period 2008-2014, which addresses visual representations and matching processes for exploring visual data, including instance-level visual search and clustering.

A B S T R A C T

This manuscript is about a journey. The journey of computer vision and machine learning research from the early years of Gabor filters and linear classifiers to surpassing human skills in several tasks today. The journey of the author's own research, designing representations and matching processes to explore visual data and exploring visual data to learn better representations.

Part i addresses instance-level visual search and clustering, building on shallow visual representations and matching processes. The representation is obtained by a pipeline of local features, hand-crafted descriptors and visual vocabularies. Improvements in the pipeline are introduced, including the construction of large-scale vocabularies [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF], spatial matching for geometry verification [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF], representations beyond vocabularies [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF] and nearest neighbor search [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. Applications to exploring photo collections are discussed, including location recognition, landmark recognition and automatic discovery of photos depicting the same scene [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF][START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

Part ii addresses instance-level visual search and object discovery, building on deep visual representations and matching processes, focusing on the manifold structure of the feature space. The representation is obtained by deep parametric models learned from visual data. Contributions are made to advancing manifold search over global or regional CNN representations. This process is seen as graph filtering, including spatial [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] and spectral [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF]. Spatial matching is revisited with local features detected on CNN activations [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF]. Finally, a method is introduced for object discovery from CNN activations over an unlabeled image collection [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

Part iii addresses learning deep visual representations by exploring visual data, focusing on limited or no supervision. It progresses from instance-level to category-level tasks and studies the sensitivity of models to their input. It introduces methods for unsupervised metric learning [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] and semi-supervised learning [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF], based again on the manifold structure of the feature space. It contributes to few-shot learning [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF], studying activation maps and learning multiple layers to convergence for the first time. Finally, it introduces an attack as an attempt to improve upon the visual quality of adversarial examples in terms of imperceptibility [START_REF] Zhang | Smooth Adversarial Examples[END_REF].

Part iv summarizes more of the author's past and present contributions, reflects on these contributions in the present context and consolidates the ideas exposed in this manuscript. It then attempts to draw a road map of ideas that are likely to come.

ix P U B L I C AT I O N S

The contributions presented in this manuscript have appeared previously in the following articles. All articles are available online. For easy access, the titles are hyperlinked.

"If I have seen further it is by standing on the shoulders of giants."

-Sir Isaac Newton A C K N O W L E D G E M E N T S

I am grateful to Patrick Pérez, Horst Bischof and Gabriela Csurka Khedari for accepting to review my manuscript and for returning a report that felt like a reward after a long period of hard work. I am also grateful to Eric Marchand for being the president and the only member of my jury that was physically present at my defense in Rennes, in July 3, 2020. Many thanks to Rémi Gribonval, Jiri Matas, Nikos Paragios and Cordelia Schmid for participating, even remotely. I could not dream of a better jury.

I owe an anonymous acknowledgement to the entire scientific community of computer vision and machine learning for such a tremendous progress during the last two decades. It would have been impossible for an individual researcher or team to work in isolation in this more and more exciting field. Giorgos Tolias and Yannis Kalantidis deserve most of my gratitude for being there, from students starting their career to well-respected scientists. I have learned so much from you and I am so proud of you. The collaboration with Christos Varytimidis and Kostas Rapantzikos has been equally important and I am so sorry our joint work did not make it into the strict selection of topics to present in this manuscript. To all other undergraduate and PhD students, post-doc researchers and engineers of the Image and Video Analysis team, it has been an amazing experience to work with you through the years. Kimon Kontosis has been so influential, despite having no joint publications. Special thanks go to Stefanos Kollias, director of the Image, Video and Multimedia Laboratory of the National Technical University of Athens, for trusting me in leading one of the pillars of the lab for more than a decade. I thank Ioannis Emiris and his team at the National and Kapodistrian University of Athens for a short period of joint work in a transition between NTUA and Inria and for the continuous collaboration over the years, until today. I am grateful for getting an entirely new perspective on supposedly well-known problems, that of the theoretical computer scientist.

Guillaume Gravier and Laurent Amsaleg have welcomed me at Inria Linkmedia team and have been instrumental in helping me integrate into a new environment in France, as well as being scientific xiii collaborators and co-advisors of PhD students, despite their other responsibilities. I am so proud to have had Oriane Siméoni, Hanwei Zhang and Yann-Raphaël Lifchitz as a first generation of PhD students in this new environment. I thank the numerous other Masters and PhD students, post-doc and permanent researchers of the Linkmedia team, including Ahmet Iscen, Teddy Furon, Ronan Sicre, Ewa Kijak, Mateusz Budnik and Miaojing Shi for their collaboration, both in the team and in their following steps. The collaboration with Ahmet, years after he moved on, is beyond words.

There are so many people I have collaborated with outside my team over the years and I am honoured to have published with more than 100 co-authors. I thank Hervé Jégou and Ondrej Chum for being the most prominent ones in the last few years, which is largely due to Giorgos. More recently, I am so happy to have worked with Filip Radenović, Zhaohui Yang, Andrei Bursuc, Sylvaine Picard, Julien Rabin, Frédéric Jurie, Cordelia Schmid and Vittorio Ferrari.

I thank Giorgos and Maria for going through the effort to provide feedback and partially proof-read my manuscript, respectively. I would not be who I am without the support and love of my family. Maria, Savvas, Apostolos, it is so hard being far away. We had such a great time during Christmas 2019, while I had to complete my manuscript. There is also Savvas senior, Mary, Nena, Panos, Christina, Marina, Rodolfos, Yannis, Natassa, Angelina, Marianna, Yangos, Apostolis, Miranda. And, there is the memory of Tolis. This manuscript is about designing representations and matching processes to explore visual data and exploring visual data to learn better representations. This chapter begins by motivating such an endeavor in seeking to replicate the functionalities of the human visual system. It sets the goals of this manuscript and walks through the subjects studied in its different parts, while providing relevant background from the fields of computer vision and machine learning. Finally, it includes a more detailed structure of the manuscript, a list of chapter dependencies and a reading guide.

motivation

We are born into this world and we open our eyes. Everything is

The impatient reader wondering how to read this manuscript may want to jump to Section 1.4 right away.

blurry but we can discriminate colors from white and recognize the face of our mother within days. We develop a perception of depth a few weeks later, long before we begin crawling. Although we can focus on particular objects at two months, it is only at about six months that we can perceive sharp images. As our visual system develops, we begin to encode, store and retrieve detailed mental images of vast amounts of objects, places or people. This occurs over a time range spanning from eye movements to years. As we grow up, little recollection remains from the first few years of our life. We maintain an up-to-date mental image of beloved persons and we are at times surprised to find out in photographs what they looked like several years ago.

How do we perceive the world around us? What is the form of mental representations stored in our visual memory? How do we organize objects into abstract categories, while at the same time recognizing particular ones instantly? Is there a precise location in the brain where a particular object, place or person is stored, or is the information distributed? What mechanisms limit the capacity of our memory and dictate the evolution of stored representations over time? What is the role of other senses in the development of vision?

What would be the impact if we could unlock the mysteries of the human visual system and we had access to hardware powerful enough to replicate its functionalities, including visual memory? What if we could afford such an artificial system for every person suffering from impaired vision or memory disorders? What if such systems could be networked into a collective memory of gigantic capacity, still able to recognize instantly?

While the unlocking part is the subject of neuroscience, the replicating part is primarily the subject of computer vision and machine learning. Often taking inspiration from the former over decades, the latter two are currently undergoing a period of spectacular growth. and machine learning research from the early years of Gabor filters and linear classifiers to surpassing human skills in several tasks today. The journey of my own research, designing representations and matching processes to explore visual data and exploring visual data to learn better representations. My personal journey, exploring this vibrant scientific field and learning from my own experience and the experience of colleagues over time.

This manuscript summarizes the research I have been conducting

Contributions.

with collaborators during roughly the last 10 years, focusing more on recent work, and draws perspectives on future directions. Clearly, putting together work of such broad thematic and temporal range is challenging in terms of size, coherence and terminology. For this reason, I am focusing on a small number of selected articles that I expose to some length. A complete and updated list of my publications and professional activities is maintained online 1 . This manuscript is a requirement for the qualification of Habilitation HDR.

à Diriger des Recherches (HDR) by University of Rennes 1. According to a recommendation by Institut de Recherche en Informatique et Systèmes Aléatoires (IRISA), an HDR manuscript is a "synthetic document of 30 to 50 pages (excluding attached articles and appendices)." On one hand, a document of such length requires significant ef-Design.

fort to prepare, without giving the reader a good understanding of the presented methods. On the other hand, attaching articles has little value since articles are too detailed to be read in large quantity. Besides, they are available online. For this reason, I choose to also include a self-contained summary of each selected article, mostly one per chapter. These technical chapters are intermixed with non-technical chapters corresponding to the "synthetic document." The latter provide context and background as well as summary and positioning of the contributions.

This design, discussed in Section 1.4, serves a dual purpose: meeting the HDR expectations and providing a succinct, unified presentation of my research contributions in the broad context of developments in computer vision, machine learning and other fields of study over a period of several decades. It is thus my hope that the manuscript will be read by more people than my jury.

The manuscript consists of three technical parts, each containing

Structure.

a collection of contributions corresponding to a different period or subject; and a fourth part consolidating the contributions and drawing perspectives on future work. Section 1.3 discusses in detail the contents of the different parts. This manuscript studies solutions to similar problems in two different periods: before and after the establishment of deep learning as the dominant paradigm in representing and understanding visual data.

This provides an excellent opportunity to appreciate the impact of deep learning on the relative importance given to different research directions by the scientific community.

Even though I am the author of this manuscript, hence take full "We".

responsibility for its content, most contributions are the result of my collaboration with colleagues. For this reason, I use first-person plural from this point onward.

exploring and learning

This section serves as a mild introduction to the subjects studied in this manuscript. It introduces a road map and terminology, then guides through the different parts. Aspiring to be read easily by nonexperts, it is kept abstract and does not include references other than to our own contributions. It is essentially a high-level summary of Chapters 2, 8, 13, 18 and 19. As explained in Section 1.4, this sequence of chapters provides a non-technical synthesis of our contributions and is recommended to follow, at least at first reading.

road map We begin with representations and matching processes for exploring visual data: first based on shallow representations in Part i, then based on deep learned representations in Part ii. We then turn to learning visual representations by exploring visual data in Part iii, progressing towards category-level tasks. A discussion of past, present and future ideas is attempted in Part iv.

By exploring we refer to processes that are linear in the data size,

Exploring.

like searching for (or retrieving) the most similar images within a dataset to a given image; and processes that are quadratic, like clustering or discovering objects within a dataset. These processes rely on the definition of visual similarity between two images. By representations we refer to simple, compact visual representa-

Representations.
tions, like a vector or a set of vectors per image. A representation may be learned on some training data. We focus on explicit mappings from the input image to the representation space, such that extending to images unseen during training is straightforward. Representations are expected to capture appearance and geometry, while at the same time being invariant to factors like viewpoint.

By matching processes we refer to processes for evaluating the visual Matching processes.

similarity of two images. This can be based on a simple Euclidean distance or cosine similarity in the case of one vector per image. A set of vectors per image allows for partial similarity, i.e. searching for a part of an image that is similar to a part of the other.

In instance-level tasks, two images are visually similar if they depict

Instance-level and category-level tasks.

two views of the same instance, e.g. object or scene; in category-level tasks, they are similar if they depict different instances of the same semantic category, e.g. birds or cars. The distinction refers mainly to the amount of appearance variability to which the representation should be invariant to.

exploring Part i addresses instance-level visual search and clus-This is a summary of Chapter 2.

tering, building on shallow visual representations and matching processes. The representation is obtained by a pipeline of local features, hand-crafted descriptors and visual vocabularies.

By local features we refer to a large set of small geometric primitives

Local features.

per image, like points, circles or ellipses. Those features are stable to lighting and viewpoint changes, in the sense that a geometric transformation of the image causes the same transformation on the features. Splitting images into many small pieces and finding which pieces they have in common is an efficient way to measure partial similarity, providing robustness to occlusion and background clutter.

By visual descriptor we refer to a vector representation of an im-Visual descriptors.

age or part of an image. Local descriptors are obtained on rectangular patches aligned with local features. Descriptors are stable to lighting changes and discriminative in the sense that proximity in the descriptor space reliably indicates correspondence of the associated features. Hand-crafted descriptors commonly take the form of orientation histograms, inspired by biological visual systems. Hand-crafted, introduced in retrospect, means not learned.

By visual vocabulary we refer to a set of descriptors learned with-Visual vocabularies.

out supervision from visual data, which allows the formation of a second layer of representation on top of visual descriptors. This representation takes again the form of histograms in the high-dimensional descriptor space. It is a compact vector representation of regions or entire images, invariant to geometric configuration, but not very discriminative. While vocabularies are learned, this representation is still referred to as shallow, a term introduced in retrospect as opposed to deep, i.e., consisting of more layers.

We introduce improvements to this pipeline: a mechanism to learn Improving representations and matching.

large-scale vocabularies that automatically adjusts the size of the vocabulary to the data [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF]; an extremely fast spatial matching process to identify common objects between two views, allowing for multiple objects or deformation [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF]; and a high-dimensional representation beyond vocabularies that is very discriminative [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

Using the same representation pipeline, we also introduce improve-

Exploring visual data.
ments in exploring visual data: a compression and indexing scheme that adapts to the distribution in the descriptor space and allows for fast search in the compressed domain at a very large scale [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]; and a clustering scheme that finds multiple views of the same scene, builds a joint representation without supervision and uses this representation to improve search [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF]. The latter is applied to exploring photo collections, supporting location and landmark recognition [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

exploring deeper Part ii addresses instance-level visual search

This is a summary of Chapter 8.

and object discovery, building on deep visual representations and matching processes, focusing on the manifold structure of the feature space. The representation is obtained by deep parametric models learned from visual data. By deep parametric models we refer to functions mapping images Deep parametric models.

to a representation space, composed of several linear and nonlinear operations, determined by a significant amount of parameters. The most common is a Convolutional Neural Network (CNN), where each operation is called a layer. Operations apply densely, resulting in 3dimensional activation tensors that can be seen as one visual descriptor per location. Gradually, the dimension of the representation increases, while the spatial resolution decreases, introducing invariance to deformation. This is reminiscent of the encoding and pooling steps of histograms in shallow models. By learning we refer to the process of determining the parameters Learning and optimization.

of a deep model, by jointly optimizing an objective function on a target task on some training data. The most common task is supervised classification, where the model maps directly to a vector of class scores and the objective is optimal when the maximum score is attained for a given class label per image. At large scale, the most common form of optimization is Stochastic Gradient Descent (SGD), relying on firstorder derivatives only. These derivatives are computed efficiently by automatic differentiation, a dynamic programming approach known as back-propagation in this context. The difficulties lie in optimizing in the absence of higher-order derivatives and in designing operations such that propagated derivatives are "stable". One of the interesting aspects of deep models is that, regardless of

Learning for instance-level search.

the initial task, representations obtained at intermediate layers may be re-used for or adapted to new tasks, where less training data is available; the latter is called transfer learning. For instance-level search, it is common to obtain one or more feature vectors2 per image by spa-Features.

tial pooling of activation maps, globally or over rectangular regions respectively. Alternatively, one may obtain a large set of local descriptors by some form of sampling.

The new target task may still be supervised classification at instance level: All views of an object belong to the same class. Importantly, the classes are different at training and testing; hence at least the last layer, a parametric classifier, is discarded. Alternatively, it may be a form of metric learning, where the objective function is defined on two or more images and is optimal when the ranking obtained by the representations is correct. Most importantly, the supervision may be provided by existing hand-crafted mechanisms.

The power of deep representations lies in that one or few vectors Manifold search.

per image are discriminative enough to outperform shallow representations consisting of thousands of vectors per image. This allows us to explore the global manifold structure of the feature space by means of nearest-neighbor graphs and search beyond Euclidean distance, according to manifold similarity. We introduce a number of improvements that make this search process practical at large scale and make con-nections to Graph Signal Processing (GSP), casting search as graph filtering [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

When searching for small objects in large images, partial similarity is Spatial matching revisited.

still needed. In this case, a representation consisting of thousands of deep learned local descriptors is even more discriminative and allows for spatial matching much like shallow counterparts. We observe that local features may be obtained simply at local maxima over the 3d activation maps without training and we use such local features for spatial matching, even without descriptors [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF].

An alternative approach to partial similarity is to identify which Object discovery.

regions of an image may correspond to objects of interest and focus attention to those regions only, suppressing background clutter. This is only feasible using the entire image collection to be searched. We introduce a nonparametric attention mechanism that is learned without supervision on the target collection, capturing discriminative and frequent patterns according to activation maps [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

learning Part iii addresses learning visual representations by This is a summary of Chapter 13.

exploring visual data, focusing on solutions assuming only limited or no supervision. It progresses from instance-level to category-level tasks and studies the sensitivity of models to their input. The better deep models and learning processes are understood, the more interesting things happen. Learning with less supervision becomes a priority: Representations are learned by observing visual data without human involvement, or with human supervision on tiny fractions of the data, or with noisy supervision from different sources on the Internet. Category-level and instance-level tasks converge in the sense that they are mostly treated the same way, the only difference lying in the data and annotations defining the task. New tasks appear; or, tasks of marginal interest become very popular. The research landscape changes within months.

One of the tasks resisting recent trends is metric learning. Originat-Metric learning.

ing in unsupervised nonlinear dimension reduction or manifold learning from pairs of "similar" and "dissimilar" examples, supervised metric learning arises as a task where such pairs are specified by humans.

In early deep metric learning, supervision appears as a natural choice. In most cases, two examples are defined as "similar" if they belong to the same class. But then, one wonders what is the difference from supervised classification. Building on our work on searching by manifold similarity, we intro-

Unsupervised metric learning.
duce one of the first unsupervised deep metric learning solutions [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF]. We treat manifold similarity as a teacher model and we use it to learn a representation by a student model. The goal is that the ranking obtained by the representation of the student is in agreement with the ranking obtained by manifold similarity in the representation space of the teacher. This approach is equally effective in fine-grained classification and instance-level search.

Obtaining large amounts of training data annotated by humans for

Semi-supervised learning.

every single task is both impractical and error prone. Unsupervised solutions are still inferior to solutions on data laboriously annotated by humans. Semi-supervised learning becomes increasingly important because it can combine limited data carefully labeled by humans with abundant unlabeled data.

Searching by manifold similarity is a well-known solution for semi-Inductive label propagation.

supervised learning, known as label propagation: In short, unlabeled examples receive class scores according to average manifold similarity to labeled examples of each class. This is a transductive solution, i.e. only makes predictions on the training set. We introduce an inductive version [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] shot learning [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF]. Inspired by partial similarity, we introduce a classification loss that implicitly acts as a data augmentation mechanism, essentially searching over all possible locations in an image. At the same time, inspired by mechanisms of increasing model capacity in incremental learning, we widen the last layers of the model to accommodate for new, task-specific features with reduced risk of overfitting. These solutions are complementary to some extent.

With the impressive performance of deep models and the resulting

Adversarial examples.

routine application to safety-critical tasks like driving cars, it becomes imperative to study and understand the sensitivity of these models to their input. Adversarial examples, i.e., failure cases obtained by imperceptible perturbations on legitimate input images, become extremely important in this respect.

Our smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF] are an attempt to improve

Smooth adversarial examples.

upon the visual quality of adversarial examples in terms of imperceptibility by an image-dependent smoothing process. This work lies at the heart of studying the manifold structure of visual data because natural images are piecewise smooth. Robustness to adversarial examples on this manifold is essentially generalization. Hence this study is important in learning better representations regardless of the standard threat model of adversarial examples.

beyond Part iv summarizes more of our past and present con-This is a summary of Chapters 18 and 19.

tributions, reflects on our contributions in the present context and consolidates the ideas exposed in this manuscript. It then attempts to draw a road map of ideas that are likely to come.

On one hand, improvements in visual representation and matching

From representing to exploring and back.

processes yield improved search by visual similarity and improved lenges before establishing its power. The most recent challenge has been to convince the scientific community in investing the required amounts of data, supervision and computing power. One remaining challenge may be to invest in storage capacity and to find memory mechanisms translating this capacity to better performance. We put forth a vision for future research whereby data becomes a first-class citizen in visual recognition tasks. Around this vision, we build a number of research directions, all on learning visual representations from data with limited supervision and applying them to visual recognition tasks. The most challenging is learning while memorizing, a form of instance-level incremental learning.

how to read

We assume a basic background on computer vision and machine learn-Assumed background and structure.

ing. Other than that, this manuscript can be read by a non-expert. We include the necessary background to make the text as self-contained as possible. The text is organized in a modular structure, allowing reading in different ways. Figure 1.1 shows the chapter dependencies.

For instance, apart from top-down, one may read Part i, Part ii or Part iii in any order, as long as a few chapters are read first. It is recommended, at least at first reading, to follow the sequence of chapters indicated in green. Apart from the current introductory chapter, this comprises one 'outline' chapter at the beginning of each of the first three parts, including context, background, our contributions and the structure of the text; and the two chapters of Part iv, providing a general synthesis and drawing perspectives beyond this work. This content is high-level and non-technical. In fact, it corresponds to the "synthetic document of 30 to 50 pages (excluding attached articles and appendices)" that is recommended by IRISA.

The background material included in the 'outline' chapters is by no means complete; it is selected according to importance, personal preference and relevance to the subjects studied in each part.

The remaining chapters assume reading of the 'outline' chapter of Technical chapters and publications.

the corresponding part. Each of these chapters-except Chapter 9-is essentially a short version of one or two clearly indicated publications; the association of chapters to publications also appears at the end of the corresponding 'outline' chapters. This text is technical but includes enough background to be self-contained. The material is presented in a unified treatment and unified notation to some extent. The intent is that in this way, the reader may get a good understanding of the methods exposed without having to go through all details of the original publications.

In particular, although these chapters include experimental results, Experiments and related work.

these are only a tiny part of the original results and are meant only to illustrate the main achievements of the corresponding methods. Also, these chapters do not include a related work section. Instead, the 'outline' chapters of the corresponding part provide a much broader historical and more recent background and context to allow positioning of our contributions in light of the current state of affairs. These chapters may in general be read in any order, with the ex-Dependencies.

ception of dependencies indicated in Figure The current version 0.4, released in October 2020, is the first to be made public. It includes the addition of a title page with the jury, acknowledgements, as well as this section on the document's history. It also includes minor revisions made since March 2020. Further revisions may follow.

Part I E X P L O R I N G

Building on hand-crafted local descriptors and visual vocabularies, we study visual representations and matching processes for exploring visual data, including instancelevel visual search and clustering, as well applications to exploring photo collections.

O U T L I N E

This chapter serves as an outline or road map of Part i. We present historical and more recent background on hand-crafted local features and descriptors as well as visual vocabularies developed in the 2000s. In this context, we position our own contributions developed in 2008-2014. Building on such methods, our work addresses visual representations and matching processes for exploring visual data, including instance-level visual search and clustering, as well as applications to exploring photo collections. We discuss common evaluation measures and outline the structure of Part i in terms of methods, key publications and chapters.

context

The work of Lowe known as Scale-Invariant Feature Transform (SIFT) in 1999 [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF][START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF] is arguably a landmark of computer vision research in the 2000s. Even if none of the ideas is entirely new, Lowe puts together three elements that make visual recognition "really" work under occlusion, clutter, lighting and viewpoint changes:

1. a sparse local feature detector, building on scale-space theory [START_REF] Witkin | Scale-Space Filtering[END_REF] and Lindeberg's automatic scale selection [START_REF] Lindeberg | Feature Detection with Automatic Scale Selection[END_REF];

2. a local descriptor in the form of orientation histograms, as an approximation of Gabor filter banks [START_REF] John G Daugman | Complete Discrete 2-D Gabor Transforms By Neural Networks for Image Analysis and Compression[END_REF][START_REF] Mark R Turner | Texture Discrimination By Gabor Functions[END_REF] followed by spatial pooling [START_REF] Bs Manjunath | Texture Features for Browsing and Retrieval of Image Data[END_REF][START_REF] Oliva | Global Semantic Classification of Scenes Using Power Spectrum Templates[END_REF]; and

3. a spatial matching process using a variant of the Generalized Hough Transform (GHT) [START_REF] Ballard | Generalizing the Hough Transform to Detect Arbitrary shapes[END_REF], where each feature correspondence between two views casts a single vote.

Considering local descriptors as a first layer in a visual representation pipeline, the Bag of Words (BoW) model [START_REF] Csurka | Visual Categorization With Bags of Keypoints[END_REF][START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] builds a second layer that simplifies and accelerates several visual recognition tasks. With few adaptations in terms of local feature sampling and the size of visual vocabularies, this pipeline is applied to tasks including both instance-level (different views of the same object or scene) and categorylevel (different objects or scenes in the same category).

In this context, Part i presents part of our work carried out in the 1. improvements in the pipeline, including the construction of large scale visual vocabularies [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF], spatial matching for geometry verification [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF], representations beyond vocabularies [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF] and nearest neighbor search [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]; and outline 2. applications to exploring photo collections, including location recognition, landmark recognition, automatic discovery and visualization of photos depicting the same scene [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF][START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

Importantly, as a result of discovering photos depicting the same scene, we improve the representation of each photo itself.

background and contributions local features Dense image registration, studied by Lucas and

Why local features?

Kanade in 1981 [START_REF] Lucas | An Iterative Image Registration Technique With an Application to Stereo Vision[END_REF], is a low-level vision task, where for each location in an image, the objective is to find a displacement with respect to another reference image. It is appropriate for small displacements, e.g. in stereopsis or optical flow. However, not all locations are equally good for estimating displacement, which is known as the aperture problem. By formulating registration as a least squares problem, Tomasi and Kanade find in 1991 [START_REF] Tomasi | Detection and Tracking of Point Features[END_REF] that it is more reliable to detect and track a sparse set of point features. The criteria are similar to corner detection, as studied by Harris and Stephens in 1988 [155].

In wide-baseline matching, studied by Pritchett and Zisserman in 1998 [START_REF] Pritchett | Wide Baseline Stereo Matching[END_REF], an object or scene is seen from two arbitrary viewpoints, so every part of one view may be seen anywhere in the other. Objects may be largely occluded or seen at very different scales. In this more challenging problem, it makes even more sense to focus on a sparse set of local features. In establishing correspondences between the two views, it also makes sense to attach to local features more geometric information like scale and orientation, as well appearance information like visual descriptors.

The Laplacian of Gaussian (LoG) kernel plays a central role in the the-How to detect them?

ory of edge detection by Marr and Hildredth in 1980 [START_REF] Marr | Theory of Edge Detection[END_REF]. It serves as a model of retinal cells, with a hypothesized biological implementation as a Difference of Gaussians (DoG) having a scale ratio close to 1.6. Following studies by Witkin in 1987 [START_REF] Witkin | Scale-Space Filtering[END_REF], Lindeberg shows in 1994 [START_REF] Lindeberg | Scale-Space Theory: A Basic Tool for Analyzing Structures at Different Scales[END_REF] that the Gaussian kernel and its derivatives are the only possible smoothing kernels for scale-space analysis and uses them in 1998 [START_REF] Lindeberg | Feature Detection with Automatic Scale Selection[END_REF] to detect local features with automatic scale selection. In this context, the LoG is an idealized blob-like pattern that is searched for exhaustively over all possible locations and scales.

The SIFT detector by Lowe in 1999 [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF][START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF] is such an LoG blob detector, approximated by a DoG with a scale ratio of exactly 1.6. SIFT features are equipped with coordinates, scale and orientation. On one hand, further approximations target faster implementations, for instance Speeded-up Robust Features (SURF) [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] approximate Gaussian derivatives by integral images [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF]. On the other hand, improved repeatability under affine transforms is sought, giving rise to affine local features [START_REF] Mikolajczyk | A Comparison of Affine Region Detectors[END_REF]. Based on iterative affine shape adaptation [START_REF] Lindeberg | Shape-Adapted Smoothing in Estimation of 3-D Shape Cues From Affine Deformations of Local 2-D Brightness structure[END_REF], the Hessian affine detector [START_REF] Mikolajczyk | Scale & Affine Invariant Interest Point Detectors[END_REF] becomes the most popular.

visual descriptors Following the experiments by Hubel and Wiesel in 1959 [START_REF] David | Receptive Fields of Single Neurones in the Cat's Striate Cortex[END_REF] and subsequent neuroscience results [START_REF] Marcelja | Mathematical Description of the Responses of Simple Cortical Cells[END_REF][START_REF] Anthony Movshon | Spatial Summation in the Receptive Fields of Simple Cells in the Cat's Striate Cortex[END_REF], 2d Gabor filters are introduced by Daugman in 1985 [START_REF] John G Daugman | Uncertainty Relation for Resolution in Space, Spatial Frequency, and Orientation Optimized By Two-Dimensional Visual Cortical Filters[END_REF] and confirmed in 1987 [START_REF] Jones | An Evaluation of the Two-Dimensional Gabor Filter Model of Simple Receptive Fields in Cat Striate Cortex[END_REF] as a mathematical model for the receptive fields of simple cortical cells. Gabor filter banks are used thereafter for texture analysis [START_REF] John G Daugman | Complete Discrete 2-D Gabor Transforms By Neural Networks for Image Analysis and Compression[END_REF][START_REF] Mark R Turner | Texture Discrimination By Gabor Functions[END_REF] and generic visual representation [START_REF] Bs Manjunath | Texture Features for Browsing and Retrieval of Image Data[END_REF][START_REF] Oliva | Global Semantic Classification of Scenes Using Power Spectrum Templates[END_REF].

Because Gabor filters are orientation-sensitive, a filter bank acts

Gabor filters and local scale/orientation histograms.

as an encoding, effectively assigning each location in an image to an orientation at that location (and similarly for scale). When followed by spatial pooling (or aggregation), this is roughly equivalent to a histogram over orientations (or scales). Pooling, hence histograms, can be global over the image [START_REF] Bs Manjunath | Texture Features for Browsing and Retrieval of Image Data[END_REF][START_REF] Oliva | Global Semantic Classification of Scenes Using Power Spectrum Templates[END_REF] or regional [START_REF] Oliva | Building the Gist of a Scene: the Role of Global Image Features in Recognition[END_REF].

The SIFT descriptor by Lowe in 1999 [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF][START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF] is essentially a shortcut, where the orientation at each location is determined directly by the local gradient and histograms are computed directly, without Gabor filters. Despite several competitors [START_REF] Mikolajczyk | A Performance Evaluation of Local Descriptors[END_REF], the SIFT descriptor dominates in most vision tasks until learned representations take over. In modern terminology, all such histogram-like descriptors are called hand-crafted or shallow in retrospect (as opposed to learned or deep) and are only the first layer in a visual representation pipeline.

Local descriptors, i.e. descriptors computed at patches aligned with

Instance-level and category-level tasks.

local features at the appropriate scale and orientation, provide a powerful mechanism for establishing correspondences between two views of the same object or scene. This mechanism is robust to occlusion, clutter, lighting and viewpoint changes, and opens the door to solving several instance-level tasks including object recognition, retrieval and 3d reconstruction. The Histogram of Oriented Gradients (HOG) [START_REF] Dalal | Histograms of Oriented Gradients for Human Detection[END_REF] is another variant, where the descriptor is computed densely over a grid. As confirmed independently using visual vocabularies [START_REF] Fei | A Bayesian hierarchical model for learning natural scene categories[END_REF][START_REF] Jurie | Creating Efficient Codebooks for Visual Recognition[END_REF], this choice is superior to using a local feature detector in category-level tasks, including scene categorization and object detection. The situation is the same in concurrent models inspired by the visual cortex [START_REF] Mutch | Multiclass Object Recognition With Sparse, Localized features[END_REF][START_REF] Serre | Object Recognition with Features Inspired By Visual Cortex[END_REF], which also maintain filter banks like modern architectures.

We focus on instance-level tasks, adopting local features and hand-crafted descriptors in Part i and learned global, regional or local representations in Part ii. Then, Part iii addresses learning representations for both instance-level and categorylevel tasks.

visual vocabularies In modern terminology, visual vocabular-

What are they? ies (or codebooks) are used to define a second layer in a visual representation pipeline on top of, usually hand-crafted, visual descriptors. Learned without supervision by vector quantization of a collection of visual descriptors, a visual vocabulary acts as a set of bins over which histograms may be computed in the visual descriptor space. An image is then represented by encoding (assigning) its visual descriptors outline to bins, called visual words, and then pooling (aggregating) them into a global histogram vector, called Bag of Words (BoW).

The term BoW originates in a linguistic context [START_REF] Zellig | Distributional structure[END_REF], where words

Why do we need them?

are naturally discrete. In the first layer of visual representation, histograms over scales and orientations are achieved by scalar quantization, resulting in a 2d grid of bins. In fact, the Gabor filter bank corresponds to a log-polar sampling grid in the frequency domain [START_REF] John G Daugman | Complete Discrete 2-D Gabor Transforms By Neural Networks for Image Analysis and Compression[END_REF]. By contrast, the visual descriptors handled in the second layer are E.g. 18d for a Gabor filter bank of 3 scales and 6 orientations or 128d for a SIFT descriptor.

high-dimensional. Histograms over this high-dimensional vector space would be intractable without vector quantization. We are essentially learning the distribution of visual descriptors of natural images, such that we are not left with countless empty bins. An early form of visual vocabulary on Gabor filter bank responses sampled densely over a single image is suggested by Daugman in 1988 [START_REF] John G Daugman | Complete Discrete 2-D Gabor Transforms By Neural Networks for Image Analysis and Compression[END_REF], applied to texture segmentation. The same idea with the same application is given the name textons by Malik et al. in 1999 [281], defining precisely what was described by Julesz in 1981 [START_REF] Julesz | Textons, the Elements of Texture Perception, and Their Interactions[END_REF] as "basic elements of pre-attentive human texture perception."

Vocabularies on local descriptors follow, applied to category-level [START_REF] Csurka | Visual Categorization With Bags of Keypoints[END_REF] and instance-level [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] recognition. The former are small (coarse) E.g. few thousands [START_REF] Van Gemert | Kernel Codebooks for Scene Categorization[END_REF].

to compensate for in-class variations, resulting in compact representations that can be used e.g. by a Support Vector Machine (SVM) for object categorization. Descriptors sampled densely over a grid eventually replace local descriptors in this case [START_REF] Fei | A Bayesian hierarchical model for learning natural scene categories[END_REF][START_REF] Jurie | Creating Efficient Codebooks for Visual Recognition[END_REF]. The latter are large (fine) to maintain discriminative power, resulting in sparse im-E.g. even a million [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF].

age representations. Borrowing ideas from text retrieval including Term Frequency (TF)-Inverse Document Frequency (IDF) [START_REF] Baeza | Modern information retrieval[END_REF] and inverted files [START_REF] Ian H Witten | Managing gigabytes: compressing and indexing documents and images[END_REF], instance-level search becomes very efficient [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF].

Vocabularies for category-level tasks commonly use a Gaussian Mixture Model (GMM) [START_REF] Farquhar | Improving bag-of-keypoints image categorisation: Generative models and pdf-kernels[END_REF][START_REF] Perronnin | Adapted Vocabularies for Generic Visual Categorization[END_REF], learned by the Expectation-Maximization (EM) algorithm [START_REF] Arthur P Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF]. For instance-level tasks, learning at a scale of e.g. one million visual words is mostly constrained to variants of kmeans [START_REF] Gray | Vector quantization[END_REF]. Our Approximate Gaussian Mixture (AGM) [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] is the first Our Approximate Gaussian Mixtures (AGMs) [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF]. attempt to apply a GMM at this scale, employing Approximate Nearest Neighbor (ANN) search in EM. The size of the vocabulary is dynamically estimated and AGM only needs to run once. This is extremely important because for each size k of vocabulary conventionally tested, one needs to not only learn the vocabulary, but also re-index a very large collection of images to evaluate performance.

spatial matching The term 'bag' in BoW refers to dropping all

From 'bags' back to local features.

geometrical information from local features and keeping only appearance information as represented by visual descriptors. This makes visual representation completely invariant to geometric transformations, but naturally drops discriminative power as well.

Establishing correspondences between two views of the same object or scene can be much more informative than the similarity of two histograms. However, correspondences found by pairwise matching of visual descriptors are noisy, even more so when descriptors are quantized. True correspondences, called inliers, may be found by measuring consistency with a rigid geometric transformation model having a given number of Degrees of Freedom (DoF). Unfortunately, es-E.g., 2 DoF for translation, 4 for similarity, 6 for affine transformation.

tablishing a transformation in turn depends on the correspondences and is challenging when the inliers are only a small percentage.

RANdom SAmple Consensus (RANSAC), introduced by Fischler and Bolles in 1981 [START_REF] Fischler | Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography[END_REF], is a robust estimator that iteratively evaluates transformation hypotheses by sampling the minimum required number of correspondences depending on the DoF, fitting the model and measuring inliers. Alternatively, the Generalized Hough Transform (GHT), introduced by Ballard in 1981 [START_REF] Ballard | Generalizing the Hough Transform to Detect Arbitrary shapes[END_REF], finds promising transformations by casting a set of votes for each correspondence and performing mode seeking in the transformation space. Both are problematic when the number of DoF is large and RANSAC is even more so when the percentage of inliers is small.

The easiest case is when a transformation hypothesis is possible Single correspondence hypotheses.

from a single correspondence (for RANSAC), or there is a single vote cast for every correspondence (for GHT). For point correspondences (e.g. corners), this happens for 2-DoF translation only. This is used e.g. for category-level recognition with vocabularies and GHT [START_REF] Leibe | Interleaved Object Categorization and segmentation[END_REF]. More interestingly, it also happens for more DoF when local features are equipped with additional geometric information. For instance, SIFT (resp. Hessian affine) feature correspondences give rise to transformations of 4 (resp. 5) DoF, which is used for instance-level recognition with GHT by Lowe [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF][START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF] (resp. with RANSAC by Philbin et al. [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF]). Both solutions require inlier verification and are therefore quadratic in the number of correspondences.

In instance-level image retrieval, a query image needs to be matched against millions others. Typically, this happens by a first stage of filtering according to BoW using inverted files, followed by a second stage of more expensive geometry verification, only on a short list of topranking images of the first stage. To accelerate the latter, our Hough Pyramid Matching (HPM) [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] uses GHT alone, without inlier verifi-Our Hough Pyramid Matching (HPM) [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF]. cation, and is linear in the number of correspondences, reaching thousands of image matches per second. HPM accommodates for multiple surfaces or flexible objects, improving accuracy over any rigid motion model, including homography.

beyond vocabularies The BoW model is praised for its effi-From 'words' back to descriptors.

ciency, but vector quantization is found detrimental in category-level tasks [START_REF] Boiman | In Defense of Nearest-Neighbor Based Image Classification[END_REF], because the most informative descriptors are the most infrequent, leading to high quantization error. Ideally, we would like to approximate the optimal descriptor correspondences. The Pyramid Match Kernel (PMK) [START_REF] Grauman | The Pyramid Match Kernel: Discriminative Classification With Sets of Image Features[END_REF][START_REF] Grauman | The Pyramid Match Kernel: Efficient Learning with Sets of Features[END_REF] attempts this by using a hierarchical partition of the descriptor space. However, using the true descriptors is superior [START_REF] Boiman | In Defense of Nearest-Neighbor Based Image Classification[END_REF].

Similarly, in instance-level retrieval, there is a progression from hierarchical vocabularies [START_REF] Nistér | Scalable Recognition With a Vocabulary Tree[END_REF] to flat, fine vocabularies [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], then to Multiple Assignment (MA) of descriptors to visual words [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF], then outline to even finer vocabularies equipped with visual synonyms, giving rise to visual words of arbitrary shape [START_REF] Mikulík | Learning a fine vocabulary[END_REF]. However, if one can afford the required space, using actual compressed descriptors, for instance with Hamming Embedding (HE) [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF], is superior.

At the same time, pooling (aggregation) mechanisms beyond his-Global pooling.

tograms are sought. The idea is to pool vectors rather scalars (frequencies) for each visual word, resulting in a very high-dimensional representation, which is followed by dimension reduction by Principal Component Analysis (PCA). Such models are Fisher vectors [START_REF] Csurka | Fisher Vectors: Beyond Bag-of-Visual-Words Image Representations[END_REF][START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF][START_REF] Perronnin | Fisher Kernels on Visual Vocabularies for Image Categorization[END_REF] and the Vector of Locally Aggregated Descriptors (VLAD) [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF][START_REF] Jégou | Aggregating local descriptors into compact codes[END_REF], applied to both instance-and category-level tasks. However, performance in instance-level tasks with small vocabularies is poor. We are the first to explore matching mechanisms like HE and pooling mechanisms like VLAD with large vocabularies for instance-level retrieval, developing a common model that incorporates both as special cases. In doing so, we borrow ideas from both models to introduce the Aggregated Selective Match Kernel (ASMK) [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF]. This Our Aggregated Selective Match Kernel (ASMK) [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

model achieves the last known state of the art before the advent of deep learning. It is still used by current state of the art methods using modern learned representations [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF][START_REF] Teichmann | Detect-to-retrieve: Efficient regional aggregation for image search[END_REF].

nearest neighbor search With the use of encoded descriptors, the problem of image retrieval boils down to ANN search in high dimensions, where points (descriptors) are compressed. One of the most successful search methods in the compressed domain is Product Quantization (PQ) [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF], which decomposes the space into a Cartesian product of subspaces and independently applies vector quantization to each. An improvement is Optimized Product Quantization (OPQ) [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF], which additionally optimizes subspace decomposition via an orthogonal transformation. However, the distribution of images in the descriptor space is arbitrary. We cannot expect to fit this distribution with a grid of points, not even a rotated one. Our Locally Optimized Product Quantization (LOPQ) [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF] applies the above ideas locally, finding an optimal sub-Our Locally Optimized Product Quantization (LOPQ) [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF].

space decomposition, rotation and quantizer for each region in the descriptor space. Like local PCA [START_REF] Kambhatla | Dimension Reduction By Local Principal Component Analysis[END_REF], this model can fit distributions exhibiting manifold structure.

LOPQ remains for several years the state of art on a challenging dataset of one billion SIFT descriptors. In 2017, using a CNN image representation, Yahoo! Research chooses LOPQ to index and provide a similar image search1 functionality on its entire Flickr2 collection, consisting of more than 10 billion images.

exploring photo collections An interesting application of methods discussed so far, is to datasets originating from community photo collections, in particular depicting outdoor urban scenes. Build-ings have typically a level of detail for local descriptors to be discriminative enough and non-deformable structure such that different views are related by a rigid transformation model. Such datasets are frequently accompanied by additional, non-visual information, like tags or geo-tags. In many cases they focus on landmarks [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF][START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF], in which case they exhibit high redundancy.

Given a query image, search by visual similarity against a dataset

Applications.

accompanied by geo-tags gives rise to location recognition [START_REF] Steinhoff | How Computer Vision Can Help in Outdoor Positioning[END_REF][START_REF] Zhang | Image Based Localization in Urban Environments[END_REF], which can work reliably up to city scale [START_REF] Schindler | City-Scale Location Recognition[END_REF]. Early efforts on landmark recognition [START_REF] Crandall | Mapping the World's Photos[END_REF][START_REF] Kennedy | How Flickr Helps Us Make Sense of the World: Context and Content in Community-Contributed Media Collections[END_REF] are mostly based on metadata, with the contribution of visual features being small. Visual clustering [START_REF] Chum | Large-Scale Discovery of Spatially Related Images[END_REF][START_REF] Simon | Scene Summarization for Online Image Collections[END_REF] is a popular way to discover landmark images without supervision. It can be made more efficient by performing geographic clustering first [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF][START_REF] Quack | World-Scale Mining of Objects and Events From Community Photo Collections[END_REF], assuming geo-tags. Structure-from-Motion (SfM) allows for visionbased reconstruction and navigation of a 3d scene [START_REF] Snavely | Photo Tourism: Exploring Photo Collections in 3D[END_REF], which can again work at city scale [4].

Most such applications are limited to clusters of popular images like landmarks and, even if they use efficient indexing by inverted files, they do not use the mined information to improve the indexing itself. We rather perform visual clustering by Kernel Vector Quanti-Our scene maps [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF]. zation (KVQ) [START_REF] Tipping | A Kernel Approach for Vector Quantization with Guaranteed Distortion Bounds[END_REF], guaranteeing that isolated images are preserved, and that all images in a cluster share at least a rigid object or surface with one particular reference image. By projecting them on that image plane, we then construct a scene map [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF] for each cluster. Now, indexing scene maps instead of individual images not only saves space, but increases recall performance as well.

Based on these ideas, our application Visual Image Retrieval and Localization (VIRaL)3 [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF] supports automated location estimation and Our application VIRaL [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF], which has been online since 2008.

geo-tagging, recognition of landmarks and points of interest, and visualization of photo clusters and tourist paths on an online map. evaluation Several tasks including retrieval, metric learning and object detection are evaluated by mean Average Precision (mAP). Another evaluation measure for retrieval is mean Precision (mP) at k. Because most chapters in Part i and Part ii, as well as Chapter 14 in Part iii, experiment on image retrieval, we define both here.

Average Precision (AP) [START_REF] Manning | Introduction to Information Retrieval[END_REF] evaluates ranking: Given a ranked list of items, it evaluates to what extent a set of positive items are ranked first. Informally, AP is the area under the precision-recall curve, where precision (resp. recall) at every position in the list is the ratio of retrieved positive to retrieved (resp. positive) items. Then, mAP is the average AP over a number of queries (for retrieval or metric learning) or classes (for object detection). This means that for every query (or class), we need to know a corresponding set of positive items.

Formally, let X := {x 1 , . . . , x n } be an ordered list of items and

We write

[n] = {1, . . . , n} for n ∈ N.

X k := {x 1 , . . . , x k } for k ∈ [n], with |X k | = k.
Let also P ⊆ X be a outline subset of positive elements of X; the remaining elements in X \ P are negative. For k ∈ [n], precision at k of X relative to P is defined as

p k (X; P ) := |X k ∩ P | |X k | (2.1) = 1 k k i=1 1 P (x i ).
(2.2)

Then, the AP of X relative to P is defined as

1A : X → {0, 1} is the indicator function of A ⊆ X: for x ∈ X, 1A(x) is 1 if x ∈ A
and 0 otherwise.

AP(X; P

) := 1 |P | n k=1 1 P (x k )p k (X; P ). (2.3)
AP satisfies 0 ≤ AP(X; P ) ≤ 1, with AP(X; P ) = 1 iff X |P | = P , that is, all elements of P are ranked before elements of X \ P . Mean Precision (mP) at k, denoted by mP@k, is the average, over a number of queries, of precision at k. This measures the proportion of top k retrieved items that are positive, for small k. For instance, our Revisited Oxford and Paris (RevOP) image retrieval benchmark [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] includes mP@10 as a standard evaluation measure along with mAP.

structure

Chapter 3 deals with building large-scale visual vocabularies for BoW models, based on the Gaussian Mixture Model (GMM). It presents our Expanding Gaussian Mixture (EGM) model and its approximate version Approximate Gaussian Mixture (AGM) [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF].

Chapter 4 addresses spatial matching for geometry verification, based on the Generalized Hough Transform (GHT). It presents our Hough Pyramid Matching (HPM) [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF].

Chapter 5 revisits visual representations and discusses solutions beyond BoW, based on local descriptors. It builds a common framework for previous models and introduces our Aggregated Selective Match Kernel (ASMK) [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

Chapter 6 goes beyond visual search to the more generic problem of Approximate Nearest Neighbor (ANN) search, based on quantization. It discusses our Locally Optimized Product Quantization (LOPQ) [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF].

Finally, Chapter 7 addresses the more challenging problem of exploring photo collections to automatically discover groups of photos depicting the same scene. It presents our scene maps representation [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF] and discusses Visual Image Retrieval and Localization (VIRaL), our online application for location and landmark recognition [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

V I S U A L V O C A B U L A R I E S

We introduce a clustering method that combines the flexibility of Gaussian mixtures with the scaling properties needed for visual vocabularies in image retrieval [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF]. It is a variant of Expectation-Maximization (EM) that dynamically estimates the number of components. We employ approximate nearest neighbor search to speed-up the E-step and exploit its iterative nature to make search incremental, boosting speed and precision.

introduction

The k-means algorithm is one of the most popular in learning visual vocabularies, or codebooks, needed by the Bag of Words (BoW) model. For image retrieval, fine vocabularies are needed, e.g. 10 6 visual words. Clustering options are limited at this scale, with the most common being variants like Approximate k-Means (AKM) [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and Hierarchical k-Means (HKM) [START_REF] Nistér | Scalable Recognition With a Vocabulary Tree[END_REF].

The Gaussian Mixture Model (GMM), along with Expectation-Maximi-

In addition to position, GMM models cluster population and shape.

zation (EM) [START_REF] Arthur P Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] learning, is a generalization of k-means that has been applied to class-level recognition [START_REF] Perronnin | Universal and Adapted Vocabularies for Generic Visual Categorization[END_REF]. It assumes pairwise 'interaction' of all points with all clusters and is slower to converge. By contrast, a point is assigned to the nearest cluster via Approximate Nearest Neighbor (ANN) search in AKM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF].

In Robust Approximate k-Means (RAKM) [START_REF] Li | Large-Scale Robust Visual Codebook Construction[END_REF], the nearest neighbor in one iteration is re-used in the next, with less effort being spent for new neighbor search. This motivates us to keep a larger number m of nearest neighbors. Thus, enough information is available for an Approximate Gaussian Mixture (AGM) [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] As shown in Figure 3.1, we compute the overlap of neighboring clusters and purge the ones that appear redundant, after each EM iteration. This algorithm, Expanding Gaussian Mixture (EGM) [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF], can dynamically estimate the number of clusters by starting with a large number and purging along the way. Focusing on spherical Gaussians, Contrary to typical GMM, indexed descriptors are assigned only to the nearest visual word.

we apply its approximate version, AGM, to large scale visual vocabulary learning for image retrieval.

gaussian mixtures

The density p(x) of a Gaussian mixture distribution is a convex combination of k d-dimensional normal densities or components,

p(x) := k j=1 π j N (x|µ j , Σ j ) (3.1)
for x ∈ R d , where π j , µ j , Σ j are the mixing coefficient, mean and covariance matrix respectively of the j-th component. Interpreting π j as the prior probability p(j) of component j, quantity

γ j (x) ← π j N (x|µ j , Σ j ) k =1 π N (x|µ , Σ ) , (3.2)
expresses the posterior probability p(j|x) given observation x ∈ R d . We say that γ j (x) is the responsibility of j for x. Now, given a set of i.i.d. observations {x 1 , . . . , x n }, the Maximum Likelihood (ML) estimate for the parameters of each component j ∈ [k] is [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF] We write [n] = {1, . . . , n} for n ∈ N.

π j ← n j n (3.3) µ j ← 1 n j n i=1 γ ij x i (3.4) Σ j ← 1 n j n i=1 γ ij (x i -µ j )(x i -µ j ) , (3.5)
where γ ij ← γ j (x i ) for i ∈ [n], and n j ← n i=1 γ ij is the effective number of points assigned to component j. The EM algorithm is an iterative process: Given an initial set of parameters, compute responsibilities γ ij according to (3.2) (E-step); then, re-estimate parameters according to (3.3)-(3.5), keeping responsibilities fixed (M-step).

Here we focus on the particular case of spherical (isotropic) Gaus-This is still more flexible than k-means, and efficient because σj is a scalar.

sians, with covariance matrix Σ j = σ 2 j I. In this case, update equation (3.5) reduces to an overlap measure. Purging is dynamic in the sense that it takes place during parameter learning. This idea introduces a P-step in EM, to be applied after the E-and M-steps in every iteration. A component j can be represented by the function p j determining its contribution to the GMM distribution (3.1),

σ 2 j ← 1 dn j n i=1 γ ij x i -µ j 2 . ( 3
p j (x) := π j N (x|µ j , Σ j ) (3.7) for j ∈ [k], x ∈ R d . Now, let pj is not normalized unless πj = 1.
p, q := p(x)q(x)dx (3.8) be the L 2 inner product of real-valued, square-integrable functions p, q, where the integral is over R d . The corresponding L 2 norm of function p is given by p := p, p . When p, q are normal distributions with p(x) = N (x|a, A) and q(x) = N (x|b, B) for x ∈ R d , the integral in (3.8) can be evaluated in closed form [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] p, q = N (a|b, A + B).

(3.9)

Hence, given components represented by p j , p , their overlap, as measured by inner product, is

p j , p = π j π N (µ j |µ , (σ 2 j + σ 2 )I) (3.10)
under the spherical Gaussian model, while p j 2 = π 2 j (4πσ

(x) = π 2 N (x|µ 2 , σ 2
2 ), in one dimension. When the latter is reduced to a single point, p 2 (x) collapses to δ(xµ 2 ), and inner product p 1 , p 2 is reduced to p 1 (µ 2 ). This gives rise to the P-step is outlined in Algorithm 3.1. We choose to process components in descending order of mixing coefficients, starting from the most populated cluster, which we always keep. We perform the P-step right after E-and M-steps in each EM iteration. We also constrain the E-and M-steps to components in C. Now the This behavior resembles agglomerative clustering [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction[END_REF].

number of components k ← |C| decreases in each iteration.

expanding When a component, say j, is purged, data points that were better 'explained' by j prior to purging will have to be assigned to neighboring components that remain. These components will then have to expand and cover the space populated by such points. Towards this goal, we modify (3.5) to overestimate the extent of each component Components will then tend to fill in as much empty space as possible.

as much as this does not overlap with its neighboring components. Details can be found in [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF]; here we only observe the 'space-filling' behavior of the two clusters on the left in Figure 3 What is important is to measure the performance of the resulting vocabulary in the retrieval task vs. required processing.

approximate gaussian mixtures

Counting d-dimensional vector operations, the complexity of the Eand M-steps O(nk), where k ≤ n is the current number of components, and the complexity of the P-step (Algorithm 3.1) is O(k2 ). This is not practical for large k, especially in the order of n. Similarly to [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], the approximate version of our algorithm involves indexing the entire set of clusters C according to their center µ j and performing an ANN query for each data point x i , prior to the E-step. For typical tree-based ANN search methods, the former is O(k log k) and the latter O(n log k). For a query point x ∈ R d , distances to cluster centers are effectively replaced by metric

d m (x, µ j ) := x -µ j , if j ∈ NN m (x) 0, otherwise, (3.13)
where NN m (x) ⊆ C denotes the approximate m-nearest neighbors of x. The overall complexity per iteration is then O(n log k). Now, similarly to [START_REF] Li | Large-Scale Robust Visual Codebook Construction[END_REF], we not only use approximate search to speed up clustering, but we also exploit the iterative nature of the clustering algorithm to enhance the search process itself. To this end, we maintain a list of the m best neighbors B(x i ) found so far for each data point x i , and re-use it across iterations. This results in an in-This generalizes RAKM [START_REF] Li | Large-Scale Robust Visual Codebook Construction[END_REF], which restricts to m = 1 and k-means only. cremental m-nearest neighbors algorithm [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF]. It is an N-step, to be performed prior to the E-step. As in [START_REF] Li | Large-Scale Robust Visual Codebook Construction[END_REF], the rationale is that by keeping track of the best neighbors found so far, we may significantly reduce the effort spent in searching for new ones.

experiments

setup We compare AGM against AKM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and RAKM [START_REF] Li | Large-Scale Robust Visual Codebook Construction[END_REF] on large scale vocabulary learning for image retrieval. We use Oxford buildings1 [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and world cities 2 [START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] datasets. The distractor set of world cities consists of 2M images; we use the first one million as distractors. We use SURF [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] features and descriptors, of dimensionality 64. We learn vocabularies from 6.5M descriptors of an independent set of 15k images depicting urban scenes. At learning, we use Fast Library for ANN (FLANN) [START_REF] Muja | Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration[END_REF] for all methods. We assign database descriptors to only one visual word. We apply MA [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] on the query side only as in [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF], keeping the first 1, 3, or 5 neighbors. We measure mAP.

results We first choose the best competing method for up to 20k distractors as shown in Table 3 then extend to 1M distractors for the RAKM 550k and AGM 857k vocabularies. AGM maintains a clear overhead.

discussion

We manage to get competitive performance on large-scale image retrieval with a set of parameters that work even on our very first two-

The main parameter is τ = 0.55.

dimensional example. In most alternatives one needs to tune at least the vocabulary size. Even with spherical components, the added flexibility of Gaussian mixtures appears to boost discriminative power. Yet, learning is as fast as approximate k-means, both in terms of EM iterations and underlying vector operations. Our solution appears to avoid both singularities and overlapping components that are inherent in ML estimation of GMM.

More can be found at our project home page3 , including software.

S PAT I A L M AT C H I N G

We develop a simple spatial matching model inspired by Hough voting in the transformation space, where votes arise from single feature correspondences. Using a histogram pyramid, we effectively compute pair-wise affinities of correspondences without ever enumerating all pairs. Our Hough pyramid matching [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] algorithm is linear in the number of correspondences and allows for multiple matching surfaces or non-rigid objects.

introduction

Despite the success of the BoW model [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF], spatial matching is still needed to boost image retrieval performance. A second stage of spa-Re-ranking is linear in the number of images to match, so its speed is crucial.

tial verification and geometry re-ranking is the de facto solution of choice, where approximations of RANSAC [START_REF] Fischler | Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography[END_REF] dominate. Exploiting the local shape of features (e.g. local scale or orientation), it is either possible to generate RANSAC hypotheses by single correspondences [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], or to see correspondences as Hough votes in a transformation space [START_REF] Lowe | Distinctive Image Features from Scale-Invariant Keypoints[END_REF]. In the former case one still has to count inliers, so the process is quadratic in the number of (tentative) correspondences. In the latter, voting is linear in the number of correspondences but further inlier count appears unavoidable.

We develop a relaxed spatial matching model, which applies pyramid matching [START_REF] Grauman | The Pyramid Match Kernel: Discriminative Classification With Sets of Image Features[END_REF] to the transformation space. Using local feature shape to generate votes, it is invariant to similarity transformations, free of inlier-count verification and linear in the number of correspondences. It imposes one-to-one mapping and is flexible, allowing nonrigid motion and multiple matching surfaces or objects.

Figure 4.1 compares our Hough Pyramid Matching (HPM) [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] to Fast Spatial Matching (FSM) [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF]. Both foreground and background surfaces are matched by HPM, whereas inliers from one surface are only found by FSM. But our major achievement is speed: In a given query time, HPM can re-rank one order of magnitude more images.

problem formulation

We represent an image by a set P of local features. For each feature p ∈ P , its position and local shape is given by the 3 × 3 matrix

We restrict discussion to scale and rotation covariant features. Given two images P, Q, an assignment or correspondence c = (p, q) is a pair of features p ∈ P, q ∈ Q. The relative transformation from p to q is a similarity transformation given by That is, a 4-DoF transformation consisting of translation, rotation and scale.

F (c) := F (q)F (p) -1 = M (c) t(c) 0 1 , (4.2)
where M (c) = σ(c)R(c), t(c) = t(q)-M (c)t(p); and σ(c) := σ(q)/σ(p), R(c) := R(q)R(p) -1 are the relative scale and orientation respectively from p to q. This is a 4-Degrees of Freedom (DoF) transformation represented by parameters

f (c) := (x(c), y(c), σ(c), θ(c)), (4.3)
where [x(c) y(c)] := t(c) and θ(c) := θ(q)θ(p). Hence assignments are points in a d-dimensional transformation space F with d = 4.

We are also given an set C of tentative correspondences. Two fea-An alternative is proximity of features in the descriptor space.

tures correspond when assigned to the same visual word:

C := {(p, q) ∈ P × Q : u(p) = u(q)}, (4.4)
where u(p) is the visual word of p. Each correspondence c = (p, q) ∈ C

We use the IDF of visual words for w.

is given a weight w(c) measuring its relative importance.

[ October 7, 2020 at 12:04classicthesis version 0.4 ] Given a pair of assignments c, c ∈ C, we assume an affinity measure α(c, c ) as a non-increasing function of their distance in the transformation space. Two assignments are conflicting if they map two features of one image to the same feature of the other.

Our problem is then to identify a subset of non-conflicting assign-

We are looking for one or more transformations that make parts of one image align to parts of the other.

ments that maximizes the sum of affinity over all assignment pairs. This is a binary quadratic programming problem [START_REF] Olsson | Solving Large Scale Binary Quadratic Problems: Spectral Methods vs. Semidefinite Programming[END_REF], but we only target a very fast, approximate solution. In fact, we want to group assignments according to affinity without ever enumerating pairs.

hough pyramid matching

We assume that transformation parameters are mapped to [0, 1], hence F := [0, 1] d . We construct a hierarchical partition Similar to [START_REF] Grauman | The Pyramid Match Kernel: Efficient Learning with Sets of Features[END_REF], there are g(b k )g(b k-1 ) new correspondences in a group with c at level k, giving rise to the strength of c up to level :

B := {B 0 , . . . , B L-1 } (4.5) of F into L levels. Each B ∈ B
The strength of c depends on the size of the groups it participates in and the level at which they are formed. The contributions of c, c to s(C) may only differ up to level -1.

bin in the hierarchy with c, c ∈ b, we impose the one-to-one mapping by keeping the strongest one up to level -1 according to (4.9).

The total operations per level are linear in n := |C|. Hence the time complexity of hpm is O(nL). The dotted line between c 6 , c 9 denotes a group that is formed at level 0 and then broken up at level 2, since c 6 is erased.

p q strength c 1 (2 + 1 2 2 + 1 4 2)w(c 1 ) c 2 (2 + 1 2 2 + 1 4 2)w(c 2 ) c 3 (2 + 1 2 2 + 1 4 2)w(c 3 ) c 4 (1 + 1 2 3 + 1 4 2)w(c 4 ) c 5 (1 + 1 2 3 + 1 4 2)w(c 5 ) c 6 0 c 7 0 c 8 1 4 6w(c 8 ) c

experiments setup

We compare HPM against 4-DoF FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] in pairwise matching and re-ranking in large scale search. In the latter case, we use two filtering models: BoW [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] and Weak Geometric Consistency (WGC) [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF]. We use our World Cities1 dataset, comprising a test set referred to as Barcelona and a 2M distractors set mostly depicting urban scenes exactly like the test set, but from different cities. We use SURF [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] features and descriptors.

results Here we only present mAP performance vs. query time for the re-ranking task, as shown in Figure 4.4 for varying number of reranked images. HPM re-ranks ten times more images in less time than FSM. With BoW, its mAP is 10% higher than FSM for the same re-ranking time, on average. At the price of 7 additional seconds for filtering, FSM eventually benefits from WGC, while HPM is clearly unaffected. After 3.3 seconds, mAP performance of BoW+HPM reaches saturation after re-ranking 7k images, while WGC does not help.

discussion

HPM is a simple spatial matching algorithm that can be easily integrated in any image retrieval engine. It boosts performance by allowing flexible matching. It is arguably the first time geometry re-ranking reaches saturation in a practical query time. The practice so far has been to stop re-ranking such that queries do not take too long, without studying further improvement as in Figure 4.4.

After its introduction [START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF], we have extended HPM to use MA [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF], applied to the query image only [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF]. This extension further boosts performance [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF]. In fact, HPM can work perfectly well even when Finding correspondences from descriptors is much more expensive than matching itself.

descriptors are not quantized. Hence it can be applied to matching scenaria other than image retrieval. More can be found at our project page2 , including real retrieval examples along with comparisons for one query and the entire 2M World Cities dataset.

B E Y O N D V O C A B U L A R I E S

We consider a family of metrics to compare images based on their local descriptors. It encompasses aggregated representations like VLAD [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF] and selective match kernels like Hamming Embedding [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF]. Making the bridge between these approaches leads us to introducing the Aggregated Selective Match Kernel (ASMK) [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF], which takes the best of existing worlds. Finally, the representation underpinning this kernel is approximated, providing precise and scalable image search.

introduction

To partially recover from the quantization loss incurred by the use of a vocabulary, it is common to assign descriptors to multiple visual words. This is known as Multiple Assignment (MA) [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] and is preferably applied to the query image only [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF] in an image retrieval context. Alternatively, one can learn a very fine vocabulary from data [START_REF] Mikulík | Learning a fine vocabulary[END_REF], where visual words have arbitrary shape in the descriptor space.

However, at the cost of some additional space, a more precise representation of the individual local descriptors works better than any vocabulary alone. One such solution is Hamming Embedding (HE) [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF], where descriptors are represented by short binary codes apart from visual word. Such methods exhibit selectivity, i.e., a correspondence contributes to the image-level similarity to a different extent, depending on an approximate distance in the descriptor space.

By contrast, aggregation or pooling operators, such as Fisher vectors [START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF][START_REF] Perronnin | Fisher Kernels on Visual Vocabularies for Image Categorization[END_REF] or the Vector of Locally Aggregated Descriptors (VLAD) [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF], depart from BoW by pooling vectors (e.g., descriptors) rather than scalars (e.g., frequencies). Compressing the resulting aggregated vectors [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF][START_REF] Perronnin | Large-Scale Image Retrieval with Compressed Fisher Vectors[END_REF] yields a very compact global image representation, which however is inferior to local image representation.

Here we bridge the gap between matching approaches, such as HE, and aggregated representations, in particular VLAD. We introduce a class of match kernels that encompasses both. Selectivity is only ex-Aggregation may still be beneficial in light of burstiness [START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF][START_REF] Jégou | On the burstiness of visual elements[END_REF]. ploited in matching approaches, which however do not use aggregation. We introduce a new representation that exploits the best of both worlds: the Aggregated Selective Match Kernel (ASMK) [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

match kernels

We represent an image by a set X := {x 1 , . . . , x n } of n d-dimensional local descriptors, quantized by a k-means quantizer q : R d → C, where C := {c 1 , . . . , c k } ⊂ R d is a codebook comprising k visual words. We denote by X c := {x ∈ X : q(x) = c} the subset of descrip-tors in X that are assigned to visual word c. Given two images X, Y , we consider a family of image-level similarity functions

K(X, Y ) := γ(X) γ(Y ) c∈C w c κ (X c , Y c ) , (5.1)
where scalar w c is a weight of visual word c, κ is a match kernel used

For instance, IDF weights.

as visual word-level similarity function and γ is a normalization function such that the self-similarity of an image is K(X, X) = 1. This family encompasses several well known methods.

bag-of-words [START_REF] Csurka | Visual Categorization With Bags of Keypoints[END_REF][START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] represents each local descriptor x solely by its visual word. As observed in [START_REF] Bo | Efficient Match Kernel between Sets of Features for Visual Recognition[END_REF][START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF], bag-of-words with cosine similarity can be expressed in terms of (5.1), by defining

κ BoW (X c , Y c ) := |X c | × |Y c | = x∈Xc y∈Yc
1.

(5.2)

hamming embedding (he) [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF][START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF] extends BoW by representing each local descriptor x with both its quantized value q(x) and a binary code b x ∈ {-1, +1} B of B bits. It matches all pairs of descriptors assigned to the same visual word by

κ HE (X c , Y c ) := x∈Xc y∈Yc w (d H (b x , b y )) , (5.3)
where w is a non-increasing similarity function and d H is the Hamw can be for instance a step [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF] or Gaussian function [START_REF] Jégou | On the burstiness of visual elements[END_REF].

ming distance, which can be expressed as

d H (a, b) = B 2 (1 -â b).
Here â denotes the 2 -normalized counterpart of vector a. vlad [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF] aggregates the descriptors per visual word into a dk-

vector representation V (X) ∝ [v(X c 1 ), . . . , v(X c k )], where v(X c ) := x∈Xc r(x) = x∈Xc x -q(x), (5.4)
and r(x) is the residual vector of x. Then, VLAD with cosine similarity

The power-law normalization of Fisher vectors [START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF] cannot be expanded as (5.5).

can again be expressed in terms of (5.1) by

κ VLAD (X c , Y c ) := v(X c ) v(Y c ) = x∈Xc y∈Yc
r(x) r(y).

(5.5)

towards a common model

The match kernels discussed above can be classified into two kinds.

non-aggregated kernels individually match all the elements occurring in the same vocabulary cell. They are written as

κ N (X c , Y c ) := x∈Xc y∈Yc
σ φ(x) φ(y) .

(5.6) encompassing all variants discussed so far. Here φ is a vector representation function, possibly nonlinear or including normalization, and σ : R → R is a scalar selectivity function.

aggregated kernels, in contrast, are written as

κ A (X c , Y c ) := σ    ψ x∈Xc φ(x) ψ   y∈Yc φ(y)      (5.7) = σ Φ(X c ) Φ(Y c ) , (5.8)
where ψ is another vector representation function, again possibly nonlinear or including normalization. Φ(X c ) is the aggregated vector rep-Aggregation, i.e., computing Φ(Xc), is an offline operation.

resentation of a set X c of descriptors in a cell. In contrast to (5.6), selectivity σ is applied after aggregation. a common model Table 5.1 summarizes the BoW, HE and VLAD kernels and expresses them in a common model. It also identifies in each case options for functions φ, σ, ψ and Φ. BoW and VLAD both fit into (5.6) and (5.7), with σ simply being identity. This is not the case for HE matching, which has a nonlinear σ, hence only fits into (5.6). This analysis suggests other potential strategies.

model κ(X c , Y c ) φ(x) σ(u) ψ(z) Φ(X c ) BoW (5.2) κ N or κ A 1 u z |X c | HE (5.3) κ N bx w B 2 (1 -u) - - VLAD (5.5) κ N or κ A r(x) u z V (X c ) ASMK (5.11) κ A r(x) σ α (u) ẑ V (X c )

aggregated selective match kernel

Our ASMK is motivated observing that VLAD employs a linear combination in (5.5) of the contributions of individual descriptor pairs (x, y) to κ, while HE applies a nonlinear weighting function σ to the similarity φ(x) φ(y) in (5.3), but involves no aggregation.

selectivity Without loss of generality, we consider a thresholded polynomial selectivity function σ α : R → R + of the form

σ α (u) := sgn(u)|u| α if u > τ 0 otherwise, (5.9)
and typically set α = 3, while τ ≥ 0. function σ α when matching features between two images, for different values of the exponent α. A larger α increases the selectivity and drastically down-weights false correspondences. This advantageously replaces hard thresholding as initially proposed in HE [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF]. mitigate this effect, it is common to down-weight [START_REF] Jégou | On the burstiness of visual elements[END_REF] or normalize [9] the contribution per visual word. This resembles binary BoW [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] or max pooling [START_REF] Boureau | Learning Mid-Level Features for Recognition[END_REF], which account at most one vote per visual word. We follow the same approach but for large vocabularies.

our asmk We aggregate the residual vectors as in VLAD (5.4) and then 2 -normalize the sum:

The database vectors v(Xc) are computed offline.

Φ ASMK (X c ) := v(X c ) = v(X c )/ v(X c ) .
(5.10)

ASMK is an aggregated kernel (5.8) with selectivity function σ α :

κ ASMK (X c , Y c ) := σ α v(X c ) v(Y c ) . (5.11)
ASMK is also summarized in Table 5.1. It handles burstiness by keeping only one representative of all bursty descriptors per cell, which is represented by the normalized mean residual.

binarization For the sake of compactness, HE represents the residual r(x) := xq(x) by a binary vector b x . We develop ASMK using full d-dimensional descriptors to investigate the upper bound in performance; in practice, we use an approximated version ASMK , binarizing v(X c ) (5.10) before applying the selectivity function.

experiments setup

We evaluate ASMK against fine vocabularies [START_REF] Mikulík | Learning a fine vocabulary[END_REF] and descriptor-We do not use any post-processing like spatial verification or query expansion.

based HE variants [START_REF] Jain | Asymmetric Hamming Embedding: Taking the best of our bits for large scale image search[END_REF][START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF] on Oxford Buildings [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], Paris [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] and Holidays [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF] datasets. We use a modified Hessian-Affine feature detector [START_REF] Perdoch | Efficient Representation of Local Geometry for Large Scale Object Retrieval[END_REF] and SIFT descriptors with component-wise square root [8,[START_REF] Jain | Hamming embedding similarity-based image classification[END_REF]. We use flat k-means to learn visual vocabularies of 65k visual words on an independent dataset. We combine with MA using 5 nearest visual words on query side only [START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF].

results As shown in Table 5.2, ASMK outperforms the binarized ASMK , which outperforms all other methods by a large margin. ASMK relies on full descriptors and does not scale to Oxford105k; only ASMK does. ASMK uses less memory and is faster than HE, having less features after aggregation.

discussion

By building a common model for match kernels like BoW, HE and VLAD, it becomes evident that different kernels have different properties. ASMK combines the selectivity of HE with the aggregation per visual word of VLAD, successfully combating burstiness. After its introduction [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF], we have extended ASMK to operate Aggregation across images [START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

across images [START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF], identifying local features in different images corresponding to the same physical structure. We do this by pairwise matching descriptors of all images in a dataset per visual word, connecting nearby pairs and forming the connected components of the resulting graph. We aggregate their representation into a single vector per component. We thus further compress the indexing structure and implicitly perform feature augmentation [START_REF] Turcot | Better matching with fewer features: The selection of useful features in large database recognition problems[END_REF].

Another extension is early burst detection [START_REF] Shi | Early Burst Detection for Memory-Efficient Image Retrieval[END_REF], where bursts are ex-Early burst detection [START_REF] Shi | Early Burst Detection for Memory-Efficient Image Retrieval[END_REF].

plicitly detected before quantizing to visual words. The simplest and most effective way to do so is to compute all pairwise feature similarities (according to scale, orientation and descriptor), connect features into components and aggregate descriptors per component. This applies to VLAD, ASMK and any aggregated kernel. Performance is on par with the state of the art but with significantly reduced complexity, thanks to the fewer descriptors kept. An interesting finding is the benefit of asymmetric aggregation, i.e., aggregating the database descriptors and not those of the query.

More can be found at our project home page1 , including software.

N E A R E S T N E I G H B O R S E A R C H

We present a simple vector quantizer that combines low distortion with fast search and apply it to Approximate Nearest Neighbor (ANN) search in high dimensional spaces [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. Leveraging the very same data structure that is used to provide non-exhaustive search, i.e., inverted lists or a multi-index, the idea is to locally optimize over rotation and space decomposition an individual product quantizer per cell and use it to encode residuals.

introduction

By experimenting with different vocabulary options [START_REF] Mikulík | Learning a fine vocabulary[END_REF][START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] and representations alternative to BoW [START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF][START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF][START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF][START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF], it becomes evi-This is in contrast to search methods that maintain all data uncompressed [START_REF] Muja | Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration[END_REF].

dent that image retrieval boils down to high-quality, compact encoding of visual descriptors that allows for fast search. This is ANN search in high-dimensional spaces, a recurring problem in computer vision.

Product Quantization (PQ) [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF] is a compact encoding method that can be used for exhaustive or non-exhaustive search through inverted indexing or multi-indexing [START_REF] Babenko | The Inverted Multi-Index[END_REF]. Better fitting to the underlying distribution is critical in search performance, as in Optimized Product Quan-

The same is true for most hashing methods [START_REF] He | K-Means Hashing: an Affinity-Preserving Quantization Method for Learning Binary Compact Codes[END_REF]. tization (OPQ) [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. The principle is that all bits allocated to data points should be used sparingly. Such methods should be ultimately seen as (lossy) data compression targeting minimal distortion.

As such, k-means, depicted in Figure 6.1(a), allows log 2 k bits to represent a data point in R d by specifying k centroids. But naïve search is O(dk) and low distortion means very large k. By constraining centroids on an axis-aligned, m-dimensional grid, PQ achieves k m centroids keeping search at O(dk); but as in Figure 6.1(b), many centroids may remain without data support. OPQ allows arbitrary rotation and re-ordering of dimensions to better align to data and balance variance across subspaces. But as illustrated in Figure 6.1(c), a multimodal distribution may not benefit from such alignment.

Our solution is Locally Optimized Product Quantization (LOPQ) [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. Following [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF], a coarse quantizer is used to index data, and residuals between data points and centroids are PQ-encoded. But within-cell

We also combine with multi-index [START_REF] Babenko | The Inverted Multi-Index[END_REF], which is essential for large scale datasets. distributions are largely unimodal; hence, as in Figure 6.1(d), we locally optimize an individual product quantizer per cell. Under no assumptions on the distribution, all centroids are supported by data, resulting a lower distortion.

background

vector quantization A quantizer is a function q that maps a d-dimensional vector x ∈ R d to vector q(x) ∈ C, where C ⊂ R d is a ), following a mixture distribution. (c) and (d) also reorder dimensions, which is not shown in 2D.

codebook with k := |C|. Each vector c ∈ C is called a centroid. Given a set X := {x 1 , . . . , x n } of n data points in R d , q induces distortion E := x∈X x -q(x) 2 . (6.1)
According to Lloyd's first condition [START_REF] Gray | Vector quantization[END_REF], regardless C, a quantizer that minimizes distortion should map x ∈ R d to its nearest centroid:

Hence, an optimal quantizer should minimize distortion E as a function of codebook C alone.

q(x) := arg min c∈C x -c . (6.2)
product quantization Assuming that dimension d is a multiple of m, write any vector

x ∈ R d as a concatenation (x 1 , . . . , x m ) of m sub-vectors, each of dimension d/m. If C 1 , . . . , C m are m sub- codebooks in subspace R d/m
, each of k sub-centroids, a product quantizer [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF] constrains C to the Cartesian product

C = C 1 × • • • × C m , (6.3)
i.e., a codebook of

k m centroids c = (c 1 , . . . , c m ) with each sub-centroid c j ∈ C j for j ∈ [m].
An optimal product quantizer q should minimize

We write [n] = {1, . . . , n} for n ∈ N.

E (6.1) as a function of C, subject to C being of the form (6.3) [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF].

In this case, for each

x ∈ R d , the nearest centroid in C is q(x) = (q 1 (x 1 ), . . . , q m (x m )), (6.4)
where q j (x j ) is the nearest sub-centroid of sub-vector x j in C j , for j ∈ [m] [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. Hence an optimal product quantizer q in d dimensions incurs m subproblems of m optimal sub-quantizers q j , j ∈ [m], each in d/m dimensions. We write q = (q 1 , . . . , q m ) in this case.

optimized product quantization [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF][START_REF] Norouzi | Cartesian k-Means[END_REF] refers to optimizing the subspace decomposition too. Constraint (6.3) is relaxed to

C = {Rĉ : ĉ ∈ C 1 × • • • × C m , R R = I}, (6.5)
where orthogonal d × d matrix R allows for rotation and permutation of dimensions. Optimization with respect to C 1 , . . . , C m and R is either joint as in Cartesian k-Means (CKM) [START_REF] Norouzi | Cartesian k-Means[END_REF] and in the non-parametric OPQ np of [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF], or decoupled, as in the parametric OPQ p of [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF].

exhaustive search Given a product quantizer q = (q 1 , . . . , q m ), let each encoded data point q(x) be represented by tuple

(i 1 , . . . , i m ) of m sub-centroid indices (6.4), each in index set [k]. Given a query
This PQ-encoding requires m log 2 k bits per point.

vector y, the (squared) Euclidean distance to x ∈ X may be approximated by the Asymmetric Distance Computation (ADC) [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF] δ q (y, x)

:= y -q(x) 2 = m j=1 y j -q j (x j ) 2 , (6.6)
where

q j (x j ) ∈ C j := {c j 1 , . . . , c j k } for j ∈ [m]. Distances y j -c j i 2 are precomputed for i ∈ [k] and j ∈ [m]
, so (6.6) amounts to only O(m) lookup and add operations.

indexing The residual vector of x ∈ R d quantized by q is r q (x) := xq(x).

(6.7)

Non-exhaustive search involves a coarse quantizer Q of K centroids, or cells. Each point x ∈ X is quantized to Q(x), and its residual vector

An inverted list of points and PQ-encoded residuals are maintained per cell.

r Q (x) is quantized by a product quantizer q. A query point y is quantized to its w nearest cells, and approximate distances to residuals are then computed (6.6) only within those cells. This is referred to as Inverted File -ADC (IVFADC) [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF].

multi-indexing applies the idea of PQ to the coarse quantizer used for indexing. A second-order inverted multi-index [START_REF] Babenko | The Inverted Multi-Index[END_REF] comprises two subspace quantizers over R d/2 , each of K sub-centroids. A cell is now a pair of sub-centroids. There are K 2 cells on a 2-dimensional grid, inducing a fine partition of R d . Quantization and distance computation is done independently per subspace. At query time, cells are traversed in increasing order of distance to the query by the multisequence algorithm.

A solution involving PQ-encoding of residuals is called Multi-Index ADC (Multi-D-ADC) [START_REF] Babenko | The Inverted Multi-Index[END_REF]. A solution using OPQ np to globally optimize both the residuals and the data prior to multi-index construction, is referred to as Optimized Multi-Index ADC (OMulti-D-OADC) [START_REF] Ge | Optimized Product Quantization[END_REF].

locally optimized product quantization

We develop our solution for a single coarse quantizer first. We then outline our solution for a multi-index.

single index Given X, we optimize a coarse quantizer Q, with codebook

B := {b 1 , . . . , b K } of K cells. For i ∈ [K], we collect the residuals of points quantized to cell b i LOPQ incurs O(K(d 2 + dk)) space overhead and O(wd 2 )
query time overhead comparing to IVFADC [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF].

Z i := {x -b i : x ∈ X, Q(x) = b i }. (6.8)
For each cell i ∈ [K], we locally optimize PQ encoding of residuals in Z i , yielding an orthogonal matrix R i and a product quantizer q i . Residuals are then locally rotated and PQ-encoded as q i (R i z) for z ∈ Z i . At query time, the query point y is soft-assigned to its w nearest cells in B. For each such cell b i , asymmetric distances

δ q i (R i r Q (y), R i z) of
The computation is exhaustive within Zi, but is performed in the compressed domain.

the rotated query residual

R i r Q (y) = R i (y -b i )
to rotated residuals R i z for z ∈ Z i are then computed according to (6.6), using the underlying local product quantizer q i . local optimization Let Z ∈ {Z 1 , . . . , Z K } be the set of residuals of data points quantized to some cell in B. Contrary to [START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF], we PQ-encode these residuals by locally optimizing both space decomposition and sub-quantizers per cell. Given m and k, this problem is expressed as minimizing distortion as a function of orthogonal matrix R ∈ R d×d and sub-codebooks

C 1 , . . . , C m ⊂ R d/m per cell, minimize z∈Z min ĉ∈ Ĉ z -Rĉ 2 subject to Ĉ = C 1 × • • • × C m R R = I, (6.9)
where

|C j | = k for j ∈ [m]. Sub-codebook C j determines a sub- Given solution R, C 1 , . . . , C m , codebook C is given by (6.5).
quantizer q j by q j (x) = arg min ĉj ∈C j

xĉj (6.10)

for j ∈ [m], x ∈ R d/m
, as in (6.2); collectively, sub-quantizers determine a product quantizer q = (q 1 , . . . , q m ) by (6.4). Local optimization can then be seen as a mapping Z → (R, q). We follow the parametric solution of [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF][START_REF] Norouzi | Cartesian k-Means[END_REF] that we briefly describe here.

parametric solution OPQ p [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF] assumes a zero-mean normal distribution N (0, Σ) of residual data Z and minimizes the theoretical lower distortion bound as a function of R alone [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. That is, R is optimized independently prior to codebook optimization, which follows by independent k-means per subspace, exactly as in PQ. Given the covariance matrix Σ, empirically measured on Z, the solution for R is found in closed form, in two steps.

First, rotating data by ẑ ← R z should yield a block-diagonal covariance matrix Σ, with the j-th diagonal block being sub-matrix Σjj of j-th subspace, for j ∈ [m]. That is, subspace distributions should be pairwise independent. This is accomplished by diagonalizing Σ as U ΛU . Second, determinants | Σjj | should be equal for j ∈ [m], i.e., variance should be balanced across subspaces. This is done by eigenvalue allocation [START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]: Eigenvalues in Λ are traversed in descending order and greedily allocated to the subspace of minimal variance. This

The solution for R represents a re-ordering of the eigenvectors of Σ. yields a permutation π of the set [d] of dimensions. Finally, the solution is R := U P π , where P π is the permutation matrix of π.

multi-index

In the case of a second-order multi-index, the space overhead is prohibitive to locally optimize per cell. Hence, we separately optimize a rotation and a set of sub-quantizers per cell of the two subspace quantizers and PQ-encode two sub-residuals per data point. At query time, the query needs to be rotated independently for [START_REF] Babenko | The Inverted Multi-Index[END_REF]. The query time overhead is O(Kd 2 ) in the worst case, but much lower in practice.

Multi-LOPQ incurs O(K(d 2 + dk)) space overhead comparing to Multi-D-ADC
each such cell. Rotations are lazy-evaluated i.e. computed on demand by the multi-sequence algorithm and stored for re-use. We call this solution Multi-Index LOPQ (Multi-LOPQ).

This solution is more constrained than in the case of a single index: each rotation matrix is constrained to be block-diagonal, keeping rotations within-subspace. By contrast, the rotation matrix in [START_REF] Ge | Optimized Product Quantization[END_REF] is unconstrained, but it is fixed for all cells. Table 6.1: Recall@r for r ∈ {1, 10, 100} on SIFT1B with 128-bit codes, k = 256, i.e. 8 bits per sub-quantizer, m = 16 subspaces, and subspace quantizers with K = 2 14 ; t is the target number of points fetched by multi-sequence.

experiments

setup Here we only include large-scale results of non-exhaustive search with a multi-index. We use the SIFT1B [START_REF] Jégou | Searching in One Billion Vectors: Re-Rank with Source Coding[END_REF] 1 dataset, containing 1 billion SIFT vectors and 10K queries. We evaluate Multi-LOPQ against Multi-D-ADC [START_REF] Babenko | The Inverted Multi-Index[END_REF] and OMulti-D-OADC [START_REF] Ge | Optimized Product Quantization[END_REF]. All methods PQencode the residuals of the subspace quantizers with 128-bit codes.

As in related work [START_REF] Babenko | The Inverted Multi-Index[END_REF][START_REF] Ge | Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF][START_REF] Jégou | Product Quantization for Nearest Neighbor Search[END_REF][START_REF] Jégou | Searching in One Billion Vectors: Re-Rank with Source Coding[END_REF][START_REF] Norouzi | Cartesian k-Means[END_REF][START_REF] Norouzi | Fast Search in Hamming Space with Multi-Index Hashing[END_REF], we measure search performance via recall@r, i.e. the proportion of queries having their Alternatively, recall@r is the fraction of queries for which the nearest neighbor would be correctly found if we verified the r top-ranking vectors using exact distances.

nearest neighbor ranked in the first r positions. Recall@1 is the most important, and is equivalent to the precision of [START_REF] Muja | Fast Approximate Nearest Neighbors with Automatic Algorithm Configuration[END_REF].

results As shown in Table 6.1, the optimized OMulti-D-OADC [START_REF] Ge | Optimized Product Quantization[END_REF] outperforms Multi-D-ADC [START_REF] Babenko | The Inverted Multi-Index[END_REF]. However, the performance of Multi-LOPQ is unprecedented, enjoying nearly 10% gain over OMulti-D-OADC on the most important measure of precision (recall@1).

overhead With K = 2 14 , the space overhead of Multi-LOPQ on top of Both space and time overhead is constant in data size n.

Multi-D-ADC is 500MB for rotation matrices and 2GB for sub-quantizer centroids, compared to 21GB that is the total SIFT1B index space with 128-bit codes. The query time overhead is the time needed to rotate the query for each cell. On average, this is 0.776 and 4.04ms respectively for t = 10k and 100k, compared to 7 and 49ms respectively for a Multi-D-ADC query.

discussion

Beneath LOPQ lies the very simple idea that no single centroid should be wasted by not representing actual data. Rather, each should contribute to lowering distortion. Hence, to take advantage of PQ, one should attempt to use and optimize product quantizers over parts of the data only.

LOPQ resembles a two-stage fitting of a mixture distribution: component means followed by conditional densities via PQ. Joint optimization of coarse and local quantizers might bring further improvement, but its training cost would be prohibitive.

More can be found at our project home page2 , including software.

E X P L O R I N G P H O T O C O L L E C T I O N S

We introduce an image clustering scheme that compresses a large corpus of images by grouping visually consistent ones, while providing a guaranteed distortion bound. This allows representing thousands of images depicting a landmark, while still being able to retrieve isolated non-landmark images.

Starting from a geo-tagged dataset, we group images geographically and then visually. We align all views to a reference image and construct a 2d scene map [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF]. Indexing and retrieval then operates directly on scene maps. We apply to location and landmark recognition and we demonstrate several integrated methods through our online application, VIRaL1 [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

introduction

Billions of images are available online along with metadata such as location, time and tags. Applications are emerging, for instance location [START_REF] Hays | IM2GPS: Estimating Geographic Information From a Single Image[END_REF] only estimates a geolocation probability map, while [START_REF] Zheng | Tour the World: Building a Web-Scale Landmark Recognition Engine[END_REF] only works on landmarks.

estimation [START_REF] Hays | IM2GPS: Estimating Geographic Information From a Single Image[END_REF], virtual tourism [START_REF] Snavely | Photo Tourism: Exploring Photo Collections in 3D[END_REF], and landmark recognition [START_REF] Zheng | Tour the World: Building a Web-Scale Landmark Recognition Engine[END_REF].

Here we are interested in location recognition given a single image, be it landmark or not. Unfortunately, current solutions either do not scale well, or focus on points of interest like landmarks.

Our work lies between generic image retrieval and clustering. While large image clusters of popular places help in terms of efficiency, a distortion bound can guarantee that isolated images are still found as in a generic retrieval engine. For instance, when clustering geo-tagged images by location [START_REF] Crandall | Mapping the World's Photos[END_REF], two images taken 2km apart are unlikely to depict the same building. Likewise, in spatial matching [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], 20 images each having 15 inliers with a reference image, may all depict similar views of a single scene.

We use Kernel Vector Quantization (KVQ) [START_REF] Tipping | A Kernel Approach for Vector Quantization with Guaranteed Distortion Bounds[END_REF], along with an appropriate metric to group images by location and then by visual similarity. Contrary to other solutions, this guarantees that no image in To speed up mining, we apply visual clustering only within each geo-cluster and we use sub-linear indexing for pairwise matching.

a cluster is too "far away" (depending on the metric) from the rest. Given a visual cluster, we align all images to a reference image and construct a 2d scene map by grouping local features, giving rise to another application of KVQ. Finally, we extend the entire search pipeline operate on scene maps rather than images. This not only provides memory savings, but increases recall too.

background

It is common to perform geo-clustering by location (latitude, longi-Views of the same scene are not expected in photos taken too far apart, so geo-clustering helps accelerate visual clustering.

tude) followed by visual clustering. The objective is to identify photos depicting views of the same scene. For instance, [START_REF] Crandall | Mapping the World's Photos[END_REF][START_REF] Li | Landmark Classification in Large-Scale Image Collections[END_REF] perform geo-clustering alone by mean-shift [START_REF] Cheng | Mean Shift, Mode Seeking, and Clustering[END_REF], while visual clustering follows using e.g. k-means [START_REF] Kennedy | How Flickr Helps Us Make Sense of the World: Context and Content in Community-Contributed Media Collections[END_REF] and agglomerative clustering [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF][START_REF] Quack | World-Scale Mining of Objects and Events From Community Photo Collections[END_REF][START_REF] Zheng | Tour the World: Building a Web-Scale Landmark Recognition Engine[END_REF]. The drawback of k-means and agglomerative clustering is that there is no control over the maximal intra-cluster distance, while mean-shift [START_REF] Crandall | Mapping the World's Photos[END_REF][START_REF] Li | Landmark Classification in Large-Scale Image Collections[END_REF] requires seeding and fixed tiles [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF][START_REF] Quack | World-Scale Mining of Objects and Events From Community Photo Collections[END_REF] do not adjust to data. We use KVQ [START_REF] Tipping | A Kernel Approach for Vector Quantization with Guaranteed Distortion Bounds[END_REF], which guarantees an upper bound on distortion and adjusts the number of clusters accordingly. kernel vector quantization Let (X , d) be a metric space. Given a set X := {x 1 , . . . , x n } ⊆ X , we are looking for a small subset Q(X) ⊆ X such that no point in X is too far away from some point in

Q. Define kernel function k : X × X → R by Br(x) is the open ball of radius r centered at x; 1A : X → {0, 1} is the indicator function of A ⊆ X . k(x, y) := 1 Br(x) (y), (7.1)
to indicate whether points x, y ∈ X lie within distance r, where r > 0 is a scale parameter. The Gram matrix is the n × n matrix K with elements

K ij := k(x i , x j ). For x ∈ X, define cluster C(x) := X ∩ B r (x)
as the set of points y ∈ X within distance r from x. If there is a weight vector w ∈ R n such that Kw > 0, then all points x ∈ X lie in C(x j ) for some point x j ∈ X with positive weight w j > 0. An 1 penalty

A weighted penalty also favors large clusters.

encouraging sparsity yields the problem

min w∈R n w 1 (7.2) subject to Kw ≥ 1, (7.3)
easily reduced to linear programming. Given the optimal solution w , the codebook Q(X) is defined as

Q(X) := {x j ∈ X : w j > 0}. (7.4)
We refer to points in Q(X) as cluster centers. Clearly, Q(X) ⊆ X, and the collection of clusters C(x) for x ∈ Q(X) is a cover for X2 but not a partition. That is, clusters are overlapping. By construction, the

Overlap is useful for both geo-and visual clustering, especially in case of gradual view transitions.

distortion induced by Q(X) is upper bounded by r. The number of clusters is adjusted accordingly.

view clustering

Now let X and X refer to photos. Each photo x ∈ X is represented by

We also refer to photos as images, or views.

location (latitude and longitude) and a set of local features, including position, local shape and visual word over a vocabulary W.

geo-clustering

We apply KVQ to X in metric space (X , d g ) with scale parameter r g , where metric d g : R 2 × R 2 → R is the geodesic distance on the Earth surface3 . Let Q g (X) be the resulting geo-codebook,

In practice, we use spatial bucketing on a uniform grid and keep one sample per bucket. visual clustering As in [START_REF] Simon | Scene Summarization for Online Image Collections[END_REF], we say that any two photos x, y ∈ X are connected if at least one rigid object is visible in both, possibly under different viewpoints. A scene is a subset S ⊆ X of connected photos. We use FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] to match a pair of images x, y under a ge-

The model may vary from 3-to 5-DoF.

ometric model. The number of inliers is used as visual similarity. The visual metric d v can be any non-increasing function of the similarity, since kernel function k is discrete (7.1). We apply KVQ to each geocluster

C g (x) for x ∈ Q g (X) in space (X , d v ) with scale parameter r v . An example is shown in Figure 7.2. Let Q v (G) be the visual codebook
The objective is not summarization or canonical view selection [START_REF] Simon | Scene Summarization for Online Image Collections[END_REF], but to align images in each cluster.

of geo-cluster G, and C v (y) the visual cluster of y ∈ G. The complete codebook Q(X) is the union over all geo-clusters

Q(X) := x∈Qg(X) Q v (C g (x)). (7.5)
The bottleneck is the construction of Gram matrix K, which is quadratic in n. The same complexity appears in [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF][START_REF] Simon | Scene Summarization for Online Image Collections[END_REF][START_REF] Zheng | Tour the World: Building a Web-Scale Landmark Recognition Engine[END_REF]; other options are to use small spatial tiles of 200m [START_REF] Quack | World-Scale Mining of Objects and Events From Community Photo Collections[END_REF] or to not use local features at all [START_REF] Kennedy | How Flickr Helps Us Make Sense of the World: Context and Content in Community-Contributed Media Collections[END_REF]. Indexing is efficient enough to even work without geo-clustering [START_REF] Chum | Large-Scale Discovery of Spatially Related Images[END_REF], but then isolated photos are unlikely to be discovered. Our solution is geo-cluster specific indexing: We use an inverted file indexed by both visual word and geo-cluster. Given a query image in geo-cluster G, we find all connected images x ∈ G in constant time. Computing a sparse K is now linear in |G|.

scene maps

We align all images in a visual cluster using a homography model and construct a scene map, a 2d spatial map of features associated to different views of the same scene. Scene maps are then used directly for retrieval.

view alignment During visual clustering, images are geometrically verified by FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF]. For each pair of matching images (x, y) in a geo-cluster, we store the best model T xy that transforms y to x. Each image x ∈ Q(X) is then treated as a reference in its visual cluster C v (x).

We now align each image y ∈ C v (x) to x and compute a 3×3 homography matrix H xy , starting from the stored model T xy and using the "iterative" method of Locally Optimized RANSAC (LO-RANSAC) [START_REF] Chum | Locally Optimized RANSAC[END_REF]. An example is shown in This feature collection bears similarities to view clustering for 3D object recognition [START_REF] Lowe | Local Feature View Clustering for 3D Object Recognition[END_REF], feature tracks for extracting canonical views [START_REF] Simon | Scene Summarization for Online Image Collections[END_REF] of a scene, visual cluster alignment for landmark detection [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF] and a latent model for Query Expansion (QE) [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF]. In our case, the objective is a compact representation of all collections P w (x) to be used directly Ideally, a query should match a scene map whenever it matches any single image in the map.

for retrieval. Unlike [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF], the construction is offline. We want to group local features from all image regions unlike [START_REF] Gammeter | I Know What You Did Last Summer: Object-Level Auto-Annotation of Holiday Snaps[END_REF], and control the distance between features in a group, unlike [START_REF] Lowe | Local Feature View Clustering for 3D Object Recognition[END_REF][START_REF] Simon | Scene Summarization for Online Image Collections[END_REF]. This gives rise to yet another use of KVQ for spatial clustering, which we apply separately to the position components of each P w (x) in the image plane R 2 with scale parameter r s . The scene map S(x) is the We use rs = θ, the error threshold used in spatial matching. collection of the resulting spatial codebooks Q s (P w (x)) over all visual words w ∈ W. An example is shown in Figure 7.4. indexing and retrieval A scene map has exactly the same representation as a single image, i.e., a set of features. We therefore treat scene maps as images for indexing and retrieval. In particular, S(x) contains |Q s (P w (x))| features assigned to visual word w. We index scene maps by visual word in an inverted file using these cardinalities as a term frequency vector. At query time, all images y ∈ C v (x)

This makes it possible to retrieve images that would not match by themselves, increasing recall.

are ranked at the same position as the corresponding scene map S(x), without verifying them individually.

experiments

setup We evaluate our method on European Cities 1M 4 , a one-million urban image dataset that we contribute [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF], comprising a test set referred to as Barcelona and a 900k distractor set depict urban scenery like the test set. We use SURF features and descriptors [START_REF] Bay | SURF: Speeded Up Robust Features[END_REF] and a

Indexing by scene maps takes 1.2GB instead of 1.61GB for the baseline, a 25% compression.

75k vocabulary learned by flat k-means [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] on an independent set. We compare against baseline BoW and two QE [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF] variants: QE results As shown in Table 7.1, our method outperforms all others, even QE 1 : The expanded set comes from entire dataset in QE 1 , while scene maps are constructed by querying a single geo-cluster. Scene maps are only slightly slower than the baseline, which is due to spatial matching on more features, while QE is even slower.

application: viral

The method introduced here [ the world, along with their metadata (i.e. geographic location, user tags, image title and description). It also uses databases of landmarks and points of interest from Wikipedia 7 and GeoNames 8 . Figure 7.5 shows the result of a VIRaL query. The suggested tags are all correct and automatically linked to Wikipedia. Landmark recog-Quantitative evaluation of location and landmark recognition is conducted [START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF].

nition is very accurate because it relies on different sources of information. In particular, it compares Flickr metadata and geo-tags to landmark names and coordinates in landmark databases.

VIRaL Explore 9 enables browsing of the entire VIRaL image collection on the world map. Starting in a given city, it places icons of grouped photos, along with landmark names and links to Wikipedia, if applicable. The VIRaL collection is processed offline to identify groups of photos depicting the same object, building, or scene, using our scene maps [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF]. Most popular groups are shown on the map, according to the zoom level.

VIRaL Routes 10 offers a unique browsing experience of photo collections. Personal photo sets are processed offline to identify where they were taken and group them by scene. A route is then laid out on the map, showing icons of visited places. Route construction is based on geographic coordinates and timestamps. The density of photos is adjusted to the zoom level.

discussion

Typically, sub-linear indexing is not exploited in landmark recognition applications, while geo-tags are not exploited in large scale 3d reconstruction applications. We combine both here, along with a novel scene representation that is directly encoded in our retrieval engine. The result is significant gain in retrieval performance, even compared to query expansion methods, at the cost of a slight increase in query time. Index space is also reduced. Contrary to landmark recognition applications, we can still retrieve any isolated image, allowing location recognition at any region where geo-tagged photos are available. We also recognize landmarks and points of interest by comparing location, photo title and frequent tags to landmark databases. More results, both for landmark and non-landmark scenes, can be found online in our project homepage 11 .

Part II

E X P L O R I N G D E E P E R
Building on Convolutional Neural Network (CNN) features, we study visual representations and matching processes for exploring visual data, including instance-level visual search and object discovery, focusing on the manifold structure of the feature space.

O U T L I N E

This chapter serves as an outline or road map of Part ii. We present historical and more recent background on deep learning for visual representations developed in the 2010s. In this context, we position our own contributions developed in 2017-2019. Building on Convolutional Neural Network (CNN) features, our work addresses visual representations and matching processes for exploring visual data, including instance-level visual search and object discovery, focusing on the manifold structure of the feature space. We outline the structure of Part ii in terms of methods, key publications and corresponding chapters.

context

The work of Krizhevsky et al. known as AlexNet [START_REF] Krizhevsky | Im-ageNet Classification with Deep Convolutional Neural Networks[END_REF] in 2012 is arguably a landmark of machine learning research in the 2010s. Even if none of the ideas is entirely new, Krizhevsky et al. put together four elements that make learning visual representations from raw data "really" work at scale: [START_REF] Fukushima | Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected By Shift in Position[END_REF][START_REF] Le Cun | Handwritten Digit Recognition with a Back-Propagation Network[END_REF] trained on a supervised classification task using Stochastic Gradient Descent (SGD) [START_REF] Robbins | A stochastic approximation method[END_REF] with momentum and back-propagation [START_REF] David E Rumelhart | Learning Representations By Back-Propagating Errors[END_REF];

1. a CNN architecture
2. the ILSVRC dataset [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], comprising more than one million images, each with a class label over 1000 classes;

3. a massively parallel implementation on GPUs [START_REF] Kumar Chellapilla | High performance convolutional neural networks for document processing[END_REF]; and

4. the ReLU non-saturating nonlinearity [START_REF] Nair | Rectified Linear Units Improve Restricted Boltzmann Machines[END_REF], facilitating backward gradient flow through a depth of 8 layers.

Representations learned on ILSVRC yield astounding performance on a multitude of category-level or instance-level computer vision tasks [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF][START_REF] Sharif Razavian | CNN features off-the-shelf: an astounding baseline for recognition[END_REF]. In 2015-16, three more elements unleash the full power of deep architectures beyond 100 layers: careful initialization [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF], activation normalization [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] and skip connections [START_REF] He | Deep residual learning for image recognition[END_REF].

In instance-level search, different views of the same object should be mapped to similar representations. Shallow representations are quickly outperformed by CNNs trained on ILSVRC and then fine-tuned to new domains, supervised by noisy class labels [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF] or by the very same shallow representations [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF].

In this context, Part ii presents part of our work carried out in the Our contributions.

period 2017-2019, which addresses visual representations and matching processes for exploring visual data, including instance-level visual search and object discovery, focusing on the manifold structure of the feature space.

outline Our contributions consist of:

1. making advances in manifold search over global or regional CNN representations seen as graph filtering, including spatial [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF], spectral [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF] and hybrid [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF];

2. revisiting spatial matching with local features detected on CNN activations in the simplest possible way [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF]; and

3. discovering objects from CNN activations over an unlabeled image collection, seen again as graph filtering [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

Importantly, as a result of discovering objects in a collection, we improve the representation of each image itself by focusing on objects and suppressing background clutter.

background and contributions neural networks Rosenblatt introduces the term perceptron in

Due to Minsky and Papert [START_REF] Marvin | Perceptrons: an Introduction to Computational Geometry, Expanded Edition[END_REF], perceptron is known today as a linear classifier and an algorithm for that classifier.

the 1960s [START_REF] Rosenblatt | Principles of Neurodynamics[END_REF], referring to a wide range of network architectures, learning algorithms and hardware implementations. Although not widely appreciated even today, he lays the foundations of modern neural networks by studying early forms of multi-layer networks, continuous activation functions, back-propagating errors, convolution, skip connections, recurrent networks, selective attention, program learning, and multimodality. The perceptron algorithm is an instance of (online) Stochastic Gradient Descent (SGD), introduced in 1951 by Robbins and Monro [START_REF] Robbins | A stochastic approximation method[END_REF]. Paramount to the optimization of network parameters is the efficient evaluation of derivatives over arbitrary architectures. Given an arbitrary computational graph and a set of values for the input variables, automatic differentiation [START_REF] Günes Baydin | Automatic differentiation in machine learning: a survey[END_REF][START_REF] Edwin | A simple automatic derivative evaluation program[END_REF] allows the construction of an-One of its first applications to machine learning is maybe by Werbos [START_REF] John | Beyond Regression: New Tools for Prediction and Analysis in the Behavioral Sciences[END_REF].

other graph for the evaluation of derivatives at the given inputs. learn an 8-layer CNN from random initialization on the 1000-class classification task of the ImageNet Large-Scale Visual Recognition Challenge (ILSVRC) [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF], having a training set of 1.2M high-resolution images. This network is now named AlexNet after Alex Krizhevsky. Among the most influential design choices of AlexNet is that of the Rectified Linear Unit (ReLU) nonlinearity [START_REF] Nair | Rectified Linear Units Improve Restricted Boltzmann Machines[END_REF] over tanh (or sigmoid). In fact, ReLU is similar to a flipped version of the loss function of three

ReLU is used at least as early as 1980 by Fukushima [START_REF] Fukushima | Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected By Shift in Position[END_REF].

well-known binary classifiers: the perceptron, logistic regression (sigmoid + cross-entropy), and SVM (hinge function) [START_REF] Bishop | Pattern Recognition and Machine Learning[END_REF]. These functions are easy to optimize because they are not saturating. In retrospect, it becomes clear that saturating nonlinearities like sigmoid would not allow enough backward gradient flow for deep networks to learn, at least without skip connections [START_REF] Nguyen | On the loss landscape of a class of deep neural networks with no bad local valleys[END_REF].

In deep learning, "depth" refers to the number of layers. It is of-

For the same number of units, deep networks partition the input space into exponentially more linear regions than shallow ones [START_REF] Guido F Montufar | On the number of linear regions of deep neural networks[END_REF][START_REF] Telgarsky | Representation benefits of deep feedforward networks[END_REF].

ten admitted that improvements come from just "stacking more layers" [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. In 2014, a network named after the Visual Geometry Group (VGG) [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and another after the film Inception [START_REF] Szegedy | Going deeper with convolutions[END_REF] reach 19 and 22 layers respectively, the latter winning ILSVRC 2014. However, both face difficulties at training. As a workaround, the former uses pre-trained layers of shallow models to initialize deeper ones, while the latter uses auxiliary classifiers at training, attached to intermediate layers.

The difficulties encountered at such depth are related to the problem of exploding/vanishing gradients [START_REF] Glorot | Understanding the Difficulty of Training Deep Feedforward Neural Networks[END_REF]. A solution is careful weight Learning long-term dependencies in recurrent networks suffers from the same problem [START_REF] Bengio | Learning long-term dependencies with gradient descent is difficult[END_REF].

initialization, which is studied in the linear regime of nonlinearities by Glorot and Bengio in 2010 [START_REF] Glorot | Understanding the Difficulty of Training Deep Feedforward Neural Networks[END_REF] and for ReLU by He et al. [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification[END_REF] in 2015, who train a 30-layer network from scratch. A more effective solution is batch normalization, an operation introduced by Ioffe and Szegedy also in 2015 [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF], which normalizes activations of all layers using mini-batch statistics. Apart from preventing exploding/vanishing gradients and allowing higher learning rates, it reduces the need for regularization, e.g. by weight decay or dropout [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF].

With all problems resolved in training a deep network from scratch, the principle of "stacking more layers" is challenged when e.g. 56layer networks are found inferior to 20-layer networks [START_REF] He | Deep residual learning for image recognition[END_REF]. Developed by He et al. in 2015 [START_REF] He | Deep residual learning for image recognition[END_REF], reaching 152 layers and winning ILSVRC 2015, the Residual Network (ResNet) addresses this issue by in-Skip connections "convexify" the loss landscape [START_REF] Li | Visualizing the loss landscape of neural nets[END_REF] and make local minima equally good [START_REF] Nguyen | On the loss landscape of a class of deep neural networks with no bad local valleys[END_REF].

troducing skip connections to the standard chain-structured network architecture. While skip connections date back at least to Rosenblatt [START_REF] Rosenblatt | Principles of Neurodynamics[END_REF], it is with ResNet that they become a key ingredient of modern deep network architectures, including Inception-ResNet [START_REF] Szegedy | Inception-v4, Inception-ResNet and the impact of residual connections on learning[END_REF] and the Densely Connected Network (DenseNet) [START_REF] Huang | Densely Connected Convolutional Networks[END_REF].

Apart from the astounding performance of CNNs, deep learning quickly develops into a "Swiss-army knife" for most computer vision tasks [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF][START_REF] Sharif Razavian | CNN features off-the-shelf: an astounding baseline for recognition[END_REF], allowing training of different modules on different outline datasets and tasks, combining modules into new architectures, finetuning in several stages, jointly training modules on several tasks, defining and solving new tasks by proper supervision and loss functions, self-learning on unlabeled data and fictitious tasks, with imagination being the only limit.

As a concrete example, Girshick et al. introduce Regions with CNN Object detection.

features (R-CNN) [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] for object detection. They use AlexNet [START_REF] Krizhevsky | Im-ageNet Classification with Deep Convolutional Neural Networks[END_REF], as pre-trained on ILSVRC and fine-tuned on the target classes with its last Fully-Connected (FC) layer removed, to replace shallow representations in an pipeline involving region proposals [START_REF] Koen | Segmentation As Selective Search for Object Recognition[END_REF][START_REF] Zitnick | Edge Boxes: Locating Object Proposals From Edges[END_REF] and an SVM classifier. Unfortunately, this uses the network thousands of times, once for each region. He et al. [START_REF] He | Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition[END_REF] rather use the network only once on the entire image and extract regional features by spatial max-pooling on projected regions on the activations of the last convolutional layer.

Based on this idea, Girshick introduces Fast R-CNN [START_REF] Girshick | Fast R-CNN[END_REF], replacing the SVM classifier by new FC layers, while Ren et al. extend to Faster R-CNN [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF], replacing hand-crafted proposals by a Region Proposal Network (RPN) and training everything end-to-end.

learning for image retrieval A CNN of e.g. 101 layers learned on a category-level classification task can be seen as a linear classifier on top of a representation extracted by the first 100 layers. The representation space can be used for similarity search as already exhibited on few examples for FC layer 7 of AlexNet [START_REF] Krizhevsky | Im-ageNet Classification with Deep Convolutional Neural Networks[END_REF], retrieving images of the same class as the queries.

Learning for instance-level tasks is possible by treating e.g. each

The term neural codes [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF] indicates that using a representation out of a neural network for similarity search appeared exotic at the time.

building or landmark as a class and using a classification loss like cross-entropy. Babenko et al. [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF] use AlexNet, as pre-trained on ILSVRC, and fine-tune it like that on a Landmarks dataset. They evaluate for the first time instance-level retrieval on a representation learned on raw data and they find that FC layer 6 performs best. It easily escapes the reader that experiments on convolutional layer 5 by Babenko et al. [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF] assume flattening of the 3d convolutional activation tensor into a vector of length 9,216, which is a non-invariant representation. The next obvious attempt by Azizpour et al. [START_REF] Azizpour | From generic to specific deep representations for visual recognition[END_REF] drop FC layers and apply spatial max-pooling to the activation tensors of the last convolutional layer. Then, Razavian et al. [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF][START_REF] Sharif Razavian | Visual Instance Retrieval with Deep Convolutional Networks[END_REF] split images into patches on a grid, feed each patch into a CNN and match the resulting regional vector representations exhaustively pairwise. With a lower-dimensional representation based on a modified version of VGG, this solution outperforms for the first time the best known representation based on hand-crafted local descriptors, which happens to be our ASMK [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF]. The matching process by Razavian et al. [START_REF] Sharif Razavian | Visual Instance Retrieval with Deep Convolutional Networks[END_REF] is invariant but expensive. The extraction process is expensive too. With the discriminative power of CNNs, a global representation with a single pass through the network is more appealing. Tolias et al. [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]

obtain a

This progression is the same as the progression from R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] to Fast R-CNN [START_REF] Girshick | Fast R-CNN[END_REF] for object detection.

single activation tensor from the last convolutional layer corresponding to the entire image and, using a similar spatial grid as [START_REF] Sharif Razavian | Visual Instance Retrieval with Deep Convolutional Networks[END_REF], per-form a two-level spatial pooling called Regional Maximum Activation of Convolutions (R-MAC): max-pooling over cells on the grid followed by average pooling. This is found superior to Maximum Activation of Convolutions (MAC), a name given in retrospect to global max-pooling as introduced by Azizpour et al. [START_REF] Azizpour | From generic to specific deep representations for visual recognition[END_REF].

To adapt representations for the image retrieval task, rather than classifying images like Babenko et al. [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF], requires learning to compare or rank them. This is the task of metric learning, which is discussed in more detail in Section 13.2. The two most common methods are to consider images in pairs or triplets with the contrastive [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF] or triplet loss [START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF], respectively. But how are we to define labels on pairs or triplets of examples? This is certainly a more difficult task than a label per example as in classification.

Gordo et al. [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF] clean the the Landmark dataset by Babenko et al. [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF] by pairwise matching using hand-crafted local descriptors and spatial verification, then finding the different profiles of each landmark (e.g. inside/outside) as connected components of a graph and keeping the largest one. On this clean dataset, they learn fine-grained Gordo et al. also use an RPN [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] to define the regions, as discussed below.

similarity with a network pre-trained on ILSVRC by attaching R-MAC pooling [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] and fine-tuning it using the triplet loss [START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF].

Concurrently, Radenović et al. [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] work on an independent unlabeled dataset of urban scenes. They apply a similar pairwise matching process as well as a SfM pipeline, resulting in 3d models and camera positions per image. They use 3d models as labels and select hard positive and negative pairs using camera positions and matching scores. They attach MAC pooling [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] to a pre-trained network and fine-tune it using the contrastive loss [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF]. They extend to Generalized Mean (GeM) pooling [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF], which allows for contributions from more than one spatial location.

We focus on instance-level tasks, adopting learned CNN representations in Part ii. The representations are global, regional or local, taking advantage of convolutional activations. Then, Part iii addresses learning representations for both instancelevel and category-level tasks. Our own contribution to metric learning for image retrieval is the subject of Chapter 14.

searching on manifolds Both Gordo et al. [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF] and Radenović et al. [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF], using a global vector representation per image, out-E.g., a vector of length 512 or 2,048 for VGG or ResNet network backbone respectively. perform representations of thousands of hand-crafted local descriptors per image, including our ASMK [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF]. This is a game changer. Not only is image retrieval reduced to nearest neighbor search, but searching on image manifolds becomes a reality. To appreciate how, we go back to studies in psychology and social sciences in the 1950s.

In 1949, Seeley [START_REF] John R Seeley | The Net of Reciprocal Influence. A Problem in Treating Sociometric Data[END_REF] studies a nonnegative square matrix W with elements representing endorsement between entities, for instance one (zero) meaning "like" ("don't like"). He observes that it is important to be liked by someone who is in turn liked a lot, and so on. Recursively, he defines an index of importance or centrality over entities, outline given by an eigenvector of W . In 1953, Katz [START_REF] Katz | A New Status Index Derived From Sociometric Analysis[END_REF] uses W to represent a directed graph over entities and a power series of W to measure

The column sum of this power series is known as Katz centrality.

the number of paths between entities, representing recursive endorsements. He introduces a damping factor α to ensure convergence of the series and expresses the power series as a matrix inversion. In 1965, Hubbell [START_REF] Charles | An input-output approach to clique identification[END_REF] introduces a boundary condition or initial preference vector to estimate the similarity of entities given the graph.

Vigna [START_REF] Vigna | Spectral Ranking[END_REF] puts together all this line of work, calling it spectral ranking, connecting the eigenvector and matrix inversion formulations and listing a number of rediscoveries of the same theory in differ-

The most well-known is arguably PageRank by Page et al. [START_REF] Page | The PageRank citation ranking: bringing order to the web[END_REF].

ent fields. Two more rediscoveries include Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF], who use a symmetric form of W to represent similarity and an iterative process to avoid matrix inversion; and Pan et al. [START_REF] Pan | Automatic Multimedia Cross-Modal Correlation Discovery[END_REF], who give the name Random Walk with Restart (RWR) to the same process. Both treat the initial preference vector as a query representation and use the iterative process to estimate the similarity of entities to the query given the graph, which represents a manifold. Several such iterative processes, called diffusion processes, are studied for image retrieval by Donoser and Bischof [START_REF] Donoser | Diffusion processes for retrieval revisited[END_REF]. To maintain efficiency, graphs need to be small and image representations simple. For large-scale image retrieval using local descriptors, it is more common to apply Query Expansion (QE) [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF] online, without a graph. In QE originates in text retrieval [START_REF] Buckley | Automatic query expansion using SMART: TREC 3[END_REF].

its simplest form, QE is non-iterative and can be thought of as just the first term of the power series formulation [START_REF] Katz | A New Status Index Derived From Sociometric Analysis[END_REF].

With powerful and compact global CNN vector representations, it is easy to construct a k-Nearest Neighbor (k-NN) graph of a large dataset offline. Following the regional matching of Razavian et al. [START_REF] Sharif Razavian | Visual Instance Retrieval with Deep Convolutional Networks[END_REF], one can even construct a k-NN graph of image regions, having few (e.g. 20) regions per image. This graph represents the image (region) manifold and diffusion explores this manifold online.

Our regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] is the first method to apply diffusion at Our regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

large scale on such a representation. We return to the matrix inversion formulation and we observe that it is only needed to solve a linear system. We reveal that the RWR process [START_REF] Pan | Automatic Multimedia Cross-Modal Correlation Discovery[END_REF][START_REF] Zhou | Ranking on Data Manifolds[END_REF] is an instance of the Jacobi [START_REF] Hackbusch | Iterative solution of large sparse systems of equations[END_REF] solver and we rather use Conjugate Gradient (CG) [START_REF] Nocedal | Numerical optimization[END_REF], which is also iterative but more efficient.

It turns out that the mapping from the input query vector to the output manifold similarity vector is a linear graph filtering operation, smoothing in particular, as defined in Graph Signal Processing (GSP) [START_REF] Sandryhaila | Discrete Signal Processing on Graphs[END_REF][START_REF] David I Shuman | The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains[END_REF]. We study this analogy between manifold search and smoothing in our Fast Spectral Ranking (FSR) [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF]. We introduce a scalable offline Our Fast Spectral Ranking (FSR) [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF].

computation of an approximate Fourier basis of the graph and perform filtering online in the frequency domain, which is extremely fast. The basis is a low-dimensional and sparse explicit embedding of the diffusion similarity kernel. Alternatively, kernel PCA [START_REF] Bourrier | Explicit Embeddings for Nearest Neighbor Search with Mercer Kernels[END_REF][START_REF] Schölkopf | Nonlinear component analysis as a kernel eigenvalue problem[END_REF] would need to compute the kernel first, which we do not.

The combination of powerful CNN representations and manifold search nearly solves the Oxford [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and Paris [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] image retrieval benchmarks. To facilitate further research, we introduce the Revis-ited Oxford and Paris (RevOP) benchmark [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF]. Among other improve-Our RevOP [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF].

ments, we create a new set of one million challenging distractors. At this scale, it turns out that CG [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] and FSR [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF] take too much time or space respectively. Our hybrid diffusion [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF] allows full control of Our hybrid diffusion [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF].

the space-time trade-off between the two extremes.

local features and spatial matching Convolutional activations on images are 3d tensors, where the two dimensions correspond to spatial dimensions of the input image, though at lower resolution, and the third dimension to features (descriptors). They can be thought of as a set of dense local descriptors per spatial location.

Or, as a collection of 2d activation maps, one per channel (see below).

Spatial pooling yields global or regional descriptors that can be used to estimate similarity, but, can the original activations be used to estimate accurate correspondences?

This question is studied and answered in the affirmative by Long et al. in 2014 [START_REF] Long | Do Convnets Learn Correspondence?[END_REF]: Although they have large receptive fields, those descriptors carry local information at a fine scale. Hence, pairwise matching can yield dense correspondence and alignment between two views. Importantly, these views are not necessarily of the same object or scene as discussed in Section 2.2. They can be of two different instances of the same category.

However, again as discussed in Section 2.2, we know that not all locations are equally good for establishing correspondences. Sparse local features remain the best choice in handling occlusion and estimating relative pose in wide-baseline matching. In instance-level retrieval, spatial matching is a key ingredient of methods based on hand-crafted local descriptors. CNN representations already encode geometry via interleaved convolution and pooling. Can spatial matching on sparse local features help further, and how?

One of the earliest learned local feature detectors is by Dias et al. [START_REF] Dias | A neural network based corner detection method[END_REF] in 1995, using a 3-layer neural network to detect corners on 8 × 8 patches after edge detection and thinning. In a more modern setting, Verdie et al. [START_REF] Verdie | TILDE: a temporally invariant learned detector[END_REF] introduce in 2014 the Temporally Invariant

The regressor is essentially a 2-layer CNN with max-out [START_REF] Goodfellow | Maxout networks[END_REF] nonlinearity.

Learned DEtector (TILDE), a piece-wise linear regressor mapping input images to a score map where features are detected at local maxima. The Learned Invariant Feature Transform (LIFT) [START_REF] Moo | LIFT: Learned invariant feature transform[END_REF] integrates TILDE into a complete pipeline comprising the detector, patch cropping, orientation estimation and descriptor extraction. SuperPoint [START_REF] Daniel Detone | SuperPoint: Self-supervised interest point detection and description[END_REF] operates on the entire image instead with a single and deeper network backbone encoder and two different upsampling decoders serving as detector and descriptor.

Most learned detectors need ground truth coordinates or correspondences, which are commonly provided by hand-crafted detectors and matching processes on carefully designed datasets. An exception is DEep Local Features (DELF) by Noh et al. [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF], which is trained with image-level labels only. Following again a single network backbone, an attention branch is selecting the spatial locations where descriptors are to be extracted from the activation tensor.

outline Still, all learned detectors operate on a single 2d score map and detect point features, without local shape. However, Tolias et al. [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] illustrate that the spatial locations of maximum activation per channel can yield correspondences, while Generalized Mean (GeM) pooling [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] suggests that more locations may be important.

Building on these findings, we observe that local features emerge on activation maps without particular effort, i.e., without modifying the network architecture and without training. Simply put, we see the activation 3d tensor as a collection of 2d maps, one per channel, and we detect local features at local maxima of these 2d maps, independently per channel. By fitting affine regions, we also equip local features with geometric information to allow generation of transformation hypotheses from single correspondences. Our Deep Spatial Matching (DSM) [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF] applies these ideas to geometry verification for Our Deep Spatial Matching (DSM) [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF].

instance-level image retrieval. By treating activation channels as visual words, we do not even use local descriptors or vocabularies.

visual attention and object discovery Primates, including human, use attention mechanisms to analyze visual stimuli. In the 1960s, Yarbus [START_REF] Alfred | Eye movements and vision[END_REF] uses an eye-tracking device to study the role of eye movements in visual perception. In 1980, Treisman and Gelade [START_REF] Treisman | A Feature-Integration Theory of Attention[END_REF] study visual search for targets among distractors. Their findings sug-This latter form of attention is independent of eye movements.

gest that in a first pre-attentive stage, simple features are processed in parallel, while in a second attentive stage, the focus of attention is shifted to different locations in a sequence. Computational models of selective visual attention follow. In 1985, Koch and Ullman [START_REF] Koch | Shifts in Selective Visual Attention: Towards the Underlying Neural Circuitry[END_REF] introduce a model comprising a saliency map that combines individual feature maps into a global conspicuity measure and a Winner-Take-All (WTA) mechanism that sequentially routes WTA is implemented via softmax over spatial locations.

the properties at the most conspicuous location from individual feature maps to a central representation. ily [START_REF] Girshick | Fast R-CNN[END_REF][START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] are reminiscent of this approach: Region proposals [5] obtained by bottom-up grouping like selective search [START_REF] Koen | Segmentation As Selective Search for Object Recognition[END_REF] and edge boxes [START_REF] Zitnick | Edge Boxes: Locating Object Proposals From Edges[END_REF] are used to focus the attention of a classifier on top of deep convolutional features extracted densely over the entire visual field. RPN [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF] is a learned attention mechanism, in the form of a RPN requires ground-truth bounding boxes.

class-agnostic detector applied densely. One-stage detectors are applied densely too, but with appropriate weighting [START_REF] Lin | Focal Loss for Dense Object Detection[END_REF].

Compact global representations for instance-level tasks can be obtained by pooling over attended regions too, replacing the fixed regions of R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]: for instance, using selective search [START_REF] Reddy | Object level deep feature pooling for compact image representation[END_REF], RPN as pre-All these solutions are either hand-crafted or require localization or image-level ground-truth. trained on ILSVRC [START_REF] Salvador | Faster R-CNN features for instance search[END_REF], or RPN fine-tuned for retrieval [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF]. An alternative is weighted average pooling using a saliency map. This may be supervised, e.g. learning to predict eye fixations [START_REF] Mohedano | Saliency Weighted Convolutional Features for Instance Search[END_REF] or from labeled nearest neighbors [START_REF] De Campos | Images As Sets of Locally Weighted Features[END_REF]; or unsupervised, e.g. by focusing on channels with sparse activations [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF] or implicitly learning an attention layer on image-level labels [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF].

The global representation should focus on foreground objects, suppressing clutter and occlusions. Ideally, the definition of foreground For instance, focusing on animals is not interesting for a target dataset of city scenes.

should depend on the target dataset: Objects appearing frequently are likely to be foreground. In this case, any learning should be fully unsupervised. In instance-level tasks using local descriptors, this can be done by pairwise matching and spatial verification over the dataset and selecting inlier features per image [START_REF] Turcot | Better matching with fewer features: The selection of useful features in large database recognition problems[END_REF].

In category-level tasks, object discovery is the task of discovering

From attention to discovery and back.

the categories and localizing foreground objects per category in a fully unsupervised way [START_REF] Sivic | Discovering Object Categories in Image Collections[END_REF]. Using region proposals, one solution is again pairwise spatial matching, interleaved with region selection [START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF].

By representing pairwise region interactions by a graph, it makes sense to use a graph centrality measure to identify regions appearing frequently, hence likely to depict foreground [START_REF] Kim | Unsupervised detection of regions of interest using iterative link analysis[END_REF].

Using CNN representations, our Graph-based Object Discovery (GOD) [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF] is a visual attention mechanism learned in a fully unsuper-Our Graph-based Object Discovery (GOD) [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

vised way on the target dataset for instance-level retrieval. Beginning with a saliency map capturing discriminative patterns based on convolutional activations alone [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF], we discover common patterns by graph centrality on a k-NN region graph. We thus learn a non-parametric model of patterns that are both discriminative and

We detect region proposals from saliency maps using our EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF].

common in the dataset. The result is a global representation that focuses on objects and suppresses background clutter.

structure

Chapter 9 provides background on graph filtering, including notation, definitions and interpretations. This background is then used in Chapters 10, 12, 14, 15 and 17.

Chapter 10 addresses manifold search. It presents two solutions, regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] and Fast Spectral Ranking (FSR) [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF]. It also briefly discusses recent work on revisiting a popular benchmark [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] and a hybrid solution that is more appropriate at large scale [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF].

Chapter 11 revisits spatial matching, now equipped with CNN representations. It presents Deep Spatial Matching (DSM) [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF], which extracts local features and quantized representations directly from CNN activations, without descriptors or vocabularies.

Finally, Chapter 12 addresses unsupervised object discovery from unlabeled image collections. It presents Graph-based Object Discovery (GOD) [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF], which discovers discriminative and frequent patterns and uses them to improve image representation for retrieval.

G R A P H F I LT E R I N G

This chapter provides background on graph filtering, including notation, definitions and interpretations. We define a particular filter that performs smoothing over a graph. This background is used in different ways in Chapters 10, 12, 14, 15 and 17. For instance, we compute a similarity measure between two vertices of a nearest neighbor graph over a dataset. This graph represents a manifold in continuous space, hence we call this measure manifold similarity. Alternatively, we compute a separate similarity measure per class and use it for classification. Finally, we compute a measure of graph centrality and we smooth an image guided by another image.

introduction

Consider an Exponential Moving Average (EMA), given by recurrence

z i := αz i-1 + (1 -α)y i (9.1)
for i ∈ Z. Here, y, z are the input and output respectively. They are discrete-time signals: y i denotes the sample of y at time i, and similarly for z. EMA is an example of a low-pass filter in signal processing [START_REF] Oppenheim | Discrete-Time Signal Processing: Pearson New International Edition[END_REF] and the output z can be regarded as a smoothed version of y.

Parameter α ∈ [0, 1) is a smoothing factor: A lower α discounts older observations faster. What we present in this chapter is a generalization of EMA on graphs, which is the subject of Graph Signal Processing (GSP) [START_REF] Sandryhaila | Discrete Signal Processing on Graphs[END_REF][START_REF] David I Shuman | The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains[END_REF]. The generalization consists in replacing time instances with vertices of a graph and time shift (delay) with an operation called graph shift. This operation replaces the sample at a vertex with a weighted linear combination of the samples at its neighbors.

We use this smoothing operation in different contexts for different purposes. For instance, we compute a similarity measure between two vertices of a nearest neighbor graph over a dataset. In turn, this graph represents a manifold in continuous space, hence we call this measure manifold similarity. Alternatively, we compute a separate similarity measure per class and use it for semi-supervised classification. Finally, we compute a measure of graph centrality and we smooth an image guided by another image.

The nearest neighbor relation is taken as reciprocal (mutual), giving rise to an undirected graph with symmetric adjacency matrix W . This simplifies the formulation and the implementation because W is diagonalizable and a particular linear system may be solved by the Conjugate Gradient (CG) [START_REF] Nocedal | Numerical optimization[END_REF] method.

As discussed by Vigna [START_REF] Vigna | Spectral Ranking[END_REF] and summarized in Section 8.2, this operation has a long history originating in social sciences in the 1950s and the most well-known form is PageRank [START_REF] Page | The PageRank citation ranking: bringing order to the web[END_REF]. Here we mainly follow Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF] who use a single-input operation to rank data on manifolds and Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF] who use a multiple-input operation applied to semi-supervised learning. We also refer to few elements from frequency analysis in GSP [START_REF] Sandryhaila | Discrete Signal Processing on Graphs: Frequency Analysis[END_REF] and spectral graph theory [START_REF] Rk | Spectral graph theory[END_REF].

definitions

graph We are given a weighted undirected graph G with n vertices represented by its n × n symmetric nonnegative adjacency matrix W . The graph contains no self-loops, i.e. W has zero diagonal. We define the n × n degree matrix

D := diag(W 1), (9.2)
where 1 is the all-ones vector, and the symmetrically normalized adjacency matrix

W := D -1/2 W D -1/2 , (9.3)
with the convention 0/0 = 0. We also define the n × n Laplacian L := D -W and normalized Laplacian filtering We define the n × n regularized Laplacian

L := D -1/2 LD -1/2 = I -W. ( 9
L α := (1 -α) -1 (I -αW), (9.6)
where α ∈ [0, 1) is a parameter. This matrix is positive-definite since I -αW = αL + (1α)I αL 0. Then, given an n × 1 observation vector y, the linear system

L α z = y (9.7)
has a unique solution z := h α (W)y, where

h α (W) := (1 -α)(I -αW) -1 . (9.8)
The mapping y → h α (W)y is a linear graph filtering, in particular smoothing of y on G and h α is the transfer function of the filter. Similarly, given an n × c observation matrix Y , the linear system izable:

L α Z = Y (9.
W = U ΛU , (9.10)
where n × n matrices U, Λ hold the eigenvectors and eigenvalues respectively of W. Since U is orthogonal and I -αW and I -αΛ are nonsingular, it follows that

This can be confirmed by direct multiplication of I -αW and

U (I -αΛ) -1 U . h α (W) = U h α (Λ)U . (9.11)
In fact, because Λ is diagonal, h α applies element-wise to the eigenvalues as a scalar function

h α (λ) := 1 -α 1 -αλ (9.12) for λ ∈ [-1, 1]. That is, h α (Λ) = diag(h α (λ 1 ), . . . , h α (λ n )).
Then, given y, the linear mapping

y → U h α (Λ)U y (9.13)
has the following interpretation:

Seen as a signal in a space domain, y is mapped by U to the frequency domain, multiplied element-wise by (h α (λ 1 ), . . . , h α (λ n )), and mapped by U back to the space domain. The columns of U are the Fourier basis of the graph.

This interpretation is understood in the context of frequency analysis in GSP [START_REF] Sandryhaila | Discrete Signal Processing on Graphs: Frequency Analysis[END_REF], where the Discrete Fourier Transform (DFT) [START_REF] Oppenheim | Discrete-Time Signal Processing: Pearson New International Edition[END_REF] is generalized by replacing the DFT matrix by U and its inverse by U . Indeed, the columns of the n-point inverse DFT matrix are the eigenvectors of the n × n cyclic permutation matrix C, which is the adjacency Multiplication by C represents the standard time shift (delay).

matrix of a directed graph having an edge from time i to i + 1 for i ∈ {0, . . . , n -2} and edge from n -1 back to 0 [START_REF] Sandryhaila | Discrete Signal Processing on Graphs: Frequency Analysis[END_REF].

This reveals that function h α is a low-pass filter, as shown in Figure 9.1. By varying α from 0 to 1, the frequency response varies from all-pass to sharp low-pass. Hence the name smoothing.

random walks Consider the iterating process

This interpretation is used in Section 10.3. z (t) := αWz (t-1) + (1α)y.

(9.14)

for t = 1, 2, . . . . Regardless of the choice of z (0) ∈ R n , state z (t) converges to z := h α (W)y as t → ∞ provided αW has spectral radius (αW) < 1 [START_REF] Zhou | Learning with local and global consistency[END_REF], which is indeed the case. Again, α controls how

Recall that ρ(W) = 1 and |a| < 1.
much z is affected by vector y, called boundary condition [START_REF] Vigna | Spectral Ranking[END_REF] in this context: z equals y for α = 0, while in the limit α → 1, z tends to a dominant eigenvector of W. Indeed, for α = 1, (9.14) becomes a power iteration.

Similarly, given an n × c observation matrix Y , the iterating process

Z (t) := αWZ (t-1) + (1 -α)Y (9.15) for t = 1, 2, . . . converges to Z := h α (W)Y as t → ∞.
In the case where W is a row-stochastic transition matrix and x (0) , y express distributions over vertices, process (9.14) can be interpreted as a random walk on a (directed) graph: At each iteration a particle moves to a neighboring vertex with probability α or jumps to a vertex according to distribution y with probability 1-α. This is referred to as Markov chain with restart [START_REF] Boldi | Graph Fibrations, Graph Isomorphism, and PageRank[END_REF] or RWR [START_REF] Pan | Automatic Multimedia Cross-Modal Correlation Discovery[END_REF]. We shall call process (9.14) RWR too, even though W is symmetric in our case.

energy minimization The quadratic energy function

This interpretation is used in Sections 15. 3 and 17.4.

E α (z) := 1 2 z L α z -y z, (9.16)
is minimized at z := L -1 α y, that is, the unique solution of (9.

7). If we

This is evident by letting the gradient ∇zEα(z) = Lαz -y vanish.

expand E α (z) using (1α)L α = αL + (1α)I, we find [START_REF] Zhou | Learning with local and global consistency[END_REF] that it has the same minimizer as

Q α (z) := α 2 i,j w ij ẑi -ẑj 2 + (1 -α) z -y 2 , (9.17)
where ẑ := D -1/2 z. The first pairwise smoothness term encourages z to vary little across edges of the graph with large weight whereas the second unary fitness term to stay close to observation y. Here, α controls the trade-off: z equals y for α = 0, while for α → 1, it tends to be constant over connected components of G.

usage

in the literature Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF] use the RWR process (9.14) to rank data on manifolds. They define the vector y with elements as 1 at queries and 0 elsewhere. After smoothing, the data are ranked by descending order of the elements of the solution z. Similarly, Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF] use the matrix version of the RWR process (9.15) for semisupervised classification. They define the label matrix Y with rows as one-hot labels at labeled examples and 0 elsewhere. Labels are inferred according to row-wise maximum of the solution Z. We refer to this as Label Propagation (LP). Kim et al. [START_REF] Tae Hoon | Generative Image Segmentation Using Random Walks with Restart[END_REF] use the same idea for Label Propagation (LP).

interactive segmentation with pixel-wise label matrix Y defined according to user-specified seeds (strokes) per object.

in this manuscript We use the definitions of this Chapter as follows. In Chapter 10 we use the linear system solution (9.7) and the frequency-domain solution (9.13) to rank images on manifolds. In Chapter 12 we use the linear system (9.7) to compute the Katz centrality [START_REF] Katz | A New Status Index Derived From Sociometric Analysis[END_REF] for unsupervised object discovery. In Chapter 14, we use the linear system (9.7) to compute the manifold similarity of images for unsupervised metric learning. In Chapter 15, we use the matrix version of the linear system (9.9) to build an inductive version of LP [START_REF] Zhou | Learning with local and global consistency[END_REF] for semisupervised learning of a CNN classifier. Finally, in Chapter 17, we use the matrix version of the linear system (9.9) to smooth a perturbation image guided by an input image in generating an adversarial example for a classifier. In all cases, we solve the linear systems by the Conjugate Gradient (CG) [START_REF] Nocedal | Numerical optimization[END_REF] method, which applies because L α is positive-definite. In all cases except Chapter 17, the observation vector y or matrix Y is sparse with nonzero elements at queries or labels. In Chapter 17, it is an arbitrary real-valued signal to be smoothed. harmonic solution For completeness, we mention an alternative version of LP by Zhu and Ghahramani [START_REF] Zhu | Learning From Labeled and Unlabeled Data with Label Propagation[END_REF] for semi-supervised classification, which iteratively propagates labels and clamps the labeled data. As discussed by Zhu et al. [START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF], this is the harmonic solution of a Laplace equation with Dirichlet boundary conditions on the labeled examples. Grady [START_REF] Grady | Random Walks for Image Segmentation[END_REF] uses the same idea [START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF] for interactive segmentation, with labels defined like Kim et al. [START_REF] Tae Hoon | Generative Image Segmentation Using Random Walks with Restart[END_REF]. Pérez et al. [START_REF] Pérez | Poisson Image Editing[END_REF] use the same formulation for image interpolation within some domain with real-values specified on its boundary; they generalize to guided interpolation, giving rise to a Poisson equation.

The version of Zhu and Ghahramani [START_REF] Zhu | Learning From Labeled and Unlabeled Data with Label Propagation[END_REF] retains the labels on the labeled data, assuming they are noise free. By contrast, in the version of Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF], new labels are allowed on labeled examples. This can be useful e.g. when classes overlap or labels are noisy. This version is more useful in ranking, where the observation vector originates in distances or similarities in the feature space and is not to be trusted like human supervision. It makes even more sense when the observation vector is an arbitrary signal.

detailed usage

In the following, we provide more details on how the definitions of this Chapter apply in each case. This material is given for reference only and it is suggested to skip it at least at first reading. ranking data on manifolds Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF] use the RWR iterating process (9.14) to rank data on manifolds. They are given a set of points V := {v 1 , . . . , v n } ⊂ R d , a subset of which are queries. They define the adjacency matrix W = (w ij ) with w ii = 0 and w ij according to a decreasing function of the distance v iv j for i = j. They also define the vector y := (y 1 , . . . , y n ) where y i = 1 if v i is a query and y i = 0 otherwise. They perform RWR until convergence and they use the resulting solution z as a ranking score: They rank each point v i according to z i , greatest first. regional diffusion In Section 10.3, we use the linear system Regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF]. solution (9.7) to rank images on manifolds as above according to a regional CNN representation. We are given a collection of feature vectors V := {v 1 , . . . , v n } ⊂ R d , with either one global or multiple regional vectors per image. We define the adjacency matrix W of a nearest neighbor graph as described in Section 10.2. The queries are not assumed to belong to the collection; rather, we compute y as the sum over queries of the similarities to the k-nearest neighbors in V per query (10.6), where the similarities are defined by (10.1). We solve linear system (10.8) by CG [START_REF] Nocedal | Numerical optimization[END_REF] to obtain z . Each image is associated with several elements of z , one per region; we pool these scores by taking a linear combination per image.

fast spectral ranking In Section 10.4, having the same prob-FSR [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF]. lem as in Section 10.3, we use instead an approximation of the frequency-domain solution (9.13) to obtain z . In particular, we obtain an approximate low-rank Fourier basis by a randomized algorithm [START_REF] Halko | Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions[END_REF][START_REF] Rokhlin | A Randomized Algorithm for Principal Component Analysis[END_REF], approximating W ≈ U r Λ r U r with rank r n, then filter y in the frequency domain by (10.11). The pooling operation from regions to images is integrated with U r offline. object discovery In Section 12.4, we use the linear system solu-GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF]. tion (9.7) to compute the Katz centrality [START_REF] Katz | A New Status Index Derived From Sociometric Analysis[END_REF] of image regions, according again to a CNN representation, and we apply to unsupervised object discovery. In particular, having the same graph as in Section 10.2, we define the all-ones vector y := 1 and we solve the linear system (12.8) by CG to obtain the vector z , where each element z i expresses the centrality of vector v i . In turn, z is used to compute a saliency map per image, according to (12.9). Finally, rectangular salient regions are detected as described by Section 12.3. metric learning In Section 14.2, having the same graph as in MoM [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF].

Section 10.2, we solve the linear system (9.7) to compute the manifold [ October 7, 2020 at 12:04classicthesis version 0.4 ] similarity of an image to all others in the collection. By forming the manifold nearest neighbors for a set of anchor images and comparing to Euclidean nearest neighbors, we generate positive and negative pairs of images to perform unsupervised metric learning. In particular, for each anchor v i in V , we set the vector y := e i , where e i is the standard n-dimensional basis vector and obtain the solution z i of linear system (14.1) by CG. Then, for each vector v j in V , the manifold similarity of v i , v j is read off as the j-th element of z i (14.2). The manifold nearest neighbors of v i in V are the elements of V corresponding to the k greatest elements of z i . label propagation (transductive) Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF] use the matrix version of the RWR iterating process (9.15) and apply it to transductive semi-supervised classification. They are given a set of points V := {v 1 , . . . , v n } ⊂ R d , a subset of which are labeled over c classes. Like Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF], they define the adjacency matrix W = (w ij ) with w ii = 0 and w ij according to a decreasing function of the distance v iv j for i = j. They also define the n × c label matrix Y = (y ij ) with y ij = 1 if v i is labeled in class j and y ij = 0 otherwise. They perform RWR until convergence, obtaining as solution the n × c matrix Z . They use this matrix to infer a label ŷi for each point v i as the class corresponding to greatest element of the i-th row of Z , i.e. ŷi := arg max j z ij (15.5).

label propagation (inductive) In Section 15.4, we use the DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF].

matrix version of the linear system solution (9.9) to build an inductive version of LP [START_REF] Zhou | Learning with local and global consistency[END_REF] above for semi-supervised learning of a parametric CNN classifier. We are given the same collection of feature vectors V and the same graph as in Section 10.2, only now the CNN representation V is updated as the classifier is trained. We are also given the same label matrix Y as above. We solve the linear system (15.6) using the CG method. Using the solution Z , we infer a pseudo-label ŷi := arg max j z ij (15.5) for each unlabeled example v i as in LP. We use those pseudo-labels along with the true labels of the labeled examples and certainty weights (15.7) to train the classifier for one epoch, optimizing the weighted cost function (15.8). By doing so, the feature vectors V and the graph are updated, hence we iterate. adversarial examples In Section 17.5, we use the matrix ver-Smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF].

sion of the linear system solution (9.9) to smooth a perturbation image y ∈ R n×d , guided by an input image x ∈ R n×d , while minimizing a cost function with respect to y in an attempt to generate an adversarial The perturbation is a matrix but denoted by y rather than Y here. example for a given classifier f , starting at input image x with class label t. As discussed in Section 17.2, the graph is defined over the pixels of the input image x according to (17.11). The system is solved by CG and the smooth output, denoted by s α (y), is row-normalized according to (17.12). The cost function (17.15) consists of a distortion term s α (y) 2 and a classification loss term (17.9) such that, when the smooth perturbation s α (y) is added to the input image x, the classi-

S E A R C H I N G O N M A N I F O L D S

Using powerful CNN representations, we explore the manifold structure of the feature space, bringing dramatic gains in standard image retrieval benchmarks. We are the first to study a diffusion mechanism on CNN representations, which can be seen as a recursive form of query expansion. We introduce a number of solutions that differ in the amount of offline pre-processing and space used to accelerate online search. We also introduce a novel view of search as smoothing of a sparse signal on a graph.

introduction

When searching in an image collection, the query is often connected to relevant images by a sequence of images, where pairs of consecutive images are similar. These images form a manifold in the feature space. Query Expansion (QE) [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF][START_REF] Tolias | Visual Query Expansion with or Without Geometry: Refining Local Descriptors By Feature Aggregation[END_REF] has been an early method to exploit this idea. Average Query Expansion (AQE) is common with CNN representations [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF], but only explores a small neighborhood of the query. Diffusion [START_REF] Donoser | Diffusion processes for retrieval revisited[END_REF][START_REF] Page | The PageRank citation ranking: bringing order to the web[END_REF][START_REF] Zhou | Ranking on Data Manifolds[END_REF] is using a k-Nearest Neighbor (k-NN) graph of the dataset constructed offline to efficiently search on the manifold online in a principled way.

The use of a k-NN graph has been prohibitive on conventional repre-

A workaround is to operate on small sub-graphs using text [START_REF] Jing | VisualRank: Applying PageRank to large-scale image search[END_REF].

sentations of thousands of local features per image. In this work, we investigate diffusion on CNN representations of one or few features per image for the first time. Using a number of regional features per image effectively recovers small objects, which are a common failure case of CNN-based retrieval. This regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] incurs no extra cost compared to diffusion on global features. We introduce a novel mechanism to handle unseen queries and we use a closed-form solution that has been avoided so far [START_REF] Donoser | Diffusion processes for retrieval revisited[END_REF], solving a linear system online by the Conjugate Gradient (CG) [START_REF] Nocedal | Numerical optimization[END_REF] method.

We then introduce Fast Spectral Ranking (FSR) [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF], shifting more computation offline: We exploit a low-rank spectral decomposition of We thus reduce manifold search to a two-stage Euclidean/dot product search.

the graph adjacency matrix to express the linear system solution as a sequence of matrix multiplications. Equivalently, we treat the query as a signal to be smoothed over the graph in the frequency domain, connecting query expansion to graph signal processing [START_REF] Sandryhaila | Discrete Signal Processing on Graphs[END_REF]. We provide a truly scalable solution to computing an approximate Fourier basis of the graph offline, accompanied by performance bounds.

nearest neighbor graph

We are given a set of n feature vectors

V := {v 1 , . . . , v n } ⊂ R d ,

with

These definitions are used in Sections 12. 4, 14.2 and 15.4. each v i associated to vertex of graph G. An image collection may be represented by either one global or multiple regional feature vectors per image. By using a single region per image, global feature vectors are a special case of regional ones.

Graph G is a k-Nearest Neighbor (k-NN) similarity graph, where edges are pairs of vectors that are reciprocal (mutual) nearest neighbors [START_REF] Kontschieder | Beyond pairwise shape similarity analysis[END_REF]. In particular, we assume a symmetric non-negative similar-

Given v, v ∈ R d , we use the monomial kernel s(v, v ) = [v v ] 3 + [435].
ity measure s :

R d × R d → R. Given v ∈ R d , let s k (v|v ) := s(v, v ), if v ∈ NN k (v ) 0, otherwise (10.1) be the similarity of v ∈ V to v if v is in the k-nearest neighbors NN k (v ) of v in V , and zero otherwise. Then, if v ∈ V , NN k (v ) excludes v itself. s k (v, v ) := min{s k (v|v ), s k (v |v)} (10.2) equals s(v, v ) if v, v
ij := s k (v i , v j ) for i, j ∈ [n].
We write [n] := {1, . . . , n} for n ∈ N.

10.3 regional diffusion background: diffusion In the work of Zhou et al. [START_REF] Zhou | Ranking on Data Manifolds[END_REF], an observation vector y = (y i ) ∈ R n specifies a set of queries in V , with y i = 1 if v i is a query and y i = 0 otherwise. With this definition of y, the iterative process (9.14) is used

z (t) := αWz (t-1) + (1 -α)y, (10.3)
where W is normalized according to (9.3) and α ∈ [0, 1) is a parameter. We refer to this process as Random Walk with Restart (RWR) [START_REF] Pan | Automatic Multimedia Cross-Modal Correlation Discovery[END_REF]. Zhou et al. [START_REF] Zhou | Learning with local and global consistency[END_REF][START_REF] Zhou | Ranking on Data Manifolds[END_REF] show that regardless of the choice of z (0) , vector z (t) converges to z = (z i ) defined by

z := L -1 α y (10.4)
as t → ∞, where L α is defined by (9.6). This results in a ranking

In this work, we use the closed form solution (10.4) rather than its derivation from (10.3).

score z i for each vector v i ∈ V , expressing a similarity of v i to the set of queries. The benefit is that this similarity captures the intrinsic manifold structure represented by the graph, while multiple queries are combined without additional cost.

handling new queries Prior work on diffusion usually assumes a query vector q ∈ R d to be contained in the dataset V [START_REF] Donoser | Diffusion processes for retrieval revisited[END_REF][START_REF] Zhou | Learning with local and global consistency[END_REF]. This does not hold in a retrieval scenario. A query can be included in the graph at query time [START_REF] Zhang | Query specific fusion for image retrieval[END_REF], but this incurs additional cost, especially to maintain reciprocity (10.2) in the presence of q.

Here we introduce an alternative approach which defines observation vector y in a new way rather than updating the graph. In particular, instead of searching for q, we are searching for its k-nearest neighbors NN k (q) in V , weighted by their similarity to q: A query is also represented by a single vector q ∈ R d .

y i := s k (v i |q) ( 10
the observation vector y by (10.5) is referred to as global diffusion.

Figure 10.1(a) shows a toy 2-dimensional example, where the knearest neighbors of q taken into account in (10.5) are depicted.

regional diffusion While global diffusion fits perfectly with the early CNN-based global features [START_REF] Babenko | Aggregating Deep Convolutional Features for Image Retrieval[END_REF][START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], it may still fail under severe occlusion or when the object of interest is small. Local CNN features from multiple regions have been investigated for this purpose, either aggregated [START_REF] Gong | Multi-scale orderless pooling of deep convolutional activation features[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] or represented as a set [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF].

Following the latter choice, an image is represented by a set of m feature vectors in R d , one for each region. Dataset V is the union of such sets over all images; n still denotes its size. The query image is also represented by a set Q of m vectors, and (10.5) becomes

Computing y involves searching for each query q individually in V .

y i := q∈Q s k (v i |q) (10.6) for i ∈ [n]. Each v i ∈ V is
assigned a scalar that is the sum of similarities over all queries q of which v i is a k-nearest neighbor in V , and zero if it is not a k-nearest neighbor of any query. Given y, diffusion (10.4) is now performed jointly for all queries in q ∈ Q, at no additional cost compared to (10.5). After diffusion, each image is associated with several elements of the ranking score vector z , one for each region. A linear combination of these scores is taken, by average Details are given in [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

pooling or Generalized Max Pooling (GMP) [START_REF] Iscen | Memory vectors for similarity search in high-dimensional spaces[END_REF][START_REF] Murray | Generalized max-pooling[END_REF]. The above process is called regional diffusion. efficient solution Iteration (10.3) is slow at large scale. To use the closed-form solution (10.4), one may compute the inverse L -1 α offline, but this matrix is not sparse like L α . Connecting (10.3) to linear system solvers, we propose a more efficient solution here.

Diffusion is an iterative solver. Eq. ( 10.3) is in fact an iteration of the Jacobi solver [START_REF] Hackbusch | Iterative solution of large sparse systems of equations[END_REF]. Given a linear system Ax = b, Jacobi decomposes A as ∆ + R, where ∆ = diag(A) and iterates according to it follows that ∆ = I n and R = -αW, re-deriving (10.3).

x (t) := ∆ -1 (b -Rx (t-1) ). ( 10
CG [START_REF] Nocedal | Numerical optimization[END_REF] is the method of choice for linear systems like (9.7)

L α z = y, (10.8)
where L α is positive-definite, in particular for graph-related problems [START_REF] Nisheeth | Laplacian Solvers and Their Algorithmic Applications[END_REF]. It has been used for random walk problems [START_REF] Grady | Random Walks for Image Segmentation[END_REF], but explicitly avoided in diffusion-based retrieval [START_REF] Donoser | Diffusion processes for retrieval revisited[END_REF].

Here we argue, as in [START_REF] Amy | Deeper Inside PageRank[END_REF], that it is the solution of (10.4) that we

In fact, CG minimizes the quadratic function Eα(z) (9.16) over a family of Krylov subspaces [START_REF] Lloyd N Trefethen | Numerical linear algebra[END_REF].

seek, rather than the path followed by iteration (10.3). We use CG to approximate this solution. Contrary to other iterative methods including (10.3), CG terminates in n steps. Remarkably, it provides good approximations in very few steps.

scaling up Here we address issues concerning space and online processing at large scale.

Compact representation. To keep the number of region features per

This also reduces the redundancy of region features.

image as low as possible, we learn a GMM on the original features of each image and represent the image by the 2 -normalized means.

Truncating the adjacency matrix. We first search through the dataset using global descriptors. We then truncate W , keeping only the rows Thus, diffusion only re-ranks the top ranked images.

and columns corresponding to the regions of the top ranked images, and re-normalize it by (9.3). We similarly truncate vector y.

fast spectral ranking

As we have seen in Section 9.2, we can write the solution of (10.8) as

z * := h α (W)y, (10.9)
where the transfer function h α (W) = L -1 α is defined by (9.8). The problem is then to compute z efficiently, in the sense that h α (W) is never explicitly computed or stored: W is given in advance and we can preprocess it offline, while y is given online. In particular, we are looking for a more efficient solution than solving linear system (10.8).

We are based on the spectral decomposition (9.11), whereby

z * = U h α (Λ)U y. (10.10)
Here U, Λ represent the eigenvectors and eigenvalues of W, respec-Graph spectral filtering is well-known [START_REF] Sandryhaila | Discrete Signal Processing on Graphs[END_REF][START_REF] David I Shuman | The Emerging Field of Signal Processing on Graphs: Extending High-Dimensional Data Analysis to Networks and Other Irregular Domains[END_REF], but search as smoothing a sparse signal is a new view.

tively. A low-rank approximation involving the leading eigenvector and eigenvalues can indeed be computed offline. Under the spectral filtering interpretation of Section 9.3, (10.10) represents a smoothing operation of the sparse signal y on graph G in the frequency domain, and U is the Fourier basis of the graph. Figure 10.2 depicts 1d and graph miniatures of this interpretation. offline: fourier basis Our solution is based on a low-rank ap-This is similar to Nyström sampling [START_REF] Drineas | On the Nyström Method for Approximating a Gram Matrix for Improved Kernel-Based Learning[END_REF] but with performance guarantees [START_REF] Halko | Finding Structure with Randomness: Probabilistic Algorithms for Constructing Approximate Matrix Decompositions[END_REF][START_REF] Witten | Randomized Algorithms for Low-Rank Matrix Factorizations: Sharp Performance Bounds[END_REF].

proximation of W computed offline. The approximation is based on a randomized algorithm [START_REF] Rokhlin | A Randomized Algorithm for Principal Component Analysis[END_REF]. In the following, rank r n and number of iterations q are given parameters.

(Range basis)

Using simultaneous iteration [442, §28], find an n × r matrix Q with orthonormal columns that represents an approximate basis for the range of W, i.e. QQ W ≈ W. In particular [151, §4.5]: Randomly draw an n × r standard Gaussian matrix B (0) and repeat for t = 0, . . . , q -1:

a) Compute QR factorization Q (t) R (t) = B (t) .
Stage 1 is embarrassingly parallelizable.

b) Define the n × r matrix B (t+1) := WQ (t) .

Finally, set Q := Q (q-1) .

(Fourier

WQQ = QCQ = QV r Λ r V r Q = U r Λ r U r [151, §9.4].
online: spectral filtering Given y, compute z := U r h α (Λ r )U r y. (10.11) We are actually approximating h α (W) by U r h α (Λ r )U r . Therefore, it is

A similar situation appears in [440, §3.3].

|h α (λ r+1 )| that governs the error rather than |λ r+1 |. Since we are using the leading eigenvectors and eigenvalues of W, this approximation makes sense because h α is nondecreasing, as shown in Figure 9.1.

Vector z ∈ R n contains the ranking score z i of each region feature v i . To obtain a score per image, we perform a linear pooling operation [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] represented as z := Σz, where Σ is a sparse N × n pooling matrix and N is the number of images. We then directly compute z := U r h α (Λ r )U r y online, where the N × r matrix U r := ΣU r is computed offline.

Computing y involves Euclidean search in R d . Then, U r projects y

The number of nonzero elements of y and rows of U r , whence the cost, are the same for global or regional search.

onto R r . With Λ r being diagonal, h α (Λ r ) is computed element-wise as discussed in Section 9.3. Finally, multiplying by U r and ranking z amounts to a dot product similarity search in R r .

We thus reduce manifold search to Euclidean followed by dot product search.

experiments setup We compare Euclidean to manifold search with global or

We apply supervised whitening [START_REF] Mikolajczyk | Improving Descriptors for Fast Tree Matching By Optimal Linear Projection[END_REF] to both global and regional features, as in [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF].

regional CNN features. We use d-dimensional features by VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] (d = 512) and ResNet101 [START_REF] He | Deep residual learning for image recognition[END_REF] (d = 2,048), fine-tuned for image retrieval [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF]. Global features are R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] at 3 scales; regional use the same 21 regions per image without pooling, optionally reduced to 5 by GMM. Regional features are matched pairwise as in Regional Matching (R-Match) [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF]. The similarity measure is s(v, v ) := [v v ] 3 + . We use GMP [START_REF] Iscen | Memory vectors for similarity search in high-dimensional spaces[END_REF][START_REF] Murray | Generalized max-pooling[END_REF] to pool regional diffusion scores per image. We set α = 0.99, and k = 50 [START_REF] Jégou | Negative Evidences and Cooccurences in Image Retrieval: The Benefit of PCA and Whitening[END_REF] for global (regional) diffusion. We use Oxford5k [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], Paris6k [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF] and Instre [START_REF] Wang | INSTRE: a new benchmark for instance-level object retrieval and recognition[END_REF] benchmarks.

We introduce a new evaluation protocol for Instre.

For large-scale experiments, we add 100k distractor images [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] to Oxford5k and Paris6k, referred to as Oxford105k and Paris106k respectively. We measure performance by mAP. Times exclude construction of the observation vector y. results Figure 10.3 compares our CG solution to iterative diffusion (10.3). CG converges in as few as 20 iterations, while (10.3) reaches the same performance only after 110 iterations. Figure 10. 4 shows that in all datasets, the optimal FSR performance is already reached at rank r = 1k. On Paris6k in particular, this happens as soon as r = 100. FSR reaches the same mAP as CG, 150 times faster on Oxford5k and Paris6k and 300 times faster on Instre.

Table 10.1 compares our diffusion Euclidean search and AQE [START_REF] Chum | Total Recall: Automatic Query Expansion With a Generative Feature Model for Object Retrieval[END_REF].

Regional diffusion significantly outperforms all other methods in all

AQE is common with global representation [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]; we extend it to regional. datasets. Global diffusion performs well on Paris because query objects mostly cover the image. This does not hold on Instre, which contains a lot of small objects. FSR performs similarly to CG but at dramatic speed-up, almost as fast as Euclidean search: Dataset truncation is no longer needed and this improves mAP.

discussion

The power of CNN features allows representing an image by just one or a few vectors. This, combined with our efficient solutions, allows for the first time to perform large scale diffusion at reasonable query times. Its effect is dramatic, especially when images contain small objects. The closed form solution, approximated by CG, is significantly faster than the iterative process RWR (10.3); but FSR offers a dramatic speed-up, at the expense of space for the eigenvectors U . [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF] (R-Match [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF]) for global (regional) representationand AQE, using ResNet101 [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF] features (d = 2,048). Regions per image reduced from m = 21 to 5 by GMM. Truncation at top 10k images at large scale. Rank r = 5k for FSR.

The excellent performance of our methods motivates us to revisit Oxford [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and Paris [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF], introducing the Revisited Oxford and Revisited Oxford and Paris (RevOP) [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF].

Paris (RevOP) benchmark [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF]. We provide new annotation for both datasets, correcting previous errors. We introduce 15 new, more difficult queries per dataset and update the evaluation protocol by introducing three new protocols of varying difficulty. We also create a new set of one million challenging distractors. At this scale, it turns out that CG either is too slow or incurs loss by truncation, while FSR either takes too much space or fails at lowrank approximations. We therefore introduce hybrid diffusion [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF], al-Hybrid diffusion [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF].

lowing full control of the space-time trade-off between these two extremes. This approach performs on par with the state of the art, with lower space compared to FSR and faster than CG.

Diffusion code can be found on github 1 , as well as the revisited Oxford and Paris benchmark 2 . Our version of the Instre dataset is also available online 3 . https://github.com/ahmetius/diffusion-retrieval/ https://github.com/filipradenovic/revisitop/ ftp://ftp.irisa.fr/local/texmex/corpus/instre/instre.tar.gz

S PAT I A L M AT C H I N G

Observing that convolutional activations of deep networks are sparse and consistent across different views of a scene, we approximate activation tensors by collections of local features, which we robustly match to find the optimal alignment of two views. We thus introduce Deep Spatial Matching (DSM) [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF] for image retrieval. This happens without any network modification, additional layers or training and without using local descriptors or visual vocabularies.

introduction

Image retrieval based on hand-crafted local descriptors [START_REF] Nistér | Scalable Recognition With a Vocabulary Tree[END_REF][START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] typi-A 3d activation tensor can be perceived as a collection of 2d response maps of pattern detectors.

cally relies on Bag of Words (BoW) [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] or aggregated descriptors [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF], followed by spatial verification [START_REF] Perdoch | Efficient Representation of Local Geometry for Large Scale Object Retrieval[END_REF][START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF]. CNN-based retrieval [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] mostly relies on global or regional spatial pooling of the 3d activation tensor [START_REF] Babenko | Aggregating Deep Convolutional Features for Image Retrieval[END_REF][START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF], which yields a compact and invariant representation like BoW, but does not allow spatial verification.

It is known that dense correspondence can be recovered by corre-Regional pooling followed by pairwise matching was the first to beat conventional retrieval pipelines [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF].

lating activation tensors [START_REF] Christopher B Choy | Universal correspondence network[END_REF][START_REF] Long | Do Convnets Learn Correspondence?[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF][START_REF] Rocco | Neighbourhood Consensus Networks[END_REF]. Combined with integral image computation, regional pooling allows fast sliding window-style spatial matching [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]. However, the activation tensor is too large to be stored, especially for large-scale applications. Sparse local feature representations are possible by imitating conventional detector pipelines [START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF][START_REF] Mikolajczyk | Scale & Affine Invariant Interest Point Detectors[END_REF]. Two dominant paradigms are detect-then-describe, LIFT [START_REF] Moo | LIFT: Learned invariant feature transform[END_REF]? (c) DELF [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF]? Or (d) local maxima on vanilla activation maps, without descriptors or vocabularies?

as in Learned Invariant Feature Transform (LIFT) [START_REF] Moo | LIFT: Learned invariant feature transform[END_REF], and describe-thendetect, as in DEep Local Features (DELF) [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF]. Given enough memory, and using conventional vocabularies, inverted files and spatial verification, DELF is state of the art [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], but not compatible with the global representations used for search. In this

We use no local descriptors or vocabularies.

work [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF], we attempt to reduce this gap by extracting from CNN activations a representation that allows spatial verification, yet it has a trivial relation to global representations: Instead of pooling, we detect local maxima. Referring to Figure 11.1, the correct answer is (d):

We "read off" information directly from activation maps.

Figure 11.2: Three views (columns) of a scene, overlaid with two feature maps (rows) of the last convolutional layer of VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF]. All activations are sparse and spatially consistent across views.

motivation

The success of max-pooling of convolutional activations (MAC) [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF],

Sparsity is studied in Cross-Dimensional Weighting (CroW) [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF], which only provides a coarse saliency map.

at least for image retrieval, can be connected to the sparsity of the activations [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF]. More interestingly, the locations of maxima can identify correspondences [7,[START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]. Generalized mean pooling (GeM) [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] is superior, which can be attributed to the fact that it allows for more than one locations contributing to the representation, while still being more selective than average pooling. As illustrated in Figure 11.

2, in most cases activation maps in all

We observe all channels, not just those contributing most to image similarity [7,[START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF].

channels are not just sparse. They also respond at consistent locations with consistent local shape across views, exhibiting translation and scale covariance to some extent. Hence the question:

Instead of reducing each channel to a single scalar, why not keep all the peaks of the responses in each channel along with geometric information? Instead of attaching an entire descriptor to each such geometric entity, why not just attach the channel it was extracted from, as if it was a visual word?

deep spatial matching

The preceding ideas give rise to the DSM architecture, illustrated in Figure 11.3. We consider a fully convolutional network f that, when followed by spatial pooling e.g. MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] or GeM [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF], extracts a global feature for retrieval [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF]. Two input images x 1 , x 2 yield

We see an activation tensor a collection of 2d maps rather then a 2d array or feature vectors, as in image registration [START_REF] Christopher B Choy | Universal correspondence network[END_REF][START_REF] Long | Do Convnets Learn Correspondence?[END_REF], optical flow [START_REF] Dosovitskiy | FlowNet: Learning optical flow with convolutional networks[END_REF] or semantic alignment [START_REF] Kim | FCSS: Fully convolutional self-similarity for dense semantic correspondence[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF].

3d activation tensors A 1 := f (x 1 ), A 2 := f (x 2 ) at the last convolutional layer of f , where A i ∈ R w i ×h i ×c , w i × h i is the spatial resolution of A i for i = 1, 2 and c is the number of channels (features).

From each activation tensor A 1 , A 2 , feature detector g extracts a sparse collection of local features P 1 := g(A 1 ), P 2 := g(A 2 ) respectively, independently per channel. Then, P 1 , P 2 undergo spatial matching, returning a collection of inliers and a geometric transformation, where tentative correspondences are formed again independently per channel. In retrieval, activation tensors are discarded and images are represented by local features alone.

The entire mechanism takes place without adapting the network and without any additional learning. gions (MSER) [START_REF] Matas | Robust wide baseline stereo from maximally stable extremal regions[END_REF] over activation map A ( ) of tensor A independently for each channel = 1, . . . , c. These are connected regions of arbitrary shape having higher activation than their neighborhood and satisfying a stability criterion. local feature representation Each MSER detected in channel gives rise to a local feature p with activation strength a(p) := pool r∈R(p) A ( ) (r), where R(p) is the MSER, pool is a spatial pooling operation and A ( ) (r) the element of A ( ) at position r. We also fit an

For instance, maximum, mean, or GeM.

ellipse with 2 × 1 mean (position) vector µ(p) and 2 × 2 covariance matrix (local shape) Σ(p). We collect local features P = (P (1) , . . . , P (c) ), where P ( ) contains the local features p found in channel .

correspondences Given the local features P 1 , P 2 of two images x 1 , x 2 , we form correspondences, i.e. pairs (p 1 , p 2 ) of local features of the two images. We allow pairs only between local features in the same channel, that is, p 1 , p 2 are in P ( )

1 , P ( )
2 respectively for some channel . The collection (P (1)

1 × P (1) 2 , . . . , P (c) 1 × P (c)
2 ) of all such pairs is the set of tentative correspondences.

We thus treat channels as visual words, as if local features were assigned descriptors that were vector-quantized against a vocabulary.

Feature channels are correlated: One filter may respond to a variety of patterns, while several filters may respond to the same pattern. For this reason we apply Non-Maximum Suppression (NMS) over channels on detected regions of database images. We do not apply NMS to the query image to allow matches from any channel. spatial matching We use FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] to find the geometric transformation between the two images and the subsets of P 1 , P 2 that are consistent with it. FSM is a variant of RANSAC [START_REF] Fischler | Random Sample Consensus: A Paradigm for Model Fitting With Applications to Image Analysis and Automated Cartography[END_REF] that generates a

As in [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], we use LO-RANSAC [START_REF] Chum | Locally Optimized RANSAC[END_REF], which iteratively evaluates promising hypotheses by least squares-fitting of a transformation to all inliers.

transformation hypothesis from a single correspondence. A hypothesis is evaluated by the number of inliers, that is, correspondences that are consistent with it. All possible hypotheses are enumerated and the transformation with the most inliers is returned. We adopt the linear 5-DoF transformation allowing for translation, anisotropic scale and vertical shear but no rotation. Images are assumed in "upright" orientation: Given a correspondence of two local features p 1 , p 2 , one finds from the ellipses defined by (µ(p 1 ), Σ(p 1 )), (µ(p 2 ), Σ(p 2 )) the transformations T 1 , T 2 that map them to the unit circle while maintaining the y-direction, and defines the transformation hypothesis T = T -1 2 T 1 .

retrieval and re-ranking In image retrieval, n database images X = {x 1 , . . . , x n } are given in advance. For each image x i with activation tensor A i , its local features P i := g(A i ) are computed along with a global descriptor v i spatially pooled directly from A i , which is then discarded. At query time, given query image x with activation tensor A, local features P := g(A) and global descriptor v, we first rank {v 1 , . . . , v n } by cosine similarity to v, and then the top-ranking images undergo spatial matching against P and re-ranked according to the number of inliers found.

We apply query-time diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] after spatial re-ranking. The precision of top-ranking images is important for diffusion [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], so spatial re-ranking is expected to help.

experiments

setup We use VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] and ResNet101 [START_REF] He | Deep residual learning for image recognition[END_REF] as trained by [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] with GeM pooling, denoted by VGG (ResNet), or V (R) for short. We also re-train them, denoted by , with max-pooling (MAC) [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF], using the setup of [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF]. We upsample ResNet by a factor of 2, denoted by ↑, without re-training.

We initially rank images by cosine similarity on multi-scale global

We apply supervised whitening [START_REF] Mikolajczyk | Improving Descriptors for Fast Tree Matching By Optimal Linear Projection[END_REF] to global features, as in [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF].

representations extracted at scales related by factors 1, 1/ √ 2, and 1/2, and spatially pooled then pooled over scales by MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] or GeM [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF]. We use the VLFeat [START_REF] Vedaldi | VLFeat: An Open and Portable Library of Computer Vision Algorithms[END_REF] [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], using DELF [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF], ASMK* [START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF], FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and diffusion (D) [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] on graph obtained by [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF]. Remaining results use diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] on our global features; GeM as trained by [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF].

We perform spatial verification of the 100 top-ranked images according to global representations with transfer error threshold of 2 pixels in the activation map, and re-rank according to the number of inliers. We compute the product of the global cosine similarity and number spatial matching of inliers on the 10 top-ranking spatially verified images and initiate global diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] on top 5 images according to this product.

Diffusion is very powerful but sensitive

to the quality of the initial top-ranked results, which we improve.

We evaluate performance by mAP and mP@10 on the medium and hard setups of the revisited ROxf and RPar benchmarks [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], as well as the large-scale benchmarks ROxf+R1M and RPar+R1M, combining a set of 1M distractor images with the two small ones. results Table 11.1 only shows results on ROxf and ROxf+R1M using diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF]. All baselines are significantly improved by reranking, with a gain of up to 5 mAP or 6 mP@10 points.

We also compare to the best performing version of DELF [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF] as evaluated by [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], which we outperform in several cases. Apart from spatial verification by [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] on the 100 top-ranking images, this version is using two independent representations. One is our ASMK* [START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF] on 128-dimensional descriptors of 1000 DELF features per image, used for initial ranking. Another is a global descriptor by ResNet-R-MAC [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF], used for diffusion as in this work. By contrast, our global and local representations are obtained from the same activation tensor, and we do not use any local descriptors or vocabularies.

discussion

Our representation arises naturally in existing convolutional activations of off-the-shelf or fine-tuned networks, without particular effort to detect local features or extract descriptors on image patches. It does not require network modification or retraining. It is a significant step towards bridging the gap between global descriptors, which are efficient for initial ranking by nearest neighbor search, and local representations, which are compatible with spatial verification.

Of course, the activation channels are not the most appropriate by construction to replace a visual vocabulary. Our representation, while being very compact, is not as powerful as storing e.g. hundreds of local descriptors per image. Nonetheless, we demonstrate that it is enough to provide high-quality top-ranking images to initiate diffusion, which then brings excellent results.

DSM code can be found on github1 .

D I S C O V E R I N G O B J E C T S

Background clutter is challenging for image retrieval, especially when using global representations. In this work [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF], we detect salient regions in an unsupervised manner, capturing discriminative and frequent patterns. Saliency is based on a centrality measure of a nearest neighbor graph of regional CNN representations on an image collection. Pooling features over salient regions improves retrieval performance, mostly for small objects.

12.1 introduction

Particular object retrieval becomes very challenging when the object

Large objects may be partially occluded, while small may be on background clutter covering most of the image.

of interest is covering a small part of the image. Representations based on local features [START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] are naturally insensitive to occlusion and background clutter. Locality allows matching small parts of image containing the object of interest, while the incorrect matches are removed by spatial verification [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF].

CNN representations have dominated modern image retrieval because they are very discriminative and have excellent performance, while being very compact [START_REF] Babenko | Aggregating Deep Convolutional Features for Image Retrieval[END_REF][START_REF] Babenko | Neural Codes for Image Retrieval[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]. Using several regions per image [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF][START_REF] Salvador | Faster R-CNN features for instance search[END_REF], local features [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF][START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF], or the entire activation tensor [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] can provide robustness to clutter, but is expensive. However, when the object is small, global CNN features fail [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

Ideally, the representation should focus on the relevant part of the image, suppressing clutter and occlusions. Deep networks learn, to some extent, what is discriminative in an image. This can be used for saliency detection [START_REF] Jimenez | Class-Weighted Convolutional Features for Visual Instance Search[END_REF][START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF][START_REF] Mohedano | Saliency Weighted Convolutional Features for Instance Search[END_REF][START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF]. Alternatively, methods inspired by object detection are using a number of region proposals [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Reddy | Object level deep feature pooling for compact image representation[END_REF][START_REF] Salvador | Faster R-CNN features for instance search[END_REF].

In both cases, a global representation is obtained by pooling.

In this work [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF], we introduce a pooling mechanism inheriting the properties of both saliency detection and region proposals. It applies to each image independently, focusing on discriminative parts. We also introduce an unsupervised Graph-based Object Discovery (GOD) mechanism, considering the image collection as a whole and capturing Our approach outperforms more expensive regional representations [START_REF] Ali S Razavian | Visual instance retrieval with deep convolutional networks[END_REF], even in the presence of diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

frequent objects. As shown in Figure 12.1, we find what is relevant for the dataset, which is not possible by considering images independently.

In both cases, we derive a powerful global representation.

feature saliency

Before looking for frequent patterns in a dataset, we first need to find

Otherwise, we risk "discovering" uninformative but frequently appearing "stuff"-like patches, e.g. sky.

discriminative regions. Fortunately, this is possible without training or bounding box annotations. Inspired by Cross-Dimensional Weighting (CroW) [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF], we construct a 2d Feature Saliency (FS) map of an image based on its convolutional activation alone.

We represent an activation map as a 3d tensor A ∈ R w×h×c , where w, h, c are the height, width and number of feature channels, respectively. F according to [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF]. Figure 12.2(b) shows FS examples.

salient regions

Given a 2d saliency map S ∈ R w×h , either FS (Section 12.2) or OS (Section 12.4), the next step is to detect a small set of rectangular regions per image, allowing us to build a region k-NN graph in Section 12.4.

Each region is associated with a FS score and a feature vector.

region detection We use the Expanding Gaussian Mixture (EGM) model [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] discussed in Section 3.3 to detect a number of salient rectangular regions. The original algorithm applies to point sets and isotropic Gaussian components. Here we extend it to functions, considering that a saliency map S can be seen as a function S : Ω → R. We fit a number of components, each modeling a region in 2d coordinate space. We also extend it to a diagonal covariance model, so that a region is a modeled by an axis-aligned rectangle.

In particular, given 2d saliency map S ∈ R w×h , we represent it as a set of Gaussian sample functions q(r) : R 2 → R with q(r)(x) := S(r)N (x|r, σI)

(12.3) for r ∈ Ω, x ∈ R 2 ,
where N is the normal density and σ is a scale σ determines how coarse or fine the representation is.

parameter. Similarly to (3.7), we represent components as Gaussian functions p j : R 2 → R with

Mean µj represents the center, while the square root of eigenvalues of Σj represent the height and width of region (component) pj.

p j (x) := π j N (x|µ j , Σ j ) (12.4)

for j ∈ [k], x ∈ R 2 ,
where k is the number of components and π j ∈ R, µ j ∈ R 2 and Σ j ∈ R 2×2 are the mixing coefficient, mean and diagonal covariance matrix respectively of component j. We initialize components as {q(r) : r ∈ Ω}, with k ← r. In the expectation (E)-step, following (3.11), we find the generalized responsibility f, g is the L 2 inner product of f, g : R d → R, given in closed form for Gaussian functions by (3.9). γj (r) ← q(r), p j k =1 q(r), p

(12.5) of component j ∈ [k]
for sample q(r), r ∈ Ω. In the maximization (M)-step, we update π j ← r j r , µ j ← 1 r j r∈Ω γj (r)r, and

Σ j ← 1 r j r∈Ω γj (r) diag ((r -µ j ) • (r -µ j )) , (12.6)
where r j := r∈Ω γj (r) is the effective number of points assigned to component j and • is the Hadamard product. Finally, in the purge (P)k may decrease at each iteration, and overlap is measured similarly to (12.5).

step, similarly to NMS, we process components by Algorithm 3.1 in descending order of mixing coefficient and keep a component unless it overlaps with previously kept ones. Figure 12.3 shows how regions are formed during EGM iterations.

Although less salient, background regions cannot be removed based on FS alone.

We get six clean regions on the ground truth building and five regions on background objects. More examples of region detection on FS and OS maps are shown in Figure 12.2.

region representation Given a region R of an image with feature saliency map F ∈ R w×h and activation map A ∈ R w×h×c , we associate it with feature saliency f := 1 |R| r∈R F (r) by average pooling of F over R, and with feature vector v := φ A (R) ∈ R d , where

A(r) ∈ R c collects elements of all channels of A at position r. φ A (R) := ω max r∈R A(r) (12.7)
denotes max-pooling [START_REF] Azizpour | From generic to specific deep representations for visual recognition[END_REF][START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] of A over R followed by supervised R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] uses uniformly sampled regions at different scales, while ours are based on saliency.

whitening ω : R c → R d . The latter is performed by [START_REF] Mikolajczyk | Improving Descriptors for Fast Tree Matching By Optimal Linear Projection[END_REF], as in [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF].

object saliency

Given an image dataset, we detect a set of regions {R 1 , . . . , R n } from FS maps and we extract a feature saliency vector f := (f 1 , . . . , f n ) ∈ R n from FS maps and a set of feature vectors V := {v 1 , . . . , v n } ⊂ R d from activation maps. Based on this information, we construct a k-NN Region centrality expresses how often the depicted pattern appears in the dataset.

graph of the regions and compute a centrality [START_REF] Newman | Networks: an introduction[END_REF] score per region. This allows us to form a dense Object Saliency (OS) map, capturing discriminative patterns appearing frequently in the dataset.

graph centrality We form the adjacency matrix W of the k-NN of V according to Section 10.2. Then, following the definitions of Chapter 9, we normalize it into W by (9.3) and compute the n × n regularized Laplacian L α by (9.6), where α ∈ [0, 1). The objective is to compute a vector z ∈ R n where each element z i represents the significance of vertex v i in the graph, for i ∈ [n]. We define this centrality As in Chapter 10, we solve this system by the CG [START_REF] Nocedal | Numerical optimization[END_REF] method, but this takes place offline.

vector as the solution z * ∈ R n of the linear system

L α z = 1.
(12.8)

The solution z * is a graph centrality measure [START_REF] Newman | Networks: an introduction[END_REF], Katz centrality [START_REF] Katz | A New Status Index Derived From Sociometric Analysis[END_REF] in particular. Its history is summarized in [START_REF] Vigna | Spectral Ranking[END_REF].

saliency map The problem is then to compute a dense OS map This is a regression problem and we use a non-parametric k-NN solution.

O ∈ R h×w for a new image represented by activation map A ∈ R h×w×c . The value O(r) at r ∈ Ω is found as a linear combination of the centrality values of the k-NN in V of a square patch P (r) centered at r, weighted by similarity and feature saliency

O(r) := F (r) γ n i=1 s k (v i |φ A (P (r)))f γ i z * i , (12.9)
where similarity s k is defined by (10.1). The sum is weighted by FS,

The sum is computed over NN k (φA(P (r))) only in practice.

hence OS captures both discriminative and frequent patterns. Exponents γ and γ control the importance of feature saliency of the current patch and the centrality of neighbors, respectively. Figure 12 image with activation map A. For each region R ∈ R, we apply spatial max-pooling of A and 2 -normalization, denoted by η. We obtain a global representation φ A (R) by summing over R and whitening:

φ A (R) := ω R∈R η max r∈R A(r) (12.10)
The difference from (12.7) is that we apply whitening on the aggre-This is the same as R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF], but our regions are detected by saliency rather than uniformly distributed.

gated vector and not separately per region. 12.5 experiments setup We evaluate our global representation on regions detected by EGM on FS and OS maps, denoted as FS.EGM and OS.EGM, respectively. We use the revisited ROxford and RParis benchmarks [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] as well as Instre [START_REF] Wang | INSTRE: a new benchmark for instance-level object retrieval and recognition[END_REF] with the protocol of [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF]. We apply detec-

Regions at different scales form new vertices in the graph;

the final image representation is pooled over scales.

tion to dataset images only, using the provided bounding boxes on queries. We use VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] with GeM [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] pooling, fine-tuned by [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF].

We adopt a multi-scale approach [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF]. We set σ = 2.5 (12.3), α = 0.99 (9.6), γ = 2 and γ = 3 (12.9). We search by cosine similarity and manifold similarity, using global diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

Using the bounding box annotation of Instre, we evaluate the qual- [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] similarity.

results Figure 12.4 shows histograms of precision of FS and OS maps on Instre. The improvement of OS is impressive. Table 12.1 compares against GeM [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF]. OS.EGM boosts performance in most cases. The gain is more pronounced on Instre, which contains small objects on severe background clutter.

discussion

Our region detection approach is dataset specific but requires no supervision and no learning other than computing the k-NN graph centrality. We precisely localize objects without ground truth bounding boxes. We avoid indexing of regional features: Our global features perform well under severe background clutter and occlusions. After its introduction [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF], we have extended GOD to use multi-scale GeM [START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] representation and shown that Katz centrality performs best among a number of alternatives [START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

Part III L E A R N I N G

Building on deep network models, we address learning visual representations by exploring data, focusing on solutions assuming only limited or no supervision. We progress from instance-level to category-level tasks and conclude with a study of the sensitivity of models to their input.

O U T L I N E

This chapter serves as an outline or road map of Part iii. We present historical and more recent background on learning with different levels of supervision developed in 2014 or later, after the deep learning outburst. In this context, we position our own contributions developed in 2018-2019. Our work addresses learning visual representations by exploring data, focusing on solutions assuming only limited or no supervision. It progresses from instance-level to category-level tasks and concludes with a study of the sensitivity of models to their input. We outline the structure of Part iii in terms of methods, key publications and corresponding chapters.

context

After or during the development of ideas leading to training deep architectures from scratch in 2012-2016 as outlined in Section 8.1, deep learning is established as the framework of choice for most computer vision tasks. The process commonly involves collecting and annotating task-specific datasets and optionally adapting architectures and defining new loss functions. The network backbone remains the same with few exceptions like feature pyramids [START_REF] Lin | Feature pyramid networks for object detection[END_REF] for dense tasks or entirely new designs for different inputs, e.g. 3d point sets [START_REF] Hua | Pointwise convolutional neural networks[END_REF].

With most of the research community relying on and improving the same framework, the period from 2014 until today sees unprecedented productivity and progress. All components of the framework are being scrutinized and ideas spread as a matter of days. What is interesting is that further increase of the network depth and the amount of annotated visual data is becoming less important than other trends. We focus here on three trends:

1. learning with less supervision; 2. the convergence of category-level and instance-level tasks; and 3. research directions going from niche to mainstream.

Unsupervised learning is historically common practice for individ-

Reducing supervision.

ual components or initialization of the representation. Learning of vocabularies for BoW [START_REF] Csurka | Visual Categorization With Bags of Keypoints[END_REF][START_REF] Sivic | Video Google: A Text Retrieval Approach to Object Matching in videos[END_REF] is unsupervised. Neocognitron [START_REF] Fukushima | Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected By Shift in Position[END_REF] and layer-wise pre-training [START_REF] Bengio | Greedy Layer-Wise Training of Deep Networks[END_REF][START_REF] Hinton | A Fast Learning Algorithm for Deep Belief Nets[END_REF][START_REF] Marc | Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition[END_REF] are unsupervised too. Backpropagation allows end-to-end learning of the entire representation, but for this representation to be of high quality, full supervision is common, requiring large amounts of labeled data. Reducing this requirement is an important subject of research.

End-to-end unsupervised learning is common in autoencoders [START_REF] Vincent | Extracting and Composing Robust Features with Denoising Autoencoders[END_REF] and generative models [START_REF] Goodfellow | Generative adversarial nets[END_REF][START_REF] Diederik | Auto-encoding variational Bayes[END_REF]. By defining simple tasks where labels outline can be obtained without human supervision, it is possible to train arbitrary networks as if learning was supervised. Such tasks include for instance watching moving objects in video [START_REF] Pathak | Learning Features by Watching Objects Move[END_REF][START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF] or predicting geometric layout [START_REF] Doersch | Unsupervised Visual Representation Learning By Context Prediction[END_REF], rotation [START_REF] Gidaris | Unsupervised Representation Learning by Predicting Image Rotations[END_REF] or cluster assignment [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF]. This is commonly referred to as self-supervision.

Learning on limited labeled data may be combined with unsupervised objectives on unlabeled data or more supervision on other datasets, domains or tasks. In such combinations, learning may be joint or in stages, essentially decoupling representation learning from the end task. Examples include transfer learning [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF][START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF][START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF], domain adaptation [START_REF] Ganin | Unsupervised Domain Adaptation by Backpropagation[END_REF][START_REF] Tzeng | Deep domain confusion: Maximizing for domain invariance[END_REF], multi-task learning [START_REF] Kokkinos | UberNet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory[END_REF][START_REF] Mallya | Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights[END_REF], incremental learning [START_REF] Li | Learning without forgetting[END_REF][START_REF] Sylvestre-Alvise | iCaRL: Incremental classifier and representation learning[END_REF][START_REF] Yoon | Lifelong Learning with Dynamically Expandable Networks[END_REF], few-shot learning [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF], semi-supervised learning [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF][START_REF] Weston | Deep Learning via Semi-Supervised Embedding[END_REF], active learning [START_REF] Sener | Active Learning for Convolutional Neural Networks: A Core-Set Approach[END_REF][START_REF] Settles | Active learning literature survey[END_REF], learning from noisy labels [START_REF] Li | Learning without forgetting[END_REF][START_REF] Sun | Revisiting Unreasonable Effectiveness of Data in Deep Learning Era[END_REF] and weakly-supervised learning [START_REF] Bilen | Weakly supervised deep detection networks[END_REF][START_REF] Zhou | Weakly Supervised Instance Segmentation Using Class Peak Response[END_REF].

There are tasks where classes at inference are different from classes

Convergence of category-level and instance-level tasks.

at learning, for instance fine-grained classification [START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF], face recognition [START_REF] Schroff | Fa-ceNet: A unified embedding for face recognition and clustering[END_REF], person re-identification [7], local descriptor learning [START_REF] Han | MatchNet: Unifying Feature and Metric Learning for Patch-Based Matching[END_REF] and instance retrieval [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF]. Those are typically tasks of fine-grained similarity and commonly addressed by metric learning, supervised by humans or other algorithms. However, treating them as supervised classification is attracting increasing attention [START_REF] Liu | SphereFace: Deep hypersphere embedding for face recognition[END_REF][START_REF] Zhang | Heated-up softmax embedding[END_REF].

Few-shot learning is a category-level task but it commonly relies on variants of a nearest neighbor classifier and treated as metric learning too [START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Vinyals | Matching networks for one shot learning[END_REF]. Instance-level formulations are also common in selfsupervised representation learning [START_REF] Wu | Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination[END_REF] or classification [START_REF] Meyer | Nearest Neighbour Radial Basis Function Solvers for Deep Neural Networks[END_REF]. Most elements are now common in category-level and instance-level tasks, including architectures, loss functions and representation learning. The main difference remains in the data and labels. Those define precisely the task in terms of factors of variation to which invariances need to be learned, e.g. within-class appearance variation for category-level and viewpoint changes for instance-level tasks.

There are emerging research directions that are part of the natural

Research directions going mainstream.

evolution of the deep learning field, for instance Neural Architecture Search (NAS) [START_REF] Liu | DARTS: Differentiable architecture search[END_REF][START_REF] Zoph | Learning transferable architectures for scalable image recognition[END_REF] and differentiable programming [START_REF] Bunel | Adaptive neural compilation[END_REF][START_REF] Feser | Neural Functional Programming[END_REF]. However, there are also research directions that have existed long before deep learning without attracting much attention and are now mainstream for different reasons. We focus on two examples: few-shot learning [START_REF] Vinyals | Matching networks for one shot learning[END_REF] and adversarial examples [START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. The former, generalizing from few examples, is becoming increasingly important as it is one of the few challenging tasks where the human skill is not understood yet and, if solved, it would allow instant inference in domains with scarce data. The latter, studying the sensitivity of models to their input by crafting imperceptible perturbations that make them fail, is already essential in the quest for robust models for safety-critical applications.

In this context, Part iii presents part of our work carried out in the Our contributions.

period 2018-2019, which addresses learning visual representations by exploring visual data, focusing on solutions assuming only limited or no supervision and progressing from instance-level to category-level tasks. Our contributions consist of:

1. methods for category-level and instance-level tasks using limited supervision, in particular unsupervised deep metric learning [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] and semi-supervised learning [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF], both based on manifold similarity as developed in Part ii;

2. a few-shot learning method [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF] that studies activation maps and learns multiple layers to convergence for the first time; and

3. an adversarial attack [START_REF] Zhang | Smooth Adversarial Examples[END_REF] that generates invisible perturbations and improves the standard success rate vs. distortion performance despite the additional smoothness constraint.

Importantly, using the same machinery that we use for exploring data, we are now able to improve the representation and at the same the improved representation helps explore better. Our attack is also an important step in understanding the role of adversarial examples on the data manifold, which has not been explored much.

background and contributions

metric learning Principal Component Analysis (PCA), introduced by Pearson in 1901 [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF] uses an orthogonal transformation to find a coordinate system for a dataset such that variables are uncorrelated and given by descending order of variance. As such, PCA can be used for dimension reduction, yielding the minimal reconstruction error assuming a linear data model. The classical version of Multidimensional Scaling (MDS), introduced by Young and Householder in 1938 [START_REF] Young | Discussion of a set of points in terms of their mutual distances[END_REF], is identical but expressed in terms of the distance matrix rather than the covariance matrix.

This subtle difference opens the door to nonlinear dimension reduction by replacing the Euclidean by some other metric. An example is Isomap [START_REF] Tenenbaum | Mapping a manifold of perceptual observations[END_REF], which represents the dataset by a k-NN graph and uses a graph distance instead. Kernel PCA [START_REF] Schölkopf | Nonlinear component analysis as a kernel eigenvalue problem[END_REF]

is obtained similarly by

The graph distance is defined in terms of shortest paths found by the Dijkstra algorithm [START_REF] Edsger | A note on two problems in connexion with graphs[END_REF].

expressing PCA in terms of the Gram matrix and then replacing the Euclidean by some other inner product. Hence, it is defined in terms of similarities rather than distances.

Rather than preserving all pairwise distances or similarities in the dataset, topology preservation methods focus on neighbors only. For instance, Laplacian eigenmaps [START_REF] Belkin | Laplacian eigenmaps for dimensionality reduction and data representation[END_REF], formulated as eigenvalue decomposition like all previous methods, minimizes pairwise distances weighted by proximity in the input space. Interestingly, the decomposition obtained by our FSR [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF] discussed in Section 10.4 is the same, except for the eigenvalues being soft-weighted by function h α (9.12) rather than hard-thresholded for dimension reduction.

All the above methods are unsupervised. The nonlinear ones are See Lee and Verleysen [START_REF] John | Nonlinear dimensionality reduction[END_REF] for a survey of unsupervised methods.

collectively referred to as nonlinear dimension reduction, manifold learning or unsupervised metric learning. By contrast, supervised metric learning, studied by Xing et al. in 2003 [490] for supervised classification, outline minimizes (resp. maximizes) distances between positive (resp. negative) pairs of examples, which are given and typically defined as pairs See Yang and Jin [START_REF] Yang | Distance Metric Learning: A Comprehensive Survey[END_REF] for a survey of supervised methods.

having the same (resp. different) class label. Such learning is useful for k-NN classifiers, using for instance Neighborhood Component Analysis (NCA) [START_REF] Goldberger | Neighbourhood components analysis[END_REF] or a triplet loss [START_REF] Kilian Q Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF].

Given a training set in some input space, the goal of metric learning

Embedding.

in general is to learn a mapping from the input space to an embedding space. In all methods discussed so far, this mapping is either only linear or restricted to the training set. In the latter case, out-of-sample

A linear mapping means that the distance metric is Mahalanobis [START_REF] Chandra | On the generalized distance in statistics[END_REF]. extension, i.e., extension to unseen examples, is not possible unless the training set is memorized [START_REF] Bengio | Out-of-sample extensions for LLE, Isomap, MDS, eigenmaps, and spectral clustering[END_REF]. Learning explicit parametric nonlinear mappings such that the training set is not explicitly memorized is where neural networks excel.

Processing one example at a time, Ackley et al. derive in 1985 [2] compact representations using nonlinear autoassociative mapping and a bottlenect layer. This is an explicit, learned nonlinear version of PCA including an encoder and a decoder. In 2006, Hinton and Salakhut-Autoencoders.

dinov [START_REF] Hinton | Reducing the Dimensionality of Data with Neural Networks[END_REF] extend to a deep autoencoder architecture that is pretrained layer-wise and fine-tuned end-to-end, both unsupervised. But how about loss functions requiring at least two examples at a time to specify pairwise distances or similarities?

Baldi and Chauvin introduce in 1993 [START_REF] Baldi | Neural Networks for Fingerprint Recognition[END_REF] an architecture where two training examples are mapped by the same network to respective representations, then a binary classifier predicts the pair as matching or non-matching based on the distance between the two representations. This architecture is given the name Siamese by Bromley et al. in Siamese architecture.

1994 [START_REF] Bromley | Signature Verification Using a "Siamese" Time Delay Neural Network[END_REF]. It is then used for supervised metric learning by Chopra et al. in 2005 [72] and for unsupervised dimension reduction by Hadsell et al. in 2006 [149], who introduce the contrastive loss.

Modern deep metric learning, used in tasks like fine-grained classification [START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF], face recognition [START_REF] Schroff | Fa-ceNet: A unified embedding for face recognition and clustering[END_REF], person re-identification [7], local descriptor learning [START_REF] Han | MatchNet: Unifying Feature and Metric Learning for Patch-Based Matching[END_REF] and image retrieval [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], is mostly supervised by labels provided by humans or other algorithms. Most common loss functions are contrastive [START_REF] Hadsell | Dimensionality Reduction By Learning an Invariant Mapping[END_REF] and a modern version of triplet [START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF]. Because possible pairs and triplets are too many, most of the difficulty shifts from the loss function to hard example mining [START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Chao-Yuan | Sampling matters in deep embedding learning[END_REF], which depends on nearest neighbor search.

Our Mining on Manifolds (MoM) [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] is one of the first unsuper-Our Mining on Manifolds (MoM) [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF].

vised deep metric learning methods that does not assume external algorithms [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] or some structure of the data [START_REF] Doersch | Multi-Task Self-Supervised Visual Learning[END_REF][START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF]. It learns similarity in the embedding space guided by manifold similarity in some initial feature space, as developed in Chapter 10. Compared to supervised methods, it is competitive on fine-grained classification and superior on instance-level retrieval, with labels provided by humans and other algorithms respectively. semi-supervised learning In 1965, Scudder [START_REF] Scudder | Probability of error of some adaptive patternrecognition machines[END_REF] introduces maybe one of the first semi-supervised classification methods, which learns a classifier on some initial set of labeled examples and then iteratively makes predictions on unlabeled examples and updates the classifier accordingly.

In the 1970s, generative models are studied in a semi-supervised See Chapelle et al. [START_REF] Chapelle | Semi-Supervised Learning[END_REF] for a survey of semi-supervised methods before deep learning.

setting. Even before the introduction of EM [START_REF] Arthur P Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], Hosmer studies in 1973 [START_REF] Hosmer | A comparison of iterative maximum likelihood estimates of the parameters of a mixture of two normal distributions under three different types of sample[END_REF] the iterative Maximum Likelihood (ML) estimation of a Gaussian mixture distribution with one component per class, where only a small part of the data is known to originate in a given component. Again, the unlabeled examples participate in the estimation by being soft-assigned to components. Also in the 1970s, Vapnik [START_REF] Vapnik | Theory of pattern recognition[END_REF] advocates transductive inference, making predictions only on the unlabeled test set-whereas in inductive inference, a general decision rule is inferred first. Because the entire unlabeled test set is used, transductive implies semi-supervised, but not conversely. Vapnik also introduces in 1998 [START_REF] Vapnik | Statistical learning theory[END_REF] an early trans- ductive version of the SVM, maximizing the margin over unlabeled as well as labeled examples. In 1999, Joachims [START_REF] Joachims | Transductive inference for text classification using support vector machines[END_REF] introduces what is known today as transductive SVM, allowing class overlap and finding an approximate solution in the form of local search.

In the 2000s, two transductive methods known as Label Propagation

The manifold assumption:

High-dimensional data lies on low-dimensional manifolds.

(LP) model the manifold structure of the data by a k-NN graph. One is by Zhu and Ghahramani in 2002 [START_REF] Zhu | Learning From Labeled and Unlabeled Data with Label Propagation[END_REF], where the solution retains the labels on labeled examples and is harmonic on unlabeled ones [START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF]. The other is by Zhou et al. in 2003 [515], where new labels are allowed on labeled examples. This can be useful e.g. when classes overlap or even when labels are noisy. The latter solution uses the same kind of graph filtering as discussed for manifold search in Chapter 10, but with one query per class.

Deep learning naturally follows the inductive approach, in the sense that not only the learned model is expected to work on unseen examples, but the training set is expected to be discarded. learning approaches. Like metric learning [START_REF] Belkin | Laplacian eigenmaps for dimensionality reduction and data representation[END_REF][START_REF] Hadsell | Dimensionality Reduction By Learning an Invariant Mapping[END_REF], it adds an unsupervised loss to minimize the distance of the embeddings of two examples that are close in the input space. Pseudo-label, introduced by Lee in 2013 [START_REF] Lee | Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks[END_REF], is a modern incarnation of the earliest approaches [START_REF] Scudder | Probability of error of some adaptive patternrecognition machines[END_REF], treating predictions as if they were true labels. Many modern approaches add an unsupervised consistency loss on all data. For instance, temporal ensemble [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF] and mean teacher [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] minimize the distance of two embeddings of the same example obtained by two similar models. This suggests a dual to the smoothness assumption: If two models are close in the parameter space, then so should be the corresponding predictions.

Our Deep Label Propagation (DLP) [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] is an inductive incarnation

Our Deep Label Propagation (DLP) [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF].

of LP in a modern deep learning setting. It uses pseudo-labels on unlabeled data too. However, our pseudo-labels are inferred by LP [START_REF] Zhou | Learning with local and global consistency[END_REF] rather than by the classifier directly [START_REF] Lee | Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks[END_REF]. Because the graph is based on the network embedding, DLP alternates between performing LP outline and updating the embedding. Weighting unlabeled examples by certainty provides robustness to incorrect pseudo-labels. DLP is found to be complementary to consistency losses.

few-shot learning Humans can learn new concepts from just

Motivation from human skills.

one example. This skill is developed quite early and applies to language, vision or associations between language and vision. handwritten digits from one example by learning a density of transformations on a corpus of digits. In 2003, Fei-Fei et al. [START_REF] Fei-Fei | A Bayesian approach to unsupervised one-shot learning of object categories[END_REF] learn a mixture distribution over appearance and shape from one example of a novel generic object category, e.g. face or motorbike, by inducing a conjugate prior from other categories. In 2005, Bart and Ullman [START_REF] Bart | Cross-Generalization: Learning Novel Classes From a Single Example By Feature Replacement[END_REF] learn novel classes from one example by using informative patches of similar, familiar classes and adapting them to the example at hand. In 2011, Lake et al. [START_REF] Lake | One shot learning of simple visual concepts[END_REF] tinue learning. Meta-learning [START_REF] Schmidhuber | Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta[END_REF][START_REF] Vilalta | A perspective view and survey of meta-learning[END_REF] refers to learning at two levels, where generic knowledge is acquired before adapting to more specific tasks. In few-shot learning, this translates to learning on base classes how to learn from few novel-class examples without overfitting. Metric learning is then seen as learning how to compare queries with support examples [START_REF] Vinyals | Matching networks for one shot learning[END_REF] or class prototypes [START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Waltner | Bacon: Building a Classifier From Only N Samples[END_REF]. Optimization meta-learning [START_REF] Finn | Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] amounts to learning a model that is easy to fine-tune in few steps. Augmentation meta-learning [START_REF] Hariharan | Low-shot Visual Recognition by Shrinking and Hallucinating Features[END_REF][START_REF] Wang | Low-Shot Learning from Imaginary Data[END_REF] refers to learning how to generate novel-class examples.

The meta-learning setup is challenged by Gidaris and Komodakis

The challenge applies equally to metric learning [START_REF] Zhang | Heated-up softmax embedding[END_REF] and face recognition in particular [START_REF] Ranjan | L2constrained softmax loss for discriminative face verification[END_REF][START_REF] Wang | NormFace: L2 hypersphere embedding for face verification[END_REF]. [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF], who rather learn on base classes a simple parametric classifier, based on cosine similarity. The same classifier is introduced independently by Qi et al. [START_REF] Qi | Low-Shot Learning With Imprinted Weights[END_REF], who further fine-tune the network, assuming access to the base class training set.

Our Dense Classification (DC) [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF] builds on this simple classifier

Our Dense Classification (DC) [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF].

by applying it densely over all locations in an image. This is a form of implicit data augmentation of dense shifts and crops with a single network evaluation. As a result, we obtain smoother activation maps that are more aligned with objects. This is the first time that activation maps are studied in the context of few-shot learning.

In the same work [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF], we introduce implanting. Neural implants

Our implanting [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF].

are convolutional filters in a new processing stream parallel to a previously trained network, which remains fixed. Few implants learn new, task-specific features with reduced risk of overfitting. Implanting thus enables training of multiple layers to convergence in the few-shot regime, departing from prior methods that train either only the last layer [START_REF] Hariharan | Low-shot Visual Recognition by Shrinking and Hallucinating Features[END_REF] or only for a few iterations [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF]. the SIFT detector either by manipulating pixels in the input image to introduce duplicate local extrema, or by replacing patches by similar patches from an image database, where no features are detected. A similar search over a SIFT descriptor database is used by Weinzaepfel et al. [START_REF] Weinzaepfel | Reconstructing an Image From Its Local Descriptors[END_REF] to reconstruct the input image from its collection of SIFT descriptors. With CNNs being differentiable, Szegedy et al. [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] rather modify the input image by using back-propagation to minimize a loss function. The objective is to modify the classifier prediction while keeping the input distortion low. Adversarial examples are not particular to nonlinearities either: In 2015, Goodfellow et al. [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] attribute their existence in linear models to the high dimensionality of the input space. The reason why their

adversarial

Why study adversarial examples?

study becomes so important can be rather sought in the fact that we now trust CNNs in safety-critical tasks like driving cars. To make matters worse, adversarial examples are transferable from one model to another [START_REF] Liu | Delving into Transferable Adversarial Examples and Black-box Attacks[END_REF] and there exist Universal Adversarial Perturbations (UAPs) that apply to several models [START_REF] Moosavi-Dezfooli | Universal Adversarial Perturbations[END_REF].

Goodfellow et al. [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] on a simple data model that adversarial robustness comes at the additional cost of dropping accuracy on benign (not adversarial) examples. But this data model is just chosen to simplify the theoretical proof. So, is this result true in general? Stutz et al. [START_REF] Stutz | Disentangling Adversarial Robustness and Generalization[END_REF] show empirically that, under the manifold assumption, adversarial examples leave the manifold; on-manifold adversarial examples exist and using the them for adversarial training boosts generalization. Stutz et al. [START_REF] Stutz | Disentangling Adversarial Robustness and Generalization[END_REF] use an autoencoder [START_REF] Boesen | Autoencoding beyond pixels using a learned similarity metric[END_REF] to approximate the underlying manifold of each class. As a result, experiments are constrained to tiny images. To generate more realistic on-manifold adversarial examples, a general property of natural images can be used instead:

Smoothness.
smoothness. Previous attempts in this direction include e.g. spatial filtering [START_REF] Zhou | Transferable Adversarial Perturbations[END_REF], harmonic functions [START_REF] Heng | Harmonic Adversarial Attack Method[END_REF] and attacking in the Fourier domain [START_REF] Guo | Low Frequency Adversarial Perturbation[END_REF]. All of these constructs are independent of the input image and none of these attacks is competitive.

Our smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF] are an attempt to improve the Our smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF].

visual quality of adversarial examples by smoothing the noisy gradient guided by the input image, similarly to style transfer [START_REF] Luan | Deep photo style transfer[END_REF][START_REF] Puy | A Flexible Convolutional Solver for Fast Style Transfers[END_REF]: The perturbation is locally smooth on flat areas of the input image, but it may be noisy on textured areas and sharp across edges. Smoothing follows Chapter 9 and the graph is defined over pixels.

Despite the additional smoothness constraint, which reduces the effective dimensionality of the perturbation, our attack has the same success rate at lower distortion compared to a regular attack. Even when the distortion is higher, the perturbation is totally invisible.

structure

Chapter 14 addresses metric learning for ranking tasks including finegrained classification and image search. It discusses Mining on Manifolds (MoM) [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF], an unsupervised method based on differences between Euclidean and manifold similarity.

Chapter 15 studies Semi-Supervised Learning (SSL), where labels are limited. It presents Deep Label Propagation (DLP) [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF], an inductive version of the classic Label Propagation (LP) [START_REF] Zhou | Learning with local and global consistency[END_REF].

Chapter 16 discusses the more challenging problem of Few-Shot Learning (FSL), where not only labels but raw data is limited too. It presents two solutions, Dense Classification (DC) and implanting [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF], which do not rely on meta-learning.

Finally, Chapter 17 concludes with a study of adversarial examples, that is, imperceptible perturbations of a given input that make a model fail. We argue that popular attacks have a fundamental limitation in terms of imperceptibility and present smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF], an attempt to address this limitation.

M E T R I C L E A R N I N G

We introduce an unsupervised framework for hard example mining [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF]. The only input is a collection of images and an initial CNN representation. Positive examples are distant points on a manifold; negative are nearby points on different manifolds. Both are revealed by disagreements between Euclidean and manifold similarities, and can be used with any discriminative loss. We apply to fine-grained classification and image retrieval.

introduction

It is common to start with a pre-trained network [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF] and apply metric learning [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF][START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF] to fine-tune the network for a task like fine-grained classification [START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Wang | Mining discriminative triplets of patches for fine-grained classification[END_REF], particular object retrieval [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], or person re-identification [7]. Loss functions like contrastive [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF], triplet [START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF] or batch-level [START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF] are typically used on hard examples, found by sampling or mining mechanisms [START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Movshovitz-Attias | No Fuss Distance Metric Learning Using Proxies[END_REF]. Training labels may be found by existing algorithms [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], spatial [START_REF] Doersch | Multi-Task Self-Supervised Visual Learning[END_REF] or temporal [START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF][START_REF] Wang | Transitive invariance for selfsupervised visual representation learning[END_REF] structure of the data, but in most cases by human supervision [START_REF] Babenko | Neural Codes for Image Retrieval[END_REF][START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF]. On one hand, by using class labels, we miss the opportunity of learning from unlabeled data, learned class representations are unimodal [START_REF] Rippel | Metric learning with adaptive density discrimination[END_REF] and the problem remains supervised classification. On the other hand, conventional manifold learning methods are unsupervised [START_REF] Belkin | Laplacian eigenmaps for dimensionality reduction and data representation[END_REF][START_REF] Saul | Think globally, fit locally: unsupervised learning of low dimensional manifolds[END_REF][START_REF] Tenenbaum | A Global Geometric Framework for Nonlinear Dimensionality Reduction[END_REF] but have difficulties general-Most do not learn an explicit mapping from the input to the embedding space.

izing to new data. We attempt to bridge this gap by introducing Mining on Mani-Learning a new representation dispenses the need to store the dataset and compute the manifold similarity, and generalizes better to unseen datasets.

folds (MoM) [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF], an unsupervised hard example mining mechanism using manifold similarity [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] on an initial representation space of unlabeled data. As illustrated in Figure 14.1, given an anchor point, neighbors on the manifold that are not Euclidean neighbors are considered positive examples to be attracted to the anchor. Conversely, Euclidean neighbors that are not manifold neighbors are considered negative and should be repelled. We apply our method to fine-grained classification and particular object retrieval.

preliminaries

problem We are given a training set X = {x 1 , . . . , x n } ⊂ X , where

An example may be an image, video, image region, local patch, etc.

X is an input space. Function f θ : X → R d maps an example x ∈ X to a vector u = f θ (x) in a d-dimensional embedding space U, where θ is a set of parameters to be learned. Examples X are represented by a set of features V = {v 1 , . . . , v n } ⊂ V, where V is a feature space and

v i = φ(x i ) for i ∈ [n]. Function φ may be f θ 0 , i.e. the same model f We write [n] := {1, . . . , n} for n ∈ N.
with parameters θ 0 , supervised or not, or a different model. The goal of metric learning is to learn the parameters θ such that matching examples are mapped to nearby points in the embedding Two images are matching if they are visually similar, where "similarity" depends on the application.

space U, while non-matching examples are well separated. For instance, a matching (non-matching) pair consists of a reference or anchor example x r and a positive (negative) example x + (x -). Alternatively, we may use a triplet (x r , x + , x -). Assuming only the input examples X and their features V , our goal is to mine such labels without any supervision [START_REF] Chopra | Learning a Similarity Metric Discriminatively, with Application to Face Verification[END_REF][START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Movshovitz-Attias | No Fuss Distance Metric Learning Using Proxies[END_REF][START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF], existing algorithms [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] or assumptions on the structure of the training data [START_REF] Doersch | Multi-Task Self-Supervised Visual Learning[END_REF][START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF][START_REF] Wang | Transitive invariance for selfsupervised visual representation learning[END_REF].

manifold similarity By NN k (v) (NN m k (v))
we denote the k Euclidean (manifold) nearest neighbors of vector v in V , i.e., the k most similar examples in V by Euclidean (manifold) similarity function s : V 2 → R (s m : V 2 → R). To define s m , we form the adjacency In fact, s m is a function of the entire set V .

matrix W of the Euclidean k-NN graph of V as in Section 10.2. Then, following Chapter 9, we normalize it into W by (9.3) and compute the n × n regularized Laplacian L α by (9.6), where α ∈ [0, 1). Following Section 10.3 and (10.8), the solution z i of the linear system

As in Chapter 10, we solve this system by CG [START_REF] Nocedal | Numerical optimization[END_REF].

L α z = e i , (14.1)
where e i is the standard n-dimensional basis vector, expresses the That is, ei is the i-th column of an n × n identity matrix.

manifold similarity of v i ∈ V to all vectors in V . We thus define the manifold similarity of v i , v j ∈ V as

s m (v i , v j ) := z i (j), (14.2)
i.e., the j-th element of z i . Observe that s m is symmetric because in fact s m (v i , v j ) is the (i, j)-th element of L -1 α , which is symmetric.

L -1 α is dense but we never compute it; we only compute its columns as needed.

Finally, the manifold nearest neighbors NN m k (v i ) of v i in V are the elements of V corresponding to the k maximum elements of z i . positives Given an anchor or query example x r and the corresponding feature v r = φ(x r ), we define the positive pool N + (x r ) of x r as the examples in X that correspond to the manifold neighbors of v r that are not Euclidean neighbors in the feature space: k controls the diversity of positives, with larger values corresponding to harder examples.

N + (x r ) := {x ∈ X : φ(x) ∈ NN m k (v r ) \ NN k (v r )}. (14.3)
As illustrated in Figure 14.1(c), these examples are assumed to be matching with x r , yet not retrieved well in the feature space. In the embedding space, positives should be attracted to the anchor so that

Mining hard positive examples, in contrast to negatives, is not common.

Euclidean and manifold neighbors agree.

negatives Similarly, we define the negative pool N -(x r ) of anchor x r as the examples in X that correspond to the Euclidean neighbors of v r that are not manifold neighbors in the feature space:

N -(x r ) := {x ∈ X : φ(x) ∈ NN k (v r ) \ NN m k (v r )}. (14.4)
As illustrated in Figure 14. 1(d), these examples are assumed to be non-It is common practice, and known to be beneficial [START_REF] Simo-Serra | Fracking deep convolutional image descriptors[END_REF], to select hard negative examples.

matching with x r , yet too close in the feature space. In the embedding space, negatives should be repelled from the anchor.

anchors We need a diverse collection of anchors that have many neighboring examples in the feature space V . This is achieved by the modes of the k-NN graph G. Following [START_REF] Cho | Mode-Seeking on Graphs via Random Walks[END_REF], we find those as the local maxima of the stationary distribution on G. Details are given in [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF]. We denote by Q the resulting anchor or query set.

training

At a given training epoch, for each anchor x r ∈ Q, a positive example x + is drawn at random from its positive pool N + (x r ), and a negative x -from a subset of its negative pool N -(x r ). This subset consists of the examples corresponding to the Euclidean nearest While Q, N + , N -are computed once, hard negative sampling uses the current network representation.

neighbors of f θ (x r ) in the embedding space, where θ is the current set of parameters. The training set for the epoch is the set of tuples (x r , x + , x -) for x r ∈ Q. Given such a tuple with corresponding embeddings u r := f θ (x r ), u + := f θ (x r ) and u -:= f θ (x -), we use the contrastive loss [START_REF] Hadsell | Dimensionality Reduction By Learning an Invariant Mapping[END_REF],

c (u r , u + , u -) := u r -u + 2 + [m -u r -u -] 2 + , (14.5)
combining a positive and a negative pair, or the triplet loss [START_REF] Wang | Learning Fine-Grained Image Similarity with Deep Ranking[END_REF] We write [x]+ := max(x, 0) for the positive part of x ∈ R.

t (u r , u + , u -) := [m + u r -u + 2 -u r -u -] 2 + , (14.6)
where m is a margin parameter.

x r N + (x r ) N -(x r )

applications fine-grained classification We use the Caltech-UCSD Birds

The goal is to learn embeddings that discriminate instances of the same class from instances of different classes.

(CUB)200-2011 dataset [START_REF] Wah | The Caltech-UCSD Birds-200-2011 dataset[END_REF], comprising 200 bird species as classes. An embedding is learned on 100 classes, while the remaining 100 are used for testing [START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF]: Given a test query, the remaining test images of the same class should be top-ranked according to embedding similarity to the query [START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Wang | Mining discriminative triplets of patches for fine-grained classification[END_REF]. This is evaluated by Recall@k, while Normalized Mutual Information (NMI) [START_REF] Manning | Introduction to Information Retrieval[END_REF] measures clustering quality. We only use labels for testing.

The feature space V may have been learned using supervision on some other training set/domain. particular object retrieval We use a 1M subset of the same Flickr7M collection used by [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] to learn the embedding. Test sets comprise small objects (Instre [START_REF] Wang | INSTRE: a new benchmark for instance-level object retrieval and recognition[END_REF]), natural scenery (Holidays [START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF]) and buildings/landmarks (Oxford5k [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] and Paris6k [START_REF] Philbin | Lost in Quantization: Improving Particular Object Retrieval in Large Scale Image Databases[END_REF]). For large-scale experiments, we add 100k distractor images [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] to Ox-ford5k and Paris6k, referred to as Oxford105k and Paris106k respectively. Given a test query, the remaining test images depicting the Similarity is even more fine-grained than in bird species. Objects are less deformable, but there is extreme diversity in viewpoint, lighting, occlusion and clutter.

same instance should be top-ranked according to embedding similarity [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], as evaluated by mAP. Modern methods use existing algorithms to mine pairs [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] or triplets [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF] on the training set. We only use the provided features.

experiments

setup In both applications, the initial model f θ is a network pretrained on ImageNet and the feature set V consists of R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF] on

All vector representations are

2-normalized.
the last convolutional layer of the same model. We use Euclidean similarity s(v, v ) := [v v ] 3 + and set α = 0.99 as in Section 10.5 for manifold similarity. We set k to 30 in the k-NN graph, 50 for positives (14.3) and 100 (10,000) for negatives (14.4) for classification (retrieval). We use the triplet loss (14.6) with m = 0.5 for classification and contrastive loss (14.5) with m = 0.7 for retrieval. Training hyperparameters are detailed in [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF]. method labels r@1 r@2 r@4 r@8 NMI Baseline [START_REF] Li | Unsupervised visual representation learning by graph-based consistent constraints[END_REF] and baseline as pre-trained on Ima-geNet. "Labels" indicates using ground-truth labels at training.

fine-grained classification We use GoogLeNet [START_REF] Szegedy | Going deeper with convolutions[END_REF] baseline as pre-trained on ImageNet, we reduce to an embedding dimensionality of d = 64 by adding a fully-connected layer and we fine-tune with triplet loss. We use all training images as anchors.

This is affordable since the training set is small (6k images).

As shown in Table 14.1, our approach outperforms the unsupervised approach [START_REF] Li | Unsupervised visual representation learning by graph-based consistent constraints[END_REF] and competes or even outperforms supervised methods that are using ground-truth labels on the training set. Comparing our MoM against SfM-based fine-tuning [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] and baseline as pre-trained on ImageNet. Fine-tuning performed for MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF].

particular object retrieval We use VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] baseline as

We use unsupervised whitening [START_REF] Jégou | Negative Evidences and Cooccurences in Image Retrieval: The Benefit of PCA and Whitening[END_REF] for the baseline and supervised whitening [START_REF] Mikolajczyk | Improving Descriptors for Fast Tree Matching By Optimal Linear Projection[END_REF] for fine-tuned networks, as in [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF].

pre-trained on ImageNet and we fine-tune MAC representation with contrastive loss. At testing, we use both MAC and R-MAC [START_REF] Tolias | Particular object retrieval with integral max-pooling of CNN activations[END_REF]. We compare against [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF], which mines labels via a Structure-from-Motion (SfM) pipeline based on conventional features. Anchor selection is essential here, due to the size of the training set.

As shown in Table 14.2, we improve over the baseline as well as [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] in most cases. We even improve on Holidays and Instre, where [START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] shows little improvement or is even inferior to the baseline.

discussion

In this work, we depart from using ground-truth labels or conventional algorithms in metric learning for fine grained similarity. We experiment with standard contrastive and triplet loss, but there are many other losses that are functions of more than two or three examples [START_REF] Bautista | Deep Unsupervised Similarity Learning using Partially Ordered Sets[END_REF][START_REF] Bg Kumar | Learning local image descriptors with deep siamese and triplet convolutional networks by minimising global loss functions[END_REF][START_REF] Law | Quadruplet-Wise Image Similarity Learning[END_REF][START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF][START_REF] Ustinova | Learning deep embeddings with histogram loss[END_REF][START_REF] Chao-Yuan | Sampling matters in deep embedding learning[END_REF]. It is possible to iterate our approach, updating the graph and the pools based on the current embedding space, then updating the embeddings, and so on.

S E M I -S U P E R V I S E D L E A R N I N G

Label Propagation (LP) [START_REF] Zhou | Learning with local and global consistency[END_REF] is a transductive method for semi-supervised classification. In this work, we extend it to an inductive setting, introducing Deep Label Propagation (DLP) [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF]. We generate pseudo-labels for the unlabeled training examples by LP and train a deep neural network to classify unseen examples. LP uses a nearest neighbor graph of the training set that we create based on the embeddings of the same network. Therefore we alternate between performing LP and updating the network.

introduction

Deep neural networks require large amounts of training examples. Visual data is available in large quantities, but supervision is provided either by humans, which is expensive, or automatically on proxy tasks [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF][START_REF] Doersch | Unsupervised Visual Representation Learning By Context Prediction[END_REF][START_REF] Gidaris | Unsupervised Representation Learning by Predicting Image Rotations[END_REF][START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF][START_REF] Pathak | Learning Features by Watching Objects Move[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF][START_REF] Wang | Transitive invariance for selfsupervised visual representation learning[END_REF][START_REF] Wu | Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination[END_REF], which is inferior or not appropriate for high-level tasks like classification.

Semi-Supervised Learning (SSL) is becoming increasingly important because it can combine data carefully labeled by humans with abundant unlabeled data to train powerful networks. In transductive inference [START_REF] Zhou | Learning with local and global consistency[END_REF][START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF], prediction is restricted to the unlabeled examples of the training set. In modern inductive learning, the goal is generalization to unseen data, while the training data is discarded. This is achieved e.g. by combining a supervised loss on labeled data with unsupervised objectives on all data [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF][START_REF] Weston | Deep Learning via Semi-Supervised Embedding[END_REF]; or, an existing classifier Using a powerful classifier trained on carefully labeled data can provide high-quality pseudo-labels, opening the door to learning from large scale unlabeled data [START_REF] Radosavovic | Data Distillation: Towards Omni-Supervised Learning[END_REF]. can be used to assign pseudo-labels [START_REF] Lee | Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks[END_REF][START_REF] Sajjadi | Regularization with stochastic transformations and perturbations for deep semi-supervised learning[END_REF].

Classic transductive methods have not been fully exploited in the inductive setting. The same holds for the manifold assumption-that similar examples should get the same prediction. In this work, we use Label Propagation (LP) [START_REF] Zhou | Learning with local and global consistency[END_REF], a transductive classification method based on the manifold assumption, to infer pseudo-labels for unlabeled data and use them to train a classifier. We call this inductive extension Deep Label Propagation (DLP) [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF]. DLP alternates between two steps: (a) Train the network on labeled and pseudo-labeled data. (b) Use the k-NN graph of the embeddings of the network to perform LP, inferring pseudo-labels for unlabeled examples, as well as a certainty-based weight per example. Training is performed on all data, weighted by certainty.

preliminaries

problem We are given a training set X := {x 1 , . . . , x n } ⊂ X , where X is an input space. A subset

X L := {x i : i ∈ L} with L ⊂ [n] is labeled We write [n] = {1, . . . , n} for n ∈ N.
by

y L := (y i ) i∈L with y i ∈ [c],
where c is a number of class labels. The remaining examples

X U := X \ X L with U := [n] \ L are unlabeled.
The goal is to use all examples X and labels y L to train a classifier that maps new examples from X to class labels.

classifier The classifier network f θ : X → R c , where θ are the network parameters, maps an input example x i ∈ X to a vector

f θ (x i ) ∈ R c of class probabilities. The corresponding prediction ŷi ∈ [c]
is the class of maximum probability

f θ (xi)j denotes the j-th element of vector f θ (xi). ŷi := arg max j∈[c] f θ (x i ) j . (15.1)
As a by-product of learning, we assume access to an embedding net-For example, f θ may consist of φ θ followed by an FC layer and softmax.

work φ θ : X → R d , mapping x i to a feature vector v i := φ θ (x i ).

J s (X L , y L ; θ) := i∈L (f θ (x i ), y i ) . (15.2)
A standard choice for the loss function is cross-entropy, given by (p, j)

:= -log p j for p ∈ R c + , j ∈ [c].
15.3 background pseudo-labeling By assigning a pseudo-label ŷi to each example x i for i ∈ U and writing ŷU := (ŷ i ) i∈U , the following pseudo-label cost function applies to unlabeled examples X U :

J p (X U , ŷU ; θ) := i∈U (f θ (x i ), ŷi ) . (15.3)
An example is [START_REF] Lee | Pseudo-label: The simple and efficient semisupervised learning method for deep neural networks[END_REF], where θ is learned by (15.2) on X L and then pseudo-labels are assigned by network predictions (15.1) on X U .

15.4 deep label propagation label propagation Following the definitions of Chapter 9, given an n × n adjacency matrix W of the training set, we normalize it into W by (9.3) and compute the n × n regularized Laplacian L α by (9.6), where α ∈ [0, 1). We also define an n × c label matrix Y by

The i-th row of Y is an one-hot encoding of label yi if i ∈ L and zero otherwise.

y ij := 1, if i ∈ L ∧ y i = j 0, otherwise (15.4)
for i, j ∈ [n]. Defining the n × c matrix Z as the solution of linear

We write the (i, j)-th element of matrix A as aij.

system (9.9), LP [START_REF] Zhou | Learning with local and global consistency[END_REF] makes a prediction for x i ∈ X according to ŷi := arg max

j∈[c] z ij . (15.5)
As illustrated in the toy example of Figure 15.1, LP is a transductive method: Its predictions are constrained to X.

To extend to unseen examples in X , one needs to store the training set X.

Similarly to (9.17), Z minimizes a quadratic cost where a smoothness term encourages nearby examples get the same predictions, while a fitness term encourages predictions to respect labels Y [START_REF] Zhou | Learning with local and global consistency[END_REF].

deep label propagation

In an inductive framework, the smoothness term becomes an unsupervised loss encouraging consistency between nearby example predictions. Indeed, such solution is adopted by [START_REF] Weston | Deep Learning via Semi-Supervised Embedding[END_REF]. This is not very effi-Solving (15.6) for Z is efficient because it does not need feeding data to the network. Then, the learning process drives all examples directly to Z .

cient because the gradient propagates from each example to its neighbors only at each iteration. Our idea is thus the following: Instead of just encouraging nearby examples to get the same predictions, we encourage all examples to get predictions directly as obtained by label propagation, as if they were all labeled. pseudo-labels Given network parameters θ, let V := {v 1 , . . . , v n } be a feature set, where v i := φ θ (x i ) for i ∈ [n]. We form the adjacency matrix W of the k-NN graph of V as discussed in Section 10.2. Defining matrices L α (9.6) and Y (15.4) as discussed in Section 15.3, we

The same solution has been used for interactive image segmentation [START_REF] Tae Hoon | Generative Image Segmentation Using Random Walks with Restart[END_REF], semantic image segmentation [START_REF] Chandra | Fast, Exact and Multi-Scale Inference for Semantic Image Segmentation with Deep Gaussian CRFs[END_REF] and image retrieval [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF].

perform LP by solving the linear system (9.9)

L α Z = Y (15.6)
using the CG method, which is faster than the iterative process of [START_REF] Zhou | Learning with local and global consistency[END_REF].

Given the solution Z , we infer pseudo-labels ŷU := (ŷ i ) i∈U , where ŷi is given by (15.5).

certainty and balancing Pseudo-labels are not equally certain. Apart from prediction ŷi (15.5), LP can give a vector of class probabilities η 1 (z i ), where z i is the i-th row of Z and η 1 denotes

We write η1(z) := z/ z 1 for z ∈ R c .

1 -normalization. We can then measure the certainty of ŷi and assign example x i a weight ω i ∈ [0, 1] for i ∈ [n], defined by

ω i := 1 - H(η 1 (z i )) log(c) , (15.7)
where H(p) := -j∈[c] p j log p j is the entropy of distribution p ∈ R c + . As illustrated in Figure 15.1(b), the manifold containing "red" labels is still predicted as "red", although several examples are far away from the labels, thus having little certainty.

In addition, to encourage balance of pseudo-labels over classes, we assign class j a weight β j := (|L j | + |U j |) -1 , where L j (U j ) is the number of examples labeled (pseudo-labeled) as class j for j ∈ [c].

weighted cost Given the above definitions of example and class weights, the following weighted cost function applies to both labeled and pseudo-labeled examples

In contrast to (15.3), pseudo-labels originate in label propagation rather than network predictions.

J w (X, y L , ŷU ; θ) := i∈L

β y i (f θ (x i ), y i )+ i∈U ω i β ŷi (f θ (x i ), ŷi ) . (15.8)
This is the sum of weighted versions of J s (15.2) and J p (15.3), where ŷi is now defined by (15.5).

embedding φ θ FC + softmax classifier f θ train with Js(X L , y L ; θ) (7.2)
for T epochs train with Jw(X, y L , ŷU ; θ) (7.8) for 1 epoch iterative training Starting with randomly initialized parameters θ, we train the network f θ for T epochs on the labeled examples X L and labels y L using the supervised cost (15.2). Using the learned embedding φ θ , we then iterate the following process T times:

features V := φ θ (X) adjacency W (5.
1. Extract features V := φ θ (X) on the entire training set X and form the Laplacian L α (9.6) of the k-NN graph of V .

2. Perform LP by solving linear system (15.6) and infer pseudolabels ŷU by (15.5) and weights by (15.7).

3.

Train f θ for one epoch on the training set X, labels y L and pseudo-labels ŷU using the weighted cost J w (15.8).

This iterative learning process is illustrated in Figure 15.2.

experiments

setup We evaluate our DLP against or combined with Mean Teacher (MT) [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF], which applies (15.2) to the labeled examples and an unsupervised consistency loss between two models to the entire training set. The combination uses the sum of the two cost functions. We

As detailed in [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF], we introduce a SSL setup for miniImageNet, which has been previously used for FSL [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF][START_REF] Ravi | Optimization as a model for few-shot learning[END_REF].

use Canadian Institute for Advanced Research (CIFAR)-10 [233], CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and miniImageNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF], containing 10, 100 and 100 classes respectively; we draw 10, 3 and 3 splits respectively of the training set into labeled and unlabeled subsets. All training (test) sets contain 50k (10k) images. The labeled subsets are uniform over classes and vary between 1-20% of the training set.

We use the same 13-layer convolutional network that is used in prior work [START_REF] Laine | Temporal ensembling for semisupervised learning[END_REF][START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF]-referred to as C13-on CIFAR-10 and CIFAR-100, and Resnet-18 [START_REF] He | Deep residual learning for image recognition[END_REF] on miniImageNet. Pre-processing, data augmen-In both networks, f θ consists of φ θ followed by 2-normalization, an FC layer and softmax.

tation and training hyperparameters are detailed in [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF]. We set k = 50 for the k-NN graph and α = 0.99 in (9.6). We train for 180 epochs in total. We evaluate on the test set by mean classification error and standard deviation over the splits. As shown in Table 15.1, either our DLP or the combination with MT [START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] performs the best out on all datasets, demonstrating that the two approaches are complimentary. The gain over MT alone is more pronounced when the number of labels is small. DLP alone performs best on miniImageNet with 4k labels.

discussion

Most recent approaches for deep SSL rely on training with unsupervised objectives on both labeled and unlabeled examples. Our approach relies on graph-based label propagation to infer pseudo-labels for the unlabeled examples. These pseudo-labels are shown to be more accurate than the ones inferred by the network itself. This idea is in principle complementary to unsupervised objectives, which is experimentally confirmed.

F E W -S H O T L E A R N I N G

In Few-Shot Learning (FSL), we target knowledge transfer from a training set with abundant data to sets with few available examples. In this work [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF], we introduce two simple solutions: (i) Dense Classification (DC) over feature maps, which for the first time studies local activations in the domain of FSL, and (ii) implanting, that is, attaching new neurons to a previously trained network to learn new, task-specific features. Implanting enables training of multiple layers in the few-shot regime, departing from most related methods that train only the final layer.

introduction

Even if it is possible to learn with limited or no supervision, there are tasks and classes with even limited raw data, i.e. from the longtail [START_REF] Wang | Learning to model the tail[END_REF]. Deep neural networks pose several challenges in the lowdata regime, in particular in terms of overfitting and generalization. The subject of FSL is to learn to recognize previously unseen classes only with very few annotated examples. This is not a new problem [START_REF] Fei-Fei | One-shot learning of object categories[END_REF], yet there is a recent interest in meta-learning [START_REF] Bertinetto | Learning feed-forward one-shot learners[END_REF][START_REF] Finn | Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks[END_REF][START_REF] Koch | Siamese neural networks for one-shot image recognition[END_REF][START_REF] Santoro | Meta-learning with memoryaugmented neural networks[END_REF][START_REF] Vinyals | Matching networks for one shot learning[END_REF]: Even when there is single large This is inspired by early work on learning to learn [174,[START_REF] Thrun | Lifelong learning algorithms[END_REF].

training set with a fixed set of classes, it is treated as a collection of sets of different classes, where each class has a few annotated examples. Here we argue instead that a simple pipeline using all available classes and data with a parametric classifier is equally effective.

Most few-shot learning approaches do not deal explicitly with spatial information since feature maps are usually flattened or pooled before the classification layer. Instead, we show that performing Dense Classification (DC) over feature maps during representation learning consistently improves performance on novel tasks.

While incremental learning touches similar aspects with FSL by learning to adapt a network to new tasks [START_REF] Mallya | Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights[END_REF][START_REF] Mallya | Packnet: Adding multiple tasks to a single network by iterative pruning[END_REF] or extending a network with new layers for each new task [START_REF] Rusu | Progressive neural networks[END_REF], few of these ideas have been adopted in FSL. We introduce neural implants, which are layers attached to an already trained network, enabling it to quickly adapt to new tasks with few examples.

preliminaries

problem We are given a training set X := {x 1 , . . . , x n } ⊂ X where X is an input space, and corresponding labels y := (y 1 , . . . , y n ) with y i ∈ [c], where c is a set of base classes. On this training set we are

We write

[n] = {1, . . . , n} for n ∈ N. allowed to learn a representation of the domain X such that we can solve new tasks. This representation learning we call stage 1.

In a new Few-Shot Learning (FSL) task, we are given a set of few support examples X := {x 1 , . . . , x n } ⊂ X and corresponding labels y := (y 1 , . . . , y n ) with y i ∈ [c ], where c is a number of novel classes classifier The classifier network f θ : X → R c (resp. R c ), where θ are the network parameters, maps an input example x ∈ X to a vector f θ (x) ∈ R c (resp. R c ) of base (resp. novel) class probabilities. The prediction for input x ∈ X is the class of maximum probability f θ (x)j denotes the j-th element of vector f θ (x).

arg max j∈[c] f θ (x) j (resp. [c ]). As a by-product, we assume an embedding network φ θ : X → R r×d . Since we study the spatial properties of the input, the embedding is a tensor, where r is the spatial dimen-For 2d images, the embedding is a 3d tensor in R w×h×d , where r = w × h is the spatial resolution.

sions and d the feature dimension. It can still be a vector if r = 1.

background

prototypes In Prototypical Networks (PN) [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], one finds a single prototype per novel class and classifies a query to the nearest prototype. Given X , y and an index set S ⊂

s(v, v ) := v, v /( v v ) for v, v ∈ R r×d ; •, • and
• are Frobenius inner product and norm respectively.

In stage 2, the full index set S := [n ] is used and computing class prototypes (16.1) is the only learning to be done.

In stage 1, a number of fictitious tasks called episodes are generated by randomly sampling a number classes from [c] and then a number of examples per class from X with their labels from y. These collections are denoted as X , y respectively, of length n . The set [n ] is partitioned into a support set S ⊂ [n ] and a query set Q := [n ] \ S. The classifier is then trained by minimizing over θ the cost function

J p (X , y ; θ) := i∈Q (f θ,C (x i ), y i ) (16.3)
on the query set Q, where is the cross-entropy loss function given by (p, j) :=log p j for p ∈ R m + , j ∈ [m] and m ∈ N.

dense classification

There are two common ways of handling an embedding tensor φ θ (x) ∈

Alternatively, the weight parameter is a tensor of the same shape as the embedding.

R r×d . The first is an FC layer [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF][START_REF] Snell | Prototypical networks for few-shot learning[END_REF][START_REF] Vinyals | Matching networks for one shot learning[END_REF], which can be seen as flattening the tensor into a vector of length r × d, as in Figure 16.1(a). This is discriminative, but not invariant. The second is Global Average Pooling (GAP) [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF][START_REF] Mishra | A simple neural attentive meta-learner[END_REF][START_REF] Boris N Oreshkin | TA-DAM: Task dependent adaptive metric for improved few-shot learning[END_REF], reducing the tensor to a vector of length d, as in Figure 16.1(b). This is invariant, but less discriminative.

architecture Dense Classification (DC), a new approach, is shown

For 2d images, Ω = [w] × [h].
in Figure 16.2. We view the embedding φ θ (x) as a collection of vectors [φ θ (x)(r)] r∈Ω , where Ω is the spatial domain and φ θ (x)(r) ∈ R d

Given A ∈ R r×d , A(r) ∈ R d collects elements of all channels of A at position r ∈ Ω.

represents a single spatial position r ∈ Ω.

In stage 1, we adopt a cosine classifier [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF][START_REF] Qi | Low-Shot Learning With Imprinted Weights[END_REF], where weight parameters are vectors in R d , shared over all spatial positions, and the classifier f θ,W : X → R r×c maps an input to a tensor in R r×c . If Mapping (CAM) [START_REF] Zhou | Learning Deep Features for Discriminative Localization[END_REF] in that softmax suppresses all but the strongest responses at each position. training The cost function (16.5) becomes

J d (X, y; θ, W ) := n i=1 r∈Ω (f θ,W (x i )(r), y i ). (16.7)
As shown in Figure 16.3, by encouraging the classifier to make correct DC differs from semantic segmentation [START_REF] Long | Fully Convolutional Networks for Semantic Segmentation[END_REF][START_REF] Noh | Learning deconvolution network for semantic segmentation[END_REF] in that we use an image-level label.

predictions everywhere, DC results in smoother activation maps that are more aligned with objects. DC behaves like implicit data augmentation of dense shifts and crops with a single network evaluation.

implanting

From stage 1, we only keep the embedding network φ θ . By implanting, we find new discriminative features for a new task in stage 2. we widen it by adding new convolution kernels, called implants, in a number of its top convolutional layers. As illustrated in Figure 16.4, we are creating a new stream of data in parallel to the base stream, learning additional connections for the new tasks. Let A l (A l ) be the output activation of base (implant) layer l. Then the input of an implant at layer l + 1 is the depth-wise concatenation [A l , A l ] if A l exists, and A l otherwise. The set of all new parameters is θ := (θ l 0 , . . . , θ L ), where θ l are the parameters of the l-th implant, l 0 is the first layer with an implant and L the network depth. The widened embedding network is φ θ,θ .

training We use fictitious subtasks as in PN [START_REF] Snell | Prototypical networks for few-shot learning[END_REF], however we This process is deterministic. Because only one support example is missing, the true task is approximated well.

are now working on novel classes in stage 2. In each subtask, we use each true support example alone as query and the rest as support: For each i ∈ [n ], we define a query set Q i := {i} and a support set S i := [n ] \ Q i . We compute prototypes C i on index set S i according to (16.1), replacing θ by (θ, θ ). The widened classifier f θ,θ ,C i is similarly given by (16.2). Similarly to (16.3), we freeze the base parameters θ and learn the implants by minimizing over θ the cost function Prototypes are recomputed at each iteration based on the current version of implants.

J(X , y ; θ, θ ) := n i=1 (f θ,θ ,C i (x i ), y i ).

(16.8)

inference

We perform GAP to the embeddings of the support examples and com-Inference is the same whether the embedding network has been implanted or not.

pute class prototypes by (16.1). Given a query x ∈ X , we can perform GAP to its embedding φ θ,θ (x) too and classify it to the nearest prototype. Alternatively, we can densely classify the embedding φ θ,θ (x), by soft-assigning the embedding φ θ,θ (x)(r) of each spatial position r independently, averaging over all positions r ∈ Ω according to

f θ,θ ,C (x) := 1 r r∈Ω σ τ [s(φ θ,θ (x)(r), c j )] c j=1 (16.9)
and classifying to arg max j∈[c ] f θ,θ ,C (x) j .

experiments

setup We use ResNet-12 [START_REF] Boris N Oreshkin | TA-DAM: Task dependent adaptive metric for improved few-shot learning[END_REF] as an embedding network. We evaluate on the split [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] of miniImageNet [START_REF] Vinyals | Matching networks for one shot learning[END_REF] results Table 16.1 shows that using DC rather than GAP in stage 1 improves novel tasks significantly. In stage 2, we add implants of

The most gain appears in 1-shot classification.

16 channels to all convolutional layers of the last residual block of the network, which helps further. This gain does not come from increased feature dimensionality, because just widening in stage 1 actually hurts performance. As shown in the bottom part, we outperform other methods by a large margin. Our baseline GAP is already competitive with tadam [START_REF] Boris N Oreshkin | TA-DAM: Task dependent adaptive metric for improved few-shot learning[END_REF], the previous state-of-the-art using the same

Our best results are at least 3% better in all settings.

network, while using a simpler cosine classifier in stage 1.

discussion

In this work we build upon a simplified process for learning on the base classes using a standard parametric classifier. We investigate activation maps for the first time in FSL and devise a new way of handling spatial information, improving the spatial distribution of the activation. We further adapt the network for new tasks by implanting neurons with limited new parameters. For the first time in FSL, we train several convolutional layers to convergence.

A D V E R S A R I A L E X A M P L E S

In this work, we investigate the visual quality of the adversarial examples. We introduce smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF], where the perturbation is locally smooth on flat areas of the input image, while it may still be noisy on textured areas and sharp across edges. This operation relies on Laplacian smoothing, which we integrate in the attack pipeline. Despite the additional constraint of smoothness, our attack has the same probability of success at lower distortion under a white-box scenario.

introduction

Adversarial examples result from applying an imperceptible perturbation to an image that can change a neural network's prediction [START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. Despite the progress in understanding the sensitivity of neural net-

This work has spawned research on adversarial attacks and defenses including team competitions [START_REF] Kurakin | Adversarial Attacks and Defences Competition[END_REF].

works to their input, assessing imperceptibility remains elusive: User studies show that p-norms are largely unsuitable [START_REF] Sharif | On the Suitability of L p -Norms for Creating and Preventing Adversarial Examples[END_REF]. But human assessment of whether an image is adversarial is hard when the p-norm of the perturbation is small. We thus ask the following question:

Given a single image, can the effect of a perturbation be magnified to the extent that it becomes visible and a human may decide whether this example is benign or adversarial?

Figure 17.1 shows that the answer is affirmative for a range of popular attacks. Assuming that natural images are piecewise smooth, we devise a simple adversarial magnification mechanism [START_REF] Zhang | Smooth Adversarial Examples[END_REF]. Without knowledge of the original image, it can reveal not only the presence One can recognize the pattern of Fig. 4 of [START_REF] Moosavi-Dezfooli | Universal Adversarial Perturbations[END_REF] in our Figure 17.1(g), revealing a Universal Adversarial Perturbation (UAP).

of an adversarial perturbation but also identify the attack.

Motivated by this example, we argue that popular adversarial attacks have a fundamental limitation in terms of imperceptibility that we address by smooth adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF]. More than just being "natural" [START_REF] Zhao | Generating Natural Adversarial Examples[END_REF] or smooth [START_REF] Guo | Low Frequency Adversarial Perturbation[END_REF][START_REF] Heng | Harmonic Adversarial Attack Method[END_REF], they are consistent with the smoothness pattern of the input image. They are photorealistic, lowdistortion, and invisible even under magnification. [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] (c) DeepFool [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF] (d) FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] (e) magnified (f) sCW (ours) (g) UAP [START_REF] Moosavi-Dezfooli | Universal Adversarial Perturbations[END_REF] (h) I-FGSM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] Figure problem Let x ∈ X be an image with true label t. An adversarial example a ∈ X is an image such that the distortion xa and the probability f (a) t are small. This problem takes two forms:

1. Target distortion, minimal probability:

We use the Frobenious norm to measure distortion (2- The performance is measured by the expected distortion E( ax ).

We focus on this form.

background: attacks

target distortion Given a distortion target , the Fast Gradient

We assume d = 1 in this section: Color channels are treated independently.

Sign Method (FGSM) [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] performs a single step in the direction of the (element-wise) sign of the loss gradient with ∞-norm , a := proj X (x + sign ∇ x (f (x), t)) , (17.6) where is the cross-entropy loss (p, k)

:= -log p k for p ∈ R c , k ∈ [c].
Iterative FGSM (I-FGSM) [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] initializes a 0 := x and then iterates proj X (a) := arg min a ∈X aa , where X is closed and convex.

[ October target success Szegedy et al. [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] propose a Lagrangian formulation of (17.4)- (17.5), minimizing the cost function

J(a, c) := x -a 2 + λ (f (a), t), (17.8)
where λ is a Lagrange multiplier for (17.5). The attack of Carlini & Wagner (CW) [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] pertains to this approach. It uses the loss function log p j for j = k by at least margin m ≥ 0. A change of variable eliminates the box constraint, replacing a ∈ X by σ(w), where w ∈ R n×d and σ is the element-wise sigmoid function. The CW attack then uses the Adam optimizer [START_REF] Kingma | Adam: A Method for Stochastic Optimization[END_REF] to minimize the cost function σ(x) := 1/(1 + e -x ) for x ∈ R.

J(w, λ) := σ(w)x 2 + λ m (f (σ(w)), t).

(17.10) over w ∈ R n×d . This is repeated for different λ by line search.

When the margin is reached, m vanishes and the distortion term pulls σ(w) back to x.

guided smoothing

To make our perturbation invisible even under magnification, we smooth it guided by the input image. Guiding is similar to guided filtering [START_REF] He | Guided Image Filtering[END_REF][START_REF] Petschnigg | Digital photography with flash and no-flash image pairs[END_REF], but we use graph filtering [START_REF] Tae Hoon | Generative Image Segmentation Using Random Walks with Restart[END_REF][START_REF] Zhou | Ranking on Data Manifolds[END_REF] for the filtering operation itself, as discussed in Chapter 9.

graph We build a weighted undirected graph having the n pixels of the input image x as vertices. The i-th vertex is associated with feature x i ∈ [0, 1] d , the i-th row of x, and position r i ∈ Ω, where Ω := w, h are the width and height of x.

[w] × [h] is the spatial domain. Edge (i, j) has weight

w ij := κ f (x i , x j )κ s (r i , r j ), if i = j 0, if i = j (17.11) for i, j ∈ [n]
, where κ f is a feature kernel and κ s is a spatial kernel. The kernels are truncated such that the adjacency matrix W is sparse.

filtering Following Chapter 9, we normalize W into W by (9.3) and compute the regularized Laplacian L α by (9.6), where α ∈ [0, 1). Then, given a signal y ∈ R n×d , the smoothed version is L -1 α y (9.9). To preserve the dynamic range of y, we row-wise normalize by

Here, 1 ∈ R n is an all-ones vector.

s α (y) := diag(L -1 α 1)) -1 L -1 α (y).

(17.12)

The smoothing function s α of course depends on x. We say s α is smoothα controls the bandwidth: sα is all-pass for α = 0 and low-pass as α → 1 (cf. Figure 9.1).

ing guided by x and the output is smooth like x.

In interactive segmentation [START_REF] Grady | Random Walks for Image Segmentation[END_REF][START_REF] Tae Hoon | Generative Image Segmentation Using Random Walks with Restart[END_REF][START_REF] Vernaza | Learning Random-Walk Label Propagation for Weakly-Supervised Semantic Segmentation[END_REF] (resp. semi-supervised classification [START_REF] Zhou | Ranking on Data Manifolds[END_REF][START_REF] Zhu | Semi-Supervised Learning Using Gaussian Fields and Harmonic Functions[END_REF]), y represents labels at few pixels (resp. examples) and is zero elsewhere. Here it is an arbitrary real-valued signal.

energy minimization Similarly to (9.17), s α (y) is proportional to the minimizer, over z ∈ R n×d , of the quadratic function

• F is the Frobenius norm. Q α (z, y) := α 2 i,j w ij ẑi -ẑj 2 + (1 -α) z -y 2 F , (17.13)
where ẑ := D -1/2 z and D is given by (9.2) [START_REF] Zhou | Ranking on Data Manifolds[END_REF]. The first pairwise ẑi is encouraged to stay close to ẑj when wij is large.

smoothness term encourages z to be smooth wherever x is. The second unary fitness term encourages z to stay close to y.

smooth adversarial examples

Our key idea is that the perturbation z := a-x is smooth like the input image x. This is achieved by a smoothing operation guided by x, which we integrate into CW (17.10). Instead of representing z implicitly as a function σ(w)-x of another parameter w, we minimize over z ∈ R n×d This is because we need to process the perturbation z independently of x. directly, while using the element-wise clipping function proj X (a) = min([a] + , 1) to satisfy the box constraint a ∈ X (17.4): troduce a pairwise loss term µ i,j w ij ẑiẑj 2 into (17.14). A problem is that the spatial kernel is typically narrow, so it would take a lot of iterations to achieve smoothness globally, each requiring a forward and backward pass through the classifier network.

min z λ z 2 + m (f (proj X (x + z)), t). ( 17
smoothness constraint Instead, we introduce the loss term µQ α (z, y) (17.13) where y ∈ R n×d is unconstrained, while z should be close to y and smooth like x (17.13), as well as small (17.14). Then, by letting µ → ∞, this term becomes a hard constraint z = s α (y) (17.12), imposing a globally smooth solution at each iteration: min y λ s α (y) 2 + m (f (proj X (x + s α (y))), t).

(17.15)

During optimization, every iterate of perturbation z is indeed smooth like x. We call this the smooth Carlini & Wagner (sCW) attack.

optimization Problem (17.15) has the same form as (17.14), where z has been replaced by s α (y). This implies that we can use the same optimization method as the CW attack, where we initialize by y = 0 n×d and we apply function s α at each iteration. We use CG [START_REF] Nocedal | Numerical optimization[END_REF] to solve the linear system L α z = y (9.7) for z. Matrix L α is fixed during optimization, depending only on the input image x. Gradients are easy to compute because smoothing is linear:

In the backward pass, we auto-differentiate through the forward CG iterations.

In the backward pass, the gradient of objective (17.15) w.r.t. y is obtained from the gradient w.r.t. z (or a) by smoothing, much like how z is obtained from y in the forward pass, z = s α (y) [START_REF] Zhang | Smooth Adversarial Examples[END_REF].

As shown in Figure 17.2, sCW produces smooth perturbations that are totally invisible in natural images. The reason is the 'phantom' of the original, revealed when the perturbation is isolated. known to the attacker. We compare target distrortion ∞-norm attacks FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] and I-FGSM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] as well as target success 2-norm attacks CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] and our smooth version sCW. We use Inception-v3 [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF] and ResNet-v2-50 [START_REF] He | Identity mappings in deep residual networks[END_REF] networks on dataset [START_REF] Kurakin | Adversarial attacks and defences competition[END_REF], comprising 1,000 Im-ageNet [START_REF] Russakovsky | ImageNet large scale visual recognition challenge[END_REF] images. We use a Laplacian feature kernel and a 3 × 3 neighborhood spatial kernel (17.11); we set α = 0.997 and use 50 iterations of CG (17.12). Parameters of the attacks are detailed in [START_REF] Zhang | Smooth Adversarial Examples[END_REF]. nal CW in terms of distortion, while keeping the probability of success roughly the same. This is surprising: We would expect a price to be paid for a better invisibility as the smoothing is adding an extra constraint on the perturbation.

evaluation
CW internally optimizes its parameter λ independently per image, while for FGSM and I-FGSM we externally set a small set of target distortions on the dataset. Our new evaluation protocol is fair, given that As a result, the operating characteristic is piecewise constant in Figure 17.3.

CW is more expensive. As an interesting finding, distortion-targeting attacks are more competitive than previously thought.

discussion

Smoothing helps mask the adversarial perturbation, when it is 'like' the input image. It allows the attacker to delude the classifier thanks to larger distortions, while still being invisible. It is impressive how sCW improves upon CW in terms of distortion and imperceptibility at the same time, given that it is more constrained. The question raised in the introduction is still open: Figure 17.1(f) shows that a human cannot make the difference between the input image and its adversarial example even under magnification. This does not mean that an algorithm will not detect some statistical evidence.

Part IV B E Y O N D

We summarize more of our past and present contributions, reflect on our contributions in the present context and consolidate the ideas exposed in this manuscript. We then put forth a vision for future research and attempt to draw a road map of ideas that are likely to come.

R E F L E C T I O N

In this chapter we first discuss briefly more of our relevant contributions spanning a period of 20 years as well as more recent and ongoing work that did not make it into the main part of this manuscript. We then revisit our contributions, discuss them in the present context, interpret the properties of different representations, make connections between different ideas, highlight limitations and hint to future work. Finally, we consolidate the ideas exposed in this manuscript by summarizing our contributions and structuring them into a road map, opening the way to new directions.

what else?

Only a few articles are selected to be exposed in the three technical parts of this manuscript. There are several others that fit perfectly the main storyline. We include here a synopsis of a wider selection of articles organized by subject and time, ranging over a period of 20 years. In the interest of space, there is very limited to no context or background in this case, even though there are problems we have not discussed so far. gional descriptors into a global image representation. We combine image partitions obtained by criteria like color and motion statistics, giving rise to a multidimensional histogram [14]. Adding a depth map partition in the case of stereo sequences, we obtain a highly accurate and temporally consistent object support [START_REF] Doulamis | Efficient Summarization of Stereoscopic Video Sequences[END_REF]. We first apply this global representation to automated video abstraction by analyzing the temporal evolution of video sequences in the representation space and detecting extremal points [14]. By introducing a correlation measure on sets of video frames, we then attack the same problem by means of combinatorial optimization [13].

We then apply the same representation to video retrieval, introducing a relevance feedback mechanism in 1999 [START_REF] Doulamis | Interactive Content-Based Retrieval in Video Databases Using Fuzzy Classification and Relevance Feedback[END_REF]. In retrospect, this is an early supervised metric learning [START_REF] Xing | Distance Metric Learning with Application to Clustering with Side-Information[END_REF] approach, where positive and negative pairs are obtained by user feedback. The mapping is linear, giving rise to a closed-form solution. It is a transductive method, since the metric is learned on the test set. Relevance feedback is a standard motivating example of transductive inference [START_REF] Joachims | Transductive inference for text classification using support vector machines[END_REF]. al. [START_REF] Itti | A Model of Saliency-Based Visual Attention for Rapid Scene Analysis[END_REF] to video, we develop spatiotemporal saliency models 1 that employ competition across different feature channels, spatiotemporal lo-cations and scales. We apply these models to a wide range of problems, including visual attention modeling [START_REF] Rapantzikos | Vision, Attention Control, and Goals Creation System[END_REF], salient event detection [START_REF] Rapantzikos | Spatiotemporal features for action recognition and salient event detection[END_REF], movie summarization [START_REF] Evangelopoulos | Multimodal Saliency and Fusion for Movie Summarization based on Aural, Visual, and Textual Attention[END_REF] and video classification [START_REF] Rapantzikos | Spatiotemporal Saliency for Video Classification[END_REF].

By selecting points at local extrema of the saliency map, we extend to spatiotemporal feature detection 2 , balancing well between sparsity and discriminative power. We apply to human action recognition [START_REF] Rapantzikos | Dense saliencybased spatiotemporal feature points for action recognition[END_REF], with spatiotemporal descriptors, a BoW model and a nearest neighbor classifier. We outperform by a large margin previous sparser Space-Time Interest Point (STIP) detectors [START_REF] Dollar | Behavior Recognition via Sparse Spatio-Temporal Features[END_REF][START_REF] Schuldt | Recognizing Human Actions: a Local SVM Approach[END_REF][START_REF] Wong | Extracting Spatiotemporal Interest Points Using Global Information[END_REF]. object proposals Alexe et al. first class-agnostic method to find candidate regions for object detection in 2010. Such a region is called object proposal, region proposal or Region of Interest (RoI). Initially, region proposals are found by bottomup cues like hierarchical segmentation [START_REF] Koen | Segmentation As Selective Search for Object Recognition[END_REF] or edges [START_REF] Zitnick | Edge Boxes: Locating Object Proposals From Edges[END_REF]. Their use becomes widespread also in CNN-based detectors [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] and they are subsequently learned, e.g. by a RPN [START_REF] Shaoqing Ren | Faster R-CNN: Towards real-time object detection with region proposal networks[END_REF].

What is less known is that we investigate this idea in 2008 [217] using hierarchical grouping of interest points, in particular Harris corners [START_REF] Harris | A Combined Corner and Edge Detector[END_REF]. Interest points are considered as a baseline by Alexe et al. [5], but without grouping. sidered too unstable for local feature detection. Yet, using computational geometry constructions, repeatable local features arise. Starting with unstable, single-scale edges, we develop two such solutions using Delaunay triangulation on local maxima of the Euclidean distance transform 3 [START_REF] Rapantzikos | Detecting Regions from Single Scale Edges[END_REF] and weighted α-shapes 4 [START_REF] Varytimidis | WαSH: Weighted α-Shapes for Local Feature Detection[END_REF][START_REF] Varytimidis | αshapes for local feature detection[END_REF].

Alternatively, starting with single-scale image gradient, we compute the exact weighted distance transform and weighted medial axis and partition the image similarly to watershed segmentation. The resulting Medial Feature Detector (MFD) 5 [16] provides subpixel-accurate features capturing the Gestalt principle of closure.

Our detectors yield regions of arbitrary shape and scale that provide good coverage of the image at a fraction of the total number of features compared to other detectors. Most importantly, while most related work is limited to matching experiments, we additionally provide experimental evaluation on the end task of instance-level search, outperforming existing detectors with only modest requirements in terms of inverted index size and query time. vel search, geometry is traditionally only considered in a sequential process of spatial verification that is applied to a short list of topranking images [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] according to similarities by global repre-sentations like BoW. Representations incorporating geometry are either not invariant [START_REF] Lazebnik | Beyond bags of features: spatial pyramid matching for recognizing natural scene categories[END_REF] or limited to weak constraints [START_REF] Csurka | Generic Visual Categorization Using Weak Geometry[END_REF][START_REF] Jégou | Improving bag-of-features for large scale image search[END_REF].

By exploiting the shape (affine) parameters of local features, our Feature Map Hashing (FMH) 6 [17] is the first to incorporate global feature geometry in the index, while enjoying invariance to affine transforms. While queries are very fast, the obtained representation is large. Thus, FMH originally scales only up to 50k images. One way to obtain a more compact representation is feature selection by matching multiple views of the same object or scene and focusing on the common parts. Consequently, FMH scales up to 1M images [START_REF] Tolias | Towards large-scale geometry indexing by feature selection[END_REF].

But, what happens to isolated views, having nothing to match with? Our SymCity7 [START_REF] Tolias | SymCity: Feature Selection by Symmetry for Large Scale Image Retrieval[END_REF] selects features from isolated views by detecting symmetries and repeating patterns. The underlying assumption is that in urban scenes, man-made structures exhibit a significant amount of such structure, while 'transient' objects like vehicles and persons standing in front of buildings do not. Building on our HPM [START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF], detection is blazing fast. close relation between clustering and nearest neighbor search as well as the success of one helping to solve the other, we introduce Dimensionality-Recursive Vector Quantization (DRVQ) [12], a paradigm where both problems are solved simultaneously. Traditionally, in the assignment step of the k-means algorithm, one needs to search for the nearest centroid for each data point. In DRVQ, we rather start from centroids and construct a distance map over the entire space. Thus, search reduces to a lookup operation. Consequently, DRVQ is orders of magnitude faster than any other solution.

In previous work [12,[START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF], we cluster local descriptors to construct visual vocabularies. By contrast, powerful global CNN representations allow scaling up to image clustering. By quantizing them, we introduce Inverted-Quantized k-Means (IQM) [15], an extremely efficient web-scale image clustering method, subsuming the properties of EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] (dynamic estimation of the number of centroids) and DRVQ [12] (inverted search from centroids to points). We achieve clustering of 100M images in less than an hour on a single machine. location recognition A particular application of image re-

2017.

trieval is visual location recognition. We introduce a panorama-to-panorama matching process for location recognition from CNN representations of street-view images [START_REF] Iscen | Panorama to Panorama Matching for Location Recognition[END_REF], reaching near-perfect performance on a challenging standard benchmark.

discovery of mid-level parts Learning mid-level discrimina-

2017.

tive parts for image classification is very common, even before deep learning. Our automatic discovery of discriminative parts [START_REF] Sicre | Automatic discovery of discriminative parts as a quadratic assignment problem[END_REF] casts part learning as a quadratic assignment problem, allowing the use of a number of known relaxations and optimization algorithms. It is based on pre-trained networks for feature extraction and makes predictions by a linear classifier on a part-based encoding.

An unsupervised version of this work is applied both to scene classification and instance-level search [START_REF] Sicre | Unsupervised part learning for visual recognition[END_REF]. By unsupervised, we mean that class labels are not used in part learning, while they are indeed used in learning the classifier on top of part-based encodings.

current work

There is also current unpublished work, carried out in 2019. This is even more relevant since it paves the way towards future ideas discussed in Chapter 19. We summarize this work here, accompanied with limited context and motivation. active learning. We contribute an immediate extension of DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] to deep active leaning [START_REF] Gal | Deep Bayesian active learning with image data[END_REF][START_REF] Sener | Active Learning for Convolutional Neural Networks: A Core-Set Approach[END_REF]. In active learning [START_REF] Settles | Active learning literature survey[END_REF], one begins learning on limited labeled data, selects which examples to label next from a large unlabeled pool and iterates in multiple cycles. Using this pool for selection only is the opposite of what would normally work well when learning a deep model from scratch.

We depart from this setting by using both labeled and unlabeled Using all available data at model training.

data during model training [START_REF] Siméoni | Rethinking deep active learning: Using unlabeled data at model training[END_REF]. We do so by using unsupervised feature learning at the beginning and semi-supervised learning at every cycle. We find that this brings a spectacular accuracy improvement compared to the differences between selection strategies. We also find that in the new setting, active learning is not effective when the quality of the representation is low, i.e. exactly when labeled data is limited. These findings suggest that we should revisit the standard settings and the evaluation protocol of deep active learning.

object detection. Another extension of the semi-supervised paradigm is to object detection. Here, the most common setting advocating less supervision is Weakly-Supervised Object Detection (WSOD) [START_REF] Bilen | Weakly supervised deep detection networks[END_REF], assuming no bounding box annotation but image-level labels on all data. There are mixed settings where e.g. few images come with bounding boxes and labels, and a large amount with image-level labels only [START_REF] Yan | Weakly-and semi-supervised object detection with expectation-maximization algorithm[END_REF]. These are sometimes called semi-supervised.

We believe that the true analogue to semi-supervised learning in Few images with image-level labels and plenty completely unlabeled images.

object detection is to have few clean images with image-level labels and a large amount of completely unlabeled images. We attack this more challenging problem for the first time by our Nano-Supervised Object Detection (NSOD) [START_REF] Yang | Training Object Detectors from Few Weakly-Labeled and Many Unlabeled Images[END_REF]. In particular, we first learn a teacher classifier by semi-supervised classification and then a student detector, training a WSOD model on pseudo-labels obtained by the teacher. By using more unlabeled images, we achieve performance competitive or superior to many state of the art WSOD solutions. attack called Boundary Projection (BP) [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] that improves upon existing methods by a large margin. Our key idea is that the classification boundary is a manifold in the image space: We thus quickly reach the boundary and then optimize distortion on this manifold. We also study the effect of quantization, which is ignored in most Quantization.

related work by treating perturbations as real. This is important because the attacker's goal is to publish adversarial images (e.g. on the Internet), and publishing implies encoding in bytes.

some thoughts

In the following, we revisit our contributions including some of those summarized in Sections [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF]. However, these results are on a dataset of SIFT descrip-LOPQ has a precision of 47% with 128-bit codes and a query time of 53ms at this scale [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF].

tors and an image has thousands of such descriptors.

One can of course use aggregated representations like BoW [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF], VLAD [START_REF] Jégou | Aggregating local descriptors into a compact image representation[END_REF] or Fisher vectors [START_REF] Perronnin | Improving the Fisher kernel for Large-scale image classification[END_REF][START_REF] Perronnin | Fisher Kernels on Visual Vocabularies for Image Categorization[END_REF]. However, none works as well as having access to local descriptors [START_REF] Jégou | Exploiting Descriptor Distances for Precise Image Search[END_REF][START_REF] Jégou | Hamming embedding and weak geometric consistency for large scale image search[END_REF] or aggregating over large vocabularies, as in our ASMK [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF].

CNN-based representations [START_REF] Gordo | End-to-End Learning of Deep Visual Representations for Image Retrieval[END_REF][START_REF] Radenović | Fine-tuning CNN Image Retrieval with No Human Annotation[END_REF] work better even than ASMK with a single vector representation of length e.g. 512. This is a game changer. Indeed, searching over one billion images becomes routine [START_REF] Babenko | Efficient Indexing of Billion-Scale Datasets of Deep Descriptors[END_REF] and the entire Flickr collection is indexed by LOPQ on deep descriptors in 2017 8 . Importantly, CNN-based representations are not constrained to instance-level similarity; they can be easily adapted to any similarity depending on the training task.

Of course, using powerful CNN representations and local features or descriptors improves performance further as shown in our benchmark [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] and DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF], but by increasing the cost. We need to always examine performance relative to resources taken, including the size of the representation and the query time.

on pooling So, why and how does it work? For one thing, shallow representations are just two layers deep, where the second layer involves learning a vocabulary on a training set; deep ones are based on learning highly nonlinear maps to optimize an objective like classification or ranking on a training set plus supervision.

More than that, the first layer of shallow representations is based on image patches, sparsely or densely sampled, and the second layer pools (aggregates) everything globally in one step, without taking into One-step global pooling.

account the appearance or geometry of the neighboring patches. One exception is Spatial Pyramid Matching (SPM) [START_REF] Lazebnik | Beyond bags of features: spatial pyramid matching for recognizing natural scene categories[END_REF], where pooling takes place at several levels of different spatial resolution, hence considering neighborhoods of different size. In both cases however, what is pooled is the quantized encodings over the vocabulary; the discriminative power of the original descriptors is lost. In SPM, pooling across all neighborhoods uses the same vocabulary. By contrast, CNNs pool the representation gradually over several lay-Gradual pooling over different neighborhoods.

ers. At each layer, neighborhoods of different size are used. What is pooled is continuous activations over filter kernels rather than quantized encodings. Different kernels are used at each layer, hence at each neighborhood. As a result, a deep descriptor at a particular location encodes appearance and geometry over a large receptive field and is very discriminative.

The same argument applies even in the absence of pooling, e.g. in the all convolutional network [START_REF] Tobias Springenberg | Striving for simplicity: The all convolutional net[END_REF]. In this case, hand-crafted spatial pooling is replaced by strided convolution, allowing the network to learn its own spatial downsampling. In any case, there is always a final global spatial pooling in order to extract a global representation, and there is apparently less interference among different locations compared to shallow representations.

follow the data By measuring similarity based on Euclidean distance, we make assumptions on the data distribution. In a highdimensional space, such assumptions may not be realistic. The input space of a 1 megapixel color image has 3 million dimensions. Are all This number is 75-150 million in the case of the human retina.

inputs equally likely? Certainly not: Natural images do not look like noise. This space is extremely sparsely populated and wildly inappropriate for any task other than capturing or displaying.

Then, all about representation learning is manifold learning [START_REF] Bengio | Representation Learning: A Review and New Perspectives[END_REF]: learning a nonlinear mapping to a lower-dimensional, more densely populated space, where dimensions act as natural coordinates in the input space. For instance, when the whole input shifts, few components should change in the representation space.

So, in the representation space, shallow or deep, is the distribution more well-behaved, like uniform, Gaussian, unimodal or clustered? Not really:

1. SIFT descriptors are a shallow representation of small image pa-128 dimensions.

tches and our experiments with LOPQ [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF] demonstrate that the statistics are different in each location in the descriptor space. In terms of compression efficiency, we need to adapt the encoder locally. Implicitly, LOPQ learns a nonlinear, non-smooth mapping before quantizing. images and our experiments with diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] demonstrate that in terms of similarity measurements, we need to adapt the metric locally. Implicitly, FSR [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF] learns a nonlinear, smooth mapping.

So, can we improve the representation space to make it more well-Manifold learning.

behaved, following the data distribution? This is what we do in MoM [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] and DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF]: We represent the manifold structure of the data by a nearest neighbor graph, propagate similarities on the graph and use them to learn a new representation space, with or without supervision. This is done iteratively in DLP; although we have not tested, the same may apply to MoM 9 .

Of course, doing this already assumes a compact representation Diffusion on shallow representations.

9 Recent results on unsupervised metric learning show that MoM fails when learning from scratch [START_REF] Cao | Unsupervised Deep Metric Learning via Auxiliary Rotation Loss[END_REF]. This might be fixed by iteratively updating the graph.

as we discuss above. Using shallow representations, the closest work to our diffusion is VisualRank [START_REF] Jing | VisualRank: Applying PageRank to large-scale image search[END_REF]. To measure similarity reliably, geometric verification is used, which makes similarity computation expensive. For this reason, truncation of the top 1000 items only takes place using text queries and then individual adjacency matrices are pre-computed offline per query, using hashing. By contrast, a single matrix over 1 million images is easily computed in our case [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF] using CNN-based descriptors without any approximation.

It is clear that a representation of a single vector per image is even more important in quadratic-complexity tasks like graph-based methods [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF][START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF][START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] or clustering, e.g. by IQM [15], than in linearcomplexity tasks like search. Of course, scaling up diffusion to billions is an open problem discussed in Chapter 19. 1000 words A picture is worth a thousand words. What does that mean for us? Literally, with enough resolution, we can arrange 32 × 32 arbitrary small images of objects in a large one. The large image becomes a container and its descriptor is pooled over the descriptors of the small ones, causing interference. This is an extreme example, but looking for a small object among clutter is very common. We are back to estimating partial similarity. In particular, we can: Partial similarity.

1. split into arbitrary regions and match independently, as in our regional diffusion [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF];

2. discover the objects offline and match independently, as discussed below;

3. use spatial matching on local features, as in DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF]; or 4. increase the dimensionality to reduce interference [START_REF] Iscen | Memory vectors for similarity search in high-dimensional spaces[END_REF][START_REF] Zaheer | Deep sets[END_REF].

All solutions have increased cost, either offline or online, space or time. The point is, we cannot expect to fit any number of arbitrary objects into a small vector. The argument on gradual pooling into descriptors that encode appearance and geometry over the receptive field makes sense up to the level of objects, which have some common appearance and geometry. It does not apply to clutter that may appear in any possible layout.

Interestingly, in our work on location recognition [START_REF] Iscen | Panorama to Panorama Matching for Location Recognition[END_REF], we average descriptors of images that are cropped out of a panoramic image, or pool over the activation tensor of the panoramic image directly. This works well but of course is a different situation. discover, discover, discover Of the solutions presented above on estimating partial similarity, discovering the objects is appealing because it may be done offline and does not require representing a collection of objects or searching online. A lot of our work is based on this idea. Candidate object regions may be found in isolated images, but not very precisely:

Hypothesizing in isolated images.

[ October 7, 2020 at 12:04classicthesis version 0.4 ]

1. Local features may be selected e.g. by detecting symmetries or repeating patterns, performing spatial matching of an image to itself, as we do in SymCity [START_REF] Tolias | SymCity: Feature Selection by Symmetry for Large Scale Image Retrieval[END_REF].

2. Object proposals may be found by bottom-up grouping of primitives, like local features or segments [START_REF] Kapsalas | Regions Of Interest for Accurate Object Detection[END_REF].

3.

A saliecy map may be obtained by bottom-up operations [START_REF] Rapantzikos | Spatiotemporal features for action recognition and salient event detection[END_REF] or directly from a CNN activation map as in CroW [START_REF] Kalantidis | Cross-Dimensional Weighting for Aggregated Deep Convolutional Features[END_REF]; regions may be detected on a saliency map by our EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF], as we do in GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

Objects may be discovered more precisely in image collections:

Discovering in image collections.

1. In video sequences, stereo or depth input, we can delineate objects very precisely, guided by motion or depth [START_REF] Doulamis | Efficient Summarization of Stereoscopic Video Sequences[END_REF].

2. In collections containing multiple views of the same object or scene, we can perform spatial matching pairwise to find inliers and focus on the common parts, as we do in feature selection [START_REF] Tolias | Towards large-scale geometry indexing by feature selection[END_REF].

3. Alternatively, we can find common parts of several views and represent all of them jointly, as we do in scene maps [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF].

4. Alternatively, we can match regions pairwise using CNN a representation. Optionally, we can find frequent objects according to graph centrality and weight objects accordingly, as we do in GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF].

Clearly, in all cases where we use exhaustive pairwise matching, Exploiting data structure.

this is because we assume unstructured image collections. Video sequences simplify the problem, because only consecutive frames need to be matched and displacements are small. Stereo or depth input is even easier; accurate objects nearly pop-out in this case.

In detect-to-retrieve [START_REF] Teichmann | Detect-to-retrieve: Efficient regional aggregation for image search[END_REF], a class-agnostic object detector is trained on a dataset annotated with bounding boxes. This is an efficient solution but, given a new dataset or domain, we cannot assume human annotation. GOD is unsupervised and can generate pseudo-annotation to train a detector instead.

It is also interesting to extend detection to instance segmentation. Given a collection comprising images or video depicting objects of unknown categories, optionally also a few images annotated with bounding boxes, a challenging goal would be to train an instance segmentation network to segment out all objects in the collection.

all-to-one or all-to-all? In classification, it is standard to All-to-one.

have an FC layer with one prototype vector per class and a loss function that is softmax followed by cross-entropy. Each example is treated independently and the objective is that its representation vector is closer to the prototype of the correct class than any other. Thus, all examples are compared to one prototype per class.

By contrast, the objective in metric learning is that each example, All-to-all. treated as an anchor, is closer to positives than to negatives, optionally by some margin. The amount of positives and negatives varies according to hard example mining [START_REF] Harwood | Smart Mining for Deep Metric Learning[END_REF][START_REF] Chao-Yuan | Sampling matters in deep embedding learning[END_REF]. However, it is becoming clear with global loss functions [START_REF] Cakir | Deep Metric Learning to Rank[END_REF][START_REF] Meyer | Nearest Neighbour Radial Basis Function Solvers for Deep Neural Networks[END_REF][START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF][START_REF] Revaud | Learning With Average Precision: Training Image Retrieval With a Listwise Loss[END_REF][START_REF] Ustinova | Learning deep embeddings with histogram loss[END_REF][START_REF] Wang | Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning[END_REF] that, if we consider mining as part of the loss function, the actual objective is that an anchor is attracted to or repelled from each other example by a different weight depending on the label and the distance [START_REF] Wang | Multi-Similarity Loss with General Pair Weighting for Deep Metric Learning[END_REF]. Thus, all examples are compared to all other examples.

There are intermediate situations where there are number of pro-All-to-many.

totypes (centroids) per class, that is, each class is modeled by a multimodal distribution [START_REF] Kostinger | Joint Learning of Discriminative Prototypes and Large Margin Nearest Neighbor Classifiers[END_REF][START_REF] Mensink | Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost[END_REF][START_REF] Movshovitz-Attias | No Fuss Distance Metric Learning Using Proxies[END_REF][START_REF] Rippel | Metric learning with adaptive density discrimination[END_REF][START_REF] Wohlhart | Optimizing 1-Nearest Prototype Classifiers[END_REF]. Thus, all examples are compared to many prototypes. The situation is the same in unsupervised methods that rely on pseudo-labels. In this sense, unsupervised representation learning can be seen as unsupervised metric learning in an all-to-all [START_REF] Wu | Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination[END_REF][START_REF] Ye | Unsupervised Embedding Learning via Invariant and Spreading Instance Feature[END_REF] or all-to-many [START_REF] Cao | Unsupervised Deep Metric Learning via Auxiliary Rotation Loss[END_REF][START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF] setting.

The argument for all-to-all metric learning is that by considering

Class separation and margin.

all pairs of examples, the class distributions become more separated. However, a similar effect can be achieved by applying a margin to the classification loss. In this sense, there are several all-to-one methods normalizing the class prototypes and computing cosine similarity with a temperature [START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF][START_REF] Qi | Low-Shot Learning With Imprinted Weights[END_REF][START_REF] Ranjan | L2constrained softmax loss for discriminative face verification[END_REF][START_REF] Zhang | Heated-up softmax embedding[END_REF], as well as methods adding different margins [START_REF] Deng | ArcFace: Additive angular margin loss for deep face recognition[END_REF][START_REF] Liu | SphereFace: Deep hypersphere embedding for face recognition[END_REF][START_REF] Wang | CosFace: Large margin cosine loss for deep face recognition[END_REF]. If all-to-one methods work well, then why do we need the more expensive all-to-all methods? Unfortunately, the situation is not clear because different methods are common in different tasks, e.g. finegrained classification, face recognition, image retrieval or even fewshot learning. In any case, there is no guarantee of class separation, since classes at learning and inference are different.

consolidation

It is time to consolidate the ideas and the discussion presented thus far. We provide a summary and open the way to new ideas discussed in Chapter 19.

summary We summarize our main contributions, connecting them and suggesting possible extensions. The contributions are ordered by coherence rather than following the separation of shallow vs. deep representations as in the technical parts of the manuscript.

1.

Although developed as a clustering method for vocabulary learning, EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] applies generally to fitting a mixture distribution and finding the number of components. We use it equally to fit regions to saliency maps in GOD. It could be used for object detection, non-maximum suppression or mode seeking problems, even in HPM. It is easy to differentiate and could be used anywhere in combination with EM, e.g. in capsules [START_REF] Hinton | Matrix capsules with EM routing[END_REF].

2. Although developed on shallow representations, ASMK [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF] is a generic method adapting descriptors to vocabularies. It still applies to deep representations [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF]; in fact recent work extends ASMK in this context [START_REF] Teichmann | Detect-to-retrieve: Efficient regional aggregation for image search[END_REF].

3. Assuming single-correspondence hypotheses, HPM [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] is a faster and more flexible spatial matching method than RANSAC, allowing for multiple objects. In DSM we experiment with FSM [START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] for spatial matching, but HPM could be used equally. Because it does not count inliers, HPM is also easy to differentiate, easier than RANSAC [START_REF] Brachmann | DSAC-differentiable RANSAC for camera localization[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF].

4. DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF] is primarily a local feature detector and may be used with any matching method. We treat feature channels as visual words, but descriptors could be taken as well, from the same locations in the activation map. In retrospect, since local features are so well separated in individual channels, it is surprising how all other detectors map everything to a single channel first, where local features are not well separated any more10 .

5.

LOPQ [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF] compresses data for nearest neighbor search; it works equally well on shallow local and deep global descriptors [START_REF] Babenko | Efficient Indexing of Billion-Scale Datasets of Deep Descriptors[END_REF]. It adapts locally the quantizers to the data distribution and since distributions in the representation space are unlikely to become uniform11 , LOPQ will probably remain relevant.

6. With only one or few vectors per image, diffusion [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF][START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] is becoming standard in manifold search [START_REF] Gu | Towards Optimal CNN Descriptors for Large-Scale Image Retrieval[END_REF][START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] and there are several extensions [START_REF] Chang | Explore-Exploit Graph Traversal for Image Retrieval[END_REF][START_REF] Liu | Guided Similarity Separation for Image Retrieval[END_REF]. It applies equally to instancelevel and category-level tasks; we use the same model in MoM for unsupervised metric learning and DLP for semi-supervised learning. Extending to billions is a challenge.

7.

MoM [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] is one of the first methods principally designed as unsupervised deep metric learning and there are now follow-up methods that are adapting supervised methods with pseudolabels [START_REF] Ye | Unsupervised Embedding Learning via Invariant and Spreading Instance Feature[END_REF] or combining with unsupervised feature learning [START_REF] Cao | Unsupervised Deep Metric Learning via Auxiliary Rotation Loss[END_REF]. MoM is essentially training a student model having manifold similarity as a teacher. This idea could be easily combined with any other loss function. Extending to semi-supervised learning would be very interesting.

8. DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] lies at the heart of the main theme of our work, with exploration of the manifold structure of the data improving the representation and vice versa. It is meant for semi-supervised classification, a category-level task, despite being based on diffusion, originally developed for instance-level search.

9. Our work on few-shot learning [START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF] is the first to consider activation maps 12 for implicit data augmentation and train several layers to convergence on the few novel-class examples. We consider alternative settings using more data at pre-training [START_REF] Lifchitz | Few-shot Few-shot Learning and the role of Spatial Attention[END_REF] or at meta-training [START_REF] Iscen | Graph convolutional networks for learning with few clean and many noisy labels[END_REF].

10. In scene maps [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF], we explore an unstructured image collection, find common parts and collect these parts coming from several images into a single representation. In GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF], we explore and find common parts similarly, then find repeating patterns and weight them according to centrality in each image. Naturally, we could combine the two ideas: representing images jointly and weighting by centrality. It would be interesting to extend to an inductive version to handle unseen images without having access to the collection. [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] .

road map We complete this chapter by building a road map of the ideas presented in this manuscript, summarizing everything in one phrase and opening the way to new ideas. On one hand, there is a path from representing individual items to

From representing to exploring.

exploring collections:

1. Improvements in visual representation and matching processes yield improved search by visual similarity, as measured quantitatively against human annotation. For instance, EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] and ASMK [START_REF] Tolias | To Aggregate or not to Aggregate: Selective Match Kernels for Image Search[END_REF][START_REF] Tolias | Image search with selective match kernels: aggregation across single and multiple images[END_REF] improve the representation, while HPM [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF][START_REF] Tolias | Speeded-Up, Relaxed Spatial Matching[END_REF] and DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF] improve matching.

2. Improved search yields improved discovery of global structure.

For instance, it yields improved nearest neighbor graphs, used by diffusion [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF][START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] and by GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF] to discover objects. It also yields improved clustering, used by scene maps [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF] to discover views of the same scene.

On the other hand, there is an opposite path from exploring collec-From exploring to representing.

tions to representing individual items:

1. Improvements in the quality of global structure yield improved search or improved representation of individual images or groups of images. For instance, diffusion [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF][START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF][START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF] explores data by manifold similarity; GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF][START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF] represents images by focusing on objects and suppressing clutter; and a scene map [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF] jointly represents aligned views of the same scene. However, this requires access to the target image collection.

2. Based on the global structure, deep parametric models can learn new representations and generalize to unseen images or entire collections. For instance, MoM [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] learns to respect the manifold structure of the data by unsupervised metric learning and DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] similarly by semi-supervised learning. Ideas on few-shot learning [START_REF] Iscen | Graph convolutional networks for learning with few clean and many noisy labels[END_REF][START_REF] Lifchitz | Few-shot Few-shot Learning and the role of Spatial Attention[END_REF][START_REF] Lifchitz | Dense Classification and Implanting for Few-shot Learning[END_REF] and adversarial examples [START_REF] Zhang | Smooth Adversarial Examples[END_REF][START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] are not as mature yet, but in the same direction.

We can summarize these observations in the following:

Exploring data and learning the representation are mutually beneficial.

So, where does that lead next? Most of our work considers completely unstructured data collections, this is why we need to search for structure. On one hand, the situation is easier in sequential data like video, where each frame is automatically associated to the next, although we still need to search for recurring patterns. On the other hand, the situation is more difficult if we do not have access to all data at all times. We would need to organize the continuous input into stored representations.

Chapter 19 attempts to pave the way towards lifting the boundary between training and test sets, as well as the boundary between models and data.

O U T L O O K

This last chapter is an attempt to draw a road map of ideas that are likely to come and research directions that may be worth exploring. We motivate discussion by discussing the role of computing power and data in the development of deep learning and the relative shortage of storage capacity in current hardware and memory mechanisms in modern networks. We put forth a vision for future research towards mechanisms translating storage capacity to better performance. Finally, we present a number of research directions that can be summarized as a differentiable, incremental version of most ideas we have discussed in this manuscript.

motivation

Progressing hand in hand, machine learning and computer vision have made huge leaps towards interpreting the world around us, automating tasks and solving problems beyond human reach. There are two catalysts in this development: computing power and data.

Once a mechanism is in place to translate more computing power

Computing power.

into better performance, the community instantly becomes enthusiastic in shifting from the pursuit of efficiency [START_REF] Lampert | Beyond Sliding Windows: Object Localization By Efficient Subwindow Search[END_REF][START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] to exhaustive parallel search [START_REF] Chen | Ten-sorMask: A foundation for dense object segmentation[END_REF][START_REF] Lin | Focal Loss for Dense Object Detection[END_REF], deeper architectures [START_REF] He | Deep residual learning for image recognition[END_REF][START_REF] Jacobsen | i-RevNet: Deep Invertible Networks[END_REF] and large-scale distributed training [START_REF] Goyal | Accurate, large minibatch SGD: Training ImageNet in 1 hour[END_REF][START_REF] You | Large batch training of convolutional networks[END_REF].

With the wide adoption of GPUs, the amount of computing power available to the average PhD student arguably grows by at least a factor of 100 between 2012 and 2017. The use of supercomputers for academic research becomes more and more common and the performance of the top system worldwide rises from 1.76 to 148 peta-FLoating-point Operations Per Second (FLOPS) between 2009 and 20191 . Yet, this is incomparable to the power of biological visual systems and is only expected to grow by further progress in hardware. A possible future is analog electronics [START_REF] Haensch | The Next Generation of Deep Learning Hardware: Analog Computing[END_REF][START_REF] Hu | Memristor-based analog computation and neural network classification with a dot product engine[END_REF][START_REF] Angad S Rekhi | Analog/Mixed-Signal Hardware Error Modeling for Deep Learning Inference[END_REF][START_REF] Shafiee | ISAAC: A convolutional neural network accelerator with in-situ analog arithmetic in crossbars[END_REF].

We often motivate a new method or task by arguing on human

Visual data.

skills that we do not yet understand [START_REF] Gülçehre | Knowledge matters: Importance of prior information for optimization[END_REF][START_REF] Brenden | Human-level concept learning through probabilistic program induction[END_REF][START_REF] Yip | Sparse Representations for Fast, One-Shot Learning[END_REF]. However, since the outburst of deep learning in 2012, the growth of visual data used in learning representations is less impressive than the growth of computing power, until 2017 [START_REF] Sun | Revisiting Unreasonable Effectiveness of Data in Deep Learning Era[END_REF]. Certainly, the average amount and quality of such data is also beyond comparison to the amount and quality of visual data that a grown-up human has seen. One obstacle is the need for supervision by humans, which is very Supervision.

expensive especially when it includes localization as in object detection [START_REF] Kuznetsova | The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale[END_REF][START_REF] Dim | Extreme Clicking for Efficient Object Annotation[END_REF] or dense annotation as in instance segmentation [146, outlook 239]; even more so in video [START_REF] Sergi Caelles | The 2018 DAVIS challenge on video object segmentation[END_REF][START_REF] Cordts | The Cityscapes Dataset for Semantic Urban Scene Understanding[END_REF]. Fortunately, via transfer learning [START_REF] Donahue | Decaf: A deep convolutional activation feature for generic visual recognition[END_REF][START_REF] Oquab | Learning and transferring mid-level image representations using convolutional neural networks[END_REF][START_REF] Yosinski | How transferable are features in deep neural networks?[END_REF] and progress in self-supervision [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF][START_REF] Doersch | Unsupervised Visual Representation Learning By Context Prediction[END_REF][START_REF] Gidaris | Unsupervised Representation Learning by Predicting Image Rotations[END_REF][START_REF] Pathak | Learning Features by Watching Objects Move[END_REF][START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF], noisy supervision [START_REF] Li | Learning without forgetting[END_REF][START_REF] Sun | Revisiting Unreasonable Effectiveness of Data in Deep Learning Era[END_REF], weak supervision [START_REF] Bilen | Weakly supervised deep detection networks[END_REF][START_REF] Zhou | Weakly Supervised Instance Segmentation Using Class Peak Response[END_REF] and mixed supervision settings [START_REF] Hu | Learning to segment every thing[END_REF][START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF][START_REF] Zhu | Object Recognition with and without Objects[END_REF], it is becoming increasingly clear that representation learning and potential human supervision on a given task can be essentially decoupled. This allows representation learning on larger scale datasets [START_REF] Caron | Unsupervised Pre-Training of Image Features on Non-Curated Data[END_REF][START_REF] Mahajan | Exploring the Limits of Weakly Supervised Pretraining[END_REF][START_REF] Zeki Yalniz | Billion-scale semi-supervised learning for image classification[END_REF]. Synthetic data [START_REF] Qiu | UnrealCV: Connecting computer vision to unreal engine[END_REF][START_REF] Richter | Playing for Benchmarks[END_REF] is helping in this direction too. Still, models and data are treated as distinct entities. It is standard Memory mechanisms.

practice to discard the training set once a representation has been learned. This is what makes incremental learning [START_REF] Li | Learning without forgetting[END_REF][START_REF] Sylvestre-Alvise | iCaRL: Incremental classifier and representation learning[END_REF][START_REF] Yoon | Lifelong Learning with Dynamically Expandable Networks[END_REF] challenging. Neural networks can memorize the training set under certain conditions, but not when learning to generalize from large training sets [START_REF] Radhakrishnan | Memorization in Overparameterized Autoencoders[END_REF][START_REF] Sablayrolles | Déjà Vu: an empirical evaluation of the memorization properties of ConvNets[END_REF]. Memory networks are an explicit memory mechanism for sequences like text or video [START_REF] Na | A Read-Write Memory Network for Movie Story Understanding[END_REF][START_REF] Sukhbaatar | End-To-End Memory Networks[END_REF]. But such mechanisms are far from becoming a standard component of the otherwise stateless networks used in vision tasks. By contrast, the human visual long-term memory has a massive capacity for details [START_REF] Brady | Visual long-term memory has a massive storage capacity for object details[END_REF].

In image retrieval [START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF], embeddings of a massive collection are in-Explicit storage and retrieval.

deed explicitly stored. However, this collection is again distinct from the training set, limiting the chances of optimizing the representation to the particular collection-e.g. GOD [START_REF] Siméoni | Graph-based Particular Object Discovery[END_REF] learns attention on a fixed representation. The reason is that any update in the representation would invalidate the stored embeddings. The stored collection is seen as a test set, hence any optimization would be seen as overfitting [START_REF] Liu | Guided Similarity Separation for Image Retrieval[END_REF]. While representation learning is end-to-end, storage still involves external post-processing like PCA and whitening. This is what makes retrieval on a dynamic collection challenging.

a vision

This discussion leads us back to the questions raised in Chapter 1.

What if there were no boundaries between training and testing? What if there was a continuous process of observing high-quality visual data, some times accompanied by supervision? What if we could not just learn tasks incrementally but memorize incrementally? What if we could recall instantly? What if we could learn the representation while memorizing? A vision for future research that we put forth in this last chapter is to progress towards making data a first-class citizen in visual recognition tasks. This refers to data representations becoming explicit part of a model rather than just its training process.

Just as there are mechanisms to automatically translate more computing power to better performance, the same should happen with storage capacity.

To make it more concrete: While the computing power may grow by a factor of 100 by using GPUs, the storage capacity drops by a factor of 10 at the same time; what if it could grow by a factor of 1000 instead? Are there mechanisms to use it in improving performance in visual recognition tasks, just like "stacking more layers"? Such mechanisms could lead for instance to an artificial long-term visual memory that learns what to store and what to forget for a given capacity. All operations would be online, continuously associating real-time input to stored data and keeping information organized at all times. Learning and inference would be one.

In light of this vision, Section 19.3 discusses a number of suggested research directions starting from the current state of the art. Just like the main goal of this manuscript, all directions are about learning visual representations from data with limited supervision and applying them to visual recognition tasks.

directions

This journey is certainly not over yet and there are too many ideas to be explored. Only a few are discussed here, in full conscience that ideas tend to be quickly invalidated or replaced by new in the ever changing landscape of machine learning and computer vision research. Some are natural extensions of the work discussed in this manuscript. Some rather lead to the formulation of new tasks.

rethinking metric learning There are many tasks where supervised metric learning (e.g. with pairs or triplets) appears to have a similar objective with supervised classification (e.g. with variants of cross-entropy), but classes at inference are different from classes at learning. These include e.g. fine-grained classification, face recognition, person re-identification, local descriptor learning and instance retrieval, as discussed in Section 13.2. Few-shot learning also includes two training stages with different classes and is treated as either metric learning or classification.

The connection between supervised metric learning and supervised classification is often unclear. Their relative performance is also unclear as comparisons often do not involve more than one tasks. Increasingly complex loss functions involving tuples of examples [START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF] raise the problem of sampling from a seemingly endless choice of tuples [START_REF] Chao-Yuan | Sampling matters in deep embedding learning[END_REF]. Ranking loss functions on even larger tuples [START_REF] Cakir | Deep Metric Learning to Rank[END_REF][START_REF] Revaud | Learning With Average Precision: Training Image Retrieval With a Listwise Loss[END_REF] appear to suffer less from this problem, but still both sampling and the loss function rely on supervision.

Our MoM [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF] is unsupervised and limits both positive and negative

Originally, contrastive loss [START_REF] Hadsell | Dimensionality Reduction By Learning an Invariant Mapping[END_REF] is applied in an unsupervised setting too.

pairs by nearest neighbor search. The unsupervised setting is gaining momentum [START_REF] Wu | Unsupervised Feature Learning via Non-Parametric Instance-level Discrimination[END_REF][START_REF] Ye | Unsupervised Embedding Learning via Invariant and Spreading Instance Feature[END_REF]. Ideally, metric learning should be explored in all settings where classification has been explored: e.g. semi-supervised, few-shot and incremental learning, seen or unseen categories, as well as distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF][START_REF] Radosavovic | Data Distillation: Towards Omni-Supervised Learning[END_REF]. In particular, ranking loss functions [START_REF] Cakir | Deep Metric Learning to Rank[END_REF][START_REF] Revaud | Learning With Average Precision: Training Image Retrieval With a Listwise Loss[END_REF] would be most interesting to explore in all such settings. In the unsupervised setting, it would allow e.g. self-learning to rank.

outlook Distillation would amount to training a student model to rank like a teacher model. In classification, each example is processed independently, while in metric learning, the loss is a function of more than one examples, resulting in greater cost. It is necessary to study, under all settings, the relative performance of the two approaches as a function of the amount and distribution of available data (e.g. multimodal or not). A better understanding of the properties of the two approaches will allow a smoother progress towards more challenging problems like long-tail [START_REF] Kang | Decoupling representation and classifier for long-tailed recognition[END_REF][START_REF] Wang | Learning to model the tail[END_REF] and open-set recognition [START_REF] Bendale | Towards open set deep networks[END_REF][START_REF] Liu | Large-Scale Long-Tailed Recognition in an Open World[END_REF].

It is natural to extend the above studies to localization tasks includ-Localization tasks.

ing spatial attention, object detection and instance segmentation. Different supervision settings have not been explored as much as in classification. One interesting example is our NSOD [START_REF] Yang | Training Object Detectors from Few Weakly-Labeled and Many Unlabeled Images[END_REF], which is in a sense a true analogue of semi-supervised learning in object detection. Also, a local version of metric learning has not been explored much. For instance, pairs (or ranking) of image regions in this problem would play the same role as pairs (or ranking) of images in metric learning.

sparse activations and alignment Among other tasks, convolutional networks are able to perform correspondence [START_REF] Christopher B Choy | Universal correspondence network[END_REF][START_REF] Long | Do Convnets Learn Correspondence?[END_REF] as discussed in Section 8.2. The applications range from dense optical flow in video [START_REF] Dosovitskiy | FlowNet: Learning optical flow with convolutional networks[END_REF] to sparse correspondence of object parts, even across semantic categories [1]. This has revived the interest in end-to-end trainable networks inspired from the conventional pipeline of feature detection, descriptor extraction and spatial matching, e.g. using RANSAC [START_REF] Brachmann | DSAC-differentiable RANSAC for camera localization[END_REF][START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF]. However, candidate correspondences are often dense and exhaustive over four dimensions [START_REF] Rocco | End-to-end weakly-supervised semantic alignment[END_REF], which is very expensive, or we are back to hundreds of high-dimensional descriptors per image, losing the compactness of the representation [START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF]. Sparse local feature detection takes place on a single channel and requires a significant amount of engineering and supervision [START_REF] Daniel Detone | SuperPoint: Self-supervised interest point detection and description[END_REF][START_REF] Moo | LIFT: Learned invariant feature transform[END_REF].

By introducing DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF], we rather show that (i) sparse local features arise in individual channels of convolutional activations without explicit training, and (ii) feature channels behave like visual words, dispensing the need for descriptors or vocabularies. Essentially, a sparse approximation of the activation tensor is a compact representation that gives rise to efficient pairwise matching under geometric alignment. Such matching is beyond the standard matching of vectors ob-E.g. by Euclidean distance, dot product or cosine similarity.

tained by global spatial pooling. In metric learning, images are considered in pairs, so alignment may become a natural part of the matching process. In classification (or detection), images (or regions) are matched against class prototypes-for instance, by an FC layer containing one prototype vector per class. Extending this matching to incorporate alignment is more challenging because prototypes involve averaging over classes. To allow deformation, matching should be flexible. To allow appearance variation, a multimodal class representation [START_REF] Mensink | Distance-Based Image Classification: Generalizing to New Classes at Near-Zero Cost[END_REF][START_REF] Rippel | Metric learning with adaptive density discrimination[END_REF] may be necessary, having multiple prototype tensors per class, potentially distributed over different layers.

Such extensions may open the door to end-to-end learning using

Why alignment?

geometrically aligned tensors in category-level tasks. Tensors are more discriminative than vectors obtained by spatial pooling, but they are not invariant. Explicit semantic alignment can answer the invariance vs. discriminative power dilemma. In semi-supervised settings, pairwise alignment can be used to transfer predictions from one image to another more reliably than global image similarity. This can improve graph-based methods like our DLP [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF], even when training a standard classifier. Alignment may also act as an attention mechanism to foreground objects. This is already useful in video, where objects may be identified through motion [START_REF] Pathak | Learning Features by Watching Objects Move[END_REF][START_REF] Wang | Unsupervised learning of visual representations using videos[END_REF] and in few-shot learning [START_REF] Hao | Collect and Select: Semantic Alignment Metric Learning for Few-Shot Learning[END_REF][START_REF] Hou | Cross Attention Network for Few-shot Classification[END_REF], where there are not enough examples to learn the foreground implicitly. It can of course help in unsupervised object discovery, e.g. by initializing regions via pairwise image matching [START_REF] Cho | Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals[END_REF] rather than individually per image as in our GOD [START_REF] Simeoni | Unsupervised object discovery for instance recognition[END_REF]. More interestingly, it may help in weakly supervised learning, where pixel-wise matching to a prototype vector notoriously focuses on the most discriminative object parts, failing to discover entire objects [START_REF] Hou | Self-erasing network for integral object attention[END_REF][START_REF] Kim | Two-Phase Learning for Weakly Supervised Object Localization[END_REF].

high-dimensional convolution The development of CNNs assumes that the input lies in a 2d Euclidean space. Handling higherdimensional or non-Euclidean data is often treated by Graph Convolutional Networks (GCNs) [START_REF] Defferrard | Convolutional neural networks on graphs with fast localized spectral filtering[END_REF][START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF], where only pair-wise scalar affinity is used between input elements. This works well when only scalar information is available, but is not appropriate when the input originates E.g. in the case of documents and citations.

in Euclidean space, e.g. 2d images, 3d video (where one dimension is time), or 3d shapes (point sets or surfaces).

Early approaches to 3d data operate on multiple 2d views [10]. This can take advantage of networks pre-trained on visual data, but incurs information loss due to projection. PointNet [START_REF] Charles R Qi | PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation[END_REF] operates directly on 3d point clouds but does not allow for local interaction between points. The latter can be achieved by centering objects on the origin and expressing data and the convolutional operation in spherical coordinates [START_REF] Taco | Spherical CNNs[END_REF][START_REF] Esteves | Learning SO(3) equivariant representations with spherical CNNs[END_REF]. This also yields 3d rotation invariance but still involves projection on a sphere.

More principled approaches generalize 2d convolution to 3d, while maintaining the sparse representation [11,[START_REF] Puy | Unifying local and non-local signal processing with graph CNNs[END_REF][START_REF] Wang | Dynamic graph CNN for learning on point clouds[END_REF]. Unlike GCN [START_REF] Defferrard | Convolutional neural networks on graphs with fast localized spectral filtering[END_REF][START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF], such generalizations preserve coordinates and use them to define local interactions via a kernel, just like standard convolution. However, they are still restricted to the same point set as the input. A 2d analogue of this idea is to represent 2d sketches by a binary image and restrict the output of convolution to the sketch [START_REF] Graham | Submanifold sparse convolutional networks[END_REF]. This is against the principle of hierarchical representation of CNNs, i.e., activations at certain locations in a layer give rise to activations at different locations in the next [START_REF] Fukushima | Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected By Shift in Position[END_REF]. However, dense 3d activations [START_REF] Zhou | VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection[END_REF] are too large to handle over a deep architecture. volution whereby activations are sparse but localized onto arbitrary locations. This could be accomplished by using a mixture model for both activations and convolution kernels [11], but then fitting a new output mixture instead of restricting to the input point set. There is a connection of this approach to capsules [START_REF] Hinton | Matrix capsules with EM routing[END_REF] in that EM is used to fit a model of capsules in a layer to votes from the previous layer. By allowing an arbitrary number of points, the challenge is to dynamically control the number of output components. Our EGM [START_REF] Avrithis | Approximate Gaussian mixtures for large scale vocabularies[END_REF] could be investigated in this direction.

Such sparse parametric convolution would be useful not just for highdimensional input, but for 2d images too. For instance, it would allow a sparse representation of a scale-space, which is inherently 3d. More generally, it would allow a transformation or pose space where the size of the representation would depend on the input data only and not on the dimensionality of the space.

Clearly, work on high-dimensional data is not as mature as on 2d

New supervision settings.

visual data. For instance, networks for 3d point sets are typically used for tasks like classification or semantic part segmentation under full supervision. A straightforward direction is then to explore new tasks and supervision settings like metric learning for similarity retrieval [START_REF] He | Triplet-Center Loss for Multi-View 3D Object Retrieval[END_REF], semi-supervised, weakly supervised and few-shot learning. Of course, self-supervision will allow the use of large-scale unlabeled 3d data collections for pre-training. This is very important since labeled 3d data is not as abundant as 2d images.

manifolds, indexing and geometry As discussed in Chapter 10, CNNs provide a powerful representation with only one or few vectors per image, allowing efficient manifold similarity search. We develop a spatial (or 'temporal') approach [START_REF] Iscen | Efficient Diffusion on Region Manifolds: Recovering Small Objects with Compact CNN Representations[END_REF] based on a linear system solution, a spectral approach [START_REF] Iscen | Fast Spectral Ranking for Similarity Search[END_REF], where similarity is based on dot product in an embedding space, and a hybrid approach [START_REF] Iscen | Hybrid Diffusion: Spectral-Temporal Graph Filtering for Manifold Ranking[END_REF], controlling the space-time trade-off between these two extremes. As discussed in Chapter 14, we also use manifold similarity for unsupervised metric learning [START_REF] Iscen | Mining on Manifolds: Metric Learning without Labels[END_REF], where the objective is that Euclidean similarity in the learned embedding space behaves like manifold similarity in the original feature space. Finally, as discussed in Chapter 15, we use manifold similarity for semi-supervised classification [START_REF] Iscen | Label propagation for Deep Semi-supervised Learning[END_REF] and active learning [START_REF] Siméoni | Rethinking deep active learning: Using unlabeled data at model training[END_REF], assigning pseudo-labels to unlabeled data according to manifold similarity to labeled data per class.

However, all solutions are based on a nearest neighbor graph. How

Billion-scale manifold similarity search.

can manifold similarity search scale further up e.g. to billions of vectors? Currently, the only such mechanism is to truncate the adjacency matrix, essentially only re-ranking the elements of the corresponding subgraph with no hope of retrieving beyond that. Unfortunately, truncation still relies on Euclidean distance in the feature space. As discussed in Chapter 6, indexing schemes based on quantizers like our LOPQ [START_REF] Kalantidis | Locally Optimized Product Quantization for Approximate Nearest Neighbor Search[END_REF] can scale up to billions of vectors in the Euclidean space. It is interesting to investigate truncation based on quantizers in the spectral embedding space. Further improvement is possible by defining the adjacency matrix Geometry-driven manifold similarity.

according to spatial matching, which is more reliable than matching global vector representations [START_REF] Chang | Explore-Exploit Graph Traversal for Image Retrieval[END_REF]. Still, using just a scalar gives little clue whether similarity is transitive, hence diffusion may suffer from drift. For instance, by walking along a street and looking sideways, diffusion will tell us that everything we see is similar. A solution may be to generalize the operations. In the adjacency matrix, we replace scalar similarities by transformations found via spatial matching, like our DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF]. In matrix-vector multiplication, we replace scalar multiplication by composition of transformations and addition by mode seeking in the transformation space, like our This process bears similarities to iconoid shift [START_REF] Weyand | Discovering Favorite Views of Popular Places with Iconoid Shift[END_REF].

HPM [START_REF] Avrithis | Hough Pyramid Matching: Speeded-Up Geometry Re-Ranking for Large Scale Image Retrieval[END_REF]. By doing so, we keep track of the transformation between the query and each image in the dataset, such that we can stop propagating when the object of interest goes out of sight.

According to Chapter 10, the diffusion process that we follow for Graph convolution [START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF] and geometry.

manifold similarity search is linear and shallow. A nonlinear, deep version is a GCN [START_REF] Thomas | Semi-supervised classification with graph convolutional networks[END_REF]: Every propagation step becomes a layer and is accompanied by a linear mapping over a feature space and a nonlinearity. Of course, learning a GCN for manifold similarity search [START_REF] Liu | Guided Similarity Separation for Image Retrieval[END_REF] is in fact overfitting the test set and can handle unseen queries but not unseen datasets. Nevertheless, this approach still makes sense in an incremental learning scenario, where the training set is memorized and the graph is dynamically updated. Another difficulty is that learning a GCN typically assumes that the entire dataset resides in memory such that propagation is performed over the entire graph at each layer. For back-propagation, all activations reside in memory as well. At large scale, the standard solution is stochastic optimization using mini-batches, which means storing all intermediate activations of the dataset. A challenging scenario is to compute the graph dynamically per layer [START_REF] Wang | Dynamic graph CNN for learning on point clouds[END_REF], according to spatial matching. Then, activations become tensors and a compact representation as in our DSM [START_REF] Siméoni | Local Features and Visual Words Emerge in Activations[END_REF] becomes indispensable. learning while memorizing The dominant paradigm in category-level tasks like classification and detection is that the training set is disposed of at inference, leaving all accumulated knowledge in the network parameters. The dominant paradigm in instance-level tasks like retrieval is that the test set is indexed against a fixed representation, because any network update would necessitate re-computing embeddings and re-indexing the entire test set.

Both paradigms may be challenged by a memory that is growing as we learn. In category-level tasks, a 'summary' of the training set e.g. a subset or a multimodal distribution per class can be made accessible while learning a new task. In instance-level tasks, training and test sets become part of a continuously growing knowledge, while the representation is updated as more data is stored. outlook Summarizing the training set of the current task by means of a subset or a multimodal parametric distribution in the input or feature space is common e.g. in metric learning [START_REF] Kostinger | Joint Learning of Discriminative Prototypes and Large Margin Nearest Neighbor Classifiers[END_REF][START_REF] Movshovitz-Attias | No Fuss Distance Metric Learning Using Proxies[END_REF][START_REF] Rippel | Metric learning with adaptive density discrimination[END_REF], few-shot learning [START_REF] Zhang | Variational Few-Shot Learning[END_REF] and adversarial defenses [START_REF] Caldelli | Exploiting CNN Layer Activations to Improve Adversarial Image Classification[END_REF][START_REF] Sitawarin | Defending Against Adversarial Examples with K-Nearest Neighbor[END_REF]. The same is common for the training set of previous tasks e.g. in incremental learning [START_REF] Belouadah | IL2M: Class Incremental Learning With Dual Memory[END_REF][START_REF] Francisco | End-to-End Incremental Learning[END_REF][START_REF] Sylvestre-Alvise | iCaRL: Incremental classifier and representation learning[END_REF] and few-shot learning [3,[START_REF] Qi | Low-Shot Learning With Imprinted Weights[END_REF]. In the latter case, the data is represented in the input space in order to update the network. Of course, this approach does not scale well.

In metric learning [START_REF] Cakir | Deep Metric Learning to Rank[END_REF][START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF] and retrieval [START_REF] Gordo | Deep Image Retrieval: Learning global representations for image search[END_REF][START_REF] Radenović | CNN Image Retrieval Learns from BoW: Unsupervised Fine-Tuning with Hard Examples[END_REF] tasks, it is standard to memorize the entire test set explicitly in the feature space. Following a common paradigm of few-shot learning [START_REF] Qi | Low-Shot Learning With Imprinted Weights[END_REF], the embeddings of the test set may be thought of as an extremely wide FC layer. To our knowledge, storing the training set and incremental learning have not been explored in this context.

How should the data be represented? If it is meant to use the data Storing data.

in updating the network, then we should choose some layer before any layers to be updated (e.g., input images if the entire network is to be updated); spatial information needs to be preserved and averaging is difficult. If it is meant to search by similarity to new examples, then a representation should be chosen among the deepest layers; spatial pooling into global vectors and averaging over the data is possible, but any network updates invalidate the data. It would be interesting to investigate a compromise between these two extremes, i.e. use the representation of some intermediate layer, allowing both search by similarity and updates of all subsequent layers. Sparse activation tensors can keep the representation compact and averaging over similar examples can be an option, assuming alignment. Alternatively, invertible architectures [START_REF] Behrmann | Invertible Residual Networks[END_REF][START_REF] Gomez | The reversible residual network: Backpropagation without storing activations[END_REF] would allow reconstruction from deep features.

How should stored data be used while learning? In incremental Recalling data.

classification, it is common to apply a distillation loss [START_REF] Francisco | End-to-End Incremental Learning[END_REF][START_REF] Sylvestre-Alvise | iCaRL: Incremental classifier and representation learning[END_REF] to preserve predictions on previous tasks. Alternatively, it is possible to constrain the network, e.g. with synaptic plasticity mechanisms [6, 227, These ideas date back to Hebb [START_REF] Olding | The organization of behavior: a neuropsychological theory[END_REF].

292] to prevent updates of connections that are important for previous tasks, or network expansion mechanisms [START_REF] Fernando | PathNet: Evolution channels gradient descent in super neural networks[END_REF][START_REF] Rusu | Progressive neural networks[END_REF][START_REF] Yoon | Lifelong Learning with Dynamically Expandable Networks[END_REF] that guarantee predictions on previous tasks. The more explicit the mechanism, the larger the required architecture. An interesting direction to investigate is to apply such mechanisms to metric learning and instance-level tasks. The challenge is the sheer volume of stored embeddings that need to be preserved. A possible approach is to focus, for each new example, to similar data previously stored. By doing so, we maintain a summary of stored representations. All previous ideas apply in this case, including search by manifold similarity, sparse activations and geometry. colophon This document was typeset using the typographical look-and-feel classicthesis developed by André Miede. The style was inspired by Robert Bringhurst's seminal book on typography "The Elements of Typographic Style". classicthesis is available for both L A T E X and L Y X: https://bitbucket.org/amiede/classicthesis/ Happy users of classicthesis usually send a real postcard to the author, a collection of postcards received so far is featured here: http://postcards.miede.de/

1 O

 1 R I E N TAT I O N

orientation 1 .

 1 2 about this manuscriptThis manuscript is about a journey. The journey of computer vision A journey.

Figure 1 . 1 ,

 11 between December 2019 and January 2020. I submitted a complete version 0.3 to the reviewers in February and received the reviews in March 2020. I defended my HDR in Rennes, on July 3, 2020.

F

  p) = σ(p)R(p) and σ(p), R(p), t(p) stand for isotropic scale, orientation and position, respectively. R(p) is an orthogonal 2 × 2 matrix with det R(p) = 1, represented by an angle θ(p).

Figure 4 . 1 :

 41 Figure 4.1: (Top) Inliers found by 4-Degrees of Freedom (DoF) FSM. (Bottom) HPM matching, with all tentative correspondences shown. The ones in cyan have been erased. The rest are colored according to strength, with red (yellow) being the strongest (weakest).

  g(b) := [| ĥ(b)| -1] + , (4.7) where ĥ(b) := h(b) \ E are the correspondences we have kept in b. Let b 0 ⊆ • • • ⊆ b be the sequence of bins containing a correspondence c up to level such that b k ∈ B k for k = 0, . . . , . For each k, we approximate the affinity α(c, c ) of c to any other correspondence c ∈ b k by a fixed quantity, assumed a non-negative, non-increasing λ controls the relative importance of successive levels, i.e. how relaxed matching is. level affinity function of k, α(k) := 2 -λk . (4.8)

s

  (c) := g(b 0 ) + k=1 α(k){g(b k )g(b k-1 )}.

  s(c) := s L-1 (c) is the strength of c, and excluding all erased assignments E := E L-1 , we define the similarity score of images P, Q s(C) := c∈C\E w(c)s(c). (4.10) If c, c are two conflicting assignments and b ∈ B is the first (finest)

2 [

 2 April 22, 2019 at 3:22classicthesis version 0.1 ]

Figure 4 . 2 :

 42 Figure 4.2: Matching of nine assignments on a 3-level pyramid in 2d space.Colors denote visual words, and edge strength denotes affinity. The dotted line between c 6 , c 9 denotes a group that is formed at level 0 and then broken up at level 2, since c 6 is erased.

9 1 4 6w(c 9 )[[

 949 April 18, 2019 at 12:50classicthesis version 0.1 ] April 21, 2019 at 16:47classicthesis version 0

Figure

  Figure 4.3: Referring to Figure 4.2: (Left) Assignment labels, features and strength. Vertices and edges denote features and assignments, respectively. Assignments c 5 , c 6 and c 7 , c 8 are conflicting, and c 6 , c 7 are erased. Assignments c 1 , . . . , c 5 join groups at level 0; c 8 , c 9 at level 2.(Right) Affinity matrix equivalent to strengths (4.9). Assignments have been arranged so that groups are contiguous. Groups formed at levels 0, 1, 2 have affinity 1, 1 2 , 1 4 respectively.

Figure 5 . 1

 51 shows the effect of this σα plays the same role as w in (5.3), applied to similarities instead of distances.

Figure 5 . 1 :

 51 Figure 5.1: Matching features with descriptors assigned to the same visual word and similarity above threshold τ = 0.25. (Top) α = 1; (bottom) α = 3. Color denotes descriptor similarity σ α (r(x) r(y)), with yellow (red) being low (high).

Figure 5 . 2 :

 52 Figure 5.2: Examples of features mapped to the same visual word. Top 25 visual words are drawn, each in different color, according the number of features mapped to them.

[Figure 6 . 1 :

 61 Figure 6.1: Four quantizers of 64 centroids (

Figure 7 . 1 :

 71 Figure 7.1: Map of Athens illustrating geo-clusters for r g = 700m. Black dots, red markers and red circles stand for photos, codebook vectors and cluster boundaries, respectively.

Figure 7 . 2 :

 72 Figure 7.2: Photos in a sample of visual clusters from Pantheon, Rome, one cluster per row. The first image in each cluster is its center.

Figure 7 . 3 .

 73 spatial clustering For each reference image x ∈ Q(X) and corresponding visual cluster C v (x) we collect a set of features P w (x) per visual word w ∈ W as the union of features over all images y ∈ C v (x), after aligning with the referenceP w (x) := y∈Cv(x){H xy F (p) : p ∈ P w (y)}.(7.6)Here, P w (y) is the set of local features of image y assigned to visual word w ∈ W, and 3 × 3 matrix F (p) expresses the position and local shape of feature p, as in (4.1).

Figure 7 . 3 :

 73 Figure 7.3: View alignment of 10 photos of Palau Nacional, Montjuic, Barcelona, in a visual cluster.

Figure 7 . 4 :

 74 Figure 7.4: Scene map: Features within red box of Figure 7.3, before (left) and after (right) spatial clustering. Colored by visual word.

Figure 7 . 5 :

 75 Figure 7.5: VIRaL response to a query image. (Top left) Map depicting locations of similar images (blue markers) and estimated location of the query image (red). (Top right) Query image with sets of frequent and suggested tags, the latter linking to Wikipedia articles. (Bottom) Retrieved visually similar images.

Figure 7 . 6 :

 76 Figure 7.6: VIRaL Routes. A personal photo set from a trip to Venice is displayed on a map. The route is automatically inferred.

Figure 7 . 6

 76 depicts an example.

. 4 ) 1 .

 41 Both are singular and positive-semidefinite. The eigenvalues of L are in the interval [0, 2] [77] and those of W in [-1, 1]. Hence, if λ 1 , . . . , λ n are the eigenvalues of W, its spectral radius Each eigenvector u of L associated to eigenvalue 0 is constant within a connected component of G, e.g., L1 = D1 -W 1 = 0 if G is connected. The corresponding eigenvector of L is D 1/2 u.

. 5 )[Figure 10 . 1 :

 5101 Figure 10.1: Diffusion on a synthetic dataset in R 2 . Dataset points, query points and their k-NN are shown in blue, red, and green respectively. The adjacency matrix is defined according to (10.2) with k = 15. Diffusion follows by (10.4) with α = 0.99 and y by (10.6).Contour lines correspond to ranking scores after diffusion.

Figure 10 .

 10 1(b) illustrates diffusion on multiple query points. It is evident that multiple manifolds are captured in this case.

. 7 )

 7 Substituting x ← z, b ← (1α)y, and A ← (1α)L α = I -αW (9.6), (10.7) then becomes z (t) := αWz (t-1) + (1 -α)y.

[[[[Figure 10 . 2 :

 102 Figure 10.2: Retrieval as smoothing. (a) Input signal y. (b) Output of an Exponential Moving Average (EMA) filter given by recurrence z i := αz i-1 + (1α)y i (9.1). Compare to (10.3). This assumes a directed graph G, in blue. (c), (d) We use an undirected graph G instead. Information "flows" in both directions, controlled by edge weights. The sample in red is the query, and output signal z is its similarity to all samples.

  basis) Find a rank-r eigenvalue decomposition U r Λ r U r ≈ W, where n × r matrix U r has orthonormal columns and r × r matrix Λ r is diagonal. In particular[151, §5.3]: a) Form the r × r matrix C := Q WQ. Even if n is very large, r is small enough such that this decomposition is tractable. b) Compute its eigendecomposition V r Λ r V r = C.c) Define the matrix U r := QV r . An average-case bound on QQ W -W for the approximation of stage 1 decays to |λ r+1 | exponentially fast in the number of iterations q [151, §9.3,10.4]. Since W is symmetric, stage 2 indeed yields W ≈ QQ

[ 1 ]Figure 10 . 3 :

 1103 Figure 10.3: mAP of regional diffusion vs. number of iterations by solving (10.8) with CG against iterative process RWR (10.3), using VGG features (d = 512). Labels indicate diffusion time (s).

  [x]+ := max(x, 0) is the positive part of x ∈ R. [ October 7, 2020 at 12:04classicthesis version 0.4 ] 10.6 discussion

  searching

Figure 11 . 1 :

 111 Figure 11.1: Fast Spatial Matching (FSM)[START_REF] Philbin | Object Retrieval With Large Vocabularies and Fast Spatial Matching[END_REF] aligns two views based on a local features. Inlier correspondences shown, colored by "visual word". Are we using (a) Hessian-affine[START_REF] Mikolajczyk | Scale & Affine Invariant Interest Point Detectors[END_REF] + SIFT[START_REF] David G Lowe | Object recognition from local scale-invariant features[END_REF]? (b) LIFT[START_REF] Moo | LIFT: Learned invariant feature transform[END_REF]? (c) DELF[START_REF] Noh | Large-scale image retrieval with attentive deep local features[END_REF]? Or (d) local maxima on vanilla activation maps, without descriptors or vocabularies?

  local feature detection We use Maximally Stable Extremal Re-On images, both maxima (MSER + ) and minima (MSER -) are detected. On CNN activations, only maxima are of interest.

[

  July 28, 2019 at 0:35classicthesis version 0.1 ]

Figure 11 . 3 :

 113 Figure 11.3: DSM architecture. Two input images x 1 , x 2 are mapped by network f to activation tensors A 1 , A 2 respectively. Sparse local features P 1 , P 2 extracted independently per channel by detector (g) undergo spatial matching (s), resulting in a collection of inliers.In retrieval, only P 1 , P 2 are stored and s applies at re-ranking.

Figure 12 . 1 :

 121 Figure 12.1: Saliency map (right) computed for input image (left) based on our salient region detection on Instre dataset. Background clutter and objects not relevant for Instre are removed.

  Let Ω := [w] × [h] be the spatial domain, r := |Ω| = wh the spatial We write [n] := {1, . . . , n} for n ∈ N. resolution and A ( ) (r) the element of A at position r ∈ Ω and channel ∈ [c]. Following CroW, we use an IDF-like weight b :=log a + c j=1 (a k + ) (12.1) for ∈ [c], where a := 1 wh r∈Ω 1[A ( ) (r) = 0] is the average number of nonzero elements of channel ∈ [c].We then compute a weighted sum over activation channelsF (r) := c =1 b A ( ) (r)(12.2)for r ∈ Ω. Finally, we obtain the 2d FS map F ∈ R w×h by normalizingContrary to CroW, we use the feature channel weights when computing the 2d spatial weights.

Figure 12 . 2 :

 122 Figure 12.2: Oxford5k (top) and Instre (bottom) images, along with superimposed FS and OS maps and regions detected by EGM, in red.

9 Figure 12 . 3 :

 9123 Figure 12.3: Evolution of regions during EGM iterations on the FS map of the image on the left; i: iteration; k: number of regions.

[Figure 12 . 4 :

 124 Figure 12.4: Histogram of saliency precision for FS and OS maps on Instre.

Figure 14 . 1 :

 141 Figure 14.1: Given an anchor point (black) and its k nearest Euclidean (NN k ) and manifold (NN m k ) neighbors in a dataset (cyan), we choose positive examples as NN m k \ NN k , and negative as NN k \ NN m k . The selection is fully unsupervised, including anchors.

Figure 14 . 2 :

 142 Figure 14.2: CUB200-2011 anchor images x r , positives N + (x r ) and negatives N -(x r ) used for fine-grained classification. True positives (negatives) in green (red).

Figure 14 . 2

 142 Figure 14.2 shows examples of anchors with positives and negatives. True positives (negatives) are examples with same (different) label as the anchor. Despite the absence of labels at training, we achieve a very clean negatives and a reasonably clean positives.

Figure 14 . 3

 143 shows examples of anchors with positives and negatives. Positives are challenging, while negatives are interesting, depicting different objects, which still look similar.

Figure 14 . 3 :

 143 Figure 14.3: Flickr7M anchor images x r , positives N + (x r ) and negatives N -(x r ) used for particular object retrieval.

Figure 15 . 1 :

 151 Figure 15.1: Toy example of Label Propagation (LP). Triangles (circles) indicate labeled (un-labeled) examples. (a) Gray (color) indicates unlabeled (labeled by color) examples. (b) LP predictions by (15.5) (certainty by (15.7)) indicated by color (size).

  supervised loss In supervised classification, network parameters θ are learned by minimizing a supervised cost function of the form This function applies only to labeled examples XL.

[Figure 15 . 2 :

 152 Figure 15.2: Deep Label Propagation (DLP). We first learn a classifier f θ on labeled examples X L (15.2). We then iterate (a) computing a k-NN graph of the current features V := φ θ (X) and its regularized Laplacian L α (9.6), (b) performing LP (15.6), and (c) training f θ with true labels y L on labeled examples X L and pseudo-labels ŷU (15.5) + weights (15.7) on unlabeled examples X U (15.8).

[Figure 15 . 3 :

 153 Figure 15.3: Accuracy of pseudo-labels predicted by LP (15.5) against network (15.1) using C13 on CIFAR-10 with 500 labels.

[[ 90 Figure 15 . 4 :

 90154 Figure 15.4: Distribution of weights ω i (15.7) for unlabeled images at different epochs during training of C13 on CIFAR-10 with 500 labels.

  [n ], let S j := {i ∈ S : y i = j} index the support examples in S labeled in class j ∈ [c ]. The prototype c j ∈ R r×d of class j is given by the average of those examples c j = 1 |S j | i∈S j φ θ (x i ) (16.1) for j ∈ [c ]. If C := (c 1 , . . . , c c ), the classifier is defined as For n ∈ N, we write [e(i)] n i=1 := (e(1), . . . , e(n)) for expression e(i) of variable i ∈ [n]. f θ,C (x) := σ [s(φ θ (x), c j )] c j=1 (16.2) for x ∈ X , where s is cosine similarity and σ : R m → R m is the softmax function σ(a) := (e a 1 , . . . , e am ) m j=1 e a j for a ∈ R m , m ∈ N.

  These collections are supposed to be support examples and queries of novel classes; queries are now labeled and the goal is to classify them correctly.

16. 4 [[Figure 16 . 1 :[Figure 16 . 2 :

 4161162 Figure 16.1: An embedding φ(x) is compared to class weights w j by similarity (s); softmax (σ) and cross-entropy ( ) follow. (a) Flattening is equivalent to class weights having the same shape r × d as φ(x). (b) By global pooling, φ(x) is reduced (Σ) to vector a ∈ R d before being compared to class weights, which are in R d too.

  w j ∈ R d is the weight parameter of class j ∈ [c], (16.4) becomesThis operation is 1 × 1 convolution followed by depth-wise softmax.

f

  θ,W (x) := σ τ [s(φ θ (x)(r), w j )] c j=1 r∈Ω (16.6) for x ∈ X . Then, f θ,W (x)(r) ∈ R c is a vector of class probabilities at position r ∈ Ω, while f θ,W (x) (j) ∈ R r is the probability of class j ∈ [c] as a function of position. This differs from Class Activation Given A ∈ R r×c , A (j) ∈ R r collects elements of A at all positions in channel j ∈ [c].

Figure 16 . 3 :

 163 Figure 16.3: Examples overlaid with CAM [514] using ResNet-12 (cf . §16.7) trained with GAP or DC (16.6). Blue (red) is low (high) activation for ground truth. Top: Base classes (walker hound, tile roof ). Bottom: Novel classes (king crab, ant).

Figure 16 . 4 :

 164 Figure 16.4: Neural implants are convolutional filters in a new processing stream parallel to the base network. The input of an implant is the depth-wise concatenation of activations from both streams. Parameters learned in stage 1 are frozen in stage 2.

17

 17

  .2 preliminaries classifier Let X := [0, 1] n×d denote images of n pixels and d color The parameters of the classifier are not shown: They remain fixed. channels. A classifier f : X → R k maps an image x ∈ X to a vector f (x) ∈ R c of class probabilities over c given classes. The prediction π : X → [c] maps x ∈ X to the class of maximum probability: [n] := {1, . . . , n} for n ∈ N. π(x) := arg max k∈[c] f (x) k . (17.1) If a true label t ∈ [c] is known, the prediction is correct if π(x) = t.

17 . 1 :

 171 Figure 17.1: (Zoom in for better view.) Magnified versions of adversarial examples generated by different attacks (b-d), (f-g) on input image (a), revealing the adversarial perturbation. Our smooth adversarial example, even when magnified (f), is indistinguishable from the magnified version of the original (e).

  [x]+ := max(x, 0) is the positive part of x ∈ R.m (p, k) := [log p kmax j =k log p j + m] + (17.9) for p ∈ R c , k ∈ [c],encouraging logit log p k to be less than any other m is a hard version of the negative cross-entropyfor m = 0.

59 Figure 17 . 2 :

 59172 Figure 17.2: Attacks against Inception-v3 on ImageNet. (d,e) Perturbation z, scaled to [0, 1] independently per channel. Despite its higher distortion, z is 'smooth like' x and totally invisible for sCW.

4 ,Figure 17 . 3 :

 4173 Figure 17.3: Operating characteristics against Inception-v3 (solid) andResNet-v2-50 (dashed) on ImageNet[START_REF] Kurakin | Adversarial attacks and defences competition[END_REF].

  video abstraction Our early research focuses on pooling re-1998-2000.

  spatiotemporal saliency By extending the model ofItti et 2005Itti et -2013. .

[ 5 ]

 5 are known to introduce the 2008.

  local feature detection Image gradient and edges are con-2010-2016.

  geometry indexing and feature selection In instance-le-2010-2014.

  clustering and nearest neighbor search Inspired by the 2013-2015.

2 .

 2 CNN descriptors are a deep representation of regions or entire E.g., 512 or 2,048 dimensions.

outlook

  An interesting direction is to investigate a generalization of con-Parametric convolution.

  Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 15.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 15.3 Background . . . . . . . . . . . . . . . . . . . . . . . . . 15.4 Deep label propagation . . . . . . . . . . . . . . . . . . . 15.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 15.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . 16 few-shot learning 16.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 16.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 16.3 Background . . . . . . . . . . . . . . . . . . . . . . . . . 16.4 Dense classification . . . . . . . . . . . . . . . . . . . . . 16.5 Implanting . . . . . . . . . . . . . . . . . . . . . . . . . . 16.6 Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . 16.7 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 16.8 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . .

	contents	xvii
	15 semi-supervised learning
	15.1
	xiv

17 adversarial examples 17.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . 17.2 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . 17.3 Background: Attacks . . . . . . . . . . . . . . . . . . . . 17.4 Guided smoothing . . . . . . . . . . . . . . . . . . . . . 17.5 Smooth adversarial examples . . . . . . . . . . . . . . . 17.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . 17.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . iv beyond 18 reflection 18.1 What else? . . . . . . . . . . . . . . . . . . . . . . . . . . 18.2 Current work . . . . . . . . . . . . . . . . . . . . . . . . 18.3 Some thoughts . . . . . . . . . . . . . . . . . . . . . . . . 18.4 Consolidation . . . . . . . . . . . . . . . . . . . . . . . . 19 outlook 19.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . 19.2 A vision . . . . . . . . . . . . . . . . . . . . . . . . . . . 19.3 Directions . . . . . . . . . . . . . . . . . . . . . . . . . . bibliography [ October 7, 2020 at 12:04classicthesis version 0.4 ]

  whereby a deep model is learned according to supervision provided by label propagation. This model can then make predictions on unseen data without accessing the training set. What if both annotations and raw data are scarce? Few-shot learn-Few-shot learning. ing, i.e. learning from few examples, is challenging because learning a deep representation undoubtedly needs a lot of data. It is often treated by nearest neighbor classifiers and indeed many solutions rely on metric learning. However, modern benchmarks require generalizing way beyond the few given examples to recognize examples of entirely different appearance. This is harder than standard categorylevel tasks in terms of within-class variability. We study activation maps for the first time in the context of few-

	Implicit data augmentation, increased capacity.

  On the other hand, improving the quality of global structure yields improved representations, assuming access to the target image collection. In addition, deep parametric models can learn new representations based on the global structure of the data and generalize to unseen images.

		The recurrent theme is that exploring data and learning the rep-resentation are mutually beneficial.
	Memory mechanisms.	Learning deep parametric models has undergone a number of chal-

[ October 7, 2020 at 12:04classicthesis version 0.4 ] discovery of global structure.

  Chapters 4 and 11 are both on different aspects of spatial matching; in the experiments of Chapter 11 we are using manifold search as introduced in Chapter 10. In these cases, reading the chapter indicated as a dependency helps, but is not necessary.Throughout the text, margin paragraphs are used to provideIwrote the main technical content of this manuscript, that is, chapters 3-7, 9-12 and 14-17, between April and July 2019. I submitted a draft version 0.2 to University of Rennes 1 in August 2019, along with candidate reviewer names. I received a positive response and Patrick Pérez, Horst Bischof and Gabriela Csurka Khedari were named as reviewers in October 2019.

	orientation
	2. references to other chapters;
	3. definitions of notation;
	4. examples; or
	5. information, explanations or discussion.
	In particular, apart from Chapter 9, definitions of notation used in
	more than one chapters are repeated in each chapter to keep it self-
	contained. Margin paragraphs in red highlight our contributions.
	1.5 history
	Also, the graph defined in Section 10.2 is used in Chapters 12, 14 and
	15.
	Margin paragraphs.

1.1. There is only one strong dependency: Chapter 9 provides background, notation and definitions on graph filtering that is required by five other chapters. The remaining dependencies are weak: Chapter 3 presents a method initially introduced for clustering that is used by Chapter 12 for region detection; 1. the topic discussed in the corresponding paragraph(s); [ October 7, 2020 at 12:04classicthesis version 0.4 ] I completed the remaining non-technical chapters 1, 2, 8, 13, 18 and 19, shown in green in

  .1. The best size for RAKM is 550k. AKM is more or less equivalent; their difference is in speed. AGM is slightly better with its vocabulary size 857k being automatically inferred. We

	visual vocabularies
	method			RAKM			AKM	AGM
	Vocabulary	350k 500k 550k 600k 700k 550k 857k
	No distractors 0.471 0.479 0.486 0.485 0.476 0.485 0.492
	20k distractors 0.439 0.440 0.448 0.441 0.437 0.447 0.459
	1M distractors	-	-	0.250	-	-	-	0.280
	Table 3.1: mAP comparisons for vocabularies of different sizes on Oxford Buildings with a varying number of distractors, using m = 100, 40 iterations for AKM/RAKM, and 15 for AGM.

  is a partition of F into 2 kd bins (hyper-

	cubes), where k := L -1 -, obtained by partitioning each dimension into 2 k equal intervals of length 2 -k . B 0 is at the finest (bottom) level; B L-1 is at the coarsest (top), with a single bin. Given bin b, let h(b) := {c ∈ C : f (c) ∈ b} (4.6)	Each partition B is a refinement of B +1 . We define a histogram pyramid of correspondences into bins.
	be the set of correspondences with parameters falling into b.
	matching process We recursively split correspondences into bins
	in a top-down fashion, and then group them bottom-up. To impose a one-to-one mapping constraint, we detect conflicting correspondences at each level, choose one to keep and mark the remaining as erased. Let E be the set of erased correspondences up to level . If b ∈ B is a bin at level , we define its group count as	We write [x]+ := max(x, 0) for the positive part of
		x ∈ R.

  A toy 2d example is used for illustration. Figure4.2 shows three groups of assignments at level 0: {c 1 , c 2 , c 3 }, {c 4 , c 5 } and {c 6 , c 9 }. The first two are joined at level 1. Assignments c 7 , c 8 are conflicting, and c 7 is erased. Assignments c 5 , c 6 are also conflicting, but are only compared at level 2; c 5 is stronger, being in a group of 5. Hence {c 6 , c 9 } is broken up, c 6 is erased and c 8 , c 9 join c 1 , . . . , c 5 in a group of 7 at level 2. For instance, c 1 , . . . , c 5 contribute from all 3 levels, while c 8 , c 9 only from level 2. These contributions are arranged in an n × n affinity

	Figure 4.3 illustrates how the similarity score (4.10) is formed, with
	λ = 1. The upper triangular part of A corresponds to the edges of matrix A. Summing affinities over a row of A and multiplying by the corresponding weight yields the assignment strength-the diagonal Figure 4.2, with edge proportional to affinity. strength being is excluded from the summation (4.7).
	mAP	0.5 0.6	0.1 0.5 1 0.1 2	5 0.5	10	1	0.1 0.5 1 20 0.1 2	2	5 10 0.5 1	3	20 2	3
		0.4	0 0				0 0						WGC+HPM BoW+HPM WGC+FSM
		0.3	0	2	4		6	8	10		12	16 BoW+FSM 14
					average time to filter and re-rank (s)
	Figure 4.4: mAP vs. total (filtering + re-ranking) query time on Barcelona with 2M World Cities distractors for a varying number of re-ranked images (shown with labels near markers, in thousands).
							[ July 11, 2019 at 0:00 -classicthesis version 0.1 ]
				4.3: Referring to Figure 4.2: (Left) Assignment labels, features and strength. Vertices and edges denote features and assignments, re-spectively. Assignments c 5 , c 6 and c 7 , c 8 are conflicting, and c 6 , c 7 are erased. Assignments c 1 , . . . , c 5 join groups at level 0; c 8 , c 9 at level 2. (Right) Affinity matrix equivalent to strengths (4.9). As-signments have been arranged so that groups are contiguous. Groups formed at levels 0, 1, 2 have affinity 1, 1 2 , 1 4 respectively.

example

Table 5 .

 5

1: Existing and new solution ASMK for the match kernel κ, classified as aggregated κ A (5.7), non-aggregated κ N (5.6), or both. φ(x): Scalar or vector representation of descriptor x. σ: Scalar selectivity. ψ(z): Vector representation of aggregated descriptor z. Φ(X c ): Equivalent representation of descriptor set X c per cell.

Table 5 .

 5

	dataset	MA	oxf5k oxf105k paris6k holidays
	HE [202]		51.7	-	-	74.5
	HE [202]		56.1	-	-	77.5
	HE-burstiness [197]		64.5	-	-	78.0
	HE-burstiness [197]		67.4	-	-	79.6
	Fine vocabulary [297]		74.2	67.4	74.9	74.9
	ASMK		76.4	69.2	74.4	80.0
	ASMK		80.4	75.0	77.0	81.0
	ASMK		78.1	-	76.0	81.2
	ASMK		81.7	-	78.2	82.2

2: mAP comparison of ASMK variants against state-of-the-art with α = 3, τ = 0 and k = 65k visual words.

Table 7 .

 7 1 , requerying using the retrieved results and merging for three iterations; and QE 2 , creating a scene map using the retrieved results and requerying once more. We evaluate performance via mAP.

	method	avg. query time mAP
	Baseline BoW	1.03s	0.577
	QE 1	20.3s	0.757
	QE 2	2.51s	0.620
	Scene maps	1.29s	0.807

1: Average query time and mAP of the four benchmarked methods on the European Cities 1M dataset including all distractors.

  [START_REF] Avrithis | Retrieving Landmark and Non-Landmark Images from Community Photo Collections[END_REF]] and others are made accessible through our online application VIRaL 5[START_REF] Kalantidis | VIRaL: Visual Image Retrieval and Localization[END_REF], a content-based image search engine. Given a single query image, it retrieves visually similar images from its database and estimates where the photo is taken on a map. It suggests tags, identifies known landmarks or points of interest, and provides links to relevant Wikipedia articles.VIRaL uses a dataset of 2.7M Flickr 6 images from 44 cities around

	The dataset is crawled from Flickr by requesting geo-tagged photos within a bounding box of city centers.

  The former perform convolution with parameters that are learned in an unsupervised fashion and the latter spatial average pooling and sub-sampling without parameters, introducing invariance to deformations. LeCun et al. study a similar architecture in the 1990s[START_REF] Le Cun | Handwritten Digit Recognition with a Back-Propagation Network[END_REF][START_REF] Lecun | Gradient-Based Learning Applied to Document Recognition[END_REF], rather learning the parameters by SGD and back-propagation on a classification loss function, establishing the term Convolutional Neural Network (CNN) and advocating end-to-end feature learning from raw data. Serre et al.[START_REF] Serre | Object Recognition with Features Inspired By Visual Cortex[END_REF] establish max-pooling as the operator of choice for spatial pooling in 2005.deep learning During the 2000s, unsupervised layer-wise pretraining is used to initialize e.g. 4-layer networks[START_REF] Bengio | Greedy Layer-Wise Training of Deep Networks[END_REF][START_REF] Hinton | A Fast Learning Algorithm for Deep Belief Nets[END_REF][START_REF] Marc | Unsupervised Learning of Invariant Feature Hierarchies with Applications to Object Recognition[END_REF] and massively parallel CNN implementations appear on Graphics Processing Units (GPUs)[START_REF] Kumar Chellapilla | High performance convolutional neural networks for document processing[END_REF][START_REF] Dan C Cireşan | High-performance neural networks for visual object classification[END_REF], but experiments are limited to tiny images. In 2012, Krizhevsky et al.[START_REF] Krizhevsky | Im-ageNet Classification with Deep Convolutional Neural Networks[END_REF] use a two-GPU implementation to

	AlexNet outperforms all previous methods by a large margin on the
	ILSVRC 2012
	competition.
	Convolutional networks.

Its reverse accumulation mode becomes widely known as back-propagation in 1986 by Rumelhart et al.

[START_REF] David E Rumelhart | Learning Representations By Back-Propagating Errors[END_REF]

, who also advocate for optimization strategies that are standard today, including random initialization and batch or online gradient descent with momentum.

Inspired by the findings of Hubel and Wiesel on the visual nervous system in 1959

[START_REF] David | Receptive Fields of Single Neurones in the Cat's Striate Cortex[END_REF]

, Fukushima introduces neocognitron in 1980

[START_REF] Fukushima | Neocognitron: A Self-Organizing Neural Network Model for a Mechanism of Pattern Recognition Unaffected By Shift in Position[END_REF]

, a network consisting of alternating layers of simple and complex cells that learn a feature hierarchy.

  is to

	Spatial pooling on convolutional activations treats feature channels as pattern detectors and can be interpreted as a histogram over such patterns, like BoW.

  Function h α is a low-pass filter:If λ ∈ [-1, 1] is an eigenvalue of W, then 1λ ∈ [0, 2] is an eigenvalue of the Laplacian L, with values near 0 (2) representing low (high) frequencies.

						0.2
						0
	h α (λ)	1 0.4 0.6 0.8				-0.2 0 h α (λ) = 1 -α 0.2 0.4 0.6 0.8 α = 0.7 1 α = 0.9 1 -αλ α = 0.99	1.2 1.4 1.6 1.8	2	2.2
		0.2
		0	0.2	0.4	0.6	0.8	1	1.2	1.4	1.6	1.8	2
							1 -λ
												This interpretation explains smoothing. It is used in Section 10.4.

9) has a unique solution Z := h α (W)Y , and the mapping Y → h α (W)Y is a linear smoothing of Y on G.

[ August 4, 2019 at 19:35classicthesis version 0.1 ] Figure 9.1: 9.3 interpretations spectral filtering Being real symmetric, matrix W is diagonal-

  are the k-nearest neighbors of each other in V , and zero otherwise. The n × n adjacency matrix W = (w ij ) of G is then defined by w

  Table 10.1: mAP of our CG and FSR diffusion against Euclidean search-

			on manifolds
	method	m instre oxf5k oxf105k par6k par106k
			global features: R-MAC [136]
	Euclidean	1	62.6	83.9	80.8	93.8	89.9
	AQE	1	70.5	89.6	88.3	95.3	92.7
	CG	1	80.5	87.1	87.4	96.5	95.4
	FSR	1	80.5	87.5	87.9	96.4	95.3
			regional features: R-Match [362]
	Euclidean 21	71.0	88.1	85.7	94.9	91.3
	AQE	21	77.1	91.0	89.6	95.5	92.5
	CG	5	88.4	95.0	90.0	96.4	95.8
	FSR	5	88.5	95.1	93.0	96.5	95.2

R-MAC

Table 11 .

 11 version of MSER, adjusting parameter ∆ per network/dataset according to the cumulative distribution of activation values over the dataset. We use a multi-scale local at the same relative factors as global, keeping the topranking 512 (2,048) local features on VGG (ResNet) according to activation, discarding activation maps with more than 20 features. We use an Intersection over Union (IoU) threshold 0.2 for NMS. 1: mAP and mP@10 state-of-the-art on benchmark[START_REF] Radenović | Revisiting Oxford and Paris: Large-Scale Image Retrieval Benchmarking[END_REF]. We use VGG (V) and ResNet (R) with MAC and GeM pooling. ↑: Upsampling; : Our re-training. DSM: This work. Top two rows as reported in

	At re-ranking, we perform spatial matching on all 9 combinations of query and database image scales, keeping the best according to inliers.

representation

  .2(c) shows OS examples. Looking at the input image and the FS map alone, it is not evident what is of interest and what is clutter. This is only found by discovering other instances of the same object in the dataset, as represented by the graph. global representation Finally, as discussed in Section 12.3, we detect a set of regions R on a saliency map (FS or OS) in a dataset

	Regions detected on OS are more likely to appear in new images than regions detected on FS.

Table 12 .

 12 1: mAP of our global representation on regions detected by EGM on FS and OS maps against GeM, by cosine or manifold

	High precision means that a saliency map is well aligned to the ground truth bounding boxes.	ity of saliency maps by precision, defined as the sum of saliency over ground truth boxes, normalized by the sum over the entire image. We evaluate retrieval performance by mAP.
		method		medium		hard
			instre Roxford Rparis Roxford Rparis
		GeM [351]	57.0	62.0	69.3	33.7	44.3
		FS.EGM	57.7	63.0	68.7	34.5	43.9
		OS.EGM	61.3	64.2	69.9	35.9	46.1
				diffusion [193]
		GeM [351]	75.0	69.3	83.9	41.1	73.9
		FS.EGM	74.6	71.0	84.1	40.6	72.5
		OS.EGM	77.4	69.0	85.4	41.9	72.3

  Weston et al. introduce in 2008 [481] maybe one of the first deep

	The smoothness assumption: If two examples are close in a high-density region, then so should be the corresponding predictions.

  Berko shows in 1958 [41] that we unconsciously learn linguistic rules by studying how children, aged 4-7, can solve linguistic tasks like forming the plural, past tense or possessive of nonsense words. Carrey studies in 1978[START_REF] Carey | Acquiring a single new word[END_REF] how children, aged 3, acquire a new word associated with a particular color.Landau et al. show in 1988 [244] that children, aged 2-3, learn new objects of particular shape more easily when named with nonsense words than when not.Early computational models attribute this human skill to "prior " and devise different ways of encoding it. In 1997, Yip and Sussman[START_REF] Yip | Sparse Representations for Fast, One-Shot Learning[END_REF] induce constraints on linguistic representations of speech from a corpus of common nouns and verbs. In 2000, Miller et al.[START_REF] Miller | Learning From One Example Through Shared Densities on Transforms[END_REF] generalize over geometric transformations in learning novel

	Early models.
	For instance, italicized digits are modeled by a shearing transformation.

knowledge

  study a generative model of handwritten character composition from strokes and parse novel examples using a library of strokes learned on different alphabets. In 2012, Mensink et al. [289] use metric learning for the Nearest Class Mean (NCM) classifier to learn novel classes from few examples.In the deep learning era, learning novel object categories from one This is the same as the NCM classifier, which is used by Waltner et al. in 2016[START_REF] Waltner | Bacon: Building a Classifier From Only N Samples[END_REF] with a linear embedding on top of CNN features. The problem is essentially supervised metric learning, particularly when the loss function is defined beyond pairs or triplets[START_REF] Meyer | Nearest Neighbour Radial Basis Function Solvers for Deep Neural Networks[END_REF][START_REF] Song | Deep metric learning via lifted structured feature embedding[END_REF][START_REF] Ustinova | Learning deep embeddings with histogram loss[END_REF].However, given the support examples, there is opportunity to con-

	Deep learning.
	Meta-learning.

or few examples becomes popular in 2016 when Vinyals et al.

[START_REF] Vinyals | Matching networks for one shot learning[END_REF]

form mini-batches called episodes from a labeled set of base classes to mimic classification tasks from a limited support set over a distinct set of novel classes. Inference is based on a soft nearest neighbor classifier. Snell et al. improve this model in 2017

[START_REF] Snell | Prototypical networks for few-shot learning[END_REF]

by averaging the embeddings of support examples into class prototypes.

  also introduce a single-step attack that is used[START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF], adversarial training is easily broken[START_REF] Tramèr | The Space of Transferable Adversarial Examples[END_REF]. As a result, training robust models becomes very expensive.The situation becomes more complex when Tsipras et al.[START_REF] Tsipras | Robustness may be at odds with accuracy[END_REF] prove

	Adversarial robustness vs. generalization.
	Adversarial training.
	fast enough to generate adversarial examples on the fly during train-
	ing. Such adversarial training is a defense mechanism that improves

the robustness to adversarial attacks. Kurakin et al. then introduce in 2016 [236] a more powerful iterative attack. Unless such a powerful it-outline erative attack is

45.3 57.8 68.6 78.4 55.0

		35.0 46.8 59.3 72.0 48.1
	Cyclic match [255]	40.8 52.8 65.1 76.0 52.6
	Triplet+semi-hard [389]	42.3 55.0 66.4 77.2 55.4
	Lifted-structure [321]	43.6 56.6 68.6 79.6 56.5
	Triplet+ [157]	45.9 57.7 69.6 79.8 58.1
	Clustering [415]	48.2 61.4 71.8 81.9 59.2
	Triplet+++ [157]	49.8 62.3 74.1 83.3 59.9

MoM (ours)

Table 14 . 1 :

 141 Recall@k and NMI on CUB-200-2011 using GoogLeNet. Comparing our MoM against

  Table 14.2: mAP on particular object retrieval using VGG.

	method	hol instre oxf5k oxf105k par6k par106k
			testing on MAC [438]
	Baseline	79.4	48.5	58.5	50.3	73.0	59.0
	SfM [350]	81.4	48.5	79.7	73.9	82.4	74.6
	MoM (ours) 82.6	55.5	78.7	74.3	83.1	75.6
			testing on R-MAC [438]
	Baseline	87.0	55.6	68.0	61.0	76.6	72.1
	SfM [350]	84.4	47.7	77.8	70.1	84.1	76.8
	MoM (ours) 87.5	57.7	78.2	72.6	85.1	78.0

Table 15 . 1 :

 151 Error rate (mean±std over splits) of our DLP against or combined with MT[START_REF] Tarvainen | Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results[END_REF] for different numbers of labels |L| using C13 on CIFAR-10 and CIFAR-100 and ResNet-18 on miniImageNet. Supervised baseline uses (15.2) only on the labeled examples X L . The remaining methods are semi-supervised on the entire set X.

	dataset	cifar-10	cifar-100	miniimagenet
	# labels	500	1,000	4,000	10,000	4,000	10,000
	Supervised	49.08±0.83 40.03±1.11 55.43±0.11 40.67±0.49 53.07±0.68 38.28±0.38
	DLP	32.40±1.80 22.02±0.88 46.20±0.76 38.43±1.88 47.58±0.94 36.14±2.19
	MT [425]	27.45±2.64 19.04±0.51 45.36±0.49 36.08±0.51 49.35±0.22 32.51±1.31
	MT [425] + DLP 24.02±2.44 16.93±0.70 43.73±0.20 35.92±0.47 50.52±0.39 31.99±0.55

  with this new data, the goal is to learn a classifier that maps a new query example from X to a class label in [c ]. The latter classifier learning we call stage 2.Classification is called c -way; if there is a fixed number k of support examples per novel class, it is called k-shot. FSL is typically evaluated on a large number of new tasks, with queries and support examples randomly sampled from (X , y ).

	Base and novel classes are disjoint. n; Novel classifier and n learning does not exclude continuing the representation learning.

Table 16 . 1 :

 161 , consisting of 64, 16, 20 for training, validation and testing respectively. To generate the support set X of a novel task, we randomly sample c classes from the validation or test set and from each class we sample k images. We report the average accuracy and the corresponding 95% confidence interval over 5k (10k) such tasks for implanting (remaining) experiments, where each task contains 30 queries per class. Average 5-way novel-class accuracy on miniImageNet using except for[START_REF] Gidaris | Dynamic Few-Shot Visual Learning without Forgetting[END_REF]. GAP, DC and wide (last residual block widened by 16 channels): Stage 1. imp (implanting): Stage 2, using GAP on both support and queries. At testing, we use GAP on support and GAP or DC on queries, depending on the choice of stage 1.

	miniImageNet is a subset of ImageNet ILSVRC-12 [376], containing 60k images of resolution 84 × 84 in 100 classes.
	method	1-shot	5-shot 10-shot
	GAP	58.61±0.18 76.40±0.13 80.76±0.11
	DC (ours)	62.53±0.19 78.95±0.13 82.66±0.11
	DC + wide	61.73±0.19 78.25±0.14 82.03±0.12
	DC + imp (ours)	-	79.77±0.19 83.83±0.16
	Gidaris et al. [124] 55.45±0.70 73.00±0.60	-
	PN [327]	56.50±0.40 74.20±0.20 78.60±0.40
	tadam [327]	58.50±0.30 76.70±0.30 80.80±0.30

classes

  7, 2020 at 12:04classicthesis version 0.4 ] := proj X ∩B∞[x; ] a (t) + α sign ∇ x (f (a (t) ), t) .(17.7) In both attacks, projection is element-wise by clipping.

	17.4 guided smoothing	123
	a (t+1) Bp[x; ] is the closed p-norm ball of radius , center x.

  protocol We introduce a new protocol that is more elaborate and more fair in comparing different types of attacks as discussed in Section 17.2. Given a test image set, we only consider its subset X of N images that are classified correctly without any attack. Let X suc be the subset of X where the attack succeeds and dist(x) := ax the distortion for image x ∈ X suc . The operating characteristic function P suc : R + → [0, 1] measures the probability of success as a function of a given upper bound ∆ on distortion: R + . For distortion targeting attacks, we run an attack over the test set with different target distortions . The attack succeeds on image x ∈ X if it succeeds in at least one run. For x ∈ X suc , the distortion dist(x) is the minimum distortion over all runs.results As shown in Figure17.2, our sCW improves a lot the origi-

	This function increases from 0 to |Xsuc| /N as ∆ ranges from 0 to ∞.	P suc (∆) :=	1 N	|{x ∈ X suc : dist(x) ≤ ∆}|	(17.16)
	for ∆ ∈ Resnet-v2 is more robust than Inception-v3: The operating characteristics are shifted to the right.

  few-shot learning. Few-shot learning is often motivated by the ability of humans to learn new tasks from few examples. However, standard benchmarks assume that the representation is learned on a limited amount of base class data, ignoring how much prior knowledge a human may have accumulated before learning new tasks. At the same time, even if a powerful representation is available, it may happen in some domain that base class data is limited.This motivates us to study a new setting: The representation is the few clean support examples, there exists a large amount of novel-class data, annotated with noisy labels. We attack this problem by forming a nearest neighbor graph over both clean and noisy data and training a Graph Convolutional Network (GCN) to predict class relevance of noisy examples. We then learn a classifier for the end task, weighting each noisy example by its relevance[START_REF] Iscen | Graph convolutional networks for learning with few clean and many noisy labels[END_REF]. This is particularly effective in one-shot learning.

	obtained from a classifier pre-trained on a large-scale dataset of a different domain, while the base class data is limited to few examples per class. Their role is to adapt the representation to the domain at	Pre-trained representation on different domain, limited base-class data.
	hand rather than learn from scratch. We call this new setting few-shot few-shot learning [259]. At the other extreme, it may happen in some domain where, apart	Few clean labeled support examples and plenty novel-class data with noisy labels.

from adversarial examples. When white-box attacks are successful, it is typically only the distortion that matters in their evaluation. We argue that speed is important as well, especially when considering that fast attacks are required by adversarial training. Given more time, iterative methods can always find better solutions.

We investigate this speed-distortion trade-off and introduce a new Speed vs. distortion.

  18.1 and 18.2 above, discuss them in the present context, interpret the properties of different representations, make connections between different ideas, highlight certain limitations or unclear situations and hint to potential future work. Some of the discussion is continued in Section 19.3. representation quality = efficiency The introduction of our work on scene maps in 2010 [19] begins by "Images in community photo collections have scaled to billions [. . . ] State of the art visual image retrieval has not yet scaled to permit searching into such huge collections." But, can't we search quickly in a collection of one billion items? Of course we can, if each item has a compact description. PQ-based solutions easily work at this scale since 2011 [205], including our LOPQ

  11. Our smooth adversarial examples [504] are closer to the manifold of natural images than unconstrained adversarial examples. It would be interesting to use them in adversarial training to investigate if this improves generalization. For this to happen, it makes sense to combine with our fast attack BP

[ October 7, 2020 at 12:04classicthesis version 0.4 ] [ October 7, 2020 at 12:04classicthesis version 0.4 ]

[ October 7, 2020 at 12:04classicthesis version 0.4 ]

[ October 7,

at 12:04classicthesis version 0.4 ]

https://avrithis.net [ October 7,

at 12:04classicthesis version 0.4 ]

Feature means representation here. In shallow representations, feature is a geometric primitive, while representation is referred to as (appearance) descriptor. To avoid confusion, we use local feature for the geometric primitive unless obvious from context, and descriptor more often than feature for the representation.[ October 7, 2020 at 12:04classicthesis version 0.4 ]

[ October 7, 2020 at 12:04classicthesis version 0.4 ]1.4 how to read

https://yahooresearch.tumblr.com/post/158115871236/ introducing-similarity-search-at-flick

https://flickr.com/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://viral.image.ntua.gr/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

[ October 7, 2020 at 12:04classicthesis version 0.4 ]3.3 expanding gaussian mixtures

C ← K // updated components

http://www.robots.ox.ac.uk/~vgg/data/oxbuildings/

http://image.ntua.gr/iva/datasets/wc/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://image.ntua.gr/iva/research/agm/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://image.ntua.gr/iva/datasets/wc/ [ October 7,

at 12:04classicthesis version 0.4 ]

http://image.ntua.gr/iva/research/relaxed _ spatial _ matching/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

[ October 7, 2020 at 12:04classicthesis version 0.4 ] 5.4 aggregated selective match kernel

http://image.ntua.gr/iva/research/asmk/ [ October 7,

at 12:04classicthesis version 0.4 ]

http://corpus-texmex.irisa.fr/ [ October 7,

at 12:04classicthesis version 0.4 ]

http://image.ntua.gr/iva/research/lopq/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://viral.image.ntua.gr/

A subsequent pruning step removes from Q(X) any point such that the cluster collection of the remaining points is still a cover for X.

3 http://en.wikipedia.org/wiki/Great-circle _ distance [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://image.ntua.gr/iva/datasets/ec1m/ http://viral.image.ntua.gr/ https://www.flickr.com/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt _ Georeferenzierung/ Wikipedia-World/en http://www.geonames.org/export/wikipedia-webservice.html#wikipediaSearch http://viral.image.ntua.gr/?explore http://viral.image.ntua.gr/?routes [ October 7, 2020 at 12:04classicthesis version 0.4 ]

http://www.image.ntua.gr/iva/research/scene _ maps/ [ October 7, 2020 at

12:04classicthesis version 0.4 ]

[ October 7, 2020 at 12:04classicthesis version 0.4 ] 9.3 interpretations

[ October 7, 2020 at 12:04classicthesis version 0.4 ] 11.3 deep spatial matching

https://github.com/osimeoni/dsm [ October 7,

at 12:04classicthesis version 0.4 ]

[ July 25, 2019 at 15:02classicthesis version 0.1 ] [ July 25, 2019 at 15:02classicthesis version 0.1 ] [ July 25, 2019 at 15:02classicthesis version 0.1 ] [ July 25, 2019 at 15:02classicthesis version 0.1 ] [ July 25, 2019 at 15:02classicthesis v [ July 25, 2019 at 15:02class [ July 25, 2019 at [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis v [ July 25, 2019 at 15:03class [ July 25, 2019 at [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis v [ July 25, 2019 at 15:03class [ July 25, 2019 at [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:03classicthesis version 0.1 ] [ July 25, 2019 at 15:04classicthesis version 0.1 ] [ July 25, 2019 at 15:04classicthesis version 0.1 ] [ July 25, 2019 at 15:04classicthesis v [ July 25, 2019 at 15:04class [ July 25, 2019 at

[ October 7, 2020 at 12:04classicthesis version 0.4 ]14.6 experiments

[ October 7, 2020 at 12:04classicthesis version 0.4 ] 15.5 experiments

[ July 28, 2019 at 14:03classicthesis version 0.1 ] [ July 28, 2019 at 14:03classicthesis version 0.1 ] [ July 28, 2019 at 14:03classicthesis version 0 [ July 28, 2019 at 14:03cla [ July 28, 2019 at 14:03classicthesis version 0.1 ] [ July 28, 2019 at 14:03classicthesis version 0.1 ] [ July 28, 2019 at 14:03classicthesis version 0 [ July 28, 2019 at 14:03cla

[ October 7, 2020 at 12:04classicthesis version 0.4 ] 16.6 inference

f (x) k is the k-th element of vector f (x).

[ October 7, 2020 at 12:04classicthesis version 0.4 ]17.6 experiments

http://image.ntua.gr/iva/research/visual _ saliency

http://image.ntua.gr/iva/research/spatiotemporal _ feature _ detection http://image.ntua.gr/iva/research/edge _ based _ feature _ detection http://image.ntua.gr/iva/research/wash/ http://image.ntua.gr/iva/research/medial _ features/ [ October 7, 2020 at 12:04classicthesis version 0.4 ] 18.1 what else?

http://image.ntua.gr/iva/research/feature _ map _ hashing

http://image.ntua.gr/iva/research/symcity/ [ October 7, 2020 at 12:04classicthesis version 0.4 ]

https://yahooresearch.tumblr.com/post/158115871236/ introducing-similarity-search-at-flick [ October 7, 2020 at 12:04classicthesis version 0.4 ] 18.3 some thoughts

Concurrently, two more works[START_REF] Cieslewski | Matching Features without Descriptors: Implicitly Matched Interest Points[END_REF][START_REF] Dusmanu | D2-Net: A Trainable CNN for Joint Description and Detection of Local Features[END_REF] detect local features independently per channel, but only as point features at global maxima, one per channel.

 11 In fact, there are attempts to make the distribution uniform[START_REF] Sablayrolles | Spreading vectors for similarity search[END_REF][START_REF] Zhang | Learning spread-out local feature descriptors[END_REF].[ October 7,

2020 at 12:04classicthesis version 0.4 ]

Concurrently, two works consider attention mechanisms to focus on the foreground[START_REF] Wertheimer | Few-Shot Learning With Localization in Realistic Settings[END_REF][START_REF] Zhang | Few-Shot Learning via Saliency-Guided Hallucination of Samples[END_REF].[ October 7, 2020 at 12:04classicthesis version 0.4 ]

Compare Jaguar (https://www.top500.org/lists/2009/11/) with Summit (https: //www.top500.org/lists/2019/11/).

[ October 7, 2020 at 12:04classicthesis version 0.4 ] B I B L I O G R A P H Y

H e r Y 9 p 6 Y J V 9 u y B P 2 F 9 / g B 0 G J z m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 1 K N S C x e k s 8 q 5 l M 5 E 0 h d g M w 5

H e r Y 9 p 6 Y J V 9 u y B P 2 F 9 / g B 0 G J z m < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 3 1 K N S C x e k s 8 q 5 l M 5 E 0 h d g M w 5 m q L S D L q s E I X u Z 9 S A 4 A q 6 y F F A v 9 R A Z S a g l 0 2 + N 3 r v H r T h h f q J 0 x J S S e 8 U z z m j 6 K h h 8 C t R 8 M A K K a k a 2 Q b X g z i 1 t h 3 X e 3 W r k w 9 t g m N A W u 8 n k u I 4 y + 1 j / b X 1 l / 7 W J P 5 Y X n o 6 S T n m 7 8 c 7 8 5 T T 6 / + 4 a B i 0 o m q L S D L q s E I X u Z 9 S A 4 A q 6 y F F A v 9 R A Z S a g l 0 2 + N 3 r v H r T h h f q J 0 x J S S e 8 U z z m j 6 K h h 8 C t R 8 M A K K a k a 2 Q b X g z i 1 t h 3 X e 3 W r k w 9 t g m N A W u 8 n k u I 4 y + 1 j / b X 1 l / 7 W J P 5 Y X n o 6 S T n m 7 8 c 7 8 5 T T 6 / + 4 a B i 0 o

e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " D 6 n M D a W O e e i H 7 F 5 s 2 p V r K 7 H S 1 t Y = " >

H e i n f g n X i n / p p / 5 J / 7 F 3 O r 7 y 0 y O + R F + V e / A X 1 A x 5 g = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " e 7 b a 0 X T n z h

Z x k Y 7 0 8 7 I D g 4 7 t I R + d k p e A G o + j Z 8 z 9 8 X F p e W f 3 U + r y 2 v r E Z b G 3 f m q L S D L q s E I X u Z 9 S A 4 A q 6 y F F A v 9 R A Z S a g l 0 2 + N 3 r v H r T h h f q J 0 x J S S e 8 U z z m j 6 K h h 8 C t R 8 M A K K a k a 2 Q b X g z i 1 t h 3 X e 3 W r k w 9 t g m N A W u 8 n k u I 4 y + 1 j / b X 1 l / 7 W J P 5 Y X n o 6 S T n m 7 8 c 7 8 5 T T 6 / + 4 a B i 0 o