François Chloé

Keywords: aeroplane, bicycle, boat, bottle, car, cat, chair, diningtable, dog, horse, person, pottedplant, sheep, train, tvmonitor person, bicycle, car, motorcycle, airplane, bus, train, boat, bird, cat, dog, horse, sheep, cow, bottle, chair, couch, potted plant, dining table, tv truck, tra elephant, bear, zebra, gira e, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli

Je tiens tout

only, yet the transferability of the announced performance is not guaranteed for applications on other kinds of images. We demonstrate this with an in-depth analysis of existing FSOD methods on aerial images and observed a large performance gap compared to natural images. Small objects, more numerous in aerial images, are the cause for the apparent performance gap between natural and aerial images. As a consequence, we improve FSOD performance on small objects with a carefully designed attention mechanism. In addition, we also propose a scale-adaptive box similarity criterion, that improves the training and evaluation of FSOD methods, particularly for small objects. We also contribute to generic FSOD with two distinct approaches based on metric learning and netuning. Impressive results are achieved with the ne-tuning method, which encourages tackling more complex scenarios such as Cross-Domain FSOD. We conduct preliminary experiments in this direction and obtain promising results. Finally, we address the deployment of the detection models inside COSE's systems. Detection must be done in real-time in extremely large images (more than 100 megapixels), with limited computation power. Leveraging existing optimization tools such as TensorRT, we successfully tackle this engineering challenge. Finalement, nous abordons le déploiement de modèles de détection au sein des systèmes de COSE qui doivent détecter les objets en temps réel sur de très grandes images (plus de 100 mégapixels), avec des ressources de calcul limitées. F q Features extracted with a backbone from a query image I q .

Keywords-

Mots

F c s Features extracted with a backbone from a support image I c s , it represents examples from the class c.

A q and A c s Aligned features in AAF framework for a query image and a support image of class c, respectively.

xv γ s and γ q Global attention operators in AAF framework for query and support images, respectively. H q and H c s Highlighted features in AAF framework for a query image and a support image of class c, respectively.

λ s and λ q Spatial alignment operators in AAF framework for query and support images, respectively. M c q Merged query features with support features for class c in AAF framework.

Ω

Fusion operators in AAF framework.

α t and β t Di usion noise variance parameters, α t = 1 -β t .

ς Di usionDet noise clamping parameter for box generation.

o Objectness score for an object. Φ c and Ψ c Prototype vectors for class c in the RPN and the detection of Prototypical Faster R-CNN, respectively. ξ i

Features extracted from Region of Interest i. z

Intermediary features within a network, e.g., embedding vector or latent variable.

C

A box similarity criterion, e.g., IoU, GIoU, or SIoU. γ Scale-Adaptative Intersection over Union strength parameter. κ I If a machine is expected to be infallible, it cannot also be intelligent.

-Alan Turing

As an introduction to this thesis manuscript, we present the industrial context and the motivation behind this project. First, we introduce the company COSE and the Laboratoire de Traitement et Transport de l'Information (L2TI) that collaborated on this CIFRE PhD project. Then, we brie y describe what object detection is and how the industrial constraints that weigh upon COSE in uenced our study toward low-data regimes and few-shot learning. Next, we carry out an overview of the structure of the manuscript, with an individual summary describing each chapter. Finally, we gather the contributions that came out of this project. This includes research articles, accepted or submitted to peer review conferences and journals, as well as open-source code contributions.

Industrial Context, Motivation and Objectives

This PhD thesis originates from a collaboration between the L2TI laboratory from Université Sorbonne Paris Nord (USPN) and the company COSE. The L2TI was founded in 1998 and is a member of the CNRS Research Federation MathSTIC (FR 3734) which includes the Laboratoire Analyse, Géométrie et Applications (LAGA), UMR 7539 and the Laboratoire d'Informatique de Paris Nord (LIPN), UMR 7030. Two main research teams coexist in the L2TI. The Multimedia team focuses on visual information analysis and processing, while the Network team targets information transport and network questions. This thesis falls within the scope of the Multimedia team. COSE 1 is a highly innovative SMB with around 20 employees. It is a rst-tier government provider in the aeronautic and defense sector. COSE was born from an INRIA start-up in the 1990s and has integrated research excellence at the heart of its industrial process. While being relatively small, COSE has multidisciplinary teams with expertise in various elds such as mechanic, electronic, navigation, automation and embedded software. Its size gives COSE remarkable agility and coste ectiveness in comparison to its main competitors. This competitive advantage has allowed COSE to build strong partnerships with major actors in the aeronautic and defense areas. COSE develops, produces and supports aerial observation camera systems and onboard equipment.

These products are mainly designed for military use and must therefore conform to strict quality criteria. The relationship with military forces is handled by the Directorate General of Armaments (DGA), which is one of the main clients of COSE. Among others, COSE currently relies on three products that are in use by French military forces around the world (see Fig. 1.1):

-GlobalScanner: a high-resolution imaging embedded system that provides real-time and georeferenced images. It consists of a high-resolution, stabilized linear sensor that can be integrated into various carriers such as helicopters, aircraft or UAVs. It comes with powerful software to operate the camera and manage image streams.

-Strike: a stabilization arm for helicopters to improve high-precision ri e accuracy. It improves shot accuracy and drastically reduces collateral damage.

-POD Xplorer: a multifunctional pod for various carriers. Its purpose is to embed various types of payloads such as optical sensors, LIDAR, scienti c equipment or inertial sensors.

Recently, COSE started the CAMELEON project to replace the decades-old GlobalScanner system.

Its objective is to improve GlobalScanner in every aspect. First, the linear sensor will be replaced by high-resolution CMOS matrix sensors. With up to six sensors per system, CAMELON will be able to cover extremely large areas with high resolution. The images will also largely overlap, enabling precise 3D reconstruction of the own-over areas, which is especially important for mission prepa- ration and risk analysis. CAMELEON will also come with an improved software stack from mission planning to image analysis and visualization. This PhD project is part of this software redesign.

The amount of image data acquired each second by the new system will be overwhelming for a single photo interpreter as done with GlobalScanner. Furthermore, standard communication streams

will not be su cient to send entire images in real-time. Therefore, relevant information must be extracted from the images, automatically and on edge. To this end, CAMELEON must integrate intelligent algorithms able to nd relevant structures and information inside the mass of pixels acquired each second. These pieces of information are often called Geospatial Intelligence (GEOINT).

They consist of evidence of human activity precisely georeferenced, with any kind of supplementary metadata (e.g., weather conditions or user annotation). Such evidence can be buildings, crop elds, vehicles, or even animals. It is illustrated in Fig. 1.2. In most cases, these are salient objects and can be detected in the images. Once an object has been localized in an image, its precise location can be derived from the carrier position, the direction of the camera and the digital elevation model used, which produces a GEOINT. The GEOINT is then enriched with relevant information about the object: what is the object? Is it dangerous? Is it moving? Even though this seems to require a human appraisal, some of these questions can be answered automatically. The main objective of this PhD project is to develop models that will be able to produce GEOINT automatically. It will need to localize objects and infer relevant metadata about them. It should drastically increase the e ciency of photo-interpreters who will then be able to manage the ever-increasing amount of data generated by aerial intelligence systems, in particular within the CAMELEON project.

I

Object Detection (OD) is a crucial part of creating GEOINTs. In computer vision, object detection is the task of localizing and classifying all objects visible in an image. Of course, the notion of an object needs to be de ned more precisely, otherwise anything in the image can be considered of interest.

A pre-de ned set of semantic classes C is xed so that a clear distinction can be made between objects of interest (i.e., the ones we want to detect, also called foreground objects) and background objects (i.e., those we are not interested in). Based on this distinction, the task of detecting objects can be split into two sub-tasks. 1) Localizing all the objects (foreground and background): this can be done by nding the coordinates of the center of the objects, a rectangular bounding box or even a precise segmentation mask for each object. In general, the object detection task in computer vision is associated with bounding box localization. 2) Classifying the objects localized in step 1).

It consists in rst ltering out background objects and then, assigning a class label c ∈ C to each foreground object. Research interest in the detection task dates back to the early 2000s when the Viola-Jones object detector [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] was rst introduced. Since then, plenty of algorithms have been proposed to improve both the speed and quality of the detection. A breakthrough occurred in 2013 with the rst uses of deep convolutional networks for detection, namely OverFeat [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF] and R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF].

These methods paved the way for more elaborated deep-learning-based detectors. Deep-learning detectors are often referred to as learning-based approaches as they mainly rely on the learning from data paradigm and supervised learning. They contrast with earlier detection methods (also called traditional methods) which often build upon hand-crafted features. A thorough review of both traditional and learning-based object detectors is available in Sec. 2.1. Learning-based approaches have now established complete dominance over traditional methods in terms of detection quality while having reasonable speed performance. Therefore, most of this project will focus on learningbased algorithms.

The choice of deep-learning-based detectors may seem puzzling for COSE on-edge applications.

Computing resources are limited inside the carrier. The payload must be as light as possible, so we cannot a ord to embed heavy Graphical Processing Units (GPU) enabled machines, designed to run deep learning models. In addition, on-board power supplies cannot provide enough energy to run such hardware. However, light-weight, energy-e cient GPUs exist, such as the Nvidia Xavier and Orin Series, which are perfectly suited for deep-learning inference. Nevertheless, another constraint remains, images must be processed in real-time. The CAMELEON system is designed to take about 1 image every second. It is a rather low frame rate but due to the sensor size and the number of cameras, this represents a data stream of several hundreds of megapixels per second. To process such a massive amount of images every second, the detection models must be as light and e cient as possible. Fortunately, tools capable of optimizing the inference of deep learning models exist.

Chap. 9 will present these tools and how they can be leveraged to build detection models fast enough for COSE's applications. This solves the issues related to the deployment and inference of such models; however, a major concern remains: how to train these object detectors?

1.1 I

C , M O

Learning-based methods and especially deep learning models heavily rely on data to be trained. In general, the overall performance of a model highly depends on the amount and quality of annotated data available during the training. For the detection task, collecting large annotated datasets is timeconsuming and expensive. In some cases, it is even impossible. In the medical domain, for instance, privacy-preserving regulations often prevent the use of personal data. For military applications, this is even harder as potential training data are classi ed. This is problematic for the training of datahungry methods such as deep learning. Fortunately, there are some learning strategies much more data-e cient. These methods are usually referred to as few-shot or low-shot learning, and thorough reviews of these methods will be presented in Sec. 2.2. While there are plenty of approaches to Few-Shot Learning (FSL), all follow the same basic principle. First, learn generic knowledge about a related task (source task), second, adapt to the target task. These two training phases are referred to as base training and ne-tuning. In the case of detection, a task can designate a set of classes to be detected, this problem is then called Few-Shot Object Detection (FSOD). A large annotated dataset containing annotations of objects belonging to C base is available. The source task is to detect these objects. Then, the target task is to detect objects from the so-called novel classes, only provided with a limited number of annotations. Chap. 3 provides an in-depth review of existing work in this area. Generally, the target task is performed on similar images as the ones seen during base training. However, the target task can also be done with di erent kinds of images, e.g., the source task can be learned from natural images while the target task on aerial or medical images. This is called Cross-Domain Adaptation. It complexi es signi cantly the problem, but it is a much more realistic scenario in the industry. Collected datasets can only approximate the real data distributions.

Discrepancies between the acquisition settings (i.e., camera, lights etc.) and the application settings almost always produce a performance drop. In medical imagery, this is a typical issue as di erent scanners will not produce exactly similar images. This prevents training models on scan collection from one hospital and deploying them in another. The military use case is another critical example.

The con dentiality of the images, and the ever-changing environment and objects of interest make it di cult to build robust detection algorithms.

Given the constraints of COSE, the main objective of this project is to develop data-e cient object detection methods based on few-shot learning. We orient our research on the Few-Shot Object Detection problem, i.e., the adaptation to novel classes. While a detailed overview of the thesis will be presented in the next section, we outline here the main parts of this work. provides details about the deployment of object detection models according to the needs of COSE. I

Overview of the thesis

This section outlines the content of each chapter of this thesis. This takes the form of a small abstract per chapter. These abstracts will be repeated for convenience at the beginning of the corresponding chapters.

Part I: Literature Review on Object Detection, Few-Shot Learning and Few-Shot Object Detection

The rst part of this thesis is composed of three chapters. The two rst present the literature about Object Detection, Few-Shot Learning and Few-Shot Object Detection. Then, the third chapter explores the challenges of applying Few-Shot Object Detection on aerial images and presents our rst contribution: an analysis of these di culties.

Chap. 2: Object Detection, Few-Shot Learning and Cross-Domain Adaptation Object Detection and Few-Shot Learning are two relevant sub elds from the Computer Vision and Machine Learning elds. Both are necessary to build detection techniques able to generalize from limited data. Hence, this chapter reviews both Object Detection and Few-Shot Learning. Both problems are de ned, and detailed reviews of the respective literature are conducted.

Chap. 3: Few-Shot Object Detection

This chapter presents the task of detection in the few-shot regime and reviews the existing literature about it. Few-Shot Object Detection (FSOD) is at the crossroads of Object Detection and Few-Shot Learning, and therefore, extensively relies on these two elds explored in Chap. 2. Just as for classication, various directions are explored in the literature to tackle the detection task in the few-shot regime which will be presented in detail. Finally, this chapter focuses on the aerial image application of FSOD methods and extensions of the few-shot setting.

Chap. 4: Understanding the Challenges of Few-Shot Object Detection

The detection task becomes extremely challenging when limited annotated data is available. In this chapter, we explore the reasons behind this di culty. In particular, we focus on the case of aerial images for which it is even harder to apply FSOD techniques. It turns out that small objects are especially challenging for the FSOD task and are the main source of error in remote sensing images.

Chapter's contributions: Part II: Improving Few-Shot Object Detection through Various Approaches

P.
The second part of this thesis presents our main contributions to the Few-Shot Object Detection (FSOD) eld. Each chapter proposes a novel approach to addressing the FSOD problem and discusses its pros and cons compared to existing methods. These contributions led to several accepted articles in international and national conferences and journals.

Chap. 5: Experience Feedback about Metric Learning for FSOD Prototypical Faster R-CNN (PFRCNN) is a novel approach for FSOD based on metric learning. It embeds prototypical networks inside the Faster R-CNN detection framework, speci cally in place of the classi cation layers in the RPN and the detection head. PFRCNN is applied to synthetic images generated from the MNIST dataset and to real aerial images with DOTA dataset. The detection performance of PFRCNN is slightly disappointing but sets a rst baseline on DOTA. However, the experiments conducted with PFRCNN provide relevant information about the design choices for FSOD approaches.

Chapter's contributions: fair comparison between various mechanisms can be made. Using the framework, we also propose a novel attention mechanism speci cally designed for small objects.

Chapter's contributions:

Ƽ P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal. P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepted at the IEEE International Conference on Image Processing 2023 (ICIP).

Chap. 7: Few-Shot Di usion Detector via Fine-Tuning Previous chapters explore few-shot object detection with metric learning and attention-based techniques. This chapter logically focuses on the last major approach for FSOD: ne-tuning. Based on Di usionDet, a recent detection model leveraging di usion models, we build a simple but e cient I ne-tuning strategy. The resulting method, called FSDi usionDet, achieves state-of-the-art FSOD on aerial datasets and competitive performance on natural images. Extensive experimental studies explore the design choices of the ne-tuning strategy to better understand the key components required to achieve such quality. Finally, these impressive results allow considering more complex settings such as cross-domain scenarios, which are especially relevant for COSE.

Chapter's contributions: This chapter describes very recent work, and we plan to submit research articles to present these results.

Part III: Rethinking Intersection Over Union

This part contains only one chapter which presents a contribution orthogonal to the approaches proposed in Part II as it questions the relevance of the Intersection over Union, a key component of object detection pipelines.

Chap. 8: Scale-Adaptative Intersection Over Union Intersection over Union (IoU) is not an optimal box similarity measure for evaluating and training object detectors. For evaluation, it is too strict with small objects and does not align well with human perception. For training, it provides a poor balance between small and large objects to the detriment of small ones. We propose Scale-adaptative Intersection over Union (SIoU), a parametric alternative that solves the shortcomings of IoU. We provide empirical and theoretical arguments for the superiority of SIoU through in-depth analysis of various criteria.

Chapter's contributions:

Ƽ P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV). P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", Accepted at GRETSI 2023.

Part IV: Prototyping and Industrial Application

Finally, the last part of this thesis presents our industrial contributions. This part is crucial for COSE as it bridges the gap between research advancements and real-world applications. Therefore, the only chapter of this part discusses the engineering aspects of object detection and is not associated with any academic contribution.

Chap. 9: Integration in COSE Prototypes Detection models are often heavy and are not well suited for COSE's application. In this chapter, we rst present in detail the CAMELEON system and its constraints. Then, we study the in uence of the model size on the performance and present useful tools and tricks to accelerate the inference.

Finally, we explain how the detection models are deployed inside the CAMELEON prototype and how they perform on aerial images.

Oral Presentations

During the PhD, I had the opportunity to give talks in various occasions listed below: -L2TI's scienti c day (Dec. 2020).

-Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images, DeepLearn Summer School 2021, Las Palmas de Gran Canaria (Jul. [START_REF] Krizhevsky | Imagenet classi cation with deep convolutional neural networks[END_REF]2021).

-Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images at a GDR-ISIS meeting: Vers un apprentissage pragmatique dans un contexte de données visuelles labellisées limitées, Paris, (Nov. [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF]2021).

-L2TI's Doctoral seminar (Mar. 2022 andFeb. 2023).

-Few-Shot Object Detection on Aerial Images, Seminar at ETS Montreal (Sep. [START_REF] Girshick | Object detection with grammar models[END_REF]2022).

I

Internships Supervision I supervised four internships over the three years of this PhD, three inside the company and one at within the L2TI:

-Conception et mise en oeuvre d'algorithmes de suivi d'objets dans des images aériennes (March-August 2021 -COSE).

-Optimisation et intégration d'algorithmes de détection d'objets dans un système embarqué (March-August 2022 -COSE).

-Self-supervised learning for Few-shot Object Detection (April-August 2022 -L2TI au travers du LabCom IRISER).

-Détection d'objets few-shot par visual transformers sur des images Aériennes (March-August 2023 -COSE and L2TI through the LabCom IRISER).

In addition to the supervision of two internships, I am actively involved inside the LabCom IRISER 2 which is a joint laboratory between COSE, the L2TI and the LIPN. It was created one year after the beginning of my PhD at the instigation of my academic and industrial supervisors.

Open-source Software

In the course of the various project I conducted during this PhD, I wrote multiple open-source Python packages that can be found on GitHub: Prototypical Faster R-CNN AAF framework Pycocosiou FSDi usionDet 2 Link to the LabCom IRISIER's website

I (F )

Si une machine doit être infaillible, alors elle ne peut pas aussi être intelligente.

-Alan Turing

Pour introduire ce manuscrit de thèse, le contexte industriel et les motivations de ce projet sont présentés. D'abord, sont introduits l'entreprise COSE et le Laboratoire de Traitement et Transport de l'Information (L2TI) qui ont collaboré sur cette thèse CIFRE. Ensuite, nous décrivons ce qu'est la détection d'objets dans le cadre de la vision par ordinateur et comment les contraintes industrielles liées à COSE ont orienté la thèse vers l'apprentissage frugal (dit few-shot). Dans un second temps, la structure de ce manuscrit est exposée en présentant un résumé individuel pour chaque chapitre.

En n, une dernière partie liste les di érentes contributions apportées au cours de ce projet, cela inclut des articles de recherche publiés ou soumis dans des conférences nationales et internationales. -GlobalScanner : un système de caméra embarquée qui produit des images haute résolution et géoréférencées en temps réel. Le système est constitué d'un capteur linéaire de très grande résolution. Ce capteur est stabilisé et intégré au sein d'une enceinte mécanique qui peut être intégrée sous di érents types d'aéronefs (hélicoptère, avion, drone, etc.). Le capteur est connecté à un poste de contrôle et une suite logicielle permettant de piloter la caméra et de gérer les ux d'images.

Contexte industriel, motivation et objectifs

-Strike : un bras de stabilisation d'arme à feu pour hélicoptère. Il améliore sensiblement la précision des tireurs et réduit les risques de dommages collatéraux.

-POD Xplorer : un pod multifonction pouvant être attaché en dessous de di érents types de porteurs. Il permet d'embarquer simplement des charges utiles variées comme des capteurs optiques, des LIDARs, ou des équipements scienti ques.

Récemment, COSE a lancé le projet CAMELEON a n de remplacer GlobalScanner. Son objectif premier est de surpasser GlobalScanner dans tous les aspects. Premièrement, le capteur linéaire sera remplacé par un capteur matriciel CMOS de haute résolution. CAMELEON pourra embarquer jusqu'à six capteurs avec des orientations di érentes a n de couvrir des très grandes zones au sol tout en conservant une grande dé nition. Les images ainsi acquises auront beaucoup de recouvrement a n de permettre la reconstruction 3D des zones survolées. C'est un aspect extrêmement important de la préparation de mission et la gestion des risques pour les forces armées. CAMELEON Jones [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF] a été introduit pour la première fois. Depuis, de nombreux algorithmes ont été proposés pour améliorer à la fois la vitesse et la qualité de la détection. Une avancée remarquable s'est produite en 2013 avec les premières utilisations de réseaux de neurones convolutifs pour la détection, notamment avec OverFeat [START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF] et R-CNN [START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF]. Ces méthodes ont ouvert la voie à des détecteurs de plus en plus élaborés basés sur l'apprentissage profond. Ces détecteurs reposent principalement sur le paradigme de l'apprentissage machine et notamment l'apprentissage supervisé. Ils di èrent des méthodes de détection antérieures (appelées méthodes traditionnelles) qui s'appuient souvent sur des caractéristiques manuelles. Une revue détaillée des détecteurs d'objets traditionnels et basés sur l'apprentissage est disponible dans la section 2.1. Les approches basées sur l'apprentissage ont désormais établi une domination complète sur les méthodes traditionnelles en termes de qualité de détection, tout en o rant des temps d'exécution plus rapides. Par conséquent, la majeure partie de ce projet se concentrera sur les algorithmes basés sur l'apprentissage.

Le choix de détecteurs basés sur l'apprentissage profond peut sembler compliqué pour les applications embarquées telles que celles de COSE. Les ressources de calcul sont limitées une fois en vol. La charge utile doit être aussi légère que possible, COSE ne peut donc pas nous permettre

Présentations orales

Au cours de cette thèse, j'ai eu l'opportunité de donner plusieurs présentations orales :

-Journée scienti que du L2TI (Déc. 2020).

-Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images DeepLearn Summer School 2021, Las Palmas de Gran Canaria (29 Juil. 2021).

-Prototypical Faster R-CNN for Few-Shot Object Detection on Aerial Images à la journée du GDR-ISIS : Vers un apprentissage pragmatique dans un contexte de données visuelles labellisées limitées, Paris, (26 Nov. 2021).

-Séminaires des doctorants (Mar. 2022 and Fév. 2023).

-Séminaire à l'ETS Montreal: Few-Shot Object Detection on Aerial Images (28 Sep. 2022). En plus des supervisions de deux stages, je suis activement impliqué dans le LabCom IRISER 2 , un laboratoire commun entre COSE, le L2TI et le LIPN. Ce laboratoire commun a été crée un an environ après le début de ma thèse sous l'impulsion de mes superviseurs académique et industriel.

Logiciels libres

Au travers des di érents projets qui ont constitués cette thèse, j'ai développé plusieurs package Python qui se trouvent en accès libre sur GitHub : As brie y presented in the introduction, this PhD project lies at the intersection of three sub-domains of Machine Learning: Object Detection (OD), Few-Shot Learning (FSL), and Cross-Domain Adaptation (CDA). In this chapter, we de ne more precisely what these three elds are and outline the main existing contributions in the literature. We start by introducing the main computer vision problems these elds address and the related notations adopted in this manuscript. Then, we provide a review of existing works in each area, and nally, we link them with the industrial needs of COSE. class could be ambiguous for multiple reasons. Given the quality of the image, it can be di cult to determine the class of the object depicted. For instance, in a satellite photograph, a car could be so small that it cannot be perceived by a human observer. Another issue is the slackness of our concept of objects, one word can refer to multiple objects (e.g., spring, game, or chest), and our vocabulary is organized hierarchically (e.g., the class vehicle contains many classes including car and truck). One could go even further by questioning the very concept of objects in our mind (see for instance exemplar-based vs. prototype-based concept theories), but it would have more to do with cognitive science than computer vision. Generally, for object detection, these complications are not considered. One object can only belong to one class and whether it belongs to the class or not falls under the common sense of the observer. Most of the time, this is established with the ground truth annotations of human experts. This explanation is rather obvious, but keep in mind that this is a simpli cation, this will be useful when the notion of an object gets blurrier in the case of few-shot learning and few-shot object detection.

The detection task consists in nding all occurrences of the objects of interest in the images, i.e., the image coordinates of each object. This can be done in several ways, by locating a single pixel inside each object, by drawing a rectangular bounding box, or by computing a precise mask around it.

The former setting is barely used as it can be quite ambiguous, all points in an object are equivalent and no supplementary information (size and shape) can be inferred from this representation.

Traditionally, object detection leverages bounding boxes to localize the objects, and precise masks are reserved for the Instance Segmentation task. A bounding box is generally determined by four coordinates, it can be the coordinates of two diagonally opposed points on the box or the coordinates of one point plus the width and height of the box, see Fig. Many more representations exist but they will not be presented here. In this manuscript, the second representation, top-left point width and height, will be used exclusively.

An image may contain more than one object and therefore, each image I is associated with a set of detection labels Y = {y i } N I i=1 , where N I is the number of objects in image I. Hence, solving an object detection task is to nd a detector model F(•, θ) with parameters θ able to output a set of predicted labels given an input image I:

F(I, θ) = Ŷ = {ŷ i } M I i=1 . (2.3)
We employ here a slight abuse of notation by calling the output of a detector y. Indeed, a detector predicts a classi cation score l c for each class c ∈ C. Therefore, ŷ = ( b, { lc } c∈C ) and ĉ = arg max c∈C l c . For convenience, we denote l ∈ [0, 1] |C| the vector of classi cation scores.

Hatted symbols represent the model's outputs. Note that the number of detections found by the model M I may not equal the number of objects present in the image as the detector can either miss some objects or output false detections. The proximity between the predicted labels Ŷ and the ground truth labels Y determines the performance of the model F(•, θ). Hence, nding an optimal detection model is to nd a set of optimal parameters, which minimizes the distance between predicted and ground truth labels:

θ * = arg min θ d( Ŷθ , Y), (2.4)
where d is a distance measure between Ŷθ , the labels predicted by F(•, θ) and the ground truth labels Y. Of course, there are plenty of valid approaches to measure the proximity between two sets of detection labels, some will be introduced in Sec. 2.1.2.

For COSE, detecting the objects of interest in an image is a crucial step. This step is necessary for the creation of GEOINTs. From the bounding boxes coordinates in the image, and the carrier position (latitude, longitude, and altitude) and attitude (pitch, roll, and yawn), one can determine the precise locations of the objects on Earth. These computations also involve camera properties and orientations, but they will not be addressed in this manuscript.

Evaluation of Object Detectors

Before jumping into the Object Detection literature, let's introduce the most commonly used metrics employed to assess the quality of the detection models. As mentioned in the previous section assessing the detection performance of a model consists in comparing the set of predicted detection labels Ŷ with the set of ground truth labels Y (typically made by a human observer). In the previous section, we de ned the set of detection labels over an image. However, to better assess the generalization capabilities of the detectors, their evaluation is always conducted on a relatively large test set of images. Therefore, we extend the notation Ŷ and Y as the sets of predicted and ground truth labels (respectively) over all test images.

Average Precision and mean Average Precision

The most commonly used metrics for Object Detection are the Average Precision (AP) and its extension in the multiclass setting, the mean Average Precision (mAP). The AP is formally de ned as the area under the precision-recall curve:

AP = 1 0 Prec(r) dr, (2.5)
where Prec denotes the Precision and r, the Recall. The Precision and Recall are two well-known metrics often used to evaluate classi cation problems. They are de ned respectively as the ratio of true positive labels over the positive predicted labels and the ratio of the positive labels over the positive true labels. Figure 2.2a clearly illustrates these de nitions with a classi cation confusion matrix. Note that the notion of True Positive (TP), True Negative (TN), False Positive (FP) and False Negative (FN) introduced in the gure will be leveraged throughout the manuscript.

For now, forget about the classi cation part of the detection and consider only the localization problem. How can we de ne the four corners of the confusion matrix for bounding boxes? An approach is to consider a detection as TP if the predicted bounding box has the same coordinates as one true detection label. However, it is extremely challenging to get a perfect positioning of Then, TPs are the boxes that have an IoU with a true box above a xed threshold (typically 0.5).

However, the IoU may not be an optimal criterion in certain cases as we will discuss in Chap. 8.

The Intersection over Union, also known as the Jaccard index, is a well-known similarity measure between sets A and B:

IoU(A, B) = |A ∩ B| |A ∪ B| . (2.6)
Besides its application in statistics, the IoU is widely used in computer vision to assess the quality of visual tasks such as detection and segmentation. The IoU can compute how close two sets of pixels are and thus gives a similarity measure between ground truth and the model prediction. Here, we focus on the detection task, therefore the IoU can be written in terms of coordinates of the boxes b

1 = [x 1 , y 1 , w 1 , h 1 ] T and b 2 = [x 2 , y 2 , w 2 , h 2 ] T : A inter = max 0, max(x 1 , x 2 ) -min(x 1 + w 1 , x 2 + w 2 ) * max 0, max(y 1 , y 2 ) -min(y 1 + h 1 , y 2 + h 2 ) , (2.7)
IoU(b 1 , b 2 ) = A inter w 1 h 1 + w 2 h 2 -A inter . (2.8)
The IoU is, therefore, a crucial part of the evaluation protocol of the object detectors as it conditions which predicted bounding boxes are TP, and which are FP. The IoU threshold limit determines how close to the ground truth the predicted boxes must be to be considered TP. In the Pascal VOC detection challenge [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF], this threshold was set to 0.5, this has been the gold standard for a few years.

However, it changed when the more challenging Microsoft COCO dataset [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] was proposed. The authors of the COCO challenge compute the AP at several thresholds (ranging from 0.5 to 0.95)

and average the values. While this is the current standard for object detection evaluation, the fewshot object detection literature still uses the former Pascal VOC AP as it is an easier metric. Hence, we will use both of these metrics in the manuscript. We will denote them as AP 0.5 and AP 0.5:0.95 respectively.

Before computing the precision-recall curve, the predicted labels must be ranked by con dence 3. Therefore, it may not be easy to compute the area under the curve, i.e., the AP. A few tricks were introduced in [START_REF] Salton | Introduction to modern information retrieval[END_REF], and later popularized with the Pascal VOC challenge [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF]. They consist in taking an interpolated precision-recall curve:

Prec interpolated (r) = max r≥r Prec(r), (2.9)
and discretize the area computation over 11 equally spaced points along the recall axis. Hence, the original AP de nition becomes:

AP = 1 11 10 i=0
Prec interpolated (0.1 × i).

(2.10)

So far, we only discuss the evaluation without taking into account the class of the bounding boxes.

In order to take this into account, the AP is computed independently for each class and noted AP c .

The predicted boxes are now considered true positive only if they have a su cient IoU with a ground truth box and if they have the same class. The mean Average Precision is de ned as follows:

mAP = 1 |C| c∈C AP c . (2.11)
The mAP is largely the most employed metric for object detection and most of our analysis will be based on it. However, there exist complementary metrics that are worth presenting here.

mean Average Precision per object size

The AP and mAP can be computed only on certain object sizes. The principle is simple, simply lter the sets of predicted and true labels to keep boxes of a certain size before the AP computation. This distinction was introduced in the COCO challenge [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], with three di erent object sizes:

-Small: boxes with an area a = wh smaller than a ≤ 32 2 pixels.

-Medium: boxes with 32 2 < a ≤ 96 2 .

-Large: boxes with a > 96 2 .

This distinction is extremely relevant for COSE applications as the objects of interest in aerial images are often small and because most detectors struggle to detect them. Even worse, this issue is reinforced in the few-shot regime as we will see in the following chapters. Therefore, mAP S , mAP M and mAP L will be extensively employed in our analysis. where υ denotes the IoU threshold (ranging from 0.5 to 1) for the recall computation. Similarly to the AP for di erent object sizes, the average recall can be declined according to the maximum number of detection allowed. Basically, this controls the size of Ŷ, the more detection the model can output, the less likely it is to miss an object of interest. Although it can be a critical metric in some applications (e.g., lesion detection), this is not essential for COSE's applications.

Average Recall

Average Precision shortcomings and Alternative

Even though AP is widely used in the computer vision community, it has several shortcomings.

Similar values of AP can be obtained from very di erent precision-recall curves, hiding di erent detectors' behavior. The ranking of the con dence scores makes the AP sensitive to the prediction con dence. Finally, the interpolation trick from [START_REF] Salton | Introduction to modern information retrieval[END_REF] may cause large errors when the number of instances of the class is small. These drawbacks were highlighted in [START_REF] Oksuz | Localization recall precision (LRP): A new performance metric for object detection[END_REF], which proposes an alternative metric, the Localization-Recall-Precision (LRP). This metric is an aggregation of three metrics based on the box regression error, the precision, and the recall, under a certain con dence threshold.

Hence, LRP xes some of the AP's shortcomings.

More recently, [START_REF] Jena | Beyond mAP: Towards better evaluation of instance segmentation[END_REF] also outlined two detection issues that are not spotted by AP, namely spatial hedging and category hedging. Spatial hedging comes from the fact that low-con dence duplicates (slightly perturbated spatially) of a box do not degrade the AP value, instead having a lot of these duplicates generally improves the AP. However, these duplicates are mostly burdensome from an application viewpoint. The authors even highlight some tricks in recent object detectors that boost AP while increasing the number of duplicates. Category hedging comes from duplicated boxes with di erent classes. Consequently, the authors proposed two novel metrics to speci cally assess spatial and category hedging: Duplicate Confusion (DC) and Naming Error (NE). Note that LRP partly assesses spatial hedging as well.

C 2 O D , F S L C D A

Literature review about Object Detection

In this section, we review the Object Detection literature but only present the most ground-breaking works. For an exhaustive review of object detection, we defer the reader to two popular surveys [START_REF] Sahil | A survey of modern deep learning based object detection models[END_REF][START_REF] Zou | Object detection in 20 years: A survey[END_REF]. This section is divided into three parts, traditional object detection, CNN-based OD and Transformer based OD. These correspond to three phases in the development of object detection techniques. This is highlighted by the timeline in Fig. 2.4, which summarizes the history of the object detection eld.

Traditional Object Detection Approaches

The very beginnings of the object detection eld date back to the early 1990s. It began with an easier one-class problem: face detection. Of course, there were prior works addressing this task, but they mainly focused on face recognition and not detection. The di erence is slight, the recognition task only asks whether there is a face or not in an image. This eld gained substantial interest over the 1970s and 1980s with seminal works such as [START_REF] Jay Goldstein | Identi cation of human faces[END_REF][START_REF] Jay Goldstein | Man-Machine Interaction in Human-Face Identi cation[END_REF][START_REF] Kaya | A basic study on human face recognition[END_REF][START_REF] Loddon Yuille | Feature extraction from faces using deformable templates[END_REF]. However, it was only in 1991 that the detection task was rst tackled by EigenFaces [START_REF] Turk | Eigenfaces for recognition[END_REF]. In this work, the authors perform a Principal Component Analysis (PCA) on a set of face images. The PCA returns a set of eigenvectors (denoted as EigenFaces) that span the face space. Applying the EigenFaces on a sliding window over the images allows creating faceness maps and therefore localizing faces. Following this pioneer contribution, many face detectors were proposed in the 1990s, for instance [START_REF] Bichsel | Human face recognition and the face image Set s topology[END_REF][START_REF] Sung | Example-based learning for view-based human face detection[END_REF][START_REF] Vaillant | Original approach for the localisation of objects in images[END_REF][START_REF] Henry A Rowley | Neural network-based face detection[END_REF]. We called this section "Traditional Object Detection Approaches" in contrast with the two following sections that are deep learning-based approaches. However, note that a signi cant proportion of the methods developed during the 1990s actually leverage neural networks. In EigenFaces [START_REF] Turk | Eigenfaces for recognition[END_REF], for instance, the authors discuss an implementation of their system using Multi-Layer Perceptron (MLP)

for fast parallel computation. Similarly, [START_REF] Vaillant | Original approach for the localisation of objects in images[END_REF][START_REF] Henry A Rowley | Neural network-based face detection[END_REF] exploit and train neural networks for the task of face detection. We will see in the next section that plenty of these ideas will be re-used 20 years later 2.1 O D by current deep-learning-based detectors. The missing pieces in these early days of object detection were large annotated datasets and dedicated hardware such as Graphical Processing Units (GPUs).

In the late 1990s, the object detection task as we know it today with multiple classes of interest was still far o , but some research groups began to apply it to objects other than faces. As an example, [START_REF] Moghaddam | Probabilistic visual learning for object detection[END_REF] introduced a general probabilistic model for the object of interest used for the visual search of faces and hands. Later, [START_REF] Constantine P Papageorgiou | A general framework for object detection[END_REF] stepped aside from the PCA-like object representations and proposed generic learnable features based on Haar wavelets transform. This was successfully applied to pedestrian detection. An extension with slightly more elaborated features proposed the Viola-Jones detector [START_REF] Viola | Rapid object detection using a boosted cascade of simple features[END_REF]. But it mainly provided several tricks for fast computation, achieving robust real-time detection.

In the 2000s, plenty of works shared the same strategy as the Viola-Jones detector: learning a set of elaborated features and classifying regions in the input images by comparison with the set of features. Improvements were made using more and more complex feature sets [START_REF] Ronfard | Learning to parse pictures of people[END_REF][START_REF] Mikolajczyk | Human detection based on a probabilistic assembly of robust part detectors[END_REF] and the popularization of Support Vector Machines (SVM) classi ers [START_REF] Papageorgiou | A trainable system for object detection[END_REF]. This strategy led to the wellknown Histogram of Orient Gradient (HOG) [START_REF] Dalal | Histograms of oriented gradients for human detection[END_REF], which was rst applied to pedestrian detection.

However, this method was one of the rst to tackle multi-class object detection in the rst Pascal VOC challenge [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] in 2005.

The Pascal VOC challenge quickly became a reference in the Object Detection eld, with increasing di culty over the years (more classes and more images). The winners of the following editions 2006, 2007 and 2008 all took inspiration from the HOG features. In particular, [START_REF] Felzenszwalb | A discriminatively trained, multiscale, deformable part model[END_REF] employs several tricks to improve the detection quality based on HOG features. Among those, Deformable Part Models (DPM), i.e., representing each object as a set of its parts provide signi cant improvements.

It also leverages pyramid features and hard examples mining which are common components of recent object detectors. DPM were then re ned with for instance the Grammar Models [START_REF] Girshick | Object detection with grammar models[END_REF] and

Star Models [START_REF] Girshick | Object detection with grammar models[END_REF].

Object Detection in the Deep Learning Era

While there were a few attempts to solve object detection with neural networks during the 1990s, all were limited to single-class problems and lagged behind state-of-the-art in terms of detection quality and speed. However, this changed with the popularization of fast parallel processing units (GPUs)

and the creation of large image datasets. In 2012, AlexNet [START_REF] Krizhevsky | Imagenet classi cation with deep convolutional neural networks[END_REF] was introduced for image classication with deep convolutional networks (CNNs). Since AlexNet, deep learning was successfully applied to most tasks in computer vision including Object Detection.

From the beginning of the deep object detection era, two schools of thought emerged: one-stage The main principle of YOLO is to split the input image into an S × S grid and predict bounding boxes, con dence scores and class probabilities for each cell in the grid. Each cell is "in charge" of detecting objects that are located within its boundaries. To deal with cases where more than one object is visible in one cell, B bounding boxes and con dence scores are predicted per cell (B = 2 in the original paper). To keep model size constrained, the class probabilities are predicted only once per cell. This assumption limits the model to predict boxes of one class only per cell. The YOLO architecture is based on a deep CNN followed by two fully connected layers. The grid separation is directly implemented inside the architecture since the input size is xed. YOLO is trained in an end-to-end fashion with a typical detection loss function. This loss function includes a regression part for box coordinates and a classi cation part, both implemented as L2 loss functions. Just like other object detectors, YOLO has an example selection strategy to compute its loss. Each ground truth box is attributed to the cell where its center is located and then to the box with the highest IoU. Thus, YOLO is extremely fast compared to the two-stage approaches (50 to 100 fps depending on the con guration for YOLO compared to less than 15 fps for Faster R-CNN). However, this speed improvement comes at the cost of slightly lower detection quality.

Just like the R-CNN family, YOLO was extended several times by its original authors and even later perpetuated by other research groups. In YOLOv2 [START_REF] Redmon | YOLO9000: better, faster, stronger[END_REF], several improvements are introduced, including a lighter and fully convolutional architecture, decoupled class probabilities for each box, anchors boxes as in Faster R-CNN and a hierarchical word structure for re ned classi cation. It also proposes several tricks and loss function adjustments to stabilize training. YOLOv2 hence achieves both higher detection performance and speed. YOLOv3 [START_REF] Redmon | Yolov3: An incremental improvement[END_REF] is then introduced in an unpublished paper by the same authors, presenting several incremental improvements. Again, it outperforms its previous version both in quality and speed. After YOLOv3, its authors decided to quit the Object Detection research for ethical reasons, but many other groups continued to re ne the YOLO framework. The race for the best performance and speed includes numerous versions of YOLO: YOLOv4

[37], PP-YOLO [START_REF] Long | PP-YOLO: An e ective and e cient implementation of object detector[END_REF], YOLOX [START_REF] Ge | YOLOX: Exceeding YOLO Series in 2021[END_REF], YOLOv6 [START_REF] Li | YOLOv6: A single-stage object detection framework for industrial applications[END_REF], YOLOR [START_REF] Wang | You only learn one representation: Uni ed network for multiple tasks[END_REF] and YOLOv7 [START_REF] Wang | YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors[END_REF]. Each of these works has its share of marginal changes involving elaborated loss design, structure change, augmentation

C 2 O D , F S L C D A
techniques and re ned anchors generation. Note that YOLOv51 and YOLOv82 also exist but only as popular code repositories on GitHub, without any detailed report about their contributions.

YOLO models have a rich development history but are now reduced to marginal changes and implementation tricks. This is extremely useful from an engineering view, but it is less relevant from a research perspective. Nevertheless, the YOLO framework inspired plenty of other one-stage detectors. In particular, some detectors drop the use of anchors boxes and instead detect objects with keypoints. CornerNet [START_REF] Law | Cornernet: Detecting objects as paired keypoints[END_REF] for instance produces heatmaps to determine the position of two corner points for each object, preventing the use of boxes at all. CenterNet [START_REF] Duan | Centernet: Keypoint triplets for object detection[END_REF] is a re nement of Corner-Net that only outputs a center point and infer the box dimensions from the keypoint features. Both

CornerNet and CenterNet output a keypoint heatmap for each class and involve sophisticated postprocessing to obtain the predicted bounding boxes. FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF] simpli es this by directly predicting boxes from each feature location.

It is also worth mentioning Single Shot Detector (SSD) [START_REF] Liu | Ssd: Single shot multibox detector[END_REF], which was proposed slightly after YOLO.

It is also a one-stage detector, but unlike the rst YOLO version, it is fully convolutional. This has several advantages as it predicts boxes densely on the images (higher recall, better detection of small objects) and it adapts better to di erent input image sizes. But most importantly, SSD leverages features from various scales for the predictions which dramatically improves the detection of small targets. Although this idea was introduced by SSD, it was popularized later with Feature Pyramid Networks (FPN) [START_REF] Lin | Feature pyramid networks for object detection[END_REF]. We will discuss these advancements in the following section as well as the choice of the CNN architecture choice.

One-stage object detectors were at rst lagging behind two-stage detectors in terms of detection performance. However, the recent progress tends to close this gap, making the one-stage detectors the standard choice in the industry as they o er the best speed/performance tradeo .

Backbone network choice

In the OD literature, it is common to denote the main features' extractor CNN as the backbone of the network. Then, the lightweight module designed for classi cation and box regression on top of the backbone is logically called the detection head. What is placed between the backbone and the head (e.g., FPN and RPN) are sometimes referred to as the neck. Fig. 2.5 highlights these three main components of the object detector structure and outlines some design choices for each component. The backbone has an extremely important role in object detection as it extracts the features on which the classi cation and regression modules work. The choice of the backbone has been driven by the advances in classi cation, speci cally the most common backbones have largely proven their capacities on ImageNet. First AlexNet [START_REF] Krizhevsky | Imagenet classi cation with deep convolutional neural networks[END_REF] was used by R-CNN, then VGG networks [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF] for Fast R-CNN, Faster R-CNN, YOLO, SSD and many others. These were quickly replaced by Residual Networks (ResNets) [START_REF] He | Deep residual learning for image recognition[END_REF] which provide a large improvement in ImageNet classi cation, and consequently in 2.1 O D the detection task. Following the extensions upon ResNets, object detectors successfully adopted WideResNet [START_REF] Zagoruyko | Wide residual networks[END_REF], ResNext [START_REF] Xie | Aggregated residual transformations for deep neural networks[END_REF] or E cientNet [START_REF] Tan | E cientnet: Rethinking model scaling for convolutional neural networks[END_REF]. More recently, the backbone network shifted from CNN to visual transformers, but this review will be conducted in Sec. This was introduced rst in SSD [START_REF] Liu | Ssd: Single shot multibox detector[END_REF], but it was popularized with Feature Pyramid Networks (FPN) [START_REF] Lin | Feature pyramid networks for object detection[END_REF]. FPNs combine features with various resolutions from the input image at a single resolution (i.e., it is not necessary to perform the forward pass on multiple rescaled versions of the same image). FPNs extract intermediate feature maps in the backbone and aggregate them in a bottom-up manner (in contrast to the top-down processing of the forward pass). This bottom-up computation path is generally implemented with deconvolution layers, such as in Deconvolution SSD [START_REF] Fu | Dssd: Deconvolutional single shot detector[END_REF]. FPNs are plug-and-play modules that can be attached to most backbone networks and signi cantly improves the detection performance, especially for small objects. In two-stage detectors, FPN are often combined with Region of Interest Alignment Layer (RoI Align) to extract RoI features from the most relevant feature level (i.e., according to the RoI size). RoI Align was introduced in Mask R-CNN [START_REF] He | Mask r-cnn[END_REF],

an extension of Faster R-CNN for Instance Segmentation, that uses a FPN. In one-stage detectors, the detection is carried out on all feature levels output by the FPN, deeper levels are responsible for detecting larger objects.

Of course, plenty of contributions were proposed to improve upon FPNs. Path Aggregation Net (PANet) [START_REF] Liu | Path aggregation network for instance segmentation[END_REF] for instance adds another top-down path before aggregating features from multiple levels with an Adaptive Feature Pooling layer. However, the design of the FPN architecture is not trivial and requires lots of trial and error. To nd optimal FPN designs, NAS-FPN [START_REF] Ghiasi | Nas-fpn: Learning scalable feature pyramid architecture for object detection[END_REF] proposed to apply principles of Neural Architecture Search for the design of FPNs and achieved superior detection performance. However, these questions are not so relevant to us as they mainly focus on Auto-ML problematics.

Non-Maximal Suppression

Note that these methods often output numerous detections and require elaborated ltering schemes to prevent duplicates. Among others the Non-Maximal Suppression (NMS) became quite popular.

In the case of largely overlapping boxes (i.e., when the IoU is above a xed threshold), NMS keeps only the most con dent box to prevent duplicates. This is done separately for each class so that objects from di erent classes may overlap. While the previous methods leverage transformers to make detections, they still rely on large CNN models as the backbone. This choice is also questioned by recent advances in visual transformer architectures. On the one hand, Data-e cient image Transformers DeiT [START_REF] Touvron | Training data-e cient image transformers & distillation through attention[END_REF] and Bidirectional Encoder representation from Image Transformers (BEiT) [START_REF] Bao | BEiT: BERT Pre-Training of Image Transformers[END_REF] both improve ViT's accuracy and training strategy. Both DeiT and BEiT show similar ne-tuning properties as CNNs, unlocking their application for various downstream tasks including object detection. On the other hand, ConViTs [START_REF] Stéphane D'ascoli | Convit: Improving vision transformers with soft convolutional inductive biases[END_REF] and Swin Transformers [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF][START_REF] Liu | Swin transformer v2: Scaling up capacity and resolution[END_REF] make the attention computation much faster with spatial inductive bias and hierarchical structure respectively.

Another source of improvement for the detection backbones comes from self-supervised training.

Recent advances in large-scale self-supervised training for classi cation are now being adapted to other visual tasks. DINO [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF][START_REF] Oquab | DINOv2: Learning Robust Visual Features without Supervision[END_REF] pre-trains both CNN and Transformer based backbones in a contrastive way with carefully designed augmentation schemes to obtain more robust visual features.

Using these pre-trained backbones generally boosts the performance on many visual tasks, at least when applied to su ciently similar images. The very recent Segment Anything Model (SAM) [START_REF] Kirillov | Segment Anything[END_REF] also falls under the same category called foundation models. Even though no derivative work has been published yet, the capacities of SAM are promising for object detection. To summarize, recent advances in Object Detection have largely followed the Transformer "revolution". First, with improved detection heads (the DETR family), and then, with improved backbones, based on Transformers but also revamped CNNs.

Object Detection on Aerial Images

Most of the works presented in the above sections focus on the object detection task applied to natural images. Aerial images di er signi cantly from natural images, they do not contain any perspective, objects can be arbitrarily rotated, and they have a greater object size variance. Given this, it seems obvious that some adjustments are required to adapt popular detectors to aerial images.

Oriented Bounding Boxes

As objects can be oriented in any direction, some aerial object detection datasets give annotations as oriented bounding boxes. This slightly changes the problem, but most detectors can easily be extended to deal with rotated boxes. The bounding box formulation can be extended so that it is rotated, the regression layer must then be adapted to predict a rotation angle [START_REF] Liu | Learning a rotation invariant detector with rotatable bounding box[END_REF][START_REF] Xie | Oriented R-CNN for object detection[END_REF], more than four coordinates [START_REF] Xia | DOTA: A large-scale dataset for object detection in aerial images[END_REF][START_REF] Xu | Gliding vertex on the horizontal bounding box for multi-oriented object detection[END_REF], or to use rotated RoI, for instance with RoI Transform [START_REF] Ding | Learning RoI transformer for detecting oriented objects in aerial images[END_REF].

Small Object Detection

Aerial images contain objects with great size variance due to discrepancies in the shot conditions (altitude, sensor resolution, camera focal length, etc.). In addition, they also have smaller objects than natural images. To deal with the object size variance, it is possible to leverage supplementary information such as the ground sample distance (GSD). The ground sample distance represents the size of one pixel on the ground. Based on GSD, a model can infer the size of the RoI and therefore re ne its predictions, as done by GSDet [START_REF] Li | GSDet: Object Detection in Aerial Images Based on Scale Reasoning[END_REF]. However, object size variance is generally a limited issue compared to detecting small objects, which remains an open challenge in object detection.

Many attempts were made to solve this issue using speci c architecture design [START_REF] Deng | Extended Feature Pyramid Network for Small Object Detection[END_REF], multiscale training [START_REF] Singh | Sniper: E cient multi-scale training[END_REF][START_REF] Singh | Scale normalized image pyramids with AutoFocus for object detection[END_REF], data-augmentation [START_REF] Kisantal | Augmentation for small object detection[END_REF] or super-resolution [START_REF] Rabbi | Small-object detection in remote sensing images with end-to-end edgeenhanced GAN and object detector network[END_REF][START_REF] Bai | SOD-MTGAN: Small Object Detection via Multi-Task Generative Adversarial Network[END_REF][START_REF] Syeda Nyma Ferdous | Super resolution-assisted deep aerial vehicle detection[END_REF]. Additionally, Normalized Wasserstein Distance (NWD) [START_REF] Xu | Detecting Tiny Objects in Aerial Images: A Normalized Wasserstein Distance and A New Benchmark[END_REF] proposes an alternative to the IoU loss speci cally designed for detecting small objects. It consists in computing the Wasserstein distance between two Gaussian distributions tted on the two bounding boxes compared. Moreover, NWD is not only used as a loss function but also as an example selection criterion.

Training Object Detectors

While we brie y presented how object detectors are trained in the previous sections, we did not give much detail about the loss functions and the optimization process. We remedy this here by reviewing the loss functions of several popular object detection benchmarks.

O D

Loss functions for Object Detection

As object detectors must solve both classi cation and regression tasks, most detection loss functions are divided into two components, a classi cation loss and a regression loss. Plenty of choices exist for both components. For the classi cation loss, the most common choice is the Cross-Entropy loss:

L cls CE ( ŷi , y i ) = -log( lc i i ), (2.13)
where lc i i is the predicted probability that the box i contains an object of class c i , c i being the true label (y i = (b i , c i )). However, some alternatives such as the L1 or L2 losses over the class probabilities are also employed:

L cls L1 ( ŷi , y i ) = li -l i 2 2 , (2.14)
where li denotes the class probability vector ( li = { lc i } c∈C ) and l i is the one-hot encoded true probability vector of box i. Another very common classi cation loss function in recent detector is the Focal Loss (FL) function [START_REF] Lin | Focal loss for dense object detection[END_REF], which was introduced with the RetinaNet object detector. Focal loss is designed to address the class imbalance issue that is inherent to dense object detectors (the background class is much more represented than other classes). Focal loss reduces the relative loss of well-classi ed examples so that the learning process focuses on misclassi ed objects. It is de ned as follows:

L cls FL ( ŷi , y i ) = -α c i (1 -lc i i ) γ log( lc i i ), (2.15)
where α c i is an inverse class-frequency parameter and γ controls how much FL reduces the contribu-

Example selection strategy

In the previous paragraph, we presented the various loss functions employed in the Object Detection literature. For simplicity, we de ned these losses for a couple of predicted and ground truth detection labels ( ŷi , y i ). In reality, the losses are computed as the sum of all such couples (over one or multiple images). However, it is not straightforward to build these couples as there may be more predictions than ground truths, missed objects, or false detection. Each detector has its own strategy to operate the matching between prediction and ground truth. These strategies were brie y presented in the previous sections, but we regrouped them inside Tab. 2.2 for clarity.

Matching Strategy

Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] -Select RoI with at least 0.5 IoU with a GT as Positive Samples (PS) and RoI with low IoU (< 0.1) as Negative Samples (NS). -Classi cation loss is computed on all selected samples (PS with the corresponding GT class and NS with the background class). -Regression Loss is only computed with PS.

YOLO [START_REF] Redmon | You only look once: Uni ed, real-time object detection[END_REF] -Select PS as grid cells in which there is a GT center point and assign the highest IoU boxes in case of multiple GT in one cell. All others are NS. -Classi cation done separately on PS and NS.

-Regression loss with PS only.

FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF] -Select PS as feature map locations that fall inside a GT, all others are NS.

If multiple GT for the same location take the smallest one. -Classi cation on PS and NS.

-Regression on PS only.

DETR [59]

-1-to-1 optimal prediction and GT assignment according to localization and classi cation cost.

-No-object are added to the GT set when the predictions are more numerous.

-Classi cation loss is computed for all matched couples. -Regression loss only for couples with an actual GT.

H-DETR [62]

-DETR matching.

-Supplementary 1-to-many matching with duplicated and augmented GT for training.

Di usionDet [START_REF] Chen | Di usiondet: Di usion model for object detection[END_REF] -DETR matching

Datasets Presentation

There exist numerous object detection datasets in the literature. We present in this section four of them in detail as they will be extensively used in this manuscript. These datasets are Pascal VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF], MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], DOTA [START_REF] Xia | DOTA: A large-scale dataset for object detection in aerial images[END_REF] and DIOR [START_REF] Li | Object detection in optical remote sensing images: A survey and a new benchmark[END_REF]. We choose these datasets because they were the most relevant and widespread datasets of natural (Pascal VOC and COCO) and aerial (DOTA and DIOR)

images at the beginning of this project. Some other datasets will be punctually used, especially for the cross-domain experiments in Chap. 7, they will be presented in detail there. However, we draw up a non-exhaustive list in Tab.

F(I, θ) = ĉ ∈ C. (2.18)
Deep Learning based models proved to be particularly adapted to the classi cation task in a fully supervised setting (i.e., provided with su ciently large annotated datasets). This was supported amongst others by LeNet [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] for digit classi cation, and by AlexNet [START_REF] Krizhevsky | Imagenet classi cation with deep convolutional neural networks[END_REF] and ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] for ImageNet classi cation. However, the classi cation task in this form is not a topic for this section, and we refer to [START_REF] Rawat | Deep convolutional neural networks for image classication: A comprehensive review[END_REF] for a complete review of existing works in this eld.

In the few-shot setting, the classi cation goal remains the same, predicting the class of an image.

The input image to an FSL model is usually denoted as a query image, and therefore, the test set is called the query set. What changes between the few-shot and regular settings is the amount (noted C base ) and novel classes set (noted C novel ) respectively. Speci cally, we have:

D base = {(I i , c i )} 1≤i≤|D base | c i ∈ C base , (2.19)
D novel = c∈C novel {(I c k , c)} 1≤k≤K .
(2.20)

As mentioned above, D base is used to train the model during a rst phase called base training. During this phase, the model has access to plenty of annotated data and is trained in a supervised manner to classify images within C base . It is noteworthy to point out that this supervised base training is not the only possible choice. Recent advances in Self-Supervised Learning (SSL) [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Grill | Bootstrap your own latent-a new approach to self-supervised learning[END_REF][START_REF] He | Momentum contrast for unsupervised visual representation learning[END_REF][START_REF] Caron | Unsupervised learning of visual features by contrasting cluster assignments[END_REF] proved that SSL is a competitive alternative to supervised base training.

After base training, the novel dataset is leveraged to adapt the model to classify the novel classes.

Hence, the few-shot classi cation task can be seen as predicting the class label from the input image and the novel dataset:

F(I, D novel ) = ĉ ∈ C. (2.21)
The model adaptation generally starts with small architectural modi cations, such as replacing the nal classi cation layer with a novel layer randomly initialized and with the right number of outputs (e.g., if the numbers of base and novel classes di er). Then, several approaches exist for adjusting the model to the novel classes given the novel dataset. We identify here four di erent adaptation strategies and will present each of them in the next sections. These strategies are: ne-tuning, metric-learning, meta-learning and attention-mechanisms. However, there are no clear boundaries between these four areas, Fig. 2.8 illustrates the interactions between the various strategies and gives a few examples for each category. We propose this taxonomy as it suits well the few-shot object detection eld. Hence, reviewing FSC through this lens helps to understand how these techniques could be extended for detection. However, there exist much more detailed taxonomies and reviews about FSC in the literature, [START_REF] Wang | Generalizing from a few examples: A survey on few-shot learning[END_REF][START_REF] Song | A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities[END_REF] are worthy examples. Note that the novel dataset can also be used during inference, so that adaptation is done "on the y". This is called transductive inference and will be presented in Sec. 2.2.1.6.

In the most common few-shot setting, we have C base ∩C novel = ∅, meaning that there are no common classes between the base and novel sets. Of course, some works focus on relaxing these assumptions, we will outline some of them in Sec. However, the authors of [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF] propose regularization penalties to be applied during ne-tuning that alleviate both the over tting and catastrophic forgetting. Speci cally, the penalty prevents the netuned weights from being too far from the pre-trained weights. Similarly, [START_REF] Yoo | E cient k-shot learning with regularized deep networks[END_REF] proposes several regularization loss functions and a grouped parameter update during ne-tuning to overcome the over tting. Closely related, [START_REF] Trianta Llou | Few-shot learning through an information retrieval lens[END_REF] leverages Direct Loss Minimization's theorem [START_REF] Hazan | Direct loss minimization for structured prediction[END_REF] to optimize the model's weights with an Information Retrieval Loss at inference. Although ne-tuning is a relatively simple approach for FSL, it was not much explored in the case of classi cation. We will see in Chap. 3 that it has gained more attention recently for more complex tasks.

Metric Learning

Metric learning is a branch of deep learning which consists of learning self-organized representation spaces, i.e., similar inputs should have similar representation in the embedding space. It was rst introduced with Siamese Networks for signature veri cation [START_REF] Bromley | Signature veri cation using a" siamese" time delay neural network[END_REF] and later for face identi cation [START_REF] Chopra | Learning a similarity metric discriminatively, with application to face veri cation[END_REF]. The idea behind the Siamese Networks is to leverage two copies of the same model and feed them two di erent images. The output of the two networks should be similar if the input D , F S L C D A images look similar as well. Siamese networks were then applied for one-shot image classi cation by [START_REF] Koch | Siamese neural networks for one-shot image recognition[END_REF]. It was one of the rst attempts to solve this task using deep neural networks. Features from the query and support images are extracted by the siamese nets and then compared by a nal prediction layer. This nal layer takes as input the di erence between the features of the query and support images. Its role is to assess whether the features are similar enough so that the two images belong to the same class. Following Siamese Networks, a series of works was proposed based on the same principle. These contributions are embodied by Prototypical Networks [START_REF] Snell | Prototypical Networks for Few-shot Learning[END_REF]. The episodic strategy in the context of metric learning is rst used in Matching Networks [START_REF] Vinyals | Matching Networks for One Shot Learning[END_REF],

an earlier work than Prototypical Network. Matching Networks are more inspired by the metalearning techniques, hence the episodic strategy. Two networks are trained jointly, one to extract the support features and one for the query features. However, the way query and support features are combined di ers from other metric-learning methods. The authors leverage an attention mechanism to compute the predicted class probabilities as a similarity-aware weighting of the support examples labels. Closely related, Task Dependent Adaptive Metric (TADAM) [START_REF] Oreshkin | Tadam: Task dependent adaptive metric for improved few-shot learning[END_REF] learns a task representation and adapts its embedding network through a task-conditioning layer which resembles an attention mechanism. The class prototypes are then task-dependent and an image is classi ed according to the most similar prototype. Matching Network and TADAM are therefore at the intersection of three approaches for FSL: metric-learning, meta-learning and attention mechanism. These are reviewed in the two following sections.

Meta-Learning

As hinted at the end of the previous section, meta-learning's paradigm is "learning to learn". This was the main motivation for the episodic training strategy described there. Each episode forces the model to adapt to new classes, repeating these episodes should overall increase the adaptation

2.2 F S L : L L D
capacities of the model. However, the concept of meta-learning goes even further.

This concept was introduced and popularized during the 1990s [START_REF] Schmidhuber | Evolutionary principles in self-referential learning, or on learning how to learn: the meta-meta[END_REF][START_REF] Thrun | Learning to learn: Introduction and overview[END_REF]. At the time, meta-learning was mostly applied in the context of policy learning, with evolutionary or reinforcement learning methods. It was brought up-to-date for the few-shot classi cation by Model-Agnostic Meta-Learning (MAML) [START_REF] Finn | Model-agnostic meta-learning for fast adaptation of deep networks[END_REF] which proposes to directly train the initial weights of a classi er so that it will quickly adapt to a given task. The optimization is done in a nested manner. At the inner level, a task is sampled (like with the episodic training strategy) and the classi er is initialized with the current initial weights. A few gradient steps are performed on the classi er with respect to the task objective function. Then, at the outer level, the initial weights are updated through gradient descent on the task loss value computed with the trained classi er. The meta-update converges to a set of initial weights that make the classi er "easy to train" on any task. However, MAML does not take task information into account for the weight initialization, and it is unrealistic to nd truly task-agnostic initializations. Therefore, [START_REF] Lee | Gradient-based meta-learning with learned layerwise metric and subspace[END_REF] extends MAML to only choose a subset of the optimal initial parameters to initialize the classi er based on task information. Orthogonally, some contributions integrate uncertainty in the weight initialization [START_REF] Franceschi | Bilevel programming for hyperparameter optimization and metalearning[END_REF][START_REF] Kim | Bayesian model-agnostic meta-learning[END_REF], others make the training easier [START_REF] Singh Behl | Alpha maml: Adaptive model-agnostic meta-learning[END_REF][START_REF] Rusu | Meta-learning with latent embedding optimization[END_REF] or provide a theoretical framework and guarantees [START_REF] Fallah | Generalization of model-agnostic meta-learning algorithms: Recurring and unseen tasks[END_REF].

Similarly, some meta-learning techniques propose learning the optimization process instead of the weight initialization. This is the case of Optimization as a Model [START_REF] Ravi | Optimization as a model for few-shot learning[END_REF] which trains a LSTM metalearner to output gradient updates for the classi er network. The meta-learner takes as input the weights of the classi er and the gradients computed on a given task. The recurrent nature of the meta-learner helps to keep track of the previous error signals and update consequently the weights of the classi er. Close to MAML, the meta-learner is updated after several weight updates (with di erent tasks) based on the loss values of the classi er on a test set.

Another meta-learning direction is introduced with Ridge Regression Di erentiable Discriminator (R2D2) [START_REF] Bertinetto | Meta-learning with di erentiable closed-form solvers[END_REF]. It consists in teaching a model "to use" standard machine learning tools such as Ridge or Lasso Regression. These techniques often have closed-form solutions and are fast to compute when few data are available. In R2D2, a CNN is trained as a feature extractor as a meta-learner, while the classi er's weights are computed with a ridge regression from the support set. The metalearner CNN is trained to extract features that will generate optimal classi er weights through ridge regression.

Attention-Based methods

As an alternative to the rather complex and heavy meta-learning methods, a line of work followed the MAML principle but focused only on some layers rather than on the complete classi er. This originates with LearNets [START_REF] Bertinetto | Learning feed-forward one-shot learners[END_REF] that are trained to output the weights of a convolution kernel from a support example. The kernel is then used in a Dynamic Convolution Layer (DCL) inside the classi er, which in the end predicts a class-membership score (according to the class of the support example).

When multiple classes are available in the support set, the DCL is applied with each class features D , F S L C D A independently and the classi cations are done in a binary fashion. This can be understood as an attention mechanism between the query and support features. The dynamic convolution acts as a lter responsive to the support features. To put it another way, the locations in the query feature map that are similar to the support features will be highlighted. The dynamic convolution sets the focus of the classi er on features from the support class. Self-Attention (SA) is probably the most common form of attention mechanism used in the current deep learning literature. It highlights similarity links between subparts of the input (e.g., di erent locations in an image, or two words in a sentence). Here, with DCL, this is done with two distinct images: a query and a support image. Several other works build upon this idea. Simple Neural Attentive Learner (SNAIL) [START_REF] Mishra | A simple neural attentive meta-learner[END_REF] designs more complex attention blocks, based on transformers, to perform the classi er's adaptation. Although their primary goal is to tackle few-shot reinforcement learning with temporal convolutional layers (to deal with causality), they apply it successfully to FSC as well. The N K images in the support set are fed in random order and the query image is given last. Very similar to SNAIL, CrossTransformers [START_REF] Doersch | Crosstransformers: spatially-aware few-shot transfer[END_REF] assemble an attention module to combine query and support features. The crucial di erence with SNAIL is the preservation of the spatiality of the features. Most previous works aggregate the support features to perform the adaptation, losing the spatial information of the support image. Inspired by the recent progress of ViT, CrossTransformers manage to adapt the classi er while preserving spatial information. Just like many metric learning methods presented in the previous section, some of the attention-based techniques discussed above borrow an episodic training strategy from meta-learning.

The query-support attention mechanism can also be interpreted as a conditioning of the classi er's input based on the support features. That is the view adopted by Dynamic Conditional Network [START_REF] Zhao | Dynamic conditional networks for few-shot learning[END_REF]. The general idea is very similar to LearNet except for the training which is not done in an episodic manner. Conditionally Shifted Neurons (CSN) [START_REF] Munkhdalai | Rapid adaptation with conditionally shifted neurons[END_REF] see the adaptation as the conditioning of the classi er activations. The meta-learner outputs shift values that are added to the pre-or post-activation values in the network. The shifts are computed from a task description stored in a memory. The task description regroups the activation of all the layers of the classi er fed with the support images. The use of a memory bank is widespread along with attention-based mechanisms for FSC. Memory Augmented Neural Network (MANN) [START_REF] Santoro | Meta-learning with memory-augmented neural networks[END_REF] epitomizes this line of work. It leverages a controller (i.e., a small network) to read and write in the memory. The controller generates a key from an input, which is then used to either add a new entry in the memory or retrieve already stored information. The retrieval is done through an attention-like mechanism. The memory is built throughout a task episode adaptively. When a new memory is added, if a similar memory is already stored, the new memory re nes the existing one to build more relevant representations.

Plenty of contributions took inspiration from MANN. Adaptive Posterior Learning (APL) [START_REF] Ramalho | Adaptive posterior learning: few-shot learning with a surprise-based memory module[END_REF] renes the memory writing process to store only "surprising" memories. Meta Networks [START_REF] Munkhdalai | Meta networks[END_REF] also leverage external memory only accessible to the meta-learner in charge of adapting the classi er.

[152] proposes a second abstract memory which stores re ned information relevant for the current task. All these memories are generally wiped when the task is modi ed. However, life-long memory

2.2 F S L : L L D
[153] also bene ts the FSC even though it is mostly exploited for slightly more challenging tasks such as Generalized FSL or Continual Learning (see Sec. 2.2.1.7).

Transductive Inference

A recent line of work tackles the few-shot classi cation problem with transductive inference. Transductive inference, in contrast to inductive inference, consists of leveraging labeled and unlabeled images and classifying all unlabeled data points at the same time. Conversely, inductive inference deals with each data point independently. Of course, deploying such methods in an industrial scenario requires having multiple data points available at test time. For real-time applications, it is generally not practical. However, in the case of COSE and the detection task, this is largely applicable. Very large images cannot be processed as a whole, they must be divided into smaller images. In addition, for the detection task, an image is often seen as a collection of objects or regions of interest. COSE's use case is therefore rather well-suited for transductive inference. Therefore, we brie y review the recent advances of transductive learning for FSC in this section.

Transductive inference is an old concept of statistical learning that was popularized under this name and for machine learning applications by Vladimir Vapnik in the 1990s [START_REF] Vladimir | An overview of statistical learning theory[END_REF]. As mentioned above, in the transductive setting, an entire unlabeled dataset (e.g., a test dataset or a query set in the fewshot context) is available at test time. Transductive methods leverage information contained both in the support and query set to make predictions. Before application on FSC, transduction was applied to regular classi cation on small-size datasets, yielding signi cant improvements over inductive methods. Amongst them, Transductive Support Vector Machines [START_REF] Joachims | Transductive inference for text classi cation using support vector machines[END_REF] extends the well-known SVM [START_REF] Cortes | Support-vector networks[END_REF] to make use of unlabeled information to re ne the class separation margins. Another direction is taken by [START_REF] Zhou | Learning with local and global consistency[END_REF] which derives an iterative method that propagates known labels to unlabeled data points according to their similarity. Recently, TransBoost [START_REF] Belhasin | TransBoost: Improving the Best Ima-geNet Performance using Deep Transduction[END_REF] even applied transduction to the entire ImageNet dataset with signi cant accuracy gain over inductive methods. The authors propose a ne-tuning approach to re ne trained neural networks to perform better on a speci c test set. It takes both the training set and the unlabeled test set to compute a regularization loss function that penalizes similar images to be classi ed di erently by the network.

Transductive inference is especially e ective in the few-shot context as the limited labeled data is often not enough to provide su cient supervision. Leveraging additional unlabeled data is therefore highly bene cial. Various approaches were proposed to make use of this supplementary information within the already existing few-shot frameworks. Probably the most straightforward approach is to ne-tune pre-trained models with additional regularization loss based on the labeled and unlabeled data. This is the direction taken by [START_REF] Guneet S Dhillon | A baseline for few-shot image classi cation[END_REF] which compares the few-shot performance of several methods against a simple transductive ne-tuned baseline. Similarly, Transductive Episodic-wise Adaptive Metric (TEAM) [START_REF] Qiao | Transductive episodic-wise adaptive metric for few-shot learning[END_REF] and Transductive Information Maximization with Gradient Descent (TIM-GD) [START_REF] Boudiaf | Information maximization for few-shot learning[END_REF] also both leverage ne-tuning objectives to re ne the model before transductive inference. This resembles semi-supervised learning which ne-tunes models with additional unla-D , F S L C D A beled data, i.e., di erent from the training and test set. However, the essence of transductive learning lies more in adapting the inference based on the additional information rather than ne-tuning the model. This can be done by direct optimization of an objective function with regularization as in LaplacianShot [START_REF] Ziko | Laplacian regularized few-shot learning[END_REF], TIM [START_REF] Boudiaf | Information maximization for few-shot learning[END_REF] or Cross-Attention Networks [START_REF] Hou | Cross attention network for few-shot classi cation[END_REF]. Many propositions iteratively propagate the known labels to unlabeled data points within a graph structure [START_REF] Kim | Edge-labeling graph neural network for few-shot learning[END_REF][START_REF] Liu | Learning to propagate labels: Transductive propagation network for fewshot learning[END_REF][START_REF] Hu | Graph-based interpolation of feature vectors for accurate few-shot classi cation[END_REF][START_REF] Rodríguez | Embedding propagation: Smoother manifold for few-shot classi cation[END_REF][START_REF] Lazarou | Iterative label cleaning for transductive and semi-supervised few-shot learning[END_REF]. But there also exist contributions that exploit transductive inference through metric learning with e.g., Prototype Recti cation [START_REF] Liu | Prototype recti cation for few-shot learning[END_REF] or Meta-Con dence Transduction [START_REF] Seong | Meta-learned con dence for few-shot learning[END_REF], which meta-learn a distance metric. Meta-learning based methods also get their transductive extension, such as Reptile [START_REF] Nichol | On rst-order meta-learning algorithms[END_REF] which extends MAML to perform transductive inference by leveraging information shared by test samples through the batch normalization layers.

Extending the Few-Shot Setting

The few-shot setup that we described in previous sections is limited and makes a few assumptions:

1. The set of base and novel classes are known in advance.

2. At test time, only the performance on novel classes matters.

3. Novel classes are only added once and all at the same time.

These assumptions signi cantly simplify the problem, but these are relaxed by di erent sub-elds of few-shot learning. In some aspects, the few-shot detection can be seen as a relaxation of these assumptions. Various tasks, similar to few-shot classi cation, exist in the literature. [START_REF] Geng | Recent advances in open set recognition: A survey[END_REF] provides a comprehensive taxonomy of these tasks. We will brie y present in this section some relevant extensions of the few-shot classi cation for COSE's application and the detection task. Tab. 2.4

provides an overview of these tasks and their di erences in terms of goal and available data. proaches for Open-set Recognition, but we will not review them here in detail and refer the reader to a complete survey [START_REF] Geng | Recent advances in open set recognition: A survey[END_REF] about it. Instead, we simply outline the general principle behind algorithms that tackle this problem. Two main approaches coexist in the literature, discriminative and generative approaches. The former ones propose techniques to distinguish between known and unknown classes using discriminative information, e.g., distance to class representations [START_REF] Bendale | Towards open set deep networks[END_REF]. The latter leverage generating models to hallucinate negative examples as additional training data [START_REF] Inhyuk | Open set recognition by regularising classi er with fake data generated by generative adversarial networks[END_REF].

Of course, transduction also helps in this case and outlierness score [START_REF] Boudiaf | Open-Set Likelihood Maximization for Few-Shot Learning[END_REF] This holds for open-set recognition, but in few-shot there are additional complexities. Not only the classes from the query set may be unknown (i.e., not even in the support set), but the support set could provide irrelevant information for the current task. This setup, introduced in [START_REF] Martin | Towards Practical Few-Shot Query Sets: Transductive Minimum Description Length Inference[END_REF], is not common in the FSL literature even if it is of great interest from an industrial perspective. It is also quite relevant from the few-shot detection point of view as the detection support examples can embed irrelevant information for the task.

Generalized Few-Shot Classi cation

Up to now, we presented the few-shot classi cation problem as only adapting a model to classify novel classes. However, it can sometimes be relevant to keep the possibility of classifying classes from the base dataset. Often the adaptation signi cantly reduces the performance on base classes, this phenomenon is known as the catastrophic forgetting [START_REF] Kirkpatrick | Overcoming catastrophic forgetting in neural networks[END_REF]. When both base and novel classes are of interest, the task is called generalized few-shot learning. This can be achieved with several tricks such as doing the inference with both base-trained and ne-tuned models. But it is also possible with careful extension and ne-tuning of the model, e.g., via disentangling base and novel class predictions [START_REF] Gidaris | Dynamic few-shot visual learning without forgetting[END_REF].

Continual Learning

Generalized few-shot is an intermediary step toward continual or life-long learning which consists in continuously adapting the model with novel classes. This is way more challenging but also resembles the industrial setting better. While extremely relevant from COSE's perspective, we choose not to tackle this problem in this PhD project as it seems more sensible to address rst the already challenging few-shot setting for the detection. In addition, continual learning often leverages complex learning scheme such as task rehearsal [START_REF] Robins | Catastrophic forgetting, rehearsal and pseudorehearsal[END_REF] or adaptive model architectures [START_REF] Li | Learn to grow: A continual structure learning framework for overcoming catastrophic forgetting[END_REF] to prevent forgetting classes or tasks. D , F S L C D A

Cross Domain Adaptation

Sometimes, there are signi cant discrepancies between images from train and test datasets. We Hence, when the task changes from source and target, the set of classes changes as well. We denote these sets as C source and C target to comply with the CDA notations. Note that these sets of classes correspond to the base and novel classes in the FSC context. Generally, in the CDA literature, a limited amount of annotated data is available for the target domain which prevents direct supervised training. However, if a closely related source domain with su cient available data is available, adaptation to the target domain is possible with limited data. Accordingly, cross-domain adaptation and few-shot learning are closely related problems. In this section, we review the two kinds of CDA, with and without label shift. COSE's industrial application contains CDA's problematics as the imaging systems can be deployed to di erent theaters of operations for which no images were available during training. This distinction is not relevant as in both cases, there is not enough target data to perform directly supervised training (although additional unlabeled target data is often leveraged). Therefore, we choose to review both DA and FSDA at once. This review is not exhaustive, and we refer the reader to [START_REF] Wang | Deep visual domain adaptation: A survey[END_REF] for a more complete overview of Domain Adaptation. Following this survey, we divide our review into two parts, discrepancy-based adaptation and generative modeling approaches.

Domain Adaptation without class shift

Discrepancy-based Adaptation

The simplest way to adapt a model to a target domain is to ne-tune it on the few available target data. The model is rst trained on the source domain to learn the task. Then, ne-tuning is done on the target domain with some tricks to avoid over tting. These tricks consist in reducing the discrepancies between source and target features. For instance, [START_REF] Tzeng | Simultaneous deep transfer across domains and tasks[END_REF] ne-tunes on the target domain with

2.2 F S L : L L D
a regular cross-entropy loss but leverages additional loss functions to minimize domain confusion with additional unlabeled target images. Similarly, [START_REF] Tzeng | Simultaneous deep transfer across domains and tasks[END_REF] has been extended with semi-supervised consistency [START_REF] Gebru | Fine-grained recognition in the wild: A multitask domain adaptation approach[END_REF] and contrastive [START_REF] Motiian | Uni ed deep supervised domain adaptation and generalization[END_REF] losses. Following the same principle, a number of works [START_REF] Ghifary | Domain adaptive neural networks for object recognition[END_REF][START_REF] Long | Learning transferable features with deep adaptation networks[END_REF][START_REF] Tzeng | Deep domain confusion: Maximizing for domain invariance[END_REF][START_REF] Long | Deep transfer learning with joint adaptation networks[END_REF] leverage additional losses based on the Maximum Mean Discrepancy (MMD) or close extensions. MMD is a distance measure between probability distributions. In the context of DA, it can be leveraged to assess the shift from source to target domain for a given class. Employing MMD-based loss functions allows these methods to learn domain invariant features and therefore improve cross-domain generalization. As an example, Central Moment Discrepancy (CMD) [START_REF] Zellinger | Central moment discrepancy (cmd) for domain-invariant representation learning[END_REF] proposes an approximation of MMD to derive a discrepancy regularizer. This regularization is computed over all layers of the model to enforce features from all levels to be domain invariant. Other contributions developed relatively similar techniques based on other criteria such as Kullback-Leiber divergence [START_REF] Zhuang | Supervised representation learning: Transfer learning with deep autoencoders[END_REF], or correlation alignment [START_REF] Sun | Return of frustratingly easy domain adaptation[END_REF].

The methods presented above all ne-tune the models from feature discrepancies. However, as the task remains the same, it is reasonable to assume that optimal weights for the source and target domains are related. Following this idea, [START_REF] Rozantsev | Beyond sharing weights for deep domain adaptation[END_REF] proposes a weight regularization to prevent netuning to nd weights too di erent from source weights. Closely related, [START_REF] Li | Revisiting batch normalization for practical domain adaptation[END_REF] proposes to only change Batch Normalization's statistics to adapt to the target domain.

Finally, advances in adversarial learning provided new ways to address DA by minimizing source and target discrepancies in an adversarial setup. This is embodied by [START_REF] Ganin | Domain-adversarial training of neural networks[END_REF] and [START_REF] Tzeng | Adversarial discriminative domain adaptation[END_REF] which both jointly train a domain discriminator along with the target feature extractor in an adversarial fashion.

The trained extractor embeds images in a shared source-target feature space on which the source classi er can perform well.

Generative Modeling

Another approach to domain adaptation is to arti cially generate target data. This is particularly easy with discriminative approaches based on Generative Adversarial Networks [START_REF] Goodfellow | Generative adversarial networks[END_REF]. GANs were extended to perform domain translation with CoGAN [START_REF] Liu | Coupled generative adversarial networks[END_REF], Pix-2-Pix [START_REF] Isola | Image-to-image translation with conditional adversarial networks[END_REF] and CycleGAN [START_REF] Zhu | Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks[END_REF].

The source domain images can then be converted into source-target image pairs which greatly facilitate domain adaptation with methods similar to the ones described in the previous paragraph. This is done for instance in CyCADA [START_REF] Ho | Cycada: Cycle-consistent adversarial domain adaptation[END_REF]. Of course, GANs are not the only available generative models suitable for this task. Recent advances in image generation leveraging Di usion Processes [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF][START_REF] Ho | Denoising di usion probabilistic models[END_REF] unveil new possibilities for domain adaptation following existing work about generative domain adaptation as done very recently by [START_REF] Zhang | Di usion-based Target Sampler for Unsupervised Domain Adaptation[END_REF].

Closely related, Deep Reconstruction Networks (DRCN) [START_REF] Ghifary | Deep reconstruction-classi cation networks for unsupervised domain adaptation[END_REF] jointly learn to classify and reconstruct images from multiple domains. The model is trained to classify source images and reconstruct target images. This strategy enforces the learning of domain-invariant features and largely improves domain adaptation. Similar approaches have been proposed with disentangled domaininvariant and domain-speci c representations [START_REF] Bousmalis | Domain separation networks[END_REF], or adversarial reconstruction [START_REF] Kim | Learning to discover cross-domain relations with generative adversarial networks[END_REF]. D , F S L C D A

Cross-Domain Adaptation with class shift

Cross-Domain Few-Shot Classi cation (CD-FSC) designates problems where both classes and domain change at the same time. This complexi es further the learning, but it is closer to real-case scenarios and developing such techniques will ease the deployment of classi cation techniques. It is particularly interesting for COSE as it solves two major issues regarding training visual recognition systems for surveillance applications: unde ned objects of interest and changing image appearance.

This setting is relatively new in the few-shot literature and has been popularized in particular by the creation of Meta-Dataset [START_REF] Trianta Llou | Meta-dataset: A dataset of datasets for learning to learn from few examples[END_REF]. Meta-Dataset is a benchmark for CD-FSC. It gathers 10 existing classi cation datasets and proposes a simple testing scenario: pre-train on ImageNet then ne-tune on each dataset individually with limited annotations.

Most of the proposed techniques for solving CD-FSC borrow from both the few-shot learning and domain adaptation elds. Plenty of approaches are then based on the meta-learning strategy, pretraining on the source dataset and ne-tuning episodically on the target domain and novel classes.

Meta-FDMixup [START_REF] Fu | Meta-fdmixup: Cross-domain few-shot learning guided by labeled target data[END_REF] for instance trains episodically a classi er with additional domain discriminant losses computed on an augmented query set (mixing-up source and target domain -MixUp [START_REF] Zhang | mixup: Beyond empirical risk minimization[END_REF] is a well-known augmentation technique). Meta-FDMixup, is later extended with a dynamic mixup strategy by Target Guided Dynamic Mixup (TGDM) [START_REF] Zhuo | TGDM: Target Guided Dynamic Mixup for Cross-Domain Few-Shot Learning[END_REF]. Another merger of FSL and DA techniques is Domain-Adaptive Prototypical Networks (DAPN) [START_REF] Zhao | Domain-adaptive few-shot learning[END_REF], which extends prototypical networks with a domain adaptation module for prototype alignment, trained in an adversarial fashion. Closely related, [START_REF] Chen | Cross-Domain Cross-Set Few-Shot Learning via Learning Compact and Aligned Representations[END_REF] proposes a bi-directional prototype alignment. Another line of work tackles CD-FSC through the prism of distillation, for instance, [START_REF] Fu | ME-D2N: Multi-Expert Domain Decompositional Network for Cross-Domain Few-Shot Learning[END_REF] rst trains two "experts" networks to perform the FSC task on both domains independently. Then, a student network is trained to match the output of both teachers using distillation techniques. It results in a student network able to deal with both domains identically. Similarly, Universal Representation Learning (URL) [START_REF] Li | Universal representation learning from multiple domains for few-shot classi cation[END_REF] distills knowledge learned from K classi ers trained on K distinct domains into a single cross-domain model. This is achieved by adding lightweight domain adaptation modules between the feature extraction module and the classi cation layer. Overall these techniques all involve complex training strategies and architectural designs which are not very convenient for industrial deployment, replication, or future extensions. To counter this, ReFine [START_REF] Oh | ReFine: Re-randomization before Fine-tuning for Cross-domain Few-shot Learning[END_REF] proposes a simple ne-tuning strategy that only re-initializes the last layers of the model before ne-tuning to facilitate domain adaptation.

Much simpler than concurrent approaches, it yields competitive results.

Finally, some other works [START_REF] Yao | Cross-domain few-shot learning with unlabelled data[END_REF][START_REF] Zhang | Few-Shot Adaptation of Pre-Trained Networks for Domain Shift[END_REF][START_REF] Hu | Adversarial Feature Augmentation for Cross-domain Few-Shot Classi cation[END_REF] study an even harder task when target domain data are completely unlabeled. We will not review this kind of approach as it is out of the industrial scope of COSE.

C

Conclusion

This chapter presents the object detection and few-shot learning elds, both necessary to the conception of few-shot object detectors. For object detection, notations and problem de nitions are given in detail, as well as a list of popular evaluation metrics and datasets. A thorough review of Few-Shot Object Detection (FSOD) task. In this chapter, we properly de ne the FSOD setting and present an exhaustive review of the current literature. We also explain how detection datasets can be leveraged for FSOD and how the proposed methods are evaluated.

Problem de nition

Unsurprisingly, the Few-shot Object Detection task aims to detect objects just as regular object detection but under the few-shot constraints. Speci cally, given an input image I, FSOD's goal is to learn a detection model F(

D base = {(I c i i , Y c i i )} 1≤i≤|D base | c i ∈ C base , (3.1)
D novel = c∈C novel {(I c k , Y c i )} 1≤k≤K , (3.2)
where I c k is an image containing at least one instance of the class c, and Y c k is the corresponding annotation set (bounding box and label) for the image I c k , ltered to contain only class c instances. Note that there could be more than K annotations per class as multiple objects of the same class can be visible on one image. This setting is commonly used in the FSOD literature and called N -ways K-shots object detection. Conversely, keeping only one annotation to comply with the few-shot classi cation setting can be problematic as it provides wrong supervision to the model. This issue will be elaborated further in Chap. 4. Hence, based on the input image and the support set, the fewshot detection model F(•, θ) should predict bounding boxes and labels for all instances of classes C novel :

F(I, D novel ) = Ŷ = {ŷ i } M I i=1 = {( bi , ĉi )} M I i=1 , with ĉi ∈ C novel . (3.3)
This setup resembles the FSC setting described in Chap. 2, but brings some complications. While the sets of base and novel classes are disjoint, FSOD must deal with the background. Any object that does not belong to either the base or novel class sets is considered background. Therefore, an object detector can encounter unknown classes at test time and must be able to ignore them. No information about the background classes is available which makes it even more di cult to discriminate between classes of interest and background. From this perspective, FSOD is closer to the few-shot open-set recognition problem than FSC. In addition, multiple di erent classes of interest can be depicted within a single image. Distinct objects can overlap in the image and their features (potentially from di erent classes) can blend, making recognition challenging. This is reinforced as the objects get smaller, their features get noisier and can be misclassi ed more easily. This stands for the query images but also for support images which increases the di culty compared to FSC.

Review of the Few-Shot Object Detection Literature

Even though FSOD is a natural extension of FSC, the di culties mentioned above prevent the direct use of FSC techniques, just as classi cation techniques may be extended for detection. Of course, the main principles for adapting classi cation models to the few-shot setting can be reused, but they need to be carefully adjusted to take care of the supplementary challenges of the detection task. Hence, as for FSC, the detection models are rst trained on the base dataset and then adapted to novel classes with the support set. This adaptation can be done in many ways, often based on FSC approaches. Therefore, we adopt the same organization as for Sec. 2.2 and divide our review into four distinct parts: ne-tuning, metric learning, meta-learning and attention-based approaches. FSOD for more thorough reviews. However, note that these surveys are already a few years old, which is already a lot compared to the recency of the eld.

Fine-tuning

Fine-tuning is the simplest approach to tackle FSOD, the principle is quite straightforward and sim- Similarly, [START_REF] Kaul | Label, verify, correct: A simple few shot object detection method[END_REF] leverages a kNN classi er to "verify" the predicted labels and lightweight bounding regressors to "correct" the predicted localizations. Multi-scale Positive Sample Re nement (MSPSR)

[257] also proposes a proposal re nement strategy by leveraging a multiscale re nement branch. It Similarly, Synthetic object IMPLantation (SIMPL) [START_REF] Xu | SIMPL: Generating Synthetic Overhead Imagery to Address Zero-shot and Few-Shot Detection Problems[END_REF] leverages 3D models for each novel class to generate high-quality augmented images. SIMPL completely blends the augmented object inside the image, whereas [START_REF] Lin | An E ective Crop-Paste Pipeline for Few-shot Object Detection[END_REF] pastes some background around the novel object as well. SIMPL leverages external information about the classes and requires access to 3D models of the classes which is not always possible. However, this opens an opportunity for addressing the even more challenging zeroshot setting. Pushing even further, [START_REF] Zhang | Hallucination improves few-shot object detection[END_REF] proposes a generative model to enrich the support set and improve detection quality. The hallucinator model is trained jointly with the detector in an EM-like procedure. First, the hallucinator is trained with the detector classi cation loss (the detector is kept frozen). Then, the detector is trained while the hallucinator provides more support examples (with the hallucinator now frozen).

Finally, some other works leverage the ne-tuning strategy with other tricks. Novel Instances Mining with Pseudo-Margin Evaluation (NIMPE) [START_REF] Liu | Novel instance mining with pseudo-margin evaluation for few-shot object detection[END_REF] build a mining network to extract pseudo-labels Considering FSOD as a hierarchical re nement [START_REF] She | Fast Hierarchical Learning for Few-Shot Object Detection[END_REF] is also a viable option as it breaks down base classes into novel classes. While this setup is certainly relevant for many applications, it di ers from the commonly adopted FSOD setting. adapt the query features. Negative prototypes can also be learned to better deal with the background class [START_REF] Yang | Restoring negative information in few-shot object detection[END_REF]. Finally, some contributions manage two sets of prototypes, arguing that one set is not optimal for adapting features for both the classi cation and the regression. Decoupled

Metric-learning based methods

Metric Network [START_REF] Lu | Decoupled metric network for single-stage few-shot object detection[END_REF] introduces a decoupled representation transformation to adapt class prototypes for either classi cation or regression. Likewise, [START_REF] Wu | Generalized and discriminative few-shot object detection via SVDdictionary enhancement[END_REF] splits the representations using Singular

Value Decomposition. Eigenvectors corresponding to the largest singular values represent the main source of variance. The authors argue that this accounts for the general adaptation between base and novel classes. They are leveraged for adapting query features both for regression and classi cation. Other eigenvectors only represent the inter-class variance and therefore, are only used in the classi cation branch. The methods described in this paragraph are slightly di erent from the ones at the beginning of this section. They all use their representation vectors to update query features before a learnable classi cation and regression module, instead of using them for direct classi cation (e.g., distance to the closest prototype). They highly resemble some attention-based methods that will be presented in the next section.

Attention-based methods

As we brie y broached at the end of the previous section, a common technique for FSOD is to adapt the features from the query image based on the support images. This can be understood as an attention mechanism between the query and support features as it highlights locations in the query feature map that are similar to the support images. Another way to see this is to think of the attention mechanism as an adaptive ltering layer. It lters the query features map according to the support features. Highlighted locations in the query map show features similar to the support images. Following the attention module, the regression and classi cation are performed independently per class, often using a shared, class-agnostic detection head (see Fig. 3.2). The query-support combination module takes the query feature maps and the features from all novel classes as input and outputs class-speci c query feature maps. This will be explained in more detail in Chap. 6 which presents a general framework to subsume existing attention mechanisms for FSOD. To summarize, what we call here attention-based FSOD methods are techniques that leverage support information to adapt query features before the classi cation and regression branches. Following this de nition the methods presented at the end of Sec. 3.2.2 can be interpreted as attention methods. However, they are presented from a metric learning perspective which is why they are not discussed in the current section (they will be classi ed as "Attention/Metric Learning" methods in Tab. 3.1).

A seminal work in this eld introduces Feature ReWeighting (FRW ) [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF], which trains a reweighting module along with a YOLO detector. The reweighting module produces class-speci c feature vectors with a Global Pooling layer (GP) applied on the support feature maps. These are then channel-wise multiplied by the query features extracted by the backbone. Hence, class-speci c query features are generated, and the detection head computes predictions for each class separately.

This technique has been widely re-used in the following literature, with other detection frameworks:

Faster R-CNN [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF][START_REF] Fan | Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector[END_REF][START_REF] Xiao | Few-Shot Object Detection With Self-Adaptive Attention Network for Remote Sensing Images[END_REF][START_REF] Xiao | Few-shot object detection and viewpoint estimation for objects in the wild[END_REF][START_REF] Kim | Few-Shot Object Detection via Knowledge Transfer[END_REF][START_REF] Ting | One-Shot Object Detection with Co-Attention and Co-Excitation[END_REF], CenterNet [START_REF] Perez-Rua | Incremental few-shot object detection[END_REF][START_REF] Zhang | PNPDet: E cient few-shot detection without forgetting via plug-andplay sub-networks[END_REF] or FCOS [START_REF] Li | One-shot object detection without ne-tuning[END_REF]. The class reweight-S O D This channel-wise multiplication between query and support features is a simple form of attention.

It can be thought of as an adaptive convolution layer, whose weights depend on the support features. Incidentally, it is often interpreted and implemented as such by existing works, approaching the meta-learning paradigm. However, more complex attention mechanisms have been leveraged in the literature. The incentive behind this improvement was the loss of spatial support information and background feature contamination with the GP layer. First, [START_REF] Xu | Few-Shot Object Detection via Sample Processing[END_REF] proposes a self-attention module to better highlight the support object features and prevent background contamination. Very similar, Dual AwareNess Attention (DANA) [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF] introduces a background attenuation block for the same reason. However, DANA also leverages an alignment mechanism to combine query and support features without losing spatial information. This alignment module is quite close to the visual transformers' attention. It encodes the features from the query images as queries and the features from a support image as keys and values. Queries here refer to the query-key-value (QKV) formulation of the transformers, the correspondence with the query features is fortuitous. Queries and keys are combined to form an attention map, which represents the similarity between the query and support image patches. Then, the dot product between the attention map and the values produces the aligned support features. It can be understood as an alignment as it re-organizes spatially the support features to match the spatial dimension of the query map, according to the similarity between query and support. The underlying idea is that the same class objects in the query and support images will likely have di erent aspects or poses. Therefore, a direct comparison between the feature maps is often irrelevant. The alignment procedure moves support features to similar locations in the query map. This is illustrated in Fig. 3.3, but more details will be given in Chap. 6. Hence, this alignment mechanism combines query and support features, highlighting their similarity without losing spatial information. The same technique is leveraged by several works [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF][START_REF] Han | Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment[END_REF][START_REF] Han | Few-shot object detection with fully cross-transformer[END_REF][START_REF] Zhao | Semantic-aligned fusion transformer for one-shot object detection[END_REF] with slight variations. Similarly, [START_REF] Lee | Few-shot object detection by attending to persample-prototype[END_REF] uses QKV attention with globally pooled support features. Instead, the authors propose to compute attention between the query images and all support images for a class at once. In other works, the attention is generally aggregated per class.

Following the same idea, Meta-DETR [START_REF] Zhang | Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation[END_REF] computes attention between a query image and all support images at once. However, the authors do this for all classes at the same time and replace the binary classi cation with a multi-class classi cation layer (unlike most methods discussed above).

To achieve this, a task encoding module adapts the features for a speci c task (i.e., to the classes of interest) before the classi cation head.

Of course, these are not the only attention mechanisms existing in the FSOD literature. Some works derive other kinds of attention achieving competitive results. Kernelized FSOD (KFSOD) [START_REF] Zhang | Kernelized few-shot object detection with e cient integral aggregation[END_REF] proposes elaborated kernel functions to combine query and support features in various ways, which can be interpreted as attention. Di erently, [START_REF] Fan | Few-Shot Object Detection with Attention-RPN and Multi-Relation Detector[END_REF] trains three distinct branches that combine query and support in di erent fashions, globally, locally, and patch-to-patch. Dynamic Relevance Learning (DRL) [START_REF] Liu | Dynamic Relevance Learning for Few-Shot Object Detection[END_REF] proposes a simpler way to combine query and support features by simple point-wise operations (concatenation, multiplication, and subtraction) after global pooling.

In addition to these attention mechanisms, some works also propose additional loss functions to improve the quality of the extracted features (query and support). As an example, Transformation

Invariant FSOD (TI-FSOD) [START_REF] Li | Transformation invariant few-shot object detection[END_REF] leverages two losses to enforce robust feature extraction. These losses are implemented as a distance between original and augmented query or support features.

The same principle is also proposed for application on remote sensing images by [START_REF] Liu | Transformation-Invariant Network for Few-Shot Object Detection in Remote Sensing Images[END_REF]. Another technique is proposed by [START_REF] Jiang | Few-Shot Object Detection via Improved Classi cation Features[END_REF], which computes a regularization loss between two randomly sampled subsets of an RoI feature. This regularization enforces consistency and robustness in the feature space making the detection easier. Likewise, [START_REF] Chu | Joint Co-Attention And Co-Reconstruction Representation Learning For One-Shot Object Detection[END_REF] derives a reconstruction loss function by com-S O D puting a low-rank matrix and reconstructing the extracted query and support features. It enforces relevant latent structure and alignment between query and support features. Finally, [START_REF] Ouyang | Few-shot object detection based on positive-sample improvement[END_REF] introduces a loss function to promote orthogonality between classes in the feature space.

Meta-Learning

While Meta-Learning methods are very common for FSC, they are much rarer in the FSOD literature. It is explained easily by the di culty of the task and the complexity of meta-learning approaches. They often require training a meta-learner model for generating weights or gradient updates to a smaller classi cation network. However, detection models are much larger than the classi cation ones, which makes the meta-learning approaches impractical for FSOD. Nevertheless, the FSOD literature borrows some techniques from the meta-learning eld. In particular, most attention and metric-learning-based methods for FSOD are trained using an episodic training strategy (see Tab. 3.1). In addition, many works are presented within the meta-learning perspective because of the episodic training, here are some examples: Meta Faster R-CNN [START_REF] Han | Meta faster r-cnn: Towards accurate few-shot object detection with attentive feature alignment[END_REF], Meta R-CNN [START_REF] Wu | Meta-rcnn: Meta learning for few-shot object detection[END_REF],

Meta-DETR [START_REF] Zhang | Meta-DETR: Image-Level Few-Shot Detection with Inter-Class Correlation Exploitation[END_REF], and GenDet [START_REF] Liu | Gendet: Meta learning to generate detectors from few shots[END_REF].

Yet, there are a few attempts at solving FSOD with meta-learning approaches. MetaDet [START_REF] Wang | Meta-learning to detect rare objects[END_REF] extends Then a ne-tuning phase occurs, the meta-learner is trained to predict the weight of the detection head only provided with support examples for the novel classes. At the same time, the detector is ne-tuned on the support set, with all class-agnostic parts frozen. At test time, the meta-learner predicts weights for the novel classes to extend the detector's head and the detector can be used as a regular detector. Similarly, Sylph [START_REF] Yin | Sylph: A hypernetwork framework for incremental few-shot object detection[END_REF] applies the same idea but only to the classi cation branch of the detector, assuming that the regression is also class-agnostic.

This section draws an almost exhaustive list of the contributions to the FSOD eld (see Tab. 3.1). As for FSC, several research tracks explore FSOD. However, meta-learning is a lot less popular approach for detection compared to classi cation. Instead, attention-based methods (often trained with an episodic strategy) are the mainstream approaches. Nevertheless, there is no consensus about the best way to tackle FSOD, and ne-tuning or metric-learning contributions are often simpler and still competitive.

Few-Shot Object Detection on Remote Sensing Images

Few-Shot Object Detection is a relatively recent eld in computer vision and so far, it has been applied mostly to natural images and in particular on Pascal VOC and MS COCO datasets (Sec. 3.5 explains how they are prepared for the few-shot setting). However, there are a few contributions that apply FSOD techniques on Remote Sensing Images (RSI), these are highlighted in green in Tab. 3.1.

F S O D S
RSI are notoriously more challenging than natural images for the detection task. Objects are smaller and more numerous, they can be arbitrarily oriented, and the background is often more complex.

Therefore, object detection methods applied to RSI often comprise some tricks to better deal with the speci cities of RSI. FSOD techniques applied to RSI follow the same trend. Among these tricks, the use of multiscale features is certainly the most common. For instance, FSOD-RSI [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF] extends FRW with three levels features map to better deal with small objects. Similarly, [START_REF] Gao | A Fast and Accurate Few-Shot Detector for Objects with Fewer Pixels in Drone Image[END_REF][START_REF] Zhou | FSODS: A Lightweight Metalearning Method for Few-Shot Object Detection on SAR Images[END_REF][START_REF] Wang | Context Information Re nement for Few-Shot Object Detection in Remote Sensing Images[END_REF] leverage multiscale features for either the query image, support images or both. FSOD applied to RSI is based on attention-mechanisms [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF][START_REF] Gao | A Fast and Accurate Few-Shot Detector for Objects with Fewer Pixels in Drone Image[END_REF][START_REF] Zhou | FSODS: A Lightweight Metalearning Method for Few-Shot Object Detection on SAR Images[END_REF][START_REF] Huang | Few-Shot Object Detection on Remote Sensing Images via Shared Attention Module and Balanced Fine-Tuning Strategy[END_REF][START_REF] Chen | Few Shot Object Detection for SAR Images via Feature Enhancement and Dynamic Relationship Modeling[END_REF][START_REF] Liu | Transformation-Invariant Network for Few-Shot Object Detection in Remote Sensing Images[END_REF] or ne-tuning strategies [START_REF] Wolf | Double Head Predictor based Few-Shot Object Detection for Aerial Imagery[END_REF][START_REF] Xu | SIMPL: Generating Synthetic Overhead Imagery to Address Zero-shot and Few-Shot Detection Problems[END_REF][START_REF] Wang | Context Information Re nement for Few-Shot Object Detection in Remote Sensing Images[END_REF][START_REF] Zhou | Few-shot object detection via context-aware aggregation for remote sensing images[END_REF], but to our knowledge, there is no FSOD method based completely on a metric-learning approach.

Among these contributions, some tackle the detection problem with other modalities than visible light. Indeed, this is a problem of interest as earth observation is often conducted with non-visible light. Image quality is highly dependent on the weather conditions, and half the earth at night is unobservable with visible light. Therefore, a lot of applications rather use infrared light or Synthetic-Aperture Radar (SAR). Two articles tackle few-shot detection in SAR images [START_REF] Chen | Few Shot Object Detection for SAR Images via Feature Enhancement and Dynamic Relationship Modeling[END_REF][START_REF] Zhou | FSODS: A Lightweight Metalearning Method for Few-Shot Object Detection on SAR Images[END_REF], yet without any notable adjustment to account for the modality change.

These contributions are of particular interest for COSE as the goal is to design e cient detection methods for high-resolution images. Some extensions for the CAMELEON project are already planned with multi-spectral images and LIDAR. Hence, methods able to adapt from one modality to another are especially valuable.

Extension of the Few-Shot Object Detection Setting

As for Few-Shot Classi cation, the Few-Shot Object Detection setup has several extensions. These settings are more challenging but re ect better real-life use cases.

One-shot and Zero-shot Object Detection First, in the case of extremely limited annotations, object detection is still achievable. One-Shot OD has been addressed by several works [START_REF] Li | One-shot object detection without ne-tuning[END_REF][START_REF] Ting | One-Shot Object Detection with Co-Attention and Co-Excitation[END_REF][START_REF] Chu | Joint Co-Attention And Co-Reconstruction Representation Learning For One-Shot Object Detection[END_REF][START_REF] Zhao | Semantic-aligned fusion transformer for one-shot object detection[END_REF] that we present in the above section.

These approaches are not di erent from the few-shot setting, it simply is more di cult. However, in the zero-shot setting, it becomes even more challenging as no image example is available for the novel classes. The common approach in this setting is to leverage semantic representations from the class labels and condition the detection on this information [START_REF] Bansal | Zero-shot object detection[END_REF]. Recently introduced large language-visual models such as CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF] provide strong improvements for various zeroshot tasks and object detection is no exception. For instance, [START_REF] Han | Multimodal few-shot object detection with meta-learning based crossmodal prompting[END_REF] trains a prompt generator in a meta-learning fashion to condition the detection on novel classes. Alternatively, DINO [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] and DETReg [START_REF] Bar | Detreg: Unsupervised pretraining with region priors for object detection[END_REF] conceive strong self-supervised pre-training schemes speci cally adapted for object detection, which translate into impressive performance in a low shot setting. Finally, [START_REF] Sha N Rahman | Transductive learning for zero-shot object detection[END_REF] leverage a transductive pseudo-labeling approach to improving zero-shot detection. To our knowledge, this is the only transductive method applied to few-shot detection. S O D

Generalized and Incremental FSOD

The goal of FSOD is to adapt to novel classes; however, in many cases, the performance on base classes matters as well. This is the case in the Generalized FSOD (G-FSOD) setting where we are interested in detecting both base and novel classes. The incremental setting extends G-FSOD with several adaptations to novel classes without forgetting the previously seen classes. For G-FSOD, a naive approach is to train two detectors: one on base classes and one on all classes (base and novel), as a ne-tuned version of the rst one. Outputs from both detectors are combined at test time to achieve better performance on base and novel classes. This is adopted by [START_REF] Fan | Generalized Few-Shot Object Detection without Forgetting[END_REF] with two Faster R-CNN. However, other contributions propose more sophisticated methods. CFA [START_REF] Guirguis | CFA: Constraint-based Finetuning Approach for Generalized Few-Shot Object Detection[END_REF], for instance, proposes a regularization loss to prevent forgetting base classes. Alternatively, [START_REF] Gao | Decoupling Classi er for Boosting Few-shot Object Detection and Instance Segmentation[END_REF] duplicates the detection head to process separately the foreground and background samples and prevents classication bias toward base classes. In the incremental setting, the naive approach from [START_REF] Fan | Generalized Few-Shot Object Detection without Forgetting[END_REF] does not scale as it would require duplicating the detector each time novel classes are added. Instead, [START_REF] Perez-Rua | Incremental few-shot object detection[END_REF][START_REF] Yin | Sylph: A hypernetwork framework for incremental few-shot object detection[END_REF] train a meta-network to generate classi er weights for novel classes on-the-y unlocking convenient adaptation. Incremental DETR [START_REF] Dong | Incremental-detr: Incremental few-shot object detection via self-supervised learning[END_REF] adopts a di erent strategy based on ne-tuning and distillation to prevent forgetting previously seen classes.

Cross-domain Few-Shot Object Detection

Last but not least, Cross-Domain FSOD (CD-FSOD) tackles the few-shot object detection task in the context of domain adaptation. CD-FSOD aims at designing methods able to generalize to new kinds of images. Just as for classi cation, two sub-tasks have been explored in the literature: CD-FSOD with and without class shift. For COSE, both tasks are relevant but CD-FSOD with class shift precisely corresponds to their application. Indeed, once a system is in operation, it will likely encounter new objects and domains. While the images will always be taken from above, their general aspect may change a lot due to weather conditions, di erent landscapes or carrier altitude. Therefore, solving CD-FSOD with class shift is crucial for the CAMELEON system.

However, this eld remains barely untouched. To our knowledge, only a few contributions tackle CD-FSOD. First, without class shift, several works address this problem with augmentation-based approaches. The idea is to leverage the few target examples to augment the source images so that they become plausible samples from the target domain. For instance, [START_REF] Gao | AcroFOD: An Adaptive Method for Cross-Domain Few-Shot Object Detection[END_REF] proposes a directive data augmentation procedure that optimally augments the source examples, so their features are close to the features of the target examples. The detector is then trained as a regular detector on the augmented examples. Likewise, [START_REF] Wang | Few-shot adaptive faster r-cnn[END_REF][START_REF] Zhao | OA-FSUI2IT: A Novel Few-Shot Cross Domain Object Detection Framework with Object-Aware Few-Shot Unsupervised Image-to-Image Translation[END_REF] propose source-to-target translation networks that convert source images into target images. These networks are trained adversarially with a discriminator that aims to distinguish between domains. Closely related, Cross-Domain CutMix [START_REF] Nakamura | Few-shot Adaptive Object Detection with Cross-Domain CutMix[END_REF] crafts an augmentation technique that mixes two domains by cropping and pasting objects from the target domain into source images and vice-versa.

D F S

The methods discussed above assume that the source and target domains share the same label space, i.e., they have the same classes. This assumption allows for building simple domain augmentation approaches as once the source images are translated into the target domain, their annotations can be directly used for training. When classes shift between source and target domains, this is no longer an option. In this very challenging setting, most FSOD methods showcase poor results and adaptation capabilities. Naive ne-tuning is often the best alternative in this case. To our knowledge, there are still no contributions that tackle this task. Only two articles [START_REF] Xiong | CD-FSOD: A Benchmark for Cross-domain Few-shot Object Detection[END_REF][START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] pave the way for future research in this direction with dedicated benchmarks and baselines. Both propose to study the generalization capabilities of FSOD methods on various datasets after a large base training on COCO dataset. First, [START_REF] Xiong | CD-FSOD: A Benchmark for Cross-domain Few-shot Object Detection[END_REF] combines three datasets: 1) ArTaxOr [START_REF] Drange | Arthropod Taxonomy Orders Object Detection Dataset[END_REF], a close-up insect images dataset, 2) UODD [START_REF] Jiang | Underwater species detection using channel sharpening attention[END_REF], an underwater image dataset, and 3) DIOR. The authors propose a simple self-distillation strategy, similar to self-supervised approaches, during the ne-tuning on the new domains. Second, [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] builds a more complete benchmark called Multi-dOmain FSOD (MoF-SOD) with 10 di erent target domains. However, the authors study the in uence of two di erent source datasets COCO and LVIS [START_REF] Gupta | LVIS: A Dataset for Large Vocabulary Instance Segmentation[END_REF]. They also provide a domain distance measure that assesses the similarity between a dataset and COCO. This measure is the recall of a detector trained on COCO applied to a dataset in a class-agnostic manner. Intuitively, if a dataset is close to COCO (in terms of classes and aspects), the trained detector will detect a lot of objects (even if the classes are wrong)

and will have a high recall. Based on this similarity measure they study the impact of freezing some layers of the detector during the ne-tuning. Previous works recommend only ne-tuning the detection head while keeping the backbone frozen. However, [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] shows that this is true only for su ciently similar datasets. In other words, when the source-target gap is large, it is better to ne-tune the model entirely for better adaptation. Unfortunately, the authors did not provide an easy-to-use meta-dataset for future research and addressing CD-FSOD remains challenging due to complex initial data processing.

3.5 Dataset preparation and evaluation in the Few-Shot setting

Adapting detection datasets in the Few-Shot setting

There are no speci c datasets for Few-Shot Object Detection. Instead, regular detection datasets can be adapted to the few-shot setting. In this section, we describe this process in the case of the four datasets on which this PhD project mainly focuses: DOTA [START_REF] Xia | DOTA: A large-scale dataset for object detection in aerial images[END_REF], DIOR [START_REF] Li | Object detection in optical remote sensing images: A survey and a new benchmark[END_REF], Pascal VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF].

The conversion of a dataset for the N -ways K-shots setting is straightforward. This choice is motivated by the presence of distractors. This concept and further explanations about how they in uence the few-shot training in detection will be presented in Sec. 4.1. It is important to note that as FSOD is a recent eld, several preparation techniques coexist in the literature. However, the one described above seems the most reasonable and common in current FSOD works.

Nevertheless, the existence of various preparation settings makes the comparison with existing methods di cult as this choice is rarely discussed in the articles. It may be challenging to gure out precisely how the datasets were prepared, not to mention the choice of the training strategy (i.e., episodic or direct ne-tuning). The reported FSOD performance in the literature should then be regarded with a critical eye.

Evaluation protocol for Few-Shot Object Detection

The common practice in FSC is to randomly sample a support set from the training split of the whole dataset, adapt the model with it (through ne-tuning or direct adaptation), and nally, make the predictions and compute the relevant metrics on the test split. This is repeated many times with di erent support sets and the scores are nally averaged to give a robust evaluation of the generalization capabilities of the models.

For detection, the same principle should be applied to get a robust assessment of the models' performance. However, the adaptation of such models is often quite long compared to classi cation models. Indeed, detection models are much larger than classi cation ones, thus they take more time to adapt to novel classes. Before going further, we need to distinguish two approaches, on the one hand, the ne-tuning strategy and on the other hand all other strategies (i.e., metric learning, meta-learning and attention-based methods). The main distinction is that the latter use the support set during inference whereas ne-tuning approaches only leverage it during the second phase of training. Fig. 3.4 illustrates the two di erent approaches for FSOD model evaluation and exhibits a time estimation for training and evaluating one model, following the general recommendations of FSC (i.e., at least 100 repetitions with various support sets).

Evaluation of ne-tuning FSOD approaches

Repeated evaluation requires ne-tuning the base model (i.e., the model after base training) with various support sets. Fine-tuning FSOD methods can take up to a few hours and repeated evaluation may take days1 , which is not practical. A reasonable compromise is to perform a limited number of runs (between 10 and 30), which is su cient according to empirical studies in [START_REF] Wang | Frustratingly Simple Few-Shot Object Detection[END_REF]. Even though, robust evaluation is still an intensive process in this setting.

Evaluation of other FSOD approaches

Other methods adapt to novel classes at inference time given a support set. Therefore, they can be more robustly evaluated (in a reasonable time) than their ne-tuning counterparts. Adaptation is often fast compared to the ne-tuning phase and can be more easily repeated. However, detection models that are based on metric learning or attention still require a ne-tuning phase at least for the regression branch. A support set must be used for this as well and its choice in uences the performance of the model, even if adaptation is repeated multiple times after the ne-tuning. Yet this ne-tuning step is often even more time-consuming than basic ne-tuning approaches as the models are augmented with costly adaptation modules. In this case, the common setting is to repeat the adaptation at inference multiple times after only a single ne-tuning of the model. These settings provide a sweet spot between evaluation robustness and computation time. They will be employed in our all experiments unless speci ed, both for ne-tuning, metric learning and attention-based approaches.

Conclusion

In this chapter, we reviewed the FSOD literature. This eld is relatively recent and fastly growing.

It has signi cantly evolved since the beginning of this project. In 2020, most FSOD works were based on attention-based approaches, yet ne-tuning techniques are now getting more and more interest. This review helps to understand the main directions that have already been explored and the relevant tracks that need to be pursued.

C 4 U C F S O D A
The detection task becomes extremely challenging when limited annotated data is available.

In this chapter, we explore the reasons behind this di culty. In particular, we focus on the case of aerial images for which it is even harder to apply FSOD techniques. It turns out that small objects are especially challenging for the FSOD task and are the main source of poor performance in remote sensing images. In this chapter, we present our rst contribution to the FSOD eld. Speci cally, this section presents an analysis of the di culties of going from a regular to a few-shot data regime for the detection task, especially for aerial images. are by de nition unknown at this point. However, in this setup, the model is speci cally trained to consider these objects as background, whereas if no co-occurrences of the novel and base classes were allowed, no background supervision would be given to novel class instances. This could be achieved by removing all images containing such co-occurrences from the dataset. However, this type of distractor is much less frequent than the self-distractors (see Fig. 4.1). In addition, these mostly occur during the base training phase, and even if they provide incorrect supervision, netuning will rectify it. Therefore, we choose to keep images with co-occurrence distractors during base training.

P. Le

The Increased Challenge of Aerial Images

The di culties described in the previous section are not speci c to any kind of image. However, it appears from the scarce literature and our experiments that applying FSOD on aerial images is much more di cult than on natural ones. ). The baseline model is FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF], trained on all classes (base and novel) and with all available annotations in each dataset. Then the mAP is computed on base and novel classes separately.

At the beginning of this PhD, only very few works addressed FSOD in aerial images. Among those, Few-Shot Object Detection via Feature Re-Weighting (FRW) [START_REF] Xiao | Few-shot object detection and viewpoint estimation for objects in the wild[END_REF] and Few-Shot Object Detection With Self-Adaptive Attention Network for Remote Sensing Images (WSAAN) [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF] were the most popular.

These two works were not evaluated on the same datasets, preventing any useful comparison. In addition, many architectural choices di er from one another, for instance, the underlying detection frameworks and the backbones. Therefore, we choose to implement these methods within a single framework, preventing most architectural discrepancies. Our proposed framework will be described thoroughly in Chap. 6. We also re-implemented Dual AwareNess Attention (DANA) [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF] as it was one of the best-performing methods on COCO dataset at that time. We analyze here the behaviors of these three FSOD techniques both on natural and aerial images. The main idea is to compare the performance of the three methods in regular and few-shot data regimes. The regular data regime corresponds to the vanilla detector (i.e., without any modi cation for the few-shot setting) and with full access to the novel class annotations in the dataset (i.e., no annotation ltering).

In our re-implementation of FRW, WSAAN and DANA, the underlying detector is FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF]. Thus, The height of the rectangle is set as the mAP on either the base or novel classes (in blue and red respectively). This clearly illustrates the di erent behaviors of FSOD methods applied on aerial or natural images. It is generally irrelevant to compare the performance of a method from one dataset to another, especially with images of di erent natures. Each dataset has its own characteristics (resolution, intra-class variety, color range, etc.) and therefore a given model will not perform equally well on two distinct datasets according to a pre-de ned performance metric. Hence, we cannot compare the absolute performance of a FSOD method on Pascal VOC and DOTA and the previous extrapolation is not valid. Nevertheless, there is a pattern: FSOD methods work consistently better on natural images compared to aerial images. To understand this phenomenon, we need a way to fairly compare the FSOD performance across several datasets. To this end, we propose to look at the relative performance of the FSOD methods against the regular baseline (i.e. FCOS in our case) using the following metric: We hypothesize that this performance gap is mainly due to di erences in the object sizes within the datasets. In aerial images, objects are much smaller on average. This is already an issue for object detection: small objects are challenging to nd. The paradigm of current vision models is to have deep feature representations with increasing elds of view. The Field of View (FoV) of a speci c layer is the area in the input image that in uences the value of one location in the feature map of that layer. In deeper layers, the FoV is often quite large compared to small objects' size and object features are diluted with their irrelevant and noisy surroundings. Thus, it reduces the activation strength at the object location, and the object can easily be missed. Feature Pyramidal Networks [START_REF] Lin | Feature pyramid networks for object detection[END_REF] and various other tricks were introduced to solve this issue, as discussed in Sec.

the
RmAP = mAP FSOD -mAP Baseline mAP Baseline . ( 4

C

To support this hypothesis, we rst conduct a brief size analysis of the four datasets DOTA, DIOR, Pascal VOC and MS COCO (see Fig. 4.3). Aerial datasets contain far smaller objects than natural ones. Plus, in aerial datasets, the size of objects in di erent classes di ers a lot. Some classes contain only small objects, while others only large objects. In Pascal VOC, this class' size variance is limited.

We argue that it is more di cult for the model to extract relevant information from small support examples but also to learn more diverse features to deal with greater objects' size variance. Incidentally, this partly explains the greater di culty of MS COCO. To support this claim, we conduct a per-class performance analysis on DOTA, DIOR and Pascal VOC. The results of this comparison are available in Fig. 4.4. In this gure, the performance is reported per class against the average size of the class. The rst row reports absolute mAP values (with 0.5 IoU threshold) both for FRW and FCOS (baseline). In the second row, the mAP gap between the FRW and the baseline is plotted against the objects' size. We did not report RmAP values for the sake of visualization. RmAP can take large values (e.g. when the regular baseline mAP is low) and this squeezes the interesting part of the plot in a narrow band around 0. Larger objects are easier to detect. It is true in both data regimes, but this trend is reinforced in the few-shot regime (in the rst row, the blue trend lines are steeper than the black ones). This is observed for base classes but not always for novel classes, probably because the trends on novel classes are not reliable due to the limited number of points.

Fig. 4.5 shows a more reliable trend for novel classes when the results from the three datasets are aggregated. Finally, the few-shot methods, which leverage support information to condition the detection can surpass the baseline in some cases. All three methods here are attention-based, and therefore, bene t from having support examples available during inference to condition the detection. This would not be the case with ne-tuning approaches. However, this seems advantageous only when the objects are large. On the contrary, when the objects are small, the performance is degraded. It con rms that small objects are poor examples to condition the detection on. For novel classes; however, the performance is always below the baseline, even if the gap shrinks with larger objects. This is expected as the network only received weak supervision for these classes. This comparative analysis con rms that detecting small objects is a very di cult task in the few-shot regime.

It is hard to extract useful information from small support objects. Even worse, this information can be detrimental for the detection. Existing FSOD methods are not designed to deal with small objects, hence the application of these methods on aerial images does not yield satisfactory results.

It is therefore crucial to develop FSOD techniques that target speci cally small objects. Incidentally, we will address this point in Sec. 6.3 and in Chap. 8.

Conclusion

In this chapter, we presented our rst contribution to the FSOD eld with an analysis of the challenges raised by the few-shot regime for the detection task. These di culties are reinforced when FSOD is applied to aerial images as they contain smaller objects. This gives a clear direction for this PhD project: improving the handling of small objects in FSOD methods. To this end, we dedicate Sec. 6.3 and the entire Part III of this thesis. As a rst step into the Few-Shot Object Detection eld, we proposed a naive approach to solve the detection task in the few-shot regime. To give some context, at the beginning of this project, FSOD was a very recent domain and very few articles tackled this challenging task, especially applied to aerial images. Therefore, we took inspiration from the Object Detection and Few-Shot Classi cation literature embodied respectively by Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] and Prototypical Networks [START_REF] Snell | Prototypical Networks for Few-shot Learning[END_REF]. This chapter presents Prototypical Faster R-CNN, our rst attempt at solving FSOD. We begin by presenting the motivation behind this contribution and its main principle. Then, the training procedure and several tricks are proposed to improve training stability and detection quality. Finally, Prototypical

Faster R-CNN is applied to synthetic and aerial images to assess its generalization capabilities and understand its limitations.

Motivation and Principle

In 2020, most of the FSOD literature was focused on attention-based approaches (see Tab. Networks can adapt to novel classes without any ne-tuning. Hopefully, this property would transfer to Faster R-CNN by replacing its classi cation layer with such malleable modules. For COSE's application, this would be ideal as the detection model could adapt "on the y" at a low cost.

Prototypical Faster R-CNN for FSOD

Before explaining in detail how the prototypical networks can be embedded into Faster R-CNN, let us de ne a few notations and detail the functioning of Faster R-CNN. The backbone, RPN and detection head are respectively denoted as f , g, and h. The backbone extracts feature F q from the input -or query -image I q : f (I q ) = F q .

(5.1)

P F R CNN FSOD

The backbone extracts features at multiple scales using an FPN, but for simplicity, we regroup all these features into one notation: F q . The backbone is a ResNet-50, the FPN extracts features from 3 di erent levels with respective strides 8, 16, and 32. The RPN takes F q as input and computes both proposals boxes bi and objectness scores o i for all locations in the feature maps:

g(F q ) = ( bi , o i ) M i=1 , (5.2)
where M is the number of generated boxes. It changes with the number of anchor boxes de ned per location, in our case, it is set to 3. Hence, M is three times the number of locations in the feature map. Then, the top 1000 boxes with the highest objectness scores are selected to extract the proposals features ξ i with the RoI Align layer:

RoIAlign(F q , bi ) = ξ i . (5.3)
Finally, the detection head h outputs classi cation scores for each proposal from its features and re nes its box coordinates:

h(ξ i ) = ŷi = ( bi , li ), (5.4)
where l i ∈ [0, 1] |C|+1 is a vector of classi cation scores. There is one more element in l i than in C

because Faster R-CNN deals with background as a class.

Extending Faster R-CNN with Prototypical Networks

To replace the classi cation layer in Faster R-CNN by prototypical networks, we propose to change the output dimension of the last layer in the classi cation branches of both the RPN and the head.

That way, instead of producing a classi cation (or objectness) score per box, these networks output embedding vectors. Each vector represents the information contained inside the corresponding box. We denote these embedding vectors of the RPN and the classi cation head z RPN i and z head i respectively. Their dimension is set to 128 (z • i ∈ R 128 ) and is kept xed in all our experiments. Hence, the outputs of the RPN and the detection head become: extract its features F c k and then the example features are extracted with RoI Align:

g(F q ) = ( bi , z RPN i ) , (5.5)
h(ξ i ) = ( bi , z head i ). ( 5
z RPN k,c = g(F c k ), (5.7)
Φ c k = RoIAlign(z RPN k,c , b c k ), (5.8)
z head k,c = h(F c k ), (5.9)
Ψ c k = RoIAlign(z head k,c , b c k ). ( 5
Φ c = 1 K K k=1 Φ c k , (5.11)
Ψ c = 1 K K k=1 Ψ c k .
(5.12)

F R CNN FSOD

For each proposal, we compute the classi cation score for class c as the likelihood of the region in the input image representing an object of class c. To do so, we suppose that the class distributions over the embedding space are Gaussian distributions centered on the class prototypes. Hence, the classi cation score of the proposal i for class c is:

lc i = p(z head i |c) = exp -d(z head i , Ψ c ) 2 2σ 2 , (5.13)
where d is a distance measure over the representation space, in our experiments, d is the Euclidean distance. Note that in our case, the embeddings are normalized after their computation, therefore the Euclidean distance is equivalent to the Cosine Similarity. σ is the standard deviation of the distribution and is set to 0.5 in our experiments. In Faster R-CNN, the background is considered as a class as well, the corresponding score can be derived from the other class scores as follows:

l∅ i = p(z head i |∅) = 1 -max c∈C lc i , (5.14)
where ∅ denotes the background class.

In the RPN the objectness computation is very similar to the classi cation score in the head. However, only two classes are considered: foreground and background. The foreground class is seen as a mixture of Gaussians (i.e., a mixture of all foreground classes) and is approximated as the maximum score among all classes for stability reasons:

ôi = max c∈C lc i .
(5.15)

These modi cations make Faster R-CNN able to adapt to novel classes. Computing prototypes for novel classes allows direct adaptation of the whole detection model and not simply the detection head as in RepMet. However, with these changes, the model also requires a di erent training scheme to ensure that the prototypes are properly leveraged and classes are not only memorized.

Training Procedure

Before presenting the changes with the Faster R-CNN training procedure, we present here what remains unchanged: the loss functions and the example selection. Faster R-CNN is trained using four distinct loss functions, two for the RPN and two for the detection head:

L RPN reg (b RPN i , bRPN i ) = SmoothL1Loss(b RPN i , bRPN i ), (5.16
) Once the base training is done, the network can directly be applied to novel classes through direct adaptation from the prototypes (see Fig. 3.4). However, the adaptation is only performed in the classi cation parts of the model, regression branches are not modi ed. This is certainly sub-optimal and therefore, we provide a ne-tuning scheme to remedy this. This ne-tuning is done exactly as the base training phase, in an episodic manner except that the episode classes are sampled from

L RPN obj (o i , ôi ) = ôi log(o i ) + (1 -ôi ) log(1 -o i ), (5.17)
L head reg (b head i , bhead i ) = SmoothL1Loss(b head j , bhead j ), (5.18)
L head cls (c i , ĉi ) = -log(l c i ), ( 5

Iterative improvements

PFRCNN, as described in the previous section, denoted the baseline, does not perform well on aerial images (see Tab. 5.1). Therefore, we introduce a series of improvements to improve the performance of the model.

In this section, we propose a series of improvements on top of the PFRCNN baseline described above. Indeed, when tested on aerial images, vanilla PFRCNN yields relatively poor performance (see baseline performance in Tab. 5.1). To remedy this, we introduce several training tricks.

Hard negative example mining

Moving average prototypes

Another issue with the baseline is that the prototypes can change abruptly, either when the network is updated or when the support set changes. We argue that this causes some training instabilities.

To prevent such rapid modi cation of the prototypes, an exponential moving average is introduced to smooth the disruption. Hence,

Φc t+1 = αΦ c t + (1 -α) Φc t .
α is set to 0.1 in our experiments. Φc t is the averaged prototype for class c at iteration t, while Φ c t is the prototype computed from the support set, for class c at iteration t.

Background clustering

Lastly, the baseline shows a poor separation of novel class representations (see Fig. 5.5). This leads to poor performance with novel classes at test time. In order to solve this, inspiration is drawn from F M L FSOD [START_REF] Caron | Deep clustering for unsupervised learning of visual features[END_REF]. At each iteration, they t a K-means on the learned representations. This gives pseudo-labels to train the network for classi cation in a self-supervised manner. Similarly, we propose to t a Kmeans on the negative embeddings (i.e. representing boxes not matched by any ground truth object).

From the resulting pseudo-labels a contrastive loss function (Triplet Loss [START_REF] Ho | Deep metric learning using triplet network[END_REF]) is computed. The triplets are formed with embeddings labeled identically by the K-means. It encourages the network to organize the negative examples into tight and separated clusters. This will eventually discover semantic clusters that represent novel objects.

Ablation study

In order to assess the relevance of the tricks formulated in previous paragraphs, a small ablation study is conducted on DOTA dataset. The results of this analysis can be found in Tab. 5.1. On the one hand, the introduction of hard examples mining and moving average prototypes improves consistently the novel classes mAP in the one-shot setting. On the other hand, background clustering greatly reduces the performance on base classes, while achieving similar results on novel classes.

According to this analysis, we chose to keep only hard example mining and the moving average as it combines the best base and novel classes performance.

Performance on Arti cial Data

Before applying PFRCNN on aerial images, we test it on an arti cial dataset with reduced di culty.

This gives a hint about the capacities of the model on real data.

MNIST-LOC Dataset

As an arti cial dataset, we leveraged MNIST-LOC. This dataset is not a published work but rather a toy example sometimes mentioned in the literature. It consists in creating arti cial images with the handwritten digit images from the original MNIST dataset [START_REF] Deng | The mnist database of handwritten digit images for machine learning research[END_REF]. For each image in MNIST-LOC, a random number of MNIST digits are sampled and placed randomly in the image with a random scale.

This creates a potentially in nite dataset but with limited variability. For our experiments, we build a dataset with 20k images in the training split and 2k images both for the test and validation splits. Compared to a real dataset, MNIST-LOC is far more simple. The background is uniform which simpli es the localization of the objects. Then, the class occurrences are uniformly distributed. And nally, the intra-class variance is reduced as MNIST is an easy dataset for the classi cation task.

Implementation details

We provide here some of the implementation details for training Prototypical Faster R-CNN, but a complete list of the hyperparameters and their values is available in our GitHub 1 . The optimization is done with Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] and a learning rate of 1e-4. The backbone network is pre-trained on ImageNet and its rst two layers are kept frozen during training. Three classes are selected as novel classes and are reserved for evaluation, the 7 others are kept as base classes. Each episode is constituted of K query = 5 images per class, i.e., 15 images per episode.

Detection performance on MNIST-LOC

We present the performance results on MNIST-LOC in Tab. First, it can be seen from this table that the performance in a regular data regime (i.e., vanilla Faster R-CNN with all annotations) is high. This con rms that MNIST-LOC is a fairly simple dataset and that the detection task is way easier on this dataset than on real ones. It is important to note that these values cannot be directly compared with the performance values in the few-shot regime as the number of classes is di erent. In the regular regime, the classi cation problem has 10 classes whereas, in the few-shot regime, it only has three (3-ways K shots setting, even for base classes).

Then, the few-shot performance of PFRCNN on base classes is also quite high, approaching one as the number of shots grows. However, for novel classes, this is di erent, the mAP values are way lower in this case and fall below an acceptable threshold for any industrial use case. To get a better grasp on these results, Fig.

Di culties on Aerial Images

While the Prototypical Faster R-CNN is challenged on synthetic images, it has more serious di culties with real images. In this section, we present the detection result of PFRCNN applied on aerial images, speci cally on DOTA and DIOR datasets. Nonetheless, these experiments are not useless and provide relevant insights about the FSOD task and its di culties. For instance, with the MNIST-LOC dataset, almost no di erence could be seen between splits. With DOTA, much better performance is achieved on Split B than Split A. It indicates some interactions between classes, some combinations are more di cult than others. These considerations were not taken into account in the design of PFRCNN and should be overcome to achieve reasonable few-shot detection.

Despite its limited performance, Prototypical Faster R-CNN is one of the rst approaches to tackle FSOD on remote sensing images from a metric learning perspective. In addition, this method does not need any ne-tuning. All previous results were given from a simple adaptation to the novel classes at inference time with novel prototypes. We also experimented with an additional ne- and that makes them di cult to distinguish. To better understand why this confusion happens, we investigate the embedding space of PFRCNN through TSNE visualization (see Fig. 5.5). This gure is made by collecting the embedding vectors of all proposals over an entire query set, and then by reducing their dimension using the TSNE algorithm [START_REF] Van Der Maaten | Visualizing data using t-SNE[END_REF]. Class-speci c clusters are well-formed in the representation space, but some classes overlap which explains the confusion. Representations of these classes may be close to another class prototype and get misclassi ed. This is especially true for novel classes which overlap over base classes, explaining their poor performance. For the example above, the misclassi cation of the two water tanks is easily understood from the TSNE plot as these two classes almost perfectly overlap (class storage-tank in dark green and roundabout in pink).

Insights and conclusion

From the results presented above, one question arises: is representation learning a suitable choice for few-shot object detection? Metric-learning methods are competitive with state-of-the-art for few-shot classi cation but seem less appropriate for FSOD. Prototypical Faster R-CNN is a rst attempt to apply prototypical networks to FSOD. The very few FSOD approaches based on Metric- To conclude, this chapter presents Prototypical Faster R-CNN, a fully metric-learning-based approach for the Few-Shot Object Detection task. This is one of the rst methods proposed in this category. Despite its relatively poor performance on real images, it can adapt to novel classes without any ne-tuning, which is still a rare property in the current literature. Finally, the experiments conducted with PFRCNN provide relevant insights about the FSOD task and will help in the design of future approaches.

C 6

A F F FSOD C

A Fair comparison is extremely challenging in the Few-Shot Object Detection task as plenty of architectural choices di er from one method to another. Attention-based approaches are no exception, and it is di cult to assess which mechanisms are the most e cient for FSOD. In this chapter, we propose a highly modular framework to implement existing techniques and design new ones. It allows for xing all hyperparameters except for the choice of the attention mechanism. Hence, a fair comparison between various mechanisms can be made. Using the framework, we also propose a novel attention mechanism speci cally designed for small objects.

Ƽ P. Le Jeune and A. Mokraoui, "A Comparative Attention

Framework Presentation and Motivation

As seen in Chap. 5, metric-learning approaches are not the optimal choice for the FSOD task. The

Alignment Attention Fusion Framework

In Sec. The AAF framework takes as input the features from the query image F q as well as the features extracted from every support image F c s for c ∈ C (if more than one example is available per class -K > 1 -, their features are averaged). It outputs class-speci c query features M c q in which features relative to class c are highlighted. The framework is divided into three parts as shown in Fig. 6.2, which provides an overview of the framework. Each component is described below in dedicated sections with examples of possible design choices. Even though this framework is presented from the perspective of object detection, it could be applied to any kind of few-shot visual tasks (e.g., classi cation or segmentation).

Query-Support Alignment

The alignment module, denoted Λ, spatially aligns the features from the query and the support.

It is unlikely that objects of the same class appear at the same position inside query and support F F FSOD C Figure 6.2: The Alignment Attention Fusion (AAF) framework is composed of three components: spatial alignment Λ, global attention Γ and a fusion layer Ω. Examples for each module are depicted, these come from FSOD methods in the literature. Ex. A is presented in [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF], Ex. B in [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF] and Ex.

C in [START_REF] Liu | Dynamic Relevance Learning for Few-Shot Object Detection[END_REF]. The colors chosen in this diagram match the colors used in Eqs. (6.1) to (6.5).

images (in comparison to the classi cation task where objects to classify are in the center of the image). This issue is commonly avoided by pooling the support map and using it as a class-speci c reweighting vector. But as discussed in Sec. 3.2.3, this trick loses the spatial information about the support object, which can be detrimental for detection. Instead, an alignment based on similarity can be done between query and support feature maps. The idea is to re-organize one feature map by comparing it with the other so that similar features are spatially close in the maps (see Fig. 3.

3).

The alignment module is de ned as follows:

A c q = λ q (F q , F c s ) F q , (6.1)

A c s = λ s (F q , F c s ) F c s . (6.2) Query Features ∈ R n×d Support Features for class c ∈ R m×d A nity matrices ∈ R n×m

Aligned features

The de nition of the matrices λ q and λ s determines how features are aligned. They are mostly derived from a similarity measure between query and support features. This formulation is close to the non-local blocks described in [START_REF] Wang | Non-local neural networks[END_REF] and is at the heart of visual transformers [START_REF] Vaswani | Attention is all you need[END_REF]. Transformers attention can be understood as an alignment of the value to match the query-key similarity. This formulation allows easy implementation of di erent feature alignments by changing the expression of the a nity matrices. As an example, Meta Faster R-CNN [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF] leverages an alignment module with a nity matrices λ s (F q , F c s ) = F q • (F q s ) T and λ q (F q , F c s ) = I (see Example A in Fig. 6.2). Only the support features are aligned so that they match query features. It is important to mention that alignment alone does not combine query and support features. It rather reorganizes spatially the query or support features. However, once the features are aligned, their sizes match, which allows direct comparison through element-wise operations (within the fusion layer).

A

A F F

Global Attention

The global attention module, denoted Γ, combines global information of the supports and the query.

It highlights class-speci c features and softens irrelevant information for the task. This module is de ned as follows:

H c q = γ q (A c q , A c s ), (6.3)
H c s = γ s (A c q , A c s ). (6.4)

Global attention operators Highlighted features

The global attention operators γ q and γ s combine the global information from their inputs and highlight features accordingly. This is generally done through channel-wise multiplication. In this way, class-speci c features are highlighted, while features not relevant to the class are softened.

Changing the de nition of γ q and γ s allows the implementation of a wide variety of global attention mechanisms. This technique largely resembles the principle of the Learnet [START_REF] Bertinetto | Learning feed-forward one-shot learners[END_REF] introduced for FSC. For instance, reference [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF] pools the support maps with a global max pooling operation (GP) into a reweighting vector and reweights the query features through channel-wise multiplication:

γ q (A c q , A c s ) = A c q GP (A c s ) and γ s (A c q , A c s ) = A c s (see Example B in Fig. 6.2).

Fusion Layer

The purpose of the fusion component is to combine highlighted query and support maps. This is only applicable when the maps have the same spatial dimension. It is mostly used alongside the alignment module as it does not combine the information from the support and the query but only reorganizes the maps. In particular, when support and query maps do not have the same spatial dimension, aligning support maps with query maps xes the size discrepancy. The fusion module is then de ned as follows:

M c q = Ω (H c q , H c s ). (6.5)

Fusion operator Merged query features

The highlighted maps can be combined through any point-wise operation, addition ⊕, multiplication , subtraction , concatenation [•, •] or more sophisticated ones. An example of such a fusion module is presented in [START_REF] Liu | Dynamic Relevance Learning for Few-Shot Object Detection[END_REF]. The fusion operator concatenates the results of the addition and the subtraction of the highlighted features: 6.2). The point-wise operators can also contain small trainable models such as in [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF], where small CNNs (e.g., ψ dot , ψ sub , and ψ cat ) are applied after the point-wise operators, but before the concatenation:

Ω(H c q , H c s ) = [H c q ⊕ H c s , H c q H c s ] (see Example C in Fig.
Ω(H c q , H c s ) = [ψ dot (H c q H c s ), ψ sub (H c q H c s ), ψ cat ([H c q , H c s ])].
Except for the fusion layer which must be applied last, spatial alignment and global attention can be applied in any order. This exibility is required to cover methods that apply global attention before alignment, such as DANA [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF]. The whole architecture of the AAF framework is illustrated in Fig. 6.2, in which examples from the previous sections are also depicted. The modular structure of F F FSOD C the framework enables the implementation of a wide variety of attention mechanisms while keeping all other hyperparameters xed. In this way, it is a useful tool for FSOD methods comparison.

Implementation details

Before presenting the results of a fair comparison between several FSOD approaches re-implemented in the AAF framework, we must review its implementation. To make the comparisons fair, some design choices are kept xed in the framework. The backbone is a ResNet-50 with a 3-layers Feature Pyramid Network on top. It extracts features at 3 di erent levels, which helps the network to detect objects of various sizes. The detection head is based on FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF], a one-stage detector (a choice motivated by insights from Chap. 5). The head is composed of a few convolutional layers with ReLU activations followed by two branches (convolutional as well) for classi cation and regression respectively. The AAF framework is applied between the backbone and the detection head. As features are extracted at multiple levels, attention mechanisms are also implemented to work at di erent scales. This may di er from the original implementations, but most methods are already designed to work at multiscale (see Tab.

Augmentations and Cropping Strategies

Some existing works leverage sophisticated training strategies (e.g. auxiliary loss functions [START_REF] Fan | Few-shot object detection with attention-RPN and multi-relation detector[END_REF],

hard examples mining [START_REF] Zhang | Hallucination improves few-shot object detection[END_REF] or multiscale training [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF]). While this certainly improves the quality of the detections, it introduces new parameters to tune and makes the comparison with other works di cult. As the focus of this study is on attention mechanisms, we choose not to reimplement all these improvements. However, to remain competitive with existing works, we propose a novel augmentation pipeline speci cally designed for object detection. It is de ned in the next paragraph which includes a cumulative study of its di erent components on DOTA. In addition, we discuss the choice of the support extraction strategy. Basically, this refers to how the support examples are 6.2 A A F F extracted from the support images since most parts of these images are irrelevant for the task. From our analysis, it seems that this design choice signi cantly in uences the model performance (see Tab. 6.2). However, it is barely discussed in the FSOD literature. We nd that the best strategy is to crop the support example and resize it to a xed-size patch. This strategy is thus xed for all our experiments.

Augmentations

To improve the performance of the methods implemented in the AAF framework and be competitive with existing works, we propose an augmentation pipeline speci cally designed for detection.

Some regular augmentation techniques cannot be directly applied for object detection as they can completely mask objects from the image. This deteriorates the training as the model will not be able to detect hidden objects, but it will be penalized anyway.

First, we apply random horizontal and vertical ips (only for aerial images) and color jitter. As it does not remove entire objects, these can be applied directly to the images. However, some other classical techniques such as random crop-resize and random cut-out cannot be applied directly. Therefore, we developed object-preserving random crop-resize and cut-out. The main idea is to apply these transformations at the object level and not at the image level. This ensures that objects of interest are still visible in the transformed image. For crop-resize, a non-empty subset of the objects in the image is randomly sampled. An overall bounding box is computed around all these objects and the cropped area is randomly drawn between this box and the image borders. Hence, it guarantees the presence of at least one object inside the cropped image. For cut-out, the principle is similar, instead of cutting out a random part of the image, the cut is applied at the object level so that it does not hide out entire objects. Fig. 6.3 compares the two proposed augmentations with their naive implementations.

We performed a cumulative study to assess the bene ts of each component of the augmentation pipeline. To do so, we implemented Feature ReWeighting (FRW) [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF], a well-known FSOD technique, within the AAF framework. FRW is then trained on DOTA dataset. This experiment is summarized in Tab. 6.1. It shows that the augmentation is bene cial for the performance on novel classes but detrimental for base classes. Surprisingly, performance drops on base classes with augmentation. More speci cally, it seems that image ips are responsible for the performance loss on base classes (see rst and second columns in Tab. 6.1). Base classes performance drops when adding ips but remains mostly constant when adding other types of augmentations. One crucial di erence between ips and other augmentations is that we choose to apply ips also on support examples.

This choice was made to increase the diversity of the support set during ne-tuning. For novel classes, only a few images are available as support during ne-tuning, and we want to avoid over tting these examples. Although other types of augmentations could have been employed for this, we wanted to avoid disrupting too much of the information in the support. This choice may be the cause of the performance drop on base classes. To verify this hypothesis, we conduct a few more experi-F F FSOD C Figure 6.3: Di erence between naive augmentation techniques (left) and our adaptation to object detection (right). The proposed transformations are applied at the object-level to preserve objects integrity.

ments disabling the ip in the support set. With the default cropping strategy (see next paragraph), the experiments con rm the hypothesis: no performance drop is observed when supports are not ipped. However, support augmentation certainly interacts with the support cropping strategy, thus we also tried with the same-size cropping strategy. Surprisingly, it does not produce similar results, and in this case, ipping support examples is actually bene cial for base classes performance. This suggests a complex interaction between augmentation on the support set and the cropping strategy.

The choice made in our experiments may not be optimal in this regard, and a deeper analysis of this interaction should be conducted in future work (e.g., studying the e ect of various augmentations on a synthetic dataset to have better control over the images). Finally, the base class performance loss is compensated by clear improvements on novel classes. As this is the main goal of FSOD, we choose to adopt the original augmentation pipeline, including ips in the support set, for all our experiments. Other augmentations are not applied to the support set to prevent disrupting their representations and therefore the conditioning of the model. Table 6.1: Cumulative study of the proposed augmentation techniques on DOTA using FRW [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF]. mAP 0.5 is reported for di erent number of shots.

Support example cropping strategy

The support information is only located inside the example bounding box. The remaining part of the support image mostly contains irrelevant information concerning the object class. Therefore, extracting features from the whole support images is not necessary. But features contained only inside the object's bounding box might not be su cient as well. Context can be extremely valuable in certain cases, especially for small objects. For instance, a car and a small boat could easily be mistaken without context. Close surroundings of the objects can help for recognition.

A common strategy for support information extraction is proposed by [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF]. They concatenate in the channel direction the support image with the support object's binary mask (rectangular, computed from the bounding box) and pass this to an extractor network. This has two main drawbacks. First, it is necessary to compute features from the entire support image, which is a loss of resources. Second, the same network cannot be used for extracting features in query and support images as it needs an additional input channel to process the mask. Hence, the network cannot be pretrained beforehand.

This design choice is rarely discussed, if ever mentioned, in the literature.

In this section, we explore this design choice by implementing several extraction strategies. We did not reimplement the technique from [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF] as it requires to have two di erent networks for -Context-padding: the cropping occurs exactly as with the default strategy, but areas around the objects are masked out. This is close to what was proposed by [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF].

-Re ection: context is replaced by re ection of the object. In the case of small objects, the support patch can easily be dominated either by irrelevant information or by zeros when using the latter two extraction methods. F F FSOD C Figure 6.4: Illustration of the di erent cropping strategies tested. The mixed strategy is not illustrated as it is a combination of default and same-size.

-Same-size: all objects are resized to ll the support patch entirely (preserving the aspect ratio). It does not change the process for large objects, but it prevents smaller objects from being dominated by irrelevant information.

-Multi-scale: the object is resized and cropped at 3 di erent scales. Each scale is responsible for matching small, medium and large objects in query images.

-Mixed: it is a combination of the default strategy and same-size. Small objects (i.e.

√ wh < 32)

are extracted using the default strategy. Larger objects ( √ wh ≥ 32) are resized into a patch of 128 × 128 pixels. Therefore, small objects are not upscaled more than 4 times, as they are using the resize strategy.

These strategies are compared in Table 6 strategy does not perform very well as it introduces small objects features which seems detrimental for the good conditioning of the network. On the contrary, same-size only generates large objects as support which is a better strategy. Finally, re ection performs surprisingly well for small objects while preserving their small size. The redundancy generated by the re ection of such small objects certainly reinforces the object's features.

Fair comparison of Few-Shot Object Detection Methods with AAF

To showcase the exibility of the proposed AAF framework, we reimplement and compare multiple existing works. Some methods described in Chap. 3 are selected: FRW [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF], WSAAN [START_REF] Xiao | Few-Shot Object Detection With Self-Adaptive Attention Network for Remote Sensing Images[END_REF],

DANA [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF], Meta R-CNN [START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF] and DRL [START_REF] Liu | Dynamic Relevance Learning for Few-Shot Object Detection[END_REF] (see Tab. course, some details di er from the original implementations, but the purpose of this comparison is to compare only the query-support combination module. In particular, the backbone and the training strategy (losses and episode tasks) may di er. We rst conduct such a comparative experiment on Pascal VOC [START_REF] Everingham | The pascal visual object classes (voc) challenge[END_REF] and MS COCO [START_REF] Lin | Microsoft coco: Common objects in context[END_REF] datasets. On these datasets, the performance achieved by our implementations is close (i.e., within 2 points of mAP 0.5 ) to the values reported in the original papers. Then, we use the framework to compare the performance of some methods on two aerial datasets: DOTA [START_REF] Xia | DOTA: A large-scale dataset for object detection in aerial images[END_REF] and DIOR [START_REF] Li | Object detection in optical remote sensing images: A survey and a new benchmark[END_REF].

Evaluation protocol

The evaluation is also conducted in an episodic manner, following recommendations from [START_REF] Huang | A Survey of Self-Supervised and Few-Shot Object Detection[END_REF]. Such a class similarity makes the novel class detection much easier. Some authors do leverage this fact, as for instance [START_REF] Cao | Few-Shot Object Detection via Association and DIscrimination[END_REF] which rst associates a base class to each novel class before learning to discriminate between them.

The behavior just described is expected from any few-shot object detection method. Moreover, performance values are close to what is reported in the original papers of the reimplemented methods.

These results are not the same as many architectural choices di er from the proposed methods (e.g. backbone, class splits, etc.). Nevertheless, it con rms that the proposed AAF framework is exible enough to implement a wide variety of attention mechanisms. It is therefore an appropriate tool to compare and design query-support attention mechanisms.

DRL is arguably the simplest method among the ve selected as it leverages only a fusion layer.

It combines query features with the features of each support image through concatenation and point-wise operations, creating class-speci c query features. It is therefore the closest to the regular Table 6.4: Performance comparison between WSAAN [START_REF] Xiao | Few-Shot Object Detection With Self-Adaptive Attention Network for Remote Sensing Images[END_REF] and DANA [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF] on MS COCO. mAP 0.5:0.95 (MS COCO mAP, with IoU thresholds ranging from 0.5 to 0.95) and mAP 0.5 values are reported for base and novel classes separately and for di erent numbers of shots: K ∈ {1, 5, 10, 30}.

MS COCO. MS COCO is more di cult, therefore, the information extracted from the supports better helps the models. Finally, WSAAN outperforms DANA on Pascal VOC but performs slightly worse on MS COCO. It can be noted that the results obtained on a dataset cannot be extrapolated to another without taking into account the characteristics of the datasets. A method that performs best on a dataset is not guaranteed to do so on another dataset. This reinforces the need of a exible framework that allows fair and easy comparison between FSOD methods. That way, the best-performing method can be easily selected for a given problem. Without such a framework, it is di cult to nd out from the literature which method is the most promising for a given application as most FSOD works focus on natural images. For COSE, this framework is highly valuable as it will serve as an objective comparison tool for attention-based FSOD methods.

From these experiments on natural images, it seems clear that DANA performs best. Therefore, it highlights the importance of feature alignment for query-support matching. Global attention loses spatial information in support features which is detrimental to detection. However, global attention methods should not be overlooked. WSAAN shows impressive performance and even outperforms slightly DANA on Pascal VOC. It could be interesting to combine both methods, but this does not seem to be trivial as demonstrated by the results of Meta R-CNN, which leverages the alignment from DANA and the attention from FRW, but does not yield better results.

Aerial images

To our knowledge, very few works evaluate FSOD methods on aerial images at the time we proposed the AAF framework. Among those we select FSOD-RSI [START_REF] Xiao | Few-shot object detection and viewpoint estimation for objects in the wild[END_REF], which simply applies FRW to aerial images (we will refer to it as FRW), WSAAN [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF] and our PFRCNN. In addition, we include DANA inside this comparison as it was the best-performing technique on natural images. All these methods are evaluated on di erent datasets, making their performance comparison challenging. Using the proposed AAF framework, we compare the performance of these methods on both DOTA and DIOR. These methods are reimplemented inside the framework and all other design choices are xed during the experiments (as described in Sec. 6.2.4). Tab. 6.5 regroups the results of the comparison.

These results show a slight improvement over the state-of-the-art on DIOR (WSAAN [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF]). Our implementation of WSAAN outperforms (8 mAP 0.5 points on base classes and 0.4 on novel classes) the result reported in the original paper. However, the attention mechanism employed in WSAAN is not optimal for aerial images. WSAAN is outperformed by both FRW and DANA. While this was expected for DANA in the light of results from Sec. 6.2.5.2, it was not for FRW. The superiority of DANA over the other methods on DOTA and DIOR is clear and coherent with the results on natural images. The more sophisticated attention mechanism, in particular the alignment, from DANA is better at extracting and leveraging the information from the support examples. Hence, the detection performance is higher. It is particularly bene cial for small numbers of shots: the extracted information is semantically robust.

These results con rm the analysis conducted in Chap. 4, the performance gap between the classical baseline (i.e. FCOS) and the few-shot approaches is larger on aerial images. On natural images, the performance drop between the few-shot approach and the regular baseline is nearly inexistent for base classes and around 25% for novel classes. On aerial images these drops are largely increased: ∼ 15% and ∼ 50% for base and novel classes respectively. This can be guessed from Tabs. 6.3 and 6.5, but detailed gaps are provided in Tab. 6.6. Following the analysis from Chap. 4, the main reason behind this performance gap between natural and aerial images is the size of the objects in F F FSOD C the image. Small objects are much more di cult to detect and also are poor representatives of a semantic class as they contain little information. Therefore, the most sensible direction to pursue is to design new attention mechanisms speci cally built for small objects. As it happens, this will be discussed in the next section.

Cross-Scales Query-Support Alignment for Small FSOD

From the analysis in Chap. 4 and the previous section, it is clear that a new attention mechanism speci cally designed for small objects is required to get reasonable performance on aerial images.

To this end, we propose a novel alignment method that combines information from multiple scales:

Cross-Scales Query-Support Alignment (XQSA). This di ers from existing methods which often work independently at di erent scales. Conversely, XQSA combines the information from various scales and sources (i.e., query and support images). Its original motivation is to unlock matching support examples with query objects belonging to the same class even though their sizes di er. With existing methods, this was prohibited as same-class objects with di erent sizes have non-similar features and are not matched by similarity-based attention mechanisms.

XQSA de nition

In this section, we detail the functioning of our proposed Cross-Scales Query-Support Alignment module. First, features are extracted from the query and support images with a backbone network f . In our implementation, f is a ResNet-50 with an FPN attached. It outputs feature maps at three distinct resolution levels:

{F q,0 , F q,1 , F q,2 } = f (I q ), (6.6)

{F c s,0 , F c s,1 , F c s,2 } = f (I c s ). (6.7)
All query features F q,i ∈ R w q,i ×h q,i ×d , for i ∈ {0, 1, 2} (i.e. from di erent levels) are attened and concatenated into a unique representation F q ∈ R nq×d , with n q = i w q,i h q,i . Here, w q,i and h q,i 6.3 C S S A S FSOD denote the size of the query feature map at level i, this size depends on the query image size and the stride of the corresponding level in the backbone. The same operation is performed for all support features F c s,i . When more than one shot is available per class, support features are average at each level i:

F c s,i = 1 K K k=1 F c,k s,i . (6.8)
Then, following the ViT paradigm, the support and query features are linearly projected into the queries, keys and values matrices Q, K and V . Speci cally, the query features are used to produce the queries while keys and values are computed from the support features:

Q = F q W Q = [ F q,0 , F q,1 , F q,2 ] W Q , (6.9
)

K c = F c s W K = [ F c s,0 , F c s,1 , F c s,2 ] W K , (6.10
)

V c = F c s W V = [ F c s,0 , F c s,1 , F c s,2 ] W V , (6.11)
Per level features

Concatenated multiscale features

Learned projection matrices where W Q , W K and W V are learnable projection matrices, which are implemented as linear layers in practice. From this, an a nity matrix is computed between the queries and the keys, and then the aligned support features A c s are computed as:

λ c s = Softmax( QK cT √ d
), (6.12)

A c s = λ c s V c . (6.13)
For completeness with the de nition of the AAF framework, λ q = I, meaning that the query features are not modi ed. The aligned features are nally processed by a two-layer MLP with skip connections. LayerNorm [START_REF] Lei Ba | Layer normalization[END_REF] is applied before alignment and the MLP. These supplementary computations can be seen as fusion operations in the AAF framework, similar to what was proposed in [START_REF] Li | One-shot object detection without ne-tuning[END_REF][START_REF] Yan | Meta r-cnn: Towards general solver for instance-level low-shot learning[END_REF][START_REF] Wu | Universal-Prototype Enhancing for Few-Shot Object Detection[END_REF]. This resembles the ViT attention, but with a major di erence, it combines features from di erent images and di erent levels (see Fig. 6.7). This allows better object matching when there are size discrepancies between support and query images.

Small objects have a limited footprint in feature maps which make them hard to detect but also hard to match with support examples. XQSA's multiscale alignment enhances the chances of matching as each query feature is compared with support features at all scales. Finally, in order to fairly compare XQSA with DANA, we leverage their BackGround Attenuation block (BGA) on the support features before alignment. They conduct a thorough ablation study which shows the positive impact of BGA on the few-shot performance of their method. We also carry out an ablation study about our cross-F F FSOD C Figure 6.7: Diagram illustrating the proposed cross-scales query-support alignment method. Features are extracted from the query and support images at multiple scales and combined to form an a nity matrix. For each query feature position, the a nity is computed with any position in the support features. This allows object matching across di erent feature levels.

scales method (see Tab. 6.7) and nd that BGA also improves performance in this case. XQSA is implemented inside the AAF framework, split into three modules: alignment, attention and fusion, following the description from Sec. 6.2. BGA is implemented within the global attention block, which can handle as well self-attention module as well and is applied before alignment.

Ablation study XQSA

To con rm the bene ts of each component of our attention methods, we conduct a brief ablation experiment, adding separately the di erent modules of our proposed attention mechanism. The ablation is conducted on DOTA and the results are available in Tab.

Application to aerial and natural images

To assess the capabilities of the proposed method, we compare it with the best methods from Sec. 6. Hence, the methods can be compared speci cally on small objects. XQSA performs consistently better on small and medium novel objects, compared with FRW and DANA. This performance gain comes at the expense of slightly lower detection quality for larger objects. In XQSA, the shallow query features are compared to all support features (i.e., not only shallow support maps). As deeper maps are smaller, this increases moderately the number of potential detections for small objects. However, for large objects, the deep query feature map is compared with all support maps, including the shallow ones. It increases a lot the number of potential matches between query and support features (see an illustration of this phenomenon in Fig. 6.8). For large objects that are already well detected, this mostly adds wrong matches and deteriorates the performance. For small objects, however, this is useful as very few correct matches are found by current FSOD methods. A potential solution for this issue would be to down-weight the contributions of shallower features in the a nity matrix's bottom rows (i.e., the left and bottom blocks of the matrix). The a nity matrix could even be made upper triangular to avoid taking into account the contributions of shallower levels at all. For similar reasons, XQSA demonstrates a slight drop in base classes performance. However, the actual goal of few-shot learning is to maximize performance on novel classes. The large number of available examples during base training is enough to learn robust query-support matching even for small objects. However, our goal here is to improve the generalization capabilities of the model on novel objects. The performance on base classes is simply a safety check and relates more to the Generalized FSOD problem (see Sec. existing works for aerial images. On DIOR, this corresponds to a 10 mAP point increase compared to previous state-of-the-art [START_REF] Li | Few-Shot Object Detection on Remote Sensing Images[END_REF]. However, this is not su cient to ll the performance gap with natural images as presented in Fig. 6.9. This gure extends Fig. 4.2 with XQSA results. While XQSA improves upon other methods on aerial images, it is still far behind the performance of the regular baseline. XQSA is better for small and medium objects but at the cost of lower performance on large objects and base classes. Progress is still required to get more versatile FSOD solutions able to handle small, medium, and large objects at the same time.

In complement to Tab. 6.8, we also provide the comparison between XQSA, DANA and FRW with mAP 0.5:0.95 metric and on the four datasets. These results are provided in Tab. 6.9. mAP 0.5:0.95 is a more demanding metric for object detection. It is especially hard for small objects as a few pixels shift from ground truth can greatly reduce the IoU and therefore lead to a missed detection. This intensi es as the IoU threshold increases in the mAP computation.

For DOTA and DIOR, results with mAP 0.5:0.95 are in agreement with results from Tab. 6.8 (i.e. with mAP 0.5 ). However, XQSA does not perform better than DANA on Pascal VOC and MS COCO novel classes with mAP 0.5:0.95 . This is mainly due to the metric being too strict on small objects. This questions the soundness of these metrics for FSOD, especially when dealing with small objects. We will tackle this question in Chap. 8. F F FSOD C

XQSA Implementation Challenges and Extensions

XQSA is inspired by the ViT and resembles some FSOD techniques that leverage transformer attention as well. It is well-known that these mechanisms are computationally heavy and scale quadratically in terms of the number of locations in the feature maps. Here, it is slightly di erent as we combine the features from the support and query images. Support images are 4× smaller than query images (see Sec. 6.2.4.1), therefore the complexity of the attention module is greatly reduced. According to the notations introduced in Sec. 6.3.1, the complexity of the XQSA alignment block is O(n q n s ). As n s is way smaller than n q (roughly 16 times), this is way better than a self-attention mechanism which would be O(n 2 q ). However, with XQSA, as features from all scales are concatenated, this remains computationally heavy, especially as this process must be repeated for each support image. Unfortunately, this does not scale well as we increase the number of support examples. Time complexity rapidly becomes prohibitive, but memory complexity is more limiting. The gradients of the Softmax used for the computation of the similarity matrices are extremely large (O(n q n 2 s )) and do not t on GPU memory when the number of support examples increases. To bypass these limitations, we propose several tricks.

-Pytorch manual gradient computation: automatic di erentiation in Pytorch is not always optimal. When successive computations involve the same gradients, Pytorch often computes and stores them separately, wasting precious resources. To this end, we re-implemented the XQSA block as a self-contained operation, with custom gradients computation to prevent duplicated gradients. This results in slight memory and performance gains, but it is still not enough to scale e ciently with the number of support examples.

-Pytorch gradient checkpointing: Pytorch has an API to checkpoint the gradients during the backward pass. It copies back the gradients on the CPU memory to prevent over owing the GPU memory. It solves the out-of-memory issues, but makes the training much slower.

-Deformable XQSA block: The alignment block is expensive due to the comparison between all query feature locations and all support locations. Thus, we extend the XQSA block with a deformable attention mechanism, inspired from ConViT [START_REF] Stéphane D'ascoli | Convit: Improving vision transformers with soft convolutional inductive biases[END_REF]. Speci cally, we introduce an inductive bias inside the attention module by adding a layer that selects the locations that will be compared between support and query feature maps. This resembles Deformable Convolutions [START_REF] Dai | Deformable convolutional networks[END_REF] and Deformable-DETR [START_REF] Zhu | Deformable {DETR}: Deformable Transformers for End-to-End Object Detection[END_REF]. While this was solving both the memory over ow and training slowness, we were not able to achieve reasonable detection performance with it.

-Support class averaging: while this solution seems sub-optimal, it saves a lot of time and memory by avoiding a lot of computation. Of course, it does not completely solve the scaling issues (e.g., as the number of support classes increases). However, it allows adding a large number of support images per class without any issues and performs well on the FSOD task.

Finally, we only keep the support class averaging as it is the simplest and best-performing alternative tested.

Qualitative Comparison within the AAF framework

In this section, we provide a qualitative comparison of the four attention-based methods that we compared previously in Fig. 6.9: FRW, WSAAN, DANA and ours XQSA. To get a fair comparison, we sampled the same support examples for each method and performed the detection on the same query images. For convenience, we split the comparison in two, rst on base classes and then on novel classes. In both cases, we provide the support examples used for the detection in the rst gure and the detection on a handful of images in the second gure. These comparisons are visible in Figs. [START_REF] Lin | Microsoft coco: Common objects in context[END_REF].10 and 6.11 for base classes and in Figs. [START_REF] Lin | Microsoft coco: Common objects in context[END_REF].12 and 6.13 for novel classes.

Base Classes Detection Quality

From Fig. 6.11, it is quite di cult to assess which method is superior to the other. It seems quite obvious that DANA and XQSA perform slightly better than FRW and WSAAN. However, there are cases where neither DANA nor XQSA is the best (see the second row for instance). This is in line with the quantitative results from the previous sections. The gap between these methods on base classes is tight, it is therefore quite di cult to correctly assess the quality of these methods from only a handful of examples. It is worth noting that these techniques work quite well on the base classes. Of course, there are some false positives, but the number of false negatives is limited, which is quite important for intelligence applications.

Novel Classes Detection Quality

With novel classes, the performance di erences are more visible than with base classes. First, it can be seen that XQSA produces less false positives which indicates a higher accuracy. Then, XQSA also provides more small detection than other methods. It sometimes gives false positives, but overall it improves the detection of small targets (see the last row for instance). Interestingly, XQSA and features with the query features and detect only entire objects. It seems that this kind of matching is much more robust than the global attention alone (FRW and WSAAN) as they showcase many more false positives overall. Of course, the detection of small objects is still very challenging in the FSOD setting which explains the relatively poor detection quality in these images. It is not easy to objectively determine which method is the best at this, but a slight advantage seems in favor of XQSA, con rming quantitative results.

Conclusion

In this chapter, we have introduced a highly modular framework for implementing attention-based FSOD methods. First, this framework allows fair comparison between the various attention mechanisms proposed in the literature. From our analysis, it seems that spatial alignment is crucial to achieving high-quality FSOD, mostly because it does not lose the spatial information contained in the support examples. Secondly, the AAF framework is a practical tool to design new attention mechanisms. For that matter, we developed a novel cross-scales alignment layer within the framework to speci cally increase the detection performance on the small objects. The so-called XQSA alignment allows us to achieve large improvements compared to the contemporary literature on several datasets. It works especially well on aerial images as they contain smaller and more objects than natural images. Speci cally, XQSA outperforms the state-of-the-art on DOTA and DIOR datasets at that time.

Nevertheless, the attention-based methods are not completely satisfactory from an industrial perspective. While they achieve reasonable performance on aerial images, they have some disadvantages. First, even if they can adapt to novel classes from a few support examples at test time, they still require extensive ne-tuning to perform correctly. This ne-tuning can take up to several hours, which is not convenient for "on-the-y" adaptation. Then, the episodic training strategy is somewhat cumbersome and generates unrealistic scenarios. Indeed, during each episode, query images are sampled so that they contain at least one instance of one of the episode classes. In real-case applications, no object of interest can be visible in an image which makes the detection task more challenging. But most importantly, only the episode classes are detected during the episode, the detection task is, therefore, simpler as it only classi es objects among a smaller number of classes.

Given these drawbacks, we investigate in the next chapter FSOD methods that do not employ the episodic training strategy and therefore solely rely on a ne-tuning scheme. F F FSOD C This chapter focuses on the last major approach for FSOD: ne-tuning. Based on Di usionDet, a recent detection framework leveraging di usion models, we build a simple but e cient ne-tuning strategy. The resulting method, called FSDi usionDet, achieves state-of-the-art FSOD on aerial datasets and competitive performance on natural images. Extensive experimental studies explore the design choices of the ne-tuning strategy to better understand the key components required to achieve such quality. Finally, these impressive results allow considering more complex settings such as cross-domain scenarios, which are especially relevant for COSE. Cross-Domain FSOD task. We emphasize that the rst two sections of this chapter present existing works in the literature, while the last three sections discuss our contributions: the FSDi usionDet strategy, thorough experimental analysis of the strategy on several datasets, and its application in Cross-Domain scenarios.

Di usion Probabilistic Models Principle

Di usion Probabilistic Models have been introduced in 2015 by Sohl-Dickstein et al. [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF]. Their principle is simple, to approximate a complex and intractable probability distribution, they model a di usion process from the original distribution to a normal distribution as gradual Gaussian noise addition. Then, the goal is to nd the reverse process to approximate the original distribution by iterative denoising. In this section, we present the main concepts of the DPM [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF] and their recent advances in generative tasks, mostly led by Denoising Di usion Probabilistic Models (DDPM) [START_REF] Ho | Denoising di usion probabilistic models[END_REF].

Forward and Reverse Di usion Processes

First, let's introduce a few notations. Our objective is to be able to e ciently sample elements x from a distribution P. When P is an arbitrary distribution, its Probability Density Function (PDF) is intractable and sampling often relies on expensive Monte Carlo techniques. Here, we suppose that there exists a random process q that transforms P into a normal distribution:

q(x) ∼ N (0, I). (7.1)
This is called the Di usion Process and refers to the eponym physical phenomenon. The main hypothesis is to assume that this process is a Markov Chain that adds Gaussian noise progressively:

x T = q(x 1:T |x 0 ), (7.2)

7.1 D P M P where x 0 ∼ P is an element sampled from the original distribution and x T ∼ N (0, I) is sampled from a normal distribution. T denotes here the number of steps in the Markov chain and q(x 1:T )

represents the joint distribution of variables x 1 to x T . The di usion process adds Gaussian noise iteratively and therefore is de ned as:

q(x 1:T |x 0 ) := T t=1 q(x t |x t-1 ) with q(x t |x t-1 ) := N ( 1 -β t x t , β t I). (7.
3)

The β t denote the variance schedule, i.e., the amount of gaussian noise added at each step. For convenience, we also de ne α t and ᾱt :

α t = 1 -β t , (7.4
)

ᾱt = t s=1 α s . (7.5)
q is called the forward di usion process as it transforms x 0 into noise (this is true only asymptotically when T → ∞). Thus, we can write for 1 ≤ t ≤ T :

x t = √ α t x t-1 + √ 1 -α t t-1 , (7.6)
= √ α t α t-1 x t-2 + 1 -α t α t-1 t-2 , (7.7)
= . . .

= √ ᾱt x 0 + √ 1 -ᾱt 0 , (7.8)
where the i (0 ≤ i ≤ t -1) are sampled from a normal distribution.

The reverse process instead transforms Gaussian noise into elements sampled from P. It is also a

Markov chain with Gaussian transitions (this is ensured for su ciently small β t ):

q(x 0:T ) := T t=1 q(x t-1 |x t ) with q(x t-1 |x t ) := N (µ t , Σ t ). (7.9)
Unfortunately, the reverse process, or rather µ t and Σ t , are highly intractable and cannot be easily estimated. However, we can approximate this process with a parametrized model p θ :

p θ (x 0:T ) := p(x T ) T t=1 p θ (x t |x t-1 ) with p θ (x t |x t-1 ) := N (µ θ (x t , t), Σ θ (x t , t)). (7.10)
Hence, if we nd an optimal set of parameters θ so that the model is able to approximate the reverse process expectation µ t and variance Σ t from variable x t and timestep t, then the reverse process can be computed. Now, we must derive e cient ways to estimate the reverse process. S D D F T One solution is to leverage deep neural networks which are well suited for these kinds of tasks.

We will explain how such models can be trained to approximate the reverse process conditional probabilities q(x t-1 |x t ). Note that the reverse process is initialized with a normal distribution: p(x T ) = N (0, I). The forward and reverse di usion processes are illustrated in Fig. 7.1, in the context of image denoising. Before jumping to the next section to see how we can estimate such models, we can observe that the reverse process distribution is tractable when conditioned on x 0 , it will be useful for training:

q(x t-1 | x t , x 0 ) := N (μ t (x t , x 0 ), βt I), (7.11) with μt (x t , x 0 ) = √ ᾱt-1 β t 1 -ᾱt x 0 + √ α t (1 -ᾱt-1 )
1 -ᾱt x t , (7.12)

and βt = 1 -αt-1 1 -αt β t . ( 7

DDPM Training and Sampling

Now that we have introduced the forward and reverse di usion processes, we can dive into the core of DDPM and see how we can train models to approximate the reverse process. The overall goal is to maximize the log-likelihood of the modeled data distribution p θ (x 0 ):

θ * = arg max θ log(p θ (x 0 )). (7.14)
Unfortunately, the log-likelihood is not easily optimizable and several tricks are required to get a more tractable objective. First, [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF] makes use of the well-known Evidence Lower BOund [START_REF] Diederik | Auto-encoding variational bayes[END_REF] (ELBO), which is lower bound to log-likelihood objective and more easily computable. In practice, -log(p θ (x 1:T )) = -log p θ (x 0:T )dx 1:T , (7.15)

= -log p θ (x 0:T ) q(x 1:T |x 0 ) q(x 1:T |x 0 ) dx 1:T , (7.16)
= -log E q p θ (x 0:T ) q(x 1:T |x 0 ) , (7.17)

≤ E q -log p θ (x 0:T ) q(x 1:T |x 0 ) = L ELBO . (7.18)
Now, using the de nition of the process p θ (x 0:T ) and q(x 1:T |x 0 ), and Bayes' rule, we can split the objective for each timestep t and make it tractable:

L ELBO = E q -log p(x T ) q(x T |x 0 ) + t>1 -log p θ (x t-1 |xt) q(xt|x t-1 ,x 0 ) -log p θ (x 0 |x 1 ) . ( 7

.19)

Objective for timestep T , LT

Objective for timestep t -1, Lt-1 L0
The previous equation is only valid because q(x 0:T ) if a Markov chain and because the Markov Property states that q(x t |x t-1 , x 0 ) = q(x t |x t-1 ). This is crucial, otherwise, the Bayes' rule introduces intractable terms (q(x t ) and q(x t-1 )). We refer the reader to Appendix A from [START_REF] Ho | Denoising di usion probabilistic models[END_REF] and Appendix B from [START_REF] Sohl-Dickstein | Deep unsupervised learning using nonequilibrium thermodynamics[END_REF] for the detailed derivation of Eq. (7.19). Finally, the terms highlighted in gray and blue in the above Eq. ( 7. [START_REF] Vaillant | Original approach for the localisation of objects in images[END_REF]) can be interpreted as Kullback-Leiber Divergence terms:

L ELBO = D KL (q(x T |x 0 )||p(x T )) + t>1 D KL (q(x t |x t-1 , x 0 )||p θ (x t-1 |x t )) + E q [-log p θ (x 0 |x 1 )] . (7.20) LT Lt-1 L0
These KL divergence terms are easy to compute as they compare only Gaussian distributions. This gives an easy-to-optimize upper bound to train the di usion models. Di usion models are meant to approximate the reverse di usion process. One way to achieve this with neural network models is to use the reparametrization trick introduced in [START_REF] Diederik | Auto-encoding variational bayes[END_REF]. The idea is to train a neural network to output the mean and variance parameters of a Gaussian distribution and sample elements from the estimated distribution. That way, the gradients can be computed through the stochastic sampling operation. Here, we speci cally learn two models able to predict the mean and variance parameters µ θ (x t , t) and Σ θ (x t , t) conditioned on x t and timestep t. These two estimators are trained following the ELBO objective which compares the estimated and reverse process distributions at each timestep.

In practice, this is achieved by randomly sampling a timestep and optimizing the model with the corresponding loss function. S D D F T

To stabilize the training, authors from [START_REF] Ho | Denoising di usion probabilistic models[END_REF] propose to x Σ θ (x t , t) = β 2 t I and introduce a few simpli cations in the loss. First, as the loss only involve KL divergence between Gaussians, it can be written analytically:

L t = E t,x 0 , 1 2 Σ θ 2 2 μt (x t , x 0 ) -µ θ (x t , t) 2 .
(7.21)

Another simpli cation follows from observing that μt (x t , x 0 ) and µ θ (x t , t) can be re-written as:

μt (x t , x 0 ) = 1 √ α t x t - 1 -α t √ 1 -ᾱt t , (7.22
)

µ θ (x t , t) = 1 √ α t x t - 1 -α t √ 1 -ᾱt θ (x t , t) . (7.23)
This follows from Eqs. (7.8) and (7.11). Given this expression of µ θ (x t , t), it is only necessary for the model to estimate θ (x t , t), the amount of gaussian noise added to x t-1 to produce x t . This explains why di usion models are especially well suited for denoising applications. Therefore, the loss can be further simpli ed as (using Eq. (7.8)):

L simple t = E t,x 0 , t -θ (x t , t) 2 , (7.24)
= E t,x 0 , t -θ ( √ ᾱt x 0 + √ 1 -ᾱt t , t) 2 . (7.25)
Note that the scaling term has been omitted from the last equation as the authors from [START_REF] Ho | Denoising di usion probabilistic models[END_REF] obtained better results without it. The training procedure can be summarized in Algorithm 1. Finally, once the model is trained, the sampling can be done from random noise and repeatedly applying the reverse process. Using the model to estimate the noise added at each time step, we have:

x t-1 = 1 √ α t x t - 1 -α t √ 1 -ᾱt θ (x t , t) . (7.26)
The sampling procedure is then de ned in Algorithm 2. Please note that the formalism employed in this section is identical to the one used in [START_REF] Ho | Denoising di usion probabilistic models[END_REF]. We could have referred the reader directly to this article, but it seemed essential to recall the basic principles of di usion models. For completeness, we also cite the excellent blogpost [START_REF] Weng | What are di usion models?[END_REF] from which we drew some inspiration for the two previous sections.

Algorithm 1 Di usion Training procedure

while not converged do x 0 ∼ q(x 0 ) t ∼ Uniform({1, ..., T }) ∼ N (0, I) Take a gradient descent step on ∇ θθ ( √ ᾱt x 0 + √ 1 -ᾱt , t) 2 end while x T ∼ N (0, I) for t = T, ..., 1 do z ∼ N (0, I)

x t-1 = 1 √ αt x t -1-αt √ 1-ᾱt θ (x t , t) + β 2 t z end for return x 0 7.1.

Recent Advances with Di usion Models

In the previous sections, we have presented the di usion models and how they can be trained to learn complex data distributions. It was originally leveraged for image generation in [START_REF] Ho | Denoising di usion probabilistic models[END_REF], which samples 2D random noise and progressively generates a sensible image. Plenty of consecutive works have done the same with various improvements. In DDPM, the authors employ a denoising U-Net to estimate the noise at each time step. This U-Net is replaced with visual transformers in recent di usion models [START_REF] Rombach | High-resolution image synthesis with latent di usion models[END_REF]. With the above formulation, the sampling is expensive as it requires iterative application of the denoising model to the whole image. Instead, Latent Di usion Models (LDM) [START_REF] Rombach | High-resolution image synthesis with latent di usion models[END_REF] proposes to apply the di usion process to the latent space to greatly reduce sampling time.

The authors leverage an encoder-decoder scheme to map the image space onto the latent space and back. In addition, their formulation is well suited for latent manipulation and conditioning the generation process with additional information such as text, other images, layout, etc. Other approaches speed-up DMs with improved sampling such as strided sampling schedule [START_REF] Quinn | Improved denoising di usion probabilistic models[END_REF], ODEbased sampling [START_REF] Lu | Dpm-solver: A fast ode solver for di usion probabilistic model sampling in around 10 steps[END_REF][START_REF] Lu | Dpm-solver++: Fast solver for guided sampling of di usion probabilistic models[END_REF], and careful variance scheduling [START_REF] Kong | On fast sampling of di usion probabilistic models[END_REF]. Alternatively, some contributions reconsider the denoising di usion process and leverage other corruption processes such as blurring [START_REF] Hoogeboom | Blurring di usion models[END_REF] or masking [START_REF] Daras | Soft di usion: Score matching for general corruptions[END_REF]. Another approach is to leverage non-Markovian di usion process with for instance Denoising Di usion Implicit Models (DDIM) [START_REF] Song | Denoising di usion implicit models[END_REF]. Similarly to LDM, [START_REF] Ho | Cascaded Di usion Models for High Fidelity Image Generation[END_REF] derives a cascaded framework to scale up the generated image size. With these iterative improvements, DMs largely outperform the state-of-the-art image generation in terms of quality. Up to now, this eld was mostly dominated by GANs (e.g., [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF][START_REF] Karras | Analyzing and Improving the Image Quality of StyleGAN[END_REF]). GANs run faster, but the gap is getting smaller and DMs overcome the GANs main issues: lack of diversity, training instabilities and mode collapse.

These techniques recently got a lot of attention out of the computer vision eld with their association with Large-Language-Models (LLMs) (e.g., CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF], GPT [START_REF] Brown | Language models are few-shot learners[END_REF] or T5 [START_REF] El | Exploring the limits of transfer learning with a uni ed text-to-text transformer[END_REF]). These models are referred to as Visual Language Models (VLM), and combine the rich semantic latent space of LLMs with image representation to perform text-to-image generation. They are embodied by Dall-E [START_REF] Ramesh | Zero-shot text-to-image generation[END_REF],

Flamingo [START_REF] Alayrac | Flamingo: a visual language model for few-shot learning[END_REF] and Imagen [START_REF] Saharia | Photorealistic text-to-image di usion models with deep language understanding[END_REF], among others. These models are able to generate almost indistinguishable images (at least for the human eye) in an extremely controllable way. It is great for plenty of applications, including for creative purposes. However, it also has a large societal impact as such models can easily be misused (e.g., for deepfake generation) and are subject to questionable biases.

As an example for the previous claims, Fig. 7.2 provides a few examples of real and fake images generated with various VLMs, guessing correctly which pictures are fake is quite challenging. S D D F T Figure 7.2: Examples of real and fake images generated with di usion approaches. We encourage the reader to guess which images are real and which are fake. We provide the list of answers in a footnote 1 on the next page to prevent any con rmation bias.

D D O D

Of course, image generation is not the only task that can be handled with di usion models. Generation of all kinds of modalities can be performed with DMs: music [START_REF] Mittal | Symbolic music generation with di usion models[END_REF][START_REF] Liu | Di singer: Singing voice synthesis via shallow di usion mechanism[END_REF], voice [START_REF] Huang | Prodi : Progressive fast di usion model for high-quality text-tospeech[END_REF], text [START_REF] Lovelace | Latent Di usion for Language Generation[END_REF], time series [START_REF] Li | Generative time series forecasting with di usion, denoise, and disentanglement[END_REF] or graphs [START_REF] Vignac | DiGress: Discrete Denoising di usion for graph generation[END_REF]. In addition, DMs can also be leveraged for non-generative tasks, such as image translation [START_REF] Sasaki | Unit-ddpm: Unpaired image translation with denoising di usion probabilistic models[END_REF], inverse image problems [START_REF] Rombach | High-resolution image synthesis with latent di usion models[END_REF] (application to inpainting), 3D

modelling [START_REF] Karnewar | HOLODIFFUSION: Training a 3D Di usion Model Using 2D Images[END_REF]. Last but not least, they can also be used for predictive tasks such as segmentation [START_REF] Rahman | Ambiguous Medical Image Segmentation using Di usion Models[END_REF], and, of course, detection with Di usionDet [START_REF] Chen | Di usiondet: Di usion model for object detection[END_REF]. Thus, the next section will be dedicated to explaining the principle behind Di usionDet.

Di usionDet for Object Detection

Di usionDet [START_REF] Chen | Di usiondet: Di usion model for object detection[END_REF] is a recently proposed model for object detection. It tackles the OD task using a generative approach instead of seeing it as a regression task. The latter predicts the box coordinates from the input image while the former generates the box coordinates conditioned on the image.

The di erence is subtle, but not seeing the detection as a regression problem unlocks new designs.

The main idea of Di usionDet is to apply the di usion principle to the box generation. Random boxes are rst sampled, and a model is trained to re ne iteratively the size and position of the boxes so that they localize the objects in the input image, this is illustrated in Fig. 7.3. Speci cally, the boxes are iteratively denoised by the model. The di usion process considered here is the same as in Denoising Di usion Implicit Models (DDIM) [START_REF] Song | Denoising di usion implicit models[END_REF], which as mentioned in the previous section, proposes a non-Markovian forward process that leads to the same objective as DDPM. The non-Markovian property of the novel di usion allows for much faster denoising. DDIM sampling is then leveraged in Di usionDet to iteratively denoise the boxes.

Speci cally, the denoising part of Di usionDet is a lightweight hybrid network, it consists of a self-attention layer (transformer-like) followed by a dynamic layer (called an Instance Interaction layer). The di usion/detection head is nally split into two branches, one for classi cation and one for regression. Both branches are implemented as small MLPs. The input to the head is computed from the input images features extracted with a backbone network. The backbone is a ResNet-50 with a three level FPN attached on top. Before being fed to the detection head, object features are pooled from the entire feature map with RoI Align module. The detection head processes object features independently, but the Instance Interaction layer enables interactions between instances.

The detection head is applied iteratively to re ne the bounding boxes. The initial bounding boxes are sampled randomly from a normal distribution. The regression branch of the head is trained to predict the noise between the true boxes and the current boxes. After each iteration, the boxes are updated following the DDIM sampling strategy. Only a small number of iterations is required to get satisfactory boxes (the original paper provides experiments with between 1 and 8 iterations).

A renewal process also replaces boxes with small con dence scores with random ones after each Di usionDet has a hybrid structure, it only has one stage, yet it does not predict boxes and labels densely as common one-stage detectors. The box denoising formulation allows for replacing the rst stage (i.e., the RPN) with a much more naive approach: random box sampling. The iterative renement of the boxes is able to compensate for the poor initial positioning of the boxes. In a sense, Di usionDet resembles two-stage detection frameworks that leverage iterative bounding box regression such as CRAFT [START_REF] Yang | Craft objects from images[END_REF], Object Detection via Multi-region model [START_REF] Gidaris | Object detection via a multi-region and semantic segmentation-aware cnn model[END_REF], or Cascade R-CNN [START_REF] Cai | Cascade r-cnn: Delving into high quality object detection[END_REF]. The main di erence between these models and Di usionDet is the direct box prediction.

Instead of outputting re ned box location, Di usionDet is trained to predict the shift between the current boxes and the corresponding ground truth. This does not seem signi cant, but it is much more adapted to the iterative regression procedure. First iterative methods propose using the same detection head repeatedly to get better and better boxes. However, the head is trained directly to output correct boxes, no matter how o they are in the rst place. Di usionDet instead conditions the model with the timestep embedding so that it knows how much noise should be removed from the boxes. With this trick, it can reuse the same head without any issues. Conversely, Cascade R-CNN makes use of decoupled heads for each iteration to account for di erent re nement magnitudes; however, it signi cantly increases the model size.

Few-Shot Di usionDet

Now that we have reviewed the basic principles of di usion models and presented Di usionDet we can see how it can be leveraged for FSOD. In this chapter, we propose an adaptation of Di usion-Det in the few-shot setting, based on ne-tuning. Fine-tuning has become increasingly popular in FSOD throughout the last two years (see Tab. 3.1). Simple ne-tuning strategies are now competitive with elaborated attention mechanisms. Another motivation for trying a ne-tuning approach is to study techniques from the three main directions in FSOD. Chaps. 5 and 6 respectively focus on metric-learning and attention-based methods. The last kind of FSOD approach in the literature relies on ne-tuning. While such a strategy is not very innovative, this is one of the rst applications of a di usion-based approach to a few-shot predictive task. In addition to the ne-tuning strategy, we propose a transductive inference scheme to boost the performance of the ne-tuned model. However, these are only preliminary work and do not yield the expected results yet. Finally, we also investigate an attention-based extension of Di usionDet, while promising on paper the experimental study demonstrates poor results.

Fine-tuning Strategy for Few-Shot Di usionDet

We present in this section the ne-tuning strategy that we propose to adapt Di usionDet to the few-shot regime:

1. Train Di usionDet in a regular fashion on a dataset containing only examples of the base classes.

2. Once base training is done, replace the classi cation layer with a randomly initialized layer with as many output neurons as the number of novel classes. S D D F T 3. Freeze the entire backbone and let only the detection head be updatable. This choice is not optimal and will be discussed in the next section, however, we present here the baseline conguration.

4. Reset the learning rate scheduler, so it goes again through a warmup phase. The scheduling is a simple linear warmup starting at 1 1000 of the base Learning Rate (LR) and linearly increasing up to the base LR value during 1000 iterations.

5.

Fine-tune the model with K images for each novel class. All instances of the novel classes are kept while instances from other classes are discarded. This corresponds to the distractor-free sampling scheme discussed in Sec. 4.1.

As our goal is not to tackle the Generalized Few-Shot setting, we are mostly interested in the performance on novel classes. Of course, one might want to detect base classes as well, in this case, it is possible to keep a version of the model after base training and leverage it for base classes detection.

Of course, it would require twice as much time to perform the inference to detect both base and novel classes, but this is a mild compromise compared to common issues raised by G-FSOD.

Fine-tuning is part of most FSOD methods as the adaptation of the regression part of the models cannot be easily done on the y (conversely to the classi cation part). However, ne-tuning attention-based or metric learning models is often quite long in comparison with "simple" netuning strategies which directly ne-tune object detectors on the support set without expensive additional components (e.g., a query-support attention block). This makes the ne-tuning faster and unlocks much quicker iterations and experiments. Nevertheless, ne-tuning approaches cannot be adapted at inference time and therefore, it is di cult to measure the robustness to various support examples. Thus, multiple ne-tunings are required to get a relevant evaluation of a model, otherwise, the randomness of the support set can introduce some variance and the comparison is less reliable. In practice, ne-tuning with di erent support does not signi cantly change the performance of FSDi usionDet.

While the proposed strategy is fairly simple, it yields impressive results. We provide in Tab. 7.1 a comparison between the FSDi usionDet baseline and the discussed methods from previous chapters.

It outperforms largely the metric-learning and attention-based methods on aerial images. On natural images, the gains are reduced but FSDi usionDet is still superior (especially for MS COCO where the problem is now a 20-ways detection problem and not a 5-ways task as in previous chapters). A detailed analysis is conducted in the next section to understand why it performs so well and how it can be improved further.

Another advantage of FSDi usionDet compared with attention methods is its memory e ciency. Indeed, query-support combination blocks and support embedding models require a lot of memory while training and often scale linearly with the number of classes (N ) and the number of shots K.

FSDi usionDet is not limited by the number of shots and therefore, we can explore much higher shot settings than with metric-learning or attention-based methods. Tab. performance on our four datasets of interest. The base class performance is not reported here as they do not depend on the number of shots, they can be found in the last row of Tab. 7.1. One can observe a smooth increase in performance with the number of shots with a plateau above 50 shots. To better visualize this trend and compare it with attention-based methods studied in Chap. 6, we plot in Fig. 7.5 the performance against the number of shots. From this, it can be seen that the performance is much lower in the one-shot setting with FSDi usionDet compared to attention-based approaches.

However, FSDi usionDet quickly catches up and outperforms largely other methods in higher shots settings. In addition, we can observe a much quicker increase in performance as the number of shots increases with FSDi usionDet. This is a highly desirable property in an industrial application because this means that the model has more potential for improvements. On the contrary, attentionbased approaches do not display such a strong trend, they are better suited for extremely low-shot regimes, but become less e ective with higher shots.

Experimental Study of FSDi usionDet

In the previous section, we have presented an e cient ne-tuning strategy for Di usionDet along with an analysis of its few-shot performance on several datasets. However, this is only a baseline for FSDi usionDet and its performance can be improved further. conducting more experiments, which was too expensive with metric-learning and attention-based methods. Thus, we present in this section a series of experiments that we conducted to explore the capabilities of FSDi usionDet but also to answer more general questions about FSOD.

Backbone Weights Initialization

First, in classi cation, it is now well-known that using self-supervised pre-trained backbones often boost a lot the few-shot performance of a method [START_REF] Chen | A simple framework for contrastive learning of visual representations[END_REF][START_REF] Gidaris | Boosting few-shot visual learning with self-supervision[END_REF][START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF][START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]. While plenty of studies show this for classi cation, for detection, the transferability of the learned features is not so evident. As a matter of fact, a few contributions actually show that using such backbones is sub-optimal. For instance, InsLoc [START_REF] Yang | Instance localization for self-supervised detection pretraining[END_REF] and SoCO [START_REF] Wei | Aligning pretraining for detection via object-level contrastive learning[END_REF] propose object-level self-supervised techniques and prove empirically that image-level SSL is not optimal and in some cases can even be detrimental to the detection task. In this section, we study empirically the in uence of using SSL pre-trained weights for the backbone initialization, before base training and ne-tuning. We do not consider the recent object-level techniques and instead leverage four di erent initialization strategies for the backbone weights:

-Scratch: weights are randomly initialized.

-ImageNet: weights are initialized from a ResNet-50 trained in a supervised manner for Ima-geNet classi cation.

-DINO: weights from the DINO [START_REF] Caron | Emerging properties in self-supervised vision transformers[END_REF] pretraining on ImageNet.

-CLIP: weights taken from the CLIP [START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]

Plasticity Analysis

The number of parameters frozen in the model is sometimes called the plasticity of the model in the continual learning eld, but this concept may also be useful in the few-shot setting. For simplicity, we measure the plasticity of the model as the ratio between the number of trainable parameters over the total number of parameters. Plasticity close to 1 means that the model is malleable and could learn new complex tasks. However, when it is close to 0, the model can barely change and learning new tasks may be di cult.

The plasticity is commonly explored in ne-tuning strategies for few-shot tasks. The underlying principle is that the task is learned during base training, and ne-tuning is only used to adapt the task to novel classes. Hence, the behavior of the entire model should not change dramatically. Therefore, the plasticity of the models is often quite low in the FS literature. In practice, the early stages of the model are kept frozen while only the deeper layers are trained. This trick is well-motivated as it drastically reduces the capacity of the model and thus prevents over tting, which is particularly severe in low-shot regimes. In addition, it also reduces catastrophic forgetting, which can be quite a challenge in G-FSOD.

However, this may be inadequate for the detection task. As the detection is primarily a problem of nding what is and what is not an object of interest, the backbone is trained as a feature lter.

Features from classes of interest are highlighted while others are faded out. Conversely, for classication, backbones are not required to learn such a ltering process as all classes are "of interest", and there is always one object of interest in the image. In addition, some recent experiments [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] about freezing settings in cross-domain scenarios show that improved performance is achieved with increased plasticity. In this work, the authors only study three freezing settings: ne-tuning only the last layer (as proposed in TFA [START_REF] Wang | Frustratingly Simple Few-Shot Object Detection[END_REF]), ne-tuning only the detection head (proposed in FSCE [START_REF] Sun | FSCE: Few-shot object detection via contrastive proposal encoding[END_REF]), and ne-tuning the whole model (their proposition). Here, we investigate the freezing setting in a more detailed manner with several intermediary setups, but the main di erence is that we conduct this analysis on the same image domain:

-Fine-Tune last layer only: ne-tune only the last regression and classi cation layers.

-Fine-Tune head only: ne-tune only the detection head.

-Up to stage i: Freeze backbone up to stage number i (i ∈ 1, 5 as ResNets have 5 stages).

-Fine-Tune whole: ne-tune the whole model.

-Bias only: ne-tune only the backbone biases.

-BatchNorm only: ne-tune only the backbone BatchNorm parameters.

The results of this comparison can be found in Tab. 7.4. It reports the mAP with di erent freezing strategies on DOTA, DIOR, Pascal VOC and MS COCO. Additionally, the plasticity rate is reported for each freezing strategy. Two distinct behaviors are observed here. First, on DOTA, as the plasticity increases, the FSOD performance increases as well. On DIOR, Pascal VOC and MS COCO, lower plasticity is optimal ( ne-tuning the detection head and the last stage of the backbone). Therefore, the ne-tuning strategy cannot be set once for all datasets. di ers in DOTA from the other datasets. As the task and the images remain similar between base training and ne-tuning, the only source of variability comes from the class splits. Our hypothesis here is that base and novel classes in DOTA are less compatible (i.e., less alike) than in the other datasets. For Pascal VOC, we brie y discuss this aspect in Sec. 6.2.5.2, where we observed surprisingly high mAP for the novel class sheep as the class horse was in the base set. A more quantitative way of measuring the compatibility between the base and novel class sets would be required to draw reliable conclusions about this. We are currently working on this.

In addition, we observe that ne-tuning the backbone entirely is often a sub-optimal choice. Instead, higher (or at least competitive) results are achieved by ne-tuning only the biases or the batch normalization parameters of the backbone. Fine-tuning only the biases or the batch normalization parameters in the backbone does not change much the plasticity as only a few parameters are concerned, yet it seems to provide a bene cial adaptability to the entire backbone. On DIOR, Pascal VOC and MS COCO, it provides very high mAP compared to other settings with similar plasticity. Finally, ne-tuning only the very last layer of the classi cation and regression branches is completely sub-optimal. Strangely, this contradicts some FSOD models that adopt this strategy and achieve reasonable performance (e.g., TFA [START_REF] Wang | Frustratingly Simple Few-Shot Object Detection[END_REF]). With FSDi usionDet, this strategy achieves extremely poor detection, having too small plasticity must be avoided. Thus, a plasticity compromise must be found depending on the dataset and its split compatibility.

Number of Proposals

Another set of experiments explores the in uence of the number of proposals for FSOD. The proposals are the boxes sampled at the beginning of the di usion process. The number of proposals N p represents the maximum number of objects that the model can detect in one image. This number is chosen large compared to the average number of objects in the images. Intuitively, sampling more random boxes reduces the chances of missing an object. However, having a higher number of proposals generates more duplicates which can be detrimental as well. More proposals also lead to a higher training time and memory usage as the denoising process is applied on all boxes. parts are the self-attention layers that scale in O(N 2 p ). Thus, we investigate the few-shot performance of FSDi usionDet with various numbers of proposals. The results of these experiments are available in Tab. 7.5. We notice two di erent behaviors between natural and aerial images. For natural images (Pascal VOC and MS COCO), it seems better to set the number of proposals relatively low compared to aerial images. This makes sense as there are more objects in aerial images. For natural images, the detection quality increases as the number of proposals is reduced, and it may be relevant to test what happens with even fewer proposals. However, with aerial images, the performance does not seem to correlate well with the number of proposals. It is relevant to mention that the results on MS COCO are opposite to what the authors of Di usionDet found in the regular data regime (increasing the number of proposals increases the mAP). This could be explained by the reduced number of objects in the images, as in the few-shot regime we consider only the novel classes, many instances are discarded and fewer proposals are required to detect the objects.

Other Experiments and Future Directions

In addition to the previous experiments, we conduct several other studies to further improve the detection capabilities of FSDi usionDet. However, some of these studies did not yield very relevant insights, some others were not explored deeply enough due to the time constraint of this PhD project. We brie y present these experiments that will pave the way for future improvements of FSDi usionDet.

Learning Rate Sweeping. First, just as proposals, freezing sweet spot and backbone pre-training, we studied the in uence of the Learning Rate (LR) and its schedule on the FSOD performance. Indeed, the choice of the LR value during the ne-tuning is not trivial. Therefore, to make sure we get a good t for our experiments we conduct a LR sweeping, i.e., we try several di erent values for the LR. This experiment is conducted only on DOTA with K = 10 for simplicity, but in theory, it should be done for every new experiment. Indeed, following the nomenclature from the recent Deep Learning Tuning Playbook [START_REF] Godbole | Deep Learning Tuning Playbook[END_REF], the learning rate is a nuisance parameter, meaning that to make a fair comparison between various settings, the optimal LR should be found for all runs individually. As we change some hy- perparameters, it is likely that the optimal learning rate changes as well, therefore fair comparison can only be achieved if the LR is optimal for all runs, e.g., the optimal learning rate for K = 1 or K = 100 shots may not be the same. Even though ne-tuning methods are fast to adapt, running such an LR analysis is very expensive. Nevertheless, running an LR sweeping on DOTA provides insights into how it in uences the FSOD performance. The results can be found in Tab. 7.6, and show an optimal value around 5e-5. But most importantly, it shows a relatively large area where performance is satisfactory. This comforts us in our choice of xing the LR for all our experiments.

While this is probably not the optimal choice, it is reasonable. We also tried a cosine annealing scheduler, but it yields consistently inferior results and was then rejected. Its only advantage is that it seems to deal better with higher LR, which makes the training slightly faster.

Proposal Prior Distribution.

In Di usionDet, the coordinates of the proposals are sampled randomly following a normal distribution. The coordinates of the boxes are clamped with a scale parameter ς to make sure the center of each box remains within the image limits. Speci cally, we have:

w = (clamp( w , -ς, ς)/ς + 1) /2, (7.27)
h = (clamp( h , -ς, ς)/ς + 1) /2, (7.28)
x = (clamp( x , -ς, ς)/ς + 1) /2 - w 2 , (7.29)
y = (clamp( y , -ς, ς)/ς + 1) /2 - h 2 , (7.30)
with x , y , w , h ∼ N (0, 1).

By default, ς is set to 2, but it would be interesting to explore how it changes the FSOD performance.

Indeed, as ς → 0 the boxes are more and more identical and their centers tend to approach the image corners, as ς → ∞, the boxes are more and more aligned with the image center. It could also be relevant to explore the use of a uniform sampling instead of a Gaussian distribution. This would prevent having a bias toward the image center as it happens with high values of ς. In fact, the use of ς close to 1, is relatively close to a uniform distribution. This is illustrated in Fig. 7.6. S D D F T

The setting of ς slightly changes the di usion process as the boxes are generated from clamped Gaussian distributions and not regular Gaussians. Most of the derivations detailed in Sec. 7.1 hold only for gaussian distributions and therefore using small values of ς or uniform prior may disrupt the di usion process. We did not have time to conduct these experiments yet, but this is planned as future work. Another consideration is the size of the generated boxes. In the above, the size of the proposal is randomly sampled and has an expected value of half the image size. This may not be optimal, especially when applying FSDi usionDet on aerial images with small objects. Hence, we propose to introduce a proposal scaling parameter that divides the width and height of the proposals:

w = w , (7.32)
h = h , (7.33)
where w and h are the scaled width and height of the sampled boxes. Of course, as small proposals may not cover the whole image, their number must be increased to prevent missing objects. Going further, we can also imagine a mixture of width and height distribution to sample proposals with signi cant size di erences, which is not achieved in practice yet as shown in Fig. 7.6. Transductive Inference.

To further improve the FSOD performance of FSDi usionDet, we also consider designing a transductive inference scheme inside the detection framework. To our knowledge, this would be a rst in the FSOD domain. Of course, the transductive setting is slightly di erent from the regular fewshot inference as it requires access to a large set of query images during the inference. The goal is to detect objects in these images, but these unlabelled images can be leveraged to improve the detection on the entire set. This setup makes a lot of sense for COSE's application. Indeed, the very large images of COSE cannot be processed as a whole, instead, they must be cropped into smaller patches. This means that a relatively large number of images are to be processed at the same time (11600 × 8700 pixels images can be cropped into roughly 400 patches of 512 × 512 pixels). Hence, studying transductive inference is of particular interest to the company.

First of all, in classi cation, the transductive inference is often used in replacement of a ne-tuning step and allows for direct adaptation of a model to a new task or domain (see Sec. 2.2.1.6 for more details). This is prohibited in the detection context as the regression branch must be ne-tuned anyway. Thus, our goal is to improve the classi cation part once the model has been ne-tuned. To

L LSM (Y) = |Q| i=1 |C novel | c=1 l c i d(z i , m c ) + 1 2 i,j η(z i , z j )l T i l j , (7.34)
where z q and l q are the extracted features and classi cation score vector for object q, respectively. Q represents the set of object representations in the whole query set, d is a distance measure (e.g., the euclidean distance between objects representations), and η is a similarity function in the embedding space (in practice, it is chosen as a binary k-NN, i.e., a vector has a similarity of 1 with its k nearest neighbors and 0 with all others). Finally, m c is the representation of class c, it is computed as the average over the support representations of that class. Intuitively, this objective function nds a compromise between assigning to an unlabelled object the label of the closest support example and assigning the same label as its neighbors.

As a rst comparison, we leverage four distinct inference setups for detection:

-Fine-tuning Inference (FI): boxes and classi cation scores output by the ne-tuned model.

-Transductive Inference (TI): boxes from ne-tuned model and classi cation score from the Laplacian Shot Module.

-Hybrid Transductive Inference (HTI): boxes from ne-tuned model and classi cation scores as a combination (e.g., element-wise multiplication) of LSM and ne-tuned model.

-Optimal Classi cation (OC): boxes from the model and optimally matched labels from the ground truth. It can be seen as an oracle, it is a performance upper-bound given the quality of the regression.

We assume above that for the detection task, the regression branch must be ne-tuned otherwise performance is highly degraded. To con rm this assumption, we compare the four inference settings described above using a model that has only been base-trained against a model that was ne-tuned on the novel classes. This is done on DOTA with K = 10 shots. The results are available in Tab. 7.7.

From that table, it is clear that the ne-tuning of the regression branch is crucial to achieving reasonable performance. In particular, the oracle (OC) is highly degraded when the regression branch is not ne-tuned. Interestingly, in this case, TI achieves higher mAP than the non-ne-tuned classi cation branch of the model. It makes sense as this layer is only initialized with random weights.

However, one can see that the TI is largely under the FI when using the ne-tuned regression branch.

The classi cation made by the LSM is therefore worse than the ne-tuned classi cation branch. A quick investigation of the classi cation scores shows that the scores output by TI di er signi cantly from FI (see Fig. 7.9). The ne-tuned model outputs a large number of very small classi cation scores which mostly correspond to background objects. Hence, they are ltered out by the post-processing (score thresholding and NMS), and the remaining ones will have a negligible impact on the mAP computation. TI, however, outputs much higher scores, with a greater variance. It struggles to distinguish foreground and background objects, and for good reason, it was designed for classi cation

7.3 F S D D
and not detection. To this end, we propose HTI, a hybrid classi cation inference that leverages both the score from the ne-tuned model and the transductive inference. Hopefully, it will x the mistakes from the ne-tuned classi cation layer while avoiding the TI's pitfall. To do so, we simply multiply the scores from FI and TI together. Thus, the good foreground/background distinction from FI is embedded in the new score distribution. This helps a lot for the classi cation; however, it is still under the FI performance. From this, it seems that TI is only detrimental to classi cation. TSNE representations of the embedding space help to make sense of these results. These can be found in Fig. 7.8 for FI, TI and HTI, and the Oracle.

One can see large patches of the cluster representing the class 2 misclassi ed as class 14 by both TI and HTI. The prototype of class 2 seems closer to these misclassi ed points in the TSNE visualizations, but the distances must be interpreted carefully as two dimensions are not enough to represent the entire complexity of the representation space (dimension 256). Yet, it seems that the distributions of the classes in the embedding space are multimodal, therefore it may be impossible to accurately classify the objects with only one prototype per class. Leveraging multiple prototypes per class should be investigated in future work.

In addition to HTI, we also tried to lter the objects with low FI scores (with a threshold at 0.05), this greatly helps for the transductive inference although it reduces slightly the performance of FI (some objects are correctly detected but have a low score but are ltered anyway). With ltering and HTI, we almost reach the same performance as the FI which is encouraging. However, the goal is to bene t from the transductive inference and this is not achieved yet. More analysis needs to be conducted to better understand the reason behind the poor classi cation score of TI. In addition, the transductive inference should be extended to account for the detection challenges: a great foreground/background imbalance and an increased intra-class diversity. To this end, we have a few ideas that we did not have time to explore yet:

-Geometric priors: leverage the geometrical features of the objects (e.g., size, aspect ratio, etc) to nd outliers and exclude them without using a hard score thresholding. This could signi cantly reduce the number of objects and help to lter ill-formed boxes.

-Multiple prototypes per class: instead of aggregating all support examples of one class as one prototype, use multiple prototypes per class and extend LSM as a mixture model.

-Background class: Introduce a background class within the LSM module to prevent the poor foreground/background distinction. These tracks will be explored during the last months of this PhD project and will hopefully improve further the detection capabilities of FSDi usionDet. In addition, we plan to apply the transductive inference in Cross-Domain scenarios, as it could help against the performance drop in these settings.

Support Attention.

Finally, we also try to extend FSDi usionDet with an attention mechanism (e.g., XQSA). The main motivation is to be able to compare the in uence of the detection framework on attention-based 155 7.7: Naive comparison between a model only trained on base classes against a model that has been ne-tuned as well. All 4 inference setups are compared as well. Bold values represent the best-performing method between FI, TI and HTI, the Oracle (OC) is not included. Performance is reported as the mAP 0.5 FSOD methods (in Chap. 6, we studied the in uence of the attention mechanism with a xed detection framework). Given the impressive results of FSDi usionDet with a simple ne-tuning strategy, this is promising. Thus, we extend the detection head with a query-support block which is meant to incorporate the support features within the detection head. The head is then split and boxes are produced for each class independently (following the attention-based FSOD principle, see Fig. 6.1). Unfortunately, this does not yield satisfactory results and slows down the training a lot. Considering the time already spent on attention mechanisms since the beginning of the project and the very good performance of FSDi usionDet with the ne-tuning, we decided not to explore this direction further. Nonetheless, this remains an interesting direction for future work.

Comparison with existing FSOD Methods

In the previous section, we explored several design choices for FSDi usionDet and analyzed how they in uence the detection performance on novel classes. We compare here the best settings for FSDi usionDet according to our experiments conducted in the previous section. These experiments are averaged over 5 distinct seeds to get more reliable results. This contrasts with the above experiments which are mostly done with one seed only. However, the limited variance observed over the multiple runs con rms that previous results are reliable as well. FSDi usionDet is compared with PFRCNN and XQSA that we proposed in Chaps. 5 and 6 and some relevant works from the literature. Table 7.8: Detection results of FSDi usionDet on DOTA, DIOR, Pascal VOC and MS COCO datasets. The models employed to produce this gure have been netuned with K = 10 shots and following the best ne-tuning strategy found in Sec. 7.3.2.4 for each dataset. The mAP 0.5 is reported as an average of over 5 distinct runs. The top rows include methods from the literature while the bottom rows designate our proposed methods. This comparison can be found in Tab. 7.8. This table also includes the Small, Medium, and Large size distinctions from the previous chapters. FSDi usionDet largely outperforms other methods disregarding the object sizes. For small objects, FSDi usionDet achieves impressive performance on aerial images but lags slightly behind XQSA on natural images. It is particularly noteworthy as it was not designed speci cally for small object detection. Another surprising result can be observed on DOTA where medium size objects are better detected than large ones, which is not the case for other datasets. This is unusual compared to all our previous experiments, including attention-based methods. It might result from having too few proposals boxes (N p = 300 in Tab. 7.8 for DOTA), then the model can only focus on small and medium objects as they are more numerous than larger ones. Given the size distribution in DOTA, this is a better compromise as it yields higher overall mAP. From Tab. 7.8, it seems that FSDi usionDet performs slightly worse than XQSA, DANA and FRW on MS COCO. This is not true as FSDi usionDet tackles MS COCO as a 20-ways detection problem whereas other approaches only consider an easier 5-ways problem. It is not possible to perform 5-ways episodic evaluation with FSDi usionDet as all classes must be included during ne-tuning. However, it could be interesting to observe how well attention-based methods perform in the 20-ways settings (it is often challenging to do so with attention-based methods due to memory constraints and long inference time).

Finally, we also provide a qualitative assessment of the performance of FSDi usionDet on DOTA, DIOR, Pascal VOC and MS COCO in Fig. 7.10. This gure presents novel class detection results on 5 images from each dataset. It is certainly stronger than all previously studied methods (see Fig. 6.13). However, it is not perfect. Some objects are misclassi ed (see the third row in MS COCO column), some are not detected (DOTA, rst row), and there are still false detections (Pascal VOC, 4th row).

Nonetheless, these qualitative results are much better than our other approaches and can be considered for actual industrial applications. It strengthens the need for more elaborate ne-tuning strategies and insight into how to design them without too much trial and error. FSDi usionDet achieves impressive FSOD results, especially on DOTA and DIOR, but this performance was achieved through expensive exploration. It would be of great help to know in advance how a strategy will perform on a given dataset. We started some investigation in this direction with the design of a compatibility

Application to Cross-Domain FSOD

Given the impressive performance of FSDi usionDet on DOTA and DIOR, it seems tempting to try more di cult setups. Up to now, the methods studied in this project were barely reaching a satisfactory point from an applicative perspective. With FSDi usionDet, we are past that, and can now consider the Cross-Domain setting. Cross-Domain is especially important for COSE, given the prohibited access to test-time images. The ability to adapt to new domains would be an extremely valuable property for a surveillance system such as CAMELEON. Of course, the domain change would be limited in COSE's applications as the only change between two missions would be the general aspect of the image (i.e., weather, GSD, luminosity, etc.). However, the images will always be aerial taken pointing nadir.

In this section, we tackle the challenging Cross-Domain Few-Shot Object Detection (CD-FSOD) task which is barely untouched in the literature. To this end, we focus on two distinct scenarios, one introduced by [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] with a rst training on MS COCO and one speci cally designed for COSE's applications where both the source and target domains are aerial datasets. For both scenarios, we rst present the dataset used as source and target domains and the experimental setup. Then we provide some experimental results with the FSDi usionDet baseline. These results are preliminary and promising, further experimentation in this direction is required to better understand this task and further improve FSDi usionDet in this context. Therefore, we end this section with a summary of the future work that is planned.

MS COCO → Anything

First, we study a general Cross-Domain (CD) setting introduced in the literature by [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF]. It consists of training rst on MS COCO and then ne-tuning on another dataset with a restricted number of shots. Unlike in the FSOD setting, there is no separation between base and novel classes in CD, all classes of the target domain are considered novel. The benchmark introduced by [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] contains a list of 10 datasets (VisDrone2019 [START_REF] Zhu | Detection and Tracking Meet Drones Challenge[END_REF], DeepFruits, iWildCam [START_REF] Beery | The iwildcam 2021 competition dataset[END_REF], SIXray [START_REF] Miao | Sixray: A large-scale security inspection x-ray benchmark for prohibited item discovery in overlapping images[END_REF], Fashionpedia [START_REF] Jia | Fashionpedia: Ontology, segmentation, and an attribute localization dataset[END_REF], Oktoberfest [START_REF] Ziller | Oktoberfest Food Dataset[END_REF], LogoDet-3K [START_REF] Wang | Logodet-3k: A large-scale image dataset for logo detection[END_REF], CrowdHuman [START_REF] Shao | CrowdHuman: A Benchmark for Detecting Human in a Crowd[END_REF], ClipArt [START_REF] Inoue | Cross-domain weakly-supervised object detection through progressive domain adaptation[END_REF], KITTI [START_REF] Geiger | Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite[END_REF]).

Cross-Domain Scenarios

The pool of datasets proposed by [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] has a large variety of images, therefore it constitutes a relevant benchmark for CD-FSOD. However, it does not contain any aerial dataset. VisDrone is an aerial image dataset but di ers greatly from DOTA or DIOR as its images are taken from a much lower altitude and contain perspective. In addition, 10 datasets make the experiments expensive to run. Thus, we propose a lighter benchmark using VisDrone2019, DeepFruits, SixRay, DOTA, DIOR and ClipArt. We emphasize that dealing with that many datasets is quite challenging as almost every dataset has its own annotation format and data structure. When [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] proposed this benchmark, the authors only provided the list of datasets, without any information about their preparation and split, which makes their experiments very hard to reproduce. On the contrary, we propose for the convenience of future research on CD-FSOD a prepared version of this "meta-dataset" 2 under the same format (the MS COCO format, which is relatively common in the OD community). In addition to the prepared meta-dataset, we also extend the popular Python package pycocotools to help load and explore the datasets.

As mentioned above, we study here 6 cross-domain scenarios with a common base training on MS-COCO. Every scenario has a di erent target domain represented with one of the following datasets. Fig. 7.11 provides some image examples (without annotations) for VisDrone2019, Deep-Fruits, SixRay, and ClipArt. We refer the reader to Fig. 2.7 for a presentation of DOTA and DIOR.

For convenience, we denote these 6 scenarios as COCO → X scenarios.

Experimental Results

We only experimented with the CD scenario with the baseline of FSDi usionDet. Speci cally, only the detection head is ne-tuned, with 500 proposals. The weights of the model are initialized using the pre-trained weights available on the Di usionDet repository, except for the last layer of the classi cation branch which is initialized randomly with the right number of outputs. While this is certainly not optimal for every scenario, this gives a strong baseline to compare with in future experiments.

The results of our experimentation with the cross-domain scenarios can be found in Tab. 7.9 and Fig. 7.12. As for the regular FS experiments, a clear pattern is visible as the number of shots increases. Obviously, the more shots, the better the detection. However, this pattern di ers from one scenario to another. For instance, COCO → ClipArt scenario sees very little improvements as the number of shots increases unlike COCO → DOTA and COCO → SIXRay. It is also noteworthy to observe the di erent behaviors between DOTA and DIOR. Even if DIOR is an "easier" dataset than DOTA (in the sense that higher performance is achieved on DIOR in a regular detection setting), there is a larger di erence in the COCO → X cross-domain scenario. Relatively low performance is observed for ClipArt and VisDrone, this is probably due to di erences in data preparation compared with [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF]. As the author did not provide any information about all the datasets' splits and preparation, we can only guess what they did. For ClipArt, it is slightly di erent as they leveraged a GAN-augmented version of the dataset which might sensibly boost the detection performance.

Finally, for each scenario 5 distinct training were done with varying seeds to check the consistency of our results. Tab. 7.9 gives the average over the 5 runs and a 95% con dence interval. A limited variance between di erent runs is observed, this means that FSDi usionDet is not very sensitive to the examples chosen in the support set. This is a crucial property as some few-shot methods de- is probably not optimal, instead, we could leverage an aerial dataset as a source model as well.

Incidentally, this will be the subject of the next section.

Cross-Domain Perspectives

The previous sections have been devoted to cross-domain experiments. These are preliminary but interesting results. They give insight into how di cult this setup is and how ne-tuning strategies can perform. However, plenty of experiments are still necessary. We detail here some of the most relevant perspectives for future CD-FSOD research that we brie y hinted in the previous sections:

1. Comparison with other FSOD methods: it would be interesting to compare with other existing FSOD methods, in particular, with attention-based techniques that we studied in depth in Chap. 6. In addition, studying other ne-tuning approaches is required to validate the results found in our experiments.

2.

Transductive inference: Even though our naive transductive detection did not outperform the ne-tuning strategy in FSOD, it could help in cross-domain scenarios. Indeed, leveraging query images during inference can reduce the discrepancies between source and target domains and improve performance. This has been empirically shown for the classi cation task, but it remains to be adapted for detection.

3. Source-target domain compatibility score: [START_REF] Lee | Rethinking Few-Shot Object Detection on a Multi-Domain Benchmark[END_REF] proposes to choose the ne-tuning sweet spot according to the distance between the source and target domains. Speci cally, more plasticity is required when domains are farther apart. They compute such distance as the recall of S D D F T a pre-trained detection model on MS COCO, applied to the target dataset in a class-agnostic manner. This could be generalized to any source and target domains with a detection model trained on the source domain. However, we would like to emphasize that this should not be called a distance measure between domains as it does not satisfy the symmetry property. Instead, it is a compatibility measure as it evaluates how bene cial the source domain is for the adaptation to the target domain. Our cross-domain scenarios on aerial images clearly demonstrate this, as we obtain contradictory conclusions for DOTA → DIOR and DIOR → DOTA scenarios. More plasticity is required for DIOR → DOTA than for DOTA → DIOR, hence, the compatibility measure cannot be the same for these two scenarios. In addition, this distance is highly in uenced by the detector chosen in the rst place, in particular, some models are known to output a lot of duplicate boxes which often boost the recall signi cantly (see such an analysis in [START_REF] Jena | Beyond mAP: Towards better evaluation of instance segmentation[END_REF]). It would be helpful to come up with a properly de ned compatibility measure for a given scenario that does not rely on a detection model and gives coherent hints to obtain an optimal ne-tuning strategy. This measure should also be able to assess the compatibility of the base and novel class set in the regular FSOD setting, as a special case of the cross-domain scenario (source and target domains are identical but the classes change). We are currently working on such a compatibility score based on an overall discrepancy measure between source and target domains and a source-target classes compatibility score.

Conclusion

In this chapter, we have presented thoroughly the basic principle of di usion models and how they can be leveraged for detection. Then, we proposed a simple ne-tuning strategy to apply Di usion-Det in the few-shot setting. FSDi usionDet achieves sensibly higher performance than all previous methods studied in this PhD on aerial images. To understand why, we conducted extensive experimental studies on crucial design choices of our strategy. It highlighted a strong but complex connection between the plasticity of the model and the detection performance. Finally, we applied FSDi usionDet in several cross-domain scenarios and observed promising results. Again, the plasticity has a great in uence on the performance and more experiments must be conducted to understand this relation completely. A possible direction would be to design a compatibility measure between domains and between sets of classes to determine the optimal amount of plasticity required for a given scenario. thoroughly and highlight some of its aws when employed as a loss function or as a cornerstone of the evaluation process of detection models. These aws mainly concern small objects for which IoU is too strict. Therefore, it is particularly relevant to tackle these issues for aerial images and COSE's applications. To address these weaknesses, we propose Scale-Adaptive Intersection over Union (SIoU), a parameterizable criterion that can be set to favor small objects as needed. We start by de ning and analyzing the IoU and its variants. Then, we propose our novel criterion SIoU and its properties. Sec. 8.3 presents an original empirical and theoretical study of several box similarity criteria and argues for the superiority of SIoU. Finally, we conduct a user study and experimental analysis to further consolidate the advantages of SIoU over IoU.

Analysis of Intersection over Union

In this section, we rst review the de nition of IoU and present some of its variants that are available in the literature. Then, we analyze why IoU is not optimal for small objects.

Intersection over Union and its Variants

To begin, let us review the de nition of existing criteria for box similarity. Originally, the IoU is de ned as the intersection area of two sets divided by the area of their union:

IoU(A, B) = |A ∩ B| |A ∪ B| , (8.1)
where A and B are two sets. Even if there are plenty of applications where IoU is useful (e.g., in statistics where IoU is better known as the Jaccard index), we are mostly interested here in its ap- frameworks (e.g., [START_REF] Yu | Unitbox: An advanced object detection network[END_REF][START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF]). IoU is also involved in the process of example selection during training of most detection methods, i.e., all the ones inspired either by Faster R-CNN [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF] or YOLO [START_REF] Redmon | You only look once: Uni ed, real-time object detection[END_REF]. In these frameworks, regression loss is computed from the coordinates of proposed boxes and ground truth. Not all pairs of proposals and ground truth are kept for the computation. Only proposals with a su cient IoU with a ground truth box are selected. Finally, IoU is also used at the heart of the evaluation process. A proposed box is considered a positive detection if it meets two conditions: 1)

an IoU greater than a given threshold with a ground truth box, and 2) the same label as this ground truth (see Sec. 2.1.2).

Several attempts were made to improve IoU but existing works mostly focus on the regression loss part, disregarding the other IoU uses in the detection task. First, [START_REF] Rezato | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF] proposed a generalized version of IoU which yields negative values when boxes do not overlap:

GIoU(A, B) = IoU(A, B) - |C\(A ∪ B)| |C| , (8.2)
where C is the convex hull around A and B. This criterion is employed as a loss function by several detection frameworks [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF][START_REF] Bochkovskiy | Yolov4: Optimal speed and accuracy of object detection[END_REF][START_REF] Zhang | Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection[END_REF]. It is sometimes also combined with other regression loss as in [START_REF] Li | Generalized focal loss: Learning quali ed and distributed bounding boxes for dense object detection[END_REF][START_REF] Carion | End-to-end object detection with transformers[END_REF], which both combine it with an L1 regression on box coordinates. Combining IoU loss with other regression terms was also proposed by [START_REF] Zheng | Distance-IoU loss: Faster and better learning for bounding box regression[END_REF]. They introduce two losses Distance-IoU (DIoU)

and Complete-IoU which respectively add an L2 regression term and an aspect ratio penalty to the IoU loss. Recently, α-IoU [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF] extends DIoU [START_REF] Zheng | Distance-IoU loss: Faster and better learning for bounding box regression[END_REF] by proposing a family of losses following the same structure as DIoU with the IoU term raised to the power α:

α-IoU(A, B) = IoU(A, B) α . (8.3)
Balanced IoU (BIoU) also extends upon DIoU by measuring shifts between the corners of the boxes instead of their centers. Alternatively, Bounded IoU [START_REF] Tychsen | Improving object localization with tness nms and bounded iou loss[END_REF] computes an IoU upper bound between a proposal and a ground truth. Other approaches, such as Scale Balanced Loss [START_REF] Sun | A scale balanced loss for bounding box regression[END_REF], try to design distance-based loss functions which share properties with IoU, especially its scale-invariance.

All these IoU variants are proposed to improve the regression part of the models. However, IoU is involved in other parts of the framework including example selection, Non-Maximal Suppression, and evaluation. A recent user study [START_REF] Stra Orello | Humans disagree with the IoU for measuring object detector localization error[END_REF] indicates that IoU does not completely align with human perception. Humans have strong positional and size preferences based on conceptual information contained in the boxes. It suggests that IoU is not an optimal choice either for example selection or for evaluation as it will lead to detections that do not satisfy human users.

Inadequation of IoU for Small Objects and Few-Shot Regime

Object detection is a fundamental task in industry and has applications in many domains such as medical imaging, agriculture, or autonomous driving. However, it is often impracticable or too expensive to build su ciently large annotated datasets to train detection models. It is therefore A I O U for the detection of small objects partially solves this issue (see Sec. 6.3.1), but is not enough. One of the reasons for the poor FSOD performance on small objects is the extensive use of the IoU. Just as for detection, most FSOD pipelines employ IoU as a regression loss [START_REF] Yu | Unitbox: An advanced object detection network[END_REF][START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF]; for example selection [START_REF] Ren | Faster r-cnn: Towards real-time object detection with region proposal networks[END_REF][START_REF] Redmon | You only look once: Uni ed, real-time object detection[END_REF][START_REF] Liu | Ssd: Single shot multibox detector[END_REF]; or as an evaluation criterion, but IoU is not an optimal choice when dealing with small objects.

IoU has a remarkable property: scale invariance. It means that scaling all coordinates of two bounding boxes by the same amount will not change their IoU. At rst glance, this seems a desirable property as all objects will be treated identically no matter their size. In practice, it has a fundamental drawback: small boxes are prone to large IoU changes from only small position or size modi cations.

To clarify, let us consider a simple example. Two square boxes of width ω are shifted diagonally by ε loc pixels. In this setup, a 1-pixel shift leads to a larger decrease in IoU when boxes are smaller. This comes from the scale invariance property, IoU stays constant as the ratio ε loc ω remains xed. However, this ratio is not constant for trained detection models, it increases as objects get smaller (see is not an optimal box similarity measure.

Only a handful of works question the adequation of IoU for object detection. Among those, [START_REF] Rezato | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF] proposed a generalization of IoU when boxes do not overlap, [START_REF] Xu | Detecting Tiny Objects in Aerial Images: A Normalized Wasserstein Distance and A New Benchmark[END_REF] introduced a novel loss function to target small objects, and [START_REF] Du | Scale-sensitive IOU loss: An improved regression loss function in remote sensing object detection[END_REF] proposed a Scale-Sensitive IoU which extended CIoU with an area regulatory factor. In addition, [START_REF] Stra Orello | Humans disagree with the IoU for measuring object detector localization error[END_REF] showed that human perception and IoU are not fully aligned.

This lack of interest in new criterion design is explained by the great detection performance in the regular setting (i.e., natural images with su cient annotations). In the few-shot regime, and when targets are small, the aws of IoU become critical. Therefore, we revisit IoU to improve FSOD methods and focus on aerial images which mostly contain small objects. We propose Scale-adaptive Intersection over Union (SIoU), a novel criterion that can replace IoU for training and evaluating detection models. However, for training we mostly aim at few-shot detection models as small objects are particularly di cult for them. To demonstrate the superiority of the proposed SIoU, Sec. 8.3 compares it with various existing criteria. This section analyzes criteria distributions when exposed to randomly shifted boxes. To our knowledge, this is the rst attempt to empirically and theoretically study the distributions of these criteria. The conclusions of this analysis are then compared with human perception through a user study which shows that SIoU aligns better with human appraisal than IoU (see Sec. 8.4). The comparison of these criteria also highlights that SIoU as a loss function can guide training towards small objects better than other criteria and in a more controlled fashion.

SIoU loss can be tuned to improve the detection of small objects just as it can be tuned to align with human perception. Finally, these analyses are con rmed by extensive experiments on both aerial images (DOTA and DIOR datasets) and natural images (Pascal VOC and COCO datasets).

large if √ w i h i > 96.
IoU is scale-invariant, hence if IoU(b 1 , b 2 ) = u, scaling all coordinates of both boxes by the same factor k will produce the same IoU:

IoU(b 1 , b 2 ) = IoU(kb 1 , kb 2 ) = u. (8.4)
However, detection models are not scale-invariant, they do not localize equally well small and large objects. 3. NMS does not lter duplicates of small boxes as their overlap is not high enough.

A way to alleviate these issues is to relax the invariance property of the IoU so it favors more small objects without penalizing large ones. To this end, we propose a novel criterion called Scale-adaptive Intersection over Union (SIoU):

SIoU(b 1 , b 2 ) = IoU(b 1 , b 2 ) p with p = 1 -γ exp - √ w 1 h 1 + w 2 h 2 √ 2κ , (8.5)
where p is a function of the object sizes. Thus, the scores are rescaled according to the object size.

γ ∈] -∞, 1] and κ > 0 are two parameters that control the strength and direction of the rescaling (hence, p ≥ 0). γ governs the scaling for small objects while κ controls how fast the behavior of regular IoU is recovered for large objects. Of course, there are many valid choices for the exponent p. However, we want to ensure some properties for SIoU, which translate into constraints for p:

-SIoU should either be higher or lower than IoU when objects are small, but should remain nite, so p(0) ∈ R * + . -For large objects, SIoU should behave like IoU, lim

τ →∞ p(τ ) = 1.
-To prevent complete inversion of the order and smooth changes, p should be positive, continuous, and monotonic. Thus, an exponential response is a natural choice for the design of p. Similar forms could be achieved with hyperbolic functions. For instance, p(τ ) = 1 -γ 1+κτ would be a sensible alternative. An inconvenient of these designs is the possibility to only focus on either small or large objects. This is mainly due to the monotonicity of p. It can be relaxed to unlock the possibility of targeting objects of a speci c size, for instance, with a bell-shaped exponent e.g., p(τ ) = 1 -γ exp(-κ(τ -τ target ) 2 ).

Where κ could be understood as a bandwidth parameter around objects of size τ target . We did not investigate the design of p, but experimenting with it would be relevant to better understand the balance between small and large objects during training.

S

A I U

SIoU Properties

This new criterion follows the same structure as α-IoU [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF], but di ers greatly as it sets di erent powers for di erent object sizes. SIoU provides a solution for small object detection in the few-shot regime while α-IoU only aims to improve general detection. However, SIoU inherits a few properties from α-IoU.

Property 1 (SIoU Relaxation) Let b 1 and b 2 be two bounding boxes and introduce τ = w 1 h 1 +w 2 h 2 2 their average area. SIoU preserves the behavior of IoU in certain cases such as:

-

IoU(b 1 , b 2 ) = 0 ⇒ SIoU(b 1 , b 2 ) = IoU(b 1 , b 2 ) = 0 -IoU(b 1 , b 2 ) = 1 ⇒ SIoU(b 1 , b 2 ) = IoU(b 1 , b 2 ) = 1 -lim τ →+∞ SIoU(b 1 , b 2 ) = IoU(b 1 , b 2 ) -lim κ→0 SIoU(b 1 , b 2 ) = IoU(b 1 , b 2 )
Property 1 shows that SIoU is sound: it equals IoU when boxes have no intersection and when they perfectly overlap. Therefore, the associated loss function (see Property 2) will take maximal values for boxes that do not overlap and minimum values for identical boxes. In addition, SIoU behaves similarly to IoU when dealing with large objects (i.e., when τ → ∞). When boxes are large, the power p that rescales the IoU is close to 1. Hence, this change of criterion only impacts small objects. However, when discussing the properties of SIoU, the limit between small/medium/large objects is relative to the choice of κ. If κ √ wh, even large objects will be rescaled. On the contrary, when κ -→ 0, all objects are treated as large and are not rescaled. In practice, κ and γ are chosen empirically, but Sec. 8.3 provides useful insights for the choice of these parameters.

L IoU (b 1 ,b 2 ) . Similarly, W ∇ (b 1 , b 2 ) = |∇L SIoU (b 1 ,b 2 )| |∇L IoU (b 1 ,b 2 )
| denotes the ratio of gradients generated from SIoU and IoU losses:

W L (b 1 , b 2 ) = 1 -IoU(b 1 , b 2 ) p 1 -IoU(b 1 , b 2 ) , (8.6)
W ∇ (b 1 , b 2 ) = pIoU(b 1 , b 2 ) p-1 , (8.7)
W L and W ∇ are increasing (resp. decreasing) functions of IoU when p ≥ 1 (resp. p < 1) which is satis ed when γ ≤ 0 (resp. γ > 0). As the IoU goes to 1, W L and W ∇ approaches p:

lim IoU(b 1 ,b 2 )→1 W L (b 1 , b 2 ) = p, (8.8)
lim

IoU(b 1 ,b 2 )→1 W ∇ (b 1 , b 2 ) = p. (8.9)
We employ the same tools as in [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF] to analyze how SIoU a ects the losses and associated gradients.

We show in property 2 that their results hold for a non-constant power p as well. From this, it can A I O U be observed that when IoU is close to 1, losses and gradients are both rescaled by p. Hence, the gradients coming from objects of di erent sizes will be rescaled di erently. The setting of γ and κ allows to balance the training towards speci c object sizes. Experimental results are provided in Sec. 8.5 to support these ndings. Proofs for properties 1 and 2 are available in App. A.

However, order preservingness is not satis ed by using power value changing with the size of the objects. This property ensures that the order given by the IoU is preserved with the novel criterion, e.g.,

IoU(b 1 , b 2 ) < IoU(b 1 , b 3 ) ⇒ α-IoU(b 1 , b 2 ) < α-IoU(b 1 , b 3
). α-IoU preserves the order of IoU, but SIoU does not. We show in App. A that even though this property is not always satis ed, a large proportion of boxes meet the conditions for the order to hold.

Extensions and Generalization of SIoU

Finally, SIoU can very well be extended as IoU was with GIoU or DIoU. Note that we only focus on GIoU extension here as DIoU and its variants are composite loss (i.e., sum of multiple loss functions).

We provide here an extension following GIoU as it appears especially well-designed for small object detection. When detecting small targets, it is easier for a model to completely miss the object, producing an IoU of 0 no matter how far the predicted box is. On the contrary, GIoU yields negative values for non-intersecting boxes. This produces more relevant guidance during the early phase of training when the model outputs poorly located boxes. Therefore, we extend SIoU by raising GIoU to the same power p as in Eq. (8.5):

GSIoU(b 1 , b 2 ) =    GIoU(b 1 , b 2 ) p if GIoU(b 1 , b 2 ) ≥ 0 -|GIoU(b 1 , b 2 )| p if GIoU(b 1 , b 2 ) < 0 . ( 8
.10)

Scale-Adaptive Criteria Analysis

This section analyzes both empirically and theoretically the behaviors of IoU, GIoU [START_REF] Rezato | Generalized intersection over union: A metric and a loss for bounding box regression[END_REF], α-IoU [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF], NWD [START_REF] Xu | Detecting Tiny Objects in Aerial Images: A Normalized Wasserstein Distance and A New Benchmark[END_REF], SIoU and GSIoU. We investigate the desirable properties of such criteria for model training and performance evaluation.

Response Analysis to Box Shifting

As mentioned in Sec. 8.2, IoU drops dramatically when the localization error increases for small objects. Shifting a box a few pixels o the ground truth can result in a large decrease in IoU, without diminishing the quality of the detection from a human perspective. This is depicted in Fig. 8.1 (left),

where plain lines represent the evolution of IoU for various object sizes. These curves are generated by diagonally shifting a box away from the ground truth. Boxes are squares, but similar curves would be observed otherwise. In this plot, boxes have the same size, so when there is no shift in between (ε loc = 0), IoU equals 1. However, if the sizes of the boxes di er by a ratio r, IoU would peak at 1/r 2 . Other line types represent other criteria. SIoU decreases slower than IoU when ε loc increases

8.3 S A C A
and this is especially true when boxes are small. This holds because γ > 0, if it was negative, SIoU would adopt the opposite behavior. In addition, the gap between IoU and SIoU is larger when objects are small. Only NWD shares this property, but it only appears when boxes have di erent sizes (all lines coincide for NWD). Hence, SIoU is the only criterion that allows controlling its decreasing rate, then entirely de ned by its horizontal position x and its width ω. If a detector is not perfect, it will produce bounding boxes slightly shifted horizontally from the ground truth. To model the detector's inaccuracy, we suppose that the predicted box position is randomly sampled from a Gaussian distribution centered on the ground truth location (which is chosen as 0 without loss of generality):

S

A C A controlling how much small objects are favored (see Fig. 8.5). NWD is not included in these plots as its expected value and variance are constant when dealing with same-size boxes.

In uence Analysis on the Performance Evaluation

If the expected value of a criterion is too small, it is likely that the boxes will be considered negative detections during evaluation and therefore reduce the performance. Therefore, having a criterion with larger expected values for small objects would better re ect the true performance of a detector.

One might think that it would be equivalent to scale-adaptive IoU thresholds during the evaluation, but this is not completely true as the variance of the criteria also di ers.

Having an accurate criterion (i.e., with low variance) is crucial for evaluation. Let us take a detector that produces well-localized boxes on average, i.e., on average the criterion computed between the boxes and their corresponding ground truths is above a certain threshold. As the detector is not perfect, it will randomly produce boxes slightly better or slightly worse than the average. If the criterion has a high variance, it will be more likely that poor boxes get scores below the criterion threshold and therefore will be considered negative detections. This will reduce the performance of the detector even though on average, it meets the localization requirements. In addition, a criterion with a higher variance will be less reliable and would produce more inconsistent evaluations of a model.

The fact that the IoU variance is high for small objects partly explains why detectors have much lower performance on these objects. Hence, SIoU seems more adapted for evaluation. Of course, using this criterion for evaluation will attribute higher scores for less precise localization of small objects. However, this aligns better with human perception as demonstrated in Sec. 8.4. Employing SIoU in the evaluation process also allows tweaking it for the needs of a speci c application. thus the optimization focuses on small objects. This also explains why generalizations of existing criteria (i.e., with negative values for non-overlapping boxes) often outperform their vanilla version.

Taking IoU and GIoU as examples, the gap between their expected values for small and large objects is greater with GIoU. It nudges the optimization towards small objects.

Inaccuracy Tolerance Assumptions

Several assumptions were made in Sec. 8.3.2 to analyze the criteria for box similarity:

1. Boxes are shifted only horizontally.

2. Boxes have the same size.

3. The detector's inaccuracy is xed and does not depend on the object size. The rst two assumptions are relatively harmless. Allowing diagonal shifts simply accelerates the IoU drop rate. A 1-pixel diagonal shift is equivalent to a vertical and a horizontal shift. Intuitively, this is similar to a 2-pixels horizontal shift. However, this is not true because, with a 1-pixel diagonal shift, the area of intersection decreases slower than with a 2-pixels horizontal shift. Following the notations from Sec. 8.3.2, the intersection between two boxes of width ω diagonally shifted by ρ pixels is (ω -ρ) 2 = ω 2 -2ωρ + ρ 2 while the intersection between same boxes horizontally shifted by 2ρ pixels is ω(ω -2ρ) = ω 2 -2ωρ. To ensure that this does not question the conclusions of Sec. 8. It is less straightforward that the conclusions hold without the last assumption. In the analysis from Sec. 8.3.2.1, we assume that the inaccuracy of the detector is xed. This means that the detector generates randomly shifted boxes by the same number of pixels on average no matter the size of the object. This is certainly false, in practice, in terms of absolute distance, detectors are better with smaller objects. However, the inaccuracy cannot simply be proportional to object sizes because small objects would then be perfectly detected. Thus, we tried to change the inaccuracy of the detector as an a ne function of the box width: σ(ω) = σ 0 + λω. We choose to set σ 0 to the xed value of σ used in Sec. 8.3.2.1 and λ = 1/4. This setting re ects better the inaccuracy of a true detector.

The expected values and standard deviations of IoU, SIoU, GIoU, and GSIoU with this inaccuracy setting are plotted in Fig. 8.4c. The main di erence with xed inaccuracy is that expected values do not approach 1 as object size gets larger, instead they tend towards lower values. It also leads to non-zero variance for large objects. However, for small objects, the curves of the di erent criteria are mostly unchanged, and the conclusions formulated in Sec. 8.3.2.1 are still valid.

Theoretical study of GIoU

In the previous section, we derived the statistics of several box similarity criteria from empirical simulations. However, the criteria probability distribution functions and rst moments can also be derived theoretically. We provide such results for GIoU in Proposition 1.

Proposition 1 (GIoU's distribution) Let b 1 = (0, y 1 , w 1 , h 1 ) be a bounding box horizontally centered and b 2 = (X, y 2 , w 2 , h 2 ) another bounding box randomly positioned, with X ∼ N (0, σ 2 ) and σ ∈ R * + . Let's suppose that the boxes are identical squares, shifted only horizontally (i.e., w 1 = w 2 = h 1 = h 2 and y 1 = y 2 ). Let Z = C(X), where C is the generalized intersection over union. The probability density function of Z is given by:

d Z (z) = 2ω (1 + z) 2 √ 2πσ exp - 1 2 ω(1 -z) σ(1 + z) 2 . ( 8

.11)

The two rst moments of Z exist and are given by:

E[Z] = 2 π 3/2 G 2,3 3,2 2a 2 0 1 2 1 2 1 2 0 , (8.12) E[Z 2 ] = 1 - 8a √ 2π + 16a 2 π 3/2 G 2,3 3,2 2a 2 -1 1 2 -1 2 1 2 0 , (8.13)
where G is the Meijer G-function [START_REF] Beals | Meijer G-Functions: A Gentle Introduction[END_REF] (see its de nition in Eq. (B.4), App. B) and a = σ ω .

The proof of this proposition and derivations for other criteria are available in App. B. The theoretical expressions completely agree with empirical results, which con rms the soundness of our simulations.

Other criteria do not have closed forms for their rst and second moments. Nonetheless, we provide in Tab. 8.1 their expressions keeping the integrals as simple as possible, which allows relatively easy evaluation. In addition, we provide the expression of the pdf for each criterion. The setup remains identical as in Proposition 1, the boxes are only horizontally shifted and have the same width ω. For clarity, we also give simple expressions of each criterion in such a setup (see Tab. 8.1).

In uence of γ and κ on SIoU and GSIoU

In the previous discussion, SIoU and GSIoU are parametrized with γ = 0.5 and κ = 64; however, these two parameters have an in uence on the distribution and moments of SIoU and GSIoU.

First, following the analysis from Sec. 8.3.2.1, Fig. 8.5 investigates the in uence of γ and κ on SIoU behavior. Fig. 8.5a shows the value of p, the expectation and variance of SIoU against object size for γ ∈ {1.0, 0.75, 0.5, 0.1, 0.0, -0.25, -2, -4}. p is a function of the average area of the boxes

(τ = w 1 h 1 +w 2 h 2 2
) and for simplicity we suppose here that the boxes are squares of the same width ω, hence √ τ = ω. Then p can be viewed as a function of ω: p(ω) = 1 -γ exp(-ω/κ). For negative values of γ, p decreases from p(0) = 1 -γ to 1, small objects get higher exponents in comparison with larger objects. On the contrary, when γ > 0, p increases from p(0) = 1 -γ to 1. Changing γ

S A C A

Criterion Expression

Probability Density Function For SIoU and GSIoU, we xed γ = -4 and κ = 16, for α-IoU, α = 3 (as recommended in the original paper [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF]). The inaccuracy of the detector is set to σ = 16. Note that the empirical pdfs were smoothed using a Kernel Density Estimator method. This a ects particularly IoU, SIoU and α-IoU as the actual PDF is de ned only on [0, 1]. For the sake of visualization, GIoU and GSIoU were rescaled between 0 and 1 for the expectation and standard deviation plots.

E[Z] E[Z 2 ] IoU C : R → [0, 1] x → max 0, ω -|x| ω + |x| d Z (z) = 2 (1 -F X (ω 1 -z 1 + z )δ 0 (z) + 1 R+ (z) 4ω (1 + z) 2 d X (ω 1 -z 1 + z )) E[Z] = 4 √ 2πa 1 0 1 1 + u e -u 2 2a 2 du - erf( 1 √ 2a ) E[Z 2 ] = erf( 1 √ 2a ) - 8a √ 2πa 1 - 1 2 e -1 2a 2 -2 1 0 1 (1 + u) 3 e -u 2 2a 2 du GIoU C : R → [-1, 1] x → ω -|x| ω + |x| d Z (z) = 4ω (1 + z) 2 √ 2πσ exp - 1 2 ω(1 -z) σ(1 + z) 2 E[Z] = 2 π 3/2 G 2,3 3,2 2a 2 0 1 2 1 2 1 2 0 E[Z 2 ] = 1 - 8a √ 2π + 16a 2 π 3/2 G 2,3 3,2 2a 2 -1 1 2 -1 2 1 2 0 SIoU C : R → [0, 1] x → ω-|x| ω+|x| p if ω -|x| ≥ 0 0 otherwise d Z (z) = 2 (1 -F X (ω 1 -z 1/p 1 + z 1/p )δ 0 (z) + 1 R+ (z) 4ωz 1/p-1 (1 + z 1/p ) 2 d X (ω 1 -z 1/p 1 + z 1/p )) E[Z] = 2ω 1 0 1 -u 1 + u p e -u 2 2a 2 du E[Z 2 ] = 2ω 1 0 1 -u 1 + u 2p e -u 2 2a 2 du GSIoU C : R → [-1, 1] x →    ω-|x| ω+|x| p if ω -|x| ≥ 0 -|x|-ω ω+|x| p otherwise d Z (z) = 2 (1 -F X (ω 1 -z 1/p 1 + z 1/p )δ 0 (z) + 1 R+ (z) 4ωz 1/p-1 (1 + z 1/p ) 2 d X (ω 1 -z 1/p 1 + z 1/p )) -(1 -F X (ω 1 + |z| 1/p 1 -|z| 1/p )δ 0 (z)) + 1 R-(z) 4ω|z| 1/p-1 (1 -|z| 1/p ) 2 d X (ω 1 + |z| 1/p 1 -|z| 1/p )) E[Z] = 2ω 1 0 1 -u 1 + u p e -u 2 2a 2 du - +∞ 1 1 -u 1 + u p e -u 2 2a 2 du E[Z 2 ] = 2ω 1 0 1 -u 1 + u 2p e -u 2 2a 2 du - +∞ 1 1 -u 1 + u 2p e -u 2

SIoU Alignment with Human Perception

As discussed in Sec. 8.3.2.1, having an accurate criterion i.e., one with low variance, is crucial for evaluation. However, such a criterion must also align with human perception. Most image processing models are destined to assist human users. Thus, to maximize the usefulness of such models, the evaluation process should align as closely as possible with human perception. To assess the agreement between the criteria and human perception, we conducted a user study in which participants had to rate on a 1 to 5 scale (i.e., from very poor to very good) how a bounding box localizes an object.

Speci cally, an object is designated by a green ground truth box and a red box is randomly sampled around the object (i.e., with random IoU with the ground truth). Then, the participants rate how well the red box localizes the object within the green one. The study gathered 75 di erent participants and more than 3000 individual answers. We present here the main conclusion of this study. image analysis, we will refer to this group as the expert group. On average, the response time is 10.3s per evaluated image during the rst phase of the experiment (when a background image is visible).

It drops to 7.2s when the image is replaced with a uniform background during the second phase.

This time di erence suggests that humans do take into account the contextual information of the image inside their decision-making process, which agrees with the ndings of [START_REF] Stra Orello | Humans disagree with the IoU for measuring object detector localization error[END_REF].

User Study Insights

Human perception does not fully align with IoU. People tend to be more lenient than IoU towards small objects. Speci cally, comparing a small and a large box with the same IoU with respect to their own ground truth, people will rate the small one better. This suggests that IoU is too strict for small objects in comparison with human perception. From a human perspective, precise localization seems less important for small objects. Fig. 8.9 represents the relative gap of IoU (left) and SIoU (right) values for each object size and rating. The relative di erences c s,r are computed against the average IoU (or SIoU) value per rating:

c s,r = C s,r - s C s,r s C s,r , (8.14)
where C s,r is the average criterion value (C ∈ {IoU, SIoU}) for objects of size s and rating r. IoU values for small objects (in orange) are lower than for large objects (in red) for all rating r. For a human to give a rating r to a box, it requires that a box overlaps less with the ground truth (according to IoU) if the boxes are small. SIoU compensates for this trend (see Fig. 8.9). However, according to SIoU, the same overlapping value triggers the same human rating no matter the object sizes (see Fig. 8.9). Or at least the required criterion value gap between small and large objects is reduced. We can observe that it is more di cult to compensate the gaps for higher rating values (especially with r = 4). This means that human appraisal is not identically in favor of small objects. Instead, it seems more and more in favor of the small objects until the boxes are nearly identical (rating r = 5). It would be relevant to extend SIoU further to take this into account and achieve even better alignment with human perception. Anyway, SIoU is much better aligned than IoU with the human rating as for a rating r, the di erence of SIoU for small and large objects is below 5% while with IoU, it is always above 15%. This means that with IoU, if we have two predicted boxes for one ground truth, the IoUs of the predicted boxes with the ground truth can vary from about 15% without changing the human rating. Such a di erence in SIoU would result in di erent human ratings for the two predicted boxes. This phenomenon is much more problematic with α-IoU and NWD. This makes them poor choices for the evaluation process as they are poorly aligned with human perception.

Similar charts are available in Fig. 8.10, only with SIoU but with various values of γ. In the previous paragraph, we set γ = 0.2. Choosing higher γ values would reverse the trend and produce a criterion even more lenient than humans for small objects. It will also decrease further SIoU's variance.

However, this setting has been chosen to maximize the alignment with human perception. SIoU with γ = 0.2 correlates better with human rating compared with other criteria. As the rating is an ordered categorical variable, we use the Kendall rank correlation to make the comparison. The correlation between the human rating r and each criterion can be found in Tab. 8.2. SIoU with γ = 0.2 and κ = 64 aligns best with human perception and has a low variance (see Sec. 8.3.4). This

showcases the superiority of SIoU over existing criteria. It should be preferred over IoU to assess the performance of models on all visual tasks that employ IoU within their evaluation process. It supports recent ndings that show misalignment between IoU and human preference [START_REF] Stra Orello | Humans disagree with the IoU for measuring object detector localization error[END_REF].

Factor Analysis

To validate our previous experiments, we investigate the potential in uence of external factors on human ratings which could bias the results. Speci cally, we are interested here in several variables: the object size, the presence of contextual information, the expertise, and the age of the participants. Of course, the object size seems to have an in uence on the human rating (according to the previous section), but the idea is to show it quantitatively. To this end, Tab. 8.3 gathers the average rating r under di erent groupings (by object size, presence of contextual information, the expertise of the participants, and age of the participants). In addition, the average value for each criterion is given for each group. IoU value is close to 0.5 for every group as expected (boxes were chosen to have a uniform IoU distribution). However, values of other criteria vary from one group to another. This is especially true for scale-dependent criteria (SIoU and NWD) on di erent object size groups. to give lower ratings. Our goal here is not to infer anything about the reasons for this di erence, but this fact should be kept in mind before drawing any conclusion. In addition, this suggests the need of distinct alignments given what population group is the end user of a system. Having a parameterizable evaluation process (e.g., with SIoU) could help design models that better satisfy their users.

To visualize better the alignment of the various criteria with the human perception, Fig. 8.11 plots the rating values against the criteria value. For clarity, random vertical shifts are added to rating values to distinguish between the values of each variable and data points. From this gure, it is clear that the IoU is not a perfect criterion as a wide range of IoU values is attributed to the same rating value. It seems also clear that contextual information and participant expertise do not introduce much change in the human rating. However, age does have a small in uence on the rating, but this is more blatant with the object size: the average IoU value for a rating r decreases with the object size (this is visible with the black vertical lines in Fig. 8.9). This completely agrees with the statistical test results. SIoU compensates for this trend and produces more aligned averages for the di erent object sizes. NWD has the same e ect but largely reverses the trend in the other direction.

Experimental Results

To support our analysis from Sec. 8.3, we conduct various experiments, mainly on aerial images with DOTA and DIOR datasets. To showcase the versatility of SIoU, we also experiment with natural images on Pascal VOC and COCO datasets. We would like to emphasize that the goal of SIoU, as a loss, is primarily to improve the performance of FSOD and not regular object detection. While SIoU is bene cial for any detection task for evaluation purposes, it is designed to address the extreme challenge of detecting small objects in the few-shot regime. Therefore, most of our experiments focus on the few-shot setting. However, we also report results in regular object detection to display the SIoU as the evaluation criterion in Sec. 8.5.5. They agree with the IoU evaluation and strengthen the conclusions of our experiments. While these results are promising, we must emphasize a few limitations of SIoU and our study. First, SIoU requires a slight tuning to get the best performance, even if that tuning is quite straightforward and mostly depends on the size distribution in the target images. SIoU allows being more lenient with small objects for evaluation (γ ≥ 0), and stricter for training (γ ≤ 0) to prioritize the detection of small targets. However, this is a small price to pay compared to the performance gains obtained on aerial datasets and especially on small objects. Second, another limitation is its application to regular object detection. While this works relatively well with FCOS, it does not show consistent results with other frameworks. It could likely be explained as, in these frameworks, IoU plays a crucial role in the example selection and loss computation. More investigation is required to answer this question. Similarly, Non-Maximal Suppression leveraged IoU as well, and therefore, an implementation of NMS with SIoU instead could also help greatly for the detection task, especially for small objects. Finally, we discussed the alignment of SIoU with human perception for evaluation, i.e., inside the computation of the mAP. However, recent works [START_REF] Oksuz | Localization recall precision (LRP): A new performance metric for object detection[END_REF][START_REF] Jena | Beyond mAP: Towards better evaluation of instance segmentation[END_REF] question the soundness of this metric and propose alternatives that are not necessarily based on IoU. It would be relevant to study them as well and understand how well they align with human perception in order to design more user-oriented detection models.

Conclusion

In this chapter, we highlighted the weaknesses of Intersection over Union both for training and evaluating few-shot object detection models. As an alternative, we proposed Scale-adaptative Intersection over Union (SIoU), a criterion that changes with the object size. We performed an in-depth empirical and theoretical study of several criteria and showed that SIoU has desirable properties for model evaluation that other criteria have not. This is con rmed by a user study that shows a better alignment of SIoU with human appreciation. In addition, we experimented thoroughly with SIoU as a loss function and obtained impressive performance gains on small object detection in the few-shot regime. This is particularly helpful for applications on aerial images, especially as it is compatible with the attention mechanism that we presented in Chap. 6 to improve small object detection in the few-shot regime. Therefore, in this chapter, we present the engineering part of this project, which focuses on applying object detection algorithms inside the CAMELEON system. In particular, we detail the architecture of the CAMELEON system and the constraints associated. In light of these constraints, we can adapt COSE P object detection algorithms for COSE's applications. This mainly involves reducing the size of the models and their inference speed while preserving their detection quality. Of course, this is no easy task, but fortunately, there exist tools to help with this process. We present these tools and their principle brie y, before explaining how they can be leveraged for deploying object detection models on edge devices within the CAMELEON system. Finally, we provide some guidelines for future improvements of optimized detection models and how to deploy few-shot models as well.

P IV

9.1 CAMELEON Aerial Intelligence System 9.1.1 Presentation of the system CAMELEON is a high-resolution aerial camera system that aims to be embedded on various types of carriers such as tactical helicopters (e.g., Airbus H215), patrol or intelligence aircraft (e.g., Airbus A400M or ATL-2), and tactical drones (e.g., Safran's Patroller). Its main objectives are to provide precise 2D and 3D models of geographical areas at needs. Often, satellite imagery is insu cient as it has a low ground resolution, can be outdated, or simply unavailable (e.g., due to weather conditions). In such situations, airborne reconnaissance systems are vital. In practice, these kinds of systems have military applications (theater cartography, tactical intelligence, etc.) but also civil and commercial (maritime surveillance, search and rescue, re monitoring, etc.). Such aerial surveillance systems already exist (e.g., GlobalScanner product by COSE as well). However, their speci cations are often limited in light of the recent progress in sensor resolution and quality. As an example, optronics systems often provide extreme ground resolution but are limited to a small area. CAMELEON aims at improving the compromise between the swath and ground resolution. Speci cally, CAMELEON will embed up to 6 camera sensors (each producing ∼ 100M pixels images). This allows for an extra wide eld of view and signi cant overlapping between images (which is crucial for 3D modelization). This involves dealing with huge amounts of data, which requires carefully designed hardware and software. The CAMELEON system is constituted of two major components, the on-board components and the ground component. Both will be presented in the following sections.

The on-board segment of CAMELEON is the heart of the system, it includes a sensor block, a dedicated computer and a user interface:

-Sensor Block: this unit gathers all the sensors of the system along with the mechanical parts and motors used for controlling their orientations. All the sensors are attached at the core of a 3-axis gimbal suspension. These three axes are controlled by motors to compensate for parasite motions (low and high frequencies) of the carrier and to be able to orient the sensors in any direction. In addition, a forward motion compensation module equips each sensor to improve image quality. Without this, the image quality is degraded due to motion blur induced by the displacement of the carrier during the exposition of the sensor. The Sensor Block also contains an Inertial Navigation System (i.e., an Inertial Measurement Unit coupled with a GNSS sensor), whose goal is to precisely measure the position and orientation of the camera to determine accurately the position of pointed objects on the ground. -On-board Computer: it manages the data stream from the sensors to the User Interface and memory storage. Speci cally, it consists of an host system that controls an FPGA programmed to read into the camera memory bu er and retrieves the images in the host memory. The host is then in charge of storing the images and their associated metadata in a geospatial database.

It also feeds the user interface with the latest image for visualization. Finally, it also controls a set of lightweight GPUs (Nvidia Xavier or Orin) to execute complex treatments such as object detection. A special memory allocation allows for fast data transfer from the host to the kernel to process the image from the system in real-time. This is illustrated in Fig. 9.1, which depicts a simpli ed overview of the system's architecture.

-User Interface: it is an application that displays the images and metadata acquired by the system to the user. It also monitors a set of variables about the ight and the mission.

The Graphical Processing Units (GPUs) selected to be part of the system are the Nvidia edge computing devices Xavier and Orin (see Fig. 9.2). These are lightweight and power-e cient GPUs that can be used as an independent device or as an end-point within a more elaborate system. These

GPUs have the computation capacity of mid-range commercial GPU (e.g., Nvidia RTX 3060) while consuming less than 50W, which is roughly four times as e cient. That makes them particularly well-suited for COSE application as only limited resources are available inside the carrier but a strong computation power is still required. Nevertheless, such an e ciency is not enough to run regular detection models in real-time on 100 of megapixels images. Thus, a lot of e ort is required to adapt the models and their inference, we will elaborate this in Sec. 9.2. The second component of the CAMELEON system is mainly constituted of the ground station. It is a high-power workstation, and its goal is to store and process data between missions. The low resources available during the ight are not su cient for heavy treatments such as 3D modelization of the over own areas. Mission disks lled during the ight inside the on-board segment can be transferred to the ground station for further analysis of the collected data. The ground station can also be used to execute more demanding detection algorithms or advanced treatments (e.g., segmentation, change detection, etc.).

CAMELEON Image Speci cations and Constraints

In addition to the limited computation resource available on-board, CAMELEON has requirements that make object detection even more challenging. To assist the operators on-board e ciently, objects must be detected in real-time. The cameras are set to acquire at least one image every second, and each image is 11600 × 8700. These images are 16-bit Bayer matrices, which amount to roughly 192 megabytes per image. This means that more than 1 GB of data is produced every second in the system when all the cameras are used. However, the current prototype is designed with only one camera, which makes the real-time detection a little less challenging. Aside from the model inference, dealing with such a large amount of data requires carefully designed datastreams between the various parts of the system to retrieve, store and display the images produced by the cameras. This is what motivated the use of a Field Programmable Gate Array (FPGA) to orchestrate the transfer between the camera and the host. transfer time, pre-and post-processing which must also t under the one-second time limit. Furthermore, the high-end GPU used for these benchmarks (available on YOLOv5's repository1 ) have much more computing power than the Nvidia Xavier selected for CAMELEON (10 times more according to the theoretical capabilities on Nvidia's website). Therefore, one must adapt the detection models signi cantly to ful ll CAMELEON's requirements. This can be done in two di erent ways:

rst reducing the model size and second accelerating the inference. Both approaches are employed in CAMELEON's prototype, and they will be discussed in the following two sections.

Reducing Object Detection Model Size for Edge Computing

The most straightforward way to increase the throughput of a detection model is to reduce its size.

However, this often comes at the cost of lower accuracy. In this section, we analyze to nd the speed/accuracy tradeo of YOLOv5 [346] on aerial images. Then we propose a simple knowledge distillation approach to improve this tradeo and achieve higher detection quality at a xed size.

Object Detection Accuracy/Speed Tradeo

First, we compare the speed/accuracy tradeo for multiple detection frameworks on natural images. The results of this comparison can be found in Fig. 9.3. To make this gure, we collected the performance metrics and model size information directly from the articles presenting the various detection methods. We select the mAP with various IoU thresholds on MS COCO. This is rather simple as this is the most common evaluation benchmark in the detection literature. Then, the most relevant way to assess the speed of the models is to measure the latency (in ms) which represents the time required to process one image. However there are multiple complications with this measure: it depends greatly on the hardware used, the size of the input images and even the version of the library used during inference (in a less pronounced manner). In addition, the latency is not always reported in the articles which makes the task even more di cult. Thus, we leverage a surrogate for the latency: the total number of parameters in the model. Of course, it does not correlate fully with the latency, but it is a sound approximation, and it is easier to collect. In Fig. 9.3, we plot the performance against the number of parameters (left) and against the latency (right). There are missing values in the latency plot as this measure was not reported in the original articles. We compare 8 distinct detection frameworks: YOLOv5 [346], CornerNet [START_REF] Law | Cornernet: Detecting objects as paired keypoints[END_REF], CenterNet [START_REF] Duan | Centernet: Keypoint triplets for object detection[END_REF], Mask R-CNN [START_REF] He | Mask r-cnn[END_REF],

DETR [START_REF] Carion | End-to-end object detection with transformers[END_REF], FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF], SwinTransformers [START_REF] Liu | Swin transformer: Hierarchical vision transformer using shifted windows[END_REF] (based on Cascade R-CNN), DynamicHeads [START_REF] Dai | Dynamic Head: Unifying Object Detection Heads with Attentions[END_REF],

and Di usionDet [START_REF] Chen | Di usiondet: Di usion model for object detection[END_REF]. The conclusion of this comparison is blatant: YOLOv5 has a much better speed/accuracy tradeo than other detection frameworks. Of course, our comparison is not exhaustive, yet it includes recent one-stage detectors that are fast and perform well. We are aware that very recent developments in the YOLO family outperform YOLOv5 (e.g., YOLOv82 ); however, we could not include them in our comparison. The comparison from Fig. 9.3 demonstrates the superiority of YOLOv5 over other approaches on MS COCO. However, it is necessary to con rm that the same behavior is observed on aerial images as well. To this end, we train the di erent versions of YOLOv5 on DOTA and DIOR datasets and plot similar curves as in Fig. 9.3. The resulting plots are available in Fig. 9.4. In this analysis, we include even smaller models than the nano version of YOLOv5 (YOLOv5-N). We call these models YOLOv5-P and YOLOv5-F (for Pico and Femto following the nomenclature from YOLOv5). These models have respectively 0.68 and 0.32 millions of parameters which is much lower compared to traditional detection models. Nevertheless, strong performance is achieved on DOTA and DIOR datasets (see Tab. 9.1). Also, YOLOv5-X is not included in the analysis as it is too large for COSE's application.

From Fig. 9.4 and Tab. 9.1, it is clear that the detection performance is strongly correlated with the number of parameters of the model and that this connection holds also for very small models (< 1M parameters). However, it seems that the performance drops faster under a certain model size.

This is observed both for DOTA and DIOR, even though it is more pronounced with DIOR. We also observed that the latency plot does not follow identically the parameters plot. For instance, with Given these tradeo curves, YOLOv5-N is the most promising model as it is closest to the top-left corner of the plots. However, it is relevant to investigate other model sizes and verify how well they comply with COSE's constraints.

Knowledge Distillation

Of course, the smaller the models, the lower the detection performance. However, there exist techniques to boost the performance of any network when we have access to a similar but larger model with increased performance. This is called Knowledge Distillation (KD). The main principle behind this technique is to train a student model to mimic a teacher which is often larger and has better performance. It was rst introduced by Hinton et al. [START_REF] Rey Hinton | Distilling the knowledge in a neural network[END_REF] in 2015. Originally, it consisted in training the student model with an additional loss measuring how close the logits from the student and the teacher are. Then, it was extended multiple times, with for instance intermediary layer activation distillation [START_REF] Romero | FitNets: Hints for Thin Deep Nets[END_REF], relational distillation [START_REF] Park | Relational knowledge distillation[END_REF], adversarial distillation [START_REF] Wang | Kdgan: Knowledge distillation with generative adversarial networks[END_REF] or multiple teachers [START_REF] Liu | Adaptive multi-teacher multi-level knowledge distillation[END_REF]. Most of these techniques are designed for classi cation applications and provide limited performance boosts for detection models. Fortunately, knowledge distillation can be extended for ne-grained tasks and in particular for object detection [START_REF] Wang | Distilling object detectors with ne-grained feature imitation[END_REF][START_REF] Li | Mimicking very e cient network for object detection[END_REF][START_REF] Chawla | Data-free knowledge distillation for object detection[END_REF][START_REF] Wei | Quantization mimic: Towards very tiny cnn for object detection[END_REF][START_REF] Wang | Gan-knowledge distillation for one-stage object detection[END_REF]. For more details about existing knowledge distillation methods, we defer the reader to this complete survey [START_REF] Gou | Knowledge distillation: A survey[END_REF].

As a rst try with KD, we applied Fine-grained Feature Imitation (FFI) [START_REF] Wang | Distilling object detectors with ne-grained feature imitation[END_REF] to improve the training of YOLOv5-P and YOLOv5-N with larger teachers trained on DOTA and DIOR. FFI proposes an additional loss function that measures the disparity between the teacher and student feature maps, but the computation only includes regions close to a ground truth annotation: Here, M is the imitation mask which is 1 where in neighboring regions of each ground truth annotation and 0 elsewhere. H and W represent the height and width of the feature map respectively. f adapt is a mapping function that converts the feature of the student, denoted s i,j , to the same size as the teacher's ones (in terms of the number of channels), denoted t i,j . Indeed, features maps of the student often have fewer channel, which prevents direct comparison. The intuition behind this loss is that student outputs should only match teacher outputs in regions where there is an object of interest. The background is too noisy and the student model may not have the capacity to mimic the teacher everywhere, it should focus only in relevant regions.

L F F I = 1 2m W i=1 H j=1 M i,j f adapt (s i,j ) -t i,j 2
The results obtained with aerial images do not agree with the performance gains reported in the original paper [START_REF] Wang | Distilling object detectors with ne-grained feature imitation[END_REF] for natural images (see Tab. 9.2). Signi cant performance drops are observed with distillation on aerial images. Similar drops are observed for YOLOv5-P and YOLOv5-N both on DOTA and DIOR datasets. KD methods designed for natural images may not be well adapted for aerial images, one reason for this could be the presence of much smaller objects which have smaller and noisier representations inside the feature maps. In a sense, that could be linked to the di culty of applying FSOD methods on aerial images, as described in Chap. 4. These results deserve to be investigated further in future work as distillation is a promising direction for detection improvements.

Inference Acceleration with TensorRT

TensorRT is a tool provided by Nvidia to optimize the inference of deep learning models on Nvidia's hardware. Recent Nvidia GPUs have dedicated modules for deep learning inference, called Tensor Cores. They contrast from CUDA cores, which are principally made for parallel computing. TensorRT unlocks the potential of the tensor cores by generating an engine that can be run most e ciently on a speci c GPU. This di ers from the vanilla Python inference (i.e., with any deep learning library), which calls CUDA kernels that are executed on the CUDA cores of the GPU. TensorRT also leverages the CUDA cores and thus leverages the maximum GPU capabilities. In addition, TensorRT proposes several optimization tricks for increasing the inference speed even more. We detail such tricks in the following paragraphs and explain how they can be used for object detection models.

Speci cally, TensorRT takes as input a neural network model from any common library in a suitable format such as Open Neural Network Exchange (ONNX) and converts it into an engine. This engine contains the initial network along with speci c instructions about how to run inference in an e cient way given the available hardware. TensorRT provides APIs for various programming languages (e.g., Python and C++) to run the engine directly from any application.

Operation Fusion and Scheduling

TensorRT merges di erent layers and re-organizes the forward pass of the model. Speci cally, the forward pass of a model can be represented as a computation graph (see Fig. fused vertically. Instead of creating distinct CUDA kernels for each layer, only one kernel is created, saving a lot of time, particularly in data transfer. This is illustrated in Fig. 9.5 (right) where, convolution, bias, and activation layers are merged as "CBR" blocks. Then, layers can also be merged horizontally. Horizontal fusion happens when similar operations are in parallel and have the same input (e.g., three distinct parallel 1×1 convolutions as illustrated in Fig. 9.5, right). This also reduces the creation of unnecessary CUDA kernels and maximizes the utilization of the resources. Finally, when there are parallel paths in the computation graph, the order in which the di erent branches can be irrelevant. In this case, some clever scheduling can maximize the utilization of the GPU and speed-up inference. This scheduling is not trivial as it is subject to the memory constraint of the hardware, and all parallel operations cannot be done at once.

TensorRT searches for optimal scheduling and fusion according to the available hardware and the input size that will be used at inference (e.g., batch size and image size). This implies that the input size is xed for a given engine, but it is a small price compared to the bene ts of the TensorRT conversion. TensorRT does have a variable size option, but it does not seem to be compatible with the reduced precision that we will detail in the following paragraphs.

Floating point precision

Then, TensorRT reduces the precision of the model's weights and activations. In Python, common libraries store models and images as oat numbers which take 4 bytes of memory (i.e., 32 bits, denoted F32). TensorRT can convert a model with half-precision oats (denoted F16), which only take 16 bits of memory. This reduces the size of the model by a factor of two, but most importantly it reduces the computation power required to perform a forward pass as well. Indeed, arithmetic operations are faster with half-precision numbers as they require fewer basic operations. Thus, it sensibly reduces the overall inference time. This trick is becoming quite common as well for training neural networks; however, mixed precision (i.e., keeping some parts of the models as regular oats) is required to prevent convergence issues [START_REF] Micikevicius | Mixed precision training[END_REF]. TensorRT instead converts the whole model into Hence, the quantization and dequantization functions can be written as:

x = Q(x) = S(x) = ax + b , (9.3)
x ≈ S -1 (x) = 1 a (x -b).

This can be further simpli ed when both the weights interval and the quantized interval are centered (e.g., α = β): S -1 (x) = 1 a x. In this case, one can easily see that the matrix multiplication, the basic operation of neural networks' forward pass can be computed using mostly integer multiplications and additions. If we de ne three matrices X ∈ R n×p , Y ∈ R p×m , and Z ∈ R n×m , such that Z = XY , then we have:

z i,j = k x i,k y k,j ≈ k S -1
x (x i,k )S -1 y (ȳ k,j ) = 1 a x a y k x i,k ȳ k,j , ( Similarly, activations of the model are also quantized per layer. However, the range of the activation highly depends on the input of the model and cannot be known in advance. To alleviate this, a calibration cache can be computed from a calibration set. TensorRT automatically builds this cache by running a forward pass on all images of the calibration set. Of course, this set must contain COSE P images that are similar to the images on which the model will be deployed. Concerning COSE, this complexi es the domain adaptation problem as the calibration of the model should be adapted to the domain. Even if it only requires non-annotated images to compute the calibration cache, it can sometimes be challenging to obtain an appropriate calibration set due to con dentiality constraints.

The resulting calibration cache gathers information about the quantization of the activation of the di erent layers of the model. It is then used during inference, which slightly increases memory usage during inference.

We brie y presented the linear quantization technique, yet TensorRT also provides more elaborated quantization to minimize the loss of information. For instance, it has an Entropy Calibration technique that adapts the density of bins according to the density of weights values. It is very similar to entropy coding techniques that assign smaller code to the most frequent symbols.

C++ Implementation

Finally, the TensorRT conversion allows for using C++ as a backend instead of Python. This is not strictly speaking a TensorRT trick, but it signi cantly reduces the inference time. As TensorRT provides a C++ API, the whole inference pipeline can be written in C++ as well. C++ is known to be much faster than Python and implementing pre-and post-processing in this language can speed up the inference.

Object Detection Pipeline for CAMELEON Prototype

Now that we have discussed various improvements for accelerating the inference time of the detection model, we present how we make use of them speci cally for COSE application. First, we select the most interesting models from our analysis in Sec. 9.2: YOLOv5-N and YOLOv5-P. Both achieve fast inference while preserving high detection performance. We then perform the conversion in a

TensorRT engine to speed up the inference. This was done only for the model trained with DOTA dataset, but it would be identical with DIOR.

Deployment with TensorRT

We started with the nano version as the goal is to satisfy the time constraint of processing one image per second. If it does not t under the time constraint, we would do the same with the pico version.

With the help of TensorRT, we reduce the precision of the model into 8-bits integers. The calibration is performed with DOTA validation images. However, to nd the most e cient utilization of the GPUs, it was necessary to explore what were the best combinations of batch size and input image size. CAMELEON's images have 11600 × 8700 pixels and doing the inference on the entire image at once demands too much GPU memory, even with our smallest models. The images must be tiled before being fed to the model. The tiling has a major downside: it can cut object in half which make them more di cult to detect. Adding some overlapping between tiles can prevent this but it also increases exponentially the amount of data to process (see Fig. 9.6). In addition, having to many tiles requires processing multiple batches. This is equivalent to performing the inference multiple times adding a small overhead at each iteration. Thus, we want to minimize the overlap and have tiles as large as possible.

We tried several combinations of batch size and tile size, but this is a tedious task as an entire engine must be created for each setting. We did not mention it before, but the engine generation is a heavy optimization process that requires several hours on the Xavier GPUs. Based on these insights and multiple practical trials, we set the tile size to half the height of a CAMELEON image: 4350, with a batch size of 1. This maximizes the utilization of the GPU, requires few inferences and limits object splitting. This results in 6 tiles organized as shown in Fig. 9.7 with slight horizontal overlap.

Overlapping also induces the need for NMS after the detection. Indeed, in areas that will be processed multiple times, detections can be duplicated. To remove these undesired boxes, an NMS operation must be performed at the whole image level. As it scales in O(M 2 ) with M the number of detected boxes, it can become expensive. In addition, it depends on the number of detected objects in the image, which can be considerable in CAMELEON-size images. Fortunately, there exist fast GPU implementations of the NMS that can be leveraged easily. It could be further optimized by rst performing the NMS on each tile individually and then only in the overlapping areas to drastically reduce the number of boxes. Nonetheless, the NMS computation time is not prohibitive as it is and the test images contain far more objects than what will be encountered in real-case applications. Therefore, we choose not to spend e ort on improving the NMS process. The preand post-processing are less expensive than the inference itself, but they still consume a signi cant amount of time. To accelerate them, we choose to implement them in C++ as it is much faster than Python, especially for data management and transfer.

Detection Performance Evaluation and Pro ling

Now that we have a TensorRT engine, we must validate its capacity, both in terms of inference speed and detection quality. Indeed, the precision reduction of the model often leads to reduced performance.

First, we check if the inference is fast enough to comply with COSE's constraints. To this end, a simple pro ling of the various steps of the execution is realized. Speci cally, we isolate the data copy, tiling, pre-processing, inference and post-processing. The results of this pro ling can be found in Tab. 9.3. The timings are averaged over 100 runs on an Nvidia AGX Xavier in high consumption mode (30W + overclocking). It results in the overall processing of one image in roughly 750ms. This falls under the 1s barrier and thus ful lls the application constraints. Larger versions of YOLOv5 do not respect this constraint. However, it would be relevant to investigate smaller ones to get degraded performance mode to process more than one image per second. The current prototype has only one sensor, but the nal system will y with 6 identical cameras. While it may not be realistic to process all these images in real-time it might be useful to have access to faster models able to manage the image from more than one sensor. We also experiment with a new generation of embedded GPU: the Nvidia AGX Orin, which recently replaced the Xavier. Compatibility issues between TensorRT and the new GPU prevent us from experimenting more with it, but we were able to create a similar engine for the Orin. This produces signi cant inference speed gains as we process an entire image in a little less than 500ms. Yet, the Orin consumes more power than the Xavier (50W).

To assess how the precision reduction changes both the inference speed and detection performance, we compare multiple deployment strategies with the YOLOv5-N model. are available in Tab. 9.4. First, there is an impressive speed gap between the TensorRT engines and Pytorch models: TensorRT engines are much faster. This is expected given all the optimizations conducted and the relative slowness of Python. However, it is worth noting di erent behaviors between the two GPUs. Of course, the RTX 3090 is faster, but changing the engine precision from F32 to F16 does not produce similar gains for the AGX Xavier. This is explained by the change of microarchitecture between these two GPUs. The AGX Xavier is based on the Volta architecture while the RTX 3090 leverages the newer Ampere architecture. In particular, Ampere introduces a new generation of Tensor core, speci cally designed for F32 computations. Reducing the precision of the models generates a slight overhead and higher gains are observed when the GPU utilization is higher, i.e., when using larger images or increased batch size. This is why we report the results both for 512 and 4096 image widths. Inference gains are more important with larger image sizes, fortunately for our application. Then, we also report the mAP for each model. We observe no performance drop when switching from F32 to F16. But, there is a slight decrease of mAP with the quantization in INT8. This is expected as a lot of approximations are made in the computations. On the other hand, we also observe a performance drop with the TensorRT conversion, with the F32 engine this is not expected as no approximation should be done. While the drop is acceptable, it should be investigated in depth to better understand its origin. Finally, we report some qualitative results of the INT8 inference on a CAMELEON-size image in Fig. 9.8. The inference on the whole image was performed in less than a second on an AGX Xavier.

More experiments are planned in future work to nd an even better compromise between inference speed and detection quality, especially with the Orin GPUs. However, the compromise attained with YOLOv5-N is already satisfactory from an industrial perspective and will be integrated into the rst CAMELEON prototype. The connection with the CAMELEON user interface Fig. 9.9 has already been done and the models wait for the rst ight to be tested on real images.

Conclusion

In this chapter, we have presented some tools and tricks that are leveraged to greatly accelerate the inference of neural networks. Thanks to these tools, we are able to achieve real-time detection on images of hundreds of megapixels, embedded on a low-consumption GPU. Of course, better compromises could be found with a more complete study of the YOLOv5 framework and especially by exploring the in uence of the model size on the performance with greater granularity. Aside from nding a better compromise, it would be relevant to investigate deeper knowledge distillation and unlock its full potential for object detection. It is also required to explore new detection frameworks (e.g., YOLOv8), in particular with the recent AGX Orin. Its improved computation power could allow for larger model sizes and increased performance while satisfying the time constraint. Finally, this deployment has been conducted for regular detection models, it should be done as well for Few-Shot detection models, even though it complexi es greatly the analysis and the creation of TensorRT engines. While it is unrealistic to deploy attention-based models in this manner (due to increased inference time), ne-tuned models are much more adapted. It would increase the adaptation time by adding the engine creation after ne-tuning, but the inference will remain unchanged. Of course, it would be necessary to study the in uence of the model size on the performance and see if that is compatible with CAMELEON's requirements.

C P

In a nutshell, we rst summarize the various contributions made to the Few-Shot Object Detection eld. Then, we take a step back from these contributions and discuss what are the most promising research directions that should be followed in future work. Finally, we present the remaining industrial challenges that COSE will face before achieving robust and adaptable object detection in an embedded environment.

Contribution Summary

In the rst part of the manuscript we have reviewed thoroughly the literature about Object Detection, Few-Shot Learning and especially Few-Shot Object Detection. These three domains are the foundations of this PhD project. The knowledge presented in the corresponding chapters is crucial for the development of future FSOD algorithms and to meet the industrial needs of COSE.

These three elds are growing rapidly and it is necessary to remain up-to-date with state-of-the-art both for academic and industrial research. It is especially important for FSOD as it is a very recent problem and lacks consensus on how to address it. Even if the FSOD literature is growing, it is still a small eld and most contributions are primarily designed for natural images. However, it is not guaranteed that such techniques will perform well on other kinds of images. In Chap. 4, we highlight especially that with an in-depth analysis of the FSOD performance on aerial images, the performance drops signi cantly when the methods are applied to these kinds of images. The main reason behind this phenomenon is the smaller object size in aerial images. Even if small objects are already di cult for regular object detection, the challenge is much greater in the few-shot regime.

In addition, Part I highlights the organization of the FSOD literature into three kinds of approaches.

We divided Part II into three chapters accordingly, each focusing on a di erent FSOD approach: metric learning, attention-based methods and ne-tuning strategies. Speci cally, in Chap. promising results and should be extended in future work. All detection models heavily rely on the Intersection over Union (IoU) for training and evaluation. However, we showed in Chap. 8 that it is not an optimal choice, especially when dealing with small objects. Thus, it becomes critical when applying FSOD to aerial images. Therefore, we proposed Scale-adaptative Intersection over Union to replace IoU. It signi cantly improves the training of few-shot object detectors, as it allows for a better balance between small and large objects. In addition, it aligns better with human perception and is then a better choice for the evaluation of object detectors. Then, with a more industrial mindset, we optimized and deployed several object detectors inside the CAMELEON prototype. We explained our process, the tools leveraged, and the compromises made in Chap. 9. This chapter focuses only on regular object detection as it is a rst step before deploying more complex models able to generalize either to new classes or new domains.

Finally, we take a step back and compare the three approaches that we proposed in this thesis: Prototypical Faster R-CNN, XQSA and FSDi usionDet. Each has its pros and cons, even if PFRCNN does not perform well it can be adapted at test time which is a highly desirable property. On the contrary, FSDi usionDet is less exible but achieves signi cantly higher performance. This comparison is available in Tab. 10.1, it compares the three methods on various criteria, grouped into four categories: Performance, Flexibility, Training and Deployment. These categories encompass the desired capabilities for a detector model inside the CAMELEON system.

Future Research Tracks

In this PhD, various directions have been explored for improving Few-Shot Object Detection. We believe that we have provided signi cant contributions and answered relevant questions in the FSOD eld. However, our analysis raises new questions and problems. First, we have proposed several improvements toward the detection of small objects in the few-shot regime. Nonetheless, there is still a signi cant performance gap between small and large objects and more e ort should be put in this direction. To this end, we have explored the attention mechanisms and loss function design, but it has a drawback, it requires the setting of two parameters and their optimal choices require some trials and errors. It would be of great use to come up with an automatic strategy that could provide the optimal parameters for a given problem or dataset.

The main motivation of Few-Shot Learning is to mimic the human ability to learn from very few examples. Of course, we are still far from solving FSOD, and a lot of e orts are still needed to close the gap with fully supervised learning, not to mention human-level perception. Nevertheless, the improvements made through this project encourage the study of more complex yet more realistic settings. One is particularly relevant for COSE's use case: Cross-Domain FSOD. This problem is still mostly untouched in the literature. We conducted some early experiments in this direction in Chap. 7 and demonstrated promising results. However, dedicated designs are required to get real COSE P improvements as applying FSOD methods directly on this task is certainly not optimal. Many directions are worth a try, taking inspiration from classi cation methods, e.g., generative modeling or discrepancy-based adaptation. The transductive inference is another promising approach, it makes some additional assumptions, but it is still realistic at least from COSE's perspective The transductive approach has not been explored for detection and could bring desirable properties to the FSOD eld, especially test-time adaptability which is still very challenging.

Remaining Industrial Challenges

Now from COSE's perspective, plenty of challenges are still to be addressed. First, the current CAMELEON prototype only has a single camera but the nal product is meant to have ve more. It would probably be quite di cult to achieve real-time detection on all images. Of course, we could continue optimizing the models and nd better speed/accuracy tradeo s. The recent advances in the YOLO family are promising, and substantial gains are achieved with newer GPUs. However, it may not be su cient to multiply the throughput of the detection pipeline by six. Fortunately, as there is some overlapping between the images, it is not required to process all six data streams independently. Instead, we could determine the overlapping areas to avoid processing them multiple times. But that is not trivial and requires image registration techniques which are also expensive, especially at the size of CAMELEON's images.

While the current prototype achieves satisfactory detection performance, there is still room for improvement. Our attempt to apply Knowledge Distillation (KD) to aerial images has been fruitless.

Nevertheless, distillation is a promising direction to improve detection performance while maintaining a xed computation budget. Some e ort should be spent on developing novel KD methods that are also bene cial with aerial images and small objects.

Currently, only regular object detection algorithms have been deployed inside the prototype. While this is certainly a major step, it is still limited by the training classes and domain. As it is not possible to know in advance the kind of images that will be encountered during missions, mainly because of con dentiality constraints, we cannot guarantee the quality of the detections in operation. ) and the function h : x → x p 1,3 is monotonically increasing. Hence, when γ ≤ 0 the order is preserved.

Case 2: γ > 0

We have p 1,3 > p 1,2 as τ 1,3 ≤ τ 1,2 and γ > 0. Therefore, p 1,3 = p 1,2 + ε, with ε > 0.

In this case, the order does not always hold, counter-examples can be found. However, it is useful to study the conditions for it to hold: can be large.

IoU(b 1 , b 2 ) p 1,2 ≤ IoU(b 1 , b 3 ) p 1,
The order-preservigness property holds in many cases. This is sensible as IoU is still a reliable metric that has been used extensively for the training and evaluation of detection models. However, in the rare cases where this order is broken, the IoU(b 1 , b 2 ) and IoU(b 1 , b 3 ) are close, so IoU does not discriminate much between the boxes. On the contrary, SIoU prefers the smallest one (or largest one, according to the choice of γ). This stronger discrimination is probably bene cial for training and evaluation.

A B

T GI U' D A

In this appendix, we give a proof of the formulas of the probability density function, expected value, and variance of GIoU. We also provide some non-closed-form expressions for other criteria.

Proposition 1 (GIoU's distribution) Let b 1 = (0, y 1 , w 1 , h 1 ) be a bounding box horizontally centered and b 2 = (X, y 2 , w 2 , h 2 ) another bounding box randomly positioned, with X ∼ N (0, σ 2 ) and σ ∈ R * + . Let's suppose that the boxes are identical squares, shifted only horizontally (i.e., w 1 = w 2 = h 1 = h 2 and y 1 = y 2 ). Let Z = C(X), where C is the generalized intersection over union. The probability density function of Z is given by:

d Z (z) = 4ω (1 + z) 2 √ 2πσ exp - 1 2 ω(1 -z) σ(1 + z) 2 . (B.1)
The two rst moments of Z exist and are given by:

E[Z] = 2 π 3/2 G 2,3 3,2 2a 2 0 1 2 1 2 1 2 0 , (B.
2)

E[Z 2 ] = 1 - 8a √ 2π + 16a 2 π 3/2 G 2,3 3,2 2a 2 -1 1 2 -1 2 1 2 0 , (B.3)
where G is the Meijer G-function [START_REF] Beals | Meijer G-Functions: A Gentle Introduction[END_REF] and a = σ ω .

Proof First, in the setup described in Proposition 1, GIoU can be expressed in terms of the width of the boxes ω and the shift in between x. Let's call this function C:

C : R → [-1, 1] x → ω -|x| ω + |x|
The shifts are sampled from a Gaussian distribution: X ∼ N (0, σ 2 ), therefore we are interested in the distribution of the variable Z = C(X).

B T GI U' D A

The cumulative density function of Z is given by:

F Z (z) = P (Z ≤ z) = P ( ω -|X| ω + |X| ≤ y) = P (ω 1 -z 1 + z ≤ |X|) = 2P (ω 1 -z 1 + z ≤ X) = 2(1 -F X (ω 1 -z 1 + z )) = 2(1 -F X (g(z)))
With g(z) = ω( 1-z 1+z ). Hence, the density function of Z can be derived by taking the derivative of F Z :

d Z (z) = d dz F Z (z) = -2g (z)F X (g(y)) = 4ω (1 + z) 2 √ 2πσ exp - 1 2 ω(1 -z) σ(1 + z) 2
To determine the rst and second moments of Z, we make use of the change of variable formula: From line 2 to 3, we used the parity of function C, between 6 and 7, a change of variable u = x/ω is done and a is set to σ/ω. Finally, in the second-to-last line, we identify a Meijer-G function [START_REF] Beals | Meijer G-Functions: A Gentle Introduction[END_REF] evaluated at 2a 2 . Unfortunately, there exist no closed-form of the integral +∞ 0 1 1+u e -u 2 2a 2 dx. In this case, a Mellin transform of this integral allows recognizing a Meijer-G function. For other criteria, their rst two moments cannot be expressed in a similar closed form. That is why we only provide the theoretical expressions of the expectation and variance of GIoU.

E[Z] = E[C(X)] =
A similar derivation leads to the expression of the second moment of Z: From line 2 to 3, we again use the parity of C, from 4 to 5, an integration by parts is done, and nally, from 5 to 6, we apply the change of variable u = x/ω. Once again, we get an integral that does not have any closed form but can be expressed by another Meijer-G function.

E[Z 2 ] = +∞ -∞ C(x) 2 d X (x) dx = +∞ -∞ 1 √ 2πσ ω -|x| ω + |x| 2 e -x 2 2σ 2 dx = 2 √ 2πσ +∞ 0 ω -x ω + x 2 e -x 2 2σ 2 dx = 1 - 8ω √ 2πσ +∞ 0 x (ω + x) 2 e -
For completeness, we recall here the de nition of the Meijer-G function: where L is the integration path and Γ is the gamma function. m, n, p and q are integers while a j and b j are real or complex numbers. There are some constraints on these parameters, but we do not detail them here, they can be found in [START_REF] Beals | Meijer G-Functions: A Gentle Introduction[END_REF].

  et soutenue le 03-10-2023 par : P LE JEUNE en vue de l'obtention du grade de D I École doctorale Galilée (ED 146), Université Sorbonne Paris Nord Laboratoire de Traitement et Transport de l'Information (L2TI, UR 3043) COSE Few-Shot Object Detection (FSOD) evaluate their methods on natural images

  Detection, Few-Shot Learning, Few-Shot Object Detection, Cross-Domain Adaptation, Deep Learning, Computer Vision, Intersection over Union, Attention Mechanism, Di usion, Query-Support Alignment R La plupart des contributions en Détection d'Objets Few-Shot (FSOD) évaluent leurs méthodes uniquement sur des images naturelles, ne garantissant pas la transférabilité de leur performance à d'autres types d'images. Nous démontrons ceci avec une analyse des méthodes FSOD existantes sur des images aériennes et observons un large écart comparé aux images naturelles. Les petits objets, plus nombreux dans les images aériennes, sont responsables de cet écart. Ainsi, nous proposons d'améliorer la détection des petits objets avec un mécanisme d'attention dédié. En plus, nous proposons un nouveau critère de similarité pour boîtes englobantes, adaptatif à la taille. Il améliore l'entraînement et l'évaluation des modèles FSOD, en particulier pour les petits objets. Nous contribuons aussi au FSOD classique avec deux approches distinctes basées sur le metric learning et le ne-tuning. Des résultats impressionnants sont obtenus avec cette dernière méthode, ce qui encourage son application à des scénarios plus complexes comme la détection Few-Shot Cross-Domain.

G

  -Clé-Détection d'objet, Apprentissage profond, Apprentissage frugal, Adaptation au domaine, Mécanisme d'attention, Di usion, Intersection over Union, Alignement query-support i List of Symbols f Backbone of a visual model (for classi cation or detection). Often implemented as a large CNN or Transformer model. D base A set containing training examples of the base classes. b A ground truth bounding box: b = [x, y, w, h]. b A predicted bounding box: b = [x, ŷ, ŵ, ĥ]. c A semantic class, c ∈ C. l A vector of classi cation scores, l ∈ [0, 1] C . C A set of classes. C base Base class set. C novel Novel class set. y A detection label, constituted of a bounding box and a class label (or a classi cation score vector). h Detection or classi cation head, often implemented as a lightweight CNN or MLP. I An image. K Number of shots, i.e., the number of examples available for a novel class. F(•, θ) General purpose model with parameters θ. N Number of novel classes in the few-shot setting. g Neck of a detection model, it generally consists of a Feature Pyramidal Network. D novel A set containing examples of the novel classes and their annotations, used for adaptation and/or ne-tuning, a.k.a support set. θ Parameters of a model, e.g., a neural network. N p Number of proposals boxes in Di usionDet.

  1 https://www.cose.fr/ I (a) GlobalScanner Camera and Observation Unit. (b) Strike stabilization arm mounted on a Gazelle helicopter. (c) POD Xplorer next to SAFRAN's Patroller at 2019 Paris Air Show.

Figure 1 . 1 :

 11 Figure 1.1: Illustration of the three main products developed at COSE: GlobalScanner, Strike and POD Xplorer.

Figure 1 . 2 :

 12 Figure 1.2: Description of Geospatial Intelligence (GEOINT).

  First, we conduct in Part I a thorough review of the literature about object detection, few-shot learning and nally fewshot object detection. Then, we propose three distinct FSOD approaches in Part II. This part includes experiments in the Cross-Domain setting, inside Chap. 7. Our experiments mainly focus on publicly available aerial datasets due to the lack of private datasets inside the company. These datasets contain detection annotations and will be presented in Chap. 2. Part III presents an alternative to the Intersection over Union, a bounding box similarity measure extensively employed in Object Detection. Its use for both model evaluation and training are discussed in Chap. 8. Finally, Chap. 9

  P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation Learning for Few-Shot Object Detection on Aerial Images, " 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110. Chap. 6: Attention Framework for Fair FSOD Comparison Fair comparison is extremely challenging in the Few-Shot Object Detection task as plenty of architectural choices di er from one method to another. Attention-based approaches are no exception, and it is di cult to assess which mechanisms are the most e cient for FSOD. In this chapter, we propose a highly modular framework to implement existing techniques and design new ones. It allows for xing all hyperparameters except for the choice of the attention mechanism. Hence, a

  National Conference Articles P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles. " GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France. P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France. Submitted Articles Ƽ P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal. Ƽ P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV).

  Cette thèse a pour origine la collaboration entre le laboratoire L2TI de l'Université Sorbonne Paris Nord (USPN) et la société COSE. Le L2TI a été fondé en 1998 et est un membre de la Fédération de Recherche MathSTIC du CNRS (FR 3734) qui inclut également deux laboratoires CNRS: le Laboratoire Analyse, Géométrie et Applications (LAGA), UMR 7539 et le Laboratoire d'Informatique de Paris Nord (LIPN), UMR 7030. Ces laboratoires sont tous rattachés à l'Institut Galilée. Deux équipes de recherche cohabitent dans le L2TI. D'abord, l'équipe Multimédia, qui se concentre sur le traitement et l'analyse de l'information visuelle et audio. Ensuite, l'équipe Réseaux, qui travaille sur le transport de l'information et les communications. Ce projet de thèse s'inscrit logiquement dans l'équipe Multimédia. COSE 1 est une PME innovante d'environ 20 salariés. C'est un fournisseur de rang 1 de l'état dans le secteur de l'aéronautique et de la défense. COSE est né en tant que startup de l'INRIA dans I les années 90 et la recherche est toujours au coeur de son processus industriel. Bien que relativement petite, COSE possède des équipes pluridisciplinaires de haut niveau dans des domaines tels que la mécanique, l'électronique, la navigation, l'automatique et les systèmes embarqués. La taille de COSE lui confère une agilité et une e cacité remarquable comparée à ces principaux compétiteurs. Cet avantage permet à l'entreprise de créer des partenariats forts avec les acteurs majeurs de l'aéronautique et de la défense. COSE développe, produit et maintient des systèmes de renseignements aéroportés et des équipements embarqués en tout genre. Ces produits sont principalement destinés à un usage militaire et sont donc soumis à des critères de qualité stricts. La relation entre les forces armées et COSE est gérée par la Direction Générale de l'Armement (DGA) qui est de fait l'un des principaux clients de COSE. COSE a pour l'instant trois produits principaux dans sa gamme que les forces françaises utilisent pour di érentes missions (voir Fig. 1.1) :

  proposera également une amélioration complète du logiciel d'observation et notamment en ce qui concerne l'analyse et le traitement des images. La quantité d'images obtenues chaque seconde par le système sera trop importante pour être analysée par un seul photo-interprète. De plus, les moyens de communication standards n'ont pas un débit su sant pour transmettre les images en temps réel. Ainsi, il est nécessaire d'extraire les informations stratégiques des images, automatiquement et à bord. CAMELEON doit donc être doté d'algorithmes intelligents et e caces a n d'extraire les informations pertinentes en temps réel. Cette thèse s'inscrit dans la refonte logicielle de CAMELEON et tente de répondre aux contraintes du projet. Les informations extraites des images sont sou-Caméra GlobalScanner et son poste d'observation. (b) Le bras de stabilisation Strike monté sur un hélicoptère Gazelle. (c) Le Pod Xplorer à côté du futur drone Patroller de SAFRAN au salon du Bourget 2019.

Figure 1 . 1 :Figure 1 . 2 :

 1112 Figure 1.1: Illustration des trois produits phares de COSE: GlobalScanner, Strike et le POD Xplorer.

1. 3 R

 3 Supervision de stages J'ai supervisé 4 stages au cours de ces trois ans de thèse, 3 au sein de l'entreprise et un au L2TI : -Conception et mise en oeuvre d'algorithmes de suivi d'objets dans des images aériennes (Mars-Août 2021 -COSE). -Optimisation et intégration d'algorithmes de détection d'objets dans un système embarqué (Mars-Août 2022 -COSE). -Apprentissage auto-supervisée pour la détection d'objets few-shot (Avril-Août 2022 -L2TI au travers du LabCom IRISER). -Détection d'objets few-shot par visual transformers sur des images aériennes (Mars-Août 2023 -COSE et L2TI via le LabCom IRISER).

1

 1 Problem De nitionObject detection consists in localizing and classifying all objects of interest visible in an image I.There are multiple terms to explain this statement. First, the notion of the object of interest is de ned according to a xed set of semantic classes C. The objects of interest are the ones that belong to one class c ∈ C. Of course, one can question the belonging of an object to a class. A

Figure 2 . 1 :

 21 Figure 2.1: Three di erent box representations: top-left and bottom-right points, top-left point, width and height, and center point width and height.Many more representations exist but they will not be presented here. In this manuscript, the second representation, top-left point width and height, will be used exclusively.

  Classi cation confusion matrix and de nition of the Precision, Recall and Accuracy metrics. (b) IoU for rectangular bounding boxes.

Figure 2 . 2 :

 22 Figure 2.2: De nition of the Precision, Recall and Accuracy metrics (a) as well as the box similarity criterion Intersection over Union (b).

  scores. It is usually possible to derive a con dence score along with the bounding box coordinates and labels from a detection model. This can be for instance the highest class probability score. Once the detections are ranked according to con dence scores, one can compute the running precision Prec k and recall r k by taking only the top-k bounding boxes. Then, it is possible to plot the precision as a function of the recall by plotting the points (r k , Prec k ), for 1 ≤ k ≤ | Ŷ|. This generally gives D zag shaped curve Prec(r) as visible in Fig. 2.

Figure 2 . 3 :

 23 Figure 2.3: Example of a Precision-Recall curve and its interpolated variant.

Figure 2 . 4 :

 24 Figure 2.4: Timeline of the Object Detection literature, it includes some of the most relevant works in the eld of Object Detection. Papers marked with a ag are the most ground-breaking works, some will be detailed in Sec. 2.1.3

  of annotated images available to train the model. In the literature, the expression N -way K-shot classi cation designates the task of classifying images amongst N di erent classes only provided with K annotated examples per class. The N K images constitute the novel dataset, in contrast to the base dataset which contains an arbitrary number of annotations for another set of classes. In the few-shot literature, the novel dataset is often called the support set, and its elements support examples. Similarly, the sets of classes of the base and novel datasets are called the base classes set D Examples of Pascal VOC images and class repartition on the training split. (b) Examples of MS COCO images and class repartition on the training split.

Figure 2 . 6 :

 26 Figure 2.6: Image examples for the Natural images dataset Pascal VOC and MS COCO.

  Examples of DOTA images and class repartition on the training split. (b) Examples of DIOR images and class repartition on the training split.

Figure 2 . 7 :

 27 Figure 2.7: Image examples for the Aerial images dataset DOTA and DIOR.

  ProtoNets replace the nal prediction layer of the siamese networks with a linear classi er and extend the metric learning framework for multi-class and increased number of shots. Speci cally, the features of all support examples of one class are aggregated to form class prototypes and query features are classi ed according to the class of the closest prototype. Many variants of the ProtoNets were then proposed. Inspired by Siamese Networks, Relation Networks[START_REF] Flood | Learning to compare: Relation network for few-shot learning[END_REF] replace the linear classi er of the Prototypical Network with a small MLP trained to predict a similarity score based on the query features and a prototype. The di erence with Siamese Networks is that this is done with each class prototype allowing Relation Nets to address multi-class problems. Other extensions include prototypes recti cation for intra and extra-class variance[START_REF] Liu | Prototype recti cation for few-shot learning[END_REF], semi-supervised prototypes re nement[START_REF] Ren | Meta-learning for semi-supervised few-shot classi cation[END_REF] and multiple prototypes per class[START_REF] Deuschel | Multi-prototype few-shot learning in histopathology[END_REF]. It is essential to note that Prototypical Networks and their extensions leverage episodic training strategies borrowed to meta-learning. This strategy consists in dividing the training into shorter episodes. During each episode, the model is trained for a random task, generally a subset of the novel classes (only the D novel dataset is considered by these approaches). The episodic strategy follows the "learning to learn" paradigm and mimics the adaptation process the model will undergo at test time.

Few-shot

  Open-set Recognition Open-set classi cation assumes that some classes are unknown during training (i.e., the training dataset is incomplete) and deals with these classes. Instances of unknown classes can be rejected or identi ed as unknown classes. It models real use cases better as test data can be contaminated by classes not included in the training set. Object detection can be assimilated as an open-set problem as objects belonging to a xed set of classes must be localized while rejecting everything else as background. The training set can only contain a limited variety of background examples and new instances of background will be presented to the model at test time. There are plenty of ap-

  discussed in the previous section the discrepancies in terms of classes: classes encountered at test time may di er from annotated training classes. However, training and test images can also have di erent aspects. For instance, autonomous vehicle perception systems could be trained only with daylight images and encounter nighttime images once deployed. The train and test image spaces are denoted as source domain and target domain in the Cross-Domain Adaptation (CDA) literature. Speci cally, a domain consists in an image space I and a marginal probability distribution p(I) over it: M = {I, p(I)} , I ∈ I.(2.22)CDA aims at adapting a model trained for a speci c task on a source domain M source = {I source , p source (I)} to perform the same (or another) task on the target domain M target = I target , p target (I) . For simplicity, we restrain the scope of this section to the classi cation task.

  There exists a slight di erence between Domain Adaptation (DA) and what is sometimes called Fewshot Domain Adaptation (FSDA) in the literature. This di erence lies in the amount of available data in the target domain. FSDA methods have access to fewer target examples than regular DA.

A

  existing works redraws decades of progress in this eld and helps understand how state-of-the-art detection has been achieved. Similarly, for Few-Shot Learning, this chapter gives the key de nitions to understand the stakes of the few-shot problem. An overview of the few-shot literature also provides relevant insights about how to adapt perception models in low-data regimes. This prospecting work greatly helps in understanding what is relevant from a research perspective and what directions to follow according to the industrial needs of COSE.This chapter presents the task of detection in the few-shot regime and reviews the existing literature about it. Few-Shot Object Detection (FSOD) is at the crossroads of Object Detection and Few-Shot Learning, and therefore, extensively relies on these two elds explored in Chap. 2. Just as for classi cation, various directions are explored in the literature to tackle the detection task in the few-shot regime which will be presented in detail. Finally, this chapter focuses on the aerial image application of FSOD methods and extensions of the few-shot setting.

C 3 . 1

 31 Problem de nition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Review of the Few-Shot Object Detection Literature . . . . . . . . . . . . . . . 3.2.1 Fine-tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Metric-learning based methods. . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Attention-based methods . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Meta-Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Few-Shot Object Detection on Remote Sensing Images. . . . . . . . . . . . . . 3.4 Extension of the Few-Shot Object Detection Setting . . . . . . . . . . . . . . . 3.5 Dataset preparation and evaluation in the Few-Shot setting . . . . . . . . . . . 3.5.1 Adapting detection datasets in the Few-Shot setting . . . . . . . . . . . . 3.5.2 Evaluation protocol for Few-Shot Object Detection . . . . . . . . . . . . . 3.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The company COSE is developing CAMELEON, an intelligent airborne surveillance system to automatically detect objects of interest. The detection algorithm must be adaptative as the objects can change from one operation to another. Therefore, the most relevant direction to explore is the

Figure 3 . 1 :

 31 Figure 3.1: Timeline of the FSOD literature, several works are included as milestones for each of the four kinds of approaches to FSOD: Fine-tuning, Metric Learning, Meta-Learning and Attentionbased approaches. The yellow hatched rectangle represents the duration of this PhD project.

Fig. 3 .

 3 Fig. 3.1 outlines the organization and the temporality of the FSOD eld. FSOD is a relatively new challenge and only started 2 years before this PhD project. Tab. 3.1 provides an almost exhaustive overview of the literature about FSOD. The reader can refer to several surveys [217, 218, 219] about

  provides a better balance between positive and negative samples and makes both base training and ne-tuning more e cient. Another line of work addresses the FSOD problem through an augmentation perspective. It circumvents the low few-annotated examples and the over tting risk by enriching the support set with more or less elaborated augmentations. An easy and e ective solution is to crop and paste novel instances directly inside base images [274]. During ne-tuning, images from the base dataset and support examples are randomly sampled. The support examples are cropped and pasted into the base images. This signi cantly boosts the ne-tuning procedure and improves FSOD performance.

  the base dataset. Reference[START_REF] Wu | Multi-faceted Distillation of Base-Novel Commonality for Few-Shot Object Detection[END_REF] ne-tunes both the classi er and the regressor of Faster R-CNN with an additional distillation loss based on pseudo-labels. Pseudo-labels are computed in a metric learning fashion between the query feature and the prototype features stored in a memory bank. Few-shot object detection via Association and DIscrimination (FADI)[START_REF] Cao | Few-Shot Object Detection via Association and DIscrimination[END_REF] splits the netuning step into association and discrimination steps. During the association step, the network is ne-tuned to map novel classes onto base classes. It leverages the well-structured base class representation space learned during base training and separates novel classes. Then, the discrimination step disentangles base and novel class representations with a dedicated margin loss.

Figure 3 . 2 :

 32 Figure 3.2: Attention-based FSOD principle

Figure 3 . 3 :

 33 Figure 3.3: Spatial alignment between query and support feature maps. Similarity matrix is computed as an outer product between the feature maps. For sake of clarity, maps are reshaped as 2-D matrix where the rst dimension controls the spatial position in the map: n q positions for the query and n s for the support. d is the number of channels. Similar colors mean that features are similar.

  Faster R-CNN with the MAML framework. Speci cally, they choose to generate weights only for the detection head of Faster R-CNN, considering that the feature extractor and RPN are class-agnostic and do not need adaptation. This signi cantly reduces the size of the generated weights and makes it possible to use MAML. During base training, only the detector is trained on the base dataset.

  First, the set of classes is divided into two sets: the base and novel class sets (with |C novel | = N ). The class split for each dataset is xed by common practices (for Pascal VOC and COCO) or taken at random (for DOTA and DIOR) when no convention is set in the literature. Tab. 3.2 gives the class split that will be used throughout this PhD thesis. Then, the instances of the novel classes are ltered from the dataset to keep only K images per novel class. This ltering operation is performed in two steps: S O D 1. for each novel class c ∈ C novel , K images containing at least one instance of class c are selected as the support examples. They constitute the support set.2. the instances of the novel classes are removed from all other images in the dataset.

4 . 1

 41 Distractors in the Few-Shot Data Regime First, changing the set of classes of interest during the training procedure (i.e., between base training and ne-tuning) is problematic. Classes considered as background can become objects of interest, which goes against the knowledge acquired during the base training phase. This is embodied by the concept of distractors, introduced in [263]. It refers to examples that provide wrong supervision to a model during training. We choose to re ne this concept in two categories: self-distractors and co-occurrence distractors. Self-distraction occurs when annotated and non-annotated instances of a class are visible in the same image. The non-annotated instances are called self-distractors. This can happen when there are annotation mistakes in a dataset, but it can also happen in the fewshot settings. For instance, if annotations of support examples are ltered (e.g., to keep only one annotation per support image), all the ltered instances will become self-distractors during netuning. When ne-tuning on the support examples, the annotated instances of the novel classes willprovide correct supervision to the model. However, the non-annotated instances will be considered as background and wrong supervision will be propagated in the model. This explains why it is more sensible to keep all annotations of the novel classes in the support set even though it does not fully comply with the original N -ways K-shots setting. In the literature, this choice is barely discussed and early works in the eld employ either the strict one annotation per image sampling or the self-distractor-free sampling described above. Using self-distractor-free sampling often results in improved performance; however, no analysis was conducted to explain the origins of these gains (either coming from more examples or thanks to more coherent supervision). In our experiments, both setups were used as this issue was encountered in the middle of this project. We will clearly specify what setting is used for all our experiments. The second type of distraction, co-occurrence distraction, happens when novel class instances are visible in images during base training. Their annotations have been ltered out, therefore they are considered as background. Of course, this makes sense during base training as the novel classes

Figure 4 . 1 :

 41 Figure 4.1: Co-occurrences between the classes of the four datasets of interest DOTA, DIOR, Pascal VOC and MS COCO. Novel classes are highlighted in red. For MS COCO only novel class labels are shown for clarity.

  regular baseline is an FCOS detector trained on the full datasets. To conduct this experiment, we only select DOTA, DIOR and Pascal VOC as they have roughly the same number of classes. COCO however has 4 times more classes which brings additional complexities. The performance results of the regular baseline are available in Tab. 4.1. Speci cally, FCOS is trained on each dataset with full access to the annotations for both base and novel classes. Then the mAP is computed on each class individually and averaged on base and novel classes separately. This gives an overview of the performance of the model respectively on base and novel classes in a regular data regime.It seems tempting here to extrapolate the FSOD performance on DOTA and DIOR from the performance on Pascal VOC. The regular baseline (FCOS) achieves similar performance on these two datasets, which contain the same number of classes and roughly the same number of images. Thus, one could have expected close FSOD performance on these datasets. This is quite di erent from the actual results reported in Tab. 4.2. The FSOD performance on DOTA and DIOR is signi cantly lower compared with the results on Pascal VOC. To better visualize this nding, Fig.4.2 represents the few-shot performance as dark bars while regular baseline performance as lighter rectangles.

Table 4 . 2 :

 42 Comparison of mAP 0.5 of several methods on DOTA, DIOR and Pascal VOC datasets. For each method, mAP is reported for di erent numbers of shots K ∈ {1, 3, 5, 10} and separately for base and novel classes. Blue and red values represent the best performance on base and novel classes respectively, for each dataset.

Figure 4 . 2 :

 42 Figure 4.2: Performance comparison between FCOS trained in a regular data regime versus three few-shot baselines, FRW, WSAAN and DANA (all based on FCOS as well) on three datasets: DOTA, DIOR and Pascal VOC.

  well a FSOD method is performing on di erent datasets compared with the regular detection performance. Hence, it represents how much performance is lost when switching from the regular to the few-shot regime. This is exactly what is illustrated in Fig.4.2, white percentages are RmAP values. RmAP is signi cantly lower on DOTA and DIOR compared to Pascal VOC, both for base and novel classes. This way, we can quantitatively con rm the intuition emerging from Tab. 4.2: FSOD works better on natural images.

  2.1.3.4. However, this problem is largely ampli ed for FSOD. It is still di cult to detect small objects, but in addition, they are poor examples for adapting the model (either through ne-tuning or direct adaptation).

Figure 4 . 3 :

 43 Figure 4.3: Box plot of objects size in DOTA, DIOR and Pascal VOC and MS COCO. On the left side, boxes represent the overall size distribution in each dataset. On the right side, the distributions are split by class and ordered by average size. As MS COCO contains 80 classes, we choose not to include the per class box plots for it in this plot.

Figure 4 . 4 : 3 CFigure 4 . 5 :A

 44345 Figure 4.4: Performance comparison between FRW baseline -with 10 shots -(blue and red dots) and regular baseline (black stars) on three di erent datasets: DOTA, DIOR and Pascal VOC. (top) Mean average performance of the two methods plotted per class against average object size. (bottom) gap between FRW baseline and regular baseline, per class. Positive values indicate better performance than the regular baseline.

C 5 . 1

 51 Motivation and Principle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 5.2 Prototypical Faster R-CNN for FSOD . . . . . . . . . . . . . . . . . . . . . . . 90 5.2.1 Extending Faster R-CNN with Prototypical Networks. . . . . . . . . . . . 91 5.2.2 Training Procedure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.2.3 Iterative improvements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 5.3 Performance on Arti cial Data . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.3.1 MNIST-LOC Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.3.2 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.3.3 Detection performance on MNIST-LOC . . . . . . . . . . . . . . . . . . . 97 5.4 Di culties on Aerial Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.5 Insights and conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 F M L FSOD

  3.1); however, the simplicity and success of the metric learning classi cation models was tempting. Thus, we proposed Prototypical Faster R-CNN (PFRCNN), an extension of Faster R-CNN based on metric learning. The key idea is to replace the classi cation layers from Faster R-CNN (i.e., in the Region Proposal Network (RPN) and in the Classi cation head) with prototypical networks. It is similar to RepMet [251] that leverages class-representative vectors in the classi cation head. However, there are two major di erences with PFRCNN. First, RepMet only replaces the classi cation layer in the second stage of Faster R-CNN, not in the RPN. Hence, the adaptation to novel classes is only done in the second stage. Even if the RPN is presented as a class-agnostic detector, it specializes in the classes seen during training. As only base classes are annotated during the rst phase of training, objects from novel classes will be ltered out by the RPN, leaving no chance for the second stage to detect them. Even if it is trained to have a high recall, the RPN will mostly generate proposals on base classes, which is harmful in a few-shot regime. Second, RepMet learns the class-generative vectors from ne-tuning on the few available examples of the novel classes. Instead, a prototypical network computes its prototypes directly from the few available examples. Finally, Prototypical

. 6 )Figure 5 . 1 :

 651 Figure 5.1: Illustration of the architecture of Prototypical Faster R-CNN.

. 10 )

 10 This gives RPN features and classi cation features for each support image, denoted Φ c k and Ψ c k respectively. Note a slight abuse of notation here, when only the embedding part of g and h is used to project the features extracted by the backbone (i.e., not the regression part). When multiple examples are available for a class (i.e., K ≥ 1), their embeddings are averaged to get one prototype per class:

  i and o i are the ground truth targets for the regression and classi cation branches of the RPN. Similarly, b head i and c i are the target for the detection head. During training, not all boxes are selected for computing the losses. The generated boxes (or proposals for the RPN) are separated into two groups: positive examples, i.e. boxes with an overlap of at least 0.7 with a ground truth annotation, and negative examples which represent the background class. The classi cation losses are computed over all examples, while the regression losses only take into account the positive boxes. This remains unchanged for Prototypical Faster R-CNN. However, the training of PFRCNN is done episodically, following the Meta-Learning paradigm and the training scheme proposed in Prototypical Networks [126]. The motivation behind such training is to mimic the setup that will be encountered at test time and prevent base classes memorization. Indeed, during base training, only annotations from the base classes are available. Training the model with all base classes at the same time could lead to over t the base class set, at the cost of adaptation. The episodic training consists in sampling a subset of classes C ep ⊂ C base and train the model to detect only these classes for a few training steps. Such a training phase is called an episode. The episodes are then repeated over and over until convergence. During each episode, a query set and a support set are sampled from the original dataset. The support set contains the examples that will be leveraged for the prototypes computation. On the other hand, the query set is exploited as a small training set. The loss is computed on the query set and between each update of the model, the prototypes are re-computed from the same support set. The update of the prototype is not necessary between each training step, but since the model's weights are updated, the class representations also change. Additionally, the episodic strategy allows for mimicking the test time setting. If there are N novel classes with K support images at test time, the episodes can reproduce this even though the dataset has a lot more classes and data. Episode after episode, the model will encounter new class combinations and support examples, in the end, it should learn to generalize to novel classes from a few examples, according to Meta-Learning claims.To build the support set, for each class c ∈ C ep , we select images containing objects of class c and disregard all other objects (i.e. their annotations are not included in the support set but the image is not masked, so they are still visible). If there is more than one object c in the image, only one is selected randomly as the annotated example. This prevents having more than K examples per class.The query set contains K query images for each of its N classes, this means at least K query examples for each class, but this number can be larger as more than one object is present in the images. As for the support set, the annotations with class labels not in C ep are discarded. This sampling procedure prevents the occurrence of self-distractors but not co-occurrence distractors (see Sec. 4.1).

  novel class sets: C ep ⊂ (C base ∪ C novel ). In this case, the examples of novel classes are the same in the query and support sets so that the total number of support examples remains xed.

  One issue encountered with the baseline is the detection of the base classes regardless of support examples. Basically, it detects base classes even though no prototypes are provided for these classes: this is base class memorization. Although this improves performance when base class prototypes are provided, it produces lots of false positive detections when novel classes are wanted. To address this, we propose to sample hard negative examples to encourage the model to detect support classes only. The main idea is to take advantage of the annotations for classes not selected in the current task to nd hard negative examples, i.e. classes that the network could have memorized from previous tasks but should not be detected during this episode. When starting a new episode, it is likely that the model still produces detection for objects annotated in one of the previous episodes if it does not rely on the support information. Even though these objects are not annotated in the new task, their annotations are available in the dataset (because they belong to base classes). Therefore, these annotations can be used to nd examples that should be considered as background for the current task. They are di erent from the background examples that do not contain any class of the dataset, which are referred to as easy negative examples and are much more numerous. Explicitly sampling these hard negative examples encourages the network to detect only objects annotated in the support set.

Figure 5 . 2 :

 52 Figure 5.2: Images of MNIST-LOC dataset and classes repartition.

5 . 3

 53 gives detection examples on MNIST-LOC dataset for base and novel classes. For base classes, two distinct support sets are leveraged between rows 1 and 2 (with di erent classes, i.e., C 1 ep = C 2 ep ). For base classes, the detection is almost perfect, which represents well the scores from Tab. 5.2. However, for novel classes, there are undesired detections of base classes and a lot of confusion between novel classes.

First, Tab. 5 .Figure 5 . 3 :

 553 Figure 5.3: Prototypical Faster R-CNN qualitative detection results on MNIST-LOC dataset, on base and novel classes. Predictions are done without ne-tuning and with K = 1.

Table 5 . 4 :

 54 tuning step, especially to re ne the regression branches of the model. This was performed on DOTA with Split A and the results are available in Tab. 5.4. Fine-tuning with the few available support examples helps signi cantly to boost the detection quality on novel classes, but it remains insu cient for COSE's application. Interestingly, after ne-tuning a common property of few-shot methods emerges: the more examples are provided, the higher the performance. It was not the case without ne-tuning. With Split A, the best performance is achieved with K = 1, with Split B, it increases until K = 5 and then decreases with K = 10. This indicates that the management of more shots is di cult within PFRCNN. It suggests that support examples features may not be trivially aggregated as it can produce irrelevant prototypes. This can happen when a class has a great variety and thus a multimodal distribution in the embedding space.Just as for MNIST-LOC, we provide qualitative results of the FSOD on DOTA with PFRCNN. These are available in Fig.5.4. The detection is satisfactory (but not perfect) on the base classes. However, the bounding boxes and labels for novel classes (bottom 2 rows) are mostly incorrect. Some 5PFRCNN performance comparison with and without ne-tuning.

Figure 5 . 4 :Figure 5 . 5 :

 5455 Figure 5.4: Prototypical Faster R-CNN qualitative detection results on DOTA dataset, on base and novel classes. Predictions are done without ne-tuning and with K = 1.

5. 5 I

 5 Learning often leverage other tricks such as carefully designed ne-tuning or attention mechanisms. Furthermore, at the beginning of this PhD, there were only two contributions solving FSOD with metric learning: RepMet[START_REF] Karlinsky | Repmet: Representative-based metric learning for classi cation and few-shot object detection[END_REF] and RN-FSOD[START_REF] Yang | Restoring negative information in few-shot object detection[END_REF]. More investigation was therefore needed in this direction. Of course, the poor results of PFRCNN alone are not enough to conclude that all metric-learning-based approaches are inappropriate. Nevertheless, metric-learning FSOD methods are now in the minority in the current literature, which indirectly con rms their inadequacy.Despite the relatively poor performance of PFRCNN, our experiments provide useful insights for future designs of FSOD methods. First ne-tuning is crucial for FSOD. It yields signi cant performance gains compared to models only trained on base classes. This makes sense as the adaptation of the model with the prototypes is only performed in the classi cation branches. The regression branches are therefore unprepared for the localization of novel classes. Of course, having a method that does not require any ne-tuning is highly desirable from an industrial perspective, but that should not come at the cost of poor performance. Then, Faster R-CNN may not be the best detector choice for few-shot extensions. Indeed, its two-stage structure duplicates the number of modications required for the adaptation to novel classes. Even if some works argue that the RPN is class-agnostic, it is still trained to only detect base classes while discarding everything else, including potential novel classes. The RPN must then be adapted to novel classes as well. It makes the few-shot extension more cumbersome, with more parameters and more causes for failure. One-stage detectors certainly are a more sensible choice. Finally, the episodic training strategy may also be inadequate for detection. It complexi es greatly the training and introduces distractors (this concept is explained in Sec. 4.1). At the beginning of each episode, a subset of classes (either base or novel depending on the training phase) is sampled. Annotations from all other classes are discarded during the episode, yet the training images still contain instances of other classes. These distractors are confusing for the model. Of course, for classi cation, the episodic strategy forces the model to establish connections between support examples and the query images. But it is much simpler as the query images only contain one object belonging to one of the episode classes. The presence of already-seen classes (and potential future classes) inside query images certainly makes the episodic training strategy suboptimal for the detection task. The latter paragraph formulates a few assumptions based on our observations of the design and training of PFRCNN. Of course, it would be wise to conduct dedicated experiments to con rm these hypotheses. For instance, carefully designed synthetic images could help to experiment with the distractors' in uence in a controlled way.

Figure 6 . 1 :

 61 Figure 6.1: Attention-based Few-Shot Object Detection principle. Query and support images are processed by the backbone before being fed to the query-support combination block. Detection is then performed independently on each class.

3 . 2 . 3 ,

 323 three main components of attention mechanisms for FSOD have been identi ed: Spatial Alignment, Global Attention and Direct Feature Fusion. Most attention-based FSOD methods rely on one or more of these components. Thi section will cover the Alignment, Attention and Fusion (AAF) framework, whose purpose is to provide a exible environment to implement attention-based FSOD methods. Before jumping into the de nition of the AAF framework, let's recall brie y the main principle of attention-based FSOD, illustrated by Fig. 6.1. The goal of the attention module is to combine the information from the query image and the support examples. Speci cally, the query features are compared with class-speci c features computed from the support set. This comparison highlights similar parts in the query image and the support examples, yielding class-speci c query features. The detection is then performed separately for each class.

  3.1). The model is trained in an episodic manner. During each episode, a subset C ep ⊂ C of the classes is randomly sampled. A query set is sampled for each episode, containing 100 images per class. This set only contains annotations of the episode classes and is leveraged for the loss computation and the optimization of the model. A support set is also sampled at the beginning of each episode containing K examples for each episode class. The support examples are used through the attention mechanisms to condition the detection on the episode classes. The training is divided into two phases base training and ne-tuning. During base training, only base classes can be sampled (C ep ⊂ C base ) and one image per class is drawn for the support set (K = 1). The optimization is done with SGD and a learning rate of 1 × 10 -3 for 1000 episodes. During ne-tuning, the backbone is frozen, the learning rate is divided by 10, and the episode classes can be sampled from C base ∪ C novel , with at least one novel class per episode. Examples from novel classes are selected among the K examples sampled once before ne-tuning. Each model is netuned separately for di erent values of K ∈ 1, 3, 5, 10. During both training phases, the same loss function is optimized, as de ned in FCOS (see Tab. 2.1).

  support and query feature extraction. However, some of our techniques are quite close to what they proposed. These techniques are described below and Fig. 6.4 illustrates most of them: -Default: the most naive extraction technique. It consists in cropping the support image around the support object at a xed size (e.g. 128 × 128). Objects larger than this are simply resized to t in the patch.

3 . 1 )

 31 . They have been chosen because they represent well the variety of attention mechanisms available in the literature and according to their popularity. FRW is based on class-speci c reweighting vectors, WSAAN has a more sophisticated global attention and computes reweighting vectors inside a graph structure. DANA and Meta R-CNN leverage query-support alignment in di erent manners and DRL only uses a sophisticated fusion layer. Each of these methods has been reimplemented within the AAF framework. Of

Fig. 6 .

 6 Fig. 6.5 displays these trends clearly, both for base and novel classes. In addition, Fig. 6.6 provides the same results split by class. An interesting observation from this last gure is the very good detection performance for the novel class sheep. This can be explained easily from the presence of the class horse in the base set. The model has seen a lot of examples of horses during base training, which makes it learn visual attributes common with a sheep (e.g., four legs, hair and grassy background).

Figure 6 . 5 :Figure 6 . 6 :

 6566 Figure 6.5: Evolution of mAP 0.5 with the number of shots averaged on base and novel classes separately. Each line represents one of the reimplemented methods.

Figure 6 . 8 :Figure 6 . 9 :

 6869 Figure 6.8: XQSA small and large objects matching asymmetry.

Figure 6 . 10 :

 610 Figure 6.10: Support examples for base classes (K = 10). These are the examples used during the detection inference that produced results from Fig. 6.13.

Figure 6 . 11 :

 611 Figure 6.11: Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with K = 10 shots on base classes.

Figure 6 . 12 :

 612 Figure 6.12: Support examples for novel classes (K = 10). These are the examples used during the detection inference that produced results from Fig. 6.13.

Figure 6 . 13 :A

 613 Figure 6.13: Qualitative assessment of the detection quality of FRW, WSAAN, DANA and XQSA on DOTA with K = 10 shots on novel classes.

C 7 . 1

 71 Di usion Probabilistic Models Principle. . . . . . . . . . . . . . . . . . . . . . 7.1.1 Forward and Reverse Di usion Processes . . . . . . . . . . . . . . . . . . 7.1.2 DDPM Training and Sampling . . . . . . . . . . . . . . . . . . . . . . . . 7.1.3 Recent Advances with Di usion Models . . . . . . . . . . . . . . . . . . . 7.2 Di usionDet for Object Detection . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Few-Shot Di usionDet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3.1 Fine-tuning Strategy for Few-Shot Di usionDet. . . . . . . . . . . . . . . 7.3.2 Experimental Study of FSDi usionDet . . . . . . . . . . . . . . . . . . . . 7.3.3 Comparison with existing FSOD Methods . . . . . . . . . . . . . . . . . .

7. 4

 4 Application to Cross-Domain FSOD . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 MS COCO → Anything. . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Aerial Cross-Domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.3 Cross-Domain Perspectives. . . . . . . . . . . . . . . . . . . . . . . . . . 7.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 we have proposed respectively metric-learning and attention-based approachesto tackle the FSOD problem. Both of these directions were sensible choices given the state of the FSOD literature at the beginning of this project. Since then, however, ne-tuning approaches have gained a lot of interest with competitive performance and reduced complexity. Following this trend, we explore in this chapter a simple ne-tuning strategy for FSOD. Based on the recent Di usionDet[START_REF] Chen | Di usiondet: Di usion model for object detection[END_REF] model, we propose an e ective ne-tuning scheme for FSOD which outperforms all previous methods on DOTA and DIOR datasets while being competitive with state-of-the-art on natural images. We begin with a brief presentation of the Di usion Probabilistic Models (DPM) and their recent progress in various generative tasks. Then we present in detail Di usionDet, which tackles OD with a refreshing perspective, as a box-denoising problem. Following this, we present our ne-tuning strategy called Few-Shot Di usionDet (FSDi usionDet) and the results of multiple experiments conducted to improve our strategy. Given the impressive performance of FSDi usion-Det in the few-shot regime, we broaden the scope of our analysis and study the more challenging

. 13 )Figure 7 . 1 :

 1371 Figure 7.1: Di usion Processes illustration in the context of image denoising. Image taken from [200, 302].

  negative log-likelihood and leverage the evidence upper bound:

Algorithm 2

 2 Di usion Sampling Procedure

Figure 7 . 3 :

 73 Figure 7.3: Illustration of Di usionDet principle, gure taken from [74]. (a) di usion process, (b) di usion process for image generation with DDPM and (c) Di usionDet principle, random boxes are denoised to locate objects in the image.

Figure 7 . 4 :

 74 Figure 7.4: Di usionDet architecture and detailed detection head design.

Its fast training time allows for 145 Figure 7 . 5 :

 14575 Figure 7.5: Performance of FSDi usionDet, XQSA, FRW, DANA and WSAAN on DOTA, DIOR, Pascal VOC and MS COCO against the number of shots. Performance is reported with mAP 0.5 .

  model, trained in a contrastive way on a 400 million image-text pairs dataset. Then, FSDi usionDet is trained following our two steps training scheme (base training and netuning). The results are available in Tab. 7.3. From this, it is quite clear that training from scratch is not a sensible option, even though base training is correct, the ne-tuning on base classes provides really poor detection performance. Then, between Imagenet, DINO and CLIP the di erences are tight. Of course, CLIP's weights are slightly worse than ImageNet and DINO, but it is still a strong baseline. Between ImageNet and DINO, however, it is di cult to conclude as both achieve the best performance on one aerial and one natural dataset. As the performance gap is thin between ImageNet and DINO, we choose to conduct our next experiments with ImageNet weights which have stood the test of time and are now the default choice in computer vision.

Fig. 7

 7 .7 shows how the proposals would change with , with ς = 2 xed.

Figure 7 . 6 :

 76 Figure 7.6: Di usionDet initial random boxes with various values of ς, the parameter that controls the spread over the images. 75 proposals are sampled per image.

Figure 7 . 7 :

 77 Figure 7.7: In uence of on the size of the proposals. Note that here 200 proposals are sampled, for visualization purposes.

Figure 7 . 8 :Figure 7 . 9 :

 7879 Figure 7.8: Comparisons of the TSNE visualizations of the ne-tuned model predictions (FI), transductive inference (TI), hybrid inference (HTI) and the oracle. Note that background predictions are only available for the oracle as the models' inferences only provide class scores.

Figure 7 . 10 :

 710 Figure 7.10: Qualitative detection results of FSDi usionDet on DOTA, DIOR, Pascal VOC and MS COCO datasets. The models employed to produce this gure have been ne-tuned with K = 10 shots and following the best ne-tuning strategy found in Sec. 7.3.2.4 for each dataset.

Figure 7 . 11 :

 711 Figure 7.11: Presentation of the datasets used in our Cross-Domain benchmark: VisDrone2019, DeepFruits, SixRay, and ClipArt.

2

  Link to the Meta-Dataset and Python API package

  pend a lot on the choice of the support set. Of course, most of our experiments should be repeated the same way to strengthen the results, but this quickly becomes expensive in terms of computing resources. These preliminary results are promising, FSDi usionDet achieves satisfactory performance with only a few-annotated examples on various datasets. These datasets are constituted of various kinds of images, therefore it demonstrates well the adaptation capabilities of FSDi usionDet. Now for COSE's application, these results on aerial datasets are particularly encouraging. FSDi usionDet achieves impressive performance with only a base training on MS COCO and few examples of either DOTA or DIOR. Thus, this model could be rapidly ne-tuned for a speci c mission by the forces without declassifying any image. Of course, this is only a baseline and FSDi usionDet can surely be improved further. In addition, this scenario starts with a base training on natural images, which

  plication in computer vision. In this case, A and B are sets of pixels, and the IoU measures how close they are. When A and B are rectangular boxes, IoU can be computed easily with simple operations on box coordinates (see Eq. (2.7)). This explains why IoU is such a widespread criterion for object detection. It is used as a loss function (L reg = 1 -IoU) by several well established detection 8

Figure 8 . 1 :

 81 Figure 8.1: (Left) Evolution of IoU, NWD [88], the proposed SIoU and α-IoU [94] when a box is shifted from the ground truth box by ε loc pixels, for various box width ω ∈ {4, 16, 64, 128} (boxes are squares). (Right) Ratio between pixel localization error ε loc and object size ω for a trained detection model on DOTA dataset. Each point represents the localization error of one object in DOTA test set.

Fig. 8 . 1 8

 81 Fig. 8.1, right), leading to lower IoU values for smaller objects. Hence, small objects are much more likely to fall under the IoU thresholds which decide if a box is a true or false detection, even though being satisfactory from a human perspective (see the user study in Sec. 8.4). In addition, Secs. 8.3.1

Fig. 8 .

 8 1 (right) clearly shows that the ratio between the localization error (ε loc = b 1 -b 2 1 ) and the object size (ω = √ w 2 h 2 ) increases as the object becomes smaller. This gure is made with a A I O U detection model trained on DOTA with all annotations. Each point represents the ratio ε loc ω for one object in the test set. If the detection model was indeed scale-invariant, the ratio should not change signi cantly with the object sizes. Hence, because of the scale-invariance property, IoU scores are lower for small objects. It then has several consequences: 1. Bounding boxes output by the model are not considered positive examples during evaluation. 2. Bounding boxes are not selected as positive examples for loss computation, which biases the training towards larger objects.

Fig. 8 . 5 ( 2 .

 852 left) in Sec.8.3.4 shows the evolution of p with object size for various γ and κ. For convenience, we will denote the average object size i.e., , the average size of boxes b 1 and b 2 , by τ = w 1 h 1 +w 2 h 2

Property 2 (

 2 Loss and gradients reweighting) Let L IoU (b 1 , b 2 ) = 1-IoU(b 1 , b 2 ) and L SIoU (b 1 , b 2 ) = 1-SIoU(b 1 , b 2 ) be the loss functions associated respectively with IoU and SIoU. Let us denote the ratio between SIoU and IoU losses by W L (b 1 , b 2 ) = L SIoU (b 1 ,b 2 )

  i.e., how much SIoU is lost for a 1-pixel shift. As GIoU and GSIoU values range in [-1, 1], they were not included in Fig.8.1, but for completeness, they are plotted in Fig.8.2 along with other criteria.

Figure 8 . 2 :

 82 Figure 8.2: Evolution of various criteria (IoU, GIoU, and GSIoU) when a box is shifted from the ground truth box by ρ pixels for various box sizes ω ∈ {4, 16, 64, 128}. With boxes of the same size (left) and di erent sizes (right).

8. 3 . 2 . 3

 323 In uence Analysis on TrainingAll criteria discussed above are employed as regression losses in the literature. The loss associatedwith each criterion C is L C (b 1 , b 2 ) = 1 -C(b 1 , b 2 ). Therefore, the expected value of the criterion determines the expected value of the loss and thus the magnitude of the gradients. Large values of the criterion give low values for the loss. Now, as the expected values of the criteria change with the object size, the expected values of the losses also change. Small objects generate greater loss values than larger ones on average. However, this is balanced by the fact that fewer small objects are selected as positive examples because the IoU is involved in the selection process. To achieve better detection, training must focus more on small objects. One way to ensure this is to set larger loss values for small objects. Thus, the equilibrium is shifted toward smaller objects and gradients will point to regions where the loss of small objects is lower. As shown in Fig.8.6 (Sec. 8.3.4), with the right parameters, SIoU can do that. It attributes lower values for small objects while keeping similar values for large ones. The contrast between small and large objects is accentuated and optimization naturally focuses on smaller objects. SIoU's parameters control which object size gets more emphasis. This is closely linked to Property 2 which states that employing SIoU (compared to IoU) reweights the loss and the gradient by p. If γ < 0, p decreases with the size of the objects and A I O U

  Fig. 8.4b).

  IoU and GIoU expected values and standard deviation with horizontally and diagonally shifted boxes. (b) IoU and GIoU expected values and standard deviation with and without boxes of the same size. (c) IoU, SIoU, GIoU, and GSIoU expected values and standard deviation with the detector's inaccuracy modeled as an a ne function, σ(ω) = σ 0 + λω (σ 0 = 16, λ = 1 4 ).

Figure 8 . 4 :

 84 Figure 8.4: Relaxing the constraints for criteria' distribution analysis.

2a 2 du 8 . 1 : 3 Figure 8 . 5 :

 281385 Figure 8.5: In uence of γ and κ on the expected value and standard deviation of SIoU.

Fig. 8 .

 8 Fig.8.5b shows the same curves for κ ∈ {8, 16, 32, 64, 256}. κ controls how fast p approaches 1 and therefore, changing κ simply shifts the curves of expectation and variance accordingly. As κ increases, IoU's behavior is retrieved for larger objects reducing the expected value of SIoU. The variance is not changed much by κ, but it slightly shifts the maximum of the curve, i.e., the object size for which SIoU's variance is maximum.

Fig. 8 .

 8 Fig. 8.6 also provides pdfs plots for various object sizes for SIoU and GSIoU, in addition to expectation and variance comparison between existing criteria. For this gure, γ = -3 and κ = 16. This gure echoes Fig. 8.3 which plots the same curves but with γ = 0.5 and κ = 64.

Figure 8 . 6 :

 86 Figure 8.6: Analysis of the distribution of IoU, SIoU, GIoU, GSIoU and α-IoU when computed on inaccurately positioned boxes. This is done by observing the probability distribution functions (pdfs) for various ω values (left), the expectation (middle) and standard deviation (right) for all criteria.For SIoU and GSIoU, we xed γ = -4 and κ = 16, for α-IoU, α = 3 (as recommended in the original paper[START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF]). The inaccuracy of the detector is set to σ = 16. Note that the empirical pdfs were smoothed using a Kernel Density Estimator method. This a ects particularly IoU, SIoU and α-IoU as the actual PDF is de ned only on [0, 1]. For the sake of visualization, GIoU and GSIoU were rescaled between 0 and 1 for the expectation and standard deviation plots.

Figure 8 . 7 :

 87 Figure 8.7: Rating page of the Web App with an example of an image and two rectangular bounding boxes. The user is asked to rate the quality of the red box compared to the green one on a 5-levels scale going from very poor to very good.

Figure 8 . 8 :

 88 Figure 8.8: Examples given to the participants of the user study. The IoU between the green and red boxes are 0.1, 0.25, 0.5, 0.75, and 0.9 for the ratings from very poor to very good respectively.

Figure 8 . 9 :

 89 Figure 8.9: Criteria' scores for di erent object sizes and human ratings r ∈ {1, 2, 3, 4, 5} (top). Relative gap with the criterion value averaged over the object sizes (c r = 1/3(c S,r + c M,r + c L,r )) (bottom).

Figure 8 . 10 :

 810 Figure 8.10: Relative gap with the SIoU values averaged over the object sizes, for various γ values.

Figure 8 . 11 :

 811 Figure 8.11: Rating against IoU, SIoU (γ = 0.2, κ = 64), NWD and α-IoU (α = 3) values, overall and for di erent groupings of the variables of interest (object size, presence of contextual information, expertise and age of the participants). Colors represent di erent values for each variable. A legend for each row is included in the right-most column of the gure. For the Age variable, the participants have been separated into three groups of the same size.

Figure 9 . 1 :

 91 Figure 9.1: Simpli ed illustration of the on-board computer architecture in CAMELEON.

Figure 9 . 2 :

 92 Figure 9.2: Nvidia AGX Orin, development kit (left) and production (right) GPUs.

Figure 9 . 3 :

 93 Figure 9.3: Inference speed / detection accuracy tradeo s comparison for multiple detection frameworks on MS COCO. Latency, i.e., wall-clock time for inference of one image is directly used (right) but this measure is not always reported in the literature. As a surrogate, the number of parameters is used to have a more thorough overview (left). N, S, M, L and X designate the various model sizes.

Figure 9 . 4 :

 94 Figure 9.4: YOLOv5 tradeo between model size and performance (left). Between latency and performance (right). 6 model sizes are compared: Femto, Pico, Nano, Small, Medium and Large on DOTA and DIOR. The horizontal axis of the left plot is in log-scale to better distinguish between the smallest models. Latency is computed with 512× 512 images on a RTX 3090.

2 ,

 2

9 . 5 )Figure 9 . 5 :

 9595 Figure 9.5: Illustration of a computation graph and its optimization by TensorRT. Figure taken from an Nvidia technical blog post.

  . In practice, this loss of information is not an issue for inference and performance is almost unchanged.Integer Quantization and CalibrationPushing even further, TensorRT enables the conversion of the model in 8-bit integers. This again reduces the amount of memory and computation power required to store the model and perform inference. However, the model's weights and intermediary computation can only take 256 di erent values with integer precision. This is certainly not enough to represent the entire range of values found in a model. A solution for this is to map the weights and activations of the model into the-2 b-1 , 2 b-1 -1 interval,where b is the number of bits used for the quantization (generally b = 8). If the dynamic range of the weights is [α, β], this can be achieved with a linear transformation: S(x) = ax + b, with a = 2 b -1 β -α , and b = -α(2 b -1) β -α -2 b-1 . (9.2)

  )where a x and a y are the quantization scale parameters for the respective quantization of matrices X and Y . As hinted in the previous equation, di erent quantization functions are necessary to keep the exibility of the model. The number of parameters inside a model is large compared to the number of quantized values (256 for integer quantization). Quantizing all weights of the model at once would result in a much looser approximation of the actual weight values. Instead, quantization is performed at the layer-level or even lower (at the column or channel level).

Figure 9 . 6 :

 96 Figure9.6: Show the number of tiles against the overlap, for di erent tile size (left) and the ratio of pixels that need to be processed given the overlap (right). As an example, choosing an overlap of 0.7 will produce 10 times more data to process compared with the original image. The computations were done with CAMELEON image size (11600 × 8700 pixels).

Figure 9 . 7 :

 97 Figure 9.7: Illustration of the tiling implemented in our prototype, the background image is a test image, it is constituted of DOTA images pasted together to get the right dimensions. Green squares represent the tiles.

Figure 9 . 8 :

 98 Figure 9.8: Qualitative detection performance on CAMELEON-size image with YOLOv5-N INT8 engine.

Figure 9 . 9 :

 99 Figure 9.9: CAMELEON user interface.

  5, wehave proposed an original FSOD method entirely based on metric learning. It embeds prototypical networks into the well-known Faster R-CNN detection model. This naive approach achieves mitigated results but is highly adaptable, it can adapt to novel classes without ne-tuning or heavy computation. The experiments conducted with this model are instructive for the development of future FSOD methods. Then, in Chap. 6, we focused on the attention-based mechanisms for FSOD. We COSE P proposed a modular framework to implement and compare existing attention-based FSOD methods.Thorough experiments on both aerial and natural images showed the superiority of local attention mechanisms, called alignment. To reduce the performance gap between natural and aerial images, a novel alignment technique is developed within the framework to speci cally address the detection of small objects. The resulting FSOD approach outperforms existing work in the literature on aerial image datasets as it improves largely the detection of small targets. Finally, Chap. 7 studied the last FSOD approach: the ne-tuning strategy. Leveraging the recent Di usionDet detector based on di usion models, we propose a simple ne-tuning approach that outperforms signi cantly previous techniques. The simplicity of this approach and its impressive detection quality allows for tackling more di cult scenarios such as Cross-Domain FSOD. Fine-tuning based FSOD is also better suited for transductive inference than metric-learning or attention approaches as it does not change the detection model much. Both transductive inference and Cross-Domain scenarios are of particular interest to COSE as they match better the real application. Our experiments in these directions show

IoU(b 1 , b 2 ) p 1 , 2 = 3 ≤ 3 Line 3

 1212333 It would be extremely valuable to achieve real-time class and domain adaptation from a few examples. It would result in a single model that could be adapted on-the-y (literally) according to the operation's needs. However, state-of-the-art FSOD is still far from this and COSE should probably not invest in this direction in the short run, especially as such adaptative methods will probably come at the cost of lower detection quality and slower inference. Instead, COSE should focus on developing an e cient ne-tuning platform that could be used by the forces to train their own models for speci c missions. It could not be done during a mission, but afterward using the images collected and a few annotations. It would help to analyze the mission data faster and could produce new real-time models for subsequent similar missions. The key would be to conceive a unique application that can handle image annotation, model ne-tuning and deployment, all at once.Then, IoU(b 1 , b 2 ) p 1,3 +ε = IoU(b 1 , b 2 ) p 1,3 IoU(b 1 , b 2 ) ε ≤ IoU(b 1 , b 2 ) p 1,IoU(b 1 , b 3 ) p 1,holds because 0 < IoU(b 1 , b 2 ) ε ≤ 1. Line 4 is true because IoU(b 1 , b 2 ) ≤ IoU(b 1 , b 3

3 ⇔ 1 ,3 p 1 , 2

 3112 ln IoU(b 1 , b 2 ) ln IoU(b 1 , b 3 ) ≥ p In practice, the right condition is often true as p 1,2 and p 1,3 are close due to scale matching, a trick present in most detection frameworks to prevent comparison of proposals and ground truth of very di erent sizes. In addition, the ratio of log values gets large rapidly, even if the gap between IoU(b 1 , b 2 ) and IoU(b 1 , b 3 ) is small, the ratio ln IoU(b 1 ,b 2 ) ln IoU(b 1 ,b 3 )

  )d X (x) dx E[Z 2 ] = E[C(X) 2 ] = +∞ -∞ C(x) 2 d X (x) dxLet's start with E[Z]:

  Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images, " 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878. P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles. " GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France

	1.2 O

  de lourdes machines dotées de cartes graphiques (GPU) conçues pour exécuter des modèles d'apprentissage profond. De plus, les alimentations embarquées ne peuvent pas fournir suisamment d'énergie pour faire fonctionner un tel matériel. Cependant, il existe des GPU légers et économes en énergie, tels que les gammes Nvidia Xavier et Orin, qui conviennent parfaitement à des systèmes embarqués. Néanmoins, une autre contrainte subsiste : les images doivent être traitées en temps réel. Le système CAMELEON est conçu pour prendre environ une image par seconde. Il s'agit d'une fréquence d'image assez faible, mais en raison de la taille du capteur et du nombre de caméras, cela représente un ux de données de plusieurs centaines de mégapixels par seconde. Pour traiter une telle quantité d'images par seconde, les modèles de détection doivent être aussi légers et e caces que possible. Heureusement, il existe des outils capables d'optimiser l'inférence des modèles d'apprentissage profond. Le chapitre 9 présentera ces outils et comment ils peuvent être utilisés pour construire des modèles de détection su samment rapides pour les applications de Cela complexi e considérablement le problème, mais c'est un scénario beaucoup plus réaliste dans l'industrie. Les ensembles de données collectés ne peuvent qu'approximer la distribution réelle des données pour un problème spéci que. Les divergences entre les paramètres d'acquisition (appareil photo, éclairage, etc.) et les conditions réelles entraînent presque toujours une baisse des performances. En imagerie médicale, il s'agit d'un problème classique car di érents scanners ne produiront pas exactement les mêmes images. Cela empêche de former des modèles sur une collection d'images provenant d'un hôpital et de les déployer dans un autre. Le cas d'utilisation militaire est un autre exemple critique. La con dentialité des images, l'environnement et les objets d'intérêt en constante évolution rendent di cile la construction d'algorithmes de détection robustes. Compte tenu des contraintes industrielles de COSE, l'objectif principal de ce projet de thèse est de développer des méthodes de détection d'objets e caces en termes de données, basées sur des stratégies d'apprentissage few-shot. Nous avons choisi d'orienter nos recherches principalement sur Chap. 2 : Détection d'objets, apprentissage few-shot et adaptation aux domaines La détection d'objets et l'apprentissage few-shot sont des sous-domaines de la vision par ordinateur et de l'apprentissage automatique. Les deux sont nécessaires pour développer des techniques de détection capables de généraliser à partir de données limitées. Par conséquent, ce chapitre passe en revue à la fois la détection d'objets et l'apprentissage few-shot. Les deux problèmes sont dé nis et des revues détaillées des littératures respectives sont réalisées. Chap. 3 : Détection d'objets few-shot Ce chapitre présente la tâche de détection dans le régime few-shot et passe en revue la littérature existante sur ce sujet. La détection d'objets few-shot (FSOD) se situe à l'intersection de la détection d'objets et de l'apprentissage few-shot, et repose donc largement sur ces deux domaines explorés dans le chapitre 2. Tout comme pour la classi cation, di érentes approches sont explorées dans la littérature pour aborder la tâche de détection en régime few-shot. En n, ce chapitre se concentre Elle intègre des réseaux de prototypes (prototypical networks) à l'intérieur de Faster R-CNN, plus précisément à la place des couches de classi cation dans le RPN et la tête de détection. PFRCNN est appliqué à des images synthétiques générées à partir de l'ensemble de données MNIST et à des images aériennes réelles avec le jeu de données DOTA. Les performances de détection de PFRCNN sont légèrement décevantes, mais elles établissent un premier point de repère sur DOTA. Les expériences menées avec PFRCNN fournissent des informations pertinentes sur les choix de conception pour les approches FSOD.Contributions liées à ce chapitre : Ce chapitre décrit des travaux très récents et nous plani ons de soumettre des articles de recherche qui les présenterons.Cette partie ne contient qu'un seul chapitre qui présente une contribution indépendante des approches proposées dans la partie précédente. Ce chapitre remet en question la pertinence de l'Intersection sur Union (IoU), un élément clé des modèles de détection d'objets.

	I	1.1 C	,
	Part III: Repenser l'Intersection over Union 1.3 Résumé des contributions Articles de conférences internationales P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representa-tion Learning for Few-Shot Object Detection on Aerial Images, " 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: 10.1109/ICMLA52953.2021.00110. d'intégrer Contributions liées à ce chapitre : P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance
		Analysis on Aerial and Natural Images, " 2022 30th European Signal Processing Conference
		P. L. Jeune, M. Lebbah, A. Mokraoui and H. Azzag, "Experience feedback using Representation (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878.
		Learning for Few-Shot Object Detection on Aerial Images, " 2021 20th IEEE International Confer-P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small ence on Machine Learning and Applications (ICMLA), Pasadena, CA, USA, 2021, pp. 662-667, doi: Object Detection in the Few-Shot Regime", Accepté à the IEEE International Conference on 10.1109/ICMLA52953.2021.00110. Image Processing 2023 (ICIP).
		chapitre présente d'abord en détail le système CAMELEON et ses contraintes. Ensuite, nous étudions
		l'in uence de la taille du modèle sur ses performances et présentons des outils et des astuces utiles
		pour accélérer l'inférence. En n, nous expliquons comment les modèles de détection sont déployés
		à l'intérieur du prototype CAMELEON et comment ils se comportent sur des images aériennes.

COSE. Cela résout les problématiques liées au déploiement et à l'inférence des modèles, cependant, une préoccupation majeure subsiste : comment entraîner ces détecteurs d'objets ?

Les méthodes basées sur l'apprentissage, et en particulier les modèles d'apprentissage profond, reposent fortement sur les données pour leur entraînement. En général, les performances globales d'un modèle dépendent de la quantité et de la qualité des données annotées disponibles lors de l'entraînement. Pour la détection, la collecte de grands ensembles de données annotées est chronophage et coûteuse. Dans certains cas, il est même impossible de rassembler de tels ensembles de données pour l'entraînement. Dans le domaine médical, par exemple, les réglementations empêchent souvent l'utilisation de données personnelles. Dans le domaine militaire, cela est encore plus di cile car les données d'entraînement sont classi ées et ne peuvent être divulguées en aucune circonstance. Cela est problématique pour l'entraînement des méthodes d'apprentissage profond, gourmandes en données. Heureusement, il existe des stratégies d'apprentissage beaucoup plus efcaces en termes de données. Ces méthodes sont généralement désignées comme l'apprentissage frugal (few-shot learning (FSL) en anglais). Une revue détaillée de ces méthodes sera présentée dans la section 2.2. Bien qu'il existe de nombreuses approches di érentes pour l'apprentissage few-shot, elles suivent souvent le même principe de base. Premièrement, elles apprennent des connaissances générales sur une tâche connexe (tâche source), puis elles s'adaptent à une tâche cible. Ces deux phases d'entraînement sont appelées entraînement de base et ne-tuning. Dans le cas de la détection, une tâche désigne un ensemble de classes à détecter, et ce problème est alors appelé détection d'objets few-shot (FSOD en anglais). Un grand ensemble de données annotées contenant des annotations d'objets appartenant à C base est disponible. La tâche source consiste à détecter ces objets. Ensuite, la tâche cible consiste à détecter des objets des classes nouvelles, en disposant uniquement d'un nombre limité d'annotations. Le chapitre 3 fournit une revue approfondie des travaux existants dans ce domaine. Dans le cas général, la tâche cible est réalisée sur des images similaires à celles vues pendant l'entraînement de base. Cependant, la tâche cible peut aussi être réalisée avec di érents types d'images. Par exemple, la tâche source peut être apprise à partir d'images naturelles tandis que la tâche cible porte sur des images aériennes ou médicales. On appelle cela l'adaptation au I domaine. le problème de la détection d'objets few-shot, c'est-à-dire l'adaptation aux nouvelles classes. Bien qu'un aperçu détaillé de la thèse soit présenté dans la prochaine section, ici les principales parties de ce travail sont décrites. Tout d'abord, dans la partie I présente une revue approfondie de la littérature sur la détection d'objets, l'apprentissage frugal et en n la détection d'objets few-shot. Ensuite, trois approches distinctes sont proposées pour la détection d'objets few-shot dans la partie II. Cette partie comprend également des expériences dans le cadre de l'adaptation au domaine, principalement dans la section 7.4. Nos expériences se concentrent principalement sur des jeux de données d'images aériennes disponibles publiquement, faute d'ensembles de données privées disponibles au sein de l'entreprise. Ces jeux de données contiennent des annotations de détection et seront présentés dans le chapitre 2. La partie III présente une alternative à l'Intersection over Union (IoU), une mesure de similarité des boîtes englobantes largement utilisée en détection d'objets. Son utilisation pour l'évaluation et l'entraînement des modèles est discutée dans le chapitre 8. En n, le chapitre 9 fournit des détails sur le déploiement des modèles de détection d'objets en fonction des contraintes de COSE. 1.2 Plan de la thèse Cette section présente un aperçu du contenu de chaque chapitre de cette thèse. Cela prend la forme d'un court résumé par chapitre. Ces résumés seront répétés pour plus de commodité au début des chapitres correspondants. Partie I : Revue de la littérature sur la détection d'objets, l'apprentissage few-shot et la détection few-shot La première partie de cette thèse est composée de trois chapitres. Les deux premiers présentent la littérature sur la détection d'objets, l'apprentissage few-shot et la détection d'objets à faible échantillonnage. Le troisième chapitre, quant à lui, explore les dé s liés à l'application de la détection d'objets few-shot à des images aériennes et présente notre première contribution : une analyse détaillée de ces di cultés. 1.2 P sur l'application de la détection d'objets few-shot sur des images aériennes et sur les extensions du régime few-shot. Chap. 4 : Analyse des di cultés liées à la détection few-shot La tâche de détection devient extrêmement di cile lorsque les données annotées sont limitées. Dans ce chapitre, les raisons derrière ces di cultés sont explorées. En particulier, nous nous concentrons sur le cas des images aériennes pour lesquelles il est encore plus di cile d'appliquer des techniques de détection few-shot. Il s'avère que les petits objets sont particulièrement di ciles à localiser en régime few-shot et sont la principale source d'erreur dans les images aériennes. Contributions liées à ce chapitre : P. Le Jeune and A. Mokraoui, "Improving Few-Shot Object Detection through a Performance Analysis on Aerial and Natural Images, " 2022 30th European Signal Processing Conference (EUSIPCO), Belgrade, Serbia, 2022, pp. 513-517, doi: 10.23919/EUSIPCO55093.2022.9909878. P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles. " GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France. Part II : Amélioration de la détection few-shot à travers plusieurs approches La deuxième partie de cette thèse présente nos principales contributions dans le domaine de la détection d'objets few-shot (FSOD). Chaque chapitre propose une nouvelle approche pour aborder le problème de FSOD et discute de ses avantages et inconvénients par rapport aux méthodes existantes. Ces contributions ont abouti à plusieurs articles acceptés et soumis dans des conférences et des revues internationales et nationales. Partie 5 : Retour d'expérience sur l'apprentissage de métrique pour FSOD Prototypical Faster R-CNN (PFRCNN) est une approche innovante pour la détection d'objets few-shot I (FSOD) basée sur l'apprentissage de métriques. Chap. 6 : Un environnement modulaire pour la détection few-shot basée sur des mécanismes d'attention Comparer de manière équitable di érents modèles est extrêmement di cile en détection d'objets few-shot car de nombreuses options architecturales di èrent d'une méthode à une autre. Les approches basées sur l'attention ne font pas exception, et il est di cile d'évaluer quels mécanismes sont les plus e caces pour le FSOD. Ce chapitre présente un environnement modulaire pour réimplémenter les techniques existantes et concevoir de nouvelles approches. Il permet de xer tous les hyperparamètres à l'exception du mécanisme d'attention et de les comparer de manière équitable. En utilisant cet environnement, nous proposons également un nouveau mécanisme d'attention spéci quement conçu pour les petits objets. Contributions liées à ce chapitre : Ƽ P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Soumis à Elsevier Pattern Recognition journal. P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepté à the IEEE International Conference on Image Processing 2023 (ICIP). Chap. 7 : FSDi usionDet: un détecteur few-shot basé sur les modèles de di usion et une stratégie de ne-tuning Les chapitres précédents explorent la détection d'objets few-shot en utilisant l'apprentissage métrique et les techniques basées sur l'attention. Ce chapitre se concentre logiquement sur la dernière grande approche pour le FSOD : le ne-tuning. En nous basant sur Di usionDet, un récent modèle de détection utilisant des modèles de di usion, nous construisons une stratégie de netuning simple et e cace, baptisée FSDi usionDet. FSDi usionDet surpasse état de l'art en FSOD sur des jeux de données aériens et obtient des performances compétitives sur les images naturelles.

Des études expérimentales approfondies explorent les choix de conception de la stratégie de ne-1.2 P tuning a n de mieux comprendre les composantes clés nécessaires pour atteindre une telle qualité. En n, ces résultats impressionnants permettent de considérer des scénarios plus complexes comme l'adaptation à de nouveaux domaines, ce qui est particulièrement pertinent pour COSE. Partie 8: Intersection over Union adaptable à la taille des objets L'Intersection sur Union (IoU) n'est pas une mesure de similarité de boîte englobante optimale pour l'évaluation et l'entraînement des détecteurs d'objets. Pour l'évaluation, elle est trop stricte avec les petits objets et ne correspond pas bien à la perception humaine. Pour l'entraînement, elle crée un déséquilibre entre les petits et grands objets souvent au détriment des petits. Nous proposons l'Intersection sur Union adaptative à l'échelle (appelée SIoU), une alternative paramétrable qui résout les lacunes de l'IoU. Des arguments empiriques et théoriques sont avancés pour démontrer la supériorité de la SIoU grâce à une analyse approfondie de celle-ci et d'autres critères existants. Contributions liées à ce chapitre : Ƽ P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Soumis à International Conference on Computer Vision 2023 (ICCV). P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France. Partie IV: Prototypage et déploiement industriel En n, la dernière partie de cette thèse présente nos contributions industrielles. Cette partie est cruciale pour COSE car elle comble l'écart entre les avancées de la recherche et les applications du monde réel. Par conséquent, le seul chapitre de cette partie aborde les aspects techniques de la détection d'objets et n'est associé à aucune contribution académique. Chap. 9: Intégration dans les prototypes de COSE Les modèles de détection sont souvent lourds et ne conviennent pas bien à l'application de COSE. Ce Articles de conférences nationales P. Le Jeune and A. Mokraoui, "Amélioration de la détection d'objets few-shot à travers une analyse de performances sur des images aériennes et naturelles. " GRETSI 2022, XXVIIIème Colloque Francophone de Traitement du Signal et des Images, Nancy, France. P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France. Articles soumis Ƽ P. Le Jeune and A. Mokraoui, "A Comparative Attention Framework for Better Few-Shot Object Detection on Aerial Images", Soumis à Elsevier Pattern Recognition journal. Ƽ P. Le Jeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Soumis à International Conference on Computer Vision 2023 (ICCV).

  First, the detector generally leverages regression techniques to predict box coordinates and outputs continuous box coordinates. This is incompatible with the annotated box coordinates, which are usually discrete. Rounding errors can cause TP to become FP. Then, from an application viewpoint, pixel-perfect bounding boxes are not necessary. Therefore, it is generally admitted that a true positive detection is a box close enough to a ground truth box. The similarity between two bounding boxes is almost always computed with the Intersection over Union (IoU).

	2.1 O	D
	the predicted boxes.

  Two-Stage DetectorsRegions with CNN features or R-CNN[START_REF] Girshick | Rich feature hierarchies for accurate object detection and semantic segmentation[END_REF] is one of the rst attempts to tackle the task of Object Detection with CNNs. This marks a signi cant performance improvement over the previous methods (about 20% mAP improvement over the best DPMs on the 2010 Pascal VOC challenge).

	C	2 O	D	, F	S	L	C	D	A 2.1 O	D
		One-Stage Detectors
		One-stage detectors appeared slightly later than two-stage ones. The reason for this is probably
		because two-stage models were the logical continuity of the sliding-window-based older detectors.
		The idea These approaches are highly ine cient as they process parts of the input images many times. While
	behind R-CNN is to leverage the classi cation power of CNNs such as AlexNet for the detection this is reduced in modern two-stage detectors, they still have redundancies that limit their inference
	task. It rst employs Selective Search [30], a class-agnostic object locator, to generate region pro-speed. In 2015, Redmon et al. proposed You Only Look Once (YOLO) [34], a rst detector to avoid
	posals. For each region proposal, the corresponding part in the input image is cropped and fed to a all redundancy as it only needs to look once at each part of the image. The main idea behind YOLO
	CNN pre-trained on ImageNet. The CNN outputs high-dimensional feature vectors which are then is to reformulate OD as a regression problem and not a classi cation one. Prior detectors solve OD
	classi ed by an SVM for each class. The CNN is ne-tuned for the detection task by replacing its by classifying regions of the input image, i.e., classifying given a region of interest. YOLO instead
	nal classi cation layer with N+1 class logits (one additional class for the background) and training regresses the box coordinates and classi es the object jointly.
	with a Cross-Entropy (CE) loss function. Proposal regions with an IoU of 0.5 with a ground truth
	box are selected as positive proposals and the model is trained to classify these regions with the label
	of their corresponding ground truth. The other proposals (denoted negative proposals) are selected
	as background examples. The authors train classi cation SVMs instead of using the classi cation
	outputs of the CNN as they observe higher performance with SVMs. In addition, they train a linear
	bounding box regressor to re ne the coordinates of each region of interest, following the most re-
	cent advances with DPM. OverFeat [3] was another attempt to solve detection with CNN. Although
	it did win the ImageNet Detection Challenge in 2013, it is largely outperformed by R-CNN and the
	corresponding paper was never published. Following R-CNN, the rst author Ross Girscick pro-
	posed two successive extensions. Firstly, Fast R-CNN [31], mainly improves over R-CNN in terms
	of speed. It introduces a Region of Interest (RoI) Pooling layer which extracts RoI features directly
	from the features maps of the entire image. This is largely inspired by Spatial Pyramid Pooling [32]
	which consists in pooling the features of an RoI with multiple binning resolutions and concatenat-
	ing the outputs. RoI Pooling saves a lot of unnecessary forward passes through the CNN (R-CNN
	performs a forward pass for each RoI). Then, they dropped the SVM classi ers for the CNN outputs
	and integrated a bounding box regressor at the end of the CNN as a parallel branch to the classi -
	cation layer. The training is done in a similar fashion as in R-CNN, they simply added a regression
	loss function, computed only on the positive proposals to train the bounding box regression branch.
	Secondly, Faster R-CNN [33] introduced the Region Proposal Network (RPN) to replace Selective
	Search as the proposal generation algorithm. Selective Search provides almost exhaustive propos-
	als but is slow. The RPN is a lightweight CNN that densely predicts proposal coordinates and an
	objectness score at each position in the feature map. The box predictions are done as coordinate
	shifts from a set of pre-de ned anchor boxes. After objectness ltering, this produces a reasonable
	number of proposals that can be processed with Fast R-CNN. The RPN is trained like Fast R-CNN
	with a similar loss function. A binary classi cation loss as the RPN classi es each location between
	objects or background (with the objectness score), and a regression loss. Example selection remains detectors and two-stage detectors. As two-stage detectors were proposed rst, we will present them unchanged compared to Fast R-CNN. The training of both the RPN and Fast R-CNN is done in an rst here. end-to-end manner. Faster R-CNN achieves superior performance compared to its predecessors, but
	most importantly, it unlocks real-time detection with deep learning-based two-stage approaches.

Table 2

 2

	.2: Brief description of some existing prediction ground truth matching strategy in existing
	object detectors.

  2.3 of the most well-known object detection datasets in the literature.

	D	, F	S	L 2.2 F	S	L	C	D : L	A	L	D
	Pascal VOC 2012 includes images of various sizes and aspect ratios. Each image is annotated with tion is not the primary interest of COSE, it is worth exploring the FSC literature as it is an older eld,
	horizontal bounding boxes around objects belonging to 20 classes. Examples of images and a list of much more developed than FSOD and because FSC lays the foundation for tackling more complex
	classes, ordered by the number of occurrences, are available in Fig. 2.6a. tasks in the few-shot regime. On the other hand, domain adaptation consists in adapting to di er-
	ent kinds of images, e.g., di erent seasons, weather conditions, general environments, etc. In the MS COCO [6] -MS COCO is an extension of Pascal VOC which includes much more images and strict de nition of domain adaptation, the classes of interest remain the same. However, the setting classes. The set of images is completely distinct from Pascal VOC, but all classes in Pascal VOC are when both the classes and the domain change is also studied in the literature. It is more challenging, included in COCO. Similarly, Fig. 2.6b presents image examples and a list of COCO classes. but it better re ects the industrial needs such as COSE's. In this section, we review both kinds of
	2.1.4.2 Aerial Images adaptations for the classi cation problem. Even though it is not a task of interest for COSE, under-
	The overall goal of this project is to detect objects from aerial images. Aerial images are sometimes standing few-shot adaptation strategies is crucial before addressing the more challenging problem
	Image Type / Application associated with low-altitude drone images. These images are halfway between natural and aerial Dataset Name # Classes # Images # Instances of Few-Shot Object Detection which we reserve for Chap. 3.
	Natural images as they often preserve some perspective. Remote Sensing Images (RSI), i.e., acquired from Pascal VOC [5] 20 11.5k 27k COCO [6] 80 117k 1.5M LVIS [96] 1203 100k 2.2.1 Few-shot Classi cation planes or satellites with nadir-oriented cameras are much closer to COSE's application. In this 1.3M Object365 [97] 365 2M manuscript, we refer to this kind of image both as aerial or RSI images. There exist a few publicly 2.2.1.1 Problem De nition 30M Autonomous Vehicle KITTI [98] 11 7k available datasets of such images. We have chosen two of them based on the ground resolution of Classi cation is a simpler problem than detection. Given a set of classes C and an input image 80k BDD100k [99] 10 400k 3M the images (in agreement with COSE systems) and their availability at the beginning of this project. I, one wants to nd the class c ∈ C that is depicted by I. Of course, the higher considerations
	Pedestrian brie y presented in Sec. 2.1.1 about how to properly de ne the membership of an image to a class CityPerson [100] 1 3k 19k TinyPerson [101] 1 1610 DOTA [77] -DOTA contains images coming from Google Earth and distinct satellites Jilin-1 and still holds. For classi cation as well, the class membership is determined by human appraisal and 72k CrowdHuman [102] 1 15k Gaofen-2 (with roughly 1m spatial resolution GSD). Images from DOTA are large, ranging from common sense. Solving a classi cation task is to nd a model F(•, θ) that outputs a class label for 340k Aerial COWC [103] 1 33k DOTA [77] 16 2.8k (megapixels) DIOR [95] 20 23k 190k overlap and converted the annotations to horizontal bounding boxes. Fig. 2.7a presents images and 220k 800 to 4000 pixels in width and objects are annotated with oriented bounding boxes. To ease the handling of the images, we prepared DOTA by tiling all images into 512 × 512 patches with a 50% a given input image I:
	the class list for DOTA.		xView [104]			60	1.1k (megapixels)		1M
	FAIR -1M [105] DIOR [95] -DIOR is very similar to DOTA. It contains only images scrapped from Google Earth 37 15k (megapixels) 1M
	Agricultural / Food and has slightly more classes than DOTA. The images are already tiled at 800×800 pixels and boxes DeepFruits 7 457 2.5k Oktobeerfest [106] 15 1k 2.5k are horizontal. Fig. 2.7b presents images from DIOR and the list of classes.
			ClipArt [107]			32		5k		13k
	LogoDet [108] SIXray [109] DroneVehicle [110] 2.2 Few-Shot Learning: Learning with Limited Data 3000 159k Other Modalities 6 9k 5 56k		194k 1M 191k
	As presented in the introduction, COSE faces a substantial challenge in the design of its imaging Table 2.3: Overview of existing detection datasets systems: the lack of real-case images and unknown objects of interest. All methods described in
	Sec. 2.1 require large annotated training sets to achieve reasonable detection performance, which is
	2.1.4.1 Natural Images misaligned with COSE's constraints. This issue is common in the industry, most computer vision
	Natural images are the kind of image humans are the most familiar with, therefore it is logically the problems lack large annotated datasets, and therefore the direct application of research contribu-
	most common application in Computer Vision. Object detection is no exception and most proposed tions is often challenging. Fortunately, there exists an entire research eld dedicated to learning
	detectors are developed to process natural images. Hence, our analysis must be conducted as well with limited annotated data. The main paradigm in this eld is to learn a closely related task with
	on natural images even though the industrial interest of COSE is more towards aerial imagery. To su cient data and adapt to the real task with limited annotations. Two kinds of adaptation can be
	this end, we choose Pascal VOC and MS COCO as our main sources of natural images. considered: class adaptation and domain adaptation. Given a computer vision task such as classi -
	Pascal VOC [5] -The Pascal VOC challenge took place every year between 2005 and 2012. This cation, the former consists in learning to classify objects or images among a set of classes and then
	competition de ned the object detection problem as we know it today. The last version of the dataset adapt to another set of classes. This is usually called Few-Shot Classi cation (FSC). While classi ca-

  Taxonomy of the Few-Shot Classi cation literature. To illustrate each category of the taxonomy, a few papers are selected as representatives among others. Papers marked with a * are not solving the FSC task but are included in this gure as no contribution in the literature tackles classi cation from this perspective.

	2.2 F	S	L	: L	L	D
	Figure 2.8: will be way lower on the test set than on the training set. Catastrophic forgetting is a more subtle
	issue. It happens when the performance of the ne-tuned model on the base classes drops. In the
	case of the simple FSC it is a subtle issue, but it becomes more challenging when dealing with
	extended setups such as generalized FSL and Continual Learning (see Sec. 2.2.1.7 for more details).
	2.2.1.7
	2.2.1.2 Fine-tuning
	Probably the most straightforward way to tackle the FSC task is to employ ne-tuning or transfer
	learning. This method trains the model on D base and then the model weights are ne-tuned using
	D novel with only the few examples available. It works well but the ne-tuned models are prone to
	strong over tting and catastrophic forgetting [119]. Over tting on the novel set is problematic as
	it means that the ne-tuned model will have poor generalization capabilities, i.e., its performance

Table 2 .

 2 4: Summary of the various avors of classi cation tasks existing in the literature. The second column, classes of interest, denotes what is the overall goal of the task. The last column presents the possible class setup encountered both in the query and support set (C query and C support respectively.). C unknown represents additional classes that should be identi ed in the open-set setting. In the detection task, C background denotes all object classes that can be present in the background and that should not be detected.

				2.2 F	S	L	: L	L	D
	Task	Classes of interest		Novel supervision	Query-support interaction
	Regular Classi cation		C base			None	None
	Few-shot Classi cation	C novel	K examples per novel class	C query = C support
	Zero-shot Classi cation	C novel	External information (e.g., class labels)	None
	Generalized FSC	C base ∪ C novel	K examples per novel class	C query = C support
	FS Open-set Recognition	C novel ∪ C unknown	K examples per novel class None for unknown classes	C query ⊂ C support or C query ⊃ C support
	Continual Learning	C base ∪	i C i novel	K examples per novel class	None
	Few-Shot Object Detection	C novel \C background	K annotated images per novel class	C query = C support

can be computed using the unlabelled examples available at test time.

  •, θ), with parameters θ, able to adapt to new classes (C novel ) from only a limited number of examples. Just as for the few-shot classi cation problem, two datasets are available, a base dataset with plenty on annotations of base classes instances C base and a novel dataset (also called support set) with K annotated images for each novel class:

Table 3 .

 3 1: List of the most relevant contributions to the Few-Shot Object Detection eld. These works are grouped according to the general approach employed to tackle FSOD and sorted by their year of publication. Green rows signify that the methods were applied to aerial images and § indicates that these images are SAR images. † signals that it was applied to generalized FSOD while ‡ means that it was developed in an incremental setting. with the few available annotations. This is leveraged by Low Shot Transfer Detector (LSTD)[START_REF] Hao | Lstd: A low-shot transfer detector for object detection[END_REF], a pioneer work on FSOD. It rst trains a Faster R-CNN on a base dataset and ne-tunes it on a support set containing only some examples of the novel classes. Regularization losses are introduced to prevent over tting. Before ne-tuning, the last layer of the classi er branch is replaced with a randomly initialized layer with the right number of outputs (i.e., the number of novel classes |C novel |). Closely related,[START_REF] Wang | Frustratingly Simple Few-Shot Object Detection[END_REF] leverages the same idea without any additional loss. It applies orthogonality constraints to the gradient during ne-tuning to prevent forgetting base knowledge. This mostly helps for the generalized FSOD setting, but it is also bene cial for regular FSOD. Another extension on top of the basic ne-tuning approaches is to add a re nement

	Approach	Abbreviation	Venue	Date Detection Framework Multiscale	Datasets	Aerial / Natural Images
		FRW [220]	ICCV	2019	YOLO	No	Pascal / COCO	Natural
		OSOD-CACE [221] NEURIPS 2019	Faster RCNN	Yes	Pascal / COCO	Natural
		Meta R-CNN [222]	ICCV	2019	Faster RCNN	No	Pascal / COCO	Natural
		FSOD-RSI [223]	TGRS	2020	YOLO	Yes	DIOR / NWPU VHR	Aerial
		ARPN [224]	CVPR	2020	Faster RCNN	Yes	COCO	Natural
		VEOW [225]	ECCV	2020	Faster RCNN	Yes	Pascal / COCO	Natural
		KT [226]	SMC	2020	Faster RCNN	Yes	Pascal	Natural
		OSOD-WFT [227]	Preprint 2020	FCOS	Yes	Pascal / COCO / ImageNet Loc	Natural
		ONCE [228] ‡	CVPR	2020	Center Net	No	Pascal / COCO / Deepfashion	Natural
		WSAAN [229]	TAEORS 2021	Faster RCNN	Yes	RSOD / NWPU VHR	Aerial
	Attention	FSOD-FPDI [230] Meta-FRCNN [231]	MDPI AAAI	2021 2022	FCOS Faster RCNN	Yes Yes	DOTA / NWPU VHR Pascal / COCO	Aerial Natural
		Meta-DETR [232]	TPAMI	2021	DETR	No	Pascal / COCO	Natural
		DRL [233]	Preprint 2021	Faster RCNN	Yes	Pascal / COCO	Natural
		DANA [234]	TM	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		SP [235]	Access	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		JCACR [236]	ICIP	2021	YOLO	Yes	Pascal / COCO	Natural
		TI-FSOD [237]	CVPR	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		SAM [238]	MDPI	2021	Faster RCNN	No	NWPU VHR-10 / DIOR	Aerial
		FSOD-FCT [239]	CVPR	2022	Faster RCNN	No	Pascal / COCO	Natural
		SAR-DRM [240]	TGRS	2022	Faster RCNN	No	FUSAR-GEN	Aerial §
		FSOD-PSI [241]	JDT	2022	YOLO	Yes	Pascal / COCO	Natural
		SAFT [242]	CVPR	2022	FCOS	Yes	Pascal / COCO	Natural
		APSP [243]	WACV	2022	Faster RCNN	No	Pascal / COCO	Natural
		KFSOD [244]	CVPR	2022	Faster RCNN	Yes	Pascal / COCO	Natural
		FSODS [245]	TGRS	2022	YOLO	Yes	SMCDD-FS	Aerial §
		TIN-FSOD [246]	Arxiv	2023	Faster RCNN	Yes	NWPU VHR/ DIOR / HRRSD	Aerial
		FSOD-ICF [247]	WACV	2023	Faster RCNN	Yes	Pascal / COCO	Natural
	Attention / Metric Learning	PNPDet [248] UPE [249] GenDet [250]	WACV ICCV NNLS	2021 2021 2021	Center Net Faster RCNN FCOS	No Yes Yes	Pascal / COCO Pascal / COCO Pascal / COCO	Natural Natural Natural
		RepMet [251]	CVPR	2018	Faster RCNN	Yes	Pascal / ImageNet Loc	Natural
		RN-FSOD [252]	NEURIPS 2020	Faster RCNN	Yes	Pascal / ImageNet Loc	Natural
	Metric learning	MDODD [253] †	ICCV	2021	Faster RCNN	No	Pascal / COCO	Natural
		FSCE [254]	CVPR	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		GD-FSOD [255]	NEURIPS 2021	Faster RCNN	Yes	Pascal / COCO	Natural
		LSTD [256]	AAAI	2018	Faster RCNN	Yes	Pascal / COCO / ImageNet Loc	Natural
		MSPSR [257]	ECCV	2020	Faster RCNN	Yes	Pascal / COCO	Natural
		TFA [258]	ICML	2020	Faster RCNN	Yes	Pascal / COCO / LVIS	Natural
		WOFG [259] †	CVPR	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		Hallu-FSOD [260]	CVPR	2021	Faster RCNN	Yes	Pascal / COCO	Natural
		DHP [261]	ICCVW	2021	Faster RCNN	Yes	iSAID / NWPU VHR	Aerial
		LVC [262]	CVPR	2021	Faster RCNN	No	Pascal / COCO	Natural
		FSCN [263]	CVPR	2021	Faster RCNN	Yes	Pascal / COCO	Natural
	Fine-tuning Strategy	FADI [264] DeFRCN [265] SIMPL [266]	NEURIPS 2021 ICCV 2021 TAEORS 2022	Faster RCNN Faster RCNN YOLO	Yes Yes No	Pascal / COCO Pascal / COCO xView (plane only)	Natural Natural Aerial
		DETReg [267]	CVPR	2022	Deformable DETR	Yes	COCO	Natural
		CFA [268] †	CVPRW 2022	Faster RCNN	No	Pascal / COCO	Natural
		CIR [269]	TGRS	2022	Faster RCNN	Yes	NWPU VHR-10 / DIOR	Aerial
		NIMPE [270]	ICASSP	2022	Faster RCNN	Yes	COCO	Natural
		HDA [271]	IROS	2022	Faster RCNN	Yes	COCO	Natural
		MDB [272]	LNCS	2022	Faster RCNN	No	Pascal / COCO	Natural
		DCB [273] †	NEURIPS 2022	Faster RCNN	Yes	Pascal / COCO	Natural
		CPP-FSOD [274]	Preprint 2023	Faster RCNN	Yes	Pascal / COCO	Natural
		I-DETR [275] ‡	AAAI	2023	Deformable DETR	No	Pascal / COCO	Natural
	Meta-Learning	MetaDet [276] Sylph [277] ‡	ICCV CVPR	2019 2022	Faster RCNN Faster RCNN	No No	Pascal / COCO COCO / LVIS	Natural Natural
	Zero-shot	TL-ZSOD [278]	ICCV	2019	RetinaNet	Yes	COCO	Natural
	Object Detection	ML-CMP [279]	Preprint 2022	Faster RCNN	No	Pascal / COCO	Natural
		OA-FSUI2IT [280]	AAAI	2022	Faster RCNN	Yes	Multiple datasets	Natural
		Acro FOD [281]	ECCV	2022	YOLO	Yes	Multiple datasets	Natural
	Cross-domain	CD-CutMix [282]	ACCV	2022	Faster RCNN	No	Multiple datasets	Natural
		CD-FSOD [283]	Preprint 2022	Faster RCNN	Yes	Multiple datasets	Aerial
		CD-MDB [284]	ECCV	2022	Faster RCNN	Yes	Multiple datasets	Aerial

ilar to FSC: train a detection model to detect base classes on a large dataset and then ne-tune it S O D step to lter the bounding boxes predicted by the ne-tuned network. For instance,

[START_REF] Li | Few-shot object detection via classi cation re nement and distractor retreatment[END_REF]

proposes a Few-Shot Correction Network (FSCN) whose goal is to assist the detector classi cation branch. It is trained directly on the false positive of the detector to speci cally target challenging situations.

  Plenty of other works leverage class prototypes for the classi cation part of the detector. However, various tricks are proposed in the literature to improve the quality and use of the prototypes. For

	F	S	O	D	L
	instance, Universal Prototype Enhancement (UPE) [249] re nes prototypes with a ne transforma-
	tion to convert image-level representations into object-level prototypes much more adapted to the
	detection task. Also, it does not leverage the prototypes directly as a classi er but rather uses them
	to adapt query features before classi cation and regression. Similarly, GenDet (Generate Detectors

Metric-learning-based methods are extensively employed for few-shot classi cation. Metric learning is designed especially for classi cation and cannot handle bounding box regression. Thus, it cannot be directly applied to object detection. However, several attempts were made to tackle FSOD with metric-learning techniques, mostly replacing the classi cation branch of the model with prototypical networks or closely related variants and keeping the regression branch unchanged. Of course, even the classi cation adaptation is not straightforward as object detection includes a special background class that should be processed with care. Among these attempts, RepMet

[START_REF] Karlinsky | Repmet: Representative-based metric learning for classi cation and few-shot object detection[END_REF]

learns class representative vectors to classify Regions of Interest (RoI) in Faster R-CNN according to their distance to the closest class prototype. Class vectors are initialized with support image representations and then ne-tuned via backpropagation. The ne-tuning is based on a cross-entropy loss and a margin metric loss which favors tight clusters in the embedding space. The background class probability is computed as the complementary probability of the most probable class. Closely related, Plug-and-Play Detectors (PNPDet)

[START_REF] Zhang | PNPDet: E cient few-shot detection without forgetting via plug-andplay sub-networks[END_REF]

learns prototype vectors as well as scale factors. In addition, they replace the Euclidean distance from RepMet with a Cosine similarity measure. Similarly, FSCE

[START_REF] Sun | FSCE: Few-shot object detection via contrastive proposal encoding[END_REF]

adds a contrastive head on top of a pre-trained detector during ne-tuning. This head outputs embedding for each RoI. A contrastive loss is optimized to bring closer the representations of same-class RoI and repel RoI with no objects. Likewise,

[START_REF] Zhao | Morphable detector for object detection on demand[END_REF]

leverages prototypes as well but deals with the background class separately with a learnable binary classi er. from Few Shots)

[START_REF] Liu | Gendet: Meta learning to generate detectors from few shots[END_REF]

combines the technique from RepMet and UPE, i.e., learnable prototypes to

Table 3 . 2

 32

			plane, ship, baseball-diamond,
			basketball-court, ground-track-eld,
	DOTA	storage-tank, tennis-court, soccer-ball-eld	harbor, bridge, small-vehicle, large-vehicle, roundabout,
			swimming-pool, helicopter,
			container-crane
			airport, basketball court, bridge,
			chimney, dam, expressway service
	DIOR	airplane, baseball eld, tennis court, train station, wind mill	area, expressway toll station, golf course, ground track eld, harbor,
			overpass, ship, stadium, storage tank,
			vehicle

: Base / Novel class splits for the di erent datasets used throughout this thesis. The novel classes in the COCO dataset correspond to all classes in Pascal VOC. S O D Figure 3.4: Illustration of two evaluation processes existing in the literature. One for ne-tuning methods (left) and one available for all other methods (right).

.

  24 13.35 45.55 12.19 49.78 12.52 56.67 16.92 56.41 15.48 58.78 20.64 59.92 28.22 61.70 30.94 62.58 32.82 3 46.50 25.32 44.18 24.42 49.67 20.70 58.05 25.08 51.72 13.84 59.14 27.26 63.34 31.12 63.52 42.19 64.18 33.95 5 48.60 29.57 47.56 31.44 53.49 24.96 60.75 32.58 60.79 30.32 62.12 34.16 64.35 46.33 64.68 46.16 65.20 42.59 10 48.52 37.10 46.72 35.12 53.25 34.39 61.47 35.56 61.88 33.41 62.49 36.43 63.16 48.71 65.27 51.70 65.03 50.30

					4.2 T I		C	A	I
		DOTA			DIOR			Pascal VOC
	FRW	WSAAN	DANA	FRW	WSAAN	DANA	FRW	WSAAN	DANA
	K Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel
	1 47

Table 5 .

 5

	PFRCNN Baseline	+HEM	+MA		+BC
	K Base	Novel	Base Novel	Base Novel	Base Novel
	1 35.5	2.1	31.2	4	26.5	6.9	13.3	4.3
	3 35.9	2.7	35.6	2.3	33.9	3.5	14.5	4.1
	5 34.3	3.8	41.2	3.3	37	4.2	18.2	4.7
	10 30.4	4.1	34.3	2.6	35.1	5.9	14.8	2.6

1: Ablation study about the training tricks described in section 5.2.3. Each column corresponds to the addition of each trick on top of the previous one. HEM, MA and BC correspond respectively to Hard Example Mining (HEM), Moving Average (MA) prototypes and Background Clustering (BC). Detection performance is reported as mAP with a 0.5 IoU threshold. Blue and red values represent the best performance on base and novel classes respectively.

Table 5 .

 5 5.2. This table reports the mean AveragePrecision (mAP) with an IoU threshold of 0.5 (see Sec. 2.1.2 for more details about mAP). The results are given with multiple values of K, the number of support examples, and two distinct splits of the base and novel classes. The evaluation is done on an unseen test set, from which the support examples are sampled as well. The table provides the mAP both for base and novel classes separately as we do not consider the generalized few-shot setting. 2: PFRCNN performance on MNIST-LOC dataset with two distinct class splits. On the left, novel classes are 0, 1 and 4, while on the right novel classes are 3, 5, and 8. In both cases, all other classes belong to the base class set. Performance is reported as mAP 0.5 . The last row reports the performance of a vanilla Faster R-CNN trained in a regular data regime, i.e., with all available annotations in the dataset. For Faster R-CNN, per-class performance is averaged over base and novel classes separately to compare with the few-shot techniques.

	F	M	L	FSOD
		Split 1: [0, 1, 4]	Split 2: [3, 5, 8]
	Data regime Base Classes Novel Classes Base Classes Novel Classes
	1 shot	94.86	21.86	92.46	19.43
	3 shots	95.70	20.39	94.82	21.22
	5 shots	95.10	24.34	94.95	21.73
	10 shots	95.86	23.19	93.11	20.17
	Faster R-CNN	76.86	96.33	84.29	79.01

1

Link to Prototypical Faster R-CNN repository.

Table 5 .

 5 3: PFRCNN performance on DOTA dataset with two distinct class splits. Split A has classes plane, ship and tennis-court and Split B has harbor, roundabout and helicopter. In both cases, all other classes belong to the base class set. Performance is reported as mAP 0.5 . The last row reports the performance of a vanilla Faster R-CNN trained in a regular data regime, i.e., with all available annotations in the dataset. For Faster R-CNN, per-class performance is averaged over base and novel classes separately to compare with the few-shot techniques.

much lower than the regular setup (i.e., Faster R-CNN trained on the whole DOTA). For novel classes, a similar performance drop is observed, making PFRCNN un t for any industrial application.

  Fair comparison of Few-Shot Object Detection Methods with AAF. . . . . 115 6.3 Cross-Scales Query-Support Alignment for Small FSOD . . . . . . . . . . . . . 120 6.3.1 XQSA de nition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 6.3.2 Ablation study XQSA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 6.3.3 Application to aerial and natural images . . . . . . . . . . . . . . . . . . . 123 6.3.4 XQSA Implementation Challenges and Extensions . . . . . . . . . . . . . 126 6.4 Qualitative Comparison within the AAF framework . . . . . . . . . . . . . . . 127 6.4.1 Base Classes Detection Quality . . . . . . . . . . . . . . . . . . . . . . . . 127 6.4.2 Novel Classes Detection Quality . . . . . . . . . . . . . . . . . . . . . . . 127 6.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129

	F	F	FSOD C
	6.2.5
			. 106
	6.2 Alignment Attention Fusion Framework . . . . . . . . . . . . . . . . . . . . . 107
	6.2.1 Query-Support Alignment . . . . . . . . . . . . . . . . . . . . . . . . . . 107
	6.2.2 Global Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
	6.2.3 Fusion Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
			110

Framework for Better Few-Shot Object Detection on Aerial Images", Submitted at the Elsevier Pattern Recognition journal. P. Le Jeune and A. Mokraoui, "Cross-Scale Query-Support Alignment Approach for Small Object Detection in the Few-Shot Regime", Accepted at the IEEE International Conference on Image Processing 2023 (ICIP).

C 6.1 Framework Presentation and Motivation . . . . . . . . . . . . . . . . . . . . 6.2.4 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Table 6 .

 6 .2. Even though same-size gets the best overall results on novel classes (regardless of object sizes), there is no clear conclusion. It is outperformed by both re ection and mixed for base classes. No method outperforms the others on all object sizes, not even the ones designed to be more robust to size (multiscale and mixed). The latter two techniques introduce discrepancies in the features: objects of similar size (e.g., from both sides of the size limit) are processed di erently, resulting in really di erent features. As same-size gives the best results 2: Comparison of support extraction strategies on base and novel classes with DOTA dataset and FRW method with 10 shots. mAP 0.5 is reported on all objects and separately on objects of di erent sizes: small (S), medium (M) and large (L).

				6.2 A	A		F	F
	Cropping Strategy		Base classes			Novel classes
		All	S	M	L	All	S	M	L
	Default	49.80 24.26 57.36 63.90	24.97	7.66	24.31 34.48
	Context padding	50.03 24.96 59.72 63.03	29.60 10.71 26.76 50.91
	Same-size	50.63 30.83 59.44 62.47	32.19 8.32	33.21 56.54
	Multiscale	51.44 29.03 59.78 63.27	26.95	8.44 33.48 45.63
	Re ection	50.28 26.05 59.36 62.49	25.50	7.22	20.66 44.29
	Mixed	50.95 27.16 60.48 60.67	27.96	9.51	26.52 48.89

on novel classes, we choose to use this strategy for all our experiments. Yet, in the light of our performance analysis in Chap. 4, we can understand some results from Tab. 6.2. The multiscale

Table 6 .

 6 3: Performance comparison between ve selected methods on Pascal VOC. All are reimplemented with the proposed AAF framework. Mean average precision is reported for each method on base and novel classes separately and for various numbers of shots (K ∈ {1, 3, 5, 10}).the support examples. To ensure a fair comparison between the various methods, the same random seed is used for all evaluations, thus the support and query examples are the same.

	The performance is averaged over multiple episodes, each containing 500 examples for each class
	and this operation is repeated multiple times with randomly sampled support sets. The query and
	on

support examples are drawn from the test set, thus the support examples are di erent from the ones used during ne-tuning. This prevents overestimations of the performance due to over tting 6.2.5.2 Natural images Tab. 6.3 gather the results on Pascal VOC. First, compared to the FCOS baseline, a slight performance drop on base classes is observed. This is expected, even if the model has seen a lot of examples of these classes during training, its predictions are still conditioned on a few examples, which can sometimes be misleading. On the other hand, performance on novel classes is signi cantly lower than the FCOS baseline, especially for low numbers of shots. The number of shots is crucial for performance on novel classes. The higher the number of shots, the better the network performs. On average, with 10 examples per class, the network achieves 0.2 higher mAP 0.5 than with 1 example.

More examples provide more precise and robust class representations, improving the detection. The same phenomenon is observed with base classes to a lesser extent (+0.04 mAP from 1 to 10 shots).

Table 6 .

 6 5: Comparison of mAP 0.5 of several methods on DOTA and DIOR datasets. For each method, mAP is reported for di erent number of shots K ∈ {1, 3, 5, 10} and separately for base and novel classes. Blue and red values represent the best performance on base and novel classes respectively, for each dataset. † denotes results taken directly from the original papers.

					6.2 A	A	F	F
		DOTA				DIOR
	FRW	WSAAN	DANA	PFRCN	FRW	WSAAN	DANA	PFRCN	WSAAN †
	K Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel	Base Novel
	1 47.24 13.35	45.55 12.19	49.38 12.52	34.98 7.51	56.67 16.92	56.41 15.48	58.78 20.64	40.66 6.07	-	-
	3 46.50 25.32	44.18 24.42	49.67 20.70	34.58 9.33	58.05 25.08	51.72 13.84	59.14 27.26	40.48 7.51	-	-
	5 48.60 29.57	47.56 31.44	52.49 24.96	36.09 11.33	60.75 32.58	60.79 30.38	62.12 34.16	41.97 8.55	-	0.25
	10 49.04 35.29	46.72 35.12	53.99 36.50	36.32 11.55	61.30 37.29	62.79 32.38	62.71 38.18	42.37 9.16	0.54	0.32

Table 6 .

 6 6: mAP 0.5 and RmAP values for some reimplemented methods and XQSA with K = 10 shots.

			DOTA	DIOR	Pascal VOC
			Base Novel	Base Novel	Base Novel
		FCOS baseline 60.87	69.69	72.82	81.48	65.47 68.02
	0.5 mAP	FRW WSAAN DANA	49.04 46.72 53.99 36.50 35.29 35.12	61.30 62.79 32.38 37.29 62.71 38.18	63.16 48.71 65.27 51.70 65.17 52.26
	RmAP (%)	FRW WSAAN DANA	-19.43 -49.36 -23.24 -49.60 -11.30 -47.63	-15.83 -54.23 -13.78 -60.26 -13.88 -53.14	-3.52 -28.39 -0.30 -24.00 -0.46 -23.17

Table 6 .

 6 7: Ablation study of the XQSA attention method on DOTA dataset. mAP 0.5 scores are reported for base and novel classes with K = 10 shots.

	Baseline
	Cross-scale Alignment
	Fusion Layer
	Query-Support Self-Attention
	Base classes	49.20 49.46 49.13 51.11
	Novel classes	36.52 38.84 40.31 41.01

6.7

. From this table, it is clear that each component plays a role in the improved performance of our method. Both the fusion (with the MLP) and the skip connections around fusion and alignment are bene cial for the performance on novel classes. It is worth noting that Background Attenuation proposed by

[START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF]

helps both for base and novel classes, which con rms the experiments conducted by the authors of this work.

Table 6 .

 6 8: Performance comparison between XQSA, FRW, and DANA. mAP 0.5 values are reported separately for base (top) and novel (bottom) classes on DOTA, DIOR, Pascal VOC, and MS COCO with K = 10 shots. mAP values are reported for All, Small (

	√	wh < 32), Medium (32 ≤	√	wh < 96)

  2.5: FRW and DANA on DOTA, DIOR, Pascal VOC and MS COCO. The results of these experiments are available in Tab. 6.8. The mAP values are reported separately for small (

	medium (32 ≤	√	wh < 96) and large ( √	wh ≥ 96) objects.	√	wh < 32),

Table 7 .

 7 FSDi Det 69.[START_REF] Parmar | Image transformer[END_REF] 52.05 81.71 54.32 74.63 52.64 51.91 24.99 1: FSDi usionDet baseline compared with other FSOD methods. mAP is reported with a 0.5 IoU threshold and all methods leverage 10 shots.

	7.2 provides the novel classes

Table 7 .

 7 2: In uence of the number of shots on the few-shot object detection performance of FSDif-fusionDet on DOTA, DIOR, Pascal VOC and MS COCO. Performance is reported with mAP 0.5 .

Table 7 .

 7 3: Study of the in uence of the backbone pre-training. mAP 0.5 is provided only for base classes, therefore the blue and red colors to distinguish between base and novel classes are no longer required.

	VOC MS COCO

Table 7 .

 7 4: In uence of the amount of plasticity on the FS performance on DOTA, DIOR, Pascal VOC and MS COCO. mAP is reported with a 0.5 IoU threshold.

	It is therefore crucial to understand what

Table 7 .

 7 5: Analysis of FSDi usionDet performance (mAP 0.5 ) against the number of proposals on DOTA, DIOR, Pascal VOC and MS COCO datasets.

	The critical

Table 7 .

 7 6: Learning rate sweeping on DOTA dataset with K = 10 shots. Two distinct schedulers are considered: constant and cosine annealing. Performance is reported with mAP 0.5 .

	7.3 F	S	D	D

Table 7 .

 7 9: Cross-domain performance results on 6 scenarios COCO → DIOR / DOTA / DeepFruits / SIXRay / ClipArt / VisDrone. Results are given for di erent numbers of shots. Experiments are repeated 5 times for each scenario and shot setting. The average mAP 0.5 is reported with a 95% con dence interval. Figure 7.12: Cross-domain performance of FSDi usionDet on multiple scenarios with MS COCO as the source domain. Light areas denote the 95% con dence interval.

	7.4 A	C	D	FSOD

Table 7 .

 7 10: FSDi usionDet Cross-domain results on the scenario DIOR → DOTA. Two settings are compared: with the backbone frozen (left) and the backbone fully ne-tuned (right). Bold values denote the best setting for overall performance on novel classes. Performance is reported with mAP 0.5 values.Besides experimenting with COCO → X scenario, we propose another setting speci cally designed for aerial images and COSE's application. The idea is to leverage two distinct aerial datasets as source and target domains. In particular, we use DOTA and DIOR to get two scenarios: DOTA → DIOR and DIOR → DOTA. Of course, it would be interesting to leverage other kinds of datasets as well (e.g., xView, VisDrone, etc.), especially as DOTA and DIOR are very similar (mostly overhead urban images and shared classes). Yet, this gives insights into how FSDi usionDet behaves in fairly simple cross-domain scenarios. We report experiments with these two scenarios in Tabs. 7.10 and 7.11. TheThe key takeaway from this experiment is that higher performance is achieved in the aerial crossdomain scenarios than with the COCO → X scenario. It seems more pro table to perform base training on a source domain that is similar to the target domain. Of course, compared to the FS performance on DOTA and DIOR, lower quality is achieved in cross-domain scenarios. This is explained rst because the images from the two datasets di er, but also because the task is now slightly more complex as all classes of the target dataset are novel. The detection task becomes a 16-way K-shots problem in DIOR → DOTA scenario for instance. In the regular FS setting studied throughout this project, only three classes were selected as novel classes for DOTA, making the classi cation much easier. Then, from Tabs. 7.10 and 7.11, a contradiction arises, in DIOR → DOTA scenario, the fully ne-tuned model outperforms the model with the backbone frozen, which agrees with the experiments from Sec. 7.3.2.4. However, in the DOTA → DIOR scenario, the inverse is observed. This clearly shows that the freezing sweet spot depends on the source and target domains and cannot be set once and for all. It works in the case of regular few-shot when the source and target domains are identical. So, one could expect the same behavior in cross-domain scenarios with similar source and target domains as it is with DOTA and DIOR. However, in the regular few-shot setting, the number

	7.4.2 Aerial Cross-Domain
	results are given for multiple numbers of shots ranging from 1 to 50. In addition, we studied two
	freezing strategies by ne-tuning only the detection head (i.e., frozen backbone) or the whole model
	(fully ne-tuned).

Table 7 .

 7 11: FSDi usionDet Cross-domain results on the scenario DOTA → DIOR. Two settings are compared: with the backbone frozen (left) and the backbone fully ne-tuned (right). Bold values denote the best setting for overall performance on novel classes. Performance is reported with mAP 0.5 values.of classes in the source domain (i.e., the base classes) is always larger than the number of classes in the target domain (i.e., the novel classes). Here, DIOR has more classes than DOTA and this difference may explain the opposite results between the two CD scenarios. Speci cally, ne-tuning the model entirely may be bene cial only when the target domain contains fewer classes than the source domain. It could also be caused by di erent class separations between the datasets. If classes are easily di erentiable in DIOR but not in DOTA, it might be di cult to transfer from DIOR to DOTA. These are only conjectures, and they should be taken carefully especially as complex interactions between source and target classes may also cause such behavior. More experiments would be required to analyze and understand this surprising result.

  Intersection over Union (IoU) is not an optimal box similarity measure for evaluating and training object detectors. For evaluation, it is too strict with small objects and does not align well with human perception. For training, it provides a poor balance between small and large objects to the detriment of small ones. We propose Scale-adaptative Intersection over Union (SIoU), a parametric alternative that solves the shortcomings of IoU. We provide empirical and theoretical arguments for the superiority of SIoU through in-depth analysis of various criteria.Ƽ P. LeJeune and A. Mokraoui, "Rethinking Intersection Over Union for Small Object Detection in Few-Shot Regime", Submitted at the International Conference on Computer Vision 2023 (ICCV). P. Le Jeune and A. Mokraoui, "Extension de l'Intersection over Union pour améliorer la détection d'objets de petite taille en régime d'apprentissage few-shot", GRETSI 2023, XXIXème Colloque Francophone de Traitement du Signal et des Images, Grenoble, France. Analysis of Intersection over Union . . . . . . . . . . . . . . . . . . . . . . . . 170 8.1.1 Intersection over Union and its Variants . . . . . . . . . . . . . . . . . . . 170 8.1.2 Inadequation of IoU for Small Objects and Few-Shot Regime . . . . . . . . 171 8.2 Scale-Adaptive Intersection over Union . . . . . . . . . . . . . . . . . . . . . . 173 8.2.1 De nition of the novel box similarity criterion . . . . . . . . . . . . . . . 173 8.2.2 SIoU Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175 8.2.3 Extensions and Generalization of SIoU. . . . . . . . . . . . . . . . . . . . 176 8.3 Scale-Adaptive Criteria Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . 176 8.3.1 Response Analysis to Box Shifting . . . . . . . . . . . . . . . . . . . . . . 176 8.3.2 Resilience Analysis to Detector Inaccuracy . . . . . . . . . . . . . . . . . 177 8.3.3 Theoretical study of GIoU . . . . . . . . . . . . . . . . . . . . . . . . . . 182 8.3.4 In uence of γ and κ on SIoU and GSIoU . . . . . . . . . . . . . . . . . . . 182 8.4 SIoU Alignment with Human Perception . . . . . . . . . . . . . . . . . . . . . 185 8.4.1 User Study Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . 185 A I O U 8.4.2 User Study Insights . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 8.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190 8.5.1 Comparison with Existing Criteria . . . . . . . . . . . . . . . . . . . . . . 192 8.5.2 Application on Aerial and Natural Images . . . . . . . . . . . . . . . . . . 192 8.5.3 γ and κ in uence on FSOD Performance . . . . . . . . . . . . . . . . . . 194 8.5.4 Additional Experiments with GSIoU Loss . . . . . . . . . . . . . . . . . . 195 8.5.5 Evaluation with SIoU . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196 8.5.6 Discussions and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . 196 8.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198 Intersection over Union (IoU) is a box similarity criterion, it measures how well two bounding boxes overlap each other. We already de ned it in Chap. 2, but in this chapter we explore its properties

	C	8
	S	A	I	O	U
	A
	P	III
	R	I	O	U
	C
	8.1

  Before introducing the proposed criterion, let us de ne two bounding boxes b1 = [x 1 , y 1 , w 1 , h 1 ] T and b 2 = [x 2 , y 2 , w 2 , h 2 ] T (the prediction box and ground truth respectively), following the box de nition from Chap. 2. Similarly, the adjectives small, medium, and large keep the same meaning as in previous chapters: the box b i is small if √ w i h i ≤ 32 pixels, medium if 32 < √ w i h i ≤ 96, and

	8.2 Scale-Adaptive Intersection over Union
	8.2.1 De nition of the novel box similarity criterion

  8.3.2 Resilience Analysis to Detector InaccuracyKnowing how a criterion responds to shifts and size variations is important to understand what makes a sensible box similarity measure. Pushing beyond the shift analysis, we study empirically and theoretically the criteria's distributions when exposed to detector inaccuracies, i.e., randomly shifted boxes. This setting mimics the inaccuracy of the model either during training or at test time.

	8.3.2.1 Empirical Protocol
	To simplify, let us suppose that all boxes are squares of the same size ω and can be shifted only
	horizontally. Similar results are observed by relaxing these constraints, see Sec. 8.3.2.4. A box is

  boxes drawn on it. A green one, which represents the ground truth annotation of an object, and a red one randomly shifted and deformed. Each participant must rate how well the red box is detecting the object inside the green box. The rating is done on a 5-levels scale, going from very poor to very good. A set of 50 di erent images is shown to each participant. After 25 images, the experiment changes slightly: the background image is replaced by a completely black image. This would remove any contextual bias coming from the variety of objects inside the green box. We refer to the two phases of the experiment as respectively, the phases with and without context. The red boxes are sampled around the green box, but to enforce a uniform distribution of the IoU with the green box, a random IoU value u is rst uniformly sampled between 0 and 1. Then, we randomly generate a red box that has an IoU u with the green box (direct box sampling does not produce uniformly distributed IoU values). Participants are instructed to answer quickly and are provided with examples for each possible rating (see Fig.8.8). The images and the annotations are randomly picked from the DOTA dataset.8.4.1.2 General Statistics about ParticipantsThe study gathered 75 participants and a total of 3136 individual answers (because some participants did not complete the entire experiment). The age of the participants ranges from 21 to 64 years old with an average of 31. Approximately half[START_REF] Bochkovskiy | Yolov4: Optimal speed and accuracy of object detection[END_REF] of the participants are versed in computer vision or

	A	I	O	U
	with two bounding
	8.4.1 User Study Presentation
	8.4.1.1 Experimental Protocol
	To carry out the user study about detection preferences, we developed a Web App 1 to gather partic-
	ipants' answers. Each participant had to sign in with a brief form. They are asked about their age
	and whether they are familiar with image analysis. This information is only meant to detect rating
	di erences between di erent population groups (see Fig. 8.11, last two rows). After completing the
	form, participants are brought to the rating page (see Fig. 8.8). On this page, one image is visible
	1 The web app is available here.

Table 8 .

 8 2: Kendall's τ correlation between various criteria and human rating r. For SIoU, γ = 0.2 and κ = 64, for α-IoU, α = 3.

	A	I	O	U
		r	IoU SIoU α-IoU NWD
	r	1.000 0.674 0.701 0.674	0.550
	IoU	0.674 1.000 0.892	0.997	0.474
	SIoU 0.701 0.892 1.000	0.892	0.576
	α-IoU 0.674 0.997 0.892	1.000	0.472
	NWD 0.550 0.474 0.576	0.472	1.000

5, 0.75, and 0.9 for the ratings from very poor to very good respectively.

Table 8 .

 8 3: Average rating and criteria values for di erent groupings of the variables of interest (object size, presence of contextual information, expertise and age of the participants). limited compared to object sizes. This con rms values inside Tab. 8.3 as the older age groups tend

	To

  Detection models are often heavy and are not well suited for COSE's application. In this chapter, we rst present in detail the CAMELEON system and its constraints. Then, we study the in uence of the model size on the performance and present useful tools and tricks to accelerate the inference. Finally, we explain how the detection models are deployed inside the CAMELEON prototype and how they perform on aerial images. Object Detection Pipeline for CAMELEON Prototype . . . . . . . . . . . . . . 212 9.4.1 Deployment with TensorRT . . . . . . . . . . . . . . . . . . . . . . . . . 212 9.4.2 Detection Performance Evaluation and Pro ling . . . . . . . . . . . . . . 215 9.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218 Up to this point, the contributions of this project have been mostly research-oriented and a signi cant amount of work is still required to apply the developed techniques on real-case scenarios.

	C	9
	I	COSE P
	A
	P	I	A
			199

C 9.1 CAMELEON Aerial Intelligence System . . . . . . . . . . . . . . . . . . . . . . 202 9.1.1 Presentation of the system . . . . . . . . . . . . . . . . . . . . . . . . . . 202 9.1.2 CAMELEON Image Speci cations and Constraints . . . . . . . . . . . . . 204 9.2 Reducing Object Detection Model Size for Edge Computing . . . . . . . . . . . 205 9.2.1 Object Detection Accuracy/Speed Tradeo . . . . . . . . . . . . . . . . . 205 9.2.2 Knowledge Distillation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 208 9.3 Inference Acceleration with TensorRT . . . . . . . . . . . . . . . . . . . . . . 209 9.4

Table 9 .

 9 1: Performance (mAP 0.5 ) on DOTA and DIOR for various YOLOv5 sizes. Numbers of parameters and latency are provided along with the performance measures. Latency is computed with 512× 512 images on a RTX 3090.

		9.2 R	O	D	M	S	E	C
	Model	# params Latency (ms) mAP DOTA mAP DIOR
	YOLOv5-F	3.19e+5	5.2		42.8	41.4
	YOLOv5-P	6.75e+5	5.4		49.1	63.2
	YOLOv5-N	1.78e+6	6.0		68.5	80.3
	YOLOv5-S	7.05e+6	6.6		72.7	85.2
	YOLOv5-M	2.09e+7	8.7		74.6	88.1
	YOLOv5-L	4.62e+7	11.1		75.4	89.1

Table 9 .

 9 KD 49.1 30.8 53.8 48.9 63.2 21.3 50.0 76.6 w/ KD 46.4 27.9 51.7 45.9 48.5 17.5 43.0 56.4 2: Detection performance (mAP 0.5 ) comparison with and without Knowledge Distillation (KD). Performance is reported for di erent objects sizes and on two datasets, DOTA and DIOR. Both YOLOv5-P and YOLOv5-N are compared.

			9.3 I	A			T	RT
		DOTA			DIOR
	All	S	M	L	All	S	M	L
	YOLOv5-P w/o YOLOv5-N w/o KD 67.9 51.0 70.9 69.8 w/ KD 47.5 30.3 52.4 45.5	80.6 35.9 69.2 90.8 55.9 22.0 51.4 65.3

Table 9 .

 9 3: Pro ling for the inference of YOLOv5-N engine with integer precision on one CAMELEON image. Timings are measured on a Nvidia AGX Xavier.

Table 9 .

 9 Speci cally, we compare the inference speed of the Pytorch model (with F32 and F16 precisions) with the TensorRT engines (F32, F16 and INT8 precisions). The inference times are measured both on an RTX 3090 and an Nvidia AGX Xavier, with distinct image widths: 512 and 4096 pixels. The results of this comparison RTX 3090 Latency AGX Xavier Latency 4: Inference speed comparison (in ms) between various deployment strategies and precisions for YOLOv5-N. Inference times are measured with two image widths and on two GPUs. A high-end desktop GPU: RTX 3090, and an embedded GPU: AGX Xavier. mAP with a 0.5 IoU threshold is also reported on DOTA.

	Image Size (pixels) 512	4096	512	4096	mAP 0.5
	Pytorch F32	6	36.3	87.5	2638.6	0.679
	Pytorch F16	8.3	20.4	90.7	2616.5	0.679
	TensorRT F32	0.75	17.16	4.65	199.22	0.565
	TensorRT F16	0.44	7.22	4.64	106.2	0.565
	TensorRT INT8	0.43	5.77	2.48	68.65	0.523

Table 10 .

 10 1: Comparison table between the three proposed approaches in this thesis: Prototypical Faster R-CNN, XQSA and FSDi usionDet. The ratings are given according to the experiments conducted throughout this project and the insights generated. Green and red colors denote pros and cons, respectively, best viewed in colors. * Episodic training is more complex and often subject to instabilities. Technical complexity of the methods is a subjective criterion that measures how intricate a method is. † TensorRT optimization is not compatible with custom layers or complex data streams which greatly complexi es the optimization of more elaborated methods such as attentionbased approaches. but we have not looked into the training examples selection strategy even though it may have a signi cant in uence on the small and large objects' balance during training. Incidentally, it would be relevant to replace IoU inside the example selection process with SIoU. This would favor smaller objects and certainly improves overall performance on aerial images. At least, it would o er more control over the training balance between small and large objects. SIoU is a controllable criterion,

	9.5 C

https://www.cose.fr/

https://github.com/ultralytics/yolov5

https://github.com/ultralytics/ultralytics

a typical setup in FSC is to repeat 100 times the adaptation, even if an FSOD model takes only 30 minutes to adapt to the novel classes through ne-tuning, the robust evaluation would take almost

days.

https://github.com/pierlj/aaf_framework

p stands here for the p-value of the statistical test here, not the exponent from SIoU's de nition.

https://github.com/ultralytics/yolov5

Link to YOLOv8 repository

 [START_REF] He | Alpha-IoU: A Family of Power Intersection over Union Losses for Bounding Box Regression[END_REF]). The inaccuracy of the detector is set to σ = 16. Note that the empirical pdfs were smoothed using a Kernel Density Estimator method. This a ects particularly IoU, SIoU , and α-IoU for which the actual pdf is de ned only on [0, 1]. For the sake of visualization, GIoU and GSIoU were rescaled between 0 and 1 for the expectation and standard deviation plots.

X ∼ N (0, σ 2 ) where σ controls how inaccurate the model is. We are interested in the distribution of C ∈ {IoU, GIoU, SIoU, GSIoU, α-IoU, NWD} and how it changes with ω. To this end, let Z = C(X). More precisely, we are interested in the Probability Density Function (PDF) of Z and its two rst moments (which exist because C is continuous and bounded). When building detection models, we hope to detect equally well objects of all sizes, this means having a constant expected IoU, no matter the objects' size. This would require the localization error to be an a ne function of ω. Of course, the localization error of the detector is likely to depend on ω. However, it cannot be an a ne function, otherwise, small objects would be perfectly detected, which is not observed (see Fig.

Comparison with Existing Criteria

To begin, we compare the few-shot performance on DOTA with various loss functions based on the criteria discussed in Sec. 8.3. The result of these experiments is available in Tab. 8.5. The criteria are divided into two groups, generalized (i.e., which is not 0 when boxes do not overlap; it therefore includes NWD) and vanilla criteria. As discussed in Sec. 8.3.2.1, the generalized versions of the criteria outperform their original counterparts and therefore should be compared separately. Scaleadaptive criteria (SIoU and GSIoU) largely outperform other losses on novel classes and especially on small objects. For SIoU and GSIoU, we choose γ = -3 and κ = 16 according to a series of experiments conducted on DOTA to determine their optimal values (see Sec. 8.3.4). It is important to point out the relatively good performance of NWD despite not checking all the desirable properties highlighted in Sec. 8.3.

Application on Aerial and Natural Images

As the previous set of experiments was only carried out on DOTA, we showcase the versatility of GSIoU on three other datasets: DIOR, Pascal VOC and COCO. As it is clear that generalized criteria achieve higher performance, the comparison here is only done between GIoU and GSIoU. The results of this comparison are available in Tab. 8.6.

The large improvements found for DOTA translate to DIOR as well, especially for small objects.

For Pascal VOC and COCO, similar gains are observed for small objects, but the improvement is This shows that the optimal values for DOTA dataset are γ = -3 and κ = 8. However, κ = 16 is also a good choice and is more consistent across datasets. Thus, we choose to keep γ = -3 and κ = 16

for other experiments. An exhaustive grid search should be done to nd even better settings. Our search was sparse and a better combination of γ and κ probably exists, yet our sub-optimal setup already yields signi cant improvement for small object detection in the FS regime.

Base classes

In uence of γ and κ on FSOD performance on Pascal VOC dataset The search conducted above was carried out on DOTA dataset. It transposes nicely on DIOR dataset as well. Yet, these two datasets are similar. They both contain aerial images with some classes in common, but most importantly, the size of their objects are highly similar. When applied to di erent datasets, these results may not hold. For instance, Pascal VOC requires other combinations of γ and κ to outperform the training with GIoU. This is shown in Tab. 8.9.

Additional Experiments with GSIoU Loss 8.5.4.1 Changing the Few-Shot Approach

To support the versatility of GSIoU, we also experiment with several few-shot approaches. We selected three FSOD techniques that we implemented within the AAF framework: Feature Reweighting [START_REF] Kang | Few-shot object detection via feature reweighting[END_REF] (FRW), Dual-Awareness Atention [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF] (DANA) and our Cross-scale Query-Support Alignment (XQSA). We train all of them with GIoU and GSIoU as regression losses and provide the results in Tab. 8.10. Except for DANA, GSIoU sensibly improves detection performance, especially for small objects. The results with DANA are surprising, and it would be of great interest to investigate the reasons behind this below-par performance.

Base classes

Novel (FRW), Dual Awareness Attention [START_REF] Chen | Dual-Awareness Attention for Few-Shot Object Detection[END_REF] (DANA) and our Cross-scale Query-Support Alignment (XQSA), trained with GIoU and GSIoU. mAP is reported with a 0.5 IoU threshold for small (S), medium (M), large (L) and all objects.

Regular Object Detection on DOTA and DIOR

GSIoU is not only bene cial for FSOD, but it also improves the performance of regular object detection methods. Tab. 8.11 compares the performance of FCOS [START_REF] Tian | Fcos: Fully convolutional one-stage object detection[END_REF] trained on DOTA and DIOR with GIoU and GSIoU. The same pattern is visible as we get better performance with GSIoU. However, the gain for small objects is not as large as for FSOD. Nevertheless, it suggests that other tasks relying on IoU could also bene t from GSIoU.

Obviously GSIoU (γ = -3 and κ = 16) losses. mAP is computed with several IoU thresholds (0.5 to 0.95) as it is commonly done in regular detection.

One possibility for this is the use of IoU in the example selection process. In most detection frameworks, only the best predicted bounding boxes, according to the IoU, are selected to compute the loss (this is detailed in Sec. 2.1.3.5 and especially Tab. 2.2). As mentioned previously, it would be relevant to study the impact of SIoU on this process as well. However, FCOS, on which the AAF framework is based, does not rely on IoU for the example selection. Instead, it selects as positive examples all predicted boxes whose center falls into a ground truth box. Hence, an IoU-based example selector probably hinders the bene ts of SIoU while the FCOS's selection is likely a better match.

This should be investigated in depth in future work.

Evaluation with SIoU

In this section, we present some of the results reported in previous section using SIoU as the evaluation criterion. Speci cally, instead of choosing an IoU threshold to decide if a box is a positive or negative detection, an SIoU threshold is employed. For the sake of comparison, we kept the same thresholds as in Tabs. 8.5, 8.6 and 8.11, i.e., 0.5 for Few-Shot methods and 0.5:0.95 for regular object detection. The results are available in Tabs. 8.12 to 8.14. The conclusions from Sec. 8.5 still hold, and the superiority of GSIoU over other criteria is clear. However, a few changes are noticeable. First, SIoU loss seems to perform better than IoU. This is expected since the model is directly optimized to satisfy this criterion. Then, when evaluated with SIoU, models trained with NWD perform well.

Indeed, NWD puts a lot of emphasis on size matching during training, and less on position. Therefore, it is logical to observe better performance compared to other losses when using SIoU as the evaluation criterion.

One crucial point is that SIoU evaluation mostly changes the score for small objects. SIoU behaves like IoU for large objects, therefore relatively small changes are visible for medium and large objects.

Overall, the scores are higher than with IoU as the expected value of SIoU is higher than IoU. The important point to note is that the gap between small and large objects performance is reduced and aligns better with human perception.

Discussions and Limitations

As mentioned in Sec. A A

P SI U' P

In this appendix, we provide the proofs for Properties 1 and 2, and discuss the order-preservigness of SIoU.

Property 1 (SIoU Relaxation) Let b 1 and b 2 be two bounding boxes and introduce τ = w 1 h 1 +w 2 h 2 2 their average area. SIoU preserves the behavior of IoU in certain cases such as:

-

κ . τ > 0 because boxes cannot be empty and as γ ∈] -∞, 1] and κ ∈ R * + , we have p > 0.

From this, the rst two items of Property 1 follow clearly. ). However, this property is often true in practice. Denoting by τ i,j the average area between boxes i and j, we can study the conditions for the order to hold. We will also note p i,j = 1 -γ exp -

Let's suppose, without loss of generality, that τ 1,2 ≤ τ 1,3 . Otherwise, cases 1 and 2 would be swapped.

Case 1: γ ≤ 0 We have p 1,2 > p 1,3 as τ 1,2 ≤ τ 1,3 and γ ≤ 0. Therefore, p 1,2 = p 1,3 + ε, with ε > 0.