Lucas Jordan Frecon

Yuan Anquetil

Gilles Liu

Stéphane Gasso

The different contributions have been proposed in the following papers

Machine learning is revolutionizing the world in many ways, enabling the creation of artificial systems capable of performing complex tasks. In medicine, machine learning systems are used to analyze medical data and identify patterns helping doctors diagnose diseases more accurately. In retail, machine learning systems are widely used to personalize customers' shopping experience by recommending products based on their browsing history and behavior. In the transportation sector, machine learning is at the heart of the development of autonomous cars, vehicles that would be capable of driving without human intervention, for which a robustness assessment is of paramount importance.

The problem addressed in this thesis is the robustness evaluation of deep neural networks as image classifiers using adversarial examples, which are original examples to which a perturbation, imperceptible to the human eye, is added changing the classifiers' output. More precisely, we propose a method to compute these adversarial perturbations using a dictionary, a set of images learned from a classifier, and a dataset. We also question the quality of the metrics used to compute distances between two images, leading to reliable estimation of a relevant robustness measure of image classifiers.

In a first contribution, we propose a new way to fool classifiers. While competitive with stateof-the-art methods, this approach learns a dictionary of images for which a certain linear combination produces such an adversarial perturbation. This dictionary makes it possible to construct universal adversarial perturbations, independent of the original examples, and transferable, effective for fooling other classifiers. The learned dictionary also makes it possible to visualize the elementary shapes underlying the creation of adversarial perturbations.

In a second contribution, we propose to learn a similarity measure between images by optimizing the cost matrix of the Wasserstein distance between their representations derived from a neural network. Using image representations enables to compute more relevant and realistic similarities than using their pixels. Pixel-based image distances don't take semantics into account. Our proposal makes it possible to explore the richness of learned representations, to construct a relevant Wasserstein distance. Having such a relevant similarity measure between images may reveal useful when working on life-at-risk computer vision applications.

Résumé

L'apprentissage automatique révolutionne le monde de nombreuses manières en permettant la création de systèmes intelligents capables d'effectuer des tâches complexes. En médecine, les systèmes d'apprentissage automatique sont utilisés pour analyser des données médicales et identifier des motifs qui peuvent aider les médecins à diagnostiquer les maladies de manière plus précise. Dans le domaine du commerce, les systèmes d'apprentissage automatique sont largement utilisés pour personnaliser l'expérience d'achat des clients en recommandant des produits en fonction de leur historique et de leur comportement de navigation. Dans le secteur du transport, l'apprentissage automatique est au coeur du développement des voitures autonomes, des véhicules qui seraient en capacité de circuler sans intervention humaine pour lesquels une évaluation de la robustesse est primordiale.

Le problème abordé dans cette thèse est celui de l'évaluation de la robustesse des réseaux de neurones profonds en tant que classifieur d'images à l'aide d'exemples adversaires, c'est-à-dire des exemples originaux auxquels est ajoutée une perturbation, imperceptible à l'oeil nu, changeant la sortie des classifieurs. Plus précisément, nous proposons une méthode de calcul de ces perturbations adversaires à l'aide d'un dictionnaire, un ensemble d'images apprises à partir d'un classifieur et d'un jeu de données. Nous remettons également en question la qualité des métriques utilisées pour calculer les distances entre deux images, afin d'estimer de manière fiable une mesure de robustesse pertinente des classifieurs d'images.

Dans une première contribution, nous proposons une nouvelle manière de tromper les classifieurs. Tout en étant compétitive avec les méthodes de l'état de l'art, cette approche permet d'apprendre un dictionnaire d'images dont une certaine combinaison linéaire produit une telle perturbation adversaire. Ce dictionnaire permet de construire des perturbations adversaires universelles, indépendantes des exemples originaux, et transférables, efficaces pour tromper d'autres classifieurs. Le dictionnaire appris, permet aussi de visualiser les formes élémentaires à la base des perturbations adversaires.

Dans une deuxième contribution, nous proposons d'apprendre une mesure de similarité entre images en optimisant la matrice de coût de la distance de Wasserstein entre leurs représentations issues d'un réseau de neurones. Utiliser les représentations des images permet de calculer des similarités plus pertinentes et réalistes qu'en utilisant leurs pixels. En effet, les distances d'images basées sur les pixels ne considèrent pas la sémantique. Notre proposition permet d'explorer la richesse des représentations apprises, afin de construire une distance de Wasserstein pertinente. Disposer d'une telle mesure de similarité pertinente entre images peut s'avérer utile lorsque l'on travaille sur des applications de vision par ordinateur pour lesquelles des vies sont en jeu.

iii

Introduction viii

Introduction xiii

Introduction Context and motivation

In recent years, the field of artificial intelligence (AI) has experienced unprecedented growth. Problems once considered out of reach, are now easily solved thanks to deep neural networks1 [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF].

In particular, deep networks played a central role in the recent revolution in computer vision. The massive adoption of deep neural networks dates back to 2012, when the Alexnet network [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] largely won the ImageNet image classification challenge, setting a new performance standard in the field.

Since then, numerous neural network based innovations emerged, solving image and video related tasks. This led to the mass adoption of deep neural networks for everyday tasks such as virtual assistants [START_REF] Tulshan | Survey on virtual assistant: Google assistant, siri, cortana, alexa[END_REF], facial recognition payment systems, driving assistants2 , and it is highly likely that in the coming years we will see the advent of fully autonomous driving systems [START_REF] Janai | Computer vision for autonomous vehicles: Problems, datasets and state of the art[END_REF].

Autonomous vehicles

Research into autonomous cars has been going on for several decades, with the first experiments and prototypes developed in the 1920s. The first attempt at driverless vehicles dates back to 1925 3 , as illustrated with the Chandler car shown in Figure 1a, and gained momentum in the 1980s when researchers succeeded in developing automated highway systems [START_REF] Fenton | Automated highway studies at the ohio state university-an overview[END_REF][START_REF] Ioannou | Automated highway systems[END_REF]. This led to semiautonomous and autonomous vehicles directly connected to the road infrastructure. Improvements in autonomous vehicles were largely achieved in Germany and the USA between 1980 and 2000 [3,[START_REF] Lantos | Nonlinear control of vehicles and robots[END_REF].

Autonomous vehicles owe much to defense industry research into unmanned equipment, led by the famous US Defense Advanced Research Projects Agency (DARPA) challenge [START_REF] Seetharaman | Unmanned vehicles come of age: The darpa grand challenge[END_REF]. The DARPA challenge is a turning point for autonomous vehicles for two reasons. Firstly, it was the first time a car was able to drive autonomously for more than 200 kilometers at an average speed of over 30 km/h, demonstrating that it is indeed possible for cars to drive on their own, without human supervision. Secondly, from a technical point of view, the DARPA challenge is a revolution because Stanford's winning car, shown Figure 2, actually won thanks to the use of machine learning based tools. It was proof that the best way to program an automatic driving system is through machine learning.

Then, private companies became interested in the field and began to carry out in depth research. Google's driverless car drew attention to autonomous vehicles and attracted talent from many disciplines to work on them. Tesla, a competitor of Google's in the autonomous car field, is an electric car company that helped spark interest in autonomous vehicles with its Autopilot product, a semi-autonomous driving system capable of steering, braking and accelerating the car. Two years after the launch of Tesla's Autopilot, Google followed with its Waymo driverless car project, offering a driverless taxi service in Phoenix, Arizona's Chandler neighborhood. Figure 1b shows the inside of such a robo-taxi system. In doing so, it became the first company to offer a commercial service using fully autonomous vehicles. One of the main concerns of autonomous cars is safety. Indeed, autonomous cars incorporate deep neural networks to make driving decisions. However, these models show their limitations in terms of robustness against adversarial examples [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF].

Adversarial examples

With the recent success of deep learning, several authors [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF] highlighted their vulnerability to adversarial attacks. Adversarial attacks are functions of the classifier neural network and an original example producing an adversarial perturbation, i.e. a slight perturbation imperceptible to the human eye, which when added to the original example leads to a misclassification as illustrated in Figure 3. These adversarial perturbations enable neural networks to be attacked by any hacker wishing to corrupt them anonymously. However, thanks to these adversarial perturbations, it is also possible to estimate the vulnerability and robustness of a neural network. In order to estimate the robustness of neural networks, some authors proposed to compute pseudo-random adversarial perturbations which, thanks to efficient optimization, enable the precisely draw the neural network's limits. These perturbations have been criticized for not representing realistic image perturbations, observable in real life, thus leading to an irrelevant estimate of the neural network robustness. In response to these critics, other types of adversarial perturbations have been proposed, such as common corruptions [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF], or adversarial patches [START_REF] Brown | Adversarial patch[END_REF]

Contributions

In this thesis, we proposed two contributions. First, we proposed to model the adversarial noise space using a dictionary of adversarial atoms, creating an adversarial attack bridging the gap between specific and universal adversarial attacks. Through multiple submissions and revisions, our proposition evolved into an efficient modeling of the adversarial noise space, enabling to derive a robust and transferable adversarial attack. Indeed, the derived attack proves to be competitive with specific state-of-the-art attacks, while being more robust, i.e. more resistant to adversarial example detection mechanisms. Besides, the computed adversarial examples are more transferable, they allow fooling other neural networks. Thanks to the universal model of the adversarial noise space, our experiments show that the computed adversarial perturbations are more universal than state-of-the-art methods, i.e. they can fool a neural network on several original examples. The learned dictionary also makes it possible to visualize the elementary shapes underlying the creation of adversarial perturbations. By relying on a dictionary, our proposition allows us to reveal the global weaknesses and dynamics leading to neural network misclassification. In addition, our proposition also questions the currently used adversarial metrics. Generally speaking, our proposition questions the consideration currently given to pseudo-random adversarial perturbations to effectively assess a relevant robustness estimation of deep neural networks.

In the second contribution, we questioned the currently used similarity measures between images. More precisely, we propose learning a similarity measure between images by optimizing the cost matrix of the Wasserstein distance between the images' representation extracted from a neural network. The similarity measures derived from our proposition are more interesting than the distances currently used, as they use image representations rather than pixels, leading to more relevant and realistic image similarities. Indeed, pixel based image distances do not consider semantics. Our proposition allows to explore the wealth of learned representations, in order to build a relevant Wasserstein distance. Especially, we tested our proposition in different application contexts to make it more robust and efficient, according to the hyper-parameters. Having such a relevant similarity measure between images can be useful when working on life-at-risk computer vision applications. However, due to lack of time, we were unable to address all the problems associated with applying our proposition to more complex and large-scale datasets.

Introduction Contexte et motivation

Ces dernières années, le domaine de l'intelligence artificielle (IA) a connu une croissance sans précédent. Des problèmes autrefois considérés comme insurmontables, peuvent désormais être résolus facilement grâce aux réseaux de neurones profonds 4 [START_REF] Silver | Mastering the game of go with deep neural networks and tree search[END_REF].

En particulier, les réseaux de neurones profonds ont joué un rôle central dans la récente révolution de la vision par ordinateur. L'adoption massive de ces réseaux de neurones profond remonte à 2012 lorsque le réseau Alexnet [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF] a largement remporté le défi de classification d'images à partir d'ImageNet, établissant ainsi un nouveau standard de performances dans le domaine.

Depuis, de nombreuses innovations reposant sur les réseaux de neurones ont émergé, résolvant des tâches liées à l'image et à la vidéo. Cela a conduit à l'adoption massive de réseaux de neurones profonds pour des tâches quotidiennes telles que les assistants virtuels [START_REF] Tulshan | Survey on virtual assistant: Google assistant, siri, cortana, alexa[END_REF], les systèmes de paiement par reconnaissance faciale, les assistants de conduite 5 , et il est fort probable que dans les années à venir nous verrons l'avènement de systèmes de conduite complètement autonomes [START_REF] Janai | Computer vision for autonomous vehicles: Problems, datasets and state of the art[END_REF].

Autonomous vehicles

La recherche sur les véhicules autonomes existe depuis plusieurs décennies avec les premières expérimentations et prototypes développés dans les années 1920. Une des première tentative de véhicules sans conducteur remonte à l'année 1925 6 avec la voiture Chandler présentée Figure 4a. Ces travaux ont pris de l'ampleur dans les années 1980 lorsque des chercheurs ont réussi à développer des systèmes autoroutiers automatisés [START_REF] Fenton | Automated highway studies at the ohio state university-an overview[END_REF][START_REF] Ioannou | Automated highway systems[END_REF]. Cela a conduit à des véhicules semiautonomes et autonomes directement connectés à l'infrastructure routière. Les améliorations des autonomous-vehicles xiii CONTENTS véhicules autonomes ont été largement réalisées en Allemagne et aux États-Unis entre 1980 et 2000 [3,[START_REF] Lantos | Nonlinear control of vehicles and robots[END_REF].

Les véhicules autonomes doivent beaucoup à la recherche menée par le secteur de la défense sur les équipements sans pilote, dirigée par le célèbre défi de l'Agence pour les projets de recherche avancée de défense (DARPA) des États-Unis [START_REF] Seetharaman | Unmanned vehicles come of age: The darpa grand challenge[END_REF]. Le défi DARPA est un tournant pour les véhicules autonomes pour deux raisons. Tout d'abord, c'est la première fois qu'une voiture a pu conduire de manière autonome sur plus de 200 kilomètres à une vitesse moyenne supérieure à 30 km/h, ce qui a démontré qu'il est effectivement possible que les voitures circulent seules, sans supervision humaine. Deuxièmement, d'un point de vue technique, le défi DARPA est une révolution car la voiture gagnante de Stanford, présentée Figure 5, a en réalité remporté grâce à l'utilisation d'outils basés sur l'apprentissage automatique. C'était la preuve que la meilleure manière de programmer un système de conduite automatique est l'apprentissage automatique.

Ensuite, des entreprises privées se sont intéressées au domaine et ont commencé à mener des recherches approfondies. La voiture sans conducteur de Google a suscité une attention sur les véhicules autonomes et a attiré de nombreux talents issus de plusieurs disciplines pour y travailler. Tesla, un concurrent de Google dans le domaine de la voiture autonome, est une entreprise de voitures électriques, qui a contribué à susciter l'intérêt pour les véhicules autonomes avec son produit Autopilot, un système de conduite semi-autonome capable de diriger la voiture, de freiner et d'accélérer. Deux ans après le lancement de l'Autopilot de Tesla, Google a suivi avec son projet de voiture sans conducteur Waymo, en proposant un service de taxi sans conducteur à Phoenix, en Arizona, dans le quartier de Chandler. La figure 4b montre l'intérieur d'un tel système de robotaxi. Elle est ainsi devenue la première entreprise à proposer un service commercial utilisant des véhicules entièrement autonomes. L'une des principales préoccupations des voitures autonomes est la sécurité. En effet, les véhicules autonomes intègrent des réseaux de neurones profonds pour prendre des décisions de conduite. Cependant, ces modèles ont montré leurs limites en termes de robustesse face à des exemples adversaires [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF].

Examples adversaires

Avec le récent succès de l'apprentissage profond, plusieurs auteurs [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF] ont souligné leur vulnérabilité aux attaques adversaires. Les attaques adversaires sont des fonctions du réseau de neurones classifieur et d'un exemple original, produisant une perturbation adversaire, c'est-àdire une légère perturbation, imperceptible l'oeil nu, qui lorsqu'elle est ajouté à l'exemple original conduit à une mauvaise classification, comme illustré Figure 6. Ces perturbations adversaires perxiv mettent d'attaquer les réseaux de neurones, pour quiconque voudrait les pirater anonymement. Cependant, grâce à ces perturbations adversaires il est aussi possible d'estimer les vulnérabilités et la robustesse d'un réseau de neurone. Afin d'estimer la robustesse des réseaux de neurones, certains auteurs ont proposé de calculer des perturbations adversaires pseudo-aléatoires qui, grâce à des optimisations efficaces, permettent de délimiter de manière précise le domaine de viabilité des réseaux. Ces perturbations ont été critiquées [START_REF] Sen | Should adversarial attacks use pixel pnorm?[END_REF] car elles ne représentent pas des perturbations d'images réalistes, observables dans la vraie vie, menant ainsi à une estimation de la robustesse des réseaux de neurones trop abstraite. Face à ces critiques, d'autres types de perturbations adversaires ont été proposés comme les perturbations communes (où common corruptions en anglais) [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF], ou bien les patchs adversaires [START_REF] Brown | Adversarial patch[END_REF] visant à calculer une estimation plus pertinente de la robustesse des réseaux de neurones.

+ =

Original image Adversarial perturbation Adversarial image

Contributions

Dans cette thèse, nous avons proposé deux contributions permettant de mieux caractériser et retrouver des exemples adversaires.

Dans le deuxième chapitre, nous avons proposé de modéliser l'espace des bruits adversaires à l'aide d'un dictionnaire d'atomes adversaires, créant ainsi une attaque adversaire faisant le lien entre les attaques adversaires spécifiques et les attaques adversaires universelles. À travers plusieurs soumissions et remises en question, notre proposition a évolué vers une modélisation efficace de l'espace des bruits adversaires permettant de dériver une attaque adversaire robuste et transférable. En effet, l'attaque adversaire construite à partir d'un dictionnaire se révèle compétitive par rapport aux attaques spécifiques de l'état de l'art, tout en étant plus robuste, c'est-à-dire qu'elle résiste mieux aux mécanismes de détection d'exemples adversaires. De plus, les exemples adversaires ainsi calculés se révèlent plus transférables, ils permettent de tromper d'autres réseaux de neurones. Du fait d'être calculé à partir d'une modélisation universelle de l'espace des bruits adversaires, nos expériences montrent que les perturbations adversaires calculées sont plus universelles que les méthodes de l'état de l'art, c'est-à-dire qu'elles peuvent tromper un réseau de neurones sur plusieurs exemples originaux. Avoir modélisé l'espace des bruits adversaires par un dictionnaire, nous permet de visualiser les formes élémentaires à la base des perturbations calculées. En s'appuyant sur un dictionnaire, notre proposition nous permet de révéler les faiblesses globales et la dynamique menant à tromper les réseaux neuronaux. En outre, notre proposition remet également en question les métriques adversaires actuellement utilisées. Da manière générale, notre proposition remet en question la considération actuellement donnée aux perturbations adversaires pseudo-aléatoires pour évaluer efficacement et de manière pertinente la robustesse des réseaux de neurones profonds.

Dans la deuxième contribution, nous remettons en question les mesures de similarité entre images actuellement utilisées. Plus précisément, nous proposons d'apprendre une mesure de xv CONTENTS similarité entre images en optimisant la matrice de coût de la distance de Wasserstein entre leurs représentations issues d'un réseau de neurones. Les mesures de similarité tirées de notre proposition sont plus intéressantes que les mesures actuellement utilisées car elles utilisent les représentations des images plutôt que leurs pixels, permettant de calculer des similarités entre images plus pertinentes et plus réalistes. En effet, les distances d'images basées sur les pixels, ne considèrent pas la sémantique. Notre proposition permet d'explorer la richesse des représentations apprises, afin de construire une distance de Wasserstein pertinente. Nous avons notamment testé notre proposition dans différents contextes d'application afin de le rendre plus robuste et plus efficace selon le choix des hyperparamètres. Disposer d'une telle mesure de similarité pertinente entre images peut s'avérer utile pour évaluer la robustesse d'un réseau face à des exemples adversaires, notamment lorsque l'on travaille sur des applications de vision par ordinateur dans lesquelles des vies sont en danger. Cependant, par manque de temps, nous n'avons pas pu résoudre tous les problèmes liés à l'application de notre proposition sur des ensembles de données plus complexes et à plus grande échelle.

Les différentes contributions ont été présentées dans les articles suivants :

1. Jordan Frecon, Lucas Anquetil, Yuan Liu, Gilles Gasso, Stéphane Canu "Adversarial Dictionary Learning", publié à Conférence sur l'Apprentissage Automatique (CAp), 2021.

2. Lucas Anquetil, Yuan Liu, Jordan Frecon, Stéphane Canu, Gilles Gasso "LIMANS: Linear Model of the Adversarial Noise Space", soumis à International Conference on Learning Representations (ICLR), 2023.

Structure du manuscrit

Cette thèse est divisée en quatre chapitres précédés par cette introduction :

• Le premier chapitre introduit les connaissances de base sur les réseaux neuronaux profonds et formalise les différentes notations, métriques, attaques et défenses adversaires.

• Le chapitre deux présente l'attaque adversaire basée sur un dictionnaire que nous avons proposée.

• Le troisème chapitre détaille notre proposition d'apprendre une mesure de similarité entre images plus pertinente.

• Enfin, nous concluons notre travail et présentons les perspectives futures.

Chapter 1

Background and preliminaries

This chapter relies on the notations introduced in [START_REF] Goodfellow | Deep Learning[END_REF], making the current work self-sufficient. The work in [START_REF] Goodfellow | Deep Learning[END_REF] is considered one of the most comprehensive introductions to deep learning.

Readers are invited to refer to it for further details.

Deep neural network classifiers

Broadly speaking, there have been three waves of deep learning development: at first, deep learning was referred to as cybernetics in the 1940s-1960s [START_REF] Mcculloch | A logical calculus of the ideas immanent in nervous activity[END_REF][START_REF]The organization of behavior: A neuropsychological theory[END_REF], then deep learning was known as connectionism in the 1980s-1990s [START_REF] Rumelhart | Learning representations by backpropagating errors[END_REF], and in 2006 [START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF][START_REF] Hinton | A fast learning algorithm for deep belief nets[END_REF][START_REF] Ranzato | Efficient learning of sparse representations with an energy-based model[END_REF] the deep learning and representation learning rise, contrasting from shallow learning which was fancy at the time, to be the current trend we are witnessing today.

The deep learning algorithms we recognize today were first intended to be computational models of biological learning that is, how learning happens in the brain. As a result, one of the names that deep learning has gone by is artificial neural networks (ANNs). While neural networks have sometimes been used to understand brain function [START_REF] Hinton | Lesioning an attractor network: investigations of acquired dyslexia[END_REF], they are generally not designed to be realistic models mimicking brain function.

The recent (third wave) rise of deep learning has been made possible thanks to two elements combined, the increase of available large-size datasets and the improvement of computer infrastructure in hardware and software. The age of "Big Data" has made machine learning much easier because the key burden of statistical estimation, generalizing well to new data after observing only a small amount of data, has been considerably lightened [START_REF] Deng | What does classifying more than 10,000 image categories tell us?[END_REF][START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF][START_REF] Karpathy | Largescale video classification with convolutional neural networks[END_REF][START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF][START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF][START_REF] Netzer | Reading digits in natural images with unsupervised feature learning[END_REF][START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF].

Alongside the data explosion, both the hardware and software enabled a faster training of neural network models and also to build deeper neural networks, that is to increase the complexity of the model, allowing to learn much more complicated patterns within the data and thus reaching a new standard in terms of performances [START_REF] Chellapilla | High performance convolutional neural networks for document processing[END_REF][START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF].

Neural networks archictures

In this sub-section, we review the different neural network architectures, incrementally according to their complexity.

Feedforward neural networks

In the simplest form, neural networks f : R P → R C are called feedforward networks, deep feedforward neural networks, or multilayer perceptrons (MLPs). Considering a feedforward network f θ , parameterized by θ parameters, and a dataset D = {(x (i ) , y (i ) )} N i =1 of labeled examples with x (i ) ∈ X ⊂ R P and its corresponding label y ∈ Y = {1, ..., C}, the goal of optimizing f is to find the best θ parameters, best modeling the conditional relationship of the data examples x (i ) to their corresponding label y (i ) (for ease of reading, f will implicitly refer to f θ lightening the notation).

A feedforward network defines a mapping from the input space X to the label space Y, and we define the prediction of the example x (i ) by f with f (x (i ) ) = ŷ(i) .

Feedforward neural networks are called networks because they define the composition of different functions (called hidden layer) connected only to one after another. These models are prefixed feedforward because at example x = [x 1 , x 2 , . . . , x P ] T , its information flows through the intermediate computations of f to finally its output ŷ, ŷ = f (x) = f (3) ( f (2) ( f (1) (x))), (1.1) supposing f has three hidden layers. In reality, f can be composed of any number of hidden layers, going up to thousands in some currently used network architecture. The overall number of functions composing f gives the depth of the model. The name deep learning arose from this terminology. Each hidden layer of the network is typically vector-valued. The dimensionality of these hidden layers determines the width of the model. One way to understand feedforward networks is, to begin with linear functions as hidden layers. Linear functions, such as linear regression, logistic regression, or linear support vector machine, are appealing because they can efficiently and reliably be optimized, either in closed form or with convex optimization. Likewise, we can apply the kernel trick with some nonlinear activation function σ, to obtain a nonlinear learning of the hidden layers producing a nonlinear feedforward network. The previous feedforward network f of three hidden layer shown in Figure 1.1 would be defined as,

h (1) = σ (1) (W (1)T x + b (1) ), h (2) = σ (2) (W (2)T h (1) + b (2) ), h (3) = σ (3) (W (3)T h (2) + b (3) ), ŷ = σ (4) (h (3) ), (1.2)
defining θ = [W (1) , b (1) , W (2) , b (2) , W (3) , b (3) ], while σ (1) , σ (2) and σ (3) are activation functions such as relu, σ(x) = relu(x) = max(0, x) enabling the hidden layers to be non-linear functions. The activiation function σ (4) is quite different as it displays the final prediction of f therefore, for classification σ (4) would likely be the softmax function, that is component-wise σ(x) (4) i = softmax(x) i = e x i j e

x j enabling ŷ to define a probability distribution over the labels.

𝒙 𝒉 𝟏 𝒉 𝟐 𝒉 𝟑 # 𝒚
𝝈 𝟏 (𝑾 𝟏 𝑻 𝒙 + 𝒃 (𝟏) )

𝝈 𝟐 (𝑾 𝟐 𝑻 𝒉 𝟏 + 𝒃 (𝟐) )

𝝈 𝟑 (𝑾 𝟑 𝑻 𝒉 𝟐 + 𝒃 (𝟑) ) 𝝈 (𝟒) (𝒉 𝟑 )

.1: Schema of an MLP, composed of three hidden layers. Input and output dimension as well as the dimension of the hidden layers are set arbitrarily for clarity purposes.

Convolutional neural networks

With feedforward neural networks being the simplest case of neural network architecture, the next more complicated architecture type is convolutional neural networks (CNNs), also called convolutional networks [START_REF] Lecun | Generalization and network design strategies[END_REF].

The name "convolutional neural network" indicates the use of a convolution operation. Convolutional networks are simply neural networks that use convolution in place of general matrix multiplication in at least one of their hidden layers. A convolution operation applies a kernel (or filter) to an input signal to extract features or perform spatial transformations.

The output of the convolution s(x, K) of an input x ∈ R P and a kernel K ∈ R k , k P, called the feature map, is the sum of the product of each element of K to each element of x within its range.

The output of the convolution s(x, K) i that is component-wise, Equation 1.3 and Figure 1.2 define and illustrate a convolution operation over a one-dimensional signal, but the operation can be extended to more dimensions as well. It is important to note that the input of the convolution can be padded with zeros to provide an output feature map of identical shape, preventing edge effects. Including a convolution operation within a neural network architecture, implies that the kernel parameters K j , j ∈ {1, . . . , k}, are now part of the θ parameters of f to be found during its optimization.

s(x, K) i = k j =1 x i -j K j . (1.3) 𝑲 𝐱 𝒔(𝒙, 𝑲)
Adding convolutions improves neural networks in two ways, first by lowering the complexity of the model (size of θ, that is the number of parameters to be found) and second by extracting only the most discriminative features as a pre-process of the next hidden layers. Besides, from a more practical viewpoint, convolution provides a way to deal with different sizes of inputs.

Opposite to feedforward networks, CNNs can set the kernel size much smaller than the input size, resulting in sparse interactions. For example, when working with large-scale data such as high-resolution images, the input size can be 10 5 or 10 6 , with convolutions we can detect the small meaningful features in the images, such as corners or edges in the objects, using kernels of size 10 2 or 10 3 only. It means that we need to store fewer parameters, which both reduces the memory requirements of the model and fastens the prediction ŷ computation by running fewer operations, improving the runtime complexity. When setting the kernel size to be lower than the input size, the convolution exploits the local connection patterns in the input. These local connections help capture spatial or temporal dependencies within the input, which is particularly useful for tasks such as image and video processing. In practice, convolution is most often used with a pooling operation [START_REF] Zhou | Computation of optical flow using a neural network[END_REF], a downsampling operation that reduces the spatial dimensions of the input by aggregating neighboring values, typically taking the maximum or average value.

Overall CNNs have tremendously outperformed feedforward neural networks thanks to the convolution operation extracting the most discriminative characteristics from the inputs and thus enabling the overall neural network function to precisely locate and discriminate more complicated patterns.

Residual Networks

CNNs [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Lecun | Backpropagation applied to handwritten zip code recognition[END_REF] have led to a series of breakthroughs for image classification [START_REF] Krizhevsky | Imagenet classification with deep convolutional neural networks[END_REF][START_REF] Sermanet | Overfeat: Integrated recognition, localization and detection using convolutional networks[END_REF][START_REF] Zeiler | Visualizing and understanding convolutional networks[END_REF]. At the same time, Simonyan and Zisserman, Szegedy et al. [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF] revealed that the network's depth is of crucial importance, as a higher depth leads to better results [START_REF] He | Delving deep into rectifiers: Surpassing humanlevel performance on imagenet classification[END_REF][START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF][START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF][START_REF] Szegedy | Going deeper with convolutions[END_REF] on the challenging ImageNet dataset [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF].

However, up to this point, convolutional neural networks cannot be "too deep" as they become too difficult to optimize. The depth training problem, also known as the vanishing gradient problem, refers to the phenomenon where gradients in deep neural networks diminish exponentially as they backpropagate through multiple hidden layers during the training, making it challenging for earlier hidden layers' weights to be efficiently optimized. Residual connections [START_REF] He | Deep residual learning for image recognition[END_REF] solve the vanishing gradient problem. Residual connections are links within the deep neural network between the early hidden layers and the final hidden layers as Figure 1.3 shows. As well as proposing the residual connections idea, authors of the original work [START_REF] He | Deep residual learning for image recognition[END_REF] introduce a series of neural network architectures including residual connections, termed ResNets. Figure 1.3 shows a neural network including two residual connections. Even though the residual connections here are only set for display purposes, we can see the hidden layer h j 's parameters will be updated with a gradient of the same magnitude as those of the last layer h n . By composition, we understand through the other residual connection, that the first hidden layer h 1 's parameters will be updated with a gradient only an order of magnitude lower than those of the last layer h n . This way, early hidden layers can receive insightful gradients allowing us to update them efficiently and therefore solving the vanishing gradient issue.

𝒙 𝒉

𝟏 𝒉 𝒋 = 𝜎 # (𝑊 (#) ℎ #&' + 𝑏 (#) ) + 𝒉 𝟏 𝒉 𝒏 + 𝒚 ⋯ ⋯ + 𝒉 𝟏 𝒉 𝒋 + 𝒉 𝒋 𝒉 𝒏 = 𝜎 ) (𝑊 ()) ℎ )&' + 𝑏 ()) ) + 𝒉 𝒋
The neural network architectures used in this thesis involve VGG [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], Inception [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF], Google-Net [START_REF] Szegedy | Going deeper with convolutions[END_REF], MobileNet [START_REF] Howard | Mobilenets: Efficient convolutional neural networks for mobile vision applications[END_REF], ResNet [START_REF] He | Deep residual learning for image recognition[END_REF] that are all composed of the presented neural networks architecture tools as well as little specific tricks making them efficient for their purpose at their publication date.

Recent state-of-the-art neural network architectures, such as NAS [START_REF] Zoph | Neural architecture search with reinforcement learning[END_REF], CoCa [START_REF] Yu | Coca: Contrastive captioners are image-text foundation models[END_REF], ViTs [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] involve new kinds of technology such as variants of attention mechanisms [180] [165] or masking [START_REF] Fang | Eva: Exploring the limits of masked visual representation learning at scale[END_REF] for example. Those technologies are out of the scope of this thesis, however, readers are invited to refer to current benchmark1 as shown in Figure 1.4, reviewing the current state of the art neural networks models.

= {(x (i ) , y (i ) )} N i =1 with x (i ) ∈ X ⊂ R P and y (i ) ∈ Y = {1, . . . , C} this process is called empirical risk minimization, min θ 1 N N i =1 L( ŷ, y), (1.4) with ŷ = softmax([ f 1 (x (i ) ), f 2 (x (i ) ), . . . , f C (x (i ) )]) T
the predicted label probability vector of x (i ) through f . The predicted label is given by choosing the label receiving the maximum probability, argmax k f k (x) = argmax k ŷk .

In equation 1.4, the loss function is denoted by L represents a real positive value, a penalty of a bad prediction. The loss is a real value indicating how bad the model's prediction is on a single example. If the model's prediction is perfect then the loss is zero otherwise, the loss is positive.

The goal of training a model is to find the best parameters θ, producing the lowest loss, on average, across all examples in D.

The choice of the loss function is of the utmost importance when building machine learning models as it plays a central role in its optimization. Losses are usually associated with the task at hand. It means that some regression loss might not be suited to train a classifier. Here we will review some of the most common classification losses, as other task losses are out of scope for this thesis.

0-1 loss

The simplest classification loss is the 0-1 loss, which simply counts the number of bad predictions,

L( ŷ, y) = 1 if argmax k ŷk = y 0 if argmax k ŷk = y.
(1.5)

Even though the 0-1 loss is quite simple to understand, this loss is not relevant as it can hardly be incorporated into an optimization scheme. Indeed, as we will see in section Section 1.1.3 the 0-1 loss does not allow us to compute gradients with respect to the model's parameters θ, preventing us from updating them.

Hinge loss

Very popular in the early 2000s, Support Vector Machines (SVMs) [START_REF] Cortes | Support-vector networks[END_REF] were at the time, the stateof-the-art classifier models. SVMs are mainly optimized using the hinge loss (also called margin loss). The core idea of hinge loss is to provide a margin-based measure for training classifiers, by maximizing the margin between the decision boundary of the classifier and the training data points. With the hinge loss, the loss is zero if the example is correctly classified, and positive if its prediction lies within a chosen margin from the decision boundary.

L( ŷ, y) = max(0, ρ -1 .{y = argmax k ŷk }), (1.6)
with ρ as the allowed margin hyper-parameter of the hinge loss. Figure 1.5 illustrates the difference between the Hinge and the 0-1 loss. Those two losses are presented because of the impact and fundamental influence they've had in the machine learning community, however, nowadays neural network classifier optimizations do not use them. Instead, the most likely loss involved to optimize neural network classifier is the cross-entropy.

Cross-entropy loss

The cross-entropy loss, as its name suggests, takes roots from the entropy dissimilarity measure. Entropy is a measure of uncertainty or disorder in a probability distribution. The entropy of a discrete probability distribution S is defined as, entropy(S) =event p(event)l og (p(event)), (1.7) with p(event) representing the probability of the realization of an event in the distribution and summing over all possible events. While entropy measures the disorder in a probability distribution, cross-entropy measures the difference between two probability distributions. In the context of classification, cross-entropy is used to compare the classifier's probability distribution to the true label distribution represented by a Dirac when considering single-label classification,

L( ŷ, y) = - C k=1 o k l og ( ŷk ), (1.8)
with ŷk representing the predicted probability of the label k for the example x through f , and o a one-hot vector with 1 at index y representing the true label of x. Cross-entropy is minimized when the probability distribution ŷ matches the true distribution o perfectly, which is when ŷ puts a probability of 1 on the true label y and 0 elsewhere. Minimizing the cross-entropy effectively maximizes the likelihood of the true labels under the probability distribution ŷ.

Triplet loss

So far, the presented losses mainly cover the classification task, but in this thesis, we explored the metric learning problem too. The metric learning problem involves learning a distance or similarity metric that captures the relationship or dissimilarity between points. In classification, it is often essential to have an appropriate distance metric d : R P × R P → R that can effectively measure the similarity or distance between points. In most research and applications, p norm distances are set as standard distances however, p norm distances may not always be suitable [START_REF] Sharif | On the suitability of lp-norms for creating and preventing adversarial examples[END_REF], as it treats all dimensions equally and may not reflect the underlying structure or semantics of the data.

Metric learning methods typically leverage labels or pairwise constraints to guide the learning process. A metric learning algorithm optimizes a loss that encourages the desired properties of the learned metric, such as being low for points of the same label, and high for points of different labels. To this end, the triplet loss [START_REF] Schroff | Facenet: A unified embedding for face recognition and clustering[END_REF] has been proposed. Given a dataset of labeled examples D = {(x (i ) , y (i ) )} N i =1 with x (i ) ∈ R P and y (i ) ∈ Y = {1, . . . , C}, the triplet loss generates triplets {(x a , x p , x n ) (i ) } from D, with x a an anchor point, x p a positive point of same label as x a and x n a negative point of different label. From these triplets, the triplet loss explicitly minimizes the distance between points of the same label d (x a , x p ) and maximizes the distance between points of different label d (x a , x n ) under a margin,

L(d,D) = (x a ,x p ,x n )∈D max(0, ρ -d (x a , x n ) + d (x a , x p )),
(1.9) with ρ the allowed margin of error.

Optimal transport loss

In recent years, the research community has seen a surge of interest in optimal transport distances [START_REF] Villani | Optimal transport: old and new[END_REF]. Optimal transport distances, also known as Wasserstein distances (WD) or Earth Mover's distances [START_REF] Rubner | The earth mover's distance, multidimensional scaling, and color-based image retrieval[END_REF], are mathematical measures that quantify the dissimilarity or distance between two probability distributions. The basic idea behind optimal transport distances is to consider the cost of moving each unit of mass from a source point to a destination point. Originally, the cost function of transporting points is assumed to be known and set as a parameter. In the machine learning context, the WD quantifies the minimum cost required to transform one probability distribution into another, taking into account the distances between the individual points in the distributions.

The WD thus provides a measure of dissimilarity between distributions that considers both their shapes and the geometric structure of their support. With one dimensional point, the WD reads WD C : R N × R M → R, and computes the association of N points from one probability distribution a to M points of another probability distribution b, such that the sum of the overall transportation under the cost function C is minimized,

WD C (a, b) = min T∈R N ×M N i =1 M j =1 T i , j C i , j ,
such that T1 = a and T T 1 = b, (1.10) with the matrix C ∈ R N ×M being the cost-matrix, that quantifies the effort of moving points, with C i j the effort of moving points from location a i to location b j . Figure 1.6 shows a valid transport plan between two points clouds. Its associated WD would be the sum of all the black arrows' costs, representing the cost of moving all the points a i to all the points b j . Even though it is very appealing for the geometric understanding and semantic-based definition, computing the exact WD can be computationally expensive, especially for high-dimensional, large-scale problems. To make the problem tractable and computationally efficient, researchers resort to its relaxed version, the Sinkhorn divergence [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF]. The Sinkhorn divergence, is an approximation of the WD that leverages the Sinkhorn-Knopp algorithm, a matrix scaling algorithm computing an approximate transport plan efficiently. The Sinkhorn divergence relaxes the strict constraints of the WD by introducing regularization parameters, which control the trade-off between computational efficiency and the accuracy of the approximation. The Sinkhorn divergence WD α C simply consists of a regularization term added to the original distance WD C ,

𝒂 𝟐 𝒂 𝟒 𝒂 𝟑 𝒂 𝟓 𝒂 𝟏 𝒂 𝒃 𝟐 𝒃 𝟒 𝒃 𝟑 𝒃 𝟓 𝒃 𝟏 𝒃 𝑻
WD α C (a, b) = min T∈R N ×M N i =1 M j =1 T i , j C i , j + α N k=1 M t =1 T i , j (log T i , j -1), such that T1 = x s and T T 1 = x t , (1.11)
with α a regularisation parameter. Usually, the optimization considers instead the dual formulation of the Sinkhorn divergence explicitly including the constraints with the dual variables. Readers are invited to see Section A for details of the dual formulation as well as the Sinkhorn-Knopp algorithm definition. The design of an appropriate loss according to the objective, the constraints, and the dataset at hand, is complex and requires careful attention. Loss conception and definition is still an ongoing research, see [START_REF] Wang | A comprehensive survey of loss functions in machine learning[END_REF] for a survey and more details on the recent improvements in this research field.

Optimization

Once we defined a neural network architecture's parameters θ and the appropriate loss function L, we have the following optimization problem,

min θ 1 N N i =1 L( f θ (x (i ) ), y (i ) ) = L( ŷ, y).
(1.12)

The simplest and the historically most used strategy to update the parameters θ is to proceed with a stochastic gradient descent (SGD) optimization scheme,

θ ← θ -γ∇ θ L(f θ (x (i ) ), y (i ) ), (1.13)
with (x (i ) , y (i ) ) randomly chosen in the dataset, and γ a fixed step size. Quite basic but efficient, SGD optimization is still used, especially with optimization tricks such as averaged mini-batch gradient computation, or mini-batch sampling strategy. One of the main improvements that have been proposed was that rather than choosing a fixed step size γ, dynamically computing a new step size γ at each gradient update, according to the gradient's magnitude. The Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] optimization strategy has been proposed to dynamically update the step size according to the gradient's magnitude, outperforming the original SGD optimization by far. To calculate the step size, Adam performs an optimization problem on its own by using two moving average variables, namely the first moment of the gradients, the mean with a first hyper-parameter β 1 , and the second moment, the variance of the gradients with a second hyper-parameter β 2 . These moving averages are updated at each iteration of the optimization process. Readers are invited to refer to Algorithm 1 on page 2 of [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] for a more precise definition of the performed computation of γ.

During the past decade, the Adam optimizer has been the most used and effective, but some variants have been proposed, such as Nadam [START_REF] Dozat | Incorporating nesterov momentum into adam[END_REF] or Radam [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF] for instance. These have been outperformed by other step size computations such as Sophia [START_REF] Liu | Sophia: A scalable stochastic second-order optimizer for language model pre-training[END_REF], which enabled us to reach new state-of-the-art performances with transformer models. New state-of-the-art performances are reached using new and better optimizers [START_REF] Chen | Symbolic discovery of optimization algorithms[END_REF][START_REF] Mishchenko | Prodigy: An expeditiously adaptive parameter-free learner[END_REF]. Further information and other optimization procedures are given in the fast-ai2 and Fidle3 courses.

Generative Adversarial Networks

Another way to train neural networks is to resort to the generative adversarial networks (GAN) frameworks, proposed in [START_REF] Goodfellow | Generative adversarial networks[END_REF]. A GAN framework consists of two models competing against each other. On one side of the framework is a generator g trying to generate synthetic data as similar as possible to true data, and on the other side is a discriminator d acting as a classifier, trying to distinguish the true data from the synthetically generated data. More formally a GAN learns to map the simple latent distribution z ∼ p z to the more complex data distribution, the true data x ∼ p d at a . By having its generator compete against its discriminator, the GAN's objective results in a min/max objective, min

g max d E x∼p d at a [log d (x)] + E z∼p z [log 1 -d (g (z))].
(1.14)

As Figure 1.7 shows, both d and g are optimized from equation 1.14 by backpropagating their loss with respect to their respective model. GAN framework received a lot of attention due to its capacity to optimize a generator producing almost-real synthetic data as Figure 1.8 shows (see [START_REF] Gui | A review on generative adversarial networks: Algorithms, theory, and applications[END_REF] for a recent survey on GAN). Even though a GAN generator does not create so-called adversarial examples 1.19, the core objective of the generator is still to generate imperceptible to the human eye, fake data.

Frameworks

During the past decade, machine learning research has been extensively supported by various companies such as Google, Meta, and Microsoft for example. Fortunately for academic research these companies released very powerful cutting-edge open-source machine learning frameworks such as Tensorflow [1], Pytorch [START_REF] Paszke | Pytorch: An imperative style, high-performance deep learning library[END_REF], Apache MXNet [START_REF] Chen | Mxnet: A flexible and efficient machine learning library for heterogeneous distributed systems[END_REF] or the new in town JAX [START_REF] Frostig | Compiling machine learning programs via high-level tracing[END_REF] for example. Each optimizer has its own strengths and weaknesses, depending on either research or production, one can be more appropriate than the other. Figure 1.9 gives a comparison of the usage for the last five years. As of today, the Pytorch framework is on the rise.

The work proposed in this thesis heavily relies on the framework Pytorch, which has many efficient and powerful features enabling us to perform extensive experiments. Pytorch has the main advantage of including the Autograd [START_REF] Paszke | Automatic differentiation in pytorch[END_REF] computation package, which as the name suggests, automatically computes reliable gradients of any differentiable loss with respect to its input. In this thesis, we focused on adversarial examples and resorted to publicly released and peer-reviewed validated benchmarks of the adversarial attack, namely Torchattacks [START_REF] Kim | Torchattacks: A pytorch repository for adversarial attacks[END_REF] and RobustBench [START_REF] Croce | Robustbench: a standardized adversarial robustness benchmark[END_REF]. All the implementations of this thesis are made available 4 and we thank the Pytorch community for releasing such crucial tools to us. Figure 1.9: Comparison of deep learning framework use, from their Google keyword search. On the x-axis is the time and on the y-axis is displayed a score of the keyword usage in the Google search engine. Source: trends.google.com

Datasets

During this thesis, our experiments were exclusively performed on image datasets, but to some extent, can be extended to other data modalities.

We performed experiments on the widely used benchmark dataset MNIST [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] ("Modified National Institute of Standards and Technology"). This dataset describes a collection of handwritten digit images, specifically a training set with 60,000 examples and a test set with 10,000 examples. Each example in the dataset is a grayscale image of size 28x28 pixels, representing a handwritten digit from 0 to 9. Quickly becoming a limiting dataset, MNIST derived more complex datasets such as Fashion-MNIST [START_REF] Xiao | Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms[END_REF], EMNIST [START_REF] Cohen | Emnist: Extending mnist to handwritten letters[END_REF], KMNIST [START_REF] Clanuwat | Deep learning for classical japanese literature[END_REF] or even QMNIST [START_REF] Yadav | Cold case: The lost mnist digits[END_REF].

The CIFAR-10 dataset [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] is another widely used benchmark dataset in the field of computer vision. CIFAR stands for "Canadian Institute For Advanced Research". The CIFAR-10 dataset includes labeled RGB images of 10 classes. The CIFAR-10 dataset comprises 60,000 RGB images divided into a training set with 50,000 images and a test set with 10,000 images. Each image in the dataset has a size of 32x32 pixels and is labeled with one of ten classes: (airplane, automobile, bird, cat, deer, dog, frog, horse, ship, or truck). The images in CIFAR-10 exhibit more complexity compared to the MNIST dataset. They contain color information and represent a broader range of objects and backgrounds. The dataset is designed to reflect real-world scenarios and challenges in object recognition and classification tasks.

As a step further real-world scenarios, we performed experiments on the ImageNet dataset [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF]. It is designed for object recognition and classification tasks and has been crucial in advancing state-of-the-art in computer vision. The ImageNet dataset originally consisted of millions of labeled images from a vast range of object categories. However, the most widely known and used subset of ImageNet is the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF], which focuses on a subset of 1.2 million images across 1000 object classes. Each image in the ImageNet dataset can vary in size and resolution. The dataset covers a wide variety of objects, including animals, vehicles, household items, natural scenes, and more. The dataset also includes a significant number of challenging images, such as those with multiple objects or cluttered backgrounds.

Most of the datasets currently used are available on the Kaggle5 website. Kaggle is an online platform hosting machine learning competitions, it provides datasets for practice and offers a collaborative environment for machine learning projects. Readers are invited to take a look out of curiosity.

The current trend in deep learning

State-of-the-art performances are reached using neural networks to learn complex representations of the data making the final predictions very accurate.

The current trend is to blindly increase the number of parameters of the model (the model's size), allowing it to learn much more complex representations, and achieve outstanding performances. Figure 1.10 shows the rise of the neural network size for different tasks according to their publication date. As we can see, in spite of their consequences, researchers and private companies chose to increase, without any limit, the neural network size. However, training neural networks is expensive and requires a lot of resources. Training deep neural networks is computationally intensive and requires significant computational resources, including powerful GPUs or specialized hardware. Large models with millions of parameters can be challenging to train and deploy in resource-constrained environments, requiring a significant amount of electrical power and therefore displaying a very poor carbon footprint.

Besides, in this thesis, we empirically show once more, that neural networks are not almighty, and are sensitive to small changes in the distribution of the input data. We show that neural network performances can easily collapse under very small perturbations of the data, showing real harm in releasing them on real-world life at-risk applications.

Overall, this thesis aims to temper the growth of neural network size and to bring the esteemed research institutions, perhaps blinded by the pursuit of gain, back to reason. We believe the focus should be placed on ensuring the security, thorough understanding, and effective control of a bit smaller neural networks but still exhibiting excellent performances.

Adversarial attacks and examples

In some applications, neural networks are reaching human-level performances when evaluated on an i.i.d. (independent and identically distributed) test set. However, several authors [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF] highlighted the vulnerability of neural networks to adversarial attacks. Adversarial attacks are functions of the deep classifier and original example producing adversarial example, i.e. a slightly perturbed original example misleading the classification. With images, an adversarial perturbation refers to an "imperceptible" perturbation of an image, i.e. a slight change in the pixels, so that this image remains unchanged for the human eye while causing misclassification as shown in Figure 1.12. These adversarial examples question the trust we put in deep learning models in our everyday tasks, especially in life-at-risk applications such as autonomous cars.

In this thesis, we mainly focus on images, even though adversarial examples could be of any modality (for instance trajectory attack shown in [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF]).

Adversarial examples are interesting to anyone who wants to evaluate the robustness of a classifiers and certify their use [START_REF] Vidot | Certification of embedded systems based on machine learning: A survey[END_REF], or to some ill-intentioned hacker who wants to jeopardize the system.

The goal of a hacker is to corrupt the system by any means. In [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF], the authors studied the adversarial harm of data-driven trajectory prediction systems within autonomous cars. They devise an adversarial attack framework generating realistic adversarial trajectories. Authors showed their adversarial attack can lead an autonomous car to drive off-road or collide with other vehicles in simulation.

Figure 1.11: Example of adversarial trajectory proposed by the adversarial attack of [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF]. By driving along the crafted adversarial history trajectory, the adversarial agent (red car) misleads the prediction of the autonomous car (AV) for both itself and the other agent (yellow car). Consequently, autonomous car planning based on the wrong prediction results in a collision. Source: AdvDO: Realistic Adversarial Attacks for Trajectory Prediction [START_REF] Cao | Advdo: Realistic adversarial attacks for trajectory prediction[END_REF]. Indeed, in Figure 1.11, by shifting the left yellow car into an adversarial agent that only goes in the opposite direction, and does not cross the trajectory of the green autonomous car, they make the trajectory prediction fail to predict a safe path leading to a crash with the right yellow car.

Opposite to what we study in this thesis, the hacker did not have to perturb the system itself but only its environment, highlighting the fragility of current models used in life-at-risk applications, and our need to assess and improve their robustness.

Adversarial examples are interesting to anyone attempting to evaluate the robustness of a model before releasing it into life-at-risk applications Overall researching the origin and the computation of adversarial examples is of the utmost importance and crucial to the integration of the model into life-at-risk applications.

Evaluating the robustness of a classifier with adversarial examples is tricky because, among all the adversarial attacks, different families co-exist with very different goals regarding the adversarial examples. There are pseudo-random adversarial attacks, common corruptions, adversarial patch attacks, and other types that are out of the scope of this thesis. Among these families, we mainly explored the pseudo-random adversarial attacks, attacks that generate very small pseudorandom adversarial perturbations, uncorrelated to the semantics of the data.

In this section, we first define the adversarial examples optimization problems, and the different visions of it leading to different goals with adversarial examples. In these definitions, we highlight the differences between pseudo-random adversarial examples, common corruptions, and adversarial patches. After defining adversarial examples we formalize their different desired properties (universality and transferability). Next, we define adversarial metrics that measure the quality of an adversarial example generation. Then, as we deeply explored it, we reviewed some of the most important pseudo-random adversarial attacks (white box, black box, universal attacks, and other types of pseudo-random attacks). Finally, we present the different families of adversarial defenses, mechanisms aiming to protect classifiers from adversarial harm.

Definitions

Exact adversarial attacks

Adversarial attacks are algorithms generating adversarial examples. They are functions of an original example x ∈ X of which we want to find its adversarial example x and a classifier f that misclassifies x . Originally, the goal of an adversarial example is to find the closest point x ∈ X to an original example x ∈ X that maximizes the misclassification gap (lowering as much as possible the right classification) of a given neural network classifier f . Given an observed (also called clean) example

x labeled as k = argmax k f k (x) by f , its adversarial example for f reads, min

ω x ∈Ω dist(x, x + ω x ), (1.15a) max ω x ∈Ω (max k =k f k (x + ω x )) -f k (x + ω x ), (1.15b)
with dist a similarity metric between data points (most commonly used distances are p norms, but we discuss this choice in Chapter 3). In this definition we assumed that the adversarial example x ∈ X takes the form x = x + ω x ∈ X as shown in Figure 1.12, which makes the adversarial example generation additive. Rather than being additive, other types of adversarial attacks exist, with a different form of the adversarial example such as the Adversarial patch, detailed in Section 1.2.1. However the additive type of adversarial attacks remains the most studied and is the one we focus on in this thesis therefore, unless explicitly rectified, assume x = x + ω x . Originally adversarial attacks were proposed for binary SVM classifiers by [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF], but the latter were extended to more class classifiers. See that (1.15) defines a bi-criteria optimization in which both criteria are opposing. This makes the problem very difficult to solve thus relaxations have been proposed.

Note that we formalized the adversarial noise space with Ω. This formalism has rarely been done and in this thesis, we explore different constraints we can impose on this space such that interesting adversarial noises can be computed. In an unconstrained setting, pseudo-random adversarial attacks consider Ω = R P , while common corruptions [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF] define Ω = ω = αD with α ∈ R 15 , D = [ Gaussian Noise, Shot Noise, ..., JPEG ] , we review them more precisely in Section 1.2.4, and Section 1.2.1 respectively. In equation (1.15), the goal of (1.15b) is to maximize the difference between any other label prediction and the original label prediction. Such an attack is coined untargeted adversarial attack, but its targeted version is just a slight modification and reads, min

ω x ∈Ω dist(x, x + ω x ), (1.16a) max ω x ∈Ω f t (x + ω x ) -f k (x + ω x ), (1.16b)
with t the targeted label, we want the adversarial example to be classified as.

In practice, the misclassification goal (1.15b) is enforced through the minimization of an adversarial loss function l making the optimization of the adversarial examples easier, more manageable, and more efficient. In a way, equation (1.15b) already is an adversarial loss function. Carlini and Wagner [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] studied multiple adversarial loss functions that we review and detail through the presentation of the adversarial attacks in Section 1.2.4.

In this thesis, unless highlighted otherwise, we mainly explore untargeted adversarial attacks, as some of the most used adversarial attacks simply set the targeted label t as the second best label of the original example t = argmax k =k f k (x), k = argmax k f k (x). Because solving equation (1.15) is too complicated for state-of-the-art classifiers and high dimensional data, researchers instead proposed to solve relaxation versions.

Adversarial attacks focusing on the quality of adversarial example

The first relaxation of equation (1.15) was proposed by [START_REF] Wang | Analyzing the robustness of nearest neighbors to adversarial examples[END_REF], which target, for a classifier f and a given original example x, its best adversarial pair, min (1.17)

ω x ∈Ω dist(x, x ) such that argmax k f k (x ) = argmax k f k (x),
In this definition, the second objective (1.15b) has been replaced with a constraint of misclassification. See that this relaxation eases the original formulation by considering adversarial any example classified with another label than the one of the original example. The gap between the original and adversarial predictions is accepted to be neglected as the purpose of the adversarial examples is to overcome the classifier's decision boundary, by whatever margin. However, in this relaxation, the first objective (1.15a) of the original problem remains thus, these attacks mainly focus on finding the closest adversarial point, that can be seen as the quality of the crafted adversarial examples (the smallest noise possible, overcoming the classifier's boundaries).

When finding the exact smallest adversarial noise of x overcoming the classifier's boundaries, these adversarial attacks find the exact adversarial example of x. Adversarial attacks producing exact adversarial examples are not that much explored as neural networks are very complex and highly not convex functions, thus making the problem very difficult to solve and intractable for high dimensional data. However, some exact adversarial attacks exist for certain types of neural networks, such as the reluplex [START_REF] Katz | Reluplex: An efficient smt solver for verifying deep neural networks[END_REF] relying on satisfiability modulo theories (SMT) solvers [START_REF] Bastani | Measuring neural net robustness with constraints[END_REF][START_REF] Huang | Safety verification of deep neural networks[END_REF][START_REF] Pulina | Challenging smt solvers to verify neural networks[END_REF] to reach exact adversarial examples. The main flaw of the reluplex is that the attack can only be performed on neural networks using only the non-linear relu activation function between the hidden layers. Such restriction of the reluplex attack makes it impractical for state-of-the-art classifiers, preventing it from becoming a standard adversarial attack.

The other envisioned relaxation of original problem 1.15 considers the performance of adversarial attacks to be over multiple original examples while neglecting the precision on each single computed adversarial perturbation. The attacker's objective over multiple original examples is to fool the classifiers on a maximum number of data points, which is measured by the fooling rate (1.24).

Adversarial attacks focusing on the fooling rate

The other proposed relaxation of the original problem is to relax the first objective (1.15a), by allowing all adversarial perturbations to live in a certain p norm ball of radius, max

ω x ∈Ω max k =k f k (x ) -f k (x )
such that dist(x, x ) ≤ , and x ∈ X . (1.18) This relaxation emphasizes maximizing the probability of finding any adversarial example x at maximum distance from its original example x. In some sense, the first constraint of (1.18) can be seen as setting Ω = {ω x ∈ R P , dist(x, x ) ≤ }. For ease of reading, because lots of adversarial attacks rely on this relaxation with an p norm as dist similarity metric, we set Ω = {ω ∈ R P , ||ω x || p ≤ } that is equivalent to the first constraint of (1.18).

This relaxation of the original objective considers the efficiency of the adversarial attack as the maximum number of points for which the adversarial attack found an adversarial example at distance from them.

See that two relaxations (1.17) and (1.18) have a different quality metric, whereas one evaluates its performance on a single point, the other needs on all of the points to estimate its performance.

The main advantage of the second relaxation compared to the first one, is that all adversarial attacks are given the same adversarial budget, and can therefore be fairly compared and benchmarked for comparison. Even though both relaxation problems are very appealing most of the state-of-the-art adversarial attacks have their specific optimization strategy and stop the optimization of ω when all of the following conditions are valid,

argmax k f k (x ) = argmax k f k (x), (1.19a) dist(x, x ) ≤ , (1.19b)
x ∈ X , (1.19c)

ω ∈ Ω. (1.19d)
All these conditions are necessary regarding the purpose of the adversarial example. The first condition (1.19a) enforces to overcome the classifier's decision boundaries, as it is its original main purpose. The second condition (1.19b) stipulates that the adversarial examples should be close to the original example, at an distance imperceptible to the human eye. Setting an authorized adversarial budget allows us to fairly compare and benchmark all the different adversarial attacks. Condition (1.19c) is necessary regarding the feasibility of the problem. Indeed, if the device capturing the images displays only positive pixels then negative values in the adversarial examples are prohibited as they are not to be seen in a real-world scenario and therefore, violate the original assumptions. Finally, as a formalism, we define with condition (1.19d) the space Ω in which adversarial noises ω live so that the different attacks' optimization are all made clear and distinct.

Figure 1.13 schematizes an adversarial example along with its optimization direction for 2dimensional point clouds. The exact adversarial example would lie on the decision boundary in the direction of the arrow from the point x while an adversarial example x computed with the second relaxation is displayed and could be anywhere behind the decision boundary and inside the ball.

So far, we have seen different optimization problems targeting pseudo-random adversarial perturbations. Pseudo-random adversarial perturbations have the advantage of exploring in depth the original input space X such that a higher number of adversarial examples can be found. It allows a more precise evaluation of the robustness of the classifier. Besides, from an ill-intentioned person, crafting pseudo-random adversarial perturbations is the easiest and therefore, is the most obvious choice for corrupting the system. Pseudo-random adversarial attacks highlight the attacking vision of adversarial examples. However, these pseudo-random perturbations are especially criticized because they do not make sense from a real-world viewpoint. Indeed, in real-world scenarios, it is unlikely to find a pink pixel in a garden picture. What's the point of evaluating the robustness of classifiers on data points that will never be seen in real-world applications? Another kind of adversarial example generation is adversarial patch attacks. These attacks are not of the additive type. They are looking for the solution of a simple human corruption, the stick of a harmful patch on real objects making the classifiers fail at recognizing them. We next detail the patched adversarial examples.

Adversarial patch attacks

To answer the question of a more realistic adversarial example, researchers proposed to compute, instead of pseudo-random adversarial examples, patched adversarial examples (also known as Adversarial Patch) [START_REF] Brown | Adversarial patch[END_REF].

Adversarial patch attacks are algorithms generating, for a target classifier f , a small patch that, added at a particular position of each image, produces an adversarial example fooling f . Figure 1.14 illustrates such an adversarial patch added to a car image. Once the patch is found and optimized, anyone can print the patch and stick it to any object to make the classifiers fail to recover the right classification of those objects. Adversarial patch attacks are not of the additive type adversarial attacks but more complicated optimization algorithms. Instead of considering every pixel of the image, the adversarial patch attack creates adversarial examples by completely replacing a part of the image with a patch, which can be seen as a mask of the image. The patch is allowed to take any shape and is optimized using gradient descent over a variety of images, subject to random translation, scaling, and rotation for each image. As quoted in [START_REF] Brown | Adversarial patch[END_REF], "For a given image x ∈ R P , patch p, patch location l , and patch transformations t (e.g. rotations or scaling) the attack defines a patch application operator A(p, x,l , t ) which first applies the transformations t to the patch p, and then applies the transformed patch to the image x at location l ", illustrated in Figure 1.15. The patch p is optimized using a variant of the Expectation over Transformation (EOT) framework of Athalye, Engstrom, Ilyas and Kwok [START_REF] Athalye | Synthesizing robust adversarial examples[END_REF], that is

p = argmax p E x∈D,t ∈T,l ∈L l og f y t (A(p, x,l , t ))], s.t. ||x -x || ≤ , (1.20)
where y t is the target label we want the adversarial example to be classified as, D is a training set of images, T is a set of path transformations, and L is a set of possible locations in the image.

Notice this definition differs from what we saw before as the adversarial attack is not additive. Indeed, the adversarial patch attack considers x = A(p, x,l , t ) = x+ω. This framework allows much more flexibility in the crafting of the adversarial perturbation but doesn't permit it to be fairly benchmarked to other adversarial attacks as its goal is not the same. However, note that this expectation of equation 1.20 is over the dataset of images, which encourages the trained patch to work regardless of what is in the background. It makes the adversarial patch somewhat universal to every image. Universality is a property we define in Section 1.2.2 and we present universal adversarial attacks in Section 1.2.4. Figure 1.16 shows other types of patched adversarial examples. In Figure 1.16a, an adversary altered a speed limit sign by adding a little patch, to force an autonomous vehicle to fail at recovering the right speed limit, while in Figure 1.16b an adversary added small black and white patches to a stop sign, making autonomous vehicle failing at recognizing it as a stop sign. As we can see these adversarial examples make a lot more sense from our human eye judgment than pseudo-random adversarial examples, highlighted in Figure 1.12.

While [START_REF] Brown | Adversarial patch[END_REF] is the first work introducing Adversarial patches, newer and fancier patch attacks have been proposed, such as the carpet-bombing patch attack, which requires less knowledge from the targeted model or training data. For newer adversarial patch attacks, readers are invited to refer to [START_REF] Sharma | Adversarial patch attacks and defences in vision-based tasks: A survey[END_REF] for a comprehensive survey on vision-based adversarial patch attacks and defenses.

Opposite to pseudo-random adversarial perturbations and adversarial patches, are common corruptions. The common corruptions are not an attack but a set of real-world perturbations of the data likely to be expected in real-world scenarios. However, as they define a limited number, fixed in stone, of perturbations, shared to all examples and all classifiers, they are mainly limited to modeling the adversarial harm. Therefore, common corruptions are closer to real-world application harm but are too limited to precisely evaluate the robustness of a classifier.

Common corruptions

Alongside adversarial patches, common corruptions are another class of adversarial perturbations that aims to be more realistic than pseudo-random adversarial perturbations. Common corruptions presented in [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF] are real-world modifications of the input data, shown in Figure 1.17, that are set in stone and used to measure the robustness of classifiers against real-world perturbations. In their work [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF] benchmarked the robustness of classifiers on the dataset ImageNet-C through a brand new instance of the dataset, ImageNet-P composed of the original images perturbed by a set of a priori chosen perturbations such as adding Gaussian Noise, adding Zoom Blur or modification of the brightness of original images, detailed in Figure 1.17.

To some extent, common corruptions can be seen as adversarial noise ω ∈ Ω, living in a constrained space Ω. Rather than setting Ω = R P and allowing adversarial noises to be pseudo-random, common corruptions constraint the adversarial noise space to be defined as Ω = ω = αD with α ∈ R 15 , D = [ Gaussian Noise, Shot Noise, ..., JPEG ] , resulting in more realistic adversarial noises.

Figure 1.17: Originally proposed common corruptions. Source: Benchmarking neural network robustness to common corruptions and perturbations [START_REF] Hendrycks | Benchmarking neural network robustness to common corruptions and perturbations[END_REF].

As quoted in [START_REF] Michaelis | Benchmarking robustness in object detection: Autonomous driving when winter is coming[END_REF], "Autonomous cars need to be able to cope with wildly varying outdoor conditions such as fog, frost, snow, and night setting, just to name a few (as visualized in Figure 1.18). One of the major weaknesses of autonomous cars is the inability of their recognition models to function well in adverse weather conditions [START_REF] Dai | Dark model adaptation: Semantic image segmentation from daytime to nighttime[END_REF]. Getting data for unusual weather conditions is hard and while many common environmental conditions can (and have been) modeled, including fog [START_REF] Sakaridis | Semantic foggy scene understanding with synthetic data[END_REF], rain [START_REF] Bernuth | Simulating photo-realistic snow and fog on existing images for enhanced cnn training and evaluation[END_REF], snow [START_REF] Bernuth | Simulating photo-realistic snow and fog on existing images for enhanced cnn training and evaluation[END_REF] and daytime to nighttime transitions [START_REF] Dai | Dark model adaptation: Semantic image segmentation from daytime to nighttime[END_REF], it is impossible to foresee all potential conditions that might occur "in the wild"". Some authors previously proposed a dataset of road scenes under fog captured by polarimetric cameras to model real-world conditions of autonomous cars [START_REF] Blin | The polarlitis dataset: Road scenes under fog[END_REF]. But later [START_REF] Michaelis | Benchmarking robustness in object detection: Autonomous driving when winter is coming[END_REF] extended this idea, by proposing several datasets containing common corruptions specifically designed for object recognition tasks (autonomous cars applications included). By setting these object detection common corruptions, Michaelis et al. of [START_REF] Michaelis | Benchmarking robustness in object detection: Autonomous driving when winter is coming[END_REF] allow benchmarking any type of object detection system within autonomous cars, which can be composed solely of one deep learning model or a deep pipeline of small unit model in the shallow learning spirit. Along with the common corruptions, [START_REF] Michaelis | Benchmarking robustness in object detection: Autonomous driving when winter is coming[END_REF] allows the building of more robust classifiers by optimizing them over these common corruptions, which is in the way of improving the classifiers' robustness by modeling more concrete real-world adversarial perturbations. In the same spirit [START_REF] Dong | Benchmarking robustness of 3d object detection to common corruptions[END_REF] proposed to establish a set of 27 common corruptions specifically suited to 3D object detection for both LiDAR (Light Detection and Ranging) and camera inputs considering real-world driving scenarios. These common corruptions are illustrated in Figure 1.19.

Figure 1.19: Overview of 27 corruptions for 3D object detection proposed by [START_REF] Dong | Benchmarking robustness of 3d object detection to common corruptions[END_REF]. See that the common corruptions are categorized into weather, sensor, motion, object, and alignment and that some corruptions are effective for one modality, while others are applied to both. Source: Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving [START_REF] Dong | Benchmarking robustness of 3d object detection to common corruptions[END_REF].

By synthesizing these corruptions on public datasets that are KITTI [START_REF] Geiger | Are we ready for autonomous driving? the kitti vision benchmark suite[END_REF], nuScenes [START_REF] Caesar | nuscenes: A multimodal dataset for autonomous driving[END_REF], Waymo [START_REF] Sun | Scalability in perception for autonomous driving: Waymo open dataset[END_REF], authors establish three corruption robustness benchmarks namely KITTI-C, nuScenes-C, and Waymo-C. Authors of [START_REF] Dong | Benchmarking robustness of 3d object detection to common corruptions[END_REF] also conducted extensive experiments of state-of-the-art 3D object recognition. Authors found that camera-only models are more easily affected by common corruptions, demonstrating the indispensability of LiDAR point clouds for reliable 3D detection or the necessity of developing more robust camera-only models. Overall, the different common corruptions propositions share the same goal, evaluating the robustness of deep models (classifiers and object detectors) on a more concrete basis.

These perturbations are interesting for two reasons. First, the classifiers' robustness is evaluated on a more concrete basis as the common corruptions are perturbations that can happen in a real-world application therefore, they are much more interesting from an application standpoint and can be understood by the non-experts.

Secondly, these perturbations are set in stone and are the same as every original example, making them example-agnostic like the pseudo-random universal adversarial perturbations (reviewed in Section 1.2.4). This universality property is desired as it exploits the flaws and weaknesses of classifiers that are always true for every example including the future unseen ones. Exploiting these flaws is of the utmost importance as they will likely remain true when releasing the classifier in production on real-world data examples. Universality is detailed in Section 1.2.2. Common corruptions are a first attempt to compute more realistic adversarial noises ω, by constraining the space Ω they live in.

Some new work has been proposed trying to learn real-world perturbations [START_REF] Gowal | Achieving robustness in the wild via adversarial mixing with disentangled representations[END_REF]. In AdvMix [START_REF] Gowal | Achieving robustness in the wild via adversarial mixing with disentangled representations[END_REF], authors propose to consider as real-world perturbations, the disentangled latent representations computed by a StyleGAN [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF] generator (the GAN framework is detailed in Section 1.1.4). Even though the presented results seem appealing, the problem with these "optimized real-world" perturbations is that they are specific to a s generator, a target classifier, and the dataset used to optimize these perturbations. Compared to the common corruptions, which are real-world perturbations, this proposition is still a step away from computing the real-world robustness of a classifier. Besides these optimized perturbations are not controlled in magnitude which makes impossible any comparison with other adversarial attacks. Indeed, authors do not compare their performances with other adversarial attacks but instead highlight the use of their optimized perturbation within an adversarial defense mechanism (more precisely defined in Section 1.2.5).

As we saw, the definition of the searching space Ω of the adversarial noises is primordial before looking for adversarial examples. Depending on the given constraints and goal, the definition of Ω can be significantly different. In our contributions, we mainly focused on pseudo-random adversarial examples, computed with the second relaxation problem therefore, we used Ω p , under which we added some constraints that will be detailed in Chapter 2.

Adversarial properties

So far, we have seen adversarial attacks as functions of a classifier f and an original example x, but other adversarial example generations exist that either are not restricted to a specific original example x or to a single classifier f .

Universality

Universal adversarial perturbations are adversarial perturbations ω that respect the conditions of the adversarial example (1.15) applied to multiple original examples,

argmax k f k (x (i ) + ω) = argmax k f k (x (i ) ) ∀i ∈ I, (1.21a) dist(x (i ) , x (i ) ) ≤ ∀i ∈ I, (1.21b) x (i ) ∈ X ∀i ∈ I, (1.21c) ω ∈ Ω, (1.21d)
with I an index set.

The more universal the adversarial perturbation is, the more harmful it is. Having a perfectly universal adversarial perturbation signifies that applied to every example, the classifier f cannot ever recover the right prediction for all possible original examples. To some extent, such universal perturbation is the most harmful perturbation possible. Universality is an important property of adversarial examples, that should be included in the performance evaluation of an adversarial attack generation.

However, crafting an adversarial perturbation that is valid for all the original examples is too complicated therefore, the problem is relaxed by lowering the number of points for which the adversarial perturbation is valid. Universal adversarial perturbation generation algorithms are called universal adversarial attacks, we review some of the most important in Section 1.2.4.

Transferability

A transferable adversarial perturbation is an adversarial perturbation ω that respects all the adversarial examples conditions applied to every classifier,

argmax k f (i ) k (x ) = argmax k f (i ) k (x) ∀i ∈ I, (1.22a) dist(x, x ) ≤ , (1.22b) x ∈ X , (1.22c) ω ∈ Ω, (1.22d)
with I an index set.

Crafting such transferable adversarial perturbation seems very challenging. In practice, when evaluating the performances of an adversarial attack, we craft all the possible adversarial perturbations on a given classifier f and empirically see if the perturbations remain adversarial for other classifiers g , making the adversarial attack more or less transferable.

The transferability property is also not to be neglected when evaluating the performance of an adversarial attack. If one can craft perfectly transferable adversarial perturbations, all life-atrisk applications relying on classifiers are put at risk, as the perturbation remains adversarial to any classifier. Transferability must also be considered when evaluating the performance of an adversarial attack as it reveals some weaknesses/leaks of the classifier remaining true to every original example and therefore, should be one of the first things to consider when evaluating its robustness.

Both the universality and transferability properties of adversarial examples are important to consider and aim for.

Adversarial metrics

We saw that different optimization problems were proposed to craft adversarial examples, implying different visions and different goals with adversarial examples. Depending on the chosen adversarial problem and who solves it, the most relevant performance metric differs and may evaluate only some aspects of the adversarial perturbations. In this section, we explain in detail where the different adversarial metrics come from, what they are measuring, and to whom they are relevant.

From an attacker's point of view

From an attacker's point of view, the adversarial metric that matters the most is the number of data points for which the classifier is fooled, that is maximizing the probability of finding an adversarial example, max Pr (x,y)∈D x ∈ B(x, ), argmax

k f (x ) = argmax k f (x) (1.23)
with B(x, ), the p norm ball of radius centered on x. Empirically this probability measure is estimated with the fooling rate (FR) defined as

FR( f , D) = 1 N N i =1 1{argmax k f k (x (i ) ) = argmax k f k (x (i ) )}, (1.24)
with

x (i ) = x (i ) + ω x , ω x ∈ Ω .
This metric is mainly suited to the optimization of the second relaxation problem which allows adversarial perturbations to live in an ball ω ∈ Ω . The FR metric doesn't evaluate the performance of the adversarial attack over a single point x but over a whole set of points D = {x (i ) ,

y (i ) } N i =1
which typically consist of the test dataset, the classifier has not been trained on.

However, this metric doesn't take into account the performances of f on clean data. Indeed, if the FR only mattered then a constant classifier would be the most robust to adversarial attacks, which is in practice a bad classifier.

From a defender's point of view

While the FR metric evaluates the performances of the adversarial attack, the robustness of a classifier is measured with another metric. From a defender's point of view when building a classifier f , the goal is to keep a high classification accuracy even though being perturbed by adversarial examples. It leads to a slight modification of the conditions for x to be an adversarial example of x. It must originally be well classified, argmax k ŷ = y (see [START_REF] Yang | A closer look at accuracy vs. robustness[END_REF] and [START_REF] Wang | Analyzing the robustness of nearest neighbors to adversarial examples[END_REF] for more details)

       argmax k f k (x + ω x ) = argmax k f k (x) = y, ω x ∈ Ω , x ∈ X . (1.25)
This point of view leads to consider the original performance of the classifier, that is SA the Standard Accuracy

SA( f , D) = 1 N N i =1 1{argmax k f k (x (i ) ) = y (i ) }, (1.26)
as the empirical counterpart of the expected loss.

Then, the robustness of the classifier (opposite to the efficiency of the adversarial attack) can be measured as the number of well classified clean examples for which the adversarial attack could not find an adversarial example in a ball around them. This quantity is estimated by the Robust Accuracy RA as the empirical counterpart of the astuteness as defined in [START_REF] Wang | Analyzing the robustness of nearest neighbors to adversarial examples[END_REF],

RA( f , D) = 1 N N i =1 1{A {(x (i ) ,y (i ) )} = ∅}, (1.27) with A {(x (i ) ,y (i ) )} = {(x (i ) , y (i ) )} the set of adversarial examples associated with (x (i ) , y (i ) ). It is clear that RA( f , D) ≤ SA( f , D)
, the equality being reached when the adversarial attack is completely inefficient. Therefore, the lower RA is, the more efficient the adversarial attack is.

Like the FR metric, the RA metric evaluates the classifier's robustness not on a single point but over a whole set of points D, typically the test set that has not been used for training the classifier.

A fair evaluation of the adversarial examples generation

Yet, the latter metric is questioned by Lorenz et al. [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF], who emphasizes the importance of taking into account the ability to easily detect the generated adversarial examples. They suggest using an adversarial example detector g plugged before the classifier f when computing the RA. By doing so, Lorenz et al. [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF] showed that state-of-the-art adversarial attacks produced easily detectable adversarial examples, making the attacks inefficient when a detector is used. Therefore, they suggest relying on another metric called the Attack Success Rate under Defense (ASRD) (equation 2 of [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF]), we rename Robust Accuracy Under Defense (RAUD) for clarity purposes, which stands as a natural performance metric from both the attacker's and defender's points of view. The RAUD is defined as,

RAUD( f , D) = 1 N N i =1 1{∀x ∈ A {(x (i ) ,y (i ) )} , g (x ) = 0}, (1.28)
where g : R P → {0, 1} is an adversarial example detector with 0 as the clean example label.

According to the RAUD, an adversarial example is considered when it fools the classifier f and the detection from d . By doing so, the RAUD evaluates the transferability of the produced adversarial example. The RAUD not only is fair for both the attacker and the defender but also evaluates the transferability of the adversarial attack allowing it to more precisely measure the harm of the adversarial attack.

Pseudo-random adversarial attacks

Other than pseudo-random adversarial perturbations, we previously saw that other families of adversarial perturbations exist such as adversarial patch (reviewed in Section 1.2.1 or common corruptions (reviewed in Section 1.2.1) however, in our contributions we mainly studied the pseudorandom adversarial perturbations therefore, here we define them in depth.

Pseudo-random adversarial attacks aim to generate additive adversarial perturbation ω ∈ Ω = R P that targets every pixel of the image without any restriction. Some of the pseudo-random adversarial attacks rely on the second relaxation optimization (see Section 1.2.1) to craft adversarial examples, thus setting Ω = {ω ∈ R P , ||ω x || p ≤ }. It makes it easier to fool classifiers and thus evaluates their robustness more precisely, but computes adversarial perturbations that are less realistic and far from real-world harm.

Table 1.1 summarizes all the attacks we review and highlights their differences. [START_REF] Silva | Opportunities and challenges in deep learning adversarial robustness: A survey[END_REF], [START_REF] Bai | Recent advances in adversarial training for adversarial robustness[END_REF], [START_REF] Zhang | A survey on universal adversarial attack[END_REF] and [2].

Attacks Knowledge Goal p -norm Optimization L-BFGS [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] White-box Specific 2 One-shot FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] White-box Specific ∞ One-shot BIM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] White-box Specific

∞ Iterative PGD [102] White-box Specific 2 , ∞ Iterative DeepFool [113] White-box Specific 2 Iterative C & W [24]
White-box Specific 0 , 2 , ∞ Iterative Autoattack [START_REF] Croce | Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[END_REF] White-box Specific 2 , ∞ Iterative Local Substitute Model Attack [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF] Black-box Specific 2 , ∞ One-shot MIM [START_REF] Dong | Boosting adversarial attacks with momentum[END_REF] Black-box Specific ∞ Iterative Zoo [START_REF] Chen | Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[END_REF] Black-box Specific p Iterative Boundary Attack [START_REF] Croce | Minimally distorted adversarial examples with a fast adaptive boundary attack[END_REF] Black-box Specific 1 , 2 , ∞ Iterative UAP [START_REF] Moosavi-Dezfooli | Universal adversarial perturbations[END_REF] White-box Universal 2 , ∞ Iterative Fast-UAP [START_REF] Dai | Fast-uap: An algorithm for expediting universal adversarial perturbation generation using the orientations of perturbation vectors[END_REF] White-box Universal 2 , ∞ Iterative UAP-PGD [START_REF] Shafahi | Universal adversarial training[END_REF] White-box Universal 2 , ∞ Iterative CW-UAP [START_REF] Benz | Universal adversarial training with class-wise perturbations[END_REF] White-box Universal 2 , ∞ Iterative NAG Attack [START_REF] Mopuri | Nag: Network for adversary generation[END_REF] Black-box Universal ∞ One-shot JSMA [START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF] White-box Specific 2 Iterative Adversarial Patch [START_REF] Brown | Adversarial patch[END_REF] White-box Specific

∞

Iterative

The very first work proposing a pseudo-random adversarial attack is [START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF] which defined the specification of the adversary, its goal, its knowledge, and its capabilities. Every one of these specifics will give birth to a sub-family of pseudo-random adversarial attacks.

An adversary's objective can be a misclassification loss function of one or multiple examples. In (1.15b) we set the misclassification goal as max

ω x ∈Ω (max k =k f k (x + ω x )) -f k (x + ω x )
because it is the direct objective of maximizing the FR, but in principle, it could be any loss function enforcing the misclassification of the adversarial example.

The adversary knowledge can include the training set, or part of it, used to build the classifier, the feature representation of each example, the type of the classifier's learning algorithm and the form of the classifier's decision function, the classifier itself, or the classifier's feedback. The adversary's capabilities are limited to modifications only of test data. Adversaries are not allowed to modify the training data as it would be another problem to let adversaries penetrate the training of the classifier. Hence, the adversary can only operate modifications to the test examples, modifications of the test examples' features, or independent modifications to specific features (the semantics of the examples may dictate that certain features are interdependent).

Differences in the adversary's knowledge separate adversarial attacks into two categories: Whitebox and Black-box attacks and differences in the adversary's goal separate adversarial attacks into two categories: Specific attacks and Universal attacks.

All these differences are highlighted in Table 1.1.

White-box adversarial attacks

A white box adversarial attack is a type of attack on a classifier in which the adversary has complete knowledge of the classifier's architecture, parameters, and inputs. In other words, the adversary has full access to the "white box" of the classifier, hence the name. White-box attacks are predominately gradient-based adversarial attacks since they usually rely on the gradient of the adversary's goal with respect to the classifier's parameters to optimize it. This is the first type of proposed adversarial attack. The very first adversarial attacks date back to a few years ago with [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] who first discovered the vulnerabilities of classifiers to adversarial perturbations by solving the following optimization problem min

ω x ∈Ω ∥ ω x ∥ 2 s.t. argmax k f k (x + ω x ) = t , (x + ω x ) ∈ X , (1.29)
with t a target label. With this problem being tough to solve, authors propose to approximate a solution using the Limited Memory Broyden Fletcher Goldfarb Shanno (L-BFGS) algorithm [START_REF] Liu | On the limited memory BFGS method for large scale optimization[END_REF], giving the attack its name, the L-BFGS attack. To solve it, authors solve instead a relaxed version, including Lagrange multiplier c, min

ω x ∈Ω c ∥ ω x ∥ 2 + L(x + ω x , t ) s.t. (x + ω x ) ∈ X , (1.30)
with L(•,•) as the classifier's loss, and c the dual variable found performing a line-search to get the minimum c > 0 for which an adversarial example x can be found. The authors showed that their attack transfers well between different classifiers. The transfer of an adversarial attack refers to the application of an adversarial example ω x targeted to fool the original classifier f , on another classifier g (see Section 1.2.2 for more details). This intriguing transfer property of adversarial attacks attracted lots of interest from researchers in the subsequent years and this work.

Solving (1.30) for a large number of images can often be expensive and computationally disastrous therefore, to lighten the complexity of the problem the Fast Gradient Sign Method (FGSM) adversarial attack [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] has been proposed to compute adversarial perturbations more efficiently, as a solution of

ω x = γsign(∇ x L(x, y)). (1.31)
The FGSM adversarial attack is a one-step gradient-based method to compute adversarial examples efficiently. The attack focuses on the "efficiency" of the perturbations rather than achieving high fooling rates. The FGSM attack is among the most influential attacks in the existing literature, it inspired many improved/tweaked versions such as FGVM [START_REF] Rozsa | Adversarial diversity and hard positive generation[END_REF], P-FGVM [START_REF] Chatzikyriakidis | Adversarial face de-identification[END_REF], I-FGSM [START_REF] Dong | Boosting adversarial attacks with momentum[END_REF], MI-FGSM [START_REF] Xie | Improving transferability of adversarial examples with input diversity[END_REF], DI 2 -FGSM [START_REF] Dong | Boosting adversarial attacks with momentum[END_REF], M-DI 2 -FGSM [START_REF] Dong | Boosting adversarial attacks with momentum[END_REF], and many others. The FGSM attack gave rise to one of the first adversarial defense mechanisms, a new way to train classifiers more robustly to adversarial examples, called adversarial training. An overview of adversarial defenses is given in Section 1.2.5.

By being a one-shot adversarial attack, the FGSM is limited in the optimization of the adversarial examples. To enhance adversarial example optimization, the Basic Iterative Method (BIM) [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] has been proposed as an iterative version of FGSM, computing adversarial examples by repeating

ω i +1 x = clip {ω i x + γsign(L(x, y))}, (1.32)
where i denotes the i th iteration, clip performing clipping at . Adversarial performances were greatly enhanced and very close to BIM, is the Projected Gradient Descent (PGD) attack [102], which is, as of today, widely considered one of the most powerful attacks in the literature. PGD is a standard optimization technique that projects gradients to the p norm ball of radius ,

ω i +1 x = Proj p {ω i x + γsign(L(x, y))}, (1.33)
with the p norm being either the 2 or the ∞ norm. Even though the PGD attack seems very close to the BIM attack, the proposed Projection operator is a great improvement compared to BIM. It allows us to select another similarity metric than the p norm, which can make the adversarial perturbation very different. We present in Section 1.2.6, that there currently is no consensus on which p norm should be used and whether even using an p norm is a good choice. Instead of restricting the adversarial perturbations to be in an norm ball, the DeepFool attack [113] aims at minimizing the norm of the adversarial perturbation by resorting to the first presented adversarial example relaxation problem, min

ω x ∈Ω ∥ ω ∥ 2 s.t. argmax k f k (x + ω x ) = y.
(

The main motivation behind resorting to this relaxation was to effectively quantify the adversarial robustness of the target classifier f . The proposed adversarial attack is an iterative algorithm linearizing the class boundaries around the current image to form a convex polyhedron and push the image toward the closest hyperplane to change the class label. Like the FGSM authors, Deepfool authors also proposed a defense mechanism built upon their proposed attack.

Considering the weaknesses of deep classifiers against adversarial attacks [START_REF] Szegedy | Intriguing properties of neural networks[END_REF], researchers proposed defense mechanisms to protect against such harm. One of the first defense mechanisms proposed is Defensive distillation [START_REF] Papernot | Distillation as a defense to adversarial perturbations against deep neural networks[END_REF] (formalized in Section 1.2.5), however Carlini & Wagner (C&W) [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] proposed a set of adversarial attacks completely breaking the defensive distillation defense. To compute the adversarial perturbations, the (C&W) attack solves the following optimisation problem, min

ω x ∈Ω ∥ ω x ∥ p +c.l (x + ω x ) s.t. x + ω ∈ X , (1.35)
with c an hyper-parameter and l a function satisfying the misclassification argmax k f k (x ) = y.

Multiple l functions were proposed and discussed by the authors. All of the proposed adversarial attacks rely on the second presented relaxation problem therefore, all adversarial perturbations live in the p norm ball of radius . Even though minimizing the norm of the adversarial perturbation is part of the objective, all of their presented algorithms stop the optimization when all the adversarial conditions are met (1.19). It makes the attacks very powerful and allows us to compare all the adversarial attacks as they are given the same adversarial budget.

As the work wasn't innovative enough, authors propose multiple p norms as magnitude metrics of the adversarial perturbations (especially 2 norm associated with = 0.5 and ∞ norm associated with = 8/255), which impose future adversarial attacks to allow using different similarity metrics to find adversarial examples.

Proposing multiple p norm constraints is an improvement in the computer vision community where there was no consensus on which norm should be used to compute distances, highlighted in Table 1.1.

Even though its computation cost is very high, the C&W attack is still considered one of the strongest adversarial attacks.

Finally, as of today, state-of-the-art performances are achieved by [START_REF] Croce | Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[END_REF] who proposed Autoattack an ensemble of diverse parameter-free attacks. An Autoattack instance encapsulates two versions of improved PGD adversarial attack plus two other attacks previously proposed by the same authors (FAB [START_REF] Croce | Minimally distorted adversarial examples with a fast adaptive boundary attack[END_REF] and SquareAttack [START_REF] Andriushchenko | Square attack: a query-efficient black-box adversarial attack via random search[END_REF]). An adversarial example computed with Autoattack is an adversarial example computed from one of these four adversarial attack roots, which naturally dope the performances of the overall Autoattack instance.

White-box adversarial attacks make the assumption the adversary has complete knowledge about the target classifier, including its architecture, parameters, and training data which makes the evaluation of the classifier's robustness very precise. While white-box attacks can be very effective in fooling the target classifier, they are not without their flaws. First, the assumption that complete knowledge of the adversary is often not realistic to real-world scenarios. In practice, adversaries usually have limited access to the target classifier and sometimes may not be able to query it, making white box attacks irrealistic. White box attacks produce adversarial examples targeting one classifier f but most likely do not transfer well to other classifiers. It is again a step away from the real-world scenarios. Finally, as white box attacks are designed to exploit specific weaknesses of a target classifier, they are susceptible to countermeasures and defensive mechanisms. It has been shown in [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF] that current state-of-the-art white box attacks are made harmless under tiny defense mechanisms.

Black-box adversarial attacks

In contrast to white-box attacks, black-box attacks do not access the classifier's internal parameters. The attack algorithm relies on alternative information like query access to the classifier [START_REF] Chen | Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[END_REF], knowing the training dataset [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF], or transferring adversarial examples from one trained classifier to another [START_REF] Zhou | Dast: Data-free substitute training for adversarial attacks[END_REF]. Notice that a black-box attacker represents a more practical adversary [START_REF] Chen | Hopskipjumpattack: A query-efficient decision-based attack[END_REF] and one that is closer to real-world scenarios [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF].

The very first adversarial black-box attacks were proposed in [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF], [START_REF] Chen | Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[END_REF] and [START_REF] Brendel | Decision-based adversarial attacks: Reliable attacks against black-box machine learning models[END_REF]. Each attack computes adversarial examples from a different source of information therefore, these three works gave rise to three main families of black-box attacks: transfer-based, score-based, and decisionbased black-box attacks.

Transfer based attacks

In transfer-based attacks, the adversary is allowed to query the target classifier and/or access some of the target classifier's training dataset. The adversary then uses this information to create a synthetic classifier from which the attacker generates adversarial examples using a white-box attack. This white-box adversarial example from the synthetic classifier is then transferred to the target classifier, hoping it remains adversarial (respecting conditions of (1.19)).

One of the first black-box adversarial attacks to be proposed is called the local substitute model attack [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF]. The idea of the local substitute model attack is to create a copy-cat classifier from which to generate an adversarial example in the hope that it will remain adversarial to the target classifier. The transfer property of adversarial examples arose from this work. In this paper, authors allowed the attacker to access a percentage of the training dataset

D = {(x (i ) , y (i ) )} k i =1 , k N,
and to query the target classifier f . They proposed to create a synthetic classifier g optimized to mimic the prediction of the target classifier f on the given fragment of the original dataset. Therefore the synthetic classifier g is not optimized to predict the original labels {y

(i ) )} k i =1 but to predict the target classifier's prediction {argmax c f c (x) (i ) )} k i =1 .
The copy-cat classifier g is optimized using, min

g k i =1 L( f (x (i ) ), g (x (i ) )), (1.36)
with L a classification loss (e.g. cross-entropy as seen in Section 1.1.2). Once the synthetic classifier g is optimized, authors proposed to compute adversarial examples targeting g using the FGSM attack and hope those adversarial examples transfer to the target classifier f . Authors found that by using only 0.3% of D, an adversary can fool any target classifier. However, this last setting is questioned by researchers who proposed the mixed [START_REF] Mahmood | Buzz: Buffer zones for defending adversarial examples in image classification[END_REF] or adaptive black-box attack [START_REF] Mahmood | Beware the black-box: On the robustness of recent defenses to adversarial examples[END_REF]. While in the original attack, 0.3% of the training data is used, the adaptive version increases the knowledge of the adversary by using anywhere from 1% to 100% of the original training dataset. Besides, they used newer and fancier white-box attacks than FGSM having better transferability. Authors claimed they found the most effective white-box attack to be the Momentum Iterative Method (MIM) [START_REF] Dong | Boosting adversarial attacks with momentum[END_REF] to transfer the adversarial examples to f . By integrating a momentum term into the iterative process of the adversarial example generation, the MIM attack stabilizes from each iteration update and escapes from poor local maxima during the iterations, resulting in more transferable adversarial examples. To further improve the transfer success rates for black-box attacks, MIM is applied to an ensemble of synthetic classifiers, maximizing the chances that the computed adversarial examples Indeed, transfer to the target classifier f . The MIM iterative optimization for the adversarial examples x (i ) takes the form of,

v t +1 = µv t + ∇ x L(g (x), y) ||∇ x L(g (x), y)|| 1 , x (i ) t +1 = x (i ) t + γsign(v t +1 ), (1.37)
with v t acting as a velocity vector of the gradient optimization at iteration t , L a loss function, µ a decay factor, and γ the learning rate.

The two previous black-box attacks require to model of at least one substitute classifier g however, a substitute classifier can be difficult to obtain, particularly if a considerable amount of data labeled by the target classifier is needed.

Score based attacks

Another direction taken by the adversarial black-box community is the score-based attacks.

Score-based adversarial attacks allow the attacker to query the defense with input x and receive the corresponding pre-softmax logits f 1 (x), . . . , f C (x) or the post-softmax probability outputs ŷ1 (x), . . . , ŷC (x). One of the first score-based adversarial attacks to be proposed is ZOO (Zeroth Order Optimization) [START_REF] Chen | Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models[END_REF] which relies on queries to create adversarial examples. Zeroth order methods are derivative-free optimization methods, where only the zeroth order oracle (the objective function value f (x) at any x) is needed during the optimization process. By evaluating the objective function values at two very close points f (x + αv) and f (x -αv) with a small α, a proper gradient along the direction of the vector v, can be estimated. For a target classifier f and a target label t , ZOO finds adversarial examples by optimizing the C&W proposed objective, with the minimization of the following adversarial loss function l ,

l (x + ω x ) = max max i =t log f i (x + ω x ) -log f t (x + ω x ), -ρ , (1.38)
with ρ > 0 a margin hyper-parameter.

In order to optimize (1.38), ZOO employs a zeroth order optimization and approximate the gradient

δ f (x+ω x ) ω i
x by using the symmetric difference quotient [START_REF] Lax | Calculus with applications[END_REF],

δ f (x + ω x ) ω i x ≈ f (x + ω x + αe i ) -f (x + ω x -αe i ) 2α , (1.39)
with α a small constant and e i a standard basis vector with only the i -th component as 1. Estimating the gradient this way requires 2 P queries of f which could be inefficient. Decision based attacks Decision-based adversarial attacks go one step further into the blindness of the adversary. In such an attack algorithm, the adversary is allowed to query the defense with input x but receives only the defense's final predicted label argmax k ŷk . In contrast to score-based attacks, the adversary does not receive any probabilistic or logit outputs from the defense, only the predicted label.

The first prominent decision-based attack is the Boundary Attack [START_REF] Brendel | Decision-based adversarial attacks: Reliable attacks against black-box machine learning models[END_REF]. The basic intuition behind the Boundary Attack is, for an original example x (i ) , to randomly initialize its adversarial example, and then perform a random walk along the classifier's decision boundary between the adversarial label and the non-adversarial labels region such that the adversarial example stays in the adversarial label region while minimizing the distance from its original counterpart. In other words, Boundary Attack performs rejection sampling with a suitable distribution P centered on an adversarial example at each step to find progressively smaller adversarial perturbations according to a given adversarial distance dist.

Algorithm 1 Boundary Attack

Require: Original example x; classifier's label prediction F; MAX-STEPS, Adversarial distance dist(•, •)

1: k = 0 2: x 0 = U(0,1) s.t. F(x k-1 ) = F(x) 3: for k < MAX-STEPS do 4:
draw random perturbation from distribution centered on

x k-1 , µ k ∼ P(x k-1 ) 5: if dist(x k-1 + µ k ) = F(x) and dist(x k-1 + µ k , x) < F(x k-1 , x) then 6:
x

k = x k-1 + µ k 7:
else 8:

x k = x k+1 9:
end if 10: end for

11: return Adversarial example x k minimizing dist(x , x)
In their presented work, authors discuss the definition of P. Being the first of its kind, the Boundary Attack is a quite simple optimization and much stronger decision-base black-box attacks have been proposed such as the Geometric decision-based attacks (GeoDA) [START_REF] Rahmati | Geoda: a geometric framework for black-box adversarial attacks[END_REF]. GeoDA is a subset of decision-based black-box attacks that can achieve high attack success rates while requiring a smaller number of queries.

Universal adversarial attacks

So far, every adversarial attack presented consists of finding one adversarial perturbation ω x (i ) associated with one original example x (i ) producing its paired adversarial example x (i ) . This goal makes these attacks specific as they find specific adversarial perturbations to each original example. Other adversarial attacks have been proposed, aiming to compute one single adversarial perturbation ω capable of producing adversarial examples for a batch of multiple original examples. Such adversarial perturbation is, therefore, universal (as we saw in Section 1.2.2) to the original example, also called example-agnostic. Adversarial attacks producing these universal adversarial perturbations are called universal adversarial attacks. The existence of universal adversarial perturbations (UAP) to fool deep classifiers for most images has first been demonstrated in [START_REF] Moosavi-Dezfooli | Universal adversarial perturbations[END_REF]. The authors proposed the first universal attack coined UAP. Given a set of labeled images D = {(x (i ) , y (i ) )} N i =1 with x (i ) ∈ X ⊂ R P , and a classifier f , the overall UAP objective is to find a single perturbation vector ω ∈ Ω = R P , such that the classifier f is fooled for most encountered images. More formally, UAP seeks a ω such that, max

ω N i =1 1 . argmax k f k (x (i ) ) = argmax k f k (x (i ) + ω) such that x (i ) + ω ∈ X ,
and ||ω|| p ≤ .

(1.40)

Obviously, from this objective, the main focus of universal adversarial perturbations is not the quality of the adversarial noises, but rather the quantity of examples on which the classifier is fooled. No universality metric has yet been proposed, therefore, the best adversarial metric is the fooling rate. We previously saw that the choice of the norm p constraining the adversarial perturbation magnitude was not a consensus in the community. UAP set it to ∞ as the adversarial perturbations show interesting patterns, displayed in Figure 1.20. The proposed algorithm computing UAP adversarial perturbations, incrementally accumulates specific DeepFool (1.34) adversarial perturbations over each example separately, hoping that the projection sum of the perturbations in the radius p norm ball will remain adversarial to each specific example. Quite simple, this computation of universal adversarial perturbations has since been improved. Next to the very first universal adversarial attack UAP is, Fast-UAP [START_REF] Dai | Fast-uap: An algorithm for expediting universal adversarial perturbation generation using the orientations of perturbation vectors[END_REF]. This attack follows the UAP procedure but, instead of aggregating all the specific DeepFool perturbations at each iteration, it only adds the perturbation with the closest orientation to the current iterate. Among all the specific DeepFool adversarial perturbations ω DF x (i ) , Fast-UAP adds to the universal perturbation ω only the one with the maximum cosinus to the universal perturbations,

ω t +1 = ω t + max i ∈[1,N] cos(ω t , ω (i ) DF ), (1.41)
with t the optimization iteration.

Elaborating on the previous optimizations procedure, UAP-PGD [START_REF] Shafahi | Universal adversarial training[END_REF] frames the universal perturbation as the solution of the following optimization problem,

max ω∈Ω N i =1 l ( f (x (i ) + ω, y (i ) ) s.t. ||ω|| p ≤ , (1.42)
with l an adversarial loss.

Recently, UAP-PGD has been extended to class-wise UAP (CW-UAP) [START_REF] Benz | Universal adversarial training with class-wise perturbations[END_REF] where a single universal perturbation is built for each class. max

ω k ,k∈Y k∈Y N k i =1 l ( f (x (i ) + ω k , k) s.t. ∀k ∈ Y, ||ω k || p ≤ . (1.43)
The solution amounts to learning multiple independent UAP-PGD perturbations, one for each class. A different algorithm to craft universal adversarial perturbations has been proposed in [START_REF] Khrulkov | Art of singular vectors and universal adversarial perturbations[END_REF]. Their method is based on the computation of the singular vectors of the Jacobian matrices of the classifier's feature maps to obtain universal adversarial perturbations. The proposed approach is much more data efficient, with universal perturbations reaching fooling rates of more than 60% on the ImageNet validation set by using only 64 images to optimize the universal adversarial perturbation.

The idea of the semantic captured within the universal adversarial perturbations is again shown by empirical evidence of [START_REF] Khrulkov | Art of singular vectors and universal adversarial perturbations[END_REF] as Figure 1.21 shows. Indeed, the forms and contours within the perturbations indicate that the universal adversarial perturbations crafted on singular vectors of the hidden layers target the features learned by those layers, such as edges, corners, or rounds. This figure is interesting as it empirically suggest that universal adversarial perturbations might rely on the most meaningful features of the object to fool the classifier. The unrealistic issue of perturbations produced by white-box adversarial attacks would thus be addressed. The new performances shown in [START_REF] Khrulkov | Art of singular vectors and universal adversarial perturbations[END_REF] indicate that, by being universal to the examples a universal perturbation has been crafted on, there are good chances that this very same universal perturbation will remain adversarial to other examples. Therefore, the transferability of universal perturbations is not to be neglected and becomes another metric that must be measured by uni-versal adversarial attacks. The RAUD metric seems to be the most appropriate as it measures both the capacity to fool a targeted classifier and to transfer to other classifiers. The RAUD measures the fooling rate of the adversarial attack on originally well-classified examples, but for adversarial examples also fooling an adversarial example detector, which to some extent measures the transferability of these adversarial examples (see Section 1.2.2 for more details). Unfortunately, the RAUD has only been considered very recently.

Inspired by Generative Adversarial Networks (GAN) framework [START_REF] Goodfellow | Generative adversarial networks[END_REF] (presented in Section 1.1.4), Network for Adversary Generation (NAG) was introduced by [START_REF] Mopuri | Nag: Network for adversary generation[END_REF]. NAG aims to model the distribution of universal adversarial perturbations. Therefore, the authors modify the GAN framework by replacing the discriminator with the (frozen) target classifier and introducing a novel loss to train the generator. The novel loss function L is composed of an adversarial loss l and a diversity loss L d ,

L = l + λ L d , with l = -l og (1 -f y (x + ω)),
and

L d = - B j =1 dist( f i (x ( j ) + ω ( j ) ), f i (x ( j ) + ω (K) )), (1.44)
with λ a hyper-parameter, B the batch size, K ∈ [1, B], K = j a random index, ω ( j ) and ω (K) two generated perturbations, x ( j ) an original example, i the index of the classifier's hidden layer, and dist a distance metric (e.g. the 2 norm). As we previously saw, the adversarial loss l is designed such that the generated perturbations produce adversarial examples fooling the target classifier while the diversity objective encourages the diversity of perturbations by increasing the distance of their feature embeddings predicted by the target classifier.

It is worth noticing that NAG is the first and among the few black-box universal adversarial attacks. The combination of both properties was made possible thanks to the modeling of the adversarial noise space by the generator. It indicates that modeling the adversarial noise space is a good idea as it allows us to explicit some adversarial perturbations constraints, enabling more realistic adversarial examples. Another variant of universal adversarial noise generation is generative adversarial perturbations (GAP) using a generator to craft universal adversarial perturbations [START_REF] Poursaeed | Generative adversarial perturbations[END_REF]. Concurrently, the authors of [START_REF] Hayes | Learning universal adversarial perturbations with generative models[END_REF] also explored the idea of generating adversarial perturbations with a generator network. Overall, the objective of the three last attacks is to train a (nonlinear) generative network that transforms a random pattern to an image-dependant perturbation or universal adversarial perturbation. Unfortunately, all these attacks relied on a non-linear deep neural network to model the adversarial noise space which complicates the analysis of the model thus, preventing us from inspecting and visualizing this adversarial noise space. A more global view of the proposed universal adversarial attacks is given in the survey [START_REF] Zhang | A survey on universal adversarial attack[END_REF], readers are invited to refer to it for a wider understanding of what researchers proposed.

Overall, universal adversarial perturbations remain not enough explored. In this thesis, we bring in a new adversarial attack in the middle of specific and universal adversarial attacks.

Other types of adversarial attacks

Whereas the majority of adversarial attacks focus on perturbing clean images while enforcing the imperceptibility of the perturbation by restricting their 2 or ∞ norms, the Jacobian-based Saliency Map Attack (JSMA) [START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF] and One-pixel attack [START_REF] Su | One pixel attack for fooling deep neural networks[END_REF] deviate from this practice by restricting the perturbations to smaller regions of the image. Opposite to back-propagating the gradient of the adversarial perturbation through the classifier f , JSMA computes the forward gradient of f as,

∇ f (x) = δ f j (x) x i , (1.45)
with j = 1, . . . , C the prediction of f for label j , and x i the i -th composant of x.

Essentially, (1.45) computes the Jacobian of the function learned by the f . The Jacobian matrix is used as a saliency map to select only a few most influential pixels according to the alteration of the model prediction. [START_REF] Su | One pixel attack for fooling deep neural networks[END_REF] demonstrated that an image classifier can even be fooled by restricting the perturbation to a single pixel. The authors also studied the performances of the attack when increasing the number of pixels used for the adversarial example.

Adversarial defenses

Adversarial defenses have been proposed to protect against adversarial attacks harm. There are three main ways to defend against adversarial examples. The first way is to robustly optimize classifiers so they are more robustly trained for adversarial example danger. The second way is, given a classifier, to use an adversarial examples detector protecting the classifier beforehand. Finally, the third option is to use gradient masking [START_REF] Silva | Opportunities and challenges in deep learning adversarial robustness: A survey[END_REF] techniques, complicating the adversarial example creation, or at least complicating its functioning.

Robust optimization

Robust

f - C c=1 o c log( f c (x )), (1.46)
with x ∈ B(x, ), a valid or adversarial example in the ball centered on x.

Overall, the goal of a robust optimization is to expand the decision boundaries of the classifier, to make it more robust, while still maintaining good classification performances. New adversarial training methods have been proposed with more sophisticated approaches, see [START_REF] Bai | Recent advances in adversarial training for adversarial robustness[END_REF] for a survey.

Adversarial examples detection

Another way to defend against adversarial examples is to recognize them before performing classification. This way if adversarial examples are recognized, they are ignored, preventing possible misclassification. The efficiency of adversarial examples detector is highlighted in [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF] which proposes to use them not only as a defensive module but as an adversarial attack evaluation metric (see 1.2.3 for more details).

Gradient masking techniques

The last mainly used defense mechanism is gradient masking. The core idea of gradient masking is to build classifiers such that the adversarial examples crafting becomes troublesome. It can be done by using classifiers that are not differentiable, in that case, adversarial example generation via gradient computation is impossible. Another way to perturb the adversarial example generation is to add some tricks inside the classifier's rugs to obfuscate the gradients [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF] (e.g. adding noise between each hidden layer). The noise injected in the classifier's internal parameters can, always be injected, or only injected at inference time [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF].

Along with the publication of new, more powerful adversarial attacks, researchers also proposed new, fancier adversarial defenses. As Figure 1.24 shows, new adversarial defenses are more efficient in building more robust classifiers, but still working to find better ways to build robust classifiers should be a top priority. Readers are invited to refer to [2] for a survey of adversarial defenses.

Figure 1.24: Published adversarial defense's robust accuracy according to the year of publication. Source: www.robustbench.io

No community consensus

On the one hand, common corruptions are too limited in the modeling of harmful examples to reliably estimate the robustness of classifiers and on the other hand, adversarial patches are too practical applications of adversarial examples making too many other adversarial perturbations out of reach when performing a robustness evaluation. We thus consider these two adversarial examples visions out of our scope and purposefully ignore them in our research.

As [START_REF] Sharif | On the suitability of lp-norms for creating and preventing adversarial examples[END_REF] highlights, p norm-bounded adversarial noises imply that the similarity between objects, for instance, images, can be efficiently measured by an p norm. However, two images don't need to be close to each other as measured by an p norm to be perceptually similar. We believe that Overall, the universal adversarial perturbations are free to fully explore the radius p norm ball around each example, which is unrealistic, but has the constraint to be universal to most of the examples, which is in the way to compute a realistic robustness evaluation. We believe universal adversarial perturbations act as a good trade-off between both worlds, allowing us to take advantage of the p norm-bound computations and to draw conclusions that are useful for real-world applications.

Chapter 2

Adversarial attack through dictionary learning

In this chapter, we present the main contribution of this thesis, which is the modeling of adversarial noise using a dictionary learning framework. First, we present the goal and its motivations. Then, we formalize the dictionary learning problem of the adversarial noise space. Next, we present the three relaxation optimizations we proposed, which are an ADiL (Adversarial Dictionary Learning) proximal gradients solution, an ADiL projected gradient solution, and a LIMANS (Linear Modeling of the Adversarial Noise Space) stochastic gradient solution. For each, we present the proposed optimization problem, give algorithmic solutions, and present its empirical results. The third relaxation offers more results as it is the one we ended up with after reviewing the results of the two previous ones. Finally, we conclude the overall idea and propose its extensions for future promising research.

Motivations

Among the proposed p norm-bounded adversarial attacks, on the one hand, specific adversarial attacks are the most efficient but the produced adversarial examples struggle to transfer to other examples, on the other hand, universal adversarial attacks do produce adversarial noises suited to multiple examples but they suffer from lower attack success. To reconcile universality and efficiency, we proposed to linearly model the adversarial noise space Ω through as set M adversarial directions D ∈ R M × P , allowing to frame any specific adversarial perturbation as a linear combination of the M universal adversarial directions. The idea behind an adversarial attack through dictionary learning is to, model the adversarial noise space Ω with a dictionary and then, to look for specific adversarial noises corresponding to the coding vectors of the dictionary within its modeled space.

By modeling the adversarial noise space Ω, we can find adversarial parameters that are exampleagnostic and therefore contain all information needed to fool the classifier on any examples. While still modeling the overall space, we still have the possibility, for any original examples x, to navigate within that space to craft an efficient adversarial noise ω x suited to this examples, to ensure good adversarial performances. This proposed framework then bridges the gap between universal and specific adversarial attacks, hoping to retain the good adversarial performances of specific attacks, and the transfer property of universal attacks.

Learning the adversarial space

Some studies on the overall structure of deep classifiers tried to clarify and explain the rise and origin of adversarial attacks. Research in [START_REF] Fawzi | Empirical study of the topology and geometry of deep networks[END_REF] illustrated the decision boundaries of a classifier are in the vicinity of examples and flat in most the directions. More recently, [START_REF] Li | Defense against adversarial attacks via controlling gradient leaking on embedded manifolds[END_REF] claimed that adversarial noise is caused by gradient leakage and the adversarial directions are perpendicular to the classifier boundaries. Baluja and Fischer [START_REF] Baluja | Adversarial transformation networks: Learning to generate adversarial examples[END_REF] trained feed-forward neural networks to generate adversarial examples against other targeted networks or sets of networks. The trained models were termed Adversarial Transformation Networks (ATNs). In the same direction, Hayes and Danezis [START_REF] Hayes | Machine learning as an adversarial service: Learning blackbox adversarial examples[END_REF] also used an attacker neural network to learn adversarial examples for black-box attacks. In some sense, generating adversarial examples with a neural network can be seen as modeling the adversarial example space by this neural network. Tabacof and Valle [155] empirically demonstrated that adversarial examples appear in large regions of the pixel space, which is in line with the similar claim in [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF]. Then, [START_REF] Tramèr | The space of transferable adversarial examples[END_REF] discovered that the decision boundaries of different classifiers are closed and proposed to establish a transferable subspace of the space across different classifiers. They proposed a method to estimate the dimensionality of the space of the adversarial examples. However, the space is inferred based on the adversarial noise generated by the FGSM method which impacts the precision of the found space. The hypothesis of the transferability depending only on the dimensionality of this space limited its performance on CNN classifiers.

Most of the previous work tried to model the space of adversarial examples, which is quite different than the space of adversarial noises itself.

Moosavi-Dezfooli, Fawzi, Fawzi and Frossard [START_REF] Moosavi-Dezfooli | Universal adversarial perturbations[END_REF] initially argued that universal adversarial perturbations exploit geometric correlations between the decision boundaries induced by the classifiers. Their existence partly owes to a subspace containing normals to the decision boundaries, such that the normals also surround the natural images. In [START_REF] Moosavi-Dezfooli | Robustness of classifiers to universal perturbations: a geometric perspective[END_REF], they built further on their theory and showed the existence of common directions (shared across data points) along which the decision boundary of a classifier can be highly positively curved.

Manifold hypothesis

The manifold hypothesis suggests that high-dimensional data, such as images or text, often lies on or near a lower-dimensional manifold embedded within that high-dimensional space. It implies that data points with similar characteristics or semantic meaning are closer to each other in this lower-dimensional manifold. Autoencoders [START_REF] Badrinarayanan | Segnet: A deep convolutional encoderdecoder architecture for image segmentation[END_REF] can help address the manifold hypothesis problem by learning an efficient data representation that captures the underlying structure or manifold of the data. An autoencoder is a type of neural network that is trained to reconstruct its input data. It consists of two main components: an encoder and a decoder as shown in Figure 2.1. During training, the autoencoder learns to encode the input data into a compressed representation in the latent space, while the decoder learns to reconstruct the original input from this representation. By doing so, the autoencoder is forced to capture the essential features of the data and discard the noise or irrelevant information. After training, the autoencoder effectively modeled the underlying structure or manifold of the data. See that the code outputted by the encoder is of lower dimension than the images but allows for full reconstruction of the image, therefore containing all the information within the image. Source: Autoencoders [START_REF] Bank | Autoencoders[END_REF].

Using an autoencoder for adversarial examples has already been proposed by [START_REF] Makhzani | Adversarial autoencoders[END_REF] in 2015. Although quite compelling, their approach mainly relied on a GAN generator, which is not exactly the goal we are after.

Here, we assumed that the manifold hypothesis holds for adversarial noise space Ω and can be modeled with a dictionary. Rather than setting Ω = R P , we assume this space, if well constrained, can be of lower dimension while maintaining decent adversarial performances. By modeling the underlying structure or manifold of the adversarial noise space, the adversarial dictionary captures the essential features and relationships of the adversarial noises, making the produced adversarial noises both specific to each example, and universal to the modeled adversarial noise space.

As the simplest possible case of modeling, an adversarial dictionary aims to be a linear model of the adversarial noise space, so that both the problem is tractable and the parameters could be inspected.

Adversarial dictionary learning objective

Let a classifier f : R P → R C outputting f (x) ∈ R C , the vector of scores for an example x ∈ X ⊂ R P to belong to a class of y ∈ Y = {1, • • • , C}. The predicted class is given by argmax k f k (x). Recall that, given an example x, an adversarial attack seeks an adversarial perturbation ω ∈ Ω such that x = x + ω, the adversarial example, is a valid example i.e. x ∈ X , close to x, and induces misclas-

sification argmax k f k (x) = argmax k f k (x ) (respecting the adversarial example constraints 1.19).
The originality of our proposition is to express the adversarial noise paired to x as ω x = Dv(x) where D ∈ R P × M is a dictionary composed of M normalized adversarial noise atoms and v(x) ∈ R M a coding vector (further on we write v for readability). While the dictionary D is universal to every example, the coding vector v is specifically tailored to any given x. By setting M = 1 atom the learned adversarial perturbation is universal while, by setting M = P (the dimension of the examples) the adversarial perturbation becomes specific.

In a way, an adversarial dictionary attack's novelty can be seen as, constraining the adver-

sarial noise space such that ω ∈ Ω = {ω k , ω k = Dv k , v k ∈ R M }, with D ∈ R P × M
acting as a base spanning this space. Given a trained DNN classifier f , the adversarial dictionary attack consists of two stages. First a training stage where the dictionary D is learned using a labeled set T = {(x (i ) , y (i ) )} N i =1 . Then an inference stage where, given D, for any new example x (k) (yet unseen), the corresponding coding vector v (k) is crafted to make ω = Dv (k) an adversarial perturbation of x (k) . Notice that as M P, the searching space of the adversarial dictionary attack is a low dimensional space (spanned by the atoms) which is much lower than the original space X . We frame the learning procedure of the adversarial dictionary attack as maximizing the fooling rate under the adversarial example constraints 1.19.

Problem 2.2.1 (adversarial dictionary training objective). Given the classifier f and the training

set T = {(x (i ) , y (i ) )} N i =1 , find D ∈ R P × M and V ∈ R M × N solution of max D∈R P × M V∈R M × N N i =1 1{argmax k f k (x (i ) ) = argmax k f k (x (i ) )}, s.t .      x (i ) = x (i ) + Dv (i ) ∈ X i = 1, . . . , N, ||Dv (i ) || p ≤ p i = 1, . . . , N, D j p = 1 j = 1, . . . , M, (2.1)
where 1 A denotes the indicator function of the set A. This optimization problem optimizes the dictionary D as well as all the coding vectors of the training set V ∈ R M × N , but the latter variable is not interesting as it is useless for future adversarial crafting, it only allows to optimize D.

Once the optimization of D is completed, the adversarial noise space is now modeled and, for any new (yet unseen) adversarial example x, we can navigate in the space spanned by D to find the best specific to x (i ) adversarial noise ω (i ) = Dv (i ) . Inference problem only optimizes the coding vector v (i ) ∈ R M , Problem 2.2.2 (adversarial dictionary inference objective). Given the classifier f , an optimized dictionary of adversarial directions D ∈ R P × M and an original labeled example

(x, y), find v ∈ R M solution of max v∈R M 1{argmax k f k (x ) = argmax k f k (x)}, s.t . x = x (i ) + Dv ∈ X ||Dv|| p ≤ p , (2.2)
Both objective 2.1 and 2.2 have the main drawback to be intractable problems. Therefore, they must be relaxed with a differentiable loss function to ensure good optimization.

During our work, we proposed three different relaxations. Namely, we proposed a first solution, Adversarial Dictionary Learning (ADiL) solved using proximal gradients, a second solution Adversarial Dictionary Learning (ADiL) solved using projected gradients, and finally Linear Model of the Adversarial Noise Space (LIMANS), each presented with some extensions. In the following sections, we review the different proposed solutions, in the respective order.

ADiL proximal gradients solution

At first, we tried to optimize problem 2.2.1 under the scope of a dictionary learning framework, and formulated the optimization as such.

The first ADiL optimization relies on the first relaxation of the adversarial example problem therefore, the adversarial noises are not constrained within an ball, although the minimization of their norm is at the core of the adversarial example optimization.

Optimization

We proposed a first framework to find a dictionary of shared universal attacks D, named ADiL for Adversarial Dictionary Learning [START_REF] Frecon | Adversarial dictionary learning[END_REF]. We proposed to address the following optimization problem reminiscent of classical dictionary learning problems (see, e.g., [START_REF] Mairal | Supervised dictionary learning[END_REF][START_REF] Rakotomamonjy | Direct optimization of the dictionary learning problem[END_REF]).

minimize D∈D V∈V N N i =1 l ( f (x (i ) + Dv (i ) ), y (i ) ) + λ 1 ||v (i ) || 1 + λ 2 ||Dv (i ) || 2 , (2.3)
where l is an adversarial loss enforcing misclassification, whereas D ⊂ R P × M and V ⊂ R M encode some constraints on D and the specific coding vectors v (i ) , respectively.

See that in equation (2.3), the adversarial loss l replaces the non-convex and hard to optimize 1 indicator function of equation (2.1). This relaxation shifts the maximization of the number of adversarial examples to the minimization of an adversarial loss function l , which is a typical procedure in machine learning and allows smoother and more manageable optimization procedures. In ADiL we specifically selected the reverse cross-entropy (minus eq. (1.8) as a convenient adver-

sarial loss L [25, see function f in Section VI.A]).
Even though equation (2.1) explicit three distinct constraints, equation (2.3) does not aim at respecting them. Indeed, at the time, we did not perfectly understood these constraints and thus incorporated them within our proposition. Instead of these three constraints, equation (2.3) does enforce to compute adversarial perturbations ω (i ) = Dv (i ) of low magnitude thanks to the second regularization term. We believed it to be the best option to craft descent and acceptable adversarial perturbations. This optimization problem does not enforce a fixed magnitude budget for the adversarial noises but instead, relies on regularization with the variables λ 2 and λ 1 to ensure respectively, that the magnitude of every adversarial noise stays within the budget and a good sparsity of the coding vectors v (i ) , an essential component of dictionary learning problems. The second regularization term, managed with λ 1 , enforces sparsity within the coding vector's entry. This regularization technique is key to well-defined dictionary learning problems, which is in line with our first idea, to frame the original problem under the dictionary learning scope. Therefore as a good practice, we included these constraints within the optimization problem even though equation (2.1) does not state any particular sparsity constraints.

Note that equation (2.3) is simply the lagrangian formulation of the original problem (2.1), with λ 1 and λ 2 acting as dual variables. Objectively and without context, optimizing the lagrangian formulation of a constrained problem is among the best options. Therefore, before stepping back from the adversarial example definition and criticizing the state-of-the-art propositions, we considered this formulation, the best choice.

Algorithmic solution

Minimizing the objective in (2.3) is a challenge due to the nonconvexity inherent to the dictionary learning formulation and the neural network f . We stress that we are only interested in finding a good stationary point in a limited time. Although classical dictionary learning problems are nonconvex, they are usually solved by alternating the optimization over D and V since each alternating problem is convex. However, here this is no longer the case because of the added term promoting adversarial examples. Hence, we embrace a direct optimization scheme over (D, V) in the spirit of the nonconvex proximal splitting framework of [START_REF] Sra | Scalable nonconvex inexact proximal splitting[END_REF] which has also been applied in the context of classical dictionary learning in [START_REF] Rakotomamonjy | Direct optimization of the dictionary learning problem[END_REF]. To that purpose, we begin by recasting equation (2.3) into, minimize

D∈R P×M V∈R M×N F(D, V) + R(D, V), (2.4)
with

F(D, V) = N i =1 λ 2 Dv (i ) 2 2 + L f (x (i ) + Dv (i ) ), y (i ) , R(D, V) = N i =1 λ 1 v (i ) 1 ,
where F is smooth, provided that f is smooth as well.

Here, we promote the linesearch based proximal-gradient method of [START_REF] Bonettini | On the convergence of a linesearch based proximal-gradient method for nonconvex optimization[END_REF]. For the ease of reading, in what follows we only focus on the proximal-gradient step. Given some sequence of stepsizes {γ k } k∈N , each step amounts in finding

(D (k+1/2) , V (k+1/2) ) = argmin D∈R P×M ,V∈R M×N h (k) (D, V),
where

h (k) (D, V) = R(D, V) -R(D (k) , V (k) ) + ∇ D F(D (k) , V (k) ) (D -D (k) ) + γ -1 k D -D (k) 2 /2 + ∇ V F(D (k) , V (k) ) (V -V (k) ) + γ -1 k V -V (k) 2 /2.
The overall step can be recast as

D (k+1/2) V (k+1/2) = prox γ k R D (k) V (k) -γ k ∇F(D (k) , V (k) )
where the gradient is computed jointly over D and V. Note that since R is separable, it yields that

prox γ k R (D, V) = Proj D (D), Soft γ k λ 1 (V),
where Soft γ k λ 1 (V) is the soft-thresholding operator [START_REF] Daubechies | An iterative thresholding algorithm for linear inverse problems with a sparsity constraint[END_REF]. The full scheme is sketched in Algorithm 2. At inference, for new unseen examples, the same algorithm is used, only the optimization is only done over the coding vectors v (i ) .

Algorithm 2 ADiL

Require:

Parameter δ ∈]0, 1[ Set D (0) ∼ N (0 P×M , 1 P×M ) and V (0) = 0 M×N for k = 0 to K -1 do Provide rough estimate of γ k > 0 Proximal-gradient step D (k+1/2) = Proj D (D (k) -γ k ∇ D F(D (k) , V (k) )) V (k+1/2) = Soft γ k λ 1 (V (k) -γ k ∇ V F(D (k) , V (k) )) Compute the difference d (k) D = D (k+1/2) -D (k) & d (k) V = V (k+1/2) -V (k) Armijo-like backtracking loop i k = 0 and h k = h (k) (D (k+1/2) , V (k+1/2) ) repeat D(k) = D (k) + δ i k d (k) D & Ṽ(k) = V (k) + δ i k d (k) V i k = i k + 1 until L( D(k) , Ṽ(k) ) ≤ L(D (k) , V (k) ) + δ i k h k D (k+1) = D(k) and V (k+1) = Ṽ(k) end for return Dictionary D (K) , coding vector V (K)

Results

It is worth noting that the adversarial noise constraints are respected through both D and V regularization with the regulation variables λ 1 and λ 2 . Therefore, very different results can be obtained according to the tuning of each of the two regulation variables. As Figure 2.2 shows, by enforcing a large λ 2 hyper-parameter, the magnitude of the adversarial noises is hugely constrained which results in a decrease in fooling rate performances. At the same time, when lowering the λ 1 hyperparameter, coding vectors are allowed to be less sparse and thus more freely explore the space spanned by D to compute adversarial noises, which increases the performances of the adversarial attack, but at the cost of producing adversarial noises of higher magnitude. In the proposed framework, the number of atoms M composing the dictionary acts as a hyperparameter controlling to what extent the adversarial noise information is compressed and therefore universal. On the one hand, when M = 1 each adversarial noise only differs by the intensity of the added perturbation through the quantity v (i ) . On the other hand, for M = P, the degree of freedom is high enough to let adversarial noises be completely specific to each image. In that case, each atom targets one pixel of the adversarial perturbation, making D a basis of R P . The choice of M has been investigated and indeed impacts the performance of the attacks. Figure 2

Learning ADiL with less data

As a sanity check of the optimization and looking for insights, we tried to optimize problem (2.3) using fewer training data points N. Indeed, the idea of adversarial perturbation modeling was to capture the most common patterns of the data fooling the classifier. Therefore by increasing the number of training points, the inference performances on unseen data points should increase as D captures more generalized fooling patterns. And we empirically confirm this insight. Indeed, in Figure 2.4, we see that by increasing the training set size N the performances D increase for various numbers of atoms M. For M = 5, M = 10, and M = 15 atoms, the test fooling significantly increases but for only M = 1 atom the performance only marginally increases. This artifact is a result of a too-small modeling space. Indeed, with only M = 1 atom, D is, to a coefficient factor, universal to every example, which constrains the problem too much to see that big of improvement when varying M. However, for higher values of M, the modeling space is large enough to see significant changes in the test performances according to the training set size N.

Overall and as expected test performances increase according to the number of atoms M, but because of the tuning of the regularization parameters λ 1 and λ 2 and the random seed, we see some discontinuities in the order of performances for the three number of atoms. This variability in performances is confirmed as the higher the number of atoms, the smoother the performances increase. Indeed, for M = 15, performances almost increase linearly with the number of training data points. Whereas for M = 5, test performances are globally growing with the number of training data points, even though we see some light "abnormal" decrease due to external factors. The dictionary D is optimized on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of ADiL, while on the x-axis is displayed the number of training data points.

ADiL projected gradients solution

The main drawback of the first proposed dictionary learning optimization is that it does not respect the fixed adversarial budget, which makes the comparison with state-of-the-art attacks biased. It is worth noting that plenty of adversarial attacks do not originally respect this constraint either. In the revisited version of ADiL, we proposed a framework that allows controlling the norm of every adversarial noise to be within an . This second version of ADiL relies on the second adversarial crafting relaxation, such that ||ω (i ) || p = ||Dv (i ) || p ≤ ∀i .

Optimization

In order to learn the dictionary of shared attacks, we propose to address the following optimization problem reminiscent of classical dictionary learning problems (see, e.g., [START_REF] Mairal | Supervised dictionary learning[END_REF][START_REF] Rakotomamonjy | Direct optimization of the dictionary learning problem[END_REF]).

min D∈D,V∈V N i =1 l f (x + Dv (i ) ), y (i ) , s.t . ||Dv (i ) || p ≤ p i = 1, . . . , N, (2.5)
Compared to the firstly proposed optimization, we used two adversarial loss L, namely the logit loss and the reverse cross-entropy, both proposed in [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF]. In Section 2.4.1, we detail a deeper analysis of these two losses. Depending on the choice of the constrained sets D and V, one can ensure that the adversarial noises are all p -norm constrained. We integrated this constraint within the parameters D, V which ends up in the following proposition, Proposition 2.4.1 ( p -Attacks). Given some budget > 0, we have that Dv p ≤ for every D ∈ D and v ∈ V where

D = {D ∈ R P×M | (∀m ∈ {1, . . . , M}), d m p ≤ 1, p ∈ P}, (2.6)
and

V = {v ∈ R M | v 1 ≤ }.
(2.7)

Note that, for such a choice, both D and V are convex sets. The proof that by framing ω = Dv, with D ∈ D,v ∈ V results in ||ω|| p ≤ is given in Section A.1. The integration of this proposition within the optimization problem is solved by Algorithm 4.

Inference optimization

We now consider the definition of the dictionary-based adversarial attack to unseen examples, coined the ADiL attack.

Provided that the dictionary D ∈ D is known, for a new yet unseen original example x of original label y, we propose two ways to learn its paired adversarial perturbation x + Dv such that the dictionary D is considered a parameter, and the optimization is performed only over v,

(unsupervised) v ∼ P (2.8) (supervised) min v∈V l ( f (x + Dv, y),
(2.9) Equation 2.8 derives a black-box ADiL adversarial attack. In the meantime, equation 2.9 gives a white-box ADiL adversarial attack. Equation 2.9 can be solved by resorting to Algorithm 4 and ignoring the optimization steps over D.

However, in practice, we found the optimized adversarial perturbation norm ||Dv|| p might still be far from the p norm budget . This artifact prevents us from fully exploring the space spanned by D, leading to sub-optimal results. When all the adversarial attacks are given an p norm budget, every adversarial perturbation will be of norm . Therefore by resorting to an algorithm that disables us from reaching this norm budget, the ADiL adversarial attack is unfairly compared to other adversarial attacks and could therefore never reach their performances.

To benefit from the proposed framework, while still being competitive with other adversarial attacks, we proposed to recast the objective of the ADiL white-box adversarial attack (2.9). Given D ∈ R P × M with M P and by assuming that D is of full rank, we may write v = D † z with D † = (D D) -1 D , therefore, problem (2.9) becomes, min z∈R P l ( f (x + DD † z, y), s.t. z p ≤ .

(2.10) Equation (2.10) takes advantage of all the possibilities offered by D, navigating in all of the spanned space, allowed to find a maximum number of adversarial perturbation under the p norm budget constraint. Finally, to ensure that the adversarial example x + Dv is a valid image, we additionally perform a projection onto the input manifold X ⊆ R P , i.e., x = Proj X (x + Dv). The biggest flaw of recasting the problem into equation (2.10) is that now the optimization in on z ∈ R P rather than v ∈ V ⊂ R M , which includes as many variables z i , i ∈ {1, . . . , P} as the number of pixels, which is the problem solved by state-of-the-art adversarial attacks. Indeed, the recast loses the computational gained offered by the storage of a complete dictionary. Algorithm 4 presents the optimization of the white-box ADiL adversarial attack, and details the different performed projections.

Algorithmic solution

Herein we propose a procedure for solving Problem (2.5). Note that minimizing the objective in (2.5) is challenging due to the nonconvexity inherent to the dictionary learning formulation and the neural network f . We highlight that we are only interested in finding a convenient stationary point in a limited time. Since classical dictionary learning problems are bi-convex, they usually are solved by alternating the optimization over D and V since each alternating problem is convex. However, here this is no longer the case because of the adversarial loss involving a neural network. Hence, we embrace a direct optimization scheme over (D, V) in the spirit of the nonconvex proximal splitting framework of [START_REF] Sra | Scalable nonconvex inexact proximal splitting[END_REF] which has also been applied in the context of classical dictionary learning in [START_REF] Rakotomamonjy | Direct optimization of the dictionary learning problem[END_REF]. Algorithm 4 performs a simple projected gradient descent over z by minimizing an adversarial loss l such that the algorithm outputs a valid adversarial example

x = x + Dv = x + DD † z,
The projection operator proj ,p of Algorithm 4 is a simple yet effective rescaling of the variable by its norm, that is proj (z) = z * /||z|| p . Given a pre-process of the original image pixels values, Algorithm 3 ADiL using projected gradients

Require: Step-sizes {γ k } K-1 k=0 Set D (0) ∼ D Set V (0) = 0 M×N for k = 0 to K -1 do loss = N i =1 l ( f (x + Dv (i ) ), y (i ) ) D (k+1) = Proj D (D (k) -γ k ∇ D loss) V (k+1) = Proj V (V (k) -γ k ∇ V loss) end for return Adversarial dictionary D (K)

Algorithm 4 White-box ADiL adversarial attack

Require: An optimized dictionary D ∈ D, an unseen original example x, norm budget , a stepsize γ, a max number of iteration T z = N (0 P , 1 P )

D † = (D D) -1 D for t = 1 to T do z = proj ,p (z) x = proj X (x + DD † z ) loss = l ( f (x ), y (i ) ) z = z -γ∇ z loss if argmax k f k (x + DD † z ) = argmax k f k (x) then return Adversarial perturbation ω = DD † z
end if end for the original manifold is expressed X = [0 : 1] P . We, therefore, set the projection into manifold operator as proj X (x) = min(max(x, 1), 0), which clamps negative and above 1 adversarial examples pixels values to their bounds. Such projection function is not smooth and could potentially corrupt the optimization, but in practice, we found the p norm budget constraint to be already too constraining. It organically enforces the pixels of the adversarial examples to be within the manifold.

ADiL in between specific and universal adversarial attacks

In this section, we further investigate how the revisited ADiL performs in a more challenging largescale scenario such as ImageNet.

The revisited ADiL problem allows it to be fairly compared to other state-of-the-art adversarial attacks given the same adversarial budget. We now compare our approach with the baseline attacks on multiple pre-trained models on ImageNet from the TorchVision repository. Namely, we consider MobileNetV2, DenseNet121, InceptionV3, ResNet18, GoogleNet and VGG11, achieving accuracy of 71.88%, 74.43%, 69.54, 69.76%, 69.78% and 69.02%, respectively. Performance in terms of fooling rate (fr), mean squared error (mse) and time are reported in Table 2.1.

On the one hand, the found results once again shed light on the superiority of specific attacks (e.g. AutoAttack [START_REF] Croce | Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks[END_REF]) managing to achieve up to 100% fooling rate. However, such performance comes at a price of high computational cost, up to 10 times more than that required to perform universal attacks. The only two exceptions are FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] and its variant FFGSM [START_REF] Wong | Fast is better than free: Revisiting adversarial training[END_REF], both being one-shot specific attacks. Still, their efficiency and low complexity come at the expense of the highest mse of any attacks.

On the other hand, universal attacks perform relatively well in large-scale experiments, mainly because of the many existing classes allowing them to easily find an adversarial label to target.

Nonetheless, their fooling rates are still significantly far behind those of specific attacks. Regarding ADiL, it displays competitive results at a much lower computational cost than specific attacks while also benefiting from low mse. Indeed, it performs at most 5 times faster than iterative specific attacks and still achieves relatively high fooling rates (e.g., [START_REF] Liu | On the variance of the adaptive learning rate and beyond[END_REF].14% for ADiL-ce on MobileNet). In addition, ADiL always drastically improves upon universal attacks in terms of fooling for at most twice or three times their execution times and a comparable mse. Finally, it is unclear whether using the logit loss over the cross-entropy is beneficial since both yield equivalent results on average. We display in Figure 2.5 a selection of 5 atoms of ADiL-100 learned on ImageNet for fooling either a MobileNet or a GoogleNet network. Observe that they all exhibit strong structured patterns reminiscent of those found in the UAP perturbation [START_REF] Shafahi | Universal adversarial training[END_REF]. However, contrary to universal attacks which merge all individual perturbations into a single one, the proposed dictionary framework allows to split the individual contributions into multiple diverse atoms.

Inspecting the quality of the adversarial space modeling

In practice, when setting the norm constraint to , among all the adversarial attack baselines, all of their computed adversarial perturbations are of norm . In other words, to maximize the performances, adversarial attacks exploit all the possible space Ω to find a maximum of adversarial perturbations to maximize the FR. Allowing the adversarial perturbations norm to reach the adversarial budget is key to being competitive with state-of-the-art adversarial attacks. In ADiL's We inspected the inferred adversarial noises to better understand the crafted adversarial perturbations beneath a simple FR performance. We found that ADiL's crafted adversarial perturbations are far from the allowed norm bound. Figure 2.6 displays ADiL's crafted adversarial perturbation average 2 norm for various norm bounds.

Through Figure 2.6 we understand that the optimization process is flawed as it does not allow the adversarial perturbations to reach the norm budget. This artifact is not a flaw as every adversarial constraint is indeed respected. The adversarial perturbation generation is well designed according to the rules set for a fair comparison, but in practice, it prohibits ADiL from reaching state-of-the-art adversarial attack performances. We see that the gap between ADiL's crafted adversarial perturbation mean 2 norm and is not negligible, for = 1, the mean 2 norm of ADiL's adversarial perturbations is only 0.84 which means that the 16 most promising percentage of the allowed adversarial perturbations norms are unexploited. Indeed, this range is the most promising as obviously the higher the norm of the adversarial noises the more harmful the adversarial perturbation is the the more likely it will fool the classifier while respecting the other adversarial constraints.

The 2 norm constraint allowed budget accepted by the community is = 0.5. Every adversarial attack considered this bound as it allowed every adversarial attack to be fairly compared. In ADiL's case, when setting = 0.5, we empirically observe that the crafted adversarial perturbations are of magnitude 0.41, which is too far from state-of-the-art adversarial attacks perturbations norm, always equal to = 0.5, to let ADiL be competitive. From this observation we considered two solutions. First, we could align every adversarial attack to be of the same norm constraint bound as the one we empirically found with ADiL. Therefore in the latter example, we would set = 0.5 for ADiL, find a mean adversarial norm of 0.41, and then compute the performances of the adversarial attack baselines with = 0.41. This proposition has not been explored as it would be cheating, unfair to other baselines, and bad scientific behavior.

The other proposition was to compute adversarial attack baseline performances with = 0.5 just like it is supposed to be, but in ADiL's case, as the inference problem is flawed, we allow a greater norm bound, making it reach adversarial perturbations norms of 0.5. In Figure 2.6, we empirically observe that when setting = 0.6 we observe a mean 2 norm of ADiL's adversarial perturbation equal to 0.5. Therefore, the original adversarial constraints are respected and ADiL would be allowed to be competitive and "fairly" compared with other adversarial attack baselines. In practice, this idea is impossible to consider because it is "cheating" to dig into the inference perturbations and re-adjust the original problem to maximize the inference performances. Second, Figure 2.6 displays the average 2 norm of the ADiL's adversarial perturbations, meaning that some norms are higher, and some are lower than . However, higher perturbation norms are forbidden making this idea impossible in practice.

ADiL's projected gradient solution lacks an efficient inference problem allowing it to be competitive with state-of-the-art adversarial attacks while respecting all the adversarial constraints.

Changing the adversarial loss

In [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF], multiple adversarial loss l have been proposed and explored for multiple p norm budget constraints, but regarding the ADiL optimization framework, we assumed necessary to review different adversarial losses within (2.5) to see if there are significant changes or if the overall optimization is robust to the choice of the adversarial loss. We selected the two most used and most effective adversarial losses. We first ran experiments with the reverse cross-entropy adversarial loss that is, for an original example x with an original prediction f

(x) = y, l f (x ), y = -L f (x ), y = + C k=1 o k l og f (x ) k , (2.11)
with f (x ) k representing the predicted probability of the label k for the adversarial example

x through f , and o a one-hot vector with 1 at index y representing the true predicted label of x.

The goal of the reverse cross-entropy as the adversarial loss is to decrease as much as possible the original prediction. While the original objective of adversarial examples is to change the classifier's final output, the reverse cross-entropy only intends to lower the prediction of original labels such that other labels receive confidence, up until another label represents the maximum probability value over the possible labels, acting as the prediction shift. While the reverse cross-entropy is an easy-to-use and straightforward adversarial loss, [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] proposed other losses with interesting properties. The reverse cross-entropy focuses on minimizing the original label prediction but does not intend to maximize any other targeted label prediction. The optimization of the reverse cross-entropy may not be optimal as it does not allow increasing other labels prediction. Carlini & Wagner however, proposed in [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] an adversarial loss capable of both minimizing the original label prediction and maximizing another adversarial label prediction. This loss is called the logit loss and is defined as, for an original example x with an

original prediction f (x) = y, l ( f (x ), y) = max( f t (x ) -f y (x), -κ),
(2.12)

with κ a margin hyper-parameter and t = argmax k =y f k (x) the second best label predicted for the original example, becoming the targeted adversarial label to predict. The logit loss minimizes the original label prediction of the adversarial example and at the same time, maximizes the second-best label prediction of the adversarial example. Carlini & Wagner studied multiple adversarial losses including the reverse cross-entropy suggest relying on the logit loss for 2 norm bounded adversarial attacks. We assumed it was necessary to compare both adversarial losses, for both the 2 and ∞ norm constraints. Indeed, in our optimization framework, we do not have any insights on which is the best therefore, a comparison of both losses is needed.

Table 2.1 compares ADiL's performances to specific and universal adversarial attacks. First, it is very promising to see that even though ADiL does not reach state-of-the-art specific attack performances, ADiL is always a much stronger attack than universal baselines. These results place ADiL where we intended it to be, within both adversarial attack families and acting as a good tradeoff between a transferable universal attack and a very effective and harmful specific attack. We also studied the two selected adversarial losses, the logit loss, and the reverse cross-entropy (rce) loss. As Table 2.1 shows, we do not see any differences between both losses. It means that the ADiL learning framework is robust to the chosen adversarial loss function, and promises to be effective for other tasks loss such as an object detection loss, or semantic segmentation loss. Such problems are yet to be studied, but the ADiL framework could easily be applied and promises to be effective with any other adversarial loss.

Time consumption analysis

Table 2.1 also compares the time consumption of the different adversarial attacks. We draw similar conclusions for the position of ADil's time consumption compared to other baselines. On the one hand, universal attacks compute adversarial examples very quickly as only one adversarial perturbation is optimized over all the original examples. This optimization averages to a tiny time consumption per example. On the other hand, specific attacks are very effective at finding adversarial perturbations for most of the original examples, but these optimizations are not free. Indeed, we see the best specific attack Autoattack, sometimes more than 10 times longer than universal attacks to compute all of the adversarial perturbations. It empirically shows a significant gap in the literature to be filled. As expected, ADiL perfectly fits in the middle of both families, computing adversarial examples always faster than specific attacks but longer than universal attacks.

ADiL fastens the computation of adversarial perturbations by finding a vector v ∈ R M , composed of only M P atoms, drastically lowering the number of variables to optimize.

Black-box ADiL attack

Along with the white-box attack derived from the optimization of equation 2.9, we also tried to derive a black-box attack. Black-box attacks are reviewed and detailed in Section 1.2.4.

Having an ADiL black-box attack is more appealing than an ADiL white-box attack because it reflects real-world scenarios where an attacker has limited or no knowledge about the classifier in use. Besides black-box attacks are more challenging for defenders to detect and mitigate, making the ADiL black-box attack a very desired proposition.

Deriving a black-box attack means that we can optimize D using equation 2.5 in the same way, but at inference time, for new unseen examples x (i ) , the corresponding coding vector v (i ) is optimized without the access of the internal parameters of the classifier f . Without any possibility to optimize the coding vectors v by gradient descent, we opted for a rejection sampling strategy, in which we try T sampling of v according to a certain probability distribution and stop the trials once a v produces an acceptable adversarial perturbation ω = Dv. This black-box attack is detailed in Algorithm 5.

In our experiments, we mainly focused on sampling the coding vector on the 1 norm sphere.

As highlights line 3 and 4 of Algorithm 5, we sampled v from P = {v ∼ N (0 M , 1 M ), ||v|| 1 = 1}. The

Algorithm 5 Black-box ADiL attack

Require: An optimized dictionary D ∈ D, an unseen original example x, a max trial number T

for t = 1 to T do v ∼ N (0 M , 1 M ) v = v/||v|| 1 if argmax k f k (x + Dv) = argmax k f k (x)
then return Adversarial perturbation ω = Dv end if end for choice comes from our choice to respect the adversarial budget constraint of equation (2.5). Indeed, by modeling D ∈ D, any coding vector vector v of a 1 1 norm produces an adversarial perturbation Dv (i ) within the ball. Therefore the inference sampling would likely be successful if the inference coding vectors do share the same shape.

Figure 2.7 illustrates the results of the black-box adversarial attacks derived from the optimization of equation (2.5). This figure shows the test performances of the black-box ADiL attack for various trial number limits. With M = 1 atom, the same conclusion is drawn as in Section 2.3. When the modeling space is too limited, no real conclusions can be drawn from a possible enhancement of the original version. However, with higher numbers of atoms M = 5, 10, 15, we see that the black-box ADiL attack performances increase according to the number of trials, which proves both the attacks potential and the sampling strategy ability to produce effective v. The left figure shows a plateau is not far ahead regarding the 2 norm of the adversarial perturbations although, according to the ∞ norm constraint, the test performance increases almost linearly according to the number of trials.

Overall, deriving a black-box attack from the original proposition is a good idea as it derives a more realistic adversarial attack from the ADiL modeling. We also tried to rely on the optimized training coding vectors V ∈ V (2.7) of equation (2.5).

Indeed, we tried some preliminary experiments in which we sampled the inference coding vectors v from a normal distribution of the optimized training coding vectors, but unfortunately, no better results were found.

ADiL adversarial defense

As the guideline of [START_REF] Tramer | On adaptive attacks to adversarial example defenses[END_REF], we proposed, along with the ADiL adversarial attack, an

loss = L( f (x (i ) ), y (i ) ) θ = θ -γ∇ θ l oss Optimize on adversarial data find v (i ) such that x (i ) = x (i ) + Dv (i ) is adversarial to f loss = L( f (x (i ) ), y (i ) )
θ = θ -γ∇ θ l oss end for end for return Robustly trained classifier f As Table 2.2 and Figure 2.8 show, training a classifier with the proposed defense including a dictionary of M atoms, not only makes it more robust to ADiL attacks of M atoms but also ADiL attacks of any number of atoms. Indeed, the robustness of the classifier is hugely improved against dictionaries that are composed of fewer or as many atoms as the dictionary used to train the classifier, but in a lesser manner, the robustness is also improved against dictionary attacks containing more atoms than the number used to train the classifier. One of the main weaknesses of adversarial training defenses is that when relying on a single adversarial attack to add its produced adversarial examples to the training set of the classifier, the produced classifier is indeed more robust, but only to the adversarial attack used to create the training adversarial examples. By relying on an ADiL attack to augment the training dataset, our empirical proofs, Table 2.2 and Figure 2.8 show that the defense mechanism profits from the modeling of the adversarial noise space, letting users build classifiers that are more robust to not a single bu any adversarial attacks. Indeed, we see classifiers more robust to similar or simpler adversarial attacks (ADiL with less or as many atoms), but also more complex adversarial attacks (ADiL with more atoms).

Optimization configuration

In this new implementation of the adversarial dictionary problem, because we all have the same adversarial budget, we can be fairly compared and benchmarked to other adversarial attacks. However, almost every adversarial attack includes either hyper-parameters or parameters that need to be fine-tuned, so that every attack reaches its best performance. In our case, because ADiL includes the modeling of the adversarial noise space, we need to fine-tune plenty of critical parameters that could jeopardize the performance of the attack if not carefully tuned. As well as maximizing the performances of ADiL's attack, finding the best ADiL parameters on a given set of experiments allows us to bring future users a manual, or at least insights, on the right parameters according to their applications. In this section, we review the most critical optimization parameters and evaluate their sensitivity to the correctness of ADiL's modeling and attack performances.

Optimizer

We previously saw in Section 1.1.3 that the optimizer is a critical parameter to the optimization process. We recall that an optimizer is simply the implementation of the strategy to find the right optimization step size at each iteration. Recent works [START_REF] Schmidt | Descending through a crowded valley -benchmarking deep learning optimizers[END_REF] showed that a slight change of optimizer could enhance or decrease the performances by far, therefore we conducted an intensive analysis on which optimizer would be best for the ADiL modeling problem, that is optimizing D and V over the training set.

Regarding the optimization of ADiL, we were looking for an optimizer that is both efficient and robust, as many other parameters are free, such as the number of atoms M, the norm used within the adversarial constraints, the number of training points N and the complexity of the problem at hand. In the case of ADiL, the robustness of the optimizer is as important as the efficiency in reaching the best possible solution Therefore we selected three optimizers, Adam [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], Adam Weighted (AdamW) [101] and SGD [START_REF] Sutskever | On the importance of initialization and momentum in deep learning[END_REF] that are widely used and enough different to see if ADiL's optimization is agnostic to the choice of optimizer or if there is one optimizer that is prominently better than the others. Because we needed the optimizer to be robust in the task it could handle well, we selected the three mainly used optimizers and were not interested in some lesser-used ones as we do not have any long-term trust and reliability over those.

To make ADiL easy to use for future users we were looking for only one optimizer that meets our requirements for both the 2 and ∞ norm constraints.

Figure 2.9 displays the evolution of the training loss and the relative norm of the parameters D and V that is evaluated as norm(D t -1 )-norm(D t ) norm(D t -1 -D t )

+ norm(V t -1 )-norm(V t ) norm(V t -1 -V t )
, at the iteration t . These two metrics allow us to assess the minimization of the training loss and the convergence towards a minimum. In Figure 2.9, it is clear that for both 2 and ∞ , the SGD optimizer is a bit too basic compared to other optimizers. Indeed, with the 2 norm, the SGD optimizer does not reach the best minima and is shown to be too unstable when using the ∞ norm. Between Adam and Adam Weighted, the choice is open to debate. Indeed, neither exceeds the other in terms of loss minimization or robustness to the initial step size. To achieve a good balance between the optimizer performances and descent robustness to the choice of the learning rate, we chose to carry out our experiments with the Adam optimizer. Now that we selected the Adam optimizer as the right tool to use under both the 2 and ∞ norm, we must select the best initial learning rate, possibly one for each norm. Indeed, Figure 2.9 highlights the need for a different learning rate for each norm. Thus we conducted an initial learning rate experiment comparison under the Adam optimizer for both the 2 and ∞ norm.

Learning rate

Even though we purposefully select the Adam optimizer as the most efficient and robust optimizer, the learning rate remains a sensitive parameter. Therefore we conducted experiments with the Adam optimizer paired with different learning rates to select the best setup, hopefully, the same for both the 2 and ∞ norm. Figure 2.10 shows us the evolution of the training loss and the relative norm of ADiL's parameters D and V, which is computed the same as in the Optimizer experiments. We understand through Figure 2.10 that a different learning rate must be used according to the p norm constraint used. This choice is mainly drawn from the Adam-0.1 curve that could be a decent choice for the ∞ norm constraint, compared to 2 norm constraint for which 0.1 is a really bad choice showcasing an under-fitting. According to the different curves, we selected using Figure 2.10a the learning rate for ∞ norm constraint to be 0.05 and for 2 norm constraint with Figure Figure 2.11 presents the result of the ADiL optimization procedure under the best hyperparameters. The atoms are visually interesting and allow us to unveil the common patterns involved in the crafting of adversarial perturbations. Note that absolutely no constraint on the "readability" of the atoms was included in the optimization process but we still end up with these visual atoms. Possessing such visual atoms allows us to understand the roots of the adversarial perturbations, which are to some extent the universal pitfalls of the neural network classifiers, according to which we can infer new adversarial perturbations. It is worth noting that the atoms differ a lot according to the p norm constraint used. Indeed, on the one hand, Figure 2.11a shows a dictionary optimized using the ∞ norm constraint, and we see that the atoms capture the corners and edges of the objects to allow computing adversarial perturbations based on those. On the other hand, Figure 2.11b presents atoms optimized under the 2 norm constraint. We see the atoms captured some random noise allowing us to craft adversarial perturbations that are less "understandable" but still universally effective at fooling the classifier.

It is desired for these atoms to be visually available for inspection, as for explaining to a nonexpert audience, the origin, and computation of the adversarial perturbations become very clear and easy to understand. These atoms allow us to investigate deeper into the origin of the adversarial noise so we can incorporate those within a defense mechanism producing more robust classifiers. The proposed ADiL adversarial defense is detailed in Section 2.4.1.

Figure 2.11 shows the result of the training procedure, but for new unseen original exam-ple, the computation of their adversarial perturbations from the learned D needs another hyperparameter to tune, that is the inference learning rate.

Inference learning rate

As for the inference problem, we assumed the Adam optimizer to be a decent choice as we previously sanity-checked its efficiency and robustness for the training procedure. However, we could not make the same assumption with the learning rate as the number of variables to optimize drastically changes. Indeed, the inference problem only optimizes the coding vector v ∈ R M such that ω = Dv is an effective adversarial perturbation, conditioned to the learned D.

The number of variables to optimize crucially influences the choice of the learning rate therefore, we deep analyzed its impact.

Figure 2.12 displays the evolution of the inference loss along with the final FR reached at the end of the inference optimization. Those two metrics are our only key metrics allowing us to assess a good optimization. The same conclusions can be drawn regarding the inference learning rate, it largely impacts the optimization process and therefore needs to be carefully tuned to ensure good inference performances. Through Figure 2.12b and 2.12a we understand that a different inference learning rate must be used depending the p norm constraints used. From Figure 2.12b we selected for the 2 norm constraints, an inference learning rate of 0.01 which reaches the best FR and visually displays the best optimization behavior, with a plateau for the inference original example for which no adversarial perturbations could be found. As for the ∞ norm constraints, we select through Figure 2.12a an inference learning rate of 0.005 because of the same reasons, the best FR is reached, while displaying a descent loss minimization.

While the latter ADiL projected gradient solution is interesting and led to good results, the full potential of the idea is not quite reached, as a-posteriori, we observe that the training adversarial noises norm is quite far from the limit, ||ω (i ) || p = ||Dv (i ) || p (see Section 2.4.1 for more details). This artifact limits the modeling of the adversarial noise space by D and prevents the overall attack from reaching its full potential.

Even though it is very promising and theoretically appealing, the empirical results did not match our expectations, and therefore ADiL has only been a stepping stone in the process leading to LIMANS.

(D t -1 )-norm(D t ) norm(D t -1 -D t ) + norm(V t -1 )-norm(V t ) norm(V t -1 -V t )
, at the iteration t , with M = 100 during the training procedure, according to different optimizers (ADAM, ADAM-W and SGD) for multiple initial step-sizes and both 2 and ∞ norm constraints.

norm(D t -1 )-norm(D t ) norm(D t -1 -D t ) + norm(V t -1 )-norm(V t ) norm(V t -1 -V t )
, at the iteration t , with M = 100 during the training procedure, using the Adam optimizer couple with various learning rates for both 2 and ∞ norm constraints. Figure 2.12: Evolution of ADiL's inference loss with M = 100, using the Adam optimizer coupled with various inference learning rates for both 2 and ∞ norm constraints.

LIMANS stochastic gradient solution

After a fruitful discussion with prestigious researcher Nicholas Carlini 1 , there was a two-sides goal regarding the improvements of ADiL. First, given a budget , the adversarial noises must exploit this freedom to the fullest, meaning the adversarial noises p norm should be close or equal to , to model the adversarial noise space efficiently. Second, an empirical analysis of the model space should be emphasized as if well modeled, the transfer property of universal attacks should hold using the modeling, allowing an adversarial attack to be both efficient to specific adversarial noise crafting and transferable to fooling other classifiers g than the one through which D has been optimized. Therefore, the original assumption of problem (2.1), that dictionary learning tools are relevant to the modelization of the adversarial noise space, is now questioned. Indeed, the sparsity constraint on the coding vectors is not quite fundamental and should therefore be removed to lower the bias during the optimization. It led to deeply questioning our initial assumptions with ADiL and pivoting toward a new lecture on the problem, engaging other optimization solutions such as stochastic gradients.

To solve the original problem (2.1), we proposed two relaxation optimizations, namely Regularized-LIMANS, a stochastic gradient descent optimization enforcing the constraints through regularization, and Simple-LIMANS, a plain loss function incorporating all the constraints and optimizing both D and V at the same time in a deep learning spirit. A high-level overview of the proposed LIMANS adversarial attack is given in Figure 2.13. It is highlighted the adversarial model D is universal to every adversarial example, while a specific coding vector v is crafted for each adversarial example.

UNIVERSAL ATTACK SPECIFIC ATTACK

LIMANS

Regularized-LIMANS solver

In the original problem (2.1), the indicator function 1 being non-smooth and highly non-convex makes the optimization untractable. Instead of maximizing the fooling rate, it is usual to consider minimizing a surrogate adversarial loss function L γ ( f (x ), f (x)), parameterized by γ, more amenable to optimization. A typical adversarial loss function of interest is

l γ ( f (x ), f (x)) = max -γ, f c (x ) -max k =c f k (x ) , (2.13)
Still, the problem is hard to solve because of the non-convexity of the constraint ||Dv (i ) || p ≤ δ p or the account of the constraint x (i ) ∈ X . Hence, for computational tractability, we propose to rely instead on the following regularized relaxation problem,

min D∈D V∈R M × N N i =1 l γ ( f (x (i ) + Dv (i ) ), f (x (i ) )) + λh (δ p ,p) (D, v (i ) ) s.t . D ∈ D, (2.14) with λ ∈ R + a regularisation parameter, D = {D ∈ R P × M , ||D j || p = 1, ∀ j ∈ {1, .
. . , M}} and h (δ p ,p) representing a penalty function. We consider the p -norm, with p = 2 or p = ∞, as penalty function

leading to h (δ 2 ,2) (D, v) = max(||Dv|| 2 -δ 2 , 0) and h (δ ∞ ,∞) (D, v) = k max(|(Dv) k | -δ ∞ , 0).
Enforcing a constraint through regularization is one of the most common ways, and the most obvious when considering instead the optimization of a Lagrangian formulation. Therefore, problem (2.14) made total sense as the most natural regularized optimization of the original problem (2.1). Note that the constraint x (i ) ∈ X is not integrated into this formulation and therefore has to be dealt with post-processing. We proposed to optimize (2.1) with a stochastic gradient descent optimization. Algorithm 7 sketches its optimization as well as the post-processing ensuring x (i ) = x (i ) +Dv (i ) 7: ŷadv = f (x (i ) ); ŷ = f (x (i ) ) 8: loss i =L 0 ( ŷadv , ŷ)+λh (δ p ,p) (D, v (i ) end for 17: end for 18: Dv (i ) ← Proj {Dv | Dv p ≤δ} (Dv (i ) )

x (i ) ∈ X .
19:

x (i ) ← Proj X (x (i ) + Dv (i ) ) 20: return {x (i ) } N i =1 , (D, V)
Here, we get rid of the constraints x (i ) ∈ X , ∀i and enforce small magnitude of ||Dv (i ) || p through the regularizer h ( p ,p) . Empirically this promotes x (i ) + Dv (i ) ∈ X to be nearly close to X .

The Regularized-LIMANS optimizes (D, V) in a stochastic fashion, and specifically, D is updated using a projected gradient descent that ensures that the constraints ||D j || p = 1, ∀ j are satisfied. A grid search on λ allows to control of the generalization of the model. In practice, for the selected value of λ, the constraint on Dv p might slightly be violated. If needed a post-processing is performed to ensure the respect of the constraint.

Simple-LIMANS solver

The Regularized-LIMANS requires the cumbersome tuning of the hyper-parameter λ. To alleviate that, we propose a second relaxation of LIMANS that involves an objective function encompassing two of the constraints, the last one being taken care of by post-processing. Specifically, the method termed Simple-LIMANS solves the following problem, min

D∈R P × M V∈R M × N N i =1 l ( f (proj X (x (i ) + p Dv (i ) + b Dv (i ) + b p )), f (x (i ) )) (2.15)
where proj X denotes the projection operator that maps its input x (i ) = x (i ) +Dv (i ) +b onto X and b an offset variable.

Note that the Simple-LIMANS adds a new variables b to he definition of the adversarial perturbation, that is ω (i ) = Dv (i ) + b. This offset is entirely universal to every adversarial perturbation and, just like in an affine function, it allows to uplift of every adversarial perturbation whatever the value of v. In a way the variable b can be seen as a rewriting of both D and v as [D, b] T and [v, 1] T . Such relaxation of the variables is commonly used within linear classifiers to allow greater flexibility in decision boundary optimization. The role and necessity of the offset b is questioned and analyzed in Section 2.6.7. To ease the reading and facilitate the comparison between Simple-LIMANS and Regularized-LIMANS we will consider b to be part D as its relaxation and therefore omit to mention it when unneeded to avoid confusion.

Simple-LIMANS trades off the constraint D ∈ D, i.e. the unit norm constraint over the atoms of D, for the explicit guarantee that the adversarial example x (i ) is valid (i.e. belongs to X ) and that the adversarial noise is utmost of magnitude p by defining x (i ) i as:

x (i ) = proj X (x (i ) + p Dv (i ) Dv (i ) p ).
Here proj X is the projection operator onto X . Simple-LIMANS solves (2.15) by iteratively updating D and V using a gradient descent procedure as shown in Algorithm 8. It proves computationally efficient as it does not require hyperparameter tuning. At termination, a post-processing is used to ensure D ∈ D without changing the adversarial examples by rescaling V accordingly.

Due to their nature, both Regularized-LIMANS and Simple-LIMANS' stochastic optimization apply to large-scale datasets.

Inference optimization

At inference time, provided that D is known, we seek for an unseen example x (k) , its adversarial counterpart x (k) = x + Dv (k) . For both proposed regularization, the original inference problem (2.2) is relaxed in the same way, by optimizing (2.14) and (2.15) only over v (k) . Therefore at inference time, the complexity of the adversarial example crafting is on M ≤ P which significantly improves the complexity of the adversarial example problem. Once D is known, for a new unseen original example x (k) , both original relaxation optimization problem boil down to, min

v (k) ∈R M l ( f (proj X (x (k) + p Dv (k) + b Dv + b p )), f (x (k) )), (2.16)
for the Simple-LIMANS algorithm and to, min (D, v (k) ).

v (k) ∈R M l γ 2 ( f (x (k) + Dv (k) ), f (x (k) )) + λh (δ p ,p)
(2.17)

for the Regularized-LIMANS algorithm. Note that for the Regularized-LIMANS algorithm, there is still the γ 2 hyper-parameter to tune, which is not the same as γ previously used for the optimization of both D and V during the training stage. Similar to the training of both D and V, the above problems (2.16) and (2.17) can both be solved directly by gradient descent as in Algorithm 8 and Algorithm 7, but here with the D fixed, considered as a parameter of the optimization. noise (i ) = Dv (i ) 7: V ← Optim(∇ V loss) (Update)

x (i ) = proj X (x (i ) + δ p noise (i ) noise (i ) p ) 8: ŷadv = f (x (i ) ); ŷ = f (x (i ) ) 9: loss i = L ∞ ( ŷadv , ŷ
V ← [||D • j || p V j • ] ∀ j ∈ {1, . . . , M} 19: D ← Proj D (D) 20: return {x (i ) } N i =1 , (D, V)
All these properties make both relaxations very appealing as solutions to the original problem by theoretically breaking the complexity of the adversarial attack problem.

LIMANS results

This section presents the experimental evaluations of the adversarial noise space and adversarial perturbations generated with it, providing a comparison with the state-of-the-art attacks on benchmark datasets.

Our experiments are conducted on the three datasets: MNIST, CIFAR-10, and ImageNet all presented in Section 1.1.6. As suggested in [START_REF] Zhang | A survey on universal adversarial attack[END_REF], we perform the experiments only on the validation set and split it into three parts, the first set for training the model D, the second for the tuning of λ when using Regularized-LIMANS and the last one for testing. MNIST Experiments. The number of original examples for training, validation, and testing is respectively 8000, 1000, and 1000. The experiments of the proposed model and the robustness estimation were conducted on the trained LeNet classifier [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] achieving a validation accuracy higher than 98.8%. These experiments have been implemented in Pytorch on a MacBook Pro with 2,3 GHz Intel Core i9, 8 cores, and a GPU Nvidia RTX 2080. CIFAR-10 Experiments. The number of original examples for training, validation, and testing is respectively 8000, 1000, and 1000. The experiments of the proposed model and the robustness estimation were conducted on the pre-trained VGG11 with batch normalization and the robust ResNet-18 [START_REF] Sehwag | Robust learning meets generative models: Can proxy distributions improve adversarial robustness?[END_REF] classifier. The transferability of the proposed model has been evaluated over four vanilla DNNs, i.e., MobileNet-V2, ResNet50, DenseNet121 and the VGG11 as aforementioned, and two robust DNNs, robust ResNet-18 2 and robust WideResNet-34-10 (R-wr-34-10) 2 [START_REF] Sehwag | Robust learning meets generative models: Can proxy distributions improve adversarial robustness?[END_REF]. These experiments have been implemented in Pytorch on a MacBook Pro with 2,3 GHz Intel Core i9, 8 cores, and a GPU Nvidia RTX 2080. ImageNet Experiments. The number of original examples for training, validation, and testing is respectively 10000, 2000, and 5000. We select here the 4 vanilla classifiers, ResNet-18, MobileNet-V2, DenseNet121, and VGG11 and two robust classifiers available on RobustBench 2 , robust ResNet-18 and robust WideResNet-50 [START_REF] Salman | Do adversarially robust imagenet models transfer better?[END_REF] (that we respectively rename as R-r18 and R-wr50 for simplicity). The experiments on a large-scale dataset were performed on a server equipped with 4 GPU Volta V100-SXM2-32GB.

Bridging the gap between specific and universal attacks

With LIMANS, we had better chances to reach a good modeling of the adversarial noise space, than the one that resulted from the ADiL projected gradient solution. Indeed, by relying on projection functions, the adversarial budget is always reached (sometimes overcome with Regularized-LIMANS), allowing D to capture the most relevant information to fool the classifiers.

To check the good modeling of the adversarial noise space, we solved the original problem (2.1) with the fast and reliable Simple-LIMANS solver to check that by increasing the size of the modeling (the number of atoms), its performances increased, sanity checking the overall proposition. Figure 2.14 displays Simple-LIMANS solutions for various numbers of atoms ranging from 1 to 4000.

The four figures highlight that when setting M = 1, LIMANS always reach a D solution spanning adversarial noises more effectively than universal attack solutions. This point is essential to sanitychecking the good modeling of the adversarial noise space. Indeed, by having both the b offset relaxation and the v (i ) coefficient to the universal D, LIMANS is always supposed to be better than the universal attacks solution. This necessary condition is indeed met, by a large margin for a Standard classifier and a smaller margin for a Robust classifier, which is natural because of the more complicated decision boundaries, making universal adversarial perturbations even harder to find. When setting the number of atoms to the input dimension M = P, a solution D exists being the basis of R P , from which all possible adversarial noises can be computed. Therefore, a second necessary condition of the LIMANS good modeling is that when setting M = P, LIMANS performances should get very close to state-of-the-art specific adversarial attacks. As we can see in Figure 2.14a and 2.14b, when applied on both a Standard and a Robustly trained deep classifier VGG11, the LIMANS 2 performances do reach the state-of-the-art specific adversarial attack baselines performances when M = P ≈ 4000. However, for the ∞ norm, things are a bit more sensitive. Indeed, we see a significant gap with the state-of-the-art specific attacks even when M = P. This bias comes from our Algorithm 8, which does not look for a solution D as a basis of R M , thus introducing a bias when the atoms are juxtaposing, preventing to possibility to reach all possible specific adversarial perturbation from D. Empirically when considering the ∞ norm, all atoms found after training are juxtaposing, confirming the introduced bias and explaining the gap found with state-of-the-art specific attacks, whereas, with the 2 norm, the atoms do not look alike. These atoms are displayed and analyzed in Section 2.6.2.

From these conclusions, we understand that the optimization problem is well-defined and empirical modeling results are acceptable. However, training performances do not assess a fair comparison with the state-of-the-art baselines, test performances only do. Therefore we used these D solutions found in Figure 2.14 and solved its paired inference problem (2.16) on the test set to fairly compare LIMANS with the state-of-the-art baselines.

Figure 2.15 tracks the test fooling rate of LIMANS for different numbers of the adversarial atoms under the 2 and ∞ norm constraint. It shows that the LIMANS attack is always stronger than universal baselines as even for only a single atom M = 1, LIMANS allows tuning the v coefficient factor making it more efficient. We see that from M = 500, LIMANS closes the gap with state-of-the-art specific adversarial attacks. This result is interesting as it empirically shows that by tuning the number of atoms M, the proposed LIMANS does bridge the gap between specific and universal adversarial attacks.

Besides reaching specific attack performances, by setting M = 500, the LIMANS attacks achieve competitive results by only optimizing for a coding vector v of dimension 500 whereas specific attacks need to optimize 3072 variables. Furthermore, such a result confirms the manifold hy- pothesis of the adversarial noise space thus confirming the efficiency of the quick, parameter-free Simple-LIMANS solver. By relying on a linear LIMANS model, users are offered the to visually inspect the adversarial model's parameters. Figure 2.16 displays the optimized model's atoms when M = 5. It shows how the atoms, the roots of LIMANS's adversarial leaks, are structured. This structure differs according to the classifier and the considered p norm. In particular, the dictionary of LIMANS-2 on the robust classifier is reminiscent of certain Fourier decompositions.

Visually interpretable atoms

In this section, we inspect the LIMANS parameters (atoms of D) found after the training stage and empirically observe interesting patterns that we detail. We give an empirical review of the three mainly used datasets, that is CIFAR-10, ImageNet, and MNIST.

CIFAR-10 dataset

The CIFAR-10 dataset is a good dataset for analyzing the performances of adversarial perturbations as it is a decently complex dataset appropriately representing the real world by offering RGB images of 10 labels each very different from one another. Therefore the claim that the atoms are visually interpretable must be at least proven with this dataset. By modeling the adversarial noise space, we empirically observe that LIMANS' parameters capture the most meaningful information fooling the classifier. Figure 2.16 displays the optimized LIMANS' atoms when M = 5, and presents interesting results. Figure 2.16a and Figure 2.16b display LIMANS parameters for a Standard and a Robust classifier under the 2 norm constraint. As we can see, under the 2 norm constraint, the atoms do not look alike at all and target locally interesting spots within the images, such that specific adversarial perturbations can be computed, taking advantage of these "pitfalls" for which the classifier is the weakest. Figure 2.16a and Figure 2.16d however, consider the ∞ norm constraint. As previously mentioned, all atoms display the classifier's vulnerabilities for every pixel value, preventing the overall D from being a basis of R P and thus limiting the number of possible specific adversarial perturbations that can be computed from. It is interesting, however, to see that even if this modeling of the adversarial noise space is not optimal, the atoms are nevertheless interesting to inspect. Indeed, we see that for the Standard classifier, the atoms capture the corners and edges of the objects, which are the most discriminating features used by the classifier to make its prediction.

In some way, we can read the LIMANS' atoms as a capture of the semantics of the objects within the classification space. Figure 2.16 highlights very particular atoms that indeed spotlight recurring patterns in classification such as edges and corners for the ∞ atoms and local spots in the images for 2 atoms.

MNIST dataset

For atom visualization only, we also considered experiments on the MNIST dataset. Indeed, the MNIST dataset offers black-and-white images with pixels in very small manifold values, therefore making the problem too simple for an adversarial perturbation study with deep classifiers. Very few works in the adversarial community target the MNIST dataset. In our case, since we only want to analyze the parameters found after the training, it still is worth looking at the result.

Along with Figure 2.16, we also show that the same conclusions can be reached for simpler datasets such as MNIST with Figure 2.17,

In addition to these visually interesting universal parameters, this empirical observation is further proven when inspecting the LIMANS produced specific adversarial perturbations.

Figure 2.18 displays LIMANS-specific adversarial perturbations on the MNIST dataset (purposely used to ease the observations). LIMANS produces much more interesting adversarial perturbations than state-of-the-art specific adversarial perturbations that are random. LIMANS however, target the most sensitive spots to fool the classifiers, which makes LIMANS adversarial attack a much more realistic attack than state-of-the-art specific adversarial attacks.

A more robust adversarial attack

In this Section, we observe that LIMANS offers a more robust adversarial attack than state-of-theart specific attacks. Table 2.3 and 2.4 present the RAUD of LIMANS and state-of-the-art specific adversarial attacks on the CIFAR-10 dataset under respectively the ∞ and 2 norm constraint. More details about the RAUD metric are provided in Section 1.2.3.

Table 2.3 and Table 2.4 shows the performances of the robustness of the proposed ∞ and 2 LIMANS attack and its comparison to the specific adversarial attack baselines targetting a Standard and a Robust classifier. These experiments are conducted with the same settings as in [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF]. Lorenz et al. [START_REF] Lorenz | Is autoattack/autobench a suitable benchmark for adversarial robustness?[END_REF] highlights that state-of-the-art specific attacks become nearly or completely harmless when the classifier is protected by an adversarial examples detector. Under the ∞ norm constraint, we find similar results and, even the proposed LIMANS attack surpasses specific attacks when M is only 10.

D 1 D 2 D 3 D 4 D 5 D 6 D 7 D 8 D 9 D 10 (a) LIMANS-2 . D 1 D 2 D 3 D 4 D 5 D 6 D 7 D 8 D 9 D 10 (b) LIMANS-∞ .
With M ≥ 500, the proposed LIMANS attack can successfully jeopardize the classical classification system, even equipped with an attack detector. Using a detector before a classification system is one kind of adversarial defense, presented in Section 1.2.5. The robust classifier shows its stability facing adversarial attacks while the proposed LIMANS attack can also ignore the attack detectors and damage the system to some extent. It indicates thus the potential application of the proposed attacks in evaluating the effectiveness of an adversarial example detector plugged prior in a classical or robust classifier.

To a lesser extent, we draw similar conclusions for the 2 norm constraint. Indeed, when the classifier is protected by a specific attack detector (detector trained to recognize a specific attack's adversarial examples), LIMANS is almost always a more harmful attack. These conclusions are questioned when the detector is trained to recognize LIMANS' adversarial examples. Indeed, when doing so, LIMANS' performances are never better than state-of-the-art specific attacks. However, Table 2.4 tells us that using a LIMANS detector is a poor defense choice. Indeed, before considering any defense detector, Figure 2.15 highlights the natural superiority of specific attacks, which inherently leads to consider a specific attack detector as the most effective defense. Even when using a LIMANS detector, specific attacks RAUD is lowered empirically showing the LIMANS framework is relevant either as an adversarial attack or a defense mechanism.

Under the 2 norm constraint, using a LIMANS detector will indeed protect from LIMANS adversarial examples, but also from state-of-the-art specific attacks, which indirectly showcases the

Details about the RAUD detector d

Regarding the design of a descent adversarial examples detector, we followed the guidelines proposed by [START_REF] Lorenz | Is robustbench/autoattack a suitable benchmark for adversarial robustness?[END_REF] and [START_REF] Harder | Spectraldefense: Detecting adversarial attacks on cnns in the fourier domain[END_REF]. In these two works, authors propose to use a random forest binary classifier with at least 100 trees, in the Fourier domain either of the input images or of the Fourier Features of Feature-Maps from the images. Authors showed that by using either one of the inputs, the random forest binary classifier can discriminate with high precision adversarial examples computed from state-of-the-art specific adversarial attacks on several complex datasets such as CIFAR10, CI-FAR100, ImageNet, or Celeba. To lower as much as possible the bias introduced by the detector, we chose to use a random forest binary classifier with 300 trees in the Fourier domain of the input images. Our choice has been confirmed by extensive experiments and is highlighted in Table 2.5. Indeed, Table 2.5 displays the confusion matrix of the detectors involved in the results presented in Table 2.3.

The detectors have been trained on the same training dataset as the one used in the training of LIMANS and, the displayed values computed over the validation dataset. Finally, the RAUD metric of the different adversarial attacks using the detector is performed on the test dataset which none has previously seen before, making everything fair.

We empirically observe highly effective detectors discriminating almost perfectly real images rather than adversarial images, without mistaking one or the other by producing low False-Positive and False-Negative values. These performances gave us confidence in the use of these detectors as a tool of the RAUD metric, evaluating both the harmfulness and the transferability of adversarial attacks, which are essential to be measured when assessing the quality of an adversarial attack.

Transfer performances

The results in this part were generated using the algorithm Regularized-LIMANS.

We present a comparison with the specific ∞ adversarial attacks AutoAttack and three stateof-the-art transferable ∞ attacks that are, VNI-FGSM [START_REF] Wang | Enhancing the transferability of adversarial attacks through variance tuning[END_REF], NAA [START_REF] Zhang | Improving adversarial transferability via neuron attribution-based attacks[END_REF], and RAP [START_REF] Qin | Boosting the transferability of adversarial attacks with reverse adversarial perturbation[END_REF]. Besides, we consider also Translation-Invariant Attack (TI-FGSM) [START_REF] Dong | Evading defenses to transferable adversarial examples by translation-invariant attacks[END_REF], VMI-FGSM [START_REF] Wang | Enhancing the transferability of adversarial attacks through variance tuning[END_REF], and universal ∞ attacks UAP and UAPPGD. However, the well-known DI-FGSM, MI-FGSM, or NI-FGSM are not selected to compare, as all these processes are already integrated into the RAP attack. The LIMANS 2 attack is compared with the 2 version of AutoAttack, UAP, UAPPGD, and RAP, as well as the CW adversarial attack.

By adjusting the hyper-parameter λ of the Regularized-LIMANS solver, we managed to find an optimized M = 150 for CIFAR-10, which achieves almost comparable performance with AutoAttack and to analyze the transferability of the learned space without loss of precision as shown in Table 2.6. With the ImageNet dataset (Table 2.8), considering the large size of an image and the memory limitation, this dimension becomes M = 100. It is far from the dimensionality of the full adversarial noise space, and hence, does not lead to comparable performance with specific attacks such as AutoAttack. However, it still offers evidence for the transferability of the learned space on ImageNet. Moreover, for robust classifiers, the decision boundaries are more complicated, leading to failure in closing the performance gap between the LIMANS attack and AutoAttack when only 100 adversarial directions are learned. Nevertheless, it always shows good performance in transferring.

Tramèr et al. [START_REF] Tramèr | The space of transferable adversarial examples[END_REF] claimed that the distance between the decision boundaries of two classifiers trained on the same dataset is small. It means that the LIMANS model of the adversarial noise space is likely to be transferable between classifiers. The results reported in the Table 2.6 confirm this intuition. The LIMANS model builds upon a ResNet50 and a VGG showing better transfer performance over state-of-the-art attacks across other classifiers.

Besides, note that the transferable property also holds between vanilla and robust classifiers. The LIMANS model learned on a vanilla classifier used to fool a robust classifier (and vice versa) produces slightly worse results than the one learned on a classifier of the same category. It might be due to the differences in the dataset split used to train the classifiers, resulting in a larger distance between the decision boundaries of the two classifier types. Yet the performance is still remarkable, e.g., on ImageNet, FR (R-wr50→ResNet-18) = 78% FR (VGG→ResNet-18) and FR (ResNet-18→R-r18) = 35% FR (R-r18→R-r18) (with R-r18 standing for Robust ResNet-18 and R-wr50 for Robust WideResNet-50).

Furthermore, it is worth noting that the performance of LIMANS attack on a target classifier does not depend on its performance on the source classifier, but on the nature of this target classifier. In Table 2.6, the fooling rate of the LIMANS attack on ResNet is 91.3%. However, when the learned LIMANS model generates adversarial perturbation to fool MobileNet, its performance is even better, reaching 96.0%. This is because MobileNet is a simpler and easier classifier to attack. Finally, through comparison and analysis, we conclude that a LIMANS model trained on a robust classifier is more easily transferable to other classifiers. Generally, the model that is harder to be fooled, shows ascendancy performances in transferring to other models.

Time consumption analysis

By relying on a model of the adversarial noise space D ∈ R P × M with M P, the LIMANS adversarial attack problem optimizes a vector v ∈ R M rather than an adversarial perturbation ω ∈ R P . Intuitively, by optimizing a lower number of variables, the LIMANS attack breaks the time complexity of the adversarial example problem therefore, should take much less time to compute adversarial examples.

In Figure 2.19, we conducted an experiment comparison of the time consumption of LIMANS and state-of-the-art specific adversarial attacks to confirm or not this intuition. dataset. On the y-axis is displayed the Robust Accuracy of the classifier, the lower Robust Accuracy is, the more efficient the adversarial attack is. On the x-axis is the time taken (in seconds) to compute adversarial examples over all the test sets of CIFAR-10, the lower the time is, the more efficient the adversarial attack is.

As shown in Figure 2.19, LIMANS does not improve the time consumption of the adversarial example crafting problem. Indeed, LIMANS is always dominated in time consumption by almost every specific attack. The only exception is Autoattack with the 2 norm constraint, which takes much more time to reach the best Robust Accuracy. We explain this empirical counter-intuitive observation by highlighting that specific adversarial attacks rely on enhanced and efficient optimization procedures to find ω ∈ R P , whereas LIMANS optimizes problem (2.17) in a very basic way by relying on an SGD optimization that is not very efficient in time.

Figure 2.19 leads us to consider improving the inference optimization problem, as an important perspective for future work. Indeed, improving the time consumption of LIMANS would likely place it as a strong, more robust, and more transferable attack compared to state-of-the-art specific attacks. Section 2.8 details the envisioned LIMANS perspectives.

Insights on the choice of M

Figure 2.15 tells us that the choice of M is crucial regarding the performances of the derived LI-MANS attack.

From our CIFAR-10 experiments, we found it interesting to choose M = 500 as, in terms of RAUD the LIMANS 500 attack already overcomes state-of-the-art specific adversarial attacks. By setting M = 500, the adversarial examples creation problem boils down to optimizing 500 variables, representing only 16.27% of the original image size. As for ImageNet which is a much more complex and bigger dataset, our experiments suggest that setting M = 150 is a good trade-off between the LIMANS performances and the space complexity to store such an adversarial noise model in GPU memory.

Optimizations hyper-parameters

This section presents the different implementation choices made for the Regularized-LIMANS and Simple-LIMANS experiments. We conducted several sanity-check experiments to enhance the LIMANS optimization. The following subsections detail each sanity-check experiment.

Regularized-LIMANS hyper-parameters and settings

Figure 2.20 illustrates the impact of the hyper-parameter λ and the number of atoms M on the attack performances. With appropriately increasing M, the performance will be improved, which confirms the conclusions drawn from Figure 2.15 and Figure 2.19. Considering the trade-off between the attack performances and the memory limit of storing the atoms, we choose to use M = 150 for CIFAR-10 and M = 100 for ImageNet when relying on the Regularized-LIMANS solver to estimate the best λ hyper-parameter. From Figure 2.20 we understand that when considering the ∞ norm constraint, setting λ = 1 provides the best performance, while λ = 0.1 is the suitable setting when dealing with the 2 norm constraint. This conclusion is valid when extending to other classifiers, as shown in Table 2.9.

Simple-LIMANS hyper-parameters and settings

The main idea behind Simple-LIMANS is to optimize the LIMANS parameters in the simplest way possible, with the least hyperparameter tuning and optimization details possible. However, few remain. We detail here their impact and how we selected those.

Simple-LIMANS ablation study of the offset b

As of good practice, and to see that Simple-LIMANS was indeed capturing the right amount of information without promoting too much the offset b, we performed an ablation study of the LIMANS performances with and without the offset b, We can see that LIMANS' performances grow accordingly without any unseen artifacts captured by b. Besides, through this figure, we understand its relative importance alongside the atoms of D that constitute the specific adversarial noise of each original example using its corresponding coding vector. The weight of b relative to the atoms is reasonable, and the balance between both in the implication of the fooling performances is respected.

Using an offset b improves the performances of the LIMANS attack, but is not necessary for the well-being of both the modeling of the adversarial noise space and the crafting of new adversarial perturbations. The offset relaxation of D exactly does what it is intended to do and, sanity checks the optimization procedure.

Learning rate

No regularization parameter tuning is needed and the learning rate is automatically taken care of, by using the scheduler ReduceLROnPlateau. The ReduceLROnPlateau scheduler is a tool proposed by the PyTorch framework, which aims at reducing the learning rate when the loss plateaus. The optimization needs to start with a pretty high learning rate which will avoid bad random initialization local minima, and then will decrease as the optimization proceeds, allowing the parameters to reach a better minimum. The parameters of the used ReduceLROnPlateau are pa-tience=40, factor=0.1, and threshold=1e-1.

Optimizer

All the Simple-LIMANS optimizations were performed with the Adam optimizer. We found the Adam optimizer to be the best among several ones, as shown in Figure 2.22a. Indeed, similar performances can be reached with a different optimizer such as the RMSProp optimizer, but overall the Adam optimizer or one of its variants seems to be a relevant optimizer choice.

Batch size

As Figure 2.22b shows, when using the Simple-LIMANS solver, the batch size B is not a sensitive hyper-parameter. We found that different batch sizes could end up with similar performances. The only difference resides in the time consumption, higher batch sizes yield faster computations. During our experiments, the batch size B was set to B = 256 during training and B = 64 during inference.

Multi-LIMANS

The most obvious extension of LIMANS we thought of, was to make it an ensemble attack coined Multi-LIMANS. In the search for the optimal adversarial noise space model we proposed to learn a shared D on multiple classifiers f k , each possessing their specific training coding vectors {v (i k } N i =1 , and see if, at inference, the specific adversarial noises would be more efficient or more transferable. Simple-LIMANS trained D. We empirically saw that learning D on multiple classifiers rather than only one did not improve the universal adversarial direction but introduced more bias, preventing it from reaching the inference-specific adversarial noise which was originally at hand.

Another envisioned improvement that Multi-LIMANS could bring in, was to improve the transferability. We knew that by learning on multiple classifiers, the dictionary was not better at inference for those classifiers, but did it generalize more and therefore performed better on other classifiers it has not yet been applied to, Unfortunately, the bias introduced by learning D on multiple classifiers does not balance with the multi-classifiers' learned property. As Figure 2.24 shows, inference performances are never

Conceptual questions raised by our work

In this section, we detail the conceptual questions raised by LIMANS, the originality of the proposed method, and the vision implied behind it. Which gap does it aim to fill and which path does it pave the way to.

Common corruptions or p norm bounded adversarial examples?

To evaluate the robustness of classifiers, relying on perturbation to assess its empirical limits is the most straightforward choice. However, defining such perturbation remains hard to do. On the one hand, most research adopted the use of p norm bounded perturbation specific to each example to assess the almost exact limit of the classifier's decision boundaries, giving us a precise definition of its robustness (detailed in Section 1.2.4). On the other hand, some research groups criticized this choice by stating that such p norm bounded perturbation specific to each example, is not realistic and therefore produces useless and incomplete evaluations of the classifier's robustness. They proposed a set of common corruptions, allowing us to assess a more precise and realistic robustness evaluation of the classifier (detailed in Section 1.2.1). Even though such common corruptions seem appealing, they don't allow to reach the classifier's decision boundaries making the robustness evaluation more realistic but unfortunately more limited than optimized adversarial attacks.

With LIMANS, we propose to use the rely on p norm bounded perturbation specific to each example to reach the classifier's decision boundaries, making the robustness evaluation precise, although to compute specific p norm bounded perturbations on a more concrete basis than randomly drawn among each feature of the examples. This more concrete basis is modeled by the universal atoms that we reach at the end of the LIMANS training procedure (detailed in Section 2.5), and they reveal interesting universal patterns always involved in the reach of the classifier's decision boundary. In a way, the LIMANS training procedure consists of the capture of the semantics of the objects within the examples and using these extracted features to overcome the classifier's decision boundaries. By relying on the optimized semantic of the object, LIMANS allows computing more realistic adversarial perturbations (see in Section 2.6.2), thus performing a more precise robustness evaluation of classifiers, in a way that the decision boundaries are more precisely reached than with common corruptions, and realistic, in a way that the computed specific p norm bounded perturbations are much more object-related and therefore more realistic than other specific p norm bounded perturbation.

We strongly believe that neither the specific p norm bounded perturbations nor the common corruptions are necessary nor sufficient to efficiently asses a precise and credible robustness evaluation of the classifiers.

LIMANS is just a humble proposition aiming to bridge the gap between these two paradigms, welcoming a reassessment of what has already been proposed, an honest discussion on the usefulness of unrealistic robustness evaluation, and a criticism of the effectiveness of common corruptions to estimate the classifiers' robustness.

Specific or universal adversarial perturbations?

Among the p norm-bounded perturbations research community, previous criticisms have already been made. Essentially people denied the specific p norm bounded perturbations because they are totally specific to each example, allowing them to reach almost exactly the classifier's decision boundary, but on this very example alone. Evaluating the classifier's robustness on an empirical dataset using specific p norm bounded perturbations will only give a robustness measure for this dataset alone and nothing more. This limitation makes the robustness evaluation using specific p norm-bounded perturbations, impractical and useless in real-life applications. The original goal is to estimate the overall robustness of the classifiers on every example, even the one we do not have access to, yet.

To resolve this limitation of specific p norm-bounded perturbations, several authors proposed to use instead universal p norm-bounded perturbations (detailed in Section 1.2.4). These universal p norm-bounded perturbations are example-agnostic, making them most likely effective even on future examples we do not know of yet. Universal p norm-bounded perturbations are very interesting as they target universal weaknesses and pitfalls of classifiers, to which they are completely broken and are always fooled.

Therefore, universal perturbations must always be included within the robustness evaluation of classifiers as they are, by definition, always included in the set of perturbations we want to model. However, universal perturbations alone are not sufficient to entirely assess the classifiers' robustness. Indeed, by being fully universal, these perturbations do not allow us to get close to the exact classifier's decision boundaries. They are too limited, and don't explore the example space to the fullest allowing to exactly assess the robustness of classifiers.

With LIMANS, we propose to take advantage of both the specific and universal paradigms. Indeed, LIMANS is one of the first propositions bridging the gap between specific and universal p norm-bounded perturbations by computing specific perturbations upon a universal perturbation basis. LIMANS almost always capture the universal perturbations necessary to assess the most general robustness of classifiers but also allow us to explore the example space to reach more precisely the classifier's decision boundaries.

LIMANS framework allows drawing robustness evaluations that are more generalized than specific p norm bounded perturbations alone, and more efficient than universal p norm bounded perturbations alone.

We humbly believe the current interest in universal perturbation is too limited, and that a correction of their value is essential to the sanity of robustness evaluation research. Rather than blindly running after the improvement of some performance metric, a deep questioning of the current work and its importance should be made, engaging in more honest, fair, and unbiased research.

Maximizing the fooling rate or maximizing the perturbation quality?

We previously saw that defining the perturbations involved in the robustness evaluation of classifiers is hard. However, once a definition is chosen, which metric should be considered to define the robustness of the classifier? The choice of the quality metric is crucial to the robustness evaluation. Section 1.2.3 details different quality metrics and explains what vision and goal each envisions. With different considered goals and metrics, a conflict of interest may arise in the robustness evaluation according to who is performing it.

We believe that a framework and constraints (just like the adversarial budget constraint) should be imposed, allowing the different propositions to be fairly compared and benchmarked. However, such a framework alone is not sufficient. Indeed, the final metric assessing the robustness of the classifier should also but carefully considered, preventing a possible conflict of interest.

To produce the least biased possible robustness evaluation, we believe that future work should explicitly state the used quality metric and to whom it is profitable, informing new readers of the possible limits and advantages of each metric.

We currently are not aware of the best robustness metric. However, we think the perfect robustness metric should integrate every intent of the robustness, whether it is its protection from possible harm or performance under possible hardware bugs.

As the deep community would suggest, one idea could be to parameterize such robustness metric with a neural network framework. The framework would take in the classifier, the original examples, and the perturbed examples outputting a robustness score. Even though it is very straightforward and almost simplistic, this approach would be fair to everyone as the same framework metric would be used, under different classifiers with the same adversarial attack producing the perturbed dataset, or the same classifier under different perturbed datasets or different classifiers. Hence, every new proposed perturbation generation would get a fair robustness score, and every new proposed classifier would receive a fair robustness score.

We believe that fairness and avoiding possible bias should be the center of future research engaging in honest and transparent discussion about new propositions.

Perspectives

In itself, LIMANS is already a complete proposition on which we spent several months to converge, but it still offers very appealing perspectives that we, unfortunately, did not have enough time to complete. LIMANS framework is one of its kind in the robustness evaluation community, discussing several points that are widely accepted and currently considered sufficient. LIMANS' perspective includes reproducing the ADiL experiments that we did not have enough time to do, such as the training stage with fewer original examples. Do the LIMANS performances and generalization grow according to the number of training examples, or by its linear definition is it immune to the training dataset size?

One original and very interesting ADiL idea was to derive a black-box adversarial attack. We believe that LIMANS could easily be subject to the same extension. A LIMANS black-box attack would step further the realism of the attacker's behavior, estimating the classifier's robustness in an even more realistic way.

Obviously, after proposing the LIMANS adversarial attack, we must propose the LIMANS adversarial defense to contribute to research under the two scopes of attacker and defender.

We also target to impose more constraints on the LIMANS atoms. Indeed, we believe the adversarial patch attacks are very interesting and appealing to estimate a realistic classifier's robustness. We think that, after a careful design of the constraints, we could make LIMANS parameters a dictionary of atoms producing adversarial patches, which could be a real improvement and innovation brought to the research community, going further into the design of realistic yet efficient perturbations.

During our experiments, we mainly focused on images but by definition, LIMANS could be applied to any kind of data modalities as long as we can back-propagate gradients and we do have a similarity metric between the objects allowing us to respect the adversarial budget constraint.

Chapter 3

A meaningful similarity metric

Observation

Usually, colored images are mathematically encoded as arrays of dimension the number of pixels in which each pixel color is represented by triplets of RGB values. Thus for a colored 24×24 image, its mathematical representation will be defined as x ∈ R 24×24×3 = R 1728 . Therefore, to estimate the similarity of two images, mathematical tools like p norms are well suited. Indeed, p norms define distances d : R P × R P → R + , thus respecting the three following axioms, Symmetry:

∀x (1) , x (2) ∈ R P , d (x (1) , x (2)
) = d (x (2) , x (1) )

Equality:

∀x (1) , x (2)
∈ R P , d (x (1) , x (2) ) = 0 ⇔ x (1) = x (2) Triangular inequality: ∀x (1) , x (2) , x (3) ∈ R P , d (x (1) , x (3) ) ≤ d (x (1) , x (2) ) + d (x (2) , x (3) )

(3.1)
Having a distance is interesting as it offers a certain flexibility in its use, and produces unbiased results. Besides, p norm distances scale very well in dimension, which is critical when computing the similarity of two images. The image dimension can quickly increase, easily making the complexity of the problem troublesome, therefore, an appropriate metric is required.

Even though p norms are very convenient tools, considered as relevant image distances, they hardly compute the appropriate distances between the images. Indeed, in Figure 3.1, we see two cat images, then the distance between these two images should be low as the same object is displayed, only the cat's color and background change, but in principle, they both represent the same idea. Figure 3.1 displays a third image containing a plane with a vaguely similar background color to the second cat but represents a different idea. Figure 3.1 highlights the meaningless property of p norms to compute image distances as the plane is closer to both cats than the cats together.

Even though they are efficient mathematical tools, p norms hardly capture the meaning of the images and therefore represent a poor distance function choice.

Up until now, the majority of state-of-the-art pseudo-random adversarial attacks considered an p norm to be a relevant distance between images [START_REF] Sharif | On the suitability of lp-norms for creating and preventing adversarial examples[END_REF]. It led to generating incomprehensible and unrealistic adversarial perturbations still considered state-of-the-art. Opposite to pseudorandom attacks, more realistic attacks were proposed, struggling to efficiently compute meaningful distances between images and therefore, generate realistic and meaningful adversarial perturbations.

Proposition

The goal of this second, more exploratory, project of this thesis is, to come up with a meaningful distance between images leading to generating more realistic adversarial perturbations computed from the very efficient state-of-the-art pseudo-random adversarial attacks. When possessing such a meaningful distance, the original goal, to generate realistic and efficient adversarial perturbations, could be reached, letting us evaluate more precisely and on a realistic basis, the robustness of classifiers.

We mainly relied on [START_REF] Sharif | On the suitability of lp-norms for creating and preventing adversarial examples[END_REF] to outline the cases for which p norms can be used and when they should not be. Readers are invited to refer to it for a more comprehensive and detailed explanation of why p norms are neither sufficient nor necessary when used as a similarity metric of images for adversarial perturbation generation.

To ease the problem, our objective is to approximate a meaningful similarity metric, that is not a distance and thus allows us to violate the three axioms 3.1 defining a distance. Given a dataset D = {(x (i ) , y (i ) )} N i =1 of labeled examples with x (i ) ∈ X ⊂ R P and its corresponding label y ∈ Y = {1, ..., C}, we want to approximate a distance d : R P × R P → R + such that the points of same labels receive low distances (high similarity), while points of different labels receive high distances (low similarity), min The most obvious choice for such a parameterized distance is the Mahalanobis distance,

d C x (i ) ∈Y k x ( j ) ∈Y k d C (x (i ) , x ( j ) ) - x (i ) ∈Y k x ( j ) ∈D\Y k d C (x (i ) , x ( j ) ), ∀Y k (3.2) with Y k = {x (i ) } N i =1 , y (i ) = k the
d C (x (1) , x (2) ) = (x (1) -x (2) ) T C(x (1) -x (2) ), (3.3)
with C ∈ R P × P a positive semi-definite matrix, in which the diagonal values C i i serves as a weights of the difference values [x (1) x (2) ] i i .

A Mahalanobis distance defines a weighted version of the p norm. Even though it would still be interesting to explore, such a Manhattan distance only considers the identical relationships between the input entries. Only the diagonal values of the matrix C are considered to compute d (x (1) , x (2) ), resulting in a restrained distance and allowing little flexibility in the approximation. Similar to our proposed adversarial dictionary attack proposition, modeling comprehensible and understandable C parameters of the distance is a desired property. Therefore, the distance would allow us to understand why two images are close (a low distance value representing high similarity) or why they are considered different (a high distance representing low similarity).

To extend the Mahalanobis distance proposition, and offer more flexibility we resorted to optimal transport distances.

Wasserstein Distance as a meaningful distance between images

Optimal transport distances, mostly known through the Wasserstein Distance (WD), are a set of probability distribution distances, that provides a way to compare how different two distributions are, based on the cost of "moving" mass from one distribution to the other according to a cost matrix C.

Given two probability distributions represented by empirical measures a ∈ R N and b ∈ R N , the WD aims to find the most efficient way to transform the first distribution a into the second b, where efficiency is defined by a cost function C ∈ R N × N that represents the "distance" or "effort" needed to move mass from one point to another,

WD C (a, b) = min T∈R N ×M N i =1 M j =1 T i , j C i , j ,
such that T1 = a and T T 1 = b.

(3.4)
In itself, computing the WD C solves the optimization problem of finding the best transport plan T, conditioned to the cost matrix C. Figure 1.6 illustrates a valid transport plan between two points clouds.

The cost matrix plays a crucial role in understanding the transportation of mass between two distributions and has several interesting aspects regarding the modeling of a meaningful distance. First, rather than using only the diagonal values like a Mahalanobis distance, all entries of the cost matrix are necessary. Each entry in the matrix C i j represents the cost of transporting a unit of mass from one point a i to another b j . It allows more flexibility and richer modeling possibilities, as well as, an intuitive interpretation of the distance value between the distributions in terms of the transportation required and the associated costs. The WD C offers a more precise distance value, as well as, an interpretation of its computation. In some cases, the cost matrix can be designed to incorporate application-specific information about the relationships between data points. This allows for a more meaningful measurement of distance that aligns with the problem's context. All these intriguing and favorable properties make the WD C a strong candidate as a meaningful similarity metric between images.

Even though relying on the WD C to estimate the similarity between images has already been done, it always was in a limited or flawed context. In [START_REF] Wong | Wasserstein adversarial examples via projected sinkhorn iterations[END_REF], Wong et al. proposed the Wasserstein Attack, minimizing the WD C between image pixels to compute adversarial perturbations. The authors proposed an idea similar to ours, and showed meaningful on-manifold adversarial perturbation results. However, their computed WD C was revealed to be flawed. Indeed, in [START_REF] Sen | Should adversarial attacks use pixel pnorm?[END_REF], authors demonstrate the irrelevancy of any image measure based on their pixels. Distances studied by [START_REF] Sen | Should adversarial attacks use pixel pnorm?[END_REF] include the WD, preventing the former Wasserstein Attack from becoming a long-lasting relevant solution to compute meaningful adversarial perturbations of images. However, the displayed empirical results are promising and encourage our pursuit in this direction.

The WD C between images has especially mainly been considered to estimate the quality of image generations. Indeed, the GAN (defined in Section 1.1.4) community which received lots of attention in the last years, hugely adopted the Fréchet Inception Distance (FID) as the most effective tool to track the quality of the artificially generated images. The FID calculates the distance between two distributions of images f X and f Y , most often one is a minibatch of original examples drawn from the dataset, while the second distribution is a minibatch of GAN (or any other image generator) generated examples. Typically, the two distributions of images are represented by summary statistics (mean & covariance matrix) computed over the outputted values of the third pool layer of an inception neural network for all images, acting as a feature extractor. The FID between CHAPTER 3. A MEANINGFUL SIMILARITY METRIC the two distributions is defined as,

FID( f X , f Y ) = ||µ X -µ Y || 2 2 + Tr (Σ X + Σ Y -2(Σ X Σ Y ) 1/2 ). (3.5)
Specialized variants of FID have been suggested as evaluation metrics for music enhancement algorithms as Fréchet Audio Distance (FAD) [START_REF] Kilgour | Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms[END_REF], for generative models of video as Fréchet Video Distance (FVD) [START_REF] Unterthiner | Fvd: A new metric for video generation[END_REF], and for AI-generated molecules as Fréchet ChemNet Distance (FCD) [START_REF] Preuer | Fréchet chemnet distance: a metric for generative models for molecules in drug discovery[END_REF].

Even though the FID (3.5) may seem a bit different than the WD C (3.4), they are defining the same thing, only when the cost matrix C is computed using the 2 norm and under the assumption that both distributions can be represented by their mean and covariance matrix. For the curiosity of readers, the proof is given in Section A.3.

Opposite to the FID, in our case, we want to compute the WD C not between two minibatch of images but between two images alone. However, as [START_REF] Sen | Should adversarial attacks use pixel pnorm?[END_REF] outlines, the WD C defines a probability distribution similarity metric making it irrelevant to apply it directly over the images' pixels like [START_REF] Wong | Wasserstein adversarial examples via projected sinkhorn iterations[END_REF] already did. It led to consider instead, extracting features from the images to represent them. Thus when using extracted features, we operate the WD C in the meaningful framework making it an appropriate and promising tool applied in the best settings. The extracted features, by definition, contain the most meaningful information about the images, offering us more chances to end up with the definition of a meaningful distance between images. By using extracted features, we now control the complexity of the problem such that, even for high-resolution images, the size of the extracted features is controlled and tuned making it a scalable distance.

In this scenario, however, we are two times conditioned. First, the optimization of the image distance is conditioned to the efficiency of the feature extraction g θ . Indeed, if an inefficient feature extractor is used, therefore no relevant distance can be derived from it. Secondly and most troublesome, when a meaningful distance is finally modeled, then for the adversarial perturbation generation, we need to get back from the feature space to the original image space using an inverse function of the feature extractor g -1 θ . If one sees the feature extractor as an encoder projecting the images in a feature space of lower dimension, such that labels are well separated, in which we can easily approximate a distance, then its paired decoder g -1 θ is needed to retrieve images from the feature vectors. Figure 3.2 presents a high-level overview of our proposition. Both g θ and its inverse function g -1 θ are crucial to the efficiency of the proposition. Even though the second limit of representing the images by their extracted features by using a g -1 θ function, is probably the most troublesome, it still is a bit far ahead and most importantly considers a meaningful distance d already optimized, but one step at a time. First, we ignore the g -1 θ decoder issue and focus only on approximating a meaningful distance d between images from their features.

Optimization

In this section, we detail the similarity metric framework we designed and derive an overall optimization problem

Turning the similarity metric into a distance

In order to represent the images x (i ) ∈ R P we consider their features a (i ) ∈ R f with f P extracted from an external model g θ . For ease of reading we will refer to the extracted features a (i ) as an image even though its associated image is x (i ) . As previously outlined, we select the Wasserstein distance (WD C ) as the most appropriate function leading to a meaningful similarity metric between images. The Wasserstein distance (WD C ) between two images a (s) and a (t ) reads, WD C (a (s) , a (t ) ) = min

T∈R f × f f i =1 f j =1 T i , j C i , j ,
such that T1 = a (s) and T T 1 = a (t ) , (3.6) with the matrix C ∈ R f × f being the cost matrix containing the cost of pairing each feature entry of a (s) to each feature entry of a (t ) .

Recently, a lot of attention has been put to the regularised version of the Wasserstein Distance, called the Sinkhorn divergence. While the unregularized WD C and the Sinkhorn divergence are theoretically equivalent candidates, the Sinkhorn divergence is more suited to similarity metric optimization. Indeed, the Wasserstein distance involves solving a linear programming problem, which can be computationally expensive and challenging to solve, especially for high-dimensional data. In contrast, the Sinkhorn divergence utilizes the Sinkhorn-Knopp algorithm, a more efficient iterative method for approximating the optimal transport plan between distributions. For the readers' curiosity, the development from the original regularized WD C expression to the Sinkhorn-Knopp solver is given in Section A.2. Besides, the Sinkhorn divergence includes a regularization parameter helping to stabilize the optimization. This regularization encourages the transport plan to be smoother, improving convergence. By tuning the regularization parameter, Sinkhorn divergence allows for better control over the trade-off between the accuracy of the solution and the stability of the convergence. Overall the Sinkhorn divergence is a good choice as it is better suited to gradient-based optimization techniques commonly used in deep learning, and most likely used here. The Sinkhorn divergence between two images a (s) and a (t ) reads,

WD κ C (a (s) , a (t ) ) = min T∈R f × f f i =1 f j =1 T i , j C i , j + κ f i =1 f j =1
T i , j (log T i , j -1), such that T1 = a (s) and T T 1 = a (t ) , (3.7) with κ the regularization parameter. Rather than optimizing (3.7), usually, people rely on the dual formulation of the problem, min

T∈R f × f max α∈R f ,β∈R f L(a (s) , a (t ) , α, β), such that α ≥ 0 f , β ≥ 0 f with (3.8) L(a (s) , a (t ) , α, β) = min T∈R f × f f i =1 f j =1 T i , j C i , j + κ f i =1 f j =1 T i , j (log T i , j -1) + α T (T1 f -a (s) ) + β T (T T 1 f -a (t ) ),
with α ∈ R f and β ∈ R f the dual variables. This dual problem is efficiently solved using the gradient descent iterations of the Sinkhorn-Knopp solver, detailed in Section A.2.

Up until now, we supposed the cost matrix C between the image features known, however, we don't know such a suitable distance between the features living in R f it is precisely the cost matrix that contains the "meaningful" information defining the meaningful distance. The cost matrix optimization defines our similarity metric optimization.

To make the similarity metric a distance, it must respect the axioms of a distance (3.1). Therefore, the cost matrix is assumed to belong to the cone of distance matrices defined as

C = C ∈ R f × f : ∀1 < i , j , k ≤ d , C i i = 0, C i j = C j i , C i j ≤ C i k + C k j .
(3.9)

Turning the distance into a Linear Program (LP) problem

To ease the reading of the distance optimization problem, we rewrite the Sinkhorn divergence incorporating the cost matrix constraints (3.9) defining the Sinkhorn divergence as a distance, as a linear program (LP) problem that is,

WD κ C (a (s) , a (t ) ) =            min c,t (i ) c t (i ) + κt (i ) (log t (i ) -1) such that At (i ) = b (i ) , C i i = 0, C i j = C j i , C i j ≤ C i k + C k j , ∀1 < i , j , k ≤ p ,

Leveraging a Quadratically Constrained Quadratic Program (QCQP) metric learning problem

Since the images are labeled we could use these labels to solve a metric learning problem from a pseudo-truth distance w(•, •) : R f × R f → R between images based on their labels. According to such pseudo-truth distance, we then look for a cost matrix C such that its conditioned WD C closer images from the same class while pushing away images from different classes. For example, in [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF], Cuturi explored the Wasserstein Distance from a similar metric learning angle but proposed to hand-craft this pseudo-truth distance w (i ) = w(a (s) , a (t ) ) between two images a (s) and a (t ) such that w (i ) = 1/Nk if both images belong to the same label or w (i ) = -1/Nk if they belong to different label, with k being a chosen number of neighbors to consider. Given such pseudo-truth distance w (i ) between pairs of images, we can look for the cost matrix encoding the Wasserstein Distance to match these distances, Given the matrix A, for one couple of points b (i ) = [a (s) T , a (t ) T ] T and their pseudo-truth distance w (i ) , our ultimate goal is to find C and the T (i ) solution of the following LP problem

J (a (s) , a (t ) ) =                min c,t (i ) c t (i ) = Tr(C T (i ) ) such that At (i ) = b (i ) c t (i ) = w (i ) C i i = κ, C i j = C j i , C i j ≤ C i k + C k j , ∀1 < i , j , k ≤ p . (3.11)
The red term highlights our real implicit goal that is, we are looking for a WD C that matches the pseudo-truth distance for these two images. Now we leverage J (a (s) , a (t ) ) to the whole dataset of images' features D = {a (i ) , y (i ) } N i =1 . We are interested in the following J D Quadratically Constrained Quadratic Program (QCQP) problem,

given the binary encoding matrix A ∈ R 2 f × f 2 matrix, B ∈ R 2 f ×N a matrix whose columns are the vectors b (i ) , w a vector in R N a concatenation of all the w (i ) and τ ∈ R f 2 ×N a matrix whose columns are all the optimal transport plans t (i ) .

J D =                min c,τ 1 τ c = N i =1 c t (i )
such that Aτ = B τ c = w

C i i = κ, C i j = C j i , C i j ≤ C i k + C k j , ∀1 < i , j , k ≤ p .
(3.12)

The red term highlights our real implicit goal that is, we are looking for a WD C that matches the pseudo-truth distance for all pairs of images.

J D defines a bi-level optimization in which every transportation plan t included in τ are intrinsically defined by the WD C and thus conditioned to the cost matrix c. However, the cost matrix objective c is to make the overall WD C distance closer to the pseudo-truth distances w(•, •) included in w therefore, c is only conditioned to the quality of the extracted features and the modeling of the pseudo-truth distance w from the labels.

Algorithm

In this section, we present the proposed relaxation solving the original problem (3.12), and the algorithm solving it.

The original objection J D (3.12) is hard to solve because of the non-convex 1 indicator function and the strict τ c = w constraint that hard to respect in practice due to the two variables optimized. Therefore we replace them with a more manageable loss function,

         min c,τ 1 m m i =1 ||c t (i ) -w (i ) || 2 2 such that Aτ = B C i i = κ, C i j = C j i , C i j ≤ C i k + C k j , ∀1 < i , j , k ≤ p .
(3.13) Equation (3.13) optimizes the transport plan t (i ) such that they are valid with respect to the WD C condition, and optimizes the cost matrix c to lower the distance of the very WD C to the pseudo-truth distance w (i ) by minimizing their 2 norm. Equation (3.13) is optimized in two steps, one for each variable. We first will rely on the Linear Solvers available in the library POT (Python for Optimal Transport) [START_REF] Flamary | Pot: Python optimal transport[END_REF] to resolve all the transport plans t (i ) forming τ. Then we fix these variables as parameters to optimize the problem over the cost matrix c by using a sequence of cost matrix solutions to converge towards the optimal one. Finally, a rescaling of c is performed to ensure the constraint of C and thus produces a well-defined WD C distance.

At every n optimization, the plans are found using POT's solvers, then frozen to improve the cost matrix c using mini-batch stochastic gradient descent with gradients from

H c (n) = min c (n) 1 m m i =1 ||WD κ c (n) (a (s) i , a (t ) i ) -w (i ) || 2 2 , (3.14)
With m being the mini-batch size. We use problem H c (n) to improve c by creating the sequence of solutions

c (n+1) = c (n) -γ∇ c H(c (n) ), (3.15)
with γ the learning rate. This sequence of solutions aims at converging toward the optimal solution C * parameterizing a meaningful Wasserstein distance WD C * for images by using their extracted features.

The full optimization scheme is sketched in Algorithm 9.

Algorithm 9 Meaningful Wasserstein distance between images optimization.

Require: Labeled dataset of image extracted features D = {a (i ) , y (i ) } N i =1 ; pseudo-truth distance w(•, •); step-size γ, Optimizer Optim; Batch size B; a number of max iteration M 1: a (i ) = softmax(a (i ) )

Rebalance every feature vector a (i ) to make it a probability distribution a (i ) ∼ D;a ( j ) ∼ D Get pair of feature (a (i ) , a ( j ) ) from D

Experiments

Before tackling more practical applications in our experiments, we first need to study the mechanisms and behavior of cost matrix optimization on a toy problem.

Toy problem

For this purpose, we artificially generate an actual ground-truth cost matrix C * to recover, and a sample of N pairs of points D = {(a (1) i , a (2) i )} N i =1 , where a (1) , a (2) ∈ R f . By relying on the ground-truth cost matrix C * , we know the corresponding true distance WD C * (a (1) i , a (2) i ) = w (i ) . These distances are crucial and allow us to recover C * by optimizing the problem (3.13). In this toy problem, we indeed bypass the construction of pseudo-distance through Metric Learning. Nevertheless, this provides us with an initial reference point and an overview of our proposed distance optimization framework.

In reality, we may not always access feature vectors a that are defining probability distribution. Therefore we ignored line 1 of algorithm 9 and we studied the Toy problem under balanced and unbalanced pair of points (a (1)

i , a (2) i )that is, Balanced Data: a (1) i , a (2) i ∈ R K with a (1) i T 1 = a (2) i T 1, ∀i ∈ {1, . . . , N}, Unbalanced Data: a (1) i , a (2) i ∈ R K with a (1) i T 1 = a (2) i T 1, ∀i ∈ {1, . . . , N}. (3.16)
To stop the optimization at the point of convergence, we have chosen to use the relative norm of the cost matrix, .17) In this notation, and given that we only consider symmetric matrices, by C n-1 we refer to the upper triangular part of the cost matrix at the n -1st update of the same parameters, and by C n we refer to the upper triangular part of the cost matrix at the nth update of the same parameters. This criterion is the least biased and most robust measure we have to detect the convergence of the parameter to be optimized, the cost matrix C (in reality, our parameters correspond only to the n × (n -1)/2 values of the upper triangular part of the cost matrix due to the symmetry condition of C).

If ||C n-1 -C n || 2 ||C n-1 || 2 ≤ α Then: STOP. ( 3

Relaxations

For computational reasons, we have simply omitted the inequality constraint on C, the third constraint defining a distance. Furthermore, in [START_REF] Bellet | Metric Learning, Synthesis Lectures on Artificial Intelligence and Machine Learning[END_REF], Bellet et al. informs us a strict respect of these constraints is still under debate, in the pursuit of constructing a distance metric closer to human perception. Indeed, according to cognitive science research [START_REF] Ashby | Toward a unified theory of similarity and recognition[END_REF], [START_REF] Rosch | Cognitive reference points[END_REF], [START_REF] Tversky | Features of similarity[END_REF], [START_REF] Tversky | Similarity, separability, and the triangle inequality[END_REF], the enforcement of symmetry and the triangle inequality appears to be too rigid for our human perception of similarity between objects. It seems that humans are more flexible in our judgment of similarity than a mathematically perfectly defined distance would suggest.

Is POT a relevant Wasserstein distance solver?

To verify the efficiency of the POT toolbox, we handcrafted a custom mini-batched version of the Sinkhorn-knopp solver to see, which of the two is the most efficient. According to Table 3.1 presenting the results of our optimization on this toy problem for the two solvers under different learning rates, we can completely discard our custom Sinkhorn solver and completely trust the POT toolbox. Even for a small dimension, our custom Sinkhorn solver requires way too much time to achieve an acceptable level of quality, to consider it a relevant solver.

Is the optimization fast enough with small dimensions?

If the optimization framework were to be successful, it would at least converge to a solution very fast(10 seconds max) under the simplest settings (such as a small size of features f). Table 3.1 gives us a first insight into the time required to find the optimal solution with a small f = 4 using the POT solver, but the performances can be greatly enhanced by more finely tuning the hyperparameters. Table 3.2 displays the performances of the cost matrix optimization when using the best (POT) solver of the WD C problem under different learning rates. Under a small dimension (f = 4) for both balanced and unbalanced data, the optimization converges pretty quickly. Indeed, for balanced data, when tuning γ = 0.01 the optimization converges to the exact solution in 8 seconds, and for unbalanced data, when tuning γ = 0.05 the best solution is approximated in 14 seconds.

Note that all these experiments were performed on a MacBook Pro with 2,3 GHz Intel Core i9, 8 cores however, transitioning to GPU computation would drastically enhance our performance as it usually is a great optimization tool when relying on SGD. These first results confirm the minimum acceptable threshold. Table 3.2: Optimization of the cost matrix with a small dimension (f = 4), using the POT Linear Programming solver for both balanced and unbalanced data, under different learning rates. The optimization error is estimated using the 2 norm between the found solution and the true cost matrix.

(a) Balanced data. Is there a trade-off between the quality of the found solution and the time required to find it?

LR

Thanks to Table 3.2 we know that the optimization can reach the exact solution very quickly in small dimensions with either balanced or unbalanced data. However, when the feature size increases, things are a bit different. With balanced data, the optimization process is smoother, and from Table 3.3a and Table 3.4a, we don't see a trade-off between the time required for optimization and the quality of the solution. However, we observe that the learning rate is an extremely sensitive parameter.

However, with unbalanced data, things are more interesting as the optimization becomes more complex. From Table 3.3b, we can observe a trade-off between the time required for optimization and the quality of the solution. Indeed, with medium-sized features, when selecting γ = 0.005 the best solution is reached in 95 seconds although, when setting γ = 5, a descent solution is reached in under 10 seconds. We draw the same conclusion for high-sized features using Table 3.4b. When setting γ = 0.005, the best solution requires 113 seconds to be found, while an acceptable solution is retrieved at γ = 5 in only 12 seconds. Table 3.4: Optimization of the cost matrix with a medium dimension (f = 64), using the POT Linear Programming solver for both balanced and unbalanced data, under different learning rates. The optimization error is estimated using the 2 norm between the found solution and the true cost matrix.

(a) Balanced data.

LR

Error We understand through Table 3.3 and Table 3.4 that a trade-off between the quality of the found solution and the time required to find it only with unbalanced data.

Is the optimization framework robust to both balanced and unbalanced data?

We see from Table 3.2, 3.3 and 3.4 that a decent solution can be reached either with balanced or unbalanced data. As the problem is more complex with unbalanced data, the quality of the found solution is lowered, and the time required to find it is increased, but overall the optimal solution can be decently approximated in a reasonable amount of time.

Overall the optimization framework is robust to both balanced and unbalanced data, which is a desirable property when considering more realistic applications in which we are not likely to have balanced image feature vectors.

Is the learning rate a sensitive parameter?

It is very clear from Table 3.2, 3.3 and 3.4 that the learning rate is a very critical parameter. Even for unbalanced data, as a trade-off is observed, it is the learning rate parameter that allows us to navigate within the trade-off between the computational time and the solution quality.

We draw the same conclusion with balanced data. If the learning rate γ is not correctly chosen, then the best solution cannot be retrieved. This conclusion is especially illustrated with the medium-sized features under balanced data presented in Table 3.3a. When setting γ = 0.05, the best solution is reached in 13 seconds, while with γ = 0.1, the optimization requires 43 seconds to reach a worse solution.

According to the many conducted optimizations, we observed that the proposed optimization framework is sensitive to the learning rate γ parameter. When it is applied to more realistic applications, a significant time must be allocated to find the most appropriate learning rate γ. Table 3.6: Optimization of the cost matrix for both a small (f = 4) and a medium (f = 16) dimension, using the POT Linear Programming solver for balanced data with different magnitudes of the original cost matrix, employing a small learning rate (γ = 0.05).

(a) Small dimension (f = 4). To have a somewhat realistic toy problem, the question of excluding generated points that are too close has been addressed. Would we achieve better optimization of the cost matrix if we only consider points that are sufficiently distant according to the optimal WD C * using the true cost matrix? Judging from Table 3.9 displaying experiments results in both small and large dimensions, the optimization does not seem to be impacted at all by the rejection of points that are too close according to the true distance to be retrieved. Therefore, we can consider this parameter as insignificant for our upcoming more realistic applications.

Magnitude

From which dimension can we no longer guarantee a fast and reliable optimization?

It is worth noting that for larger feature dimensions, as we saw, a more detailed study of the learning rate would be more appropriate. However, these initial figures provide us with a preliminary understanding of the limit of the proposed distance optimization.

Even though for a dimension of f = 128, the computational time to reach convergence remains quite reasonable, Table 3.10a and Table 3.10b indicates that starting from a dimension of f = 256, optimizing the cost matrix becomes more challenging. It is important to note that for these larger dimensions, a more detailed study of various optimization parameters (mini-batch size, magnitude, learning rate, or even a learning rate annealing strategy) would be appropriate to achieve the most efficient optimization. Nevertheless, these figures remain illustrative and provide us with practical insights into the optimization process.

After addressing all these questions, we gained insights into the specificity of cost matrix optimization. We understand that the optimization works relatively well for both balanced and unbalanced data, but it takes, on average, 1.5 times longer to converge with unbalanced data. We know that using the POT solver is necessary. We've determined that rejecting points that are too close has no influence. The size of the mini-batch B is a relatively critical parameter to ensure op-Table 3.8: Optimization of the cost matrix for a large dimension (f = 64), using the POT Linear Programming solver for balanced data with varying magnitudes of the original cost matrix, employing both a small (γ = 0.5) and a large (γ = 5) learning rate.

(a) γ = 0.5. timization convergence, along with the magnitude of the solution to be found. However, we can manage these two parameters by choosing an appropriate learning rate, which seems to be the most critical hyperparameter of this optimization framework.

Magnitude

The outcome of this toy problem is positive; it demonstrates that within a certain working framework and under specific conditions (controlled dimension of the features f), we can converge towards the solution C * in an acceptable time frame. These findings are promising and allow us to tackle a slightly more realistic problem, such as experimenting with image features from the MNIST dataset as a next step.

MNIST images application

Now we gained a preliminary understanding of the optimization problem from the toy problem we can tackle a slightly more concrete application using images from the MNIST dataset [START_REF] Deng | The mnist database of handwritten digit images for machine learning research[END_REF]. Being one of the easiest image datasets, we considered the MNIST as the first more practical application of our distance optimization framework, which would then allow us to go further into more complex and realistic image datasets.

To reduce the dimensionality of the images and utilize only the most relevant information, we choose to employ discriminative features (referred to as features) extracted from the penultimate layer of a classifier neural network trained on these images. By doing so, we construct a fairly accurate classifier (achieving a minimum of 99% classification accuracy) to ensure that we retain the most relevant features of the images (those responsible for achieving the desired classification accuracy).

Let f : R P → R C be a classifier that takes input images of P pixels and assigns them a probability vector over the C possible classes. Since f is a neural network, we can redefine it as f (x) = g (h(x)), where h : R i nput Di mensi on → R f is a non-linear feature extraction function, and g : R f → R C is the final linear layer of f responsible for the final classification.

This feature extraction process also enables us to increase or lower the feature size f. More precisely, we consider the features from the penultimate layer after its activation function, which is the ReLU (Rectified Linear Unit, r el u(x) = max(x, 0)). This activation function has a promising of each class to assess and confirm the possible modes of each label. Figure 3.4 displays the mean feature vector of each label. Overall, both figures confirm that each class, in the feature space R f , occupies an isolated local region of the feature space, involving only a few dimensions, and is situated far enough from one another to be accurately distinguished. These features render an almost linear problem and consequently allow us to efficiently optimize our cost matrix.

Before tackling our proposed optimization framework, we attempted to eliminate the step of constructing the pseudo-ground truth distance w : R f × R f → R. Instead, we aimed to treat our Wasserstein Distance WD C directly as the distance to optimize within a metric learning optimization framework.

Optimizing the Wasserstein distance with a metric learning objective

Rather than relying on a pseudo-truth distance w : R f ×R f → R that we don't possess for the MNIST features, we consider our optimization framework of WD C as a metric learning problem in itself.

Thus at each iteration, the cost matrix is optimized according to the triplet loss objective

[164], min C relu(WD C (a (i ) , p (i ) ) -WD C (a (i ) , n (i ) ) + ρ) ∀(a (i ) , p (i ) , n (i ) ) ∈ Triplets(D), (3.18)
with ρ an arbitrary margin, and we considered the feature triplets (a (i ) , p (i ) , n (i ) ), composed of a first feature ba (i ) belonging to class y (i ) , a second feature p (i ) also belonging to class y (i ) , and finally a third feature n (i ) , not belonging to class y (i ) . The proposed triplet loss replaces the previous cost matrix update (line 8 from algorithm 9). The goal behind minimizing the triplet loss is to reduce the distance between points belonging to the same class while simultaneously increasing the distance between points belonging to different classes. This objective is widely accepted and used within the Metric Learning community, given the multitude of possible triplets possible from a single dataset. The initial results are both interesting and promising.

Results

In this section we review the results found during our experiments with the MNIST dataset, and why they reveal an ill-defined optimization problem.

An efficient optimization

During our MNIST experiments, to ensure adherence to a maximum number of distance axioms, we enforce certain conditions on our cost matrix. Specifically, we constrain the matrix to be positive, and symmetric and contain small values on the diagonal (0.5 instead of 0, as 0 is too dominant and inhibits the optimization of other variables). This approach helps achieve symmetry, identity, and positivity for our Wasserstein distance.

Figure 3.5 shows the result of the cost-matrix triplet loss optimization. In addition to achieving outstanding performance, we empirically observe from Figure 3.5b small WD C distance within classes and large WD C distance between different classes, we obtain an interesting and explainable cost matrix.

From Figure 3.5a, we see that the cost matrix contains values that can easily be interpreted in conjunction with the feature distribution. If a point C i j in the cost matrix is small, the cost between feature i and feature j is small, indicating a high correlation between these two features. This observation perfectly aligns with our findings from Figure 3

Comparison with the baseline

To some extent, the classifier's probabilities define a somewhat distance between images. Indeed, if two images are considered the same label then according to the classifier, they are somehow similar. Therefore, the optimized WD C must at least recover the classifier's performance to show any kind of improvement. Unfortunately, from Table 3.11, we see that the WD C does not catch up with the classifier's performances. It means that even though the cost matrix optimization is efficient and allows a retrieval of a decent solution, it does not reach the baseline performances and therefore does not reflect in any kind of improvement.

Intuition on the failure

The bad performances displayed in Table 3.11 might indicate that in reality, we are missing out on some important information contained in the features when optimizing the cost-matrix C.

From Figure 3.4, displaying the mean features, we observe that even though some dimensions of the features are activated (non-zero values), their values are so small that they are insignificant. This raises questions about the relevance of their activation, which could lead to some bias introduced with an overweight of them.

But more importantly, Figure 3.4 tells us that the features seem to already be linearly separable. And we confirm this hypothesis when increasing the feature size f with Figure 3 Indeed, the average features of each class appear to have very little overlap with others, suggesting a potential existing linear separation.

To explore and confirm or not this intuition, we fixed the cost matrix C with the simplest matrix that doesn't encode any correlations,

C 0 =   0.1 1 . . . 1 1 0.1 . . . 1 1 1 . . . 0.1   (3.19)
This matrix is indeed symmetric, positive, and has an almost zero diagonal, which enforces symmetry, identity, and triangularity of its parametrized Wasserstein Distance.

When using the Wasserstein distance parameterized with the simplest cost matrix, WD C 0 , we observed that this matrix yielded the best results. Thus, by initializing the cost matrix in this manner, the optimization didn't manage to improve the test performance even by updating it through triplet loss optimization.

To verify this dual observation of an existing linear separation of features, we performed the same 1-nearest neighbor (1-NN) search using the 2 norm as the distance from the mean feature of the Train dataset for each class, to the test dataset features. We observed that 99.08% of the test features are closest to the mean training feature of the same class.

MNIST images conclusion

This conclusion indicates that by extracting features from MNIST images using h(x), a function trained jointly with f (x) = g (h(x)), where g is a linear function, for a classification task achieving 99% accuracy on the Test dataset as Figure 3.7 highlights, an 2 norm is sufficient to properly separate the features from each other. When building deep neural network classifiers, the first layers' job is to discriminate the data points, while the latter layers are designed to make the prediction. If its training is correctly managed thus, by definition of the linear final layer, the inputs are already well separated before the final layer. Even though we thought we correctly formalized the problem and checked the relevance of all our hypotheses, it revealed that one was not completely accurate leading to a failed intuition.

It leads us to consider other types of features from images and, to apply our framework on more concrete and complex datasets, for which a simple training of a classifier does not reach 99% accuracy, meaning that the features before prediction are not completely linearly separated. In such cases, our optimization framework might be revealed as a relevant tool.

Conclusion & perspectives

Initially, we aimed to model a meaningful adversarial attack on images, that is, an attack taking into consideration the semantic nature of images. For this purpose, we used extracted features instead of the image pixels. To efficiently distinguish the images in the feature space, we proposed learning a Wasserstein distance.

During this process, we sanity-checked the overall proposed optimization framework on a toy problem and found it very efficient and successful.

However, when applied to a slightly more concrete scenario, we discovered that features derived from before the last layer of a neural network classifier are almost linearly separable, and therefore, an 2 norm is sufficient to accurately distinguish the features according to their label.

Therefore, we can proceed with the development of a meaningful adversarial attack on images, using an 2 norm as the distance metric between the features.

During our work on the development of a meaningful adversarial attack, we completely ignored the feature decoder that would map the adversarial features back to the image space as displayed in Figure 3.2. Instead, a relevant choice would suggest considering an autoencoder architecture type to extract the most discriminative features from the images and map them back to the image space.

It would also be worth exploring our optimization framework on more complex datasets, one for which no deep classifier training could linearly separate the images according to their label. In such cases, the modelization of a meaningful image distance with our proposed optimization would be relevant and seem very promising.

The idea we explored is at the core of the current machine learning Large Language Models (LLM) revolution. Indeed, the idea to embed the object in a space where multiple tasks can be performed seems to be a relevant choice and shows outstanding recent results [157] [42]. This current trend confirms the relevance of our idea, and even though we did not reach perfect results, our findings confirm that working to create more meaningful adversarial attacks is the right path for computing more realistic and relevant adversarial perturbations.

Chapter 4

Conclusion & perspectives

Throughout our works, we delved deep into assessing the robustness of image classifiers to certify their suitability for practical use through additive adversarial attacks. We highlighted that properties like transferability and universality are regrettably under-emphasized in current research, although they are fundamental to crafting meaningful adversarial examples leading to a meaningful robustness evaluation. We also took a step back and underscored the significance of the performance evaluation metric in achieving an unbiased and fair assessment of the adversarial example generation.

In the first work, we proposed to model the adversarial noise space using a dictionary such that an attack can be derived, maximizing the desirable properties of adversarial examples, transferability, and universality. Indeed, the derived LIMANS dictionary-based adversarial attack shows to be more robust and more transferable than current state-of-the-art methods. Moreover, our approach allows for visual inspection of the roots of adversarial perturbations, making it easier to communicate with non-expert audiences and gain a better understanding of classifier vulnerabilities. Empirical evidence supports the idea that, according to our approach, the generated adversarial perturbations are more relevant as they exhibit more structure related to objects than pseudo-random adversarial perturbations.

One of the first perspectives on this work is to evaluate the modeling of the adversarial noise space with fewer training examples to see if the LIMANS attack improves in generalization if more training examples are considered. LIMANS' primary goal is to allow for a more relevant robustness evaluation of classifiers using adversarial examples. Therefore, constructing a LIMANS black-box attack would step further the realism of the attacker's behavior, estimating the classifier's robustness in an even more realistic way. The construction of such LIMANS black-box attack is one of the most promising perspectives. Also, as advocated for instance by Carlini et al. [START_REF] Carlini | On evaluating adversarial robustness[END_REF], research on adversarial learning should be engaged in developing tools for resistance against these potential attacks. After working and proposing the LIMANS attack, we must design the LIMANS adversarial defense to contribute under the two scopes of attacker and defender, making the work an unbiased contribution. Another interesting perspective is to impose more constraints on the LIMANS atoms. Indeed, the adversarial patch attacks are interesting and appealing to estimate a more realistic classifier's robustness. We believe that, under carefully designed constraints, we could make LIMANS parameters, a dictionary of adversarial patches. It could lead to a real improvement and innovation brought to the research community, going further into crafting more realistic and efficient adversarial perturbations.

During our experiments, we mainly focused on images however, LIMANS could be applied to any data modalities as long as we can back-propagate gradients and have a similarity metric between points allowing us to respect an adversarial budget constraint. The LIMANS attack could be envisioned on any other data modalities, as long as the two previous conditions are met. Overall, this work aims to be a humble first step toward modeling and understanding the origin and nature of adversarial perturbations. This work also steps away from state-of-the-art methods to question the relevance of the robustness assessment using adversarial examples.

In a second work, we explored the modeling of a meaningful similarity metric between images. While creating adversarial perturbations, we often deal with a biased and irrelevant image distance, which inevitably hampers the creation of relevant adversarial examples, leading to an overly considered robustness estimation.

Despite numerous challenges, we demonstrated that our approach to modeling a relevant similarity metric between images is feasible and promising even though some issues need to be addressed. During our work on the development of a meaningful adversarial attack, we ignored the feature decoder that would map the adversarial features back to the image space.

Instead of considering the image representations of a neural network, an appropriate choice would be to consider an autoencoder architecture type to extract the most discriminative representations of the images while allowing to map them back to the image space.

It would also be worth exploring our optimization framework on more complex datasets, one for which no deep classifier training could linearly separate the images according to their label. In such cases, the proposed modeling of a meaningful image distance would be relevant and promising.

This second work aligns with the first, in the way that it promotes a more meaningful robustness evaluation using adversarial examples. We believe that the considered similarity metric or distance is one of the most critical choices to perform a robustness evaluation of classifiers. Therefore, before performing any meaningful robustness evaluation using adversarial examples, the distance modeling should in itself be an optimization problem to resolve.

To sum up, our work contributes to estimating the image classifier robustness using adversarial examples in a more meaningful way. We have shown that to conduct a fair and realistic robustness evaluation from an application standpoint, adversarial examples must have a certain form and, most importantly, be relevant to the context.

A.2.1 We first write down the Lagrangian formulation of the original problem

WD(a i , a j ) = min

π∈R f × f f k=1 f t =1 π k,t C k,t + f k=1 f t =1
π k,t (log(π k,t ) -1) , such that π1 = a i and π T 1 = a j .

(A.3)

The Lagrangian formulation including the constraints with, α, β ∈ R f , α i ≥ 0, β j ≥ 0 ∀i , j , reads

L(π,α,β) = f k=1 f t =1 π k,t C k,t + f k=1 f t =1
π k,t (log(π k,t ) -1) -α T (π1a i ) -β T (π T 1a j ) .

L(π * , u, v) = f k=1 f t =1 π * k,t C k,t + f k=1 f t =1
π * k,t (log(π * k,t ) -1) -α T (π * 1a i ) -β T (π * T 1a j ) ,

L(π * , u, v) = f k=1 f t =1 π * k,t [C k,t + log(π * kt ) -α k -β t ] - f k=1 f t =1 π * kt + α T a i + βa j , L(π * , u, v) = - f k=1 f t =1
π * kt + α T a i + βa j , L(π * , u, v) =u T Mv + α T a i + βa j , L(π * , u, v) =u T Mv + α T log(u) + β T log(v) .

A.2.4

We finish by highlighting the solver to efficiently solve the problem

∇ u L(u, v) = 0 ∇ v L(u, v) = 0 ⇔ -Mv + α u = 0 -M T u + β v = 0 ⇔ u * = α Mv v * = β M T u .
The optimization strategy falls back to gradient descent optimization on u and v simultaneously, once convergence is attained for these two variable we compute the optimal plan using π * k,t = diag(u)M diag(v). We fall back on the same result previously found in the solver proposed in [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF].

A.3 From the Wasserstein Distance to the Frechet Inception Distance

Let X and Y two multivariate random variables on R N following distribution f X resp. f Y both belonging to a family of distributions which is closed with respect to changes of location and scale. µ X , µ Y and Σ X , Σ Y are the respective means and standard deviations of f X and f Y . If WD( f X , f Y ) denotes the Wasserstein distance between the f X and f Y , we have

WD( f X , f Y ) = ||µ X -µ Y || 2 2 + Tr (Σ X + Σ Y -2(Σ X Σ Y ) 1/2 ).
According to [START_REF] Dowson | The fréchet distance between multivariate normal distributions[END_REF] who is the original author of the proof, the proof of such result follows these three steps:

1. We show that WD( f X , f Y ) = min f W ∈Γ( f X , f Y )
E||X -Y|| 2 2 where Γ( f X , f Y ) denotes the collection of all measures on N × N with marginals f X and f Y on the first and second factors respectively. II by doing, given a matrix P ∈ R 2N , if ∀w ∈ R 2N we have w T Pw ≥ 0, which is true when w T Pw = w T Q T Qw = ||Qw|| 2 ≥ 0, then P is positive semi definite. We exhibit such property for Σ W :

∀w ∈ R 2N , w T Σ W w = X Y T Σ X Σ XY Σ T XY Σ Y X Y = X Y T Σ X X+Σ XY Y Σ T XY X+Σ Y Y X Y T Σ X X+Σ XY Y Σ T XY X+Σ Y Y = X T Σ X X + Y T Σ T XY X + X T Σ XY Y + Y T Σ Y Y, ∃B ∈ R 2N w T Σ W w = w T B T Bw = (Bw) T (Bw) = ||Bw|| 2 = 2N i =1 (B T i w) 2 = 2N i =1 (B T i X Y ) 2 2N i =1 (B T i X Y ) 2 = 2N i =1 (B T i w) 2 = 2N i =1 (B T i X Y ) 2 = 2N i =1 (a T i X + b T i Y) 2 .
(A.9)

We decompose the matrix B, the square root matrix of Σ W , into internal components a i ∈ R N , b i ∈ R N , ∀i = 1, . . . , 2N, then B = a 1 ,..,a 2N b 1 ,..,b 2N ∈ R 2N×2N . The two red term should match, which will give us a redifinition of Σ X , Σ Y and Σ XY . We develop these terms to see how to redefine the variables (colors are used to highlight equality

) 2N i =1 (a T i X + b T i Y) 2 = 2N i =1 (a T i X) 2 + (b T i Y) 2 + 2((a T i X)(b T i Y)).
Developing each term in the sum:

2N i =1 (a T i X) 2 = 2N m=1 N j =1 a m j x j N i =1 a mi x i = 2N m=1 N j =1 N i =1 a m j x j a mi x i = N j =1 N i =1 2N m=1
a m j a mi x i x j , But we know that X T Σ X i j X = (A.12)

N i =1 x i N j =1 Σ X i j x j = N i =1 N j =1 Σ X j x i x j thus Σ X =
The red terms don't interest us since they are either already defined or not related to the dual variables and therefore become constraints. Thus we have the new problem in terms of dual variables min The KKT conditions state that :

∇S ∇a i = 0 ⇔ ∇Tr (a i b T i + b i a T i ) ∇a i + ∇Tr (a i a T i )Ω ∇a i = 0 ⇔ 2b T i + 2a T i Ω T = 0 ⇔ b T i = a T i Ω T . ∇S ∇b i = 0 ⇔ ∇Tr (a i b T i + b i a T i ) ∇b i + ∇Tr (b i b T i )M ∇b i = 0 ⇔ 2a T i + 2b T i M T = 0 ⇔ a T i = b T i M T .
(A.14)

Optimally we have:

a * i = Mb i ,
Σ Y = 2N i =1 b * i b * T i = 2N i =1 a * i Ω(Ωa * i ) T = Ω 2N i =1 a * i a * T i Ω = ΩΣ X Ω = Σ Y , Σ XY = 2N i =1 a * i b * T i = 2N i =1 a * i (Ωa * i ) T = 2N i =1 a * i a * T i Ω = Σ X Ω = Σ XY , (A.16)
By combining both we end up with a definition of

Σ XY Σ XY = Σ 2 XY Σ XY Σ XY = Σ X ΩΣ X Ω = Σ X Σ Y , Σ 2 XY = Σ X Σ Y .
(A.17)

We defined Σ 2 XY but for our original problem A.8 we are actually more interested in Σ XY . Assuming Σ X is non-singular, meaning Σ -1

X is defined, we can establish a definition of Ω,

Ω = Σ -1/2 X RΣ -1/2 X . (A.18)
From the definition of Σ Y we have a definition of R,

Σ Y = ΩΣ X Ω ⇔ Σ Y = Σ -1/2 X RΣ -1/2 X Σ X Σ -1/2 X RΣ -1/2 X ⇔ Σ Y = Σ -1/2 X RRΣ -1/2 X ⇔ R 2 = Σ 1/2 X Σ Y Σ

Notations

This section provides a concise reference describing the notation used throughout this work. The i -th example (input) from a dataset y (i ) or y (i ) The label associated with x (i )

Numbers and Arrays

X

The m × n matrix with input example x (i ) in row X i ,: + norm(V t -1 )-norm(V t ) norm(V t -1 -V t )

Adversarial

, at the iteration t , with M = 100 during the training procedure, according to different optimizers (ADAM, ADAM-W and SGD) for multiple initial step-sizes and both 2 and ∞ norm constraints. . . . . . . . . . . 2.10 Evolution of ADiL's training loss and the relative norm of D and V that is evaluated as norm(D t -1 )-norm(D t ) norm(D t -1 -D t )

+ norm(V t -1 )-norm(V t ) norm(V t -1 -V t )
, at the iteration t , with M = 100 during the training procedure, using the Adam optimizer couple with various learning rates for both 2 2.12 Evolution of ADiL's inference loss with M = 100, using the Adam optimizer coupled with various inference learning rates for both 2 and ∞ norm constraints. . . . . . .

2.13

A high-level overview of the proposed LIMANS adversarial attack and its optimization. It is highlighted the adversarial model D is universal to every adversarial example, while a specific coding vector v is crafted for each adversarial example. . . . . .

  (a) The radio-operated automobile, American Wonder in 1925. Source: Wikipedia.com (b) Waymo's robo-taxi. Source: Image courtesy of Waymo.

Figure 1 :

 1 Figure 1: Autonomous cars in 1925 and 2025, from Chandler (Motor Car Company) to Chandler (Phoenix Suburb).

Figure 2 :

 2 Figure 2: DARPA challenge winning car from Stanford, which was the first to integrate machine learning models to drive. Source: herox.com

Figure 3 :

 3 Figure 3: Example of an adversarial image in the context of objects detection by an autonomous car.

  (a) L'automobile radiocommandée, American Wonder, en 1925. Source : Wikipedia.com (b) Le robo-taxi de Waymo. Source: Image fournie par Waymo.

Figure 4 :

 4 Figure 4: Les voitures autonomes en 1925 et cent ans plus tard, de Chandler (Motor Car Company) à Chandler (banlieue de Phoenix).

Figure 5 :

 5 Figure 5: La voiture gagnante du défi DARPA de l'Université Stanford, qui a été la première à intégrer des modèles d'apprentissage automatique pour la conduite. Source: herox.com

Figure 6 :

 6 Figure 6: Exemple d'une image adversaire dans le cas d'une detection d'objets par une voiture autonome.

Figure 1 . 2 :

 12 Figure 1.2: Schema of a 1 dimensional convolution operation. The input size and kernel size are set for clarity purpose.

Figure 1 . 3 :

 13 Figure 1.3: Schema of a neural network of n hidden layers with two residual connections. The input size, number of hidden layers, their dimension, and the location of the residual connections are set for clarity purposes.

Figure 1 . 4 :

 14 Figure 1.4: Comparison of the state-of-the-art neural network classifiers on the ImageNet dataset as of 2023. Source: paperswithcode.com

Figure 1 . 5 :

 15 Figure 1.5: Comparison of the Hinge loss and the 0-1 loss. The vertical axis represents the value of the Hinge loss (in red) and zero-one loss (in blue) for fixed ρ = 1, while the horizontal axis represents the value of the prediction y.

Figure 1 . 6 :

 16 Figure 1.6: Schema of the Wasserstein distance between points from a in red, and points from b in blue. The black arrows represent the association of points from a to points from b, these associations define the transport plan T.

Figure 1 . 7 :

 17 Figure 1.7: Diagram of the GAN optimization scheme.

Figure 1 . 8 :

 18 Figure 1.8: Example of fake person images generated by GAN models. Source: thispersondoesnotexist.com

Figure 1 . 10 :

 110 Figure 1.10: Machine learning model's number of parameters (size) according to their publication date and their task. Source: Machine Learning Model Sizes and the Parameter Gap [167].

Figure 1 . 12 :

 112 Figure 1.12: Adversarial example produced with an additive pseudo-random adversarial attack. Source: Matthias Hein presentation Adversarial Robustness: Evaluation and Approaches beyond Adversarial Training in "A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning" workshop at ICML 2021.

  andx ∈ X .

Figure 1 . 13 :

 113 Figure 1.13: Schema of an adversarial example x in the ball of radius centered on the original example x, with the green arrow pointing in the optimization direction of the adversarial example, perpendicular to the classifier f decision boundary.

Figure 1 . 14 :

 114 Figure 1.14: Example of optimized adversarial patch aiming to fool the detection of light stops. Source: Meta Adversarial Training against Universal Patches[108].

Figure 1 . 15 :

 115 Figure 1.15: An illustration of the patch application operator. Source: Adversarial Patch[20].

  (a) Alteration of a speed limit sign, making autonomous vehicles recognize an 85 mph speed limit, in place of a 35 mph. Source: Macafee.com (b) Alteration of a stop sign, making autonomous vehicles fail at recognizing it. Source: Robust Physical-World Attacks on Deep Learning Visual Classification [54].

Figure 1 . 16 :

 116 Figure 1.16: Examples of real-world adversarial examples produced by someone altering sign boards.

Figure 1 . 18 :

 118 Figure 1.18: Examples of varying outdoor images conditions acting as adversarial perturbations in the case of autonomous cars object recognition. Source: Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming [109].

Figure 1 . 20 :

 120 Figure 1.20: Examples of UAP produced universal adversarial perturbations on different neural network architectures under the ∞ norm constraint. Source: Universal Adversarial Perturbations [111].

Figure 1 . 21 :

 121 Figure 1.21: Examples of singular vector UAP produced universal adversarial perturbations constructed using various layers of various DNNs on different neural network architectures under the ∞ norm constraint. Source: Art of Singular Vectors and Universal Adversarial Perturbations [83].

  optimization defenses improve the robustness of the classifier by using regularization, certification bounds, or adversarial examples during the training of the classifier. Robust optimization of classifiers is also called adversarial training of classifiers. The adversarial training has been proposed by [63], as the incorporation of adversarial examples in the training of the classifier. The loss of the classifier does not change however, training examples now consist of valid original training examples along with FGSM computed adversarial examples of the training set.

  min

  Figure 1.22 highlights such decision boundary expansion, making the classifiers more robust to adversarial examples, in the way that the computed adversarial examples are further away from the original examples, while still keeping the same high classification performances.

Figure 1 . 22 :Figure 1 . 23 :

 122123 Figure 1.22: Schema of a more robust classifier's decision boundary. In a dashed line, the classifier's decision boundary is close the both cloud points, making the computed adversarial examples very close to the original examples. In a solid line, the shown decision boundary is more robust as the computed adversarial examples are at a higher distance from the original examples. Source: Liwei Wang's talk at "A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning" work ICML 2021.

Figure 1 . 25 :

 125 Figure 1.25: Schema of the two adversarial examples vision in the community, with the universal adversarial examples in the middle.

Figure 2 . 1 :

 21 Figure 2.1: Example of an autoencoder performing compression and the decompression of an MNIST image.See that the code outputted by the encoder is of lower dimension than the images but allows for full reconstruction of the image, therefore containing all the information within the image. Source: Autoencoders[START_REF] Bank | Autoencoders[END_REF].

Figure 2 . 2 :

 22 Figure 2.2: Comparison with state-of-the-art adversarial attacks of ADiL with different hyperparameters λ 1 and λ 2 on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of adversarial attacks, while the rMSE of the produced adversarial noises is displayed on the x-axis.

Figure 2 . 3 :

 23 Figure 2.3: ADiL's performances with different numbers of atoms M on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of adversarial attacks, while on the x-axis is displayed the rMSE of the produced adversarial noises.

Figure 2 . 4 :

 24 Figure 2.4: ADiL performances with a different number of training data points N for multiple numbers of atoms M = 5, 10, 15. The dictionary D is optimized on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of ADiL, while on the x-axis is displayed the number of training data points.

Figure 2 . 5 :

 25 Figure 2.5: Illustration of ADiL's atoms when setting M = 5 on ImageNet dataset for different classifiers, under the ∞ norm constraint.

Figure 2 . 6 :

 26 Figure 2.6: ADiL's crafted adversarial perturbation mean 2 norm with different norm constraint bound. The dictionary D is optimized on a VGG classifier with M = 50.

  Considering the ∞ norm adversarial budget constraint.

Figure 2 . 7 :

 27 Figure 2.7: Black-box ADiL inference performances according to the number of trails for the sampling of v for various p norm adversarial budget constraints of equation (2.9).

Figure 2 .

 2 10b to be 0.001. Indeed, with the two values, the training loss is decreasing by far from the starting point and reaches a plateau while displaying a real convergence regarding the parameter's relative norm. Now we crafted an optimization setup that allows us to exploit ADiL's potential to the fullest for both the ∞ and 2 norm constraints. The optimization of Algorithm 3 under this setting produces dictionaries D shown in Figure2.11.

Figure 2 . 9 :

 29 Figure 2.9: Evolution of ADiL's training loss and the relative norm of D and V that is evaluated as norm(D t -1 )-norm(D t ) norm(D t -1 -D t )

Figure 2 . 10 :

 210 Figure 2.10: Evolution of ADiL's training loss and the relative norm of D and V that is evaluated as norm(D t -1 )-norm(D t )

  Considering the ∞ norm adversarial budget constraint. Considering the 2 norm adversarial budget constraint.

Figure 2 . 11 :

 211 Figure 2.11: Example of D solution with M = 10 found by optimizing Algorithm 3 with the optimization setting proposed in Section Optimizer and Learning rate, for both the ∞ and 2 norm constraint.

Figure 2 .

 2 Figure 2.13: A high-level overview of the proposed LIMANS adversarial attack and its optimization.It is highlighted the adversarial model D is universal to every adversarial example, while a specific coding vector v is crafted for each adversarial example.

Algorithm 7

 7 Regularized-LIMANS Require: Classifier f ; Learning rate ρ; Training dataset T ; p budget δ p ; Optimizer Optim; Batch size B; Regularization parameter λ 1: D = N (0, 1 M × P ); V = N (0, 1 P × M ) 2: for k = 0 to MAXEPOCH do for (x (i ) , y (i ) ) ⊂ T do 5: // Compute loss i from Problem (2.14) 6:

Algorithm 8

 8 Simple-LIMANS Require: Classifier f ; Learning rate ρ; Training dataset T ; p budget δ p ; Optimizer Optim; Batch size B 1: D = N (0, 1 M × P ); V = N (0, 1 P × M ) 2: for k = 0 to MAXEPOCH do for (x (i ) , y (i ) ) ⊂ T do 5: // Compute loss i from Problem (2.15) 6:

  ∞ norm with standard classifier.

  ∞ norm with robust classifier.

Figure 2 . 14 :

 214 Figure 2.14: Training fooling rate of adversarial attacks under the 2 and ∞ norm constraint on CIFAR-10 when fixing several atoms M (x-axis), associated to the classifier (left) VGG11 and (right) robust ResNet-18, with the Simple-LIMANS solver.

Figure 2 . 15 :

 215 Figure 2.15: Test fooling rate of adversarial attacks under the 2 and ∞ norm constraint on CIFAR-10 when fixing several atoms M (x-axis), associated to the classifier (left) VGG11 and (right) robust ResNet-18, with the Simple-LIMANS solver.

5 (

 5 c) LIMANS-∞ on Standard classifier.

5 (

 5 d) LIMANS-∞ on Robust classifier.

Figure 2 . 16 :

 216 Figure 2.16: Visualization of the learned universal adversarial directions (atoms of the dictionary D) when M = 5, on CIFAR-10 and corresponding to the classifier (left) VGG11 and (right) robust ResNet-18. All atoms are rescaled for display.

Figure 2 . 17 :

 217 Figure 2.17: Visualization of the learned universal adversarial directions (atoms of the dictionary D) when M = 10, on MNIST according to both the 2 and ∞ norm targetting a LeNet classifier achieving more than 98.8% of test accuracy. All atoms have been rescaled for display.LIMANS NoisePredicted: 0 Predicted: 7

Figure 2 . 18 :

 218 Figure 2.18: Examples of 2 adversarial perturbations produced by LIMANS and PGD on the MNIST dataset for a LeNet classifier achieving more than 98.8% of test accuracy.

  M = 1000 M = 2000 M = 4000 Robust Accuracy vs Time Consumption on the Test Set on VGG with 2 norm Autoattack PGD CW LIMANS (a) Considering the 2 norm.

M

  = 500 M = 1000 M = 2000 M = 4000 Robust Accuracy vs Time Consumption on the Test Set on VGG with norm Autoattack PGD FGSM FFGSM MIFGSM LIMANS (b) Considering the ∞ norm.

Figure 2 . 19 :

 219 Figure 2.19: Pareto front of the Robust Accuracy achieved by a standard VGG11 classifier and the time consumption taken by the state-of-the-art specific adversarial attacks and the LIMANS with a different number of atoms M attacks under the 2 and the ∞ norm constraint on the CIFAR-10dataset. On the y-axis is displayed the Robust Accuracy of the classifier, the lower Robust Accuracy is, the more efficient the adversarial attack is. On the x-axis is the time taken (in seconds) to compute adversarial examples over all the test sets of CIFAR-10, the lower the time is, the more efficient the adversarial attack is.

Figure 2 . 20 :

 220 Figure 2.20: Performance of LIMANS (left) ∞ -attacks and (right) 2 -attacks on CIFAR-10 when attacking VGG, under different settings of hyper-parameter in Regularized LIMANS λ, and different number of atoms M.

TrainingFigure 2 . 21 :

 221 Figure 2.21: Performances of the Simple-LIMANS algorithm under the 2 norm constraint, with and without the offset b.

Figure 2 .

 2 Figure 2.21 compares the training performances of a Simple-LIMANS produced solution and the same solution without the offset b. We can see that LIMANS' performances grow accordingly without any unseen artifacts captured by b. Besides, through this figure, we understand its relative

  Batch size comparison with the Simple-LIMANS solver.

Figure 2 . 22 :

 222 Figure 2.22: Evolution of LIMANS 10 training loss according to (left) different optimizers and (right) different batch sizes B using the Simple-LIMANS solver on a standard VGG classifier under the ∞ norm.

  Multi-LIMANS applied on a Standard classifier.

  Multi-LIMANS applied on a Robust classifier.

Figure 2 . 23 :

 223 Figure 2.23: Test performances of the Simple-LIMANS algorithm under the ∞ norm constraint, for both the standard classifier (VGG) and the robust classifier (Robust ResNet) with the Multi-LIMANS training on the VGG, DenseNet and robust ResNet classifiers.

Figure 2 .

 2 Figure2.23 displays test fooling rate performances between a Multi-LIMANS trained D and, a Simple-LIMANS trained D. We empirically saw that learning D on multiple classifiers rather than only one did not improve the universal adversarial direction but introduced more bias, preventing it from reaching the inference-specific adversarial noise which was originally at hand.Another envisioned improvement that Multi-LIMANS could bring in, was to improve the transferability. We knew that by learning on multiple classifiers, the dictionary was not better at inference for those classifiers, but did it generalize more and therefore performed better on other classifiers it has not yet been applied to, Unfortunately, the bias introduced by learning D on multiple classifiers does not balance with the multi-classifiers' learned property. As Figure2.24 shows, inference performances are never

  Transfert performances on DenseNet from different learning classifier set Learning from Robust ResNet and Standard VGG classifier Learning from Standard VGG classifier Learning from Robust ResNet classifier

Figure 2 . 24 :

 224 Figure 2.24: Transfer test performances of the Simple-LIMANS algorithm under the ∞ norm constraint, for both the standard classifier (VGG) and the robust classifier (Robust ResNet) with the Multi-LIMANS training on the VGG, DenseNet, and robust ResNet classifiers, but not trained on the model used during inference.

Figure 3 . 1 :

 31 Figure 3.1: Examples of the 2 norm considered as a distance between images. All of the three images come from the CIFAR-10 dataset.

  of all examples only of label k and C the parameters of the distance d .

Figure 3 . 2 :

 32 Figure 3.2: High-level overview of our meaningful adversarial attack proposition. The car image is from the CIFAR-10 dataset.

(3. 10 )

 10 with t = flatten(T) the vectorized version of the Optimal Transport Plan T between a (s) and a (t ) , c = flatten(C) the vectorized version of the cost matrix C, b ∈ R 2 f the concatenation of both feature vectors b = (a (s) T , a (t ) T ) T and A ∈ R 2 f × f 2 a fixed binary matrix encoding the right pairing between the transport plan and the feature's entry in b to respect the original constraints. Now that the canvas is formally set and we clarified that the optimization of the meaningful distance between images boils down to optimizing C ∈ C we turn its optimization problem into a metric learning problem.

2: C = 1 f × f Initialize the cost matrix C 3 :

 3 for k = 0 to M do

7 :d

 7 = WD C (a(i ) , a ( j ) ) Solve (3.13) w.r.t t using POT 8: loss = loss + ||dw(a (i ) , a ( j ) ensure C ∈ C Making the updated WD C a distance 12: end for 13: return C parameter of WD C the optimized meaningful distance between image

Figure 3 . 4 :

 34 Figure 3.4: Distribution of the features when setting f = 16 according to the 10 possible labels.

  1.2 1.3 1.2 1.4 1.2 1.7 0.5 0.6 1.6 1.2 2.0 0.9 2.1 1.2 1.2 1.2 2.3 2.0 1.6 1.9 1.6 1.8 0.6 0.5 2.2 2.3 2.1 2.2 0.9 1.3 1.5 1.4 1.9 1.1 1.0 2.4 1.5 0.9 1.6 2.2 0.5 1.7 1.5 0.9 1.3 2.1 1.41.1 1.5 1.5 2.1 2.0 1.6 0.9 1.2 2.3 1.7 0.5 1.0 1.9 1.2 1.5 1.5 1.8 1.6 1.2 1.3 1.5 1.6 2.0 2.0 2.1 1.5 1.0 0.5 1.4 1.7 0.9 1.0 0.9 1.6 1.5 2.3 1.2 1.4 0.9 0.9 2.2 0.9 1.9 1.4 0.5 1.7 1.7 1.1 1.7 1.4 2.5 1.7 2.2 1.7 1.9 2.1 0.9 1.3 1.2 1.7 1.7 0.5 1.4 1.5 2.0 1.8 1.8 1.0 1.6 1.4 1.1 1.2 1.3 2.1 1.5 0.9 1.7 1.4 0.5 Cost Matrix after optimization. label: 1 label: 2 label: 3 label: 4 label: 5 label: 6 label: 7 label: 8 label: 9 label: Bootstrap of Test point distances label per label.

Figure 3 . 5 :

 35 Figure 3.5: Triplet loss optimization results.

  .4 and Figure 3.4.

Figure 3 . 6 :

 36 Figure 3.6: Distribution of the features when setting f = 32 according to the 10 possible labels.

Figure 3 . 7 :

 37 Figure 3.7: Dichotomy of a deep network's architecture. The first layers' job is to discriminate the data points, while the latter are designed to make the prediction. Source: Leveraging sparse linear layers for debuggable deep networks [174].

IA. 2 . 2 A. 2 . 3

 2223 We find the optimal solution of the plan from this Lagrangian formulation δL(π, α, β)δπ kt = 0 ⇔ C kt + (log(π k,t )) -α k -β t = 0 , δL(π, α, β) δπ kt = 0 ⇔ log(π k,t ) = -C kt + α k + β t , δL(π, α, β) δπ kt = 0 ⇔ π * k,t = exp( α k ) exp( -C kt ) exp( β t ) .We use a rewriting of the dual variable to rewrite the Lagrangian problemLet u = exp( α ), v = exp( β ) and M = exp( -C ), then we have δL(π * , α, β) δπ kt = 0 ⇔ π * k,t = diag(u)M diag(v) .

ΣΣ 10 )

 10 b mi y i y j , But we know that Y T Σ Y Y = Y i j y i y j thus Σ Y = b m j x i y j , But we know that Y T Σ T XY X + X T Σ XY Y = XY i j x i y j thus Σ XY =We now rewrite Σ X , Σ Y and Σ XY (the red elements) in terms of a i ∈ R N , b i ∈ R N ∀i = 1, . . . , 2N, the components of the Σ W 's square root matrix and rewrite Eq. A.8 as wellWD( f X , f Y ) = min Σ XY Tr (Σ X + Σ Y -Σ XY -Σ T XY ), WD( f X , f Y ) = minconstraints with Lagrangian variables Ω ∈ R N×N and M ∈ R N×N into the Lagrangian formulation of the problem L(a,b,Ω,M) = max

  T m + b m a T m ) + Tr (( 2N m=1 a m a T m ))Ω + Tr (( 2N m=1 b m b T m ))M. (A.13)

  ∀i = 1, . . . , 2N, b * i = Ωa i , ∀i = 1, . . . , 2N. (A.15) From these two definitions and knowing that Σ X = 2N m=1 a * m a * T m and Σ Y = 2N m=1 b * m b * T m we derive a definition of Σ XY

1 {0, 1 ,

 11 Identity matrix with n rows and n columns e(i ) Standard basis vector [0, . . . , 0, 1, 0, . . . , 0] with 1 at position i diag(a)A square, diagonal matrix with diagonal entries given by a . . . , n} the set of all integers between 0 and n[a, b]The real interval including a and b(a, b]The real interval excluding a but including b A\B Set subtraction, i.e., the set containing the elements of A that are not in B Indexing a iElement i of vector a with indexing starting at 1a -iAll elements of vector a except for elements iA i , j Element i , j of matrix A A i ,: Row i of matrix A A :,i Column i of matrix A Derivative with respect to x of f : R → R ∂ f ∂x Partial derivative with respect to x of f : R → R ∇ x f Gradient of f : R n → R m with respect to x f (x)d x Definite integral over the entire domain of x Probability and Information Theory a⊥b The random variables a and b are independent P(a) A probability distribution over a discrete variable p(a) A density probability function over a continuous variable, a ∼ P Random variable a has distribution P E x∼P [ f (x)] or E f (x) Expectation of f (x) with respect to P(x) Var( f (x)) Variance of f (x) Cov( f (x), g (x)) Covariance of f (x) and g (x) H(x) Shannon entropy of the random variable x D KL (P ∥ Q) Kullback-Leibler divergence of P and Q N (x; µ, Σ) Gaussian distribution over x with mean µ and covariance Σ VIII Functions f : A → B The function f with domain A and range B f • g Composition of the function f and g f (x, θ) A function of x parametrized by θ. (Sometimes we write f (x) instead and neglect θ to lighten notation) f i (x) Element i of f (x) when f ranges in the vector space log x Natural logarithm of x e x Natural exponential of x ∥ x ∥ p L p norm of x ∥ x ∥ L 2 norm of x x + Positive part of x, i.e., max(0, x)

∆

  An adversarial attack, ∆(x) = x = x + ω x An adversarial example paired to the original example x ω x An adversarial noise paired to the original example x Ω The space in which the adversarial noise ω x lives IX 1.12 Adversarial example produced with an additive pseudo-random adversarial attack. Source: Matthias Hein presentation Adversarial Robustness: Evaluation and Approaches beyond Adversarial Training in "A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning" workshop at ICML 2021. . . . . . . . . . . . . . . . . . 14 1.13 Schema of an adversarial example x in the ball of radius centered on the original example x, with the green arrow pointing in the optimization direction of the adversarial example, perpendicular to the classifier f decision boundary. . . . . . . . . . . 17

1

 1

  and ∞ norm constraints. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.11 Example of D solution with M = 10 found by optimizing Algorithm 3 with the optimization setting proposed in Section Optimizer and Learning rate, for both the ∞ and 2 norm constraint. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. 14

 14 Training fooling rate of adversarial attacks under the 2 and ∞ norm constraint on CIFAR-10 when fixing several atoms M (x-axis), associated to the classifier (left) VGG11 and (right) robust ResNet-18, with the Simple-LIMANS solver. . . . . . . . . . 2.15 Test fooling rate of adversarial attacks under the 2 and ∞ norm constraint on CIFAR-10 when fixing several atoms M (x-axis), associated to the classifier (left) VGG11 and (right) robust ResNet-18, with the Simple-LIMANS solver. . . . . . . . . . . . . . . . . 2.16 Visualization of the learned universal adversarial directions (atoms of the dictionary D) when M = 5, on CIFAR-10 and corresponding to the classifier (left) VGG11 and (right) robust ResNet-18. All atoms are rescaled for display. . . . . . . . . . . . . . . . 2.17 Visualization of the learned universal adversarial directions (atoms of the dictionary D) when M = 10, on MNIST according to both the 2 and ∞ norm targetting a LeNet classifier achieving more than 98.8% of test accuracy. All atoms have been rescaled for display. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.18 Examples of 2 adversarial perturbations produced by LIMANS and PGD on the MNIST dataset for a LeNet classifier achieving more than 98.8% of test accuracy. . . . . . . .

Table 1 .

 1

1: Summary of the diverse presented adversarial attacks: The ' p -norm' refers to the norm used in Ω = {ω ∈ R P , ||ω x || p ≤ }. Sources of this table include

  .3 reports ADiL's test fooling rates for various numbers of M. From Figure2.3, we observe that ADiL indeed reaches better test fooling rate performances with a higher M but at the cost of becoming less universal.

	0.2 0.3 0.4 0.5 0.6 0.7 0.8 Fooling rate	2 atoms 3 atoms 5 atoms 10 atoms	20 atoms 30 atoms 50 atoms 100 atoms
	0.1
	0.0	0.010	0.012 rMSE 0.014	0.016

Table 2 .

 2 1: Performance of ∞ attacks on ImageNet. Comparisons are drawn in terms of fooling rates (fr), mean squared error (mse), and the average time for evaluating the attack (in ms). Results are divided between specific (top cell), ADiL with M = 100 atoms (middle cell) and universal attacks (bottom cell).

			VGG			DenseNet			GoogleNet
	Attacks	fr	mse	time	fr	mse	time	fr	mse	time
	APGD	99.96 95.41	689 99.99 71.67	526	100	69.57	283
	AutoAttack	100	95.45	688	100	71.63	536	100	69.57	291
	FGSM	97.84 144.43	21	94.16 144.35	55	91.96 144.38	51
	FFGSM	98.76 114.83	21	96.43 114.67	55	94.00 114.71	51
	MIFGSM	99.93 94.05	594 99.99 74.46	460	100	73.08	233
	PGD	99.96 91.74	593 99.99 73.10	458	100	71.69	234
	ADiL-ce	86.08 90.76	140 88.97 88.12	187 84.12 88.82	101
	ADiL-logit 84.52 89.02	140 87.86 86.41	183 86.76 86.05	101
	UAP-PGD 68.44 121.70	59	68.14 102.75	67	72.72 120.65	56
	UAP	60.37 84.62	71	57.52 79.28	67	42.15 74.32	48
			Inc.V3			ResNet18			MobileNet
	Attacks	fr	mse	time	fr	mse	time	fr	mse	time
	APGD	99.99 66.47	910	100	77.23	180	100	69.31	364
	AutoAttack	100	66.49	913	100	77.23	180	100	69.31	365
	FGSM	83.40 140.50	65	97.89 144.41	45	96.16 144.39	51
	FFGSM	85.35 112.93	65	99.08 114.70	45	98.77 114.74	51
	MIFGSM	99.99 68.51	899	100	79.03	171	100	73.24	354
	PGD	99.99 68.53	899	100	77.38	170	100	71.78	352
	ADiL-ce	86.16 75.41	162 92.46 89.38	88	98.14 85.25	86
	ADiL-logit 87.75 78.16	139 89.79 87.78	89	96.69 83.43	86
	UAP-PGD 52.27 96.10	66	70.85 107.20	50	91.66 106.27	46
	UAP	18.33 55.44	61	65.37 89.84	53	85.44 91.02	50

  and explained in Section 1.2.5. For the ADiL defense, we opted for an ADiL adversarial training defense. During the ADiL adversarial training defense, the classifier currently in training is optimized using original examples as well as ADiL-crafted adversarial examples. Feeding adversarial examples to the optimization of a classifier tends to push its decision boundaries away from the original data points, making the classifier's decision more robust to slight perturbations. However, to access ADiL adversarial attack, ADiL's parameters D need to be optimized as well. Therefore we chose, as a good trade-off to update ADiL's parameters every M epoch of the classifiers' optimization, making the ADiL adversarial attack always up-to-date with the classifier's decision boundaries while building it. ADiL defense is detailed in Algorithm 6.

	Algorithm 6 ADiL Defense
	Require: A classifier f parameterized by θ, a training set D = {x (i ) , y (i ) } a classification loss L, a
	step-size γ, a maximum number of iteration K, number of epoch when optimizing ADiL's pa-
	rameters M
	for k = 1 to K do
	Updating D to the current classifier f every M epoch
	if mod(k,M) = 0 then

ADiL adversarial defense. Defenses are mechanisms that make classifiers more robust to adversarial attacks. De-fenses are reviewed Optimize D from current f and D by solving equation (2.5) end if for (x (i ) , y (i ) ) ∈ D do Optimize on original data

Table 2 . 2 :

 22 Study of the proposed defense mechanism for LeNet trained on CIFAR-10. By hinging on the proposed defense with M defense = 10 atoms, we report the fooling rate of ADiL attacks for various M attacker atoms.

	M attacker	2 atoms 5 atoms 10 atoms 15 atoms 20 atoms
	Without Defense 25.78%	56.25%	60.15%	46.09%	57.81%
	With Defense	15.62% 30.46%	53.90%	44.53%	56.25%

Figure 2.8: Fooling Rate of ADiL with different number of atoms on a LeNet classifier. The dots represent the ADiL attack on a standard LeNet classifier, while the diamonds display fooling rates on a robust LeNet classifier trained with the defense. Fooling rate values are precisely reported in

Table 2.2.

Table 2 .

 2 3: Robustness performance of the LIMANS ∞ attack ( = 8/255) in terms of RAUD on the CIFAR-10 test data and against the attack detectors plugged in both standard classifier (S.C.) and robust classifier (R.C.). The shown performances are averaged over 5 random seeds.

The smaller the RAUD, the more robust the adversarial attack is.

  The best performances are marked in bold.

	Detectors d	d PGD		d Autoattack	d LIMANS 10
	Classifiers f	S.C.	R.C	S.C.	R.C	S.C.	R.C
	SA	91.1	85.1	91.1	85.1	91.1	85.1
	FGSM	91.1 ± 0.0 85.1 ± 0.0	91.1 ± 0	85.1 ± 0.0 89.3 ± 0.0 79.5 ± 0.0
	PGD	91.0 ± 0.0 84.9 ± 0.1 91.0 ± 0.0 85.0 ± 0.0 80.7 ± 0.5 73.3 ± 0.3
	Autoattack	91.0 ± 0.0 85.0 ± 0.0 91.1 ± 0.0 85.0 ± 0.0 78.2 ± 0.3 71.5 ± 0.3
	LIMANS 10	78.3 ± 2.4 79.6 ± 0.0 81.7 ± 2.0	8.0 ± 0.0	86.4 ± 1.6 79.5 ± 0.1
	LIMANS 500	26.3 ± 0.3 71.3 ± 0.1 32.1 ± 0.4 73.2 ± 0.2 36.5 ± 3.1 69.4 ± 0.2
	LIMANS 1000 24.7 ± 0.9 70.4 ± 0.2 31.6 ± 1.1 72.4 ± 0.1 36.9 ± 4.6 68.5 ± 0.2
	LIMANS 4000 23.7 ± 0.5 69.8 ± 0.0 30.8 ± 0.9 72.9 ± 0.3 35.6 ± 2.2 68.2 ± 0.1
	robustness of the LIMANS framework.

Table 2 .

 2 4: Robustness performance of the LIMANS 2 attack ( = 0.5) in terms of RAUD on the CIFAR-10 test data and against the attack detectors plugged in both standard classifier (S.C.) and robust classifier (R.C.). The shown performances are averaged over 5 random seeds.

The smaller the RAUD, the more robust the adversarial attack is.

  The best performance is marked in bold font.

	Detectors d	d PGD	d Autoattack	d LIMANS 10
	Classifiers f	S.C.	R.C	S.C.	R.C	S.C.	R.C
	SA	91.1	85.1	91.1	85.1	91.1	85.1
	PGD	64.0 ± 1.2	82.4 ± 0.0	65.3 ± 0.6 81.8 ± 0.1 42.9 ± 0.6 80.1 ± 0.0
	Autoattack	63.1 ± 1.1	82.19 ± 0.1	65.8 ± 0.6 81.1 ± 0.2 42.1 ± 0.3 79.4 ± 0.0
	LIMANS 10	83.5 ± 0.5	87.4 ± 0.1	82.9 ± 0.4 88.1 ± 0.0 86.9 ± 0.9 87.6 ± 0.1
	LIMANS 500	62.9 ± 0.2	84.0 ± 0.4	64.7 ± 1.0 83.8 ± 0.3 51.7 ± 1.3 81.7 ± 0.1
	LIMANS 1000 63.7 ± 0.6	82.7 ± 0.3	63.6 ± 0.9 82.6 ± 0.2 46.8 ± 0.7 80.1 ± 0.1
	LIMANS 4000 62.6 ± 1.2 82.16 ± 0.2 62.2 ± 0.6 82.6 ± 0.3 46.9 ± 0.2 80.0 ± 0.2

Table 2 .

 2 5: Confusion matrices of the detectors used to compute the RAUD of Table 2.3. All detectors have been trained on the same training dataset as the one used in the training of LIMANS and the displayed values of computed over the validation dataset, such that fair performances are considered. TN: True Negative, FP: False Positive, FN: False Negative, TP: True Positive.

	Detectors d	d FGSM	d PGD	d Autoattack	d LIMANS 10
	Confusion Matrix	TN FP FN TP	863 57 0 920	814 106 0 920	790 130 0 920	846 74 95 825

Table 2 . 9 :

 29 Fooling rate performances of the Regularized-LIMANS solver with M = 150 on CIFAR-10, when varying the λ hyper-parameter. The best results are marked in bold style.

			∞ norm			2 norm
	λ	VGG MobileNet R-R18 VGG MobileNet R-R18
	0.1 73.2	85.3	15.8	71.7	95.4	17.6
	1.0 91.0	97.3	25.3	45.4	49.6	12.8
	10 46.1	91.7	19.9	13.6	24.4	10.3

Table 2 .

 2 6: Transferability performance of the LIMANS ∞ attacks on CIFAR-10 ( = 8/255), in terms of fooling rates (FR). The best transferable results are marked in bold font.

	Source model \Target model MobileNet Inception ResNet50 DenseNet	VGG	R-r18 R-wr-34-10
		AutoAttack	100	87.1	37.2	32.8	22.4	1.7	1.5
		UAP	47.3	36.1	9.3	8.3	8.7	1.3	0.8
		UAPPGD	86.2	56.1	20.5	19	21.3	1.6	1.5
		TI-FGSM	87.2	25.2	25.9	29.1	16.1	2.1	2.0
		VMI-FGSM	100	87.3	53.1	49.8	38.9	2.2	2.5
	MobileNet	VNI-FGSM	100	88.1	54.8	53.1	40.7	2.4	2.9
		NAA	72.2	25.3	6.8	5.9	6.4	1.3	1.0
		RAP	86.7	60.3	38.5	35.7	25.8	3.0	1.6
		LIMANS	97.3	92.2	73.6	66.4	67.7	10.2	10.7
		AutoAttack	54.7	100	14.7	12.9	12.0	1.2	1.1
		UAP	39.2	32.9	9.3	9.7	9.4	1.5	1.1
		UAPPGD	73.9	75.5	26.3	23.8	27.3	2.2	1.5
		TI-FGSM	19.7	60.2	19.8	21	11.4	2.2	1.5
		VMI-FGSM	69.8	86.1	40.8	38.3	31.3	2.6	1.6
	Inception	VNI-FGSM	75.5	89.5	44.4	42.4	36.3	3.2	2.3
		NAA	38.7	70.5	8.4	8.1	9.2	1.1	1.5
		RAP	61.9	90.2	42.0	41.7	30.3	2.3	2.7
		LIMANS	98	95.1	79.6	73.9	75.8	10.7	10.8
		AutoAttack	63.3	52.6	100	54.6	25.1	1.2	2.4
		UAP	31.4	23.6	12.1	12.5	11.2	1.3	1.7
		UAPPGD	63.3	49.4	39.4	35.1	26.1	1.1	2.3
		TI-FGSM	18.4	17.1	74.0	38.5	20.4	2.2	3.0
		VMI-FGSM	74.9	75.3	96.0	78.1	53.5	2.1	3.2
	ResNet50	VNI-FGSM	78.3	76.9	95.9	80.3	57.2	2.7	2.1
		NAA	50.7	38.6	64.7	22.9	18.4	1.4	2.1
		RAP	49.0	45.7	75.1	52.5	35.4	1.6	2.8
		LIMANS	96.0	92.9	91.3	81.8	82.1	11.7	13.2
		AutoAttack	56.9	51.6	48.8	100	21.8	2.1	2.0
		UAP	27.6	20.6	10.6	12.8	11.4	1.6	1.4
		UAPPGD	61.1	49.9	29.3	47.4	27.3	2.7	2.1
		TI-FGSM	17.4	15.8	26.3	65.2	17	2.9	2.3
		VMI-FGSM	73.7	71.8	77.2	93.1	47.9	3.3	3.7
	DenseNet	VNI-FGSM	78.1	76.2	79.5	94.0	53.3	3.5	4.2
		NAA	37.2	31.1	23.7	74.9	12.5	1.2	1.5
		RAP	47.8	43.5	48.7	75.9	35.6	3.2	3.5
		LIMANS	96.7	93.5	88.4	85.5	82.7	12.3	13.4
		AutoAttack	62.5	58.0	43.0	44.0	100	2.7	2.7
		UAP	22.0	18.4	10.2	10.2	10.0	1.1	1.3
		UAPPGD	63.6	55.9	27.6	29.4	41.9	3.1	2.1
		TI-FGSM	19.7	16.7	25.6	27.6	74.4	3.7	2.2
		VMI-FGSM	66.2	64.2	57.5	56.9	96.5	3.0	2.6
	VGG	VNI-FGSM	69.3	68	62.6	61.4	96.5	3.0	2.6
		NAA	42.3	38.3	14.5	1.8	71.6	1.6	1.2
		RAP	46.5	44.5	39.5	40.9	73.8.	3.3	3.4
		LIMANS	97.4	95.1	87.5	81.5	91.0	11.5	12.6
		AutoAttack	17.5	15.7	17.2	15.6	17.5	44.3	23.4
		UAP	14.5	9.5	7.1	6.4	7.6	1.9	2.6
		UAPPGD	18.6	13.3	9.7	8.6	10.5	3.1	3.5
		TI-FGSM	8.4	5.5	8.2	7.8	8.6	26.2	13.1
		VMI-FGSM	24	22.9	24.2	21.9	24.8	38	22.7
	R-r18	VNI-FGSM	27.1	23.1	25.4	23.8	25.6	38.1	22.9
		NAA	16.2	11.5	11.2	10.4	10.4	18.7	7.2
		RAP	10.9	8.4	7.9	8.9	9.7	23.8	12.2
		LIMANS	81.3	73.2	71.7	68.3	61.7	25.3	21.6

Table 2 .

 2 7: Transferability performance of the LIMANS 2 -attacks on CIFAR-10 ( = 0.5), in terms of fooling rates (FR). The best transferable results are marked in bold font.

	Source model \Target model MobileNet Inception ResNet50 DenseNet	VGG	R-r18	R-wr-34-10
		AutoAttack	100	50.20	14.20	13.30	8.20	0.90	0.50
		UAP	7.50	5.20	3.00	2.50	2.40	0.30	0.40
		UAPPGD	37.90	15.20	2.00	1.10	0.90	0.30	0.20
		CW	97.50	11.00	4.20	3.20	2.40	0.30	0.00
	MobileNet	RAP	67.30	11.20	4.20	3.90	2.60	0.50	0.10
		LIMANS	95.40	91.50	61.70	59.30	51.50	4.60	5.00
		AutoAttack	32.80	100	6.60	7.90	5.50	0.50	0.50
		UAP	9.80	7.50	2.50	3.50	2.90	0.20	0.10
		UAPPGD	26.90	16.70	1.30	2.30	2.00	0.30	0.10
		CW	16.30	82.80	5.00	5.20	3.70	0.30	0.00
	Inception	RAP	13.60	43.50	3.50	3.60	2.70	0.40	0.30
		LIMANS	94.60	94.10	64.30	63.90	57.20	5.10	5.20
		AutoAttack	31.00	23.40	99.70	26.10	10.00	1.20	0.70
		UAP	5.10	3.80	2.40	1.90	2.80	0.50	0.30
		UAPPGD	4.10	3.20	2.20	2.30	2.20	0.40	0.20
		CW	13.50	9.80	82.40	13.10	6.10	0.50	0.40
	ResNet50	RAP	10.20	8.60	33.00	8.60	4.90	0.40	0.30
		LIMANS	92.60	87.50	78.10	71.70	61.70	7.90	7.50
		AutoAttack	32.60	25.20	27.30	99.50	10.20	0.50	0.50
		UAP	4.90	3.70	2.60	3.30	1.90	0.20	0.20
		UAPPGD	5.00	4.50	3.20	3.70	2.10	0.20	0.20
		CW	14.60	13.70	14.80	80.00	6.40	0.40	0.30
	DenseNet	RAP	8.00	7.30	7.70	29.00	4.50	0.30	0.30
		LIMANS	91.10	87.60	74.00	74.10	62.70	8.40	7.70
		AutoAttack	32.00	28.20	19.50	21.10	98.90	0.80	0.60
		UAP	4.70	3.80	2.20	2.70	2.00	0.50	0.40
		UAPPGD	4.80	5.60	2.00	2.70	2.80	0.40	0.40
		CW	10.00	8.20	5.70	7.40	79.10	0.60	0.30
	VGG	RAP	8.80	7.10	5.20	6.50	32.10	0.30	0.50
		LIMANS	94.20	89.00	74.80	71.00	71.70	8.00	7.10
		AutoAttack	6.70	8.30	8.00	8.50	9.60	24.60	11.00
		UAP	3.40	3.10	2.50	2.30	1.80	0.50	0.40
		UAPPGD	2.70	2.10	2.00	1.70	2.60	0.30	0.10
		CW	9.50	11.60	10.60	10.00	11.60	22.90	3.90
	R-r18	RAP	8.70	7.70	7.60	8.10	9.60	10.70	4.50
		LIMANS	58.70	53.80	50.30	50.70	41.80	17.60	14.60
		AutoAttack	7.70	7.80	8.20	7.80	8.90	15.20	22.50
		UAP	3.00	3.10	2.40	2.90	2.60	0.90	0.40
		UAPPGD	2.90	2.80	2.20	1.00	1.60	0.70	0.60
		CW	10.30	9.10	13.00	10.60	10.40	8.80	21.20
	R-wr-34-10	RAP	8.10	7.30	8.10	7.60	7.70	7.10	9.90
		LIMANS	59.10	54.80	51.80	50.00	42.50 17.00	14.70

Table 2 .

 2 8: Transferability performance of the LIMANS ∞ attack on ImageNet ( = 4/255), in terms of fooling rates. The best transferable results are marked in bold font.

	Source model \Target model MobileNet ResNet18 DenseNet	VGG	R-r18	R-wr50
		AutoAttack	100	26.38	20.44	26.94	1.64	1.24
		UAP	48.48	11.5	10.46	17.28	1.8	0.84
		UAPPGD	69.94	18.04	14.34	22.34	2.72	1.56
		TI-FGSM	99.74	36.98	31.66	31.24	3.2	2.56
		VMI-FGSM	100	44.84	37.92	42.92	2.92	2.04
	MobileNet	VNI-FGSM	99.98	44.64	36.54	43.62	2.88	2.00
		NAA	84.56	15.1	11.72	16.88	2.1	1.2
		RAP	96.52	54.58	47.24	49.16	3.72	3.16
		LIMANS	75.24	50.06	46.94	44.34	10.02	5.62
		AutoAttack	40.3	100	35.76	34.9	1.8	1.34
		UAP	13.34	11.3	9.00	11.72	1.36	0.86
		UAPPGD	25.3	47.22	18.44	23.26	2.5	1.44
		TI-FGSM	32.06	99.84	31.38	31.66	2.98	2.8
		VMI-FGSM	56.5	100	51.78	50.2	2.9	2.04
	ResNet18	VNI-FGSM	56.74	99.98	51.4	51.42	2.84	2.04
		NAA	22.54	97.94	14.84	19.3	2.12	1.2
		RAP	53.36	96.74	51.30	50.60	3.80	3.14
		LIMANS	59.16	59.16	53.14	48.28	10.48	6.62
		AutoAttack	37.72	40.4	100	30.22	1.8	1.3
		UAP	12.76	9.94	9.8	11.42	1.24	0.92
		UAPPGD	22.72	20.7	40.04	20.18	2.48	1.28
		TI-FGSM	30.1	35.56	99.66	27	3.12	2.32
		VMI-FGSM	52.22	55.44	99.98	44.82	2.9	2.06
	DenseNet	VNI-FGSM	53.88	56.9	99.98	46.16	2.64	2.1
		NAA	24.22	25.68	98.34	21.38	1.34	1.42
		RAP	48.16	54.12	96.76	42.00	3.12	3.30
		LIMANS	58.86	56.9	57.26	47.74	11.3	7.32
		AutoAttack	47.94	40.06	32.62	100	2.34	1.42
		UAP	13.34	9.8	8.82	13.6	1.34	0.78
		UAPPGD	24.42	23.16	18.12	46.26	2.54	1.6
		TI-FGSM	33.2	38.26	29.3	99.4	2.96	2.28
		VMI-FGSM	57.52	53.46	43.76	99.86	2.9	2.2
	VGG	VNI-FGSM	57.98	53.96	42.88	99.84	2.76	2.24
		NAA	19.62	14.92	12.18	79.96	2.18	1.4
		RAP	53.14	53.12	42.68	95.68	3.48	2.84
		LIMANS	57.68	54.14	50.04	51.62	10.68	6.24
		AutoAttack	13.7	15.8	10.82	14.6	71.74	10.78
		UAP	11.52	9.32	8.46	10.90	1.44	1.16
		UAPPGD	14.00	12.34	11.20	13.56	3.14	1.66
		TI-FGSM	11.88	13.42	10.08	11.02	54.46	10.14
		VMI-FGSM	17.00	17.80	12.12	16.08	64.98	11.94
	R-r18	VNI-FGSM	16.14	17.66	12.48	16.08	63.22	11.74
		NAA	11.46	10.86	9.34	11.42	21.48	4.9
		RAP	11.32	10.80	8.16	10.32	45.80	7.94
		LIMANS	37.14	33.2	33.76	29.90	29.84	12.94
		AutoAttack	20.14	22.76	17.36	19.44	15.42	59.02
		UAP	9.88	7.60	6.96	8.62	1.84	1.24
		UAPPGD	14.54	12.56	10.92	14.36	2.16	1.38
		TI-FGSM	14.16	16.34	12.68	13.68	17.16	43.66
		VMI-FGSM	24.22	26.66	20.12	23.86	17.82	54.56
	R-wr50	VNI-FGSM	23.88	26.22	19.68	23.28	18.00	52.28
		NAA	14.08	13.12	10.20	14.04	9.82	12.58
		RAP	13.82	14.06	10.52	13.5	15.54	34.1
		LIMANS	42.18	42.5	42.46	34.22	23.7	18.02

Table 3 .

 3 1: Time Consumption (in seconds) for the optimization of equation (3.13) with the POT solver and our custom Sinkhorn solver under different learning rate with f = 4.

		(a) Balanced data.	(b) Unbalanced data.
	LR	POT	Custom	LR	POT	Custom
	1e-3 73.66	152.16	1e-3 96.01	220.23
	5e-3 24.26	155.78	5e-3 34.79	445.73
	1e-2 10.91	140.62	1e-2 17.17	454.97
	5e-2 10.79	171.27	5e-2 18.04	812.61
	1e-1 11.44	360.43	1e-1 14.67	616.96
	5e-1 14.94	138.69	5e-1 25.66	395.7
	1	64.71	350.12	1	25.98	444.54

Table 3 . 3 :

 33 Optimization of the cost matrix with a medium dimension (f = 16), using the POT Linear Programming solver for both balanced and unbalanced data, under different learning rates. The optimization error is estimated using the 2 norm between the found solution and the true cost matrix.

	(a) Balanced data.	(b) Unbalanced data.
	LR	Error Time	LR	Error	Time
	1e-4	0.03	0.13	1e-4 0.023	44.98
	5e-4 0.021 44.18	5e-4 0.023	99.97
	1e-3 0.015 44.27	1e-3 0.012	99.64
	5e-3 0.001 44.49	5e-3 0.003	95.52
	1e-2 0.001 41.47	1e-2 0.003	95.63
	5e-2	0	13.77	5e-2 0.008	96.25
	0.1	0.015 43.94	0.1	0.018	99.82
	0.5	0.054 44.01	0.5	0.156	99.8
	1	0.091 44.62	1	0.099 100.25
	5	0.273 44.69	5	2.348	9.17
	10	0.397 44.15	10	4.978	9.05
	50	1.309 44.74	50	24.83	8.79

Table 3 .

 3 7: Optimization of the cost matrix for both a small (f = 4) and a medium (f = 16) dimension, using the POT Linear Programming solver for balanced data with different magnitudes of the original cost matrix, employing a medium learning rate (γ = 0.5).

				(b) Medium dimension (f = 16).
		Error Time	Magnitude Error Time
	1	0	11.45	1	0	13.77
	2	0	7.13	2	0.001 38.98
	5	0.002 11.13	5	0.012 47.73
	10	0.006 18.36	10	0.068 49.75
	50	3.022 43.81	50	1.126 47.76
				(b) Medium dimension (f = 16).
	Magnitude Error Time	Magnitude Error Time
	1	0	11.19	1	0.049 51.28
	2	0	9.56	2	0.08	44.9
	5	0	8.4	5	0.001 19.71
	10	0.002 7.32	10	0.002 23.38
	50	0.025 12.78	50	0.147 50.67

(a) Small dimension (f = 4).

Should we ignore too close neighbors points to ensure a good optimization?

Table 3 .

 3 9: Optimization of the cost matrix for both small and large dimensions, using the POT Linear Programming solver for balanced data with varying minimum distance width ρ thresholds accepted between points.

					(b) γ = 5.
		Error Time	Magnitude Error Time
	1	0.058 61.58	1	0.527 62.4
	2	0.049 66.62	2	0.531 62.48
	5	0.041 59.88	5	0.537 61.06
	10	0.045 57.43	10	0.501 61.55
	50	0.313 56.05	50	0.365 59.74
					(b) f = 64.
	ρ	Error Time	ρ	Error Time
	0	0	49.2	0	0.007 42.3
	0.001	0	44.6	0.001 0.007 40.8
	0.005	0	41.3	0.005 0.007 40.5
	0.01	0	40.5	0.01 0.007 40.1

(a) f = 4.

Table 3 .

 3 11 compares the original classifier's performances to a 1-nearest neighbors WD C classification.

	System	Précision Test
	Classifier	99.01%
	WD C	98.90%

Table 3 .

 3

11: Classification performances comparison between the original classifier and a 1-nearest neighbors according to the WD C .

  .6. 12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

	Label: 1	Label: 2	Label: 3 Mean Feature per label	Label: 4	Label: 5
		12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32
	Label: 6	Label: 7	Label: 8	Label: 9	Label: 10
	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32	12345678910 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

  1/2 X . (A.19) Let the eigenvalues and eigenvectors of R 2 be (λ 1 , . . . , λ 2N ) resp. (v 1 , . . . , v 2N ). The Singular-Value Decomposition of R 2 gives R

2 = VDV -1 , (A.20) V

  .[START_REF] Bengio | Greedy layer-wise training of deep networks[END_REF] Example of optimized adversarial patch aiming to fool the detection of light stops. Source: Meta Adversarial Training against Universal Patches[START_REF] Metzen | Meta adversarial training against universal patches[END_REF]. . . . . . . . . . . . 171.15 An illustration of the patch application operator. Source: Adversarial Patch[START_REF] Brown | Adversarial patch[END_REF]. . . . 18 1.16 Examples of real-world adversarial examples produced by someone altering sign boards. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 1.17 Originally proposed common corruptions. Source: Benchmarking neural network robustness to common corruptions and perturbations [72]. . . . . . . . . . . . . . . . 19 1.18 Examples of varying outdoor images conditions acting as adversarial perturbations in the case of autonomous cars object recognition. Source: Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming [109]. . . . . 20 1.19 Overview of 27 corruptions for 3D object detection proposed by [47]. See that the common corruptions are categorized into weather, sensor, motion, object, and alignment and that some corruptions are effective for one modality, while others are applied to both. Source: Benchmarking Robustness of 3D Object Detection to Common Corruptions in Autonomous Driving [47]. . . . . . . . . . . . . . . . . . . . . . . 20 1.20 Examples of UAP produced universal adversarial perturbations on different neural network architectures under the ∞ norm constraint. Source: Universal Adversarial Perturbations [111]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 1.21 Examples of singular vector UAP produced universal adversarial perturbations constructed using various layers of various DNNs on different neural network architectures under the ∞ norm constraint. Source: Art of Singular Vectors and Universal Adversarial Perturbations [83]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 1.22 Schema of a more robust classifier's decision boundary. In a dashed line, the classifier's decision boundary is close the both cloud points, making the computed adversarial examples very close to the original examples. In a solid line, the shown decision boundary is more robust as the computed adversarial examples are at a higher distance from the original examples. Source: Liwei Wang's talk at "A Blessing in Disguise: The Prospects and Perils of Adversarial Machine Learning" work ICML 2021. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 1.23 Schema of the adversarial examples detector adversarial defense. . . . . . . . . . . . 34 1.24 Published adversarial defense's robust accuracy according to the year of publication. Source: www.robustbench.io . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 1.25 Schema of the two adversarial examples vision in the community, with the universal adversarial examples in the middle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 2.1 Example of an autoencoder performing compression and the decompression of an MNIST image. See that the code outputted by the encoder is of lower dimension than the images but allows for full reconstruction of the image, therefore containing all the information within the image. Source: Autoencoders [11]. . . . . . . . . . . . . 38 2.2 Comparison with state-of-the-art adversarial attacks of ADiL with different hyperparameters λ 1 and λ 2 on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of adversarial attacks, while the rMSE of the produced adversarial noises is displayed on the x-axis. . . . . . . . . . . . . . . . . 42 XII 2.3 ADiL's performances with different numbers of atoms M on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of adversarial attacks, while on the x-axis is displayed the rMSE of the produced adversarial noises. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 ADiL performances with a different number of training data points N for multiple numbers of atoms M = 5, 10, 15. The dictionary D is optimized on a LeNet classifier on the CIFAR-10 dataset. On the y-axis is displayed the test fooling rate performances of ADiL, while on the x-axis is displayed the number of training data points. . . . . . 2.5 Illustration of ADiL's atoms when setting M = 5 on ImageNet dataset for different classifiers, under the ∞ norm constraint. . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 ADiL's crafted adversarial perturbation mean 2 norm with different norm constraint bound. The dictionary D is optimized on a VGG classifier with M = 50. . . . . 2.7 Black-box ADiL inference performances according to the number of trails for the sampling of v for various p norm adversarial budget constraints of equation (2.9). . 2.8 Fooling Rate of ADiL with different number of atoms on a LeNet classifier. The dots represent the ADiL attack on a standard LeNet classifier, while the diamonds display fooling rates on a robust LeNet classifier trained with the defense. Fooling rate values are precisely reported in Table 2.2. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.9 Evolution of ADiL's training loss and the relative norm of D and V that is evaluated as norm(D t -1 )-norm(D

t ) norm(D t -1 -D t )

https://chat.openai.com

https://www.tesla.com/autopilot

https://computerhistory.org/blog/where-to-a-history-of-autonomous-vehicles/?key=where-to-a-history-ofautonomous-vehicles ix

https://chat.openai.com

https://www.tesla.com/autopilot

https://computerhistory.org/blog/where-to-a-history-of-autonomous-vehicles/?key=where-to-a-history-of-

https://paperswithcode.com/sota

https://www.fast.ai/

https://fidle.cnrs.fr

https://github.com/lucasanquetil

https://www.kaggle.com/

& TRAINING: Learning 𝑫 and {𝒗 " } "$% &

https://nicholas.carlini.com/

https://robustbench.github.io/

Remerciements

Je commence par remercier Stéphane Canu, mon directeur de thèse, qui m'a accompagné et guidé durant ces trois années de travaux intensifs. Je remercie les membres du jury qui ont accepté de rapporter et d'examiner mon travail de thèse. Vos commentaires m'ont permis de prendre du recul sur mes travaux et de les reconnaitre à leur juste valeur.

Je remercie également les deux membres de mon comité de suivi de thèse, Mike Gartrell et Laurent Heutte, qui m'ont rassuré, conseillé et accompagné sur mes travaux et mes choix personnels. Je remercie avec nostalgie tous les membres du LITIS qui ont fait partie de l'équipe pédagogique qui m'a accompagné, suivi, guidé et qui m'a considéré ces huit dernières années, j'en garde un chaleureux souvenir et une haute estime à leur égard. Merci pour leur investissement, leur professionnalisme et leur sympathie Je remercie ma psychothérapeute de m'avoir soutenu et de m'avoir accompagné vers un jour plus lumineux.

Enfin je remercie tout particulièrement ma famille, mes amis et ma compagne qui m'ont écouté, aidé et profondement aimé.

Per ardua ad astra 1. We first write down the Lagrangian formulation of the original problem 2. We find the optimal solution of the plan from this Lagrangian formulation 3. We use a rewriting of the dual variable to rewrite the Lagrangian problem 4. We finish by highlighting the solver to efficiently solve the problem

Is the mini-batch size a sensitive parameter?

In stochastic gradient decent optimizations the mini-batch can sometimes be a sensitive parameter. Indeed, when not high enough, the gradients may not always point in the best direction to reach the optimal solution, leading only to a local solution. Table 3.5: Optimization of the cost matrix for a small dimension (f = 4), medium dimension (f = 16), and high dimension (f = 64), using the POT Linear Programming solver for both balanced and unbalanced data, under different minibatch sizes B. The optimization error is estimated using the 2 norm between the found solution and the true cost matrix.

B

Error Time 55.18 In Table 3.5a we perform the proposed optimization for various sizes of mini-batch. The minibatch size has absolutely no influence on the quality of the solution found, except for the time consumption. We recall that the displayed results show performances on CPU-performed optimizations, while the mini-batch strategy is especially suited to GPU computations, we do not think it is worth considering the mini-batch size as a time consumption sensitive hyper-parameter.

On the other hand, from Table 3.5c and 3.5b, we can observe that in medium and large dimensions, the mini-batch size becomes a critical parameter. Indeed, it is only when using a relatively large mini-batch size (f = 128) that we achieve rapid convergence. Beyond this required minimum size, increasing the mini-batch size doesn't further improve optimization results.

Therefore, we now know that a sufficiently large mini-batch size is required to ensure the reaching of the optimal solution for realistic feature sizes. However, it won't ultimately produce better results according to its increase.

Does the magnitude of the ground truth cost matrix to recover have any impact on the optimization?

The magnitude of the values within the ground truth cost matrix to recover may severely affect the optimization, as it could easily lead to finding a descent local solution but prevent finding the optimal solution C * . Thus we performed the same optimization with the cost matrix multiplied by a magnitude coefficient factor which increases the distance between the points values and the cost matrix' entries to recover.

At first, Table 3.6 tells us that the magnitude of the cost matrix to be retrieved has a significant impact on the quality of our optimization.

However, by increasing the learning rate, Table 3.7 demonstrates that it adjusts the optimization to the magnitude of the optimal cost matrix to be found.

Even if in small and medium dimensions, increasing the learning rate led to enhancing the optimization, by increasing the learning rate too much, Table 3.8 reveals that optimization can be damaged in the case of a large dimension.

Once again, the learning rate is a critical factor for successful optimization. We will need to choose it very carefully since it adjusts the optimization for both changes in dimension and changes in the magnitude of the matrix to be retrieved. effect for our purposes, as during the classifier training, it essentially nullifies unused nodes in the penultimate layer to achieve better classification performance. As a result, h(x) ∈ R f will be a very sparse vector. This sparsity is advantageous for us, as it achieves meaningful dimensionality reduction of the features (selecting them based on their relevance to the classification task). It further reduces the complexity of the WD C inner optimization (the transport plan objective).

Analysis of the MNIST extracted features

As good scientific behavior, data analysis of the features must be performed. Even though we don't have any tools to perform such analysis, we empirically look at the features' entries activation to see if we have some modes emerging according to the labels, which would mean relevant features.

In practice, we observe that for a layer of size f = 16, we end up with non-zero features with an average size of 5. For a layer of size f = 32, we ultimately have non-zero features with an average size of 10. Figure 3.3 displays the distribution of these features when setting f = 16. From Figure 3.3, we see that only a few components of the features are non-zero. Therefore, we understand that the classifier's training, to effectively separate the classes, has been successful. Indeed, we observe that for each class, only a few dimensions dominate the distribution of features, and we never observe complete overlap between two sets of dominant dimensions among the classes. Along with the label's feature dimensions activation, we considered the mean feature and

Proof. Assuming that D ∈ D, then we have that

Hence v 1 ≤ implies Dv p ≤ .

A.2 From a regularised Wasserstein Distance to a Sinkhorn divergence

We will demonstrate how the regularised Wasserstein Distance A.3 boils down to the Sinkhorn divergence which allows to solve it efficiently [START_REF] Cuturi | Sinkhorn distances: Lightspeed computation of optimal transport[END_REF]. The demonstration will be performed in 4 steps:

2. We show when

(a) Reformulation of the problem with W = X Y . Eq. 11 (b) Expliciting the constraints, Σ W 0. Eq. 14 (c) Formulation of the dual problem. Eq. 16 (d) Solving the dual problem. Eq. 20 (e) Coming back to the primal problem. Eq. 22 (f ) Reformulation using SVD. Eq. 25 (g) Concluding. Eq. 28 3. Coming back to the general case where µ X and µ Y are not longer assumed to be zero.

The original formulation of the WD between f X and f Y is

Where C denotes a cost function between the random variables X and Y and Γ( f X , f Y ) denotes the collection of all measures on the random multivariate variable W = (X, Y) with marginals are f X (x) and f Y (y) on the first and second factors respectively. Since we focus on the l 2 norm as cost function C, we have

From the very definition of f W being a measure on W = (X, Y), the problem simplifies to

A.3.2 We show when µ

We rewrite the problem according to the variable W

There is an implicit assumption, Σ W being positive semi definite Σ W 0. A matrix P ∈ R 2N is only positive semi definite if and only if ∃B ∈ R 2N×2N such that B T B = P. One can prove that With V ∈ R 2N×2N a matrix made of the eigenvectors as columns and D ∈ R 2N×2N a diagonal matrix which entries D i i are the eigenvalues. From this form we infer the forms of R

D 1/2 being a diagonal matrix which entries D i i are the square root of R 2 's eigenvalues λ i ∀i = 1, . . . , 2N. However each λ i has two possible square root (-λ 1/2 i , +λ 1/2 i ) which confirm 2 N possible D 1/2 matrices. Thus we encode all of them by timing each eigenvalue' square root with i ∈ {-1, 1}. With the definition of R we derive a definition of Σ XY

Σ XY has the same eigenvalues as R so the maximum and minimum value of Tr (Σ XY ) = 2N i =1 λ i are reached when either i = 1 or i = -1, ∀i = 1, . . . , 2N, which we simplify by respectively writing ±. We rewrite Σ XY with this new notation

We already saw that Σ

Hence Σ XY = (Σ X Σ Y ) 1/2 . Which conclude the proof by giving that Tr (Σ XY + Σ T XY ) is maximised or minimised by taking