Benoît Bonnet

Thèse Présentée

Christophe Rosenberger Professeur

Gildas Avoine

Pascal Frossard Professeur

Understanding, Taming, and Defending From Adversarial Examples

Keywords: Understanding, taming, and defending from adversarial examples

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • THÈSE DE DOCTORAT DE L'UNIVERSITÉ DE RENNES ÉCOLE DOCTORALE N O 601
Mathématiques, Télécommunications, Informatique, Signal, Systèmes, Électronique Spécialité : Signal, Image, Vision

ACKNOWLEDGEMENT

Je tiens à remercier de nombreuses personnes qui ont toutes eu une contribution dans l'achèvement de cette thèse.

Premièrement, je tiens à remercier mes deux directeurs de thèse Teddy Furon et Patrick Bas. Je suis extrêmement reconnaissant de l'encadrement que j'ai eu au cours de cette thèse. Au début de ma thèse, ils ont su me mettre sur les rails immédiatement pour commencer à expérimenter et publier rapidement. De plus, la crise sanitaire ayant commencé peu après le début de ma thèse, ils ont su m'assurer tout le suivi dont j'avais besoin pour continuer dans ma recherche malgré l'isolement. Tout le monde a appris à travailler de chez soi ces dernières années, je considère avoir eu beaucoup d'aide pour y parvenir. Je ne peux que souhaiter à tout doctorant et doctorante d'avoir un tel encadrement et une telle bienveillance de la part de ses directeurs de thèse.

Deuxièmement je tiens à remercier le centre IRISA et en particulier l'équipe Linkmedia de m'avoir accueilli. L'équipe a bien changé depuis mon arrivée: certains membres sont partis, d'autres sont arrivés et certains même sont partis puis revenus. Je voudrais remercier en particulier Hanwei Zhang qui m'a apporté beaucoup d'aide et m'a aidé à m'approprier les différents outils. Thibault Maho avec qui partager un bureau n'aura jamais été aussi plaisant, et qui aura été source de beaucoup de discussions sur nos recherches respectives. Guillaume Le Noé-Bienvenu avec qui j'ai pu passer de très bons moments en dehors du centre et qui a toujours su apporté beaucoup d'énergie à cette équipe. Et enfin Aurélie Patier avec qui tout est toujours facile même lorsqu'on est pas toujours des plus efficaces.

Enfin je tiens à remercier tout particulièrement Margot Pernet sans qui je n'aurais jamais pu aller au bout de cette thèse et à vrai dire sans qui je ne l'aurais même pas commencé. Elle a toujours été d'un soutien indéfectible et j'essaierai tous les jours de lui apporter le même soutien. Je remercie également toute ma famille et mes parents particulièrement qui se disent sans doute qu'après tout ça, il serait temps que leurs enfants arrêtent les études, mais qui seront toujours prêt à apporter le soutien nécessaire si cela devait encore durer. Et enfin, merci à mes deux compagnons de télétravail: Tiny et Gontrand qui auront su rendre la maison bien plus vivante pendant ces périodes d'isolement.

"Que représente cette image?" Le challenge ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] est un parfait exemple de cette tâche. C'était une compétition annuelle permettant de classer les performances des classifieurs actuels. Le but étant de classer des images dans 1 000 catégories (ou classes) prédéfinies, avec plusieurs millions d'exemples au total. C'est précisément de cette compétition qu'a émergé la domination de l'Apprentissage Profond. En 2012, Krizhevsky et al. remporte la compétition avec le Réseau de Neurones Profond (DNN pour Deep Neural Network) nommé AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF]. Il s'agissait du premier DNN à remporter le concours. Concours qui ne sera par la suite remporté que par d'autres DNNs qui apporteront leur lot d'innovations. Il s'agit du début de l'ère de l'Apprentissage Profond. La notion de sécurité n'est pas toujours présente dans une tâche de classification. Certaines applications peuvent toutefois s'avérer sensibles dans leur contexte. Il peut s'agir de défense nationale par exemple (reconnaissance et cartographie), de filtre sur des réseaux sociaux (empêcher le partage de contenu inapproprié ), ou de véhicules autonomes (systèmes de navigation). Ces véhicules se basent essentiellement sur de la Vision par Ordinateur.

Il apparaît essentiel d'explorer toutes les potentielles menaces qui nuiraient à l'intégrité de ces systèmes. Celles-ci sont nombreuses. Certaines interviennent dès la phase d'apprentissage:

-Empoisonnement: généralement dans le cas de données d'entraînement collectées automatiquement. Les données sont volontairement dégradées pour déteriorer les performances du modèle entraîné par la suite. [START_REF] Nelson | Exploiting machine learning to subvert your spam filter[END_REF]. -Porte Dérobée: La personne qui déploie le modèle peut l'avoir entraîné à réagir différement sur des données spécifiques. Il peut donc ensuite se soustraire à une reconnaissance ou détection, ou empêcher une bonne classification [START_REF] Liu | Trojaning attack on neural networks[END_REF]. D'autres menaces concernent l'intégrité d'informations importantes, liées à la confidentialité:

-Inversion de Modèle: l'attaquant cherche à recréer les données qui ont été utilisées lors de l'entraînement [START_REF] Fredrikson | Model inversion attacks that exploit confidence information and basic countermeasures[END_REF][START_REF] Zhang | The secret revealer: Generative model-inversion attacks against deep neural networks[END_REF]. -Inférence d'Appartenance: similaire à la précédente, l'attaquant essaye ici de trouver si des données spécifiques qu'il détient ont été utilisées lors de l'entraînement du modèle [START_REF] Hayes | Logan: Membership inference attacks against generative models[END_REF]. -Extraction de Modèle: une telle attaque vise à cloner un modèle existant avec la plus grande précision [START_REF] Tramèr | Stealing machine learning models via prediction {APIs}[END_REF]. Cela passe généralement par une abondance de requêtes contre laquelle il peut être nécessaire de se protéger.

Enfin il reste les menaces qui interviennent lors de l'usage final du modèle:

-Attaque par Déni de Service: ces attaques sont bien connues de tous les systèmes fonctionnant par requête. L'attaquant inonde le service de demandes pour le rendre inutilisable à d'autres utilisateurs. -Attaque par reprogrammation: l'attaquant modifie un modèle existant pour l'utiliser à d'autres fins. [START_REF] Tramèr | Stealing machine learning models via prediction {APIs}[END_REF] [START_REF] Gamaleldin F Elsayed | Adversarial reprogramming of neural networks[END_REF]. -Attaque par Évasion ou Attaque Adversaire: le sujet de cette thèse. Une telle attaque manipule les données d'entrée de manière imperceptible pour tromper un classifieur.

Les attaques par évasion créent ce que l'on appelle des exemples adversaires ou adverses. Ces exemples sont des données qui ressemblent à des données normales ou naturelles mais qui trompent le DNN. La prédiction est incohérente avec ce qu'un humain perçoit. Cette thèse propose une étude approfondie de ces exemples: leur création, leur viabilité, pourquoi ils existent et comment s'en protéger. Voici les différents chapitres autour desquels s'articule cette thèse. Les deux premiers chapitres comportent des éléments théoriques ainsi qu'une revue de la littérature. Les chapitres suivants s'appuient sur des contributions publiées au cours de cette thèse.

Résumé des Chapitres

Le chapitre 4 permet de définir la base du manuscrit. Nous expliquons dans un premier temps brièvement l'histoire de la classification d'image. Cela amène à suivre les avancées qui ont mené au début de l'ère du Deep Learning avec l'arrivée d'Alexnet. Cette première partie du chapitre définit également les bases de l'entraînement des réseaux de neurones artificiels. La deuxième partie de ce chapitre part à la découverte des vul-nérabilités adverses. Les différents cadres d'attaques sont expliqués ainsi que le principe de base des attaques boîtes-blanches. Celui-ci est intrinsèquement lié à la procédure d'entraînement. Enfin les différentes notations ainsi que les attaques utilisées tout au cours de cette thèse sont expliquées.

Le chapitre 5 revient sur l'origine des vulnérabilités adversaires. Dans une première partie, nous cherchons à comprendre la raison de leur existence. Toutes les sources potentielles de faille sont explorées. Comprendre la raison d'une faille du système permet de chercher dans la bonne direction pour corriger celle-ci. Dans une deuxième partie, nous explorons donc les défenses qui existent. Elles sont catégorisées selon leur mécanisme de fonctionnement et leur spécificité.

Le chapitre 6 correspond à notre premier chapitre de contribution. Ces travaux ont pour but d'établir un banc d'évaluation des réseaux pour des attaques en boîte-blanche et en boîte-noire. C'est le seul chapitre qui parlera d'attaque boîte-noire. Nous y définissons également les métriques et les courbes d'évaluation d'une attaque sur un réseau. Ces courbes sont utilisées tout au long de cette thèse.

Le chapitre 7 parle de travaux qui partent d'une observation simple: la plupart des travaux sur les exemples adverses ne produisent pas d'image. Même les illustrations les plus connues dans la littérature ne sont en réalité pas des images adverses. La raison est simple: les attaques sont considérées comme existant dans l'espace des données traitées par le DNN. Cet espace correspond au pré-traitement que subissent les images. Or cet espace est pseudo-continu, considéré par la plupart comme continu. Lorsque le pré-traitement est inversé, les valeurs de pixels ne sont pas entières. Il ne s'agit donc pas d'images numériques. Arrondir est dans bien des cas une mauvaise solution qui détruit le signal adverse. Ce chapitre propose une quantification de ces objets aussi bien dans le domaine spatial que JPEG. Les images ainsi créées restent bien adverses.

Le chapitre 8 est un chapitre qui est lui aussi lié à des notions de quantification. Comme pour le chapitre précédent, nous cherchons à quantifier des images adverses. Cette fois-ci en revanche, nous nous inspirons des travaux dans le domaine de la stéganographie pour le faire. Cette science a pour objectif de dissimuler des informations au sein d'un medium. Une image par exemple. Les stratégies cherchent évidemment à rendre le signal inséré le plus discret possible. Nous mettons donc au point dans ce chapitre des détecteurs pour identifier les images adverses. Certains de ces détecteurs sont employés en stéganalyse, la science antagoniste de la stéganographie. Nous montrons alors quels gains de détectabilité peuvent être obtenus selon les différentes stratégies de quantification.

Le chapitre 9 correspond à des travaux liés à la compréhension et à la défense. Plutôt que d'étudier un mécanisme de défense, nous explorons l'impact des techniques de redimensionnement employées au cours du pré-traitement des images. Les réseaux de neurones sont capables de traiter des tailles différentes d'images, définies par leur architecture et leur entraînement. Nous cherchons donc à savoir quelle stratégie est la meilleure pour la défense à ce sujet. Dans une partie théorique, nous évaluons d'abord l'impact de la taille d'image sur le signal adverse. Nous explorons ensuite les effets des différentes méthodes de redimensionnement pour trouver la meilleure stratégie à obtenir.

Résumé des Contributions

La classification d'images est la principale application de la VO. Les réseaux de neurones artificiels dominent largement le domaine. Cependant, on les sait vulnérables à des petites perturbations sur les entrées: les exemples adversaires. Dans le Chap. 4 nous avons brièvement expliqué le fonctionnement de ces réseaux, leur entraînement, et la fabrication d'images adversaires. Dans le Chap. 5 nous avons étudié la raison même de l'existence de ces exemples ainsi que des travaux qui visent à s'en défendre. Mais une des conclusions de cette première partie est que les réseaux de neurones sont vulnérables aux attaques de par leur fonctionnement même.

Le but de cette thèse était d'étudier ces exemples dans un contexte réaliste. Nous avons pris plusieurs décisions dans cette direction. Nous étudions par exemple la tâche de classification que nous jugeons la plus réaliste: Imagenet. Les images qui composent le jeu de données ressemblent à des images qui seraient utilisées pour une application réelle. Et les exemples adverses devraient également être des images réalistes. Dans les chapitres 7 et 7, nous cherchons à quantifier des exemples adversaires dont les valeurs ne sont pas entières. Dans le Chap. 7, il s'agit de minimiser la distorsion dans le domaine spatial (PNG) ou fréquentiel (JPEG). Dans le Chap. 8, nous cherchons à minimiser la détectabilité.

Dans le Chap. 8, nous jouons également le rôle de la défense en mettant au point des détecteurs en partie inspiré de la stéganalyse. On y observe que la détection est efficace lorsque l'on connaît exactement le type d'attaque (et quantification). Lorsque l'on change la stratégie de quantification, dans le but de la rendre moins détectable, les détecteurs précédents ne parviennent plus vraiment à détecter les adversaires. Et même lorsque l'on réentraîne des détecteurs sur des adversaire plus discrets, ils ne parviennent pas à retrouver leur performances initiales (données quantifiées 'naïvement').

Dans le Chap. 9, nous étudions également l'attaque et la défense. On y analyse l'impact de la taille et du redimensionnement sur les images adverses. La taille d'entrée d'un réseau varie, même pour une même tâche comme Imagenet. Et lorsque l'image n'est pas à la bonne taille, elle doit être redimensionnée. Cette étape affecte évidemment le signal adverse sur des données attaquées. On conclue dans une première partie que les réseaux utilisant des plus petites tailles semblent plus robustes aux attaques. Mais il s'agit aussi d'un équilibre entre la performance et la robustesse: le 'No Free Lunch' décrit dans le Chap. 5. Une deuxième observation est que le redimensionnement affecte logiquement plus le signal lorsque de l'anti-crénélage est employé. Celui-ci n'affecte pas les performances du réseau et devrait toujours être utilisé. Enfin, chaque attaque va être spécifique à la méthode de redimensionnement utilisée. La transférabilité est négligeable. Mais nous montrons également que l'utilisation d'un ensemble de modèle peut suffire à attaquer toutes ces méthodes simultanément. Cela rajoute évidemment de la distorsion lors de l'attaque.

Vers une Evaluation Equitable

Nous n'avons travaillé qu'avec Imagenet. Nous considérons que les données plus petites peuvent être pratique pour étudier les exemples adversaires, mais que les travaux publiés devraient s'appliquer à Imagenet pour être comparables. Mais même lorsque les articles utilisent Imagenet, les données sont rarement les mêmes. Une pratique commune est par exemple de récupérer aléatoirement 1 000 images du jeu de validation, ce que nous avons fait dans certains de nos travaux. Mais il n'y a aucun consensus sur ces données. Cette méthode n'est peut-être pas la meilleure dans le cadre d'évaluation et comparaison. Nous avons l'exemple en Sect. 5.3.2 des travaux de [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF] qui ont extrait 5 000 images. Le taux de classification correcte est de 100% sur ces images (par le DNN étudié). Ces données ne semblent donc pas réellement aléatoires. Ce qui se rapproche le plus d'une bonne pratique est d'utiliser des jeux de données dédiés à la discipline, le Neurips Adversarial Challenge [START_REF] Kurakin | Adversarial attacks and defences competition[END_REF] 2017 par exemple.

Le deuxième critère d'évaluation serait comment l'attaque est utilisée. Les examples adverses sont généralement construits selon:

1. Une contrainte de Distorsion: l'exemple est construit avec une distorsion donnée, quelle que soit sa classification à l'issue.

2. Une contrainte de Succès: l'attaque continue de pousser la distorsion jusqu'à ce que l'exemple soit adverse.

3. Une Optimisation: l'attaque minimise la distorsion tout en s'assurant que l'exemple est adverse.

Chacun de ces scenarii peut être utilisé choisi selon l'intérêt des auteurs. Par exemple pour des travaux de détection, utiliser une contrainte de Distorsion est intéressante. Le signal a toujours la même intensité et l'exemple est probablement loin derrière la frontière de classe. A l'opposée, chercher à détruire le signal adverse par traitement quelconque sera plus compliqué. Ces travaux vont plutôt travailler avec un scnerario d'Optimisation.

Dans l'optique de travailler avec le meilleur des deux mondes, nous avons principalement travaillé avec le scenario d'optimisation. Mais ce choix n'était peut-être pas toujours optimal. Notamment lorsque nous avons étudié des questions de transférabilité (chapitres 7 et 9). Nous considèrons toutefois que ce scenario reste le plus réaliste. Pour cela nous avons principalement utilisé l'attaque BP pour laquelle nous avons développé une version best-effort (Chap. 6). Cette attaque fonctionne très rapidement et produit des exemples adverses basse-distorsion de manière très fiable.

Les Images Adverses

Nous avons beaucoup étudié la création d'images adverses. Travailler sur un scenario boîte-blanche signifie connaître le modèle. Mais probablement pas d'avoir accès aux représentations intermédiaires des données. Ce qui est souvent le cas des exemples adverses qui sont construits dans le domaine continu (tenseurs) et non discret (pixels). Dans les chapitres 7 et 8, nous expliquons que cette tâche n'est pas nécessairement triviale, surtout dans le cadre d'optimisation.

Une attaque dans le monde réelle se basera probablement sur des images postées en ligne, ou bien même physiques (autocollants par exemple). Des images en ligne pourraient être utilisées afin de contourner des filtres de contenu, ou pour protéger ses propres données contre de la collecte automatique de données. Nous avons travaillé sur cette dernière problématique dans un court article [10] visible en Annexe A. Des applications dans le monde physique pourraient par exemple servir à tromper des véhicules autonomes, ou bien à éviter des systèmes de reconnaissance faciale.

Produire des images adverses est donc en concordance avec cette thèse: faire des attaques réalistes. Toutefois, le scenario boîte-blanche n'est pas toujours le plus réaliste. Mais cela pourrait le devenir avec des attaques plus avancées d'extraction de modèles [START_REF] Tramèr | Stealing machine learning models via prediction {APIs}[END_REF].

Défenses et Perspectives

Nous avons considéré l'aspect défenses plusieurs fois au cours de cette thèse. Notamment pour la détections (chapitre 8), pour les ensembles de modèles (chapitres 7 et 9), et enfin sur les bonnes pratiques quant à la taille et au redimensionnement (chapitre 9). La sécurité est un domaine itératif et chaque contribution en attaque permet de progresser en défense. Par exemple dans le chapitre 8, nous jouons le rôle itératif des deux côtés.

Nous avons également travaillé sur de la robustification de modèles récemment. Mais ces travaux restent pour le moment non publiés. Un aperçu de ces travaux est donné en Annexe B. Le principe de ces travaux est de rendre les modèles plus robustes via un réentraînement non-supervisé. Cette méthode est semblable au logit pairing (décrit dans le Chap. 5), mais l'appariement s'effectue dans le domaine des features.

Un Dernier Mot

Nous avons parlé plusieurs fois de la compétition entre attaque et défense. Cela donne parfois naissance à des critiques biaisées sur les travaux de l'autre camp. Le manque d'unification des moyens d'évaluation est bien souvent un frein à la progression. En conclusion nous allons parler de travaux récents de Guo et al. [START_REF] Guo | Adversarially trained neural representations may already be as robust as corresponding biological neural representations[END_REF]. Ils expliquent que les réseaux les plus robustes aujourd'hui pourraient être finalement aussi robustes que les neurones biologiques. Pour cela, ils caractérisent un équivalent de signal adverse pour l'oeil humain. A partir de ce modèle, ils évaluent que nous pourrions être tout aussi sensibles à des signaux adverses précis. Au cours des dernières années, les progrès ont été tels dans l'Apprentissage Profond que l'on peut souvent être subjugués par leurs performances. Cela nous mène probablement à juger leurs vulnérabilités de manière critique. Nous sommes toutefois nous-mêmes vulnérables dans notre traitement de l'information. Les illusions d'optique sont un parfait exemple d'exploitation de ces vulnérabilités. La différence reste sans doute que nous sommes généralement conscients que notre perception est trompée.

INTRODUCTION

The term Artificial Intelligence (AI) was first coined at a workshop by John McCarthy in 1955 [START_REF] Mccarthy | A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955[END_REF]. Taking place at Dartmouth College (Hanover, New Hampshire), this workshop was proposed by Claude Shannon himself, the father of information theory [START_REF] Elwood | A mathematical theory of communication[END_REF]. This event is widely considered to be the cornerstone of AI. The lack of computational power and memory however led this new field to immediately know an AI winter [START_REF] Crevier | AI: the tumultuous history of the search for artificial intelligence[END_REF], longing for breakthroughs.

Artificial Intelligence is now a broad term to designate algorithms that are not based on hard-coded rules. These programs can translate an input, i.e. data, to an output. Different families of algorithms exist in AI. In particular, Machine Learning has emerged as the most flexible and adaptive system. Deep Learning algorithms are the largest models of Machine Learning. Their specificity is that they tackle large-dimensional data. Fields of applications can be sorted into three main categories:

-Natural Language Processing (NLP) which uses text-based data.

-Audio Signal Processing (ASP) which uses audio-based data.

-Computer Vision (CV) which uses image-based data.

The latter is at the heart of this thesis. Each of these fields can be divided into multiple sub-categories. CV has recently known an explosion in popularity with image generation. Models like DALLE-2 [START_REF] Ramesh | Hierarchical text-conditional image generation with clip latents[END_REF], Imagen [START_REF] Saharia | Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding[END_REF], or StableDiffusion [START_REF] Rombach | High-Resolution Image Synthesis with Latent Diffusion Models[END_REF] have brought light on the impressive possibilities offered by AI.

Yet the first and most popular task in CV remains image classification. It answers one simple question:

"What does this image represent?"

The ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] challenge well represents this task. It is a yearly competition that establishes a classification benchmark on 1,000 object categories (or classes) over millions of images. This is precisely on this task that Deep Learning emerged and started its dominance. In 2012, Krizhevsky et (DNN) AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF]. It was the first DNN to win the contest that would then only be won by other DNN models, marking the beginning of the Deep Learning era.

While most classification tasks are not linked to any notion of security, some might have crucial applications. For example in the case of national defense and security (scouting, mapping biometrics); in the case of online content filter (possibility of letting harmful or shocking content through); or in the case of autonomous vehicles, which could affect a broader population. Navigation systems rely mostly on CV tasks tied together. On such an important ask, one must investigate the security threats. And there are many. Some involve malicious intervention during the training phase:

-Poisoning: usually in the case of automatically collected data. The data is intentionally degraded to worsen the performance of the later trained model [START_REF] Nelson | Exploiting machine learning to subvert your spam filter[END_REF]. -Backdooring: anyone who trains a model might teach the model to react differently to specific data. This lets the same person to have secret knowledge on how to break classification [START_REF] Liu | Trojaning attack on neural networks[END_REF].

Some other security threats deal with extracting crucial information, closely related to data privacy:

-Model Inversion: the attacker tries to recreate data used during the training phase [START_REF] Fredrikson | Model inversion attacks that exploit confidence information and basic countermeasures[END_REF][START_REF] Zhang | The secret revealer: Generative model-inversion attacks against deep neural networks[END_REF]. -Membership Inference: similar to the previous one, the attacker here tries to find whether the specific data in their hands was used to train a model [START_REF] Hayes | Logan: Membership inference attacks against generative models[END_REF]. -Model Extraction: such attack aims at cloning the attacked model with the highest fidelity [START_REF] Tramèr | Stealing machine learning models via prediction {APIs}[END_REF].

The last category of threats involves downstream usage of the model:

-Denial of Service (DoS): these well-known attacks apply to all request-based systems. The attack is simple: a user floods the system with requests to either bring it down or prevent other users from using it. -Reprogramming Attack: the attacker modifies an existing model to perform another signal. DNNs can process various image sizes and it is not trivial to understand which size will have the best behavior regarding its robustness. Bring theory head to head with experimentation yields best-practice to adopt for the defender to protect a model.

Part I

Introduction on Adversarial Samples

Introduction

A Deep Neural Network (DNN) as a classifier is trained to classify images by the object represented in the picture. This is for instance the well-known ImageNet challenge encompassing a thousand classes. The state-of-the-art proposes impressive results as classifiers now do a better job than humans with fewer classification errors and much faster timings. The advent of the AlexNet DNN in 2012 is often seen as the turning point of 'Artificial Intelligence' in Computer Vision.

Yet, the recent literature of adversarial examples reveals that these classifiers are vulnerable to specific image modifications. The recent field of adversarial attacks explores ways of fooling DNNs in this fashion since the works of Szegedy et al. [START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. The goal of an attack is to modify an image with little distortion so that its predicted label differs from the ground truth. The perturbation is often a weak signal usually invisible to the human eye. Almost surely, no human would incorrectly classify these adversarial images. The perturbation is a priori both classifier and image specific. This topic is extremely interesting as it challenges the 'Artificial Intelligence' qualification too soon attributed to Deep Learning.

The literature considers three setups:

-White-Box: The attacker knows the classification model architecture and parameters. Most attacks use the very core strength of DNNs to fool them: gradient back-propagation. The very first attacks were FGSM [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF], IFGSM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] and Deep-Fool [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF], later on improved by PGD [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF], CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF], or BP [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF]. -Black-Box: The attacker only queries the model and observes its output. Attacks can not exploit the gradient. They thus either locally estimate it (Hop-SkipJump [START_REF] Chen | Hopskipjumpattack: A query-efficient decision-based attack[END_REF] or GeoDa [START_REF] Rahmati | GeoDA: a geometric framework for black-box adversarial attacks[END_REF]) or probe the class frontier such as SurFree [START_REF] Maho | Surfree: a fast surrogatefree black-box attack[END_REF]. -Gray-Box: The attacker has partial knowledge of his/her target, for instance, the classification model is public but some front-end defense mechanisms are secret.

These attacks are associated with two possible goals:

-Targeted: The attacker determines which class should the classifier predict over the adversarial sample. -Untargeted: The attacker only needs the final predicted class to differ from the ground-truth.

In this thesis, we explore mostly the setup of white-box and untargeted attacks. This configuration is the most favorable for the attacker. It allows us to study further the behavior of DNNs and unfold their secrets.

Adversarial examples are an emerging field in Information Forensics and Security, addressing the vulnerabilities of Machine Learning algorithms. They are also interesting in non-security applications. They constitute a means to better understand how deep learning models work (or do not work) [START_REF] Ortiz-Jiménez | Optimism in the face of adversity: Understanding and improving deep learning through adversarial robustness[END_REF]. This comes in addition to the generalization of the model measured by the accuracy over the validation set. They are key to investigating the frontiers between classes in the image space. In this non-security perspective, the quest for more robust models often resorts to adversarial training, smooth labeling, and network architecture modifications. These mechanisms aim at pushing away the frontiers from the training samples. Deep Learning is an incremental science. Current state-of-the-art models use many bricks that were laid down over the years. Slowly at first, for a lack of hardware and thus results. And then increasingly fast since the advent of AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF].

Deep convolutional networks 4.2.1 A Brief History

In 1957, F. Rosenblatt conceptualized and simulated the first artificial neuron further theorized in his published work in 1961 [START_REF] Rosenblatt | Principles of neurodynamics. perceptrons and the theory of brain mechanisms[END_REF]. Psychologist at first and then a neurobiologist, he drew inspiration from biological neurons to model the artificial neuron. The concept is simple: a linear combination of multiple inputs is calculated (Fig. 4.1). A nonlinear activation function acts as a threshold that 'fires' or simply outputs a signal if the processed input signal is sufficient. The artificial neuron is activated. From multiple neurons, stacked in layers (or fully-connected layer), one can build a Multi-Layer-Perceptron (MLP). Layers (or hidden layers when they are intermediary) are rows of perceptrons stacked one after another (Fig. 4.2). The MLP is capable of extracting complex information and relationship within the input data.

Another notable step towards modern Deep Learning was studied by H.J. Kelley in 1960 [START_REF] Henry | Gradient theory of optimal flight paths[END_REF]. In his works, Kelley set the basics for gradient back-propagation for complex systems optimization. This contribution is arguably less visible than the perceptron. It is however the very mechanism that made learning possible. The learning phase consists in assigning the right weights θ to the model. This learning phase, further explained in Sect. 4.2.4, will long remain the main obstacle of Deep Learning.

Towards Convolutions

The very first convolutional neural network (CNN) was developed by K. Fukushima in 1970 [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition[END_REF]. All moderns Deep Neural Networks (DNNs) are CNNs. In fact, the only difference is purely arbitrary. A DNN is simply a CNN that stacked enough layers to be deemed "Deep". Called Neocognitron, this CNN introduced convolutional layers as well as pooling layers. These two new additions are very important features of DNNs. Convolutional layers are specific layers of artificial neurons. A filter is a group of neurons, that has few weights and only considers a small neighborhood of the input, for example, 5 × 5. And a layer consists of multiple filters. The first convolutional layer might extract basic information such as straight lines and colors. The second layer will calculate a combination of these extracted features which yields slightly more complex information such as angles for example. The complexity of the patterns gets increasingly important. In the final layers of a modern DNN, features may represent a class itself. The second contribution of this work is the pooling layers. A specificity of convolutional layers is that they preserve spatial dimension. On a color image, the 3-dimensional input has a size h × w × c. Where h is the number of pixels in the height dimension, w in the width dimension and c the depth dimension (or channels, e.g.: Red, Green and Blue). After a convolutional layer, this input is transformed to another object of dimension h × w × n f , where n f is the number of filters in said layer. Pooling layers are ways of reducing the size of this object. Pooling is an operation done on a small neighborhood as well, usually 2×2, which effectively reduces the size by a factor 2 (Fig. 4.3). The operation can be max-pooling: the maximum value of the neighborhood is kept, the rest is tossed; or average-pooling: values are averaged over the neighborhood. Combined with convolutions, this operation leads to a necessity in image classification: translation equivariance. This means that an object will be recognized as such by the system regardless of its position on the image.

Once all the different elements were available Lecun et al. [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] built Lenet5 in 1998. It is the first CNN with a practical application: reading hand-written digits on postcodes. It is also the first CNN to outperform the competition on any specific task. The so-called LeNet5 was trained using back-propagation, which was the missing link to make the training procedure work. Its name stems from the 5 hidden layers. There are 3 convolutional layers (with 2 pooling layers after the first two), and 2 fully-connected layers.

Convolutional layers act as feature extractors to build semantic information for the fully-connected layers. The final fully-connected layer then outputs a score for each class (before activation) called logits. These logits are transformed to a probability mapping using a Softmax activation (Eq. (4.1)). Softmax had been investigated before on MLPs [START_REF] Bridle | Training stochastic model recognition algorithms as networks can lead to maximum mutual information estimation of parameters[END_REF].

Sof tmax(x) k = e x k C i=1 e x i (4.1)
LeNet5 is the result of almost a decade of research after Lecun had proposed the first conceptual LeNet in 1989. Its success was made possible by the progress in computer hardware. This is also the reason of the stagnation over the next 14 years. Computational power was simply not enough to make significant progress.

Alexnet: A Deep Learning Revolution

Although the works of Lecun et al. [START_REF] Lecun | Gradient-based learning applied to document recognition[END_REF] brought light on the possible success of CNNs, the real Deep Learning revolution came in 2012. Krizhevsky et al. released AlexNet [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] and caused a major paradigm shift in Computer Vision. Their prowess was to win ImageNet Large Scale Visual Recognition Challenge (ILSVRC), or simply ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF] challenge this year. It is a famous event within the CV community in which classifier models compete with each other in correctly predicting images on 1,000 different classes. Alexnet in terms of architecture is similar to LeNet5, but with 8 hidden layers this time. It can be called the first Deep Neural Network.

Saying that Krizhevsky et al. merely capitalized on increased computational power would be greatly undermining their work. Alexnet brought its load of much-needed innovations to the community. Most of which tackle issues that were unsolved until then. The first is a new activation function called the Rectified Linear Unit (ReLU):

ReLu(x) = max(0, x) (4.2)
Researchers had mostly experimented with two activation functions until then: sigmoid: f (x) = (1 + e -ax ) -1 , with a > 0 the slope factor) and the hyperbolic tangent: Another innovation is the introduction of Local Response Normalization (LRN). This step normalizes the output of convolutions layers. This prevents specific filters from having a response so large that it takes over the others. This heavily impairs training and keeps the DNN stuck in non-favorable local minima. This normalization remained in use for some years. In 2015, Ioffe et al. introduced Batch Normalization [69] (or BatchNorm) which is a data-specific normalization. It adds trainable parameters that normalize outputs of convolutions with regards to the training set. Although it is more effective and helped reach higher accuracy, the drawback of BatchNorm is that it is very dependent on the training data. If test data comes from a different distribution, its effectiveness can be compromised.

f (x) = tanh(ax).
AlexNet also popularized Dropout, developed by the same team [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF]. It is a regularization technique that randomly 'turns off' neurons during training phase. This prevents decision paths from being reinforced every step and taking over the other.

Finally, the generalization of AlexNet over such diverse test data was made possible through data augmentation. Simply put, data augmentation is a technique that modifies the input to enrich the training set. Modifications vary from random cropping, translation, rotation, and vertical or horizontal flipping of the images. It can be seen as another regularization technique that prevents overfitting. All these additions made Deep Learning emerge from a potential technology to the standard of Computer Vision. In the years that followed, the competition was won by only deeper and larger DNNs. Although the challenge was discontinued in 2017, Imagenet is still widely used as a dataset today. Accuracy on test set is still a good benchmark to rank vision models, and it is the largest labeled image dataset that can often be useful for pretraining. Models have nowadays become much larger, and keep increasing in terms of parameter count. But let us not downplay the progress that was made following Alexnet.

GoogleNet or Inception V1 [START_REF] Szegedy | Going Deeper with Convolutions[END_REF] in 2014 introduced many tricks and novel techniques. They use two outputs to their DNN: one intermediary and one final. The first output serves as a way to propagate further information through the gradient that ended up being too weak otherwise. They also created Inception layers that combine and concatenate simultaneous convolutions of different sizes. Among these sizes, the surprising 1 × 1 convolution kernels were introduced, serving as depth reduction.

In 2015 Simonyan et al. introduced VGG family of DNNs [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF]. These DNNs ranging from VGG-11 to VGG-19 were parameters-heavy at the time. Their smallest implementation used 133M parameters and their largest 144M. As a comparison, Alexnet 'only' had 62.4M and GoogleNet 6.4M. The architecture did not use many innovations. Their creators argue that their main contribution is the use of small 3 × 3 convolutions kernels. The real prowess remains that they successfully trained this many layers. It was a feature itself at the time.

Lastly, Resnets were released in 2018 [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF] and are another notable contribution. Their main contribution is using 'skip' connections within the convolutional layers to transmit the information from previous layers to the rest of the network. These connections are simply an identity function that ensures a complete transmission of the information.

Many other DNNs were developed over the years and continued performing better and better as is shown on Fig. 4.5. Among the most recent models, Attention networks made a noticeable impact [START_REF] Vaswani | Attention is all you need[END_REF]. Attention networks apply priority over certain regions of an image, deeming what part of it is more relevant to classification. Another important breakthrough that tends to become the standard is the Transformer network or Self-Attention network. Inspired by NLP model Bert [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], ViT [START_REF] Dosovitskiy | An image is worth 16x16 words: Transformers for image recognition at scale[END_REF] popularized the use of transformers for vision. Transformers tend to generalize better than other DNNs and avoid biases.

We will now see notations used throughout this manuscript to define models, images, and attacks.

Model and Data Notation

Let I ∼ D be a digital image in the domain 0, 255 n where n is the number of pixels and 0, 255 := {0, 1, . . . , 255}. This image is preprocessed before feeding a DNN. This stage is defined during the training phase of the DNN to improve its learning capability Figure 4.5 -Top-1 accuracy (i.e.: correctly predicted images) on DNNs since Alexnet. The interactive chart is available at https://paperswithcode.com/sota/image-classification-onimagenet over the data. After the DNN was successfully trained, it expects data preprocessed in the same fashion to make a prediction. This preprocessing is usually done in two steps: range reduction from [0, 255] to [0, 1], and normalization:

x := I/255 -µ data σ data , ( 4.3)
where µ data and σ data are respectively the mean and standard deviation computed over the training data. These constants are usually channel-specific (i.e. each channel has its own normalization), sometimes set to arbitrary values such as 0.5. We call x ∼ X the tensor image which lies in a pseudo-continuous ensemble. Its values are in the floating-point domain.

Classifier Model

Let f : [0, 255] n → [0, 1] C be a classifier mapping a tensor image x ∼ X to class logits l for C classes. l is a C-dimensional vector. Probability vector ŷ is obtained with a Softmax function: ŷ = Sof tmax( l) (Eq. (4.1)). The predicted class is defined as:

ĉ(x) := argmax k ŷk (x). (4.4)
The classifier makes a correct prediction if ĉ(x) = c(x), where c(x) denotes the groundtruth class of x. c(x) class is embedded in a one-hot ground-truth vector y, another Cdimensional vector. The c-th value is 1, rest is 0. A perfect classifier yields ŷ = y. But to achieve this, the classifier must first undergo a training phase. This is how it learns to make a good mapping of input image to correct classification.

Training a Classifier

Training a classifier is the very first step to having a functioning model. A model does not initially ensure ĉ(x) = c(x). It is first initialized with random weights under some form of uniform Gaussian distribution. The probability of making a correct prediction is entirely random: 1/C. As discussed previously in Sect. 4.2.1, the training phase is far from easy. We will see in Sect. 4.3 that this procedure is also at the heart of White-Box attacks. We will now have a quick overview of this procedure.

DNNs are complex systems with millions to billions of parameters nowadays. The problems that DNNs try to solve are non-convex, high-dimensional, and optimized over a large volume of data. No optimal parametrization can be computed in a single step. The solution comes from iterative approximation.

To achieve this, a particular algorithm is helpful: gradient descent (shown in Eq. (4.5)). This method, when it converges successfully, finds a good approximation of the solution by minimizing an cost function or loss function. Convergence is however far from trivial and almost certainly the algorithm ends in a local minimum. It has been observed that these local minima are however close to the theoretical global minimum. Some even argue that local minima are global minima [START_REF] Kawaguchi | Every local minimum value is the global minimum value of induced model in nonconvex machine learning[END_REF].

Formally, parameters θ of the model are updated at each iteration with the following rule:

θ t+1 ← θ t -η × ∂L ∂θ t (4.5)
Where θ t represent the set of parameters θ at iteration t. L is a loss function, usually categorical cross-entropy in the case of classification (Eq. (4.9)). And η is the learning rate. The gradient is calculated using the chain rule:

∂x ∂z = ∂x ∂y × ∂y ∂z (4.6)
The gradient is calculated from the output, we call this procedure back-propagation.

Let us consider a DNN that has l layers. Parameters ω h of a hidden layer h are thus updated as follows:

ω h t+1 ← ω h t -η × ∂ω h+1 t ∂ω h t × • • • × ∂ω l t ∂ω l-1 t × ∂L ∂ω l t (4.7)
Gradient descent updates the weights θ t layer per layer in order to minimize the loss. This is the basic gradient descent algorithm. The optimal set of parameters θ is thus:

θ = argmin θ E x∼X L(θ, x, y) (4.8)
Most training procedures nowadays use slight variations of gradient descent. Stochastic Gradient Descent (SGD) with Momentum [START_REF] Ning Qian | On the momentum term in gradient descent learning algorithms[END_REF] adds an aggregation of past gradients to control the descent. This reduces bias on a single step and quickens convergence. Root Mean Square Propagation (RMSProp) [START_REF] Hinton | Neural networks for machine learning lecture 6a overview of mini-batch gradient descent[END_REF] is another optimization algorithm that uses a mobile exponential average of gradients to regularize the update term. The most popular optimizer remains Adam [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] which simply is a combination of both SGD and RMSProp.

A loss function translates the error that a classifier makes on its predictions. In multilabel classification, the categorical cross-entropy is the most common. It is defined as follows:

L = - C k=1 y k • log ŷk (4.9)
This differentiable loss thus computes the error between prediction and ground-truth. Minimizing this loss effectively improves the accuracy of the DNN. The procedure is the following: the DNN is 'fed' a batch of images, then the loss (Eq. (4.9)) is computed over this batch, and finally, parameters of the DNN are updated using Eq. (4.5). A batch of images is used rather than a single image as another regularization method. This ensures better convergence. Single images generate too much bias over one step that takes time to be corrected. At the end of the training phase, we have ideally θ ≈ θ . This means that the classifier reached the highest accuracy possible.

These were the basics of classifiers and training. We will see how adversarial examples in a white-box setup are crafted using similar methods. In fact, attacks can be seen as a form of gradient descent performed over the image rather than the DNN.

Adversarial Examples

Adversarial Vulnerabilities of DNNs

Adversarial attacks were brought to light by the works of Szegedy et al. [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. The famous image (Fig. 4.6) of a panda turned into a gibbon through an invisible perturbation has been seen by many deep learning enthusiasts. This image is an adversarial example, and the perturbation can be called the adversarial signal. It is the result of an evasion attack. Although this signal may look like noise, it is carefully crafted. The mechanism behind it is the very same one used to train a model: gradient descent using back-propagation.

Adversarial examples can be a surprising result. Especially now that state-of-the-art DNNs perform better than a single human on a classification task. This has been observed in medical diagnosis [START_REF] Buetti-Dinh | Deep neural networks outperform human expert's capacity in characterizing bioleaching bacterial biofilm composition[END_REF][START_REF] Richens | Improving the accuracy of medical diagnosis with causal machine learning[END_REF] for instance. These works [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF] gave birth to a whole new field dedicated to adversarial examples. Works vary from theoretical studies of their very existence, to how to craft them in different setups, with better performance, or how to defend against them. We will now study the crafting procedure of these adversarial examples. We will see in Chap. 7 that this panda is however not an adversarial image.

Fooling a classifier

Evasion attacks are performed at evaluation stage. The DNN is trained, and ready to operate on its task and its weights are frozen. The attack leaves DNN in its exact same state, only the input is modified to affect the prediction. The optimization problem seen in Eq. (4.5) takes place on the image instead. But this time the goal is to degrade prediction, so it can rather be seen as a gradient ascent:

x a ← x 0 + η × ∂L ∂x 0 (4.10)
Where x a is the created adversarial image from an image x 0 we call natural. The gradient is calculated using back-propagation as seen in Eq. (4.7). Chain rule is propagated one further (Fig. 4.7). Note that the attacker has access to gradients only in a white-box setup (or a gray-box setup to some extent). Unless stated otherwise, we consider white-box attacks throughout this thesis.

A completed attack forges an adversarial sample x a such that:

ĉ(x a ) = c(x 0 ). (4.11)
The resulting class ĉ(x a ) is either chosen by the attacker in a targeted scenario or any class that verifies Eq. (4.11) in an untargeted scenario. An attack optimizes the perturbation x a -x o according to a given metric, usually the 0 , 1 , 2 or ∞ -norm. This gives the following optimization problem on a generic m -norm:

x a := min ĉ(xa) =c(x 0 ) ||x a -x 0 || m .
(4.12)

Throughout this thesis, we consider attacks in their 2 -norm form. This is common practice in image processing: The PSNR gives the logarithmic scale of the 2 -norm. Indeed, for natural images (i.e. ImageNet [START_REF] Deng | Imagenet: A large-scale hierarchical image database[END_REF], but not MNIST), 2 reflects distortion perceived by humans when comparing similar images with very low distortion. Researchers also often consider the case of ∞ -norm (maximum distortion over a pixel) or more rarely 0 -norm (number of modified pixels).

Equation (4.11) displays the paradigm of an untargeted scenario. In a targeted setup, the attacker picks the predicted label or adversarial class. The optimization performed on the image is this time a normal gradient descent:

x a ← x 0 -η × ∂L ∂x 0 (4.13) L = - C i=1 y t i • log ŷi (4.14)
Where y t i is a one-hot vector. The 1-value is on the targeted class t. The difference between targeted and untargeted attacks lies mostly in this + orsign. The optimization is a gradient descent just like during the training phase. The objective is to minimize the loss characterizing the prediction of the targeted class. The following works will now be presented in their untargeted version. These definitions and optimization problems apply to classification tasks. They can however easily be transferred to other DNN-based tasks. In Computer Vision, adversariality has been studied in various scenarii such as:

1. Object segmentation [2,[START_REF] Fischer | Adversarial examples for semantic image segmentation[END_REF][START_REF] He | Segmentations-leak: Membership inference attacks and defenses in semantic image segmentation[END_REF] 2. Object detection and tracking [START_REF] Liang | Efficient adversarial attacks for visual object tracking[END_REF][START_REF] Wei | Transferable adversarial attacks for image and video object detection[END_REF] 3. Video recognition models [START_REF] Jiang | Black-box adversarial attacks on video recognition models[END_REF][START_REF] Li | Adversarial perturbations against real-time video classification systems[END_REF] But adversariality is not limited to Computer Vision. This phenomenon is seen in numerous other application fields and tasks:

1. Audio (speech-to-text) [START_REF] Carlini | Audio adversarial examples: Targeted attacks on speech-to-text[END_REF][START_REF] Cisse | Houdini: Fooling deep structured prediction models[END_REF] 2. Text (sentiment analysis) [START_REF] Di | Is bert really robust? a strong baseline for natural language attack on text classification and entailment[END_REF][START_REF] Wang | Cat-gen: Improving robustness in nlp models via controlled adversarial text generation[END_REF] 3. Reinforcement Learning (prevent the agent from learning) [START_REF] Huang | Adversarial attacks on neural network policies[END_REF][START_REF] Lin | Tactics of adversarial attack on deep reinforcement learning agents[END_REF] 4. Anomaly detection [START_REF] Kloft | Online anomaly detection under adversarial impact[END_REF] This is just a mere overview of what is possible. Adversariality has been studied in most Deep Learning application, and literature is very rich in this domain. This thesis is however about classification in Computer Vision. The next sections will dive into the construction of adversarial examples in a white-box setup.

Usual White-box Attacks

White-box attacks have access to the entire model and parameters. This is convenient to use gradients, as seen in Eq. (4.10). We now define a slightly different loss that we call adversarial loss L adv . This will prove useful to craft perturbations:

L adv (x) := ŷc (x) -ŷa (x), (4.15)
Where ŷc (x) is the predicted probability of x belonging to the ground truth class c(x 0 ). The second term ŷa (x) is the probability of the adversarial class. Note that both terms can be replaced by logits lc (x) and la (x) for similar results. In a targeted scenario, class a = t is a parameter of the attack. In an untargeted scenario, it is the best prediction excluding the ground-truth of the original:

a = arg max k =c ŷk (x). (4.16)
Using this trick, we still consider the attack to be untargeted, but the adversarial class is guided. Attacks are however iterative processes and the adversarial class can change over time. This is not the case when the probability ŷa (x) is already high. It does change more frequently when ŷc (x) ≈ 1.

The optimization problem is thus similar to a targeted scenario as shown in Eq. (4.13). Loss needs to be decreased to ensure adversariality instead of increased. Moreover, it has an interesting property that justifies its use. When L adv (x) < 0, the predicted class is not the ground-truth label c and x is adversarial. It thus provides both a direction and a stopping criterion. We will now introduce notable attacks that we use throughout this thesis. Attacks are defined with this loss L adv .

Fast Gradient Sign Method FGSM

FGSM [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF] is the first and most basic attack. It uses the sign of the gradient on the original image:

x a = clip [0,255] (x 0 -× sign (∇L adv (x 0 ))) , (4.17)
where clip I is the clipping of the component within the interval I. This attack is not iterative, it is a single-step process relying on only one parameter . Note that the adversarial sample is a quantized image only for an integer value of .

Iterative Fast Gradient Sign Method iFGSM

iFGSM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] is an iterative version of FGSM.

x (i+1) a = clip [0,255] x (i) a -× sign ∇L adv (x (i) a ) . (4.18)
This attack uses two parameters: the descent rate previously seen in FGSM and the number of iterations N iter .

Projected Gradient Descent PGD

This attack is an iterative attack whose updates are defined as follows in its 2 -norm version PGD 2 [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF]:

x (i+1) a = clip [0,255] proj α x (i) a - ∇L adv (x (i) a ) ∇L adv (x (i) a ) . ( 4.19)
PGD 2 revolves around a projection proj α on the ball centered on x 0 of radius α. This projection is effective only if the 2 -norm of the perturbation exceeds α. PGD 2 also uses an 2 normalized gradient to have better control of the perturbation update. This attack uses 3 parameters: the radius α, the descent rate , and the number of iterations N iter .

Other versions of PGD can use different norms, which affects the projection step.

Boundary Projection BP

BP [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] is a fast two-step iterative attack. The first step quickly finds an adversarial sample while the second step refines it by reducing its distortion. Stage 1 is defined as follows:

x (i+1) a = clip [0,255] x (i) a -γ i ∇L adv (x (i) a )| ∇L adv (x (i) a ) , ( 4.20)
where γ i is an acceleration term ranging from a predetermined γ min at the first iteration to 1 at the final one. Stage 2 refines the adversarial sample found through projection using the same parameters and normalized gradient of L adv .

Carlini & Wagner CW

This attack minimizes the following Lagrangian formulation in its 2 -norm version CW 2 [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF]:

J(x a , µ) = ||x a -x 0 || 2 + µ|L adv (x a ) -m| + (4.21)
where |z| + = max(0, z) and m ≤ 0 is a margin. The minimum of this equation is found with ADAM optimizer within an inner loop. The outer loop does a line search over µ.

When both outer and inner loops are done, the adversarial sample with the least distortion is returned. This attack uses five parameters: the amount of iterations over both loops, the margin m, and the learning rate and momentum for ADAM.

Chapter Conclusion

This chapter allowed us to set the basics for the rest of this thesis. We first saw an overview of the history of DNNs and how each of the essential bricks was brought to the community over a long period of time. This was until the advent of Alexnet. Then, successive DNNs started to use the same bricks and improve on them. We briefly reviewed notable innovations and architecture principles. This allowed us to define a classifier model and study the important first phase of every classification project: the training phase.

We however exposed that this training phase, the strength of DNNs, can have an inconvenient side-effect. This is the existence of adversarial examples in a white-box setup. It is also worth mentioning that many Black-Box attacks exist as well. It is a whole different paradigm that is more constrained for the attacker. Attacks can be based on prediction scores [START_REF] Ma | Simulating unknown target models for query-efficient black-box attacks[END_REF][START_REF] Yang | Learning black-box attackers with transferable priors and query feedback[END_REF]; or only decision ([24, 86, 99, 118]). Some aim at estimating the gradient to reproduce similar mechanisms, while others do not.

In this thesis, we however mostly consider DNNs to be a white-box. The only exception is Chap. 6 which partly uses black-box attacks. We then studied in little more detail some popular white-box attacks. All these attacks are studied throughout this thesis. Many others exist and listing all of them would be tedious. This was not the purpose of this chapter. We will now dive into a less practical chapter and analyze the very existence of adversarial examples. Understanding how they exist also leads to defending from them. This will be the second part of the chapter.

Introduction

Within every security topic, two sides of the coin coexist. Attack and Defense. The field of adversarial examples is no exception to this rule. Security is played as an iterative game where each party makes a move one after another. The attacking side is however usually one step ahead. A flaw in the system is found. The Attacker sees the opportunity to exploit it. The defense then fixes the issue and hopefully focuses on extending the solution to other similar potential flaws. Being one step behind as the defender is obviously not desirable. This led companies over the world to hire ethical hackers (also known as white-hat hackers). Their purpose is simple: break a system to identify threats that will need fixing. The pattern we see here is simple: Identify a flaw and attack → Gain knowledge over the system → Defend and fix the issue → Identify further risks and start over.

Defense and understanding of a system are thus closely tied together. DNNs are still vastly considered to be black-boxes. We understand the operations done within but we do not know what is the purpose of each intermediate step. Even DNNs with the fewest parameters have millions of them.

Figure 5.1 -Illustration of the iterative game between attack and defense. The frontier between both sides is always blurry. There is an everlasting race to find issues.

Understanding Adversarial Examples

A classifier is usually built from two bricks: convolutional layers stacked together and a fully-connected layer. The purpose of the convolutional layers is to extract intermediary representations or features. The fully-connected layer creates a probability map given the existing features.

The first convolutional layers extract simple information such as contours, texture, and color. The following layers piece together these small bits to extract larger and more complex patterns. The final convolutional layer builds a feature map: a high-dimensional vector representing features on the whole picture.

In terms of explainability we can for example easily analyze the first layer of convolution and understand the patterns considered by the DNN. The first layer usually looks for straight or curved lines, and color presence. But as we get to deeper layers, it is harder and harder to understand the macro-patterns, made of small patterns pieced together.

The construction of feature maps is part of the black-box. A human for instance might identify a car through the presence of two or more wheels, a parallelepiped-like structure, a windshield, etc. The features extracted and used to make a prediction by the DNN are however unknown. And they vary greatly from one model to another. ResNet-18 has a 512-dimensional feature map while ResNet-50 has a 2048-dimensional feature map. This is true for any architecture, a bigger model (more parameters) usually uses a higherdimensional feature map. Features extracted between two networks simply cannot be the same. All the intermediary phases (convolutional operations) are thus unintelligible. We can understand how the whole system works, but we do not know the steps leading to the result.

The very existence of adversarial examples can be useful to understand and unfold the black-box. Understanding how the DNN is fooled gives insight into how it was correct in the first place. First, we need to identify the possible culprits. We denote three of them: Obviously, neither of these three is the sole responsible. We need to dive into each category and investigate their share of responsibility.

Data Is The Enemy

Data is commonly pointed at when it comes to adversarial robustness. How could a DNN be robust if its data is misleading? The strength and weakness of Deep Learning is the amount of data. This can be mitigated with data augmentation, but only to some extent. To train a good classifier, one needs an enormous amount of data. Table 5.1 gives an overview of the most popular datasets in image classification. Even for small classification problems such as MNIST or CIFAR-X, the amount of data is considerable. But as we tackle harder tasks, the amount becomes overwhelming, especially in the case of ImageNet. Most of our work throughout this thesis was done using ImageNet classifiers. We will now consider this dataset for the rest of this section.

Name

Image

Data labeling

It would be tedious work to check the label of every single image within Imagenet. It would take over 38 years for someone to cover the whole dataset if they were to inspect 1,000 images a day. So how exactly were these images annotated?

It was done through crowdsourcing with Amazon's Mechanical Turk (AMT) 1 . A bot first queries the internet to find images of a specific class. Humans then validate whether the queried class is indeed present in the image. The protocol seems pretty sound at first but there are many flaws with it.

Dogs ? Dogs. And some more

The choice of classes within the 1,000 existing is questionable. It seems obvious that reaching a thousand categories was a goal in itself. It is probable that some categories were added just to reach that number. We can for instance hope that people who participated in AMT were dog enthusiasts. We count no less than 118 dog breeds within classes. Many breeds are easily distinguishable from one another. Additionally, people surely looked up what each of them looked like prior to confirming their presence. But even so, this task is likely complicated. We extracted an example of 6 different classes, displayed on Fig. 5.2. This will allow you to test your own differentiation abilities on these breeds. This task is definitely not trivial.

Labeling error is indeed found throughout the even most popular datasets as per Northcutt et al. [START_REF] Curtis G Northcutt | Pervasive label errors in test sets destabilize machine learning benchmarks[END_REF]. They estimate that the validation set on Imagenet has at least 6% error. It is also suggested that more recent models overfit on these errors. Overfitting on validation or test sets is an indirect consequence of cross-validation. While the DNN is not trained directly on these sub-sets, it is evaluated on them. Thus the model that will be picked in the end is the one that has the best accuracy on validation sets. This might come at the cost of indirect overfitting.

An improved (e.g. more curated) dataset is released along with this work [START_REF] Curtis G Northcutt | Pervasive label errors in test sets destabilize machine learning benchmarks[END_REF], available at https://labelerrors.com/. This website also offers good visualization of the existing errors in the data. It is worth seeing for yourself if you were to work with this set.

Trained with this cleaner Imagenet, a Resnet-18 outperforms off-the-shelf Resnet-50 while on cleaner CIFAR-10, VGG-11 outperforms VGG-19. Resnet-50 should perform better (resp. VGG-19). They however generalized on wrong data which hinders prediction confidence or even classification on clean data. Low prediction confidence is a strong enemy. It means the class frontier could easily be crossed. The image lies close to it, and thus a small perturbation will suffice to fool said classifier.

Finally, some labels in ImageNet are plainly duplicates:

-638:'maillot' and 639:'maillot, tank suit' -134:'crane' and 517:'crane' -657:'missile' and 744:'projectile, missile' -620:'laptop, laptop computer' and 681:'notebook, notebook computer' -836:'sunglass' and 837:'sunglasses, dark glasses, shades' Images from both 657:'missile' and 744:'projectile, missile' could be from either class (as is illustrated on Fig. 5.3). The same goes for 620 and 681, 836 and 837, and 638 and 639 (not shown for decency reasons, but we invite you to see for yourself that both class contain the exact same type of images). In the case of 134:'crane' and 517:'crane' there is however a difference since the former designate birds and the latter building machines. But surely this ambiguity led to errors. Labeling error is not the only threat here. Feature maps will more likely be extremely correlated. This results in high class proximity within the feature space. Section 5.2.3 provides a further analysis of this phenomenon.

Many people participated in labeling these images. While the effort is admirable, we can still question the quality of such work. It is hard to find statistics about this whole work, data is unavailable. We can however hope that there was a lot of overlap on reviewed images. Having each image checked by multiple people would effectively reduce the error made. Still, labeling is a long and tedious task and the community is grateful to have access to such a large dataset. Labeling errors is the human factor. But it is not the only issue with ImageNet and image datasets in general. Images themselves can be misleading and poor lessons for the DNN to learn.

Data representation

An important work by Birodkar et al. [9] analyzes redundancy in datasets. They for instance estimate that ImageNet or CIFAR-10 contain at least 10% of semantic redundancy. Images within the same class overlap a lot in visual information which hinders learning and proper generalization. A basic example of redundancy is the DNN that learns to recognize the street to predict a car. Streets are not a 'car' per se, but appear to be closely related. If the classifier is then shown a car on a beach, it might struggle. This is an example of 'background' redundancy as is shown on Fig. Other issues are pointed at by Berer et al. [8]. The main complaint is the single annotation per image. Many images have multiple possible labels represented. Having a single label is misleading because the DNN might find the exact features of a secondary class but it is taught to ignore it. This in turn is detrimental to the learning procedure. Features of all present classes are mixed and the fully-connected layer adjusts its decision based on them. The harm caused is twofold:

-Secondary classes count for the prediction of the primary one. An image containing only an object from a secondary class will have less confidence on its correct label. The primary class might have a non-null and even non-negligible probability.

-Features from secondary classes are not discriminant for prediction. The construction of their features is tuned down. This results in poor learning of these off-classes.

These issues will especially be problematic in the case of strongly correlated classes. Figure 5.5 from their work illustrates some of these situations. In the case of 477:"carpenter's kit, tool kit", one can assume that 784:'screwdriver' will be present in many instances (Fig. 5.6 displays a few examples).

Figure 5.5 -Illustration from [8] showing images with multiple classes within. Red: original labels found in the dataset. Green: suggested improved labels if multiple annotations are considered. Playfully titled 'Are we done with ImageNet?' [8], their work questions the relevance of ImageNet nowadays. We just showed that there is indeed room for improvement. A lot of improvement. But it is still the best data there is for image classification. A DNN is likely to be trained on it to at least learn a diversity of features, before being fine-tuned on another specific task. This is especially convenient if this new task contains little data. Finally and mostly, it is still the ultimate benchmark in the community. Hopefully, it will be supplanted in a near future, but it is our best option at the time.

Learning the Hard Way

The way information is treated stems from the training phase that we originally defined. The DNN is taught to minimize an error, without clear instructions on how to achieve this. Training a DNN is entirely empirical and usually requires multiple attempts before having a satisfying result. Convergence of gradient descent is never ensured and can suffer from many phenomenons detrimental to the training procedure:

-Overshoot: learning rate too high, the gradient descent is unable to remain in a minimum and diverges. -Undershoot: learning rate too small, the loss function gets stuck in a local minimum. -Gradient explosion: the back-propagation of the gradient increases its value multiplicatively which makes the algorithm diverge. -Gradient vanishing: the back-propagation of the gradient decreases its value multiplicatively which stops the convergence.

Finally, a DNN in its converged state is inherently dependent on initialization. This first step is done in a well-thought-out fashion but remains random. So how do we know that two different DNNs (same architecture) that underwent the exact same training phase will end up in the same state? The answer is simple: we know that they do not and that they are not the same classifier. The landscape of a loss function is highly non-smooth. This is due to the high dimensionality of the problem tackled. Two DNNs might end up with the same loss scores on training and validation sets. But they will not be equal. This is one of the prowess of Deep Learning: the loss of a converged DNN finishes in a local minimum that is close in value to the global minimum. Yet we cannot identify this global minimum and can only assume that gradient descent made the loss close to it.

Generalization

The strength of DNNs lies in their ability to generalize. Images are complex data and extracting the right information is not an easy task. This led Computer Vision community to investigate invariance in their classification systems, well before DNNs. Classifiers are usually built from two bricks: feature extraction and category sorting. It so happens that DNNs have both within a single network even if both tasks are not done with the same layers (for instance convolutional layers for feature extraction and fully-connected for classification). Before DNNs, independent feature extractors were used for this first half of classification. Notably Scale-Invariant Feature Transform (SIFT) [START_REF] David G Lowe | Distinctive image features from scale-invariant keypoints[END_REF] that introduced, as its name suggests, scale invariance. Another Oriented FAST and Rotated BRIEF2 (ORB) [START_REF] Rublee | ORB: An efficient alternative to SIFT or SURF[END_REF] introduced rotation robustness. Complete invariance is equivalent to a perfect conceptualization of an object. Formally, let us note x ∈ R n a class representation, f (x) : R n → R n f a feature extractor and g : R n → R n an input transformation:

f (x) = f (g(x)) (5.1)
In a perfect extractor, this is true for any transformation or composition of transformations. A transformation is a modification of the representation of a given object. This includes translation, rotation, color change, scale, viewpoint, contrast, saturation, etc. No extractor is perfect to the point of reaching perfect equality between both terms of Eq. (5.1).

Figure 5.7 -Illustration of a perfect feature extractor. Every feature vector is equal, regardless of the transformation applied to the image. The object is perfectly conceptualized.

The role of convolutional and pooling layers is to bring scale and translation invariance to the DNN. Other invariances are however acquired during the learning stage. These are taught through the sheer amount of data as well as data augmentation. The amount of data required by DNN is mostly to improve generalization and push the DNN away from training set biases. But how much generalization is too much generalization?

Excessive Invariance

The work of Jacobsen [START_REF] Jacobsen | Excessive invariance causes adversarial vulnerability[END_REF] points this excess in state-of-the-art models. DNNs develop invariances to a lot of contextual information that should not be ignored. They propose an invariance-based attack akin to adversarial attacks. Their attack however adds perturbation to an image such that its logits matches the logits of another image. Images shown on Fig. 5.8 display the surprising inability of a DNN to differentiate entirely different images (more examples are available in the full article [START_REF] Jacobsen | Excessive invariance causes adversarial vulnerability[END_REF]). They study this phenomenon through two different tools. The first one is a toy dataset called adversarial spheres [START_REF] Gilmer | Adversarial Spheres[END_REF] It is a simple classification problem in which both categories are two concentric high-dimensional spheres of different radii. This dataset was introduced to study the detrimental effect of high dimensions inherent to image data. This translates into high data sparsity and many degrees of freedom for the attack to be performed. This is a prime example of curse of dimensionality [6] The second one is an iRevNet [START_REF] Jacobsen | i-revnet: Deep invertible networks[END_REF] model. It is a DNN invertible up until the last projection onto class vectors. By removing this last projection, convert it to a fully-invertible network called fi-RevNet. This network has two outputs z s for classification (semantic variables) and z n (nuisance variables). Its classification performances on ImageNet are only slightly lower than the competition. It is thus a decent classifier.

It is demonstrated that a perfect classifier should maximize the conditional mutual information I(y; z s |z n ) while decreasing the mutual information I(y; z n ) [START_REF] Jacobsen | Excessive invariance causes adversarial vulnerability[END_REF]. The identified culprit is cross-entropy loss. A training phase using this cost function will only increase I(y; z s ). This hinders the capacity of DNNs to understand the presence of possible nui-sance that could be linked to a given class. This can be read in echo with the presence of redundancies studied above in Sect. 5.2.1.

Adversarial Spheres [START_REF] Gilmer | Adversarial Spheres[END_REF] were also used in other works to blame BatchNorm [START_REF] Galloway | Batch normalization is a cause of adversarial vulnerability[END_REF]. We briefly mentioned BatchNorm [START_REF] Ioffe | Batch normalization: Accelerating deep network training by reducing internal covariate shift[END_REF] in Sect. 4.2.1 as an improvement to LRN. It is a normalization, applied batch-wise onto data after going through a layer. BatchNorm operate a 0-mean and unit standard-deviation as well as the following operation:

x BN ← γ * x N orm + β.
Where x BN is the input after BatchNorm and x N orm after normalization. Both gamma and beta are parameters that were however learned during the training phase. Adversarial images are not part of the same distribution as natural images. It is easy to see how this could backfire, and [START_REF] Galloway | Batch normalization is a cause of adversarial vulnerability[END_REF] provides experimental observation of it.

We have so far identified many possible culprits for adversarial vulnerabilities. Among them: data labels, data itself, data dimension, cross-entropy loss, BatchNorm, etc. It seems that there is room for improvement. However, any classifier, even a human, will misclassify an image at a certain level of distortion. It is time to look into this capacity in DNNs. The works of Cubuk et al. in 2017 titled "Intriguing Properties of Adversarial Examples" [START_REF] Ekin | Intriguing properties of adversarial examples[END_REF] as a nod to Szegedi et al. [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] state in their abstract: "Here we argue that the origin of adversarial examples is primarily due to an inherent uncertainty that neural networks have about their predictions. We show that the functional form of this uncertainty is independent of architecture, dataset, and training protocol; and depends only on the statistics of the logit differences of the network, which do not change significantly during training."

Adversarial Examples Are Inevitable

It is a strong statement about adversarial vulnerability. In the previous section, we identified bricks of DNNs that increased an existing flaw in the system. But this work [START_REF] Ekin | Intriguing properties of adversarial examples[END_REF] offers a pessimistic view of the problem. They argue that the success of an attack is affected by two elements. The first one is the input-logit Jacobian of the model and the logits themselves. In the case of a semi-random attack, this had previously been quantized by [START_REF] Fawzi | Robustness of classifiers: from adversarial to random noise[END_REF]. There is a clear dependency on dimensions and distances in the feature space. This subspace is very informative to understand adversariality. It is convenient to have a look at the construction of logits to gain further knowledge on the problem.

Class Projection and Logits

Logits are the scores calculated for each class considered by a model. This is before going through a SoftMax activation that turns logits into class probabilities. So what are these scores exactly?

Let us consider a DNN f (x) built from two bricks: convolutional layers and a fullyconnected layer. This DNN has l layers. We call f i (x) the output of the i-th layer whose weights are Ω i . An input image x is transformed into a feature vector f l-1 (x) after convolutions. The weights of the fully-connected layer are such that:

Ω l =         ω 1 l ω 2 l . . . ω C l         (5.2)
Logits are obtained through matrix multiplication: Ω l × f l-1 (x). The logit score of class k is thus a scalar projection of f l-1 (x) onto ω k l . For clarity we note these vectors: f (x), resp. ω k for the rest of this section. Seeing logits as projection eases geometric interpretation (Fig. 5.9). A scalar product is dependent on two values: norms of both vector and the cosine of the angle between them. Class logit scores are thus determined by two values ω k and cos( ω k , f (x)). Table 5.2 tells us that class vectors do not differ by a lot in norm. The most discriminant value is thus the angle cos( ω k , f (x)). We discussed earlier in Sect. 5.2.1 the problem of class proximity. Frontiers between two classes k and l is highly dependent on cos(ω k , ω l ). If this value is small, confidence can be affected by this phenomenon. A high logit score on either class yields a somewhat high value on the other. And works of Cubuk et al. [START_REF] Ekin | Intriguing properties of adversarial examples[END_REF] tell us that close logit scores are linked to high vulnerability.

It is interesting to note

that f (x) ∈ R n f + but ω k ∈ R n f , ∀k ∈ {0, 1, ...,
We can calculate cos(ω k , ω l ) easily through the definition of scalar product:

cos(ω k , ω l ) = n f i=0 ω i k ω i l ω k ω l (5.3)
Let us note Cω the symmetric matrix in R C×C such that Cω k,l = cos(ω k , ω l ). gives coefficients for a random set of consecutive labels. These are not a priori semantically related to dog breeds. The difference is flagrant. The correlation between dog breeds is sometimes really important. In the untargeted paradigm, these classes are especially vulnerable.

Then a solution appears: train class vectors to be uncorrelated to each other. But this would affect accuracy. Insisting once more on dog breeds: classes vectors from these classes need to be correlated. It appears that a trade-off exists between robustness and accuracy. It is the (in)famous No Free Lunch Theorem.

No Free Lunch Theorem

'No Free Lunch' is a phrase often seen in adversarial-related works. It states that additional robustness on a given trained system comes necessarily at the cost of accuracy. The works of Tsipras et al. [START_REF] Tsipras | Robustness May Be at Odds with Accuracy[END_REF] discussed this theorem. Through experimentation, they evidence this equilibrium. It persists even in simple settings, or a theoretical case of infinite data [START_REF] Tsipras | Robustness May Be at Odds with Accuracy[END_REF]. This even holds true on adversarially robust models (see Sect. 5.3.4). In short, DNNs are doomed to be vulnerable to adversariality. It is an indirect consequence of their efficiency.

Using toy examples, Zhang et al. [START_REF] Zhang | Theoretically principled trade-off between robustness and accuracy[END_REF] push the analysis further. They propose a theoretical model of the trade-off. Robustness is basically defined by two terms: natural error and boundary error. Natural error is a classification error on untouched images. Boundary error is a measure of the margin of confidence around natural images. But then again, further works such as [START_REF] Raghunathan | Understanding and mitigating the tradeoff between robustness and accuracy[END_REF] exploit this definition of the trade-off to increase it. Once an issue is understood, we can improve on it. The future of robust DNNs is thus not hopeless. We investigated in this section the reasons for adversarial vulnerability. There is room for improvement on some aspects of it. Data, in the case of Imagenet for instance, seems to have a lot of room for improvement. But literature is rich in novel ideas to defend DNNs. We will now study existing defenses.

Defending Against Evasion

Adversarial vulnerabilities necessarily exist. At least to some extent. This does not mean we cannot defend from them. In this final section of the chapter, we will dive into existing defenses. We can sort defenses into two distinct categories:

1. Reactive defenses: the defender acknowledges the vulnerability of a DNN. It does not try to interfere with the attack. Focus is set instead on the corruption of data. The goal is to either detect image manipulation or to remove (reform) the adversarial signal.

2. Proactive defenses: the defender tries to patch the vulnerability and/or make it more difficult to attack their system. More distortion is required to build an adversarial sample.

Both strategies are the source of numerous works. We do not try to benchmark the following defenses. The purpose of this section is to give insight into each strategy and sub-strategies.

Reactive Defenses

Two strategies exist to counter an adversarial example: detection or reformation. Attack-wise, the arms race is about reducing the distortion of the perturbation. Papers on either strategy thus assume low-distortion samples. When it comes to data reforming, this eases the job. But this might make the detection job harder to accomplish. We will now see the different approaches that have been considered by researchers.

Data Reforming

We explained in Chap. 4 that white-box attacks rely mostly on back-propagating the gradient. In a classifier, convolutional filters all overlap on the image. This generates interference in the gradient computed on the image. Moreover, attacks are usually run in a multiple-step process. This implies even local stability (image-wise) of the adversarial signal. Therefore perturbation usually lies in the high-frequency domain. This is a crucial observation from which reforming stems. It is easier to remove a noisy signal than a lowfrequency one. Within the iterative attack-defense security game, this inspired research on low-frequency attacks [START_REF] Guo | Low frequency adversarial perturbation[END_REF].

To remove high-frequencies, the most common approach is a form of compression. This technique especially aims at reducing local variance to decrease overall information. The works of Guo at al. [START_REF] Guo | Countering adversarial images using input transformations[END_REF] in particular use four different methods (Fig. 5.10). The first one is bit-depth reduction. Natural 256-level 8-bit images ({0, 1, ..., 255}) are converted to images with as low as 8-level 3-bit images. The loss of top-1 clean accuracy on Imagenet amounts to ≈ 8%. Considering the amount of compression, this is pretty low. DNNs are resilient to compression. The second method is JPEG compression. They use a quality factor of 75 for their experiments. This does not amount to a lot of compression and clean accuracy is almost unaffected. The third method is total-variation (TV) minimization. TV measures the norm ( 2 ) of the difference between pixels and their neighborhood. Minimizing TV builds an image locally smooth. The resulting compression is strong, which results in a loss of ≈ 15% of clean accuracy on Imagenet. Finally, they use image quilting as a means of compression. The image is split into patches along a predefined grid. Each patch is replaced by its closest counterpart within a database. This result in a tremendous impact on clean accuracy: a loss of ≈ 40% A greater gain in robustness is at the cost of worse clean accuracy. Except for bit-depth reduction, which gains in robustness do not compensate for the loss of accuracy. Most works in adversarial reforming use derivatives of these ideas. It is however difficult to make a fair comparison between different methods. Different norms are used to craft adversarial samples ( 2 , ∞ or less common 0 ). Different data is used as well. Interesting work from Xie et al. [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF] uses a subset of 5,000 images from the validation set. But clean accuracy on this data is 100%. This means that data was cherry-picked. This is problematic for a fair evaluation. Their idea is to perform random cropping and padding within the classifier. But without a unified evaluation, it is impossible to draw conclusions. Some works however differ from the others. A defense called PixelDefend [START_REF] Song | Pixeldefend: Leveraging generative models to understand and defend against adversarial examples[END_REF] aims at projecting data back onto a known manifold (training data). They justifiably argue that adversarial examples lie in low-probability regions of feature maps. This idea is seen in other works such as Samangouei et al. [START_REF] Samangouei | Defense-gan: Protecting classifiers against adversarial attacks using generative models[END_REF] that uses a Generative Adversarial Network (GAN [START_REF] Goodfellow | Generative adversarial networks[END_REF]) to do so.

In general, reforming data comes at a cost: natural accuracy. Unsurprisingly, these methods drag us back to No Free Lunch (Sect. 5.2.3). Hopefully, the impact of such methods is greater on manipulated data. But it will still alter clean images, which in turn decreases accuracy. Reforming data can be favorable, especially with very low distortion, but is to be used cautiously.

Adversarial Detection

Detection is a very special case of defense. It does not respond to a form of trade-off between accuracy and robustness. Neither is directly affected by detection. Accuracy will however indirectly be affected by false positives (image is not classified). An important aspect of this method is that it requires a detectable signal. If a network has very low robustness, adversarial images are crafted with very little perturbation. They become indistinguishable from natural images. Thankfully a detector is far more effective than the human eye. But a detector might require a somewhat robust network to work for. The problem of No Free Lunch is just displaced to another step of the whole process.

One of the cornerstones of detection is Feature Squeezing [START_REF] Xu | Feature squeezing: Detecting adversarial examples in deep neural networks[END_REF]. It is heavily inspired by works on reformation. It was first developed using bit-depth reduction and spatial smoothing. But any method seen previously can be used. The idea is to use a reformation method or squeezer on an image. Logits outputs of clean and reformed data are compared. If the difference is above a learned threshold, the input image is deemed adversarial.

Multiple squeezers can be used in parallel to increase confidence. Method is illustrated on Fig. 5.11. Feature Squeezing works under an important assumption: the adversarial Figure 5.11 -Illustration of Feature Squeezing from [START_REF] Xu | Feature squeezing: Detecting adversarial examples in deep neural networks[END_REF]. Two squeezers are used here, but more could be added.

image is more affected by the squeezer than a natural image. Reformation works under the same assumption. Feature Squeezing has a big advantage over its competitors as it is indirectly improved as a new squeezer is available. This detector, as well as others, comes at the cost of training time. Most other works build and train DNNs. In the paper of Metzen et al [START_REF] Hendrik | On detecting adversarial perturbations[END_REF], sub-DNNs are integrated into the original classifier. Each convolutional layer has a secondary output that goes through a sub-classifier. The spatial size of feature maps decreases along the DNN. Each sub-classifier is thus built differently. The idea behind is that the distributions of adversarial images and natural images are vastly unaligned. Resulting feature maps thus lie in improbable spots of feature spaces. This idea is seen throughout many works. Ma et al. [START_REF] Ma | Characterizing adversarial subspaces using local intrinsic dimensionality[END_REF] for instance train classifiers with natural, noisy, and adversarial images. Their argument is that noise does not bring a natural image far out of distribution. But noise counters the adversarial signal to some extent. It therefore drags adversarial images back to the distribution of natural images. Learning how either will react to added noise is key to detection.

We mentioned earlier that detection does not respond to the No Free Lunch theorem. However such methods will fail if a DNN allows very low-distortion adversarial examples to exist. Proactive defenses can be helpful either on their own or combined with detection. We will now study these defenses

Proactive Defenses

What is striking at first is how adversarial samples look unaltered even to an expert eye. This is due to the signal being diluted over a whole image, and also to the great vulnerability of DNNs. But this vulnerability can be decreased to some extent. This is the goal of proactive defenses. We identified earlier some reasons in Sect. 5.2. Proactive defenses have indeed worked at fixing most of these possible issues. This starts with data.

Enhancing Data

It has been observed times and times that DNNs do achieve human-level performances on natural images. DNNs could even surpass panels of experts. This can be seen in medical diagnosis [START_REF] Buetti-Dinh | Deep neural networks outperform human expert's capacity in characterizing bioleaching bacterial biofilm composition[END_REF][START_REF] Richens | Improving the accuracy of medical diagnosis with causal machine learning[END_REF] for instance. We can assume that there is nothing inherently wrong with the training procedure. DNNs do learn what they are taught. However humans may not be the best teachers. We saw previously (Sect. 5.2.1) that data could sometimes be improved. But we may also not be able to teach the classifier to learn the right elements. As per [START_REF] Ilyas | Adversarial examples are not bugs, they are features[END_REF], DNNs learn features that are not robust. For instance, texture or colors can be somewhat useful, but mostly non-informative. Contours and shapes seem better strategies. The authors build two different datasets. The first one is obtained by removing nonrobust features. The resulting images may come as surprising as the pictured class seems less visible (illustration (a) of Fig. 5.12). Models that learn from this data both end up having a good clean accuracy and robustness. More surprisingly, they build a dataset of visually mislabeled data (illustration (b) of Fig. 5.12). These images are adversarial examples on a teacher DNN. The student DNN ends up having poor robustness but good clean accuracy. This is really telling about the weak features that DNNs learn. Forcing a DNN to learn strong features (first set of data) gives it good robustness.

The other side of data are the labels. It is argued by some works that a one-hot encoding of the pictured class is not a good lesson for the classifier. We saw for instance the case of 477:"carpenter's kit, tool kit" and 784:'screwdriver' (Fig. 5.6). A solution some have come up with is label distillation. The idea is to use class probabilities that another classifier (called teacher) outputs. The idea dates back to 2015 [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF] and was initially introduced for dimensionality reduction. The improvement brought to this method by Papernot et al. [START_REF] Papernot | Distillation as a defense to adversarial perturbations against deep neural networks[END_REF] is to use the same architecture for both student and teacher. In an iterative fashion, the student can become the teacher of a new student, and so on.

These works increased robustness by improving the quality of the training data. But we saw earlier that data was not the only one responsible for adversarial vulnerability and we can have a look at how to modify the building bricks of DNNs to improve their robustness.

Model Tweaking

We discussed at length the problematic of class proximity in the feature subspace in Sect. 5.2.3. Some works aim at changing the way classification is performed. Instead of relying on scalar projection (Sect. 5.2.3), the works of Mustafa et al. [START_REF] Mustafa | Adversarial defense by restricting the hidden space of deep neural networks[END_REF] instead rely on building class polytopes. Each polytope is trained to be maximally separated from other polytopes. This decreases the correlation between classes, which in turn increases robustness. Their training objective includes two terms: one to minimize the distance from a feature vector to its class centroid, and the other to push centroids apart. Figure 5.13 displays a projection of different images in the feature space. We notice the direction vectors for classical correlation Softmax training and the diposition of centroids in their method. Experiments are run on MNIST and CIFAR-10. These are easy classification tasks. When it comes to ImageNet, more correlation between classes exists semantically. We mentioned in Sect. 5.2.3 that correlation between vectors may be necessary for correct classification. To insist once more on dog breeds: they do share many characteristics and could not be entirely separated in well-trained feature space. Trying to reproduce such results on Imagenet may not be possible.

The idea is interesting and the visualization is really telling of the classical class proximity paradigm. Especially when feature vectors have small norms, an adversarial example might be at a close distance (see the left side of Fig. 5.13). It is however hard to tell whether this could work on a more complex Imagenet. Other works have studied modifying models for robustness purposes. For example, Dhillon et al. [START_REF] Guneet S Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] who suggest a random pruning of activations (preferably lower magnitude ones) at inference. A good aspect of their method is that it does not require additional training and can be performed easily. But once again they do not experiment on Imagenet. We did try to perform pruning on Imagenet models to obtain absolutely no gain until accuracy degradation. Our experimental work was brief, and we do not dismiss this work as potential good practice.

Works that attempt at modifying models to earn robustness are not necessarily the most promising. In terms of proactive defenses, the next section is by far the most active.

Adversarial Training

The very idea of adversarial training (or retraining) came early with the works of Goodfellow et al. in 2015 [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF]. The idea is simple although it exists in different forms. The main idea is to train a model using adversarial examples in the training set. Then the optimization problem lies in minimizing a normal classification loss:

θ = argmin θ E x∼X max ||xa-x||≤ L(θ, x a , y) (5.4)
This optimization is done over a constrained attack. Under a given norm, ||x a -x|| can not exceed . This is akin to a classical training procedure as previously seen. The other optimization is a little different:

θ = argmax θ E x∼X min c =ĉ(xa) ||x a -x|| (5.5)
Here the goal is to maximize the margin between classes frontier. The lowest distortion (unconstrained) adversarial example x a is pushed further away from x. Of course, both of these optimizations are done in parallel with regular training. This ensures both correct classification and good robustness. In practice, only the first formulation is used for adversarial training.

Ideally, adversarial training is achieved with knowledge of the threat model. Knowing how x a is generated ensures better robustness against the attack. The first adversarial training suggested in [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF] used for instance FGSM (Sect. 4.3.3). In general, adversarial training with gradient-based attacks transfers well to other attacks. It can be however preferable to use iterative attacks. This is what was done by Madry et al. [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF] in which they both introduced and trained against PGD (Sect. 4.3.3). While their robustness is unmatched, it comes at the cost of a great loss of accuracy. Obviously, adversarial training falls right into the No Free Lunch.

Another important work by Kannal et al. [START_REF] Harini Kannan | Adversarial logit pairing[END_REF] brought many novel ideas to the field. First, they minimize the distance between logits rather than reducing classification error. The optimization problem is close to the first one (Eq. (5.4)):

θ = argmin θ E x∼X max ||xa-x||≤ || l(x a ) -l(x)|| (5.6)
This is Adversarial Logits Pairing (ALP). The adversarial example is constrained and built using a PGD attack. The norm reduced is 2 but they argue that optimization would work for other norms. Other ideas in this work are interesting. This first one is Clean Logits Pairing (CLP) in which they match logits of different images of the same class together. They empirically show that this increased model robustness. The best results were obtained with added Gaussian noise for regularization. And the final interesting result they show is Logits Squeezing in which they simply reduce the norm of logits. They obtain increased robustness as well from this experiment. This last result can be a little surprising when considering the previous illustration of feature space (Fig. 5.13). It seems preferable to have large norms to increase the distance between classes. Their experiments however show otherwise.

Plenty of other works exist in this model robustification, but most derive from these. Adversarial training (or sometimes retraining) is probably one of the hottest topics defensewise. Protecting a model from this vulnerability is preferably done upfront. This works also display really good results. Some adversarialy trained models can be attacked only through visible perturbations. To a human that is.

Chapter Conclusion

The efficiency of defenses is sometimes questioned. Some defenses rely on preventing correct gradient calculation, often through randomization. This kind of defense is largely proven ineffective by the works of Athalye et al. [3]. They developed the Backward-Pass Differentiable Approximation (BPDA) that can be used in many scenarii to approximate gradient. The attack is then performed as if the defense was absent and the results are convincing. Other critics [START_REF] Rice | Overfitting in adversarially robust deep learning[END_REF] claim that more recent works in robustification are simply mitigating overfitting in adversarial training. Finally, some argue that robustification is performed only on small distortion increments [START_REF] Papernot | Practical black-box attacks against machine learning[END_REF]. This regime would not be effective against larger increments and would bear no added robustness against black-box attacks.

Still, each form of defense is interesting differently. A combination of both proactive and reactive defenses might be desirable. The synergy between robustness and detection is strong for instance. High-distortion examples are however not desirable for reformation. These techniques are more effective against small signals. But their advantage over others is that they can usually be implemented without further training. If time is a constraint to deploy a system, this situation can be favorable. Otherwise, the first solution might yield the best results.

Finally, No Free Lunch Theorem may not be so rigid. Works have shown that the trade-off can be shifted towards one or another quantity. Notably, Xie et al. improved the accuracy of EfficientNet [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF] as well as its robustness. Either quantity is however only mildly improved. Their work is inspired by adversarial training, they introduce AdvProp as a form of adversarial crafting. Overfitting is mitigated and No Free Lunch is defeated.

Introduction

A deluge of research papers now propose defenses to block an attacker, and adaptive attacks against these defenses. This is an endless arms race, and systematic benchmarks to evaluate the state of the threat are greatly required.

It is currently extremely difficult to have a clear view on what is truly working in this domain. The cliché is that no two papers report the same statistics for the same attack against the same model over the same image set. This is mostly due to that an attack is an algorithm with many parameters. Its power is indeed highly dependent of these parameters. These values are rarely specified in research papers.

There exist benchmarks in the litterature, such as ARES [START_REF] Dong | Benchmarking Adversarial Robustness on Image Classification[END_REF], RobustBench [START_REF] Croce | RobustBench: a standardized adversarial robustness benchmark[END_REF], RobustVision [START_REF] Lab | Robust Vision Benchmark[END_REF],

ADBD [START_REF] Chen | RayS: A Ray Searching Method for Hard-Label Adversarial Attack[END_REF]. They aim at providing a better understanding of the robustness of image classifiers. Yet, they fall short because their slowness prevents them from tackling large image dataset like ImageNet. They only operate on CIFAR-10 or MNIST. Also, they resort to attacks which are not all state-of-the-art. This chapter proposes RoBIC, to consider these concerns and develop a benchmark tool to measure the robustness of image classifiers in a modern setup.

Difficulties

This section explains the difficulties for setting up a benchmark measuring the robustness of image classifiers.

Notation

An attack is a process forging an image I a = A(I 0 , M, Π), where I o is the original image, M is the target model, and Π is a set of attack parameters. The ground truth label of I 0 is denoted by y 0 . The boolean function 1(I a , y 0 ) = [M (I a ) = y 0 ] tells whether the attack deludes classifier M in the untargeted attack scenario: the prediction M (I a ) is not the ground truth. The distortion between I 0 and I a is denoted by d(I a , I 0 ). Some statistics like the probability of success and the average distortion are extracted from the adversarial images forged from the test set. They depend on the attack A and its set of parameters Π. Therefore, it can not play the role of a measure of robustness of a given model. The first difficulty is to get rid off the impact of parameters Π.

The best effort mode

The parameters Π have a huge impact on the power of an attack. For instance, some attacks like FGSM [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF], I-FGSM [START_REF] Kurakin | Adversarial examples in the physical world[END_REF], PGD [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] are distortion constrained in the sense that Π is strongly connected to a distortion budget. If this budget is small, the probability of the success of the attack is small. If it is large, this probability is close to 1 but the distortion is too big. Hence, it is hard to find the best setting to make these attacks competitive. Our strategy, so-called 'best effort mode', reveals the intrinsic power of an attack by finding the best setting for any image:

I a = A(I 0 , M, Π ) with Π = arg min Π:1(A(I 0 ,M,Π),y 0 )=1 d(A(I 0 , M, Π), I 0 ). (6.1)
The best effort mode makes the measurement of the robustness independent from an arbitrary global setting Π. Yet, it is costly in terms of computations. Attacks with few parameters are preferred since the search space is smaller.

Worst case attacks

A second difficulty is to make the robustness score independent of the attack. Ideally, we would like to know the worst case attack to certify the robustness of a model. An option proposed by benchmarks RobustVision [START_REF] Lab | Robust Vision Benchmark[END_REF] and ARES [START_REF] Dong | Benchmarking Adversarial Robustness on Image Classification[END_REF] is to consider a set of J = 11 attacks as outlined in Tab. 6.1. This is again costly as each image of the test set has to be attacked J times. Yet, a benchmark happens to be useful if it is fast enough so to assess the robustness of many models. The best effort mode over an ensemble of attacks is out of reach. This is the reason why we need to focus on fast worst case attacks in the sense that they achieve their best effort mode within limited complexity. Section 6.4 focuses on these attacks.

Metrics

In this thesis, we present results in two forms: graphs or table. We call operating curve the graph showing the success-rate of an attack (y-axis) at a given 2 -distortion (x-axis) such as in Fig. 6.1. We believe this curves to represent fair evaluation of optimized attacks. Distortion is measured in the pixel domain 0, 255 as where N is the number of pixels. This is easily interpretable: If for any pixel i, x a,i = x o,i ± (as in FGSM) then d(x a , x o ) = . The operating curve sums up the impact of an attack against a classifier over a set of test images S test by the following function:

d(x a , x o ) := x a -x o 2 / √ n, ( 6
d → P (d) = |{x o ∈ S test : d(x a , x o ) ≤ d}| |S test | . ( 6.3)
Note that P (0) = 1 -η > 0, where η is the accuracy of the classifier. This is the fraction of original images which are misclassified, hence considered as already adversarial.

Figure 6.1 shows operating curves of several attacks against two well-known classifiers. We choose a complexity budget allowing an attack to perform at its best capacity under the best effort mode: FGSM runs on N iter = 30 iterations, BP on N iter = 50, IFGSM, PGD 2 on N iter = 10 × 10, and CW 2 on 10 × 100 iterations (i.e. outer loop × inner loop). We formulate three remarks:

-CW 2 requires even more iterations to successfully attack all images against ResNet-50. -BP achieves by far the best trade-off between complexity, distortion, and success rate. -These attacks yield adversarial images with unquantized pixel values, and the average distortion is lower than 0.5 for most images. Rounding these values to 0, 255 erases the adversarial perturbation in most pixel positions so that the result is no longer an adversarial image [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF]. Chapter 7 studies this thoroughly.

The benchmark

This section justifies the recommendations made in our benchmark and defines the measure of robustness.

Pixel domain. Our benchmark is dedicated to image classification. As a consequence, the distortion is defined on the pixel domain: An image I is defined in the space 0, 255 n with n = 3RC pixels for 3 color channels, R rows and L columns. Most papers in the field measure distortion after the transformation of the image in a tensor x ∈ X n . This is a mistake preventing a fair comparison: for most models X = [0, 1], but for some others

X = [-1, 1] or X = [-3, 3].
We outline that an adversarial image is above all an image, i.e. a discrete object I a ∈ 0, 255 n . Again, most attacks output a continuous tensor x a ∈ X n , neglecting the quantization. This is a mistake: in real-life, the attacker has no access to x a , which is an auxiliary data internal of the model.

Measure of robustness.

Let us define the accuracy function η(d) := 1 -P (d). The value η(0) is the classical accuracy of the model over original images. Function η(d) is by construction non increasing and should converge to 0 as the distortion d increases. After observing many accuracy functions η for different models and attacks, we notice that they share the same prototype:

η(d) = η(0) e -λd with λ ∈ R + . (6.4)
Like in nuclear physics, we define the half-distortion d1 /2 as the distortion needed to reduce to half the initial accuracy:

η(d1 /2 ) = η(0)/2, d1 /2 = λ -1 log(2). (6.5)
This approximation is verified experimentally with an average coefficient of determination R 2 of 99%. The half-distortion d1 /2 will be the keystone of the proposed metric of robustness. A model is then characterized by three separated concepts: its generalization ability η(0) and its robustnesses d1 /2 against black-box and white-box attacks.

Fast Attacks

The recent trend in adversarial examples is to design fast attacks with state-of-the-art performances.

Fast black-box attacks

In the black-box decision based setup, the attacker can query a model and observes the predicted class. The complexity of the attack is gauged by the number of queries K needed to find an adversarial image of low distortion.

There has been a huge improvement on the amount of queries recently. Brendel et al. report in the order of one million of queries for one image in one of the first decision based black-box BA [14, Fig. 6]. Then, the order of magnitude went down to tens of thousands [START_REF] Chen | Hopskipjumpattack: A query-efficient decision-based attack[END_REF]Fig. 4] [86, Fig. 5] and even some thousands in [118, Fig. 2]. Current benchmarks use others black-box attacks, which are either decision-based (Square Attack [1] in RobustBench [START_REF] Croce | RobustBench: a standardized adversarial robustness benchmark[END_REF] is score-based), or not state-of-the-art (like Gaussian noise in RobustVision [START_REF] Lab | Robust Vision Benchmark[END_REF], or BA [START_REF] Brendel | Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models[END_REF] in ARES [START_REF] Dong | Benchmarking Adversarial Robustness on Image Classification[END_REF]).

SurFree [START_REF] Maho | Surfree: a fast surrogatefree black-box attack[END_REF] and RayS [START_REF] Chen | RayS: A Ray Searching Method for Hard-Label Adversarial Attack[END_REF] are the only decision-based papers with less than one thousand of calls on ImageNet. Yet, RayS [START_REF] Chen | RayS: A Ray Searching Method for Hard-Label Adversarial Attack[END_REF] is designed to minimize the ∞ distortion, whereas SurFree [99] targets 2 . Sect. 6.5 investigates which attack is the best candidate for a fast benchmark.

White-Box Attacks and "Best-effort Mode"

White-box attacks are a family of processes parametrized by one or more parameters. One parameter setting may not be adequate from one classifier to another, and for any image. This explains why experimental results in this literature lack reproducibility. The complexity is usually gauged by the number of gradient computations. Current benchmarks use different white-box attacks: RobustBench [START_REF] Croce | RobustBench: a standardized adversarial robustness benchmark[END_REF] relies on PGD [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] (with 2 parameters Π), RobustVision [START_REF] Lab | Robust Vision Benchmark[END_REF] use DeepFool [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF], and ARES [START_REF] Dong | Benchmarking Adversarial Robustness on Image Classification[END_REF] CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF].

In this chapter, we propose the concept of best-effort mode enabling a fair comparison of attacks and classifiers. It consists of automatically setting the attack to perform as well as possible on a given iteration budget. It finds the parameters such that the attack is successful and the 2 -norm of the perturbation is minimized. (The attack parameters are usually defined within ranges and the optimization may fail providing adversarial).

The implementation of CW is already optimized and therefore needs no tweaking. Two parameters are still defined by the user: boundaries of research and the number of iterations for iterative attacks. FGSM This attack depends on one parameter . Best-effort simply means running a binary search to find the lowest value that successfully crafts an adversarial perturbation.

iFGSM This attack also depends on one parameter , for a given number of iterations. We perform the iterative search in the same fashion as previously.

PGD Our best-effort mode runs a binary search on the radius α. The iterations budget is equally distributed between the number of iterations and the binary search (e.g. if N iter = 100, 10 radii are tested with N run = 10 iterations each), while is set to 2α/N run . With this value of , adversarial samples are not projected back onto the 2 -ball of radius α at least within the first half of the N run iterations. Our experiments confirm that this empirical choice is good. BP This attack leads to very good results when set up correctly. It finds an adversarial sample (stage 1) that is then refined (stage 2). For stability, we aim at finishing stage 1 within roughly the same number of iterations for each image. Inspired by Deepfool [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF], this is done through a first-order approximation of the loss:

L adv (x o + u) ≈ L adv (x o ) + u ∇L adv (x o ). (6.6)
Branching this linearization with (4.20) gives the value of α canceling the loss within κ iterations:

α = L adv (x o ) ∇L adv (x o ) 2 κ j=1 γ j . ( 6.7)
We set κ = 0.2 × N iter and experimentally observe that stage 1 is more or less completed when desired, leaving ≈ 0.8 × N iter iterations to stage 2. Section 6.5 compares PGD, C&W and PGD to show differences in complexity.

Quantization

The adversarial samples are quantified in the pixel domain to create images. The first option considers the quantization as a post-processing not interfering with the attack. The second option performs quantization at the end of any iteration.These options are tested on several black and white box attacks. The quantization will be a post-processing for white-box attacks as detailed in Chap. 7, whereas the second option give better results on black-box attacks.

Experiments

All the attacks are run on 1000 ImageNet images from ILSVRC2012 validation set with size n = 3 × 224 × 224.

Selecting the worst case attacks

Black box attacks: Figure 6.2 compares the evolution of the half-distortion (6.5) in function of the query amount for four decision-based black-box attacks: SurFree [START_REF] Maho | Surfree: a fast surrogatefree black-box attack[END_REF],

RayS [START_REF] Chen | RayS: A Ray Searching Method for Hard-Label Adversarial Attack[END_REF], GeoDA [START_REF] Rahmati | GeoDA: a geometric framework for black-box adversarial attacks[END_REF], and QEBA [START_REF] Li | Qeba: Query-efficient boundary-based blackbox attack[END_REF]. SurFree and RayS reach their best effort within 3000 queries, while QEBA and GeoDA do not since their d1 /2 still decrease after 5000 queries. Yet, SurFree obtains quantified adversarials with much lower distortion. Therefore, our benchmark only needs this attack. The number of queries is kept at 5000 to be sure to reach the optimal value of d 1/2 . White box attacks: Figure 6.3 compares three white-box-attacks in the best effort mode: PGD [START_REF] Kurakin | Adversarial examples in the physical world[END_REF], CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF], and BP [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] with our trick 6.7. They all reach the same d1 /2 when given a large complexity budget. Yet, BP converges faster than the others. Our benchmark uses this version of BP to evaluate the white-box-robustness.

Comparison with other benchmarks

Table 6.1 lists several benchmarks. Most of them evaluate the robustness as the successrate under a prescribed 2 or ∞ distortion budget. But, these budgets are set arbitrarily or even not constant within the same benchmark for RobustML. Our half-distortion (6.5) is parameter-free. It returns an accurate, reliable and fair measurement of robustness. Some benchmarks need many attacks to get a full vision of the robustness: ARES [START_REF] Dong | Benchmarking Adversarial Robustness on Image Classification[END_REF] and RobustVision [START_REF] Lab | Robust Vision Benchmark[END_REF] use 11 attacks. This is too time-consuming. On the contrary, ADBD [START_REF] Chen | RayS: A Ray Searching Method for Hard-Label Adversarial Attack[END_REF] focuses on a single black-box attack, which is indeed outdated. RobustBench [START_REF] Croce | RobustBench: a standardized adversarial robustness benchmark[END_REF] condenses four attacks in one measure elegantly: for a given image, if the first simple attack does not succeed within the distortion budget, then the second more complex one is launched etc. The total runtime heavily depends on the distortion budget. Yet, black-box and white-box attacks use different mechanisms. Our benchmark reports a measurement for each separately.

Benchmarking models

Table 6.2 compares standard models from timm [START_REF] Wightman | PyTorch Image Models[END_REF] and torchvision [START_REF] Marcel | Torchvision the Machine-Vision Package of Torch[END_REF] libraries. Here are some intriguing results. Robustness in white box vs. black box. One does not imply the other. Fig. 6.4 even shows a negative correlation. However, some models escape this rule. For instance, VGG16 is neither robust in black box nor in white box. EfficientNet AdvProp [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF] follows the opposite trend. We believe that black-box robustness reveals the complexity of the borders between classes, and white-box robustness indicates how close natural images are from the borders. This highlights the importance of having two different measurements.

The importance of the training procedure.

There is on average a factor 20 between the half-distortions in white and black box. This factor drops to 4 and 10 for the models adversarially trained: ResNet50 [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF], EfficientNet AdvProp [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF]. Table 6.2 lists four EfficientNet models sharing the same architecture but different training procedures. Their accuracies are similar but there is up to a factor of 2 between the robustnesses. The same holds on the three variants of Resnet50. The gaps in accuracy and robustness are noticeable with standard models from timm [START_REF] Wightman | PyTorch Image Models[END_REF] and torchvision [START_REF] Marcel | Torchvision the Machine-Vision Package of Torch[END_REF]. It is even more visible with adversarial training from [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF]: the gain in robustness is impressive but at the cost of a big drop in accuracy.

Model

Parameters

Chapter Conclusion

This chapter introduced a rigorous benchmark based on a new and independent measurement of robustness: the half distortion. RoBIC is faster than the other benchmarks. This allows to tackle larger images which is more realistic.

In addition to the accuracy, RoBIC gives the black box robustness, and white box robustness. We believe that the first indicates how far away the class boundaries lie from the images whereas the last reflects how curved are the boundaries. As the other benchmarks, two limitations hold: The network must be differentiable to run a white box attack, and deterministic to run a black box attack.

Introduction

Recent attacks aim at reducing the distortion (usually measured as 2 or ∞ norm), increasing the probability of success, and speeding up the process. Even if some learning procedures result in more robust classifiers [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF] and some images are harder to attack, recent white-box attacks craft perturbation invisible to the human eye in most cases provided their complexity budget is large enough.

Regardless their complexity, very few attacks consider the specificity of the medium. A raster image is in its digital form a 3-dimensional matrix of integers, such as the PNG image format. JPEG images [START_REF] Gregory | The JPEG still picture compression standard[END_REF] are coded as integer matrices representing DCT coefficients in different color spaces. To forge an adversarial image rather than just a sample encoded in a floating-point tensor, one needs to craft an integral perturbation. Added to the original image, the result must remain within the defined boundaries (i.e. [0, 255] n with n the number of pixels in the spatial domain).

Attacks rarely address this constraint. It is sometimes argued that the attack is performed inside the classifier in the white-box setup and thus is not required to be integral. While debatable, we consider this assumption to be very niche. Ironically every attack still clips their samples within the boundaries of a pre-processed image (i.e. [0,255]). The white-box setup means that the attacker can replicate the model in his/her garage to prepare an attack that will later on be deployed against a remote classifier service analyzing integral images.

The first idea that comes to mind is to round pixel-wise the crafted perturbation to the nearest integer. This is not working. Perturbations are so small that they are partially erased (set to zero) by rounding. Table 7.1 gives a first insight of this problem. This preliminary experiment is run on 1,000 randomly selected images from ImageNet. The same images are used throughout this chapter. The studied classifier is EfficientNet-b0. Rounding an optimized attack significantly increases the accuracy (decreases the success rate of the attack). An alternative is to round after every step of an iterative attack like PGD 2 . This requires significantly more distortion at every iteration so that the perturbation is not erased by rounding. This is displayed as PGD 2 round in Table 7.1: it succeeds in beating classification but generates 64% more distortion than our quantization. Figure 7.1 illustrates further these results. JPEG compression smoothes the image by cancelling high frequencies, especially at low quality factor. This has little effect on the accuracy of the classifier over natural images while adversarial perturbations are very sensitive to it. Even an attack with an increased distortion budget does not easily fool a classifier after a JPEG compression especially at low-quality factor. Table 7.1 shows that JPEG compressing images forged by FGSM or PGD does not create adversarial examples. This is the reason why some works propose JPEG as a defense against adversarial attacks [START_REF] Liu | Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples[END_REF][START_REF] Shaham | Defending against Adversarial Images using Basis Functions Transformations[END_REF]. Backward Pass Differentiable Approximation (BPDA [3]) was developed to beat this defense and can be used to create JPEG adversarial images as well. BPDA approximates the JPEG compression by a differentiable transformation. This approxima-tion is less accurate for low JPEG quality factors. BPDA then fails to attack some images although it generates more distortion than our quantization. Table 7.1 shows that our method JPEG quantizes images which remain adversarial almost surely with a distortion comparable to the compression itself (see section 7.4).

Contributions

This chapter proposes a quantization dedicated to adversarial perturbation so that samples can be saved as adversarial images as shown in Fig. 7.2. It is a post-process to be used on top of any attack. This method however relies on gradients available in the white-box setup. It is quick in the sense that it typically needs fewer iterations than the attack per se. Our intensive experimental study shows that forging real images, be it in the raster or JPEG format, is not a hard constraint for the attacker when quantization is properly managed. In other words, our quantization adds little to no extra distortion.

Quantization is restricted to a desired range of values providing control over ∞distortion (see Sect. 7.2). We also extend this method to the JPEG domain (see Sect. 7.3). This proves to be challenging since this compression erases high-frequencies typical of an adversarial perturbation. Finally we also propose best-effort mode for multiple attacks (see Sect. 6.4.2). This mode finds the best parameter setting for each original image in order to reveal the intrinsic power of an attack. This allows a fair comparison of the attacks.

Note that generating adversarial contents in the quantized domain can also be used in another context. The Euclidean distortion can for example be replaced by a steganographic cost to increase the undetectability of adversarial perturbation (see Chap. 8)). The attacker may also target a network which is not a classifier, like a regression function evaluating the visual quality (see Appendix A).

Outlines

Sections 7.2 and 7.3 detail our improved post-processing in both spatial and JPEG domains. Section 7.4 presents experimental results in various scenarios as well as a thorough study of the impact of each parameter. This further motivates our choice of default parameters. Example images are also displayed throughout this section.

Our code is available at gitlab.inria.fr/bbonnet/adversarial-quantization

Smart Quantization in the Spatial Domain

We saw in 4.2.2 the principle of pre-processing I 0 to x 0 . Values in tensor x 0 are encoded as floating-point variables so that their domain is pseudo-continuous. Yet, Eq. ( 4.3) shows that there are only 256 different possible values for a given entry of x 0 . White-box attacks modify x 0 into x a ∈ [0, 1] n which entries may not equal one of the 256 admissible values. This means that by reversing the preprocessing (4.3), the attacker gets x a ∈ [0, 255] n whose pixel values may not be integers. For readability, we integrate the preprocessing to our models as the first layer. The sequel focuses on images x ∈ [0, 255] n whereas original images are in 0, 255 n . In particular, x 0 ∈ 0, 255 n .

Assume that an attack has forged the adversarial sample x a ∈ [0, 255] n , which is not quantized, i.e. pixel values are a priori not in 0, 255 n . This section presents our mechanism carefully quantizing the pixel values to keep adversarial images adversarial.

Problem Statement

Our mechanism casts x a ∈ [0, 255] n to x q ∈ 0, 255 n . Its goal is to solve the optimization problem defined in (4.12) with the additional integral constraint: x a is eventually replaced by x q ∈ 0, 255 n . Since we are working in a white-box environment, our method can rely on the following quantities:

-the original image x 0 ∈ 0, 255 n , -the unquantized adversarial image x a ∈ [0, 255] n , -the adversarial loss L adv (x) and its gradient ∇L adv (x) (see Eq. (4.15)).

We introduce the following weak signals:

u := x a -x 0 , ( 7.1
)

q := x q -x a . (7.
2)

The quantization noise q plays the central role in our approach. We redefine the distortion and the loss functions w.r.t. this variable:

D(q) := x q -x 0 2 = ||u + q|| 2 (7.3) L(q) := L adv (x q ) = L adv (x a + q). (7.4)
There is obviously a trade-off between these two quantities. For instance, the choice q † = arg min D(q) = -u (7.5) cancels the perturbation and makes x q = x o not adversarial.

Finding the adversarial image x q minimizing distortion D(q) can be expressed as: min q∈Q,L(q)<0 D(q), (

where Q is the set of admissible solutions. Remark that x q -x o = q + u ∈ Z N since it is the difference of two integer vectors. This implies that q ∈ Q ⊂ Z n -u, i.e. the grid Z n shifted by translation -u. For instance, quantizing by rounding the perturbation gives

q = [x a ] -x a = [x o + u] -(x o + u) = [u] -u, ( 7.7)
where [u] is the closest integer value of u component-wise.

We add the other constraint of q being of limited amplitude. We introduce a new parameter d f ∈ N * , so-called degree of freedom which reflects the number of choices per component:

q i + u i =                    [u i ] if d f = 0 u i or u i if d f = 1 [u i ] -1, [u i ], or [u i ] + 1 if d f = 2 . . .
The case d f = 0 amounts to rounding the perturbation and there is no freedom to choose another integer. This implies that the ∞ -norm of the quantization noise is bounded by q ∞ ≤ 1/2. In the general case, this norm is bounded by q ∞ ≤ (d f + 1)/2. The case d f = ∞ means that the perturbation is quantized to integers but there is no control on the norm q ∞ .

In the end, the set of admissible quantization noises is defined by the product space

Q = ⊗ N i=1 Q i with Q i =      {[u i ] - d f 2 , . . . , [u i ] + d f 2 } -u i if d f is even, { u i - d f -1 2 , . . . , u i + - d f -1 2 } -u i if d f is odd. (7.8)
Note that the number of admissible solutions is exponential with N : |Q| = (d + 1) N .

We now assume that ∀q ∈ Q, q is small enough to make a first order approximation of the loss:

L(q) := L adv (x q ) ≈ L adv (x a ) + q g, ( 7.9)
where g := ∇L adv (x a ). For instance, the choice

q ‡ i =      min(Q i ) if sign(g i ) > 0 max(Q i ) if sign(g i ) < 0 (7.10)
minimizes the first order approximation of L(q). For d f large enough, this certainly ensures that L(q) < 0 and x q is adversarial but the distortion is large. The solution to problem (7.6) can consequently be seen as a compromise between (7.5) minimizing the distortion and (7.10) minimizing the loss.

Solution

The solution of (7.6) is given by a Lagrangian formulation. Define the following functional: J λ (q) := D(q) + λL(q), (7.11) where λ ∈ R + is the Lagrangian multiplier balancing adversariality and distortion quantities. Suppose we know how to efficiently minimize that functional by q λ := min Q J λ (q), ∀λ ∈ R + . The expected behavior along λ is for L(q λ ) to decrease while D(q λ ) increases (see Fig. 7.3). For instance, 1. When λ = 0, all importance is given to D(q). This results in a distortion-based quantization q † erasing the perturbation u as seen in (7.5).

2. When λ → +∞, all importance is given to L(q). This results in a gradient-based quantization q ‡ of (7.10).

Since the distortion strictly increases with λ, the optimal solution of problem (7.6) is then q λ where λ = min{λ : L(q λ ) < 0}. We practically compute this optimal solution in a two step approach.

Minimizing the functional

Finding the minimum of J λ is difficult, except if we rely on approximation (7.9), then we can write that J λ (q) ≈ u + q 2 + λL adv (x a ) + λq g (7.12)

is convex and thus is minimized when ∇J λ (q) = 0. This happens for q = qλ , where

qλ := - λ 2 g -u. (7.13)
Yet this solution is not admissible because it does not belong to Q a priori. We rewrite the approximation (7.12) as

J λ (q) ≈ q -qλ 2 + λ 2 4 g 2 + λ(g qλ + L adv (x a )) (7.14)
to outline that the minimizer on Q is just its closest element to qλ . This amounts to first quantize qλ onto Z N -u and then clip:

q λ,i = clip [min(Q i ),max(Q i )] ([-λg i /2] -u i ) . (7.15)

Finding the optimal λ

The relaxation of the integral constraint and the linearisation of the loss provides a first approximation of λ . Inserting (7.13) in (7.9) yields:

λ c := 2(L adv (x a ) -u g) g 2 . (7.16)
We find the value of λ * by looking around λ c . Similarly to our previous work [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF], we run a line search in the interval [0.01λ c , 100λ c ]. For every value tested, we compute the optimal perturbation (7.15), add it to x a , and submit this to the classifier. If it is adversarial, then the value of λ is decreased. It is increased otherwise. In other words, the computation of the best quantization noise q λ given λ relies on the linear approximation (7.9), but the finding of λ implies to evaluate the classifier.

Smart Quantization in the JPEG Domain

The JPEG file format represents an image as a 3-dimensional tensor of scaled DCT coefficients quantized to integers. Extending our quantization to the DCT domain enables us to craft JPEG adversarial images.

JPEG Compression

The JPEG compression is schematically done in four stages (excluding entropic source coding). A RGB image (Red, Green, Blue) is linearly converted to Y C b C r (Luminance, blue-Chroma, red-Chroma). This linear transform ensures that all values lie in the range [0, 255]. Then each channel undergoes the 8 × 8 block DCT-transform. The resulting DCT coefficients are finally divided by quantization steps which depend on their frequency bin and the quality factor. A lower quality factor increases the quantization steps s.t. the following quantization loses more information. This pipeline is linear and thus can be summarized by X = Jx + C, where X ∈ R n stores the scaled DCT coefficients and C is a constant vector encoding the shift in the conversion RGB to Y C b Cr. We have supposed here that n = 3LC where the numbers of columns C and lines L are multiple of 8. The matrix J ∈ R n×n encodes the color conversion, the block DCT, and the division by the quantization steps. It is cumbersome to express it due to the flattening of images in n dimensional vectors. The main properties are that J is invertible and that it is not an isometry, i.e. Jx = x in general. This is due to the scaling with the quantization steps but also to the color domain conversion [START_REF] Shuyuan | High-Quality Color Image Compression by Quantization Crossing Color Spaces[END_REF].

Vectors in this JPEG domain are denoted with capital letters: X o , X a , X q , U , and Q with:

U = X a -X o = J(x a -x o ) = Ju, (7.17) Q = X q -X a = J(x q -x a ) = Jq. (7.18)
The scenario is the following: As in the previous section, an attack forges x a and we have to craft its JPEG version. This amounts to convert it in the JPEG domain, X a = Jx a +C, and to quantize these coefficients with care so that the image remains adversarial. Note that X o = Jx o + C is a priori not an element of Z N , unless the original image was already quantized in the JPEG domain.

Quantization

We write the problem by focusing on the quantization noise Q in the JPEG domain. Since x q = J -1 (X q -C), Eq. (7.9) is written as:

L adv (x q ) = L(J -1 (X a + Q -C)) = L(x a + J -1 Q) ≈ L adv (x a ) + (J -1 Q) g, (7.19)
where g is the gradient of the loss function in the pixel domain. As for the Euclidean distortion, we have

x q -x o 2 = u + q 2 = u + J -1 Q 2 . (7.20)
Finally, the Lagrangian functional defined in (7.11) becomes:

D(J -1 Q) + λL(J -1 Q). (7.21)
Following the same reasoning as in the spatial domain, the minimum of this functional when relaxing the quantization constraint amounts to set J -1 Q to qλ given in (7.13).

Equivalently:

Qλ = - λ 2 Jg -U. (7.22)
Yet, this time the rounding is done with respect to X a since we need X a + Q to be an integral vector:

Q λ = - λ 2 Jg -U + X a -X a . (7.23)
Like in the spatial domain, this value is clipped to belong to the set Q, defined in (7.8) replacing u by U . Note that if the original image is in JPEG format with the same quality factor so that X o is an integer vector, then Q λ = -λ 2 Jg -U , and we recover a quantization similar to (7.15).

Experimental Work

Implementation Details and Setup

We make the following implementation choices.

One can either implement the transformation J -1 as a preprocessing layer like previously discussed in Sect. 4.2.2. This allows to directly feed the classifier with JPEG images. Attacks also naturally adapt to this new object as the gradient back-propagates through the transformation layer. Yet, this approach makes the attack domain-specific. Our choice of design is to implement our quantization separately on top of any attack. Our method forges JPEG images from a spatial adversarial sample x a resulting from an attack on x o .

Our implementation builds on the Python library Pillow to be as close as possible to the official JPEG standard. Two differences remain: JPEG may apply a sub-sampling on the C b and C r channels. Taking into account sub-sampling is straightforward with backpropagation and auto-differentiation. For the sake of simplicity, we work on JPEG images without sub-sampling and the color channels have all the same size. JPEG may apply clipping when converting from one color domain to another. These border effects produce small information losses. We do not apply this lossy step to keep the transformation linear.

The experiments use 1,000 PNG versions of images from the validation set of Imagenet ILSVR 2012 [START_REF] Russakovsky | Imagenet large scale visual recognition challenge[END_REF]. Unless stated otherwise, the attacked classifier is EfficientNet-b0 [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF]. EfficientNet in its b0 configuration is a recent and lightweight classifier that achieves high accuracy on ImageNet. Our previous work [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] shows that distortion created by quantization is proportional to the distortion created by the attack. We thus choose BP to be the default attack. It performs well with few iterations in its best-effort setup as seen in Fig. 6.1. The research on the optimal value of λ is done by default over 10 iterations. Finally, both spatial and JPEG quantization are run by default with degree of freedom

d f = 1 unless specified otherwise.
The protocol is the following. For the spatial domain, x q = x o if the original image is already misclassified, otherwise BP produces x a that our method quantizes to x q . For the JPEG domain, x o is first converted in the JPEG format. If this triggers a misclassification, then this JPEG version of x o is the adversarial image X q . Otherwise, BP produces x a from the JPEG original that our method quantizes to X q .

Operating curves in both domains are displayed as follow:

-Distortion is calculated w.r.t. the original spatial image.

-Misclassified original images in each domain are considered as already adversarial at null distortion.

Note that compressed JPEG images do not have a null distortion since they differ from the original spatial image. For the sake of clarity, we however consider they do. This choice is further motivated in Section 7.4.3. Without quantization, the distortion is the same in both domains and it strictly increases w.r.t. λ as predicted by (7.13) or (7.22). With quantization and clipping (with d = 1), the distortion increases much more in the JPEG domain because of the coarser quantization steps in the high-frequency bins. It also does not start at 0 but at the distortion induced by the regular JPEG compression of the original image.

Investigations on the Search of λ

In the spatial domain, the adversarial losses (with or without quantization and/or clipping) are well-approximated by (7.9) for small perturbation, i.e. when λ is small. In particular, they converge to L adv (x o ) when λ → 0. The linear approximation is useful for predicting when the loss cancels. Adding quantization and clipping constrains the problem and we observe that λ > λ c . This implies a stronger distortion. Of course, the linear approximation is very wrong when predicting losses below -1.

The picture is less clear in the JPEG domain. The approximation holds true in the beginning and until L adv (X q ) reaches 0. The approximation λ c remains extremely relevant. However L adv sometimes becomes non-monotonic as λ → ∞ because of the strong distortion.

For this reason, our search of λ * slightly differs. It starts from λ c given in (7.16). This is the same value for both spatial and JPEG domains. In the spatial domain, a line search

Distortion

Approx. Loss (eq.24)

Quant. Loss, d=1 (eq.30)

Unquant. Loss (eq.28)

Unquant. Distortion Quant. Distortion, d=1

Eq. (4.9)

Eq. (4.15)

Eq. ( 4

Distortion

Approx. Loss (eq.34)

Quant. Loss, d=1 (eq.38)

Unquant. Loss (eq.37)

Unquant. Distortion Quant. Distortion, d=1

Eq. ( 4. [START_REF] Carlini | Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods[END_REF] Eq. (4.23)

Eq. (4.22)

Figure 7.3 -Comparing the approximated loss (7.9) (resp. (7.19)) with the loss without rounding: L(q λ ) in (7.13) (resp. L(J -1 Qλ ) in (7.22)) and the loss with quantization L(q λ ) in (7.15) (resp. L(J -1 Q λ ) in (7.23)) as a function of λ in the (left) spatial domain, resp. (right) JPEG90. Distortion is also displayed with scale on the right. within [0.01 • λ c , 100 • λ c ] works well because the loss is almost monotonically decreasing. In the JPEG domain, the loss is less predictable and we instead sample n values in this interval:

λ i : = λ c • 10 α i , ( 7.24
)

α i := 2 • n -2i n . (7.25)
The lowest value tested that verifies L adv < 0 is necessarily the best since distortion strictly increases with λ.

It is expected that the line search in the spatial domain is more efficient than the uniform sampling in the JPEG domain. This is indeed illustrated by Fig. 7.4. Quantization in the spatial domain converges faster thanks to the line search. However, for both approaches, searching for λ with n = 10 steps is sufficient. Note that each step only makes a forward pass through the classifier network. In comparison running BP makes N iter = 50 forward and backward passes. Our quantization is thus faster than the attack. It gets slowed down by the DCT transform in JPEG domain however.

Impact of Quality Factor in JPEG Domain

The quality factor of JPEG determines the values of the quantization steps applied on the DCT coefficients. A lower quality factor leads to a bigger loss in information and degradation of the image. This is especially true for high frequencies of the image which are coarsely quantized. The gradients computed during attacks look like noisy patterns and the resulting adversarial perturbation is thus in the high-frequency range. Its distortion needs to be amplified to preserve the adversarial property of the perturbation at a lower quality factor.

Figure 7.8 shows operating curves for adversarial images crafted for different JPEG quality factors (plain curves) which confirm this last statement. For the sake of comparison, the distortion of the JPEG compression on the original images is also displayed as a cumulative sum over all the 1,000 images (dashed curves). We observe that our adversarial quantization returns a distortion very close to regular JPEG compression. This leads to the interesting result that some adversarial images quantized in JPEG often have the same distortion (or even less in few cases) than just the original image compressed. Figure 7.13 illustrates this result with JPEG75. On three of the four examples displayed, distortion of the adversarial image is equal or very close to the distortion of the compressed image. In other words, a perturbation being compliant to JPEG is not a strong constraint for the attacker as it is almost distortion-free.

The price to pay is a small extra complexity thanks to our method. This is necessary as the JPEG compression alone is not working as an attack. Table 7.1 indicates that only 17.0% (JPEG90 and JPEG100) or 19.0% (JPEG75 and JPEG60) of compressed original images are misclassified.

We plot the same operating curve as previously considering JPEG compression as an attack on Fig. 7.8 (dash plots). Its discloses which images were naturally adversarial BP+JPEG60 quant. BP+JPEG60 quant.

d(J -1 (X q ), x o ) = 5.71 d(J -1 (X q ), x o ) = 5.

900 'water tower' 650 'microphone' 727 'planetarium'

813 'spatula' after JPEG compression and which ones needed to be attacked and quantized by our method. There is no correlation between the distortion due to JPEG and the chance that it succeeds as an attack. Indeed, most of these images are already misclassified in PNG and still adversarial once in JPEG.

Figure 7.6 shows the operating curve w.r.t. to the Euclidean distance from the original image compressed to JPEG format. These curves start at a success rate of 17.0% (resp. 19.0%) since a null distortion corresponds to misclassified original JPEG images. Except for these specific images, our method forges an attacked JPEG image different than its original JPEG version although both of them are equivalently far away from the original PNG image.

Distortion is mostly imperceptible when the quality factor is high. It does start to be noticeable with JPEG75. Fig. 7.13 displays for example at the last row some little artifacts at the bottom of the lighter which are not typical from a JPEG compression. On JPEG60, quantization artifacts are more frequent and important. Figure 7.5 displays two examples. These patterns are especially visible on smooth image regions usually not affected by JPEG compression. Quantized attacks remain imperceptible when the image is highly textured.

We conclude this section by the following remarks.

-The classifier is very robust to JPEG compression alone.

-For any tested quality factor, almost all images are successfully attacked and quan- in both domains and the benefit of increasing d f seems rather low. A higher value of d f is not necessarily a better choice. This is particularly visible in Fig. 7.10 with the quality factor 100.

The reason lies in the metrics used. The distortion (6.2) is proportional to the 2norm, i.e. the square root of the squared-difference, summed over all pixels. Adding +2 on one coefficient thus costs 2 whereas adding +1 on two coefficients costs √ 2. The degree of freedom constrains the ∞ norm of the total perturbation u + q (or U + Q in the JPEG domain). This clipping increases the spreading of the perturbation over all the coefficients: since the coefficients with large gradient amplitude can not host a large perturbation, λ increases to compensate this clipping on the other components. This more uniform distribution of the perturbation energy over the coefficients yields a lower Euclidean distortion but also a lower perceptual impact. Figure 7.9 shows two images quantized with two different quality factors each quantized with d f = 1 and d f = ∞. Their Euclidean distortion w.r.t. the original image is similar but the artefact are more visible for d f = ∞ (quantization but no clipping). We notice that this holds for any image.

Quantization on Different Classifiers and Attacks

This study considers four recent classifiers: EfficientNet-b0 [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF] and its adversarially trained counterpart EfficientNet-b0-advprop [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF]; RegNetX-032 [START_REF] Radosavovic | Designing Network Design Spaces[END_REF], and the older ResNet50 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. to note that both neural networks misclassified the right image as the same class (828: tray) whereas the left image is misclassified with different labels yet semantically very close. These classification errors are also understandable from a human point of view. As a final comment on classifiers: all images misclassified by EfficientNet-b0 (170 total) are misclassified by RegNetX-032 as well, and RegNetX-032 misclassified 6 more images (176 total). Figure 7.11 shows the operating curves of all four classifiers. They are attacked with BP or PGD 2 and quantized in both spatial and JPEG90 domains. Gradients from one classifier to another vary with the number and nature of hidden layers. This affects how an attack behaves. The hierarchy between unquantized attacks however remains the same from one classifier to another as seen in Fig. 6.1. This order remains after quantization in the spatial domain: BP outperforms PGD 2 .

d f = ∞ d(J -1 (X q ), x o ) = 2.81 d(J -1 (X q ), x o ) = 2.89 130 'flamingo' 130 'flamingo' ground-truth: 129 'spoonbill' d(J -1 (X q,d f =1 ), d(J -1 (X q,d f =∞ )) = 0.67 d(J -1 (X q ), x o ) = 5.90 d(J -1 (X q ), x o ) = 5.92 432 'tank suit' 432 'tank suit' ground-truth: 776 'sax, saxophone' d(J -1 (X q,d f =1 ), d(J -1 (X q,d f =∞ )) = 0.46
Nevertheless, the differences between classifiers and attacks are barely noticeable in the JPEG domain. Distortion added by JPEG compression takes over as Sec. 7.4.3 explains and imposes the common shape of the operating curve. This shows the adaptability of our quantization w.r.t. which neural network is attacked.

Transferability

Our quantization method aims at creating an adversarial image with minimum distortion. The image therefore lies just behind the frontier of correct classification for the targeted classifier. Therefore, no transferability to other deep neural networks is guaranteed. We consider the scenario where the attacker knows that the deployed classifier belongs to an ensemble but she/he does not know which one exactly. The goal is to forge images adversarial for all the classifiers in the ensemble. Our strategy is to aggregate the losses of the classifiers with the maximum operator so that we focus on the most robust element of the ensemble and to aggregate their gradients with the average operator like in Expectation over Transformation (EoT [3,4]). Figure 7.11 shows that beating all the classifiers in the ensemble does not amount to beat the most robust one (i.e. EfficientNet-b0 advprop). More distortion is required instead in spatial domain. In JPEG domain we observe that the slope of the ensemble curve is similar to the curve of any single model. The distortion created by the quantization is still on par with compression alone. We do note however that quantizing for several classifiers is a more difficult task in JPEG. The final accuracy is 6.7% in JPEG, 0% in spatial domain.

JPEG Compression as a Defense

As mentioned several times throughout this chapter, JPEG compression usually erases adversarial perturbations while not spoiling the accuracy of the classifier over natural images. For this reason, JPEG compression has been studied as a means of defense against adversarial samples [START_REF] Liu | Feature Distillation: DNN-Oriented JPEG Compression Against Adversarial Examples[END_REF][START_REF] Shaham | Defending against Adversarial Images using Basis Functions Transformations[END_REF]. It acts as a low-pass filter reforming the input image. Table 7.2 shows this is indeed true for our spatially quantized images. While a quality factor of 90 reforms 42% of our adversarial PNG images, a quality factor of 60 reforms ≈ 70%. This proves to be a very effective defense against adversarial samples. However, our adversarial images quantized in the JPEG domain are naturally more robust to JPEG compression. The results show interesting properties:

-The performance of the attack is maximized when the quality factor matches the one used at the defense. -Compressing at the same quality factor does however reform few images (< 1% in all three cases) because JPEG is not idempotent. -Quantized adversarials at a given quality factor are robust to defenses with a higher quality factor.

Chapter Conclusion

We have proposed a method (improved from an initial work [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF]) to effectively quantize adversarial samples in order to craft adversarial images in spatial or JPEG domains. This quantization guarantees adversariality while minimizing the distortion. It runs within few forward calls to the network making it faster than simple attacks and it conveniently operates on top of any white box attacks for broader usability.

When dealing with JPEG compression, the distortion induced by the attack is a very small fraction of the distortion induced by sole compression, and crafting adversarial images in JPEG at low-quality factors also provides robustness to countermeasures based on JPEG compression.

The presented methodology is moreover ubiquitous and it could be transferred to other domains such as JPEG2000 [START_REF] Zandi | CREW: Compression with Reversible Embedded Wavelets[END_REF] or HEIF [START_REF] Miska | The High Efficiency Image File Format Standard [Standards in a Nutshell[END_REF], and the optimization setup can also be used for other metrics than classification (for example regression as proposed in [10]) and on other distances such as steganographic costs [START_REF] Bonnet | Forensics through stega glasses: the case of adversarial images[END_REF] in order to generate adversarial images that are less prone to be statistically detected.

Natural

BP+spatial quant. JPEG75 compression BP+JPEG75 quant.

2 'white shark' 3 'tiger shark' 2 'white shark' 3 'tiger shark'

d(x q , x o ) = 0.0 d(x q , x o ) = 0.02 d(J -1 (X 0 ), x o ) = 3.15 d(J -1 (X q ), x o ) = 3.15 791 'shopping cart' 161 'basset hound' 791 'shopping cart' 161 'basset hound' d(x q , x o ) = 0.0 d(x q , x o ) = 0.53 d(J -1 (X 0 ), x o ) = 6.88 d(J -1 (X q ), x o ) = 7.52
754 'radio, wireless' 766 'rotisserie' 754 'radio, wireless' 766 'rotisserie' d(x q , x o ) = 0.0 d(x q , x o ) = 0.17

d(J -1 (X 0 ), x o ) = 3.63 d(J -1 (X q ), x o ) = 3.
(x q , x o ) = 0.0 d(x q , x o ) = 0.21 d(J -1 (X 0 ), x o ) = 3.72 d(J -1 (X q ), x o ) = 3.79

Introduction

The connection between adversarial examples and forensics/anti-forensics is obvious. First, adding an adversarial perturbation to delude a processing is an image manipulation per se and therefore detecting adversarial examples is a forensic task by itself. Second, techniques forging adversarial examples are also used to fool forensics detectors as proposed in [START_REF] Güera | A counter-forensic method for CNN-based camera model identification[END_REF] [5]. In this case, the adversarial attack is a counter-forensics strategy to conceal an image manipulation.

Paper [START_REF] Quiring | Forgotten Siblings: Unifying Attacks on Machine Learning and Digital Watermarking[END_REF] makes the connection between adversarial examples and information hiding (be it watermarking or steganography). Both fields modify images (or any other type of media) in the pixel domain so that the content is moved to a targeted region of the feature space. That region is the region associated to a secret message in information hiding or to a wrong class in adversarial examples. Indeed, paper [START_REF] Quiring | Forgotten Siblings: Unifying Attacks on Machine Learning and Digital Watermarking[END_REF] shows that adversarial examples benefits from ideas proven efficient in watermarking, and vice-versa. This chapter contributes to the same spirit by investigating what both steganography and steganalysis bring to the the "cat-and-mouse" game of adversarial examples. There are two natural ideas: Steganalysis aims at detecting weak perturbations in images. This field is certainly useful for the defender.

Steganography is the art of modifying an image while being non-detectable. This field is certainly useful for the attacker.

These two sides of the same coin allow to mount a defense and to challenge it in return, as done in other studies [3,[START_REF] Carlini | Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods[END_REF][START_REF] Tramer | On Adaptive Attacks to Adversarial Example Defenses[END_REF]. This chapter aims at revealing the status of the game between the attacker and the defender at the time of writing, i.e. when both players use up-to-date tools: state-of-the-art image classifiers with premium steganalyzers, and best-in-class steganography embedders. This chapter proposes three contributions:

-Assess the robustness of very recent image classifiers, EfficientNet [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF] and its robust version [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF], -Apply one state-of-the-art steganalyzer (SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF]) for forensics purposes, i.e. to detect adversarial images, -Use the best steganographic schemes to craft counter-forensics perturbations reducing the detectability: HILL [START_REF] Li | A new cost function for spatial image steganography[END_REF] uses empirical costs, MiPod [START_REF] Sedighi | Content-Adaptive Steganography by Minimizing Statistical Detectability[END_REF] models undetectability from a statistical point of view, while GINA [START_REF] Li | A strategy of clustering modification directions in spatial image steganography[END_REF][START_REF] Wang | Non-Additive Cost Functions for Color Image Steganography Based on Inter-Channel Correlations and Differences[END_REF] synchronizes embeddings on color channels. Section 8.2 reviews the connections between forensics, steganography, and adversarial examples. Our main contribution on counter-forensics and experimental results are detailed in Sect. 8.3 and 8.4.

Related Works

Steganalysis for forensic purposes

Steganalysis has always been bounded to steganography, obviously. Yet, a recent trend is to resort to this tool for other purposes than detecting whether an image conceals a secret message. For instance, paper [START_REF] Qiu | A Universal Image Forensic Strategy Based on Steganalytic Model[END_REF] claims the universality of SRM and LBP steganalyzers for forensic purposes detecting image processing (like Gaussian blurring, gamma correction) or splicing. The authors of [START_REF] Farooq | A generic passive image forgery detection scheme using local binary pattern with rich models[END_REF] used this approach during the IEEE IFS-TC image forensics challenge. The same trend holds as well on audio forensics [START_REF] Luo | Improved Audio Steganalytic Feature and Its Applications in Audio Forensics[END_REF]. As for camera model identification, the inspiration from steganalysis (co-occurrences, color dependencies, conditional probabilities) is clearly apparent in [START_REF] Tuama | Camera model identification based machine learning approach with high order statistics features[END_REF]. This reveals a certain versatility of steganalysis. It is not surprising since the main goal is to model and detect weak signals. Modern steganalyzers are no longer based on handcrafted features like SRM [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF]. They are no more no less than Deep Neural Networks like Xu-Net [START_REF] Xu | Structural design of convolutional neural networks for steganalysis[END_REF] or SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF]. The frontier between steganalysis and any two-class image classification problem (such as image manipulation detection) is blurred. Yet, these networks have a specific structure able to focus on weak signal detection. They for example avoid subsampling or pooling operations in order to preserve high frequency signals, they also need large databases combined with augmentation techniques and curriculum learning to converge [START_REF] Yousfi | Breaking ALASKA: Color separation for steganalysis in JPEG domain[END_REF].

However, this general-purpose strategy based on steganalysis method has some drawbacks. It lacks fine-grained tampering localization, which is often an issue in forensics [START_REF] Fan | General-purpose image forensics using patch likelihood under image statistical models[END_REF]. Paper [START_REF] Chen | A gradient-based pixel-domain attack against SVM detection of global image manipulations[END_REF] goes a step further in the cat-and-mouse game with an counter-forensic method: knowing that the defender uses a steganalyzer, the attacker modifies the perturbation (accounting for a median filtering or a contrast enhancement) to become less detectable.

As for adversarial images detection, this method is not new as well. The authors of [START_REF] Schöttle | Detecting Adversarial Examples -a Lesson from Multimedia Security[END_REF] wisely see steganalysis detection as a perfect companion to adversarial re-training. This last mechanism fights well against small perturbations. It however struggles in correctly classifying coarser and more detectable attacks. Unfortunately, this idea is supported with a proof of concept (as acknowledged by the authors): the steganalyzer is rudimentary, the dataset is composed of tiny images (MNIST). On the contrary, the authors of [START_REF] Liu | Detection Based Defense Against Adversarial Examples From the Steganalysis Point of View[END_REF] outline that steganalysis works better on larger images like ImageNet (ILSVRC-2016). They however use a deprecated classifier (VGG-16 [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF]) with outdated steganalyzers based on hand-crafted features (SPAM and SRM).

Adversarial examples

As mentioned multiple times throughout this manuscript, an untargeted adversarial attack aims at finding the optimal point:

x a = arg min x:ĉ(x) =c(xo) x -x 0 , ( 8.1)
where • is usually the Euclidean distance.

Discovering this optimal point is difficult because the space dimension n is large. Even in a white-box scenario, all attacks are sub-optimal iterative processes.

As outlined in Chap. 7, definition (8.1) is very common in literature, yet it is incomplete. The final goal of the attacker is to create an adversarial image I a ∈ D in the pixel domain, not x a ∈ X . Applying the inverse pre-processing is not solving the issue because this a priori makes non integer pixel values. Rounding to the nearest integer, I a = round(x a ), is simple but ineffective. Most networks are so vulnerable that u = x a -x 0 is a weak signal partially destroyed by rounding. x a is no longer adversarial. Note that DDN is a rare example of a powerful attack natively offering quantized pixel values.

Chapter 7 proposed a post-processing Q on top of any attack that makes sure x q = Q(x a ) is (i) an image (integral constraint), (ii) remains adversarial, and (iii) has a low Euclidean distortion x q -x 0 . This chapter follows the same approach but adds another constraint: (iv) be non-detectable. Figure 8.1 shows the characteristic function measuring the probability of success of an attack [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] as a function of the distortion budget (L 2 -norm) against landmark classifiers in the history of ImageNet challenge. Unlike other chapters, distortion is not normalized w.r.t. n. The characteristic function starts at 1-η, where η is the accuracy of the classifier: a proportion 1-η of original images are naturally adversarial since there are misclassified. As we know, the accuracy of the networks increases as time goes by: AlexNet (2012) [START_REF] Krizhevsky | ImageNet Classification with Deep Convolutional Neural Networks[END_REF] < VGG-16 (2015) [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] < GoogLeNet (2015) [START_REF] Szegedy | Going deeper with convolutions[END_REF] < ResNet-50 [START_REF] He | Deep Residual Learning for Image Recognition[END_REF] (2016) < EfficientNet-b0 [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF] (2019). On the other hand, the robustness to this attack can be measured by the

Defenses

The literature proposes four types of defenses or counter-attacks against adversarial examples white-box attacks as seen in Chap. 5. In this work, we consider detection as means of defense against our quantized adversarial images. We evaluate steganalysis as a candidate for this task. We evaluate its efficiency against standard quantization from the previous chapter as well as a steganographic-oriented quantization explained below

Steganographic costs

Undetectability is usually tackled by the concept of costs in the steganographic literature: each pixel location i of a given cover image is assigned a set of costs (w i ( )) that reflects the detectability of modifying the i-th pixel by quantum. Usually, w i (0) = 0, w i (-) = w i ( ), and w i (| |) is increasing. The goal of the steganographer is to embed a message m while minimizing the empirical steganographic distortion:

D( ) := n i=1 w i ( i ). (8.2)
This is practically achieved using Syndrome Trellis Codes [START_REF] Filler | Minimizing additive distortion in steganography using syndrome-trellis codes[END_REF]. This chapter proposes to use the steganographic distortion (instead of L 1 , L 2 or L ∞ norms in adversarial literature) in order to decrease detectability.

Note that this distortion is additive, which is equivalent to consider that each pixel modification yields a detectability independent from the others. Yet, one strategy takes into account potential interactions between neighboring modifications. The image is first decomposed into disjoint lattices to be sequentially embedded where costs are then sequentially updated after the embedding over one lattice [START_REF] Li | A strategy of clustering modification directions in spatial image steganography[END_REF]. This work uses three families of steganographic costs. The first one, HILL [START_REF] Li | A new cost function for spatial image steganography[END_REF], is empirical and naive, but has nevertheless been widely used in steganography thanks to its simplicity. The cost map w associated to ±1 is computed using two low-pass averaging filters L 1 and L 2 of respective size 3 × 3 and 15 × 15 and one high pass filter H: ( * means convolution)

w = 1 |I * H| * L 1 * L 2 , with H =      -1 2 -1 2 -4 2 -1 2 -1      . ( 8.3)
The second one, derived from MiPod [START_REF] Sedighi | Content-Adaptive Steganography by Minimizing Statistical Detectability[END_REF], assumes that the residual signal is distributed as N (0, σ 2 i ) for the original image, and N ( i , σ 2 i ) for the stego image. The variance σ 2 i is estimated on each pixel using Wiener filtering and a least square approximation on a basis of cosine functions. The cost is the log likelihood ratio between the two distributions evaluated at 0, i.e.:

w i ( i ) = 2 i /σ 2 i . ( 8.4)
Unlike HILL, this model handles modifications other than ±1.

The last one is a cost updating strategy favoring coherent modifications between pixels within a spatial or color neighborhood. It is called GINA [START_REF] Wang | Non-Additive Cost Functions for Color Image Steganography Based on Inter-Channel Correlations and Differences[END_REF] and it is derived from CMD [START_REF] Li | A strategy of clustering modification directions in spatial image steganography[END_REF]. It splits the color images into 4 disjoint lattices per channel, i.e. 12 lattices. The embedding performs sequentially starting by the green channel lattices. The costs on one lattice is updated according to the modifications done on the previous ones as:

w i ( i ) = 1 9 w i ( i ) , if sign( i ) = sign(µ i ), (8.5)
with µ i the average of the modifications already performed in the spatial or colour neighborhood of location i.

Looking at Adversarial Examples with Stega Glasses

First, note that adversarial images recently became a source of inspiration for steganography: paper [START_REF] Tang | CNN-Based Adversarial Embedding for Image Steganography[END_REF] proposes the concept of steganography with an adversarial embedding fooling a DNN-based steganalyzer. References [7] and [START_REF] Shi | CNN-based steganalysis and parametric adversarial embedding: a game-theoretic framework[END_REF] propose both to cast the problem of adversarial embedding as a game-theoretical problem. A protocol to train efficiently new adversaries and to generate less detectable stego contents using a min max strategy is presented in [7]. The reference [START_REF] Shi | CNN-based steganalysis and parametric adversarial embedding: a game-theoretic framework[END_REF] solves the game between one embedder and one steganalyst using both different levels of adversarial perturbations.

Paper [START_REF] Schöttle | Detecting Adversarial Examples -a Lesson from Multimedia Security[END_REF] stresses however one fundamental difference between steganography and adverarial examples: Steganalysis has two classes, where the class 'cover' distribution is given by Nature, whereas the class 'stego' distribution is a consequence of designed embedding schemes. On the other hand, a perfect adversarial example and an original image are distributed as by the class ĉ(x a ) or c(x o ), which are both given by Nature.

We stress another major difference: Steganographic embedding is essentially a stochastic process. Two stego-contents derived from the same cover are different almost surely with STC [START_REF] Filler | Minimizing additive distortion in steganography using syndrome-trellis codes[END_REF]. This is a mean to encompass the randomness of the messages to be embedded. This is also the reason why steganographic embedders turns the costs (w i ( )) into probabilities (π i ( )) of modifying the i-th pixel by quantum. These probabilities are derived to minimize the detectability under the constraint of an embedding rate given by the source coding theorem:

R = - 1 n i i π i ( i ) log 2 (π i ( i )) bits. (8.6)
In contrast, an attack is a deterministic process always giving the same adversarial version of one original image. Adversarial imaging does not need these probabilities.

Steganographic Post-Processing

This section presents the use of steganography in our post-processing Q mounted on top of any adversarial attack.

Optimal post-processing

Starting from an original image, we assume that an attack has produced x a . The problem is that x a ∈ [0, 255] n , i.e. its pixel values are a priori not quantized. Our postprocessing specifically deals with that matter, outputting x q = Q(x a ) ∈ {0, . . . , 255} n . In this chapter, we call the quantized1 perturbation:

u := x a -x 0 ∈ R n , ( 8.7)
:= x q -x 0 ∈ Z n . ( 8.8)
The design of Q amounts to finding a good . This is more complex than just rounding perturbation u.

We first restrict the range of . We define the degree of freedom d f as the number of possible values for each i , 1 ≤ i ≤ n. This is an even integer greater than or equal to 2. The range of i is centered around u i . For instance, when d f = 2, i ∈ { u i , u i }. In general, the range is given by

L i := { u i -d f /2, . . . , u i -1, u i , . . . , u i + d f /2 -1}. ( 8.9)
Over the whole image, there are d n f possible sequences for . We now define two quantities depending on . The classifier loss at x q = x a -u + :

L( ) := ŷc 0 (x a -u + ) -ŷca (x a -u + ), (8.10)
where c 0 is the ground truth class of x o and c a is the predicted class after the attack. When the attack succeeds, it means that x a is classified as c a = c o because ŷca (x a ) > ŷc 0 (x a ) so that L(u) < 0. Our post-processing cares about maintaining this adversariality. This constrains s.t. L( ) < 0.

The second quantity is the detectability. We assume that a black-box algorithm gives the stego-costs (w i ( )) for a given original image. The overall detectability of x q is gauged by D( ) as given by (8.2). In the end, the optimal post-processing Q minimizes detectability while maintaining adversariality: = arg min :L( )<0 D( ).

(8.11)

Our proposal

The complexity for finding the solution of (8.11) a priori scales as O(d n f ). Two ideas from the adversarial examples literature help reducing this cost. First, the problem is stated as an Lagrangian formulation as in [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF]:

λ = arg min D( ) + λL( ). (8.12)
where λ ≥ 0 is the Lagrangian multiplier. This means that we must solve this problem for any λ and then find the smallest value of λ s.t. L( λ ) < 0.

Second, the classifier loss is linearized around x a , i.e. for around u: L( ) ≈ L(u) + ( -u) g, where g = ∇L(u). This transforms problem (8.12) into λ = arg min n i=1 w i ( i ) + λ(u ii ).g i .

(8.13)

The solution is now tractable because the functional is separable: we can solve the problem pixel-wise. The algorithm stores in d f × n matrix W the costs, and in d f × n matrix G the values ((u ii ).g i ) i for i ∈ L i (8.9). For a given λ, it computes W + λG and looks for the minimum of each column 1 ≤ i ≤ n. In other words, it is as complex as n minimum findings, each over d f values, which scales as O(n log d f ).

Note that for λ = 0, Q quantizes x a,i 'towards' x 0,i to minimize detectability. Indeed,

if i = 0 is admissible (0 ∈ L i holds if |u i | ≤ d f /2), then Q(x a,i ) = x 0,i at λ = 0.
On top of solving (8.13), a line search over λ is required. The linearization of the loss being a crude approximation, we make calls to the network to check that Q(x a ) is adversarial: When testing a given value of λ, λ is computed to produce x q that feeds the classifier. If x q is adversarial then L( λ ) < 0 and we test a lower value of λ (giving more importance to the detectability), otherwise we increase it. The search is performed

Simplification for quadratic stego-costs

We now assume that the stego-costs obey to the following expression: w i ( ) = 2 /σ 2 i as in (8.4). This makes the functional of (8.13) (restricted to the i-th pixel) equals to

2 i /σ 2 i -λg i i + λp i which minimizer is ˜ i = λg i σ 2 i /2.
Yet, this value in general is not an integer belonging to L i (8.9). This issue is easily solved because a quadratic function is symmetric around its minimum, therefore the minimum over L i is its value closest to ˜ i as shown in Fig. 8.2. The range L i being nothing more than a set of consecutive integers, we obtain a closed form expression:

λ,i = min(max([λg i σ 2 i /2], u i -d f /2), u i + d f /2 -1), (8.14)
where [•] is the rounding to the nearest integer. The post-processing has now a linear complexity.

In this equation, the min and max operate a clipping so that λ,i belongs to

L i . This clipping is active if ˜ i / ∈ L i , which happens if λ ≥ λi with λi :=        2 p i -d f g i σ 2 i + if g i < 0 2 p i +d f -2 g i σ 2 i + if g i > 0, (8.15)
where |a| + = a if a > 0, 0 otherwise. This remark is important because it shows that for any λ > max i λi , the solution λ of (8.14) remains the same due to clipping. Therefore, we can narrow down the line search of λ to [0, max i λi ].

Experimental Investigation

Experimental setup

Our experimental work uses 18,000 images from ImageNet of dimension 224×224×3. This subset is split in 1,000 for testing and comparing, 17,000 for training. An image is attacked only if the classifier predicts its correct label beforehand. This happens with probability equaling the accuracy of the network Acc. We measure L 2 the average Euclidean distance of the perturbation and P suc the probability of a successful attack only over correctly labeled images.

We attack the networks with 4 different attacks: FGSM [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF], PGD 2 [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF], CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] and DDN [START_REF] Rony | Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses[END_REF]. All these attacks are run in a best-effort fashion with a complexity limited to 100 iterations. This means that for FGSM and PGD 2 the distortion is gradually increased until the image is adversarial. For more complex CW and DDN attacks, different parameters are used over a total maximum of 100 iterations. The final attacked version is the adversarial image with the smaller distortion. Since DDN is the only attack that creates integer images, the other 3 are post-processed either by the enhanced quantization [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF], which is our baseline, or by our method explained in Sect. 8.3.2.

The adversarial image detectors are evaluated by the true positive rate TPR 5 when the false positive rate FPR is fixed to 5%.

Robustness of recent classifiers: there is free lunch

Our first experiment compares the robustness of the famous ResNet-50 network to the recent classifiers: the vanilla version of EfficientNet-b0 [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF] and its robust version trained with AdvProp [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF]. Note that the authors of [START_REF] Xie | Adversarial Examples Improve Image Recognition[END_REF] apply adversarial re-training for improving accuracy. As far as we known, the robustness of this version was not yet established.

Figure 8.3 shows the same characteristic function as in Fig. 8.1 with this time the vanilla EfficientNet-b0 against its robust version. Table 8.1 gives measurements P suc and L 2 as a summary of the characteristic function shown in Fig 8 .1. This confirms that modern classifiers are more accurate and more robust (lower P suc and/or bigger L 2 ). This is indeed a surprise: It pulls down the myth of 'No Free Lunch' in adversarial machine learning literature [START_REF] Dohmatob | Generalized No Free Lunch Theorem for Adversarial Robustness[END_REF][START_REF] Tsipras | Robustness May Be at Odds with Accuracy[END_REF] (the price to pay for robustifying a network is allegedly a lower accuracy). [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] (PGD in best effort with quantization) against Vanilla EfficientNet-b0 and its Robust counterpart. Distortion is not normalized for n.

Detection with forensics detectors

We use three steganalyzers to detect adversarial images. Their training set is composed of 15,651 pairs of original and adversarial images. The latter are crafted with best-effort FGSM against vanilla EfficientNet-b0.

The first detector is trained on SRM feature vectors [START_REF] Fridrich | Rich models for steganalysis of digital images[END_REF], with dimensions 34,671. SRM is a model that applies to only one channel. It is computed on the luminance of the image in our experimental work. The classifier separating these high-dimensional vectors into two classes is the linear regularized classifier [START_REF] Cogranne | Is ensemble classifier needed for steganalysis in highdimensional feature spaces?[END_REF]. The second detector is based on the color version of SRM: SCRMQ1 [START_REF] Goljan | Rich model for Steganalysis of color images[END_REF] with dimension 18,157. The classifier is the same. The third detector is SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF], one of the best detectors in steganalysis. Training is performed on 180 epochs: The first 100 with a learning rate of 10 -3 , the remaining 80 with 10 -4 . Data augmentation is also performed during training. First, there is a probability p 1 = 0.5 of mirroring the pair of images. Then, there is another probability p 2 = 0.5 of rotating them by 90 degrees.

The attacks: Table 8.2 shows the results of detection on all 4 attacks. PGD 2 achieves a high P suc at almost a third of the distortion FGSM would obtain. DDN and CW being harder to optimize attain both lower P suc and higher distortion under the given constraints. For the rest of the study we therefore focus on PGD 2 to give the best attacking setup with reasonable complexity.

The detectors: Table 8.2 gives also the TPR 5 associated to the detectors. Although [START_REF] Liu | Detection Based Defense Against Adversarial Examples From the Steganalysis Point of View[END_REF] achieves good performances with SRM, we do not obtain the high detection rates reported in the reference. This cab be due to both finer attacks (best effort mode) and quantization. Our results show also that the detectors generalize well: although trained to detect images highly distorted by FGSM, they can detect as well and sometimes even better more subtle attacks like CW. Moreover, SRNet always outperforms SCRMQ1 and is the most accurate of the three detectors. From Tab. 8.2, we can also deduce that PGD 2 + [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] is the worstcase scenario for defense. The probability of fooling both the classifier EfficientNet-b0 and the detector SRNet in this setup combines to only 0.88 × (1 -0.933) = 5.9%.

Post-processing with a Steganographic Embedder

We now play the role of the attacker. We use PGD 2 with best effort as the base attack to compare the detectability of four post-processings: The non-steganographic insertion [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] as a baseline, HILL (8.3), MiPod (8.4), and GINA (8.5). GINA uses the quadratic method explained in Sect. 8.3.3 sequentially over the 12 lattices. Quadratic stego-costs are updated with CMD strategy (8.5). Each lattice contributes to a 1/12 of the initial classification loss.

Table 8.3 illustrates how each strategy is detected by either SCRMQ1 or SRNet. Both detectors are trained on FGSM with [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] quantization as 'stego' images crafted on their respective network. Distortion increases with each method and along the degree of freedom d f . The use of Steganographic costs therefore enables to reduce the detectability while increasing the L 2 distortion.

From the attacker perspective, the best strategy to fool the detector PGD 2 is GINA costs with d f = 4. This scenario now has 48.0% chance of fooling both Vanilla EfficientNet-b0 and SRNet and 80.4% with SCRMQ1 as the detector. Fig. 8.4 shows the two examples with highest distortion on EfficientNet-b0 that still fool SRNet. The added distortion remains imperceptible to the human eye even in these cases.

The conclusion on Robust EfficientNet-b0 is however different. Since the distortion needed to attack the network is higher, it is consequently expected that the detectors will be more accurate. If SCRMQ1 detects GINA distortion slightly better than on Vanilla EfficientNet-b0, SRNet is however very efficient to detect each strategy even if it was trained on FGSM.

Training on adversarial images with GINA costs

We finally play the role of the defender again. We want to detect GINA perturbation with the highest possible TPR. To achieve this we retrain our detectors in the same setups as before, but with images using GINA perturbation as adversarial images. Since Tab. 8.3 shows that in most cases d f = 4 is indeed the worst-case for the defense side, we attacked the training set of "cover" images with PGD 2 and GINA costs with d f = 4.

The first result we report is that under the same setup, SRNet was never able to distinct both distributions of images. The average confidence on the whole test set is roughly 50%. Trying to train SRNet with a finer learning rate did not lead to any better result. There is probably a set of hyperparameters that would lead to a more effective Table 8.4 shows TPR 5 for SCRMQ1 under such training setup. The detector is able to detect GINA mechanism at a higher rate than in Tab. 8.3 but generalizes poorly on other attacks. A conclusion to this final experiment is that GINA can be stealthy to general detectors, but it is still better detected after another iteration of the defender. The detection accuracy is however lower when using GINA costs, and drops from 83.1% to 68.5%. The price of detecting GINA is also to become more specific and to lose performance on the other attacks.

Chapter Conclusion

This chapter explores both sides of adversarial image detection with steganographic glasses.

On the Attack side, our work using distortions designed for steganographic purposes is able to reduce the detection rates. Steganographic distortion target specific regions and pixels of an image to quantize the attack. The L 2 distortion increases w.r.t. the original attack, but remains imperceptible by the human eye (Fig. 8.4) and less detectable by a targeted detector. This paper consequently shows the possibility of tweaking an attack to make it harder to detect while remaining invisible.

On the Defense side, we use SRNet [START_REF] Boroumand | Deep residual network for steganalysis of digital images[END_REF], state-of-the-art in steganalysis to detect adversarial images. Training it on images attacked with the basic FGSM shows excellent performance. Detection also generalizes well even on the finest attacks such as PGD 2 [START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF] and CW [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF].

Finally both Attack and Defense are affected by the considered neural network. The effect of adversarial training on EfficientNet-b0 [Xie:2019aa] is twofold: it increases the classification accuracy as well as robustifying the network. An increased robustness translates into a higher attacking distortion, which itself translates into a higher detectability.

Introduction

The topic of adversarial examples has recently attracted a huge interest with more than 3,000 papers published in the last four years. As far as image classification is concerned, most articles consider toy datasets like MNIST composed of 28 × 28 images (10 classes) and CIFAR 32 × 32 images (10 or 100 classes). These are far from nowadays typical image sizes. Few papers deal with the more realistic ImageNet dataset composed of large images (1000 classes). Yet, classifiers usually process these images once downscaled to 224 × 224 or 256 × 256. This is still about the size of thumbnails on Internet websites, so not yet a modern image. Indeed, no work uses the original ImageNet with larger images. This chapter considers forging realistic, i.e. large, adversarial images under a white-box setup for two scenarios:

A. The image classifier can natively process large images. This means that the model underneath is a wide and deep neural network.

B. The image classifier first downscales the images to a smaller size, say 224 × 224, and then uses a neural network to make a prediction.

Scenario A questions how the energy of the adversarial perturbation evolves with the size of the input data. On one hand, the classifier gets more pixels to make a decision which stems into a higher accuracy on natural images. On the other hand, the attacker has more degrees of freedom to delude the classifier. This chapter shows experimental evidence that a bigger size indeed benefits more to the attacker (under some conditions).

In scenario B, the downscaling is part of the classifier box. The small image is thus an internal data that the attacker cannot have access to: The goal is to forge an adversarial version of the large image and not its downscaled version. White-box attacks rely on computing the gradient of a loss function w.r.t. a neural network using back-propagation. The difficulty is therefore how to propagate further this information through the downscaling back to the space of large images. Another point of interest is the choice of downscaling method for the defender and whether its knowledge is key to forge adversarial examples for the attacker.

In the end, we also compare the distortion of perturbations that delude a classifier according to scenario A or B. This yields the best practice for the defender.

Related Works

Several theoretical works study how the dimension n of the inputs makes the forgery of adversarial examples easier. However, they use different arguments and terminology. It is difficult to verify that they evidence the same phenomenon.

We denote by n the dimension of the input and call by input deviation its natural standard deviation σ X (n). If the input x ∈ R n is a centered random vector, it is typically measured by E[ x 2 ]/n where • is the Euclidean distance. In the same way, we measure the adversarial distortion σ A (n) by the typical value of E[ u 2 ]/n of the adversarial perturbation u for signals of dimension n.

The early attempt [START_REF] Gilmer | Adversarial Spheres[END_REF] considers a synthetic example where data points are uniformly distributed over spheres of constant radius R: the input deviation σ X (n) obviously scales as R/ √ n. Paper [START_REF] Gilmer | Adversarial Spheres[END_REF] then proves that the 2 norm of the adversarial perturbation in expectation scales as O( 1 / √ n) for a fixed classification accuracy. This translates into an adversarial distortion σ A (n) scaling as O(1/n).

Papers [START_REF] Dohmatob | Generalized No Free Lunch Theorem for Adversarial Robustness[END_REF][START_REF] Fawzi | Adversarial vulnerability for any classifier[END_REF] generalizes this result to many data distributions verifying the Talagrand W 2 transportation-cost inequality. They state that the 2 norm of the perturbation scales as the natural "intra-class noise level" σ X (for a fixed accuracy). For instance, if

x ∼ N (µ k , σ 2 X I n ) for class k, then the 2 norm of the attack is proportional to the power of x, which remains constant as n increases (see [START_REF] Dohmatob | Generalized No Free Lunch Theorem for Adversarial Robustness[END_REF]Sec. 2.5.2]).

We propose to summarize this literature by the following rule of thumb:

σ A (n) ∝ σ X (n)/ √ n. ( 9.1)
More precisely, [START_REF] Dohmatob | Generalized No Free Lunch Theorem for Adversarial Robustness[END_REF] shows that the proportional constant is upper bounded by κ

(n) = -2 log(1 -η(n)) + π /2
, where η(n) is the accuracy of the classifier.

How do these theoretical results transfer to images? For square color images of size , there are n = 3 2 pixel values. One argument is that the diameter of the hypercube [0, 255] n grows as 255 √ n, which reflects the typical distance between inputs, hence a constant input deviation. In the same way, up or downscaling does not change the histogram of the pixel values, yielding a constant standard deviation σ X (n). Plugging this hand-waving justification into (9.1) yields an adversarial distortion scaling as κ(n)/ √ n.

On the contrary, paper [START_REF] Dube | High Dimensional Spaces, Deep Learning and Adversarial Examples[END_REF] claims that the expectation of the 2 norm of the perturbation vanishes as O( 1 / √ n) assuming images are 1 /f 2 power spectrum processes. This would make σ A (d) = O( 1 /n) contradicting the previous paragraph.

No work provide clear experimental evidence. Paper [START_REF] Chattopadhyay | Curse of Dimensionality in Adversarial Examples[END_REF] investigates this topic on simple datasets MNIST and CIFAR. By upscaling to higher resolutions, it finds that network accuracy and attack efficiency remain the same whatever the dimension. This is not convincing because upscaling does not create any new information.

In the end, this section shows that there is no clear report about adversarial example as the dimension of the inputs grows as far as realistic image classification is concerned.

Problem formulation

This section proposes to include the downscaling inside the neural network, i.e. this is scenario B in Sect. 9.1. It outlines that running an adversarial attack in the large image space or the small image space does not produce the same effect.

Including the downscaler inside the network

The classifier is a neural network composed of several layers. Each layer applies a linear transformation (be it a convolution or a fully connected layer) followed the nonlinear activation function φ(.).

a k = φ(z k ), (9.2)
z k = W k a k-1 + b k , ( 9.3)
with a k the output of the k-th layer ∀k s.t. 1 ≤ k ≤ K. The final vector a K is the predicted logit per class (usually the last layer has no non-linearity). Since the downscaling is also a linear function, it can be incorporated inside the network as the layer 0 without activation: a 0 := x = DX, where X is the large L × L input image, and D the 3 2 × 3L 3 matrix downscaling to a smaller × image x.

For instance, if = L /2, one pixel of x o is interpolated from 4 pixels of X o , and under a proper flattening we have: where {δ i } are the positive weights of the downscaling kernel (summing up to 1). Another case is the nearest neighbor interpolation where there is single 1 in each row of D, e.g.

D =      δ 1 δ 2 δ 3 δ 4 0 0 0 0 . . . 0 0 0 0 δ 1 δ 2 δ 3 δ 4 .
δ 1 = 1, δ i = 0 for i = 1.

Attacking in the large or small image space

In an untargeted white-box setup, the attacker usually defines the loss L

(X) = a K (c o ) -max c =co a K (k)
, where c o is the ground-truth label of image X o . The attack aims at finding a perturbation u s.t. L(X o + u) is negative and u is small. White box attacks use the gradient of this loss or its first term:

∇ X L(X) := ( dL(X) /dX) = D g with (9.5)

g := W 1 φ 1 . . . W K-1 φ K-1 W K-1 ∇L(a K )
With φ k a shortcut for φ (a k ). In other words, by back-propagating through the downscaler, the gradient in the large image space is nothing more than the gradient g in the small image space through the matrix D .

The attack usually computes

X o -∇ X L(X o ) = X o -D g.
The downscaling maps this to x o -DD g. Yet, the same attack in the small input space would give x o -g.

With the nearest neighbor interpolation, DD = I and there is no difference. If L is a multiple of as in (9.4), DD = ( i δ 2 i ) I , with i δ 2 i ≤ 1. There is a loss of energy but the downscaled perturbation stays colinear with g.

If L is not a multiple of or if the downscaler is not the nearest neighbor interpolation, there is also a misalignment because DD is not proportional to identity matrix I . This is especially true when several pixels of the small image depend on the same pixel of the large image, e.g. due to anti-aliasing. This shows that the downscaling has an impact on the forgery of adversarial images.

Experimental Works

Models, data, and attacks

We experiment with four families of classifiers: EfficientNet [START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF] and its Lite implementation, EfficientNet-V2 [START_REF] Tan | EfficientNetV2: Smaller Models and Faster Training[END_REF], and NFNet family [START_REF] Brock | High-performance large-scale image recognition without normalization[END_REF]. One family is composed of several models tackling × color images with, for instance, ranging from 224 to 600 for EfficientNet, or from 256 to 576 for NFNet (see Fig. 9.1). Family members share the same architecture but with wider activation maps and are trained with the same procedure. This is important for a fair comparison. We created a subset of 1,000 images from the validation set of Imagenet 2012. Each image is the first occurrence of a class within the original dataset. Each image is first center-cropped and resized to L × L with L = 600 and bilinear interpolation (default on opencv library). This is the image size natively processed by EfficientNet-b7, the largest size used by the CNN of our experimental work.

For scenario B, we use four different downscaling methods: Nearest, Bilinear, Bicubic and Area. The first three methods interpolate one pixel from the 1, 4, or 16 (resp.) closest pixels. Area performs an average pooling on the image. Paper [START_REF] Parmar | On Buggy Resizing Libraries and Surprising Subtleties in FID Calculation[END_REF] highlights the lack of antialiasing in PyTorch implementation except for method Area. We implement two antialiasing methods: average and Gaussian. Both use a kernel size k size = /L . Gaussian values are generated with a standard deviation σ = 1.6 × k size as inspired by SIFT scale space construction [START_REF] David G Lowe | Distinctive image features from scale-invariant keypoints[END_REF]. Figure 9.1 shows the accuracies of all models and over all downscalers (except when = L). Our observations are twofold: the accuracy increases with the model size ; and the downscaling method has little impact.

We choose the white-box attack BP [START_REF] Zhang | Walking on the Edge: Fast, Low-Distortion Adversarial Examples[END_REF] in its best-effort [START_REF] Maho | RoBIC: A benchmark suite for assessing classifiers robustness[END_REF] implementation because of its high probability of success, low distortion, and speed. Distortion is measured as the Root Mean Squared Error: d(x a , x 0 ) = x a -x 0 / √ n where x 0 (resp. x a ) is the original (resp. adversarial) image. For scenario B, we implement downscaling as a first layer in order to back-propagate the gradient as detailed in Sect. 9.3.

Scenario A: comparison with theoretical results

We measure the adversarial distortion for different image sizes by downscaling the baseline images to the input size of the model with bilinear interpolation. The attacker modifies these × images in this scenario. According to section 9.2 Eq. (9.1), we measure the normalized adversarial distortion σ A (n)/κ(n) as a function of n = 3 2 . Figure 9.2 plots this function in logarithmic scale. These experimental results partially confirm the theory found in literature. For three families, in the range ∈ [224, 416], this gives almost a trend line whose slope is ≈ -1 /2. This gives more credit to papers [START_REF] Dohmatob | Generalized No Free Lunch Theorem for Adversarial Robustness[END_REF][START_REF] Fawzi | Adversarial vulnerability for any classifier[END_REF][START_REF] Gilmer | Adversarial Spheres[END_REF] rather than [START_REF] Dube | High Dimensional Spaces, Deep Learning and Adversarial Examples[END_REF] predicting -1. Yet, in the range ∈ [456, 600] the adversarial deviation is stable or even increasing, and the family EfficientNet-Lite completely violates the theory.

Scenario B: attacking through downscaling

.3 shows the success rate of the attack logically increases with distortion. As Sect. 9.3.2 suspected, there is a noticeable hierarchy in the downscaling methods, especially on smaller sizes like = 256 for NFNet-F0. With Nearest, any pixel on the smaller image matches exactly a pixel on the bigger one. The perturbation signal is crafted on these pixels and entirely goes through the downscaling. In every other method, the adversarial signal is diluted through the back-propagation on neighboring pixels. This creates a counter effect detrimental to the attack. Nearest is thus easier to attack because it requires less distortion. Conversely, methods with anti-aliasing such as Area are the best options as a defense.

The impact of the downscaling decreases as the network takes a bigger input size. This results in narrower differences in the plotted curves for = 512 with NFNet-F4 (Fig. 9.3).

We observe the same behavior with the other network families. For instance, with Efficient-Net, Tab. 9.1 second column shows that the attack distortion goes up as increases for Nearest, whereas Area is more robust when downscaling a lot, e.g. down to b0 ( = 224).

Scenario B: transferability

This section assumes that the attacker does not know which downscaling method is used. The attack targets a specific interpolation and is tested through another one. Table 9.2 shows these results when downscaling from = 600 to = 224.

Transferability of the attack is poor except for Bilinear (attack) displaying good transferability over Bicubic (defense). Our explanation is that BP adds very small perturbations. Adversarial examples thus lie right behind the frontier of the ground-truth class for

Scenario B: ensemble model

In order to achieve better transferability, the attacker fights against an ensemble of classifiers. For a given model (e.g. EfficientNet-b0), we gather all downscaling methods in an ensemble of classifiers. This gives birth to as many gradients, which need to be aggregated into a single one to be exploited by BP attack. We explore two options:

1. A basic averaging of the gradients over all the ensemble, also referred to as EOT [START_REF] Wang | Protecting neural networks with hierarchical random switching: Towards better robustness-accuracy trade-off for stochastic defenses[END_REF].

2. A worst-case gradient selection, a trick inspired by Deepfool [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF].

For a given classifier / downscaler, the second method estimates the distance d adv an image is from the frontier as if the classifier were linear [START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF]:

d adv = L adv (x)
∇L adv (x) . The classifier with the biggest distance is deemed as the worst for the attacker, who then targets it in the next iteration of the attack. In other words, the aggregated gradient is simply the gradient of the worst classifier of the ensemble.

For both methods, an image is deemed adversarial if it deludes all the classifiers of the ensemble. For this experiment we use another subset of 100 randomly picked images from the validation set of Imagenet 2012 as a matter of computational time. Table 9.1 shows the distortion at which 90% of the images are successfully misclassified. Here are the lessons: Attacking an ensemble consumes more distortion than targeting a single classifier ; so guaranteed transferability is possible at the cost of a bigger distortion. There is no clear better option for attacking an ensemble. Distortion increases as the scale goes down. A better defense strategy is to run a model on small scale images and to randomly pick a downscaler over an ensemble at each inference.

Chapter Conclusion

In this chapter, we have studied the impact of downscaling when attacking large images. The best practice appears to be scenario B: downscaling to a small size with a wide downscaling kernel like 'Area' or 'Bil+Gauss'.

The transferability of an adversarial attack to other interpolation kernels is nearly insignificant when running optimized (low-distortion) attacks. This opens the door to a random resizing method as a means of defense. The attacker can however circumvent this issue by attacking an ensemble, if the pool of downscaling methods is disclosed. Nevertheless, this defense increases the distortion by ≈ 35% compared to the best pure strategy without sacrificing accuracy.

CONCLUSION AND PERSPECTIVES

Image classification is the main application of Computer Vision. Deep Learning-based classifiers quickly became the best option to tackle a such task. They were however found to be vulnerable to specific perturbations. This created the field of adversarial examples. In Chap. 4, we briefly introduce Deep Neural Networks, how they are trained, and how they are fooled. We then describe a few White-Box attacks that are used throughout the manuscript. We then study the existence of a such vulnerability in Chap. 5. Many culprits are identified. Adversarial vulnerability comes from the shared responsibility of data, architecture, and training phase. But eventually, we conclude that DNNs are vulnerable by design.

The purpose of this thesis is to study adversarial examples in a realistic context. We make many choices in this direction. First, we work on a classification task that we estimate to be realistic: the Imagenet classification task. This task is what resembles the most a real-world application. Adversarial examples should also be realistic images. We discuss in Chap. 7 how most works on adversarial examples do not craft images. Instead, they work in the floating-point domain and generate data that can be processed by an off-the-shelf DNN but cannot be saved as an image. Chapter 7 shows that quantization is not a trivial task, especially when considering a lossy image format such as JPEG.

Chapter 8 also aims at crafting spatial images. We consider both the perspective of a defender and the attacker. First, we build detection systems that could prevent a DNN from working on a corrupted image. Then, we modify our quantization method to follow steganographic strategies. We finally build detection systems to detect these new examples. This gives insight into the iterative attack-defense game. Each time a player moves, the other answers. We conclude that detection can be prevented in many cases, even after retraining on these stealthy images. One of our detectors is still able to perform a little. The other one could not even be trained successfully on stealthy adversarial examples.

Chapter 9 is also a contribution that studies both perspectives. We analyze the impact that image size can have on adversarial examples. Images from Imagenet are processed in different sizes that are DNN-specific. We show that models using larger images usually are more accurate. They are also larger in terms of parameters, which is a reason for this performance. However, the scaling of adversarial examples (distortion-wise) is almost entirely a function of the input dimension. And this does not act in favor of larger models. The "No Free Lunch theorem" is at stake and we can not draw a hard conclusion on what model to adopt. This is obviously very task-specific. Yet, smaller models have comparable performances and offer more robustness. This should be known when deploying a classifier. The second point in our conclusion is that resizing obviously affects adversarial examples. If the image will undergo a downscaling, the adversarial signal should remain effective. An evident aspect of resizing is that it should rely on antialiasing to dilute the signal. Attacks are downscaler-specific. Defenses could thus use a random resizing method to hinder the attack. However, we show that even a such strategy can be defeated using an ensemble model if the resizing options are known. But this comes at the cost of higher distortion.

Towards Fair Evaluation

We worked solely using ImageNet. Smaller datasets can be useful to understand adversarial examples or as a mere playground. We believe however that most works should be applied to Imagenet for fair comparisons. This is not the case. But even then, data is rarely the same. A common practice, that we ourselves did, is to pick random images from the validation set. Say 1,000. But there is no consensus on which images to pick. We had the example in 5.3.2 of the works of [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF] that randomly picked 5,000 images. Accuracy on these images is 100%. Should we still consider data random? It appears to be cherry-picked. A best practice would be the use of unified data such as Neurips Vision Challenge [START_REF] Kurakin | Adversarial attacks and defences competition[END_REF].

The other main aspect is the attack used and how it is used. Adversarial examples can be crafted along:

1. Distortion constraint: create a sample with a given distortion and evaluate whether it is indeed adversarial or not.

2. Success constraint: push the attack until the sample is adversarial.

3. Optimize: ensure adversariality at minimum distortion.

Each scenario can be pushed to the authors' best interest. For instance, the distortion constraint can be convenient for detection. If the signal always is of the same intensity and )) with the loss without rounding: L(q λ ) in (7.13) (resp. L(J -1 Qλ ) in (7.22)) and the loss with quantization L(q λ ) in (7.15) (resp. L(J -1 Q λ ) in (7. We call g t (x 0 ) natural teacher features (resp. g s (x 0 ) natural student features) and g s (x a ) adversarial student features. Note that we do not need g t (x a ) the adversarial teacher features. Adversarial examples are built from f s (x 0 ) as detailed below. In these experiments, we only the feature extraction half of the DNN. Both convolutional filters and BatchNorm are retrained. We identified BatchNorm to be partially responsible for adversarial vulnerabilities in Sect. 5.2.2.

An accurate student classifier yields g s (x 0 ) ≈ g t (x 0 ). Since the fully-connected layer is not trained, the best accuracy is a priori obtained when feature extraction is identical. A robust classifier yields g s (x a ) ≈ g s (x 0 ). A robust classifier should extract similar features out of an adversarial example. We introduce two losses: L nat := dist(g s (x 0 ), g t (x 0 )) and L rob := dist(g s (x a ), g s (x 0 )). Where dist is a metrics to be defined in Sect 9.7.2. Both are optimized throughout our retraining with L tot := L nat + L rob .

Building Adversarial Examples

Our training procedure is unsupervised. We can however attack images regardless of their ground-truth class c but w.r.t. their predicted class ĉ. Adversarial samples are also model-specific. Our student model evolves during its retraining. Adversarial samples should become more and more different between the student and the teacher model. They are strictly equal only at the first iteration.

During this first phase, we therefore build adversarial samples through f s (x). We use a PGD-like attack:

x (i+1) a = x (i) a - √ n N iter ∇L adv (x (i) a ) ∇L adv (x (i) a )
.

(9.6)

With:

L adv (x) := ŷc (x) -ŷa (x), (9.7)

Where c(x) is the argmax of ŷ and a(x) its second argmax. This attack ensures a final distortion of a maximum of d(x a , x 0 ) = . Note that in the case of N iter > 1, we set a(x) to remain the same as during the first iteration. This prevents the adversarial example from going back and forth between the same two classes.

Optimization

We explore several distances to achieve our robustification: Mean Squared Error (MSE), Mean Squared Logarithm Error (MSLE), and Cosine Correlation (CC).

Mean Squared Error

σ A (g(x a )) σ X (g(x 0 )) √ n f = σ A (x a ) σ X (x 0 )
√ n (9.12)

We call σ obj the scaled distortion such that:

σ obj (g(x a )) := σ X (g(x 0 ))

x a -x 0 2 x 0 2

√ n √ n f (9.13) This could be useful to reformulate L rob . In the case of a Root Mean Square Error (RMSE) optimization normalized for dimension, we could write:

L rob := max[σ A (g(x a )), σ obj (g(x a ))] (9.14)

We however did not obtain substantial gains using this scaling yet and we do not provide results using this scaling.

Experimental Work

Experimental Setup

We use MS-COCO unlabeled 2017 dataset. There 123, 000 images that we split in 120, 000 for training and 3, 000 for validation. Our justification to using this data is that it represents complex scenes and we hope to extract rich feature vectors.

Our test set is Neurips Adversarial Challenge 2017 [START_REF] Kurakin | Adversarial attacks and defences competition[END_REF] dataset. 1,000 images akin to Imagenet compose this set. We only studied robustification on Resnet-18. In the testing phase, we attack these images with BP in best-effort mode.

The only available comparison on Resnet-18 we found comes from Salman et al. [START_REF] Hadi | Do adversarially robust imagenet models transfer better?[END_REF] (Fig. 9.6). Out of the different available model weights, we pick two: = 0.05 because clean accuracy is barely affected, and = 0.5 because it is at the frontier of a notable accuracy drop. Both stem from vanilla adversarial training from scratch.

During the training phase, we reduce L tot = L nat + L rob with Adam optimizer on every convolution and BatchNorm of Resnet-18. The learning rate is set to 10 -4 . We use a batch size of 256 images. Training is performed over 10 epochs at which point we observe convergence.

Class Vectors

Before diving into experimental work, we would like to display an interesting result. In Sect. 5.2.3 we introduced Cω the symmetric matrix in R C×C such that Cω k,l = cos(ω k , ω l ). We now study the Cosine Correlation between Cω of different classifiers.

Eff.Net-b0 Res18 Res18 Table 9.3 display correlation for these matrices for different classifiers. We draw two observations. First, distances between class vectors seem to be mostly similar from one architecture to another. Second, Distances are even closer from one training to another on the same architecture. This observation led us to believe that most robustness is to be earned on feature extraction.

Iterative Attacks

In this last experiment, we observe the effect of an iterative adversarial generation on robustification. Figure 9.7 display results of this experiment. We surprisingly do not observe any gain in either accuracy or robustness with multi-step generation.

Discussion

Our work is currently still in progress. Experiments are iterative and we are hoping to improve our results further. We have however displayed interesting gains in robustness on DNNs. Moreover, it can be difficult to compare ourselves with the existing method. There are two reasons for that. The first is that our method proposes unsupervised retraining and the best works in model robustification train models from scratch. The use case is different since our method could be used to robustify DNNs trained on any task.

We identify that our data may not be the most adequate. We chose it to bring a variety of visual representations. Pseudo-classification used to create adversarial images is however heavily biased towards certain classes. Over 5% of pseudo-classification predict 562: 'fountain' and over 5% predict 807: 'solar dish, solar collector, solar furnace'. Label 611: 'jigsaw puzzle' even appears in over 10% of all predictions. This is far above an average of 0.1% that would represent fairly all classes. These images likely do not represent solar dishes but instead fall in improbable regions of classification. Data is too different from Imagenet distribution that the DNN was trained on. We wanted to have rich data, agnostic to Imagenet classification but we did not use the best data. Representation within MS-COCO is very different and not adequate. Such examples are seemingly normal images. A human understands what it represents as if it was not manipulated. But the attacked classifier will make an incorrect prediction. In this manuscript, we study the creation of such examples, the reason for their existence, and the underlying vulnerability of classifiers. In particular, we study these examples in a realistic context. First, attacks are optimized (high success rate and low distortion). Second, we add the constraint that adversarial examples should be images. We thus work on spatially-quantized (PNG) or DCT-quantized images (JPEG).

Figure 2 . 1 -

 21 Figure 2.1 -Un classifieur permet de trier des images en catégories appelées labels. Le classifieur connaît d'abord une étape d'entraînement durant laquelle il apprend à effectuer cette répartition.

  al. won the challenge with their Deep Neural Network

Figure 3 . 1 -

 31 Figure 3.1 -A classifier sorts images in categories called labels. The classifier underwent a training procedure during which it learned to recognize a set of classes.

Figure 4 . 1 -

 41 Figure 4.1 -Illustration of an artificial neuron.

Figure 4 . 2 -

 42 Figure 4.2 -Illustration of a Multi-Layer Perceptron. Output is C-dimensional. The size and number of hidden layers are parameters chosen when designing the model.

Figure 4 . 3 -

 43 Figure 4.3 -Illustration of the first convolution operation in a DNN, followed by a pooling. Size is decreased by a factor 2.

Figure 4 .

 4 4 shows an important property of these functions: their gradient is null outside a small range centered around 0. ReLU has two possible values of gradient: 0 for negative input and 1 for positive input. This simple function allowed Krizhevsky et al. to build a deeper CNN. As is further explained in Sect. 4.2.4, the gradient is at the heart of the training procedure. Having a small value of gradient until then caused the phenomenon of vanishing gradient. This is caused by the gradient of all the layers being

Figure 4 . 4 -

 44 Figure 4.4 -Comparison of different activation functions around 0. First and second curves show the impact of the factor a in the sigmoid function

Figure 4 . 6 -

 46 Figure 4.6 -Famous illustration of an adversarial example from the work of Goodfellow et al. [53]. The perturbation signal is voluntarily amplified to understand the addition (see the factor 0.007).

Figure 4 . 7 -

 47 Figure 4.7 -Similarity and difference between an evasion attack and training phase.

-

  Data: DNNs are vulnerable to adversarial examples because the data it was fed during the training phase is incomplete or misleading. -Training Phase: The DNN is not taught correctly how to process images or given too much freedom. -DNNs themselves: adversarial examples are simply a feature of Deep Learning.

Figure 5 . 2 -

 52 Figure 5.2 -Image A: 182:'Border Terrier', Image B: 184:'Irish Terrier', Image C: 185:'Norwich Terrier', Image D: 186:'Norfolk Terrier', Image E: 187:'Yorkshire Terrier', Image F: 189:'Lakeland Terrier'.

Figure 5 . 3 -

 53 Figure 5.3 -We extracted images from the validation set for two classes. Top row: images of class 657:'missile'. Bottom row: images of class 744:'projectile, missile'.

5

 5

  .4.

Figure 5 . 4 -

 54 Figure 5.4 -Figure 1 from [9] that helps visualize different types of redundancies within ImageNet pictures. Images highlighted with a green box are kept for an improved dataset while the others are deemed redundant. They are discarded.

Figure 5 . 6 -

 56 Figure 5.6 -Looking through images of 477:"carpenter's kit, tool kit" reveals an logical abundance of 784:'screwdriver'.

Figure 5 . 8 -

 58 Figure 5.8 -Figure 1 of [71] showing visually different images that share the same logits (displayed above) through a ResNet152. The first image is unmodified.

Figure 5 . 9 -

 59 Figure 5.9 -Illustration of class projection on a 2-label problem. Dashed lines illustrate the scalar product of f l-1 (x) on each class vector. The natural image x has a greater logit score on class 1. The modified adversarial image x a has been modified so it crosses class frontier. It is now classified as label 2.

Figure 5 . 10 -

 510 Figure 5.10 -Experiments of [58] over Imagenet validation dataset (50,000 images). y-axis is the accuracy, x-axis the L 2 -dissimilarity: x 0 -xa x 0 . Resnet-50 is attacked with Deepfool (left) and C&W (right).

Figure 5 .

 5 Figure 5.12 -Illustration of training experimented by Ilyas et al. [68]. A robust dataset is used on the diagram (a) and a non-robust dataset is used on diagram (b).

Figure 5 .

 5 Figure 5.13 -2D projection of feature vectors of natural images, extracted from [105].Comparison is given between their method and classical correlation and Softmax training. The projection of natural images and their adversarial counterpart is also displayed.

Figure 6 . 1 -

 61 Figure 6.1 -Operating curves of EfficientNet-b0 (left) and ResNet-50 (right) against four attacks in best-effort mode and without quantization.

Figure 6 . 2 -

 62 Figure 6.2 -Evolution of d1 /2 with the complexity budget for black box setup. Attacks on EfficientNet[START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF]

Figure 6 . 3 -

 63 Figure 6.3 -Evolution of d1 /2 with the complexity budget for white box setup. Attacks on EfficientNet[START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF]

Figure 6 . 4 -

 64 Figure 6.4 -Black-box d1 /2 as a function of white-box d1 /2 .

Figure 7 . 1 -Figure 7 . 2 -

 7172 Figure 7.1 -Operating curves of EfficientNet-b0 against FGSM and PGD in best-effort mode with floating point (plain) or quantized (dotted) pixel values.

Figure 7 .

 7 Figure 7.3 shows the behavior of the adversarial loss and the distortion as functions of λ, for one image in the spatial and the JPEG domains. To plot these curves we use the quantized (7.15) (resp. (7.22) for JPEG) and unquantized solution (7.23) (resp. (7.13)). The same behavior is observed on other images through other classifiers up to a change of ranges of values.

Figure 7 . 4 -

 74 Figure 7.4 -The impact of the number of tested values of λ on the operating curve. EfficientNet, BP, d f = 1, spatial (left) and JPEG90 (right).

Figure 7 . 5 -

 75 Figure 7.5 -Visible artifacts on images quantized as JPEG60. Predicted class is displayed in red, ground-truth in blue.

Figure 7 . 8 -

 78 Figure7.8 -The operating curve of Efficientnet-b0 against BP + JPEG quantization with d f = 1 (plain) and against JPEG compression alone (dot). For the sake of comparison, the cumulative distribution function of the distortion due to JPEG compression is also displayed (dashed). Distortion is measured from the original spatial image x o .

Figure 7 .

 7 Figure 7.12 shows two images misclassified by two different classifiers. It is interesting

Figure 7 . 9 -Figure 7 . 10 -

 79710 Figure 7.9 -Visual artifacts for two adversarial images on JPEG90 (top) and JPEG60 (bottom) with and without clipping. Visual artifacts on JPEG60 are due to the compression factor. Distortion is naturally high. JPEG 90 however represents low compression. Notice the non-smooth sky around the flamingo. Effect is especially visible for d f = ∞.

Figure 7 . 11 -Figure 7 . 12 -

 711712 Figure 7.11 -The operating curves of several classifiers against BP (plain) and PGD 2 (dotted) with quantization in spatial (left) and JPEG90 (right) domains.

Figure 7 . 13 -

 713 Figure 7.13 -Examples of attacked images with spatial and JPEG75 quantizations. JPEG75 compression of the original image is also displayed in the third column.

Figure 8 . 1 -

 81 Figure 8.1 -Characteristic function of quantized attack (PGD in best-effort) against well known (vanilla) classifiers for ImageNet.

Figure 8 . 2 -

 82 Figure 8.2 -Rounding the minimizer when the stego-cost is quadratic.

Figure 8 . 3 -

 83 Figure 8.3 -Characteristic function of attack[START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] (PGD in best effort with quantization) against Vanilla EfficientNet-b0 and its Robust counterpart. Distortion is not normalized for n.

'Figure 8 . 4 -

 84 Figure 8.4 -Top row: Cover images with their label below. Bottom row: adversarial images with steganographic embedding GINA (d f =4). Below them are their new label and the distortion

Figure 9 . 1 -

 91 Figure 9.1 -Accuracies with downscaling. Stripes define max and min accuracies over the 6 downscalers for a given model. Numbers are the sizes of the downscaled images, n = 3 2 .

Figure 9 . 2 -

 92 Figure 9.2 -Experimental measurement of the normalized adversarial distortion σ A (n)/κ(n) as a function of log 10 (n).

Figure 9 . 3 -

 93 Figure 9.3 -Attacking NFNet-F0 (left, = 256) and NFNet-F4 (right, = 512) through every downscaling method with the attack BP.

  Tensor natural image x a = x 0 + u Tensor adversarial image u Perturbation (adversarial signal) X Distribution of natural images D Distribution of tensor natural images l = f (x, θ) Logits output ŷ = f (x, θ) Class probability output y ∈ {0, 1} C One-hot ground-truth vector n Number of pixels C Number of classes c = argmax k y Ground-truth class ĉ = argmax k ŷ Predicted class

Figure 9 . 4 -

 94 Figure 9.4 -Illustration of our training procedure. Features are extracted from a natural image x 0 . An adversarial example of this image x a is built in parallel. Features of x a are then extracted.

3 -

 3 Correlations of Cω matrices from different classifiers. Three architectures are studied: EfficientNet-b0, Resnet18, and Resnet50. Three robustification methods are studied: 1 [129]( = 0.05), 2 [129]( = 0.5), and 3 [98]( = 3.0). Results are symmetrical, only half is displayed for clarity.

Figure 9 . 7 -

 97 Figure 9.7 -Effect of the number of steps N iter used during the attack. The attack is performed with = 1 and the training procedure is optimized with MSE

Titre:

  Comprendre, Apprivoiser, et se Protéger des Exemples Adversaires Mot clés : Réseaux de Neurones Artificiels, Exemples Adversaires Résumé : L'Intelligence Artificielle est une discipline qui a connu un fort essor au cours de ces dernières années, notamment en Vision par Ordinateur où l'application la plus commune est la classification d'image. Aujourd'hui, les réseaux de neurones artificiels profonds sont d'excellents classifieurs inférant ce que représente une image. Des travaux ont cependant rapidement montré qu'ils sont vulnérables aux attaques par évasion, aussi appelés les exemples adverses. Ces exemples sont des images qui pour un humain semblent être une représentation normale d'un objet. Mais le classifieur attaqué ne parviendra pas à prédire correctement ce qu'elles représentent. Cette thèse étudie les mécanismes de création de ces exemples, la raison de leur existence et la vulnérabilité des classifieurs. En particulier, ce travail replace ces exemples adverses dans un contexte réaliste. Premièrement, il propose des attaques rapides même sur des grandes images avec un fort taux de succès et une distortion imperceptible ou indétectable. Deuxièmement, il ajoute la contrainte que les exemples adversaires sont avant tout des images, c'est à dire des signaux quantifiés dans le domaine spatial (format PNG) ou dans le domaine DCT (format JPEG). Title: Understanding, Taming, and Defending from Adversarial Examples Keywords: Deep Neural Networks, Adversarial Examples Abstract: Artificial Intelligence is nowadays one of the most essential disciplines of computer science. These algorithms perform particularly well on Computer Vision tasks, especially classification. A classifier infers what an image represents. Nowadays Deep Neural Networks are largely used for these problems. These neural networks first undergo a training phase during which they are given many examples. These images are accompanied by labels: information on what the image represents. However, it was quickly found that the same logic used during the training phase could be used maliciously. This is the creation of Adversarial Examples through an Evasion Attack.

Table 5 .

 5

		Size Image Count Classes Count
	MNIST	28 × 28	60, 000	10
	CIFAR-10	32 × 32	60, 000	10
	CIFAR-100	32 × 32	60, 000	100
	Tiny-ImageNet	64 × 64	100, 000	200
	MS-COCO	> 224 × 224	330, 000	80 + 91
	ImageNet	> 224 × 224	> 14M	1, 000
	ImageNet-21k > 224 × 224	> 14M	21, 841

1 -Most popular image classification datasets. MS-COCO has 80 object categories and 91 'other' categories such as background, sky, ground...

Table 5 .

 5 C}. Feature vector is indeed obtained after a ReLu activation, which makes all its values ≥ 0. There 2 -Class vectors in the fully-connected layer of popular models. n f is the input or feature dimension.is no such restriction on ω k 3 .

	Model	n f	ωk	std( ω k ) min( ω k ) max( ω k )
	VGG16:	4,096 1.18	0.08	0.98	1.43
	Resnet18:	512	1.57	0.10	1.30	1.80
	Resnet50:	2,048 0.87	0.05	0.72	1.06
	EfficientNet-b0: 1,280 2.38	0.16	1.83	3.01
	Class Proximity
	Label # 182 184 185 186 187 189
	182	1.00 0.42 0.43 0.37 0.23 0.39
	184	0.42 1.00 0.41 0.31 0.21 0.40
	185	0.43 0.41 1.00 0.62 0.44 0.36
	186	0.37 0.31 0.62 1.00 0.43 0.36
	187	0.24 0.21 0.44 0.43 1.00 0.28
	189	0.39 0.40 0.36 0.36 0.27 1.00

Table 5 .

 5

3 -Correlations of classes vector in the fully-connected layer of Resnet18. Studied classes are expected to have high correlations:182:Border Terrier, 184:Irish Terrier, 185:Norwich Terrier, 186:Norfolk Terrier, 187:Yorkshire Terrier, and 189:Lakeland Terrier

Table 5 .

 5 3 gives us the values of Cω for dog labels mentioned in Sect. 5.2.1. For comparison, Tab. 5.4

	Label# 182	184	185	186	187	189
	579	0.02 -0.04 0.00 0.06 0.02 -0.03
	580	-0.01 0.02 0.01 0.04 -0.02 -0.03
	581	-0.04 -0.04 -0.02 -0.10 -0.05 -0.05
	582	-0.04 0.00 -0.03 0.00 -0.06 -0.11
	583	-0.07 -0.04 0.02 -0.01 0.01 0.00
	584	-0.08 -0.06 -0.03 -0.01 0.08 -0.13

3. Features therefore lies in a tiny fraction of ω k space R n f + :

1

2 nf . In the case of even the smallest n f of Resnet18, it represents 7.46 × 10 -153 % of it.

Table 5 .

 5 4 -Correlations of the same classes vector as Tab. 5.3. New classes are expected to have low correlations:579: grand piano, grand, 580: greenhouse, nursery, glasshouse, 581: grille, radiator grille, 582: grocery store, 583: guillotine, and 584: hair slide

Table 6 .

 6 1 -Benchmarks Comparison. Average Runtimes per ImageNet Image with ResNet50 [98].

Table 6 .

 6

			Accuracy	d1 /2
		(millions)	η(0) white box black box
	AlexNet [137]	62.38	56.8	0.19	2.17
	CSPResNeXt50 [152]	20.57	84.6	0.13	4.48
	DualPathNetworks 68b [88]	12.61	83.8	0.08	3.82
	MixNet Large [144]	7.33	84.2	0.12	2.96
	MobileNetV2 [131]	5.83	80.1	0.09	2.90
	ReXNet 200 [31]	16.37	85.4	0.14	3.89
	RegNetY 032 [116]	19.44	85.8	0.11	4.94
	SEResNeXt50 32x4d [66]	27.56	85.9	0.12	5.01
	VGG16 [137]	138.00	74.9	0.09	2.44
	EfficientNet AdvProp [158]	5.29	84.3	0.31	4.35
	EfficientNet EdgeTPU Small [142]	5.44	82.8	0.15	3.16
	EfficientNet NoisyStudent [160]	5.29	82.7	0.19	2.37
	EfficientNet [142]	5.29	82.8	0.17	3.56
	ResNet50 (torchvision) [61]	25.56	77.9	0.10	2.77
	ResNet50 (timm) [61]	25.56	80.5	0.15	4.35
	ResNet50 AdvTrain [98]	25.56	60.8	2.56	9.88

2 -Benchmark of models with 1.000 ImageNet Images

Table 7 .

 7 1 -Accuracy (in %) and mean average distortion of FGSM and PGD 2 attacks against EfficientNet-b0 in best-effort mode (see section 6.4.2) over 1,000 randomly selected images from ImageNet.

		floats		PNG		JPEG90	JPEG75	JPEG60
	Attack	Acc. Dist. Acc. Dist. Acc. Dist. Acc. Dist. Acc. Dist.
	None	83.0 0.00 83.0 0.00 83.0 3.0 81.0 5.2 81.0 6.4
	FGSM	11.6 0.33 66.4 0.41 83.0 3.0 81.2 5.2 81.4 6.4
	PGD 2	0.2 0.16 81.7 0.04 83.0 3.0 81.2 5.2 81.4 6.4
	PGD 2 round	0.2 0.28 0.2 0.28	-	-	-	-	-	-
	PGD 2 BPDA	-	-	-	-	0.6	3.9	6.6	5.8 12.5 6.8
	PGD 2 ours	0.2 0.16 0.2 0.17 0.6	3.0	0.6	5.2	1.0	6.5

Table 7 .

 7 2 -Accuracy (in %) of EfficientNet-b0 equipped with a JPEG compression as a defense front-end reformer against our quantized best-effort BP.

				Defense
	Atttack	None JPEG100 JPEG90 JPEG75 JPEG60
	Spatial	0.1	4.0	42.1	63.0	70.4
	JPEG100	0.6	1.0	39.8	71.4	76.6
	JPEG90	0.6	0.7	0.6	60.0	69.8
	JPEG75	0.6	0.7	2.4	0.6	14.7
	JPEG60	1.0	1.0	1.3	6.4	1.1

Table 8 .

 8 1 -Robustness of recent classifiers against PGD 2 followed by quantization Acc (%) P suc (%) L 2

	Alexnet	57.0	100	104
	VGG-16	75.0	100	56.5
	GoogLeNet	77.2	99.8	72.9
	ResNet-50	80.0	97.2	81
	Vanilla EfficientNet-b0 [142]	82.8	99.1	115
	Robust EfficientNet [158]	84.3	98.5	192

average distortion necessary for hacking the images (cf. Tab. 8.1). This reveals a different hierarchy: ResNet-50 and VGG-16 are quite fragile contrary to the old AlexNet. Overall, the recent EfficientNet is both more accurate and more robust.

Table 8 .

 8 2 -Detection probabilities (TPR 5 ) with forensics detectors of adversarial images targeting classifier vanilla EfficientNet-b0[START_REF] Mingxing | EfficientNet: Rethinking Model Scaling for CNN[END_REF]

		P suc L 2 SRM(%) SCRMQ1(%) SRNet(%)
	FGSM+[12] 89.7 286	72.00	83.3	93.5
	PGD 2 +[12] 98.6 113	65.02	83.1	93.8
	CW+[12]	89.7 97	68.78	83.6	94.5
	DDN	83.2 186	79.53	91.9	94.8

Table 8 .

 8 3 -Undetectability of steganographic embedding on PGD 2 against the vanilla model (Van) and its robust version (Rob).

		d f	P suc (%)	L 2	SCRMQ1(%) SRNet(%)
			Van Rob Van Rob Van	Rob	Van Rob
	[12]	2 98.6 98.3 101 167 83.1	84.6	93.8 90.1
	HILL	2 98.6 98.3 113 177 78.0	76.6	87.6 88.5
	HILL	4 98.9 98.5 125 181 76.0	73.3	87.4 88.2
	MiPod 2 98.3 98.3 176 242 77.4	76.2	86.6 87.7
	MiPod 4 98.7 98.0 164 247 74.4	70.2	84.5 87.7
	GINA	2 98.5 98.1 283 337 24.4	32.4	68.3 82.9
	GINA	4 98.8 98.2 300 330 18.6 24.3 50.9 85.2

Table 8 .

 8 4 -Detection on SCRMQ1 after training on adversarial images embedded with GINA (d f =4)

		d SCRMQ1(%)
			Van	Rob
	[12]	2	55.9	56.7
	HILL	2	53.4	53.6
	HILL	4	50.4	53.9
	MiPod 2	56.1	55.9
	MiPod 4	53.9	54.9
	GINA	2 77.7 78.4
	GINA	4	68.5 79.7

Table 9 .

 9 1 -Distortion for attacking 90% of the images for the EfficientNet family

	Model		Downscaling	Ensemble
	Size		Nearest Area Average Worst
	b0: 224	0.15	0.39	0.54	0.53
	b1: 240	0.16	0.37	0.56	0.47
	b2: 260	0.17	0.37	0.47	0.49
	b3: 300	0.17	0.34	0.46	0.44
	b4: 380	0.21	0.33	0.40	0.54
	b5: 456	0.26	0.33	0.38	0.42
	b6: 528	0.25	0.31	0.36	0.37
	b7: 600	0.37	0.37	0.37	0.37
	a targeted classifier. A change in the downscaling method modifies these frontiers which
	in return classify the sample accurately again.
	Table 9.2 -Accuracy (%) when downscaling from size L = 600 to = 224 for EfficientNet-
	b0
				Attack
	Defense Bil. Bic. Area Near. BilG BilAg
	Bil		0.7 70.7 74.6 75.1 73.5	8.2
	Bic.		5.6 0.9 75.6 75.9 74.1	9.6
	Area	72.8 72.8 0.3	72.8 69.8 71.9
	Near.	75.6 75.6 35.5	0.8	69.4 75.6
	BilG	73.4 73.5 72.6 73.4 0.50 72.2
	BilAg	74.0 73.9 73.2 75.0 72.2	0.7

  6.2 Evolution of d1 /2 with the complexity budget for black box setup. Attacks on EfficientNet [142] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Evolution of d1 /2 with the complexity budget for white box setup. Attacks on EfficientNet [142] . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Black-box d1 /2 as a function of white-box d1 /2 . . . . . . . . . . . . . . . . . 7.1 Operating curves of EfficientNet-b0 against FGSM and PGD in best-effort mode with floating point (plain) or quantized (dotted) pixel values. . . . . 7.2 Example of adversarial images quantized with our method. Attacked network is EfficientNet-b0. The predicted label is displayed below. . . . . . . . 7.3 Comparing the approximated loss (7.9) (resp. (7.19

  This metrics is commonly seen in problems involving a pairing of vectors:M SE(g 1 , g 2 ) := E[(g 1 -g 2 ) 2 ] (9.8)Mean Squared Logarithmic ErrorThis metrics is more rarely used is such problems. In this specific task, it can be used since feature maps are strictly positive. It offers more stability over MSE whose values may explode.Cosine CorrelationWe worked multiple times cosine distances since Sect.5.2.3:cos(g 1 , g 2 ) := g 1 , g 2 g 1 • g 2 (9.10)Where • is the Euclidian distance. This distance needs to be maximized instead of minimized. We call Cosine Correlation our last metrics that minimizes the sine value of this angle using the trigonometric identity:CC(g 1 , g 2 ) := 1 -cos(g 1 , g 2 ) 2 (9.11)Theoretical Scaling We observed in Chap. 9 the scaling of adversarial examples. Weestablished that σ A (n) scaled as O( 1 / √ n). This yields σ A (n 1 ) × √ n 1 = σ A (n 2 ) × √ n 2 . This is true only if σ X (n 1 ) = σ X (n 2 ).Let us consider g(x) a mere downscaling operation. We can now establish the distortion that the adversarial signal should have in the feature space:

M SLE(g 1 , g 2 ) := E[(log(g 1 + 1) -log(g 2 + 1)) 2 ] (9.9)

1

  Res18 2 Res50 Res50 1 Res50 2 Res50 3

	Eff.Net-b0	1.00	0.70	0.71	0.70	0.85	0.85	0.85	0.67
	Res18		1.00	0.91	0.92	0.75	0.75	0.75	0.74
	Res18 1			1.00	0.91	0.75	0.76	0.76	0.74
	Res18 2				1.00	0.75	0.75	0.75	0.74
	Res50					1.00	1.00	1.00	0.89
	Res50 1						1.00	1.00	0.92
	Res50 2							1.00	0.85
	Res50 3								1.00

Table 9 .

 9

The name is inspired by a famous automaton named the Mechanical Turk, built in 1770. It was supposed to be a self-operating machine that played chess. It was however quickly discovered that it was a hoax and a human was operating the 'automaton', hiding within the table that displayed the chess set...

FAST being itself an improved version of SIFT[START_REF] Rosten | Fusing points and lines for high performance tracking[END_REF] and BRIEF another feature extractor[START_REF] Calonder | Brief: Binary robust independent elementary features[END_REF] robust to pixel intensity. We can certainly notice a pattern of puns well-appreciated by the community.

Notation differs from previous chapter. Here is equal to u + q seen previously

tized. A few images were unsuccessful at low quality factors (1% for JPEG60). -Little distortion is added to the inherent distortion of the JPEG compression.

-Yet, at low quality factor, the adversarial perturbation is not typical from JPEG compression artifact.

Figure 7.7 summarizes the results but this time assuming that misclassified images have null distortion. We consider this procedure to carry out the main information about the efficiency of the attack and we use this setup to display the following results.

Impact of the Degree of Freedom

A previous iteration of this work [START_REF] Bonnet | What If Adversarial Samples Were Digital Images?[END_REF] considered only quantization in the spatial domain with d f = 1. The method here is more general with higher degrees of freedom. Is that useful?

When d f = 0, quantization is equivalent to rounding the perturbation samples to the nearest integer. Figures 7.10 shows that this mostly leads to unsuccessful attack. Indeed, the perturbation is a weak signal partially destroyed by rounding. The success rate converges to approximately 20% for the different JPEG quality factors, close to the proportion of misclassified original images (see Fig. 7.8). This demonstrates the robustness of the classifier against JPEG.

When attacking EfficientNet-b0, d f = 1 seems enough to quantize almost every image Chapter 8

QUANTIZATION USING STEGANOGRAPHIC STRATEGIES

Chapter 9

probably far beyond the adversarial frontier. We observed this when reviewing literature for Chap. 8 .

Once again, in a matter of studying the best of both worlds, we mostly studied the Optimize setup. But this may not always be adequate. Transferability is a such situation. Pushing attacks further is probably a better option and we did not. Still, we consider this scenario to be the most likely. An attack like BP, in its best-effort implementation (see Chap. 6) is impressively effective with few iterations as shown in Chap. 6.

Extend the Image Paradigm

We studied at length the creation of adversarial images and not samples. We believed this last point to be an important factor. Having access to the model (white-box) does not mean you have access to inner representations of your data.

A likely attack will use images online or even in the real world. Online adversarial images could be used to bypass content filters online or to protect your data from data collection. This was the purpose of our short work [10] (see Appendix A). Real-world images could be applied to endanger passengers of autonomous vehicles or to evade facial recognition or any detection system.

Making images was thus in line with our direction in this thesis: making attack realistic. Arguably, a white-box setup is not the most likely setup. But its likelihood increase with the advent of model extraction (euphemism for stealing) attacks [START_REF] Tramèr | Stealing machine learning models via prediction {APIs}[END_REF].

About Defense and Perspectives

We considered the Defense side a few times throughout this thesis. Notably when considering detectors in Chap. 8, the effect of ensemble models (Chap. 7 and Chap. 9), or best-practice for image size and resizing in Chap. 9. Contribution on the attack side can also be considered an iteration towards better defense. Especially in Chap. 8 when we both built detection and studied the effect of steganographic strategies to fool detectors.

We also have been working on adversarial training recently but our work remains unpublished to this day. Insight into our experiments and results is displayed in Appendix B. Our method yields robustification with unsupervised retraining. It is akin to logit pairing, only it is performed on feature maps.

A Final Word

We mentioned many times that both attack and defense compete against each other. This can sometimes give birth to biased critics of opposing works. The lack of unity in the evaluation may sometimes hinder progression. As a conclusion, we would like to bring attention to a recent work by Guo et al. [START_REF] Guo | Adversarially trained neural representations may already be as robust as corresponding biological neural representations[END_REF]. They argue that currently robust DNN could be as robust as biological neurons. This paper runs experiments to characterize similar adversarial signals in the human eye. Leveraging this model, they go on to show that the human eye is as easily fooled as robust models. We can easily be baffled by the performance of DNNs and forget that we ourselves can be vulnerable to adversity. We are all familiar with optical illusions. Yet, we think that DNNs should not be fooled under any circumstances. The only difference may lie in the ability we have to understand that we are being fooled.

LIST OF FIGURES

INTRODUCTION

The internet is flooded with images. This is especially true with the growth of social networks over the last decade. All this data is used to perform analysis to bring out new trends or to train predictive models. When it comes to images, deep neural networks vastly lead the landscape of machine learning. These deep neural networks are especially known to thrive on big datasets. This leads to the idea that more data leads to better models. While there certainly is truth to that affirmation, better learning mostly comes out of better data. Good data is data that both fits the task (e.g. people, places, objects detection) and whose quality is good. Due to the amount of available data, a human could not perform this cherry-picking of good data. Automated classifiers like BIQA [4] have been trained to assess the quality of an image. This classifier was trained on images whose quality was labeled based on the perceived quality of the media (e.g. resolution, compression artifacts). To protect one's data, images can be manipulated and slightly modified to defeat the automatic quality assessment [6]. We chose an adversarial attack approach to achieve this goal.

APPROACH 2.1 Adversarial Examples

Adversarial examples were first introduced by Szegedy et al. [8] in early 2014. They are usually studied in the case of image classification: An attack effectively crafts a perturbation of an image to a small extent but enough to fool even the best classifiers. In this setup, an original image 𝑥 0 is given as an input to the trained neural network to estimate the probabilities ( p𝑘 (𝑥 0 )) 𝑘 of being from class 𝑘 ∈ {1, . . . , 𝐾 }. The predicted class is given by:

The classification is correct if ĉ (𝑥 0 ) = 𝑐 (𝑥 0 ) the ground truth class for 𝑥 0 . The goal of an attack is to craft an imperceptible perturbation 𝑝 such that the adversarial sample 𝑥 𝑎 = 𝑥 0 + 𝑝 verifies ideally:

Copyright 2020 for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). MediaEval'20, December 14-15 2020, Online

Where ∥ • ∥ is a measure of distortion, in most cases the Euclidean distance. A small distortion makes it less likely for human to perceive that the image was manipulated.

BIQA is a deep neural network and as such is vulnerable to adversarial attacks. However BIQA is not a classifier returning a class prediction but a regressor giving a quality score 𝐵𝐼𝑄𝐴(𝑥) ∈ [0, 100]. The notion of adversarial sample thus needs to be redefined. In our case, we set a target score 𝑠 𝑎 ∈ [0, 100]. Regardless of the original score 𝐵𝐼𝑄𝐴(𝑥 𝑜 ), our adversarial sample now ideally verifies:

Quantization

An original image 𝑥 0 in the spatial domain (e.g. PNG format) is a 3-dimensional discrete tensor: 𝑥 0 ∈ {0, 1, . . . , 255} 𝑛 (with 𝑛 = 3 × 𝑅 × 𝐶, 3 color channels, 𝑅 rows and 𝐶 columns of pixels). The main objective of this task is to craft images: 𝑥 𝑎 ∈ {0, 1, . . . , 255} 𝑛 . This additional constraint to the attack is yet not easy to enforce. In a deep neural network, this input image is first preprocessed onto a range domain that usually reduces variance of the data. Its purpose is to ease the learning phase and thus to increase the performance of a deep neural network. This preprocessing is defined by design before the training stage and cannot be modified at testing. In the case of BIQA, the range domain is [-0.5, 0.5] 𝑛 .

Most attacks of the literature are performed in this domain without consideration of the transformation it represents. This leads to an adversarial sample 𝑥 𝑎 ∈ [0, 255] 𝑛 after reverting the preprocessing. To save this adversarial sample 𝑥 𝑎 as an image, the first step is then to round it which will erase most of the perturbation in the case of a low-distortion attack. Rounding is therefore likely to remove the adversarial property of the sample.

Paper [1] addresses this problem presenting a post-processing added on top of any attack to efficiently quantize a perturbation: It keeps the adversarial property while lowering the added distortion. The method is based on a classification loss to ensure adversariality defined as follows:

To adapt this method to the context of BIQA, we only need to redefine it to:

For a given x, 𝐿(𝑥) < 0 ensures x scores under the target 𝑠 𝑎 .

MediaEval'20, December 14-15 2020, Online B. Bonnet, T. Furon, P. Bas

EXPERIMENTAL WORK

In this task, we know the classifier (BIQA) and its parameters. We are therefore in a white-box setup. Most modern attacks are developed in this scenario, from the most basic FGSM [3] and IFGSM [5] to the most advanced PGD [7], C&W [2], BP [10]. FGSM is a noniterative attack bringing a fast solution of the problem. Our work used this attack in the early stages as a proof of concept bringing a quick further understanding of the problem. Artifacts were visible. Instead all the results reported here are crafted using more the advanced PGD attack [7] in its 𝐿 2 optimization version. One input parameter is the distortion budget. We run the attack over 7 iterations with different distortion budgets (whose maximum value is set to 2000). A binary search quickly finds an adversarial sample with the lowest distortion.

JPEG compression

The final images will be evaluated on their JPEG [9] counterpart. This compression is done with a quality factor of 90. However there are many image compression sofwares providing different results. We used the command line $ convert to simulate this compression. Tables 1 and2 show for different methods both 𝑃 𝑃 𝑁 𝐺 and 𝑃 𝐽 𝑃𝐸𝐺 respectively the percentage of images successfully beating the target score in the PNG domain and the JPEG domain. Additionally Table 2 shows results of the jury as well.

Quantization

Spatial domain.

The work [1] serves as a baseline for quantization. We only slightly adapt it as stated in Sect. 2.2. Table 1 reports our results for two target scores: 𝑠 𝑎 = 30 and 𝑠 𝑎 = 50. It appears that the perturbation crafted in the pixel domain is fragile when facing a JPEG compression.

DCT domain.

The final image being evaluated after a JPEG compression, we explore a method adapting the quantization [1] to the DCT domain. Using the same notations [1]: Let 𝑋 𝑜 denote the image in the DCT domain, 𝑋 𝑎 = 𝑋 𝑜 + 𝑃 is the result of an initial attack like PGD, and 𝑋 𝑞 = 𝑋 𝑜 + 𝑃 + 𝑄 the final quantized transformed coeffcients. We solve a Lagrangian formulation:

where 𝜆 is the Lagrangian multiplier controlling the tradeoff between the distortion 𝐷 (𝑄) and 𝐿(𝑄) defined in (5). The distortion 𝐷 (𝑄) is defined as the squared 𝐿 2 norm of added perturbation:

The quantization noise 𝑄 is s.t. 𝑋 𝑜 + 𝑃 + 𝑄 ∈ ΔZ 𝑛 , where Δ ∈ N 𝑛 is the quantization step matrix for JPEG QF=90. If we use a first order approximation of 𝐿(𝑄), we can develop (6) in a second-degree polynomial function. For any coefficient 𝑗, this function is locally minimized by:

where 𝐺 = ∇𝐿(𝑄)| 𝑄=0 the gradient computed at 𝑄 = 0. This minimum however does not enforce (𝑃 + 𝑄 ★ ) ∈ Z 𝑛 . A simple rounding of (𝑃 + 𝑄) will then finalize the quantization. Finally we need to control a maximum allowed distortion. If 𝜆 gets big, 𝑄 ( 𝑗) become a very high value which is not desirable. The final value

RESULTS AND ANALYSIS

Tables 1 and2 show the importance of considering the JPEG compression. When the image is quantized by the 𝐿 2 optimization in the spatial domain, most images will successfully be adversarial images. However, very few of them remain adversarial after the JPEG compression. The BIQA score on most images increases up to 10 points. If the quantization is done in the DCT domain, most of them remain adversarial and the task is successful. It is however obviously more difficult to beat a lower target score 𝑠 𝑎 . An interesting property of the DCT quantization is that it creates typical JPEG artifacts as seen on Figure 1. This is especially true in low frequency images since it is harder to remain undetectable in a such situation.

DISCUSSION AND OUTLOOK

The MediaEval task was a good opportunity to extend our previous work [1] to 1) a regressor BIQA, and 2) in the DCT domain. Saving the DCT coefficients directly into a JPEG image is more consistent as it offers a better control on adversariality. Another difficulty of this task was the lack of knowledge about the compression algorithm. We therefore worked in a 'gray' box setup. The results showed that JPEG compression have a big effect on the BIQA score of, at least, adversarial images (and probably any other quality estimator). Hopefully our JPEG compression is close to the one used in the contest which allowed transferability of our adversarial images.

Part IV

Appendix B 161

Introduction and Related Works

We discussed in Chap. 5 the existence of different techniques to make a DNN more resilient to evasion attacks. We call this resilience robustness, and the process robustification. The most commonly used method is undeniably adversarial training or retraining. (Eq. (5.5))

A slightly different optimization is suggested by Kannan et al. [START_REF] Harini Kannan | Adversarial logit pairing[END_REF]. Called Logits Pairing, their method mixes both paradigms (Eq.(5.6)). The type of training they propose is akin to label distillation [START_REF] Hinton | Distilling the knowledge in a neural network[END_REF]. We saw in Chap. 5 the works of Papernot et al. [START_REF] Papernot | Distillation as a defense to adversarial perturbations against deep neural networks[END_REF] that extended this idea to improve robustness. A teacher DNN performs a prediction on data, which in turn is fed to a student DNN as labels along with the same data. Training is performed on the student as if labels were the ground-truth. Teacher DNN remains unmodified throughout the procedure. Through this distillation of knowledge, they demonstrate gains in robustness.

The works of Kannan et al. [START_REF] Harini Kannan | Adversarial logit pairing[END_REF] add to this procedure the presence of distortionconstrained adversarial examples. Distillation is this time performed on logits rather than class probabilities. Goldblum et al. [START_REF] Goldblum | Adversarially robust distillation[END_REF] also use adversarial samples but perform more classical label distillation to display gains in robustness.

Our work is in the same line of distillation. We however propose a retraining phase on features instead of logits. We use the final features (i.e. one step before logits) to do so. We estimate that most of the vulnerability lies in feature extraction rather than classification. We display results that back this idea in Sect. 9.8. Feature vectors are richer in information than logits and thus a better candidate for such unsupervised training.

Problem Formulation

Let f (x) : R n → R C be a trained DNN, built from a feature extractor g(x) : R n → R n f and fully-connected layer h(x) : R n f → R C . We initially clone this model to obtain

Optimization Method

We experiment with the different distances discussed in Sect. 9.7.2. Figure 9.5 display results for each of them. Behavior is fairly similar for each of them, gains in robustness are obtained at the cost of a small loss in clean accuracy. We note that MSE however has the best trade-off. We observe good robustness as well as better accuracy than other methods.

Adversarial Distortion

We also experiment with the effect of during the generation of the adversarial example. Figure 9.6 display results for = 0.5, = 1, and = 2. Performances of the robust model from [START_REF] Hadi | Do adversarially robust imagenet models transfer better?[END_REF] are also visible.

We observe that our method gets better robustness with higher values of . But this comes in exchange to clean accuracy. Overall, it seems that = 1 yields the best of both quantities. Its clean accuracy is roughly equal to the training with = 0.5 but with better robustness. Since there is a trade-off, the idea of a "best" method is debatable. We can see that our method is not on par with [START_REF] Hadi | Do adversarially robust imagenet models transfer better?[END_REF]. Their lower-distortion training with = 0.05 seems equally as robust as our method with = 0.5. But their clean accuracy is better. Their higher-distortion training with = 0.5 display great robustness with a loss in a clean accuracy equivalent to our method with = 1. And our retrained model is far more vulnerable.