Marine Picot

Pablo Piantanida

Protecting Deep Learning Systems Against Attack: Enhancing Adversarial Robustness and Detection

Over the last decade, Deep Learning has been the source of breakthroughs in many different fields, such as Natural Language Processing, Computer Vision, and Speech Recognition. However, Deep Learning-based models have now been recognized to be extremely sensitive to perturbations, especially when the perturbation is welldesigned and generated by a malicious agent. This weakness of Deep Neural Networks tends to prevent their use in critical applications, where sensitive information is available, or when the system interacts directly with people's everyday life. In this thesis, we focus on protecting Deep Neural Networks against malicious agents in two main ways.

The first method aims at protecting a model from attacks by increasing its robustness, i.e., the ability of the model to predict the right class even under threats. We observe that the output of a Deep Neural Network forms a statistical manifold and that the decision is taken on this manifold. We leverage this knowledge by using the Fisher-Rao measure, which computes the geodesic distance between two probability distributions on the statistical manifold to which they belong. We exploit the Fisher-Rao measure to regularize the training loss to increase the model robustness. We then adapt this method to another critical application: the Smart Grids, which, due to monitoring and various service needs, rely on cyber components, such as a state estimator, making them sensitive to attacks. We, therefore, build robust state estimators using Variational AutoEncoders and the extension of our proposed method to the regression case.

The second method we focus on that intends to protect Deep-Learning-based models is the detection of adversarial samples. By augmenting the model with a detector, it is possible to increase the reliability of decisions made by Deep Neural Networks. Multiple detection methods are available nowadays but often rely on heavy training and ad-hoc heuristics. In our work, we make use of a simple statistical tool called the data-depth to build efficient supervised (i.e., attacks are provided during training) and unsupervised (i.e., training can only rely on clean samples) detection methods.

Titre : Protéger les systèmes de deep learning face aux attaques : améliorer la robustesse adversaire et la détection Mots clés : Deep Learning, IA, Smart Grids, Sécurité, Attaques, Défenses Résumé : Au cours de la dernière décennie, l'apprentissage profond a été à l'origine de percées dans de nombreux domaines différents, tels que le traitement du langage naturel, la vision par ordinateur et la reconnaissance vocale. Cependant, il est désormais connu que les modèles basés sur l'apprentissage profond sont extrêmement sensibles aux perturbations, en particulier lorsque la perturbation est bien conçue et générée par un agent malveillant. Cette faiblesse des réseaux neuronaux profonds tend à empêcher leur utilisation dans des applications critiques, où des informations sensibles sont disponibles, ou lorsque le système interagit directement avec la vie quotidienne des gens. Dans cette thèse, nous nous concentrons sur la protection des réseaux neuronaux profonds contre les agents malveillants de deux manières principales.

La première méthode vise à protéger un modèle des attaques en augmentant sa robustesse, c'està-dire la capacité du modèle à prédire la bonne classe même en cas d'attaques. Nous observons que la sortie d'un réseau neuronal profond forme une variété statistique et que la décision est prise sur cette variété. Nous exploitons cette connaissance en utilisant la mesure de Fisher-Rao, qui calcule la distance géodésique entre deux distributions de probabilité sur la variété statistique auquel elles appartiennent. Nous utilisons la mesure de Fisher-Rao pour régulariser la fonction coût utilisée lors de l'apprentissage et augmenter la robustesse du modèle. Nous adaptons ensuite cette méthode à une autre application critique: les réseaux intelligents (Smart Grids), qui, en raison de divers besoins de la surveillance et de service, reposent sur des composants cybernétiques, tels qu'un estimateur d'état, ce qui les rend sensibles aux attaques. Nous construisons donc des estimateurs d'état robustes en utilisant des autoencodeurs variationnels et l'extension de notre méthode proposée au cas de la régression.

La deuxième méthode sur laquelle nous nous concentrons et qui vise à protéger les modèles basés sur l'apprentissage profond est la détection d'échantillons adverses. En ajoutant un détecteur au modèle, il est possible d'augmenter la fiabilité des décisions prises par les réseaux neuronaux profonds. De multiples méthodes de détection sont disponibles aujourd'hui, mais elles reposent souvent sur un entraînement lourd et des heuristiques ad-hoc. Dans notre travail, nous utilisons des outils statistiques simples appelés les profondeur de données (data-depth) pour construire des méthodes de détection efficaces supervisées (c'est-à-dire que les attaques sont fournies pendant l'entraînement du détecteur) et non supervisées (c'est-à-dire que l'entraînement ne peut s'appuyer que sur des échantillons propres).

Title: Protecting Deep Learning Systems Against Attack: Enhancing Adversarial Robustness and Detection Keywords: Deep Learning, AI, Smart Grids, Security, Attacks, Defences Abstract: Over the last decade, Deep Learning has been the source of breakthroughs in many different fields, such as Natural Language Processing, Computer Vision, and Speech Recognition. However, Deep Learning-based models have now been recognized to be extremely sensitive to perturbations, especially when the perturbation is well-designed and generated by a malicious agent. This weakness of Deep Neural Networks tends to prevent their use in critical applications, where sensitive information is available, or when the system interacts directly with people's everyday life. In this thesis, we focus on protecting Deep Neural Networks against malicious agents in two main ways.

The first method aims at protecting a model from attacks by increasing its robustness, i.e., the ability of the model to predict the right class even under threats. We observe that the output of a Deep Neural Network forms a statistical manifold and that the decision is taken on this manifold. We leverage this knowledge by using the Fisher-Rao measure, which computes the geodesic distance between two probability distributions on the sta-tistical manifold to which they belong. We exploit the Fisher-Rao measure to regularize the training loss to increase the model robustness. We then adapt this method to another critical application: the Smart Grids, which, due to monitoring and various service needs, rely on cyber components, such as a state estimator, making them sensitive to attacks. We, therefore, build robust state estimators using Variational AutoEncoders and the extension of our proposed method to the regression case.

The second method we focus on that intends to protect Deep-Learning-based models is the detection of adversarial samples. By augmenting the model with a detector, it is possible to increase the reliability of decisions made by Deep Neural Networks. Multiple detection methods are available nowadays but often rely on heavy training and adhoc heuristics. In our work, we make use of a simple statistical tool called the data-depth to build efficient supervised (i.e., attacks are provided during training) and unsupervised (i.e., training can only rely on clean samples) detection methods.

Résumé

Au cours de la dernière décennie, l'apprentissage profond a été à l'origine de percées dans de nombreux domaines différents, tels que le traitement du langage naturel, la vision par ordinateur et la reconnaissance vocale. Cependant, il est désormais connu que les modèles basés sur l'apprentissage profond sont extrêmement sensibles aux perturbations, en particulier lorsque la perturbation est bien conçue et générée par un agent malveillant. Cette faiblesse des réseaux neuronaux profonds tend à empêcher leur utilisation dans des applications critiques, où des informations sensibles sont disponibles, ou lorsque le système interagit directement avec la vie quotidienne des gens. Dans cette thèse, nous nous concentrons sur la protection des réseaux neuronaux profonds contre les agents malveillants de deux manières principales.

La première méthode vise à protéger un modèle des attaques en augmentant sa robustesse, c'est-à-dire la capacité du modèle à prédire la bonne classe même en cas d'attaques. Nous observons que la sortie d'un réseau neuronal profond forme une variété statistique et que la décision est prise sur cette variété. Nous exploitons cette connaissance en utilisant la mesure de Fisher-Rao, qui calcule la distance géodésique entre deux distributions de probabilité sur la variété statistique auquel elles appartiennent. Nous utilisons la mesure de Fisher-Rao pour régulariser la fonction coût utilisée lors de l'apprentissage et augmenter la robustesse du modèle.

Nous adaptons ensuite cette méthode à une autre application critique : les réseaux intelligents (Smart Grids), qui, en raison de divers besoins de la surveillance et de service, reposent sur des composants cybernétiques, tels qu'un estimateur d'état, ce qui les rend sensibles aux attaques. Nous construisons donc des estimateurs d'état robustes en utilisant des autoencodeurs variationnels et l'extension de notre méthode proposée au cas de la régression.

La deuxième méthode sur laquelle nous nous concentrons et qui vise à protéger les modèles basés sur l'apprentissage profond est la détection d'échantillons adverses. En ajoutant un détecteur au modèle, il est possible d'augmenter la fiabilité des décisions prises par les réseaux neuronaux profonds. De multiples méthodes de détection sont disponibles aujourd'hui, mais elles reposent souvent sur un entraînement lourd et des heuristiques ad-hoc. Dans notre travail, nous utilisons des outils statistiques simples appelés les profondeur de données (data-depth) pour construire des méthodes de détection efficaces supervisées (c'est-à-dire que les attaques sont fournies pendant l'entraînement du détecteur) et non supervisées (c'est-à-dire que l'entraînement ne peut s'appuyer que sur des échantillons propres). 2.7 FAB method for a linear binary classifier. In blue is the distance between the original example x orig and the projection of the current iteration x (i ) onto the hyperplane, in green is the impact of the step towards the original sample, and in red is the new distance. The left figure represents the FAB algorithm using projections onto the hyperplane, while the right one represents the FAB algorithm using projections on the hyperplane plus an extrapolation step (to go over the boundary). 3.1 Illustration of FRD between two distributions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ) over the statistical manifold C . ©2022 IEEE. . . . . . . . . . . .

CONTENTS

List of Figures

LIST OF FIGURES

3.2 Visualization of statistical manifold C defined by the model q θ (y|x) = In black, we show the solutions obtained by minimizing the risk L TRADES (θ) in Equation 3.5 ( Figure 3.6b), and the risk L FIRE (θ) in Equation 3

List of Tables

3. [START_REF] Introduction Tramer | On adaptive attacks to adversarial example defenses[END_REF] Comparison between KL and Fisher-Rao based regularizer under whitebox l ∞ threat model. Note that we do not use the same hyperparameters as presented in ZHANG and collab. [2019] for the TRADES method. I also owe a lot to all of my friends, and I would like to give a special thank you to Julie, Charlotte, Pauline, Charles, Eva, and Abdellatif, for always being there for me.

Last but not least, I would like to thank my family, my parents, Vincent and Marylène, and my siblings, Vivien, Paul, Nicolas and Mélanie, for my upbriging and their everlasting support. You helped me more than you can think to achieve this. [IQBAL and collab., 2021], autonomous cars [MOZAFFARI and collab., 2020],

bots [LOHOKARE and collab., 2020], or ad recommendations [DU and collab., 2021] among others. Such Artificial Intelligence (AI)-based systems can access critical and personal data about individuals, such as their medical history or credit card information, and their decision can impact our society or directly affect individuals' lives (for example, the autonomous car crashing with passengers inside).

In this context, many concerns about the potential failures of large neural networks, which are not trustworthy enough, limit their adoption. They are due to the poor understanding of the behavior of large neural networks, which are often seen as black boxes. An essential line of research with high industrial and societal impacts consists in designing tools to make them more reliable. These tools are crucial to ensure the wide adoption of DL-based systems. Among the problems of high interest, the undertaken research is expected to take into consideration the following aspects:

• Ensure that no transmission of data to an unauthorized recipient (data leakage) is possible, i.e., increasing the global system security.

• If it is impossible, at least ensure that the potentially leaked data does not contain sensitive information, i.e., increasing data privacy.

• Increase the reliability of the deep neural network's decision process, i.e., finding ways to determine whether the user can trust the decision process.

The primary focus of this thesis is on security aspects. These topics are extensively studied in the research community, as illustrated by the continuous growth in the number of research works addressing security aspects the machine learning which have been published in top-tier AI conferences. In what follows, we provide the reader an overview of some of the main issues that should be addressed to develop safe and trustworthy DL-based systems.

How to ensure no private information leakage

In the privacy-related domain, the main goal is to ensure that no sensitive information is available, even if malicious agents retrieve data. Many different techniques have been studied, and can be summarized in three main categories, each acting CHAPTER 1. INTRODUCTION on one of the phase on the DL-based system deployment: modifying the data [ER- LINGSSON and collab., 2014;GOLDWASSER and MICALI, 1984;HUANG and collab., 2017;SWEENEY, 2002], acting on the training phase [HESAMIFARD and collab., 2017;IYENGAR and collab., 2019;MOHASSEL and ZHANG, 2017], or using the inference outputs [GILAD-BACHRACH and collab., 2016;MIRESHGHALLAH and collab., 2020a;RIAZI and collab., 2018;WANG and collab., 2018]. We refer the reader to [MIRESHGHAL-LAH and collab., 2020b] for a more extensive survey about the threats and defensive techniques regarding privacy.

How to increase our trust on DL-based results

Another security-related problem includes measuring and ensuring the reliability of DNNs' decisions. In this field, the goal is to find ways to increase our trust in the final decision made by a DNN. This problem is particularly interesting as large neural networks often perform well when limited to input data whose probability distribution is close to the training dataset on which the models were trained. This can be an essential source of dysfunction in real-world contexts. Indeed, data characteristics are constantly evolving and particularly subject to distributional shifts.

At test time, two main scenarios can be distinguished, depending on the source of the input. In the first scenario, we cannot be sure whether the test sample is from the original data source or whether it comes from another environment. We usually call it the out-of-distribution (OOD) scenario. In contrast, in the second scenario, we are sure that the input sample is from the original data distribution . However, we do not know how much we can trust the system's decision. We usually call it the in-distribution scenario.

How to ensure trust in the environment. Let us consider the case where the system is deployed in a potentially altered environment, i.e., out-of-distribution samples can be fed to the system. Most methods in the OOD literature, to ensure that the input of a deep neural model comes from the original environment, focus on deploying a detector that distinguishes between samples that originate from the original environment and samples that do not. As in the privacy-related fields, OOD detectors can affect different phases of the classifier's deployment procedure. First, the detection method can retrain a classifier for which distinguishing between natural and OOD samples is easier, using techniques from contrastive training [HENDRYCKS and collab., 2019;WINKENS and collab., 2020], regularization [HEIN and collab., 2019;LEE and collab., 2021;NANDY and collab., 2021], generative learning [REN and collab., 2019;SCHLEGL and collab., 2017;VERNEKAR and collab., 2019;XIAO and collab., 2020;ZHANG and collab., 2021] or ensemble learning [CHOI and JANG, 2018;VYAS and collab., 2018]). It is also possible for the OOD detection method to only interact with the model outputs. It can be directly at the output layer [HENDRYCKS and CHAPTER 1. INTRODUCTION GIMPEL, 2017;HSU and collab., 2020;LIANG and collab., 2018;LIU and collab., 2020],

or at different feature levels [GOMES and collab., 2022;KIRICHENKO and collab., 2020;LEE and collab., 2018;QUINTANILHA and collab., 2019;SASTRY and OORE, 2020;ZISSELMAN and TAMAR, 2020].

How to ensure trust in the actual decision. There exists another interesting domain where the goal is to determine whether, in perfect conditions (the system is working in its training environment), we can trust the model to make adequate decisions. One straightforward method is to directly rely on the confidence of the DL model [START_REF] Hendrycks | A baseline for detecting misclassified and out-of-distribution examples in neural networks[END_REF], i.e., the higher the confidence is, the lower the uncertainty and the better we can trust the prediction. Since then, different works have tried to develop tools that better catch the relationship between uncertainty and confidence (MC-dropout [GAL and GHAHRAMANI, 2016], Laplace Approximation [DAXBERGER and collab., 2021], SWAG [MADDOX and collab., 2019], Deep Ensembles [LAKSHMINARAYANAN and collab., 2017], DUQ [ VAN AMERSFOORT and collab., 2020],

DOCTOR [GRANESE and collab., 2021]), or define new confidence scores using auxiliary models (TrustScore [JIANG and collab., 2018], ConfidNet [CORBIÈRE and collab., 2019]).

How to protect against malicious agents

A third scenario actually exists: what happens if a malicious agent interacts with the system? As a matter of fact, all computational-based systems face the threat of malicious agents that try to disrupt their normal functioning, and cyber-security is a growing field in many communities. It is particularly true in DL as DNNs have been proven to be extremely sensitive to even the slightest well-designed disturbance [SZEGEDY and collab., 2014]. This phenomenon has been shown to be a feature of Deep Neural Networks [ILYAS and collab., 2019]. It is, therefore, crucial to develop techniques to protect the systems against them.

How to handle threats. Tampering with the input can be achieved in multiple ways. One can, for example, add a specific transformation to the input to disrupt a model's behavior. If the transformation is chosen randomly, we talk about corruption [HENDRYCKS and DIETTERICH, 2018;SCHNEIDER and collab., 2020]. However, if the tampering is done according to a specific model, under a specific environment, we talk about adversarial attacks. Multiple threat methods have been presented over the years [CARLINI and WAGNER, 2017;MADRY and collab., 2018;PAPERNOT and collab., 2016c;SZEGEDY and collab., 2014]. They are extremely powerful at fooling a DNN, and protecting against them has been a hot topic since 2018. Systems deployed without any defense against those kinds of attacks are bound to fail, which could cause massive issues if the system is used in critical conditions. For example, in 2022, XIE and collab. [2022] successfully attacked a stock prediction model.

CHAPTER 1. INTRODUCTION

Focus of this manuscript

All of the aforementioned issues are of extreme importance if we want to develop trustworthy AI systems.

However, although protecting a system from OOD samples and improving its privacy could be enough in some cases, in other cases, especially in critical systems, the vulnerability of DL-based systems to adversarial attacks would prevent their use. In addition, improving the resilience of deep neural networks against attacks can be seen as a worst-case scenario and, therefore, is the most challenging and demanding task. Finally, studying the effect of attacks on DNNs decisions gives us theoretical insights into their weaknesses, and understanding why a DL-based system is fooled can help us understand how a model learns, which is a necessary step towards Artificial General Intelligence (AGI). For all those reasons, we have decided to focus on protecting DL-based systems against malicious agents.

While a plethora of very efficient and relatively easy methods to craft attacks have been developed over the years, the matter of defending against them is an arduous task, and adequate protection methods are not easily found. We, therefore, decided to focus our work on how to protect DL-based decision processes in the presence of adversarial attacks.

In the following, we will present the general threats' goal and the motivation behind our work.

Interacting with Malicious Agents: Attacking and Defending

Global Context

As previously mentioned, AI-based systems all face the issue of handling action from malicious agents. In this setup, there exist two types of scenarios of interest. The first scenario deals with the generation of threats to fool Deep Neural Networks, i.e. behaving as the malicious agent. The second scenario consists in acting as a defender.

The case of attacking neural networks has been widely studied, ending up with many very effective methods that cause DNN-based systems to completely collapse [CARLINI and WAGNER, 2017;CROCE and HEIN, 2020;MOOSAVI-DEZFOOLI and collab., 2016]. However, while many methods have also been presented in the defender role, very few techniques are acknowledged to be truly efficient at handling malicious agents [CARLINI and WAGNER, 2017;TRAMER and collab., 2020]. The role of the defender is to make the attacker's role harder. A good defense is resilient to attackers CHAPTER 1. INTRODUCTION until the capacity, knowledge, or time required for the attacker to break the defense is beyond reasonable.

Given a set of assumptions (for example, the maximal allowed perturbation), it is possible to build defenses that will not collapse [CARMON and collab., 2019;MADRY and collab., 2018;ZHANG and collab., 2019], but there is still margin for improvement, as the performances are not perfect.

We, therefore, have chosen to focus this thesis on crafting new defenses to counter the action of malicious agents.

Notations and problem definition

Adversarial attacks are known to fool any type of DNN-based systems, as attacks for image segmentation [HENDRIK METZEN and collab., 2017;XIE and collab., 2017],

object recognition [XIE and collab., 2017], speech recognition [CISSE and collab., 2017], recurrent neural networks [PAPERNOT and collab., 2016a], reinforcement learning [LIN and collab., 2017], generative models [KOS and collab., 2018], variational autoencoder [TABACOF and collab., 2016], language classification [ALZANTOT and collab., 2018;LI and collab., 2020] have been created. Indeed, there exist plenty of applications sensitive to perturbed inputs. Still, techniques developed for image classification can be applied to any sort of application, such as NLP, image segmentation, or signal processing. In addition, the literature on attacking image classifiers is the most developed among all the possible applications. Therefore, we decided to focus mostly on defending against attacks in the image classification setting.

In what follows, we formalize the problem of image classification.

The goal of an image classifier is to predict to which class a given input belongs among a certain number of known classes (see Section 2.1 for a quick overview of the Deep Learning methodology).

Notations. First, let us define two spaces: an input space X ⊂ R d and a label space Y = {1, . . . , C}. In the classical supervised learning problem, data are composed of a pair (x, y) ∈ X × Y , where x is an image, and y is its associated label. The unknown joint data distribution is denoted by p(x, y).

Let us also define the L-layer classifier f θ parametrized by θ ∈ Θ. Let f l θ : X → R d l with l ∈ {1, . . . , L}, denote the output of the l -th layer of the classifier, where d l is the dimension of the latent space induced by the l -th layer. The class prediction, i.e., the final decision of the classifier, is obtained from the L-th layer softmax output as follows:

f θ (x) = f L θ (x) ≜ arg max c∈Y q θ (c|x) with q θ (•|x) = softmax( f L-1 θ (x)). CHAPTER 1. INTRODUCTION

Attacking a neural network

Context. As mentioned earlier, this thesis focuses on defending against adversarial attacks. We will start by defining the threat objective to better understand key challenges.

A brief history. Adversarial samples are well-designed modifications of a given input that aims at disrupting the functioning of a DNN-based system. In 2014, SZEGEDY and collab. [2014] found that this kind of perturbation, already known in the ML community, also disrupted neural networks. Since then, a variety of methods to generate potent attacks have been developed [CARLINI and WAGNER, 2017;CROCE and HEIN, 2020;GOODFELLOW and collab., 2015;MADRY and collab., 2018;MOOSAVI-DEZFOOLI and collab., 2016;PAPERNOT and collab., 2016c;SZEGEDY and collab., 2014].

Problem Formulation. The adversarial generation problem has been defined as follows [SZEGEDY and collab., 2014]. For a given input x with associated label y, we want to find an adversarial example x ′ according to:

x ′ = arg min x ′ ∥x ′ -x∥ p s.t. f θ (x ′ ) = t (1.1) x ′ ∈ [0, 1] d ,
where t can be a specific class, i.e., targeted attacks, or any class other than y, i.e., untargeted attacks. The condition x ′ ∈ [0, 1] d enforces that the new created sample is still an image.

Having this weakness exposed, researchers started to try to develop defenses to protect deep neural networks. These defenses can be separated into two main groups: robust models, where the goal is to preserve the decision integrity, and detection methods, where the goal is to ensure that the input is clean.

Defense 1: protecting DNNs' decision integrity against threats

Context. Following the emergence of the threat domain, researchers started to focus on how to defend against these attacks. The first method that caught the community's eye is crafting robust models, i.e., models which make the right decision even under attack.

A brief history. While presenting the generalization of adversarial samples to DNN models, SZEGEDY and collab. [2014] is also the first to mention that back-feeding malicious samples to train a model can help with dealing with them. GOODFELLOW and collab. [2015] is the first to mention Adversarial Training. Since then, a wide variety of defense mechanisms to increase robustness has been proposed [CARMON CHAPTER 1. INTRODUCTION and collab., 2019;MADRY and collab., 2018;PAPERNOT and collab., 2016b;WANG and collab., 2019;ZHANG and collab., 2019], some of which have been proven inefficient [CARLINI and WAGNER, 2017].

Problem Formulation.

Adversarial training [GOODFELLOW and collab., 2015] is the most commonly used defense and was the first to be recognized as truly effective.

The idea behind this method is as follows. To classify adversarial examples correctly, it is possible to train a classifier on this specific task. At each training iteration, perturbations are generated, and the model learns to classify them with their true associated label, i.e., the label of the original image.

The classifier, therefore, tries to solve the following optimization problem:

min θ∈Θ L(θ), (1.2)
where L(θ) is the risk function used to train the model.

Any risk function can be used to train the model. In their original paper, GOODFEL-LOW and collab. [2015] proposed to regularize the natural loss with another computed on the Fast-Gradient-Sign-Method attack. However, using this method has been shown to be inefficient. Later, MADRY and collab. [2018] chose to use the expectation of the classical cross-entropy but applied it to the adversarial sample instead of the natural one, i.e., L(θ) = E p(x,y) max

x ′ :∥x ′ -x∥ p ≤ε log q θ (y|x ′ ) . (1.3) This method is the first one that has been proven efficient. Further improvements of the original method have been proposed [CARMON and collab., 2019;WANG and collab., 2019;ZHANG and collab., 2019]. One of them ([ZHANG and collab., 2019])

defines a new risk function based on the trade-off between correctly classifying the clean samples and ensuring natural and adversarial samples are similarly classified.

The first part is ensured through the minimization of the classical cross-entropy risk.

The second is through the minimization of the Kullback-Leibler divergence between natural and adversarial predictions. The trade-off is controlled using λ, which is a hyperparameter. To summarize,

L(θ) = E p(x,y) -log q θ (y|x) + λ E p(x) max x ′ :∥x ′ -x∥ p ≤ε KL(q θ (•|x)||q θ (•|x ′ )) , (1.4) where KL(q θ (•|x)||q θ (•|x ′ )) = E q θ (y|x) q θ (y|x)
q θ (y|x ′ ) is the Kullback-Leibler (KL) divergence between the clean and the disturbed predictions.

The Kullback-Leibler divergence between two probability distributions p and q is a statistical measure measuring how the two probability distributions differ from one another. In other words, it can be seen as how much uncertainty we can expect from q if we know p.

CHAPTER 1. INTRODUCTION

In the case of adversarial learning, finding the disturbance that maximizes the KL divergence between the natural and the perturbed samples will tend to create samples whose predictions are independent of the clean predictions, while minimizing it will force the model to learn how to find a prediction function keeping the information shared between the clean and the attacked samples. Given a fixed model (i.e., fixed θ), the family {q θ (•|x) : x ∈ X } forms a statistical manifold. DL-based systems' output space therefore forms a manifold and the final decision of this given model is taken on it. Therefore, we can wonder: How can we use the internal structure of DNN's output space to improve its robustness ?

Defense 2: ensuring DNNs' input integrity

Context. The second main type of defense consists in ensuring that the input of the DNN is valid (i.e., is not corrupted). To do so, it is possible to craft detection methods to deploy on top of the model to protect.

A brief history. One of the first methods to detect adversarial samples is BENDALE and BOULT [2016], that only uses the output of the penultimate layer to protect their network. Since then, methods have started to use different mechanisms to detect adversarial samples, from the study of input statistics [KHERCHOUCHE and collab., 2020], to the use of different model outputs [START_REF] Li | Adversarial examples detection in deep networks with convolutional filter statistics[END_REF]MA and collab., 2018],

to the training of ad-hoc models [GONG and collab., 2017;GROSSE and collab., 2017;MENG and CHEN, 2017], to the analysis of the changes in output when the input is modified [HU and collab., 2019;XU and collab., 2018].

Problem Formulation. Crafting a detector is equivalent to finding a function g : R d → {0, 1} that will associate to the normal behavior the class 0 and the abnormal one the class 1. In practice, we want to find a dissimilarity function s : R d → R, that will output an abnormality score. The higher the score, the more likely the input is corrupted. Using a thresholding step, we can link g and s for a given input x and a given threshold γ, thanks to: Depending on the paradigm, the detection methods can be either supervised, i.e., at training time, the detector has information about the attacker, or unsupervised, i.e., the detector only has access to the natural training samples. In both cases, detection methods tend to rely on ad-hoc heuristics [FEINMAN and collab., 2017;KHERCHOUCHE and collab., 2020;XU and collab., 2018], or on heavy training [MA CHAPTER 1. INTRODUCTION and collab., 2018;MENG and CHEN, 2017;RAGHURAM and collab., 2021].

g (x) = 1 s(x)>γ =    0 if s(x) ≤ γ 1 if s(x) > γ.
To bypass these limitations, we tried to answer a simple research question: How can we craft an efficient and effective detection method based on simple tools ?

Contribution and Outlines

During this Ph.D., as mentioned in Section 1.2, we addressed two distinct questions.

• Q1: How can we use the internal structure of DNN's output space to improve its robustness?

• Q2: How can we craft an efficient and effective detection method based on simple tools?

How can we use the internal structure of DNN's output space to improve its robustness ?

In order to address Q1, we studied the impact of a new information-geometric measure to improve the robustness of neural networks: the Fisher-Rao measure.

We investigate the problem of optimizing the trade-off between accuracy and robustness. We derive an explicit characterization of the Fisher-Rao Distance (FRD) based on the information-geometric properties of the soft predictions induced space of the neural classifier and propose a new formulation of adversarial defense, called FIsher-rao REgularizer (FIRE).

In addition, we apply a similar method to another practical domain: the state estimation of Smart Grid systems. Indeed, to ensure the appropriate functioning of the electrical grid, monitoring its behavior is crucial. To perform that, we usually estimate the states of the network thanks to the measurable physical quantities available by using a state estimator [ABUR and EXPOSITO, 2004]. However, this cyber component of the power grid makes it highly vulnerable to attacks, such as False Data Injection Attacks (FDIAs) [LIU and collab., 2011], like in Ukraine in 2015 [ALDERSON and DI PIETRO, 2016] when a cyberattack in the electrical network caused a blackout for several hours. We, therefore, extended FIRE method to craft state estimator robust to attacks.

How can we craft an efficient and effective detection method based on simple tools?

In order to address Q2, we first had to find a simple tool that allows us to distinguish between natural and attacked behaviors.

CHAPTER 1. INTRODUCTION

In the statistical community, there exists a simple tool called the data-depth, first introduced by TUKEY [1975], that computes a center-outward ordering of a new sample with respect to a given probability distribution. This can be seen as how close is this new sample to the examples of a known probability distribution? Since then, multiple definitions of data-depths have been proposed, including halfspace depth [TUKEY, 1975], the simplicial depth [START_REF] Liu | On a notion of data depth based on random simplices[END_REF], the projection depth [LIU, 1992] or the zonoid depth [KOSHEVOY and MOSLER, 1997]. We refer the reader to [STAERMAN, 2022] for an in-depth explanation of the data-depths.

In our work, we decided to use the data-depths to create new methods to detect adversarial examples, depending on the available setting. We studied the detection capabilities of our proposed method under different threat scenarios and provided insights into why our proposed method outperforms others.

Outlines

The rest of this manuscript is organized as follows.

In Chapter 2, we present the different state-of-the-art methods to craft, defend against and detect adversarial examples.

Next, we present our solutions to increase the trade-off between natural performances and robustness in Part I. It is divided as follows.

• In Chapter 3, we present the Fisher-Rao distance in the case of classification, define the new robust risk based on this information-geometric distance, and experimentally compare it to previously used ones.

• In Chapter 4 and Chapter 5, we extended the method presented in Chapter 3 to the Smart Grid problem. After presenting the state estimation problem, we give a quick review of the available techniques to attack and defend against attacks.

Finally, we present our proposed method to defend against attack using the Fisher-Rao distance, first under linear approximation, then in the general case.

In in Part II, we present our proposed data-depths detection methods. It is organized as follows.

• In Chapter 6, we describe our data-depth method in the case of supervised adversarial detection. In this case, the defender can access full or partial knowledge about the threats it will encounter. We show that thanks to particular characteristics of the data-depths, our method can remain efficient against attackers with full knowledge about the deployed defense mechanism.

• In Chapter 7, we propose a fully unsupervised detection method. In this case, the defender has no knowledge whatsoever about whether it will be attacked.

CHAPTER 1. INTRODUCTION

We also show that our approach remains partially efficient against attackers with complete knowledge about the deployed defense mechanism.

Finally, in Chapter 8, we present our concluding remarks, the limitations of our work, and the interesting future work that could be done based on our work.

List of Publications

Conferences

f θ (x) = f L θ (x).
(2.1)

Classification. In the case of supervised classification, the input space X ⊂ R d , and the output space Y = {0, . . . , C}. In that case, the (L-1) t h -layer is called the logits layer, and the L t h layer output the predicted class of a given input. The final decision of the classifier f θ can be written as follows for a given input x:

f L θ (x) = arg max c∈Y q θ (c|x) (2.2)
where q θ (•|x) = softmax( f L-1 θ (x)) can be seen as soft-probabilities, i.e., each component can be seen as the probability of the input belonging to this specific class.

Specificities of the Computer Vision problem.

In the case of CV, the inputs are three-dimensional images, and therefore the input space X ⊂ R w×h×c where w and h are the width and the height of the input image, and c is the number of channels ( 1for gray-scaled images, 3 for RGB images).

Regression. In the case of supervised regression, the input space X ⊂ R d , and the output space Y ⊂ R m . The main goal of regressors is to estimate a continuous quantity y ∈ Y using observations x.

Architectures.

Multi-Layer Perceptron (MLP). One of the first and most classical neural network

architectures is what we call the Multi-Layer Perceptron [START_REF] Haykin | Neural networks: a comprehensive foundation[END_REF]. It is a feedforward model, where the information goes only in one way (i.e., from the input to the output). It is composed of at least three layers of aggregated neurons. A layer, composed of multiple neurons, each connected to each neuron of the previous and the next layer, is called a fully-connected layer. Each neuron k in each layer l is characterized by its learnable weights w l ,k ∈ R d l -1 and biases b l ,k ∈ R. The relationship between the input of the neuron k in each layer l and its output can be summarized as follows:

f l ,k θ (x) = σ(w l ,k • f l -1 θ (x) + b l ,k ), (2.3)
where σ is a non-linear function called the activation function (for example, the ReLU function [NAIR and HINTON, 2010]). The global structure of an MLP is presented in

(x) = [ f l ,1 θ (x), f l ,2 θ (x), . . . , f l ,d l -1 θ (x), f l ,d l θ (x)
]. This type of architecture requires 1-dimensional inputs. To use it in image classification, it would therefore be necessary to transform images into 1-dimensional variables. However, the spatial dependencies of the images would be lost, and for complicated CV tasks, losing such important information could lead to poor results. This is one of the reasons why Convolutional Neural Networks (CNNs) have been considered.

Convolutional Neural Networks. One of the main characteristic of Convolutional

Neural Networks (CNNs) [LECUN and collab., 1989] is that they keep spatial dependencies of inputs. They are composed of two main parts. The first one extracts the important features of the input, and the second learns a good way to predict for an illustration of CNN's structure). A convolutional layer l is characterized by the filters {w l ,k } k∈{1,...,K} that compose it, their size and number K. The parameters of each filter are learnable, and the output of the layer is the convolution between its input and each of the filters of the layer. We can write the convolutional layer l 's operation as:

f l θ (x) = ( f l -1 θ * w l ,k )(x)
(2.4)

CNNs have gained much importance following the introduction of AlexNet [KRIZHEVSKY and collab., 2012], the first model to vastly outperform previous models on the ImageNet dataset. A few years later the ResNet architecture [HE and collab., 2016] was introduced to overcome some of the issue that extremely deep CNN could face during training (vanishing gradients, for example).

ResNets. The ResNet architecture [HE and collab., 2016] Transformers were originally introduced for neural machine translation [VASWANI and collab., 2017]. They are composed of multiple transformer cells. The transformer cell is based on self-attention [LIN and collab., 2017] and can be easily parallelized, contrarily to previous used recurrent cells (e.g., LSTM [HOCHREITER and SCHMID-HUBER, 1997], GRU [CHUNG and collab., 2014]) in seq2seq models [CHO and collab., 2014]. The input of a transformer in NLP applications is a sentence, where each word is one of its basic units. In CV applications, the basic unit is the pixel. However, applying the transformer using the pixel as a basic unit is too time and computa- Since their first successful use, ViTs have become the state-of-the-art architecture in classification tasks.

Other architectures. Even though our work mainly focuses on supervised vision classification, we would also like to mention a special case of unsupervised architectures: the AutoEncoders (AE) [BOURLARD and KAMP, 1988]. AE aims at finding a meaningful representation z of an input x without supervision. To do so, it will first learn an encoder function f θ 1 , where z = f θ 1 (x) then try to reconstruct x from z through the training of a decoder function f θ 2 , where x = f θ 2 (z). A good AutoEncoder will output as x a good approximation of x (i.e., f θ 2 ( f θ 1 (x)) ≈ x) and as z a meaningful representation of x. The quality of the reconstruction will be controlled by the minimization of a reconstruction loss, while the quality of the representation will be controlled by the minimization of a regularization term. This type of structure is used in many fields, such as data compression [CHENG and collab., 2018;THEIS and collab., 2017], feature extraction [START_REF] Gogna | Discriminative autoencoder for feature extraction: Application to character recognition[END_REF], image denoising [VINCENT and collab., 2008;YASENKO and collab., 2020], for examples.

Variational AutoEncoders [REZENDE and collab., 2014]. Variational AutoEncoder (VAE) is a particular case of AutoEncoders where instead of learning directly the representation z of x, the model learns the parameters of the probability distribution of the representations. The representation is then sampled from the distributions, and the decoder tries to reconstruct x from the sampled representation.

The training procedure is key to having reasonably good performing models, whether for classification or regression tasks. The following will present the classical training losses, evaluation metrics, optimizers, and regularization methods used in classical DL problems.

Training procedures. We will first focus on the classical training losses and risks (i.e., the training risk is the expectation over the data of the training loss), which are highly related to the evaluation metrics.

The goals of classifiers and regressors are widely different, therefore, their evaluation metrics are as well. In regression tasks, the goal is to approximate a quantity, usually continuous. An L p -norm difference between the estimated and the true quantity works well to quantify the performances of a regressor. Depending on the specificities of the problem, L 1 or L 2 -norms are usually employed. Defined and differentiable almost everywhere (not at 0 for the L 1 -norm), their expectations over the data are both suitable risks to train regression models and evaluate their performances.

Defining suitable training losses and risks is more challenging in classification tasks. The natural way to assess the performances of a given classifier is by counting the number of times it predicts the right class for given inputs. This metric is called CHAPTER 2. PRELIMINARIES the accuracy (acc.) and can be computed as follows:

acc. = 1

n sampl es i =1,••• ,n sampl es 1 { f θ (x i )=y i } , (2.5)
where y i is the true label of the input x i . Yet, this quantity is not suitable as a training risk due to its non-differentiability. The cross-entropy risk has been chosen as a differentiable surrogate of the accuracy. It works efficiently in classification tasks and has interesting connections to other information theory measures. The cross-entropy (CE) risk is defined as, for a given input classifier f θ with soft-probability distribution

q θ : CE(θ) = E p(x,y) -log q θ (y|x) . (2.6)
Once the training risk is chosen, the DL models are trained using classical optimizers, such as the Stochastic Gradient Descent (SGD) optimizer, the Adaptive Gradient Descent (AdaGrad), the Adaptive Momentum (Adam), or the Root Mean Square Propagation (RMS Prop.) optimizer, chosen depending on the specificities of the problem. To avoid overly fitting the training data and generalizing poorly on new incoming input (i.e., overfitting), a variety of techniques can be used. The most widely-spread ones are Dropout (randomly putting input's components, along with some weights, to 0 to force the model to minimize its dependence on specific components) and weight-decay (forcing the weight of each layer to remain small).

Now that we have discussed deploying Deep Neural Networks, we will focus on how to attack them.

Attacking Neural Networks

Global adversarial problem. SZEGEDY and collab. [2014a] define the adversarial problem as: arg min

x ′ ∥x ′ -x∥ p s.t. f θ (x ′ ) = t (2.7) x ′ ∈ [0, 1] d ,
where t is either the target class (equivalent to targeted attacks) or any class different from the original label y (untargeted attacks). The condition x ′ ∈ [0, 1] d means that we want the adversarial example to represent an image still. In Figure 2.5, the effect of an adversarial example on ImageNet, where the targeted classifier is a GoogLeNet, is represented. This problem is challenging to solve. Therefore, different relaxations arg min

x ′ c • ∥x ′ -x∥ p + L (x, x ′ , t ; θ) (2.8) s.t. x ′ ∈ [0, 1] d ,
where L (x, x ′ , t ; θ) is the objective of the attacker, and c > 0 is to find. It should be carefully chosen so that minimizing L (x, x ′ , t ; θ) will imply that f θ (x ′ ) = t .

Another common relaxation was proposed by MADRY and collab. [2018]:

arg max

x ′ :∥x ′ -x∥ p <ε L (x, x ′ , t ; θ).
(2.9)

In this case, we no longer try to minimize the L p -norm of the difference between natural and adversarial sample, but we define a maximal allowed perturbation ε given a specific L p -norm.

Whitebox attacks

Fast Gradient Sign Method (FGSM). One of the first and simplest method to generate adversarial examples is the Fast Gradient Sign Method (FGSM) [GOODFELLOW and collab., 2015]. The adversarial example x ′ is generated thanks to:

x ′ = x -α sgn∇ x L (x, x ′ , t ; θ), (targeted),
(2.10)

x ′ = x + α sgn∇ x L (x, x ′ , y; θ), (untargeted), (2.11)
where α ≤ ε is the amplitude of the perturbation, and sgn is the sign function.

This attack scheme is designed to be fast but not necessarily potent.

Projected Gradient Descent. An iterative version of FGSM, called Projected

Gradient Descent (PGD) [MADRY and collab., 2018], has been designed such that:

x ′(0) = x + n, n ∼ Unif([-ε; ε]),
(2.12) and i ) , y; θ), (untargeted).

x ′(i +1) = x ′(i ) -α sgn∇ x ′(i ) L (x, x ′(i ) , t ; θ), (targeted), (2.13) x ′(i +1) = x ′(i ) + α sgn∇ x ′(i ) L (x, x ′(
(2.14)

At each step of the algorithm, the condition x ′ ∈ [0, 1] d is enforced using clipping.

The PGD method has been proven to be both faster and more efficient than the FGSM method. It, therefore, has been widely used. However, PGD has a few issues that can lead to underperforming attacks, as explained in [CROCE and HEIN, 2020b].

Basic Iterative Method. As PGD, the Basic Iterative Method (BIM) [KURAKIN and collab., 2018] is an iterative extension of FGSM. The main difference is that contrary to the initialization of the PGD algorithm, the BIM method starts at the natural sample.

Auto-PGD. According to CROCE and HEIN [2020b], PGD has three main flaws: • The step size is fixed.

• It is agnostic of the budget, i.e., the algorithm does not know how many steps it can perform.

• It is unaware of the trend, i.e., it does not know if the direction it is taking is actually improving the attack objective.

To overcome those issues, they introduced a variant of the PGD algorithm called Auto-PGD (APGD).

It is composed of two phases: one exploration phase, where the algorithm searches for good initial points, and one exploitation phase, where the algorithm tries to maximize the attack strength according to the knowledge accumulated so far.

The APGD method falls to being the PGD algorithm with two main differences.

The first one is the computation of the perturbation at each step i :

x′(i+1) = x ′(i ) + η (i ) ∇ x ′(i ) L (x, x ′(i ) , y; θ), (2.15) x ′(i +1) = x (i ) + α * (x ′(i +1) -x ′(i ) ) + (1 -α) * (x ′(i ) -x ′(i -1) ), (2.16)
where η (i ) corresponds the learned step size at step i . Equation 2.15 is the PGD algorithm, and Equation 2.16 incorporates a momentum with α controlling the influence of the past.

The second difference is that the step size η (i ) is learnt and no longer fixed. To ensure that the algorithm creates stronger attacks, events are defined during which, if the previous iterations did not improve the attacker objective, the step size is modified, with a restart to the best previous point.

These two modifications of the PGD algorithm tend to create more potent attacks in a similar amount of time.

DeepFool. DeepFool (DF) is a method introduced by MOOSAVI-DEZFOOLI and collab. [2016]. It is an untargeted iterative method based on the gradient of a loss with Figure 2.7: FAB method for a linear binary classifier. In blue is the distance between the original example x orig and the projection of the current iteration x (i ) onto the hyperplane, in green is the impact of the step towards the original sample, and in red is the new distance.

The left figure represents the FAB algorithm using projections onto the hyperplane, while the right one represents the FAB algorithm using projections on the hyperplane plus an extrapolation step (to go over the boundary). Source: [CROCE and HEIN, 2020a], Fig. 1 respect to the input. The basic principle of the DF method is presented in Figure 2 and the projection of the example at the current iteration x (i ) (in blue in Figure 2.7).

Then, they compute a step towards the original point (in green in Figure 2.7) to get the new sample x (i +1) . The distance between x orig and x (i +1) is represented in red.

Carlini and Wagner's attack. The Carlini and Wagner's attack (C&W) was introduced by CARLINI and WAGNER [2017]. It can be either a targeted or an untargeted attack. They try to solve the original optimization problem (cf. Equation 2.7).

To that extend, they introduce a function g such that f θ (x ′ ) = t if and only if g (x ′ ) ≤ 0. Since the condition can be seen as a minimization problem, we can rewrite the problem as: (2.17) where c ≥ 0 is an hyperparameter.

min x ′ ∥x ′ -x∥ p + c g (x ′ ) s.t. x ′ ∈ [0, 1] d ,

CHAPTER 2. PRELIMINARIES

In their paper, the authors introduce multiple g functions, but the one they recommend for the untargeted case under L ∞ -norm constraint is: (2.18) where z = f L-1 θ (x), (z) + = max(0, z), z i is the i t h component of z, and z y is the component of z corresponding to the true class of x. To remove the condition on the norm constraint for p > 1, they use the change-of-variable approach with a hyperbolic tangent transformation.

g (x) = (z y -[max i ̸ =y z i )]) + ,
Despite creating potent attacks, the C&W algorithm tends to be highly computationally expensive. They propose to compute the gradient of each class with respect to each component of the input to extract the sensitivity direction. Then, a saliency map is computed to select the main directions, i.e., the directions in which modifying the pixel will impact the classification the most. The saliency map can be written as:

Jacobian-based

S(x, t )[i ] = 0, if ∂z t ∂x i (x) < 0 or j ̸ =t ∂z j ∂x i (x) > 0, ∂z t ∂x i (x)| j ̸ =t ∂z j ∂x i (x)|, otherwise,
where z = f L-1 θ (x) is the logit of x according to the specific model parametrized by θ, x i is the i t h component of x, z j its j t h component of z and z t the component of z corresponding to the targeted class t .

JSMA has been extended to become a blackbox attack [PAPERNOT and collab., 2017] where, instead of crafting the samples on the targeted model, they train a substitute model to attack and use the transferability of neural networks to attack the targeted system.

Blackbox attacks

As previously mentioned, whitebox attackers can create an extremely harmful attack.

However, they require much knowledge about the targeted system: they need not only the output of the model but also access to the entire model and its gradients.

To bypass this necessity, it is possible to craft attackers that rely on less information.

L (x, x ′ , y; θ) with x ′ = T(x; δu, δv, φ), (2.20)
where T(x; δu, δv, φ) is the transformation applied to x. The transformation can be written, for a given pixel position (u, v) of the image x, the adversarial new position

(u ′ , v ′ ) is: u ′ v ′ = cos φ -sin φ sin φ cos φ u v + δu δv .
(2.21)

Hop Skip Jump The Hop Skip Jump attack (HOP) [START_REF] D'amour | Underspecification presents challenges for credibility in modern machine learning[END_REF] is a blackbox attack based on gradient estimation. In this case, the attack has only access to whether the sample is rightfully or wrongfully classified. The method is iterative, and each iteration i can be divided into four steps:

• Binary search to approach the decision boundary.

• Estimation of the gradient at the decision boundary and computation of the direction of the gradient v (i ) .

• Computation of the minimum step size η (i ) .

• x ′(i +1) = x ′(i ) + η (i ) * v (i ) .

HOP is the attack presented here that requires the least amount of knowledge about the targeted system to attack.

CHAPTER 2. PRELIMINARIES

All of the aforementioned methods are only a small subset of all the possible methods to create adversarial examples. Given the wide variety of choices, how can we have a common method to evaluate and compare defensive methods ?

AutoAttack

When the community began to have a huge interest in defensive methods, the question of how to evaluate and compare the methods arose. To answer this question CROCE and HEIN [2020b] introduced AutoAttack.

AutoAttack is a collection of 4 attacks based on the principle of the worst-case scenario.

The four chosen attacks are as follows. First, they use two versions of the APGD algorithm. One where the attacker's objective is the adversarial cross-entropy, and another using the Difference of Logits Ratio (DLR) objective defined as:

L (x, x ′ , t ; θ) = DLR(x ′ , t ) = - z ′ y -z ′ t z ′ π 1 -(z ′ π 3 + z ′ π 4 )/2 , (2.22)
where

z ′ = f L-1 θ (x ′
) are the logits of x ′ , z ′ t is the logit of the targeted class, z ′ y is the logit of the original class, and π represents the ordering of the components of z ′ in decreasing order. The goal is to push the classification of x ′ to t rather than y.

The two final attacks are FAB and SA.

To aggregate the influence of the four attacks, CROCE and HEIN [2020b] proceed as follows. An adversarial example is deemed successful if at least one of the four considered attacks is successful. In other words, the defense is deemed successful for a specific clean input x if none of the four methods can find a successful adversarial sample based on x that fools the defense.

Since its introduction in 2020, AutoAttack has become the reference to compare robust methods. A year later, CROCE and collab. [2020] introduced RobustBench to rank the different defensive schemes according to their performances when evaluated using AutoAttack.

It is clear from all of this that there exist plenty of different methods to attack neural networks and that protecting against them is not straightforward. In the following, we will see the main techniques to create defenses to protect DL-based systems against threats.

Protecting Neural Network's Decisions

Recently, several works focused on improving the robustness of neural networks by investigating various defense mechanisms.

Robustness: different defense mechanisms

Adversarial robustness can be based on plenty of different mechanisms.

Randomness.

To build efficient defenses, some papers proposed to leverage randomness. Randomness could be applied at different levels of the system. It is possible to add randomness at the input level [XIE and collab., 2017], at the output of each hidden layer [LIU and collab., 2018a], or directly to the parameters of the model using a Bayesian Neural Network [LIU and collab., 2018b].

Distillation. We could also mention distillation, initially introduced in [HINTON and collab., 2015], and further studied in [PAPERNOT and collab., 2016a] to increase a model's robustness. The idea of distillation is to use a large DNN (the teacher) to train a smaller one (the student), which can perform with similar accuracy while utilizing a temperature parameter to reduce sensitivity to input variations. The resulting defense strategy may be efficient for some attacks but can be defeated with the standard C&W attack. and collab., 2018]. We will therefore focus on this particular defensive scheme, which is the most popular strategy for enhancing robustness.

Adversarial

Note that, to overcome the lack of guarantees on the task performance beyond standard evaluation metrics, a new line of work called Certified Robustness has emerged. The certified defenses aim at training classifiers whose predictions at any input feature will remain constant within a set of neighborhoods around the original input, through different mechanisms: randomized smoothing [CAO and GONG, 2017;COHEN and collab., 2019;LECUYER and collab., 2019;LIU and collab., 2018a], relaxation and duality [RAGHUNATHAN and collab., 2018b;WONG and collab., 2018],

constraining the global [CISSE and collab., 2017;GOUK and collab., 2021] or local [HEIN and ANDRIUSHCHENKO, 2017] Lipschitz constant of the model, mixed integer linear programming [BUNEL and collab., 2018;LOMUSCIO and MAGANTI, 2017], or adding complementary certification mechanisms to robust training [RAGHUNATHAN and collab., 2018a;WONG and KOLTER, 2020]. Although these methods are promising, they either do not scale to high-dimensional datasets and models or do not achieve SOTA yet.

Adversarial training

As previously mentioned, Adversarial Training has been introduced by GOODFELLOW and collab. [2015]. It is based on an attack-defense scheme. The attacker aims at creating perturbed inputs by maximizing a loss to fool the classifier, while the CHAPTER 2. PRELIMINARIES defender's goal is to classify those attacked inputs correctly.

The robust optimization problem solved can be written as follows.

min θ E p(x,y) max x ′ :∥x ′ -x∥ p ≤ε ℓ(x ′ , y; θ) , (2.23)
where ε denotes the maximal distortion allowed in the adversarial examples according to the L p -norm, and ℓ(x, x ′ , y; θ) is the training loss. Since the exact solution to the above inner max problem is generally intractable, a relaxation is proposed by generating an adversarial example.

The inner attacker design is vital to AT since it has to create meaningful attacks.

The chosen mechanism to craft an adversarial sample was the FGSM attack. However, it w quickly shown that, although it provided some successes, it was possible to defeat this defense with stronger attackers [TRAMÈR and collab., 2017].

Later, MADRY and collab. where ACE is the Adversarial Cross-Entropy risk,

ACE(θ) . = E p(x,y) max x ′ :∥x ′ -x∥ p ≤ε -log q θ (y|x ′ ) . ( 2
L TRADES (θ) . = E p(x,y) -log q θ (y|x) + λ E p(x) max x ′ :∥x ′ -x∥ p ≤ε KL(q θ (•|x)∥q θ (•|x ′ )) , (2.26)
where

KL(q θ (•|x)∥q θ (•|x ′ )) . = E q θ (y|x) log q θ (y|x) q θ (y|x ′ ) .
(2.27)

MART. WANG and collab. [2019] uses a robustness regularizer that considers the misclassified inputs and boosted losses. They consider two sets: the set where the original input is rightfully classified and the set where it is misclassified. For each set, they propose a regularizer, equal to 1{f θ (x

′ ) ̸ = y} + 1{f θ (x ′ ) ̸ = f θ (x)}, meaning
that we want, at the same time, the adversarial example to be rightfully classified, and the natural and adversarial examples to be classified in the same way. Given

that, for rightfully classified clean examples, 1{f θ (x ′ ) ̸ = y} = 1{f θ (x ′ ) ̸ = f θ (x)}, the total considered risk is: L MART (θ) = E p(x,y) 1{f θ (x ′ ) ̸ = y} + λ 1{f θ (x) ̸ = y} • 1{f θ (x ′ ) ̸ = f θ (x)} , (2.28)
where λ is a hyperparameter controlling the trade-off between rightfully classifying

the adversarial examples and classifying the natural and adversarial examples in the same way.

Since the indicator's function is not differentiable, they relax the three terms as follows. The first one -1{f θ (x ′ ) ̸ = y} -is approximated as the boosted crossentropy(BCE), which is equal to:

BCE(x ′ , y; θ) = -log q θ (y|x ′ ) -log(1 -max c̸ =y q θ (c|x ′ )).
(2.29)

The second term -

1{f θ (x ′ ) ̸ = f θ (x)} -is approximated as the Kullback-Leibler divergence (Eq. (2.27)), i.e. 1{f θ (x ′ ) ̸ = f θ (x)} ≈ KL(q θ (•|x)∥q θ (•|x ′ )).
Finally, 1{f θ (x) ̸ = y} is approximated by 1q θ (y|x), since it will be large for misclassified inputs and small for rightfully classified inputs.

So, the MART risk can be written as:

L MART (θ) = E p(x,y) [ max x ′ :∥x ′ -x∥ p ≤ε BCE(x ′ , y; θ) + λ (1 -q θ (y|x)) • KL(q θ (•|x)∥q θ (•|x ′ ))].
(2.30) It should be noted that the main disadvantage of adversarial training-based methods remains the required computational expenses.

Improvement

Ensuring the Input's Integrity

Inspired by the concept of rejection channels [START_REF] Chow | An optimum character recognition system using decision functions[END_REF], which was proposed over and collab., 2017;KHERCHOUCHE and collab., 2020;MA and collab., 2018]. The extracted features can be computed either directly on the images [KHERCHOUCHE and collab., 2020] or extracted at the targeted network's layer [CARRARA and collab., 2018;LEE and collab., 2018;LU and collab., 2017;METZEN and collab., 2017]. The supervised detection methods can rely on statistical characteristics linking them to in-training or out-of-training distributions/manifolds [FEINMAN and collab., 2017;GROSSE and collab., 2017;[START_REF] Li | Adversarial examples detection in deep networks with convolutional filter statistics[END_REF]MA and collab., 2018]). All these methods depend on the defender's knowledge of the threats it will face.

Unsupervised Detection. Knowledge about the attacker is not always available to the defender. To overcome this lack of information, unsupervised detection methods do not rely on prior knowledge of attacks and only learn from clean data at the time of training [MENG and CHEN, 2017;XU and collab., 2018] PANG and collab., 2018] or the rejection option [ALDAHDOOH and collab., 2021a;SOTGIU and collab., 2020].

In our work, we decided to focus on methods that do not modify the underlying classifier to protect. In the following, we will present the detection methods we will compare our work with in more details.

Supervised detection methods

As previously mentioned, supervised detection methods rely on the knowledge and availability of adversarial samples. They can be separated into two categories: the attack-aware methods, where one detector is trained per specific attack, and the blind-to-attack methods, where a single detector is trained to detect all the threats. or from another one. The estimation of LID is computed as the inverse of the mean of the log of the distance between a given point to its k nearest neighbors. If a given point is similar to the training distribution, then its distance to either of its k nearest neighbors will always be quite close, and the LID approximate will therefore be close to 0. However, if a sample is not quite similar to the training distribution, its distance .

Local Intrinsic Dimensionality (LID). The Local Intrinsic

to at least one of its nearest neighbors will likely be big, and the LID approximate will increase.

Following this idea, in practice, three versions of each natural sample are used:

the clean one, a noisy version, and an attacked one. The LID approximate for each of those points to the clean distribution will be computed at the output of each layer.

Those variables will then be used to train a detector that will distinguish between adversarial and normal samples (either clean or noisy samples).

Each strategy to craft adversarial samples can have very different LID characteristics, a detector per type of threat is therefore necessary. LID therefore lies in the attack-aware category.

Kernel Density and Bayesian Uncertainty (KD-BU). The Kernel Density and

Bayesian Uncertainty (KD-BU) method [FEINMAN and collab., 2017] also relies on the idea that adversarial samples lie off the original data manifold. To detect adversarial samples, they first perform a kernel density estimate at the last hidden layer level to detect samples that are far from the original manifold. Then, a bayesian uncertainty estimate is computed to detect when points lie in low-confidence regions of the input space. Both of those characteristics are later used to train a detector that distinguishes between natural and adversarial examples. Once again, the kernel density estimates and the bayesian uncertainty values for different types of attacks can differ a lot, therefore, this method has been created to be attack-aware.

Natural Scene Statistics (NSS). The Natural Scene Statistics method

[ KHERCHOUCHE and collab., 2020] relies on the extraction of the natural scene statistics at the image level. Natural scene statistics are statistics that will be very different CHAPTER 2. PRELIMINARIES for natural and attacked images. Indeed, applying the natural scene statistics will output an image with meaning for clean images. However, for attacked samples, the output image will have no meaning. In Figure 2.8 is presented an example of the extraction of natural scene statistics on a natural sample and various adversarial ones, where we clearly see that, while the NSS image still represents a leopard, the ones for the adversarial examples do not. The Natural Scene Statistics extraction is then used to train a detector to distinguish between natural and attacked samples.

To overcome the need to have a specific detector per attack, NSS decided to train their detector using the natural scene statistics of various attacks. It therefore lies in the blind-to-attack category. For a new input, they look at the reconstruction error. If it is small, then the example is clean, else it is adversarial.

Unsupervised detection methods

The reformer, also an autoencoder, will, given an input, try to find an approximation of it that is on or close to the original manifold. This projection is then fed to the underlying classifier to estimate the class.

The training of the two necessary autoencoders makes this method rather long and computationally and memory expensive to deploy but extremely fast to test.

Review of the Smart Grid Case

In 2015, the United Nation organization edited 17 goals to change our world. Amongst those goals, goal 7 aims at ensuring access to affordable, reliable, sustainable and modern energy for all. We would need to widely modify our grid's structure to include However, smart grid systems can be sensitive to attacks, and more specifically to False Data Injection Attacks (FDIAs) [LIU and collab., 2011].

In the following, we will provide a quick review of Smart Grid systems, and more specifically of the state estimation problem, how to attack it and how to protect it against attacks.

The state estimation problem

Let x = |x|e i arg x ∈ C n be the state vector (latent variables) of the power grid, assumed to be random with a prior probability density function (pdf) p(x), and y = |y|e i arg y ∈ C m the measurement vector. In general, these are related through the following nonlinear equation:

y = h(x) + z, (2.31)
where h is the measurement function (dependent on the grid topology, line impedances, etc.) and z ∈ C m is additive noise. This is called the AC model in the literature [ABUR and EXPOSITO, 2004;GIANNAKIS and collab., 2013].

Usually, under certain assumptions about the grid and its operating point, the CHAPTER 2. PRELIMINARIES problem can be linearized as follows:

y = Hx + z, (2.32)
where H ∈ C m×n is the linearized Jacobian measurement matrix. This is called the DC model.

We define a state estimator as a function

f φ : C m → C n such that x = f φ (y).
State estimators are crucial components for the functioning of smart grid systems, but they are known to be sensitive to faulty samples. Bad data detectors are usually implemented to overcome this issue. They are a-posteriori detectors based on the l p -norm (where typically p = 2) of the residual vector

r = y -h(x). (2.33)
Concretely, the sample is detected as faulty when the ∥r∥ p > τ, where τ is a threshold chosen appropriately to control the trade-off between missed and false detections.

Nevertheless, bad data detectors are insufficient to overcome the problem of welldesigned attacks. In particular, a theoretical bound on the maximal number of meters that one can attack in order to remain undetected is derived in JIN and collab. [2018].

Attacking and defending state estimator

One of the most critical types of cyber-attacks are FDIAs. In LIU and collab.

[2011],

the authors present various methods to generate such corrupted samples to bypass the bad data detector under the DC assumption. Let us consider

y a = y + a, (2.34)
where a is the attack vector and y a the attacked observation. It has been showed that, to remain undetected, the a should be a combination of the column vectors of H, i.e., a = Hc, for all c. The specific values for the vector c is imposed by the considered scenario, i.e. the goal of the attack.

Most of the research has been done according to the DC model assumption [DÁN and SANDBERG, 2010;KOSUT and collab., 2010;LIU and collab., 2011;PASQUALETTI and collab., 2011;SANDBERG and collab., 2010;XIE and collab., 2010;YUAN and collab., 2011], which leads to a simple linear model for the measurements as a function of the state. LIANG and collab. [2016] present a comprehensive review of the security problem under the DC-model assumption.

Crafting an FDIA under the AC model (Equation 2.31) is a more complex problem.

A few works have focused on it, as [HUG and GIAMPAPA, 2012;JIN and collab., 2018;KEKATOS and collab., 2017;LIANG and collab., 2014;TEIXEIRA and collab., 2015;ZHU CHAPTER 2. PRELIMINARIES and GIANNAKIS, 2012], which considers the nonlinear relation between the state and the measurements. Joint cyber and physical attacks have also been studied [SOLTAN and collab., 2016], leading to the study of detection and recovery of information from line failure under DC and AC assumption [SOLTAN and collab., 2018;SOLTAN and ZUSSMAN, 2018]. JIN and collab. [2018] presented the attacker's optimization problem as:

max x a ,a g (x a ) s.t. h(x a ) + z = y + a = y a (2.35) ∥a∥ 0 ≤ c,
where x a is the corrupted state, a is the attack vector, y a is the attacked observation, and g (•) is the objective of the attacker. The condition on the l 0 -norm of the attacked vector allows the attacker to bypass the bad data detector by attacking only a subset of sensors.

Notice that the attacker objective can be different based on its specific goal, for example:

• Target state attack, where g (x a ) = ∥x ax target ∥ 2 2 , which will put the corrupted state to the targeted value x target .

• Voltage collapse attack, where g (x a ) = ∥x a ∥ 2 2 , which will lead the estimator to believe that the voltage is low.

• State deviation attack, where g (x a ) = -∥x a -x∥ 2 2 , which will force the corrupted state to be different from the original predicted one.

Three main types of defense strategies have been developed in the literature.

The first one is the detection of the FDIAs [ KOSUT and collab., 2010;PASQUALETTI and collab., 2011], which can be used to discard the compromised measurements.

The second one focuses on protecting the communication channel between the meters and the control center using encryption, authentication and key management [DÁN and SANDBERG, 2010;TEIXEIRA and collab., 2015;WANG and LU, 2013]. Finally, Robust State Estimation (RSE) has been created to develop state estimations that are robust against bad data. CELIK and ABUR [1992]; KOTIUGA and VIDYASAGAR [1982]; MILI and collab. [1994]; ZHU and GIANNAKIS [2012] focused on the DC model while ZHANG and collab. [2017] investigated a AC-based solution that requires many relaxations.

The Fisher-Rao measure, and the data-depths

In the following, we will mathematically define the two objects we used during this Ph.D.

The Fisher-Rao distance (FRD)

Consider the family of probability distributions over the classes C . = q θ (•|x) : x ∈ X parametrized by x. Assume that the following regularity conditions hold [ATKINSON and MITCHELL, 1981]:

(i) ∇ x q θ (y|x) exists for all x, y and θ ∈ Θ;

(ii) y∈Y ∇ x q θ (y|x) = 0 for all x and θ ∈ Θ;

(iii) G(x) = E Y∼q θ (•|x) ∇ x log q θ (Y|x)∇ ⊺ x log q θ (Y|x) is positive definite for any x and θ ∈ Θ.
The variance of the differential form ∇ ⊺ x log q θ (Y|x)d x can then be interpreted as the square of a differential arc length d s 2 in the space C , which yields

d s 2 = 〈d x, d x〉 G(x) = d x ⊺ G(x)d x.
(2.36) Thus, G, which is the Fisher Information Matrix (FIM), can be adopted as a metric tensor. We now consider a curve γ : [0, 1] → X in the input space connecting two arbitrary points x and x ′ , i.e., such that γ(0) = x and γ(1) = x ′ . Notice that this curve induces the following curve in the space C : q θ (•|γ(t )) for t ∈ [0, 1]. The Fisher-Rao distance between the distributions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ) will be denoted as d R,C (q θ , q ′ θ ) and is formally defined as:

d R,C (q θ , q ′ θ ) . = inf γ 1 0 d γ ⊺ (t ) d t G(γ(t )) d γ(t ) d t , (2.37)
where the infimum is taken over all piecewise smooth curves. This means that the FRD is the length of the geodesic between points x and x ′ using the FIM as the metric tensor. Several examples for simple families of distributions can be found in ATKINSON and MITCHELL [1981].

The data-depths

A data depth function, formally defined as: (2.38) where P (R d ) denotes the space of all probability distributions on R d , measures the centrality of any element in x ∈ R d w.r.t. a probability distribution P (respectively, a data set). It provides a center-outward ordering of points in the support of P and can be straightforwardly used to extend the notions of rank or order statistics to multivariate data. The higher D(x, P), the deeper x is in P.

D : R d × P (R d ) -→ [0, 1] , (x, P) -→ D(x, P),
Since data depth naturally way defines a non-parametric pre-order on R d w.r.t.

a probability distribution, it can be seen as a centrality-based alternative to the cumulative distribution function for multivariate data. Many different definition of data-depths, respecting Equation 2.38, has been proposed CHEN and collab.

[2015]; CUEVAS and collab. [2007]; KOSHEVOY and MOSLER [1997]; [START_REF] Liu | On a notion of data depth based on random simplices[END_REF]; RAMSAY and collab. [2019]; STAERMAN and collab. [2021]; [START_REF] Zuo | Projection-based depth functions and associated medians[END_REF], each presenting its own theoretical and practical properties.

However, several axioms have been developed throughout the recent decades a "good" depth function should satisfy DYCKERHOFF [2004]; [START_REF] Liu | On a notion of data depth based on random simplices[END_REF]; ZUO and SERFLING [2000]: CHAPTER 2. PRELIMINARIES

(D 1 ) (AFFINE INVARIANCE)

Chapter 2 Conclusion

This chapter first presented an overview of the Deep Learning background.

Then, we introduced the current state-of-the-art methods to attack neural networks. Later, we reviewed the existing methods to increase the models' robustness against adversarial threats. This will be useful to appreciate better the contribution of Part I. Then, we presented existing detection mechanisms we will compare against in Part II. Then, we presented quickly the state estimation problem for Smart Grids. Finally, we provided a quick definition of the two main tools we used during this thesis. Now that we have presented what exists in the literature, we will present our first contribution on leveraging the knowledge about the models' output space to increase their robustness. • In Chapter 3, we provide our solution to improve adversarial robustness through the use of a regularizer based on an information-geometric measure: the Fisher-Rao measure. The output space of a neural network forms a statistical manifold, and the decision is taken on it. To leverage this knowledge, we decided to use a measure that captures the distance between two outputs following the shape of the statistical manifold at hand: the Fisher-Rao distance. We derived a close-form of the Fisher-Rao distance in the case of a binary classification as well as for the multi-class scenario. On a toy example, we show that it is possible to achieve Pareto-optimum points unachievable using other regularizers.

The experimental results show that our regularizer consistently enhances the robustness of neural networks.

• In Chapter 4, we apply our method to the state estimation study in Smart Grids applications. Due to monitoring and service needs, Smart

Grid systems include a cyber component that makes them vulnerable to potential threats. In this work, we made use of a Variational AutoEncoder and the physical knowledge of the grid to train an unsupervised state estimator. We design a defensive scheme similar to a min-max problem to craft a robust state estimator using a similar method as the one used in Chapter 3. We experimentally show that it is possible to train robust state estimators for multiple electrical systems in the linearized case.

• In Chapter 4, we assume that the system in linear. Since this assumption might not be realistic for all the bus systems, we propose an extension of the previously introduced framework in Chapter 5. This extension to the non-linear case (i.e., under the AC model assumption) of the state estimation problem is a more general scenario given that all power systems are intrinsically non-linear.

Chapter 3

Adversarial Robustness via Fisher-Rao

Regularization Chapter 3 Abstract

This chapter addresses the first research question dedicated to increasing adversarial robustness. We present our first contribution to the scope of image data. One way to ensure that an image classifier remains trustworthy under threats is to force the model to classify similarly clean and attacked inputs. To do so, a successful line of work is to use a modified training procedure that will take into account perturbed samples via a custom risk. The output of a neural network forms a statistical manifold, and the decision mechanisms depend on this manifold. To leverage this knowledge, we decided to use an informationgeometric distance, i.e., the Fisher-Rao distance, that measures the distance between two possible outputs and to use it to train a robust classifier. In this work, we derived the closed form of the Fisher-Rao distance on both deep binary classifiers and multi-class deep classifiers. We later used this formula of the Fisher-Rao distance to define a regularizer that, when minimized, will force the resulting deep classifier to estimate the natural and attacked input in a similar manner. We experimentally showed that using this regularizer will allow the resulting model to achieve Pareto-optimum points impossible to reach when using SOTA regularizers. We also experimentally showed that using Fisher-Rao as a robust regularizer allowed us to increase the robustness of neural networks. COHEN and collab., 2019;CROCE and collab., 2019;GOWAL and collab., 2019;LECUYER and collab., 2019;LI and collab., 2019;MIRMAN and collab., 2018;WONG and collab., 2018;ZHANG and collab., 2020]. These methods aim at training classifiers whose predictions at any input feature will remain constant within a set of neighborhoods around the original input. However, these algorithms do not achieve state-of-the-art performance yet. Also, some approaches tend to rely on convex relaxations of the original problem [RAGHUNATHAN and collab., 2018;WONG and KOLTER, 2018] since directly solving the adversarial problem is not tractable. Although these solutions are promising, it is still not possible to scale them to high-dimensional datasets. Finally, we could mention distillation, initially introduced in HINTON and collab. [2015], and further studied in PAPERNOT and collab.

Contents

[2016]. The idea of distillation is to use a large DNN (the teacher) to train a smaller one (the student), which can perform with similar accuracy while utilizing a temperature parameter to reduce sensitivity to input variations. The resulting defense strategy may be efficient for some attacks but can be defeated with the standard Carlini-Wagner attack.

In this work, we will focus on the most popular strategy for enhancing robustness,

Summary of contributions

Our work investigates the problem of optimizing the trade-off between accuracy and robustness and advances state-of-the-art methods in very different ways.

• We derive an explicit characterization of the Fisher-Rao Distance (FRD) based on the information-geometric properties of the soft-predictions of the neural classifier. That leads to closed-form expressions of the FRD for the binary and multiclass cases (Theorem 1 and Theorem 2, respectively). We further relate them to well-known regularization metrics (presented in Proposition 1). is an iterative attack: the output at each step is the addition of the previous output and the sign of the loss gradient modulated by a fixed step size. The loss maximized in PGD is often the same loss that is minimized for the defense.

Robust defense loss. An essential choice of the defense mechanism is the robust loss used to attack and defend the network. Initially, GOODFELLOW and collab.

[2015]; MADRY and collab.

[2018] used the adversarial cross-entropy. However, it was shown that one way to improve adversarial training is through the choice of this loss.

TRADES [ZHANG and collab., 2019] It should be noted that the main disadvantage of adversarial training-based methods remains the required computational expenses. Nevertheless, as will be shown in Section 3.5, FIRE can significantly reduce them.

Background

We consider a standard supervised learning framework where x ∈ X ⊆ R n denotes the input vector on the space X and y ∈ Y the class variable, where Y := {1, . . . , M}. The unknown data distribution is denoted by p(x, y) = p(x)p(y|x). We define a classifier to be a parametric soft-probability model of p(y|x), denoted as q θ (y|x), where θ ∈ Θ are the parameters. This can be readily used to induce a hard decision:

f θ : X → Y with f θ (x) := arg max y∈Y q θ (y|x)
. Adversarial examples are denoted as x ′ = x + δ ∈ X , where ∥δ∥ ≤ ε for an arbitrary norm ∥ • ∥. Loss functions are denoted as ℓ(x, x ′ , y, θ) and the corresponding risk functions by L(θ). We also define the natural missclassification probability as P e (θ) . = P( f θ (X) ̸ = Y), the adversarial missclassification probability as

P ′ e (θ) . = P( f θ (X ′ ) ̸ = Y).

Adversarial learning

We provide some background on adversarial learning, focusing on adversarial defense's most popular proposed loss functions. Adversarial examples are slightly modified inputs that can fool a target classifier. Concretely, SZEGEDY and collab.

[2014] define the adversarial generation problem as:

x ′ = arg min

x ′ : ∥x ′ -x∥≤ε ∥x ′ -x∥ s.t. f θ (x ′ ) ̸ = y, (3.1)
where y is the true label (supervision) associated to the sample x. This formulation shows that the vulnerable points of a classifier are the ones close to its decision boundaries. Since this problem is difficult to tackle, it is commonly relaxed as follows [MADRY and collab., 2018]:

x ′ = arg max x ′ : ∥x ′ -x∥≤ε ℓ(x, x ′ , y, θ). (3.2)
Once adversarial examples are obtained, they can be used to learn a robust classifier as discussed next.

The adversarial problem has been presented in MADRY and collab. [2018] as follows:

min θ E p(x,y) max x ′ :∥x ′ -x∥ p ≤ε ℓ(x, x ′ , y, θ) , (3.3)
where ε denotes the maximal distortion allowed in the adversarial examples according to the l p -norm. Since the exact solution to the above inner max problem is generally intractable, a relaxation is proposed by generating an adversarial example using an iterative algorithm such as PGD.

Adversarial Cross-Entropy (ACE)

If we take the loss to be the Cross-Entropy (CE), i.e., ℓ(x, x ′ , y, θ) =log q θ (y|x ′ ), we obtain the ACE risk:

L ACE (θ) . = E p(x,y) max x ′ :∥x ′ -x∥≤ε
log q θ (y|x ′ ) .

(3.4)

TRADES

Later ZHANG and collab.

[2019] defined a new risk based on a trade-off between natural and adversarial performances, controlled through an hyperparameter λ. The resulting risk is the addition of the natural cross-entropy and the Kullback-Leibler (KL) divergence between natural and adversarial probability distributions:

L TRADES (θ) . = E p(x,y) max x ′ :∥x ′ -x∥≤ε -log q θ (y|x) + λ KL(q θ (•|x)∥q θ (•|x ′ )) , (3.5)
where It is worth to mention that the concept of statistics on manifolds is very different from manifold learning which is a branch of machine learning where the goal is to learn a latent manifold from valued data. In this paper, we are interested in the statistical manifold obtained when fixing the parameters θ of a DNN and changing its feature input. We consider the following statistical manifold:

KL(q θ (•|x)∥q θ (•|x ′ )) . = E q θ (y|x) log q θ (y|x) q θ (y|x ′ ) . ( 3
C . = q θ (•|x) : x ∈ X .
In particular, the focus is on changes in a neighborhood of a particular sample in an adversarial manner (i.e., considering a worst-case perturbation). Please notice that the statistical manifold is different from the loss landscape. The loss landscape is defined as the changes of the risk function with respect to changes in the model parameters (i.e., L(θ) vs θ), while the statistical manifold refers to the changes of the soft-probabilities of the classifier with respect to changes in the input (i.e., q θ (y|x) vs x). In order to understand the effect of a perturbation on the input, we first need to be able to capture the distance over the statistical manifold between different probability distributions, i.e., between two different feature inputs. That is precisely

CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION d R,C (q θ , q ′ θ ) q θ (•|x) q θ (•|x ′ ) C dEcli d (qθ , q ′ θ )
Figure 3.1: Illustration of FRD between two distributions q θ = q θ (•|x) and

q ′ θ = q θ (•|x ′ ) over the statistical manifold C . ©2022 IEEE.
what the FRD computes, as illustrated by the red curve in Figure 3.1. It is worth to mention that FRD can be very much different from the euclidean distance since the later does not depend on the shape of the manifold. For a formal mathematical definition of FRD and a short review of basic concepts in information geometry, we refer the reader to Subsection 3.6.1.

As discussed in Section 3.2, the robustness of a classifier is related to the distance between natural examples and the decision boundaries (i.e., points x such that q θ (y|x) ≈ q θ (y ′ |x) for y ̸ = y ′ ). In fact, if a natural example is far from the decision boundaries, a norm-constrained attack will clearly fail (in this case, the optimization problem in Equation 3.1 will be infeasible). Since the decision boundaries are given by the soft-probabilities q θ (y|x), this can be equivalently studied by analyzing the shape of the statistical manifold C (which should not be confused with the loss landscape). In fact, if q θ (y|x) is relatively flat (i.e., does not change much) with respect to perturbations of x around x 0 , it is clear that adversarial perturbations will not modify the classifier decision at this point. In contrast, if q θ (y|x) changes sharply with perturbations of x around x 0 , an adversarial can easily leverage this vulnerability to fool the classifier. This notion of robustness is related to the Lipschitz constant of the network, as discussed in various works (e.g., CISSE and collab. [2017]). To illustrate these ideas clearly, let us consider the logistic regression model q θ (y|x)

= 1/[1 + exp(-y θ ⊺ x)],
where n = 2 and Y = {-1, 1}, as a simple example. One way to visualize the statistical manifold C is to plot q θ (1|x) as a function of x (since q θ (-1|x) = 1q θ (1|x), this completely characterizes the manifold). This is shown in Figure 3.2a for the value of θ which minimizes the natural missclassification probability P e under a conditional Gaussian model for the input x (see Section 3.4 for details). As can be seen, the manifold is quite sharp around a particular region of X . This region corresponds to the neighborhood of the points for which θ ⊺ x ≈ 0 as x is perturbed in the direction of θ. Therefore, we can say that this model is clearly non-robust, as its output can be significantly changed by small perturbations on the input. Consider now the same model but with the values of θ obtained by minimizing the adversarial missclassification probability P ′ e . As can be seen in Figure 3.2b, the statistical manifold is much flatter than in Figure 3.2a, which means that the model is less sensitive to adversarial perturbations on the input. Therefore, it is more robust.

Let us now consider the FRD of the two models around the point x = 0, which gives a point that lies in the decision boundary, by letting δ = x ′x vary in the ℓ ∞ ball B ∞,ε = {δ : ∥δ∥ ∞ ≤ ε}, with ε = 0.1. Figure 3.3a displays the FRD for the parameters θ which minimize the misclassification error probability P e , and Figure 3.3b shows the FRD for the parameters θ which minimize the adversarial misclassification error probability P ′ e . Clearly, the abrupt transition of q θ (1|x) in Figure 3.2a corresponds to a sharp increase on the FRD as ∥δ∥ ∞ increases. On the contrary, for a flatter manifold as in Figure 3.2b, the FRD increases much more slowly as ∥δ∥ ∞ increases.

This example shows how FRD reflects the shape of the statistical manifold C .

Our goal in this work is to use the FRD to control the shape of the statistical manifold by regularizing the misclassification risk.

The rest of this section is organized as follows. We begin by introducing the FIRE risk function, which is our main theoretical proposal to improve the robustness of neural networks. We continue with the evaluation of the FRD given by Equation 3.26 for the binary and multiclass classification frameworks and provide some exciting properties and connections with other standard distances and well-known information divergences.

The FIRE risk function

The main proposal of this paper is the FIRE risk function, defined as follows:

L FIRE (θ) . = E p(x,y) max x ′ :∥x ′ -x∥≤ε -log q θ (y|x) + λ • d 2 R,C (q θ • |x), q θ (•|x ′ ) , (3.7)
where λ > 0 controls the trade-off between natural accuracy and robustness to the adversary.

In Figure 3.4, we show the shape of the statistical manifold C as λ is varied for the logistic regression model discussed in Subsection 3.3.1 (see also Section 3.4).

Notice that when no FRD regularization is used, the manifold in Figure 3. In what follows, we derive closed-form expressions of the FRD for general classification problems. However, for the sake of clarity, we begin with the binary case.

FRD for the case of binary classification

Let us first consider the binary classification setting, in which X ⊆ R n and Y = {-1, 1}

are input and label spaces, respectively. Consider an arbitrary given model:

q θ (y|x) = 1
1 + e -h θ (x)y .

(3.8)

Here h θ (x) represents an arbitrary parametric representation or latent code of the input x. As a matter of fact, we only need to assume that h θ is a smooth function.

The FRD for this model can be computed in closed-form, as shown in the following result. The proof is relegated to Subsection 3.6.2.

Theorem 1 (FRD for binary classifier). The FRD between soft-predictions q θ ≡ q θ (•|x) and q ′ θ ≡ q θ (•|x ′ ), according to Equation 3.8 and corresponding to inputs x and x ′ , is given by d R,C (q θ , q ′ θ ) = 2 arctan(e h θ (x ′ )/2 )arctan(e h θ (x)/2 ) .

(3.9)

For illustration purposes, Figure 3.5a shows the behavior of the FRD with respect to changes in the latent code compared with the Euclidean distance. It can be observed that the resulting FRD is rather sensitive to variations in the latent space when h θ (x) ≈ 0 while being close to zero for the region in which |h θ (x)| is large and

|h θ (x ′ )| ≪ |h θ (x)|.
This asymmetric behavior is in sharp contrast with the one of the Euclidean distance. However, these quantities are related as shown by the next proposition.

Proposition 1 (FRD vs. Euclidean distance). The Fisher-Rao distance can be bounded as follows:

d R,C (q θ , q ′ θ ) ≤ 1 2 h θ (x ′ ) -h θ (x) , (3.10)
for any pair of inputs x, x ′ ∈ X .

The proof of this proposition is relegated to Subsection A.3.1.

CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION

Logistic regression:

A particular case of significant importance is that of logistic regression:

h θ (x) = θ ⊺ x.
In this case, the FRD reduces to: d R,C (q θ , q ′ θ ) = 2 arctan(e θ ⊺ x ′ /2 )arctan(e θ ⊺ x/2 ) .

(3.11)

A first-order Taylor approximation in the variable δ = x ′x and maximization over δ

such that ∥δ∥ ≤ ε yields d R,C (q θ , q ′ θ ) ≈ 1 2 cosh(θ ⊺ x/2) ε∥θ∥ * , (3.12)
where ∥ • ∥ * is the dual norm of ∥ • ∥, which is defined as ∥z∥ * .

= sup{|z ⊺ w| : ∥w∥ ≤ 1}.

Therefore, in this case, we obtain a weighted dual norm regularization on θ, with the weighting being large when θ ⊺ x is close to zero (i.e., points with large uncertainty in the class assignment), and being small when |θ ⊺ x| is large (i.e., points with low uncertainty in the class assignment). An even more direct connection between the FRD and the dual norm regularization on θ can be obtained from Proposition 1, which leads to

d R,C (q θ , q ′ θ ) ≤ 1 2 |δ ⊺ θ| ≤ 1 2 ε∥θ∥ * .
(3.13) Equation 3.13 formalizes our intuitive idea that ℓ p -regularized classifiers tend to be more robust. This is in agreement with other results, e.g., TORKAMANI and LOWD [2014].

FRD for the case of multiclass classification

Consider the general M-classification problem in which Y = {1, . . . , M}, and let Theorem 2 (FRD multiclass classifier). The FRD between soft-predictions q θ ≡ q θ (•|x) and q ′ θ ≡ q θ (•|x ′ ), according to Equation 3.14 and corresponding to inputs x and x ′ , is

q θ (y|x) = e h y (x,θ) y ′ ∈Y e h y ′ (x,θ) , ( 3
CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION
given by d R,C (q θ , q ′ θ ) = 2 arccos y∈Y q θ (y|x)q θ (y|x ′ ) .

(3.15)

Remark. Although not obvious, the FRD for the multiclass case (Equation 3.15) is indeed consistent with the FRD for the binary case (Equation 3.9), i.e., they are equal for the case M = 2 (for further details the reader is referred to Subsection 3.6.3).

Comparison between FRD and KL divergence

The Fisher-Rao distance (Equation 3.15) has some interesting connections with other distances and information divergences. We are particularly interested in its relation with the KL divergence, which is the adversarial regularization mechanism used in the TRADES method [ZHANG and collab., 2019]. The next theorem summarizes the mathematical connection between these quantities. The proof of this theorem is relegated to Subsection A.3.2.

Theorem 3 (Relation between FRD and KL divergence). The FRD between softpredictions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ), given by Equation 3.15 is related to the KL divergence through the inequality:

1 -cos d R,C (q θ , q ′ θ ) 2 ≤ 1 2 KL(q θ , q ′ θ ), (3.16)
which means that the KL divergence is a surrogate of the FRD. In addition, it can also be shown that the KL divergence is a second-order approximation of the FRD, i.e.,

KL(q

θ ∥q ′ θ ) = 1 2 d 2 R,C (q θ , q ′ θ ) + O (d 3 R,C (q θ , q ′ θ )), (3.17)
where O (•) denotes big-O notation.

The above result shows that the KL is a weak approximation of the FRD in the sense that it gives an upper bound and a second-order approximation of the geodesic distance. However, in general, we are interested in distances over arbitrarily distinct softmax distributions, so it is clear that the KL divergence and the FRD can behave very differently. In fact, only the latter measures the actual distance on the statistical manifold C . = q θ (•|x) : x ∈ X (for further details, the reader is referred to Subsection 3.6.1). In the next section, we show that this has an important consequence on the set of solutions obtained by minimizing the respective empirical risks while varying λ.

Accuracy-Robustness Trade-offs and Learning in the

Gaussian Model

To illustrate the FIRE loss and the role of the Fisher-Rao distance to encourage robustness, we study the natural-adversarial accuracy trade-off for a simple logistic regression and Gaussian model and compare the performance of the predictor trained on the FIRE loss with those of ACE and TRADES losses, in Equation 3.4 and Equation 3.5, respectively.

Accuracy-robustness trade-offs

Consider a binary example with a simplified logistic regression model. Therefore, in this section we assume that Y = {-1, 1} and the softmax probability

q θ (y|x) = 1 1 + exp(-y θ ⊺ x) , (3.18)
We choose the standard adversary obtained by maximizing the cross-entropy loss, i.e.,

x ′ * = argmax

x ′ :∥x ′ -x∥≤ε -log q θ (y|x ′ ). (3.19)
For simplicity 1 , in this section we assume that the adversary uses the 2-norm (i.e,

∥ • ∥ = ∥ • ∥ 2 ).
In such case, x ′ * can be written as x ′ * = x -ε y θ/∥θ∥ 2 .

We also assume that the classes are equally likely and that the conditional inputs given the class are Gaussian distributions with the particular form x|y ∼ N (y, µ, Σ).

In this case, we can write the natural and adversarial misclassification probabilities CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION as:

P e (θ) . = P( f θ (X) ̸ = Y) = Φ -θ ⊺ µ θ ⊺ Σθ , (3.20) P ′ e (θ) . = P( f θ (X ′ * ) ̸ = Y) = Φ ε∥θ∥ 2 -θ ⊺ µ θ ⊺ Σθ , ( 3
Φ ε λ 1/2 max (Σ) + Φ -1 (P e (θ)) ≤ P ′ e (θ), (3.22) Φ ε λ 1/2 min (Σ) + Φ -1 (P e (θ)) ≥ P ′ e (θ). (3.23)
The proof of this proposition is relegated to Subsection 3.6.4.

Learning

Let us consider the 2-dimensional case, i.e., n = 2. From Equation 3.20 and Equation 3.21, it can be noticed that both P e (θ) and P ′ e (θ) are independent of the 2-norm of θ. Therefore, we can parameterize θ as θ = [cos(α), sin(α)] ⊺ with α ∈ [0, 2π) without any loss of generality. Thus, (1 -P e (θ), 1 -P ′ e (θ)) for all values of α gives a curve that represents all the possible values of the natural and adversarial accuracies for this setting. Figure 3.6a shows this curve for a particular choice 2 of ε, µ, and Σ. As can be observed, the solution which maximizes the natural accuracy gives poor adversarial accuracy and viceversa 3 . The set of Pareto-optimal points (i.e., the set of points for which there is no possible improvement in terms of both natural and adversarial accuracy or, equivalently, the set {max θ β(1 2 In this experiment, we obtained the components in µ by sampling N (0, 1/400). We also defined a matrix A with samples of the same distribution and constructed Σ as Σ = AA ⊺ .

-P e (θ)) + (1 -β)(1 -P ′ e (θ)) : 0 ≤ β ≤ 1})
3 The (normalized) value of θ that maximizes 1 -P e (θ) is θ * nat = Σ -1 µ/∥Σ -1 µ∥ 2 (see, for instance, BISHOP [2006]) and corresponds to α * nat ≈ 1.814, giving 1 -P e (θ) ≈ 0.784 and 1 -P ′ e (θ) ≈ 0.183. The value of θ that maximizes 1 -P ′ e (θ) is α * adv ≈ 2.783, giving 1 -P e (θ) ≈ 0.608 and 1 -P ′ e (θ) ≈ 0.309. This is a metric of particular importance, which combines with equal weights both natural and adversarial accuracies.

In addition, we present the solution of the (local) Empirical Risk Minimization (ERM)4 for the TRADES risk function as defined in Equation 3.5 for different values of λ in Figure 3.6b. As can be seen, the curve obtained in the (1 -P e (θ), 1 -P ′ e (θ)) space covers a large part of the Pareto-optimal points expect for a segment for which the solution does not behave well. Finally, in Figure 3.6c we present the result for the proposed FIRE risk function in Equation 3.7 for different values of λ. In this case, we observed that the curve of solutions in the (1 -P e (θ), 1 -P ′ e (θ)) space covers all the Pareto-optimal points. Moreover, we have observed that, for some λ, the θ which minimizes the FIRE risk matches the θ which achieves the MAA defined in Equation 3.24, while TRADES method fails in achieving this particularly relevant point. It should be added that none of the methods cover exactly the set of Pareto-optimal points, which is expected, since all loss functions can be considered surrogates for the quantity βP e (θ) + (1 -β)P ′ e (θ), where 0 ≤ β ≤ 1. We performed a similar comparison between FRD and KL using standard datasets. The results are reported in Appendix Section A.1 CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION

Experimental Results

In this section, we assess our proposed FIRE loss' effectiveness to improve neural networks' robustness.5

Setup

Datasets: We resort to standard benchmarks. First, we use MNIST [LECUN and collab., 2010], composed of 60,000 black and white images of size 28×28 -50,000 for training, and 10,000 for testing -divided into 10 different classes. Then, we test on CIFAR-10, and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], composed of 60,000 color images of size 32×32×3 -50,000 for training and 10,000 for testing -divided into 10 and 100 classes, respectively. Finally, we also test on CIFAR-10 with additional data thanks to 80 Million Tiny Images [TORRALBA and collab., 2008], an experiment that will be detailed later. equally about natural and adversarial accuracies, we also compute the average sum of the two, i.e., the Average Accuracy ( Avg. Acc.). This is an empirical version of the Maximum Average Accuracy (MAA) defined in Equation 3.24. Finally, we report the runtime of each method as the time required to complete the adversarial training. To provide fair comparisons between runtimes, we run the official code of each method on the same 4 NvidiaV100 GPUs (for further details, see Section 3.5.2).

Experimental results

Kullback-Leibler versus Fisher-Rao regularizer:

To disentangle the influence of different regularizers, we compare the Fisher-Raobased regularizer to the Kullback-Leibler-based regularizer used in TRADES with CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION the exact same model and hyperparameters (as detailed previously) on CIFAR-10, CIFAR-100, and MNIST datasets. We used λ = 6 to train the TRADES method, since it is the value presented in the original paper [ZHANG and collab., 2019]. The results are averaged over 5 tries and summarized in Table 3.1. Those results confirm the superiority of the proposed regularizer. Specifically, the natural and adversarial accuracies increase up to 1% each under AutoAttack, improving the trade-off up to 1%.

Comparison with state-of-the-art

We compare FIRE with the state-of-the-art methods using adversarial training under l ∞ -norm attacks. Due to its effectiveness on adversarial performances, we use the self-adaptive scheme from HUANG and collab.

[2020] along with the FIRE loss to smooth the one-hot labels on CIFAR-10, and report the results under FIRE + Self Adaptive. We do not include the Adversarial Weight Perturbation (AWP) [WU and collab., 2020] method since it leverages two networks to increase the robustness of the model. We trained all methods from the state-of-the art on 5 tries and report the mean the standard variance of the results in and collab. [2020] but, given that their method requires four times FIRE's runtime to complete its training, the gain of our method appears to be more significant. Besides, FIRE outperforms RICE and collab.

[2020] on the more challenging CIFAR-100 in both Avg. Acc., with more significant gain (1.12%) and runtime. Taking all metrics into account, FIRE appears to be the best overall method.

Runtimes: Interestingly, our method exhibits a significant advantage over previous state-of-the-art methods using similar backbones. Our method outruns methods with similar performances by 20% on average. We presume that the difference between the different runtimes comes from the fact that our proposed FIRE loss comprises 5 different operations. In contrast, the KL divergence, proposed in ZHANG and collab.

[2019], is composed of 6 operations.

Ablation studies

Influence of λ: We study the influence of the hyperparameter λ on the performances of the FIRE method.

Proofs of Theorems and Propositions

Review of Fisher-Rao Distance (FRD)

Consider the family of probability distributions over the classes 8 C . = q θ (•|x) : x ∈ X .

Assume that the following regularity conditions hold [ATKINSON and MITCHELL, 1981]:

(i) ∇ x q θ (y|x) exists for all x, y and θ ∈ Θ;

(ii) y∈Y ∇ x q θ (y|x) = 0 for all x and θ ∈ Θ;

(iii) G(x) = E Y∼q θ (•|x) ∇ x log q θ (Y|x)∇ ⊺
x log q θ (Y|x) is positive definite for any x and θ ∈ Θ.

Notice that if (i) holds, (ii) also holds immediately for discrete distributions over finite spaces (assuming that y∈Y and ∇ x are interchangeable operations) as in our case.

When (i)-(iii) are met, the variance of the differential form ∇ ⊺ x log q θ (Y|x)d x can be interpreted as the square of a differential arc length d s 2 in the space C , which yields

d s 2 = 〈d x, d x〉 G(x) = d x ⊺ G(x)d x.
(3.25)

CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION Thus, G, which is the Fisher Information Matrix (FIM), can be adopted as a metric tensor. We now consider a curve γ : [0, 1] → X in the input space connecting two arbitrary points x and x ′ , i.e., such that γ(0) = x and γ(1) = x ′ . Notice that this curve induces the following curve in the space C : q θ (•|γ(t )) for t ∈ [0, 1]. The Fisher-Rao distance between the distributions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ) will be denoted as d R,C (q θ , q ′ θ ) and is formally defined by

d R,C (q θ , q ′ θ ) . = inf γ 1 0 d γ ⊺ (t ) d t G(γ(t )) d γ(t ) d t , (3.26)
where the infimum is taken over all piecewise smooth curves. This means that the FRD is the length of the geodesic between points x and x ′ using the FIM as the metric tensor. In general, the minimization of the functional in Equation 3.26 is a problem that can be solved using the well-known Euler-Lagrange differential equations. Several examples for simple families of distributions can be found in ATKINSON and MITCHELL [1981].

Proof of Theorem 1

To compute the FRD, we first need to compute the FIM of the family C with q θ (y|x)

given in Equation 3.8. A direct calculation gives:

G(x) = e h θ (x) 1 + e h θ (x) 2 ∇ x h θ (x)∇ ⊺ x h θ (x). (3.27)
It is clear from this expression that the FIM of this model is of rank one and therefore singular. This matches the fact that q θ (y|x) has a single degree of freedom given by h θ (x). Therefore, the statistical manifold C has dimension 1.

To proceed, we consider the following model:

q θ (y|u) = 1 1 + e -u y , (3.28)
where we have defined u = h θ (x). This effectively removes the model ambiguities because q(y|u) ̸ = q(y|u ′ ) if and only if u ̸ = u ′ . Note that q θ (y|u), for fixed θ, can be interpreted as a one-dimensional parametric model with parameter u. Its FIM is a scalar that can be readily obtained, yielding:

G(u) = e u (1 + e u ) 2 .
(3.29)

Clearly, the FIM G(u) is non-singular for any u (i.e., G(u) > 0 for any u) as required.

Let D = {q θ (•|u) : u ∈ U }, where U = h θ (X ) and consider two distributions in D:

q θ = q θ (•|u) and q ′ θ = q θ (•|u ′ ).
Then, the FRD can be evaluated directly as follows CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION [ATKINSON and MITCHELL, 1981][Eq. (3.13)]:

d R,D (q θ , q ′ θ ) = u ′ u G 1/2 (v)d v = 2 arctan e u ′ /2 -arctan e u/2 . (3.30)
Therefore, the FRD between the distributions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ) can be directly obtained by substituting u = h θ (x) and u ′ = h θ (x ′ ) in Equation 3.30, yielding the final result in Equation 3.9.

Proof of Theorem 2

As in the binary case developed in Subsection 3.3.3, the FIM of the family C with q θ (y|x) given in Equation 3.14 is singular. To shows this, we first notice that q θ (y|x) can be written as

q θ (y|x) = e g y (x,θ) y ′ ∈Y e g y ′ (x,θ) , (3.31)
where g y (x, θ) = h y (x, θ)h 1 (x, θ). Since g 1 (x, θ) = 0, this shows that C has M -1 degrees of freedom: g 2 (x, θ), . . . , g M (x, θ). A direct calculation of the FIM gives

G(x) = M y=2 q θ (y|x)∇ x g y (x, θ)∇ ⊺ x g y (x, θ) (3.32) - M y,y ′ =2 q θ (y|x)q θ (y ′ |x)∇ x g y (x, θ)∇ x g ⊺ y ′ (x, θ).
Let v ∈ X be an arbitrary vector and define

β y . = ∇ ⊺ x g y (x, θ) v for y = [2 : M]. Notice that G(x)v = M y=2 q θ (y|x)β y ∇ x g y (x, θ) - M y=2 M y ′ =2 q θ (y ′ |x)β y ′ q θ (y|x)∇ x g y (x, θ).
(3.33)

Therefore, the range of G(x) is a subset of the span of the set {∇ x g 2 (x, θ), . . . , ∇ x g M (x, θ)}.

Thus, it follows that rank(G(x)) ≤ M -1, which implies that it is singular.

The singularity issue can be overcome by embedding C into the probability simplex P defined as follows:

P = q : Y → [0, 1] M : y∈Y q(y) = 1 . (3.34)
To proceed, we follow CALIN and UDRI ŞTE [2014][Section 2.8] and consider the CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION following parameterization for any distribution q ∈ P :

q(y|z) = z 2 y 4 , y ∈ {1, . . . , M}. (3.35)
We then consider the following statistical manifold: where {e y } are the canonical basis vectors in R M and I is the identity matrix. From Equation 3.37 we can conclude that the Fisher metric is equal to the Euclidean metric.

D = q(•|z) : ∥z∥ 2 = 4, z y ≥ 0, ∀y ∈ Y . ( 3
Since the parameter vector lies on a sphere, the FRD between the distributions q = q(•|z) and q ′ = q(•|z ′ ) can be written as the radius of the sphere times the angle between the vectors z and z ′ . This leads to d R,D (q, q ′ ) = 2 arccos z ⊺ z ′ 4

= 2 arccos y∈Y q(y|z)q(y|z ′ ) .

(3.38)

Finally, we can compute the FRD for distributions in C using:

d R,C (q θ , q ′ θ ) = 2 arccos y∈Y q θ (y|x)q θ (y|x ′ ) . (3.39) Notice that 0 ≤ d R,C (q θ , q ′ θ ) ≤ π, ∀ x, x ′ ∈ X , being zero if q θ (•|x) = q θ (•|x ′
) and π when [q θ (1|x), . . . , q θ (M|x)] and [q θ (1|x ′ ), . . . , q θ (M|x ′ )] are orthogonal vectors.

Proof the consistency between Theorem 1 and Theorem 2. This consists in

showing the equivalence between the FRD for the binary case, given by Equation 3.9, and the multiclass case, given by Equation 3.15, for M = 2. First, notice that the models Equation 3.8 and Equation 3.14 coincide if we consider the following correspondence: h θ (x) = h 2 (x, θ)h 1 (x, θ), y = 1 ↔ y = -1 and y = 2 ↔ y = 1. Then, using CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION standard trigonometric identities, we rewrite the FRD for the multiclass case:

d R,C (q θ , q ′ θ ) = 2 arccos 1 1 + e h θ (x) 1 1 + e h θ (x ′ ) + 1 1 + e -h θ (x) 1 1 + e -h θ (x ′ )
= 2 arccos cos(arctan(e h θ (x)/2 )) cos(arctan(e h θ (x ′ )/2 ))

+ sin(arctan(e h θ (x)/2 )) sin(arctan(e h θ (x ′ )/2 ))

= 2 arccos cos arctan(e h θ (x)/2 )arctan(e h θ (x ′ )/2 ) = 2 arctan(e h θ (x ′ )/2 )arctan(e h θ (x)/2 ) , (3.40)

where the last step follows by | arctan(α)| ≤ π/2, ∀α ∈ R, so the argument of the cosine function belongs to [-π, π] and arccos(cos(α)) = |α| for |α| ≤ π, which completes the proof.

Proof of Proposition 2

For completeness, we first present the derivation of the misclassification probabilities Equation 3.20 and Equation 3.21:

P e (θ) = P( f θ (X) ̸ = Y) = Φ - θ ⊺ µ θ ⊺ Σθ , (3.41) P ′ e (θ) = P( f θ (X ′ * ) ̸ = Y) = Φ ε∥θ∥ 2 -θ ⊺ µ θ ⊺ Σθ . (3.42)
Notice that Φ, i.e., the cumulative distribution function of the standard normal random variable, is a monotonic increasing function and ε∥θ∥ 2 / θ ⊺ Σθ ≥ 0, so we have P ′ e (θ) ≥ P e (θ), as expected. Furthermore, observe also that Φ is invertible so we can write P ′ e (θ) explicitly as a function of P e (θ):

P ′ e (θ) = Φ ε∥θ∥ 2 θ ⊺ Σθ + Φ -1 (P e ) .
(3.43)

We now proceed to the proof of the proposition. Notice that by the Rayleigh theorem [HORN and JOHNSON, 2013][Theorem 4.2.2], we have that

λ min (Σ)∥θ∥ 2 2 ≤ θ ⊺ Σθ ≤ λ max (Σ)∥θ∥ 2 2 , (3.44)
where λ min (Σ) and λ max (Σ) are the minimum and maximum eigenvalues of Σ, respectively. Therefore, we can bound ε∥θ∥ 2 / θ ⊺ Σθ as follows:

ε λ 1/2 max (Σ) ≤ ε∥θ∥ 2 θ ⊺ Σθ ≤ ε λ 1/2 min (Σ) . ( 3

Summary and Concluding Remarks

We introduced FIRE, a new robustness regularizer-based method on the geodesic distance of softmax probabilities using concepts from information geometry. The main innovation is to employ Fisher-Rao Distance (FRD) to encourage invariant softmax probabilities for both natural and adversarial examples while maintaining high performances on natural samples. Our empirical results showed that FIRE consistently enhances the robustness of neural networks using various architectures, settings, and datasets. Compared to the state-of-the-art methods for adversarial defenses, FIRE increases the Average Accuracy (Avg. Acc.). Besides, it succeeds in doing so with a 20% reduction in terms of the training time.

Interestingly, FRD has rich connections with Hellinger distance, the Kullback-Leibler divergence, and even other standard regularization terms. Moreover, as illustrated via our simple logistic regression and Gaussian model, the optimization based on FIRE is well-behaved and gives all the desired Pareto-optimal points in the natural-adversarial region. This observation contrasts with the results of other state-of-the-art adversarial learning approaches. Further theoretical explanation of this change in behaviour is worth exploring in a future work.

Chapter 3 Conclusion

In this chapter, we addressed the problem of leveraging the knowledge about the output space to increase an image classifier's robustness. We introduced a new robustness regularizer-based method on the Fisher-Rao distance. The Fisher-Rao Distance (FRD) is the geodesic distance of softmax probabilities using concepts from information-geometry that will encourage invariant softmax probabilities for both natural and adversarial examples while maintaining high performances on natural samples. Our empirical results showed that using the Fisher-Rao distance consistently enhances the robustness of neural networks using various architectures, settings, and datasets.

Our framework is powerful and generic enough to be adapted to various applications. In what follows, we rely on the same idea to increase the safety of the cyber-components of Smart Grid systems. We focused on the protection of the state estimator, a component necessary to monitor the grid in real-time that is highly sensitive to attacks. In Chapter 4, we restrict ourselves to the linearized case, while in Chapter 5, we extend it in the non-linear setting.

CHAPTER In this chapter, we still address the first research question dedicated to increasing adversarial robustness. We present our second contribution, which falls the scope of smart grid systems. We follow the same idea as in Chapter 3, and we modify the training of the state estimator to ensure that it remains trustworthy under threats.

This chapter adapts the information-geometric regularizer previously introduced to build a robust state estimator in Smart Grid Systems. Due to their cyber component, such systems are highly vulnerable to threats. It is, therefore, crucial to enhancing their capability to face attacks. Indeed, through their interaction with citizens' lives, Smart Grid systems are critical systems.

In the following, we will present the state estimator problem in the linearized

Abstract

Smart Grids are critical cyber-physical systems where monitoring is crucial, especially the process of state estimation. Since this task strongly depends on the reliability of power grid meters and their communication channels, it is vulnerable to cyber-attacks and, particularly, false data injection attacks (FDIAs), which are modifications on the meter readings that are often hard to detect. In this paper, we propose a method to construct a robust state estimator based on a variational autoencoder trained on the Fisher-Rao distance, which is a measure of dissimilarity between probability distributions. Then, we introduce a novel method to generate FDIAs that exploits knowledge of the state estimator and its learning procedure, for which we show effectiveness. Finally, numerical results

and comparison with state-of-the-art methods confirm that our approach can archive similar estimation errors for clean and noisy (attacked) measurements.

Introduction

Smart Grids are complex systems composed of a physical layer, including generation, transmission, and distribution of electrical power, and a communication layer encompassing sensors and communication systems. Monitoring the network is essential to ensure self-healing, maintenance, interaction with the consumer while providing various target services [ABUR and EXPOSITO, 2004]. Consequently, power grid

CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS

systems surveillance requires strategic meters to measure different quantities, such as bus voltages, active or reactive power injections. These data are then processed at control centers, where general grid supervision is performed through accurate state estimations. This computational aspect of the Smart Grid management makes them vulnerable to unreliable data.

A bad data detector is usually implemented [START_REF] Monticelli | Reliable bad data processing for real-time state estimation[END_REF]] to secure the system. However, it is not difficult to bypass this defense. For instance, attackers can alter the reliability of state estimation by constructing malicious (noisy) samples without being uncovered. The impact of attacks on power systems has been studied in SGOURAS and collab. [2014] and RICE and ALMAJALI [2014]. One potentially dangerous type of cyber-attacks that has gained attention over the last decade is known as False Data Injection Attacks (FDIAs) [LIU and collab., 2011].

FDIAs introduce attacks that are based on the actual power flow of the targeted grid. This knowledge allows the attacker to construct malicious manipulations of a subset of collected measurements across the grid. The main goal of an FDIA is, for instance, to disrupt the expected behavior or operation of the Smart Grids. In 2015, for example, a malicious attack was launched against a Ukrainian power plant. It deprived hundreds of thousands of houses of electricity for several hours [ALDERSON and DI PIETRO, 2016].

FDIAs can either be random, e.g., the goal is only to disrupt the state estimator, or targeted, e.g., the goal is to fool the estimator by inducing specific (target) values to the states. Stealth attacks [DÁN and SANDBERG, 2010;SUN and collab., 2019] are FDIAs designed so that their detection by the control center is made complex or even infeasible. Indeed, the issue of preventing cyber-attacks from disturbing the whole power system has recently been studied. Defense mechanisms are based on detecting FDIAs [ZONOUZ and collab., 2012], or introducing robustness against the loss of meters [ASHOK and collab., 2016]. The method investigated in HU and collab.

[2017] estimates the error vectors to denoise the measurements before performing state estimation. This paper introduces novel methods to generate attacks assuming full knowledge of both the state estimator and its learning procedure, and to defend against those powerful attacks. Our contributions are summarized as follow:

• We present a defensive scheme to prevent attacks from fooling the state estimator.

• We propose a method to create FDIAs using knowledge about the estimator's structure and learning procedure.

• Experimentally, we show that our attacks are highly efficient at disrupting state estimators.

CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS

• Finally, we design a robust estimator for various well-known electrical networks: IEEE 14/57/118-bus system and test it on state-of-the-art attack.

The rest of this article is organized as follows. In Section 4.2, we present an overview of the standard state estimation components, FDIAs, and MILP attack. We derive the defensive and attack schemes in Section 4.3. Experimental results are relegated to Section 4.4 where we prove the efficiency of the considered attack/defense mechanisms.

Background on State Estimation and Attacks

Bayesian approach for power state estimation

In general, the state is related to the measurement vector Y ∈ R m through the following nonlinear equation:

Y = h(X) + Z, (4.1)
where h is the measurement function and Z ∈ R m is additive noise. This is called the AC model in the literature [ABUR and EXPOSITO, 2004;GIANNAKIS and collab., 2013].

Usually, under certain assumptions about the grid and its operating point, the problem can be linearized as follows:

Y = HX + Z, (4.2)
where H ∈ R m×n is the linearized Jacobian measurement matrix. This is called the DC model, and we will adopt it for the remainder of the paper.

State estimators are crucial components for the functioning of Smart Grids systems, but they are known to be sensitive to faulty samples. Bad data detectors are usually implemented to overcome this issue. They are a-posteriori detectors based on the l p -norm (where typically p = 2) of the residual ∥Y -H X∥ p , where X is the estimated state. The sample is detected as faulty when the l p -norm of the residual is greater than a threshold τ. Nevertheless, as we discuss next, bad data detectors are insufficient to overcome the problem of well-designed attacks.

False data injection attack

One of the most critical types of cyber-attacks are FDIAs. In LIU and collab.

Defense Against False Data Injection Attacks

A Variational Autoencoder-based state estimator

We introduce a Variational Autoencoder (VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF] to build a robust state estimator. Our network is expected to solve Equation 4.2 by learning X ∼ q φ (X|Y) (4.4) from the measurements Y in an unsupervised way, where φ are the parameters of the encoder, and Ŷ is built from the estimate X using the physical knowledge of H and of the noise Z ∼ N (0, Σ ZZ ):

Ŷ ∼ p( Ŷ| X) = N ( Ŷ; H X, Σ ZZ ). (4.5)
The conditional pdf of X|Y will be modeled as Gaussian,

q φ (X|Y) = N (X; µ φ (Y), Σ φ (Y)), (4.6)
where the mappings µ φ and Σ φ are learnt through a deep neural network encoder.

To increase the robustness of a state estimator against FDIAs, we need to ensure that the estimator makes similar predictions for clean and attacked samples. To this end, we build on our recent work [PICOT and collab., 2021] using tools from information geometry to introduce a training regularizer based on Fisher-Rao distance (FRD) [ATKINSON and MITCHELL, 1981] from which our robust VAE-based estimator is derived.

A new metric improving robustness against attacks

To obtain an estimator that is robust to corrupted samples, µ φ and Σ φ , the learned autoencoder (see Subsection 4.2.1) has to perform similar on clean and noisy (attacked) measurements. Therefore, the defender must pursue two different goals:

• Correctly reconstruct the measurement vector from the estimated states. The estimator minimizes a reconstruction loss L (Y, H f φ (Y)), where X = f φ (Y) is the estimated state vector and f φ (Y) is the sampling of q φ (•|Y).

• Provide similar results through the mappings µ φ and Σ φ for both clean and corrupted measurement vectors, respectively Y and Y A , through the minimization of the distance between the natural and corrupted distributions.

As we mentioned, one natural distance to consider is the FRD [ATKINSON and MITCHELL, 1981], which captures the distance between pdfs over the underlying For a given measurement vector Y and a sample of attacked measurements Y A , we can compare the pdfs q φ (X|Y) and q φ (X|Y A ).

If we assume that the problem is uni-dimensional and that q φ (X|Y) ∼ N (µ, σ)

and q φ (X|Y A ) ∼ N ( μ, σ), the Fisher-Rao distance can be written as follows [ATKINSON and MITCHELL, 1981]:

d F (q φ (•|Y), q φ (•|Y A )) = 2 log ∥( µ 2 , σ) -( μ 2 , -σ)∥ + ∥( µ 2 , σ i ) -( μ 2 , σ)∥ ∥( µ 2 , σ) -( μ 2 , -σ)∥ -∥( µ 2 , σ i ) -( μ 2 , σ)∥ , (4.7)
where µ and σ are the mean and variance of the clean estimated states, respectively, and μ and σ are the corresponding attacked states ones.

For any dimension d ≥ 2, if we assume that Σ φ is diagonal, the FRD can be written as follows:

d R q φ (•|Y), q φ (•|Y A ) = ∀ i d F (q i φ (•|Y), q i φ (•|Y A ) 2 , (4.8)
where q i φ indicates the i -th component of q φ . The defender's objective then becomes:

φ * = arg min φ E[L (Y, H f φ (Y)] + β E d R (q φ (•|Y), q φ (•|Y A )) , (4.9)
where L (•, •) is the reconstruction loss, and β is a hyperparameter controlling the trade-off between clean and attacked performances. It remains to define a defensive scheme using the above objective to learn a robust state estimator.

In order to train a state estimator as robust as possible, being able to generate the most powerful attacks is extremely important. The best possible method is to give the attacker full knowledge about the underlying training procedure.

Novel attacks exploiting the state-estimator knowledge

Knowledge about the state estimator is required to generate harmful attacks. In this setting, where the attackers have access to the state-estimator function and parameters φ, the attacker wants to maximize the expected error between the clean (noncorrupted) estimated state X and the corrupted estimated state XA , i.e. E[ℓ( X, XA )].

While at the same time, the attacker must guarantee that the bad data detector will not detect the corrupted samples (based on the generated noise), i.e., the attackers should minimize ∥Y A -H XA ∥ p . Hence, the attacker's objective can be written as follows: where f φ (•) is the state estimator and λ is an hyperparameter controlling the trade-off between the power of the attack and the detection of this attack. In order to build such attack, we can use an arbitrary fidelity measure ℓ(•, •), e.g. an l p -norm.

Y * A = arg max Y A ℓ( f φ (Y), f φ (Y A )) -λ∥Y A -H f φ (Y A )∥ p , ( 4
In order to create the attacks that will be the most harmful to a state estimator that defends according to Equation 4.9, the attacker should choose the Fisher-Rao distance between clean and attacked probability distributions as its fidelity measure. • Then, the defender updates its parameters φ by approximately solving Equation 4.9.

General defensive framework

Y A ← Y for i = 1..n do Y A ← Y A + δ sgn(∇ Y A (g (Y, Y A ))), where g (Y, Y A ) = L ( f φ (Y), f φ (Y A )) -λ∥Y A -H f φ (Y A )∥ p
From this methodology, we derive an approximate solution to a min-max problem which yields our robust estimator.

Numerical Results

In this section, we assess our proposed attack method's strength and evaluate the robust estimator's effectiveness on both our proposed attack and the MILP attack defined in KHEZRIMOTLAGH and collab. [2019].

Experimental set-up

Power network: We begin by considering the IEEE 14 bus system. The state vector is composed of the bus angles α. Later we present results on bigger IEEE bus systems (i.e., IEEE 57/118 bus systems), typically used as benchmarks. For each dataset, we define a training set (8000 samples used to train the model), a validation set ( 2000samples used for the hyperparameters choices), and a testing set (1000 samples used for testing the resulting model).

Considered Meters: We use the active power P at each bus as the meters.

Autoencoder model:

• Since the DC model is linear, we employ a linear neural network to build the encoder, composed of a single layer using the measurement and the state size as input and output sizes.

• We use a physical knowledge-based decoder, defined by the Kirchhoff and Ohm laws. • To train the robust VAE model, we need to choose the fidelity measure ℓ(•, •) in Equation 4.9 and the reconstruction loss L (•, •) in Equation 4.10. For the latter, we have to ensure that the chosen network-along with its loss-can predict the states accurately. Hence, we chose L (•, •) as the l 1 -norm between the groundtrue measurement vector and the estimated one since it gives us the best state estimation results. Consequently, we also consider the l 1 -norm for the bad data detector and the attacker. Then, we choose the FRD (see Equation 4.8)

as the fidelity measure ℓ(•, •) to ensure that the attack/defense scheme is as balanced as possible.

Optimization set-up. We train for 200 epochs, using the Adam optimizer with a learning rate of 10 -4 for the first 3/4 of learning and 10 -5 afterward, and a weight decay of 5.10 -4 .

Generation of the attack. We apply the formula in Equation 4.10, with the PGD method described in Algorithm 1. We set the step parameter δ to 0.03 and the number of iterations n to 10. In the final simulation, we consider the MILP attack presented in Subsection 4.2.3 which also has access to the knowledge of the state estimator.

Performance of the proposed attack

We aim at assessing the strength of the attack mechanism. To this end, we train a defenseless state estimator for the IEEE 14 bus system, where the mean angle clean error is equal to 0.0867 degrees. We generate the noisy (attacked) samples using the method presented in Subsection 4. For λ = 100, the number of detected samples is equal to 0. At the same time, the state estimation error caused by the attack based on the state estimator equals 1.08.

Experimentally, it confirms that maximizing the Fisher-Rao distance between the original and the corrupted state attack is effective and can be undetected.

Influence of the number of attacked meters

It is reasonable to assume that the attacker often can only affect a subset among all meters in real-world scenarios. We, therefore, propose investigating the influence of the portion of attacked meters in the measurement vector. Once again, the attackinduced mean angle error and the percentage of detected samples are reported in Figure 4.2. We set λ = 10 to have a powerful attack and highlight the number of meter's influence on the power and the detection of the attack. We select the meters so that only the c most influential meters are kept, i.e., the l 0 of the attack vector is below c, with c ∈ [0, 1].

As expected, the larger the portion of meters the attacker can affect, the more degrees of freedom it has, and thus, the more powerful the attack becomes. Nevertheless, the percentage of detected samples increases as well. Since a detected attack cannot be considered successful, the best trade-off between attack detection and power seems to be around 50% of corrupted meters. In the remaining, we assume that the attacks use λ = 10, and the attacker will affect 50% of the meters for each measurement vector. Now, we focus on the defensive aspect of the state estimator.

Defensive performances

On the 14-bus net

We train a robust state estimator, using the method detailed in Subsection 4.3.4.

On other classical power networks

We choose to follow the classical IEEE bus systems with three different numbers of buses, namely: small (14 buses), medium (57 buses), and large (118 buses). We train a defenseless and a robust estimator for λ = 10 and 50% of corrupted meters. The optimal β value is 0.1 for all cases. We then test both estimators on clean observations, attacked observation using our proposed method, and attacked observation using the MILP attack [KHEZRIMOTLAGH and collab., 2019].

We report the mean absolute state error on clean and attacked observations for both a defenseless and a robust state estimator for all considered power networks in Table 4.1. The clean and attacked errors are pretty similar for all networks and for both considered attacks (our attack and the MILP attack) when a robust estimator is trained, while the amount of natural performances lost by the robust training is between 0 and 0.04 degrees. Therefore, it is possible to construct a robust state estimator that improves robustness against state estimator-aware attacks while not decreasing the performance significantly on clean samples.

Conclusion

In this paper, we introduced a novel method to generate powerful stealth false data injection attacks based on the structure and training procedure of the state estimator that the attacker aims at fooling. We have experimentally proven that this new generation of attacks introduces state estimation error.

We also introduced a method to learn a robust state estimator using a geometric information distance, known as the Fisher-Rao distance, based on a min-max game between the estimator and the attacker. On multiple benchmarks of power networks, we have experimentally proven that it is possible to train a state estimator that improves its robustness against attacks while not decreasing the performance significantly on clean samples.

Finally, it is worth mentioning that our results are based on the DC model assumption. However, we plan to generalize the proposed method to the more general

Chapter 4 Conclusion

In this chapter, we addressed the problem of leveraging the knowledge about the output space to increase a state estimator's robustness. We modified our regularizer-based method on the Fisher-Rao distance to apply it to smart grid systems.

We are among the first to use deep models to train a state estimator, based on Variational AutoEncoders, that estimates the appropriate state even under threat. In this work, we restricted ourselves to the linearized case. However, this hypothesis is not necessarily realistic for every bus system. In the next chapter, we will extend our work to be applicable to non-linear state estimators.

In However, this linear assumption does not always hold.

In the following, we will present the state estimator problem in the nonlinearized case, i.e., the AC model, which is the realistic case since all electrical systems are actually always non-linear. We present our solution to build robust non-linear state estimators. Our extensive numerical experiments prove that our method is able to construct state estimators that are robust to state-of-theart attacks in the case of multiple bus systems.

Abstract

Smart Grids are critical systems where state estimation is critical due to monitoring purposes. Through this computational aspect, Smart Grids are vulnerable to cyber-attacks, and in particular, to False Data Injection Attacks (FDIAs). In this paper, we begin by introducing two novel methods to generate

FDIAs that exploit knowledge of the state estimator: a deterministic process, creating undetectable attack for each observation vector, and a random process, learning a probability distribution depending on the observations. Then, we present a method to construct a robust state estimator based on a variational autoencoder trained using ideas from adversarial deep learning to improve its robustness capabilities. Finally, we test our robust estimator for three different CHAPTER 5. ROBUST STATE ESTIMATION AGAINST ADVERSARIAL NOISE power systems: the IEEE 14/57/118 buses systems. The results show that our approach can provide similar estimation errors for clean and noisy (attacked) measurements, also showing that the estimator is not sensitive to the attacker strategy.

Introduction

Motivation

Smart Grids are cyber-physical electrical systems where monitoring is an essential task. Real-time analysis of the network is made possible through the strategic deployment of meters on the grid. These meters are then processed to assess the state of the grid through accurate state estimation. This computational aspect, while allowing self-healing, maintenance, and interaction with the consumer while providing various services [ABUR and EXPOSITO, 2004], also increases the vulnerability of smart grids to unreliable data and malicious agents.

To prevent unreliable data from disrupting the normal operation of a smart grid, bad data detectors are usually implemented and included in the state estimation process. However, it is well-known that malicious agents can bypass this protection using well-crafted attacks and alter the reliability of state estimation without being uncovered. In these cyber-attacks, known as False Data Injection Attacks (FDIAs) [LIU and collab., 2011], the attacker makes use of the knowledge about the power grid and the monitoring system to craft the attack. FDIAs attempt to disrupt the state estimation process through the manipulation of specific measurements. The attacks can either be random, e.g., the goal is only to disrupt the state estimator, or targeted, e.g., the goal is to induce specific (targeted) values to the states. Stealth attacks [DÁN and SANDBERG, 2010;SUN and collab., 2019] are FDIAs designed so that their detection by the control center is made complex or even infeasible. The problem of defense against FDIAs is of critical importance in practice. For instance, in 2015, a malicious attack was successfully launched against a Ukrainian power plant. This deprived hundreds of thousands of houses of electricity for several hours [ALDERSON and DI PIETRO, 2016].

Related work

FDIAs have been widely studied since their introduction by LIU and collab. [2011].

Power system vulnerability studies have been conducted by developing strategies to efficiently design attack schemes [HUG and GIAMPAPA, 2012;KOSUT and collab., 2010;LIU and collab., 2011;RAHMAN and MOHSENIAN-RAD, 2013;YUAN and collab., 2011].

The vulnerability has been defined as the minimum number of compromised meters

Contributions

In this paper, we propose a novel method to craft robust state estimators based on a game between the estimator and a simulated attacker using a deep learning approach.

The main idea is that if the simulated attacker and the estimator are trained jointly, the estimator will become robust in the process. Our idea is inspired by the field of adversarial machine learning [START_REF] Vorobeychik | Adversarial Machine Learning, Synthesis Lectures on Artificial Intelligence and Machine Learning[END_REF].

Our contributions can be summarized as follows:

• We present a scheme to train a robust state estimator to prevent attacks from disrupting the state estimator based on an adversarial training approach.

• We propose two frameworks to generate attacks: a deterministic framework, where the attack vector is constructed using the gradients of the loss function with respect to the input, and a random gaussian framework, where the mean and the standard variation both depend on the measurements. The latter attack can be seen as a generalization of the gaussian attacks presented in SUN and collab. [2019].

• We compare the deterministic and random attacks on the IEEE 14 bus system and find that random attacks seem better suited to train a robust state estimator.

• Finally, we test our defense scheme against several attacks on the IEEE 14/57/118 bus systems and compare its performance with a state-of-the-art method. We find that our approach yields a more robust estimator.

The rest of the paper is organized as follows. In Section 5.2, we present a review of the state estimation problem, followed by an overview of false data injection attacks.

In particular, we review a state-of-the-art defense mechanism and an attack method to later use as our benchmark. In Section 5.3, we present our defensive framework along with the proposed attacks. Finally, in Section 5.4, we experimentally show the effectiveness of our robust state estimator.

Background on State Estimation and Attacks

Bayesian approach for power state estimation

Let x = |x|e i arg x ∈ C n be the state vector (latent variables) of the power grid, assumed to be random with a prior probability density function (pdf) p(x), and y = |y|e i arg y ∈ C m the measurement vector. In general, these are related through the following nonlinear equation:

y = h(x) + z, (5.1)
where h is the measurement function (dependent on the grid topology, line impedances, etc.) and z ∈ C m is additive noise. This is called the AC model in the literature [ABUR and EXPOSITO, 2004;GIANNAKIS and collab., 2013].

We define a (parametric) state estimator as a function f φ : C m → C n such that x = f φ (y). The parameters φ can be obtained by minimizing a risk function (see Section 5.3).

State estimators are crucial components for the functioning of smart grid systems, but they are known to be sensitive to faulty samples. Bad data detectors are usually implemented to overcome this issue. They are a-posteriori detectors based on the l p -norm (where typically p = 2) of the residual vector r = yh(x).

(5.2)

Concretely, the sample is detected as faulty when the ∥r∥ p > τ, where τ is a threshold chosen appropriately to control the trade-off between missed and false detections.

Nevertheless, bad data detectors are insufficient to overcome the problem of welldesigned attacks. In particular, a theoretical bound on the maximal number of meters that one can attack in order to remain undetected is derived in JIN and collab. [2018].

False Data Injection Attacks

In order to craft an FDIA under the AC model (Equation 5.1), the attacker can solve the following optimization problem [JIN and collab., 2018]:

max x a ,a g (x a ) s.t. h(x a ) + z = y + a = y a (5.3) ∥a∥ 0 ≤ c,
where x a is the corrupted state, a is the attack vector, y a = y + a is the attacked observation, and g (•) is the objective of the attacker. The condition on the l 0 -norm of the attacked vector allows the attacker to bypass the bad data detector by attacking only a subset of sensors. Since the minimization problem in Equation 5.3 is widely non-convex, different relaxations are proposed in JIN and collab. [2018] to solve Equation 5.3.

Notice that the attacker objective can be different based on its specific goal, for example:

• Target state attack, where g (x a ) = ∥x ax target ∥ 2 2 , which will put the corrupted state to the targeted value x target .

• Voltage collapse attack, where g (x a ) = ∥x a ∥ 2 2 , which will lead the estimator to believe that the voltage is low.

• State deviation attack, where g (x a ) = -∥x a -x∥ 2 2 , which will force the corrupted state to be different from the original predicted one.

The optimization problem in Equation 5.3 requires to solve precisely the state estimation problem for each attack vector a in order to obtain x a . If we assume that the attacker knows the structure of the state estimator f φ , it is reasonable and convenient to replace x a with f φ (y + a). This leads to the following problem for the attack design:

max a g ( f φ (y + a)) s.t. ∥a∥ 0 ≤ c, (5.4)
We focus on this formulation for the remaining of the paper.

A variety of attacks has recently been proposed to attack AC State Estimator.

While some try to attack without any knowledge about the grid to attack, we focus on attacks with full knowledge about it. This can be considered as a worst-case scenario, which is a reasonable criterion to study the vulnerability of critical systems such as smart grids.

JIN and collab.

[2017] presented a semi-definite relaxation to create a convex attack optimization to create powerful and sparse attacks. The sparsity constraint is relaxed as an l 1 -norm constraint on the additive noise. While their method has been presented for targeted attacks, it can be easily extended to any of the three attacker's objectives presented in Subsection 5.2.2.

While crafting attacks that disrupt the state estimator's functioning is relatively easy, building efficient defenses against said attacks is a more challenging task.

Review of the LASSO robust state estimators

There exists in the literature a few works that focus on crafting robust state estimators [JIN and collab., 2019;ZHANG and collab., 2017]. We decided to focus on the LASSO solution proposed in JIN and collab. [2019]. First, they introduce a new basis for the state estimation problem, where the AC problem is, in fact, linear. On this new basis, the state is no longer directly optimized. Instead, the new state is a combination of the original one. In order to obtain the robust state estimator, two steps are needed.

The first one is the optimization of a lasso regression, where the l 2 reconstruction error is regularized by the l 1 norm of the noise. The second step consists of the retrieval of the original state from the new basis one.

Proposed Robust State Estimator

A Variational AutoEncoder based state estimator

We propose to use a Variational AutoEncoder (VAE) [START_REF] Kingma | Auto-encoding variational bayes[END_REF] to build a robust state estimator. Our network is expected to provide a state estimator by learning a conditional distribution q φ (x|y) such that x = f φ (y) ∼ q φ (x|y) from the measurements y in an unsupervised way. Here, φ are the parameters of the encoder.

Notice that we can then reconstruct the estimate observation ŷ from the estimate x using the physical knowledge of the grid (to obtain h) and of the noise distribution z (more specifically, its covariance matrix Σ z ). The equivalent minimization problem is: (5.5) where L(φ) is the risk of the state estimator. To obtain an estimator that is robust to corrupted samples, we need to ensure similar performance for clean and attacked measurements. Therefore, the risk should be defined based on two goals:

φ * = arg min φ L(φ),
1. Correctly estimate the states from the observations y.

2. Provide similar results for both clean and corrupted measurement vectors, respectively y and y a .

To ensure both goals are met, we propose the following risk: (5.6) where minimizing L clean (φ) will ensure that the clean states are rightfully estimated, while minimizing L attacked (φ) will ensure clean and attacked mapping are similar. β is a fixed hyperparameter controlling the trade-off between the goals.

L(φ) = L clean (φ) + βL attacked (φ),
To rightfully estimate the clean states, we consider:

L clean (φ) = E[L (y, h(x)],
(5.7)

where L (•, •) is the reconstruction loss. In general, this optimization problem is highly non-convex and has an infinite of solutions. To force the state estimator to find the solution we expect, we impose the condition that the amplitude of the predicted state x has to be within 10% of 1 p.u. The new risk therefore becomes:

L clean (φ) = E L (y, h(x)) + c • ||1 -|x| ||

Attack mechanisms using full knowledge about the state estimator

The attacker's goal is to disrupt the regular operation of the state estimator. We assume that its goal is to maximize the expected error between the original predicted state values x and the state estimation of the attacked observation xa , corresponding to the 3rd case of the objectives in Subsection 5.2.2.

In the setting where full access to the state estimator to attack is available, the attacker wants to maximize, for each observation y, the error between the clean (noncorrupted) estimated state x and the corrupted estimated state xa , i.e. ℓ(x, xa ; φ), where ℓ(•, •; •) is an error loss. At the same time, the attacker must guarantee that the generated sample will remain undetected by the bad data detector, i.e., the attacker should minimize ∥y ah(x a )∥ p . The attacker's objective can therefore be written as: (5.13) where λ is a hyperparameter controlling the trade-off between the power of the attack and its capability to remain undetected.

l (y, y a ; φ) = ℓ(x, xa ; φ) -λ∥y a -h(x a )∥ p ,
The attacker's objective can be optimized in two ways, either in a deterministic way or following a random process. In the deterministic case, the attacker maximizes its objective for each observation, i.e., for a given observation y, y * a = arg max y a l (y, y a ; φ).

(5.14)

The attacker can also generate random attacks following: It should be mentioned that this attacker's objective is similar to the one in SUN and collab.

[2019] if we set ℓ to be the mutual information between the attacked observation and the state.

In the following, we will present two algorithms to generate deterministic and random attacks according to the optimization problems in Equation 5.14 and Equation 5.15, respectively.

Deterministic attack mechanism

Since the problem in Equation 5.14 is extremely non-convex and hence widely intractable, we consider a simple approach to approximate its solution. Concretely, we consider a Projected Gradient Descent (PGD) method to generate the attacked measurements. This method was introduced in the field of adversarial machine learning to generate adversarial examples for computer vision problems [MADRY and collab., 2018]. It is an iterative method based on the sign of the gradient of the objective to maximize. The procedure is explained in detail in Algorithm 2. Notice that each attack is independent of the others and the computation of the attack vector a must be done from scratch for each measurement vector y.

Algorithm 3 Random attack generation (noisy channel optimization)

INPUT: Define the learning rate α c and the number of steps n c INPUT: input data {y 1 , .., y m } Initialize the parameters θ, and fix φ n = 0 while n < n c do Draw a minibatch {y π 1 , .., y π k } for all y ∈ {y π 1 , .., y π k } do Compute µ θ (y), Σ θ (y). Sample y a from N (y; µ θ (y), Σ θ (y)). end for Compute L(θ) = E l (y, y a ; φ) using the minibatch samples and the corresponding attacks.

θ ← θ + α c ∇ θ (L(θ)) n ← n + 1 end while OUTPUT: θ

Random attack mechanism

Another possible way to generate harmful attacks is to create them thanks to a parametric model, called a noisy channel, that will learn the perturbation distribution. If we consider a gaussian noisy channel, we have that y a ∼ q θ (y a |y) = N (y; µ θ (y), Σ θ (y)).

The problem in Equation 5.15 then becomes:

θ * = arg max θ:y a ∼q θ (•|y)
E l (y, y a ; φ) .

(5.16)

The optimization process is explained in Algorithm 3.

The idea behind this method is that we can model the noisy conditional distribution p y a |y as the product of independent normal distributions, where the mean and variance of each component of y a depend on the observation y. This can be seen as the generalization of gaussian attacks presented in SUN and collab. [2019] to conditional gaussian attacks. It should be noted that conditional gaussian attacks are much more powerful than gaussian attacks, thus allowing to create more harmful attacks. On the other hand, training and using a parametric model as the noisy channel will allow the attacker to use and share knowledge among all observation samples. We believe sharing this knowledge will enable the attacker to create more potent and less detectable corrupted samples.

Training of robust state estimator

Given that the parameters φ of a state estimator evolve during training, it is possible to attack the network at each training step. Therefore, to ensure the state estimator is robust, it is convenient to generate attacked samples at each training step and use them to optimize the loss.

Two different steps are needed to train a robust state estimator:

• First, corrupted samples are generated using either the deterministic attack method proposed in Algorithm 2, or the random attack method described in Algorithm 3.

• Then, the defender updates its parameters φ by approximately solving Equation 5.6.

It should be noted that this process can be repeated until convergence. Using this methodology, we derive an approximate solution to problem in Equation 5.5 leading to our robust estimator. The robust state estimator training is described in Algorithm 4. • Random attack: We use a 5-dense-layer model, with input and output sizes for each layer equal to the number of meters. The chosen optimizer is a Stochastic Gradient Descent optimizer with a learning rate of 10 -4 and a weight decay of 5.10 -4 . We perform 1 step of the channel before doing one step of the state estimator during the training phase and reinitialize it at the end of each epoch.

We train another channel from scratch for one epoch and discard the detected samples for testing.

Performance of attacks

First, we want to assess the strength of the different attack mechanisms. We attack a defenseless state estimator, trained using the set-up presented in Subsection 5.4.1, using both attacks and plot the angle and voltage attack-induced errors along with the percentage of detected samples.

1) Deterministic attack using PGD method

The results for the deterministic attack generation method are presented in Figure 5.1a. For small values of λ, the hyperparameter that controls the trade-off between effectiveness and detection of the attack, the attacks are widely detected, which is uninteresting. For big values of λ, the attacks are undetected but significantly less powerful. A good trade-off between the two phenomena seems to be for λ = 10 3 since the attack creates a 7 degree and 0.8 P.U. error, while only 45% of the attacks are detected.

2) Random attack using a noisy channel

The results for the random attack generation method are presented in Figure 5.1b.

The same behavior as for the deterministic attack is visible in the results. A good trade-off seems to be for λ = 10, for which the attacks create a 4-degree and 0.4-P.U.

error while 47% of the attacks are detected.

CHAPTER 5. ROBUST STATE ESTIMATION AGAINST ADVERSARIAL NOISE particular case, the optimal value of β is equal to 0.03, for which the mean errors are similar for natural and corrupted samples, and their values are 0.27 degrees and 0.015 P.U. on clean samples and 0.016 P.U. on attacked ones, respectively.

2) Random attack using a noisy channel: We perform the same simulations as before, now considering the corrupted samples from the noisy channel's training.

The natural and corrupted angle and voltage errors are presented in Figure 5.3b.

The same phenomena as the ones described before are happening in this case.

The optimal β value here is also equal to 0.03, which is not surprising since the same defender's objective will be the same no matter how the corrupted samples are generated. When training a defender based on the random attack, we can find a robust VAE for which the mean errors are similar for natural and corrupted samples, and their values are 0.26 degrees and 0.014 P.U. on clean samples, and 0.016 P.U. on attacked ones, respectively.

From all these simulations, we can conclude that using either method to generate attacks, it is possible to create robust state estimators that will have similar state predictions for both natural and attacked samples.

Comparison between the two attack schemes

Finally, to finish comparing the two attack mechanisms, we decided to attack the robust estimator trained with a method using the other. When we attack the robust estimator trained on the PGD method, the robust performances increases to 0.39 degrees as the mean angle error and 0.032 as the mean voltage error. For the other way around, i.e., testing PGD on the estimator trained with the noisy channel, the results are 0.17 degrees and 0.035 P.U., respectively. The two defenses are quite similar.

However, training a robust classifier on the noisy channel attack takes approximately three times less time than training a robust classifier on the PGD method. This phenomenon is because it is possible to train the random attacker once per estimator step while we have to complete 10 steps of PGD for every estimator step. In order to increase the power of the deterministic attack, we would have to increase the number of PGD steps, which will be even more computationally costly. Therefore, we choose the noisy channel attack as the best attack since, while having equivalent performances, it requires a lot less training time.

Comparison between state estimation methods

We now test our method on widely known and used power networks: the IEEE 14/57/118 bus systems. We trained a defenseless, a robust, and a LASSO state estimator following the method presented in Subsection 5.2.3 for each of the datasets and reported the results for natural, random attacked and state-of-the-art (SOTA)

Chapter 5 Conclusion

In this chapter, we addressed the problem of leveraging the knowledge about the output space to increase a state estimator's robustness in the nonlinearized case. We presented two procedures to create powerful stealth attack, one deterministic and one random. We compared our two attack frameworks and showed that they were indeed powerful. We used this attacks to craft robust state estimators and experimentally showed that they were efficient against attacks.

Part I Conclusion

Part I focused on the first research question: how can we leverage the information about the internal structure of DNN's output space to improve its robustness? This research question has been addressed for two different applications:

1. For image data. We tackled the problem of crafting deep classifiers that are robust to adversarial attackers. We decided to use the Fisher-Rao distance, an information-geometric distance that measures the geodesic between two probability distributions on a statistical manifold. We use this distance as a regularizer to force a given deep classifier to predict natural and adversarial inputs in similar ways. Using the Fisher-Rao distance as a regularizer builds stronger classifiers than using previously introduced regularizers.

2. For state estimation in Smart Grid systems. In the critical case of smart grids, where protecting against attacks is crucial to ensure the grid's reliability, we proposed a new framework to build a robust state estimator. We used a similar framework than in Chapter 3 to show that it was possible to construct robust estimators, both under linear and non-linear assumptions. Our contribution is among the first to make use of a deep neural network (i.e., a variational autoencoder in our case)

to build defenses against stealth attacks.

Part I was dedicated to increasing our trust in the deep models' decision process by increasing their robustness against attacks.

As previously said, an alternative way to increase our trust in the models is to rely on detecting methods. The detector is an additional module that can be plugged on top of the neural network to protect against malicious agents.

Given an input, the detector chooses to reject it based on chosen characteristics. In Part II, we describe our contributions to the field of adversarial detection applied to image data.

Part II Abstract

This part is dedicated to our proposed answers to the second question asked in Chapter 1: How can we craft an efficient and effective detection method based on simple tools ? This part is split into two chapters consisting of two different contributions.

• In Chapter 6, we present our first solution to craft a detection method.

We introduce the data-depths, statistical tools usually used in anomaly detection in the framework of supervised detection of adversarial examples. Data-depths provide a center-outward ordering of points w.r.t.

a reference distribution. We specifically use the halfspace-mass depth and design a detection method to leverage the relevant class-wise information present in the different layers of a given model. We also make use of the available knowledge about the possible threats to craft an efficient method to discard attacked inputs. Experimentally, we validate our method in different scenarios: attack-aware, blind-to-attack, and against adaptive attacks, and show the superiority of our method compared to other state-of-the-art detectors.

• In Chapter 7, we present our second and last detection method. In the unsupervised setting, i.e., without any knowledge about the possible threats, we design a detection method that leverages both the internal structure of the Vision Transformers and the available information provided by the use of a specific data-depth: the Integrated Rank-Weighted (IRW) depth. After explaining our different choices, we experimentally prove that our method outperforms state-of-the-art methods and that our method makes the adaptive attacker's job more complicated. We here present our first contribution concerning the second research question. We made use of the data-depths to detect adversarial examples in a supervised setting. We propose, in the following, a detection method that leverages the particular class-wise information available at different layer of the network , and the halfspace-mass depth to efficiently detect adversarial examples trying to fool an underlying classifier. We experimentally show that the attacks have different class-wise behavior. We compare our method with state-of-the-art detection methods, and experimentally prove its efficiency.

Finally, we show that it is more complicated for an adaptive attacker to fool our detection method than others. perturbations to deceive a trained model. This phenomenon may lead to disastrous consequences in sensitive applications such as autonomous driving, aviation safety management, or health monitoring systems [GEIFMAN and EL-YANIV, 2019;GEIFMAN and collab., 2019;GUO and collab., 2017;[START_REF] Meinke | Neural networks that provably know when they don't know[END_REF].

Over time, a vast literature has been produced on defense methods against adversarial examples [ALDAHDOOH and collab., 2021b;ATHALYE and collab., 2018b;CROCE and HEIN, 2020;ZHENG and collab., 2019]. On the one hand, techniques to train models with improved robustness to upcoming attacks have been proposed

in MADRY and collab. [2018]; ZHENG and collab. and collab., 2019], auxiliary model [ALDAHDOOH and collab., 2021a;SOTGIU and collab., 2020;ZHENG and HONG, 2018]). Detection methods of adversarial examples differ as well according to whether the underlying classifier is assumed to be pretrained or not: when further training of the classifier is allowed, the methods present a novel training procedure (e.g., with reverse cross-entropy [PANG and collab., 2018];

with the rejection option [ALDAHDOOH and collab., 2021a;SOTGIU and collab., 2020]) and a thresholding test strategy. Finally, the learning task of the underlying network also impacts the adversarial examples detection methods (e.g., detection of adversarial examples for human recognition tasks [TAO and collab., 2018]).

Adversarial detection can be related to the anomaly detection problem. Indeed, anomaly detection aims to identify abnormal observations without previously knowing them, possibly including adversarial attacks. A plethora of techniques has been designed to address this problem ranging from machine learning algorithms such as

Isolation Forest [LIU and collab., 2008;STAERMAN and collab., 2019], Local Outlier

Factor [BREUNIG and collab., 2000] or One-Class SVM [SCHÖLKOPF and collab., 2001] to statistical tools such as kernel density estimation [FEINMAN and collab., 2017] or Data Depth [ZUO and SERFLING, 2000] (see CHANDOLA and collab. [2009] for an extensive review of anomaly detection methods). In particular, data depth stands out as a natural candidate to detect anomalies [CHEN and collab., 2009].

The idea of statistical depth has grown in popularity in multivariate data analysis since its introduction by John Tukey [TUKEY, 1975]. For a distribution on R d with d > 1, by transporting the natural order on the real line to R d , a depth function provides a center-outward ordering of points w.r.t. the distribution. The higher the point depth score, the deeper the point is in the distribution. In addition to anomaly detection [CHEN and collab., 2009;SERFLING, 2006;STAERMAN andcollab., 2020, 2021b], the notion of depth has been used to extend the notions of (signed) rank or order statistics to multivariate data, which find numerous applications in statistics and machine learning (e.g. robust inference [CUEVAS and collab., 2007], classification [LANGE and collab., 2014], hypothesis testing [START_REF] Oja | Descriptive statistics for multivariate distributions[END_REF], clustering [START_REF] Jörnsten | Clustering and classification based on the l1 data depth[END_REF]STAERMAN and collab., 2021a]. To the best of our knowledge, it has not been investigated yet through the lens of adversarial attack detection. This paper aims to leverage this overlooked notion to build an adversarial attack detector.

Contributions.

Our contribution is threefold:

1. We propose applying the halfspace-mass depth notion in the context of the adversarial detection problem. To the best of our knowledge, we are the first to both explore and successfully apply data depth for adversarial detection. 3. We extensively evaluate HAMPER's performance across popular attack strategies and computer vision benchmark datasets (e.g., SVHN, CIFAR10, and CIFAR100).

As shown by Figure 6.1, HAMPER largely outperforms SOTA detection methods and consistently detects attacks that SOTA approaches fail to identify.

The paper is organized as follows. First, in Section 6.2, we describe the adversarial detection problem and provide a detailed overview of the SOTA supervised detection methods and the attack mechanisms considered throughout the paper. In Section 6.3, after recalling the concept of data depth by focusing on the halfspace-mass depth, we introduce HAMPER, our proposed supervised detector method based on the HM depth. In Section 6.4, we provide insights on the underlying classifier's behavior under threats. In Section 6.5, we extensively evaluate HAMPER through numerical experiments on benchmarks on visual datasets and compare it to SOTA methods.

Finally, concluding remarks are gathered in Section 6.6.

Background

After defining the problem formulation, we present the SOTA detection methods and the attack mechanisms that we will consider throughout this paper.

Problem formulation

Let (X, Y) be a random tuple of variables valued in X × Y with unknown data distribution p XY ; X ⊂ R d represents the feature space and Y = {1, . . . , C} represents the labels attached to elements in X , where C ∈ N, C ≥ 2. The training dataset

D = {(x i , y i )} n i =1 is defined as n ≥ 1 independent identically distributed (i.i.d.
) realizations of p XY . Subsets of the feature space associated with a label c ∈ Y are denoted by S c = {x i ∈ S :

y i = c} with S = {x i } n i =1 . Given a parametric model with L ≥ 1 layers; let f ℓ θ : X → R d ℓ with ℓ ∈ {1, .
. . , L}, denotes the output of the ℓ-th layer of the deep neural network (DNN) parametrized by θ ∈ Θ where the dimension of the latent space induced by the ℓ-th layer is d ℓ . The class prediction is obtained from the L-th layer softmax output as follows:

f L θ (x; D) ≜ arg max c∈Y q θ (c|x) with q θ (•|x) = softmax( f L-1 θ (x; D)).
The adversarial problem. Given x ∈ X and p ≥ 1, the adversarial generation problem ATTACK DETECTION can be defined as producing x ′ such as [SZEGEDY and collab., 2014]:

x ′ = arg min x ′ ∈R d : ∥x ′ -x∥ p <ε ∥x ′ -x∥ s.t. f L θ (x ′ ; D) ̸ = y, (6.1)
where y is the true label associated to the sample x, and ∥•∥ p is the p-norm operator.

Since this problem is computationally infeasible in general, it is commonly relaxed as follows:

x ′ = arg max

x ′ ∈R d : ∥x ′ -x∥ p <ε L (x, x ′ ; θ), (6.2)
where L (x, x ′ ; θ) is the objective of the attacker, representing a surrogate of the constraint to fool the classifier, i.e., f L θ (x ′ ; D) ̸ = y. The variety of attacks differs with the choice of the norm (e.g., p = 1, 2, ∞) and the value of ε.

Crafting a detector. Given a new observation x ∈ R d , detecting adversarial attacks boils down to build a binary rule g : R d → {0, 1}. Namely: a new observation x ∈ R d is considered as 'normal' (or 'natural', 'clean'), i.e. generated by p XY , when g (x) = a with a ∈ {0, 1}, and x is considered as an adversarial example when g (x) = 1a. For a given scoring function s : R d → R, and a threshold γ ∈ R, we have

g (x) = I{s(x) > γ} =    1 if s(x) > γ, 0 if s(x) ≤ γ. (6.3)

Supervised detection methods

Supervised detection methods, when the defender has access to the future threats that it is going to face, can be separated into two main groups: attack-aware and blind-to-attack methods.

Attack-aware methods. In the attack-aware setting, the methods are going to face a single threat, and they have full knowledge about them. Therefore, it is possible to train one detector per attack. In this setting fall two detection methods: KD-BU [FEINMAN and collab., 2017] and LID [MA and collab., 2018]. KD-BU is based on the intuition that the adversarial examples lie off the data manifold. To train the detector, a kernel density estimation in the feature space of the last hidden layer is performed, followed by an estimation of the bayesian uncertainty of the input sample. LID extracts the local intrinsic dimensionality features for natural and attacked samples for each layer of the classifier and trains a detector on them.

Blind-to-Attack setting. In the blind-to-attack setting, the defender has knowledge about the fact that it is going to be attacked, but do not know exactly how. In that case, a single detector is trained, deployed and tested against all possible threats.

NSS [KHERCHOUCHE and collab., 2020] falls into that category. It is based on the ATTACK DETECTION extraction of the natural scene statistics from the clean and adversarial samples from different threats models, later used to train a detector to discriminate between natural inputs and adversarial examples. Natural scene statistics are regular statistical properties that are altered by adversarial perturbations.

A brief review of attack mechanisms

Multiple methods to generate adversarial examples have been developed in recent years. The attack mechanisms can be divided into two main categories: whitebox, where the attacker has complete knowledge about the targeted classifier, and blackbox, where the attacker does not know about the targeted classifier.

Whitebox attacks. The simplest one is Fast Gradient Sign Method (FGSM), introduced by Goodfellow et al. [GOODFELLOW and collab., 2015]. It consists in modifying the examples in the direction of the gradient of a specific objective, w.r.t. the input on the targeted classifier. Two iterative versions of FGSM have been proposed: Basic Iterative Method (BIM; KURAKIN and collab., 2018) and Projected Gradient Descent (PGD; MADRY and collab., 2018). The main difference is that BIM initializes the adversarial example to the natural sample while PGD initializes it to the natural example plus random noise. Although PGD was initially created under an L ∞ constraint, it is possible to extend the method to any L p -norm constraint. Later, Moosavi-Dezfooli et al. [MOOSAVI-DEZFOOLI and collab., 2016] introduced DeepFool (DF), an iterative method based, at each step, on a local linearization of the model, resulting in a simplified problem. Finally, Carlini-Wagner [CARLINI and WAGNER, 2017] presents the CW method to find the smallest noise solving the original adversarial problem.

They proposed a new relaxed version of the adversarial problem that optimizes an attack objective, chosen according to a specific task.

Blackbox attacks.

Without any knowledge about the targeted classifier or its gradients, blackbox attacks are expected to rely on different mechanisms. Square Attack (SA; ANDRIUSHCHENKO and collab., 2020) employs a random search for perturbations that maximize a given objective, Spatial Transformation Attack (STA; ENGSTROM and collab., 2019) applies small translations and rotations to the original image while Hop Skip Jump (HOP; CHEN and collab., 2020) estimates the gradient-based direction to perturb through a query on the targeted classifier.

Adaptive attacks. There exists a third type of attacks called Adaptive Attacks [ATHA-LYE and collab., 2018a;CARLINI and WAGNER, 2017;TRAMER and collab., 2020;YAO and collab., 2021]. Adaptive attacks have full knowledge about not only the underlying classifier to attack, but also the defense mechanisms one may have deployed. ATTACK DETECTION To build efficient adaptive attacks, it is therefore crucial to understand the mechanisms involved into the defense, and finding ways to bypass them. For examples, the Backward-Pass Differentiable Attack (BPDA; ATHALYE and collab., 2018a) has been developed to overcome the non-differentiability of the defense mechanisms by finding a suitable surrogate to the non-differential parts of them.

A Depth-Based Detector

After presenting the data depth in Subsection 6.3.1, with an emphasis on the halfspacemass depth, we introduce our depth-based detector in Subsection 6.3.2.

Background on data-depth

A data depth function D(•, P) : R d → [0, 1] measures the centrality of any element in x ∈ R d w.r.t. a probability distribution P (respectively, a data set). It provides a centeroutward ordering of points in the support of P and can be straightforwardly used to extend the notions of rank or order statistics to multivariate data. The higher D(x, P), the deeper x ∈ R d is in P. The earliest proposal is the halfspace depth introduced by John Tukey in 1975 [TUKEY, 1975]. This depth is very popular due to its appealing properties and ease of interpretation. Assume that P is defined on an arbitrary subset K ⊂ R d and denote by P(H) ≜ P(H ∩ K ) the probability mass of the closed halfspace H. The halfspace depth of a point x ∈ R d with respect to a probability distribution P on R d is defined as the smallest probability mass that can be contained in every closed halfspaces containing x: D H (x, P) = inf H∈H (x) P (H) , (6.4) where H (x) is the set of all closed halfspaces containing x.

However, the halfspace depth suffers from three critical issues: (i ) finding the direction achieving the minimum to assign it a score induces a significant sensitivity to noisy directions, (i i ) assigning the zero score to each new data point located on the outside of the convex hull of the support of P makes the score of these points indistinguishable, and (i i i ) as the dimension of data increases, an increasing percentage of points will appear at the edge of the convex hull covering the data set leading to have low scores to every points.

To remedy those drawbacks, alternative depth functions have been independently

introduced in CHEN and collab. [2015] andRAMSAY andcollab. [2019]. In this regard, the extension of Tukey's halfspace depth, recently introduced and referred to as the halfspace-mass (HM) depth [CHEN and collab., 2015] (see also RAMSAY andcollab., 2019 andSTAERMAN andcollab., 2021b), offers many advantages. ATTACK DETECTION Authors proposed to replace the infimum by an expectation over all possible closed halfspaces containing x, following in the footsteps of [START_REF] Cuevas | On depth measures and dual statistics. a methodology for dealing with general data[END_REF]. More precisely, given a random variable X following a distribution P and a probability measure Q on H (x), it is defined as follows:

D HM (x, P) = E H∼Q [P(H)] . (6.5) In addition to basic properties a depth function should satisfy, the halfspace-mass depth possesses robustness properties: it has a unique (depth-induced) median with an optimal breakdown point equal to 0.5 [CHEN and collab., 2015] which means that the halfspace-mass depth provides a stable ordering of the 'normal' data even when polluted data belong to the training set. In addition, it has been successfully applied to anomaly detection in CHEN and collab. [2015] making it a natural choice to adversarial attack detection. When a training set {x i , . . . , x n } is given, Equation 6.5 boils down to:

D HM (x, P n ) = E Q 1 n n i =1 I{x i ∈ H} , (6.6)
where P n is the empirical measure defined by 1 n n i =1 δ x i . The expectation can be conveniently approximated with a Monte-Carlo scheme in contrast to several depth functions that are defined as solutions of optimization problems, possibly unfeasible in high dimension. The aim is then to approximate Equation 6.6 by drawing a finite number of closed halfspaces containing x (see Section B.1 for the approximation algorithm for training and testing).

Our Depth-Based Detector

The methodology we propose here is based on the halfspace-mass depth that exhibits attractive mathematical and computational properties, as described in the previous section.

Our depth-based detector, HAMPER, relies on the information available in a subset Λ of DNNs' layers, i.e., the mapped data z ℓ,i = f ℓ θ (x i ; D), ℓ ∈ Λ ⊂ {1, . . . , L -1}. We denote by S ℓ = z ℓ,i [2015], we use an efficient training/testing procedure in order to compute D HM (summarized in Algorithm 5 and Algorithm 6 in Section B.1). These algorithms are repeated for each class c and each considered layer ℓ leading to |Λ| × C scoring functions. Three parameters with low sensitivity are involved: K which is the number of sampled halfspaces in order to approximate the expectation of Equation 6.5; the size n s of the sub-sample drawn at each projection step; and the λ hyperparameter which controls the extent of the choice of the hyperplane. In this paper, we follow the advice given in CHEN and collab. [2015] by choosing the following parameters K = 10000, n s = 32 and λ = 0.5 offering a good compromise between performance and computational efficiency.

Aggregation procedure. Following the supervised setting scenario, as in FEINMAN and collab. [2017]; KHERCHOUCHE and collab. [2020] or MA and collab. [2018], the score is obtained through an aggregation which is performed between halfspacemass scores using a small validation dataset composed of 'normal' and 'adversarial' samples. Our scoring function is then formally defined as: Referring to the problem formulation notations (see Subsection 6.2.1), given a ATTACK DETECTION threshold γ, and supposing a = 1, the detector is provided by the Equation 6.3. The overview of HAMPER can be summarized in Algorithm 7 in Section B.1.

s(x) = ℓ∈Λ C c=1 α ℓ,c D HM (z ℓ , S ℓ c ), ( 6

Comparison to state-of-the-art detection methods.

We benchmark our approach with three supervised detection methods: LID, KD-BU, and NSS. We chose these baselines because they are supervised and do not modify the model to protect. We could consider ABUSNAINA and collab. HAMPER. Our proposed HAMPER detector is computing, thanks to the halfspacemass depth, the distance of a given x to a reference training distribution. In the ATTACK DETECTION sense that it compares between a novel point and a reference, our method is close to the LID method, however, as explained in the previous section, we do not compute our anomality score (i.e., the HM depth) at each layer's output, but we only use a subset of layers Λ. In addition, it is possible to use our proposed detector under both scenarios, i.e., the attack-aware and the blind-to-attack scenarios.

Analyzing Statistical Information of the Networks'

Behavior under Threats

In this section, we provide insights into the attackers' and defenses' behavior from the classifier's perspective. This section aims to provide justification and insights on the choice of making the linear weights of Equation 6.7 dependant on both the class and the layer. In particular, we analyze the behavior of the classifier on attacks in Subsection 6.4.2, while in Subsection 6.4.3 we explore which subset of layers of the classifier carries the relevant information to build an efficient supervised data-depth based detector. we use a ResNet-18 trained for 100 epochs, using an SGD optimizer with a learning rate of 0.1, weight decay of 10 -5 , and a momentum of 0.9; for CIFAR100 we chose a

Experimental setting

ResNet-110 pre-trained using an SGD optimizer with a learning rate of 0.1, weight decay of 10 -5 , and a momentum of 0.9.

Once trained, all classifiers are frozen.

Attacks & choice of the maximal allowed perturbation ε. Takeaways. The variability of the per-class behavior of the classifier under threats suggests that class is an important characteristic and should be leveraged to detect adversaries. This observation further motivates the per-class computation of the halfspace-mass depths and then the class dependency of the linear regressor of Equation 6.7.

Analyzing the networks' per layer behavior under threats

In this section, we investigate the per-layer behavior of the image classifier to motivate and justify the choice of the layer behavior of the proposed aggregation procedure (see Equation 6.7). Analysis. From Figure 6.4, we observe of the decision of the detector is based on several layers. This a-posteriori analysis justifies the layer dependency of the linear regressor weights. As the L 2 weight regularization increases, the last layers receive more weights, suggesting they are good candidates to build a detector. Although it is possible to rely on the whole set of layers, motivated by efficiency and Figure 6.4, we select the 11 final layers of the classifier, i.e., we base our experiments on the subset Λ = {L -12, . . . , L -1}.

Takeaways. Through Figure 6.4, we have motivated both the per-layer computation of the halfspace-mass depths and then the per-layer dependency of the linear regressor of Equation 6.7.

Experiments

In this section, we assess the effectiveness of our proposed depth-based detection method. The code will be made available at github.com. This section is organized as follows: we first describe the experimental setting in Subsection 6.5.1 and then we provide a detailed discussion of the results in Subsection 6.5.2.

Experimental setting

We refer to Subsection 6.5.1 for the datasets, the classifiers and the attacks we considered for our evaluation.

Evaluation metrics. For each threat scenario, the performance is measured in terms of two metrics: AUROC↑ (higher is better): the Area Under the Receiver Operating Characteristic curve (ROC;DAVIS and GOADRICH, 2006) Remark. The ideal classifier would reach 100% of AUROC↑ and 0% of FPR↓ 95% .

Detecting adversarial examples

We recall from Subsection 6.2.2 that we distinguish between two settings in the supervised context: the attack-aware scenario and the blind-to-attack scenario. Therefore, we conduct two sets of experiments. In the attack-aware scenario, for each attack we train a detector on a validation set -composed of the first 1000 samples of the testing set -and tested on the remaining samples. We refer here to HAMPER-Attack-Aware (HAMPER AA ) and we compare it with LID and KD-BU. In the blind-to-attack scenario, we train a unique detector and we test it on all the possible attacks. We refer here to HAMPER-Blind-to-Attack (HAMPER BA ) and we compare it with NSS. Note that, while our competitors assign to each input sample the probability of being adversarial, i.e., adversarial samples are labeled as 1, in HAMPER the detector outputs the depth score. This means that a high score corresponds to a deep sample, i.e., a natural one, hence clean samples are labeled as 1. ATTACK DETECTION SVHN,14.6(25.3) on CIFAR10 and 31.8(81.6) on CIFAR100 in terms of AUROC↑(FPR↓ 95% ).

Under L 1 , L 2 and L ∞ -norm constraints, HAMPER BA outperforms NSS on all considered datasets. The increase goes up to 30.2(78.0) in L 1 -norm, 37.4(89.9) in L 2 -norm and 30.8(81.7) in L ∞ -norm in terms of in AUROC↑(FPR↓ 95% ).

Performance analysis per ε. Overall, the results in Table B.2 show that, in contrast to HAMPER BA , NSS' performances are increasing with the value of maximal allowed perturbation ϵ. As a matter of fact, the performances of HAMPER BA on L 1 and L 2norm constraints first decrease as ε increases (ε ∈ [5, 15] for L 1 -norm constraint, and ε ∈ [0.125, 0.3125] for L 2 -norm constraint), until it starts increasing. On L ∞norm constraint, our method follows the expected behavior, i.e., the performances increases with ε.

Performance analysis per type of threat. Table 6.2 suggests that both HAMPER BA and NSS are globally better at detecting attacks with L ∞ -norm constraints, particularly those created with PGD and BIM as an attack strategy. However, on SVHN NSS finds more difficult to spot the FGSM-based attacks. SA and DeepFool threats are the thoughest to detect for NSS. On the contrary, while HAMPER BA consistently detect SA-based attack, it shows a slight drop in performance for DeepFool-based attacks on CIFAR10. Finally, NSS presents poor performances at detecting CW 2 attacks, while it is not the case for our proposed method.

Summary. While NSS's performances vary with the dataset, the ε and the norm used to construct the attacks, HAMPER BA consistently detect them. In particular, we note that HAMPER BA is well suited to larger datasets (e.g., CIFAR100). Conversely, NSS' performance highly decreases when passing from the datasets with 10 classes to the one with 100.

Attacking HAMPER using adaptive adversary

The importance of attacking defenses with adaptive attacks has increased recently [ABUSNAINA and collab., 2021;RAGHURAM and collab., 2021]. As mentionned in Subsection 6.2.3, the Backward-Pass Differentiable Attack (BPDA; ATHALYE and collab., 2018a) is based on the possibility to find a suitable surrogate to the non-differentiable parts of any defense. However, deriving a suitable differentiable surrogate of the halfspace-mass depth remains a open research question which has never been tackled. As a matter of fact, the only attempts to approximate a non-differentiable depth was performed on the Tuckey depth in SHE and collab. [2021], with very poor results [DYCKERHOFF and collab., 2021]. Finding a differentiable suitable surrogate to attack HAMPER would, therefore, require a substantial effort and should be rigorously handled. As a consequence, we have to rely on adaptive blackbox attackers, as suggested in TRAMER and collab. [2020], to attack HAMPER.

Experimental Setting. We designed a blackbox adaptive attacker by using SA, which is both effective and computationally efficient [ENGSTROM and collab., 2019].

To extend SA into an adaptive attack, we modified the success criterion to not only fool the classifier but also the detector, and the loss, which becomes a trade-off between minimizing the difference between the logits of the two most probable classes and maximizing the detector's prediction. In Table 6.3, we present the results of the adaptive SA attack on CIFAR10 for both NSS and HAMPER BA .

Analysis. On Table 6.3, we varied the value of the parameter α controlling the trade-off between the classifier and the detector performances. As α increases, the performance of the attack on the classifier decreases while the detector has more and more trouble detecting them, as one can expect. On the considered adaptive attacks, it is clear that it is easy for the attacker to find powerful and undetected samples to attack NSS. However, it is more difficult to fool our proposed method. To decrease the AUROC↑ to a value below 50 (which is equivalent to a random detector), the attacker is only able to fool the classifier 25% of the time.

Takeaways. HAMPER BA is more robust to adaptive attacks than NSS.

Further analysis of the detector behaviors

In this section, we first investigate the per-class behavior of the detectors to further asses the effectiveness of the proposed method (cf. Section 6.5.4). Then, we study the AUROC↑/FPR↓ 95% trade-off (cf. Section 6.5.4) and the time and resource constraints (cf. Section 6.5.4) for each of the considered methods. this experiment, we study the performance distribution per class.

Analyzing the detectors' per-class behavior

Simulation. In Figure 6.5, we examine the accuracy of the detectors on the testing samples (natural and adversarial) w.r.t. the class predicted by the classifier. Note that, for visualization purposes, we select the 10-most/least populated classes in the case of CIFAR100. For this simulation, we select the threshold γ for which TPR is at 95%. This is demonstrated by Figure 6.5a where NSS, KD-BU and LID obtain the highest accuracy in class 4 on SVHN, which is also the most populated class (cf. Figure 6.2a).

Analysis

A similar behavior is observable in CIFAR10 (cf. classes 3 and 4 in Figure 6.5b and Figure 6.2b). Moreover, Figure 6.5 suggests that, in terms of accuracy per class, the SOTA methods show similar performances on SVHN; on the contrary, on CIFAR10 the detector performing the best is NSS while on CIFAR100 it is LID.

Takeaways. Differently from the competitors, the HAMPER detectors are not affected by the per-class distribution of the samples. In particular, and regardless of the dataset, the proposed detectors show a uniform behavior overall the classes.

Further confirmation is given from the plots in Figure 6.5c and Figure 6.5d where the accuracies of the detectors remain constant even when focusing on only the most and the least-populated classes respectively.

Studying the AUROC↑-FPR↓ 95% relationship

As commonly done in anomaly detection, we measure the detection performances in terms of AUROC↑ and FPR↓ 95% . However, to a large AUROC↑ does not necessarily ATTACK DETECTION correspond a low FPR↓ 95% . In this experiment, we study the trade-off between AUROC↑ and FPR↓ 95% performances for each considered detection method.

Simulation. In Figure 7.5, we analyze the trade-off between AUROC↑ and FPR↓ 95% .

We translate on the Cartesian planes the results presented in Table B.1 and in Table B.2. The perfect detector will have the points in (100, 0).

Analysis. Interestingly, the HAMPER detectors behave closely to the perfect detector for all the considered attacks. This is particularly true for HAMPER AA . For NSS, KD-BU and LID, a large AUROC↑ does not necessarily correspond a low FPR↓ 95% . In SVHN (Figure 6.6a), five of the NSS points are exhibiting high AUROC↑ (between 70% and 92%) while presenting extremely high FPR↓ 95% (between 81% and 99.6%). A similar behavior is presented in CIFAR10 with LID.

Takeaways. Contrary to other detection method, which can exhibit high FPR↓ 95% for high AUROC↑, our proposed detectors behave similarly to the perfect detector for both attack-aware and blind-to-attack scenarios.

Time and resources constraints

For some applications, time and resource necessity can be critical. We, therefore, decided to measure the constraints of each considered method. Simulations. In Table 6.4, We report the time and resource constraints needed for each method. HAMPER AA V100-16G 02m00s 02m00s

HAMPER BA V100-16G 02m00s 02m00s

Takeaways. HAMPER's deployment requires comparable time and resources to the other considered detectors.

Concluding Remarks and Future Work

In this paper, we introduced HAMPER, a simple and effective method to detect adversarial attacks. One of the keys of HAMPER is to rely on the halfspace-mass depth, a statistical tool that remains overlooked by the machine learning community. and collab., 2017;[START_REF] Meinke | Neural networks that provably know when they don't know[END_REF]) or in autonomous driving [START_REF] Bojarski | End to end learning for self-driving cars[END_REF]GUO and collab., 2017]. Therefore, it is crucial to deploy a proper strategy to defend against adversarial attacks [AMODEI and collab., 2016].

In this context, the task of distinguishing adversarial samples from clean ones is becoming increasingly challenging as developing new attacks is getting more attention from the community [CROCE and HEIN, 2020;DENG and KARAM, 2020a,b;DONG and collab., 2019;DUAN and collab., 2020;GAO and collab., 2021;JIA and collab., 2020;LIN and collab., 2019;NASEER and collab., 2021;WANG and collab., 2021a;WU and collab., 2020b;ZHAO and collab., 2020] FEINMAN and collab., 2017;KHERCHOUCHE and collab., 2020;MA and collab., 2018]; the unsupervised ones, instead, do not rely on prior knowledge of attacks and, in general, only learn from clean data at the time of training [MENG and CHEN, 2017;XU and collab., 2018].

In this work, we focus on the unsupervised scenario, which is often a reasonable approach to real-world use-cases. We model the adversarial detection problem as an anomaly detection framework [BREUNIG and collab., 2000;CHANDOLA and collab., 2009;LIU and collab., 2008;SCHÖLKOPF and collab., 2001;STAERMAN andcollab., 2019, 2020], where the aim is to identify abnormal observations without seeing them during training. Statistical tools called data depths are natural similarity score in this context. Data depths have a simple geometric interpretation as they provide center-outward ordering of points with respect to a probability distribution [TUKEY, 1975;ZUO and SERFLING, 2000]. Geometrically speaking, the data depths measure how deep a sample is in a given distribution. Although data depths have received attention from the statistical community, they remain overlooked by the machine learning community.

Contributions.

Our contributions can be summarized as follows:

1. Building on novel tools: data depths. Our first contribution is to introduce AP- FEINMAN and collab., 2017;KHERCHOUCHE and collab., 2020;MA and collab., 2018;MENG and CHEN, 2017;XU and collab., 2018] focusing on ResNets [HE and collab., 2016], we choose to benchmark APPROVED on vision transformers models [CHEN and collab., 2021;DOSOVITSKIY and collab., 2021;STEINER and collab., 2021;TOLSTIKHIN and collab., 2021;ZHAI and collab., 2022].

Indeed, such networks achieve state-of-the-art results on several visual tasks, including object detection [HE and collab., 2021], image classification [WANG and collab., 2021b] and generation [PARMAR and collab., 2018], largely outperforming ResNets.

Interestingly enough, we empirically observe that transformers behave differently from ResNets, which justifies the need to develop detection techniques such as AP-PROVED, that leverage the specific features of transformers' architectures. Moreover, to avoid overfitting on a specific attack, we test our detection method on a wide range of attack mechanisms.

Ensuring reproducibility.

We provide the open-source code of our method, attacks, and baseline to ensure reproducibility and reduce future research computation and coding overhead.

Organization of the paper. The paper is organized as follows. In Section 7.2, we review detection methods along with attack mechanisms. In Section 7.3, we introduce our detector APPROVED and focus on the description of the data depth on which it relies. In Section 7.4, we study the performance of adversarial attacks on vision transformers and give insights on models' behavior under threat. In Section 7.5, we evaluate APPROVED through numerical experiments and concluding remarks are relegated to Section 7.6.

Background and Related Work

ℓ c = 1 |D c | x∈D c δ f ℓ θ (x)
where δ u is the dirac mass at point u. Let f ℓ θ : X → R d ℓ with ℓ ∈ {1, . . . , L}, denote the output of the ℓ-th layer of the DNN, where d ℓ is the dimension of the latent space induced by the ℓ-th layer. The class prediction is obtained from the L-th layer softmax output as follows:

f L θ (x) ≜ arg max c∈Y q θ (c|x) with q θ (•|x) = softmax( f L-1 θ (x)).

Review of attack mechanisms

The existence of adversarial examples and their capability to lure a deep neural network have been first introduced in SZEGEDY and collab. [2014]. The authors define the adversarial generation problem as:

x ′ = arg min x ′ ∈R d : ∥x ′ -x∥ p <ε ∥x ′ -x∥ p s.t. f L θ (x ′ ) ̸ = y, ( 7.1)
where y is the true label associated to a natural sample x ∈ X being modified, ∥•∥ p is the L p -norm operator, and ε is the maximal allowed perturbation. and collab., 2018;LU and collab., 2017;METZEN and collab., 2017] and CHEN, 2017], network invariant [MA and collab., 2019] or training an auxiliary model [ALDAHDOOH and collab., 2021b;SOTGIU and collab., 2020;ZHENG and HONG, 2018]. RAGHURAM and collab.

[2021] uses dimension reduction, kNN and layer aggregation to distinguish between natural and adversarial samples. In this paper, we only focus on unsupervised methods that cannot act on the model to be protected.

Our Proposed Detector

Background on data depth

The 1975], the simplicial depth [START_REF] Liu | On a notion of data depth based on random simplices[END_REF], the projection depth [LIU, 1992] or the zonoid depth [KOSHEVOY and MOSLER, 1997]. Despite their applications in statistics and machine learning (e.g., regression [HALLIN and collab., 2010;[START_REF] Rousseeuw | Regression depth[END_REF], classification [MOZHAROVSKYI and collab., 2015], automatic text evaluation [STAERMAN and collab., 2021b] or anomaly detection [START_REF] Rousseeuw | Anomaly detection by robust statistics[END_REF]SERFLING, 2006;STAERMAN andcollab., 2022, 2020]) the use of data depth with representation models, and more generally to deep learning, remains overlooked by the community. The halfspace depth is the first and the most studied depth in the literature probably due to its appealing properties [START_REF] Donoho | Breakdown properties of location estimates based on half space depth and projected outlyingness[END_REF]ZUO and SERFLING, 2000] as well as its connections with univariate quantiles. However, it suffers from computational burden in practice [START_REF] Dyckerhoff | Exact computation of the halfspace depth[END_REF][START_REF] Rousseeuw | Computing location depth and regression depth in higher dimensions[END_REF]]. Indeed, it requires solving an optimization problem over the unit hypersphere of a non-differentiable quantity. To remedy this drawback, [RAMSAY and collab., 2019] introduced the Integrated Rank-Weighted (IRW) depth (see also CHEN and collab. [2015]; STAERMAN and collab. [2021a]), which involves an expectation as an alternative to the infimum over the unit hypersphere of the halfspace depth, making it easier to compute. The IRW depth is scale and translation invariant and has been successfully applied to anomaly detection [CHEN and collab., 2015;STAERMAN and collab., 2021a] making it a good candidate for our purposes. Formally, the IRW depth is defined as:

D IRW (z, p Z ) = S d -1
min {F u (〈u, z〉) , 1 -F u (〈u, z〉)} du, (7.5)

where the unit hypersphere is denoted as S then linear in all of its parameters, is defined as:

D MC IRW (z, S n ) = 1 n proj n proj k=1 min 1 n n i =1 I {〈u k , z i -z〉 ≤ 0} , 1
n n i =1 I {〈u k , z i -z〉 > 0} , (7.6)

APPROVED: Our depth-based detector

Intuition. Our detector tries to answer this simple question: can we find a metric that will be able to distinguish between natural and arbitrary adversarial samples? At the logit layer, we want to compare the new input to the training samples of its predicted class to measure whether the new sample is behaving as expected. Data depths, particularly the IRW depth, are serious candidates as they measure the 'distance' θ (x), p L-1 ŷ ) > γ, otherwise, it is considered as adversarial. A classical way to select γ it by selecting an amount of training samples the detector can wrongfully detect.

Comparison with existing detectors

We benchmark our approach with two unsupervised detection methods: FS and MagNet. We chose these baselines because they are unsupervised and do not modify the model to protect. We could consider NIC [START_REF] Ma | NIC: detecting adversarial samples with neural network invariant checking[END_REF] but extracting features at each layer is computationally expensive.

The Feature Squeezing method (FS; [XU and collab., 2018]). It computes the feature squeezing of the input, extracts its prediction, and compares it to the original prediction. The further away they are, the more likely the input is adversarial. In practice, four versions of the input are needed: the original input, a low-precision version, a median-filtered version, and a denoising-filtered version. One inference on the model is required for each of the four inputs. Later, the maximal L 1 difference between the original prediction and each of the other three is picked. FS is, therefore, parameter-free and does not require training. However, the necessary time to extract the essential features and the memory needed to store all the input modifications and their prediction are quite high.

MagNet [MENG and CHEN, 2017]. It is based on the training of two components:

first, a detector that detects if a sample x is clean or not, then, a reformer that tries to find an approximation of the input closer to the training manifold. In practice, each of those components is an autoencoder that must be trained on clean samples before testing new inputs. MagNet requires three inferences at testing time, one on the detector, one on the reformer, and one on the original model to protect. We also observe that for APPROVED the most challenging task is to distinguish natural and adversarial samples when they are crafted with DeepFool. However, it is the best choice even in this case as it reaches better performances than the other detectors.

Adaptive attacks

In this experiment, we evaluate APPROVED against adaptive attacks, which has knowledge about the defense [ATHALYE and collab., 2018;CARLINI and WAGNER, 2017;TRAMER and collab., 2020]. to allow the attacker to fool both the detection method as well as the classifier. We rely on an hyperparameter α that weights the relative importance of the two parts of the objective. It is straightforward (cf. Figure 7.4) that APPROVED is less sensitive to adaptive attacks than FS. This results further validates the use of the IRW depth to craft detection method and further assesses the superiority of APPROVED. corner. On CIFAR100 and Tiny ImageNet, the results for our method are slightly more spread out in the top left and center of the figure, while for FS and MagNet, they are still in the center and bottom right, respectively. Note that FS has a different behavior than expected, i.e., the line connecting the top left corner with the bottom right corner. This behavior change can be observed for FPR↓ 90% between 25%-35% on CIFAR10 and between 50%-75% on CIFAR100 and Tiny ImageNet. On CIFAR10, FS presents a lower AUROC↑ for a fixed FPR↓ 90% than expected, whereas, on CIFAR100, it presents a lower AUROC↑ (for FPR↓ 90% values between 50%-60%) or higher (for FPR↓ 90% values around 75%) than expected.

Conclusions and Limitations

We introduced APPROVED, an efficient unsupervised detection method designed to defend against adversarial attacks. In contrast with previous detection methods, which were built for ResNet architectures, APPROVED is well suited for vision transformers which nowadays represent the state-of-the-art. While the relevant information about the discrepancy between clean and adversarial samples is distributed across all layers We conduct extensive numerical experiments and prove that APPROVED outperforms the other state-of-the-art methods significantly.

Future Research. We think our method paves the way for future research efforts.

Indeed, there is still room for improvement: even if the AUROC↑ performances are good, the FPR↓ 90% are also fairly high. We believe the idea of leveraging information contained in layers of transformers through data depths can be fruitful in improving defense mechanisms against adversarial attacks. Our research is expected to have a positive societal impact by protecting the integrity of AI systems, especially necessary in critical systems such as autonomous cars [MORGULIS and collab., 2019] or stock predictions [XIE and collab., 2022].

Chapter 7 Conclusion

In this work, we presented an efficient unsupervised detection method designed to defend Vision Transformers from adversarial attacks. While distributed across all layers of ResNets, the information about the difference between natural and attacked samples is concentrated in the logit layer for the transformers. This motivated us to build a detection method on top of this layer by computing a similarity score between an input sample and the training distribution based on the statistical notion of data depth. We chose to use the Integrated Rank-Weighted depth, which lends itself to fast inference computations and is non-differentiable, making it harder for gradient-based adversarial methods to craft malicious samples. Experimentally, we proved that APPROVED outperforms the other state-of-the-art methods significantly. In the unsupervised case, where the defense does not have any knowledge about the possible threats it is going to face, our proposed data-depth method named APPROVED significantly outperforms other state-of-the-art methods, both in terms of performances under classical and adaptive attacks, and in terms of computational requirements. While there are still improvements to provide in all cases, our different experiments suggest that data-depths are valuable tools to distinguish between natural and attacked samples.

Limitations and Potential Future Works

This section will put our work in perspective and mention some potential future work.

Limitations of our proposed methods

Using the Fisher-Rao distance to improve defenses. While the Fisher-Rao distance has a simple geometric interpretation, and its use over the Kullback-Leibler divergence as a regularizer improves the trade-off between natural and robust performances of deep classifiers, we do not have theoretical guarantees over the performances. In addition, while deeply connected, our experiments suggest that the Fisher-Rao distance and the Kullback-Leibler divergence behave differently. A theoretical explanation of this behavior change could be interesting.

Protecting state estimators in Smart Grid systems. We proposed methods to protect state estimators against attacks. While our results suggest that our method is CHAPTER 8. DISCUSSION OF FINDINGS, LIMITATIONS, POTENTIAL FUTURE WORKS AND GLOBAL SUMMARY efficient, we could not test it on real-world data since they were unavailable.

Building strong detection methods. While significantly improving the state-ofthe-art results on multiple threats, the proposed methods are not perfect. Adversarial attacks, while representing the worst-case scenario, are not the only possible alteration of the input. However, we do not know if our proposed methods can detect all potential threats in all possible scenarios, as we focused solely on protecting deep classifiers against adversarial attacks.

Potential future work related to: How can we use the internal

structure of DNN's output space to improve its robustness?

• Extention of our method to multimodal models. The will of the community to build multimodal systems, which learn from more than one type of modality,

typically combining text and image data, has gained importance over the last few months [ALAYRAC and collab., 2022;GARCIA and collab., 2019;[START_REF] Radford | Learning transferable visual models from natural language supervision[END_REF]RAMESH andcollab., 2022, 2021]. Recently, methods to attack such systems have been developed [EVTIMOV and collab., 2020;YI and collab., 2021;ZHANG and collab., 2022]. Protecting those networks will therefore be necessary when these models are deployed in practical scenarios. However, protecting such systems is not trivial, as textual data is discrete and based on a dictionary while image data is not, therefore, we need to ensure that all modalities are taken into account and protected.

• Protecting Vision Transformers. Vision Transformers (ViTs) are achieving state-of-the-art performances on multiple vision tasks today [bey;FAYYAZ and collab., 2021;GRAHAM and collab., 2021;HEO and collab., 2021;LEE and collab., 2021;LI andcollab., 2021, 2022;[START_REF] Mehta | Mobilevit: Light-weight, general-purpose, and mobile-friendly vision transformer[END_REF]RENGGLI and collab., 2022;SANDLER and collab., 2022;TOUVRON andcollab., 2022, 2021;TU and collab., 2022;YANG and collab., 2022]. However, it has been widely overlooked by the robustness community, just a few works have been studying

whether ViT where, by design, more robust than classic convolutional networks [ALDAHDOOH and collab., 2021a;BENZ and collab., 2021;MAHMOOD and collab., 2021]. It would therefore be interesting to adapt our method to improve the robustness of Vision Transformers on multiple vision tasks. This task in not trivial as Vision Transformers are a huge amount of parameters and their training on natural samples is already heavy.

• Additional theoretical comparison between Fisher-Rao distance and other metrics. We have shown, in Chapter 3 using a toy example, that the optimization based on FIRE is well-behaved and gives all the desired Pareto-optimal CHAPTER 8. DISCUSSION OF FINDINGS, LIMITATIONS, POTENTIAL FUTURE WORKS AND GLOBAL SUMMARY points in the natural-adversarial region. This observation contrasts with the results of other state-of-the-art adversarial learning approaches. Further theoretical explanations of this behavior change would be worth exploring.

8.2.3

Potential future work related to: How can we craft an efficient and effective detection method based on simple tools?

• Extention to different underlying tasks. Our proposed methods can be extended to any underlying tasks. Attacks have started to be designed for a wide range of vision tasks, such as image segmentation [HENDRIK METZEN and collab., 2017;XIE and collab., 2017], object recognition [XIE and collab., 2017],

speech recognition [CISSE and collab., 2017]. It, therefore, would be interesting to try to protect these underlying tasks. Moreover, other fields have protection necessities such as securing textual pre-trained models [CHAPUIS and collab., 2020;COLOMBO and collab., 2021a;DEVLIN and collab., 2018;DINKAR and collab., 2020;YANG and collab., 2019], which remains in its infancy. In NLP, many opportunities exist where attacks could help quantify model robustness. [2022]; COLOMBO and collab. [2021bCOLOMBO and collab. [ , 2022aCOLOMBO and collab. [ ,c, 2021d]]; STAERMAN and collab.

[2021] as they are widely used and often deployed in real-world scenarios. As each architecture and each application presents its specificities, adapting our proposed methods to other tasks is not straightforward.

• Using information tools to protect systems. As shown in Part II, the datadepth is an interesting tool to distinguish between normal and modified inputs. learning models can be extracted from the network's layers using the samples directly [CARRARA and collab., 2018;COLOMBO, 2021b;LU and collab., 2017;METZEN and collab., 2017], or pre-process them using kernel density estimation or uncertainty measure [FEINMAN and collab., 2017], computer vision specific characteristics such as natural scene statistics [KHERCHOUCHE and collab., 2020], PCA [START_REF] Li | Adversarial examples detection in deep networks with convolutional filter statistics[END_REF] or also local intrinsic dimensionality [MA and collab., 2018]. Regarding (R2), one can arguably say that these methods are satisfying as the inference time of simple machine learning models is fast.

However, as they do not satisfy (R3), these methods need to make some assumptions on the nature of adversarial attacks to generate malicious samples at the risk of overfitting and misgeneralizing. Moreover, most of these methods rely on the hidden layers of the networks, which makes them unrealistic for practical black-box applications (R1) where only softmax are available.

Global research directions

Adapting the adversarial problem to the real world. The adversarial problem represents a worst-case scenario and is an exciting problem to solve. However, this scenario is not necessarily representative of what could happen in real-world applications. For example, providing full knowledge about a system to an attacker is probably too ambitious. In addition, defenses can be combined, i.e., it is possible to craft a detector on top of a robust network. Finally, defending directly on the neural network to protect is not the only type of defense that exists. For example, one can monitor the activities of its users to ensure that no strange behavior is happening, such as an abnormal number of queries in a short amount of time. While the developed tools for the adversarial problem are powerful and interesting and give us insights into the functioning of neural networks, there is still a lot to do to craft efficient defensive methods that can be applied to the industry in more practical APPENDIX A. APPENDIX OF CHAPTER 3 The Hellinger distance between two distributions q and q ′ on Y is defined as follows:

H(q, q ′ ) . = 2 1 -y∈Y q(y)q(y) ′ Theorem 4 (Relation between FRD and Hellinger distance). The FRD between softpredictions q θ = q θ (•|x) and q ′ θ = q θ (•|x ′ ), given by Equation 3.15 is related to the Hellinger distance through the relation H 2 (q θ , q ′ θ ) = 2 1cos d R,C (q θ , q ′ θ ) 2 . (A.2)

Since 0 ≤ d R,C (q θ , q ′ θ ) ≤ π, it is clear that H 2 (q θ , q ′ θ ) is a monotonically increasing function of d R,C (q θ , q ′ θ ).

We conclude from this result that the FRD and the Hellinger distance are theoretically equivalent regularization mechanisms. However, it is clear that the empirical optimization of these distances may be different. This is further explored and confirmed in the following.

Experimental comparison

Since the Hellinger distance and the Fisher-Rao distance are theoretically equivalent metrics (see Subsection A.2.1), we investigate the empirical difference between those two distances. First, we perform the same simulations as those presented in Draw S ℓ c,n s , a sub-sample of S ℓ c with size n s without replacement.

3:

Draw randomly and uniformly a direction u k in S d -1 .

4:

Compute 〈u k , z ℓ,i 〉 for every z ℓ,i ∈ S ℓ c,n s such that p k,i ≜ 〈u k , z ℓ,i 〉. Project z ℓ onto u k and such that p ℓ k = 〈z ℓ , u k 〉. Draw K closed halfspaces containing z ℓ using Algorithm 5.

4:

Compute D HM (z ℓ , S ℓ c ) using Algorithm 6.

C.6 Per Class Analysis

As for CIFAR10 (see Section 3.5), the detector performances depend on the predicted class. Some classes are easy to detect (i.e., classes 0, 21, 53, 75, and 94), others are more difficult (i.e., classes 3, 10, 33, 47, 60, 74, and 93). Some have low variance (i.e., 0, 1, 24, 34, 75, 82 and, 94) while others have an extremely large dispersion (i.e., 11, 35,47, 52, 96, and 98).

2. 1

 1 Illustration of a Multi-Layer Perceptron . . . . . . . . . . . . . . . . . . . 2.2 CNN structure. More specifically, LeNet's [LECUN and collab., 1989] structure. Source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3 Representation of a ResNet residual block, source: [HE and collab., 2016], Fig.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Representation of the Vision Transformer's structure. Source: [DOSO-VITSKIY and collab., 2020], Fig.1 . . . . . . . . . . . . . . . . . . . . . . . . 2.5 Example of the effect of an adversarial sample on ImageNet using a GoogLeNet as the targeted model. Source: [GOODFELLOW and collab., 2015], Fig.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6 DF method for a linear binary classifier. Source: [MOOSAVI-DEZFOOLI and collab., 2016], Fig.2 . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Source:[CROCE and HEIN, 2020a], Fig.1 . . . . . . . . . . . . . . . . . . . 2.8 Natural Scene Statistics extraction on a natural sample, and various adversarial ones. Source: [KHERCHOUCHE and collab., 2020], Fig.1a . . 2.9 Our current grid' structure needs to change to fulfill UN goals to change our world. Source: [HEINRICH, 2018], p.33. . . . . . . . . . . . . . . . . .

1 /

 1 [1+exp(-y θ ⊺ x)] with different values of θ: (a) Parameters minimizing the natural misclassification error probability P e , (b) Parameters minimizing the adversarial misclassification error probability P ′ e . ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.3 FRD between the distributions q θ (•|x) and q θ (•|x ′ ) as a function of δ using the logistic model with different values of θ: (a) Parameters minimizing the natural misclassification error probability P e , (b) Parameters minimizing the adversarial misclassification error probability P ′ e . ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 3.4 Visualization of statistical manifold C defined by the model q θ (y|x) = 1/[1 + exp(-y θ ⊺ x)] when minimizing the FIRE risk function for different values of λ: (a) No adversarial FRD regularization, (b) Medium adversarial FRD regularization, (c) High adversarial FRD regularization. ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 3.5 Comparison between FRD and Euclidean distance. ©2022 IEEE. . . . . 99 3.6 Plot of all the possible points (1-P e (θ), 1-P ′ e (θ)) for the Gaussian model with ε = 0.1, µ = [-0.0218; 0.0425] and Σ = [0.0212, 0.0036; 0.0036, 0.0042] shown in blue. In red, we show the Pareto-optimal points (Figure 3.6a).

  .7 (Figure 3.6c).©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Influence of the hyperparameter λ on the natural and adversarial accuracies for FIRE regularizer on CIFAR-10. ©2022 IEEE. . . . . . . . . 4.1 Error between natural and noisy samples using the state estimatoraware attack. ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Errors between natural and corrupted performances as a function of the percentage of perturbed meters, for λ = 10 and without defense. ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Estimated-angles mean errors and portions of detected samples for a robust estimator as a function of the β parameter. The natural errors are in plain lines, while the attacked ones are in dashes. ©2022 IEEE. . 5.1 Averaged angle and tension attack-induced errors, and percentage of detected noisy samples under (a) the deterministic attack, (b) the random attack with no defense, as a function of λ, the attack hyperparameter. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LIST OF FIGURES 5.2 Influence of the percentage of attacked meters on the angle and tension attack-induced errors and percentage of detected samples under (a) the deterministic attack with λ = 1000, (b) the random attack with λ = 10 with no defense. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.3 Natural and attacked angle and tension estimation errors averaged over the buses and the samples for (a) the deterministic attack with λ = 1000 and 25% of attacked meters, (b) the random attack with λ = 10 and 25% of attacked meters. Natural values are in plain line, attacked ones are in dashes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1 Average performances of our method (i.e., HAMPER) in an attack-aware and single detector setting, along with the performances of state-of-theart detection mechanisms (i.e., NSS, LID, KD-BU), on three classically considered datasets (i.e., SVHN, CIFAR10 and CIFAR100). Below the dataset names is the accuracy of their underlying classifiers. In addition to outperforming other methods on all three considered datasets, our method, contrary to the others, does not lose performances as the classifier's accuracy decreases. . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Calibrating the maximal allowed perturbation ε on CIFAR100. Accuracy on adversarial examples created using PGD 1 for the SVHN, CI-FAR10 and CIFAR100 classifiers. On CIFAR100, to ensure high successes of the attacks, one must allow the attacker to have larger values of ε, compared to the CIFAR10 and SVHN ones. . . . . . . . . . . . . . . . . . 6.2 Per class behavior analysis. Average number of adversarial examples per class on each of the considered datasets. . . . . . . . . . . . . . . . . 6.4 Per layer behavior analysis. Evolution of ᾱℓ = 1 C c α ℓ,c the average and the standard deviation over the classes of the regressor weight as a function of the layer, for different value of the regularization parameter. 6.5 Accuracies of the detectors per predicted classifier class. For visualization reasons, we restrict the plot to the 10-most/least populated classes for CIFAR100. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 FPR↓ 95% as a function of the AUROC↑ for all five considered methods, (a) on SVHN, (b) on CIFAR10, and (c) on CIFAR100. . . . . . . . . . . . . 7.1 Percentage of successful attacks depending on the L p -norm constraint, the maximal perturbation ε and the attack algorithm on ResNet18 (green) and ViT (orange). . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 Difference between natural and adversarial IRW depth values as a function of the layer on ViT (top) and on ResNet18 (bottom), averaged over the attacks. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 APPROVED's AUROC↑ and FPR↓ 90% per class, averaged over CIFAR10. . .

  ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Test robustness on different datasets under white-box l ∞ attack. We ran all methods on 5 different tries and reported the mean and standard deviation. The codes for UAT and Atzmon et al. are not publicly available. Note that retraining the SOTA methods modifies slightly the experimental results. ©2022 IEEE. . . . . . . . . . . . . . . . . . . . . . . 4.1 Clean and Attacked Mean Error for Defenseless and Robust Estimator for different Bus Systems. ©2022 IEEE. . . . . . . . . . . . . . . . . . . . 5.1 Natural and Attacked Mean Error for Defenseless, Robust and LASSO state estimators for different Bus Systems . . . . . . . . . . . . . . . . . . 6.1 Attack-aware performances on the three considered datasets -SVHN, CIFAR10 and CIFAR100 -of HAMPER AA detector together with the results of the SOTA detection methods: LID, and KD-BU, averaged over the L p -norm constraint. The best results among the detectors are shown in bold. The results are presented as AUROC↑ ±FPR ↓ 95% % and in terms of mean (µ) and standard deviation (σ). . . . . . . . . . . . . . . . . . . . . 6.2 Blind-to-Attack detector performances on the three considered datasets -SVHN, CIFAR10, and CIFAR100 -of the HAMPER BA detector together with the results of the state-of-the-art detection methods, i.e., NSS, averaged over the L p -norm constraint, along with the average and global performances. The best results among the detectors are shown in bold. The results are presented as AUROC↑% ±FPR ↓ 95% % and The results are presented as AUROC↑ ±FPR ↓ 95% % and in terms of mean (µ) and standard deviation (σ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . LIST OF TABLES C.4 AUROC↑ and FPR↓ 90% for each considered attack mechanism, L p -norm constraint and ε on CIFAR100 for APPROVED, FS and MagNet. The best result for each attack is shown in bold. . . . . . . . . . . . . . . . . . . . 252 C.5 AUROC↑ and FPR↓ 90% for each considered attack mechanism, L p -norm constraint and ε on Tiny ImageNet for APPROVEDand FS. The best result for each attack is shown in bold. . . . . . . . . . . . . . . . . . . . . . . . 253 Acknowledgments They are many people I would like to thank for their contribution during my Ph.D., in many different ways. First, I would like to express the most profound appreciation for my two supervisors, Prof. Fabrice Labeau and Prof. Pablo Piantanida, for their guidance, and trust during those last four years. Their valuable feedback and our discussions were a great part of this doctorate. I also want to thank the French Ministère de l'Éducation Nationale, de l'Enseignement Supérieur, de la Recherche et de l'Innovation and McGill University for funding my Ph.D. I am thankful to all my co-authors, Pierre Colombo, Francisco Messina, Federica Granese, Malik Boudiaf, Ismail Ben Ayed, Marco Romanelli, Guillaume Staerman, Nathan Noiry, and Eduardo Gomes, for our multiple discussions and their hard work. I learnt a lot from each of them. A special thank you to my dear colleagues, Ralph, Pierre, Nicolas, Mickael, Daouda, and Stephano. I do not know how I could have survived those last four years without our breaks.

  Context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Deep Neural Networks: powerful tools with loopholes . . . . 1.1.2 How to ensure no private information leakage . . . . . . . . . 1.1.3 How to increase our trust on DL-based results . . . . . . . . . 1.1.4 How to protect against malicious agents . . . . . . . . . . . . 1.1.5 Focus of this manuscript . . . . . . . . . . . . . . . . . . . . . 1.2 Interacting with Malicious Agents: Attacking and Defending . . . 1.2.1 Global Context . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.2 Notations and problem definition . . . . . . . . . . . . . . . . 1.2.3 Attacking a neural network . . . . . . . . . . . . . . . . . . . . 1.2.4 Defense 1: protecting DNNs' decision integrity against threats 1.2.5 Defense 2: ensuring DNNs' input integrity . . . . . . . . . . . 1.3 Contribution and Outlines . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 How can we use the internal structure of DNN's output space to improve its robustness ? . . . . . . . . . . . . . . . . . . . . 1.3.2 How can we craft an efficient and effective detection method based on simple tools? . . . . . . . . . . . . . . . . . . . . . . . 1.3.3 Outlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 List of Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.1 Journals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.2 Conferences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4.3 Preprints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Neural Networks: powerful tools with loopholes Deep Neural Networks have allowed impressive breakthroughs in different fields, such as Natural Language Processing (NLP) or Computer Vision (CV). Deep Learning (DL)-based systems are more and more used in many applicative domains, as in medicine

  scoring function to detect adversarial attacks is still an open question.

Figure 2 . 1 .

 21 Figure 2.1. The output of each layer is then the aggregation of the output of each neuron, i.e., f l θ(x) = [ f l ,1 θ (x), f l ,2 θ (x), . . . , f

Figure 2

 2 Figure 2.2: CNN structure. More specifically, LeNet's [LECUN and collab., 1989] structure. Source

Figure 2 . 4 :

 24 Figure 2.4: Representation of the Vision Transformer's structure. Source: [DOSOVITSKIY and collab., 2020], Fig.1

  tionally expensive. The first successfully trained Vision Transformer [DOSOVITSKIY and collab., 2020] considered a patch of the input as the basic unit instead of the pixel. They separated their images into 16x16 patches to feed to the model. An overview of the structure of a ViT is presented in Figure 2.4. Overall, ViTs work as follows. After patching the input image, a linear embedding of each patch is performed, and a position embedding is added. The transformed patches and positions are later fed to the transformer encoder, which is composed of L transformers cells (left side of Figure 2.4 for a detailed presentation of the structure of a transformer cell), which extract the features of the embeddings. After the encoder, as in MLP or CNNs, fully-connected layers are then used to extract useful classification information and output the soft-probabilities q θ (•|x).

Figure 2 . 5 :

 25 Figure 2.5: Example of the effect of an adversarial sample on ImageNet using a GoogLeNet as the targeted model. Source: [GOODFELLOW and collab., 2015], Fig.1

Figure 2 . 6 :

 26 Figure 2.6: DF method for a linear binary classifier. Source: [MOOSAVI-DEZFOOLI and collab., 2016], Fig.2

  Training. Finally, we want to mention Adversarial Training. Adversarial training (AT) was first introduced by GOODFELLOW and collab. [2015]. It is based on augmenting the original data with attacked samples. It is today one of the only defenses that have been proven efficient against adversarial attacks [ATHALYE

  [2018] improved the Adversarial Training framework by modifying the attack to craft adversarial samples. They proposed to no longer use the single-step algorithm FGSM but to use its iterative version, i.e., the PGD algorithm.An essential choice of the defense mechanism is the robust loss used to attack and defend the network. While GOODFELLOW and collab.[2015] proposed to use a trade-off between the original cross entropy and its adversarial version, i.e., when the input is a perturbed version of the clean one,MADRY and collab. [2018] decided to only use the adversarial cross-entropy, i.e., the cross-entropy between the original label and the corrupted sample.Considering all the made choices, it is possible to write the optimization problem as the authors in[MADRY and collab., 2018] solved:

  .25) However, it was shown that one way to improve adversarial training is through the choice of this loss. Improvement based on the loss. TRADES. ZHANG and collab. [2019] introduces a robustness regularizer based on the Kullback-Leibler divergence. They defined a new risk optimizing a trade-off between natural and adversarial performances, controlled through a hyperparameter λ. The resulting risk is the addition of the natural cross-entropy and the Kullback-CHAPTER 2. PRELIMINARIES Leibler (KL) divergence between natural and adversarial probability distributions:

  based on additional data.Whether it is Madry's version of Adversarial Training, TRADES, or MART, they all have been improved in different ways. One of the most successful improvements uses unlabeled data to enhance generalization[ALAYRAC and collab., 2019; CARMON and collab., 2019]. The theoretical argument of this data augmentation is that ad-CHAPTER 2. PRELIMINARIES versarial generalization requires more examples than natural generalization, i.e., it requires a larger dataset at training time.To improve robustness,CARMON and collab. [2019] proposes the use of additional data when training on CIFAR-10. Specifically, they use 500k additional images from 80M-TI 1 . Those images have been selected such that their L 2 -distance to images from CIFAR-10 are below a fixed threshold. They then use a classifier to predict the pseudo-label of each new image before adding them to the dataset. Finally, they use this augmented dataset to perform adversarial training using the TRADES loss. ALAYRAC and collab. [2019] propose two methods. The first one is similar to Carmon et al.'s method, but they consider the sum of natural and Adversarial Cross-Entropy weighted by a hyperparameter λ as the loss. The other method does not use pseudo labels. For unlabeled data, only the smoothness of the loss with respect to the adversarial examples is considered. In other words, they consider a natural loss computed only on natural -labeled -examples from the original CIFAR dataset, and for all examples -both labeled and unlabeled -the considered loss is the Kullback-Leibler divergence between natural and adversarial probability distributions, i.e., KL(q θ (•|x)∥q θ (•|x ′ )). The total sum is a weighted sum of the natural and the adversarial losses. Other improvements Other types of improvement have been studied in recent years. Pretraining [HENDRYCKS and collab., 2019], early stopping [RICE and collab., 2020], curriculum learning [ATZMON and collab., 2019], adaptative models [HUANG and collab., 2020], or additional perturbations on the model weights [WU and collab., 2020] are a few examples of these improvements.

  Dimensionality method (LID) [MA and collab., 2018] is based on the intuition that adversarial examples lie outside of the clean data manifold. By computing the Local Intrinsic Dimensionality, it is possible to check whether the new point is close to the original data manifold

Figure 2 . 8 :

 28 Figure 2.8: Natural Scene Statistics extraction on a natural sample, and various adversarial ones. Source: [KHERCHOUCHE and collab., 2020], Fig.1a

Feature

  Squeezing (FS) [XU and collab., 2018]. The key idea of the Feature Squeezing (FS) method is to compare the model's prediction of the original sample with its prediction of the sample after multiple squeezing. The further away they are, the more likely the input is adversarial. In practice, four versions of the input are needed: the original input, a low-precision version, a median-filtered version, and a denoising-filtered version. One inference on the model is required for each of the four inputs. Later, the maximal L 1 difference between the original prediction and each of the other three is picked. FS is, therefore, parameter-free and does not require training. MagNet [MENG and CHEN, 2017]. MagNet is based on the idea that adversarial examples do not lie on the data manifold. It uses a detector and a reformer to detect adversarial examples. The detector, an autoencoder trained to reconstruct rightfully natural samples, will try to reject examples that are far from the natural manifold.

Figure 2 . 9 :

 29 Figure 2.9: Our current grid' structure needs to change to fulfill UN goals to change our world. Source: [HEINRICH, 2018], p.33.

  Denoting by P X the distribution of any r.v. X taking its values in R d , we have: ∀x ∈ R d , D(Ax + b, P AX+b ) = D(x, P X ), for any d × d nonsingular matrix A with real entries and any vector b in R d . (D 2 ) (MAXIMALITY AT CENTER) For any P ∈ P (R d ) that has a symmetry center x * (in a sense to be specified), the depth function D(., P) takes its maximum value (MONOTONICITY RELATIVE TO DEEPEST POINT ) For any P ∈ P (R d ) with deepest point x * , the depth at any point x in R d decreases as one moves away from x * along any ray passing through it: ∀ξ ∈ [0, 1], D(x * , P) ≥ D(x * + ξ(xx P ), P). (D 4 ) ( VANISHING AT INFINITY ) For any P ∈ P (R d ), the depth function D vanishes at infinity: D(x, P) → 0, as ||x|| → ∞.

  This part is dedicated to our proposed answer to the first question asked in Chapter 1: How can we use the internal structure of DNN's output space to improve its robustness ? This part is split into three chapters consisting of three different contributions.

  which is based on adversarial training, i.e., learning with an augmented training set containing adversarial examples[GOODFELLOW and collab., 2015].

1

 1 Information geometry and statistical manifoldStatistics on manifolds and information geometry are two different ways in which differential geometry meets statistics. A statistical manifold can be defined as a parameterized family of probability distributions (or density functions) of interest.

  (a) θ = [-6.4290, 25.7487] ⊺ . (b) θ = [-0.9364, 0.3509] ⊺ .

Figure 3 . 2 :

 32 Figure 3.2: Visualization of statistical manifold C defined by the model q θ (y|x) = 1/[1 + exp(-y θ ⊺ x)] with different values of θ: (a) Parameters minimizing the natural misclassification error probability P e , (b) Parameters minimizing the adversarial misclassification error probability P ′ e . ©2022 IEEE.

Figure 3

 3 Figure 3.3: FRD between the distributions q θ (•|x) and q θ (•|x ′ ) as a function of δ using the logistic model with different values of θ: (a) Parameters minimizing the natural misclassification error probability P e , (b) Parameters minimizing the adversarial misclassification error probability P ′ e . ©2022 IEEE.

  CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION (a) λ = 0. (b) λ = 3.8957. (c) λ = 98.1730.

Figure 3

 3 Figure 3.4: Visualization of statistical manifold C defined by the model q θ (y|x) = 1/[1 + exp(-y θ ⊺ x)] when minimizing the FIRE risk function for different values of λ: (a) No adversarial FRD regularization, (b) Medium adversarial FRD regularization, (c) High adversarial FRD regularization. ©2022 IEEE.

  Figure 3.4c, setting λ to a very high value causes the statistical manifold to become extremely flat. This means that the model is basically independent of the input, and the classification performance will be poor. Notice the similarities between Figure 3.2 and Figure 3.4.

Figure 3 . 5 :

 35 Figure 3.5: Comparison between FRD and Euclidean distance. ©2022 IEEE.

  .14) be a standard softmax output, where h : X × Θ → R M is a parametric representation function and z y denotes the y-th component of the vector z. The FRD for this model can also be obtained in closed-form as summarized below. The proof is given in Subsection 3.6.3.

Figure 3 . 6 :

 36 Figure 3.6: Plot of all the possible points (1 -P e (θ), 1 -P ′ e (θ)) for the Gaussian model with ε = 0.1, µ = [-0.0218; 0.0425] and Σ = [0.0212, 0.0036; 0.0036, 0.0042] shown in blue. In red, we show the Pareto-optimal points (Figure 3.6a). In black, we show the solutions obtained by minimizing the risk L TRADES (θ) in Equation 3.5 ( Figure 3.6b), and the risk L FIRE (θ) in Equation 3.7 (Figure 3.6c).©2022 IEEE.

  .21) where Φ denotes the cumulative distribution function of the standard normal random variable. The following result provides lower and upper bounds for P ′ e (θ) in terms of ε and the eigenvalues of Σ. Notice that the bounds get sharper as ε or the spread of Σ decreases and are tight if Σ = σ 2 I. Proposition 2 (Accuracy-robustness trade-offs). The adversarial misclassification probability P ′ e (θ) satisfies the inequalities:

  are also shown in Figure3.6a. In particular, this set contains the Maximum

Architectures:

  In order to provide fair comparisons, we use standard model architectures. For the MNIST simulations, we use the 7-layer CNN as in ZHANG and collab.[2019]. For CIFAR-10 and CIFAR-100, we use a WideResNet (WRN) with 34 layers, and a widen factor of 10 (shortened as WRN-34-10) as in MADRY and collab.[2018];WU and collab. [2020];ZHANG and collab. [2019]. For the simulations with additional data, we use a WRN-28-10 as inCARMON and collab. [2019].Training procedure: For all standard experiments (without additional data), we use a Stochastic Gradient Descent (SGD) optimizer with a momentum of 0.9, a weight decay of 5 • 10 -4 and Nesterov momentum. We train our models on 100 epochs with a batch size equal to 256. The initial learning rate is set to 0.01 for MNIST and 0.1 for CIFAR-10 and CIFAR-100. FollowingZHANG and collab. [2019], the learning rate is divided by 10 at epochs 75 and 90. For our experiments with additional data, we follow the protocol introduced by CARMON and collab. [2019], using a cosine decay[LOSHCHILOV and HUTTER, 2017], and training on 200 epochs.Generation of adversarial samples:For all experiments, we use PGD[MADRY and collab., 2018] to generate the adversarial examples during training. The loss which is maximized during the PGD algorithm is the Fisher-Rao distance (FRD) for our experiments, and the Kullback-Leibler divergence for the TRADES method. For MNIST, we use 40 steps and 10 steps for the rest. The step size is set to 0.01 for MNIST and 0.007 for the others. The maximal distortion in l ∞ -norm ε allowed is 0.031 for CIFAR-10 and CIFAR-100, and 0.3 for MNIST. This setting is used in most methods, such asCARMON and collab. [2019];GOODFELLOW and collab. [2015].Additional data: To improve performance,CARMON and collab. [2019] propose the use of additional data when training on CIFAR-10. Specifically,CARMON and collab. [2019] use 500k additional images from 80M-TI 6 . Those images have been selected such that their l 2 -distance to images from CIFAR-10 are below a threshold.Hyperparameters:The Rao regularizer used to improve the robustness of neural networks introduces a hyperparameter λ to balance natural accuracy and adversarial robustness. We select λ = 12 from the CIFAR-10 simulations and use this value for all the other datasets. Further study on the effect of λ is provided in Section 3.5.2.Test metrics: First, we provide the accuracy of the model on clean samples after adversarial training (Natural). Second, we test our models using the recently introduced AutoAttack [CROCE and HEIN, 2020], keeping the same hyperparameters as the ones used in the original paper and code 7 . AutoAttack tests the model under a comprehensive series of attacks and provides a more reliable assessment of robustness than the traditionally used PGD-based evaluation. Given that we care

Figure 3 . 7 :

 37 Figure 3.7: Influence of the hyperparameter λ on the natural and adversarial accuracies for FIRE regularizer on CIFAR-10. ©2022 IEEE.

  .36) Notice that the parameter vector z belongs to the positive portion of a sphere of radius 2 and centered at the origin in R M . As a consequence, the FIM follows by

  .45) CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION Using this fact together with the monotonicity of Φ in Equation 3.43, we obtain the desired result.

  This work was supported by the Natural Sciences and Engineering Research Council of Canada and McGill University in the framework of the NSERC/Hydro-Quebec Industrial Research Chair in Interactive Information Infrastructure for the Power Grid (IRCPJ406021-14).

  [2011],the authors present various methods to generate such corrupted samples to bypass the bad data detector under the DC assumption. Let us consider Y A to be the attacked sample. We consider thatY A = Y + A, (4.3) CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS

CHAPTER 4 .

 4 ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS statistical manifold. Consider an attack mechanism p(Y A |Y).

Figure 4 . 1 :

 41 Figure 4.1: Error between natural and noisy samples using the state estimator-aware attack. ©2022 IEEE.

Algorithm 1

 1 Review of the PGD Algorithm INPUT: Define the step δ and the number of iterations n INPUT: Y

  and sgn is the sign function. end for OUTPUT: Y A Given that the parameters φ of a state estimator evolve during training, it is theoretically possible to attack the network at each training step. Therefore, the defender should generate attacked samples at each training step and then defend against them to ensure robustness. Two different steps are needed to train a robust state estimator: • First, corrupted samples are generated using the proposed attack method in Equation 4.10. Since the underlying maximization problem Equation 4.10 is untractable, we generate noisy samples using the Projected Gradient Descent method (PGD) [MADRY and collab., 2018] (for further details, see Algorithm 1).

Figure 4 . 2 :

 42 Figure 4.2: Errors between natural and corrupted performances as a function of the percentage of perturbed meters, for λ = 10 and without defense. ©2022 IEEE.

Figure 4

 4 Figure 4.3: Estimated-angles mean errors and portions of detected samples for a robust estimator as a function of the β parameter. The natural errors are in plain lines, while the attacked ones are in dashes. ©2022 IEEE.

  3.3, and report the attack-induced mean angle error and the percentage of detected samples in Figure 4.1. Please observe that the impact of the λ hyperparameter (Equation 4.10) that controls the trade-off between the power of the attack and its detection is apparent in Figure 4.1. The bigger λ is, CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS the smaller attack-induced mean error and the number of detected samples are.

First, we need

  to study the influence of the hyperparameter β that controls the tradeoff between natural and attacked performances. The mean angle errors and the percentage of noisy detected samples are reported in Figure 4.3. The trade-off between clean and attacked performances is visible. While increasing the β hyperparameter provides better performances on attacked samples, it also worsens the clean ones. For β ≥ 1, the VAE starts to give more importance to estimating the attacked and clean natural samples in the same way and no longer tries to predict the clean measurements correctly. The best trade-off between clean and attacked performances was found for β = 0.1.CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS

CHAPTER 4 .

 4 ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS AC model setup.

CHAPTER 5 .

 5 ROBUST STATE ESTIMATION AGAINST ADVERSARIAL NOISE y * a ∼ q * (y a |y) = arg max q(y a |y): y a ∼q(y a |y) E[l (y, y a ; φ)], (5.15)

Figure 5 . 1 :

 51 Figure 5.1: Averaged angle and tension attack-induced errors, and percentage of detected noisy samples under (a) the deterministic attack, (b) the random attack with no defense, as a function of λ, the attack hyperparameter.

1

 1 Problem formulation . . . . . . . . . . . . . . . . . . . . . . . . 165 6.2.2 Supervised detection methods . . . . . . . . . . . . . . . . . . 166 6.2.3 A brief review of attack mechanisms . . . . . . . . . . . . . . . 167 6.3 A Depth-Based Detector . . . . . . . . . . . . . . . . . . . . . . . . . 168 6.3.1 Background on data-depth . . . . . . . . . . . . . . . . . . . . 168 6.3.2 Our Depth-Based Detector . . . . . . . . . . . . . . . . . . . . 169 6.3.3 Comparison to state-of-the-art detection methods. . . . . . . 171 ATTACK DETECTION

Figure 6 . 1 :

 61 Figure 6.1: Average performances of our method (i.e., HAMPER) in an attack-aware and single detector setting, along with the performances of state-of-the-art detection mechanisms (i.e., NSS, LID, KD-BU), on three classically considered datasets (i.e., SVHN, CIFAR10 and CIFAR100). Below the dataset names is the accuracy of their underlying classifiers. In addition to outperforming other methods on all three considered datasets, our method, contrary to the others, does not lose performances as the classifier's accuracy decreases. derived from monitoring clean and adversarial characteristics (e.g., model uncertainty[FEINMAN and collab., 2017], natural scene statistics[KHERCHOUCHE and collab., 2020]). In the unsupervised detection setting, the detector does not rely on the prior knowledge of the attacks, and it only learns from the clean data at training time. Different techniques are used to extract the meaningful features (e.g., feature squeezing[LIANG and collab., 2021; XU and collab., 2018], denoiser approach[MENG and CHEN, 2017], network invariant[MA and collab., 2019], sensitivity to noise [HU

2.

  Through an analysis of the classifier's behavior under threat, we show how to leverage the halfspace-mass depth to build an anomaly score. To that end, we introduce HAMPER, a simple supervised method to detect adversarial examples given ATTACK DETECTION a trained model. Given an input sample, HAMPER relies on a linear combination of the halfspace-mass depth score. These depth scores are computed w.r.t. a reference distribution corresponding to the training data conditioned per-class and per-layer.

ni

  =1 and S ℓ c = z ℓ,i ∈ S ℓ : y i = c , the ℓ-th and the ℓ-th classconditionally representations of the training dataset, respectively. Our approach aims to construct a score function s : R d → [0, 1] providing a confidence level to a new observation x indicating its degree of abnormality w.r.t. to the training dataset. HAMPER leverages appealing properties of the HM depth detailed in Subsection 6.3.1 and can be summarized into two distinct steps. The function s is built by first constructing |Λ| × C intermediate scoring functions s ℓ,c : R d ℓ → [0, 1] designed for each considered layer and each class. The map s ℓ,c assigns a value to any element of the ATTACK DETECTION embedded space of the ℓ-th layer represented somehow its 'distance' to the class c of the mapped training set. Thereafter, an aggregation is performed between scores using a small validation dataset composed of both 'normal' and 'adversarial' samples. These two parts of the proposed approach are detailed below. Intermediate score functions. Given a new observed image x ∈ R d mapped into |Λ| representations {z ℓ } ℓ∈Λ such that z ℓ = f ℓ θ (x; D), we propose to use the HM depth as intermediate scoring functions s ℓ,c . Precisely, we compute D HM (z ℓ , S ℓ c ), for each considered layer ℓ and each class c, i.e., the HM depth between z ℓ = f ℓ θ (x; D) and the classconditionally probability distribution of the training dataset S ℓ c = { f ℓ θ (x i ; D) : x i ∈ S c }. Following the approximation algorithm of the HM introduced in CHEN and collab.

  .7)where the weights α ℓ,c are obtained through the training of a linear regressor in a supervised manner. It is worth noting that the anomaly score s from Equation6.7 results from both class and layer dependent linear combination. The class dependency of s is motivated by (1) the per class behavior classifier which is highlighted in Subsection 6.4.2 and by (2) the monotonicity relative to deepest point property of the halfspace-mass depth and depth functions in general (see e.g. property (D 3 ) in Section B.2 of the Appendix orSTAERMAN [2022];ZUO and SERFLING [2000]). The layer dependency of s is motivated by the per layer behavior classifier which is displayed inSubsection 6.4.3.

  [2021] but the codes are not publically available.Local Intrinsic Dimensionality (LID).LID is based on the intuition that adversarial examples lie outside of the clean data manifold. By computing the Local Intrinsic Dimensionality, it is possible to check whether the new point is close to the original data manifold. Following this idea, three version of each natural samples are used.The clean one, a noisy version of it, and an attacked one. The LID approximate for each of those points to the clean distribution will be computed at the output of each layers. Those variables will then be used to train a detector that will distinguish between adversarial samples, and normal ones (normal samples are either clean or noisy samples). Each of the strategy to craft adversarial samples can have very different LID characteristics, a detector per type of threats is therefore necessary. LID therefore lies in the attack-aware category.Kernel Density and Bayesian Uncertainty (KD-BU). KD-BU also relies on the idea that adversarial samples lie off the original data manifold. To detect adversarial samples, they first perform a kernel density estimate at the last hidden layer level to detect samples that are far from the original manifold. Then, a bayesian uncertainty estimate is computed to detect when points lie in low-confidence regions of the input space. Both of those characteristics are later used to train a detector that distinguish between natural and adversarial examples. Once again, the kernel density estimates and the bayesian uncertainty values for different types of attacks can differ a lot, therefore this method have been created to be attack-aware.Natural Scene Statistics (NSS). NSS relies on the extraction of the natural scenestatistics at the image level. Natural scene statistics are statistics that will be very different for natural and attacked images. Indeed, for clean image, applying the natural scene statistics will output an image with meaning, however, for attacked samples, the resulting image will have no meaning. The Natural Scene Statistics extraction is then used to train a detector to distinguish between natural and attacked samples. To overcome the need to have a specific detector per attack, the authors of NSS decided to train their detector using the natural scene statistics of various attacks. It therefore lies in the blind-to-attack category.

Figure 6 . 3 :

 63 Figure 6.3: Calibrating the maximal allowed perturbation ε on CIFAR100. Accuracy on adversarial examples created using PGD 1 for the SVHN, CIFAR10 and CIFAR100 classifiers. On CIFAR100, to ensure high successes of the attacks, one must allow the attacker to have larger values of ε, compared to the CIFAR10 and SVHN ones.

Figure 6 . 4 :

 64 Figure 6.4: Per layer behavior analysis. Evolution of ᾱℓ = 1 C c α ℓ,c the average and the standard deviation over the classes of the regressor weight as a function of the layer, for different value of the regularization parameter.

  represents the relation between True Positive Rate (TPR) -i.e. adversarial examples detected as adversarial -and False Positive Rate (FPR) -i.e. natural samples detected as adversarial. As can be checked from elementary computations the AUROC↑ corresponds to the probability that an natural example has higher score than an adversary sample. FPR at 95% TPR↓ or FPR↓ 95% (lower is better): represents the percentage of natural examples detected as adversarial when 95% of the adversarial examples are detected. The FPR↓ 95% is of high interest in practical applications.

Subsection 6 .Figure 6 . 5 :

 665 Figure 6.5: Accuracies of the detectors per predicted classifier class. For visualization reasons, we restrict the plot to the 10-most/least populated classes for CIFAR100.

.

  HAMPER performances are not affected either by the class nor by the dataset and they consistently outperform the competitors. However, the competitor's performance are class-dependant: they tend to better distinguish between natural and attacked samples for classes with the highest number of adversarial examples.

Figure 6 . 6 :

 66 Figure 6.6: FPR↓ 95% as a function of the AUROC↑ for all five considered methods, (a) on SVHN, (b) on CIFAR10, and (c) on CIFAR100.

Analysis.

  All methods have quite comparable training time. However, LID, due to the extraction of the LID parameters to all layers, takes a lot more time to test than the others.

7. 1 Introduction

 1 Recent years have seen a rapid development of Deep Neural Networks (DNNs), which have led to a significant improvement over previous state-of-the-art methods (SOTA) in numerous decision-making tasks. However, together with this growth, concerns have been raised about the potential failures of deep learning systems, which limit their large-scale adoption [ALVES and collab., 2018; JOHNSON, 2018; SUBBASWAMY and CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES SARIA, 2020]. In Computer Vision, a particular source of concern is the existence of adversarial attacks [SZEGEDY and collab., 2014], which are samples created by adding to the original (clean) image a well-designed additive perturbation, imperceptible to human eyes, with the goal of fooling a given classifier. The vulnerability of DNNs to such kinds of attacks limits their deployment in safety-critical systems as in aviation safety management [ALI and collab., 2020], health monitoring systems [LEIBIG

Notations.

  Let us consider the classical supervised learning problem where x ∈ X ⊆ R d denotes the input sample in the space X , and y ∈ Y = {1, . . . , C} denotes its associated label. The unknown data distribution is denoted by p XY . The training DETECT ADVERSARIAL IMAGES dataset D = {(x i , y i )} n i =1 is defined as n ≥ 1 independent identically distributed (i.i.d.) realizations of p XY . Consider D c = {(x i , y i ) ∈ D : y i = c}, the training data for a given class c ∈ Y . We define the empirical training distribution for the class c at layer ℓ as p

  notion of data depth goes back to John Tukey in 1975, who introduced the halfspace depth[TUKEY, 1975]. Data depth functions are useful nonparametric statistics allowing to rank elements of a multivariate space R d w.r.t. a probability distribution (or a dataset). Given a random variable Z which follows the distribution p Z , a data depth can be defined as:D : R d × P (R d ) -→ [0, 1] , (z, p Z ) -→ D(z, p Z ).(7.4)The higher the value of the depth function, the deeper the element is in the reference distribution. Various data depths have been introduced over the year CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES (see Chapter 2 of STAERMAN [2022] for a survey), including halfspace depth [TUKEY,

Figure 7

 7 Figure 7.5: AUROC↑ as a function of FPR↓ 90% for APPROVED, FS, and MagNet on all considered datasets.

  CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES of ResNets, for the transformers, it was empirically shown to be concentrated in the logit layer. This motivated us to build APPROVED on top of this logit layer by computing a similarity score between an input sample and the training distribution based on the statistical notion of data depth. We chose to use the Integrated Rank-Weighted depth, which lends itself to fast inference computations and is non-differentiable, making it harder for gradient-based adversarial methods to craft malicious samples.

  Fields of interest include sequence generation task COLOMBO and collab. [2021c, 2022d]; COLOMBO* and collab. [2019]; COLOMBO and collab. [2021e]; JALALZAI* and collab. [2020], translation models COLOMBO and collab. [2021a]; GUER-REIRO and collab. [2023], multimodal emotion analysis COLOMBO and collab. [2022d]; GARCIA and collab. [2019], and sentence similarity CHHUN and collab.

  However, other metrics could be well suited for this taskCOLOMBO [2021a];COLOMBO and collab. [2022b];PICHLER and collab. [2022]. Information measures compute the resemblance between two probability distributions and are gaining attention in the deep learning community. It would be interesting to find out if such measures can help protect our systems.• Combining detection methods. We know that the different state-of-the-art methods and our methods capture different characteristics of clean and adversarial inputs, as they tend to behave differently depending on the type of threats they face. It would be interesting to find a simple method to combine the scores provided by different methods to leverage the different characteristics of natural and adversarial input instead of focusing on only one. CHAPTER 8. DISCUSSION OF FINDINGS, LIMITATIONS, POTENTIAL FUTURE WORKS AND GLOBAL SUMMARY • Real time detector. To deploy detection methods in real-world scenarios, we believe they need to meet three separate requirements. (R1) Black-box scenario. Systems already deployed in production are generally opaque to the end user, who only has access to the softmax predictions of the networks. (R2) Low resources / computation time. In many real-world applications, AI systems are making real-time predictions at a high frequency (e.g. face recognition for airport security). As a result, any relevant detector should have a low inference time and require low computation resources. (R3) No oracle on the nature of the attackers. Any relevant detector should be unsupervized, meaning it should not require any training phase with access to attack examples. Indeed, the landscape of existing attackers is moving fast, making the availability of adversarial examples not realistic in practice. Existing detection methods tend to fail at meeting at least one of those requirements. Supervised methods -not satisfying (R3). Supervised methods usually consist in training simple machine learning algorithms, such as SVMs or logistic regressions, to discriminate adversarial examples from natural ones, using examples from both classes. The features used for training these machine

  Unsupersived methods -satisfying (R3). Unsupervised methods only rely on clean samples to build a detector, making them very attractive for real-life applications. Let us explore existing works in light of requirements (R1) and (R2). Some detectors require access to intermediate layers representation [AL-DAHDOOH and collab., 2021b; MA and collab., 2019; SOTGIU and collab., 2020; WORKS AND GLOBAL SUMMARY ZHENG and HONG, 2018], which makes them unsuitable for use in the context of (R1). Two methods satisfy (R1) but are arguably less effective regarding (R2): the Feature Squeezing (FS) of XU and collab. [2018] and the Mag-Net detector of MENG and CHEN [2017] which relies on a denoising autoencoder. Let us also mention JTLA [RAGHURAM and collab., 2021], a refinement of FS which unfortunately does not satisfy (R1).

  detection methods. As adversarial attack detection can be considered an extreme case of the out-of-distribution (OOD) detection problem, it would be interesting to extend ODD methods to adversarial detection.In particular, a line of OOD methods is based on extracting relevant information from the softmax probabilities, making them very attractive as they meet all the requirements. This line of work has been launched by the seminal work ofDARRIN and collab. [2023a,b]; GOMES and collab.; HENDRYCKS and GIMPEL[2016], who proposed to focus on the Maximum Softmax Probability(MSP) to discriminate between in-and out-of-distribution samples. The underlying idea of MSP is that the more spiky the probabilities, the more confident the network is and, therefore, the cleaner the input. Let us also mention the DOCTOR detector, recently introduced by GRANESE and collab.[2021], which computes the Gini coefficient of the softmax probabilities. Both methods satisfy the three requirements (R1) -(R2) -(R3), but fail at detecting some attacks.

  15, we readily obtain the following relation between the Hellinger distance and the Fisher-Rao distance.

Figure 3 .Algorithm 5

 35 Figure 3.6 but using the Hellinger distance as a robust regulatizer. Then, we perform comparison between Hellinger and FRD on real datasets (similar to the simulations given in Section 3.5.2).

  D HM (z ℓ , S ℓ c ) = HM/K. Algorithm 7 HAMPER INPUT: : test representations {z ℓ } L-1 ℓ=1 ; sample representations S ℓ c = {z ℓ,i ∈ S ℓ : y i = c}. INPUT: Number of closed halfspaces K; sub-sample size n s ; hyperparameter λ. 1: for ℓ = 1, . . . , L -

  a linear regression to find weights α ℓ,c such that s(x) = L-1 ℓ=1 C c=1 α ℓ,c D HM (z ℓ , S ℓ c ). OUTPUT: the score function s. B.3 Detailed Results on Perfect Knowledge about the attacker Table B.1: Performances on the three considered datasets SVHN, CIFAR10, and CIFAR100-of the HAMPER AA detector together with the results of the state-of-the-art detection methods: LID, and KD-BU, on multiple threat scenarios with multiple maximal perturbations ε. The best results among the detectors are shown in bold. The results are presented as: AUROC↑ ±FPR ↓ 95% %. * stipulates the non-gradient based attacks.

  Dans un second temps, nous adaptons cette méthode à une autre application critique: les réseaux intelligents (Smart Grids), qui, en raison de divers besoins de la surveillance et de service, reposent sur des composants cybernétiques, tels qu'un estimateur d'état, ce qui les rend sensibles aux attaques. Nous construisons donc des estimateurs d'état robustes en utilisant des autoencodeurs variationnels et l'extension de notre méthode proposée au cas de la régression. Que ce soit dans le cas des hypothèses linéraire (appelé modèle DC) ou sans hypothèses (appelé modèle AC), nous montrons qu'il est possible de créer des estimateurs d'états robustes aux attaques ayant une connaissance parfaite du réseau électrique à attaquer, surpassant significativement les méthodes de protection présentées précédemment. La deuxième méthode sur laquelle nous nous concentrons et qui vise à protéger les modèles basés sur l'apprentissage profond est la détection d'échantillons adverses. Inspiré par le concept des canaux de réjection, il est possible d'augmenter la fiabilité des décisions prises par les réseaux neuronaux profonds en ajoutant un détecteur au modèle initial. De multiples méthodes de détection sont disponibles aujourd'hui, mais elles reposent souvent sur un entraînement lourd et des heuristiques ad-hoc. Dans notre travail, nous utilisons des outils statistiques simples appelés les profondeurs de données (data-depth) pour construire des méthodes de détection efficaces. Les profondeurs de données nous fournissent un ordonnancement des points vis-à-vis du "centre" d'un distribution référence. En d'autre termes, calculer les profondeurs de données permet de déterminer à quel point un point est "profond" dans une distribution référence, i.e., est similaire à la référence. Nous developpons deux méthodes distinctes de détections, dans deux cas spécifiques: le cas dit "supervisé", c'est-à-dire que les attaques sont fournies pendant l'entraînement du détecteur, et le cas "non supervisées", c'est-à-dire que l'entraînement ne peut s'appuyer que sur des échantillons propres. Dans le cas supervisé, nous présentons HAMPER, méthode basée sur l'utilisation de la profondeur de masse du demi-espace (Halfspace-Mass depth) et la connaissance des attaques qui seront perpétrées afin de construire un détecteur d'attaques adverses. Nous présentons dans ce travail deux scénarios distincts: le scénario connaissant parfaitement l'attaque perpétrée, et le scénario aveugle aux attaques. Dans le deux cas, notre méthode permet de mieux détecter les exemples adverses, tout en necessitant un temps et des ressources similaires aux autres attaques. De plus, dans le cas d'attaques ayant un connaissance parfaite du modèle à attaquer et de la défense choisie, les performances de notre détecteur surpassent les performances des méthodes état-de-l'art. Dans le cas non-supervisé, nous présentons APPROVED, méthode basée uniquement sur l'utlisation de la profondeur intégrée pondérée par le rang (Integrated Rank-Weighted depth) appliqué à la sortie d'un classifieur profond (c'est-à-dire aux

Mass Depth-Based Detector for Adversarial Attack Detection.

	1.4.1 Journals
	1.	Adversarial Robustness via Fisher-Rao Regularization. M. Picot, F. Messina,
		M. Boudiaf, F. Labeau, I. BenAyed, P. Piantanida. IEEE Transactions on Pattern
		Analysis and Machine Intelligence (TPAMI).
	2.	A Halfspace-

M. Picot * , F. Granese * , G. Staerman, M. Romanelli, F. Messina, P. Piantanida, P.

Colombo. Transactions on Machine Learning Research (TMLR).

the Uncertainty of a Deep Neural Network Enhances its Adver- sarial Robustness

	2. Modelling 3. MEAD: A

1. Robust Autoencoder-based State Estimation in Power Systems. M. Picot, F. Messina, F. Labeau, P. Piantanida. 2022 IEEE Power and Energy Society Innovative Smart Grid Technologies Conference (ISGT). . M. Picot, P. Piantanida, F. Messina, F. Labeau. 2019 Groupe de Recherche et d'Etudes de Traitement du Signal et des Images Conference (GRETSI).

Multi-Armed Approach for Evaluation of Adversarial Examples Detectors

. F. Granese * M. Picot * , M. Romanelli, F. Messina, P. Piantanida. 2022 European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML PKDD).

Contents 2.1 Deep Learning Background . . . . . . . . . . . . . . . . . . . . . . . 48 2.2 Attacking Neural Networks . . . . . . . . . . . . . . . . . . . . . . . 53

  Entropy, vol. 19, n o 12, p. 656. 27 ILYAS, A., S. SANTURKAR, D. TSIPRAS, L. ENGSTROM, B. TRAN and A. MADRY. 2019,

	CHAPTER 1. INTRODUCTION
	CORBIÈRE, C., N. THOME, A. BAR-HEN, M. CORD and P. PÉREZ. 2019, «Addressing
	failure prediction by learning model confidence», Advances in Neural Information 2019, «Detecting out-of-distribution samples using low-order deep features statis-
	Processing Systems, vol. 32. 28 Neural Information Processing Systems, vol. 34, p. 5669-5681. 28 on Learning Representations. 28, 30, 31, 32 tics», . 28 preprint arXiv:2205.01094. 28
	CROCE, F. and M. HEIN. 2020, «Reliable evaluation of adversarial robustness with GROSSE, K., P. MANOHARAN, N. PAPERNOT, M. BACKES and P. MCDANIEL. 2017, «On «Adversarial examples are not bugs, they are features», Advances in neural infor-MENG, D. and H. CHEN. 2017, «Magnet: A two-pronged defense against adversarial RAGHURAM, J., V. CHANDRASEKARAN, S. JHA and S. BANERJEE. 2021, «A general XU, W., D. EVANS and Y. QI. 2018, «Feature squeezing: Detecting adversarial examples
	an ensemble of diverse parameter-free attacks», in International Conference on Machine Learning, PMLR, p. 2206-2216. 29, 31 the (statistical) detection of adversarial examples», arXiv preprint arXiv:1702.06280. 33 mation processing systems, vol. 32. 28 2020, «Contextualized perturbation for textual adversarial attack», arXiv preprint arXiv:2009.07502. 30 examples», in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, édité par B. M. Thuraisingham, D. Evans, T. Malkin and framework for detecting anomalous inputs to dnn classifiers», in International Conference on Machine Learning, PMLR, p. 8764-8775. 34 in deep neural networks», in 25th Annual Network and Distributed System Security Symposium, The Internet Society. 33 Chapter 2
	DAXBERGER, E., A. KRISTIADI, A. IMMER, R. ESCHENHAGEN, M. BAUER and P. HEN-NIG. 2021, «Laplace redux-effortless bayesian deep learning», Advances in Neural mitigate the problem», 2019 IEEE/CVF Conference on Computer Vision and Pattern computer vision, p. 5764-5772. 33 MAEILZADEH. 2020a, «A principled approach to learning stochastic representations in Advances in Neural Information Processing Systems, vol. 32, édité par H. Wal-Conference on Machine Learning, p. 1-11. 30, 32 Information Processing Systems, vol. 34, p. 20 089-20 103. 28 HEIN, M., M. ANDRIUSHCHENKO and J. BITTERWOLF. 2019, «Why relu networks yield high-confidence predictions far away from the training data and how to IQBAL, S., G. F. SIDDIQUI, A. REHMAN, L. HUSSAIN, T. SABA, U. TARIQ and A. A. ABBASI. 2021, «Prostate cancer detection using deep learning and traditional tech-niques», IEEE Access, vol. 9, p. 27 085-27 100. 26 LI, X. and F. LI. 2017, «Adversarial examples detection in deep networks with con-volutional filter statistics», in Proceedings of the IEEE international conference on D. Xu, ACM, p. 135-147. 33, 34 MIRESHGHALLAH, F., M. TARAM, A. JALALI, A. T. ELTHAKEB, D. TULLSEN and H. ES-REN, J., P. J. LIU, E. FERTIG, J. SNOEK, R. POPLIN, M. DEPRISTO, J. DILLON and B. LAKSHMINARAYANAN. 2019, «Likelihood ratios for out-of-distribution detection», ZHANG, H., Y. YU, J. JIAO, E. P. XING, L. E. GHAOUI and M. I. JORDAN. 2019, «Theo-retically principled trade-off between robustness and accuracy», in International Preliminaries
	DU, C., Z. GAO, S. YUAN, L. GAO, Z. LI, Y. ZENG, X. ZHU, J. XU, K. GAI and K.-C. LEE. 2021, «Exploration in online advertising systems with deep uncertainty-Recognition (CVPR), p. 41-50. 27 for privacy in deep neural inference», arXiv preprint arXiv:2003.12154. 27 lach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox and R. Garnett, Curran IYENGAR, R., J. P. NEAR, D. SONG, O. THAKKAR, A. THAKURTA and L. WANG. 2019, «Towards practical differentially private convex optimization», in 2019 IEEE Sym-LIANG, S., Y. LI and R. SRIKANT. 2018, «Enhancing the reliability of out-of-ZHANG, Y., W. LIU, Z. CHEN, J. WANG, Z. LIU, K. LI and H. WEI. 2021, «Out-of-Associates, Inc. 27 distribution detection with distance guarantee in deep generative models», . 27 Chapter 2 Abstract
	ZISSELMAN, E. and A. TAMAR. 2020, «Deep residual flow for out of distribution This thesis explores the use of statistical and information-geometric tools to
	detection», in The IEEE Conference on Computer Vision and Pattern Recognition improve Deep Neural Networks security when facing threats. This chapter
	(CVPR). 28 aims to provide the reader with the necessary background to apprehend the
	different contributions of this thesis. In this chapter, after a quick review of the
	Deep Learning background, we present an overview of the current methods
	to (i) attack neural networks, (ii) protect the neural network's decisions and
	(iii) detect attacks. We will present in more detail the various methods used to
	either attack our proposed defenses or to compare ourselves. We first focus on
	images, and then present the Smart Grid case. Finally, we quickly present the
	two main tools we use throught this thesis: the Fisher-Rao distance and the
	data-depths.
	p. 201-210. 27
	27 GOLDWASSER, S. and S. MICALI. 1984, «Probabilistic encryption», Journal of computer and system sciences, vol. 28, n o 2, p. 270-299. 27 2.2.1 Whitebox attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 55 doctorat, Institut polytechnique de Paris. 35 HESAMIFARD, E., H. TAKABI and M. GHASEMI. 2017, «Cryptodl: Deep neural networks 2.2.2 Blackbox attacks . . . . . . . . . . . . . . . . . . . . . . . . . . 58
	1.4.3 Preprints over encrypted data», arXiv preprint arXiv:1711.05189. 27 Digital Entertainment, vol. 16, p. 322-324. 26 SWEENEY, L. 2002, «k-anonymity: A model for protecting privacy», International GOMES, E. D. C., F. ALBERGE, P. DUHAMEL and P. PIANTANIDA. 2022, «Igeood: An information geometry approach to out-of-distribution detection», in International Conference on Learning Representations. 28 HSU, Y.-C., Y. SHEN, H. JIN and Z. KIRA. 2020, «Generalized odin: Detecting out-of-distribution image without learning from out-of-distribution data», 2020 IEEE/CVF 2.2.3 AutoAttack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 journal of uncertainty, fuzziness and knowledge-based systems, vol. 10, n o 05, p. MA, X., B. LI, Y. WANG, S. M. ERFANI, S. N. R. WIJEWICKREMA, G. SCHOENEBECK, D. SONG, M. E. HOULE and J. BAILEY. 2018, «Characterizing adversarial subspaces 557-570. 27 2.3
	1. GONG, Z., W. WANG and W.-S. KU. 2017, «Adversarial and clean data are not twins», Robust State Estimation Against Adversarial Noise. M. Picot, F. Messina, F. Conference on Computer Vision and Pattern Recognition (CVPR), p. 10 948-10 957. using local intrinsic dimensionality», in 6th International Conference on Learning SZEGEDY, C., W. ZAREMBA, I. SUTSKEVER, J. BRUNA, D. ERHAN, I. GOODFELLOW
	Representations. 33

Labeau, P. Piantanida. Submitted at IEEE Transactions on Smart Grid (TSG).

2.

A Simple Unsupervised Data Depth-based Method to Detect Adversarial Images. M. Picot, G. Staerman, N. Noiry, F. Messina, P. Piantanida, P. Colombo. Submitted at Transactions on Machine Learning Research (TMLR 2023). aware learning», in Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, p. 2792-2801. 26 ERLINGSSON, Ú., V. PIHUR and A. KOROLOVA. 2014, «Rappor: Randomized aggregatable privacy-preserving ordinal response», in Proceedings of the 2014 ACM SIGSAC conference on computer and communications security, p. 1054-1067. 27 FEINMAN, R., R. R. CURTIN, S. SHINTRE and A. B. GARDNER. 2017, «Detecting adversarial samples from artifacts», arXiv preprint arXiv:1703.00410. 33 GAL, Y. and Z. GHAHRAMANI. 2016, «Dropout as a bayesian approximation: Representing model uncertainty in deep learning», in international conference on machine learning, PMLR, p. 1050-1059. 28 GILAD-BACHRACH, R., N. DOWLIN, K. LAINE, K. LAUTER, M. NAEHRIG and J. WERNS-ING. 2016, «Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy», in International conference on machine learning, PMLR, arXiv preprint arXiv:1704.04960. 33 GOODFELLOW, I. J., J. SHLENS and C. SZEGEDY. 2015, «Explaining and harnessing adversarial examples», International Conference on Learning Representations. 31, 32 CHAPTER 1. INTRODUCTION GRANESE, F., M. ROMANELLI, D. GORLA, C. PALAMIDESSI and P. PIANTANIDA. 2021, «Doctor: A simple method for detecting misclassification errors», Advances in HENDRIK METZEN, J., M. CHAITHANYA KUMAR, T. BROX and V. FISCHER. 2017, «Universal adversarial perturbations against semantic image segmentation», in Proceedings of the IEEE international conference on computer vision, p. 2755-2764. 30 HENDRYCKS, D. and T. DIETTERICH. 2018, «Benchmarking neural network robustness to common corruptions and perturbations», in International Conference on Learning Representations. 28 HENDRYCKS, D. and K. GIMPEL. 2016, «A baseline for detecting misclassified and out-of-distribution examples in neural networks», arXiv preprint arXiv:1610.02136. 28 HENDRYCKS, D. and K. GIMPEL. 2017, «A baseline for detecting misclassified and out-of-distribution examples in neural networks», in International Conference on Learning Representations. 27 HENDRYCKS, D., M. MAZEIKA and T. DIETTERICH. 2019, «Deep anomaly detection with outlier exposure», in International Conference on Learning Representations. 28 HU, S., T. YU, C. GUO, W.-L. CHAO and K. Q. WEINBERGER. 2019, «A new defense against adversarial images: Turning a weakness into a strength», Advances in Neural Information Processing Systems, vol. 32. 33 CHAPTER 1. INTRODUCTION HUANG, C., P. KAIROUZ, X. CHEN, L. SANKAR and R. RAJAGOPAL. 2017, «Contextaware generative adversarial privacy», posium on Security and Privacy (SP), IEEE, p. 299-316. 27 JIANG, H., B. KIM, M. GUAN and M. GUPTA. 2018, «To trust or not to trust a classifier», Advances in neural information processing systems, vol. 31. 28 KHERCHOUCHE, A., S. A. FEZZA, W. HAMIDOUCHE and O. DÉFORGES. 2020, «Natural scene statistics for detecting adversarial examples in deep neural networks», in 22nd IEEE International Workshop on Multimedia Signal Processing, IEEE, p. 1-6. 33 KIRICHENKO, P., P. IZMAILOV and A. G. WILSON. 2020, «Why normalizing flows fail to detect out-of-distribution data», in Advances in Neural Information Processing Systems, vol. 33, édité par H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan and H. Lin, Curran Associates, Inc., p. 20 578-20 589. 28 KOS, J., I. FISCHER and D. SONG. 2018, «Adversarial examples for generative models», in 2018 ieee security and privacy workshops (spw), IEEE, p. 36-42. 30 KOSHEVOY, G. and K. MOSLER. 1997, «Zonoid trimming for multivariate distributions», The Annals of Statistics, vol. 25, n o 5, p. 1998-2017. 35 LAKSHMINARAYANAN, B., A. PRITZEL and C. BLUNDELL. 2017, «Simple and scalable predictive uncertainty estimation using deep ensembles», Advances in neural information processing systems, vol. 30. 28 LEE, K., K. LEE, H. LEE and J. SHIN. 2018, «A simple unified framework for detecting out-of-distribution samples and adversarial attacks», in Advances in Neural Information Processing Systems 31, édité par S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, Curran Associates, Inc., p. 7167-7177. 28 CHAPTER 1. INTRODUCTION LEE, S., C. PARK, H. LEE, J. YI, J. LEE and S. YOON. 2021, «Removing undesirable feature contributions using out-of-distribution data», in International Conference on Learning Representations. 27 LI, D., Y. ZHANG, H. PENG, L. CHEN, C. BROCKETT, M.-T. SUN and B. DOLAN. distribution image detection in neural networks», in International Conference on Learning Representations. 28 LIN, Y.-C., Z.-W. HONG, Y.-H. LIAO, M.-L. SHIH, M.-Y. LIU and M. SUN. 2017, «Tactics of adversarial attack on deep reinforcement learning agents», arXiv preprint arXiv:1703.06748. 30 LIU, R. Y. 1990, «On a notion of data depth based on random simplices», The Annals of Statistics, vol. 18, n o 1, p. 405-414. 35 LIU, R. Y. 1992, Data Depth and Multivariate Rank Tests, North-Holland, Amsterdam, p. 279-294. 35 LIU, W., X. WANG, J. OWENS and Y. LI. 2020, «Energy-based out-of-distribution detection», Advances in Neural Information Processing Systems. 28 LIU, Y., P. NING and M. K. REITER. 2011, «False data injection attacks against state estimation in electric power grids», ACM Transactions on Information and System Security (TISSEC), vol. 14, n o 1, p. 1-33. 34 LOHOKARE, A., A. SHAH and M. ZYDA. 2020, «Deep learning bot for league of legends», in Proceedings of the AAAI Conference on Artificial Intelligence and Interactive MADDOX, W. J., P. IZMAILOV, T. GARIPOV, D. P. VETROV and A. G. WILSON. 2019, «A simple baseline for bayesian uncertainty in deep learning», Advances in Neural Information Processing Systems, vol. 32. 28 CHAPTER 1. INTRODUCTION MADRY, A., A. MAKELOV, L. SCHMIDT, D. TSIPRAS and A. VLADU. 2018, «Towards deep learning models resistant to adversarial attacks», in International Conference MIRESHGHALLAH, F., M. TARAM, P. VEPAKOMMA, A. SINGH, R. RASKAR and H. ES-MAEILZADEH. 2020b, «Privacy in deep learning: A survey», arXiv preprint QUINTANILHA, I. M., R. DE M. E. FILHO, J. LEZAMA, M. DELBRACIO and L. O. NUNES. RIAZI, M. S., C. WEINERT, O. TKACHENKO, E. M. SONGHORI, T. SCHNEIDER and F. KOUSHANFAR. 2018, «Chameleon: A hybrid secure computation framework for machine learning applications», in Proceedings of the 2018 on Asia conference on computer and communications security, p. 707-721. 27 SASTRY, C. S. and S. OORE. 2020, «Detecting out-of-distribution examples with Gram matrices», in Proceedings of the 37th International Conference on Machine Learning, Proceedings of Machine Learning Research, vol. 119, édité par H. D. III and A. Singh, PMLR, p. 8491-8501. 28 SCHLEGL, T., P. SEEBÖCK, S. M. WALDSTEIN, U. SCHMIDT-ERFURTH and G. LANGS. 2017, «Unsupervised anomaly detection with generative adversarial networks to guide marker discovery», . 27 SCHNEIDER, S., E. RUSAK, L. ECK, O. BRINGMANN, W. BRENDEL and M. BETHGE. 2020, «Improving robustness against common corruptions by covariate shift adaptation», Advances in Neural Information Processing Systems, vol. 33, p. 11 539-11 551. 28 STAERMAN, G. 2022, Functional anomaly detection and robust estimation, thèse de and R. FERGUS. 2014, «Intriguing properties of neural networks», International Conference on Learning Representations. 28, 31 TABACOF, P., J. TAVARES and E. VALLE. 2016, «Adversarial images for variational autoencoders», arXiv preprint arXiv:1612.00155. 30 CHAPTER 1. INTRODUCTION XIE, Y., D. WANG, P.-Y. CHEN, J. XIONG, S. LIU and S. KOYEJO. 2022, «A word is worth a thousand dollars: Adversarial attack on tweets fools stock prediction», arXiv

Protecting Neural Network's Decisions . . . . . . . . . . . . . . . . 60

	2.3.1 Robustness: different defense mechanisms . . . . . . . . . . 61
	2.3.2 Adversarial training . . . . . . . . . . . . . . . . . . . . . . . . . 61
	2

.4 Ensuring the Input's Integrity . . . . . . . . . . . . . . . . . . . . . . 64

	CHAPTER 2. PRELIMINARIES
	Figure 2.1: Illustration of a Multi-Layer Perceptron
	2.4.1 Supervised detection methods . . . . . . . . . . . . . . . . . . 65
	2.4.2 Unsupervised detection methods . . . . . . . . . . . . . . . . 67

2.5 Review of the Smart Grid Case . . . . . . . . . . . . . . . . . . . . . 67 2

.5.1 The state estimation problem . . . . . . . . . . . . . . . . . . . 68 2.5.2 Attacking and defending state estimator . . . . . . . . . . . .

69 2.6 The Fisher-Rao measure, and the data-depths . . . . . . . . . . . . 71

	2.6.1 The Fisher-Rao distance (FRD) . . . . . . . . . . . . . . . . . . 71
	2.6.2 The data-depths . . . . . . . . . . . . . . . . . . . . . . . . . . 71
	2

.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 2.1 Deep Learning Background General supervised models. Let us first recall the previous notations. Let us consider an input space X ⊂ R d , where d is the dimension of the input, and output space

	Y .We define as p(x, y) the true joint data distribution.
	Let us define a deep neural model f θ parametrized by θ ∈ Θ with L layers. The
	output of each layer l is denoted by f l θ (•) and belongs in R d l where d l is the output
	dimension of the layer. Therefore, we have:

Saliency Mapping Attack The

	Jacobian-based Saliency Map
	Attack (JSMA) was introduced by PAPERNOT and collab. [2016b]. It is a targeted attack.

  KOTIUGA, W. W. and M. VIDYASAGAR. 1982, «Bad data rejection properties of weughted least absolute value techniques applied to static state estimation», IEEE Transactions on Power Apparatus and Systems, , n o 4, p. 844-853. 70 KRIZHEVSKY, A., I. SUTSKEVER and G. E. HINTON. 2012, «Imagenet classification with SHI, A. CUNNINGHAM and F. HUSZÁR. 2017, «Lossy image compression with compressive autoencoders», arXiv preprint arXiv:1703.00395. 52 TRAMÈR, F., N. PAPERNOT, I. GOODFELLOW, D. BONEH and P. MCDANIEL. 2017, «The space of transferable adversarial examples», arXiv preprint arXiv:1704.03453. 62 LI and K. REN. 2011, «Modeling load redistribution attacks in power systems», IEEE Transactions on Smart Grid, vol. 2, n o 2, p. 382-390. 69 ZHANG, H., Y. YU, J. JIAO, E. P. XING, L. E. GHAOUI and M. I. JORDAN. 2019, «Theoretically principled trade-off between robustness and accuracy», in International

		CHAPTER 2. PRELIMINARIES CHAPTER 2. PRELIMINARIES	CHAPTER 2. PRELIMINARIES
	HEIN, M. and M. ANDRIUSHCHENKO. 2017, «Formal guarantees on the robustness LIU, R. 1990, «On a notion of data depth based on random simplices», The Annals of MILI, L., M. G. CHENIAE and P. J. ROUSSEEUW. 1994, «Robust state estimation of
	of a classifier against adversarial manipulation», Advances in neural information Statistics, vol. 18, p. 405-414. 72 electric power systems», IEEE Transactions on Circuits and Systems I: Fundamental
	processing systems, vol. 30. 61 Theory and Applications, vol. 41, n o 5, p. 349-358. 70
		LIU, X., M. CHENG, H. ZHANG and C.-J. HSIEH. 2018a, «Towards robust neural
	HEINRICH, B. S. 2018, «Energy atlas 2018. figures and facts about renewables in networks via random self-ensemble», in Proceedings of the European Conference MOOSAVI-DEZFOOLI, S., A. FAWZI and P. FROSSARD. 2016, «Deepfool: A simple
	deep convolutional neural networks», Advances in neural information processing on Computer Vision (ECCV), p. 369-385. 61 and accurate method to fool deep neural networks», in 2016 IEEE Conference on Conference on Machine Learning, p. 1-11. 62 europe», . 11, 68 HENDRYCKS, D., K. LEE and M. MAZEIKA. 2019, «Using pre-training can improve systems, vol. 25. 50 KURAKIN, A., I. J. GOODFELLOW and S. BENGIO. 2018, «Adversarial examples in LIU, X., Y. LI, C. WU and C.-J. HSIEH. 2018b, «Adv-bnn: Improved adversarial defense through robust bayesian neural network», arXiv preprint arXiv:1810.01279. VASWANI, A., N. SHAZEER, N. PARMAR, J. USZKOREIT, L. JONES, A. N. GOMEZ, Computer Vision and Pattern Recognition (CVPR), ISSN 1063-6919, p. 2574-2582, Ł. KAISER and I. POLOSUKHIN. 2017, «Attention is all you need», Advances in ZHANG, Y., R. MADANI and J. LAVAEI. 2017, «Conic relaxations for power system state doi: 10.1109/CVPR.2016.282. 11, 56 neural information processing systems, vol. 30. 51 estimation with line measurements», IEEE Transactions on Control of Network model robustness and uncertainty», in International Conference on Machine Learn-ing, PMLR, p. 2712-2721. 64 the physical world», in Artificial intelligence safety and security, Chapman and Hall/CRC, p. 99-112. 55 61 LIU, Y., P. NING and M. K. REITER. 2011, «False data injection attacks against state Systems, vol. 5, n o 3, p. 1193-1205. 70 NAIR, V. and G. E. HINTON. 2010, «Rectified linear units improve restricted boltz-VINCENT, P., H. LAROCHELLE, Y. BENGIO and P.-A. MANZAGOL. 2008, «Extracting and mann machines», in Icml. 49 composing robust features with denoising autoencoders», in Proceedings of the ZHENG, Z. and P. HONG. 2018, «Robust detection of adversarial attacks by mod-HINTON, G., O. VINYALS and J. DEAN. 2015, «Distilling the knowledge in a neural LECUN, Y., B. BOSER, J. DENKER, D. HENDERSON, R. HOWARD, W. HUBBARD and estimation in electric power grids», ACM Transactions on Information and System PANG, T., C. DU, Y. DONG and J. ZHU. 2018, «Towards robust detection of adversarial 25th international conference on Machine learning, p. 1096-1103. 52 eling the intrinsic properties of deep neural networks», in Advances in Neural network», in NIPS Deep Learning and Representation Learning Workshop. 61 Security (TISSEC), vol. 14, n o 1, p. 1-33. 68, 69 examples», in Advances in Neural Information Processing Systems 31, p. 4584-4594. Information Processing Systems 31, p. 7924-7933. 65
		Representations. 65
		arXiv:1708.03981. 69
	MADRY, A., A. MAKELOV, L. SCHMIDT, D. TSIPRAS and A. VLADU. 2018, «Towards KHERCHOUCHE, A., S. A. FEZZA, W. HAMIDOUCHE and O. DÉFORGES. 2020, «Natural deep learning models resistant to adversarial attacks», in International Conference 2016, «A review of false scene statistics for detecting adversarial examples in deep neural networks», in on Learning Representations. 54, 55, 62 RAGHUNATHAN, A., J. STEINHARDT and P. LIANG. 2018a, «Certified defenses against 69, 70 data injection attacks against modern power systems», IEEE Transactions on Smart 22nd IEEE International Workshop on Multimedia Signal Processing, IEEE, p. 1-6. 11, 65, 66 61 MENG, D. and H. CHEN. 2017, «Magnet: A two-pronged defense against adversarial Grid, vol. 8, n o 4, p. 1630-1638. 69 adversarial examples», in International Conference on Learning Representations. IEEE, p. 226-231. 69
	LIANG, J., O. KOSUT and L. SANKAR. 2014, «Cyber attacks on ac state estimation: examples», in Proceedings of the 2017 ACM SIGSAC Conference on Computer and XU, W., D. EVANS and Y. QI. 2018, «Feature squeezing: Detecting adversarial examples
	KOSHEVOY, G. and K. MOSLER. 1997, «Zonoid trimming for multivariate distribu-Unobservability and physical consequences», in 2014 IEEE PES General Meeting| Communications Security, édité par B. M. Thuraisingham, D. Evans, T. Malkin and RAGHUNATHAN, A., J. STEINHARDT and P. S. LIANG. 2018b, «Semidefinite relaxations in deep neural networks», in 25th Annual Network and Distributed System Security
	51	65

ABUR, A. and A. G. EXPOSITO. 2004, Power system state estimation: theory and implementation, CRC press. 68 ALAYRAC, J.-B., J. UESATO, P.-S. HUANG, A. FAWZI, R. STANFORTH and P. KOHLI. 2019, «Are labels required for improving adversarial robustness?», in Advances in Neural Information Processing Systems, p. 12 214-12 223. 63, 64 ALDAHDOOH, A., W. HAMIDOUCHE and O. DÉFORGES. 2021a, «Revisiting model's uncertainty and confidences for adversarial example detection», arXiv preprint arXiv: 2103.05354. 65 ALDAHDOOH, A., W. HAMIDOUCHE, S. A. FEZZA and O. DÉFORGES. 2021b, «Adversarial example detection for DNN models: A review», arXiv preprint arXiv:2105.00203. 64 ANDRIUSHCHENKO, M., F. CROCE, N. FLAMMARION and M. HEIN. 2020, «Square attack: a query-efficient black-box adversarial attack via random search», in European Conference on Computer Vision, Springer, p. 484-501. 58 ATHALYE, A., N. CARLINI and D. WAGNER. 2018, «Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples», in International conference on machine learning, PMLR, p. 274-283. 61 ATKINSON, C. and A. F. S. MITCHELL. 1981, «Rao's distance measure», Sankhyā: The Indian Journal of

Statistics, Series A (1961-2002), vol. 43, n o

3, p. 345-365, ISSN 0581572X. 71 ATZMON, M., N. HAIM, L. YARIV, O. ISRAELOV, H. MARON and Y. LIPMAN. 2019, «Controlling neural level sets», in Advances in Neural Information Processing Systems, p. 2034-2043. 64 BOURLARD, H. and Y. KAMP. 1988, «Auto-association by multilayer perceptrons and singular value decomposition», Biological cybernetics, vol. 59, n o 4, p. 291-294. 52 BUNEL, R. R., I. TURKASLAN, P. TORR, P. KOHLI and P. K. MUDIGONDA. 2018, «A unified view of piecewise linear neural network verification», Advances in Neural Information Processing Systems, vol. 31. 61 CAO, X. and N. Z. GONG. 2017, «Mitigating evasion attacks to deep neural networks via region-based classification», in Proceedings of the 33rd Annual Computer Security Applications Conference, p. 278-287. 61 CARLINI, N. and D. WAGNER. 2017, «Towards evaluating the robustness of neural networks», in 2017 IEEE Symposium on Security and Privacy (SP), ISSN 2375-1207, p. 39-57, doi: 10.1109/SP.2017.49. 57 CARMON, Y., A. RAGHUNATHAN, L. SCHMIDT, J. C. DUCHI and P. S. LIANG. 2019, «Unlabeled data improves adversarial robustness», in Advances in Neural Information Processing Systems, p. 11 192-11 203. 63, 64 CARRARA, F., R. BECARELLI, R. CALDELLI, F. FALCHI and G. AMATO. 2018, «Adversarial examples detection in features distance spaces», in Computer Vision -ECCV 2018

Workshops -Munich, Germany, Proceedings, Part II, vol. 11130, Springer, p. 313-327. 65 CELIK, M. K. and A. ABUR. 1992

, «A robust wlav state estimator using transformations», IEEE Transactions on Power Systems, vol. 7, n o 1, p. 106-113. 70 CHEN, B., K. M. TING, T. WASHIO and G. HAFFARI. 2015, «Half-space mass: a maximally robust and efficient data depth method», Machine Learning, vol. 100, n o 2, p. 677-699. 72 CHEN, J., M. I. JORDAN and M. J. WAINWRIGHT. 2020, «Hopskipjumpattack: A queryefficient decision-based attack», in 2020 ieee symposium on security and privacy (sp), IEEE, p. 1277-1294. 59 CHENG, Z., H. SUN, M. TAKEUCHI and J. KATTO. 2018, «Deep convolutional autoencoder-based lossy image compression», in 2018 Picture Coding Symposium (PCS), IEEE, p. 253-257. 52 CHAPTER 2. PRELIMINARIES CHO, K., B. VAN MERRIËNBOER, C. GULCEHRE, D. BAHDANAU, F. BOUGARES, H. SCHWENK and Y. BENGIO. 2014, «Learning phrase representations using rnn encoder-decoder for statistical machine translation», arXiv preprint arXiv:1406.1078. 51 CHOW, C.-K. 1957, «An optimum character recognition system using decision functions», IRE Transactions on Electronic Computers, , n o 4, p. 247-254. 64 CHUNG, J., C. GULCEHRE, K. CHO and Y. BENGIO. 2014, «Empirical evaluation of gated recurrent neural networks on sequence modeling», arXiv preprint arXiv:1412.3555. 51 CISSE, M., P. BOJANOWSKI, E. GRAVE, Y. DAUPHIN and N. USUNIER. 2017, «Parseval networks: Improving robustness to adversarial examples», in International Conference on Machine Learning, PMLR, p. 854-863. 61 COHEN, J., E. ROSENFELD and Z. KOLTER. 2019, «Certified adversarial robustness via randomized smoothing», in International Conference on Machine Learning, PMLR, p. 1310-1320. 61 CROCE, F., M. ANDRIUSHCHENKO, V. SEHWAG, E. DEBENEDETTI, N. FLAMMARION, M. CHIANG, P. MITTAL and M. HEIN. 2020, «Robustbench: a standardized adversarial robustness benchmark», arXiv preprint arXiv:2010.09670. 60 CROCE, F. and M. HEIN. 2020a, «Minimally distorted adversarial examples with a fast adaptive boundary attack», in International Conference on Machine Learning, PMLR, p. 2196-2205. 11, 57 CROCE, F. and M. HEIN. 2020b, «Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks», in International Conference on Machine Learning, PMLR, p. 2206-2216. 55, 60 CUEVAS, A., M. FEBRERO and R. FRAIMAN. 2007, «Robust estimation and classification for functional data via projection-based depth notions», Computational Statistics, vol. 22, n o 3, p. 481-496. 72 DÁN, G. and H. SANDBERG. 2010, «Stealth attacks and protection schemes for state estimators in power systems», in 2010 first IEEE international conference on smart grid communications, IEEE, p. 214-219. 69, 70 DEVLIN, J., M.-W. CHANG, K. LEE and K. TOUTANOVA. 2019, «BERT: Pre-training of deep bidirectional transformers for language understanding», in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short HOCHREITER, S. and J. SCHMIDHUBER. 1997, «Long short-term memory», Neural computation, vol. 9, n o 8, p. 1735-1780. 51 HUANG, L., C. ZHANG and H. ZHANG. 2020, «Self-adaptive training: beyond empirical risk minimization», Advances in Neural Information Processing Systems, vol. 33. 64 HUG, G. and J. A. GIAMPAPA. 2012, «Vulnerability assessment of ac state estimation with respect to false data injection cyber-attacks», IEEE Transactions on smart grid, vol. 3, n o 3, p. 1362-1370. 69 JIN, M., J. LAVAEI and K. H. JOHANSSON. 2018, «Power grid ac-based state estimation: Vulnerability analysis against cyber attacks», IEEE Transactions on Automatic Control, vol. 64, n o 5, p. 1784-1799. 69, 70 KEKATOS, V., G. WANG, H. ZHU and G. B. GIANNAKIS. 2017, «Psse redux: Convex relaxation, decentralized, robust, and dynamic approaches», arXiv preprint tions», The Annals of Statistics, vol. 25, n o 5, p. 1998-2017. 72 KOSUT, O., L. JIA, R. J. THOMAS and L. TONG. 2010, «Malicious data attacks on smart grid state estimation: Attack strategies and countermeasures», in 2010 first IEEE international conference on smart grid communications, IEEE, p. 220-225. 69, 70 CHAPTER 2. PRELIMINARIES L. JACKEL. 1989, «Handwritten digit recognition with a back-propagation network», Advances in neural information processing systems, vol. 2. 11, 49, 50 LECUYER, M., V. ATLIDAKIS, R. GEAMBASU, D. HSU and S. JANA. 2019, «Certified robustness to adversarial examples with differential privacy», in 2019 IEEE Symposium on Security and Privacy (SP), IEEE, p. 656-672. 61 LEE, K., K. LEE, H. LEE and J. SHIN. 2018, «A simple unified framework for detecting out-of-distribution samples and adversarial attacks», in Advances in Neural Information Processing Systems 31, édité par S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett, Curran Associates, Inc., p. 7167-7177. 65 LI, X. and F. LI. 2017, «Adversarial examples detection in deep networks with convolutional filter statistics», in IEEE International Conference on Computer Vision, ICCV, IEEE Computer Society, p. 5775-5783. 65 LIANG, B., H. LI, M. SU, X. LI, W. SHI and X. WANG. 2021, «Detecting adversarial image examples in deep neural networks with adaptive noise reduction», IEEE Trans. Dependable Secur. Comput., vol. 18, n o 1, p. 72-85. 65 LIANG, G., J. ZHAO, F. LUO, S. R. WELLER and Z. Y. DONG. Conference & Exposition, IEEE, p. 1-5. 69 LIN, Z., M. FENG, C. N. D. SANTOS, M. YU, B. XIANG, B. ZHOU and Y. BENGIO. 2017, «A structured self-attentive sentence embedding», arXiv preprint arXiv:1703.03130. LOMUSCIO, A. and L. MAGANTI. 2017, «An approach to reachability analysis for feed-forward relu neural networks», arXiv preprint arXiv:1706.07351. 61 LU, J., T. ISSARANON and D. A. FORSYTH. 2017, «Safetynet: Detecting and rejecting adversarial examples robustly», in IEEE International Conference on Computer Vision, IEEE Computer Society, p. 446-454. 65 MA, S., Y. LIU, G. TAO, W. LEE and X. ZHANG. 2019, «NIC: detecting adversarial samples with neural network invariant checking», in 26th Annual Network and Distributed System Security Symposium, NDSS 2019, San Diego, California, USA, February 24-27, 2019, The Internet Society. 65 MA, X., B. LI, Y. WANG, S. M. ERFANI, S. N. R. WIJEWICKREMA, G. SCHOENEBECK, D. SONG, M. E. HOULE and J. BAILEY. 2018, «Characterizing adversarial subspaces using local intrinsic dimensionality», in 6th International Conference on Learning D. Xu, ACM, p. 135-147. 65, 67 METZEN, J. H., T. GENEWEIN, V. FISCHER and B. BISCHOFF. 2017, «On detecting adversarial perturbations», in 5th International Conference on Learning Representations. 65 PAPERNOT, N., P. MCDANIEL, I. GOODFELLOW, S. JHA, Z. B. CELIK and A. SWAMI. 2017, «Practical black-box attacks against machine learning», in Proceedings of the 2017 ACM on Asia conference on computer and communications security, p. 506-519. 58 PAPERNOT, N., P. D. MCDANIEL and I. J. GOODFELLOW. 2016a, «Transferability in machine learning: from phenomena to black-box attacks using adversarial samples», CoRR, vol. abs/1605.07277. 61 PAPERNOT, N., P. D. MCDANIEL, S. JHA, M. FREDRIKSON, Z. B. CELIK and A. SWAMI. 2016b, «The limitations of deep learning in adversarial settings», 2016 IEEE European Symposium on Security and Privacy (EuroS&P), p. 372-387. 58 PASQUALETTI, F., R. CARLI and F. BULLO. 2011, «A distributed method for state estimation and false data detection in power networks», in 2011 IEEE International Conference on Smart Grid Communications (SmartGridComm), IEEE, p. 469-474. for certifying robustness to adversarial examples», Advances in Neural Information Processing Systems, vol. 31. 61 RAMSAY, K., S. DUROCHER and A. LEBLANC. 2019, «Integrated rank-weighted depth», Journal of Multivariate Analysis, vol. 173, p. 51-69. 72 THEIS, L., W. WANG, W. and Z. LU. 2013, «Cyber security in the smart grid: Survey and challenges», Computer networks, vol. 57, n o 5, p. 1344-1371. 70 WANG, Y., D. ZOU, J. YI, J. BAILEY, X. MA and Q. GU. 2019, «Improving adversarial robustness requires revisiting misclassified examples», in International Conference on Learning Representations. 63 WONG, E. and J. Z. KOLTER. 2020, «Learning perturbation sets for robust machine learning», arXiv preprint arXiv:2007.08450. 61 WONG, E., F. SCHMIDT, J. H. METZEN and J. Z. KOLTER. 2018, «Scaling provable adversarial defenses», in Advances in Neural Information Processing Systems, p. 8400-8409. 61 WU, D., S.-T. XIA and Y. WANG. 2020, «Adversarial weight perturbation helps robust generalization», Advances in Neural Information Processing Systems, vol. 33. 64 XIE, C., J. WANG, Z. ZHANG, Z. REN and A. YUILLE. 2017, «Mitigating adversarial effects through randomization», arXiv preprint arXiv:1711.01991. 61 XIE, L., Y. MO and B. SINOPOLI. 2010, «False data injection attacks in electricity markets», in 2010 First IEEE International Conference on Smart Grid Communications, Symposium, The Internet Society. 65, 67 YASENKO, L., Y. KLYATCHENKO and O. TARASENKO-KLYATCHENKO. 2020, «Image noise reduction by denoising autoencoder», in 2020 IEEE 11th International Conference on Dependable Systems, Services and Technologies (DESSERT), IEEE, p. 351-355. 52 YUAN, Y., Z. ZHU, H. and G. B. GIANNAKIS. 2012, «Robust power system state estimation for the nonlinear ac flow model», in 2012 North American Power Symposium (NAPS), IEEE, p. 1-6. 69, 70 ZUO, Y. 2003, «Projection-based depth functions and associated medians», The Annals of Statistics, vol. 31, n o 5, p. 1460-1490. 72 ZUO, Y. and R. SERFLING. 2000, «General notions of statistical depth function», The Annals of Statistics, vol. 28, n o 2, p. 461-482. 72

  Adversarial learning . . . . . . . . . . . . . . . . . . . . . . . . 94 Accuracy-robustness trade-offs . . . . . . . . . . . . . . . . . . 102

	CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION
	in terms of clean and robust performances while reducing the training time by
	20% over the best-performing methods.
	3.3 Adversarial Robustness with Fisher-Rao Regularization . . . . . . 95
	3.3.1 Information geometry and statistical manifold . . . . . . . . 95 3.1 Introduction
	3.3.2 The FIRE risk function . . . . . . . . . . . . . . . . . . . . . . . 98 Deep Neural Networks (DNNs) have achieved several breakthroughs in different
	3.3.3 FRD for the case of binary classification . . . . . . . . . . . . 99 fields such as computer vision, speech recognition, and Natural Language Processing
	3.3.4 FRD for the case of multiclass classification . . . . . . . . . . 100 (NLP). Nevertheless, it is well-known that these systems are extremely sensitive to
	3.3.5 Comparison between FRD and KL divergence . . . . . . . . . 101 small perturbations on the inputs [SZEGEDY and collab., 2014], known as adversarial
	3.4 Accuracy-Robustness Trade-offs and Learning in the Gaussian Model102 examples. Formally, an adversarial example represents a corrupted input, character-
	ized by a bounded optimal perturbation from the original vector, designed to fool a specified neural networks' task. Adversarial examples have already proven threatful A simple approach to cope with corrupted examples is to detect and discard 3.4.1 responding to natural and perturbed input features. Based on the information-geometric properties of the class of softmax distributions, we derive an explicit them before classification. For instance, FEINMAN and collab. [2017], ZHENG and
	characterization of the Fisher-Rao Distance (FRD) for the binary and multiclass HONG [2018], and GROSSE and collab. [2017] present different methods to detect
	cases, and draw some interesting properties as well as connections with stan-corrupted inputs. Although these ideas can be useful to ensure robustness to outliers

3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 3.1.1 Summary of contributions . . . . . . . . . . . . . . . . . . . . 92 3.1.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 3.2.1 3.4.2 Learning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 3.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.5.1 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105 3.5.2 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . 106 3.6 Proofs of Theorems and Propositions . . . . . . . . . . . . . . . . . 108 3.6.1 Review of Fisher-Rao Distance (FRD) . . . . . . . . . . . . . . 108 3.6.2 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . 109 3.6.3 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . 110 3.6.4 Proof of Proposition 2 . . . . . . . . . . . . . . . . . . . . . . . 112 3.7 Summary and Concluding Remarks . . . . . . . . . . . . . . . . . . 113 3.8 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114 Abstract Adversarial robustness has become a topic of growing interest in machine learning since it was observed that neural networks tend to be brittle. We propose an information-geometric formulation of adversarial defense and introduce FIRE, a new Fisher-Rao regularization for the categorical cross-entropy loss, which is based on the geodesic distance between the softmax outputs cordard regularization metrics. Furthermore, we verify on a simple linear and Gaussian model, that all Pareto-optimal points in the accuracy-robustness region can be reached by FIRE while other state-of-the-art methods fail. Empirically, we evaluate the performance of various classifiers trained with the proposed loss on standard datasets, showing up to a simultaneous 1% of improvement in several domains, including vision and NLP [ALZANTOT and collab.], hence leading to the emergence of the rich area of adversarial machine learning [VOROBEYCHIK and collab., 2018]. The effectiveness of adversarial examples has been attributed to the linear regime of DNNs [GOODFELLOW and collab., 2015] and the data manifold geometrical structure itself [GILMER and collab., 2018], among other hypotheses. More recently, it has been related to the existence of valuable features for classification but meaningless for humans [ILYAS and collab., 2019].

In this paper, we focus on the so-called white-box attacks, for which the attacker has full access to the model. However, it should be noted that black-box attacks, in which the attacker can only query predictions from the model without access to further information, are also feasible

[PAPERNOT and collab., 2017]

. The literature on adversarial machine learning is extensive and can be divided into three overlapping groups, studying the generation, detection, and defense aspects. The simplest method to generate adversarial examples is the Fast Gradient Sign Method (FGSM)

[GOODFELLOW and collab., 2015]

, including its iterative variant called Projected Gradient Descent (PGD)

[MADRY and collab., 2018]

. Although widely used, PGD has a few issues that can lead to overestimating the robustness of a model. AutoAttack

[CROCE and HEIN, 2020]

has been recently developed to overcome those problems, enabling an effective way to test and compare the different defensive schemes. (i.e., inputs with large deviations with respect to clean examples), they do not seem to be satisfactory solutions for mild adversarial perturbations. In addition, adversarial detection can generally be bypassed by sophisticated attack methods [CARLINI and WAGNER, 2017]. Recently, several works focused on improving the robustness of neural networks CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION by investigating various defense mechanisms. For instance, certified defense mechanisms addressed the need for more guarantees on the task performance beyond standard evaluation metrics [

Table 3 .

 3 1: Comparison between KL and Fisher-Rao based regularizer under white-box l ∞ threat model. Note that we do not use the same hyperparameters as presented inZHANG and collab. [2019] for the TRADES method. ©2022 IEEE.

	Defense	Dataset	ε	Structure	Natural	AutoAttack	Avg. Acc.	RunTime
	TRADES FIRE	MNIST	0.3	CNN CNN	99.27 ± 0.03 99.22 ± 0.02	94.27 ± 0.18 94.44 ± 0.14	96.77 ± 0.09 96.83 ± 0.10	2h22 2h06
	TRADES FIRE	CIFAR-10	8/255	WRN-34-10 WRN-34-10	85.84 ± 0.31 85.98 ± 0.09	50.47 ± 0.36 51.45 ± 0.32	68.15 ± 0.23 68.72 ± 0.22	13h49 11h00
	TRADES FIRE	CIFAR-100	8/255	WRN-34-10 WRN-34-10	59.62 ± 0.42 61.03 ± 0.21	25.89 ± 0.26 26.42 ± 0.21	42.76 ± 0.26 43.73 ± 0.12	13h49 11h10

Table 3 .

 3 2: Test robustness on different datasets under white-box l ∞ attack. We ran all methods on 5 different tries and reported the mean and standard deviation. The codes for UAT and Atzmon et al. are not publicly available. Note that retraining the SOTA methods modifies slightly the experimental results. ©2022 IEEE.

	Defense	Dataset	ε	Structure	Natural	AutoAttack	Avg. Acc.	Runtime
			Without Additional Data
	Madry et al. [MADRY and collab., 2018]			CNN	98.53 ± 0.06	88.62 ± 0.23	93.58 ± 0.14	2h03
	Atzmon et al.[ATZMON and collab., 2019] TRADES [ZHANG and collab., 2019]	MNIST	0.3	CNN CNN	99.35 99.27 ± 0.03	90.85 94.27 ± 0.18	95.10 96.77 ± 0.09	-2h22
	FIRE			CNN	99.22 ± 0.02	94.44 ± 0.14	96.83 ± 0.10	2h06
	Madry et al. [MADRY and collab., 2018]			WRN-34-10	87.56± 0.09	44.07 ± 0.27	65.82 ± 0.15	10h51
	Self Adaptive[HUANG and collab., 2020]			WRN-34-10	83.39 ± 0.19	53.11 ± 0.29	68.25 ± 0.14	13h57
	TRADES [ZHANG and collab., 2019] FIRE + Self Adaptive	CIFAR-10	8/255	WRN-34-10 WRN-34-10	84.79 ± 0.24 83.70 ± 0.36	52.12± 0.28 53.26 ± 0.19	68.45 ± 0.12 68.48 ± 0.13	17h49 11h12
	Overfitting [RICE and collab., 2020]			WRN-34-10	85.64 ± 0.55	51.72 ± 0.56	68.68 ± 0.44	42h01
	FIRE			WRN-34-10	85.98 ± 0.09	51.45 ± 0.32	68.72 ± 0.22	11h00
	Overfitting [RICE and collab., 2020]			RN-18	53.83	18.95	36.39	-
	Overfitting [RICE and collab., 2020]	CIFAR-100	8/255	WRN-34-10	59.22 ± 0.61	25.99 ± 0.51	42.61 ± 0.28	42h08
	FIRE			WRN-34-10	61.03 ± 0.21	26.42 ± 0.21	43.73 ± 0.12	11h10
			With Additional Data Using 80M-TI
	Pre-training[HENDRYCKS and collab., 2019]			WRN-28-10	86.93 ± 0.79	53.35 ± 0.81	70.14 ± 0.54	40h00 + 0h20
	UAT [ALAYRAC and collab., 2019]			WRN-106-8	86.46	56.03	71.24	-
	MART [WANG and collab., 2019]	CIFAR-10	8/255	WRN-28-10	87.39 ± 0.12	56.69 ± 0.28	72.04 ± 0.15	13h53
	RST-adv [CARMON and collab., 2019]			WRN-28-10	89.49 ± 0.41	59.69 ± 0.26	74.59 ± 0.27	22h12
	FIRE			WRN-28-10	89.73 ± 0.04	59.97 ± 0.11	74.86 ± 0.05	18h30

Table 3

 3

.2. Note that the codes for UAT and

Atzmon et al.

are not publicly available, therefore we reported the results available on RobustBench

[CROCE and collab., 2020]

.

Average Accuracy (Avg. Acc.): Overall, FIRE exhibits the best Avg. Acc. among compared methods in all settings. On CIFAR-10, at equivalent method, our FIRE method outperforms HUANG and collab.

[2020]

. The gain on natural examples is close to 0.3% and 0.15% in adversarial performances, giving an improvemnt of 0.23% of Avg. Acc. Moreover, our FIRE method performs slightly better (0.04%) than RICE

  3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATIONIn reference to IEEE copyrighted material which is used with permission in this thesis, the IEEE does not endorse any of Université Paris-Saclay's or McGill University's products or services. Internal or personal use of this material is per-YU, J. JIAO, E. P. XING, L. E. GHAOUI and M. I. JORDAN. 2019, «Theoretically principled trade-off between robustness and accuracy», in International

	CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION CHAPTER 3. ADVERSARIAL ROBUSTNESS VIA FISHER-RAO REGULARIZATION
	the 34th International Conference on Machine Learning, Proceedings of Machine HORN, R. and C. JOHNSON. 2013, Matrix Analysis, Matrix Analysis, Cambridge Uni-
	Learning Research, vol. 70, édité par D. Precup and Y. W. Teh, PMLR, p. 854-863. 96 versity Press, ISBN 9780521839402. 112
	COHEN, J., E. ROSENFELD and Z. KOLTER. 2019, «Certified adversarial robustness via Conference on Machine Learning, p. 1-11. 15, 93, 95, 101, 104, 105, 106, 107, 108 HUANG, L., C. ZHANG and H. ZHANG. 2020, «Self-adaptive training: beyond empirical mitted. If interested in reprinting/republishing IEEE copyrighted material for ad-randomized smoothing», in International Conference on Machine Learning, PMLR, risk minimization», Advances in Neural Information Processing Systems, vol. 33. 93, ZHENG, Z. and P. HONG. 2018, «Robust detection of adversarial attacks by modeling vertising or promotional purposes or for creating new collective works for resale or redistribution, please go to http://www.ieee.org/publications_standards/ publications/rights/rights_link.html to learn how to obtain a License from p. 1310-1320. 92 CROCE, F., M. ANDRIUSHCHENKO and M. HEIN. 2019, «Provable robustness of relu 105, 107 ILYAS, A., S. SANTURKAR, D. TSIPRAS, L. ENGSTROM, B. TRAN and A. MADRY. 2019, the intrinsic properties of deep neural networks», in Neural Information Processing Systems, p. 7924-7933. 91 Chapter 4
	RightsLink. If applicable, University Microfilms and/or ProQuest Library, or the Archives of Canada may supply single copies of the dissertation. «Adversarial examples are not bugs, they are features», in Advances in Neural networks via maximization of linear regions», in the 22nd International Conference on Artificial Intelligence and Statistics, PMLR, p. 2057-2066. 92 Information Processing Systems, p. 125-136. 91 Robust Autoencoder-based State
	3.8 References CROCE, F., M. ANDRIUSHCHENKO, V. SEHWAG, E. DEBENEDETTI, N. FLAMMARION, M. CHIANG, P. MITTAL and M. HEIN. 2020, «Robustbench: a standardized adver-KRIZHEVSKY, A. 2009, «Learning multiple layers of features from tiny images», cahier de recherche. 105 Estimation in Power Systems
	Chapter 4 Abstract
	2034-2043. 93, 105
	BISHOP, C. M. 2006, Pattern Recognition and Machine Learning, Springer. 103 on Learning Representations. 91, 93, 94, 105, 106
	JELOVIC, T. MANN and P. KOHLI. 2019, «Scalable verified training for provably ro-CALIN, O. and C. UDRI ŞTE. 2014, Geometric modeling in probability and statistics, MIRMAN, M., T. GEHR and M. VECHEV. 2018, «Differentiable abstract interpretation bust image classification», in Proceedings of the IEEE/CVF International Conference Springer. 110 on Computer Vision, p. 4842-4851. 92 for provably robust neural networks», in International Conference on Machine
	CARLINI, N. and D. WAGNER. 2017, «Adversarial examples are not easily detected: Learning, p. 3578-3586. 92
	GROSSE, K., P. MANOHARAN, N. PAPERNOT, M. BACKES and P. D. MCDANIEL. 2017, Bypassing ten detection methods», in Proceedings of the 10th ACM Workshop «On the (statistical) detection of adversarial examples», CoRR, vol. abs/1702.06280. PAPERNOT, N., P. MCDANIEL, I. GOODFELLOW, S. JHA, Z. B. CELIK and A. SWAMI. on Artificial Intelligence and Security, AISec '17, ACM, New York, NY, USA, ISBN 91 2017, «Practical black-box attacks against machine learning», in Proceedings of 978-1-4503-5202-4, p. 3-14, doi: 10.1145/3128572.3140444. 91 the 2017 ACM on Asia Conference on Computer and Communications Security,
	CARMON, Y., A. RAGHUNATHAN, L. SCHMIDT, J. C. DUCHI and P. S. LIANG. 2019, «Un-HENDRYCKS, D., K. LEE and M. MAZEIKA. 2019, «Using pre-training can improve ASIA CCS '17, ACM, New York, NY, USA, ISBN 978-1-4503-4944-4, p. 506-519, doi:
	10.1145/3052973.3053009. 91

ALAYRAC, J.-B., J. UESATO, P.-S. HUANG, A. FAWZI, R. STANFORTH and P. KOHLI. 2019, «Are labels required for improving adversarial robustness?», in Advances in Neural Information Processing Systems, p. 12 214-12 223. 93, 105 ALZANTOT, M., Y. SHARMA, A. ELGOHARY, B.-J. HO, M. SRIVASTAVA and K.-W. CHANG. «Generating natural language adversarial examples», in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 91 ATKINSON, C. and A. F. S. MITCHELL. 1981, «Rao's distance measure», Sankhyā: The Indian Journal of Statistics, Series A (1961-2002), vol. 43, n o 3, p. 345-365, ISSN 0581572X. 108, 109, 110 ATZMON, M., N. HAIM, L. YARIV, O. ISRAELOV, H. MARON and Y. LIPMAN. 2019, «Controlling neural level sets», in Advances in Neural Information Processing Systems, p. labeled data improves adversarial robustness», in Advances in Neural Information Processing Systems, p. 11 192-11 203. 93, 105, 106 CISSE, M., P. BOJANOWSKI, E. GRAVE, Y. DAUPHIN and N. USUNIER. 2017, «Parseval networks: Improving robustness to adversarial examples», in Proceedings of sarial robustness benchmark», arXiv preprint arXiv:2010.09670. 107 CROCE, F. and M. HEIN. 2020, «Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks», in International Conference on Machine Learning, PMLR, p. 2206-2216. 91, 106, 108 FEINMAN, R., R. R. CURTIN, S. SHINTRE and A. B. GARDNER. 2017, «Detecting adversarial samples from artifacts», CoRR, vol. abs/1703.00410. 91 GILMER, J., L. METZ, F. FAGHRI, S. S. SCHOENHOLZ, M. RAGHU, M. WATTENBERG and I. J. GOODFELLOW. 2018, «Adversarial spheres», . 91 GOODFELLOW, I. J., J. SHLENS and C. SZEGEDY. 2015, «Explaining and harnessing adversarial examples», International Conference on Learning Representations. 91, 92, 93, 106 GOWAL, S., K. D. DVIJOTHAM, R. STANFORTH, R. BUNEL, C. QIN, J. UESATO, R. ARANDmodel robustness and uncertainty», in International Conference on Machine Learning, PMLR, p. 2712-2721. 93, 105 HINTON, G., O. VINYALS and J. DEAN. 2015, «Distilling the knowledge in a neural network», in NIPS Deep Learning and Representation Learning Workshop. 92 LECUN, Y., C. CORTES and C. BURGES. 2010, «Mnist handwritten digit database», ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, vol. 2. 105 LECUYER, M., V. ATLIDAKIS, R. GEAMBASU, D. HSU and S. JANA. 2019, «Certified robustness to adversarial examples with differential privacy», in 2019 IEEE Symposium on Security and Privacy (SP), IEEE, p. 656-672. 92 LI, B., C. CHEN, W. WANG and L. CARIN. 2019, «Certified adversarial robustness with additive noise», in Advances in Neural Information Processing Systems, p. 9464-9474. 92 LOSHCHILOV, I. and F. HUTTER. 2017, «Sgdr: Stochastic gradient descent with warm restarts», International Conference on Learning Representations. 106 MADRY, A., A. MAKELOV, L. SCHMIDT, D. TSIPRAS and A. VLADU. 2018, «Towards deep learning models resistant to adversarial attacks», in International Conference PAPERNOT, N., P. D. MCDANIEL and I. J. GOODFELLOW. 2016, «Transferability in machine learning: from phenomena to black-box attacks using adversarial samples», CoRR, vol. abs/1605.07277. 92 ZHANG, H., Y.

  Review of the MILP Attack . . . . . . . . . . . . . . . . . . . . . 123

	CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER
	SYSTEMS
	4.2.3 4.3 Defense Against False Data Injection Attacks . . . . . . . . . . . . . 124
	4.3.1 A Variational Autoencoder-based state estimator . . . . . . . 124
	4.3.2 A new metric improving robustness against attacks . . . . . . 124
	4.3.3 Novel attacks exploiting the state-estimator knowledge . . . 125
	4.3.4 General defensive framework . . . . . . . . . . . . . . . . . . . 126
	4.4 Numerical Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
	4.4.1 Experimental set-up . . . . . . . . . . . . . . . . . . . . . . . . 127
	4.4.2 Performance of the proposed attack . . . . . . . . . . . . . . . 128
	4.4.3 Influence of the number of attacked meters . . . . . . . . . . 129
	4.4.
	tors. It is worth noting that this work is among the first to rely on deep neural
	networks (i.e., variational autoencoders in our case) in the field of robust
	state estimators for smart grid systems. Our extensive numerical experiments
	prove that our method is able to construct state estimators that are robust to
	state-of-the-art attacks in the case of multiple bus systems.

case, i.e., the DC model, and present our solution to build robust state estima-Contents 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120 4.2 Background on State Estimation and Attacks . . . . . . . . . . . . 122 4.2.1 Bayesian approach for power state estimation . . . . . . . . . 122 4.2.2 False data injection attack . . . . . . . . . . . . . . . . . . . . . 122 4 Defensive performances . . . . . . . . . . . . . . . . . . . . . . 129 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 4.6 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

Table 4 .

 4 1: Clean and Attacked Mean Error for Defenseless and Robust Estimator for different Bus Systems. ©2022 IEEE.

	Bus System	14buses 57buses 118buses
	Defenseless	Clean Error	0.0867	0.1171	0.0969
	Estimator	Attacked Error	1.37	7.4910	2.1568
		MILP Attack	0.4412	0.1248	0.1149
	Robust	Clean Error	0.0989	0.1527	0.1028
	Estimator	Attacked Error 0.1021	0.1535	0.1020
		MILP Attack	0.0982	0.1531	0.0994

  IEEE Transactions on Smart Grid, vol. 3, n o 4, p. 1790-1799.

	121
	Chapter 5
	Robust State Estimation Against
	Adversarial Noise
	Chapter 5 Abstract
	ABUR, A. and A. G. EXPOSITO. 2004, Power system state estimation: theory and
	implementation, CRC press. 120, 122
	ALDERSON, D. and R. DI PIETRO. 2016, «Operational technology: Are you vulnera-
	ble?», Governance Directions, vol. 68, n o 6, p. 339-343. 121
	ASHOK, A., M. GOVINDARASU and V. AJJARAPU. 2016, «Attack-resilient measurement
	design methodology for state estimation to increase robustness against cyber
	attacks», in 2016 IEEE Power and Energy Society General Meeting (PESGM), IEEE, p.
	1-5. 121

ATKINSON, C. and A. F. MITCHELL. 1981, «Rao's distance measure», Sankhyā: The Indian Journal of Statistics, Series A, p. 345-365. 124, 125 DÁN, G. and H. SANDBERG. 2010, «Stealth attacks and protection schemes for state CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS critical infrastructures», This chapter represents the final part of our answer to the first research question dedicated to increasing adversarial robustness. We present our third and final contribution. In the previous chapter, we present the extension of the framework presented in Chapter 3 to the state estimator's problem in smart grid systems. To build our robust state estimator in Chapter 4, we made the DC model assumption, i.e., that the state estimation problem was linear.

  CHAPTER 5. ROBUST STATE ESTIMATION AGAINST ADVERSARIAL NOISEThe remaining question is how to generate powerful attacks y a on which to train our robust state estimator. In the following, we will present two attack mechanisms that make use of the full knowledge about the targeted state estimator.

	Algorithm 2 Deterministic attack generation (PGD Algorithm)
	INPUT: Define the step δ and the number of iterations n PGD
	INPUT: y
	y a ← y + a with a ∼ Unif([-0.01, 0.01])
	for i = 1..n PGD do
	Compute l (y, y a ; φ) from Equation 5.13.
	a ← δ • sgn(∇ y (l (y, y a ; φ))), where sgn is the element-wise sign function.
	y a ← y a + a
	end for
	OUTPUT: y a

1 , (5.8)

Table 5 .

 5 1: Natural and Attacked Mean Error for Defenseless, Robust and LASSO state estimators for different Bus Systems

	Estimator		14buses Angle Voltage Angle Voltage Angle Voltage 57buses 118buses
		Natural	0.11	0.006	0.12	0.007	0.30	0.003
	Defenseless Random Attack 2.01	0.17	0.87	0.09	1.34	0.04
		SOTA Attack	2.39	0.21	0.71	0.07	0.41	0.07
		Natural	0.26	0.014	0.21	0.009	0.26	0.003
	Robust	Random Attack 0.26	0.016	0.21	0.013	0.26	0.009
		SOTA Attack	0.26	0.014	0.21	0.013	0.28	0.01
		Natural	30.2	0.050	60.4	0.052	87.4	0.044
	LASSO	Random Attack	-	-	-	-	-	-
		SOTA Attack	37.4	0.050	69.5	0.054	93.1	0.043

  ., O. KOSUT and L. SANKAR. 2014, «Cyber attacks on ac state estimation: Unobservability and physical consequences», in 2014 IEEE PES General Meeting| Conference & Exposition, IEEE, p. 1-5. 137 LIU, Y., P. NING and M. K. REITER. 2011, «False data injection attacks against state estimation in electric power grids», ACM Transactions on Information and System

	Security (TISSEC), vol. 14, n o 1, p. 1-33. 136, 137
	MADRY, A., A. MAKELOV, L. SCHMIDT, D. TSIPRAS and A. VLADU. 2018, «Towards
	deep learning models resistant to adversarial attacks», in International Conference
	on Learning Representations. 144
	MILI, L., M. G. CHENIAE and P. J. ROUSSEEUW. 1994, «Robust state estimation of
	electric power systems», IEEE Transactions on Circuits and Systems I: Fundamental
	Theory and Applications, vol. 41, n o 5, p. 349-358. 137
	Entropy, vol. 22, n o 4, p. 404. 142
	ABUR, A. and A. G. EXPOSITO. 2004, Power system state estimation: theory and
	implementation, CRC press. 136, 139 RAHMAN, M. A. and H. MOHSENIAN-RAD. 2013, «False data injection attacks against
	nonlinear state estimation in smart power grids», in 2013 IEEE Power & Energy
	ALDERSON, D. and R. DI PIETRO. 2016, «Operational technology: Are you vulnera-Society General Meeting, IEEE, p. 1-5. 136
	ble?», Governance Directions, vol. 68, n o 6, p. 339-343. 136 SANDBERG, H., A. TEIXEIRA and K. H. JOHANSSON. 2010, «On security indices for
	ATKINSON, C. and A. F. MITCHELL. 1981, «Rao's distance measure», Sankhyā: The state estimators in power networks», in First Workshop on Secure Control Systems
	Indian Journal of Statistics, Series A, p. 345-365. 142 (SCS), Stockholm, 2010. 137
	SOLTAN, S., M. YANNAKAKIS and G. ZUSSMAN. 2016, «Power grid state estimation fol-
	CELIK, M. K. and A. ABUR. 1992, «A robust wlav state estimator using transforma-lowing a joint cyber and physical attack», IEEE Transactions on Control of Network
	Systems, vol. 5, n

tions», IEEE Transactions on Power Systems, vol. 7, n o 1, p. 106-113. 137 DÁN, G. and H. SANDBERG. 2010, «Stealth attacks and protection schemes for state estimators in power systems», in 2010 first IEEE international conference on smart grid communications, IEEE, p. 214-219. 136, 137 LIANG, JPASQUALETTI, F., R. CARLI and F. BULLO. 2011, «A distributed method for state estimation and false data detection in power networks», in 2011 IEEE International Conference on Smart Grid Communications (SmartGridComm), IEEE, p. 469-474. 137 PICOT, M., F. MESSINA, M. BOUDIAF, F. LABEAU, I. B. AYED and P. PIANTANIDA. 2021, «Adversarial robustness via fisher-rao regularization», arXiv preprint arXiv:2106.06685. 142 PINELE, J., J. E. STRAPASSON and S. I. COSTA. 2020, «The fisher-rao distance between multivariate normal distributions: Special cases, bounds and applications», o 1, p. 499-512. 137 SOLTAN, S., M. YANNAKAKIS and G. ZUSSMAN. 2018, «React to cyber attacks on power grids», IEEE Transactions on Network Science and Engineering, vol. 6, n o 3, p. 459-473. 137

  To have a wide range of attacks to test, we use all the methods mentioned in Subsection 6.2.3. For FGSM, BIM and PGD, we consider the L ∞ -norm, with multiple ε in {0.0315, 0.0625, 0.125, 0.25, 0.3125, 0.5}. We also generate perturbed examples using PGD under the L 1https://github.com/bearpaw/pytorch-classification ATTACK DETECTION outliers, i.e., adversarial examples behaving differently from the others.

Table 6 .

 6 3: Detector performances and Attack's success under adaptive blackbox attacks for NSS and HAMPER BA on CIFAR10. We present the results as: AUROC↑% ±FPR ↓ 95% % for the detector performances.

		Adaptive Attacks
			CIFAR10
		NSS	HAMPER BA
		Detector	Attack	Detector	Attack
		Performances	Success (%)	Performances	Success (%)
	α = 10 -3	9.4 ±100	95.82	79.3 ±67.5	98.93
	α = 10 -2	7.2 ±100	96.12	95.4 ±26.5	99.36
	α = 10 -1	5.5 ±100	93.60	74.6 ±65.8	99.32
	α = 1	7.8 ±100	95.88	85.5 ±66.1	98.77
	α = 10 1	3.2 ±100	93.28	63.6 ±87.1	95.83
	α = 10 2	3.5 ±100	93.83	68.0 ±75.6	42.42
	α = 10 3	3.7 ±100	93.44	47.7 ±86.1	25.06

Table 6 .

 6

	4: Time and computational con-
	straints to train and test each detection
	method.
	Method	GPUs	Training Time Testing Time
	NSS	V100-16G	00m30s	00m55s
	KD-BU	V100-16G	00m30s	02m00s
	LID	V100-16G	04m00s	35m00s

  Like all the supervised adversarial detection methods in the literature, HAMPER requires knowledge of the kind of attack evaluated at testing time, meaning that they are generally validated by assuming a single implicitly known attack strategy, which does not necessarily account for real-life threats. In future work, we will investigate how HAMPER and its main competitors perform when facing samples crafted through multiple unknown attack strategies at test time. We will mainly make an effort to understand whether the notion of depth is well suited to better generalize the detection of adversarial examples to attacks that are not involved in the supervised framework. Our depth-based detector . . . . . . . . . . . . . . 7.3.3 Comparison with existing detectors . . . . . . . . . . . . . . . Set-up . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Adversarial attack calibration . . . . . . . . . . . . . . . . . . . 7.4.3 Locating the relevant information . . . . . . . . . . . . . . . . Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Adaptive attacks . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.3 Finer analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO
	ATTACK DETECTION DETECT ADVERSARIAL IMAGES
	KRIZHEVSKY, A. 2009, «Learning multiple layers of features from tiny images», cahier de recherche. 172 KURAKIN, A., I. J. GOODFELLOW and S. BENGIO. 2018, «Adversarial examples in 7.3.2 APPROVED: 7.4 Adversarial Attacks on Vision Transformers (ViT) . . . . . . . . . .
	Through our extensive experiments based on two scenarios, attack-aware, and blind-to-attack, we demonstrate that HAMPER achieves state-of-the-art performances. On average, it outperforms the existing best detector by 13.1% (29.0%) in terms of AUROC↑ (resp. FPR↓ 95% ) on SVHN, 14.6% (25.7%) on CIFAR10, and 22.6% (49.0%) on CI-FAR100. Interestingly, HAMPER exhibits class-independence, and less dependence on the norm-attack and the threat scenarios than other adversarial attack detection methods. the physical world», in Artificial intelligence safety and security, Chapman and Hall/CRC, p. 99-112. 167 LANGE, T., K. MOSLER and P. MOZHAROVSKYI. 2014, «Fast nonparametric classifica-tion based on data depth», Statistical Papers, vol. 55, n o 1, p. 49-69. 164 LI, X. and F. LI. 2017, «Adversarial examples detection in deep networks with con-volutional filter statistics», in IEEE International Conference on Computer Vision, ICCV, IEEE Computer Society, p. 5775-5783. 163 LIANG, B., H. LI, M. SU, X. LI, W. SHI and X. WANG. 2021, «Detecting adversarial image examples in deep neural networks with adaptive noise reduction», IEEE Trans. Dependable Secur. Comput., vol. 18, n o 1, p. 72-85. 163 LIU, F. T., K. M. TING and Z.-H. ZHOU. 2008, «Isolation forest», in 2008 eighth ieee international conference on data mining, IEEE, p. 413-422. 164 LU, J., T. ISSARANON and D. A. FORSYTH. 2017, «Safetynet: Detecting and rejecting adversarial examples robustly», in IEEE International Conference on Computer Vision, IEEE Computer Society, p. 446-454. 163 Chapter 7 7.4.1 7.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A Simple Unsupervised Data Depth-based Method to Detect Adversarial Images Chapter 7 Abstract We here present our final contribution. In the previous chapter, we made use of data-depths to detect adversarial examples in a supervised setting. In this work, we wanted to free our method from the necessity to gather knowledge 7.5.1 Although the research community has developed a large amount of effective
	about the possible attacker. In addition, we observed that, no other previous attacks, the detection problem has received little attention. Existing detection
	MA, S., Y. LIU, G. TAO, W. LEE and X. ZHANG. 2019, «NIC: detecting adversarial work focused on developing detection methods on Vision Transformer, in methods either rely on additional training or on specific heuristics at the risk of
	samples with neural network invariant checking», in 26th Annual Network and spite of it being the SOTA architecture to classify images. We therefore propose overfitting. Moreover, they have mainly focused on ResNet architectures while
	Distributed System Security Symposium, The Internet Society. 163 a detection method that leverages the particular structure of Vision Trans-transformers, which are state-of-the-art for vision tasks, have not been properly
	formers, and the data-depths (more precisely the Integrated-Rank-Weighted investigated. In this paper, we overcome these limitations by introducing AP-MA, X., B. LI, Y. WANG, S. M. ERFANI, S. N. R. WIJEWICKREMA, G. SCHOENEBECK, depth) to efficiently detect adversarial examples. We experimentally show the PROVED, a simple unsupervised detection method for transformer architectures. D. SONG, M. E. HOULE and J. BAILEY. 2018, «Characterizing adversarial subspaces advantages of our method compared to previous state-of-the-art unsupervised It leverages the information available in the logit layer and computes a similarity using local intrinsic dimensionality», in 6th International Conference on Learning score with respect to the training distribution. This is accomplished using a data method. Moreover, we show that directly attacking our proposed detector is far Representations. 163, 166, 170 MADRY, A., A. MAKELOV, L. SCHMIDT, D. TSIPRAS and A. VLADU. 2018, «Towards depth that is: (i) computationally efficient; and (ii) non-differentiable, making it from being straightforward and that our method makes the adaptive attacker's harder for gradient-based adversaries to craft malicious samples. Our extensive job more complicated. experiments show that APPROVED consistently outperforms previous detectors deep learning models resistant to adversarial attacks», in International Conference Broader Impact Statement on Learning Representations. 163, 167 on CIFAR10, CIFAR100 and Tiny ImageNet.
	MEINKE, A. and M. HEIN. 2020, «Neural networks that provably know when they
	Many concerns have been raised about the potential failures of Deep Learning: large don't know», in 8th International Conference on Learning Representations, ICLR.
	neural networks are not trustworthy enough, limiting their adoption in high-risk
	applications. This paper's main contribution aims at improving the reliability of
	Deep Learning by designing a tool to prevent a malicious agent from disrupting the
	functioning of the system. Thus we believe our work will have a positive impact on
	society.

163 MENG, D. and H. CHEN. 2017, «Magnet: A two-pronged defense against adversarial examples», in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, édité par B. M. Thuraisingham, D. Evans, T. Malkin and D. Xu, ACM, p. 135-147. 163 Contents 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 7.2 Background and Related Work . . . . . . . . . . . . . . . . . . . . . 194 7.2.1 Review of attack mechanisms . . . . . . . . . . . . . . . . . . . 195 7.2.2 Review of detection methods . . . . . . . . . . . . . . . . . . . 196 7.3 Our Proposed Detector . . . . . . . . . . . . . . . . . . . . . . . . . . 197 7.3.1 Background on data depth . . . . . . . . . . . . . . . . . . . . 197 7.6 Conclusions and Limitations . . . . . . . . . . . . . . . . . . . . . . 7.7 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Abstract

Deep neural networks suffer from critical vulnerabilities regarding robustness, which limits their exploitation in many real-world applications. In particular, a serious concern is their inability to defend against adversarial attacks.

  CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES PROVED, A simPle unsuPeRvised method fOr adVersarial imagE Detection. Given an input, APPROVED considers its embedding in the last layer of the pre-trained classifier and computes the depth of the sample w.r.t the training probability distribution. The deeper it is, the less likely it is to be adversarial. Contrarily to existing methods that in-

	volve additional networks training [MENG and CHEN, 2017] or heavily rely on opaque
	feature engineering [XU and collab., 2018], APPROVED is computationally efficient and
	has a simple geometrical interpretation. Moreover, data depths non-differentiability
	making it harder for to gradient-based attackers to target APPROVED.
	2. A truly upgraded experimental setting. Motivated by practical considerations
	which are different from previous works [

  CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES will be perpetrated, and unsupervised methods, where the detector can only rely on clean samples, which is not desired in practice.Generally, supervised methods use simple machine learning algorithms (e.g., SVM or a logistic regressor) to distinguish between natural and adversarial examples.

	The effectiveness of such methods heavily relies on natural and adversarial feature
	extraction. They can be extracted directly from the network's layers [CARRARA
	Multiple techniques have since been crafted to solve this problem. They can be
	divided into two main groups of attack mechanisms depending on the knowledge
	they have of the DNN model: whitebox and blackbox attacks. The former has full
	access to the model, its weights, and gradients, while the latter can only rely on
	queries.
	Carlini & Wagner's (CW) [CARLINI and WAGNER, 2017] attack is among the strongest
	whitebox attacks developed yet. It attempts to solve the adversarial problem in Equa-
	tion 7.1 by regularizing the minimization of the perturbation norm by a surrogate
	of the misclassification constraint. DeepFool (DF) [MOOSAVI-DEZFOOLI and collab.,
	2016] is an iterative method that solves a locally linearized version of the adversarial
	problem and takes a step in that direction.
	The authors of GOODFELLOW and collab. [2014] relax the problem as follows:
	x ′ = arg max	L (x, x ′ ; θ),	(7.2)
	x ′ ∈R d : ∥x ′ -x∥ p <ε

where L (x, x ′ ; θ) is the objective of the attacker, which is a surrogate of the misclassification constraint, and propose the Fast Gradient Sign Method (FGSM) that approximates the solution of the relaxed problem in Equation

7

.2 by taking one step in the direction of the sign of the gradient of the attacker's objective w.r.t. the input.

Basic Iterative Method (BIM)

[KURAKIN and collab., 2018]

and Projected Gradient

  Therefore, even though, at inference time, little time is necessary to output the prediction, MagNet requires careful training, which is time-consuming, and storing two autoencoders, which is highly memory-consuming. CHAPTER 7. A SIMPLE UNSUPERVISED DATA DEPTH-BASED METHOD TO DETECT ADVERSARIAL IMAGES in terms of FPR↓ 90% on CIFAR10. Finally, by looking at the detailed results presented in Table C.3 and Table C.4, we can deduce that FS and APPROVED have opposite behaviors: when the performances of FS decrease, APPROVED's performances tend to improve. For example, under the L ∞ -norm constraint, APPROVED has more trouble detecting attacks with small perturbations, while FS has more difficulty detecting attacks with large perturbations. Indeed, since APPROVED measures the depth of a sample within a distribution, it will be able to recognize the strongest attacks well. In Table 7.3 we give the overall idea of the results on all three datasets per attack mechanism by showing them in terms of mean and standard deviation (std) on the AUROC↑ and on the FPR↓ 90% . APPROVED turns out to consistently outperform the state-of-the-art detectors forall datasets. In particular, we notice that the FGSM attacks that are the easiest to generate are the ones that present the highest diversity among the results in the methods examined. Indeed, by looking at Table 7.3, we can find larger values of the standard deviation in correspondence to that attack. Moreover, APPROVED is capable of recognizing attacks that are more difficult for the competitors (e.g., STA for MagNet or FGSM for FS).

	Performances per attack.

  Two scenarios can be considered with adaptive attacks:

	whitebox and blackbox. Whitebox attacks (e.g. BPDA [ATHALYE and collab., 2018])
	are not straightforward to adapt in our case since finding a differentiable surrogate of
	IRW remains a very challenging open research question in the statistical community,
	which has never been tackled. As a matter of fact, the only attempts to approximate
	a non-differentiable depth was performed not on the IRW depth but on the Tuckey
	depth in DYCKERHOFF and collab. [2021], with very poor results as pointed out in SHE
	and collab. [2021]. Thus, in this experiment, we rely on blackbox attacks and present
	the results in Figure 7.4. We attacked both APPROVED and FS using a modified version
	of SA [ENGSTROM and collab., 2019], for which the attack objective has been modified

  new input compared to a reference probability distribution, and are non-differentiable, making the attacker's job more difficult. In the case of supervised methods, where the defense has either complete or partial knowledge about the threats it is going to face, we proposed HAMPER, a supervised method that leverages the halfspace-mass depth score at different layer levels. We presented two versions of our solution, one where complete knowledge about the attacker is available and another where only partial knowledge is available. Our method is somehow similar to the Local Intrinsic Dimensionality method as we compare a new point to a reference. However, we only compute our anomality score on a subset of layers and not all of them. Experimentally, our proposed detection method significantly and consistently outperforms previous state-of-the-art methods in all considered scenarios, namely the attack-aware scenario, the blind-to-attack scenario, or under adaptive attacks.In terms of computational time, our method is comparable to the fastest methods, namely the Natural Scene Statistics method[KHERCHOUCHE and collab., 2020] and the Kernel Density and Bayesian Uncertainty method[FEINMAN and collab., 2017].

	CHAPTER 8. DISCUSSION OF FINDINGS, LIMITATIONS, POTENTIAL FUTURE
	WORKS AND GLOBAL SUMMARY
	ordering of a
	ALAYRAC, J.-B., J. UESATO, P.-S. HUANG, A. FAWZI, R. STANFORTH and P. KOHLI. 2019,
	«Are labels required for improving adversarial robustness?», in Advances in Neural
	Information Processing Systems, p. 12 214-12 223. 196
	196

ALDAHDOOH, A., W. HAMIDOUCHE and O. DEFORGES. 2021a, «Reveal of vision transformers robustness against adversarial attacks», arXiv preprint arXiv:2106.03734.

Table 8 .

 8 1: Summary of Detector's requirements meets

Table A .

 A 1: Comparison between Hellinger and Fisher-Rao based regularizer under white-box l ∞ threat model.

	Defense	Dataset	ε	Structure	Natural	AutoAttack	Avg. Acc.	RunTime
	Hellinger FIRE	MNIST	0.3	CNN CNN	99.31 ± 0.03 99.22 ± 0.02	94.03 ± 0.24 94.44 ± 0.14	96.67 ± 0.07 96.83 ± 0.10	2h06 2h06
	Hellinger FIRE	CIFAR-10	8/255	WRN-34-10 WRN-34-10	85.96 ± 0.28 85.98 ± 0.09	50.52 ± 0.31 51.45 ± 0.32	68.24 ± 0.15 68.72 ± 0.22	11h02 11h00
	Hellinger FIRE	CIFAR-100	8/255	WRN-34-10 WRN-34-10	60.79 ± 0.88 61.03 ± 0.21	25.58 ± 0.27 26.42 ± 0.21	43.18 ± 0.54 43.73 ± 0.12	11h12 11h10
	A.2

Comparison between Fisher-Rao and Hellinger dis- tances A.2.1 Theoretical relation

  Randomly and uniformly select κ k in mid k -λ 2 range k , mid k + λ 2 range k . Testing algorithm for the approximation of D HM .

	5:	Set mid k = max i	p k,i + min i	p k,i /2 and range k = max i	p k,i -min i	p k,i .
	7:	Set m left k =	|{z ℓ,i ∈ S ℓ c,n s : p k,i < κ k }| n s	and m	right k =	|{z ℓ,i ∈ S ℓ c,n s : p k,i ≥ κ k }| n s	.
				right k	} K k=1 .
	Algorithm 6 right k	} K k=1 .
	INPUT: : HM=0.

6:

8: end for OUTPUT:

: {u k , κ k , m left k , m INPUT: : test observation z ℓ ; {u k , κ k , m left k , m 1: for k = 1, . . . , K do 2:

Detailed Results on No Knowledge about the attacker

  TableB.2: Performances on the three considered datasets -SVHN, CIFAR10, and CIFAR100of the HAMPER BA detector together with the results of the state-of-the-art detection method: NSS, on multiple threat scenarios with multiple maximal perturbations ε. The best results among the detectors are shown in bold. The results are presented as: AUROC↑ ±FPR ↓ 95% %. * stipulates the non-gradient based attacks. ±0.3 ε = 0.3125 (ε ⋆ = 15) 52.5 ±94.1 59.0 ±88.2

	Norm L1 B.4 Norm L 1	LID SVHN CIFAR10 CIFAR100 NSS SVHN CIFAR10 CIFAR100 ⋆	KD-BU SVHN CIFAR10 CIFAR100 SVHN CIFAR10 CIFAR100 HAMPERAA HAMPER BA SVHN CIFAR10 CIFAR100 ⋆
	PGD1 PGD 1
	ε = 5 (ε ⋆ = 40) ε = 5 (ε ⋆ = 40) ε = 10 (ε ⋆ = 500) ε = 10 (ε ⋆ = 500) ε = 15 (ε ⋆ = 1000) ε = 15 (ε ⋆ = 1000) ε = 20 (ε ⋆ = 1500) ε = 20 (ε ⋆ = 1500) ε = 25 (ε ⋆ = 2000) ε = 25 (ε ⋆ = 2000) ε = 30 (ε ⋆ = 2500) ε = 30 (ε ⋆ = 2500)	78.6 ±76.9 63.5 ±84.3 48.6 ±95.1 50.1 ±94.3 66.6 ±89.1 52.2 ±92.9 51.5 ±94.6 56.7 ±90.0 47.2 ±94.8 48.9 ±93.7 59.4 ±91.7 62.5 ±85.2 61.6 ±92.7 46.8 ±95.0 69.2 ±85.1 67.7 ±80.3 65.7 ±92.0 47.0 ±95.0 78.1 ±71.7 72.0 ±74.5 61.6 ±94.1 68.9 ±82.5 84.8 ±55.5 75.9 ±68.6	49.6 ±94.1 51.6 ±94.3 36.0 ±97.1 53.1 ±94.2 76.2 ±66.2 58.3 ±93.3 90.1 ±35.0 66.5 ±91.8 95.4 ±22.2 75.5 ±89.2 96.4 ±17.1 83.7 ±84.4		81.6 ±57.8 32.2 ±98.4 95.7 ±20.4 88.4 ±47.5 75.0 ±70.0 41.1 ±96.9 87.6 ±51.0 90.4 ±38.8 76.7 ±83.6 64.9 ±91.4 95.1 ±23.5 89.2 ±49.0 84.3 ±64.1 77.9 ±83.7 96.0 ±21.8 98.3 ±8.5 88.8 ±55.2 87.8 ±70.5 96.4 ±19.8 98.3 ±8.2 91.2 ±54.6 94.2 ±32.0 95.7 ±27.5 98.8 ±5.4	40.5 ±96.3 99.9 ±0.4 100 ±0.0 40.2 ±98.6 100 ±0.0 99.9 ±0.3 64.9 ±84.6 100 ±0.0 99.9 ±0.2 72.3 ±80.7 100 ±0.0 99.9 ±0.2 80.9 ±74.0 100 ±0.0 99.9 ±0.2 88.1 ±64.0 100 ±0.0 100 ±0.1	±0.0 ±0.0 ±0.0 ±0.0 ±0.0 ±0.0	100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.0
	ε = 40 (ε ⋆ = 5000) ε = 40 (ε ⋆ = 5000)	73.2 ±91.3 77.7 ±70.2 92.9 ±24.3 82.1 ±57.1	99.8 ±0.1 99.4 ±0.2		95.3 ±25.5 97.9 ±10.1 99.3 ±2.5 98.7 ±5.0	99.7 ±0.2 100 ±0.0 100 ±0.0	±0.0	100 ±0.0
	Norm L2 Norm L 2 PGD2 PGD 2 ε = 0.125 (ε ⋆ = 5) ε = 0.25 (ε ⋆ = 10) ε = 0.125 (ε ⋆ = 5) ε = 0.3125 (ε ⋆ = 15) 68.9 ±84.3 51.2 ±93.2 SVHN CIFAR10 CIFAR100 SVHN CIFAR10 CIFAR100 ⋆ APPROVED AUROC FPR PGD ∞ 80.8 ±76.0 63.6 ±84.3 45.3 ±95.2 74.8 ±79.1 59.0 ±86.3 39.7 ±96.5 49.2 ±95.0 49.7 ±94.2 51.5 ±94.3 APPROVED APPROVED FS ε = 0.03125 89.6 28.8 35.5 ±97.0 ε = 0.5 (ε ⋆ = 20) 64.3 ±89.3 47.7 ±95.1 76.6 ±66.1 ε = 1 (ε ⋆ = 30) 79.1 ±79.6 69.3 ±87.2 83.8 ±50.0 ε = 1.5 (ε ⋆ = 40) 84.7 ±70.7 89.2 ±45.9 92.5 ±27.8 ε = 2 (ε ⋆ = 50) 87.3 ±72.0 95.0 ±23.9 94.9 ±23.3 ε = 0.25 (ε ⋆ = 10) 49.6 ±94.9 55.7 ±90.5 51.7 ±94.0 99.9 52.8 ±94.2 SVHN CIFAR10 CIFAR100 SVHN CIFAR10 CIFAR100 SVHN FS MagNet CIFAR10 CIFAR100 ⋆ AUROC FPR AUROC FPR 84.8 ±48.5 30.3 ±98.6 40.9 ±96.0 100 ±0.0 ±0.0 100 ±0.0 76.6 ±66.6 40.6 ±98.6 41.3 ±96.2 100 ±0.0 ±0.0 93.0 ±27.7 94.6 ±26.1 FS MagNet MagNet 96.0 8.2 49.7 90.0 99.9 ±0.6 100 ±0.0 76.9 ±68.3 39.2 ±97.8 40.9 ±97.8 100 ±0.0 ±0.0 100 ±0.0 80.2 ±83.5 78.2 ±83.1 39.4 ±99.9 100 ±0.0 ±0.0 100 ±0.0 95.4 ±28.9 98.8 ±3.6 65.9 ±84.2 100 ±0.0 ±0.0 100 ±0.0 98.2 ±5.7 99.9 ±0.0 73.1 ±79.2 100 ±0.0 ±0.0 100 ±0.0 99.2 ±0.0 99.9 ±0.0 82.6 ±72.2 100 ±0.0 ±0.0 100 ±0.0 90.4 ±37.7 89.6 ±45.1 91.9 ±34.0 79.9 ±69.9 AUROC↑ FPR↓90% AUROC↑ FPR↓90% AUROC↑ FPR↓90% ε = 0.0625 99.1 1.9 93.8 11.9 49.8 89.9 AUROC↑ FPR↓90% AUROC↑ FPR↓90% AUROC↑ FPR↓90% ε = 0.125 99.9 0.0 89.2 47.1 49.9 89.6 99.8 ±0.8 ε = 0.5 (ε ⋆ = 20) 66.4 ±87.4 67.5 ±79.9 54.4 ±94.4 94.0 ±30.7 94.7 ±30.6 PGD ∞ PGD ∞ ε = 0.25 99.9 0.0 85.5 73.6 50.0 89.5 99.9 ±0.2 ε = 1 (ε ⋆ = 30) 92.1 ±29.6 83.1 ±54.4 59.1 ±93.6 98.8 ±6.5 99.4 ±3.3 ε = 0.03125 96.5 6.4 78.7 42.9 50.3 89.6 ε = 0.03125 75.4 51.5 76.0 74.8 50.2 89.7 ε = 0.5 99.9 0.0 83.6 82.2 50.1 89.4 100 ±0.1 ε = 0.0625 99.1 2.1 73.4 64.7 51.0 88.4 ε = 0.0625 88.1 26.0 68.9 75.0 50.6 88.9 BIM ε = 1.5 (ε ⋆ = 40) 98.0 ±5.9 91.7 ±32.8 67.2 ±92.4 98.9 ±4.4 100 ±0.0 99.9 ±0.2 ε = 0.125 99.7 0.8 71.8 68.6 52.9 85.5 ε = 0.125 93.3 14.9 65.5 75.0 52.1 86.5 ε = 0.03125 80.7 43.1 86.0 44.8 49.5 90.1 ε = 2 (ε ⋆ = 50) 99.4 ±1.6 96.2 ±16.1 77.5 ±88.4 99.5 ±2.4 100 ±0.0 100 ±0.1 ε = 0.25 99.8 0.5 70.9 70.0 54.3 83.4 ε = 0.25 94.4 12.8 64.3 75.0 53.0 84.9 ε = 0.0625 95.1 15.1 90.3 33.4 49.9 89.9 DeepFool * No ε 93.2 ±29.4 71.4 ±81.1 96.2 ±12.2 95.0 ±15.9 85.7 ±73.5 70.9 ±72.5 100 ±0.0 ±0.0 ε = 0.5 99.8 0.5 70.8 70.1 54.4 83.3 ε = 0.5 89.7 26.4 64.2 75.0 53.1 84.8 ε = 0.125 99.6 1.0 87.4 61.4 49.9 89.8 DeepFool * BIM BIM ε = 0.25 99.9 0.0 84.9 79.9 50.0 89.5 100 ±0.0 CW2 * ε = 0.01 61.9 ±92.8 51.0 ±94.0 31.7 ±98.9 42.7 ±88.8 45.5 ±93.4 35.9 ±96.8 100 ±0.0 ±0.0 No ε 58.2 ±93.4 55.9 ±92.3 73.0 ±72.6 90.8 ±29.0 79.7 ±62.5 ε = 0.03125 88.3 27.0 74.0 64.5 50.3 89.6 ε = 0.03125 63.1 72.9 67.6 75.0 50.2 89.7 ε = 0.5 99.9 0.0 83.9 82.5 50.2 89.1 100 ±0.0 CW 2 ε = 0.0625 97.1 5.4 70.2 72.3 50.7 88.9 ε = 0.0625 70.5 64.8 63.0 81.1 50.5 89.2 FGSM * ε = 0.125 99.0 2.2 70.0 72.2 51.8 87.2 ε = 0.125 87.2 28.1 62.1 82.7 51.3 87.8 ε = 0.03125 74.5 55.9 56.3 75.5 49.7 90.2 100 ±0.0 HOP ε = 0.01 61.8 ±92.0 56.0 ±91.2 64.6 ±90.0 92.7 ±30.9 89.2 ±44.6 ε = 0.25 99.7 0.7 70.7 70.5 53.6 84.4 ε = 0.25 93.2 15.4 63.7 75.4 52.5 85.7 ε = 0.0625 80.8 43.5 58.0 71.8 50.4 89.6 99.9 ±0.1 ε = 0.5 99.9 0.2 71.2 68.4 56.4 80.1 ε = 0.5 96.5 8.3 65.3 75.0 54.6 82.2 ε = 0.125 87.1 30.4 53.6 75.1 50.9 88.7
	ε = 0.1 Norm L∞ FGSM HOP ε = 0.1 ε = 0.03125 80.8 87.8 ±64.3 69.3 ±82.3 SVHN CIFAR10 CIFAR100 68.7 ±88.2 FGSM ε = 0.25 91.1 22.3 87.6 ±64.0 65.4 ±87.9 73.2 ±87.9 ε = 0.03125 78.1 69.5 48.1 61.9 62.5 94.3 ±16.8 87.1 ±73.5 SVHN CIFAR10 CIFAR100 SVHN CIFAR10 CIFAR100 71.2 ±81.7 100 ±0.0 ±0.0 100 ±0.0 48.1 78.8 52.6 86.2 93.8 ±22.5 95.6 ±19.6 75.2 38.8 51.9 88.1 51.0 88.8 ε = 0.5 94.4 15.2 50.9 74.2 60.7 72.1 99.8 ±0.5 ε = 0.0625 82.4 60.2 77.2 37.5 53.0 86.1 ε = 0.0625 86.5 33.0 61.3 61.4 52.1 86.8 SA
	PGD∞ ε = 0.03125 ε = 0.0625 Norm L ∞ ε = 0.125 ε = 0.25 ε = 0.5 PGD ∞ SA	93.7 ±25.5 86.1 ±47.1 98.7 ±3.7 94.6 ±25.2 SVHN CIFAR10 CIFAR100 47.4 ±95.2 48.8 ±93.9 ε = 0.125 93.1 16.6 90.4 24.0 54.8 50.0 95.6 ±27.3 99.6 ±0.0 SVHN 78.9 55.8 99.0 ±3.5 100 ±0.0 31.2 ε = 0.125 77.0 49.1 48.7 78.5 CIFAR10 CIFAR100 58.6 ±88.1 100 ±0.0 61.3 ±86.3 100 ±0.0 57.3 79.2 80.2 50.6 89.4 ε = 0.25 99.1 1.6 69.6 25.0 70.6 54.8 ε = 0.5 99.7 0.6 67.7 31.2 80.4 18.0 95.7 10.3 49.6 50.0 66.4 60.4 98.6 4.1 46.2 56.2 86.6 24.2 APPROVED FS MagNet No Norm SA AUROC FPR AUROC FPR AUROC FPR	±0.0 ±0.0	100 ±0.0 100 ±0.0
	ε = 0.125 ε = 0.25 ε = 0.3125 ε = 0.03125 ε = 0.0625 ε = 0.125 ε = 0.125 CW ∞ ε = 0.3125	99.7 ±0.8 99.6 ±2.0 99.4 ±3.2 99.3 ±1.7 91.3 ±34.2 97.7 ±11.2 99.0 ±3.7 99.1 ±3.5 99.9 ±0.2 99.0 ±4.4 99.9 ±0.2 99.9 ±0.3 ε = 0.125 98.2 CW ∞ ε = 0.3125 90.4 89.6 26.0 STA 81.7 42.2 No ε 80.2	50.3 ±92.5 74.5 ±77.0 77.9 ±75.6 53.1 ±93.2 55.9 ±91.8 61.5 ±89.9 3.3 30.6 63.3 67.0 42.5	100 ±0.0 100 ±0.0 100 ±0.0 97.4 ±13.8 99.5 ±2.0 99.9 ±0.3 72.0 78.8 50.0 54.9 100 ±0.0 100 ±0.0 100 ±0.0 99.4 ±2.3 99.3 ±3.6 99.8 ±0.7 25.0 37.5 82.6 50.0 50.0 89.8 53.0 77.5	68.3 ±83.5 79.4 ±76.1 83.3 ±72.5 99.9 ±0.4 100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.1 99.9 ±0.4 55.1 82.4 50.6 89.3 34.9 95.6	±0.0 ±0.0 ±0.0	100 ±0.0 100 ±0.0 100 ±0.0
	ε = 0.5 BIM ε = 0.25 ε = 0.3125 No Norm	98.7 ±6.2 99.9 ±0.2 99.9 ±0.2 No Norm 99.6 ±1.1 99.9 ±0.1 99.9 ±0.1 APPROVED AUROC↑ FPR↓90% AUROC↑ FPR↓90% AUROC↑ FPR↓90% 87.2 ±56.3 100 ±0.0 100 ±0.0 90.8 ±56.1 100 ±0.0 71.3 ±84.9 99.7 ±1.7 100 ±0.0 99.9 ±0.4 75.4 ±81.3 100 ±0.1 99.8 ±0.8 APPROVED FS MagNet FS MagNet 99.8 ±0.8 AUROC↑ FPR↓90% AUROC↑ FPR↓90% AUROC↑ FPR↓90%	±0.0	100 ±0.0
	ε = 0.03125 ε = 0.5 STA	91.5 ±32.6 79.7 ±63.3 99.9 ±0.2 99.9 ±0.1 STA	47.4 ±95.2 84.7 ±67.6		92.2 ±40.4 95.8 ±22.0 100 ±0.2 100 ±0.1	58.7 ±88.4 99.8 ±0.5 100 ±0.0	±0.0	100 ±0.0
	ε = 0.0625 BIM No ε	98.5 ±6.3 No ε 87.4	89.2 ±42.8 94.9 32.1	48.6 ±94.0 10.5 65.4	99.2 ±0.6 78.8 50.0	100 ±0.0 37.5 38.3	92.8	60.7 ±86.8 39.4	100 ±0.0 93.5	±0.0	100 ±0.0
	ε = 0.125 ε = 0.25 ε = 0.03125 ε = 0.3125 ε = 0.0625 ε = 0.5 ε = 0.125 ε = 0.25	99.6 ±1.6 99.7 ±1.2 99.0 ±3.0 89.3 ±41.4 95.8 ±21.4 98.6 ±4.7 99.4 ±3.2 99.1 ±3.5 99.8 ±0.3 97.9 ±9.2 99.2 ±4.3 99.7 ±1.0 99.9 ±0.2 99.7 ±0.9 99.9 ±0.2 99.9 ±0.1	49.9 ±92.8 74.3 ±78.3 53.0 ±93.3 78.3 ±73.8 55.8 ±91.9 85.4 ±64.1 61.4 ±90.3 71.2 ±85.1		99.9 ±0.0 100 ±0.0 96.7 ±21.2 96.8 ±14.4 100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.0 99.2 ±3.8 99.8 ±0.8 100 ±0.0 100 ±0.0 99.9 ±0.7 99.8 ±1.1 100 ±0.0 99.9 ±0.4	67.1 ±83.6 78.8 ±76.5 99.9 ±0.3 100 ±0.0 100 ±0.0 83.4 ±72.8 100 ±0.0 99.9 ±0.2 91.6 ±53.6 100 ±0.0 99.9 ±0.4 99.9 ±0.4	±0.0 ±0.0 ±0.0 ±0.0	100 ±0.0 100 ±0.0 100 ±0.0 100 ±0.0
	FGSM ε = 0.3125	99.9 ±0.2	99.9 ±0.1	75.3 ±81.0		99.9 ±0.6	99.8 ±1.0	99.9 ±0.4
	ε = 0.03125 ε = 0.5	95.4 ±19.4 86.6 ±49.0 99.9 ±0.2 99.9 ±0.1	48.9 ±94.4 84.9 ±67.7		87.8 ±44.0 24.9 ±99.7 99.9 ±0.2 99.7 ±1.4	40.0 ±96.5 99.9 ±0.1 100 ±0.0	±0.0	100 ±0.0
	ε = 0.0625 ε = 0.125 FGSM	99.2 ±3.8 99.7 ±0.2	97.3 ±12.7 99.4 ±2.9	47.3 ±94.2 47.2 ±92.0		90.7 ±31.3 81.4 ±75.4 92.7 ±22.6 93.0 ±46.3		38.5 ±96.9 36.0 ±97.9	100 ±0.0 100 ±0.0	±0.0 ±0.0	100 ±0.0 100 ±0.0
	ε = 0.25 ε = 0.03125	99.8 ±0.0 99.6 ±0.9 93.6 ±28.2 98.4 ±3.5	82.3 ±64.2 53.1 ±93.6		93.6 ±18.0 94.3 ±26.2 96.9 ±15.7 98.8 ±5.0	67.1 ±80.6 99.8 ±0.6 100 ±0.0	±0.0	100 ±0.0
	ε = 0.3125 ε = 0.0625	99.4 ±0.0 98.7 ±0.2	99.1 ±1.8 99.6 ±1.5	86.7 ±54.3 57.1 ±90.9		6.2 ±99.5 95.5 ±20.3 94.6 ±27.0 99.2 ±3.2	69.3 ±77.9 99.9 ±0.2 100 ±0.0	±0.0	100 ±0.0
	ε = 0.5 ε = 0.125	99.9 ±0.0 82.0 ±100	100 ±0.0 99.9 ±0.1	93.7 ±30.2 67.2 ±87.0		5.8 ±99.5 98.6 ±6.5	99.6 ±1.6 99.7 ±1.1	73.3 ±73.9 99.9 ±0.1 100 ±0.0	±0.0	100 ±0.0
	CW∞ * ε = 0.25	70.5 ±100	99.9 ±0.1	82.1 ±74.0		99.4 ±3.2	99.9 ±0.5	99.9 ±0.4
	ε = 0.3125 ε = 0.3125 ε = 0.5	85.9 ±51.3 71.5 ±80.5 76.1 ±100 99.9 ±0.1 88.7 ±97.6 99.9 ±0.1	78.1 ±81.3 86.9 ±64.5 94.1 ±36.9		90.0 ±32.7 79.5 ±76.0 98.5 ±7.2 100 ±0.1 99.8 ±0.7 100 ±0.0	67.7 ±79.7 99.9 ±0.4 100 ±0.0 99.9 ±0.2	±0.0	100 ±0.0
	SA ε= 0.125 CW ∞ *	92.3 ±31.8 93.1 ±45.6	84.6 ±82.0		93.0 ±22.5 90.0 ±73.4		68.9 ±78, 4	100 ±0.0	±0.0	100 ±0.0
	ε = 0.3125	67.9 ±90.9 64.6 ±89.9	70.0 ±85.3		90.5 ±33.0 90.5 ±40.9	99.9 ±0.1
	No norm SA No ε ε= 0.125	SVHN CIFAR10 CIFAR100 99.1 ±4.4 91.7 ±36.6 98.4 ±4.2 91.3 ±82.3 11.6 ±99.9 67.4 ±89.3	SVHN CIFAR10 CIFAR100 SVHN CIFAR10 CIFAR100 92.8 ±21.9 81.4 ±76.2 76.1 ±61.3 100 ±0.0 ±0.0 100 ±0.0 99.0 ±4.9 97.9 ±12.7 99.9 ±0.2
	No norm	SVHN CIFAR10 CIFAR100	SVHN	CIFAR10 CIFAR100
	No ε	99.8 ±6.4 93.8 ±20.2	92.9 ±24.7		98.5 ±0.4	80.3 ±57.1	100 ±0.0

Images available at https://github.com/yaircarmon/semisup-adv

The analysis can be extended to an arbitrary norm but it is somewhat simplified for the

2-norm case.

For experiments, we used 10 4 samples for each different class and a BFGS Quasi-Newton method for optimization. The initial value of θ is zero for both TRADES and FIRE. We do not report the result using ACE risk in Equation3.4 because in this setting we would obtain the trivial solution θ = 0, which is a minimizer of L ACE (θ).

Codes are available on GitHub at https://github.com/MarinePICOT/ Adversarial-Robustness-via-Fisher-Rao-Regularization

Images available at https://github.com/yaircarmon/semisup-adv

The AutoAttack code is available on https://github.com/fra31/auto-attack

Since we are interested in the dependence of q θ (•|x) with changes in input x and, particularly, its robustness to adversarial perturbations, we consider x as the "parameters" of the model over which the regularity conditions must be imposed.

LIST OF TABLES

List of Abbreviations

AC

LIST OF TABLES LIST OF TABLES

For the manuscript in Chapter 3, I designed the experiments, developed the code, wrote the manuscript, and helped with the theoretical part. F. Messina and P.

Piantanida provided the most significant part of the theoretical part and helped with the writing, while M. Boudiaf provided assistance with the experimental aspect and gave feedback on the manuscript, and I. BenAyed and F. Labeau provided feedback.

For the manuscripts in Chapter 4 and Chapter 5, I designed the experiments, provided the code, and wrote the manuscripts while F. Messina, P. Piantanida, and F.

Labeau provided scientific and writing suggestions.

For the manuscript in Chapter 6, G. Staerman provided the tool, P. Colombo, F.

Granese and I found the application, F. Granese and I designed the experiments, developed the code, and wrote the paper. P. Colombo and G. Staerman greatly helped with the writing. M. Romanelli, F. Messina, and P. Piantanida provided scientific suggestions and feedback on the writing part.

Finally, for the manuscript in Chapter 7, I developed the experiments and the code and wrote the manuscript. P. Colombo provided scientific and writing suggestions.

G. Staerman and N. Noiry, actively helped with the writing, while F. Messina and P. Piantanida provided feedback.

CHAPTER 4. ROBUST AUTOENCODER-BASED STATE ESTIMATION IN POWER SYSTEMS

where A is the attack vector. It has been showed that, not to be identified as faulty by the bad data detector, the attack should be a combination of the column vectors of H, i.e., A = Hc, for all c.

There exist different scenarios where the attacker can, for example, only tamper with a certain number of measurements either of its choosing or imposed by the system. The attack can also be random or targeted. All these scenarios impose specific values for the vector c. Intuitively, knowledge about the state estimator should give more power to the attacker. In the following, we present an attack method that uses that knowledge.

Review of the MILP Attack

The authors of KHEZRIMOTLAGH and collab. [2019] proposed an algorithm to create an undetected attack using the knowledge about the state estimator.

Their algorithm is composed of two parts. The first minimization problem allows the attacker to select which lines to overflow under the constraints that the generators' active powers are not modified, and the power flow behavior is respected.

The second minimization problem tries to minimize the attack vector's norm under the condition that perturbed lines (selected according to the resolution of the first minimization problem) are overflowed, meaning that their power flows are at most 20% above their maximum values. The attack vector should respect some conditions in order not to be detected. The power flow equations should be respected, and each component of the attack should not be too different from the value of the original observation. Finally, the total norm of the attack vector must be zero to satisfy the condition of zero-sum of all active power in the power system is met.

It is pretty easy to generate FDIAs that disrupt state estimation. Since state estimation is critical to ensure the proper functioning of Smart Grid systems, it is essential to ensure the reliability of the power system state estimator. Given that attacks are designed to remain undetected by the bad data detector, it appears to be fundamentally essential to improve the robustness of state estimators against all kinds of attacks, especially the most powerful ones. The following section aims at further developing this approach. required to successfully disrupt the state estimation [SANDBERG and collab., 2010].

Unfortunately this is a complex cardinality minimization problem which is hard to solve, particularly for large grids with several meters installed. In fact, the authors in HENDRICKX and collab. [2014] showed that this problem is an NP-hard problem and presented a relaxation hypothesis to solve it efficiently. A similar approach has been considered by the authors in SOU and collab. [2013]. The impact caused by FDIAs has been studied from different perspectives such as the energy deceiving aspect [LIANG and collab., 2016] or the economic losses [LIANG and collab., 2016;TAJER, 2017;XIE and collab., 2010].

Most of the research has been done according to the DC model assumption [DÁN and SANDBERG, 2010;KOSUT and collab., 2010;LIU and collab., 2011;PASQUALETTI and collab., 2011;SANDBERG and collab., 2010;XIE and collab., 2010;YUAN and collab., 2011], which leads to a simple linear model for the measurements as a function of the state. These works show precisely how FDIAs can bypass bad data detectors. A comprehensive review of the security problem under the DC-model assumption can be found in LIANG and collab. [2016]. A few works have focused on AC-based FDIAs [HUG and GIAMPAPA, 2012;JIN and collab., 2018;KEKATOS and collab., 2017;LIANG and collab., 2014;TEIXEIRA and collab., 2015;ZHU and GIANNAKIS, 2012], which considers the nonlinear relation between the state and the measurements.

On the other hand, joint cyber and physical attacks have also been studied [SOLTAN and collab., 2016], leading to the study of detection and recovery of information from line failure under DC and AC assumption [SOLTAN and collab., 2018;SOLTAN and ZUSSMAN, 2018].

Three main types of defense strategies have been developed in the literature.

The first one is the detection of the FDIAs [ KOSUT and collab., 2010;PASQUALETTI and collab., 2011], which can be used to discarding the compromised measurements.

The second one is based on encryption, authentication and key management [DÁN and SANDBERG, 2010;TEIXEIRA and collab., 2015;WANG and LU, 2013] in order to protect the communication channel between the meters and the control center.

Finally, Robust State Estimation (RSE) under the DC assumption has been created to develop state estimations that are robust against bad data [CELIK and ABUR, 1992;KOTIUGA and VIDYASAGAR, 1982;MILI and collab., 1994;ZHU and GIANNAKIS, 2012].

An AC-based solution to the RSE problem has been investigated by ZHANG and collab.

[2017], but it requires many relaxation assumptions. It should be noted that these defense strategies are complementary to each other and can be used simultaneously in practice. In this work, we focus on the latter approach to protect the grid against FDIAs. (5.9)

The conditional distribution p(x|y) will be modeled as Gaussian, i.e., q φ (x|y) = N (x; µ φ (y), Σ φ (y)) where the mappings µ φ and Σ φ are learnt through a deep neural network encoder. Notice that the decoder is fixed (i.e., it is not required to learn it) since p(y|x) = N (y; h(x), Σ z ).

The question now is how we should choose L attacked (φ) to ensure that clean and attacked states are similar. To answer this question, we build on our recent work [PICOT and collab., 2021] using tools from information geometry to introduce a risk function based on Fisher-Rao distance (FRD) [ATKINSON and MITCHELL, 1981;PINELE and collab., 2020] between two distributions. We now develop this idea to define L attacked (φ) for our robust VAE-based state estimator.

A new metric to improve robustness against attacks

As previously mentioned, the choice of the risk to ensure that clean and attacked states are similarly estimated is important. The output of the variational autoencoder being a probability distribution, one natural distance to consider is the FRD [ATKINSON and MITCHELL, 1981;PINELE and collab., 2020], which captures the distance between distributions over the underlying statistical manifold (defined by the encoder neural network in this case).

Consider an attack mechanism that outputs an attack observation y a given a clean one y. We can compare the predicted distributions q φ (x|y) and q φ (x|y a ). If we first assume that the problem is one-dimensional and that q φ (x|y) = N (x; µ, σ 2 ) and q φ (x|y a ) = N (x; μ, σ2 ), the Fisher-Rao distance can be written as follows [ATKINSON and MITCHELL, 1981]:

(5.10)

Now, if we assume that Σ φ is diagonal for simplicity, the FRD can be written as follows: (5.11) where q i φ is the i -th component of q φ . A natural choice for L attacked(φ) , using FRD, is:

(5.12)

Experiments

First, we consider a small well-known system -the IEEE 14-bus system -to assess the strength of our proposed attack/defense scheme and to choose all their related hyperparameters. Later, we test our robust state estimator on two additional systems, the IEEE 57 and 118 bus systems.

Set-up

AutoEncoder:

• Encoder: We use an encoder composed of 3 dense layers, with ReLU as activation functions.

• Decoder: We use a physical knowledge-based decoder, defined using Kirchhoff and Ohm laws.

• Considered losses: We need to choose 3 different losses: the attacker's objective ℓ(•, •; •), the state estimator reconstruction loss L (•, •), and the l p -norm of the bad data detector. We chose the l 1 error as L (•, •) since it gave us the best l 2reconstruction error. Therefore, we also use it for the bad data detector. To have the attacker-defender problems as symmetrical as possible, we use the Fisher-Rao distance as the attacker's objective, i.e., ℓ(x, xa ; φ) = d R q φ (•|y), q φ (•|y a ) .

Optimization set-up:

The Stochastic Gradient Descent optimizer is chosen to optimize our VAEs, with a learning rate of 10 -2 , decayed by 10 after 1/100 t h of the training and by 100 after 1/3 of the training; and a weight decay of 10 -5 .

Generation of the attacks:

• Deterministic attack: We use the method detailed in Algorithm 2 to corrupt samples, using δ = 0.1 and n = 10. In real-life scenarios, the attacker does not necessarily have access to all the meters. In the following, we will investigate the effect of the number of attacked meters on the attacks.

Influence of the number of attacked meters

In Figure 5.2a and Figure 5.2b, we present the influence of the percentage of attacked meters on our two attack mechanisms, using the λ value chosen previously. The attacker can attack the n% of meters with the highest l 1 -norm ∥a∥_1.

In this case, the two attack mechanisms behave similarly. Indeed, the more meters we can attack, the more powerful the attack is, but the more detectable it is. A good trade-off for both the deterministic and the random attack is 25% of attackable meters.

Effect of β on estimation performance

Now we study the influence of the β parameter to increase the robustness of the state estimator to corrupted observations.

1) Deterministic attack using PGD method:

We now train a robust state estimator, using the method detailed in Subsection 5. As expected, the β hyperparameter controls the trade-off between natural and attacked performances. For small values of β, the state estimator focuses on estimating the natural performances and does not significantly impact the noisy performances.

On the opposite, for large values of β, the state estimator focuses on approximating the natural and noisy samples in the same way but no longer tries to estimate the natural observations correctly. Therefore, there exists an optimal β value. In this attacked performances presented in Subsection 5.2.2 on Table 5.1. It is clear from this table that our proposed defense mechanism is able to protect our VAE against attack by rightfully estimating the states from both the clean and the noisy observations, whether they are created using our method or the state-of-the-art one. However, the state-of-the-art defense mechanism, using all the hyperparameters defined in the original paper, has more difficulty estimating the clean states and is more sensitive to attacks (i.e., they exhibit a difference of 7 degrees under attacks) than our proposed method.

To conclude, our method seems to be well-suited to craft robust state estimators, for which the state is estimated similarly for clean and attacked samples.

Conclusion

In this paper, we introduced novel methods to generate powerful false data injection attacks based on the knowledge of the estimator's training procedures. We have experimentally proven that these methods introduce significant state estimation errors. We also introduced a method to learn a robust state estimator using a geometric information distance, known as the Fisher-Rao distance, based on a min-max game between the estimator and the attacker. On multiple benchmarks of power networks, we have experimentally proven that it is possible to train a state estimator that improves its robustness against attacks while not significantly decreasing the performance on clean samples.

Abstract

Despite the widespread use of deep learning algorithms, vulnerability to adversarial attacks is still an issue limiting their use in critical applications.

Detecting these attacks is thus crucial to build reliable algorithms and has received increasing attention in the last few years. In this paper, we introduce the HalfspAce Mass dePth dEtectoR (HAMPER), a new method to detect adversarial examples by leveraging the concept of data depths, a statistical notion that provides center-outward ordering of points with respect to (w.r.t.) a probability distribution. In particular, the halfspace-mass (HM) depth exhibits attractive properties such as computational efficiency, which makes it a natural candidate for adversarial attack detection in high-dimensional spaces. Additionally, HM is non differentiable making it harder for attackers to directly attack HAMPER via gradient based-methods. We evaluate HAMPER in the context of supervised adversarial attacks detection across four benchmarks datasets. Overall, we empirically show that HAMPER consistently outperforms SOTA methods. In particular, the gains are 13.1% (29.0%) in terms of AUROC↑ (resp. FPR↓ 95% ) on SVHN, 14.6% (25.7%) on CIFAR10 and 22.6% (49.0%) on CIFAR100 compared to the best performing method.

Introduction

In most machine learning applications, deep models have achieved state-of-the-art performance. However, an important limitation to their widespread use in critical systems is their vulnerability to adversarial attacks [SZEGEDY and collab., 2014], i.e., the introduction of maliciously designed data crafted through minor adversarial ATTACK DETECTION 1 1 1 2 1 3 1 4 1 5 1 6 1 7 1 8 1 9 1 norm constraint with ε varying in {5, 10, 15, 20, 25, 30, 40}, for CIFAR10 and SVHN, and in {40, 500, 1000, 1500, 2000, 2500, 5000} for CIFAR100. Moreover, we generate perturbed examples using PGD under the L 2 -norm constraint with ε varying in {0. 125,0.25,0.3125,0.5,[START_REF] Introduction Tramer | On adaptive attacks to adversarial example defenses[END_REF]1.5,2},for CIFAR10 and SVHN,and in {5,10,15,20,30,40,50} for CIFAR100. In order to attack with PGD (L 1 and L 2 norm) the classifier trained on CIFAR100, we chose different epsilon values than those used for CIFAR10 and SVHN since the attacks generated with those epsilons were not able to fool the network (see Figure 6.3). CW attacks are generated under the L ∞ and L 2 constraint, with ε equals to 0.3125 and 0.01 respectively. Finally, we perturb samples using DF which is an L 2 attack without any constraint on ε. Concerning blackbox attack, SA is an L ∞norm attack ε = 0.125, HOP is an L 2 attack with ε = 0.1. Finally, STA is not concerned by a norm constraint nor a maximal perturbation, the attacker strength is limited in rotation (maximum of 30 o ) and in translation (maximum of 8 pixels).

Analyzing the networks' per class behavior under threats

In this section, we investigate the per-class behavior of the image classifier to motivate and justify the choice of the class dependency of the proposed aggregation procedure (see Equation 6.7).

Simulation. We examine the distribution of the adversarial examples w.r.t. the class predicted by the classifier. For this purpose, in Figure 6.2, we plot the distribution of adversarial samples per predicted class (N c ) as a function of the class (c).

Analysis. In SVHN, CIFAR10 and CIFAR100 natural images are balanced. However, on these datasets, the per-class distribution of the adversary is not uniform over the classes: in both SVHN (Figure 6.2a) and CIFAR10 (Figure 6.2b), classes 3 and 4 are overly represented on average, on the contrary of class 1 and 8 for SVHN and CIFAR10 respectively. Similarly, in CIFAR100 (Figure 6.2c), the classes 11, 24, 45, 68, 80, 81, and 97 are the overrepresented whilst the classes 37, [START_REF] Feinman | Detecting adversarial samples from artifacts[END_REF][START_REF] Haykin | Neural networks: a comprehensive foundation[END_REF]67,76 and 98 are the most underrepresented on average. Note that the diamond points in the plots denote the ATTACK DETECTION Table 6.1: Attack-aware performances on the three considered datasets -SVHN, CIFAR10 and CIFAR100 -of HAMPER AA detector together with the results of the SOTA detection methods: LID, and KD-BU, averaged over the L p -norm constraint. The best results among the detectors are shown in bold. The results are presented as AUROC↑ ±FPR ↓ 95% % and in terms of mean (µ) and standard deviation (σ).

Attack-aware scenario

Here, we study the performance of the different detectors in the attack-aware scenario and show that HAMPER AA outperforms existing detectors.

Global analysis. We present the attack-aware evaluation of HAMPER AA together with LID and KD-BU on SVHN, CIFAR10 and CIFAR100. In Table 6.1, we group the results from Table B.1 (that we relegate to Section B.3 due to space constriction) according to the attack-norm (e.g., L 1 , L 2 , L ∞ , No norm), and we express them in terms of the mean on the AUROC↑ and the mean on the FPR↓ 95% . We also report the average performances per dataset (Average) and over the datasets (Global).

In general, HAMPER AA outperforms the SOTA detectors by maintaining performance close to 100% AUROC↑ and 0% FPR↓ 95% on all the four datasets regardless of the attack-norm considered.

Performance analysis per ε. Overall, the results in Table B.1 show that the smaller the perturbation magnitude ε to craft the attack is, the more complex the attack detection. For example, the worst result of LID is with PGD1 and ε = 15 for SVHN and PGD1 with ε = 20 for CIFAR10 (AUROCs smaller than 50%). KD-BU exhibits the same attitude but for FGSM on SVHN, where it reaches its minimum (5.9% AUROC↑ and 99.5% FPR↓ 95% ) with ε = 0.5. Note that, the high value of the standard deviation in Table 6.1 could implies the detector is more susceptible to the ε changes. This is particularly true in the case of the L 1 -norm group since all the attacks considered are created with the same algorithm (PGD). On this regard, KD-BU turns out to be the detector most susceptible (e.g., on CIFAR10 its standard deviation on the FPR↓ 95% is 33.4 whilst the one of LID is 8.6 and the one of HAMPER AA is 0.0).

Performance analysis per type of threat. On average between the norm based ATTACK DETECTION Table 6.2: Blind-to-Attack detector performances on the three considered datasets -SVHN, CIFAR10, and CIFAR100 -of the HAMPER BA detector together with the results of the state-ofthe-art detection methods, i.e., NSS, averaged over the L p -norm constraint, along with the average and global performances. The best results among the detectors are shown in bold. The results are presented as AUROC↑% ±FPR ↓ 95% % and The results are presented as AUROC↑ ±FPR ↓ 95% % and in terms of mean (µ) and standard deviation (σ). attacks, LID and KD-BU more easily detects the L ∞ -norm attacks. Interestingly, with L ∞ both LID and KD-BU have the best performance on CIFAR10 whilst on SVHN and CIFAR100 the detectors have the best performance in the no norm case (cf. Table 6.1).

NSS

Consistently over the datasets, KD-BU poorly behaves on CW 2 as the AUROCs do not reach 50% (cf. Table B.1).

Summary. Table 6.1 suggests LID and KD-BU have similar behaviors on SVHN, whilst KD-BU improves on CIFAR10. A closer look at Table B.1 also suggests KD-BU has higher variance in the results w.r.t. LID on all the three datasets. Thus KD-BU performances are most affected by the perturbation magnitude to craft the adversarial examples. HAMPER AA , on the other side, is hereby confirmed as the best detector since it does not change its performances no matter the perturbation or the norm considered in the attacks.

Blind-to-attack scenario

Here, we study the performance of the different detectors in the blind-to-attack scenario and show that HAMPER BA outperforms existing detectors.

Global analysis.

We present the blind-to-attack evaluation of HAMPER BA together with NSS on SVHN, CIFAR10, CIFAR100. As in Section 6.5.2, in Table 6.2, we group the results from Table B.2 according to the attack-norm and we express them in terms of mean / standard deviation on the AUROC↑ and mean / standard deviation on the FPR↓ 95% . Moreover we report the average performances per dataset (Average) and over all the dataset (Global). On average, HAMPER BA outperforms NSS by 13.1(31.7) on

Chapter 6 Conclusion

In this work, we presented an efficient supervised detection method designed to defend DNNs against adversarial attacks. Given that attackers have different behavior with respect to the predicted class, we leverage this knowledge by constructing a class-wise detection method. In addition, we observed that using only the output of the final layers of the network to protect was enabling us to design an efficient detection method. We experimentally proved its efficiency compared to other state-of-the-art method on two scenario: attack-aware and blind-to-attack. Finally, we showed that our method was toughening the adaptive attacker's goal to fool both the network and the defense, compared to other detection methods.

In this work, we set ourselves in the supervised setting. In what follows, we will present our proposed framework to detect adversarial examples in the unsupervised case, focusing on protecting a widely overlooked classifier architecture: the Vision Transformers. [MADRY and collab., 2018] are two iterative extensions of the FGSM algorithm. Their main difference relies on the initialization of the attack algorithm, i.e., while BIM initializes the adversarial examples to the original samples PGD adds a random uniform noise on it. Although created for L ∞ -norm constraints, these three methods can be extended to any L p -norm constraint.

To overcome the absence of knowledge about the model to attack, Hop Skip Jump (HOP) [START_REF] D'amour | Underspecification presents challenges for credibility in modern machine learning[END_REF] tries to estimate the model's gradient through queries.

Square Attack (SA) [ANDRIUSHCHENKO and collab., 2020] is based on random searches for a perturbation. If the perturbation doesn't increase the attacker's objective, it is discarded. Finally, Spatial Transformation Attack (STA) [ENGSTROM and collab., 2019] rotates and translates the original samples to fool the model.

Review of detection methods

Defending methods against adversarial attacks have been widely studied for classical CNNs [ALAYRAC and collab., 2019;ATZMON and collab., 2019;CARMON and collab., 2019;HENDRYCKS and collab., 2019;HUANG and collab., 2020;MADRY and collab., 2018;RICE and collab., 2020;WANG and collab., 2019;WU and collab., 2020a;ZHANG and collab., 2019]. Whereas a few works have focused on studying the robustness of vision transformers to adversarial samples [ALDAHDOOH and collab., 2021a;BENZ and collab., 2021;MAHMOOD and collab., 2021]. Meanwhile, to protect adversarial attacks from disrupting DNNs' functioning, it is possible to craft detectors to ensure that the sample can be trusted.

Building a detector falls down to finding a scoring function s : R d → R and a threshold γ ∈ R to build a binary rule g : R d → {0, 1}. For a given test sample

If s is an anomaly score, g (x) = 0 implies that x is considered as 'natural', i.e., sampled from p XY , and g (x) = 1 implies that x is considered as 'adversarial', i.e., perturbed, and if s is a similarity score, the opposite decision is made.

A detection method can act on the model to be protected by modifying its training procedure using tools such as reverse cross-entropy [PANG and collab., 2018] or the rejection option [ALDAHDOOH and collab., 2021b;SOTGIU and collab., 2020]

Adversarial Attacks on Vision Transformers (ViT)

In the following, we provide insights on the behavior of vision transformers under the threat of adversarial attacks, along with a comparison to the classically used ResNets models. Remark. We discard the perturbed samples that do not fool the underlying classifier. Indeed, detecting a sample that does not perturb the classifier's functioning as either natural or adversarial is a valid answer.

Set-up

Attacks.

For the experiments, we will evaluate the different detection methods on the attacks presented in Subsection 7.2.1. Under L 1 -norm constraint, we craft attacks following PGD 1 scheme. For the L 2 -norm constraint, we consider PGD 2 , DF and HOP. Under L ∞ -norm constraint, we study PGD ∞ , BIM and FGSM attacks, CW ∞ and SA. Finally, we create STA attacks, which are not subject to a norm constraint.

The values of the maximal allowed perturbation are discussed in the next section. Given that the variety of attacks comes from choosing the L p -norm constraint and the maximal allowed L p -norm perturbation ε, it is crucial to select them carefully.

Adversarial attacks and defense mechanisms have been widely studied for classical convolutional networks, particularly for ResNets [GOODFELLOW and collab., 2014;MADRY and collab., 2018;MENG and CHEN, 2017;MOOSAVI-DEZFOOLI and collab., 2016;XU and collab., 2018;ZHANG and collab., 2019]. Hence, choosing the maximally allowed perturbation ε for ViT comes naturally from comparing the success attack rates between attacks on ResNets and ViTs.

In

Locating the relevant information

In the previous section, we saw that the attacks behave differently w.r.t. the classifier on which they are perpetrated. We now continue this investigation by looking at the differences between the two models from the depth scores' perspective. In this framework, we define the layer to have relevant information when the difference between the depth score on the naturals and the depth score on the adversarial is significant. Indeed, the higher the differ- ence, the more evident the shift between the distributions of the natural and the adversarial induced by the depth score will be, and hence the easier it will be to find a threshold that distinguishes natural from adversarial samples. We start by computing layer per layer the differences between the IRW depth on the natural samples (D IRW ( f ℓ θ (x); p ℓ ŷ )) and on the adversarial samples (D IRW ( f ℓ θ (x ′ ); p ℓ ŷ )) both for ViT and for ResNet18. In Figure 7.2, we plot the mean and standard deviation for each layer and each network.The diamond points represent the outliers. Figure 7.2 shows that the information about whether a sample is natural or adversarial, based on the study of the IRW depth, is significantly spread across the ResNet18 model: in each layer, the values range between 0 and 0.06. On the contrary, on ViT, this information is concentrated in the logit layer, where the values range between 0.05 and 0.2 while the values range from 0 to 0.05 for the other layers. To summarize, while relevant

Experiments

Results

Performances of APPROVED compared to other unsupervised detection methods.

In Table C.3, Table C.4, and Table C.5 relegated to Section C.5 , we report the detailed results for each considered detection method under the threat of each attack mechanism, L p -norm constraint and maximum perturbation ε. In Table 7.2, we report the averaged AUROC↑ and FPR↓ 90% on each of the considered L p -norm, along with the global average for each detector, on CIFAR10, CIFAR100, and Tiny ImageNet. Overall, APPROVED shows better results than the SOTA detection methods. On CIFAR10, AP-PROVED creates an increase of AUROC↑ of 18.9% and a decrease of FPR↓ 90% of 30.7% compared to the best performing state-of-the-art detector, i.e., FS. On CIFAR100, the improvements are 17.1% and 24.8%, respectively, while they are 18.0% and

Part II Conclusion

Part II focused on the second research question: how can we craft an efficient and effective detection method based on simple tools? This research question has been addressed in two different ways:

1. In the supervised case. We tackled the problem of crafting a detection method to distinguish between natural and attacked samples. We decided to use the data-depths, simple statistical tools providing a centeroutward ordering of points w.r.t. to a reference distribution. We leveraged the class-wise information present in the model's different layers to protect and the knowledge about the potential threats to craft an efficient detector. We experimentally showed the superiority of our proposed method.

2. In the unsupervised scenario. We also proposed a method to discard attacks in the case where no information about the threats is available. We made use of the information extracted by the data-depths as well as the internal structure of the Vision Transformer, which behaves differently than the classically used ResNet, to construct an efficient method to detect adversarial examples. Experimentally, we proved that our method outperforms other state-of-the-art detectors and complicates the attacker's job.

Part II was dedicated to increasing our trust in the deep models' inputs by ensuring they come from a clean environment.

In Section 7.6, we will provide the reader with concluding remarks about the work we produced during the last four years. We will also put our contributions into perspective and discuss some of the limitations as well as the perspectives of the different parts of our work.

Chapter 8 This manuscript presented the research conducted over the last four years on improving the reliability of neural networks under attack. In this chapter, we will provide some concluding remarks.

Discussion of Findings

Research problem

In this thesis, the work presented focused on two central axes.

In the first axis, we focused on answering the following research question: (Q1)

"How can we use the internal structure of DNN's output space to improve its robustness?".

In the second axis, we focused on providing answers to the following research question: (Q2) "How can we craft an efficient and effective detection method based on simple tools".

In the following, we will briefly summarize our key results.

Key results

Our proposed solution for (Q1).

In Part I, we present our solution to the first research question for two different applications: computer vision and Smart Grid systems. We proposed a method based on the regularization of the training loss by an information-geometric measure, namely the Fisher-Rao distance, which computes the geodesic distance between two probability distributions on the underlying probabilistic manifold that the outputs of a DNN form. We connected it to standard metrics and experimentally verified that using the Fisher-Rao distance as a regularizer achieved more Pareto-optimum points than the standard regularization measure. On image data, we experimentally proved that our method consistently improved the DNNs robustness to adversarial attacks.

We later adapted our method in the context of the Smart Grids. Due to monitoring and service necessity, intelligent power grids must rely on cyber-component, making them vulnerable to cyber-attacks. State estimators, necessary components to monitor the grid, are known to be particularly sensitive to stealth attacks. During this work, we presented a DNN-based, more specifically a Variational AutoEncoderbased, solution to improve the robustness of state estimators in the context of Smart Grid systems to stealth attacks.

Our proposed solution for (Q2).

In Part II, we presented two solutions to try and answer the second research question suited to two different scenarios: the supervised and unsupervised cases.

In both cases, we used the data-depths, simple statistical tools that provide a centeroutward ordering of a point with respect to a reference distribution. For a given new input, the data-depth will measure how deep it is in the reference distribution. In the supervised case, we used a specific depth, the halfspace-mass depth, and leveraged the class-wise information contained in the different layers of a given model, to CHAPTER 8. DISCUSSION OF FINDINGS, LIMITATIONS, POTENTIAL FUTURE WORKS AND GLOBAL SUMMARY craft an efficient detection method. We experimentally showed that our method consistently improved the state-of-the-art method on different kinds of attacks. In the unsupervised case, we used another data-depth, called the Integrated Rank-Weighted depth, and leveraged the particular structure of Vision Transformers to build an efficient method to detect adversarial samples. Experimentally, in addition to proving the superiority of our approach compared to other state-of-the-art methods, we showed that using our method would toughen the attacker's job at attacking both the targeted classifier and the defense.

Comparison between our work and previous state-of-the-art methods.

Globally, our proposed methods show an improvement compared to previous stateof-the-art methods.

Using the Fisher-Rao distance to improve defenses. Regarding robustness as a defense against adversarial attacks, theoretically, we have shown that the Kullback-Leibler divergence is a surrogate of the Fisher-Rao distance. In addition, compared to the use of the Kullback-Leibler divergence [ZHANG and collab., 2019] as a robust regularizer, using the Fisher-Rao distance to force clean and adversarial predictions to be similar experimentally suggests that it is possible to achieve better trade-offs between natural and adversarial accuracies, both on simple data and on more realistic ones. Finally, as for the Kullback-Leibler divergence, combining the Fisher-Rao regularizer with other methods to improve robustness is possible.

Protecting against False Data Injection Attacks in Smart Grid systems. In the case of building robust state estimators to enhance the protection of Smart Grids systems, using a Variational AutoEncoder, combined with the online generation of false data injection attacks (FDIAs) [LIU and collab., 2011] and our proposed robust loss, is allowing us to build strong state estimators, with good robust qualities. In the case of DC assumptions, our method can rightfully estimate the state even in the case of an attack. In the case of AC assumptions, our method clearly outperforms the LASSO solution proposed by JIN and collab. [2019], which is one the most recently proposed method to improve state estimators' defenses against FDIAs. Are the attack and defense problems underspecified? Recently, the problem of underspecification and its implications has been formalized by D'AMOUR and collab.

[2020]; TENEY and collab. [2022]. A problem is considered underspecified if there exist many different solutions that solve it without any change of results on the main task but exhibiting widely different characteristics. Most DL problems are, in fact, underspecified, and it could cause problems with the reliability of machine learning, as every solution can behave differently when deployed in real-world applications where the main task can slightly vary from the training one. We believe the attack problem is, in fact, underspecified, as attacking two different models trained on the same task can result in vastly different attacked samples. Moreover, we also believe the defense problem is underspecified, as the performances of the defenses on attacks that have never been seen before can widely change depending on the chosen defense [D'AMOUR and collab., 2020][Section 5]. We, therefore, believe it would be interesting to test attack and defense mechanisms under the scope of underspecification.

A.1 Comparison between the Fisher-Rao distance and the KL divergence on real data

In order to confirm on real data the difference between the Kullback-Leibler divergence and the Fisher-Rao distance, we reproduce the simulation presented on Figure 3.6 based on the CIFAR-10 dataset. We trained multiple classifier using both TRADES and FIRE methods, varying λ ∈ [0, 50]. We use the dataset CIFAR-10 with a ResNet-18 for the model, and train the models for 100 epochs. The optimizer is the Stochastic Gradient Descent (SGD) with a learning rate of 0.01, with a decay of 0.1 at epoch 75, 90 and 100. We also use a weight decay of 5.10 -4 , a momentum of 0.9. The results are averaged over 2 tries.

We present the results on Figure A.1. On the left figure, we observe the entire curve, and on the right we zoomed on the zone of interest. We can see that Fisher-Rao distance presents a better trade-off between natural and adversairal accuracies than the Kullback-Leibler divergence on real data, confirming the improvements presented on Figure 3.6. For natural accuracies below 80%, the Fisher-Rao distance and the Kullback-Leibler divergence seem to behave quite similarly. For natural accuracies above 80%, which is actually the zone of interest, the improvement caused by the use of the Fisher-Rao distance seem to be quite consistent among all the training points. At fixed adversarial accuracies, the Fisher-Rao distance can increase the results by up to 1% of natural accuracies. Note that, in this simulation the Kullback-Leibler divergence can achieve a better adversarial accuracy that the Fisher-Rao distance (44.21% compared to 43.57%), but with a cost of sighly more than 2.5% for natural accuracies (78.69% compared to 81.23%). The Fisher-Rao divergence therefore achieves a better trade-off between natural and adversarial accuracies than the Kullback-Leibler divergence.

Accuracy-Robustness trade-offs in the Gaussian case

As we defined the FIRE risk, it is possible to define the Hellinger risk as : L HEL (θ) . = E p(x,y) max

x ′ :∥x ′ -x∥ p ≤ε log q θ (y|x)

where H(q θ , q ′ θ ) is defined in Equation A.1. In Figure 3.6b, we present the solution of the (local) Empirical Risk Minimization (ERM) for the Hellinger risk function as defined in Equation A.3 for different values of λ. As can be observed, the curve obtained for all pairs of (1 -P e (θ), 1 -P ′ e (θ)) covers about half of the Pareto-optimal points while the curve of all solutions for (1 -P e (θ), 1 -P ′ e (θ)) corresponding to FIRE risk (see Equation 3.7) covers all the Pareto-optimal points. This simulation shows that even if the Hellinger distance and the Fisher-Rao distance are theoretically equivalent, their dynamics in training are actually quite different. In addition, FRD seems to be better suited for training in the Gaussian case.

A.2.2 Comparison based on real data

We now study the empirical difference between those two distances on real datasets.

We therefore perform the same simulations as those provided in Section 3.5.2 using the squared Hellinger distance between the natural and the adversarial predictions as the robustness regularizer. The results are summarized in Table A.1. Even though using the Hellinger distance as a robust regularizer performs better than using the Kullback-Leibler divergence, it still performs worse that the Fisher-Rao distance. In the case of real data, FRD appears to be better suited for training than the Hellinger distance.

In conclusion, FRD provides a better regularization objective than the Hellinger distance.

A.3 Proof of Proposition 1 and Theorem 3 A.3.1 Proof of Proposition 1

Notice that the FRD in Equation 3.9 can be written as follows:

where we have defined the function f : R → R as f (z) .

= arctan(e z/2 ). Notice that f is a smooth function. Using the mean value theorem, we have

where f ′ (z) = e z/2 /(2e z +2) is the derivative of f and c is a point in the (open) interval with endpoints h θ (x) and h θ (x ′ ). Notice that 0 ≤ f ′ (z) ≤ 1/4 for any z. Then, we have

(A.7)

A.3.2 Proof of Theorem 3

Consider first the Hellinger distance between two distributions q and q ′ over Y , defined as follows:

Using Equation 3.15, we readily obtain the following relation: .8) We now use the following inequality relating the Hellinger distance and the KL divergence [TSYBAKOV, 2008, Lemma 2.4]:

H 2 (q, q ′ ) ≤ KL(q, q ′ ). (A.9)

Using the relation Equation A.8, we obtain the desired bound in Equation 3.16. The second-order approximation (cf. Equation 3.17) follows directly from CALIN and UDRI ŞTE [2014][Theorem 4.4.5].

A.4 References

B.1 Approximation algorithms

In this part, we present algorithms, originally proposed in CHEN and collab. [2015] and adapted to our problem, that are used in steps 3 and 4 in HAMPER (see Algorithm and MOSLER [1997]; [START_REF] Liu | On a notion of data depth based on random simplices[END_REF]; RAMSAY and collab. [2019]; STAERMAN and collab.

[2021]; ZUO [2003]. While these differ in theoretical and practically related properties such as robustness or computational complexity, several postulates have been developed throughout the recent decades the "good" depth function should satisfy.

Such properties have been thoroughly investigated in [START_REF] Liu | On a notion of data depth based on random simplices[END_REF]; ZUO and SERFLING [2000] and DYCKERHOFF [2004] with slightly different sets of axioms (or postulates)

to be satisfied by a depth function. They are recalled below. We trained two different models: a ViT, and a ResNet18. The ResNet18 has been trained on 100 epochs, with a Stochastic Gradient Descent (SGD) optimizer, with a learning rate of 0.1, a momentum of 0.9, and a weight decay of 10

C.2 Approximation Algorithm

In this appendix, we display the algorithm used to compute the IRW depth (see Algorithm 8). is linear in all of its parameters.

Remarks.

Given that the algorithm is linear in all its parameters, computing the IRW depth can be scaled to any datasets. Note that the IRW data depth makes no assumption on the training distribution. In line 3 of Algorithm 8, "rank values" consists in ranking the elements of the projection of each input on U. This is achieved by a sorting algorithm. This step allows us to define an ordering of the projected inputs, which is used to compute the final depth score.

C.3 Time and Computational Requirements

C.3.1 To generate attacks

We here present the computational requirements to generate the attacks on the transformer, along with the required time to generate them. We use the Adversarial-Robustness Toolbox (ART) [NICOLAE and collab., 2018] to generate the attacks.

C.3.2 To deploy detectors

This section presents the computational requirements, along with the time needed to deploy each of the studied detection methods on CIFAR10. For FS and MagNet, we use the codes available at https://github.com/aldahdooh/detectors_review.

C.4 Success Rates of Attacks on CIFAR10

We here report the success rate per attack for all the different threat mechanisms (i.e., PGD 1 , PGD 2 , PGD ∞ , BIM, FGSM, CW ∞ , SA, STA, DF and HOP). In orange are the attack performances on ViT while the ones on ResNet are in green (see Section 7.4

for a detailed analysis). In Table C Appendix D

Résumé Étendu

Au cours de la dernière décennie, l'apprentissage profond a été à l'origine de percées dans de nombreux domaines différents, tels que le traitement du langage naturel, la