Rafael Pinot

Laurent Meunier

Alexandre Araujo

Hisashi Kashima

Florian Yger

Cédric Gouy-Pailler

Jamal Atif

Theoretical evidence for adversarial robustness through randomization

This paper investigates the theory of robustness against adversarial attacks. It focuses on the family of randomization techniques that consist in injecting noise in the network at inference time. These techniques have proven effective in many contexts, but lack theoretical arguments. We close this gap by presenting a theoretical analysis of these approaches, hence explaining why they perform well in practice. More precisely, we make two new contributions. The first one relates the randomization rate to robustness to adversarial attacks. This result applies for the general family of exponential distributions, and thus extends and unifies the previous approaches. The second contribution consists in devising a new upper bound on the adversarial generalization gap of randomized neural networks. We support our theoretical claims with a set of experiments.

Introduction

Adversarial attacks are some of the most puzzling and burning issues in modern machine learning. An adversarial attack refers to a small, imperceptible change of an input maliciously designed to fool the result of a machine learning algorithm. Since the seminal work of [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] exhibiting this intriguing phenomenon in the context of deep learning, a wealth of results have been published on designing attacks [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF][START_REF] Moosavi-Dezfooli | Deepfool: a simple and accurate method to fool deep neural networks[END_REF][START_REF] Kurakin | Adversarial examples in the physical world[END_REF][START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Moosavi-Dezfooli | Universal adversarial perturbations[END_REF] and defenses [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Papernot | Distillation as a defense to adversarial perturbations against deep neural networks[END_REF][START_REF] Guo | Countering adversarial images using input transformations[END_REF][START_REF] Meng | Magnet: a two-pronged defense against adversarial examples[END_REF][START_REF] Samangouei | Defense-GAN: Protecting classifiers against adversarial attacks using generative models[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]), or on trying to understand the very nature of this phenomenon [START_REF] Fawzi | Empirical study of the topology and geometry of deep networks[END_REF][START_REF] Simon-Gabriel | Adversarial vulnerability of neural networks increases with input dimension[END_REF][START_REF] Fawzi | Adversarial vulnerability for any classifier[END_REF][START_REF] Fawzi | Robustness of classifiers: from adversarial to random noise[END_REF]. Most methods remain unsuccessful to defend against powerful adversaries [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF][START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF]. Among the defense strategies, randomization has proven effective in some contexts. It consists in injecting random noise (both during training and inference phases) inside the network architecture, i.e. at a given layer of the network. Noise can be drawn either from Gaussian [START_REF] Liu | Towards robust neural networks via random self-ensemble[END_REF][START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF][START_REF] Rakin | Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack[END_REF], Laplace [START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF], Uniform [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF], or Multinomial [START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] distributions. Remarkably, most of the considered distributions belong to the Exponential family. Albeit these significant efforts, several theoretical questions remain unanswered. Among these, we tackle the following, for which we provide principled and theoretically-founded answers:

Q1: To what extent does a noise drawn from the Exponential family preserve robustness (in a sense to be defined) to adversarial attacks?

A1: We introduce a definition of robustness to adversarial attacks that is suitable to the randomization defense mechanism. As this mechanism can be described as a non-deterministic querying process, called probabilistic mapping in the sequel, we propose a formal definition of robustness relying on a metric/divergence between probability measures. A key question arises then about the appropriate metric/divergence for our context. This requires tools for comparing divergences w.r.t. the introduced robustness definition. Renyi divergence turned out to be a measure of choice, since it satisfies most of the desired properties (coherence, strength, and computational tractability). Finally, thanks to the existing links between the Renyi divergence and the Exponential family, we were able to prove that methods based on noise injection from the Exponential family ensures robustness to adversarial examples (cf Theorem 1) .

Q2: Can we guarantee a good accuracy under attack for classifiers defended with this kind of noise?

A2: We present an upper bound on the drop of accuracy (under attack) of the methods defended with noise drawn from the Exponential family (cf. Theorem 2). Then, we illustrate this result by training different randomized models with Laplace and Gaussian distributions on CIFAR10/CIFAR100. These experiments highlight the trade-off between accuracy and robustness that depends on the amount of noise one injects in the network. Our theoretical and experimental conclusion is that randomized defenses are competitive (with the current state-of-the-art [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]) given the intensity of noise injected in the network.

Outline of the paper: We present in Section 2 the related work on randomized defenses to adversarial examples. Section 3 introduces the definition of robustness relying on a metric/divergence between probability measures, and discusses the key role of the Renyi divergence. We state in Section 4 our main results on the robustness and accuracy of Exponential family-based defenses. Section 5 presents extensive experiments supporting our theoretical findings. Section 6 provides concluding remarks.

Related works

Injecting noise into algorithms to improve their robustness has been used for ages in detection and signal processing tasks [START_REF] Zozor | Stochastic resonance in discrete time nonlinear AR(1) models[END_REF][START_REF] Chapeau-Blondeau | Noise-enhanced performance for an optimal bayesian estimator[END_REF][START_REF] Mitaim | Adaptive stochastic resonance[END_REF]. It has also been extensively studied in several machine learning and optimization fields, e.g. robust optimization [START_REF] Ben-Tal | Robust optimization[END_REF] and data augmentation techniques [START_REF] Perez | The effectiveness of data augmentation in image classification using deep learning[END_REF]. Recently, noise injection techniques have been adopted by the adversarial defense community, especially for neural networks, with very promising results. Randomization techniques are generally oriented towards one of the following objectives: experimental robustness or provable robustness.

Experimental robustness: The first technique explicitly using randomization at inference time as a defense appeared during the 2017 NIPS defense challenge [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF]. This method uniformly samples over geometric transformations of the image to select a substitute image to feed the network. Then [START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF] proposed to use stochastic activation pruning based on a multinomial distribution for adversarial defense. Several papers [START_REF] Liu | Towards robust neural networks via random self-ensemble[END_REF][START_REF] Rakin | Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack[END_REF] propose to inject Gaussian noise directly on the activation of selected layers both at training and inference time. While these works hypothesize that noise injection makes the network robust to adversarial perturbations, they do not provide any formal justification on the nature of the noise they use or on the loss of accuracy/robustness of the network.

Provable robustness: In [START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF], the authors proposed a randomization method by exploiting the link between differential privacy [START_REF] Dwork | The algorithmic foundations of differential privacy[END_REF] and adversarial robustness. Their framework, called "randomized smoothing"1 , inherits some theoretical results from the differential privacy community allowing them to evaluate the level of accuracy under attack of their method. Initial results from [START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF] have been refined in [START_REF] Li | Second-order adversarial attack and certifiable robustness[END_REF], and [START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF]. Our work belongs to this line of research. However, our framework does not treat exactly the same class of defenses. Notably, we provide theoretical arguments supporting the defense strategy based on randomization techniques relying on the exponential family, and derive a new bound on the adversarial generalization gap, which completes the results obtained so far on certified robustness. Furthermore, our focus is on the network randomized by noise injection, "randomized smoothing" instead uses this network to create a new classifier robust to attacks.

Since the initial discovery of adversarial examples, a wealth of non randomized defense approaches have also been proposed, inspired by various machine learning domains such as adversarial training [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF], image reconstruction [START_REF] Meng | Magnet: a two-pronged defense against adversarial examples[END_REF][START_REF] Samangouei | Defense-GAN: Protecting classifiers against adversarial attacks using generative models[END_REF] or robust learning [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]. Even if these methods have their own merits, a thorough evaluation made by [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF] shows that most defenses can be easily broken with known powerful attacks [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF][START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Chen | Ead: elastic-net attacks to deep neural networks via adversarial examples[END_REF]. Adversarial training, which consists in training a model directly on adversarial examples, came out as the best defense in average. Defense based on randomization could be overcome by the Expectation Over Transformation technique proposed by [START_REF] Athalye | Synthesizing robust adversarial examples[END_REF] which consists in taking the expectation over the network to craft the perturbation. In this paper, to ensure that our results are not biased by obfuscated gradients, we follow the principles of [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF][START_REF] Carlini | On evaluating adversarial robustness[END_REF] and evaluate our randomized networks with this technique. We show that randomized defenses are still competitive given the intensity of noise injected in the network.

3 General definitions of risk and robustness

Risk, robustness and probabilistic mappings

Let us consider two spaces X (with norm . X ), and Y. We consider the classification task that seeks a hypothesis (classifier) h : X → Y minimizing the risk of h w.r.t. some ground-truth distribution D over X × Y. The risk of h w.r.t D is defined as

Risk(h) := E (x,y)∼D [1 (h(x) = y)] .
Given a classifier h : X → Y, and some input x ∈ X with true label y true ∈ Y, to generate an adversarial example, the adversary seeks a τ such that h(x + τ ) = y true , with some budget α over the perturbation (i.e with τ X ≤ α). α represents the maximum amount of perturbation one can add to x without being spotted (the perturbation remains humanly imperceptible). The overall goal of the adversary is to find a perturbation crafting strategy that both maximizes the risk of h, and keeps the values of τ X small. To measure this risk "under attack" we define the notion of adversarial α-radius risk of h w.r.t. D as follows

Risk α (h) := E (x,y)∼D sup τ X ≤α 1 (h(x + τ ) = y) .
In practice, the adversary does not have any access to the ground-truth distribution. The literature proposed several surrogate versions of Risk α (h) (see [START_REF] Diochnos | Adversarial risk and robustness: General definitions and implications for the uniform distribution[END_REF] for more details) to overcome this issue. We focus our analysis on the one used in e.g [START_REF] Szegedy | Intriguing properties of neural networks[END_REF], or [START_REF] Fawzi | Adversarial vulnerability for any classifier[END_REF] denoted α-radius prediction-change risk of h w.r.t. D X (marginal of D for X ), and defined as

PC-Risk α (h) := P x∼D X [∃τ ∈ B(α) s.t. h(x + τ ) = h(x)]
where for any α ≥ 0, B(α) := {τ ∈ X s.t. τ X ≤ α} .

As we will inject some noise in our classifier in order to defend against adversarial attacks, we need to introduce the notion of "probabilistic mapping". Let Y be the output space, and F Y a σ-algebra over Y. Let us also denote P(Y) the set of probability measures over (Y, F Y ). Definition 1 (Probabilistic mapping). Let X be an arbitrary space, and (Y, F Y ) a measurable space.

A probabilistic mapping from X to Y is a mapping M : X → P(Y). To obtain a numerical output out of this probabilistic mapping, one needs to sample y according to M(x).

This definition does not depend on the nature of Y as long as (Y, F Y ) is measurable. In that sense, Y could be either the label space or any intermediate space corresponding to the output of an arbitrary hidden layer of a neural network. Moreover, any mapping can be considered as a probabilistic mapping, whether it explicitly injects noise (as in [START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF][START_REF] Rakin | Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack[END_REF][START_REF] Dhillon | Stochastic activation pruning for robust adversarial defense[END_REF]) or not. In fact, any deterministic mapping can be considered as a probabilistic mapping, since it can be characterized by a Dirac measure. Accordingly, the definition of a probabilistic mapping is fully general and equally treats networks with or without noise injection. There exists no definition of robustness against adversarial attacks that comply with the notion of probabilistic mappings. We settle that by generalizing the notion of prediction-change risk initially introduced in [START_REF] Diochnos | Adversarial risk and robustness: General definitions and implications for the uniform distribution[END_REF] for deterministic classifiers. Let M be a probabilistic mapping from X to Y, and d P(Y) some metric/divergence on P(Y). We define the (α, )-radius prediction-change risk of M w.r.t. D X and d P(Y) as

PC-Risk α (M, ) := P x∼D X ∃τ ∈ B(α) s.t. d P(Y) (M(x + τ ), M(x)) > .
These three generalized notions allow us to analyze noise injection defense mechanisms (Theorems 1, and 2). We can also define adversarial robustness (and later adversarial gap) thanks to these notions. Definition 2 (Adversarial robustness). Let d P(Y) be a metric/divergence on P(Y). The probabilistic mapping M is said to be d

P(Y) -(α, , γ) robust if PC-Risk α (M, ) ≤ γ.
It is difficult in general to show that a classifier is d P(Y) -(α, , γ) robust. However, we can derive some bounds for particular divergences that will ensure robustness up to a certain level (Theorem 1).

It is worth noting that our definition of robustness depends on the considered metric/divergence between probability measures. Lemma 1 gives some insights on the monotony of the robustness according to the parameters, and the probability metric/divergence at hand.

M is d 2 -(α, , γ)-robust =⇒ M is d 1 -(α, φ( ), γ)-robust
As suggested in Definition 2 and Lemma 1, any given choice of metric/divergence will instantiate a particular notion of adversarial robustness and it should be carefully selected.

On the choice of the metric/divergence for robustness

The aforementioned formulation naturally raises the question of the choice of the metric used to defend against adversarial attacks. The main notions that govern the selection of an appropriate metric/divergence are coherence, strength, and computational tractability. A metric/divergence is said to be coherent if it naturally fits the task at hand (e.g. classification tasks are intrinsically linked to discrete/trivial metrics, conversely to regression tasks). The strength of a metric/divergence refers to its ability to cover (dominate) a wide class of others in the sense of Lemma 1. In the following, we will focus on both the total variation metric and the Renyi divergence, that we consider as respectively the most coherent with the classification task using probabilistic mappings, and the strongest divergence. We first discuss how total variation metric is coherent with randomized classifiers but suffers from computational issues. Hopefully, the Renyi divergence provides good guarantees about adversarial robustness, enjoys nice computational properties, in particular when considering Exponential family distributions, and is strong enough to dominate a wide range of metrics/divergences including total variation.

Let µ 1 and µ 2 be two measures in P(Y), both dominated by a third measure ν. The trivial distance

d T (µ 1 , µ 2 ) := 1 (µ 1 = µ 2 )
is the simplest distance one can define between µ 1 and µ 2 . In the deterministic case, it is straightforward to compute (since the numerical output of the algorithm characterizes its associated measure), but this is not the case in general. In fact one might not have access to the true distribution of the mapping, but just to the numerical outputs. Therefore, one needs to consider more sophisticated metrics/divergences, such as the total variation distance

d T V (µ 1 , µ 2 ) := sup Y ∈F Y |µ 1 (Y ) -µ 2 (Y )|.
The total variation distance is one of the most broadly used probability metrics. It admits several very simple interpretations, and is a very useful tool in many mathematical fields such as probability theory, Bayesian statistics, coupling or transportation theory. In transportation theory, it can be rewritten as the solution of the Monge-Kantorovich problem with the cost function c(y 1 , y 2 ) = 1 (y

1 = y 2 ): inf Y 2 1 (y 1 = y 2 ) dπ(y 1 , y 2 )
, where the infimum is taken over all joint probability measures π on (Y × Y, F Y ⊗ F Y ) with marginals µ 1 and µ 2 .

According to this interpretation, it seems quite natural to consider the total variation distance as a relaxation of the trivial distance on [0, 1] (see [START_REF] Villani | Optimal transport: old and new[END_REF] for details). In the deterministic case, the total variation and the trivial distance coincides. In general, the total variation allows a finer analysis of the probabilistic mappings than the trivial distance. But it suffers from a high computational complexity.

In the following of the paper we will show how to ensure robustness regarding TV distance.

Finally, denoting by g 1 and g 2 the respective probability distributions w.r.t. ν, the Renyi divergence of

order λ [41] writes as d R,λ (µ 1 , µ 2 ) := 1 λ-1 log Y g 2 (y) g1(y) g2(y) λ dν(y).
The Renyi divergence is a generalized measure defined on the interval (1, ∞), where it equals the Kullback-Leibler divergence when λ → 1 (that will be denoted d KL ), and the maximum divergence when λ → ∞. It also has the very special property of being non decreasing w.r.t. λ. This divergence is very common in machine learning, especially in its Kullback-Leibler form as it is widely used as the loss function (cross entropy) of classification algorithms. It enjoys the desired properties since it bounds the TV distance, and is tractable. Furthermore, Proposition 1 proves that Renyi-robustness implies TV-robustness, making it a suitable surrogate for the trivial distance. Proposition 1 (Renyi-robustness implies TV-robustness). Let M be a probabilistic mapping, then ∀λ ≥ 1:

M is d R,λ -(α, , γ)-robust =⇒ M is d T V -(α, , γ)-robust with = min   3 2 1 + 4 9 -1 1/2 , exp( + 1) -1 exp( + 1) + 1   .
A crucial property of Renyi-robustness is the Data processing inequality. It is a well-known inequality from information theory which states that "post-processing cannot increase information" [START_REF] Cover | Elements of information theory[END_REF][START_REF] Beaudry | An intuitive proof of the data processing inequality[END_REF].

In our case, if we consider a Renyi-robust probabilistic mapping, composing it with a deterministic mapping maintains Renyi-robustness with the same level. Proposition 2 (Data processing inequality). Let us consider a probabilistic mapping M : X → P(Y).

Let us also denote ρ : Y → Y a deterministic function. If U ∼ M(x) then the probability measure M (x) s.t ρ(U ) ∼ M (x) defines a probabilistic mapping M : X → P(Y ).

For any λ > 1 if M is d R,λ -(α, , γ) robust then M is also is d R,λ -(α, , γ) robust.
Data processing inequality will allow us later to inject some additive noise in any layer of a neural network and to ensure Renyi-robustness.

4 Defense mechanisms based on Exponential family noise injection

(z, θ) = exp { t(z), θ -u(θ) + k(z)} where t(z) is a sufficient statistic, k a carrier measure (either for a Lebesgue or a counting measure) and u(θ) = log z exp {< t(z), θ > +k(z)} dz.
To show the robustness of randomized networks with noise injected from the Exponential family, one needs to define the notion of sensitivity for a given deterministic function: Definition 4 (Sensitivity of a function). For any α ≥ 0 and for any ||.|| A and ||.|| B two norms, the α-sensitivity of f w.r.t. ||.|| A and ||.|| B is defined as

∆ A,B α (f ) := sup x,y∈X ,||x-y|| A ≤α ||f (x) -f (y)|| B .
Let us consider an n-layer feedforward neural network N (.

) = φ n • ... • φ 1 (.). For any i ∈ [n],
we define N |i (.) = φ i • ... • φ 1 (.) the neural network truncated at layer i. Theorem 1 shows that, injecting noise drawn from an Exponential family distribution ensures robustness to adversarial example attacks in the sense of Definition 2. Theorem 1 (Exponential family ensures robustness). Let us denote N i X (.) = φ n • ... • φ i+1 (N |i (.) + X) with X a random variable. Let us also consider two arbitrary norms ||.|| A and ||.|| B respectively on X and on the output space of N i X .

• If X ∼ E F (θ, t, k) where t and k have non-decreasing modulus of continuity ω t and ω k . Then for any α ≥ 0,

N i X (.) defines a probabilistic mapping that is d R,λ -(α, ) robust with = ||θ|| 2 ω B,2 t (∆ A,B α (φ)) + ω B,1 k (∆ A,B α (φ))
where ||.|| 2 is the norm corresponding to the scalar product in the definition of the exponential family density function and ||.|| 1 is the absolute value on R. The notion of continuity modulus is defined in the supplementary material.

• If X is a centered Gaussian random variable with a non degenerated matrix parameter Σ.

Then for any α ≥ 0, N i X (.) defines a probabilistic mapping that is d R,λ -(α, ) robust with

= λ∆ A,2 α (φ) 2 2σmin(Σ) where ||.|| 2 is the canonical Euclidean norm on R n .
In simpler words, the previous theorem ensures stability in the neural network when injecting noise w.r.t. the distribution of the output. Intuitively, if two inputs are close w.r.t. . A , the output distributions of the network will be close in the sense of Renyi divergence. It is well known that in the case of deterministic neural networks, the Lipschitz constant becomes bigger as the number of layers increases [START_REF] Gouk | Regularisation of neural networks by enforcing lipschitz continuity[END_REF]. By injecting noise at layer i, the notion of robustness only depends on the sensitivity of the first i layers of the network and not the following ones. In that sense, randomization provides a more precise control on the "continuity" of the neural network. In the next section, we show that thanks to the notion of robustness w.r.t. probabilistic mappings, one can bound the loss of accuracy of a randomized neural network when it is attacked.

Bound on the generalization gap under attack

The notions of risk and adversarial risk can easily be generalized to encompass probabilistic mappings. Definition 5 (Risks for probabilistic mappings). Let M be a probabilistic mapping from X to Y, the risk and the α-radius adversarial risk of M w.r.t. D are defined as

Risk(M) := E (x,y)∼D E y ∼M(x) [1 (y = y)] Risk α (M) := E (x,y)∼D sup τ X ≤α E y ∼M(x+τ ) [1 (y = y)] .
The definition of adversarial risk for a probabilistic mapping can be matched with the concept of Expectation over Transformation (EoT) attacks [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF]. Indeed, EoT attacks aim at computing the best opponent in expectation for a given random transformation. In the adversarial risk definition, the adversary chooses the perturbation which has the greatest probability to fool the model, which is a stronger objective than the EoT objective. Theorem 2 provides a bound on the gap between the adversarial risk and the regular risk: Theorem 2 (Adversarial generalization gap bound in the randomized setting). Let M be the probabilistic mapping at hand. Let us suppose that M is d R,λ -(α, ) robust for some λ ≥ 1 then:

| Risk α (M) -Risk(M)| ≤ 1 -e -E x e -H(M(x))
where H is the Shannon entropy

H(p) = -i p i log(p i ) .
This theorem gives a control on the loss of accuracy under attack w.r.t. the robustness parameter and the entropy of the predictor. It provides a tradeoff between the quantity of noise added in the network and the accuracy under attack. Intuitively, when the noise increases, for any input, the output distribution tends towards the uniform distribution, then, → 0 and H(M(x)) → log(K), and the risk and the adversarial risk both tends to 1 K where K is the number of classes in the classification problem. On the opposite, if no noise is injected, for any input, the output distribution is a Dirac distribution, then, if the prediction for the adversarial example is not the same as for the regular one, → ∞ and H(M(x)) → 0. Hence, the noise needs to be designed both to preserve accuracy and robustness to adversarial attacks. In the Section 5, we give an illustration of this bound when M is a neural network with noise injection at input level as presented in Theorem 1.

Numerical experiments

To illustrate our theoretical findings, we train randomized neural networks with a simple method which consists in injecting a noise drawn from an Exponential family distribution in the image during training and inference. This section aims to answer Q2 stated in the introduction, by tackling the following sub-questions: Q2.1: How does the randomization impact the accuracy of the network? And, how does the theoretical trade-off between accuracy and robustness apply in practice? Q2.2: What is the accuracy under attack of randomized neural networks against powerful iterative attacks? And how does randomized neural networks compare to state-of-the-art defenses given the intensity of the injected noise?

Experimental setup

We present our results and analysis on CIFAR-10, CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF] and ImageNet datasets [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF]. For CIFAR-10 and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], we used a Wide ResNet architecture [START_REF] Zagoruyko | Wide residual networks[END_REF] which is a variant of the ResNet model from [START_REF] He | Deep residual learning for image recognition[END_REF]. We use 28 layers with a widen factor of 10. We train all networks for 200 epochs, a batch size of 400, dropout 0.3 and Leaky Relu activation with a slope on R -of 0.1. We minimize the Cross Entropy Loss with Momentum 0.9 and use a piecewise constant learning rate of 0.1, 0.02, 0.004 and 0.00008 after respectively 7500, 15000 and 20000 steps. The networks achieve for CIFAR10 and 100 a TOP-1 accuracy of 95.8% and 79.1% respectively on test images.

For ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF], we use an Inception ResNet v2 [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF] which is the sate of the art architecture for this dataset and achieve a TOP-1 accuracy of 80%. For the training of ImageNet, we use the same hyper parameters setting as the original implementation. We train the network for 120 epochs with a batch size of 256, dropout 0.8 and Relu as activation function. All evaluations were done with a single crop on the non-blacklisted subset of the validation set.

To transform these classical networks to probabilistic mappings, we inject noise drawn from Laplace and Gaussian distributions, each with various standard deviations. While the noise could theoretically be injected anywhere in the network, we inject the noise on the image for simplicity. More experiments with noise injected in the first layer of the network are presented in the supplementary material.

To evaluate our models under attack, we use three powerful iterative attacks with different norms: ElasticNet attack (EAD) [START_REF] Chen | Ead: elastic-net attacks to deep neural networks via adversarial examples[END_REF] with 1 distortion, Carlini&Wagner attack (C&W) [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] with 2 distortion and Projected Gradient Descent attack (PGD) [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] with ∞ distortion. All standard deviations and attack intensities are in between -1 and 1. Precise descriptions of our numerical experiments and of the attacks used for evaluation are deferred to the supplementary material.

Attacks against randomized defenses: It has been pointed out by [START_REF] Athalye | Synthesizing robust adversarial examples[END_REF][START_REF] Carlini | On evaluating adversarial robustness[END_REF] that in a white box setting, an attacker with a complete knowledge of the system will know the distribution of the noise injected in the network. As such, to create a stronger adversarial example, the attacker can take the expectation of the loss or the logits of the randomized network during the computation of the attack. This technique is called Expectation Over Transformation (EoT) and we use a Monte Carlo method with 80 simulations to approximate the best perturbation for a randomized network.

Experimental results

Trade-off between accuracy and intensity of noise (Q2.1): When injecting noise as a defense mechanism, regardless of the distribution it is drawn from, we observe (as in Figure 1(a)) that the accuracy decreases when the noise intensity grows. In that sense, noise needs to be calibrated to preserve both accuracy and robustness against adversarial attacks, i.e. it needs to be large enough to preserve robustness and small enough to preserve accuracy. Figure 1(a) shows the loss of accuracy on CIFAR10 from 0.95 to 0.82 (respectively 0.95 to 0.84) with noise drawn from a Gaussian distribution (respectively Laplace) with a standard deviation from 0.01 to 0. Performance of randomized networks under attacks and comparison to state of the art (Q2.2):

While Figure 1(b) and 1(c) illustrated a theoretical robustness against growing adversarial perturbations, Table 1 illustrates this trade-off experimentally. It compares the accuracy obtained under attack by a deterministic network with the one obtained by randomized networks with Gaussian and Laplace noises both with low (0.01) and high (0.5) standard deviations. Randomized networks with a small noise lead to no loss in accuracy with a small robustness while high noise leads to a higher robustness at the expense of loss of accuracy (∼ 11 points).

Finally, Table 2 compares the accuracy and the accuracy under attack of randomized networks with Gaussian and Laplace distributions for different standard deviations against adversarial training from Madry et al. [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]. We observe that the accuracy on natural images of both noise injection methods are similar to the one from [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]. Moreover, both methods are more robust than adversarial training to PGD and C&W attacks. As with all the experiments, to construct an EoT attack, we use 80 Monte Carlo simulations at every step of PGD and C&W attacks. These experiments show that randomized defenses can be competitive given the intensity of noise injected in the network. Note that these experiments have been led with EoT of size 80. For much bigger sizes of EoT these results would be mitigated. Nevertheless, the accuracy would never drop under the bounds illustrated in Figure 5.2, since Theorem 2 gives a bound that on the worst case attack strategy (including EoT).

Conclusion and future works

This paper brings new contributions to the field of provable defenses to adversarial attacks. Principled answers have been provided to key questions on the interest of randomization techniques, and on their loss of accuracy under attack. The obtained bounds have been illustrated in practice by conducting thorough experiments on baseline datasets such as CIFAR and ImageNet. We show in particular that a simple method based on injecting noise drawn from the Exponential family is competitive compared to baseline approaches while leading to provable guarantees. Future work will focus on investigating other noise distributions belonging or not to the Exponential family, combining randomization with more sophisticated defenses and on devising new tight bounds on the adversarial generalization gap.

Supplementary Material Table of Contents

A Notations and definitions

Let us consider an output space Y, and F Y a σ-algebra over Y. We denote P(Y) the set of probability measures over (Y, F Y ). Let (Y , F Y ) be a second measurable space, and φ a measurable mapping from (Y, F Y ) to (Y , F Y ). Finally Let us consider µ, ν two measures on (Y, F Y ). Dominated measure: µ is said to be dominated by ν

(denoted µ ν) if for all Y ∈ F Y , ν(Y ) = 0 =⇒ µ(Y ) = 0. If µ is dominated by ν, there is a measurable function h : Y → [0, +∞) such that for all Y ∈ F Y , µ(Y ) = Y h dν. h
is called the Radon-Nikodym derivative of µ w.r.t. ν and is denoted dµ dν . Push-forward measure: the push-forward measure of ν by φ

(denoted φ#ν) is the measure on (Y , F Y ) such that ∀Z ∈ F Y , φ#ν(Z) = ν(φ -1 (Z)).
Convolution product: the convolution of ν with µ, denoted ν * µ is the push-forward measure of ν ⊗ µ by the addition on Y. Since the convolution between functions is defined accordingly, we use * indifferently for measures and simple functions.

Modulus of continuity:

Let us consider f : (E, . E ) → (F, . F ). f admits a non-decreasing modulus of continuity regarding . E and . F if there exists a non-decreasing function ω E,F f : R + → R + such as for all x, y ∈ E,

||f (y) -f (x)|| F ≤ ω E,F f (||x -y|| E ).

B Main proofs

As a trade-off between completeness and conciseness, we delayed the proofs of our theorems to this section.

B.1 Proof of Lemma 1

Lemma 1. Let M be a probabilistic mapping, and let d 1 and d 2 be two metrics on P(Y). If there exists a non decreasing function φ : R → R such that ∀µ 1 , µ 2 ∈ P(Y), d 1 (µ 1 , µ 2 ) ≤ φ(d 2 (µ 1 , µ 2 )), then the following holds:

M is d 2 -(α, , =⇒ M is d 1 -(α, φ( ), γ)-robust
Proof. Let us consider a probabilistic mapping M, x ∼ D, and τ ∈ B(α), one has

d 1 (M(x), M(x + τ )) ≤ φ(d 2 (M(x), M(x + τ ))) ≤ φ( ). Hence P x∼D [∀τ ∈ B(α), d 1 (M(x + τ ), M(x)) ≤ φ( )] ≤ 1 -γ.
By inverting the inequality, one gets the expected result.

B.2 Proof of Proposition 1

Proposition 1 (Renyi-robustness implies TV-robustness). Let M be a probabilistic mapping, then for all λ ≥ 1:

M is d R,λ -(α, , γ)-robust =⇒ M is d T V -(α, , γ)-robust with = min   3 2 1 + 4 9 -1 1/2 , exp( + 1) -1 exp( + 1) + 1   .
Proof. Given two probability measures µ 1 and µ 2 on (Y, F Y ), and λ > 0 one wants to find a bound on

d T V (µ 1 , µ 2 ) as a functional of d R,λ (µ 1 , µ 2 ).
Thanks to [START_REF] Gilardoni | On pinsker's and vajda's type inequalities for csiszár'sf -divergences[END_REF], one has

d KL (µ 1 , µ 2 ) ≥ 2d T V (µ 1 , µ 2 ) 2 + 4d T V (µ 1 , µ 2 ) 4 9 .
From which it follows that

d T V (µ 1 , µ 2 ) 2 ≤ 9 4 1 + 4d KL (µ 1 , µ 2 ) 9 - 1
One thus finally gets:

d T V (µ 1 , µ 2 ) ≤ 3 2 1 + 4d KL (µ 1 , µ 2 ) 9 -1 1/2
Moreover, using inequality from [START_REF] Vajda | Note on discrimination information and variation[END_REF], one gets:

d KL (µ 1 , µ 2 ) ≥ log 1 + d T V (µ 1 , µ 2 ) 1 -d T V (µ 1 , µ 2 ) - 2d T V (µ 1 , µ 2 ) 1 + d T V (µ 1 , µ 2 )
For the sake of simplicity, since the second part of the right hand side of the equation is non increasing given d T V (µ 1 , µ 2 ), and since 0 ≤ d T V (µ 1 , µ 2 ) ≤ 1 one gets:

d KL (µ 1 , µ 2 ) + 1 ≥ log 1 + d T V (µ 1 , µ 2 ) 1 -d T V (µ 1 , µ 2 )
Hence, one gets:

exp(d KL (µ 1 , µ 2 ) + 1) -1 exp(d KL (µ 1 , µ 2 ) + 1) + 1 ≥ d T V (µ 1 , µ 2 )
By combining the two results, one obtains:

d T V (µ 1 , µ 2 ) ≤ min   3 2 1 + 4d KL (µ 1 , µ 2 ) 9 -1 1/2 , exp(d KL (µ 1 , µ 2 ) + 1) -1 exp(d KL (µ 1 , µ 2 ) + 1) + 1   .
To conclude for λ > 1 it suffices to use Lemma 1, and the monotony of Renyi divergence regarding λ.

B.3 Proof of Proposition 2

Proposition 2 (Data processing inequality). Let us consider a probabilistic mapping M : X → P(Y). Let us also denote ρ : Y → Y a deterministic function. If U ∼ M(x) then the probability measure M (x) s.t. ρ(U ) ∼ M (x) defines a probabilistic mapping M : X → P(Y ).

For any λ > 1 if M is d R,λ -(α, , γ) robust then M is also is d R,λ -(α, , γ) robust.
Proof. Let us consider M d R,λ -(α, , γ) robust algorithm. Let us also take x ∈ X , and τ ∈ B(α). Without loss of generality, we consider that M(x), and M(x + τ ) are dominated by the same measure µ. Finally let us take ρ a measurable mapping from (Y, F Y ) to (Y , F Y ). For the sake of readability we denote M(x) = ν 1 and M (x + τ ) = ν 2 (therefore M (x) = ρ#ν 1 , and

M (x + τ ) = ρ#ν 2 ). Since µ >> ν 1 , ν 2 , one has ρ#µ >> ρ#ν 1 , ρ#ν 2 . Hence one has d R,λ (ρ#ν 1 , ρ#ν 2 ) = 1 λ -1 log Y dρ#ν 1 dρ#µ λ dρ#ν 2 dρ#µ 1-λ dρ#µ = 1 λ -1 log Y dρ#ν 1 dρ#ν 2 λ dρ#ν 2
Simply using the transfer theorem, one gets

d R,λ (ρ#ν 1 , ρ#ν 2 ) = 1 λ -1 log Y dρ#ν 1 dρ#ν 2 • ρ λ dν 2 Since dρ#ν1 dρ#ν2 • ρ = E dν1 dν2 ρ -1 (F Y ) one easily gets the following: d R,λ (ρ#ν 1 , ρ#ν 2 ) = 1 λ -1 log Y dρ#ν 1 dρ#ν 2 • ρ λ dν 2 = 1 λ -1 log Y E dν 1 dν 2 ρ -1 (F Y ) λ dν 2
Finally, by using the Jensen inequality, and the property of the conditional expectation, one has

d R,λ (ρ#ν 1 , ρ#ν 2 ) ≤ 1 λ -1 log Y E dν 1 dν 2 λ ρ -1 (F Y ) dν 2 = 1 λ -1 log Y dν 1 dν 2 λ dν 2 = d R,λ (ν 1 , ν 2 ).

B.4 Proof of Theorem 1

Lemma 2. Let ψ : X → R n be a mapping. For any α ≥ 0 and for any norms ||.|| A and ||.|| B , one can define

∆ A,B α (ψ) := sup x,y∈X ,||x-y|| A ≤α ||ψ(x) -ψ(y)|| B .
Let X be a random variable. We denote M(x) the probability measure of the random variable ψ(x) + X.

• If X ∼ E F (θ, t, k) where t and k have non-decreasing modulus of continuity ω t and ω k . Then for any α ≥ 0, M defines a probabilistic mapping that is

d R,λ -(α, ) robust with = ||θ|| 2 ω B,2 t (∆ A,B α (φ)) + ω B,1 k (∆ A,B α (φ))
where ||.|| 2 is the norm corresponding to the scalar product in the definition of the exponential family density function and ||.|| 1 is here the absolute value on R.

• If X is a centered Gaussian random variable with a non degenerated matrix parameter Σ. Then for any α ≥ 0, M defines a probabilistic mapping that is d R,λ -(α, ) robust with

= λ∆ A,2 α (φ) 2 2σmin(Σ) where ||.|| 2 is the canonical Euclidean norm on R n .
Proof. Let us consider M the probabilistic mapping constructed from noise injections respectively drawn from 1) an exponential family with non-decreasing modulus of continuity, or 2) a non degenerate Gaussian. Let us take x ∈ X , and τ ∈ B(α). Without loss of generality, we consider that M(x), and M(x + τ ) are dominated by the same measure µ. Let us also denote, p F the Radon-Nikodym derivative of the noise drawn in 1) with respect to µ, p G the Radon-Nikodym derivative of the noise drawn or in 2) with respect to µ and δ a the Dirac function mapping any element to 1 if it equals a and to 0 otherwise.

1) d R,λ (M(x), M(x + τ )) = d R,λ ν * δ ψ(x) , ν * δ ψ(x+τ ) ≤ d R,∞ ν * δ ψ(x) , ν * δ ψ(x+τ ) = log sup z∈R n (p F * δ ψ(x) )(z) (p F * δ ψ(x+τ ) )(z) = log sup z∈R n exp(< t(z -ψ(x)) -t(z -ψ(x + τ )), θ > + k(z -ψ(x)) -k(z -ψ(x + τ ))) ≤ sup z∈R n ||θ|| 2 ||t(z -ψ(x)) --ψ(x + τ ))|| 2 + |k(z -ψ(x)) -k(z -ψ(x + τ ))| ≤ ||θ|| 2 ω B,2 t (||ψ(x + τ ) -ψ(x)|| B ) + ω B,1 k (||ψ(x + τ ) -ψ(x)|| B ) ≤ ||θ|| 2 ω B,2 t (∆ A,B α (ψ)) + ω B,1 k (∆ A,B α (ψ)) 2) d R,λ (M(x), M(x + τ )) = 1 λ -1 log R n p G * δ ψ(x) λ × p G * δ ψ(x+τ ) 1-λ dµ = 1 λ -1 log R n exp -1/2 λ (z -ψ(x)) Σ -1 (z -ψ(x)) + (1 -λ) (z -ψ(x + τ )) Σ -1 (z -ψ(x + τ )) (2π) n/2 |Σ| 1/2 dz = -λψ(x) Σ -1 ψ(x) + (1 -λ)ψ(x + τ ) Σ -1 ψ(x + τ ) -(λψ(x) + (1 -λ)ψ(x + τ )) Σ -1 (λψ(x) + (1 -λ)ψ(x + τ )) 2λ -2 = λ 2 -λ 2(λ -1) (ψ(x) -ψ(x + τ )) Σ -1 (ψ(x) -ψ(x + τ )) ≤ λ 2 σ max Σ -1 ||(ψ(x) -ψ(x + τ ))|| 2 2 ≤ λ∆ A,2 α (ψ) 2 2σ min (Σ)
.

Theorem 1 (Exponential family ensures robustness). Let us denote N i X (.) = φ n • ... • φ i+1 (N |i (.) + X) with X a random variable. Let us also consider ||.|| A , and ||.|| B two arbitrary norms respectively on X and on the output space of N i X .

• If X ∼ E F (θ, t, k) where t and k have non-decreasing modulus of continuity ω t and ω k . Then for any α ≥ 0, N i X (.) defines a probabilistic mapping that is d R,λ -(α, ) robust with = ||θ|| 2 ω B,2 t (∆ A,B α (φ)) + ω B,1 k (∆ A,B α (φ)) where ||.|| 2 is the norm corresponding to the scalar product in the definition of the exponential family density function and ||.|| 1 is here the absolute value on R. The notion of continuity modulus is defined in the preamble of this supplementary material.

• If X is a centered Gaussian 2 random variable with a non degenerated matrix parameter Σ. Then for any α ≥ 0, N i X (.) defines a probabilistic mapping that is d R,λ -(α, ) robust with

= λ∆ A,2 α (φ) 2 2σmin(Σ) where ||.|| 2 is the canonical Euclidean norm on R n .
Proof. This theorem is a direct consequence of Lemma 2 and Proposition 2. By applying Lemma 2 to ψ = N |i and Proposition 2 to ρ = φ n • ... • φ i+1 , we immediately get the result.

B.5 Proof of Theorem 2

Theorem 2 (Adversarial generalization gap bound in the randomized setting). Let M be the probabilistic mapping at hand. Let suppose that M is d R,λ -(α, ) robust for some λ ≥ 1 then:

| Risk α (M) -Risk(M)| ≤ 1 -e -E x e -H(M(x))
where H is the Shannon entropy: H(p) =i p i log(p i )

Proof. Let M be a randomized network with a noise X injected at layer i. We have:

|Risk α (M) -Risk(M)| = E (x,y) sup τ /||τ ||≤α E y ∼M(x+τ ) [1 (y 1 = y)] -E y2∼M(x) [1 (y = y)] = E (x,y) sup τ /||τ ||≤α E y1∼M(x+τ ),y2∼M(x) [1 (y 1 = y) -1 (y 2 = y)] ≤ E (x,y) τ /||τ ||≤α E y1∼M(x+τ ),y2∼M(x) [|1 (y 1 = y) -1 (y 2 = y)|] ≤ E (x,y) sup τ /||τ ||≤α E y1∼M(x+τ ),y2∼M(x) [1 (y 1 = y 2 )] = E (x,y) sup τ /||τ ||≤α P y1∼M(x+τ ),y2∼M(x) (y 1 = y 2 )
For two discrete random independent variables of law P = (p 1 , ..., p K ) and Q = (q 1 , ..., q K ), thanks to Jensen's inequality:

P(P = Q) = K i=1 p i q i ≥ exp ( K i=1 p i log q i ) = exp (-d KL (P, Q) -H(P ))
Then we have: ≤ E (x,y) 1 -e --H(M(x))

E (x,
= 1 -e -E x e -H(M(x))

C Additional results and discussions

In this section, we give some additional results on both the strength of the Renyi-divergence and a bound on the generalization gap for TV-distance.

C.1 About Renyi divergence

In the main submission, we chose to use the Renyi-robustness as the principled measure of robustness. Since Renyidivergence is a good surrogate for the trivial distance (which is a generalization of the 0 -1-loss for probabilistic mappings), we supported this statement by showing that Renyi-divergence is stronger than TV-distance. In this section, we extend this result to most of the classical divergences used in Machine Learning and show that Renyi-divergence is stronger than all of them.

Let us consider an output space Y, F Y a σ-algebra over Y, and µ 1 , µ 2 , ν three measures on (Y, F Y ), with µ 1 , µ 2 in the set of probability measures over (Y, F Y ) denoted P(Y). One has ν >> µ 1 , µ 2 and one denotes g 1 and g 2 the Radon-Nikodym derivatives with respect to ν.

because, for any

i ∈ [K], p i -q i ≤ d T V (P, Q).
Then, the proof is a simple adaptation of the model of proof from Theorem 2.

D Additional empirical evaluation

Due to space limitations, we had to defer the thorough description of our experimental setup and the results of some additional experiments.

D.1 Architectures & Hyper-parameters

We conduct experiments with 3 different dataset:

• CIFAR- For CIFAR-10 and CIFAR-100 [START_REF] Krizhevsky | Learning multiple layers of features from tiny images[END_REF], we used a Wide ResNet architecture [START_REF] Zagoruyko | Wide residual networks[END_REF] which is a variant of the ResNet model from [START_REF] He | Deep residual learning for image recognition[END_REF]. We used 28 layers with a widen factor of 10. We trained all the networks for 200 epochs, a batch size of 400, dropout 0.3 and Leaky Relu activation with a slope on R -of 0.1. We used the cross entropy loss with Momentum 0.9 and a piecewise constant learning rate of 0.1, 0.02, 0.004 and 0.00008 after respectively 7500, 15000 and 20000 steps.

The networks achieve for CIFAR10 and 100 a TOP-1 accuracy of 95.8% and 79.1% respectively on test images.

For ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF], we used an Inception ResNet v2 [START_REF] Szegedy | Inception-v4, inception-resnet and the impact of residual connections on learning[END_REF] which is the sate of the art architecture for this dataset and achieved a TOP-1 accuracy of 80%. For the training of ImageNet, we used the same hyper parameters setting as the original implementation. We trained the network for 120 epochs with a batch size of 256, dropout 0.8, Relu as activation function. All evaluations were done with a single crop on the non-blacklisted subset of the validation set.

D.2 Evaluation under attack

We evaluate our models against the strongest possible attacks from the literature using different norms ( 1 , 2 and ∞ ) which are all optimization based attacks. On their guide to evaluate robustness, Carlini et al. [START_REF] Carlini | On evaluating adversarial robustness[END_REF] proposed the three following attacks for each norm: where g is a function such that g(y) ≥ 0 iff f (y) = l with l the target class. The authors listed some g functions. we choose the following one:

g(x) = max(F k(x) (x) -max i =k(x) (F i (x)), -κ)
where F is the softmax function and κ a positive constant.

Instead of using box-constrained L-BFGS [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] as in the original attack, the authors use instead a new variable for x + r:

x + r = 1 2 (tanh(w) + 1)
Then a binary search is performed to optimize the constant c and ADAM or SGD for computing an optimal solution.

1 -ElasticNet attack is an adaptation of 2 C&W attack where the objective is adaptive to 1 perturbations:

min x+r∈X c 1 × r 1 + c 2 × r 2 + g(x + r)
∞ -PGD attack. The PGD attack proposed by [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] is a generalization of the iterative FGSM attack proposed in [START_REF] Kurakin | Adversarial examples in the physical world[END_REF]. The goal of the adversary is to solve the following problem:

argmax r p ≤ L(F θ (x + r), y)
In practice, the authors proposed an iterative method to compute a solution:

x t+1 = P x⊕r (x t + α sign(∇ x L(F θ (x t ), y)))
Where x ⊕ r is the Minkowski sum between {x} and {r s.t. r p ≤ }, α a gradient step size, P S is the projection operator on S and x 0 is randomly chosen in x ⊕ r.

Figure 3(a) presents the trade-off accuracy versus intensity of noise for the CIFAR-100 dataset. As for CIFAR-10, we observe that the accuracy decreases from 0.79 with a small noise (0.01) to ∼0.55 with a higher noise (0.5). The Figures 3(b) and 3(c) are coherent with the theoretical guarantee of accuracy (Theorem 2) that the model can achieve under attack with a given perturbation and noise.

Table 3 and 4 summarize the results on the accuracy and accuracy under attack of CIFAR-10 and CIFAR-100 datasets with a Randomized Wide ResNet architecture given the standard deviation of the injected noise and the number of iterations of the attack. For PGD, we use an epsilon max of 0.06 and a step size of 0.006 for an input space of between -1 and +1. We show that injecting noise empirically helps defending neural networks against adversarial attacks.

D.5 Experiments with noise on the first activation

The aim of the following experiments is empirically illustrate the Data processing inequality in Proposition 2.

Table 6 and 7 present the experiments conducted with the same set of parameters as the previous ones on CIFAR-10 and CIFAR-100, but with the noise injected in the first activation layer instead of directly in the image. We observe from Table 6 that we can inject more noise with a marginal loss on accuracy. The accuracy under attack is presented in Table 7 for a selection of models.

Figure 1 :

 1 Figure 1: (a) Impact of the standard deviation of the injected noise on accuracy in a randomized model on CIFAR-10 dataset with a Wide ResNet architecture. (b) and (c) illustration of the guaranteed accuracy of different randomized models with Gaussian (b) and Laplace (c) noises given the norm of the adversarial perturbation.

A Notations and definitions 12 B Main proofs 13 B. 1 15 C 20 E

 121311520 Proof of Lemma 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.2 Proof of Proposition 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 B.3 Proof of Proposition 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 B.4 Proof of Theorem 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 B.5 Proof of Theorem 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Additional results and discussions 16 C.1 About Renyi divergence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 C.2 Generalization gap with TV-robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 D Additional empirical evaluation 18 D.1 Architectures & Hyper-parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D.2 Evaluation under attack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 D.3 Detailed results on CIFAR-10 and CIFAR-100 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 D.4 Large scale robustness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 D.5 Experiments with noise on the first activation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Additional discussions on the experiments 21 E.1 On the need for injecting noise in the training phase . . . . . . . . . . . . . . . . . . . . . . . . . . 21 E.2 Reproducibility of the experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21

  y) sup τ /||τ ||≤α P y1∼M(x+τ ),y2∼M(x) (y 1 = y 2 ) ≤ E (x,y) sup τ /||τ ||≤α 1 -e -d KL (M(x),M(x+τ ))-H(M(x))

2 -

 2 Carlini & Wagner attack and 1 -ElasticNet attack The 2 Carlini & Wagner attack (C&W ) introduced in [7] is formulated as: min x+r∈X c × ||r|| 2 + g(x + r)

D. 3

 3 Detailed results on CIFAR-10 and CIFAR-100

Figure 2 :

 2 Figure 2: (a) Impact of the standard deviation of the injected noise on accuracy in a randomized model on CIFAR-100 dataset with a Wide ResNet architecture. (b) and (c) illustration of the guaranteed accuracy of different randomized models with Gaussian (b) and Laplace (c) noises given the norm of the adversarial perturbation.

  Lemma 1. Let M be a probabilistic mapping, and let d 1 and d 2 be two metrics on P(Y). If there exists a non decreasing function φ : R → R such that ∀µ 1 , µ 2 ∈ P(Y), d 1 (µ 1 , µ 2 ) ≤ φ(d 2 (µ 1 , µ 2 )), then the following assertion holds:

Table 1 :

 1 [START_REF] Ben-Tal | Robust optimization[END_REF]. Figure1(b) and 1(c) illustrate the theoretical lower bound on accuracy under attack of Theorem 2 for different distributions and standard deviations. The term in entropy of Theorem 2 has been estimated using a Monte Carlo method with 10 Accuracy under attack on the CIFAR-10 dataset with a randomized Wide ResNet architecture. We compare the accuracy on natural images and under attack with different noise over 3 iterative attacks (the number of steps is next to the name) made with 80 Monte Carlo simulations to compute EoT attacks. The first line is the baseline, no noise has been injected.

	Distribution	Sd	Natural	1 -EAD 60	2 -C&W 60	∞ -PGD 20
	-	-	0.958	0.035	0.034	0.384
	Normal	0.01 0.50	0.954 0.824	0.193 0.448	0.294 0.523	0.408 0.587
	Laplace	0.01 0.50	0.955 0.846	0.208 0.464	0.313 0.494	0.389 0.589

4

simulations. The trade-off between accuracy and robustness from Theorem 2 thus appears w.r.t the noise intensity. With small noises, the accuracy is high, but the guaranteed accuracy drops fast w.r.t the magnitude of the adversarial perturbation. Conversely, with bigger noises, the accuracy is lower but decreases slowly w.r.t the magnitude of the adversarial perturbation. These Figures also show that Theorem 2 gives strong accuracy guarantees against small adversarial perturbations. Next

Table 2 :

 2 Accuracy under attack of randomized neural network with different distributions and standard deviations versus adversarial training by Madry et al.[START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF]. The PGD attack has been made with 20 step, an epsilon of 0.06 and a step size of 0.006 (input space between -1 and +1). The Carlini&Wagner attack uses 30 steps, 9 binary search steps and a 0.01 learning rate. The first line refers to the baseline without attack.

	Attack	Steps	Madry et al. [31]	Normal 0.32 Laplace 0.32 Normal 0.5 Laplace 0.5
	-	-	0.873	0.876	0.891	0.824	0.846
	∞ -PGD	20	0.456	0.566	0.576	0.587	0.589
	2 -C&W	30	0.468	0.512	0.502	0.489	0.479

paragraph shows that in practice, randomized networks achieve much higher accuracy under attack than the theoretical bound, and keep this accuracy against much larger perturbations.

  10 and CIFAR-100 datasets, which are composed of 50K training samples, 10000 test samples and respectively 10 and 100 different classes. Images are trained and evaluated with a resolution of 32 by 32 pixels. • ImageNet dataset, which is composed of ∼ 1.2M training examples, 50K test samples and 1000 classes. Images are trained and evaluated with a resolution of 299 by 299 pixels.

Table 3 :

 3 Accuracy and Accuracy under attack of CIFAR-10 datasetAdversarial training fails to generalize to higher dimensional datasets such as ImageNet. We conducted experiments with the large scale ImageNet dataset and compared our randomized neural network against large scale adversarial

				1 -EAD			2 -C&W			∞ -PGD
		Natural	20	50	60	20	50	60	10	15	20
	Normal (Sd)
	0.010	0.954		0.208	0.193	0.172	0.271	0.294	0.411	0.428	0.408
	0.050	0.950	0.265	0.347	0.367	0.350	0.454	0.423	0.638	0.549	0.486
	0.130	0.931	0.389	0.401	0.411	0.443	0.495	0.515	0.710	0.636	0.553
	0.200	0.913	0.411	0.456		0.470	0.481	0.516	0.724	0.629	0.539
	0.320	0.876	0.442	0.450	0.445	0.475	0.522	0.499	0.720	0.641	0.566
	0.500	0.824	0.453	0.513	0.448	0.503	0.494	0.523	0.694	0.608	0.587
	Laplace (Sd)
	0.010	0.955	0.167	0.190	0.208	0.184	0.279	0.313	0.474	0.423	0.389
	0.050	0.950	0.326	0.315	0.355	0.387	0.458	0.448	0.630	0.534	0.515
	0.130	0.929	0.388	0.426	0.435	0.461	0.515	0.493	0.688	0.599	0.538
	0.200	0.919	0.417		0.464	0.484	0.481	0.501	0.730	0.600	0.569
	0.320	0.891	0.460	0.443	0.448	0.472	0.499	0.520	0.750	0.665	0.576
	0.500	0.846	0.454	0.471	0.464	0.488		0.494	0.721	0.650	0.589
	Exponential (Sd)
	0.010	0.953	0.153	0.174		0.228	0.292	0.306	0.443	0.404	0.395
	0.050	0.953	0.312	0.326	0.330	0.343	0.468	0.435	0.616	0.575	0.479
	0.130	0.940	0.373	0.402	0.411	0.424		0.504	0.679	0.585	0.526
	0.200	0.936	0.394		0.414	0.455	0.510	0.501	0.701	0.623	0.550
	0.320	0.919	0.429	0.426	0.416	0.494	0.492	0.513	0.739	0.638	0.564
	0.500	0.900	0.423	0.454	0.470	0.488	0.494	0.516	0.752	0.699	0.594
	D.4 Large scale robustness

Table 4 :

 4 Accuracy and Accuracy under attack of CIFAR-100 dataset. Kurakin et al.[START_REF] Kurakin | Adversarial examples in the physical world[END_REF]. One can observe from Table5that the model from Kurakin et al. is neither robust against recent 1 nor 2 iterative attacks such as EAD and C&W. Moreover, it offers a small robustness against ∞ PGD attack. Our randomized neural network with EoT attacks offers a small robustness on 1 and 2 attacks while being less robust against PGD.

				1 -EAD			2 -C&W			∞ -PGD
		Natural	20	50	60	20	50	60	10	15	20
	Normal (Sd)
	0.010	0.790	0.235	0.234	0.228	0.235	0.318	0.316	0.257	0.176	0.187
	0.050	0.768	0.321	0.294	0.320	0.357	0.377	0.410	0.377	0.296	0.254
	0.130	0.726	0.357	0.371	0.349	0.387	0.427	0.428	0.414	0.319	0.260
	0.200	0.689	0.338	0.350	0.384	0.394	0.381		0.439	0.356	0.277
	0.320	0.627	0.334	0.344	0.350	0.328	0.364	0.400	0.441	0.366	0.299
	0.500	0.553	0.322	0.331	0.331	0.349	0.342	0.351	0.408	0.374	0.308
	Laplace (Sd)
	0.010	0.782	0.199	0.227	0.243	0.225	0.311	0.321	0.236	0.190	0.177
	0.050	0.763	0.326	0.317	0.331	0.354	0.377	0.409	0.368	0.319	0.256
	0.130	0.723	0.337	0.357	0.344	0.408	0.414	0.408	0.420	0.346	0.293
	0.200	0.695	0.355	0.349	0.361	0.393	0.405	0.393	0.445	0.340	0.303
	0.320	0.647	0.324	0.373	0.357	0.388	0.387	0.373	0.460	0.381	0.303
	0.500	0.572	0.310	0.308	0.323	0.358	0.351	0.361	0.425	0.403	0.329
	Exponential (Sd)
	0.010	0.785	0.218	0.251	0.217	0.247	0.278	0.321	0.250	0.214	0.169
	0.050	0.767	0.323	0.337	0.317	0.346	0.380	0.402	0.356	0.291	0.235
	0.130	0.749	0.330		0.356	0.403	0.444	0.421	0.400	0.328	0.266
	0.200	0.731	0.345	0.361	0.357	0.388	0.424	0.406	0.427	0.340	0.267
	0.320	0.703	0.349	0.351	0.340	0.388	0.439	0.399	0.433	0.351	0.280
	0.500	0.673	0.387	0.378	0.378	0.396	0.435		0.485	0.370	0.322
	training proposed by

Table 5 :

 5 Accuracy under attack of the Adversarial model training by Kurakin et al. [27] and an Inception Resnet v2 model training with normal 0.1 noise injected in the image on the ImageNet dataset.

		Baseline	1 EAD 60	2 C&W 60	∞ PGD
	Kurakin et al. [27]	0.739	0.097	0.100	0.239
	Normal 0.1	0.625	0.255	0.301	0.061

Table 6 :

 6 Impact of the distribution and the intensity of the noise with randomized networks with noise injected on the first activation

	Sd	Normal	Sd	Laplace	Sd	Exponential
	0.01	0.956	0.01	0.955	0.01	0.953
	0.23	0.943	0.05	0.947	0.08	0.943
	0.45	0.935	0.10	0.933	0.15	0.938
	0.68	0.926	0.15	0.916	0.23	0.925
	0.90	0.916	0.20	0.911	0.30	0.919
	1.00	0.916	0.25	0.897	0.38	0.903
	1.34	0.906	0.30	0.889	0.45	0.897
	1.55	0.900	0.35	0.882	0.53	0.886
	1.77	0.893	0.40	0.867	0.60	0.885
	2.00	0.886	0.45	0.855	0.68	0.875

Preprint. Under review.

Name introduced in[START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF] which came later than[START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF].

Although the Gaussian distribution belongs to the exponential family, it does not satisfy the modulus of continuity constraint on t and its robustness has to be proved differently.

The Separation distance:

{z}) .

The Hellinger distance:

.

The Prokhorov metric:

Theorem 3. Let M be the probabilistic mapping, then for all λ > 1 if M is d R,λ -(α, , γ)-robust the following assertions holds:

(

(

, exp( +1)-1 exp( +1)+1 .

(

Proof.

(1) The proof is a simple adaptation of Proposition 1 using the inequality d H (µ 1 , µ 2 ) 2 ≤ d KL (µ 1 , µ 2 ) [START_REF] Gibbs | On choosing and bounding probability metrics[END_REF] and Lemma 1.

(2) Using the inequalities

, the proof is immediate, using Theorem 1 and Lemma 1. (3) The proof is adapted from Proposition 1 using the inequality

The result is a straightforward application of Lemma 3, and Lemma 1.

C.2 Generalization gap with TV-robustness

In the main paper, we give a bound on the generalization gap based on the Renyi-robustness. We extend this result to the TV-robustness, highlighting the fact that generalization gap could be derived from any of the above divergences. Theorem 4. Let M be the probabilistic mapping at hand. Let us suppose that M is d T V -(α, ) robust then:

where H c is the collision entropy:

Proof. For two discrete random independent variables of law P = (p 1 , ..., p K ) and Q = (q 1 , ..., q K ), thanks to Jensen's inequality:

E Additional discussions on the experiments

For the sake of completeness and reproducibility, we give some additional insights on the noise injection scheme and comprehensive details on our numerical experiments.

E.1 On the need for injecting noise in the training phase

Robustness has always been thought as a property to be enforced at inference time and it is tempting to focus only on injecting noise at inference. However, simply doing so ruins the accuracy of the algorithm (as it becomes an instance of distribution shift [START_REF] Sugiyama | Machine learning in non-stationary environments: Introduction to covariate shift adaptation[END_REF]). Indeed, making the assumption that the training and test distributions matches, in practice, injecting some noise at inference would result in changing the test distribution.

Distribution shift occurs when the training distribution differs from the test distribution. This implies that the hypothesis minimizing the empirical risk is not consistent, i.e. it does not converge to the true model as the training size increases.

A way to circumvent that is to ensure that training and test distributions matches using importance weighting (in the case of covariate-shift) or with noise injection in the training phases as well (in our case).

E.2 Reproducibility of the experiments

We emphasize that all experiments should be easily reproducible. All our experiments are developed with TensorFlow version 1.12 [START_REF] Abadi | TensorFlow: Large-scale machine learning on heterogeneous systems[END_REF]. The code is available as supplemental material and will be open sourced upon acceptance of the paper. The archive contains a readme file containing a small documentation on how to run the experiments, a configuration file which defines the architecture and the hyper-parameters of the experiments, python scripts which generate a bash command to run the experiments. The code contains Randomized Wide Resnet used for CIFAR-10 and CIFAR100, Inception Resnet v2 used for ImageNet, PGD, EAD and C&W attacks used for evaluation under attack. We ran our experiments, on a cluster with computers each having 8 GPU Nvidia V100.


Xavier Juanes
email: xavier.juanes@cirad.fr

Véronique Alary
email: veronique.alary@cirad.fr

Mona A Osman

Adel Aboul-Naga

Dataset for describing the diversity of household farming systems and the degree of crop-livestock diversification and integration in the Western part of Nile valley (Egypt)

Keywords:

The CLIMED household farm dataset comes from a data collection conducted from 2013 to 2014 in five zones of the New Reclaimed Lands in the western part of the Nile Delta (Egypt). The main objective was to describe the diversity of household farms' assets and activities, the degree of crop and livestock integration at the farm level to assess the link between integration, diversification, efficiency, and livelihoods. This data set permitted to compare the diversity of farming systems of 175 household farms and to assess the economic and technical performances of crop-livestock systems along a geographical transect of reclaimed desert lands in Egypt. This dataset was the primary material in the research paper on "Multi-criteria assessment of the sustainability of farming systems in the reclaimed desert lands of Egypt" (See the related research article.). Data described the three main components of the family farm system, i.e., the land, livestock, and household systems, respectively. The description of each activity (mainly crop, animal, or off-farm) by detailing all the incoming and outgoing flows of inputs and outputs allowed

investigating the economic and financial contribution of each activity and the degree of dependence or complementarity between them. The dataset provided two tables of analyzed data related to, respectively, 'diversification and integration' and 'efficiency and wellbeing.' Moreover, this dataset constitutes an original material regarding the living conditions and farm functioning in the new lands reclaimed over the last 50 years in Egypt. The survey data were entered into an Access database, checked with statistical cross-checking variables, and completed by field return for missed or noncoherent data.

© 2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY-NC-ND license. ( http://creativecommons.org/licenses/by-nc-nd/4.0/ )

Specifications Table

Subject area Household farm system; household's activities; Crop-livestock integration; efficiency; diversification; wellbeing; efficiency; Specific subject area New reclaimed lands (NRL); Egypt; Type of data Five tables with raw and analyzed/calculated data How data were acquired

The household farm survey has been based on a structured questionnaire with quantitative and qualitative parameters; Data format Excel file with one table (data matrix) per sheet.

Raw and analyzed/calculated data.

Parameters for data collection

The main conditions for starting the data collection at the household level were that the family head was present, preferably accompanied by his spouse and children. Three locations were privileged for data collection: at home, under the animal shelter, or in the border of the parcel. The family head and his wife were the primary respondents, even if some adult sons or girls have completed the information for specific activities. All the answers were filled on the paper support during the interviews.

Value of the data

The CLIMED dataset provides a complete description of the farm and off-farm activities at the household level to assess global indicators related to integration, diversification, efficiency, and wellbeing; The dataset can be used by research or public bodies to capture the diversity of farm systems and to work on sustainable innovations in the studied zones of Egypt; The information in the dataset about the family farm organization and activities can be used to support policy markers or development agencies in prioritizing and developing their operations for more sustainable development option of these rural zones; The dataset can be used as the basis to design and implement further agronomic or zootechnic experiments or to identify prototypes of farm systems to test innovations in the zone;

Data

As shown in Table 1 , the survey was administered to 175 household heads representing the sample size used in the five selected zones of New Reclaimed Lands in Egypt. Among the sample, 90% (158) of them are small and medium land beneficiaries (with 1-2 ha, maximum), and the remained 10% (17) were composed of medium and large land farms who have invested in land or livestock in the zone. All of them have a livestock activity.

The three first tables ( table 2 -4 ) contain the raw data describing the household and family labor characteristics, the land access and size, and the animal stock per species. These tables are mainly extracted from the set of raw data. Tables 5 and6 give the calculated data compiled to assess the sustainability of the family farms based on indicators related to the degree of diversification and integration ( Table 5 ) and indicators of wellbeing and efficiency ( Table 6 ).

Experimental design, materials, and methods

The household farm survey has been implemented following an exploratory field study based on open interviews with agricultural technical staff working in the zone and farmers. This exploratory study allowed identifying the critical criteria of diversity (notably regarding the type of land access and livestock size) and the sampling protocol along a gradient of settlement in the zone (See Fig. 1 , [ 1 , 2 ]). So, five zones have been chosen according to the date of land reclamation and settlement: from the reclaimed lands settled in the sixties located in the Southwest of Alexandra (El-Nardha) to the newly reclaimed lands settled at the end of the nineties in the Tiba and Bustan extension zones. In-between, two zones have been considered: Sukharel-Bangar (called here 'Bangar') reclaimed mainly in the eighties and Hammam in the nineties. Except for the Bustan zone, three villages have been selected to reflect the diversity of land beneficiaries in each zone. In the Bustan zone, a fourth selected village allowed to consider the particular case of a village settled by graduates. In the two more recent settled locations (i.e., Tiba and Bustan extension), it is usual to distinguish "graduates" and "common beneficiaries" village. In each village, ten farmers have been selected based on the method of snowball sampling [START_REF] Goodman | Snowball Sampling[END_REF] and respecting a certain proportion of very small, small, and medium farms regarding livestock size. One hundred fifty-eight farmers have been surveyed in 2013/14. We added 17 large farmers settled in the region to understand the global dynamics in the different zones. The total sample counts 175 farms. We organized the household farm survey with an appointment with the heads of the household. Generally, the local technician contacted a set of farmers (according to the criteria given in our protocol), and we organized a joint meeting in the meeting room of one local association or of one farmer. The research team introduced to the farmers the research project, its objective, and the expectations of the household farm survey. This presentation allowed us to have a general discussion about the main constraints or opportunities in the studied village.

The French-Egyptian research team was composed of 8 researchers. In the majority of cases, a group of two researchers followed one farmer at his house to conduct the interview and fill the questionnaire.

The household farm survey has been based on a structured questionnaire organized on six parts (See the supplementary file with the questionnaire): -Part 1: Family and house description to assess the family living conditions -Part 2: Land and crop system. This part consists of a story and description of the land access and crop management over the seasons; -Part 3: Livestock structure and management including the feeding system, animal movements (inc. livestock transactions), animal performance and health care; -Part 4: Mode of funding (formal or informal credit or donation); -Part 5: Main changes during the last ten years; -Part 6: Social capital, including family and professionals networks.

According to the composition of the research group, the questionnaire was filled in Arabic or English. The survey was conducted from March 2013 to February 2014.

After each session of fieldwork, two researchers were in charge of data entry at the research office. This data entry has been organized on Microsoft ACCESS. A storage database and input screens specific to this survey had been developed. This information system thus guaranteed the coherence of the data and their integrity through an Information Technology (IT) structure. A data checking has been done from June 2013 to February 2014 using cross-checking variables, statistical tests (mainly based on the distribution for each variable), and tests of coherence (e.g., the cropland allocation according to land access). A systematic return to each farm has been organized from May to September 2014 to validate or to correct incoherent data. All the questionnaires in paper forms are available at APRI (Animal production research institute) in Egypt.

Fig. 1 .

 1 Fig. 1. The geographical location of the five selected areas in the western part of the Nile Delta (Egypt) (Alary et al., 2016, [3] ).

Table 1

 1 Description of the sample.

	Data source location	Institution: The French International center of agricultural research for
		development (CIRAD), the Egyptian Animal Production Research Institute
		(APRI) and International center for Agricultural Research in Dry Areas (ICARDA)
		City/Town/Region: Five zones in the Western part of Nile Valley: El-Nahda,
		El-Hammam, Banger, Tiba, and Bustan zone, along a gradient from the
		Northwest zone to the South (see fig. 1 below).
		Country: Egypt
	Data accessibility	Repository name: CIRAD dataverse:
		https://dataverse.cirad.fr/privateurl.xhtml?token = c86e100e-c674-4110-bd82-
		cf0571b53a6d
		Data identification number: doi:10.18167/DVN1/UDTX1Y
		Provisional access before publication:
		https://dataverse.cirad.fr/privateurl.xhtml?token = c86e100e-c674-4110-bd82-
		cf0571b53a6d
	Related research articles	Author's name: Véronique Alary, Samir Messad, Adel Aboul-Naga, Mona A.
		Osman, Taha Hosni Abdelsabour, Ehab Salah, Xavier Juanes
		Title: Multi-criteria assessment of the sustainability of crop-livestock farming
		systems in the reclaimed desert lands of Egypt
		Journal: Agricultural System
		DOI: 10.1016/j.agsy.2020.102863

Description of data collection An Egyptian-French research team, including animal scientists, agronomists, and socio-economists, collected the data using a commonly structured questionnaire with closed, semi-opened, and open questions. The research team involved in the data collection entered the answers into a database developed on Microsoft Access.

Table 2

 2 Main variables describing household characteristics.

	Short name of			Range preview
	the variable	Full name of the variable	Content of the variable	Min -Max *
	Edu_H	Education of the family head	By educational level: 1. No read	2.6-4.8
			no write; 2. Coranic school; 3.
			primary school; 4. Secondary
			school; 5. High school or
			professional school
	Age_H	Age of the family head	Number of years	45.5-54.3
	Fs_hh	Family size	Number of persons	6.7-11.1
	Per_school	Schooled children/total children	%	40% -70%
		number in the family
		Number of children out of school who	Number of persons	0.1-0.3
	Fw_child_nschool	work in the farm
	Amw_hh	Number of potential male workers in	Number of persons	2.4-3.2
		the family (more than 16 years old and
		no schooled)
	Afw_hh	Number of potential female workers in	Number of persons	0.5-1.3
		the family
	Tw_out	Number of workers from the family	Number of persons	0.4-0.7
		working outside the farm
	Tw_out_pot	Number of workers from the family	Number of persons	0.7-1.3
		working outside the farm and persons
		looking for a job

* Min and Max are the minima and maximum of the mean by zone

Table 3

 3 Main variables describing the land system (access and size).

	Short name of		Content of the	Range preview
	the variable	Full name of the variable	variable	Min -Max
	Atot	Total area owned by the family	Feddan *	3.10-9.1
	Acult	Total seasonal area use for the crops by	Feddan *	6.9-17.8
		the family
	Prent	Percentage of rent area / seasonal	%	1% -19%
		cultivated area (AA)
	Area_purch	Land purchased since the arrival	Feddan *	1.7-4.9
	Area_ben_grad	Land access as beneficial or graduate	Feddan *	1.2-3.8
	Table 4
	Main variables describing the livestock system.
	Short name of		Content of the	Range preview
	the variable	Full name of the variable	variable	Min -Max
	TLU_farm	Number of Total Livestock Unit (TLU * )	TLU	6.1-24.5
		per farm
	Fat_TLU	Number of fattening large ruminant in	Heads	0.7-6.8
		the farm
	Dairy_farm	Number of dairy large ruminants per	Heads	2.03-8.6
		farm (buffaloe, local cows, and
		crossbred)
	Perbuff_dairy	Percentage of dairy buffaloes per farm/	%	16% -49%
		total dairy animals
	Percross_dairy	Percentage of dairy crossbred per farm/	%	32% -62%
		Total dairy animals
	SR_head	Number of small ruminant per farm	Heads	0.57-6.3

* One feddan = 0.42 ha. * Total Livestock Unit (TLU) of 250 kg live weight.

Table 5

 5 Description of the synthetic variables for assessing diversification and integration at the farm level.

	Short name of		Content of the	Range preview
	the variable	Full name of the variable	variable	Min -Max
		% animal cash flow / total family cash	%	29% -37%
	Receipt_Anl_perc	flow
	Dairy_totprod	Percentage of dairy products/ total	%	3% -9%
		animal production (in value)
	FeedPur_Tlu	Purchased feed cost (inc. concentrates)	EGP/TLU	1202-1977
		per TLU per year
	FodderPro_Tlu	Self-produced fodder cost per TLU per	EGP/TLU	352-859
		year
	ConcPur_Tlu	Concentrate cost per TLU per year	EGP/TLU	897-1791
		Production cost for fodder/total feed	%	24% -39%
	Selffeed_cost_perc	costs (produced and purchased)
	N_org_perc	Organic nitrogen (N)/ total Nitrogen	%	29% -49%
		supply (chemical and organic)
	Perc_Nfarm	On-farm Nitrogen supply/Organic	%	33% -92%
		nitrogen supply
	FWU	Number of family workers on the farm	Full-time work	2.6-3.8
			unit
	AW_tot	Number of salaried agricultural	Full-time work	1.2-5.6
		workers (Number of days of	unit
		agricultural workers /260 days/year)
	WAWU	The salaried workforce in the total	%	23% -45%
		farm workforce
	AWU	The family and salaried workforce in	Full-time work	3.8-8.6
		the farm	unit
	Pfodder	Total area cultivated with fodder per	% TAA *	10% -22%
		year
	Ptree	Total area cultivated with tree crop per	% TAA	0% -67%
		year
	PAnnualCrop	Total area cultivated with annual crops	% TAA	5% -28%
		per year
	Pwheat	Total area cultivated with wheat per	% TAA	8% -27%
		year
	Pmaize	Total area cultivated with maize per	% TAA	9% -24%
		year
	Pcashcrop	Total area cultivated with cash crop per	% TAA	1% -28%
		year

* TAA for Total Agricultural Area).

Table 6

 6 Description of the synthetic variables for assessing wellbeing and efficiency at the farm level.

	Short name of the		Content of the	Range preview
	variable	Full name of the variable	variable	Min -Max
	Net_inc	Gross margin per feddan	EGP/feddan	27,721-170,214
	Net_inc_FWU	Net income/ familial work unit	EGP/family	25,332-89,229
			work unit
	Net_inc_salmin	Net income/ minimum govermental salary	ratio	4-12
		(1200 EGP/month * 12 months)
	Net_inc_cap	Total net income per Total family members	EGP/person	7736-21,427
		Meat and milk income per total family and	%	47-67%
	Anl_CF_FarmExpenses	farm annual expenses
		Ruminant net income/ minimum salary (fixed	ratio	0.75-2.22
	Employement_Ruminant	at 1200 EGP per month)
	NutFam_P	Protein supply/family protein needs based on	%	26-38%
		FAO requirement i.e., 60 g/person/day)
		Milk daily income/minimum family daily	%	12-39%
	Milk_CF_capitaneeds	monetary needs
	Trans_farmjob	Number of feddan/total workforce (AWU)	ha	1.01-1.76
	Trans_family	Farm capital (Owned land value and livestock	EGP	45,537-122,016
		capital at selling price) divided by the number
		of children
	Net_inc_fed	Net income per feddan	EGP	12,323-32,085
	Profit	Net income / total product	%	36-48%
	Bov_inc_K	Meat and dairy net income/livestock capital	%	25-134%
	Eff_feed_liter	Total feed costs/ Milk production	EGP/liter	1.52-2.95
	Milk_yield_liter	Total milk production per dairy animal per year	liter/head/year	1217-1530
	AAdairyprod_fed	Total milk production per feddan of fodder	liter/feddan	2034-6513
		crops (mainly, maize and berseem)

Note: Economic and financial indicators are related to 174 farmers. One farmer with an intensive poultry farm has been removed from the sample.

Acknowledgments

The material presented here has been collected in the framework of a collaborative research project named CLIMED on 'The Future of Mediterranean Livestock Farming Systems: Opportunity and Efficiency of Crops-Livestock Integration,' within the research program ARIMnet (2012-2016). This work was supported by the Agency of National Research (ANR) in France (Grant ARIMnet Program 2011, ANR-2012-2015, 12-AGRI-0 0 07 ). We especially thank all the researchers from APRI's sheep and goat research department for their active participation and all the farmers and stakeholders in the Nubaria area for the time they spent with us. We thank also the CGIAR Research Program on Livestock and their support.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships which have, or could be perceived to have, influenced the work reported in this article.

Supplementary materials

Supplementary material associated with this article can be found, in the online version, at doi:10.1016/j.dib.2020.105879 .