Alice Lasserre
email: alice.lasserre@etu.u-bordeaux.fr

Raymond Namyst
email: raymond.namyst@u-bordeaux.fr

Pierre-André Wacrenier
email: pierre-andre.wacrenier@u-bordeaux.fr

EASYPAP: a Framework for Learning Parallel Programming

Keywords: parallel programming, visualization, monitoring, education, OpenMP, MPI

This paper presents EASYPAP, an easyto-use programming environment designed to help students to learn parallel programming. EASYPAP features a wide range of 2D computation kernels that the students are invited to parallelize using Pthreads, OpenMP, OpenCL or MPI. Execution of kernels can be interactively visualized, and powerful monitoring tools allow students to observe both the scheduling of computations and the assignment of 2D tiles to threads/processes. By focusing on algorithms and data distribution, students can experiment with diverse code variants and tune multiple parameters, resulting in richer problem exploration and faster progress towards efficient solutions. We present selected lab assignments which illustrate how EASYPAP improves the way students explore parallel programming.

I. INTRODUCTION

During the last decade, the High Performance Computing community had a hard time coping with the evolution of parallel architectures toward massively parallel heterogeneous multicore machines. In fact, all software developers are currently concerned about this manycore trend which has impacted every commodity hardware, from smartphones to desktop machines. To get the most out of nowadays computers, people must be trained to parallel programming. It is no surprise that integrating HPC into undergraduate and postgraduate courses to expose all students to basic parallel programming skills has been identified as a priority by the European Technology Platform for HPC in their 3rd Strategic Research Agenda [START_REF]Strategic research agenda[END_REF].

Unfortunately, learning parallel programming is intrinsically more difficult than learning sequential programming, especially because students lack convenient and easy-to-use tools to get familiar with non-determinism and to visualize what happened during a parallel execution.

We present EASYPAP, our attempt to provide students with a simple and attractive programming environment to facilitate their discovery of the main concepts of parallel programming. EASYPAP is a framework providing interactive visualization, real-time monitoring facilities, and off-line trace exploration utilities. Students focus on parallelizing 2D computation kernels using Pthreads, OpenMP, OpenCL, MPI, intrinsics instructions, or a mix of them. EASYPAP was designed to make it easy to implement multiple variants of a given kernel, and to experiment with and understand the influence of many parameters related to the scheduling policy or the data decomposition. During our undergraduate and postgraduate lab sessions, students enjoyed the feedback provided by the graphical tools and were able to gain a deeper understanding of both parallel programming and computer architecture.

II. THE EASYPAP FRAMEWORK

EASYPAP is a C programming environment that relies on the SDL library [START_REF]SDL: Simple directmedia layer[END_REF] to interactively render the results of 2D computations. EASYPAP's main philosophy is to let students focus on computation kernels while hiding most of the implementation details related to program initialization (at the notable exception of memory allocation), code instrumentation and interactive display. The main program loop is thus controlled by EASYPAP.

A. Kernels and variants

In EASYPAP, functions performing computations on images are called kernels. EASYPAP comes with a large set of predefined kernels (e.g. Transpose, Invert, Blur, Pixelize, Game Of Life, Mandelbrot, Abelian SandPile). New kernels can obviously be easily added. Let us take the mandel kernel, which computes the Mandelbrot set, as an illustration. The outer loop (line 3) performs the requested nb_iter iterations in a row. Lines 4-6 illustrate how the contents of the image are accessed during an iteration. For the sake of simplicity, EASYPAP works on square shape images. The pixels of the image are accessed through the cur_img(row, col) macro. Here is how to run the seq variant of the mandel kernel on a 2048 × 2048 image:

easypap --kernel mandel --variant seq \ --size 2048
This action brings a window on the screen which displays an animation consisting of the series of images computed at each iteration. The animation can be paused, or can be slightly accelerated by skipping frames.

Since checking the belonging to the Mandelbrot set may be computed independently for any (i, j) pixel, the mandel kernel can be trivially parallelized. To develop a straightforward OpenMP version designed as an incremental evolution of the sequential variant, we can simply duplicate the sequential variant, rename it mandel_compute_ omp, insert a single "#pragma omp parallel for" clause before the for loop iterating over lines, recompile EASYPAP, and launch:

easypap --kernel mandel --variant omp

The obtained graphical animation allows student to visually check if this new variant produces the expected output and if it runs faster.

The simplicity with which students are able to implement while maintaining many different variants of given kernel is an essential feature of EASYPAP. Indeed, it makes it very convenient to compare variants against each other and explore their robustness when changing some parameters, as we further discuss in the next sections.

B. Interactive monitoring

In order to get more feedback about the parallel execution of a variant, the code needs to be slightly instrumented. To do so, sequential portions of code computing image chunks (called tiles) have to be bracketed by calls to monitoring_ {start/end}_tile. Fig. 2 shows a typical OpenMP tiled implementation of the mandel kernel where the do_tile function has been instrumented. This function sequentially computes all the pixels inside an arbitrary rectangle defined by (x,y,width,height). The last parameter is the rank (from 0 to #threads-1) of the thread which computes the tile. With OpenMP, we just pass omp_get_thread_num(). Once the code has been instrumented, real-time monitoring can simply be activated: easypap --kernel mandel --variant omp_tiled \ --tile-size 16 --monitoring

The monitoring mode pops up two additional side windows as displayed in Fig. 3.

The Activity Monitor window reports the realtime load of each CPU. This load is a percentage representing the amount of time spent in computations over the duration of the iteration. At the bottom of the window, a history diagram reports the evolution of cumulated idleness over time. In Fig. 3, we clearly observe a load imbalance between CPUs. The static distribution of tiles is indeed inappropriate because the large black area at the bottom of the image, which contains a lot of pixels belonging to the Mandelbrot set, involves much more computations than other areas.

The Tiling window reflects the way tiles have been assigned to threads at each iteration. Each thread is assigned a different color which is consistent with the color assigned to CPUs in the Activity Monitor window. By observing Fig. 3, we see that the tiles have been assigned to threads in contiguous blocks, in accordance to the static loop scheduling policy.

The tiling window is a precious tool to observe the different loop scheduling policies of OpenMP when combined with the collapse clause. In Fig. 4, we examine various loop scheduling policies through the tiling window. Fig. 4a shows that the static clause evenly distributes tiles to threads in contiguous chunks. Fig. 4b reveals the opportunistic nature of the dynamic clause. Fig. 4c illustrates the behavior of the new OpenMP 5 nonmonotonic clause: tiles are first distributed in a static manner, but work-stealing is eventually used to correct load imbalance. Finally, Fig. 4d shows how size of chunks assigned to threads decreases over time with the guided policy.

C. Performance mode

In order to accurately benchmark and compare the performance of multiple variants, we need to completely eliminate the overhead of graphical updates. When invoked with the --no-display option, EASYPAP runs silently and reports the overall wall clock time after completion of the requested number of iterations.

> easypap --kernel mandel --variant omp_tiled \ --tile-size 16 --iterations 50 \ --no-display 50 iterations completed in 579ms

Moreover, the completion time, together with all execution and configuration parameters, are reported in a Comma Separated Values (CSV) file. Students can customize simple python scripts to automate their experiments by specifying parameter ranges, as illustrated in Fig 5.

Students can then exploit their data and produce the desired graph or histogram thanks to the easyplot command. A key feature of easyplot is that the legend is automatically generated from the data. Once data have been filtered, constant parameters are put aside, and the names of plotlines are set using the remaining ones (see Fig. 6). This guarantees that experiments conducted in different conditions will not silently be incorporated in the same graph, a common mistake among students's reports.

D. Post mortem trace analysis

Although the monitoring facilities greatly help to detect and understand flaws in the execution of kernels, it cannot always capture some subtle properties such as the heterogeneity of tasks duration or the correct implementation of task dependencies. When a deeper analysis is required, students use the --trace option to record tile-related profiling events at execution time (i.e. start/end time, tile coordinates, cpu) into a trace file:

easypap --kernel mandel --variant omp --trace \ --no-display --iterations 10
To visually explore and interact with the trace, we provide the EASYVIEW utility (Fig. 7). Its graphical interface is subdivided in two parts.

The left side presents a view widely adopted by many trace viewers: a Gantt chart displays per-CPU sequences of tasks for a selectable range of iterations. Tiles computed by the same CPU have the same color, and are displayed on the same timeline. When moving the mouse over a task, a pop-up bubble displays the task duration. The right side displays a reduced view of the surface computed at the selected iteration (see thumbnails of Mandelbrot set appearing in Fig. 7). Whenever the x-axis of the mouse intersects tasks in the Gantt chart, the corresponding tiles are highlighted over this reduced image, helping to localize computations. As a consequence, starting on the left side of the Gantt chart and moving smoothly the mouse towards the right side reveals the order in which tiles have been computed.

In addition, students can toggle between this vertical mouse mode and a horizontal mode in which the y-axis of the mouse allows to select a particular CPU and highlights the tiles computed during the displayed period. Basically, this allows to observe the "coverage map" of a given CPU during one or multiple iterations, and to check the locality of computations across iterations. This feature is further detailed in Section III-B.

EASYVIEW is a powerful mean for students to understand how the scheduling of computations are performed, to see which image areas are the most time-consuming, to check if the computations were evenly balanced over computing units, and even to track down synchronization issues. We highlight a series of such situations in Section III.

III. EXAMPLE OF ASSIGNMENTS

We have used EASYPAP and EASYVIEW both with undergraduate students during parallel programming introduction courses, and with postgraduate students during parallel and distributed computing courses, where students explore advanced features of multicore, GPU and cluster programming. Even if students usually start with simple, straightforward implementations of basic Figure 8. When using OpenMP dynamic loop scheduling of small tiles, the tiling window reveals two noticeable patterns. kernels, they quickly dive into more subtle codes where they encounter bugs and performance issues. Using various case studies, we now explore to what extent EASYPAP and EASYVIEW help students to better understand the behavior of their code and visualize things which are traditionally very difficult to observe. Animated screenshots of EASYPAP and EASYVIEW are available on the project site [START_REF] Namyst | The EASYPAP web site[END_REF].

A. Computing the Mandelbrot Set

After a first hands-on session during which undergraduate students discover the EASYPAP environment using very simple kernels, their first assignment is usually devoted to the computation of the Mandelbrot Set. As we previously mentioned, parallelizing the mandel is trivial, but achieving good speedups requires to pay attention to load balancing. The assignment is thus mostly about performing experiments to find the best combination of loop scheduling policy, tile shape and tile size.

Almost all students end up with a tiled implementation, either using Pthreads or OpenMP, similar to the one described in Fig. 2 using dynamic distribution of squared tiles. The size of tiles depends on the dimension of the image as well as on the underlying hardware. After a few iterations, students observe interesting patterns appearing in the tiling window, as spotted in Fig. 8.

Pattern 1 reveals horizontal stripes of the same color together with a few stripes featuring an alternation of two colors. These stripes correspond to one or two threads computing several tiles in a row. Such a situation happens because 1) these tiles correspond to areas located far away from the In "heat map" mode, the brightness of tiles displayed in the Tiling Window reflects the duration of the corresponding tasks: the brighter an area is, the more time-consuming it is. On picture (a) we can distinguish the shape of the Mandelbrot set as depicted in Fig. 8. On picture (b), we observe that border tiles take a longer time to be processed than inner tiles.

Mandelbrot set, where computations take only a few iterations to complete and 2) the other threads are all busy computing time-consuming tiles in the top-right black corner.

In contrast, Pattern 2 features a quasi-perfect cyclic distribution of colors. This is due to the fact that all tiles require the same amount of (heavy) computations. Therefore, the dynamic distribution turns into a regular, cyclic one in such areas.

B. Picture Blurring: a simple 2D stencil code

During their discovery of parallel computing, our students are quickly exposed to simulations involving Stencil computations. We use an assignment based on a Picture Blurring kernel to introduce students to the parallelization of 2D stencil codes. The sequential version of the blur kernel uses two images. At each iteration, all pixels from the 3 × 3 square centered in (i, j) are read from the first image, and the average is written to the second one. The two images are swapped between iterations.

Since every pixel is read multiple times at each iteration, students are encouraged to implement a tiled parallel version to maximize cache reuse. To avoid out-of-bounds image accesses for pixels located on the borders (which have less than 9 neighbours), their code includes several conditional branches which leads to poor performance.

By observing that tests are only required for tiles located on the edges (i.e. outer tiles), students implement different codes for outer and inner tiles. After implementing this optimization, they can quickly check its effectiveness by using the "heat map" mode of the tiling window: Fig. 9b reveals that inner tiles involve less computations than tiles located on the edges.

Running EASYPAP in performance mode tells us that the gain achieved is beyond expectations: the new variant is 3 times faster! To analyze this performance boost, EASYVIEW offers a nice trace comparison feature, as shown in Fig. 10. We notice that many tasks are approximately 10 times faster than their original version. By moving the mouse over those tasks, students immediately get the confirmation that short durations do always correspond to inner tiles. The ×10 speedup not only comes from the removal of conditional branches: it is mostly imputable to compiler auto-vectorization (×8 on AVX2-capable Intel processors).

Another interesting observation can be made when switching to the "coverage map" mode provided by EASYVIEW, using mouse horizontal mode to select all displayed tasks for a given CPU. In Fig. 10, the mouse cursor is over the CPU#3's timeline, so the purple squares displayed over the top-right thumbnail reveal the area covered by all tasks executed on this CPU during iteration range [7..9]. We observe that the squares are mostly regrouped in a single area, with only a few ones scattered in other places, which highlights the good locality property of the new nonmonotonic:dynamic scheduling policy.

C. Identification of Connected Components

In more advanced courses, we introduce the students to tasks and dependencies concepts. After experimenting with OpenMP tasks on small programs, students are asked to parallelize a Connected Components Detection algorithm on 2D images. The main goal is to identify the different connected components (i.e. separated by transparent pixels) by coloring each of them in a unique color. The proposed algorithm first reassigns each pixel a unique color and then propagates the maximum between neighbours until reaching a steady state. The sequential implementation uses a sequence of two phases per iteration: the first phase propagates local maxima to the right and to the bottom, and the second one proceeds to an up-left propagation.

Parallelizing this algorithm without introducing extra iterations is quite challenging. A possible solution is to use a tiled implementation in which tiles are processed with some constraints: during the bottom-right phase (resp. up-left), a tile cannot be executed until its left and upper (resp. right and lower) neighbours have not completed. With OpenMP tasks, these constraints directly translate into task dependencies, as sketched in Fig. 11.

However, because it takes time to get familiar with the subtleties of task dependencies in OpenMP, students usually achieve a correct implementation only after several attempts. Most of the time, they over-constrain the problem and end up with a sequential execution of tasks. In such cases, EASYVIEW greatly helps to figure out if the dependencies were correctly enforced, as illustrated in Fig. 12. One can observe the order in which tiles were processed by just moving the mouse.

D. Game of Life: Putting it All Together

In addition to Pthreads, OpenMP and OpenCL, EASYPAP also provides support for MPI programs, and most notably for debugging such programs using monitoring facilities. To illustrate this feature, we present an assignment to students who attend advanced courses on HPC programming.

The goal is to implement an efficient version of Conway's Game of Life [START_REF] Games | The fantastic combinations of john conway's new solitaire game "life" by martin gardner[END_REF] able to cope with large, potentially sparse simulations. Therefore, we ask them to pay attention to memory usage optimization, by using their own, low memory footprint data structures for computations. EASYPAP allows kernels to use arbitrary data structures for computations. Such kernels simply have to update the current image when a graphical refresh is needed.

In addition, students have to develop a lazy evaluation algorithm that avoids computing tiles whose neighbourhood was in a steady state at the previous iteration. Once they end up with an effective Pthreads or OpenMP lazy variant, students can look at the tiling window to make sure that areas where "nothing changes" are not computed.

Finally, students extend their implementation in order to cope with distributed architectures by using MPI. They learn how to exchange ghostcells between MPI processes, including metainformations regarding the state of tiles (steady or lively). The whole code is less than 150 lines. EASYPAP helps by integrating the mpirun process launcher and only displaying the main window of the master process by default.

For debugging purposes, EASYPAP can display all the windows for each process. The following command launches two MPI processes executing the mpi_omp variant in debugging mode. for (int j = 0; j < NUM_TILES; j++) for (int i = 0; i < NUM_TILES; i++) #pragma omp task depend(in: tile The monitoring windows (Fig. 13) reveal that each process contains 4 threads and works on half of the image. Most importantly, since the sparse dataset consists in planers evolving along the diagonals of the image, we can check that only tiles located near diagonals are computed.

[i -1][j], \ tile[i][j -1]) \ depend(inout: tile[i][j]) \ firstprivate(i, j) tile_down_right (i, j);

E. Discussion

Since we introduced EASYPAP in our lab sessions, in 2018, it is clear from our side that students find parallel programming much more attractive and fun. Graphical tools make their debugging sessions less painful and more effective. The fact that it took postgraduate students less than a dozen hours to come up with an efficient MPI+OpenMP implementation of the Game of Life kernel using lazy evaluation (see Section III-D) impressed us. This highlights the importance of allowing students to quickly prototype preliminary variants of the code and analyze their parallel behavior interactively.

On the downside, EASYPAP provides the students with an integrated environment where all the low-level details of configuration, compilation and initialization of various components are hidden. So it is necessary to conduct more conventional lab assignments as well, involving writing small applications from scratch to show student how a complete OpenCL program looks like for instance.

IV. RELATED WORK

There have been many contributions to the field of developping programming environments for teaching parallel programming [START_REF] Adams | Patterns and exemplars: Compelling strategies for teaching parallel and distributed computing to cs undergraduates[END_REF], [START_REF] Danner | Paravis: A library for visualizing and debugging parallel applications[END_REF].

Like the authors of [START_REF] Adams | Patterns and exemplars: Compelling strategies for teaching parallel and distributed computing to cs undergraduates[END_REF], we are convinced by the pedagogical benefits of using exemplars. EASYPAP is also built around the notion of exemplars that students can parallelize using multiple paradigms.

Regarding visualization, we adhere to the same philosophy as the ParaVis [START_REF] Danner | Paravis: A library for visualizing and debugging parallel applications[END_REF] and TSGL [START_REF] Adams | Tsgl a thread safe graphics library for visualizing parallelism[END_REF] efforts, which provide easy-to-use C/C++ interfaces to visualize 2D animations produced by parallel computations. These libraries are versatile and can be interfaced with almost any existing 2D simulation. EASYPAP follows a different approach by providing an integrated educational framework with monitoring and trace exploration capabilities, experience automation and plot generation assistance.

Many outstanding tools have been developed to visualize and analyze execution traces, such as Aftermath [START_REF] Drebes | Language-centric performance analysis of openmp programs with aftermath[END_REF], Grain Graphs [START_REF] Muddukrishna | Grain graphs: Openmp performance analysis made easy[END_REF], Intel Vtune Profiler [START_REF]Intel Vtune Profiler[END_REF], TAU [START_REF] Shende | The tau parallel performance system[END_REF], Vampir [START_REF] Knüpfer | The vampir performance analysis tool-set[END_REF] or ViTE [START_REF]ViTE: Visual trace explorer[END_REF]. We think EASYPAP represents a smooth and attractive first contact with trace analysis tools, before beeing introduced to more complex ones. An original aspect of both EASYPAP and EASYVIEW is that they establish a graphical link between computations (i.e. tasks) and their associated data (i.e. image tile).

V. CONCLUSION AND FUTURE WORK

EASYPAP is a framework designed to make learning parallel programming more accessible and attractive to students. A comprehensive set of tools allows to quickly get visual feedback about the parallel behavior of their code, to analyze the locality of the computations, and to understand performance issues. The use of EASYPAP during undergraduate and postgraduate courses on parallel programming at University of Bordeaux was very successful. Students were able to understand very subtle aspects of scheduling, synchronization and compiler optimizations. We have also used EASYPAP to popularize parallel programming for middle school students. It made it possible to easily illustrate concepts such as load imbalance.

Currently, EASYPAP only partially supports OpenCL: users can observe animated output of kernels, but monitoring and trace exploration are not yet implemented. These features will soon be developed by leveraging OpenCL profiling events. In a near future, we also intend to further extend the EASYVIEW trace explorer to integrate per-task cache usage information using the PAPI library [START_REF] Terpstra | Collecting performance data with papi-c[END_REF].

Fig. 1 voidFigure 1 .

 11 Figure 1. Sequential version of kernel mandel

Figure 3 .

 3 Figure 3. The monitoring mode displays two additionnal windows: a tiling window (top) and a CPU monitoring window.

Figure 4 .

 4 Figure 4. During execution, students observe how the OpenMP loop scheduling policy impacts the assignment of tiles to threads. Note that (a) reflects a steady state, whereas (b), (c) and (d) are dynamically changing between iterations.

Figure 5 .Figure 6 .

 56 Figure 5. Typical experiments automation script

Figure 7 .

 7 Figure 7. EASYVIEW brings interactive exploration of traces. Moving the mouse over a task in the Gantt diagram displays its duration (bubble at top of window). Tasks intersecting the mouse x-axis have their corresponding tile highlighted over the image thumbnail, allowing to link computations and their data.

  Figure 9. In "heat map" mode, the brightness of tiles displayed in the Tiling Window reflects the duration of the corresponding tasks: the brighter an area is, the more time-consuming it is. On picture (a) we can distinguish the shape of the Mandelbrot set as depicted in Fig.8. On picture (b), we observe that border tiles take a longer time to be processed than inner tiles.

Figure 10 .

 10 Figure 10. Comparison of two execution traces of the blur kernel using EASYVIEW. The bottom trace corresponds to the execution of a basic OpenMP implementation using uniform tiles. The top trace corresponds to an optimized OpenMP version where conditional code was removed from inner tiles. This later version is approximately 3 times faster in this setup (iteration 3 with the basic version is as long as iterations [7..9] with the optimized version).

Figure 11 .

 11 Figure 11. Snippet showing the implementation of the downright propagation using OpenMP tasks with dependencies.

Figure 12 .

 12 Figure 12. EASYVIEW allows to visualize the wave of tasks moving forward during the execution of code depicted in Fig.11. These three screenshots were taken while moving the mouse from left to right over the Gantt window.

Figure 13 .

 13 Figure 13. When launched in debugging mode, monitoring windows of every MPI process show up and help to visualize which area is processed by each of them.


Alexandre Araujo

Laurent Meunier

Rafael Pinot

Benjamin Negrevergne

Robust Neural Networks using Randomized Adversarial Training

This paper tackles the problem of defending a neural network against adversarial attacks crafted with different norms (in particular ∞ and 2 bounded adversarial examples). It has been observed that defense mechanisms designed to protect against one type of attacks often offer poor performance against the other. We show that ∞ defense mechanisms cannot offer good protection against 2 attacks and vice-versa, and we provide both theoretical and empirical insights on this phenomenon. Then, we discuss various ways of combining existing defense mechanisms in order to train neural networks robust against both types of attacks. Our experiments show that these new defense mechanisms offer better protection when attacked with both norms.

Introduction

Deep neural networks achieve state of the art performances in a variety of domains such as natural language processing [START_REF] Radford | Language models are unsupervised multitask learners[END_REF], image recognition [START_REF] He | Deep residual learning for image recognition[END_REF] and speech recognition [START_REF] Hinton | Deep neural networks for acoustic modeling in speech recognition[END_REF]. However, it has been shown that such neural networks are vulnerable to adversarial examples, i.e. imperceptible variations of natural examples, crafted to deliberately mislead the models [START_REF] Globerson | Nightmare at test time: robust learning by feature deletion[END_REF][START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. Since their discovery, a variety of algorithms have been developed to generate adversarial examples (a.k.a. attacks), for example FGSM [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF], PGD [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] and C&W [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF], to mention the most popular ones.

Because it is difficult to characterize the space of visually imperceptible variations of a natural image, existing adversarial attacks use surrogates that can differ from one attack to another. For example, [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] use the ∞ norm to measure the distance between the original image and the adversarial image whereas Carlini and * Contact Author [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] use the 2 norm. When the input dimension is low, the choice of the norm is of little importance because the ∞ and 2 balls overlap by a large margin, and the adversarial examples lie in the same space. An important insight in this paper is to observe that the overlap between the two balls diminishes exponentially quickly as the dimensionality of the input increases. For typical image datasets with large dimensionality, the two balls are mostly disjoint. As a consequence, the ∞ -bounded and the 2 -bounded adversarial examples lie in different area of the space, and it explains why ∞ defense mechanisms perform poorly against 2 attacks and vice-versa.

We show that this insight is crucial to design defense mechanisms that are robust against both types of attacks, and we advocate for the design of models that incorporate defense mechanisms against both ∞ and 2 attacks. Then we evaluate strategies (existing and new ones) to mix up existing defense mechanisms. In particular, we evaluate the following strategies:

(a) Mixed Adversarial Training (MAT), a training procedure inspired by Adversarial Training [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF]. It is based on augmenting training batches using both ∞ and 2 adversarial examples. This method defends well against both norms for PGD attacks, but fails against C&W attacks. (b) Mixed noise injection (MNI), a technique that consists in noise injection at test time [START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF][START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF]. We evaluate different noises and their mixture. This method defends better against C&W attacks, but does not obtain good results against PGD attacks for ∞ norm. (c) Randomized Adversarial Training (RAT), a solution to benefit from the advantages of both ∞ adversarial training, and 2 randomized defense. As we will show, RAT offers the best trade-off between defending against PGD and C&W attacks. The rest of this paper is organized as follows. In Section 2, we recall the principle of existing attacks and defense mechanisms. In Section 3, we conduct a theoretical analysis to show why the ∞ defense mechanisms cannot be robust against 2 attacks and vice-versa. We then corroborate this analysis with empirical results using real adversarial attacks and defense mechanisms. In Section 4, we discuss various strategies to mix defense mechanisms, conduct comparative experiments, and discuss the performance of each strategy.

Preliminaries on Adversarial Attacks and Defense Mechanisms

Let us first consider a standard classification task with an input space X = [0, 1] d of dimension d, an output space Y = [K] and a data distribution D over X × Y. We assume the model f θ has been trained to minimize a loss function L as follows:

min θ E (x,y)∼D [L(f θ (x), y)] . (1)
In this paper, we consider N -layers neural network models, therefore the model is a composition of N non-linear parametric functions φ θi (i.e.

f θ = φ (N ) θ N • • • • • φ (1)
θ1 ).

Adversarial attacks

Given an input-output pair (x, y) ∼ D, an adversarial attack is a procedure that produces a small perturbation τ ∈ X such that f θ (x + τ ) = y. To discover the damaging perturbation τ of x, existing attacks can adopt one of the two following strategies: (i) maximizing the loss L(f θ (x + τ ), y) under some constraint on τ p , with p ∈ {0, • • • , ∞} (a.k.a. loss maximization); or (ii) minimizing τ p under some constraint on the loss L(f θ (x + τ ), y) (a.k.a. perturbation minimization).

(i) Loss maximization. In this scenario, the procedure maximizes the loss objective function, under the constraint that the p norm of the perturbation remains bounded by some value , as follows:

argmax τ p ≤ L(f θ (x + τ ), y).
(2)

The typical value of depends on the value p of the norm • p considered in the problem setting. In order to compare ∞ and 2 attacks of similar strength, we choose values of ∞ and 2 (for ∞ and 2 norms respectively) which result in ∞ and 2 balls of equivalent volumes. For the particular case of CIFAR-10, this would lead us to choose ∞ = 0.03 and 2 = 0.8 which correspond to the maximum values chosen empirically to avoid the generation of visually detectable perturbations. The current state-of-the-art method to solve Problem (2) is based on a projected gradient descent (PGD) [START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF] of radius . Given a budget , it recursively computes

x t+1 = Bp(x, ) x t + α argmax δ s.t. ||δ||p≤1 ∆ t |δ (3)
where

B p (x, ) = {x + τ s.t. τ p ≤ }, ∆ t = ∇ x L (f θ (x t ) , y)
, α is a gradient step size, and S is the projection operator on S. Both PGD attacks with p = 2, and p = ∞ are currently used in the literature as state-of-the-art attacks for the loss maximization problem.

(ii) Perturbation minimization. This type of procedures search for the perturbation that has the minimal p norm, under the constraint that L(f θ (x + τ ), y) is bigger than a given bound c:

argmin L(f θ (x+τ ),y)≥c τ p .
(4)

The value of c is typically chosen depending on the loss function L. For example, if L is the 0/1 loss, any c > 0 is acceptable. Problem (4) has been tackled by [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF], leading to the strongest method known so far. (Denoted C&W attack in the rest of the paper.) It aims at solving the following Lagrangian relaxation of Problem (4):

argmin τ τ p + λ × g(x + τ ) (5)
where

min θ E (x,y)∼D max τ p ≤ L (f θ (x + τ ), y) . (6)
In the case where p = ∞, this technique offers good robustness against ∞ attacks [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF]. AT can also be performed using other kinds of attacks (including strong 2 attacks such as C&W albeit at a much higher computational cost). However, as we will discuss in Section 3, ∞ adversarial training offers poor protection against 2 adversarial attacks and vice-versa. Noise injection mechanisms. Another important technique to design robust models against adversarial attacks is to inject noise in the model. Injecting a noise vector η at inference time results in a randomized neural network fθ := f θ (x + η).

In contrast with Adversarial Training, noise injection mechanisms are, in certain cases, provably robust against adversarial examples as discussed by [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF]; [START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF]. Empirical results have also demonstrated their efficiency against 2 adversarial attacks [START_REF] Siraj Rakin | Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack[END_REF]. These works focus however on Gaussian and Laplace distributions a.k.a generalized Gaussian of order 2, and 1 respectively. As the limit of a generalized Gaussian density (Dytso et al., 2018) when p → ∞ is a Uniform distribution, we also investigate the injection of uniform noise to defend against ∞ attacks.

3 No Free Lunch for adversarial defenses

Theoretical analysis

Let us consider a classifier f ∞ equipped with an ideal defense mechanism against adversarial examples bounded with an ∞ norm of value ∞ . It guarantees that for any input-output pair (x, y) ∼ D and for any perturbation

τ such that τ ∞ ≤ ∞ , f ∞ is not misled by the perturba- tion (i.e. f ∞ (x + τ ) = f ∞ (x))
. We now focus our study on the performance of this classifier against adversarial examples bounded with an 2 norm of value 2 .

Using Figure 1(a), we observe that any 2 adversarial example that is also in the ∞ ball, is guaranteed to be protected by the ∞ defense mechanism of f ∞ , but not if it is outside the ∞ ball. To characterize the probability that an 2 perturbation is guaranteed to be protected by an ∞ defense mechanism in the general case (i.e. any dimension d), we measure the ratio between the volume of intersection of the ∞ ball of radius ∞ and the 2 ball of radius 2 . As Theorem 1 shows, this ratio depends on the dimensionality d of the input vector x, and rapidly converges to zero when d increases. Therefore a defense mechanism that protects against all ∞ bounded adversarial examples, is unlikely to be efficient against 2 attacks. Theorem 1 (Probability of the intersection goes to 0). Let B 2,d , and B ∞,d be two d dimensional balls, respectively for 2 norm and ∞ norm. If for all d, one constrains B 2,d , and B ∞,d to have the same volume, then

Vol(B 2,d B ∞,d ) Vol(B ∞,d ) → 0 when d → ∞.
Proof. Without loss of generality, let us fix the radius of the ∞ ball to 1 (denoted B ∞,d (1)). One can show that for all d,

Vol (B 2,d (r 2 (d))) = Vol (B ∞,d (1)). Where r 2 (d) = 2 √ π Γ( d 2 + 1) 1/d , Γ is the gamma function, and B 2,d (r 2 (d)) is the 2 ball of radius r 2 (d). Then, thanks to Stirling's formula, r 2 (d) ∼ 2 πe d 1/2
. Finally, if we denote U S , the uniform distribution on set S, by using Hoeffding inequality between Eq. ( 9) and (10), we get:

Vol(B 2,d (r 2 (d)) B ∞,d (1)) Vol(B ∞,d (1)) (7) =P x∼U B ∞,d (1) [x ∈ B 2,d (r 2 (d))] (8) =P x∼U B ∞,d (1) d i=1 |x i | 2 ≤ r 2 2 (d) (9) ≤ exp -d -1 r 2 2 (d) -dE|x 1 | 2 2 (10) ≤ exp - 2 πe - 2 3 2 d + o(d) . (11)
Then the ratio between the volume of intersection of the ball and the volume of the ball converges towards 0.

Theorem 1 states that, when d is large enough, 2 bounded perturbations have a null probability of being also in the ∞ ball of the same volume. As a consequence, for any value of d that is large enough, a defense mechanism that offers full protection against ∞ adversarial examples is not guaranteed to offer any protection against 2 attacks, and vice-versa1 .

Remark that this result defeats the 2-dimensional intuition: if we consider a 2 dimensional problem setting, the ∞ and the 2 balls have an important overlap (as illustrated in Figure 1(a)) and the probability of sampling in the intersection of the two balls is bounded by approximately 98%. However, as we increase the dimensionality d, this probability quickly becomes negligible, even for very simple image datasets such as MNIST. An instantiation of the bound for classical image datasets is presented in Table 1. The probability of sampling in the intersection of the ∞ and 2 balls is close to zero for any realistic image setting. In large dimensions, the volume of the corner of the ∞ ball is much bigger than it appears in Figure 1(a).

No Free Lunch in practice

Our theoretical analysis shows that if adversarial examples were uniformly distributed in a high dimensional space, then any mechanism that perfectly defends against ∞ adversarial examples has a null probability of protecting against 2 -bounded adversarial attacks and vice-versa. Although existing defense mechanisms do not necessarily assume such a distribution of adversarial examples, we demonstrate that whatever distribution they use, it offers no favorable bias w.r.t the result in Theorem 1. As we discuss in Sec. 2, there are two distinctive attack settings: loss maximization (PGD) and perturbation minimization (C&W). We analyse the first setting in details and conduct a second series of experiments to demonstrate that the results are similar if we consider the second setting.

Adversarial training vs. loss maximization attacks

To demonstrate that ∞ adversarial training is not robust against PGD-2 attacks, we measure the number of 2 adversarial examples generated with PGD-2 , lying outside the ∞ ball. (Note that we consider all examples, not just the ones that successfully fool the classifier). To do so, we use the same experimental setting as in Section 4 with ∞ and 2 such that the volumes of the two balls are equal. Additionally, we also measure the average ∞ and 2 norms of these adversarial examples, to understand more precisely the impact of adversarial training, and we report the accuracy, which reflects the number of adversarial examples that successfully fooled the classifier (cf. The results are unambiguous: none of the adversarial examples generated with PGD-2 are inside the ∞ ball (and thus in the intersection of the two balls). As a consequence, we cannot expect adversarial training ∞ to offer any guaranteed protection against 2 adversarial examples. We illustrate this phenomenon using Figure 1 (b): notice that the 2 adversarial example represented in this figure cannot be protected using ∞ adversarial training which is only designed to push the decision boundary (red line) outside of the ∞ ball (square), but not outside of the 2 ball (circle). Our results demonstrate that all PGD-2 examples are already in this upper area (outside the intersection), before ∞ adversarial training. Therefore ∞ adversarial training is unnecessary.

The second experiment naturally demonstrates a similar behaviour. We first observe that adversarial examples generated with PGD-∞ lying outside the 2 ball cannot be eliminated using 2 adversarial training (as illustrated in Figure 1 (c)). However, Table 2 shows that all examples are already outside the 2 ball, clustered around the corner of the ∞ ball (average distance is 1.64 compared to 0.031 × √ 3072 = 1.71 for the corner). Therefore, any defense method (including 2 adversarial training) that would eliminate only adversarial examples inside the 2 balls, cannot be efficient against ∞ adversarial examples.

The comparison of accuracy under PGD-2 attack of a classifier defended by either ∞ or 2 adversarial training corroborate our analysis. In fact, when defended with AT-∞ the accuracy of the classifier under attack is 0.37, while the AT-2 defends the classifier up to 0.52 i.e. 40% better. Similarly, a classifier defended with AT-∞ with an accuracy under PGD-∞ attack of 0.43 performs 16% better than the one defended with AT-2 which obtains 0.37 accuracy under attack. These results keep confirming our claim: 2 -based defenses are inadequate to defend against ∞ attacks, and vice-versa.

Building Defenses against Multiple Adversarial Attacks

So far, we have shown that adversarial defenses are able to protect only against the norm they have been trained on. In order to solve this problem, we propose several strategies to build defenses against multiple adversarial attacks. These strategies are based on the idea that both types of defense must be used simultaneously in order for the classifier to be protected against multiple attacks. In this section we evaluate several of these defense strategies, and compare them against state-of-the-art attacks using a solid experimental setting (the detailed description of the experimental setting is described in Section 4.4).

MAT -Mixed Adversarial Training

Earlier results have shown that ATp improves the robustness against corresponding p -bounded adversarial examples, and the experiments we present in this section corroborate this observation (See As discussed by [START_REF] Tramèr | Adversarial training and robustness for multiple perturbations[END_REF], there are several possible strategies to mix the adversarial training examples. The first strategy (MAT-Rand) consists in randomly selecting one adversarial example among the two most damaging ∞ and 2 , and to use it as a training example, as described in Equation 12: MAT-Rand:

min θ E (x,y)∼D E p∼U ({2,∞}) max τ p ≤ L (f θ (x + τ ), y) .
(12) An alternative strategy is to systematically train the model with the most damaging adversarial example ( ∞ or 2 ). As described in Equation 13: MAT-Max:

min θ E (x,y)∼D max p∈{2,∞} max τ p ≤ L (f θ (x + τ ), y) . (13)
The accuracy of MAT-Rand and MAT-Max are reported in Table 4 (Column: MAT). As expected, we observe that MAT-Rand and MAT-Max offer better robustness both against PGD-2 and PGD-∞ adversarial examples than the original AT does. More generally, we can see that AT is a good strategy against loss maximization attacks, and thus it is not surprising that MAT is a good strategy against mixed loss maximization attacks. However, AT is very weak against perturbations minimization attacks such as C&W, and MAT is no better against such attacks. This weakness makes MAT of little practical use.

MNI -Multiple Noise Injection

Another important technique to defend against adversarial examples is to use Noise Injection (NI). [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF] demonstrated that injecting noise in the network can give provable defense against adversarial examples. Furthermore, we found that NI offers better protection than AT against perturbation minimization attacks such as C&W, thus, they are good candidates to obtain models robust to multiple attacks. In this work, besides the generalized Gaussian noises, already investigated in previous works, we evaluate the efficiency of uniform distributions which are generalized Gaussian of order ∞. As shown in Table 4 (Columns: NI), noise injection from this distribution gives better results than Gaussian noise injection against all the attacks except PGD-∞ .

To obtain the best out of both noises, we propose to combine them (MNI) either by convolution (Conv) or by mixture (Mix). Hence, the final noise vector comes from one of the following probability density functions: MNI-Conv:

1 2πσ 2 1 exp -x 2 2σ 2 1 * 1{|x| ≤ σ 2 } 2σ 2 (14)
MNI-Mix:

1 8πσ 2 1 exp -x 2 2σ 2 1 + 1{|x| ≤ σ 2 } 4σ 2 . ( 15
)
Following the literature [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF], we choose σ 1 = 0.25. Accordingly, we take σ 2 = 0.2. The results are presented in Table 4 (Column: MNI). We found that MNI offers comparable results against the experimental setting in [START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF], but does not improve over NI with a uniform distribution.

RAT -Randomized Adversarial Training

We now examine the performance of Randomized Adversarial Training (RAT) which mixes Adversarial Training with Noise Injection. We consider the two symmetric settings: RAT-∞ and a noise from a normal distribution, as well as RAT-2 and a noise from a uniform distribution. The corresponding loss function is defined as follows:

min θ E (x,y)∼D max τ ≤ L fθ (x + τ ), y) . ( 16
)
where fθ is a randomized neural network with noise injection as described in Section 2.2. The results of RAT are reported in Table 4 (Columns: RAT-∞ and RAT-2 ). We can observe that the first setting offers the best extra robustness, which is consistent with previous experiments, since AT is generally more effective against ∞ attacks whereas NI is more effective against 2 -attacks. Overall, RAT-∞ and a noise from uniform distribution offer the best minimal robustness with at least 0.39 accuracy, 16 points above the second best (NI with noise from a normal distribution, with 0.22).

Experimental setting

To compare the robustness provided by the different defense mechanisms, we use strong adversarial attacks and a conservative setting: the attacker has a total knowledge of the parameters of the model (white-box setting) and we only consider untargeted attacks (a misclassification from one target to any other will be considered as adversarial).

To evaluate defenses based on noise injection, we use Expectation Over Transformation (EOT), the rigorous experimental protocol proposed by [START_REF] Athalye | Synthesizing robust adversarial examples[END_REF] and later used by [START_REF] Athalye | Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples[END_REF]; [START_REF] Carlini | On evaluating adversarial robustness[END_REF] to identify flawed defense mechanisms.

To attack the models, we use state-of-the-art algorithms PGD and C&W (see Section 2). We run PGD with 20 iterations to generate adversarial examples and with 10 iterations when it is used for adversarial training. We run C&W with 60 iterations to generate adversarial examples. For bounded attacks, the maximum ∞ bound is fixed to 0.031 and the maximum 2 bound is fixed to 0.83. As discussed in Section 2, we chose these values so that the ∞ and the 2 balls have similar volumes. Note that 0.83 is slightly above the values typically used in previous publications in the area, meaning the attacks are stronger, and thus more difficult to defend against.

All experiments are conducted on CIFAR-10 with the Wide-Resnet 28-10 architecture. We use the training procedure and the hyper-parameters described in the original paper by [START_REF] Zagoruyko | Wide residual networks[END_REF]. Training time varies from 1 day (AT) to 2 days (MAT) on 4 GPUs-V100 servers.

Related Work

Adversarial attacks have been an active topic in the machine learning community since their discovery [START_REF] Globerson | Nightmare at test time: robust learning by feature deletion[END_REF][START_REF] Biggio | Evasion attacks against machine learning at test time[END_REF][START_REF] Szegedy | Intriguing properties of neural networks[END_REF]. Many attacks have been developed. Most of them solve a loss maximization problem with either ∞ [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF][START_REF] Kurakin | Adversarial examples in the physical world[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF], 2 [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Kurakin | Adversarial examples in the physical world[END_REF][START_REF] Madry | Towards deep learning models resistant to adversarial attacks[END_REF], 1 [START_REF] Tramèr | Adversarial training and robustness for multiple perturbations[END_REF] or 0 [START_REF] Papernot | The limitations of deep learning in adversarial settings[END_REF] surrogate norms.

Defending against adversarial examples is a challenging problem since the number of layers makes it difficult to understand the geometry of the decision boundary. Despite empirically proven efficient, Adversarial training [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] gives no formal defense guarantees. Besides this line of work, randomization and smoothing [START_REF] Xie | Mitigating adversarial effects through randomization[END_REF][START_REF] Lecuyer | Certified robustness to adversarial examples with differential privacy[END_REF][START_REF] Pinot | Theoretical evidence for adversarial robustness through randomization[END_REF][START_REF] Cohen | Certified adversarial robustness via randomized smoothing[END_REF] have gained popularity since they provide guarantees, but so far, the efficiency of these methods remains limited against ∞ -based attacks.

An open question so far is to build an efficient defense against multiple norms. Concurrently to our work, [START_REF] Tramèr | Adversarial training and robustness for multiple perturbations[END_REF] proposed to tackle this issue by mixing randomized training with attacks for different norms to defend against multiple perturbations. Then, [START_REF] Hadi Salman | Provably robust deep learning via adversarially trained smoothed classifiers[END_REF] proposed to mix adversarial training with randomized smoothing to have better certificates against adversarial attacks. These methods are closely related respectively to MAT and RAT. Aside from these similarities, we propose a new geometric point of view for robustness against multiple perturbations, that is backed up theoretically and experimentally. We also conduct a rigorous and full comparison of RAT and MAT as defenses against adversarial attacks. Finally, we propose MNI, that adds mixture of noise to our network and gets promising results. To the best of our knowledge, this is the first work that covers mixtures and convolution of noises with different natures.

Conclusion

In this paper, we tackle the problem of protecting neural networks against multiple attacks crafted from different norms. First, we demonstrate that existing defense mechanisms can only protect against one type of attacks. Then we consider a variety of strategies to mix defense mechanisms and to build models that are robust against multiple adversarial attacks. We show that Randomized Adversarial Training offers the best global performance.

Figure 1 :

 1 Figure 1: Left: 2D representation of the ∞ and 2 balls of respective radius and . Middle: a classifier trained with ∞ adversarial perturbations (materialized by the red line) remains vulnerable to 2 attacks. Right: a classifier trained with 2 adversarial perturbations (materialized by the blue line) remains vulnerable to ∞ adversarial examples.

Table 1 :

 1 Bounds of Theorem 1 on the volume of intersection of 2 and ∞ balls at equal volume for typical image classification datasets. When d = 2, the bound is 10 -0.009 ∼ 0.98.

	Dataset	d	Inter. (in log 10 )
	-	2	-0.009
	MNIST	784	-144
	CIFAR	3072	-578
	ImageNet 150528	-28946

Table 2 (

 2 top)). The same experiment is conducted for 2 adversarial training against PGD-∞ and the results are presented in Table 2 (bottom). All experiments in this section are conducted on CIFAR-10, and the experimental setting is fully detailed in Section 4.4.

	PGD-2 vs. →	Unprotected AT-∞
	Examples inside 2 ball	100%	100%
	Average 2 norm	0.83	0.83
	Examples inside ∞ ball	0%	0%
	Average ∞ norm	0.075	0.2
	Accuracy under attack	0.00	0.37
	PGD-∞ vs. →	Unprotected	AT-2
	Examples inside 2 ball	100%	100%
	Average 2 norm	1.4	1.64
	Examples inside ∞ ball	0%	0%
	Average ∞ norm	0.031	0.031
	Accuracy under attack	0.00	0.37

Table 2 :

 2 (Top) number of PGD-2 adversarial examples inside the ∞ and inside the 2 ball, without and with ∞ adversarial training. (Bottom) number of PGD-∞ adversarial examples inside the ∞ and inside the 2 ball, without and with 2 adversarial training. On CIFAR-10 (d = 3072).

Table 3 :

 3 This table shows the amount of adversarial examples inside the ∞ ball and inside the 2 ball but outside the ∞ ball. We can observe a clear shift between a baseline model (no defense) and a model trained with Adversarial Training PGD ∞ attacked with C&W attack[START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF]. inside the intersection of the two balls. This phenomenon is due to the fact that C&W minimizes the 2 norm of the perturbation. Therefore without AT, the attack is able to discover adversarial examples that are very close to the original image, where the ∞ and the 2 balls overlap. When the model is trained with AT-∞ , we observe a clear shift: 71% of the examples are now outside the ∞ , but still inside the 2 ball, as illustrated in Figure 1 (b). This means that C&W attack still minimizes the 2 norm of the perturbation while updating its search space to ignore the examples in the ∞ ball. Since C&W was always able to discover adversarial examples in this area, AT-∞ offers no extra benefit in terms of robustness (0% Accuracy). Together, these results and Theorem 1 confirm that ∞ -based defenses are vulnerable to 2 -based perturbation minimization attacks.

		Unprotected AT-∞
	Examples inside intersection	70%	29%
	Examples outside intersection	30%	71%
	Accuracy under attack	0.00	0.00
	Adversarial training vs. perturbation minimization
	attacks. We now study the performances of an 2 per-
	turbation maximization attack (C&W) with and without
	AT-

∞ . It allows us to understand in which area C&W discovers adversarial examples and the impact of AT-∞ . The results are reported in Table 3. First, when the classifier is undefended, we observe that 70% of adversarial examples lie

  Table 4, column: AT). Building on this observation, it is natural to examine the efficiency of Mixed Adversarial Training (MAT) against mixed ∞ and 2 attacks. MAT is a variation of AT that uses both ∞ -bounded adversarial examples and 2 -bounded adversarial examples as training examples.

Table 4 :

 4 This table shows a comprehensive list of results consisting of the accuracy of several defense mechanisms against 2 and ∞ attacks. This table main objective is to compare the overall performance of 'single' norm defense mechanisms (AT and NI presented in the Sec. 2.2) against mixed norms defense mechanisms (MNI, MAT & RAT mixed defenses presented in Sec. 4). The red values present all accuracy below 30% which shows that all defense mechanisms have 'weaknesses' with the exception of RAT.

		Baseline	AT		MAT	NI			MNI	RAT-∞	RAT-2
		-	∞	2	Max	Rand	N	U	Mix	Conv	N	U	N	U
	Natural examples	0.94	0.85	0.85	0.80	0.80	0.79	0.87	0.84	0.79	0.74	0.80	0.79	0.87
	PGD-∞ 20	0.00	0.43	0.37	0.37	0.40	0.23	0.22	0.19	0.20	0.35	0.40	0.23	0.22
	PGD-2 20	0.00	0.37	0.52	0.50	0.55	0.34	0.36	0.33	0.32	0.43	0.39	0.34	0.37
	C&W-2 60	0.00	0.00	0.00	0.00	0.00	0.33	0.53	0.41	0.32	0.30	0.41	0.33	0.34
	Min Accuracy	0.00	0.00	0.00	0.00	0.00	0.23	0.22	0.19	0.20	0.30	0.39	0.23	0.22

Th. 1 can easily be extended to any two balls with different norms. For clarity, we restrict to the case of ∞ and

norms.