Jordi Molgó
email: jordi.molgo@inaf.cnrs-gif.fr

Evelyne Benoit

Rómulo Aráoz

Armen Zakarian

Bogdan I Iorga
email: bogdan.iorga@cnrs.fr

SPIROLIDES AND OTHER CYCLIC IMINE TOXINS: TOXICOLOGICAL PROFILE

In this chapter, we review available evidence on the toxicological profile of spirolides and other lipophilic cyclic imine toxins, highlighting their chemical structure, the phytoplankton species involved in their production, their pharmacokinetics/toxicokinetics and experimental toxicity, and their molecular targets and mechanisms of action. These phycotoxins belong to an emerging class of chemical agents associated with marine algal blooms and shellfish toxicity. Their chemical structure is represented by a macrocycle, with the ring size between 14 and 27, and two conserved features that include the cyclic imine group and spiroketal ring system. The producers of spirolides, gymnodimines and pinnatoxins have been identified as being the dinoflagellates Alexandrium ostenfeldii/peruvianum, Karenia selliformis and Vulcanodinium rugosum. Their acute toxicity, appraised by the mouse bioassays, classifies them as "fast-acting" toxins because they induced rapid onset of neurological symptoms followed by death within a few minutes. The spirolide congeners are the most toxic after intraperitoneal injection, while there are indications that pinnatoxins are the most toxic group after oral administration. The neurotoxic effects reported for these phycotoxins are mostly due to their specific interaction with the muscle-and neuronal-types of nicotinic acetylcholine receptors which are the principal molecular targets of spirolides, gymnodimines and pinnatoxins, so far studied. Hence, these phycotoxins exhibit both high affinity and broad specificity on nicotinic receptors, indicating that their sites of interaction in the receptors include amino acid residues highly conserved among animal species.

INTRODUCTION

Spirolides were first discovered in the 1990s from the Atlantic coast of Nova Scotia in Canada when unusual toxicities were detected by mouse bioassay in mussel and scallop extracts. Since then, the spirolides have been found to have a worldwide distribution, with a remarkable diversity in their chemical profiles. Spirolides belong to the so-called cyclic imine group of neurotoxins that also include the gymnodimines, pinnatoxins, pteriatoxins, portimine, spiro-prorocentrimine and prorocentrolides (for a recent review, see [START_REF] Stivala | Synthesis and biology of cyclic imine toxins, an emerging class of potent, globally distributed marine toxins[END_REF]. The generic chemical structure of the cyclic imine toxins is represented by a macrocycle, with the ring size between 14 and 27, and two conserved features that include two subunits: the cyclic imine group (which in most cases is found as a spiroimine) and the spiroketal ring system. The cyclic imine, which has been shown to be a key pharmacophoric feature, can be a 5-membered (portimine), 6-membered (gymnodimines, spiroprorocentrimine, prorocentrolides) or 7-membered (spirolides, pinnatoxins, pteriatoxins) ring, that in many cases is substituted with one or two methyl groups. The spiroketal ring system can be a simple tetrahydrofuran (i.e. portimine, gymnodimine) or tetrahydropyran group (i.e. spiroprorocentrimine, prorocentrolides), but also a more complex 6,5-(spirolides H and I), 6,6,5-(spirolide G), 6,5,5-(spirolides A-F) or 6,5,6-spiroketal (pinnatoxins, pteriatoxins). For each sub-family, the first member is shown and the positions for which variations in the substitution pattern were reported are numbered and highlighted in orange. The characteristic spiroimine and cyclic imine features are colored in blue and magenta, respectively, and the size of the macrocyclic ring system in each sub-family is shown in magenta.

Spirolides and the cyclic imine toxins studied so far are "fast-acting" phycotoxins, because of the rapid beginning of neurological symptoms and the short time to death. The basis for these neurotoxic effects is in direct relation to their molecular antagonist action on nicotinic acetylcholine receptors. In this chapter, we summarize our current understanding of the toxicological profile of spirolides and other cyclic imine toxins, with emphasis on their chemical structure, producing organisms, pharmacokinetics/toxicokinetics, experimental toxicity, molecular targets and mechanism of action.

SPIROLIDES -Chemical characteristics of spirolide toxins

Spirolides constitute the largest family among cyclic imine toxins. As shown in Figure 2, to date sixteen spirolide analogues have been reported, which can be classified into four distinct groups according to the structure of the spiroimine and spiroketal ring systems. The first group of spirolides has 10 members with a characteristic 6,5,5-spiroketal ring system. As compared to spirolides A and B, the spirolides C and D and their analogues have an extra methyl group at the position 31 on the imine ring, which is believed to render them more resistant to enzymatic and acidic hydrolysis [START_REF] Christian | Are spirolides converted in biological systems?-A study[END_REF]. Spirolides A and C and their analogs have a double bond between the positions 2 and 3 on the lactone ring which is not present in the structure of spirolides B and D and their derivatives. Other positions where structural variability was observed for spirolides C and D are 13 and 19 on the spiroketal ring system, with the presence or absence of methyl substituents, and 27, where hydroxy and keto substituents were evidenced in some cases. The second spirolide group contains the biologically inactive spirolides E and F, which present the same chemical features as spirolides A and B, respectively, except for the cyclic imine that becomes an open ring amino ketone (reviewed by [START_REF] Guéret | Spiroimine shellfish poisoning (SSP) and the spirolide family of shellfish toxins: Isolation, structure, biological activity and synthesis[END_REF]. These structural changes suggest that they are probably shellfish metabolites of spirolides A and B. The third spirolide group, represented by spirolide-G and 20-methyl spirolide-G, has a 6,6,5-spiroketal ring system with a hydroxyl group at C17 for which the stereochemistry could not be determined [START_REF] Mackinnon | Spirolides isolated from Danish strains of the toxigenic dinoflagellate Alexandrium ostenfeldii[END_REF]) and a methyl substituent at C20 that is present only in the second member of this group. The rest of the structure is identical with spirolide-C, with the dimethyl substituted 7-membered cyclic imine and the unsaturated lactone ring. The fourth spirolide group includes spirolides H and I whose structures are identical to spirolides C and D, respectively, apart from the presence of a 6,5-instead of the 6,5,5spiroketal ring system [START_REF] Roach | Characterization of a dispiroketal spirolide subclass from Alexandrium ostenfeldii[END_REF]. Spirolide-H is the 2,3-unsaturated counterpart of spirolide I.

-Spirolide producing organisms

The producers of spirolides and other cyclic imine toxins were identified as dinoflagellates, a large and diverse group of marine eukaryotic microalgae known to be the major causative agents of harmful algal blooms. Although many dinoflagellate species can produce various natural toxins that impact humans, those involved in the production of Cyclic Imine toxins are limited to mainly Alexandrium, Karenia (formely Gymnodinium), Vulcanodinium and Prorocentrum species (for reviews, see Molgó et al., 2014a;[START_REF] Stivala | Synthesis and biology of cyclic imine toxins, an emerging class of potent, globally distributed marine toxins[END_REF].

All known spirolides have been so far linked to Alexandrium species, and differences in spirolides content and profiles have been reported for dinoflagellate strains from different coastal regions.

Hence, the globally distributed species Alexandrium ostenfeldii [START_REF] Cembella | Spirolide composition of micro-extracted pooled cells isolated from natural plankton assemblages and from cultures of the dinoflagellate Alexandrium ostenfeldii[END_REF] was formerly identified as the biological source of all known spirolides (Table 1). However, during the past six years, another dinoflagellate species, Alexandrium peruvianum, was also recognized as the producing organism for some spirolide congeners in Northwestern Mediterranean Sea waters, Irish coastal waters [START_REF] Touzet | Morphogenetic diversity and biotoxin composition of Alexandrium (dinophyceae) in Irish coastal waters[END_REF] and in the United States coastal waters (see Table 1). It is worth noting that spirolides E and F were not detected in dinoflagellates, but were attributed to be metabolically modified products present in shellfish [START_REF] Hu | Characterization of biologically inactive spirolides E and F: Identification of the spirolide pharmacophore[END_REF].

Table 1 near here -Pharmacokinetics/toxicokinetics of spirolide toxins

Among the spirolides, both 13-desmethyl spirolide-C and 13,19-didesmethyl spirolide-C have been shown to cross a Caco-2 cell monolayer [START_REF] Espiña | 13-desmethyl spirolide-c and 13,19didesmethyl spirolide-c trans-epithelial permeabilities: human intestinal permeability modelling[END_REF]. Caco-2 trans-epithelial permeability assays represent a good model to predict the intestinal absorption of drugs and toxins. When administered orally with a single non-lethal dose to mice deprived of food (24 hours before, but given a nutritive solution with salts and glucose) both spirolides were detected in blood and urine samples after 15 minutes and 1 hour of administration, respectively as revealed by liquid chromatography-tandem mass spectrometry [START_REF] Otero | Pharmacokinetic and toxicological data of spirolides after oral and intraperitoneal administration[END_REF]. The 13-desmethyl spirolide-C and 13,19-didesmethyl spirolide-C were also detected 1 hour after intraperitoneal (i.p.) administration in urine, and trace amount of both toxins in blood; after 24 hours important amounts were detected in feces. These results indicate that spirolides are absorbed when administered orally, and are excreted through the urine and feces. The histological evaluation by light microscopy of the heart, kidneys, liver, spleen, stomach, and intestines, performed 24 hours and 7 days after oral administration, did not reveal any observable alteration [START_REF] Otero | Pharmacokinetic and toxicological data of spirolides after oral and intraperitoneal administration[END_REF]. Further studies, using ultra-performance liquid chromatography-mass spectrometry and proton magnetic resonance spectroscopy, have shown that 13-desmethyl spirolide-C is found in the brain a few minutes after its intraperitoneal injection to 3xTg mouse (a well-known model of Alzheimer's disease), and remains measurable even 24 hours post-administration, indicating that the spirolide traverses the blood brain barrier.

Furthermore, a decrease in the intracellular amyloid beta levels in the hippocampus of treated 3xTgmice was observed, emphasizing the positive effects of the molecule in this mouse model [START_REF] Alonso | Benefit of 13-desmethyl spirolide C treatment in triple transgenic mouse model of Alzheimer disease: beta-amyloid and neuronal markers improvement[END_REF].

-Toxicity of spirolide toxins

Although in vitro cytotoxicity tests have been used, the most appropriate assays for appraising the toxicity of Cyclic Imine toxins and estimating their risk to human health are in vivo methods.

Hence, to date, most if not all studies regarding the acute toxicity of cyclic imines have been performed using mainly the mouse bioassays -with synthetic toxins, or toxins extracted from either cultured dinoflagellates or contaminated shellfish, administrated to animals by i.p. injection, gavage administration, or voluntary consumption (feeding) -and determining their 50% lethal dose (LD 50 ) values. It is worth noting that cyclic imine oral administration, with feeding taking priority over gavage, is a more relevant method since, first, it closely reproduces the situation in humans, and second, it avoids the use of high concentrations of solvent (i.e. ethanol) to dissolve the toxins (for reviews, see Molgó et al., 2014a;[START_REF] Stivala | Synthesis and biology of cyclic imine toxins, an emerging class of potent, globally distributed marine toxins[END_REF].

Spirolides, as the other cyclic imines, are classified as "fast-acting" toxins because they induced rapid onset of neurological symptoms -including hyperactivity, jumping, piloerection, hyperextension of the back, stiffening and arching of the tail toward the head, tremors progressing to spams, as well as skeletal muscle paralysis and extension of the hind limbs, respiratory distress with marked abdominal breathing, tremors of the whole body and severe dyspnea -followed by death due to respiratory arrest within 3-50 minutes, whatever the route of administration is. It is worth noting that animals that did not die usually fully recovered with no detectable long-term effects, confirming that the Cyclic Imine toxins are rapidly detoxified and/or excreted (see above).

The LD 50 of some spirolide congeners, after i.p. injection to mice, revealed that these compounds are the most toxic among cyclic imines toxins. In particular, spirolide-C as well as, although contested (Munday et al., 2012;[START_REF] Otero | Pharmacokinetic and toxicological data of spirolides after oral and intraperitoneal administration[END_REF], 13-desmethyl spirolide-C and 20-methyl spirolide-G were the most toxic with LD 50 values as low as 0.007-0.008 mg/kg mouse, followed by 13,19-didesmethyl spirolide-C and spirolide-A (with LD 50 of 0.032 and 0.037 mg/kg mouse, respectively). The other spirolide congeners exhibit an apparently much lower toxicity (Table 2).

Interestingly, the comparison of the acute toxicity of spirolides and their chemical structure points out the importance of some structural requirements, such as the imine ring with the C31 methyl group, and the methyl group at C19, for the toxicological potency of the molecule. Although

showing similar rank orders in the acute toxicity response, the LD 50 of spirolide congeners after oral administration to mice via gavage are higher than those by i.p. injection, and even higher when administered by feeding (Table 2). The significant reduction in oral toxicity versus i.p.

administration of pure spirolide congeners is consistent with the absence of harmful effects in humans consuming shellfish contaminated with these cyclic imine toxins, with the exception of non-specific symptoms, such as gastric distress and tachycardia, reported in Canada [START_REF] Richard | Investigations into the toxicology and pharmacology of spirolides, a novel group of shellfish toxins, in Harmful Algal Blooms[END_REF]. However, the risk to human health of spirolide accumulation in shellfish is evidenced by the two following findings. First, the LD 50 of lipophilic shellfish extracts, containing mainly 13desmethyl spirolide-C, was shown to be higher after i.p. than oral administration to mice [START_REF] Richard | Investigations into the toxicology and pharmacology of spirolides, a novel group of shellfish toxins, in Harmful Algal Blooms[END_REF], and second, examination of the brains of mice and rats after i.p. injection with this spirolide congener revealed neuronal damage in the hippocampus and brain stem, with increased mRNA levels of muscarinic and nicotinic acetylcholine receptors [START_REF] Gill | Neural injury biomarkers of novel shellfish toxins, spirolides: A pilot study using immunochemical and transcriptional analysis[END_REF].

Table 2 near here -Molecular targets and mechanism of action of spirolide toxins

From the large number of compounds found in nature known to interact with nicotinic acetylcholine receptors (nAChRs), including a number of alkaloids and peptides from various origins, only a few are known to be produced by dinoflagellates. Dinoflagellate toxins known to interact with nAChRs include the spirolides, gymnodimines and pinnatoxins (for a recent review, see [START_REF] Stivala | Synthesis and biology of cyclic imine toxins, an emerging class of potent, globally distributed marine toxins[END_REF].

The nAChRs are ligand-gated ion channels mediating fast neurotransmission in the central and peripheral nervous systems (reviewed in [START_REF] Corringer | Nicotinic receptors at the amino acid level[END_REF][START_REF] Albuquerque | Mammalian nicotinic acetylcholine receptors: From structure to function[END_REF]. At the skeletal neuromuscular junction, the nAChR is a pentamer consisting of two α1-subunits, single β1and δ-subunits, and either a γ or a ε subunit in the embryonic or adult receptor-type, respectively (Fig. 3). The neuronal nAChRs in the peripheral and central nervous systems are composed of pentameric combinations of  and  subunits and exhibit a great heterogeneity in subtypes. To date, nine nAChR α subunits (α2-α10) and three nAChR β subunits (β2-β4) have been cloned from distinct animal species. In the peripheral nervous system, the primary subtype is the heteromeric α3β4 subtype, which is present in autonomic ganglia neurons. In addition, the 3 subunit may assemble with α5 and/ or β2 subunit to form a variety of nAChRs. The other major subtype of neuronal nAChR in the peripheral and central nervous system comprised 7 subunits forming a homomeric nAChR, as well as heteromers made up from the combination of different α subunits, or α and β subunits. These arrangements include the  nAChR, which is abundant and widely distributed in the central nervous system, and more complex subunit combinations with the presence of additional subunit types may exist. Thus, a major characteristic of neuronal nAChRs is their wide heterogeneity, based on both diverse stoichiometry and subunit composition.

GYMNODIMINES -Chemical characteristics of gymnodimine toxins

The gymnodimines present a 6-membered cyclic imine, bearing no methyl substituents, as part of the spiroimine ring system, an unsaturated lactone and a tetrahydrofuran ring (Fig. 5). Two groups can be distinguished according to the substitution pattern at the positions C17 and C18. The first gymnodimine group has a methyl substituent at C17 and an E-configured endocyclic double bond between C17 and C18. Gymnodimine-A and its 12-methyl derivative are part of this group. The second gymnodimine group contains gymnodimines B and C, which have an exocyclic double bond at C17 and a hydroxyl substituent at C18 with S and R stereochemistry, respectively.

-Gymnodimine producing organisms

The production of the gymodimines identified so far (gymnodimes-A, -B and -C) was exclusively attributed to the dinoflagellate Karenia selliformis [START_REF] Haywood | Comparative morphology and molecular phylogenetic analysis of three new species of the genus karenia (dinophyceae) from New Zealand[END_REF]) (formely known as Gymnodinium selliforme) in New Zealand and Australia (Table 3). However, the gymnodimine congener 12-methyl gymnodimine was reported as originating from another dinoflagellate in the United States, Alexandrium peruvianum (Van Wagoner et al., 2011) (Table 3), which was proven to also produce some spirolide congeners (see Table 1).

Table 3 near here -Toxicity of gymnodimine toxins

The gymnodimines are among the least toxic cyclic imines according to the mouse bioassay irrespective of the route of administration is (i.p. injection, gavage or feeding administration). Their acute oral toxicity, in particular that of gymnodimine-A, was shown to be between about 8-and more than 78-fold lower than the i.p. toxicity [START_REF] Munday | Acute toxicity of gymnodimine to mice[END_REF]) (Table 4). It is worth noting that gymnodimine-A was reported to be about 10-fold more toxic than gymnodimine-B after i.p. injection to animals. Remarkably, the LD 50 for gymnodimine-A was decreased by about 30-fold when administered to mice by intracerebroventricular administration, as compared to i.p. injection (i.e. 0.003 and 0.08-0.096 mg/kg mouse, respectively [START_REF] Kharrat | The marine phycotoxin gymnodimine targets muscular and neuronal nicotinic acetylcholine receptor subtypes with high affinity[END_REF]) (Table 4). This indicates that gymnodimine-A preferentially affected the rodent central nervous system, further highlighting the risk of the Cyclic Imine toxins to human health may have when accumulated in shellfish at high concentrations.

Table 4 near here -Molecular targets and mechanism of action of gymnodimine toxins

Gymnodimine-A at nanomolar concentrations was reported to produce a concentration-dependent block of twitch tension responses evoked by nerve stimulation in isolated mouse phrenic-nerve hemidiaphragm-muscle preparations, without affecting muscle excitability and directly elicited muscle contraction. Also, gymnodimine-A, according to the nanomolar concentration used, either reduced the amplitude or completely blocked miniature-endplate potentials generated by the release of a single ACh quantum [START_REF] Kharrat | The marine phycotoxin gymnodimine targets muscular and neuronal nicotinic acetylcholine receptor subtypes with high affinity[END_REF]. Similarly, gymnodimine-A blocked the amplitude of endplate potentials without affecting the resting membrane potential of muscle fibers. Such an action prevented endplate potentials to reach the threshold potential for action potential generation in muscles fibers, strongly suggesting that gymnodimine-A blocked endplate nAChRs. Direct evidence for such an action was obtained, using both the patch-clamp technique and iontophoretic ACh pulses of constant duration delivered by a micropipette on the surface of Xenopus skeletal myocytes expressing the embryonic muscle-type nAChR in their surface [START_REF] Kharrat | The marine phycotoxin gymnodimine targets muscular and neuronal nicotinic acetylcholine receptor subtypes with high affinity[END_REF].

Gymnodimine-A was shown both to have no agonist action on such receptors, but to block reversibly the inward nicotinic currents elicited by the iontophoretic ACh pulses. Furthermore, the inhibition of ACh-activated currents by gymnodimine-A was clearly independent of the holding membrane potential, in other terms had no voltage-dependence. Interestingly, the neuromuscular block produced by gymnodimine-A could be reversed by continuous washout of the phycotoxin from the medium within 30 min. Also, the block of K + channels in motor nerve terminals by 3,4diaminopyridine (100 µM), which increases evoked quantal ACh release by nerve impulses (Van der Kloot and [START_REF] Van Der Kloot | Quantal acetylcholine release at the vertebrate neuromuscular junction[END_REF], was able to antagonize the action of the phycotoxin. In addition, gymnodimine-A, in a reversible manner, blocked ACh-evoked currents in Xenopus oocytes expressing heterologous human 7 nAChRs, and this block was independent of the holding membrane potential [START_REF] Kharrat | The marine phycotoxin gymnodimine targets muscular and neuronal nicotinic acetylcholine receptor subtypes with high affinity[END_REF]. The X-ray crystal structure of gymnodimine-A in complex with Aplysia-AChBP has been determined [START_REF] Bourne | Structural determinants in phycotoxins and AChBP conferring high affinity binding and nicotinic AChR antagonism[END_REF]. The structures obtained indicate that gymnodimine-A was bound at the subunit interface, between loops C and F from opposing faces, in the same region where ACh binds. In addition, the structures highlighted the characteristic hydrogen bond between the iminium nitrogen in gymnodimine-A and the carbonyl oxygen of Trp 147 from loop C, and also the anchoring interactions of the tetrahydrofuran component in gymnodimine-A with the apical and membrane extremities of the binding site, along the subunit interface.

PINNATOXINS, PTERIATOXINS AND PORTIMINE -Chemical characteristics of pinnatoxins and pteriatoxins

To date, eight pinnatoxins (A-H) and three pteriatoxins (A-C) have been described. They all share the same common scaffold with a dimethyl substituted 7-membered cyclic imine as part of a spiroimine ring system, a 6,5,6-spiroketal ring system and a bridged ketal which is unique within the cyclic imine phycotoxins family (Fig. 6). The pinnatoxins and pteriatoxins are differentiated by the substituent at C33, which might be carboxyl (pinnatoxin-A), vinyl (pinnatoxins G and H), 5). Recently, a new pinnatoxin analog, pinnatoxin-H, was purified from a culture of the dinoflagellate Vulcanodinium rugosum from the South China Sea [START_REF] Selwood | Pinnatoxin H: a new pinnatoxin analogue from a South China Sea Vulcanodinium rugosum isolate[END_REF]. It is worth noting that the fatty acid esters of pinnatoxins A and G isolated from Canadian mussels were reported to likely originate from the shellfish than from the dinoflagellate (McCarron et al., 2012). Similarly, there is no conclusive evidence for the dinoflagellate source of pteriatoxins, isolated from the Japanese bivalve Pteria penguin [START_REF] Shumway | A review of the effects of algal blooms on shellfish and aquaculture[END_REF].

Table 5 near here -Toxicity of pinnatoxins and pteriatoxins

Acute toxicological studies with purified samples of pinnatoxin congeners, administered by i.p.

injection, revealed that pinnatoxin F was between about 2.7 and 5.1 times more potent than pinnatoxins E, G and H (Table 6), showing the following rank order of potency in the acute toxicity response: pinnatoxin F > pinnatoxin G > pinnatoxin E > pinnatoxin H (Munday et al., 2012b). A different rank order in the acute toxicity response (i.e. pinnatoxin F > pinnatoxin G > pinnatoxin H

> pinnatoxin E) was obtained for gavage administration. The oral toxicity of pinnatoxin-F was only between 5-and 6.5-fold higher than that of pinnatoxins G and H, but was about 93.3-fold higher than that of pinnatoxin E. As detailed previously for spirolide congeners and gymnodime-A, the oral toxicity of pinnatoxin-E was at least 49-fold lower than the i.p. toxicity, while comparatively smaller differences were observed for pinnatoxins F, G and H between the two routes of administration (Table 6). These findings are of great interest regarding the risk these toxins may have to human health. Only a few reports were concerned with the acute toxicity of pteriatoxins, but they revealed that these toxins produce animal lethality at doses between 0.008 and 0.1 mg/kg when administrated to mouse by i.p. injection, with toxic symptoms resembling those of pinnatoxins (see [START_REF] Takada | Structural determination of pteriatoxins A, B and C, extremely potent toxins from the bivalve Pteria penguin[END_REF]. The atypically stability of the aliphatic cyclic imine in pinnatoxins [START_REF] Jackson | Stability of cyclic imine toxins: interconversion of pinnatoxin amino ketone and pinnatoxin A in aqueous media[END_REF] is probably associated to their high oral toxicity, highest among the Cyclic Imine toxins.

-Molecular targets and mechanism of action of pinnatoxins and pteriatoxins

The use of successful strategies for the chemical synthesis of pinnatoxins A and G [START_REF] Beaumont | Marine Toxins with Spiroimine Rings: Total Synthesis of Pinnatoxin A[END_REF][START_REF] Stivala | Synthesis and biology of cyclic imine toxins, an emerging class of potent, globally distributed marine toxins[END_REF] allowed the production of substantial amounts of these toxins in pure form which permitted performing electrophysiological and functional studies, as well as competition binding studies together with computational molecular modeling, leading to the revision of the mode of action previously proposed for pinnatoxins. Thus, based on such analyses it has been unequivocally established that pinnatoxins A and G are potent antagonists of nAChRs [START_REF] Aráoz | Total synthesis of pinnatoxins A and G and revision of the mode of action of pinnatoxin A[END_REF]. The rank order of potency for pinnatoxin-A on nAChR subtypes determined by functional analyses was the following: human 7 > Torpedo 2 β1γδ > human 4β2. The reversibility of the pinnatoxin-A activity was faster for the  nAChR, while it was slowly reversible for the 7 nAChR subtype. Interestingly, pinnatoxin-A exhibited higher affinity than pinnatoxin-G for the human neuronal α7 nAChR subtype. In contrast, a synthetic aminoketone derivative of pinnatoxin-A with an open imine ring showed no activity on the various nAChR subtypes studied, a feature supporting the central role of the seven-membered cyclic imine ring for the potent antagonism of this family of toxins [START_REF] Aráoz | Total synthesis of pinnatoxins A and G and revision of the mode of action of pinnatoxin A[END_REF].

Also, pinnatoxins E, F and G were reported to produce concentration-dependent reductions in nerve-evoked twitch tension in isolated rat hemidiaphragm preparations, with a rank order of potency of F > G > E, while no effects were detected on directly muscle-evoked twitch responses.

Pinnatoxins F and G moreover reduced the amplitude of spontaneous miniature endplate potentials and evoked endplate potentials at skeletal neuromuscular junctions, without affecting miniature endplate potential frequency or the resting membrane potential of the muscle fibers. These results

show that pinnatoxins E, F and G are potent neuromuscular blockers and cause toxicity by acting as antagonists at muscle-type nicotinic acetylcholine receptors [START_REF] Hellyer | Neuromuscular blocking activity of pinnatoxins E, F and G[END_REF].

Further ligand-binding studies were performed to examine whether pinnatoxin-A had any action on other neurotransmitter receptors, including those coupled to G-proteins, and neurotransmitter transporters. However, no significant action, or very little action, was detected using a high concentration of pinnatoxin-A (10 µM) on the different receptors studied including: adrenergic receptors (Alpha1A, Alpha1B, Alpha1D, Alpha2A, Alpha2B, Alpha2C, Beta1, Beta2, Beta3) ;

PROROCENTROLIDES AND SPIRO-PROROCENTRIMINE -Chemical characteristics of prorocentrolide and spiro-prorocentrimine toxins

Prorocentrolides are the only members of the cyclic imine phycotoxins family that do not contain a spiroimine group. Their chemical structure includes a 26-membered carbo-macrocycle and a 28membered macrocyclic lactone arranged around a 3,4,4a,5,8,8a-hexahydroisoquinoline that incorporates the characteristic cyclic imine group. Six prorocentrolides have been reported to date (Fig. 7), which differ by the substituents at the positions 4 (hydrogen, hydroxy or sulfonate), 9

(methyl or exocyclic vinyl), 14 (hydroxy or acetate ester), 29 (methyl or hydrogen) and 30 (hydroxy or sulfonate). Prorocentrolide-B shows specific structural modifications at the positions 10 (exocyclic vinyl), 15 (hydrogen instead of hydroxy), 16 (hydroxy instead of hydrogen) and 29

(unsubstituted endocyclic E double bond).

A single spiro-prorocentrimine compound is currently known (see Fig. 1), whose chemical structure contains a 25-membered carbo-macrocycle and a 15-membered macrocyclic lactone arranged around a 6,6-spiroimine ring system. The overall structure and the stereochemistry of substituents have been unambiguously determined by X-ray crystallography [START_REF] Lu | Spiro-prorocentrimine, a novel macrocyclic lactone from a benthic Prorocentrum sp. of Taiwan[END_REF].

-Prorocentrolide and spiro-prorocentrimine producing organisms

To date, the production of prorocentrolide congeners and spiro-prorocentrimine is well-known to be restricted to primarily benthic or epiphytic Prorocentrum sp. which, unlike their pelagic counterparts, does not form dense blooms. Hence, prorocentrolide A was isolated from Prorocentrum lima in Japan and Taiwan, and the structurally related cyclic imine prorocentolide B from Prorocentrum maculosum in tropical waters. The Taiwanese Prorocentrum lima also produces two prorocentrolide analogs, 4-hydroxy and 14-O-acetyl-4-hydroxy prorocentrolides. Finally, spiroprorocentrimine was isolated along with prorocentrolide A from another Taiwanese, still unidentified, strain of Prorocentrum sp. (Table 7).

Table 7 near here -Toxicity of prorocentrolide and spiro-prorocentrimine toxins

Although only a few reports were concerned with the acute toxicity of prorocentrolide and spiroprorocentrimine toxins, they revealed that, when administrated to mouse by i.p. injection, prorocentrolide A had an animal lethality of 0.4 mg/kg mouse [START_REF] Torigoe | Prorocentrolide, a toxic nitrogenous macrocycle from a marine dinoflagellate, Prorocentrum lima[END_REF], while spiro- prorocentrimine exhibited much less toxicity (i.e. animal lethality occurred at 2.5 mg/kg mouse) than other known marine Cyclic Imine toxins [START_REF] Lu | Spiro-prorocentrimine, a novel macrocyclic lactone from a benthic Prorocentrum sp. of Taiwan[END_REF].

-Molecular targets and mechanism of action of prorocentrolide and spiro-prorocentrimine toxins

The molecular targets for these Cyclic Imine toxins have not yet been determined, because of the lack of purified or synthetic toxins.

CONCLUSION

In conclusion, this short review shows that the well-chemically characterized natural products derived from some toxic dinoflagellates species constitute a novel source of potent nAChR-ligands molecules. The distinguishing feature of these phycotoxins is the presence of a cyclic imine moiety in their structure, which constitutes a chemical signature associated with their capacity to interact with muscle-types and neuronal-nAChR subtypes. Thus, spirolides, gymnodimines and pinnatoxins, which are recognized "fast-acting toxins", are known to broadly target muscle-and neuronal-type nAChRs, generally with high affinity for the various receptor subtypes studied. The sites of interaction of these toxins on nAChRs include amino acid residues highly conserved in the sequences of muscle-and neuronal-type receptors. Another important aspect in the action of these phycotoxins is the lack of animal species-selectivity, since similar actions have been reported on frog, fish, chick, rodent, and human nAChRs. The toxicological profile reported for these phycotoxins, during rodent bioassays, is mostly due to their specific interaction with the nAChRs.

The competitive binding to nAChRs has been exploited to design and develop new tests to detect spirolides, gymnodimines and pinnatoxins in contaminated shellfish with better accuracy than the broad spectrum mouse bioassay. These tests are important in the food safety field, since shellfish constitutes a worldwide rich food resource that may be contaminated by toxins produced by harmful dinoflagellates. Additional appropriate toxicological studies must be developed to assess the risk these phycotoxins may have to human health, since at present there is no strong evidence linking the presence of spirolides, gymnodimines and pinnatoxins in shellfish tissues with human intoxications. Also, more data is needed on the risks of chronic exposure to these phycotoxins. Shellfish frequently contains varying amounts of these toxins, at low or high concentrations, which is a risk factor for human health that cannot be neglected. Furthermore, shellfish concentrate these toxins in their edible tissues, and can act as vector for transferring these phycotoxins through the marine food web, thus menacing wildlife.

Figure 1 near hereFigure 1 .

 1 Figure 1 near here

Figure 2

 2 Figure 2 near here

Figure 2 .

 2 Figure 2. Chemical structures of the known spirolide toxins.

Figure 3

 3 Figure 3 near here

Figure 3 .

 3 Figure 3. Three-dimensional structure of the muscle-type nAChR (Torpedo marmorata) obtained by cryoelectron microscopy (Protein Data Bank code 2BG9). The five subunits are colored in red, yellow, green, cyan and blue. The two antagonist sites where the spiroimine toxins can bind are shown as magenta-colored surfaces. Left: lateral view showing the extracellular (ECD), transmembrane (TMD) and intracellular (ICD) domains. Right: top view showing the organization of the five subunits arranged around the central pore.

Figure 4

 4 Figure 4 near here

Figure 4 .

 4 Figure 4. Experimental approaches used to study the action of spirolides and other spiroimine phycotoxins using electrophysiological techniques. (A) Mature female Xenopus laevis frog. (B) Oocytes at stages V to VI of development, surgically removed from an anesthetized mature female X. laevis under anesthesia. (C) Western blot of Torpedo marmorata purified membranes showing the g and d subunits of the nAChR and the rapsyn ( r ) protein. (D) Manually defolliculated oocytes in which one is microinjected with 50 nL of purified Torpedo membranes using a Nanoliter 2010 Micro 4 Controller injector mounted on a microscope. (E, F) Staining of nAChRs incorporated to the membrane of an oocyte using fluorescent α-Bungarotoxin, Alexa Fluor® 488 conjugate. (G) Inward nicotinic current triggered by an ACh pulse of 15 seconds duration, recorded in an oocyte micro-injected 4 days previously with purified Torpedo membranes. Perfusion of 13-desmethyl spirolide C (13-SPX-C) had no agonist action (blue tracing above recording), but complete blocked the ACh-evoked current

Figure 5 near hereFigure 5 .

 5 Figure 5 near here

Further insight into the

  interaction between gymnodimine-A and the muscle-type    nAChR was gained in competition binding studies using [ 125 I]-bungarotoxin as a tracer and HEK-293 cells expressing this nAChR. Gymnodimine-A totally displaced, in a concentration-dependent manner, [ 125 I]-bungarotoxin from HEK-293 cells expressing the muscle-type   nAChR or the chimeric neuronal 7-5-HT3 nAChR. Furthermore, gymnodimine-A inhibited the specific [ 3 H]epibatidine binding on HEK293 cells expressing either  or  neuronal nAChRs. These data are consistent with the fact that gymnodimine-A was highly toxic when applied directly into the mouse central nervous system by intracerebroventricular injection. The rank order of potency for gymnodimine-A on the various nAChRs studied was: 7-5HT3 > 1 2  >  > [START_REF] Kharrat | The marine phycotoxin gymnodimine targets muscular and neuronal nicotinic acetylcholine receptor subtypes with high affinity[END_REF]. In another study the rank order for binding affinity (Ki) was α7, α6β3β4α5 > rat α3β4 > human α3β4, α4β4 > rat α4β2, human α4β2[START_REF] Hauser | Comparison of acetylcholine receptor interactions of the marine toxins, 13-desmethylspirolide C and gymnodimine[END_REF]. Some 6,6-spiroimine analogs of gymnodimine-A have been chemically synthesized, and were shown to block ACh-evoked nicotinic currents in Xenopus oocytes having incorporated the Torpedo muscle-type (  ) nAChR in their membrane, or expressing the human nAChR[START_REF] Duroure | 6,6-Spiroimine analogs of (-)-gymnodimine A : synthesis and biological evaluation on nicotinic acetylcholine receptors[END_REF]. Although these spiroimine analogs are about 40-45 times less active than gymnodimine-A, the fact that they block both types of nAChRs indicate that the 6,6-spiroimine moiety is a key structural factor required for blocking nAChRs in the gymnodimine family.

  glycinyl (pinnatoxins B and C), butyrolactone (pinnatoxin-F) or hydroxy/keto-butyric acid (pinnatoxins D and E), as well as S-cysteinyl derivatives (pteriatoxins A-C). Other positions with substituent variability are 21, 22 and 28, for which we observe two substitution patterns: H, H and OH (pinnatoxins A, B, C, G and pteriatoxins A, B, C), and CH 3 , OH and H (pinnatoxins D, E, F, H), respectively.

Figure 6 near hereFigure 6 .

 6 Figure 6 near here

  muscarinic receptors (M1, M2, M3, M4 and M5); serotoninergic receptors (5-HT1A, 5-HT1B, 5-HT1D, 5-HT1E, 5-HT2A, 5-HT2B, 5-HT2C, 5-HT3, 5-HT5A, 5-HT6, 5-HT7); GABAA receptor, dopaminergic receptors (D1, D2, D3, D4 and D5), Histamine receptors (H1 and H2), Opioid (DOR), Opioid (KOR), µ-Opioid (MOR) and the dopamine transporter (DAT), norepinephrine transporter (NET) and serotonin transporter (SERT). Furthermore, no activity on Ca V 1.1 calcium channels (type-L) originally proposed to be the target of pinnatoxins, was observed[START_REF] Aráoz | Total synthesis of pinnatoxins A and G and revision of the mode of action of pinnatoxin A[END_REF]. These experimental results reveal an extraordinary selectivity of pinnatoxin-A for the nAChRs.The evidence that spirolides, gymnodimines and pinnatoxins target nAChRs prompted the development of target-directed functional method, based on the mechanism of action of competitive antagonists on nAChRs. Thus, several non-radioactive ligand-binding methods have been developed(Vilariño et al., 2009; Fonfría et al., 2010; Otero et al., 2011; Aráoz et al., 2012; Rodriguez et al., 2013) to replace the traditional mouse bioassay which has numerous drawbacks related to sensitivity and specificity, apart from the ethical concern of using large number of animals, as recently discussed (Daneshian et al., 2013).-PortimineA new Cyclic Imine toxin, named portimine, has been recently discovered from the pinnatoxin producing dinoflagellate Vulcanodinium rugosus from Northland (New Zealand)[START_REF] Selwood | Portimine: A bioactive metabolite from the benthic dinoflagellate Vulcanodinium rugosum[END_REF]. The structure of portimine, including the relative configurations, was elucidated by spectroscopic analyses. The cyclic imine moiety consists of an unprecedented five-membered ring with a spiro-link to a cyclohexene ring (see Fig.1). Portimine contains only 14 carbons in its macrocycle, making it the smallest Cyclic Imine toxin isolate to date. Such structural factors have a large impact on the toxin toxicity, since it has a much lower toxicity in rodent bioassays after i.p. administration (LD 50 = 1.57 mg/kg mouse) than any other Cyclic Imine toxin. In contrast, portimine exhibits high toxicity in vitro to mammalian cells, with an LC 50 of 2.7 nM to mouse leukemia P388 cells, and activation of caspases indicates that it has apoptotic activity.

Figure 7 near hereFigure 7 .

 7 Figure 7 near here

Table 1 .

 1 Dinoflagellate species known to produce spirolides, their location and produced compounds.

		Canada	Spirolides A, B, C, C2, C3, D, D2, D3, H and I 13-desmethyl Spirolides C and D
		Norway	20-methyl Spirolide G
			Spirolides D and G
		Denmark	13-desmethyl Spirolides C and D
			13,19-didesmethyl Spirolide C
			Spirolide D
	Alexandrium ostenfeldii	Italy	13-desmethyl Spirolides C and D 13,19-didesmethyl Spirolide C 27-hydroxy-13,19-didesmethyl Spirolide C
			27-oxo-13,19-didesmethyl Spirolide C
			27-hydroxy-13-desmethyl Spirolide C
		Scotland	Spirolide C
			Spirolides C and D
		Ireland	13-desmethyl Spirolide C
			20-methyl Spirolide G
		Mediterranean Sea	Spirolides B, C and D 13-desmethyl Spirolides C and D
	Alexandrium peruvianum	Ireland	13-desmethyl Spirolides C and D
		Unired States	Spirolide D 13-desmethyl Spirolides C and D

Table 2 .

 2 The 50% lethal doses (LD 50 ) of spirolide congeners according to their administration route.

	Spirolide	Administration mode	LD 50 (mg/kg mouse)	Ratio (compared to i.p.)
		Intraperitoneal injection	0.037	1
	Spirolide A	Gavage	0.24-0.55	6.5-14.9
		Feeding	1.2-1.3	32.4-35.1
	Spirolide B	Intraperitoneal injection Gavage	0.099 0.440	1 4.4
		Intraperitoneal injection	0.008	1
	Spirolide C	Gavage	0.053-0.18	6.6-22.5
		Feeding	0.5-0.78	62.5-97.5
		Intraperitoneal injection	0.007-0.028	1
	13-desmethyl Spirolide C	Gavage	0.13-0.16	4.6-22.9
		Feeding	0.5-1	1 7 . 9 -142.9
	13,19-didesmethyl Spirolide C	Intraperitoneal injection	0.032	1
	27-hydroxy-13-desmethyl Spirolide C	Intraperitoneal injection	>0.027	1
	27-oxo-13,19-didesmethyl Spirolide C	Intraperitoneal injection	>0.035	1
	Spirolide E	Intraperitoneal injection	>1.000	1
	Spirolide F	Intraperitoneal injection	>1.000	1
		Intraperitoneal injection	0.008->0.063	1
	20-methyl Spirolide G	Gavage	0.088-0.16	<1.4
		Feeding	0.5-0.63	<7.9

Table 3 .

 3 Dinoflagellate species known to produce gymnodimine congeners.

	Karenia selliformis	New Zealand Australia	Gymnodimines A, B and C Gymnodimine A
	Alexandrium peruvianum Unired States	12-methyl Gymnodimine

Administration mode LD 50 (mg/kg mouse) Ratio (compared to i.p.)

	Gymnodimine A	Intraperitoneal injection	0.08-0.096	1
		Intracerebroventricular injection	0.003	0.03-0.038
		Gavage	0.755	7.9-9.4
		Feeding	4.057->7.5	42.3->78.1
	Gymnodimine B	Intraperitoneal injection	0.800	1

Table 4 .

 4 The 50% lethal doses (LD 50 ) of gymnodimine congeners according to their administration route.

Table 6 near here

 6

	Vulcanodinium rugosum	Australia	Pinnatoxins A, E, F and G
		New Zealand	Pinnatoxins E and F
		Japan	Pinnatoxin G
		France	Pinnatoxin G
		China	Pinnatoxin H

Table 5 .

 5 Dinoflagellate species known to produce pinnatoxin congeners.

	Pinnatoxin	Administration mode	LD 50 (mg/kg mouse)	Ratio (compared to i.p.)
		Intraperitoneal injection	0.045-0.057	1
		Gavage	2.800	49.1-62.2
		Intraperitoneal injection	0.013-0.016	1
		Gavage	0.025-0.03	1.6-2.3
		Feeding	0.05-0.077	3.1-5.9
		Intraperitoneal injection	0.043-0.05	1
		Gavage	0.150	3-3.5
		Feeding	0.400	8-9.3
		Intraperitoneal injection	0.067	1
		Gavage	0.163	2.4

Table 6 .

 6 The 50% lethal doses (LD 50 ) of pinnatoxin congeners according to their administration route.

	Pinnatoxin E
	Pinnatoxin F
	Pinnatoxin G
	Pinnatoxin H

Table 7 .

 7 Dinoflagellate species known to produce prorocentrolide congeners and spiroprorocentrimine.

		Japan	Prorocentrolide A
	Prorocentrum lima	Taiwan	Prorocentrolide A 4-hydroxy Prorocentrolide
			14-O -acetyl-4-hydroxy Prorocentrolide
	Prorocentrum maculosum Tropical waters	Prorocentrolide B
	Prorocentrum sp.	Taiwan	Prorocentrolide A Spiro-prorocentrimine

Acknowledgements

Work performed in the laboratory of authors was funded by the "Agence Nationale de la Recherche" (France, grant AQUANEUROTOX ANR-12-ASTR-0037-1), and by the National Institutes of Health (USA, grant NIGMS R01 GM077379 to A.Z).


Léonard Hussenot

Matthieu Geist

Olivier Pietquin

CopyCAT: Taking Control of Neural Policies with Constant Attacks

We propose a new perspective on adversarial attacks against deep reinforcement learning agents. Our main contribution is CopyCAT, a targeted attack able to consistently lure an agent into following an outsider's policy. It is pre-computed, therefore fast inferred, and could thus be usable in a real-time scenario. We show its effectiveness on Atari 2600 games in the novel read-only setting. In this setting, the adversary cannot directly modify the agent's state -its representation of the environment-but can only attack the agent's observation -its perception of the environment. Directly modifying the agent's state would require a write-access to the agent's inner workings and we argue that this assumption is too strong in realistic settings.

INTRODUCTION

We are interested in the problem of attacking sequential control systems that use deep neural policies. In the context of supervised learning, previous work developed methods to attack neural classifiers by crafting so-called "adversarial examples". These are malicious inputs particularly successful at fooling deep networks with high-dimensional input-data like images. Within the framework of sequential-decision-making, previous works used these adversarial examples only to break neural policies. Yet the attacks they build are rarely applicable in a real-time setting as they require to craft a new adversarial input at each time step. Besides, these methods use the strong assumption of having a write-access to what we call the agent's state, the actual input of the neural policy built by the algorithm from the observations. When taking this assumption, the adversary -the algorithm attacking the agent-is not placed at the interface between the agent and the environment where the system is the most vulnerable. We wish to design an attack with a more general purpose than just shattering a neural policy as well as working in a more realistic setting.

Our main contribution is CopyCAT, an algorithm for taking full-control of neural policies. It produces a simple attack that is: (1) targeted towards a policy, i.e., it aims at matching a neural policy's behavior with the one of an arbitrary policy; [START_REF] Tom B Brown | Adversarial patch[END_REF] only altering observation of the environment rather than complete agent's state;

(3) composed of a finite set of pre-computed state-independent masks. This way it requires no additional time at inference hence it could be usable in a real-time setting.

We introduce CopyCAT in the white-box scenario (Sec. 3), with read-only access to the weights and the architecture of the neural policy. This is a realistic setting as prior work showed that after training substitute models, one could transfer an attack computed on these to the inaccessible attacked model [START_REF] Papernot | Transferability in machine learning: from phenomena to black-box attacks using adversarial samples[END_REF]. The context is the following: [START_REF] Marc G Bellemare | The arcade learning environment: An evaluation platform for general agents[END_REF] We are given any agent using a neural-network for decision-making (e.g., the Q-network for value-based agents, the policy network for actor-critic or imitation learning methods) and a target policy we want the agent to follow. [START_REF] Tom B Brown | Adversarial patch[END_REF] The only thing one can alter is the observation the agent receives from the environment and not the full input of the neural controller (the state). In other words, we are granted a read-only access to the agent's inner workings. In the case of Atari 2600 games, the agents builds its state by stacking the last four observations. Attacking the agent's state means writing in the agent's memory of the last observations. (3) The computed attack should be inferred fast enough to be used in real-time.

We stress the fact that targeting a policy is a more general scheme than untargeted attacks where the goal is to stop the agent from taking its preferred action (hoping for it to take the worst). It is also more general than the targeted scheme of previous works where one wants the agent to take its least preferred action or to reach a specific state. In our setting, one can either hard-code or train a target policy. This policy could be minimizing the agent's true reward but also maximizing the reward for another task. For instance, this could mean taking full control of an autonomous vehicle, possibly bringing it to any place of your choice.

We exemplify this approach on the classical benchmark of Atari 2600 games (Sec.5). We show that taking control of a trained deep RL agent so that its behavior matches a desired policy can be done with this very simple attack. We believe such an attack reveals the vulnerability of autonomous agents. As one could lure them into following catastrophic behaviors, autonomous cars, robots or any agent with high dimensional inputs are exposed to such manipulation. This suggests that it would be worth studying new defense mechanisms that could be specific to RL agents, but this is out of the scope of this paper.

Section 6 presents experiments studying various aspects of the proposed method, as the use of CopyCAT in the black-box setting or in environments with a more complex perception. These results assess the possibility of using the proposed method in various settings even though some of these experiments are still preliminary.

BACKGROUND

In Reinforcement Learning (RL), an agent interacts sequentially with a dynamic environment so as to learn an optimal control. To do so, the problem is modeled as a Markov Decision Process. It is a tuple {S, A, P, r , γ } with S the state space, A the action space we consider as finite in the present work, P the transition kernel defining the dynamics of the environment, r a bounded reward function and γ ∈ (0, 1) a discount factor. The policy π maps states to distributions over actions: π (•|s). The (random) discounted return is defined as G = t ≥0 γ t r t . The policy π is trained to maximize the agent expected discounted return. The function

V π (s) = E π [G |s 0 = s]
denotes the value function of policy π (where E π [•] denotes the expectation over all possible trajectories generated by policy π ). We also call µ 0 the initial state distribution and ρ(π ) = E s∼µ 0 [V π (s)] the expected cumulative reward starting from µ 0 . Value-based algorithms [START_REF] Hessel | Rainbow: Combining improvements in deep reinforcement learning[END_REF][START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF] use the value function, or more frequently the action-value function Q π (s, a) = E π [G |s 0 = s, a 0 = a], to compute π . To handle large state spaces, deep RL uses deep neural networks for function approximation. For instance, value-based deep RL parametrizes the action-value function Q ω with a neural network of parameters ω and deep actor-critics [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF] directly parametrize their policy π θ with a neural network of parameters θ . In both cases, the taken action is inferred by a forward-pass in a neural network.

Adversarial examples were introduced in the context of supervised classification. Given a classifier C, an input x, a bound ϵ on a norm ∥.∥, an adversarial example is an input [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] is a widespread method for generating adversarial examples for the L ∞ norm. From a linear approximation of C, it computes the attack η as:

x ′ = x + η such that C(x) C(x ′ ) while ∥x -x ′ ∥ ≤ ϵ. Fast Gradient Sign Method (FGSM)
η = ϵ • sign(∇ x l(θ, x, y)), (1)
with l(θ, x, y) the loss of the classifier and y the true label.

As an adversary, one wishes to maximize the loss l(θ, x + η, y) w.r.t. η. Presented this way, it is an untargeted attack. It pushes C towards misclassifying x ′ in any other label than y. It can easily be turned into a targeted attack by, instead of l(θ, x + η, y), optimizing for -l(θ, x + η, y target ) with y target the label the adversary wants C to predict for x ′ . This attack, optimized for the L ∞ norm can also be turned into an L 2 attack by taking:

η = ϵ • ∇ x l(θ, x, y) ∥∇ x l(θ, x, y)∥ 2 . ( 2
)
As shown by Eqs. ( 1) and ( 2), these attacks are computed with one single step of gradient, hence the term "fast". These two attacks can be turned into -more efficient, yet slower-iterative methods [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF][START_REF] Dong | Boosting adversarial attacks with momentum[END_REF] by taking several successive steps of gradients. These methods will be referred to as iterative-FGSM.

When using deep networks to compute its policy, an RL agent can be fooled the same way as a supervised classifier. As a policy can be seen as a mapping S → A, untargeted FGSM (1) can be applied to a deep RL agent to stop it from taking its preferred action: a * = arg max a ∈A π (a|s). Similarly targeted FGSM can be used to lure the agent into taking a specific action. Yet, this would mean having to compute a new attack at each time step, which is generally not feasible in a real-time setting. Moreover, with this formulation, it needs to directly modify the agent's state, the input of the neural policy, which is a strong assumption.

THE COPYCAT ATTACK

In this work, we propose CopyCAT. It is an attack whose goal is to lure an agent into having a given behavior, the latter being specified by another policy. CopyCAT's goal is not only to lure the agent into taking specific actions but to fully control its behavior. Formally, CopyCAT is composed of a set of additive masks ∆ = {δ a } a ∈A that can be used to drive a policy π to follow any policy π target . Each additive mask δ a is pre-computed to lure π into taking a specific action a when added to the current observation regardless of the content of the observation. It is, in this sense, a universal attack. CopyCAT is an attack on raw observations and, as ∆ is pre-computed, it can be used online in a real-time setting with no additional computation.

Notations. We denote π the attacked policy and π target the target policy. At time step t, the policy π outputs an action a t taking the state s t as input. The agent state is internally computed from the past observations and we denote f the observations-to-state function:

s t = f (o t , o 1:t -1 ) with o 1:t -1 = (o 1 , o 2 ...o t -1 ).
Data Collection. In order to be pre-computed, CopyCAT needs to gather data from the agent. By watching the agent interacting with the environment, CopyCAT gathers a dataset D of K episodes made of observations:

D = (o k t ) k ∈(1:K ) t ∈(1,T k ) .
We recall that the objective in this setting is for CopyCAT to work with a read-only access to the inner workings of the agent. We thus stress that D is made of observations rather than states. If CopyCAT is successful, π is going to behave as π target and thus may experience observations out of the distribution represented in D. Yet, as will be shown, CopyCAT transfers to unseen observations. We hypothesize that, as we build a universal attack, the learned attack is able to move the whole support of observations in a region of R N where π chooses a precise action.

Training. A natural strategy for building an adversarial example targeted towards label ŷ is the following. Given a classifier P(y|x) parametrized with a neural network and an input example x, one computes the adversarial example x = x + δ by maximizing log P( ŷ| x) subject to the constraint: ∥δ ∥ ∞ = ∥xx ∥ ∞ ≤ ϵ. The adversary then performs either one step of gradient (FGSM) or uses an iterative method [START_REF] Kurakin | Adversarial examples in the physical world[END_REF] to solve the optimization problem.

Instead, CopyCAT is built for its masks to be working whatever the observation it is applied to. For each a ∈ A we build δ a , the additional masks luring π into taking action a, by maximizing over δ a :

E o k t ∈D log π (a| f (o k t +δ a , o k 1:t -1 ))+α ∥δ a ∥ 2 s.t. ∥δ a ∥ ∞ < ϵ . (3)
We restrict the method to the case where f , the function building the agent's inner state from the observations, is differentiable. The Eq. 3 can be optimized by alternating between gradient steps with adaptive learning rate [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] on mini-batches and projection steps onto the L ∞ -ball of radius ϵ. Unlike FGSM, CopyCAT is a full optimization method. It does not take only one single step of gradient.

CopyCAT has two main parameters: ϵ ∈ R + , a hard constraint on the L ∞ norm of the attack and α ∈ R + , a regularization parameter on the L 2 norm of the attack.

Inference. Once ∆ is computed, the attack can be used on π to make it follow any policy π target . At each time step t and given past observations, π target infers an action a target t . The corresponding mask δ a target t ∈ ∆ is applied to the last observation o t before being passed to the agent. No optimization is made at inference and CopyCAT can thus be used in a setting where the adversary is not let unlimited time to compute an attack between two time-steps.

RELATED WORK

Vulnerabilities of neural classifiers were highlighted by Szegedy et al. [START_REF] Szegedy | Intriguing properties of neural networks[END_REF] and several methods were developed to create the socalled adversarial examples, maliciously crafted inputs fooling deep networks. In sequential-decision-making, previous works use them to attack deep reinforcement learning agents. However these attacks are not always realistic. The method from Huang et al. [START_REF] Huang | Adversarial attacks on neural network policies[END_REF] uses fast-gradient-sign method [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF] for the sole purpose of destroying the agent's performance. What's more, it has to craft a new attack at each time step. This implies back-propagating through the agent's network, which is not feasible in real-time. Moreover, it modifies directly the state of the agent by writing in its memory, which is a strong assumption to take on what component of the agent can be altered. The approach of Lin et al. [START_REF] Lin | Tactics of adversarial attack on deep reinforcement learning agents[END_REF] allows the number of attacked states to be divided by four, yet it uses the heavy optimization scheme from Carlini and Wagner [START_REF] Carlini | Towards evaluating the robustness of neural networks[END_REF] to craft their adversarial examples. This is, in general, not doable in a real-time setting. They also take the same strong assumption of having a "read & write-access" to the agent's inner workings. To the best of our knowledge, they are the first to introduce a targeted attack. However, the setting is restricted to targeting one "dangerous state". Pattanaik et al. [START_REF] Pattanaik | Robust deep reinforcement learning with adversarial attacks[END_REF] proposes a method to lure the agent into taking its least preferred action in order to reduce its performance but still uses computationally heavy iterative methods at each time step. Pinto et al. [START_REF] Pinto | Robust Adversarial Reinforcement Learning[END_REF] proposed an adversarial method for robust training of agents but only considered attacks on the dynamic of the environment, not on the visual perception of the agent. Zhang et al. [START_REF] Zhang | Learning to design games: Strategic environments in reinforcement learning[END_REF] and Ruderman et al. [START_REF] Ruderman | Uncovering Surprising Behaviors in Reinforcement Learning via Worst-case Analysis[END_REF] developed adversarial environment generation to study agent's generalization and worst-case scenarios. Those are different from this present work where we enlighten how an adversary might take control of a neural policy.

ATARI EXPERIMENTS

We wish to build an attack targeted towards the policy π target . At a time step t, the attack is said to be successful if π under attack indeed chooses the targeted action selected by π target . When π is not attacked, the attack success rate corresponds to the agreement rate between π and π target , measuring how often the policies agree along an unattacked trajectory of π .

Note that we only deal with trained policies and no learning of neural policies is involved. In other words, π and π target are trained and frozen policies.

What we really want to test is the ability of CopyCAT to lure π into having a specific behavior. For this reason, measuring the attack success rate is not enough. Having a high success rate does not necessarily mean the macroscopic behavior of the attacked agent matches the desired one as will be shown further in this section.

Cumulative reward as a proxy for behavior. We design the following setup. The agent has a policy π trained with DQN [START_REF] Mnih | Human-level control through deep reinforcement learning[END_REF]. The policy π target is trained with Rainbow [START_REF] Hessel | Rainbow: Combining improvements in deep reinforcement learning[END_REF]. We select Atari games [START_REF] Marc G Bellemare | The arcade learning environment: An evaluation platform for general agents[END_REF] with a clear difference in terms of performance between the two algorithms (where Rainbow obtains higher average cumulative reward than DQN). This way, in addition to measuring the attack success rate, we can compare the cumulative reward obtained by π under attack ρ(π ) to ρ(π target ) as a proxy of how well π 's behavior is matching the behavior induced by π target . In this setup, if the attacked policy indeed gets cumulative rewards as high as the ones obtained by π target , it will mean that we did not simply turned some actions into other actions we targeted, but that the whole behavior induced by π under attack matches the one induced by π target . This idea that, in reinforcement learning, cumulative reward is the right way to monitor an agent's behavior has been used and developed by the inverse reinforcement learning literature. Ng et al. [START_REF] Andrew | Algorithms for inverse reinforcement learning[END_REF] argued that the value of a policy, i.e. its cumulative reward, is the most compact, robust and transferable description of its induced behavior. We argue that measuring cumulative reward is thus a reasonable proxy for monitoring the behavior of π under attack. At this point, we would like to carefully defuse a possible misunderstanding. Our goal is not to show that DQN's performance can be improved by being attacked. We simply want to show that its behavior can be fully manipulated by an opponent and we use the obtained cumulative reward as a proxy for the behavior under attack.

Baseline. We set the objective of building a real-time targeted attack. We thus need to compare our algorithm to baselines applicable to this scenario. The fastest state-of-the-art targeted method can be seen as a variation of Huang et al. [START_REF] Huang | Adversarial attacks on neural network policies[END_REF]. It applies targeted FGSM at each time step t to compute a new attack. It first infers the action a target and then back-propagates through the attacked network to compute their attack. CopyCAT only infers a target and then applies the corresponding pre-computed mask. Both methods can thus be considered usable in real-time yet CopyCAT is still faster at inference. We set the objective of attacking only observations rather than complete states so we do not need a write-access to the agent's inner workings. DQN stacks four consecutive frames to build its inner state. We thus compare CopyCAT to a version of the method from Huang et al. [START_REF] Huang | Adversarial attacks on neural network policies[END_REF] where the gradient inducing the FGSM attack is only computed w.r.t the last observation, so it produces an attack comparable to CopyCAT, i.e., on a single observation. The gradient from Eq. 1: ∇ s t l(θ, s t , a target ) becomes ∇ o t l(θ, f (o t , o 1:t -1 ), a target ). To keep the comparison fair, a target is always computed with the exact same policy π target as in CopyCAT. The policy π target : S → A is fixed.

FGSM-L ∞ has the same parameter ϵ as CopyCAT, bounding the L ∞ norm of the attack. CopyCAT has an additional regularization parameter α allowing the attack to have, for a same ϵ, a lower energy and thus be less detectable. We will compare CopyCAT to the attack from Huang et al. [START_REF] Huang | Adversarial attacks on neural network policies[END_REF] showing how behaviors of π under attacks match π target when these attacks are of equal energy.

Full optimization-based attacks would not be inferred fast enough to be used in a sequential decision making problem at each time step.

Experimental setup. We always turn the sticky actions on, which make the problem stochastic [START_REF] Marlos | Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents[END_REF]. An attacked observation is always clipped to the valid image range, 0 to 255. For Atari games, DQN uses as its inner state a stack of four observations:

f (o i , o 1:i-1 ) = [o i , ..., o i-3 ].
For learning the masks of ∆, we gather trajectories generated by π in order to fill D with 10k observations. We use a batch size of 8 and the Adam optimizer [START_REF] Diederik | Adam: A method for stochastic optimization[END_REF] with a learning rate of 0.05. Each point of each plot is the average result over 5 policy π seeds and 80 runs for each seed. Only one seed is used for π target to keep comparison in terms of cumulative reward fair.

Attack energy. As

CopyCAT has an extra parameter α, we test its influence on the L 2 norm of the produced attack. For a given ϵ, FGSM-L ∞ computes an attack η of maximal energy. As given by Eq. 1, its L 2 norm is ∥η∥ 2 = √ N ϵ 2 with N the input dimension. For a given ϵ, CopyCAT produces |A| masks. We show in Fig. 1 the largest L 2 norm of the |A| masks for a varying α (plain curves) and compare it to the norm of the FGSM-L ∞ attack (dashed lines). We want to stress that the attacks are agnostic to the training algorithm so the results are easily transferred to other agents using neural policies trained with another algorithm. As can be seen on Fig. 1, for a given ϵ and for the range of tested α, the attack produced by CopyCAT has lower energy than FGSM-L ∞ . This is especially significant for higher values of ϵ, e.g higher than 0.05.

Influence of parameters over the resulting behavior. We wish to show how the agent behaves under attack. As explained before, this analysis is twofold. First, we study results in terms of attack success rate -rate of action chosen by π matching a target when shown attacked observations-as done in supervised learning. Second, we study the behavior matching through the cumulative rewards under attack ρ(π ).

What we wish to verify in the following experiment is Copy-CAT's ability to lure an agent into following a specific behavior. If the attack success rate is high (close to 1), we know that, on a supervised-learning perspective, our attack is successful: it lures the agent into taking specific actions. If, in addition, the average cumulative reward obtained by the agent under attack reaches ρ(π target ) it means that the attack is really successful in terms of behavior. We recall that we attack a policy with a target policy reaching higher average cumulative reward.

We show on Fig. 2, 3 and 4 (three different games) the attack success rate (left) and the cumulative reward (right) for CopyCAT (plain curves) for different values of the parameters α and ϵ, as well as for unattacked π (green dashed line) and π target (black dashed lines). We observe a gap between having a high success rate and forcing the behavior of π to match the one of π target . There seems to exist a threshold corresponding to the minimal success rate required for the behaviors to match. For example, as seen on the left, CopyCAT with ϵ = 5 and α < 10 -5 (green curve) is enough to get a 85% success rate on the attack. However, as seen on the right, it is not enough to get the behavior of π under attack to match the one of the target policy as the reward obtained under attack never reaches ρ(π target ).

Overall, we observe on Fig. 2-right, Fig. 3-right and Fig. 4-right that with ϵ high enough ϵ ≥ 0.04 and α < 10 -6 , CopyCAT is able to consistently lure the agent into following the behaviour induced by π target .

Comparison to Huang et al. [START_REF] Huang | Adversarial attacks on neural network policies[END_REF]. We compare CopyCAT to the targeted version of FGSM on a setup where the gradient is computed only on the last observation. As in the last paragraph, we study both the attack success rate and the average cumulative reward under attack. We ask the question: is CopyCAT able to lure the agent into following the targeted behavior? Is it better at this task than FGSM in the real-time and read-only setting? We show on Fig. 5, 6 and 7 (three different games) the success rate of CopyCAT and FGSM (y-axis, left) and the average cumulative reward under attack (y-axis, right). These values are plotted (i) against the L 2 norm of the attack for FGSM and (ii) against the largest L 2 norm of the masks: max a ∈A ∥δ a ∥ 2 for CopyCAT. We only plot the standard deviation on the attack success rate because it corresponds to the intrinsic noise of CopyCAT. We do not plot it for cumulative reward for the reason that one seed of π target has a great variance (with the sticky actions) and matching π target , even perfectly, implies matching the variance of its cumulative rewards. The same phenomenon can be observed on Fig. 2 and3: CopyCAT is not itself unstable (left figures, when α decreases or ϵ increases, the rate of successful attacks consistently increases). Yet the cumulative reward is noisier, as the behavior of π is now matching with a highvariance policy. As observed on Fig. 5-right, Fig. 6-left and Fig. 7left, FGSM is able to turn a potentially significant part of the taken actions into the targeted actions (maximal success rate around 75% on Space Invaders). However, it is never able to make π 's behavior match with π target 's behavior as seen on Fig. 5-right, Fig. 6-right and Fig. 7-right. The average cumulative reward obtained by π under FGSM attack never reaches the one of π target . On the contrary, CopyCAT is able to successfully lure π into following the desired macroscopic behavior. First, it turns more than 99% of the taken actions into the targeted actions. Second, it makes ρ(π ) under attack reach ρ(π target ). Moreover, it does so using only a finite set of masks while the baselines compute a new attack at each time step.

An example of CopyCAT is shown on Fig. 8. The patch δ a aiming at action a: "no-op" (i.e. do nothing) is applied to an agent playing Space Invaders. The patch itself can be seen on the right (gray represents a zero pixel, black negative and white positive). The

ADDITIONAL EXPERIMENTS

In this section, we provide additional experiments to study further various aspects of the proposed approach. We build experiments to assess the possibility that CopyCAT would be successfully applicable in harder contexts, with very dissimilar policies π and π target , in the much more complex black-box setting or in environments with more realistic images than Atari.

Towards black-box targeted attacks

Papernot et al. [START_REF] Papernot | Transferability in machine learning: from phenomena to black-box attacks using adversarial samples[END_REF] observed the transferability of adversarial examples between different models. Thanks to this transferability, one is able to attack a model without having access to its weights. By learning attacks on proxy models, one can build black-box adversarial examples. However Kos and Song [START_REF] Kos | Delving into adversarial attacks on deep policies[END_REF] enlightened the difficulty for the state-of-the-art methods to build targeted adversarial examples in this black-box setting. Starting from the intuition that universal attacks may transfer better between models, we enhanced CopyCAT for it to work in the black-box setting.

We consider a setting where the adversary (1) is given a set of proxy models {π 1 , ..., π n } trained with the same algorithm as π , (2) can also query the attacked model π , but (3) has no access to its weights. In the black-box setting, CopyCAT is divided into two steps: first training multiple additional masks, and second selecting the highest performing ones.

Training. The ensemble-based method from Kos and Song [START_REF] Kos | Delving into adversarial attacks on deep policies[END_REF] computes its additional mask by attacking the classifier given by the mean predictions of the proxy models. We instead consider that our attack should be efficient against any convex combination of the proxy models' predictions. For each action a ∈ A, we compute the mask δ a by maximizing over 100 epochs on the dataset D:

E o k t ∈D (α 1 ...α p )∼∆ log 1≤p ≤n α p π p (a| f (o k t + δ a , o k 1:t -1 )) + α ∥δ a ∥ 2 s.t. ∥δ a ∥ ∞ < ϵ . (4)
with ∆ the uniform distribution over the n-simplex. By sampling uniformely over the simplex, we hope to build an attack fooling any classifier with predictions in the convex hull of the proxy models' predictions.

For each action, 100 masks are computed this way. These masks are just computed with different random seeds.

Selection. We then compute a competition accuracy for each of these random seeds. This accuracy is computed by querying π on states built as follows. We take four consecutive observations in D, apply 3 masks randomly selected among the previously computed masks on the first 3 observations; the mask δ a that is actually being tested is applied on the last observation. The attack is considered successful if π outputs the action a corresponding to δ a . For each action, the mask with the highest competition accuracy among the 100 computed masks is selected.

Inference. The selected masks are then used online as in the white-box setting.

Results. We provide preliminary results for the considered blackbox setting. Four proxy models of DQN are used to attack π . Again, it is attacked to make it follow the policy π target given by Rainbow. The results can be found in Fig. 9. Each dot is an attack tested over 80 new episodes. Y-axis is the mean success rate (middle) or the cumulative reward (right). X-axis is the maximal norm of the attack. The figure on the left gives the value of α (on the y-axis) corresponding to each color.

We can observe on Fig. 9 that the proposed black-box attack is effective, even if less efficient than its white-box counterpart. The proposed black-box CopyCAT could certainly be improved, and we let this for future work.

Attacking untrained DQN

In Sec. 5, the attacked agent was a trained DQN agent, while the target policy was a trained Rainbow agent. If these agents have clearly different behaviors, one could argue that they were initially trained to solve the same task (getting the highest score as possible), Rainbow achieving better results. To further assess CopyCAT's ability to lure a policy into following another policy, we therefore attack an untrained DQN, with random weights, to follow the policy π target still obtained from a trained Rainbow agent. These two policies are dissimilar as one is random while the other is trained to maximized the game's score. We see on Fig. 10, 11, 12 that in this case, FGSM is able to lure π into following π target at least as well as CopyCAT. This shows that it is easier to fool an untrained network than a trained one. As expected, trained networks are more robust to adversarial examples.

CopyCAT is also able to lure the agent into following π target .

More realistic images

Reinforcement learning led to great improvements for games [START_REF] Silver | Mastering chess and shogi by self-play with a general reinforcement learning algorithm[END_REF] or robots manipulation [START_REF] Levine | End-to-end training of deep visuomotor policies[END_REF] but is not able yet to tackle realistic-image environments. While this paper focuses on weaknesses of reinforcement learning agents, the relevance of the proposed method would be diminished if one could not compute universal adversarial examples on realistic datasets. We thus present this proof-of-concept, showing the existence of universal adversarial examples on ImageNet [START_REF] Deng | ImageNet: A Large-Scale Hierarchical Image Database[END_REF] dataset. Note that Brown et al. [START_REF] Tom B Brown | Adversarial patch[END_REF] already showed the existence of universal attacks but considered a patch completely covering a part of the image rather than an additional mask on the image.

We computed a universal attack on VGG16 [START_REF] Simonyan | Very deep convolutional networks for large-scale image recognition[END_REF], targeted towards the label "tiger_shark", the same way CopyCAT does. It is trained on a small training set (10 batches of size 8), and tested on a random subset of ImageNet validation dataset. The network is taken from Keras pretrained models [START_REF] Chollet | Keras[END_REF] and attacked in a white-box setting. The same procedure as CopyCAT is used. 1000 images are randomly selected in the validation set. Only 80 are used for training and the rest is used for testing. The attack is trained with the same loss, same learning rate and same batch size as CopyCAT, for 200 epochs. The (rescaled) computed attack is shown in Fig. 13. Examples of attacked images from the test set are visible on Fig. 14. After 200 epochs, the train accuracy is 90% and the test accuracy 88.44%. This proof-of-concept experiment validates the existence of universal adversarial examples on realistic images and shows that CopyCAT's scope is not reduced to Atari-like environments. More generally, the existence of adversarial examples have been shown to be a property of high-dimensional manifolds [START_REF] Goodfellow | Explaining and harnessing adversarial examples[END_REF]. Going towards more realistic images, hence higher dimensional images, should on the opposite, allow CopyCAT to more easily find universal adversarial examples.

CONCLUSION

In this work, we built and showed the effectiveness of CopyCAT, a simple algorithm designed to attack neural policies in order to manipulate them. We showed its ability to lure a policy into having a desired behavior with a finite set of additive masks, usable in a real-time setting while being applied only on observations of the environment. We demonstrated the effectiveness of these universal masks in Atari games, in the white-box setting. We also investigated an extension of CopyCAT in the black-box setting and validated the possibility of using it on potentially more complex environments with realistic observations. As this work shows that one can easily manipulate a policy's behavior, a natural direction of work is to develop robust algorithms, either able to keep their normal behaviors when attacked or to detect attacks to treat them appropriately. Notice however that in a sequential-decision-making setting, detecting an attack is not enough as the agent cannot necessarily stop the process when detecting an attack and may have to keep outputting actions for incoming observations. It is thus an exciting direction of work to develop algorithm that are able to maintain their behavior under such manipulating attacks. Another interesting direction of work in order to build real-life attacks is to further develop targeted attacks on neural policies in the black-box scenario, with no access to network's weights and architecture. However, targeted adversarial examples are harder to compute than untargeted ones and we may experience more difficulties in reinforcement learning than supervised learning. Indeed, learned representations are known to be less interpretable and the variability between different random seeds to be higher than in supervised learning. Different policies trained with the same algorithm may thus lead to S → A mappings with very different decision boundaries. Transferring targeted examples may not be easy. In order to attack a policy π , training imitation models like the one from Hester et al. [START_REF] Hester | Deep q-learning from demonstrations[END_REF] to obtain proxy policies that are both (i) solving the same task as π and (ii) similar mappings S → A may be the key to compute transferable adversarial examples.

Figure 1 :

 1 Figure 1: Influence of parameters ϵ and α on the maximal L 2 norm of ∆: max a ∈A ∥δ a ∥ 2 . CopyCAT is attacking DQN on HERO.

Figure 2 :

 2 Figure 2: Influence of parameters ϵ and α on the average behavior of the agent playing Space Invaders under CopyCAT attack . On the left, the attack success rate. On the right, the average cumulative reward.

Figure 3 :

 3 Figure 3: Influence of parameters ϵ and α on the average behavior of the agent playing HERO under CopyCAT attack . On the left, the attack success rate. On the right, the average cumulative reward.

Figure 4 :

 4 Figure 4: Influence of parameters ϵ and α on the average behavior of the agent playing Air Raid under CopyCAT attack . On the left, the attack success rate. On the right, the average cumulative reward.

Figure 5 :

 5 Figure 5: CopyCAT against FGSM for policy targeted attacks in Space Invaders. On the left, the attack success rate. On the right, the average cumulative reward under attack.

Figure 6 :

 6 Figure 6: CopyCAT against FGSM for policy targeted attacks in HERO. On the left, the attack success rate. On the right, the average cumulative reward under attack.

Figure 7 :

 7 Figure 7: CopyCAT against FGSM for policy targeted attacks in Air Raid. On the left, the attack success rate. On the right, the average cumulative reward under attack.

Figure 8 :

 8 Figure 8: DQN playing Space Invaders. On the left, the unattacked current observation. In the middle, a mask of CopyCAT is applied to lure the agent into taking the "no-op" action. Parameters used are: ϵ = 3 • 10 -2 , α = 1.3 • 10 -5 . The L 2 norm of the corresponding mask (shown on the right) is 0.78. For this same ϵ, FGSM-L ∞ would produce an attack of L 2 norm: 2.52

Figure 9 :

 9 Figure 9: CopyCAT attacking DQN on Air Raid, in the black-box setting to make it match Rainbow's policy.

Figure 10 :

 10 Figure 10: Untrained DQN attacked by CopyCAT to follow Rainbow's policy on Air Raid. Left: the attack success rate. Right: the average cumulative reward.

Figure 11 :

 11 Figure 11: Untrained DQN attacked by CopyCAT to follow Rainbow's policy on Space Invaders. Left: the attack success rate. Right: the average cumulative reward.

Figure 12 :

 12 Figure 12: Untrained DQN attacked by CopyCAT to follow Rainbow's policy on HERO. Left: the attack success rate. Right: the average cumulative reward.

Figure 13 :

 13 Figure 13: The rescaled attack for the target class "tiger_shark".

Figure 14 :

 14 Figure 14: Top: images from the test set, unseen during the training of the attack. Down: attacked images: (image + attack). Below each image, the label predicted by VGG16.