Clément Zanolli

David M Alba

Christopher M Dean

Josep Fortuny

Roberto Macchiarelli

Lorenzo Rook

Oreopithecus bambolii is still an "enigmatic anthropoid"

à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Oreopithecus bambolii lived in the Tusco-Sardinian archipelago between c. 8.3 and 6.7 Ma. Its evolution in an insular context was likely responsible for the development of a number of peculiar cranio-dental features which render the assessment of its phylogenetic relationships quite problematic compared to other late Miocene Eurasian catarrhines. Even if its dentition shows a blend of hominoid, cercopithecoid-like and unique features (such as the development of a molar centroconid), Oreopithecus is regarded by several researchers as a derived member of the dryopithecine subfamily. By using high resolution X-ray microtomography, we detailed the inner structural morphology of three Oreopithecus lower permanent molar crowns (the M1 FS1996#Fi98, the M2 FS1996#Fi99 and the M3 FS1996#Fi97) from the Sardinian site of Fiume Santo and compared their 3D enamel thickness (ET) topographic distribution and enamel-dentine junction (EDJ) morphology with those from representatives of some European fossil anthropoids (pliopithecoid taxa belonging to Pliopithecus and the hominoids Dryopithecus and Ouranopithecus), extant hominoids (Homo, Pan, Gorilla, Pongo, Hylobates), and Papio. In Oreopithecus, the pattern of ET topographic distribution, where the thickest tissue is found along the mesiodistal axis of the crown, is unique among the taxa considered in this study. The results of the 3D geometric morphometric analysis (GMA) discriminate the relatively high dentine horns and EDJ conformation of Oreopithecus from the typical cercopithecoid condition, on the one hand, and from the fossil and extant hominoid patterns (including in Dryopithecus). Oreopithecus also differs from the pliopithecoid condition, except for the M1 IPS41955 that closely resembles the former.

Funding and support provided by: French CNRS, Generalitat de Catalunya (2014-SGR-416, GRC), Spanish Ministerio de Economía y Competitividad (CGR2014-54373-P), National Geographic Society (grant #7484-03), University of Berkeley (project NSF-BCS-0321893).

UMR 5288 CNRS, Université Toulouse III Paul Sabatier, France,

Institut Català de Paleontologia Miquel Crusafont, Universitat Autònoma de Barcelona, Spain,

Department of Cell and Developmental Biology, University College of London, UK,

Centre de Recherches en Paléobiodiversité et Paléoenvironnements, Muséum National d'Histoire Naturelle, Paris, France,

UMR 7194 CNRS, Muséum National d'Histoire Naturelle, Paris, France,

Unité de Formation Géosciences, Université de Poitiers, France,

Dipartimento di Scienze della Terra, Università di Firenze, Italy.


Clémentine Côte

Anne Piram

Alexandre Lacoste

Denis Josse

Pierre Doumenq

Sorption of nerve agent simulants vapors on human scalp hair Authors

Keywords: Chemical warfare agents, triethyl phosphate, diisopropyl fluorophosphate, G-agents, passive sensor

Human scalp hair is a biological matrix that can trap chemical vapors from explosives (TNT), drugs (THC) and chemical weapons (yperite). The external contamination of human's hair following exposure to organophosphorus (OP) nerve agent was simulated by model compounds: triethyl phosphate (TEP) and diisopropyl fluorophosphate (DFP). In this work were exposed strands of hair to vapors of TEP and DFP (3 and 7 ppmv) to model sorption kinetics. Sorption isotherms were also investigated at several contamination levels (80 to 3000 mg.min.m -3 ). So following hair exposure extractions of OP nerve agent simulants from hair were conducted by soaking in 5 mL of DCM for 10 minutes under orbital mixing. Raw extracts were analyzed in GC-MS/MS to quantify each simulant content in hair. To conclude results are fitted by applying isotherm or kinetic equations and the best model is bimodal first-order to fit kinetic data, suggesting the co-existence of two different mechanisms of sorption. The best equation to describe OP vapors incorporation on hair is Freundlich model. Thus hair can be used as a passive sensor able to trap chemical warfare agents and can also offer valuable information regarding both individual contamination and proof of exposure to chemical weapons.

Introduction

Chemical warfare agents (CWA) are highly toxic substances whose properties have been used to kill, injure or incapacitate people during conflicts or terrorist attacks [START_REF] Johnson | Chapter 2 -Historical Perspective of Chemical Warfare Agents A2 -Gupta[END_REF][START_REF] Smart | Chapitre 2 History of chemical and biological warfare : an american perspective[END_REF]. The issues surrounding chemical weapons are not new. Indeed from ancient Greece (431-404 b.c.) during Peloponnesian Spartans used ignited pitch and sulphur at Plataea and Delium [START_REF] Szinicz | History of chemical and biological warfare agents[END_REF][START_REF] Bertrand | Risques chimiques: accidents, attentats[END_REF]. In 1675 the first agreement was signed by Germany and France in Strasbourg (France) to ban the use of poisoned bullets [START_REF] Johnson | Chapter 2 -Historical Perspective of Chemical Warfare Agents A2 -Gupta[END_REF][START_REF] Szinicz | History of chemical and biological warfare agents[END_REF].

Various CWA including phosgene, yperite and lewisite were widely employed during World War I and caused 100'000 deaths and 1.2 million injured people. Subsequently, during World War II, millions of innocent civilians were killed in concentration camps by carbon monoxide, hydrogen cyanide and Zyklon B. For the 1970s people use more the chemistry on the weapons and their potential damage to fight [START_REF] Smart | Chapitre 2 History of chemical and biological warfare : an american perspective[END_REF][START_REF] Szinicz | History of chemical and biological warfare agents[END_REF][START_REF] Robinson | The Problem of Chemical and Biological Warfare[END_REF]. To have an operational global control, Organisation for the Prohibition of Chemical Weapons was entered into force in 1997, implementing Chemical Weapons Convention [START_REF] Pitschmann | Overall View of Chemical and Biochemical Weapons[END_REF]. This convention was ratified definitively by 192 Member States through the world in October 2015 (some of other countries still negotiate about it nowadays) and they promised to stop the development, the production, the storage and the use of all chemical weapons [START_REF] Opcw | Convention on the prohibition of the development, production, stockpiling and use of chemical weapons and on their destruction[END_REF]. Unfortunately exposures to CWA recently occurred [START_REF] Pita | The Use of Chemical Weapons in the Syrian Conflict[END_REF]. The threat of use of chemical agents has not been permanently ruled out and has been proved during conflicts (Syria, 2013) [START_REF] Bajgar | Chapter 3 -Global Impact of Chemical Warfare Agents Used Before and After 1945 A2 -Gupta[END_REF], terrorist attacks (Tokyo subway, 1995) [START_REF] Bajgar | Chapter 3 -Global Impact of Chemical Warfare Agents Used Before and After 1945 A2 -Gupta[END_REF][START_REF] Okumura | Chapter 4 -The Tokyo Subway Sarin Attack: Acute and Delayed Health Effects in Survivors A2 -Gupta[END_REF], assassinations (Kim Jong-Nam, 2017) [START_REF] Patočka | KIM JONG-NAM? WAS IT THE AGENT VX?[END_REF] and assassination attempts (Skripal, 2018) [START_REF] Vale | Novichok: a murderous nerve agent attack in the UK[END_REF].

The organophosphorus (OP) nerve agents are the largest group of CWA and are highly toxic substances [START_REF] Bajgar | Laboratory diagnosis of organophosphates/nerve agent poisoning[END_REF][START_REF] Bajgar | Organophosphates/nerve agent poisoning: mechanism of action, diagnosis, prophylaxis, and treatment[END_REF][START_REF] Ellison | Handbook of Chemical and Biological Warfare Agents, Second Edition[END_REF][START_REF] Lillie | Potential military chemical/biological agents and compounds[END_REF][START_REF] Watson | Chapter 9 -Organophosphate Nerve Agents A2 -Gupta[END_REF] because they exhibit lethal doses of exposures (LCt50) lower than 400 mg.min.m - 3 and they inhibit the enzyme acetylcholinesterase (AChE) which is the primary enzyme responsible for the hydrolytic metabolism of the neurotransmitter acetylcholine (ACh) into choline and acetate.

That causes the accumulation of acetylcholine in the cholinergic synapses [START_REF] Bajgar | Laboratory examination nerve agent intoxication[END_REF]. The first clinical signs of OP nerve agents poisoning are the appearance of headaches, breathing difficulties and miosis (if the eyes are exposed). In the hardest cases are noticed important clinical signs : muscular paralysis, hypersalivation and death by asphyxia [START_REF] Vucinic | Environmental exposure to organophosphorus nerve agents[END_REF].

An exposure to CWA can be proved by using biological matrices such as blood and urine [START_REF] Bajgar | Laboratory examination nerve agent intoxication[END_REF]. But their usages are limited to a very short time frame (2-3 days in urine and a few weeks in blood) what restricts their utilization as a proof of CWA exposure for living persons. Then the key point of this work is the identification of a sensor that could confirm an exposure to CWA after a long period. For this purpose human scalp hair is an interesting alternative compared to blood and to urine hair. In fact hair has been already used to demonstrate the drug consumption or the toxin exposure months and sometimes years after the occurrence [START_REF] Pragst | State of the art in hair analysis for detection of drug and alcohol abuse[END_REF][START_REF] Moeller | Hair analysis as evidence in forensic cases[END_REF][START_REF] Henderson | Mechanisms of drug incorporation into hair[END_REF]. Thus the detection time frame of contaminants in hair can be significantly increased compared to blood and to urine [START_REF] Pragst | State of the art in hair analysis for detection of drug and alcohol abuse[END_REF].

The hair surface area is estimated at 0.40 m².g -1 [START_REF] Hessefort | True porosity measurement of hair: a new way to study hair damage mechanisms[END_REF]. The accessible surface on hair depends on the amount available (5-200 g on average) and consequently hair offers an accessible surface from 2 to 80 m². As a comparison, in previous studies, the unprotected naked skin surface (head, neck, arms and hands) was 0.48 m² and the full-body surface is on average 1.8 m² for a uniformed soldier [START_REF] Baker | Role of body surface area in dosing of investigational anticancer agents in adults, 1991-2001[END_REF][START_REF] Wester | Predicted Chemical Warfare Agent VX Toxicity to Uniformed Soldier Using Parathion in Vitro Human Skin Exposure and Absorption[END_REF]. Thus the advantage of the hair surface area admit to achieve high sampling rate and then low detection limits. Scalp hair can bring information about exogenous contamination and can offer a valuable diagnostic tool to prove that a suspect was or is still in contact with illicit substances [START_REF] Duvivier | Evidence based decontamination protocols for the removal of external Δ9-tetrahydrocannabinol (THC) from contaminated hair[END_REF][START_REF] Emídio | Validation of an analytical method for analysis of cannabinoids in hair by headspace solid-phase microextraction and gas chromatography-ion trap tandem mass spectrometry[END_REF][START_REF] Madry | Metabolite to parent drug concentration ratios in hair for the differentiation of tramadol intake from external contamination and passive exposure[END_REF][START_REF] Tsanaclis | Differentiation between consumption and external contamination when testing for cocaine and cannabis in hair samples[END_REF]. This concept was developed by Spiandore et al. who showed sulfur mustard simulant is detected in hair after exposure to vapors [START_REF] Spiandore | Desorption of sulphur mustard simulants methyl salicylate and 2-chloroethyl ethyl sulphide from contaminated scalp hair after vapour exposure[END_REF][START_REF] Spiandore | Efficacy of scalp hair decontamination following exposure to vapours of sulphur mustard simulants 2-chloroethyl ethyl sulphide and methyl salicylate[END_REF][START_REF] Spiandore | Hair analysis as a useful procedure for detection of vapour exposure to chemical warfare agents: simulation of sulphur mustard with methyl salicylate[END_REF]; and by Oxley et al. who showed military explosive compounds adsorb on hair [START_REF] Oxley | Accumulation of Explosives in Hair-Part 3: Binding Site Study*[END_REF][START_REF] Oxley | Hair as Forensic Evidence of Explosive Handling[END_REF][START_REF] Oxley | Accumulation of Explosives in Hair-Part II: Factors Affecting Sorption*[END_REF].

It is compulsory to practice decontamination after an exposure to CWA or industrial toxics in order to decrease the risk of adverse health effects for healthcare professionals and all people around.

In the UK [START_REF] Chilcott | UK's initial operational response and specialist operational response to CBRN and HazMat incidents: a primer on decontamination protocols for healthcare professionals[END_REF] proposed for the first time a protocol to integrate the hair decontamination after an individual exposure to liquid or to particulate substances of CWA and toxic industrial. That is why decontamination material should be used to clean head including hair and face before doing a protocol for hands and all potentially exposed area [START_REF] Chilcott | UK's initial operational response and specialist operational response to CBRN and HazMat incidents: a primer on decontamination protocols for healthcare professionals[END_REF].

Investigations were performed with surrogates that are able to mimic chemical and/or physical interactions with hair of the real G-agents -one of OP nerve agent chemical classes -to determine the ability of hair to trap OP nerve agents. This work focuses on the hair external contamination only through vapors allowing to check the quantity adsorbed on the surface of hair and to evaluate sorption kinetics and sorption isotherms. In this case the incorporation of vapors of triethyl phosphate (TEP) and diisopropyl fluorophosphate (DFP) on hair was examined.

Materials and methods

Chemicals, reagents and sample collection

Analytical grade dichloromethane (DCM ≥ 99.8 %) was purchased from Merck (VWR, France). Triethyl phosphate (TEP) (CAS No. 78-40-0) and diisopropyl fluorophosphate (DFP) (CAS No. 55-91-4) were supplied by Sigma Aldrich. Working solutions were prepared by dissolving TEP and DFP separately into DCM (0.5 mg.mL -1 ). Natural blond hair strands (30 cm length) were obtained from Secher-Fesnoux (Chaville, France).

2.2.Sorption experiments and extraction process

First, a 150 mg hair strand (around 5 cm length) was hung in the center of a 1 L roundbottomed flask. Next the contaminant (working solution) was separately added and was evaporated by gentle heating. Then the experiment was placed in a thermostated bath (20 °C).

Second, sorption kinetics were evaluated over 1 hour by exposing scalp hair strands to 50 µL (50 mg.min.m -3 to 1500 mg.min.m -3 ) and 100 µL (100 mg.min.m -3 to 3000 mg.min.m -3 ) of working solutions. Each experiment was performed in duplicate (n=2).

Third, sorption isotherms were performed over 30 minutes, scalp hair strands were exposed to simulant vapors emitted by 5 to 200 µL of working solutions (80 mg.min.m -3 to 3000 mg.min.m - 3 ). Each experiment was performed in triplicate (n=3).

Finally, at the end of the simulant exposure every hair extraction was conducted by soaking hair in 5 mL of DCM for 10 minutes under orbital mixing at a shaking speed of 250 rpm in order to succeed in this experiment.

Analytical method

1000 µL of raw extract was spiked with 10 µL of an internal standard (50 µg.mL -1 in DCM).

TEP was used as internal standard for DFP sorption experiments, and DFP was used as internal standard for TEP. The concentrations of each simulant content in hair (qt) were determined by GC-MS/MS analysis. The amount (qt) of simulant adsorbed per mass unit of hair (µg.g -1 ) was calculated according to Eq. [START_REF] Johnson | Chapter 2 -Historical Perspective of Chemical Warfare Agents A2 -Gupta[END_REF].

𝑞 𝐶 𝑉 𝑚 1
Where Ct and V are respectively the concentration (µg.mL -1 ) and the volume (mL) of raw extract and m is the mass of hair strands (g).

The analyses were performed by Gas Chromatography (7890A Agilent Technologies) coupled to tandem Mass Spectrometer (7000 Agilent Technologies) equipped with an autosampler (7693, Agilent Technologies). The separation was carried out on HP5-MS column (5%-phenyl, 95% dimethylpolysiloxane; 30 m × 0.25 mm × 0.25 μm) with a constant Helium flow rate of 1 mL.min -1 .

For each analysis, 1 μL of the solution was injected into the injector at 250 °C in a splitless mode with a purge time of one minute. The oven temperature program was applied as follows: 1 min at 50 °C, then the temperature increases at a rate of 100 °C.min -1 in order to arrive to 80 °C held for 2 min then new temperature ramp at 20 °C.min -1 in order to arrive to 280 °C as a final temperature and held for 3 min for a complete run of 16.3 minutes with a solvent delay at 4 min. The transfer line was maintained at 260 °C and the mass spectrometer was equipped with an Electron Impact (EI, 70 eV) ionization source at 230 °C and two quadrupole analyzers at 150 °C. The mass transitions were performed in multiple reactions monitoring (MRM) mode and detection parameters were optimized and are summarized in Table 1.

Quality assurance

A quality control procedure was constructed to ensure the reproducibility of the analytical method. For each analytical sequence a calibration curve (0.5 to 2000 ng.mL -1 ) was carried out with a minimum of 6 points (R² > 0.99 and ± 10 % accuracy of each standard). Samples were separated by solvent blanks (DCM) in order to control the absence of cross contamination. A quality control solution was injected every 12 samples. Their concentrations were back calculated from the calibration curve in order to check the accuracy of the method. The unspiked human hair scalps were also analyzed to check the absence of carryover in blank contamination. The target compounds have never been detected.

Instrumental detection and quantification limits (LOD and LOQ) were determined with a standard solution and by calculating respectively a signal to noise ratio (S/N) of three and ten (Table 2). The sample's quantification was possible above 0.4 and 0.04 ng.mL -1 . So we can detect values until 0.02 µg.g -1 for TEP and 0.002 µg.g -1 for DFP. The Method Quantification Limit (MQL) was estimated following Eq. ( 2):

𝑀𝑄𝐿 5 𝐿𝑂𝑄 𝑚 2
Where LOQ is the instrumental quantification limit and m the mass of the strand of hair (Table 2). In this work uncertainties to each measurement point correspond to 3 times the standard deviation.

Results and discussion

3.1.Choice of simulants for G-agents

This work was carried out with simulants due to the toxicity and restriction of the OP nerve agents utilization [START_REF] Vucinic | Environmental exposure to organophosphorus nerve agents[END_REF][START_REF]OPCW, Fact sheet 7: Monitoring Chemicals with Possible Chemical Weapons Applications[END_REF]. The selection of compounds that can correctly mimic OP nerve agent behaviors in gas phase and their interactions with human's hair was then crucial. So, in order to extrapolate our results to G-agents, an important attention is given to the analysis of similarities concerning physico-chemical properties (vapor pressure, Henry constant and partition coefficient) or chemical structure (Table 3). As claimed by Bartelt-Hunt et al. a single simulant cannot mimic all Gagents properties [START_REF] Bartelt-Hunt | A Review of Chemical Warfare Agent Simulants for the Study of Environmental Behavior[END_REF].

Thus, two specific simulants were carefully selected to better cover physical and chemical properties of OP G-agents: TEP and DFP. They both have been often used as G-agent simulants [START_REF] Bartelt-Hunt | A Review of Chemical Warfare Agent Simulants for the Study of Environmental Behavior[END_REF] and they present a semi-persistent behavior as G-agents because of their volatility (Table 3). allows to correctly mimic some chemical relationships with hair (Table 3). Moreover as shown in Table 3 Henry's law constant are in the same range as G-agents (1.5.10 -7 -5.7.10 -7 ) and simulants (3.6.10 -8 -3.2.10 -6 ).

3.2.Sorption kinetics

Hair contamination by TEP and DFP was evaluated immediately after the hair vapor exposure.

The amounts of surrogate accumulated in hair at different time frames are presented in Figure 1. As expected TEP and DFP content in hair increased both with time of exposure and concentration of simulant in the vapor phase (Figure 1). It was noteworthy that even for very short exposure times (5 min) simulant residues have been quantified in a 150 mg hair strand. As a very small amount of hair is required to perform this analysis, scalp hair sampling could reasonably be performed on a victim in the theatre of operation. The amount of both simulants recovered in hair increased with exposure time. At 3 ppmv exposure 1.50±0.2 µg.g -1 and 0.130±0.009 µg.g -1 were detected after five minutes of TEP and DFP exposure against 6±2 µg.g -1 and 0.24± 0.01 µg.g -1 after one hour. At 7 ppmv exposure 4.7±0.9 µg.g -1 and 0.5±0.3 µg.g -1 were detected after five minutes of TEP and DFP exposure against 9.2±1.5 µg.g -1 and 1.3± 0.1 µg.g -1 after one hour. A plateau was observed after 30 min of exposure to 7 ppmv TEP vapors what suggested an equilibrium reached state.

Data were fitted according to the most common kinetic models in order to assess the sorption mechanisms: pseudo-first-order (PFO) [START_REF] Ho | A Comparison of Chemisorption Kinetic Models Applied to Pollutant Removal on Various Sorbents[END_REF], bimodal first-order [START_REF] Li | Organophosphate flame retardants emitted from thermal treatment and open burning of e-waste[END_REF], Elovich [START_REF] Largitte | A review of the kinetics adsorption models and their application to the adsorption of lead by an activated carbon[END_REF] and intra-particle diffusion models [START_REF] Weber | Kinetics of Adsorption on Carbon from Solution[END_REF].

The PFO equation based on Lagergren's model was widely used in sorption kinetics [START_REF] Azizian | Kinetic models of sorption: a theoretical analysis[END_REF][START_REF] Vasanth Kumar | Equilibrium data, isotherm parameters and process design for partial and complete isotherm of methylene blue onto activated carbon[END_REF][START_REF] Tseng | Characteristics and applications of the Lagergren's firstorder equation for adsorption kinetics[END_REF] which only involve the substrate concentration in the rate determining step. This model was generally applied over initial time of the sorption process (20-30 minutes) [START_REF] Ho | A Comparison of Chemisorption Kinetic Models Applied to Pollutant Removal on Various Sorbents[END_REF][START_REF] Tran | Mistakes and inconsistencies regarding adsorption of contaminants from aqueous solutions: A critical review[END_REF] and was used to experimental data as follows Eq.( 3). This specific model was based on the sorption capacity of solids.

ln q q ln q k . t 3 Where qe and qt (µg.g -1 ) represent simulant amount sorbed at equilibrium and at any time t (min) and k1 is the PFO rate constant (min -1 ).

It was therefore relevant to also consider the combination of multiple first order processes resulting from two regimes of sorption: one fast and the other slow. Then a bimodal first-order equation was applicable for our experimental data as follows Eq.( 4) [START_REF] Schuhfried | Desorption kinetics with PTR-MS: Isothermal differential desorption kinetics from a heterogeneous inlet surface at ambient pressure and a new concept for compound identification[END_REF].

q 𝑞 . e 𝑞 . 𝑒 𝑏 4

Where qt (µg.g -1 ) represent simulant amount sorbed at any time t (min); q1 and q2 (µg.g -1 ) represent simulant amount partial sorbed at equilibrium; k2 and k3 are the bimodal first-order rate constant (min - 1 ) and b the intercept related to the sorption step.

Elovich equation was also another model often used to chemisorption data as follows Eq.( 5) [START_REF] Largitte | A review of the kinetics adsorption models and their application to the adsorption of lead by an activated carbon[END_REF][START_REF] Wu | Characteristics of Elovich equation used for the analysis of adsorption kinetics in dye-chitosan systems[END_REF]. This model was an empirical equation: the applicable equation to experimental data was developed by Chien and Clayton [START_REF] Chien | Application of Elovich equation to the kinetics of phosphate release and sorption in soils[END_REF].

𝑞 1 𝛽 ln 𝑡 1 𝛽 ln 𝛼. 𝛽 5
Where qt (µg.g -1 ) represents simulant amount sorbed at any time t (min); 𝛼 and 𝛽 are respectively the initial rate (mg.g -1 .min) and the desorption constant during the experiments (mg.g -1 ).

Weber and Morris proposed also a model to describe internal diffusion [START_REF] Weber | Kinetics of Adsorption on Carbon from Solution[END_REF] Eq.( 6). This model was used to describe several reaction mechanisms such as surface diffusion and bulk diffusion [START_REF] Weber | Kinetics of Adsorption on Carbon from Solution[END_REF][START_REF] Tran | Mistakes and inconsistencies regarding adsorption of contaminants from aqueous solutions: A critical review[END_REF].

q k . t 𝑐 6
Where qt (µg.g -1 ) represents simulant amount sorbed at any time t (min); ki is the intraparticle diffusion rate constant (mg.g -1 .min -1/2 ) and c is the intercept related to the sorption step.

A regression was applied for all models (Figure 2). The model relevance was evaluated by the determination coefficient (R²) and the p-value in ANOVA tests (Table 4). The model that gave the highest determination coefficient and the lowest p-value was considered like the best to describe experimental data. All kinetic sorption equations were specified in Figure 2 and analysis parameters are determined (Table 4). PFO, Elovich and intra-particle diffusion models were all rejected -considering sorption kinetics of TEP on scalp hair -because the determination coefficient R² was below 0.7. It was consequently considered that these models could not fit experimental data. The bimodal first-order fitting was presented a high determination coefficient above 0.7 at low and at high exposure doses (respectively 0.8697 and 0.7162). The nonlinear adjustments of the rate of time on the amount content in hair (q) as a function of time were leaded to a satisfactory coefficient of determination.

Models

Furthermore, the p-values were less than the significance level of 0.05 and the residues were randomly distributed according to this model what validates the relevance of this model.

Satisfactory results were obtained for all fittings with the DFP data excepted with the PFO for the experiment realized at 3 ppmv. Indeed, we proved that the value of R² is not enough and that is why it invalidated the relevance of this model. Furthermore, bimodal first-order, Elovich and intraparticle diffusion gave us acceptable coefficient of determination (R² >0.7) and p-values (under the threshold value of 0.05). Bimodal first-order model described a better experimental data because it offered both the highest R² value and the lowest p-value as we observed for TEP.

The main differences between the two agents are volatility propriety and chemical structure (Table 1).

Even though the volatility of TEP (3 953 mg.m -3 ) is lower than this of DFP (5 748 mg.m -3 ) but the amount of TEP found in hair is around ten times higher than this of DFP (Figure 1). For a given chemical compound this suggested an increase of the accumulation in hair with a decreasing volatility. Consequently, we could expect that low volatility G-agents could be more persistent in hair and should prove to us a less labile behavior. TEP and DFP volatilities are in the same range as Gagents (497-20 660 mg.m -3 ): they are liquid at ambient temperature, and they all emit denser vapors than air (vapor density: 4.8-6.4).

All models described above (PFO, bimodal first-order, Elovich and intraparticle diffusion) are models that involve the combination of multiple sorption phenomena, either multilayered or both chemical and physical interactions which can be partly explained by the complexity of the scalp hair matrix.

Indeed human hair is a natural fiber of α-keratin divided in three main components: cuticle, cortex and medulla -on average 50-100 µm thickness -and has a specific elementary composition: around 51% carbon, 21% oxygen, 17% nitrogen and the rest are hydrogen and sulfur [START_REF] Satish | Macroporous carbon from human hair: A journey towards the fabrication of high energy Li-ion capacitors[END_REF][START_REF] Bhushan | Biophysics of Human Hair: Structural, Nanomechanical, and Nanotribological Studies[END_REF][START_REF] Popescu | Hair-the most sophisticated biological composite material[END_REF]. Furthermore the cortex is highly porous with various functional groups like carbonyl and amino groups [START_REF] Satish | Macroporous carbon from human hair: A journey towards the fabrication of high energy Li-ion capacitors[END_REF][START_REF] Yu | Structure and mechanical behavior of human hair[END_REF].

So hair is a particularly complex substrate with an irregular and heterogeneous surface and hairsimulant interaction of different energies can coexist according to the nature of sorption sites. Besides several kinetics specific to each type of interaction can intervene in the sorption.

The acceptable correlation was observed for bimodal first-order for both simulants. The Elovich model brings to light the intra-particle diffusion and was often used to describe surface diffusion and bulk diffusion [START_REF] Weber | Kinetics of Adsorption on Carbon from Solution[END_REF][START_REF] Tran | Mistakes and inconsistencies regarding adsorption of contaminants from aqueous solutions: A critical review[END_REF]. The heterogeneity of the hair surface [START_REF] Robbins | Chemical and Physical Behavior of Human Hair, Fifth Edition[END_REF] could lead to multiple interactions with OP nerve agent simulants depending on the position of the sorption site. Modeling by two sorption regimes from the hair can be coherent with this structure. Then we conclude that the sorption kinetics of OP nerve agent simulants is better represented by bimodal first-order model.

These two compounds TEP and DFP were both chosen to mimic nerve organophosphorus Gagents comportment. Thus it is reasonable to deduce and to assume that G-agents vapors will be trapped by hair and that the quantity recovered in hair will increase with the time of exposure. These results prove us that hair can be used as a passive sensor and bring some information about OP nerve agents vapor exposure.

3.3.Sorption Isotherms

Hair contamination by TEP and DFP vapors were revealed by hair extraction immediately after exposure. Sorption isotherms were conducted at 20 °C for a fixed exposure time (30 minutes). As we expected TEP and DFP content in hair increased with concentration of simulant in the vapor phase.

The amounts of simulant accumulated in hair along doses of exposure are presented in Figure 3. It is noteworthy that even for very short doses of exposure (0.02 Pa) simulant residues were quantified in hair (0.02±0.01 and 0.006±0.003 µg.g -1 for TEP and DFP). The sorption isotherms of TEP and DFP were investigated. The isotherm data were fitted by applying the Langmuir and Freundlich isotherm relations. The Langmuir model is commonly used on sorption isotherms and the linear equation form is shown in Eq.( 7) [START_REF] Tran | Mistakes and inconsistencies regarding adsorption of contaminants from aqueous solutions: A critical review[END_REF][START_REF] Langmuir | [END_REF] and also applied recently by Wang and al [START_REF] Wang | Comparison of Langmuir and Freundlich adsorption equations within the SWAT-K model for assessing potassium environmental losses at basin scale[END_REF].

C 𝑞 1 𝑄 𝐶 1 𝑄 𝐾 7
Where qe (µg.g -1 ) represents simulant amount sorbed; Ce (Pa) is simulant amount exposed; Q o max (µg.g -1 ) is the maximum saturated monolayer sorption capacity of sorbent and KL (Pa) is a constant related to the affinity between a sorbent and a sorbate. The Freundlich model is one of the most commonly used equilibrium relations to describe heterogeneous systems [START_REF] Freundlich | Over the Adsorption in Solution[END_REF][START_REF] Ho | Review of second-order models for adsorption systems[END_REF]. The linear form is described by Eq.( 8).

log 𝑞 log 𝑘 1 𝑛 . log 𝐶 8

Where qe (µg.g -1 ) represents simulant amount sorbed; Ce (Pa) is simulant amount exposed, kf is the Freundlich constant (µg.g -1 .Pa -n ), the sorption capacity of the sorbent and 1/n without dimension represents the heterogeneity factor to sorption intensity.

A linear regression was applied (Figure 4) for all models and the model relevance were evaluated with the determination coefficient (R²) and the p-value in ANOVA test. good (0.9313 and 0.9952) and the p-values are less than the significance level of 0.05 (8.10 -7 and 1.10 -6 ). The linear adjustments logarithmic of the amount detected in hair (log q) as a function of logarithmic amount exposed (log C) leads to validate this model for both simulants. Furthermore the Freundlich model is consistent for multilayer adsorptions and take into account the heterogeneity of the adsorbent that can be expected for hair substrate [START_REF] Wang | Comparison of Langmuir and Freundlich adsorption equations within the SWAT-K model for assessing potassium environmental losses at basin scale[END_REF].

So the sorption isotherms of TEP and DFP are best fitted by the Freundlich model. The sorption of G-agents vapor on hair could have a similar behavior to these two simulants by extrapolation. Thus the contaminant levels detected could depend on the dose of exposure and the Freundlich model could be used to interpret the data when the hair is in contact with a G-agent. Specific models offer a good and a better fitting for G-agents (bimodal first-order and Freundlich) reminding that we consider both models very important: sorption kinetic model and sorption isotherm model. Even if hair is a particularly complex substrate with an irregular and heterogeneous surface we can see that hair-simulants interactions of different energies can coexist according to the sorption sites. Our findings show that hair can be used as a passive sensor for organophosphorus substances despite the complexity of this matrix. The G-agents simulants TEP and DFP could be identified and quantified in hair when exposure concentrations were close to the Gagents LCt50.

We demonstrated that after vapor exposure to OP comprised between 80 and 3000 mg.min.m -3 hair analysis can offer a valuable diagnostic tool for the identification of OP and an evaluation of the level of contamination in this study. In fact hair analysis can provide both qualitative and quantitative information even below the lethal dose. Finally hair samples could be used to diagnose an exposure to nerve agents.

Hair surface adsorption has already been investigated with other organic molecules, i.e. sulfur mustard [START_REF] Spiandore | Hair analysis as a useful procedure for detection of vapour exposure to chemical warfare agents: simulation of sulphur mustard with methyl salicylate[END_REF], cannabinoids [START_REF] Duvivier | Evidence based decontamination protocols for the removal of external Δ9-tetrahydrocannabinol (THC) from contaminated hair[END_REF], cocaine [START_REF] Wang | Testing human hair for drugs of abuse. IV. Environmental cocaine contamination and washing effects[END_REF] and explosives [START_REF] Oxley | Accumulation of Explosives in Hair-Part 3: Binding Site Study*[END_REF][START_REF] Oxley | Hair as Forensic Evidence of Explosive Handling[END_REF][START_REF] Oxley | Accumulation of Explosives in Hair-Part II: Factors Affecting Sorption*[END_REF]. The hair capacity to trap external vapor contamination was demonstrated for all these target substances after a specific extraction procedure. Concerning [START_REF] Spiandore | Hair analysis as a useful procedure for detection of vapour exposure to chemical warfare agents: simulation of sulphur mustard with methyl salicylate[END_REF], they worked on a useful procedure for sulfur mustard detection on hair -another CWA. Their project underlined the hair capacity to trap CWA vapor and confirmed our statement: the intensity and the time of exposure are crucial. However the kinetic and isothermal models assigned were different. In their researches they were applied a pseudo-second-order equation and Langmuir isotherm model what implies a variable affinity with the hair for OP nerve agents and sulfur mustard. These variations are easily explained by their own structure and physicochemical properties which generate variable behaviors with human's hair. All these and their results encouraged us to use hair as a passive sensor for CWA and identification of these agents following an individual exposure. Finally all the data obtained in our work allow us to get information of the sorption of OP nerve agents concerning human's hair which are important for exposure risk assessments. Hair is able to capture a wide range of volatile compounds and various chemical properties (atmospheric pollutants, chemical weapons) [START_REF] Emídio | Validation of an analytical method for analysis of cannabinoids in hair by headspace solid-phase microextraction and gas chromatography-ion trap tandem mass spectrometry[END_REF][START_REF] Spiandore | Efficacy of scalp hair decontamination following exposure to vapours of sulphur mustard simulants 2-chloroethyl ethyl sulphide and methyl salicylate[END_REF][START_REF] Spiandore | Hair analysis as a useful procedure for detection of vapour exposure to chemical warfare agents: simulation of sulphur mustard with methyl salicylate[END_REF][START_REF] Oxley | Hair as Forensic Evidence of Explosive Handling[END_REF][START_REF] Oxley | Accumulation of Explosives in Hair-Part II: Factors Affecting Sorption*[END_REF]. That is why we can affirm that during a chemical attack, exposed people's hair will capture these organic compounds and the detection of a G-agent would bring the proof that a specific OP nerve agent was used after hair sampling and analysis.

Conclusion

To summarize, external contamination of scalp hair are observed when hair is exposed to vapor simulants (3 and 7 ppmv) for various times (80 to 3000 mg.min.m -3 ). This shows the usefulness of hair analysis as a proof of exposure to chemical weapons. The experimental results demonstrate that the amount of both simulants recovered in hair after different times of TEP and DFP exposure increases with exposure time. Results also show that sorption kinetics follows a bimodal first-order kinetic and that isotherms are fitted better when using the Freundlich model for both simulants. By extrapolation, we consider the sorption of G-agents vapor on hair similar of the two simulants (TEP and DFP).

Finally all the data obtained in our work allow us to get information of the sorption of OP nerve agents concerning human's hair which are important for exposure risk assessments. We demonstrated hair can be used especially to reveal an exposure or handling of prohibited substances like organophosphorus nerve agents and thus to offer valuable information for health and for security services.

  The use of TEP (VP = 0.39 mmHg) and DFP (VP = 0,519 mmHg) are in good agreement with G-agent's volatility (range 0.057 -2.74 mmHg). Furthermore values of octanol-water partition coefficients for G-agents are between 0.299 -1.824 against 0.800 and 1.173 for TEP and DFP respectively. All the log Kow are in the same range. Thus this value is an estimation of a chemical's trend to compare polarity to mimic interactions with the hair fiber. Their amphiphilic character involves rapid sorption, moderate storage, and therefore a rapid resorption in hair. They also present similar structures (C2H5-O-P=O; (C3H8)iso-O-P=O) with G-agents (C2H5-O-P=O for Tabun ; (C3H8)iso-O-P=O for Sarin) what

Figure 1 :

 1 Figure 1 : TEP (a) and DFP (b) sorption kinetics on hair at 2 vapor concentrations:  3 ppmv and  7 ppmv

Figure 2 :

 2 Figure 2 : Models for the sorption of TEP (a-d) and DFP (e-h) on hair; (a)(e) pseudo-first-order model, (b)(f) bimodal first-order, (c)(g) Elovich model and (d)(h) intraparticle diffusion model - 3 ppmv and  7 ppmv

Figure 3 :

 3 Figure 3 : Effects of TEP (Δ) and, DFP (Ο) sorption capacity on hair at different initial dose at vapor exposure after 30 minutes of exposure (n=3)

Figure 4 :

 4 Figure 4 : Langmuir (a) and Freundlich (b) models for the sorption of TEP () and DFP () on hair

Table 1 :

 1 Fragmentation and detection parameters used for target compounds (retention time, MRM quantifier and qualifier transitions and collision energies)

			Transitions for Quantification	Transitions for Qualification
	Compound	Retention time (min)	Precursor ion (m/z)	Product ion (m/z)	CE (V)	Precursor ion (m/z)	Product ion (m/z)	CE (V)
	TEP	6.35	154.9	98.9	10	154.9	127.0	2
	DFP	4.60	126.9	100.9	10	140.9	101.0	5

Table 2 :

 2 Quality parameters of the analytical method

	Compound	LOD (ng.mL -1 )	LOQ (ng.mL -1 )	MQL (µg.g -1 )
	TEP	0.12	0.4	0.02
	DFP	0.01	0.04	0.002

Table 3 :

 3 Physical, chemical and toxicological properties of OP nerve agents and simulants[START_REF] Ellison | Handbook of Chemical and Biological Warfare Agents, Second Edition[END_REF][START_REF] Lillie | Potential military chemical/biological agents and compounds[END_REF][START_REF] Watson | Chapter 9 -Organophosphate Nerve Agents A2 -Gupta[END_REF][START_REF] Bartelt-Hunt | A Review of Chemical Warfare Agent Simulants for the Study of Environmental Behavior[END_REF][START_REF] Czerwinski | Effect of octanol:water partition coefficients of organophosphorus compounds on biodistribution and percutaneous toxicity[END_REF][START_REF] Czerwinski | A Method for Measuring Octanol:water Partition Coefficients of Highly Toxic Organophosphorus Compounds[END_REF][START_REF] Munro | The sources, fate, and toxicity of chemical warfare agent degradation products[END_REF][START_REF]Subcommittee On Acute Exposure Guideline Levels, Acute Exposure Guideline Levels for Selected Airborne Chemicals[END_REF][START_REF] Singer | Indoor Sorption of Surrogates for Sarin and Related Nerve Agents[END_REF]

Table 4 :

 4 Kinetic model parameters for TEP and DFP sorption on hair.

	Models	Agents	TEP		DFP
		Exposition	3 ppmV	7 ppmV	3 ppmV	7 ppmV
	PFO	R²	0.9192	0.1071	0.6703	0.7257
		p-value	7.5.10 -6	0.1587	6.9.10 -4	2.7.10 -4
	Bimodal first-order	R²	0.8697	0.7162	0.8127	0.9551
		p-value	1.0.10 -4	1.1.10 -6	3.3.10 -8	3.5.10 -8
	Elovich	R²	0.7473	0.4043	0.7415	0.9309
		p-value	7.7.10 -4	0.0156	2.0.10 -4	2.5.10 -7
	Intra-particles	R²	0.8662	0.2374	0.7154	0.8424
	diffusion
		p-value	5.8.10 -5	0.0618	3.2

.10 -4 1.6.10 -5

Acknowledgments

The authors gratefully acknowledge the DGA (Direction Générale de l'Armement) and AMU (Aix-Marseille University) for this partnership and their financial support.

DFP Diisopropyl fluorophosphate OP Organophosphorus PFO Pseudo-first-order TEP Triethyl phosphate Funding This work was supported by the Direction Générale de l'Armement and Aix-Marseille University through a PhD grant.


Laurent Meunier

Herilalaina Rakotoarison

Pak Kan Wong

Baptiste Roziere

Jeremy Rapin

Olivier Teytaud

Antoine Moreau

Carola Doerr

Black-Box Optimization Revisited: Improving Algorithm Selection Wizards through Massive Benchmarking

Existing studies in black-box optimization for machine learning suffer from low generalizability, caused by a typically selective choice of problem instances used for training and testing different optimization algorithms. Among other issues, this practice promotes overfitting and poor-performing user guidelines. To address this shortcoming, we propose in this work a benchmark suite, OptimSuite, which covers a broad range of black-box optimization problems, ranging from academic benchmarks to real-world applications, from discrete over numerical to mixed-integer problems, from small to very large-scale problems, from noisy over dynamic to static problems, etc. We demonstrate the advantages of such a broad collection by deriving from it Automated Black Box Optimizer (ABBO), a general-purpose algorithm selection wizard. Using three different types of algorithm selection techniques, ABBO achieves competitive performance on all benchmark suites. It significantly outperforms previous state of the art on some of them, including YABBOB and LSGO. ABBO relies on many high-quality base components. Its excellent performance is obtained without any task-specific parametrization.

The OptimSuite benchmark collection, the ABBO wizard and its base solvers have all been merged into the open-source Nevergrad platform, where they are available for reproducible research.

Introduction: State of the Art

Many real-world optimization challenges are black-box problems; i.e., instead of having an explicit problem formulation, they can only be accessed through the evaluation of solution candidates. These evaluations often require simulations or even physical experiments. Black-box optimization methods are particularly widespread in machine learning [SGZ + 16, WFT20], to the point that it is considered a key research area of artificial intelligence. Black-box optimization algorithms are typically easy to implement and easy to adjust to different problem types. To achieve peak performance, however, proper algorithm selection and configuration are key, since black-box optimization algorithms have complementary strengths and weaknesses [Ric76, Smi09, Kot14, BKK + 16, [START_REF] Kerschke | Automated Algorithm Selection on Continuous Black-Box Problems By Combining Exploratory Landscape Analysis and Machine Learning[END_REF][START_REF] Kerschke | Automated Algorithm Selection: Survey and Perspectives[END_REF]. But whereas automated algorithm selection has become standard in SAT solving [START_REF] Xu | SATzilla: Portfolio-based algorithm selection for SAT[END_REF] and AI planning [START_REF] Vallati | On the effective configuration of planning domain models[END_REF], a manual selection and configuration of the algorithms is still predominant in the broader black-box optimization context. To reduce the bias inherent to such manual choices, and to support the automation of algorithm selection and configuration, sound comparisons of the different black-box optimization approaches are needed. Existing benchmarking suites, however, are rather selective in the problems they cover. This leads to specialized algorithm frameworks whose performance suffer from poor generalizability. Addressing this flaw in black-box optimization, Cobyla [START_REF]A direct search optimization method that models the objective and constraint functions by linear interpolation[END_REF] For all other cases and all details, please refer to the source code 2 Sound Black-Box Optimization Benchmarking

We summarize desirable features and common shortcomings of black-box optimization benchmarks and discuss how OptimSuite addresses these.

Generalization. The most obvious issue in terms of generalization is the statistical one: we need sufficiently many experiments for conducting valid statistical tests and for evaluating the robustness of algorithms' performance. This, however, is probably not the main issue. A biased benchmark, excluding large parts of the industrial needs, leads to biased conclusions, no matter how many experiments we perform. Inspired by [START_REF] Recht | Do CIFAR-10 classifiers generalize to CIFAR-10[END_REF] in the case of image classification, and similar to the spirit of cross-validation for supervised learning, we use a much broader collection of benchmark problems for evaluating algorithms in an unbiased manner. Another subtle issue in terms of generalization is the case of instance-based choices of (hyper-)parameters: an experimenter modifying the algorithm or its parameters specifically for each instance can easily improve results by a vast margin. In this paper, we consider that only the following problem properties are known in advance (and can hence be used for algorithm selection and configuration): the dimension of the domain, the type and range of each variable, their order, the presence of noise (but not its intensity), the budget, the degree of parallelism (i.e., number of solution candidates that can be evaluated simultaneously). To mitigate the common risk of over-tuning, we evaluate algorithms on a broad range of problems, from academic benchmark problems to real-world applications. Each algorithm runs on all benchmarks without any change or task-specific tuning.

Use the ask, tell, and recommend pattern. Formalizing the concept of numerical optimization is typically made through the formalism of oracles or parallel oracles [START_REF] Rogers | Theory of recursive functions and effective computability[END_REF]. A recent trend is the adoption of the ask-and-tell format [CHP + 10]. The bandit literature pointed out that we should distinguish ask, tell, and recommend : the way we choose a point for gathering more information is not necessarily close to the way we choose an approximation of the optimum [START_REF] Bubeck | Pure exploration in finitely-armed and continuous-armed bandits[END_REF][START_REF] Coulom | Clop: Confident local optimization for noisyblack-box parameter tuning[END_REF][START_REF] Decock | Noisy optimization complexity under locality assumption[END_REF]. We adopt the following framework: given an objective function f and an optimizer, for i ∈ {1, . . . , T }, do x ← optimizer.ask and optimizer.tell(x, f (x)). Then, evaluate the performance with f (optimizer.recommend). The requirement of a recommend method distinct from the ask is critical in noisy optimization. A debate pointed out some shortcomings in the the noisy counterpart of BBOB [START_REF] Auger | Benchmarking the (1+1)-CMA-ES on the BBOB-2009 noisy testbed[END_REF] which was assuming that ask = recommend: [Bey12a, Bey12b, Cou12a] have shown that in the noisy case, this difference was particularly critical, and a framework should allow algorithms to "recommend" differently than they "ask". A related issue is that a run with budget T is not necessarily close to the truncation of a run in budget 10T .

Translation-invariance. Zero frequently plays a special role in optimization. For example, complexity penalizations often "push" towards zero. In control, numbers far from zero are often more likely to lead to bang-bang solutions (and therefore extract zero signal, leading to a needle-in-the-haystack optimization situation), in particular with neural networks. In one-shot optimization, [CCD + 19, MDRT20] have shown how much focusing to the center is a good idea in particular in high-dimension. Our experiments in control confirm that the scale of the optimization search is critical, and explains the misleading results observed in some optimization papers (Section 4.2). In artificial experiments, several classical test functions have their optimum in zero. To avoid misleading conclusions, it is now a standard procedure, advocated in particular in [START_REF] Hansen | Real-parameter black-box optimization benchmarking[END_REF], to randomly translate the objective functions. This is unfortunately not always applied.

Rotation and symmetrization. Some optimization methods may perform well on separable objective functions but degrade significantly in optimizing non-separable functions. If the dimension of a separable objective function is d, these methods can reduce the objective function into d one-dimensional optimization processes [START_REF] Salomon | Re-evaluating genetic algorithm performance under coordinate rotation of benchmark functions. a survey of some theoretical and practical aspects of genetic algorithms[END_REF]. Therefore, [HAFR09, HRM + 11] have insisted that objective functions should be rotated to generate more difficult non-separable objective functions. However, [BGK + 17] pointed out the importance of dummy variables, which are not invariant per rotation; and [START_REF] Holland | Adaptation in natural and artificial systems[END_REF] and more generally the genetic algorithms literature insist that rotation does not always makes sense -we lose some properties of a realworld objective function, and in some real-world scenarios rotating would, e.g., mix temperature, distance and electric intensity. Permutating the order of variables is also risky, as their order is sometimes critical -k-point crossovers a la Holland [START_REF] Holland | Adaptation in natural and artificial systems[END_REF] typically assume some order of variables, which would be broken. Also, users sometimes rank variables with the most important first -and some optimization methods do take care of this [CCD + 19]. In OptimSuite, we do include rotations, but include both cases, rotated or not.

Table 1: Properties of selected benchmark collections (details in appendix). "+" means that the feature is present, "-" that the feature is missing, and an empty case that it is not applicable. Far-optimum refers to problems with optimum far from the center or on the side of the domain; such benchmarks test the ability of optimization algorithms to answer promptly to a bad initialization [START_REF] Alexandre | Cumulative Step-size Adaptation on Linear Functions[END_REF]. "Translations" applies only to artificial benchmarks. A "+" in rows "Multimodal", "symmetrization", and "real-world" does not imply that all test functions have this property. "Open sourced" refers to open access to most algorithms involved in the published comparison; here, "-" refers to license issues for the benchmark itself. For composite functions which use various objective functions on various subsets of variables, we consider the case with rotations -without excluding the non-rotated case. An extension of symmetrization that we will integrate later in ABBO, which makes sense for replicating an objective function without exact identity, consists in symmetrizing some variables: for example, if the i th variable has range [a, b], we can replace x i by b + a -x i . Applying this on various subsets of variables leads to 2 d symmetries of an objective function, if the dimension is d. This variation can reduce the bias toward symmetric search operations [LTO + 13].

Benchmarking in OptimSuite. We summarize in Table 1 some existing benchmark collections and their (desirable) properties. We inherit various advantages from Nevergrad, namely the automatic rerun of experiments and reproducibility in one line. Our fork includes PBT (a small scale version of Population-Based Training [JDO + 17]), Pyomo [HLW + 17], Photonics (problems related to optical properties and nanometric materials), YABBOB and variants, LSGO [LTO + 13], MLDA [START_REF] Gallagher | Exploratory landscape analysis of the mlda problem set[END_REF], PowerSystems, FastGames, 007, Rocket, SimpleTSP, Realworld [LMP + 20], MuJoCo [START_REF] Todorov | Mujoco: A physics engine for model-based control[END_REF], and others, including a (currently small) benchmark of hyperparameters of Scikit-Learn [PVG + 11] and Keras-tuning (underlined means: the benchmark is either new, or, in the case of PowerSystems or SimpleTSP, significantly modified compared to previous works, or, in the case of LSGO or MuJoCo, included for the first time inside Nevergrad. For Mu-JoCo, we believe that interfacing with Nevergrad is particularly useful, to ensure fair comparisons, which rely very much on the precise setup of MuJoCo. We note that, at present, we do not reproduce the extreme black-box nature of [START_REF] Loshchilov | Black box optimization competition[END_REF]. Still, by integrating such a wide range of benchmarks in a single open source framework, which, in addition, is periodically re-run, we believe that Nevergrad/OptimSuite provides a significant contribution to benchmarking, and this both for the optimization and the machine learning community, where most of the benchmark suites originate from.

A New Algorithm Selection Wizard: ABBO

Black-box optimization is sometimes dominated by evolutionary computation. Evolution strategies [BS02, Bey01, Rec73] have been particularly dominant in the continuous case, in experimental comparisons based on the Black-Box Optimization Benchmark BBOB [START_REF] Hansen | Real-parameter black-box optimization benchmarking[END_REF] or variants thereof. Parallelization advantages [SGZ + 16] are particularly appreciated in the machine learning context. However, Differential Evolution [START_REF] Storn | Differential evolution -a simple and efficient heuristic for global optimization over continuous spaces[END_REF] is a key component of most winning algorithms in competitions based on variants of Large Scale Global Optimization (LSGO [LTO + 13]), suggesting a significant difference between these benchmarks. In particular, LSGO is more based on correctly identifying a partial decomposition and scaling to ≥ 1 000 variables, whereas BBOB focuses (mostly, except [VAB + 18]) on ≤ 40 variables. Mathematical programming techniques [Pow64, Pow94, NM65, Art15] are rarely used in the evolutionary computation world, Table 2: Nevergrad maintains a dashboard [START_REF]Dashboard of results for Nevergrad platform[END_REF]. For each experiment, there are many configurations (budget, objective function, possibly dimension, and noise level). We separate benchmarks used for designing ABBO, benchmarks used for validation, and those only used for testing. "*" denotes benchmarks used for designing Shiwa (which is used inside ABBO). We present the rank based on the winning rate of ABBO in the dashboard. Since the submission of this paper, several variants of bandit-based algorithms have been added for high-dimensional noisy optimization. They outperform ABBO, hence its poor rank for these cases. Detailed plots are available in Fig. 4, 7. As expected, DE variants are strong on LSGO and CMA-ES variants are strong for YABBOB. ABBO also performs well on YABBOB, which was used for designing its ancestor Shiwa (see Fig. 1). For the MuJoCo testbed, details are available in Table 3 and Figure 3. Our modifications in the codebase implies an improvement of Shiwa compared to the version published in [LMP + 20]; for example, our chaining implies that the (k + 1) th code starts from the best point obtained by the k th algorithm, which significantly improves in particular the chaining CMA-ES+Powell or CMA-ES+SQP. Experiments with " † " in the ranking of Shiwa correspond to this improved version of Shiwa. (1) Better use of chaining [START_REF] Molina | Memetic algorithm with local search chaining for continuous optimization problems: A scalability test[END_REF] and more intensive use of mathematical programming techniques for the last part of the optimization run, i.e., the local convergence, thanks to Meta-Models (in the parallel case) and more time spent in Powell's method (in the sequential case). This explains the improvement visible in Section 4.1.

(2) Better performance in discrete optimization, using additional codes recently introduced in Nevergrad, in particular adaptive step-sizes.

(3) Better segmentation of the different cases of continuous optimization. We still entirely rely on the base algorithms as available in Nevergrad; that is, we did not modify the tuning of any method. We acknowledge that our method only work thanks to the solid base components available in Nevergrad, which are based on contributions from various research teams.

The obtained algorithm selection wizard, ABBO, is presented in Algorithm 1. The performances of ABBO is summarized in Table 2 and a detailed dashboard is available at https://dl.fbaipublicfiles. com/nevergrad/allxps/list.html.

Experimental Results

When presenting results on a single benchmark function, we present the usual average objective function value for different budget values. When a collection comprises multiple benchmark problems, we present our aggregated experimental results with two distinct types of plots:

(1) Average normalized objective value for each budget, averaged over all problems. The normalized objective value is the objective value linearly rescaled to [0, 1].

(2) Heatmaps, showing for each pair (x, y) of optimization algorithms the frequency at which Algorithm x outperforms Algorithm y. Algorithms are ranked by average winning frequency. We use red arrows to highlight ABBO.

Benchmarks in OptimSuite Used for Designing and Validating ABBO

YABBOB (Yet Another Black-Box Optimization Benchmark [RDP + 19]), is an adaptation of BBOB [START_REF] Hansen | Real-parameter black-box optimization benchmarking[END_REF], with extensions such as parallelism and noise management. It contains many variants, including noise, parallelism, high-dimension (BBOB was limited to dimension < 50). Several extensions, for the high-dimensional, the parallel or the big budget case, have been developed: we present results in Figures 1 and4. The high-dimensional one is inspired by [LTO + 13], the noisy one is related to the noisy counterpart of BBOB but correctly implements the difference between ask and recommend as discussed in Section 2. The parallel one generalizes YABBOB to settings in which several evaluations can be executed in parallel. Results on PARAMULTIMODAL are presented in Figure 6 (left). In addition, ABBO was run on ILLCONDI & ILLCONDIPARA (ill conditionned functions), HDMULTIMODAL (a multimodal case focusing on high-dimension), NOISY & RANKNOISY (two noisy continuous testbeds), YAWIDEBBOB (a broad range of functions including discrete cases and cases with constraints).

Figure 1: Average normalized loss and heatmap for YABBOB. Additional plots for High-dimensional (HD), NoisyHD, and Large budgets are available in Fig. 4. Other variants include parallel, differences of budgets, and combinations of those variants, with excellent results for ABBO. Up-to-date results from the periodically rerun experiments are available on Nevergrad's dashboard https://dl.fbaipublicfiles.com/nevergrad/ allxps/list.html, where ABBO is merged under the name NGOpt8.

AllDEs and Hdbo are benchmark collections specifically designed to compare DE variants (AllDEs) and high-dimensional Bayesian Optimization (Hdbo), respectively [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF]. These benchmark functions are Realworld: A test of ABBO is performed on the Realworld optimization benchmark suite proposed in [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF]. This suite includes testbeds from MLDA [START_REF] Gallagher | Exploratory landscape analysis of the mlda problem set[END_REF] and from [LMP + 20]. Results for this suite, presented in Figure 6, confirm that ABBO performs well also on benchmarks that were not explicitly used for its design -however, this benchmark was used for designing Shiwa, which was the basis of our ABBO. A rigorous cross-validation, on benchmarks totally independent from the design of Shiwa, is provided in the next sections.

New Benchmarks in OptimSuite Used Only for Evaluating ABBO

Pyomo is a modeling language in Python for optimization problems [HLW + 17]. It is popular and has been adopted in formulating large models for complex and real-world systems, including energy systems and network resource systems. We implemented an interface to Pyomo for Nevergrad. Experimental results are shown in Figure 2. They show that ABBO also performs decently in discrete settings and in constrained cases.

Additional new artificial and real-world functions: LSGO (large scale global optimization) combines various functions into an aggregated difficult testbed including composite highly multimodal functions. Correctly decomposing the problem is essential. Various implementations of LSGO exist; in particular we believe that some of them do not match exactly. Our implementation follows [LTO + 13] , which introduces functions with subcomponents (i.e., groups of decision variables) having non-uniform sizes and non-uniform, even conflicting, contributions to the objective function. Furthermore, we present here experimental results on SequentialFastgames from the Nevergrad benchmarks, and three newly introduced benchmarks, namely Rocket, SimpleTSP (a set of traveling salesman problems), power systems (unit commitment problems [START_REF] Prasad | Unit commitment-a bibliographical survey[END_REF]). Experimental results are presented in Figures 2, 7, and8. They show that ABBO performs well on new benchmarks, never used for its design nor for that of the low-level heuristics used inside ABBO.

MuJoCo. Many articles [SK20, WFT20

] studied the MuJoCo testbeds [START_REF] Todorov | Mujoco: A physics engine for model-based control[END_REF] in the black-box setting. MuJoCo tasks correspond to control problems. Defined in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF][START_REF] Mania | Simple random search provides a competitive approach to reinforcement learning[END_REF], the objective is to learn a linear mapping from states to actions. It turned out that the scaling is critical [START_REF] Mania | Simple random search provides a competitive approach to reinforcement learning[END_REF]: for reasons mentioned Figure 3: Results on the MuJoCo testbeds. Dashed lines show the standard deviation. Compared to the state of the art in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF], with an algorithm adapted manually for the different tasks, we get overall better results for Humanoid, Ant, Walker. We get worse results for Swimmer (could match if we had modified our code for the 3 easier tasks as done in [WFT20]), similar for Hopper and Cheetah: we reach the target for 5 of the 6 problems (see text). Runs of Shiwa correspond to the improvement of Shiwa due to chaining, as explained in Table 2.

in Section 2, solutions are close to 0. We chose to scale all the variables of the problem at the power of 0.1 the closest to 1.2/d, for all methods run in Figure 3. We remark that ABBO and Shiwa perform well, including comparatively to gradient-based methods in some cases, while having the ability to work when the gradient is not available. When the gradient is available, black-box methods do not require computation of the gradient, which saves time.

We use the same experimental setup as [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF] (linear policy, offline whitening of states). We get results better than LA-MCTS, in a setting i.e., does not use any expensive surrogate model (Table 3). Our runs with CMA-ES and Shiwa are better than those in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF]. We acknowledge that LMRS [START_REF] Sener | Learning to guide random search[END_REF] outperforms our method on all MuJoCo tasks, using a deep network as a surrogate model: however, we point out that a part of their code is not open sourced, making the experiments not reproducible. In addition, when rerunning their repository without the non open sourced part, it solved Half-Cheetah within budget 56k, which is larger Table 3: Results for a linear policy in the black-box setting from the latest black-box paper [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF] and references therein, compared to results from ABBO. Two last columns = average reward for the maximum budget tested in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF], namely 1k, 4k, 4k, 40k, 30k, 40k, respectively. "ioa" = iterations on average for reaching the target. "iter" = iterations for target reached for median run. "*" refers to problems for which the target was not reached by [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF]: then BR means "best result in 10 runs". ABBO reaches the target for Humanoid and Ant whereas previous (black-box) papers did not; we get nearly the same ioa for Hopper and HalfCheetah (Nevergrad computed the expected value instead of computing the ioa, so we cannot compare exactly; see Figure 3 for curves). ABBO is slower than LA-MCTS on Swimmer. Note that we keep the same method for all benchmarks whereas LA-MCTS modified the algorithm for 3 rows. On HDMULTIMODAL, ABBO performs better than LA-MCTS, as detailed in the text, and as confirmed in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF], which acknowledges the poor results of LA-MCTS for high-dimensional Ackley and Rosenbrock. ours. For Humanoid, the target was reached at 768k, which is again larger than our budget. Results from ABBO are comparable to, and usually better than (for the 3 hardest problems), results from LA-MCTS, while ABBO is entirely reproducible. In addition, it runs the same method for all benchmarks and it is not optimized for each task specifically as in [START_REF] Sener | Learning to guide random search[END_REF][START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF]. In contrast to ABBO, [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF] uses different underlying regression methods and sampling methods depending on the MuJoCo task, and it is not run on other benchmarks except for some of the HDMULTIMODAL ones. On the latter, ABBO performances are significantly better for Ackley and Rosenbrock in dimension 100 (expected results around 100 and 10 -8 after 10k iterations for Rosenbrock and Ackley respectively for ABBO, vs 0.5 and 500 in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF]). From the curves in [START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF] and in the present work, we expect LA-MCTS to perform well with an adapted choice of parametrization and with a low budget, for tasks related to MuJoCo, whereas ABBO is adapted for wide ranges of tasks and budgets.

Conclusions

This paper proposes OptimSuite, a very broad benchmark suite composed of real-world and of artificial benchmark problems. OptimSuite is implemented as a fork of Nevergrad [START_REF] Rapin | Nevergrad -A gradient-free optimization platform[END_REF], from which it inherits a strong reproducibility: our (Python) code is open source, tests are rerun periodically, and up-to-date results are available in the public dashboard [START_REF]Dashboard of results for Nevergrad platform[END_REF]. A whole experiment can be done as a one-liner. OptimSuite fixes several issues of existing benchmarking environments. The suite subsumes MuJoCo, Pyomo, LSGO, YABBOB, MLDA, and several new real-world problems. We also propose ABBO, an improved client-server interaction [START_REF] Loshchilov | Black box optimization competition[END_REF]. A possible combination between our platform and such a challenge, using the dashboard to publish the results, could be useful, to offer a meaningful way for cross-validation. Further improving ABBO is on the roadmap. In particular, we are experimenting with the automation of the still hand-crafted selection rules. Note, though, that it is important to us to maintain a high level of interpretability, which we consider key for a wide acceptance of the wizard. Another avenue for future work is a proper configuration of the low-level heuristics subsumed by ABBO. At present, some of them are merely textbook implementations, and significant room for improvement can therefore be expected. Newer variants [Los14, AH16, LGB18] of CMA-ES, of LMRS [START_REF] Sener | Learning to guide random search[END_REF], recent Bayesian optimization libraries (e.g. [EPG + 19]), as well as per-instance algorithm configuration such as [START_REF] Belkhir | Per instance algorithm configuration of cma-es with limited budget[END_REF] are not unlikely to result in important improvements for various benchmarks. We also plan on extending OptimSuite further, both through interfacing existing benchmark collections/problems, and by designing new benchmark problems ourselves.

A APPENDIX

We specify the properties mentioned in Table 1.

• Large scale: includes dimension ≥ 1 000.

• Translations: in unbounded continuous domains, a standard deviation σ has to be provided, for example for sampling the first and second iterates of the optimization algorithm. Given a standard deviation σ, we consider that there is translation when optima are randomly translated by a N (0, σ 2 ) shift. Only interesting for artificial cases.

• Far-optimum: optima are translated far from the optimum, with standard deviation at least N (0, 25× σ 2 ).

• Symmetrizations / rotations (here assuming an optimum, up to translation, in 0). Rotation: with a random rotation matrix M , the function x → f (x) is replaced by x → f (M (x)). Symmetrization:

x → f (x) can be replaced by x → f (S(x)), with S a diagonal matrix with each diagonal coefficient equal to 1 or -1 with probability 50%. We do not request all benchmarks to be rotated: it might be preferable to have both cases considered.

• One-line reproducibility: Where reproducibility requires significant coding, it is unlikely to be of great use outside of a very small set of specialists. One-line reproducibility is given when the effort to reproduce an entire experiment does not require more than the execution of a single line. We consider this to be an important feature.

• Periodic automated dashboard: are algorithms re-run periodically on new problem instances? Some platforms do not collect the algorithms, and reproducibility is hence not given. An automated dashboard is convenient also because new problems can be added "on the go" without causing problems, as all algorithms will be executed on all these new problem instances. This feature addresses what we consider to be one of the biggest bottlenecks in the current benchmarking environments.

• Complex or real-world: Real-world is self-explanatory; complex means a benchmark involving a complex simulator, even if it is not real world. MuJoCo is in the "complex" category.

• Multimodal: whether the suite contains problems for which there are local optima which are not global optima.

• Open sourced / no license: Are algorithms and benchmarks available under an open source agreement? BBOB does not collect algorithms, MuJoCo requires a license, LSGO and BBOB are not realworld, Mujoco requires a license, BBComp is no longer maintained, Nevergrad does not include complex ML problems without license issue before our work: some people have already applied Nevergrad to MuJoCo, but with our work MuJoCo becomes part of Nevergrad so that people can upload their code in Nevergrad and it will be run on all benchmarks, including MuJoCo.

• Ask/tell/recommend correctly implemented [CHP + 10, BMS11]: The ask and tell idea (developped in [CHP + 10]) is that an optimization algorithm should not come under the format Optimizer.minimize(objective -f unction) because there are many settings in which this is not possible: you might think of agents optimizing concurrently their own part of an objective function, and problems of reentrance, or asynchronicity. All settings can be recovered from an ask/tell optimization method. This becomes widely used. However, as well known in the bandit literature (you can think of pure exploration bandits [START_REF] Bubeck | Pure exploration in finitely-armed and continuous-armed bandits[END_REF]), it is necessary to distinguish ask, tell and recommend: the "recommend" method is the one which proposes an approximation of the optimum. Let us develop an example explaining why this matters: the domain is {1, 2, 3, 4}, and we have a budget of 20 in a noisy case. NoisyBBOB assumes that the optimum is found when "ask" returns the optimum arm: then, the status remains "found" even if the algorithm has no idea where is the optimum and never comes back nearby. So an algorithm which just iteratively "asks" 1, 2, 3, 4, 1, 2, 3, 4, . . . reaches the optimum in at most 4 iterations. This does not mean anything in the noisy case, as the challenge is to figure out which of the four numbers is the optimum. With a proper ask/tell/recommend, the optimizer chooses an arm at the end of the budget. A simple regret is then computed. Actually this also matters in the noise-free case, but the issue is much more critical in noisy optimization. The case of continuous noisy optimization also has counter-examples and all the best noisy optimization algorithms use ask/tell/recommend. We add the reference to the paper above.

• Human excluded / client-server: The problem instances are truly black-box. Algorithms can only suggest points and observe function values, but neither the algorithm nor its designer have access to any other information about the problem apart from the number of variables, their type, ranges, and order. It is impossible to repeat experiments for tuning hyperparameters without "paying" the budget of the HP tuning. This is something we could not do, as everything is public and open sourced: however, we believe that we mitigate this issue by considering a large number of benchmarks.

  only Noisy optimization with categorical variables Genetic algorithm mixed with bandits [HMI09, LMP + 20]. alphabets of size < 5, sequential evaluations (1 + 1)-Evolutionary Alg. with linearly decreasing stepsize alphabets of size < 5, parallel case Adaptive (1 + 1)-Evolutionary Alg. [DDL19]. Other discrete cases with finite alphabets Convert to the continuous case using SoftMax as in [LMP + 20] and apply CMandAS2 [RDP + 19] Presence of infinite discrete domains FastGA [DLMN17] Numerical decision variables only, evaluations are subject to noise d > 100 progressive optimization as in [Ber16]. d ≤ 30 TBPSA [HB16] b > 100 sequential quadratic programming Other cases TBPSA [HB16] Numerical decision variables only, high degree of parallelism Parallelism > b/2 or b < d MetaTuneRecentering [MDRT20] Parallelism > b/5, d < 5, and b < 100 DiagonalCMA-ES [RH08] Parallelism > b/5, d < 5, and b < 500 Chaining of DiagonalCMA-ES (100 asks), then CMA-ES+meta-model [AST05] Parallelism > b/5, other cases NaiveTBPSA as in [CT20] Numerical decision variables only, sequential evaluations b > 6 000 and d > 7 Chaining of CMA-ES and Powell, half budget each. b < 30d and d > 30 (1 + 1)-Evol. Strategy w/ 1/5-th rule [Rec73] d < 5 and b < 30d CMA-ES + meta-model [AST05] b < 30d

Figure 2 :

 2 Figure 2: Additional problems: Pyomo (covering Knapsack, P-median and others), SequentialFastgames (presented as heatmaps due to the high noise: GuessWho, War, Batawaf, Flip). Rockets, SimpleTSP, PowerSystems, and LSGO plots are available in Figures 7, and 8. Pyomo and SimpleTSP include discrete variables. Pyomo includes constraints. Rocket, PowerSystems, SequentialFastGames are based on open source simulators and are already merged from OptimSuite to Nevergrad.

Figure 4 :

 4 Figure 4: YAHDBBOB (dimension ≥ 50) and YANOISYHDBBOB (noisy + dimension ≥ 50) heatmaps.

Figure 5 :

 5 Figure 5: ABBO vs specific families of optimization algorithms (DE, and BO in the high-dimensional case) on Cigar, Hm, Ellipsoid, Sphere functions. Not all run algorithms are mentioned, for short. Bayesian optimization (Nevergrad uses [Nog ]), often exploring boundaries first, is outperformed in high dimension [WFT20].

Figure 6 :

 6 Figure 6: Up: experiments for the parallel multimodal setting PARAMULTIMODAL. Budget up to 100 000, parallelism 1 000, Ackley+Rosenbrock+DeceptiveMultimodal+Griewank+Lunacek+Hm. Bottom: Realworld benchmark from Nevergrad: games, Sammon mappings, clustering, small traveling salesman instance, small power systems.

Figure 7 :

 7 Figure 7: Additional problems (1): on left, Rocket (26 continuous variables, budget up to 1600, sequential or parallelism 30) and on right, SimpleTSP (10 to 1 000 decision variables).

  Algorithm selection was applied to continuous black-box optimization and pushed in Nevergrad [LMP + 20]: their optimization algorithm combines many optimization methods and outperforms each of them when averaged over diverse test functions. Closer to machine learning, efficient global optimization[START_REF] Jones | Efficient global optimization of expensive black-box functions[END_REF] is widely used, although it suffers from the curse of dimensionality more than other methods[START_REF] Snoek | Practical bayesian optimization of machine learning algorithms[END_REF];[START_REF] Wang | Learning search space partition for blackbox optimization using Monte Carlo Tree Search[END_REF] presented a state-of-the-art algorithm in black-box optimization on MuJoCo, i.e., for the control of various realistic robots[START_REF] Todorov | Mujoco: A physics engine for model-based control[END_REF]. We propose ABBO, which extends [LMP + 20] by the following features:

	Benchmark	Use for ABBO # of configs		ranking		ABBO best competitor
				ABBO Shiwa CMA-ES
	HDBO	Designing	24	2/21	1 †	2	Shiwa
	PARAMULTIMODAL	Designing	112	1/27	3 †	6	DiagonalCMA-ES [RH08]
	Realworld	Designing	486	1/6	2 †	3	Shiwa [LMP + 20]
	Illcondi	Designing	12	1/24 3 †	8	Cobyla
	Illcondipara	Designing	12	5/28	7 †	3	DiagonalCMA-ES
	YABBOB	Designing*	630	1/8	2	5	Shiwa
	YAPARABBOB	Designing*	630	1/6	4	5	MetaModel
	YAHDBBOB	Designing*	378	2/19	3	18	(1 + 1)-ES
	YANOISYBBOB	Designing*	630	2/11	6	10	SQP
	YAHDNOISYBBOB	Designing*	630	4/24	2	13	SQP
	YASMALLBBOB	Designing*	378	1/8	2	7	Shiwa
	HdMultimodal	Validation	42	1/14	2 †	4	Shiwa
	Noisy	Validation	96	16/28 19 †	NA	RecombiningOptimisticNoisyDiscrete(1 + 1)
	RankNoisy	Validation	72	4/25 NA	19	ProgD13
	AllDEs	Validation	60	1/28	2 †	3	Shiwa [LMP + 20]
	Pyomo	Evaluating	104	1/19 3 †	10	Shiwa [LMP + 20]
	Rocket	Evaluating	13	5/18	4 †	3	DiagonalCMA-ES [RH08]
	SimpleTSP	Evaluating	52	3/15	2 †	7	PortfolioDiscrete(1 + 1)
	Seq. Fastgames	Evaluating	20	3/28	4 †	23	OptimisticDiscrete(1 + 1)
	LSGO	Evaluating	45	1/6	4 †	6	MiniLHSDE
	Powersystems	Evaluating	48	10/26 NA	25	(1 + 1)-ES
	but they have won competitions [Art15] and significantly improved evolution strategies through memetic
	methods [RS94].

Acknowledgments

This work was supported by the Paris Ile-de-France Region.