Henry Mackeown

Junias Adusei Gyamfi

Morgan Delaporte

Vital Kaat

Marnix Schoutteten

Liesbeth Verdickt

Baghdad Ouddane

Justine Criquet
email: justine.criquet@univ-lille.fr

Removal of disinfection by-product precursors by ion exchange resins

Keywords: chlorination, bromide, iodide, DBP, NOM

The removal of dissolved organic carbon (DOC) is a key factor in the control of disinfection by-product (DBP) formation in drinking water. Ion exchange process deals with the removal of naturally negatively charged organic matter. A fluidized bed column test has been chosen to test in parallel the efficiency of 4 different anionic exchange resins (Purolite PPA860S, Dowex TAN-1, Amberlite IRA-958 and IRA-410) in terms of DOC fraction removal and related DBP formation reduction. IRA-410 was shown to be the best performing resin in terms of DOC and DBP formation potential reduction followed by PPA860S. These resins removed respectively 41 and 37% of DOC, with humic substances as the main target of the two resins (68 and 72% reduction, respectively, based on size exclusion chromatography with an organic carbon detector). The UV absorbance was reduced in a higher proportion than the DOC demonstrating a preference of the resins for the relatively hydrophobic compounds. The reduction in trihalomethane and haloacetic acid formation was higher than the DOC removal for IRA-410 and PPA860S (53 and 59%, respectively); whereas IRA-958 on the other hand showed a lower reduction in DBP formation potential than its DOC removal. All the resins showed a much higher reduction of iodinated trihalomethanes formation potential (66 to 96% reduction compared to raw water) due to the removals of iodide in addition of organic precursors. However, in a context of increasing halide concentrations, the different resins showed a reduction of their efficiency to control the formation of DBPs.

Introduction

The most commonly applied methods for natural organic matter (NOM) removal from water are coagulation, 1 adsorption 2 and membrane filtration [START_REF] Sillanpää | Natural Organic Matter in Water[END_REF] . Ion exchange processes (IEX) -where negatively charged NOM fractions are removed by reversible exchange of a counter ion, usually chloride, from the anionic resin surface [START_REF] Bolto | Ion exchange for the removal of natural organic matter[END_REF] -were introduced more recently, with magnetic ion exchange resin (MIEX) early in the 2000s as first industrial application in drinking water. [START_REF] Levchuk | Removal of natural organic matter (NOM) from water by ion exchange -A review[END_REF] Since then, different types of resins are regularly developed and tested. Different processes such as suspended ion exchange (SIX) [START_REF] Galjaard | SIX®: A New Resin Treatment Technology for Drinking Water[END_REF] and fluidized ion exchange (FIX) [START_REF] Cornelissen | Fluidized ion exchange (FIX) to control NOM fouling in ultrafiltration[END_REF] have been explored as well. Ion exchange is of particular interest as a pretreatment before coagulation as it has been shown to significantly reduce the coagulant dose required (by 50 to more than 80%), [START_REF] Singer | Enhanced coagulation using a magnetic ion exchange resin[END_REF][START_REF] Fearing | NOM control options: the next generation[END_REF][START_REF] Grefte | Natural organic matter removal by ion exchange at different positions in the drinking water treatment lane[END_REF] as well as increase the floc size and strength. [START_REF] Jarvis | Pilot Scale Comparison of Enhanced Coagulation with Magnetic Resin Plus Coagulation Systems[END_REF] IEX is also more efficient than coagulation for the removal of charged organic compounds. [START_REF] Bolto | Removal of natural organic matter by ion exchange[END_REF] Using IEX in combination with coagulation, can thus remove both a higher amount of dissolved organic carbon (DOC), and a broader spectrum of DOC as hydrophilic and low molecular weight fractions (LMW) fractions are more difficult to remove using conventional coagulation treatment. [START_REF] Allpike | Size Exclusion Chromatography To Characterize DOC Removal in Drinking Water Treatment[END_REF][START_REF] Mergen | Characterisation of natural organic matter (NOM) removed by magnetic ion exchange resin (MIEX® Resin)[END_REF] The higher DOC removal is often accompanied with an enhanced reduction in disinfection by-product (DBP) formation potential. [START_REF] Singer | Enhanced coagulation using a magnetic ion exchange resin[END_REF]15 Furthermore, anionic resins can also remove bromide from certain water sources with low alkalinity 15 , thereby potentially reducing the formation of brominated DBPs during disinfection. The removal efficiency of NOM by IEX resins was mainly studied through measuring colour and NOM characteristics such as DOC, TOC, UV absorbance at 254 nm [START_REF] Bolto | Removal of natural organic matter by ion exchange[END_REF][START_REF] Croué | Removal of hydrophobic and hydrophilic constituents by anion exchange resin[END_REF][START_REF] Humbert | Performance of selected anion exchange resins for the treatment of a high DOC content surface water[END_REF] and more recently NOM fraction removal. [START_REF] Grefte | Natural organic matter removal by ion exchange at different positions in the drinking water treatment lane[END_REF] While these studies offer useful guidance in understanding the removal of NOM fractions, less has been done on DBP formation potential, with studies focusing on THMFP and HAAFP generally on one single resin type. [START_REF] Tan | Dissolved organic matter removal and disinfection byproduct formation control using ion exchange[END_REF][START_REF] Boyer | Bench-scale testing of a magnetic ion exchange resin for removal of disinfection by-product precursors[END_REF][START_REF] Finkbeiner | Understanding the potential for selective natural organic matter removal by ion exchange[END_REF][START_REF] Brezinski | Ion-exchange for trihalomethane control in potable water treatment -a municipal water treatment case study in Rainy River, Ontario, Canada[END_REF] In this study, the removal of DBP precursors from raw surface water used for drinking water production was investigated on four different ion exchange resins in a fluidized bed column test. The 4 different resins have been compared in term of NOM fraction removal efficiency. These results have been related to the trihalomethane and haloacetic acid formation, including iodinated disinfection byproducts. The interest of the IEX process in different scenarios of increasing halide concentrations has been studied in order to evaluate such processes on raw water impacted by high levels of halides and to gain more knowledge on DBP precursors.

Materiel and Methods

Water resource and pilot unit

The raw water is drawn from the river IJzer (Yser) (approximately 80%) downstream of the town of Ieper and from surrounding lowlands (approximately 20%), and stored in a 3 million m³ reservoir. [START_REF] Watergroep | Annual report[END_REF] The water is characterized by a high NOM content, with an average DOC content of 12 mgC L -1 , combined with a high alkalinity and hardness. The Belgian part of the hydrographic basin of the IJzer river covers an area holding 110,000 inhabitants, and a largely agrarian activity, i.e. intensive farming and intensive cattle breeding. [START_REF] Baert | Proceedings _ 4th International Conference On Environmental Compliance And Enforcement[END_REF] Due to the combination of high DOC concentration and relatively high alkalinity, the water requires large amounts of coagulant and sulfuric acid during coagulation. [START_REF] Verdickt | Applicability of ion exchange for NOM removal from a sulfate-rich surface water incorporating full reuse of the brine[END_REF] The four resins tested were Purolite PPA860S, Dowex TAN-1, Amberlite IRA-958 and Amberlite IRA-410. The three first mentioned resins are strong base anion type I resins with a quaternary ammonium group, and the last one is a strong base type II resin where a methyl group is replaced with an alcohol group. Their main properties are presented in Table S1. A column experiment was performed in order to better translate the results to a full scale plant. Four parallel transparent PVC columns, with an internal diameter of 86.4 mm were used for performing breakthrough experiments. The resins were pre-treated extensively to avoid possible leaching of residual monomers and to avoid irreversible adsorption of NOM occurring on virgin resins. This procedure consisted of a jar pre-treatment and a dummy loading run in the columns. For the jar pre-treatment, the following regeneration procedure was conducted three times consecutively: regeneration using 3 bed volumes (BV) of a 10% NaCl brine followed by decantation, and a triple rinsing with 3 BV of tap water followed by decantation. After the jar pretreatment procedure, the resins were transferred inside the columns, and a dummy loading run was performed using 500 BV of filtered raw water at an empty bed contact time (EBCT) of 1.73 min, followed by a regeneration with 1 BV of 10% NaCl at an EBCT of 30 minutes and a rinse with 1 BV of tap water at an EBCT of 30 minutes. The latter regeneration procedure has been developed from the long practice on a IEX industrial scale pilot. [START_REF] Verdickt | Applicability of ion exchange for NOM removal from a sulfate-rich surface water incorporating full reuse of the brine[END_REF] After this extensive pretreatment procedure, the resins were considered to be suitable to perform a representative IEX cycle. The breakthrough experiments were run in parallel for the 4 resins in a fluidized bed modus (i.e. bottom -up, Fig. S1) using raw water from the reservoir which had been prefiltered over a 25 µm bag filter (Filtration CVBA, Drongen, Belgium), at an EBCT of 1.73 min.

Sample collection and analysis prior to chlorination

Samples were taken at 0 BV and every recurring 50 BV, until 500 BV of raw water were treated. To simulate the average treated water quality which would be obtained after 500 BV, a mixed effluent sample was created by mixing equal volumes (208 mL) of each of the 11 taken samples for each resin, for the DBP experiments and LC-OCD measurement. The samples were characterized by size exclusion chromatography (SEC) coupled to organic carbon detection (LC-OCD) at the certified laboratory "Het Waterlaboratorium (HWL)" (Haarlem, Noord-Holland) according to the LC-OCD procedure described in literature. [START_REF] Huber | Characterisation of aquatic humic and non-humic matter with size-exclusion chromatography -organic carbon detection -organic nitrogen detection (LC-OCD-OND)[END_REF] The raw water and mixed effluent samples for all four resins were filtered through 0.45 µm polyethersulfone (PES) membrane filters within a few hours after sampling. All samples were stored at 4 °C until the chlorination experiments or analysis. Iodide, iodate and bromide were measured using a liquid chromatograph coupled to an inductively coupled plasma mass spectrometer (HPLC-ICP-MS -Agilent 7900). The protocol has been detailed in a previous study. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] The quantification limit was 0.2 µg L -1 for all three species. Total iodine and bromine in the water samples were determined by ICP-MS (Varian 820 MS). The quantification limit level was 0.5 µg L -1 for both iodine and bromine.

Chlorination tests

Two different trihalomethane formation potentials have been performed (THM7d and THM24h). The THM7d tests were carried out using a 7-day chlorination procedure according to the Standard Methods 5710 [START_REF]Standard Methods For the Examination of Water and Wastewater[END_REF] with minor modifications. The samples were placed in either 25 or 50 mL glass reagent bottles with PTFE lined screw caps, buffered with a phosphate solution at pH 7.0 ± 0.2 and chlorinated using a stock solution of 5 g L -1 sodium hypochlorite at different doses in order to ensure samples with free chlorine residuals between 3 and 5 mg L -1 at the end of the chlorination test. Samples were incubated in a thermostatically controlled cabinet at 25 ± 2 °C for 7 days, after which the free chlorine residual of each sample was measured using the N, N-diethyl-p-phenylenediamine (DPD) colorimetric method. [START_REF] Rodier | [END_REF] In the samples containing a free chlorine residual between 3 and 5 mg L -1 , the chlorine residual was quenched using sodium thiosulfate and the samples were stored at 4 °C in the dark until the THMs were analysed. The 7-day test is the classical procedure to account for the total possible formation of DBPs, however, the high chlorine dose of this test is not suitable for the assessment of iodinated DBPs.

To avoid this bias, chlorination experiments have also been performed with a lower chlorine dose, i.e. 0.3 mg Cl2 L -1 residual after a contact time of 24 h and at a temperature of 20°C to assess the evolution of THMs and HAAs after IEX treatment. DBP formation potential tests have been performed on the IEX treated waters either without modifications or with bromide and iodide concentrations equalised in order to compare the efficiency of the different resins in the same halide concentration conditions. In addition, the concentration of halide has been varied in the IEX treated waters. The concentrations of bromide was varied from 2.5 to 27.5 μM (228 to 2200 µg L -1 ) while the iodide concentration was kept constant (0.5 μM i.e. 63 µg L -1 ). This led to molar bromide/iodide ratios ranging from 5.7 to 55, comparable to a previous study [START_REF] Allard | Mechanistic Study on the Formation of Cl-/Br-/I-Trihalomethanes during Chlorination/Chloramination Combined with a Theoretical Cytotoxicity Evaluation[END_REF] (Table S2). In a second set of experiments, the concentration of iodide was varied from 0.06 to 2 µM (7 to 254 µg L -1 ) with a constant bromide concentration (228 µg L -1 i.e. the highest residual bromide concentration among the IEX treated waters).

Disinfection by-product analysis

Our previous study developed high sensitivity headspace-trap methods for the analysis of the 10 THMs and 7 HAAs. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] The same methods were used in this study on a TraceTM 1300 GC ISQ-LT single quadrupole mass spectrometer (Thermo Scientific®) coupled to a TurboMatrix Headspace 40 Trap (Perkin Elmer®) and are briefly described. For THMs analysis, 10 mL of the water samples were transferred into 22 mL glass vials containing 4 g of sodium sulfate (2.8 M), and were spiked with 1,2dibromopropane as internal standard (6.4 µg L -1 ). The vials were immediately sealed and stirred in a vortex mixer until complete salt dissolution. Haloacetic acids (HAA5 + iodoacetic and bromochloroacetic acids) were analysed by a one-step procedure. HAAs in aqueous medium form ionpairs with tetrabutylammonium hydrogensulfate and react with dimethylsulfate to produce methyl haloacetates which are directly volatized in the headspace. 9.84 mL of the water samples (without pH adjustment) were spiked with bromopropionic acid (10 µg L -1 ) as internal standard and placed in a 22 mL glass vial containing 4 g of sodium sulfate. 100 µL of a 0.5 M ion pairing agent (tetrabutylammonium hydrogensulfate, final amount of 2.3 µM) and 60 µL of derivatization reagent (dimethylsulfate) were then added. The vials were immediately sealed and stirred before being placed into an autosampler.

The analysis was performed after 2 to 4 hours of reaction time, this time should not exceed 5 hours to avoid the hydrolysis of the methylesters formed.

Results and discussion

Effect of ion exchange treatment on NOM and inorganic compounds

Raw water composition

The DOC of the raw water was relatively high, around 12 mgC L -1 (Fig. 1A), but at a typical level for this site. The SUVA was moderate, around 2 indicating the presence of hydrophilic and/or low molecular weight compounds. LC-OCD fractionation indicated that humic substances (HS) dominated (40%), followed by biopolymers, building blocks (BB) and low molecular weight neutrals (LMWN), these 3 fractions each representing approximately 14% of the DOC (Fig. 1A). No low molecular weight acids (LMWA) were found in the feed water by LC-OCD. Additionally, around 15% of the DOC is not eluted through the column presumably due to hydrophobic interactions; this represents the so-called hydrophobic organic carbon fraction (HOC). The mineralization of the water is relatively high, mainly with bicarbonate (247 mg L -1 ) but also with quite a large amount of chloride (147 mg L -1 ) and sulfate (104 mg L -1 ); the cations are dominated by sodium (105 mg L -1 ) and calcium (72 mg L -1 ) (Table S3).

IEX treated waters

The treatment of raw water by ion exchange removed from 23 up to 41% of the dissolved organic carbon content based on DOC measurement depending on the type of resins (Fig. 1a). The best removal is obtained with the type II strong base resin IRA-410, a styrene divinyl benzene type with a dimethyl ethanol ammonium functional group. This resin has the best total exchange capacity (≥ 1.25 eq L -1 ) according to the manufacturer information (Table S1). This parameter however does not explain the difference of removals obtained for the 3 other resins, which are all type I resins with a quaternary ammonium group (from 0.7 to ≥ 0.8 eq L -1 ). The DOC removal by such resins is usually in the 40-70% range. In some cases, lower removal rates due to competition with sulfates were observed (Table S4).

In this water, sulfate is effectively competing for the ion exchange (see details below), in addition, the low SUVA NOM could also be a poor target of exchange sites. Considering the removal of the different fractions of NOM, the humic substances fraction is the main one removed by the different IEX resins, both in terms of DOC (from 41 to 72%) and UV254 absorbing fraction (58-91%), followed by the building block fraction (12-41% for DOC and 35-79% for UV254) and LMWN fraction for DOC removal (13-36% while the UV254 LMWN showed no removal.) (Fig. 1A, Table S5). After IEX treatment, the concentration of the biopolymer fraction was found to slightly increase by 6-32% in terms of DOC, which may have been caused by the release of polymeric IEX material [START_REF] Cornelissen | Selection of anionic exchange resins for removal of natural organic matter (NOM) fractions[END_REF] but also in some extent by a different deconvolution due to the lowering of the following HS peak. Overall, the SUVA was lowered by the IEX treatment system by 32-62%, which is higher than the DOC removal, indicating preferential removal of higher-SUVA components. The final SUVA values (0.75-1.34) indicate that the remaining NOM is mainly hydrophilic. These results are in agreement with previous studies (Table S4) that show effective removal of high charge density organic fractions such as HS and less effective removal of BB, LMWN, and no removal of biopolymers due to size exclusion, although one study did find PPA860S to efficiently remove most major NOM fractions, including biopolymers. 31 The most effective resins for the removal of HS was Purolite PPA860S (72% removal) and Amberlite IRA-410 (68%). The later exhibits the most effective removal of BB and LMWN fractions (41% and 36% removal respectively), which then explain its overall best performance (Table S5). The molecular weight of HS in waters treated by 3 resins (IRA 958, TAN-1 and PPA860S) decreased by 15-22% with the lowest MW for PPA860S (centre on 400 g mol -1 ), indicating a preference for the removal of the highest molecular weights HS. In fact, the macroporous acrylic structure of PPA 860S is generally better for the removal of large anions. [START_REF] Bolto | Ion exchange for the removal of natural organic matter[END_REF][START_REF] Bolto | Removal of natural organic matter by ion exchange[END_REF] On the opposite, the molecular weight of HS after treatment by the IRA-410 did not change. Removal of HS and BB fractions by the anionic resins can mainly be explained by ionic interactions while the removal of the LMWN could possibly be explained by physical adsorption onto the resin material. [START_REF] Cornelissen | Selection of anionic exchange resins for removal of natural organic matter (NOM) fractions[END_REF] The removal of this later fraction is higher for IRA-410 and TAN-1 resins (36 and 18% respectively) compared to the others (13%). This should be related to the resin types as styrenic structure based resins (structures of TAN-1 and IRA-410) exhibiting more hydrophobic interaction than the acrylic resins (structures of IRA 958 and PPA860S, Table S1).

The cation removal for all the resins was low or inexistent as these resins are designed for the exchange of anions. Considering iron, while its concentration is quite low in the raw water (15 µg L -1 ), the concentrations after treatment were systematically below that level (by 13-33%); its retention possibly happens by complexation with humic substances. 32,[START_REF] Adusei-Gyamfi | Natural organic mattercations complexation and its impact on water treatment: A critical review[END_REF] A significant increase in Cl -concentrations after ion exchange treatment is obviously observed for all resins (Table S3). However, a large part of this release is attributable to the ion exchange with sulfate (from 44 up to 89% of the initial 104 mg L -1 of sulfate; Table S3). Indeed, sulfate ions are smaller than NOM molecules, hence the ion exchange process is kinetically faster for this ion. 32 A nearly 50 % reduction in DOC removal when increasing the sulfate level to a concentration of 100 mg L -1 for Purolite A860 34 (PPA860S corresponds to a sieved fraction of A860). Then, a major part of the practical exchange capacity of the resins goes to the removal of sulfate (0.96-1.94 meq L -1 , which represents 64-77% of the actual ion exchange), and another small part to bicarbonate (0.08-0.36 meq L -1 ). The potential release of chloride from incomplete washing of the resin is taken into account considering an equivalent increase of sodium concentrations (0.28-3.47 meq L -1 ). Finally, only 10 to 19% (0.25-0.32 meq L -1 ) of the total amount of chloride exchanged could be attributed to NOM removal for 3 of the 4 resins. For IRA-958, the calculated amount of chloride exchanged for NOM was negative (-0.05 meq L - 1 ) and close to zero, likely due to the uncertainties on the analytical measurements of the ion concentrations. Finally, the concentration of chloride in the treated waters (from 264 to 305 mg L -1 ) exceeds the European quality guideline of 250 mg L -1 for three of the four resins in the conditions used here. However, in industrial processes, extensive rinses would have to be performed to avoid unnecessary chloride release from salts trapped in micropores that only slowly diffuse when the resin swells in a lower ionic strength water compared to the saline regeneration solutions.

In contrast to what has been observed on MIEX resins, 15 bromide was not removed in our experiments and even increased by up to 17%, perhaps due to some bromide contamination during resin regeneration (bromide being a common contaminant of commercially available NaCl). The bromide anions may also have been outcompeted for binding sites by the other anions. 15 This is confirmed by the proportional increase between chloride and bromide concentrations observed for 3 of the 4 resins (the concentration is constant after IRA-410 treatment ; Fig. S2). However, iodide was significantly exchanged onto the resin (18-85%) for IRA-410, TAN-1 and PPA860S (Table S3), which has not been previously reported in the literature for these 3 resins. IRA-958 was not able to significantly remove iodide (3% reduction).

Removal of DBP precursors

THM7d, THM24h and HAA24h speciation

The formation potential of the THM4 after 7 days (THM7d) is around 550 µg L -1 in the raw water (Fig. S3), which is in the same range as noticed for previous experiments on the same site. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] CHCl3 is the main THM formed in all waters after the 7-day formation potential test. The reduction in THM7d compared to the raw water is in the range 30-60%. The order of THMFP reduction efficiency for the resins was: IRA-958 < TAN-1 ≈ PPA860S < IRA-410 (Fig. S3). Figure 1 shows the decrease in DBP formation potential after 24 hours (residual chlorine 0.3 mgCl2 L -1 ) through the treatment of water with the four IEX resins. The THM24h varies from around 80 to 150 µg L -1 in the different IEX treated waters compared to 175 µg L -1 in the raw water. CHBrCl2 is the major compound, due to the lower chlorine dose used leading to a higher bromide to chlorine ratio. Compared to the 7-day test, not only are the formation potentials 2.5-3.5 times lower after the 24-h test, but the reduction in THMFP compared to the raw water is also lower, by around 25% (mass or molar balance) (Fig. 1B). This observation should be then related to the preferential removal by IEX resins of DBP precursors which have a slow reactivity towards chlorination and THM formation. This could be related to the preferential removal of high molecular weight HS. The sum of 7 HAAs was 2 to 3 times lower than for THMs (i.e. 35 to 85 µg L -1 ; Fig. 1D). Compared to the THMs the reduction in HAAFP is higher, in the range 23-63% based on the 24hour HAA formation potential tests (Fig. 1D). DCAA, TCAA and BCAA are the main HAAs formed, representing 78 to 93% of the measured HAAs. The mono-HAAs are at much lower levels, as is usually the case in the literature. [START_REF] Bond | Examining the interrelationship between DOC, bromide and chlorine dose on DBP formation in drinking water -A case study[END_REF] The order of removal efficiency of THM24h and HAA24h precursors by the different resins was the same as for the DOC and THM7d with PPA860S and IRA-410 the best performing resins. The decrease in THM4 and HAA FPs followed the trends of DOC and SUVA reductions, a strong correlation (R 2 > 0.99) was observed between THM7d of the raw water and the 4 different treated waters and the DOC as well as UV254 and then SUVA (Fig. S4). These trends show a higher reduction of the THMFP compared to the DOC, thus the different IEX resins are more efficient at removing THM precursors than overall DOC. The THMFP/DOC ratio varies from 47 µg mgC -1 for the raw water to 32 to 42 µg mgC -1 for the treated waters. Considering UV absorption, its reduction is lower than THMFP (Fig. S4), demonstrating that a fraction of the precursors do not absorb UV, and come from more hydrophilic fractions.

According to the curve, if the UV absorbance was equal to zero, approximately 140 µg L -1 of THMs would still theoretically be formed. On the contrary, 3.4 mgC L -1 (about 30 %) of the DOC does not contribute to the THM formation (Fig. S5). Combining these results, a direct proportionality between THMFP and SUVA is observed (slope of 1 between THM7d/THM7d-raw vs. SUVA/SUVAraw) which underlines the relevance of this parameter in the IEX process. The formation potential of THMs and HAAs after the 24-h test also correlated with the DOC, UV and SUVA (R 2 from 0.85 to 0.93, Fig. S5), but the correlations were not as strong as for THM7d (Fig. S4).

I-THM24h

Iodinated DBPs are usually more cytotoxic and genotoxic than the brominated and chlorinated analogues. [START_REF] Richardson | Occurrence, genotoxicity, and carcinogenicity of regulated and emerging disinfection by-products in drinking water: A review and roadmap forresearch[END_REF] Their low odour threshold could also bring issues of medicinal taste and odours to the produced waters. [START_REF] Cancho | Determination of the odor threshold concentrations of iodinated trihalomethanes in drinking water[END_REF] From this raw water, the overall I-THM formation is relatively low (< 1 µg L -1 ) certainly due to the low level of iodide in the raw water (7.2 µg L -1 ) but also to its oxidation to iodate (see section 3.3.2). The major I-THMs formed are CHCl2I followed by CHBrClI and CHBr2I (Fig. 2B). These compounds correspond to the mono-iodinated compounds which are predominant for low iodide levels. [START_REF] Zhang | A comparison of iodinated trihalomethane formation from chlorine, chlorine dioxide and potassium permanganate oxidation processes[END_REF] In comparison with THM4, the reduction in I-THMFP compared to raw water is 2-10 times higher, in the range 76-96% and much higher than the DOC removal. In fact, the concentration of iodide is reduced after IEX treatment, with different efficiency depending on the resin type. IRA-410 which is the best performing resin for the reduction of I-THMFP reduces the concentration of iodide by more than 80% down to 1.1 µg L -1 . Indeed, the I-THM concentration can be correlated to the iodide concentration (R 2 = 0.96; Fig. S6) for all 4 treated waters. However, the formation of I-THMs in the raw water sample is not correlated to the treated waters with much higher I-THMs formed than the correlation with the 4 treated water would have predicted. The content of ammonia in the water (as chloramines favours I-DBP formation compared to chlorine [START_REF] Dong | Formation of Iodinated Disinfection Byproducts (I-DBPs) in Drinking Water: Emerging Concerns and Current Issues[END_REF] ) does not explain this behaviour either as the raw water shows the same concentration as the waters treated by the IRA-410 and IRA-958 resins (around 0.17 mg L -1 ) and a lower concentration than the waters treated by PPA860S and TAN-1 resins (around 0.27 mg L -1 ). Then, the removal of NOM DBP precursors is a key factor in the control of I-DBP formation. In fact, the IRA-958 resin treated water which shows a low iodide decrease compared to the raw water (<3%) still exhibits a high decrease of I-THMs (76%). This could be due to the removal of specific NOM precursors but also maybe more likely to the increase of iodate formation by favouring the reaction of chlorine (and/or bromine) with iodine reactive species (i.e. mainly HOI) compared to the reaction of chlorine (and/or bromine) with NOM. [START_REF] Criquet | Iodate and Iodo-Trihalomethane Formation during Chlorination of Iodide-Containing Waters: Role of Bromide[END_REF] Thus, all the resins by lowering iodide and/or by removing specific precursors show a beneficial effect, lowering the concentration of I-DBPs.

Comparison to the classical treatment line

Sampling campaigns have been performed on the full-scale treatment plant made of nitrification, coagulation-flocculation decantation, intermediate chlorination, sand filtration, activated carbon filtration and final chlorination steps. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] In the classical treatment line, the highest DOC removal occurs during the coagulation step which mainly removes the highest molecular weight fraction, i.e. biopolymers and humic substances. The treatment line removes around 70% of DOC for both sampling campaign, with 57 to 65% occurring during the coagulation step. The THMFP is reduced by 80% showing a regular decrease of the ratio THMFP/DOC due to a quite good removal of DBP precursors. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] The IEX treatment shows a limited efficiency for the DOC removal compared to the classical treatment.

Similarly, the THMFP/DOC ratio is reduced, linked to the decrease of SUVA after IEX treatment (Fig. 2). When looking to the THMFP in function of DOC or SUVA values (Fig. 2) of treated waters, two different regression lines are obtained with similar slopes with each of them achieving good coefficients of determination (R 2 from 0.9 to 0.99, Fig. 2). The existing shift between these two correlations is to be related to the presence of the biopolymer fraction in the raw and IEX treated waters for which the experiments were performed in autumn compared to the others performed in spring before the algal bloom. Then, the SUVA value of the whole water is reduced (2.0) while the SUVA value of the humic substance fraction is 3.5. In fact, this fraction represents 18 to 28% of the DOC for the IEX treated waters. This also explain why the THMFP/DOC ratios are rather low for the IEX treated waters (32 to

Figure 1 :

 1 Figure 1: Assessment of IEX process efficiency for the 4 different resins compared to raw water for dissolved organic carbon removal and disinfection by-product formation potential (24h -low chlorine residual). A) NOM fraction removal assessed by LC-OCD B) THM4, C) I-THMs and D) HAAs. Numbers above bars correspond to the reduction percentage compared to the concentration of each parameter in raw water.

Figure 2 :

 2 Figure 2: a)THMFP (7d) in function of DOC and b) THMFP/DOC ratio vs. SUVA values in the full scale treatment plant and IEX treated waters. 1. Raw water 2. nitrification 3. Coagulation-decantation 4. Intermediatechlorination and sand filtration 5. GAC 6. final chlorination. Data from GAC (5) are excluded from the regression equation. Two campaigns from the full scale treatment plant are presented in figure a) while SUVA values were only measured in one of the 2 campaigns in figure b).

Acknowledgments

This work was supported by Interreg 2 Seas program via DOC2C's project -contract No 2S01-013. Authors also thank De Watergroep, the Region "Hauts-de-France", CPER Climibio and Chevreul Institut for support. David Dumoulin, Veronique Alaimo and Romain Marie are thanked for their technical support.

while the THMFP is higher (218 to 384 µg L -1 ) for the IEX treated waters compared to the full scale plant for the 7d test (106 µg L -1 ). [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] Data of fluorescence acquired either from Fluorescence Excitation Emission Matrices analysed by PARAFAC or the humic like signal (excitation at 244 nm and emission at 420 nm) or protein like signal (excitation at 230 nm and emission at 345 nm) of the SEC chromatogram do not improve the characterization of DBP precursors (data not shown).

The IEX treatment alone is therefore not appropriate for the treatment of such waters but could still be useful to reduce the amounts of reactants used for clarification. This treatment is however limited in case of sulfate rich waters.

Assessment of the IEX resins to reduce the formation of DBPs under different halide concentration conditions

The IEX resin treatment has been tested in terms of increasing halide concentrations for the control of DBP formation. The concentration of iodide and bromide of the treated waters have been taken into account and equalized to make the comparison of the resin efficiency without considering the variation of halides between the different samples. 4 different scenario with increasing bromide concentrations from 228 up to 2200 µg L -1 with a constant iodide concentration of 63 µg L -1 (which corresponds to molar Br-/I-ratio from 5.7 up to 55) have been first performed to study the increase of bromide at iodide constant concentration. Additional experiments with extreme iodide concentration (i.e. 7 and 250 µg L -1 in addition to 63 µg L -1 performed in the first set of experiments) with a constant concentration of bromide (228 µg L -1 ) have also been performed in order to assess the I-DBP evolution.

In the following paragraph, the results of IRA-410, the best performing resin is presented; the results of other resins are available in the SI. The lowest bromide concentration tested corresponds to the highest level encountered in the IEX treated waters. For this already quite high concentration (228 µg L -1 ), the main THM4 were found to be a mix of CHBrCl2, CHCl3 and CHBr2Cl for the 4 resins (Figs. 3 A and S6-7), with a lower proportion of CHBr2Cl for the raw water. Some CHBr3 was also formed, especially for IRA-410 (Fig. 3A) but was a minor species. When the bromide concentration was increased, naturally, the THM4 shifted to a higher proportion of brominated species with a concomitant decrease of the chloroform formation. When the bromide concentration was increased to 2200 µg L -1 , CHBr3 was predominant in all 5 samples. However, a different trend is noticed between the raw water and the IEX treated waters. In fact, the quantity of THM4 (in molar concentration) did not change noticeably for the raw water (average concentration of 1.21 ± 0.01 µM for a concentration of Br -up to 800 µg L -1 and 1.30 µM for the highest bromide concentration). On the opposite, the formation of THM4 increased for the 4 IEX treated waters with the increase of bromide (from 37 to 44 % for an increasing bromide concentration from 228 up to 2200 µg L -1 compared to only 6.5% for the raw water, based on molar concentration).

Increasing bromide concentration

Considering the increasing molecular weight of brominated compounds compared to chlorinated ones, the increase is even higher (from 110 to 120% compared to 68% for raw water). Then, the concentration of THM4 formed in the IRA-958 treated water with the highest bromide concentration equalized the one achieved in the raw water for the same Br -concentration despite a substantial decrease of the DOC (23%) (Fig. S7). Bromine reacts more with hydrophilic compounds than with hydrophobic compounds, [START_REF] Awad | Characterization of dissolved organic matter for prediction of trihalomethane formation potential in surface and sub-surface waters[END_REF] which may be the reason for an overall increase in THM formation from the treated waters where part of hydrophobic compounds were removed. Considering the I-THMs, two different trends are noticed. In fact, for IRA-410, TAN-1 and to a lower extent IRA-958, the concentration of I-THMs first increases with the increase of bromide from 228 to 400 µg L -1 due to the higher formation of brominated substances (mainly CHBrClI), a phenomenon which has already been underlined in the literature. [START_REF] Allard | Mechanistic Study on the Formation of Cl-/Br-/I-Trihalomethanes during Chlorination/Chloramination Combined with a Theoretical Cytotoxicity Evaluation[END_REF] With even higher bromide concentrations, the formation of I-THMs then decreases, which is due to the fastest formation of iodate in presence of increasing concentration of bromide. [START_REF] Criquet | Iodate and Iodo-Trihalomethane Formation during Chlorination of Iodide-Containing Waters: Role of Bromide[END_REF] On the other hand, the I-THM formation uniformly decreases for the raw water and PPA860S treated water with the increase of bromide concentration (Fig. S8). These two waters correspond to the highest I-THM formation with the sums equalling to respectively 1.33 and 0.44 µg L - 1 for the raw and PPA860S treated waters (having the lowest bromide concentration). The same observation is made considering the molar concentration (Fig. S9). It has to be noticed that the overall I-THM formation is quite low, especially considering the iodide spiking (63 µg L -1 ). This low I-DBP formation is due to favourable conditions for the iodate formation considering the quite high bromide background concentration (228 µg L -1 ).

The reduction efficiency in I-THM formation in IEX treated water compared to the raw water dropped with the increase of bromide concentration (Fig. S9). For example, for IRA-410, the I-THM formation is lower than in the raw water by a factor of 16 ([Br -] = 228 µg L -1 ), but this factor is reduced to 6.5 for a bromide concentration of 2200 µg L -1 . Indeed, the higher competition of DOC for reactive bromine in raw water than in the treated waters at the lower bromide concentrations is less apparent at higher bromide concentrations.

The sum of the 7 HAAs remains more or less stable with increasing bromide concentrations as illustrated in the case of resin IRA-410 (Fig. 3C), but the molar concentrations decrease (Fig. S10). This could be due to the trihaloacetic acids (tri-HAAs) speciating towards more brominated tri-HAAs which could not be measured in this study. However, when considering the relative toxicity of the different compounds (dividing the DBP concentration by their respective LC50, Fig. S11) the toxicity related to the THM4 and HAA5 increases with the increase of bromide, as the main toxicity driver is the bromoacetic acid which concentration increases.

Increasing iodide concentration

IRA-410 treated water (24h chlorination test). B. % of iodate formation compared to iodide initial concentration for raw and the 4 IEX treated waters for increasing iodide concentration. [Br-] = 228 μg L-1; ND: no data.

The effect of increasing iodide concentration (0.06, 0.5 and 2 µM, corresponding to 7, 63 and 254 µg L -1 ) has been studied with a bromide concentration kept constant to the minimum background level possible (2.5 µM or 228 µg L -1 ). The results of I-THMs formation in the IRA-410 treated water (Fig. 4a)

show that the increase of I-DBP is not linear with the increasing concentration of iodide. In fact the I-THMs show a moderate increase for a concentration of iodide increasing from 0.06 to 0.5 µM (factor of 1.7 similar to the increase of I -concentration), but a much larger increase with the increase of iodide by a factor of 4 (from 0.5 to 2 µM), where the I-THM concentration is multiplied by a factor of 5.2. This behaviour is also noticed for the raw water and IRA-958 treated waters. However, for TAN-1 and the PPA860S, the increase is linear and asymptotic respectively. The 3 main I-THMs formed at low Iconcentration (CHCl2I, CHBrClI and CHBr2I) are shown to increase in the same proportion for the 3 compounds. A small quantity of CHClI2 appears from 63 µg L -1 I -. No CHI3 was detected which is in agreement with literature where much higher amounts of iodide were needed to form a substantial amount of CHI3. [START_REF] Hua | Effect of Bromide and Iodide Ions on the Formation and Speciation of Disinfection Byproducts during Chlorination[END_REF] The increase of I-THM formation is much lower than the one predicted by the linear regression obtained in previous section (Fig. S6). This regression was then limited to very low iodide concentration. Considering the iodate formation (Fig. 4b), the raw water exhibits a high formation of iodate at low iodide level, which is even higher than the initial iodide concentration. A part of organic iodine (total iodine equals to 35 µg L -1 ) is then transformed into iodate. This phenomena has also been observed on the full scale treatment plant. [START_REF] Mackeown | Formation and removal of disinfection by-products in a full scale drinking water treatment plant[END_REF] The % of iodate formation compared to the initial iodide content is lower for the IEX treated water (from 74 to 92%). It is interesting to note that the IEX treatment also removes a part of the organic iodine (removal of 40 to 86 % of total iodine). The iodate formation yield is reduced with increasing iodide concentration for raw and IEX treated waters (around 50% for 63 and 254 µg L -1 ). In terms of resin efficiency to remove I-DBP precursors, the IRA-410 resin has a particular behaviour compared to the other resins, with a decreasing capacity to reduce the formation of I-DBPs compared to the DBPs formed in the raw water. For example, the formation of I-THMs is reduced by a factor of almost 20 at low concentration of iodide for the IRA-410 treated water compared to the raw water, this factor dropped to 10 for the highest concentration of iodide tested.

On the contrary, the other resins show their capacity of minimizing the I-DBP increasing with the increase of iodide concentration. However, in most of the conditions tested here, IRA-410 treated water provides the lowest I-THMs formation (only TAN-1 with 254 µg L -1 of iodide show a lower I-THM formation). Low concentrations of iodoacetic acid (<30 ng L -1 ), one of the most genotoxic DBP compound identified to-date, [START_REF] Wagner | CHO cell cytotoxicity and genotoxicity analyses of disinfection by-products: An updated review[END_REF] has only been found in the PPA860S treated waters (spiked at 63 and 254 µg L -1 ) and in TAN-1 tread water (spiked at 254 µg L -1 ). Considering the yields of iodate formation and the moderate increase of I-THMs, the iodination of a wider type of active moieties of NOM not leading to THM formation could be supposed.

Considering the overall DBP formation and the removal of DBP precursors, it is worthwhile to notice that the reduction in DBP formation potential is reduced with increasing concentrations of bromide. It has been shown that bromine is generally a better agent for electrophilic substitution and addition than chlorine. [START_REF] Allard | Mechanistic Study on the Formation of Cl-/Br-/I-Trihalomethanes during Chlorination/Chloramination Combined with a Theoretical Cytotoxicity Evaluation[END_REF][START_REF] Criquet | Reaction of bromine and chlorine with phenolic compounds and natural organic matter extracts -Electrophilic aromatic substitution and oxidation[END_REF] The removal of NOM favouring the bromination, could be worrying in terms of toxicity of DBP formed as brominated DBPs generally exhibit higher toxicity than the chlorinated analogues. 45 The advantage of resins in the removal of specific DBP precursor is then reduced in presence of high concentration of halides. The effect of increasing concentrations of iodide alone, which may hardly be found in natural waters without a concomitant increase of bromide however shows here that the precursors of I-DBP and/or reactivity of HOI are different from brominated and chlorinated analogues. This would deserve more research to identify the different properties of DBP precursors responsible for DBP speciation.

Conclusions

Four ion exchange resins was tested for the removal of NOM and the concomitant assessment of reduction of disinfection by-product formation. It was confirmed that all resins preferentially removes the humic substances fraction. In addition, the SUVA value was lowered by the IEX treatment to a higher extend than the DOC removal demonstrating the preference of resins for the highly hydrophobic compounds exchange. In this source water, a large part of the removal efficiency of the resins is dramatically reduced by the high content of sulfate, which would be a key negative factor for the further upscaling of this process on such raw water. In terms of disinfection by-product formation, the reduction of DBP formation potential compared to the one achieved in raw water is related to the DOC removal efficiency of the resins. The IRA-410 resin showed the best performance followed by the PPA860S. However, from the comparison of the different resins, it has been demonstrated that the efficiency in DBP reduction is better linked to the SUVA removal than to DOC removal. The experiments also showed that the resins preferentially remove the DBP precursors which have a slow reactivity towards chlorination.

All resins show a beneficial effect in lowering the concentration of I-DBPs by lowering iodide and/or by the removing of specific precursors. Finally, the advantage of DBP reduction by the IEX treatment is reduced in presence of an increasing concentration of halides. The reduction of efficiency is highly dependent on the resin type and then this aspect would have to be taken into account in addition to the DOC removal efficiency in case of source water impacted by high halide levels or sources showing a risk of increasing halide concentrations in the future due to anthropogenic pressure.

Conflicts of interest

There are no conflicts to declare.


Julien Girard-Satabin

Guillaume Charpiat

Zakaria Chihani

Marc Schoenauer

CAMUS: A Framework to Build Formal Specifications for Deep Perception Systems Using Simulators

The topic of provable deep neural network robustness has raised considerable interest in recent years. Most research has focused on adversarial robustness, which studies the robustness of perceptive models in the neighbourhood of particular samples. However, other works have proved global properties of smaller neural networks. Yet, formally verifying perception remains uncharted. This is due notably to the lack of relevant properties to verify, as the distribution of possible inputs cannot be formally specified. We propose to take advantage of the simulators often used either to train machine learning models or to check them with statistical tests, a growing trend in industry. Our formulation allows us to formally express and verify safety properties on perception units, covering all cases that could ever be generated by the simulator, to the difference of statistical tests which cover only seen examples. Along with this theoretical formulation, we provide a tool to translate deep learning models into standard logical formulae. As a proof of concept, we train a toy example mimicking an autonomous car perceptive unit, and we formally verify that it will never fail to capture the relevant information in the provided inputs.

Introduction

Recent years have shown a considerable interest in designing "more robust" deep learning models. In classical software safety, asserting the robustness of a program usually consists in checking if the program respects a given specification. Various techniques can output a sound answer whether the specification is respected or not, provided it is sufficiently formally formulated. However, the deep learning field is different, since the subject of verification (the deep learning model) is actually obtained through a learning algorithm, which is not tailored to satisfy a specification by construction. In this paper, we will denote as "program" the deep learning model that is the result of a learning procedure. Such a program aims to perform a certain task, such as image classification, using statistical inference. For this, it is trained through a learning algorithm, usually involving a loss minimization by gradient descent over its parameters. This way, in most deep learning applications, there exists no formal specification of what the program should achieve at the end of the learning phase: instead, the current dominant paradigm in statistical learning consists in learning an estimator that approximates a probability distribution, about which little is known. Failures of learning procedures, such as overfitting, are hard to quantify and describe in the form of a specification. 1 CEA LIST, France, julien.girard2@cea.fr, zakaria.chihani@cea.fr 2 TAU, INRIA, LRI -Université Paris-Saclay, France, First.Last@inria.fr A particular flaw of deep learning, namely adversarial examples, has been the subject of intensive research [START_REF] Szegedy | Intriguing properties of neural networks[END_REF][START_REF] Carlini | Towards Evaluating the Robustness of Neural Networks[END_REF][START_REF] Papernot | Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples[END_REF][START_REF] Ilyas | Adversarial Examples Are Not Bugs, They Are Features[END_REF]. Recently, the quest for provable adversarial robustness has been bringing together the machine learning and formal methods communities. New tools are written, inspired by decades of work in software safety, opening new perspectives on formal verification for deep learning. However, the bulk of these works has focused on the specific issue of adversarial robustness. Apart from well-defined environments where strong prior information exists on the input space (see [START_REF] Katz | Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks[END_REF]), little work has been made on formulating and certifying specific properties of deep neural networks.

The goal of this work is to propose a framework for the general problem of deep learning verification that will allow the formulation of new properties to be checked, while still benefiting from the efforts of the formal methods community towards more efficient verification tools. We aim to leverage the techniques developed for adversarial robustness and extend the scope of deep learning verification to working on global properties. Specifically, we focus on a still unexplored avenue: models trained on simulated data, commonplace in the automotive industry. Our contribution is twofold: we first propose a formalism to express formal properties on deep perception units trained on simulated data; secondly we present an open source tool that directly translates machine learning models into a logical formula that can be used to soundly verify these properties, hence ensuring some formal guarantees. Recent work proposed to analyse programs trained on simulators [START_REF] Dreossi | Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems[END_REF]. Although their motivations are similar to ours, they work on abstract feature spaces without directly considering the perception unit, and they rely on sampling techniques while we aim to use sound, exhaustive techniques. Their aim is to exhibit faulty behaviour in some type of neural network controllers, while we can formally verify any type of perception unit.

The paper is structured as follows: We first describe our formulation of the problem of verification of machine learning models trained on simulated data. We then describe the translator tools, and detail its main features. Finally, we present as a first use case a synthetic toy 'autonomous vehicle' problem. We conclude by presenting the next issues to tackle.

Related work

Adversarial robustness: a local property

Adversarial perturbations are small variations of a given example that have been crafted so that the network misclassifies the resulting noisy example, called an adversarial example. More formally, given a sample x0 in a set X , a classification function C : x ∈ X → R d , a distortion amplitude ε 0 and a distance metric . p , a neural network is locally ε-robust if for all perturbations δ s. Since their initial discovery in [START_REF] Szegedy | Intriguing properties of neural networks[END_REF], adversarial examples became a widely researched topic. New ways to generate adversarial examples were proposed [START_REF] Carlini | Towards Evaluating the Robustness of Neural Networks[END_REF][START_REF] Kurakin | Adversarial examples in the physical world[END_REF][START_REF] Yao | Trust Region Based Adversarial Attack on Neural Networks[END_REF], as well as defenses [START_REF] Araujo | Robust Neural Networks using Randomized Adversarial Training[END_REF][START_REF] Madry | Towards Deep Learning Models Resistant to Adversarial Attacks[END_REF]. Other works focus on studying the theory behind adversarial examples. While the initial work [START_REF] Goodfellow | Explaining and Harnessing Adversarial Examples[END_REF] suggests that adversarial examples are a result of a default in the training procedure, "bugs", recent investigations ( [START_REF] Ford | Adversarial Examples Are a Natural Consequence of Test Error in Noise[END_REF][START_REF] Ilyas | Adversarial Examples Are Not Bugs, They Are Features[END_REF][START_REF] Simon-Gabriel | First-order Adversarial Vulnerability of Neural Networks and Input Dimension[END_REF]) suggest that (at least part of the) adversarial examples may be inherently linked to the design principles of deep learning and to their resulting effects on programs: using any input features available to decrease the loss function, including "non robust" features that are exploited by adversarial examples generation algorithms. It is important to note that their very existence may be tied with the fact that we employ deep neural network on highly-dimensional perceptual spaces such as images, where we witness counter-intuitive behaviours. In any case, their imperceptibility for humans and their capacity to transfer between networks and datasets [START_REF] Papernot | Transferability in Machine Learning: from Phenomena to Black-Box Attacks using Adversarial Samples[END_REF] make them a potentially dangerous phenomenon regarding safety and security. For example, an autonomous car sensor unit could be fooled by a malicious agent to output false direction in order to cause accidents.

Proving global properties in non-perceptual space

It is possible to express formal properties in simpler settings than adversarial robustness. By simpler, we mean two main differences: (i) the dimensionality of the input is much lower than in typical perception cases, where most of adversarial examples occur, and (ii) the problem the program aims to solve provides an explicit description of the meaning of the inputs and outputs, making a formulation of safety property much simpler. Rephrased otherwise, the program is working on inputs whose semantics is (at least partially) defined. Since deep neural networks use simple programming concepts (e.g., no loops), it is quite easy to translate them directly to a standard verification format, such as SMT-LIB [START_REF] Barrett | The SMT-LIB Standard[END_REF]. Provided the inputs are sufficiently well defined, it is then possible to encode safety properties as relationships between inputs and outputs, such as inequality constraints on real values.

An example of such setting can be seen in the Anti Collision Avoidance System for Unmanned aircrafts (ACAS-Xu) [START_REF] Manfredi | An introduction to acas xu and the challenges ahead[END_REF]. Inputs correspond to aircraft sensors, and outputs to airplane commands. In such case, specifications can be directly encoded as a set of constraints on the inputs and outputs. In [START_REF] Katz | Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks[END_REF], the authors proposed an implementation of ACAS-Xu as a deep neural network, and they were able to formally prove that their program respected various safety properties.

It is important to note that the inputs of the program are here highlevel information (existence of an intruder together with its position), which completely bypasses the problem of perception (as airplanes have direct access to this information, in a low-level form, through their sensors, and through communications with ground operators).

Tools for provable deep learning robustness

Critical systems perform operations whose failure may cause physical harm or great economical loss. A self-driving car is a critical sys-tem: failure of embedded software may cause harm, as seen in accidents such as [START_REF] Hawkins | Tesla didn't fix an autopilot problem for three years, and now another person is dead[END_REF]. In the automobile industry, one expects the airbag to resist to a given pressure, the tires to last for a lower-bounded duration, etc. As software is more and more ubiquitous in vehicles, it is natural to have high expectations for software safety as well. An attempt to meet these expectations makes use of formal methods. This general term describes a variety of techniques that aim to provide mathematically sound guarantees with respect to a given specification. In less than a decade, an impressive amount of research was undertaken to bring formal verification knowledge and tools to the field of adversarial robustness. Deep learning verification has developed tools coming from broadly two different sets of techniques; this taxonomy is borrowed from [START_REF] Bunel | A Unified View of Piecewise Linear Neural Network Verification[END_REF].

The first set is the family of exact verification methods, such as Satisfiability Modulo Theory [START_REF] Barrett | Satisfiability modulo theories[END_REF]. SMT solvers perform automated reasoning on logical formulae, following a certain set of rules (a logic) on specific entities (integers, reals, arrays, etc.) described by a theory. An SMT problem consists in deciding whether, for a given formula, there exists an instantiation of the variables that makes the formula true. Programs properties and control flow are encoded as logical formulae, that specialized SMT solvers try to solve. It is possible to express precise properties, but since most SMT solvers try to be exhaustive over the search space, a careful formulation of the constraints and control flow is necessary to keep the problem tractable. It was formally proven in [START_REF] Katz | Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks[END_REF] that solving a verification problem composed of conjunction of clauses by explicit enumeration for a feedforward network is NP-hard. However, the NP-hardness of a problem does not prevent us from designing solving schemes. In this same work, the authors introduced ReLuPlex, a modified solver and simplex algorithm, that lazily evaluates ReLUs, reducing the need to branch on non-linearities. Their follow-up work [START_REF] Katz | The Marabou Framework for Verification and Analysis of Deep Neural Networks[END_REF] improves and extends the tool to support more complex networks and networklevel reasoning. Others [START_REF] Bunel | A Unified View of Piecewise Linear Neural Network Verification[END_REF] rephrase the problem of adversarial robustness verification as a branch-and-bound problem and provide a solid benchmark to compare current and future algorithms on piecewise linear networks. Other exact techniques are based on mixed integer linear programming (MILP). The verification of adversarial robustness properties on piece-wise linear networks can indeed be formulated as a MILP problem [START_REF] Tjeng | Evaluating Robustness of Neural Networks with Mixed Integer Programming[END_REF], and a pre-conditioning technique drastically reduces the number of necessary calculations. Adversarial robustness properties were thus checked on ResNets (a very deep architecture) with l∞-bounded perturbations on CIFAR-10.

The second set of techniques in formal methods is based on overapproximating the program's behaviour. Indeed, since solving the exact verification problem is hard, some authors worked on computing a lower bound of ε, using techniques building overapproximations of the program, on which it is easier to verify properties. Abstract interpretation (first introduced in [START_REF] Cousot | Abstract Interpretation: A Unified Lattice Model for Static Analysis of Programs by Construction or Approximation of Fixpoints[END_REF]) is an example of such technique. It is a mathematical framework aiming to prove sound properties on abstracted semantics of program. In this framework, a program's concrete executions are abstracted onto less precise, but more computationally tractable abstract executions, using numerical domains. Finding numerical domains that balance expressiveness, accuracy and calculation footprint is one of the key challenges of abstract interpretation.

The first instance of specifically-tailored deep learning verification described how to refine non-linear sigmoid activation function to help verification [START_REF] Pulina | An Abstraction-Refinement Approach to Verification of Artificial Neural Networks[END_REF]. [START_REF] Wong | Provable defenses against adversarial examples via the convex outer adversarial polytope[END_REF] proposes an outer convex envelope for ReLu classifiers with linear constraints, expressing the robustness problem as a Linear Programming (LP) problem. [START_REF] Mirman | Differentiable Abstract Interpretation for Provably Robust Neural Networks[END_REF] and [START_REF] Singh | An Abstract Domain for Certifying Neural Networks[END_REF] propose a framework for building abstract interpretations of neural net-works, which they use to derive a tight upper bound on robustness for various architectures and for regularization. On MNIST, both works displayed a robustness of 97% bounded by a l∞ = 0.1 perturbation. On CIFAR-10, they achieved a 50% robustness for a similar net with a l∞ = 0.006 perturbation. Symbolic calculus on neural networks is performed in [START_REF] Wang | Formal Security Analysis of Neural Networks using Symbolic Intervals[END_REF], allowing symbolic analysis and outperforming previous methods. A verification framework based on bounding ReLu networks with linear functions is proposed in [START_REF] Boopathy | CNN-Cert: An Efficient Framework for Certifying Robustness of Convolutional Neural Networks[END_REF].

The boundary between these two families of techniques can be blurry, and both techniques can be combined. For instance, [START_REF] Singh | Robustness Certification with Refinement[END_REF] combines overapproximation and MILP techniques to provide tighter bounds on exact methods. Competing with complete methods, they verify a hard property on ACAS-Xu faster and provide precise bounds faster than other methods.

All these techniques are employed either for proving local properties (local adversarial robustness), or on simpler, non perceptual input spaces. On the opposite, our work proposes a framework to prove global properties on perceptual inputs.

3 CAMUS: a new formalism to specify and verify machine learning models

Motivation

In most deep learning application domains, such as image classification [START_REF] He | Deep Residual Learning for Image Recognition[END_REF], object detection [START_REF] Chabot | Deep manta: A coarse-to-fine manytask network for joint 2d and 3d vehicle analysis from monocular image[END_REF], control learning [START_REF] Bojarski | End to end learning for selfdriving cars[END_REF], speech recognition [START_REF] Ravanelli | The PyTorch-Kaldi Speech Recognition Toolkit[END_REF], or style transfer [START_REF] Karras | A style-based generator architecture for generative adversarial networks[END_REF], there exists no formal definition of the input. Let us consider the software of an autonomous car as an example. A desirable property would be not to run over pedestrians. This property can be split in i) all pedestrians are detected, and ii) all detected pedestrians are avoided. For a formal certification, the property should be expressed in the form "For any image containing pedestrians, whatever the weather conditions or camera angle could be, all pedestrians present in that image are detected and avoided". Such a formulation supposes one is able to describe the set of all possible images containing pedestrians (together with their location). However, there exists no exact characterization of what a pedestrian is or looks like, and certainly not one that takes into account weather condition, camera angle, input type or light conditions. Any handmade characterization or model would be very tiresome to build, and still incomplete. On the upside, machine learning has demonstrated its ability to make use of data that cannot be formally specified, yielding impressive results in all above-mentioned application domains, among others; on the downside, it has also been demonstrated that ML models can easily fail dramatically, for instance when attacked with adversarial examples. Thus, manufacturers of critical systems need to provide elements that allow regulators, contractors and end-users to trust the systems in which they embed their software.

Usually, car manufacturers rely on test procedures to measure their system's performances and safety properties. But testing can, at best, yield statistical bounds on the absence of failures: The efficiency of a system against a particular situation is not assessed before this situation is actually met during a real-world experiment. As the space of possible situations is enormous (possibly infinite) and incidents are rare events, one cannot assess that an autonomous vehicle will be safe in every situation by relying on physical tests alone.

A current remedy is to use artificial data, and to augment the actual data with data generated by a simulation software, with several benefits: Removing the need to collect data with expensive and time consuming tests in the real world; Making it possible to generate potentially hazardous scenarios precisely, e.g., starting with the most common crash cases. Examples of such simulators are Carla [START_REF] Dosovitskiy | CARLA: An open urban driving simulator[END_REF] and the NVIDIA Drive Constellation system. However, even if it is possible to artificially generate corner cases more easily, the space of possible scenarios is still enormous, and some accidents remain completely unpredictable a priori by human test designers. For instance, in a recent car accident involving partially self-driving technology, the manufacturer admitted that the camera failed to distinguish a white truck against a bright sky [START_REF] Hawkins | Tesla didn't fix an autopilot problem for three years, and now another person is dead[END_REF], causing the death of the driver. Such a test case is difficult to come up with for a human, because it is the conjunction of specific environmental conditions and specific driving conditions.

Our motivation is to bring an additional layer of trust, not relying on statistical arguments, but rather on formal guarantees. Our long term objective is to be able to formalize a specification and to provide guarantees on every possible scenario, automatically finding violations of the specification. Because practitioners are now relying more and more on simulators, we propose as a first step to study such simulated setting, and to formalize it. The idea is to rephrase the verification problem in order to include both the deep learning model and the simulator software within the verification problem. As said earlier, a simulator offers more control on the learning data by providing explicit parameters (for instance: number and positions of pedestrians on the image).

Problem formulation and notations

Let f : X → Y be an algorithm taking a perceptual input x ∈ X and yielding a decision y ∈ Y. The perceptual space X will typically be of the form R d or [0, 1] d . In the general framework of this work, f is a program trained with a learning procedure on a finite subset of X to perform a specific task (e.g., drive the passengers safely home). In our example, the task would be to output a command from an image, in which case, for a given image x, f (x) would be the driving action taken when in environment x.

Let us denote by g : S → X the simulator, that is, a function taking as input a configuration s ∈ S of parameters, and returning the result of the simulation associated to these parameter values. A configuration s of parameters contains all information needed by the simulator to generate a perceptual input; each parameter may be a discrete or continuous variable. Let us take as running example a simulator of autonomous car images: s would contain the road characteristics, the number of pedestrians and their positions, the weather conditions. . . , that is, potentially, thousands or millions of variables, depending on the simulator realism.

The problem to solve here is the following: For a model f trained on data belonging to X generated by g to perform a certain task, how can we formulate and formally verify practical safety properties for all possible x ∈ X , including samples never seen during training?

Including the simulator in the verification

In standard settings, such as the ones schematized in Figure 1, specifications express relationships from X to Y using a formulation of f . But X is such a huge space that formulating properties that are non trivial, let alone verify these, is prohibitively difficult, especially in the case of perceptive systems where the domain of x cannot be specified: all matrices in [0, 255] 3 #pixels are images, technically, but few of them make sense, and one cannot describe which ones. Moreover, given an image x, the property to check might be difficult to express, as, to state that all pedestrians were detected and avoided, one needs to know whether there are pedestrians in x and where, which we do not know formally from just the image x. And if one had a way to retrieve such information from x (number and location of pedestrians) without any mistake, one would have already solved the initial problem, i.e., safe self-driving car. To summarize, in this setting, it is impossible to express a relevant space for x and a property to verify Φ:

∀x ∈ ?, Φ? f (x)
In the setting of simulated inputs, though it remains difficult to formulate properties on the perceptual space X , we know that this space is produced by g applied to parameters in S. On the contrary to X , S is a space where there exists an abstract, albeit simplistic characterization of entities. Indeed, setting parameters for a pedestrian in the simulated input yields a specification of what a pedestrian is in X according to the inner workings of g. The procedure g transforms elements s ∈ S, that represent abstracted entities, into elements x ∈ X that describe these entities in the rich perceptual space. To output values in Y, f has to capture the inner semantics contained in X , that is to say, to abstract back a part of S from X .

The above remark is the key to the proposed framework: If we include S and g alongside f , X and Y in the verification problem, then all meaningful elements of S are de facto included. It then becomes possible to formulate interesting properties, such as "given a simulator that defines pedestrians as a certain pattern of pixels, does a model trained on the images generated by this simulator avoid all pedestrians correctly?". Formally, to ensure that the output y = p • g(s) satisfies a property Φ for all examples x = g(s) that can ever be generated by the simulator, the formula to check is of the form:

∀s ∈ S, Φ s, p • g(s)
The property Φ may depend on s indeed, as, in our running example, s explicitly contains the information about the number of pedestrians to be avoided as well as their locations. Including S and g in a formal property to check requires to formulate at least partially the multiple functions that compose g. De-scribing precisely these procedures is a key problem that we plan to address later.

As our framework relies on including the simulator in the verification problem, we call it Certifying Autonomous deep Models Using Simulators (CAMUS).

Separating perception and reasoning

Before the rise of deep learning, the perception function (which, e.g., recognizes a certain pattern of pixels as a pedestrian) and the control, or reasoning function (which, e.g., analyzes the location of a pedestrian and proposes a decision accordingly) in vehicles were designed and optimized separately. However, work such as [START_REF] Bojarski | End to end learning for selfdriving cars[END_REF] showed that end-to-end learning can in general be a much more efficient alternative; there exist many incentives to adopt this end-to-end architecture, mixing and training jointly the perception and control functions. However, combining perception and reasoning into one model makes the formulation of safety properties more difficult.

Thus in our description (see Fig. 3), we choose to separate the perception and the reasoning functions, respectively in the components p and r. The perception part p is in charge of capturing all relevant information contained in the image, while the reasoning part r will make use of this relevant information to output directives accordingly to a specification.

One way to make sure that p retrieves all relevant information is to require it to retrieve all information available, that is, to reconstruct the full simulator parameter configuration s. In this setting, the output s of the perception module p lies in the same space as the parameter configuration space S, and the property we would like to be satisfy can be written as p • g = Id, which can be rewritten as:

∀s ∈ S, p • g(s) = s (1)
This way, we ensure that the perception module p correctly perceives all samples that could ever be generated by the simulator. In the case some parameters are known not to be relevant (image noise, decoration details, etc.), one can choose not to require to find them back, therefore asking to retrieve only the other ones. For the sake of notation simplicity, we will here consider the case where we ask to reconstruct all parameters. This separation between perception p and further reasoning r brings modularity as an additional benefit: even when dealing with different traffic regulations, it is only necessary to prove p once; the verification of compliance towards local legislations and specifications by r can be done separately. It allows to reuse the complex perception module with different reasoning modules r without needing to re-prove it. Note also that r does not need to be as complex as p, since it will work on much smaller spaces; multiple verifications of r are then easier.

One could argue that this formulation makes the problem more complex, and it indeed may be the case. However, our proposition is aimed at safety, and in order to provide additional trust, it is sometimes necessary to formulate the problem differently. For instance, there are good practices to structure the code to provide some safety guarantees: bounded loops, correctly allocated and de-allocated references, ban of function references, . . . are constructs that voluntarily restrain the expressive power of the programming language to ensure a safer behaviour. Hence, although this formulation might seem like a step back with regard to the state-of-the-art, we argue that it provides a new way to formulate safety properties, and hence will be beneficial in the long run. between perception and reasoning: p learns to capture all the relevant parameters; r learns to respect the specification. Verifying φ 1 proves the perception module once and for all; verifying φ 2 can be done when the specification changes (e.g., for different driving rules).

Properties Formulation

Considering jointly the simulator g and the machine learning model f , split in p and r, two families of properties are amenable to formal checking:

• Φ1: perception module p has captured sufficient knowledge from X ; • Φ2: reasoning module r respects a specification property regarding Y.

Families of property φ2 have been addressed in the literature -see Section 2.2. The key point of the proposed approach is thus to obtain a representation space that reliably yields semantic meaning, which is the objective of Φ1. Since the simulator is included in the verification problem, properties of family Φ1 can be written as relationships between input parameter configurations s ∈ S and retrieved parameter configurations s ∈ S, outputs of the perception module p. Strict equality between s and s may be difficult to achieve, and is actually not needed as long as the reasoning module r is able to deal with small estimation errors. Expressed in the proposed formalism, the perception task is equivalent to finding (a good approximation of) S. Thus, a relaxed version of property 1 to satisfy could be formalized as some tolerance ε > 0 on the reconstruction error s -s (for some metric • ):

∀s ∈ S, s -p • g(s) ε (2)

Discussion

As stated earlier, it is not always necessary to retrieve all parameters of configuration s. For instance, one could seek to retrieve only the correct number of pedestrians and their locations, from any image generatable by the simulator. In this case, the output of p would be just a few coefficients of s, and must consequently be characterized differently (e.g., as belonging to a given subspace of S). This would allow to express more flexible properties than simply reconstruct all parameters.

For the model f to correctly generalize, the simulated data must yield two characteristics:

1. they need to be sufficiently realistic (that is to say, they should look like real-world images); if not the network could overfit the simplistic representation provided by the simulator; 2. they must be representative of the various cases the model has to take into account, to cover sufficiently diverse situations.

Additional characterization of the simulator would be difficult. For instance, one could suggest to require the simulator g to be either surjective or injective, in order to cover all possible cases x ∈ X , or for parameters to be uniquely retrievable. Yet, the largest part of the perceptual space X is usually made of nonsensical cases (think of random images in [0, 255]3 #pixels with each pixel color picked independently: most are just noise), and the subspace of plausible perceptual inputs is generally not characterizable (without which the problem at hand would already be solved). Regarding injectivity, being one-to-one is actually not needed when dealing with properties such as 2.

Finally, let us consider the case where several simulators are available, and where, given a perceptive system p, we would like to assert properties of type Φ1 for each of the simulators. At first glance, as the output of p consists of retrieved parameters, this would seem to require that all simulators are parameterized exactly identically (same S). However, for real tasks, one does not need to retrieve all parameters but only the useful ones (e.g., number of pedestrians and their locations), which necessarily appear in the configuration of all simulators. As it is straightforward to build for each simulator a mapping from its full list of parameters towards the few ones of interest, a shared space for retrieved parameters can be defined, a unique perception system p can be trained, and formal properties for all simulators can be expressed.

Translating neural networks into logical formulae

In previous section, we introduced two families of properties: Φ1 involves the simulator g, its parameter space S, the perceptual space X where simulated data lie, and the perception unit p; Φ2 involves the representation space learned by p (which should be a copy of S), the reasoning unit r and its output space Y.

In order to be able to actually formulate properties of these families, we must first be able to represent all these elements as logical formulae. The goal of this section is to introduce ONNX2SMT, a tool to do so automatically.

ONNX2SMT provides an interface to all machine learning models that use the Open Neural Network Exchange format [START_REF] Bai | Onnx: Open neural network exchange[END_REF], and translates them into the standard language SMT-LIB [START_REF] Barrett | The SMT-LIB Standard[END_REF], allowing all stateof-the-art generalist SMT solvers and deep learning verification specialized tools to work on a direct transcription of state-of-the-art neural networks. ONNX2SMT will be open-sourced to further help the community effort towards safer deep learning software 3 .

ONNX and SMT-LIB

The Open Neural Network eXchange format (ONNX) 4 is a community initiative kickstarted by Facebook and Microsoft, that aims to be an open format for representing neural networks, compatible across multiple frameworks. It represents neural networks as directed acyclic graphs, each node of the graph being a call to an operation. Common operations in machine learning and deep learning are tensor multiplications, convolutions, activations functions, reshaping, etc. 5 . Operations have predecessors and successors, describing the flow of information in the network. The network parameters are also stored in the ONNX graph. A wide variety of deep learning frameworks support ONNX, including Caffe2, PyTorch, Microsoft CNTK, Mat-Lab, SciKit-Learn and TensorFlow. Examples of use cases presented in the official page 6 include benchmarking models coming from different frameworks, converting a model prototyped using PyTorch to Caffe2 and deploying it on embedded software.

SMT-LIB2 is a standard language used to describe logical formulae to be solved using SMT solvers. Most state-of-the-art solvers implement a SMT-LIB support, which facilitates benchmarks and comparisons between solvers. SMT-COMP [START_REF] Barrett | Smt-comp: Satisfiability modulo theories competition[END_REF] is a yearly competition using SMT-LIB as its format. This challenge is a unique opportunity to present different techniques used by solvers, to increase the global knowledge of the SMT community. SMT-LIB2 supports expressing formulae using bit vectors, Boolean operators, functional arrays, integers, floating points and real numbers, as well as linear and nonlinear arithmetic. In this work, only the Quantifier-Free Non linear Real Arithmetic (QF NRA) theory will be used. Since the language aims to be compatible with a wide variety of solvers, expressivity is limited compared to languages such as Python, used by most deep learning platforms. In particular, there is no built-in Tensor type, and it is hence necessary to adapt the semantics of tensors to SMT-LIB2. This adaptation is performed by ONNX2SMT.

Features

Features of ONNX2SMT include the support of the most common operations in modern neural networks, such as tensors addition and multiplication, maxpooling and convolution on 2D inputs. Support for a wider range of operators (such as reshaping or renormalization operators) is on-going work.

ONNX being a low-level standard, ONNX2SMT uses a Neural IntErmediate Representation (NIER) to perform optimizations and modifications of the NN structure, e.g., by avoiding superfluous variable creation during the flattening of the tensors that results from broadcast. While ONNX is considered a standard, it is a rather permissive one, as networks generated by PyTorch and TensorFlow, for example, will not store the same information in the same fields. Our intermediate representation therefore provides the unification necessary to the more strict SMT-LIB standard. Finally, NIER also allows our planned work of offering an ouput to CP and MIP solvers that are not always compatible with SMT-LIB standard.

The conversion from ONNX to NIER is performed thanks to the reference protobuf description of ONNX, converted to OCaml types using the piqi 7 tool suite. ONNX2SMT provides straightforward conversion from ONNX to SMT-LIB, using NIER as an intermediate representation.

All the features described above allow us to encode machine learning models (p and r) as SMT formulae. X and Y can be expressed directly using QF NRA existing primitives. Future work will provide an additional mechanism to encode g and S.

Usage

ONNX2SMT workflow can be summarized as follows: Input: an ONNX file created using an ML framework; 1. Convert the ONNX model to NIER (onnx parser ); 2. Convert NIER to a SMT-LIB ( smtifyer ) string, written on disk; 3. Add the property to validate to the existing SMT-LIB file; Output: An SMT-LIB file that can be solved to prove the property.

Experiments

As a proof of concept for the proposed framework, it is applied it to a simple synthetic problem. Neural networks are trained with PyTorch [START_REF] Paszke | Automatic differentiation in PyTorch[END_REF], then converted into ONNX using the built-in ONNX converter, and finally converted into the intermediate representation in SMT-LIB format with ONNX2SMT. We use z3 [START_REF] De | Z3: An efficient smt solver[END_REF], CVC4 [START_REF] Barrett | CVC4[END_REF], YICES [START_REF] Dutertre | The yices smt solver[END_REF] and COLIBRI [START_REF] Marre | Real behavior of floating point numbers[END_REF] SMT solvers as standard verification tools.

Let us consider here the perception module of an autonomous vehicle, whose goal is to output driving directives that result in safe driving behaviour. The perception module is modeled as a deep neural network with one output node, taking as input an image. If an obstacle lies in a pre-defined "danger zone", the network should output a "change direction" directive. Otherwise, it should output a "no change" directive.

The "simulator" is here a Python script, taking as input the number and the locations on the image of the one-pixel wide obstacles and generating the corresponding black-and-white images.

The verification problem consists in the formulation of the network structure and constraints on the inputs, and in the following properties to check:

1. verify that an input with an obstacle (or several ones) in the danger zone will always lead to the "change direction" directive; 2. verify that an input without obstacle on the danger zone will never lead to the "change direction" directive.

If both properties are verified, our model is perfect for all the inputs that can be generated. If the first one is not verified, our verification system will provide examples of inputs where our model fails, which can be a useful insight on the model flaws. Such examples could be used for further more robust training, i.e., integrated into a future training phase to correct the network misclassifications. Similarly, if the second property is not verified, the solver will provide false positives, that can help designers reduce erroneous alerts and make their tools more acceptable for the end-user.

Experimental setting

In this toy example, input data are N × N black-and-white images (see Fig. 4 for examples). The space of possible simulated data g(S) ⊂ X can simply be described by the constraint that each pixel can only take two values (0 and 1). In real life, data are much more complex, possibly continuous; such data can also be handled in our framework, though experimenting with realistic simulators is the topic of future work. The neural network is fully-connected with two hidden layers. The number of neurons in the first and second hidden layers are respectively one half and one quarter of the flattened size of the input (N 2 ). We used Glorot to initialize weights, with a gain of 1. The network was trained with Adam optimizer for 2000 epochs, batch size of 100, using the binary cross entropy loss. The danger zone is defined as the bottom half part of the image. Any image with at least one white pixel in this zone should then yield a "change direction" directive.

Here, constraints on inputs are encoded as statements on the SMT-LIB variables. A fragment of property to check is presented on Figure 5. On such a simple problem, the decomposition perception/reasoning is not needed, since there exists a formal characterization of what an obstacle is. All network's parameters were converted using the QF NRA theory. )) (or (= actual_input_0_0_0_7 1.) (= actual_input_0_0_0_8 1.)) . . . ;; Formulate constraint on outputs ;; ''... then output for detected ;; obstacle always higher ;; than for no obstacle'' (assert (> actual_output_0_0_0_0 actual_output_0_0_0_1))

Figure 5: A SMTLIB2 file describing our problem. First part is a full description of the network, automatically produced by ONNX2SMT.

Handmade annotations describe the property to check, i.e. there are no false negatives in our network. The goal for the solver is to find a counterexample.

Results

We present preliminary results for N ∈ {3, 5, 7}. The goal was to return UNSAT, meaning no counterexample to the property was found. Under the hypothesis that the space of all possible data is described by the simulator (which is the case here), the model will never fail to detect obstacles. Runtimes are available in Table 2. For N = 5 and N = 7, most of possible input configurations were not seen in the training or test sets. Yet, the network is proven always correct, which shows interesting generalization abilities. We were thus able to verify quickly that this perceptive module will never miss obstacles, at least for inputs generable by the simulator.

Discussion and perspectives

We introduced CAMUS, a formalism describing how to formally express safety properties on functions taking simulated data as input.

We also proposed ONNX2SMT, a tool soon to be open sourced, that leverages two standards used by the communities of formal methods and machine learning, to automatically write machine learning algorithms as logical formulae. We demonstrated the joint use of ONNX2SMT and CAMUS on a synthetic example mimicking a selfdriving car perceptive unit, as a proof of concept of our framework. This toy example is of course still simplistic and much work on scalability is needed before real self-driving car simulators can be incorporated into formal proofs. Among future work, ONNX2SMT will be released (see https: //www.lri.fr/ ˜gcharpia/camus/) and gain support for more deep learning operations. While we provide a toolkit to translate neural networks directly in our framework, a way to easily represent a simulator is yet to include. It is not an easy task, since the simulator must be describable with sufficient granularity to allow the solver to use the simulator internal working to simplify the verification problem. A scene description language with a modelling language for simulators is a possible answer to these issues. Further theoretical characterization of the simulator procedure and its link with the perceptive unit will be undertaken, for instance to encompass stochastic processes. Besides, on more complex simulators, programs and examples, the problem to verify will remain computationally difficult. Various techniques to enhance solvers performances will be developed and integrated in CAMUS, taking advantage of the domain knowledge provided by the simulators parameters. Finally, our current framework checks properties for all possible inputs, including anomalous ones such as adversarial attacks. A possible extension would be to identify "safe" subspaces instead, where perception is guaranteed to be perfect, and "unsafe" subspaces where failures may happen.

Figure 1 :

 1 Figure 1: Natural inputs with huge perceptual space: no characterization of the input nor property can be formulated.

Figure 2 :

 2 Figure 2: Generated inputs with integration of the generation procedure in the verification problem. There are now new properties to check since we have a formal characterization of the perceptual elements.

Figure 3 :

 3 Figure 3: Integration of the generation procedure in the verification, with splitbetween perception and reasoning: p learns to capture all the relevant parameters; r learns to respect the specification. Verifying φ 1 proves the perception module once and for all; verifying φ 2 can be done when the specification changes (e.g., for different driving rules).

Figure 4 :

 4 Figure 4: Example of inputs for the toy problem. White pixels represent obstacles. If they are in the top half of the image, no alert should be fired (first two exemples), while an alert should be fired if at least one lies in the (dashed) bottom half of the image (last two examples). 9x9 picture is depicted here for clarity.

  To provably assert adversarial robustness of a network, the goal is then to find the exact minimal distortion ε. Note that this property is local, tied to sample x0. A global adversarial robustness property could be phrased as: A deep neural network is globally εrobust if for any pair of samples(x1, x2) ∈ X 2 s.t. x1 -x2 p ε, C(x1) = C(x2).Verifying this global property is intractable, thus all the work has focused on local adversarial robustness.

	C(x0 + δ).
	t. δ p	ε, C(x0) =

Table 1 :

 1 Total number of possible samples for each N .

		N		3	5	7
		Train set size		461	500	500
		Test set size		51	500	500
	Total possible number of samples	512	2672	19650
	N	z3	CVC4	YICES		COLIBRI
	3	0.04s	0.08s	TIMEOUT	UNKNOWN
	5	30.2s	61.2s	TIMEOUT	TIMEOUT
	7	434s	TIMEOUT	TIMEOUT	TIMEOUT

Table 2 :

 2 Runtimes of solvers to answer UNSAT. TIMEOUT is one hour.

See https://www.lri.fr/ ˜gcharpia/camus/ for future release

https://onnx.ai/

Full list of supported operators is available at: https://github.com/onnx/onnx/blob/master/docs/Operators.md

https://github.com/onnx/tutorials/

https://github.com/alavrik/piqi

Acknowledgment

The work of J. Girard-Satabin was partially funded by a Ph.D. grant of the French Ministry of Defence, while Z. Chihani was partially funded by projects SPARTA and CPS4EU.