Marianne Le Bail
email: lebail@inapg.inra.fr

Philippe Verger

Thierry Doré

Jean-François Fourbet

Agnès Champeil

Renaud Ioos

Jean-Charles Leblanc

Simulation of consumer exposure to deoxynivalenol according to wheat crop management and grain segregation: Case studies and methodological considerations

Keywords: deoxynivalenol, consumer exposure, crop management, grain segregation, wheat, mycotoxin, Fusarium

niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Deoxynivalenol (DON) is a trichothecene occurring predominantly in cereal crops such as wheat, barley, oats, rye and maize. DON is produced mainly in the field by phythopathogenic or opportunistic species of fungi, which belong to the genus Fusarium. The toxicity of DON has been reviewed by WHO (1990), IARC (1993) and more recently by [START_REF] Jecfa | Safety evaluation of certain mycotoxins in food, Prepared by the Fiftysixth Meeting of the Joint FAO/WHO Expert Committee on Food Additives[END_REF]. The acute effects of DON (nausea, vomiting, diarrhoea, abdominal pain etc.) are difficult to distinguish from the gastrointestinal effects of microbes and have been attributed to the presence of DON at concentrations of 3-93 mg.kg -1 in grain for human consumption. DON has long-term toxic effects on the immune system, growth and reproduction. JECFA (2001) established a provisional maximum tolerable daily intake (PMTDI) of 1 µg.kg -1 body weight based on a two-year study in mice and a safety factor of 100. Now that the risk has been assessed, risk management procedures are required to ensure that consumers are not exposed to levels of toxin exceeding this PMTDI. One of the key issues involved in setting standards is consideration of both the variability of measurement of DON concentration in grain and annual variation of the mean concentration of this mycotoxin, which may vary by a factor of five [START_REF] Lauren | A survey of the natural occurence of fusarium mycotoxins in cereals in New Zealand in 1986-1989[END_REF][START_REF] Lauren | Mycotoxins in New Zealand maize: A study of some factors influencing contamination level in grain[END_REF]Scott, 1997). In practice, assuming a consumption of 175 g of wheat flour per day (GEMS Food, 2003) and a PMTDI of 60 µg/day for an adult weighing 60 kg, the maximum permissible concentration in wheat is 340 µg.kg -1 . However, it is difficult to respect this limit every year at field scale.

In recent years, agronomists have investigated the effects of farmers' practices on harvest quality. Most studies have focused on processing and organoleptic qualities (Habib et al., 1997;Virender, 2000); but recent studies have also considered aspects of food safety [START_REF] Doré | Pratiques agricoles et sécurité sanitaire des aliments en production végétale[END_REF][START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF]. They have also taken into account the fact that crops produced by farmers are often collected by firms acting as middle-men and that production is generally divided into several batches with different levels of quality for different agro-industrial markets. For cereals for example, two main strategies may be used to improve the quality of production in the supply area: improvements in cropping systems and the use of good agricultural practices at field scale and/or optimizing crop segregation in country grain elevators (Habib et al., 1997;[START_REF] Bail | Maîtrise de la qualité des céréales à l'échelle du bassin d'approvisionnement d'une entreprise de collecte-stockage. Approche agronomique[END_REF]. For example, it has been shown that in years in which DON levels exceed the limits set (Dowell et al., 1999), some US millers spend considerable amounts of money attempting to blend wheat to meet FDA (Food and Drug Administration) guidelines. It seems likely that it will become increasingly difficult in the future to use blending as an opportunist solution.

Research programmes carried out in the last 20 years have shown that wheat crop management affects the risk of contamination with Fusarium toxins at harvest. Preceding crop (Teich & Hamilton, 1985;[START_REF] Krebs | Fusarien-und Toxinbelastung des Weizens bei Direktsaat[END_REF][START_REF] Dill-Macky | The effect of previous crop residues and tillage on Fusarium head blight of wheat[END_REF], soil tillage (Teich & Hamilton, 1985;[START_REF] Krebs | Fusarien-und Toxinbelastung des Weizens bei Direktsaat[END_REF][START_REF] Schaafsma | The effect of agronomic practice on the accumulation of deoxynivalenol (DON) in winter wheat fields in Ontario, 1996-1997[END_REF][START_REF] Dill-Macky | The effect of previous crop residues and tillage on Fusarium head blight of wheat[END_REF][START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF][START_REF] Koch | Overcoming constraints of conservation tillage -Fusarium head blight in winter wheat[END_REF], fungicide application (Hardy et al., 1999;Simpson et al., 2001;Ioos et al., 2005), and wheat genotype [START_REF] Jenny | Getreidefusariosen: Sortenresistenz und Toxingehalte[END_REF]Young, 2002) have all been shown to affect the level of grain contamination with DON.

In this study, we aimed to use current knowledge to identify and to discuss the effects on consumer exposure to the toxin DON of changes in farming practices or in crop segregation practices at country grain elevators.

Materials and Methods

Defining crop management scenarios

These scenarios concern the changes in crop management on plots harvested in the supply area of a country elevator. Reduced tillage and no-tillage systems are being used in increasingly large areas throughout the world [START_REF] Derpsch | Situation of conservation agriculture in the world[END_REF]. As they are also known to have a most important impact on DON contamination, we therefore decided to assess the effects of the large-scale adoption of reduced tillage systems on consumer exposure to DON. Previous studies of wheat-grain DON contamination have reported differences between ploughed and no-tillage plots of various magnitudes, depending preceding crop (and year for the Champeil study): 1) zero to eight times more contamination in no-tillage plots than in ploughed plots [START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF]; 2) zero to three times more contamination in notillage plots [START_REF] Koch | Overcoming constraints of conservation tillage -Fusarium head blight in winter wheat[END_REF], mean of a two-year experiment); 3) zero to 1.5 times more contamination in no-tillage plots (Dill-Macky & Jones, 2000, mean of a three-year experiment with two levels of irrigation); 4) zero to eight times more contamination in notillage plots [START_REF] Krebs | Fusarien-und Toxinbelastung des Weizens bei Direktsaat[END_REF]. In all cases, the use of maize as the preceding crop led to the highest levels of grain contamination with DON. Thus, the adoption of no-tillage practices is likely to have a much greater effect on grain contamination with DON in maize cropping areas than in areas in which maize is rare. Based on these previous studies, no-tillage systems seem to display an average of three times more DON contamination of grain than ploughed systems.

We set up three scenarios, differing in the extent of conversion to no-tillage systems.

Scenario 1: all the plots of the area are ploughed (0% no-tillage) Scenario 2: half the area is ploughed (50% no-tillage) Scenario 3: all the plots are direct-drilled (100% no-tillage)

In each of these scenarios, all the wheat crops of the supply area are blended in a country grain elevator.

Defining segregation scenarios

The segregation of cereal crops in the crop supply system affects final batch quality, as batch contamination depends on mean grain contamination levels for each plot and plot production.

The basic segregation scenario (scenario a) corresponds to the blending of all the wheat crops from a given crop supply area. This basic scenario was used for crop management scenarios 1-3. Two alternative scenarios were studied to assess the effects of alternative supply strategies on consumer exposure. Such alternative strategies have already been successfully explored to face processing quality problems in cereal crops [START_REF] Bail | Maîtrise de la qualité des céréales à l'échelle du bassin d'approvisionnement d'une entreprise de collecte-stockage. Approche agronomique[END_REF]. The first alternative scenario (scenario b) was defined based on preceding crop, which is known to have an effect on DON contamination. We studied preceding crop (maize vs. other crops) as it is easier to obtain precise information for this variable than for soil tillage for real farms. A second alternative scenario (scenario c) was defined based on data for the real grain contamination of each plot: all the plots of the crop supply area with grain contamination levels below 1250 µg.kg -1 (discussed as a reference value in the EU for commercial transactions) were used to make a first batch, and the other plots were used to make a second batch. This strategy may be of value for limiting consumer exposure to DON, as grain contamination in farmers' fields within a crop supply area does not follow a normal distribution. The various data series available, recorded at national or regional level, show that this distribution seems to be log-normal (EU SCOOP 3.2.10;[START_REF] Dgal | Résultats des plans de surveillance 2002 et 2003 de la contamination des céréales françaises par certaines mycotoxines[END_REF][START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF].

If this is indeed the case, then the rare fields with very high toxin content in a sample of fields are likely to have a major effect on the mean contamination level of the sample, and the segregation of these fields from the other fields should have a large, positive effect, reducing consumer exposure. The three scenarios are summarised as follows:

Scenario a: all the wheat crops of the area are blended in a single batch A Scenario b: the plots of the area are divided into two classes according to preceding crop: batch M (maize) for the plots with maize as a preceding crop and batch O (others) for the other plots Scenario c: the plots of the area are divided in two classes according to the real grain contamination value to segregate two batches: batch L (low) for the plots with grain contamination levels lower than 1250 µg.kg -1 , and batch H (high) for the plots with contamination levels at or exceeding this threshold.

These three scenarios were applied to two distributions of grain contamination. For samples in DON was not detected (40% of censored data in dataset 1, and 5% in dataset 2) we assumed contamination at half the detection limit (not detected = ½ limit of detection (LOD)). For plots with DON levels below the quantification limit, we assumed contamination at half this limit (< limit of quantification (LOQ) = ½ LOQ) according to the IPCS/Gems/food recommendations of the 1995 Euro Workshop. The first data set used was a sample of 17 organic wheat fields harvested in the western part of the Paris Basin in 2002. Grain DON content for dataset 1 ranged from 15 to 2250 µg.kg -1 (15 is half the LOD for dataset 1), with 15 µg.kg -1 as a median value. The second data set was a sample of 21 wheat fields harvested in France in 2002. Grain DON content for dataset 2 ranged from 10 to 16685 µg.kg -1 (10 is half the LOD for dataset 2), with 135 µg.kg -1 as a median value. The mean values of grain contamination for the various batches, for both data sets, are shown in table 1. The means were estimated assuming equal yields for all plots.

Consumer exposure assessment

We assessed consumer exposure levels based on the combination of food consumption data and the various levels of grain contamination obtained with the six scenarios. Food consumption data are released by the French national and individual survey of food consumption, called -INCA‖, conducted in 1999 in 3003 subjects aged three years and over [START_REF]Enquête INCA individuelle et nationale sur les consommations alimentaires[END_REF]. In this survey, 2492 consumers of cereal products were identified. As deoxynivalenol is mostly found in wheat flour and we had contamination data only for the raw material used to produce flour, wheat grains we introduced a weighting factor to take into account the percentage of flour in cereal-based products. Consumption data for cereal-based products were then recalculated from the original consumption level reported in each food survey, by integrating the percentage of wheat flour needed to produce cereal-based products. We did this by assuming that 40% of biscuits, composite dishes (such as pizza or sandwiches) and breakfast cereals, 80% of bread, 95% of pasta and 100% of cereals (semolina, couscous etc.) are constituted by wheat flour [START_REF] Leblanc | Simulation of exposure to deoxynivalenol of French consumers of organic and conventional foodstuffs[END_REF]. The mean, median, 90 th , 95 th and 97.5 th percentiles of the distribution of daily wheat flour consumption were examined (Table 2). Estimated daily exposure was assessed by multiplying the mean concentrations of DON in wheat by the amount of food, correcting for the approximate proportion of flour in the food consumed daily by each consumer. The DON content in flour was determined using a corrective coefficient of 0.44 [START_REF] Neira | The effect of bakery processing on natural deoxynivalenol contamination[END_REF] applied to grain content, to take into account the effects of processing. This calculation provides a realistic estimation of exposure, for comparisons with long-term toxicological effects (JEFCA, 2001).

Results and discussion

We assessed consumer exposure to DON according to flour consumption and the three scenarios of crop management at a regional scale (Table 3), for a basic mean grain contamination of 400 µg.kg -1 in ploughed plots, which seems to be a realistic value according to published values for France lasting recent years [START_REF] Dgal | Résultats des plans de surveillance 2002 et 2003 de la contamination des céréales françaises par certaines mycotoxines[END_REF]. With this basic reference for scenario 1, the values used for scenarios 2 and 3 were 800, and 1200 µg.kg -1 , respectively.

For scenario 1, estimated exposure was lower than the PMTDI value, except for the 97.5th percentile. In contrast, even if only 50% of the area was subjected to reduced tillage practices (scenario 2), estimated mean exposure was always at least higher than the PMTDI, and estimated exposure was much higher than the PMTDI for direct-drilling over the whole area (scenario 3). Scenario 3 is unlikely in Europe, as deep tillage remains a good means of solving other agronomic problems, such as weed contamination. In contrast, scenario 2 is a realistic possibility, particularly in areas in which agronomic conditions (soil and crop rotation characteristics) do not necessitate annual deep tillage or in conditions in which deep tillage is not recommended, as a means of preventing soil erosion. However, our results should not be used as a direct argument for regulating crop management practices minimise the health risks.

The results obtained are highly sensitive to two factors: the basic grain contamination value for scenario 1, and the ratio (R) of grain DON content for ploughed fields to grain DON content for direct-drilled fields in the same area. Figure 1 presents the values obtained for exposure assessment under scenario 2 , in which grain contamination levels are 100 to 800 μg.kg -1 , and R ranges from 1 to 8, corresponding to the range reported in previous studies.

Even for the highest value of R, mean exposure did not exceed the PMTDI if the basic contamination level was 100 μg.kg -1 . In contrast, an R value of 2 was sufficient to attain this PMTDI for mean exposure for contamination values of 400 μg.kg -1 and over. We therefore need to determine why these two variablesbasic contamination level and R valuediffer so greatly from year to year and from place to place. Climate must account for some of this variation (see for example [START_REF] Blaney | Production of zearalenone, deoxynivalenol, nivalenol, and acetylated derivated by Australian isolates of Fusarium graminearum and F. pseudograminearum in relation to source and culturing conditions[END_REF][START_REF] Hope | Relationship between growth and mycotoxin production by Fusarium species, biocides and environment[END_REF] but other elements of the cropping system may also play a significant role. If farmers choose an appropriate set of techniques (for example, avoiding reduced tillage after a maize crop), they are likely to decrease both the R value and consumer exposure, even in scenario 2. However, it remains important to increase agronomic knowledge, to improve our understanding of contamination at the regional scale and variations in R values.

We estimated daily consumer exposure according to the three different grain segregation procedures, and for the two datasets (Table 4). For dataset 1 (Table 4a), scenario a resulted in levels of exposure below the PMTDI. Thus, the two alternative scenarios, b and c, gave no advantage over the simple blending of grain from all the fields. Moreover, each of these scenarios generated a batch resulting in estimated exposure levels exceeding 1 µg.day -1 .kg -1 body weight (batch M and batch H for scenarios b an c respectively), except for the mean exposure for scenario b.

The results for dataset 2 were very different (Table 4b). The simplest scenario (scenario a) systematically resulted in exposure levels exceeding the PMTDI. In this dataset, scenarios b and c were of great value as they made it possible to create batches (batch O for scenario b, batch L for scenario c) for which estimated exposure was below 1 µg.day -1 .kg -1 bodyweight, regardless of consumption value for scenario c, and up to the 95 th percentile for scenario b.

Scenario c is potentially the most useful as the creation of batch L not only reduces exposure, but also corresponds to a very large proportion of the fields (90% of the fields in the sample, whereas only 47% were retained in batch O from scenario b). However, scenario c is also much more difficult to implement based on current knowledge. It requires either the direct determination of grain DON content at harvest for each field or the development of a tool for use in the growing crop that can predict final DON content at harvest. Direct measurement remains costly, rapid methods (immunoassays, ELISA tests) are not still totally reliable but the accuracy of methods is generally limited by sampling errors because mycotoxins are not homogeneously distributed in batches [START_REF] Koch | State of the art of trichothecenes analysis[END_REF][START_REF] Boulc'h | Evaluation de la qualité sanitaire du blé : à propos des mycotoxines et des moyens de les détecter[END_REF]. However, no operational predictive tool similar to those for predicting other quality characteristics, such as grain protein content (Le [START_REF] Bail | Is it possible to forecast the grain quality and yield of different varieties of wheat from Minolta SPAD meter measurements[END_REF], is yet available for predicting DON content. In particular, there is no simple relationship between Fusarium head blight scores in the field and mycotoxin concentrations in grains [START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF]. Climatic models predicting DON contamination at harvest exist (Hooker et al., 2002;Hook & Schaafsma, 2003), but their validity domain seems not to be large enough to use them at a wide scale [START_REF] Champeil | Contribution à la compréhension des effets des systèmes de culture sur l'infection des cultures de blé tendre d'hiver par la fusariose et la contamination des grains par les mycotoxines associées[END_REF], for instance in Europe. In the absence of technical innovations in this area of predictive tools, scenario b seems satisfactory in terms of the decrease in exposure obtained but remains very costly as a significant fraction of the harvest may be lost to human consumption (as in dataset 2). Scenario b is based on the nature of the previous crop, in the absence of soil tillage data. If a more complete dataset, with more information on crop management and soil and climatic conditions were available, alternative grain segregation scenarios could be tested. Moreover, if informations about the climate were sufficient for at least predict the general pattern of the distribution of DON content that should occur in an area, it could be decided lately if segregation is useful.

Comparing the results obtained with the two datasets highlights the huge importance of the distribution of DON content. Low exposure values were obtained with scenario a for dataset 1, because a large number of fields accounted for very low levels of contamination, and no very high contamination value was observed. The results obtained for scenarios b and c with dataset 2 reflect very high levels of grain contamination for one field in which the preceding crop was maize. These two datasets should be considered only as examples for testing the methodology used. They show that with a grain DON content distribution of this type, the proportion of fields with very high levels of contamination has a major effect. It is therefore of prime importance to determine why some fields are so much more heavily contaminated than others. It would also be of interest to determine how our well two datasets reflect the diversity of real distributions in supply areas. Data are lacking in this domain. Finally, it would be useful to be able to anticipate the situation, based on climatic and cropping system conditions in a given area in a given year. If we knew in advance whether the situation was closer to that for dataset 1 or dataset 2, we would be able to decide whether crop segregation was necessary (scenarios b or c).

We estimated exposure using a coefficient of 0.44 for the decrease in DON content due to processing. This value is open to debate, as it is known to depend on the type of processing (JEFCA, 2001;[START_REF] Shollenberger | Fusarium toxins in wheat flour collected in an area of southwest Germany[END_REF][START_REF] Molinié | Les mycotoxines dans les céréales : les points importants de contrôle de la production au stockage, le devenir dans les produits dérivés[END_REF]. Our results would clearly be affected by changes in this coefficient, but such changes would not invalidate the approach.

Finally, the study demonstrates the difficulties associated with establishing a rational and general system combining cropping system and grain segregation procedure for preventing crop contamination by Fusarium toxins and consumer exposure. Dowell, F.E., Ram, M.S. and Seitz, L.M., 1999. Predicting scab, vomitoxin, and ergosterol in single wheat kernels using near-infrared spectroscopy. Cereal Chemistry 76, 573-576. Habib, R., Triboi, E., Génard, M. and Le Bail, M., 1997. La nutrition azotée des cultures et la qualité des produits, in B. Nicolardot (ed.). Gestion de l'azote dans les agrosystèmes, INRA, Paris, pp. 141-160. Hardy, A., Silva Fernandes, A., Speiijers, G., Hans, R., Delcour, M-P., Kuiper, H., Führ, F., Carere, A., Richard-Molard, D. and Thomas, M., 1999. Opinion on

  the relationship between the use of plant protection products on food plants and the occurrence of mycotoxins in food. Scientific Committee on Plants. European Commission Health and Consumer Protection directorate-General SCP/RESI/063-Final, Brussels, Belgium. Hooker, D.C. and Schaafsma, A.W., 2003. The DONcast model: using weather variables pre-and post-heading to predict deoxynivalenol content in winter wheat. Aspects of Applied Biology, 68, 117-122. Hooker, D.C., and Tamburic-Ilincic, L., 2002. Using weather variables pre-and postheading to prodict deoxynivalenol content in winter wheat. Plant disease 86(6), 611-619. IARC, 1993. Monographs on the evaluation of carcinogenic risks to humans, vol. 56, some naturally occurring substances, food items and constituents, heterocyclic aromatic amines and mycotoxins. World Health Organisation, Lyons, pp 397-433. International Programme on Chemical Safety/Gems/food Euro workshop on reliable evaluation of low level contamination of food, Appendix 5, Kulmbach, Germany, May 1995. Ioos, R., Belhadj, A., Menez M. and Faure, A., 2005. The effects of fungicides on Fusarium spp. and Microdochium nivale and their associated trichothecene mycotoxins in French naturally infected cereal grains. Crop Protection, in press.

Figure 1 :

 1 Figure1: Assessment of consumer exposure for various values of DON content in grains at a regional scale, and different values of the ratio (DON content in direct-drilled fields)/(DON content in ploughed fields), according to scenario 2.

Table 1 -

 1 Characteristics of the various batches for three scenarios of grain segregation and two datasets

Table 2 :

 2 Distribution of daily individual wheat flour consumption in the French population (g.j -1 )

Table 3 :

 3 Estimation of daily consumer exposure according to crop management practices, with 400 μg.kg -1 as the basic level of contamination for scenario 1.

Table 4 :

 4 Estimation of consumer exposure according to grain segregation procedure Figure1: Assessment of consumer exposure for various values of DON content in grains at a regional scale, and different values of the ratio (DON content in direct-drilled fields)/(DON content in ploughed fields), according to scenario 2.

Table 1 -

 1 Characteristics of the various batches for three scenarios of grain segregation and

	two datasets
	Scenario a		Scenario b			Scenario c
	Batch A	Batch M	Batch O	Batch H
	% plots	Mean DON	% plots	Mean DON	% plots	Mean DON	% plots	Mean DON	%
	concerned	contamination	concerned	contamination	concerned	contamination	concerned	contamination	conc
		(μg.kg -1 )		(μg.kg -1 )		(μg.kg -1 )		(μg.kg -1 )
	Data
	100	173	18	760	82	48	6	2250
	set 1
	Data
	100	1112	53	1765	47	393	10	9120
	set 2

Table 2 :

 2 Distribution of daily individual wheat flour consumption in the French population (g.j -1 )

	Mean	163
	Standard Deviation	85
	Median	147
	90th perc.	264
	95th perc.	313
	97.5th perc.	380

Table 3 :

 3 Estimation of daily consumer exposure according to crop management practices, with 400 μg.kg -1 as the basic level of contamination for scenario 1.

			90th percentile	95th percentile	97.5th percentile
		Mean exposure	exposure	exposure	exposure
		(µg.day -1 .kg -1 body	(µg.day -1 .kg -1 body	(µg.day -1 .kg -1 body	(µg.day -1 .kg -1 body
		weight)	weight)	weight)	weight)
	Scenario 1	0.5	0.8	0.9	1.1
	Scenario 2	1.0	1.5	1.8	2.2
	Scenario 3	1.4	2.3	2.8	3.3

Table 4 :

 4 Estimation of consumer exposure according to grain segregation procedure a. Dataset 1

				90th percentile	95th percentile	97.5 th percentile
			Mean exposure	exposure	exposure	exposure
			(µg.day -1 .kg -1	(µg.day -1 .kg -1	(µg.day -1 .kg -1	(µg.day -1 .kg -1
			body weight)	body weight)	body weight)	body weight)
	Scenario a	Batch A	0.2	0.3	0.4	0.5
		Batch M	0.9	1.5	1.7	2.1
	Scenario b	Batch O	0.1	0.1	0.1	0.1
		Batch H	2.7	4.4	5.2	6.3
	Scenario c	Batch L	0.1	0.1	0.1	0.1
	b. Dataset 2
						97.5 th
				90th percentile	95 th percentile	percentile
			Mean exposure	exposure	exposure	exposure
			(µg.day -1 .kg -1	(µg.day -1 .kg -1 body	(µg.day -1 .kg -1	(µg.day -1 .kg -1
			body weight)	weight)	body weight)	body weight)
	Scenario a	Batch A	1.3	2.2	2.6	3.1
		Batch M	2.1	3.4	4.1	4.9
	Scenario b	Batch O	0.5	0.8	0.9	1.1
		Batch H	10.9	17.7	20.9	25.4
	Scenario c	Batch L	0.3	0.5	0.6	0.7

Acknowledgements

This research was funded by the DGAL (French Ministry of Agriculture) and by INRA. We would also like to thank Alex Edelman & Associates for editorial advice.

Scott, P.M. 1997. Multi-year monitoring of Canadian grains and grain-based foods for trichothecenes and zearalenone. Food Addit. Contam.14, 333-339. Simpson, D.R., Weston, G.E., Turner, J.A., Jennings, P. and Nicholson, P., 2001


Anne Benoit

Aurélien Cavelan

Yves Robert

Hongyang Sun

Optimal resilience patterns to cope with fail-stop and silent errors

Keywords: resilience, fail-stop errors, silent errors, multilevel checkpoint, verification, optimal pattern

This work focuses on resilience techniques at extreme scale. Many papers deal with fail-stop errors. Many others deal with silent errors (or silent data corruptions). But very few papers deal with fail-stop and silent errors simultaneously. However, HPC applications will obviously have to cope with both error sources. This paper presents a unified framework and optimal algorithmic solutions to this double challenge. Silent errors are handled via verification mechanisms (either partially or fully accurate) and in-memory checkpoints. Fail-stop errors are processed via disk checkpoints. All verification and checkpoint types are combined into computational patterns. We provide a unified model, and a full characterization of the optimal pattern. Our results nicely extend several published solutions and demonstrate how to make use of different techniques to solve the double threat of fail-stop and silent errors. Extensive simulations based on real data confirm the accuracy of the model, and show that patterns that combine all resilience mechanisms are required to provide acceptable overheads.

I. INTRODUCTION

Fault-tolerance techniques are mandatory at extreme scale [START_REF] Cappello | Toward Exascale Resilience[END_REF], [START_REF] Cappello | Toward exascale resilience: 2014 update[END_REF]. The first source of problems is the frequent striking of fail-stop (unrecoverable) errors. This phenomenon is well understood: regardless of the robustness of each individual resource, aggregating too many resources will cause trouble at some point. Specifically, if the MTBF (Mean Time Between Failures) of each resource is ten years (a pretty optimistic figure for, say, a socket or a processor node), then the MTBF of a platform comprising one million of such resources is only five minutes 1 . The standard approach to cope with fail-stop errors is checkpoint and rollback recovery, and many protocols are available (see [START_REF] Hérault | Fault-Tolerance Techniques for High-Performance Computing, Computer Communications and Networks[END_REF] for a survey).

The second source of problems is the frequent striking of silent errors (or SDCs, for Silent Data Corruptions). This phenomenon is not so well understood, but has been recently identified as one of the major challenges for Exascale [START_REF] Cappello | Toward exascale resilience: 2014 update[END_REF]. There are several causes of silent errors, such as cosmic radiation, packaging pollution, among others. In contrast to a fail-stop error whose detection is immediate, a silent error is identified only when the corrupted data leads to an unusual application behavior. Such a detection latency raises a new challenge: if the error struck before the last checkpoint, and is detected after that checkpoint, then the checkpoint is corrupted and cannot be used for rollback. In order to avoid corrupted checkpoints, an effective approach consists in employing some verification mechanism and combining it with checkpointing [START_REF] Chen | Online-ABFT: An online algorithm based fault tolerance scheme for soft error detection in iterative methods[END_REF], [START_REF] Sao | Self-stabilizing iterative solvers[END_REF], [START_REF] Benoit | Efficient checkpoint/verification patterns[END_REF]. This verification mechanism can be general-purpose (e.g., based on replication [START_REF] Fiala | Detection and correction of silent data corruption for large-scale high-performance computing[END_REF] or even triplication [START_REF] Lyons | The use of triple-modular redundancy to improve computer reliability[END_REF]) or application-specific (e.g., based on Algorithm-based fault tolerance (ABFT) [START_REF] Bosilca | Algorithm-based fault tolerance applied to high performance computing[END_REF], on approximate re-execution for ODE and PDE solvers [START_REF] Benson | Silent error detection in numerical time-stepping schemes[END_REF], or on orthogonality checks for Krylov-based sparse solvers [START_REF] Chen | Online-ABFT: An online algorithm based fault tolerance scheme for soft error detection in iterative methods[END_REF], [START_REF] Sao | Self-stabilizing iterative solvers[END_REF]).

Verification mechanisms are typically costly; in fact, replication is the only alternative in an application-agnostic framework. Guaranteeing accurate and efficient detection of silent errors for scientific applications is one of the hardest challenges in extreme-scale computing [START_REF] Cappello | Toward exascale resilience: 2014 update[END_REF]. For many parallel applications, alternative techniques exist that are capable of detecting some but not all errors. We call these techniques partial verifications, while a guaranteed verification is capable of detecting all silent errors. One example is the lightweight SDC detector based on data dynamic monitoring [START_REF] Gomez | Detecting silent data corruption through data dynamic monitoring for scientific applications[END_REF], designed to recognize anomalies in HPC datasets based on physical laws and spatial interpolation. Similar fault filters have also been designed to detect silent errors based on time series predictions [START_REF] Berrocal | Lightweight silent data corruption detection based on runtime data analysis for HPC applications[END_REF]. Although not completely accurate, these partial verification techniques nevertheless cover a substantial amount of silent errors, and more importantly, they incur low overhead. These properties make them attractive candidates for designing more efficient resilient protocols.

Altogether, the detection of silent errors seriously complicates the design of resilience protocols. What is the best type of verification, either guaranteed or partial? And what is the best combination with checkpoints? To further complicate the story, silent errors naturally call for in-memory checkpointing, because a local copy of the data can be used after corruption has been detected. On the contrary, fail-stop errors require to store the checkpoints on remote stable storage (disks) because the whole memory content can be lost when such a failure strikes. Granted, multi-level checkpointing protocols have been designed for several years, but we face two major difficulties when combining fail-stop and silent errors. First, and to the best of our knowledge, the interplay of verification mechanisms with two types of checkpoints, in-memory and disk-based, has never been investigated. Second, the inherent detection latency of silent errors renders the problem quite different from traditional multi-level checkpointing, where each failure, regardless of its level, is detected immediately upon striking. In this work, after some quite technically involved derivations, we provide the optimal solution to the problem, either with guaranteed or with partial verifications. This was somewhat unexpected, because no optimal solution is known for two-level checkpointing with two levels of failstop errors; state-of-the-art protocols in that latter context rely on heuristics [START_REF] Di | Optimization of multi-level checkpoint model for large scale HPC applications[END_REF].

Our approach to solving the double problem of fail-stop and silent errors is to partition the execution of the application into periodic patterns, i.e., computational units that repeat over time. Each pattern ends with a guaranteed verification, an inmemory checkpoint and a disk checkpoint, so that errors do not propagate from a given pattern to the next one. Inside each pattern, there are several segments, each ending with a guaranteed verification and an in-memory checkpoint. In turn, each segment is partitioned into work chunks (possibly of different sizes) that are separated by partial verifications. See Figure 1(b) for an example with three segments and a total of six chunks. Several parameters should be given to fully characterize a pattern, namely the number of segments, and the number and size of each chunk inside each segment. The shape of a pattern is quite flexible, which enables us to provide the first model including two levels of checkpoints.

The main objective is to design an optimal pattern. Informally, consider a pattern P that includes W units of work (the cumulated size of all the chunks within the pattern). Without loss of generality, assume unit speed computation, so that we can speak of time or work interchangeably. In the presence of fail-stop or silent errors, the expected execution time of the pattern will be E(P): we have to take expectations, as the computation time is no longer deterministic. Note that E(P) > W for two reasons: the time spent in checkpoints and verifications, even if there is no error, and the time lost due to recovery and re-execution after an error. An optimal pattern is defined as the one minimizing the ratio E(P) W , or equivalently the ratio E(P)-W W = E(P) W -1. This latter ratio is the relative overhead paid for executing the pattern. The smaller this overhead, the faster the progress of the execution.

The main contributions of this work are the following: • The design of a detailed model based upon the computational patterns described above.

• The determination of the optimal pattern, first in some particular cases (one-chunk segments, one segment with multiple chunks), and then in the general case. The comprehensive list of results summarized in Table I extends and unifies many results from the literature.

• An extensive set of simulations that use data collected on real platforms, and extrapolate them to exascale platforms. The results confirm the accuracy of the model, as long as the MTBF is large enough in front of the resilience parameters. They also help assess the impact of each resilience mechanism, and show that patterns that combine all mechanisms (partial and guaranteed verifications and two checkpoint types) are required to provide acceptable overheads.

The rest of the paper is organized as follows. Section II introduces the model and notations. The following section shows how to determine the optimal pattern. We start with the simplest pattern (a single one-chunk segment) in Section III-A, extending Young and Daly's formula to two error sources. We discuss patterns with multiple one-chunk segments in Section III-B, and the most general pattern in Section III-C. Simulation results are presented in Section IV. Section V briefly discusses related work. Finally, Section VI provides concluding remarks and hints for future directions.

II. MODEL

Failure model. We consider a realistic scenario in large-scale systems, where hardware faults and silent data corruptions coexist. They are commonly referred to as fail-stop errors and silent errors in the literature. Since these two types of errors are caused by different sources, we assume that they are independent and that both occurrences follow a Poisson process with arrival rates λ f and λ s , respectively. The probability of having at least a fail-stop error during a computation of length w is given by p f = 1 -e -λ f w and that of having at least a silent error during the same computation is p s = 1 -e -λsw . We also assume that both error rates are in the same order, i.e., λ f = Θ(λ), and λ s = Θ(λ), where λ = λ f + λ s = 1/µ denotes the reciprocal of the platform MTBF µ while accounting for both types of failures.

Two-level checkpointing. To deal with both fail-stop and silent errors, resilience is provided through the use of a twolevel checkpointing scheme coupled with an error detection (or verification) mechanism. The protocol is enforced by a periodic computing pattern as discussed in Section I. When a fail-stop error strikes inside a pattern, the computation is interrupted immediately due to a hardware fault, so all the memory content is destroyed: we then roll back to the beginning of the pattern and recover from the last disk checkpoint (taken at the end of the previous pattern, or the initial data for the first pattern). On the contrary, when a silent error is detected inside a pattern, either by a partial verification or by a guaranteed one, we roll back to the nearest memory checkpoint in the pattern, and recover from the memory copy there, which is much cheaper than recovering from the last disk checkpoint. We enforce the following two properties for a pattern:

• A memory checkpoint is always taken immediately before each disk checkpoint. Since performing an I/O operation requires first flushing the data to a memory buffer, this process incurs little extra overhead and hence has a natural justification. Indeed, such a property has been enforced in some practical multi-level checkpointing systems [START_REF] Bautista-Gomez | FTI: High performance fault tolerance interface for hybrid systems[END_REF]. Similarly, when we recover from a disk checkpoint, we also restore the corresponding memory copy, which was destroyed due to the last fail-stop error.

• A guaranteed verification is always executed immediately before each memory checkpoint. Since storing a checkpoint can be expensive even for the memory, this property guarantees that all (memory and disk) checkpoints are valid, and hence avoids the need of maintaining multiple checkpoints, which is known to be difficult to recover from (one has to decide which checkpoint is valid, etc.). With this property, only one memory checkpoint and one disk checkpoint need to be maintained at any time during the execution of the application.

To simplify the presentation, we assume that errors only strike the computations, while verifications, memory copies, and I/O transfers are protected from failures. However, all optimality results hold without this assumption [START_REF] Benoit | Optimal resilience patterns to cope with fail-stop and silent errors[END_REF].

Notations. Let C D denote the cost of disk checkpointing, C M the cost of memory checkpointing, R D the cost of disk recovery, and R M the cost of memory recovery. Recall that when a disk recovery is done, we also need to restore the memory state, hence a cost R D + R M is paid. Also, let V * denote the cost of guaranteed verification and V the cost of a partial verification. The partial verification is also characterized by its recall, which is denoted by r and represents the proportion of detected errors over all silent errors that have occurred during the execution. If multiple partial verifications are available, our previous work [START_REF] Cavelan | Assessing the impact of partial verifications against silent data corruptions[END_REF], [START_REF] Bautista-Gomez | Which verification for soft error detection?[END_REF] has suggested to use the one with the largest accuracy-to-cost ratio, which is defined as

r 2-r / V V * +C M .
Note that the guaranteed verification can be considered as one with recall r * = 1 and hence with accuracyto-cost ratio C M V * +1. Since a partial verification usually incurs a much smaller cost yet has a reasonable recall, its accuracyto-cost ratio can be orders of magnitude (e.g., 100x) better than that of the guaranteed verification [START_REF] Gomez | Detecting silent data corruption through data dynamic monitoring for scientific applications[END_REF], [START_REF] Berrocal | Lightweight silent data corruption detection based on runtime data analysis for HPC applications[END_REF]. Hence partial verifications are highly attractive for detecting silent errors, and we make use of them between memory checkpoints in the pattern.

For clarity, we refer to the computation between two consecutive memory checkpoints as a segment, and refer to the computation between two consecutive verifications as a chunk. Formally, a pattern P(W, n, α, m, β 1 , . . . , β n ) is defined by the following parameters: • W : total amount of computation (or work) of the pattern.

• n: number of memory checkpoints inside the pattern (also number of segments within the pattern).

• α = [α 1 , α 2 , . . . , α n ]: proportion of the segment sizes, i.e., α i = w i /W , where w i denotes the amount of work in the i-th segment of the pattern. Hence

n i=1 α i = 1. • m = [m 1 , m 2 , .
. . , m n ]: number of verifications inside each segment (also number of chunks in that segment).

• β i = [β i,1 , β i,2 , . . . , β i,mi ] ∀i = 1, 2, .
. . , n: proportion of the chunk sizes in the segments, i.e., β i,j = w i,j /w i , where w i,j denotes the amount of work in the j-th chunk of the i-th segment. Hence mi j=1 β i,j = 1 for all i = 1, 2, . . . , n. The simplest pattern is illustrated in Figure 1(a), and it consists of a single segment (n = 1, W = w 1 ), which comprises a single chunk (m = [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF]). By construction, this chunk is followed by a guaranteed verification, followed immediately by a memory checkpoint and a disk checkpoint. With our notations, this pattern is denoted as P(W, 1, [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ), or P D (only disk checkpoints, which are always preceded by a guaranteed verification and a memory checkpoint). Figure 1(b) shows a more complicated pattern, with three segments. The first segment has three chunks, the second segment has one chunk, and the third segment has two chunks. Therefore, if a silent error is detected by the guaranteed verification at the end of the second segment, it is possible to recover from the memory checkpoint preceding it, rather than starting the whole pattern again. Moreover, silent errors, if occurred in the first and third segments, may be detected earlier thanks to the additional partial verifications.

Objective. The objective is to find a pattern that minimizes the expected execution time of the application. Let W base denote the base execution time of an application without any overhead due to resilience techniques (without loss of generality, we assume unit-speed execution). Since the execution is divided into periodic patterns, defined by P(W, n, α, m, β 1 , . . . , β n ), let E(P) be the expected execution time of the pattern. For large jobs, the expected makespan W final of the application when taking failures into account can then be approximated by

W final ≈ E(P) W • W base . Now, define H(P) = E(P)
W -1 to be the expected overhead of the pattern. We obtain W final ≈ W base +H(P)•W base . Thus, minimizing the expected makespan is equivalent to minimizing the pattern overhead H(P). Hence, we will focus on minimizing the overhead in this paper.

Time W V * CM CD V * CM CD (a) Pattern P D = P(W, 1, [1], [1], [1] ) Time 1st segment 2nd segment 3rd segment V * CM CD V V V * CM V * CM V V * CM CD (

III. OPTIMAL PATTERNS

A. Pattern P D , or revisiting Young and Daly

In this section, we revisit Young [START_REF] Young | A first order approximation to the optimum checkpoint interval[END_REF] and Daly [START_REF] Daly | A higher order estimate of the optimum checkpoint interval for restart dumps[END_REF] on computing the optimal periodic checkpointing interval, and extend their formula to include both fail-stop and silent errors. The result on the order of the optimal interval and the observations established in this case will pave the way for the subsequent analysis on more advanced patterns.

The classical formula by Young and Daly gives the optimal disk checkpointing interval without considering silent errors, thus does not include verifications and memory checkpoints in the pattern. To cope with both fail-stop and silent errors, we analyze the pattern P D = P(W, 1, [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ), which contains one single segment with a unique chunk followed by a guaranteed verification, a memory checkpoint and a disk checkpoint (see Figure 1(a)). Obviously, the only parameter to determine is the work length W , which is commonly referred to as the checkpointing period in the literature. The following proposition shows the expected execution time of a pattern with a fixed work length.

Proposition 1. The expected execution time of a given pattern P(W, 1, [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) is

E(P) = W + V * + C M + C D + λ s + λ f 2 W 2 + λ s W (V * + R M ) + λ f W (R M + R D ) + O(λ 2 W 3 ) . (1)
Proof. Let p f = 1 -e -λ f W and p s = 1 -e -λsW denote the probabilities of having at least one fail-stop error and at least one silent error, respectively, in the pattern. The expected execution time obeys the following recursive formula:

E(P) = p f E(T lost ) + R D + R M + E(P) + (1 -p f ) W + V * + p s (R M + E(P)) + (1 -p s )(C M + C D ) , (2)
where E(T lost ) denotes the expected time loss during the execution of the pattern if a fail-stop error strikes. Equation ( 2) can be interpreted as follows: if a fail-stop error occurs, we lose E(T lost ) time, perform a recovery from both disk and memory, and then re-execute the pattern (Line 1). If no failstop error strikes during the execution, we run the guaranteed verification to detect silent errors, which if indeed occurred involves a memory recovery only followed by a re-execution (Line 2). Otherwise, if no silent error strikes either, we can proceed with the memory and disk checkpointing (Line 3). To derive the expected execution time, we need to compute E(T lost ), which can be expressed as follows:

E(T lost ) = ∞ 0 xP(X = x|X < W )dx = 1 P(X<W ) W 0 xP(X = x)dx,
where P(X = x) denotes the probability that a fail-stop error strikes at time x. By definition, we have P(X = x) = λ f e -λ f x and P(X < W ) = 1 -e -λ f W . Integrating by parts, we get

E(T lost ) = 1 λ f - W e λ f W -1 . (3)
Substituting Equation (3) into Equation ( 2) and solving the recursion, we obtain

E(P) = e (λ f +λs)W -e λsW λ f + e λsW V * + C D + C M + e (λ f +λs)W -e λsW R D + e (λ f +λs)W -1 R M .
By approximating

e λx = 1 + λx + λ 2 x 2 2
up to the secondorder term, we can further simplify the expected execution time, which turns out to be given by Equation (1). Theorem 1. A first-order approximation to the optimal work length in pattern P(W, 1, [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) is given by

W * = V * + C M + C D λ s + λ f 2 . ( 4
)
The optimal expected overhead in this case is

H * (P) = 2 λ s + λ f 2 (V * + C M + C D ) + O(λ) . ( 5
)
Proof. From the result of Proposition 1, the expected overhead of the pattern can be computed as

H(P) = V * + C M + C D W + λ s + λ f 2 W + λ s (V * + R M ) + λ f (R M + R D ) + O(λ 2 W 2 ) . (6)
Assuming that the platform MTBF µ = 1/λ is large in front of the resilience parameters, consider the first two terms of H(P) (Line 1 of Equation ( 6)): the overhead is minimal when the pattern has length W = Θ(λ -1/2 ), and in that case both terms are in the order of λ 1/2 , so that we have

H(P) = Θ(λ 1/2 ) + O(λ).
Indeed, the last term O(λ 2 W 2 ) becomes also negligible compared to Θ(λ 1/2 ). Hence, the optimal pattern length W * can be obtained by balancing the first two terms of Equation ( 6), which gives rise to Equation (4). Then, by substituting W * back into H(P), we get the optimal expected overhead as shown by Equation ( 5).

When only fail-stop errors exist, we retrieve the classical formula by Young [START_REF] Young | A first order approximation to the optimum checkpoint interval[END_REF] and Daly [START_REF] Daly | A higher order estimate of the optimum checkpoint interval for restart dumps[END_REF], which is given by W * = 2C D /λ f . When there are only silent errors, the optimal work length is given by

W * = (V * + C M )/λ s .
We observe from Theorem 1 that the optimal work length W * of a pattern is in the order of λ -1/2 and the optimal overhead H * (P) is in the order of λ 1/2 . Theorem 1 also shows that we can express the expected execution overhead of a pattern as

H(P) = oef W + o rw W + O(λ)
, where o ef and o rw are two key parameters that characterize two different types of overheads in the execution, and they are defined below. Definition 1. For a given pattern, o ef denotes the error-free overhead due to the resilience operations (e.g., verification, checkpointing), and o rw denotes the re-executed work overhead, in terms of the fraction of re-executed work due to errors.

In the simple pattern P(W, 1, [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) analyzed above, these two overheads are given by

o ef = V * + C M + C D and o rw = λ s + λ f
2 , respectively. The optimal pattern length and the optimal expected overhead can thus be expressed as

W * = o ef o rw and H * (P) = 2 √ o ef • o rw + O(λ) . ( 7
)
We see that minimizing the expected execution overhead H(P) of a pattern becomes equivalent to minimizing the product o ef • o rw up to the dominating term. Intuitively, including more resilience operators reduces the re-executed work overhead but adversely increases the error-free overhead, and vice versa. This requires a resilience protocol to find the optimal trade-off between o ef and o rw . We will make use of this observation in the subsequent sections to derive the optimal patterns in more complicated protocols.

B. Pattern

P DM = P(W, n, α, [1, . . . , 1], [1], . . . , [1] )
We first consider a pattern with multiple segments, but each segment has only one chunk. In other words, Time w1 w2 wn the protocol performs multiple memory checkpoints between two disk checkpoints but without any intermediate verification. Figure 2(a) depicts the pattern P DM = P(W, n, α, [1, . . . , 1], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], . . . , [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) in this protocol. The goal is to determine the work length W , the number of memory checkpoints n, and the relative lengths of the segments α in the pattern. The following proposition shows the expected execution time of a pattern when these parameters are fixed.

• • • • • • V * CM CD V * CM V * CM V * CM V * CM CD (a) Pattern P DM = P(W, n, α, [1, . . . , 1], [1], . . . , [1] ) Time w1,1 w1,2 w1,m • • • • • • V * CM CD V V V V * CM CD (b) Pattern P DV = P(W, 1, [1], [m], β ) Time w1,1 w1,m 1 wn,1 wn,m n • • • • • • • • • • • • • • • • • • V * CM CD V V V * CM V * CM V V V * CM CD (
Proposition 2. The expected execution time of a given pattern P(W, n, α, [1, . . . , 1], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], . . . , [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) is

E(P) = W + n(V * + C M ) + C D + λ s n i=1 α 2 i + λ f 2 W 2 + O( √ λ) . (8)
Proof. Define E i as the expected time to execute the i-th segment of the pattern up to the memory checkpoint at the end of the segment. We first show the following result:

E i = w i +V * +C M +λ s w 2 i +λ f w 2 i 2 + i-1 k=1 w k w i +O( √ λ) ,
where w i = α i W denotes the work length of the i-th segment. We prove the above claim by induction on i. For the base case, the problem is reduced to the simple pattern shown in Section III-A, except that there is no disk checkpoint. Since we know from Theorem 1 that the work length of a pattern is in the order of λ -1/2 , we get the following result from Proposition 1:

E 1 = w 1 + V * + C M + λ s w 2 1 + λ f 2 w 2 1 + O( √ λ)
. Suppose the claim holds up to E i-1 . Then, E i can be expressed recursively as follows:

E i = p f i E(T lost i )+R D +R M + i-1 k=1 E k +E i +(1 -p f i ) w i +V * +p s i (R M +E i )+(1 -p s i )C M
, where E(T lost i ) denotes the expected time loss during the execution of segment i when a fail-stop error strikes, which according to Equation ( 3) is given by E(

T lost i ) = 1 λ f -wi e λ f w i -1
, and p f i = 1 -e -λ f wi and p s i = 1 -e -λswi denote the probabilities of having at least one fail-stop error and at least one silent error in segment i, respectively. By following the reasoning of the proof of Proposition 1, we obtain:

Ei = wi +V * +CM +λsw 2 i + λ f 2 w 2 i +λ f wi i-1 k=1 E k +O( √ λ) = wi +V * +CM +λsw 2 i + λ f 2 w 2 i +λ f wi i-1 k=1 (w k +O(1))+O( √ λ) = wi +V * +CM +λsw 2 i +λ f w 2 i 2 + i-1 k=1 w k wi +O( √ λ) .
Now, we can compute the expected execution time of the pattern by summing up all the E i 's as follows:

E(P) = n i=1 Ei +CD = n i=1 wi +n(V * +CM )+CD +λs n i=1 w 2 i +λ f n i=1 w 2 i 2 + i-1 k=1 w k wi +O( √ λ) = W +n(V * +CM )+CD + λs n i=1 α 2 i + λ f 2 W 2 +O( √ λ) , since n i=1 w 2 i + 2 i-1 k=1 w k w i = W 2 .
Theorem 2. A first-order approximation to the optimal parameters in pattern P(W, n, α, [1, . . . , 1], [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF], . . . , [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] ) is given by

α * i = 1 n * for 1 ≤ i ≤ n * , W * = n * (V * +C M )+C D λs n * + λ f 2
, and n * is either max(1, n * ) or n * , where n * = 2λs

λ f • C D V * +C M . The optimal expected overhead in this case is H * (P) = 2 λ s (V * + C M ) + 2λ f C D + O(λ).
Proof. (Sketch) Given the number of segments n and subject to n i=1 α i = 1, we know that n i=1 α 2 i is minimized when α i = 1 n for all 1 ≤ i ≤ n. Hence, we can derive the two types of overheads from Proposition 2 as follows:

o ef = n(V * + C M ) + C D and o rw = λ s n + λ f 2 .
For a given n, we can then retrieve the value of the optimal work length W * = oef orw . Now, minimizing

F (n) = o ef •o rw = n(V * + C M ) + C D λs n + λ f
2 , we get the optimal value of n * as shown in the theorem. Since the number of segments can only be a positive integer, and F (n) is a convex function of n, the optimal integer solution is either max(1, n * ) or n * , whichever one leads to a smaller value of F (n). Substituting all these values back into

H * (P) = 2 √ o ef • o rw + O(λ), we
obtain the optimal expected overhead.

C. General patterns

Due to lack of space, we refer to [START_REF] Benoit | Optimal resilience patterns to cope with fail-stop and silent errors[END_REF] for a detailed (and technical) proof for the characterization of the optimal pattern in the general case. We report all results in Table I, which contains six optimal patterns. In addition to P D (one singlechunk segment, Section III-A) and P DM (several one-chunk segments, Section III-B), we have: • P DV , the optimal pattern with a single segment that has multiple chunks in it. Each chunk ends with a partial verification, except the last one, which ends with a guaranteed verification followed by a memory checkpoint and a disk checkpoint (see Figure 2(b)). In the optimal solution, all chunks have same length, except the first and last one, which are larger.

Pattern W * n * m * H * (P) PD V * +CM +CD λs+ λ f 2 - - 2 λs + λf 2 (V * + CM + CD) PDV * m * V * +CM +CD 1 2 (1+ 1 m * )λs+ λ f 2 - λs λs+λf • CM +CD V * 2(λs + λf )CM + CD + √ 2λsV * PDV (m * -1)V +V * +CM +CD 1 2 1+ 2-r (m * -2)r+2 λs+ λ f 2 - 2 -2 r + λs λs+λf • 2-r r V * +CM +CD V -2-r r 2(λs + λf ) V * -2-r r V + CM + CD + 2λs 2-r r V PDM n * (V * +CM )+CD λs n * + λ f 2 2λs λf • CD V * +CM - 2 λs(V * + CM ) + 2λf CD PDMV * n * m * V * +n * CM +CD 1 2 (1+ 1 m * ) λs n * + λ f 2 λs λf • CD CM CM V * 2λf CD + √ 2λsCM + √ 2λsV * PDMV n * (m * -1)V +n * (V * +CM )+CD 1 2 1+ 2-r (m * -2)r+2 λs n * + λ f 2 λs λf • CD V * -2-r r V +CM 2 -2 r + 2-r r V * +CM V -2-r r 2λf CD + 2λs V * -2-r r V + CM + 2λs 2-r r V
• P DV * , the variant of P DV with only guaranteed verifications. For this variant, all chunks have same length.

• P DM V , the optimal pattern in the general case, with several multiple-chunk segments (see Figure 2(c)). In the optimal solution, all segments are identical. In each segment, all chunks have the same length, except the first and last one, which are larger.

• P DM V * , the variant of P DM V with only guaranteed verifications. For this variant, all chunks have the same length.

IV. PERFORMANCE EVALUATION

In this section, we conduct a set of simulations whose goal is twofold: (i) corroborate the theoretical study, and (ii) assess the relative efficiency of each checkpoint and verification type under realistic scenarios. We rely on simulations to evaluate the performance of the patterns at extreme scale, and we instantiate the model with three scenarios. In the first scenario, we compare all patterns using real parameters from the literature. The second scenario is a weak scaling experiment, whose purpose is to assess the scalability of the approach on increasingly large platforms. In the last scenario, we study the impact of varying error rates on the overhead. The simulator code is publicly available at http://graal.ens-lyon.fr/ ∼ yrobert/two-level.zip, so interested readers can experiment with it and build relevant scenarios of their choice.

A. Simulation setup

We make several assumptions on the input parameters. First, we assume that the recovery cost is equivalent to the corresponding checkpointing cost, i.e., R D = C D and R M = C M . This is reasonable because writing a checkpoint and reading one typically takes the same amount of time. Then, we assume that a guaranteed verification must check all the data in memory, making its cost in the same order as that of a memory checkpoint, i.e., V * = C M . Furthermore, we assume partial verifications similar to those proposed in [START_REF] Berrocal | Lightweight silent data corruption detection based on runtime data analysis for HPC applications[END_REF], [START_REF] Gomez | Detecting silent data corruption through data dynamic monitoring for scientific applications[END_REF], [START_REF] Gomez | Detecting and correcting data corruption in stencil applications through multivariate interpolation[END_REF], with very low cost while offering good recalls. In the following, we set V = V * 100 and r = 0.8. All these choices are somewhat arbitrary and can easily be modified in the simulator; we believe they represent reasonable values for current and next-generation HPC applications.

The simulator generates errors following an exponential distribution of parameter λ f for fail-stop errors and λ s for silent errors. An experiment goes as follows. We feed the simulator with the description of the platform, consisting of the parameters λ f , λ s , C D and C M (since the other parameters can be deduced from the above assumptions). For each pattern, we compute the optimal length W * , the optimal overhead H * (P), as well as the optimal number of memory checkpoints n * and the optimal number of verifications m * (when applicable), using the formulas from Table I. The total amount of work for the application is set to that of 1000 optimal patterns, and the simulator runs each experiment 1000 times. For each pattern, it outputs the simulated overhead, the simulated number of disk checkpoints, memory checkpoints, verifications, disk recoveries and memory recoveries, by averaging the values from the 1000 runs.

B. Assessing resilience mechanisms on real platforms

In the first scenario, we assess the performance of the six optimal patterns shown in Table I on four different platforms with real parameter settings.

1) Platform settings: Table II presents the four platforms used in this experiment and their main parameters. These platforms have been used to evaluate the Scalable Checkpoint/Restart (SCR) library by Moody et al. [START_REF] Moody | Design, Modeling, and Evaluation of a Scalable Multi-level Checkpointing System[END_REF], who provide accurate measurements for λ f , λ s , C D and C M using real applications. Note that the Hera platform has the worst error rates, with a platform MTBF of 12.2 days for fail-stop errors and 3.4 days for silent errors. In comparison, and despite its higher number of nodes, the Coastal platform features a platform MTBF of 28.8 days for fail-stop errors and 5.8 days for silent errors. In addition, the last platform uses SSD technology for memory checkpointing, which provides more data space, at the cost of higher checkpointing costs.

2) Pattern overhead: The first row of Figure 3 presents, for each pattern, the predicted overhead H * (P) (in blue) versus the simulated one (in yellow) on each platform. Remember that the derivation of the expected overhead uses first-order approximation, ignoring some low-order terms in the computation. As a result, the predicted overhead, being a little optimistic, is always slightly less than the simulated one. However, the difference between the two is very small (less than 1%), which validates the model quite satisfactorily. Overall, the overhead oscillates between 4% and 7% on Hera, where checkpoints are relatively cheaper, to just over 15% on Coastal SSD, where checkpoints are more expensive. Regardless of the platform, the more advanced patterns always result in smaller overheads. In particular, we observe a significant difference between the first three patterns (P D ,P DV * ,P DV ), which use single-level checkpointing, and the last three patterns (P DM ,P DM V * ,P DM V ), which use twolevel checkpointing. The gap is more visible for Atlas (5%) and Coastal (4%), where the difference between the costs of a disk checkpoint and a memory checkpoint is larger, thus making memory checkpoints more valuable.

3) Pattern periods: The second row of Figure 3 shows the work lengths (periods) of the patterns on each platform. Single-level patterns are associated with shorter periods (around 3 hours on Hera and 10 hours on Coastal), as opposed to the longer periods shown by the two-level patterns (around 8 hours on Hera and 20 hours on Coastal). Indeed, when a fail-stop error strikes, the only choice is to recover from the last disk checkpoint, losing all the work done so far. In that case, a short period helps to mitigate the amount of time lost. However, silent errors are more prominent on these platforms and when a silent error occurs, two-level patterns can recover from an intermediate memory checkpoint instead. Not only does that provide a faster recovery, but also it does not require the application to restart from the very beginning of the pattern. As a result, disk checkpoints are only used for fail-stop errors, and a longer period is favored in order to accommodate more but cheaper intermediate memory checkpoints.

4) Pattern checkpoints and verifications:

The third row of Figure 3 presents the average number of disk checkpoints, memory checkpoints and verifications taken by each pattern. We take all checkpoints and verifications into account, including the ones performed in recoveries and re-executions. Since a partial verification is much cheaper than a guaranteed one, the two patterns that are allowed to use them (P DV and P DM V ) tend to take full advantage of this mechanism. On Hera, P DV generates an average of 13 verifications per hour, which is slightly more than its two-level counterpart (P DM V ), which generates 12 verifications per hour. On Coastal, there are more than 20 verifications per hour for P DV and 19 for P DM V . For the two-level patterns, whose periods are longer, the disk checkpointing frequencies become lower. However, their memory checkpointing frequencies are higher, because the cheaper memory checkpoints are favored in these twolevel schemes in order to better protect the application from silent errors. Lastly, we observe that the Coastal SSD platform requires very few verifications and memory checkpoints. This is because the cost of a memory checkpoint is much larger on this platform (180s) as opposed to the costs on other platforms (15.4s on Hera and 4.5s on Coastal).

5) Pattern recoveries: The last row of Figure 3 shows the number of recoveries performed per day by each pattern on each platform. The number of disk recoveries follows closely the fail-stop error rate of a given platform, and it is not affected by the pattern used. Indeed, when a fail-stop error strikes, a disk recovery is performed regardless of the pattern. On Hera, we observe 0.083 disk recovery per day on average, translating to approximately one recovery every 12 days, which is in accordance with the platform MTBF of 12.2 days for failstop errors. The same applies to Atlas and Coastal, which show respectively 0.044 and 0.034 disk recoveries per day on average (equivalent to a platform MTBF of 22 days and 29 days). The number of memory recoveries is more complicated to analyze, because a memory recovery is not performed immediately after the occurrence of a silent error. Instead, it is performed only when an alarm is raised by a verification, or when a fail-stop error strikes. In both cases, more than one silent error could have occurred before the memory recovery. In the latter case, a memory recovery is triggered right after a disk recovery, possibly without any silent error. In general, the memory recovery frequency could well depend on the pattern used, which explains the slight difference under different patterns. Nevertheless, the simulation results show that the silent error rate is a good indicator of the memory recovery frequency. For instance, on Hera, we observe 0.285 memory recovery per day on average, which is approximately one memory recovery every 3.5 days. This is very close to the MTBF of 3.4 days for silent errors.

C. Weak scaling experiment

We now present the results of a weak scaling experiment to assess the scalability of the model. This experiment is based on the Hera platform, whose disk checkpoint cost is the closest to state-of-the-art platforms (5 minutes).

1) Platform settings: We first calculate the MTBF of one computing node, which is 8.57 years for fail-stop errors and 2.4 years for silent errors. The platform MTBF is obtained by dividing the per-node MTBF by the number of nodes. For example, when 2 17 nodes are used, the MTBF decreases to 2064s for fail-stop errors and 577s for silent errors. Under weak scaling, the problem size grows linearly with the number of nodes, so the time to perform a memory checkpoint C M remains constant. In addition, we make the optimistic 2) Results: Figure 4 presents the simulation results for the simplest pattern P D and the most advanced pattern P DM V . We can see that the overheads remain acceptable up to 2 15 = 32768 nodes, which are 100% for P D and 64% for P DM V . Beyond that, the overheads increase drastically for both patterns, eventually exceeding 500% for 2 18 = 262144 nodes. However, compared to the simple pattern P D , the twolevel pattern P DM V improves the overhead by a few percent on 256 nodes up to over 150% on 2 18 nodes.

We also observe the difference between the simulated overhead and the predicted one, which starts negligible for a small number of nodes but reaches more than a factor of three for 2 18 nodes. The reason is the use of first-order approximation to compute the predicted overhead, which is only accurate when the platform MTBF is large in front of the other parameters. Obviously, this is no longer the case for a large number of nodes. For instance, when the number of nodes reaches 10 5 (almost 2 17 nodes), the MTBF of the whole platform reduces to less than 10 minutes, which is in the same order as the pattern period. At this point, the application experiences more than half a dozen errors per hour. In order to minimize the impact of errors, the pattern P DM V places a large number of verifications and more than 10 memory checkpoints per hour. As a result, a lot of time is wasted on resilience operations. When the error rate is this high, however, there is not much flexibility in the optimization, and no pattern is able to offer satisfying performance.

Similar results are also obtained when we repeat the weak scaling experiment with a disk checkpointing cost of 90s instead of 300s to account for improved disk technology [START_REF] Benoit | Optimal resilience patterns to cope with fail-stop and silent errors[END_REF].

D. Impact of error rates

Finally, we study the impact of the error rates on the performance of the patterns. Again, we focus on the Hera platform but scale its number of nodes to 10 5 . We vary the error rates λ f and λ s with respect to their nominal values while keeping the other parameters fixed.

The first row of Figure 5 presents the impact of λ f and λ s on the simulated overheads of two patterns P D and P DM V . For the P DM V pattern, the overhead is affected more by the fail-stop errors than by the silent errors. This is because the intermediate memory checkpoints better protect the application from silent errors. On the other hand, the overhead of the single-level pattern P D is affected more by the silent errors, because silent errors have a much higher rate. The figure also shows the difference between the overheads of the two patterns, which is small when most errors are fail-stop, due to the relatively small rate. However, when the silent error rate increases, the two-level pattern achieves a much better performance than the single-level pattern, by saving up to 200% on the overhead.

The second row of Figure 5 presents the impacts of λ f and λ s separately on the periods and checkpointing frequencies of the two patterns. First, when the silent error rate is fixed at its nominal value, the period of P D remains constant with varying λ f , while the period of P DM V decreases as λ f increases. This is because the high silent error rate has already driven the period of P D very low (< 10 minutes), so increasing the fail-stop error rate has a limited impact. On the other hand, the period of P DM V is primarily driven by the fail-stop error rate, so it decreases quickly, allowing more disk checkpoints to be taken. In addition, the number of checkpoints successfully taken in an hour remains stable for both patterns. Since the period of P DM V decreases while the period of P D remains constant, it implies degraded performance for the two-level pattern and stable performance for the single-level one, corroborating the previous analysis. The role is reversed when the fail-stop error rate is fixed at the nominal value and the silent error rate is changed. Since the P DM V pattern is equipped with more memory checkpoints and verifications, silent errors have little impact on its period. On the contrary, the period of P D decreases in order to detect silent errors earlier, which is the only way to protect the application from increased silent error rate. Lastly, the number of memory checkpoints performed by P DM V increases with the silent error rate in order to compensate for the fixed number of disk checkpoints. For the P D pattern, the checkpointing frequency remains the same, implying degraded performance with decreased period.

E. Summary

From the simulation results, we conclude that the first-order approximation for the resilience patterns provides an accurate performance model for systems with up to tens of thousands of nodes. Overall, the complex pattern that combines all resilience mechanisms offers significantly better performance, improving the base pattern by up to 150% in the execution overhead. The findings are consistent on different platforms and with varying error rates. The results nicely corroborate the analytical study, and demonstrate the benefit of using twolevel patterns for dealing with both fail-stop and silent errors.

V. RELATED WORK

Due to lack of space, we refer to [START_REF] Benoit | Optimal resilience patterns to cope with fail-stop and silent errors[END_REF] for a detailed list of related work on multi-level checkpointing and silent error detection. Here we only outline differences with our previous work. In a nutshell, our previous work investigates the design of special patterns. Aupy et al. [START_REF] Aupy | On the combination of silent error detection and checkpointing[END_REF] analyzed two simple patterns: one with k checkpoints and one guaranteed verification, and the other with k verifications and one checkpoint. Benoit et al. [START_REF] Benoit | Assessing general-purpose algorithms to cope with fail-stop and silent errors[END_REF] studied the latter pattern and gave explicit formulas to accommodate both fail-stop and silent errors. The idea of interleaving p checkpoints and q verifications has also been explored in [START_REF] Benoit | Efficient checkpoint/verification patterns[END_REF] to achieve more optimized computing patterns. The first analysis of a pattern using partial verification for silent error detection was given by Cavelan et al. [START_REF] Cavelan | Assessing the impact of partial verifications against silent data corruptions[END_REF]. This analysis has been recently extended to the case with multiple partial verifications [START_REF] Bautista-Gomez | Which verification for soft error detection?[END_REF]. All these results apply to a single level of checkpointing only, which considerably simplifies the design of optimal solutions. To the best of our knowledge, this work is the first to investigate the combination of memory checkpoints, disk checkpoints, partial verifications and guaranteed verifications.

Finally, we stress that this work is along the same direction as multi-level checkpointing, but the two levels we propose target different error sources, namely, fail-stop errors and silent errors. This dramatically changes the computation of the expected re-execution time, because we do not have to distinguish which error type strikes first. Moreover, as in Young [START_REF] Young | A first order approximation to the optimum checkpoint interval[END_REF] and Daly [START_REF] Daly | A higher order estimate of the optimum checkpoint interval for restart dumps[END_REF], we provide explicit formulas on the optimal checkpointing intervals for both levels (up to a firstorder approximation), while previous work relies on numerical methods to find the optimal solution [START_REF] Di | Optimization of multi-level checkpoint model for large scale HPC applications[END_REF].

VI. CONCLUSION

When computing at extreme scale, both fail-stop errors and silent errors are major threats to executing HPC applications with acceptable overhead. While several techniques have been developed to cope with either threat, few approaches are devoted to addressing both of them simultaneously. Although surprising-because dealing with both error sources is unavoidable on large-scale platforms-, this lack of solutions may be explained by the new challenges raised by silent errors, whose detection is not immediate and requires the use of verification mechanisms, either partial or guaranteed. Also, the interplay of two levels of checkpoints and of two types of verifications raises difficult optimization challenges. The major contribution of this paper is the characterization of the optimal computational pattern. The derivation is technically involved, but the results are easy to use in real-life scenarios: one has just to look at Table I and pick the optimal pattern that fits their resilience needs. The accuracy of our model as well as the analysis have been nicely corroborated by extensive simulations. The results show acceptable difference in the predicted overhead and the simulated one on systems with up to tens of thousands of nodes. Also, the complex pattern that combines all resilience mechanisms provides up to 150% improvement in the execution overhead compared to the base pattern dictated by the classical Young/Daly's formula.

Finally, our approach is application-agnostic. Future work will be devoted to the study of application-specific verification and checkpoint mechanisms, in particular for sparse iterative solvers. It will be interesting to assess the impact of ad-hoc techniques (ABFT, orthogonality checks, incremental checkpointing, etc) on the overhead of computational patterns in the latter framework.

Figure 1 .

 1 Figure 1. Two examples of patterns.

Figure 2 .

 2 Figure 2. Patterns P DM , P DV and P DM V .

PatternsP

  D P DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV Patterns P D P DV* P DV P DM P DMV* P DMV P DV* P DV P DM P DMV* P DMV P DV* P DV P DM P DMV* P DMV P DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV DV* P DV P DM P DMV* P DMV

Figure 3 .

 3 Figure 3. Performance of the six optimal patterns on the four platforms. Each column represents one platform.

Figure 4 .

 4 Figure 4. Weak scaling experiment on the Hera platform.

Figure 5 .

 5 Figure 5. Impact of error rates λ f and λs on the performance of the patterns on the Hera platform with 10 5 nodes.

Table I THE

 I SIX OPTIMAL PATTERNS. P D , P DV * AND P DV HAVE ONLY ONE LEVEL OF CHECKPOINTING, WHILE P DM , P DM V * AND P DM V HAVE TWO LEVELS. THE TABLE REPORTS THE OPTIMAL PATTERN LENGTH W * , THE OPTIMAL OVERHEAD H * (P) (IGNORING LOWER-ORDER TERMS), THE OPTIMAL NUMBER OF MEMORY CHECKPOINTS n * FOR THE TWO-LEVEL CHECKPOINTING PATTERNS, AND THE OPTIMAL NUMBER OF VERIFICATIONS m * WITHIN A SEGMENT WHEN ADDITIONAL VERIFICATIONS ARE ADDED.

The MTBF µp with p resources is µp = µ ind /p, where µ ind is the MTBF of each resource, see[START_REF] Hérault | Fault-Tolerance Techniques for High-Performance Computing, Computer Communications and Networks[END_REF] Proposition 1.2].

In actual systems, the I/O bandwidth could become a bottleneck, resulting in increased disk checkpointing cost. This would further widen the performance gap between single-level and two-level patterns.

Acknowledgment: This research was funded in part by the European project SCoRPiO, by the LABEX MILYON (ANR-10-LABX-0070) of Université de Lyon, within the program "Investissements d'Avenir" (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR), by the PIA ELCI project, and by the ANR RESCUE project. Yves Robert is with Institut Universitaire de France.