Anne-Sophie Sophie Lambert
email: annesophielambert@orange.fr

Aymeric Dabrin
email: aymeric.dabrin@irstea.fr

Arnaud Foulquier
email: arnaud.foulquier@univ-grenoble-alpes.fr

Soizic Morin
email: soizic.morin@irstea.fr

Christophe Rosy
email: christophe.rosy@irstea.frmarina.coquery@irstea.fr

Marina Coquery

Stéphane Pesce
email: stephane.pesce@irstea.fr

: Irstea

Influence of temperature in pollution-induced community tolerance approaches used to assess effects of copper on freshwater phototrophic periphyton

Keywords: Biofilm, Metal, Microbial ecotoxicology, Combined stress, PICT, Short-term toxicity tests

come

Influence of temperature in pollution-induced community tolerance approaches used to assess effects of copper on freshwater phototrophic periphyton

Introduction

Periphyton is a complex assemblage of phototrophic and heterotrophic microorganisms that can be a valuable indicator of toxic pollution in aquatic systems [START_REF] Sabater | Monitoring the effect of chemicals on biological communities. The periphyton as an interface[END_REF]. However, it is necessary to in situ distinguish toxicant effects from those due to other environmental variables. The pollutioninduced community tolerance (PICT) approach offers a way to isolate, at least partially, the effects of individual toxicants or classes of toxicants (according to their mode of action) within an ecosystem subjected to multiple stressors by studying shifts in community sensitivity (Schmitt-Jansen et al., 2008;[START_REF] Tlili | Pollution-induced community tolerance (PICT) towards an ecologically relevant risk assessment of chemicals in aquatic systems[END_REF]. PICT assumes that chronic exposure of a biological community to toxicants can drive a species succession in which sensitive species are superseded by more tolerant ones [START_REF] Blanck | A critical review of procedures and approaches used for assessing pollution-induced community tolerance (PICT) in biotic communities[END_REF], and/or specific mechanisms of adaptation such as detoxification occur (Tlili and Montuelle, 2011). The resulting tolerance at community level can then be evaluated using short-term toxicity tests on selected microbial functions to determine effective concentrations (e.g. EC50) after acute exposure to the toxicant exerting the selection pressure (or a toxicant with a similar mode of action), using the dose-response toxicology model (Schmitt-Jansen et al., 2008).

Heavy metals, which are common contaminants of freshwaters, are potentially toxic for aquatic microbial communities (e.g. [START_REF] Gustavson | Tolerance induction and succession in microalgae communities exposed to copper and atrazine[END_REF][START_REF] Fechner | Adapting an enzymatic toxicity test to allow comparative evaluation of natural freshwater biofilms' tolerance to metals[END_REF][START_REF] Rotter | Active bio-monitoring of contamination in aquatic systems-an in situ translocation experiment applying the PICT concept[END_REF]. Copper (Cu) is a widespread contaminant in rivers, especially those draining agricultural catchments, notably due to its use as a fungicide in both conventional and organic agriculture [START_REF] Barón | Copper and photosystem II: a controversial relationship[END_REF].

Copper is an essential element for microorganisms. It serves as a cofactor in a number of enzymes that catalyze a wide variety of functions such as photosynthetic and mitochondrial electron transport or several redox reactions [START_REF] Ladomersky | Copper tolerance and virulence in bacteria[END_REF][START_REF] Adams | Copper Uptake, Intracellular Localization, and Speciation in Marine Microalgae Measured by Synchrotron Radiation X-ray Fluorescence and Absorption Microspectroscopy[END_REF]. However, it is known since several decades that Cu can also exert toxic effects on microorganisms. Toxicity mechanisms of Cu notably involve the generation of reactive oxygen species [START_REF] Okamoto | Antioxidant modulation in response to metal-induced oxidative stress in algal chloroplasts[END_REF][START_REF] Sabatini | Oxidative stress and antioxidant defenses in two green microalgae exposed to copper[END_REF]. Toxic effects of Cu on phototrophic microorganisms also include lipid peroxidation [START_REF] Rijstenbil | Oxidative stress induced by copper: defense and damage in the marine planktonic diatom Ditylum brightwellii (Grunow) West, grown in continuous cultures with high and low zinc levels[END_REF] and increased membrane permeability [START_REF] Cid | In situ feeding assay with Gammarus fossarum (Crustacea): modelling the influence of confounding factors to improve water quality biomonitoring[END_REF], decrease in chlorophyll and accessory pigments content [START_REF] Rijstenbil | Oxidative stress induced by copper: defense and damage in the marine planktonic diatom Ditylum brightwellii (Grunow) West, grown in continuous cultures with high and low zinc levels[END_REF], and reduction in growth rates [START_REF] Prasad | Toxicity of cadmiun and copper in Chlamydomonas reinhardtii wild-type (wt 2137) and cell wall deficient mutant strain (cw 15)[END_REF]. At community level, chronic exposure to Cu can functionally impair phototrophic microbial communities by reducing photosynthetic activity [START_REF] Soldo | Long-term effects of copper on the structure of freshwater periphyton communities and their tolerance to copper, zinc, nickel and silver[END_REF][START_REF] Lambert | Structural and functional recovery of microbial periphyton after a decrease in copper exposure: Influence of the presence of pristine communities[END_REF]. It can also impact community structure via changes in the distribution of algal classes and the taxonomic composition of diatom communities [START_REF] Morin | Diatom immigration drives biofilm recovery after chronic copper exposure[END_REF][START_REF] Serra | Effects of chronic copper exposure on fluvial systems: Linking structural and physiological changes of fluvial biofilms with the in-stream copper retention[END_REF]. These changes can ultimately increase phototrophic community tolerance to Cu, in line with the PICT concept [START_REF] Soldo | Long-term effects of copper on the structure of freshwater periphyton communities and their tolerance to copper, zinc, nickel and silver[END_REF][START_REF] Tlili | PO 4 3dependence of the tolerance of autotrophic and heterotrophic biofilm communities to copper and diuron[END_REF]. In lotic ecosystems, tolerance levels of phototrophic periphyton communities to metals are generally assessed by measuring photosynthetic parameters (e.g. Dorigo et al., 2010;[START_REF] Foulquier | Effects of mixtures of dissolved and particulate contaminants on phototrophic biofilms: new insights from a PICT approach combining toxicity tests with passive samplers and model substances[END_REF].

Several recent case studies have demonstrated the potential of PICT approaches to help assess the ecological status of aquatic ecosystems [START_REF] Larras | Pollutioninduced community tolerance (PICT) as a tool for monitoring Lake Geneva long-term in situ ecotoxic restoration from herbicide contamination[END_REF][START_REF] Pesce | Pollution-induced community tolerance for in situ assessment of recovery in river microbial communities following the ban of the herbicide diuron[END_REF][START_REF] Tlili | Micropollutant-induced tolerance of in situ periphyton: Establishing causality in wastewater-impacted streams[END_REF]. However, integrating PICT in monitoring programs still faces a number of challenges [START_REF] Tlili | Pollution-induced community tolerance (PICT) towards an ecologically relevant risk assessment of chemicals in aquatic systems[END_REF] including (i) the need to define tolerance baselines which are the tolerance levels for communities which have not been previously exposed to a selected toxicant or other toxicants with the same mode of action, [START_REF] Pesce | Pollution-induced community tolerance for in situ assessment of recovery in river microbial communities following the ban of the herbicide diuron[END_REF], and (ii) the need to improve the distinction between effects of toxicants and confounding effects due to other environmental factors so as to obtain a better understanding of how communities respond to combined stress [START_REF] Schmitt-Jansen | Induced community tolerance of periphyton towards combined salt and toxic stress[END_REF]. Both tolerance baseline and toxicant-induced tolerance can be modified by external factors, such as light (Guasch et al., 1998;Guasch and Sabater, 1998), nutrients [START_REF] Tlili | PO 4 3dependence of the tolerance of autotrophic and heterotrophic biofilm communities to copper and diuron[END_REF] or other toxicants favoring co-tolerance effects (Schmitt-Jansen et al., 2008), and by internal characteristics such as periphyton biomass [START_REF] Guasch | Contrasting effects of organic and inorganic toxicants on freshwater periphyton[END_REF][START_REF] Lambert | Improved short-term toxicity test protocol to assess metal tolerance in phototrophic periphyton: toward standardization of PICT approaches[END_REF] or community composition and species interactions (Guasch et al., 1998;[START_REF] Bérard | Seasonal variations in the sensitivity of Lake Geneva phytoplankton community structure to atrazine[END_REF].

Global warming and increasing frequency of extreme climate events [START_REF] Easterling | Climate extremes: observations, modeling, and impacts[END_REF][START_REF] Wreford | Adaptation in Agriculture: Historic Effects of Heat Waves and Droughts on UK Agriculture[END_REF] mean that freshwaters may increasingly undergo strong fluctuations in water temperature that can extend beyond normal seasonal and daily temperature variations [START_REF] Smith | An ecological perspective on extreme climatic events: a synthetic definition and framework to guide future research[END_REF]. In lotic ecosystems, such variations in water temperature can impact periphyton, and thus potentially influence community tolerance to toxicants. [START_REF] Boivin | Effects of copper and temperature on aquatic bacterial communities[END_REF] showed that periphytic bacterial communities growing at 14 °C and 20 °C under Cu exposure were respectively three and six times more Cu-tolerant than communities growing at 10 °C. [START_REF] Larras | The effect of temperature and a herbicide mixture on freshwater periphytic algae[END_REF] showed that spring lacustrine phototrophic periphyton growing at 18 °C was more sensitive to a herbicide mixture than that growing at higher temperatures (i.e. 21 °C, 24 °C and 28 °C). Contrastingly, [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF] recently suggested a decrease in Cu tolerance of winter phototrophic periphyton with temperature (from 8 °C to 23 °C), where higher temperatures decreased diatom species diversity and exacerbated community sensitivity towards Cu. While being scarce, these earlier results suggest that temperature is an important environmental parameter to consider when interpreting results from PICT approaches seeking to establish firm causal links between chronic Cu exposure and effects on phototrophic periphyton.

Since little is known about the influence of temperature on the response of phototrophic periphytic communities to acute and chronic exposure to Cu [START_REF] Lambert | Temperature modulates phototrophic periphyton response to chronic copper exposure[END_REF][START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF], the aim of this microcosm study (Fig. 1) was to assess the effects of two contrasting water temperatures (18 °C and 28 °C) on (i) the baseline level of Cu tolerance in non-exposed phototrophic periphyton (i.e. effect of temperature on tolerance baseline), (ii) Cu tolerance acquisition by phototrophic periphyton in response to 3-week chronic exposure to Cu at a nominal concentration of 60 µg L-1 (i.e. effect of temperature on PICT selection) and (iii) the resulting Cu tolerance levels measured after short-term toxicity test on photosynthetic efficiency (i.e. effect of temperature on PICT detection). Based on previous studies showing that a 10 °C temperature increase could significantly modify the community structure of phototrophic periphyton [START_REF] Pippo | Effect of light and temperature on biomass, photosynthesis and capsular polysaccharides in cultured phototrophic biofilms[END_REF][START_REF] Larras | The effect of temperature and a herbicide mixture on freshwater periphytic algae[END_REF][START_REF] Lambert | Temperature modulates phototrophic periphyton response to chronic copper exposure[END_REF][START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF] and select diatom species exhibiting lower tolerance to Cu [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF], we hypothesized that communities growing at 28 °C, with or without Cu exposure, would be less tolerant to subsequent acute exposure to Cu during PICT detection than communities growing at 18 °C.

Materials and Methods

Experimental set-up

The microcosm experiment was carried out using 12 independent aquariums (glass, 40 × 20 × 25 cm) incubated in two tanks (polyethylene, 250 L, 121 × 81 × 33 cm) containing water thermoregulated at 18 °C (i.e. close to the average temperature measured in the field during the sampling period) and 28 °C, respectively (Fig. 1). This high 28 °C temperature was chosen as a worst-case scenario, based on the temperatures recorded in the Morcille river, where the periphyton used in this study was sampled (26.9 °C recorded at 6:00 pm on 21/07/2015 in the downstream section of the Morcille river, unpublished data). In each tank, six microcosms were filled with 16 L (i.e. 20 cm depth) of reconstituted water consisting of 3:1 (v/v) distilled water:groundwater supplemented with nutrients to adjust conductivity (i.e. about 180 µS cm-1) and nutrient concentrations (i.e. 15 mg L-1 silica; 8 mg L-1 nitrates; 0.2 mg L-1 phosphates) to the characteristics of the Morcille river (Dorigo et al., 2010) (Table 1). Each microcosm was fitted with a submerged pump (New Jet 800) to keep the water constantly mixed and oxygenated, and each tank was fitted with three submerged pumps to homogenize water temperature. Highpressure sodium lamps were used to deliver a constant light intensity of 3500 lx (42.7 µmol m-2 s-1) under a 13 h/11 h light/dark photoperiod, corresponding to the field conditions at the sampling date (May 19th, 2014). For each temperature condition (i.e. 18 °C and 28 °C), three microcosms were used as control microcosms (without Cu addition), and three "Cu" microcosms were supplemented with CuSO4•5H2O to obtain a Cu concentration close to 60 µg L-1. This high Cu exposure level was chosen based on previous studies (e.g. [START_REF] Tlili | PO 4 3dependence of the tolerance of autotrophic and heterotrophic biofilm communities to copper and diuron[END_REF][START_REF] Lambert | Structural and functional recovery of microbial periphyton after a decrease in copper exposure: Influence of the presence of pristine communities[END_REF] in order to promote the selection of tolerant species and induce significant changes in community composition compared with the non-exposed control microcosms. To limit Cu adsorption by the experimental equipment during the exposure period, Cu microcosms (including glass slides and pumps) were contaminated using the same Cu concentration for 24 h before the start of the experiment. All the microcosms were filled with uncolonized artificial substrates (glass slides) to allow periphyton settlement during the experiment. Just before the experiment started, stones were collected at the reference upstream site of the Morcille river to retrieve periphyton (Beaujolais, Eastern France, see [START_REF] Montuelle | The periphyton as a multimetric bioindicator for assessing the impact of land use on rivers: an overview of the ArdiSres-Morcille experimental watershed (France)[END_REF] for details). The periphyton was scraped and suspended in river water to obtain a periphytic inoculum, which was homogenized, and then added in equal volumes to all the microcosms at the start of the experiment (Week 0). The study lasted 5 weeks. During Week 1, the water level of each microcosm was adjusted, and each nutrient was added to maintain the initial trophic conditions. Water was then renewed weekly to maintain Cu exposure and avoid nutrient depletion. From Week 2 to Week 5, the main physical and chemical water parameters were measured in each aquarium before and 2 h after each water renewal. Conductivity, pH, and dissolved oxygen concentrations were measured using portable meters (WTW, Germany). Water samples were collected at the same time for subsequent laboratory analyses.

Water temperature was recorded every hour with data loggers (HOBO® Pendant Temperature/Light, Prosensor). Water temperatures in the microcosms were close to the target temperatures, with mean values of 18.3 °C (± 0.5; min 17.5, max 19.9) and 27.0 °C (± 1.8; min 22.9, max 29.4), respectively, with no significant difference between control and Cu microcosms.

Chemical analyses

Standard operating procedures were followed to determine the concentrations of orthophosphates (PO4; NF EN ISO 6878), nitrates (NO3; NF EN ISO 10304), nitrites (NO2; NF EN 26777), ammonium (NH4; NF T 90-015-2), silica (SiO2; NF T 90-007) and dissolved organic carbon (DOC; NF EN 1484).

Microcosm experimental design and strategy.

To measure dissolved Cu concentrations, 30 mL of microcosm water was sampled before and after each water renewal, filtered (0.45 µm polyvinylidene difluoride (PVDF) filter, Whatman), acidified with 0.5% (v:v) nitric acid (Suprapur, Merck), and stored at 4 °C until analysis. Filtered water samples were analyzed using inductively coupled plasma -mass spectrometry (ICP-MS XSeries II, Thermo Electron). Quality controls were routinely checked using a certified reference material (Environment Canada, TM 27-3, Lake Ontario natural water) to check analytical accuracy (97%) and precision (± 12%).

In the control microcosms, dissolved Cu concentrations remained very low (0.4 ± 0.1 µg L-1) throughout the experiment. In all Cu microcosms, at 2 h after water renewal, mean dissolved Cu concentrations were 63.0 ± 8.8 µg L-1 with no significant difference among treatments. As previously observed in similar microcosm experiments [START_REF] Lambert | Structural and functional recovery of microbial periphyton after a decrease in copper exposure: Influence of the presence of pristine communities[END_REF][START_REF] Lambert | Temperature modulates phototrophic periphyton response to chronic copper exposure[END_REF], Cu concentrations decreased strongly between each water renewal (from 43% to 60% according to the sampling time). Mean dissolved Cu concentrations before water renewal were 32.2 ± 8.0 µg L-1, with no significant difference among treatments.

Periphyton characterization and tolerance assessment

The biofilm growth cycle includes successive phases namely initial cell attachment to a surface, microcolony formation, biofilm maturation and detachment (also termed dispersal). Given the importance of the growth state on PICT measurement [START_REF] Lambert | Improved short-term toxicity test protocol to assess metal tolerance in phototrophic periphyton: toward standardization of PICT approaches[END_REF], the sampling strategy used to collect periphyton in the microcosms was based on growth curve analysis to perform PICT analysis on biofilms collected during their advanced stage of maturation (and before cell detachment). Periphytic communities were collected at the end of the exponential growth (i.e. mature state) to assess the tolerance levels on mature assemblages while limiting the cell detachment process. In the control microcosms, algal biomass rapidly increased, and a mature periphyton was obtained within three weeks. By contrast, Cu induced a slower algal growth, and periphyton became mature only after 5 weeks. Accordingly, periphyton characterization and tolerance assessment were performed after 3 weeks of development for control communities (to limit cell detachment) and after 5 weeks of development for Cu communities. About 15 slides were randomly collected in each microcosm, and periphyton was carefully scraped off with a razor blade, suspended in 1:1 (v/v) demineralized:mineral water (Volvic, France), and homogenized to obtain one periphyton suspension per microcosm. Each suspension was divided up into different volumes to measure the following parameters.

Physical-chemical characteristics of water in the control and Cu microcosms for the four tested conditions before (i.e. conditions after 1 week) and 2 h after each water renewal (i.e. initial conditions), for 5 weeks (means value ± standard deviation [s.d.], n = 3).

DO, dissolved oxygen; DOC, dissolved organic carbon.

To measure algal cell density, 2 mL of periphyton suspension was immediately fixed in formalin (37% formaldehyde, Prolabo, France) for counting. Each sample was counted using a Nageotte counting chamber (Marienfeld, Germany). After homogenization, 125 µL of sample was placed in the counting chamber, and the total number of cells was recorded in 10 fields of the gridded area (1.25 µL each, 0.5 mm depth) under light microscopy at × 400 magnification (Olympus BX51 upright microscope, UK). Microalgae were counted individually (i.e. enumeration of all cells when forming colonies) and discriminated, based on their morphological features, into diatoms, chlorophytes and cyanobacteria. For diatoms, dead and live organisms before fixation were distinguished by observation of the turgidity and color of the chloroplast, as described in [START_REF] Morin | Recovery potential of periphytic communities in a river impacted by a vineyard watershed[END_REF].

Aliquots of periphyton suspension (3 mL) were also used to determine total chlorophyll a (chl a) and photosynthetic efficiency by multiwavelength pulse-amplitude-modulated (PAM) fluorimetry on a Phyto-PAM system (H. Walz, Germany) as described in Schmitt-Jansen and Altenburger (2008). Photosynthetic efficiency was based on measurement of the maximum PSII quantum yield (PSII) (Schmitt-Jansen and Altenburger, 2008), a measure of the physiological state of phototrophic communities [START_REF] Tlili | PO 4 3dependence of the tolerance of autotrophic and heterotrophic biofilm communities to copper and diuron[END_REF].

Photosynthetic efficiency was also used as endpoint to assess the tolerance of phototrophic periphyton communities to Cu in the PICT approach, following [START_REF] Lambert | Improved short-term toxicity test protocol to assess metal tolerance in phototrophic periphyton: toward standardization of PICT approaches[END_REF] with slight modifications. Given the influence of periphyton biomass on PICT measurement [START_REF] Lambert | Improved short-term toxicity test protocol to assess metal tolerance in phototrophic periphyton: toward standardization of PICT approaches[END_REF], we diluted the initial periphyton suspension with an equivalent mixture of demineralized water and mineral water (Volvic, France) to obtain a periphyton suspension concentration of about 2000 µg chl a L-1 (as measured with the Phyto-PAM fluorimeter) for each sample before exposure of periphyton to increasing Cu concentrations.

A semi-logarithmic series of Cu concentrations was freshly prepared in the same 1:1 demineralized:mineral water mixture to obtain nine Cu concentrations, ranging from 0.32 to 3200 mg L-1. The Cu concentration of each solution was analyzed by ICP-MS as described above; 1.8 mL of the periphyton suspension was then exposed for 2 h to 0.9 mL of the different Cu solutions, to obtain final Cu concentrations ranging from 0.1 to 1000 mg L-1.

The short-term (2 h) incubation was performed in climatic chambers (MLR-350 Versatile Environmental Test Chamber, Sanyo) under artificial light (1400 lx). To evaluate the influence of growth temperature on tolerance acquisition (i.e. during the PICT selection phase), the short-term incubation was carried out at the temperature of the microcosms during growth (i.e. 18 °C and 28 °C, respectively) (Fig. 1). To assess the influence of incubation temperature on tolerance measurement (i.e. during the PICT-method detection phase), the short-term incubation was also carried out at the intermediate temperature between the two temperature conditions tested (i.e. 23 °C) (Fig. 1).

After 2 h incubation with the nine Cu solutions (three independent analytical replicates per replicate sample and per concentration) and the water blank (five independent analytical replicates per replicate sample), periphyton suspensions were kept for 30 min in a dark chamber, and maximum quantum yield of PSII was then measured on the Phyto-PAM (665 nm).

Statistical analysis

Variations in water and periphyton characteristics (physical and chemical characteristics, chl a, PSII yield, cellular density) between temperature conditions (during growth and toxicity tests) were tested by ANOVA followed by a post hoc Tukey test using R version 2.15.0 (R Development Core Team, 2012). Data were log-transformed before statistical analysis to satisfy the conditions of normality and homogeneity of variances. The analysis of results from short-term bioassays was conducted using functions from the 'drc' package [START_REF] Ritz | Bioassay analysis using R[END_REF] in R version 2.15.0 (R Development Core Team, 2012). For each microcosm, the dose-response curve was fitted to the data using the fourparameter log-logistic model given by the formula:

, where b is slope of the curve around e, c and d are the lower and upper limits of the curve, respectively, and e is EC50, the dose producing a response half-way between the upper and the lower limit of the curve. The response variable was expressed as the percentage of the yield of the uncontaminated control assay. EC50 was derived from each dose-response curve. ANOVAs followed by Tukey HSD tests were used to determine significant between-treatment differences in EC50, upper and lower limits, slope and percentage of inhibition.

Results and discussion

Effect of temperature on tolerance baseline

One limitation to the use of PICT in monitoring programs is the lack of knowledge about theoretical tolerance baselines that would preclude the need for reference samples in field studies [START_REF] Pesce | Pollution-induced community tolerance for in situ assessment of recovery in river microbial communities following the ban of the herbicide diuron[END_REF]. The difficulty defining baseline values is due to spatial and seasonal variability in basal community tolerance related to the effects of various environmental factors, which cannot be controlled during in situ exposure and can act as confounding factors [START_REF] Cid | In situ feeding assay with Gammarus fossarum (Crustacea): modelling the influence of confounding factors to improve water quality biomonitoring[END_REF]. To evaluate the potential influence of a drastic change in temperature on the baseline tolerance of phototrophic periphyton to Cu, we assessed the effect of a 10 °C temperature rise on the Cu tolerance levels of non-Cu-exposed communities. The magnitude and range of the temperature increase were chosen as representative of extreme climatic events in the Morcille river where the periphyton used in this study was sampled (e.g. increase in water temperature from 18.3 °C to 25.8 °C between 21/06/2015 and 01/07/2015, unpublished data).

To avoid temperature stress during the measurement of baseline tolerance, toxicity tests were performed at growth temperature (i.e. 18 °C and 28 °C for periphyton growing at 18 °C and 28 °C, respectively). After a growing period of three weeks at 18 °C, which was representative of the river temperature during the initial sampling of periphyton, the basal tolerance of control phototrophic communities was characterized by an EC50(18 °C) value for Cu of 2.58 ± 0.31 mg L-1 (Table 2). EC50(28 °C) values obtained with non-Cu-exposed communities growing at 28 °C (0.71 ± 0.41 mg L-1) were about three orders of magnitude lower than those obtained with 18 °C communities, revealing a notable decrease in their tolerance to Cu even if it was not statistically significant (p > 0.99). This result is consistent with the recent findings of [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF] who observed that warming (from 8 °C to 23 °C) significantly decreased the basal tolerance of phototrophic biofilms towards acute exposure to Cu. In their study, the change in basal tolerance was mainly attributable to a change in phototrophic community structure since temperature appeared to be an important driver for several diatom species [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF] as previously shown [START_REF] Nicola | Periphyton responses to temperature at different ecological levels[END_REF][START_REF] Larras | The effect of temperature and a herbicide mixture on freshwater periphytic algae[END_REF]. In the present study, the increase in the sensitivity to Cu at higher temperature in control phototrophic communities may stem, at least partially, from structural changes. In our experimental conditions, the 10 °C temperature increase was an important driver of phototrophic community evolution, where we found an increase in algal densities (Fig. 3), in particular in cyanobacteria, either solitary or forming short filaments. The observed decrease in basal tolerance may also have been due to physiological stress induced by the marked temperature increase, which could have weakened phototrophic communities, thus increasing their sensitivity to subsequent stress (i.e. Cu exposure during the toxicity test). This hypothesis is supported by the significant decrease observed in PSII values at 28 °C (p = 2), which revealed an inhibitory effect of temperature on photosynthesis. The influence of temperature on periphyton photosynthesis has been sparsely researched, and results differ significantly among studies. For example, at variance with our results, [START_REF] Baulch | Effects of warming on benthic communities in a boreal lake: implications of climate change[END_REF] reported an increase in photosynthetic activity of periphyton when temperature increased by 4.5 °C (with initial temperatures ranging between 9.3 °C and 14.3 °C), whereas [START_REF] Larras | The effect of temperature and a herbicide mixture on freshwater periphytic algae[END_REF] found that increasing the growing temperature from 18 °C to 28 °C had no effect on the maximum PSII quantum yields of lacustrine periphyton. This inconsistency among studies could be due to differences in initial taxonomic composition, and/or to differences in temperature gradients tested. It is well known that the thermal optimum for photosynthesis varies according to algal species [START_REF] Necchi | Photosynthetic responses to temperature in tropical lotic macroalgae[END_REF]. In our case, the 10 °C temperature rise probably exceeded a temperature-mediated threshold for community tolerance.

These results reveal that changes in temperature can notably modify the basal capacity of phototrophic periphyton to tolerate subsequent exposure to Cu. This means that temperature is an important factor to consider when modeling the influence of environmental parameters to determine theoretical basal tolerance to metals. However, given the extreme temperature rise tested in the present work as well as the lack of statistical significance of the observed trend effect, further research is needed to make a more accurate assessment of how the basal tolerance can be influenced by more realistic temperature variations (e.g. seasonal and daily temperature variations or gradual increase following a prolonged heat event).

Effect of temperature on PICT selection

The literature reports strong evidence that several-week exposure to Cu at environmental concentrations of some tens of µg L-1 (e.g. about 10 µg L-1 in Tlili et al., 2011; about 25 µg L-1 in [START_REF] Lambert | Structural and functional recovery of microbial periphyton after a decrease in copper exposure: Influence of the presence of pristine communities[END_REF] can induce structural changes in phototrophic periphytic communities, leading to an increase in their Cu tolerance (i.e. PICT selection). In our experiment, phototrophic communities were exposed to Cu concentrations between 24 µg L-1 (i.e. lowest concentration recorded before water renewal) and 69 µg L-1 (i.e. highest concentration recorded after water renewal). This exposure

Fig. 2. PSII yield (mean ± s.d., in relative units; n = 3) of control and Cu-exposed communities at week 3 and week 5 for growth temperatures 18 °C and 28 °C. Asterisks indicate significant differences (p < 0.05) between 18 °C and 28 °C for each Cu treatment (control and Cu).

level remains environmentally relevant given that Cu concentrations can reach several hundred µg L-1 in highly contaminated rivers (e.g. [START_REF] Hirst | Comparing the responses of diatoms and macro-invertebrates to metals in upland streams of Wales and Cornwall[END_REF][START_REF] Cantonati | Achnanthidium minutissimum (Bacillariophyta) valve deformities as indicators of metal enrichment in diverse widely-distributed freshwater habitats[END_REF]. Unsurprisingly, phototrophic periphyton growing for 5 weeks at 18 °C with no heat stress treatment and under such Cu exposure exhibited high tolerance levels to Cu (EC50(18 °C) of 44.8 ± 14.1 mg L-1; Table 2). The EC50 value was thus 17 times higher than that observed in control communities growing for 3 weeks at 18 °C (2.6 ± 0.3 mg L-1). Despite the difference in growth duration between control and Cuexposed communities, which prevents statistical comparison, this large difference evidenced an adaptation of phototrophic periphyton to Cu following chronic exposure. This result is in accordance with [START_REF] Soldo | Long-term effects of copper on the structure of freshwater periphyton communities and their tolerance to copper, zinc, nickel and silver[END_REF], who reported a 6-fold increase in tolerance of phototrophic periphyton communities (based on EC50 values using photosynthesis rates measured by the 14C technique) after 12 weeks of exposure to Cu at 63.5 µg L-1 (nominal concentration). To assess the effect of temperature on PICT selection, phototrophic periphytic communities were simultaneously exposed to Cu and subjected to a 10 °C increase. The EC50(28 °C) of Cu-exposed communities at 28 °C was over 30 times lower (1.41 ± 0.14 mg L-1) than the EC50(18 °C) of Cu-exposed communities at 18 °C (44.8 ± 14.1 mg L-1; Table 2), highlighting that the 10 °C increase in temperature led to a sharp, significant decrease in tolerance levels of Cu communities (p < 0.001). Furthermore, the lower limit observed in the dose-response curves showed that PSII yield was almost completely inhibited (i.e. 98%) when Cu-exposed communities growing at 28 °C were exposed to the highest Cu concentration tested (3200 mg L-1) in bioassays, whereas the PSII yield of Cu-exposed communities growing at 18 °C was only reduced by about 64% (p < 0.001; Table 2). Moreover, the upper limit revealed that basal PSII yield of Cu-exposed communities was lower at 28 °C (0.32 ± 0.03) than at 18 °C (0.53 ± 0.01; p < 0.001). The same significant decrease in PSII yield between 18 °C-and 28 °C-Cu-exposed communities was also observed when the measurement was made immediately after periphyton sampling in the microcosms (p < 0.001; Fig. 2). This strong decrease in PSII yield revealed a significant effect of the temperature increase on the physiological state of the chronically Cu-exposed phototrophic communities. This effect could have weakened the Cu-exposed communities growing at 28 °C, decreasing their capacity to tolerate a subsequent acute exposure to copper, as suggested by the EC50 values. As mentioned above for non-exposed communities, changes in phototrophic community composition may also explain the increased vulnerability of periphyton towards acute exposure to Cu [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF]. With long-term Cu exposure, diatoms were almost eliminated from the community: at 18 °C; diatoms only accounted for about 0.01% of the communities (Fig. 3), with a mortality of 15.7 ± 9.7%. This suggests that green algae and cyanobacteria were more Cu-tolerant. However, the 10 °C increase in temperature led to changes in the growth forms of both green algae and cyanobacteria. At 18 °C, soft algae in Cu-exposed communities were mostly solitary, or formed short 3-5 cell fragments, whereas the higher temperatures favored long filamentous colonies. The more heterogeneous algal community growing at 18 °C, mostly composed of solitary cells with probably different sensitivity levels to Cu, could explain the higher tolerance to low Cu concentrations, and the sharp decrease in PSII yield around the EC50, as shown by the significantly higher slope (Table 2) at 18 °C (1.90 ± 0.37) than at 28 °C (1.00 ± 0.38) (p = 0.022). These results thus suggest that in our experimental conditions, temperature and metal stress acted antagonistically on community tolerance acquisition, with a reduced tolerance to copper of the Cuexposed communities grown at 28 °C. These findings are consistent with our initial hypothesis which was based on the results of [START_REF] Morin | Changes in copper toxicity towards diatom communities withexperimental warming[END_REF], who found that a temperature gradient increase from 8 °C to 23 °C selected for less Cu-tolerant diatom species, leading to a decrease in Cu-induced community tolerance. Assessing in laboratory conditions the effects of Cu on the structure of mature biofilms collected in Dutch rivers, [START_REF] Barranguet | Development of photosynthetic biofilms affected by dissolved and sorbed copper in a eutrophic river[END_REF][START_REF] Barranguet | Copper-induced modifications of the trophic relations in riverine algal-bacterial biofilms[END_REF] observed that phototrophic communities collected in summer (at a water temperature of 19.5 °C) were more strongly impacted by Cu than communities collected in spring (11.5 °C) exposed mature biofilms collected in Dutch rivers to copper in laboratory conditions. Their work showed that water temperature may lead to contrasting responses of diatoms to Cu exposure. Performing a 6-year in situ study in a river contaminated by cadmium and zinc, [START_REF] Morin | Cumulative stressors trigger increased vulnerability of diatom communities to additional disturbances[END_REF] demonstrated that the effects of metal pollution on diatom diversity were more drastic during the hottest summers. Altogether, present and previous results evidence the influence of temperature on the acquisition of tolerance of phototrophic periphytic communities to Cu and other metals. That means that temperature is an important factor to consider when using PICT to establish causal links between chronic Cu exposure and effects on phototrophic periphyton.

Effect of temperature on PICT detection

Besides the influence of growth temperature on tolerance baseline and PICT selection, we also evaluated the influence of incubation temperature on tolerance measured during short-term toxicity tests, both for determining tolerance baseline (non-Cu exposed communities) and for assessing induced tolerance (Cu exposed communities). To this end, phototrophic periphyton communities were incubated for 2 h with increasing concentrations of Cu at their respective growth temperatures (18 °C or 28 °C) and at an intermediate temperature (23 °C) (Fig. 1).

Short-term toxicity tests performed at 18 °C and 23 °C for control and Cu-exposed communities growing at 18 °C found similar upper limits (i.e. yields of uncontaminated control assay; C0) irrespective of incubation temperature (p > 0.75 for control and p > 0.99 for Cu; Table 2), suggesting a lack of any direct effect of short-term incubation temperature on photosynthetic efficiency. Moreover, incubation temperature had no effect (p between 0.64 and 1.00) on the lower limit and slope of the dose-response curves (Fig. 4, Table 2). Furthermore, there was no significant difference between EC50 values obtained via short-term toxicity tests at 18 °C or 23 °C for either controls (about 2 mg L-1, p > 0.99) or Cu-exposed communities (about 40 mg L-1, p > 0.93). Likewise, at a growth temperature of 28 °C, there was no significant difference between EC50 values obtained from shortterm toxicity tests at 28 °C and 23 °C for either controls (about 1 mg L-1, p > 0.99) or Cu-exposed communities (about 1.5 mg L-1, p = 1.00). This study thus demonstrated that a 5 °C variation in incubation temperature applied during short-term toxicity tests had no significant influence on measured tolerance, regardless of the growth temperature and Cu exposure history. Our investigation on the influence of thermal conditions on PICT measurement addresses the challenge of improving the standardization of PICT approaches for integration into regulatory frameworks for assessing ecological and chemical status of aquatic ecosystems [START_REF] Pesce | Pollution-induced community tolerance for in situ assessment of recovery in river microbial communities following the ban of the herbicide diuron[END_REF][START_REF] Tlili | Pollution-induced community tolerance (PICT) towards an ecologically relevant risk assessment of chemicals in aquatic systems[END_REF]. In our experimental context, and in the absence of any influence of temperature during short-term incubation, we have demonstrated that Cu tolerance measurements can be performed at a fixed intermediate temperature without interfering with the interpretation of results. Nevertheless and despite the lack of significant statistical difference between incubation temperatures in our experimental conditions (p < 0.05) it is interesting to note that, whatever the growth temperature (i.e. 18 °C and 28 °C) and the Cu treatment (i.e. Control and Cu), mean EC50 values were always lower at higher incubation temperature during short-term toxicity tests (i.e. 23 °C vs 18 °C and 28 °C vs 23 °C). This observed trend may suggest an increase in the sensitivity of phototrophic communities to acute toxicity of copper in response to temperature increase during short-term toxicity tests. Keeping in mind that in the field, the winter-summer temperature swing can exceed 10 °C (e.g. [START_REF] Pesce | Longitudinal changes in microbial planktonic communities of a French river in relation to pesticide and nutrient inputs[END_REF][START_REF] Pesce | In situ relationships between spatio-temporal variations in diuron concentrations and phototrophic periphyton tolerance in a contaminated river[END_REF], it will be necessary to further test the influence of incubation temperature for greater differences in growth temperature (i.e. difference between growth and incubation temperatures > 5 °C). We also note that intra-daily temperature variations may be an important factor to consider in studies assessing the combined effects of temperature and toxicants: on the downstream reach of the Morcille river; the difference between minimum and maximum daily water temperature can reach 6 °C on sunny summer days (unpublished data), and comparable orders of magnitude have been observed in other rivers in Slovakia [START_REF] Pekárová | Long-term trend and multi-annual variability of water temperature in the pristine Bela River basin (Slovakia)[END_REF] and in Montana [START_REF] Gammons | Diel behavior of iron and other heavy metals in a mountain stream with acidic to neutral pH: Fisher Creek, Montana, USA[END_REF].

Conclusions 5.

Assessing the ecological effects of pollutants is an important aspect of regulations dealing with the sustainable management of water resources. In this context, scientists and regulators face the challenge of going beyond the estimation of pollutant concentrations to take into account the ecotoxicological effects on exposed aquatic communities. Among microbial ecotoxicological approaches, the pollution induced community tolerance (PICT) concept has been proven to be a powerful metric to better link the assessment of ecological and chemical status of ecosystems [START_REF] Pesce | Pollution-induced community tolerance for in situ assessment of recovery in river microbial communities following the ban of the herbicide diuron[END_REF][START_REF] Tlili | Pollution-induced community tolerance (PICT) towards an ecologically relevant risk assessment of chemicals in aquatic systems[END_REF]. However, prior its implementation in regulatory frameworks as a suitable biomonitoring tool, further work is required to standardize PICT measurement and define baseline tolerance levels at large geographical scales. The present results obtained by testing an extreme environmental scenario (in terms of temperature increase and Cu exposure) evidenced the potential influence of temperature on both the basal capacity of phototrophic periphyton to tolerate subsequent exposure to Cu (i.e. influence on tolerance baseline), and its capacity to acquire tolerance after chronic exposure to Cu (i.e. influence on PICT selection). They reveal that temperature should be an important factor to consider when using PICT at a large geographical or temporal scale, especially for integrating PICT within regulatory frameworks to assess ecological and chemical status of aquatic ecosystems. Accordingly, it would be relevant to develop strategies for modeling the influence of temperature on PICT approaches (using Cu as model compound) to make measurements comparable in space and time by considering robust reference conditions defined at a larger scale [START_REF] Cid | In situ feeding assay with Gammarus fossarum (Crustacea): modelling the influence of confounding factors to improve water quality biomonitoring[END_REF].

Fig

  Fig. 1.

Fig

  Fig. 3.Cellular density (mean ± s.d.; cells cm-2; n = 3) of live diatoms and other microalgae in control and Cu-exposed communities at week 3 and week 5 for growth temperatures 18 °C and 28 °C. Lack of asterisk indicates no significant differences (p < 0.05) between 18 °C and 28 °C for each treatment (control and Cu) and each kind of biological parameter (Live diatoms and Other microalgae).

Fig. 4 .

 4 Fig. 4.Dose-response curves showing inhibition of PSII yield with increasing concentrations of copper for control (A) and Cu-exposed communities (B).For each treatment, first and second temperatures in the legend are growth temperature and shortterm toxicity temperature, respectively. For clarity, data from the three replicate bioassays were pooled to produce a single dose-response curve per treatment.

Table 1 .

 1

Table 2 . Tolerance measurements: summary of parameters derived from the dose-response curves obtained from short-term bioassays for control and Cu-exposed community microcosms. For a given parameter and a given Cu exposure context, different lower case letters indicate significant differences (p < 0.05) between temperature. Values are means ± standard deviation for each microcosm (n = 3).

 2

			Control			Cu
	Growth temperature (°C)	18		28		18		28
	Incubation temperature (°C)	18	23	28	23	18	23	28	23
	EC50 (mg/L)	2.58 ± 0.31 (a)	2.01 ± 0.41 (a)	0.71 ± 0.41 (a)	1.35 ± 0.68 (a)	44.82 ± 14.13 (a)	40.02 ± 2.32 (a)	1.41 ± 0.14 (b)	1.76 ± 0.90 (b)
	Slope	0.57 ± 0.11 (a)	0.64 ± 0.09 (a)	1.09 ± 0.47 (a)	1.00 ± 0.45 (a)	1.90 ± 0.37 (a)	2.34 ± 0.20 (a)	1.00 ± 0.38 b)	0.89 ± 0.30 (b)
	Upper limit (yield; relative unit)	0.52 ± 0.01 (a)	0.56 ± 0.02 (a)	0.46 ± 0.02 (a)	0.45 ± 0.07 (a)	0.53 ± 0.01 (a)	0.52 ±0.01 (a)	0.32 ± 0.03 (b)	0.28 ± 0.07 b)
	Lower limit (yield; relative unit)	0.00 ± 0.00 (a)	0.00 ± 0.00 (a)	0.00 ± 0.00 (a)	0.03 ± 0.05 (a)	0.19 ± 0.06 (a)	0.15 ± 0.04 (a)	0.01 ± 0.01 (b)	0.00 ± 0.01 (b)
	PSII (% of control)	100 ± 0 (a)	100 ± 0 (a)	100 ± 0 (a)	93 ± 13 (a)	64 ± 11 (a)	71 ± 8 (a)	98 ± 3 (b)	99 ± 2 (b)

Acknowledgments

The authors thank ATT for English-language editing. We also thank Bernard Motte and Bernadette Volat (EMHA, Irstea) for technical support, and the staff at the Aquatic Chemistry Laboratory (LAMA, Irstea) for water chemistry analysis. This project is part of LTER-funded research program "Zone Atelier du Bassin du Rhône" (ZABR).


Pascale Minet

Éric Renault
email: eric.renault@telecom-sudparis.eu

Ines Khoufi

Selma Boumerdassi
email: selma.boumerdassi@cnam.fr

Analyzing Traces from a Google Data Center

Keywords: Data analysis, data center, big data application, resource allocation, scheduling

Traces collected from an operational Google data center over 29 days represent a very rich and useful source of information for understanding the main features of a data center. In this paper, we characterize the strong heterogeneity of jobs and the medium heterogeneity of machine configurations. We analyze the off-periods of machines. We study the distribution of jobs per category, per scheduling class, per priority and per number of tasks. The distribution of job execution durations shows a high disparity, as does the job waiting time before being scheduled. The resource requests in terms of CPU and memory are also analyzed. The distribution of these parameter values is very useful to develop accurate models and algorithms for resource allocation in data centers.

I. CONTEXT AND MOTIVATION

In High Performance Computing (HPC), it is assumed that all machines are homogeneous in terms of CPU and memory capacities, and that, the tasks making up the jobs have similar resource requests. It has been shown in [START_REF] Reiss | Heterogeneity and dynamicity of clouds at scale: Google trace analysis[END_REF] that this homogeneity relating both to machine capacity and workload , although generally valid for HPC, does no longer apply to data centers. This explains why the publication of data gathered in an operational Google data center over 29 days [START_REF] Wilkes | More Google cluster data[END_REF] has aroused such great interest among researchers. For jobs and tasks, researchers want to characterize their submission, their structure and the workload requested, while for machines, they study the distribution of off-periods. They also highlight any periodic patterns and tendencies, and detect correlations between memory usage and CPU usage if such correlations exist. More generally, they validate or invalidate some simplifying assumptions that are usually made when reasoning about models. Such results are needed to make the models more accurate for jobs and tasks as well as for available machines. Having validated these models on real data centers, they are then used for extensive evaluation of placement and scheduling algorithms and more generally for resource allocation (i.e. CPU and memory). These algorithms can then be applied in real data centers. Another possible use of this data set is to consider it as a learning set in order to predict some feature of the data center, such as the workload of hosts or the next arrival of jobs. As a conclusion, it is crucial to have real traces of a Google data center publicly available that are representative of the functioning of real data centers. Our goal in this paper is to analyze the data collected and to draw useful conclusions about machines, jobs and tasks as well as resource usage. In a further step, these results will be integrated in models used in a general framework designed for high performance resource allocation in a data center.

II. RELATED WORK

All the studies so far published relating to this data set [START_REF] Wilkes | More Google cluster data[END_REF] include data analysis. In this paper, we also focus on data analysis and draw original conclusions on machine availability, as in [START_REF] Liu | Characterizing machines and workloads on a Google cluster[END_REF], [START_REF] Beaumont | Analyzing real cluster data for formulating allocation algorithms in Cloud platforms[END_REF], jobs and tasks, as in [START_REF] Beaumont | Analyzing real cluster data for formulating allocation algorithms in Cloud platforms[END_REF], [START_REF] Alam | Analysis and clustering of workload in google cluster trace based on resource usage[END_REF], [START_REF] Yousif | Clustering cloud workload traces to improve the performance of cloud data centers[END_REF] and resource usage, as in [START_REF] Reiss | Heterogeneity and dynamicity of clouds at scale: Google trace analysis[END_REF], [START_REF] Alam | Analysis and clustering of workload in google cluster trace based on resource usage[END_REF], [START_REF] Gbaguidi | Characterizing servers workload in Cloud Datacenters[END_REF], [START_REF] Di | Host load prediction in a google compute cloud with a bayesian model[END_REF]. The first lessons drawn from a shorter data set of 6 hours were published in [START_REF] Chen | Analysis and lessons from a publicly available google cluster trace[END_REF]. Although the authors pointed out the presence of peaks in job submissions, the existence of both short jobs and very long ones, the size of this data set is too small to generalize. Using the 29-day data set published in [START_REF] Wilkes | More Google cluster data[END_REF], the authors of [START_REF] Beaumont | Analyzing real cluster data for formulating allocation algorithms in Cloud platforms[END_REF] show that on average 5.8% of jobs running in a given time window account for 94.7% of CPU usage and 89% of memory usage. They also show that actual resource usage can be approximated by a lognormal distribution. With regard to CPU usage, two thirds of the dominant jobs belonging to normal production exhibit a daily pattern and half of the jobs exhibit a weekly pattern. In addition, the distribution of machine removals shows that the number of removals observed in the trace is higher than that provided by a Poisson distribution, suggesting some spatial or temporal correlations for some machines. However, independent failures with a failure rate of 10 -5 per hour is a reasonable approximation. A characterization of machines and jobs is provided in [START_REF] Liu | Characterizing machines and workloads on a Google cluster[END_REF]. The authors observe that while most machine downtimes are short (i.e less than 25 mn), others may last more than 2 hours and a half. They also show that there are many jobs with a single task, and that most jobs have fewer than 100 tasks, but also that there is a small number of jobs with 2000 tasks. They also point out that on average, a job has 20 scheduling constraints associated with its tasks, which makes such tasks more difficult to place and schedule. An important result is that up to 60% of CPU resource is wasted by tasks that do not complete successfully. The authors of [START_REF] Alam | Analysis and clustering of workload in google cluster trace based on resource usage[END_REF] use K-Means to classify the jobs into three categories according to their duration: short, medium and long. They show that short jobs are the most frequent and use less resources. Medium jobs are frequent and are memory-intensive, i.e. they have strong requirements in terms of memory. Long jobs, the least frequent, have strong requirements in terms of CPU, and are termed CPU-intensive. The authors of [START_REF] Reiss | Heterogeneity and dynamicity of clouds at scale: Google trace analysis[END_REF] show that 2% of jobs account for 80% of resource usage (i.e. CPU or memory) and that about 75% of jobs run a single task. More generally, they observe that the number of tasks per job is a power of two. The high diversity of resource requests per task is highlighted. The overall memory usage is only 53% of the memory allocation, whereas the CPU usage corresponds to only 40% of the CPU allocation. In [START_REF] Yousif | Clustering cloud workload traces to improve the performance of cloud data centers[END_REF], the tasks are classified by means of K-Means and density-based clustering into two categories: CPU-intensive and memory-intensive. The authors propose associating a virtual machine with each task, and their placement algorithm puts two virtual machines of different categories on the same physical machine. However, no performance evaluation is done to validate this approach. In [START_REF] Di | Host load prediction in a google compute cloud with a bayesian model[END_REF], the authors predict the average load of a host with a Bayesian model. Their method provides a greater accuracy than other methods based on moving averages or autoregression.

III. DATA PREPROCESSING A. Real traces

The real traces were collected in an operational Google data center over a period of 29 consecutive days. The goal of the authors of [START_REF] Reiss | Google cluster-usage traces: format + schema[END_REF] was to point out the complexity of scheduling the jobs submitted to a data center. This complexity is due to the variety of job types, the existence of scheduling constraints for some jobs, the heterogeneity of machines and a bad estimation of resource usage by users. The confidentiality of these traces is ensured by obfuscation techniques. Each information element recorded has a timestamp expressed in microseconds. The traces are organized into different tables related to machines, jobs and tasks as well as resource usage. Each table corresponds to several files in the CSV format.

• The M achine events table contains the following events: add a machine to the data center, remove a machine from the data center, and update the available resources of a machine. It also gives the CPU capacity and the memory capacity of the machine. It has a volume of 2.9 MBytes. • The M achine attributes table gives the kernel version, clock speed, etc. of the machine considered. • The Jobs events table and the T ask events table describe the events related to a job and a task, respectively. These events are submit, schedule, evict, fail, finish, kill, update pending and update running. The T asks events table also contains the scheduling class, the priority, the resource request in terms of CPU and memory, as well as some placement constraint of each task. This table has a size of 15.4 GBytes. • The T asks resource usage includes information about the mean and maximum usage of memory and CPU during each measurement period of 5 minutes. It has a size of 159 GBytes.

B. Data cleaning

Before being analyzed, data are cleaned up. Any record with missing information is discarded. The outliers are discarded, for example events occurring at time 0 that were artificially added by the measurement process. To make the processing of records faster, the columns in the different tables that are not analyzed are removed.

IV. MACHINE ANALYSIS

A. Memory and CPU capacities of machines

The total number of machines in the data center is 12582. The CPU capacity of any machine is expressed relative to the CPU capacity of the most powerful machine. As a consequence, three types of machines are distinguished, as depicted in Figure 1.

• The most powerful machines with a CPU capacity equal to 1. There are 796 machines of this type, which corresponds to 6.32% of the total number of machines. These machines have consecutive identifiers, which could mean that they have been replaced recently. • Machines with medium CPU capacity, equal to 0.5. These represent the largest number of machines (11632) corresponding to 92.449% of the total number of machines. • The least powerful machines with small CPU capacity, equal to 0.25. There are 123 such machines, which represents 0.977% of the total number of machines. As an initial approximation, machines can be considered as homogeneous in terms of having a CPU capacity equal to 0.5. Similarly, the Memory capacity of any machine is expressed relative to the highest Memory capacity. 6 types of machines are distinguished, as depicted in Figure 2. There is a greater heterogeneity of machines in terms of memory capacity than in CPU capacity. Assuming that the most recent machines have the highest identifiers, we notice that the most recent machines have the highest memory capacity. We can now focus on the different machine configurations. Each machine configuration is represented by a pair (CP U capacity, memory capacity), as illustrated in Figure 3. We observe the existence of 10 different configurations in the data center. The most frequent configuration, 53% of the machines, corresponds to a CPU capacity ≥ 0.5 and a memory capacity ≥ 0.5. We observe that the machines with the most powerful CPU also have the largest memory capacity. 30% of machines have a CPU capacity of 0.5 and a memory capacity of 0.25. Machine heterogeneity (i.e. 10 different configurations) should be taken into account in the scheduling algorithm. In addition, a uniform distribution model is not appropriate.

Table I shows the percentage of machines per category, given in decreasing order. In a first approximation, we can consider only the first six configurations, the last four configurations concerning only 0.10% of the machines.

B. Study of machine events

We now focus on the number of changes in machine status. Google defined three types of machine events:

• Add (also called event 0) the machine is added to the set of operational machines. • Remove (also called event 1) the machine is no longer operational. It has either failed or is undergoing maintenance.

• Update (also called event 2) the resources (e.g. CPU or memory or system version) of the machine are updated.

Figure 4 depicts the number of machines on a logarithmic scale having a given number of adds, removes and updates per machine. The number of machines having had a given number of removes is equal to the number of machines having had the same number of adds. This seems to indicate that each machine that has been removed has finally been added. The number of updates is higher than the number of adds and removes. We observe that very few machines, 4, have a number of changes higher that 50, which corresponds to 0.03%. A very large number of machines, 12540, corresponding to 99.66% of all the machines have a number of changes less than or equal to 10. The three curves (i.e. add, remove and update) follow a hyperbolic pattern.

There were 8860 remove events followed by an add event, corresponding to machine restarts. The duration of these offperiods is depicted in Figure 5. It varies from 5 seconds to 1044365 seconds, corresponding to 11.5 days. 1 machine was unavailable for 11.5 days. 50% of off-periods were less than or equal to 1000 seconds. 1% of the off-periods has a duration equal to 1000 seconds. The number of off-periods equal to a given duration higher than 1000 seconds is equal to 1 or 2.

V. PATTERN OF JOB SUBMISSIONS

We compute the number of jobs submitted per minute. observing the distribution of job submissions over the 29 days a repetitive and fractal pattern becomes apparent, as depicted in Figure 6. We observe a periodicity of 10, 100, 1000 and 10000. This means that every 10 minutes we observe an absence of submissions for 1 minute; every 100 minutes an absence of job submissions for 10 minutes; every 1000 minutes an absence of job submissions for 100 minutes; every 10000 minutes an absence of job submissions during 1000 minutes. An absence of job submissions does not mean that no job arrives in the data center but that no job is released by the global scheduler. This regularity cannot be explained by randomness. It should correspond to some periodic management task run in the Google data center.

VI. ANALYSIS OF JOBS AND TASKS

From the T ask Event table, we compute the following three distributions: the priority of jobs, the scheduling class of jobs and the number of tasks per job.

The priority of jobs ranges from 0 to 11. Depending on its priority, a job belongs to one of the following five categories defined in [START_REF] Reiss | Heterogeneity and dynamicity of clouds at scale: Google trace analysis[END_REF]:

• F ree with the lowest priorities 0 and 1. Such jobs incur little charging. • P roduction with priority 9.

• M onitoring with priority 10. These jobs are in charge of monitoring the health of other jobs. • Inf rastructure with the highest priority 11. These jobs deal with the infrastructure of the data center. • Other with priorities 2 to 8. Table II provides the distribution of jobs in the different categories. Figure 7 depicts the number of jobs per priority on a logarithmic scale. There is no job in priority 7. As expected, there is only 1 job having the highest priority 11 which is reserved to the infrastructure. The number of jobs with a priority equal to 3 or 5 is equal to 100, for both. The numbers of jobs having a priority of 0, 1, 4, 6, 8 or 9 is close to 100000. Here again, a uniform distribution of jobs in the different priorities does not apply. The scheduling class of a job indicates the latency sensitivity of this job, where 0 denotes a non-production task and 3 a latency-sensitive task. Figure 8 depicts the number of jobs per scheduling class. As expected, there is a very small number of jobs, 2364, that are latency-sensitive, with a scheduling class equal to 3. Most jobs, 178959, belong to the scheduling class 1, whereas remaining jobs are more or less evenly distributed in the scheduling classes 0 and 2, with about 12000. Table III provides the percentage of jobs per scheduling class. The number of tasks per job is depicted in Figure 9 on a logarithmic scale on both axes. 92.05% of jobs have a single task. 95.75% of jobs have fewer than 10 tasks, 98.6% of jobs have fewer than 50 tasks and 99% of tasks have fewer than 92 tasks. We also notice that 1 job has a number of tasks equal to 10500 and 12 jobs have a number of tasks equal to 5000. The number of tasks per job is frequently a multiple of 10.

VII. RESOURCE REQUESTS AND JOB EXECUTION

From the Task Event table, we compute the distribution of CPU requests and memory requests per job shown in Figure 10. We observe that all the tasks of a same job request the same amount of CPU and the same amount of memory. We notice that: • 1.54% of jobs have a CPU request higher than or equal to 10% • 1.74% of jobs have a memory request higher than or equal to 10%. • 0.11% of jobs have a memory request and a CPU request higher than or equal to 10%. In Figure 10, where the x-axis and the y-axis are represented with a log base 2 scale, many CPU and memory requests are "aligned". This means that some specific values are preferred over others. An in-depth analysis shows that most of the "lines" are powers of 2. In other words, memory requests and CPU requests are often expressed as powers of 2. We first focus on the time jobs must wait before being scheduled. This time is called job schedule time. The distribution of job schedule times is depicted in Figure 11. 60% of jobs wait 1 second before being scheduled. 94.25% of jobs wait less than 10 seconds. Surprisingly, there are 50 jobs (0.013%) that wait more than 1000 seconds. One possible explanation could be that they request specific resources that are not immediately available.

The job execution time is evaluated from the schedule time up to the completion time of the last task. The distribution of the job execution times is illustrated in Figure 12. 35122 jobs do not finish within the 29-day period. considered in the dataset. 384362 jobs finish successfully, corresponding to

VIII. CONCLUSION

The data set obtained from an operational data center over 29 days provides very interesting information. Data analysis allows us to draw the following conclusions.

• Although 92.45% of machines have a CPU capacity of 0.5, there are 10 machine configurations in the data center, each configuration is characterized by a pair (CP U capacity, memory capacity). The most frequent configuration is supported by only 53.34% of machines. • Over the 29 days, all the machines in the data center that were removed, were restarted later after an offperiod. 50% of these periods have a duration less than or equal to 1000 seconds (i.e. 16.66 minutes), suggesting a maintenance operation.

• The study of the number of job submissions per minute shows an unexpected fractal pattern with a periodicity of 10, 100, 1000 and 10000 minutes, where every 10 minutes there is an absence of submissions for 1 minute, and so on. This could be explained by the fact that the global scheduler could be periodically unavailable to allow some management of the data center infrastructure to take place. scheduled. However, some of them wait for more than 1000 seconds. Such large values could be explained by the existence of placement constraints for the jobs, making them harder to place and schedule. 49% of jobs have an execution time less than 100 seconds. These features should be reflected in the job sets and the models used to evaluate the performances of scheduling placement algorithms in data centers.

Fig. 1 .

 1 Fig. 1. CPU capacity

Fig. 2 .

 2 Fig. 2. Memory capacity.

Fig. 3 .

 3 Fig. 3. Number of machines per (CPU, Memory) configuration.

Fig. 4 .Fig. 5 .

 45 Fig. 4. Number of adds(0), removes(1) and updates(2) per machine.

Fig. 6 .

 6 Fig. 6. Repetitive pattern in the job submission.

Fig. 7 .

 7 Fig. 7. Number of jobs per priority.

Fig. 8 .

 8 Fig. 8. Number of jobs per scheduling class.

Fig. 9 .

 9 Fig. 9. Distribution of the number of tasks per job.

Fig. 10 .

 10 Fig. 10. Distribution of CPU and memory requests.

Fig. 11 .

 11 Fig. 11. Distribution of job schedule times.

Fig. 12 .

 12 Fig. 12. Distribution of job execution times.

  Memory capacity in the interval [0.4995, 0.5]. There are 6712 of these machines, corresponding to 53.34% of the total number of machines. • Machines with a Memory capacity in the interval [0.2493, 0.25]. There are 3858 machines, which corresponds to 30.66% of the total number of machines. • Machines with a Memory capacity of 0.1241. There are 54 such machines, which corresponds to 0.429% of the total number of machines. • The least powerful machines with a Memory capacity in the interval [0.03885, 0.06158]. There are 6 machines of this type, corresponding to 0.047% of the total number of machines.

• Machines with the highest memory capacity have a Memory capacity in the interval [0.9678, 1]. There are 907 machines of this type, which corresponds to 7.02% of the total number of machines. • Machines with a Memory capacity of 0.75. These are 1001 such machines, which corresponds to 7.95% of the total number of machines.

• Machines with a

  91.62% of all the jobs. 4 jobs have an execution time less than 10 seconds. About 36000 jobs, representing 9% of the jobs have an execution time within the interval [20, 25] seconds. Most jobs, 49%, have an execution less less than 100 seconds. 90% of jobs have a duration less than 1000 seconds. We observe that the number of jobs with an execution time equal to 1000×n, with n a positive integer, is approximately divided by two, each time n is increased by 1.

  • The distribution of jobs per category reveals only one job, representing 0.002%, for the Infrastructure, 0.13% of jobs for Monitoring, 9.91% of jobs for Production, 56.30% of jobs for Other, and 33.63% of jobs for Free. 92.05% of jobs have a single task. 95.75% have fewer than 10 tasks. But 12 jobs have 5000 tasks and 114 jobs have around 1000 tasks.• With regard to resource requests, 0.11% of jobs have a memory request and a CPU request higher than or equal to 10%. • 94.25% of jobs wait less than 10 seconds before being


Pascale Minet

Eric Renault

Ines Khoufi

Selma Boumerdassi

Data analysis of a Google data center

published or not. The documents may come

Data Analysis of a Google Data Center

Pascale Minet * Éric Renault † Ines Khoufi * and Selma Boumerdassi ‡ * Inria, 2 rue Simone Iff, CS 42112, 75589 Paris Cedex 12, France. Email: firstname.name@inria.fr † SAMOVAR, Telecom SudParis, CNRS, Universite Paris-Saclay, 91011 Evry, France. Email: eric.renault@telecom-sudparis.eu ‡ CNAM/CEDRIC, 292 rue Saint Martin, 75003 Paris, France. Email: selma.boumerdassi@cnam.fr

Abstract-Data collected from an operational Google data center during 29 days represent a very rich and very useful source of information for understanding the main features of a data center. In this paper, we highlight the strong heterogeneity of jobs. The distribution of job execution duration shows a high disparity, as well as the job waiting time before being scheduled. The resource requests in terms of CPU and memory are also analyzed. The knowledge of all these features is needed to design models of jobs, machines and resource requests that are representative of a real data center.

I. CONTEXT AND MOTIVATION

In High Performance Computing (HPC), on the one hand all machines are assumed to be homogeneous in terms of CPU and memory capacities, and on the other hand, the tasks comprising jobs have similar resource requests. It has been shown in [START_REF] Reiss | Heterogeneity and dynamicity of clouds at scale: Google trace analysis[END_REF] that this homogeneity related to both machines capacities and workloads that has been validated for HPC is no longer valid in data centers. This explains why the publication of data gathered in an operational Google data center during 29 days [START_REF] Wilkes | More Google cluster data[END_REF] has raised a large interest among researchers.

For jobs and tasks, researchers want to characterize their submission, their structure and the workload requested. For machines, they study the distribution of the off-periods. They also highlight periodic patterns and tendencies, if any. They detect correlations between memory usage and CPU usage if such correlations exist. More generally, they validate or invalidate some simplifying assumptions usually made when reasoning on models. Such results are needed to make the models more accurate for jobs and tasks as well as for available machines. These models being validated on real data centers are then used for extensive evaluation of placement and scheduling algorithms and more generally for resource allocation (i.e. CPU and memory). These algorithms can then be applied in real data centers.

Another possible use of this data set is to consider it as a learning set in order to predict some feature of the data center like for instance the workload of hosts or the next arrival of jobs.

As a conclusion, it is crucial to have real traces of a Google data center publicly available that are representative of the functioning of real data centers. Our goal in this paper is to analyze the collected data and to draw pertinent conclusions about jobs and tasks as well as resource usage. In a further step, these results will be integrated in models used in a general framework designed for a high performance resource allocation in a data center.

II. ANALYSIS OF JOBS AND TASKS

The Jobs events table and the T ask events table describe the events related to a job and a task, respectively. These events are submit, schedule, evict, fail, finish, kill, update pending and update running. The T asks events table also contains the scheduling class, the priority, the resource request in terms of CPU and memory, as well as some placement constraint of each task. This table has a size of 15.4 GBytes.

Before being analyzed, data are cleaned. Any record with missing information is discarded. The outliers are discarded, like for instance the events occurring at time 0 that have been artificially added by the measurement process. To make faster the processing of records, the columns in the different tables that are not analyzed are withdrawn.

From the T ask Event table, we compute the distribution of the number of tasks per job, the CPU request and memory request per job, the job execution time and the job schedule time.

The number of tasks per job is represented in Fig. 1 in a logarithmic scale on both axes. 92.05% of jobs have a single task. 95.75% of jobs have less than 10 tasks, 98.6% of jobs have less than 50 tasks and 99% of tasks have less than 92 tasks. We also notice that 1 job has a number of tasks equal to 10500 and 12 jobs have a number of tasks equal to 5000. The number of tasks per job is frequently a multiple of 10. From the Task Event table, we compute the distribution of CPU request and memory request per job that is represented in Fig. 2. We observe that all tasks of a same job request the same amount of CPU and the same amount of memory. We notice that:

• 1.54% of jobs have a CPU request higher than or equal to 10%. • 1.74% of jobs have a memory request higher than or equal to 10%. • 0.11% of jobs have a memory request and a CPU request higher than or equal to 10%. On Fig. 2, where the x-axis and the y-axis are represented with a log base 2 scale, we notice that lots of CPU and memory requests are "aligned". This means that some specific values are preferred over the other. An in-depth analysis shows that most of the "lines" are powers of 2. In other words , memory requests and CPU requests are often expressed as powers of 2. The distribution of the job execution times is illustrated in Fig. 3. 35122 jobs do not finish in the considered 29 days in the dataset. 384362 jobs finish successfully, corresponding to 91.62% of jobs. 4 jobs have an execution duration less than 10 seconds. About 36000 jobs, representing 9% of jobs have an execution duration in the interval [20, 25] seconds. Most jobs, 49%, have an execution duration less than 100 seconds. 90% of jobs have a duration less than 1000 seconds. We observe that the number of jobs with an execution duration equal to 1000 × n, with n a positive integer, is approximately divided by two, each time n is increased by 1.

To explain these large values of job execution durations, we focus on the waiting time of jobs before being scheduled. This time is called job schedule time. The distribution of job schedule times is depicted in Fig. 4. 60% of jobs wait 1 second before being scheduled. 94.25% of jobs wait less than 10 seconds. Surprisingly, there are 50 (0.013%) of jobs that wait more than 1000 seconds. A possible explanation could be that they request specific resources that are not immediately available.

III. CONCLUSION

The data set provided from an operational data center during 29 days contains very interesting information. Data analysis allows us to draw the following conclusions. • 92.05% of jobs have a single task. 95.75% have less than 10 tasks. But 12 jobs have to 5000 tasks and 114 jobs have around 1000 tasks. • With regard to resource requests, 0.11% of jobs have a memory request and a CPU request higher than or equal to 10%. • 49% of jobs have an execution duration less than 100 seconds. • 94.25% of jobs wait less than 10 seconds before being scheduled. However, some of them wait more than 1000 seconds. Such large values could be explained by the existence of placement constraints for the jobs making them harder to place and schedule. These features should be reflected in the job sets and the models used to evaluate the performances of scheduling placement algorithms in data centers.

Fig. 1 .

 1 Fig. 1. Distribution of the number of tasks per job.

Fig. 2 .

 2 Fig. 2. Distribution of CPU and memory requests.

Fig. 3 .Fig. 4 .

 34 Fig. 3. Distribution of job execution time.