Fluoride HCl Hydrogen chloride HCO Bicarbonate K Potassium Mg Magnesium Mn Manganese Na Sodium NO Nitrate Pb Lead SiO Silicon dioxide SO

Keywords: Data preprocessing, Data analysis, Environmental Data, Water Quality Assessment, water pollution. v vi IQR Inter Quartil Range

Agradezco a mis muy queridos amigos Miguel Angel, Carlos, Jorge, Paulina y Margarita por su ayuda incondicional, sus consejos, nuestras platicas existenciales, nuestros chistes de barrio. Gracias por estar siempre presentes en forma física, electrónica o espiritual.

Enfin, je termine avec un remerciement espécial pour la personne qui a partagé avec moi dès le début le stress d'une réponse de bourse, pour me soutenir dans mon bourdel administrative, pour les jours épluchantes (avec et sans marques sur le front). Merci Batu.

Acknowledgments -

Remerciements -Agradecimientos

Et puis...il y a ceux que l'on croise, que l'on conaît à peine, qui vous disent un mot, une phrase, vous accordent une minute, une demi-heure et changent le cours de votre vie. Katherine Pancol.

I have the pleasure to take some lines to acknowledge the people who have been with me all along these three years.

First, I thank my mentors Dr. Laure Berti-Equille and Dr. Ma. Aurora Armienta Hernandez who believe in this project and who always know how to encourage me and give me the best advises to improve my work. Thank you for your support, guidance and cheering, but mainly to have accepted to be my mentors in this project that take us extra-job to « translate » English to Spanish, Spanish to French, chemistry to computing science. I thank Dr. Hélène Fenet, Dr. Karina Giber, Dr. François Pinet and Dr. Nathalie Villa-Vialaneix for have accepted to evaluate my thesis work. The experience of defending my PhD was very pleasant despite the stress and many hair left on my shower sink.

Je tiens sincèrement à remercier à Corinne Grac et Jean-Christophe Desconnets mes encadrants. Je suis vraiment ravi d'avoir été encadré et d'avoir travaillé avec vous. A la fois pour vos inestimables conseils, vos relectures précieuses (surtout dans les deadlines très fermés) et vos encouragements dans mes moments de faiblesse. Je remercie spécialement à Corinne Grac pour son support dans la recherche de financement, des solutions pour mes multiple et variées problèmes administratives, je n'aurai pas pu commencer sans ton précieuse aide.

Mes remerciements vont aussi à Isabelle Mougenot et Florence Le Ber pour avoir fait partie de mon comité de suivi de thèse en évaluant l'avancement de mes travaux.

Cette thèse a été effectué sur plusieurs sites en commençant par la belle alsace en france puis en partant à mon bien aimé (et chaotique) Mexique et en finissant pour le jovial Montpellier. Tout ça n'as pas pu être possible sans l'accueil des responsables d'équipes et directeurs de laboratoires. Je remercie donc Frédérique Seyler (ESPACE-DEV, IRD, Montpellier) et Jean-François Quéré (ENGEES) pour m'avoir accueilli et m'avoir offert un cadre de travail idéal.

Également, je remercie à tout l'équipe de ESPACE-DEV pour leur conseils, leur précieuses commentaires envers mon travail de thèse et surtout pour l'ambiance familiale qui fait que l'on a vraiment mal à quiter le labo.

Un grand merci à l'équipe MICADO qui m'a accueilli très chaleureusement, pour tout iii ces moments des échanges et discussions. Pour notre café matinale et les délicieux croissants ou sucreries. Mais aussi pour tout ces moments que l'on a partagé sans climatisation en été ou sans chauffage en hiver ça a donnée plus d'animation dans nos journées. Tout d'abord merci à mes collègues doctorants, aux spicy girls Minh et Yi. Il n'y a que vous pour partager ma passion pour la nourriture épicé ou les petit piments avec des fruits ;). Minh pour nos après midi-culturelles et pour m'avoir fait partie de l'arrivé d'Emma, Yi ma accompagnante des dernière jours de souffrance et pour avoir resté avec moi "enfermé dehors" au labo. Moejdeh et Phuong, vous Un remerciement particulier à ma chère coloc et copine Cléo, qui m'a supporté, soutenu et nourri. Les petit plats, les soirée clignotage et tous les moments de complicité ont nourri pas seulement mon estomac mais ils m'ont permis d'arriver vivante jusqu'à la fin. Merci mamacita.

Agradezco, al Instituto de Geofísica por haberme permitido trabajar en sus instalaciones. Gracias a las muchachas que hacen vivir el laboratorio. A la maestra Alejandra Aguayo y a las químicas Olivia Cruz, Nora Ceniceros y Aurelia Juárez por su ayuda, consejo estimulo paciencia y compañía. Gracias a mis colegas Chuy, Esther, Juan, Israel, Azu. El trabajo en el laboratorio se vuelve una tarea agradable y ligera en su compañía. Gracias a mi estudiante Atzin por haber compartido el trabajo de conteo de cuerpos desmembrados, cabezas volantes y patas destazadas de nuestros bichitos. Gracias a la doctora Claudia A. Ponce de Leon Hill por haberme permitido trabajar y hacer uso de las instalaciones en su laboratorio. Agradezco especialmente al maestro Manuel Hernández Quiroz por su disponibilidad, su tiempo y dedicación para introducirme en el mundo de los plaguicidas. Gracias a todos los chicos del laboratorio de ecologia, trabajar en su labo siempre fue muy placentero.

Agradezco a mis padres, hermanas y mis sobrinos, quienes siempre han creído en mi y que a pesar de la distancia siempre han estado presentes para apoyarme y ayudarme. Estar lejos de ustedes siempre ha sido el mas grande desafío, todas nuestras sesiones de skype dominicales me dieron la fuerza para continuar en este proyecto personal que se volvió nuestro proyecto. Los quiero mucho.

Résumé

Les données acquises lors des surveillances environnementales peuvent être sujettes à différents types d'anomalies (i.e., données incomplètes, inconsistantes, inexactes ou aberrantes). Ces anomalies qui entachent la qualité des données environnementales peuvent avoir de graves conséquences lors de l'interprétation des résultats et l'évaluation des écosystèmes. Le choix des méthodes de prétraitement des données est alors crucial pour la validité des résultats d'analyses statistiques et il est assez mal défini. Pour étudier cette question, la thèse s'est concentrée sur l'acquisition des données et sur les protocoles de prétraitement des données afin de garantir la validité des résultats d'analyse des données, notamment dans le but de recommander la séquence de tâches de prétraitement la plus adaptée. Nous proposons de maîtriser l'intégralité du processus de production des données, de leur collecte sur le terrain et à leur analyse, et dans le cas de l'évaluation de la qualité de l'eau, il s'agit des étapes d'analyse chimique et hydrobiologique des échantillons produisant ainsi les données qui ont été par la suite analysées par un ensemble de méthodes statistiques et de fouille de données. En particulier, les contributions multidisciplinaires de la thèse sont : ( ) en chimie de l'eau : une procédure méthodologique permettant de déterminer les quantités de pesticides organochlorés dans des échantillons d'eau collectés sur le terrain en utilisant les techniques SPE-GC-ECD (Solid Phase Extraction -Gas Chromatography -Electron Capture Detector) ; ( ) en hydrobiologie : une procédure méthodologique pour évaluer la qualité de l'eau dans quatre rivières Mexicaines en utilisant des indicateurs biologiques basés sur des macroinvertébrés ; ( ) en science des données : une méthode pour évaluer et guider le choix des procédures de prétraitement des données produites lors des deux précédentes étapes ainsi que leur analyse ; et enfin, ( ) le développement d'un environnement analytique intégré sous la forme d'une application développée en R pour l'analyse statistique des données environnementales en général et l'analyse de la qualité de l'eau en particulier. Enfin, nous avons appliqué nos propositions sur le cas spécifique de l'évaluation de la qualité de l'eau des rivières Mexicaines Tula, Tamazula, Humaya et Culiacan dans le cadre de cette thèse qui a été menée en partie au Mexique et en France.

Mots clés : prétraitement des données, analyse des données, données environnementales, évaluation de qualité de l'eau, pollution de l'Eau.

vii viii

Resumen

Los datos obtenidos de monitoreos ambientales pueden estar sujetos a diferentes tipos de anomalías (i.e., datos incompletos, inconsistencias, inexactitudes, o valores extremos). Estas anomalías que afectan la calidad de los datos ambientales pueden tener consecuencias graves al ser utilizados en la interpretación de resultados y en la evaluación de los ecosistemas. La elección de métodos de pre-tratamiento de datos resulta crucial en la validación de resultados de análisis estadísticos sin embargo, esta esta mal definida. Para estudiar esta problemática, esta tesis se concentro en la adquisición y en los protocolos de pre-tratamiento de datos para garantizar la validez de los resultados de análisis, con la finalidad principal de dar recomendaciones sobre las secuencias de pre-tratamiento mas adecuadas. Nosotros proponemos un control integral en el proceso de producción de datos, desde la colecta en el terreno de estudio hasta su análisis. En el caso de la evaluación de la calidad del agua, se trata de las etapas de análisis químico e hidrobiológico de muestras, cuyos datos producidos han sido posteriormente analizados a través de un una serie de métodos estadísticos y de exploración de datos. Las contribuciones multidisciplinarias de la tesis son: ( ) en química ambiental: un procedimiento metodológico que permite determinar el contenido de plaguicidas organoclorados en muestras de agua utilizando las técnicas de SPE-GC-ECD (Solid Phase Extraction -Gas Chromatography -Electron Capture Detector); ( ) en hidrobiología: un procedimiento metodológico para evaluar la calidad del agua en cuatro ríos Mexicanos, utilizando indicadores biológicos basados en macro-invertebrados; ( ) en ciencia de los datos: un método para evaluar y guiar la elección de procedimientos de pre-tratamiento de datos producidos luego de las dos etapas precedentes, así como su análisis; y ( ) el desarrollo de un ambiente analítico integrado en forma de aplicación desarrollada en R para el análisis estadístico de datos ambientales en general y el análisis de la calidad del agua en particular. Finalmente, dentro del contexto de esta tesis desarrollada entre México y Francia, hemos aplicado nuestras propuestas en el caso especifico de la evaluación de la calidad del agua de los rios mexicanos Tula, Tamazula, Humaya y Culiacan.

List of Figures

. General water Experimental procedure for assessment of outlier detection and processing.

. NRMSE results of the imputation of missing values using the Hot-deck (hd), IRMI (ir), k-NN (kn) and Mice (mi) imputation methods on datasets N (A), N A. Trent Biotic Index (TBI). Key groups for the estimation of the Trent River Board Biotic Index and Biotic Index values related to the total number of groups present in a sample (

CHAPTER

Introduction

According to UNESCO, rivers, lakes and coastal areas of developing countries are polluted by more than % of untreated sewage. In , the World Health Organization (WHO) estimated that , human deaths were caused by contaminated drinking water and soil (WHO, ). Regarding freshwater biodiversity, globally, more than % of vertebrates species have been threatened by pollution (Assessment, ; Vörösmarty et al., ), % of the world's amphibian species are threatened with extinction (Dudgeon et al., ) and it is estimated that rates of species loss have a tendency to increase with higher values in tropical latitudes (Groombridge and Jenkins, ).

Faced with the need to reduce water pollution, intensive studies to monitor the quality of water are continuously implemented. Monitoring activities consist in measuring various water quality components at many locations during numerous time periods. Though data from environmental surveys may be prone to have different anomalies (i.e., incomplete, inconsistent, inaccurate or outlying data). Anomalies on data are ubiquitous, they arise due to experimental problems, human errors or system failures.

Bad data quality, may be significant costly (Haug et al., ) and have considerable consequences when assessing environmental ecosystems (Wahlin and Grimvall, ). To generate quality data and reduce data anomalies, it is necessary both: to acquire quality data and preprocess data (Han and Kamber, ; Berti-Équille, a).

The acquisition of quality data can be obtained by following standardized sampling and analytical protocols, and using advanced analytical tools (i.e., Inductively Coupled Plasma Mass Spectrometry (ICP-MS) , Gas Chromatography Mass Spectrometry (GC-MS), Liquid Chromatography Mass Spectrometry (LC-MS) or Liquid Chromatography tandem Mass Spectrometry (LC-MS-MS)) (UNEP,

). However, in developing countries access to advanced analytical tools may be limited or standardized protocols are difficult to implement. It is thus, necessary to adapt tools that can be easy and cheap to use without compromising the quality of data.

Concerning data preprocessing, it consists of all actions necessary to generate quality data. These actions include but are not limited to:

. Cleaning data (i.e., removing outliers) and inconsistencies; . Resolving data conflicts (i.e., settling discrepancy); . Recovering incomplete data (i.e, filling missing values); . Reducing data (i.e., selecting relevant attributes) or creating new aggregate data.

In fact, a vast number of data preprocessing procedures are available rending the selection of an optimal procedure a very arduous task.

There are many inherent challenges in this respect, for instance:

• Conventional approaches treat each data anomaly as isolated cases however, data anomalies co-occur in data. Therefore it is necessary to develop preprocessing procedures that deal with multiple data anomalies within a dataset; • The ordering in which preprocessing procedures should be executed is under studied.

It is necessary to define a data preprocessing ordering to produce the least biased results; • The impact of data preprocessing on statistical analysis results has not been widely studied, it is necessary to conduct studies in this area to produce accurate results; • An extensive work that provides information about the accuracy on the application of preprocessing procedures is needed.

Data preprocessing, will definitely improve the quality of data. But, good data acquisition practices are also necessary to prevent data anomalies. An important problem on the acquisition of quality environmental data is the lack of standardized sampling and analytical protocols particularly on laboratories where the access to advanced analytical instruments is limited. Thus, it is necessary to adapt analytical procedures that are cost-effective and easy to implement in order to get quality data.

We are interested in developing a new approach that combines both; good data acquisition and data preprocessing practices. Our main motivation is to provide tools and methodological approaches to the scientific community for the acquisition, preprocessing and analysis of environmental data. In this manner we were concentrated in two main challenges: acquire quality data and preprocess data. Our purposes are: ( ) to provide methodological approaches and tools for the acquisition of water quality data and ( ) to provide a general overview of the advantages and disadvantages of preprocessing procedures on statistical results.

Concerning the acquisition of data we focused on the development of tools in Environmental Chemistry and Hydrobiology. We were especially interested in: ( ) water pollution problems caused by heavy metals, pesticides, pharmaceutical and personal care products, and ( ) the use of biomonitoring metrics using macroinvertebrates for water quality assessment of Mexican rivers. In Mexico, the monitoring of these type of pollutants is limited regardless their toxic effects on ecosystems and human health. Such limitation is due to the restricted access to advanced analytical instruments for their quantification. By adapting analytical techniques we will be able to: quantify pesticides, pharmaceuticals and personal care products in water; acquire quality data by reducing the presence of anomalies such as missing values, censored or outlying data; and identify sources of pollution that may affect ecosystems and human health. Use of biomonitoring metrics based on macroinvertebrates is an interesting approach for the assessment of water quality in rivers. It is a low-cost and easy to implement tool that provides valuable information about the ecological state of aquatic ecosystems. Nonetheless, in Mexico biomonitoring metrics are scarcely used. In fact, well-described sampling and analytical protocols for their implementation do not exist yet. We were interested in developing sampling and analytical protocol to use biomonitoring metrics based on macroinvertebrates for the water quality assessment of Mexican rivers.

Regarding data preprocessing, we were interested in highlighting the importance of Chapter having quality data by implementing appropriate data preprocessing procedures. In previous works (Serrano Balderas, ; Serrano Balderas, ) we observed that the quality of data has a significant impact on analytical results. These works conducted us to perform a study where we demonstrated the importance of having quality data in an environmental application framework (Berrahou et al., ). Data quality is undoubtedly important in data analysis, we are concerned now about the procedures to get quality data namely, preprocessing procedures.

. Objectives

Our principal objective is to provide to the scientific community the methodological approaches and tools for the acquisition, preprocessing and analysis of environmental data by guaranteeing the quality of data and analysis results. We aim at applying our methodological approaches for a specific case: water quality assessment of four Mexican rivers (Tula, Tamazula, Humaya and Culiacan).

Our purpose on providing an integrated approach that combines both; good data acquisition and data preprocessing practices, is to control the entire pipeline this is, from the production and to the analysis of data. To achieve our aim, we have defined specific objectives related to the three areas of our study. The objectives are:

Environmental Chemistry

• Acquire data by deploying reliable and low-cost methods of analysis for the quantification of organochlorine pesticides, pharmaceuticals and personal care products in water. These methods aimed at acquiring quality data for water quality assessment in four Mexican rivers (Tula, Humaya, Tamazula and Culiacan); • Specify the sampling and methodological protocols for the analysis of organochlorine pesticides, pharmaceuticals and personal care products; • Conduct the sampling campaign and analytical procedures for the analysis of water samples from the four Mexican rivers (Tula, Humaya, Tamazula and Culiacan).

Hydrobiology

• Acquire data by defining a methodological approach using macroinvertebrates-based biomonitoring metrics as new complementary tools for water quality assessment of Mexican rivers; • Conduct the sampling campaign and analytical procedures for the acquisition of hydrobiological data.

Data science

• Define a methodological approach for the selection of data preprocessing procedures to treat the most common data anomalies and data problems (missing values, outliers, feature selection and normalization); • Evaluate the impact of data preprocessing procedures on subsequent statistical analysis; • Determine the most appropriate data preprocessing procedures to get the less biased analytical results; • Specify the procedures to preprocess and analyse data that are necessary to ensure the reliability of results on environmental studies in general and for water quality data analysis in particular.

. Outline

The document is organized as follows:

Chapter presents an overview of related work including: a) the acquisition of water quality data for physico-chemical, chemical and biological parameters and b) preprocessing and analysis of environmental data. In Chapter we describe our methodological approaches for the acquisition of chemical and biological data. We also describe the study sites and the methods followed to collect and analyse the samples. Our methodological approach for the assessment of data preprocessing procedures is detailed in Chapter . In Chapter we present EvDa, a Shiny/R application designed to preprocess and analyse environmental data. Results of EvDa application to water quality data from the Mexican rivers: Tula Tamazula, Humaya and Culiacan are also described in Chapter .

Finally, conclusions and future work associated to our contributions are given in Chapter .

CHAPTER

Environmental data analysis: the case of water quality assessment with a multidisciplinary survey

. Introduction

The main tasks presented in this manuscript are: data acquisition, data preprocessing and data analysis. Concerning data acquisition we aimed at developing and applying tools in environmental chemistry and hydrobiology to acquire water quality data. For data preprocessing, we aimed at developing a methodological framework to prepare the data and analyse them. As a consequence our bibliographic study ranges from environmental chemistry (Section . ), hydrobiology (Section . ), computer science and statistics (Section . ).

.

Data collection in physico-chemical and chemical water quality assessment

Collection of water quality data involves a set of tasks that include but are not limited to: design of experiment, selection of sites of study, sample collection, development and/or adaptation of analytical methods for analysis of pollutants and analysis of data (Bartram and Ballance, ). For a better understanding of the characteristics of water quality data and an appropriate subsequent analysis, we provide a general overview of methods and tasks that are implemented to get water quality data. We focused mainly on methods to acquire data related to pesticides, pharmaceuticals and beauty products as pollutants.

. . Water quality assessment of rivers

Water quality monitoring programs (WQMP) are conducted, in general, to get information about the characteristics of water resources, identify pollution problems, evaluate and describe water management actions (WMO, ). Assessment of water quality is commonly carried out following standardized methods and protocols. Globally, the WHO have proposed a practical guide for designing and implementing freshwater quality studies (Bartram and Ballance,

). Today, each country have established their own protocols, water quality standard and permissible limits.

Water Quality Monitoring Programs refer to the acquisition of quantitative and representative information on the existing conditions (physical, chemical and biological characteristics) of a water body. They are designed following, in general, twelve steps according to the diagram shown in Figure . . Identification and definition of objectives are the first two steps in monitoring programs (Smith et al., ). Then the necessary experiments are designed, these need to be structured to meet specific needs according to the objectives defined. The experiments are composed of subsequent steps including: selection of scale, determination of frequency, location and design of stations, selection of variables, selection of sample type, definition of sample collection, definition of analysis methods and definition of land use monitoring.

Scale is defined whether at local, regional or national level. Frequency of sampling varies according to the purpose of the monitoring, the characteristics of the water body and the importance of the sampling station location. In general, monthly and seasonal samplings are made for characterizing water quality over long time periods (e.g., over a year) whereas weekly samplings are done for control purposes. On cases where significant differences are suspected or detected, samples may be collected on a continuous basis. Traditionally, the design of stations is done subdividing the locations into macrolocation and micro-locations. Macro-locations are river reaches sampled within a watershed and micro-locations are sampling points belonging to a macro-location that represent specific points (e.g., discharge of a river, source of pollution) (Harmancioglu et al., ). Today, design of stations is done using remote sensing techniques. By integrating remotely sensed data, GPS or GIS technologies natural resource managers are able to geographically locate a monitoring site (Ritchie et al., ).

Selection of variables depend directly to the objectives established on the monitoring program and their importance vary with the type of water body and uses. Commonly the variables that characterize water quality are classed into: general water quality parameters, dissolved salts, organic matter, microbial pollution, nutrients and inorganic variables. The development of new analytical technologies have allowed the identification of chemical compounds that were not possible to identify with traditional techniques. These compounds are classed as emergent pollutants and are monitored whether in surface waters (Geissen et al., b) or ground waters (Lamastra et al., ). Such organic compounds include among others: pesticides, fungicides, oils, hydrocarbons, organic solvents, phenols, surfactants, pharmaceuticals, fats and hormones (c.f. Table . ).

In addition to the aforementioned variables, ecological indicators (e.g., fish, macroinvertebrates, macrophytes, diatoms, phytoplankton.) are integrated as part of biological monitoring. Though the use of ecological indicators is advantageous (c.f. Section . . ), they only provide information about indirect causes for a given change therefore it is recommended to use them as complementary tools in WQ monitoring activities (Sponseller et al., ; Serrano Balderas et al., ).

Concerning sample type and sample collection, they depend on the variables to be measured (Pegram et al., ). Commonly; water, particular matter and living organisms are collected. Quality of water and particular matter are analysed by measuring physical and chemical parameters whereas living organisms can be used in different ways, for instance: as ecological indicators studying communities of organisms or as toxicological indicators by studying the physiology, morphology or chemical composition of tissues on specific organisms (Dolédec and Statzner, ).

Analytical methods are determined according to the type of sample, the quality parameter to be analysed, amount of sample and whether the sample is analysed in situ

Chapter

Table . -Water quality variables. a Units are in m 3 s -1 . b Units are in mg L -1 . c Units are in µg L -1 . d Units are in • C. e Units are in µS cm -1 . f Units are in mV. g Units are in mg/CaCO 3 . h Units are in number/100 mL. i Units are in mg L -1 Pt/Co [Hazen units]. j Units are in dilution rate at stated temperature. k Units are in FTU (Formazin Turbidity Units) or NTU (Nephelometric Turbidity Units) or JTU (Jackson Turbidity Units).

General water quality

or at laboratory. Generally, water sampling operations include in situ measurements, sample pre-treatment, conservation, identification and transportation. There are numerous methods for field monitoring they range from simple devices to sophisticated equipment (WMO, )(UNEP, )(Meyers, )(APHA/AWWA/WPCF, ). The selection of methods and devices to measure the selected variables depend on the availability of equipment and reagents, degree of expertise and the accuracy needed according to the objectives of the program.

Data management (DM) is an integral part of a monitoring program: it aims at converting data into information that will meet monitoring objectives. DM in WQMP includes analysis, distribution and storage of data. As an important part, data analysis allows to have a basic understanding of data and to determine data quality and quantity (EPA, ). A first insight to understand data can be obtained through summary statistics. Some of the characteristics that can be described include: normality, variability, symmetry of data distribution, missing values or outliers. This kind of information can be useful in order to define the suitability of data to support the intended use and, in most of the cases, to identify problems on data that may affect conclusions on the environmental assessment program (Helsel and Hirsch, ).

. . Collection of data about emerging pollutants in rivers

Today, there are numerous synthetic organic compounds that are used for industrial, agricultural or domestic purposes. They can enter to natural effluents through infiltration, draining, natural deposition or by direct discharge. Except for specific removal by wastewater treatment processes, they may be released into running waters at trace level concentrations (in the nanogram (ng) or microgram per liter (µg l -1 ) range). Some trace pollutants are referred as Emerging Pollutants (EP). Emerging pollutants (EPs) are synthetic or naturally occurring chemicals that are rarely part of environmental monitoring practices but which enter the environment and cause known or suspected adverse ecological and (or) human health effects (Geissen et al., a). Common wastewater processes are not able to treat EPs.

Though release of EPs have been occurred for a long time, the interest to include them in environmental assessment programs has increased over the last decades. Some of the EPs that are found in aquatic environment have been recognized as hazardous to ecosystems (von der Ohe et al.,

). The persistence of EPs, their metabolites and transformation products in natural waters has increased environmental and human health concern, since little is known about their fate, behaviour and ecotoxicological effects (Petrie et al., ). The EPs most constantly studied in the freshwater environments include: industrials, pesticides, pharmaceuticals and personal care products (PPCPs) (Murray et al., ). In Table . we provide a list of some of these compounds.

Industrials are organic compounds used in manufacturing and production processes, pesticides are substances or mix of substances used to destroy, suppress or alter the life cycle of any pest and PPCPs are compounds administrated internally or externally to the bodies of humans and domestic animals. PPCPs are part of the compounds of human health concern because they can exhibit multixenobiotic resistance and show resistance to microbial degradation (Smital et al., ).

As well as in a traditional water quality monitoring program (c.f. Subsection . . ), collection of data for assessment of EPs includes twelve steps (c.f. Figure . ). Due http://www.epa.nsw.gov.au/pesticides/pestwhatrhow.htm to the physico-chemical properties of EPs (e.g., adsorption, volatility, polarity), sample collection and analytical methods are the critical steps. Water quality data that includes EPs is numerical, volume of data may depend on the objectives and scale of study (i.e., local, regional, national) thus, it can vary from to over thousand of individuals (or sampling sites) and from one to hundred of compounds.

Chapter

Uncertainties may occur all along data collection and could be linked mainly to: field sampling problems, transportation of samples, bad sample manipulation or problems with analytical instrumentation. Data anomalies such as missing values, outliers and nondetected data are frequently found when EPs are included in a water quality monitoring program. Missing values and outlying data may occur by different reasons (e.g., difficulties on field sample collection, lost of sample, equipment failure, anomalous conditions). Non-detected concentrations or ND are frequently reported when a concentration of a compound in a sample is less than a specified limit of detection (LOD), the resulting left-censored data preclude subsequent statistical analysis (Clarke, ). For the case of missing values and outlying data different data preprocessing procedures could be implemented to improve the quality of data and facilitate statistical analysis (c.f. Subsection . ). Concerning non-detected data, use of advanced analytical techniques could be advantageous, since advanced techniques allow the detection of EPs at very low concentrations (µg and ng l -1 ).

. . Chemical analysis in water

Samples collected for environmental assessment constitute a representative fragment of the site under study. Such sample contains the target specie called the analyte, which in general is mixed with a number of other compounds constituting the matrix. To identify or quantitatively determine an analyte, separation methods can be used. Such methods are achieved by differences in physical or chemical properties between the constituents of a mixture (Wilson et al., ).

There are different separation methods including: chemical precipitation, distillation, extraction by solvents, ion exchange, electrolysis, and chromatography (Skoog Douglas et al., ; Harvey, ).

• Chemical precipitation is based on the solubility of the compounds. It requires to have big differences of solubility between the analyte and the other compounds. The main drawback is that undesired co-precipitates can be formed making the separation process more complex and slow. • Distillation is used to separate volatile analytes from non-volatile interferences. Big differences of volatility are required to separate the target compound therefore, the use of this method is limited when compounds in a mixture have similar or very close volatility characteristics.

• Extraction by solvents is based on solubility differences between two immiscible liquids. This method involves the selective transfer of an analyte from one liquid phase to another by using a solvent with a polarity similar to the analyte. In water samples, non-polar solvents are used to extract the analyte from water. Compared to precipitation, the extraction method is less complex and faster however, large quantities of organic solvents are needed, emulsions may be formed and it is not suitable for thermally unstable compounds. The process occurs when a water matrix is in contact with an ion exchange resin which contains charged ions. The ions of the matrix are then replaced by the ions of the resin. Resins are composed of organic polymer chains which contain charged functional groups with either positive or negative charge. In a matrix containing multiple compounds the ion exchange separation will not be highly efficient for different reasons: ) the sample need to be passed through different resins in order to separate the analyte from the matrix, causing a lost of sample and analyte; ) it is necessary to know the physico-chemical characteristics of the compounds present in a matrix in order to use specific resins; and ) resins composed of mixed polymer chains are indispensable but they can be costly and there is not guarantee of getting good separation results. • Electrolysis occurs when an electric current is passed through a mixture containing ions. The electric current induce chemical reactions and separates compounds of the matrix. The process occurs in an electrolytic cell which is a device with positive and negative electrodes. The electrodes are then introduced in a solution containing charged ions. When the electric current is passed through the solution, cations travel to the electrode charged negatively (cathode) and are transformed to neutral molecules. While anions travel to the electrode charged positively (anode) to be transformed into neutral molecules. Separation by electrolysis is widely used in metallurgical processes. • Chromatography is a separation process where substances are distributed between two phases named stationary and mobile phase. The mobile phase is a samplefree phase that is passed over a second sample-free phase that remains stationary (stationary phase). The chromatography is achieved by placing a sample into the stationary phase and putting it in contact with the mobile phase, as the mobile phase moves the components of the sample are partitioned between the mobile and the stationary phases.

The basic chromatographic process can be described in the following steps (c.f. Figure . ):

. First, the stationary phase, which is a finely powdered solid, is placed into a vertical hollow glass tube. Then a sample matrix containing the components to be separated is placed at the top of the glass tube; . Then, the mobile phase is added continuously to the glass tube. In this manner, the different constituents of the mixture are carried on along with the mobile phase. This process is known as elution; . Finally, the separated components of the mixture can be recovered. The basic chromatographic process has been largely used for the separation and purification of divers compounds. This technique has been improved greatly. Nowadays chromatographic techniques are composed of divers accessories designed to automatize, control and assure reproducibility of the entire separation process.

In general, the chromatography instrumentation is composed by a mobile phase supply system, a chromatographic column, a detector and an equipment for data acquisition and processing (Figure . ). The mobile phase supply systems is designed to transfer the mobile phase through the chromatographic column. To achieve the separation of the compounds present in a mixture, the mobile phase should be composed of divers solvents or solvent mixtures that will interact with the molecules of the sample to proceed with the elution of compounds. With the help of the mobile phase the components of a mixture are carried out to the chromatographic column. The column contains the stationary phase that allows the separation of compounds. Once the different compounds are eluted from the column they are identified by a detector.

The detector along with the column is an important part of the chromatographic system, it identifies the components separated by the column and allows their quantification. Detectors provide whether a single information (e.g., retention time or combined information (e.g., retention time and structure of the analyte). For this reason, chromatographic systems can be equipped with two or three detectors. The detectors must frequently used include: Thermal conductivity detector (TCD), flame ionization detector (FID), Nitrogen phosphorus detector (NPD), Electron capture detector (ECD), Photo-ionization detector (PID), Mass spectrometry detector (MSD), Infrared detector (IR) and ultraviolet detector (UV). The compounds identified by the detector are represented in a chromatogram. The chromatogram shows the variation of the amount of the analyte in the mobile phase in specific period of time. The chromatogram consist of a peak for each of the separated compounds identified by the detector (Rouessac and Rouessac,

). Considering the physical nature of the two phases involved, chromatographic techniques can be classed into Liquid phase chromatography (LC) and Gas phase chromatography (GC). The liquid phase chromatography uses a liquid as the mobile phase. Depending on the retention phenomenon this category can be sub-divided into liquid/solid chromatography (or adsorption chromatography), ion chromatography (IC), liquid/liquid chromatography (or partition chromatography, LLC), and liquid/bound chromatography. In gas chromatography the mobile phase is an inert gas (i.e., helium, hydrogen or nitrogen). Gas chromatography can be sub-divided into gas/liquid/chromatography (GLC) and gas/solid chromatography (GSC).

When measuring an analyte at trace level within a matrix, interferences between analyte and other compounds may occur. Therefore it is necessary to meticulously prepare a sample before its analysis. Some methodologies for sample pretreatment include: solid phase extraction (SPE), immunoaffinity extraction, microextraction procedures, gas extraction on a cartridge or a disc, headspace, supercritical phase extraction and microwave reactors. Among these methodologies the SPE is the most frequently used. SPE is used to purify or concentrate an extract prior to the analysis of its constituents (Pawliszyn, ). It is implemented using a plastic cartridge similar to a syringe which contains a solid sorbent upon which a known volume of liquid sample is passed.

In the initial step of the SPE method the target compound is retained by the sorbent, then undesired substances are eliminated by rising them. Finally a small volume of a strongly enriched solution is recovered using an appropriate solvent. This solution contains the analyte. The SPE procedure isolates and pre-concentrates the analyte, this is particular useful when analysing traces. An analogous methodology to SPE is the liquid/liquid extraction, where large volume of solvent is used to dilute the analyte. The main drawbacks of the liquid/liquid extraction are: ) large volume of solvents are necessary and ) impurities along with the analyte are dissolved by solvent so the extracted solution contains both the analyte and impurities which makes the method unacceptable.

Today, numerous analytical techniques are available, most of them include the use of

The time a solute takes to move from the point of injection to the detector (Harvey, ). Inert gas is a gas that do not undergo chemical reactions, they are used to avoid unwanted chemical reactions A compound is considered to be in trace amount when its concentration in the environment is less than 1000 ppm (1 µL L -1 or 1 mg kg -1 ) Liquid Chromatography tandem Mass Spectrometry (LC-MS-MS) or Gas Chromatography Mass Spectrometry (GC-MS). They have been used for the analysis of various compounds in wastewater, surface and ground waters (Robles-Molina et al.

; Bruzzoniti et al.

; Ma et al. ; Nebot et al. ; Mansilha et al. ).

In Mexico, different efforts to advance on the study of EPs have been made. Some of the analytical methods that have been proposed for the analysis of EPs in Mexico include: Supercritical-Fluid Chromatography coupled with Diode-Array Detection (SFC-DAD) (Salvatierra-Stamp et al., a), Selective Elution and analysis by Gas Chromatography-Mass Spectrometry (GC-MS) (Gibson et al., ), Liquid Chromatography with a Photodiode Array Detector (LC-PAD) (Salvatierra-Stamp et al., b), Solid Phase Micro Extraction followed by Gas Chromatography-Mass Spectrometry (SPME-GC-MS) (Peña-Álvarez and Castillo-Alanís,

), Liquid Chromatography (LC) with post-column fluorescence detection (García de Llasera and Bernal-González, a) and Liquid Chromatography diode array UV detection coupled on-line to a Solid Phase Extraction (SPE) (García de Llasera and Bernal-González, a). These methods can be used to analyse different EPs in water samples including: pharmaceuticals (e.g., carbamazepine, carbofuran, glyburide), endocrine disruptors (e.g., αethinyl estradiol, bisphenol A, β-estradiol), bactericides (i.e., triclosan) and pesticides (carbamates, organochlorine, organophosphorus). In addition to the above mentioned methods, some others were also developed for the analysis of EPs in soils (García de Llasera and Bernal-González, b) and sediments (Alcántara-Concepción et al.,

).

The works published on environmental assessment of EPs in Mexico are still limited, but the already published work gives evidences of the importance to conduct studies on this topic, and most importantly, to include EPs in the monitoring programs to assess the quality of Mexican surface waters.

.

Data collection from biomonitoring in water quality assessment

The growing need for controlling good surface water quality and healthy aquatic ecosystems demands an increasing effort to monitor the quality of aquatic ecosystems. This section presents the importance of using macroinvertebrates-based monitoring metrics as a rapid, low cost, and reliable alternative to monitor the quality of surface waters. In the following subsections a state-of-the art on biomonitoring and collection of data from biomonitoring practices using macroinvertebrates is presented.

. . Biological organisms in biomonitoring

Aquatic ecosystems are affected by a number of interrelated physical, chemical, and biological factors. The continuous degradation of surface waters due to natural or anthropogenic causes prompts an interest to the monitoring and management from a regulatory approach to a holistic ecosystem approach (Pinto et al., ). In this manner, biological monitoring has become an effective tool for the assessment of water quality and aquatic ecosystem integrity (Barbour et ).

Biomonitoring or biological monitoring is an environmental survey using biological variables. Biotic communities present changes in their structure, composition and behaviour when the physical, chemical or biological attributes of a river are altered. Therefore, aquatic ecosystems are increasingly monitored by observing the overall responses of a biocenosis able to integrate over space and time all the stressors undergone by the ecosystem.

Usually biomonitoring tools concern only a part of biocenosis, e.g., fish. Their application supposes to know biological organisms currently present in a geographic site and includes a listing of these organisms. Today, different aquatic organisms are used to assess biotic integrity of streams including: bacteria, protozoans, diatoms, algae, macrophytes, macroinvertebrates, and amphibians (Amaral Pereira et al.

; Barbour et al. ; Kane et al. ; Lane and Brown ). Each group responds to disturbances differently, providing diverse information. Thus, the group of assemblage is chosen depending on the objective of the study and the characteristics of the ecosystem to be assessed. Biomonitoring of lotic systems; i.e., rivers, is done using metrics based on periphyton, benthic macroinvertebrates and fish communities (Li et al., ). Metrics based on periphyton and fish communities are good indicators of habitat/hydro-morphological alterations. While metrics based on macroinvertebrates have a better response when enrichment of pollutants (like NO ) is the main stressor affecting streams integrity (Johnson et al., ).

Use of macroinvertebrates on bioassessment practices present different advantages (Bonada et al. ; Rosenberg and Resh ) including:

. their benthic nature, which allows a spatial analysis of pollutants;

. their relatively long-live cycles, that allows to follow environmental changes over long periods of time (they provide evidence of conditions of about last months), contrary to diatoms that can reflect environmental changes only for the last months;

a self-sufficient community of naturally occurring organisms occupying and interacting within a specific biotope.

Chapter

. their sensitivity to different types of chemical and/or hydro-morphological alterations; . their abundance and diversity: a large number of species produces a range of responses to a wide variety of disturbances; . they are easy to identify to a family level (at this level of identification, they provide in general, enough information of alteration level), compared to periphyton; . sampling is easy to implement, requires few people, and equipment is relatively inexpensive; compared to fishes; . and their ubiquitous occurrence.

. . Ecological assessment of rivers using macroinvertebrates

Assessment of environmental condition using biological organisms has been historically practised in developed countries (Cairns and Pratt, ). Today, analysis of macroinvertebrates communities in rivers is carried out using mainly structural and taxonomical approaches, this means relying on the presence/absence, sensitivity, richness, abundance, and diversity of particular taxa. All this information is then converted into numerical values such as indices and scores.

The main steps on rivers assessment using macroinvertebrates include: ( ) field sampling, ( ) sample treatment, and ( ) interpretation of data.

. Field sampling: Sampling can be implemented in a representative fragment of the water body. To select a representative fragment, it is necessary first to record the specific locality, topography, and drainage characteristics of the aquatic ecosystem habitats. Locality can be specified by latitude, longitude, section number, habitat location (country, state, province or township), and elevation of the study area above the sea level. The topographic description of the study area includes: the type of water body (creek, river, pond, lake or reservoir); surface features (slope and form of the surrounding terrain and shoreline, form of stream channel and formations such as riffles, rapids, falls or islands); size of the water body, and its approximate centre and average depths; current velocity and sediment structure. For drainage characteristics, the major drainage system should be identified along with the name of the water body as well as the smaller watershed that drains the body water (Brower et al., ). The characteristics of the aquatic habitats are then used to select the sampling methodology and tools. All habitats or specific habitats (e.g., habitats with uniform and clearly defined conditions) can be sampled (Jáimez-Cuéllar et al., ). Within the studied fragment of body water, samples are collected generally from mineral and vegetable (alive or dead) substrates and from slow and fast water's course speed. Samples can be collected with the help of different devices (APHA ; Ghetti ; Rosenberg and Resh ). The most commonly-used ones include: nets (e.g., handnet, surber net, kick net, D-frame net, dredges), grabs, core samplers, artificial substrates and drift. Important for the nets is the mesh size which should be between 250 µm and 500 µm. Sampling can be completed by hand, picking macroinvertebrates that may be situated on hard substrates (stones or plants). Biological organisms are frequently preserved in % ethanol for posterior examination.

. Sample treatment: Examination of macroinvertebrate samples can be performed in situ or soon after capture, or most frequently later in a laboratory, (Bervoets et al., ). Samples are sorted using a pile of sieves of , , , or 1250 µm in order to separate different size of substrates and of macroinvertebrates, to make easier the visualization of organisms. The sample can be cleaned with running tap water. Sieve contents should be placed in white trays to sort the organisms. The organisms can be examined under a stereoscope and finally be conserved in glass or plastic containers with % ethanol (APHA, )(De Pauwn and Vanhooren, )(Ghetti, )(U.S.E.P.A, ). Family or genus are the level of taxa identification the most adopted in many biomonitoring programs (Thorne and Williams, ) where family is considered the minimal identification level recommended (Hilsenhoff, ). Taxa is identified by the use of keys of local taxa.

. Interpretation of data: All the information obtained from the identification and quantification of taxa is then treated to compute various metrics (i.e., richness, diversity, similarity, biotic indices, functional traits, multimetric approaches). Finally, analysis of data is performed applying different statistical methods. The selection of the statistical method depends mainly on the purpose of the study and may eventually combine the biological metrics with environmental metrics (e.g. chemical or physicochemical parameters). For instance, graphical techniques are used to provide rapidly accessible information of a selected number of variables while multivariate statistical techniques can be used to test the degree of similarities in a larger number of variables. Some differences on the performance of graphical, multivariate and classification methods are described in Güler et al. ( ) and Walley and S.Džeroski (

).

The idea of using aquatic organisms as biological indicators of environmental condition was first introduced by Kolkwitz and Marsson (

). They introduced the concept of saprobity which refers to a measure of organic pollution in rivers and the associated decrease in dissolved oxygen. Since then, different approaches have been developed. Among them, Metcalfe ( ) has distinguished three major approaches: the saprobic, diversity, and biotic approaches. In recent years, however, new approaches have been developed, e.g., multiple biological traits, multimetric and predictive approaches. Below we describe the most common approaches. They are classed into five groups (Resh and Jackson, ) which are: ( ) measures of richness; ( ) enumerations; ( ) diversity and similarity indices; ( ) biotic indices, and ( ) functional traits. In addition to these groups, there are multiple biological traits, multimetric, and predictive approaches.

. Measures of richness: There are as many measures of richness as there are macroinvertebrates orders and levels of identification: the most frequently-used for macroinvertebrates are family, genus and species. Orders of EPT taxa are considered to be sensible to anthropogenic stressors. Therefore, they have been widely used as indicators of environmental disturbances (Wallace et al., ). Five measures of richness are the most commonly used (c.f. Table . )

Table . -Measures of richness (Barbour et al., ) (Resh and Jackson, ). . Enumerations: Concerning enumeration metrics, it is considered that some stresses increase or decrease the total numbers of individuals of some taxa. Some measures are based on the identification of pollution-sensible groups and their relative abundances to the total abundance of macroinvertebrates (e.g., Ratio of EPT abundance to Chironomidae abundance). The most commonly used include five metrics (c.f. Table . ) (Merrit et al., b). They simple consist on counting and classifying individuals, to finally compute the corresponding ration of each metric. . Diversity and similarity indices: Diversity approaches use richness, evenness, and abundance to evaluate the community structure with respect to the occurrence of species (Li et al. ; Merrit et al. b). These approaches are based on the principle that stressed or polluted water will lead to a reduction in diversity on the community. The main problems of diversity approaches is the unclear reference level and the considerable variations on the diversity of natural undisturbed waters. The use of diversity on water quality monitoring relies on the idea that balanced, stable communities will be represented by high diversity index values. Similarity indices compare community structure between sites. Thus, communities at disturbed and undisturbed sites, will present remarkable dissimilarities. Diversity and similarity indices have been strongly criticized when employed separately in assessment of river systems (Metcalfe, ) and today, they are preferably used jointly with other metrics (e.g., multimetric approaches) in order to integrate the behaviour of the elements and processes of biological systems (Karr, ). In Table . , the most commonly used Similarity and Diversity metrics are listed. It must be noticed that Bray-Curtis distance is a dissimilarity index, its interpretation is teh opposite of teh results obtained with the other similarity indices but provides relevant information. . Biotic indices: Biotic approaches combine measures of diversity with the ecological sensitivity of individual taxa into a single numerical expression. The principle of biotic approaches is that macroinvertebrate groups disappear as pollution increases and a reduction of the number of taxonomic groups is observed when pollution increases. The main disadvantage of these approaches are the difficulty to determine representative reference communities and the uncertainties that can be generated when biotic approaches are adopted in geographic regions different from which they were originally designed. Biotic indices are calculated as follows: first, different values are assigned to macroinvertebrate taxa according to their sensitivity or tolerance: then, all individual values of the taxa present in the studied sample are summed up to finally generate a single index or score. The values assigned to macroinvertebrates differ among taxa according to their sensitivity and tolerance to pollutants. Those values are constructed more commonly taking into account the tolerance of macroinvertebrates to: pH, organic pollution, eutrophication, heavy metals, and pesticides.

Measure name Common expression

From the numerous biotic indexes that have been developed (Cairns and Pratt ; Metcalfe ) six are the most widely used (c.f. Table . ). Among them, the Biological Monitoring Working Party (BMWP) has been adopted and modified for effective use in several countries, such as Spain (Alba-Tercedor and Sánchez-Environmental data analysis (Cairns et al., ) Margalef Index D = S -1 log N (Margalef, ) Similarity indices Jaccard's Coefficient Ortega, ), Australia (Chessman, ), India (De Zwart and Trivedi, ), Costa Rica (Mafla, ), and Colombia (Roldán Pérez, ). . Functional traits: Macroinvertebrate traits are characteristics that have been adopted as a complementary and indirect approach to reflect ecological integrity (Statzner et al., b). This approach is based on the habitat characteristics and the biological and ecological functions of species. Thus, by relating species traits to habitat characteristics, important insights into the structure and functioning of streams communities can be observed (Bremner et al., ; Kilbane and Holomuzki, ; Poff et al., ; Statzner et al., a). Traits are classified in two categories mainly: biological and ecological. Biological traits (i.e., size, body form, life cycle, food and feeding habits, reproductive strategies, mobility, etc.) describe the biology of the species. Ecological traits (i.e., pH and temperature tolerances, bio-geographic distribution, tolerance to organic pollution, etc.) are related to habitat preferences.Usseglio-Polatera et al. (

C j = j a+b-j ( Jaccard
) have proposed twenty-two (eleven biological and eleven ecological traits).

Functional traits measures organize species into functional feeding froups (FFGs) according to their morphological behaviour, food-gathering mechanisms or locomotionattachment adaptation (Merrit et al., b; Tachet et al., ). Each of these FFGs are expected to vary significantly, they can increase or decrease upon accumulation or loss of particular food sources, presence of pollutants or with particular habitat types. FFGs indicate perturbation of the community when deviations of expected abundances of FFGs or habit groups occur (Merrit et al., b). FFGs have been applied by several authors to assess perturbations on the aquatic ecosystem such as: land use effects (Dolédec et al., ), high-flow disturbance (Holomuzki and Biggs,

), anthropogenic effects (Usseglio-Polatera and Beisel, ), organic pollution (Lafont et al., ) or monitoring of water quality (Bady et al., ; Charvet et al., ; Statzner et al., ) and, in recent years, they have been applied together with other metrics (i.e., multimetric approaches). The most commonly used are: the percentage of filtering collectors, the percentage of scrapers, the percentage of gathering collectors, the percentage of predators and the ratio of scrapers to filtering collectors (scrapers/filtering collectors).

. Multimetric approaches: Multimetric approaches provide robust and integral responses of an assemblage to natural and anthropogenic stressors. They combine various measures of richness, enumerations, pollution-tolerance values, functional feeding groups, dominance, life cycle and density. The Index of Biotic Integrity (IBI), introduced by Karr ( ) was the first multimetric approach, which was designed to assess fish assemblages. Inspired by the IBI index, different multimetric approaches have been proposed and adopted in different countries including: France (I M (Mondy et al., )), Germany (Vlek et al., ), India (Sivaramakrishnan et al., ), East Africa (IBI-LVB (Masese et al.,

)), Brazil (Multimetric approach for Central Amazonia region (Couceiro et al., )), Panama (NLSMI (Helson and Williams, )) and Mexico (IBIwest-central-Mx (Weigel et al., )). Multimetric indices are potential and efficient assessment tools because they integer individual metrics that consider different attributes of communities in order to respond to different types of pressure. Nevertheless, these multimetric approaches are region-specific and cannot be implemented globally.

. Predictive approaches: Predictive models assess river health by comparing reference sites with altered ones. The assumption on these models is that the least-impacted sites with similar environmental characteristics should have similar fauna patterns in the absence of anthropogenic impact. In this context, predictive models have been proven to be useful in biomonitoring activities.

Both multimetric and predictive approaches use biotic indices. The predictive model approaches most frequently used are the Biological Monitoring Working Party score (BMWP) and the Average Score Per Taxon (ASPT) (Norris and Hawkins, ). The multivariate predictive models that are widespread used include:

• RIVPACS (River Invertebrate Prediction and Classification System (Wright et al., )); • AusRivAS (Australian Rivers Assessment System (Smith et al., )); • BEAST (Benthic Assessment Sediment (Reynoldson et al., )) and • ANNA (Assessment by Nearest Neighbor Analysis (Linke et al., ))

Inspired by RIVPACS, some other models have also been developed such as: Mondego (Feio et al., ), Medpacs (Poquet et al., ) or the predictive model for Bolivian streams (Moya et al., ). But so far, due mainly to the absence of reference sites and incomplete information on rivers functions, there is no equivalent model developed in Mexico. Some of the foreign biomonitoring metrics that have been used to assess Mexican aquatic ecosystems using macroinvertebrates include: the Hinselhoff Family-level Biotic Index (Henne et al., ) (Barba-Álvarez et al., ), the Beck Biotic Index (Rosas et al., ) and the Extended Biotic Index (López-Hernández et al., ). Nonetheless, they have been used only on isolated cases or for a particular study case. In fact, the use of macroinvertebrates for bioassessment of lotic systems in Mexico is still scarce (Mathuriau et al., ). This is due mainly to: untrained personnel, unknown information of pristine communities, absence of sampling material, lack of identification keys, absence of local experts and incomplete information on macroinvertebrates functions on Mexican rivers (Resh, ).

Applicability of biological monitoring methods in developing countries can be always debated. For instance, in a study published by Damanik-Ambarita et al., a carried out in different tropical river basins (Ecuador, Ethiopia and Vietnam), it was observed that different biological metrics, presence and absence of sensitive/tolerant taxa as well as relative abundance of macroinvertebrates families could be used to assess the quality of the studied rivers. They particularly observed that functional traits, like FFG (Functional Feeding Groups) were promising. In other study published by Damanik-Ambarita et al., b different biomonitoring metrics were used in Middle and South America Rivers. To study the ecological water quality in Ecuador (Guayas river basin), they selected physicochemical data (temperature, conductivity, turbidity, Chlorophyll, nutrients, organic matters) and two macroinvertebrates indices: the BMWP-CO (Roldán Pérez,

) and the Neotropical Low-land Stream Multimetric Index (NLSMI) (Helson and Williams, ). The NLSMI's metrics are : % of scrapers, % of shredders, Margalef's index, % of chironomidae and % of Diptera, % of Trichoptera and Shannon Index, which are among the metrics we recommend. They used these metrics because macroinvertebrates' responses towards environmental changes might vary across sites and habitats (García-de la Parra et al., ; Helson and Williams, ). However in the above mentioned publications they did not study micro-pollutants. In our study we plan to study macro-pollutants and micro-pollutants as chemical variables, we choose to keep all the biotic metrics and their potential correlation with abiotic data because, ) as showed by Damanik-Ambarita et al., a the relationship between macroinvertebrates communities and habitat disturbance are still lacking and poorly understood in South America, and ) as explained previously, bioassessment of freshwaters in Mexico is scarce.

. . Biomonitoring data: general characteristics, common uncertainties and anomalies

Biomonitoring data consist of a list of fauna living in the rivers. The term taxa is used to define the family, sub-family or species from which a giving individual belongs. Fishes, oligochaetes or macroinvertebrates could be part of the fauna on a biomonitoring study. The resulting list of taxa is used to compute biological indices that provide a numerical value. This numerical value is then used as an indicator of the water quality characteristics of river. In the previous section (c.f. . . ), we described the different indices that can be computed. Each index provides different information that scientist can integer to have a general overview of the characteristics of a river.

Uncertainties in biomonitoring data are ubiquitous. They may occur from collection to Chapter analysis of data. Uncertainties may occur due to differences on: sampling protocols, number of volume of samples, type of substrates, analysis of samples by inexperienced personal, sorting method, level of taxa identification or rare specimens consideration (Wiederkehr, ).

Misclassification of taxa, is one of the most common errors found on biomonitoring data. It has an important impact on the boundary between good and moderate rivers water quality status (Wiederkehr et al. ; Haase et al. ; Metzeling et al. ). Another common error is missing data, that may occur mainly to lost of samples.

.

Preprocessing and analysis of environmental data

Data exploration and analysis are the basis to discover knowledge, understand studied phenomena, solve problems, and make decisions. Data quality is an important issue on data analysis, bad data quality may be significant costly (Haug et al., ) and induce misleading conclusions (Wahlin and Grimvall, ). Quality data is mandatory particularly on environmental data because erroneous data may lead to faulty conclusions leading expensive decisions and causing dramatic consequences on environmental systems. Aiming at improving the quality of data for the particular case of water quality, Rieger et al. (

), Alferes et al. ( ), Alferes and Vanrolleghem ( ) have proposed to evaluate and validate data from in situ measurements. Such evaluation allows to detect data anomalies and implement corrective actions for on-line monitoring systems however, most of the monitoring systems are not on-line. Thus, different approaches are necessary in order to correct data anomalies and improve the quality of data.

Anomalies in data are ubiquitous and may have dramatic consequences in data analysis (Eppler and Helfert, ), limit performance of statistical procedures (Cortes et al., ; Coussement et al., ) and produce misleading analytical results (Wahlin and Grimvall, ). To mitigate the impacts of data anomalies such as missing values, inconsistencies, outlying data, duplicates, etc., it is a first necessary step to preprocess data (Famili et al., ; Gibert et al., ).

Data preprocessing is a fundamental and critical step in data analysis, it consists of all actions necessary to prepare the data before data analysis. In general it is performed to: ( ) solve data problems, ( ) understand the nature of data and perform more meaningful data analysis, and ( ) extract meaningful knowledge from a given dataset and application domain (Famili et al., ).

Although, numerous procedures to preprocess data anomalies are available, there are many inherent challenges in this area, for instance:

. Conventional approaches usually treat data anomalies as independent types of anomaly and handle them in isolation. Preprocessing approaches that take into consideration jointly different types of data anomalies still need to be developed; . Anomalies co-occur in data. It is crucial to define a data preprocessing ordering that produce the results with the least bias. This means that preprocessing procedures should not impact statistical results; . The impact of data preprocessing procedures on statistical analysis results is under studied. Thus, it is important to conduct this type of studies; . It is necessary to define a systematic framework that will allow users to perform the most appropriate preprocessing procedure according to a specific statistical analysis task. . Very few research is focused on environmental data preprocessing.

In this thesis, we focus on the different strategies to preprocess data anomalies and particularly, on their impact on statistical analysis results. We have conducted a bibliographic study related to data anomalies, data preprocessing procedures, and their impact on statistical analysis. In the following section, we briefly describe data anomalies and related detection methods. Then, we describe data preprocessing procedures for missing values, outliers, and duplicates.

. . Data anomalies and their detection . . . Data anomalies

Abnormal patterns in data (or data glitches) are errors in the measurement and recording of data that negatively impact analysis. They are ubiquitous and occur due to a variety of reasons from human errors (e.g., typos) to software and hardware problems (e.g., inconsistencies due to failures of automated equipment) (Berti-Equille et al.,

). Data may be distorted during the collection step, or when data is transcribed, merged, transferred or copied (De Veaux and Hand, ).

Different types of data glitches may be found. According to the taxonomy of data distortion proposed by Kim et al. ( ), we can identify three main classes: missing data, not missing but wrong data, and not missing and not wrong but unusable.

We focus on data glitches related to missing data and not missing but wrong data. Hereafter, we refer only to numerical data problems. Data glitches, detection methods, and preprocessing procedures related to non-numerical data are out-of-the scope of this work and will not be addressed. Hereafter, we describe the data glitches most commonly encountered.

Missing data: Missing data are instances where no data is stored for a given variable in the current observation. They may occur due to failure in equipment or human errors. Let us consider a dataset which composed by a table with rows (observations) and columns(variables). A dataset is considered incomplete if there are at least one variable with at least one missing data. A complete dataset is then defined as

Y com = (Y obs , Y mis ) ( . )
where Y obs corresponds to the "observed" part and Y mis to the missing part.

To better understand the behaviour of missing data it is necessary to identify them within a dataset Y com . In the dataset indicator variables R are defined, they identify what is known (R = 1) and what is missing (R = 0). R is known as the missingness which is represented by a matrix composed by the indicator variables linked to the variables of Y com . The pattern of missingness depends on the structure of missing values of an incomplete dataset. It can be: (a) univariate if only one variable has missing values, (b) monotone if the variables in the dataset can be classified from left to right so that if a value is missing from Y mis i so it si also for Y mis i+1 , and (c) arbitrary (c.f. Figure . ). Missing data are considered to be MAR when the distribution of missingness does not depend on the missing parts. This can be expressed as

P (R/Y com ) = P (R/Y obs ) ( . )
This equation indicates that all the information on the missing part (Y mis ) of the data is contained in the observed part (Y obs ).

An special case of MAR is MCAR, which occurs when the distribution of missingness does not depend on both Y mis and Y obs .

P (R/Y com ) = P (R) ( . )
Missing data is considered as MNAR when the distribution of missingness depends on Y mis (Schafer and Graham, ).

Considering for example the structure of univariate data in Figure . , where the variable X is known for all individuals but the variable Y is missing for some individuals. MCAR means that the probability that Y is missing for an individual it does not depend of values in X or Y . MAR means that the probability that Y is missing may depend on X but not on Y , and MNAR means that the probability of the existence of the missing values depend on Y .

The previous definitions describe the statistical relationships between a set of data and their missing values while ignoring the causes of the gaps. It is assumed now that these causes can be encoded by a group of variables Z. These group of variables may have, for example, variables that will explain the reasons for which some individuals have missing values. It is possible that the variables that cause missing values are not present in a dataset, however some of these variables may be related to X and Y and so, relation between X, Y and R can be deduced.

By setting Z as a variable of cause that is unrelated to X and Y , MCAR, MAR and MNAR data can therefore be represented by the graphical relationship of Figure . . On MCAR it is necessary to have the causes of missing data entirely contained in Z. MAR allows some causes of being related to X. On MNAR, it is required to have some causes related to Y once the relationship between X and R have been taken into account (Schafer and Graham, ).

In general a dataset is necessarily either MNAR or MAR.

Inconsistent and faulty data: Inconsistent data are considered as data that do not respect certain constraints (e.g., domain expert or business constraint). Outliers for instance may be considered as inconsistencies in the case that data do not respect a given statistical model as a constraint such as distribution or density. And our focus is on outlying values as a type of inconsistencies that are model-based. Inconsistencies may arise from misreading instruments or misrecording values at any level. For instance, when "John F. Kennedy" is recorded as "Jhon F. Kenedy", this may be easy identified as a typo error. There may be more complex cases for instance: " ---" that is entered as " ---". Another example may be when subjects do not follow questionnaire instructions by answering questions that they were supposed to Outliers: These are unexpected data that do not conform to a given model. According to Chen et al. (

), an outlier is considered as a pattern in data that does not comply with the general behaviour of the data. Park et al. ( ) identify two types of outliers: (a) outliers caused by equipment failure or measurement errors and (b) outliers reflecting ground truth. Outliers occur due to human errors, instrument errors, natural deviations in populations, fraudulent behaviours and changes or faults in the systems (Hodge and Austin, ) Duplicates: Duplicates are repeated entries. They may not share common identities (Sarawagi and Bhamidipaty, ; Berti-Équille, b). These kind of glitches occur frequently in data integration from different databases, they are defined as data heterogeneity problems (Chatterjee and Segev, ). Two types of data heterogeneity can be distinguished: structural and lexical (Elmagarmid et al., ). Structural heterogeneity arise when databases are structured differently. For instance, user name might be recorded in one field Name, while, in another database the same information might be recorded in multiple fields, say, First Name, Last Name. Lexical heterogeneity occurs when different representations are used to refer to the same object in multiple databases (e.g., George Walker Bush versus G.W. Bush).

Undocumented data: Data may be unusable when proper data documentation is lacking. Undocumented data is a very important problem because, without an appropriate data dictionary, many suppositions can be made. For instance, a record can be composed of a set of numbers as follows:

. One can suppose that is a telephone number, or a membership number, or many other possibilities. However, this is still a guess: without proper data documentation this type of data is unusable (Berti-Équille and Dasu, ; Dasu, ).

. . . Detection of data anomalies

There are different tools in the fields of statistics, data mining, and data management to detect problems in data. Detection methods are mainly classified into two categories (Dasu, ): ( ) quantitative methods based on statistical and data mining approaches, and ( ) methods based on constraints developed from data properties and functional dependencies, or manually defined by experts.

We focused mainly on statistical methods and particularly on methods to detect missing and outlying data. We review techniques for the detection of such anomalies. We also briefly review some approaches for detection of inconsistent and duplicated data.

Data profiling is performed by using statistical methods to get standard characteristics of data (i.e., data types, granularity, format patterns, content patterns, value sets, and cross-column relationships) (Sebastian-Coleman, ; Abedjan et al., ). This first exploration of data allows to detect mistakes, check assumptions, preselect appropriate models and determine relationships among the explanatory variables. The statistical methods used in data profiling are either univariate or multivariate.

Missing data. Number, pattern of missing observations (MAR, MNAR, MCAR) and plausible reasons for missing data are useful information for an appropriate treatment and analysis (Pigott, ). Number of missing observations is an important information. Lower percentage of missing observations may imply the use of simple methods to process them though, there is not a consistent definition of "low percentage of missing observations" they may go to % to lower values to be processed (Little and Rubin, ).

Currently numerous statistical software's allow exploratory data analysis (e.g., SAS, ADaMSoft, IMB SPSS Modeler, JASP, R, RapidMiner, STATISTICA, MATLAB, WEKA). Must of them include packages/modules that allow the detection of missing values. We can mention for instance the package VIM available in the R environment for statistical computing which allows the exploration and analysis of missing observations (Templ et al., a).

Outliers detection. Considering the definition given by Hawkins (

) "An outlier is an observation which deviates so much from the other observations as to arouse suspicions that it was generated by a different mechanism". Thus, the purpose is to detect observations that fit into this description. There are many methods to detect outliers, some of them include: statistics-based (proximity-based, parametric, non-parametric and semiparametric), distance-based, density-based, deepness-based, deviation-based, clusteringbased, neural networks (supervised and unsupervised), machine learning, etc. (Huang et al., ; Kriegel et al., ). We will just describe three outlier detection approaches in detail. A more detailed description of the different methods can be consulted on the survey of Kriegel et al., .

• Statistic-based outlier detection approach. This approach assumes a distribution or probability model for the given dataset then a discordancy test is used to identify outliers with respect to the model. A discordancy test examines a null or working hypothesis and an alternative hypothesis. The working hypothesis, H 0 :

x i ∈ F, i = 1, 2, ..., n,
assumes the entire dataset of n records comes from the same distribution model F .

In the alternative hypothesis, H 1 it is assumed that data comes from another distribution model G, H 1 :

x i ∈ G, i = 1, 2, ..., n.
The test of discordancy verifies whether an object x i is significantly large or small in relation to the distribution F . Many tests have been proposed. A review of the most frequently used has been done by Barbato et al. ( ). Briefly, they include: Interquartile range (IQR), modified IQR, Pierce, Chauvenet, Grubbs and Dixon's methods. Some of the major disadvantage of the test of discordancy are: ( ) they assume that data follows a normal distribution. This may be misleading, particularly when data does not follow a normal distribution; ( ) results depend on chosen model F . Thus, x i may be an outlier under one model and a valid value under another and; ( ) they are appropriate for one-dimensional data but not applicable to multi-dimensional data (Barnett and Lewis, ; Barbato et al., ). Detection of outliers in multivariate data is more complex than univariate data because outliers may be masked in the bulk of multi-dimensional data. Multivariate outlier detection methods consider the variations of multiple variables, they depend frequently on a distance measure. Such distance measure attempts to measure the distance between an observation and the centre of the data distribution, by considering the inherent variability and correlation in the data. Multivariate outlier detection methods are frequently based on Mahalanobis distance (Filzmoser, ; Riani et al., ; Berti-Équille and Dasu, ; Kriegel et al., ).

• Distance-based outlier detection approach. Distance-based approaches measure similarity between two objects with the help of distance between them in the data space.

When this distance exceeds a particular threshold, the object is considered as an outlier. In a given dataset X, an outlier DB(p, d) with parameters p and d, will be considered as such if a fraction of p of the object x in X lies at a distance greater than d from x. This approach is suitable for datasets that do not fit any standard distribution model, it discovers outliers effectively and it can be used on multivariate data. The k-NN approaches such as (Knorr and Ng, ; Knorr et al., ; Ramaswamy et al., ; Byers and Raftery, ) are the most popular.

• Density-based outlier detection approach. In this approach, the degree of an object to be an outlier is measured with respect to the density of the local neighbourhood. This degree named local outlier factor (LOF) is assigned to each object (Breunig et al., ). Compared to other approaches, the density-based approach assigns an outlier factor, which is the degree the object is being outlier. Different density-based outlier mining algorithms have been proposed, for instance: the local correlation integral (LOCI) (Papadimitriou et al., ), the relative density factor (RDF) (Ren et al., ) or the density-similarity-neighbour based outlier factor (DSNOF) (Cao et al., ) to mention some. The singularity of the density-based outlier detection and its ability to detect outliers has become an attractive approach to detect outliers (Chen et al., ).

Although, the selection of appropriate outlier detection methods is an important task when determining relationships among the studied variables (Alameddine et al., ; Filzmoser et al., ) an universal outlier detection method has not been identified, rather, different aspects may be considered for the selection of a particular method including: characteristics of the data (e.g., dimension, type, sample size), scalability, speed, model accuracy, algorithm efficiency and performance. A detailed insight of the pros and cons of different outlier detection techniques has been provided by different authors (Reimann et al., ; Pasha, ; Chen et al., ; Filzmoser, ; Zimek et al., ; Barbato et al., ; Ghosh and Vogt, ; Cousineau and Chartier, ).

For the case of the detection of inconsistent and duplicate data, various techniques based mainly, on constraints have been developed, some of them are described below.

Detection of inconsistencies.

There are various approaches for the resolution of inconsistencies (Dongilli et al., ; Kolev et al., ). Data inconsistency occurs at the level of data representation. This kind of conflict is based generally on the content and not the schema and so it can be resolved by simple conversion in the case of numerical values.

Detection of duplicated data. Typically, detection of duplicated data relies on string comparison techniques to deal with typographical variations. A detailed survey of methods to detect typographical variations has been made by Elmagarmid et al. ( ). Briefly, they are classed into: character-based similarity metrics, token-based similarity metrics, phonetic similarity metrics and numeric similarity metrics. Some other approaches to detect duplicates in data streams have been proposed by Deng and Rafiei ( ), Shen and Zhang ( ), Gopalan and Radhakrishnan ( ) and Wei et al. ( ).

For duplicated data where multi-instance tuples are versions of each other, a tuple similarity measure is adopted. The tuple similarity measure is used to map each tuple to a binary vector in a semantic vector space, then the similarity between tuples of the multi-instance is computed as one of the vector coefficients (e.g., matching coefficient, Dice coefficient, overlap coefficient or Jaccard coefficient). Two tuples can have high probabilities of being versions of each other if they are the most similar (Anokhin et al., ).

In data from water quality assessment, the anomalies most frequently found are missing values, outlying data, duplicate data and inconsistent data.

. . Dealing with data anomalies and main preprocessing procedures

The best way to improve the quality of data is to prevent errors since the data collection step. Though, even if good data collection practices are implemented, data glitches may inevitable occur. When data glitches occur, different procedures can be implemented to process them. It is important to notice that the preprocessing may differ depending mainly on the purpose of the study and the statistical method to be used. Below we give an overview of features selection and normalization procedures then, we continue presenting an overview of relevant methods to process data anomalies, particularly missing values and outliers.

Data preprocessing is a mandatory task needed to convert raw data into new data that serve as input for a certain data mining (DM) algorithm. Data preprocessing involves different subtasks mainly: data integration, data cleaning, data reduction and data transformation (García et al., ; Hellerstein, ).

In data integration, multiple data sources (e.g., databases, data cubes, flat files) are combined to construct new tuples and values. Data cleaning involves all operations necessary to correct anomalous data (e.g., treating missing or outlying data, removing noise, resolving inconsistencies). Data reduction include techniques (e.g., dimension reduction, data compression, discretization and concept hierarchy generation) to reduce the volume or dimensions (number of attributes) in a dataset without compromising the integrity of the original data and yet producing quality knowledge. In Data transformation, data is converted or consolidated into forms appropriate for mining. Subtasks in data transformation include: normalization, smoothing, aggregation, feature construction and generalization of the data.

We focus our overview on the procedures for data cleaning, particularly to solve missing and outlying data on data transformation specifically normalization and feature selection for data reduction.

Missing values. Today various imputation methods are available (Olinsky et al., ). Most of them use indirect approaches to replace the missing values by an imputed form. In general, the imputation methods are focused on MAR data. We can identify two Chapter main types: univariate (single imputation) and multivariate (multiple imputation) methods (Olinsky et al., ; Donders et al., ; Farhangfar et al., ). One important difference between univariate and multivariate methods is that univariate methods are based on the estimation of parameters in order to maximize the similarity between variables when replacing the value, whereas multivariate imputation methods are based on similarities among the objects and/or variables.

The univariate imputation methods most commonly used include: the mean, hot-deck and regression (Olinsky et al., ; Schafer and Graham, ). Multivariate imputation methods are based either on the k-NN, the Expectation-Maximization (EM) algorithms or they are multiple imputations.

k-NN methods are based on the use of a distance measure in order to find the nearest neighbour, subsequently, the average weighted value found is used to replace the missing value. In general, the average weight value used for numerical data is the mean (Troyanskaya et al., ; Jerez et al., ; García-Laencina et al., ; Hron et al., ).

The main advantages of k-NN are: ( ) it predicts both quantitative and qualitative features and ( ) the training dataset is used as a 'lazy' model so explicit predictive models are not created. The major disadvantage is that the algorithm searches through all the dataset to look for the most similar instances. This is a big limitation for large databases. When applying the k-NN approach different distance metrics are used to define similarities between target and reference records. The most frequently used are based on absolute differences, Euclidean or Mahalanobis distance functions (Eskelson et al., ). Absolute differences are computed as:

The Expectation-Maximization approach iterates through two main steps (Expectation, E, and Maximization, M). An incomplete data matrix have observed data Y obs , missing data Y mis and a vector of parameters, θ. Complete data Y com is then defined as Y com = (Y obs , Y mis ). The expected complete data log-likelihood function is defined as

Q(θ|θ ) = E{ln[f (Y com |θ)]Y obs , θ }.
Where the complete data log-likelihood function and the observed data log-likelihood function are defined as:

L(θ) = f (Y com |θ) and L(θ) = f (Y obs|θ ) respectively.
The EM algorithm alternates between the following steps, where θ is initialized at some value:

. Expectation step (E step): Computing Q(θ|θ (t) ) as a function of θ; . Maximization step (M step): Find θ (r+1) that maximizes Q(θ|θ (t) )

In the M step, the maximum log-likelihood estimation is performed as if there were no missing data. Then, in the E step, the conditional expectation of the missing values is found, such value is predicted given the observed data and current estimated parameters. The values obtained from the expectation are used to replace the missing values. The approach iterates until there is convergence in the parameters estimates (Ho et al., ; Olinsky et al., ; Palarea-Albaladejo and Martín-Fernández, ). The EM is a simple and easy to implement approach. The algorithm is numerically stable and it provides fitted values for the complete data during the E step which do not demand further computation. The main disadvantage is its slow linear convergence in some cases in addition the EM approach tends to underestimate the variability of the estimate (Couvreur,

).

Multiple imputation (MI) involves three steps: the generation of completed datasets, the computation of overall estimates and standard errors, and the combination of results. Each missing value is replaced by a list of simulated values, which generates m possible versions of the complete datasets then, each m possibility is analysed by standard proce-dures which are specific for handling complete data. Finally, the results are combined to obtain overall estimates and standard errors that will provide information about missingdata uncertainty and finite-sample variation. According to (Rubin, ) the three steps of MI are:

Step -Imputation: An appropriate model that includes random variation is used to impute missing values. 'Completed' datasets are created by setting plausible values for missing observations. The set of values is used M times to create M 'completed' datasets.

Step -Analysis: The M datasets are analysed using standard complete-data methods.

Step -Combination: In the final step, results are combined in order to take into consideration the uncertainty of the imputation. The following estimations are involved in the procedure:

• produce a single-point estimate (i. The purpose of the imputation is to perform statistical analysis taking into account the uncertainty due to the presence of missing values present in the incomplete dataset and preserving the main characteristics and structure of the data (Farhangfar et al., ; Abayomi et al., ; Cottrell et al., ). As the EM approach, MI is advantageous because it incorporates uncertainty associated to the imputation, it is simpler to adapt however, when the percentage of missing information is large, several imputations are required to get precise estimates, and the results are very sensitive to a misspecification of the model (Heitjan, ).

In the selection of an imputation method to replace missing values, different aspects can be considered such as: the characteristics and structure of data (e.g., percentage of missing values, distribution, pattern of missing values), robustness of method, speed and suitability (univariate or multivariate). Aiming at identifying the most performing imputation method, different comparative studies have been done (Kadengye et al., ; Hron et al., ; Zhang et al., ; Borgoni and Berrington, ; Olinsky et al., ; Engels and Diehr, ; Mercer et al., ), results of such studies indicate that multiple imputation methods perform better than single imputation methods.

Outliers: As previously mention, outliers are patterns in data that are not necessarily incorrect but likely suspicious. Once an outlier has been detected, the way to proceed for its treatment should be taken cautiously. Outlying data can be whether eliminated from data, retained with an appropriate label or replaced by an imputed form (e.g., mean, median and mode), using single imputation methods. The way in which an outlier is treated will mainly depend on the application domain and expert knowledge (Hodge and Austin, ).

Feature selection is performed to find the minimum set of attributes that are necessary to extract the same knowledge as the one obtained using all attributes. It allows to speed the learning stages and facilitates the understanding of data patterns. Feature selection techniques are classified into three categories: Filter, Wrapper and Hybrid (Liu and Motoda, ; Sutha and Tamilselvi, ).

• Filter methods look at the intrinsic properties of the data to evaluate the relevance of features. In general, the score of feature relevance is computed so low-scoring features are identified and removed. Filter methods are advantageous because they can be used in high-dimensional datasets, are computationally simple, fast and independent of the classification algorithm. However, most of the available methods are univariate and ignore feature dependencies. • Contrary to filter techniques where relevant features are found independently of the model selection step, wrapper methods embed the model selection within the feature subset search. In wrapper techniques, different possible feature subsets are searched, evaluated, and compared. A specific classification model is trained and tested to evaluate a specific subset of features. To search for possible feature subsets, a search algorithm is 'wrapped' around the classification model. To guide the search of an optimal subset and thus avoid the generation of exponential feature subsets, heuristic search methods are used. The main advantages of wrapper techniques is that they take into account feature dependencies and that they integer feature subsets search with model selection. However, they are computationally expensive and have high probabilities of overfitting. • In Hybrid methods, the optimal subset of features is search while the classifier is constructed. As in wrapper methods, hybrid methods are specific to a given learning algorithm; however, hybrid methods are less computationally expensive.

Interesting results of a comparative study on feature selection methods has been published by Bolón-Canedo et al. (

). In accordance to their results, Bolón-Canedo et al. (

) suggest to use filtering methods over wrapper and hybrid methods. However, when selecting a method, Murtaugh ( ) suggest to base the selection on the appropriateness for the task at hand. Some other aspects can also be considered such as: characteristics of data, computational costs and robustness of the method (Murtaugh, ; Saeys et al., ; Sutha and Tamilselvi, ).

In addition to the above mentioned techniques, there are some approaches that automatically perform feature selection or variable selection e.g., within a regression model. We can distinguish three important classes of methods: ( ) Subset selection, ( ) Shrinkage, and ( ) Dimension reduction.

• Subset selection In this approach a subset of p predictors that we believe to be related to the response are identified. A model using least squares is then fitted on the reduced subset. Subset selection methods include best subset and stepwise model selection.

In the best subset selection a least squares regression is fitted for each possible combination of the p predictors. Then the resulting models are compared in order to identify the one that fit the best. Its algorithm is described as follows:

Algorithm . . Best subset selection

. Let M 0 denote the null model, which contains no predictors. This model simply predicts the sample mean for each observation. . For k = 1, 2, ...p:

(a) Fit all p k models that contain exactly k predictors (b) Pick the best among these p k models, and call it M k . Here best is defined as having the smallest Residual sum of squares (RSS), or equivalently largest R 2 . . Select a single bet model from among M 0 , ..., Mp using cross-validated prediction error, Cp (AIC), BIC, or adjusted R 2 .

The main disadvantage with best subset selection is that it can not be applied with very large p. When p is large, there are higher chances to overfitting and high variance of the coefficient estimates. An alternative to best subset selection, stepwise methods can be used. They explore a more restricted set of models and include the forward stepwise selection and backward stepwise selection.

Forward stepwise selection starts with a model with no predictors and add predictors one-at-a-time until all predictors are included in the model. Contrary to forward stepwise selection Backward stepwise selection begins the least squares model containing all p predictors and removes the least useful predictor one-at-a-time. Below we provide the algorithm for Forward stepwise selection, more details on the the backward stepwise selection can be consulted at (Harrell, ).

Algorithm . . Forward stepwise selection

. Let M 0 denote the null model, which contains no predictors.

. For k = 0, ...p -1: (a) Consider all p -k models that augment the predictors in M k with one additional predictor.

(b) Choose the best among these p -k models, and call it M k+1 . Here the best is defined as having smallest RSS or highest R 2 . . Select a single bet model from among M 0 , ..., Mp using cross-validated prediction error, Cp (AIC), BIC, or adjusted R 2 .

Forward stepwise can be used when n < p, while backward selection requires that the number of observations n to be larger than the number of variables p. Subset selection, forward selection and backward selection provide a set of models each one containing a subset of p predictors. To select the best model with respect to test error the C p , Akaike information criterion (AIC), Bayesian information criterion (BIC) and Adjusted R 2 can be used. From a standard linear model

Y = β 0 + β 1 X 1 + ... + β p X p + ( . )
The C p estimate of test MSE is computed as follows:

C p = 1 n (RSS + 2dσ 2 ) ( . )
where σ2 is an estimate of the variance of the error associated with each response measurement in . . In the selection of the best model, the model with the lowest C p value is chosen. The AIC criterion is defined as:

AIC = 1 nσ 2 (RSS + 2dσ 2 ) ( . )
C p and AIC are proportional to each other. The BIC is given by

BIC = 1 n (RSS + log(n)dσ 2 ) ( . )
Similarly to C p the model that has the lowest BIC values is selected.

For a least square model with d variables the adjusted R 2 is defined as:

Adjusted R 2 = 1 - RSS/(n -d -1) T SS/(n -1) ( . )
where T SS = (y i -ȳ)

(y i -β 0 - p j=1 β j x ij ) 2 ( . )
In ridge regression the coefficients are estimated by minimizing

n i=1 (y i -β 0 - p j=1 β j x ij ) 2 + λ p j=1 β 2 j = RSS + λ p j=1 β 2 j ( . )
The objective in ridge regression is to make the RSS small in order to get coefficient estimates that fit the data well. The term λ j β 2 j named shrinkage penalty will be small when β 1 , ..., β p are close to zero, therefore it will shrink the estimates of β j towards zero. The advantage of ridge regression over least squares is based on the bias-variance trade-off. In fact as λ increases ridge regression fit decreases, this leads to decreased variance but increased bias. In addition, ridge regression is advantageous over best subset selection computationally speaking, because ridge regression only fits a single model for any fixed value of λ while best subset selection requires searching through 2 p models. One important disadvantage of ridge regression is that it includes all p predictors in the final model and so λ β 2 j will shrink all the coefficients towards zero, but any of them will be set exactly to zero (unless λ = ∞). This is problematic particularly in model interpretation where the number of variables p is large. An alternative to ridge regression that overcomes this disadvantages is lasso. The lasso coefficients estimates are the values that minimize

n i=1 (y i -β 0 - p j=1 β j x ij ) 2 + λ p j=1 |β j | = RSS + λ p j=1 |β j | ( . )
The difference with respect to lasso is that the β 2 j term in the ridge regression has been replaced by |β j |. Similarly to ridge regression, lasso shrinks the coefficient estimates towards zero. In the case of lasso some of the coefficient estimates are forced to be exactly equal to zero when λ is sufficiently large. Lasso performs variable selection which generated models are easier to interpret compared to those produced by ridge regression. It is expected that lasso perform better when a relatively small number of predictors have substantial coefficients, and the remaining predictors equal zero. Ridge regression performs better when the response variable is a function of many predictors with coefficients of equal size. The main problem is that in real-datasets the number of predictors that are related to the response is a priori unknown.

Subset selection and shrinkage methods control variance either by using a subset of variables or by shrinking the coefficients toward zero. An alternative to these methods are the dimension reduction methods which transform the predictors to then fit a least square model with the transformed variables.

• Dimension Reduction By considering Z 1 , Z 2 , ..., Z M represent M < p linear combinations of original p predictors, that is

Z m = p j=1 φ jm X j ( . )
for some constants φ 1m , φ 2m , ...,φ pm , m = 1, ..., M . Using least squares, the linear model is then fitted as

y i = θ 0 + M m=1 θ m z im + i , i = 1, ..., n ( . )
The dimension reduction approach reduces the problem of estimating the p + 1 coefficients β 0 , β 1 , ..., β p to estimating the M + 1 coefficients θ 0 , θ 1 , ..., θ M where M < p. Dimension reduction methods include two steps. In the first step the transformed predictors Z 1 , Z 2 , ...Z M are obtained and in the second step, the model is fitted using the M predictors previously obtained. Principal components and partial least squares are two dimension reduction approaches.

Subset selection, shrinkage and dimension reduction are approaches to perform feature selection or variable selection from a regression model. Within the purpose of this manuscript we assess approaches that are not automatically performed in statistical methods, except for lasso, they were not used in this study.

Normalization Transformation and normalization is performed when attributes values refer to different measurement units or have different scales. It is carried out to give all attributes equal weight and also to smooth the outliers. To this aim, attributes are scaled to range within a specific scale (e.g. -1.0 to 1.0, or 0 to 1.0). Different methods for data normalization are available, the most commonly used are: min -max normalization, z -score normalization, and normalization by decimal scaling (García et al., ).

. . Impact of data preprocessing procedures on statistical analysis results

Data preprocessing has been acknowledged as a primary task to counteract the negative effects that data glitches may produce on analytical results (Van den Broeck et al., ; Dasu,

; Gibert et al., ). Indeed, various publications indicate that more performing results can be obtained when preprocessing procedures are performed previous to statistical analysis (Crone et al., b; Crone et al., a; Nawi et al., ; Kotsiantis et al., ; Praveena et al., ).

Different published works have demonstrated that results from statistical analysis can be improved when procedures such as imputation of missing values (Farhangfar et ) most of the studies have assessed a particular preprocessing method for a particular statistical analysis. In the comparative study of Farhangfar et al. ( ), the effect of different missing data imputation methods on six classifiers was assessed. Their results indicate that improvements on classification may vary according to the imputation method used to replace missing values. Their results highlight the importance of conducting comparative studies to define the specific methods necessary to preprocess data for a particular statistical analysis.

Another aspect that is important to mention when preprocessing data is the absence of tools to: ( ) measure the accuracy of preprocessing procedures and ( ) frame a methodological workflow to preprocess data according to specific statistical analyses. Aiming at providing tools to measure the impact of data preprocessing procedures Dasu Chapter and Loh (

) have developed the statistical distortion metric. The statistical distortion was proposed to measure the effectiveness of data cleaning strategies, it includes three dimensions: glitch improvement, statistical distortion and cost-related criteria. Another interesting approach proposed by Berti-Equille et al. (

), is the masking index which is an indicator for quantifying hidden glitches. However, these metrics do not assess the impact on statistical methods. Concerning a methodological data preprocessing framework, to our knowledge, there are still no publication on this topic. This remains an interesting topic to be explored which actually motivated my work. Finally, few research has been focused on preprocessing of environmental data. Some approaches that has been published to assess and control the quality of data have been published by Rieger et ) have provided guide lines to non-expert user for the selection of pre-processing techniques. Though a robust methodological approach to guide on the selection of preprocessing techniques remains a topic to be explored.

As previously mentioned, presence of glitches is an ubiquitous problem, their detection and processing is an important task to obtain reliable results on data analysis. However there are still many open questions:

• Among the different detection methods, which one is the most suitable one according to the statistical analysis to be applied? • Which data preprocessing procedure should be applied for a specific statistical analysis and dataset? • How different are the analysis results when different preprocessing procedures are applied? • Do preprocessing procedures adversely impact analytical results and conclusions?

In the data preprocessing study that we present in Chapter we intend to answer these questions in general, and also in the particular application use cases of water quality assesment and environmental data analysis.

. Summary

In this chapter, we have presented an overview of the state-of-the art related to three disciplines of my work. The state-of-the art that we present refers to two main topics: data acquisition and data preprocessing. For data acquisition we were interested in the different tools in environmental chemistry and hydrobiology to acquire data specifically for water quality assessment. While in data preprocessing, we were interested in data anomalies that occur in environmental datasets, their detection, and processing.

Related to environmental chemistry, Section . . gives a general overview of water quality assessment, later in section . . we present important aspects on pollution by emerging pollutants and relevant works related to the assessment of pollution in water. We put an emphasis on the pollution of Mexican rivers by emerging pollutants because they are introduced to the natural effluent without following an specific treatment and they are not included in national monitoring programs. The restricted access to advanced analytical instrumentation is one of the main causes of the limited analysis of emerging pollutants. We present in Section . . , our bibliographic study related to analytical methods for the quantification of emerging pollutants.

Concerning hydrobiology, we present a state-of-the art on biomonitoring practices (Section . . ), usefulness of macroinvertebrates in bio-assessment practices (Section . . ) and uncertainties on biomonitoring data (Section . . ). In fact, as we described in our bibliographic study, the use of macroinvertebrates-based biomonitoring metrics are very pertinent for the assessment of Mexican rivers. However, their use is limited and the challenges are vast: description of sampling and analytical protocols, development of local or regional metrics, development of specific taxonomical keys. It is necessary to solve this issues in order to acquire quality biomonitoring data.

Finally, to complete our bibliographic survey in Section . we present different data aspects related to data anomalies, their detection (Section . . ), procedures to process data anomalies (Section . . ) and impact of data preprocessing on statistical analysis (Section . . ).

CHAPTER

Acquisition of Environmental Data

. Introduction

In environmental studies it is essential to have quality analytical results to provide accurate responses or solutions to an specific problem. The quality of analytical results relies mainly on the quality of data therefore, good laboratory and field practices are necessary to ensure good data quality.

The assessment of water quality with respect to the content of emerging pollutants such as pesticides, pharmaceuticals, and personal care products is at its early stage in Mexico.

One important problem when analysing these type of pollutants is the elevated cost and the difficulty to analyse them specially in laboratories with limited access to specialized analytical instrumentation. Therefore, development or adaptation of tools as well as the definition of protocols to analyse these type of pollutants are necessary.

Data from biomonitoring water quality assessment is a regular practice in developed countries however, in Mexico the use of biomonitoring metrics for the water quality assessment of rivers is scarce despite their advantages (i.e., easy to implement, cheap). An essential problem is the absence of biomonitoring protocols to sample and analyse biological samples. To acquire quality biomonitoring data is indispensable to define such protocols.

Within the interest to acquire quality data, we propose to control the entire pipeline. This means from the collection of samples on the field and until the analysis of data on the laboratory. We focused on the acquisition of water quality data with respect to emerging pollutants and biomonitoring data. Our aim is to provide a methodological approach for good laboratory practices to acquire quality data.

In this chapter, we describe our methodological approach for the acquisition of water quality data in Mexico with respect to: pollution by pesticides, pharmaceuticals, and personal care products, a biomonitoring using macroinvertebrates. To our purpose we have collected and analysed water and biological (macroinvertebrates) samples from four Mexican rivers (Tula, Culiacan, Humaya and Tamazula). In addition, we have acquired physico-chemical and chemical data. This chapter is organized as follows: we present the sites of our study in Section . . In Section . , we describe the sampling and analytical methods used for the acquisition of physico-chemical and chemical data. In Section . . , we present our analytical method for the analysis of organochlorine pesticides. Finally, in Section . , we present the sampling and analytical protocol for the bio-assessment of Mexican river waters using macroinvertebrates.

.

Description of the study sites

We have studied four rivers in Mexico the Tula, Culiacan, Humaya, and Tamazula.

The sub-watershed of Tula River is located at the sud-west of Hidalgo State, Mexico. This sub-watershed has km length, semi-arid climate, mean annual temperature of °C, between mm and mm of rainfall annually (between May and October in storms) and the mean evaporation rate is . mm (INEGI, ).

The Tula river is a water flow system that runs from the State of Mexico to the southcentral part of Hidalgo state in Mexico. The Tula river receives the untreated wastewater from Mexico City and from different municipalities in the State of Hidalgo, including south of Zimapán, South-East of Tasquillo, South-West of Ixmiquilpan, Progreso de Obregón, Mixquihuala de Juárez, Tezontepec de Aldama, Tula de Allende, and Tepeji del Río de Ocampo. As a result of wastewater irrigation, the flow of the Tula river increased from . to . m s -in the last decades (DFID, ).

Water from the Tula River is used mainly for agricultural purposes. Municipalities of Hidalgo state that use a vast soil extension for agriculture are Mixquihaula de Juárez ( . %), Tezonepec de Aldama ( . %), Progreso de Obregón ( . %), Ixmiquilpan ( %), Tula de Allende ( . %), Tasquillo ( . %), and Zimapán ( . %) (INEGI,

).

Five sampling sites (named H to H ) placed along the river were selected to collect samples (c.f. Figure . ). Sites H and H are located closed to agricultural fields and were selected to assess the quantities of organochlorine pesticides in the water of the Tula River. Sites H , H , and H were located near urbanized areas, they were selected to assess the levels of pollutants, mainly PPCPs. Details of sampling site location is given in Table . . We have selected a total of eleven sampling sites to assess the water quality along the rivers Tamazula, Humaya, and Culiacan (c.f. Figure . ). The location of each sampling site is detailed in Table . . We selected four sampling sites along the longitudinal axis of the Humaya river (C to C ) to assess the levels of pollutants mainly from domestic wastes. Three sampling sites (C -C ) were selected on the Tamazula River to represent the least polluted state. Sampling sites C and C were selected to assess the levels of domestic wastes and anthropogenic activities in the Culiacan River and sites C and C were chosen to assess pollution by organochlorine pesticides. .

Physico-chemical and chemical data acquisition . . Sampling of water samples

Due to a limited budget and time we did the sampling campaign only for one year and two seasons (dry and post-rainy). Five sampling stations were sampled on April (dry season) and October (post-rainy season) at the Tula River. While sampling stations were sampled on January (dry season) at the Tamazula, Humaya, and Culiacan rivers. The sampling sites described in the previous section were chosen according to accessibility, ease of recognition in the field and ecological relevance.

Water samples were collected to determine contents of: major elements pH, conductivity, carbonate (CO ), bicarbonate (HCO ), sulfate (SO ), chloride (Cl), fluoride(F), sodium (Na), potassium (K), calcium (Ca), magnesium (Mg), boron (B), silicon dioxide (SiO ), nitrate (NO ) arsenic (As), heavy metals cadmium (Cd), lead (Pb), iron (Fe), copper (Cu), manganese (Mn), zinc (Zn), eighteen organochlorine pesticides, namely: I-BHC, II-BHC, III-BHC, IV-BHC, heptachlor, aldrin, heptachlor epoxide, I-endosulfan, II-endosulfan, dieldrin, DDE, endrin, DDD, endrin aldehyde, endosulfan sulfate, DDT, endrin ketone and methoxychlor, and eight PPCPs (ibuprofen, -benzyl--chlorophenol, naproxen, triclosan, ketoprofen, diclofenac, bisphenol A and estrone).

For analysis of major elements, one liter of water was collected and kept at °C until analysis. For analysis of heavy metals and As mL of water were collected and added HNO . Polypropilene bottles used were previously rinsed first with % HNO and then three times with deionized water. Samples were taken by duplicates.

One liter of water was collected by duplicate in amber glass bottles for analysis of pesticides, while mL were collected for analysis of PPCPs. Samples were kept at °C and analysed right after collection. Glass bottles were washed and rinsed times with Chapter deionized water then, they were placed in a container with HCl % during hours. After that time, bottles were rinsed with deionized water and dried. Finally, dried glass bottles were rinsed first with acetone and then with hexane. The above mentioned conditions of cleaning were carry out to reduce pollution effects produced by the presence of external compounds that may not be present in the sampling sites and to get the best recovery of all pesticides under study.

. . Analysis of major elements, heavy metals, and arsenic in water samples

Physico-Chemical Parameters Temperature, pH, conductivity and redox potential (Eh) parameters were determined at each sampling station using an ORION handheld Multiparameter for the determination of temperature, pH, and Eh. A portable conductometer conductronic PC-was used to determine conductivity.

Determination of major elements, heavy metals and arsenic concentrations

Determination of major species was done following the protocol of Armienta et al. ( ). Alkalinity (CO , HCO ) was determined by titration, boron by carmine method, Ca and Mg by titration with EDTA, Cl and F by potentiometry, SiO through the molybdosilicate method, sulfate by turbidimetric method, Na, and K by Atomic Absorption Spectrometry (AAS), and NO by HPLC.

Analysis of arsenic was performed using atomic absorption hydride generation with a Perkin Elmer AAnalyst Spectrometer and FIAS . Analysis of lead was done using an Atomic Absorption Spectrometer Perkin Elmer AAnalyst with graphite furnace. And analysis of Cd, Fe, Cu, Mn, and Zn was performed by Atomic Absorption Spectrometry with a Perkin Elmer AAnalyst Spectrometer.

The physico-chemical and chemical data that we acquired is composed of individuals (sampling sites) and variables (physicochemical and chemical parameters) of numerical values. Only the sites from the Tula river where sampled for two seasons (dry and rainy). An example of data obtained from these analysis is presented on Tables . and. .

Chapter

. .

Analysis of organochlorine pesticides and PPCPs in water samples

Quantification of pollutants in water at low concentrations (in the range of µg L -1 and ng L -1 ), has been a challenging problem in water quality assessment studies. A vast number of techniques have been developed to overcome this problem however, most of them include the use of advanced analytical tools (i.e., ICP-MS, GC-MS, LC-MS or LC-MS-MS). In laboratories where such tools are not available it is necessary to adapt or develop new tools that are cheaper and easy to use without compromising the quality of data. In Mexico monitoring of pesticides is scarce partially due to the limited access to specialized analytical instrumentation. To provide an accessible tool for the analysis of pesticides in water we have adapted an analytical method based on Solid Phase Extraction (SPE) followed by Gas-Chromatography with an Electron Capture Detector (GC-ECD) for the quantification of organochlorine pesticides.

We adapted the method described by Guardia Rubio et al. (

). Analysis was performed using a solid phase extraction (SPE) and gas chromatography with electron capture detection (GC-ECD). Contrary to the method proposed by Guardia Rubio et al. (

), we used ethyl acetate-hexane : %, v/v to dissolve the extract obtained from the solid phase extraction. We have performed an analysis using different combinations of the solvents hexane, cyclohexane and ethyl acetate-hexane, and the ethyl acetate-hexane provided the best results. The solid phase extraction step that we proposed in our method was compared to a liquid-liquid extraction (LLE) method which was previously used in the laboratory. With our methodological approach we obtained better recoveries than with the LLE (recovery ranging from % to %). Details of the analytical performance of the proposed methodology are given in Table . . Analysis of PPCPs was performed using a new method based on SPME and GC-MS. The method used for PPCPs analysis was tested and developed in collaboration with a group of researchers of the Faculty of Chemistry at UNAM, Mexico under the direction of Dr.Araceli Peña A. The new SPME-GC-MS that we proposed is a simple, rapid, efficient, and sensitive alternative for the simultaneous analysis of PPCPS (ibuprofen, -benzyl--chlorophenol, naproxen, triclosan, ketoprofen, diclofenac, bisphenol A and estrone) in water at trace levels (ng L -1 ). In this manuscript, we only describe the procedure followed for the analysis of organochlorine pesticides. Details about the SPME-GC-MS method for PPCPs analysis on water is presented by Diaz-Flores et al.(under revision, Determination of pharmaceuticals and personal care products (PPCPs) in river water and sediment by solid phase extraction followed by gas chromatography-mass spectrometry (SPME-GC-MS), submitted to Analytical Chemistry). In Table . we provide details of the analytical performance of the SPME-GC-MS method for the analysis of PPCPs. The new methodological approaches that we proposed allowed to: ( ) simultaneously detect eighteen organochlorine pesticides (SPE-GC-ECD) and eight PPCPs (SPME-GC-MS), ( ) perform analysis in a shorter period of time, ( ) reduce sample manipulation which reduces errors on data and, ( ) reduce volume of solvents to be used.

Solid Phase Extraction

Solid Phase Extraction was performed for the extraction of pesticides in water samples, the procedure followed is as follows. Samples were first filtered under vacuum through a . µm Millipore filter to eliminate particulated material until the sample remained transparent. C cartridge were placed in a -port Visiprep SPE vacuum manifold and conditioned by passing 5 mL of dichloromethane, 5 mL of methanol and 5 mL of Milli-Q water at a flow rate of 1 mL min -1 . Then, 500 mL of water sample was passed through the cartridge at a flow rate of 1 mL min -1 . The solid phase in the cartridge was not allowed to become dry at any moment. After loading the sample, the SPE cartridge was dried for 20 min under vacuum. Pesticides were eluted using 2 mL of dichloromethane by gravity and finally under vacuum. The extract was brought to dryness in a nitrogen stream and redissolved in 1 mL of ethyl acetate-hexane (25 : 75%, v/v). Analyses of a spiked blank sample and a blank sample were performed together with every set of samples. Figure .  schematizes the SPE procedure for extraction of OCPs in water samples. directly on the GC injector at °C for h.

Solid Phase Micro Extraction

Water samples were firstly filtered under vacuum through a . µm Millipore filter. Then, samples were fortified with analytes at (1000 ng l -1 ), mL of water were transferred to a glass vial and adjusted to pH with . % concentrated formic acid( %), . g sodium chloride were added and a stir bar was introduced. The glass vial was closed and placed on a water bath at °C and maintained min for temperature equilibrium. The analytes were extracted by SPME in immersion mode for min at °C and rpm. Following SPME, the analytes were derivatized on head-space mode by exposing the fiber to the vapors of µL of N,N-tertbutyldimethylsilil-N-methyltrifluoroacetamide (MTBSTFA) in order to obtain the volatile derivatives. After derivatization, the fiber was retracted and inserted directly on the GC injector. Analytes desorption was carried out at °C for min.

Analytical instrumentation and operating conditions

An Agilent Technology N gas chromatograph equipped with an ECD detector was used for pesticide quantification. The chromatograph was equipped with an autosampler and split/splitless injector. The column used was a capillary column (Phenyl Methyl Siloxane m× µm× . µm i.d.), with nitrogen as carrier gas at a constant flow of 1 mL min -1 . A 1 µL aliquot of the extract was injected in a splitless mode into the gas chromatograph. After each injection, syringe was first rinsed with acetone and then with ethyl acetatehexane ( : %, v/v). Temperature programme was the following: initial temperature 80 • C, held for 1 min, 30 • C min -1 ramp to 175 • C then 6 • C min -1 to 215 • C, finally by 15 • C min -1 to 290 • C and held for 5 min.

GC-MS: A Hewlett Packard HP

A gas chromatograph equipped with a mass spectrometer (Agilent Technologies, U.S.A.) was used for PPCPs analysis. The chromatographic column used was a ZB-( m x . mm i.d. x . µm film thickness), with helium as carrier gas at a constant flow of mLmin -and injector temperature of °C. Column temperature program was °C ( min), then increased at °C min-to °C ( min), then °C min-to °C ( min); total running time min. The sample was injected in splitless mode ( min). Mass spectrometric detection was carried out by single ion monitoring (SIM), selecting two fragment ions for identification and quantification of each analyte: ibuprofen y m/z; naproxen y m/z; diclofenac y m/z; ketoprofen y m/z; -benzyl--chlorophenol y m/z; triclosan y m/z; bisphenol A y m/z; estrone y m/z. Quantification was carried out by external standardization by integrating area of the chromatographic peaks. Peak identity was confirmed with retention time and mass spectrum of each analyte, where characteristic fragment ions were selected.

With the use of analytical methods described above we were able to analyse for the first time the content of organochlorine pesticides and PPCPs in the Mexican rivers (Tula, Tamazula, Humaya and Culiacan). Data from the analysis of PPCPs and organochlorine pesticides is composed of individuals (sampling sites) and variables ( PPCPs and Oragnochlorine Pesticides) of numerical values. An example of these data is presented on Tables . and. .

Table . -Example of data from the analysis of organochlorine pesticides on water samples from the Tula river. Only ten out of the six-teen compounds are shown. .

Hydrobiological data acquisition

As explained in Section . currently, in Mexico, the use of macroinvertebrates for ecological assessment of aquatic systems is scarce due partially to the absence of biomonitoring protocols and undefined metrics. Aiming at providing guidelines and practical tools for the biomonitoring of aquatic systems, we have done a study whose objectives were: ) identify the macroinvertebrate-based monitoring approaches with potential application to the ecological assessment of Mexican streams, and ) describe sampling and analytical procedures necessary to implement such approaches.

Results of this study have been published (Serrano Balderas et al.,

) and are part of the work of this manuscript. Briefly, thirty-five biomonitoring metrics were identified as potential approaches to be applied for the ecological assessment of Mexican streams. A description of the sampling and analytical procedures to compute the selected metrics is also given in the above mentioned publication. We have applied for the first time the guidelines described by Serrano Balderas et al. (

) in order to acquire biological data for Tula, Tamazula, Humaya, and Culiacan rivers. Below we described the sampling and analytical procedures followed for this purpose.

. . Sampling of macroinvertebrates

Macroinvertebrate sampling was performed first by a visual inspection of a m stretch of the river channel to identify the different habitats and substrates. Samples were collected from multi-habitats as suggested by Jáimez-Cuéllar et al. (

), using a surber net (surface cm , mesh size µm). For each sampling point, replicates were obtained. Samples were stored in plastic bags and preserved in % ethanol.

. . Analysis of macroinvertebrates

In laboratory, macroinvertebrate samples were separated from vegetable and mineral substrates and sorted using a µm mesh sieve. Macroinvertebrates were examined under a stereoscope and identified at family level using different taxonomic keys (Heckman, ; Heckman, ; Heckman, ; Merrit et al., a; Tachet et al., ; Novelo-Gutiérrez, b; Novelo-Gutiérrez, a). Thirty-five biomonitoring metrics were calculated. They include: metrics of richness, enumeration metrics, diversity and similarity indices, biotic indices , functional feeding metrics and multimetric approach. For the case of similarity indices, they were computed by comparing two stations. In fact, during the sampling campaign one station was selected to represent the least polluted state of the river, this station was located far from anthropogenic and urban activities and upstream. This station considered as "non-polluted" or "the least polluted" was used to compute similarity indices by comparing it against the other stations.

Details concerning the taxonomic resolution, definition, and expected response to increasing perturbation for each metric are given in Table . .

Family or Genus

Description of the pollution level of streams, according to the sensitivity of key groups to pollution. Aquatic ecosystems are described using "quality classes". Ghetti, Beck Biotic Index (BI)

Genus

Classification of streams according to their organic pollution. Organisms are classed according to their tolerance to organic pollution. The taxonomic level indicated for the diversity and similarity indices is the level preconceived by the author of each metric. However, they can be computed using some other taxonomic level (e.g., species-level).

Metric of richness is the number of specimens found in each sample site, the richness metrics calculated includes: number of total taxa, number of EPT taxa, number of Ephemeroptera taxa, number of Plecoptera taxa, and number of Trichoptera taxa.

Enumerations were computed by estimating the relative abundance of certain pollutionsensible groups to the total abundance of macroinvertebrates they consisted of: number of families in common, %EPT, %Ephemeroptera, %Plecoptera, %Trichoptera, %Coleoptera, %Diptera, %Chironomidae, EPT:Chironomidae, % of most dominant genera, and % of dominant taxa.

Four diversity indices (Shannon's Index, Simpson's Index, Margalef Index, and Sequential Comparison Index) and two similarity indices (Jaccard's Coefficient and Sørensen Coefficient) were computed. The formulaes of three diversity indices are given below:

Shanon_Index : H = i=S i=1 p i log p i ( .
)

Simpson_Index : D = i=S i=1 n i (n i -1) N (N -1) ( . ) M argalef _Index : D = S - 1 log N ( . )
Where p i is the proportion of individuals of the ith taxon (p i = n i /N ), n i is the total number of individuals of the ith taxon, N is the total number of individuals for all ith taxa and S is the total number of taxa.

Sequential Comparison Index (SCI) was computed using Equation . (Cairns et al., ). The procedure to estimate the SCI is as follows: Let us consider a sample containing Chapter macroinvertebrates. These macroinvertebrates are placed in a container for examination (c.f., Figure . ). Counting from left to right in row A, we can observe that the first four organisms are similar and, therefore, are part of the same run. Organisms six and seven are similar but both are different compared to organism five and, therefore, the last two organisms are part of a new run. In row A there are 3 runs for 7 organisms. Compared to row A, row B presents 5 runs for 10 organisms. The Jaccard's (Jaccard, ) and (Sørensen, ) coefficients are similarity indices used to evaluate the mean differences between sampling stations. They use presenceabsence data and were computed using the following equations:

Jaccard_Coef f icient : C j = j a + b -j ( . ) Sørensen_Coef f icient : C s = 2j a + b ( . )
where j is the number of taxa in common between the stations A and B, a is the number of all taxa in station A and b is the number of all taxa in station B.

Biotic indices and the Macroinvertebrate-based Index of Biotic Integrity (IBI) for westcentral Mexican streams were computed according to authors description. Biotic indices include: Trent Biotic Index (Metcalfe, ), Extended Biotic Index (Ghetti, ), Beck Biotic Index (Beck, ), Family Biotic Index (Hilsenhoff, ), Biological Monitoring Working Party (National Water Council,

) and its associated Average Score per Taxon (Armitage et al., ). Details on the computation of each metric are given in Appendix A.

Six Functional Feeding Groups were computed including: filtering collectors, scrapers, shredders, predators, and gathering collectors. Each organism was counted and grouped in feeding groups according to Merrit et al. ( a).

Hydrobiological data from the rivers Tula, Tamazula, Humaya, and Culiacan is composed of individuals (sampling sites) and numeric parameters ( biomonitoring metrics). Only the Tula river contains information related to two sampling periods (dry and rainy). Below we provide an example of data from biomonitoring of the rivers Tula, Tamazula, Humaya and Culiacan.

. Introduction

Data Mining (DM) methods include non supervised methods (i.e., clustering, PCA, classifiers) and supervised methods (i.e., regression, classification), through the use of such methods, we can discover relevant patterns in the data and gain more knowledge. To use DM methods it is necessary to provide input data in the amount, structure and format that suits DM methods perfectly. However, real-world data such as data from environmental surveys are highly affected by anomalies such as missing values, inconsistencies, inaccuracies, outlying data, etc. To provide high quality data and thus quality analytical results it is necessary to preprocess data. Data preprocessing is a critical step in data mining processes. Results from data preprocessed inappropriately may lead to misleading conclusions and, in the worst case, dramatic consequences and wrong decision making that can affect the survival of certain environmental ecosystems in our application domain.

We aim at providing to data and environmental scientists a guide to inspect, preprocess and analyse environmental data. We focused our study on the best practices to preprocess data with respect to an specific statistical analysis in order to get quality analytical results.

To our aim, we have developed a comprehensive study to assess the impact of preprocessing procedures on accuracy of different statistical methods namely regression, classification, and clustering. We focused our study on procedures to select features, normalize data, and deal with missing values and outliers because these are the anomalies most frequently find in water quality data.

The main objectives of the approach are:

. Assess the robustness of methods to process missing and outlying data;

. Evaluate the effect of feature selection, normalization, missing data imputation, and outlying processing on the accuracy of subsequent classification, clustering, and regression analysis; . Identify the best data preprocessing procedures for a particular statistical analysis.

We have structured our study in three parts: ( ) generation of synthetic datasets, ( ) data preprocessing, and ( ) statistical analysis (c.f. Figure . ).

In the first part, we construct synthetic data to assess preprocessing procedures then, in step two we generated preprocessed data by executing different preprocessing procedures on the synthetic data of step one, and we assessed the robustness of preprocessing procedures. In the third step we used the preprocessed data to assess the impact of preprocessing procedures on the results of statistical analysis. In the following sections we describe each part of our study.

The chapter is organized as follows: in Section . we briefly describe the preprocessing procedures used in our study, in Section . we describe our synthetic datasets. Then, in Section . we detail the methodological procedure followed to study the robustness of the preprocessing procedures and, in Section . we describe our analysis of the impact of data preprocessing procedures on the results of the main statistical methods. Finally, our results and concluding remarks are given in Section . respectively.

Generation of

Synthetic datasets

Statistical analysis

Data Preprocessing

Original

Chapter

.

Data preprocessing procedures . .

Feature selection

Feature selection is performed to reduce the dataset by removing redundant or irrelevant features (or variables). It increases the speed of the learning methods and facilitates the understanding of patterns. We have studied three feature selection methods, two filter and one wrapper method. They were chosen because they belong to two different families of techniques that allow us to have an heterogeneous suite of methods to carry out our experiments. The methods are the following:

• Correlation-based Feature Selection (CFS) is a filter algorithm that ranks features according to a correlation-based evaluation (Hall, ; Liu et al., ). Relevant features are selected according to their correlation among classes and with each other. Features are considered relevant when they present a strong correlation with the class and they are not correlated with each other. Acceptance of a feature will depend on its efficiency to predict classes in the instance space that are not predicted by other features.

• Linear Correlation is an attribute evaluation method. This univariate filter method provides a ranking of all features for a certain threshold. Here, we set up the threshold to select features with strong correlation values (Eid et al., ). • Wrapper Subset Evaluator evaluates usefulness of a feature set by using a learning algorithm. Cross-validation is used to estimate accuracy of the subsets (Kohavi and John, ). Finally, the feature subset which provides the best learning performance is chosen. In this work, we have used a simple regression model.

. . Normalization

Normalization is performed in order to give all variables equal weight. By normalizing data, all variables are expressed in the same measurement units, therefore measurement units can not affect data analysis. We have performed three normalization methods on numerical data: min-max, z-score and decimal-scale normalization. They were chosen due to their differences on the computation and their popularity.

• In Min-max normalization. Let us define min A and max A be the minimum and maximum values of an attribute A. In min-max normalization a value v i of A is transformed to v i in the range [new_min A , new_max A ] by computing

v i = v i -min A max A -min A (new_max A -new_min A ) + new_min A ( . )
• Z-score normalization (or zero-mean normalization) is performed based on the mean and standard deviation of an attribute A. A value v i of A is normalized to v i i as follows:

v i = v i - Ā σ A ( . )
where Ā is the mean and σ A the standard deviation of A. • In decimal scale normalization, the values of an attribute v i is normalized to v i by moving the decimal point which depends on the maximum absolute value of v i . Decimal scale normalization is computed using the following formula:

v i = v i 10 j ( . )
where j is the smallest integer such that M ax(|v i |) < 1.

. . Imputation methods

Imputation methods, are used to replace missing values on the dataset. This alternative can be used when an statistical method can not be performed under the presence of missing values. To our purpose, we have tested four imputation methods: two distance-based imputations: Hot-deck and K-NN, and two model-based imputation methods: Multiple Imputation by Chained Equations (MICE) and Iterative Robust Model-based Imputation (IRMI). They were chosen because they represent different techniques of imputation that allow us to have an heterogeneous comparison of methods.

• Hot-deck imputation method uses an actual value from a dissimilar case in the current dataset to replace the missing value. To determine the similar case, the user selects classification variables. The cases that agree with the case under consideration on these classification variables are placed into a pool from which one case is chosen randomly (Olinsky et al., ). The values used to impute in Hot-deck preserve the distributional characteristics of the data. This approach is effective essentially when the data are MAR. However, there is little empirical work which can determine the accuracy of this imputation method, it also can provide large bias on the estimates of the error variances.

• K-nearest neighbour imputation (K-NN) is based on the application of a distance measure in order to find the closest neighbours. Once the k-nearest neighbours have been found, a replacement value is estimated and used as a substitute for the missing value. The replacement value is calculated depending on the type of data, the mode is used for qualitative data and the mean for continuous data (Troyanskaya et al., ). One important issue when applying the K-NN approach is the selection of the appropriate distance metric. The distance between two observations was defined as

d i,j = p k=1 w k δ i,j,k p k=1 w k ( . )
where w k is the weighted mean of the contributions of each variable and δ i,j,k is the contribution of the kth variable. For continuous variables the absolute distance was computed as follows:

δ i,j,k = |x i,k -x j,k |/r k ( . )
where x i,k is the value of the kth variable of the ith observation, and r k the range of the kth variable (Templ et al., a). The advantage of K-NN is that it provides a robust procedure for missing data estimation, it can make prediction for both discrete and continuous attributes, it can treat multiple missing values easily because the creation of predictive models are not needed however, the K-NN algorithm is particularly problematic for large datasets because the algorithm search on the entire dataset to find the most similar instances (Batista et al., ). • The Multiple Imputation by Chained Equations (MICE) is based on the assumption that missing data is MAR. The procedure consists of the computation of a set of regression models where each variable with missing values is modelled conditionally Chapter upon the other variables in the data (Buuren and Groothuis-Oudshoorn, ). The algorithm can be summarized into the following steps:

Algorithm . . Multiple Imputation by Chained Equations (MICE)

-Step : Initialization of a temporal imputation using a simple imputation method (e.g., mean); -Step : For one of the variables y i set back the temporal imputation into missing values; -Step : Regression based on the observed values from the variable y i in Step on the other variables in the imputation model. This means that in a regression model y i will be considered as the dependent variable and the other variables as independent; -Step : Replace missing values of y i by predictions drawn from the regression model. When y i is subsequently used as independent variable in the regression models for other variables, both these imputed values and the observed will be used; -Step : Steps -are repeated for each variable with missing values.

-Step : The loop through each variables where steps -are repeated constitutes one iteration.

At the end of each iteration, all missing values have been replaced. Steps -are repeated for a number of iterations defined by user. The final imputations at the end of all iterations are kept as the final result for one version of imputed dataset.

• The Iterative Robust Model-based Imputation (IRMI) (Templ et al., b) is based on the EM algorithm, where, in the "Expectation" step, the regression method is applied iteratively. For each iterative step, one variable is used as the response variable and the other variables are used as regressors. In this way, "all" the information of the variables is used for the imputation of the responding variables. The algorithm is summarized into seven steps as follows:

The main differences between MICE and IRMI is that IRMI includes a robust regression which reduces the influence of outlying observations and protects against poorly initialized missing values. Another difference is that in each step of MICE the predictive values are used to update former missing values while IRMI uses predictive values to update expected values, this allow to keep track of convergence of sequential methods. IRMI also provides (co-)variances that are included in the final iteration.

The four selected imputation methods are available in R. Scripts involving the studied methods and their respective packages, namely: VIM for hot-deck, K-NN and IRMI, and mice for MICE have been implemented for testing and assessing the methods. I/{l} and X m l I/{l} will consist of ones, and an intercept term x o l l = X o l I/l β + in the regression problem should be considered. Where β is the regression coefficient and is the error term. In each regression the distribution of the response x o l l is considered to fit if the response is: * continuous, the link is µ and a robust regression method is applied; * categorical, a generalized linear regression is applied; * binary, a logistic linear regression is applied when the link is log( µi 1-µi ) for i = ,...,n; * semi-continuous, a two-stage approach is applied, in the first stage a logistic regression is applied and in the second stage a robust regression based on the continuous (non-constant) part of the response is used to impute; * count, a robust generalized linear regression of family Poisson is applied and the link is log(µ i ), for i = ,...,n.

-

Chapter

. . Outlier detection methods

Four detection methods based on different approaches were implemented including; an statistic-based approach (Inter quartile Range, IQR), two multivariate outlier detection approaches (Adjusted-Quantile), and an algorithm that uses Principal Components decomposition (PCOUT) (Filzmoser et al., ), and a density-based approach (Local Outlier Factor, LOF; (Breunig et al., )). They were chosen because they belong to different families of methods that allow us to have an heterogeneous comparison of methods.

• Inter Quartile Range (IQR) covers the central % of data, it is determined as the range between the and % of the quantiles Q 1 (lower quantile value) and

Q 3 (upper quartile value), IQR = Q 3 -Q 1 .
When an observation falls below Q 1 -1.5(IQR) or above Q 3 + 1.5(IQR) it is then considered as an outlier.

• Adjusted-Quantile (AQ). Outliers are detected through the adjusted quantile as follows (Filzmoser et al., ): First, a chi-squared plot is used to visualize the deviation of the data distribution from multivariate normality in the tails. Then the tails of the empirical distribution function G n (u) of the squared robust distances RD 2 i and the distribution function G(u) of χ 2 ρ are compared to detect outliers. Let us denote the empirical distribution function of the squared robust distances RD 2 i as G n (u) and G(u) as the distribution function of χ 2 ρ . G n will converge to G for multivariate normally distributed samples. The tails of G n and G are compared to detect outliers. The tails are defined by δ = χ 2 ρ ; 1 -α for small α (e.g., α = . ), and

ρ n (δ) = sup u≥δ (G(u) -G n (u)) + ( . )
where + indicates the positive differences. p n (δ) is not used directly as a measure of outliers. Instead a critical value P crit is introduced to distinguish between outliers and extremes where extremes of the distribution are considered as observations with a large RD. The measure of outliers in the sample is defined as

α n (δ) = 0 if p n (δ) ≤ p crit (δ, n, p), p n (δ) if p n (δ) > p crit (δ, n, p) ( . ) c n (δ) = G -1 n (1 -α n (δ))

-Step

Transform the data by subtracting the median and dividing by the median absolute deviation (MAD), in each dimension. Compute the covariance matrix of the transformed data. The MAD is defined for a sample {x 1 , ..., xn} ⊂ R as

M AD(x 1 , ..., xn) = 1.4826 • med j |x j -med j x i | ( . )

-Step

Calculate the principal component decomposition of the semi-robust covariance matrix from Step . Retain the p * values/eigenvectors that contribute to at least % of the total variance. Sphered the transformed data through the median and MAD.

-Step

Compute the robust kurtosis weights for each components and weighted norms for the sphered data as follows

w j = 1 n n i=1 (z ij -med i z ij ) 4 M AD(z 1j,...,znj ) 4 -3 ( . )
By scaling data through MAD, the Euclidean norms in principal component space are equivalent to Mahalanobis distances. Transform these distances using the following equation

d i = RD i • χ 2 p * ,0.5 median{RD i } ( . )
where χ 2 p * ,0.5 is the χ 2 p * 0.5 quantile. -Step Determine weights w 1i for each robust distances as follows:

w 1i =      0, d i ≥ c 1 -( di-M c-M ) 2 2 , M < d i < c 1, d i ≤ M ( . )
where i = 1, ..., n and M is the . quantile of the distances {d 1 , ..., dn} and c = median{d 1 , ..., dn} + 2.5 • M AD{d 1 , ..., dn}.

-Step Compute the (unweighted) Euclidean norms of the data using the semi-robust principal component decomposition used in Step . Transform using Eq. ( . ) to get a set of distances. -Step For each robust distance, determine weights w 2i according to Eq. ( . ) with c 2 = χ 2 p * 0.99 quantile and M 2 = χ 2 p * 0.25 quantile. -Step Determine final weights for all observations according to Eq. ( . ) using weights of Steps and

w i = (w 1i + s)(w 2i + s) (1 + s) 2 ( . )

Chapter

The local reachability density (lrd) of object p is the inverse of the average reach-dist of the kNNs of p which is defined as:

lrd k (p) = l/ o kN N (p) reach -dist k (p, o) Card(kN N (p)) ( . )
The local outlier factor (LOF) of object p is the average ratio of local reachability distance of neighbours of p and lrd of o

LOF k (p) = o kN N (p) lrd k (o) lrd k (p) Card(kN N (p)) ( . )
. The first step of our comprehensive study for the assessment of preprocessing procedures is the construction of synthetic datasets. Our experiments can not rely only on our dataset produced using the methods described in the previous chapter (c.f. Section ) because the results would not be meaningful. Thus, we needed to construct synthetic datasets. Our synthetic datasets were constructed in order to be the most similar to environmental data in terms of distribution, correlation, etc., specifically, water quality data. As illustrated in Figure . , we created non-deformed datasets that we named "original" then, we deformed the original datasets by injecting anomalies (i.e., missing values and outlying data) or by deforming the original characteristics of the datasets. For instance, creating a new dataset with a non-normal distribution or by generating high correlated variables. We created different datasets for a better control on our experimentations. The resulting deformed datasets were subsequently used on the second part of our study (c.f. Section . ). Below we detail the characteristics of our datasets.

Synthetic data description

Generation of Synthetic datasets

Synthetic data for Feature Selection

Four synthetic datasets were used to assess the impact of feature selection procedures on subsequent regression, classification, and clustering analysis. Each dataset follows a normal distribution and is composed of different number of observations n, numerical variables p, and one categorical variable of five classes with an uniform distribution, variables include irrelevant features (c.f. Table . ). These irrelevant features are highly correlated variables.

Chapter

Synthetic datasets for Normalization

Experiments to assess the impact of normalization were performed using four synthetic datasets. Each dataset is composed of different number of observations n, numerical variables p from a Weibull distribution and one categorical variable of five classes from an uniform distribution (c.f. Table . ).

Synthetic datasets for missing data

Assessment of imputation methods was carried out using four synthetic datasets. Each dataset is composed of different number of observations n, numerical variables p from a normal distribution and one categorical variable of five classes from an uniform distribution. Missing data were introduced randomly in numerical variables, using the MCAR mechanism, into each of the datasets. The missing values were introduced into all variables in all datasets in the following six amounts: %, %, %, %, % and %. For each missing ratio, we generated ten different missing datasets from the original complete dataset to ensure that experimental results were statistically acceptable. Table .  summarizes the characteristics of our experimental datasets.

Synthetic datasets for outlying data

The experiments were performed using three synthetic datasets. Each dataset is composed of different number of observations n, numerical variables p from a normal distribution and one categorical variable of five classes from an uniform distribution. Outlying data were introduced randomly into each of the datasets in the following five amounts:

. %, . %, %, % and %. A detailed description of the synthetic datasets used for assessment of outlier detection methods is given in Table . .

Hereafter we provide the characteristics of the first ten variables of the datasets used for our experimentations.

Table . -Synthetic datasets used to assess feature selection methods. Only the µ and σ 2 values of the first ten variables are shown. Highly correlated variables are denoted as Y k .

Dataset Variables FS n = 21 p = 8 X X X X X Y Y Y µ . . . . . . . . σ 2 . . . . . . . . FS n = 600 p = 30 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . FS n = 4000 p = 53 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . FS n = 20000 p = 98 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . .
Table . -Synthetic datasets used to assess normalization methods. Only the µ and σ 2 values of the first ten variables are shown.

Dataset Variables AL n = 21 p = 8 X X X X X X X X µ . . . . . . . . σ 2 . . . . . . . . AL n = 600 p = 30 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . AL n = 4000 p = 53 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . AL n = 20000 p = 98 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . .
Table . -Synthetic datasets used to assess imputation methods. Only the µ and σ 2 values of the first ten variables are shown.

Dataset Variables N n = 21 p = 8 X X X X X Y Y Y µ . . . . . . . . σ 2 . . . . . . . . N n = 600 p = 30 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . N n = 4000 p = 53 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . N n = 20000 p = 98 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . .
Table . -Synthetic datasets used to assess outlying preprocessing. Only the µ and σ 2 values of the first ten variables are shown. In the need to implement useful tools to evaluate the effectiveness of restoration and/or protection actions of French rivers, the FRESQUEAU was conceived. The project brought together experts on computational intelligence and data mining methods (from the four laboratory involved, LHYGES , TETIS , LSIIT and LIRMM .) and experts on hydroecology: LHYGES, TETIS as well as two engineering consultants, AQUASCOP and AQUABIO. The principal goals of this project were: to develop a tool that will allow the assessment of rivers global function by considering the different parts of the ecosystem and to broaden the knowledge of data mining from complex, heterogeneous and large databases with temporal and spatial variations. Within the objectives of the project, physical, chemical and biological data produced by the French Water Agencies DREAL and ONEMA from the Rhin-Meuse (RM) and Rhône-Méditerranée-Corse (RMC) basins were collected.

Dataset Variables N n = 21 p = 8 X X X X X X X X µ . . . . . . . . σ 2 . . . . . . . . N n = 600 p = 30 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . N n = 4000 p = 53 X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . .

. Semi-synthetic data description

Generation of Semi-synthetic datasets

Original

The FRESQUEAU project (« Fouille de données pour l'évaluation et le suivi de la qualité hydrobiologique des cours d'eau ») which was a French multidisciplinary project financially supported by the Research National Agency (ANR) -Laboratory of Hydrology and Geochemistry of Strasbourg A joint research centre of Territories, Environment Remote Sensing satellite and Spatial Information (TETIS from its French acronym).

LSIIT from its French abbreviation is the Laboratory of Science of Images, Informatics and Remote Sensing Satellite of Strasbourg.

LIRMM from its French abbreviation is the Laboratory of Informatics, Robotics and Microelectronics of Montpellier

The Regional Direction for the Environment, Development and Housing (DREAL from its French abbreviation).

The National Office of Water and Aquatic Environment (ONEMA from its French abbreviation).

Chapter

The FRESQUEAU dataset was composed of monitoring stations of the RM area dating from to . It concerned physical and physico-chemical information. The data contained the annual statistical summary values of each parameter (arithmetic mean, median, percentile , percentile ) obtained from streams water quality monitoring done every month or every two months.

The physico-chemical data were formed by the monitoring of parameters analysed for each station. Out of these parameters, were considered as classic physicochemical parameters called now with the Water Framework Directive (DCE) "physicochemical parameters supporting biology" (in the text those parameters will be mentioned as macro pollutants) and as micro pollutants. The list of macro pollutants is made up of the parameters describing the content of oxygen (e.g. DBO , Dissolved Oxygen O , saturated oxygen O%), organic matter (e.g., Organic Carbon, CO -) phosphorus (total phosphorus, phosphates PO -), nitrogen (e.g., Kjendhal Nitrogen, NH -, NO -, NO -), minerals (e.g., Na, K, Ca, Cl) or eutrophication (e.g. chlorophyl a, pheopigments).

The FRESQUEAU dataset contained the couple station-year (which was considered as an independent station) and its corresponding summary statistic value for each parameter.

We have used the data which is composed of stations-year with variables for macro pollutants related to the mean summary statistic value. This dataset was treated in order to remove noisy data. Therefore outliers, duplicate data and missing values were removed. The cleaned dataset was composed of observations (stations-year) and variables that we named FQ . From this dataset, two semi-synthetic datasets were generated. The first semi-synthetic dataset, named FQ is a reduced presentation of FQ , it is composed of observations and variables. Observations and variables were selected randomly from the FQ dataset. The second semi-synthetic dataset, FQ was constructed by introducing new observations following the distribution of the FQ dataset. FQ dataset is composed of observations and variables.

The cleaned semi-synthetic dataset (FQ , FQ and FQ ) were subsequently deformed by injecting anomalies (i.e., missing values, outlying data) in the same fashion as described in Section . for the synthetic datasets. Details of the µ and σ 2 of the semi-synthetic datasets are given in Appendix B.

. In the second part of our comprehensive study, we aimed at assessing the robustness of methods to process missing and outlying data and preprocess data by feature selection and normalization. To our purpose, we have used our deformed data constructed in the previous step (c.f. Section . ), next we have executed four data preprocessing procedures including: feature selection, normalization, imputation of missing values and outlying processing (c.f. Figure . ). We were interested in these procedures because they treat the data anomalies most frequently found in environmental data. The resulting preprocessed datasets were subsequently used on the third part of our study (c.f. Section . ).

Robustness study of data preprocessing procedures

Data

Additional to the construction of preprocessed datasets, we have compared the performance of the selected imputation and outlier detection methods. We did a comparative study for the selected methods because, we did not find in the literature a study that compares them. Concerning feature selection and normalization, previous studies have compared the performance of different feature selection (Bolón-Canedo et al.,

) and normalization methods (Mustaffa and Yusof, ; Grimvall et al., ) and thus we did not studied them in here. Below we detail our experimental procedure.

. . Selection of features

Feature selection is a process to select an optimal subset of features. An optimal subset could be a subset that gives the best estimate on a predictive model. It is implemented for different purposes including: ( ) to improve performance of a model in terms of simplicity of the model, speed, etc.; ( ) to better visualize data; ( ) to remove noise and reduce dimensionality.

We have performed three feature selection methods a Correlation-based feature selection (CFS), a linear-based correlation (LC) and a Wrapper subset evaluator (WR) to select an optimal data subset. Selection of features on each deformed dataset was performed as follows: First, we have aleatory chosen one variable that we used as independent variable then, by using the selected feature selection methods we have identified the best data subset with respect to our independent variable. The resulting data subsets were subsequently used to assess the impact of feature selection procedures on statistical results (c.f. Section . ).

. . Normalization of data

Water quality data is represented by a set of variables with different dimensions and units. To give all variables equal weight and obtain accurate predictive model we need to normalize data. We have performed three normalization methods: min-max (MM), Zscore (ZS) and decimal scale normalization (DS). Normalization was applied to numerical data. The resulting normalized datasets were subsequently used for our study described in Section . . . Experimental procedure to assess robustness of missing data imputation is presented in Figure . . First, the 'original' synthetic datasets were deformed by injecting six different amounts of missing values. Next, the missing values in the 'deformed' datasets were imputed using five imputation methods. The imputation accuracy was evaluated by computing the normalized root mean squared error (NRMSE) (Oba et al., ) as follows:

. . Handling missing data

N RM SE = mean[(y imputed -y complete ) 2 ] variance[y complete ] ( . )
where the mean and variance are calculated over missing entries in the whole matrix. y complete is known because the missing entries are artificial. The larger the NRMSE is, the less is the prediction accuracy.

The experiments, which include datasets, times amounts of missing values, imputation methods gives us a total of 4 × 10 × 6 × 4 = 960 preprocessed datasets. All the experiments were performed using the R environment for statistical computing (version . . ; R Core Team ( )). Experimental procedure to assess robustness of outlying processing is presented in Figure . . First, 'original' synthetic datasets were constructed, then outlying datasets were generated by introducing the five different amounts of outlying data. Next, outliers were detected using four methods to detect outliers. Finally, outliers were replaced using the imputation methods Hot-deck, k-NN, mice and IRMI. The resulting imputed datasets were subsequently used to study the impact of outlier preprocessing on statistical results (c.f. Subsection . ). Outlier detection methods were assessed by computing detection rate and precision of detection as follows (Chen et al., ),

. . Handling outlying data

detection rate = the number of outliers correctly detected total number of true outliers × 100 ( . )

precision = t p t p + f p × 100 ( . )
where t p is the number of points correctly labelled as outlier and f p is the number of points wrongly labelled as outlier.

The experiments, which include three datasets, five amounts of outlying data, four outlier detection methods and four methods of imputation gives us a total of 3 × 5 × 4 × 4 = 240 preprocessed datasets. All the experiments were performed using different functions available in the R environment for statistical computing (version . . ; R Core Team ( )).

. . Results and discussion on robustness of data preprocessing procedures . . . Imputation methods robustness

Results of our assessment on imputation methods are graphically illustrated in Figure . . We observed that, from the two distance-based imputation methods, the K-NN is more accurate than the Hot-deck method. While for the two model-based imputation methods, IRMI showed up to be more accurate than MICE. To our surprise, the K-NN method has the lowest NRMSE values. We consider that this results is due to the characteristics of the datasets. More experiments that include heterogeneous set of datasets (e.g., non normal distribution, correlated variables) will be necessary to have a more general view of the behaviour of the tested methods. Except for K-NN, our results are congruent to previous studies (Cottrell et al., ; Templ et al., b; Hron et al., ) in that multivariate imputation methods (here: MICE and IRMI) are more robust when Chapter comparing with more simple imputation methods (here: Hot-deck). It must be noticed that the results that we obtained depend on the characteristics of the data, in our case data with multivariate normal distribution. It must be necessary to perform other experiments with a more heterogeneous set of data to get a more complete vision of the behaviour of the tested imputation methods. For our subsequent comparative study (c.f. . ), we decided to continue using the imputation methods described in this section.

. . . Outlier detection assessment

Results of our outlier detection assessment are as follows: As shown in Figure . , we observed that precision of outlier detection methods was in the following ordering: PCOUT > Adjusted quantile > LOF > IQR for N 4000 and N 600 datasets. For N 21 dataset the LOF detection method show the best results. We observed that the IQR method has negative values, this indicated that IQR is precise but the rate of outliers detected is low.

This assumption was confirmed with the results of detection rate where the IQR method show, in general, the lowest values.

Concerning outlier detection rates, detection performance is observed in the following order: LOF > PCOUT > Adjusted quantile > IQR. In general, LOF method provides the best results for datasets N and N for the five outlying data rates. For dataset N , we observed that at low outlying data rates ( . % and . %) methods Adjusted quantile and PCOUT perform well but at higher outlying data rates ( % and %) method LOF has the best detection results. For our synthetic datasets that have multivariate normal distribution and individual outliers LOF shows to be the best method. Additional experiments that may include heterogeneous set of datasets will be necessary. We want to emphasize that the results that we observed were characteristic for our datasets (multivariate normal distribution and individual outliers injected at random). And further analysis may be necessary to observe the behaviour of the tested methods on datasets with different distribution (e.g., Weibul, Cauchy) and with other type of outliers (e.g., cluttered, patterns). . Our experimental procedure to assess the impact of data preprocessing procedures on statistical analysis results is shown in Figure . . First, we have used synthetic deformed datasets (c.f. Section . ) that were preprocessed using: feature selection, normalization, imputation and outlier processing (c.f. Section . ). Then, the resulting preprocessed datasets were used to execute regression, classification and clustering methods. The selected statistical methods were chosen because they are frequently used on environmental studies to predict phenomena, find patterns and discover relationships among variables and observations (Wiseman, ). Finally, we have compared the statistical results of each preprocessed data by computing different statistical errors. We have performed our experiments using a combination that, seems the most simple to us. This is, only one type of data anomaly was studied at a time the resulting preprocessed dataset was then used to implement the selected statistical methods. For example, we have four deformed datasets for feature selection, they were preprocessed using three feature selection methods. Finally the resulting twelve preprocessed datasets were used to implement three regression methods, three classifiers and two clustering methods.

Study of the impact of data preprocessing procedures on statistical results

Statistical

In the following sections we detail the procedures followed to implement the statistical methods and the computation of the errors.

. . Impact on regression results

We have performed three regression methods on our preprocessed data that resulted from feature selection (c.f. Subsection . . ), normalization (c.f. Subsection . . ), imputation of missing values (c.f. Subsection . . ), and outliers processing (c.f. Subsection . . ). Regression methods were chosen for their diversity of representation and learning style. They include:

• Penalized regression: Least Absolute Shrinkage and Selection Operator (LASSO);

• Linear regression: Ordinary Least Squares Regression (OLSR);

• Non-Linear regression: Multivariate Adaptive Regression Splines (MARS).

Regression analysis for each dataset was implemented as follows:

. We begin by randomly splitting the observations into a training and test set where % of data was used for training and % for testing, . Regression model was fit on the training set, and the fitted model was used to predict the responses for the observations in the testing set, . Resulting validation was assessed using the RMSE as in Equation ..

RM SE = mean[(y -ŷ)] 2 ( . )
where ŷ is the predicted value and y is the actual value. . Finally, we estimated the preprocessing error rate by comparing the RMSE value of the original non-deformed dataset and the preprocessed dataset as follows:

Error RM SE processing = RM SE P reprocessed -RM SE original RM SE original * 100% ( . )
Low values of Error RM SE processing will be indicative of small data preprocessing impact on regression results.

. . Impact on classification results

Three classification methods based on different learning style were chosen. they are:

• Linear classification: Linear Discriminant Analysis (LDA);

• Non-Linear classification: Naïve Bayes (NB);

• Non-Linear classification with Regression Trees: Classification and Regression Trees (CART)

Similarly to regression analysis (c.f. Section . . ) we have applied the classification method to different alternatives of data preprocessing outputs. Classification analysis was performed in four steps:

. We randomly split the observations into a training and test sets where % of data was used for training and % for testing; . Classification models were fit on the training set, and the fitted model was used to predict the responses for the observations in the testing set; . Resulting validation of the previous step was assessed through computation of accuracy, and Cohen's Kappa coefficient (Cohen, ). Accuracy is the number of correct predictions made divided by the total number of predictions Accuracy = number of correct classif ications total number of classif ications ( . )

The kappa coefficient (Cohen, ) is commonly used as a measure of agreement between two classifications, it ranges from -1 to +1. A value of 1 indicates perfect agreement. Let us consider the following results from two classifiers;

Chapter

The resulting classifications can be assimilated into two qualitative variables of p terms (here p = 3) giving the following contingency table :   Table . -Contingency table obtained from the results of two classifiers.

A B C A B C
In the case of perfect agreement between the two classifiers, the contingency table would have been zero out of the diagonal. By denoting the contingency table as N = (n ij ) i,j=1,...,p and n the total observations, the observed proportional agreement is defined as:

p 0 = 1 n p i=1 n ii ( . )
In the case where the two variables are independent (i.e., if the agreement between the two classifiers was perfectly random), the theoretical proportion of observed agreements is estimated by :

p e = 1 n 2 p i=1 n i .n. i ( . )
kappa is then defined as:

κ = p 0 -p e 1 -p e ( . )
. Lastly, preprocessing error was computed using Eq ( . ) and Eq ( . ) as follows:

Absolute Error Accuracy processing = |Accuracy P reprocessed -Accuracy original | ( . )

Absolute Error Kappa processing = |Kappa P reprocessed -Kappa original | ( . )

. .

Impact on clustering results

Clustering analysis was performed using the K-means (KM) and Hierarchical clustering (HC) methods on our preprocessed data that resulted from feature selection (c.f. Subsection . . ), normalization (c.f. Subsection . . ), imputation of missing values (c.f. Section . . ) and outliers processing (c.f. Section . . ). The two clustering methods were chosen for their algorithmic differences.

To perform K-means clustering, we first specified a number of clusters K using the elbow method (Thorndike, ). The elbow method was applied first to the original dataset and, the K number of clusters found was then used on the processed dataset. Then, we performed the K-means algorithm with the previously specified number of clusters. Hierarchical clustering was computed using the Ward's agglomeration method (Ward Jr, ).

Clustering results of K-means on preprocessed data were compared against the clustering results of the original non-deformed datasets using the Rand Index (RI, Equation .), Adjusted Rand Index (AR, Equation .) and Jaccard Index (JI, Equation . ) (Meilă, ).

From resulting cluster C k of the original non-preprocessed dataset D, the number of data points in D and in C k are defined as n and n k , respectively thus,

n = K k=1 n k ( . )
A second cluster of the preprocessed dataset D is defined as C = {C 1 , C 2 , ..., C k }, and the cluster sizes as n k .

To compare clustering from the pair (C, C ), we use a confusion matrix. Within the confusion matrix (which is a K × K matrix) the kk th element is the number of points in the intersection of clusters

C k of C and C k of C . Thus n kk =| C k ∩ C k |.
We compare clusters by counting the pairs of points on which two clusterings agree/disagree. They were obtained from the contingency table [n kk ]. The four cases that can be found are: N 11 the number of pairs that are in the same cluster under both C and C ; N 00 the number of pairs in different clusters under both C and C ; N 10 the number of pairs in the same cluster under C but not under C ; N 01 the number of pairs in the same cluster under C but not under C;

The four counts satisfy

N 11 + N 00 + N 10 + N 01 = n(n -1)/2 ( . )
We use the following indices to represent the probability that a pair of points belonging to a cluster under C is also in the same cluster C .

Rand Index : RI(C, C ) =

N 11 + N 00 n(n -1)/2 ( . )

Adjusted Rand Index :

AR(C, C ) = R(C, C ) -E[R] 1 -E[R] = k k=1 K k =1 n kk 2 -[ K k=1 n k 2 ][ K k =1 n k 2 ]/ n 2 [ k=1 K n k 2 + K k =1 n k 2 ]/2 -[ K k=1 n k 2 ][ K k =1 n k 2 ]/ n 2 ( . ) Jaccard index : J(C, C ) = N 11 N 11 + N 01 + N 10 ( . )
For Hierarchical clustering we have compared the clustering results of preprocessed dataset against the clustering results of the original non-preprocessed dataset using the Cophenetic correlation coefficient (Sokal and Rohlf, ). Datasets D were modelled using the hierarchical clustering method to produce a dendrogram T which distance measures are defined as: x(i, j) =| D i -D j | , the Euclidean distance between the ith and jth observations; t(i, j) = the distance between the model points T i and T j ; and the Cophenetic correlation coefficient is defined as:

c = i<j (x(i, j) -x)(t(i, j) -t) [ i<j (x(i, j) -x) 2 ][ i<j (t(i, j) -t) 2 ] ( . )
where x is the average of x(i, j) and t the average of t(i, j). The Cophenetic correlation coefficient allow us to estimate the similarities between dendograms of preprocessed and original datasets. Where Cophenetic values close to indicate high similarity between two dendrograms.

. .

Results and discussion about preprocessing procedures on statistical results -Experimental procedure for assessment of the impact of data preprocessing procedures on results from regression analysis.

. . . Regression analysis

Statistical

Our results report the preprocessing error values for three different regression methods (LASSO, OLSR and MARS) against (c.f. Figure . ): It is known that noise features that are not truly associated with the response will lead to a deterioration of the fitted model and will increase the test-set error and the risk of over-fitting. This is important in the analysis of highly dimensional data which is known as "curse of dimensionality". This was the main reason to have performed feature selection before analysis on regression methods. However, it must be noticed that LASSO already includes feature selection. To make our analysis comparable we have performed feature selection before analysis on LASSO but this step should be omitted on a regression analysis by LASSO.

For our synthetic datasets we observed that in general, Linear correlation-based feature selection show the lowest RMSE values for LASSO and OLSR regression methods. For MARS regression, none of the three feature selection methods stand up as the best for F S21, F S4000 and F S20000 datasets. For F S600 dataset the Linear correlationbased method show the best results. Concerning our semi-synthetic datasets the linear correlation-based feature selection show the lowest RMSE values for the three regression methods.

Results for normalization

Predictive variables can be normalized in order to obtain a numerical stability that enables one to compare effects across multiple explanatory variables. In other words, by normalizing predictive variables one assure to give the same weight to all the predictors. One important aspect in linear regression is that linear transformations are not expected to affect regression results because the model coefficients (intercept and slope terms in the linear model) are estimated in order to convert the units of predictive variables into the units of response variable. In this manner, linear transformation of predictive variables does not seem to be meaningful in linear regression. Our goal in this section was to assess the variability induced by different non-linear transformations on the final results of regression analysis. To our purpose we have performed three non-linear normalizations they are: decimal scale and two sigmoidal normalizations. They were computed using the following formulas:

• Decimal scale normalization:

v i = v i 10 j ( . )
• Sigmoidal normalization using logistic sigmoid function:

v i = 1 1 + e - v i -µ i σ i ( . )
• Sigmoidal normalization using the hyperbolic tangent function:

v i = 1 -e - v i -µ i σ i 1 + e - v i -µi σ i ( . )
where v i denotes the values of an attribute, v i is the normalization of v i and j is the smallest integer such that M ax(|v i |) < 1.

We have selected one response variable and the remaining variables were used as predictors. Only predictor variables were transformed (non Gaussian initially). Figures . and . show that there is not a significant difference when non-linear transformations are applied to predictor variables on LASSO, OLSR and MARS regressions.

Our results indicate that, similarly to linear transformations, non-linear transformations do not affect linear regression results. Such behaviour can be explained through the estimation of the model coefficients because they produce estimates in order to transform the units of each predictor variable into units of the response variable appropriately. Our results suggest that transformations on data are not relevant when regression by LASSO, OLSR and MARS are performed.

Results for missing values preprocessing

A side-by-side comparison between different imputation methods, respective of the regression algorithms, is given in Figures . and. . It shows the preprocessing error rates computed for RMSE over the three regressions methods and the four datasets, resulting from imputation against different amounts of missing values. The preprocessing error rates are provided for the four imputation methods.

Preprocessing error results on our synthetic datasets show that the impact of imputation varies for different regression methods. In general, at high amounts of missing values ( % and %) imputation methods Hot-Deck and MICE give the lowest error values. The lowest error rates were observed for LASSO and OLSR regression methods (with % and % on average respectively). In general, the highest RMSE values were observed for the dataset of small size (N ) and the highest error values were observed on dataset N for MARS regression. High preprocessing error values indicate high probabilities to get inaccurate regression results. From this observation, we could conclude that inaccurate results may be obtained when regression by MARS is performed after imputing missing values particularly for small datasets (< observations) and missing data percentage superior to %. Imputation is required for LASSO and OLSR and it was observed that Hot-Deck and MICE imputation methods impact the least LASSO and OLSR regression results. LASSO and OLSR methods are the least impacted by imputation methods when the dataset size is greater than observations and the percentage of missing values is lower than %.

Whit respect to the semi-synthetic data (c.f., Figure . ) we observed that error values increase with respect to the increased missing data rate. We observed that the imputation methods Hot-Deck and K-Nearest Neighbour have the highest error for the three datasets. Results about the imputation methods are not the same between the two groups of dataset (synthetic and semi-synthetic) such behaviour is justified by the fact that datasets have different characteristics (different number of observations and variables) however, we noticed that both groups of datasets have the same tendency. This means that error values increase with respect to the increased missing rates.

Results on outliers preprocessing

A side-by-side comparison between different outlier detection and imputation methods, respective of the regression algorithms is given in Figures . and. . It shows the preprocessing error rates of RMSE over the three regression methods and the synthetic and semi-synthetic datasets, resulting from outlier detection-imputation with respect to different amounts of outlying data. The preprocessing error rates are provided for the sixteen combinations of outlier detection and imputation methods.

In general, for our synthetic datasets we observed that at small amounts of outlying data ( . % and . %) and small length dataset ( N ), the combined methods PCOUT-Hot-Deck and PCOUT-IRMI give the lowest preprocessing error values. While at high amounts of outlying data ( %, %, or %) and large datasets (e.g., N and N ), detection methods PCOUT and LOF combined with imputation methods MICE and IRMI give the lowest preprocessing error values. We noticed that, in general, at high amounts of outlying data ( % and %) preprocessing error rates were higher. For our semi-synthetic datasets (c.f., Figure . ), we observed that error rates increase at higher amounts of outlying data. For LASSO the detection method IQR combined with IRMI and MICE show the lowest error values. While for OLSR and MARS we did not observe remarkable differences among the outlying preprocessing methods.

It has been widely suggested to process outliers carefully because, depending of the observer they may provide different information according to the domain of study and could be considered either as noise or as signal (Huang et al., ). According to our results, and if the observer considers to preprocess outlying data, we could suggest the use of multivariate methods (i.e., PCOUT and LOF combined with MICE and IRMI).

For the specific characteristics of our synthetic datasets, we observed that, in general, multivariate methods give the best results and particularly on large datasets with high amounts of missing values and outlying data. We interestingly observed that, simple methods (e.g., K-NN imputation method or Inter Quartile Range) provide the best results on our small datasets with small amounts of data anomalies. -Preprocessing errors of RMSE of the analysis of regression after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). In these experiments, we report the preprocessing error values for three different classification methods: Classification and Regression Trees (CART), Linear Discriminant analysis (LDA) and Naïve Bayes (NB) with respect to (c.f. Figure . ):

. . . Classification analysis

Statistical analysis Data Preprocessing

Chapter

Results on our synthetic datasets for accuracy show that the filter method Correlationbased feature selection give the most accurate results on the four datasets for the three classifiers. Concerning Kappa, the best results for LDA were observed on the Correlationbased feature selector. While for NB classification, the Linear Correlation-based feature selection show the best results. For CART classifier none of the feature selection methods stand-up as the best one.

For our semi-synthetic datasets the wrapper subset evaluator give the best results for LDA, while the linear correlation-based feature selection give best results for NB. Finally, non of the feature selection methods stand-up as the best for CART classifier.

CART is known to be highly non robust this explains its behaviour in our results. We consider that the correlation-based feature selection methods show the best results on our synthetic dataset due to the characteristics of our datasets (e.g., variables with a correlation > 0.7, multivariate distribution). The results on our synthetic datasets show that, decimal-scale normalization provides the best results for AL21 and AL20000 datasets on the LDA and NB classifiers, while for AL600 and AL4000 datasets, the min-max and z-score normalizations stand up as the best for LDA and NB respectively. Except for AL4000 dataset the z-score normalization gives the best results for CART classifier. For our semi-synthetic datasets the z-score and min-max normalitzations give the best results for LDA and NB respectively. For CART, none of the normalization strategies stand up as the best.

Results for normalization

From Kappa results we observed that, for small datasets decimal-scale and z-score give the best results for synthetic and semi-synthetic datasets respectively. While for long datasets (e.g., AL4000, AL20000 and F Q1000) none of the methods stand up as the best. Our results suggest that the selection of normalization method will provide different classification results. We assume that the difference in our synthetic and semi-synthetic datasets is due to the characteristics of our datasets (e.g., distribution, size) and on the differences of the learning style of the three classifiers.

Results for missing values preprocessing

A side-by-side comparison between different imputation methods, with respect to the regression algorithms, is given in Figures . and. . They show the preprocessing absolute errors computed for accuracy and Kappa over the three classification methods and the synthetic and semi-synthetic datasets, resulting from imputation against different amount of missing values. The preprocessing absolute errors are provided for the four imputation methods.

Results show that, in general, for small datasets (N 21, F Q16), imputation methods Hot-Deck and k-Nearest Neighbour give low preprocessing error rates at small amounts of missing values ( %, %, and %). For large datasets (N , N , FQ and FQ

) imputation methods IRMI and MICE show the lowest preprocessing error values in both, accuracy and Kappa, for the six amounts of missing values. For the three classification methods the highest preprocessing error values of accuracy were observed on N and FQ datasets for the four imputation methods.

Concerning the preprocessing error results for Kappa, we observed that small datasets (N 21, F Q16) and small amounts of missing values ( %, %, and %) the imputation method Hot-Deck have the lowest error values on the three classification methods for the seven datasets. From these results, we could conclude that for datasets with similar characteristics as ours and small amounts of missing values ( %, %, or %), the imputation by Hot-Deck will have the lowest impact on CART, LDA, and NB classification methods. While for large datasets (more than observations and variables), IRMI and MICE imputation methods should be preferred. We inferred that this behaviour depends directly on the characteristics of the datasets that we use for our study. Therefore, results may change if datasets have different characteristics than ours.

Results on outlying data preprocessing

In Figures . and. , we present a side-by-side comparison between different outlier detection and imputation methods, with respect to the classification algorithms. It shows the preprocessing absolute errors computed for accuracy and Kappa over the three classification methods and the synthetic and semi-synthetic datasets, resulting from outlier detection-imputation against different amounts of outlying data. Sixteen combinations of outlier detection and imputation methods are shown.

Results show that, on average, there is no universally best method to detect and impute outliers. The impact of detection-imputation of outliers varies for different classifiers. In general, we observed that classification results were the most impacted on the small dataset (N and FQ ), where preprocessing error values of accuracy were, on average, over 0.4. CART classifier seems to be the most affected on N , F , N and FQ datasets for the five amounts of outlying data.

Concerning the preprocessing error values for Kappa, in general, small datasets (N and FQ ) gives the highest errors for LDA and NB classifiers. While for CART, the highest errors were observed on N and FQ dataset for the five amounts of outlying data. Among the sixteen combinations assessed, we noticed that outlier detection methods PCOUT and LOF combined with IRMI and MICE imputation methods provide the lowest error values on the three classifiers and on the five amounts of outlying data. For synthetic datasets, the detection method PCOUT combined with Hot-Deck imputation also shows low preprocessing error values on the three classifiers particularly for N and N datasets at outlying data rates of . %, %, and %.

Our experiments on synthetic and semi-synthetic datasets show that results on outliers preprocessing depend on the characteristics of the datasets. We can see that multivariate methods (e.g., PCOUT and MICE) give the best results for our datasets which have a multivariate distribution with variables somehow correlated. The results observed for CART classifier, indicate us that this method is the most impacted by the preprocessing procedure which is explained due to its poor robustness. Figure .

-Accuracy and Kappa preprocessing error from the analysis of classification after imputation of missing data on semi-synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). -Preprocessing errors of Accuracy and Kappa of the analysis of classification after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). -Preprocessing errors of Accuracy and Kappa of the analysis of classification after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). -Experimental procedure for assessment of the impact of data preprocessing procedures on results from clustering.

Chapter

. . . Clustering results

Statistical analysis Data Preprocessing

Our experiments report the Rand Index (RI), Adjusted Rand Index (AR), Jaccard Index (JI) for K-means clustering and Cophenetic coefficient for Hierarchical clustering with respect to (c.f. Figure . ): Clustering by K-means was performed for all synthetic datasets and for FQ and FQ semi-synthetic datasets. Hierarchical Clustering is not robust for datasets which number of observations are larger than therefore, only the datasets N , N , N , FQ and FQ were used for this method.

Results for feature selection

A side by side comparison between different feature selection methods, with respect to the clustering algorithms is given in Figures . and. . It shows the Rand index, Adjusted Rand index and Jaccard index over the K-means on the synthetic and semisynthetic datasets, and the Cophenetic coefficient over Hierarchical clustering on datasets FS , FS , FS , FQ and FQ .

We observed that Wrapper Subset evaluator show to have the best results on both clustering methods for datasets FS , FS , and FS . While for datasets FQ and FQ the Linear Correlation-based feature selection show the best results on both clustering methods. It can be observed that clustering results on the studied datasets after feature selection are different. It was assumed that such differences are due mainly to the characteristics of the datasets (e.g., distribution, size).

Results for missing values preprocessing

A side-by-side comparison between the different imputation methods, respective of the clustering algorithms, is given in Figures . and. . It shows the RI, AR, JI, and Cophenetic coefficient values over the two clustering methods and the synthetic and semi-synthetic datasets resulting from imputation against different amounts of missing values.

In general, it was observed that the precision of the clustering methods reduced with an increasing amount of missing data. Our results show that the Hot-Deck and MICE imputation methods give the best results on both clustering methods on small datasets (N and FQ ) and low missing data rates ( % and %). K-Nearest Neighbour imputation method shows, in general, the best results on K-means clustering for the synthetic datasets and the six amounts of missing data. While for F Q16 and F Q1000 datasets the MICE and IRMI imputation methods show the best results on K-means clustering. Except for N dataset at % missing data, MICE and IRMI imputation methods provide the best results on Hierarchical clustering for the three dataset and the six amounts of missing data. For N dataset and % missing data, Hot-Deck imputation shows the best results. We consider that the observed results depend on the characteristics of the data. We observed that the multivariate imputation methods MICE and IRMI stand up as the best since our datasets are multivariate. On the particular case of N and FQ datasets, the Hot-Deck method provide the best results, we consider that this is due in part to the type of missing values (Missing at Random) and the characteristics of our datasets.

Results for outlying data preprocessing

A side-by-side comparison between different outlier detection and imputation methods, respective of the regression algorithms is given in Figures . and. . It shows the Rand index, Adjusted Rand index, Jaccard index, and Cophenetic coefficient over the two clustering methods and the three datasets resulting from outlier detection-imputation against different amounts of outlying data. The three indices and coefficient values are shown for the sixteen combinations of outlier detection and imputation methods.

Results show that, on average, there is no universally best method to detect and impute outliers. The impact of detection-imputation of outliers varies for the two clustering methods and for the synthetic and semi-synthetic datasets. Rand index, Adjusted Rand index, and Jaccard index results show that, Inter Quartile Range (IQR) combined with K-Nearest Neighbour and MICE imputation give the best results on the two clustering methods for N dataset at . % of outlying data. While for dataset FQ , Adjusted Quantile (AQ) combined with IRMI imputation give the best results at . % and . % of outlying data. -Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after imputation of missing data on semi-synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). bined with IRMI, MICE and Hot-Deck shows the best results for K-means clustering. While for N and FQ datasets for the five amounts of outlying data, IQR combined with K-Nearest Neighbour and MICE are the best outlier preprocessing procedures for K-means clustering. Concerning Hierarchical clustering, the IQR detection method combined with K-Nearest Neighbour and Hot-Deck imputation methods show the best Cophenetic coefficient values for N and N datasets for the five amounts of outlying data. For dataset N , IQR combined with Hot-Deck or IRMI imputation methods show to be the best outlying data preprocessing option. The combined methods AQ-MICE and IQR-MICE provide the best results for datasets FQ and FQ respectively.

According to our results on Section . . , the IQR method, provided a very low detection rate, but a high precision. This means that the probability that the detected point using IQR were a real outlier was high. When comparing our results from Section . . and . . . we observe that in general the IQR outlier detection method stands as the best when combined with imputation methods such as MICE and IRMI. We consider that these results depend to the characteristics of the datasets and also on the preprocessing methods. It seems that a precise outlier detection method such as the IQR combined with multivariate imputation methods will be a good option to preprocess outliers. However, it is probable that different results may be obtained on dataset with characteristics different from ours. Thus, our results can not be generalized for all type of datasets but are useful to visualize the impact of the preprocessing and statistical methods that we studied on our synthetic and semi-synthetic datasets. -Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ), and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). Figure .

-Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ), and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

. Summary and concluding remarks

A comprehensive study to assess the impact of preprocessing procedures on accuracy of three statistical methods (regression, classification and clustering) was presented. Our methodological approach allowed us to identify the best data preprocessing strategies for each statistical method for our synthetic and semi-synthetic datasets. We discovered that there is not a universal data preprocessing procedure since it depends on the characteristics of the dataset in terms of size, distribution, Skewness, Kurtosis, etc.

Results obtained from this study were collected and summarized in a matrix. It contains the main characteristics of our datasets and the preprocessing procedures that provides the best results for each statistical method. With the help of such matrix we could construct a set of rules for combining optimally data preprocessing and data analysis. This rules were subsequently used on the construction of a fully integrated analytics environment in R for statistical analysis of environmental data in general, and for water quality data analytics, in particular (c.f. Chapter ). Table . gives an extract of our summary table. For simplicity purposes, only results for N dataset on one regression, one classification and one clustering methods are shown. The set of rules that we obtained can be used as the basis to extract knowledge related to data preprocessing and most importantly, it can be completed and extended to study: other type of datasets, data anomalies and data preprocessing procedures.

It should be noticed that our study was performed using datasets with a multivariate normal distribution and on four procedures to prepare data (feature selection and normalization) and process anomalies (missing values and outliers). Further studies will be necessary to cover a wider spectrum of datasets including temporal or spatial series, different distribution along with other types of data preprocessing and data mining techniques. Moreover, the approach that we presented was performed studying each data anomaly as isolated cases. For instance, datasets including missing values have been preprocessed only using imputation methods. However, multiple anomalies may be present in a real-world dataset thus requiring the application of divers preprocessing strategies. An important issue when multiple anomalies are present in a dataset is the order in which preprocessing strategies need to be applied. Below we provide an example to illustrate the importance of data preprocessing ordering.

Non-normalized datasets with missing values were preprocessed by different order. The synthetic datasets have different length (ALNA ; n = 21 p = 8; ALNA ; n = 600 p = 30) with a Weibul distribution and different amounts of missing at random values (i.e., %, %, %, %, % and %). These datasets were preprocessed by ( ) normalization followed by imputation of missing values and, ( ) imputation of missing values followed by normalization. Min-max normalization was used. Imputation of missing values were done using the IRMI, MICE, Hot-deck and k-NN methods. To compare the results of preprocessing strategies a classification by Naïves Bayes was performed following the same methodology described in Section . . .

Results of classification by Naïves

Bayes on the different ordering of preprocessing strategies is shown in Figure . . It was observed that the differences between the two preprocessing ordering are small but, in general, lower error rates are obtained when normalization is performed after imputation of missing values. Such differences may be due to divers aspects including: distribution of data before and after the the application of the first preprocessing procedure, general assumption of preprocessing and statistical procedures, etc. Therefore an extended study in this respect is necessary in order to have a better understanding of preprocessing ordering.

Though ordering of preprocessing strategies is out of the scope of this thesis work, with this example we want to highlight the importance of conducting studies in this topic. -Results of Naïves Bayesian classification on datasets preprocessed by: (A) normalization followed by imputation of missing values and (B) imputation of missing values followed by normalization. Min-max normalization (MM).Imputation methods: Hot-deck (HD), IRMI (IR), K-NN (KN) and MICE (MI).

Table . -Extract of summary results from the study of data preprocessing procedures on regression, classification and clustering methods. FS (Feature Selection), N (Normalization), I (Imputation method), O (Outlier processing).

Regression

. Introduction

In the previous chapter, we have presented our methodology to collect water quality data as well as our approach to assess data preprocessing quality.

In order to provide to the scientific community the tools to better preprocess and analyse environmental data, we have developed a fully integrated environment in R that integrates our data preprocessing approach.

The dataset collected for assessing water quality in Mexican rivers was used as input to the tool that we describe in this chapter. In this chapter, we present our results in our application domain of Water Quality Assessment. This chapter is organized as follows: in Section . we present related work on scientific workflow systems as our tool fits in the same line. Then, in Section . we present the main functionalities of our prototype. In Section . , we present the results of the analysis of water quality data of the Mexican rivers by using our prototype. Finally, we provide some conclusions of this chapter in Section .

.

Related work on scientific workflow systems

Advances in scientific computation allow scientist to conduct their analysis by using workflow systems. Scientific workflow systems (SWFS) provide an infrastructure to perform and monitor a set of data manipulation steps in a goal-oriented scientific application.

A scientific workflow system is composed of a set of tasks linked together to create a final product in therms of derivative/aggregate data, or hypothesis validation. In computing sciences, workflows are derived from programming models that allow the integration of software routines, datasets and services for the scientific discovery process. SWFS are generally applied to: ( ) describe complex scientific procedures, ( ) automate data derivation processes, ( ) improve through put and performance on high performance computing and ( ) manage and query of provenance (Zhao et al., ).

There are several software environments that provide tools to define, compose, map, and execute workflows. They have been grouped into two classes: script-like systems and graphical-based systems (Neubauer et al., ).

Script-like systems describe workflows using a textual programming languages such as Java, Perl or C ++ . Tasks, parameters, constraints and data dependencies are established in order to build up a workflow. Examples of these type of systems are GridAnt (von Laszewski and Hategan, ), Karajan (Von Laszewski et al., ) or Askalon (Fahringer et al., ) to mention some. Some of their advantages include:

• Configuration of specialized process model; • Control of sequences, constraints, loops and parallel tasks;

• Parallel execution of workflows;

• Monitoring of the execution.

Though, Script-like systems often have complex semantics, require a deep knowledge of the workflow engine functionality and knowledge on programming languages.

Graphical-based systems specify workflows through graphical elements (i.e., nodes and edges) that correspond to the graph component. Workflow tasks are often represented by nodes while data dependencies or communications between tasks are represented by links. Some of the most popular graph-based systems are: Kepler (Altintas et al., ),

Weka WS (Talia et al., ) and Triana (Taylor et al., ). Their advantages include:

• Easy and intuitive use for unskilled user;

• Model of workflows through the use of graphical interface;

• Limited use of programming languages;

One important drawback of graphical-based systems, is that complex workflows can not be represented due to the limited expressiveness on the directed acyclic graph-based languages.

Scientific workflow systems aim at supporting research by providing a framework that encompass all the necessary tasks and algorithms to access, collect, process and analyse scientific data. Numerous workflow systems are available (Talia, ; Liu et al., ), often require user have a deep knowledge of the workflow engine functionality or ability on programming languages. They also require a comprehensive knowledge of the characteristics of data, thus the designed scientific workflow can include the necessary data preprocessing tasks to manage anomalies on data. Notwithstanding, as we have mentioned in our previous chapters, the impact of data preprocessing task on subsequent data analysis is under study and as such, the integration of data preprocessing procedures on scientific workflow systems may be taken cautiously. Additional to this problem, it must be noticed that environmental scientist are not computer experts, they may not be able nor want to cope to the complexity of the use and deployment of these systems for their scientific purposes. To our knowledge, there are not a scientific workflow system specifically for environmental sciences that includes data preprocessing procedures and that provides a user-friendly interface for non computer experts. We aim to cover this need by providing a simple scientific workflow system. Our system integers the necessary tasks to inspect, preprocess and analyse environmental data. Data preprocessing procedures can be assessed and if necessary be modified in order to get valid results according to user criteria. It also take advantages of an user-friendly interface that allows users with little knowledge on R to analyse environmental data.

.

EvDA: Development of R Shiny application for environmental data preprocessing and analysis . .

Main workflow

Data collected by environmental scientists cover highly diverse topics of study going from public health studies to atmospheric phenomena. Moreover, the laborious tasks of collecting environmental data increase the probability of having anomalies in the data. As a result, the analysis of environmental data has itself become an arduous effort. The methodological approaches developed in this work aim at providing tools to support and assist the scientific community in the preprocessing and analysis of data. We have focused on procedures for preprocessing the environmental data specifically features selection, normalization, imputation of missing values, and outlying data as well as regression, classification, and clustering methods since they are frequently used to analyse environmental data.

We present a prototype named EvDA that aims to inspect, preprocess, and analyse environmental data easier. The user can upload his/her own data. Preprocessing procedures can be executed sequentially and data analysis such as regression, classification, clustering or PCA analysis produce meaningful results for the domain experts using an user-friendly interface.

Chapter

Load data Analysis Preprocessing Summary Inspection

Display data summary -select features -normalize data -impute missing values -process outlying data -Get preprocessed data -list preprocessing procedures -perform statistical analysis

Display results

Display all results

Figure . -Scientific workflow of EvDA

Implementation To this aim, we have designed and developed a scientific workflow system prototype in EvDA, based on our methodological approach described in the Section . . It was constructed to execute a set of data manipulation tasks for environmental data analysis in general and for water quality data analytics in particular. Our prototype consists of graphical user interface from which user is able to load, inspect, preprocess and analyse data (c.f. Figure . ). EvDA was implemented using Shiny web application framework (R package version . . ) for R statistics software (R Core Team,

). It uses several R packages and functions internally. In EvDA, the input is an original dataset. After inspection, a set of preprocessing tasks are executed to perform statistical analysis on the preprocessed data. Finally, the outputs include visualization and analytical results of the preprocessed data.

Data inspection.

EvDA provides results of summary statistics such as mean, median, minimum and maximum values, as well as Skewness and Kurtosis coefficients to characterize the data distribution. Information about normality, missing values, and outlying data is also provided. Results are displayed in a table and can be graphically visualized. In the data inspection step, the user can have a general overview of the characteristics of data. This information is helpful to make decisions about data preprocessing procedures that may be necessary to execute. Data preprocessing. Data preprocessing procedures such as: feature selection, normalization, imputation of missing data, and outlying data preprocessing can be performed. The most optimal data preprocessing procedures are automatically determined in EvDA: a message listing optimal data preprocessing procedures are displayed. The user could choose between the suggested options and the procedures available in the application. Three feature selection methods are available: wrapper subset evaluator, Correlationbased and linear correlation-based feature selectors, they are calculated using the caret and the mlr R packages. Feature selection outputs are: ( ) a list containing pertinent features and ( ) a new dataset containing only the selected features. Six normalization methods can be performed: decimal scale, min-max, z-score, -range normalization, sigmoidal and softmax normalization; they were adapted from the dprep R package. Five methods to impute missing values are proposed in EvDA including: mean, Hot-Deck, K-Nearest Neighbour (K-NN), Iterative Robust Model-based Imputation (IRMI), and Multiple Imputation by Chained Equations (MICE). Imputation methods are computed using their respective packages, namely ForImp for mean, VIM for Hot-Deck, K-NN and IRMI and, mice for MICE.

Outlying data are processed as follows: first outliers are detected using one of the outlier detection methods, then they could be removed from the dataset or imputed using imputation methods. The outlier detection methods available in EvDA are: inter quartile range (IQR), adjusted-quantile (AQ), Principal Components decomposition (PCOUT) and Local Outlier Factor (LOF). They are computed using the mvoutlier and Rlof R packages. And the imputation methods are the Hot-Deck, K-Nearest Neighbour (K-NN), the Iterative Robust Model-based Imputation (IRMI), and the Multiple Imputation by Chained Equations (MICE). Data analysis. Regression, classification, clustering, and PCA analysis can be performed in EvDA. Analytical methods are computed using various R packages.

Nineteen regression methods are available including: four linear regression (ordinary least square regression, stepwise linear regression, principal component regression and partial least squares regression), three penalized regression (Ridge regression, least absolute shrinkage and selection operator and, elastic net), four non-linear regression (multivariate adaptive regression splines, support vector machine, k-Nearest Neighbour and neural network) and eight decision trees (Classification and Regression Trees (CART), conditional decision trees, model trees, rule system, bagging CART, random forest, gradient boosted machine and cubist).

Sixteen classification methods can be computed they are: two linear classifiers (logistic regression and linear discriminant analysis), eight non linear classifiers (Mixture discriminant analysis, quadratic discriminant analysis, regularized discriminant analysis, neural network classification, flexible discriminant analysis, support vector machine classification, K-Nearest Neighbour classification and Naïves Bayes) and six non-linear classifies with decision trees (Classification and regression trees, bagging CART classification, Random Forest Classification, C . , C . , and PART).

The hierarchical clustering, K-means, correlation matrix and PCA can also be computed in EvDA. For clustering methods, information such as number of clusters (i.e., K-means) or the height of the cut to the dendrogram for hierarchical clustering need to be specified by user. Principal Component Analysis (PCA) and correlation matrix are performed using the FactoMineR and Hmisc R packages respectively.

. . Overview of the application

EvDA has a user-friendly interface. It consists of four main steps including: loading, inspection, preprocessing, and analysis of data. The user can click on different tabs to move between each step. Parameters are inserted through the use of widgets such as drop-down menus, sliders, check-box, etc. At each step, the user can define the settings and the results are calculated and shown automatically.

• Data input. The user's data can be uploaded by uploading a file in .csv, .txt or .xls format. Delimiters are detected automatically from the data (c.f. Figure . ) but, it is also possible to specify it manually. Sometimes the user may be interested in studying a subset of the dataset. In this case the user can select the desired rows and columns manually and use the subset as a new dataset. EvDA uses only numerical and categorical data since the methods available in EvDA are implemented mainly on these type of data. • Data inspection. Inspection of data is done through: descriptive statistics, tests of normality, analysis of missing data, and detection of outlying data. The inspection tasks are accessible by selecting the corresponding tab. The users can choose which normality test and outlier detection method to use. Results are displayed in tables and graphs (see Figure . ) and can be downloaded by users for future use. • Data preprocessing. Many real-world data may contain different type of anomalies.

To improve the quality of data and perform statistical analysis, data need to be preprocessed. EvDA provides a set of preprocessing tasks including: feature selection, normalization, imputation of missing values, and outliers processing. In order to guide the user, a panel indicating the most optimal data preprocessing procedures is shown. The user can choose one among the different methods that are available for each preprocessing task. They can be performed whether in the original input dataset or in a dataset resulted from a previous preprocessing task. An example of the information displayed on the data preprocessing tab is given in Figure . . It shows the different methods to impute missing values on a given dataset. Preprocessed data can be used for subsequent statistical analysis or downloaded for future use. • Analysis. Our prototype implements some of the functionalities that we considered the most frequently used on environmental studies. For regression analysis, independent and dependent variables can be specified by the user. The models are constructed by splitting the observations into a training and test sets. Percentage of splitting can be manually defined. Accuracy of the model is then computed and the results can be saved.

Similarly to regression, classification analysis is performed using splitted data. Prediction can be made using the resulting model and the user can obtain accuracy and Kappa measure results. The construction of clustering by K-means requires to specify the number of cluster k. For easy selection of k, EvDA performs the elbow method. A graphic representing the number of clusters against the total within-clusters sum of squares is displayed.

With the help of this visualization, the user can select the number of cluster k. Hierarchical clustering does not require to pre-specify the number of clusters. In our prototype, Hierarchical clustering is performed and the results are represented in a Development and experimental results dendrogram which is a tree-based representation of the observations. Correlation matrix is performed using all variables, the output is composed of a correlation matrix, a list of the most correlated variables and a graphical display of the correlation matrix. PCA is constructed only using numerical data. PCA output consists of a table that summarizes PCA results and, PCA plots of the observations and variables. PCA visualization is a scatterplot of the principal components corresponding to axes and and PCA plot of variables shows the projection of the variables within the first two principal components. Optionally, the user can add supplementary variables, they can be whether continuous or categorical. These variables can be used to get additional information about the variability. In the resulting plots, they are coloured differently for better visualization. In Figure . , we show an example of the information displayed when the K-means clustering analysis is performed.

In the next section we present results obtained from the application of EvDA. In the following section, we aimed at demonstrating the utility of our prototype and providing useful information to specialists regarding the water pollution of the four Mexican rivers. .

Case study: Water pollution of the Tula, Culiacan, Tamazula, and Humaya rivers By applying our methodological approaches, we aimed now at responding to a concrete applicative need in the context of water quality assessment. Precisely, we aimed at providing answers to the following specialist-driven questions:

. Considering the different sources of pollution from PPCPs and pesticides, are they found in the same place or are they found with the same concentrations in the different sampling sites? . What is the correlation between heavy metals, pesticides, PPCPs, and biomonitoring metrics? . What are the most representative or pertinent pollutants to focus on in order to assess the impact of specific anthropological activities (i.e., agricultural activities)? . Is it appropriate to analyse all the pollutants if the conclusions are similar for one type of pollutant?

. . Data description

. . . Physico-chemical and chemical data

To assess the quality of water of the Tula, Humaya, Tamazula, and Culiacan Mexican rivers, the methodological approach we designed to collect water quality data was described in Chapter . By applying our methodological approach, we have collected data that describe the physico-chemical, chemical and biological characteristics of the rivers. The physico-chemistry includes a set of physical and chemical parameters that characterize the river. Such parameters could be necessary either for aquatic biodiversity or for a pollution source. We have organized the different physico-chemical and chemical parameters into two groups: macro-pollutants and micro-pollutants.

• Macro-pollutants. They include a set of compounds that are naturally present in the river, which are necessary for good functioning of the aquatic ecosystem. We can mention for instance: organic carbon, nitrate, phosphates, sulphate or metals which are necessary for the good functioning of aquatic ecosystems. However, when their concentrations exceed normal values, pollution problems may emerge such as eutrophication or anoxia . • Micro-pollutants. They are compounds that are not naturally present in the river; they include oil, hydrocarbons, organic solvents, surfactants, pesticides, pharmaceuticals, and personal care products. They could be introduced due to filtration, runoff or direct discharge to the watercourse. A tiny dose of these compounds will poison the aquatic environment.

We have analysed water samples to determine the content of organochlorine pesticides and PPCPs. The analytical methods that we adapted for the analysis of pesticides and PPCPs are under submission.

. . . Biological data

It concerns the biological organisms (flora and fauna) living in rivers. In hydrobiology the biological organisms are grouped into: macrophytes, diatoms, fishes, macroinvertebrates, and oligochaetes. Macrophytes and diatoms are representative of the aquatic flora while fishes, macroinvertebrates, and oligochaetes are representative of the aquatic fauna. We have chosen macroinvertebrates in order to compute biological indices to assess the quality of the aquatic ecosystems of the Mexican rivers.

Macroinvertebrates are biological organisms living at the bottom of rivers and lakes. Their size is superior to . mm and they are visible without the help of special lens. Macroinvertebrates include different species: for example; molluscs, crustaceans or larvae of insects.

Macroinvertebrates are inventoried and the resulting list of taxa has been then used to compute different indices that provide information about the diversity, fauna richness, and quality characteristics of the aquatic ecosystem.

In our study case, we have collected and analysed macroinvertebrates from each sampling site. A total of indices were computed as suggested by Serrano Balderas et al. ( ).

Our dataset is composed of observations and a total of numerical variables that include macro-pollutants, micro-pollutants, and biological indices. Collection of samples was done in two seasons (rainy and dry). For the sake of simplicity, we present only the results of the dry season. In Table . , we list the different parameters that compose our dataset. Below we will refer to this dataset as the Mexican dataset.

. . Data preprocessing

Data collected from the analysis of samples can not be used in its original form. We need to perform a set of preprocessing tasks in order to initialize data properly to serve as input for subsequent statistical analysis. The preprocessing task that were implemented are ( ) feature selection, ( ) normalization, ( ) imputation of missing values, and ( ) outlying data processing. The preprocessing procedures that we applied were those proposed by EvDa.

. Normalization. Our dataset contains variables with different measurement units; to scale all numerical variables to an specific range, we have performed z-score normalization. By applying this transformation, values of variables have a mean equal to 0 and a standard deviation of 1. . Missing values. Our data subsets contained an amount of . % of missing data.

Missing data were due mainly, to some technical accidents during the manipulation of samples. To obtain a complete dataset, we have imputed missing data using the Multiple Imputation by Chained Equations (MICE) imputation method. . Outlying data processing. Detection of outliers was performed using the Inter Quartile Range (IQR) method. A total amount of . % of outlying data was detected. We decided to process outliers by imputing them using the K-Nearest Neighbour imputation method. However, they will be studied conscientiously for an extended future analysis.

. . Results of the analysis

In order to respond to the questions listed at the beginning of Section . , we have applied a set of statistical analyses to our preprocessed data. Below, we present the results.

. Considering the different sources of pollution from PPCPs and pesticides, are they found in the same place or are they found with the same concentrations in the different sampling sites?

For the sake of simplicity, we show exclusively the results obtained for the analysis of pesticides. We have computed a linear regression. First we have done a selection of variables by performing a Linear correlation-based feature selection. Feature selection was performed to obtain a regression model that represented the best our phenomena. A total of variables over the corresponding to the concentrations of organochlorine pesticides were used as predictors and a variable named distance was used as response. We have created the variable distance using Google Earth, it measures the distance between agricultural areas and each sampling site. In Figure . , we illustrate the distance taken between agricultural areas and the sampling sites C to C .

We have graphically represented the concentrations of the pesticides for each sampling site (c.f. Figure . ). In Tables . and. , we provide details of the linear model for the regression of distance between sampling sites and agricultural areas on the concentration of pesticides for the Mexican data. The R 2 value indicates that our model fits well our data, the F-statistics value suggests that at least one of the pesticides must be related to the distance between sampling sites and the agricultural areas. However, the p-values of the variables used for the analysis (c.f. Table . ) indicate that changes on the distance are not related to changes in the pesticides variables. The linear regression model using only the distance variable and the concentration of pesticides only provides us information about the relationship between the proximity of agricultural fields and the content of pesticides in the samples. This means that, the closest an agricultural field is from a sampling site, the content of pesticides in a water sample will be higher. In Figure . , we observed that site C has the highest content of the analyzed pesticides. We also observed that site C collects the water discharges of the flow and is the closest of an agricultural area (c.f. Figure . ). From these observations, we can deduce that the quantities of pesticides depend in part, on the proximity of pollution sources in this case from the agricultural fields thus, high concentrations of pesticides will be found at sampling stations closer to agricultural fields. It must be noticed that the regression model built using only the distance variable as the response and the concentrations of organochlorine pesticides as predictors does not provide complete information to model pesticides fate in surface waters. To this purpose additional parameters (i.e., organic matter content, particle size of sediment, concentration of pesticide, polarity, etc.) and complex models should be further considered (Nowell et .

.

Which is the correlation between heavy metals, pesticides, PPCPs, and biomonitoring metrics?

To distinguish relationships among the different variables, we have subsetted the original dataset into three datasets named macro, micro, and metals. The macro subset contains the general chemistry variables related to the macro-pollutants listed on Table . , while micro and metals datasets contain the variables of Micro-pollutants and Heavy metals respectively. A correlation matrix and a PCA analysis were performed on each data subset. We have constructed the PCA using the biomonitoring metrics as supplementary variables.

From the correlation matrix (c.f. Figure . ), we observed that, out of the biomonitoring metrics, only are positively correlated to the macro-pollutants, to the micro-pollutants, and to metals. A better visualization of the correlation between the different variables was obtained from a PCA analysis. Figure . shows the first two principal components for the macro-pollutants, micro-pollutants, and metals subsets respectively. For macro-pollutants and micro-pollutants, we observed that, some biomonitoring . metrics show a negative correlation with these pollutants. This observation indicates that at high concentrations of pollutants, low values of biomonitoring metrics will be observed. Such observation is consistent with biomonitoring metrics themselves indeed, certain biomonitoring metrics (i.e., BMWP, EBI, Shannon or Simpson's indices) show low values when poor aquatic biodiversity is observed therefore a polluted aquatic system. The Family Biotic Index (FBI) show a similar behaviour compared to the other metrics. In fact, FBI has high values at high concentration levels of pollutants and, as we can observe on Figure . (a) and (b) this metric is positively correlated to the macro-and micro-pollutants.

For the case of metals, our results do not provide enough information about the relationship between the metals and the biomonitoring metrics. As we observed on the PCA the percentage of explained variance provided by the two axes is % which is lower compared to the axes for macro and micro pollutants ( % and % respectively). . What are the most representative or pertinent pollutants to study in order to assess the impact of specific anthropological activities (i.e., agricultural activities)?

For this question, we focused on the impact of agricultural activities. To this, we have performed a linear correlation-based feature selection to identify the variables that are the most pertinent to assess this activity. We have used our variable distance that we previously described in Question (Considering the different sources of pollution between PPCPs and pesticides, are they found in the same place or have the same trends?).

For a detailed analysis, we have subsetted the Mexican dataset into three subsets named macro, micro, and bio. They are composed of the variables Macro-pollutants, Micro-pollutants and Biological indices as detailed in Table . . Results from the feature selection on each subset is given in Table . . For the assessment of the impact of agricultural activities, it appear that only macro-pollutants, micro-pollutants, and biological indices are pertinent to use. . Is it appropriate to analyse all the pollutants if the conclusions are similar for each type of pollutant? Certainly, each type of pollutant provide different information and scientist may be interested to study most of them. However, as we will illustrate, not all pollutants can necessarily be included to study the impact of specific anthropogenic activity. Lets us continue using the agricultural activity as the subject of the next study. To assess the impact of agricultural activities on the quality of water, scientist may be interested in determining the content of different pollutants such as metals, minerals, and pesticides. Then, the complete set of variables could be used to predict the pollution due to the proximity of agricultural activities. However, this action is not necessarily the most efficient. To exemplify this, we have performed a linear regression model. We have used the three data subsets named macro, micro, and bio as previously described in Question (What are the most representative or pertinent pollutants to study in order to assess the impact of specific anthropological activities (i.e., agricultural activities)?). We have subsetted the data in order to study separately each type of pollutant as they provide different information.

A linear correlation-based feature selection was computed for the selection of variables. The regression model was performed on the data subsets that contain the totality of variables and on the subsets after the selection of features.

In Table . , we present the results of the linear model for the regression of the distance between the sampling sites and the agricultural areas on the variables, for the macro, micro, and bio data subsets. The construction of the regression model using all variables seems to be inadequate: the results indicate that the number of observations n is not large enough to construct a model with (for macro), (micro) and (bio) variables. However, we observed that the model fits better with a reduced number of features. The most promising results were observed on the micro and bio subsets.

Our result suggest that a limited number of variables will be sufficient to construct a linear regression model to predict the impact of agricultural activities on the quality of water.

. Conclusion and future development

The analysis of environmental data demands the application of various data preprocessing tasks in order to obtain a dataset in conformance with the requirements of data analysis methods and useful for further data mining. Therefore, scientists may be confronted to two main challenges: ( ) the efficient application of data preprocessing procedures and ( ) the appropriate selection of the methods to preprocess data. To cover these challenges, we have developed a fully integrated analytics environment in R.

The originality of our prototype, is that it includes a set of data preprocessing procedures and statistical analysis for environmental data in general and for water quality data analytics in particular with user-friendly interface. The users with little knowledge on R can inspect, preprocess, and analyse their data by simply clicking on the desired action then, EvDA recommends and executes the necessary scripts to analyse data and finally returns results that can be saved. Concerning preprocessing procedures, EvDA analyses the characteristics of data and guides the user for the selection of the most appropriate preprocessing methods.

EvDA provides the following functionalities:

• Inspection of the statistical characteristics of data; • Data preprocessing procedures;

• Visual representation of analytical steps; • Display of original and preprocessed data; • Recording of the preprocessing procedures that have been executed.

We have presented our prototype EvDA for preprocessing and analysis of environmental data and its application on our case study: Water pollution of the Tula, Culiacan, Tamazula, and Humaya river. Currently our prototype has some limitations and some of the adaptations that will be implemented in a short-term future include:

• Adaptation of data preprocessing procedures for categorical data. Currently, the data preprocessing procedures are suitable only for quantitative data; • Addition of supplementary data preprocessing procedures: EvDa includes a limited number of methods to normalize, select features, impute missing values, and process outliers; • Inclusion of other data preprocessing procedures such as data transformation, discretization, instance selection, etc; • Development of more interactive options: our prototype does not allow the setting of the method parameters for most of the statistical analysis; • Development of interactive options related to visualization of data and presentation of data that may be interesting for the user; • Inclusion of spatial spatio-temporal data analysis methods: the user may also be interested in representing statistical information on the map particularly to study the dynamic phenomena.

Data obtained from environmental surveys may be prone to have different anomalies (i.e., incomplete, inconsistent, inaccurate, or outlying data). These anomalies affect the quality of environmental data and may have severe consequences when assessing environmental ecosystems. To get correct and useful results from data mining and statistical analysis tasks, it is necessary to acquire quality data and preprocessed data. Different shortcomings are associated to the acquisition and preprocessing of environmental data. In chemical data acquisition for instance, there are analytical techniques with limited performance, non standardized methods or ambiguous data collection protocols. In data science, some of the shortcomings include: the sequence of data preprocessing procedures is not handed and depends on the dataset characteristics; the impact of data preprocessing procedures on statistical results has been understudied. Within the context of this thesis, we were interested on a particular shortcoming, the impact of data preprocessing procedures on subsequent statistical analysis and in one specific applicative domain: water quality assessment. Although we have applied our approaches in this domain, they can be adapted for analysis and preprocessing of other type of environmental data.

Collection, analysis, and preprocessing of environmental data is a complex subject and it requires specialized knowledge of each domain. Environmental scientists are interested among others in developing methods and/or protocols to analyse and collect environmental samples while data scientist are interested in developing algorithms and information systems to better analyse, preprocess, and visualize any data. The goal is to provide scientific solutions for understanding environmental phenomena that involve both environmental and data science. The challenges are vast and, in this work, we focused on one main problem: How to analyse environmental data and provide accurate results? To answer this question we propose a suite of methodological approaches to collect, preprocess, and analyse environmental data in order to control the entire process and guarantee accurate analysis results.

We were interested on the development of approaches and tools that will allow scientists to: ( ) acquire quality data and ( ) perform the most appropriate data preprocessing procedures to finally obtain accurate results in the context of water quality assessment.

Our approaches are multidisciplinary and thus related to both environmental sciences and data science, they include:

• a method for the collection of water quality data, specifically for the quantification of pollutants at low concentrations, which reduces incompleteness of data;

• a methodological procedure for the collection of biological data, that diminishes data errors due to outlying data, missing values or inconsistencies; • an approach to identify the optimal data preprocessing strategies for specific statistical methods.

. Contributions

We have provided contributions related to Environmental Chemistry, Hydrobiology, and Data Science. Hereafter, we detail each one.

Environmental Chemistry:

• The analysis of pesticides in samples of water: The existing analytical methods to quantify emerging pollutants on environmental samples use advanced instrumentation such as Liquid Chromatography tandem Mass Spectrometry (LC-MS-MS) or Gas Chromatography Mass Spectrometry (GC-MS). However, in developing countries access to advanced analytical instrumentation may be limited as consequence, analysis of emerging pollutants is rarely done. We adapted an analytical method for the simultaneous analysis of eighteen organochlorine pesticides in samples of water including: I-BHC, II-BHC, III-BHC, IV-BHC, heptachlor, aldrin, heptachlor epoxide, I-endosulfan, II-endosulfan, dieldrin, DDE, endrin, DDD, endrin aldehyde, endosulfan sulfate, DDT, endrin ketone and methoxychlor. The proposed analytical method uses the Solid Phase Extraction (SPE) to extract analytes followed by Gas Chromatography-Electron Capture Detector (GC-ECD) for their quantification. The Liquid-Liquid extraction method (LLE) has been used for the analysis of organochlorine pesticides (Wu et al., ; Diaz et al., ) however, it is a laborious method, use large amount of solvent and require intensive cleaning procedures to assure contaminant free material to determine pesticides at trace levels. The analytical method that we proposed requires small amount of solvent, is simple, it provides better recovery for the majority of the pesticides (recovery ranging from % to % on average) compared to LLE (recovery ranging from % to %) and reduces incertitudes and errors that may appear due to intensive manipulation of samples.

• Analysis of Pharmaceutical and Personal Care Products (PPCPs) in samples of water: A collaborative work between the Institute of Geophysics and the Faculty of Chemistry at UNAM has also been done to develop a new Solid Phase Micro Extraction-Gas Chromatography-Mass Spectrometry (SPME-GC-MS) method for the simultaneous analysis of PPCPs (ibuprofen, -benzyl--chlorophenol, naproxen, triclosan, ketoprofen, diclofenac, bisphenol A and estrone) in river water. Compared to other methods (Huang et al., ; Yu et al., ), ours is simple, rapid and efficient. It allows to extract target analytes from a miniaturized system where volume samples reach few millilitres and where manual intervention of the analyst is reduced to the minimum. This is advantageous to reduce anomalies on data.

The analytical methods that we propose allow us to quantify pesticides at a concentration range of µg L -1 and ng L -1 for pesticides and PPCPs respectively. We propose to use our analytical methods to reduce the presence of data anomalies such as missing values, outliers, and censored data.

Hydrobiology:

Use of biomonitoring metrics for the assessment of water quality in Mexico is scarce despite their advantages (Mathuriau et al., ). Their limited used is due to: absence of sampling and analytical protocols, lack of identification keys, incomplete information of local communities and absence of monitoring metrics. These shortcomings increase the problems on data quality. We have designed a new methodological approach for the acquisition of data using biomonitoring metrics in Mexico. We have done a bibliographic study where we identified thirty five macroinvertebrate-based biomonitoring metrics of potential use for the ecological assessment of surface waters in Mexico. Metrics were selected considering the following characteristics: sensitivity, ecological relevance, representative, feasibility, metric interpretation, performance, and geographical suitability. Sampling and analytical procedures to compute the selected metrics are also described. We suggest the use of our methodological approach to increase the quality of data, reduce uncertainties, incompleteness or inconsistencies in the dataset due to undefined sampling and analytical protocols for biomonitoring assessment in Mexico.

Data Science:

When analysing environmental data, various data preprocessing tasks need to be applied to obtain a dataset in conformance with the requirement of data analysis methods, and useful for further data analysis. Numerous data preprocessing procedures are available (c.f. Chapter . ) and their appropriate selection is necessary to get the less biased statistical analysis results. An important aspect when analysing environmental data is that contrary to data scientists, environmental scientists are not familiar with the tasks to preprocess data such as: data cleaning, normalization, data transformation, feature selection, etc. Therefore, data preprocessing and data analysis may be a very critical step. We aim at providing to environmental scientists a guide to inspect, preprocess and analyse environmental data. Our contributions are:

• Selection of data preprocessing procedures: We proposed an optimal selection of data preprocessing procedures to treat common data anomalies and data problems (feature selection, normalization, missing values and outliers). They consist of procedures already available on the R environment for statistical computing. The selected preprocessing procedures respond to the need to treat data anomalies of datasets with different dimensions, providing accurate statistical results. • Assessment of data preprocessing procedures: We evaluated the impact of data preprocessing procedures on subsequent statistical analysis. From this evaluation we suggested the data preprocessing procedures that are the most appropriate to get the least biased analytical results. • Development of an integrated analytics environment in R: An integrated analytics environment, named EvDA, for statistical analysis of environmental data, was proposed. EvDA is a user-friendly Shiny/R application relying on statistics to guarantee data quality and quality results. It allows users with little knowledge of R to inspect, preprocess, and analyse their data. An stable version of our prototype will be available on Github in a short-term future.

.

Future work

The approaches developed in this work are, to our-point-of-view, an important step on environmental informatics. However, there are still more work to be done including:

For our approaches in Environmental Chemistry and Hydrobiology:

• Adapt of our analytical technique (SPE GC-ECD) for the analysis of other types of Chapter emerging pollutants and types of samples (i.e., soils and sediments). We were focused on the development of an analytical method that allow us analyse specifically organochlorine pesticides and some PPCPs but, many other pollutants affect the aquatic environment (e.g., oils, surfactants, industrial wastes,etc.) and the development or adaptation of methods to analyse them is necessary; • Development of local biomonitoring metrics. We have demonstrated the usefulness of biomonitoring metrics using macroinvertebrates for the water quality assessment of Mexican rivers. However, it will be favourable to develop a biomonitoring metric suitable to the ecological, climatological and geographic characteristics of Mexico.

Because in Mexico the diversity of biological species is vast, there are endemic species, and the climate and geographic characteristics are heterogeneous and complex.

For our approaches in Data science:

We have designed our experimentations to study the most frequent data anomalies (non normalized data, irrelevant features, missing values, outlying data) using the most general case (datasets with multivariate normal distribution). However, other data anomalies may appear on data (e.g., censored data, uncertainties), and real-world dataset may have different characteristics to those conducted in our study. Due to our limited time, we have decided to focused on the above mentioned cases but it most be noticed that further studies need to be done in order to cover the full spectrum of possibilities in order to appropriately preprocess data.

In addition, our experimentations were developed under the assumption that data anomalies appear as isolated cases. However, data anomalies may co-occur (Berti-Equille et al.,

) thus, the results of data preprocessing may be different to those observed in our study. We insist on the need to continue studying in this topic and some future work that we suggest include:

• Enlarge the study related to data anomalies and data preprocessing. In this study we were focused on a limited number of data anomalies and data preprocessing procedures. To continue contributing to this area of study, other types of data anomalies (e.g., censored data, duplicates, data redundancies, inconsistencies) and preprocessing procedures (e.g., transformation, discretization, integration) should be studied; • Adjust our approach to other environmental data such as ecological biodiversity, geological studies, air or soil pollution; • Study the effect of preprocessing procedures under the presence of multiple data anomalies; • Adjust our approach to datasets of varied characteristics.

We also have proposed a prototype to inspect, preprocess and analyse environmental data. It can be used as a basis to create robust-user-friendly tools that allow preprocess and analyse environmental data. Anomalies on data are not exclusive of environmental studies, they may occur also in biology, astrophysics, geology or engineering, to mention some. It is necessary to continue producing knowledge on this exiting and interesting area.

Related publications

• Diaz-Flores, L., Peña, A., Armienta-Hernandez, M., and Serrano-Balderas, E. (under revision). Determination of pharmaceuticals and personal care products (PPCPs) in river water and sediment by solid phase extraction followed by gas chromatography mass spectrometry (SPME-GC-MS). Submitted to Journal of Analytical Chemistry. -.

Oral communications

CHAPTER

Résumé étendue

Confrontés à la nécessité de réduire la pollution d'eau, des programmes de surveillance de la qualité de l'eau sont fréquemment mis en place. De tels programmes consistent à quantifier divers composants physiques et chimiques dans différents sites et pendant plusieurs périodes. Or les données collectés lors des programmes de surveillance sont sujets aux différents anomalies (i.e., incomplets, inconsistantes, imprécis ou aberrants). Des anomalies dans les données sont omniprésents et peuvent être présents dû aux problèmes dans les expériences, par des erreurs humains ou défaillances du système.

La mauvaise qualité de données peut être significativement coûteuse (Haug et al., ) et avoir des sévères conséquences dans l'évaluation des écosystèmes environnementaux (Wahlin and Grimvall, ). Pour produire des données de qualité et réduire des anomalies de données il est nécessaire : acquérir des données de qualité et pré-traiter des données (Han and Kamber, ; Berti-Equille, a).

L'acquisition des données de qualité peut être obtenue ) en suivant des protocoles d'échantillonnage et d'analyse normalisés et ) en utilisant des outils analytiques avancées (i.e. spectrométrie de masse couplée à un plasma inductif (SF-ICP-MS), chromatographie en phase gazeuse -spectrométrie de masse (CG-MS) ou chromatographie liquide couplée à la spectrométrie de masse en tandem (LC-MS/MS)). Cependant, dans les pays en voie de développement l'accès aux tels outils analytiques est limité ou des protocoles normalisés sont difficiles à mettre en place. Il est donc nécessaire d'adapter des outils qui peuvent être faciles à utiliser et bon marché sans compromettre la qualité des données.

Concernant le pré-traitement des données il s'agit de tout procédure qui est nécessaire afin de générer des données de qualité. Ces procédures incluent :

• Nettoyage des données (i.e., suppression de valeurs aberrantes et des incohérences) ; • Élimination des inconsistances des données (i.e. réglage des différences); • Complétude des données manquantes (i.e., remplissage des valeurs manquantes); • Réduction des données (i.e., sélection d'attributs relevants) ou création des données agrégées.

Bien que la détection des anomalies dans les données ait été largement étudiée, très peu des travaux ont été publié en particulier dans le domaine de sciences de l'environnement. En considérant que les anomalies sur des données peuvent apparaître tout au long d'une étude environnementale il est nécessaire d'identifier, contrôler et dans le cas nécessaire Chapter designer des protocoles d'acquisition et pré-traitement des données pour garantir des résultats précis.

Le travail de thèse présenté ici est positionné sur l'informatique environnementale puisqu'il a une approche pluridisciplinaire entre chimie de l'environnement, hydrobiologie, statistique, science des données et informatique. Ce travail a été centré dans deux défis principaux : l'acquisition des données de qualité et la définition des procédures de prétraitement des données appropriés pour finalement analyser des données.

. Motivations

Lors Les procédures de pré-traitement de données amélioreront certainement la qualité de données cependant, des bonnes pratiques d'acquisition des données sont aussi nécessaires pour éviter des anomalies de données. Un problème important sur l'acquisition de données environnementales est l'absence des protocoles standardisés pour l'échantillonnage et l'analyse en particulière dans des laboratoires où l'accès aux instrumentations analytiques est limité. Il est donc nécessaire d'adapter des procédures analytiques qui sont rentables et faciles de mettre en oeuvre pour obtenir des données de qualité.

Dans ce travail de thèse un intérêt particulier a été porté sur le développement d'une nouvelle approche qui combine à la fois des bonnes pratiques d'acquisition et de prétraitement des données.

. Objectifs

L'objectif principal de ce travail est de fournir à la communauté scientifique des approches méthodologiques et des outils pour l'acquisition, le pré-traitement et l'analyse des données environnementales en garantissant la qualité de résultats d'analyse et de données. Les approches méthodologiques ont été appliqués dans un cas d'étude : évaluation de la qualité de l'eau dans quatre rivières mexicaines (Tula, Tamazula, Humaya et Culiacan).

Le but de fournir une approche intégrée qui combine à la fois la bonne acquisition de données et des bonnes pratiques de pré-traitement de données est de contrôler tout la chaîne de traitement de données dès leur acquisition et jusqu'à leur analyse. Pour attendre l'objectif principal des objectifs spécifiques ont été définis. Ils sont liés aux trois domaines d'étude.

Chimie de l'environnement

• Acquérir des données en déployant des méthodes fiables et bon marché pour la quantification des pesticides organochlorés, des produits pharmaceutiques et de soin dans l'eau; • Définir des protocoles pour l'échantillonnage et l'analyse des pesticides organochlorés, des produits pharmaceutiques et de soin;

• Mettre en place la campagne d'échantillonnage et les procédures analytiques pour l'analyse des échantillons d'eau des rivières mexicaines Tula, Humaya, Tamazula et Culiacan.

Hydrobiologie

• Acquérir des données en définissant une approche méthodologique à l'aide de métriques de biosurveillance (macroinvertébrés) en tant que nouveaux outils complémentaires pour la surveillance de la qualité d'eau des rivières mexicaines; • Mettre en place la campagne d'échantillonnage et les procédures analytiques pour l'acquisition des données hydrobiologiques.

Sciences des données

• Définir une approche méthodologique pour la sélection des procédures de pré-traitement des données afin de traiter des anomalies sur les données et des problèmes sur les données (i.e., valeurs manquantes, valeurs aberrantes, sélection des variables, normalisation); • Évaluer l'impact des procédures de pré-traitement des données sur des ultérieure analyses statistique; • Déterminer les procédures de pré-traitement de données les plus appropriés pour obtenir les résultats des analyses moins biaisées; • Préciser les procédures pour pré-traiter et analyser des données qui sont nécessaires pour garantir la fiabilité des résultats sur des études environnementaux en général et pour l'analyse de données de qualité de l'eau en particulier.

.

Acquisition des données

Dans le but d'acquérir des données de bonne qualité des approches méthodologiques en chimie de l'environnement et en hydrobiologie ont été proposées. Il s'agit des approches pour l'acquisition des données issue de la surveillance de la qualité de l'eau par rapport à la biosurveillance à l'aide des macroinvertébrés et à la pollution par : pesticides, produits pharmaceutiques et de soin. Pour attendre les objectifs préalablement décrit, des échantillons d'eau et des macroinvertébrés ont été prélevés dans quatre rivières mexicaines (Tula, Tamazula, Humaya et Culiacan).

. . Sites d'étude

Cinq sites (H -H ) situés au long de la rivière Tula ont été sélectionnés pour prélever des échantillons (c.f. Figure .

. . Analyse des pesticides organochlorés et PPCPs

Une méthode analytique pour l'analyse des pesticides organochlorés a été adapté en modifiant la méthode décrit par Guardia Rubio et al. ( ). L'analyse a été réalisé en utilisant une extraction en phase solide (SPE) et la chromatographie en phase gazeuse avec détection par capture d'électrons (GC-ECD). L'analyse de PPCPs a été effectué en utilisant une nouvelle méthode basée sur la micro-extraction en phase solide suivi de la chromatographie en phase gazeuse couplé à un spectromètre de masse (SPME -GC -MS). La nouvelle méthode permet la quantification de PPCPs (ibuprofen, benzyl-chlorophenol, naproxen, triclosan, ketoprofen, diclofenac, bisphenol A et estrone) dans des échantillons d'eau à niveau de traces (ng/L). Les détails sur la méthode SPME-GC-MS sont présentés par Diaz-Flores et al. (en révision, Determination of pharmaceuticals and personal care products (PPCPs) in river water and sediment by solid phase extraction followed by gas chromatography-mass spectrometry (SPME-GC-MS) soumis à Analytical Chemistry) et donc il ne sera pas détaillé ici. L'extraction en phase solide a été réalisée pour extraire des pesticides dans des échantillons d'eau. La première phase de la procédure consiste à filtrer les échantillons à travers un filtre Millipore . µm. Ensuite l'échantillon est déposé dans des cartouches C qui sont installés sur un système manifold. Les cartouches sont premièrement conditionnées en faisant passer mL de cichloromethane, mL de methanol et mL d'eau distillé avec un flux de mL/min. Ensuite mL d'échantillons d'eau sont versées à travers les cartouches (flux : mL/min). La cartouche a été séchée pendant min sous vide. En fin l'extrait a été mis à sec dans un courant d'azote et redissous dans mL d'ethyl acetate-hexane : %, v/v.

Les données obtenus lors de l'analyse de pesticides et PPCPs est constitué de individus (sites d'échantillonnage) et variables ( PPCPs et pesticides organochlorés) des valeurs numériqes.

. . Acquisition des données hydrobiologiques

Afin de fournir des lignes directrices et des outils pratiques pour la biosurveillance des systèmes aquatiques une étude à été menée dont les objectives ont été : ) identifier des méthodes de biosurveillance basées sur des macroinvertébrés qui peuvent être potentiellement utilisées dans l'évaluation écologique des rivières au Mexique, et ) décrire les procédures d'échantillonnage et d'analyse pour mettre en place les méthodes identifiées en . Les résultats de cette étude ont été publiés (Serrano Balderas et al., ) et font partie de ce manuscrit. De manière synthétique trente-cinq métriques ont été identifiées comme des outils de potentiel application pour la surveillance écologique des rivières mexicaines.

Les procédures d'échantillonnage et d'analyse pour calculer les métriques sélectionnées sont décrits dans la même publication.

Le prélèvement des macroinvertébrés a été effectué tout d'abord par une inspection visuelle d'un tronçon de la rivière afin d'identifier les différents habitats et substrats. Des échantillons ont été prélevés dans des habitats multiples à l'aide d'un filet surber (surface m 2 , maillage µm). Pour chape point d'échantillonnage, des prélèvements ont été répliqués deux fois. Les échantillons ont été stockés dans des sacs en plastique et conservés dans de l'éthanol %.

Les échantillons ont été analysés dans le laboratoire. Les macroinvertébrés ont été séparés des substrats et triés en utilisant une maille de µm. Ensuite, ils ont été examinées sous un stéréoscope et identifiés au niveau des familles à l'aide des différents clés taxonomiques ( .

Pré-traitement et analyse des données

Afin de fournir aux spécialiste de l'environnement un guide pour inspecter, pré-traiter et analyser des données environnementales une étude sur des meilleurs pratiques pour pré-traiter les données a été menée. Cette étude a été centré sur des procédures de prétraitement des données (sélection des variables, normalisation des données, imputation des valeurs manquantes et traitement des valeurs aberrants) et leur impact sur des subséquent analyses statistique principalement les méthodes de régression, classification et groupement. Les objectives de l'approche sont les suivants :

• Évaluer la robustesse des méthodes pour traiter des valeurs manquantes et aberrants ;

• Examiner l'effet des procédures de pré-traitement de données sur la précision dans des analyses de classification, groupement et régression ; • Identifier les meilleurs procédures de pré-traitement des données pour une analyse statistique particulière.

L'étude a été structurée en trois parties : ( ) génération des données synthétiques, ( ) pré-traitement des données et ( ) analyse statistique (c.f. Figure . ). Dans la première partie, des données synthétiques ont été construites afin d'évaluer les procédures de prétraitement de données puis, à l'étape deux, des données pré-traités ont été construites en utilisant différents procédures de pré-traitement sur les données synthétiques qui ont été construites dans la première partie. Dans la troisième étape, les données pré-traitées ont été utilisées pour évaluer l'impact des procédures de pré-traitement sur les résultats des analyses statistique. Une description détaillée de chaque partie de l'étude est donnée ci-dessous.

. . Données synthétiques et semi-synthétiques

Quatre jeux de données synthétiques ont été utilisés pour évaluer l'impact des procédures de sélection de variables sur des subséquentes analyses , n = , n = ), variables numériques (i.e., p = , p = , p = , p = ) suivant une loi normale et une variable catégorielle de cinq classes suivant une distribution uniforme. Des valeurs manquantes on été injectés de manière aléatoire sur les variables numériques sur chaque jeu de données dans les proportions suivantes : %, %, %, %, % et %. Pour chaque injection de valeurs manquantes dix répétitions ont été effectués pour s'assurer que les résultats expérimentaux ont été statistiquement acceptables.

En fin, des tests pour évaluer le pré-traitement de valeurs aberrants a été exécuté en utilisant trois jeux de données synthétiques composées d'un nombre différent des observations (i.e., n = , n = , n = ), variables numériques (i.e., p = , p = , p = ) suivant une loi normale et une variable catégorielle de cinq classes suivant une distribution uniforme. Des valeurs aberrantes ont été injectés de manière aléatoire sur chaque jeu de données dans les proportions suivantes: . %, . %, %, % et %.

Nous avons utilisé également des données semi-synthétiques. Tout d'abord nous avons collecté des données réelles issues de campagnes de surveillance des eaux de la région Rhin-Meuse, France. Ce jeu de données se composent de individus (stations d'échantillonnage) et variables. Cet ensemble de données a été traitée afin d'éliminer les données bruitées. Le jeu de données nettoyé se composait de individus et variables. A partir de ce jeu de données deux ensembles de données semi-synthétiques ont été générés. Le premier est une présentation réduite, il est composé de individus et variables sélectionnés de manière aléatoire. Le deuxième jeu de données a été construit en introduisant de nouvelles observations. Ces observations suivent la même répartition que le jeu de données nettoyé. Le deuxième jeu de données est composé de individus et variables.

Generation of

Synthetic datasets

Statistical analysis

Data Preprocessing

Original

. . Pré-traitement de données

Dans la deuxième partie de notre étude, nous avons cherché à évaluer la robustesse des méthodes pour traiter les valeurs manquantes et aberrantes et, pré-traiter les données par sélection des variables et normalisation. Pour attendre les objectifs de cette étude, les données déformés, construites dans l'étape précédente, ont été pré-traités en utilisant différents méthodes (imputation des valeurs manquantes, traitement des valeurs aberrants, sélection des variables et normalisation de données).

Trois méthodes de sélection de variables ont été exécutés (Correlation-based feature selection (CFS), linear-based correlation (LC) et Wrapper subset evaluator (WR)) afin de sélectionner un sous-ensemble de données optimale. La sélection de variables sur chaque jeu de données déformée a été implémenté comme suit: tout d'abord, une variable a été utilisé comme variable indépendante, puis, en utilisant les méthodes de sélection de variables et par rapport à la variable indépendante, le meilleur sous-ensemble de données a été choisi. Le sous-ensembles de données qui en résultent ont par la suite servis à évaluer l'impact des méthodes de sélection de variables sur les résultats des analyses statistiques.

Trois méthodes de normalisation ont été implémentés (min-max, Z-score et decimal scale) sur les données numériques. Les données qui en résultent ont par la suite été utilisés pour évaluer l'impact des méthodes de normalisation sur les résultats des analyses statistiques.

Les valeurs manquantes dans chaque jeux de données déformés ont été traités en utilisant quatre différents méthodes d'imputation (Hot-deck, K-NN, mice et IRMI ). Les jeux de données imputés ont par la suite été utilisés pour évaluer l'impact des méthodes d'imputation sur les résultats des analyses statistiques.

Les données déformés par des valeurs aberrantes ont été traités tout d'abord, en utilisant quatre différents méthodes de détection de valeurs aberrantes (Inter Quartil Range (IQR), Adjusted-Quantile, Principal Components decomposition (PCOUT) et Local Outlier Factor (LOF)). Par la suite, les valeurs aberrantes ont été remplacés en utilisant des méthodes d'imputation (Hot-deck, K-NN, mice et IRMI). Les données pré-traitées ont été utilisées pour évaluer l'impact du traitement des valeurs aberrants sur les résultats des analyses statistiques.

. . Analyse statistique

Pour évaluer l'impact des procédures de pré-traitement des données différents méthodes statistiques ont été appliquées aux données pré-traitées. Les méthodes statistiques utilisées ont été choisis car ils sont fréquemment utilisées dans des études environnementaux. La comparaison des résultats statistiques sur chaque jeu de donnée pré-traitée a été effectuée en calculant différents erreurs statistiques. Par simplicité, chaque anomalie de données a été étudiées en les considérant comme cas isolées. Les anomalies sur les données qui occurrent simultanément dans un jeu de données n'as pas fait l'objet d'étude dans cette thèse.

Trois méthodes de régression ont été implémentés sur les jeux de données pré-traitées. Ils incluent :

• Penalized regression : Leas Absolute Shrinkage and Selection Operator (LASSO) ; • Linear regression : Ordinary Least Squares Regression (OLSR) ; • Non-Linear regression : Multivariate Adaptive Regression Splines (MARS). L'analyse par régression sur chaque jeu de donnée a été implémentée de la manière suivante :

. Les données ont été divisées de manière aléatoire en deux sous-ensemble de L'analyse de classification sur chaque jeu de données pré-traité a été exécutée dans quatre étapes :

. Les données ont été divisées en deux sous-ensemble de données où % des données ont été utilisées pour training et % pour test ; . Le modèle de classification a été construit sur le sous-ensemble training, le modèle obtenue a par la suite été utilisé pour prédire les réponses pour les observations dans le sous-ensemble test ; . Le résultat de la classification a été évalué en calculant l'exactitude et le coefficient de Cohen's Kappa ; . Finalement, l'erreur de pré-traitement a été calculé en utilisant les formules suivantes :

Absolute Error Accuracy processing = |Accuracy P reprocessed -Accuracy original | ( . )

Absolute Error Kappa processing = |Kappa P reprocessed -Kappa original | ( . )

L'analyse de groupement a été implémenté en utilisant les méthodes K-means et regroupement hiérarchique sur les données pré-traitées. La méthode K-means a été implémenté en spécifiant tout d'abord le nombre de clusters K à l'aide de la méthode de coude. La méthode de coude à d'abord été appliquée à l'ensemble de données originaux non déformées et le nombre de clusters trouvé a été utilisé sur les données pré-traitées. La méthode de regroupement hiérarchique a été exécutée à l'aide de méthode d'agglomération de Ward.

Les résultats de regroupement de la méthode K-means de données pré-traitées ont été comparés contre les résultats de regroupement de données originaux non-déformées à l'aide des indices Rand, Adjusted Rand et Jaccard (Meila, ).

Concernant les analyses par regroupement hiérarchique les résultats des données prétraitées ont été comparées contre ceux de données non-déformées à l'aide du coefficient Cophenetic de corrélation (Sokal and Rohlf, ).

Chapter L'approche méthodologique pour évaluer l'impact des procédures de pré-traitement de données sur les résultats des analyses statistiques nous a permis d'identifier les meilleurs méthodes de pré-traiter de données pour chaque une de méthode d'analyse statistique étudiée. Nous avons découvert qu'il n'y a pas un procédure de prétraitement de données universel puisqu'il dépend des caractéristiques de données et termes de taille, distribution, Skewness, Kurtosis, etc. Les résultats de cette étude ont été utilisés pour établir un ensemble de règles permettant de combiner de façon optimale les procédures de pré-traitement et d'analyse de données. Cette règles par la suite ont été utilisés sur la construction d'un environnement analytique intégré sour la forme d'une application développée en R pour l'analyse statistique des données environnementales en général et l'analyse de la qualité de l'eau en particulier. L'étude réalisée ici a utilisé des données multivariée avec une distribution normale et sur quatre procédures pour préparer les données (normalisation et sélection des variables) et traiter des anomalies (valeurs manquantes et valeurs aberrantes). En conséquence, l'ensemble de règles qui ont été obtenus n'est pas exhaustive, mais il pourrait être utilisé comme base et encore plus important, il peut être complété par l'étude d'autres type des données, des anomalies de données et procédures de pré-traitement de données.

. Développement

Un prototype nommée EvDa est présenté ici, il vise à inspecter, pré-traiter et analyser les données environnementales d'un manière plus facile. L'utilisateur peut télécharger ses propres données. Les procédures de pré-traitement peuvent être exécutées séquentiellement et l'analyse de données telles que : régression, classification, groupement ou ACP produisent des résultats utiles pour les experts du domaine à l'aide d'une interface graphique facile à utiliser.

EvDA se compose de quatre étapes principales : téléchargement, inspection, prétraitement et analyse des données. L'utilisateur peut cliquer sur les différents onglets pour se déplacer entre chaque étape. Les différents paramètres sont insérés par l'utilisation de widgets tels que des menus déroulants, case, etc. A chaque étape, l'utilisateur peut définir les paramètres et les résultats sont calculés et montrés de manière automatique (c.f. Figure . ).

• Data input : L'utilisateur peut télécharger des fichiers en format .csv, .txt ou .xls. Dans certain cas l'utilisateur peut être intéressé en étudier un sous-ensemble de données, dans ce cas l'utilisateur peut sélectionner les colonnes et lignes et utiliser le sous-ensemble de données comme nouveaux jeu de données. EvDA utilise uniquement des données numériques et catégorielles. • Inspection de données: L'inspection de données est implémenté en utilisant des analyses statistiques descriptives, des tests de normalité, des analyses de valeurs manquantes et détection des valeurs aberrants. L'utilisateur peut sélectionner parmi les différents méthodes de normalisation et détection des valeurs aberrants. • Pré-traitement de données: Différents procédures de pré-traitement de données sont proposés y compris: sélection des variables pertinents, normalisation, imputation des valeurs manquantes, et traitement des valeurs aberrants. L'utilisateur peut choisir parmi les différentes méthodes qui sont disponibles pour chaque tâche de prétraitement. Ils peuvent être effectués dans le jeu de données d'entrée original ou dans un jeu de données qui a résulté d'une tâche précédente de pré-traitement. • Analyse de données: EvDA inclus des analyses par régression, classification, groupement et ACP. Pour une anlayse de régression, les variables indépendantes et dépendantes peu- Les approches méthodologiques développés en chimie de l'environnement, hydrobiologie et science des données ont par la suite été utilisées pour évaluer la pollution de l'eau dans quatre rivières mexicaines (Tula, Tamazula, Humaya et Culiacan). Les résultats des analyses sur les données du Mexique montrent que les activités d'agriculture, les déchets urbains et industrielles polluent l'eau des rivières étudiés. Les résultats observés indiquent que des mesures de protection et remédiation sont nécessaires pour améliorer et protéger la qualité des écosystèmes aquatiques dans les rivières mexicaines Tula, Tamazula, Humaya et Culiacan.

. Conclusions

Les contributions multidisciplinaire de la thèse sont : ( ) en chimie de l'eau : une procédure méthodologique permettant de déterminer les quantités de pesticides organochlorés dans des échantillons d'eau collectés sur le terrain en utilisant les techniques SPE-GC-ECD (Solid Phase Extraction -Gas Chromatography -Electron Capture Detection) ; ( ) en hydrobiologie : une procédure méthodologique pour évaluer la qualité de l'eau dans quatre rivière Mexicaines en utilisant des indicateurs biologiques basés sur des macroinvertébrés ; ( ) en science des données : une méthode pour évaluer et guider le choix des procédures de pré-traitement des données produites lors des deux précédentes étapes ainsi que leur analyse ; en fin ( ) le développement d'un environnement analytique intégré sous la forme d'une application développée en R pour l'analyse statistique des données environnementales en général en l'analyse de la qualité de l'eau en particulier. Enfin, nous avons appliqué nos propositions sur le cas spécifique de l'évaluation de la qualité de l'eau des rivières Mexicaines Tula, Tamazula, Humaya et Culiacan dans le cadre de cette thèse qui a été menée en partie au Mexique et en France. ). ASPT is calculated by dividing the BMWP score by the total number of contributing taxa. High ASPT values characterize clean sites with relatively large numbers of high scoring taxa and low ASPT values are distinctive of polluted sites that do not support many high scoring taxa (c.f. Table A . ). Functional Feeding Groups measures Six FFG measures are described in here, including filtering collectors, scrapers, shredders, predators and gathering collectors. The Functional feeding group approaches are based on the behavioural and feeding mechanisms by which macroinvertebrates obtain their food resources (c.f. Table A. ) (Merrit et al., ). ). The Macroinvertebrate-based Index of Biotic Integrity (IBI) designed for west-central Mexican streams was computed. It comprises eight metrics (catch per unit effort, generc richness, % EPT genera, % Chironomidae individuals, Hilsenhoff Biotic Index, % depositional individuals, % predator individuals and % gatherer genera). Macroinvertebrate IBI score was calculated by summing component metric scores (c.f. Table A . and Table A. ). The final IBI score is then used to qualify the stream. IBI classifies streams into five classes (very poor, poor, fair, good and very good). Low IBI scores correspond to streams with very poor quality. -FQ semi-synthetic dataset (n = 7520, p = 26). The µ and σ 2 values of all variables are shown.

Biological Monitoring Working

  quality monitoring workflow. . . . . . . . . . . . . . . . . . . . Basic chromatographic process. (a) The different components (C: glass column, SP: stationary phase, MP: mobile phase, S: sample); (b) introduction of the sample; (c) elution of constituents of the mixture; (d) recovering of compounds after separation. Figure taken from Rouessac and Rouessac ( ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Schematic representation of chromatographic instrumentation . . . . . . . . . Structures of missingness: (a) univariate pattern, (b) monotone pattern, and (c) artibtrary pattern. Rows correspond to observations and columns to variables (Cottrell et al., ). . . . . . . . . . . . . . . . . . . . . . . . . Representation of (a) missing completely at random (MCAR), (b) missing at random (MAR), and (c) missing not at random (MNAR) in a univariate missing-data pattern. X represents completely observed variables, Y denotes partly missing variable, Z represents the component of the causes of missingness unrelated to X and Y, and R represents the missingness (Schafer and Graham, ). . . . . . . . . . . . . . . . . . . . . . . . . . . . Sampling sites in the Tula river (Hidalgo state, Mexico) . . . . . . . . . . . . Sampling sites in the Tamazula, Humaya, and Culiacan rivers (Sinaloa state, Mexico). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Flow diagram of the SPE method for analysis of OCPs in water. . . . . . . . Example of macroinvertebrates counting process to determine the number of runs in a sample. Each form represents a different organism. The number of runs is used to compute the Seqential Compraison index (SCI). . . . . . . General procedure for assessment of data preprocessing procedures on statistical results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Flow diagram of the generation of synthetic datasets for the assessment of preprocessing procedures on statistical results. FT denotes highly correlated data, AL denotes non-normalized data, NA stands for not available/missing values and OUT denotes outlying data. . . . . . . . . . . . . . Flow diagram of the generation of semi-synthetic datasets for the assessment of preprocessing procedures on statistical results. . . . . . . . . . . . . . . . . Extract of the general workflow for assessment of data preprocessing procedures. Figure shows the workflow diagram of data preprocessing assessment. . Experimental procedure for assessment of imputation methods. . . . . . . . xv .

.

  Similarity and Diversity indices. . . . . . . . . . . . . . . . . . . . . . . . . . . Biotic indices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Location of sampling sites of the Tula river. . . . . . . . . . . . . . . . . . . . Location of sampling sites of Tamazula, Humaya and Culiacan rivers. . . . . Example of data from the physochemical and chemical values obtained from the analysis of water samples of the rivers Tula. Tamazula. Humaya and Culiacan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example of the data from the concentration of arsenic and heavy metals on water samples of the rivers Tula, Tamazula, Humaya and Culiacan. . . . . . . SPE-GC-ECD limit of detection (LOD), limit of quantification (LOQ) and recovery values for water analysis. . . . . . . . . . . . . . . . . . . . . . . . . SPME-GC-MS limit of detection (LOD), limit of quantification (LOQ) and recovery values for water analysis. . . . . . . . . . . . . . . . . . . . . . . . . Example of data from the analysis of organochlorine pesticides on water samples from the Tula river. Only ten out of the six-teen compounds are shown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example of data from the analysis of PPCPs on water samples from the Tula river. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Biomonitoring metrics for the biological assessment of Mexican rivers (Serrano Balderas et al., ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Example of data from the biomonitoring metrics computed for the rivers Tula, Tamazula, Humaya and Culican. . . . . . . . . . . . . . . . . . . . . . . Synthetic datasets used to assess feature selection methods. Only the µ and σ 2 values of the first ten variables are shown. Highly correlated variables are denoted as Y k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Synthetic datasets used to assess normalization methods. Only the µ and σ 2 values of the first ten variables are shown. . . . . . . . . . . . . . . . . . xxi . Synthetic datasets used to assess imputation methods. Only the µ and σ 2 values of the first ten variables are shown. . . . . . . . . . . . . . . . . . . . . Synthetic datasets used to assess outlying preprocessing. Only the µ and σ 2 values of the first ten variables are shown. . . . . . . . . . . . . . . . . . . Contingency table obtained from the results of two classifiers. . . . . . . . . Extract of summary results from the study of data preprocessing procedures on regression, classification and clustering methods. FS (Feature Selection), N (Normalization), I (Imputation method), O (Outlier processing). . . . . . . List of variables that were used for the assessment of water quality of the Mexican rivers Tula, Humaya, Tamazual and Culiacan. . . . . . . . . . . . . . Summary of results about the linear model for the regression of distance between sampling sites and agricultural areas on the concentration of pesticides of the Mexican dataset. . . . . . . . . . . . . . . . . . . . . . . . . . . Results from the linear regression analysis on the Mexican dataset. Variables used for the analysis were chosen using the Linear correlation-based feature selection method. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results of the linear correlation-based feature selection on the data subsets macro, micro and bio. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Results of the linear regression on the macro, micro and bio data subsets before and after feature selection. . . . . . . . . . . . . . . . . . . . . . . . . Emplacement des sites d'échantillonnage de la rivière Tula . . . . . . . . . . . Emplacement des sites d'échantillonnage des rivières Tamazula, Humaya et Culiacan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure . -

 . Figure . -Basic chromatographic process. (a) The different components (C: glass column, SP: stationary phase, MP: mobile phase, S: sample); (b) introduction of the sample; (c) elution of constituents of the mixture; (d) recovering of compounds after separation. Figure taken from Rouessac and Rouessac ().

Figure . -

 . Figure . -Schematic representation of chromatographic instrumentation

(

  Courtemanch and Davies, ) p i = proportion of individuals of the ith taxonp i = n i /N . n i =total number of individuals of the ith taxon. N = total number of individuals for all ith taxa. S = total number of taxa. j = number of taxa in common between the stations A and B. a = number of all taxa in station A and b = number of all taxa in station B. A = Number of taxa at the reference site. B = Number of taxa at the study site. C = The taxa common to both sites.

Figure . -

 . Figure . -Structures of missingness: (a) univariate pattern, (b) monotone pattern, and (c) artibtrary pattern. Rows correspond to observations and columns to variables (Cottrell et al., )..

Figure . -

 . Figure . -Representation of (a) missing completely at random (MCAR), (b) missing at random (MAR), and (c) missing not at random (MNAR) in a univariate missing-data pattern. X represents completely observed variables, Y denotes partly missing variable, Z represents the component of the causes of missingness unrelated to X and Y, and R represents the missingness (Schafer and Graham, )..

  e., θ = (1/M ) M m=1 θ(m) ) by averaging the value of the parameter estimates across the M samples; • compute standard errors by (a) getting the average of the squared standard errors of the M estimates, (b) computing the variance of the M parameter estimates, and (c) include the uncertainty due to imputation by combining the two quantities.

  Contents. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Description of the study sites . . . . . . . . . . . . . . . . . . . . . Physico-chemical and chemical data acquisition . . . . . . . . . . . Sampling of water samples . . . . . . . . . . . . . . . . . . . . . . . Analysis of major elements, heavy metals, and arsenic in water samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Analysis of organochlorine pesticides and PPCPs in water samples . Hydrobiological data acquisition . . . . . . . . . . . . . . . . . . . . Sampling of macroinvertebrates . . . . . . . . . . . . . . . . . . . . . Analysis of macroinvertebrates . . . . . . . . . . . . . . . . . . .

Figure . -

 . Figure . -Sampling sites in the Tamazula, Humaya, and Culiacan rivers (Sinaloa state, Mexico).

Filtration

  Figure . -Flow diagram of the SPE method for analysis of OCPs in water.

Figure . -

 . Figure . -Example of macroinvertebrates counting process to determine the number of runs in a sample. Each form represents a different organism. The number of runs is used to compute the Seqential Compraison index (SCI).

Figure . -

 . Figure.-General procedure for assessment of data preprocessing procedures on statistical results.

-

  Step : Initialization of a temporal imputation using a simple imputation method (e.g., mean imputation); -Step : Sort each variable x according to the amount of missing values. Set I = { ,...,p}; -Step : Set l = ; -Step : Define X o l I/{l} and X m l I/{l} the matrices with the variables of the observed and missing cells of x l , respectively, where m l ⊂ {1, ..., n} denotes the indices of missing observations in variable x l and o l = {1, ..., n}/m l the indices of the observed cells of x l . The first column of X o l

  Step : Evaluate β with the model in Step . Replace the missing parts x m l l by x m l l = X m l k β; -Step : Perform Step -for each l= ,...,p; -Step : Repeat Steps -until imputed values stabilize, i.e. until i ) 2 < δ, f or all i m l and l I is the i-th imputed value of the current iteration, and x m l l,i is the i-th imputed value from the previous iteration.

  Figure.-Flow diagram of the generation of synthetic datasets for the assessment of preprocessing procedures on statistical results. FT denotes highly correlated data, AL denotes non-normalized data, NA stands for not available/missing values and OUT denotes outlying data.

Figure . -

 . Figure . -Flow diagram of the generation of semi-synthetic datasets for the assessment of preprocessing procedures on statistical results.

Figure . -

 . Figure . -Extract of the general workflow for assessment of data preprocessing procedures. Figure shows the workflow diagram of data preprocessing assessment.

Figure . -

 . Figure . -Experimental procedure for assessment of imputation methods.

Figure . -

 . Figure.-Experimental procedure for assessment of outlier detection and processing.

Figure . -

 . Figure . -NRMSE results of the imputation of missing values using the Hot-deck (hd), IRMI (ir), k-NN (kn) and Mice (mi) imputation methods on datasets N (A), N (B), N (C) and N (D).

Figure . -

 . Figure . -Precision and detection rate results of outliers detection methods. Adjusted quantile (adj.quan), Inter Quartile Range (iqr), Local Outlier Factor (lof) and Principal Components decomposition approach (pcout).

Figure . -

 . Figure.-Experimental procedure for assessment of the impact of data preprocessing procedures on statistical analysis results.

  Example. Classification results of two classifiers for observations:Classifier : b, b, c, a, c, c, c, a, a, b, c, b, b, a, c, a, b, c, c, a. Classifier : b, b, b, a, c, c, b, a, a, c, c, b, b, a, c, b, c, c, c, a.

  Figure .-Experimental procedure for assessment of the impact of data preprocessing procedures on results from regression analysis.

Figure . -

 . Figure . -RMSE results of the analysis of regression after feature selection processing on synthetic datasets. Linear correlation-based feature selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR).

  Figure .-RMSE results of the analysis of regression after feature selection processing on semi-synthetic datasets. Linear correlation-based feature selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR).

Figure . -

 . Figure . -RMSE results of the analysis of regression after normalization on synthetic datasets. Decimal scale noramlization (DS), Logistic sigmoidal normalization (SS) and Sigmoidal normalization using the hyperbolic tangent function (SM).

  Figure .-RMSE results of the analysis of regression after normalization on semi-synthetic datasets. Decimal scale normalization (DS), Logistic sigmoidal normalization (SS) and Sigmoidal normalization using the hyperbolic tangent function (SM).

  Figure .-Preprocessing errors of RMSE of the analysis of regression after imputation of missing values on synthetic dataset. Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI)

Figure . -

 . Figure . -Preprocessing errors of RMSE of the analysis of regression after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

  Figure .-Preprocessing errors of RMSE of the analysis of regression after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

Figure . -

 . Figure.-Experimental procedure for assessment of the impact of data preprocessing procedures on results from classification.

  . It shows a side-by-side comparison of Accuracy and Kappa results over the three classification methods and the synthetic and semi-synthetic datasets resulting from feature selection.

  Figures . and .

  shows a side-by-side comparison of the results of accuracy and Kappa values over the three classification methods and the four datasets resulting from normalization.

  Figure .-Accuracy and Kappa results of the analysis of classification after feature selection on synthetic datasets. Linear Correlation-based Feature Selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR).

  Figure .-Accuracy and Kappa results of the analysis of classification after normalization on synthetic datasets. Min-Max normalization (MM), Decimal scale normalizaton (DS) and Z-score normalization (ZS).

  Figure .-Accuracy and Kappa preprocessing error from the analysis of classification after imputation of missing data on synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

  Figure .-Preprocessing errors of Accuracy and Kappa of the analysis of classification after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

  Figure .-Preprocessing errors of Accuracy and Kappa of the analysis of classification after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

  Figure .-Experimental procedure for assessment of the impact of data preprocessing procedures on results from clustering.

  Figure . -Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after feature selection processing on synthetic datasets. Feature selection methods: Correlation-based Feature Selection (HC), Linear Correlation-based Feature Selection (FI) and Wrapper Subset Evaluator (WR).

  Figure . -Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after feature selection processing on semi-synthetic datasets. Feature selection methods: Correlation-based Feature Selection (HC), Linear Correlation-based Feature Selection (FI) and Wrapper Subset Evaluator (WR).

  Figure . -Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after imputation of missing data on synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI).

  Figure .

  Figure .-Results of Naïves Bayesian classification on datasets preprocessed by: (A) normalization followed by imputation of missing values and (B) imputation of missing values followed by normalization. Min-max normalization (MM).Imputation methods: Hot-deck (HD), IRMI (IR), K-NN (KN) and MICE (MI).

Figure . -

 . Figure . -Screenshot of EvDA application. Display of the Data input tab.

Figure . -

 . Figure . -Screenshot of EvDA application. Display of the Data inspection tab.

Figure . -

 . Figure . -Screenshot of EvDA application. Display of the Data Preprocessing tab.

Figure . -

 . Figure . -Screenshot of EvDA application. Example of display of the analysis by K-means clustering.

Figure . -

 . Figure . -Google Earth image of the sampling sites C to C situated on the rivers Tamazula, Humaya and Culiacan. Distance between agricultural areas and the sampling sites are defined by a red line, sampling sites are highligted by a yellow icon. Source: "Culiacan." ° '00"N ° '00"O. Google Earth ©. August , . November , .

Figure . -

 . Figure . -Concentrations of Pesticides on sites C to C situated on the rivers Tamazula, Humaya, and Culiacan.

Figure . -

 . Figure . -Lower level correlation matrix of the Mexican dataset. Name of variables are abbreviated, see Table .for details.

Figure . -

 . Figure . -Lower level correlation matrix of the Mexican dataset. Name of variables are abbreviated, see Table .for details..

Figure . -

 . Figure . -Sites de prélèvement dans la rivière Tula (État d'Hidalgo, Mexique).

  de régression, classification et groupement. Chaque jeu suit une loi normale et est composé d'un nombre différent d'observations (i.e., n = , n = , n = , n = ), variables numériques (i.e., p = , p = , p = , p = ) et une variable catégorielle de cinq classes suivant une distribution uniforme. Les variables incluent des variables non pertinentes. Les tests pour évaluer l'impact de pré-traitement par normalisation des données ont été exécutes en utilisant quatre jeux de données synthétiques composés d'un nombre différent d'observations (i.e., n = , n = , n = , n = ), variables numériques (i.e., p = , p = , p = , p = ) et, une variable catégorielle de cinq classes. Chaque jeu de données suit une distribution Weibull. L'évaluation des méthodes d'imputation des valeurs manquantes a été effectué en utilisant quatre jeu de données synthétiques. Chaque jeu a été composé d'un nombre différent d'observations (i.e., n = , n =

Figure . -

 . Figure . -Procédure générale d'évaluation des procédures de pré-traitement des données sur les résultats statistiques.

Figure . -

 . Figure . -Capture d'écran de l'application EvDA. Affichage de l'onglet inspection de données.

  To calculate the BMWP, all the individual scores of all families present in a sample are summed. High BMWP values are characteristic of clean sites, while low BMWP values are typical of polluted sites (c.f. TableA. ).

  me manqué énormément quand vous êtes pas là. A l'équipe portugais Claudio, Alexandre, Eudes, Leandro et Savio, vous m'avez donné encore plus d'envie pour apprendre le portugais. Les vrais caipiriña c'est que avec vous. A Christian et Hatim pour leur bonne humeur, et les pose café.

  assessment of the impact of data preprocessing procedures on statistical analysis results. . . . . . . . . . . . . . . . . . . . . . Experimental procedure for assessment of the impact of data preprocessing procedures on results from regression analysis. . . . . . . . . . . . . . . . . . Preprocessing errors of RMSE of the analysis of regression after imputation of missing values on synthetic dataset. Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI) . . . . . . . . . . . . . . . . . . . . . . Preprocessing errors of RMSE of the analysis of regression after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . Experimental procedure for assessment of the impact of data preprocessing procedures on results from classification. . . . . . . . . . . . . . . . . . . . . . Accuracy and Kappa results of the analysis of classification after feature selection on synthetic datasets. Linear Correlation-based Feature Selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Accuracy and Kappa results of the analysis of classification after feature selection on semi-synthetic datasets. Linear Correlation-based Feature Selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after feature selection processing on synthetic datasets. Feature selection methods: Correlation-based Feature Selection (HC), Linear Correlation-based Feature Selection (FI) and Wrapper Subset Evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after feature selection processing on semi-synthetic datasets. Feature selection methods: Correlation-based Feature Selection (HC), Linear Correlation-based Feature Selection (FI) and Wrapper Subset Evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after imputation of missing data on synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Scientific workflow of EvDA . . . . . . . . . . . . . . . . . . . . . . . . . . . . Screenshot of EvDA application. Display of the Data input tab. . . . . . . . Google Earth image of the sampling sites C to C situated on the rivers Tamazula, Humaya and Culiacan. Distance between agricultural areas and the sampling sites are defined by a red line, sampling sites are highligted by a yellow icon. Concentrations of Pesticides on sites C to C situated on the rivers Tamazula, Humaya, and Culiacan. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Lower level correlation matrix of the Mexican dataset. Name of variables are abbreviated, see Table . for details. . . . . . . . . . . . . . . . . . . . . PCA biplot of the two principal components for the macro-pollutants (a), micro-pollutants (b), and metals (c). Biomonitoring metrics are coloured in blue. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Sites de prélèvement dans la rivière Tula (État d'Hidalgo, Mexique). . . . . Water quality variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . List of some Emerging Pollutants that are frequently found in aquatic environment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	.	B), N Precision and detection rate results of outliers detection methods. Adjusted (C) and N (D). . . . . . . . . . . . . . . . . . quantile (adj.quan), Inter Quartile Range (iqr), Local Outlier Factor (lof) and Principal Components decomposition approach (pcout). . . . . . . . . . Experimental procedure for . . . Preprocessing errors of RMSE of the analysis of regression after imputation of missing values on semi-synthetic datasets. Hot-deck (HD), Iterative Step-Preprocessing errors of Accuracy and Kappa of the analysis of classification after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iter-ative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . Results of Naïves Bayesian classification on datasets preprocessed by: (A) normalization followed by imputation of missing values and (B) imputa-tion of missing values followed by normalization. Min-max normalization (MM).Imputation methods: Hot-deck (HD), IRMI (IR), K-NN (KN) and ° '00"N ° '00"O. Google MICE (Source: "Culiacan." Earth ©. August , . November , . . . . . . . . . . . . . . . . .
	.	wise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Preprocessing errors of Accuracy and Kappa of the analysis of classification Imputation by Chained Equations (MI) . . . . . . . . . . . . . . . . . . . . after outlier detection followed by imputation of outlying data on semi-
		. synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Preprocessing errors of RMSE of the analysis of regression after outlier de-

. RMSE results of the analysis of regression after feature selection processing on synthetic datasets. Linear correlation-based feature selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RMSE results of the analysis of regression after feature selection processing on semi-synthetic datasets. Linear correlation-based feature selection (FI), Correlation-based Feature selection (HC) and Wrapper Subset evaluator (WR). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RMSE results of the analysis of regression after normalization on synthetic datasets. Decimal scale noramlization (DS), Logistic sigmoidal normalization (SS) and Sigmoidal normalization using the hyperbolic tangent function (SM). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RMSE results of the analysis of regression after normalization on semisynthetic datasets. Decimal scale normalization (DS), Logistic sigmoidal normalization (SS) and Sigmoidal normalization using the hyperbolic tangent function (SM). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . tection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . xvi . . Accuracy and Kappa results of the analysis of classification after normalization on synthetic datasets. Min-Max normalization (MM), Decimal scale normalizaton (DS) and Z-score normalization (ZS). . . . . . . . . . . . . . . . Accuracy and Kappa results of the analysis of classification after normalization on semi-synthetic datasets. Min-Max normalization (MM), Decimal scale normalizaton (DS) and Z-score normalization (ZS). . . . . . . . . . . . . Accuracy and Kappa preprocessing error from the analysis of classification after imputation of missing data on synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Accuracy and Kappa preprocessing error from the analysis of classification after imputation of missing data on semi-synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Quantile ( ), PCOUT ( ) and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . Experimental procedure for assessment of the impact of data preprocessing procedures on results from clustering. . . . . . . . . . . . . . . . . . . . . . xvii . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after imputation of missing data on semi-synthetic datasets. Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after outlier detection followed by imputation of outlying data on synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ), and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Rand index, Adjusted Rand index, Jaccard index and Cophenetic coefficient of Clustering methods after outlier detection followed by imputation of outlying data on semi-synthetic datasets. Detection methods: Inter Quartile Range ( ), Adjusted Quantile ( ), PCOUT ( ), and LOF ( ). Imputation methods: Hot-deck (HD), Iterative Stepwise Regression Imputation (IR), K-Nearest Neighbour (KN) and Multiple Imputation by Chained Equations (MI). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Screenshot of EvDA application. Display of the Data inspection tab. . . . . . Screenshot of EvDA application. Display of the Data Preprocessing tab. . . . Screenshot of EvDA application. Example of display of the analysis by K-means clustering. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviii . . . Sites de prélèvement des rivières Tamazula, Humaya et Culiacn (État de Sinaloa, Mexique). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Procédure générale d'évaluation des procédures de pré-traitement des données sur les résultats statistiques. . . . . . . . . . . . . . . . . . . . . . . . . . Capture d'écran de l'application EvDA. Affichage de l'onglet inspection de données. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix xx List of Tables . . Measures of richness (Barbour et al., ) (Resh and Jackson, ). . . . . Enumeration metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

  Number of Systematic Units observed of the taxonomic group. . . . . . . . . . . . . A. Benthic macroinvertebrates clased according to Beck's biotic Index (BBI) Classes (Beck, ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . FQ semi-synthetic dataset (n = 7520, p = 26). The µ and σ 2 values of all variables are shown. . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

		IT	Information Technology
		LC-MS Liquid Chromatography-Mass Spectrometry
		Nomenclature LC-MS-MS Liquid Chromatography-tandem Mass Spectrometry
		LC-PAD Liquid Chromatography with a Photodiode Array Detector
		MAR Missing at Random
	Acronyms MCAR Missing Completely at Random
	AAS Atomic Absorption Spectroscopy MICE Multiple Imputation by Chained Equations
	ANN Artificial Neural Networks MNAR Missing Not at Random
	BBI Beck Biotic Index OCPs Organochlorine Pesticides
	BMWP Biological Monitoring Working Party PBDEs PolyBrominated DiphenylEthers
	BMWP-ASPT BMWP -Average Score Per Taxon
	BOD Biochemical Oxygen Demand
	COD Chemical Oxygen Demand
	DM Data Management
	DO Dissolved Oxygen
	EBI Extended Biotic Index
	EDTA Ethylenediaminetetraacetic acid
	EI	Environmental Informatics
	EMB Expectation-Maximization with bootstrapping
	EP	Emerging Pollutant WQ Water Quality
	EPT Ephemeroptera Plecoptera Trichoptera WQA Water Quality Assessment
	FFG Functional Feeding Groups WQMP Water Quality Monitoring Program
	GC-ECD Gas Chromatography-Electron Capture Detector Symbols
	A. Evaluation of water quality using the Family Biotic Index (FBI) and the tolerance values for families of stream arthropods (Hilsenhoff, ). . . . GC-MS Gas Chromatography-Mass Spectrometry B Some definition of B
	A. Biomonitoring Working Party Score System (BMWP) (National Water Coun-HBI Hilsenhoff's Biotic Index
	cil; HPLC High Performance Liquid Chromatography ) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
	A. Biological Monitoring Working Party (BMWP) and Average Score Per Taxon (ASPT) scores and their related quality index (Armitage et al., ; Friedrich IBI Index of Biotic Integrity
			et al.,	; National Water Council,
			xxiv

Table taken from Metcalfe, ). . . . . . . . A. Table to calculate Extended Biotic Index (EBI) values and conversion table to transform EBI values into Quality Classes (Ghetti, ). SU: ). . . . . . . . . . . . . . . . . . . A. Functional Feeding Groups (FFG): Categorization and food resources. Coarse Particulate Organic Matter (CPOM); Fine Particulate Organic Matter (FPOM) (Merrit et al., ). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxii B. IBI-Mex Index of Biotic Integrity for west-central Mexico ICP-MS Inductively Coupled Plasma Mass Spectrometry IRD Institut de Recherche pour le Développement xxv IRMI Iterative Robust Model-based Imputation PPCPs Pharmaceutical and Personal Care Products SCI Sequential Comparison Index SEA Strategic Environmental Assessment SFC-DAD Supercritical-Fluid Chromatography -Diode-Array Detections SPE Solid Phase Extraction SPME-GC-MS Solid Phase Micro Extraction-Gas Chromatography-Mass Spectrometry TBI Trent Biotic Index UNAM Universidad Nacional Autónoma de México WHO World Health Organization

Table . -

 . List of some Emerging Pollutants that are frequently found in aquatic environment.

	Industrials	Pesticides	PPCPs
	antioxidants	carbamates	analgesics
	perfluorates	chloro-acetanilides	anti-epileptic drugs (AEDs)
	phenols	chlorophenoxy acids anticonvulsants
	phthalates	organochlorines	lipid regulators
	poly-brominated diphenylethers (PBDEs) organophosphates	antimicrobials
	triazoles	pyrethroids	polycyclic musks (PCMs)
		triazines	non-steroidal anti-inflammatory drugs (NSAIDs)
		diuron	synthetic hormones
		isoproturon	fragances
		mecoprop	insect repellant
		prometon
	Units are given in µg L -1 .

•

  Ion exchange. Water is composed of electrically charged atoms or molecules named ions. Ions can be charged positively or negatively, named cations or anions respectively. Ion exchange is used to separate undesired cations and anions within a water matrix such undesired cations are replaced with other similarly charged ions.

Table . -

 . Enumeration metrics.

	Metric name	Common expression
	Ratio of EPT individuals to total individuals	% EPT
	Ratio of Chironomidae individuals to total number of individuals	% Chironomidae
	Ratio of EPT abundance to Chironomidae abundance	EPT:Chironomidae
	Ratio of individuals in numerically dominant family to total number of	% of most dominant genera
	individuals
	Ratio of individuals in numerically dominant taxa to total number of	% of dominant taxa
	individuals

Table . -

 . Similarity and Diversity indices.

		Index name	Definition	Reference
		Shannon's	H =	i=S i=1 p i log p i	(Shannon,	)
	Diversity indices	Index Simpson's Index D =	i=S i=1 N (N -1) ni(ni-1)	(Simpson,	)
		Sequential	SCI = number of runs number of specimens
		Comparison
		Index (SCI)

Table . -

 . Biotic indices.

	Index name	Common expression Reference
	Trent biotic Index	TBI	(Woodiwiss,	)
	Extended Biotic Index	EBI	(Ghetti,	)(Hellawell,	)
	Beck Biotic Index	BBI	(Beck,	)
	Hilsenhoff's Biotic Index	HBI	(Hilsenhoff,		)
	Biological Monitoring Working Party	BMWP	(Armitage et al.,	)
	Score System
	BMWP -Average Score Per Taxon	BMWP-ASPT	(Armitage et al.,	)
	Each biotic index contains a table listing the tolerance values assigned for each taxa. Computation of
	the biotic indexes is done using the corresponding tables for each metric.

  Shrinkage methods constrains the coefficient estimates towards zero. By shrinking the coefficient estimates the variance of the model is reduced. The two best-known techniques for shrinking the regression coefficients are ridge regression and the lasso. The fitting procedure in least squares estimates β 0 , β 1 , βp uses values that minimize

	n
	RSS =
	i=1

2

. Contrary to C p , AIC and BIC where small values indicate a model with low test error, large values of adjusted R 2 indicate a model with a low test error. The above mentioned methods involve the use of least squares linear models that contains a subset of predictor variables. As an alternative shrinkage methods can be used to fit a model that contains all p predictors.

• Shrinkage.

Table . -

 . Location of sampling sites of the Tula river. Mexico in the state of Sinaloa, Mexico. Its a semiarid region with seasonal rainfall from June to September (mean annual precipitation mm) and dryness during the rest of the year. This region has a mean annual temperature of . -; however, currently its discharge rate has diminished due, mainly, to its use to irrigate agricultural fields.Water from the Culiacan river is used for agricultural, industrial, domestic and livestock activities. Though, irrigation of agricultural fields is the main use. The Culiacan valley is the biggest and the most important agricultural zone of Mexico, it comprises more

										°C and mean
	annual evapotranspiration of	mm (INEGI,	).
	Water flow of Tamazula River starts at the South-East of Durango state and ends at
	the Culiacan city, it has a length of		km and a drainage basin of ,	km .
	The Humaya River is considered as the main watercourse forming the Culiacan river. It
	has a total length of	km and a drainage basin of ,		km .
	The Culiacan River begins at the Culiacan city after the union of the rivers Tamazula
	and Humaya. It runs	. km until it reaches the Gulf of California and on its way, it
	increases its volume by groundwater recharge (INEGI,		). The Culiacan river is the
	most important perennial river in the region it has a drainage basin of ,	km , it
	receives the treated water from the sewage of Culiacan city and surrounding communities
	(approximately	,	inhabitants) at a rate of	,		m day -(INEGI,	). Until
	, the discharge rate of the Culiacan river used to be	. m s
	Name of sam-	Municipality Latitude		Longitude	Altitude (m) Focus of the
	pling site									study
	H	Tasquillo	°	.	'	° .	'	.	domestic waste
										pollution
	H	Ixmiquilpan	° .	'	° .	'	.	pesticides from
	H	Ixmiquilpan	° .	'	° .	'	.	agriculture
	H	Tlacotlapilco	° .		° . '	.	domestic waste
	H	Progreso		° .	'	° .	'	.	pollution

Figure . -Sampling sites in the Tula river (Hidalgo state, Mexico) The rivers Tamazula, Humaya, and Culiacan are located in the Pacific coast plain Chapter province of than Km of agricultural fields which represents % of Sinaloa state. The Culiacan agricultural area has ≈ , ha with % of the land irrigated (INEGI, ).

Table . -

 . Location of sampling sites of Tamazula, Humaya and Culiacan rivers.

	Name	of	Municipality	Latitude			Longitude		Altitude (m) Focus of the
	Sampling									study
	site
	Humaya river
	C		Mojolo	° .	'	° .	'	.
	C		Mojolo	° .		'	° .	'	.	domestic waste
	C		Mojolo	° .		'	° .	'	.	pollution
	C		Humaya	° .		'	° .	'	.
	Tamazula river
	C C C		Culiacan Tamazula La Limita de Itaje	° . ° . ° .	'	' '	° . ° . ° .	' ' '	. . .	least polluted state
	Culiacan river
	C		Culiacan	° .	'	° .	'	.	anthropogenic
	C		Culiacan	° .	'	° .	'	.	activities
	C		Bacurimi	° .		'	°	.	'	.	pesticides from
	C		Culiacancito	° .		'	° .	'	.	agriculture

Table . -

 . Example of data from the physochemical and chemical values obtained from the analysis of water samples of the rivers Tula. Tamazula. Humaya and Culiacan.

	Sample Season pH		Ω µS/cm	CO		HCO		SO	Cl		F
	H	.	± .	±	.	± .	.	± .	. ± .		± .	. ± .
	H	. ± .	±	-		.	±	. ± .	. ±	.	± .
	H	. ±	±	. ±	. ± .	. ± .	.	± .	.	± .
	H	. ± .	. ± .	.	± .	.	± .	. ± .	. ± .	.	± .
	H	. ± .	. ± .	. ±	.	± .	.	± .	. ±	.	± .

ND = Non detected. Concentrations are given in mg/L. Each sample was analyzed by triplicates for each parameter.

Table . -

 . Example of the data from the concentration of arsenic and heavy metals on water samples of the rivers Tula, Tamazula, Humaya and Culiacan.

	Sample Saison Cd		As		Fe		Cu	Mn	Zn	Pb
	H	Dry	.	± .	.	± .	. ± .	ND	. ±	ND	.	± .
	H		.	± .	.	± .	.	±	ND	. ±	ND	.	±
	H		.	±	.	± .	ND		ND	. ±	ND	.	± .
	H		.	± .	.	± .	. ±	ND	. ± .	ND	.	± .
	H		.	± .	.	± .	ND		ND	. ±	ND	.	± .

ND = Non detected. Concentrations are given in mg/L. Each sample was analyzed by triplicates for each element.

Table . -

 . SPE-GC-ECD limit of detection (LOD), limit of quantification (LOQ) and recovery values for water analysis.

	1 I-BHC	9.01 0.15	0.43	1.54	109.07 ± 0.55
	2 II-BHC	9.86 0.30	0.82	2.96	143.42 ± 0.40
	3 III-BHC	10.02 0.12	0.32	1.16	128.18 ± 0.49
	4 IV-BHC	10.85 0.17	0.48	1.73	177.67 ± 0.22
	5 Heptachlor	12.22 0.71	1.97	7.08	48.53 ± 0.63
	6 Aldrin	13.34 0.34	0.95	3.42	24.93 ± 0.69
	7 Heptachlor epoxide	14.64 0.85	2.36	8.48	64.80 ± 0.38
	8 I-Endosulfan	15.83 1.18	3.27	11.80	62.74 ± 0.41
	9 Dieldrin	16.75 0.54	1.51	5.44	17.70 ± 0.67
	10 DDE	16.86 0.54	1.51	5.44	17.70 ± 0.67
	11 Endrin	16.86 0.76	2.12	7.63	50.38 ± 0.40
	12 II-Endosulfan	17.68 0.70	1.95	7.02	89.66 ± 0.48
	13 DDD	18.01 0.01	0.03	0.11	79.38 ± 0.42
	14 Endrin aldehyde	18.23 1.46	4.04	14.56	35.27 ± 0.47
	15 Endosulfan sulfate	18.56 0.22	0.61	2.20	120.75 ± 0.29
	16 DDT	19.18 0.52	1.46	5.24	105.47 ± 0.61
	17 Endrin ketone	20.07 0.10	0.28	1.01	132.72 ± 0.39
	18 Methoxychlor	20.24 0.74	2.05	7.37	91.82 ± 0.90
	Spiked level of water 500 ng L -1 .
	Replicates n = 3.
	RT=Retention time.
	SD = Standard deviation.
	RSD = Residual standard deviation.

No. Pesticide RT(min) SD LOD (ng L -1 ) LOQ (ng L -1 ) Recovery (% ± RSD)

Table . -

 . SPME-GC-MS limit of detection (LOD), limit of quantification (LOQ) and recovery values for water analysis.

	1 Ibuprofen	0.4	1.1	94.1 ± 3.4	90.6 ± 3.0
	2 2-benzyl-4-chlorophenol	0.4	1.1	91.2 ± 2.3	96.1 ± 2.4
	3 Naproxen	0.7	2.0	77.4 ± 2.1	88.9 ± 1.7
	4 Triclosan	0.5	1.6	90.6 ± 3.7	97.2 ± 3.8
	5 Ketoprofec	0.4	1.2	85.6 ± 4.1	96.2 ± 8.7
	6 Diclofenac	4.2	12.8	57.5 ± 0.4	54.4 ± 2.5
	7 Bisphenol A	1.4	4.1	93.2 ± 1.4	84.0 ± 3.5
	8 Estrone	1.5	4.6	67.5 ± 1.1	76.0 ± 3.6
	Spiked level of water 20 ng L -1 .
	Spiked level of water 90 ng L -1 .
	Replicates n = 3.
	RSD = Residual standard deviation.

No. Pesticide LOD (ng L -1 ) LOQ (ng L -1 ) Recovery (% ± RSD) Recovery (% ± RSD)

  Solid Phase Micro Extraction (SPME) was performed for the extraction of PPCPs in water samples, SPME was carried out using µm poliacrylate fibers (PA) supported on a manual device for SPME (Bellefonte, PA, USA). PA fibers were previously conditioned

			Chapter
		SPE
		Conditioning
	SPE Cartridges: Strata C-18 (200 mg/6 mL)	1 x 5mL Dichlorometane 1 x 5mL Methanol 1 x 5mL MiliQ water	Flow rate: 1 mL/min
		Sample loading
		500 mL of water	Flow rate: 12 -15 mL/min
		Water remotion
		Passing air for 15 min
		Passing nitrogen for 15 min
		Sample elution
		2 x 1mL Dichloromethane	Flow rate: 1 mL/min
		Adding Sodium sulfate and
		washing with dichloromethane

Table . -

 . Example of data from the analysis of PPCPs on water samples from the Tula river.

	Sample Season Ibuprofen Naproxen	Triclosan Diclofenac Bisphenol A
	H	dry	± .	. ± .	. ± .	-	-
	H		. ± .	. ± .	. ± .	-	-
	H		. ± .	. ± .	. ± .	-	-
	H		. ± .	± .	. ± .	-	-
	H		. ± .	. ± .	. ± .	-	-

ND = Non detected. Concentrations are given in ng/L. Each sample was analysed by triplicates for each compound.

Table . -

 . Biomonitoring metrics for the biological assessment of Mexican rivers(Serrano Balderas et al., ).

	Metric	Taxonomic	Definition	Expected re-	Reference
			resolution		sponse to in-
					creasing pertur-
					bation
	Measures of	Number of total	Family	All different taxa at a site.	Decrease	Resh
	richness	taxa				and
						Jackson,
						;Bar-
						bour
						et al.,
		Number of EPT		Total number of taxa of the or-
		taxa		ders Ephemeroptera (mayflies),
				Plecoptera (stoneflies) and Tri-
				choptera (caddisflies).
		Number of	Genus or	Total number of taxa of the order
		Ephemeroptera	Species	Ephemeroptera (mayflies).
		taxa
		Number of Ple-		Total number of taxa of the order
		coptera taxa		Plecoptera (stoneflies).
		Number of Tri-		Total number of taxa of the order
		choptera taxa		of Trichoptera (caddisflies).
	Enumerations	Number of fami-	Family	Number of families in common
		lies in common		between samples.
		%EPT		Ratio of EPT abundance.
		%Ephemeroptera		Ratio of mayflies to total number
				of individuals.
		%Plecoptera		Ratio of stoneflies to total number
				of individuals.
		%Trichoptera		Ratio of caddisflies to total num-
				ber of individuals.
		%Coleoptera		Ratio of individuals of the order
				of Coleoptera to total number of
				individuals.
		%Diptera		Ratio of individuals of the order	Increase
				of Diptera to total number of
				individuals.
		%Chironomidae		Ratio of chironomidae individuals
				to total number of individuals.
		EPT / Chirono-		Ratio of EPT abundance to chi-	Decrease
		midae		ronomidae abundance.
		% of most domi-	Genus	Ratio of individuals in numerically	Increase
		nant genus		dominant genus to total numbers
				of individuals.
		% of dominant	Family	Ratio of individuals in numerically
		taxa		dominant taxa to total number of
				individuals.
	Diversity and	Shannon's Index		Description of community struc-	Decrease	Shannon,
	Similarity indices			ture (Diversity).
		Simpson's Index		Diversity index commonly used in	Increase	Simpson,
				Ecology and Biology.
		Margalef Index		Diversity index, where the number	Decrease	Margalef,
				of species in the sample and the
				total number of organisms in the
				sampling are considered.
		Sequential Com-	Order or	Description of stream quality		Cairns
		parison Index	Family	method, based upon distinguishing		et al.,
		(SCI)		the number of different types
				of organisms and the number of
				"runs".
		Jaccard's Coeffi-	Family	Description of the similarity be-		Jaccard,
		cient		tween two samples.
		Sørensen Coeffi-		Description of the similarity be-		Sørensen,
		cient		tween two samples.
	Biotic indices	Trent Biotic	Family	Description of the pollution level		Woodiwiss,
		Index (TBI)	(genus	of streams, according to the sensi-		;
			for Ple-	tivity of key groups to pollution.		Met-
			coptera and			calfe,
			Ephemeroptera
			nymphs and
			species-level
			for Annelida,
			Mollusca,
			Crustacea
			and
			Megaloptera)
		Extended Biotic
		Index (EBI)

Table . -

 . Biomonitoring metrics for the biological assessment of Mexican rivers(Serrano Balderas et al., ). (continued...)

	Metric	Taxonomic	Definition	Expected re-	Reference
			resolution		sponse to in-
					creasing pertur-
					bation
		Biological Mon-		Score system using benthic	Decrease	ISO,
		itoring Working		macroinvertebrate classed ac-		;
		Party (BMWP)		cording to their pollution tolerance		National
				for Britain rivers.		Water
						Council,
		Biological Mon-		Score system based on the BMWP		Gutiérrez-
		itoring Working		modified to include macroinverte-		Fonseca
		Party Costa Rica		brate communities of Costa Rica		and
		(BMWP CR)				Lorion,
		Biological Mon-		Average Score Per Taxon, added to		Armitage
		itoring Working		the BMWP Score System.		et al.,
		Party Average
		Score per Taxon
		(BMWP ASPT)
	Functional Feed-	%Filterer collec-		Percent of the macrobenthos that	Increase	Merrit
	ing measures	tors		filter fine organic material.		et al.,
						b
		% Scrapers		Percent of the macrobenthos that	Decrease
				feed on epiphytes.
		%Shredder		Percent of the macrobenthos that
				feed on leaf litter.
		%Predators		Percent of the predator func-
				tional feeding group. They are
				carnivores-scavangers, engulf or
				pierce prey.
		% gathering		Percent of the macrobenthos that	Variable
		collectors		collect fine deposited organic
				material.
	Multimetric	IBI-west central		Computation of multimetric ap-	Decrease	Weigel
	approach	Mexico		proaches which is summarized in a		et al.,
				single value.

Table . -

 . Example of data from the biomonitoring metrics computed for the rivers Tula, Tamazula, Humaya and Culican.

						CHAPTER
				Metric		Sampling sites
				H	H	H	H	H
	Measures	of	richness	Number of total taxa Number of Plecoptera taxa Number of EPT taxa Number of Ephemeroptera taxa Preprocessing and Analysis of
				Number of Trichoptera taxa Environmental Data
	Enumerations

Contents

. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Data preprocessing procedures . . . . . . . . . . . . . . . . . . . . . Feature selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Normalization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Imputation methods . . . . . . . . . . . . . . . . . . . . . . . . . . . Outlier detection methods . . . . . . . . . . . . . . . . . . . . . . . Synthetic data description . . . . . . . . . . . . . . . . . . . . . . . Semi-synthetic data description . . . . . . . . . . . . . . . . . .

. . Robustness study of data preprocessing procedures . . . . . . .

  Results and discussion on robustness of data preprocessing procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

	. .	Impact on classification results . . . . . . . . . . . . . . . . . . .
	. .	Impact on clustering results . . . . . . . . . . . . . . . . . . . . .
	. .	Results and discussion about preprocessing procedures on statis-
		tical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . Selection of features . . . . . . . . . . . . . . . . . . . . . . . . . . . Normalization of data . . . . . . . . . . . . . . . . . . . . . . . . . . Handling missing data . . . . . . . . . . . . . . . . . . . . . . . . . . Handling outlying data . . . . . . . . . . . . . . . . . . . . . . . . . . Study of the impact of data preprocessing procedures on statistical results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Impact on regression results . . . . . . . . . . . . . . . . . . . . . . Summary and concluding remarks . . . . . . . . . . . . . . . . .

  Results obtained from regression analysis on datasets processed by feature selection are represented in Figures. and .. They present the RMSE results on a side by side comparison between different feature selection methods, respective of the regression algorithms.

	• For feature selection
	-three feature selection methods; Correlation-based Feature Selection (CFS),
	Linear Correlation-based Feature Selection, and Wrapper Subset Evaluator.
	• For normalization preprocessing
	-three normalization methods; Decimal scale (DS), Sigmoidal normalization us-
	ing logistic sigmoid function (SS) and Sigmoidal normalization using the hy-
	perbolic tangent funciton (SM);
	• For missing values preprocessing
	-six amounts of missing values ( %, %, %, %, % and %)
	-four imputation methods; Hot-Deck (HD), K-Nearest Neighbour (KN), It-
	erative Stepwise Regression Imputation (IRMI) and Multiple Imputation by
	Chained Equation (MICE)
	• For outliers preprocessing
	-five amounts of outlying data ( . %, . %, %, % and %);
	-four outlier detection methods; ( ) Inter Quartile Range (IQR), ( ) Adjusted
	Quantile (AQ), ( ) Principal Components decomposition for multivariate out-
	lier detection approach (PCOUT) and ( ) Local Outlier Factor (LOF);
	-subsequent outlier imputation by four imputation methods; Hot-Deck (HD),

K-Nearest Neighbour (KN), Iterative Stepwise Regression Imputation (IRMI) and Multiple Imputation by Chained Equations (MICE) Results for feature selection

Table . -

 . List of variables that were used for the assessment of water quality of the Mexican rivers Tula, Humaya, Tamazual and Culiacan.

			Variables
	Macro-pollutants	Micro-pollutants		Biological indices
	General	Heavy met-	Organo-		PPCPs
	chemistry	als	chlorine
			pesticides
	pH	As	I-BHC		Ibuprofen	Number of total taxa
	conductivity	Cd	II-BHC		Naproxen	Number of EPT taxa
	CO3	Fe	III-BHC		Triclosan	Number of Ephemeroptera taxa
	HCO3	Mn	IV-BHC		Diclofenac	Number of Plecoptera taxa
	SO4	Pb	Heptachlor	Bisphenol A Number of Trichoptera taxa
	Cl	Zn	Aldrin			Number of families in common
	F		Heptachlor		% EPT
			epoxide
	Na		I-Endosulfan		% Ephemeroptera
	K		Dieldrin			% Plecoptera
	Ca		DDE			% Trichoptera
	Mg		Endrin			% Coleoptera
	B		II-			% Diptera
			Endosulfan
	SiO2		DDD			% Chironomidae
	NO3		Endrin alde-		EPT/ Chironomidae
			hyde
			DDT			% of most dominant genus
			Endrin	ke-		% of dominant taxa
			tone
			Methoxychlor		Shannon's Index
						Simpson's Index
						Margalef Index
						Sequantial Comparison Index
						Jaccard's coefficient
						Sørensen coefficient
						Trent Biotic index
						Extended Biotic index
						Beck Biotic index
						Family Biotic index
						BMWP
						BMWP-CR
						BMWP-ASPT
						% Filterer collectors
						% Scrapers
						% Shredder
						% Predators
						% gathering collectors
						% IBI-west central Mexico
	BMWP: Biological Monitoring Working Party.
	BMWP-CR: Biological Monitoring Working Party-Costa Rica.
	BMWP-ASPT: Biological Monitoring Working Party-Average Score per Taxon.

Table . -

 . Summary of results about the linear model for the regression of distance between sampling sites and agricultural areas on the concentration of pesticides of the Mexican dataset.

	Quantity	Value
	Residual standard error
	R 2	.
	F-statistic	.

Table . -

 . Results from the linear regression analysis on the Mexican dataset. Variables used for the analysis were chosen using the Linear correlation-based feature selection method.

		Coefficient	Std. error	t-statistics p-value
	Intercept	.		.	.	.
	I_BHC	-	.	.	-.	.
	I_Endosulfan	-	.	.	-.	.
	Dieldrin		.	.	.	.
	Endrin	-	.	.	-.	.
	Endrin_aldehyde		.	.	.	.
	Endosulfan_sulfate -	.	.	-.	.
	Endrin_ketone	-	.	.	-.	.
	Methoxychlor		.	.	.	.

Table . -

 . Results of the linear correlation-based feature selection on the data subsets macro, micro and bio.

	Name of subset Number of original	Number of selected	Name of selected vari-
	variables	variables	ables
	macro		pH, conductivity, SO ,
			Cl, Na, K, Ca, SiO
	micro		II-BHC,	I-Endosulfan,
			Dieldrin,	DDE, En-
			drin,	Endrin-aldehyde,
			Endosulfan-sulfate,
			Endrin-ketone, Methoxy-
			chlor,	Ibuprofen,
			Bisphenol-A
	bio		percentage of diptera,
			EPT-Chironomidae,
			Shannon's index, Simp-
			son's index, Margalef,
			SCI, Sørensen index,
			TBI, BMWP

  It must be noted that a linear model will perform well on test observations if n, the number of observations, is much larger than p, the number of variables. However, a large variability of the model exists if n is not much larger than p, which results in over-fitting thus on poor predictions. It is frequently observed that some or many of the variables used in a model are not associated with the response. Such irrelevant variables lead to complex resulting models. A model that can be easily interpreted can be obtained by removing irrelevant variables. Some of the approaches that include feature selection or variable selection include stepwise model selection procedures (e.g., Forward Stepwise selection, Backward Stepwise selection), shrinkage methods (e.g., Ridge regression, LASSO), and dimension reduction methods (e.g., Principal Component Regression, Partial Least Squares)(James et al., ).

Table . -

 . Results of the linear regression on the macro, micro and bio data subsets before and after feature selection.

		Name	of	Number of	Quantity
		data subset	variables
				Residual	R 2	F-statistic
				standard
				error
	After feature selection	macro micro bio		. . .	. . .	.	. .

  des études environnementaux il est important de considérer uniquement des données de qualité qui fournissent des informations valables. Il est évident que pour fournir des résultats précis il est nécessaire d'avoir des données de bonne qualité. Dans la science des données les procédures de pré-traitement des données sont proposées pour réduire des anomalies et ainsi améliorer la qualité des données. Le choix des méthodes de prétraitement des données est crucial pour la validité des résultats d'analyses statistiques et il est assez mal défini.

Table . -

 . ). Les sites H et H sont situés proche aux champs agricoles, ils ont été choisis pour évaluer les quantités des pesticides organochlorés dans l'eau de la rivière Tula. Les sites H , H et H sont localisés près des zones urbanisées ils ont été sélectionnés pour évaluer les niveaux des polluants principalement PPCPS (Produits pharmaceutiques et de soin). Les détails sur l'emplacement des sites d'échantillonnage sont données dans la table .Pour évaluer la qualité de l'eau des rivières Tamazula, Humaya et Culiacan once sites ont été sélectionnés (c.f. Figure.) Les détails sur l'emplacement de sites d'échantillonnage sont données dans la table . . Quatre sites localisés au long de la rivière Humaya (C -C ) ont été choisis pour évaluer les concentrations de polluants principalement de déchets domestiques. Trois sites (C -C ) ont été pris dans la rivière Tamazula pour représenter l'état le moins pollué. Les sites C et C ont été choisis pour évaluer le niveaux de Chapter Emplacement des sites d'échantillonnage de la rivière Tula

	Sites	Commune	Latitude		Longitude		Altitude (m)
	H	Tasquillo	°	.	'	° .	'	.
	H	Ixmiquilpan	° .	'	° .	'	.
	H	Ixmiquilpan	° .	'	° .	'	.
	H	Tlacotlapilco	° .		° . '	.
	H	Progreso	° .	'	° .	'	.

  Finalement, trente-cinq métriques (indicateurs biologiques) ont été calculé, ils comprennent: • indicateurs de richesse (nombre de taxons totales, nombre des taxons EPT, nombre des taxons Ephemeroptera, nombre des taxons Plecoptera et nombre des taxons Trichoptera) ; • indicateurs d'énumération (nombre des familles en commun, %EPT, %Ephemeroptera, % Plecoptera, % Trichoptera, % Coleoptera, % Diptera, % Chironomidae, % EPT: Chironomidae, % taxons plus dominants, % genres plus dominantes); • indices de diversité et similitude (Shanon, Simpson, Margalef, Sequential Comparison Index -SCI, Jaccard et Sørensen); • indices biotiques (Trent Biotic index, Extended Biotic index, Beck Biotic index, Family biotic index, Biological Monitoring Working Party et Average Score per Taxon); • groupes fonctionnelles (les filtreurs, les prédateurs, les détritivores, les herbivores et les omnivores) et; • indice multi-métrique (IBI-west central Mexico).

	Heckman,	; Heckman,	; Heckman,	; Merrit et al.,	a; Tachet
	et al., a). Le jeux des données hydrobiologiques collectées des rivières Tula, Tamazuala, Hu-; Novelo-Gutiérrez, b; Novelo-Gutiérrez,
	maya et Culiacan est composé de	individus (sites d'échantillonnage) et	paramètres
	numériques ( métriques).

  données nommées training et test où % des données ont été utilisé pour training et % pour test ; . Le modèle de régression a été calculé sur le sous-ensemble de données training. Par la suite, le modèle ajusté a été utilisé pour prédire les réponses pour les observations dans le jeu de données test ; . Le résultat de la validation a été évalué en calculant le RMSE ; . Enfin, l'erreur de pré-traitement a été calculé en comparant la valeur de RMSE du jeu de données non-deformé avec celle du jeu de données pré-traitée comme suit: Error RM SE processing = RM SE P reprocessed -RM SE original RM SE original * 100% ( . ) Les trois méthodes de classification qui ont été utilisées sont : • Linear classification : Linear discriminant analysis (LDA) ; • Non-Linear classifcation : Naïve Bayes (NB) ; • Non-Linear classifcation with Regression Trees : Classification and Regression Trees (CART)

  Chapter vent être spécifiés par l'utilisateur. Les modèles sont construits en créant des sousensemble de données pour training et test. La précision du modèle est ensuite calculée et les résultats peuvent être sauvegardés. De manière similaire l'analyse par classification est effectuée en utilisant un sousensemble de données. La prédiction peut être faite en utilisant le modèle qui en résulte et l'utilisateur peut calculer l'exactitude et la valeur de Kappa. L'analyse de regroupement par K-means est exécuté en spécifiant le nombre de clusters k. Les résultats sont affiches dans un graphique où l'utilisateur peut sélectionner le nombre de clusters k. Les résultats de la méthode de groupement hiérarchique sont fournis de manière graphique où les observations sont représentés de manière arborescente dans un dendrogramme. La matrice de corrélation est implémenté en utilisant tous les variables, les résultats sont composés d'une liste de variables le plus corrélés et d'une représentation graphique de la matrice de corrélation. L'ACP est construit en utilisant uniquement des données numériques. Les résultats de l'ACP sont représentés graphiquement pour les variables et les observations, où les deux premières composantes principales (Axe et Axe ) peuvent être visualisés. Les résultats numériques de l'ACP et les graphiques peuvent être sauvegardé.

Table A .

 A -Biomonitoring Working Party Score System (BMWP)(National Water Council; )

	Families	Score
	Siphlonuridae Heptageniidae Leptophlebiidae Ephemerellidae
	Potamanthidae Ephemeridae
	Taeniopterygidae Leuctridae Capniidae Perlodidae Perlidae
	Chloroperlidae
	Aphelocheiridae
	Phryganeidae Molannidae Beraeidae Odontoceridae
	Leptoceridae Goeridae Lepidostomatidae Brachycentridae
	Sericostomatidae
	Astacidae
	Lestidae Agriidae Gomphidae Cordulegasteridae Aeshnidae
	Corduliidae Libellulidae
	Psychomyiidae Philopotamidae
	Caenidae
	Nemouridae
	Rhyacophilidae Polycentropodidae Limnephilidae
	Neritidae Viviparidae Ancylidae
	Hydroptilidae
	Unionidae
	Corophiidae Gammaridae
	Platycnemididae Coenagriidae
	Mesoveliidae Hydrometridae Gerridae Nepidae Naucoridae
	Notonectidae Pleidae Corixidae
	Haliplidae Hygrobiidae Dytiscidae Gyrinidae
	Hydrophilidae Crambidae Helodidae Dryopidae Elimidae
	Chrysomelidae Curculionidae
	Hydropsychidae
	Tipulidae Simuliidae
	Planariidae Dendrocoelidae
	Baetidae
	Sialidae
	Piscicolidae
	Valvatidae Hydrobiidae Lymnaeidae Physidae Planorbidae
	Sphaeriidae
	Glossiphoniidae Hirudidae Eropobdellidae
	Asellidae
	Chironomidae
	Oligochaeta (whole class)

Average Score Per Taxon (ASPT) (Armitage et al.,

Table A .

 A -Biological Monitoring Working Party (BMWP) and Average Score Per Taxon (ASPT) scores and their related quality index(Armitage et al.,

					; Friedrich et al.,	; National Water
	Council,	).
	BMWP ASPT score Category	Interpretation
	-		, or less	Very poor Heavily polluted
	-	, -,	Poor	Polluted or impacted
	-	, -,	Moderate Moderately impacted
	-	, -,	Good	Clean but slightly impacted
	>		Over ,	Very good Unpolluted, unimpacted

Table A .

 A -Functional Feeding Groups (FFG): Categorization and food resources. Coarse Particulate Organic Matter (CPOM); Fine Particulate Organic Matter (FPOM)(Merrit et al., ).

					Chapter A
	Macroinvertebrate-
	Functional groups	Particle size feeding	Dominant	food	Particle size range of
		mechanisms	ressources		food (mm)
	Shredders	Chew conditioned letter	CPOM-decomposing (or	> ,
		or live vascular plant tis-	living hydrophyte) vas-
		sue, or gouge wood	culat plants
	Filtering collectors	Suspension feeders : fil-	FPOM-decomposing de-	, -,
		ter particles form the	trital particles ; algae,
		water column	bacteria, and feces
	Gathering collectors Deposit feeders : ingest	FPOM-decomposing de-	, -,
		sediment or gather loose	trital particles ; algae,
		particles in depositional	bacteria, and feces
		areas
	Scrapers	Graze rock and wood	Periphyton-attached,	, -,
		surfaces or stems of	non-filamentous	algae
		rooted aquatic plants	and associated detritus,
			microflora and fauna,
			and feces
	Predators	Catpure and engulf prey	Prey-living animal		> ,
		or tissue, ingest body
		fluids

based Index of Biotic Integrity (IBI designed for west- central Mexico streams) (Weigel et al.,

Table A .

 A -Scoring criteria of the Macroinvertebrate-based Index of Biotic Integrity (IBI designed for west-central Mexico streams)(Weigel et al., ).

	Metric		Poor( ) Fair( ) Good( )
	Catch per unit effort (CPUE)	≤	-		>
	Generic richness (GR)
	Basin area ≤	Km	≤	-		≥
	Basin area >	Km	≤	-		≥
	% EPT
	Basin area ≤	Km	<	-		>
	Basin area >	Km	<	-		>
	% Chironomidae individuals (% Midge) >	-		<
	Hilsenhoff biotic Index (HBI)	>	.	-	< .
	% depositional individuals (%Depo)	>	-		<
	% predator individuals (%Pred)	<	-		>
	%gatherer genera (%Gath)	>	-		<

Table A .

 A -Macroinvertebrates-based Index of Biotic Integrity (IBI) quality values and their related biological responses to environmental conditions(Weigel et al., ).

	Value

and qualitative rating Biological response to environmental condition

  -Very goodComparable to the minimum influence system in the region. Macroinvertebrates are abundant. GR and %EPT are near the maximum for the size of stream. Chironomids are absent or %Midge is very low. Low HBI scores and % Depo indicate no organic pollution and sedimentation are undetectable. High % Pred and low % Gath indicate a predominance of specialized feeders. Non-point-source pollution is minimum or distant from the site. -Good Generally low levels of non-point-source pollution have influences some aspects of the macroinvertebrate assemblage. GR and % EPT are usually high but may not be at the maximum for the size of the stream. Often % Midge, % Depo and % Gath increase. Some organic pollution may be reflected in the HBI scores. -Fair Point-source pollution may be present distant from the site or in low quantities. Moderate to severe non-point-source pollution or diversion for irrigation are typically the major stressors. GR and % EPT are moderate to low for the size of stream. Odonates and other depositional taxa become more prevalent. HBI scores and % Midge typically run moderate to high. -Poor Point-source pollution is generally present but is intermittent or not immediately at the site. Non-point-source pollution can be severe. Abundance is typically low but, if it is high, is due to a high % Midge. HBI scores and % Depo suggest very tolerant assemblages. -Very poor System is severely degraded by point-source pollution. Zero values can result from all water being diverted for irrigation or collecting < individuals in CPUEs. Chironomids are typically the only macroinvertebrate fauna present and are usually abundant. If other taxa are present, their abundance is low. .

									.	.	.
		X	Y	Y	Y	Y	Y	Y	Y	Y	Y
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.

Table B .

 B -N synthetic dataset (n = 4000, p = 53) used to assess feature selection methods. The µ and σ 2 values of all variables are shown. Highly correlated variables are denoted as Y k .

	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		X	X	X	X	X	X	X	X	X	X
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		X	X	X	X	X	X	X	Y	Y	Y
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		Y	Y	Y
	µ	.	.	.
	σ 2	.	.	.

Table B .

 B -N synthetic dataset (n = 20000, p = 98) used to assess feature selection methods. The µ and σ 2 values of all variables are shown. Highly correlated variables are denoted as Y k . .

								.	.	.	.
		Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		Y	Y	Y	Y	Y	Y	Y	Y	Y	Y
	µ	.	.	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.	.	.
		Y	Y	Y	Y	Y	Y	Y	Y
	µ	.	.	.	.	.	.	.	.
	σ 2	.	.	.	.	.	.	.	.

Table B .

 B -N synthetic dataset (n = 20000, p = 98) used to assess normalization methods. The µ and σ 2 values of all variables are shown.

Table B .

 B

Process in which a body of water acquires high concentrations of nutrients, specifically phosphates and nitrates. It also causes a massive development of plants.Lack of oxygen in a body of water produced by oxidation of high concentrations of organic matter.

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . σ 2 . . . . .

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X Y µ . . . . . . . . . . σ 2 . . . . .

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . .

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X Xµ . . . . . . . . . . σ 2 . . . . . . . . . . X X X µ . . . σ 2 . . .

VariablesX X X X X X X X X X

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X µ . . . σ 2 . . .

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X Xµ . . . . . . . . . . σ 2 . . . . . . . . . . µ . . . . . . σ 2 . . . . . .

VariablesX X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X X X X X µ . . . . . . . . . . σ 2 . . . . . . . . . . X X X X X X µ . . . . . . σ 2 . . . . . .

Este trabajo a sido realizado bajo la ayuda de una beca CONACyT (Beca para estudiantes mexicanos en el extranjero) iv

. . Acquisiton de données physico-chimiques et chimiques

Des échantillons d'eau ont été collectés pour déterminer le contenu de : pH, conductivité, carbonate (CO ), bicarbonate (HCO ), sulphate (SO ), chlore (Cl), fluor (F), sodium (Na), potassium (K), calcium (Ca), magnésium (Mg), bore (B), dioxyde de silicium (SiO ), nitrate (NO ) arsenic (As), cadmium (Cd), plomb (Pb), fer (Fe), cuivre (Cu), manganèse (Mn), zinc (Zn), dix-huit pesticides organochlorés: I-BHC, II-BHC, III-BHC, IV-BHC, heptachlor, aldrin, heptachlor epoxide, I-endosulfan, II-endosulfan, dieldrin, DDE, endrin,

APPENDIX A

Biotic indices

Trend Biotic Index (Metcalfe, ) The organisms used for the computation of index are identified at family, genus or species level depending on the type of organism. The index values go from 0 to 10, with 10 representing clean streams, this value decreases with increasing pollution. In Table A. , are represented the key groups and the Biotic index values necessary to estimate the Trent Biotic Index of a given sample. It is a two-entrance table: vertical entrance corresponds to the value of richness found and the horizontal entrance to the less tolerant organism.

Chapter A

Extended Biotic Index (EBI) (Ghetti, ) EBI values are determined considering the abundance of specific macroinvertebrate taxa which is ordered according to its tolerance to stress factors. EBI values go from 0 to 14. They are calculated using a two-entrance table (c.f. Table A. ): a vertical entrance -corresponding to the value of Richness found; and a horizontal entrancecorresponding to the less tolerant Systematic Units (SU) to stress factors. To determine the ecological quality of aquatic ecosystems, EBI values are calculated and converted into "quality classes". The organisms used to compute the Extended Biotic Index (EBI), are identified at family or genus level. Beck Biotic Index (Beck, ) To calculate this index, macroinvertebrates are counted and organized according to the three classifications proposed by Beck (c.f. Table A. ). The three classes categorize macroinvertebrates according to their organic pollution tolerances. They are defined as follows: Class I Organisms (Sensitive or Intolerant): organisms that exhibit rapid response to aquatic environmental degradations and are reduced in number. Class II Organisms (Facultative): organisms that are capable to survive under polysaprobic conditions. Class III Organisms (Tolerant): organisms that have high resistance to adverse conditions within the aquatic environment. Once the organisms have been counted and classed the Beck's Biotic Index is calculated using the following expression:

Where BI is the Beck's Biotic Index n is the number of Class I organisms identified n is the number of Class II organisms identified Index values range from to about and, at lower values of the index, the organic pollution is greater. (Hilsenhoff, , Plafkin et al., ). Organisms are identified at family level. Tolerance values are assigned from for organisms very intolerant to organic pollution to for organisms very tolerant to organic pollution (c.f. Table A . ). FBI is computed using the following expression:

where xi is the number of individuals in the i th taxon t i is the tolerance value of the i th taxon n is the total number of organisms in the sample. -N synthetic dataset (n = 20000, p = 98) used to assess imputation methods. The µ and σ 2 values of all variables are shown.

B. Datasets for assessment of imputation of missing values

Variables

B. Datasets for assessment of outliers processing

B.

Semi-synthetic datasets