.1: GP Parameters for the different methods

deaux, France. Specially, I thank to Dr. Frédérique Faïta-Aïnseba and her team for making an excellent job during the EEG recording sessions.

In a personal note, I would like to give special thanks to my wife, Anabel,

R E S U M E N

Los avances en el desarrollo de Interfaces Cerebro-Computadora (BCI, por sus siglas en inglés Brain-Computer Interface) se han incrementado en años recientes, principalmente porque ha evolucionado el nivel de convergencia de técnicas multidisciplinarias. La electroencefalografía (EEG), una teécnica de grabación de señales cerebrales estudiado en esta tesis, permite la construcción de BCIs, sin embargo las señales son complejas para procesar, lo cual requiere metodologías que extraigan patrones de forma eficiente. Esta tesis explora dos tópicos principales: primero, se propone un sistema para el reconocimiento de convulsiones de epilepsia usando una combinacion de metodos de procesamiento de señales para la extracción eficiente de rasgos; segundo, explora el uso de un algoritmo meta-heurístico, Programación Genética (GP, por sus siglas en inglés Genetic Programming), como una alternativa en el diseño de BCIs. Sin embargo, existen temas sin resolver in GP que esta tesis explora: ¿existe una metodologia de búsqueda en GP más eficiente?; ¿cual es una representación apropiada dependiendo del problema a estudiar?; ¿cual son los operadores de búsqueda más adecuados?. De esta forma, se presenta un estudio a fondo con la introducción de un GP memetico aplicado a problemas de regresión. Despues, se extiende adaptandolo a problemas de clasificación. Los resultados son positivos; GP se beneficia fuertemente de la combinación de una metodologia general de busqueda y una local (LS, por sus siglas en inglés Local Search). Los ultimos dos cuestionamientos se estudian simultáneamente en el desarrollo de un sistema de reconocimiento para estados mentales usando EEG. Se propone una versión de GP ( + FEGP) que evoluciona modelos de extracción de rasgos usando operadores especializados de busqueda, representación de individuos y función de aptitud. Los resultados muestran que esta combinación permite una exactitud de clasificación que aporta en el estado-del-arte para la tarea particular del reconocimiento de estados mentales.

Palabras clave: EEG, Extracción de Rasgos, Regularidad Hölderiana, Matching Pursuit, Programación Genética, Regresión, Clasificación.

i A B S T R A C T

The advances in the development of Brain-Computer Interfaces (BCI) have been increasing in recent years, mostly because the level of convergence from multi-disciplinary techniques has evolved. The electroencephalography (EEG), a brain recording method studied in this thesis, allows the construction of BCIs, however the signals are rather complex to process, which requires methodologies that efficiently extract patterns from them. This thesis explores two directions: first, a system is proposed for the epilepsy seizures recognition using a combination of signal processing methods for an efficient feature extraction; second, it explores the usage of a meta-heuristic algorithm, namely Genetic Programming (GP), as an alternative in the design of BCIs. Nonetheless, there is currently open-issues in GP that this thesis also explores: is there a more efficient search methodology in the exploration by GP?; what is a proper representation depending on the studied problem?; which are the most adequate search operators?. For the first topic, a thoroughly study is presented by introducing a memetic GP applied to regression problems. Then, it is extended by adapting it to classification problems. The results are positive; GP is greatly benefited from the combination of a general and a Local Search (LS) methodology. The last two topics are studied simultaneously in the development of a recognition system for mental states using EEG. A GP version ( + FEGP) is proposed that evolves feature extraction models by using specialized search operators, individuals representation and fitness function. The results show that the combination of these reaches a state-of-the-art accuracy for the particular task of mental states recognition.

Keywords: EEG, Feature Extraction, Hölderian Regularity, Matching Pursuit, Genetic Programming, Regression, Classification.

iii To God, my beloved parents, my wife and my daughter.

v vi A C K N O W L E D G E M E N T S
Firstly, I would like to express my gratitude to my advisor Dr. Leonardo Trujillo Reyes for the continuous support during my doctoral timeline, for his patience, determination, motivation and great vision. During the doctoral road there were several moments where I had hesitations, nevertheless his guidance showed me better paths, revealing me what a great mentor he is.

Besides my advisor, I would like to specially thank to my co-advisor Dr. Pierrick Legrand, for the uncountable times where he gave me fully selfless support with time, guidance, patience, resources and even a good friendship. I thank to his family that gives him the energy to allow him to offer the best of him.

My sincere thanks also goes to the rest of my thesis committee: Dr. Luis Nestor Coria de los Ríos, Dr. Nohé Ramón Cazarez Castro, Dr. José Ricardo Cárdenas Valdez and Dr. Victor Hugo Díaz Ramírez, for their insightful comments, but also for the tough questions which encouraged me to improve my research. My gratitude also goes to other professors from the Postgraduate program that directly or indirectly influenced this dissertation: Dr. Yazmín Maldonado Robles, Dr. Youness El Hamzaoui, Dr. Paul Valle Trujillo and Ms. Arturo Sotelo. I thank my fellow labmates for the times that we shared joy and frustration along these 4 years: Enrique, the Elder; Yuliana, the Enthusiast; Victor, the Joker; Uriel, the Kramer; Angel, the Instigator; Perla, the Dreamer; Ramón, the Xolo; Carlos, the Conspirator; Paul, the Señor; Luis, the Erudite; Pablo, the Cuban; Adrian, the Jungle boy.

I would like to thank to CONACYT, that allows young prospects to pursue their dreams providing a scholarship (No. 294213) through postgraduate programs across the country. I would like to express my gratitude to the involved people in the FP7 Left image shows a dyadic wavelet decomposition, and the right image display the actual regression calculated over the point t 0 , where each dot corresponds to each log 2 of the wavelet coefficient magnitude located approximately above t 0 . Wholesale (e) and Banknote (f). Fitness performance. Plots shows the median over 20 independent runs from the best-so-far solution. . . . . . Figure 5.6

Results of fitness performance of problem LSVT. Fitness performance. Plots shows the median over 20 independent runs from the best-so-far solution. . . . . . . . . . . . . . . . . . . . . . . . Figure 5.7

Box plots from 20 runs at the end of each run; where: (a) presents the test fitness of the best solution found during testing; and (b) the average population size. . . . . . . . . . . . . . . . . . . . Figure 6.1 Methodology of the reference system . . . . . . . Figure 6.2

Relative position of the electrodes based on the international standard system 10/10. . . . . . . . Acquisition protocol of the CNV, describing the stimuli presented to each subject. Each task consists of 50 pairs of stimuli. . . . . . . . . . . . . . Figure 6.5

Relationship of the vigilance level and the amplitude on the CNV test. Main ERP components are shown here: P1, N1, P2, N2 and P3. . . . . . . Figure 6.6

CNV obtained for subject 6. Validation interval is achieved between T1 and T2. S1 and S2 belong to the corresponding stimulus triggering time, as shown in Fig. 6.4. In this example, the EEG recordings are considered valid since the expected relaxed state has lower amplitude than the normal state. . . . . . . . . . . . . . . . . . . . Figure 6.7

Frequency response for the Butterworth filter used in this work. . . . . . . . . . . . . . . . . . . Figure 6.8

Classification performance for the reference system. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure 6.9

Methodology for the proposed enhanced system. Figure 6.10 GP flowchart in a nutshell. . . . . . . . . . . . . . Figure 6.11

Block diagram for the proposed + FEGP. . . . . . Figure 6.12 + FEGP individual representation. . . . . . . . . . Figure 6.13

+ FEGP inter-crossover, swapping subtrees between different individuals. F1 i is the local fitness; the crossover points are chosen from the subtrees with max(F1). . . . . . . . . . . . . . . . Figure 6.14 + FEGP intra-crossover, swapping subtrees within the same individual. . . . . . . . . . . . . . Figure 6.15

+ FEGP inter-individual feature crossover. . . . . Figure 6.16

+ FEGP feature insertion mutation. . . . . . . . . Figure 6.17

Operator probabilities through the search. . . . . Figure 6. 18 Class variance penalization, where (a) corresponds to the tree output, in this example f 1 , f 2 ∈ i = 1 . . . r are new features; and (b) corresponds to the data modification with a larger distribution variance used to compute the global fitness. Parameters for all algorithms; all GSGP variants (GSGP, GSGP-LS, HYBRID, HYBRID-LIN) share these parameters. . . . . . . . . . . . . A d Brain-Computer Interface (BCI) is a communication system that allows a person to send and receive control signals toward a external device (usually a computer) by means of a brain generated signal. Such systems have been valuable in different scenarios, like entertainment, security, education, neuromarketing, neuroergonomics and medical [START_REF] Abdulkader | Brain computer interfacing: Applications and challenges[END_REF], with the latter serving as the initial inspiration for the development of BCIs. Indeed, the healthcare field is one of the top priorities for human development, where a variety of applications have employed BCI including prevention, detection, diagnostic, rehabilitation and restoration.

However, the interest in other fields has been increasing over time. For example, in the area of neuroergonomics and smart environments, the exploitation of BCIs has been seen to improve the quality of the human's daily life, like the control of smart houses, workplaces or transportation [START_REF] Peng | An improved EEG de-noising approach in electroencephalogram (EEG) for home care[END_REF].

Brain signals can also potentially be used to assist the improvement of workplace conditions by the estimation of an operator' cognitive state [START_REF] Venthur | Novel applications of BCI technology: Psychophysiological optimization of working conditions in industry[END_REF]. The field of intelligent transportation has also been benefited from the cognitive state by monitoring BCI functions. In this domain, the behavior of drivers has been reviewed in numerous studies. It has been found that distraction and fatigue are considered a strong cause for most traffic accidents [START_REF] Dong | Driver inattention monitoring system for intelligent vehicles: A review[END_REF]. Being recognized that the mental state of a person is critical in several domains, it is one of the main reasons behind the motivation of this thesis.

Introduction Designing BCIs systems is a complex task that requires multidisciplinary works related to computer science, neuroscience and psychology. The design of such a system usually involves several elements: 1. Brain activity capture: Brain signals are recorded by means of a sensor, such as Electroencephalogram (EEG), the method used in this work.

2. Pre-processing: Recorded data is cleaned and denoised.

3. Feature extraction: Input data is transformed into a compact feature space.

4. Classification: An algorithm is used to build a decision model upon the feature set and the tasks to be recognized.

Translation into a command:

A command is associated to the mental task or state to control a given application.

6. Feedback: It provides a feedback to the user, which improves the user's performance.

This thesis is focused on the first four components, with the main original contributions oriented toward the feature extraction, while the last two components are beyond the scope of this work.

T H E S I S O B J E C T I V E S

The work presented in this manuscript belongs to a combination of BCI research, signal processing, machine learning and Genetic Programming (GP) areas. The main goal of this research is focused in the development of EEG recognition systems, first from a classical perspective, that is using tools from signal processing and statistical learning; and second, using an alternative meta-heuristic approach. In the former view, we will explore the current advances in the development of EEG recognition systems and thus propose our own methodology. The design of efficient BCI systems is still considered an open topic, mainly because its complexity is considered high. A wide range of tools and algorithms have been used to design different functional layers of BCIs, either separately or combined. The majority of the research done in this domain is rooted from ideas and concepts derived from traditional optimization and signal processing. An alternative to this point of view are meta-heuristic based algorithms, a superset of tools inspired by a different reasoning: the exact optimal solution is not fundamental to solve the problem at hand. In the broad group of meta-heuristic algorithms, GP stands as a powerful technique that does not require explicit prior knowledge of the data. Despite that GP has been used in different parts of BCI design, the fact remains that GP fundamentally suffers from limitations that we will try to address in the second part of the manuscript. This direction is important because a successful adaptation of GP toward a specific application depends in a large scale on how these drawbacks are handled. Finally, we will combine all prior knowledge to propose a new methodology to solve the recognition of mental states using EEG, based on a GP algorithm.

T H E S I S O R G A N I Z AT I O N A N D C O N T R I B U -T I O N S

The diagram in Figure 1.1 shows a complete synopsis of the thesis roadmap in terms of the research directions taken during this journey. This map directly correlates with the presented topics in this manuscript. Each of the roads represents different research areas that contributes to the main goals of the thesis. Understanding this diagram helps, first, to locate each event and the domain where it resides, and second, to comprehend the convergence of the secondary roads into the main road, and their interdependencies along the doctoral timeline.

Parting from this map, a type of guideline for the explored research areas, we now change the direction toward a more pragmatic viewpoint: the manuscript organization.

Part I: Epileptic seizures classification

In the first part of the document, we are going to examine the BCI design from a classical perspective, that is the current advances in EEG recognition are going to be discussed, as well as our contribution for a specific application in this domain. In this part, two chapters cover the aforementioned standpoint.

Chapter 2 of this manuscript will be focused on an overview of the current BCI design methodologies, including acquisition, preprocessing, feature extraction and classification.

As briefly mentioned before, one of the main topics that this manuscript is trying to target is the design of efficient feature extraction methods. In Chapter 3, we will present current advances in methods and algorithms used to solve the problem of feature extraction and classification of EEG signals. Moreover, we will focus to the recognition of epileptic seizures, a relative common condition present in a large percentage of the worldwide population. In particular, we propose a novel methodology to design a robust feature extraction block, by employing a set of different measures. In particular, two techniques are combined to build this feature set. First, the Matching Pursuit is used to describe the EEG signals as a global regularity measure; and second, the Hölderian regularity, that works as a local descriptor measure. In the latter measure, an approximation of the measure is calculated with a method never used before in this domain. Additional simpler statistical features are used to compose the full feature set. Our comprehensive study shows that this new methodology outperforms the majority of other recent works [START_REF] Z-Flores | Regularity and Matching Pursuit feature extraction for the detection of epileptic seizures[END_REF].

Part II: Genetic Programming

The previous part of the investigation worked as a preamble to illustrate the most noteworthy difficulties normally found when handling EEG signals and building a BCI system; which were solved for a specific problem instance. Nevertheless, the complexity of the general problem requires to explore new alternatives. In that sense, the introduction of meta-heuristic techniques are viable options, mainly because they do Introduction not necessarily requires that a global optimum solution is found to solve the problem. They are particularly useful when the underlying data size is large and complex, that is each category is not easily distinguishable between them using simpler algorithms. In this part, we introduce GP as a particular meta-heuristic algorithm of interest. GP can adapt to a wide range of applications making it a valuable candidate to address the problem at hand. However, we will state that GP itself requires a series of changes to be successful in the application. Since its conception, it was designed to work as a highly abstract algorithm that eventually could handle very complex situations, but this carries as well some disadvantages. One of these limitations is the way the search is performed [START_REF] O'neill | Open issues in Genetic Programming[END_REF]; it is not optimal in terms of convergence time or solution complexity. Thus, in this part we try to address this limitation by incorporating the term of Local Search (LS) into GP, a well known technique used in other meta-heuristic algorithms. By combining global and local search operators, the overall performance is increased.

A study of the incorporation of LS strategies into GP is presented in Chapter 4. We will explore different variants in the methodology and what are the consequences for the search by using several performance measurements. The experimental work will target symbolic regression problems as the first of a two-parts study [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF], with the second part discussed in the following chapter.

Appendix A further extends the experimentation of symbolic regression using LS with GP, applied to real-world problems [START_REF] Trujillo | Local Search is Underused in Genetic Programming[END_REF].

Appendixes B, C and D focus in the modeling of three contrasting real-world problems using the proposed memetic GP. In appendix B, experimental data gathered from an electrical gas turbine is modeled by GP with LS. Furthermore, a comparison between another GP based technique and other machine learning algorithms is presented with outstanding performance for the models produced by our memetic approach [START_REF] Enríquez-Zárate | Automatic modeling of a gas turbine using genetic programming: An experimental study[END_REF]. Moreover, the effects of amplitude and phase in Radio Frequency (RF) amplifiers are modeled in appendix C producing competitive results compared with more commonly used approaches [START_REF] Valdez | Local Search Approach to Genetic Programming for RF-PAs Modeling Implemented in FPGA[END_REF]. In the consequent appendix D, models are derived from the removal process of two chemical contaminants commonly present in wastewater. In summary, for all three case studies, GP with LS produced better and compacter models than other state-of-the-art techniques.

In the last appendix, E, a software program is presented with the aim to provide a modeling tool that is modern and user-friendly. The core of this tool is powered by GP with the capability of running in parallel hardware. Currently, it is still in development.

In Chapter 5, the previous study will be extended by addressing classification problems. In there, various modifications are introduced, like the chosen heuristic for LS selection or the proposal for a fitness function that handles binary classification problems. The methodology is tested using real-world benchmark problems [START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF].

Part III: Mental states classification

In the Part II we discussed one of the deficiencies of GP as an efficient global searcher. However, is not the only open issue that still exists in GP [START_REF] O'neill | Open issues in Genetic Programming[END_REF]. Another important topic is a proper representation of the individuals. The relationship between the representation and the problem at hand is strong if the solution needs to be found in a very efficient way. Additional parameters affect the overall performance of GP, very similarly as other population-based algorithms, like the type of population initialization or the characteristics of the search operators. All these aspects will be discussed in Part III of this manuscript through the mental states classification.

Even within the field of BCI, there are several levels of difficulties and not every problem can be handled using the same approach. In the case of mental states recognition, the problem is considered complex because apart of the common differences in the EEG signals in terms of the cognitive states, the signals carry a combination of other conditions either induced or already present during the experiment. Focusing again in the design of an efficient feature extraction, we will present a GP variant that we expect to handle this problematic in a effective manner. For that, we will introduce several modifications into GP.

Introduction

In Chapter 6, we will conclude this journey by addressing one of the original objectives of the thesis: to design a BCI system which will be able to recognize human mental states using EEG by means of GP. In particular, we will use GP to design the feature extraction model. However, we will present a full system including the necessary BCI parts like the brain activity capture, pre-processing, feature extraction and classification. This system is build upon a reference system using classical techniques [START_REF] Vézard | EEG classification for the detection of mental states[END_REF]. Since GP plays, in this chapter, the main role, we will focus heavily in the discussion of our original contribution: a new representation, new search or genetic operators, a new fitness function and a new methodology for terminal selection; we will name this contribution as Augmented Feature Extraction with Genetic Programming ( + FEGP). The algorithm will be supported by the necessary experimental evidence.

At Chapter 7, the general thesis conclusions are going to be presented.

Conference papers

1. Z-Flores, E., [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF]

Epileptic seizures classification

2

A N O V E R V I E W O F B C I 2.1 I N T R O D U C T I O N
A BCI is an artificial intelligence system that can recognize a certain set of patterns in brain signals following five consecutive stages: signal acquisition, preprocessing or signal enhancement, feature extraction, classification, and the control interface (Nicolas-Alonso and Gomez-Gil, 2012). The signal acquisition stage captures the brain signals and may also perform noise reduction and artifact processing. The preprocessing stage prepares the signals in a suitable form for further processing. The feature extraction stage identifies discriminative information in the brain signals that have been recorded. Once measured, the signal is mapped onto a vector containing effective and discriminant features from the observed signals. The extraction of this interesting information is a very challenging task. Brain signals are mixed with other signals coming from a finite set of brain activities that overlap in both time and space. Moreover, the signal is not usually stationary and may also be distorted by artifacts such as electromyography (EMG) or electrooculography (EOG) (Nicolas-Alonso and Gomez-Gil, 2012). The feature vector must also be of a low dimension, in order to reduce feature extraction stage complexity, but without relevant information loss. The classification stage classifies the signals taking the feature vectors into account. The choice of good discriminative features is therefore essential to achieve effective pattern recognition, in order to decipher the user's intentions. Finally the control interface stage translates the classified signals into meaningful commands for any connected device, such as a wheelchair or a computer. In the following sections, a brief explanation of each stage in BCI is presented, with the exception of the control interface, which is outside of the scope of this manuscript.

S I G N

A L A C Q U I S I T I O N
The human being is often referred as the subject and is considered the source of the brain signals [START_REF] Abdulkader | Brain computer interfacing: Applications and challenges[END_REF]. The measurement of the neurophysilogic state of the brain is called signal acquisition. There are two types of brain activities that may be monitored: electrophysiological and hemodynamic.

Electrophysiological activity is generated by electro-chemical transmitters exchanging information between the neurons. The neurons generate ionic currents which flow within and across neuronal assemblies. The large variety of current pathways can be simplified as a dipole conducting current from a source to a sink through the dendritic trunk. These intracellular currents are known as primary currents. Conservation of electric charges means that the primary currents are enclosed by extracellular current flows, which are known as secondary currents [START_REF] Baillet | Electromagnetic brain mapping[END_REF]. Electrophysiological activity is measured by electroencephalography, electrocorticography, magnetoencephalography, and electrical signal acquisition in single neurons.

The hemodynamic response is a process in which the blood releases glucose to active neurons at a greater rate than in the area of inactive neurons. The glucose and oxygen delivered through the blood stream results in a surplus of oxyhemoglobin in the veins of the active area, and in a distinguishable change of the local ratio of oxyhemoglobin to deoxyhemoglobin [START_REF] Laureys | Functional neuroimaging[END_REF]. These changes can be quantified by neuroimaging methods such as functional magnetic resonance and near infrared spectroscopy. These kinds of methods are categorized as indirect, because they measure the hemodynamic response, which, in contrast to electrophysiological activity, is not directly related to neuronal activity.

Most current BCIs obtain the relevant information from the brain activity through electroencephalography. Electroencephalography is by far the most widely used neuroimaging modality, owing to its high temporal resolution, relative low cost, high portability, and few risks to the users. BCIs based on electroencephalography consist of a set of sensors that acquire electroencephalography signals from different brain areas. However, the quality of electroencephalography signals is affected by scalp, skull, and many other layers as well as background noise. Noise is key to electroencephalography and to other neuroimaging methods, insofar as it reduces the SNR and therefore the ability to extract meaningful information from the recorded signals.

In general, there are two classes of brain acquisition methods: invasive and non-invasive methods. Each type will be further discussed below.

Invasive methods

Invasive modalities need to implant microelectrode arrays inside the skull that involves significant health risks, which restricts their use to experimental settings. Two invasive modalities can be found in BCI research: electrocorticography, which places electrodes on the surface of the cortex, either outside the dura mater (epidural electrocorticography) or under the dura mater (subdural electrocorticography) [START_REF] Roland | Brain mapping in a patient with congenital blindness -a case for multimodal approaches[END_REF], and intracortical neuron recording which implants electrodes inside the cortex [START_REF] Hochberg | Neuronal ensemble control of prosthetic devices by a human with tetraplegia[END_REF].

Electrocorticography (ECoG)

ECoG is a recording method that brings a less invasive option while at the same time preserves the advantages of invasive approach. It involves implanting electrode grids or strips over the cortex surface through a surgical operation [START_REF] Roland | Brain mapping in a patient with congenital blindness -a case for multimodal approaches[END_REF]. ECoG is an invasive recording modality which requires a craniotomy to implant an electrode grid, entailing significant health hazards. For that reason, the first studies on ECoG were with animals [START_REF] Yuen | Tissue response to potential neuroprosthetic materials implanted subdurally[END_REF]. In humans, ECoG has been used for the analysis of alpha and beta waves or gamma waves [START_REF] Crone | Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis II. Event-related synchronization in the gamma band[END_REF] produced during voluntary motor action.

Intracortical

Intracortical neuron recording is a neuroimaging technique that measures electrical activity inside the gray matter of the brain. It is an invasive recording modality that needs to implant microelectrode arrays inside the cortex to capture spike signals and local field potentials from neurons; represents the most invasive method. Three signals can be obtained by intracortical neuron recording: single-unit activity (SUA), multi-unit activity (MUA), and local field potentials (LFPs) [START_REF] Waldert | A review on directional information in neural signals for brain-machine interfaces[END_REF].

Non-invasive methods

These recording methods follow the approach that does not require implanting of external objects into subject's brain. Thus it avoids the surgical procedures or permanent device attachment needed by invasive acquisition. These include functional magnetic resonance imaging, magnetoencephalography, functional near-infrared spectroscopy and electroencephalography.

Functional Magnetic Resonance Imaging (fMRI)

fMRI is a non-invasive neuroimaging technique which detects changes in local cerebral blood volume, cerebral blood flow and oxygenation levels during neural activation by means of electromagnetic fields. It depends on the fact that any usage of brain part requires the increase of incoming blood flow. It uses blood-oxygen-level-dependent (BOLD) contrast, which is sensitive to the hemodynamic response [START_REF] Ayaz | An optical brain computer interface for environmental control[END_REF]. The main advantage of the use of fMRI is high space resolution.

Signal acquisition

Magnetoencephalography (MEG)

MEG is a non-invasive imaging technique that registers the brain's magnetic activity by means of magnetic induction. MEG measures the intracellular currents flowing through dendrites which produce magnetic fields that are measurable outside of the head [START_REF] Waldert | A review on directional information in neural signals for brain-machine interfaces[END_REF]. The neurophysiological processes that produce MEG signals are identical to those that produce electroencephalography signals. The magnetic signal outside of the head is currently acquired only using the Superconducting Quantum Interference Device (SQUID) [START_REF] Babiloni | Fundamentals of Electroencefalography, Magnetoencefalography, and Functional Magnetic Resonance Imaging[END_REF].

Functional Near-Infrared Spectroscopy (fNIRS)

fNIRS is a noninvasive technique that measures blood dynamic in the brain in order to detect the neuronal activity. It uses light in the nearinfrared range to determine the blood flood. Infrared light penetrates the skull to a depth of approximately 1-3 cm below its surface, where the intensity of the attenuated light allows alterations in oxyhemoglobin and deoxyhemoglobin concentrations to be measured [START_REF] Coyle | Brain-computer interface using a simplified functional near-infrared spectroscopy system[END_REF].

Electroencephalography (EEG)

EEG measures electric brain activity caused by the flow of electric currents during synaptic excitations of the dendrites in the neurons and is extremely sensitive to the effects of secondary currents [START_REF] Baillet | Electromagnetic brain mapping[END_REF]. It was first used in humans by H. Berger in 1924. EEG signals are easily recorded in a non-invasive manner through electrodes placed on the scalp, for which that reason it is by far the most widespread recording modality. However, it provides very poor quality signals as they have to cross the scalp, skull, and many other layers. This means that EEG signals in the electrodes are weak, hard to acquire and of poor quality. This technique is moreover severely affected by background noise generated either inside the brain or externally over the scalp [START_REF] Nicolas-Alonso | Brain computer interfaces, a review[END_REF].

The EEG recording system consists of electrodes, amplifiers, A/D converter, and a recording device. The electrodes acquire the signal from the scalp, the amplifiers process the analog signal to enlarge the amplitude of the EEG signals so that the A/D converter can digitalize the signal in a more accurate way. Finally, the recording device, which may be a personal computer or similar, stores, and displays the data [START_REF] Binnie | Electroencephalography[END_REF].

The EEG signal is measured as the potential difference over time between signal or active electrode and reference electrode. An extra third electrode, known as the ground electrode, is used to measure the differential voltage between the active and the reference points. The minimal configuration for EEG measurement therefore consists of one active, one reference, and one ground electrode. Multi-channel configurations can comprise up to 128 or 256 active electrodes [START_REF] Teplan | Fundamentals of EEG measurement[END_REF]. These electrodes are usually made of silver chloride (AgCl) [START_REF] Sinclair | Basic Electronics in Clinical Neurophysiology[END_REF]. Electrode-scalp contact impedance should be between 1 kΩ and 10 kΩ to record an accurate signal [START_REF] Usakli | Improvement of EEG signal acquisition: An electrical aspect for state of the Art of front end[END_REF]. The electrode-tissue interface is not only resistive but also capacitive and it therefore behaves as a low pass filter. The impedance depends on several factors such as the interface layer, electrode surface area, and temperature [START_REF] Usakli | Improvement of EEG signal acquisition: An electrical aspect for state of the Art of front end[END_REF]. EEG gel creates a conductive path between the skin and each electrode that reduces the impedance. In Figure 2.1, a cap with electrodes is being placed over a studied person. The use of the gel is cumbersome, however, as continued maintenance is required to assure a relatively good quality signal. Electrodes that do not need to use of gels, called 'dry' electrodes, have been made with other materials such as titanium and stainless-steel [START_REF] Fonseca | A novel dry active electrode for EEG recording[END_REF].

The amplitude of electrical bio-signals is in the order of microvolts. A typical example of such signals is depicted in Figure 2.2. Consequently, the signal is very sensitive to electronic noise. External sources such power-lines or halogen lamps may generate background noise and thermal, shot, flicker, and burst noises are generated by internal sources [START_REF] Leach | Fundamentals of Low-Noise Analog Circuit Design[END_REF]. Design considerations should be addressed to reduce the effects of the noise, such as electromagnetic interference shielding or reduction for common mode signal, among others [START_REF] Usakli | Improvement of EEG signal acquisition: An electrical aspect for state of the Art of front end[END_REF].

EEG comprises a set of signals which may be classified according to their frequency. Well-known frequency ranges have been defined ac- cording to distribution over the scalp or biological significance. These frequency bands are referred to as delta (δ), theta (θ), alpha (α), mu (µ), beta (β) and gamma (γ) from low to high, respectively. Relevant characteristics of these bands are detailed below.

The delta band lies below 4 Hz, and the amplitude of delta signals detected in babies decreases as they age. Delta rhythms are usually only observed in adults in deep sleep state and are unusual in adults in an awake state. A large amount of delta activity in awake adults is abnormal and is related to neurological diseases [START_REF] Kübler | Brain-computer communication: unlocking the locked in[END_REF]. Due to low frequency, it is easy to confuse delta waves with artifact signals, which are caused by the large muscles of the neck or jaw.

Theta waves lie within the 4 to 7 Hz range. In a normal awake adult, only a small amount of theta frequencies can be recorded. A larger amount of theta frequencies can be seen in young children, older children, and adults in drowsy, meditative or sleep states [START_REF] Kübler | Brain-computer communication: unlocking the locked in[END_REF]. Like delta waves, a large amount of theta activity in awake adults is related to neurological disease [START_REF] Kübler | Brain-computer communication: unlocking the locked in[END_REF]. Theta band has been associated with meditative concentration [START_REF] Aftanas | Human anterior and frontal midline theta and lower alpha reflect emotionally positive state and internalized attention: High-resolution EEG investigation of meditation[END_REF] and a wide range of cognitive processes such as mental calculation [START_REF] Fernández | EEG activation patterns during the performance of tasks involving different components of mental calculation[END_REF], maze task demands [START_REF] Caplan | Distinct Patterns of Brain Oscillations Underlie Two Basic Parameters of Human Maze Learning[END_REF], or conscious awareness [START_REF] Klimesch | Theta synchronization during episodic retrieval: Neural correlates of conscious awareness[END_REF].

Alpha rhythms are found over the occipital region in the brain [START_REF] Pineda | The functional significance of mu rhythms: Translating "seeing" and "hearing" into "doing[END_REF]. These waves lie within the 8 to 12 Hz range. Their amplitude increases when the eyes close and the body relaxes and they attenuate when the eyes open and mental effort is made [START_REF] Black | The Operant Conditioning of Central Nervous System Electrical Activity[END_REF]. Increasing mental effort causes a suppression of alpha activity, particularly from the frontal areas [START_REF] Venables | The influence of performance feedback on goal-setting and mental effort regulation[END_REF]. These rhythms primarily reflect visual processing in the occipital brain region and may also be related to the memory brain function [START_REF] Klimesch | EEG-alpha rhythms and memory processes[END_REF]. Mu rhythms may be found in the same range as alpha rhythms, although there are important physiological differences between both. In contrast to alpha rhythms, mu rhythms are strongly connected to motor activities and, in some cases, appear to correlate with beta rhythms [START_REF] Pineda | The functional significance of mu rhythms: Translating "seeing" and "hearing" into "doing[END_REF][START_REF] Pfurtscheller | Mu rhythm (de)synchronization and EEG single-trial classification of different motor imagery tasks[END_REF].

Beta rhythms, within the 12 to 30 Hz range, are recorded in the frontal and central regions of the brain and are associated with motor activities. Beta rhythms are desynchronized during real movement or motor imagery [START_REF] Pfurtscheller | Motor imagery and direct braincomputer communication[END_REF]. Beta waves are characterized by their symmetrical distribution when there is no motor activity. However, in case of active movement, the beta waves attenuate, and their symmetrical distribution changes [START_REF] Pfurtscheller | Motor imagery and direct braincomputer communication[END_REF].

Gamma rhythms belong to the frequency range from 30 to 100 Hz. The presence of gamma waves in the brain activity of a healthy adult is related to certain motor functions or perceptions, among others [START_REF] Lee | Synchronous Gamma activity: A review and contribution to an integrative neuroscience model of schizophrenia[END_REF]. Some experiments have revealed a relationship in normal humans between motor activities and gamma waves during maximal muscle contraction [START_REF] Brown | Cortical correlate of the piper rhythm in humans[END_REF]. This gamma band coherence is replaced by a beta band coherence during weak contractions, suggesting a correlation between gamma or beta cortical oscillatory activity and force [START_REF] Mima | Force level modulates human cortical oscillatory activities[END_REF]. Also, several studies have provided evidence for the role of gamma activity in the perception of both visual and auditory stimuli [START_REF] Lee | Synchronous Gamma activity: A review and contribution to an integrative neuroscience model of schizophrenia[END_REF][START_REF] Lutzenberger | Visual stimulation alters local 40-Hz responses in humans: an EEGstudy[END_REF]. Gamma rhythms are less commonly used in EEG-based BCI systems, because artifacts such as electromyography (EMG) or electrooculography (EOG) are likely to affect them [START_REF] Zhang | Improving mental task classification by adding high frequency band information[END_REF].

As explained above, EEG is recorded by electrodes. The electrodes placed over the scalp are commonly based on the International 10-20 system [START_REF] Klem | The ten-twenty electrode system of the International Federation[END_REF], as seen in Fig. 2.3 which has been standardized by the American Electroencephalographic Society. The 10-20 system uses two reference points in the head to define the electrode location. One of these reference points is the nasion, located at the top of the nose at the same level as the eyes. The other reference point is the inion, which is found in the bony lump at the base of the skull. The transverse and median planes divide the skull from these two points. The electrode locations are determined by marking these planes at intervals of 10% and 20%. The letters in each location corresponds to specific brain regions in such a way that A represents the ear lobe, C the central region, P g the Figure 2.3: Electrodes placement over scalp according with the 10-20 standard.

nasopharyngeal, P the parietal, F the frontal, F p the frontal polar, and O the occipital area.

Table 2.1 summarizes the brain acquisition methods with their advantages and disadvantages.

S I G N A L P R E -P R O C E S S I N G

The most common types of pre-processing are artifact detection, spectral filtering and spatial filtering. A summary of these methodologies and their approaches are listed as follows:

• Artifact detection. This technique is based on finding confounding signals from sources outside the brain (e.g. eye movement or muscle artifacts), and then removing them from the recording.

• Spectral filtering. In this method, noise is removed, such as slow drifts or line noise. • Spatial filtering. Here, the combination of signals from multiple electrodes for focusing on activities at a particular location of the brain. It is used either to target or reject source-nodes based upon their position.

F E AT U R E E X T R A C T I O N

Different activities result in different patterns of brain signals. BCI is seen as a pattern recognition system that classifies each pattern into a class according to its features. BCI extracts some features from brain signals that reflect similarities to a certain class as well as differences from the rest of the classes. The features are measured or derived from the properties of the signals which contain the discriminative information needed to distinguish their different types (Nicolas-Alonso and [START_REF] Nicolas-Alonso | Brain computer interfaces, a review[END_REF].

The design of a suitable set of features is a challenging issue. The information of interest in brain signals is hidden in a highly noisy environment, and brain signals comprise a large number of simultaneous sources. A signal that may be of interest could be overlapped in time and space by multiple signals from different brain tasks. For that reason, in many cases, it is not enough to use simple methods such as a band pass filter to extract the desired band power.

Brain signals can be measured through multiples channels. Not all information provided by the measured channels is generally relevant for understanding the underlying phenomena of interest. Dimension reduction techniques such as Principal Component Analysis (PCA) or Independent Component Analysis (ICA) can be applied to reduce the dimension of the original data, removing the irrelevant and redundant information. Computational costs are thereby reduced.

In the first part of this manuscript, we recognize that the most demanding part of a successful BCI design is the feature extraction, hence we particularly focus in this area for the contributions in the consequent chapters.

C L A S S I F I C AT I O N

The aim of the classification step in a BCI system is recognition of a user's intentions on the basis of a feature vector that characterizes the brain activity provided by the feature extraction step. Either regression or classification algorithms can be used to achieve this goal, but using classification algorithms is currently the most popular approach [START_REF] Lotte | A review of classification algorithms for EEG-based brain-computer interfaces[END_REF].

Classification algorithms have traditionally been calibrated by users through supervised learning using a labeled data set. It is assumed that the classifier is able to detect the patterns of the brain signal recorded in online sessions with feedback. However, this assumption results in a reduction in the performance of BCI systems, because the brain signals are inherently non-stationary.

Classifiers also have to face two main problems related to the pattern recognition task: the curse of dimensionality and the bias-variance tradeoff. The curse of dimensionality means that the number of training data needed to offer good results increases exponentially with the di-mensionality of the feature vector [START_REF] Jain | Statistical pattern recognition: a review[END_REF]. Unfortunately, the available training sets are usually small in BCI research, because training process takes a long time and is a tiring process for users. The biasvariance tradeoff represents the natural trend of the classifiers towards a high bias with low variance and vice versa. Stable classifiers are characterized by high bias with low variance, while unstable classifiers show high variance with low bias. To achieve the lowest classification error, bias and variance should be low simultaneously. A set of stabilization techniques such as the combination of classifiers or regularization can be used to reduce the variance.

C H

A P T E R C O N C L U S I O N S
In this chapter we briefly reviewed some of the techniques employed in state-of-the-art BCI systems. Different neuroimaging approaches have been successfully applied in BCI: (i) EEG, which provides acceptable quality signals with high portability and is by far the most usual modality in BCI; (ii) fMRI and MEG, which are proven and effective methods for localizing active regions inside the brain; (iii) NIRS, which is a very promising neuroimaging method in BCI; and (iv) invasive modalities, which have been presented as valuable methods to provide the high quality signals required in some multidimensional control applications e.g., neuroprostheses control.

A wide variety of signal features and classification algorithms have been tested in the BCI design. Although BCI research is relatively young, many advances have been achieved in a little over two decades, because many of these methods are based on previous signal processing and pattern recognition research. Many studies have demonstrated the valuable accuracy of BCIs and provided acceptable information bit rate, despite the inherent major difficulties in brain signal processing.

In the next chapter we will discuss a particular application of BCI, the recognition of epileptic seizures, a rather common condition. Although this being a well recognized situation from neuroscience and physiology perspectives, is still an open issue in terms of the design of robust auto-matic recognition systems. Similarly as other BCI applications that use EEG as the brain capture method, the signals also in this domain are complex and their patterns recognition is not trivial by any means. For that, we will introduce our first contribution by employing a novel combination of two different signal processing methods to develop a feature extraction model.

C L A S S I F I C AT I O N O F E P I L E P T I C S E I Z U R E S B Y M E A N S O F S I G N A L P R O C E S S I N G
Based on Z-Flores, E., [START_REF] Z-Flores | Regularity and Matching Pursuit feature extraction for the detection of epileptic seizures[END_REF]. Regularity and Matching Pursuit feature extraction for the detection of epileptic seizures. Journal of Neuroscience Methods, 266:107-125

3.1 I N T R O D U C T I O N
E pilepsy is a type of neurological disorder that it is characterized by an enduring predisposition to generate unprovoked seizures, each occurring more than 24 hours apart [START_REF] Fisher | ILAE official report: a practical clinical definition of epilepsy[END_REF]. Normal brain activity is considered to be non-synchronous, but during epileptic seizures a group of neurons begins firing in an abnormal, excessive and synchronized manner. This is opposed to what normally happens, when an excitatory neuron fires it becomes resilient to firing again for a short period of time [START_REF] Else | Disorders of the Adrenal Cortex, chapter 21[END_REF]. There are approximately 65 million people worldwide living with epilepsy [START_REF] Thurman | Standards for epidemiologic studies and surveillance of epilepsy[END_REF]. It varies from region to region, for instance in the United States the annual incidence of epilepsy is 48 per 100,000 inhabitants, whereas the prevalence approximates 710 per 100,000 [START_REF] Hirtz | How common are the "common" neurologic disorders[END_REF]. Diagnosing epilepsy after a single unprovoked seizure, when there is a high risk for recurrence, may or may not lead to a decision to initiate treatment by the epileptologist. Nonetheless, the diagnosis helps the physician assess the balance between the possible avoidance of a second seizure and the associated risks of actually occurring. According to [START_REF] Eadie | Shortcomings in the current treatment of epilepsy[END_REF] Classification of Epileptic seizures by means of signal processing about 70% of the cases can be controlled by medication, but even then the patient might suffer negative side effects.

Automated systems can help develop an appropriate therapy plan that could eventually improve the balance of the associated risks and potentially enhance the patient's quality of life [START_REF] Fisher | ILAE official report: a practical clinical definition of epilepsy[END_REF]. Such systems would be able to detect a bodily response that matches the epileptic symptoms, and distinguish them from signals present during a patient's regular activity. There are non-invasive methods that can sense epileptic states before they manifest, as well as to detect the seizure physiologic state, which is also known as the Ictal state. One such method is the Electroencephalogram (EEG), which records electrical brain activity along the scalp. Other methods are also available, such as the Electrocorticogram (ECoG) that is recorded by electrodes that are inserted trough the skull [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF][START_REF] Sotelo | Seizure states identification in experimental epilepsy using gabor atom analysis[END_REF]. ECoG provides more localized readings but can be undesirable for the patient since it is invasive [START_REF] Ball | Signal quality of simultaneously recorded invasive and noninvasive EEG[END_REF].

EEGs are very popular because they have several advantages compared to other methods, as stated before in chapter 2 [START_REF] Tzallas | Epileptic seizure detection in EEGs using time-frequency analysis[END_REF][START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF][START_REF] Güler | Adaptive neuro-fuzzy inference system for classification of EEG signals using wavelet coefficients[END_REF][START_REF] Rajendra Acharya | Use of principal component analysis for automatic classifica-Bibliography tion of epileptic EEG activities in wavelet framework[END_REF]Acharya et al., 2012a;[START_REF] Guler | Recurrent neural networks employing Lyapunov exponents for EEG signals classification[END_REF][START_REF] Ahammad | Detection of epileptic seizure event and onset using EEG[END_REF][START_REF] Orhan | EEG signals classification using the K-means clustering and a multilayer perceptron neural network model[END_REF][START_REF] Guo | Epileptic seizure detection using multiwavelet transform based approximate entropy and artificial neural networks[END_REF][START_REF] Kamath | Analysis of EEG Dynamics in Epileptic Patients and Healthy Subjects Using Hilbert Transform Scatter Plots[END_REF][START_REF] Lima | Tackling EEG signal classification with least squares support vector machines: A sensitivity analysis study[END_REF], these are: (1) hardware costs are relatively low; (2) electrodes can be positioned based on application needs; (3) it does not expose the patient to high-intensity magnetic fields like magnetoencephalography; and (4) it is non-invasive, relative with other methods. There are some drawbacks as well, like the inherent presence of noise in the signals. Nevertheless, the present work focuses on the use of EEG recordings for automatic epilepsy analysis.

Epileptic seizures can be identified in EEG signals by experienced physicians, but automatic recognition is still not a trivial task. Research for the development of computational systems that perform these tasks usually focuses on three fronts. Either by exploiting the morphology of the signals recorded during the epileptic crisis [START_REF] Yadav | Morphology-based automatic seizure detector for intracerebral EEG recordings[END_REF], by applying analytic or numerical methods [START_REF] Ramgopal | Seizure detection, seizure prediction, and closed-loop warning systems in epilepsy[END_REF], or a combination of the two [START_REF] Tzallas | Epileptic seizure detection in EEGs using time-frequency analysis[END_REF][START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF]Güler 3.1 Introduction and Ubeyli, 2005;[START_REF] Rajendra Acharya | Use of principal component analysis for automatic classifica-Bibliography tion of epileptic EEG activities in wavelet framework[END_REF]Acharya et al., 2012a;[START_REF] Guler | Recurrent neural networks employing Lyapunov exponents for EEG signals classification[END_REF][START_REF] Ahammad | Detection of epileptic seizure event and onset using EEG[END_REF][START_REF] Orhan | EEG signals classification using the K-means clustering and a multilayer perceptron neural network model[END_REF][START_REF] Guo | Epileptic seizure detection using multiwavelet transform based approximate entropy and artificial neural networks[END_REF][START_REF] Kamath | Analysis of EEG Dynamics in Epileptic Patients and Healthy Subjects Using Hilbert Transform Scatter Plots[END_REF][START_REF] Lima | Tackling EEG signal classification with least squares support vector machines: A sensitivity analysis study[END_REF][START_REF] Murugavel | An Optimized Extreme Learning Machine for Epileptic Seizure Detection[END_REF][START_REF] Divya | Classification of EEG Signal for Epileptic Seizure Detection using EMD and ELM[END_REF][START_REF] Kumar | Epileptic seizure detection using DWT based fuzzy approximate entropy and support vector machine[END_REF].

The design of BCI includes several sections, these being: brain recording method, pre-processing, feature extraction, classification, translation into commands and feedback. As stated earlier in this manuscript, we are going to focus in the first four sections, putting special importance in the feature extraction, which we consider one of the most critical parts for the efficiency of a BCI. Thus, we propose a new methodology for feature extraction, which incorporates two powerful signal analysis tools to construct specialized domain features, namely Hölderian regularity [START_REF] Mallat | Singularity detection and processing with wavelets[END_REF] and the Matching Pursuit (MP) algorithm [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF]. Each of these tools tackles the posed problem from different perspectives. The former incorporates a local signal regularity measure while the latter employs a time-frequency analysis. These two methods are related, since MP can be used for a global regularity measure. Both of these techniques are considered to be highly nonlinear in their core design, thus the proposed methodology is well suited to analyze a nonlinear process, like the one that produces an epileptic seizure. Moreover, we combine these nonlinear features with much simpler features computed as statistics of the raw signals in time domain, which have been shown to be useful in related tasks [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF].

After the proposed feature extraction process, a classifier is used to solve the automatic detection problem. Experimental results achieve a perfect performance for three of the four tested problem instances. Moreover, on the fourth test case, the most complex, classification accuracy is 97.6%, competitive with the state-of-the-art [START_REF] Tzallas | Epileptic seizure detection in EEGs using time-frequency analysis[END_REF][START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF][START_REF] Güler | Adaptive neuro-fuzzy inference system for classification of EEG signals using wavelet coefficients[END_REF][START_REF] Rajendra Acharya | Use of principal component analysis for automatic classifica-Bibliography tion of epileptic EEG activities in wavelet framework[END_REF]Acharya et al., 2012a;[START_REF] Guler | Recurrent neural networks employing Lyapunov exponents for EEG signals classification[END_REF][START_REF] Ahammad | Detection of epileptic seizure event and onset using EEG[END_REF][START_REF] Orhan | EEG signals classification using the K-means clustering and a multilayer perceptron neural network model[END_REF][START_REF] Guo | Epileptic seizure detection using multiwavelet transform based approximate entropy and artificial neural networks[END_REF][START_REF] Kamath | Analysis of EEG Dynamics in Epileptic Patients and Healthy Subjects Using Hilbert Transform Scatter Plots[END_REF][START_REF] Lima | Tackling EEG signal classification with least squares support vector machines: A sensitivity analysis study[END_REF][START_REF] Tzallas | Automatic seizure detection based on time-frequency analysis and artificial neural networks[END_REF][START_REF] Guler | Multiclass Support Vector Machines for EEG-Signals Classification[END_REF]. All of these conclusions are derived from a rigorous experimental validation and comprehensive statistical tests.

In the following subsections a brief description of epileptic states is presented, as well as a short review of previous works related to techniques employed for EEG analysis.

Epileptic states

Epiletic seizures are unintentional and disruptive events of mental activity that impair a patient's motor, sensorial and autonomic functions. Seizures develop over several states [START_REF] Franaszczuk | Time-frequency analysis using the matching pursuit algorithm applied to seizures originating from the mesial temporal lobe[END_REF] The Pre-Ictal state refers to the time period before the seizure symptomatology is evident. Here, the signal amplitude is higher than in the Basal state, with the presence of spikes and transitory activity, also called recruiting rhythms [START_REF] Kohsaka | Widespread activation of the brainstem preceding the recruiting rhythm in human epilepsies[END_REF]. The Ictal state is the prominent period where the symptomatology is evident. The EEG signal magnitude is higher than in any other state and displays a dominant low frequency rhythm. The Post-Ictal state refers to the span of time when an altered state of consciousness exists after the active portion of the seizure ended. This period is variable and depends of the seizure duration. The overall amplitude of the signal decreases and the frequency increases. Several symptoms appear during this state, like migraines, depression and a loss of motor functions [START_REF] Fisher | The postictal state: a neglected entity in the management of epilepsy[END_REF]. Finally, the Inter-Ictal state refers to the period of time between seizures.

The problem of automatically detecting the states of epileptic seizures can be posed in different ways [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF][START_REF] Sotelo | Seizure states identification in experimental epilepsy using gabor atom analysis[END_REF]. Here, we recognize that the identification of Ictal states is an important task, as a way to prevent or prepare for a seizure. Moreover, the effects on the patient are most severe in this state. Consequentially, this work focuses on the automatic recognition of Ictal activity among other EEG readings. For this work, we use the Bonn data set [START_REF] Andrzejak | Indications of nonlinear deterministic and finitedimensional structures in time series of brain electrical activity: de-Bibliography pendence on recording region and brain state[END_REF], a pub-3.1 Introduction lic database that contains several types of EEG recordings, to test the proposed approach.

Previous work

The automatic classification of epileptic seizures has received much attention over recent years. The problem has been handled from many perspectives, focusing on different aspects of the problem, while using a variety of signal processing and pattern recognition paradigms. Since the number of reported studies is large, here we review the most recent and relevant examples. Moreover, for comparative reasons, we mostly limit to works based on the Bonn data set [START_REF] Andrzejak | Indications of nonlinear deterministic and finitedimensional structures in time series of brain electrical activity: de-Bibliography pendence on recording region and brain state[END_REF]. [START_REF] Acharya | Automated EEG analysis of epilepsy: A review[END_REF] present an extensive survey that summarizes a variety of different methods on how this problem has been addressed. In that survey, the authors also summarize the classification accuracy from each reviewed work, comparing their quality and experimental work. Even though some works achieve strong performance, there is still room for developing new domain-specific patterns recognition methods. Moreover, the insights gathered in epileptic states detection might be extended to other areas of EEG analysis or application domains [START_REF] Vezard | Classification of EEG signals by evolutionary algorithm[END_REF].

Before attempting to solve the classification task, feature extraction must be performed. The simplest features are extracted in the time domain. For instance, [START_REF] Xie | Dynamic principal component analysis with nonoverlapping moving window and its applications to epileptic EEG classification[END_REF] proposed an improved Dynamic Principal Component Analysis (DPCA) by means of a nonoverlapping moving window. Because they show a highly effective feature extraction, a simple nearest neighbor classifier produces good results. This shows that one of the most difficult tasks is deriving an optimum feature extraction method. Recently, [START_REF] Kamath | Analysis of EEG Dynamics in Epileptic Patients and Healthy Subjects Using Hilbert Transform Scatter Plots[END_REF] uses the Hilbert Transform (HT) as a method for a time to time-domain transformation, where the frequency is expressed as a rate of phase change, a dispersion entropy measure, dispersion complexity and forbidden count. These features are then used to distinguish between the Ictal and Inter-Ictal states.

Classification of Epileptic seizures by means of signal processing

A more common approach is to use time-frequency analysis for epilepsy detection, such as wavelet analysis (Acharya et al., 2012a;[START_REF] Ahammad | Detection of epileptic seizure event and onset using EEG[END_REF][START_REF] Orhan | EEG signals classification using the K-means clustering and a multilayer perceptron neural network model[END_REF][START_REF] Tzallas | Epileptic seizure detection in EEGs using time-frequency analysis[END_REF][START_REF] Guo | Epileptic seizure detection using multiwavelet transform based approximate entropy and artificial neural networks[END_REF][START_REF] Chen | A random forest model based classification scheme for neonatal amplitudeintegrated EEG[END_REF][START_REF] Nunes | EEG signal classification for epilepsy diagnosis via optimum path forest -A systematic assessment[END_REF][START_REF] Kumar | Epileptic seizure detection using DWT based fuzzy approximate entropy and support vector machine[END_REF][START_REF] Gandhi | Discrete harmony search based expert model for epileptic seizure detection in electroencephalography[END_REF][START_REF] Zainuddin | Reliable epileptic seizure detection using an improved wavelet neural network[END_REF][START_REF] Murugavel | An Optimized Extreme Learning Machine for Epileptic Seizure Detection[END_REF][START_REF] Kumari | Seizure Detection in EEG Using Time Frequency Anlysis and SVM[END_REF]. A related technique is used by [START_REF] Kovacs | On application of rational discrete short time fourier transform in epileptic seizure classification[END_REF], the Short Time Fourier Transform (STFT), achieving good results and confirming that well established methods are still relevant in this domain. Packet Wavelet Decomposition is also used in this domain, since it is a variant of the same underlying concept (Rajendra [START_REF] Acharya | Automated diagnosis of epileptic EEG using entropies[END_REF]. In [START_REF] Faust | Wavelet-based EEG Processing for Computer-aided Seizure Detection and Epilepsy Diagnosis[END_REF], the authors present a recent comprehensive review of wavelet-based methods used to solve the problem of automatic seizure detection. It is clear that the robust time-frequency analysis provided by the wavelet decomposition is a popular technique among the reviewed literature, because its properties are useful in many areas of signal processing.

Other nonlinear methods not directly related to time-frequency analysis are also used in this domain, [START_REF] Acharya | Automated EEG analysis of epilepsy: A review[END_REF] suggests that given the nonlinearity of EEG signals these techniques might yield better results, although simpler methods sometimes can outperform them. For example, some works [START_REF] Bajaj | EEG signal classification using empirical mode decomposition and support vector machine[END_REF][START_REF] Alam | Detection of seizure and epilepsy using higher order statistics in the EMD domain[END_REF][START_REF] Divya | Classification of EEG Signal for Epileptic Seizure Detection using EMD and ELM[END_REF] use the Empirical Mode Decomposition (EMD) to decompose EEG signals into a set of intrinsic mode functions, a method that performs well with highly non-stationary and nonlinear signals, while using a variety of classifiers. [START_REF] Guler | Recurrent neural networks employing Lyapunov exponents for EEG signals classification[END_REF]) make use of another nonlinear approach, the Lyapunov exponent approximation as a measure of signal chaosity, with promising results. The use of different type of entropies like Kolmogorov-Sinai Entropy(KSE), Approximate Entropy (ApEn), Sample Entropy (SampEn), Spectral Entropy (SE) among others have been the subject of study in this domain (Acharya et al., 2012a;[START_REF] Nicolaou | Detection of epileptic electroencephalogram based on Permutation Entropy and Support Vector Machines[END_REF][START_REF] Acharya | Application of Non-Linear and Wavelet Based Features for the Automated Identification of Epileptic Eeg Signals[END_REF][START_REF] Martis | Application of Intrinsic Time-Scale Decomposition (Itd) To Eeg Signals for Automated Seizure Prediction[END_REF]. In [START_REF] Acharya | Application of entropies for automated diagnosis of epilepsy using EEG signals: A review[END_REF], the authors survey recent works that apply entropy methods for epilepsy detection. Other related methods include works that use the High Order Spectra (HOS), employing the third order cumulant as a feature [START_REF] Acharya | Application of Non-Linear and Wavelet Based Features for the Automated Identification of Epileptic Eeg Signals[END_REF], Recurrence Quantification 3.2 Materials and methods Analysis (RQA), an analysis based on the topologies derived from recurrence plots in dynamical systems [START_REF] Niknazar | A new framework based on recurrence quantification analysis for epileptic seizure detection[END_REF], and the Higuchi Fractal Dimension for measuring signal complexity through fractal concepts [START_REF] Martis | Application of Intrinsic Time-Scale Decomposition (Itd) To Eeg Signals for Automated Seizure Prediction[END_REF].

The above cited works recognize that a critical step in identifying a brain-related phenomenon using EEG recordings is the feature extraction phase, the main emphasis of the current work. In the following section we present our proposed feature extraction approach and afterwards we evaluate our classification results. Moreover, we will compare our results with those previously reported in the literature, showing that our proposal achieves state-of-the-art performance.

The reminder of this chapter is organized as follows. In Section 3.2 the employed EEG data set is described, the proposed feature extraction methods are presented and the classification task is posed. In Section 3.3 we discuss the experimental details and its results are presented in Section 3.4. Finally, in Section 3.6 we present our chapter conclusions.

M AT E R I A L S A N D M E T H O D S

Epilepsy EEG Data set

The data set used in this work was published by the Bonn University [START_REF] Andrzejak | Indications of nonlinear deterministic and finitedimensional structures in time series of brain electrical activity: de-Bibliography pendence on recording region and brain state[END_REF]. The data set includes five subsets of signals (denoted as Z, O, N, F and S), each containing 100 single-channel EEG segments with a duration of 23.6s. All segments were recorded using an amplifier system, digitized with a sampling rate of 173.61 Hz and 12-bit A/D resolution, and filtered using a 0.53-40 Hz (12 dB/octave) band pass filter. The normal (Basal) segments (Sets Z and O) were taken from five healthy subjects. The standard surface electrode placement scheme (the international 10-20 system) was used to obtain the EEG from the healthy cases. Volunteers were relaxed in an awake state with eyes opened (Z) and eyes closed (O), respectively. Both states exhibit different characteristics, the EEG readings with the eyes closed show a higher magnitude than when the eyes are opened, as well as the presence of alpha waves Classification of Epileptic seizures by means of signal processing [START_REF] Ball | Signal quality of simultaneously recorded invasive and noninvasive EEG[END_REF]. Both the Inter-Ictal and Ictal segments were obtained from five epileptic patients. The Inter-Ictal segments were recorded during seizure-free intervals from the depth electrodes that were implanted into the hippocampal formations (Set N) and from the epileptogenic zone (Set F). The Ictal segments (Set S) were recorded from all sites exhibiting Ictal activity using depth electrodes and also from strip electrodes that were implanted into the lateral and basal regions of the neocortex. For convenience, we refer to subsets Z, O, N, F and S, as class A, B, C, D and E respectively; see Table 3.1.

Proposed feature extraction

An appropriate feature extraction is one of the most important tasks in designing classification systems. We propose a feature extraction approach, that to the authors knowledge, has not been used before in the problem domain studied here. In particular, our approach is based on combining the MP algorithm [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF] and signal regularity analysis [START_REF] Jaffard | Wavelet methods for pointwise regularity and local oscillations of functions[END_REF]. The proposed system is depicted in Fig-ure 3.1. Some versions of each method have been used before trying to either analyze or classify EEG recordings, specially using the MP algorithm [START_REF] Durka | Adaptive time-frequency parametrization of epileptic spikes[END_REF]. However, this is the first time that they are used simultaneously to undertake the epilepsy detection problem. The motivation for the simultaneous use of both techniques is that together they can complement the level of detail extracted from the signal, from a broader to a local characterization. This span of detail might produce a robust feature extraction outcome, and its effectiveness is actually validated by experimental results.

Durka has been efficiently using MP decomposition for EEG signal analysis for several years [START_REF] Durka | Adaptive time-frequency parametrization of epileptic spikes[END_REF][START_REF] Franaszczuk | Time-frequency analysis using the matching pursuit algorithm applied to seizures originating from the mesial temporal lobe[END_REF][START_REF] Durka | Stochastic timefrequency dictionaries for matching pursuit[END_REF][START_REF] Durka | Multichannel matching pursuit and EEG inverse solutions[END_REF]. In particular, he worked on EEG signals derived from subjects exhibiting an epileptic condition with competent results [START_REF] Durka | Adaptive time-frequency parametrization of epileptic spikes[END_REF]. More recently, additional research has been focused on extending Durka's work, like improving the residual inference using source deflation [START_REF] Wu | Matching pursuit and source deflation for sparse EEG/MEG dipole moment estimation[END_REF] or by imposing a restricted dictionary on the MP [START_REF] Picot | Detection of cortical slow waves in the sleep EEG using a modified matching pursuit method with a restricted dictionary[END_REF], among other proposals [START_REF] Bénar | Consensus Matching Pursuit for multi-trial EEG signals[END_REF]. It is worth mentioning that the measures derived from the MP decomposition used in our proposal differ from Durka's works, as seen further in Section 3.2.2.2. MP is effective at detecting epileptic spikes due to its ability in finding base functions closely similar to the analyzed signal in time-frequency space [START_REF] Durka | Adaptive time-frequency parametrization of epileptic spikes[END_REF]. In our work, we use a simple approach: a canonical MP (as originally proposed by Stéphane Mallat [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF]) decomposition is performed over an EEG recording (no additional complexity is added) and several statistics are extracted from the obtained decomposition.

Similarly, regularity of EEG signals has also been studied before [START_REF] Mikaili | Assessment of the complexity/regularity of transient brain waves (eeg) during sleep, based on wavelet theory and the concept of entropy[END_REF][START_REF] Popivanov | Multifractality of decomposed EEG during imaginary and real visual-motor tracking[END_REF][START_REF] Natarajan | Nonlinear analysis of EEG signals at different mental states[END_REF][START_REF] Mathuvanesan | Performance Analysis Of Singularity And Irregular Detection In Human Health Monitoring Using Lipschitz Exponent Function[END_REF], however this topic is broad and there are several ways to measure a signal regularity; including Shannon entropy, spectral entropy, ApEn, Lempel-Ziv complexity and Higuchi fractal, among others. Some works have focused on ApEn concerning EEG analysis [START_REF] Fan | Comparison of eeg approximate entropy and complexity measures of depth of anaesthesia during inhalational general anaesthesia[END_REF][START_REF] Abásolo | Analysis of regularity in the {EEG} background activity of alzheimer's disease patients with approximate entropy[END_REF][START_REF] Chuckravanen | Approximate Entropy as a Measure of Cognitive Fatigue: An EEG Pilot Study[END_REF]. Closely related to signal regularity, there is also some research for EEG analysis by means of mutifractal theory. For instance, a processing Figure 3.1: Proposed system for automatic epileptic seizures detection. tool for extracting the regularity spectrum of a time series, the so called Multifractal Detrended Fluctuation Analysis (MFDFA) is employed in different works, by approximating the Hurst exponent [START_REF] Figliola | Study of EEG brain maturation signals with multifractal detrended fluctuation analysis[END_REF][START_REF] Zorick | Multifractal Detrended Fluctuation Analysis of Human EEG: Preliminary Investigation and Comparison with the Wavelet Transform Modulus Maxima Technique[END_REF][START_REF] Kantelhardt | Multifractal detrended fluctuation analysis of nonstationary time series[END_REF]. Another method for extracting the fractal dimension of a signal, an approach for regularity analysis, is the Wavelet Transform Modulus Maxima (WTMM), used in several works [START_REF] Ma | Sleep-stage characterization by nonlinear EEG analysis using wavelet-based multifractal formalism[END_REF][START_REF] Song | Fluctuation Dynamics in Electroencephalogram Time Series[END_REF][START_REF] Dick | Potentialities of the wavelet and multifractal techniques to evaluate changes in the functional state of the human brain[END_REF][START_REF] Popivanov | Multifractality of decomposed EEG during imaginary and real visual-motor tracking[END_REF]. However, this work represents the first time that the approach by [START_REF] Jaffard | Wavelet techniques in multifractal analysis[END_REF]) is employed to solve the problem of detecting epileptic states.

In summary, the MP algorithm and regularity analysis complement each other by accessing different layers of information embedded in the signals. Hölderian exponent calculation works as a local regularity measure while MP acts as a global regularity measure. In the following subsections we formally describe both feature extraction methods.

Hölder exponent

In general, the regularity of a signal can be described as the characterization of the singularities it contains. These singularities often carry valuable information, specially for nonlinear and non-stationary signals [START_REF] Mallat | A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way[END_REF].

There are several methods to calculate the regularity of a nonstationary time series, either of form local or pointwise [START_REF] Mallat | A Wavelet Tour of Signal Processing, Third Edition: The Sparse Way[END_REF].

The precise calculus of them requires of complex numerical methods which are not practically viable and sometimes are not even possible to calculate it. Rather, an approximation is commonly used, where the computational resources are feasible and its characterization is quite accurate. In this work we make use of the regularity measure given by the pointwise Hölder exponent, approximated through a discrete wavelet decomposition. The pointwise Hölder exponent of a function is defined as:

Definition 1 Let f : R d → R, s ∈ R + * \N and x 0 ∈ R d . Then, f ∈ C s (x 0 ) ⇔ ∃η ∈ R + * , a polynomial P of degree < s and a constant c such that ∀x ∈ B(x 0 , η), |f (x) -P (x -x 0 )| ≤ c|x -x 0 | s (3.1)
The pointwise Hölder exponent of f at x 0 is α p = sup s {f ∈ C s (x 0 )}.

In some cases, it is necessary to have information of the regularity of a signal, not in a point, but in a neighborhood of that point, namely a local Hölder exponent. Its definition is: Definition 2 Let f be a function on the neighborhood of x 0 . Let {I n } n∈N be a decreasing sequence of open intervals converging toward x 0 . The local Hölder exponent of the function f at x 0 is

α l (x 0 ) = sup n∈N [α f (I n )] = lim n→+∞ α f (I n ) (3.2) where α f (Ω) = sup{s : f ∈ C s (Ω)} (3.3) being f : Ω → R a function, with Ω a open subset of R.
There are, as well, different techniques to estimate the pointwise regularity by considering Hölder spaces. One of them is by using the oscillation method; a mechanism that closely follows the Hölder exponent definition [START_REF] Jaffard | Wavelet methods for pointwise regularity and local oscillations of functions[END_REF]. Another method is the use of discrete wavelet decomposition (DWT). Wavelet analysis can be used to compute an approximation of the Hölder exponent by estimating the decay rate of its wavelet coefficients versus the scales, corresponding to Classification of Epileptic seizures by means of signal processing Figure 3.2: Regression calculated over a point of the signal. Left image shows a dyadic wavelet decomposition, and the right image display the actual regression calculated over the point t 0 , where each dot corresponds to each log 2 of the wavelet coefficient magnitude located approximately above t 0 . the wavelets localized near the considered point [START_REF] Jaffard | Wavelet techniques in multifractal analysis[END_REF]. Here, we choose the latter method as the preferred way to achieve the regularity estimation of our signals, and specifically using a dyadic decomposition methodology.

Being Ψ a mother wavelet of the classical form {Ψ j,k } j,k that makes an orthonormal base of L 2 , wavelet coefficients are denoted as c j,k of f , with the latter being a uniformly Hölderian function, where j corresponds to the scales and k corresponds to the temporal location.

Theorem 1 Let α be the pointwise Hölder exponent of f at point t 0 , then exists a constant c > 0 such that the wavelet coefficient satisfy

|c j,k | ≤ c2 -j(α+ 1 2 ) (1 + |2 j t 0 -k|) α ∀j, k ∈ Z 2 ;
(3.4) Equation 3.4 states that the wavelet coefficients decrease in absolute value by an amount that depends on the Hölder exponent. From this theorem, if we make the hypothesis that the global and local exponents are the same then we are only interested in indices j, k such that |k -2 j t 0 | < c for the point t 0 . Then, Equation 3.5 implies the existence of coefficients of the order of 2 -j(α+ 1 2 ) . This simplifying assumption is satisfied if and only if the local exponent is equal to the pointwise exponent at t 0 [START_REF] Véhel | Signal and Image processing with Fra-cLab[END_REF].

reciprocally, if ∀j, k ∈ Z 2 we have |c j,k | ≤ c2 -j(α+ 1 2 ) (1 + |2 j t 0 -k|) α ′ , (3.5) for an α ′ < α the Hölder exponent of f at t 0 is α. A function f is uniformly Hölderian is there is ε > 0 such that f ∈ C ε (R).
Under this assumption, an estimation of the Hölder exponent is obtained by means of the slope p of the log 2 |c j,k | regression with j : α(n, t 0 ) = -p -1 2 , n = ⌊log 2 (N )⌋ being the number of decomposition levels and N the length of the signal.

Definition 3

In each point t 0 of the signal, decomposed in n scales, the regularity estimation is given by

α(n, t 0 ) = - 1 2 -K n n ∑ j=1 s j log 2 |c j,k | , (3.6)
Classification of Epileptic seizures by means of signal processing with K n = 12 n(n-1)(n+1) and s j = j -n+1 2 . c j,k are the wavelet coefficients located over t 0 . The value of k is given by ⌊ t 0 +1 2 n-j+1 ⌋.

Figure 3.2 illustrates this method [START_REF] Legrand | Débruitage et interpolation par analyse de la régularité Hölderienne. Application à la modélisation du frottement pneumatique-chaussée[END_REF].

Figure 3.3 presents an example of regularity computation on a time series. The input signal is composed by the superposition of two waveforms with different amplitudes, frequencies and starting times. The regularity measure characterizes the signal singularities. The amplitude given by α corresponds to the regularity of the signal around a given point. Clearly, the low frequency waveform is more regular than the high frequency one. The hard transition between both waveforms is also captured by the presence of highly irregular spikes.

Matching Pursuit

Matching pursuit refers to a family of greedy algorithms that compute the best nonlinear approximation of a signal (Hussain and Shawetaylor, 2009). One of these algorithms was introduced by [START_REF] Mallat | Matching pursuits with time-frequency dictionaries[END_REF], with the goal of decomposing a signal into a linear expansion of waveforms that are selected from a redundant dictionary of functions.

An unknown signal can be expanded in terms of functions g γn , called time-frequency atoms [START_REF] Chen | Atomic decomposition by basis pursuit[END_REF], given by

f (t) = +∞ ∑ n=-∞ a n g γn (t) , (3.7)
where a n is an expansion coefficient. In this way f (t) can be explained using functions g γn from a dictionary D. The family D= [(g γ (t)) γ∈Γ ], where the index γ is an element of the set Γ = R + × R 2 , it is highly redundant. In general, a family of time-frequency atoms can be generated by scaling, translating and modulating a single window function Let f ∈ H, where H is a Hilbert space, and in our case H = L 2 (R) (space of complex valued functions), the MP algorithm computes a linear expansion of f over a set of vectors selected from D, by successive approximations of f with orthogonal projections on elements of D. Let g γ0 ∈ D, the vector f can be decomposed into

g γ (t) = 1 √ s g ( t -u s ) e iξt , ( 3
f = ⟨f , g γ0 ⟩g γ0 + Rf , (3.9)
where Rf is the residual vector after approximating f in the direction of g γ0 . The MP algorithm sub-decomposes the residue Rf by projecting it on a vector of D that best matches Rf , as it was done for f .

In general, after m iterations MP decomposes a signal f into

f = m-1 ∑ n=0 ⟨R n f , g γn ⟩g γn + R m f (3.10)
or, in terms of energy,

∥f ∥ 2 = m-1 ∑ n=0 |⟨R n f , g γn ⟩| 2 + ∥R m f ∥ 2 .
(3.11)

In Figure 3.4, a visualization of the MP decomposition into timefrequency atoms of one signal is shown, by means of a Heisenberg boxes. Notice that the raw signal is the same used in the example of Figure 3.3. The boxes show specific waveform characteristics: its frequencies, positions, durations and amplitudes, by means of the atoms found during the decomposition. The color scale represents the relative atoms amplitude according with the input signal. The high frequency spikes that exist at the transition of both waveforms are captured by the MP algorithm.

Proposed feature sets

The main contribution of this paper is the exploitation of timefrequency content and regularity analysis of EEG signals for the detection of the Ictal state during an epileptic seizure. The pointwise Hölder exponent and the MP decomposition are not directly dependent on signal amplitude. This is important in real applications, since it is well known that EEG recordings can vary in amplitude as result of changes in the instrumentation, noise induction or skull thickness (Usakli, 2010). Usually, EEG recording systems require a calibration process which can be problematic [START_REF] Grizou | Calibration-Free BCI Based Control[END_REF]. Therefore, frequency and regularity-based techniques could help alleviate the previously mentioned complications. Nonetheless, it is evident that the Ictal state presents high amplitude spikes in the time-domain, a characteristic that might help simplify the detection problem. Therefore, we recognize that some useful, and probably necessary, information can only be obtained by explicitly considering the time-domain features.

Thus, we propose two sets of features extracted from each EEG epoch, which are evaluated and compared in our experimental work: (a) Hölder regularity and MP decomposition features (4 features); (b) Hölder, MP and statistical time-domain features (10 features). Additionally, we include an additional set which is automatically chosen by searching for the optimum combination of the proposed 10 feature set, using a meta-heuristic combinatorial optimization algorithm. All of these feature sets are summarized in Table 3.2 and described in the following subsections.

Hölderian regularity and MP decomposition (4 features set)

These features are presented in the second column of Table 3

(H) = (µ 1/2 ) i (|H i -(µ 1/2 ) j (H j )|) , (3.12)
where µ 1/2 is the median, i and j are indexes of samples in H.

For the MP decomposition, we propose two basic features: the Gabor Atom Density (GAD), first introduced by [START_REF] Koubeissi | Analysis of dynamics and propagation of parietal cingulate seizures with secondary mesial temporal involvement[END_REF], and the frequencies mean µ(F) of all atoms, where F is the atom frequencies vector. The frequencies mean is calculated with

µ(F) = 1 m m ∑ n=0 F n (g γn ) , (3.13)
where F n (g γn ) is the frequency of the n atom found during the decomposition.

The GAD is defined as the number of atoms m retrieved during the decomposition, divided by the size of the reconstructed time-frequency space. For each window, the range of time and frequency are, respectively, R t = N (1/F s ); R f = F s /2, where N is the number of points in the window and F s the sampling frequency. The density of atoms over the space is

GAD = m R t R f = 2m N .
(3.14)

Classification of Epileptic seizures by means of signal processing

Hölder, MP decomposition and time-domain features (10 features set)

This set is presented in the third column of Table 3.2. These include the regularity and MP features, as well as the mean amplitude of the MP decomposition

µ(A) = 1 m m ∑ n=0
A n (g γn ) ,

(3.15)

where A n (g γn ) is the amplitude of the n atom. Additionally, five additional features which were successfully used in [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF] for epilepsy state identification are considered. The proposed features are basic statistical measures calculated explicitly in the time domain over the raw epochs E, with E being the epoch vector in the time domain. They are: the mean µ(E), median µ 1/2 (E), standard deviation σ (E), skewness γ 1 (E) and kurtosis γ 2 (E).

Automatic feature selection with GA

An additional feature set is proposed, which is automatically chosen as a subset of the complete 10 feature set. To perform this automatic feature selection procedure we employ a Genetic Algorithm (GA) [START_REF] Goldberg | Genetic Algorithms in Search, Optimization and Machine Learning[END_REF], a popular algorithm from the field of Evolutionary Computation (EC) [START_REF] Eiben | Introduction to Evolutionary Computing[END_REF].

GAs are well established stochastic population-based optimization algorithms, where a large set of candidate solutions compete to survive in each iteration (generation) of the search process. Each solution is ranked by a fitness function that is defined based on the problem domain. For instance, for a classification task fitness can be based on accuracy or total classification error. Each candidate solution (individual) is encoded using a domain specific representation called a genotype, that the search process can manipulate. The genotype is then decoded into a problem domain representation (phenotype), for which fitness can be computed. By means of the search (genetic) operators, solutions evolve by randomly swapping genetic material between pairs of solutions (crossover) or by randomly altering parts of existing solu-

GAD GAD µ(F) µ(F) MP statistics µ(A) µ(E) µ 1/2 (E) σ (E) γ 1 (E)
Raw signal basic statistics

γ 2 (E)
tions (mutation). The fitness function is the main determining factor for choosing which solutions will be subjected to the search operators, and to determine which solutions will be kept in the population or will be discarded before the next iteration. This produces a selective pressure towards high-performing solutions. Normally, the search is stopped when a certain number of iterations is reached, and the best solution found at that point is returned.

GAs have been widely used for automatic feature selection [START_REF] Sun | Object detection using feature subset selection[END_REF][START_REF] Lin | Study of image retrieval and classification based on adaptive features using genetic algorithm feature selection[END_REF], in particular given that the canonical binary representation is well suited to represent solutions in this general task. Section 3.3.6 provides further details of our implementation and the resulting feature set.

Classification

Once the feature extraction process has been done and consequentially a set of features has been extracted, the classification task must be performed, depicted in the rightmost block of Figure 3.1. This task Classification of Epileptic seizures by means of signal processing solves the problem of assigning a class label to unknown data, based on the set of known features and corresponding class labels, basically a supervised learning problem [START_REF] Duda | Pattern Classification (2Nd Edition)[END_REF]. Machine learning literature includes a wide variety of supervised classification techniques, from simple Bayes classifiers, to more complex methods like hidden Markov models [START_REF] Alpaydin | Introduction to Machine Learning[END_REF] or genetic programming [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF]. In this work, we use the Random Forests™(RF) classifier by [START_REF] Breiman | Random forests[END_REF], part of the family of decision trees classifiers [START_REF] Duda | Pattern Classification (2Nd Edition)[END_REF]. Decision trees creates a model that predicts the class label of an unknown feature vector based on rules derived from the training set and expressed as a tree where each internal node performs a decision based on a particular input feature, and each leaf is labeled with a class or a probability distribution over the classes [START_REF] Rokach | Data Mining with Decision Trees: Theroy and Applications[END_REF]. RF creates a set of decision trees in an ensemble scheme, where a set of weak models work together to build a stronger classifier. The main algorithm parameter is the number of trees used.

This method exhibits several advantages that have made it popular in many domains [START_REF] Sculley | Results from a Semi-Supervised Feature Learning Competition[END_REF], including EEG analysis [START_REF] Chen | A random forest model based classification scheme for neonatal amplitudeintegrated EEG[END_REF]. It is noteworthy to mention that to the author's knowledge RF has not been applied to the Bonn data set before [START_REF] Acharya | Automated EEG analysis of epilepsy: A review[END_REF]. The advantages of RF are [START_REF] Breiman | Random forests[END_REF]: (1) accuracy on test data is superior to many classifiers when feature selection is optimal; (2) it performs implicit feature selection; (3) it can effectively estimate missing data while maintaining high accuracy; and (4) it has shown resiliency to over-fitting and class imbalance.

E X P E R I M E N TA L W O R K

This section presents our experimental validation of the proposed feature extraction methods and classification scheme, evaluating each of the proposed feature sets, the effect of signal normalization and considering several problem formulations, using standard performance measures and statistical tests. Moreover, our results are compared with state-Table 3.3: Classification problems derived from the Bonn data set.

Name Type Classes

Problem 1 Binary A -E Problem 2 Binary A,B,C,D -E Problem 3 Multi-class A -D -E Problem 4 Multi-class A -B -C -D -E
of-the-art works from recent literature. In all of the following work, the reported algorithms and tests were implemented in (MATLAB, 2014).

Classification problems

Following the suggestions in [START_REF] Acharya | Automated EEG analysis of epilepsy: A review[END_REF], the Bonn data set can be used to formulate four distinct problems of varying degrees of difficulty, these are summarized in Table 5.1. Problem 1 and Problem 2 are binary classification problems, the former between class A and E (see Table 3.1) and the latter between all Basal readings (classes A, B, C and D) and Ictal state readings (class E). On the other hand, Problems 3 and 4 are multi-class problems, the former considering classes A, D and E, while Problem 4 considers all five classes, the most difficult case. Note that each group (A-E) contains 100 epochs with 4097 samples each. Therefore, Problem 2 in particular presents an unbalanced classification task.

Epoch segmentation

Each epoch in the Bonn database is composed by 4096 samples or measurements, given the duration of each recording and the sampling frequency. However, we pose two different classification problems by treating the data differently. First, we consider each epoch as a single data vector, leaving 100 epochs per class. In the second approach, we split each epoch into 4 non-overlapping segments of roughly 5.9s each with 1024 samples, following [START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF][START_REF] Güler | Adaptive neuro-fuzzy inference system for classification of EEG signals using wavelet coefficients[END_REF]

Experimental setup

For the computation of Hölder regularity we use a Daubechies 10 orthogonal wavelet family and least square linear regression for the coefficients slope calculation in the wavelet decomposition. When we segment the epochs, the maximum decomposition level is lowered because the epoch length is smaller by a factor of 4. The MP algorithm is setup with the parameters in Table 3.4. The atom dictionary can capture different EEG patterns, this helps reduce the number of required iterations to obtain a satisfactory reconstruction.

Usually, there are two stopping criteria for the MP algorithm. One is the number of iterations and the other is the energy level of the signal reconstruction. Here the latter was chosen because we prefer to keep an uniform energy level after reconstruction for all epochs, regardless of its class.

Figures 3.5 and 3.6 show the computation of the pointwise Hölder exponent and MP on the first epoch from each class (A-E). The pointwise Hölder exponent α are similar for class A, B, C and D, with small statistical differences. Class E is distinct, its regularity is noticeably different compared with other groups. For MP decomposition, we can see a clearer difference of atom distribution for all groups. Group B captures more content in higher frequencies, opposed to class E which has a more compact atom location toward lower frequencies despite the fact that it contains more atoms. Other groups exhibit an intermediate composition of atom frequencies, but each shows distinct patterns. To calculate the Hölderian regularity, the FracLab toolbox was used1 .

Figure 3.7 depicts pairwise projections of all possible combinations of the proposed features in Table 3.2, with their corresponding class emphasized by a different color. Effectively, the complete feature space can be visualized, depicting the underlying difficulty of the classification task. From this visualization we can see clearly that no single feature pair is enough to achieve strong discriminant effects. A common pattern found is that variance of class E is large and similar for all combinations of features, although its is usually clearly separated from other classes, thus the uniqueness of the signals generated during epileptic seizures.

Pre-processing

In this work we consider a simple pre-processing stage, where the amplitude is scaled within the range [0, 1] using min-max normalization. To test the importance of this pre-processing step, the proposed features are evaluated both with and without normalization.

Classifier setup and performance measures

The experimental work uses the Random Forests™ implementation by [START_REF] Jaiantilal | RF Matlab interface[END_REF] cross-validation to obtain a robust estimate of classifier performance. The cross-validation was performed 10 times, this gives us a total of 100 runs per problem instance. The following performance measures are reported based on statistics over all runs.

The main quality measure to determine the classification performance is the accuracy. Given a confusion matrix A = [A(i, j)], where its element A(i, j) is the number of data points whose true class label was i and were classified to class j, the overall accuracy can be calculated with

Accuracy = 1 N M ∑ i=1 A(i, i) , (3.16)
where N is the number of samples in the data set and i ∈ M being the i-th label of M classes. Additionally, specificity, F-score and recall were calculated as test classification performance measures. Recall and specificity are measures developed for binary class problems, thus they are calculated for each class, where high values are desired. These are given by

Recall i = A(i, i) A(i, i) + A(i, j) (3.17)
Classification of Epileptic seizures by means of signal processing A(j, j) A(j, j) + A (j, i) .

MAD(H) µ(H) GAD µ(F) µ(A) µ(E) µ 1 2 (E) σ(E) γ 1 (E) γ 2 (E) MAD(H) µ(H) GAD µ(F) µ(A) µ(E) µ 1 2 (E) σ(E) γ 1 (E) γ 2 (E) Class A B C D E
(3.18)

The balanced F-score measure can be calculated based on precision and recall by

F-score = 2 precision • recall precision + recall (3.19)
where precision is given by

Precision i = A(i, i) A(i, i) + A(j, i)
.

(3.20)

Automatic feature selection (9 features set)

To pose the feature selection problem, the genotype of each candidate solution is given by a binary string

b = [b 1 , . . . , b 10 ],
where each b i is associated with one element in the 10 feature set (see the third column of Table 3.2). Each bit in the string determines if the i-th feature is used (b i = 1) or not (b i = 0). The fitness function is based on the classification accuracy (Equation 3.16) achieved on the training set, considering all five classes (Problem 4). Moreover, we apply pre-processing and use the complete epochs. The parameters of the GA are given in Table 3.5. The GA was executed using 10-fold cross-validation, with the goal of detecting which features were chosen with the highest frequency.

Figure 3.8 depicts the frequency of the optimal set of features found by the GA over all folds of the training data. Notice that most features were used at least 80% of the time, with seven of them used with 100% frequency. The only exception was the γ 2 (E) feature, indicating that it is likely the least useful of all the features. Therefore, in this set we only use the first nine features; hereafter we will refer to this set as the 9 feature set or the automatic feature set. 3.4 Results

γ 2 (E) γ 1 (E) σ(E) µ 1 2 (E) µ(E) µ(A) µ(F) GAD µ(H) MAD (

R E S U LT S

The experimental results are organized as follows. Firstly, we perform a statistical analysis on the complete feature set to detect any significant differences among the five classes. For this, the One-way Analysis of Variance (ANOVA) is employed over the feature matrix. These results are presented in Table 3.6. Secondly, Tables 3.7 and 3.8 summarize the performance of our classification system for each problem (1-4) using each feature set (4, 9 and 10). This table shows results obtained on the raw and normalized signals. In each case, the tables show the average Specificity, Recall and F-score computed over all runs. Moreover, the median, best, worst and inter-quartile spread is given. For the segmented approach, the same performance analysis is presented in Tables 3.9 and 3.10 as well.

Finally, to validate our results, non-parametric statistical tests are used to compare the performance behavior of each system configuration based on the accuracy results. For every problem, the proposed approach was tested considering each feature set (3 variants), whether or not pre-processing is applied (2 variants) and if it is using the full epochs or the segmented approach (2 variants); a total of twelve different groups (3×2×2). Afterward, a Friedman test is used to calculate pairwise statistical differences. The p-values for each pairwise comparison are given in Table 3.11 and Table 3.12, applying the Benjamini-Hochberg correction. For these tests, we will reject the null hypothesis that two groups share the same median value with p-values below an α = 0.05.

D I S C U S S I O N

The main contribution of the proposed methodology is the proposed feature set, with Table 3.6 providing some interesting insights. Among all features only the mean of the epoch µ(E) suffers from poor discrimination properties. Indeed, the variance of the average amplitude of the raw signals is quite similar between epochs, thus it is hard to determine a statistically significant difference between the classes. Although the me-Table 3.6 dian is related to the mean, the latter is quite sensitive to outliers while the former is more robust, thus the difference in their corresponding p-value. There are also similarities in the skewness feature among the classes, although it is not statistically significant. Furthermore, there seems to be strong agreement between these results and the GA-based feature selection described in Section 3.3.6. Both the ANOVA test and the meta-heuristic feature selection confirm that 8 of the 10 features are relevant to the classification task, partially disagreeing on only two cases (µ(E) and γ 2 (E)).

Turning to the classification results summarized in Tables 3.7, 3.8, 3.9 and 3.10 (with statistical comparisons given in Tables 3.11 and 3.12) it is clear that Problem 1 is the easiest and Problem 4 is the hardest, as expected. Indeed, for Problems 1-3, at least one configuration achieves perfect accuracy on the test set, while the best performance on Problem 4 is 97.6% accurate. Moreover, these results seem to depend on several factors. For instance, in most cases the full set of 10 features is required to achieve the best performance, with the automatic set of 9 features being the next best option. In all cases, except when using the 4 feature set, performing a pre-processing (normalizing) step is also necessary to achieve the best results.

An interesting result was the performance of the segmented approach. If we compare each row of Tables 3.7 and 3.8 with each row of Tables 3.9 and 3.10, particularly focusing on the normalized data, it is evident that the best results are achieved when we use the full epochs. This is consistent with the idea that with larger epochs the feature extraction process becomes more robust, and with a better ability to extract general properties from each class. There is one exception however, for Problem 4 the best performance is achieved using the segmented approach and the 9 feature set. It seems that the classifier benefits from having a larger amount of examples in this case.

To further validate our work, we take one of the best configurations (9 feature set, full length epochs and pre-processing) and compare it (informally) in Tables 3.13, 3.14, 3.15 and 3.16 with state-of-the-art results reported for the Bonn data set. In the case of Problem 4, we use the segmented variant that produced the best performance. This table presents the authors, the feature extraction approach, whether or not feature postprocessing was performed, the classifier used, the experimental training and testing configurations, the number of features employed, and their performance based on average accuracy, recall and specificity. For a fair comparison, only works that explicitly reported all of the above configurations and performance measures are considered.

This comparison is quite favorable for the methodology described in our current work. First, for problems 1, 2 and 3 we achieve perfect performance based on classification accuracy, matching or surpassing all other previous works. Second, the performance on Problem 4 is quite competitive, though slightly outperformed by some works [START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF][START_REF] Güler | Adaptive neuro-fuzzy inference system for classification of EEG signals using wavelet coefficients[END_REF][START_REF] Guler | Multiclass Support Vector Machines for EEG-Signals Classification[END_REF]. However, as noted in the comparison table, the experimental setup (regarding training and testing partitions) is sometimes not fully given [START_REF] Übeyli | Features extracted by eigenvector methods for detecting variability of EEG signals[END_REF][START_REF] Güler | Adaptive neuro-fuzzy inference system for classification of EEG signals using wavelet coefficients[END_REF][START_REF] Guler | Multiclass Support Vector Machines for EEG-Signals Classification[END_REF][START_REF] Kamath | Analysis of EEG Dynamics in Epileptic Patients and Healthy Subjects Using Hilbert Transform Scatter Plots[END_REF][START_REF] Guo | Epileptic seizure detection using multiwavelet transform based approximate entropy and artificial neural networks[END_REF][START_REF] Lima | Tackling EEG signal classification with least squares support vector machines: A sensitivity analysis study[END_REF][START_REF] Ahammad | Detection of epileptic seizure event and onset using EEG[END_REF][START_REF] Guler | Recurrent neural networks employing Lyapunov exponents for EEG signals classification[END_REF], making the relevance of the performance differences less clear in those cases. Furthermore, on Problem 4 our method uses a significantly smaller feature set (9) compared to other works summarized in Table 3.16, none of them use less than 20 features.

S U M M A R Y A N D C H A P T E R C O N C L U S I O N S

In this chapter we propose a new feature extraction approach for the classification of EEG signals and the detection of epileptic seizures. The detection depends on the ability of the system to recognize seizures among other EEG recordings. The proposed system performs feature extraction using the MP algorithm and Hölderian regularity analysis. This work is the first to combine both methods to effectively extract meaningful traits from epileptic events that are captured by EEG recordings.

Results show that regularity analysis based on the Hölder exponent was able to capture enough information to make a local characterization of the signals and build useful features. One important characteristic of regularity based analysis is its invariance to signal scaling or amplitude a There is missing information of how the train/test partition was done or other details in the experimental setup.

changes, which makes it ideal for EEG processing given the noisy conditions. On the other hand, the MP decomposition provided a more global characterization of the analyzed signals, by building features over the obtained set of waveforms or so called atoms. Certainly, the feature extraction process was shown to be sufficiently informative to be able to simplify the classification problem and achieve state-of-the-art performance.

The experimental work allows us to conclude the following regarding the proposed approach. First, the pre-processing step produces statistically significant performance improvements. Second, using only MP decomposition and Hölderian regularity provided strong performance on all problems, but to achieve optimal accuracy additional information was required. In particular, statistical time-domain features provided sufficient information to solve most problems with state-of-the-art results, that compared favorably with recently published works. Third, the set of proposed features provides the necessary discriminant information to solve the studied problems, with both a statistical analysis and a combinatorial feature selection algorithm suggesting that almost all features are relevant to the epilepsy detection problem in EEG signals. Indeed, on the most difficult classification task (considering five classes), our results achieve close to perfect accuracy (97.6%), while using less than half of the total features used by other approaches that achieve similar performance.

Although the proposed approach was seen as a promising choice among other signal processing/machine learning methods the fact remains that the problem to be solved is complex. In these cases, metaheuristic methods are viable options compared with classical machine learning methods. Meta-heuristic approaches are highly efficient when the problem exhibits these characteristics: the number of samples and variables in the dataset is high, the search space where the optimal solution exists is massive, an approximation of the optimal solution is sufficient to solve the problem, and generalization is more important than data fitting. Despite that in the literature exists a wide range of metaheuristic algorithms, GP is one of the most promising options due to one unique asset: the adaptation to a large number of contrasting ap-Classification of Epileptic seizures by means of signal processing plications. One of the main GP distinctions compared with other algorithms is the solutions representation; they are built using computing elements that being combined can ultimately execute complex processes according with the problem's requirements. In other words, the same algorithm can be adapted to a learning system solver (regression, classification, etc.), to code synthesis for software engineering, to automatic bug fixer, to game strategies/rules development, to electrical circuits designer, among others.

In Part II of this thesis, GP will be the main topic, as the driver for the following research in this manuscript. Nevertheless, as to be explained later, GP is not free of shortcomings. Its unique traits also carry some associated limitations, at least in the original formulation of the algorithm. The most common representation, tree based, carries a series of drawbacks that, fortunately, have been addressed with promising results. Some of these are further explored in consequent chapters, along unique contributions and experimental evidence. More specifically, the incorporation of LS strategies into GP is analyzed and applied to two different statistical learning problems: regression and classification. I n the domain of optimization algorithms, it is widely understood that effectively balancing exploration and exploitation is one of the key issues underlying any successful search process. Indeed, a useful taxonomy of search algorithms is based on the idea of differentiating between search methods that rely on exploratory search operators, and search methods that can guarantee (local) optimality by exploiting the local structure of a fitness landscape. Local search (LS) algorithms can guarantee convergence to local optima if their underlying assumptions are satisfied and are computationally efficient. However, the success of a LS strongly depends on the initial point, particularly in highly irregular, multimodal or discontinuous fitness landscapes [START_REF] Gill | Practical optimization[END_REF]. On the other hand, global search algorithms include a variety of deterministic and stochastic strategies. Here, we focus on evolutionary algorithms (EAs), metaheuristic strategies based on an abstract model of Neo-Darwinian evolution [START_REF] Eiben | Introduction to Evolutionary Computing[END_REF][START_REF] Luke | Essentials of Metaheuristics[END_REF]. EAs are particularly useful when a good initial solution is difficult to propose, or when LS strategies tend to converge on undesirable local optima. Moreover, EAs are particularly well suited when a single monolithic so-lution is not sufficient, and instead a set of different solutions is required [START_REF] Coello | Evolutionary Algorithms for Solving Multi-Objective Problems (Genetic and Evolutionary Computation[END_REF][START_REF] Dunn | Parisian camera placement for vision metrology[END_REF]. In general, EAs cannot guarantee convergence towards local optima in most realistic scenarios. Moreover, they tend to be computationally costly, and in many cases can be highly inefficient. Nevertheless, EAs have produced extremely competitive solutions in difficult domains and problem instances [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF].

Considering the strengths and weaknesses of both global and local methods, it is intuitive to conclude that the best strategy should be a hybrid approach, commonly referred to as memetic search. These methods have been extensively studied over recent years, combining EAs with a variety of local searchers [START_REF] Chen | A multi-facet survey on memetic computation[END_REF].

While all EAs are based on the same general principles [START_REF] De | Evolutionary Computation: A Unified Approach[END_REF], there is a large variety among current methods, each with their respective strengths and weaknesses. In this chapter, we focuses on an EA paradigm that presents unique properties among other EAs, genetic programming (GP) [START_REF] Koza | Genetic Programming: On the Programming of Computers by Means of Natural Selection[END_REF][START_REF] Poli | A Field Guide to Genetic Programming[END_REF]. The goal of GP is to automatically evolve specialized syntactic expressions that best solve a given tasks, which can be interpreted as mathematical functions or computer programs. In particular, in this chapter we focus on how the standard syntactic search can be improved by including a LS method. While this work is not the first to develop a memetic GP system, it does present the first systematic evaluation of what could be the best strategies to accomplish this, by considering several variants and evaluating them on current benchmark problems for symbolic regression.

The remainder of this chapter proceeds as follows. First, an overview of GP is provided in Section 4.2. Then, Section 4.3 reviews previous work on memetic optimization with EAs. Afterwards, Section 4.4 describes our proposed memetic algorithms to perform real-valued parameter optimization of evolved syntactic expressions, considering several basic variants. The experimental setup is presented in Section 4.5 and results are discussed in Section 4.6. Finally, a summary of the chapter and concluding remarks are outlined in Section 4.7.

Genetic Programming

G E N E T I C P R O G R A M M I N G

GP can be understood as a generalization of the basic genetic algorithm, its main features can be summarized as follows [START_REF] Poli | A Field Guide to Genetic Programming[END_REF]. First, GP was originally proposed as an EA that evolves simple programs, functions, operators, or in general symbolic expressions that perform some form of computation. GP is basically used to evolve solutions to different types of design problems, with examples as varied as quantum algorithms [START_REF] Spector | Automatic Quantum Computer Programming: A Genetic Programming Approach (Genetic Programming)[END_REF], computer vision operators [START_REF] Olague | Evolutionary-computer-assisted design of image operators that detect interest points using genetic programming[END_REF] and satellite antennas [START_REF] Hornby | Computerautomated evolution of an x-band antenna for nasa's space technology 5 mission[END_REF]. Second, solutions are expressed as variable length structures, such as linked lists, parse trees or graphs [START_REF] Poli | A Field Guide to Genetic Programming[END_REF]. These structures encode the syntax of each individual program. Therefore, in a canonical GP algorithm, search operators, such as crossover and mutation, perform syntactic variations on the evolving population. Third, by considering each individual in a GP run as a program, the evolutionary process is basically attempting to write the best program syntax that solves a given problem. Therefore, a finite set of basic symbols needs to be defined, which is called the primitive set P. Within the primitive set there is a subset of basic operations or functions of different arity, called the function set F, and a subset of input variables, constants or zero arity functions called the terminal set T , such that P = F ∪ T .

Given the variety of possible GP configurations and applications, this work focuses on the problem of symbolic regression using a tree representation. In symbolic regression, the goal is to search for the symbolic expression K O : R p → R that best fits a particular training set T = {(x 1 , y 1 ), . . . , (x n , y n )} of n input/output pairs with x i ∈ R p and y i ∈ R. In such problems, for instance, the function set can be defined as F = {+, -, /, * } and the terminal set can be composed by each of the features in the input data, such that T = {x i } with i = 1, . . . , p, but other terminal elements can be included such as integer or real-valued constants.

The general symbolic regression problem can then be defined as

(K O , θ O ) ← arg min K∈G;θ∈R m f (K(x i , θ), y i ) with i = 1, . . . , p , (4.1)
where G is the solution or syntactic space defined by P, f is the fitness function based on the difference between a program's output K(x i , θ) and the expected output y i , such as the mean square error, and θ is a particular parametrization of the symbolic expression K, assuming m real-valued parameters. This dual problem, of simultaneously optimizing syntax (structure) as well as its parametrization is also discussed in [START_REF] Lohmann | Proceedings of parallel problem solving from nature (ppsn i) -first workshop[END_REF][START_REF] Emmerich | Design of graphbased evolutionary algorithms: A case study for chemical process networks[END_REF]. The authors differentiate between two possible approaches towards solving such a task. The first group is hierarchical structure evolution (HSE), when θ has a strong influence on fitness, and thus a local search is required at each generation of the global (syntactic) search, configured as a nested process. The second group is called simultaneous structure evolution (SSE), when θ has a marginal effect on fitness, in such cases a single evolutionary loop could optimize both syntax and parameters simultaneously. However, such categorizations are highly abstract, and a particular implementation could easily be classified into both groups. Nevertheless, it is reasonable to state that the standard GP approach falls in the SSE group.

In standard GP, however, parameter optimization is usually not performed explicitly, since GP search operators only affect syntax. Therefore, the parameters are only implicitly considered. For instance, a GP individual might have the following syntax K(x) = x + sin(x); in this case, we might propose the following parametrization: θ = (α 1 , α 2 , α 3 ), with K(x) = α 1 x + α 2 sin(α 3 x). In a traditional GP search, these parameters are all set to 1, which does not necessarily lead to the best possible performance for this particular syntax. Indeed, if the optimal solution is K O (x) = 3.3x + 1.003sin(0.0001x), then individual K might be easily lost during the selection or survival processes1 . This seems to be a glaring weakness in the standard GP approach. While the syntactic-based search in GP has solved a variety of difficult problems, it is nonetheless very inefficient, leading to important practical limitations, such as search stagnation, bloat and uninterpretable solutions [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF]. In other words, GP performs a highly explorative search, since the search operators can produce large fitness changes with only modest syntactic modifications or vice-versa. Therefore, the inclusion of a LS procedure could enhance the performance of the search. In this chapter, we study what are the effects of including a local optimization strategy within a GP run, configured as an HSE system.

P R E V I O U S W O R K

Many works have studied how to combine an evolutionary algorithm (EA) with a local optimizer, usually referred to as memetic algorithms, with impressive results in a variety of domains [START_REF] Chen | A multi-facet survey on memetic computation[END_REF]. A useful taxonomy of memetic algorithms can be derived based on how inheritance is carried out during the evolutionary process [START_REF] Chen | A multi-facet survey on memetic computation[END_REF]. Suppose that (h, h o ) ∈ G 2 , where h is an individual solution and h o is the solution generated after a LS is applied on h. Obviously, for a minimization problem, f (h o ) ≤ f (h), where f is the objective function. Thus, h h o , unless h was in fact a local optima. Then, a memetic algorithm could proceed in two distinct ways with respect to inheritance. In a Lamarckian algorithm, the traits acquired during the local search, captured in h o , replace those of the original individual h; i.e., the inheritance of acquired characteristics (h, f (h)) → (h o , f (h o )). On the other hand, in a Baldwinian algorithm, the local optimization process only modifies the fitness of an individual; (h, f (h)) → (h, f (h o )); i,e., ontogenic evolution. In this work, we propose a Lamarckian approach for a GP system.

When applying a LS strategy to tree-based GP, there are basically two broad approaches to follow. (1) To apply a LS on the syntax of a tree or (2) to apply it numerically on the parameters of the tree, as described in Equation (4.1). Regarding the former, two recent works are noteworthy. In [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF], the authors apply a greedy search on a randomly chosen GP node, attempting to determine the best function to use in that node among all the functions in the primitive set, constrained only by the arity of the initial function. Moreover, to reduce computational overhead the authors apply a heuristic decision rule to decide which trees are subject to local optimization, preferring smaller trees in the population. This heuristic has the positive effect that it biases the search towards smaller trees, so it is adopted in our current proposal. In [START_REF] Wang | Multiobjective genetic programming for maximizing ROC performance[END_REF], the authors propose a multi-objective GP algorithm to evolve decision tree classifiers, and use two specialized mutation operators that limit their resulting phenotypic variations within a local neighborhood of the parent classifier.

Regarding the optimization of numerical parameters within the tree, the following works are relevant. In [START_REF] Topchy | Faster genetic programming based on local gradient search of[END_REF], gradient descent is used to optimize numerical constants within a GP tree, achieving good results on symbolic regression problems. However, the work only optimizes the value of the terminal elements (tree leaves), and it does not consider parameters within internal nodes. Additionally, that work only considers training fitness, a highly deceptive measure of learning. Similarly, in [START_REF] Zhang | Genetic programming with gradient descent search for multiclass object classification[END_REF] and [START_REF] Graff | Wind speed forecasting using genetic programming[END_REF] a LS algorithm is used to optimize the value of constant terminal elements. In [START_REF] Zhang | Genetic programming with gradient descent search for multiclass object classification[END_REF], gradient descent is used and tested on classification problems, while [START_REF] Graff | Wind speed forecasting using genetic programming[END_REF] uses Resilient Backpropagation (RPROP) and evaluates the proposal on a real-world problem. In [START_REF] Zhang | Genetic programming with gradient descent search for multiclass object classification[END_REF], the authors apply gradient descent on every individual of the evolving population, an obvious computational bottleneck, while [START_REF] Graff | Wind speed forecasting using genetic programming[END_REF] only applies RPROP on the best individual from each generation. However, it is not evident which strategy can offer the best results in new scenarios, particularly since both [START_REF] Zhang | Genetic programming with gradient descent search for multiclass object classification[END_REF][START_REF] Graff | Wind speed forecasting using genetic programming[END_REF] evaluate their approaches on specific problem instances.

Other recent works have completely changed the basic GP framework to include an explicit parametrization of syntactic expressions. The fast function extraction (FFX) algorithm [START_REF] Mcconaghy | FFX: Fast, Scalable, Deterministic Symbolic Regression Technology, chapter Genetic Programming Theory and Practice IX[END_REF], for instance, poses the symbolic regression problem as that of finding a linear combination of a subset of candidate basis functions. Thus, FFX builds linear in parameter models, and optimizes using a modified version of the elastic net regression technique, eliminating the evolutionary process altogether. In the prioritized grammar enumeration (PGE) tech-nique [START_REF] Worm | Prioritized grammar enumeration: Symbolic regression by dynamic programming[END_REF], dynamic programming replaces the the basic search operators of traditional GP, and numerical parameters are optimized using the non-linear Levenberg-Marquardt algorithm.

In this chapter, we propose a very simple parametrization approach for GP trees, by constraining the number of internal parameters of each tree regardless of its size. A similar approach was developed by [START_REF] Kommenda | Effects of constant optimization by nonlinear least squares minimization in symbolic regression[END_REF], with some noteworthy differences; for instance, parameters are only used to replace constants terminals, and each tree is enhanced by adding a linear upper tree, that effectively adds a weight coefficient and a bias to the entire tree. Then, the Levenberg-Marquardt optimizer is used to find the optimal values for these parameters. Similarly, in [START_REF] Tsai | Using weighted genetic programming to program squat wall strengths and tune associated formulas[END_REF], the authors propose a weighted GP by inserting coefficients in binary trees. In this work, weight optimization is done by means of GA, which requires parameter tuning itself to reach a global optimum. However, this study has some drawbacks, it lacks the study of the influence of optimized solutions in the evolutionary process, it is tested over an specific problematic and experimental reproducibility is not clear.

Finally, the work presented by [START_REF] Arnaldo | Multiple regression genetic programming[END_REF] uses a multivariate linear regression approach to optimize evolved solutions for symbolic regression problems. Particularly, the search is conducted by standard GP, but the linear regression is used as a Baldwinian memetic algorithm, such that the tree is decomposed into a set of subtrees which are then linearly combined discarding the original tree structure, a much more explorative approach then the one presented here. Then, to control code growth a multiobjective approach us included to account for solution complexity.

Here, we propose a Lamarckian memetic algorithm, where the main goal is to combine the explorative capabilities of GP without changing its canonical implementation, and incorporate stronger exploitative features by means of numerical optimization.

I N T E G R AT I N G L O C A L O P T I M I Z AT I O N S T R AT E G I E S W I T H I N G P

Intuitively, through tree parametrization and local optimization, a GP search should converge faster towards high quality solutions. As stated before, the proposal is to develop an HSE-GP, by parameterizing GP trees and including a Lamarckian memetic strategy. Therefore, the parametrization of GP trees must be defined; then, a particular LS method must be chosen. Finally, a decision strategy must be suggested to determine on which individuals should a LS be applied. The proposals for each of these issues are presented below.

Parametrization of GP trees

For this study, we propose a simple and naive approach to add parameters within GP trees. For each function in the function set g k ∈ F, we add a unique weight coefficient θ k ∈ R, such that each function is now defined by

g ′ k = θ k g k (4.2)
where g ′ k is the new parameterized function, k ∈ {1, . . . , r} and r = |F|. Note that each θ k is linked to a single g k , such that the parameter vector of tree K i is given by θ ∈ R m with m the number of different functions included in K i such that m ≤ r. Notice that if a tree contains several instances of a particular function, all instances of this function share the same coefficient. Indeed, this severely constraints the optimization process, particularly for large trees that can include many instances of the same function. To be clear, it is not argued that such a parametrization should be considered as the best possible alternative. Nonetheless, it does have one important consequence, it bounds the size of the search space for each LS process, something that could become overwhelming if the GP trees grew to large, which tends to happen frequently due to bloat [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF].

In all trees, every θ k is initialized to unity, which would be their implicit value in a standard GP. Since the proposed algorithm is a Lamarck-ian memetic search, the standard GP search operators (subtree mutation and subtree crossover) still only operate at the syntax level, exchanging g ′ k nodes without affecting their respective θ k .

Local Search mechanism

Potentially, any tree can be of linear or non-linear form; however, for convenience we treat every tree as non-linear expression. This is a multidimensional non-linear optimization problem, which can be solved using least squares.

The above problem, is formally expressed by the following cost function

min θ ∥K(θ, x) -y∥ 2 2 = min θ ∑ i (K(θ, x i ) -y i ) 2 , (4.3)
where x are the input data points, y are the output data points, i is the index for the regression instances, and K is the non-linear function. The problem posed in (4.3) can be solved using different methods [START_REF] Yuan | Numerical methods for generalized least squares problems[END_REF]) [START_REF] Lawson | Solving least squares problems[END_REF]. In this case, we chose to use a well known technique with good convergence properties and good scalability in complexity for high dimensional problems, called trust region optimization [START_REF] Sorensen | Newton's Method with a Model Trust Region Modification[END_REF].

The trust region optimization method tries to minimize a smooth non-linear function subject to bounds on the variables, given by

min θ∈R m K(θ), l i ≤ θ i ≤ u i ∀i ∈ {1..m} , (4.4)
where l i ∈ {R ∪ {-∞}}, u i ∈ {R ∪ {∞}}, and K : R m → R. Conceptually, a trust region approach replaces an m-dimensional unconstrained optimization problem by an m-dimensional constrained problem. This results in an approximate solution, since it does not needed to be solved with high accuracy. One of the appealing points of these methods is their strong convergence properties [START_REF] Coleman | On the convergence of reflective Newton methods for large-scale nonlinear minimization subject to bounds[END_REF]. The idea behind a trust region method for min θ∈R m K(θ) is simple. The increment Regression s k = x k+1 -x k is an approximate solution to a quadratic subproblem with a bound on the step size

min s∈R m { ψ k (s) def = g T k s + 1 2 s T B k s : ∥D k ∥ ≤ ∆ k } , (4.5)
where g k def = ∇f (x k ), B k is a symmetric approximation to the Hessian matrix ∇ 2 f (x k ), D k is a scaling matrix, and ∆ k is a positive scalar representing the trusted region size. Solving (5.6) efficiently is not a trivial task, see [START_REF] Shultz | A Family of Trust Region Based Algorithms for Unconstrained Minimization with Strong Global Convergence Properties[END_REF][START_REF] Sorensen | Newton's Method with a Model Trust Region Modification[END_REF][START_REF] Steihaug | The Conjugate Gradient Method and Trust Regions in Large Scale Optimization[END_REF][START_REF] Moré | Computing a trust region step[END_REF]; Figure 4.1 shows an illustration of the trust region method. Here, the method proposed in [START_REF] Coleman | An interior trust region approach for nonlinear minimization subject to bounds[END_REF] is used, which does not require the solution of a general quadratic programming subproblem at each iteration.

Integrating LS into GP

The final issue that must be considered is to determine on which individuals is the LS applied, and at what moment during the evolutionary process. For the latter point, LS is applied after a complete evolutionary loop is completed; i.e., LS is applied after the survival criterion is applied and before the following generation begins. This means that the local optimization of individuals from generation t impacts the search at generation t + 1. For the former point, several different approaches are evaluated.

First, three naive approaches are considered, two of which have been used in previous memetic GP systems. Probably the simplest is to apply a LS on all of the individuals, this method is referred to as LSALL-GP. This approach will inherently introduce a large computational overhead. Another approach is to be more selective, and only apply a LS on the best individual from each generation, this method is referred to as LSBEG-GP. Conversely, LS can also be applied on the worst individual from each generation, this variant is called LSWEG-GP. This variant might be useful for extreme cases of individuals that present the correct structure, but a highly suboptimal parametrization. 4.4, at iteration j.

1 1 0.1 0.5 0.5 1 P ER LS 0 0 100% 10% 10% 100%
The fourth variant is referred to LS Random Population or LSRP-GP, where every individual of the population is a viable candidate for a LS optimization with a probability p LS , basically implemented as a mutation operator. Here, it is assumed that a low probability is desirable, to minimize the added computational cost. The fifth variant is called LS Best Subset or LSBS-GP, which is the same as LSRP-GP, except that the individuals that are valid candidates for LS are those in the best P ER LS percentile of the population, a second algorithm parameter. Finally, the sixth variant is called LS Worst Subset or LSWS-GP, takes the same approach as LSRP-GP, but candidate individuals for a LS are those in the worst P ER LS percentile. All of the methods and their parameters are summarized in Table 4.1.

E X P E R I M E N TA L S E T U P

The algorithms that are evaluated are LSBEG-GP, LSWEG-GP, LSRP-GP, LSBS-GP, LSWS-GP and LSALL-GP; also, a standard GP is included as a control method. All the algorithms were implemented in Matlab using the GPLAB2 Toolbox [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF] modifying its core functionality to integrate the LS procedure. The set of experiments cover a series of well known benchmark symbolic regression problems. Five synthetic problems were used: Keijzer-6 (Keijzer, 2003), Korns-12 (Korns, 2011), Vladislavleva-4 (Vladislavleva et al., 2009), Nguyen-7 [START_REF] Uy | Semantically-based crossover in genetic programming: application to real-valued symbolic regression[END_REF], Pagie-1 (Pagie and Hogeweg, 1998); and one real-world 4.5 Experimental Setup problem. All of them are suggestions made in [START_REF] White | Better gp benchmarks: community survey results and proposals[END_REF], a recent survey on GP benchmarking. Two performance measures are used to evaluate the different algorithms. First, fitness evaluation over the test set for the best solution found thus far. Second, the average population size, a relevant measure regarding the bloat phenomenon and solution complexity [START_REF] Trujillo | Preliminary study of bloat in genetic programming with behavior-based search[END_REF]. The evolution of these measures is analysed with respect to the total number of fitness function evaluations instead of generations, to account for the LS iterations. The LS performs a maximum of 400 iterations. However, we do not consider any additional computational effort due to the LS, which underestimates the computational cost of performing LS. The stopping criteria is the number of function evaluations. The GP configuration parameters for all variants are shown in Table 4.2. Some of these parameters vary depending on the problem, as suggested in [START_REF] Mcdermott | Genetic programming needs better benchmarks[END_REF]. For the case of Tower problem, total samples were divided in a ratio of 30/70% for the training and testing sets. As stated before, Table 4.1 summarizes the parameter values for the memetic GP variants. 4.2, considered a simple or easy problem. The difficulty is raised with the testing data, since the solution must extrapolate outside the training domain. In general, most methods exhibit similar performance, with some notable exceptions. First, LSWEG-GP obviously achieves the worst test performance, while LSBS-GP and LSALL-GP exhibit the best. In particular, LSBS-GP and LSALL-GP converge quickly to very good performance; in fact LSBS-GP could have been halted very early in the run. However, after a million function evaluations LSALL-GP, shows a noticeable improvement, achieving the best results. In terms of size, both of these methods also exhibit the best results, with LSALL-GP producing the smaller trees.

Figure 4.3 presents the results for the Korns-12 problem, considered difficult in GP literature since standard GP usually cannot find the true expression [START_REF] Korns | Accuracy in symbolic regression[END_REF]. The problem includes redundant input data that does not influence its output. The idea is to test GP algorithms on their ability to avoid over-fitting. With respect to test error, Figure 4.3(a) shows that most algorithms perform quite similarly, all of them converging to their best median values quite early in the runs. What is noticeable is the performance of LSBS-GP, with the worst test performance, but also the best training performance, shown in Figure 4.3(c), indicating a slight over-fitting. Regarding the evolution of size, shown in Figure 4.3(e), all methods exhibit similar sizes, but code growth is noticeably slower for LSALL-GP.

For the Vladislavleva-4 problem shown in Figure 4.3, considered a difficult test case for GP [START_REF] Vladislavleva | Order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming[END_REF], over-fitting is also noticeable. In Figure 4.3(b) we can see that test fitness varies greatly across the runs for most methods, particularly for LSBS-GP. Conversely in Figure 4.3(d) we can observe a smooth monotonic convergence for the training fitness. The case of LSBS-GP is noticeable, particularly since it is clear that the method achieves its best test fitness early in the run, and then starts to over-fit. The only method that did not over-fit was LSALL-GP, exhibiting the best performance over the test data. Finally, with respect to program size (Figure 4.3(f)), once again all methods show similar trends, with LSALL-GP producing substantially smaller trees.

Problem Nguyen-7 and Pagie-1 exhibit similar outcome in terms of fitness and program size. In both, among all methods LSBS-GP exhibits a faster convergence to a better solution quality computed over test data and also achieves the best results along with LSALL-GP. Once again LSALL-GP produces the smallest trees, with LSBS-GP the second best, however in these problems the difference between both of these methods is smaller than in the other cases. All other variants show similar performance based on both measures.

Finally, the Tower problem is an industrial real-world problem on modeling gas chromatography measurements of the composition of a distillation tower. This problem contains 5000 records and 23 potential input variables. The measurements (5000 for each variable) are not treated as time series, but simply used as samples for a regression model. In this case, LSALL-GP again achieves the most interesting results, both in regards to test-fitness and solution size. From the remaining meth- ods, LSBS-GP shows the best performance, but still noticeably worse that LSALL-GP.

A final summary of the performance of each method is presented in Table 4.3, that shows three different snapshots of the median performance of each algorithm, after 250,000, 1 million and 2 million function evaluations. In general, LSALL-GP exhibits the best performance after all of the allowed function evaluations. However, if we only consider 250,000 function evaluations, then LSBS-GP exhibits the best performance on three of the six problems, and also the second best performance on two others.

From the remaining methods, those that show the worst performance are LSWEG-GP and LSWS-GP, that apply LS to the worst solutions in the population, what definitely seems to be a bad strategy. Moreover, while LSBS-GP shows good performance, the deterministic LSBEG-GP that applies LS to the best solution is notably inferior to the best methods, a noteworthy results since several previously published works have employed this memetic strategy.

C H A P T E R C O N C L U S I O N S

This work studies the problem of integrating a local optimization process into a GP, using a Lamarckian HSE memetic approach. A comparative study is performed, evaluating different ways in which to incorporate a LS during the basic evolutionary process of GP, evaluating performance on symbolic regression problems. A simple tree parametrization is proposed, bounding the size of parameter space for each individual tree, even if bloat occurs during the run. The local optimization is done by a trust region technique that determines the optimal coefficients posing a basic non-linear curve fitting problem. As stated before, it is not clear what might be the best strategy to incorporate LS into GP, so different stochastic and deterministic variants are extensively evaluated over a set of widely used benchmark problems. In particular, each method uses different heuristic decisions to determine which individuals in the GP run should be subject to a local optimization process; experimental results suggest the following.

In general, it seems that a memetic GP almost always outperforms a standard GP, in terms of both solution quality and solution size. Moreover, among the different methods that were tested, several insights can be gathered. First, it does not appear to be beneficial to use a LS as another mutation strategy, such that all individuals might be candidates for a LS, given the average performance of LSRP-GP. Second, it does not seem useful to apply LS to the worst individuals in the population, as seen by the performance of LSWEG-GP and LSWS-GP. Third, many works have used the simple deterministic heuristic of applying LS to the best solution found, either at the end of the run or at each generation. However, this to does not seem to be an adequate strategy, given the performance of LSBEG-GP. Finally, of all the methods, the best performance was achieved when LS is applied to all of the solutions (LSALL-GP) or to random individuals chosen from the top percentile (w.r.t. fitness) of the population (LSBS-GP).

For all problems, the best performance was achieved by LSALL-GP. However, if only a small amount of computational effort is feasible, then LSBS-GP seems to be the best option, given its fast convergence. More-over, the difference in computational effort between both methods is understated in the results presented here, since only the total iterations in the LS are considered, omitting the total effort devoted towards computing approximate derivatives or matrix inversions. Nevertheless, LSALL-GP also shows a substantial ability to curtail bloating during the search, this was indeed expected. Since the search process in LSALL-GP focuses primarily on parameter optimization, limiting the total of syntactic search performed by the GP crossover and mutation operators; i.e., the total number of generations is quite low for LSALL-GP, basically eliminating the possibility of bloating.

In appendix A, we extend the experimentation to real-world symbolic regression problems, providing additional insights in the evolution process. Consequent appendixes, namely B, C and D focus in three different case studies that uses the integration of LS into GP. The compilation of the results derived from all these works produced a compelling conclusion: the global search from GP is greatly benefited by a LS mechanism. Nevertheless, further investigation is required to extend the domain of representation of LS operators into other applications, which will be discussed in the future directions at Chapter 7.

In the next chapter, we will explain our contribution toward classification problems using a memetic approach to GP, gathering experimental evidence as result. We will explain the differences between the representation used in regression and the needed changes to address binary classification problems.

Based on Z-Flores, E., [START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF]. T he general goal of the genetic programming (GP) paradigm is to evolve specialized syntactic expressions that can solve a user defined problem, mostly used in supervised learning tasks. In particular, GP is widely used to generate mathematical functions, or operators, that solve symbolic regression and classification problems. In the previous chapter we explored different variants of LS integration into GP, providing useful insight on the effectiveness of the hybrid search mechanisms in symbolic regression problems. However, there was still a couple of open questions: is it possible to make a representation to handle classification problems? if so, will the memetic GP approach be efficient as well?. To try to answer these questions, in this chapter we introduce a methodology to parameterize GP trees, employ a LS method to numerically tune them and apply this process for binary classification problems.

Here, a domain specific fitness function is used, based on the Receiver Operating Characteristic (ROC) curve of each GP individual. Experimental results are encouraging in several problems, performance is significantly improved relative to a standard GP search, based both on fitness and solution size. Moreover, by using widely used real-world benchmarks, it is clear that our proposal compares favorably with other proposals from the GP and machine learning literature.

The remainder of this chapter is organized as follows. Section 5.2 describes our approach and Section 5.3 presents the experimental work. Finally, concluding comments are given in Section 5.4.

I

N T E G R AT I N G T H E L O C A L S E A R C H M E C H A - N I S M W I T H I N G P
For binary classification, GP is used as a supervised learning problem, where a training set x of n-dimensional patterns with a known classification, are used to derive a mapping function g(x) : R n → {0, 1}, with a threshold δ as classification decision. Afterwards, for convenience, we transform the classification problem into a regression one by inserting a continuous function in order to get gradient information, and to be able to use a numerical optimizer.

GP Tree Parameterization

First, we follow the strategy proposed in [START_REF] Kommenda | Effects of constant optimization by nonlinear least squares minimization in symbolic regression[END_REF], where a small linear subtree is added on top of the root node of the original tree K(x), such the new extended tree K ′ (x) is given by

K ′ (x) = θ 2 + θ 1 (K(x)) , (5.1)
where θ 1 and θ 2 are the first two parameters from θ, as shown in Figure 5.1. This subtree provides shifting and scaling properties, so the optimizer later on can have more freedom in its capabilities to find a better solution. This approach is closely related to the linear scaling technique [START_REF] Keijzer | Improving symbolic regression with interval arithmetic and linear scaling[END_REF], allowing the syntactic search operators to focus on evolving the desired shape of the evolved function without worrying about optimizing the scale or bias of the output.

Afterwards, for each node n k in a given tree we add a weight coefficient θ k ∈ R, such that each node is now defined by

n ′ k = θ k n k , (5.2)
where n ′ k is the new modified node, k ∈ {1, ..., r}, r = |Q| and Q is the tree representation. Notice that each node has an unique parameter that can be modified to help meet the overall optimization criteria of the nonlinear expression.

At the beginning of the GP run, each parameter is initialized by θ k = 1∀k, where |θ| = r. During the GP syntax search, subtrees belonging to different individuals are swapped, added or removed (following the standard crossover/mutation rules) together with its corresponding parameters, without affecting their values; a memetic search process with Lamarckian inheritance.

A Continuous Transfer Function

In a binary classification problem, the desired output is either 0 or 1, a discrete output. Indeed, this discontinuity will have an homologous discontinuity in parameter space, and thus the LS algorithm can easily fail. To provide a continuous output, and provide gradient information to the LS method, a transfer function is added as a root node.

Here, a sigmoid function is used, producing the following non-linear tree

sig(K ′ (x, θ)) = 1 1 + e ρ(K ′ (x,θ)-δ) ,
(5.3)

where K ′ is the program output evaluated over the input vector x, θ is the parameter vector corresponding to the program K ′ , ρ is the sigmoid slope parameter and δ is the classification threshold, where the function has a value of 0.5. The sigmoid function is normalized to the range [0,1]; therefore, if sig(K ′ (x, θ)) ≤ 0.5 then the class label is 0, and 1 otherwise.

The proposed tree extensions and parameterizations are depicted in

The Local Search Mechanism

In this work we treat each tree as a non-linear expression. We also assume a non-discontinuity for the evaluated tree over the interval constrained by the input training data.

The problem to be solved is an optimization one, where we want to best fit a non-linear function to an output vector containing either zero or one values (each class label). It can be formally defined as

min θ ∥sig(K ′ (θ, x)) -y∥ 2 2 = min θ ∑ i (sig(K ′ (θ, x i )) -y i ) 2 ,
(5.4)

where x is the input data vector, y ∈ {0, 1} is the known class label, sig is the sigmoid calculated over the program K ′ , i is the index for the problem instances and θ is the parameter vector. The problem presented in (5.4) can be solved using different techniques [START_REF] Yuan | Numerical methods for generalized least squares problems[END_REF][START_REF] Lawson | Solving least squares problems[END_REF]. In this work, we follow the same method used in previous chapter, the Trust Region opti- mizer, belonging to the family of non-linear Gauss-Newton methods [START_REF] Sorensen | Newton's Method with a Model Trust Region Modification[END_REF]. One example of this family is the recent approach to the Levenberg-Marquardt algorithm, which follows the Trust Region concept.

As a reminder, in the Trust Region algorithm, an iterative process is performed with the following goal

min θ∈R m ∥sig(K ′ (θ, x)) -y∥ 2 2 , l i ≤ θ i ≤ u i ∀i ∈ {1, . . . , m} , (5.5)
where l i ∈ {R ∪ {-∞}}, u i ∈ {R ∪ {∞}}, and K ′ : R m → R. Conceptually, a Trust Region approach replaces an m-dimensional unconstrained optimization problem by an m-dimensional constrained problem. This results in an approximate solution, since it does not needed to be solved with high accuracy. One of the appealing points of these methods is their strong convergence properties [START_REF] Coleman | On the convergence of reflective Newton methods for large-scale nonlinear minimization subject to bounds[END_REF]. The idea behind a Trust Region method for min θ∈R m K(θ) is simple. The increment s j = x j+1 -x j is an approximate solution to a quadratic subproblem with a bound on the step size

min s∈R m { ψ j (s) def = g T j s + 1 2 s T B j s : ∥D j ∥ ≤ ∆ j } ,
(5.6) where ψ(s) is the quadratic function, g j def = ∇f (x j ), B j is a symmetric approximation to the Hessian matrix ∇ 2 f (x j ), D j is a scaling matrix, and ∆ j is a positive scalar representing the trusted region size. Solving (5.6) efficiently is not a trivial task, see [START_REF] Sorensen | Newton's Method with a Model Trust Region Modification[END_REF][START_REF] Moré | Computing a trust region step[END_REF]. Here, the method proposed in [START_REF] Coleman | An interior trust region approach for nonlinear minimization subject to bounds[END_REF] is used, which does not require the solution of a general quadratic programming subproblem at each iteration. Figure 5.2 shows the effect of performing LS over a specific individual.

The Fitness Function

In the case of binary classification, a simple method is to consider only accuracy in the training stage [START_REF] Winkler | Advanced genetic programming based machine learning[END_REF][START_REF] Eggermont | Genetic Programming: Second European Workshop[END_REF]. In this work we use the following fitness measure

f itness = 1 -Acc , (5.7)
where Acc (accuracy; computed as the ratio of correctly classified instances among total number of cases) is calculated based on the best threshold found using the ROC curve. The ROC curve, which is commonly used by the machine learning community, can provide a robust measure of classifier performance even with an unbalanced dataset. Each operating point on a ROC curve represents the classification accuracy at a single threshold δ i . The ROC curve is constructed by changing the classification threshold and evaluating its True Positives Rate (TPR) and False Positives Rate (FPR) values. These are calculated with TPR=TP/P and FPR=FP/N, respectively, where TP is the True Positive value, FP is the False Positive value, P is the total of positive instances and N the negative ones.

In the implementation presented here, we used 10 uniformly spaced threshold values within [min(K ′ (x, θ))max(K ′ (x, θ))], which is the range given by minimum and maximum output of the extended tree, over all fitness cases. This methodology is similar as the one presented in [START_REF] Bhowan | Developing new fitness functions in genetic programming for classification with unbalanced data[END_REF].

In the ROC curve, the top-left area represents the most accurate classification results. Ideally, a perfect classifier should have a TPR value of one with zero FPR. Based on this, we propose a simple method to choose the best threshold picked up from the ROC curve. A vector with 45 degree slope is traced with coordinates [0,1]-[1,0], in ROC space, which represent our reference vector υ r . We then build a numerical array ω containing the angle differences between each threshold vector υ i , with coordinates [0,1]-[FPR i ,TPR i ,] and the reference vector, as seen in Figure 5.3. This is calculated by and

∆υ i = υ i (y) -υ i (x), ∆υ r = υ r (y) -υ r (x) (5.8) FPR 1 0 TPR 1 ω i Δυ r Δυ i reshold
ω i = acos ( (∆υ i • ∆υ r )|∆υ r | |∆υ i | ) , (5.9)
where υ i is the vector corresponding to each threshold δ i and ω i is the difference angle expressed in radians. Consequently, the array ω = [ω 1 , ..., ω k ]. The threshold corresponding to the minimum value in this array will represent our chosen threshold for the particular classifier, as seen in Figure 5.3.

Integrating LS into GP

Finally, we need to determine which individuals should be improved by the LS method. For instance, some researchers apply the LS to the best solution in the population, but [START_REF] Trujillo | Neat, there's no bloat[END_REF] showed that better results are obtained when it is applied to all the population or a subset of the best solutions in each generation. However, in our case, the number of parameters grows proportionally with the size of the trees, providing incentive to limit the amount of times the LS is applied, especially for larger trees. Therefore, in this work we use the heuristic rule proposed by Muhammad and Ryan in [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF], which they tested with a syntactic LS process. In [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF], the LS method is applied stochastically for each tree, based on a probability p(s) determined by the tree size s (number of nodes) and the average size of the population s, based on

p(s) =          1 if s < (c -1)s s s + c if (c -1)s ≤ s ≤ cs 0 otherwise .
(5.10)

Based on equation 5.10, shown in Figure 5.4, smaller trees are more likely to be optimized by the LS than larger trees; in particular, by setting c = 1.5 trees that are 50% larger than the population average are not subject to the LS process. This heuristic could provide two positive results. First, the LS method is mostly applied on small trees that have a relatively small number of parameters, simplifying the LS problem. Second, by focusing on improving smaller trees, this increases their chances for survival, increasing the chances of finding solutions that are not hampered by the bloat problem [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF]. Hereafter, we call this method Local Search Heuristic by Size (LSHS).

Experimental Setup

The algorithm LSHS was evaluated and compared with a canonical GP implementation as control method. The GP settings were the same between both variants, including the fitness function.

The LSHS algorithm was implemented using the GPLAB1 Matlab toolbox [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF]. The set of experiments covered a series of benchmark binary classification problems, taken from the UCI repository [START_REF] Lichman | UCI machine learning repository[END_REF]; real problems with distinct complexity. Table 5.1 shows a summary of each evaluated problem. A couple of quality measurements were used to compare the results of the algorithm, the accuracy calculated over the test data and average population size, needed to assess the performance in terms of bloat and consumed resources. The evolution of these measure is analyzed with respect to the total number of fitness function evaluations instead of generations, to account for the LS iterations. The evaluations performed by the iterative Trust Region algorithm were also taken in account to provide a fair comparison between standard GP and LSHS. The Trust Region algorithm performs a maximum of 500 iterations. The stopping criteria is a predefined number of function evaluations. The GP configuration used in this research is listed in Table 5.2.

Results

Fig. 5.5 and Fig. 5.6 summarizes the results of the tested algorithm over several classification problems. The figure shows convergence plots for training and testing fitness with respect to the number of fitness function evaluations, showing the median over all runs. In each of the problems, the accuracy of LSHS testing data is either similar or better compared to standard GP. At least in four of them the improvement in the method is significant. For the Sonar problem the accuracy reached in the testing data is similar to the one obtained by standard GP. In the Banknote problem, the LSHS improvement is significant, where the classification error almost reaches zero.

Figure 5.7(a) discloses descriptive statistics from test fitness, at the end of each run, using notched box plots. The Wilcoxon rank sum test was calculated for the fitness test data, shown in first column of Table 5.3. The average population size is also presented in Figure 5.7(b), and its corresponding rank sum test is given in the second column of Table 5.3.

In both cases, the null hypothesis is that the underlying distributions in each pair of experiments have equal medians.

The statistical tests show that that the differences in performance exhibited by the standard GP and LSHS-GP are significant on four of the seven problems, particularly Parkinsons, Diabetes, Wholesale and Banknote, with α = 0.05. Regarding size, statistical difference was detected on five problems, namely Parkinsons, Diabetes, Wine, Wholesale and Banknote. It is interesting to note, that on all four problems where performance improved, size was also reduced. Here we can state that for these problems, the increased average size in standard GP does indeed represent bloat, since better solutions can be found while maintaining smaller populations. The Wine problem seems relatively easy for GP, the median performance of both algorithms is practically a perfect accuracy, but still GP evolves unnecessarily large populations, since the memetic GP with LSHS can produce equivalent performance with smaller populations in terms of average program size. The two outliers seem to be the Sonar and LSVT problems, two difficult problems on which no improvements in terms of size or fitness are obtained. It could be, that this is related to the large number of problem features in both cases, Sonar has 60 features and LSVT has 309, however confirming this hypothesis is left as future research.

Table 5.4 presents an informal comparison with some recently published results for the seven problems tested here, using average classification accuracy. Thus providing a quick survey to contextualize the results presented above. The comparison is encouraging, the proposed LSHS method shows relatively strong performance on all problems, except on Parkinsons and LSVT. However, the best results on Parkinsons [START_REF] Ma | An efficient diagnosis system for parkinson's disease using kernel-based extreme learning machine with subtractive clustering features weighting approach[END_REF] uses a domain specific preprocessing stage, while other methods report similar results with our work. Similarly, [START_REF] Tsanas | Objective automatic assessment of rehabilitative speech treatment in parkinson's disease[END_REF] uses a state-of-the-art feature selection algorithm to reduce the dimensions of the problem given to the classifier, important for this dataset since it contains 309 features, while GP takes the entire feature vectors as input, a clear disadvantage.

C H A P T E R C O N C L U S I O N S

In this study, a methodology to incorporate a LS method into GP was proposed, that tackles the challenges imposed by a supervised classification task. A tree parameterization is introduced where a candidate solution in GP is extended by a simple mechanism where a weight is added to each node. Using a well known and robust non-linear opti- mizer, the solution is improved targeting a better classification rate. A heuristic is used to determine which solutions are improved by the LS method, preferring smaller than average trees within the population.

The presented results illustrate the benefits of using a LS method during the GP search. Indeed, in most problems the improvements in terms of classification error was significant, and a survey of recently published results confirm the quality of the evolved solutions. Moreover, the evolution of program growth also provides interesting insights. In particular, GP+LS with LSHS was able to produce high quality solutions, in most cases significantly better than standard GP, while maintaining smaller individuals within the population in terms of average size. This suggests that code growth is not necessary to obtain improved performance if the search space is properly exploited, which the LS method allows us to do. Moreover, it is important to notice that the proposed LSHS-GP system should not increase the complexity, or more precisely reduce the interpretability of the evolved solutions relative to standard GP. In fact, it might help produce simpler trees (results at least show that they are smaller), since the LS process allows simple numerical tuning of the evolved programs to improve fitness; something that otherwise might require complex syntactical changes. This contrast nicely with other recent approaches based on geometric semantic operators [START_REF] Moraglio | Geometric semantic genetic programming[END_REF], that improve GP performance at the cost of lowering interpretability. [START_REF] Ozcift | Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms[END_REF] 87.7 (5-fold CV) Dirichlet process mixtures Ozcift and Gulten [START_REF] Ozcift | Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms[END_REF] 87.1 (10-fold CV) CFS-RF Ma et al. [START_REF] Ma | An efficient diagnosis system for parkinson's disease using kernel-based extreme learning machine with subtractive clustering features weighting approach[END_REF] 99.49 (10-fold CV) SCFW-KELM with feature preprocessing Dufourq and Pillay [START_REF] Dufourq | A comparison of genetic programming representations for binary data classification[END_REF] 86.7 (testing) GP based (arithmetic, logical and decision trees) Koshiyama et al. [START_REF] Koshiyama | Gpf-class: A genetic fuzzy model for classification[END_REF] 89.33 (testing) GP based (GPF-CLASS) Wang et al. [START_REF] Wang | Multiobjective genetic programming for maximizing ROC performance[END_REF] 86 In the last two chapters, we tried to provide a bird-eye view in terms of the current memetic GP approaches, together with our own analysis. Before, we stated that GP with a tree representation suffered with some limitations, being the search process one of them. Indeed, the standard search operators are blind to the effect to the individual outputs. Only after evaluation and further selection the search can be steered toward the objective. This produces a slow convergence, that normally is correlated with the underlying data characteristics. Therefore, LS strategies accelerate this convergence without affecting much the solution structure.

Since, conceptually, GP allows us to use it in a wide range of applications, the need for a proper representation is of vital importance. Identifying this representation is one of the several open questions that still exist [START_REF] O'neill | Open issues in Genetic Programming[END_REF]. Additionally, appropriate search operators, types of population initialization, selection and replacement methods are as well important at the time of adapting GP to a new problem. For example, in our previous proposals, between the regression and the classification approaches, there was a slight variation in the individual representation along the necessary change in the fitness function. In the next chapter, we introduce more drastic modifications. The particular problem of feature extraction for EEG signals requires an overhaul in several GP elements. The canonical individual representation simply will not work. A new set of search operator will be needed as well; first, to fully take advantage of the solutions representation, and second, to make the search more efficient. Furthermore, the fitness function has to be adapted to the new objective. Details on the proposal will be further discussed across the next chapter.

Part III

Mental states classification

113 C L A S S I F I C AT I O N O F M E N TA L S TAT E S B Y M E A N S O F G E N E T I C P R O G R A M M I N G 6.1 I N T R O D U C T I O N
T here are human activities that are considered to have a high risk because they require a response within a given time window with an associated cost. An incorrect reaction could pose different risks in terms of security, health or financial costs, among others. For example, in a surgery, a physician is expected to be fully alert, as is the case for a heavy-machinery operator; a technician in a manufacturing process, a prolonged state of alertness is desired when safety and productivity is the target. Some of the causes of unintentionally low states of vigilance are associated to sleep deprivation [START_REF] Alhola | Sleep deprivation: Impact on cognitive performance[END_REF], monotonous tasks [START_REF] Schmidt | Assessing driver's vigilance state during monotonous driving[END_REF] or stress [START_REF] Selye | The stress syndrome[END_REF].

Human consciousness has two main components: wakefulness and awareness of the environment [START_REF] Baars | A Cognitive Theory of Consciousness[END_REF]. Wakefulness is associated to the content of consciousness and awareness to the level of consciousness. These two, in the majority of situations, are heavily correlated with normal physiological states (with the exception of dream activity during REM-sleep) [START_REF] Laureys | The neural correlate of (un)awareness: Lessons from the vegetative state[END_REF]. Derived from this, a custom model can be built where a high awareness/wakefulness represents an alert or normal state and a lower awareness represents a relaxed but awake state, which are the two conditions studied in this work.

To make use of this model, an approach would be to develop a computing system that would interact to a human and be able to identify each condition. A BCI system fits quite well in this kind of scenario.

In particular, different brain signal acquisition methods have been developed during the last decades, being the Electroencephalogram (EEG) the most used among the non-invasive approaches. It is worth mentioning that research has shown that BCI systems themselves can be affected when the alertness of the user is low [START_REF] Myrden | Effects of user mental state on EEG-BCI performance[END_REF].

However, the task of distinguishing both states by means of EEG recordings is not trivial. These type of signals exhibit several complex characteristics: they are highly irregular, non-periodic, non-stationary, divergent between persons and distinctive among trials [START_REF] Siuly | EEG Signal Analysis and Classification: Techniques and Applications[END_REF].

In this work, we propose a new methodology for the automatic classification of two mental states, involving a novel algorithm for feature extraction based on the Genetic Programming (GP) paradigm. The study is dissected in two parts: first, a reference system is built based on our previously published work [START_REF] Vézard | EEG classification for the detection of mental states[END_REF]; and second, an enhanced system is proposed with the objective of improving the already competitive results exhibited by the reference system. Both systems include these functional blocks: acquisition of signals, pre-processing, feature extraction and classification. However, this work makes the following contributions: first, the system includes a classifier independent feature extraction algorithm called + FEGP; second, the algorithm avoids over-fitting during training using a regularized fitness function, which is unique in GP based search; third, unique components in the algorithm design are introduced to evolve multidimensional transformations of the data, extending a previous approach [START_REF] Muñoz | M3GP -Multiclass Classification with GP[END_REF], with specialized search operators and dynamic parameters; fourth, this is the first time that a GP based algorithm is used in the classification of mental states.

Related work

Although the use of an Event-Related Potential (ERP) is a common practice in the development of BCI systems, is not always the case, and basically depends on the mental task that needs to be interpreted. ERPs are brain measurement captured after a cognitive, motor or sensory event has occurred. Some cognitive functions in humans can be detected after some milliseconds in response to a particular stimulus, which is where ERP based methods can be useful [START_REF] Woodman | A brief introduction to the use of event-related potentials (ERPs) in studies of perception and attention[END_REF]. However, ERPs are not suitable for the goal of this work. Certainly, mental states might not be triggered by any particular stimulus, rather they are a prolonged condition in brain activity, activated by psychological reasons like mood change, or by physiological like exhaustion.

Literature states that the most promising EEG based systems have to contain at least the following components: signal pre-processing, feature extraction/selection and classification (Nicolas-Alonso and Gomez-Gil, 2012). Moreover, hybrid systems usually are the ones with the best performance, making it difficult to classify them into well defined groups [START_REF] Amiri | A Review of Hybrid Brain-Computer Interface Systems[END_REF]. Because literature related to EEG classification can be overwhelming, we are going to discuss here only the works that try to classify some kind of mental state rather than a particular mental task (e.g. motor-imagery).

The majority of the research in this area has focused on techniques from signal processing, supervised/unsupervised learning, statistical learning or a hybridization of these fields; either for feature selection, feature extraction or classification [START_REF] Gupta | Performance enhancement of mental task classification using EEG signal: a study of multivariate feature selection methods[END_REF]. Particularly for EEG feature extraction in terms of mental states, several approaches have been followed. In [START_REF] Trejo | EEG-Based Estimation and Classification of Mental Fatigue[END_REF], a Kernel Partial Least Squares (KPLS) algorithm was used as a feature extraction step for alerted and fatigued states. Wavelet Transform (WT) coefficients and some measures of approximate coefficients, like Shannon entropy, are used in [START_REF] Zarjam | Beyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload[END_REF] for feature extraction, for a workload mental states classification problem. Garcés-Correa et al. [START_REF] Garcés Correa | Automatic detection of drowsiness in EEG records based on multimodal analysis[END_REF]) also use the WT for feature extraction in drowsiness detection. In these methods, coefficients from the WT decomposition are used as a spectral granularity measure of the signals, suitable as feature vectors. Saidatul and Sazali (A.Saidatul, Paulraj, 2013) used a modified Auto Regressive (AR) model by compensating forward and backward prediction errors, then using simpler statistics over the obtained eigenvectors, with very competitive results. Based on a valance/arousal framework, general mental states are identified by using the Power Spectral Density (PSD) in the work by [START_REF] Mallikarjun | Mental State Recognition by using Brain Waves[END_REF], focusing on feature extraction rather than classification. Other common approaches to construct new features is the Common Spatial Pattern (CSP), Source Power co-Modulation (SPoC) and Spatio-Spectral Decomposition (SSD) which are presented in the work by Schultze-Kraft et al. [START_REF] Schultze-Kraft | Unsupervised classification of operator workload from brain signals[END_REF]. Deep Learning (DL) based methodologies have been recently used successfully in this type of problems. For example, the work by Hajinoroozi et al. [START_REF] Hajinoroozi | Prediction of driver's drowsy and alert states from EEG signals with deep learning[END_REF] present different variants of deep neural networks (DNN), like Deep Belief Network (DBN), Convolutional Neural Networks (CNN) and some specific channel-wise CNNs (CCNN). By stripping the label layer, the DNNs are effectively used as feature extractor elements.

In terms of feature selection, the number of works is rather small. For example, in (A. Saidatul, Paulraj, 2013) authors use Principal Component Analysis (PCA) to extract a feature vector subset. This is opposed to what happens in mental task research, where there is a high occurrence on feature selection methods, those being filter or wrapper based [START_REF] Gupta | Performance enhancement of mental task classification using EEG signal: a study of multivariate feature selection methods[END_REF].

Regarding the classifier, the community tends to use a wide range of tools from the machine learning field: Linear Discriminant Analysis (LDA) (Schultze-Kraft et al., 2016) [START_REF] Vézard | EEG classification for the detection of mental states[END_REF], k-Nearest Neighbors (k-NN) (A. Saidatul, Paulraj, 2013) [START_REF] Guo | Automatic feature extraction using genetic programming: An application to epileptic EEG classification[END_REF], Support Vector Machine (SVM) [START_REF] Gupta | Performance enhancement of mental task classification using EEG signal: a study of multivariate feature selection methods[END_REF] [START_REF] Shen | EEG-based mental fatigue measurement using multi-class support vector machines with confidence estimate[END_REF], Artificial Neural Network (ANN) [START_REF] Zarjam | Beyond Subjective Self-Rating: EEG Signal Classification of Cognitive Workload[END_REF] and bagging-based methods [START_REF] Hajinoroozi | Prediction of driver's drowsy and alert states from EEG signals with deep learning[END_REF], among others.

According with the surveyed literature, the majority of works use classical algorithms to solve the problem of mental states recognition, however alternatives like meta-heuristic methods seem to help when the complexity of the problem is high. From an optimization perspective, algorithms from the Evolutionary Computing (EC) field have been employed as channel reduction or feature selection approaches. Genetic Algorithms (GA) have been found to be suitable, mostly in wrapper approaches [START_REF] Vézard | Detecting mental states of alertness with genetic algorithm variable selection[END_REF] [START_REF] Fang | EEG Signal Classification Using the Event-Related Coherence and Genetic Algorithm[END_REF] [START_REF] Rezaee | Applying Genetic Algorithm to EEG Signals for Feature Reduction in Mental Task Classification[END_REF] [START_REF] Hongxia | Research on the application of the improved genetic algorithm in the electroencephalogram-based mental workload evaluation for miners[END_REF]. Other algorithms like Ant Colony Optimization (ACO) [START_REF] Erguzel | Ant colony optimization based feature selection method for QEEG data classification[END_REF] or Particle Swarm Optimization (PSO) [START_REF] Lin | Classification of mental task from EEG data using neural networks based on particle swarm optimization[END_REF] have also been used. Adaptive approaches have been employed as well like the auto-reinforced system introduced in [START_REF] Hassani | An Incremental Framework for Classification of EEG Signals Using Quantum Particle Swarm Optimization[END_REF], by incorporating a feed-backed PSO and a rule-based classifier.

GP is a particular variant of EC algorithms, with the unique characteristic that solutions are concatenated computing elements. An inherent strength of the GP paradigm is its symbolic representation that can be adapted to a wide range of problems. For example, GP can be used either for feature selection, feature extraction or classification. In [START_REF] Bhardwaj | Classification of EEG signals using a novel genetic programming approach[END_REF], a GP multi-tree classifier is used in combination with Empirical Mode Decomposition (EMD) in an EEG dataset for epilepsy detection. A continuation of this work with modified search operators is presented in [START_REF] Bhardwaj | A novel genetic programming approach for epileptic seizure detection[END_REF]. A decision-tree based model is presented in [START_REF] Fernández-Blanco | Classification of signals by means of Genetic Programming[END_REF], where arithmetic and logical rules are inferred in a multi-level GP representation. Although Electrocorticogram (ECoG) was used instead of EEG by the authors in [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF], GP was employed as a classifier to solve an epileptic seizure recognition problem.

More closely related to this work, Guo et al. [START_REF] Guo | Automatic feature extraction using genetic programming: An application to epileptic EEG classification[END_REF] built a feature extraction model using GP for an epilepsy EEG dataset. In [START_REF] Firpi | On prediction of epileptic seizures by means of genetic programming artificial features[END_REF], GP is used similarly, this being a feature extraction task. These cited works are further discussed in Subsection 6.3.1, since there are some differences worth reviewing compared with our proposal. Apart from feature extraction methods, GP can be used as a simple feature selection tool like in [START_REF] Sabeti | Entropy and complexity measures for EEG signal classification of schizophrenic and control participants[END_REF]. Although some work has been done in EEG classification using GP, to the authors's knowledge this is the first time it is used to specifically tackle the mental state recognition problem.

The chapter is organized as follows. In Section 6.2 the reference system based on CSP and LDA is described and analyzed. In Section 6.3 we discuss in detail the proposed system, in particular the GP based feature extraction method called + FEGP. Experimentation and results are presented in Section 6.4. In Section 6.5 results are discussed. Finally, in Section 6.6 we present our conclusions. Programming Figure 6.1: Methodology of the reference system

R E F E R E N C E S Y S T E M

The reference system is basically the first part of our previous work [START_REF] Vézard | EEG classification for the detection of mental states[END_REF]. Particularly, a complete classification system is built based on several stages. First, a data acquisition protocol was followed; second, a preprocessing step involving spectral filtering; third, a feature extractor based on the CSP is used; and fourth, a classification step using LDA. This reference system is described in Fig. 6.1. In the following subsections, each of these elements are explained in detail.

Data acquisition

Acquisition protocol

A selected group of individuals (critical personal information was protected) were chosen to collaborate in the experiments during a 2011 campaign at the Université de Bordeaux, France. Each of the involved persons were subjected to the following procedure to build the dataset: First, a cap was placed over the scalp using 58 electrodes in positions following the international standard system 10/10 [START_REF] Jurcak | 10/20, 10/10, and 10/5 systems revisited: Their validity as relative head-surface-based positioning systems[END_REF], as shown in Fig. 6.2. This procedure lasted approximately 30-40 mins, since the electrodes are wet based, illustrated in Fig. 6.3. The used recording system was a Deltamed Coherence 3NT, by Natus1 .

Second, the participants were seated alone in a soundproof room, in front of a screen where different information was fed directly trough visual cues.

Third, the first EEG recording started at this point. The individual had to look at a fixed cross at the center of the screen, avoiding eye movements as much as possible. This recording corresponds to the alert, basal or normal state.

Fourth, a Contingent Negative Variation (CNV) protocol [START_REF] Walter | Contingent Negative Variation : An Electric Sign of Sensori-Motor Association and Expectancy in the Human Brain[END_REF] was used, depicted in Fig. 6.4. The CNV is a special protocol where specific steps are followed by using ERPs. The person is instructed to press the space bar as quickly as possible each time the cross is replaced by a square on the screen. For each appearance of the square, a warning sound (beep) is given 2.5s before it appears, allowing the person to prepare her/his response. The experimental session included 50 pairs of stimuli (S1: beep, S2: square), with a random amount of time elapsing between each pair. Depending on the experiment's requirements, number of repetitions or trial duration could vary. An average signal of the superimposition of each trial is extracted, synced at the stimuli triggering starting time. For our experiment, although the subject was instructed to react as fast as possible to the visual cue, the reaction time is not of interest, rather what happens after the audio cue, that is the prolonged mental state. The protocol is designed this way to avoid selftraining by the subjects to the studied state. In the CNV test, the amplitude of the resulting signal is directly associated to the relaxation level of the subject [START_REF] Naitoh | Modification of surface negative slow potential (CNV) in the human brain after total sleep loss[END_REF], as illustrated in Fig. 6.5.

Fifth, a relaxation session was performed in order to bring the subject to a lower vigilance state. The subject was guided by a soundtrack broadcast through speakers placed in the room. The soundtrack sug-gested the subject to perform three successive exercises of self-relaxation, based on muscular exercises and mental visualization. The first exercise is autogenic training [START_REF] Carruthers | Autogenic training[END_REF] where the subject has to mentally repeat some sentences favoring self-hypnosis. The second exercise is a progressive relaxation [START_REF] Elton | Relaxation Theory and Practice[END_REF], which consists on flexing and unflexing some muscles. The final exercise is mental visualization, where the subject imagines that she/he is moving in a familiar and lovely place.

Sixth, the subsequent EEG recording is obtained afterwards, which corresponds to the relaxed state. Approximately 3 mins of recording was captured at this step, fairly matching the normal state recording length.

Finally, a second CNV collection was gathered, using the same protocol as before.

Notice that a pair of CNV tests was performed during the data acquisition procedure. This dual measurement allows us to properly assess the validation of the actual subjects brain state compared with the expected state. Differences in amplitudes from the first and the second CNV test should display a presence of an induced relaxation state or its absence. If this difference exist then both EEG recordings are considered valid and are tagged as "normal" and "relaxed" states, respectively. To do this, a visual and numerical inspection of superimposed CNV tests was performed, exemplified in Fig. 6.6 at the interval between T1 (360 ms after the stimulus S1) and T2 (900 ms).

Subjects

The experiment involved 44 subjects (mixed gender), non-smoking, aged from 18 to 35 years old. They are right-handed, to avoid variations in the characteristics of the EEG due handedness linked to functional inter-hemispheric asymmetry. After the CNV test, several subjects were rejected since they were not in compliance with the expected state. Therefore, only 13 valid individuals were used to build the dataset.

Raw data

The recorded data, considered as the raw data in this study, contains 26 records of approximately 3 mins each (13 corresponding to the normal state and 13 more for the relaxed state) of 58 channels for each subject. A sampling frequency of 256 Hz was employed during the acquisition by the Deltamed system. Since the length of the recordings slightly varies from subject to subject, approximately 46,000 samples were obtained for each recording.

Pre-processing

For this type of systems, the automatic detection should be accomplished by just analyzing a small portion of a signal and classifying it as quickly as possible, particularly in an online scenario. By splitting the signal in small packets of data (commonly referred as trials), training and testing a supervised learning system is possible. However, the length of a trial cannot be predefined a priori, thus a quick analysis to determine an appropriate size for the trials is required. In [START_REF] Vézard | EEG classification for the detection of mental states[END_REF], different trial lengths were checked and the optimal one was found based on the classification accuracy. Lengths of 1024, 2048 and 4096 samples were analyzed and a value of 2048 samples (8 seconds) was found to be the best, resulting in ≈ 22 trials per subject/class. Consequently, this value was also used in this work. Therefore, our dataset is stored in a matrix X 58×(26×≈22)×2048 .

The recorded EEG signals usually contain noise and unrelated frequencies to the task at hand. The origin of these unrelated frequencies are partly due the oscillatory rhythms present during brain activity [START_REF] Pfurtscheller | EEG-based discrimination between imagination of right and left hand movement[END_REF]. The most studied rhythms are alpha (8-12 HZ), beta (19-26 Hz), gamma (1-3.5 Hz) and theta (4-8 Hz), which are associated to different psycho-physiological states. The alpha waves are characteristic of a diffuse awake state for healthy persons and can be used to discern between the normal and relaxed states. Actually, in some recordings alpha waves start to appear when the subject is starting to relax.

In this work, band-pass filtering is applied to discriminate frequencies outside the alpha and beta bands. Again, an analysis is required to find out which cutoff frequencies are useful to build the filter, by sweeping a range of candidate values. In [START_REF] Vézard | EEG classification for the detection of mental states[END_REF] we present such analysis, with the resulting values of 7 Hz and 30 Hz corresponding to the low and high cutoff frequencies using a 5th order Butterworth filter; matching what other researchers have found as well [START_REF] Ramoser | Optimal spatial filtering of single trial EEG during imagined hand movement[END_REF][START_REF] Blankertz | Optimizing Spatial filters for Robust EEG Single-Trial Analysis[END_REF]. The filter response is illustrated in Fig. 6.7.

Common Spatial Patterns

Like other techniques that derive a projection or transformation matrix based on some specific requirements (e.g Independent Component Analysis (ICA) or PCA), the CSP is a technique where a matrix W is built, that simultaneously maximizes the variance of a multivariate subset with an arbitrary label and minimizes the variance for another subset with dif-

Low Cutoff

High Cutoff ferent label, belonging to the same data-set. Projected data is given by F = XW, where X is the original data with dimensions p × n × T , being p the number of channels, n the number of trials and T the trials length. This technique is useful in a binary classification problem because data is projected to a space where both classes are optimally separated in terms of their variance. The CSP can be briefly defined as the following optimization problem:

w = argmax w ||wX C1 || 2 ||wX C2 || 2 (6.1) and v = argmax v ||vX C2 || 2 ||vX C1 || 2 , (6.2)
where X C1 is the matrix that contains the trials for class one and X C2 the matrix for class two; respectively the normal and relaxed conditions in our case. Coefficients found in w and v define a set of filters that project the covariances of each class orthogonally.

This problem can be solved by a single eigen-decomposition of M = Σ -1 C2 Σ C1 , where

Σ C1 = 1 n C1 X ′ C1 X C1 (6.3)
is the estimation of the covariance corresponding to the average of n C1 trials of class one. Similarly,

Σ C2 = 1 n C2 X ′ C2 X C2 (6.4)
is the homologous for class two. In a single eigen-decomposition, the first k eigenvectors (corresponding to the k largest eigenvalues) of M are also the last k eigenvectors of Σ -1 C1 Σ C2 . Sorting the eigenvalues in descending order we can build our filter matrix with

W = [w 1 , . . . , w k , v 1 , . . . , v k ],
(6.5) in which filters are considered pairwise (w j , v j ) for j = 1, . . . , k. The number of filter pairs can greatly affect the classification accuracy, thus a tuning step is needed to find the optimal number of filters. This step was performed in [START_REF] Vézard | EEG classification for the detection of mental states[END_REF] resulting in a value of k = 3, which we employed as well in this work.

Classification

To approximate a normal distribution of the data, a logarithm calculation is performed over the variance obtained from the input data transformation by XW. This is stored in a matrix Z, and is given by

Z = log(var(F)) =            log(var(X 1 W 1 )) • • • log(var(X 1 W j )) . . . . . . . . . log(var(X i W 1 )) • • • log(var(X i W j ))            n×2k . (6.6) Programming
The evaluation of the classification process is performed using the Leave One Out Cross-Validation (LOOCV) method. In this methodology, data is split into q folds (X = [X 1 , . . . , X q ]) where the training subset has a size of q -1 and the testing has a size of one. In our case q = 13, the number of subjects. The process is repeated q times by changing the index m for testing and keeping the remaining folds for training. The reason for using this type of partitioning is because the recognition is performed at the individual level (grouping trials from the same individual) rather than at the trial level. We want to avoid to train and test the classifier using trials from the same subject, which could lead to misleading results.

The classifier used is LDA. This technique is commonly used in BCI systems, due to its ease of implementation and competitive performance compared with more sophisticated algorithms [START_REF] Duda | Pattern Classification (2Nd Edition)[END_REF]. Briefly, in LDA a hyperplane is calculated based on the covariances of the data distribution that optimally separates two classes. If we assume that both classes belong to a normal distribution and have the same covariance, then we can calculate

w = Σ -1 (µ C1 -µ C2 ) (6.7) and b = w T µ C1 + µ C2 2 , (6.8)
where Σ is the shared covariance matrix for both classes, while µ C1 and µ C2 are the corresponding means for each class. Classification is performed by assigning a label (either C1 or C2) to a label vector γ depending on a score vector s, given by s = b + wZ, (6.9) computed using the hyperplane found during training but using a projected testing data based on the pre-calculated W.

and γ =        C1 if s ≥ 0 C2 otherwise . ( 6
Since this is calculated for a given fold in the LOOCV strategy, after q repetitions, an average of the accuracy results of all folds is computed and is considered as the prediction accuracy. Results from the classification performance can be seen in Fig. 6.8, where the average and fold wise accuracy are presented corresponding to the training and testing partitions. For training, an average accuracy of 76.6% was obtained, and 75.7% for testing. Please note that over-fitting exists (or a failure to generalize) for some folds, namely 3, 6, 8 and 12.

P R O P O S E D E N H A N C E D S Y S T E M

In this section we present a novel algorithm based on GP that extends the reference system to improve the classification results, particularly for the prediction over unseen data. From a general point of view, the enhanced system is described in Fig. 6.9, where it can be seen that the Programming Figure 6.9: Methodology for the proposed enhanced system.

proposed Feature Extraction with GP ( + FEGP) follows the CSP block. In other words, we perform a post-transformation of the CSP output, with the goal of improving the classification results.

Genetic Programming

Part of the broader Evolutionary Algorithms (EA) research area, GP [START_REF] Koza | Genetic Programming: On the Programming of Computers by Means of Natural Selection[END_REF] is sometimes considered to be a generalization of the popular GAs [START_REF] Goldberg | Genetic Algorithms in Search, Optimization and Machine Learning[END_REF]. In GP, a pool of candidate solutions (population) are syntactically evolved towards a specific objective by applying selection pressure based on a quality assessment or fitness. GP differentiates from other population based algorithms on the following fronts. First, the solutions or individuals are expressed with a symbolic structure. This is mainly where its strength comes from, since it can represent many things, depending on the problem's domain. The only restriction is that the expression must be computable, that is to say for a given input it produces a valid output. Second, the size of the solution is variable, meaning that during the search their length or size changes. This provides GP its second strength, solution structures are not predefined, rather they are evolved simultaneously with their performance. A general diagram for GP is presented in Fig. 6.10.

GP has been used successfully in many applications with different representations [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF], whereas the tree representation is the most common among them. In this representation, building blocks are attached following a hierarchical pattern, where the leaves are normally inputs such as variables, features or constants (terminal set), and the internal nodes are computing elements (function set). The tree evaluation is calculated in a reverse order, from leaves to the root node, being the latter where the output is produced.

Although the canonical GP achieves competitive performance compared with other algorithms [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF] [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF], including hand-made designs, it is not suitable for all types of problems, specially when the underlying data has complex properties. Practically, GP can suffer from some shortcomings, also common to other machine learning algorithms, like the stagnation of the search in a local optima [START_REF] Dauphin | Identifying and attacking the saddle point problem in high-dimensional non-convex optimization[END_REF] or model over-fitting in under-sampled data [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction[END_REF]. For these reasons, a new algorithm is proposed based on an extension of our previous works [START_REF] Muñoz | M3GP -Multiclass Classification with GP[END_REF] [START_REF] Naredo | Evolving genetic programming classifiers with novelty search[END_REF].

Augmented Feature Extraction with Genetic Programming:

+ FEGP Recognized as one of the most important tasks in a classification system, our main proposal is focused on the improvement of the feature extraction stage. One of the discovered complexities found during the implementation of the reference system is that the data of the study is highly varied and uncorrelated within the same class, and overlapped between both classes. This yields a non-trivial problem for any supervised learning system. A suitable strategy would be a non-linear model that constructs new features which further increases the inter-class distance and reduces the intra-class spread.

Based on this, a GP-based algorithm is proposed with the objective of improving the classification accuracy for the reference system. Another goal for the proposal is to be highly flexible, not explicitly depending on any specific pre-processing or classifier blocks. A block diagram for this algorithm is depicted in Fig. 6.11, hereafter referred to as + FEGP, for augmented feature extraction with GP.

Similar research has been conducted in terms of feature extraction with GP. [START_REF] Guo | Automatic feature extraction using genetic programming: An application to epileptic EEG classification[END_REF] propose first an individual representation with a single root node, where a special node called 'F' can be inserted in any part of the tree, which produces a new feature. Besides this enhancement, no special search operators were used for their system. On the other hand, [START_REF] Firpi | On prediction of epileptic seizures by means of genetic programming artificial features[END_REF]) used a canonical GP individual representation, where a single tree produces a new feature, a rather simple proposal that nevertheless produced competitive results on the classification of signals from epileptic seizures. A similar approach was followed by [START_REF] Poli | Evolution of a brain-computer interface mouse via genetic programming[END_REF], where a single root node representation with a custom fitness function was used to evolve mouse trajectories by employing EEG signals. The authors used a strongly-typed GP, a variant in which there is an explicit constraint on data type allowed in the terminals and functions. In a previous work [START_REF] Sotelo | Identification of epilepsy stages from ECoG using genetic programming classifiers[END_REF], we used GP as a classifier for the detection of epileptic stages, taking as input simple statistics of each epoch.

The present proposal builds upon our previous works in [START_REF] Muñoz | M3GP -Multiclass Classification with GP[END_REF] [START_REF] Naredo | Evolving genetic programming classifiers with novelty search[END_REF], where the M3GP algorithm is presented. M3GP also uses some additional search operators, to add or remove complete features during the run, but mostly relies on a canonical GP implementation with a single objective function that only takes in account the training accuracy and a static parametrization of the search. On the other hand + FEGP includes a wider variety of search operators, a specialized local fitness measure to account for the fact that each solution is a collection of subtrees, dynamic parameters that control how the search is performed at different moments during the run, a regularized fitness function to control over-fitting and achieve better generalization, and a specialized initialization procedure to account for the extremely large feature space in this problem domain. Each of these features will now be explained in greater detail.

1. Individual representation. The representation is also based on a tree structure, but it involves a multi-tree with a single root node, such that each individual defines a mapping of the form K : R k → R r , being k the number of input features and r the number of newly constructed features. Each of these subtrees are constructed in the same way as in a canonical GP (see Fig. 6.12). No calculation is executed at the root node, it is rather used as a container of the evolving r subtrees. Please note that each subtree works as a nonlinear model that transforms input features into a new space that is expected to simplify the classification. Moreover, note that r can vary among individuals.

2. New genetic operators. Because of the manner in which individuals are represented, it is necessary to introduce new search operators. For any of these operators the root node is avoided during the operation to preserve the representation.

(a) Inter-crossover. This is a crossover performed between subtrees belonging to different individuals and produces two offsprings. The selection for the crossover points are based on a local fitness measure (later discussed in detail), which allows the best subtrees to interchange genetic material. This can be visualized in Fig. 6.13.

(b) Intra-crossover. For a given individual, genetic material can be swapped within the same individual using this operator, depicted in Fig. 6.14. The rationale behind this is that genetic material from one feature might help the evolution of another within the same individual, improving the overall fitness. The subtree selection is performed randomly, choosing the crossover points from different subtrees. This operator generates two offsprings.

(c) Inter-individual feature crossover. It can be desirable to keep complete features with good performance. This operator interchanges complete subtrees between two selected individuals. This is performed randomly rather than using a deterministic approach. Preliminary experimentation suggested that not using a fitness-based selection of the subtree allowed the algorithm to explore the solution space better. This operator can be seen in Fig. 6.15.

(d) Subtree mutation. This is performed the same way as in canonical GP. A portion of a selected subtree is substituted with a randomly generated tree.

(e) Feature insertion mutation. This operator allows the insertion of a new randomly generated subtree expanding the dimensionality of the transformed feature space; Fig. 6.16 depicts this operator.

(f) Feature removal mutation. Similarly, a deletion operator is needed to reduce the projected feature space. The subtree to be removed is selected randomly. This operator can only be used if r > 1.

Subtree 1 Subtree 2 Subtree 3 Subtree r ... Subtree r+1

Figure 6.16: + FEGP feature insertion mutation.

Terminal selection during population initialization.

To generate the initial population, trees are built by randomly picking functions and terminals when needed. However, when the data set contains many input variables there is a high probability that some variables are less frequently selected or may not be used at all. This greatly reduces the exploration in syntax space, specially during the initial generations. Mutated trees follow the same mechanism, suffering from the same shortcomings. To avoid this, a new terminal selection method is introduced by employing a sampling without replacement procedure. When a terminal is randomly selected then it is excluded on the subsequent terminal selection step. Once all available terminals have been chosen, then the procedure is repeated by making all terminals available. This produces a uniform distribution in the selection of terminals, defined by f req_ter, which can be seen in detail in Alg. 6.3.1. The algorithm's output is a chosen terminal T from the terminal set T .

4. Genetic operator selection. In canonical GP, operators are chosen at each generation with an user-defined probability. The probability of choosing each operator is usually static throughout the search. In the work by [START_REF] Tuson | Adapting operator settings in genetic algorithms[END_REF], their operator probabilities are changed at each generation by rewarding or penalizing operators according with their success his-Algorithm 6.3.1 Terminal selection. f req_ter is a global array during the tree construction process.

Input: Terminal set (T ),Terminal frequencies (f req_ter = {0}) Output: T 1: b ← minimum element in f req_ter 2: index2f ill ← indexes from f req_ter equal to b (least frequent elements) 3: t ← all indexes in T 4: index ← random index from index2f ill 5: The frequency of the chosen terminal is incremented by one:

f req_ter(index2f ill(index)) = f req_ter(index2f ill(index)) + 1 6: Return the corresponding index in T : T = t(index2f ill(index))
tory to produce good offsprings. After an initial preliminary experimentation, we introduce probability functions that mimic this aforementioned adaptation.

Based on the implementation in [START_REF] Tuson | Adapting operator settings in genetic algorithms[END_REF] three operator probabilities (intra-crossover: p IntraC ; inter-individual feature crossover: p InterF ; and subtree mutation: p M ) have a common dynamic behavior through the initial experimental runs, with the following characteristics: it was periodic, the variation range was rather small and it was stationary. This means that the probabilities for these operators had to be almost constant to produce good offsprings during the search. Therefore, choosing static values for these operators seemed an appropriate decision for our work. Moreover, it produced almost identical results during initial experimentation compared with the algorithm by [START_REF] Tuson | Adapting operator settings in genetic algorithms[END_REF].

The other three operators (inter-crossover: p InterC ; feature insertion mutation: p K + ; and feature removal mutation: p K -) exhibited unique dynamic behaviors, which were implemented as p InterC (gen) = -∆ InterC log(gen) log(gen max + 1) + pInterC (6.11)

p K -(gen) = 1 -p InterC -p InterF -p IntraC -p M -p K (+) , (6.13)
where gen is the current generation, gen max is the maximum generation number, -∆ InterC is a decay parameter and pInterC is an initial probability. At the beginning of the search, the genetic exchange between different individual promotes a quick exploration, hence the high probability for the inter-crossover operator. However, in latter generations, this could be ineffective, where exploitation is desired; it is more beneficial to reuse material within same individual rather than from others. A decaying logarithmic function for the inter-crossover probability replicates this behavior. Programming Two of the most important operators are the feature insertion and removal mutation. Indeed, these are the ones responsible for the flexibility of the new feature space size. There is a higher change that the hyperplane of the classifier has a good class separability when the number of dimensions is high. A high probability for the generation of new features (p K + ) at the start of the run greatly promotes the search for the most promising size of the new feature space. The ratio between these operators and the static ones decreases after the initial generations, giving more importance to the evolution of the solutions without changing the feature space too much. Because evolved models stagnate at the end of the run, increasing the feature space size is required once more. Moreover, p K -is seen as the complementary part of p K + and p InterC . The operator functions are plotted in Fig. 6.17 for the case that gen max = 50.

5. Fitness calculation. The fitness is computed by a local and a global measure, which are used differently by the search. More specifically:

(a) Local fitness measure (Fisher's Discriminat Ratio). This is computed at the subtree level, which provides a separability value for each constructed feature. Certainly, the quality of a solution depends on the quality of their individual features. The local measure f ′ is only used as the criterion for the inter-crossover subtree selection, that is crossover points are selected within the subtrees with the best local fitness. Although the GP selection process chooses two individuals with similar fitness rank it does not guarantee that all individual subtrees have a good performance. We are not interested to interchange material from transformation subtrees with poor performance, rather we prefer a greedy approach where only material from the best solutions are used. This is given by

F1 i = (µ C1 i -µ C2 i ) 2 σ 2 C1 i + σ 2 C2 i i=1...r , (6.14)
where µ C1 , µ C2 , σ C1 and σ C2 are the means and standard deviations corresponding to class one and two for the ith feature.

For any given tree K, r local fitness measures are calculated.

It was found that the use of the local fitness in other operators did not help the search. (b) Global fitness measure. Due to the complexity of the problem which could lead to over-fitting, the global fitness depends on two regularization procedures. Let the global fitness be defined by

f ′ = (100 -Acc) + ϵ, (6.15)
where Acc is the classification accuracy and ϵ is a first regularization term. Moreover, Acc is calculated with

Acc = TP + TN TP + TN + FN + FP , (6.16)
where TP (True Positive), TN (True Negative), FN (False Negative) and FP (False Positive) are values from the confusion matrix as a result of the labels γtrain obtained by the LDA classifier. The term ϵ is proposed as the first regularization term, given by ϵ = 0.5 -FN FN + FP , (6.17) which becomes zero if FP and FN are equal. In preliminar experimentation it was found that the algorithm tended to avoid early over-fitting when there was a balance in the FN and FP scores. In this work, we need first to compute the second regularization, defined as K ∈ R r → β ∈ R r and β = β i ∀i ∈ {1 . . . r}, to calculate the accuracy and the confusion matrix. This is achieved by artificially changing the covariance of the tree outputs, and it is defined by

β i (x) = [α C1 i ({K(x)| C1 } -µ C1 i ) + µ C1 i , α C2 i ({K(x)| C2 } -µ C2 i ) + µ C2 i ],
(6.18) and

α C1 i = 1 + 2σ C1 i max({K(x)| C1 }) -min({K(x)| C1 }) ,
(6.19) 6.20) where {K(x)| C1 } denotes the tree output corresponding to all the data samples from class C1 (normal) and {K(x)| C2 } is the same but corresponding to class C2 (relaxed). The main reason for this is that data variance is high in this problem and the classifier training has to be relaxed so that prediction accuracy on unseen data increases. Please note that the actual penalty is given by a factor of twice the standard deviation of data produced by the tree outputs. This is exemplified in Fig. 6.18. Afterwards, the LDA hyperplane can be computed using Eqs. 6.7 and 6.8 by using δ as the input data and then obtaining predicted labels. These labels are then used to compute the accuracy.

α C2 i = 1 + 2σ C2 i max({K(x)| C2 }) -min({K(x)| C2 }) , (
Finally, it is important to mention that although the LDA classifier was used during training, it was meant only for driving the search so the class separability increases. Once + FEGP finds a new transformation model, any classifier can be used during testing. This offers flexibility in terms of the algorithmic choice, specially when dealing with a broader range of classification problems. Two examples of the data distribution of each class are depicted in figures 6.26 and 6.27, corresponding to the folds with the best and worst performance on the testing data. Left scatter plots show the first two principal components from PCA calculated over the raw signals. The middle plots correspond to the data after the CSP (matrix Z) calculation. Note a slightly different distribution between both folds, because the CSP is performed with different training data. The plots on the right depict the data distribution after a randomly chosen run of the + FEGP algorithm. The classification accuracy for Fig. 6.26 is 82.8% for training and 99.1% for testing. The homologous values for Fig. 6.27 are 81.4% (training) and 44.1% (testing).

D I S C U S S I O N

One of the main contributions of this work is the proposed feature extraction method, + FEGP. Generally speaking, the classification accuracy in the training phase, as seen in Fig. 6.19, surpasses the reference system. Furthermore, the system's classification accuracy on unseen data also improved over the reference system, as seen by the testing performance reported in Fig. 6.20.

If we take a look at the system performance for each fold (Fig. 6.21), we immediately see that some data partitions lead to better classification accuracies than others. Fig. 6.26 and Fig. 6.27 show the best and worst cases in terms of quality in more detail. After dimensionality reduction with PCA we can recognize that the volume of the overlap region for both classes is high in both cases for the raw data. Intuitively, we can deduce that for the majority of state-of-the-art classifiers the performance will be poor in such circumstances. The effectiveness of CSP can be clearly seen afterwards, with a substantial increase in the separability of the classes. At this point we can see that the testing cluster is quite different between almost all folds, being fold 4 and 8 the extreme cases. For fold 4 (Fig. 6.26), the testing data matches the distribution of the training data, making it easier for a classifier to obtain good generalization results. On the other hand, in fold 8 (Fig. 6.27), the case is the opposite; the data cluster belongs to a multi-modal distribution, where the obtained model is evolved over a different mode than where the testing data resides, suggesting that the EEG recordings for that particular subject are quite different from the rest. Lets recall that although the experiments were done in a controlled environment we cannot fully constrain the physiological activity of each subject. This is a more realistic scenario, but it makes the problem more difficult to solve. Furthermore, we can see the benefits of the + FEGP algorithm in the third scatter plot of each figure. For fold 4, the new features make the problem quite trivial for almost any classifier. The robustness here is that the testing data does not shift nor varies, rather it integrates to the counterpart training samples. The more difficult case is fold 8, which given an already problematic situation from the CSP output, the + FEGP improvement is relative small. Here, an important observation is that even for this worst case, the + FEGP performance is at least the same or better, but not worse than the performance of the reference system. Programming We can analyze further the performance for the remaining folds in the LOOCV. The Wilcoxon test infers that the majority of the folds have a similar statistical performance in terms of their medians in the training phase, with the exception of fold 3 and 13. Although these are the extreme cases for training, their performance is an improvement upon the CSP output. This can be extended to all folds; indeed + FEGP produced almost a linear improvement over all folds compared with the reference system. In the testing phase, almost all null hypothesis combinations were rejected, something expected giving the nature of CSP output. However, the improvement was not linear among all folds; in folds 1, 3, 4 and 13, the improvement by + FEGP was significant, but in folds 5, 7 and 11 there was a small reduction in performance (see figure 6.20), suggesting the obtained models over-fitted. Moreover, according to the Friedman test, the hypothesis that samples from the algorithm's outcome belong to the same distribution was accepted for almost all folds during training, with the exception of folds 3 and 13, if we compare them with the rest. During testing, the hypothesis was rejected for almost all combinations.

The performance of the enhanced system can be seen from different angles. At the bottom plot of Fig. 6.22 we can see a steady minimization of the global fitness. At the same time, we can analyze the performance of the LDA classifier calculated directly over the transformed data. Although training fitness is decreasing during the runs, specially at the end, the classification accuracy does converge, and more importantly, the accuracy on unseen data does not decrease in the final iterations of the search (Fig. 6.23).

Moreover, if we consider solution sizes in + FEGP, shown in Fig. 6.24 (plot lines represents the mean of 13 medians over 30 runs), solutions grow almost linearly during the search. This is a common result in GP with tree representations, where the increase of solution size is a product of fitness improvement [START_REF] Koza | Genetic Programming: On the Programming of Computers by Means of Natural Selection[END_REF]. A side effect of this phenomena is that sometimes fitness becomes stagnated with unnecessary growth of trees, referred to as bloat [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF]. However, in our experiments with a relatively small number of generations for the search, + FEGP does not exhibit any bloat, that is the increase in solution size is always accompanied by a steady improvement in fitness. This again is made possible by the regularized fitness measure employed by + FEGP.

The increase of solution sizes also means that there is an increment in the number of newly created features, the depth in the tree structure or both. Specifically, we can seen the frequency histogram for all runs and folds in Fig. 6.25, showing a nearly normal distribution. All individuals were started with 3 features at the beginning of the runs. The evolved models produced a minimum of 6 and a maximum of 33 new features, with an average of 17.35.

As mentioned earlier in Section 6.3, there is an inherent flexibility in terms of the classifier selection, after using the + FEGP. Therefore, additional classifiers were evaluated: Tree Bagger (TB), Random Forests3 (Jaiantilal, 2012) (RF), k-NN and SVM. All of them produces non-linear decision functions, as opposed to the relative simpler LDA. With the exception of RF, all classifiers are based on their MATLAB implementations. No parameter tuning was performed for these, rather their corresponding default settings were used. The classification accuracies are summarized in Table 6.4. A one-way Analysis of Variance (ANOVA) was performed among the classifiers to estimate statistical significance on the null hypothesis that samples belong to the same distribution. Correlating the results from Table 6.4, Fig. 6.26 and Fig. 6.27 we can deduct that for this particular problem, complexer classifiers do not produce better accuracies for unseen data, rather they all over-fit. One reason for this, is that most of these algorithms perform accurately if there is enough data to learn, however for the case of under-sampled data, sometimes simpler models produces better results. The simpler assumptions of LDA (which in this case are actually infringed: class distributions are not normal nor share the same covariance) help to relax the model and behaves better as a generalization method. p-value 0.0000 0.0567

6.6 C H A P T E R C O N C L U S I O N S
In this chapter we propose a new system for the classification of two mental state based on EEG recordings, with the introduction of a novel GP algorithm that performs as a feature extraction method. The proposed system involves several functional blocks: acquisition of signals, pre-processing, feature extraction and classification. This work is the first to use a GP based algorithm as feature extraction for the identification of mental states; specifically, normal and relaxed states.

A reference system was implemented based on our previous work [START_REF] Vézard | EEG classification for the detection of mental states[END_REF], which produced competitive results among similar research in this domain. Nevertheless, the proposed system outperformed our previous results by exploiting further the separability of classes using evolved transformation models. Certainly, apart from the appropriate choice of the classifier, one of the most important blocks in these type of systems is the feature extraction process, where the pattern recognition of the underlying data is performed.

Evidence found during experimentation allow us to conclude in the following directions. First, the + FEGP algorithm in a wrapper scheme produced improved performance compared to the system with just CSP, reaching a classification accuracy of 78.8% for unseen data. Certainly, any system could efficiently benefit from the hybridization of different tools to achieve competitive results; a sole technique usually cannot tackle such a complex problem. Second, the proposed genetic operators allows us to evolve transformation models with good testing performance. Third, the unique fitness function steered the search toward solutions that avoided over-fitting in most of the cases. Fourth, the sizes of the evolved solutions are relative small in terms of the number of new features and the number of nodes per subtree. Moreover, in our tests + FEGP was not affected by bloat [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF]. Fifth, although from the system design point of view the classifier can be interchanged, other classifiers did not produce better results compared with LDA. Nevertheless, when examining the whole scenario we can state that most of the shortcomings were derived from the underlying data properties. In this case, we could opt for three options: increase the system's complexity by incorporating an additional processing step and help reduce further issues like over-fitting, use adaptive classifiers or increase the number of EEG recordings. In either of the circumstances, we can foresee applications of the + FEGP outside of this domain, that is by merely adjusting the fitness function it can be used in any data transformation scenario.

7

C O N C L U S I O N S A N D F U T U R E W O R K 7.1 G E N E R A L D I S C U S S I O N
We can say that there were two original motivations behind this research. First, the implications that a certain human mental state can have in a scenario associated with a high risk, thus the importance to design automatic human-computer interaction systems; and second, the design of new algorithms aimed to improve existing BCI systems. Despite those being our original incentives, this journey turned out to be richer in terms of the obtained insights in several aspects. While the majority of proposals that address the problem of mental states recognition rely on methods from statistical learning and signal processing, other methods are usually required to deal with the highly complex nature of the task. In particular, meta-heuristic algorithms can be a viable alternative to achieve state-of-the-art results. However, there is only a small number of works covering meta-heuristic techniques, if we compared them with classical approaches. This fact may be due to the following reasons: there are still open issues in the meta-heuristic algorithms that prevent them to efficiently solve the problem; the field has not matured to the same level as classical statistical learning and signal processing; or simply, there is not sufficient interest to use these techniques, possible to the lack of theoretical foundations behind their success. We feel that the third reason is too harsh, since important theoretical results have been published [START_REF] Langdon | Foundations of Genetic Programming[END_REF], while other heuristics are commonly used in similar fields. Therefore, in our work we present contributions to help address the first two concerns.

This work began with the study of the classical approaches used in the design of BCI systems. The advances in such techniques have been increasing in recent years, where the combination of different techniques produce the most successful results. We recognized that designing a hybrid methodology might deliver competitive results.

In the first part of the manuscript, we proposed a new feature extraction methodology to deal with the recognition of epileptic seizures. By combining three different types of measures we build a feature set. MP allows us to extract a global characterization of the EEG signals, using density values of the number of Gabor atoms for a given epoch. On the other hand, the Hölderian regularity provides a more detailed characterization, the local regularity (around the neighborhood of a given sample) of the signals. The last set of measures, given by mere statistical calculations over the raw EEG signals, reside in the time domain, as opposed to the first measure which is produced in the frequency domain. Notice that this particular combination of measures collects different levels of information from the raw signals. The experimentation produced competitive results that outperformed, in the majority of the cases, a vast collection of other state-of-the-art methodologies. Nevertheless these encouraging results, the proposal allows us to identify the difficulties normally found when designing BCI systems, a rather common overstatement that new researchers in the field need to consider.

Afterward, we proceeded to study the current state of meta-heuristic algorithms and their application in the domain of BCI design. There is a large number of such algorithms, that are derived from very contrasting rationale. Instead of analyzing this vast group, we focused on a particular bio-inspired algorithm, namely GP. This is a paradigm from the field of EC, in which GP stands out because the solutions representation make it ideal to handle a wide variety of applications. In particular, GP can be used for feature selection, feature extraction or classification; almost every part of a complete BCI system.

However, GP has several open issues that have been the focal point of much research in the field. One of these open issues is the way the search is performed. It has been studied before that the search is not optimal in terms of the solution size or convergence time. The majority 7.1 General discussion of the meta-heuristic algorithms perform a type of exploration search, however the exploitation can become cumbersome mainly because it is not explicitly expressed as a mechanism of search. A meme, a term introduced by Richard Dawkings, was adapted to the computational field and incorporated into meta-heuristic algorithms, namely memetic algorithms, where exploration and exploitation techniques are combined. By incorporating a local neighborhood search, the synergy produced by these two mechanisms results in a faster convergence. The majority of works focus on representations that are rather simple to implement, such as real valued representations. In that sense, GP is still behind, mainly because its representation is rather general, where the adaptation of LS procedures is more difficult.

We recognized that GP is able to evolve the solutions syntax, but the optimal parametrization is never explicitly included as an objective during the search. Thus, we proposed the following hybridization:

1. parameters are added to each node in the tree; 2. an upper linear tree is inserted to the original tree; 3. a heuristic to select which individual has to be optimized is defined.

In the first item, the aforementioned space is explored by a numerical optimizer guided by the same objective function as the global searcher, essentially producing a LS in the solution (program) output landscape. The outcome is that for a particular solution, the optimal parameters are found without changing the structure.

The second item, a more trivial idea, was meant to help the search during the numerical optimization, by adding a bias and a scalar value to the non-linear part.

For the third item, the heuristic for the LS selection is considered to play an important role in the memetic mechanism. For that, we tested a series of different variants, concluding that two different options can be opted depending on the needs: a fitness and a size based heuristics. In the former, a subset of the best solutions in the population turned out to produce the best performance. In the latter, choosing relative small solutions in the population produced a faster convergence. Extensive experimental evidence was collected to validate the initial conjectures. First, in a set of regression problems and later in classification problems. Derived from these experiments, we conclude that GP greatly benefits from the incorporation of LS strategies. In summary, the best solution outperformed canonical GP in terms of quality and size. If this particular issue in GP cannot be considered as closed, we can surely state that this contribution is a small step towards this important goal.

As previously stated, the recognition of mental states is a non-trivial task, that requires the involvement of different approaches. For that, we separated the last part of the thesis in two studies. We first built a BCI system based on classical techniques. Specifically, we used CSP as feature extraction and a LDA classifier. Then we designed an improved system which incorporates a GP based algorithm ( + FEGP) with the purpose of augmenting the CSP output, in a wrapper scheme. + FEGP was built with the rationale behind to evolve transformation models of the form K : R k → K ′ : R r , being K ′ a new feature set. The search was guided with the objective of improving the classification accuracy for the obtained K ′ output. By trying to tackle the problem of mental states recognition, another open issue in GP was addressed; a proper representation, which was designed according with the needs of the problem at hand. New search operators were also introduced with independent and specific functions that combined produced a more efficient search for the analyzed problem. A fitness function was defined as well, to complement the other introduced elements into GP. This function included components that helped to reduce over-fitting issues and allowed to evolve models that performed well with testing data. If we oversee this work, we can discover two important contributions: the proposal of a novel approach for an efficient BCI design, and the improvement provided to GP as an stochastic optimization algorithm. We can satisfactorily conclude that we fulfilled the original objective of the thesis.

F U T U R E W O R K

The conclusions obtained during the development of this thesis could be considered compelling and produced new opportunities in terms of future research. For that, we will focus first on each part, following the same order as the document, and finally propose a general future research.

For the first part of this thesis, one important aspect to consider is to reduce the computational cost of the feature extraction process, for online or real time implementations. However, there are several ways to cut corners and produce a more efficient method. Hölderian regularity computation can be simplified substantially by relaxing the approximation accuracy and imposing a smaller analyzing window, as done in other works. Alternatively, accurate and efficient approximations can be obtained by using meta-heuristic methods [START_REF] Trujillo | Evolving estimators of the pointwise hölder exponent with genetic programming[END_REF], for which the computational costs are dramatically reduced. MP can also be improved by using a pre-built dictionary employing an optimized family of functions according with a subset of already recorded signals; such a task can be performed offline. Improvements of both techniques would improve the possibility for an online implementation of the proposed approach. Future work will also focus on feature postprocessing, which might further improve classification accuracy. The post-processing might be useful to fully exploit the decorrelation hidden in the extracted features. Additional pre-processing techniques might also be effective in boosting performance; e.g., Hölderian regularity has proved to be a powerful tool for denoising signals [START_REF] Legrand | Local regularity-based image denoising[END_REF].

For the second part of the manuscript, there are several future directions that can be taken. In terms of efficiency of the memetic GP approach, future work will be centered on exploring and evaluating other parametrization schemes. Moreover, a method must be implemented that determines if a tree requires linear or non-linear parameter optimization, in order to simplify the process whenever possible and to use the parameter values as decision criteria to prune unnecessary subtrees. Although our proposal focused in a single algorithm as the LS optimizer, more efficient algorithms could be more appropriate alternatives. It was found that a global optimum in the LS process is not needed, rather an approximation is sufficient. Thus, a gradient-free algorithm could replace the LS mechanism. Another idea regarding the efficiency aspect, is to develop a surrogate LS mechanism along the GP evolution, where the characteristics of the LS influence can be modeled by another GP population and then replaced as an artificial LS. This would reduce greatly the computational burden of the memetic approach. As previously stated in this manuscript, one of the open issues in GP is a proper representation, which was briefly discussed as a possible difficulty when adapting a LS mechanism to other type of data representations. For that, further research is needed to expand the LS design to cover a wide range of representations.

In the third part, there are several aspects to consider in future work. One is the introduction of feature selection into the system; ultimately, channel reduction is very important for any real-world scenario, mainly for practical reasons. It is also of our interest to increase the number of mental states in future research. Although we emphasize the benefits of the proposed + FEGP algorithm in this work, the exploration of different tools in the pre-processing and post-processing is also important. Moreover, we do not express by any means that the + FEGP algorithm in conjunction with CSP is the best option, rather a choice, that proved to be very competitive in this domain. Another aspect is the improvement of the + FEGP algorithm itself. Further investigation is required to study the evolution behavior; there are still open questions: what is the relationship between the number of features and the efficiency of individual subtrees? Is there a way to fully avoid bloat in longer searches? Are there better genetic operators that allow to search more efficiently? In our previous works, by implementing a type of LS into GP we achieved better results in regression Z-Flores et al. ( 2014) and classification problems [START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], suggesting that GP in general is benefited from the hybridization of search operators. This encourages us to extend these approaches into this domain. Despite not mentioning computational costs along the presentation of this work, which at the moment is considered as an off-line methodology, it is important to reduce algorithm complexity and at the same time increase its accuracy. In future work, a full study will be done balancing the consumed resources and efficiency, such as by using intrinsic parallel implementations, such as Graphics Processing Units (GPUs) or Field Programmable Gate Arrays (FPGAs).

Finally, generally speaking, the future research direction derived from this thesis can be quite abroad. If we want the influence of BCIs to be extended further into our daily lives, their design must be improved in terms of accuracy and efficiency. For instance, given the complex nature of EEG data for these type of problems, a GP algorithm can be designed in a way that it benefits from specific patterns extracted from the raw signals and dynamically adapts to them, without hand-tuning.

However, to develop an online system, a fast learning algorithm is needed. Here, the advances in approaches for time series forecasting can be really useful, where a delay in the processing might exists but the prediction is performed in real-time.

A more ambitious goal is to develop a complete BCI system by using meta-heuristics, memetic algorithms and a hyper-parametrization mechanisms. An example of this are the memetic algorithms of second and third generation, namely hyper-heuristic and meta-Lamarckian strategies. The actual number of works in GP addressing such systems are rather scarce, a situation that encourages us to work toward this ambitious area. version of the Matlab GP toolbox GPLab [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF]. The GP parameters are given in table A.1. In what follows, we will present results based on the median performance over all runs. The fitness function used is the RMSE, and the stopping criterion is the total number of fitness function evaluations. Function evaluations are used to account for the computational cost of the trust region optimizer, which in this case is allowed to run for 100 iterations. Results are compared with a standard GP. However, our strategy is to apply the LS on the smallest individuals of the population. This is clearly validated in figures A.1(c) and A.1(d). Figure A.1(c) shows the raw improvement of test fitness for each individual before and after the LS. A similar plot is shown in figure A.1(d), however instead of showing the raw improvement, this figure plots the improvement in rank within the population. In both plots the average program size is plotted with a white line, and individuals that were not passed to the local optimizer have a zero value. These plots reveal that: (1) most individuals below the average size are improved by the LS; and (b) the largest improvement is exhibited by individuals that are only slightly smaller than the average program size. The median of the average program size produced by standard GP on this problem is 123.576, which is substantially higher than what is shown by GP-LS.

These results present three interesting and complimentary results. First, GP-LS clearly outperforms standard GP, in terms of convergence, solution quality and average solution size. Second, the LS is clearly improving the quality of the smallest individuals in the population, in some cases substantially. On the other hand, and thirdly, the best solutions are still the largest trees in the population. This means that while the LS operator improves smaller solutions, the best solutions are not necessarily directly subjected to the LS process. This means that the LS process should be seen as an important additional operator, that complements the other search operators. While many previous works have applied a LS process on the best solutions found, our results indicate that this is insufficient, the LS should be applied more broadly to achieve the best results.

Figure A.2 summarizes how the LS operator influences the construction of the best solution. First, figure A.2(a) shows how many times the best solution in the population was chosen by the LS selection heuristic for each run. The plot indicates that the best solution was chosen about 50% of the time. Second, we track all of the ancestors of the best individual from each run, and figure A.2(b) plots the percentage of ancestors that were subjected to the LS. This plot also suggests that, on average, about half of the ancestors were subjected to the LS and half were not. Based on Enríquez-Zárate, J., Trujillo, L., de Lara, S., Castelli, M., Z-Flores, E., [START_REF] Enríquez-Zárate | Automatic modeling of a gas turbine using genetic programming: An experimental study[END_REF]. Automatic modeling of a gas turbine using genetic programming: An experimental study. Applied Soft Computing, 50:212-222

B.1 I N T R O D U C T I O N
G as turbines (GTs) are one of the key elements in the electrical power industry since they transform mechanical power into the desired electrical output. In particular, GTs are widely used in so-called Open Cycle Plants and Combined Cycle Plants. They are considered to be high-risk units since they operate at high speed, temperature and pressure, conditions that impose strict control requirements. The control systems executes the startup and shutdown sequences, control speed, power and temperature, while also performing functions related to safety and protection. Startup and shutdown sequences consist of a series of complex events where each step has to be satisfactorily completed before continuing to the next. Moreover, in central power generation, it is of utmost importance to ensure high levels of availability and reliability of the generating units, particularly due to the high costs associated to system failure or an interruption in the availability of electrical power. Therefore, condition monitoring of GTs within power plants is highly desirable to guarantee the long-term safety of the units and their efficient operation [START_REF] Sekhon | Stationary gas turbines -a real time dynamic model with experimental validation[END_REF].

For instance, a condition monitoring system can be connected in parallel to the control system for the early detection of potential abnormalities. This can allow supervisors to perform an orderly shutdown of the unit, thus avoiding unexpected stops or failures of a greater magnitude [START_REF] Sekhon | Stationary gas turbines -a real time dynamic model with experimental validation[END_REF]. An important element of such a monitoring system is the model that is used to describe or predict the behavior of the unit. If the models are sufficiently accurate, they can be used to detect unwanted deviations from what is considered to be a normal behavior. Indeed, many different physical models of GTs have been developed in the past [START_REF] Sarwar | Time series method for machine performance prediction using condition monitoring data[END_REF]. These models capture the dynamics of the turbine, but their precision can significantly vary based on the specific unit that is used. In practice, it is not always possible to have complete mathematical models that describe all of the turbine behaviors accurately. Therefore, another alternative is to develop statistical or computational models that might provide a more accurate characterization of the behavior of such units.

Several authors have developed dynamic mathematical and computational models, as well as specialized test equipment for GTs [START_REF] Sun | Application of a state space modeling technique to system prognostics based on a health Bibliography index for condition-based maintenance[END_REF]. The modeling of GTs for diagnostics and prognostics has been carried out with a variety of techniques, which can be categorized into two major groups: physics-based methods and data-driven methods. The physics-based methods are limited to the cases where failure mechanisms can be quantified [START_REF] Sun | Application of a state space modeling technique to system prognostics based on a health Bibliography index for condition-based maintenance[END_REF], and require specialized personnel to aid in the modeling process as well as a detailed description of the particular system to be modeled with specific technical information from the manufacturer of the equipment, which are often not available. The data-driven methods attempt to derive the models directly from recorded system data without digging into the physics of the system, using tools derived from artificial intelligence, machine learning, statistical analysis and computational intelligence [START_REF] Sun | Application of a state space modeling technique to system prognostics based on a health Bibliography index for condition-based maintenance[END_REF]. For instance, a popular approach is to use neural networks [START_REF] Depold | The application of expert systems and neural networks to gas turbine prognostics and diagnostics[END_REF][START_REF] Delgadillo | Modelling and dynamic simulation of gas turbine[END_REF].

This work proposes the use of genetic programming (GP) [START_REF] Koza | Genetic Programming: On the Programming of Computers by Means of Natural Selection[END_REF][START_REF] Langdon | Foundations of Genetic Programming[END_REF][START_REF] Poli | A Field Guide to Genetic Programming[END_REF] to model a Mitsubishi single shaft Turbo-Generator, Model MS6001, with a 30MW generation B.1 Introduction capacity. GP has recently gained popularity given its ability to generate syntactic models in a wide variety of problem domains [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF], in many cases outperforming more common machine learning paradigms [START_REF] Trujillo | Evolving estimators of the pointwise hölder exponent with genetic programming[END_REF]Castelli et al., 2015c,b,d) while using very little in the way of prior knowledge. While EAs have been used before in GT systems, they are mostly used for optimizing or improving system efficiency using previously derived models [START_REF] Chaquet | Using genetic algorithms to improve the thermodynamic efficiency of gas turbines designed by traditional methods[END_REF]. To the extent of our knowledge, this is the first work to address the GT modeling problem directly using a GP approach. Moreover, this work compares recent proposals in GP literature, including the fairly recent geometric semantic GP (GSGP) [START_REF] Moraglio | Geometric semantic genetic programming[END_REF]Vanneschi et al., 2014a) and improved variants of standard GP [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF]. The experimental work compares several variants of these methods, particularly emphasizing algorithms that combine the global search process provided by the basic GP paradigm with greedy local search operators. Such combinations have been shown to improve algorithm convergence and predictive accuracy (Castelli et al., 2015e), while also reducing model size and complexity [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF]. Comparisons are also made with seven non-GP modeling methods, showing that GP search outperforms all other methods in terms of predictive accuracy and model size, particularly on the most difficult modeling task.

The remainder of this work proceeds as follows. Section B.2 describes the power generation process of GTs in general and the particular unit studied in this work, describing how data was collected to apply the data-driven approach. Section B.3 focuses on the specialized variants employed in this work and developed by the authors. The experimental work and results are presented in Section B.4. Finally, conclusions are outlined in Section B.5.

B.2 G A S T U R B I N E O P E R AT I O N A N D C A S E S T U D Y

This section provides a brief introduction to GTs and how they are used for power generation, as well as a detailed description of how the dataset used in this study was extracted.

B.2.1 Process Description

A GT power unit is mainly composed of a starting device, a compressor, a combustion chamber, a turbine, an electric generator and other auxiliary components such as a fuel system, a lube oil system, a starting device and other subsystems. A graphical depiction of a generic GT power unit is shown in Figure B.1.

The starting device is a diesel engine, it assists the GT unit during the startup phase to overcome inertia and to accelerate the GT up to the ignition speed. The centrifugal compressor pressurizes the incoming air and sends it to the combustion chamber. High-pressure combustion exhaust gases are expanded within the turbine stages to rotate the unit. The electric generator that is coupled to the GT can generate the desired electricity when it operated at the rated speed. Finally, the combustion exhaust gases are discharged to the atmosphere (CFE, 1998).

B.2.2 Case Study

The case study of the current paper considers a Mitsubishi single shaft Turbo-Generator Model MS6001, which has a 30MW generation capacity. The plant is located in Cozumel, Quintana Roo, in the southeast of Mexico in the complex known as the Chankanaab Turbo-Gas Central (see Figure B.2).

For this model, the combustion zone is formed by 10 combustion chambers and nozzles with their respective baskets. It also has two spark plugs and two flame detectors. In this area the oxygen supplied by the compressor is combined with the diesel fuel delivered by spray nozzles. Afterward, the spark plugs ignite the mixed fuel to produce the required combustion, generating a large volume of gases.

The gases produced by the combustion pass through the turbine which consists of three stages of blades. The power of the exhaust gases is thus converted to mechanical energy. However, the temperature at the turbine inlet is not directly regulated or monitored. The control of the gas turbine is done by monitoring the exhaust temperature, which should not exceed 1067 • F. This temperature is then used to regulate the fuel supply to the turbine. The liquid fuel system delivers a high pressure flow modulated by a servo controlled bypass valve assembly. The servo valve controls the bypass valve stroke according to the difference requirement and the sensed fuel flow. Fuel oil goes to a flow divider whose purpose is the apportion of fuel oil to each of the fuel nozzles of the turbine. The distribution has ten pump elements (in line) with the inlet port located at the mid-point where the fuel oil enters the unit and is distributed by an internal manifold to the inlet side of each pump element. The flow from each pump element is proportional to the speed at which the unit operates.

Fuel flow is measured by magnetic pickups and represents the feedback signal in the outer control loop of the system. The speed of the flow divider is a direct measure of the fuel flow nozzles in the combustion chambers, as depicted in Figure B.3. The correct measurement of the fuel flow is crucial for a good control performance. Concerning the exhaust gases temperature, it reflects the fuel flow variations; therefore, it is also critical for the control system to maintain a temperature profile that preserves the unit from any thermal excess. In both cases (fuel flow and exhaust temperature), it is also desirable to have accurate predictive models that can be used as monitoring tools. These models can be used to detect deviations over time from the expected behavior of the GT unit, thus anticipating possible failure situations.

The unit is equipped with a control system that offers a user interface to operators. The interface collects data from the system using specialized acquisition modules, providing a historical database for analysis purposes. The monitoring process of this plant can therefore be used to collect a representative dataset related to flow and temperature in the GT. Table B.1 shows the available data related to fuel flow that is provided by the system, while Table B.2 shows the same for the exhaust gas temperature.

B.2.3 Problem Statement and Data

The goal of this work is to derive predictive models of both the fuel flow and the exhaust gas temperature. In particular, the intention of the fuel flow prediction problem is to find a function K f that takes the reference fuel flow rf and the turbine speed ts as inputs and produces as output an approximation of the actual fuel flow f f of the GT unit; i.e., K F (rf , ts) ≈ f f . Conversely, the goal of the exhaust gas temperature prediction problem is to find a function K T that takes the fuel flow f f and the compressor discharge pressure cp as inputs and produces as output an approximation of the actual exhaust gas temperature et of the GT unit; i.e., K T (f f , cp) ≈ et. In both cases, both K T and K F should approximate the fuel flow and the exhaust gas temperature, respectively, within the normal operating ranges of the independent variables rf , ts, f f and cp. In both cases, our intent was to use a small amount of independent variables that could be easily and reliably monitored, while capturing sufficient information to predict the dependent variables. Therefore, in both problems the desired models (K F (rf , ts),K T (f f , cp)) take two measurements as inputs and produce a single output.

The GT unit was allowed to operate in normal conditions after a control tuning process during the commissioning phase. At this time, the above mentioned independent variables were recorded. In the case of fuel flow prediction, the GT unit operated for approximately 23 minutes, using a sampling period of 100ms, recording a total of 13, 788 records (input/output pairs). For the prediction of the exhaust gas temperature, the unit operated for approximately the same amount of time with the same sampling period, producing a total of 13, 820 records. These datasets allowed us to pose two supervised learning problems and solve them with GP, as will be described next. Providing an exhaustive review of the cases where GP has been used to solve real-world modeling problems is beyond the scope of this work. Our goal is more modest, and provide some recent examples of successful applications of GP to solve complex modeling problems in real-world domains. In [START_REF] Cao | Evolutionary modeling of systems of ordinary differential equations with genetic programming[END_REF] the authors use a hybrid GP and GA algorithm to derive models based on systems of ordinary differential equations and higher-order differential equations, using several real-world applications to illustrate their approach which is based on tree-based GP. Recently, GP was applied to model the consumption of natural gas consumption in a real-world chemical plant [START_REF] Kovačič | Prediction of the natural gas consumption in chemical processing facilities with genetic programming[END_REF], outperforming the specialist prediction in minimizing costs to the plant. Another related example to the work presented here is [START_REF] Arellano | Prediction of jet engine parameters for control design using genetic programming[END_REF], where GP is used to model the exhaust gas temperature of a commercial jet engine, and [START_REF] Babaelahi | Analytical closed-form model for predicting the power and efficiency of stirling engines based on a comprehensive numerical model and the genetic programming[END_REF] where GP is used to aid in the practical modeling process of Stirling engines. The prediction of fuel consumption for a gasoline engine was addressed with GP in [START_REF] Togun | Genetic programming approach to predict torque and brake specific fuel consumption of a gasoline engine[END_REF], achieving very high correlation with the real behavior of the engine and a low predictive error. Modeling the startup process of an aero-engine is studied in [START_REF] Li | Linear-in-parameter models based on parsimonious genetic programming algorithm and its application to aero-engine start modeling[END_REF], using a specialized GP system to generate linear (in parameter) models, comparing with support vector machines shows that GP models outperform the standard technique and produce smaller and more parsimonious solutions. Similarly, [START_REF] Cava | Automatic identification of wind turbine models using evolutionary multiobjective optimization[END_REF] presents an approach based on a stack-based GP system for the automatic identification of the dynamics of a wind turbine. The approach posed multi-objective search, deriving models that are as accurate as those produced with competing techniques, whereas are relative more parsimonious, easier to analyze and interpret. Research has also shown that GP models can outperform standard regression techniques in predicting the material characteristics of specialized alloys [START_REF] Gusel | Application of genetic programming for modelling of material characteristics[END_REF], and focus on the fact that since GP is a population-based and stochastic method it can generate a variety of different models that can be considered for further analysis.

B.3.2 Genetic Programming with Local Search

In standard GP, only the symbolic expression is evolved by the search process, while the underlying parametrization θ is not considered. Recent works have shown that concurrently optimizing both the structure and the parametrization of the evolved models can speed up convergence and improve performance. In particular, we adopt our proposal in [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], which has been applied to both symbolic regression and classification problems. The approach can be summarized as follows.

First, as suggested in [START_REF] Kommenda | Effects of constant optimization by nonlinear least squares minimization in symbolic regression[END_REF], for each individual K in the population we add a small linear uppertree above the root node, such that

K ′ = θ 1 + θ 2 (K) , (B.1)
where K ′ represents the new program, while θ 1 and θ 2 are the first two parameters from θ.

Second, for all the other nodes n k in the tree K we add a weight coefficient θ k ∈ R, such that each node is now defined by

n ′ k = θ k n k , (B.2)
where n ′ k is the new modified node, k ∈ {3, ..., r}, r = |K| which is the total number of nodes (size) of K. Notice that each node has a unique parameter that can be modified to help meet the overall optimization criteria of the non-linear expression. At the beginning of the GP run, each parameter is initialized as θ k = 1. During the GP syntax search, subtrees belonging to different individuals are swapped, added or removed (following the standard crossover/mutation rules) together with their corresponding parameters, without affecting their values. This follows a memetic search process with Lamarckian inheritance [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], where its main goal is to preserve the information acquired during the local search process and pass it to the following generation. Therefore, we consider each tree as a non-linear expression and the local search operator will search for the best fit parameters of the model K ′ . The problem can be solved using a variety of techniques, but following [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF] we employ a trust region algorithm.

Finally, it is important to consider that the local search optimizer can substantially increase the underlying computational cost of the search, particularly when individual trees are very large. While applying the local search strategy to all trees might produce good results [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], it is preferable to reduce to a minimum the amount of trees to which it is applied. Therefore, we use the heuristic proposed in [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF], where the local search is applied stochastically based on a probability p(s) determined by the tree size s (number of nodes) and the average size of the population s, based on

p(s) =          1.5 -s s if 0.5s ≤ s ≤ 1.5s 1 if s < 0.5s 0 otherwise . (B.3)
In this way, smaller trees are more likely to be optimized than larger trees, which also reduces the computational effort and improves the convergence of the trust region optimizer by keeping the parameter vectors relatively small. We will refer to this version of GP as GP-LS.

B.3.3 Geometric Semantic Genetic Programming

In the last few years, a rising topic in the field of GP has been the use of what are referred to as semantic methods [START_REF] Moraglio | Geometric semantic genetic programming[END_REF]Vanneschi et al., 2014a;Castelli et al., 2015f, 2013. Semantics in GP can be loosely defined as a description of what a program actually does, instead of a description of its syntax. Specifically, for supervised learning problems with a training set T with n input/output pairs (x i , y i ), GP literature usually defines the semantics of a program K as the vector of program outputs s = {K(x 1 ), ..., K(x n )} [START_REF] Moraglio | Geometric semantic genetic programming[END_REF]. From this perspective, the vector of ideal outputs y = {y 1 , ..., y n } can be referred to as the target semantics and the fitness of a program K can be quantified by a distance between vectors y and s.

In particular, let us consider regression problems where x ∈ R p , and s, y ∈ R n . Then, the aim of GP is to find a program syntax that produces the target semantics of the problem. However, the search operators of standard GP operate at the syntax level with an indirect effect on the semantics of a program; i.e., standard GP operators cannot control or even constrain what will be the semantics of the offspring based on the semantics of the parents. This makes the GP search process inefficient, in the sense that while fitness is uniquely and directly defined by semantics the mapping between syntax and fitness is much more complex and difficult to predict. Therefore, Moraglio et al. [START_REF] Moraglio | Geometric semantic genetic programming[END_REF] proposed important new search operators that operate at the level of syntax but have a predictable and bounded behavior on program semantics. This is achieved by imposing geometric constraints within semantic space. These operators are referred to as geometric semantic operators (GSOs) and a GP algorithm that uses them is called geometric semantic GP or GSGP. These operators are defined as follows.

Definition 4 Geometric Semantic Crossover (GSC). Given two parent functions K 1 , K 2 : R n → R, the geometric semantic crossover returns the real function

K XO = (K 1 • K R ) + ((1 -K R ) • K 2 )
, where K R is a random function with normal distribution, domain R and codomain [0, 1].

Definition 5 Geometric Semantic Mutation (GSM). Given a parent function K : R n → R, the geometric semantic mutation with mutation step ms returns the real function K M = K + ms • (K R1 -K R2 ), where K R1 and K R2 are random real functions with normal distributions.

Using these operators the semantics of the offspring are completely defined by the semantics of the parents. The semantics of an offspring K XO produced by GSC will lie on the segment between the semantics of both parents. On the other hand, GSM defines a mutation such that the semantics of the offspring K M lies within the n-sphere of radius ms that surrounds the semantics of the parent. Finally, the computational cost of evolving a population of n individuals for g generations is O(ng) [START_REF] Vanneschi | Geometric semantic genetic programming for real life applications[END_REF], while the cost of evaluating a new and unseen in-stance is O(g) [START_REF] Vanneschi | Geometric semantic genetic programming for real life applications[END_REF]. This is the implementation used in this work.

B.3.3.1 GSGP with Local Search

While GSGP has been shown to produce strong results (Vanneschi et al., 2014a;Castelli et al., 2015f, 2013, its original formulation has several disadvantages. It is easy to see that GSC will not be useful when the semantics of the parents do not surround the target semantics. Similarly, GSM can sometimes produce offspring that have a worse fitness than the parent, which is unnecessary since the target semantics is known. Moreover, the offspring produced by GSOs will always be larger than the parents, meaning that program growth cannot be eliminated.

This work follows our proposal in (Castelli et al., 2015e), where a local search approach is integrated into GSM, an operator we will refer to as GSM-LS. This operator exploits the fact that the geometric mutation defines a linear combination of the parent K program with random programs, expressed as

K M = α 0 + α 1 • K + α 2 • (K R1 -K R2 ) (B.4)
where α i ∈ R; notice that α 2 replaces the mutation step parameter ms of GSM. GSM-LS poses a basic multivariate linear regression problem, which could be solved, for example, by Ordinary Least Square regression (OLS). However, in this case we have n linear equations given by the number of fitness cases, and only three unknowns (the α i s). This gives an overdetermined multivariate linear fitting problem, which can be solved through singular value decomposition (SVD)1 . In this way the GSM-LS will be guaranteed to produce the best linear fit based on the target semantics of the program. Hereafter, we will refer to a GSGP with GSM-LS mutation as GSGP-LS. This version of GSGP has performed very well in a variety of domains (Castelli et al., 2015c,b,d), in many cases outperforming standard GP, GSGP and several other regression techniques.

Table B.3: Parameters for all algorithms; all GSGP variants (GSGP, GSGP-LS, HYBRID, HYBRID-LIN) share these parameters. datasets were randomly partitioned before each run and, while training performance will be reported, the critical result will be the prediction accuracy of the evolved models over the test data. Table B.3 summarizes the parameters for all tested algorithms. For GSGP variants mutation and crossover rates have been chosen after a preliminary tuning phase that has been performed following the indication reported in (Castelli et al., 2015g). GP-LS parameters were set based on the values reported in [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], with only minimal manual tuning. Values for other parameters were selected based on author's suggestions and library default values. For the stopping criterion all GSGP variants were executed for a maximum of 300 generations. On the other hand, the stopping criterion for GP-LS is 100 generations. Notice also that the function sets in both cases are different, since we follow the best reported results for each algorithm. For HYBRID and HYBRID-LIN l = 10 as in (Castelli et al., 2015e,c). Finally, in GSGP variants, when the standard GSM mutation is used (instead of GSM-LS), the mutation step is randomly set at each mutation event in the range [0, 1].

It is important to mention that in all cases fitness was determined using the MAE between the real output and the estimated output of each problem when evaluated over the training set. Moreover, in all subse-quent comparisons, performance will be measured using the MAE, computed on the training or testing data. Moreover, Table B.4 summarizes these results numerically, reporting the median train and test performance of each algorithm on each problem, showing in bold the best (lowest) results. What immediately stands out from these results is that GP-LS outperforms all other algorithms in all cases except one, testing performance on the Flow problem, however there are runs that still are better than the other algorithms. Also notice that while performance on the Flow problem is very similar for all algorithms, on the Temperature problem performance varies quite substantially, with GSGP performing the worst and GP-LS performing the best. Another relevant observation is related to the variance in performance of each method over all runs. First, on the Flow problem, at one extreme HYBRID-LIN exhibits very stable behavior in all runs, suggesting that the linear scaling approach increases the robustness of GP search. The other extreme is GP-LS, which shows the largest variance on this problem, with several runs achieving substantially smaller error values than the other methods. Second, on the Temperature problem, all methods exhibit comparatively less variance, even GP-LS behaves quite consistently over all runs.

B.4.2 Results and Analysis

To test the significance of these results, we use the Friedman test to perform all pairwise comparisons of the methods, focusing specifically on test performance for both problems. The null hypothesis in each pairwise comparison is that the groups have equal medians, and the p-values are corrected using the Benjamini-Hochberg algorithm. The resulting p-values are given in Tables B.5 and B.6, the former for the test performance on the Flow problem and the latter for the test performance on the Temperature problem. In both tables an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.05 significance level. This statistical comparison suggest the following. On the Flow problem, basically all methods exhibit statistically similar results, since the null hypothesis was not rejected in all pairwise comparisons except two. In particular, the test suggests that a statistically significant difference exists between HYBRID and GSGP, and HYBRID and GSGP-LS. While the boxplots of all three methods appear quite similar (see Figure B.5), under further inspection the distribution of the results of GSGP and GSGP-LS are actually bi-modal, which explains the rejection of the null hypothesis. On the other hand, on the Temperature problem it is clear that GP-LS exhibits performance that is significantly different with respect to all other methods, confirming its superiority on this problem. Taken as a whole, we can say that GP-LS is the best method, since its median performance is not statistically different with respect to other methods on the Flow problem, while achieving the best performance on the Temperature problem.

Finally, it is important to comment on the size of the evolved models, which for GP-based systems is measured by the number of nodes of the best solution found. In particular, GSGP is known to produce extremely large models, the size of the models grows exponentially with the number of generation [START_REF] Moraglio | Geometric semantic genetic programming[END_REF]. This means that GSGP solutions are basically black boxes. On the other hand, one of the goals of GP is to produce models that can be interpreted or easily implemented in real-world systems. In this case, GP-LS produces vastly more compact solutions. The median size of the evolved models are 33 and 36 nodes for the Flow and Temperature problems, respectively. The actual best symbolic model obtained by the GP-LS technique for the Flow problem is given by f F =((θ 1 * (((θ 2 (X 1 )) + (((θ 3 ((log((θ 4 ((log((θ 5 ((sin((θ 6 ((sin(((θ

(X 2 ))))))))))/(θ 11 (X 1 )))))))))))))))))))) *
((θ 12 ((log((θ 13 (X 2 )))))) + (((θ 14 ((log((θ 15 ((log((θ 16 (X 2 ))))))))))+ ((θ 17 ((log((θ 18 ((log((θ 19 ((log((θ 20 ((log((θ ((log((θ 23 ((log((θ 24 ((log((θ

21 (X 2 ))))))))))))))))))+ (θ 22
25 (X 2 )))))))))))))))) + (θ 26 (X 2 )))))+
(θ 27 ((exp((θ 28 ((log(((θ 29

(X 2 )) + ((θ 30 (X 1 ))/(θ 31 (X 2 ))))))))))))))- (θ 32 (X 1 )))) + θ 33 ) ,
(B.5) where f F is the fuel flow function, θ 1 = 0.1377, θ 2 = -2.7002, θ 3 = 0.5654, θ 4 = -17.8777, θ 5 = 1.0151, θ 6 = -2.0230, θ 7 = -0.0188, θ 8 = 2.1919, θ 9 = 4.8047, θ 10 = 0.0550, θ 11 = -8.2834, θ 12 = -0.1644, θ 13 = -0.0255, θ 14 = -7.4487, θ 15 = 4.7472, θ 16 = 0.0472, θ 17 = -0.1766, θ 18 = 4.8017, θ 19 = 0.5157, θ 20 = 21.6877, θ 21 = 0.0480, θ 22 = 14.5261, θ 23 = 1.4830, θ 24 = 3.1747, θ 25 = 0.0689, θ 26 = 0.0085, θ 27 = 2.1211, θ 28 = 758.9091, θ 29 = -0.0101, θ 30 = 1.6733, θ 31 = 0.3230, θ 32 = -9.9588, θ 33 = -0.0007. X 1 represents the reference fuel flow and X 2 the turbine speed.

On the other hand, the model corresponding to the Temperature variant is given by f T =((θ 1 * (θ 2 ((log((((((θ 3 ((tan((((θ 4 ((log((θ (exp((θ 11 ((sin((θ 12 ((sin((θ

5 (X 2 )))))) * (θ 6 ((exp((θ 7 (X 2 ))))))) * (θ 8 ((exp((θ 9 (X 1 )))))))))))+ (θ 10 (
13 (X 1 )))))))))))))))/
((θ 14 ((tan((((θ 15 ((log((θ 16

(X 2 )))))) * (θ 17 ((exp((θ 18 (X 2 ))))))) * (θ 19 ((exp((θ 20 (X 1 )))))))))))/(θ 21 ((tan((((θ 22 (X 2 )) * (θ 23 ((exp((θ 24 (X 2 ))))))) * (θ 25 ((exp((θ 26 (X 1 ))))))))))))) * (θ 27 (X 2 )
))/((θ 28 ((exp((θ 29 ((sin((θ 30 (X 1 ))))))))))/ (θ 31 ((tan((((θ 32 ((log((θ 33

(X 2 )))))) * (θ 34 ((exp((θ 35 (X 2 ))))))) * (θ 36 ((exp((θ 37 (X 1 )))))))))))))))))) + θ 38 ) , (
θ 35 = -7
.5917, θ 36 = 310.6381, θ 37 = 3.5085, θ 38 = -2.6837. X1 represents the compressor discharge and X 2 the fuel flow.

B.4.3 Comparison with other Machine Learning methods

In order to get a better contextualization of the previous results, we compare the best GP method (GP-LS) with other common techniques for data-driven modeling taken from machine learning literature. In particular the following methods from the Weka open source library [START_REF] Hall | The weka data mining software: An update[END_REF] have been considered: (1) Linear regression (LR); (2) Isotonic Regression (IR); (3) Square Regression (SR); (4) Multilayer Perceptron (MLP); and (5) Support Vector Machines (SVM) with polynomial kernel. In all cases, we use the default configurations for each method except for MLP, in which we performed a parameter sweep of the decay learning rate and the optimal number of hidden nodes. For both problems the learning rate was left in the default one, while the number of neurons in the hidden layer was set to 28 for the Flow problem and to 8 for the Temperature problem performing a greedy hill climber search. Moreover, we also performed a comparison with the Fast Function Extraction algorithm (FFX) [START_REF] Mcconaghy | FFX: Fast, Scalable, Deterministic Symbolic Regression Technology, chapter Genetic Programming Theory and Practice IX[END_REF] and Multivariate Adaptive Regression Splines (MARS) [START_REF] Friedman | Multivariate adaptive regression splines[END_REF]. Generally speaking, FFX and MARS build a linear combination of basis functions to construct the model that best describes the training data of a supervised learning problem. On the one hand, FFX was developed as an alternative to the evolutionary search of standard GP but with the same general goals, while MARS is a well established method that continues to be widely used in difficult real-world domains [START_REF] Zhang | Multivariate adaptive regression splines and neural network models for prediction of pile drivability[END_REF][START_REF] Pramila | Comparison of multivariate adaptive regression splines and random forest regression in predicting forced expiratory volume in one second[END_REF][START_REF] Vijayaraghavan | Multivariate adaptive regression splines based prediction of peak expiratory flow with spirometric data[END_REF]. Both of these techniques are similar to GP in the sense that their goal is to produce a new symbolic expression that best fits the data, so we will also compare these methods with GP-LS based on the size of the models each method produces. For both MARS2 and FFX3 , we used the freely available toolboxes provided by the authors using their default configuration. Figure B.7 shows a boxplot comparison between all methods based on the test performance on both problems. Table B.7 compares the test performance of all of the above methods with GP-LS on both problems. Moreover, we apply the same statistical pairwise test as before ans report the results in Table B.8 for the Flow problem and in Table B.9 for the Temperature problem. As before, there is little difference between the methods on the Flow problem, with the statistical tests suggesting that the best performance is achieved by SR, SVM, FFX and GP-LS. On the other hand, on the Temperature problem two methods stand out: MARS and GP-LS achieve the lowest median performance and the statis- Finally, it is useful to compare the size of the models produced by FFX, MARS and GP-LS, since these methods are aimed at producing new symbolic expressions that describe the problem data. In this task, it is always preferable to generate the smallest or most parsimonious solutions, for ease of interpretation. Figure B.8 presents boxplots that compare the size of the solutions produced by each method over all runs. For a fair comparison, we express the models produced by MARS and FFX as syntax trees, and define the size of the models as the total number of nodes. On the Flow, we can see that the smallest solutions are produced by GP-LS and MARS, but it is important to remember that GP-LS outperforms MARS in terms of test error. Moreover, while FFX achieved similar performance to GP-LS in terms of test error, the models it generates are one order of magnitude larger. Similarly, on the Temperature problem the smallest solutions are generated by GP-LS. This is particularly important since it outperforms all other methods in terms of test MAE, and while MARS achieves similar performance it is evident that it does so at the cost of generating less parsimonious solutions.

B.5 C O N C L U S I O N S A N D F U T U R E W O R K
This work addresses the problem of modeling key aspects of a GT power system, namely the fuel flow of the system and the temperature of the exhaust gas. Both variables are critical to the proper functioning and performance of the GT power generator. In particular, this paper describes a data-driven approach to generate the models using GP, a paradigm of evolutionary computation for automatic program induction. Recent state-of-the-art variants of GP are tested. These variants integrate a powerful local search mechanism (GP-LS) or are based on geometric semantic concepts (GSGP). To pose the supervised learning problem, real data taken from a power plant in Mexico is used, resulting in a real-world difficult task. Comprehensive experimental valida- tion clearly shows that GP-LS outperforms all other GP variants on both problems, with clear statistical significance in the detected differences among the methods.

While recent literature has emphasized the power of geometric semantic versions of GP, these results show that standard GP algorithms can still be used to address difficult real-world problems with strong results, while also evolving much smaller solutions. This is achievable by integrating local search strategies into the search process, allowing standard GP operators to focus on evolving the structure of the solution and allowing the fine-tuning of these models to be performed by standard numerical methods. These results are consistent with what we showed in previous works using community benchmarks.

Future work will be devoted towards integrating the evolved models into an online monitoring system, to be used by engineers and researchers working at the power plant. The goal is to develop an automatic fault detection system, that can allow for easy monitoring and control to avoid possible damage or losses in power service delivered by the plant. A n amplifier circuit is designed to increase the power levels of an in- put signal, and is the most important component of a Radio Frequency (RF) link. There are two types of amplifiers in the RF domain that are: the power amplifier (PA) and low noise amplifier (LNA). The PA is mainly present in the transmission stage and is designed to increase the power level of the signal before to be transmitted by the antenna. This increased capacity is crucial to achieve the desired signal-to-noise ratio in the receiver side. Unfortunately, the PA is intrinsically a nonlinear system that generates four typical unintended consequences: spectral regrowth, memory effects, inter-modulation products and adjacent channel interference. This signal degradation becomes severe when multilevel modulation schemes signal are used [START_REF] Biglieri | Analysis and compensation of nonlinearities in digital transmission systems[END_REF].

Due to rapid growth and high demand for data transmission, the RF spectrum availability is becoming increasingly scarce. Additionally, there is a concern related to the requirements obliged to use higher frequencies for signals with digital modulation systems such as wireless broadband communications, for example, Wide Code Division Multiple Access (WCDMA), Worldwide Interoperability for Access Microwave (WiMAX), and others. In such systems, the transmitter chain is included within a PA as the main element in the amplification process, however, its inherent nonlinearity causes distortion to the operating band intermodulation products and expands the spectrum allocated, infringing adjacent channels [START_REF] Kenington | High Linearity RF Amplifier Design[END_REF].

The development of pre-distorters is now facilitated by the availability of a variety of behavioral models of PAs, mainly based on the Volterra series, which considers undesirable effects such as memory and nonlinearity in order to describe the nonlinearity of a PA. However, due to the high computational complexity of Volterra series it is impractical in some real applications due to the number of parameters to be estimated [START_REF] Xiaofang | On the numerical stability of RF power amplifier's digital predistortion[END_REF][START_REF] Schetzen | The Volterra and Wiener Theories of Nonlinear Systems[END_REF]. The memory polynomial model (MPM) is used to overcome these limitations by reducing the processing time for computational modeling compared to a full Volterra series [START_REF] Bennadji | Implémentation de modèles comportementaux d' amplificateurs de puissance dans des environnements de simulation système et co-simulation circuit-système[END_REF][START_REF] Pedro | A comparative overview of microwave and wireless power-amplifier behavioral modeling approaches[END_REF][START_REF] Asbeck | Augmented behavioral characterization for modeling the nonlinear response of power amplifiers[END_REF]. Furthermore, it has been shown that the MPM can reduce internal iterations during the shaping step, without losing the accuracy of the model identification [START_REF] Ku | Behavioral modeling of nonlinear RF power amplifiers considering memory effects[END_REF].

The basic RF-PA modeling techniques involves special truncations of the Volterra Series [START_REF] Li | Wiener-saleh modeling of nonlinear RF power amplifiers considering memory effects[END_REF][START_REF] Moon | 2-d enhanced hammerstein behavior model for concurrent dual-band power amplifiers[END_REF][START_REF] Misic | Volterra kernels extraction from neural networks for amplifier behavioral modeling[END_REF], but for very complex nonlinear models with wide data range is required the use of the whole Volterra Series [START_REF] Staudinger | Memory fading volterra series model for high power infrastructure amplifiers[END_REF][START_REF] Zhu | Behavioral modeling of RF power amplifiers based on pruned volterra series[END_REF], Fuzzy Neural Network [START_REF] Zhai | Behavioral Modeling of Power Amplifiers with Dynamic Fuzzy Networks[END_REF][START_REF] Mkadem | Behavioral modeling and digital predistortion of Power Amplifiers with memory using Two Hidden Layers Artificial Neural Networks[END_REF] or GP [START_REF] Sheng | An ANN model of optimizing activation functions based on constructive algorithm and GP[END_REF][START_REF] Patelli | A regressive schema theory based tool for gp evolved nonlinear models[END_REF]. By other hand, GP performs an evolutionary search within the space of possible program syntaxes, achieving the expression that best solves a given model. GP can be viewed as a biologic evolutionary inspired algorithm where a pool of symbolic expressions are built in a synergy fashion upon a target. Each expression competes for survival at each iteration by measuring its fitness value. This is usually expressed by an error metric toward the objective. In general, each symbolic expression consist of a mathematical equation that represent a potential candidate model in the imposed problem. Standard GP can solve complex problems by searching in the syntax space, however accuracy on the solutions can be stagnated through the evolution and expressions might grow in size.

C.2 AM/AM and AM/PM conversions In recent years, a few outsourcing protocols have been proposed for hardware implementation FPGA based [START_REF] Alasady | Design and hardware implementation of look-up table predistortion on ALTERA Stratix DSP board[END_REF][START_REF] Xiong | Implementation of the Quadrature Waveform Generator Based on DSP Builder[END_REF]. This work implements in hardware behavioral models based on genetic programming (GP) and MPM. The results are then compared based on normalized mean squared error (NMSE), mean squared error (MSE), mean absolute error (MAE) and Correlation coefficient. The experimental setup is based on the RF-PA Doherty 7W @ 2.11 GHz. It is important to note, that to the authors knowledge this is the first work to use GP to derive models for such a PA. Moreover, we propose the use of a recent variant of GP that incorporates numerical methods as local search methods, which greatly enhance the performance of GP [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF].

The remainder of this work proceeds as follows. Section C.2 describes the AM/AM and AM/PM conversions, describing how data was collected to apply the data-driven approach. Section C.3 describes the Volterra Series and MPM as special case of the Volterra Series. Section C.4 provides a general overview of GP, focusing on the specialized variants employed in this work. The experimental work and results are presented in Section C.5. Finally, conclusions and future work are outlined in Section C.6.

C.2 A M / A M A N D A M / P M C O N V E R S I O N S

The AM/AM distortion curve of a nonlinear system is the relationship between the output power magnitude of the system and the input magnitude. The AM/PM distortion curve for the linear system is the ratio of the phase shift between the output and the input power magnitude. The figure C.1 shows a general block diagram of a PA based on distortion curves.

A pass-band input signal, can be represented by equation C.1.

x(n) = r(n)cos(ω c n + φ(n)), (C.1)
where ω c is the center carrier frequency, r(n) is the amplitude of the signal, φ(n) is the modulated phase. Assuming the previous pass-band

Y P B = g(r(n))cos(ω 0 n + φ(n) + f (r(n)), (C.2)
where g(r(n)) is the amplitude nonlinearity or AM/AM conversions, f (r(n)) is the phase nonlinearity or AM-PM conversion.

C.3 V O LT E R R A S E R I E S

Volterra series are a combination of linear convolution and a series of nonlinear powers which can be used to describe the relationship between input/output of a nonlinear causal time invariant system with memory order.

The main disadvantage of Volterra series is the increase in the number of parameters needed regarding the nonlinearity order and memory depth; as a consequence it has a drastic increase of the complexity in the step of identifying parameters. As the bandwidth of the input signal increases, the time span memory PA becomes comparable to variations in C.3 Volterra Series time of the envelope of the input signal. The general Volterra series are expressed as following:

y(n) = M ∑ m 1 =0 h 1 (m 1 )x(n-m 1 )+ M ∑ m 1 =0 M ∑ m 2 =0 h 2 (m 1 , m 2 )x(n-m 1 )x(n-m 2 )+. . . + M ∑ m 1 =0 M ∑ m 2 =0 M ∑ m 3 =0 h 3 (m 1 , m 2 , m 3 )x(n -m 1 )x(n -m 2 )x(n -m 3 ) + . . . , (C.3)
where x(n) represents the complex signal input baseband, and y(n) is the complex baseband signal output, h n m n the coefficients of the Volterra series, n is the order of the nonlinearity and m is the depth of system memory. As we can see, the number of coefficients of the Volterra series increases exponentially with the size of the memory system and the order of the nonlinearity [START_REF] Lei | Digital predistortion of power amplifiers for wireless applications[END_REF].

C.3.1 Memory Polynomial Model

A review of the current research related to the RF-PA modeling considering memory effects, mainly based MPM as a special case of the Volterra series, give us that it is widely understood that MPM is a popular way to make the modeling work reducing coefficients required for this process, mainly obtained by the method of least square error [START_REF] Rawat | Three-Layered Biased Memory Polynomial for Dynamic Modeling and Predistortion of Transmitters With Memory[END_REF][START_REF] Liu | A robust augmented complexity-reduced generalized memory polynomial for wideband RF power amplifiers[END_REF].

The MPM is a subset of the Volterra series, it consists of several stages of delay and nonlinear static functions, representing a truncation of the general Volterra series considering only the diagonal terms of the kernels [START_REF] Ku | Behavioral modeling of nonlinear RF power amplifiers considering memory effects[END_REF]. Therefore, the number of parameters is significantly reduced compared to the original series. MPM considers is expressed as:

y(n) = Q ∑ q=0 K ∑ k=1 a 2k-1,q |x(n -q)| 2(k-1) x(n -q), (C.4)
where x(n) is the input signal, y(n) is the output signal, a k,q are the Volterra Kernels, Q is the memory depth and K the nonlinearity order.

The output system based on memory steps can be as equation C.5:

y(n) = Q ∑ q=0 F q (n -q) = F 0 (n) + F 1 (n -1) + • • • + F q (n -q) + • • • F Q (n -Q),
(C.5) where F q (n) is given by

F q = Q ∑ q=0 a 2k-1,q |x(n -q)| 2(k-1) x(n), (C.6)
and F q (n) can be expressed as extended form as Being previously stated that the GP goal is to search for syntactic expressions that perform some form of computation, attempting to find the relation between a set of independent variables (inputs) and dependent variables (outputs), but recognized that the canonical version of GP suffers from some shortcomings. That is the search operators perform at syntax level, blind to the effects of the solutions output. For that, the incorporation of LS techniques is beneficial to the search by accelerating the overall quality convergence producing smaller and better solutions. Moreover, in this work we follow the same strategies presented in Builder stage, these icons represent advanced and standard blockset that allows for high-performance HDL generation algorithms directly from the Simulink environment.

F q (n) = a 1,q x(n)+a 3,q |x(n)| 2 x(n)+a 5,q |x(n)| 4 x(n)+• • •+a 2k-1,q |x(n)| 2(k-1) x(n).

C.5.1.2 GP-LS Experimental Setup

Since GP is intrinsically a stochastic search and optimization algorithm, this means that different GP runs will lead to different models (or solutions) produced as output. Therefore, we performed 10 independent runs of GP-LS using the parameters summarized in table C.2. From these runs we choose the best model found for each problem. In particular, we chose the models that achieved the lowest training error. In both cases the evolved models are relatively small, with a total of 17 nodes each. This means that each model is accompanied by 17 real-valued parameters. Informally, we can say that the models are parsimonious and relatively simple, particularly in terms of size. Of course, their efficiency will be evaluated experimentally below, from the perspective of a hardware implementation of each.

C.5.2 Results and Analysis

Table C.3 summarizes the results numerically for MPM, showing a very high error using the mean squared error (MSE), mean absolute error (MAE) and root mean squared error (RMSE) as metrics, each computed by comparing the known behavior with the predicted behavior of each model. Results are shown for the raw model outputs, and its combination with white noise. What immediately stands out from these results is that MPM does not have the required precision for a complex model such as the one studied here, considering that our data is composed by 122,880 samples. These experimental results shows that the best solutions generated by GP-LS clearly outperforms the standard MPM approach to model the behavior of a complex PA. In terms of accuracy, it is clear that GP-LS is a better option. We now compare the hardware resources required by each mode, this is summarized in Table C.5, considering that all models were implemented using the DSP Development Board Cyclone III-ALTERA. Again, the GP-LS models clearly require less resources on the FPGA board that the standard MPM approach. The AM/AM and AM/PM must be implemented in hardware together, both models represent the RF-PA behavior in amplitude and phase respectively

C.6 C O N C L U S I O N S
This work tackles the problem of predicting the behavior of a real RF-PA, particularly a RF-PA Doherty 7W@2.1 GHz. To do so, we used a traditional MPM approach for both the AM/AM and AM/PM conversion curves, as special case of the well-known Volterra series. Moreover, we used a recently developed variant of GP search for both modeling problems, GP-LS that integrates a numerical local search method with the standard syntax-based search. The experimental results GP-LS produces much more reliable solutions than using MPM. For instance, for the AM/AM problem a MPM with NL=15 resulted in MSE=362.69, MAE= 25.25 and RMSE= 2494.1, while for GP-LS they were as low as MSE=78.26, MAE= 7.5702 and RMSE= 5377.9. Moreover, model performance was improved by combining the raw output of the evolved models by GP-LS with a white noise component, which provides a better approximation of the true PA behavior. Moreover, the MPM and GP-LS were also compared based on their efficiency, given by the amount of resources they require when they are implemented on an FPGA. Again, the GP-LS models require less resources on the boards than standard MPM. In summary, GP-LS are not only better at approximating PA behavior they do so in a more efficient way. These results are encouraging for real-world testing and deployment, which is left as future work. C urrently, the exponential growth of global population and the manner in which industrial societies are developing, have given rise to serious environmental problems. For instance, the intensive development of the chemical industry, that has been required to meet the growing demand, has produced large scale pollution from coal processing (Song et al., 2015) and carbon emissions [START_REF] Song | Analysis of carbon emissions and their influence factors based on data from anhui of china[END_REF]; global issues with significant economic, social and environmental implications [START_REF] Lee | Sustaining the low-carbon emission development in asia and beyond: Sustainable energy, water, transportation and low-carbon emission technology[END_REF][START_REF] Jaromir | Process integration: pinch analysis and mathematical programming-directions for future development[END_REF]. For these reasons, multidisciplinary research and collaboration between stakeholders should be enhanced to stimulate the transition toward a more sustainable society [START_REF] Niesten | Sustainable collaboration: The impact of governance and institutions on sustainable performance[END_REF].

Regarding our water supply, the excessive applications of phenolic compounds leads to serious repercussions. Water is widely used as a raw material in the process industries for a wide range of products. In particular, phenol and phenolic compounds are common contaminants found in effluents from the production of plastics, leather, paint, textile, paper, printing and other petrochemical processes [START_REF] Altenor | Adsorption studies of methylene blue and phenol onto vetiver roots activated carbon prepared by chemical activation[END_REF]. The modeling wastewaters from these industries are generating considerable amounts of phenolic pollutants [START_REF] Kumar | Removal of methylene blue and phenol onto prepared activated carbon from fox nutshell by chemical activation in batch and fixed-bed column[END_REF], leading to increasingly stringent environmental regulations on the discharge of effluents [START_REF] Klemeš | Water Integration for Recycling and Recovery in Process Industry[END_REF]. The main reason for this is that these contaminants are extremely toxic, even if they are present in small concentrations their impact on health includes protein degeneration, tissue erosion, paralysis of the central nervous system, vomiting, difficulty in swallowing, anorexia, liver and kidney damage, headache, fainting and various mental disturbances [START_REF] Qadeer | A study of the adsorption of phenol by activated carbon from aqueous solutions[END_REF][START_REF] Srivastava | Adsorptive removal of phenol by bagasse fly ash and activated carbon: Equilibrium, kinetics and thermodynamics[END_REF][START_REF] Takahashi | Variation of biodegradability of nitrogenous organic compounds by ozonation[END_REF]. In fact, phenolic compounds are classified as high priority pollutants due to their carcinogenic effect on human health and their harmful effects on wild life [START_REF] Altenor | Adsorption studies of methylene blue and phenol onto vetiver roots activated carbon prepared by chemical activation[END_REF]. Therefore, it is necessary to remove these phenolic compounds from industrial effluents before discharging them into the water stream.

The US Environmental Protection Agency (EPA) has instituted a regulation for lowering phenol concentration in the wastewater to less than 1.0 mg/dm 3 [START_REF] Tian | Kinetics of the electrochemical oxidation of 2-nitrophenol and 4-nitrophenol studied by in situ uv spectroscopy and chemometrics[END_REF]. Therefore, some researchers have proposed a variety of methods for their removal from wastewater [START_REF] Ukrainczyk | Oxidation of phenol in acidic aqueous suspensions of manganese oxide[END_REF]. Among current proposals, the removal of phenols by adsorption technologies is regarded as one of the best methods because it does not require a high operating temperature and can be considered to be a relatively simple process from an implementation standpoint. Indeed, various adsorbent solids have been used to remove phenolic compounds from wastewater, including activated carbon [START_REF] Nouri | Comparison of adsorption capacity of p-cresol and p-nitrophenol by activated carbon in single and double solute[END_REF], silica [START_REF] Hanna | Sorption of hydrophobic molecules by organic/inorganic mesostructures[END_REF], polymeric resins [START_REF] Abburi | Adsorption of phenol and p-chlorophenol from their single and bisolute aqueous solutions on amberlite xad-16 resin[END_REF], fly ash [START_REF] Sarkar | Removal characteristics of some priority organic pollutants from water in a fixed bed Bibliography fly ash column[END_REF] and kaolinite [START_REF] Barhoumi | Coadsorption of alkylphenols and nonionic surfactants onto kaolinite[END_REF].

D.1.1 Proposal and Contributions

This work studies the use of activated carbon for the adsorption of phenols and nitrophenols. In particular, the goal of this work is to find models that capture the nonlinear relationships between several relevant operating conditions, such as the contaminant, initial concentra-D.1 Introduction tion, pH and contact time, to predict the adsorption efficiency in an aqueous solution. Modeling phenol and nitrophenols adsorption onto activated carbon is very complex, it requires the solution of systems of equations that involve the radiant energy balance, the spatial distribution of the adsorbed radiation, mass transfer and the mechanisms of the adsorption transport problem [START_REF] Altenor | Adsorption studies of methylene blue and phenol onto vetiver roots activated carbon prepared by chemical activation[END_REF][START_REF] Kumar | Removal of methylene blue and phenol onto prepared activated carbon from fox nutshell by chemical activation in batch and fixed-bed column[END_REF]. The studied adsorption process involves forces near the surface of the molecules, as well as the attractive and repulsive mobility forces between molecules of various types and shapes, which include the effects of molecular dissymmetry on properties of matter, such as evaporation, condensation and reflection. The process depends on several factors and phenomena, exhibiting a nonlinear behavior which is difficult to describe by linear mathematical models, such as those derived by linear regression methods. In similar problems, these considerations have lead researchers to turn towards non-linear data driven modeling methods [START_REF] Nia | Modeling of reactive orange 12 (ro 12) adsorption onto gold nanoparticle-activated carbon using artificial neural network optimization based on an imperialist competitive algorithm[END_REF][START_REF] Karimi | Application of artificial neural network and genetic algorithm to modeling and optimization of removal of methylene blue using activated carbon[END_REF][START_REF] Dil | Application of artificial neural network and response surface methodology for the removal of crystal violet by zinc oxide nanorods loaded on activate carbon: kinetics and equilibrium study[END_REF]Ghaedi et al., 2015b,a).

Appropriate process models could be used to take corrective actions when the adsorption efficiency is predicted to be outside the required levels. The ability to take such actions could help reduce the environmental impact of the effluents when they are discharged into the water stream.

Therefore, the first main contribution of this work is the proposal of a relatively simple data-driven approach to derive models of the adsorption efficiency, based on a real-world experimental study. The derived models can help researchers and practitioners to predict the effect that variations in the operating conditions might have on the adsorption process, without recurring to costly and time-consuming experimental tests. The second contribution of this work relates to the specific algorithmic approach taken in this work, which is based on the use of genetic programming (GP) [START_REF] Koza | Genetic Programming: On the Programming of Computers by Means of Natural Selection[END_REF]. Basically, we pose a supervised learning problem to automatically derive descriptive models of the adsorption efficiency and solve the problem by means of GP.

In this work, we compare two state-of-the-art GP methods recently proposed by the authors: (1) GP with local search optimization (GP-LS); and (2) neat-GP, a variant that searches for parsimonious solutions. The GP algorithms are also compared with standard regression techniques, including multivariante linear, quadratic and robust regression. Moreover, GP is also compared with more powerful regression techniques, namely Multivariate Adaptive Regression Splines [START_REF] Friedman | Multivariate adaptive regression splines[END_REF] and the Fast Function Extraction algorithm [START_REF] Mcconaghy | FFX: Fast, Scalable, Deterministic Symbolic Regression Technology, chapter Genetic Programming Theory and Practice IX[END_REF]. We present a comprehensive numerical and statistical comparison of the results, showing that GP can be used to construct useful predictive models of the phenol adsorption efficiency by activated carbon, outperforming all other methods.

The remainder of this work is organized as follows. Section D.2 provides a detailed description of the process of adsorption of phenol and nitrophenols by activated carbon. Afterward, section D.3 and section D.4 describe our GP-based modeling algorithms, GP-LS and neat-GP. Section D.5 presents our experimental work, describing how we derive a representative dataset of the adsorption process, the experimental setup and a discussion of the main results. Finally, Section D.6 presents our conclusions and outlines future work.

D.2 P H E N O L A N D N I T R O P H E N O L A D S O R P T I O N B Y A C T I VAT E D C A R B O N

The process of adsorption refers to the adhesion of molecules from a mixture in a gaseous or liquid state onto a solid surface. This process creates a film of the adsorbate on the surface of the adsorbent. The present work focuses on activated carbon as an absorbent, which has been widely used in industry for many years, due to its uniform porous structure and appropriate selective adsorptive [START_REF] Ruthven | Principles of adsorption and adsorption processes[END_REF][START_REF] Speight | The chemistry and technology of petroleum[END_REF].

The surface properties of activated carbon used in this work are summarized in Table D.1, which were calculated using a t-plot analysis [START_REF] Do | Adsorption analysis: equilibria and kinetics[END_REF][START_REF] Lippens | Studies on pore systems in catalysts: V. the t method[END_REF][START_REF] Nakai | The analysis of surface and pores of activated carbons by the adsorption of various kinds of gases[END_REF]. An average diameter distribution of 0.3 nm is obtained by the method described by Horváth and Kawazoe [START_REF] Horváth | Method for calculation of effective pore size distribution in molecular sieve carbon[END_REF]. The material is characterized by scanning electronic microscopy (SEM) using a Hitachi [START_REF] Singha | The use of artificial neural networks (ann) for modeling of adsorption of cr(vi) ions[END_REF], to study the effect that important operating parameters, such as pH and the initial concentration of the contaminant, have on the adsorption efficiency W % over time. The adsorption efficiency is given by

W % = C i -C f C i × 100 (D.1)
where C i is the initial concentration of the contaminant at the absorbent, and C f is the final concentration of the contaminant in the absorbent at a particular time instant.

D.3 G E N E T I C P R O G R A M M I N G W I T H L O C A L S E A R C H
The most common application of GP is to solve what are known as symbolic regression problems, where a model that best fits a dataset is sought. In particular we adopt the approach proposed in [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF] which has been applied to both symbolic regression and classification problems. Despite its well documented success [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF], the basic GP approach suffers from a severe drawback that is commonly referred to as bloat [START_REF] Silva | Dynamic limits for bloat control in genetic programming and a review of past and current bloat theories[END_REF][START_REF] Silva | Reassembling operator equalisation: a secret revealed[END_REF]. When GP is used to evolve solutions of unconstrained size, it will usually produce excessively large solutions.

For this reason, many researchers have proposed bloat-free GP variants, ranging from the very simple to more elaborate methods. In particular, in this work we study the recently proposed neat-GP algorithm [START_REF] Trujillo | neat genetic programming: Controlling bloat naturally[END_REF], which is based on the NeuroEvolution of Augmenting Topologies (NEAT) algorithm [START_REF] Stanley | Evolving neural networks through augmenting topologies[END_REF], originally developed for the evolution of neural networks. Recently published results show that neat-GP outperforms other bloat control methods, and achieves equal or better performance that the standard GP search [START_REF] Trujillo | neat genetic programming: Controlling bloat naturally[END_REF]. However, while previous results showed that neat-GP can be competitive with the state-of-the-art, this work is the first to apply the method to a real-world data modeling problem.

D.5 Experiments

D.5 E X P E R I M E N T S

As stated before, the goal of this work is to derive a a model that describes the relationship between operating variables and adsorption efficiency of phenol and nitrophenols onto activated carbon. In particular, we use a data-driven approach, where real experimental data is generated, recorded and then used to pose a supervised-learning problem. Afterward, GP (GP-LS and neat-GP) is used to solve a supervised learning problem. Performance is compared with several baseline and state-of-the art methods to evaluate their predictive accuracy and model complexity.

D.5.1 Data Generation

To generate our ground truth data, experiments are performed with the adsorption process in a liquid phase. A constant amount of activated carbon (0.1 g) is mixed with a 10 mL solution of phenol (or 2nitrophenol, 3-nitrophenol, 4-nitrophenol) in a 15 mL centrifuge tube at a pH range between 2-10. The suspensions are shaken at room temperature from 5 minutes to 6 hours, using an orbital shaker with a constant agitation rate. The phases are separated by centrifugation for 2 min at 1500 rpm. The concentrations of phenol, 2-nitrophenol, 3-nitrophenol and 4-nitrophenol in the supernatant solution after adsorption are determined with the ultraviolet-visible (UV-Vis) spectrophotometer technique. The maximum absorbances wavelength used are 269 nm, 278, 273 and 316 nm, respectively, for phenol, 2-nitrophenol, 3-nitrophenol and 4-nitrophenol. The experimental process is summarized in Figure

D.2.
The experimental database is provided by Abatal et al. as published in [START_REF] Abatal | Comparative adsorption behavior between phenol and p-nitrophenol by na-and hdtma-clinoptiloliterich tuff[END_REF]. It contains 975 samples of input/output pairs. The inputs are: (1) the contaminant present in the suspension (phenol, 2-nitrophenol, 3-nitrophenol, 4-nitrophenol); (2) the initial contaminant concentration in the adsorption process (20, 40, 60, time (0-360 mins.). The output to be approximated is the adsorption efficiency W %. These input and output variables are summarized in Table D.2. From the collected data, we can observe that the removal of pollutants from wastewater is affected by the pH of the phase where the removal occurs; as shown in Figure D.3(a), which plots the effect of pH on the adsorption of each contaminant. The plot shows that the adsorption of phenolic compounds by activated carbon is reduced with increasing pH values. The pH of the solution has an impact on the surface charge and the degree of ionization of the absorbent, accelerating the internal surface area and increasing the creation of the porous material of activated carbon as an adsorbent solid. This might lead to change the We have observed that when the experiments are carried out for one to three hours with pH values lower than 7, the adsorption efficiency of phenolic compounds reaches the critical values (maximum). Therefore, we can confirm that the interference of activated carbon on phenolic compounds is affected by the operating conditions considered in this work, namely the pH, contact time, initial contaminant concentration and contaminant.

D.5.2 Algorithms and Experimental Setup

The regression problem posed in this work is solved with several methods. First, we use basic methods common in such works to provide a baseline for our experimental study; these are: multivariate linear regression (Linear), multivariate quadratic regression (Quadratic) and iteratively reweighted least squares linear regression (Robust). Second, we use two powerful regression methods, the Fast Function Extraction algorithm (FFX) [START_REF] Mcconaghy | FFX: Fast, Scalable, Deterministic Symbolic Regression Technology, chapter Genetic Programming Theory and Practice IX[END_REF] and Multivariate Adaptive Regression Splines (MARS) [START_REF] Friedman | Multivariate adaptive regression splines[END_REF][START_REF] Zhang | Multivariate adaptive regression splines and neural network models for prediction of pile drivability[END_REF][START_REF] Pramila | Comparison of multivariate adaptive regression splines and random forest regression in predicting forced expiratory volume in one second[END_REF][START_REF] Vijayaraghavan | Multivariate adaptive regression splines based prediction of peak expiratory flow with spirometric data[END_REF]. Generally speaking, FFX and MARS build a linear combination of basis functions to construct the model that best describes the training data. Third, we use the two GP variants presented in Section B.3, neat-GP and GP-LS.

In summary, we use seven algorithms to solve the modeling problem: Linear, Quadratic, Robust, FFX, MARS, neat-GP and GP-LS. For each algorithm we performed 30 independent runs. In each run the problem data was split into a training and testing set, where the former contains 75% of the data samples and the latter 25%. The datasets were randomly partitioned before each run, and while training performance will be reported the critical result will be the prediction accuracy of the evolved models over the test data. In particular, the algorithms are compared using the Mean Squared Error (MSE) and the coefficient of determination (R 2 ), computed over the ground truth output and the estimated output of each algorithm, for the training and testing data. Table D.3 summarizes all other algorithm parameters for neat-GP and GP-LS. Moreover, we will also report an analysis of the size of the solutions generated by the algorithms that achieved the best performance, as an approximation of model complexity.

For the Linear, Quadratic and Robust regression methods we used the implementation offered in Matlab 2014a with the default configuration. In the case of MARS 1 and FFX 2 , we used the freely available toolboxes provided by the authors with their default configuration. GP-LS is implemented using Matlab and the GPLab toolbox [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF] and neat-GP is implemented using the DEAP framework [START_REF] De Rainville | Deap: A python framework for evolutionary algorithms[END_REF], which is freely available on our website 3 . D.4 summarizes these results numerically, reporting the median, first and third quartiles of the train and test MSE. Using only visual inspection of these results, the worst performance is achieved by Linear and Robust regression methods, with the best performance clearly achieved by MARS and GP-LS. Note the poor performance by neat-GP, which is outperformed by Quadratic regression method and also shows the highest variance in performance, suggesting that the method is highly sensitive to the training data on this problem. FFX outperforms all methods, except for MARS and GP-LS. It is also important to note that while there are obvious differences among the compared algorithms, the performance of all methods does not vary substantially when comparing their training and testing MSE, this suggests than none of the methods suffered from overfitting. Among all methods, GP-LS achieves the minimum error on the present problem, while MARS exhibits a slightly lower median error.

D.5.3 Results and Discussion

To evaluate the significance of these results, we use the Friedman test to perform all pairwise comparisons of the methods, focusing specifically on test performance for both problems. The null hypothesis in each pairwise comparison specifies that both sets of results share a common median, and the p-values are corrected using the Benjamini-Hochberg defined by the number of input variables and their performance was not competitive with the more advanced methods. Moreover, we differentiate between the methods based on a linear combination of basis functions (FFX and MARS) and the GP-based methods (neat-GP and GP-LS). In the former group, size is given based on the total number of basis functions used, while in the latter group we use the total number of tree nodes in the best solution found. These results are summarized and presented as boxplots in Figure D.6, with the following observations. First, MARS produces smaller solutions than FFX, by a substantial margin, using a median of 15 basis functions while FFX uses 24. Second, while GP-LS is not designed to be a bloat control method, it nevertheless produces solutions that are at least as small as those generated by neat-GP, the median solution size is 41 nodes for GP-LS and 57 for neat-GP. These results are particularly noteworthy, since both MARS and GP-LS also outperform FFX and neat-GP based on MSE and R 2 . Moreover, it is important to mention that the size of the solutions generated by GP-LS should be considered as smaller than those generated by MARS. If a basis function (as used by either MARS or FFX) is expressed as a tree structure it would require at least 3 nodes, and sometimes as many as 5 or 7. Therefore, taking the conservative estimate of 3 nodes, and counting each of the sum operations in the linear models, MARS would require about 60 nodes to express the models it is generation, higher than the 41 required by GP-LS. Taking both the size and performance of the models into consideration, it is reasonable to state that GP-LS can produce the most efficient models of the adsorption phenomenon.

To visualize the quality of the GP-LS models, Figure D.7 shows scatter plots for all thirty models, plotting the ground truth output with respect to the estimated adsorption efficiency W %. .9 provide a visual illustration of the ability of the GP models to approximate the underlying contaminant adsorption under different conditions Finally, it is instructive to mention that the GP process performs an implicit feature subset selection stage during the search. As the GP algorithm generates different candidate models, the number of input variables (terminals) used by each model is decided by the search process. Therefore, some of the input variables might not be included in the final model returned by the search. Nevertheless, it is important to analyze which of the input variables were most often used by the best models found in all of the runs (30 models in total). Figure D.10 analyzes the usage frequency for each of the input variables from the best solutions found in each run. First,Figure D.10(a) shows a bar plot with values between 0 to 30, where the height of each bar represents the number of models that used a particular feature. The most used feature is time, followed by the contaminant, the pH and finally the initial concentration. Figure D.10(b) provides a finer level of detail. In this plot the y-axis represents the proportion that each feature represents of all the inputs to a model (as leaves on the tree representation), ordered on the x-axis based on their performance rank. Moreover, the right side of the plot shows the average proportion over all runs for each feature. The effective planning and management of industrial process that release contaminants into our ecosystem has become a primary concern over recent decades, since contamination and resource scarcity problems have led to a variety of impacts on our well being. However, achieving a reasonable and efficient management strategy is difficult since many conflicting factors have to be balanced, due to complexities of real world problems.

This work presents the first application of GP to model the adsorption of phenols and nitrophenols, one of the most used processes in industrial water treatment applications. The methodology used is to pose a supervised learning problem, after collecting relevant data from realworld experimental tests. The considered input variables are the contaminant, the pH levels, the initial concentration and the elapsed time. The desired output of the model is a prediction of the adsorption efficiency of the process W %.

D.6 Summary, Conclusions and Future Work

The experimental work considered two recently proposed GP variants, a bloat free algorithm (neat-GP) and a GP-based search that incorporates a numerical local optimizer (GP-LS). Moreover, the GP algorithms were compared with standard regression techniques (linear, quadratic and robust regression) and with more sophisticated regression methods, namely FFX and MARS. The experimental results show that GP-LS clearly outperforms all other methods based on accuracy (MSE and R 2 ), except for MARS that achieves equivalent performance. On the other hand, neat-GP performed rather poorly, with median performance similar to that of the simpler linear methods and a larger variance. Moreover, when comparing the derived models based on size, a good indicator of model complexity, GP-LS achieved the best results, generating smaller models than FFX, MARS and neat-GP. Therefore, considering both accuracy and size, we can confidently state that GP-LS can be used to model the adsorption efficiency quite effectively. These observations are further extended by showing breakthrough curves comparing the GP-LS models with the ground truth data.

Future work derived from this research will be centered on the development of an automated wastewater treatment system, sufficiently suitable to directly control the adsorption of phenolic compounds onto activated carbon. Or more generally, to derive data-driven models for other absorbents, to monitor and predict when there might be a hazardous effect caused by the pollutants in real time. The models derived by the GP algorithms have the potential to be implemented in smart sensors systems, to provide accurate predictions and help decision makers and highlevel managers of industrial process, that use phenolic compounds, develop cost-effective management policies to remove these harmful contaminants from the water. Therefore, future work will also focus on the managerial implications by examining and assessing the economic, environmental and ecological impacts of using the derived models as part of an online wastewater treatment system. I n the field of model generation for the characterization of a system's behavior by data inspection, the morphology of the model (syntax) is just as important as its performance, specially when the system is nonlinear and complex architectures are required. Although there is a vast quantity of different model generation algorithms [START_REF] Morlini | Advances in statistical models for data analysis[END_REF], most of them offers a pre-built model structure, where the algorithm searches for the best fit of this structure to input data, however only few focus on model-free approaches; that is, no prior knowledge on a specific model structure is imposed, rather the algorithm builds it at run-time. GP belongs to this specific group of model generation algorithms. The model generation process is reproducible by a supervised learning task, namely regression. Moreover, GP searches for a near-optimal symbolic expression that minimizes or maximizes a cost function adapted to the problem to be solved, by evolving a pool of candidate solutions using syntax search operators. It is noteworthy to mention that two tasks, the model syntax and its parameters, are solved simultaneously during the algorithm iterative process. A symbolic regression problem can be formally defined as K * ← min K∈G f (K(x j ), y) with j = 1, . . . , p, (E.1)

where K * is a suitable symbolic expression that satisfies the cost function f , G is the syntactic space defined by the primitive set P of func-tions and terminals, x j ∈ R p is the input data matrix, p is the number of variables in input data, y ∈ R is the output data vector and K is the symbolic expression output. Usually, f is defined as an error metric, like Mean Squared Error [MSE]). Despite that in recent years the use of meta-heuristic algorithms, like GP, has been successfully used in different domains, mainly because its outstanding results in rather complex problems [START_REF] Koza | Human-competitive Results Produced by Genetic Programming[END_REF](the work in [START_REF] Mittman | Computer Chess Programs[END_REF] won the 2014 Human-Competitive Award), the widespread deployment is still in its early stages compared with other more mature techniques. Although one might argue that theoretical formality is still lacking in some areas of Evolutionary Algorithms (EA), in particular for GP [START_REF] White | Better gp benchmarks: community survey results and proposals[END_REF], the fact is that apart from that, the spread of tools toward a broader audience helps to the overall GP influence. From a research point of view, these tools are essential for experimentation and comparison purposes, however realworld engineering problems should be also benefited from these aforementioned tools. Indeed, a truly efficient model extraction application might not require an expert to solve the problematic at hand. Specifically for GP, there are several software libraries and applications accessible to both practitioners and developers. Available libraries ranges from the simpler to the most complex ones. Either by just focusing on only GP or on several EAs, however only few are still actively updated. Some examples are:

f = 1 N ∑ N i (K(x i ) -y i ) 2 (
• BEAGLE1 -A Evolutionary Computation (EC) C++ framework.

Although it includes capabilities for GP, there is a minimalist stripped version specific for GP called Puppy.

• BEAGLE Puppy2 -Written in compliance of BEAGLE codebase, it is a fast tree-based GP C++ implementation, which this work is based on.

E.1 Introduction

• DEAP 3 -A Python framework that covers a wide range of EAs, including GP.

• DGPF 4 -A Java based GP framework that can easily extend to other EAs as well. It has a distributed workload design built-in.

• ECJ 5 -A vast EC Java based framework with a steeply learning curve, that is still actively developed.

• EO 6 -Another EC framework written in C++ with its main focus being the execution speed. It supports parallel computing through third party software.

• EpochX 7 -A specific GP Java based framework.

• GPLAB 8 -A MATLAB GP toolbox.

• µGP 9 -A C++ GP optimizer, tested in a wide range of distinct problems.

• PushGP 10 -An EA framework that uses a self designed programming language to evolve programs using a clever stack based methodology.

• GPOCL 11 -An OpenCL implementation of GP. It can run in multiple platforms.

• DataModeler 13 -A commercial product using GP as model generation.

• HeuristicLab 14 -An open source software that handles a wide range of EAs through a visual interface. Actively in development, is one of the few applications that provide several tools varying from data management to results reporting.

• Eureqa 15 -A commercial product focused on simple data model extraction and its deployment.

All of these tools offers different layers of functionality and might not be useful for all kind of tasks. Only few of them focuses on the user interaction, being either an expert on the field or just a practitioner. This work pretends to close more that gap by proposing an open source visual tool (LeafGP) for model extraction that it is intuitive and easy to use by both, a casual and an experienced user.

E.2 T H E G R A P H I C A L U S E R I N T E R FA C E

The Graphical User Interface (GUI) was implemented in Qt and programmed in C++. Qt is a cross-platform framework for the development of software, specially for rapid graphical interface deployment across multiple hardware. Currently in its 5.7 version, it is widely used in many different scenarios, ranging from purpose-specific applications to full OS desktop environments. Since GP is the sole algorithm for model generation, LeafGP is built around a modified version of GPOCL, providing very efficient properties given the parallel nature of its design. In the following subsections we will explore LeafGP throughly; first, we will briefly describe some characteristics of the GPOCL implementation and how it is utilized by LeafGP; and then, we will outline the different functionalities from the user interaction point of view.

E.2.1 GPOCL

GPOCL is a freely available high-performance implementation written in C++ and OpenCL that implements a canonical GP system using a prefix linear tree representation [START_REF] Augusto | Accelerated parallel genetic programming tree evaluation with OpenCL[END_REF]. OpenCL is an open standard for uniform and portable parallel programming across heterogeneous computing platforms. It aims to provide low and high level access to data-or task-parallel devices, either individually or simultaneously. Besides the conventional multi-core CPUs and GPUs from multiple vendors, OpenCL also supports some other parallel devices, such as FPGA, Digital Signal Processors (DSP), IBM's Cell Broadband engine, and more.

E.2.1.1 Programming model

The OpenCL's Application Program Interface (API) exposes the access to the parallel computing devices by querying the host system to determine which platforms are present. A platform is a particular OpenCL implementation installed on the system. From the platform it is possible to discover which compatible computing devices are available and then select the ones according to some criteria defined by the programmer; a criterion, for instance, may be a particular architecture, like CPU or GPU. Given one or more devices, a memory buffer can be allocated on them, and a command queue, responsible for kernel submissions to the devices, can be set up.

E.2.1.2 General structure

The main routine of GPOCL performs the following high-level procedures in the given order.

1. OpenCL initialization: this is the step where the general OpenCLrelated tasks are initialized. An OpenCL platform is discovered, a device representation is created and its specifications are extracted, and a command queue is created.

2. Calculating n-dimensional ranges: defines how much parallel work there will be and how they are distributed among the compute units; in other words, local size and global size parameters are calculated.

3. Memory buffers creation: in this phase all global memory regions accessed by the OpenCL kernels are allocated on the device and possibly initialized. The fitness cases are transferred and enough space is reserved for the population and error vectors.

4. Kernel building: an OpenCL kernel, relative to a given strategy of parallelization, is compiled just-in-time, targeting the compute device. Before the actual compilation, an optimized GP interpreter is assembled and helper functions and runtime parameters/flags are embodied in the kernel source.

5. Evolving: this iterative routine implements the actual GP dynamics.

E.2.1.3 Evolutionary algorithm

Most of the steps in the GP cycle are performed sequentially and performed by the host machine, but the costly procedure of evaluation of programs are done in parallel by the target computing device, be it the CPU or GPU. Moreover, LeafGP extends this behavior by implementing also the evaluation in a sequential procedure. This provides to the user further flexibility in the deployment stage, specifically when the parallel hardware is not available, thus allowing GP to be executed in a single CPU core.

The evaluation step itself does not do much, because the hard work is done mostly by the OpenCL kernels. Basically, three things happen within Evaluate (P):

1. Population transfer: all programs of P are transferred to the target compute device. For the GPU, due to the physical separation between the host memory and the device memory, an explicit copy or synchronization is required.

E.2.2 LeafGP

In general, LeafGP has four different functional areas: data loading and exploration, GP settings, GP execution and results analysis. Since LeafGP is intended to cover two main audiences, each area is divided in two subareas: the express, for the casual and inexperienced user that does not have to worry about changing parameters; and the advanced, for the experienced user with knowledge of GP, where every possible parameter is tunable.

LeafGP is designed to be executed over two threads. The first thread is responsible of the GUI responsiveness, while the second thread executes complete GP cycles. This scheme is quite common in graphical interfaces that requires heavy calculations underneath. The main GUI is enclosed in the parent class MainWidget and is connected through the Qt system of signal and slots to communicate among threads. This mechanism allows the asynchronous transfer of data between the GP cycle and the presentation of information. For example, once GP is started, it sends all sort of information relevant to the solutions fitness, population statistics, consumed resources, etc. to the user so the algorithm's performance can be reviewed in real-time.

A parent class is used to enclose all the data used during the execution of LeafGP, namely gpExperiment . The data in gpExperiment is organized as follows:

• Performance information of a GP run.

• Processing of input data through the Dataset class.

• GP parameters used during an experiment.

• In a hierarchical scheme, information of all GP populations is stored in this order: 1) run, 2) generation, 3) population, 4) individual.

This data is shared along all the different areas in LeafGP, available for reading or writing. In the data loading section, samples are fed to a Dataset object. Once the user has finished manipulating the data, then it is transfered to a gpExperiment object. GP parameters are also stored in the aforementioned object.

The main GP cycle is implemented in the GP class, where the second thread is executed, namely the workerGP class, providing responsiveness to the GUI. GP includes a series of signal and slots to pass information to the main thread. This mechanism has two objectives: first, to give a visual feedback to the user; and second, to store into gpExperiment partial information related to the GP run. If for any reason the GP cycle is interrupted, the user can still visualize and explore the partial run.

Once a GP run has been completed, then all sort of information is available either for analysis or for generation of reports.

In the following subsection, each part of LeafGP is described and can be used as a quick user guide.

E.2.2.1 Application navigation

As briefly mentioned before, LeafGP contains four distinctive parts, which will be explained below:

1. Data exploration. Input data can be analyzed and pre-processed before the algorithm is executed.

• Express (Fig. E.1). New or existing projects can be created or loaded, respectively. A project is the easiest way to quickly retrieve a previously executed experiment, which could contain any combination of the dataset, the GP settings, the population information or the summary from results. If a new project is created the two options can be chosen: load a prebuilt dataset from a list of benchmark problems or load a stored dataset. In the current version, the latter supports Comma Separated Values (.csv) and Matlab Uncompressed files (.mat). Once the data has been loaded, a quick summary from the data is shown.

• Advanced (Fig. three plots available depending on the selection of the variable in the dataset: pearson correlation, multiple linear regression and quantile-quantile (Q-Q). Processing, at the moment is being developed.

2. Algorithm settings. Parameters heavily influence on performance and solutions quality. These can be tweaked to better fit algorithm expectations.

• Express (Fig. E.3). GP parameters are automatically defined without the intervention of the user, either from an analysis of the data or hyper-parameterization.

• Advanced (Fig. E.4). Eight pages are included in this part.

Experiment, where the settings related to the run are defined here, like the number of runs or the type of the data partition.

Population, the size of population or type of initialization is specified here; a pre-built population can be loaded as well.

Selection, where the GP selection schemes are defined. Functions, by using a drag-and-drop the user can select the function set. Terminals, default terminals are set according with the loaded dataset, but the user can remove or add new ones.

Operators, the canonical search operators are tuned here by changing probabilities. Fitness, in the current version it is still in development. Others, accelerated processing is available depending on the current hardware where LeafGP is executed; the actual support follows closely to the GPOCL implementation, that is either CPU or GPU.

3. GP Launcher. GP launcher based on predefined settings.

• Express (Fig. E.5). The main information from the GP run is displayed here, together with data related to the best solution of the evolution, like a tree representation or a symbolic representation in a prefix syntax.

• Advanced (Fig. E.6). Two different types of information can be explored. Performance, convergence plots of quality and size are displayed. Population, an interactive population interface, where a dot-matrix graph represents all the individuals during the run; the user can review origin of operator, size and quality for any chosen individual. The colormap can be changed according with the relative fitness, operator or relative size. The genealogy for any individual can be analyzed as well by selecting a dot and pressing the ancestor button.

4. Results. Evolved solutions can be explored and analyzed.

• Express (Fig. E.7). The best solution for the last run is displayed here, with its error and the correlation with the expected data.

• Advanced (Fig. E.8). Four different sections can be selected by the user. Summary, where a quick condensation of the last run is displayed including an evolvability plot. Population, here single individuals from full populations in multiple runs can be reviewed. Statistics, in case that three or more runs are executed, then boxplots for size and quality are displayed.

Saving, where the results can be stored in a single file, with the flexibility of choosing different level of details.

• Report (Fig. E.9). In the current version, a PDF file containing user selected information can be generated.

E.2.2.2 Implementation details

The design of LeafGP reduces external dependencies as much as possible. In the current version, it contains the following dependencies:

• OpenCL SDK. Depending in the hardware vendor, a SDK is required in order to be compiled, this being either CPU or GPU hardware.

• OpenBLAS. Any variant of the linear algebra library is required, however an optimized version is preferred. The source code has been licensed as LGPL 2.1. LeafGP is freely available in a github repository 16 . Because of the nature of Qt framework cross-platform scope, the code can be compiled in Linux, OSx and Windows. However, only the deployment in Linux environment has been tested. The source code has been compiled in a Intel Xeon@3.3GHz CPU with 4-cores, running Kubuntu 17.04 64-bits, Qt version 5.7.

E.3 C O N C L U S I O N S

In this work, a graphical open source GP tool for model extraction has been presented, which is based on Qt and GPOCL. Although this software is intended as a mid-long term development, it currently packs a number of basic features that make it user friendly. It contains data exploration, GP parameterization, real-time plotting and graphical results capabilities. Parallel processing is available depending on the installed hardware. Further development will require more resources, and given the open source characteristic of LeafGP, the authors actively encourage that any enthusiastic developer join the project and contribute, either by providing new ideas or simply by coding.

Figure

  Figure 1.1 PhD roadmap and timeline. . . . . . . . . . . . . 4 Figure 2.1 EEG cap placement using wet based electrodes. . Figure 2.2 EEG digital recordings at the acquisition station. Figure 2.3 Electrodes placement over scalp according with the 10-20 standard. . . . . . . . . . . . . . . . . . Figure 3.1 Proposed system for automatic epileptic seizures detection. . . . . . . . . . . . . . . . . . . . . . . . Figure 3.2 Regression calculated over a point of the signal. Left image shows a dyadic wavelet decomposition, and the right image display the actual regression calculated over the point t 0 , where each dot corresponds to each log 2 of the wavelet coefficient magnitude located approximately above t 0 . Figure 3.3 (b) Hölderian regularity calculated over a sample signal (a), where α is the estimated Hölder exponent. . . . . . . . . . . . . . . . . . . . . . . . Figure 3.4 MP Heisenberg boxes over a sample signal. Colors correspond to the relative atom amplitudes. . Figure 3.5 The pointwise Hölder exponent α (second column) and MP Heisenberg boxes (third column) calculated over the first epoch of each group in the Bonn data set (first column). . . . . . . . . . . Figure 3.6 The pointwise Hölder exponent α (second column) and MP Heisenberg boxes (third column) calculated over the first epoch of each group in the Bonn data set (first column). . . . . . . . . . . Figure 3.7 Matrix of pairwise projections of all features for all five classes. The diagonal plots correspond to the density distribution per feature. . . . . . . . .

Figure 4. 1

 1 Figure 4.1Visualization of the trust region algorithm showing the landscape of f (θ) as the objective function, equivalent to Equation4.4, at iteration j. . .

Figure 4. 2

 2 Figure 4.2 Results for problem Keijzer-6 plotted with respect to total function evaluations: (a) Fitness over test data; and (b) Average program size. Both plots show median values over 30 independent runs. . . . . . . . . . . . . . . . . . . . . . . .

Figure 4. 3

 3 Figure 4.3 Results for problems Korns-12 (a,c,e) and Vladislavleva-4 (b,d,f): (a,b) Fitness over test data; (c,d) Fitness over training data; and (e,f) Average program size. All plots show median values over 30 independent runs. . . . . . . . . .

Figure 4. 4

 4 Figure 4.4 Results for problem Nguyen-7 plotted with respect to total function evaluations: (a) Fitness over test data; and (b) Average program size. Both plots show median values over 30 independent runs. . . . . . . . . . . . . . . . . . . . . . . .

Figure 4. 5

 5 Figure 4.5 Results for problem Pagie-1 plotted with respect to total function evaluations: (a) Fitness over test data; and (b) Average program size. Both plots show median values over 30 independent runs. . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4. 6

 6 Figure 4.6 Results for Tower problem plotted with respect to total function evaluations: (a) Fitness over test data; and (b) Average program size. Both plots show median values over 30 independent runs. . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure

  Figure 5.1 Example of tree transformation, product of parameterization and subtree addition. . . . . . . .

Figure 6. 3

 3 Figure 6.4

Figure

  Figure 6.19

Figure 1

 1 Figure 1.1: PhD roadmap and timeline.

Figure 2

 2 Figure 2.1: EEG cap placement using wet based electrodes.

Figure 2

 2 Figure 2.2: EEG digital recordings at the acquisition station.

Figure 3

 3 Figure 3.3: (b) Hölderian regularity calculated over a sample signal (a), where α is the estimated Hölder exponent.

Figure 3

 3 Figure 3.4: MP Heisenberg boxes over a sample signal. Colors correspond to the relative atom amplitudes.

  Figure 3.5: The pointwise Hölder exponent α (second column) and MP Heisenberg boxes (third column) calculated over the first epoch of each group in the Bonn data set (first column).

  Figure 3.6: The pointwise Hölder exponent α (second column) and MP Heisenberg boxes (third column) calculated over the first epoch of each group in the Bonn data set (first column).

Figure 3 . 7 :

 37 Figure 3.7: Matrix of pairwise projections of all features for all five classes. The diagonal plots correspond to the density distribution per feature.

Figure 3 . 8 :

 38 Figure 3.8: Optimal feature frequency for GA algorithm. Each bar value is the accumulative feature appearance after running over all 10 folds.

  : One-way ANOVA test for complete feature set. Mean and standard deviation per class are shown in columns 2 through 6. p-values for each feature are shown in last column.

  seizures by means of signal processing Table3.8: Summary of classification performance processed over full length epochs, employing a 10-fold cross validation and 10 independent runs. Includes raw and normalized signals. Columns show the average Specificity, Recall, F-score and rank statistics of the accuracy, including median, best, worst and Interquartile Range (IQR).

  seizures by means of signal processing Table3.12: Friedman pairwise tests, showing adjusted p-values with the Benjamini-Hochberg correction; bold values indicate that the null hypothesis is rejected at the α = 0.05 significance level.

  information of how the train/test partition was done or other details in the experimental setup. processing Table3.14: Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy. missing information of how the train/test partition was done or other details in the experimental setup.

71

  Based on Z-Flores, E.,[START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF]. Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability, Set Oriented Numerics, and Evolutionary Computation V, volume 288, pages 213-228 4.1 I N T R O D U C T I O N

Figure 4

 4 Figure 4.1: Visualization of the trust region algorithm showing the landscape of f (θ) as the objective function, equivalent to Equation 4.4, at iteration j.

  Figure 4.2: Results for problem Keijzer-6 plotted with respect to total function evaluations: (a) Fitness over test data; and (b) Average program size. Both plots show median values over 30 independent runs.

Figure 4 . 3 :Figure 4 . 5 :

 4345 Figure 4.3: Results for problems Korns-12 (a,c,e) and Vladislavleva-4 (b,d,f): (a,b) Fitness over test data; (c,d) Fitness over training data; and (e,f) Average program size. All plots show median values over 30 independent runs.

  A Local Search Approach to Genetic Programming for Binary Classification. In Proceedings of the 2015 Annual Conference on Genetic and Evolutionary Computation, GECCO'15, pages 1151-1158. ACM 5.1 I N T R O D U C T I O N

Figure 5 . 1 :

 51 Figure 5.1: Example of tree transformation, product of parameterization and subtree addition.

Figure 5 . 2 :

 52 Figure 5.2: Example of the optimization effect over an actual individual for the Parkinsons problem (Littleet al., 2007). Even though the solution was clearly a bad classifier, after optimization accuracy improved.Projection of both classes are shown for a clearer visualization on the position of instances relative to the classification threshold.

  the Local Search mechanism within GP

Figure 5 . 4 :

 54 Figure 5.4: Transfer function for p(s) corresponding to the heuristic method based on individuals tree sizes, selected for performing LS.

  Fig. 5.5 and Fig. 5.6 summarizes the results of the tested algorithm over several classification problems. The figure shows convergence plots for training and testing fitness with respect to the number of fitness function evaluations, showing the median over all runs.In each of the problems, the accuracy of LSHS testing data is either similar or better compared to standard GP. At least in four of them the improvement in the method is significant.Figures 5.5(a-c) show the results for the problems Parkinsons, Diabetes and Wine. In all of them we can see a superior solution quality evolved by the LSHS method. Notice that for the Wine problem, considered an easy classification problem, standard GP reaches almost zero error for testing data, yet the LSHS algorithm presents a perfect accuracy.

Figures 5

 5 Figures 5.5(d-f) present the performance results for Sonar, Wholesale and Banknote while Figure 5.6 presents results for LSVT problem.For the Sonar problem the accuracy reached in the testing data is similar to the one obtained by standard GP. In the Banknote problem, the LSHS improvement is significant, where the classification error almost reaches zero.Figure5.7(a) discloses descriptive statistics from test fitness, at the end of each run, using notched box plots. The Wilcoxon rank sum test was calculated for the fitness test data, shown in first column of Table5.3. The average population size is also presented in Figure5.7(b), and its corresponding rank sum test is given in the second column of Table5.3.In both cases, the null hypothesis is that the underlying distributions in each pair of experiments have equal medians.The statistical tests show that that the differences in performance exhibited by the standard GP and LSHS-GP are significant on four of the seven problems, particularly Parkinsons, Diabetes, Wholesale and Banknote, with α = 0.05. Regarding size, statistical difference was detected on five problems, namely Parkinsons, Diabetes, Wine, Wholesale and

  Figure 5.7: Box plots from 20 runs at the end of each run; where: (a) presents the test fitness of the best solution found during testing; and (b) the average population size.

Figure 6 . 2 :

 62 Figure 6.2: Relative position of the electrodes based on the international standard system 10/10.

Figure 6 . 3 :

 63 Figure 6.3: Photograph that illustrates the cap positioning procedure. Conductive gel is used to reduce the electrical impedance between the scalp and the electrode.

Figure 6 . 4 :

 64 Figure 6.4: Acquisition protocol of the CNV, describing the stimuli presented to each subject. Each task consists of 50 pairs of stimuli.

Figure 6

 6 Figure 6.5: Relationship of the vigilance level and the amplitude on the CNV test. Main ERP components are shown here: P1, N1, P2, N2 and P3.

Figure 6 . 7 :

 67 Figure 6.7: Frequency response for the Butterworth filter used in this work.

Figure 6 . 8 :

 68 Figure 6.8: Classification performance for the reference system.

Figure 6 .

 6 Figure 6.12: + FEGP individual representation.

Figure 6

 6 Figure6.17: Operator probabilities through the search.

Figure 6

 6 Figure 6.18: Class variance penalization, where (a) corresponds to the tree output, in this example f 1 , f 2 ∈ i = 1 . . . r are new features; and (b) corresponds to the data modification with a larger distribution variance used to compute the global fitness.

Figure 6 Figure 6

 66 Figure 6.19: Training performance of the reference system and the proposed enhanced system for each LOOCV fold. In the center, mean values for all fold are shown.

  Figure 6.21: Box plots of the enhanced system performance for 30 runs per fold, where (a) corresponds to the classification accuracy for the training partition and (b) is the homologous for the testing partition.

Figure 6 Figure 6 Figure 6 Figure 6 Figure 6 Figure 6

 666666 Figure 6.22: LDA classification accuracy and convergence plots for the evolution of fitness on the training data. Top plot shows the history of LDA accuracy calculated over the best individual per fold; it represents the median over 30 independent runs. The bottom plot shows the evolution of fitness from the best individual representing the mean of the 13 (folds) medians over 30 runs.

  Figure A.1 summarizes the main results. The convergence plots of GP and GP-LS are shown in figure A.1(a), showing the median training and testing performance. The figure clearly shows that GP-LS converges faster to a lower error, and at the end of the run it substantially outperforms standard GP, consistent with (Z[START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF]-Flores et al., , 2015)).

  Figure A.1(b) presents a scatter plot (each point is one individual) of all individuals generated in all runs. The individuals are plotted based on function evaluations and size. Each individual is color coded using a heat map based on test performance, with the best individuals (lowest error) in dark gray.

  Figure A.1(b) shows that the best performance is achieved by the largest individuals.

  Figure A.2: Influence of the LS operator on the construction of the best solution found for the Yacht problem. Left plot (a) shows the individual ratio of the best individuals that have been chosen for LS versus the ones that do not. The right plot (b) shows the same behavior but in a generation by generation basis, highlighting the ratio in the Z-axis.

Figure

  Figure B.1: Graphical depiction of a generic GT system.

Figure

  Figure B.2: Turbo-Generator at the Chankanaab Turbo-Gas Central.

Figure B. 3 :

 3 Figure B.3: Fuel flow path in a GT power unit.

  Figure B.4: Plots show 3D point clouds of the real data collected from the GT unit.

  Case study: Gas turbine modelingB.3 G E N E T I C P R O G R A M M I N G B.3.1 Applying GP to Model Real-World Systems

Figure

  Figure B.5 presents a summary of all results as bloxplots that show the median, first and second quartiles, as well as the maximum and minimum performance over all thirty runs for both problems. Figure B.5(a) shows the training error on the Flow problem and Figure B.5(b) shows the same for the Temperature problem. Conversely, Figure B.5(c) shows the test error on the Flow problem while Figure B.5(d) shows the same for the Temperature problem.Moreover, TableB.4 summarizes these results numerically, reporting the median train and test performance of each algorithm on each problem, showing in bold the best (lowest) results. What immediately stands out from these results is that GP-LS outperforms all other algorithms in all cases except one, testing performance on the Flow problem, however there are runs that still are better than the other algorithms. Also notice that while performance on the Flow problem is very similar for all algorithms, on the Temperature problem performance varies quite substantially, with GSGP performing the worst and GP-LS performing the best. Another relevant observation is related to the variance in performance of each method over all runs. First, on the Flow problem, at one extreme HYBRID-LIN exhibits very stable behavior in all runs, suggesting that the linear scaling approach increases the robustness of GP search. The other extreme is GP-LS, which shows the largest variance on this problem, with several runs achieving substantially smaller error values than the other methods. Second, on the Temperature problem, all methods exhibit comparatively less variance, even GP-LS behaves quite consistently over all runs.To test the significance of these results, we use the Friedman test to perform all pairwise comparisons of the methods, focusing specifically on test performance for both problems. The null hypothesis in each pairwise comparison is that the groups have equal medians, and the p-values

  Figure B.5 presents a summary of all results as bloxplots that show the median, first and second quartiles, as well as the maximum and minimum performance over all thirty runs for both problems. Figure B.5(a) shows the training error on the Flow problem and Figure B.5(b) shows the same for the Temperature problem. Conversely, Figure B.5(c) shows the test error on the Flow problem while Figure B.5(d) shows the same for the Temperature problem.Moreover, TableB.4 summarizes these results numerically, reporting the median train and test performance of each algorithm on each problem, showing in bold the best (lowest) results. What immediately stands out from these results is that GP-LS outperforms all other algorithms in all cases except one, testing performance on the Flow problem, however there are runs that still are better than the other algorithms. Also notice that while performance on the Flow problem is very similar for all algorithms, on the Temperature problem performance varies quite substantially, with GSGP performing the worst and GP-LS performing the best. Another relevant observation is related to the variance in performance of each method over all runs. First, on the Flow problem, at one extreme HYBRID-LIN exhibits very stable behavior in all runs, suggesting that the linear scaling approach increases the robustness of GP search. The other extreme is GP-LS, which shows the largest variance on this problem, with several runs achieving substantially smaller error values than the other methods. Second, on the Temperature problem, all methods exhibit comparatively less variance, even GP-LS behaves quite consistently over all runs.To test the significance of these results, we use the Friedman test to perform all pairwise comparisons of the methods, focusing specifically on test performance for both problems. The null hypothesis in each pairwise comparison is that the groups have equal medians, and the p-values

  Figure B.5: Experimental results showing the train (a,b) and test (c,d) performance of all algorithms on each problem, Flow (a,c) and Temperature (b,d).

  Figure B.6 presents a graphical depiction of the best models evolved by GSGP-LS on each problem, showing 3D point clouds of the real data compared with the predicted values of the GSGP-LS models over the entire dataset. The GP-LS models were chosen because they achieved the minimum error (see the boxplots of FigureB.5) and because the models are the most compact, which is desirable in a real-world deployment of the models. Notice that in both cases the models are able to closely approximate the real behavior of the GT unit, accurately predicting the behavior of both variables of interest: fuel flow and exhaust temperature.

  Figure B.6: Plots show 3D point clouds of the real data collected from the GT unit and the output of the best models produced by the GP-LS method.

  Figure B.7: Experimental results showing the train test performance of all algorithms on each problem, Flow (a) and Temperature (b).

  Figure B.8: Experimental results showing the size of the solutions generated by GP-LS, MARS and FFC on both problems: (a) Flow; and (b) Temperature.

C

  C A S E S T U D Y: R F -PA S M O D E L I N G Based on Cárdenas Valdez, J. R., Z-Flores, E., Núñez Pérez, J. C., and Trujillo, L. (2017). Local Search Approach to Genetic Programming for RF-PAs Modeling Implemented in FPGA. In NEO 2015, Studies in Computational Intelligence, pages 67-88. Springer C.1 I N T R O D U C T I O N

Figure C. 1 :

 1 Figure C.1: General overview of a RF-PA based on distortion curves.

  subdivided into MPM terms based on the sampling time of the input signal. The figure C.2 shows the internal structure and the delay made for each step and the overview of this model can be noted that phase shift of the signal during the first cycle is increased. In the figure C.3, equation C.7 is represented for each F q stage. C.4 G E N E T I C P R O G R A M M I N G W I T H L O C A L S E A R C H

Figure

  Figure C.5: AM/PM conversion curve for the RF-PA Doherty 7W @ 2.1 GHz.

Figure

  Figure C.7: AM/PM conversion curve expressed as voltage.

  Figure C.10 depicts the AM/AM model generated by GP-LS, which, as stated before, consists of 17 nodes and contains 17 parameters to reproduce the AM/AM behavior. Similarly, figure C.11 depicts the AM/PM model generated by GP-LS, which has the same size and the same number of parameters.

Figure C. 8 :

 8 Figure C.8: General MPM structure for RF-PA modeling.

Figure

  Figure C.9: DSP Builder chain for the specified board.

  Figure C.10: DSP Builder Tool implementation of the GP-LS model using 17 parameters for the AM/AM conversion curve.

Figure

  Figure C.12: RF-PA Doherty 7W@2.1 GHz AM/AM using a traditional modeling technique as MPM with NL=15.

Figure

  Figure C.14: RF-PA Doherty 7W@2.1 GHz AM/PM using a traditional modeling technique as MPM with NL=15.

Figure

  Figure C.15: RF-PA Doherty 7W@2.1 GHz AM/PM using a traditional modeling technique as MPM with NL=15 and white noise.

Figure

  Figure C.16: RF-PA Doherty 7W@2.1 GHz AM/AM using the GP-LS model with 17 parameters.

Figure

  Figure C.17: RF-PA Doherty 7W@2.1 GHz AM/AM using the GP-LS model with 17 parameters, with white noise.

Figure

  Figure C.18: RF-PA Doherty 7W@2.1 GHz AM/PM using the GP-LS model with 20 parameters.

Figure

  Figure C.19: RF-PA Doherty 7W@2.1 GHz AM/PM using the GP-LS model with 20 parameters, with white noise.

D

  C A S E S T U D Y: A D S O R P T I O N O F P H E N O L S A N D N I T R O P H E N O L S M O D E L I N G Based on Z-Flores, E., Abatal, M., Bassam, A., Trujillo, L., Juarez-Smith, P., and El Hamzaoui, Y. (2017). Modeling the Adsorption of Phenols and Nitrophenols by Activated Carbon using Genetic Programming. Journal of Cleaner Production D.1 I N T R O D U C T I O N

  modeling

  Figure D.1: Two example of SEM images of activated carbon at 1.50K magnification.

  Figure D.2: Schematic diagram of the experimental adsorption process used in this work. Variable Type Operating Range Units Contaminant (C) Input {1, 2, 3, 4} -Initial concentration (C i ) Input [20, 100] ppm pH Input [2, 10] -Time (t) Input [0, 360] minutes Adsorption efficiency (W %) Output [0, 100]

  Figure D.3: Analysis of the experimental data.

Figure D. 4

 4 Figure D.4 summarizes the performance of each algorithm using boxplots that depict rank statistics (maximum, minimum, first quar-

  Figure D.4(a) presents the MSE of each algorithm on the training data, while Figure D.4(b) shows the same on the test data. Moreover, Table

  Figure D.4: Boxplots that show the MSE performance of each algorithm on the training (a) and testing (b) data over all thirty runs. Numerical values represent the median.

  Figure D.6: Size comparison between basis function based methods (a) and GP-based methods (b).

  Figure D.7: Scatter plots of all 30 GP-LS models, relating the ground truth adsorption efficiency W % with the estimated value produced by each model, on both the training data (a) and the test data (b); in both cases the figures provides the value of the Pearson correlation coefficientρ.

  Figure D.8: Breakthrough curves for model K A , comparing the ground truth W % (points) with the model estimates (solid curves).

E

  L E A F G P : V I S UA L I Z AT I O N O F R E G R E S S I O N M O D E L I N G U S I N G G P E.1 I N T R O D U C T I O N

Figure E. 1 :

 1 Figure E.1: Data exploration. Express page.

Figure E. 2 :

 2 Figure E.2: Data exploration. Advanced page.

Figure E. 3 :

 3 Figure E.3: Algorithm settings. Express page.

Figure E. 4 :

 4 Figure E.4: Algorithm settings. Advanced page.

Figure E. 5 :

 5 Figure E.5: GP Launcher. Express page.

Figure E. 6 :

 6 Figure E.6: GP Launcher. Advanced page.

Figure E. 7 :

 7 Figure E.7: Results. Express page.

Figure

  Figure E.8: Results. Advanced page.

  -Marie Curie-IRSES 2013 European Commission program, through the ACoBSEC project, whom gave me full support in multiple occasions, including internships in Bor-E LeafGP: visualization of regression modeling using GP E.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . E.2 The Graphical User Interface . . . . . . . . . . . . . . . . . E.2.1 GPOCL . . . . . . . . . . . . . . . . . . . . . . . . E.2.2 LeafGP . . . . . . . . . . . . . . . . . . . . . . . . . E.3 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . Electrodes placement over scalp according with the 10-20 standard. . . . . . . . . . . . . . . . . . Figure 3.1 Proposed system for automatic epileptic seizures detection. . . . . . . . . . . . . . . . . . . . . . . . Figure 3.2 Regression calculated over a point of the signal.

		L I S T O F F I G U R E S
	Figure 1.1	PhD roadmap and timeline. . . . . . . . . . . . . 4
	Figure 2.1	EEG cap placement using wet based electrodes. .
	Figure 2.2	EEG digital recordings at the acquisition station.
	Figure 2.3
	Bibliography
	xiv	xiii

Table 3 .

 3 Figure C.2 Overview of F q (n) as block diagram. . . . . . . . Figure C.3 Description of each stage of F q with the Volterra Kernels. . . . . . . . . . . . . . . . . . . . . . . . . Figure C.4 AM/AM conversion curve for the RF-PA Doherty 7W @ 2.1 GHz. . . . . . . . . . . . . . . . . Figure C.5 AM/PM conversion curve for the RF-PA Doherty 7W @ 2.1 GHz. . . . . . . . . . . . . . . . . LS model with 20 parameters, with white noise. . . . . . . . . . . . . . . . . . . . . . . . . . Figure D.1 Two example of SEM images of activated carbon at 1.50K magnification. . . . . . . . . . . . . . . . Breakthrough curves for model K A , comparing the ground truth W % (points) with the model estimates (solid curves). . . . . . . . . . . . . . . Figure D.9 Breakthrough curves for model K B , comparing the ground truth W % (points) with the model estimates (solid curves). . . . . . . . . . . . . . . Figure D.10 Analysis of use frequency of each input feature: (a) shows how often a feature was used in the best model found; (c) the proportion of terminal elements that each feature represents in each tree. . . . . . . . . . . . . . . . . . . . . . . . . . . Figure E.1 Data exploration. Express page. . . . . . . . . . . Figure E.2 Data exploration. Advanced page. . . . . . . . . . Figure E.3 Algorithm settings. Express page. . . . . . . . . . Figure E.4 Algorithm settings. Advanced page. . . . . . . . . 14 Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy. . . . . . . . . . . . . . . . . . . . . Table 3.15 Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy. . . . . . . . . . . . . . . . . . . . . Table 3.16 Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy. . . . . . . . . . . . . . . . . . . . . Table 4.1 GP Parameters for the different methods. . . . . Table 4.2 General GP parameters . . . . . . . . . . . . . . . Table 4.3 Summary of median fitness computed over the test set of each problem. The Sample column indicates the number of function evaluations performed; bold indicates the best results. . . . . . . Table 5.1 Binary classification problems summary used to evaluate proposed algorithm in this work. . . . . Table 5.2 GP parameters . . . . . . . . . . . . . . . . . . . . Table 5.3 Wilcoxon rank sum test, with α = 0.05. Bold values are less than α. . . . . . . . . . . . . . . . . Table 5.4 Classification accuracy comparison among several state-of-art methods, including from the GP community. . . . . . . . . . . . . . . . . . . . . . FEGP parameters used in the experiments. . . . Table 6.2 Statistical pairwise tests corresponding to the training performance. The upper triangular matrix encloses the adjusted p-values with the Benjamini-Hochberg correction of the Friedman test and the lower triangular matrix consist of the p-values for a Wilcoxon rank sum test. Bold values indicate that the null hypothesis is rejected at the α = 0.05 significance level for both tests. . . . . . . . . . . . . . . . . . . . . . . . Table 6.3 Statistical pairwise tests corresponding to the testing performance. The upper triangular matrix encloses the adjusted p-values with the Benjamini-Hochberg correction of the Friedman test and the lower triangular matrix consist of the p-values for a Wilcoxon rank sum test. Bold values indicate that the null hypothesis is rejected at the α = 0.05 significance level for both tests. . . . . . . . . . . . . . . . . . . . . . . . Table 6.4 Classifiers performance for the + FEGP algorithm. Values represent means and standard deviation over 13 folds. Last row shows the oneway ANOVA test between classifiers; bold value indicates that the null hypothesis is rejected at an α = 0.05 significance level. . . . . . . . . . . . Table A.1 GP parameters. . . . . . . . . . . . . . . . . . . . Table B.1 Monitored variables in the Mitsubishi single shaft Turbo-Generator Model MS6001 that are related to fuel flow. . . . . . . . . . . . . . . . . . Table B.2 Monitored variables in the Mitsubishi single shaft Turbo-Generator Model MS6001 that are related to exhaust gas temperature. . . . . . . . . Table B.3

	Figure C.6	AM/AM conversion curve expressed as voltage. .
	Figure C.7	AM/PM conversion curve expressed as voltage. .
	Figure C.8	General MPM structure for RF-PA modeling. . .
	Figure C.9	DSP Builder chain for the specified board. . . . .
	Figure C.10	DSP Builder Tool implementation of the GP-LS
		model using 17 parameters for the AM/AM con-
	Figure C.12	RF-PA Doherty 7W@2.1 GHz AM/AM using
		a traditional modeling technique as MPM with
		NL=15. . . . . . . . . . . . . . . . . . . . . . . . .
	Figure C.13	RF-PA Doherty 7W@2.1 GHz AM/AM using
		a traditional modeling technique as MPM with
		NL=15 and white noise. . . . . . . . . . . . . . . .
	Figure C.14	RF-PA Doherty 7W@2.1 GHz AM/PM using a
		traditional modeling technique as MPM with
		NL=15. . . . . . . . . . . . . . . . . . . . . . . . .
		xxi

version curve. . . . . . . . . . . . . . . . . . . . . Figure C.11 DSP Builder Tool implementation of the GP-LS model using 20 parameters for the AM/PM conversion curve. . . . . . . . . . . . . . . . . . . . . Figure D.2 Schematic diagram of the experimental adsorption process used in this work. . . . . . . . . . . . Figure D.3 Analysis of the experimental data. . . . . . . . . . Figure D.4 Boxplots that show the MSE performance of each algorithm on the training (a) and testing (b) data over all thirty runs. Numerical values represent the median. . . . . . . . . . . . . . . . . Figure D.5 Boxplots that show the R 2 performance of each algorithm on the training (a) and testing (b) data over all thirty runs. Numerical values represent the median. . . . . . . . . . . . . . . . . . . . . . . xxii Figure D.8 Figure E.5 GP Launcher. Express page. . . . . . . . . . . . . Figure E.6 GP Launcher. Advanced page. . . . . . . . . . . . Figure E.7 Results. Express page. . . . . . . . . . . . . . . . . Figure E.8 Results. Advanced page. . . . . . . . . . . . . . . Figure E.9 Results. Report page. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Table 3.4 Configuration for MP algorithm. . . . . . . . . . Table 3.5 GA parameters. . . . . . . . . . . . . . . . . . . . Table 3.6 One-way ANOVA test for complete feature set. Mean and standard deviation per class are shown in columns 2 through 6. p-values for each feature are shown in last column. . . . . . . . . . α = 0.05 significance level. . . . . α = 0.05 significance level. . . . . xxvi xxvii Table 6.1 +

Table 2 .

 2 1: Characteristics of acquisition methods.

		Invasiveness	Spatial	Temporal	resolu-	Portability Recorded
		and medical	resolu-	tion			signal
		issues	tion
	ECoG	Invasive	∼1	mm	∼0.003 s (High)	Portable	Electrical
			(High)
			∼0.5 mm
			(LFP)
	Intracortical Invasive	∼0.1 mm (MUA)	∼0.003s (High)	Portable	Electrical
			∼0.05 mm
			(SUA)
	fMRI	Non-invasive	∼1	mm	∼1 s (Low)		Non-	Metabolic
			(High)			portable
	MEG	Non-invasive	∼5	mm	∼0.05 s (Medium)	Non-	Magnetic
			(Medium)			portable
	fNIRS	Non-invasive	∼5	mm	∼1 s (Low)		Portable	Metabolic
			(Medium)
	EEG	Non-invasive	∼10 mm	∼0.05 s (Medium)	Portable	Electrical
			(Low)

Table 3 .

 3 1: Summary of the Bonn data set. All classes have 100 epochs per class and 4096 samples per epoch.

	Data set	Class Description
	label
	Z	A	Normal, eyes open
	O	B	Normal, eyes closed
	N	C	Seizure free, depth electrodes, hippocampal forma-
			tions
	F	D	Seizure free, depth electrodes, epileptogenic for-
			mations
	S	E	Epiletic activity, depth electrodes, eptic formations
	which are common in a relaxation state

Table 3

 3

	.2: Proposed feature sets. An additional set is selected automati-
	cally, presented in Section 3.3.6
	Type	Number of features
		4	10
	Regularity statistics	MAD(H) MAD(H) µ(H) µ(H)

  Ubeyli, Classification of Epileptic seizures by means of signal processing Table 3.4: Configuration for MP algorithm.

	Description	Value
	Number of atoms per dictionary	5000
	Sampling rate	173.61 Hz
	Stopping criteria	25 dB SNR of reconstruction
	Atom types in dictionary	Gabor with Gaussian window
		Gabor with Cosine window
		Chirp with Gaussian window
		Gabor with Rectangle window

2005

). Therefore, each class contains 400 total epochs (100 × 4). Notice that this increases the number of examples we have for each class, but each example contains less information, possibly making it more difficult to characterize each epoch. Hereafter, we refer to the second variant as the segmented method.

  Classification of Epileptic seizures by means of signal processing Table3.5: GA parameters.

	Parameter	Value
	Population	100
	Generations	50
	Chromosome	Binary string
	Crossover operator	Intermediate, 0.8 prob.
	Mutation operator	Adaptive feasible, 0.15 prob.
	Elitism	0.05%
	Selection method	Tournament
	Fitness function	RF classification accuracy

Table 3 .

 3 7: Summary of classification performance processed over full length epochs, employing a 10-fold cross validation and 10 independent runs. Includes raw and normalized signals. Columns show the average Specificity, Recall, F-score and rank statistics of the accuracy, including median, best, worst and Interquartile Range (IQR).

	γ 2 (E) 3.23±0.27 3.22±0.27 3.65±0.63 4.33±2.59 3.43±1.19 0.0000	γ 1 (E) -0.02±0.11 0.06±0.13 -0.15±0.27 0.08±0.76 -0.06±0.76 0.0092	σ (E) 40.73±8.26 61.11±18.12 50.83±19.17 65.62±57.53 306.61±147.23 0.0000	µ 1/2 (E) -6.23±24.63 -13.08±30.63 -8.42±24.04 -8.72±30.09 6.67±81.36 0.0226	µ(E) -6.26±24.69 -12.51±30.57 -8.88±24.00 -6.20±23.40 -4.75±27.02 0.2437	6.46±0.74 7.12±0.69 5.30±0.44 4.90±0.24 6.50±0.49 0.0000

(a) 4 features

Table 3 .

 3 9: Summary of classification performance processed over segmented epochs, employing a 10fold cross validation and 10 independent runs. Includes raw and normalized signals. Columns show the average Specificity, Recall, F-score and rank statistics of the accuracy, including median, best, worst and Interquartile Range (IQR).

	Normalized	F-score Accuracy	median best worst IQR	0.9259	0.9260 93.5 93.7 93.1 0.60	0.9260	0.9476	0.7817 91.6 92 91 0.50	0.8647	0.8265	0.8062 83.7 0.8531 84.6 82.9 0.83	0.8286	0.4605	0.4122	0.5043 56.4 0.6194 57.2 55.5 0.75	0.7913	0.5575		1.0000	1.0000 100 100 100 0.00	1.0000	0.9962	0.9845 99.5 99.5 99.2 0.00	0.9904	0.9999	0.9999 100 1.0000 100 100 0.00	0.9999	0.9617	0.9584	0.9769 97.6 0.9807 97.8 97.5 0.25	0.9925	0.9740
		Specificity Recall		0.9260 0.9263	0.9263 0.9260	0.9261 0.9261	0.7580 0.9549	0.9549 0.7580	0.8564 0.8564	0.9018 0.8320	0.9076 0.7915	0.9100 0.8630	0.9064 0.8288	0.8070 0.4650	0.8025 0.4130	0.8383 0.4888	0.8460 0.6280	0.9001 0.7998	0.8388 0.5589		1.0000 1.0000	1.0000 1.0000	1.0000 1.0000	0.9780 0.9979	0.9979 0.9780	0.9879 0.9879	1.0000 0.9998	0.9999 1.0000	1.0000 1.0000	1.0000 0.9999	0.9926 0.9530	0.9869 0.9683	0.9953 0.9723	0.9942 0.9840	0.9980 0.9928	0.9934 0.9741
	Raw	F-score Accuracy	median best worst IQR	0.9634	0.9638 97.3 97.5 96.9 0.62	0.9636	0.9724	0.8863 95.5 95.8 95.2 0.00	0.9294	0.7977	0.7842 83.8 0.9338 84.6 83.3 0.83	0.8386	0.6312	0.6777	0.6172 67.1 0.5020 67.8 66.5 0.50	0.8885	0.6633	(b) 9 features (Automatic)	1.0000	1.0000 100 100 100 0.00	1.0000	0.9939	0.9752 99 99.2 99 0.00	0.9845	0.9307	0.9123 94.3 0.9876 95 94.2 0.42	0.9435	0.8447	0.8711	0.7708 83.2 0.6968 83.8 82.8 0.50	0.9781	0.8323
		Specificity Recall		0.9625 0.9648	0.9648 0.9625	0.9636 0.9636	0.8658 0.9781	0.9781 0.8658	0.9219 0.9219	0.8866 0.8095	0.9033 0.7663	0.9547 0.9420	0.9149 0.8393	0.8595 0.6643	0.8933 0.6745	0.8743 0.6210	0.8765 0.4725	0.9522 0.8973	0.8911 0.6659		1.0000 1.0000	1.0000 1.0000	1.0000 1.0000	0.9685 0.9956	0.9956 0.9685	0.9821 0.9821	0.9476 0.9600	0.9751 0.8815	0.9918 0.9903	0.9715 0.9439	0.9367 0.8930	0.9676 0.8593	0.9303 0.7868	0.9400 0.6570	0.9945 0.9743	0.9538 0.8341
				Class A	Class E	Average	Class A,B,C,D	Class E	Average	Class A	Class D	Class E	Average	Class A	Class B	Class C	Class D	Class E	Average		Class A	Class E	Average	Class A,B,C,D	Class E	Average	Class A	Class D	Class E	Average	Class A	Class B	Class C	Class D	Class E	Average
					Problem 1			Problem 2			Problem 3				Problem 4					Problem 1			Problem 2			Problem 3				Problem 4

(a) 4 features

Table 3 .

 3 10: Summary of classification performance processed over segmented epochs, employing a 10-fold cross validation and 10 independent runs. Includes raw and normalized signals. Columns show the average Specificity, Recall, F-score and rank statistics of the accuracy, including median, best, worst and Interquartile Range (IQR).

Table 3 .

 3 11: Friedman pairwise tests, showing adjusted p-values with the Benjamini-Hochberg correction; bold values indicate that the null hypothesis is rejected at the α =

	significance level.
	0.05

Table 3 .

 3 13: Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy.

	AccuracyRecall Specificity		93.55 --	94.3 100 100	96.88 93.3 100	98.87 94.96 99.43	99 93.2 98.9		99.7 --		99.8 100 99.2
	Training/Testing partition Number	fea-of	tures 60/40 random, 100 times 69-92	50/50 random, 10 times holdout 16	a 15	10-fold cross validation 20	50/50 cross fold 30		75/25 483		50/50 a 10 50/50 random, 10 times holdout 13-66
	Classifier		SVM	ANN	ANN	Wavelet Neural	Network Optimized	Extreme Learning	Machine Bagged	Alternating	Decision Trees MLPNN ANN
	Feature post-	processing	No	PCA	No	No	Entropy		No		Entropy PCA
	Authors Feature selection		(Nicolaou and Permutation Entropy	Georgiou, 2012) (Tzallas et al., 2009) Time Frequency	analysis (Divya, 2015) Empirical Mode	Decomposition (Zainuddin et al., DWT	2013) (Murugavel and Wavelet	Ramakrishnan,	2014) (Kovacs et al., 2014) Discrete Short Time	Fourier Transform	(Guo et al., 2010) Multiwavelet (Tzallas et al., 2007) Time Frequency	analysis

Table 3 .

 3 15: Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy.There is missing information of how the train/test partition was done or other details in the experimental setup. processing Table 3.16: Comparison of classification performance among several published approaches using the same database, including our work. Accuracy values are shown as percentile. Best value for each problem is shown in bold. Compared works are grouped by problems and sorted by their accuracy.

	AccuracyRecall Specificity		84.2 83.5 85.6	93.9 97.1 89.1	95.67 99 99.5				96 93.5 98.4		96.7 93.5 98.3		98.1 99.4 100	98.67 98.55 99.33		99 99 99	99.7 100 100			100 100 100	100 100 100
	Training/Testing partition Number	fea-of	tures 62/38 a 18	a 15	a 6				50/50 cross fold 30		random subsampling, cross 18-56	validation, 5000 times & best	model 3-fold cross validation 4	70/30 random, 20 times 30		10-fold cross validation 9	10-fold cross validation 7			60/35 + 5 of validation partition 48	10-fold cross validation 9
	Classifier		Linear	ANN	Decision Tree				Optimized	Extreme Learning	Machine ANN		Fuzzy	Error-Correction	Output Codes	Gaussian Mixture	Model Fuzzy			ANN	RF
	Feature post-	processing	No	No	No				Entropy		K-means		No	Phase Space	Reconstruc-	tion PCA	eigenvalues ANOVA	filtering		No	No
	Authors Feature selection		(Ahammad et al., Wavelet	2014) (Divya, 2015) Empirical Mode	Decomposition (Martis et al., 2013) Higuchi Fractal	Dimension, entropy	over Intrinsic	Time-Scale	Decomposition (Murugavel and Wavelet	Ramakrishnan,	2014) (Orhan et al., 2011) DWT		(Acharya et al., Entropies	2012a) (Niknazar et al., DWT, Recurrence	2013) Quantification Analysis	(Rajendra Acharya Wavelet Packet	et al., 2012) (Acharya et al., Decomposition Entropies, High Order	2012b) Spectra, Hurst	exponent, Higuchi	(Alam and Bhuiyan, Fractal Dimension Empirical Mode	2013) This work Decomposition Regularity & MP
													Problem 3
																				a

Table 4 .

 4 2: General GP parameters

	Parameter	Value
	Runs	30
	Population	500
	Function evaluations	2'000,000
	Training set	as indicated in (McDermott et al., 2012)
	Testing set	as indicated in (McDermott et al., 2012)
	Crossover operator	Standard subtree crossover, 0.9 probability
	Mutation operator	Mutation probability per node 0.05
	Tree initialization	Ramped Half-and-Half, maximum depth 6
	Function set	as indicated in (McDermott et al., 2012)
	Terminal set	Input variables, constants as indicated in (McDermott et al., 2012)
	Selection for reproduc-	Tournament selection of size 3
	tion
	Elitism	Best individual survives
	Maximum tree depth	17

Table 4 .

 4 3: Summary of median fitness computed over the test set of each problem. The Sample column indicates the number of function evaluations performed; bold indicates the best results.

	Sample S-GP LSBEG-GP LSWEG-GP LSRP-GP LSBS-GP LSWS-GP LSALL-GP	250,000 0.2 0.2 0.5 0.54 0.15 0.29 0.46	1000000 0.17 0.17 0.49 0.22 0.13 0.28 0.11	2000000 0.17 0.17 0.5 0.23 0.13 0.25 0.04	250,000 1.04 1.05 1.05 1.05 1.06 1.06 1.04	1000000 1.04 1.05 1.05 1.05 1.06 1.06 1.04	2000000 1.04 1.05 1.05 1.05 1.06 1.06 1.04	250,000 0.28 0.47 0.31 0.35 1.03 0.34 0.5	1000000 1.03 1.33 0.98 0.6 1.89 0.99 0.18	2000000 1.01 1.85 1.18 0.68 4.3 0.74 0.18	250,000 0.02 0.01 0.02 0.005 0.002 0.02 0.003	1000000 0.004 0.009 0.006 0.003 0.001 0.007 0.0006	2000000 0.002 0.008 0.004 0.002 0.001 0.007 0.0003	250,000 0.15 0.1 0.15 0.16 0.02 0.14 0.05	1000000 0.1 0.06 0.07 0.1 0.01 0.09 0.01	2000000 0.07 0.05 0.06 0.07 0.01 0.06 0.006	250,000 67.5 66.9 67 67.4 63.2 66.8 61.98	1000000 65.92 64 64.54 64.39 62 64.52 56.43	2000000 62.31 62.08 62.64 62.03 59.41 62.42 53.53
	Problem		Keijzer-6			Korns-12			Vladislavleva-4			Nguyen-7			Pagie-1			Tower

  Figure 5.3: Example of a ROC curve and its best threshold. Note that each tree will generate a different ROC curve, thus presenting a different classification threshold.

	Reference vector
	Best threshold vector
	Best threshold
	ω i Angle between reference
	vector and threshold vector

Table 5 .

 5 1: Binary classification problems summary used to evaluate proposed algorithm in this work.

	Name		# of instances # of features Brief description
	Parkinsons (Little et al.,	197	23	Biomedical voice measurements
	2007)				related to Parkinson's disease.
	Diabetes	(Lichman,	768	8	Diabetes present on Pima Indi-
	2013)				ans patients.
	Wine (Lichman, 2013)	178	13	Chemical analysis of wines.
	Sonar (Lichman, 2013)	208	60	Sonar signals off metal cylinder
					at various angles and conditions.
	Wholesale	(Lichman,	440	8	Clients information of wholesale
	2013)				distributor.
	Banknote	(Lichman,	1372	5	Image information from gen-
	2013)				uine & forged banknote-like
					specimens.
	LSVT (Tsanas et al.,	126	309	Signal data from speech rehabil-
	2014)				itation treatment on Parkinson
					patients.

Table 5 .

 5 2: GP parameters

	Parameter

Table 5 .

 5 3: Wilcoxon rank sum test, with α = 0.05. Bold values are less than α.

	Fitness	Size

Table 5 .

 5 4: Classification accuracy comparison among several state-of-art methods, including from the GP community.

	Problem	Study	Accuracy (%)	Brief description of the
				study
		Ozcift and Gulten

Table 6 .

 6 1: + FEGP parameters used in the experiments.

	Parameter	Value	Comments
	Generations	50	If the max depth for the evolved trees
			is small then more generations are not
			needed since the solution size is restricted
			and the overall search stagnates.
	Population size	100	A small population was found to be suffi-
			cient to evolve good solutions with larger
			populations reaching the same perfor-
			mance.
	Population initial-	Full	Since the search space can be huge, very
	ization		small individuals are not beneficial to ex-
			plore quickly the solution space.
	Population init max	5 levels	A good compromise between structure di-
	depth		versity and complexity.
	Initial features	3
	Search operators	Inter crossover, Inter feature
		crossover, Intra crossover,
		Mutation, Feature insertion,
		Feature removal
	Initial probabilities	0.2, 0.2, 0.2, 0.05, 0.15, 0.1	Dynamic probabilities.
	Delta values for de-	0.2	This only applies to the Inter-crossover op-
	cay functions Function set	+,-,×,/,log,cos,sin,tan, √ ,abs	erator.
	Terminal set	random [0,1], input features
	Max depth level	20	Since individuals can have several features,
			big values for the allowed depth is not de-
			sired. If a child tree is above this limit
			after the genetic operation, then it is dis-
			carded and a reproduction operation is
			performed instead.
	Selection	Tournament, size 5	A small tournament size provides a richer
			diversity in the population.
	Elitism	Only best individual survives Assures that the best solution is not lost
			from generation to generation.

Table 6 .

 6 2: Statistical pairwise tests corresponding to the training performance. The upper triangular matrix encloses the adjusted p-values with the Benjamini-Hochberg correction of the Friedman test and the lower triangular matrix consist of the p-values for a Wilcoxon rank sum test. Bold values indicate that the null hypothesis is rejected at the α =

		0.0003	0.0033	0.0001	0.0010	0.0001	0.0003	0.0010	0.0033	0.0033	0.0010	0.0003	0.0001	-
		0.5796	0.3834	0.0412	0.5796	0.5796	0.5796	0.0010	0.3834	0.5796	0.0966	0.8273	-	0.0000
		0.5796	0.2102	0.0033	0.8273	0.8273	0.5796	0.0033	0.8273	0.0966	0.5796	-	0.3023	0.0000
	0.05 significance level for both tests.	0.2102 0.0010 0.2102 0.0412 0.3834	0.3834 0.2102 0.5796 0.3834 0.8273	0.0966 0.0001 0.0412 0.0003 0.0010	0.3834 0.0412 0.3834 0.5796 0.3834	0.8273 0.0033 0.5796 0.0966 0.2102	-0.0966 0.5796 0.2102 0.8273	0.0021 -0.0033 0.0966 0.3834	0.6689 0.0005 -0.0966 0.0966	0.1130 0.0741 0.0111 -0.8273	0.2370 0.0441 0.0469 0.6326 -	0.6507 0.0005 0.2576 0.1743 0.2177	0.8014 0.0008 0.7820 0.0268 0.1445	0.0000 0.0004 0.0000 0.0000 0.0000
		0.5796	0.0966	0.0003	0.0412	-	0.8602	0.0000	0.1445	0.0221	0.0782	0.4657	0.4063	0.0000
		0.2102	0.8273	0.0003	-	0.0741	0.2683	0.0048	0.0206	0.4970	0.8405	0.5973	0.0497	0.0000
		0.0033	0.0033	-	0.0000	0.0000	0.0103	0.0000	0.0030	0.0000	0.0000	0.0004	0.0046	0.0000
		0.0412	-	0.0000	0.6327	0.0442	0.1589	0.1020	0.0180	0.9198	0.6873	0.1908	0.0559	0.0000
		1 -	2 0.0168	3 0.0001	4 0.0206	5 0.5799	6 0.6326	7 0.0001	8 0.2370	9 0.0071	10 0.0871	11 0.2176	12 0.6689	13 0.0000

Table 6 .

 6 3: Statistical pairwise tests corresponding to the testing performance. The upper triangular matrix encloses the adjusted p-values with the Benjamini-Hochberg correction of the Friedman test and the lower triangular matrix consist of the p-values for a Wilcoxon rank sum test. Bold values indicate that the null hypothesis is rejected at the α = 0.05 significance level for both tests.

Table 6 .

 6 4: Classifiers performance for the + FEGP algorithm. Values represent means and standard deviation over 13 folds. Last row shows the one-way ANOVA test between classifiers; bold value indicates that the null hypothesis is rejected at an α = 0.05 significance level.

	Name	Training	Testing
	LDA	81.9 ± 2.5 78.8 ± 16.2
	TB	100 ± 0	70.7 ± 15.9
	RF	100 ± 0	70.8 ± 16
	k-NN	100 ± 0	63.2 ± 10.9
	SVM	96.1 ± 1.2 63.5 ± 14.7

Table A .

 A 1: GP parameters.

	Parameter	Value
	Runs	30
	Population	100
	Function evaluations	2'500,000
	Training set	70% of complete data
	Testing set	30% of complete data
	Crossover operator	Standard subtree crossover, 0.9 prob.
	Mutation operator	Mutation probability per node 0.05
	Tree initialization	Full, max. depth 6
	Function set	+,-,×,÷,exp,sin,cos,log,sqrt,tan,tanh
	Terminal set	Input features, constants
	Selection for reproduction Tournament selection of size 7
	Elitism	Best individual survives
	Maximum tree depth	17

Table B .

 B 1: Monitored variables in the Mitsubishi single shaft Turbo-Generator Model MS6001 that are related to fuel flow.

	Variable	Description	Operating Range Units
	et	Average of exhaust gas temperature	[0, 1370]	• F
	cp	Compressor discharge pressure	[0, 175]	PSI
	f f	Fuel flow	[0.0, 61.1]	GPM
	al	Alarm limit of exhaust gas temperature	[0, 1370]	• F
	tl	Trip limit of exhaust gas temperature No. 1	[0, 1500]	• F
	sp	Speed set point	[0, 6000]	RPM
	t	Turbine Speed	[1, 6000]	RPM
	Table B.2: Monitored variables in the Mitsubishi single shaft Turbo-
	Generator Model MS6001 that are related to exhaust gas temperature.

Table B .

 B 5: Summary p-values of the pairwise Friedman test with Benjamini-Hochberg correction for the test performance on the Flow problem; an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.05 significance level.

		GSGP GSGP-LS HYBRID HYBRID-LIN GP-LS
	GSGP	-	0.7150	0.0000*	0.1358	0.715
	GSGP-LS	-	-	0.0000*	0.1358	0.715
	HYBRID	-	-	-	0.1358	0.715
	HYBRID-LIN	-	-	-	-	0.715
	GP-LS	-	-	-	-	-
		GSGP GSGP-LS HYBRID HYBRID-LIN	GP-LS
	GSGP	-	0.0000*	0.0000*	0.0000*	0.0000*
	GSGP-LS	-	-	0.4652	0.0000*	0.0000*
	HYBRID	-	-	-	0.0000*	0.0000*
	HYBRID-LIN	-	-	-	-	0.0000*
	GP-LS	-	-	-	-	-
	Algorithm	Flow	Flow	Temperature Temperature
		Train	Test	Train	Test
	GSGP	0.0961 0.0952	293.75	279.36
	GSGP-LS	0.0957 0.0952	85.18	108.23
	HYBRID	0.0953 0.0945	94.41	113.41
	HYBRID-LIN	0.0963 0.0980	62.72	77.97
	GP-LS	0.0909 0.1006	12.94	12.92
	Table B.4: Summary of median performance by each algorithm on each
	problem.

Table B .

 B 6: Summary p-values of the pairwise Friedman test with Benjamini-Hochberg correction for the test performance on the Temperature problem; an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.05 significance level.

  B.6) where f T is the temperature function, θ 1 = 5.7509, θ 2 = 5.2602, θ 3 = 16.2779, θ 4 = 9.2538, θ 5 = 1.0549, θ 6 = -4.4801, θ 7 = -1.2692, θ 8 = 0.6149, θ 9 = -0.3637, θ 10 = -15.6936, θ 11 = 2.6504, θ 12 = 99.5816, θ 13 = -0.0022, θ 14 = 9.3046, θ 15 = 51.3792, θ 16 = 0.7228, θ 17 = -179.6383, θ 18 = -7.6820, θ 19 = 0.0430, θ 20 = 1.1433, θ 21 = 10.0145, θ 22 = -77.0285, θ 23 = 562.4549, θ 24 = -0.3169, θ 25 = 562.6132, θ 26 = -2.1891, θ 27 = -6.0626, θ 28 = 0.3125, θ 29 = 3.2894, θ 30 = 0.4170, θ 31 = 12.0751, θ 32 = 254.8610, θ 33 = 0.7083, θ 34 = -0.5231,

Table B .

 B 8: Summary p-values of the pairwise Friedman test with Benjamini-Hochberg correction for the test performance on the Flow problem; an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.05 significance level. tical test suggesting that there is no significant difference between their median performance. However, if we analyze the boxplots of Figure B.7 we can see that the lowest error was produced by GP-LS.

	Algorithm						Flow	Temperature
							Test		Test
	LR						0.1095		224.96
	IR						0.1122		118.16
	SR						0.0947		223.80
	MLP						0.1493		15.23
	SVM						0.0948		80.98
	FFX						0.0992		15.03
	MARS						0.1111		14.81
	GP-LS						0.1006		12.92
	Table B.7: Summary of median performance by each algorithm on each
	problem, comparing GP-LS with other machine learning methods.
		LR	IR	SR	MLP	SVM	FFX	MARS	GP-LS
	LR	-	0.0068* 0.0000* 0.0000* 0.0000* 0.0000*	0.0129	0.0035*
	IR	-	-	0.0000* 0.0001* 0.0000* 0.0000*	0.5010	0.0001*
	SR	-	-	-	0.0000*	0.1104	0.0046* 0.0000*	1.0000
	MLP	-	-	-	-	0.0000* 0.0000* 0.0004* 0.0000*
	SVM	-	-	-	-	-	0.0015* 0.0000*	1.0000
	FFX	-	-	-	-	-	-	0.0000*	0.3061
	MARS	-	-	-	-	-	-	-		0.0000*
	GP-LS	-	-	-	-	-	-	-		-

Table B .

 B 9: Summary p-values of the pairwise Friedman test with Benjamini-Hochberg correction for the test performance on the Flow problem; an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.05 significance level.

		LR	IR	SR	MLP	SVM	FFX	MARS	GP-LS
	LR	-	0.0000*	0.7700	0.0000* 0.0000* 0.0000* 0.0000* 0.0000*
	IR	-	-	0.0000* 0.0000* 0.0000* 0.0000* 0.0000* 0.0000*
	SR	-	-	-	0.0000* 0.0000* 0.0000* 0.0000* 0.0000*
	MLP	-	-	-	-	0.0000*	1.0000	1.0000	0.0346*
	SVM	-	-	-	-	-	0.0000* 0.0000* 0.0000*
	FFX	-	-	-	-	-	-	0.5210	0.0346*
	MARS	-	-	-	-	-	-	-	0.0792*
	GP-LS	-	-	-	-	-	-	-	-

Table C

 C

		.2: GP-LS Parameters
	Parameter	Value
	Population	200
	Generations	40
	Crossover operator	Standard subtree, 0.9 probability
	Mutation operator	0.05 probability
	Tree initialization	Ramped Half-and-half, max depth 6
	Function set	+,-,x,/,exp,sin,cos,log,sqrt,tan,tanh
	Terminal set	Input features, random constants
	Selection type	Tournament, size 7
	Elitism	Best individual survives
	Maximum tree depth	17
	Maximum LS iterations	500

Table C .

 C 3: Summary of performance by MPM with NL=15 for AM/AM and AM/PM, with and without white noise added at the output.In figure C.12 we can see the AM/AM results using the MPM with NL=15 and figure C.13 shows the same for AM/AM with white noise added to the output. In both cases it is noticeable that the model does not follow with high precision the real PA behavior. These result demonstrate that for complex RF-PA behaviors alternative modeling methods should be used. Figures C.14 and C.15 show similar analysis for AM/PM behavior, with similar results indicating that MPM cannot provide accurate predictions of amplifier behavior.TableC.4 summarizes the performance by GP-LS on both modelling problems, showing a similar analysis as table C.3. If we compare both tables it is clear that GP-LS achieves the best overall performance in both cases, AM/AM and AM/PM. Moreover, notice that adding white noise reduces the error, particularly for the AM/AM model. Figure C.16 depicts the GP-LS behavior for the AM/AM model using the raw output, while figure C.17 shows the behavior for AM/AM after adding white noise to the output. Figures C.18 and C.19 present a similar analysis for the AM/PM model. All four plots are in agreement with the actual PA behavior, confirming the numerical results of Table C.4.

	Performance	AM/AM	AM/AM	AM/PM	AM/PM
	Measures	with MPM	with MPM	with MPM	with MPM
		using	using	using	using
		NL=15	NL=15 and	NL=15	NL=15 and
			adding		adding
			white noise		white noise
	MSE	362.6932	342.8695	5012.4	4965.4
	MAE	15.8983	15.2569	47.1811	47.0634
	RMSE	8949.8	8606.9	2494.1	25414

Table C .

 C 4: Summary of performance by GP-LS for AM/AM and AM/PM, with and without white noise added at the output.

	Performance	AM/AM	AM/AM	AM/PM	AM/PM
	Measures	with GP-LS	with GP-LS	with GP-LS	with GP-LS
			adding		adding
			white noise		white noise
	MSE	116.1207	78.2696	936.8455	911.2172
	MAE	8.9220	7.5702	8.1459	8.9996
	RMSE	6881.9	5377.9	18824	19964

Figure C.13: RF-PA Doherty 7W@2.1 GHz AM/AM using a traditional modeling technique as MPM with NL=15 and white noise.

Table C .

 C 5: FPGA overall resources for the AM/AM and AM/PM using MPM and GP-

LS Total Logic Elements Total com- binational functions Dedicated Logic Registers Total mem- ory bits

	Resources	862/119088	509/119088	724/119088	211382/
	used by	(<1%)	(<1%)	(<1%)	3981312
	GP-LS				(1%)
	Resources	950/119088	800/119088	850/119088	212890/
	used by	(<1%)	(<1%)	(<1%)	3981312
	MPM				(1%)

  3400N instrument operated at 5 kV and the magnification is adjusted to obtain a clear image. Micrographs of activated carbon are shown in Figure D.1, showing that the external surface is fairly homogeneous with cavities of different sizes. We use breakthrough curves to better understand the behavior of activated carbon during the adsorption process

	Property	Value
	BET Surface Area	232.40 m 2 /g
	Average Diameter	0.30 nm
	Adsorbent Density	2.2460 g/cc
	Surface Atom Density	38.45e 14 molec/cm 2
	Langmuir Surface Area	1.034e 03 m 2 /g
	Surface Tension	8,85 erg/cm 2
	Critical Pressure	33.50 atm
	Table D.1: Surface properties of activated carbon calculated by t-plot
	analysis.
	S-

Table D

 D

.2: Summary of collected variables in the experimental dataset.

Table D .

 D 3: Parameters for neat-GP and GP-LS. tile, median, third quartile and outliers) over all 30 runs.

	Parameter	neat-GP	GP-LS
	Population size	500	250
	Initialization	Ramped Half & Half	Ramped Half & Half
	Max.	initial	3 levels	6 levels
	depth
	Crossover rate	0.7	0.9
	Mutation rate	0.3	0.1
	Function set	+,-,×,÷,sin,exp,cos,log,sqrt,tan,tanh +,-,×,÷,sin,exp,cos,log,sqrt,tan,tanh
	Terminal set	Input features	Input features
			& random constants	& random constants
	Selection		Tournament of size 6	Tournament of size 7
	Elitism		Best individual survives	Best individual survives
	Maximum tree	none	17
	depth

Table D .

 D 4: Summary of MSE by each algorithm on the training and testing sets, showing the median, first and third quartiles.

	Algorithm	Train	Train	Train	Test	Test	Test
		Median	1st Quartile	3rd Quartile	Median	1st Quartile	3rd Quartile
	Linear	870.95	857.55	884.87	906.14	860.23	946.75
	Robust	977.975	958.02	994.55	908.08	863.29	953.21
	Quadratic	489.795	476.48	496.22	497.17	478.96	537.63
	neat-GP	614.64	523.46	921.64	612.89	513.28	938.92
	FFX	191.26	184.35	194.30	199.81	187.22	207.45
	MARS	82.3865	79.72	86.05	92.53	85.83	101.86
	GP-LS	89.915	52.57	122.78	94.14	65.44	122.11

  E.2). Three different pages are exposed. Summary, where data is displayed in a spreadsheet format; the user can edit or explore data value statistics. Metrics, with 264 E.2 The Graphical User Interface

https://project.inria.fr/fraclab/

Some GP systems do include numerical terminals or random ephemeral constants, while these constants can act as coefficients in an evolved expressions, they are not subject to optimization after they are introduced into the syntax of a particular individual.

http://gplab.sourceforge.net/

http://gplab.sourceforge.net/

http://www.natus.com/

https://github.com/jrderuiter/randomforest-matlab

7.2 Future work

In this work, the GNU Scientific Library is used http://www.gnu.org/software/gsl/

http://www.cs.rtu.lv/jekabsons/regression.html

http://trent.st/ffx/

http://www.cs.rtu.lv/jekabsons/regression.html http://trent.st/ffx/ http://www.tree-lab.org/index.php/resources-2/downloads/open-source-tools

https://github.com/chgagne/beagle

https://code.google.com/archive/p/beagle/wikis/Puppy.wiki

http://www.evolved-analytics.com/ http://dev.heuristiclab.com/ http://www.nutonian.com/products/eureqa/

E.2 The Graphical User Interface

https://github.com/emigdioz/LeafGP

Some examples of GP applications with a graphical interface are: • Discipulus 12 -A commercial product that is capable of building predictive models using GP. https://github.com/deap/deap http://dgpf.sourceforge.net/ http://cs.gmu.edu/ eclab/projects/ecj/ http://eodev.sourceforge.net/ http://www.epochx.org/ http://gplab.sourceforge.net/ http://ugp3.sourceforge.net/ http://faculty.hampshire.edu/lspector/push.html http://gpocl.sourceforge.net http://www.rmltech.com/

E X P E R I M E N TAT I O N A N D R E S U LT S

In this section we present experimental details regarding the proposed system. The reported algorithms were implemented in (MAT-LAB, 2015). Specifically for the + FEGP algorithm the GPLAB 2 [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF] toolbox was used.

Given the stochastic nature of GP, a series of multiple runs were executed in order to statistically determine the algorithm performance. Following the LOOCV scheme for data partition and validation, the experiments involve 30 independent runs per each fold. Results presented in this work contain 390 runs in total (13 × 30). Details of the values used for filtering, CSP and LDA were mainly discussed in Section 6.2. Furthermore, + FEGP parameters are summarized in Table 6.1. Some comments in the third column are derived from experimental tuning of the algorithm.

The training classification performance of the proposed enhanced system is shown in Fig. 6.19, with the overlapped performance of the reference system for comparison. Predictive performance calculated over the testing partitions is presented in Fig. 6.20. Basic statistical results for the + FEGP based system per LOOCV fold are given in Fig. 6.21. To validate our results, non-parametric Friedman and Wilcoxon rank sum tests were used to calculate pairwise statistical differences at a LOOCV fold level. The results are presented in Table 6.2 and Table 6.3 corresponding to the training and testing phases, with p-values corrected by applying the Benjamini-Hochberg correction for multi-group comparisons.

+ FEGP fitness performance and LDA classification accuracy during the training phase are shown in Fig. 6.22. Although the + FEGP algorithm only uses the fitness measure (bottom plot), the top plot shows the convergence of the actual classification accuracy achieved by LDA across the generations. Please note that the bottom convergence plot condenses all 390 runs by first computing the median of 30 runs per fold and finally calculating the mean of all fold results. Similarly, in Fig. 6.23,

Based on Trujillo, L., Z-Flores, E., Ju, P., Legrand, P., Castelli, M., [START_REF] Trujillo | Local Search is Underused in Genetic Programming[END_REF]. Local Search is Underused in Genetic Programming. In Genetic Programming Theory and Practice XIV, number January

I n this appendix we present additional experiments for the integration of Local Search into GP for regression real-world problems. In chapter 4 we showed a study for different strategies on LS selection and we tested in problems where the data was generated by pre-defined functions, the so called benchmark problems. However, in this chapter we proceed to follow the strategies in chapter 5, that is a linear subtree is added to the root node for any selected individual to be optimized and the LS selection in the population is given by a tree size related heuristic. In the following sections we discuss more detail in the implementation, experimentation and results.

A.2 P R O P O S A L

First, as suggested in [START_REF] Kommenda | Effects of constant optimization by nonlinear least squares minimization in symbolic regression[END_REF], for each individual K in the population we add a small linear uppertree above the root node, such that

Additional regression experiments using GP with LS K ′ represents the new program output, while θ 1 , θ 2 ∈ R are the first two parameters from θ. Second, for all the other nodes n k in the tree K we add a weight coefficient θ k ∈ R, such that each node is now defined by

where n ′ k is the new modified node, k ∈ {3, ..., r + 2} and r is the size of tree K. Notice that each node has an unique parameter that can be modified to help meet the overall optimization criteria of the non-linear expression. At the beginning of the GP run each parameter is initialized by θ k = 1. During the GP syntax search, subtrees belonging to different individuals are swapped, added or removed together with their corresponding parameters, often called Lamarckian inheritance [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF]. We consider each tree as a non-linear expression and the local search operator must now find the best fit parameters of the model K ′ . The problem could be solved using a variety of techniques, but following [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF] we use a trust region algorithm.

Finally, it is important to consider that the LS could increase the computational cost of the search, particularly when individual trees are very large. While applying the LS strategy to all trees might produce good results [START_REF] Z-Flores | Evaluating the Effects of Local Search in Genetic Programming. In EVOLVE -A Bridge between Probability[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF], it is preferable to reduce the amount of trees to which it is applied. Therefore, we use the heuristic proposed in [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF], where the LS is applied stochastically based on a probability p(s) determined by the tree size s and the average size of the population s (details in [START_REF] Azad | A Simple Approach to Lifetime Learning in Genetic Programming-Based Symbolic Regression[END_REF][START_REF] Z-Flores | A Local Search Approach to Genetic Programming for Binary Classification[END_REF]). In this way, smaller trees are more likely to be optimized than larger trees, which reduces the computational cost and improves the convergence of the optimizer by keeping the parameter vectors relatively small. Hereafter, we refer to this version of GP as GP-LS.

A.3 E X P E R I M E N T S A N D R E S U LT S

We evaluate this proposal on a real-world symbolic regression task, the Yacht problem that has 6 features and 308 input/output samples [START_REF] Ortigosa | A neural networks approach to residuary resistance of sailing yachts prediction[END_REF]. The experiments are carried out using a modified

B.4 E X P E R I M E N T S

This work focuses on applying GP to model a GT unit using a datadriven approach following a supervised learning formulation. Therefore, we will use the datasets described in Section B.2 to pose two symbolic regression problems and solve them with the variants of GP described in Section B.3. Hereafter, we will refer to the problem of modeling the fuel flow of the GT unit as the Fuel problem, and the problem of modeling the exhaust gas temperature as the Temperature problem.

Five different algorithms are compared based on the GP systems described in the previous section; these are:

1. The standard GSGP algorithm as described in [START_REF] Moraglio | Geometric semantic genetic programming[END_REF] and implemented in (Castelli et al., 2015a);

2. The GSGP-LS algorithm proposed in (Castelli et al., 2015e) and described in the previous section;

3. An algorithm we refer to as HYBRID, that combines GSGP and GSGP-LS, by applying GSGP-LS for the first l generations and then switches to GSGP, as suggested in (Castelli et al., 2015e); 4. HYBRID-LIN is the same as HYBRID, but the training data is preprocessed using the linear scaling approach of [START_REF] Keijzer | Improving symbolic regression with interval arithmetic and linear scaling[END_REF].

5. GP-LS is the GP algorithm with LS integrated.

All GSGP variants are implemented using the GSGP-C++ library (Castelli et al., 2015a), while GP-LS is implemented using Matlab and the GPLab toolbox [START_REF] Silva | Gplab-a genetic programming toolbox for matlab[END_REF].

B.4.1 Experimental Setup

All algorithms were used to solve both the Flow and Temperature problems, performing 30 independent runs of each. In each case, the problem data was split into a training and testing set, where the former contains 70% of the data samples and the latter the remaining 30%. The This work will focus on applying MPM and GP-LS to model the behavior of a real RF-PA. Therefore, we will use the datasets based on the table C.1 that describes the behavior of a Doherty PA. The figures C.4 and C.5 describe the AM/AM and AM/PM conversion curves, the AM/AM data was generated by plotting output power versus input power, and AM/PM data by plotting the phase differences between input and output data versus input power. However, the data will be implemented within a FPGA development board, so they must be modified to use an amplitude reference as volts. The experiments will compare MPM with a high nonlinearity order of NL=15 with the best models produced by GP-LS. In this way comparing a common modeling tool, with an approach based on GP, a paradigm that has been used only sparingly in this domain. The data is introduced to the MPM and GP-LS methods. Afterward, the models are used to predict the PA behavior using two different schemes. First, using the raw output of the models; in this case the models will try to fit all of the underlying behavior of the system. We also use a second approach, where we combine the output of the generated models with a high frequency signal. This second method allows the models to focus on the large-scale and low frequency behavior of the system, with high-frequency details assumed to be stationary. While this can be achieved in different ways, in this work we use a white noise signal. Future work will study the effects of using other noise models.

C.5.1 Experimental Setup

Table C.1 shows the RF-PA Doherty 7W @2.1 GHz specifications used for this work. This RF-PA has maintains a constant gain for low level input signals. However, at higher input power, the PA goes into algorithm. The resulting p-values are given in Table D.5. In both tables an asterisk (*) indicates that the null hypothesis can be rejected at the α = 0.01 significance level. These results allow us to make some important conclusions. First, that the difference between MARS and GP-LS is not statistically significant in terms of median performance. Second, that the difference between both of these algorithms and the rest of the methods is statistically significant. Finally, neat-GP is clearly outperformed by FFX, MARS and GP-LS.

To provide a deeper analysis of the performance of each method, we analyze the coefficient of determination R 2 of a linear fit between the expected output and the estimated output of each method. D.6 presents a numerical comparison based on R 2 , showing the median, first and second quartiles over all runs. These results are consistent with what was shown above, with MARS and GP-LS substantially outperforming all other methods, and neat-GP performing worse. The statistical significance of these results is tested using the same procedure with the corresponding p-values reported in Table D.7. These results lead to the same conclusions discussed above It is also instructive to compare the methods based on the size of the models found. In this comparison we exclude the more basic regression methods (Linear, Quadratic and Robust) since their size is completely 2. Kernel execution: for any non-trivial problem, this is the most demanding phase. Here, the entire recently transferred population is evaluated -by interpreting each program over each fitness case -on the compute device. Fortunately, this step can be done both in parallel as well accelerated by GPUs.

3. Error retrieval: after being computed and accumulated in the previous step, the prediction errors of the whole population need to be transferred to the host so that this information is available to the evolutionary process. Again, if the compute devices and the host are not the same, an explicit transfer is necessary.

E.2.1.4 GP program interpreter

The GP tree interpreter is a function that takes a GP program and its inputs, and then executes the instructions stored on the tree's nodes in a flow given by the tree's structure. An interpreter is commonly implemented as a stack-based procedure in which the partial results coming from the children nodes are pushed into a stack and pulled back when a parent operator needs the accumulated result.

A pseudo-code for a GP tree interpreter is presented by the function Interpreter (program, n), described in Algorithm E.2.1. The argument program is an array of integers encoding a tree (the structure, operators, operands and variables) via a prefix linear representation, whereas n is the index of the training data point currently being evaluated. The function INDEX() extracts the operator/operand code, VALUE() returns the actual numerical value of a constant or the index if the argument is a variable, X n is the n-th row of the training dataset, PUSH() puts a number into the stack and, finally, POP pulls back a value from the top of the stack. At the end of the interpretation process, the program's predicted output is left on the top of the stack, so this value is returned by the interpreter and then used to compute the program's prediction error.

LeafGP extends the arithmetic functions shown in Algorithm E.2.1 with a collection of conditional, logical, relational and trigonometric operators.

Algorithm E.2.1 Function