Junshi Xia

Docteur De L'universit

Présentée Par

Thèse Dirigée Par

Jocelyn Chanussot

Thèse Soutenue

M Grégoire

Mercier Professeur

M Jón

Professeur Atli Benediktsson

M Farid

Melgani Professeur

Mireille Mme

Guillaume

M Peijun Du

M Jocelyn

Chanussot Professeur

Multiple classifier systems for the classification of hyperspectral data

Keywords: multiple classifier systems, hyperspectral data, classification, spectralspatial classifiers, Rotation-based ensemble, Rotation Forest, support vector machines, manifold learning, Markov Random Field, Random subspace, extended multi-attribute profiles Asphalt, meadows, gravel, trees, metal sheets, bare soil, bitumen, bricks and shadow. . . . Classification results of Indian Pines AVIRIS image. (a) RoF-PCA. (b)

In this thesis, we propose several new techniques for the classification of hyperspectral remote sensing images based on multiple classifier systems (MCS). Our proposed framework introduces significant innovations with regards to previous approaches in the same field, many of which are mainly based on an individual algorithm. First, we proposed to use Rotation Forests with several linear feature extraction and compared them with the traditional ensemble approaches, such as Bagging, Boosting, Random subspace and Random Forest. Second, the integration of the support vector machines (SVM) with Rotation subspace framework for context classification is investigated. SVM and Rotation subspace are two powerful tools for high-dimensional data classification. Combining them can further improve the classification performance. Third, we extended the work of Rotation Forests by incorporating local feature extraction technique and spatial contextual information with Markov random fields (MRF) to design robust spatial-spectral methods. Finally, we presented a new general framework, Random subspace ensemble, to train series of effective classifiers, including decision trees (DT) and extreme learning machine (ELM), with extended multi-attribute profiles (EMAPs) for classifying hyperspectral data. Six RS ensemble methods, including Random subspace with DT (RSDT), Random Forest (RF), Rotation Forest (RoF), Rotation Random Forest (RoRF), RS with ELM (RSELM) and Rotation subspace with ELM (RoELM), are constructed by the multiple base learners. The effectiveness of the proposed techniques is illustrated by comparing with state-of the-art methods using real hyperspectral data sets with different contexts.

First and foremost I want to thank my advisor, Prof. Jocelyn Chanussot, for the patient guidance, encouragement and advice he has provided during the past three years. It has been an honor to be his first Chinese Ph.D student. I have been extremely lucky to have a supervisor who cared so much about my work, and who responded to my questions and queries so promptly. Again, I appreciate all his contributions of time, ideas, and funding to make my Ph.D experience productive and stimulating.

I must express my gratitude to my home supervisor Prof. Peijun Du, for his continued support and encouragement. He led me into the fields of remote sensing and machine learning. Without his help, I would not make such great progress.

I would also like to thank Grégoire, Jón Atli, Farid and Mireille for serving as my committee. I also want to thank you for providing brilliant comments and suggestions, thanks to you.

Completing this manuscript would have been all the more difficult without the support and friendship provided by the other members from the GIPSA-lab in Grenoble. In particular I would like to thank Prof. Mauro Mura Dalla for the suggestions he made in Chapter 6 of this work. I would like to thank Dr. Zhongyang Li for the arrangement of my documents and housing when I first came to Grenoble. In the past three years, whenever I had a problem, he always did his best to help me. I would like to thank Dr. Guanghang Song for providing her MacBook when my notebook was broken.

Those people, such as the teachers and the postgraduates from the School of Environmental Science and Spatial Informatics of China University of Mining and Technology, also deserve thanks for making my stay in Xuzhou that much more pleasurable.

I would like to thank my dear friends, also my teachers, Kai Wang, Xiyan He and their lovely daughter: Emma. I appreciate Kai for the proof reading of my book chapter and Xiyan for providing me ideas during my study in Grenoble.

I would like to thank my parents, my girlfriend: Miss. Zuo Zhang, for their constant love, understanding and encouraging.

Finally, this thesis is dedicated to my grandma in heaven, Mrs. Fengying Sun, a goodnatured and kindly woman who left me when I was writing this manuscript.

iii List of Tables

Chapter 1 Introduction

This chapter introduces the problems addressed in this thesis and gives an overview of subsequent chapters. Section 1.1 gives a general description of hyperspectral remote sensing. In Section 1.2, we introduce the pixel-wise and spectral-spatial classifiers of hyperspectral data. The issues in hyperspectral image classification are also described. Section 1.3 gives the objective of this thesis, and introduces multiple classifier systems (MCS) and highlights its distinct advantages compared to the individual classifier. Section 1.4 concludes this chapter with an overview of the subjects addressed in each subsequent chapter. Section 1.5 summarizes the significant contributions of the thesis.

Hyperspectral remote sensing

Hyperspectral remote sensing (HRS) is a technology that can provide detailed spectral information from each pixel in an image. The development of IS started in the 1980's by NASA's Jet Propulsion Laboratory (JPL) with the production of Airborne Imaging Spectrometer (AIS) [START_REF] Vane | Airborne imaging spectrometer : A new tool for remote sensing[END_REF]. The success of AIS leads to the development of new optical instrument, namely Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) [4]. Hyperspectral imaging is related to multi-spectral imaging. Fig. 1.1 illustrates the hyperspectral imaging concept in comparison of multi-spectral imaging concept. As can be seen from this Figure, multispectral imaging deals with several images at discrete, referring to narrow bands, which covers the spectrum from the visible to the long-wave infrared. On the contrary, hyperspectral images deal with narrow spectral bands at 10nm to 20nm intervals over a continuous spectral range(VIS, NIR, SWIR, MWIR and LWIR) and produces the spectrum of all pixels in the scene. Table 1.1 lists the airborne and satellite hyperspectral sensors [4,[START_REF] Staenz | Overview of terrestrial imaging spectroscopy missions[END_REF]. Typical hyperspectral sensors contain 64-250 spectral bands which cover a spectral range of 400nm to 2500nm, with a spatial resolution varying from 1-20 m/pixel and 14.5-100 m/pixel for airborne and satellite sensors, respectively.

The main advantage to hyperspectral imaging is that, because an entire spectrum is acquired at each point, the operator needs no prior knowledge of the sample, and postprocessing allows all available information from the dataset to be mined [START_REF] Chang | Hyperspectral Imaging : Techniques for Spectral Detection and Classification[END_REF]. Hyperspectral imaging can also take advantage of the spatial relationships among the different spectra in a neighborhood, allowing more elaborate spectral-spatial models for a more accurate segmentation and classification of the image [START_REF] Chang | Hyperspectral Imaging : Techniques for Spectral Detection and Classification[END_REF].

The primary disadvantages are cost and complexity. Fast computers and large data storage capacities are needed for analyzing hyperspectral data. Significant data storage capacity is necessary since hyperspectral cubes are large, hyperdimensional datasets, potentially exceeding hundreds of megabytes. All of these factors greatly increase the cost of acquiring and processing hyperspectral data [START_REF] Chang | Hyperspectral Imaging : Techniques for Spectral Detection and Classification[END_REF]. Hyperspectral remote sensing is used in a wide array of applications with various techniques, such as classification, unmixing, target detection etc. These application domains include [START_REF] Landgrebe | Multispectral land sensing ; where from, where to ?[END_REF][START_REF] Kalacska | Hyperspectral Remote Sensing of Tropical and Sub-Tropical Forests[END_REF]:

• Agriculture. Typical applications for agriculture using Hyperspectral data are to monitor the growth and health of crops, to detect the nutrient and water status of wheat in irrigated systems, to detect the animal proteins in compound feeds to avoid mad-cow disease.

• Mineralogy. Many minerals can be identified from hyperspectral remote sensing images and their relation to the presence of valuable mineral (gold and diamond) is understood. The current advance is to find the relationship between oil and gas leakages from pipelines and natural wells, and their effects on the vegetation and the spectral signatures.

• Surveillance. Hyperspectral surveillance is the implementation of hyperspectral scanning technology for surveillance purposes. Hyperspectral imaging is particularly useful in military surveillance, in which people have learned not only to hide from the naked eye, but also to mask their thermal signatures to blend into the surroundings and avoid infrared scanning.

• Environment. Hyperspectral can provide the precision classification results for the support of continuous monitoring of emissions produced by coal and oil-fired power plants, municipal and hazardous waste incinerators, cement plants, as well as many other types of industrial sources.

Hyperspectral image classification

In machine learning and statistics, classification is the problem of identifying to which of a set of categories a new observation belongs [START_REF] Michie | Machine Learning, Neural and Statistical Classification[END_REF]. Traditional classification algorithms can be categorized into two folds: unsupervised and supervised. Unsupervised classifiers, often called clustering, simply give the output from the incoming data and no training is needed. Unsupervised algorithms are generally very fast and easy to implement, while their accuracy is limited due to the lack of prior information with respect to the given training samples. The spectral classes derived from clustering are not information classes that users are interested in. The task of supervised classification is to use the available training samples to establish a model that can be used to predict the label of unseen data.

In this thesis, we focus on supervised classification. Supervised classification used in hyperspectral data can be viewed as identification of objects in a scene. Pixels in hyperspectral without or with considering the correction between spatially neighborhood pixels are often called spectral-based [START_REF] Landgrebe | Signal Theory Methods in Multispectral Remote Sensing[END_REF] and spectralspatial classification [START_REF] Fauvel | Advances in spectral-spatial classification of hyperspectral images[END_REF], respectively.

Spectral-based classification

Let X train , Y train = {x i , y i } n i=1 be of n training samples. X = x i ∈ R D , i = 1, 2, .., n , where D is the dimension of hyperspectral data. y i ∈ {1, 2, ..., C}, C is the total number of classes. The aim of classification is to develop a model which can find a relationship between input samples X train and the target Y train and use the established model to assign new pixels to one of the C class of interest.

When we apply hyperspectral data for supervised classification, the first and most important drawback, which limits the possibility to apply traditional techniques for the classification of multispectral images is the curse of dimensionality (also referred to Hughes phenomenon) [START_REF] Hughes | On the mean accuracy of statistical pattern recognizers[END_REF]. In Hughes phenomenon, classification accuracy increases gradually in the beginning as the number of spectral bands or dimensions increases, but decreases dramatically when the band number reaches some value. The features of hyperspectral data are usually redundant and strongly correlated. In order to alleviate Hughes phenomenon and extract uncorrelated features, one common strategy is to use feature selection/extraction techniques.

Feature selection returns a subset of the features, whereas feature extraction creates new features from functions of the original features [START_REF] Guyon | Feature Extraction : Foundations and Applications[END_REF]. A feature selection algorithm can be seen as the combination of a search technique for extracting new feature subsets, along with an evaluation measure scores the different feature subsets. Feature selection algorithms could be often categorized into three parts: wrappers, filters and embedded methods [START_REF] Serpico | A new search algorithm for feature selection in hyperspectral remote sensing images[END_REF][START_REF] Pal | Feature selection for classification of hyperspectral data by SVM[END_REF].

Feature extraction can be conducted in a supervised or unsupervised manner, in terms of whether the label information is utilized to guide the extraction of relevant features [START_REF] Guyon | Feature Extraction : Foundations and Applications[END_REF][START_REF] Mojaradi | Dimensionality reduction of hyperspectral data via spectral feature extraction[END_REF]. The most popular unsupervised algorithms are principal component analysis (PCA) [START_REF] Richards | Remote sensing digitial image analysis[END_REF], Independent Component Analysis (ICA) [START_REF] Wang | Independent component analysis-based dimensionality reduction with applications in hyperspectral image analysis[END_REF][START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF], Maximum Noise Fraction (MNF) [START_REF] Green | A transformation for ordering multispectral data in terms of image quality with implications for noise removal[END_REF]; supervised ones contain Linear Discriminant Analysis (LDA) [START_REF] Bandos | Classification of hyperspectral images with regularized linear discriminant analysis[END_REF], Local Fisher Discriminant Analysis (LFDA) [START_REF] Sugiyama | Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis[END_REF], Decision Boundary Feature extraction (DBFE) [START_REF] Landgrebe | Signal Theory Methods in Multispectral Remote Sensing[END_REF] and Non-parametric Weighted Feature extraction (NWFE) [START_REF] Kuo | Nonparametric weighted feature extraction for classification[END_REF]. Generally, supervised methods require a large amount of labeled training data. However, it could fail to identify the relevant features that are discriminative to different classes, provided the number of labeled samples is small. Unsupervised methods ignore the label information and therefore are often unable to identify the discriminative features. Given the high cost in manually labeling data, and at the same time abundant unlabeled data is often easily accessible, it is desirable to develop semi-supervised methods that are capable of exploiting both labeled and unlabeled data [START_REF] Song | A unified framework for semi-supervised dimensionality reduction[END_REF]. Several semi-supervised feature extraction methods were proposed over the last years, such as Semi-supervised discriminant analysis (SDA) [START_REF] Cai | Semi-supervised discriminant analysis[END_REF], Semi-supervised Local Fisher Discriminant Analysis (SELF) [START_REF] Sugiyama | Semi-supervised local fisher discriminant analysis for dimensionality reduction[END_REF].

When the dimensionality of hyperspectral data is reduced by using feature selection/extraction techniques, any classifiers (such as Gaussian maximum likelihood, minimum distance classifier, etc.) suitable for multispectral images can be used. Here, particular attention has been dedicated to support vector machines (SVMs), which is the most widely used classifier for hyperspectral data in recent years [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF][START_REF] Huang | An assessment of support vector machines for land cover classification[END_REF][START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF].

SVMs construct a hyperplane or set of hyperplanes in a high-or infinite-dimensional space, which can be used for classification, regression, or other tasks [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. Intuitively, a good separation is achieved by the hyperplane that has the largest distance to the nearest training data point of any class (so-called functional margin), since in general the larger the margin the lower the generalization error of the classifier. Whereas the original problem may be stated in a finite dimensional space, it often happens that the sets to discriminate are not linearly separable in that space. For this reason, it was proposed that the original finite-dimensional space be projected into a much higher-dimensional space, presumably making the separation easier in that space. To keep the computational load reasonable, the mappings used by SVM schemes are designed to ensure that dot products may be computed easily in terms of the variables in the original space, by defining them in terms of a kernel function selected to suit the problem. More details about SVM and its applications can be detailed in [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF][START_REF] Huang | An assessment of support vector machines for land cover classification[END_REF][START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF][START_REF] Mountrakis | Support vector machines in remote sensing : A review[END_REF].

Spectral-spatial classification

In the above subsection, spectral-based classifiers only deal with the spectral information, in which each pixel is often viewed as a vector of discrete variables. In fact, neighboring pixels have strong relationships in hyperspectral image, because homogeneous structures occurred in the image scene are generally large compared to the size of pixel and image sensors often acquire significant amount of energy from adjacent pixels [START_REF] Richards | Remote sensing digitial image analysis[END_REF]. Thus, considering the spatial contextual information in the classification of hyperspectral data can significantly improve the classification performance. If the spatial information of hyperspectral image (especially for the high spatial resolution) is not considered, the thematic map, which includes salt and pepper classification noise, looks very noisy. Accordingly, it is essential to take into account spatial information.

The pioneer work may be conducted by Landgrebe and his research team. They proposed the well-known ECHO (Extraction and Classification of Homogeneous Objects) classifier, which is recognized as the standard spatial-spectral classifier in the remote sensing community [START_REF] Kettig | Classification of multispectral image data by extraction andclassification of homogeneous objects[END_REF]. This approach is to group the pixels into an object based on region growing and then use the means and covariances of the group pixels as the input features of a maximum likelihood (ML) classifier. Since the covariance matrices are involved in the ECHO classifier, a feature extraction/ dimensionality reduction step is recommended to pre-process the hyperspectral data before applying this technique.

After that, many researchers have developed various techniques for the spatial-spectral classification of hyperspetral data. A good survey of these approaches can be found in [START_REF] Fauvel | Advances in spectral-spatial classification of hyperspectral images[END_REF]. To the best of our knowledge, Spectral-spatial classification algorithms can be divided into several groups, which are detailed in Table A.1.

Issues in hyperspectral image classification.

In summary, hyperspectral data can be viewed as a very important source to generate accurate classification maps, thanks to the continuous spectral information which is benefit to distinguish very similar classes. However, several challenges should be taken into account:

• How to avoid problems related to the ratio between number of training samples and dimensionality of data.

• How to design proper contextual information to improve the classification performance.

• How to design the optimal or sub-optimal classifier for a given task.

The first challenge sometimes can be alleviated by feature selection/extraction techniques. However, the dimensionality after applying feature selection/extraction technique is hard to determine. When we design proper contextual information for the classification of hyperspectral data, a pattern classification approach must be commonly chosen. The most popular used method in the spatial-spectral classification is the support vector machines (SVM). These approaches shows various performance in different applications. How to design the optimal or sub-optimal classifier for a given is an open question. To alleviate these challenges, we proposed to use multiple classifier systems, which can provide the complementary information of the pattern classifiers and integrate the outputs of these pattern classifiers according to a certain combination approaches.

Methods

Descriptions

Reference

Composite kernels

Multiple kernels, which represent spectral and spatial information, are combined. [START_REF] Camps-Valls | Composite kernels for hyperspectral image classification[END_REF][START_REF] Camps-Valls | Spatio-spectral remote sensing image classification with graph kernels[END_REF] Markov random field MRF uses a probabilistic model to integrate spatial in-formation. [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF][START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF] Segmentation 1) Segmentation and classification are performed to get various regions and pixel-wise classification result, re-spectively. Then, the most frequent class in a region is assigned as the final class. [START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF][START_REF] Tarabalka | Segmentation and classification of hyperspectral images using watershed transformation[END_REF] 2) Segmentation is performed to get multiple regions, and the features obtained from each region (object) is treated as the input of the supervised classifier. [START_REF] Huang | An adaptive mean-shift analysis approach for object extraction and classification from urban hyperspectral imagery[END_REF][START_REF] Van Der Linden | Classifying segmented hyperspectral data from a heterogeneous urban environment using support vector machines[END_REF] Mathematical morphology 1) Results of morphological operators over features from original images or calculated by feature selec-tion/extraction are treated as the additional inputs of supervised classifiers.

[ [START_REF] Benediktsson | Classification and feature extraction for remote sensing images from urban areas based on morphological transformations[END_REF][START_REF] Benediktsson | Classification of hyperspectral data from urban areas based on extended morphological profiles[END_REF][START_REF] Fauvel | Spectral and spatial classification of hyperspectral data using svms and morphological profiles[END_REF] 2) Multiple classification results produced by spectral information and morphological features are combined to generate a final classification map. [START_REF] Dalla Mura | Classification of hyperspectral images by using extended morphological attribute profiles and independent component analysis[END_REF] Other methods Tensor modeling, context-based classification, etc. [START_REF] Velasco-Forero | Classification of hyperspectral images by tensor modeling and additive morphological decomposition[END_REF][START_REF] Lin | Hyperspectral image processing by jointly filtering wavelet component tensor[END_REF] • Spatial information. A spatial contextual information is considered based on the spatial structures observed in a scene.

Particular attention is paid to the possibility of exploiting MCS for improving the quality of classification maps, due to little investigation done in the literature on this topic [START_REF] Benediktsson | Multiple classifier systems in remote sensing : from basics to recent developments[END_REF][START_REF] Du | Multiple classifier system for remote sensing image classification : A review[END_REF]. An important point of this work is the attempt to obtain results which could be general as possible, thus not be scenario-dependent. In order to achieve such a result, a large number of hyperspectral datasets are analyzed, provided by three different sensors, covering university area, urban area and agricultural zones.

Multiple classifier systems (MCS) is using a set of learning machines to learn partial solutions for a classification problem and then integrating these solutions in some manner to construct a final or complete solution to the original problem [START_REF] Kuncheva | Combining Pattern Classifiers : Methods and Algorithms[END_REF][START_REF] Brown | Ensemble learning[END_REF]. MCS has fast been gaining popularity among researchers for their ability to fuse together multiple classification outputs for better accuracy and classification.

In MCS, we refer to an individual learning machine as the base learners. Based on the advantages of ensemble methods and increasing complexity of real-world problems, MCS is one of the important problem-solving technique. Since the last decade, there have been much literature published on MCS approaches.

The simplicity and effectiveness of MCS take the role of key selling point in the current machine learning community. Successful applications of MCS have been reported in various fields, for instance in the context of face recognition [START_REF] Czyz | Multiple classifier combination for face-based identity verification[END_REF], image analysis [START_REF] Smits | Multiple classifier systems for supervised remote sensing image classification based on dynamic classifier selection[END_REF] and handwritten digit recognition [START_REF] Kuncheva | Combining Pattern Classifiers : Methods and Algorithms[END_REF].

MCS can take advantage of the strengths of each method, while avoiding its weaknesses. Furthermore, there are other motivations to combine several individual classifiers:

• To avoid the choice of some arbitrary but important initial condition, e.g. those involving the parameters of the individual classifiers.

• To introduce some randomness to the training process in order to obtain different alternatives that can be combined to improve the results obtained by the individual classifiers.

• To use complementary classification methods to improve dynamic adaption and flexibility.

Outline of the thesis

The remainder of this thesis consists of two main parts. A chapter in which we introduce and review the background and related work on MCS is followed by the two main parts of the thesis. The first part attempts to develop spectral-based classification methods by solving the curse of dimensionality problem based on MCS (in Chapter 3 and 4). The second part strives to develop the MCS classifier with spatial information for classifying hyperspectral data (in Chapter 5 and 6). Below, we briefly introduce the contents of each of the chapters of the thesis.

• Chapter 1 presents the research issues in hyperspectral remote sensing image classification and research methodology of the thesis.

• Chapter 2 is devoted to introduce and review the background and related works on multiple classifier system, especially on hyperspectral data. In particular, the important aspects for MCS, including the topology, classifier generation and classifier combination, are introduced. The literature of MCS on hyperspectral image classification is also reported. • Chapter 7 concludes the thesis and provides several directions of future research.

Contributions

In summary, the main contribution of the thesis are:

• The complete review for MCS and related applications of hyperspectral data is given.

• We proposed several innovation Rotation-based classifiers, including Rotation Forests and Rotation ELM, which can gain high classification performance with low computational complexity for both spectral and spatial information.

• The proposed Rotation-based SVM can significantly improve the classification results using limited training samples when compared to Random subspace SVM.

• The investigation of advanced methods making use of both spectral and spatial information, showing results in terms of accuracy comparable or better than the state-of-the-art approaches proposed in the literature.

Chapter 2

Multiple classifier systems

Abstract: Over the last two decades, multiple classifier systems (MCS) has shown great potential to improve the accuracy and reliability of remote sensing image classification.

Although there are lots of literature covering the MCS approaches, there is a lack of a comprehensive literature review which presents an overall architecture of the basic principles and trends behind the design of remote sensing classifier ensemble. Therefore, in order to give a reference point for MCS approaches, this chapter attempts to explicitly review the remote sensing implementations of MCS.

Introduction

Different classifiers, such as parametric classifiers and non-parametric classifiers, have their own strengths and limitations. The famous 'no free lunch' theorem stated by Wolpert may be extrapolated to the point of saying that there is no single computational view that solves all problems [START_REF] Wolpert | No free lunch theorems for optimization[END_REF]. In the remote sensing community, Giacinto compared the performances of different classification approaches in various applications and found that no one could always gain the best result [START_REF] Giacinto | Comparison and combination of statistical and neural network algorithms for remote-sensing image classification[END_REF]. In order to alleviate this problem, MCS can provide the complementary information of the pattern classifiers and integrate the outputs of these pattern classifiers so as to make best use of the advantages and bypass the disadvantages of these pattern classifiers. Nowadays MCS is highlighted by review articles as a hot topic and promising trend in remote sensing image classification and change detection.

The reviews include the articles by Benediktsson et al. [START_REF] Benediktsson | Multiple classifier systems in remote sensing : from basics to recent developments[END_REF] and Du et al. [START_REF] Du | Multiple classifier system for remote sensing image classification : A review[END_REF]. Most of MCS approaches focus on integrating the supervised classifiers. Few works devote to combine unsupervised classification, often called cluster ensemble [START_REF] He | A cluster ensemble method for clustering categorical data[END_REF][START_REF] Iam-On | A link-based approach to the cluster ensemble problem[END_REF]. Gao et al.

proposed an interesting work to combine multiple supervised and unsupervised models using graph-based consensus maximization [START_REF] Gao | A graph-based consensus maximization approach for combining multiple supervised and unsupervised models[END_REF]. Unsupervised models, such as clustering, do not directly generate label prediction for each individual classifier, they provide useful constraints for the joint prediction of a set of related object. Thus, they proposed to consolidate a classification solution by maximizing the consensus among both supervised predictions and unsupervised constraints based on the optimization problem on a bipartite graph. Experimental results on three real applications demonstrate the benefits of the proposed method over existing alternatives. In this chapter, we focus on the combination of supervised classifiers. In addition to MCS in the domain of classification, some concepts are similar to MCS, including multiple classifier combination, mixture of experts, classifier ensemble, decision level combination, ensemble learning, decision fusion etc. The main issues of MCS design are [START_REF] Kuncheva | Combining Pattern Classifiers : Methods and Algorithms[END_REF][START_REF] Rokach | Pattern classification using ensemble methods[END_REF][START_REF] Wozniak | A survey of multiple classifier systems as hybrid systems[END_REF]:

• MCS topology: How to interconnect individual classifiers.

• Classifier generation: How to generate and select valuable classifiers.

• Classifier combination: How to build a combination function which can exploit the strengths of the selected classifiers and combine them optimally. chitecture, multiple classifiers are designed independently without any mutual interaction and their outputs are combined according to certain strategies [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][START_REF] Rahman | Serial combination of multiple experts : A unified evaluation[END_REF][START_REF] Ranawana | Multi-classifier systems : Review and a roadmap for developers[END_REF]. Alternatively, in the concatenation topology, the classification result generated by a classifier is used as the input into the next classifier [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][START_REF] Rahman | Serial combination of multiple experts : A unified evaluation[END_REF][START_REF] Ranawana | Multi-classifier systems : Review and a roadmap for developers[END_REF]. When the primary classifier cannot obtain the satisfactory classification result, then the outputs of the primary classifier is feed to a secondary classifier, and so on. The main drawback of this topology is that the mistakes produced by the earlier classifier cannot be corrected by the later classifiers.

MCS topology

Besides concatenation and parallel combination, hierarchical combination that combines both concatenation and parallel combination is also used.

A very special case of concatenation topology is the AdaBoost [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF]. The goal of boosting is to enhance the accuracy of any given learning algorithm, even weak learning algorithms with an accuracy slightly better than chance. The algorithm process training of the weak learner multiple times, each time presenting it with an updated weights over the training samples. Then, the weights of misclassified samples are increased to concentrate the learning algorithm on specific samples. Finally, the decisions generated by the weak learners are combined into a single decision.

Classifier generation

Classifier generation aims to build mutually complementary individual classifiers that are accurate and at the same time disagree on some different parts of the input space. That means diversity of the classifier is a vital requirement for the success of the MCS.

Both theoretical and empirical studies indicate that we can ensure diversity using Ho-mogeneous and Heterogeneous approaches [START_REF] Kuncheva | Combining Pattern Classifiers : Methods and Algorithms[END_REF][START_REF] Rokach | Pattern classification using ensemble methods[END_REF]. In Homogenous approaches, we can obtain a set of classification results obtained by the same classifier by injecting randomness into the classifier, manipulating the training sample and the input features. The Heterogeneous approaches are to apply different learning algorithms to the same training set. First of all, we will start to review some diversity measures, and then will show that how to generate diverse classifiers followed to ensure the diversity in the ensemble.

Diversity measures

Diversity represents the difference among the individual classifiers [START_REF] Krogh | Neural network ensembles, cross validation, and active learning[END_REF][START_REF] Chandra | Evolving hybrid ensembles of learning machines for better generalisation[END_REF]. Fig. 2.2 presents four different classifiers combination within three classes (9 samples) using majority vote approach. The overall accuracy of all individual classifier is 6/9. The overall accuracies of the four combinations are 1, 8/9, 6/9 and 5/9, respectively. Our goal is to use diversity measures to find the classifier combination like in Figure 2.2(a) or (b) and avoid to select the third or especially the fourth classifiers combination. Kuncheva and Whitaker summarized the diversity measures in classifier ensembles [START_REF] Kuncheva | Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy[END_REF]. A special issue called "Diversity Measure in Multiple Classifier System" published in Information Fusion journal indicates that diversity measure is an important research direction in MCS [START_REF] Ludmila | Diversity in multiple classifier systems[END_REF]. Petrakos et al. applied agreement measure in decision fusion level combination [START_REF] Michail | The effect of classifier agreement on the accuracy of the combined classifier in decision level fusion[END_REF]. Foody compared the different classification results from three aspects: similarity, non-inferiority and difference using hypothesis tests and confidence interval algorithms [START_REF] Foody | Classification accuracy comparison : Hypothesis tests and the use of confidence intervals in evaluations of difference, equivalence and non-inferiority[END_REF]. It is proved that increasing diversity should lead to the better accuracy, but there is no formal proof of this dependency [START_REF] Brown | Diversity creation methods : a survey and categorisation[END_REF]. And how to measure classifier diversity is still an open question. Table 2.1 summarizes the 15 diversity measures with their types, data range and literature sources.

Diversity measures also play an important role in ensemble pruning. Ensemble pruning aims at reducing the ensemble size prior to combination while maintaining a high diversity among the remaining members in order to reduce the computational cost and memory storage. To deal with the ensemble pruning process, several approaches have been proposed such as clustering-based,ranking-based and optimization-based approaches [START_REF] Tsoumakas | An ensemble pruning primer[END_REF].

Ensuring diversity

Following the steps of pattern classification, we can enforce the diversity by the manipulation of training samples, features, outputs and classifiers.

Manipulating the training samples. In this method, each classifier is trained on the different versions of training samples by exchanging the distribution of original training samples. This method is very useful for the unstable learner (decision tree and neural Mutual Information(M I) [81] * Note: 'p' stands for 'Pairwise' and 'n' stands for 'Non pairwise'. 's' means 'similarity', 'c' means 'correlation' and 'div' means 'dissimilarity'. The arrow specifies the greater diversity if the measure is lower(↓) or higher(↑). network). Small changes in the training set will lead to a major change in the obtained classifier. Bagging and Boosting belong to this category [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF][START_REF] Breiman | Bagging predictors[END_REF]. Bagging applies sampling with replacement to obtain the independent training samples for individual classifiers. Boosting changes the weights of training samples according to the results of the previous classifiers, focusing on the wrong classified samples, making the final result using a weight vote rule.

p s/c ↓ / [72] Q-statistic(Q) p s/c ↓ [-1, 1] [65, 73] Correlation coefficient(ρ) p s/c ↓ [-1, 1]
Manipulating the training features. The most well-known algorithm of this type is Random subspace [START_REF] Ho | The random subspace method for constructing decision forests[END_REF]. The Random subspace was employed for several types of base learners, such as decision tree (Random Forest) [START_REF] Breiman | Random forests[END_REF], SVM [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]. The other development is Attribute Bagging, which establishes the appropriate size of a feature subset, and then creates random projections of a given training set by random selection of feature subsets [START_REF] Bryll | Attribute bagging : improving accuracy of classifier ensembles by using random feature subsets[END_REF].

Manipulating the outputs. Multi-classification problem can be converted into several two-class classification problems. Each problem discovers the discrimination between one class and the other classes. Error Correcting Output Coding (ECOC) adapts a code matrix to convert a multi-class problem into binary ones. Ensemble of multi-classifier classification problem can be treated as ensembles of multiple two-classifier classification problem, and then combined together [START_REF] Dietterich | Solving multiclass learning problems via error-correcting output codes[END_REF]. The other method to deal with the outputs is label switching [START_REF] Martinez-Munoz | Switching class labels to generate classification ensembles[END_REF]. This method generates an ensemble by using perturbed version of the training set where the classes of the training samples are randomly switched. High accuracy can be achieved with fairly large ensemble generated by class switching.

Manipulating the individual classifiers. In this stage, we can use different types of individual classifiers or the same classifiers with different parameters. For instance, when the support vector machine is selected as the base learner, we can gain diversity by using different kernel functions or parameters.

Classifier combination

Once the individual classifier has been designed and implemented, the next task is to combine the individual results obtained through each individual classifier.

The output of a classifier can take many forms. A formal classification for the types of outputs which can be presented below:

• Abstract level. The classifier outputs a single unique class.

• Rank level. The classifier ranks each output class according to its belief on which class the input data belongs to.

• Measurement level. The output of the classifier is an array which contains these belief values. These classifiers are also known as probabilistic classifiers.

According to the output of classifier, classifier combination approach can be divided into three levels: abstract level, rank level and measurement level [START_REF] Ruta | An overview of classifier fusion methods[END_REF]. The abstract level combination methods are applied when each classifiers outputs a unique label [START_REF] Ruta | An overview of classifier fusion methods[END_REF]. Rank level makes use of a ranked list of classes where ranking is based on decreasing likelihood.

In

Experimental results

In this subsection, we provide an experiment to access the performances of the classifier combination algorithms. Airborne OMISII hyperspectral remote sensing image with 64 bands with spectral range: 450-1090 nm is used as the data source. The spectral resolution is 10 nm. Zhongguangcun, a high-tech zone of Beijing City is chosen as the study area. The five noisy bands are removed so it is remaining 59 bands for classification. The image size is 400 × 400. Here, we selected four individual classifiers, including support vector machines (SVM), J48 Decision tree(variant of C4.5), multiple-layer perception neural network (MLPNN), radius bias function neural network (RBFNN), which are implemented in WEKA software 1 . The parameters in these methods are set to the default values in the software. The classification results of the four base learners can be seen in Fig. 2.3. Table 2.3 summarizes the accuracies of individual classifiers and classifier combination approaches. From Table 2.3, DWDCS achieved the highest accuracy. The computation time of DWDCS is proportional to the size of validation dataset and the nearest neighbors. In our experiments, the number of validation dataset is 500 and the nearest neighbor is set to be 15. The computation time of DWDCS is more than 10 min and other parallel combination strategies are less than 5 s. Among the classifier combination methods, MV and BA are the fastest ones and also gain high classification accuracy. Therefore, in this thesis, MV and BA are used to combine the hard labels and class probabilities, respectively. [START_REF] Yang | A dynamic subspace method for hyperspectral image classification[END_REF] proposed dynamic subspace method to improve random subspace method on automatically determining dimensionality and selecting component dimensions for a various subspace. Du et al. [START_REF] Du | Hyperspectral remote sensing image classification based on decision level fusion[END_REF] produced diverse classifiers using different feature extraction methods and then combined their results using evidence theory and linear consensus algorithms.

Recently, Tarabalka et al. designed a new multiple-classifier approach for spectralspatial classification of hyperspectral data [START_REF] Tarabalka | Multiple spectral-spatial classification approach for hyperspectral data[END_REF]. This approach consists of three steps:

• Step 1: Pixel-wise classification using SVM classifier and unsupervised segmentation by watershed segmentation (WS), expectation maximization (EM), and Hierarchical image SEGmentation (HSEG) are separately applied to hyperspectral data. Then, three spectral-spatial classifiers (SVM+EM, SVM+WS and SVM+HSEG) are produced by combining pixel-wise classification and three segmentation approaches, respectively.

• Step 2: Three spectral-spatial classifiers (SVM+EM, SVM+WS and SVM+RHSEG) are used to select the marker pixels, which get the same class label from all the classifiers.

• Step 3: A minimum spanning forest is built, where each tree is rooted on a classificationdriven marker and forms a region in the spectral-spatial map.

The multiple spectral-spatial classifier has introduced a powerful and commonly used spectral-spatial MCS approach for hyperspectral data classification, which can combine the advantages of three spectral-spatial classifiers and meanwhile avoid their weakness.

Stumpf et al. proposed a supervised workflow which takes the advantage of recent advances in object-oriented image analysis (OOA) and Random Forest, to investigate the [START_REF] Xia | Hyperspectral remote sensing image classification based on rotation forest[END_REF] Rotation Forest Hyperspectral images Landslide inventory mapping [START_REF] Stumpf | Object-oriented mapping of landslides using random forests[END_REF]. A sequence of image segmentation, feature selection, object classification and error balancing was developed and tested on a variety of sample datasets (Quickbird, IKONOS, Geoeye-1, aerial photographs) of four sites in the northern hemisphere recently affected by landslides (Haiti, Italy, China, France). Employing approximately 20% of the data for training, the proposed workflow resulted in accuracies between 73% and 87% for the affected areas, and approximately balanced commission and omission errors.

Puissan et al. [START_REF] Puissant | Object-oriented mapping of urban trees using random forest classifiers[END_REF] proposed a methodology to inventory and map the urban tree spaces from a mono-temporal very high resolution (VHR) optical image using a Random Forest classifier in combination with object-oriented approaches. The methodology was developed and its performance was evaluated on a dataset of the city of Strasbourg (France) for different categories of built-up areas. The results indicated a good accuracy and a high robustness for the classification of the green elements in terms of user and producer accuracies.

Introduction

A successful MCS should be one where the member classifiers are accurate as well as the diversities among them are obvious. Combining similar classification results would not further improve the accuracy [START_REF] Kuncheva | Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy[END_REF][START_REF] Waske | Mapping of hyperspectral aviris data using machine-learning algorithms[END_REF][START_REF] Ceamanos | A classifier ensemble based on fusion of support vector machines for classifying hyperspectral data[END_REF]. Two basic approaches are Boosting and Bagging [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF][START_REF] Breiman | Bagging predictors[END_REF]. Boosting processes data with iterative re-training, and the weights of misclassified samples are increased to concentrate the learning algorithm on specific samples [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF][START_REF] Gislason | Random forests for land cover classification[END_REF]. In contrast, Bagging can produce accurate ensemble by training many classifiers on boot-strapped samples from training set [START_REF] Breiman | Bagging predictors[END_REF]. Diversity in Bagging is provided with further randomization yielding Random Forest ensemble approach [START_REF] Breiman | Random forests[END_REF]. Random Forest adopts decision trees trained on bootstrap samples and the diversity is promoted with random choice of features at each node while constructing the trees. It can overcome the drawbacks of Bagging and Boosting algorithms (e.g., high computational cost and sensitivity to noise) [START_REF] Benediktsson | Multiple classifier systems in remote sensing : from basics to recent developments[END_REF]. In addition, limiting the number of variables in Random Forest used for a split, the computational complexity can be reduced and the correlation between the trees be decreased. This enables Random Forest to deal with high-dimensional datasets [START_REF] Waske | Mapping of hyperspectral aviris data using machine-learning algorithms[END_REF].

In this chapter, we proposed to use Rotation Forest, which extends the idea of Random Forest [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF]. The purpose of Rotation Forest is to encourage simultaneously both member diversities and individual accuracy within a classifier ensemble. In the framework of Rotation Forest, each classifier is independently constructed by decision tree model, and each decision tree is trained on the training samples in a rotated feature space derived from feature extraction techniques. One of the most important point of Rotation Forest is to select the feature extraction. The capabilities of different feature transformation algorithms, including PCA, independent component analysis (ICA), maximum noise fraction (MNF) and local fisher discriminant analysis (LFDA) are explored.

The remainder of this chapter is organized as follows. In Section 3.2, we introduce the base learner: classification and regression tree (CART). Traditional decision tree ensemble approaches, including Bagging, AdaBoost, Random Subspace and Random Forest, are described in Section 3.3. The main idea and the implementation of Rotation Forest is shown in Section 3.4. Experimental results are presented in Section 3.5. The general conclusion and perspective of this chapter are drawn in Section 3.6.

Classification and Regression tree

Classification and regression tree (CART) method was developed in 80s by Breiman, Freidman, Olshen, Stone in their paper "Classification and Regression Trees" [START_REF] Breiman | Classification and regression trees[END_REF]. From a geometrical point of view, the principle of tree structured models is beautifully simple. As shown in Fig. 3.1 (a), a tree is a hierarchical structure that consists of nodes and edges. Nodes can be divided into three groups: root node, internal (split) nodes and terminal (leaf) nodes. Split and leaf nodes have exactly one incoming edge while each internal node has exactly two outgoing edges. Let us define the following concepts of a tree: Definition 3.1. A tree is a directed graph G = (V, E) in which any two vertices (or nodes) are connected by exactly one path. Definition 3.2. A rooted tree is a tree in which one of the nodes has been designated as the root. Definition 3.3. If there exists an edge from t 1 to t 2 , then node t 1 is said to be the parent of node t 2 while node t 2 is said to be the child of node t 1 . Definition 3.4. In a rooted tree, a node is said to be internal if it has one or more children and terminal if it has no children. Terminal nodes are also known as leaves. Definition 3.5. A binary tree is a rooted tree where all internal nodes have exactly two children.

A decision tree is a tree used to make decisions in each terminal node. Decision tree is represented by a set of questions which split the data into smaller and smaller parts. It can be interpreted as a technique for splitting complex problem into a hierarchy of simpler ones. Just taking a simple example in Fig. 3.1 (b), we want to know a photo whether it represents an indoor scene or an outdoor one. We can start asking a question ('Is top part blue?') of root node and the right answer ('true') of the data is sent to the left or right child. Then a new test is applied until the data reaches a leaf, which contains the answer ('outdoor') [1].

In general, CART algorithm consists of tree parts:

• Construction of maximum tree.

• Choose the right tree size.

• Classification of new data using constructed tree.

Let t p be a parent node and t l , t r be a left and a right child nodes of parent node t p . Let X train be an input variable matrix with n observations and D number of variable x j . Let class vector Y train consist of n observations with total number of C classes.

Classification tree is built in accordance with splitting rule, which performs the splitting of training samples into smaller parts. In each time, the data will be divided into two parts with maximum homogeneity. This can be shown in the Fig. 3.2.

Maximum homogeneity of child nodes is defined as impurity function i(t). Since the impurity of parent node t p is constant for any of the possible splits x j ≤ x R j , j = 1, 2, ..., D, the maximum homogeneity of left and right child nodes will be equivalent to the maximization of change of impurity function ∆i(t):

∆i(t) = i(t p ) -E [i(t c )] (3.1)
where, t c means left and right child nodes of the parent node t p . Furthermore, the above equation can be formulated as follows:

∆i(t) = i(t p ) -p l i(t l ) -p r i(t r ) (3.2)
where, p l and p r are the proportion nt l nt p and nt r nt p of learning samples from t p going to t l and t r , respectively. n t l is the size of the subset t l . Thus, at each node, CART want to solve the following maximization problem: arg max

x j ≤x R j ,j=1,2,...,D [i(t p ) -p l i(t l ) -p r i(t r )] (3.3)
It is implied that CART search through all possible values of all variables in training samples for the best split x j ≤ x R j which will maximize the change of impurity measure ∆i(t).

The next step is to define the impurity function. In theory there are several impurity functions. Two of them are widely used in practice: Gini and Twoing splitting rules.

Gini index

Gini index uses the impurity function i(t) as follows:

i(t) = j =m p(k|t)p(m|t) (3.4)
where, k and m means the index of the class k, m ∈ {1, 2, ..., C}, p(k|t) is the conditional probability of class k provided by the node t. By combining the two equations, the change of impurity measure ∆i(t) is given by ∆i

(t) = - C k=1 p 2 (k|t p ) + p l C k=1 p 2 (k|t l ) + p r C k=1 p 2 (k|t r ) (3.5)
Gini index can be solved by the following equation: arg max

x j ≤x R j ,j=1,2,...,D - C k=1 p 2 (k|t p ) + p l C k=1 p 2 (k|t l ) + p r C k=1 p 2 (k|t r ) (3.6)

Twoing splitting rule

Unlike Gini index, the objective of Twoing splitting rule is to search for two classes that will make up together more than 50% of the data. Twoing splitting rule is aimed at maximizing the change of impurity measure as follows:

∆i(t) = p l p r 4 C i |p(i|t l ) -p(i|t r )| 2 (3.7)
which implies the following maximization problem: arg max

x j ≤x R j ,j=1,2,...,D   p l p r 4 C i |p(i|t l ) -p(i|t r )| 2   (3.8)
Twoing splitting rule can build more balanced trees, but the computation time is longer than Gini index [START_REF] Breiman | Classification and regression trees[END_REF]. Timofeev founds that there is a small difference between tree constructed using Gini and tree constructed via Twoing index, the difference can be seen only at the bottom of the tree where the variables are less significant in comparison with top of the tree [START_REF] Timofeev | Classification and Regression Trees (CART) Theory and Applications[END_REF]. Therefore, considering the computational complexity, we only adapt Gini index to construct CART and its ensembles.

Maximum trees may turn out to be of very high complexity and consists of hundred levels. Therefore, pruning the decision trees before being used for classification is important. Pruning trees implies cutting off insignificant nodes and even subtrees. Generally, pruning usually yields better classification results than maximum tree. For an interpretability point of view, it is also a very effective framework for simplifying decision trees and better understanding the structure in the data. However, Louppe pointed out that pruning is no longer required to achieve good generalization performance in the context of ensemble of decision trees [START_REF] Louppe | Understanding Random Forests[END_REF]. In this chapter, pruning is not applied to the CART and its ensembles.

After the classification tree is constructed, it can be used for classification of new sample. The output is an assigned class to each of the new observations. Each of the new observations will get to one of the terminal nodes of the tree by set of questions. A new observation is assigned with the dominating class of terminal node, where this observation belongs to. Dominating class is the one which has the largest amount of observations in the current node.

The success of decision tree (and by extension, of all tree-based methods) is explained by several factors that make them quite attractive in practice [START_REF] Louppe | Understanding Random Forests[END_REF]:

• Decision trees are non-parametric. They can model arbitrarily complex relations between input and outputs, without any a prior assumption.

• Decision trees can handle heterogeneous data (ordered or categorical variables, or mix of both).

• Decision trees intrinsically implement feature selection, making them robust to irrelevant or noisy variables (at least to some extent).

• Decision trees are easily interpretable, even for non-statistically oriented users.

Meanwhile, CART has several limitations [START_REF] Louppe | Understanding Random Forests[END_REF]:

• CART may have unstable decision trees. Small changes in the data can completely change the fitted tree.

• CART splits only by one variable.

• CART needs more data than parametric procedures.

• CART only produce moderately accurate classification result.

In order to reduce the generalization error of decision tree, multiple decision tree classifiers are combined. In the next section, we will provide several ensemble approaches for combining multiple decision trees.

Ensemble methods

Bagging

Bagging is the abbreviation of bootstrap aggregating. In this algorithm, several samples selected at random from an original training set by replacement and instructive iteration is exerted to create some different bags, and each bag is classified by vote to predict its class [START_REF] Breiman | Bagging predictors[END_REF]. When using sampling with replacement, we can define the size of a new training set smaller than the one of original size. In general, when drawing with replacement n values out of a set of n (different or equally likely), the expected number of unique draws is n 1 -exp -n /n . Algorithm 3.1 shows the main steps of Bagging. For the classification phase, a new sample x * will be run on the output ensemble and the class with the maximum number of votes is chosen as the label for x * .

Boosting

Boosting can process data with weights, and the weights of misclassified samples are increased to concentrate the learning algorithm on specific samples [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF]. Bagging has been shown to reduce the variance of the classification, while Boosting reduces both the variance and the bias of the classification. So in most cases, Boosting can produce more Add the classifier to the current ensemble, L = L ∪ L i . 5: end for accurate classification results than Bagging. However, the computation time of Boosting is more than Bagging, and Boosting is sensitive to noise [START_REF] Gislason | Random forests for land cover classification[END_REF]. AdaBoost improves the simple boosting algorithm via an iterative process. Sometimes, AdaBoost fails to improve the performance of the base classifier. The main reason for AdaBoost's failure is overfitting. A large number of iterations may result in an overcomplex composite classifier. One possible way to avoid overfitting is to select the number of iterations as small as possible. The basic AdaBoost algorithm deal with binary classification. Furthermore, Freund and Schapire developed two versions of the AdaBoost algorithms(AdaBoost.M1 and AdaBoost.M2), which are equivalent for binary classification and differ in their handling of multi-class problem [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF]. In this chapter, we adapt AdaBoost.M1, whose pseudo-code is described in Algorithm 3.2. The classification of a new sample x * is performed according to the following equation:

arg max y∈dom(y)   j:L j (x * )=y log 1 β j   (3.9)

Random subspace

The random subspace method (RSM) is an ensemble construction technique proposed by Ho [START_REF] Ho | The random subspace method for constructing decision forests[END_REF]. In the RSM, the training set is also modified as in bagging. However, this modification is performed in the feature space. Algorithm 3.3 shows the main steps of Random subspace. For the classification phase, a new sample x * will be run on the output ensemble and the class with the maximum number of votes is chosen as the label for x * .

Random Forest

Random forests (seen in Algorithm 3.4) combine Bagging and random subspace method to generate decision forests [START_REF] Breiman | Random forests[END_REF]. It consists of a number of decision trees, of which each Algorithm 3.2 AdaBoost.M1

Input: : n labeled samples T train = X train , Y train = {(x i , y i )} n i=1 with x i ∈ R D and y i ∈ R. Number of classifiers: T . Base classifier: L.
Output: : The classifiers L j , j = 1, 2, ..., T and β j .

1: Initialize the weights of each sample W 0 (i) = 1/n, j = 1, 2, ..., n 2: for j = 1 to T do 3:

Fit a classifier L j to the training data using the weights w 0 4:

j ← i:L j (x i ) =y i 5:
If t > 0.5 then 6:

T ← t -1 7:
exit Loop

8:

EndIf 9:

β j ← j 1-j 10: W j (i) = W j-1 (i) ×    β j L j (x i ) = y i 1 Otherwise 11:
Normalize W j (i) to be a proper distribution. 12: end for Algorithm 3.3 Random Subspace Add the classifier to the current ensemble, L = L ∪ L i . 5: end for tree is trained with the examples bootstrap sampled from the training set. In training each tree, a randomly selected subset of features is used to split each node. Random forests perform similarly as Adaboost in terms of error rate, and it is more robust with respect to noise. Another important feature of the random forests is that it is fast. Add the classifier to the current ensemble, L = L ∪ L i . 5: end for

Input: : n labeled samples T train = X train , Y train = {(x i , y i )} n i=1 with x i ∈ R D and y i ∈ R.

Rotation Forest

Rotation Forest constructs different versions of the training set by employing the following steps: the feature set is divided into disjoint sets on which the original training set is projected. Then, a random sample of classes is eliminated and a bootstrap sample is selected from every projection result. Feature extraction method is used to rotate each obtained subsample. Finally, the components are rearranged to form the dataset that is used to train a single ensemble member. The details of Rotation Forest are presented in Algorithm A.1 [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF].

The strong performance is attributed to a simultaneous improvement of 1) diversity within the ensemble, obtained by the use of feature extraction on training data and 2) accuracy of the base classifiers, by keeping all extracted features in the training data [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF][START_REF] Ozcift | Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms[END_REF].

It is essential to notice step 5 in Rotation Forest presented in Algorithm A.1, the sample size Xtrain i,j is selected smaller than X train i,j due to two reasons: one is to avoid obtaining the same coefficients when the same features are chosen and the other is to enhance the diversity within the ensemble [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF].

Given the importance of the choice regarding the algorithm for feature extraction in Rotation Forest, several alternatives are considered in this chapter. In particular, four linear feature extraction techniques, including principal component analysis (PCA), maximum noise fraction (MNF), independent component analysis (ICA) and local fisher discriminant analysis (LFDA) are considered. randomly split the features F into K subsets F i j 3:

for j = 1 : K do 4:
select the corresponding features of F i j to compose a new training features X train i,j

5:

select a new training samples Xtrain i,j using bootstrap algorithm, whose size is 75% of the original size sparse matrix R i is composed of the above coefficients

R i =         v (1) i,1 , ..., v (M 1 ) i,1 0 • • • 0 0 v (1) i,2 , ..., v (M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1)
i,j , ..., v

(M K ) i,j         9:
rearrange R i to R a i with respect to the original feature set build classifier L i using X train R a i , Y train 12: end for 13: Add the classifier to the current ensemble, L = L ∪ L i .

For linear feature extraction purpose, we often assume that there exists a mapping function f , which transforms each data point of original image x i to z i = f (x i ). This mapping is always represented by a matrix v:

z i = f (x i ) = v T x i (3.10)
In many feature extraction methods, the projection matrix V = (v 1 , v 2 , ...) can be obtained from the d eigenvectors corresponding to the d largest eigenvalues {λ 1 , λ 2 , ..., }, by solving the following eigenvalue decomposition problem:

S 1 v = λS 2 v (3.11)
where, S 1 and S 2 are the specific matrices for different feature extraction methods. PCA is the most popular linear unsupervised feature extraction method, which can keep the most information in a few components in terms of variance [START_REF] Richards | Remote sensing digitial image analysis[END_REF]. Although PCA is not optimal for dimensionality reduction in target detection and classification of hyperspectral data, PCA is still competitive for the purpose of classification because of its low complexity and the absence of parameters [START_REF] Cheriyadat | Why principal component analysis is not an appropriate feature extraction method for hyperspectral data[END_REF][START_REF] Fauvel | Kernel principal component analysis for the classification of hyperspectral remote sensing data over urban areas[END_REF]. For PCA, S 1 is the covariance matrix of the original data and S 2 is the identify matrix.

ICA is a method for separating a multivariate signal into additive sub-components. In a comparison of PCA, ICA not only decorrelates the signals but also reduces higherorder statistical dependence in order to make the signals as independent as possible. In this chapter, we use the FastICA approach, which is an efficient and popular algorithm for ICA invented by Hyvarinen [START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF]. This algorithm is based on a fixed-point iteration scheme maximizing non-Gaussian as a measure of statistical independence and also can be derived as an approximate Newton iteration. More details can be referred to [START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF].

MNF is a linear transformation that contains two separate PCA rotations and a noise whitening step. The general procedures are presented as the followings: 1) Use the noise covariance matrix to decorrelate and rescale the noise in the data (noise whitening). This results in transformed data in which the noise has unit variance and no band-to-band correlations. 2) Perform a standard principal components transformation of the noisewhitened data [START_REF] Green | A transformation for ordering multispectral data in terms of image quality with implications for noise removal[END_REF].

LDA is the best known supervised feature extraction approach. However, this method has a limitation: for C class classification problem, it can extract at maximum C -1 features [START_REF] Landgrebe | Signal Theory Methods in Multispectral Remote Sensing[END_REF][START_REF] Dell'acqua | Exploiting spectral and spatial information in hyperspectral urban data with high resolution[END_REF]. That means in Rotation Forest, we should define the value of C is greater than M . In order to solve the problem, we adapt Local Fisher Discriminant Analysis (LFDA) instead of LDA. LFDA effectively combines the ideas of LDA and Locality Preserving Projection (LPP), which leads to both maximize between-class separability and preserve with-class local structure [START_REF] Sugiyama | Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis[END_REF]. It can be viewed as the following eigenvalue decomposition problem:

S lb v = λS lw v (3.12)
where, v is an eigenvector and λ is the eigenvalue corresponding to v. S lb and S lw denote the local between-class and within-class scatter matrix. LFDA wants to find an eigenvector matrix that maximizes the local between-class scatter in the embedding space while minimizes the local within-class scatter in the embedding space. S lb and S lw can be defined:

S lb = 1 2 n i,j=1 ω lb i,j (x i -x j )(x i -x j ) (3.13) S lw = 1 2 n i,j=1 ω lw i,j (x i -x j )(x i -x j ) (3.14)
where, ω lb and ω lw are the weight matrices with:

ω lb i,j =      A i,j ( 1 n -1 ny i ) y i = y j 1 n y i = y j (3.15)
ω lw i,j =      A i,j ny i y i = y j 0 y i = y j (3.16)
where, A i,j is the affinity value between x i and x j in the local space.

A i,j = exp -x i -x j σ i σ j (3.17)

σ i = x i -x e i (3.18)
where, x e i is the e-th nearest neighbor of x i , n y i is the number of labeled samples in class y i ∈ {1, 2, 3, ..., C}.

Experimental results and analysis

Experimental settings

In order to assess the performance of Rotation Forest algorithm, we conduct the experiments with three widely used hyperspectral images obtained from NASA's Airborne Visible Infra-Red Imaging Spectrometer (AVIRIS), Reflective Optics System Spectrographic Imaging System (ROSIS) and Digital Airborne Imaging Spectrometer (DAIS) owned by the German aerospace center (DLR). AVIRIS dataset is captured over a vegetation area of Indian Pines, Indiana, USA. The image contains 145 × 145 pixels, with 200 spectral bands after removing twenty water absorption bands (104-108, 150-163 and 220). The spatial resolution is 20 m/pixel. ROSIS image with 115 spectral channels is acquired over the university of Pavia, Italy. The image size is 610 × 340 with the spatial resolution of 1.3 m. Twelve noisy channels were removed and the remaining 103 bands with a spectral range from 0.43 to 0.86 µm were used for the experiments. The DAIS image was collected at 1500m flight altitude over the city of Pavia, Italy, with ground resolution of 5m and size of 400 × 400 pixels with 80 spectral bands. Training and Test samples are detailed in Table 3.1.

In all cases, the performance achieved by Rotation Forest is illustrated using the following designs:

• Number of features in each subset: M = 10;

• Number of classifiers in the ensemble: T = 10;

• Feature extraction method: PCA [START_REF] Richards | Remote sensing digitial image analysis[END_REF], ICA [START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF], MNF [START_REF] Green | A transformation for ordering multispectral data in terms of image quality with implications for noise removal[END_REF] and LFDA [START_REF] Sugiyama | Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis[END_REF];

• Base classifier: classification and regression tree (CART).

Furthermore, the popular ensemble methods, including Bagging [START_REF] Breiman | Bagging predictors[END_REF], AdaBoost [START_REF] Freund | Experiments with a new Boosting algorithm[END_REF], Random subspace (RS) [START_REF] Ho | The random subspace method for constructing decision forests[END_REF], Random Forest (RF) [START_REF] Breiman | Random forests[END_REF], support vector Machine (SVM) [START_REF] Plaza | Recent advances in techniques for hyperspectral image processing[END_REF] and logistic regression via variable splitting and augmented Lagrangian (LORSAL) [START_REF] Li | Spectral-spatial classification of hyperspectral data using loopy belief propagation and active learning[END_REF] are added to be compared with Rotation Forest. The required parameters of SVM with RBF are the penalty factor C and kernel width γ. We use a five-fold crossvalidation grid search method to find the best combination of C and γ within the set

C ∈ {2 1 , 2 2 , 2 3 , 2 4 , 2 5 , 2 6 , 2 7 , 2 8 } and γ ∈ {2 -3 , 2 -2 , 2 -1 , 2 0 , 2 1 , 2 2 , 2 3 }.
The number of neighbors in LFDA is chosen from {1, 3, 5, 7} using five-fold cross validation. In the following experiments, we employed RoF(PCA, ICA, MNF and LFDA) as the abbreviations of Rotation Forest with PCA, ICA, MNF and LFDA transformations. Overall accuracy (OA) is used as the quantitative index. In order to investigate the impact of the labeled samples on the classification accuracy, we randomly select the labeled samples with replacement from the original training samples whose sizes correspond to four cases: 25% (case 1), 50% (case 2), 75% (case 3), 100% (case 4) rate of original size of training samples. In order to increase the statistical significance of the results, each value of OA reported in this work is obtained as the average of 10 Monte Carlo (MC) runs.

Experimental results

Table 3.2 shows the classification accuracies (OA%) obtained by the Rotation Forest approaches as well as other algorithms using different training samples. We highlight the highest OA of each case in bold font.

For AVIRIS dataset, it can be seen that RoF-PCA and RoF-ICA achieve better results than other ensemble approaches (Bagging, Adaboost, Random subspace and Random Forest), where the OA is always increased as the number of training samples is increased. For instance, in case 1, CART, Bagging, Adaboost, RS and RF acquired an OA of 57.25%, 66.5%, 66.98%, 58.14% and 71.38%, respectively. RoF-PCA and RoF-ICA respectively increased the OA to 79.65% and 76.78%, while the OA of RoF-MNF and RoF-LFDA were improved to 76.78 % and 71.66%. LORSAL yielded the highest OA and RoF-PCA gave the better performance than SVM in all cases. The classification performance of RS ensemble method is slightly better and sometimes worse than the one of CART because of low value of M . Ho [START_REF] Ho | The random subspace method for constructing decision forests[END_REF] suggested that the appropriate value of M is set to be D/2 for RS DT ensemble classifier. In addition, we have compared the computation times of these methods on an Intel(R) Xeon(R) CPU X5660 @ 2.80 GHz 2.79GHz, two processors, 12 GB memory. In case 2, the computation times of RoF-PCA, RoF-MNF, RoF-ICA, RoF-LFDA, SVM and LORSAL was 8.53s, 9.28s, 9.16s, 36.84s, 42.65s and 3.18s respectively. The computation times of Rotation Forest approaches are longer than those of Bagging, AdaBoost, Random subspace and Random Forest. For the approaches of SVM and RoF-LFDA, the computational time also included the time consumed on the parameter determination. Therefore, RoF-PCA, RoF-MNF and RoF-ICA are efficient than RoF-LFDA and SVM.

For ROSIS dataset, it can be observed (seen in Table 3.2) that RoF-PCA outperforms other algorithms in all cases. RoF-LFDA gave the better performance than RoF-MNF. In case 4, the corresponding OA of RoF-PCA achieved on the test set was 83.14%, higher than the one of SVM (79.98%) and LORSAL(80.09%). The computation complexity is similar with the previous AVIRIS experiment. The computation complexity of RoF-PCA, The Pavia Center DAIS data set was easy to classify since even the CART achieves extraordinarily high classification accuracy. Regarding the global accuracies, Rotation Forest with different transformation algorithms are all superior to other compared approaches. In case 1-3, RoF (PCA) achieved the best global accuracies with the OA (95.58%). And RoF (LFDA) yielded the highest OA (95.92%) in case 4.

Sensitivity to Parameters

Ensemble size (T ) and the number of features in a subset (M ) are the key parameters of Rotation Forests, also as an indicator of the operating complexity. In order to investigate the impacts of these parameters, we have performed the classification performances using different ensemble sizes when the number of features in a subset M is fixed to be 10 and different number of features in a subset when ensemble size T is fixed to be 10, respectively.

(a) (b) (c) (d) (e) (f) (g) (h) (i)

Summary

In this chapter, a new ensemble classifier, called Rotation Forest, was introduced into hyperspectral remote sensing image classification. It consists in splitting the feature set into K subsets, running data transformation algorithms separately on each subset and then reassembling a new extracted feature set while keeping all the components. CART classifier is used as the base classifier. Different splits of the feature set lead to different rotations. Thus diverse classifiers are obtained. Thus, diversity and accuracy are targeted together . We have applied Rotation Forests with different data transformation approaches, including PCA, MNF, ICA and LFDA to classify hyperspectral remote sensing image and compared with Bagging, AdaBoost, Random subspace and Random Forest and other advanced classifiers. Experimental results have shown that Rotation Forests outperformed other ensemble methods in terms of accuracies and are competitive with the kernel-based classifiers (SVM and LORSAL). The key parameters of Rotation Forests are also explored in chapter. Future studies will be focused on the integration of Rotation Forest with other ensemble approaches, the selection of an optimized decision tree model, and the use of spectral-spatial feature extraction algorithms.

(a) (b) (c) (d) (e) (f) (g) (h) (i)

Introduction

Support vector machines (SVM) and multiple classifier system (MCS) can gain high precision classification results based on different principles. The former is a strong single supervised learning algorithm with the purpose of finding the optimal hyperplane to separate the data from different classes while the latter is the combination of many simple outputs. It seems to be more useful to combine SVMs and MCSs further to improve the classification performance. However, far too little attention has been paid to the integra- and Ceamanos et al. [START_REF] Ceamanos | A classifier ensemble based on fusion of support vector machines for classifying hyperspectral data[END_REF] are both superior to a regular SVM, they suffer a limitation of diversity within the ensemble. Thus, a more appropriate scheme should be designed to combine multiple SVM classifiers.

In this chapter, inspired by the idea of Rotation Forest [START_REF] Xia | Hyperspectral remote sensing image classification based on rotation forest[END_REF][START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF], we present rotationbased SVM (RoSVM) ensemble that uses feature extraction and random feature selection to generate independent variants of SVM results. Random Projection (RP), which is treated as an effective and simple feature extraction technique, is introduced into RoSVM. Classification results under this MCS are expected to show significant advantages over the previous study that only use random feature selection [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]. In order to investigate the performances of RoSVM, this article seeks to address the following questions:

• Is there a significant increase in accuracy of RoSVM compared to Random subspace SVM (RSSVM) ?

• How do the two significant parameters, ensemble sizes and number of features in the subset, impact the performances of SVM ensembles? Are there recommended values for these parameters?

• Is RP better than PCA to rotate the axes for the ensemble of SVM classifiers?

To answer the above questions, we assess the performances of RoSVM ensemble based on several experiments carried on three different hyperspectral datasets, i.e. an agriculture area from Indian Pines, USA, a university area and a urban area from Pavia, Italy. The This chapter has been divided into five sections. Section 4.2 gives brief descriptions of SVMs. We introduce the RoSVM ensembles in Section 4.3. Experimental results and discussion are presented in Section 4.4. The last section presents the conclusions and perspectives.

Support Vector Machines

The Support Vector Machines (SVMs) based on structural risk minimization are supervised learning algorithms that can be used for classification and regression [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF]. The primary goal of SVMs is to find the unique hyperplane having the maximum margin that can linearly separate the two classes (Fig. 4.1).

Let X train , Y train = {x i , y i } n i=1 be training samples. F represents the features with dimension D. Y train = {-1, +1}. SVM are learned by solving the following primal optimization problem:

min w,ξ 1 2 w 2 + C n i=1 ξ i (4.1)
subject to:

y i (w • x i + b) ≥ 1 -ξ i , ξ i ≥ 0 (4.2)
where, w is the vector related to hyperplane and b is the bias. ξ i is the slack variable that deals with misclassified samples (Fig. 4.1).

In its dual the form, the optimization problem is:

max a n i=1 α i - 1 2 i,j α i α j y i y j x i x j (4.3) subject to: 0 ≤ α i ≤ C (4.4)
where, C is an hyper-parameter that controls the degree of mis-classification of the model, in case classes are not linearly separable. From the solution of dual problem, we have

w = n i=1 α i y i x i (4.5)
from which the final linear model can finally be expressed. SVM extends to non-linear classification by projecting the original input space into a high-dimensional space (the so-called kernel trick), where a separating hyperplane can hopefully be found. Interestingly, the dual optimization problem is exactly the same, expect that the dot product x i • x j is replaced by a kernel K(x i , x j ), which corresponds the dot product of x i and x j in the new space. A detailed description of the concept of SVMs can be found in [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF][START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF].

In recent years, SVMs have been employed successfully in the classification of hyperspectral data with its advantages to solve difficulties such as small-size samples, poor generalization, etc [START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF]. A review about remote sensing implementations of SVMs is detailed in [START_REF] Mountrakis | Support vector machines in remote sensing : A review[END_REF].

In this paper, one against one (OAO) strategy is chosen to classify hyperspectral data using multi-class SVM approach [START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF]. The effectiveness of SVM depends on the kernel function, the parameters of the kernel and the soft margin parameter C . A Gaussian radial basis function (RBF) with parameter γ is chosen, and the best combination of C and γ is selected by a grid-search.

Rotation-based SVM Ensemble

SVM classifiers have shown excellent classification performances for hyperspectral remote sensing images. Because SVM classifier is very stable, small changes in the training set do not produce very different SVM classifiers [START_REF] Buciu | Demonstrating the stability of support vector machines for classification[END_REF]. Therefore, it is difficult to get an SVMs ensemble that performs better than a single SVM using Bagging and Boosting framework.

Inspired by the Random subspace proposed by Ho [START_REF] Ho | The random subspace method for constructing decision forests[END_REF], Waske et al. [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF] designed a MCS based on SVM and random feature selection for the classification of hyperspectral data, which can get better classification result than any single SVM. The limitation of this MCS is to only use the random feature selection to promote the diversity within the ensemble. In order to further enhance the diversity within the ensemble, RoSVM ensemble, which combines random feature selection and data transformation, is developed to construct the diverse classifiers. Let us denote T the number of classifiers, K the number of subsets and M the number of features in each subset.

In the first step, F is split into K groups and each group has M features. Please note that the last group may have the number of features lower than M if D is indivisible by K.

In the second step, a new training set Xtrain i,j is selected from the training set X i,j using bootstrap technique with 75% size, where X train i,j represents the j th (j = 1, .., K) subset of the i th (i = 1, .., T ) classifier, corresponding to the feature subset F i,j .

In the third step, Xtrain

i,j
is transformed by a feature extraction algorithm to get the coefficients v

(1) i,j , ..., v

(M j ) i,j , the size of v (•) i,j is M × 1.
In the fourth step, a sparse rotation matrix R i is obtained with the above coefficients:

R i =            v (1) i,1 , ..., v (M 1 ) i,1 0 • • • 0 0 v (1) i,2 , ..., v (M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1) i,K , ..., v (M K ) i,K           
Then, the columns of R i are rearranged to R a i respected to the order of the original feature set.

In the fifth step, the training feature is X train R a i for the i th SVM classifier and each SVM classifier is trained in parallel topology. The final classification result is produced by combining the individual classification results using the majority vote rule.

It should be noticed that in the second step, the objective of selecting 75% size of X train i,j is to avoid obtaining the same coefficients when the same features are chosen and hence promote the diversity within the ensemble [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF].

The success of RoSVM ensemble depends on feature extraction. Feature extraction presented in this work is not to perform dimensionality reduction but just to rotate the dataset by keeping all the components. The first work of rotation-based ensemble is to construct multiple DT classifiers, called Rotation Forest, with PCA feature extraction. In our previous work [START_REF] Xia | Hyperspectral remote sensing image classification based on rotation forest[END_REF], we applied Rotation Forest to classify hyperspectral remote sensing images using four feature extraction algorithms, including principal component analysis (PCA) [START_REF] Richards | Remote sensing digitial image analysis[END_REF], Maximum Noise Fraction (MNF) [START_REF] Green | A transformation for ordering multispectral data in terms of image quality with implications for noise removal[END_REF][START_REF] Chang | Interference and noise adjusted principal components analysis[END_REF], Independent Component Analysis (ICA) [START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF] and Local Fisher Discriminant Analysis (LFDA) [START_REF] Sugiyama | Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis[END_REF]. In most cases, Rotation Forest with PCA transformation achieves the highest overall accuracies. Unfortunately RoSVM with PCA is quite expensive to compute for hyperspectral image analysis. A computationally simple method of dimensionality reduction that does not introduce a significant distortion in the data set would thus be desirable. In this chapter, we will introduce Random Projection (RP) into RoSVM ensemble classifiers. In RP, the original high-dimensional data is projected onto a lower-dimensional subspace using a random matrix whose columns have unit lengths. RP has been found to be a computationally efficient, yet sufficiently accurate method for dimensionality reduction of high-dimensional data sets.

In RP, the original D-dimensional is projected to a d-dimensional (d ≤ D) subspace through the origin, using a random d × D matrix v whose columns have unit lengths. Using matrix notation where x i is the original set of D-dimensional observation:

x RP i = v d×D x i (4.6)
is the projection of the data onto a lower d-dimensional subspace. The key idea of random mapping arises from the Johnson-Lindenstrauss (JL) lemma: if points in a vector space are projected onto a randomly selected subspace of suitably high dimension, then the distance between the points are approximately preserved [START_REF] Achlioptas | Database-friendly random projections[END_REF][START_REF] Baraniuk | A simple proof of the restricted isometry property for random matrices[END_REF]. RP is computationally very simple: forming the random matrix v and projecting the

D × N data matrix x i into d dimensions is of order O(dN ).
In most projection methods, the resulting transformation matrix can not take a d value bigger than D (e.g., PCA). However, for RPs it is possible to get a projected space which is bigger than the original, because in RPs the matrix entries are simply random numbers. Three types of RP are used in this work [START_REF] Achlioptas | Database-friendly random projections[END_REF][START_REF] Baraniuk | A simple proof of the restricted isometry property for random matrices[END_REF]:

1. Gaussian. The entire values v come from a Gaussian distribution (mean 0 and standard deviation 1).

2. Sparse. The values v in the transformation matrix are √ 3 × α, where, α is a random number generated by the following conditions: -1 with the probability 1/6, 0 with the probability 2/3 and +1 with probability 1/6. A freely Library for SVM, LibSVM with Matlab implementation, is chosen for training the SVM classifier [START_REF] Chang | LIBSVM : A library for support vector machines[END_REF]. Gaussian RBF kernel is used, and the combination of C and γ is selected by a grid search using a fivefold cross validation. Random subspace SVM (RSSVM) ensemble is added to compare RoSVM [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]. A grid search is performed in each SVM classifier within the ensemble, that means the parameters C and γ of each SVM classifier in the MCS are the optimal ones. Overall accuracy (OA), which is the percentage of correctly classified samples in the test samples and average accuracy (AA), which is the average percentage of correctly classified samples for all the individual classes, are used to measure the accuracies.

Results of AVIRIS Indiana Pines image

We first evaluated the classification performances of the proposed RoSVM ensemble methods using the Indian Pines AVIRIS image in Fig. 4.3(a) with ground truth in Fig. 4.3(b). Table 4.2 presents the average OA of SVM and SVM ensemble approaches over 10 Monte Carlo runs using different numbers of training samples and ensemble sizes. As can be seen from the table above, the classification performances of RoSVM ensembles are significantly better than those of single SVM and RSSVM ensemble. In Table 4.2, there is a clear trend that increasing the size of the training set leads to higher OA and AA of SVM and SVM ensembles. In this case, a regular SVM achieved an OA between 57.16% and 74.22% and an AA between 68.26% and 82.11%; RSSVM ensemble gained an OA between 62.46% and 78.58% and an AA between 74.14% and 85.44%. Compared to a regular SVM and RSSVM, RoSVM ensemble classifiers yield OA (AA) between 68.27% (78.15%) and 83.02% (89.99%), 66.98% (77.92%) and 82.93% (89.66%), 66.01% (76.87%) and 82.19% (89.04%), 65.79% (76.28%) and 82.69% (89. RoSVM is superior to a regular SVM and RSSVM using ten member classifiers. For each ensemble method, no significant increase was found by adding the ensemble size. The number of features in the subset significantly impacted the overall accuracy. Effects of varying M on the performance of RoSVM are shown in Fig. 4.4. It is apparent from Fig. 4.4 that RSSVM can achieve higher accuracy than a regular SVM using medium value of M (M = 50) and that using a small number of features in a subset (M = 10) is ineffective in terms of accuracies. This observation is the same with the work presented by Waske et al. [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]. Unlike RSSVM, RoSVM can enhance the classification performances for all the values of M . The maximum accuracy is acquired by generating RoSVM ensembles with M = 100. Table 4.3 shows the overall, average and class-specific accuracies for the Indian Pines AVIRIS image using SVM and SVM ensemble methods. RoSVM ensemble approach outperformed a regular SVM and RSSVM ensemble for all the classes (except for the class with accuracy of 100%). In Table 4.3, we can find the accuracies of Class Corn-no till, Grass/pasture, Soybeans-min till and Soybeans-clean till achieved by RoSVM are improved by four percentage points or even more when compared to RSSVM. The individual accuracies of other classes produced by RoSVM ensembles are slightly higher than those of RSSVM. Classification maps obtained by the regular SVM and SVM ensemble approaches are shown in Fig. 4.5. From Fig. 4.5, we can see that RoSVM classification maps generate smoother homogeneous regions (Soybeans-min till and Soybeans-no till) than other classification maps.

Results of University of Pavia ROSIS image

Experimental results carried on the University of Pavia ROSIS image (seen in Table A.3) indicated that RoSVM ensembles are superior to the regular SVM and RSSVM ensemble. SVM ensemble approaches with 10 iterations gained higher OAs than the regular SVM. RSSVM significantly improve the accuracy using very limited training set (10 samples per class), the difference between 100 iterations and 10 iterations is 3.15 percentage points. In other cases, there is no increase of OA associated with larger ensembles.

From Table A.3, it is apparent that RoSVM performed well for the limited training For a very limited number of training samples, SVM ensemble presented more accurate than the regular SVM in most classes (Fig. 4.7). In this case, the individual accuracies of Bricks, Gravel and Asphalt acquired by RoSVM-RP B are improved by more than 4 percentage points and the accuracies of Class Bricks, Bare soil and Meadow obtained from RoSVM-PCA are increased by eight percentage points. When the sample size is increased, the difference is significantly reduced. Individual accuracy of Gravel obtained from RSSVM and RoSVM-PCA is even lower than that of the regular SVM.

Results of Pavia Center DAIS image

For Pavia Center DAIS image shown, we choose very limited training samples to examine the performance of RoSVM. The total number of training samples are 18, 36, 54 and 72, respectively. It must be pointed out that, 0.75 × n should be greater than M in RoSVM- 2) SVM ensemble approaches obtained good classification results with 10 iterations and additional iterations cannot signficantly increase the accuracy. Number of features in the subset (M ) has great influence on the overall accuracy. Difference between the maximum and minimum accuracies is more than three percentage points.

3) Regarding the class-specific accuracies, SVM ensembles only improve the classification results of three classes: Trees, Meadow and Bricks. The main improvement of individual class accuracies is on the Class Meadow(improved by 25 percentage points). The accuracies of Class Asphalt, Bitumen and Parking lot are decreased by the ensembles.

Discussion

Experimental results revealed that the superiority of RoSVM ensembles using random feature selection and feature extraction over RSSVM ensemble and the regular SVM. Differences of individual class accuracies between RoSVM and the regular SVM are investigated. The key parameters are also studied. Here, we summarize the answers to the three questions of this study: 1) Is there a significant increase in accuracy of RoSVM compared to random subspace SVM?

Yes, RoSVM ensembles were found to be better than random subspace SVM.

2) How do the two important parameters: ensemble sizes and the number of features in the subset affect the performance of SVM ensembles? Are there recommendation values of the two parameters?

SVM ensembles with 10 iterations can obtain higher overall accuracies than the regular SVM. Larger ensemble size cannot improve the accuracies significantly. Number of features in a subset significantly affected the overall accuracy. For RSSVM ensemble, we recommend the medium value of M , 20%-30% number of the features. For RoSVM, there is no pattern of dependency between M and the ensemble accuracy. As M = 100 and M = 10 worked well for AVIRIS and ROSIS(DAIS) dataset, respectively. Thus, we propose to use the same values in future studies.

3) Is RP better than PCA to rotate the axes for the ensemble of SVM classifiers? It is hard to say. RoSVM-PCA and RoSVM-RP have shown various performance on different training set and experimental settings. However, compared to PCA, RP has two advantages. RP feature extraction technique can obtain the transformation matrix with any dimensions. The other is that RP has lower complexity than PCA.

Summary

In this chapter, rotation-based SVM (RoSVM) ensembles are proposed by combining SVM and MCS in order to improve the classification accuracies of hyperspectral data.

Compared to RSSVM ensemble approach, the proposed RoSVM enhance the diversity by integrating random feature selection and feature extraction techniques. Three types of Random Projections (RP) are introduced into RoSVM. We also investigated the influence of the parameters on the overall accuracy. Experimental results with different contexts and different spectral and spatial resolution on three real hyperspectral datasets demonstrate that RoSVM gives favorable performances in the comparison of the regular SVM and RSSVM, especially for the limited number of training samples. RoSVM with PCA and RP feature extraction methods have shown various performances on different training sets and experimental settings. Despite the positive experimental results offered by RoSVM, the main drawback of RoSVM is its computational complexity, which is higher than those of the regular SVM and RSSVM. Future research should concentrate on the investigation of other fast base learners (e.g. extreme learning machines) in MCSs for the classification of hyperspectral data.

Chapter 5 Rotation Forest with local feature extraction and Markov random fields

Abstract: A new spectral-spatial classifier for hyperspectral data is proposed. Firstly, Rotation Forests with three local feature extraction methods, including neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA) and linearity preserving projection (LPP), are performed to obtain the class probabilities. Secondly, spatial contextual information, which is modeled by Markov random fields (MRF) prior, is used to refine the classification results obtained from Rotation Forests, by solving a maximum a posteriori (MAP) problem using the α-Expansion Graph Cuts optimization method. Experimental results reveal that the combination of RoF with local feature extraction methods and MRF can significantly improve the classification accuracies.

Introduction

It has been demonstrated that spatial information is a crucial component for the analysis of remote sensing images [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF][START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF][START_REF] Tarabalka | Segmentation and classification of hyperspectral images using watershed transformation[END_REF]. The classification result could be very noisy if spatial information is not taken into account. In order to increase classification accuracy and regularize the classification maps, it is critical to combine spectral information with spatial information in the process. Tarabalka et al. [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF][START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF][START_REF] Tarabalka | Segmentation and classification of hyperspectral images using watershed transformation[END_REF] proposed a group of spectral-spatial classification methods to combine a pixel-wise classification result with a segmentation map. The segmentation maps can be obtained from partitional clustering, minimum spanning forest and watershed transformation techniques, respectively. Recently, Markov random fields (MRF) has become a popular tool to exploit the spatial information in the classification of hyperspectral data. MRF is a probabilistic model that is used to integrate spatial information into image classification. Tarabalka et al. [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF] used MRF model as a post-processing scheme to a probability SVM classification map. The classification process is solved by metropolis algorithm based on stochastic relaxation and annealing. Li et al. [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF] combined the class posterior probabilities and spatial information into a combinatorial optimization problem, and solved this problem by graph cuts algorithms. The class posterior probabilities are produced by a sparse multinomial logistic regression (SMLR) classifier, and the spatial information is represented by Markov random field (MRF)-based multilevel logistic (MLL) prior.

In this chapter, we develop new spectral-spatial classifiers, which contain two essential components, Rotation Forests for the pixel-wise classification and Markov random field for spatial regularization, respectively. In particular, Rotation Forests, which create sparse projection matrix using feature extraction and randomly selected subsets of the original features, are used to estimate the class probabilities. Then, spatial contextual information achieved by Markov random field is used to refine the classification results obtained from Rotation Forest classifiers. Finally, the output is produced by solving a maximum a posteriori problem using the α-Expansion Graph Cuts optimization algorithm. The main contribution of the proposed work is to introduce three manifold learning local feature extraction methods, including neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA) and linearity preserving projection (LPP) into Rotation Forests. The results of Rotation Forests with three local feature extraction techniques are further redefined with the help of spatial contextual information, which will be shown to provide a good characterization of content of hyperspectral data. Experimental results are presented on three hyperspectral airborne images recorded by the Airborne Visible/Infrared Imaging Spectrometer (AVIRIS), Reflective Optics System Imaging Spectrometer (ROSIS) and Digital Airborne Imaging Spectrometer (DAIS), re-spectively. They have different spatial/spectral resolutions and correspond to different contexts, hence demonstrating the robustness of the conclusions.

The remainder of this chapter is organized as follows. Section 5.2 presents the proposed models, including Rotation Forests and Markov random field. Section 5.3 describes the experimental results with analysis. Section 5.4 provides the summary and suggestions for future lines of research.

Proposed method

Let X = {x 1 , ..., x N } ∈ R N ×D denote an image of D-dimensional feature vectors; let Y = {y 1 , ..., y N } be an image of pixel; let X train , Y train = {(x 1 , y 1 ) , ..., (x n , y n )} be a total number of n training samples.

The proposed methods based on Rotation Forests and MRF, which are depicted in Fig. 5.1, are composed of three main steps as follows:

1) supervised pixel-wise classification using Rotation Forests.

2) spatial information extraction using Markov random fields.

3) spatial-spectral classification by solving a maximum a posteriori problem computed by the α-Expansion Graph cuts optimization algorithm.

Rotation Forest

In this study, Rotation forests are used for the pixel-wise classification of the hyperspectral data. They construct different versions of the training set by using the following steps: 1) the feature set is divided into several disjoint subsets on which the original training set is projected; 2) a rotation sparse matrix R a i is constructed by performing feature extraction on each subset with the bootstrapped samples corresponding to 75% of the initial training samples; 3) a classifier is built on the features projected by R a i ; 4) the final result is obtained by combining the outputs of the multiple classifiers, repeating the first three steps several times. The details of Rotation Forests are shown in Algorithm 5.1. randomly split the features F into K subsets 3.

For j = 1 : K 4.

select the corresponding features of F j i to compose a training set X train i,j

select a new training set Xtrain

i,j using bootstrap algorithm, whose size is 75% of the original size 6.

transform Xtrain i,j by a certain feature extraction method to get the coefficients v

(1) i,j , ..., v (M k ) i,j 7.
End for 8.

sparse matrix R i is constructed with the above coefficients

R i =         v (1)
i,1 , ..., v

(M 1 ) i,1 0 • • • 0 0 v (1)
i,2 , ..., v

(M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1)
i,j , ..., v

(M K ) i,j         9.
rearrange R i to R a i with respect to the original feature set 10.

obtain the new training samples {X train R a i , Y train } 11.

build the ith classifier L i using {X train R a i , Y train } 12. End for Output: The probability for each class is calculated by the average combination method:

p(y i |x i ) = 1 T T j=1 p(y i |x i R a j )
The excellent performances of Rotation Forests can be attributed to simultaneous improvements in two aspects. One is to promote the diversity within the ensemble by the use of feature extraction on training data and the use of the decision tree, known to be sensitive to variations in the training data. The other is to improve the accuracies of the base classifiers by keeping all extracted features in the training data. It is crucial to notice Step 5 in Algorithm 5.1, the objective of selecting sub-samples is on the one hand to avoid obtaining the same coefficients of the transformed components if the same features are selected, and on the other hand, to enhance the diversity among the member classifiers.

Two important issues for Rotation Forests are the base classifier and the feature extraction method. Classification and regression tree (CART) is adopted as the base classifier in this study, because it is unstable, sensitive to the rotations of the axes and fast [START_REF] Breiman | Classification and regression trees[END_REF]. The main idea of CART is to choose the best split that makes the data in each child nodes are as pure as possible. The Gini index is used to select the best split in this paper [START_REF] Breiman | Classification and regression trees[END_REF]. It should be pointed out that we do not employ feature extraction for dimensionality reduction but for rotation of the axes while keeping all the dimensions.

In the original Rotation Forest model, only PCA is considered. In order to consider neighborhood information around the data points, three local linear feature extraction methods, including NPE [START_REF] He | Neighborhood preserving embedding[END_REF], LLTSA [START_REF] Zhang | Linear local tangent space alignment and application to face recognition[END_REF] and LPP [START_REF] He | Locality preserving projections[END_REF] are introduced into Rotation Forests.

In linear feature extraction, we often assume that there exists a mapping matrix v, which can transform each data point x i to z i . NPE [START_REF] He | Neighborhood preserving embedding[END_REF] aims at preserving the local neighborhood structure on the data manifold. For this purpose, each data point can be presented as a linear combination of its τ neighborhoods. and the combination weights can be computed by minimizing the following objective function:

min i x i - j W ij x j , j W ij = 1, j = 1, 2, ..., τ. (5.1)
where W is a weight matrix with elements W ij having the weight of the edge from node i to node j. The projection matrix v is given by the minimum eigenvectors solution to the generalized eigenvalue problem

X (I -W) (I -W) X v = λXX v. (5.2)
LLTSA uses the tangent space in the neighborhood of a data point to represent the local geometry, and then aligns those local tangent spaces in the low-dimensional space which is linearly mapped from high-dimensional space [START_REF] Zhang | Linear local tangent space alignment and application to face recognition[END_REF]. More precisely, LLTSA defines a promotes solutions in which adjacent pixels are likely to belong to the same class [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF][START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF].

MRF assumes that any pixel is independent to others outside its defined neighborhood. The Hammersely-Clifford theorem shows that a random field is an MRF only if it follows a Gibbs distribution [START_REF] Besag | Spatial interaction and the statistical analysis of lattice systems[END_REF]. Therefore, the prior model has the following structure:

p(Y) = 1 Z e (-(i,j)∈ς Vς (Y)) , (5.8)
where (i, j) ∈ ς denotes that pixel i and pixel j are neighbors.

The MLL has been a popular spatial contextual model, in which the clique energy is defined as [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF][START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]:

-V ς (y i , y j ) =      µ if y i = y j 0 otherwise (5.9)
where µ is a positive number that controls the degree of smoothness. Thus, Eq. (5.8) can be rewritten as follows:

p(Y) = 1 Z e µ (i,j)∈ς δ(y i -y j ) , (5.10)
where δ(•) is the Dirac unit impulse function.

MAP labeling

The image classification task can then be formulated as a maximum a posterior (MAP) problem, for which maximizing the posterior p(Y|X) gives a solution, which is equivalent to maximizing p(X|Y)p(Y). It is possible to impose spatial contextual constraints by modeling p(Y) with an MRF [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF][START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. Assuming conditional independence of the features given the labels, p(X|Y) can be formulated as [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF]:

p(X|Y) = N i=1 p(x i |y i ) (5.11) = N i=1 p(y i |x i )p(x i ) p(y i ) .
(5.12)

Then, the posterior distribution can be rewritten as follows:

p(Y|X) ∝ p(X|Y)p(Y) p(X) (5.13) ∝ 1 p(X) N i=1 p(x i |y i )p(Y) (5.14) ∝ 1 p(X) N i=1 p(y i |x i )p(x i ) p(y i ) p(Y) (5.15) ∝ N i=1 p(x i ) p(X) N i=1 p(y i |x i ) p(y i ) p(Y).
(5.16)

In the proposed model,

N i=1 p(x i ) p(X)
is a factor not depending on y. The densities p(y i ) are assumed to be equally distributed. The MAP problem can then be defined as follows:

arg max log p(y i |x i ) + log p(Y) = arg min -log p(y i |x i ) -log p(Y) (5.17)
= arg min

   -log p(y i |x i ) -µ (i,j)∈ς δ(y i -y j )    .
(5.18)

An important issue of MRF-based approaches is to compute the global minimum of the objective function. MRF-based objective functions such as that in the above equations are highly non-convex. The existence of local minima causes considerable difficulties in finding the global minima in an intractably vast search space.

Various combinatorial optimization methods have been proposed to solve this problem. In this chapter, we resort to the α-Expansion graph-cut-based algorithm. This method yields good approximations to the MAP segmentation and is quite efficient from a computational viewpoint, with practical computational complexity [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF][START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. It should be noticed that α-Expansion graph-cut-based algorithm can exactly solve the binary class problem. Since more than two different classes are present, multi-class problem cannot be solved exactly but an approximate solution within a known factor of the optimal solution is found.

Experimental results and analysis

In this section, three real hyperspectral data with several different characteristics (sensors, areas, dimensions and spatial resolutions) were used for the experiments. The detailed descriptions of the three hyperspectral images and the corresponding results and analysis are shown in the next subsections.

Indian Pines AVIRIS image

In the first experiment, we investigated the performances of Rotation Forests with local feature extraction and MRF using different numbers of training samples. In this experiment, M and T are fixed to be 10, µ is fixed to be 4 and τ is fixed to be 12. Table 5.1 shows the average of overall accuracies obtained from the proposed methods using different numbers of training samples. The standard deviations of the proposed methods are also given in the table. As can be seen in Table 5.1, the overall performances of RoF-NPE, RoF-LLTSA and RoF-LPP are better than those of RoF-PCA. With the help of spatial contextual information, the combination of Rotation Forests and MRF significantly outperforms the Rotation Forests, which use the spectral information only. Moreover, it is clear that Rotation Forests with local feature extraction methods have more stable performances than RoF-PCA in most cases.

In the second experiment, the dependence of the classification accuracies on different parameters are studied. In the proposed model, there are four parameters: ensemble size (T ), number of features in the subset (M ), number of neighbors (τ ) considered by the local feature extraction methods and the regularization parameter (µ). Following our previous study [START_REF] Xia | Hyperspectral remote sensing image classification based on rotation forest[END_REF], the value T =10 is recommended. A larger size of T in the Rotation Forests has non-significant effects on the overall accuracy, while increasing the computation time. Thus, T is fixed to be 10 in all experiments. The regularization parameter (µ) is empirically set to be 4. The impacts of different values of M on the classification performances are presented in Table 5.2. The number of training samples per class is 20, and the total number of training samples is 320. τ is fixed to be 12. The classification results are significantly improved when larger values of M are used. The main reason is that an insufficient number of features in a subset (low values of M ) could not provide a reliable sparse rotation matrix based on the above feature extraction methods, resulting in a decrease of the classification performance. Table 5.3 shows the impact of different values of τ on classification accuracies. From Table 5.3, it is found that the classification performances indeed depend on the settings of τ . The optimal values of τ are between 8 and 12.

In order to compare the class-specific accuracies and overall accuracies precisely, we have chosen 30 pixels per class from the available ground truth (a total size of 423 pixels) as the training set. We also used the two standard classifiers for comprehensive comparisons: SVM and logistic regression via variable splitting and augmented Lagrangian algorithm (LORSAL), respectively [START_REF] Li | Spectral-spatial classification of hyperspectral data using loopy belief propagation and active learning[END_REF]. Global and class-specific accuracies achieved by all the compared algorithms are listed in Table 5.4. As shown in Table 5.4, RoF-PCA

University of Pavia ROSIS image

The average accuracies over 10 independent runs and the corresponding standard deviations of the proposed methods using different numbers of training samples are featured in Table 5.5. The sensitivity analysis of parameters, M and τ , are presented in Table 5.6 and Table 5.7, respectively. Experimental results reveal a number of interesting facts: 1) Rotation Forests with local feature extraction methods (e.g. NPE) provide better and more stable performances than RoF-PCA; 2) the combination of Rotation Forests and MRF significantly improves the classification accuracy when compared to the Rotation Forests; 3) classification accuracies of the Rotation Forests decrease when the value of M increase, but the classification performances of the combination of Rotation Forest and MRF are improved. 4) the optimal values of τ are between 8 and 12.

Furthermore, we test the classification performances of the proposed spectral-spatial classification algorithms using the whole training set. Table A.4 summarizes the overall accuracies, average accuracies and class-specific accuracies. Fig. A.6 presents the classification maps. We also list the classification results of the pixel-wise classifiers: SVM and LORSAL, and the following spectral-spatial classifier: LORSAL-MRF in Table A. 4. The results of SVM, LORSAL (LORSAL-MRF) reported in the table are taken from [START_REF] Tarabalka | Multiple spectral-spatial classification approach for hyperspectral data[END_REF] and [START_REF] Li | Spectral-spatial classification of hyperspectral data using loopy belief propagation and active learning[END_REF], respectively. As can be seen in Table A.4, the OAs of the four Rotation Forests are all higher than those of SVM and LORSAL. The global and most of the class-specific accuracies (except the Class Gravel ) increase thanks to the proposed methods. The Class Gravel is wrongly classified and confused with the similar Class Bricks. Therefore, when we apply MRF on a pixel-wise classification result, even more pixels of the Class Gravel are wrong assigned to the Class Bricks. That leads to a lower accuracy than that of the classifier using spectral information only. The best global accuracies are achieved by RoF-LPP-MRF. The corresponding classification map is significantly more accurate than any other classification map, according to the results of McNemar's test. In this case, the overall and average accuracies are improved by 7.39 and 4.42 percentage points respectively, compared to RoF-LPP. The use of LPP feature extraction also leads to the highest accuracies for most of the classes (five out of nine). The OAs of the proposed four schemes are all higher than standard spectral-spatial classifiers, such as SVM-Watershed segmentation [START_REF] Tarabalka | Segmentation and classification of hyperspectral images using watershed transformation[END_REF], SVMMSF [START_REF] Tarabalka | Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers[END_REF], SVMMRF-NE [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF], SVMMRF-E [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF], LORSAL-MRF [START_REF] Li | Spectral-spatial classification of hyperspectral data using loopy belief propagation and active learning[END_REF]. Corresponding results are not listed here, but can be found in the original reference.

The sensitivity to the number of features in a subset M is explored using the whole

Pavia Center DAIS image

Rotation Forest classifiers with PCA, NPE, LLTSA and LPP feature extraction techniques are performed on the Pavia Center DAIS image using the whole training set. Table 5.9

gives the classification accuracies. The OAs and AAs of four rotation forests are all higher than those of SVM and LORSAL. MRF regularization with µ = 4 was performed on the pixel-wise classification result derived from Rotation Forest ensembles. In Table 5.9, the results for the proposed methods with different feature extraction algorithms are presented. From Table 5.9, it can be seen that Rotation Forests with four feature extraction methods achieved excellent global accuracies. Again, the MRF methods perform better than the spectral-based approaches. Among them, the use of LLTSA achieves the best performances. It is consistent with the characteristic of LLTSA, which indicates that LLTSA can provide more local information than PCA. In terms of class-specific accuracies, the main improvement is achieved by the Class Shadows. The other classes are classified equally accurately.

Summary

The large number of spectral channels in a hyperspectral image increases the potential of discriminating different materials and structures in a scene. However, the huge volume of hyperspectral data often leads to challenges in image analysis. The success of hyperspectral remote sensing image classification does not only depend on the high precision pixel-wise classifier, but also needs the incorporation of the spatial information into the classifier.

In this paper, we developed new spectral-spatial classification methods, suited for hyperspectral remote sensing image. Rotation Forest are applied as the spectral classifier for hyperspectral data. Different feature extraction methods have been investigated for the construction of Rotation Forests. It is shown that, with the help of local information that obtained by NPE, LLTSA and LPP, classification accuracies can be improved. Furthermore, Rotation Forests with spatial contextual information using MRF were then proposed. This strategy can further significantly improve the performances. The proposed classification methodology succeeded in taking advantage of the spatial and spectral information simultaneously. The sensitivity of the parameters in the proposed methods was also investigated.

Future studies will focus on the integration of Rotation Forests with other spatial information regularizations, the use of semi-supervised feature extraction algorithms and the combination of ensemble learning and active learning.

Introduction

Recent studies demonstrated that spectral-spatial approaches can provide more accurate classification results by integrating the spatial and spectral information together [START_REF] Fauvel | Advances in spectral-spatial classification of hyperspectral images[END_REF]. This great interest is due to the fact that spatial features are discriminant features that can well complement the spectral features. Different approaches can be used to extract spatial features [START_REF] Benediktsson | Classification and feature extraction for remote sensing images from urban areas based on morphological transformations[END_REF][START_REF] Pesaresi | A new approach for the morphological segmentation of high-resolution satellite imagery[END_REF][START_REF] Bau | Hyperspectral region classification using a threedimensional gabor filterbank[END_REF][START_REF] Tsai | Feature extraction of hyperspectral image cubes using threedimensional gray-level cooccurrence[END_REF][START_REF] Qian | Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features[END_REF]. Among them, mathematical morphology (MM) is a powerful tool for the analysis and processing of geometrical structures in the spatial domain [START_REF] Serra | Image Analysis and Mathematical Morphology[END_REF]. Pesaresi and Benediktsson introduced the morphological profile (MP) for classifying very high spatial resolution images using a sequence of geodesic opening and closing operations [START_REF] Pesaresi | A new approach for the morphological segmentation of high-resolution satellite imagery[END_REF].

The derivative of the morphological profile (DMP) was also defined in their study. Furthermore, Benediktsson et al. proposed the extended morphological profile (EMP), in which an MP is computed on each component after reducing the dimensionality of the data [START_REF] Benediktsson | Classification of hyperspectral data from urban areas based on extended morphological profiles[END_REF]. The first few components and the EMP are stacked together and then classified by a neural network. The main drawback of the method in [START_REF] Benediktsson | Classification of hyperspectral data from urban areas based on extended morphological profiles[END_REF] is that it is constructed for classification of urban structures and it cannot fully use the spectral information of hyperspectral data [START_REF] Fauvel | Spectral and spatial classification of hyperspectral data using svms and morphological profiles[END_REF]. Fauvel et al. developed a spectral and spatial fusion methods based on EMP and the original hyperspectral data to overcome this problem [START_REF] Fauvel | Spectral and spatial classification of hyperspectral data using svms and morphological profiles[END_REF]. In the works of Dalla Mura et al. [START_REF] Dalla Mura | Morphological attribute profiles for the analysis of very high resolution images[END_REF][START_REF] Dalla Mura | Extended profiles with morphological attribute filters for the analysis of hyperspectral data[END_REF], attribute profiles (APs) [START_REF] Breen | Attribute openings, thinnings, and granulometries[END_REF] were proposed for extracting additional spatial features for the classification of remote sensing imagery, extending the MP and EMP concepts. APs were proved to extract more reliable spatial features than MPs in the classification of high-resolution images. Since then, APs and its extensions have been widely used for the classification and change detection of multi/hyper-spectral and LiDAR data. to classify EMAPs extracted from hyperspectral data [START_REF] Song | Remotely sensed image classification using sparse representations of morphological attribute profiles[END_REF].

From the above literature review, it can be seen that when the EMAPs are used for hyperspectral data classification, two strategies are often adopted:

• applying feature selection/extraction [START_REF] Pedergnana | A novel technique for optimal feature selection in attribute profiles based on genetic algorithms[END_REF] or advanced classifier to EMAPs [START_REF] Song | Remotely sensed image classification using sparse representations of morphological attribute profiles[END_REF];

• integrating EMAPs with spectral information to formulate the composite kernel for kernel-based methods [START_REF] Li | Generalized composite kernel framework for hyperspectral image classification[END_REF].

In this chapter, we propose an advanced classification scheme based on Random subspace (RS) ensembles applied to EMAPs features. Decision tree (DT) and Neural network (NN) are usually adopted as a base learner in RS ensemble because they are unstable because small changes in the training data lead to large change in the results, resulting in generating high diversity within the ensemble. Considering the computational cost, we construct the RS ensembles with two fast learning algorithms: classification and regression tree (CART) and newly NN classifier: extreme learning machine (ELM). EMAPs were generated by the combination of APs and the first several components extracted by PCA. Six classifier ensembles, including RSDT, RF, RoF, RoRF, RSELM and RoELM, are considered as shown in Table A.5. The novelty of this work consists in:

• Rotation Random Forest is introduced in the field of hyperspectral remote sensing;

• Random/Rotation subspace with ELM are proposed for the classification of hyperspectral data;

• the comparative analysis of the six aforementioned methods is performed in the classification of both spectral information and spatial information (EMAPs).

In particular, the performances in a scenario with limited training samples and high input dimensions and the computational complexity are investigated in this chapter. It should be noted that the spectral information and EMAPs are directly applied to the Random subspace ensemble methods without any preprocessing technique (feature extraction/selection and whitening).

The overall structure of this chapter takes the form of five sections, including this introductory section. Section II presents an introduction of decision tree and its ensembles. The proposed ELM ensemble methods is detailed in Section III. The main description of EMAPs is presented in Section IV. We report the experimental results in Section V. Section VI contains the conclusion of the presented work and its perspectives.

Decision tree and its ensembles

Let X train , Y train = {(x 1 , y 1 ) , ..., (x n , y n )} be a set of labeled samples, where x i ∈ R D .

1 Let F be the set of D features. In order to construct an RS ensemble, we collect T classifiers based on the subsets of the original features. Each feature set in the ensemble defines a subspace of features of cardinality M and a classifier is trained using the whole training samples [START_REF] Ho | The random subspace method for constructing decision forests[END_REF]. The final result is generated by a majority voting rule. Two parameters, including the ensemble size T and the cardinality of the feature set M , are required in the RS ensemble.

Decision tree

Decision tree is a non-parametric supervised learning algorithm used for classification and regression [START_REF] Quinlan | Induction of decision trees[END_REF]. It is composed of a root node, a set of internal nodes (split) and a set of terminal nodes (leaves). In classifications, a root node and each internal node has a splitting decision and splitting features associated with it. Class labels can then be assigned to the leaves. The creation of a DT from training samples involves two phases. At first, a splitting measure and a splitting attribute should be chosen. In the second phase, the records among the child nodes are split based on the decision made in the first phase. This process is applied recursively until a stopping criterion is met [START_REF] Narayanan | Interactive presentation : An fpga implementation of decision tree classification[END_REF]. Then, the DT can be used to predict the class label of a new sample. The prediction process starts at the root, and a path to a leaf is traced by using the splitting decision at each internal node. The class label attached to the leaf is then assigned to the new sample [START_REF] Narayanan | Interactive presentation : An fpga implementation of decision tree classification[END_REF].

A critical component of the decision tree induction process is the selection of the split. Different algorithms uses various metrics to split the nodes. The most widely used splitting criteria relies on the minimization of the Gini index of the splits [START_REF] Rokach | Data Mining with Decision Trees : Theroy and Applications[END_REF].

Decision tree ensembles

Random subspace with DT

The RS ensemble, introduced by Ho [START_REF] Ho | The random subspace method for constructing decision forests[END_REF], was proposed for constructing multiple decision trees. The objective of the RS ensemble is to sample a feature set of low dimensionality subspaces from the whole original high dimensional features space, then construct a classifier on each smaller subspace, and finally apply a majority voting rule for the final decision.

Random Forest

Random Forest, developed by Breiman [START_REF] Breiman | Random forests[END_REF], combines Bagging [START_REF] Breiman | Bagging predictors[END_REF] and Random subspace [START_REF] Ho | The random subspace method for constructing decision forests[END_REF] together to produce the decision tree ensemble. Random Forest is a particular implementation of bagging in which each model is a random tree. A random tree is grown according to the CART algorithm with one exception: for each split, only a small subset of features of randomly selected splits is considered and the best split is chosen from this subset. Since only a portion of the input features is used to split and no pruning on the tree is done, the computational complexity of Random Forest is relatively light [START_REF] Breiman | Random forests[END_REF]. The computing time is approximately of T √ M nlog(n), where T , M and n represents the number of classifiers, features in a subset and training samples, respectively.

Rotation Forest

Rotation Forest is a recently proposed ensemble method for building classifier ensembles using independently DTs with a different set of extracted features [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF]. The main heuristic of Rotation Forest is to apply feature extraction and to subsequently reconstruct a full feature set for each classifier in the ensemble. To do this, the feature space is randomly split into K subsets, then principal component analysis (PCA) is applied to each K subsets and a new set of M linear extracted features is constructed by all principal components. The data is transformed linearly into the new feature space. Individual classifier is trained with this data set. Different splits of the features will lead to different extracted features, thereby contributing to the diversity introduced by the bootstrap sampling.

Rotation Random Forest

Rotation Random Forest is a variant of Rotation Forest, which uses Random Forest as the base classifier instead of decision tree [START_REF] Stiglic | Rotation of random forests for genomic and proteomic classification problems[END_REF]. We expect that Rotation Random Forest could improve the performance of Random Forest by introducing further diversity using a feature extraction approach into the ensemble. The base classifiers in Rotation Random Forest are less diverse but more accurate with respect to Rotation Forest and this could be beneficial for the ensemble of Random Forest.

ELM and its ensemble

NN is also used as the base learner for ensemble learning. However, the main drawback of conventional NN is the high computation complexity. To address the shortcoming, Extreme learning machine (ELM) was proposed for the learning of generalized single hidden layer feed-forward neural networks (SLFNs) without tuning the hidden layers [START_REF] Huang | Extreme learning machine : a new learning scheme of feedforward neural networks[END_REF][START_REF] Huang | Extreme learning machine : theory and applications[END_REF].

Extreme learning machine

For generalized SLFNs, the output function of ELM is defined as:

f (x i ) = δ j=1 β j h j (x i ) = h(x i )β (6.1)
where, β = [β 1 , β 2 , ..., β δ ] is the vectors of weights between the hidden layer of δ nodes and the output node and h(x) = [h 1 (x), h 2 (x), ..., h δ (x)] is the vector of hidden layer. Actually, h(x i ) is the feature mapping from the D-dimensional input space to the δdimensional hidden-layer feature space. The standard SLFNs can approximate these n samples with zero error meaning that Σ i f (x i ) -y i = 0. Thus, the n equations can be written compactly as:

Hβ = Y train (6.2)
where, Y train is the target matrix and H is the hidden-layer output matrix:

H =        h(x 1 )
. . .

h(x n )        =        h 1 (x 1 ) • • • h δ (x 1 ) . . . . . . . . . h 1 (x n ) • • • h δ (x n )        (6.
3)

The output weights in equation ( 6.2) are given by the following smallest norm least-squares solution [START_REF] Huang | Extreme learning machine : theory and applications[END_REF]:

β = H + Y train (6.4)
where, H + is the Moor-Penrose generalized inverse of the hidden layer output matrix H.

In ELM, a feature mapping H from input space to a higher dimensional space is needed. The works of [START_REF] Huang | Convex incremental extreme learning machine[END_REF][START_REF] Huang | Enhanced random search based incremental extreme learning machine[END_REF] demonstrated that almost all nonlinear piecewise continuous functions can be used as output functions of the hidden-nodes. In this chapter, the Sigmoid function is adopted as the nonlinear piecewise continuous function:

g(ω, b, x i ) = 1 1 + exp (-(ω • x i + b)) (6.5)
where, {ω j , b j } δ i=1 are randomly generated values that can define a continuous probability distribution (i.e., g = 1). Thus, h(x i ) is defined based on the nonlinear piecewise continuous function g(ω i , b i ):

h(x i ) = [g(ω 1 , b 1 , x i ), ..., g(ω δ , b δ , x i )] (6.6)
The training steps of ELM are listed in Algorithm 6.1. For a new sample x * , calculate the output layer matrix: Compared to conventional feed forward NNs, ELM offer significant advantages such as: 1) fast leaning speed, 2) no need to tune the parameters, 3) better generalization performance, 4) ease of implementation, etc. [START_REF] Huang | Extreme learning machine : a new learning scheme of feedforward neural networks[END_REF][START_REF] Huang | Extreme learning machine : theory and applications[END_REF].

h(x * ) = [g(ω 1 , b 1 , x * ), ..., g(ω δ , b δ , x * )].

ELM ensembles

ELM decreases the learning time dramatically with respect to a conventional neural network due to the random selection of weights and biases for hidden nodes [START_REF] Huang | Convex incremental extreme learning machine[END_REF][START_REF] Huang | Enhanced random search based incremental extreme learning machine[END_REF]. However, the parameters in ELM are not optimized and not able to incorporate prior knowledge of the inputs, thus the generalization error might increase. Consequently, we propose constructing ensemble of several predictors on the training set using RS method in which the parameters in each predictor are randomly selected. In this work, two implementations of ELM ensemble, Random subspace-and Rotation Subspace-based are developed for hyperspectral image classification.

Random subspace with ELM

Given a training set, the parameters of ELM (activation function and number of hidden nodes), the number of features in a subset (M ) and the number of classifiers (T ), the RS with ELM algorithm can be summarized by the following three steps: 1) Generate M features in a subset from the entire feature set for T times; 2) Apply these features to ELM classifier and obtain T classification results; 3) Produce the final classification map by combining T classification results using a majority voting rule.

Rotation subspace with ELM

The proposed rotation based ELM is presented in Algorithm 6.2. The main steps of Rotation-based ELM can be summarized as follows:

• the feature space is divided into K disjoint subspaces.

• principal component analysis (PCA) is performed on each subspace with the bootstrapped samples of 75% of original training set.

• the new training set, which is obtained by rotating the original training set, is treated as the input of the individual classifier.

• the final result is generated by combing the individual classification results using a majority voting rule.

The diversity in RoELM is promoted in three aspects: 1) random selection of features; 2) feature extraction applied to the selected features using bootstrap sampling technique; 3) random selection of parameters in each ELM classifier.

EMAPs

Mathematical morphological is a powerful framework for the analysis of spatial information in remote sensing imagery [START_REF] Pesaresi | A new approach for the morphological segmentation of high-resolution satellite imagery[END_REF][START_REF] Serra | Image Analysis and Mathematical Morphology[END_REF]. In particular, attribute profiles have been successfully applied to produce classification maps of remote sensing data [START_REF] Dalla Mura | Morphological attribute profiles for the analysis of very high resolution images[END_REF][START_REF] Dalla Mura | Extended profiles with morphological attribute filters for the analysis of hyperspectral data[END_REF]. A sequence of attribute filters (AFs) are applied to a scalar image to obtain APs. AFs are randomly split the features F into K subsets F i j 3:

for j = 1 : K do transform Xtrain i,j

to get the coefficients v

end for

8:
sparse matrix R i is composed of the above coefficients

R i =         v (1)
i,1 , ..., v

(M 1 ) i,1 0 • • • 0 0 v (1)
i,2 , ..., v

(M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1)
i,j , ..., v

(M K ) i,j         9:
rearrange R i to R a i with respect to the original feature set build ELM classifier L i using X train R a i , Y train 12: end for 13: Add the classifier to the current ensemble, L = L ∪ L i . connected operators, that is they process a gray-level image by keeping or merging their connected components at different gray levels.

Denoting respectively with φ and γ an attribute thickening and thinning based on the arbitrary criterion P λ . An AP of an image X is obtained by applying several attribute thickening and thinning operators with a given sequence of thresholds {λ 1 , λ 2 , .., λ } for the predicate P as follows [START_REF] Dalla Mura | Extended profiles with morphological attribute filters for the analysis of hyperspectral data[END_REF]:

AP (X) = φ λ (X), φ λ -1 (X), ..., φ λ 1 (X), X, γ λ 1 (X), ..., γ λ -1 (X), γ λ (X) (6.7)
AP deals with only one spectral band. If we apply the full spectral bands of hyperspectral data to extract APs, the dimensionality of APs becomes extremely high. In order to address the problem, Dalla Mura et al. proposed to consider few of the first several principal components of the hyperspectral data [START_REF] Dalla Mura | Extended profiles with morphological attribute filters for the analysis of hyperspectral data[END_REF]. However, any feature extraction and selection could be also used [START_REF] Pedergnana | Classification of remote sensing optical and lidar data using extended attribute profiles[END_REF]. Thus, the expression of an EAP computed on the first C PCs from the original hyperspectral data [START_REF] Dalla Mura | Extended profiles with morphological attribute filters for the analysis of hyperspectral data[END_REF] is given by: EAP = {AP (P C 1 ), AP (P C 2 ), ..., AP (P C C )} (6.8)

An EMAP is composed of m different EAPs based on different attributes {a 1 , a 2 , ..., a m }:

EM AP = EAP a 1 , EAP a 2 , ..., EAP am (6.9)

where, EAP a = EAP a / {P C 1 , P C 2 , ..., P C C }.

Although a wide variety of attributes can be used to construct APs, only the area and standard deviation attributes are considered in this study. Fig A .8 presents the general steps of the construction of EMAPs using the area and standard deviation attributes. Firstly, PCA is performed on the original hyperspectral image and the first components with cumulative eigenvalues over 99% are kept. Then, APs with attribute and standard deviation attributes are computed on the first retained features and the output features are concatenated into a stacked vector to construct an EMAP.

According to [START_REF] Pedergnana | A novel technique for optimal feature selection in attribute profiles based on genetic algorithms[END_REF], λ s is initialized so as to cover a reasonable amount of deviation in the individual feature, which is mathematically given by:

λ s (F i ) = µ i 100 {τ min , τ min + σ s , τ min + 2σ s , ..., τ max } (6.10)
where, F i is the ith feature of the image and µ i is the mean value of the ith feature. The values of τ min , τ max and σ s are 2.5%, 27.5% and 2.5%, respectively, which leads to 11 thinning and 11 thickening operations. The construction of the attribute area is established in the following:

λ a (F i ) = 100 ν {α min , α min + σ a , α min + 2σ a , ..., α max } (6.11)
where, ν is the spatial resolution of the remote sensing image. The values of α min and α max are 1 and 14. The EAP for the area attribute contains 14 thinning and 14 thickening operations for each feature.

Experimental results and analysis

In this section, the proposed approaches are evaluated using two real hyperspectral datasets: Indian Pines AVIRIS and University of Pavia ROSIS images. Table 6.2 and Table 6.3 give the class name and the number of ground truth of AVIRIS and ROSIS hyperspectral data. Two individual classifiers, including DT and ELM, and six ensemble learning methods, including RSDT, RF, RoF, RoRF, RSELM and RoELM are applied to classify the spectral information and EMAPs of hyperspectral data. A non-parametric decision tree learning technique: classification and regression tree, are used to construct the decision tree ensemble [START_REF] Breiman | Classification and regression trees[END_REF]. The impurity measure used in selecting the variable in CART is Gini index. The Sigmoid function is selected as the functions of hidden-nodes.

In this work, only the first four components resulting from PCA (which comprise more than 99% of the data variance) were used, and the EMAPs consisted of 204 features.

We used the following measures to evaluate the performance of different classification methods:

• Overall accuracy (OA): the percentage of correctly classified samples.

• Average accuracy (AA): average percentage of correctly classified samples for individual class.

• Kappa coefficient (κ): the percentage agreement corrected by the level of agreement that could be expected to chance alone.

Results of Indian Pines AVIRIS image

Table 6.4 and 6.5 present the classification results obtained for the individual classifiers and RS ensemble methods using different size numbers of training samples when the spectral and EMAPs are used as the input, respectively. Average accuracies for each classifier are also given in the parentheses. The parameters used for each ensemble classifier are shown in Table 6.6. As shown in Table 6.4 and 6.5, the RS ensemble methods exhibit the potential to improve the classification performance by using both spectral and spatial information. The proposed RoELM outperforms ELM, RSELM and other decision tree ensemble in terms of achieving higher classification accuracies in all cases. With the help of promoting diversity using feature extraction approaches, Rotation subspace classifiers, including RoF, RoRF and RoELM, are superior to the ensemble classifiers of RF, RSDT and RSELM. In order to show the performance of RS ensemble methods under different training conditions and scenarios, in the second experiment, we evaluated the classification accuracies of the RS ensemble approaches using a fixed number of training samples in which 10% of the labeled samples per class have been used for training (a total 1036 samples) and the remaining labeled samples are used for testing. Table 6.7 and 6.8 provide the OAs, AAs, κ and class-specific accuracies obtained from the invidual and ensemble classifiers using spectral information and EMAPs, respectively. The processing times in seconds are also included for reference.

It can be seen from the results in Table 6.7 and Table 6.8 that the performance of ELM is superior to CART in terms both of testing accuracy and learning time. When the spectral information is treated as input, RoELM and RoF share the top position. The OAs (AAs) of the two methods are 84.70% (75.33%) and 81.31% (73.62%), higher than those of other methods. Class 9 produces bad results in all classifiers, the reason may be that there is insufficient information provided by class 9 using only 2 samples in the training. Compared to the results reported in Table 6.7, the classification accuracies in Table 6.8 involving the spatial information are much better than those obtained only with the spectral information, demonstrating that EMAP can accurately model spatialcontextual information in all cases. For the EMAPs as the input for this scene, RF and RSELM is slightly better than RoF and RoELM. Among them, RoRF yields the highest OA, AA and κ. Feature extraction techniques in the processing of RoF, RoRF and RoELM classifiers will lead to longer computation time than those of RSDT, RF and RSELM. The computational complexity of ELM ensemble is lower than those of DT ensemble. The computation time of RF ensemble is extremely low (less than 1s). Fig 6 .2 presents the classification maps (one of the ten Monte Carlo runs) obtained for the individual and ensemble learning methods with 10% labeled sample as the training samples in Table 6.7 and Table 6.8. As can be seen from the two figures, RS ensemble can improve the classification performance and reduce the classification noise. The classification methods based on EMAPs spatial features result in classification maps with more homogeneous regions when compared to the classification result using spectral information.

More classification results for the Indian Pines AVIRIS image based on EMAPs and other spatial-contextual information can be found in [START_REF] Qian | Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features[END_REF][START_REF] Li | Generalized composite kernel framework for hyperspectral image classification[END_REF][START_REF] Bernabe | Spectralspatial classification of multispectral images using kernel feature space representation[END_REF][START_REF] Song | Remotely sensed image classification using sparse representations of morphological attribute profiles[END_REF]. The accuracies in the previous studies are not directly compared with those given in this chapter because different experimental settings (number of features, training and testing samples) are used in these studies. However, it can be concluded RS ensemble with EMAPs performs well compared to other previously proposed classification approaches for hyperspectral data. Random subspace ensembles both with spectral information and spatial information are performed on the University of Pavia ROSIS image. For all the ensemble classifiers, the number of classifiers (iterations) is fixed to be 20. For the RF algorithm, the number of features in a subset is set to the default value √ N of the software package (10 for this scene). For the RSDT and RoELM approach (spectral and spatial information), M is set to 52 and 102, respectively. The number of features in a subset for RoF, RoRF and RoELM used for spectral and spatial information is set to 10 and 3, respectively. The number of hidden nodes in ELM and its ensemble is fixed to be 128. Table 6.9 gives the overall accuracies, average accuracies and class-specific accuracies obtained for different classification algorithms using the entire training set when applied to the spectral information of University of Pavia ROSIS image. The computational time are also given in this table. From this table, it is clear that RoF provided the best results in terms of global and individual class accuracies, followed by RoRF and RoELM. In order to enhance the classification results, RS ensemble with EMAPs are further applied to classify hyperspectral data and the global and class-specific accuracies are reported in Table A.6. It can be seen from Table A.6 that the classification results with EMAPs significantly outperformed those only considering spectral information. All the RS ensemble yields the highest precision results. The proposed RSELM and RoELM outperform ELM, RoRF and other ensemble methods in terms of achieving higher global and class-specific accuracies. Rotation subspace-based classifiers (RoRF, RoF and RoELM) generate more accurate results than those of RF, RSDT and RSELM because they introduced more diversity within the ensemble. Concerning the computational load, different observations can be made as in the former experiments. The computational cost of ELM and its ensembles is higher than those of DT and DT ensembles, because of the large size of the dataset. The spectral-spatial methods are less computationally efficient than the spectral-based methods due to the higher dimensionality of input features, but provide, in turn, higher accuracies. For illustrative purpose, In addition, Table 6.11 presents the comparisons of RSELM EMAPs and RoELM EMAPS against other state of the art spectral-spatial classification methods, such as SVM+Clustering [START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF], MLRsubMLL [START_REF] Li | Spectral-spatial hyperspectral image segmentation using subspace multinomial logistic regression and markov random fields[END_REF], Generalized composite kernels (GCK) [START_REF] Li | Generalized composite kernel framework for hyperspectral image classification[END_REF] and Mixed lasso with 3D-DWT features [START_REF] Qian | Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features[END_REF]. SVM+Clustering approach combines the results of a pixel wise SVM classification and the segmentation map obtained by partitional clustering using majority voting [START_REF] Tarabalka | Spectral-spatial classification of hyperspectral imagery based on partitional clustering techniques[END_REF]. MLRsubMLL is a Bayesian approach, which contains two main steps: 1) the posterior probability distributions are constructed by a subspace MLR classifier, and 2) segmentation, which refers to an image of class labels from a posterior distribution built on the aforementioned classifier and on a multilevel logistic (MLL) prior [START_REF] Li | Spectral-spatial hyperspectral image segmentation using subspace multinomial logistic regression and markov random fields[END_REF]. GCK combines the different kernels built on the spectral and the spatial information of the hyperspectral data without any weight parameters [START_REF] Li | Generalized composite kernel framework for hyperspectral image classification[END_REF]. The classifier in this work is the multinomial logistic regression, and the spatial information is modeled from EMAPs. Mixed lasso with 3D-DWT features is to use structured sparse logistic regression (solved by Mixed lasso) to classify three-dimensional discrete wavelet transform (3D-DWT) [START_REF] Qian | Hyperspectral image classification based on structured sparse logistic regression and three-dimensional wavelet texture features[END_REF]. The results presented in Table 6.11 are obtained using the same training and testing set. From Table 6.11, we can conclude that both RSELM EMAPs and RoELM EMAPS outperform other spatial-spectral classifiers in terms of OA, AA and κ. In particular, RoELM EMAPs gains the highest OA and κ and RSELM EMAPs achieves the highest AA.

Results of University of Pavia image

In order to assess the effectiveness of the RS ensemble for a limited training set, we have randomly extracted a few training samples from the training set. Only 10 samples for each class are used for this experiment. We have repeated the training sample selection and the classification process ten times, and the mean classification results are reported in this chapter. Table 6.12-6.13 provides the overall accuracies, average accuracies and class-specific accuracies obtained for individual and ensemble classifiers using 10 samples per class when the spectral and spatial information of University of Pavia ROSIS image used as the respectively input. The classification results in Table 6.12-6.13 are lower than those in Table 6.9-A. depicts that a large number of hidden nodes may give higher accuracies in testing, but a complex network could also overfit the training data. For instance, the generalization performance decreases when the number of hidden nodes in larger than 512.

In general, these parameters should be selected empirically in particular applications.

Summary

In this chapter, we have developed a novel framework that combines Random subspace ensemble classifiers and EMAPs for spatial-spectral remotely sensed hyperspectral data.

Considering the computational cost, we selected two fast learning algorithms: DT and ELM, to built the RS ensemble classifiers. Several conclusions can be summarized based on our experimental results:

• Although RS ensembles requires more training time than individual classifiers, their performance is superior to the individual classifiers both using spectral and spatial information as input. The computational load for RF algorithm is very low (less than 1s for AVIRIS dataset). In addition, the computation time for RS ensemble can be further reduced by decreasing the ensemble size.

• In most cases, Rotation subspace classifiers, such as RoF, RoRF and RoELM outperform RSDT, RF and RSELM. That is because we introduce more diversity in Rotation subspace ensemble classifiers by using feature extraction and random selection strategies. However, it will lead in increased computational complexity for Rotation subspace approaches.

• In general, ELM and its ensembles can achieve higher accuracies than DT and its ensemble. The computation time of ELM and its ensemble depends on the hidden nodes while the ensemble size and training samples are fixed. Nevertheless, the efficiency of the ELM ensemble could be further improved by choosing smaller size of ensemble or using less hidden nodes.

On the other hand, Random subspace ensemble is likely to have two limitations: 1) the number of features in a subset is required to be provided in advance. The optimal value for this parameter depends on the dataset; 2) the high computation time due to the highdimensionality of the input features. Therefore, our future work is to develop an effective scheme for automatically estimating the number of features in a subset for RS ensemble and a preprocessing step for both spectral and spatial information of hyperspectral data.

Chapter 7

Conclusions and Future Directions

Summary of Contributions

In this thesis, we presented new developments for the problems of remotely sensed hyperspectral image classification based on multiple classifier system (MCS), in which the ultimate goal is to accurately interpret the image data provided by remote sensed hyperspectral imaging instruments in the context of Earth observation application. Our proposed classification techniques exploit the rich spectral information available in this kind of data, while the developed spectral-spatial classifiers make combined use of both the spatial and the spectral information present in the data. Specifically, we have focused on the problems of spectral and spatial hyperspectral image classification, in which some training data is assumed to be available a priori, and particularly addressed some of the most relevant challenges that can be found in this context. These challenges can be described as follows:

• First and foremost, we have addressed the problems related with the high ratio between the high dimensionality of hyperspectral data and the limited availability of training samples in real applications, which poses critical problems for supervised algorithms in order to overcome the curse of dimensionality. In order to address this challenge, we have adopted strategies based on multiple classifier system (MCS) which allowed use to improve the classification performance using multiple learning algorithms.

• Second, we have used a particular class of ensemble classifiers based on the concept of Rotation-based ensemble classifiers, which represent an innovation with regards to previous developments in the hyperspectral imaging literature. These classifiers are able to learn directly the posterior class distributions and deal with the high dimensionality of hyperspectral data in a very effective way. The structure of Rotation-based classifiers is very open and flexible, which can combine any base classifiers. Compared to other ensemble classifiers, Rotation-based ensemble classifiers can gain higher accuracies with the help of ensuring diversity using rotation strategy.

• Third, in addition to the rich spectral information available in the hyperspectral data, the spatial information derived from the neighboring pixels is taken into account. In order to take advantage from this point, our proposed techniques have been designed to exploit spatial and spectral information in order to regularize the pixel-wise classification results.

After describing our general contributions, we describe next the specific contributions in the four main chapters of this thesis. In each case future research lines are identified. • Chapter 6 presented a new general framework to train series of effective classifiers with spatial information for classifying hyperspectral data. The proposed framework is based on the two key observations: 1) the curse of dimensionality and the high feature-to-instance ratio can be alleviated by using Random subspace (RS) ensembles; 2) the spatial-contextual information is modeled by the extended multiattribute profiles (EMAPs). Two fast learning algorithms, decision tree (DT) and extreme learning machine (ELM) are selected as the base classifiers. Six RS ensemble methods, including Random subspace with DT (RSDT), Random Forest (RF), Rotation Forest (RoF), Rotation Random Forest (RoRF), RS with ELM (RSELM) and Rotation subspace with ELM (RoELM), are constructed by the multiple base learners. Experimental results on two real hyperspectral data verify the effectiveness of the RS ensemble methods for the classification of both spectral and spatial information (EMAPs). On the University of Pavia ROSIS image, our approach, both RSELM and RoELM with EMAPs, achieve the state-of-the-art performance, which demonstrates the advantage of our approach. The key parameters in RS ensemble and the computational complexity are also investigated in this study.

Future directions

There are several potential problems with MCS for hyperspectral data classification that would be worthwhile to investigate.

• Adaptive learning. In this thesis, both Random subspace and Rotation subspace can achieve satisfactory classification results. However, the optimal values of M is hard to determine. In the future, we will develop an automatic framework to select the optimal values. Another future work worth being investigated is to adaptively select the optimal sub-classifiers from the original ensemble of classifiers.

• Multiple features learning. When we deal with very high spatial resolution images from urban areas, multiple features are required to fusion together to acquire the accurate classification result. Existing multiple feature learning techniques simply concatenate a pair of different features or combine the classification results in different features. However, how to fuse multiple features in a generalized way is still an open problem. In future studies, we will formulate the multiple feature learning as a general subspace learning problem in which we can use Random subspace, Rotation subspace and Tensor subspace techniques.

• Faster learning. This is a fundamental problem in the context of hyperspectral imaging, in which the dimensionality of the hyperspectral data in future instruments will be ever-increasing and the quick response is required to process the data in many application domains, such as disaster monitoring etc. With these issues in mind, a future research line that we are considering is related with the computationally efficient implementation of the proposed approaches in high performance computing architecture such as multi-core processors.

• Online learning. Ensemble models have largely been learned only in batch mode, in which all of the training samples are learned as a set multiple times. Online learning attempts to learn models by learning each training example only once. Online learning is especially useful when data is arriving continuously so that it is impractical to store data or when the dataset is very large, in which case the running time for multiple runs through the data would be prohibitive.

Propulsion Laboratory de la NASA (JPL) qui fabrique plus tard lâ AIS [START_REF] Vane | Airborne imaging spectrometer : A new tool for remote sensing[END_REF]au profi de celui-là. Le succès de l'AIS conduit à l'élaboration de nouveaux instruments optiques, á savoir AVIRIS [4]. L'imagerie hyperspectrale est liée á l'imagerie multi-spectrale. Ce dernier traite plusieurs images discret, se référant à des bandes étroites, qui couvre tout le spectre du visible à l'infrarouge dâondes longues. Au contraire, les images hyperspectrales traitent d'étroites bandes spectrales avec les intervalles de 10nm à 20nm sur une plage spectrale continue (VIS, NIR, MIR, MWIR et LWIR) et produisent le spectre de tous les pixels de la scène. Capteurs hyperspectrales typiques contiennent 64 à 250 bandes spectrales qui couvrent une plage spectrale de 400nm à 2500nm, avec une résolution spatiale variant de 1-20 m/pixel et de 14.5-100 m/pixel pour les capteurs aéroportés et satellitaires, respectivement. La télédétection hyperspectrale demeure bien pratique dans grand nombre de domaines, tels que la classification, lâunmixing, la détection de cible, etc [START_REF] Landgrebe | Multispectral land sensing ; where from, where to ?[END_REF][START_REF] Kalacska | Hyperspectral Remote Sensing of Tropical and Sub-Tropical Forests[END_REF].

A.1.1 Classification d'image hyperspectrale

Pixels à hyperspectrale avec ou sans compte tenu de leurs liens spatiaux sont généralemrnt défin comme base spectrale [START_REF] Landgrebe | Signal Theory Methods in Multispectral Remote Sensing[END_REF] et spectrale spatiale classification [START_REF] Fauvel | Advances in spectral-spatial classification of hyperspectral images[END_REF], respectivement.

Lorsque nous faisons la classifcation au travers de chiffres données hyperspectrales, le premier et le plus important inconvénient consiste à la malédiction de la dimension [START_REF] Hughes | On the mean accuracy of statistical pattern recognizers[END_REF]. la précision de la classification augmente progressivement au début, avec l'augmentation des dimensions, mais diminue considérablement lorsque le numéro de la bande atteint une certaine valeur. Les caractéristiques de données hyperspectrales sont généralement redondants et fortement corrélés. Afin de resodre ce problème, une stratégie courante consiste à utiliser des techniques de sélection/d'extraction de caractéristiques.

Lorsque le nombre de dimensions de données hyperspectrale est réduit en utilisant des techniques de sélection/ d'extraction de caractéristiques, des classificateurs (tels que la probabilité gaussienne maximale, minimale classificateur de distance, etc.) peuvent être utilisés. Ici, il faut faire attention car les SVM demeure le classificateur le plus largement utilisé pour les données hyperspectrales au cours des dernières années [START_REF] Vapnik | The Nature of Statistical Learning Theory[END_REF][START_REF] Huang | An assessment of support vector machines for land cover classification[END_REF][START_REF] Melgani | Classification of hyperspectral remote sensing images with support vector machines[END_REF].

Si l'information spatiale de l'image hyperspectrale ne sont pas considérés, la carte thématique, qui comprend le bruit de classification sel et le poivre, l'air très bruyant. Par conséquent, il est essentiel de prendre en compte l'information spatiale. Au meilleur de nos connaissances, spectral-spatiale algorithmes de classification peuvent être divisés en plusieurs groupes, qui sont détaillées dans le tableau A.1. • Les information spectrale. Une description détaillée spectrale de chaque pixel est assurée par le grand nombre de bandes spectrales.

• Les information spatiales. Une information contextuelle spatiale est considéré sur la base des structures spatiales observées dans une scène.

Une attention particulière est accordée à la possibilité d'exploiter MCS pour l'amélioration de la qualité des cartes de classification, en raison de petite enquête effectuée dans la littérature sur ce sujet [START_REF] Benediktsson | Multiple classifier systems in remote sensing : from basics to recent developments[END_REF][START_REF] Du | Multiple classifier system for remote sensing image classification : A review[END_REF]. Un point de ce travail important est la volonté d'obtenir des résultats qui pourraient être de nature générale que possible, donc pas dépendant de scénario. Afin d'atteindre un tel résultat, un grand nombre d'ensembles de données hyperspectrales sont analysés, fournies par trois capteurs différents, couvrant quartier de l'université, zone urbaine et les zones agricoles.

MCS utilise un ensemble de machines d'apprendre à apprendre des solutions partielles pour un problème de classification et l'intégration de ces solutions d'une certaine manière à construire une solution définitive ou complète au problème initial [START_REF] Kuncheva | Combining Pattern Classifiers : Methods and Algorithms[END_REF][START_REF] Brown | Ensemble learning[END_REF]. MCS peut tirer parti des points forts de chaque méthode, tout en évitant ses faiblesses. En outre, il existe d'autres motivations à combiner plusieurs classificateurs :

• Eviter les conditions initiales arbitraires mais importantes, par exemple, celles-ci contiennent les paramètres des classificateurs individuels.

• Introduire certaines randomisations dans le processus de formation afin d'obtenir différentes alternatives qui peuvent être combinées pour améliorer les résultats obtenus par les classificateurs.

• Utiliser les méthodes de classification complémentaires pour améliorer l'adaptation dynamique et la flexibilité.

A.1.3 Présentation de la thèse

Le reste de cette thèse se compose de deux parties principales. Un chapitre dans lequel nous introduisons et étudions le fond et les travaux connexes sur MCS est suivie par les deux parties principales de la thèse. La première partie tente de développer des méthodes de classification basé spectrales par la résolution de la malédiction de problème de dimensionnalité basé sur MCS (au chapitre 3 et 4). La deuxième partie vise à développer le classificateur MCS avec l'information spatiale pour la classification des données hyperspectrales (au chapitre 5 et 6). Ci-dessous, nous présentons brièvement le contenu de chacun des chapitres de la thèse.

• Le chapitre 1 présente les enjeux de la recherche sur la télédétection hyperspectrale classification des images et la méthodologie de la thèse.

• • Le chapitre 7 conclut la thèse et propose plusieurs pistes de recherche dans le future. • MCS topologie.

• Génération de classificateur.

• Combinaison de classificateur. d'extraction de caractéristiques est utilisé pour faire tourner chaque sous-échantillon obtenu. Enfin, les composants sont réarrangés pour former l'ensemble de données qui est utilisé pour former un seul membre de l'ensemble. Les détails de Rotation Forêt sont présentés dans Algorithme A.1 [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF]. La solide performance est attribuable à une amélioration simultanée de 1) la diversité dans l'ensemble, obtenu par l'utilisation de l'extraction de caractéristiques sur les données de formation et 2) la précision des classificateurs de base, en gardant toutes les caractéristiques extraites dans les données d'apprentissage [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF][START_REF] Ozcift | Classifier ensemble construction with rotation forest to improve medical diagnosis performance of machine learning algorithms[END_REF].

A.2.1 MCS topologie

Compte tenu de l'importance du choix en ce qui concerne l'algorithme pour l'extraction de caractéristiques dans la RoF, plusieurs options sont envisagées dans ce chapitre. En divisé de faéon aléatoire les caractéristiques F dans K sous-ensembles F i j 3:

for j = 1 : K do 4:

sélectionner les fonctions correspondantes de F i j pour composer un nouveau formation X train i,j R i matrice creuse est composée de coefficients ci-dessus

R i =         v (1) i,1 , ..., v (M 1 ) i,1 0 • • • 0 0 v (1) i,2 , ..., v (M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1)
i,j , ..., v

(M K ) i,j         9:
réorganiser R i à R a i par rapport à l'ensemble des fonctionnalités d'origine 10:

obtenir les nouveaux échantillons de formation X train R a i , Y train 11:

construire classificateur L i à l'aide X train R a i , Y train 12: end for 13: Ajouter le classificateur à l'ensemble actuel, L = L ∪ L i . • Extraction de caractéristiques : PCA [START_REF] Richards | Remote sensing digitial image analysis[END_REF] , ICA [START_REF] Hyvarinen | Independent component analysis : algorithms and applications[END_REF], MNF [START_REF] Green | A transformation for ordering multispectral data in terms of image quality with implications for noise removal[END_REF] et LFDA [START_REF] Sugiyama | Dimensionality reduction of multimodal labeled data by local fisher discriminant analysis[END_REF] ;

• Base de classificateur : classification et l'arbre de régression (CART).

Il peut être observé (voir dans le tableau 3.2) que RoF-PCA surpasse les autres algorithmes. RoF-LFDA a donné la meilleure performance de RoF-MNF. Le OA correspondant de RoF-PCA réalisé sur l'ensemble de test était 83.14% . la complexité de calcul de RoF-PCA, RoF-MNF et RoF-ICA sont beaucoup moins que celle de SVM et RoF-LFDA.

A.4 SVM ensemble sur la base de rotation

Classificateurs SVM ont montré d'excellentes performances de classification pour les hyperspectral images de télédétection. Parce que SVM classificateur est très stable, de petits changements dans l'ensemble de la formation ne produisent pas très différents classificateurs SVM [START_REF] Buciu | Demonstrating the stability of support vector machines for classification[END_REF]. Par conséquent, il est difficile d'obtenir un ensemble de SVM que plus performant que d'un seul SVM en utilisant ensachage et promouvoir le cadre. Inspiré par le sous-espace aléatoire proposé par Ho [START_REF] Ho | The random subspace method for constructing decision forests[END_REF], Waske et al. [START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF] conu un MCS basée sur SVM et la sélection de fonction aléatoire pour la classification des données hyperspectrales, qui peut obtenir un meilleur résultat de classification que tout seul SVM. la limitation de cette MCS est à utiliser uniquement la sélection de fonction aléatoire à promouvoir la diversité au sein de l'ensemble. pour renforcer la diversité dans l'ensemble, RoSVM ensemble, qui combine la sélection de fonction aléatoire et de transformation de données, est développé pour construire les diverses étapes principales classifiers. RoSVM peut être résumées comme suit :

Dans la première étape, F est divisé en groupes K et chaque groupe dispose de fonctionnalités M .

Dans la deuxième étape, une nouvelle série de formation Xtrain i,j est choisi dans la formation ensemble X i,j technique bootstrap avec 75 taille %, lorsque X train i,j représente le j th (j = 1, .., K) sous-ensemble de la i th (i = 1, .., T ) classificateur, correspondant à la fonction sous-ensemble F i,j .

Dans la troisième étape, Xtrain i,j est transformé par un algorithme d'extraction de caractéristiques pour obtenir les coefficients v (1) i,j , ..., v

(M j ) i,j , la taille de v (•) i,j est M × 1.
Dans la quatrième étape, une matrice creuse de rotation R i est obtenu avec les coeffi-cients ci-dessus :

R i =            v (1) i,1 , ..., v (M 1 ) i,1 0 • • • 0 0 v (1)
i,2 , ..., v

(M 2 ) i,2 • • • 0 . . . . . . . . . . . . 0 0 • • • v (1) i,K , ..., v (M K ) i,K           
Ensuite, les colonnes de R i sont réarrangés à R a i respecté l'ordre de l'ensemble des fonctionnalités d'origine.

Dans la cinquième étape, la fonction de formation est X train R a i pour la i th SVM classificateur et chaque SVM est formé en topologie parallèle. le résultat de la classification finale est produit en combinant les résultats de la classification individuelles en utilisant la régle de la majorité.

Il est à noter que dans la deuxième étape, l'objectif de sélection de la taille 75 % du X train i,j est pour éviter d'obtenir les mêmes coefficients lorsque les mêmes caractéristiques sont choisis et donc promouvoir la diversité au sein de l'ensemble [START_REF] Rodriguez | Rotation forest : A new classifier ensemble method[END_REF].

Le succès d'ensemble RoSVM dépend de l'extraction de caractéristiques. Extraction de caractéristiques présentées dans ce travail est de ne pas effectuer la réduction de la dimensionnalité mais juste pour faire tourner l'ensemble de données en gardant tous les composants. RoSVM-PCA est assez cher pour calculer d'analyse d'image hyperspectrale. Une méthode de calcul simple réduction de la dimension à ne pas introduire d'importantes distorsions dans l'ensemble de données serait donc souhaitable. Dans ce chapitre, nous allons introduire RP dans RoSVM ensemble des classificateurs. En RP, les données de grande dimension originale est projetée sur un sous-espace de dimension plus faible en utilisant une matrice dont les colonnes aléatoires ont des longueurs unitaires. RP a été trouvée être une méthode de calcul efficace, tout en étant suffisamment précise pour la réduction de la dimensionnalité des ensembles de données de grande dimension.

En RP, le D de dimension originale est projetée à une de dimension d (d ≤ D ) de sous-espace par l'origine, en utilisant un d × D matrice v dont colonnes ont des longueurs unitaires. Utilisant la notation matricielle où x i est l'ensemble initial de D observation de dimension :

x RP i = v d×D x i (A.1)

RP est mathématiquement très simple : la formation de la matrice aléatoire v et la projection de l'D × N matrice de données x i dans D dimensions est d'ordre O(DN ).

Dans la plupart des méthodes de projection, la matrice de transformation qui en résulte peut pas prendre un d valeur plus grande que D (par exemple, PCA). Trois types de RP sont utilisés dans ce travail [START_REF] Achlioptas | Database-friendly random projections[END_REF][START_REF] Baraniuk | A simple proof of the restricted isometry property for random matrices[END_REF] :

1. Gaussian. L'ensemble des valeurs v proviennent d'une distribution gaussienne (moyenne 0 et écart-type 1).

2. Sparse. Les valeurs v dans la matrice de transformation sont √ 3 × α, où α est un nombre aléatoire généré par les conditions suivantes : -1 avec la probabilité 1/6 , 0 avec la probabilité 2/3 et +1 avec probabilité 1/6. Les résultats expérimentaux effectués sur l'image de ROSIS (voir dans le tableau A.3 ) ont indiqué que des ensembles RoSVM sont supérieurs à la SVM et RSSVM. Ensemble SVM approche avec 10 itérations ont gagné OA supérieure à la SVM régulière. RSSVM améliorer sensiblement la précision en utilisant l'ensemble d'apprentissage très limité (10 échantillons par classe), la différence entre 100 itérations et 10 itérations est de 3.15 points de pourcentage. Dans d'autres cas, il n'y a pas d'augmentation de l'arthrose associée à de plus grands ensembles.

En outre, nous avons étudié l'influence de différentes valeurs de M sur la précision globale. Figue. A.5 présente les effets de la variation M sur la performance des ensembles SVM en utilisant trois ensembles d'apprentissage différents. Le nombre suffisant de fonctionnalités pour RSSVM est 25. Tous les quatre ensembles RoSVM peuvent améliorer les résultats de la classification de toutes les valeurs de M . La meilleure performance réalisée par RoSVM utilisant différents ensembles de formation sont divers. Par exemple, la précision maximale est atteinte par des ensembles RoSVM-PCA avec M = 25 pour l'ensemble de la formation (40 échantillons par classe), tandis que RoSVM-RP B, à donné la plus haute OA avec M = 10.

A.5 Rotation Forêt avec extraction de caractéristiques locales et champ aléatoire de Markov

Les méthodes proposées fondé sur la RoF et du MRF sont composés de trois étapes principales comme suit :

1) classification supervisée pixel par pixel en utilisant la RoF.

2) l'extraction de l'information spatiale en utilisant des MRF. 3) classification spatio-spectrale par la résolution d'un problème MAP calculée par le algorithme d'optimisation α-Expansion de Graph cuts.

La tâche de classification de l'image peut alors être formulée comme un problème MAP, pour lesquels l'agrandissement postérieur p(Y|X) donne une solution, ce qui est équivalent à maximiser p(X|Y)p(Y). Il est possible d'imposer des contraintes contextuelles spatiales par modélisation p(Y) avec un MRF [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF][START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF]. En supposant l'indépendance conditionnelle des caractéristiques données les étiquettes, p(X|Y) peut être formulé comme [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF] : (A.9) p(y i |x i ) est obtenu par RoF. Afin de tenir compte des informations de voisinage autour des points de données, trois méthodes d'extraction d'entités linéaires locaux, y compris les NPE [START_REF] He | Neighborhood preserving embedding[END_REF], LLTSA [START_REF] Zhang | Linear local tangent space alignment and application to face recognition[END_REF] et LPP [START_REF] He | Locality preserving projections[END_REF] sont introduits dans les RoF.

p(X|Y) =
Une question importante des approches fondées sur-MRF est de calculer le minimum global de la fonction objectif. Fonctions objectives à base de MRF telle que celle dans les équations ci-dessus sont fortement non convexe. Nous avons recours à l'algorithme basé sur le graphique coupe α-Expansion de Graph Cuts [START_REF] Li | Hyperspectral image segmentation using a new bayesian approach with active learning[END_REF][START_REF] Boykov | Fast approximate energy minimization via graph cuts[END_REF].

Tableau A.4 résume les précisions globales, des précisions moyennes et les précisions spécifiques à chaque classe. Figue. A.6 présente les cartes de classification. Nous listons aussi les résultats de la classification des classificateurs de pixel par pixel : SVM et LOR-SAL, et le classificateur spectral-spatial suivant : LORSAL-MRF dans le tableau A.4. Comme on peut le voir dans le tableau A.4, l'OA des quatre RoF sont tous plus élevés que ceux de SVM et LORSAL. Cela conduit à une précision inférieure à celle du classificateur à l'aide uniquement des informations spectrales. Les meilleurs mondiaux précisions sont atteints par RoF-LPP-MRF. Le plan de classement correspondant est nettement plus précise que toute autre carte de classification, en fonction des résultats du test de McNemar. Dans ce cas, la précision globale et moyenne sont améliorées par 7.39 et 4.42 points de pourcentage, respectivement, par rapport à RoF-LPP. L'utilisation de fonction extraction LPP mène également aux plus hautes précisions pour la plupart des classes (cinq sur neuf). L'OA de la proposition de quatre régimes sont tous plus élevés que les classificateurs spectrales-spatiales standards, tels que SVM-segmentation [START_REF] Tarabalka | Segmentation and classification of hyperspectral images using watershed transformation[END_REF], SVMMSF [START_REF] Tarabalka | Segmentation and classification of hyperspectral images using minimum spanning forest grown from automatically selected markers[END_REF], SVMMRF-NE [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF], SVMMRF-E [START_REF] Tarabalka | SVM and MRF-based method for accurate classification of hyperspectral images[END_REF], LORSAL-MRF [START_REF] Li | Spectral-spatial classification of hyperspectral data using loopy belief propagation and active learning[END_REF].

Figue. A.7 trace la OA en fonction de M pour les deux classificateurs de pixel par pixel et spectrales-spatiales proposées. Figue. A.7 indique que les algorithmes basés sur les RoF atteint les plus hautes performances lorsque M est configuré pour être 10. Ce parce que quand une valeur inférieure de M est utilisé, la diversité au sein des ensembles augmente. La grande diversité au sein de l'ensemble conduit souvent à de grandes précisions. Les résultats de la classification spectrales-spatiales rapporté à la

A.6 Aléatoires subspatiales d'ensemble avec les profils d'attributs morphologiques étendues

Dans ce chapitre, nous proposons un système de classification de pointe basée sur RS des ensembles appliqués à EMAPs caractéristiques. DT et NN sont généralement adoptées en tant qu'apprenant de base en ensemble de RS car ils sont instables parce que de petits changements dans la ligne de données de formation à grande variation dans les résultats, résultant dans la production grande diversité au sein de l'ensemble. Considérant le co ût On peut voir dans le tableau A.6 tout l'ensemble RS donne les résultats de précision les plus élevées. Le RSELM proposé et RoELM, RoRF et d'autres méthodes d'ensemble en termes de réalisation des précisions globales et spécifiques aux classes supérieures. RoRF, RoF et RoELM génèrent des résultats plus précis que ceux de RF, RSDT et RSELM car ils ont introduit plus de diversité au sein de l'ensemble. En ce qui concerne la charge de calcul, différentes observations peuvent être faites que dans les expériences précédentes. Les effets de l'ensemble des paramètres de RS sont représentés sur la figure A.10 . On constate qu'il n'y a pas de situation de dépendance entre M et l'exactitude ensemble. Différents RS classificateur ensemble gagnent le plus OA sur différentes valeurs de M . RoELM avec EMAPs réalise le meilleur résultat de classification lorsque M = 6 . Figure A.10 représente qu'un grand nombre de nøeuds cachés peut donner des précisions plus élevées dans les tests, mais un réseau complexe pourrait également surajustement les données d'entraînement. Par exemple, la performance de généralisation diminue lorsque le nombre de noeuds cachés dans supérieure à 512. En général, ces paramétres doivent être choisis de manière empirique dans des applications particulières.

A.7 Conclusions et orientations futures

Les conclusions générales de cette thèse sont les suivants : Il y a plusieurs problèmes potentiels avec MCS pour la classification des données hyperspectrales qui méritent la poursuite de recherche.

• L'apprentissage adaptatif. Dans cette thèse, le RS et le RoS peuvent tous obtenir de résultats satisfaisants. Toutefois, les valeurs optimales de M est difficile à déterminer. Dans l'avenir, nous allons développer un cadre automatique pour sélectionner les valeurs optimales. Un autre travail consistera á sélectionner de manière adaptative les sous-optimales classificateurs de l'ensemble de classificateurs original.

• L'Apprentissage de fonctions multiples. Les techniques actuelles d'apprentissage simplement concaténer une paire de caractéristiques différentes ou de combiner les résultats de la classification dans différentes fonctions. Cependant, la façcon de fusionner plusieurs fonctions d'une manière générale est encore un problème ouvert.

Dans les études futures, nous allons formuler la fonction apprentissage multiple comme un problème général d'apprentissage sous-espace dans lequel nous pouvons utiliser sous-espace aléatoire, Rotation sous-espace et des techniques de sous-espaces tenseur.

• D'apprentissage plus rapide. Une future ligne de recherche que nous envisageons est liée à la mise en øeuvre efficace de calcul des approches proposées dans l'architecture de calcul haute performance tels que les processeurs multi-core.

• D'apprentissage en ligne. L'apprentissage en ligne est particulièrement utile lorsque les données sont arrivent continuellement de sorte qu'il est impossible de stocker des données ou lorsque l'ensemble est très grand, dans ce cas, le temps d'exécution pour de multiples passages dans les données seraient prohibitifs.

Figure 1 . 1 :

 11 Figure 1.1: Hyperspectral vs Multispectral images of the same area. The upper-left is the cube of a hyperspectral image with 64 bands from 415 nm to 900 nm. The upperright is the spectral reflectance of several land cover types from hyperspectral image, including pond, metalic roof, river, grass and agriculture. The bottom-left is the cube of a multispectral image with 4 bands from 475 nm to 750 nm. The bottom-right is the spectral reflectance of the same land cover types from multispectral image.

Name

  Spectral range (nm) Spectral/spatial resolution (nm/m) Number of Bands Launch time

Fig. A. 1

 1 Fig. A.1 illustrates the two topologies employed in MCS design. The overwhelming majority of MCS reported in the literature is structured in a parallel style. In this ar-

Figure 2 . 1 :

 21 Figure 2.1: The topologies of MCSs.

Figure 2 . 2 :

 22 Figure 2.2: Different classifiers combination using three single classifiers. The three colors represent the different classes. The overall accuracy of all individual classifiers is 6/9. The overall accuracies of the four combinations are 1, 8/9, 6/9 and 5/9, respectively.

Fig. 2 .

 2 3(a) is the false color composite of the image by using Band 27, 25 and 2 as R, G and B components. Training samples and test samples are selected independently from the image. This image is classified into five classes: water, building, vegetation, forest and bare soil.

Figure 2 . 3 :

 23 Figure 2.3: False color composite of OMIS image and classification results of individual classifier.

Contents 3 . 1

 31 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 3.2 Classification and Regression tree . . . . . . . . . . . . . . . . 29 3.3 Ensemble methods . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3.1 Bagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.3.2 Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.3.3 Random subspace . . . . . . . . . . . . . . . . . . . . . . . . . 35 3.3.4 Random Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.4 Rotation Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 3.5 Experimental results and analysis . . . . . . . . . . . . . . . . 40 3.5.1 Experimental settings . . . . . . . . . . . . . . . . . . . . . . . 40 3.5.2 Experimental results . . . . . . . . . . . . . . . . . . . . . . . . 41 3.5.3 Sensitivity of Parameters . . . . . . . . . . . . . . . . . . . . . 45 3.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

Figure 3 . 1 :

 31 Figure 3.1: Decision tree. (a) A tree is a set of nodes and edges organized in a hierarchical structure. Root/internal nodes and terminal nodes are denoted with circles and squares, respectively. (b) An illustrative decision tree example used to examine a photo represents indoor or outdoor scene [1].

Figure 3 . 2 :

 32 Figure 3.2: Splitting algorithm of CART.

Algorithm 3. 1 2 : 3 :

 123 Bagging Input: : n labeled samples T train = X train , Y train = {(x i , y i )} n i=1 with x ∈ R D and y ∈ R. Number of classifiers: T . Base classifier: L. The ensemble L = φ. Output: : The ensemble L . 1: for i = 1 to T do Bootstrap sample from T train to form a new training set T train i with size of n , n ≤ n. Train a DT classifier L i using a new training set T train i . 4:

2 :

 2 Number of classifiers: T . Base classifier: L. The ensemble L = φ. number of features in a subspace M , feature set F. Output: : The ensemble L . 1: for i = 1 to T do Randomly selected from F without replacement to form a new training set T train i composed of M features (M < D).

3 :

 3 Train a DT classifier L i using a new training set T train i .

Algorithm 3 . 4 2 :

 342 Random Forest Input: : n labeled samples T train = X train , Y train = {(x i , y i )} n i=1 with x ∈ R D and y ∈ R. Number of classifiers: T . Base classifier: L. The ensemble L = φ. M : number of features used in each node. n : the sub-sample size. Output: : The ensemble L . 1: for i = 1 to T do Bootstrap sample from T train to form a new training set T train i with size of n , n ≤ n.

3 :

 3 Train a DT classifier L i using a new training set T train i with M in each node.

Algorithm 3 . 5

 35 Rotation Forest Input: X train , Y train = {x i , y i } n i=1 : training samples, T : number of classifier, K: number of subsets (M: number of features in each subset), L: base classifier. The ensemble L = φ. F: Feature set Output: The ensemble L 1: for i = 1 : T do 2:

  feature extraction method (e.g. PCA, ICA) to get the coefficients v

10 :

 10 obtain the new training samples X train R a i , Y train 11:

Figure 3 . 3 :

 33 Figure 3.3: (a) Three-band color composite of University of Pavia ROSIS image. (b) Reference map: Asphalt, meadows, gravel, trees, metal sheets, bare soil, bitumen, bricks and shadow.

Figure 3 . 4 :

 34 Figure 3.4: Classification results of different ensemble classifiers when applied to ROSIS image. (a) CART, OA = 64.91%. (b) Bagging, OA = 70.11 %. (c) AdaBoost, OA = 70.81%. (e) RS, OA = 65.11%.(d) RF, OA = 70.68%. (f) RoF-PCA, OA = 83.15%. (g) RoF-ICA, OA = 78.09%. (h) RoF-MNF, OA = 74.12%. (i) RoF-LFDA, OA = 76.13 %.

Figure 3 . 5 :

 35 Figure 3.5: (a) Three-band color composite of DAIS image. (b) Ground truth.

  Fig. A.4 shows the OA(%) using different numbers of T and M obtained from AVIRIS, ROSIS and DAIS image, respectively. With the increment of T , the classification results are slightly improved. Random subspace ensembles show significantly improvement when the value of M increases. For the other ensemble classifiers, the sensitivity of parameters to M in different hyperspectral images is various. For AVIRIS image, the classification accuracies of Rotation Forests are improved when the value of M increases. For ROSIS image, the classification performances of Rotation Forest are decreased when when the value of M increases. For DAIS image, Rotation Forests yield the highest OAs in a medium value of M .

Figure 3 . 6 :Figure 3 . 7 :

 3637 Figure 3.6: Classification results of different ensemble classifiers when applied to DAIS image. (a) CART, OA = 91.52%. (b) Bagging, OA = 92.2%. (c) AdaBoost, OA = 92.24%. (e) RS, OA = 92.01%.(d) RF, OA = 94.28%. (f) RoF-PCA, OA = 95.81%. (g) RoF-ICA, OA = 95.47%. (h) RoF-MNF, OA = 95.39%. (i) RoF-LFDA, OA = 96.01%.

Figure 4 . 1 :

 41 Figure 4.1: An illustrated classification example of SVM with one non separable feature vector in each class [2].

  Fig. 4.2 show the main procedure of rotation-based SVM ensemble. According to Fig. 4.2, the main steps of rotation-based SVM ensemble can be summarized as follows:

3 .

 3 Bernoulli. The values v under the Bernoulli distribution, are 1 √ M × α, where, α is a random number observed by the following conditions: -1 with the probability 1/2 and +1 with probability 1/2.

4. 4

 4 Experimental results and analysis 4.4.1 Experimental setup In this section, the RoSVM ensemble classifiers are evaluated by using three real hyperspectral data sets. For the AVIRIS dataset, labeled samples (10366 samples) are divided into training samples and test samples. For the other two datasets, the training samples are separated from the test samples. The number of train (test) samples in ROSIS and DAIS images are 3921(42776) and 1859(14585), respectively. In order to investigate the classification performances of RoSVM for limited training samples, we randomly selected 10, 20, 30, 40, 50 samples per class from Indiana Pines dataset and the University of Pavia ROSIS image, and 2, 4, 6, 8 samples per class of the Pavia Center DAIS image to compose the training set, respectively. The remaining labeled samples of Indian Pines AVIRIS dataset are treated as the test samples. Ensemble size (T ) and the number of features in a subset (M ) are the principal parameters of RoSVM. Thus, we select various combinations of T and M , T = 10, 25, 50, 100 for three datasets and M = 10, 25, 50, 100 for Indian Pines AVIRIS image, M = 10, 25, 50 for University of Pavia ROSIS image, M = 10, 20, 40 for Pavia Center DAIS image, to assess the impact of the two parameters in RoSVM.

Figure 4 . 3 :

 43 Figure 4.3: (a) Three-band color composite of AVIRIS image. (b) Ground truth: Cornno till, corn-min till, corn, soybean-no till, soybeans-min till, soybeans-clean till, alfalfa, grass/pasture, grass/trees, grass/pasture-mowed, hay-windrowed, oats, wheat, woods, bldg-grass-tree-drives, stone-steel towers.

  76%) using PCA, RP Gaussian, RP Sparse and RP Bernoulli feature extraction techniques, respectively. RoSVM-PCA outperforms RoSVM with three RP feature extraction methods for small sizes of the training set (10, 20, 30, 40 samples per class), on the contrary, RoSVM-RP shows better performance than RoSVM-PCA for medium size of the training set (50 samples per class). Different RP algorithms provide different performances in different ensemble size and training set. For instance, RoSVM-RP G with 50 iterations obtained the highest OA for the training set of 30 samples per class, and RoSVM-RP B with 100 iterations got the highest OA for the training set of 50 samples per class.

Figure 4 . 4 :

 44 Figure 4.4: Indian Pines AVIRIS image. Sensitivity to the change of M (T = 10). (a) 10 samples per class. (b) 30 samples per class. (c) 50 samples per class.

Figure 4 . 5 :

 45 Figure 4.5: Classification maps of the Indian Pines AVIRIS image achieved by SVM and SVM ensemble approaches (40 samples per class).

Figure 4 . 6 :

 46 Figure 4.6: University of Pavia ROSIS image. Sensitivity to the change of M (T = 10). (a) 20 samples per class. (b) 30 samples per class. (c) 40 samples per class

Figure 4 . 7 :

 47 Figure 4.7: University of Pavia ROSIS image. Difference between class-specific accuracy achieved by SVM ensemble approaches with 10 iterations and a regular SVM. (a) 10 samples per class. (b) 50 samples per class.

Figure 4 . 8 :

 48 Figure 4.8: Pavia Center DAIS image. Sensitivity to the change of M (T = 10). (a) 4 samples per class. (b) 6 samples per class. (c) 8 samples per class.

Figure 4 . 9 :

 49 Figure 4.9: Pavia Center DAIS image. Difference between class-specific accuracy achieved by SVM ensemble approaches with 10 iterations and a regular SVM.

Contents 5 . 1

 51 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.2 Proposed method . . . . . . . . . . . . . . . . . . . . . . . . . . 74 5.2.1 Rotation Forest . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 5.2.2 Markov random fields . . . . . . . . . . . . . . . . . . . . . . . 77 5.2.3 MAP labeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 5.3 Experimental results and analysis . . . . . . . . . . . . . . . . 79 5.3.1 Indian Pines AVIRIS image . . . . . . . . . . . . . . . . . . . . 79 5.3.2 University of Pavia ROSIS image . . . . . . . . . . . . . . . . . 84 5.3.3 Pavia Center DAIS image . . . . . . . . . . . . . . . . . . . . . 86 5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 69

Figure 5 . 1 :

 51 Figure 5.1: Flowchart of the spectral-spatial approach using Rotation Forest and Markov random field.

Algorithm 5 . 1

 51 Rotation ForestInput: T train = {X train , Y train } = {(x i , y i ) , ..., (x n , y n )}: training samples, T : number of classifier, K: number of subsets (M: number of features in each subset), base classifier L, training features F 1. For i = 1 : T 2.

Figure 5 . 2 :

 52 Figure 5.2: Classification results of Indian Pines AVIRIS image. (a) RoF-PCA. (b) RoF-NPE. (c) RoF-LLTSA. (d) RoF-LPP. (e) RoF-PCA-MRF. (f) RoF-NPE-MRF. (g) RoF-LLTSA-MRF. (h) RoF-LPP-MRF.

  Fig. A.7 plots the OA as a function of M for both proposed pixel-wise and spectral-spatial classifiers. Fig. A.7 indicates that Rotation Forests-based algorithms achieved the highest performances when M is set to be 10. That is because when a smaller value of M is used, the diversity within the ensemble increases. High diversity within the ensemble often leads to high accuracies. The spectral-spatial classification results reported in Fig. A.7(e-h) are more accurate than those of Rotation Forests presented in Fig. A.7(ad), further demonstrating the importance of spatial information.

Figure 5 . 3 :

 53 Figure 5.3: Classification results of University of Pavia ROSIS image. (a) RoF-PCA. (b) RoF-NPE. (c) RoF-LLTSA. (d) RoF-LPP. (e) RoF-PCA-MRF. (f) RoF-NPE-MRF. (g) RoF-LLTSA-MRF. (h) RoF-LPP-MRF.

Figure 5 . 4 :

 54 Figure 5.4: Impact of OA using different numbers of M obtained for the ROSIS image. (a) Rotation Forests. (b) Rotation Forests and MRF.

Algorithm 6 . 1 1 : 2 : 3 :

 61123 Assign a class label to x * via the highest output value of decision function: h(x * )β. Extreme learning machine Input: n labeled samples X train , Y train = {(x i , y i )} n i=1 , δ the number of nodes in a hidden layer. The sigmoid function g. Randomly select the {ω 1 , ..., ω δ } and {b 1 , ..., b δ } For each training sample x i , calculate the output layer matrix: h(x i ) = [g(ω 1 , b 1 , x i ), ..., g(ω δ , b δ , x i )] Calculate the output weight: β = H + Y train Output: The output weight β.

Algorithm 6. 2

 2 RoELM Input: X train , Y train = {x i , y i } n i=1 : training samples, T : number of classifier, K: number of subsets (M: number of features in each subset), L: base classifier. The ensemble L = φ. F: Feature set Output: The ensemble L 1: for i = 1 : T do 2:

4 :,j 5 :

 45 select the corresponding features of F i j to compose a new training set X train iselect a new training samples Xtrain i,j using bootstrap algorithm, whose size is 75% of the original size of training samples 6:

10 :

 10 obtain the new training samples X train R a i , Y train 11:

Figure 6 . 1 :

 61 Figure 6.1: The construction of EMAPs using the area (A) and standard (S) deviation attributes. Firstly, PCA is performed on the original hyperspectral image and the first features with cumulative eigenvalues over 99% are kept. Then, APs with area and standard deviation attributes are performed on the first features and the output features are concatenated into a stacked vector to construct EMAPs.

Figure 6 . 2 :

 62 Figure 6.2: Classification results of Indians Pines AVIRIS image (only one Monte Carlo run). Overall accuracies of the classifiers are also given.

  Fig A.9 provides the classification maps of the individual and ensemble classifiers (one of ten Monte Carlo runs). Compared to the results

Figure 6 . 3 :

 63 Figure 6.3: Classification results of University of Pavia ROSIS image (only one Monte Carlo run). Overall accuracies for each classifier are given.

Fig A. 9

 9 (a-f), the maps involving spatial information (seen in Fig A.9 (g-p)) generate more homogeneous areas (especially for the Class M eadows located at the lower left area) and reduce the classification noise.

6. 5 . 3 Figure 6 . 4 :

 5364 Figure 6.4: Indiana Pines AVIRIS image (10% of the labeled samples as training samples). Sensitivity to the change of (a) M with spectral information. (b) M with EMAPs. (c) δ of ELM and its ensembles with spectral information. (d) δ of ELM and its ensembles with EMAPs. University of Pavia ROSIS image (entire training set). Sensitivity to the change of (e) M with spectral information. (f) M with EMAPs. (g) δ of ELM and its ensembles with spectral information. (h) δ of ELM and its ensembles with EMAPs.

Figure A. 1 :

 1 Figure A.1: Les topologies de MCS.

  particulier, quatre techniques d'extraction de caractéristiques linéaires, y compris PCA, MNF, ICA et LFDA. Des résultats expérimentaux sont montrés dans la thèse sur trois images aéroportées hyperspectrales, enregistrées par les capteurs AVIRIS, ROSIS et DAIS, avec différents contextes (zones urbaines et agricoles et volcaniques), différentes résolutions spatiales (1.3 m et 20 m) et différents nombres de canaux spectraux (de 80 à 220 bandes). Cette section présente des résultats doévaluation expérimentale sur une image de University of Pavia ROSIS. L'image ROSIS avec 115 canaux spectraux est acquis sur l'université de Pavie, en Italie. La taille de l'image est de 610 × 340 avec la résolution spatiale de 1.3 m. Douze canaux bruyants ont été retirés et les 103 autres bandes avec une plage spectrale de 0.43 à 0.86 µm ont été utilisés pour les expériences. Dans tous les cas, la performance réalisée par rotation forêt est illustrée à l'aide des modèles suivants : • Nombre de caractéristiques dans un sous-ensemble : M = 10 ; • Nombre de classificateurs : T = 10 ; Algorithm A.1 Rotation Forest Input: X train , Y train = {x i , y i } n i=1 : échantillons de formation, T : nombre de classificateurs, K : nombre de sous-ensembles (M : nombre de fonctions dans chaque sous-ensemble), L : classificateur de base. l'ensemble L = ∅. F : Ensemble de fonctionnalités Output: l'ensemble L 1: for i = 1 : T do 2:

caractéristiques 5 :

 5 sélectionner une nouvelle échantillons de formation Xtrain i,jen utilisant l'algorithme d'amoréage, dont la taille est de 75 % de la taille originale feature extraction method (e.g. PCA, ICA) to get the coefficients v

Figure A. 2 :

 2 Figure A.2: (a) à trois bandes de couleur composite de ROSIS (b) Carte de référence : asphalte, des prairies, du gravier, des arbres, des feuilles de métal, un sol nu, bitume, des briques et de l'ombre.

  Fig. A.2 présenté les trois -band composite de couleur et la réalité de terrain de l'image ROSIS. les résultats de la classification avec voiturette et différents classificateurs d'ensemble sont illustrés sur la Fig. A.3. Fig. A.4 décrit l'OA (%) en utilisant des nombres différents de T et M . Avec l'augmentation de T , les résultats de la classification sont légèrement améliorées. Pour les autres classificateurs d'ensemble, la sensibilité des paramètres à M se diverse en fonction de différentes images hyperspectrales. Les performances de classification de RoF sont diminués lorsque la valeur de M augmente.

Figure A. 3 :Figure A. 4 :

 34 Figure A.3: Les résultats de classification des différents classificateurs d'ensemble. (a) CART, OA = 64.91%. (b) Bagging, OA = 70.11 %. (c) AdaBoost, OA = 70.81%. (e) RS, OA = 65.11%.(d) RF, OA = 70.68%. (f) RoF-PCA, OA = 83.15%. (g) RoF-ICA, OA = 78.09%. (h) RoF-MNF, OA = 74.12%. (i) RoF-LFDA, OA = 76.13 %.

3 . 1 √M

 31 Bernoulli. Les valeurs v en vertu de la loi de Bernoulli, sont × α, où α est un nombre aléatoire observé par les conditions suivantes : -1 avec la probabilité 1/2 et +1 avec probabilité 1/2. Afin d'étudier les performances de classification de RoSVM pour les échantillons de formation limitées, nous avons choisi au hasard 10, 20, 30, 40, 50 échantillons par classe. Diverses combinaisons de T et M , T = 10, 25, 50, 100 et M = 10, 25, 50 sont utilisés pour évaluer l'impact de ces deux paramètres dans RoSVM.

Figure A. 5 :

 5 Figure A.5: Sensibilité à la variation de M (T = 10). (a) 20 échantillons par classe. (b) 30 échantillons par classe. (c) 40 échantillons par classe.

-

  i |x i )p(x i ) p(y i ) . (A.3)Ensuite, la distribution a posteriori peut être réécrite comme suit :i |x i )p(x i ) p(y i ) i |x i ) p(y i ) p(Y). (A.7)Dans le modèle proposé,N i=1 p(x i ) p(X)est un facteur non en fonction à y. La densité p(y i ) sont supposés être également répartie. Le problème de MAP peut alors être définie comme suit : arg max log p(y i |x i ) + log p(Y) = arg min -log p(y i |x i ) -log p(Y) log p(y i |x i ) -µ (i,j)∈ς δ(y i -y j )

  Fig. A.7(e-h) sont plus précis que ceux de RoFs présenté dans la figure. A.7(a-d), démontrant une nouvelle fois l'importance de l'information spatiale.

Figure A. 6 :

 6 Figure A.6: Les résultats de classification. (a) RoF-PCA. (b) RoF-NPE. (c) RoF-LLTSA. (d) RoF-LPP. (e) RoF-PCA-MRF. (f) RoF-NPE-MRF. (g) RoF-LLTSA-MRF. (h) RoF-LPP-MRF.

Figure A. 7 :

 7 Figure A.7: L'impact de l'arthrose avec chiffres différents de M à partir de l'image ROSIS. (a) RoF. (b) RoF et MRF.

Figure A. 8 :

 8 Figure A.8: La construction de EMAPs utilisant la area (A) et standard deviation (S) attributs.

Figure A. 9 :

 9 Figure A.9: Les résultats de classification. Précision globale pour chaque classificateur sont donnés.

Figure A. 10 :

 10 Figure A.10: Sensibilité à la variation de (a) M avec EMAPs et (b) δ de ELM et ses ensembles avec EMAPs.

  1.1 Hyperspectral sensors, including running, operated and future missions. . . 1.2 Summary of spectral-spatial approaches applied in hyperspectral image classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . For ensemble approaches, only highest averages of OA using different values of M are presented. . . . . . . . . . 4.3 Overall, average and class-specific accuracies obtained from the Indian Pines AVIRIS image. M is set to be 50 and 100 for RSSVM and RoSVM, respectively. T is set to be 10. . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Average of OA using SVM and ensemble approaches (the corresponding value of AA is in bracket) applied to different numbers of training samples (University of Pavia ROSIS image). For ensemble approaches, highest averages of OA with different values of M are presented. . . . . . . . . . . Average of overall accuracies obtained from the proposed methods using different numbers of training samples (10 Monte Carlo runs) for the Indian Pines AVIRIS image. The numbers of training samples per class and the total numbers of training samples (in brackets) are also given. . . . . . . . 5.2 Average of overall accuracies obtained from the proposed Rotation Forests and MRF using different values of M (Indian Pines AVIRIS image). . . . 5.3 Average of overall accuracies obtained from the proposed Rotation Forests and MRF using different values of τ (Indian Pines AVIRIS image). . . . . 5.4 Overall, average and class-specific accuracies obtained for the Indian Pines AVIRIS image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Average of overall accuracies obtained from the proposed approaches using different numbers of training samples for the University of Pavia ROSIS image. The number of training samples per class and the total number of training samples (in brackets) are also given. . . . . . . . . . . . . . . . . . 5.6 Average of overall accuracies obtained from the proposed approaches using Individual and ensemble classification approaches considered for the study 6.2 Indiana Pines AVIRIS image: class name and number of samples in Ground truth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 University of Pavia ROSIS image: class name and number of training and test samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 Overall accuracies (in percent) and average accuracies (in the parentheses) obtained for different classification algorithms using different number of training samples per class when applied to the Spectral information of AVIRIS Indian Pines Hyperspectral data. . . . . . . . . . . . . . . . . . . . 6.5 Overall accuracies (in percent) and average accuracies (in the parentheses) obtained for different classification algorithms using different number of training samples per class when applied to the EMAPs of AVIRIS Indian Pines Hyperspectral data. The reported standard deviation is obtained from ten Monte Carlo runs. . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 The parameters used for ELM and RS ensemble classifiers (Indiana Pines AVIRIS image) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10% number of samples in Ground Truth as training samples when applied to the spectral information of Indiana Pines AVIRIS Hyperspectral data. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.8 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10% number of samples in Ground Truth as training samples when applied to the EMAPs of Indiana Pines AVIRIS Hyperspectral data. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.9 Overall accuracies, average accuracies and class-specific accuracies obtained for different classification algorithms using the entire training set when applied to the spectral information of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . 6.10 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using the entire training set when applied to the EMAPs of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . . . . . . . . . . . . . . 6.11 Overall accuracies, average accuracies and κ obtained from the proposed methods: RoELM and RSELM with other Spatial-spectral classifier for the University of Pavia ROSIS image . . . . . . . . . . . . . . . . . . . . . . . 6.12 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10 samples per class when applied to the spectral information of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . . . . . . . . 6.13 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10 samples per class when applied to

	xviii	LIST OF TABLES
	5.1
	xvii

2.1 Summary of the 15 diversity measures . . . . . . . . . . . . . . . . . . . . 2.2 Summary of classifier combination approaches. . . . . . . . . . . . . . . . . 2.3 Classification accuracy statistics using different methods when applied to OMIS hyperspectral image. . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4 Studies on High-dimensional remote sensing image classification using MCS published in Journals in recent years . . . . . . . . . . . . . . . . . . . . . 3.1 Training and test samples of the three hyperspectral images . . . . . . . . 3.2 Overall accuracies [%] for the three hyperspectral image using different classifiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1 Details of experimental settings applied for three hyperspectral datasets. . 4.2 Average of OA using SVM and ensemble approaches (the corresponding value of AA is in bracket) applied to different numbers of training samples (Indian Pines AVIRIS image). 4.5 Averages of OA and AA using SVM and ensemble approaches (the back bracket is the corresponding value of AA) applied to different numbers of training samples (Pavia Center DAIS image). . . . . . . . . . . . . . . . . different values of M (University of Pavia ROSIS image). The number of training samples per class is 20, and the total number of training samples is 180. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Average of overall accuracies obtained from the proposed methods using different values of τ (University of Pavia ROSIS image). The number of training samples per class is 20, and the total number of training samples is 180. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.8 Overall, average and class-specific accuracies obtained for the University of Pavia ROSIS image. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.9 Overall, average and class-specific accuracies obtained from the proposed approaches (Pavia Center DAIS image). . . . . . . . . . . . . . . . . . . . 6.1 the EMAPs of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs. . . . . . . . . . . . . . . . . . . . . . .

Table 1 . 1 :

 11 Hyperspectral sensors, including running, operated and future missions.

Table 1 . 2 :

 12 Summary of spectral-spatial approaches applied in hyperspectral image classification

Table 2 .

 2

	Kappa statistic(κ 1 , κ 2 )
	p
	s
	↓
	[-1, 1]
	[67, 71]

1: Summary of the 15 diversity measures Name p/n s/dis/c ↑ / ↓ Range Reference

  the measurement level, probability values of the classes provided by each classifier are used in the combination. Majority/weighted vote, fuzzy integral, evidence theory and dynamic classifier selection belong to the abstract level combination methods, while others are measurement level methods.TableA.2 summarizes classifier combination approaches. Weighted vote, fuzzy integral, Dempster-Shafer evidence theory and consensus theory require validation set to calculate the weights. Dynamic classifier selection calculates the distance between the samples so it requires the original image. And the computation time of dynamic classifier selection is more expensive than other approaches.

Table 2 . 2 :

 22 Summary of classifier combination approaches.

	Name	Class label	Class probability	validation set	Reference
	Majority vote (MV)	Y	N	N	[48]
	Weighted vote (WV)	Y	N	Y	[59, 89]
	Bayesian average (BA)	N	Y	N	[90]
	Dempster-shafer evidence theory (DS)	Y	N	Y	[48, 91]
	Fuzzy integral (FI)	Y	N	Y	[92-94]
	Consensus theory (CT)	Y	Y	Y	[95, 96]
	Dynamic classifier selection (DCS)	Y	N	Y	[51, 97, 98]
	* Note: 'Y' and 'N'

mean whether or not the class label, class probability or validation set are needed. Dynamic classifier selection method needs the original image to calculate the distance.

Table 2 .3: Classification accuracy statistics using different methods when applied to OMIS hyperspectral image.

 2

	Methods OA	κ	Methods	OA	κ
	SVM	90.72% 0.88 BA [90]	92.95% 0.91
	J48 DTC 87.86% 0.85 DS [48]	93.08% 0.91
	MLPNN 92.53% 0.91 FI [93]	92.72% 0.90
	RBFNN 89.46% 0.87 Log CT [95]	93.12% 0.91
	MV [48] 93.15% 0.91 Linear CT [95] 93.27% 0.91
	WV [59] 93.27% 0.91 DWDCS [51]	93.58% 0.92

Table 2 .

 2 4, the most popular MCS for hyperspectral images is Random Forest. Waske et al. [84] combined random feature selection and SVM classifier for the classification of hyperspectral images. Yang et al.

Table 2 . 4 :

 24 Studies on High-dimensional remote sensing image classification using MCS published in Journals in recent years

	Methods
	Study

Table 3 . 1 :

 31 Training and test samples of the three hyperspectral images

		AVIRIS			ROSIS			DAIS
	Class name	Training set	Test set	Class name	Training set	Test set	Class name	Training set	Test set
	Soybeans-min till	493	2468	Asphalt	548	6641	Water	202	4281
	Grass/pasture	99	497	Meadows	540	18649	Trees	205	2424
	Hay-windrowed	98	489	Gravel	392	2099	Meadows	206	1251
	Soybeans-clean till	123	614	Trees	524	3064	Bricks	315	2237
	Corn-no till	287	1434	Metal Sheets	265	1345	Soil	205	1475
	Grass/tree	149	747	Soil	532	5029	Asphalt	204	1704
	Soybeans-no till	194	968	Bitumen	375	1330	Bitumen	202	685
	Woods	259	1294	Bricks	514	3682	Parking lot	201	287
	Corn-min till	167	834	Shadows	231	947	Shadows	119	241

Table 3 . 2 :

 32 Overall accuracies [%] for the three hyperspectral image using different classifiers

			DAIS				ROSIS				AVIRIS
	Case 4 91.57 92.17 92.61 92.01 94.8 95.81 95.48 95.28 95.92	Case 3 91.25 92.09 92.22 91.78 93.95 95.78 95.64 95.15 95.57	Case 2 90.51 91.49 92.45 90.54 93.93 95.76 95.36 94.91 95.6	Case 1 87.95 90.89 91.83 88.75 93.12 95.64 95.2 95.06 95.52	Case 4 64.93 70.11 70.3 63.82 71.11 83.14 78.04 73.28 75.57	Case 3 63.39 69.64 70.13 64.12 69.9 82.89 75.91 72.59 75.73	Case 2 62.82 68.26 67.8 63.35 68.9 80.71 76.37 71.48 75.2	Case 1 59.33 67.38 66.83 61.14 66.77 78.38 71.92 70.45 73.02	Case 4 68.57 80.76 80.35 70.13 83.96 88.6 88.36 86.59 84.01	Case 3 67.74 76.86 77.6 66.82 80.31 87.51 86.58 84.39 81.26	Case 2 62.26 73.12 73.3 61.25 75.82 84.87 84.52 82.03 77.78	Case 1 57.25 66.5 66.98 58.14 71.38 79.65 76.1 76.78 71.66	(PCA) (ICA) (MNF) (LFDA)	CART Bagging AdaBoost RS RF RoF RoF RoF RoF
	95.1 94.8	94.67 94.47	94.17 94.45	93.95 93.76	79.98 80.09	77.86 78.08	77.35 76.97	76.42 71.28	87.06 90.01	84.57 89.13	82.02 87.46	76.82 84.3		SVM LORSAL

  tion of SVMs and MCSs, especially in the context of remote sensing. SVM is a stable classifier. Hence, Bagging-and Boosting-based SVM ensemble cannot improve the accuracy than a single SVM[START_REF] Buciu | Demonstrating the stability of support vector machines for classification[END_REF]. Waske and Benediktsson fused Support Vector Machines (SVM)

for classifying SAR and multi-spectral imagery from agricultural areas

[START_REF] Waske | Fusion of support vector machines for classification of multisensor data[END_REF]

. Waske et al. applied random feature selection technique to construct multiple SVM classifiers for hyperspectral remote sensing image classification

[START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]

. Ceamanos et al. created a SVM classifier ensemble method based on several spectral-similarity band subsets for classifying hyperspectral data

[START_REF] Ceamanos | A classifier ensemble based on fusion of support vector machines for classifying hyperspectral data[END_REF]

. Although SVM ensemble methods proposed by Waske et al.

[START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]

Table 4 . 1 :

 41 Details of experimental settings applied for three hyperspectral datasets.

		AVIRIS	ROSIS	DAIS
	Number of samples per class	10,20,30,40,50	2,4,6,8
	Ensemble size		10,25,50,100
	Number of features in a subset 10,25,50,100 10,25,50	10,20,40
	Base classifier		SVM
	Feature extraction	PCA, RP with Gaussian, Sparse and Bernoulli

Table 4 .

 4

	Methods	Ensemble size	Number of samples per class
		10	20	30	40	50
	SVM	57.18 (68.26)	65.18 (77.26)	70.47 (81.36)	73.55 (83.07)	74.22 (82.11)
		59.35 (72.55)	69.89 (79.39)	74.90 (85.04)	76.95 (85.67)	78.58 (85.44)
	RSSVM	60.28 (73.15)	71.34 (79.32)	75.81 (85.56)	75.91 (85.12)	78.52 (85.39)
		61.46 (74.14)	70.67 (80.01)	74.82 (84.74)	76.04 (86.03)	78.52 (85.37)
		61.43 (73.85)	71.39 (80.19)	75.61 (85.09)	76.15 (86.78)	78.28 (85.13)
		67.28 (77.93)	77.00 (86.75)	78.63 (88.1)	80.14 (88.82)	80.74 (88.83)
	RoSVM-PCA	68.15 (78.11)	76.24 (86.87)	79.13 (88.42)	81.15 (89.11)	81.93 (89.17)
		68.27 (78.15) 77.47 (87.14)	79.65 (88.42)	81.18 (89.14)	82.01 (88.93)
		67.94 (78.46)	76.92 (86.25)	79.68 (88.42)	81.17 (88.67)	82.02 (88.99)
		66.82 (77.12)	75.72 (85.08)	79.39 (88.05)	79.79 (88.87)	80.93 (89.20)
	RoSVM-RP G	67.12 (77.32)	76.04 (85.92)	80.12 (88.15)	80.62 (88.71)	81.64 (89.35)
		67.14 (77.48)	76.72 (85.78)	81.14 (88.74) 81.93 (88.12)	81.88 (89.27)
		66.98 (77.92)	76.34 (85.13)	80.23 (88.69)	81.12 (88.87)	82.93 (89.66)
		64.04 (75.11)	76.76 (85.63)	78.98 (87.84)	79.49 (88.17)	81.18 (89.18)
	RoSVM-RP S	65.25 (76.14)	77.71 (86.04)	79.91 (88.02)	81.11 (88.61)	81.48 (89.18)
		65.85 (76.48)	77.87 (86.18) 80.15 (88.62)	81.52 (88.72)	81.42 (89.21)
		66.01 (76.87)	77.82 (86.13)	79.13 (88.00)	81.00 (88.88)	82.19 (89.04)
		64.49 (75.08)	76.01 (85.32)	79.49 (87.99)	79.78 (88.93)	81.65 (89.03)
	RoSVM-RP B	65.19 (75.87)	76.71 (85.62)	78.99 (87.62)	81.00 (88.49)	81.86 (89.13)
		64.99 (76.01)	77.07 (85.92)	79.02 (87.99)	81.83 (88.98)	82.05 (89.23)
		65.79 (76.28)	77.48 (85.99)	80.11 (88.02)	81.96 (89.02) 82.69 (89.76)

2: Average of OA using SVM and ensemble approaches (the corresponding value of AA is in bracket) applied to different numbers of training samples (Indian Pines AVIRIS image). For ensemble approaches, only highest averages of OA using different values of M are presented.

Table 4 . 3 :

 43 Overall, average and class-specific accuracies obtained from the Indian Pines AVIRIS image. M is set to be 50 and 100 for RSSVM and RoSVM, respectively. T is set to be 10.

	Class	Train Test SVM RSSVM		RoSVM
						PCA	RP G RP S	RP B
	Alfalfa	40	14	92.86	100.00 100.00 100.00 100.00 100.00
	Corn-no till	40	1294 69.15	73.10	74.10 72.67	72.67	73.60
	Corn-min till	40	794	64.74	69.14	76.32	75.94	74.69	76.45
	Bldg-Grass-Tree-Drives	40	194	88.66	90.72	92.78	92.78	93.3	92.78
	Grass/pasture	40	447	86.00	88.84	93.87 93.00	93.22	92.56
	Grass/trees	40	717	93.35	95.90	96.46 96.74 96.04	96.61
	Grass/pasture-mowed	13	13	100.00 100.00 100.00 100.00 100.00 100.00
	Corn	40	449	96.00	96.44	97.10	96.88	96.88	97.33
	Oats	10	10	100.00 100.00 100.00 100.00 100.00 100.00
	Soybeans-no till	40	928	67.89	76.62	78.88 78.56	77.69	78.23
	Soybeans-min till	40	2428 56.87	60.87	65.69	66.23	64.83	66.85
	Soybeans-clean till	40	574	73.17	77.00	85.89 84.67	84.49	83.80
	Wheat	40	172	95.34	95.35	99.42	99.42	99.42	99.42
	Woods	40	1254 87.88	90.91	92.34 91.71	91.55	90.91
	Hay-windrowed	40	340	65.29	66.18	68.53	66.47 69.41	68.82
	Stone-steel towers	40	45	100.00 100.00 100.00 100.00 100.00 100.00
	OA			73.02	76.80	80.14 79.75	79.26	79.97
	AA			83.58	86.32	88.84 88.44	88.39	88.58

Table 4 . 4 :

 44 Average of OA using SVM and ensemble approaches (the corresponding value of AA is in bracket) applied to different numbers of training samples (University of Pavia ROSIS image). For ensemble approaches, highest averages of OA with different values of M are presented.

	Methods	Ensemble size		Number of samples per class
			10	20	30	40	50
	SVM		63.97 (74.03)	65.05 (78.07)	70.73 (80.15)	71.91 (82.87)	73.86 (83.36)
		10	64.38 (75.85)	68.38 (80.22)	71.69 (80.63)	72.99 (83.05)	74.10 (83.44)
	RSSVM	25	65.67 (76.13)	70.12 (81.12)	71.98 (80.93)	73.09 (83.15)	74.70 (83.54)
		50	66.48 (76.13)	70.38 (80.89)	71.74 (80.91)	73.25 (83.63)	74.96 (83.73)
		100	67.12 (76.99)	70.01 (80.32)	71.86 (80.63)	73.15 (83.65)	74.92 (83.97)
		10	72.37 (78.61)	72.20 (81.52)	75.43 (80.97)	75.57 (83.97)	75.89 (85.54)
	RoSVM-PCA	25	73.12 (78.89) 73.42 (81.74) 76.14 (81.11)	76.27 (84.02)	75.82 (85.43)
		50	73.01 (78.94)	73.18 (81.92)	76.64 (81.23) 76.13 (84.00)	75.99 (85.77)
		100	72.98 (78.69)	73.28 (81.73)	76.01 (80.97)	76.42 (83.92)	75.98 (85.68)
		10	69.74 (77.82)	70.74 (82.07)	75.00 (81.9)	77.09 (85.02)	76.66 (86.20)
	RoSVM-RP G	25	70.51 (77.94)	71.28 (82.34)	75.97 (82.4)	76.93 (84.98)	76.99 (86.23)
		50	70.64 (78.02)	71.11 (82.52)	76.06 (82.67)	76.95 (84.98)	77.01 (86.42)
		100	70.28 (77.91)	71.64 (82.67)	76.02 (82.53)	77.12 (85.12) 77.00 (86.45)
		10	70.05 (77.71)	70.76 (80.15)	74.64 (80.60)	75.62 (84.19)	76.79 (86.37)
	RoSVM-RP S	25	70.65 (77.99)	71.25 (81.02)	74.11 (80.96)	75.92 (84.28)	76.92 (86.42)
		50	70.99 (78.11)	71.34 (80.92)	75.16 (81.45)	76.37 (84.67)	76.95 (86.51)
		100	70.42 (77.90)	71.02 (81.13)	75.24 (81.62)	76.58 (84.74)	76.63 (86.64)
		10	69.94 (77.99)	70.57 (80.38)	74.86 (81.07)	75.13 (84.45)	77.48 (86.69)
	RoSVM-RP B	25	70.12 (78.08)	71.24 (80.99)	75.77 (81.97)	75.98 (84.97)	77.12 (86.52)
		50	70.43 (78.11)	71.38 (81.30)	75.22 (81.45)	76.02 (85.02)	77.53 (86.61)
		100	70.38 (77.92)	71.46 (81.41)	75.41 (81.22)	75.82 (84.86)	77.42 (86.67)

for the training set (40 samples per class) while RoSVM-RP B yielded the highest OA with M = 10.

Table 4 . 5 :

 45 Averages of OA and AA using SVM and ensemble approaches (the back bracket is the corresponding value of AA) applied to different numbers of training samples (Pavia Center DAIS image).

			Methods		Ensemble size	Number of samples per class
								2	4	6	8
			SVM					71.51 (70.63)	80.03 (80.22)	86.05 (84.53)	90.60 (87.99)
						10		73.85 (73.85)	85.98 (84.07)	89.39 (88.29)	92.70 (89.44)
			RSSVM		25		74.13 (73.12)	86.13 (84.12)	89.78 (88.42)	92.43 (89.35)
						50		74.22 (74.93)	86.07 (84.14)	89.97 (88.54)	93.10 (89.56)
						100		74.16 (73.97)	85.99 (84.05)	89.65 (88.66)	92.97 (89.16)
						10		76.70 (74.85)	86.01 (86.84)	90.53 (89.84)	94.66 (91.87)
			ROSVM-PCA	25		76.79 (74.93)	86.81 (87.02)	91.47 (89.95)	94.56 (91.76)
						50		76.98 (75.02)	86.94 (87.11)	91.55 (89.99)	95.00 (92.02)
						100		76.70 (74.85)	86.25 (86.92)	91.11 (89.90)	94.78 (91.99)
						10		74.69 (73.39)	89.76 (89.86)	90.78 (89.70)	93.38 (90.50)
			RoSVM-RP G	25		75.39 (73.49)	89.12 (89.90)	90.12 (89.14)	93.97 (90.78)
						50		75.47 (73.51)	89.92 (89.67)	90.17 (89.02)	93.87 (90.61)
						100		75.58 (73.67)	89.97 (89.78)	90.68 (89.09)	93.71 (90.54)
						10		77.66 (75.87)	90.31 (90.08)	91.68 (91.10)	93.47 (90.52)
			RoSVM-RP S	25		77.97 (75.96)	90.75 (90.64)	91.79 (91.23)	93.59 (91.12)
						50		78.02 (76.00) 91.11 (90.79) 91.98 (91.45)	93.67 (91.38)
						100		77.99 (76.02)	91.01 (90.54)	92.02 (91.87)	93.52 (90.78)
						10		76.80 (74.85)	89.24 (88.92)	91.22 (91.04)	93.66 (90.82)
			RoSVM-RP B	25		77.32 (75.99)	90.13 (89.24)	91.78 (91.54)	94.12 (90.13)
						50		77.01 (75.21)	90.01 (89.64)	91.62 (91.47)	94.32 (91.12)
						100		77.34 (76.08)	90.12 (89.22)	91.99 (91.78) 94.30 (91.27)
		90
		88
		86
	OA(%)	84
		82		SVM
				RSSVM
		80		RoSVM-PCA
				RoSVM-RP G
		78		RoSVM-RP S
				RoSVM-RP B
		76
		10	15	20	25	30	35	40
			Number of features in a subset (M)
				(a)

Table 5 . 1 :

 51 Average of overall accuracies obtained from the proposed methods using different numbers of training samples (10 Monte Carlo runs) for the Indian Pines AVIRIS image. The numbers of training samples per class and the total numbers of training samples (in brackets) are also given.

	Methods			Number of training samples
		10 (160)	20 (320)	30 (423)	40 (583)	50 (723)	60 (830)
	RoF-PCA	50.68±1.82	60.57±1.93	64.98±1.59	66.86±1.67	68.65±1.49	70.21±1.23
	RoF-NPE	52.18±0.98	62.08±1.40	66.38±1.58	67.58±1.41	70.47±1.12	71.48±1.02
	RoF-LLTSA	53.20±1.04	63.61±1.49	67.27±1.58	68.62±1.58	70.56±1.48	71.47±0.89
	RoF-LPP	54.81±1.87	63.92±1.91	66.86±1.65	67.96±0.69	70.31±0.88	72.61±1.17
	RoF-PCA-MRF	71.14±3.65	77.41±2.94	80.92±3.85	83.79±1.40	86.74±2.91	89.51±1.88
	RoF-NPE-MRF	73.91±2.28	80.47±1.85	85.23±2.65 88.31±2.01 88.07±1.37 90.97±1.58
	RoF-LLTSA-MRF 73.39±2.64	81.91±1.32	83.87±3.04	84.57±2.22	87.79±2.86	90.06±1.84
	RoF-LPP-MRF	76.88±1.74 83.37±1.33 85.12±2.30	87.32±1.78	88.25±1.92	90.57±2.32

Table 5 .

 5

	Methods		M
		10	25	50	100
	RoF-PCA	60.57±1.93	63.28±1.71	64.31±2.22	66.08±1.55
	RoF-NPE	62.08±1.40	66.86±1.22	67.63±1.75	70.09±2.09
	RoF-LLTSA	63.61±1.49	64.19±1.50	66.74±1.51	64.51±1.53
	RoF-LPP	63.92±1.91	64.46±1.72	66.59±2.41	67.85±1.51
	RoF-PCA-MRF	77.41±2.94	82.03±2.97	82.30±3.07	83.31±4.26
	RoF-NPE-MRF	80.47±1.85	83.25±2.56 84.47±3.44	85.34±2.58
	RoF-LLTSA-MRF 81.41±1.32	83.14±3.43	83.01±3.20	83.17±3.97
	RoF-LPP-MRF	83.37±1.33 82.93±2.78	84.87±1.74	83.86±2.72

2: Average of overall accuracies obtained from the proposed Rotation Forests and MRF using different values of M (Indian Pines AVIRIS image).

Table 5 . 3 :

 53 Average of overall accuracies obtained from the proposed Rotation Forests and MRF using different values of τ (Indian Pines AVIRIS image).

	Methods

Table 5 . 4 :

 54 Overall, average and class-specific accuracies obtained for the Indian Pines AVIRIS image.

	LORSAL-MRF		100	68.13	89.33	100	98.19	100	96.15	98.98	100	81.71	69.33	97.23	100	89.95	99.47	98.95	84.13	92.96
	RoF RoF-MRF	PCA NPE LLTSA LPP PCA NPE LLTSA LPP	92.59 92.59 94.44 87.04 94.44 98.15 98.15 96.3	63.88 72.03 70.92 61.02 73.92 81.31 82.29 72.66	60.43 58.15 59.35 67.87 56.83 74.70 89.20 88.84	81.62 87.61 86.75 85.9 100 100 99.57 98.29	87.12 81.69 85.51 90.94 90.14 86.52 92.76 92.15	90.63 92.77 87.82 86.88 99.33 100 97.32 96.65	92.31 96.15 100 96.15 96.15 100 100 96.15	88.75 90.59 86.71 89.57 97.95 99.39 98.98 97.55	100 100 100 95 100 100 100 100	75.62 67.25 72.52 76.65 84.09 83.88 92.05 84.61	58.51 57.98 53.77 61.39 84.85 85.53 84.04 88.7	66.78 69.87 54.72 72.14 98.53 95.93 97.88 98.37	99.06 98.58 98.11 99.53 99.53 99.52 100 100	85.47 78.9 91.42 84.31 82.61 76.82 75.89 81.68	60.53 75.26 47.89 52.37 95.26 98.16 98.95 97.63	98.94 100 97.89 100 100 100 100 100	72.11 72.18 72.89 73.01 84.73 86.53 88.36 87.92	81.39 82.47 82.49 81.67 90.85 92.49 94.19 93.6
	Train Test SVM LORSAL		30 54 90.74 98.15	30 1434 67.78 60.53	30 834 63.43 67.51	30 234 90.17 84.19	30 497 86.92 95.57	30 747 85.27 95.45	13 26 92.31 96.15	30 489 95.91 96.11	10 20 85 100	30 968 73.86 72.93	30 2468 58.55 55.88	30 614 60.91 79.8	30 212 99.53 100	30 1294 83.62 87.87	30 380 70.26 78.95	30 95 96.84 94.74	72.6 74.26	81.32 85.24
	Class		Alfalfa	Corn-no till	Corn-min till	Bldg-Grass-Tree-Drives	Grass/pasture	Grass/trees	Grass/pasture-mowed	Corn	Oats	Soybeans-no till	Soybeans-min till	Soybeans-clean till	Wheat	Woods	Hay-windrowed	Stone-steel towers	OA	AA

Table 5 . 5 :

 55 Average of overall accuracies obtained from the proposed approaches using different numbers of training samples for the University of Pavia ROSIS image. The number of training samples per class and the total number of training samples (in brackets) are also given.

	Methods		The number of training samples
		10 (90)	20 (180)	30 (270)	40 (360)	50 (450)
	RoF-PCA	61.81±3.42	67.00±2.87	69.71±3.56	69.06±3.20	73.17±3.93
	RoF-NPE	61.05±2.49	65.59±2.79	68.70±3.98	71.18±2.93	73.43±3.54
	RoF-LLTSA	63.81±1.63	66.36±2.56	72.83±2.57	71.81±2.28	74.07±3.28
	RoF-LPP	61.04±2.45	69.34±2.00	73.90±3.25	71.71±2.74	74.03±2.89
	RoF-PCA-MRF	67.91±4.49	74.45±3.02	77.21±3.23	77.76±4.83	80.22±4.38
	RoF-NPE-MRF	66.26±3.51	72.24±2.93	76.12±3.13	79.88±4.16	81.13±3.65
	RoF-LLTSA-MRF 72.30±3.63 74.35±2.80	82.61±2.93	80.88±3.87	82.41±3.92
	RoF-LPP-MRF	69.14±3.56	76.86±3.16 83.45±3.65 79.91±3.73	82.94±3.50

Table 5 .

 5

	Methods		M
		10	25	50
	RoF-PCA	67.00±2.87	66.64±2.76	66.22±3.83
	RoF-NPE	65.59±2.79	66.51±3.05	66.36±3.22
	RoF-LLTSA	66.36±2.56	66.77±2.68	64.07±2.98
	RoF-LPP	69.34±3.00	67.09±2.23	65.88±3.92
	RoF-PCA-MRF	74.45±3.02	74.73±4.48	77.80±4.45
	RoF-NPE-MRF	74.24±2.93	77.61±3.48 79.48±3.87
	RoF-LLTSA-MRF 74.35±2.80	76.65±3.60	77.60±3.31
	RoF-LPP-MRF	76.86±3.16 76.77±4.12	77.02±3.81

6: Average of overall accuracies obtained from the proposed approaches using different values of M (University of Pavia ROSIS image). The number of training samples per class is 20, and the total number of training samples is 180.

Table 5 . 7 :

 57 Average of overall accuracies obtained from the proposed methods using different values of τ (University of Pavia ROSIS image). The number of training samples per class is 20, and the total number of training samples is 180.

	Methods

Table 5 . 8 :

 58 Overall, average and class-specific accuracies obtained for the University of Pavia ROSIS image.

			Bitumen	Asphalt	Gravel	Meadows	Trees	Bare Soil	Metal Sheets	Shadows	Bricks		Class
	AA	OA	392	548	265	532	231	540	375	514	524		Train
	88.28 87.7	80.99 80.11	1330 81.58 74.27	6631 83.71 82.43	2099 70.32 99.26	18649 70.25 69.08	3064 97.81 95.46	5029 92.25 92.96	1345 99.41 89.85	947 96.62 89.54	3682 92.59 96.41		Test SVM LORSAL
	87.25 87.82 88.19 88.64 91 91.61 92.23 93.06	82.53 83.41 83.40 84.76 90.44 90.98 90.84 92.15	90.53 89.1 91.43 91.2 94.59 95.26 95.41 95.49	83.44 83.86 84.29 82.78 93.27 93.43 93.64 92.91	61.51 62.41 66.75 67.65 55.55 62.98 69.80 66.46	75.32 76.5 76.42 79.69 87.78 87.34 86.76 89.77	91.51 93.9 91.12 93.02 91.74 96.18 91.97 95.53	95.63 95.8 95.33 95.76 99.26 99.68 99.68 99.92	99.11 99.33 99.41 99.11 99.63 99.55 99.93 99.93	96.83 98.1 97.36 97.57 98.52 97.15 98.94 99.37	91.39 92.29 91.63 90.98 98.64 98.51 98.34 98.18	PCA NPE LLTSA LPP PCA NPE LLTSA LPP	RoF RoF-MRF
	92.54	85.57	72.42	96.71	99.78	72.36	97.57	98.35	98.5	99.29	97.85		LORSAL-MRF

Table 5 . 9 :

 59 Overall, average and class-specific accuracies obtained from the proposed approaches (Pavia Center DAIS image).

	LORSAL-MRF		100	95.3	98.08	83.05	93.37	92.68	97.99	98.25	99.59	96.02	95.37
		LPP	100	95.17	98.16	86.71	93.02	98.95	98.48	99.12	99.59	96.59	96.62
	RoF RoF-MRF	PCA NPE LLTSA LPP PCA NPE LLTSA	100 100 100 100 100 100 100	93.89 94.8 95.54 95.3 94.88 95.59 95.05	97.04 97.84 98.16 97.20 98.96 98.16 99.2	87.72 84.74 91.86 85.22 86.03 87.46 93.36	89.85 92.08 92.72 91.84 93.19 92.66 95.54	97.5 96.52 98.26 98.95 98.95 97.91 98.95	97.45 99.11 98.44 97.85 97.91 97.76 98.88	98.39 97.23 98.25 98.39 97.23 96.5 97.23	92.12 91.29 87.14 92.53 99.17 100 95.85	95.65 95.78 96.98 96.07 96.34 96.46 97.52	94.78 94.85 95.75 95.32 96.14 96.34 97.12
	Train Test SVM LORSAL		202 4281 100 100	205 2424 94.42 95.17	206 1251 97.11 96.4	315 2237 75.45 75.66	205 1475 92.28 93.78	204 1704 90.1 95.12	202 685 96.82 98.39	201 287 93.47 92.85	119 241 92.01 86.31	OA 94.56 94.8	AA 92.31 92.63
	Class		Water	Trees	Meadows	Bricks	Soil	Asphalt	Bitumen	Parking lot	Shadows

  Dalla Mura et al. presented a technique based on EAPs and independent component analysis (ICA) for the classification of urban hyperspectral images [43]. Prashanth et al. explored the use of APs based on three supervised and two unsupervised feature extraction techniques for the classification of hyperspectral data with SVM and Random Forest classifiers [148]. Pedergnana et al. proposed a classification approach of features extracted with EAPs computed on both optical and LiDAR images, leading to the integration of spectral, spatial and elevation data [149]. Pedergnana et al. proposed a novel iterative technique based on genetic algorithm to select the optimal features from the EMAPs [150]. Falco et al. investigated the performance of change detection in very high resolution image based on APs [151]. Li et al. presented a generalized composite kernel framework for hyperspectral image classification by combing spectral and spatial

information (EMAPs)

[START_REF] Li | Generalized composite kernel framework for hyperspectral image classification[END_REF]

. Bernabe et al. proposed a new strategy combing EMAPs and kernel principal components analysis (KPCA) for the classification of multi/hyperspectral images [153]. Song et al. applied a sparse representation-based learning approach

Table 6 . 1 :

 61 Individual and ensemble classification approaches considered for the study

	Individual classifiers	(Notation)	Classifier ensembles	(Notation)
			Random subspace with DT	RSDT
	Decision tree	DT	Random Forest	RF
			Rotation Forest	RoF
			Random Rotation Forest	RoRF
	Extreme learning machine	ELM	Random subspace with ELM	RSELM
			Rotation subspace with ELM	RoELM

•

  Computation time: all methods were implemented in Matlab on a computer having Inter(R) Xeon(R) 2 CPU, 2.8 GHz and 12GB of memory. Random Forest implementation in Matlab Mex files written in C is downloaded from the website:

http://code.google.com/p/randomforest-matlab/. The source code of ELM can be assessed from the website: http://www.ntu.edu.sg/home/egbhuang/elm_ codes.html.

Table 6 . 2 :

 62 Indiana Pines AVIRIS image: class name and number of samples in Ground truth

		Class	Number
	Number	Name	Ground Truth
	1	Alfalfa	54
	2	Corn-no till	1434
	3	Corn-min till	834
	4	Bldg-Grass-Tree-Drives	234
	5	Grass/pasture	497
	6	Grass/trees	747
	7	Grass/pasture-mowed	26
	8	Corn	489
	9	Oats	20
	10	Soybeans-no till	968
	11	Soybeans-min till	2468
	12	Soybeans-clean till	614
	13	Wheat	212
	14	Woods	1294
	15	Hay-windrowed	380
	16	Stone-steel towers	95

Table 6 . 3 :

 63 University of Pavia ROSIS image: class name and number of training and test samples

		Class	Number of samples
	Number	Name	Train	Test
	1	Bricks	524	3682
	2	Shadows	514	947
	3	Metal Sheets	375	1345
	4	Bare Soil	540	5029
	5	Trees	231	3064
	6	Meadows	532	18649
	7	Gravel	265	2099
	8	Asphalt	548	6631
	9	Bitumen	392	1330

Table 6 . 4 :

 64 Overall accuracies (in percent) and average accuracies (in the parentheses) obtained for different classification algorithms using different number of training samples per class when applied to the Spectral information of AVIRIS Indian Pines Hyperspectral data.

	Samples per class DT	RSDT	RF	RoF
	5	29.64±3.61(39.62) 36.41±3.49(47.17) 42.87±3.65(53.79) 47.79±3.23(61.05)
	10	38.85±4.03(49.87) 46.57±2.48(58.33) 49.89±3.16(60.86) 57.33±2.26(69.95)
	15	40.43±2.42(51.79) 49.62±1.18(61.91) 51.13±1.52(63.73) 63.03±1.65(73.79)
	20	43.13±2.35(55.15) 53.82±2.07(65.78) 55.52±2.50(66.92) 68.98±2.71(79.89)
	25	46.59±1.32(57.54) 55.71±1.18(66.78) 57.23±1.56(68.52) 71.81±1.80(81.19)
	30	47.49±1.35(58.24) 58.23±2.36(67.98) 60.07±2.10(70.47) 72.65±2.27(81.36)
	35	47.90±2.56(57.75) 59.82±1.31(69.12) 61.48±1.59(71.10) 74.36±0.58(82.54)
	40	49.05±2.06(58.33) 60.85±1.27(70.10) 62.66±1.52(71.95) 74.46±1.19(82.97)
	45	50.42±2.14(60.79) 62.22±1.24(72.08) 63.68±0.93(73.25) 76.58±1.26(84.90)
	50	50.51±2.63(59.48) 62.63±0.99(71.60) 64.20±0.58(77.51) 75.96±1.06(84.39)
	Samples per class RoRF	ELM	RSELM	RoELM
	5	51.14±3.22(62.66) 51.15±2.58(65.8)	55.39±3.08(69.69)	58.72±2.06(71.87)
	10	58.39±1.78(69.58) 57.17±1.92(71.06) 65.11±1.83(76.35)	69.84±1.27(79.67)
	15	61.49±1.71(73.01) 58.51±1.80(71.99) 68.69±1.59(79.7)	72.93±1.07(83.43)
	20	67.56±2.55(78.54) 57.68±2.02(70.75) 70.73±1.18(80.36)	75.95±0.82(85.22)
	25	70.17±1.46(79.43) 62.03±1.58(76.34) 72.67±0.97(83.71)	77.13±0.94(86.51)
	30	70.79±1.77(80.03) 61.88±1.10(73.96) 75.61±1.11(85.49)	78.24±0.67(86.87)
	35	72.78±1.17(81.23) 66.73±1.62(76.98) 74.86±1.53(84.31)	78.89±1.07(87.17)
	40	73.45±1.53(81.64) 66.44±1.13(77.79) 75.46±0.80(85.05)	80.08±0.5(88.24)
	45	74.81±1.37(83.37) 67.46±1.05(77.31) 77.43±0.31(86.64)	80.34±0.25(88.08)
	50	74.56±1.26(82.21) 67.65±1.69(77.11) 77.85±1.09(87.57)	81.19±0.7(89.00)

Table 6 . 5 :

 65 Overall accuracies (in percent) and average accuracies (in the parentheses) obtained for different classification algorithms using different number of training samples per class when applied to the EMAPs of AVIRIS Indian Pines Hyperspectral data. The reported standard deviation is obtained from ten Monte Carlo runs.

	Samples per class DT	RSDT	RF	RoF
	5	55.07±6.68(65.95) 57.48±5.51(70.24) 65.69±5.58(77.51) 66.22±4.41(76.91)
	10	70.22±5.01(80.19) 73.84±4.59(82.36) 77.21±3.43(85.77) 78.29±2.94(76.91)
	15	76.04±1.12(83.34) 80.34±2.28(85.96) 83.18±1.78(89.26) 83.26±1.9(88.25)
	20	80.82±2.32(85.98) 82.69±1.75(87.84) 84.46±1.80(89.44) 85.36±1.53(89.29)
	25	81.62±2.24(87.64) 84.37±2.14(89.95) 87.54±1.34(92.1)	87.92±1.31(92.10)
	30	84.17±2.23(88.38) 87.07±2.82(90.27) 88.66±1.06(92.94) 89.32±1.67(92.60)
	35	85.13±3.11(89.15) 87.51±2.25(91.72) 88.93±1.33(92.66) 89.84±1.06(93.07)
	40	85.14±1.71(88.52) 87.72±1.82(90.91) 89.85±1.23(93.29) 90.44±0.98(92.98)
	45	85.68±1.48(89.31) 89.12±1.91(92.42) 90.81±0.97(93.69) 91.34±0.96(93.87)
	50	87.08±1.53(90.43) 90.27±1.21(92.82) 91.61±0.79(94.01) 92.20±0.85(94.59)
	Samples per class RoRF	ELM	RSELM	RoELM
	5	70.81±4.59(81.64) 73.24±5.28(81.44) 74.24±5.38(81.55)	75.97±4.01(83.59)
	10	80.98±2.45(88.31) 82.47±2.71(87.17) 83.36±2.33(87.69)	85.19±2.32(89.43)
	15	86.01±1.65(90.78) 83.92±2.94(88.02) 85.76±2.67(89.25)	87.45±2.05(91.4)
	20	87.41±1.45(91.55) 83.98±2.41(87.31) 87.67±1.49(90.15)	89.35±1.09(92.19)
	25	89.30±1.83(93.48) 87.02±2.60(90.7)	88.37±1.69(91.61)	90.70±1.33(94.02)
	30	91.05±1.60(94.46) 89.65±1.78(92.28) 90.73±1.60(92.82)	92.14±1.45(94.71)
	35	91.29±0.74(94.00) 89.48±1.70(91.73) 90.76±1.20(92.45)	92.77±0.98(94.99)
	40	91.99±1.20(94.45) 88.76±1.17(91.03) 91.01±1.25(92.54)	92.99±0.64(94.8)
	45	92.71±0.93(95.16) 91.4±1.24(93.73)	93.71±0.65(95.41)	93.97±0.60(95.61)
	50	93.31±0.42(95.33) 91.32±0.86(93.35) 94.29±0.43(95.31)	94.53±0.41(95.25)

Table 6 . 6 :

 66 The parameters used for ELM and RS ensemble classifiers (Indiana Pines AVIRIS image)

	Features Methods T	M	δ	Features Methods T	M	δ
		RSDT	20 110 -		RSDT	20 102 -
		RF	20 15	-		RF	20 15	-
		RoF	20 110 -		RoF	20 3	-
	Spectral	RoRF	20 110 -	EMAPs	RoRF	20 3	-
		ELM	-	-	256		ELM	-	-	256
		RSELM	20 110 256		RSELM	20 102 256
		RoELM	20 110 256		RoELM	20 3	256

Table 6 . 7 :

 67 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10% number of samples in Ground Truth as training samples when applied to the spectral information of Indiana Pines AVIRIS Hyperspectral data. The reported results are achieved by ten Monte Carlo runs.

	Class	DT RSDT RF	RoF RoRF ELM RSELM RoELM
	1	38.16 28.77 15.71 53.06 29.80 15.11	25.71	49.39
	2	50.76 65.79 61.82 79.86 74.38 73.01	79.95	83.13
	3	45.09 53.86 50.55 69.31 62.88 59.01	61.89	71.85
	4	26.87 34.69 30.76 63.32 50.76 36.59	54.17	62.09
	5	67.63 79.04 79.98 88.37 83.06 90.56	93.17	91.48
	6	78.23 92.14 92.37 94.72 93.99 95.55	97.49	94.64
	7	20.43 16.09 14.78 59.13 50.87	3.04	8.02	46.52
	8	85.93 92.77 96.16 97.73 98.20 99.43	99.57	98.32
	9	1.67	0.56	6.11	0	2.78	4.44	12.22	37.77
	10	47.01 63.85 61.76 77.19 76.87 64.02	68.56	75.67
	11	62.02 80.56 83.88 87.35 90.53 80.77	87.72	87.87
	12	29.73 42.28 47.25 69.58 68.72 67.09	75.51	82.64
	13	80.94 90.94 92.93 97.91 97.07 99.63	99.58	98.53
	14	89.57 93.67 94.13 96.22 96.36 95.55	96.94	97.36
	15	35.96 39.77 39.33 55.85 45.09 59.97	60.26	54.77
	16	54.59 71.88 81.06 88.35 89.53 46.12	70.59	73.29
	OA	59.77 72.53 72.84 83.14 81.31 77.46	82.38	84.70
	AA	50.79 57.98 59.29 73.62 69.12 61.89	68.23	75.33
	κ	54.13 68.44 68.70 80.70 78.49 74.11	79.74	82.44
	Time(s) 1.49	9.25	0.85 26.95 18.11	0.22	6.18	14.63

Table 6 . 8 :

 68 Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10% number of samples in Ground Truth as training samples when applied to the EMAPs of Indiana Pines AVIRIS Hyperspectral data. The reported results are achieved by ten Monte Carlo runs.

	Class	DT RSDT RF	RoF RoRF ELM RSELM RoELM
	1	82.65 83.67 87.14 87.35 87.14 74.08	86.94	87.76
	2	86.14 91.01 91.00 91.11 91.55 90.21	90.12	90.33
	3	92.65 95.5 95.31 95.63 96.51 97.48	98.75	98.95
	4	74.55 87.11 89.15 87.57 91.66 88.34	94.27	95.02
	5	89.53 92.37 92.51 93.31 93.20 91.28 94.63	94.36
	6	94.15 95.33 97.22 96.95 98.07 97.75	99.12	99.32
	7	23.48 23.04 73.91 40.43 84.78 88.28 96.09	96.09
	8	100	99.77 99.77 99.75 99.80 96.23	99.39	99.55
	9	69.44 61.11 92.77 62.78 98.33 80.56	89.44	95.56
	10	84.43 86.89 88.43 87.50 89.06 92.61	90.55	91.56
	11	94.67 96.18 97.94 96.74 98.55 96.53	98.49	98.66
	12	85.14 90.29 92.28 90.22 93.06 86.20	89.19	89.17
	13	99.16 98.84 99.11 99.42 99.53 98.95	99.48	99.48
	14	98.57 99.22 99.25 99.23 99.24 96.29	99.16	99.42
	15	93.27 96.40 97.63 96.35 98.63 75.38	92.40	94.36
	16	96.12 97.41 98.00 97.65 98.12 0.71	50.51	32.71
	OA	91.57 94.05 95.17 94.56 95.83 92.59	95.46	95.40
	AA	85.23 87.13 93.21 90.00 94.83 84.43	92.32	91.39
	κ	90.41 93.22 94.36 93.80 95.24 91.64	94.82	94.73
	Time(s) 0.77	4.55	0.63 13.99 14.07 0.21	4.08	13.59

Table 6 . 9 :

 69 Overall accuracies, average accuracies and class-specific accuracies obtained for different classification algorithms using the entire training set when applied to the spectral information of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs.

	Class	DT RSDT RF	RoF RoRF ELM RSELM RoELM
	1	83.46 91.68 90.17 92.55 93.29 90.7	95.05	91.76
	2	92.93 97.42 97.44 98.30 99.60 99.65	99.69	99.89
	3	96.95 98.99 98.82 99.58 99.55 85.64	98.95	99.85
	4	76.29 81.56 77.80 95.60 95.55 94.92	96.14	97.62
	5	97.75 98.67 98.58 95.62 98.79 96.68	97.11	95.33
	6	52.35 53.12 56.10 74.61 65.38 58.76	64.32	69.19
	7	54.79 51.32 53.79 58.49 57.54 70.18	68.06	63.43
	8	71.93 79.60 80.07 84.55 85.34 77.21	80.50	76.02
	9	76.62 83.68 84.63 89.93 90.39 88.01	91.08	90.90
	OA	67.30 70.44 71.37 82.66 79.04 74.56	78.45	79.44
	AA	78.11 81.78 81.93 87.69 87.27 84.64	87.88	87.11
	κ	60.15 63.90 64.79 78.09 73.98 68.75	72.84	74.25
	Time(s) 1.98 20.50 2.33 44.74 53.41	1.56	34.65	51.45

Table 6 .

 6 10: Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using the entire training set when applied to the EMAPs of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs.

	Class	DT RSDT	RF	RoF	RoRF ELM RSELM RoELM
	1	98.02 98.95 98.94 98.61 99.16 98.96	99.51	99.58
	2	85.96 92.47 97.33 97.00 99.32 98.37	99.31	98.38
	3	99.55 99.58 99.62 99.62 99.62 96.51 99.67	99.56
	4	98.95 96.55 96.34 99.39 97.35 97.61 99.96	99.90
	5	89.69 97.10 99.12 94.55 99.23 94.54	98.52	97.45
	6	90.85 91.66 97.28 93.65 97.42 96.42	98.35	98.55
	7	67.13 80.63 73.05 85.45 75.15 87.88	98.08	99.38
	8	91.34 94.26 95.14 93.52 95.32 97.16 97.69	97.54
	9	99.32 100.00 100.00 100.00 100.00 99.92 99.93	99.92
	OA	91.67 93.61 96.08 94.85 96.47 96.49	98.67	98.69
	AA	91.20 94.58 95.20 95.75 95.84 96.37 99.00	98.92
	κ	89.14 91.71 94.83 93.26 95.34 95.37	98.21	98.25
	Time(s) 1.38 29.72	2.82	37.93 70.24 1.83	37.66	71.59

Table 6 .

 6 11: Overall accuracies, average accuracies and κ obtained from the proposed methods: RoELM and RSELM with other Spatial-spectral classifier for the University of Pavia ROSIS image

	Classifier	SVM+Clustering [36]	MLRsubMLL [163]	GCK [152]
	OA	94.68	94.10	98.09
	AA	95.21	93.45	97.76
	κ	92.92	92.24	97.46
	Classifier Mixed lasso 3D-DWT [143] RSELM EMAPs RoELM EMAPs
	OA	98.15	98.67	98.69
	AA	97.56	99.00	98.92
	κ	97.48	98.21	98.25
	using only spectral information presented in

Table 6 .

 6 12: Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10 samples per class when applied to the spectral information of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs.

	Class	DT	RSDT	RF	RoF	RoRF	ELM	RSELM RoELM
	1	64.06	66.11	69.78 73.67 80.33 52.83	61.17	54.08
	2	87.71	91.22	96.58 95.21 98.36 92.83	93.36	97.56
	3	92.92	96.39	95.73 98.61 99.20 40.37	13.37	94.59
	4	44.66	51.76	51.06 82.51	77.54	52.54	55.26	84.55
	5	75.78	84.35	88.54 89.24 96.81 55.41	41.34	89.13
	6	36.08	39.71	46.02 47.65	48.54	50.08	59.19	48.29
	7	37.66	40.96	43.83 48.64	46.34	50.27	56.91	61.98
	8	58.84	60.97	65.06 63.85 66.71 47.99	54.97	49.75
	9	64.08	68.71	78.63 84.31 87.94 61.36	67.21	73.96
	OA	49.79	53.79	58.24 63.32 64.77 51.67	56.41	60.22
	AA	62.45	66.69	70.58 75.97 77.98 55.97	55.86	72.63
	κ	40.02	44.36	49.24 55.81 57.51 40.96	45.74	52.12
	Time	0.29	2.51	0.63	10.13	19.74	2.6	41.86	80.67

Table 6 .

 6 [START_REF] Guyon | Feature Extraction : Foundations and Applications[END_REF]: Overall accuracies, average accuracies and class-specific obtained for different classification algorithms using 10 samples per class when applied to the EMAPs of University of Pavia ROSIS image. The reported results are achieved by ten Monte Carlo runs.

	Class	DT	RSDT	RF	RoF	RoRF	ELM	RSELM RoELM
	1	84.63	87.66	89.04 89.14	90.85	56.65	71.11	94.81
	2	82.48	89.59	98.26 90.61	98.59	95.77	99.88	97.42
	3	85.92	91.23	94.71 90.21	97.70	88.91	99.43	91.44
	4	84.85	85.74	82.26 87.52	87.11	73.24	95.85	95.27
	5	84.35	88.55	92.84 92.25	94.62	87.00	94.80	93.18
	6	79.48	82.18	87.37 86.81	87.72	81.80	90.79	88.38
	7	63.08	67.38	76.10 77.73	76.21	66.61	85.83	89.39
	8	80.62	85.15	90.71 87.01	91.18	88.61	95.26	95.75
	9	98.66	99.44	99.10 99.21	99.45	99.62	99.95	99.91
	OA	81.14	84.24	88.11 88.40	89.30	80.43	91.19	91.93
	AA	82.67	86.32	90.04 89.20	91.49	82.02	92.55	93.95
	κ	75.93	79.77	84.54 84.95	86.10	74.75	88.54	89.55
	Time	0.32	2.47	0.79	15.02	27.56	2.87	58.35	114.35

  [START_REF] Chang | Hyperspectral Imaging : Techniques for Spectral Detection and Classification[END_REF], due to the limited training set. For instance, the OA and AA of RoRF EMAPs are 96.47% and 95.87% for the original training set, whereas using limited training samples, the OA and AA of RoRF EMAPs are 89.30% and 91.49%. Nevertheless, with a very small training set, the results using the combination of RS ensembles and EMAPs are still very good. Considering the processing time, with the limited training set, the processing time of DT and DT ensemble is significantly reduced. The computational cost of ELM and its ensembles with limited training samples is higher than those of ELM and its ensemble with entire training set, because we used more hidden nodes (δ = 512) to generate better performances.

•

  In Chapter 3, Rotation Forest, has been applied to hyperspectral remote sensing image classification. The framework of Rotation Forest is to project the original data into a new feature space using transformation methods for each base classifier (decision tree), then the base classifier can train in different new spaces for the purpose of encouraging both individual accuracy and diversity within the ensem-RoSVM), which combines SVM and MCS together. The basic idea of RoSVM is to generate diverse SVM classification results using random feature selection and feature extraction, which can enhance both individual accuracy and diversity within the ensemble simultaneously. Two simple feature extraction methods: principal component analysis (PCA) and random projection (RP), are chosen for feature extraction in RoSVM. Empirical study on three hyperspectral datasets demonstrates that the proposed RoSVM ensemble methods outperform the regular SVM and random subspace SVM (RSSVM). The impacts of the parameters on the overall accuracy of RoSVM (different training sets, ensemble sizes and numbers of features in the subset) are also investigated in this study.• Chapter 5 extends the work in Chapter 3 by incorporating spatial contextual information with Markov random field to design robust spatial-spectral methods. In the first step, the weak classifier of hypersepctral data, classification and regression tree (CART) is selected as the base classifier because it is unstable, fast and sensitivity to rotation axes. We adapt four feature extraction methods, including principal component analysis (PCA), neighborhood preserving embedding (NPE), linear local tangent space alignment (LLTSA) and linearity preserving projection (LPP) into Rotation Forest. In the second step, spatial contextual information, which is modeled by Markov random field prior, is used to redefine the classification results obtained in the first step, by solving a maximum a posteriori problem using the α-Expansion Graph Cuts optimization method. The numerical studies have been conducted to evaluate this algorithm with many other approaches.

	ble simultaneously. Principal component analysis (PCA), maximum noise fraction
	(MNF), independent component analysis (ICA) and local fisher discriminant analy-
	sis (LFDA) are introduced as feature transformation algorithms in the original Ro-
	tation Forest. The performance of Rotation Forest was evaluated based on several
	criteria: different data sets, sensitivity to the number of training samples, ensemble
	size and the number of features in a subset. Experimental results revealed that
	Rotation Forest, especially with PCA transformation, could produce more accurate
	results than Bagging, AdaBoost, Random subspace, Random Forest. They indicate
	that Rotation Forests are promising approaches for generating classifier ensemble of
	hyperspectral remote sensing.
	• With different principles, support vector machines (SVMs) and multiple classifier
	systems (MCSs) have shown excellent performances for classifying hyperspectral
	images. In Chapter 4, we propose a novel ensemble approach, namely rotation-based
	SVM (

Table A . 1 :

 A1 Résumé des approches spectrales spatiale appliquée à la classification d'images hyperspectrales

	A.1.2 Objectifs
	Dans cette thèse, des méthodologies avancées sont proposées pour traiter deux importantes
	sources d'information, ces premières consisient àla classification spectrale à base spectrale
	et spatiale des données hyperspectrales :
	Référence	[32, 33]		[34, 35]		[36, 37]				[38, 39]				[40-42]			[43]				[44, 45]
	Méthodes Descriptions	Noyaux composites Noyaux multiples, qui représentent l'information spec-	trale et spatiale, sont combinés.	Champs aléatoires de Markov MRF utilise un modèle probabiliste pour intégrer l'in-	formation spatiale.	Segmentation 1) Segmentation et classification sont effectuées pour ob-tenir diverses régions et le résultat de classification pixel	par pixel, respectivement. Ensuite, la classe la plus fré-	quente dans une région est désigné comme la dernière	classe.	2) Segmentation est effectuée pour obtenir plusieurs ré-	gions, et les caractéristiques obtenues de chaque région	(objet) est considéré comme l'entrée du classificateur su-	pervisé.	Morphologie mathématique 1) Résultats des opérateurs morphologiques plus de fonc-tionnalités à partir d'images originales ou calculées par	la sélection de fonction / extraction sont traitées comme	des entrées supplémentaires de classificateurs dirigés.	2) Résultats de la classification multiples produits par	des informations spectrales et des caractéristiques mor-	phologiques sont combinées pour générer une carte de	classification finale.	Autres Modélisation de Tensor, classification basée sur le	contexte, etc.

  chapitre 2 est consacré à la presentation du contexte et aux travaux concernant le système de classification multiple, surtout les données hyperspectrales. En particulier, les aspects importants de MCS sont introduits, tels que la combinaison topologie, et la génération de classificateur classificateur. Le document de MCS sur la classification d'image hyperspectrale est également souligné. • Dans le chapitre 3, Rotation Forêt est proposé pour la classification de données hyperspectrales. PCA, MNF, ICA et LFDA sont introduits sous forme d'algorithmes fonction de transformation. Le chapitre 5 étend le travail du chapitre 3 en incorporant la méthode d'extraction de caractéristiques linéaire locale et l'information contextuelle spatiale avec le champ de Markov à établir des méthodes spatio-spectrales robustes. Nous introduisons trois méthodes d'extraction de caractéristiques locales linéaires, y compris la NPE, la LLTSA et la LPP Les informations contextuelles spatiales sont modélisée par MRF priori, celles-ci s'appliquent pour redéfinir les résultats de classification obtenus dans la première étape, par la biais de résoudre le problème de MAP, en utilisant la méthode d'optimisation de α-Expansion Graph Cuts .

	• Le chapitre 6 présenté un nouveau cadre pour pratiquer une série de classifica-
	teurs efficaces avec des EMAPs pour classifier des données hyperspectrales. Deux
	méthodes d'apprentissage rapides : l'DT et l'ELM sont sélectionnées comme les
	classificateurs de base. Six RS méthodes d'ensemble, y compris les RSDT, RF, RoF,

• Dans le chapitre 4, nous proposons une nouvelle approche des ensembles, à RoSVM. Deux méthodes simples d'extraction de caractéristiques sont choisies dans RoSVM : la PCA et la RP. Une étude empirique sur l'ensemble de trois données hyperspectrales démontre que les méthodes d'ensemble RoSVM sont plus pratiques que le SVM et RSSVM. • RoRF, RSELM et RoELM, sont établis par apprenants de base multiples.

  En variante, dans la topologie de concaténation, le résultat de classification généré par un classificateur est utilisée comme entrée dans le prochain classificateur[START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][START_REF] Rahman | Serial combination of multiple experts : A unified evaluation[END_REF][START_REF] Ranawana | Multi-classifier systems : Review and a roadmap for developers[END_REF].Génération de classificateur a pour but de construire des classificateurs individuels qui sont mutuellement complémentaires et précis en même temps désaccord sur certains différentes parties de l'espace d'entrée. Après les étapes de classification de motifs, nous pouvons appliquer la diversité par la manipulation de formation échantillons, les caractéristiques, les sorties et les classificateurs.Manipulation des échantillons d'apprentissage. Dans cette méthode, chaque classificateur est formé sur les différentes versions d'échantillons de formation par l'échange de la distribution des échantillons originaux de formation. Bagging et Boosting le site appartiennent à cette catégorie[START_REF] Freund | Experiments with a new Boosting algorithm[END_REF][START_REF] Breiman | Bagging predictors[END_REF].

	A.2.2 Génération de classificateur.

Figue. A.1 illustre qu'il y a deux topologies dans la conception MCS. Lequel le plus populaire dans les documents demeure le style parallèle. Dans cette architecture, plusieurs classificateurs sont conéus de faéon indépendante, sans aucune interaction mutuelle et leurs sorties sont combinés selon certaines stratégies

[START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][START_REF] Rahman | Serial combination of multiple experts : A unified evaluation[END_REF][START_REF] Ranawana | Multi-classifier systems : Review and a roadmap for developers[END_REF]

. Manipulation des fonctions de formation L'algorithme le plus connu de ce type est RSe

[START_REF] Ho | The random subspace method for constructing decision forests[END_REF]

. Le sous-espace aléatoire a été utilisée pour plusieurs types d'apprenants de base, telles que l'DT (Random Forest)

[START_REF] Breiman | Random forests[END_REF]

, SVM

[START_REF] Waske | Sensitivity of support vector machines to random feature selection in classification of hyperspectral data[END_REF]

.

Manipulation des sorties. Problème multi-classification peut être converti en plusieurs problèmes de classification à deux de classe. Chaque problème découvre la discrimination entre une classe et les autres classes. L'autre méthode pour traiter avec les sorties est Label Switching

[START_REF] Martinez-Munoz | Switching class labels to generate classification ensembles[END_REF]

. Manipulation des classificateurs. A ce stade, nous pouvons utiliser différents types de classificateurs ou les mêmes classificateurs avec des paramètres différents.

A.2.3 Combinaison de classificateur.

Table A .

 A 2 résume combinaison de classificateur approche. WV, FI, DS et CT nécessitent validation afin de calculer les poids. DCS calcule la distance entre le échantillons, il nécessite donc l'image originale. Et le temps de calcul de DCS est plus cher que les autres approches. Méthodes de combinaison, tandis que d'autres sont des méthodes de mesure de niveau.

	A.3 Rotation Forêt

RoF construit les différentes versions de l'ensemble d'apprentissage en utilisant les étapes suivantes : l'ensemble des fonctionnalités est divisé en ensembles disjoints sur lequel l'ensemble de la formation initiale est projetée. Ensuite, un échantillon aléatoire de classes est éliminé et un échantillon bootstrap est choisi dans chaque résultat de projection. Méthode

Table A . 2 :

 A2 Résumé de la combinaison de classificateur approches. Remarque : 'Y' et 'N' si oui ou non l'étiquette de la classe moyenne, la probabilité de classe ou d'un ensemble de validation sont nécessaires.

	Nom	Étiquette Probabilité Validation	Référence
	Vote majoritaire (MV)	Y	N	N	[48]
	Vote pondéré (WV)	Y	N	Y	[59, 89]
	Moyenne bayésienne (BA)	N	Y	N	[90]
	Théorie de l'évidence de Dempster-Shafer (DS)	Y	N	Y	[48, 91]
	Fuzzy intégrante (FI)	Y	N	Y	[92-94]
	La théorie du consensus (CT)	Y	Y	Y	[95, 96]
	Sélection de classificateur dynamique (DCS)	Y	N	Y	[51, 97, 98]

*

Table A . 3 :

 A3 Moyenne de l'arthrose en utilisant SVM et approches d'ensemble (la valeur correspondante de AA). Pour les approches d'ensemble, moyennes les plus élevées de l'arthrose avec différentes valeurs de M sont présentés.

	méthodes	T	Nombre d'échantillons par classe
		10	20	30	40	50
	SVM	63.97 (74.03)	65.05 (78.07)	70.73 (80.15)	71.91 (82.87)	73.86 (83.36)
		64.38 (75.85)	68.38 (80.22)	71.69 (80.63)	72.99 (83.05)	74.10 (83.44)
	RSSVM	65.67 (76.13)	70.12 (81.12)	71.98 (80.93)	73.09 (83.15)	74.70 (83.54)
		66.48 (76.13)	70.38 (80.89)	71.74 (80.91)	73.25 (83.63)	74.96 (83.73)
		67.12 (76.99)	70.01 (80.32)	71.86 (80.63)	73.15 (83.65)	74.92 (83.97)
		72.37 (78.61)	72.20 (81.52)	75.43 (80.97)	75.57 (83.97)	75.89 (85.54)
	RoSVM-PCA	73.12 (78.89) 73.42 (81.74)	76.14 (81.11)	76.27 (84.02)	75.82 (85.43)
		73.01 (78.94)	73.18 (81.92)	76.64 (81.23) 76.13 (84.00)	75.99 (85.77)
		72.98 (78.69)	73.28 (81.73)	76.01 (80.97)	76.42 (83.92)	75.98 (85.68)
		69.74 (77.82)	70.74 (82.07)	75.00 (81.9)	77.09 (85.02)	76.66 (86.20)
	RoSVM-RP G	70.51 (77.94)	71.28 (82.34)	75.97 (82.4)	76.93 (84.98)	76.99 (86.23)
		70.64 (78.02)	71.11 (82.52)	76.06 (82.67)	76.95 (84.98)	77.01 (86.42)
		70.28 (77.91)	71.64 (82.67)	76.02 (82.53)	77.12 (85.12) 77.00 (86.45)
		70.05 (77.71)	70.76 (80.15)	74.64 (80.60)	75.62 (84.19)	76.79 (86.37)
	RoSVM-RP S	70.65 (77.99)	71.25 (81.02)	74.11 (80.96)	75.92 (84.28)	76.92 (86.42)
		70.99 (78.11)	71.34 (80.92)	75.16 (81.45)	76.37 (84.67)	76.95 (86.51)
		70.42 (77.90)	71.02 (81.13)	75.24 (81.62)	76.58 (84.74)	76.63 (86.64)
		69.94 (77.99)	70.57 (80.38)	74.86 (81.07)	75.13 (84.45)	77.48 (86.69)
	RoSVM-RP B	70.12 (78.08)	71.24 (80.99)	75.77 (81.97)	75.98 (84.97)	77.12 (86.52)
		70.43 (78.11)	71.38 (81.30)	75.22 (81.45)	76.02 (85.02)	77.53 (86.61)
		70.38 (77.92)	71.46 (81.41)	75.41 (81.22)	75.82 (84.86)	77.42 (86.67)

Table A . 4 :

 A4 Précision de la classification de l'image ROSIS.

	LORSAL-MRF		97.85	99.29	98.5	98.35	97.57	72.36	99.78	96.71	72.42	85.57	92.54
	RoF RoF-MRF	PCA NPE LLTSA LPP PCA NPE LLTSA LPP	91.39 92.29 91.63 90.98 98.64 98.51 98.34 98.18	96.83 98.1 97.36 97.57 98.52 97.15 98.94 99.37	99.11 99.33 99.41 99.11 99.63 99.55 99.93 99.93	95.63 95.8 95.33 95.76 99.26 99.68 99.68 99.92	91.51 93.9 91.12 93.02 91.74 96.18 91.97 95.53	75.32 76.5 76.42 79.69 87.78 87.34 86.76 89.77	61.51 62.41 66.75 67.65 55.55 62.98 69.80 66.46	83.44 83.86 84.29 82.78 93.27 93.43 93.64 92.91	90.53 89.1 91.43 91.2 94.59 95.26 95.41 95.49	82.53 83.41 83.40 84.76 90.44 90.98 90.84 92.15	87.25 87.82 88.19 88.64 91 91.61 92.23 93.06
	Test SVM LORSAL		3682 92.59 96.41	947 96.62 89.54	1345 99.41 89.85	5029 92.25 92.96	3064 97.81 95.46	18649 70.25 69.08	2099 70.32 99.26	6631 83.71 82.43	1330 81.58 74.27	80.99 80.11	88.28 87.7
	Classe Train		1 524	2 514	3 375	4 540	5 231	6 532	7 265	8 548	9 392	OA	AA

Table A . 5 :

 A5 Classement individuel et ensemble approches envisagées pour l'étude de calcul, nous construisons les RS ensembles avec deux algorithmes d'apprentissage rapide : CART et nouvellement NN classificateur : ELM. Six ensembles de classificateurs, y compris RSDT, RF, RoF, RoRf, RSELM et RoELM, sont considérés comme indiqué dans le tableau A.5. Figure A.8 présente les étapes générales de la construction de EMAPs utilisant la zone et les attributs de l'écart type. Tout d'abord, PCA est effectué sur l'image hyperspectrale d'origine et les premiers éléments avec les valeurs propres cumulés de plus de 99% sont conservés. Ensuite, les points d'accès avec l'attribut et l'écart type des attributs sont calculés sur les caractéristiques premier retenus et les caractéristiques de sortie sont enchaînés dans un vecteur empilés pour construire un EMAPs.

	Classificateurs individuels	(Notation)	Ensembles de classificateurs	(Notation)
			Sous-espace aléatoire avec DT	RSDT
	Arbre de décision	DT	Forêts Aléatoires	RF
			Rotation Forêt	RoF
			Rotation aléatoire Forêt	RoRF
	D'apprentissage machine Extreme	ELM	Sous-espace aléatoire avec ELM	RSELM
			Sous-espace rotation avec ELM	RoELM

Table A . 6 :

 A6 Précision de la classification obtenus pour les RS avec EMAPs. .26 95.14 93.52 95.32 97.16 97.69 97.54 9 99.32 100.00 100.00 100.00 100.00 99.92 99.93 99.92 OA 91.67 93.61 96.08 94.85 96.47 96.49 98.67 98.69 AA 91.20 94.58 95.20 95.75 95.84 96.37 99.00 98.92 κ 89.14 91.71 94.83 93.26 95.34 95.37 98.21 98.25 Le coût de calcul d'ELM et ses ensembles est supérieure à celle de la DT et DT ensembles, en raison de la grande taille de l'ensemble de données. Les méthodes spectrales-spatiales sont moins efficaces que les calculs des méthodes en raison de la dimension supérieure de la base de caractéristiques d'entrée-spectraux, mais fournissent à leur tour des précisions élevées. À titre illustratif, la figure A.9 fournit les cartes classification des classificateurs et d'ensemble. Par rapport aux résultats obtenus en utilisant uniquement l'information spectrale présentée dans la figure A.9(a-f), les cartes portant l'information spatiale (voir sur la figure A.9(g-p)) générer des zones plus homogènes (en particulier pour la classe M eadows située à la partie inférieure gauche) et réduire le bruit de classification.

	Class	DT RSDT	RF	RoF	RoRF ELM RSELM RoELM
	1	98.02 98.95 98.94 98.61 99.16 98.96	99.51	99.58
	2	85.96 92.47 97.33 97.00 99.32 98.37	99.31	98.38
	3	99.55 99.58 99.62 99.62 99.62 96.51 99.67	99.56
	4	98.95 96.55 96.34 99.39 97.35 97.61 99.96	99.90
	5	89.69 97.10 99.12 94.55 99.23 94.54	98.52	97.45
	6	90.85 91.66 97.28 93.65 97.42 96.42	98.35	98.55
	7	67.13 80.63 73.05 85.45 75.15 87.88	98.08	99.38
	8 91.34 94Time(s) 1.38 29.72	2.82	37.93 70.24 1.83	37.66	71.59

•

  Dans le chapitre 3, RoF, a été appliquée pour la classification d'images hyperspectrales de télédétection. PCA, MNF, ICA et LFDA sont introduits sous forme d'algorithmes de transformation dans la forêt de rotation d'origine. Les résultats expérimentaux montrent que la rotation des forêts, en particulier avec la transformation PCA, pourrait produire des résultats plus précis que d'ensachage, AdaBoost, RS, Random Forest. • Dans le cadre de situations différentes, SVM et MCS ont montré leurs performances excellentes en terrmes de la classification d'images hyperspectrales. Dans le chapitre 4, nous proposons une nouvelle approche des ensembles, qui sâappelle RoSVM. Celleci combine lâensemble de SVM et MCS. De plus, deux méthodes simples sont choisis pour l'extraction de caractéristiques : la PCA et la RP,Une étude empirique sur trois données hyperspectrales démontre que les méthodes d'ensemble RoSVM sont plus privillégés que le SVM et RSSVM. • le chapitre 5 étend le travail au chapitre 3 en incorporant l'information contextuelle spatiale avec champ de Markov à concevoir des méthodes de spatio-spectrale robustes. Nous nous adaptons quatre méthodes d'extraction de caractéristiques, y compris PCA, NPE, LLTSA et LPP. Dans la deuxième étape, l'information contextuelle spatiale, qui est modélisée par MRF prior, est utilisé pour redéfinir les résultats de la classification obtenus dans la première étape, par la résolution d'un maximum d'un problème posteriori à l'aide du α-Expansion Graph Cuts méthode d'optimisation. • le chapitre 6 présenté un nouveau cadre général pour former série de classificateurs efficaces avec l'information spatiale pour la classification des données hyperspectrales. Le cadre proposé est basé sur les deux observations principles : 1) la malédiction de la dimension peuvent être atténués à l'aide de RS des ensembles ; 2) les informations spatiales contextuelles peuvent être obtenues par le biais d'EMAPs. Deux algorithmes d'apprentissage rapide, DT et ELM sont sélectionnées comme les classificateurs de base. Six RS méthodes d'ensemble, y compris les RSDT, RF, RoF, RoRF, RSELM et RoELM, sont construits par les apprenants de base multiples. A partir de l'image de ROSIS, notre approche, y compris RSELM-EMAPs ainsi que RoELM-EMAPs, atteint au niveau de performances state-of-the-art, ce qui révèle l'avantage de ces deux méthodes.

http://www.cs.waikato.ac.nz/ml/weka/

y i is different between DT and ELM. In DT, y i is a scalar. In ELM, y i is a vector of labels in which j th column is set to be 1 if the sample belongs to class j while the other columns are set to be 0.

Acknowledgements

MRF. (h) RoF-LPP-MRF. . . . . . . . . and MRF. . . . . . . . . . . . .

Chapter 4 Rotation-based SVM ensemble

Abstract: With different principles, support vector machines (SVMs) and multiple classifier systems (MCSs) have shown excellent performances for classifying hyperspectral images. In this paper, we propose a novel ensemble approach, namely rotation-based SVM (RoSVM), which combines SVMs and MCSs together. The basic idea of RoSVM is to generate diverse SVM classification results using random feature selection and feature extraction, which can enhance both individual accuracy and diversity within the ensemble simultaneously. Two simple feature extraction methods: principal component analysis (PCA) and random projection (RP), are chosen for feature extraction in RoSVM. Empirical study on three hyperspectral datasets demonstrates that the proposed RoSVM ensemble methods outperform the regular SVM and random subspace SVM (RSSVM). The impacts of the parameters on the overall accuracy of RoSVM are also investigated. neighborhood graph on the data and estimates the local tangent space Θ i at each data point x i . Subsequently, it forms the alignment matrix B by performing the summation as follows:

Rotation-based SVM ensemble

where the entries of the alignment matrix B are obtained by iterative summation (for all matrices V i and starting from B ij = 0 for ∀ij), J τ is the centering matrix of size τ , ν i is a selection matrix that contains the indices of the nearest neighbors of data point x i . The cost function is minimized in a linear manner by solving the generalized eigenvalue problem:

LPP is a technique that aims at combining the benefits of linear techniques and local nonlinear method for feature extraction by finding a linear mapping that minimizes the cost function [START_REF] He | Locality preserving projections[END_REF]:

(5.5)

In the above cost function, large weights W ij correspond to small distances between the data points x i and x j . LPP starts with the construction of a nearest graph in which each point x i is connected to its τ nearest neighbors. The weights of the edges in the graph are computed as follows:

Subsequently, LPP solves the generalized eigenvalue problem

in which Q is the diagonal matrix whose entries are column, Q ii = j W ji , and L = Q-W is the Laplacian matrix.

Markov random fields

Rotation Forests are supervised classifiers that only focus on spectral information, without considering any spatial correction, which may consequently bring some classification noises. In order to alleviate the problem, we integrate the contextual information with spectral information by using an isotropic multi-level logistic model (MLL) prior to modeling the image of the class label. This approach exploits the fact that, in real-world images, it is often likely that spatially neighboring pixels belong to the same class. This prior, which belongs to the MRF class, encourages piecewise smooth segmentation and