Laurent Heutte

Robert Sabourin

Keywords: Dissimilarity space, random forest, multi-view learning, high dimension, low sample size, dissimilarity learning, dynamic selection

The work of this thesis was initiated by a Radiomic learning problem. Radiomics is a medical discipline that aims at the large-scale analysis of data from traditional medical imaging to assist in the diagnosis and treatment of cancer. The main hypothesis of this discipline is that by extracting a large amount of information from the images, we can characterize the specificities of this pathology in a much better way than the human eye. To achieve this, Radiomics data are generally based on several types of images and/or several types of features (from images, clinical, genomic). This thesis approaches this problem from the perspective of Machine Learning (ML) and aims to propose a generic solution, adapted to any similar learning problem. To do this, we identify two types of ML problems behind Radiomics: (i) learning from high dimension, low sample size (HDLSS) and (ii) multiview learning. The solutions proposed in this manuscript exploit dissimilarity representations obtained using the Random Forest method. The use of dissimilarity representations makes it possible to overcome the well-known difficulties of learning high dimensional data, and to facilitate the joint analysis of the multiple descriptions, i.e. the views.

The contributions of this thesis focus on the use of the dissimilarity measurement embedded in the Random Forest method for HDLSS multi-view learning. In particular, we present three main results: (i) the demonstration and analysis of the effectiveness of this measure for HDLSS multi-view learning;

(ii) a new method for measuring dissimilarities from Random Forests, better adapted to this type of learning problem; and (iii) a new way to exploit the heterogeneity of views, using a dynamic combination mechanism. These results have been obtained on radiomic data but also on classical multi-view learning problems.

i

Résumé

Les travaux de cette thèse ont été initiés par des problèmes d'apprentissage de données radiomiques. La Radiomique est une discipline médicale qui vise l'analyse à grande échelle de données issues d'imageries médicales traditionnelles, pour aider au diagnostique et au traitement des cancers. L'hypothèse principale de cette discipline est qu'en extrayant une grande quantité d'informations des images, on peut caractériser de bien meilleure façon que l'oeil humain les spécificités de cette pathologie. Pour y parvenir, les données radiomiques sont généralement constituées de plusieurs types d'images et/ou de plusieurs types de caractéristiques (images, cliniques, génomiques).

Cette thèse aborde ce problème sous l'angle de l'apprentissage automatique et a pour objectif de proposer une solution générique, adaptée à tous problèmes d'apprentissage du même type. Nous identifions ainsi en Radiomique deux problématiques d'apprentissage: (i) l'apprentissage de données en grande dimension et avec peu d'instances (high dimension, low sample size, a.k.a. HDLSS) et (ii) l'apprentissage multi-vues. Les solutions proposées dans ce manuscrit exploitent des représentations de dissimilarités obtenues à l'aide des Forêts Aléatoires. L'utilisation d'une représentation par dissimilarité permet de contourner les difficultés inhérentes à l'apprentissage en grande dimension et facilite l'analyse conjointe des descriptions multiples (les vues).

Les contributions de cette thèse portent sur l'utilisation de la mesure de dissimilarité embarquée dans les méthodes de Forêts Aléatoires pour l'apprentissage multi-vue de données HDLSS. En particulier, nous présentons trois résultats: (i) la démonstration et l'analyse de l'efficacité de cette mesure pour l'apprentissage multi-vue de données HDLSS; (ii) une nouvelle méthode pour mesurer les dissimilarités à partir de Forêts Aléatoires, plus adaptée à ce type de problème d'apprentissage; et (iii) une nouvelle façon d'exploiter l'hétérogénèité des vues, à l'aide d'un mécanisme de combinaison dynamique. Ces résultats ont été obtenus sur des données radiomiques mais aussi sur des problèmes multi-vue classiques.

Mots-clés: Espace de dissimilarité, forêt aléatoire, apprentissage multi-vue, dimension élevée, taille réduite de l'échantillon, apprentissage de dissimilarité, sélection dynamique

General Introduction

In many real-world Pattern Recognition problems, the data available are complex, in the sense that they cannot be described by a single numerical representation. For example, they can come from multiple sources as in video surveillance problems ([119]), where multiple cameras are used to captured the same scene, from different angles. This is necessary to avoid blind spot or occlusions of object or persons. The images captured by these multiple cameras are expected to complement each other to have a more accurate and complete representation of the scene.

Another widespread situation is problems for which the raw data are described via multiple feature extractors, with the goal to better capture their complexity. In certain image recognition task for example, an image is described by multiple feature representations, e.g. colour descriptors, shape descriptors, texture descriptors, etc. Each of these descriptor families is used to capture a particular characteristic of the images, and using them all together is expected to help to better address the complexity of the recognition task.

The starting point of this work is a medical imaging problem of this type, i.e. for which the data are derived from several image modalities and/or several feature extractors. This problem is computer-aided cancer diagnosis/treatment based on Radiomic features, a key application of the DAISI project, co-financed by the European Union with the European Regional Development Fund (ERDF) and by the Normandy Region. As part of the DAISI project, this thesis has been initiated around the Radiomics application in collaboration with experts and doctors from the Henri Becquerel center, one of the French Comprehensive Cancer Centers (FCCCs).

The objective of Radiomics is to describe radio-graphic medical images with a large number of heterogeneous image features, in the hope of discovering characteristics of the disease that cannot be discerned with the naked eye. The main hypothesis is that exploiting numerous heterogeneous features may be useful for predicting prognosis and therapeutic response for various conditions, thus providing valuable information for personalized therapy. This thesis work addresses the Radiomics problem from a Machine Learning (ML) point of view. In doing so, Radiomics can be considered a Multi-View Learning (MVL) problem, in which a 'view' is the set of features obtained from one modality and/or one feature extractor. Each instance of the problem is therefore described by multiple views, and the goal is to learn a predictive model by taking into account the complementarities of all views. MVL tasks usually needs dedicated methods because it exhibits specific ML difficulties. The first difficulty often relates to the overall dimension of the problem. Considering all the features extracted in all views, most MVL problems are (very) high dimensional learning problems. This will largely narrow down the choice of machine learning methods. For example, the popular machine learning method Support Vector Machine (SVM) can suffer from data piling problem on very high dimensional features ([163]). The high feature dimension usually requires a large number of labelled training instances. However, it is impossible to collect a lot of training instances in various fields, especially in the medical field. Due to the problem of data collection and sharing, the number of patients is always very small compared to the high feature dimension, which usually leads to the High Dimension Low Sample Size (HDLSS) problem. Due to the small sample size, the instances are very sparse in the feature space, which also makes it harder to deal with noise or outliers.

When multi-view problems are coupled with HDLSS problems, it becomes more difficult because many multi-view techniques are unable to handle the HDLSS problems while HDLSS solutions do not take multi-view information into consideration.

Radiomics is a typical example of HDLSS multi-view problem. Each view of

Radiomic problem can easily have hundreds or thousands of features while the sample size is usually smaller than one hundred. The state-of-the-art works in Radiomics, however, concatenate the multi-view information into one view and use feature selection techniques to decrease the feature dimension. These feature selection methods may lose some important information, especially when only few features are selected. Furthermore, these methods also overlook the multi-view challenge, which deviates from the original intention of extracting heterogeneous information. Features from different views generally have distinct descriptions of the same instance and can provide additional information for the learning task, making it necessary to use multi-view alternatives to take benefit of the complementary information.

Contents

In the first part of the thesis, we aim at finding a solution to deal with both HDLSS and multi-view challenges. A literature review on multi-view learning methods with a focus on the HDLSS problem is given in Chapter 1. One straightforward solution is to find a low dimensional intermediate representation that can be comparable among all the views to tackle the HDLSS challenge. Then, these intermediate representations can be merged into a joint representation to tackle the multi-view challenge. In this work, we propose to use dissimilarity representations as the solution because it can naturally reduce the feature dimension to a lower space. The dissimilarity values are also directly comparable from one view to another and easy to integrate.

As the only information shared by the different views is the class information, we propose to use a dissimilarity measure that can take this information into account. Among different supervised dissimilarity learning methods, we propose to use the Random Forest Dissimilarity (RFD). Random Forest (RF) is a successful machine learning method due to its interpretability and its accuracy in classification and regression tasks. RF can also naturally handle HDLSS problems thanks to the implicit feature selection mechanism as well as the bootstrap procedure ([103]). RF also embed a dissimilarity measure, the RFD measure, which reflects both feature and class dissimilarities. A first solution of RFD based intermediate integration is proposed in Chapter 2 as the solution for HDLSS multi-view problem: the features from each view are firstly projected into a dissimilarity space built with RFD, then these dissimilarity spaces are merged to form a joint dissimilarity space, used as the new representation for learning. The proposed solution is compared to multiple state-of-the-art Radiomic and multi-view solutions during the experiments on real world datasets. We show that this proposed approach is accurate and competitive on several real-world HDLSS multi-view problems.

In the second part of this thesis, we mainly focus on deepening this first solution and make it more efficient for the HDLSS multi-view problem. We identify two ways to improve the proposed method in Chapter 2: one in the way the dissimilarity is computed from the RF and the other in the way the view-specific dissimilarity representations are merged. For the first one, we propose a finer and a more accurate dissimilarity measure based on Random Forests, by better exploiting the tree structures. Each tree in a Random Forest estimates the dissimilarity between two instances with a binary value, which may not be precise enough. Furthermore, the RFD estimate is a simple average over these binary values, while different trees may contribute differently to the final dissimilarity value. To overcome these limitations, our proposal is based on the evaluation of each terminal node confidence so that we can tell which tree to trust for each test instance. Based on this proposal, we also propose another more refined confidence measure using instance hardness to make the dissimilarity measure more adaptive to the HDLSS problem.

The second improvement is to propose a more adaptive dissimilarity matrix combination method to take better advantage of the multi-view information.

For the experiments in Chapter 2 and Chapter 3, the joint dissimilarity representation is calculated by averaging over all the dissimilarity representations from each view. With averaging, different views are assumed to have the same importance for the classification task. However, for multi-view problem, different views usually offer very heterogeneous information and may have different importance for the given task. To estimate this importance, two methods based on static weighting and dynamic view selection respectively are proposed in Chapter 4. We firstly propose to use the Out Of Bag (OOB) accuracy of the Random Forest classifier used to built the dissimilarity representation to estimate the importance of each view. The intuition is that if a Random Forest has better generalization performance, the corresponding dissimilarity measure also has better quality. Secondly, a dynamic view selection method is proposed to select different view combinations for different test instances with the intuition that the information provided from one view for one test instance may not be as useful for another test instance.

Finally, we summarize our contributions in the last chapter, along with the future works we are interested in for both short and long terms. data using numerous sensors. It is also very common to have data from different sensors to describe the same object in self driving car scenarios. In [186], the authors use data from camera (Figure 1.1a) and lidar ( Figure1.1b) to have a better pedestrian detection system. Data from different sensors usually can provide complementary information (color and depth information in the example of [186]) to improve the accuracy of machine learning models.

From the perspective of software, a large amount of new algorithms have been proposed for both information extraction and data analysis. To make the process of data analysis more efficient, many different feature extractors have been proposed to better represent the characteristics of instances. Multiple feature extractors are usually used together to represent the data heterogeneity for complex problems. For example in Figure 1.2, the AWA dataset used in [143] for natural scene image classification, is build with six different groups of traditional features including RGB color histograms ([143]), ), rgSIFT ([229]), Pyramid Histogram of Oriented Gradients (PHOG [START_REF] Bosch | Representing shape with a spatial pyramid kernel[END_REF]), Speeded Up Robust Features (SURF [START_REF] Bay | Speeded-up robust features (SURF)[END_REF])

and local self-similarity histograms ([207]), which results in a total of 10940 features.

There are also a lot of "genuine" multi-view problems. For example, social media data from Facebook or Instagram contain image data along with textual data such as hashtags; videos are made up of visual data such as images and texts (subtitles) as well as audio data; news data can always be found in multiple different languages, etc. These multiple views generally convey additional information, and successfully combining them often makes it possible to obtain better overall performance than treating them individually.

In real world applications, the feature dimension is becoming higher and higher in each view, which makes the feature dimension extremely high when all multi-view features are concatenated. For example in [145], a total of 1403 handcrafted features and 98304 deep features from magnetic resonance imaging (MRI) are extracted. Because of the curse of dimensionality, a lot of machine learning methods will suffer from over-fitting problem ([163, 256]). To avoid this problem, a large amount of training instances are required, especially in multi-view learning when extra validation instances are often needed

The example of Radiomics

In this section, the HDLSS multi-view problem is introduced and analyzed in detail through the medical example of Radiomics. Although this work is not restricted to this particular application, it is important here to present in detail what Radiomics is and why it is a typical HDLSS multi-view learning problem. The main mission of the thesis is to propose efficient new solutions for the HDLSS multi-view problems such as Radiomics.

Background and workflow

Background of Radiomics

One of the main difficulties in the diagnosis and treatment of cancer rises from the fact that tumors can show very heterogeneous profiles. This phenomenon is called the tumour heterogeneity and may occur for both between tumours (inter-tumour heterogeneity) and within tumours (intra-tumour heterogeneity). The tumor heterogeneity makes the diagnosis and treatment of cancer more difficult.

The usual process of cancer detection is from certain signs and symptoms to the further investigation by medical imaging and at last confirmed by biopsy ( [3]). However, with the improvement of medical imaging technology, more and more attention has been paid on the data collected from medical images such as computed tomography (CT), positron emission tomography (PET) or magnetic resonance imaging (MRI) in medical research during the last two decades. Compared to the traditional procedure, medical imaging has the advantage of being easy to perform, low cost, and non-invasive ( [START_REF] Coroller | CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma[END_REF]). One of the most important reason which makes medical imaging popular in diagnosis and treatment of cancer is that tumor phenotype characteristics can be visualized. One typical example can be found in Figure 1.3: representative CT images of lung cancer are shown on the left and the corresponding 3D visualizations are shown on the right. It can be seen that strong phenotypic heterogeneity can be visualized ( [3]): some tumors are smaller, some are bigger; some are round shaped while others are more spiky.

have also been done. Furthermore, the information provided by Radiomics is thought nowadays to be complementary to clinical, pathological, and genomic information ([146, 234]).

Workflow of Radiomics

The process of Radiomics mainly contains three steps: data acquisition and segmentation , feature extraction and data analysis. One typical example is shown in Figure 1.4:

• From step 1) to 3) the data are collected from different medical imaging devices and the regions of interest (ROI) are segmented. Collecting data is quite hard in medical fields due to different data acquisition protocols, data sharing policies, data privacy, etc. In many Radiomics studies, researchers use no more than 50 patients in their dataset ([18, 43, 48, 61, 87]). In [START_REF] Cameron | MAPS: a quantitative radiomics approach for prostate cancer detection[END_REF], they have only 13 prostate cancer patients for example.

• Then from step 4) to 6), different feature extraction techniques are used to extract multiple heterogeneous feature groups that can be complementary to each other. Due to the limited number of cancer patients, it is common to have Radiomic datasets with many features but very small number of training instances in comparison. Before the data analysis, Radiomic features can also be combined with other features such as genomics, clinics or protein features in practice, which makes the problem "more multi-view" with both multiple data sources and feature extractors, leading to a even higher feature dimension at the meantime.

• Finally, machine learning methods are used to build predictive, diagnostic or prognostic models to help realize the personalized treatment. Due to the large amount of features, it is common for traditional machine learning methods to suffer from the curse of dimensionality. Hence, feature selection methods are usually used before machine learning model.

The most used machine learning methods in Radiomics include Random Forest (RF), support vector machine (SVM) and neural networks ([84, 125, 211, 242]).

Even though the choices of feature extractors may be different across Radiomics studies, there are two aspects in common. The first one is that multiple heterogeneous feature extractors are often used to better represent the tumor heterogeneity. Aerts et al have used four feature groups for lung and head-and-neck cancer captured by CT images in [3] (shown in Figure 1.5), including features calculated with first-order statistics from the histogram, shape features, texture features and wavelet features. Similar feature extraction method can also be found in [173,174] (CT scans for lung and headand-neck cancer), [START_REF] Coroller | Radiomic phenotype features predict pathological response in non-small cell lung cancer[END_REF]248] (CT scans for non-small-cell lung carcinoma), [START_REF] Farhidzadeh | Classification of progression free survival with nasopharyngeal carcinoma tumors in SPIE Medical Imaging[END_REF] (MRI scans for head-and-neck cancer) and [237] (DCE-MRI for breast cancer). However, for Radiomics in brain tumor, three different feature extractors are used including local binary patterns (LBP), histogram of oriented gradients (HOG) and SIFT features ([259]). Despite of the different feature extractors for different cancers, there are at least two or three feature groups extracted in Radiomic works. These features are also often combined with additional information, like clinical and genomic information for example ([3]).

The second aspect in common for handcrafted Radiomic features is the "large quantity of features". However, "large quantity" is a qualitative expression and there is no study that defines the minimum number of features to be large. In the workflow of Radiomics, we have pointed out that the sample size of Radiomic problem is usually very small (fewer than one hundred patients), while the number of features is usually at least 4 or 5 times above the number of learning instances ([3, 64, 173]). Hence, the large number of features generally means that the feature size is much bigger than the sample size in Radiomics. For example, the Radiomic dataset used in [258] is made up of 84 patients and 6746 features.

Automatic feature learning

As manually extracted features would contain inevitable bias, researchers try to develop some automatic feature extraction methods. Deep learning is the most used, especially convolutional neural network (CNN). The advantage of using CNN is that given the region of interest as input, and an objective as output (e.g. classification), the features can be learned automatically.

175 handcrafted features are used which result in a total of 33903 features (details about this work can be found in Appendix A).

Machine Learning for Radiomics

The previous section has detailed the first steps of the workflow of Radiomics presented in Figure 1.4, until the feature extraction. This section focus on how the literature addresses the next step of this workflow: the learning phase.

In the previous section, it has been shown that Radiomic problems are typical HDLSS multi-view problems. In a large majority of Radiomic works, the multiple views are concatenated to form a single-view feature vector. Among the multiple views, it is very often to have some high dimensional views. When these multiple feature groups are concatenated, it usually worsens the HDLSS problem. This constitutes the major obstacle for the traditional machine learning methods. For example, linear discriminant analysis (LDA) is not suitable for HDLSS problem as the pooled covariance matrix is not invertible ([256]). Support vector machine (SVM) is also proved to have data piling problem on very high dimensional data ([163]), which may adversely affect the generalization performance of SVM in some HDLSS situations.

As a consequence, feature selection methods are systematically used to overcome the difficulty of learning in high dimension. The goal is to reduce the redundancy, noise, or irrelevant features while at the same time keep good performance. There are mainly two feature selection strategies used in Radiomic works. The first one is to select features by some predefined criteria and then build machine learning classifiers on the selected features. The second one combines the feature selection procedure with classification by using the performance of a predefined classifier as the feature selection criteria.

Most of the studies of Radiomics belong to the first category, where feature selection is used independently from the machine learning methods. Usually a feature score showing the relevance or reliability (such as in [START_REF] Coroller | Radiomic phenotype features predict pathological response in non-small cell lung cancer[END_REF], [174] and [3]) is calculated to rank all the features. Then, the features with the lowest ranks are removed. The number of features or the threshold should be predefined. In Radiomics, most studies choose no more than 15 features among methods to find the most accurate combination. For example, in [173], the authors use 14 feature selection methods along with 12 classifier families on CT images of lung cancer patients. Two similar studies has been done in [174] and in [248]. The experimental result of [173] is shown in Figure 1.7: it can be seen that feature selection criteria Minimum Redundancy Maximum Relevance (MRMR) and Wilcoxon (WLCX) have the best general performance; machine learning methods Bagging (BAG), RF and SVM have good overall performance with selected features. But choosing the proper classifier on selected features is very important too. For example, the features selected by MRMR have the best performance if fed into BAG, RF or SVM. But when Bayesian (BY) is used as the classifier on the same selected features, the performance is the worst, which motivates the second category of feature selection in Radiomics by taking the interaction of classifiers into consideration.

The second category of feature selection in Radiomics integrate the classifier into the selection procedure to improve the classification performance. Generally speaking, a classifier needs to be chosen in advance and some feature weighting or ranking approach is usually embedded in the training process.

The feature elimination is then realized according to the weight/rank of features. Few Radiomic works have successfully used Support Vector Machine

Recursive Feature Elimination (SVM-RFE [107]) and obtained very good performance. This approach differs from the most popular approaches used in Radiomics by embedding the feature selection into the learning procedure of the SVM, so that it can take the resulting classifier performance into account.

In [238,257], they showed that SVM-RFE had very good performance on Radiomic problem. The workflow of [257] is shown in Figure 1.8: SVM-RFE is used to rank all the features, then SVM classifier is trained to select the best size of feature subset. For other HDLSS problems, SVM-RFE also had very good performance ( [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]).

different views should be considered to improve the performance. This section presents a taxonomy of multi-view learning methods with an emphasis on HDLSS problems to provide better solutions to the HDLSS multi-view learning problems. Our goal here is not to give an exhaustive survey on this machine learning field but to present a panorama of the different multi-view approaches, especially for the methods that can deal with the HDLSS problem. According to [204], there are three main kinds of multi-view approaches: early integration, late integration and intermediate integration. Each of these approaches is detailed in the following sections.

Problem statement

Before introducing the different multi-view learning approaches, this section gives a formal definition of this type of problems, and details all the notations that will be used in the rest of this manuscript.

Supervised learning

Supervised learning tasks strive to infer a function h, often called a model, that maps an input domain X to an output domain Y:

h : X → Y
For cases where Y = R, the problem is called a regression problem. For cases where Y is a finite set of classes, the problems is called a classification problem.

In the latter case, the C classes are denoted {ω 1 , ω 2 , . . . , ω C }.

For simplicity, and because it concerns most of the Radiomics tasks found in the literature, this manuscript mainly focus on classification. However, note that most of the methods described in this section also suit to regression tasks.

As for the input domain X , it is typically a m-dimensional space, i.e. X =

X 1 × X 2 × • • • × X m
, where X i is the domain of the ith feature of the problem. Consequently, an instance x ∈ X is a m-dimensional vector noted:

x = (x 1 , x 2 , . . . , x m-1 , x m )
where x j is the value of the jth feature of x.

In supervised learning, h is said to be learnt from a set T of labeled instances. This set is usually called a training set, and is composed of n input-output pairs: T = (x 1 , y 1 ), (x 2 , y 2 ), . . . , (x n , y n )

Such a training set is often written as a n × m matrix:

X =        x 1,1 x 1,2 x 1,3 . . . x 1,m x 2,1 x 2,2 x 2,3 . . . x 2,m . . . . . . . . . . . . . . . x n,1 x n,2 x n,3 . . . x n,m        (1.1)
where x i,j is the value for the jth feature of the ith instance in T . In the same way, the n output values are gathered in a vector y = y 1 , y 2 , . . . , y n , where y i is the class label of the ith instance in T .

Multi-view learning

Multi-view learning is a learning task where each instance is described by Q different input vectors instead of only one. Formally, the task is to infer a model h:

h : X (1) × X (2) × • • • × X (Q) → Y
where the X (q) are called the views. These views constitute different description spaces of different dimensions, noted m 1 to m Q . In such learning framework, the training set T is actually decomposed in Q training set:

T (q) = (x (q) 1 , y 1 ), (x (q) 2 , y 2 ), . . . , (x (q) n , y n ) , ∀q = 1..Q
Similarly to T , a multi-view dataset can be written as Q matrices:

X (q) =         x (q) 1,1 x (q) 1,2 x (q) 1,3 . . . x (q) 1,m q x (q) 2,1 x (q) 2,2 x (q) 2,3 . . . x (q) 2,m q . . . . . . . . . . . . . . . x (q) n,1 x (q) n,2 x (q) n,3 . . . x (q) n,m q         ∀q = 1..Q (1.2)
where x (q) i,j is the value of the jth feature of the ith instance in the qth view.

High dimension, low sample size (HDLSS)

HDLSS problems are learning problems for which the dimension m of the input space is very high in regards to n, the number of instances available in the training set T . Machine learning problems described in high-dimensional spaces are known in the scientific community to be particularly difficult, since an enormous amount of training instances is typically required for learning an accurate classifier. A typical rule of thumb is that there should be at least 5 training examples for each dimension in the representation ([224]). It is well known that such "curse of dimensionality" problem leads to serious breakdown in many algorithms with an under-determined problem. According to [163], in the context of HDLSS, classical multivariate analysis is useless due to the need of the root inverse of the covariance matrix, which does not exist (because the covariance is not of full rank). Many traditional machine learning techniques such as Logistic Regression (LR), discriminant analysis or K-Nearest Neighbors (KNN) are not able to give a solution to HDLSS problem due to the ill-posedness ([103]). For example, Linear Discriminant Analysis (LDA) is not suitable for HDLSS problem as the pooled covariance matrix is not invertible ([256]). Support Vector Machine (SVM) is also known to have data piling problem on very high dimensional data ([163]). When the sample size is small, the distribution of data in the high dimensional space is very sparse, which makes it also harder to deal with outliers ( [START_REF] Branden | Robust classification in high dimensions based on the SIMCA method[END_REF]) and may easily cause overfitting problem. Unlike these classifiers discussed above, Random

Forest can deal well with high dimensional data due to the implicit feature selection mechanism during the tree construction ( [START_REF] Chen | Random forests for genomic data analysis[END_REF]222]).

Nevertheless, as already extensively discussed in the previous section, many machine learning problems are naturally described in high-dimensional spaces but with very few training instances; that is to say with m >> n. These problems are called HDLSS learning problems.

When transposed to multi-view learning problems, the HDLSS setting is even more critical. The reason is that the different m q values are all potentially individually greater than n: m q >> n, ∀q = 1..Q. Besides, if it is not strictly the case for all views, one can reasonably assume that ∑ Q q=1 m q >> n. As it has been explained previously and as it will be further discussed in the following, concatenating all the views together, to form a new joint description space, is a principle often encountered in the literature when dealing with real-world HDLSS multi-view learning problems. In such a case, the dimension m of this joint description space is m = ∑ Q q=1 m q , which exacerbates the difficulties that stem from the HDLSS setting. As an example, when compiling works from the Radiomics literature, the values for m q are often between 400 and 2000 while the values for n are from 50 to 200 ([59, 122, 179, 225]). Furthermore, q is usually bigger than 3 which could lead to a joint description space of dimension m easily bigger than 1000 ([65, 87, 173, 174, 248]).

The state-of-the-art Radiomic solutions usually ignore the multi-view challenge by concatenating all the views together. To take better advantage of multi-view information, we firstly review the multi-view literature with the focus on the state-of-the-art methods offered for HDLSS problem. In the following sections, three different categories of multi-view solutions (early, late and intermediate integration) are introduced.

Early integration

Early integration methods directly concatenate different views together and treat the multi-view learning as single-view learning ([204]). All the Radiomics works discussed in the previous section belong to this category. The flowchart of early integration is given in Figure 1.9. The dimension of the description space formed by this concatenation is inherently high and a lot of machine learning methods can suffer from it. This would be especially the case, if the not take the correlation among features into consideration, the optimal feature subset may not be unique.

One of the most direct and simplest ranking criteria is the Pearson correlation coefficient ([106]) to detect the dependencies between a feature and the class label. Given the ith feature vector X •,i and the class label vector y, the Pearson correlation coefficient can be calculated with:

P(i) = cov(X •,i , y) var(X •,i ) * var(y) (1.3)
where cov() is the covariance and var() is the variance. This correlation based ranking is simple but limited to detect only the linear dependencies between the feature and the class label.

During the last decades, a large number of filter methods have been proposed, especially methods based on Mutual Information (MI), Relief and its variant RELF ([223]). MI is one of the most popular feature selection criteria due to its computational efficiency and simple interpretation ( [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]223]). Relief selects features that help to separate instances from different classes. RELF adds the ability of dealing with multi-class problems and is also more robust and capable of dealing with incomplete and noisy data ( [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]). RELF can be interpreted as margin maximization, which explains why it has superior performance in many applications ([64, 223, 248]).

To summarize, filter methods are computationally simple and fast, which makes it very popular when facing HDLSS problems ([223]). However, the major disadvantage is that filter methods ignore the interaction with the classifier, the search in the feature subset space is separated from the search in the hypothesis space ([251]). In contrast to filter methods, we will introduce wrapper methods in the next section which obtain a feature subset relying on the classification.

Wrapper methods

Wrapper methods consist in using a classifier for selecting a subset of features, the objective being to optimize the classifier performance by searching for the best subset ( [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]). The general scheme of wrapper methods for feature selection is shown in Figure 1.10b. Given a predefined classifier, wrapper feature selection methods usually include the following steps: the first step consists in finding a particular subset of features among the 2 m possible subsets. The second step usually consists in training the classifier from a given subset of features and estimating its performance. The estimated performance is usually accessed by a validation dataset or cross-validation. The final subset of features, retained at the end of this procedure, is the one that has allowed to obtain the best performance.

However, exhaustive search systematically enumerates all possible combination of features and find the best feature subset, which is computationally intensive. For dataset with m features, the size of searching space is O(2 m ).

In the case of HDLSS problem, the feature dimension m is normally very big, an exhaustive search is intractable. This problem is known to be NP-hard ([106]). Wrapper methods usually adopt sub-optimal searches, such as sequential search, or heuristics algorithms ( [START_REF] Chandrashekar | A survey on feature selection methods[END_REF]). The sequential selection methods are iterative algorithms that add or remove features at each iteration until the maximum objective function is obtained ([188, 196]). For example, the Sequential Forward Selection (SFS) adds one feature each time so that the maximum classification accuracy is obtained until the required number of features are obtained ([196]). However, SFS suffers from producing nested subsets since the forward inclusion is always unconditional ( [START_REF] Chandrashekar | A survey on feature selection methods[END_REF]). The feature selected in the next iteration is highly dependent on the previous selected features. Heuristics algorithms such as Genetic Algorithm (GA) are also very often used for feature subset selection ([9, 63, 97]). The global maximum for the objective function (classification accuracy for example) can be found, which

gives the best sub-optimal subset ( [START_REF] Chandrashekar | A survey on feature selection methods[END_REF]).

To summarize, the main disadvantage of wrapper methods is the intensive computational cost. For each new subset, the classifier needs to be retrained to evaluate the performance. And overfitting may occur easily when the training instances are not enough. To provide better generalization ability, extra validation datasets are usually needed. For the HDLSS setting, it may be impossible to use an independent validation dataset for the search of the best feature subset. In the next section, the third category of feature selection methods, namely embedded methods, are presented, which are usually considered to be better alternatives to filter and wrapper methods.

Embedded methods

Embedded methods consist in selecting features during the training process without splitting the instances into training and validation sets ( [START_REF] Chandrashekar | A survey on feature selection methods[END_REF]). The general scheme of embedded methods for feature selection can be found in Figure 1.10c. Compared to wrapper methods, embedded methods have the main advantage to be less computationally intensive ([201]); while compared to filter methods, embedded methods take into account the interaction with classifiers.

One of the most used embedded methods are pruning methods. Pruning methods firstly train the classifier with the entire feature set and eliminate features gradually by some ranking criteria while maintaining the performance of the classifier. SVM-RFE ([107]) is the most famous pruning based embedded method. It is a recursive feature elimination method using the learned feature weight as ranking criterion ([107]). As shown in Algorithm 1, firstly, all the features are used to train the SVM classifier, then the weight and the ranking criterion of each feature is calculated. The feature with the smallest ranking is eliminated. This process continues until all the features are eliminated, and at last a ranked list can be given, so that we can choose how many features we want in the ranked list. This approach differs from the filter approaches by embedding the feature selection into the learning procedure, so that it can take the resulting classifier performance into account. SVM-RFE method is known to be efficient and accurate on many kinds of HDLSS applications ( [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]238,257]).

Apart from using SVM as the predefined classifier, neural networks can also be used for feature selection. A saliency measure calculated from trained multilayer perceptron networks is used to calculate the feature weights ([198, 205]). Network Pruning commonly used to obtain the optimum network architecture for neural networks can be used for feature selection ([52]). In [205],

a penalty is applied for features with small magnitude at the node and the nodes connecting to these input features are excluded. 4. Add the feature with lowest ranking criteria to r, and remove the feature from s. 8: Output: The updated feature ranks.

Discussion

Filter methods have the advantage not to require a validation set to perform feature selection, which is probably the reason why they are mostly used in many real world applications. However, they ignore the resulting classification performance and therefore, are usually less accurate than embedded and wrapper methods. On the other side, these two other families of approaches usually have a higher computational cost and are not very suitable for very low sample size problems.

Late integration

Late integration methods firstly build separate models on each view and combine them afterwards. These methods are named "late" because the data fusion process is done in the late stage of classification after the classifiers are trained. Most late integration methods belong to Multiple Classifier System (MCS) approach, which can be divided into two main categories: co-training and Independent Classifier Combination (ICC). The major difference between these two approaches is the interaction among classifiers: co-training methods re-train classifiers multiple times taking into account the information of classifiers from other views, while most ICC methods train classifiers once for each view independently and then merge the classification outputs. In the following sections, the details of these two approaches are given. confidently labelled instances by H (2) are chosen (while keeping the class ratio) to be added to the labelled training set T . Finally, 2p instances randomly drawn from U are then added to replenish U ′ . The updated T is then used to re-train H (1) and H (2) . This re-training process will stop until a termination condition (e.g. maximum iteration) is satisfied.

Co-training explores the relation between classifiers built on each view by maximizing the mutual agreement on two distinct views of the data in a semisupervised way. Usually, different views can provide different useful information, which indicates the differences between classifiers H (1) and H (2) during the first iterations. As the learning process is going on, the two classifiers are used to predict more and more instances from U , and their disagreement is expected to be smaller and smaller ([240]). Through information exchange between views, the final optimized classifier can be obtained ([29]). Nigam and Ghani [171] showed experimentally that even for single-view data, co-training on multiple views manually generated by random splits of features can still improve performance.

The unlabelled instances play an important role in co-training methods, which enable the information exchange between classifiers. However, as explained in the first chapter, HDLSS multi-view problems are usually composed of very few labeled instances and no additional unlabeled instances are available. Secondly, the co-training method is originally proposed to solve problems with two views. When there are more views, the solution will be much more complex. Thirdly, it is very hard for real world HDLSS multi-view problem applications to fulfill the three assumptions, which may lead to the failure of cotraining methods. As a consequence, co-training approaches are not straightforwardly applicable to HDLSS multi-view problems.

Independent Classifier Combination

Co-training makes many assumptions on the given multi-view problem and uses unlabelled instances to exchange information between classifiers, which limits its use in many real world applications. ICC based methods are more flexible by training one classifier for each view independently at first and then combining these classifiers in a proper way. In [111], the authors recall that posteriori probability produced by the probabilistic models embodied by the classifiers is the most popular support function fusion method ([247]). Dynamic weighting ( [START_REF] Cao | Dynamic voting in multiview learning for radiomics applications in Joint IAPR International Workshops on Statistical Techniques in Pattern Recognition (SPR) and Structural and Syntactic Pattern Recognition[END_REF]) can also be used to combine the decisions, but in the cost of using some validation dataset for most cases.

Aside from the rule based combination, trainable combination is also used in ICC. The new feature vector is constructed by combining the outputs of all the base classifiers, then a 'secondary' classifier can be built on these new features to give the final decision. For example, neural networks are used in [START_REF] Brunelli | Person identification using multiple cues[END_REF] to learn the combined matching score based on the classifiers built from face data and voice data. Other classifiers such as SVM, decision trees, Multilayer Perceptrons (MLP) and Random Forest are also used in [START_REF] Chatzis | Multimodal decision-level fusion for person authentication[END_REF][START_REF] Fierrez | Multiple classifiers in biometrics. Part 1: Fundamentals and review[END_REF]105,169,197,232]. Combinations with higher complexity can potentially provide better classification results. But validation datasets to train the secondary classifier are always required to guarantee the improvement. The small number of training instances will limit the choice of combination methods. Hence, choosing different combination approaches is a trade-off between the classifying capabilities of combination functions and the training sample size ([228]).

Discussion

Compared to co-training, ICC is more suitable for HDLSS multi-view problems because fewer assumptions are made beforehand and no extra unlabelled are needed to re-train the classifiers. ICC is usually faster compared to Co-training as all the classifiers are trained only once. However, in some cases, validation datasets are needed for a better combination of classifiers.

For both late integration methods, attention should be paid for the choice of classifiers due to the fact that a lot of classifiers can not deal well with high dimensional problems. For low sample size problem, Co-training method is not a good choice due to the need of many unlabelled instances while ICC may not be a good choice neither when validation datasets are required for the combination of classifiers.

Shared representation learning aims at finding a latent low dimensional space shared by all the views (see Figure 1.13): feature transformation functions for each view are learned with the assumption that all the views are generated from the latent space. Shared representation learning is an efficient multi-view dimensionality reduction technique as the dimension of the latent space is lower than any view, but it is mostly unsupervised and ignores the supervised information, which may lead to a subspace with weak predictive ability ( [START_REF] Chen | Predictive subspace learning for multi-view data: a large margin approach[END_REF]).

Canonical correlation analysis (CCA) is the most well-known shared representation learning method ([113, 131]). CCA works by seeking for a projection for each view so that the correlation among the projected views is maximized.

For multi-view problems with two views {X (1) , X (2) }, CCA learns two projections w 1 ∈ R m 1 and w 2 ∈ R m 2 for view 1 and view 2 respectively so that the following correlation between two projections is maximized:

ρ = w T 1 X (1) X (2)T w 2 (w T 1 X (1) X (1)T w 1 )(w T 2 X (2) X (2)T w 2 ) (1.4)
As ρ is invariant to the scaling of w 1 and w 2 , Equation 1.4 can be formulated as: max

w 1 ,w 2 w T 1 X (1) X (2)T w 2 s.t. w T 1 X (1) X (1)T w 1 = 1, w T 2 X (2) X (2)T w 2 = 1 (1.5)
However, CCA can not be applied directly to many real world datasets which exhibit non-linear characteristics, hence the kernel variant of CCA, namely KCCA ([112, 140]), was proposed to firstly map each instance to a higher space in which linear CCA can be applied. With the replacement of kernel matrices

K 1 = X (1)
T X (1) and K 2 = X (2)T X (2) , the optimization in Equation 1.5 can be rewritten as:

max w 1 ,w 2 w 1 T K 1 K T 2 w 2 s.t. w 1 T K 1 K T 2 w 1 = 1, w T 2 K 2 K T 2 w 2 = 1 (1.6)
Apart from the basic CCA and KCCA, there are also a lot of other studies related to CCA. In [260], the authors propose MKCCA for dimensionality reduction by performing PCA followed by CCA to better remove noises and handle the issue of trivial learning. More recently, more and more deep learning based CCA such as Deep CCA [START_REF] Andrew | Deep canonical correlation analysis in International Conference on Machine Learning[END_REF] and its variant [241] have also been proposed.

Multi-representation fusion

In contrast to shared representation learning methods, multi-representation fusion provides a more transparent way to take advantage of the complementary information among different views (see Figure 1.14). Multi-representation fusion projects each view in a space in which every instance is described by its (dis)similarities to all the training instances. In that way, each view is separately projected in the same description space so that linear or non-linear combinations can be applied directly. Multi-representation fusion is very flexible and efficient, and can be applied to many different types of data.

In [177], the authors proposed an SVM with heterogeneous kernel function, which firstly computes separate kernels for each view and then sums the results. Their proposed kernel is an attempt to incorporate prior knowledge into the task at hand. The kernel function is shown in Equation 1.7, where g and p stand for two different views (gene expression and phylogenetic profiles), and K is a local kernel. In the experiments, they compared the performance of single view data as well as three multi-view integration methods (early, intermediate and late integration), and showed that multi-representation fusion based method is the best performing among all the tested methods.

K combined (x 1 , x 2 ) = K g (x (g) 1 , x (g) 2 ) + K p (x (p) 1 , x (p) 2 ) (1.7)
From the heterogeneous kernel K combined , it is easy to tell the difference between shared representation learning and multi-representation fusion. Shared representation learning methods find the data projection to maximize the view correlation and the dimension of projected space needs to be predefined. The For multi-view problems, MKL uses one kernel for each view and the final kernel K η is used for SVM:

K η (x h , x k ) = f η ({K (q) (x (q) h , x (q) k )} Q q=1 ) (1.8)
where f η is a linear or non-linear combination function. The most successful kernels in the literature include linear kernel, polynomial kernel and gaussian kernel. Linear combination is the most popular approach in MKL ([100]), which contains simple sum or average of kernels as well as weighted combination (Equation (1.9)).

K η (x h , x k ) = f η ({K (q) (x (q) h , x (q) k )} Q q=1 ) = Q ∑ q=1 η q K (q) (x (q) h , x (q)
k )

(1.9) Lanckriet et al. [144] show how the kernel matrix can be learned from data via semidefinite programming (SDP) techniques. Better results are obtained than SVMs trained with each single kernel in 9 out of 13 experiments. Sim-pleMKL use the optimization of linear combination with kernel weights on a simplex ([195]). According to their comparison results, using multiple kernels is better than using a single one in terms of accuracy. However, trained linear combination is not always better than averaging for simple linear kernels.

For the combination of complex gaussian kernels, linear combination is better than nonlinear combinations, but still not better than unweighted combination. Their results show that simple average of kernels is a strong baseline.

Similar to the conclusion of [195], in [START_REF] Aiolli | EasyMKL: a scalable multiple kernel learning algorithm[END_REF], they also find out that the mean of kernels can obtain very good results. Hence they proposed a time and space efficient MKL method named EasyMKL by maximizing the distance between positive and negative examples ([255]). Their experimental results are shown to perform significantly better than the simple kernel averaging.

Apart from the combination of kernels, dissimilarity measures are also used to merge the information from different views. There are a large quantity of dissimilarity measures in the literature for all different feature types such as binary, categorical, ordinal, symbolic or quantitative features. There are no restrictions like being symmetric or positive semi-definite (PSD) for dissimilarity measures neither. Hence, a lot of studies have also tried to use multiple dissimilarity fusion.

In [153] for example, the authors propose a method to combine multiple dissimilarity measures together. They firstly define a distance measure D a based on Kullback-Leibler (KL) divergence, which measures directly the difference between a query and a prototype. Then, they define another distance measure D d which only considers inter-relations between different training instances. Similarly, Heterogeneous Auto-Similarities of Characteristics (HASC) is proposed to deal with heterogeneous data with a combination of covariance matrix of features (COV), and Entropy and Mutual Information (EMI) matrix ([202]). Many other similar works of combining multiple dissimilarity matrices with heuristic rules can be also found in [START_REF] Amerise | Weighting Distance Matrices Using Rank Correlations tech[END_REF]121,150,185]. An adaptive bilinear mixing of dissimilarities is also proposed in [128]. They claim that if the data are heterogeneous, a single dissimilarity measure might not be sufficient to describe the relations between the data. They focus on the prototype based learning like learning vector quantization dataset. Similar to the idea of MKL, the combination of dissimilarities most adequate for the classification task is also learned. Generalized Learning Vector Quantization (GLVQ) is used to integrate bilinear mixing and weighting of dissimilarities in prototype-based classification learning.

Discussion

In summary, shared representation learning is an efficient multi-view dimensionality reduction technique. However CCA based methods are unsupervised and ignore the supervised information, which may lead to a joint space with weak predictive ability. Multi-representation fusion projects each view in a space in which every instance is described by its similarities (kernel functions) or dissimilarities to all the training instances. In that way, each view is separately projected in comparable lower description spaces, which provides a good solution for HDLSS problem ( [START_REF] Costa | The dissimilarity approach: a review[END_REF]). Then the joint data representation can be obtained by searching for the best combination of the new data representations, which provides a good solution for multi-view problems.

Conclusion and contributions

In this chapter, we have introduced the HDLSS multi-view problem through the example of Radiomics. From the perspective of multi-view learning, Radiomic features are always from multiple distinct feature groups or different imaging modalities to better represent the tumor heterogeneity. From the perspective of HDLSS, Radiomic features are easily to be high dimensional (over 1000 features), especially when combined with transfer learning features. But the sample size of Radiomics is usually very small, normally fewer than 100 instances. Most of the state-of-the-art works in Radiomics concatenate all the feature groups together as a single feature vector, which often results in a very high dimension. Hence feature selection is the most used method to reduce the dimension. However, if only a small subset of the features are chosen, certainly a lot of useful information is lost and the heterogeneity can not be well represented. By concatenating all feature groups together, the complementary information from different feature groups is often ignored.

To make better use of multi-view information, we have reviewed the stateof-the-art multi-view solutions including early integration, intermediate integration and late integration with an emphasis on HDLSS problems. We have shown that early integration ignores the potential complementary information that different views may offer, while late integration methods are not very suitable for low sample size problem since they may require additional training instances to optimize the combination of classifiers. In our opinion, intermediate integration methods offer a better way to deal with the HDLSS multiview problem by studying the relations between views and combining the views together so that traditional machine learning methods can be applied in a joint low dimensional space. Among the different intermediate integration approaches introduced above, we have shown that multi-representation fusion is the most appropriate solution for the HDLSS multi-view problem for the following reasons: to deal with the HDLSS problem, features from each view can be represented by (dis)similarity matrix, which leads to a lowdimensional description space for HDLSS problems; it also makes the fusion of each view very straightforward since dissimilarities are always comparable from one view to another; then, the multi-view problem can be solved in a more transparent way by searching for the best combination.

Multi-representation fusion can be divided as a two stage learning process: the first stage is to learn the new data representation to project data from each view into a lower common space; the second stage is to learn the combination of views to better exploit the complementary information among views. Most multi-representation fusion methods focus on the second step only. For example, multiple kernel learning, the richest literature in multi-representation fusion, focus more on the second step about how to combine different kernels while ignoring the first step about how to learn the appropriate kernel for each view (most MKL methods just choose the well-known predefined kernels such as linear polynomial or gaussian kernels without identifying the nature of the data). In this work, we believe that more attention needs to be paid on the first step: firstly, data from different views normally have different nature, type or complexity. An appropriate (dis)similarity representation should be learned accordingly. Secondly, learning the proper (dis)similarity representation for each view makes the data combination more meaningful because the only information shared by all the views (the class information) will be included in each (dis)similarity representation.

The main contributions of this thesis are:

• Random Forest Dissimilarity (RFD) is chosen as an intermediate representation for multi-representation fusion. To deal better with HDLSS problem, the parameterization of RFD is also studied. In the experiments in Chapter 2, by comparing with different early, late and intermediate integration methods on real world datasets, RFD based intermediate integration method show the potential of being a good solution for HDLSS multi-view problem.

• Some limitations and possible modifications of the classic RFD measure • Instead of using simple averaging to form a joint dissimilarity matrix, static weighting and dynamic weighting methods are explored to take better advantage of the complementary information. A static weighting based on OOB accuracy and a dynamic view selection method are proposed. From the experimental results in chapter 4, both methods can improve the classification performance while the dynamic view selection is significantly better than averaging.

Introduction

The previous chapter introduced the HDLSS multi-view problem. In particular, we have shown that a key aspect of multi-view learning is to efficiently exploit the complementarity of the views. To do so, multi-view methods based on the fusion of multiple intermediate representations are the most relevant from our point of view, especially for HDLSS problems. Multi-representations fusion methods deal with HDLSS multi-view problems through two steps : i) building new low dimensional representations of the data from each view separately, and ii) merging these view-specific representations into a joint representation.

Most multi-representation fusion methods focus on learning a good (parametric) combination operator rather than learning good view-specific intermediate representations. Dissimilarity representations are usually chosen as the intermediate representation because they offer the pairwise information between instances indicating if these two instances are similar or dissimilar. As the pairwise information is comparable across the views, it makes the representation merging task easier. Another advantage of using dissimilarity representation is that the original high dimensional n × m data (n is sample size, m is feature size) will be presented as a n × n matrix, which offers a natural solution to HDLSS problem.

There are a lot of different dissimilarity measures in the literature that can be used for multi-representation fusion. In general, they can be divided into two groups: learning free measure and learning based measure. Learning free measures are general purpose measures (e.g., the Euclidean distance and the cosine similarity for feature vectors). They are mostly defined without specific context and are problem independent (not learned from data). Learning based measures learn a specific measure for the data by taking the class information into account. For example, in the case of classification tasks, the goal would be to make the distance reflect the best possible the class membership: instances from the same class should be close to each other while instances from different classes should be far from each other [START_REF] Bellet | Supervised metric learning with generalization guarantees[END_REF]206]. For multi-view problems, the features from different views may be very diverse, and the only information shared by all the views is the output information. Using learning based dissimilarity measure as the intermediate representation can help to filter the "noisy" information from each view that is not related to the classification task and make the fusion task more efficient and transparent. Hence, in this work we firstly focus on generating the proper intermediate representation with class information and secondly focus on the combination. The experimental results of this chapter have been published in [START_REF] Cao | Random forest dissimilarity based multi-view learning for Radiomics application[END_REF].

The remainder of this chapter is organized as follows. In section 2.

Related works

In the literature, methods that learn a similarity or dissimilarity measure from a dataset can be generally called dissimilarity learning. Dissimilarity is a very general term which relates to many notions such as distance, kernel, similarity etc. With different problem formulations or constraints, there exists a considerable number of approaches that aim at learning the dissimilarity, among which the most useful ones include metric learning, kernel learning and random partitions. The details of each of these methods are given in the following sections.

We first give the definitions of distance, kernel, similarity and dissimilarity. In terms of classification, instances from the same class should be similar in some way and instances from different classes should be dissimilar. The notion of "(dis)similarity" plays a pivotal role in pattern recognition and machine learning. Similarity measure is a numerical measure of how close two instances are. The value is bigger when two objects are closer. On the contrary, dissimilarity measure is a numerical measure of how different two instances are. The value is smaller when two instances are closer. Normally, it's possible to transfer a similarity value into a dissimilarity value and vice versa. The notion of proximity usually refers to similarity or dissimilarity. A specific dissimilarity form (distance) and a specific similarity form (kernel) are defined in the following.

Distance function

A distance function over the domain X is a pairwise function d(•, •): X × X → R which satisfies the following properties:

• Reflexivity: d(x i , x i ) = 0 • Definiteness: d(x i , x j ) = 0 ⇒ x i = x j • Nonnegativity: d(x i , x j ) ≥ 0 • Symmetry: d(x i , x j ) = d(x j , x i ) • Triangle inequality: d(x i , x k ) ≤ d(x i , x j ) + d(x j , x k )
When all the previous properties are respected except for the definiteness, the function is called a pseudo distance.

Kernel function

A symmetric similarity function K(•, •) is a kernel if K(•,
•) can be written as an inner product in Hilbert space H with the mapping function Φ : X → H:

K(x i , x j ) = Φ(x i ), Φ(x j ) (2.1) Equivalently, K(•, •) is a kernel function if it is positive semi-definite (p.s.d): n ∑ i=1 n ∑ j=1 a i a j K(x i , x j ) ≥ 0 (2.2)
for all finite sequences x 1 , . . . , x n ∈ X and a 1 , . . . , a n ∈ R ( [START_REF] Bellet | Supervised metric learning with generalization guarantees[END_REF]).

The most popular kernels in the literature are the Linear kernel, the Polynomial kernel and the Gaussian Radial Basis Function (RBF) kernel. :

Linear kernel : K lin (x i , x j ) = x T i x j Polynomial kernel : K poly (x i , x j ) = (x T i x j + 1) p Gaussian kernel : K gau (x i , x j ) = exp(- ||x i -x j || 2 2σ 2 ) (2.3)
where p is the hyperparameter (degree) of polynomial kernels; σ is the hyperparameter (bandwidth parameter) for gaussian kernel.

Compared to the term of distance or kernel, dissimilarity and similarity are more general terms, which do not have the constraints to be a metric or positive semi-definite. A dissimilarity can be asymmetric, non-PSD or can violate the triangle inequality ([182]).

Metric learning

The goal of metric learning is to learn a distance metric function d(x i , x j ), for all (x i , x j ) ∈ T × T , which mostly consists in estimating some parameters from the data, in order to make a generic distance function suit the best possible to some constraints defined by the ground truth. The most popular form of metric learning methods is based on the Mahalanobis distance due to its simplicity and nice interpretation in terms of a linear projection ( [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF]).

Mahalanobis distance

d M (x i , x j ) = (x i -x j ) T M(x i -x j ) (2.4)
Originally from [161], the Mahalanobis distance is a measure that takes into account the correlation between features. The Generalized Mahalanobis Distance (GMD) formula is shown in Equation (2.4): when M = Ω -1 (where Ω is the covariance matrix), Equation (2.4) represents the original Mahalanobis distance. Most metric learning methods strive to estimate the parameter M from a training set, and with the goal to make the resulting distance measure reflect the output, e.g. the class membership for classification tasks. According to the survey of [START_REF] Bellet | A survey on metric learning for feature vectors and structured data[END_REF], the positive semi-definite matrix M is mostly learned from pair constraints or triplet constraints, which usually have the following form:

• Must-link / cannot-link constraints (sometimes called similar set / dissimilar set):

S p = {(x i , x j ), x i and x j should be similar}

D p = {(x i , x j ),
x i and x j should be dissimilar}

• Relative constraints (sometimes called training triplets):

R t = {(x i , x j , x k ), x i should be more similar to x j than to x k }
All the methods in metric learning try to optimize at least one of the following objectives ([148]):

min M ∑ (x i ,x j )∈S p d M (x i , x j ) (2.5) max M ∑ (x i ,x j )∈D p d M (x i , x j ) (2.6)
The optimization can be written in a more general way:

min M l(M, S p , D p , R t ) + λR(M) (2.7)
where l() is a loss function to measure the loss when specified constraints are violated; λ is the parameter used to control the regularizer R(M). Generally speaking, the Mahalanobis metric learning formulations differ by the choice of constraints, loss function and regularizer ([24]).

One simple example for the illustration of metric learning is shown in Figure 2.1: the original instances in the Euclidean space are shown on the left pane.

With constraints of making instances from the same class closer and instances from different class farther, the metric learning objective is to get instances on the right pane with fewer violations of constraints.

Kernel learning

Kernel functions normally have the form of Equation 2.1, representing the inner product of two vectors (see in the beginning of 2.2). If these two vectors are unit vectors, the inner product also represents the angle between these two vectors. Therefore, a kernel is also often interpreted as the similarity between instances and kernel learning is often seen as similarity learning ( [1]). Unlike distance metric learning methods, the biggest constraint of learning a kernel is the positive semi-definiteness ([129]). According to [1], there are generally three families of kernel learning: data-dependent, nonparametric and parametric.

Data-dependent kernel learning

For using the polynomial and the RBF kernel, both hyperparameters p and σ need to be set a priori and are sometimes quite complicated to be tuned ([239]). In contrast, data-dependent kernels such as Fisher kernel ([123, 230])

or marginalization kernel ([116]) strive to learn the parameters of the kernel from training instances with generative models. However, the positive semidefiniteness still needs to be proven mathematically for any data-dependent kernel, which sometimes constitutes an obstacle for the development of new kernels of the kind, and which also makes this approach less popular than using pre-defined kernels.

Nonparametric kernel learning

In contrast to metric learning methods, nonparametric kernel learning does not use any prior model. The kernel matrix is learned without a pre-defined kernel form that implicitly generates it ([117, 118, 194]). The objective function of the nonparametric kernel learning is usually expressed as a set of userdefined criteria that aim at finding the best kernel. The downside of these methods is that during testing, the best kernel must be built from the training and test examples, which makes them impossible to be applied to new instances.

Parametric kernel learning

Most of the available approaches in the literature of kernel learning lie in this family. Parametric kernel learning is usually formed as an optimization problem to find the parameters of a predefined model with respect to the userdefined criteria ( [1]). Methods in this family can be divided into two categorizes based on the number of predefined kernels used:

• Single kernel: With a single predefined kernel, the objective of parametric learning is to improve the base kernel and make it optimal for the learning task ([2, 10, 55]). There are generally two ways to realize this objective: the first one is to find the appropriate hyperparameters for the kernel and the second one is to find the transformation from the base kernel to the optimal kernel. Kernel alignment ( [START_REF] Cortes | Algorithms for learning kernels based on centered alignment[END_REF]239]) methods are very often used in this category.

• Multiple kernels: With a set of available kernels, the objective of multiple kernel learning is to find the best combination of these kernels linearly or non-linearly ( [START_REF] Argyriou | A DC-programming algorithm for kernel selection[END_REF][START_REF] Bach | Multiple kernel learning, conic duality, and the SMO algorithm[END_REF]). Multiple kernel learning methods are the most popular solutions in this category, which have been introduced in the previous chapter.

Random partitions

Most of the methods presented so far in this chapter follow the same core principle: optimizing some parameters of a predefined generic model, with respect to some constraints. Random partitions adopt a different approach in the sense that the method strives to infer the model from the training instances only, without any prior formulation of the measure. The key idea of Random Partitions is to define multiple randomized partitions of the input space that group the instances according to their class membership. It has been proven that such random partitions can be used to define kernels, and as a consequence to define dissimilarity measures.

Given a dataset T , a cluster C is a non-empty subset of T . The partitions of T divide T into multiple non-overlapping clusters ̺ = {C 1 , . . . , C N } with respect to:

C i ∩ C j = ∅ ∪ i C i = T (2.8)
A random partition of T is a sample from the partition distribution P, where P is a discrete probability density function (pdf) that represents how likely a given clustering is ( [START_REF] Davies | The random forest kernel and other kernels for big data from random partitions[END_REF]). For any instance x, ̺(x) gives the cluster x belongs to.

A lot of works have been done on random partitions especially in the field of non-parametric Bayesian statistics [START_REF] Aldous | École d'Été de Probabilités de Saint-Flour[END_REF]141,184,199]. In recent years, the relation between random partitions and dissimilarity measures has been highlighted. In [START_REF] Davies | The random forest kernel and other kernels for big data from random partitions[END_REF], the authors find out that kernels can be generated from random partitions. Given a random partition distribution P, a kernel can be defined as:

K P = E[I[̺(x i ) = ̺(x j )]] ̺∼P (2.9)
Random partitions can be easily constructed from existing machine learning methods. For example, any clustering algorithm such as K-means ([160]) or DBSCAN ([86]) can be transformed to stochastic clustering algorithm by adding the randomness (different initializations, number of clusters, feature projections, etc.). The output of stochastic clustering algorithm is a random partition. Another example is decision tree based ensemble methods. The nodes of a decision tree divide instances into different non-overlapping partitions, which makes the ensemble of trees such as Random Forest ( [START_REF] Breiman | Random forests[END_REF]), boosted decision trees ( [START_REF] Freund | A decision-theoretic generalization of online learning and an application to boosting[END_REF]) or bayesian additive regression trees ( [START_REF] Chipman | Bayesian additive regression trees[END_REF]) a natural solution to construct random partitions. A simple kernel generated from the random partitions of Random Forest is given in [START_REF] Davies | The random forest kernel and other kernels for big data from random partitions[END_REF] by generating the random partition from the tree depth sampled randomly of the trained decision tree.

The similarity measure generated from random partitions can be easily transformed to the dissimilarity measure. In [START_REF] Dudoit | Bagging to improve the accuracy of a clustering procedure[END_REF], the authors proposed a partitioning clustering procedure with bootstrap learning sets to improve the accuracy of a given clustering procedure. M bootstrap sets are used to create a dissimilarity matrix: for each bootstrap dataset, if object i and object j belong to the same cluster, the similarity between them S ij adds 1. At last the dissimilarity matrix is obtained by:

D ij = 1 - S ij M (2.10)

Discussion

In this section, different dissimilarity learning methods have been introduced. Generally speaking, most distance metric learning methods are based on Mahalanobis distance learning due to its simplicity and nice interpretation, but these methods require some pre-specified free parameters, and most of them involve some expensive computational procedures such as eigenvalue decomposition or semi-definite programming. Although a lot of metric learning methods have been proposed and shown to perform well in many different applications, few of them try to deal with HDLSS problem ([24, 148, 252]). Since most methods learn O(m 2 ) parameters , metric learning methods are intractable for real-world high dimensional applications. Kernel learning learns the dissimilarity measure in the form of a kernel which can be seen as a similarity function. However, kernel learning methods usually require a large labelled dataset either for defining an ideal kernel or for cross validation ( [1]).

Compared to these two methods, dissimilarity learning based on random partitions is more flexible in the way that it can learn both a dissimilarity measure and a similarity (kernel) measure.

To deal with HDLSS problem, Random Forest Dissimilarity (RFD) is introduced in the next section. RFD allows to overcome the aforementioned drawbacks as it is proved to be particularly robust to high dimensions and as it does not require an exponential amount of training instances.

Random Forest dissimilarity measure 2.3.1 Random Forest

Random Forest (RF) has been a very popular data mining and statistical tool for years due to its transparency and great success in classification and regression tasks as well as in unsupervised learning ( [4,209]) or active learning tasks ([101, 162]). In the past 15 years, it has shown to be among the most accurate general purpose machine learning methods on a wide variety of realworld problems as illustrated by the consequent experimental comparison in [START_REF] Fernández-Delgado | Do we need hundreds of classifiers to solve real world classification problems[END_REF]. One other important property of RF is that it does not overfit if there are enough trees in the ensemble according to Breiman's work ([35]). We now recall the RF principle.

The name "Random Forest" in this chapter refers to Breiman's work in [START_REF] Breiman | Random forests[END_REF].

The algorithm works by growing M different (randomized) trees with the following rules ( [START_REF] Biau | A random forest guided tour[END_REF]). Firstly, a bootstrap sample is generated for each tree by randomly selecting n instances, with replacement, from the initial training set made up of n different instances. Each of these bootstrap samples is then used to build one tree. During this induction phase, at each node of the tree, a splitting rule is designed by selecting a feature over mtry features chosen uniformly at random among the m initial features. This selection can be performed by maximizing the well-known Gini impurity criterion. At last, while the induction of a single tree is usually prematurely stopped by a stopping criterion, e.g. a minimum number of training instances in the node, Random Trees in Random Forest classifier are grown to their maximum depth. As for the final prediction of the RF, it is obtained via majority voting over the component trees ( [START_REF] Biau | A random forest guided tour[END_REF]). The resulting Random Forest classifier with M decision trees is typically noted as: 2.11) where h k (x) is a random tree grown using the process discussed above. We refer the reader to [START_REF] Biau | A random forest guided tour[END_REF][START_REF] Breiman | Random forests[END_REF] for more details about this procedure. Note however that there exist many different RF learning methods that differ from the one in [START_REF] Breiman | Random forests[END_REF] by the use of different randomization techniques for growing the trees. We choose to use this reference method since it is the most commonly used in the literature and since each RF learned is mainly used to compute the dissimilarities and not only to exhibit the best accuracies.

H(x) = {h k (x), k = 1, . . . , M} (
For predicting the class of a given test instance x i with a random tree, x i goes down the tree structure, from its root till its terminal node. The descending path is decided by successive tests on the values of the features of x i , one per node. The prediction is given by the terminal node (or leaf node) in which

x i has landed. We refer the reader to [START_REF] Biau | A random forest guided tour[END_REF] for more information about this process.

Hence if two test instances land in the same terminal node, they are likely to belong to the same class and they are also likely to share similarities in their feature vectors, since they have followed the same descending path. This is the main motivation behind using Random Forest for measuring dissimilarities between instances, by using the procedure explained in the following.

Random Forest Dissimilarity (RFD)

The RFD measure is inferred from a RF classifier H, learned from T . Let us firstly define a dissimilarity measure inferred by a decision tree d (k) : let L k denote the set of leaves of the k th tree, and let l k (x) denote a function from X to L k that returns the leaf node of the k th tree where a given instance x i lands when one wants to predict its class. The dissimilarity measure d (k) , inferred by the k th tree in the forest is defined as in Equation (2.12): if two training instances x i and x j land in the same leaf of the k th tree, then the dissimilarity between both instances is set to 0, else set to 1.

d (k) (x i , x j ) =    0, if l k (x i ) = l k (x j ) 1, otherwise (2.
12)

The RFD measure d (H) (x i , x j ) between x i and x j consists in calculating d (k) for each tree in the forest, and in averaging the resulting dissimilarity values over

RFD matrix

Let D denote a n × n matrix, called a dissimilarity matrix, built from a given RFD measure d and from a training set T , and defined as in Equation (2.14): In the same way, using the dissimilarity to each of the training instances, any new instance x i can be mapped into a n dimensional dissimilarity space DS. For HDLSS problem, the dimension of this dissimilarity space is necessarily smaller than the dimension of the original feature space.

D =       

Properties of RFD matrix

In the following, we give the proof that the RF similarity matrix is symmetric and positive semi-definite. These two properties are essential since they ensure that such a matrix can be used as a kernel matrix in kernel methods like non-linear SVM ([181]). Note that RF similarity matrices are easily obtained from the RFD matrices defined in the previous section, by S H = 1 -D H . RF similarity is also called the RF proximity in the literature.

Let us firstly recall the two following theorems from [147], that gives the basic conditions for determining whether a matrix is p.s.

MA ij ≤ MA ii , (1 ≤ i < j ≤ n) (2.15) MA il + MA jl ≤ MA ll + MA ij , (1 ≤ i < j ≤ n, l = i, j) (2.16)
Using these theorems, let us demonstrate that the similarity matrix inferred by a single tree k, noted S (k) is p.s.d. From Equation (2.12), one can see that S (k) has the following properties:

• S (k) is a symmetric matrix with principal diagonal values equal to 1.

• The off diagonal entries in S (k) are either 0 or 1.

One can reasonably consider that the number of training instances available is greater than 3, and as a consequence, that S (k) is a symmetric binary matrix ∈ (0, 1) n×n , with n ≥ 3. According to Theorem 2.3.2, for this matrix to be p.s.d., it needs to satisfy both Equation (2.15) and Equation (2.16):

1. As S (k) is a symmetric binary matrix with principal diagonal values

S (k) ii equal to 1, hence S (k) ij ≤ S (k)
ii , which satisfies Equation (2.15).

2. To prove S (k) satisfies Equation (2.16), two situations need to be considered:

(a) If S (k) ij = 1, then S (k) ll + S (k) ij = 2. Since S (k) il ≤ 1 and S (k) lj ≤ 1, then S (k) il + S (k) lj ≤ S (k) ll + S (k) ij . (b) If S (k) ij = 0, then S (k) ll + S (k) ij = 1. At the same time, S (
(k) il + S (k)
lj is necessarily less or equal to 1 and as a consequence, S (k) also satisfies Equation (2.16). This proves that S (k) meets the requirements of Theorem 2.3.2, and is a p.s.d. matrix. It follows from Theorem 2.3.1 that the sum of all S (k) , ∀k = 1..M, is also p.s.d., meaning the RF similarity matrix S H or any linear combination of S H is also p.s.d.

Multi-view learning with RFD matrix

For multi-view learning, one needs now to fuse the dissimilarity matrices built on each view and to learn a classifier from the resulting joint dissimilarity matrix.

A natural way to fuse the dissimilarity matrices is to compute the unweighted average matrix. For multi-view learning tasks, the training set T is composed of Q views: T (q) = {(x (q) 1 , y 1 ), . . . , (x (q) N , y N )}, q = 1..Q. From these views, Q RFD matrices are computed following Equation (2.14) and noted {D (q) H , q = 1..Q}. For multi-view learning, the joint dissimilarity matrix D H can be computed as in Equation (2.17).

D H = 1 Q Q ∑ q=1 D (q) H (2.17)
According to the work in [START_REF] Duin | The dissimilarity space: Bridging structural and statistical Pattern Recognition[END_REF], learning from a dissimilarity matrix D H can be done in two different ways: (i) by using the corresponding similarity matrix Multi-view Random Forest kernel SVM (RFSVM): Instead of using traditional kernels, such as the Gaussian Radial Basis Function kernel, SVM classifiers can be efficiently trained on user-defined kernels. For example, in [108], the authors proposed a problem dependent distance measure to construct a substitution Gaussian kernel. Such a user-defined kernel can be supplied to SVM classifiers as a kernel matrix as long as it is positive semi-definite (p.s.d).

For RFSVM, the joint similarity matrix S H is used as a kernel matrix. The proof that it is p.s.d. has been given in the previous section. Then, given a test following hyperparameters are assumed to be crucial for having a "good" RFD measure:

• Forest size M: As explained in [START_REF] Bernard | Using random forests for handwritten digit recognition in Ninth International Conference on Document Analysis and Recognition[END_REF][START_REF] Breiman | Random forests[END_REF], it is now known that the RF accuracy converges for an increasing number of trees in the forest. One can naturally wonder if the same goes for the quality of the corresponding RFD measure. As explained in the previous section, RFD between two instances is computed by averaging over the dissimilarity values inferred by each tree. If the number of trees is very small, say 5 trees for example, the RFD estimate would always be one of the 5 following values: 0, 0.2, 0.4, 0.6, 0.8 or 1. Obviously, this is not accurate enough for describing (dis)similarities between instances. When the number of trees increases, the RFD value is expected to be more accurate and reliable.

• Tree depth δ: The rationale behind studying the influence of the tree depth on the RFD quality is less obvious. When the node is deeper down the tree structure, it is usually more "pure", that is to say it gathers training instances from the same class mostly. This is desirable since it means the RFD values will reflect the class membership: two instances from the same class will be considered quite similar. However, at the same time, the deeper the node, the smaller it will be, that is to say the fewer instances it will gather. As a consequence, the resulting RFD matrix is likely to be sparse, and the dissimilarity measure too loose.

To illustrate the influence of these hyperparameters, an RF classifier is built on the dissimilarity matrix induced from different combinations of numbers of trees and tree depths on three toy datasets. The three toy datasets, composed of 100 instances, two features and two classes, have different shapes and complexities, as shown in the first column of Figures 2.4a and 2.4b: (i) the first row is a dataset with two isotropic Gaussian classes to show how the RFD measure behaves differently from a traditional Euclidean distance measure;

(ii) the second row is a donut-shaped dataset, more complex with regards to similarity measures because, contrary to the RFD measure, a distance-based dissimilarity would fail to represent the class membership; and (iii) the third row is a banana-shaped dataset used to confirm that the RFD measure can Figure 2.4a shows the influence of the number of trees when their depth is set to the maximum. For the dataset (i) (first row), 8 trees are enough to achieve good performance. But for the two other datasets, the influence of the number of trees can be better highlighted. For an increasing number of trees in the forest, the quality of RFD gets better as shown in Figure 2.4a, and it is also reflected on the decision frontier. It can be seen that for both datasets, the decision frontiers better suit to the classes (i.e. describe more and more correctly the data structure) when the number of trees increases. Therefore if we want the RFD measure to be accurate for each of the cases, it is necessary to have as many trees as possible.

Figure 2.4b shows the influence of the tree depth for a forest of 1024 trees.

For the dataset (i), there is no difference in the three scatter plots because the maximum depth is equal to 1. For the two other toy datasets, it can be seen that the decision frontiers are sharper and better fit the training set when the tree is deeper. In particular, when the tree depth is not maximum, the decision boundaries are not sharp enough: this is because, in this case, the trees fail to capture the class membership of similar instances.

In summary, these results show that if we want the RFD measure to be accurate, it is necessary to have the maximum tree depth, with a large number of trees in the forest.

Experiments on real-world datasets

To confirm the trends observed on the toy datasets, the hyperparameters are further studied on real-world multi-view datasets. A general description of these datasets can be found in Table 2. All these datasets are multi-view datasets, that is to say they are supplied with several views of the same instances. However, as the goal of this first experiment is to study the effect of hyperparameters on the quality of the RFD measure, we considered the 71 views (coming from the 15 datasets) separately, as independent datasets. The reason we decided to use multi-view dataset for this experiment is to be able to re-use the same datasets in our next experiments.

Both hyperparameters M and δ have been tested with the following values:

• Forest size M ∈ {8, 16, 32, 64, 128, 256, 512, 1024}: first, an RF with 1024 trees is built; the performance is then monitored with the first 8 trees, the first 16 trees, and so on, until all the 1024 trees are used in the RF. Recall that for training a Random Forest, trees are grown independently from each other. Therefore, retaining a subset of trees in a forest already built is just a mean to save computation time.

• Tree depth δ : an RF is firstly built with fully grown trees. For each RF, the maximum tree depth δ max is computed. Then, the quality of the RFD is measured by only considering nodes above depth i ∈ {1, 2, 3, . . . , δ max }, that is to say by considering that each branch of each tree has not been grown beyond depth i.

Following the conclusion of [START_REF] Duin | The dissimilarity space: Bridging structural and statistical Pattern Recognition[END_REF], the quality of the RFD measure obtained with different combinations of these hyperparameters is now assessed with a 1-Nearest Neighbor classifier (1NN): 1NN selects the nearest neighbor for a test instance according to the dissimilarity values and assign the label of

Results on real-world datasets

The results on the 71 views are presented in this section as mean and standard deviations of the classification rates over the 50 runs. To better illustrate the results, a 2D color-map is drawn for each dataset, as in Figure 2.5. The warm color (yellow) stands for a relatively high quality of the RFD as measured by the 1NN accuracy while the cold color (blue) stands for relatively low quality. The y-axis corresponds to the number of trees, and the x-axis corresponds even with very few instances for learning, the trend is still observable on these figures.

Discussion

From this study on the parametrization, one can see that the general trend for all datasets is similar: the RFD measure is more reliable when the RF contains more trees and when these trees are fully grown. From the overall comparison on the real-world datasets, the maxmax setting (1024 trees with maximum depth) appeared to be better than the maxhalf setting (128 trees with maximum depth) but not statistically significantly, which means that 128 trees already allow to obtain a quite good RFD measure for most of the views. For a better insight into this, Figure 2.7 shows the result of the Nemenyi post-hoc test when focusing on the number of fully grown trees. It shows that the performance gaps for forests from 256 to 1024 trees are not statistically significant.

However, these differences in terms of average ranks, observable on this fig-

ure, are still important enough from our point of view to consider using more than 256 trees. It is worth noting that the computational cost of learning an RF classifier is directly proportional to the number of trees ([157]). Hence, in all the remaining experiments, all the RF have been learned with 512 trees as a good compromise between reliability and computational costs. And of course, all the trees have been fully grown all along these experiments.

RFD-based multi-view learning

As far as we know, only one method has already been proposed that perform learning from a joint RF dissimilarity matrix: the method in [START_REF] Gray | Random forest-based similarity measures for multi-modal classification of Alzheimer's disease[END_REF], named RFMDS in this chapter. This method is similar to RFDis except for two aspects.

First, the computation of the joint similarity matrix differs in that, for RFSVM and RFDis, it is an unweighted average combination whereas in RFMDS, it is a linear combination with weights optimized through a coarse-grained grid search. Second, RFSVM works in a kernel space and RFDis works in a dissimilarity space, whereas RFMDS works in an embedded space obtained with For this experimental validation, six methods are compared: one state-of-theart Radiomics solution based on feature selection, namely SVM-RFE ([238, 257]); four intermediate integration methods, i.e. the proposed RFSVM and RFDis presented in section 2.5, the RFMDS method proposed in [START_REF] Gray | Random forest-based similarity measures for multi-modal classification of Alzheimer's disease[END_REF] and the MKL method EasyMKL ( [START_REF] Aiolli | EasyMKL: a scalable multiple kernel learning algorithm[END_REF]); and one RFD-based late integration method, namely LateRFDis, from [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]. This latter method is a basic MCS architecture, which firstly builds an RFD matrix on each view, then trains an RF classifier on each of these dissimilarity matrices, and finally combines these RF classifiers by majority voting (However, we also explore the potential of some more complex late integration methods and the results can be found in Appendix B). All these methods are sum up in Table 2.3. For the SVM-RFE method, the number of features to select, which is a hyperparameter of the method, is set according to the total number of features following the rules described in [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF]. An RF classifier is then built from the selected features. For all the RF classifiers used in this experiment, the number of trees is set to 512 as explained in section 2.4.3, while the other parameters are set by default as proposed in the Scikit-learn machine learning framework ([178]). As for the SVM based method, the usual hyperparameter C is used to define the penalty factor. Its value is classically set using a grid search with cross-validation. For EasyMKL, a similar grid search with cross-validation strategy is used to find the best combination of kernels among a pool of linear kernels, gaussian kernels and polynomial kernels with different hyperparameters, following the protocol given in [195].

Finally, similar to the preliminary study, a stratified random splitting procedure is repeated 10 times, with 50% of the instances for training, 50% for testing. In order to compare the methods, the mean and standard deviations of accuracy are evaluated over 10 runs.

Results and discussions

The results of this experimental comparison are given and discussed in the following, firstly for non-Radiomic datasets and secondly on the Radiomic datasets.

Results on non-Radiomic datasets

The average classification rates over the 10 repetitions, along with standard deviations, are shown in Table 2.4. From the average ranking, it can be seen that the RFSVM method performs globally the best among the six methods, followed by the RFDis and EasyMKL methods, while the state-of-the-art Radiomics solution (i.e. SVM-RFE) is ranked the worst.

Comparison of the multi-view solutions and the feature selection method:

From Table 2.4, one can see that all the multi-view methods are globally better than the feature selection method SVM-RFE. To better assess the difference, a pairwise analysis based on the Sign test is computed on the number of wins, ties and losses as in [START_REF] Cruz | Enhanced online pruning of base classifiers for dynamic ensemble selection[END_REF]. The result is shown in Figure 2.8 (a). All the multiview solutions are compared to SVM-RFE: each vertical line indicates the critical value corresponding to a confidence level α equal to 0.10 and 0.05. If the number of wins is above these lines, the corresponding method can be considered to be significantly better than the baseline method. Comparison of the RFD-based methods (RFSVM, RFDis, RFMDS): Let us recall that each of these methods exploits the RFD measure in the three different possible ways for dissimilarity-based classification according to [START_REF] Duin | The dissimilarity space: Bridging structural and statistical Pattern Recognition[END_REF]180]:

RFSVM uses a kernel space, RFDis a dissimilarity space, and RFMDS an embedded space. From Table 2.4, it can be seen that RFSVM clearly outperforms the two other methods which indicates that kernel space seems to be the best approach. Note that the RFMDS method is quite accurate on 2-class problems, but not so much for multi-class problems, while the proposed RFSVM and RFDis are as accurate for both. The reason may be that the RFMDS is based on an embedded space as explained in section 3, which may suffer from a loss of information as the dimension is reduced. This information loss is more ob- with very small sample size (126 instances and 94 instances respectively). This stresses that the proposed RFSVM method, as well as the RFDis method, manage to better handle HDLSS datasets than the state-of-the-art MKL approach.

Let us also recall that, contrary to EasyMKL, the RFSVM method does not require a greedy optimization of the kernel combination, neither requires to choose a priori the different kernel to use in the combination.

Results on Radiomic datasets

In the following, the previous analysis is confirmed on the real-world Radiomic datasets. The results are gathered in is also given in Figure 2.8 (b), and the same conclusion holds: the two proposed methods, RFSVM and RFDis, significantly outperform SVM-RFE with α = 0.05.

Comparison of the RFD-based methods (RFSVM, RFDis, RFMDS):

Here again, the same conclusion goes with the Radiomic datasets: the RFSVM is still the best method, followed by the RFMDS method. The RFMDS method is still slightly better than RFDis on Radiomic datasets that are all 2-class problems, which also confirms the previous conclusion that RFMDS works well for 2-class problems.

Comparison of the RFSVM method and the state-of-the-art MKL method (EasyMKL): Table 2.5 shows in particular that the EasyMKL method has much worse performance on Radiomic problems than on non-Radiomic problems, which seems to confirm that EasyMKL hardly handles very small datasets like those found in the medical field. The proposed RFSVM and RFDis on the other side still work well for both Radiomics or non-Radiomic problems.

Discussion

From the results on both non-Radiomics and Radiomic problems, the following conclusions can be drawn: (i) in general, the multi-view solutions outperform the state-of-the-art Radiomics solution that uses feature selection in the concatenated feature space (early integration), and the differences are always statistically significant for the two proposed RFD-based methods; (ii) by comparing three different possibilities of learning with dissimilarity, learning in kernel space seems to be a better choice for multi-view learning problems, while one can note that the RFMDS method, that uses an embedded space, is less accurate for multi-class problems; (iii) by comparing RFSVM to MKL method, one can see that even though both methods use kernels, the RFSVM method is better than MKL, especially for very small datasets like in the Radiomics application. These results also stress that the RF kernel outperforms the traditional gaussian, linear, and polynomial kernel in the HDLSS context.

Let us finally recall that the RFSVM method has the strong advantage to not require the optimization of the kernel combination, neither to choose the different kernels to use beforehand.

Conclusion

In this chapter, we proposed to use RFD as the intermediate representation for multi-representation fusion method and showed its potential compared to other state-of-the-art methods. Because multi-view datasets share the class information among views, learning a proper dissimilarity measure that can reflect the class becomes necessary. Among various dissimilarity learning methods, RFD was chosen thanks to the following two reasons. Firstly, it can take advantage of both feature and class dissimilarities due to the tree structure.

Secondly, it can deal well with the HDLSS problem, which makes it a better choice compared to other dissimilarity learning methods such as distance metric learning or kernel learning.

A preliminary experiment has been proposed to better understand how the RFD measure behaves according to the most important hyperparameters. We have shown that when there are more trees in the forest, and the trees are deeper, the resulting RFD estimate is more accurate. In the second set of ex- For the two proposed methods, the RFD measure used for each view shares the same parameter settings, while we think it could be further fruitful to make the RFD measure suit to the specificities of each view. Therefore, we propose in the next two chapters some improvements of the proposed methods. Firstly, we will focus on how to better capture the dissimilarity information using RF in the next chapter. For the current RFD measure, each tree in the forest gives binary information (i.e. 1 or 0). However, we believe that values with higher precision would be more appropriate to measure the dissimilarities at the tree level. It would also be important to take into account the tree confidence in the dissimilarity measure. Different improvements will be proposed in the next chapter. Furthermore, the two dissimilarity-based intermediate integration methods treat all the views with the same importance, while a weighted combination could also be used to generate a better joint dissimilarity matrix. These view combination solutions will be presented in chapter 4.

Chapter 3

Towards a better RFD measure Contents

Introduction

From the experimental results in the previous chapter, we showed the potential of the multi-representation fusion approach for HDLSS multi-view learning. This approach is a two-step process: i) project each view in a new intermediate representation space, based on dissimilarities and ii) combine the resulting dissimilarity spaces to form a merged representation from which a new classifier can be learned. However, the multi-view learning literature mainly focuses on the second step only ([6, 98, 128, 133]), considering that the dissimilarity measure is problem-dependent and supplied beforehand. Yet, as explained in the 'related works' section of the previous chapter, the RFD measurement does not require choosing a dissimilarity formulation beforehand, since it is fully learned from the data. As a consequence, we think that this first step deserves to be further studied in this context, and in particular, that the RFD computation should be modified to suit better to our task. This is the purpose of this chapter.

The process of construction, the mathematical properties and the parametrization of RFD measure were discussed in details in the previous chapter. The dissimilarity values between instances are firstly generated from each decision tree and then combined by averaging over all the trees in the Random Forest. However, it is possible to make the RFD measure more accurate. Firstly, estimating the dissimilarity between two instances with a binary value may not be accurate enough and the precision of dissimilarity value depends on the number of trees. We believe that if each tree is able to provide a real value between 0 and 1, the final averaged dissimilarity value would be more accurate and the precision would depend less on the number of trees. Secondly, the dissimilarity value provided by a forest is calculated with a simple average. Yet, it is known for a RF that all the trees are not as reliable as each others for prediction, and one could think that the same phenomenon holds for the dissimilarity estimation: the RFD measure could then benefit from a weighting process to make the trees contribute differently to the final dissimilarity computation.

With the objective of proposing more elaborate dissimilarity measures based on Random Forest, the rest of this chapter is organized as follows: in section 2, the binary dissimilarity output of each decision tree is modified using information from the tree structure. A simple dissimilarity measure namely RFD NC is proposed by taking advantage of terminal node confidence. In section 3, a more elaborated dissimilarity measure namely RFD I H is proposed using instance hardness information beyond the tree structure. Different methods are tested and compared on real world datasets. Finally, in section 4 the conclusion and future work are discussed.

An accurate RFD measure from the tree structure

Random Forest based similarity and dissimilarity measures have been used in several works. For example, RFD is used in [START_REF] Auret | Change point detection in time series data with random forests[END_REF] for the change point detection in time series data and is also used in [START_REF] Gray | Random forest-based similarity measures for multi-modal classification of Alzheimer's disease[END_REF] for the classification of Alzheimer's disease. Several studies also show the use of unsupervised version of Random Forest similarity for data clustering [4,209,210]. However, very few works have been proposed to modify the RFD measure introduced in Chapter 2.

In [START_REF] Englund | A novel approach to estimate proximity in a random forest: An exploratory study[END_REF], the authors state that the binary dissimilarity values provided by each tree is a very simple binary measure, which needs to be more refined. They propose a novel approach to estimate the Random Forest similarity for small sized forest. They try to make the similarity value non-binary for each tree by taking into account the distance between terminal nodes. Let us assume x i ends in node n i and x j ends in node n j . The distance g ijk between n i and n j of the kth tree is the number of tree branches between n i and n j . The similarity between instances in n i and n j of tree k is calculated with:

s (k) ij = 1 e wg ijk (3.1)
where the parameter w controls the influence of the distance between two terminal nodes on the final dissimilarity. When w is very big (over ten for example), the similarity between two instances in different nodes is close to 0. In this case, g ijk has no effect on the dissimilarity measure. However, if w is very small, g ijk has a very strong influence on the final similarity value. One proximity estimate, especially when RF is made of a small number of trees. However, in the previous chapter, we have shown that for HDLSS problem, a bigger number of trees is necessary. Thirdly, the proposed method is controlled by a new hyperparameter, potentially difficult to tune in the HDLSS setting, due to the lack of validation datasets. In the next section, we propose to focus on the information provided by each terminal node to obtain a more accurate RFD measure.

New RFD measure with terminal node confidence

The motivation of using the node confidence is that each tree is a weak learner and not all terminal nodes can provide useful dissimilarity information. To illustrate this, another example is given in Figure 3.3a and 3.3b. The decision tree is built on a synthetic dataset with two classes and two features. All terminal nodes from Figure 3.3a are shown as partitions of the feature space in For example, when a test instance lands in node #2 (the red triangle in node #2 shown in Figure 3.3b), we are quite confident that it belongs to class 1 (blue circles). However, if a test instance lands in node #8 (the red triangle in node #8 shown in Figure 3.3b), it is difficult to tell which class it belongs to because the node #8 is surrounded by many instances from other classes and it is easy to make a mistake.

The main task here is to identify these difficult terminal nodes and to assign them a lower confidence. However, it is hard to evaluate the quality of leaf nodes only according to the tree structure (instances in the node, node depth, etc.) because the structure of the tree can be easily affected by noise or outliers when the decision trees are fully grown ( [START_REF] Breiman | Random forests[END_REF]). One advantage of using Bagging for building a Random Forest, as it is done in the reference method used HDLSS setting makes often impossible to obtain additional instances for that purpose. When a decision tree is fully grown, all its terminal nodes are "pure", that is to say contain instances from only one class. As OOB instances are not seen during the training process, each decision tree may make mistakes on OOB instances. Hence, the weight of the kth tree for any test instance x t can be defined as:

w (k) (x t ) = ∑ x i ∈N (k) (x t ) I(h (k) (x i ) = y i ) ∑ x i ∈N (k) (x t ) 1 (3.2)
where N (k) (x t ) is the neighborhood of test x t on tree k, which includes all the instances (both training and OOB instances) landing in the same terminal node as x t . I() equals to 1 if the expression is true and 0 otherwise. Hence, w (k) represents the posterior probability assigned to the terminal node where the test instance lands in using OOB instances. This weighting procedure is summarized in Algorithm 2.

Algorithm 2 RFD NC : RFD with node confidence Measure the weight of each tree w (k) (x t ) (as in Equation 3.2) 7:

The final dissimilarity:

d(x t , x i ) = 1 - ∑ M k=1 w (k) (x t )×s (k) (x t ,x i ) ∑ M k=1 w (k) (x t )
8: Output: The dissimilarity vector d(x t ) between x t and all training instances.

It can be seen that the node confidence does not depend on the entire tree, but only depends on the terminal node the test instance ends in. This procedure enables a "personalized weighting": each test instance will have different weights over M decision trees because it is very unlikely that two different instances always end in the same terminal node for all M trees, especially when M is big and the feature dimension is very high.

According to the results in the previous chapter, two RFD based methods both work very well with SVM and RF as the classifier. However, when the RFD measure is modified as proposed, the resulting RFD matrix is not p.s.d. anymore. Indeed, let us recall that this property has been proven for the binary dissimilarity values given by a single tree. Since these values are not binary anymore, the p.s.d. property can not be proven anymore. Nevertheless, even if the RFSVM can not be used here because it requires the RFD matrix to be p.s.d., the RFDis methods is still usable, and will be used in the following to test the new RFD measure and compare it to state-of-the-art methods.

Experiments and results

Protocol

In this section, we compare the proposed method RFD NC to two Random Forest based dissimilarity measures: the RFD measure which is introduced in the previous chapter and proved to work very well in HDLSS case and the RFD PB proposed in [START_REF] Englund | A novel approach to estimate proximity in a random forest: An exploratory study[END_REF]. We also compare the proposed dissimilarity learning For all the RF classifiers used in this experiment, the number of trees is set to 512 following the conclusions from the previous chapter, while other parameters are set to the default values proposed in the Scikit-learn machine learning framework ([178]).

With RFDis, all the five chosen dissimilarity measures (summarized in Table 3.1) are firstly used to build the dissimilarity matrix for each view. Then these dissimilarity matrices are averaged to form the joint dissimilarity matrix, which is then used as the input of the Random Forest for the classification task.

Two non-RFD based dissimilarity measures (Euclidean distance and LMNN)

provide unbounded distance values. Hence, they are re-scaled to the [0,1] interval by dividing each dissimilarity vector by its maximum value before the averaging. The results are presented in the following for multi-view datasets only, but single-view results are also reported in Appendix B, which share the same conclusions as the multi-view results. Proposed RF based measure using node confidence

The datasets used in this chapter are the same as in the previous chapter. We recall that a stratified random splitting procedure is repeated 10 times, with 50% of the instances for training, 50% for testing. In order to compare the methods, the mean and standard deviations of accuracy are evaluated over 10 runs.

Results

The average classification rates over the 10 repetitions, along with standard deviations, are shown in Table 3.2. Bold numbers on each row show the best classification results among all five methods. The proposed method RFDis NC with node confidence wins 6 times the first place among 15 datasets. The method RFDis PB proposed in [START_REF] Englund | A novel approach to estimate proximity in a random forest: An exploratory study[END_REF] wins 5 times the first place among 15 datasets.

From the average ranking, it can be seen that the proposed method RFDis NC and RFDis PB are the best among all the methods. Among all three Random Forest based dissimilarity measures, the original RFD is the worst ranked; while among all five methods, there is no surprise that EUDis is ranked as the worst solution because it is the only learning free dissimilarity measure.

Many studies have shown that the Euclidean distance measure can suffer from the curse of dimensionality, especially when the feature dimension is bigger than the sample size ( [START_REF] Angiulli | On the Behavior of Intrinsically High-Dimensional Spaces: Distances, Direct and Reverse Nearest Neighbors, and Hubness[END_REF][START_REF] Feldbauer | A comprehensive empirical comparison of hubness reduction in high-dimensional spaces[END_REF]). For the highest feature dimension datasets such as BBC (13628 features), the performance gap between learning based and learning free dissimilarity measures is thus important.

The result of Nemenyi post-hoc test is shown in Figure 3.4: there are no significant difference among the three RF based methods, but RFDis NC and RFDis PB are significantly better than EUDis and LMNNDis.

The performance gap between proposed RFDis NC and RFDis is quite small.

It may be due to the following limitations of the proposed RFDis NC method:

• When sample size is small, the OOB sample size is of course also small, which may result in the case that some terminal nodes have no OOB instances. In this case, the node posterior probability is 1 but it does not mean that the corresponding sub-region is relevant for learning the dissimilarity.

needs to be used instead. To give a more refined weight, the concept of Instance Hardness (IH) is introduced in 3.3.1, which can be used to measure the confidence at instance level. Then in 3.3.2, a new dissimilarity measure based on IH, namely RFD I H , is proposed to overcome the limitations of RFD NC .

The comparison with all the methods tested in the previous experiment and a further result analysis are given in 3.3.3 and 3.3.4 respectively.

Instance hardness

In [216], the authors provide an empirical definition of instance hardness.

Generally speaking, the IH measure helps to identify which instances are likely to be misclassified and why they are hard to classify ([156, 187]). High values of instance hardness usually indicate difficult instances such as outliers or instances close to a decision boundary. The advantage of IH measure is that it reveals the difficulty of a problem at an instance level rather than at the dataset level ([127, 155]).

A summary of different hardness measures is given in Table 3.5. The "+" and "-" symbols indicates whether the corresponding measure is positively or negatively correlated to IH ([216]). All hardness measures are designed to understand why an instance is hard (a positive correlation with instance hardness)

or easy (a negative correlation with instance hardness) to classify. For example, Class Likelihood (CL) measures the probability of an instance belonging to a certain class. For "harder" instances, their CL values are lower.

Among the different IH measures listed in Table 3.5, the most used is the k-Disagreeing Neighbors (kDN) measure. The kDN value is simply the percentage of instances from other classes in the neighborhood of a given instance:

kDN(x i ) = |x k : x k ∈ KNN(x i ) ∩ y k = y i | K (3.3)
where KNN(x i ) is the K nearest neighbors of x i . The distance measure chosen here is usually the Euclidean distance. kDN is straightforward and easy to understand. If the nearest neighbors of an instance are all from the same class as the given instance, the kDN value is 0 and it is easy to classify. On the other hand, if the nearest neighbors of an instance are all from other classes, the subspaces for leaf nodes. This example shows that different leaf nodes have different decision paths, which normally leads to different selected feature subsets. This feature subset defines the feature subspace for all the instances landing in the corresponding leaf node. From the example of Iris dataset (3.6),

we have shown the importance of the neighborhood in different subspaces.

Hence, the kDN measure can be used to measure the instance hardness in this low dimensional subspace to solve the problems presented in the previous paragraph. Compared to the node confidence weighting scheme proposed in the previous section, the IH provides more refined weights on instance level instead of node level.

The IH weighting scheme is summarized in Algorithm 3. for each tree h k in H do

4: If x t and x i land in different leaf nodes, s (k) (x t , x i ) = 0 5: If x t and x i land in the same leaf node, s (k) (x t , x i ) = 1 6:
Measure the instance hardness of x i in the corresponding subspace I H (k) (x i ) based on kDN defined in Equation 3.3.

7:

The final dissimilarity:

d(x t , x i ) = 1 - ∑ M k=1 (1-IH (k) (x i ))×s (k) (x t ,x i ) ∑ M k=1 (1-IH (k) (x i ))
8: Output: The dissimilarity vector d(x t ) between x t and all training instances.

Experiments and results

The experimental protocol in this section is the same as in the previous section. We add the results of the proposed RFD I H to Table 3.2. The average classification rates with standard deviations are shown in Table 3.3. The average ranking of all the methods shows that the proposed RFDis I H is the best among all methods, followed by RFDis PB and RFDis NC . Among all four Random Forest based dissimilarity measures, the original RFD is ranked the worst. More detailed comparison is given in the following paragraphs.

Comparison to RFDis

As RFDis is our baseline from the previous chapter, to better assess the progression between the proposed methods and the baseline, a pairwise analysis based on the Sign test is computed on the number of wins, ties and losses between RFDis and all the other methods. The result is shown in Figure 3.8, which demonstrates that RFDis is a strong baseline as most methods are not able to be significantly better. With α = 0.10, the proposed two methods RFDis NC and RFDis I H are significantly better than RFDis. With α = 0.05 or α = 0.01, only RFDis I H is significantly better than RFDis. Compared to RFDis NC , RFDis I H has better performance, which confirms our hypothesis that more refined weighting can lead to better result.

The Nemenyi post-hoc test with Critical Differences (CD) is also done to have an overall statistical comparison. The result is shown in Figure 3.9: only node #7 is associated with class 0. As all the trees are fully grown, each leaf node is pure (all the bootstrap instances are correctly classified), which shows that x i is a misclassified OOB instance. It results in wrong information when we try to measure the dissimilarity between other instances and x i . From its neighborhood, it can be told that all the 7 nearest neighbors in the projected subspace (pointed with arrow lines) belong to class 0, which makes the IH value of x i equal to 1. The test instance x t is from class 0. With the classic RFD measure, their similarity on this tree is 1 because they land in the same leaf node. However, with RFD I H , their similarity is 0 because x i is considered as a very hard instance with IH = 1. From this example, it confirms our hypothesis that RFD I H can help to identify if a tree provides a reliable dissimilarity measure for a given instance.

Conclusion

RFD based multi-representation fusion methods have been shown to be a successful solution for HDLSS multi-view problem in the previous chapter. In this chapter, with the aim to address the first possibility to improve the classification performance, the limitations of classic RFD have been analyzed. We argued that the original binary dissimilarity value provided by each decision tree can be refined with more accurate values. Based on the idea that not all trees can provide reliable dissimilarity information for a given instance, two different weighting schemes have been proposed: the first one is to weight each leaf node of the decision tree using the node confidence because the leaf node contains all the information of the dissimilarity induced from the decision tree; the second one provides a more refined weighting scheme based on instance hardness to address the disadvantages of the first proposal.

The experiments on 15 multi-view datasets have been realized to compare the proposed two measures to other dissimilarity measures including learning free measure and learning based measures. The results have shown that: 1.

compared to learning free measure, learning based measures have better performance for multi-representation fusion on HDLSS multi-view problem; 2.

Random Forest based dissimilarity measures are better than the metric learning method LMNN; 3. among all Random Forest based measures, the proposed method RFD I H is significantly better than the classic RFD measure. The two weighting methods for RFD lead to more accurate dissimilarity measures without the limitations of being metric or p.s.d. However, this also limits the use of the proposed measures as they are not symmetric and not p.s.d, which means that they can not be used as a kernel for SVM classifier. But they can always be used as new feature vectors to train the classifiers such as Random Forest.

In this chapter, the kDN has been chosen as the instance hardness measure.

The parameter k was fixed for all the datasets according to the suggestion from other studies. The effect of k can be studied in short term to see its influence on the resulting dissimilarity measure on different datasets, especially those datasets with very small sample size. Usually, larger k can provide more refined weights and differentiate better the confidences of different trees. Hence, an adaptive k for different datasets may help to improve the classification performance instead of using a fixed number. According to [159], kDN does not take class imbalance into account and may not be a good choice for imbalanced data. The authors propose a Bayes Imbalance Impact Index, which can be studied and tested for the imbalanced datasets.

In the long term, the instance hardness for test instances can be proposed. One However, how to adapt the measure for HDLSS problem remains a challenge.

Introduction

The second chapter of this manuscript showed the great potential of RFD based multi-representation fusion methods for HDLSS multi-view learning. It also introduced two areas for improvement: (i) proposing a finer measure of dissimilarity that better suits to our problem and (ii) proposing a mechanism for merging representations that better takes into account the complementarities of views. As the first goal has been explored in the previous chapter, it is the second goal that we are now focusing on in this chapter.

The RFD based multi-view methods proposed so far are merging the viewspecific representations in a quite naive way: by simply averaging them. Let us recall that these view-specific representations are built by describing each instance through its dissimilarities with each of the training instances. For example, any instance x is represented by Q vectors:

{d (q) (x, x 1 ), d (q) (x, x 2 ), . . . , d (q) (x, x N )}, ∀q = 1, 2, . . . Q (4.1)
where d (q) (x, x i ) is the dissimilarity between the instance x and the training instance x i , computed from the qth view. These values are fully comparable from a view to another and as a consequence, can be straightforwardly averaged over the views. The merged representation is given by :

{d m (x, x 1 ), d m (x, x 2 ), . . . , d m (x, x N )} (4.2)
where

d m (x, x i ) = 1 Q Q ∑ q=1 d (q) (x, x i ) (4.3)
The averaged dissimilarity is a simple, yet meaningful way to merge the representations built from all the views. However, it intrinsically considers that all views are equally reliable with regard to the task, and that the resulting dissimilarities are as important as each other. This is likely to be wrong, from our point of view. In multi-view learning problems, the different views are meant to be complementary in some ways, that is to say to vehicle different types of information regarding the classification task. These different types of information may not have the same contribution to the final predictions. That is the reason why it may be important to differentiate these contributions, for example by weighting them according to some criterion taking into account their reliability.

Generally speaking, there are two ways of weighting dissimilarity matrices: static weighting and dynamic weighting. Static weighting methods assign a fixed weight to each view with the assumption that the importance of each view is the same for all test instances, while dynamic weighting methods assign "personalized" view weights depending on the instance to predict. In section 4.2, different static weighting methods are reviewed and a weighting method based on the OOB accuracy is proposed. In section 4.3, several dynamic weighting methods are reviewed. A dynamic view selection method is proposed in section 4.4 to select the best combination of views for each instance to predict.

Static view combination

Static weight calculation

Given a set of dissimilarity matrices {D (1) , D (2) , . . . , D (Q) } built from Q different views, our goal here is to find the best non-negative weight set {w (1) , w (2) , . . . , w (Q) }, so that the joint dissimilarity matrix is:

D = Q ∑ q=1 w (q) D (q) (4.4)
where w (q) ≥ 0 and

∑ Q q=1 w (q) = 1.
In the literature, there exists many approaches that allow to find such an optimal or near-optimal set of weights, to combine dissimilarity matrices. The most natural one is to deduce the weights from a quality score measured on each view. For example in [150] this principle is used for multi-scale image classification where each view is a version of the image at a given scale. For this application, the weights are derived directly from the scale factor associated with the view. Obviously, this only makes sense with regard to the application, for which the scale factor gives an indication of the reliability for each view.

Another, more generic and classification-specific approach, is to evaluate the quality of the dissimilarity matrix using the performance of a classifier. This makes it possible to estimate whether a dissimilarity matrix sufficiently reflects class membership ( [START_REF] Duin | The dissimilarity space: Bridging structural and statistical Pattern Recognition[END_REF]150]). For example, in [150], the authors propose to train a linear SVM from each view and to use the cross-validation performance for the estimation of the weight of each view:

α (q) = max(acc (q) -rnd (q) , 0) (4.5)
where acc (q) is the cross validation accuracy on view q and rnd (q) is the random classification accuracy. In this case, if the cross validation accuracy is somehow random, the weight assigned to the corresponding view is zero and the view is not taken into account in the combination.

KNN is also very often used to evaluate the quality of a dissimilarity matrix due to its easy interpretation: a good dissimilarity measure should propose good neighborhoods (i.e, the most similar instances should belong to the same class). For example, a 1 Nearest Neighbor (1NN) classifier is used to assess the performance of dissimilarity matrices in [START_REF] Duin | The dissimilarity space: Bridging structural and statistical Pattern Recognition[END_REF]. In the field of metric learning, 3NN is widely used to evaluate the quality of learned metrics ( [130, 148, 190, 243]).

In the field of kernel combination, a different heuristic rule is used. In [START_REF] Cristianini | On kerneltarget alignment[END_REF], the notion of Kernel Alignment (KA) is defined to measure the similarity between two kernels with the following formulation:

A(K 1 , K 2 ) = K 1 , K 2 F K 1 , K 1 F K 2 , K 2 F (4.6)
where •, • F is the Frobenius norm which is given by:

K 1 , K 2 F = N ∑ i=1 N ∑ j=1 K 1 (x i , x j )K 2 (x i , x j ) (4.7)
Transposed to vectors instead of matrix, this KA measure can be interpreted as the angle between two vectors, which reflects their correlation. To compute the KA, an ideal target matrix must be defined beforehand. This ideal target matrix is an optimal theoretical similarity matrix, in regards to the task. For example, for binary classification, the ideal target matrix is usually defined as K * = yy T , where y = {y 1 , y 2 , . . . , y N } is the label vector and y i ∈ {-1, +1}. Thus, each value in K * is:

K * ij =    1, if y i = y j -1, otherwise (4.8)
In other words, instances are considered similar (K * ij = 1) if and only if they belong to the same class. In [233], the authors proposed an adaptation of this matrix to multi-class classification:

K * ij =    1, if y i = y j -1 C-1 , otherwise (4.9)
where C is the number of classes. To use the KA method in our experiments, the dissimilarity are transformed into kernel as explained in chapter 2 (section 2.3.3).

In [191], the authors propose an heuristic rule with KA to weight kernel matrices for binary classification problem. The weight assigned to the kernel matrix

K (i)
of the ith view is given by:

w (q) = A(K (q) , yy T ) ∑ Q h=1 A(K (h) , yy T ) (4.10)
In [1,[START_REF] Cristianini | On kerneltarget alignment[END_REF]191], the authors proved that if a kernel matrix is p.s.d, the corresponding alignment to the target kernel is non-negative, and the corresponding w (q) is also non-negative. However, as our similarity matrix generated by RFD I H is not p.s.d. anymore, we cannot make sure that w (q) in Equation 4.10 is non-negative. To avoid negative weights, we propose to use the softmax function to normalize the weights. The corresponding weights for each view are measured as in Equation 4.11.

w (q) = exp(A(K (q) , K * )) ∑ Q h=1 exp(A(K (h) , K * )) (4.11)
Both classification accuracy (Equation 4.5) and KA (Equation 4.11 and 4.9) could be used in our framework to estimate the quality of the dissimilarity matrix D (q) . The disadvantage of using the classification accuracy is that a validation dataset is required for many classifiers to guarantee a good performance, which is less adapted to the HDLSS problem. In the next section, we propose another way of measuring the quality of each dissimilarity matrix without using a validation dataset for static weighting.

Weighting with OOB accuracy

All the static weighting methods calculate the importance of each view from the dissimilarity matrices only, which may not generalize well on the test instances if there is no validation dataset. To overcome this problem, we propose to use OOB estimation in this section.

In the previous chapter, OOB instances were used to evaluate the confidence of each terminal node. Here, the OOB instances are used to estimate the performance of each Random Forest. For each OOB instance, the trees that have not used this instance as training data are chosen to form a sub-forest to predict its label. When the labels of all OOB instances are predicted, they are compared to the ground truth labels to give a classification accuracy. This estimate is known to be a reliable estimate of the generalization performance [START_REF] Breiman | Out-of-bag estimation[END_REF].

We propose to use the Out-Of-Bag accuracy of the Random Forest classifier from each view as the weight of the corresponding dissimilarity matrix. The intuition behind this choice is that if the trained Random Forest classifier has good performance, the dissimilarity measure generated from this Random Forest should be good too. The main advantages of using OOB accuracy to weight each view in the combination is that, in terms of computational cost, OOB measure adds no extra cost as it can be obtained during the training process. OOB measure does not need extra validation neither, which better suits the HDLSS problem.

Experiments and Results

Experimental protocol

In the previous chapter, we showed that RFDis I H achieved the best results

with simple averaging over all the dissimilarity matrices, and therefore, it is used as the baseline method in this experiment. The proposed method based on OOB weighting (SW OOB ) is compared to two additional methods from the literature: 1. Using kNN classifiers to estimate the quality of each dissimilarity matrix and 2. Using Kernel Alignment. For the first method, a 3NN classifier has been chosen as it is the most popular method in the metric learning literature ([130, 148, 190, 243]). The training accuracy rate of these classifiers is used as weight directly. For the second method, dissimilarity matrices are transformed into similarity matrices as previously explained, and the Equation 4.11 is used to compute the weights; this method is called Dissimilarity Alignment (DA) in the following. A summary of all these Static Weighting (SW) methods are shown in Table 4.1. The experimental protocol and datasets are the same as the previous chapter. We recall that for the RF classifiers used in this experiment, the number of trees is set to 512 according to the recommendation of previous chapters, while the other parameters are set to the default values as proposed in the Scikit-learn machine learning framework ([178]).

Results

The average classification rates over the 10 repetitions, along with standard deviations, are shown in Table 4.2. The bold numbers are the best results

for each dataset. The proposed approach SW OOB is the best for 9 of the 15 datasets, while SW DA wins 5 times and SW 3NN wins 2 times over 15 datasets. For the nus3 dataset, no static weighting method is able to improve the classification performance over the baseline method Avg. The average ranking results show that SW OOB has the best results, followed by SW DA and SW 3NN . All three static weighting methods are better than the simple averaging. Even though the difference of average ranking between the best method (SW OOB ) and the worst method (Avg) is obvious, the accuracy differences on each dataset are quite small, rarely above 1%. The difference is even below 0.1% for 7 of the datasets. To see more clearly if any static weighting method is significantly better than the baseline, the pairwise comparison result is shown in Figure 4.1. None of these three static weighting methods is significantly better than Avg. In a nutshell, these results show that the best performing method SW OOB is not able to have the significant performance improvement. One possible reason is that the views in these datasets are all equally informative over all the instances, which may lead to very similar OOB weights in the combination.

And as a consequence, the corresponding combination is very similar to our baseline method, with a simple average. To verify this, the differences between SW OOB and Avg are analyzed in three different ways (Table 4.3). Firstly, the Weight di f f values are reported, as the average difference of weight values between SW OOB and Avg; Secondly, Dissimilarity di f f , as the average difference of dissimilarity values between the joint dissimilarity matrices generated from SW OOB and Avg; and Thirdly, Prediction di f f , as the average percentage of test instances that have different prediction results between SW OOB and Avg. The results in Table 4.3 show that all three indicators are very low in a They all contribute to the good predictions, at least for some of the instances in the problems. However, since the weights provided by the static weighting methods are very close to the uniform weights of the simple average combination, this does not allow us say whether or not some instances could be better predicted if we relied on some views rather than others. This only makes it possible to say that no single view is undoubtedly better or worse than all the others. In the next section, a dynamic view combination method is proposed to enforce larger weight differences in each weighted combination, by using different matrices combinations for different test instances.

Dynamic view combination

In opposition to static weighting, dynamic weighting aims at assigning different weights to each view for each instance to predict ([164, 189]). The intuition behind dynamic weighting is that, due to the heterogeneity among instances, different instances may rely on different information sources. In Radiomics problem for example, for a patient A, there may be more useful information in one view (e.g. texture or shape features) with regards to the classification task while for a patient B, there may be more useful information in another view (e.g. intensity or wavelet features).

Nevertheless, dynamic weighting is particularly complex in our framework.

Let us recall that the multi-representation fusion approach is made up with two stages: 1. inferring the dissimilarity matrices from each view, and 2. combining these dissimilarity matrices to form a new training set. The weights we want to compute with dynamic weighting are the weights used to compute the final joint dissimilarity matrix in stage 2. As a consequence, if these weights change for each test instance, the joint dissimilarity matrix needs to be entirely re-calculated and a new classifier needs also to be re-trained afterwards. This means that, for every new instance to be predicted, a whole training procedure has to be performed. This is computationally expensive and quite inefficient from our point of view.

To avoid this problem, the dynamic view weighting can be replaced by a dynamic view selection procedure. Dynamic Selection (DS) is one of the most successful approaches in Multiple Classifier Systems (MCS) ( [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]). DS aims at selecting the most competent classifier(s) for each test instance. When a single classifier is selected, it is called Dynamic Classifier Selection (DCS). When multiple classifiers are selected and combined, it is called Dynamic Ensemble Selection (DES). In this work, we are more interested in DCS to select the best view combination. There are essentially two steps for DCS ( [START_REF] Britto | Dynamic selection of classifiers-a comprehensive review[END_REF]): the generation of a pool of classifiers and the selection of the most adequate classifier, as shown in Figure 4.2. In the following sections, we propose a procedure to generate a pool of classifiers with different combinations of views and a selection criterion for dynamic view selection.

such a task, traditional grid search or random search strategies can be used to generate the candidate weight tuples. However, the number of such tuples increases exponentially with respect to the number of views. For example, suppose we use a grid search strategy with weights discretized with 10 different values in [0, 1]. For Q views, it will result in 10 Q different weights tuples. Six views would imply to generate 1 million weight tuples and thus 1 million classifiers to train from each of them. This is too computationally expensive in our point of view.

The alternative approach we propose is to select a subset of views for every candidate in the pool, instead of considering a weighted combination of all of them. By doing so, for each instance to predict, we intend to use only the views that are considered informative enough. The selected views are then combined by averaging, since it has been shown in the previous section that the simple average is a strong baseline. For example, if a problem has three views, there are 2 3 -1 = 7 different situations by only considering the presence or absence of views. The weights assigned by simple averaging could thus take their values in the following ensembles: {[ 1 3 , 1 3 , 1 3 ], [0, 1 2 , 1 2 ], [ 1 2 , 0, 1 2 ], [ 1 2 , 1 2 , 0], [1,0,0], [0, 1, 0], [0, 0, 1]} (the situation where all views are not selected is obviously ignored). Based on these weights, a pool of 7 classifiers C = {C 1 , C 2 , . . . , C 7 } is formed. From this example, it can be seen that compared to a grid search, dynamic view selection has the advantage of using a pool of much smaller size, which reduces significantly the computational cost.

Another advantage of dynamic view selection is that it can deal well with missing views. In this chapter, for a fair comparison with the static weighting methods, the Random Forest classifier is chosen to form the pool.

Once the pool of classifiers is generated, the best classifier needs to be selected for each instance to predict. In the next section, a new classifier selection method for HDLSS problems is proposed.

Selection criteria

The selection of the most competent classifier is a key aspect of DS. Generally speaking, the selection procedure can be divided into two steps: 1. define the region of competence, 2. compute a selection criterion to select the best classifier.

The region of competence Θ t of each test instance x t is the region used to estimate the competence of each classifier for that instance. There exist four categories of methods for defining the region of competence: Clustering, kNN, Decision Space, Potential Functions ( [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]). The Clustering and kNN methods are the most widely used in the literature. For Clustering ([138, 217]), the principle is usually to define the region of competence of an instance x as the closest cluster, according to the distances of x to the centroids of all the clusters. As the clusters are fixed, many different instances might share the same region of competence. In contrast, kNN methods give different regions of competence from one instance to another, since they are defined by their neighborhood. This allows for more flexibility but at the expense of a higher computational cost ( [START_REF] De Souto | Empirical comparison of dynamic classifier selection methods based on diversity and accuracy for building ensembles[END_REF]).

The most important part of the selection process is to define the criterion to measure the competence level of each classifier in the pool. There are a lot of methods according to the type of information used to measure the competence, including ranking ([200, 246]), accuracy ([69, 246]), probabilistic ([95, 139, 245]), behavior ([49, 96]), oracle ([134]), data complexity ( [START_REF] Brun | Contribution of data complexity features on dynamic classifier selection in[END_REF]) and Metalearning ([75,183]). We do not give an exhaustive survey of these methods but detail in the following, one of the most representative work, namely Local Classifier Accuracy (LCA,[246]). LCA is a widely used method in the field of dynamic selection with good and robust classification results ([71]).

LCA uses the local accuracy of classifier C i with respect to the predicted label ŷt for a given test instance x t :

w i,t = ∑ x k ∈Θ t, ŷt I(C i (x k ) = ŷt ) ∑ x k ∈Θ t I(y k = ŷt ) (4.12)
where Θ t = {x 1 , . . . x k , . . . , x K } is the competence region of the given test instance. These K instances usually come from a validation dataset instead of the training dataset. Θ t, ŷt gathers the instances in Θ t with label ŷt . I() equals to 1 if the expression is true and 0 otherwise. Equation 4.12 simply represents the percentage of correct classifications within the region of competence, but considering only those instances where the classifier has given the same class as the one it gives for x t . Many DS methods share the similar idea as LCA, they mainly differ in how to calculate the competence of the classifier in the region of competence.

However, traditional dynamic selection methods demand a validation dataset ( [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]), which is not very suitable for HDLSS data. In the following part, a selection criterion that does not rely on validation data is proposed. The proposed dynamic view selection method using local confidence is summarized in Algorithm 4.

Experiments and Results

Experimental protocol

For the experimental validation of DS local , the neighborhood size K is set to 7 according to the suggestion in [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]. The base classifier for each view combination is the Random Forest classifier. The other parameters are the same as in the previous section. Table 4.4 gathers the results of DS local , as the average classification rates and standard deviations, and all the results from Table 4.2, for comparison purposes. All the four methods reported in this table have been run on the same datasets, with the same replicates.

Results

Algorithm 4 Dynamic view selection (DS local )

1: Input: Q N × N training dissimilarity matrices {D (1) , D (2) , . . . ,

D (Q) } 2: Q n × N test dissimilarity matrices {D (1)
T , D

T , . . . , D T s : the set of l test instances The pairwise analysis based on the Sign test is used here again to better assess the difference between the baseline method and the proposed dynamic view selection method. The result is shown in Figure 4.3: with α = 0.05, the dynamic view selection method DS local is significantly better than the simple average combination.

From our point of view, these results underlines two essential results: First of all, the simple average combination, when using dissimilarity representation for multi-view learning, is a quite strong baseline. This is consistent with conclusions one can find in the Multiple Kernel Learning literature, as it has already been discussed in the first chapter of this manuscript. Secondly however, the dynamic selection procedure proposed in this chapter allows predominantly to improve the accuracy over this baseline. It shows that all the views do not participate in the same extent to the good prediction of every instance. Some instances are better recognized when the dissimilarities are computed by relying on some views more than the others. These views are certainly not the same from one instance to another, and some instances may need the dissimilarity informations from all the views. Nevertheless, this highlights that the confusion between the classes is not always consistent from one view to another, and we think that this may be amplified for HDLSS problems. In that sense, the views complement each others, and this can be efficiently exploited for multi-view learning provided that we can identify the views that are the most reliable for every instance, one by one.

Conclusion

In this chapter, we proposed several adaptive ways to combine dissimilarity matrices instead of using the simple average combination as in the previous chapters. As dissimilarity matrices built from different views are comparable, it is more meaningful to combine them in a linear way with their weights representing the importance of each view. Hence different linear combination methods including static weighting and dynamic weighting have been investigated.

Two hypotheses has been tested in this chapter. The first one is that all the views are not globally as informative as each other. A static weighting method SW OOB has been proposed to reduce the influence of the less informative views and to increase the influence of the more informative views. The second hypothesis is that the views do not all contribute in the same way to the good prediction of all instances. With this hypothesis, a dynamic view selection method DS local has been proposed to remove views that are irrelevant for a particular instance to classify.

For the first hypothesis, the experimental results show that, even though the proposed static weighting method is better than the baseline methods on most of the datasets, the performance difference is not significant enough. All the views contribute in the similar way to the problem, at least for the problems that we considered in our experiments. It is rare that one view is indisputably better or worse than all the others and particularly on data extracted from real-world applications like those we tested. As for the second hypothesis, in contrast to static weighting methods, the dynamic weighting method based on local accuracy has more potential (DS local got better performance than SW OOB on 10 out of 15 datasets). The results presented in this chapter show that a significant part of the instances can be better recognized if we only consider a well selected subset of views.

However, in its current form, the dynamic selection method proposed in this chapter strongly depends on the number of candidate classifiers in the pool. This pool is limited by the number of views available in the datasets. However, as explained in this chapter, the diversity in the pool is a key property for the DCS approach to be relevant. It allows in particular to propose many different candidate solutions from which the best one is selected. To allow for more versatility, we think it could be interesting to decompose each view into several sub-views. It could be done for example, by using Bagging and Random Subspaces principles before computing the view-specific dissimilarities.

In such a way, the dynamic combination could only select some specific part of each view, instead of considering the views as a whole.

Multi-view data are now very common in real world applications. Whether they arise from multiple sources or from multiple feature extractors, the multiple views are supposed to provide a more accurate and complete description of objects than a single description would do. The starting point of this thesis work is one of such real-world applications: Radiomics. This medical imaging classification problem is a typical multi-view problem because the purpose of Radiomics is precisely to "describe" the patients through several image modalities and several types of features extracted from these images.

In addition to the multi-view aspect, another machine learning challenge is underlying the Radiomics problems: High feature Dimension, Low Sample Size (HDLSS). As a medical related Pattern Recognition problem, the amount of data available for learning to solve Radiomics problems is always very limited, and at the same time, the features extracted from the medical images are always very numerous.

The goal of the thesis was to provide solutions to HDLSS multi-view learning problems, such as Radiomics problems but not exclusively. The main challenge is that most of the state-of-the-art multi-view learning methods are not always suitable for HDLSS problems. Therefore, our proposal in this work was to address HDLSS multi-view problems using methods based on dissimilarities. Dissimilarity strategies allow to overcome the well-known issues that arise from HDLSS problems, and to give an efficient way to handle the heterogeneity of the multiple views at the same time. Three major contributions have been proposed in this direction, and are summarized below.

Conclusion 123

The second contribution, described in Chapter 3, focuses on proposing a new dissimilarity measure, still based on Random Forest, that better deals with HDLSS issues. As our initial RFD measure was based on the Decision Tree dissimilarity estimates that take their values in {0, 1}, two approaches are proposed to make the RFD value more accurate : the first one (RFD NC ) uses leaf node confidence estimates to weight the dissimilarity between instances that land in this node; the second one (RFD I H ) uses an Instance Hardness estimator, applied on the training instances in the same node to refine their dissimilarity values. For both HDLSS single-view and HDLSS multi-view situations, the proposed methods achieve better performance than the initial RFD measure.

The third and final contribution, presented in Chapter 4, concerns the design of a dynamic view selection method that provides a better way of merging the per-view dissimilarity representations. In the experiments of Chapter 2 and 3, the multi-view dissimilarity matrices were merged with a simple average. Even though the flat average is considered as a strong baseline in the literature, it does not take the view importance into account. The rationale behind using weighted combinations of views is to base the decision on the relevant views primarily, and to ignore as much as possible the irrelevant views. We have shown that the dynamic selection of views we propose achieves significant improvement against the simple average.

In a nutshell, the RFD based intermediate integration method enables to face both the HDLSS and the multiview issues. However, there are still open issues and room for improvement. We detail the most important and promising ones from our point of view in the following.

Even though the high dimensional challenge is overcame by the dissimilarity space, the low sample size challenge may be still an issue for the proposed framework. Indeed the mechanism we propose in this work strongly rely on the OOB measure. However, this measure is very sensitive to the size of the initial training set : the measure is based on about one third of the training instances as explained in Chapter 3 and if there are not enough instances (which is the case when dealing with LSS data), this measure may not be reliable enough. This is even more critical for imbalanced LSS datasets for which there is still a strong risk with our proposed framework to ignore under-represented classes in the dissimilarity space: only few dimensions of this space would be related to the minority class. Therefore we believe it deserves to be further explored. One possible direction to investigate would be the use of ad-hoc ensemble techniques. For example data augmentation methods such as the Synthetic Minority Over-Sampling (SMOTE) technique ( [START_REF] Chawla | SMOTE: synthetic minority over-sampling technique[END_REF]) could be tested.

SMOTE was originally proposed to create artificial instances for the minority class of imbalanced datasets. It can also be used to create artificial instances for all the classes to overcome the LSS issue in general. Note however that this would result in higher dissimilarity space dimension since it corresponds to the number of instances in the training set. One possible solution to face this issue would be to select a subset of reference instances (or prototypes) to which dissimilarities are measured, as it is commonly done in the dissimilarity space approach ( [START_REF] Costa | The dissimilarity approach: a review[END_REF]).

As we pointed out in Chapter 4, our framework seems to be all the more successful when there is heterogeneity and disagreement between views, similar to the diversity concept in ensemble learning. The diversity in the pool of classifiers is a key property for the dynamic selection approach to be relevant, as it allows in particular to propose many different candidate solutions from which the best one is selected. Popular ensemble learning methods could be transposed in our framework to enhance this diversity. For example, before computing the view-specific dissimilarities, we could generate random subviews by using the random subspaces mechanism along with Bagging, instead of considering each view as a whole. Doing so, the final predictions could be given by selecting the more reliable parts of each view and by ignoring the irrelevant ones, using the same dynamic selection mechanism as in our framework.

Finally, as explained in Chapter 1, the approach that has been adopted for this work is the intermediate integration method, the most relevant approach in the literature for dealing with HDLSS multi-view problem. However, in preliminary works on Radiomics datasets we report in Appendix B, we have also shown that there is some potential to use the late integration method. The method proposed in this preliminary work was to first generate dissimilarity space for each view, then learn one classifier per dissimilarity space and combine these classifiers at last. Compared to the multi-view late integration methods proposed in the literature, this can be seen as a simple and naive procedure since the classifiers do not cooperate together. Indeed, the state-of-the-art methods in multiview learning are co-training based, that is to say classifiers are trained jointly to maximize their agreement. This could be adapted to our framework by "co-generating" the dissimilarity spaces. The main obstacle as explained in Chapter 1 would be the lack of additional instances that could be circumvent by the OOB mechanism.

A.1 Introduction

The detection and treatment of cancer are still very challenging. The normal process of cancer detection is from certain signs and symptoms to the further investigation by medical imaging and at last confirmed by biopsy ( [3,[START_REF] Coroller | CT-based radiomic signature predicts distant metastasis in lung adenocarcinoma[END_REF]). The diagnosis of breast cancer usually uses the biopsy tissue. The pathologists can histologically assess the microscopic structure and elements of the tissue from breast tissue biopsies ( [START_REF] Araújo | Classification of breast cancer histology images using Convolutional Neural Networks[END_REF]).

One of the most important method for tumor histological examination in pathology is Hematoxylin and eosin (H&E) staining ( [START_REF] Chan | The Wonderful Colors of the Hematoxylin-Eosin Stain in Diagnostic Surgical Pathology[END_REF]). However, manual analysis is experience based, qualitative and always causes intra-or interobservers variation even for experienced pathologists ([165]). Hence developing a more efficient, accurate, quantitative and automated system is necessary and urgent. Due to the high performance of deep learning networks, more and more studies used deep learning for the classification of breast cancer images ([221]). However, the number of images available has always been an obstacle for the use of deep learning. Many studies divide images into patches for data augmentation, but the new problem is that there are no label information for patches.

In this paper, transfer learning without fine-tuning is proposed to solve the above problems. Six different feature extractors are compared, including five deep learning architectures and a traditional feature extractor combining PF-TAS (Parameter-Free Threshold Adjacency Statistics) and GLCM (Gray Level Co-Occurrence Matrices) features. When all features are combined, there are mainly three challenges from the machine learning point of view: (i) small sample size: size: like most other medical applications, the number of breast cancer histology images is very small (400 images); (ii) high dimensional feature space: as six groups of features may be combined, the size of the feature space may be up to 31855, which is over 80 times bigger than the sample size; (iii) multiple feature groups: it may be hard to improve the learning performance by exploiting the complementary information that different groups contain ([44]). To deal with these three challenges, we propose to treat breast cancer histology image classification as a multi-view learning problem.

A multi-view RFSVM method proposed in our previous work ([44]) is then used as a solution.

The remainder of this paper is organized as follows: the six feature extractors are detailed in Section II; in Section III, the dissimilarity based multi-view learning solution is introduced; we describe in Section IV the data sets chosen in this study and provide the protocol of our experimental method; we analyze in Section V the results of our experiments; the final conclusion and future works are drawn in Section VI.

A.2 Feature extractors

In total six different feature extractors are used in this work: handcrafted features, ResNet-18, ResNeXt, NASNet-A, ResNet-152 and VGG16. In this section, a brief introduction of each feature extractor is given. The handcrafted features include PFTAS and GLCM and have been chosen due to their good performance on breast cancer histology image classification ([220]). The five deep learning networks have been chosen for their performance and because they are built on different structures with different depths, and the pre-trained models are available online 12 .

Handcrafted features:

Two kinds of feature extractors are combined together to form the handcrafted feature group: PFTAS and GLCM. TAS (Threshold Adjacency Statistics) is a simple and fast morphological measure for cell phenotype image classification presented by Hamilton et al. in [110].

ResNeXt:

ResNeXt is one of the state-of-the-art techniques for object recognition. It builds upon the concepts of repeating layers while exploiting the split transform merge strategy to bring about a new and improved architecture ([250]).

The input space of ResNeXt is a {3, 224, 224} RGB image and we use the network pretrained on ImageNet with 1000 classes 5 . 2048 features are extracted from the average pool layer (i.e. before the last classification layer).

NASNet-A:

In the work of [262], the authors proposed to search for an architectural building block on a small dataset and then transfer the block to a larger dataset to re- with 1001 classes (ImageNet+background) 6 . 4032 features are extracted from the last layer before the classification layer.

VGG16:

The VGG Network was introduced by the researchers at Visual Graphics Group at Oxford ([212]). This network is specially characterized by its pyramidal shape. VGG16 takes as input a {3, 224, 224} RGB image and we use the network pretrained on ImageNet with 1000 classes 7 . Features are extracted from the last max pooling layer, which results in 512x7x7 features.

A.3 Dissimilarity-based learning

In our previous work ([44]), we proposed to use RFSVM to integrate information from different views together (each feature group is a view in multi-view learning framework). We have shown that RFSVM offers a good performance on Radiomics data. The RFSVM method can deal well with high dimensional low sample size multi-view data because: (i) RFSVM uses random forest dissimilarity measure to transfer each view of the data to a dissimilarity matrix so that the data dimension is reduced without feature selection, and at the same time the data in each view become directly comparable; (ii) RFSVM can take advantage of the complementary information contained in each view by combining the dissimilarity matrices together. We now recall the RFSVM method.

Random forest: Given a training set T , a Random Forest classifier H is a classifier made up of M trees denoted as in Equation (A.1):

H(x) = {h k (x), k = 1, . . . , M} (A.1)
where h k (x) is a random tree grown using the Bagging and the Random Feature Selection techniques as in [START_REF] Biau | A random forest guided tour[END_REF]. For predicting the class of a given query point x with such a tree, x goes down the tree structure, from its root till its of Equation (A.4), one can calculate the joint similarity matrix S H as in Equation (A.5):

S H = 1 -D H (A.5)
where 1 is a matrix of ones. SVM is one of the most successful classifier. Apart from the most used gaussian kernel, a lot of custom kernels can also be used:

we use the joint similarity matrix S H inferred from the RF classifier H as a kernel in a SVM classifier.

A.4 Experiments

The The protocol of the experiments is as follows:

• First, the 6 feature extractors described in Section 2 are used to extract features from histology image data. As there is no patch label provided, to simplify the feature extracting process, all images are rescaled to the network input size.

• Second, for each group of features, a random forest with 500 trees is built. The performance of each feature group is measured by the classification accuracy of the random forest. The random forest dissimilarity matrix is calculated for each group too.

• Finally, the RFSVM method described in Section 3 is used to combine all the groups together. Two RFSVMs are used: RFSVM (DL only) combines the five deep learning based feature groups; RFSVM-All combines all the six feature groups. For RFSVM, the search range of parameter C for SVM is {0.01, 0.1, 1, 10, 100, 1000}.

histology images

Note that in [START_REF] Bill | A Comparative Analysis of Predictive Learning Algorithms on High-Dimensional Microarray Cancer Data[END_REF], the authors found that when dealing with high dimensional low sample size data, stratification of the sampling is central for obtaining minimal misclassification. In this work, the stratified random splitting procedure is repeated 10 times, with 75% as training data and 25% as testing data.

In order to compare the methods, the mean and standard deviations of accuracy were evaluated over the 10 runs. However, for the contest, only one model can be submitted. Hence the best performance among the 10 runs is also presented and chosen as the model for the contest.

A.5 Results

The results of the experiments are shown in

A.6 Conclusion

In this work, we firstly compared the popular handcrafted features used in breast cancer histology image classification with five deep learning based feature extractors pretrained on ImageNet. Not surprisingly, the experimental results show that the deep learning based features are better than the handcrafted. To improve the performance of transfer learning, we tackled the problem of breast cancer histology image classification as an HDLSS multi-view learning task and applied an RFSVM method previously proposed for the classification of Radiomics data. The results obtained with RFSVM (DL only)

show that the performance of transfer learning can be improved by combining multiple feature extractors together. The results obtained with RFSVM-All

show that even though deep learning based features have better performance than handcrafted features for breast cancer histology image classification, the accuracy can be improved significantly when they are combined together and surpass the state of the art performance on the dataset used.

B.1 Introduction

One of the biggest challenges of cancer treatment is the inter-tumor heterogeneity and intra-tumor heterogeneity. It demands for more personalized treatment. In Radiomics, a large amount of features from standard-of-care images obtained with CT (computed tomography), PET (positron emission tomography) or MRI (magnetic resonance imaging) are extracted to help the diagnosis, prediction or prognosis of cancer [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]. Many medical image studies like [215,219] have already tried to use quantitative analysis before the existence of Radiomics. However, with the development of medical imaging technology and more and more available softwares allowing for more quantification and standardization. Radiomics focuses on improvements of image analysis, using an automated high-throughput extraction of large amounts of quantitative features [142]. Radiomics has the advantage of using more useful information to make optimal treatment decisions (personalized medicine)

and make cancer treatment more effective and less expensive.

Radiomics is a promising research field for oncology, but it is also a challenging machine learning task. In the work [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF], the authors identify Radiomics as a challenge in machine learning for the three following reasons: (i) small sample size: due to the difficulty in data sharing, most of Radiomics data sets have no more than 200 patients; (ii) high dimensional feature space: the feature space for Radiomics data is always very high dimensional compared to the sample size; (iii) multiple feature groups: different sources and different feature extractors are used in Radiomics -the most used features include tumor intensity, shape, texture, and so on [3] -and it may be hard to exploit the complementary information brought by these different views [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF].

When the three challenges are encountered in a classification task, it can be seen as an HDLSS (High dimension low sample size) Multi-View learning task. Now most studies in Radiomics ignore the third challenge and propose to simply concatenate different feature groups and to use a feature selection method to reduce the dimension. However, a lot of useful information may be lost when only a small subset of features is retained [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF], and the complementary information that different feature groups can offer may be ignored [START_REF] Cao | Improve the performance of transfer learning without fine-tuning using dissimilarity-based multi-view learning for breast cancer histology images in International Conference Image Analysis and Recognition[END_REF].

In contrast to the current studies that treat Radiomics data as a single-view machine learning task, we have proposed in our previsous work to cope with Radiomics complexity using an HDLSS multi-view paradigm [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]: we have The remainder of this paper is organized as follows. Related works in Radiomics and multi-view learning are discussed in Section 2. In section 3, the proposed dynamic voting solution is introduced. Before turning to the result analysis (Section 5), we describe the data sets chosen in this study and provide the protocol of our experimental method in Section 4. We conclude and give some future works in Section 6.

B.2 Related Works

In the state of the art of Radiomics, groups of features are most often concatenated into a single feature vector, which results in an HDLSS machine learning problem. In order to reduce the high dimensionality, some feature selection methods are used : in the work of [174] and [3], they used feature stability as a criterion for feature selection While in the work of [218], they used a SVM (Support Vector Machine) classifier as a criterion to evaluate the predictive value of each feature for pathology and TNM clinical stage. Different filter feature selection methods have also been compared along with reliable machine learning methods to find the optimal combination [174]. Generally speaking,

B.3 Proposed MCS based solutions

As explained in the Introduction, the simple MCS based late integration used in [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF] has shown a good potential for Radiomics. In this section, we use several more intelligent voting methods including static voting and dynamic voting to test if they can get significantly better.

For multi-view learning tasks, the training set T is composed of Q views:

T (q) = {(X (q)
1 , y 1 ), . . . , (X (q)

N , y N )}, q = 1..Q. Generally speaking, the MCS based late integration method builds a classifier C (q) for each view T (q) . During test time, for each test data X t , C (q) will predict the class label label (q) t of X t . Finally, the predicted labels from all the views {label Here Random forest is chosen as the classifier for each view T (q) because it can deal well with different data types, mixed variables and high dimensional data [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]. Random forest can also offer the OOB measure, which can be used as a measure for static weight and also to replace extra validation dataset for dynamic voting methods. In addition, random forest also provides a proximity measure, which can be used to calculate the neighborhood of a test sample [227].

Firstly, for each view q, a Random Forest H (q) is built with M decision trees, and is denoted as in Equation (B.1): H(X) = {h k (X), k = 1, . . . , M} (B.1) where h k (X) is a random tree grown using bagging and random feature selection. We refer the reader to [START_REF] Biau | A random forest guided tour[END_REF][START_REF] Breiman | Random forests[END_REF] for more details about this procedure.

For a J-class problem with label (q) t = i, where i ∈ {1, 2, . . . , J}, a weight W (q) is used for each view q (for the case of majority voting, all W (q) = 1). The final decision is made by:

y t = Max
(q) t = j) × W (q) ) (B.2)
different patients. One easy access to this kind of "personalized" information is the prediction probability of each test sample as it shows generally how confident the classifier C q is on the test data.

The predicted class probabilities of a test sample X t for random forest are computed as the mean predicted class probabilities of the trees in the forest. The class probabilities of a single tree is the fraction of samples of the same class in a leaf. The global weight W (q)

global of view q for each test data X t is simply the predicted probability (posterior probability obtained from H (q) ) for the most confident class of random forest, which measures the overall confidence rate of label prediction based on all the training data:

W (q) global = P(label (q) t | X t , H (q) ) (B.4) W (q)
global generally reflects how confident the classifier H (q) is when predicting the label of a test sample. But it also means the global measure is not very personalized. To capture more personalized information, we propose in the next subsection the local weight measure.

B.3.3 LDV (Local Dynamic Voting)

A local weight usually means the performance or confidence of a classifier in a smaller neighborhood in validation data of a test sample. It usually demands two measures: firstly, a distance measure to find the neighborhood; secondly the competence measure to evaluate the performance of the classifier in the neighborhood. RFD (random forest dissimilarity) in this work is used as a distance measure to find the neighborhood of a given test sample, while OOB measure is used to replace the validation dataset.

The RFD measure D H is inferred from a RF classifier H, learned from training data T. For each tree in the forest, if two samples end in the same terminal node, their dissimilarity is 0 otherwise 1. This process goes over all trees in the forest, and the average value is the RFD value (more details are given in [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]). It can be told that compared to other dissimilarity measures, RFD takes the advantage of class information to measure the distance [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF].

The reason why we choose to multiply global weight and local weight for deriving a combined weight, is that, as it is explained previously, W global lacks personalized information, but it can be counter-balanced by W local to give more preference in some situations. For example, when W (q) global agrees with W (q) local on a particular view q, if both weights are small, then W (q) GL becomes even smaller as we do not have confidence on this view; if both weights get bigger and bigger, then W (q)

GL gets closer and closer to both weights, especially local weight. On the contrary, when W (q) global disagrees with W (q) local , it is hard to make a decision with a disagreement (as we need prior knowledge to decide to choose global or local weight); hence we penalize W (q) GL as long as there is a disagreement (W (q) GL is smaller than 0.5) but still with a preference to W (q) local .

B.4 Experiments

In this study, we use several publicly available Radiomics datasets. A general description of all datasets can be found in Table B The main objective of the experiment is to compare the state of the art Radiomics methods to static and dynamic voting methods. In total six methods are compared: one state of the art Radiomics method, i.e. SVMRFE; two static weighting methods, i.e. MVRF (combines RF results with majority voting as in [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]) and WRF (combines RF results with weights as in Section 3.1, the weights are the OOB accuracy of each H (q) ); three dynamic weighted voting methods, i.e. GDV, LDV and GLDV as described in the previous section.

applications

For the two dynamic voting methods that use local weights, LDV and GLDV, the neighborhood size n neighbor is set to 7 according to the work of [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]. For SVMRFE, the number of selected features is defined as in [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF] according to the experiments of [START_REF] Bolón-Canedo | A review of feature selection methods on synthetic data[END_REF] and a Random forest classifier is then built on the selected features. For all random forest classifiers, the tree number is set to 500 while the other parameters are set to the default values given by the Scikit-Learn package for Python.

Similar to our previous work [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF][START_REF] Cao | Improve the performance of transfer learning without fine-tuning using dissimilarity-based multi-view learning for breast cancer histology images in International Conference Image Analysis and Recognition[END_REF], a stratified repeated random sampling approach was used to achieve a robust estimate of the performance. The stratified random splitting procedure is repeated 10 times, with 50% sample rate in each subset. In order to compare the methods, the mean and standard deviations of accuracy are evaluated over 10 runs. The results of mean accuracies, along with the corresponding standard deviation, over the 10 repetitions are shown in Table B.2. GDV and the two static voting methods have almost the same results over the four datasets, but these results are different from the two dynamic weighted voting methods LDV and GLDV. It is not surprising that there is no difference between MVRF and WRF because the datasets we use in this work have only five views, which means that there is no situation like even votes (the worst case would be 3 against applications GLDV is always better than both LDV and GDV, which means that for different datasets, the best combination of LDV and GDV should be different. To further study the preference of global weight W global and local weight W local for different datasets, a new combination is formed as:

B.5 Results

Dataset

W (q) GLnew = (W (q) global ) 1-a × (W (q) local ) a (B.7)
From Equation B.7 it can be told that when a = 1, the combination is only affected by local accuracy while when a = 0 the combination is only affected by global accuracy. The results of W (q)

GLnew are shown in Table B.3, from which we can confirm our conclusion that for IDHCodel1 and nonIDH data, they get better results when they use more global weight. For lowGrade and progression data, they get better results when they use more local weight.

B.6 Conclusions

In the state of art works of Radiomics, most studies used feature selection methods as a solution for the HDLSS problem. In this work, we have treated Radiomics as a multi-view learning problem and investigated the potential of MCS based late integration methods, proposed earlier in [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF]. In particuler,

we have investigated some dynamic voting based MCS methods, that can give each patient a personalized prediction by dynamically integrating the classification result from each view. We believe these methods have a great potential and can significantly outperform early integration methods that make use of feature selection in the concatenated feature space.

To confirm our hypothesis, a representative early integration method, five MCS methods including three dynamic voting methods and two static voting methods, have been compared on four Radiomics datasets. We conclude from our experiments that all MCS based late integration methods are generally better than the state of art Radiomics solution, but only LDV and GLDV are significantly better, which shows the potential of MCS based late integration methods of being a better solution than the state-of-art Radiomics solutions.

C.2 Results

The results of average accuracy over 10 repetitions as well as the standard deviation on 21 datasets are shown in Table C.2. From the average ranking, it can be seen that RFD I H is the best, followed by RFD NC and then RFD PB . Four RFD based methods are better than LMNN and Euclidean distance. There is no surprise that Euclidean distance is ranked at the last place.

RFD I H wins the first place on 13 datasets, while RFD NC wins the first place on 5 datasets. However, Euclidean distance and LMNN win the first place once each. It can be seen that RFD based methods are the best for the majority of data sets.

1 21 1. 3 . 1 36 1. 4

 12131364 Background and workflow . . . . . . . . . . . . . . . 11 1.2.2 Radiomic features . . . . . . . . . . . . . . . . . . . . 14 1.2.3 Machine Learning for Radiomics . . . . . . . . . . . . 18 1.3 HDLSS multi-view learning: a literature review . . . . . . Problem statement . . . . . . . . . . . . . . . . . . . . 22 1.3.2 Early integration . . . . . . . . . . . . . . . . . . . . . 25 1.3.3 Late integration . . . . . . . . . . . . . . . . . . . . . . 31 1.3.4 Intermediate integration . . . . . . . . . . . . . . . . . Conclusion and contributions . . . . . . . . . . . . . . . . . 42

  are studied. Two more accurate dissimilarity measures are proposed based on Random Forest to improve the classification performance of multi-representation fusion. Instance hardness measure in the subspace defined by each leaf node is used to weight the dissimilarity values so that they are no more binary and more accurate. The experimental results in Chapter 3 show significant improvement over classic RFD based multi-representation fusion.

  k) ij = 0 means that the i th and j th instances fall in different terminal nodes, which implies that S (k) il and S (k) lj can not be both equal to 1. Thus S

S H = 1 -

 1 D H as a kernel matrix in a kernel-based learning method, e.g. a SVM classifier (named RFSVM in the following and illustrated in Figure 2.3a) and (ii) by using the dissimilarity matrix D H as a new training set (named RFDis in the following and illustrated in Figure 2.3b).

1 .

 1 The first four datasets are Radiomics problems. The others 11 datasets are non-Radiomic datasets but relate to similar HDLSS multi-view applications. For Radiomic datasets, there are 5 views for each of these 4 datasets: 4 texture feature groups from axial T1-weighted MR images before and after gadolinium-based CE material administration as well as axial T2-weighted and axial T2-weighted fluid attenuated inversion recovery (FLAIR) images. The fifth view are Visually AcceSAble Rembrandt Images (VASARI) features ([104]). More details about the Radiomic datasets can be found in [258]. As for the non-Radiomic datasets: LSVT is a dataset on vocal performance degradation of Parkinson's disease subjects ([226]); Metabolomic contains biomarkers (CEA and TIMP), fluorescence concentration (PF) and NMR profiles for early detection of colorectal cancer ([37]); BBC and BBCSport are text classification problems constructed from the news article corpora by splitting articles into related segments of text ([249]); the remaining datasets (Cal7 and 20, Mfeat, NUS-WIDE2 and 3, and AWA8 and 15) are classical image classification datasets obtained using different feature extractors. Similar to [151], these latter datasets have been randomly downsampled to simulate the HDLSS setting.

Figure 2 .

 2 8 (a) shows that except for RFMDS, all the methods are significantly better than SVM-RFE with α = 0.05 and 0.01. RFSVM and RFDis are the ones that win the most against SVM-RFE.

Figure 3 .

 3 Figure 3.3b, where training instances are represented by circles while red triangles represent test instances. These six terminal nodes are very different: node #2 and node #10 are the two biggest sub-regions while other nodes occupy much smaller sub-regions; node #10 has the tree depth of 1 while node #8 has the tree depth of 5; node #2 contains a lot of training instances while node #7 contains only 1 training instance. Due to the diversity in terminal nodes of a decision tree, we have different confidences on different nodes too.

  method to the classic Euclidean distance measure and a metric learning measure based on the recommendation of the very recent survey([148]): Large Margin Nearest Neighbors (LMNN[START_REF] Domeniconi | Large margin nearest neighbor classifiers[END_REF]) along with PCA. LMNN is one of the most widely-used Mahalanobis distance learning methods. The constraints of LMNN are defined for each training instance: the "target neighbors" of a training instance are its k nearest neighbors based on Euclidean distance while the "impostors" are the instances from other classes. The objective is to make the target neighbors belong to the correct class of the training instance and keep impostors away with a margin. According to the suggestions in [148]: the number of components for PCA is chosen as 300; the size of neighborhood for LMNN is set to 25, but for datasets with less than 25 instances for some class, the size of neighborhood is set as the number of training minority class size.

  For each training instance, we firstly find the corresponding decision path and leaf node. Then, we project all training instances into the subspace defined by the decision path and use kDN to measure the instance hardness in the subspace. According to the suggestion of [72], the neighborhood size of kDN can be set to 7. It can be seen that IH assigns different weights to the training instances even though they are in the same terminal node. Algorithm 3 RFD I H : RFD with instance hardness 1: Input: Test instance x t , trained RF H <-RandomForest(mtry = √ m, n trees = M, T ) 2: for each training instance x i in T do 3:

  disadvantage of most instance hardness measures is that the class information is needed. In this case, we can only measure the IH on training instances and measure the confidence only based on training instances. But dissimilarity reflects the pairwise information, the instance hardness of both training instances and the given test instances should be taken into account. In [126], the authors have tried to estimate the classifier's trust score for each test instance by measuring the agreement between the classifier and the modified nearest neighbor classifier on the test instance with some data density requirement.

FIGURE 4 . 1 :

 41 FIGURE 4.1: Pairwise comparison between different combination methods and the simple averaging. The vertical lines illustrate the critical values considering a confidence level α = {0.10, 0.05, 0.01}.

FIGURE 4 . 3 :

 43 FIGURE 4.3: Pairwise comparison between different combination methods and the simple averaging. The vertical lines illustrate the critical values considering a confidence level α = {0.10, 0.05, 0.01}.

  duce the computation cost and improve the efficiency. They used NAS (Neural Architecture Search) framework from [261] as the main search method for their NASNets. The three networks constructed from the best three searches are named NASNet-A, NASNet-B and NASNet-C respectively. In this work, a NASNet-A pretrained on ImageNet is used. The input space of NASNet-A is a {3, 331, 331} RGB image and we use the network pretrained on ImageNet

  used a naive MCS (Multiple Classifier Systems) based method which turns out to work well for Radiomics data but not significantly better than the state of the art methods used in Radiomics. Here we want to further investigate the potential of the MCS multi-view approach. Hence we propose several less simplistic MCS based methods including static voting and dynamic voting methods to combine classification results from different views. Our main contribution in this paper is thus to propose a new dynamic voting scheme to give a personalized diagnosis (decision) from Radiomics data. This dynamic voting method is designed for small sample sized dataset like Radiomics data and uses a large number of trees in random forest to provide OOB (Out Of Bag) samples to replace the validation dataset.

  be combined either by majority voting or weighted voting.

Chapter 2 Random Forest Dissimilarity for intermediate integration Contents 2

.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 2.2 Related works . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 2.2.1 Metric learning . . . . . . . . . . . . . . . . . . . . . . 49 2.2.2 Kernel learning . . . . . . . . . . . . . . . . . . . . . . 52 2.2.3 Random partitions . . . . . . . . . . . . . . . . . . . . 53 2.2.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 55 2.3 Random Forest dissimilarity measure . . . . . . . . . . . . . 56 2.3.1 Random Forest . . . . . . . . . . . . . . . . . . . . . . 56 2.3.2 Random Forest Dissimilarity (RFD) . . . . . . . . . . 57 2.3.3 RFD matrix . . . . . . . . . . . . . . . . . . . . . . . . 59 2.4 The parametrization of RFD . . . . . . . . . . . . . . . . . . 62 2.4.1 Experiments on real-world datasets . . . . . . . . . . 65 2.4.2 Results on real-world datasets . . . . . . . . . . . . . 67 2.4.3 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . 70 2.5 RFD-based multi-view learning . . . . . . . . . . . . . . . . 70 2.5.1 Experimental protocol . . . . . . . . . . . . . . . . . . 71 2.5.2 Results and discussions . . . . . . . . . . . . . . . . . 73 2.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

  d 11 d 12 d 13 . . . d 1n d 21 d 22 d 23 . . .

	d 2n . . . d n1 d n2 d n3 . . . d nn . . . . . . . . . . . .	      	(2.14)

where d ij denotes d(x i , x j ), for all (x i , x j ) ∈ T × T . D is non-negative and respects the reflexivity condition. Such a dissimilarity matrix can be viewed as a new training set, where each training instance x i is described by a vector {d i1 , d i2 , . . . , d in }.

  This follows immediately from the equation x T (A + B)x = x T Ax + x T Bx ≥ 0. Consequently any sum of p.s.d. matrices is p.s.d.

	Theorem 2.3.2 A symmetric binary matrix MA ∈ (0, 1) n×n , with n ≥ 3, is p.s.d. if and only if it satisfies the following inequalities:

d. or not Theorem 2.3.1 If both A and B are two p.s.d. matrices then so is A + B.

TABLE 2 . 1 :

 21 Overview of the real-world datasets used in our experiments. IR (imbalanced ratio) is the number of instances of the majority class over the number of instances of the minority class.

		#features #instances #views #classes	IR
	nonIDH1[258]	6746	84	5	2	3
	IDHcodel[258]	6746	67	5	2	2.94
	lowGrade[258]	6746	75	5	2	1.4
	progression[258]	6746	84	5	2	1.68
	LSVT[226]	309	126	4	2	2
	Metabolomic[37]	476	94	3	2	1
	Cal7[151]	3766	1474	6	7	25.74
	Cal20[151]	3766	2386	6	20	24.18
	Mfeat[93]	649	600	6	10	1
	BBC[249]	13628	2012	2	5	1.34
	BBCSport[249]	6386	544	2	5	3.16
	NUS-WIDE2[60]	639	442	5	2	1.12
	NUS-WIDE3[60]	639	546	5	3	1.43
	AWA8[143]	10940	640	6	8	1
	AWA15[143]	10940	1200	6	15	1

the nearest neighbor to the test instance. This method can well reflect the quality of the dissimilarity matrix, because the idea behind 1NN is that the most similar instances should belong to the same class. A stratified random splitting strategy has been used to obtain a robust estimate of the performance of these 1NN classifiers. Each dataset has been randomly split 50 times, with 50% of the instances for training and 50% for test. A grid search has been performed on M and δ over the 50 random splits.

TABLE 2 .

 2 3: An overview of all the methods compared in this work

	Methods	Integration Method Learning space
	SVM-RFE [257] Early	Reduced feature space
	RFSVM	Intermediate	Kernel space
	RFDis	Intermediate	Dissimilarity space
	RFMDS [99]	Intermediate	Embedded dissimilarity space
	EasyMKL [6]	Intermediate	Kernel space
	LateRFDis	Late	Dissimilarity space

  Let us recall that RFSVM and EasyMKL both adopt the same kind of kernel-based principle. From average ranking in Table2.4, one can see that both RFSVM and RFDis globally outperform EasyMKL. EasyMKL is accurate for most datasets except for the two medical datasets(LSVT and Metabolomic)

vious for multi-class data (Cal7: 7 classes, Cal20: 20 classes, Mfeat: 10 classes, AWA8: 8 classes, AWA15: 15 classes).

Comparison of the RFSVM method and the state-of-the-art MKL method (EasyMKL):

Table 2 .

 2 5. By looking at the average ranking, the RFSVM method is still ranked first and the RFMDS method is ranked second. As for the feature selection method SVM-RFE, it is still ranked last.

	Comparison

of the multi-view solutions and the state-of-the-art Radiomics solution:

  From Table2.5, one can see that all the multi-view solutions are generally better than the Radiomics solution SVM-RFE. Similar to the analysis on non-Radiomics problems, a pairwise analysis based on the Sign test

TABLE 2 .

 2

		5: Classification accuracy with 50% training in-
		stances and 50% test instances for Radiomic datasets
	Dataset	SVM-RFE RFSVM	RFDis	RFMDS	EasyMKL LateRFDis
	nonIDH1	76.28%	80.69%	79.53%	82.55%	76.04%	80.93%
	IDHcodel	±4.39 73.23%	±2.76 76.76%	±3.57 76.47%	±4.55 73.82%	±2.37 72.35%	±2.51 76.17%
	lowGrade	±5.50 62.55%	±4.52 63.95%	±3.95 63.48%	±4.26 62.55%	±2.35 64.65%	±2.06 65.11%
	progression	±3.36 62.36%	±4.56 65.52%	±3.76 63.42%	±5.53 65.00%	±4.26 59.73%	±5.20 58.94%
	±3.73 Average Rank 4.875	±4.47 2.000	±6.49 3.250	±5.95 3.125	±6.00 4.750	±6.02 3.000

TABLE 3 .

 3 1: An overview of all the methods compared in this chapter

	Methods	Dissimilarity measure
	EUDis	Euclidean distance
	LMNNDis PCA+LMNN
	RFDis	RFD
	RFDis PB RFDis NC	RF based dissimilarity in [85]

TABLE 3 .

 3 22% ± 2.55 42.28% ± 3.13 56.06% ± 1.35 56.38% ± 1.47 56.34% ± 1.68 awa15 24.80% ± 0.97 28.25% ± 1.60 37.90% ± 1.49 37.62% ± 1.40 37.93% ± 1.50 Metabo 69.38% ± 2.29 67.08% ± 4.04 67.71% ± 5.12 67.50% ± 5.76 67.08% ± 6.31 mfeat 96.00% ± 1.45 96.87% ± 0.79 97.56% ± 0.99 97.63% ± 0.95 97.63% ± 1.00 nus2 89.52% ± 1.44 90.33% ± 1.55 92.49% ± 2.01 92.49% ± 1.81 92.67% ± 1.47 bbc 85.89% ± 1.33 93.02% ± 1.29 92.82% ± 0.67 93.00% ± 0.67 92.33% ± 0.49 lowGrade 63.72% ± 5.12 62.33% ± 7.04 63.48% ± 3.76 63.72% ± 4.67 63.95% ± 3.64 nus3 73.92% ± 2.40 78.02% ± 2.69 79.41% ± 1.94 79.64% ± 2.19 79.91% ± 2.14 progression 58.42% ± 4.82 62.63% ± 5.86 63.42% ± 6.49 63.42% ± 7.48 63.95% ± 6.56 LSVT 82.86% ± 2.11 85.24% ± 2.84 83.33% ± 3.97 82.70% ± 3.44 83.49% ± 3.56 IDHCodel 73.53% ± 5.42 71.47% ± 2.30 76.47% ± 3.95 76.47% ± 4.16 76.18% ± 3.82 nonIDH1 79.07% ± 3.45 73.26% ± 3.49 79.53% ± 3.57 79.53% ± 3.72 79.77% ± 3.46 bbcsport 80.11% ± 1.69 73.77% ± 5.45 81.75% ± 2.70 82.56% ± 2.85 79.93% ± 3.11 Cal20 84.04% ± 0.82 87.50% ± 0.78 89.12% ± 0.69 89.27% ± 1.01 89.06% ± 1.19 Cal7 92.67% ± 0.63 95.09% ± 0.66 95.21% ± 0.67 95.51% ± 0.50 95.34% ± 0.48

		2: The classification accuracies of multi-view in-
	termediate integration with different dissimilarity measures.
	Random Forest is built on the joint dissimilarity matrix (av-
		erage over views) to evaluate the performance.
		EUDis	LMNNDis	RFDis	RFDis PB	RFDis NC
	awa8 39.Avg rank	4.23	3.90	2.60	2.13	2.13

TABLE 3 .

 3 08% ± 4.04 67.71% ± 5.12 67.50% ± 5.[START_REF] Davies | The random forest kernel and other kernels for big data from random partitions[END_REF] 67.08% ± 6.31 69.17% ± 5.80 Mfeat 96.00% ± 1.45 96.87% ± 0.79 97.56% ± 0.99 97.63% ± 0.95 97.63% ± 1.00 97.53% ± 1.00 NUS-WIDE2 89.52% ± 1.44 90.33% ± 1.55 92.49% ± 2.01 92.49% ± 1.81 92.67% ± 1.47 92.82% ± 1.93 BBC 85.89% ± 1.33 93.02% ± 1.29 92.82% ± 0.67 93.00% ± 0.67 92.33% ± 0.49 95.46% ± 0.65 lowGrade 63.72% ± 5.12 62.33% ± 7.04 63.48% ± 3.76 63.72% ± 4.67 63.95% ± 3.64 63.95% ± 5.62 NUS-WIDE3 73.92% ± 2.40 78.02% ± 2.69 79.41% ± 1.94 79.64% ± 2.19 79.91% ± 2.14 80.32% ± 1.95 progression 58.42% ± 4.82 62.63% ± 5.86 63.42% ± 6.49 63.42% ± 7.48 63.95% ± 6.56 65.79% ± 4.71 LSVT 82.86% ± 2.11 85.24% ± 2.84 83.33% ± 3.97 82.70% ± 3.44 83.49% ± 3.56 84.29% ± 3.51 IDHCodel 73.53% ± 5.42 71.47% ± 2.30 76.47% ± 3.95 76.47% ± 4.16 76.18% ± 3.82 76.76% ± 3.59 nonIDH1 79.07% ± 3.45 73.26% ± 3.49 79.53% ± 3.57 79.53% ± 3.72 79.77% ± 3.46 80.70% ± 3.76 BBCSport 80.11% ± 1.69 73.77% ± 5.45 81.75% ± 2.70 82.56% ± 2.85 79.

		3: The classification accuracies of multi-view in-
	termediate integration with different dissimilarity measures.
	Random Forest is built on the joint dissimilarity matrix (av-
	erage over views) to evaluate the performance.
	EUDis	LMNNDis	RFDis	RFDis PB	RFDis NC	RFDis I H
	AWA8 AWA15 Metabo 69.38% ± 2.29 67.Avg rank 39.22% ± 2.55 42.28% ± 3.13 56.06% ± 1.35 56.38% ± 1.47 56.34% ± 1.68 56.22% ± 1.01 24.80% ± 0.97 28.25% ± 1.60 37.90% ± 1.49 37.62% ± 1.40 37.93% ± 1.50 38.23% ± 0.83 5.20 4.83 3.67 2.83 2.93 1.53

93% ± 3.11 90.18% ± 1.96 Cal20 84.04% ± 0.82 87.50% ± 0.78 89.12% ± 0.69 89.27% ± 1.01 89.06% ± 1.19 89.76% ± 0.80 Cal7 92.67% ± 0.63 95.09% ± 0.66 95.21% ± 0.67 95.51% ± 0.50 95.34% ± 0.48 96.03% ± 0.53

TABLE 4 .

 4 1: An overview of all the methods compared in this section.

	Name	Combination method
	Avg	Simple average over all matrices
	SW 3NN SW DA SW OOB	Static weighting by 3NN performance Static weighting by kernel alignment (Equation 4.11) Static weighting by OOB accuracy (Section 4.2.2)

TABLE 4 .

 4 2: The average classification rates of multi-view intermediate integration with different static weighting methods. The dissimilarity measure is RFD I H proposed in the previous chapter. Random Forest is built on the joint dissimilarity matrix to evaluate the performance.

		Avg	SW 3NN	SW DA	SW OOB
	awa8 awa15 Metabo	56.22% ± 1.01 56.22% ± 0.99 56.12% ± 1.42 56.59% ± 1.41 38.23% ± 0.83 38.13% ± 0.87 38.27% ± 1.05 38.23% ± 1.26 69.17% ± 5.80 68.54% ± 5.

85 70.00% ± 4.86 70.00% ± 6.12 mfeat

	nus2 bbc lowGrade	97.53% ± 1.00 97.53% ± 1.09 97.53% ± 1.09 97.57% ± 1.01 92.82% ± 1.93 92.86% ± 1.88 92.60% ± 2.12 92.97% ± 1.72 95.46% ± 0.65 95.52% ± 0.64 95.36% ± 0.74 95.46% ± 0.60 63.95% ± 5.62 62.56% ± 6.

10 63.95% ± 3.57 63.95% ± 5.01 nus3 80.32% ± 1.95 79

  .95% ± 2.40 80.09% ± 2.07 80.14% ± 2.20 progression 65.79% ± 4.71 65.79% ± 4.71 65.79% ± 4.99 66.32% ± 4.37

	LSVT IDHCodel nonIDH1 bbcsport Cal20 Cal7 Avg rank	84.29% ± 3.51 84.29% ± 3.65 84.60% ± 3.54 84.76% ± 3.63 76.76% ± 3.59 77.06% ± 3.43 77.35% ± 3.24 76.76% ± 3.82 80.70% ± 3.76 80.47% ± 3.32 80.00% ± 3.15 80.93% ± 4.00 90.18% ± 1.96 90.29% ± 1.83 90.26% ± 1.78 90.26% ± 1.95 89.76% ± 0.80 89.88% ± 0.82 89.77% ± 0.68 90.00% ± 0.71 96.03% ± 0.53 96.10% ± 0.57 96.11% ± 0.60 96.10% ± 0.60 2.93 2.77 2.57 1.73

TABLE 4 .

 4 3: The average difference between SW OOB and Avg in terms of: weight value difference, the corresponding joint dissimilarity matrices difference and the percentage of differently predicted test instances.

	Weight di f f	Dissimilarity di f f Prediction di f f
	awa8 awa15 Metabo mfeat nus2 bbc lowGrade nus3 progression 1.17% ± 0.77 1.26% ± 1.16 1.72% ± 1.18 1.85% ± 1.47 1.62% ± 0.48 0.52% ± 0.43 0.19% ± 0.12 1.61% ± 0.97 1.27% ± 0.63 LSVT 0.88% ± 0.55 IDHCodel 0.70% ± 0.64 nonIDH1 0.75% ± 0.49 bbcsport 0.48% ± 0.28 Cal20 1.09% ± 0.37 Cal7 0.57% ± 0.20 Avg 1.05%	0.05% ± 0.01 0.03% ± 0.01 0.43% ± 0.34 0.15% ± 0.01 0.20% ± 0.05 0.00% ± 0.00 0.42% ± 0.14 0.20% ± 0.05 0.36% ± 0.15 0.19% ± 0.08 0.26% ± 0.17 0.39% ± 0.18 0.03% ± 0.02 0.19% ± 0.01 0.30% ± 0.02 0.22%	3.44% ± 1.42 9.65% ± 1.88 1.67% ± 1.56 0.70% ± 0.48 0.37% ± 0.57 0.30% ± 0.15 1.86% ± 1.74 1.53% ± 0.91 1.58% ± 1.75 0.48% ± 1.02 0.59% ± 1.18 0.23% ± 0.70 0.37% ± 0.23 1.56% ± 0.32 0.31% ± 0.21 1.64%
	large majority of cases, which indicates that the best static weighted combina-

tion found is always very close to the simple average combination. The most plausible interpretation of these results is that no single view is more or less informative in general than all others with regard to the classification task.

  To define the appropriate region of competence, we choose the KNN technique. KNN neighborhood is usually based on Euclidean distance. However, the experimental results from the previous chapter have shown that the three RF based dissimilarity measures are significantly better than the Euclidean distance, among which the best performing method is RFD I H . Hence, we propose to use RFD I H as the dissimilarity measure to define the region of competence θ t of each test instance x t . To calculate the local confidence w local

			i,t
	for the RF classifier H i , the OOB accuracy is estimated on the region of com-
	petence θ t :
	w local i,t	= OOB accuracy (H i , θ t )	(4.13)

5 :

 5 {w 1 , w 2 , . . . , w 2 Q -1 } := { all the Q-sized binary vectors, apart from all zeros } ⊲ Generation of the candidate subsets of views

	6: D m 1 , D m 2 , . . . , D m 2 Q -1 , where D m i = dissimilarity matrices 7: H = {H 1 , H 2 , . . . , H 2 Q -1 }, where H i = RandomForest(D m ∑ Q j=1 w i,j ×D (j) ⊲ Generation of the joint ||w i || ⊲ i , y) Generation of the pool of classifiers
	8: for each x t inT s do
	9: 10: 11: 12:	for each H i in H do d m t = ∑ Q (j) T j=1 w i,j ×d ||w i || θ t = KNN(RFD I H (H i , d m t )) w local i,t = OOB accuracy (H i , θ t )	⊲ Region of competence for x t
	13: 14:	w local t idx = argmax(w local = {w local 1,t , w local 2,t , . . . , w local 2 Q -1,t } t )
	15: Output: w idx

Table 4 .

 4 4 firstly shows that the DS local outperforms the four competitors for 10 over the 15 datasets. It also shows that DS local is the best ranked over all the datasets, followed by the static weighting method SW OOB . Even though the difference between the average ranks is very small, these two methods are very differents from each others. DS local usually achieves a much higher improvement in classification accuracy than SW OOB . For example, on datasets such as awa8, Metabo, lowGrade, progression and IDHCodel, DS local improves the average accuracy between 1 to 2 percent, while SW OOB only improves the average accuracy between 0.1 to 0.5 percent. On other datasets such as bbcsport, LSVT, nus2, nus3 and awa15, DS local also has better performance than SW OOB .

TABLE 4 .

 4 4: The experimental results of multi-view intermediate integration with different weighting methods. The dissimilarity measure is RFD I H proposed in the previous chapter.

	Avg	SW 3NN	SW DA	SW OOB	DS local
	awa8 awa15 Metabo mfeat nus2 bbc lowGrade nus3 progression 65.79% ± 4.71 65.79% ± 4.71 65.79% ± 4.99 66.32% ± 4.37 66.84% ± 5.29 56.22% ± 1.01 56.22% ± 0.99 56.12% ± 1.42 56.59% ± 1.41 57.28% ± 1.49 38.23% ± 0.83 38.13% ± 0.87 38.27% ± 1.05 38.23% ± 1.26 38.82% ± 1.56 69.17% ± 5.80 68.54% ± 5.85 70.00% ± 4.86 70.00% ± 6.12 70.21% ± 4.85 97.53% ± 1.00 97.53% ± 1.09 97.53% ± 1.09 97.57% ± 1.01 97.63% ± 0.99 92.82% ± 1.93 92.86% ± 1.88 92.60% ± 2.12 92.97% ± 1.72 93.30% ± 1.58 95.46% ± 0.65 95.52% ± 0.64 95.36% ± 0.74 95.46% ± 0.60 95.42% ± 0.59 63.95% ± 5.62 62.56% ± 6.10 63.95% ± 3.57 63.95% ± 5.01 65.81% ± 5.31 80.32% ± 1.95 79.95% ± 2.40 80.09% ± 2.07 80.14% ± 2.20 80.77% ± 2.06 LSVT 84.29% ± 3.51 84.29% ± 3.65 84.60% ± 3.54 84.76% ± 3.63 84.44% ± 3.87 IDHCodel 76.76% ± 3.59 77.06% ± 3.43 77.35% ± 3.24 76.76% ± 3.82 77.65% ± 3.77 nonIDH1 80.70% ± 3.76 80.47% ± 3.32 80.00% ± 3.15 80.93% ± 4.00 79.77% ± 2.76 bbcsport 90.18% ± 1.96 90.29% ± 1.83 90.26% ± 1.78 90.26% ± 1.95 90.44% ± 1.89 Cal20 89.76% ± 0.80 89.88% ± 0.82 89.77% ± 0.68 90.00% ± 0.71 89.15% ± 0.97 Cal7 96.03% ± 0.53 96.10% ± 0.57 96.11% ± 0.60 96.10% ± 0.60 94.65% ± 1.09 Avg rank 3.67 3.50 3.30 2.40 2.13

  Similar to the work of [220], we use the ResNet is one of the deepest deep learning architectures proposed by Microsoft researchers. The deep residual nets based methods have won the first places on the tasks of ImageNet detection, ImageNet localization, COCO detection, and COCO segmentation as well as the first place on the ILSVRC 2015 classification task ([115]). We use two ResNet in this work: ResNet-18 and ResNet-152. Both networks take as input a {3, 224, 224} RGB image and are pretrained on ImageNet with 1000 classes 34 . Features are extracted from the average pool layer (i.e. before the last classification layer), which results in 512 features for ResNet-18 and 2048 features for ResNet-152.

	ResNet-18 and ResNet-152:
	Parameter-Free Threshold Adjacency Statistics (PFTAS) from the python li-
	brary Mahotas ([62]) to build a 162-dimensional PFTAS-feature vector. GLCM
	features are widely used to describe the texture of tumor in cancer applica-
	tions. Same as PFTAS, the library Mahotas is used to calculate the GLCM
	features leading to a 175-dimensional GLCM-feature vector.

  dataset used in this work is from ICIAR 2018 Grande Challenge on BreAst Cancer Histology images8 . It is composed of Hematoxylin and eosin stained breast histology microscopy images. Microscopy images are labeled as nor-

mal, benign, in situ carcinoma or invasive carcinoma according to the predominant cancer type in each image. It is a balanced dataset with in total 400 images.

  Table A.1. We can tell that the best feature extractor is ResNet-152 with an average accuracy of 79.30% and best accuracy of 83.00%. Followed by ResNeXt with an average accuracy of 78.60% and the best accuracy of 81.00%. Surprisingly, the worst feature extractor is handcrafted features with PFTAS and GLCM with an average accuracy of 67.00%. In the work of[220], PFTAS and GLCM are the best features for breast cancer histology image classification. By comparing the performance of the six feature extractors, we can see that even though the deep learning networks are pretrained on ImageNet dataset, which is very different from histology images, they still have a better performance as a feature extractor for breast cancer data than the best handcrafted feature extractor used in the field of breast cancer histology image classification.

	With RFSVM (DL only) integrating all the five deep learning based feature
	groups together, the average accuracy is improved to 82.90% and the best per-
	formance is improved to 85.00%. However, when all feature groups are com-
	bined with RFSVM-All, the average accuracy is improved to 87.10% and the
	best performance is improved to 93.00%. It shows that even though the hand-
	crafted features do not have a very good performance individually, they can
	still provide useful complementary information for breast cancer classification
	when combined with deep learning based feature groups.

TABLE A .

 A 1: The image wise classification results with 75% training data and 25% test data. Average is the average accuracy over 10 runs, Best is the best accuracy among the 10 runs.

		Average	Best
	Handcrafted	67.00% ± 5.46	76.0%
	ResNet-18	75.10% ± 5.46	78.0%
	ResNeXt	78.60% ± 1.74	81.0%
	NASNet-A	74.70% ± 2.33	78.0%
	ResNet-152	79.30% ± 3.20	83.0%
	VGG16 RFSVM(DL only)	68.00% ± 5.04 82.90% ± 1.37	78.0% 85.0%
	RFSVM-All	87.10% ± 2.17	93.0%

TABLE A

 A

		.2: The confusion matrix, sensitivity and specificity
			of our best model.
		Benign	InSitu	Invasive	Normal
	Benign	23	1	0	1
	InSitu	0	23	0	2
	Invasive	1	0	24	0
	Normal	2	0	0	23
	Sensitivity	92%	92%	96%	92%
	Specificity	85%	96%	100%	85%

  .1 where IR stands for the imbalance ratio of the dataset. More details about these datasets can be found in the work of[258].

		#features #samples #views #classes	IR
	nonIDH1	6746	84	5	2	3
	IDHcodel	6746	67	5	2	2.94
	lowGrade	6746	75	5	2	1.4
	progression	6746	75	5	2	1.68

TABLE B .

 B

1: Overview of each dataset.

TABLE B .

 B 3: The results of new combinations WIn general, all MCS based late integration methods are better than feature selection methods. Majority voting is simple and efficient. GLDV is only better than majority voting on two datasets. But LDV and GLDV are preferable for Radiomics applications in the following three ways: (i) they give different weights of each view to each test sample, so that each test sample uses a different combination of classifiers to give a personalized decision; (ii) they are significantly better than the state of art work in Radiomics; (iii) the performance of GLDV can be further improved by adjusting the proportion of local weight and global weight. Note that other parameters like the neighborhood size can also be adjusted to optimize the performance. Compared to static voting, the disadvantage of dynamic voting is that it is more complex and less efficient.

					different a value.		(q) GLnew with
	Dataset	a=0	a=0.1	a=0.2	a=0.3	a=0.4	a=0.5	a=0.6	a=0.7	a=0.8	a=0.9	a=1
		(GDV)										(LDV)
	nonIDH	82.79%	82.79%	82.79%	82.32%	81.16%	80.23%	79.99%	79.30%	77.90%	77.44%	76.98%
	± 2.37 IDHCodel1 76.76%	± 2.37 76.76%	± 2.37 76.76%	± 2.13 75.88%	± 3.02 75.58%	± 2.80 75.29%	± 3.15 75.29%	± 2.42 75.29%	± 2.38 75.00%	± 2.33 75.00%	± 1.93 74.11%
	lowGrade	± 2.06 64.41%	± 2.06 64.41%	± 2.06 64.41%	± 1.76 64.65%	± 1.34 64.41%	± 1.44 64.41%	± 1.44 64.65%	± 1.95 64.18%	± 1.97 63.48%	± 1.97 63.48%	± 1.17 64.415%
	± 3.75 progression 61.57%	± 3.75 61.57%	± 3.75 61.84%	± 3.57 62.10%	± 3.45 62.36%	± 3.45 62.10%	± 3.72 62.36%	± 4.18 63.42%	± 3.75 62.89%	± 3.45 62.63%	± 3.45 62.36%
		± 4.27	± 4.27	± 3.57	± 3.56	± 3.91	± 4.43	± 4.41	± 4.62	± 4.77	± 4.37	± 4.56

TABLE C .

 C 1: Overview of the real-world datasets used in our experiments. IR (imbalanced ratio) is the number of instances of the majority class over the number of instances of the minority class.

		#features #samples #views #classes	IR
	AWA8[143]	10940	640	6	8	1
	AWA15[143]	10940	1200	6	15	1
	Bio[231]	1776	1000	1	2	1
	CNAE[93]	856	540	1	8	1
	Game[231]	1000	600	1	2	1
	Metabolomic[37]	476	94	3	2	1
	Mfeat[93]	649	600	6	10	1
	LU[231]	10937	250	1	2	1.02
	NUS-WIDE2[60]	639	546	5	2	1.12
	arcene[93]	10000	200	1	2	1.27
	BBC[249]	13628	2012	2	5	1.34
	lowGrade[258]	6746	84	5	2	1.4
	NUS-WIDE3[60]	639	442	5	3	1.43
	progression[258]	6746	75	5	2	1.68
	LSVT[226]	309	126	4	2	2
	GAs[93]	1396	76	1	3	2.67
	IDHcodel[258]	6746	67	5	2	2.94
	nonIDH1[258]	6746	84	5	2	3
	BBCSport[249]	6386	544	2	5	3.16
	Cal20[151]	3766	2386	6	20	24.18
	Cal7[151]	3766	1474	6	7	25.74

TABLE C .

 C 2:The result over 10 repetitions comparing different distance measures. First 2 methods are Euclidean distance and metric learning method PCA+LMNN. RFD is the classic RFD measure, RFD NC is RFD weighted by posterior probability in the leaf node. RFD I H is weighted by instance hardness. The best result of each data set is presented in bold.

https://github.com/Cadene/pretrained-models.pytorch

https://github.com/pytorch/vision/tree/master/torchvision

https://download.pytorch.org/models/resnet18-5c106cde.pth

http://data.lip6.fr/cadene/pretrainedmodels/fbresnet152-2e20f6b4.pth

http://data.lip6.fr/cadene/pretrainedmodels/resnext101-64x4d-e77a0586.pth

https://data.lip6.fr/cadene/pretrainedmodels/nasnetalarge-a1897284.pth

https://download.pytorch.org/models/vgg16-397923af.pth

https://iciar2018-challenge.grand-challenge.org/dataset/

Acknowledgements

Acknowledgment

This work is part of the DAISI project, co-financed by the European Union with the European Regional Development Fund (ERDF) and by the Normandy Region.

Acknowledgment

This work is part of the DAISI project, co-financed by the European Union with the European Regional Development Fund (ERDF) and by the Normandy Region.

Published as: Cao, H., Bernard, S., Heutte, L. and Sabourin, R. Improve the performance of transfer learning without fine-tuning using dissimilarity based multi-view learning for breast cancer histology images. In Proc. of ICIAR 2018, LNCS 10882, Springer, pp. 779-787, 2018. Abstract: Breast cancer is one of the most common types of cancer and leading cancer-related death causes for women. In the context of ICIAR 2018 Grand Challenge on Breast Cancer Histology Images, we compare one handcrafted feature extractor and five transfer learning feature extractors based on deep learning.We find out that the deep learning networks pretrained on ImageNet have better performance than the popular handcrafted features used for breast cancer histology images. The best feature extractor achieves an average accuracy of 79.30%. To improve the classification performance, a random forest dissimilarity based integration method is used to combine different feature groups together. When the five deep learning feature groups are combined, the average accuracy is improved to 82.90% (best accuracy 85.00%). When handcrafted features are combined with the five deep learning feature groups, the average accuracy is improved to 87.10% (best accuracy 93.00%).

histology images terminal node. The prediction is given by the terminal node (or leaf node) in which x has landed. We refer the reader to [START_REF] Biau | A random forest guided tour[END_REF] for more information about this process. Hence if two query points land in the same terminal node, they are likely to belong to the same class and they are also likely to share similarities in their feature vectors, since they have followed the same descending path.

Random Forest Dissimilarity (RFD): the RFD measure is inferred from a RF classifier H, learned from T . Let us firstly define a dissimilarity measure inferred by a decision tree d (k) : let L k denote the set of leaves of the kth tree, and let l k (x) denote a function from X to L k that returns the leaf node of the kth tree where a given instance x lands when one wants to predict its class. The dissimilarity measure d (k) , inferred by the kth tree in the forest is defined as in Equation (A.2): if two training instances x i and x j land in the same leaf of the kth tree, then the dissimilarity between both instances is set to 0, else set to 1.

The RFD measure d (H) consists in calculating the d (k) value for each tree in the forest, and to average the resulting dissimilarity values over the M trees, as in Equation (A.3):

Multi-view learning dissimilarities: For multi-view learning tasks, the training set T is composed of K views:

N , y N )}, k=1..K. Firstly, for each view T (k) , the RFD matrix is computed and noted as {D k H , k = 1..K}. In multi-view learning, the joint dissimilarity matrix can typically be computed by averaging over the Q matrices as in Equation (A.4):

Multi Random Forest kernel SVM (RFSVM): From the joint RFD matrix D H

histology images

The confusion matrix, sensitivity and specificity of our best model are shown in Table A.2. From the results we can see that our model has very high sensitivity on all four classes, and very high specificity too for two classes, i.e. InSitu and Invasive.

Note that the state of the art performance on this dataset is considered to be from [START_REF] Araújo | Classification of breast cancer histology images using Convolutional Neural Networks[END_REF]. In this work, the authors used CNN patch-wise training on a previous version of the dataset with 249 images for training and 20 images for testing (7.4% of the whole dataset as test data). They obtained as best performance an accuracy of 85.00%. In our work, 300 images are used for training and 100 images are used as test data. Hence, even if the results are not directly comparable with [START_REF] Araújo | Classification of breast cancer histology images using Convolutional Neural Networks[END_REF], the accuracy of our best model is 8% higher than the accuracy reported in [START_REF] Araújo | Classification of breast cancer histology images using Convolutional Neural Networks[END_REF] while using 25% of the whole dataset as test data, which is much more than 7.4% in [START_REF] Araújo | Classification of breast cancer histology images using Convolutional Neural Networks[END_REF]. In particular, we propose a random forest based dynamic weighted voting scheme, which personalizes the combination of views for each new patient to classify. The proposed method is validated on several real-world Radiomics problems.

Appendix

applications the embedded feature selection method SVMRFE shows good performance on different Radiomics applications [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF].

A lot of studies have been done on multi-view learning and according to the work of [204], there are three main kinds of solutions: early integration, intermediate integration and late integration. Early integration concatenates in- In [START_REF] Cao | Dissimilarity-based representation for radiomics applications[END_REF], MCS based late integration methods (with simple majority voting) have shown a big potential and a lot of flexibility on Radiomics data. In this work, to further investigate the potential of MCS for Radiomics applications, both static and dynamic combinations are tested. The intuition behind static weighted voting is that different views have different importances for a classification task. While the intuition behind proposing dynamic voting methods is that, due to the heterogeneity among patients, different patients may rely on different information sources. For example, for a patient A, there may be more useful information in one view (e.g. texture or shape features) while for a patient B, there may be more useful information in another view (e.g. intensity or wavelet features). Three dynamic integration methods were considered in the work of [227]: DS (Dynamic Selection), DV (Dynamic Voting), and DVS (Dynamic Voting with Selection). The difficulty in multi view combination is that the number of views is fixed and usually very small. In this case, dynamic selection methods may not be applicable. Hence, we focus on dynamic voting method in this work. However, traditional dynamic voting methods demand a validation dataset [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]. In Radiomics, the data size is too small to have a validation dataset. In the next section, we propose a dynamic voting method based on the random forest dissimilarity measure and the Out-Of-Bag (OOB) measure, without the need of validation dataset. applications I() is an indicator function, which equals to 1 when the condition in the parenthesis is fulfilled and 0 otherwise.

B.3.1 WRF (Static Weighted Voting)

To calculate the weights for static voting, we need a measure to reflect the importance of each view to give a final decision. Usually, the prediction accuracy over a validation dataset can be used for that. However, Radiomics data have very small sample size, and it is impossible to have extra validation data.

Hence we propose to use the OOB accuracy of each random forest H (q) as the static weight W (q) for each view:

When Bagging is used in a random forest, each bootstrap sample used to learn a single tree is typically a subset of the initial training set. This means that some of the training instances are not used in each bootstrap sample (37% in average; see [START_REF] Breiman | Out-of-bag estimation[END_REF] for more details). For a given decision tree of the forest, these instances, called the Out-of-bag (OOB) samples, can be used to estimate its accuracy. To use OOB to measure the accuracy of a random forest, the concept of sub-forest is used. When the forest size is big, all training data have a high probability to be an OOB sample at least once. Hence, for each OOB sample X OOB , the trees that did not use this data as training sample are grouped together as a sub-forest H sub(X OOB ) (which can be seen as a representative of the complete random forest H) to give a prediction on X OOB . The overall accuracy of the sub-forests predictions on all OOB samples is then used as OOB accuracy for a random forest H. We refer the reader to the work of [START_REF] Breiman | Out-of-bag estimation[END_REF] for further information about OOB measure.

B.3.2 GDV (Global Dynamic Voting)

In static voting, we believe that different views have different importances for classification. However, with dynamic voting, we can personalize this importance with an assumption that the importances of views are different for applications To calculate the local weight

local , RFD is used to find the neighborhood θ X of each test instance X by choosing the most n neighbor similar instances in training data. The OOB measure over θ X is then used to calculate the local weight. Unlike in the work of [227] using OOB to measure the individual tree accuracy, here OOB is used to measure the performance of the RF classifier.

With θ X , the local weight can be easily calculated with OOB measure:

The idea of local weight here is similar to OLA (Overall Local Accuracy) used in dynamic selection [START_REF] Cruz | Dynamic classifier selection: Recent advances and perspectives[END_REF]. There are two main differences: firstly, LDV uses the random forest dissimilarity as a distance measure which carries both feature information and class label information while OLA uses Euclidean distance which may suffer from the concentration of pairwise distance [5] in high dimensional space; secondly, OLA requires a validation dataset while LDV does not.

B.3.4 GLDV (Global&Local Dynamic Voting)

From the previous two subsections, we can see that W (q)

global uses global information from all training data and measures the confidence of the classifier. But it has also the risk of being too generalized and lacks of personalized information. On the other hand, W (q) local uses information on the neighborhood of the test sample to give a more personalized measure which can better represent the heterogeneity among cancer patients but may lose the global vision at the same time. Hence we propose a measure that takes both measures into account.

With each H (q) , the global weight W GL is calculated by taking advantage of both global and local information together:

Appendix C

Single view results for the experiment in Chapter 3

C.1 Datasets and protocol

A general description of all datasets can be found in Table C.1. There are in total 21 HDLSS datasets used in this work. All these datasets are publicly available. 15 of these datasets are multi-view data (with #views >1). More details of these datasets can be found in Chapter 2. Six other public single view HDLSS datasets are added. When comparing proposed methods with other distance metrics, the multi-view data are treated as single view by concatenating all the views.

All the methods compared in this section are the same as in Chapter 3 and the parameter settings are the same too. To test the performance of the proposed dissimilarity measure, the classification accuracy is evaluated with KNN following the suggestion in many dissimilarity learning studies. The choice of K is 3 here as in many works in metric learning [130,148,190,243]. A stratified random splitting procedure is repeated 10 times, with 50% of the instances for training, 50% for testing. In order to compare the methods, the mean and standard deviations of accuracy are evaluated over the 10 runs. From the experiment results, it can be seen that all RF based methods are better than learning free method Euclidean distance and classic metric learning method LMNN on HDLSS data. Comparing to classic metric learning methods, RF based measures have the advantage of dealing with high feature dimension efficiently in the subspace without using PCA. Among four RF based measures, proposed RFD I H is the best, which is significantly better than classic RFD. This single view result corresponds to the multi-view result in Chapter 3.