Haijie Fu

Xiaodong Yue

Wei Liu

Thierry Denoeux
email: thierry.denoeux@utc.fr

STABLE CLUSTERING ENSEMBLE BASED ON EVIDENCE THEORY

As an unsupervised ensemble learning strategy, clustering ensemble combines multiple base clusterings into a high-quality one and has achieved successful applications in image analysis and data mining. However, extant clustering ensemble methods are ineffective to handle the data uncertainty in clustering consensus process, which may mislead to poor clustering ensemble results. To tackle the problem, we propose a stable clustering ensemble (SCE) method based on evidence theory (Dempster-Shafer theory) in this paper. Specifically, we construct a belief function of cluster membership to measure the uncertainty and stability of data instances in clustering ensemble and thereby implement the stable clustering ensemble algorithm. We test the proposed stable clustering ensemble method in the tasks of structural data clustering and image segmentation. The experimental results validate the proposed method is effective to process the uncertain data and produce high-quality data clusterings.

Index Terms-Clustering ensemble, stability, evidence theory 1. INTRODUCTION Clustering ensemble methods have been proposed to combine multiple base clusterings into a single one, called the consensus, which aims at producing a more accurate and robust clustering of data [START_REF] Strehl | Cluster ensembles-a knowledge reuse framework for combining multiple partitions[END_REF]. Clustering ensemble has been successfully applied in the areas of image analysis [START_REF] Zhang | Spectral clustering ensemble applied to sar image segmentation[END_REF], computer vision [START_REF] Zhang | Weighted clustering ensemble: A review[END_REF], multimedia [START_REF] Yue | An efficient color quantization based on generic roughness measure[END_REF] and data mining [START_REF] Alexander Topchy | Clustering ensembles: Models of consensus and weak partitions[END_REF].

Although previous research works have achieved a great progress, extant clustering ensemble methods ignored the uncertainty of data in the clustering consensus process, which may mislead to poor clustering ensemble results. Specifically, if a data instance is partitioned unsteadily into different clusters in multiple base clusterings, it is indicated that the data This work was supported by National Natural Science Foundation of China (Serial Nos. 61976134, 61991410, 61991415) and Natural Science Foundation of Shanghai (Serial No. 21ZR1423900).

instance is unstable in the consensus process and has much uncertainty in clustering ensemble. For the tasks of image analysis, the unstable pixels in the clustering ensemble correspond the uncertain regions of images. To improve the performances of data analysis, it is required to formulate and handle the data uncertainty in clustering ensemble.

To tackle the problem, we propose a stable clustering ensemble (SCE) method based on evidence theory (Dempster-Shafer theory) in this paper. In the proposed method, we construct a belief function of cluster membership to measure the uncertainty and stability of data instances in clustering ensemble and thereby implement the stable clustering ensemble algorithm. Based on the stability measure, we can divide data instances into two categories: cluster core and cluster halo [START_REF] Rodriguez | Clustering by fast search and find of density peaks[END_REF]. Cluster core consists of the stable data instances with little uncertainty, which represent the structure of data distribution. In contrast, cluster halo contains the unstable data instances with high uncertainty, which denote the uncertain boundaries of clusters. To implement the stable clustering ensemble, we first detect the cluster core in clustering consensus as the certain fundamental clusters and then gradually distribute the uncertain data instances in the cluster halo into certain clusters. The main contributions of this paper are summarized below.

• Propose a measure of data stability in clustering ensemble based on evidence theory.

• Implement a stable clustering ensemble algorithm with stability measure to handle uncertain data in clustering consensus.

The rest of the paper is organized as follows. Section 2 briefly reviews the foundations of clustering ensemble and evidence theory. In Section 3, we introduce the proposed stable clustering ensemble method in detail, which include the stability measure with evidence theory and the clustering ensemble algorithm. In Section 4, the experiments are conducted to verify the proposed method. Section 5 concludes the paper work.

BACKGROUND

Basics of Clustering Ensemble

Let X = {x 1 , ..., x n } be a data set and Π = {π 1 , ..., π B } denotes B base clusterings generated by multiple clustering procedures. Clustering ensemble aims at combining multiple base clusterings Π into an accurate and robust clustering. In general, clustering ensemble consists of two stages of generating base clusterings and assembling the consensus clustering result.

The diversity and quality of base clusterings are the key factors to affect the performances of clustering ensemble. Three kinds of strategies were investigated to generate diverse and accurate base clusterings, which include: 1) diverse parameter setting strategy [START_REF] Ayad | Cumulative voting consensus method for partitions with variable number of clusters[END_REF] that uses one clustering algorithm with random cluster centers or cluster numbers; 2) diverse algorithm strategy [START_REF] Yang | Temporal data clustering via weighted clustering ensemble with different representations[END_REF] that uses various clustering algorithms to generate diverse base clusterings; 3) diverse feature strategy [START_REF] Tao | From ensemble clustering to multi-view clustering[END_REF] that represents data clusterings in different feature spaces. For assembling the base clusterings, the methodologies can be categorized into following four kinds [START_REF] Iam-On | Comparative study of matrix refinement approaches for ensemble clustering[END_REF]. Feature based approach that transforms the clustering ensemble problem into the clustering of categorical data [START_REF] Carpineto | Consensus clustering based on a new probabilistic rand index with application to subtopic retrieval[END_REF]. Direct approach, which is based on the relabeling process to find the best matched clustering [START_REF] Fischer | Bagging for path-based clustering[END_REF]. Graph-based approach that utilizes the graph representation to solve the clustering ensemble problem [START_REF] Zhou | Clustering ensemble via structured hypergraph learning[END_REF]. Co-association approach that creates the pairwise correlation matrix among data instances to assemble base clusterings [START_REF] Fred | Combining multiple clusterings using evidence accumulation[END_REF].

Preliminaries of Evidence Theory

Evidence theory, also referred to as Dempster-Shafer (D-S) theory or theory of belief functions [START_REF] Shafer | A mathematical theory of evidence[END_REF][START_REF] Denoeux | Representing uncertainty on set-valued variables using belief functions[END_REF] is a theoretical framework for reasoning with partial and unreliable information. Let a variable w taking values in a finite set Ω, a mass function on Ω is defined as a mapping from 2 Ω to [0,1], satisfying the following condition

A∈Ω m (A) = 1. (1)
Each quantity m (A) can be interpreted as the probability that the evidence supports w ∈ A. In particular, m (Ω) is the probability that the evidence tells us nothing about w, i.e., the unknown probability. A subset A of Ω such that m (A) > 0 is called a focal set of m. The mass function for which Ω is the only focal set is said to be vacuous, it represents total ignorance. Given a mass function m, belief bel and plausibility function pl are defined by

bel (A) = φ =B⊆A m (B), (2)
pl (A) = B∩A =φ m (B). (3)
For all A ⊆ Ω, the quantities bel (A) and pl (A) denote the degree of total support in A and the degree that the evidence consistent with A, respectively.

METHODS

Measuring data stability in clustering ensemble

Given B base clusterings for ensemble Π = {π 1 , ..., π B }, the uncertainty of a data instance in the clustering ensemble is related to the stability of the data instance belonging to a cluster under the partitions of different base clusterings [START_REF] Li | Clustering ensemble based on sample stability[END_REF]. Suppose some data instances are partitioned into a cluster in one base clustering, but in other base clusterings, these data instances are distributed into different clusters. It is natural to consider that the hypothesis of these data instances belonging to the same cluster is inconclusive. Next we utilize the evidence theory to measure the stability of belongingness of data to clusters. We define a discernment frame Ω = {c, ¬c} to discern whether data belong to a cluster c or not (¬c). For a data instance x i belonging to a cluster c πg in the base clustering π g , its mass function of cluster membership can be defined as

m πg i (A) =                    t =g πt∈Π xj ∈c πg f (i, j, π t ) (B -1) • |c πg | , A = {c πg } 1 - t =g πt∈Π xj ∈c πg f (i, j, π t ) (B -1) • |c πg | , A = Ω (4
) where |c πg | is the number of data instances in the cluster c πg . f (i, j, π t ) is defined by

f (i, j, π t ) =
1, x i , x j in the same cluster of π t 0, otherwise.

Through accumulating the evidences of data co-occurrence in a cluster from other B -1 base clusterings, m πg i ({c πg }) denotes the probability mass of the instance x i certainly belonging to the cluster c πg in the base clustering π g , and m πg i (Ω) denotes the unknown mass (uncertainty).

Based on the mass function above, we can further formulate the pairwise relationship between data instances using Dempster's combination rule. Suppose Θ = {s, ¬s} is a frame of discernment, in which s denotes a pair of data instances belonging to the same cluster, and ¬s means that they belong to different clusters. For a pair of data instances x i and x j , the belief and uncertainty about the hypothesis that x i and x j belong to the same cluster are defined below. Considering all the base clustering, the average belief and uncertainty about that x i and x j belong to a same cluster are obtained by

bel πg ij ({s}) = m πg i ({c πg }) • m πg j ({c πg }), (6)
m πg ij (Θ) = m πg i (Ω) • m πg j (Ω). (7)
bel ij ({s}) = 1 B B t=1 bel π t i,j ({s}), (8)
m ij (Θ) = 1 B B t=1 m π t i,j (Θ). (9)
In general, if a data instance is certainly assigned to a cluster by most base clusterings, we consider that the instance is stable in the clustering ensemble. Therefore, we measure the stability of a data instance x i through accumulating its probability mass in all base clusterings.

Stability(x i ) = πt∈Π m πt i ({c πt }) (10)
Moreover, based on the stability measure, we can find the highly stable data instances from data sets to form the cluster core. Here we adopt the average stability degree of all data instances as the threshold to select the stable data in clustering ensemble.

Clustering ensemble with stable data

Based on the stability measure of data instances, we can implement the stable clustering ensemble to handle the uncertainty in clustering consensus. The process of the stable clustering ensemble consists of the following two stages.

1. In the fist stage, we select the highly stable data instances as the cluster core and perform clustering of these stable data to capture the certain clusters of data distribution.

2. In the second stage, considering the remained unstable data as cluster halo, we distributed each data instance in the cluster halo into a cluster in the certain clustering obtained in the first stage.

For a selected stable instance x i among n data instances, its belief and uncertainty of pairwise relationship belonging to the same cluster are defined in equation ( 8) and [START_REF] Tao | From ensemble clustering to multi-view clustering[END_REF]. We utilize the pairwise belief and uncertainty to form the feature vector R(i) of x i , R(i) = {bel i1 ({s}), ..., bel in ({s}), m i1 (Θ), ..., m in (Θ)}.

(11) For each pair of stable instances from cluster core x i , x j , we can measure their similarity using cosine measure and construct a evidential similarity matrix (ESM), in which each element is computed as Get a data instance from cluster halo, assign it into the nearest cluster in C; 8:

ESM ij = < R(i), R(j) > < R(i), R(i) > • < R(j), R(j) > . ( 12
)
Update the cluster and remove the instance from the cluster halo; 9: end while 10: return the updated clusters.

Based on ESM of all the stable data instances, we simply utilize a hierarchical clustering (HC) algorithm [START_REF] Johnson | Hierarchical clustering schemes[END_REF] to form the clusters of cluster core. These clusters can be considered as the certain part of the final clustering result. For the remained unstable data instances in cluster halo, we assign each instance into its nearest cluster of stable data instances and update the cluster iteratively until all the unstable data instances are assigned. The process of the stable clustering ensemble is shown in Algorithm 1. Using the algorithm, we can obtain a clustering ensemble result P = {pc 1 , ..., pc k }, in which pc i denotes the ith cluster and k is the cluster number. If it is required to further merge the clusters in P , we can adopt the pairwise similarity measure between clusters pc i and pc j and perform HC algorithm again to merge the clusters in P .

EXPERIMENTS

In the experiments, we implement two tests to validate the superiority of the proposed stable clustering ensemble (SCE) method. The first test aims to verify the effectiveness of the SCE method for data clustering, we perform the SCE method on 10 structural data sets including both synthetic data sets (Flame, 2d-3c-no123, Aggregation, Chainlink, Wingnut) and UCI data sets (Ecoli, Segmentation, Glass, Knowledge Modeling, Yeast), and compare the clustering results with other 6 representative clustering ensemble methods, which include WTQ [START_REF] Iam-On | A link-based approach to the cluster ensemble problem[END_REF], WCT [START_REF] Iam-On | A link-based approach to the cluster ensemble problem[END_REF], CSM [START_REF] Iam-On | A link-based approach to the cluster ensemble problem[END_REF], MCLA [START_REF] Strehl | Cluster ensembles-a knowledge reuse framework for combining multiple partitions[END_REF], CSPA [START_REF] Strehl | Cluster ensembles-a knowledge reuse framework for combining multiple partitions[END_REF] and HGBF [START_REF] Fern | Solving cluster ensemble problems by bipartite graph partitioning[END_REF]. In the second test, we validate the ability of SCE for image analysis, we utilize the SCE method for image segmentation on Berkeley Segmentation Dataset and compare the segmentation results produced by other kinds of clustering methods.

For the experiment implementation, K-means algorithm is used to generate base clusterings and the cluster number of each base clustering is set as √ n, n is the data instance number. In the comparative experiments, we run each algorithm 10 times and present the average results. Besides, the decay factor parameter in the comparative methods WCT, WTQ are set to 0.9.

In the first test on structural data sets, we adopt the wellknown criteria ARI [START_REF] Hubert | Comparing partitions[END_REF] and NMI [START_REF] Strehl | Cluster ensembles-a knowledge reuse framework for combining multiple partitions[END_REF] to evaluate the clustering quality and generate 50 base clusterings for ensemble on each data set. Table 1 and 2 list the ARI and NMI evaluations of clustering results produced by all the comparative clustering ensemble methods. It is obvious that the proposed SCE method achieves the best performance. Besides the structural data, we also test the SCE method on unstructural images. We perform the clustering ensemble methods on the images from Berkeley Segmentation Dataset for image segmentation. Comparing with traditional clustering ensemble methods, SCE can effectively fuse multiple clustering-based segmentation results and detect the uncertain regions based on the stability measure. To illustrate this, we generate 20 segmentations using Chan-Vese method [START_REF] Chan | Active contours without edges[END_REF] with random initial contours. The stability of each pixel in an image is calculated by equation ( 10), Fig. 1 shows the uncertain image regions (marked by gray color) that consist of the unstable pixels in multiple segmentation results. Utilizing different clustering ensemble methods for image segmentation, we evaluate the clustering-based segmentation results by the measurements of BDE [START_REF] Freixenet | Yet another survey on image segmentation: Region and boundary information integration[END_REF], PRI [START_REF] Unnikrishnan | Toward objective evaluation of image segmentation algorithms[END_REF], VOI [START_REF] Meilǎ | Comparing clusterings: an axiomatic view[END_REF] and GCE [START_REF] Martin | A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics[END_REF]. To accelerate the clustering-based segmentation, we compress image pixels to superpixels [START_REF] Achanta | Superpixels and polygons using simple non-iterative clustering[END_REF] for clustering ensemble. Moreover, referring to [START_REF] Lei | Superpixel-based fast fuzzy c-means clustering for color image segmentation[END_REF], we initialize the cluster number from 2 to 6 for each image and determine the optimal cluster number to form the final image segmentation according to the highest PRI. Table 3 lists the detailed evaluations and Fig. 2 presents some comparative segmentation results. We can find that our method produces more precise image segmentations than other clustering methods.

CONCLUSION

To tackle the drawback of handling the data uncertainty in clustering ensemble, we propose a stability measure of data in clustering ensemble based on evidence theory and thereby implement a stable clustering ensemble (SCE) method with the stability measure. The experiments of structural data clustering and image segmentation validate the superiority of the proposed SCE method. Our future work will focus on the acceleration of the stability computation.

If x i

  and x j belong to different clusters, m πg ij ({¬s}) = 1, bel πg ij ({s}) and m πg ij (Θ) are both zero.

Fig. 1 .

 1 Fig. 1. Example of image segmentation based on SCE, (b-c) two different segmentation results, (d) the ensemble segmentation in which uncertain regions are marked by gray color.

Fig. 2 .

 2 Fig. 2. Segmentation results based on different clustering ensemble methods.

  Algorithm 1 Clustering ensemble with stable data Input: Stability degrees stability(x i ) of data instances {x 1 ..., x i ..., x n } among B base clusterings; Output: Clusters of data instances; 1: Average the stability degree as the threshold T ; 2: cluster core = {x i |stability(x i ) > T, i = 1, ..., n}; 3: cluster halo = {x i |stability(x i ) ≤ T, i = 1, ..., n}; 4: Construct the similarity matrix ESM of cluster core; 5: Use HC algorithm on ESM to form clusters C of stable data; 6: while |cluster halo| > 0 do

	7:

Table 1 .

 1 ARI evaluations of comparative clustering methods

	Data sets	MCLA HBGF CSPA CSM WTQ WCT	SCE
	Flame	0.5190 0.4858 0.4514 0.7171 0.6511 0.8081 0.8392
	2d-3c-no123 0.6045 0.5265 0.5547 0.8617 0.8132 0.9109 0.9849
	Aggregation 0.5613 0.5173 0.528 0.9338 0.9563 0.971 0.9920
	Chainlink	0.4276 0.2110 0.1962 0.1988 0.3100 0.3751 0.4784
	Wingnut	0.8276 0.899 0.769 0.8807 0.8304 0.8501 0.9843
	Glass	0.2250 0.1884 0.1523 0.2527 0.1968 0.2098 0.2572
	Ecoli	0.3637 0.2941 0.2816 0.3625 0.4359 0.3619 0.7540
	KM	0.1722 0.1863 0.1849 0.2654 0.2412 0.2574 0.3006
	Yeast	0.0909 0.0695 0.0673 0.0997 0.1075 0.0919 0.1490
	Segmentation 0.4210 0.4283 0.3958 0.4811 0.3577 0.3405 0.4769

Table 2 .

 2 NMI evaluations of comparative clustering methodsData sets MCLA HBGF CSPA CSM WTQ WCT SCE Flame 0.4784 0.4599 0.4295 0.6389 0.5638 0.7284 0.7833 2d-3c-no123 0.6337 0.5849 0.6253 0.8608 0.7944 0.8947 0.9575 Aggregation 0.7681 0.7246 0.7290 0.9666 0.9566 0.9803 0.9884

	Chainlink	0.2891 0.3455 0.0842 0.4318 0.4769 0.5939 0.5025
	Wingnut	0.7697 0.8463 0.6673 0.8267 0.7445 0.7656 0.9478
	Glass	0.3498 0.3160 0.2836 0.3807 0.3570 0.3606 0.3822
	Ecoli	0.5465 0.4938 0.4921 0.5822 0.5763 0.5680 0.7130
	KM	0.2700 0.2956 0.2981 0.3704 0.3331 0.3528 0.4142
	Yeast	0.2023 0.1794 0.1725 0.2092 0.2280 0.2121 0.2507
	Segmentation 0.5397 0.5398 0.5095 0.6059 0.5882 0.5336 0.6418

Table 3 .

 3 Evaluations of image segmentation results

	Methods	PRI ↑	VOI ↓	GCE ↓	BDE ↓
	WTQ	71.10	3.42	43.44	15.49
	WCT	71.38	3.40	42.88	15.36
	CSM	71.34	3.40	42.83	15.57
	MCLA	72.05	3.43	43.72	14.73
	CSPA	71.10	3.54	45.35	15.47
	HBGF	71.64	3.49	44.43	15.11
	SCE	75.81	3.07	36.95	13.81


Wei Liu

Xiaodong Yue

Yufei Chen
email: yufeichen@tongji.edu.cn

Thierry Denoeux
email: thierry.denoeux@utc.fr

Trusted Multi-View Deep Learning with Opinion Aggregation

Multi-view deep learning is performed based on the deep fusion of data from multiple sources, i.e. data with multiple views. However, due to the property differences and inconsistency of data sources, the deep learning results based on the fusion of multi-view data may be uncertain and unreliable. It is required to reduce the uncertainty in data fusion and implement the trusted multi-view deep learning. Aiming at the problem, we revisit the multi-view learning from the perspective of opinion aggregation and thereby devise a trusted multiview deep learning method. Within this method, we adopt evidence theory to formulate the uncertainty of opinions as learning results from different data sources and measure the uncertainty of opinion aggregation as multi-view learning results through evidence accumulation. We prove that accumulating the evidences from multiple data views will decrease the uncertainty in multi-view deep learning and facilitate to achieve the trusted learning results. Experiments on various kinds of multi-view datasets verify the reliability and robustness of the proposed multi-view deep learning method.

Introduction

In real-world applications, data is usually represented with different views, including multiple modalities or various types of features, which leads a growing interest in multiview learning. With the development of the deep learning, most of the existing multi-view learning methods tend to integrate multi-view information with deep neural networks to achieve state-of-the-art performance in various application domains (Wang et al. 2015a;[START_REF] Andrew | Deep canonical correlation analysis[END_REF][START_REF] Wang | Partial multi-view clustering via consistent GAN[END_REF][START_REF] Tao | Adversarial graph embedding for ensemble clustering[END_REF][START_REF] Tian | Contrastive multiview coding[END_REF][START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Sun | Multi-view learning for visual violence recognition with maximum entropy discrimination and deep features[END_REF][START_REF] Zhang | CPM-Nets: Cross Partial Multi-View Networks[END_REF][START_REF] Zhang | Deep Partial Multi-View Learning[END_REF][START_REF] Sun | Multi-view representation learning with deep gaussian processes[END_REF]. However, due to the property differences and inconsistency of multiple data sources, the results learned from multi-view deep learning method may be uncertain and unreliable, because the traditional convolutional neural networks focus on the accuracy of the classifications but ignore the credibility of the results, which makes a great limitation in various kinds of applications, especially safety-critical applications (e.g., medical diagnosis or autonomous driving).

To address this limitation, an uncertainty-aware trusted Multi-view Classification (TMC) method [START_REF] Han | Trusted Multi-View Classification[END_REF]) was proposed recently. TMC focuses on combining different views at an evidence level in terms of the Dempster's rule of combination to produce a reliable classification result. However, it does not guarantee the decrease of the overall uncertainty when integrating the uncertain information extracted from multi-view data and does not consider the "consistency" in multi-view learning for avoiding the conflict across information captured from different views. Moreover, the fusion with Dempster's rule will produce counter-intuitive results [START_REF] Zadeh | Review of Shafer's: A Mathematical Theory of Evidence[END_REF]. Therefore, this has motivated us to revisit the multi-view learning from the perspective of opinion aggregation and thereby develop a trusted multi-view deep learning method.

Opinion aggregation aims at aggregating multiple opinions within a group in support of group decision making, which is the same as the fusion process for multi-view learning. However, the opinions from multiple views about the same domain are always unreliable because of the various sensor qualities or environmental factors, which adds more uncertainty to the decision-making process. A good opinion aggregation process should consist of two necessary parts: 1) a trusted aggregation strategy, which can reduce the overall uncertainty after aggregation, 2) maximization of consistency across views for avoiding conflict between multiple opinions. Therefore, from the perspective of a good opinion aggregation structure, we devise a trusted multiview deep learning method. Within this method, we adopt the evidence theory to represent opinions as beliefs about the truth of propositions under degrees of uncertainty. In this opinion representation, the beliefs mean the evidences support for those class probabilities and uncertainty means the vacuity of evidence, which allows explicit expression of level of trust for the results learned from different views. Then, guided by the mapping between opinions and Dirichlet PDFs, we integrate the opinions in terms of the evidence accumulation, which can increase the evidences support for class probabilities and decrease the vacuity of evidence and thereby increase the reliability of multi-view deep learning results. In summary, our contributions of this paper are:

(1) We construct a trusted multi-view deep learning method through simulating opinion aggregation mechanism to achieve trusted learning results. The proposed method adopts the evidence theory to formulate the opinions as learning results from different data sources and represents the integrated opinion as multi-view learning result through opinion aggregation with evidence accumulation, which can precisely estimate the uncertainty of results in multi-view deep learning. (2) We theoretically prove that accumulating the evidences from multiple data views will decrease the overall uncertainty and prediction error of multi-view learning results, which facilitates to produce trusted and accurate learning results. Moreover, we further extend our method by minimizing the opinion entropy across views for guaranteeing the consistency across multiple views.

(3) We conduct extensive experiments over various kinds of real-world data to validate the effectiveness of the proposed model in accuracy, reliability and robustness.

Related Work

Multi-View Learning: Multi-view learning receives increasing interest in recent years to analyze complex data.

The traditional representative methods are canonical correlation analysis (CCA) [START_REF] Harold | Relations between two sets of variates[END_REF]) and its variants [START_REF] Bach | Kernel independent component analysis[END_REF][START_REF] Hardoon | Sparse canonical correlation analysis[END_REF][START_REF] Wang | Variational Bayesian approach to canonical correlation analysis[END_REF]. CCA maximizes the correlation between different views to find a common representation. Kernel CCA [START_REF] Bach | Kernel independent component analysis[END_REF] develops CCA to nonlinear conditions, which makes the CCA more robust. Sparse CCA [START_REF] Hardoon | Sparse canonical correlation analysis[END_REF] learns sparse representation to reduce the effect of noisy data. BCCA presents [START_REF] Wang | Variational Bayesian approach to canonical correlation analysis[END_REF] a Bayesian model selection algorithm for CCA based on a probabilistic interpretation. Different from CCA, some methods [START_REF] Zhao | Multi-view clustering via deep matrix factorization[END_REF]Zhang et al. 2018b;[START_REF] Liu | Lowrank multi-view learning in matrix completion for multilabel image classification[END_REF] obtain hierarchical representation from multiview data through matrix factorization. Multi-view dimensionality co-reduction (MDcR) [START_REF] Zhang | Flexible multi-view dimensionality co-reduction[END_REF]) applies the kernel matching to regularize the dependence across views. Nonparametric sparse learning method (NSMD) [START_REF] Liu | Nonparametric sparse matrix decomposition for cross-view dimensionality reduction[END_REF]) develops an effective sparse learning method for cross-view dimensionality reduction. Consensus and complementarity based maximum entropy discrimination (MED-2C) (Chao and Sun 2016) proposes a multi-view classification based on the two principles consensus and complementarity. Furthermore, Self-representation is also introduced to better incorporate multi-view information [START_REF] Xie | On unifying multi-view self-representations for clustering by tensor multi-rank minimization[END_REF][START_REF] Xie | Robust kernelized multiview self-representation for subspace clustering[END_REF]. Kernelized version of tensor-based multiview subspace clustering (Kt-SVD-MSC) [START_REF] Xie | On unifying multi-view self-representations for clustering by tensor multi-rank minimization[END_REF] jointly learns self-representation coefficients in mapped high-dimensional spaces. Moreover, with the development of the deep learning, some works [START_REF] Andrew | Deep canonical correlation analysis[END_REF]Wang et al. 2015a[START_REF] Wang | Partial multi-view clustering via consistent GAN[END_REF][START_REF] Tao | Adversarial graph embedding for ensemble clustering[END_REF][START_REF] Tian | Contrastive multiview coding[END_REF][START_REF] Bachman | Learning representations by maximizing mutual information across views[END_REF][START_REF] Sun | Multi-view learning for visual violence recognition with maximum entropy discrimination and deep features[END_REF][START_REF] Sun | Multi-view representation learning with deep gaussian processes[END_REF] [START_REF] Wang | Partial multi-view clustering via consistent GAN[END_REF] or impose prior information [START_REF] Tao | Adversarial graph embedding for ensemble clustering[END_REF]. However, these methods achieve a great performance on multi-view classification, but they rarely consider the reliability of the classification result. Recently, a trusted multiview classification method [START_REF] Han | Trusted Multi-View Classification[END_REF]) has been proposed, which focuses on the uncertainty estimation problem and produces a reliable classification result. Nonetheless, it does not guarantee the decrease of overall uncertainty after fusion of different views and does not consider the consistency across views. Moreover, the fusion with Dempster's rule in TMC will produce counter-intuitive results [START_REF] Zadeh | Review of Shafer's: A Mathematical Theory of Evidence[END_REF]. In contrast, our method explores the consistency between different views from the perspective of opinion aggregation and reduces the overall uncertainty after fusing different opinions, which guides an accurate, robust and trusted result.

Opinion Aggregation: Decision making is a pervasive part of life. Every day we are confronted with deciding between multiple choices. Opinion aggregation aims at aggregating multiple opinions within a group, which is very useful for group decision making. Due to its effectiveness, opinion aggregation has been widely used in various applications [START_REF] Zadeh | A simple view of the Dempster-Shafer theory of evidence and its implication for the rule of combination[END_REF][START_REF] Ding | The utility of linguistic rules in opinion mining[END_REF][START_REF] Liu | Low-quality product review detection in opinion summarization[END_REF][START_REF] Martini | Opinion aggregation and individual expertise[END_REF][START_REF] Iso | Convex Aggregation for Opinion Summarization[END_REF][START_REF] Belluti | Smallmolecule inhibitors/modulators of amyloid-β peptide aggregation and toxicity for the treatment of Alzheimer's disease: a patent review (2010-2012)[END_REF].

Mechanism of Opinion Aggregation

Due to the property differences and inconsistency of data sources, results from multiple views about the same domain may be unreliable, which adds more uncertainty to the decision-making process. For reducing the uncertainty in data fusion to obtain the trusted multi-view deep learning results, we revisit the multi-view deep learning from the perspective of opinion aggregation and thereby implement a trusted multi-view deep learning method. In this section, we will describe the mechanism of opinion aggregation for the proposed trusted multi-view deep learning, as shown in Figure 1. Within this mechanism, we first adopt the evidence theory to formulate the information extracted from different views with neural networks as corresponding opinions (step (1) in Figure 1), which can precisely estimate the uncertainty of results from different views. Then we integrate multi-opinions to obtain a unified opinion with evidence accumulation (step (2) in Figure 1), which can decrease the overall uncertainty. Furthermore, we measure the consistency across opinions based on the opinion entropy (step (3) in Figure 1), which can avoid the conflict between different opinions. Details are described as following.

Opinion Representation under Evidence Theory

Traditional neural classification networks usually use the softmax as the standard output. However, using the softmax only obtains the class probabilities but ignores the reliability of the results. To address this problem, the softmax layer is replaced by an activation layer (i.e. ReLU) to obtain a nonnegative output, termed as evidence [START_REF] Sensoy | Evidential deep learning to quantify classification uncertainty[END_REF] b k + u = 1. Then the probability that the data is assigned to class k is defined by

P (k) = b k + a k u, for k = 1, ..., K.
Typically, all values of the a are set to 1/K when there is no preference over class.

Let the expected probability distribution derived from Dirichlet distribution be equal with the projected probability distribution derived from the opinion in evidence theory. Then we have a mapping between opinion and Dirichlet distribution [START_REF] Jøsang | Subjective Logic: A formalism for reasoning under uncertainty[END_REF],

ω = (b, u, a) ↔ Dir (P |α ) , (1)
where P = (P 1 , ..., P k ) T is the probability that the data is assigned to k th class, α = (α 1 , ..., α k ) T represents the Dirichlet parameters and we have α = e + aK, where e = (e 1 , . . . , e k )

T indicates the amount of support evidence collected from neural network in favor of a sample to be classified into k th class. Noted that, when there is no preference over class, the Dirichlet parameters α = e + 1. Then the belief b and uncertainty mass u are calculated as

b k = e k S = α k -1 S , u = K S , (2)
where

S = K k=1 (e k +1) = K k=1 α k is the Dirichlet strength.
That is, the Dirichlet distribution parametrized over evidence represents the density of such probability assignment, it represents the predictions of the learner as a distribution over possible softmax outputs, which models the second-order probabilities to indicate the uncertainty of the neural network result.

Finally, according to equations 1 and 2, we could translate the output of neural network e = (e 1 , . . . , e k )

T into opinion ω = (b, u, a) (step (1) in Figure 1), which allows us flexibly integrate multiple views for trusted decision making.

Opinion Aggregation with Evidence Accumulation

The opinion of single view has been formalized above, which allows explicit expression of uncertainty degree. Now, we begin to focus on the opinion aggregation with multi-view deep learning. Particularly, we use the evidence accumulation in evidence theory to combine multiple opinions, which can reduce the overall uncertainty. The Definition of opinion aggregation with evidence accumulation is described as below. Definition 1. Opinion aggregation with evidence accumulation. The opinion aggregation with evidence accumulation simply consists of evidence parameter addition. Given a data with M multiple views for the same K-classification problem, we can obtain a set of evidences {e m } M m=1 , collected from M neural networks and a set of opinions {ω m } M m=1 in terms of the equation 2. Then we have an integrated opinion

ω ♦(M ) = ⊕ m=1,...,M (ω m ) = b ♦(M ) , u ♦(M ) , a ♦(M ) . (3) For k = 1, ..., K, we have b ♦(M ) k = e ♦(M ) k S ♦(M ) , u ♦(M ) = 1 - K k=1 b ♦(M ) k , a ♦(M ) k = 1 K . (4) Where S ♦(M ) = K k=1 e ♦(M ) k + 1 is the Dirichlet strength, e ♦(M ) k = M m=1 e m
k represents the process of evidence accumulation.

Following the Definition 1, we can obtain the integrated opinion ω ♦(M ) = b ♦(M ) , u ♦(M ) , a ♦(M ) . The corresponding integrated parameters of the Dirichlet PDF are induced as α

♦(M ) k = e ♦(M ) k + 1.

Measure of Consistency across Opinions

Our method for multi-view fusion has been described, which projects the outputs of neural networks to opinions at evidence level and then combines these opinions in terms of evidence accumulation. However, in some cases, the opinions collected from multiple views are inconsistent. To avoid the conflict between multiple opinions, we introduce a consistency measure named opinion entropy in Definition 2, which can guarantee the consistency across multiple views. The definition of opinion entropy is described as follows. Definition 2. Opinion Entropy across two opinions. Given any view denoted by the opinion ω = (b, u, a) for Kclassification task, the entropy of the opinion [START_REF] Jøsang | Subjective Logic: A formalism for reasoning under uncertainty[END_REF]) is defined as

H(ω) = - K k=1 P k log 2 (P k ), (5)
where

P k = b k + a k u.
Then the opinion entropy between two opinions ω 1 and ω 2 is computed as

E ω 1 , ω 2 = H ω 1 + ω 2 2 - 1 2 H ω 1 - 1 2 H ω 2 , ( 6
)
where

H ω 1 +ω 2 2 = - K k=1 P 1 k +P 2 k 2 log 2 P 1 k +P 2 k 2 .
Trusted Multi-view Deep Learning with Opinion Aggregation

In this section, we will discuss how to train our multiview deep learning network. The neural network can capture the evidence from input to induce a classification opinion. Therefore, the traditional neural network can be naturally transformed into the evidence-based neural network (S ¸ensoy, Kaplan, and Kandemir 2018) with minor changes which only replace the softmax layer with an activation layer (e.g., ReLU) to provide non-negative output, termed as the evidence e = (e 1 , . . . , e k ) T . Accordingly, the parameters of the Dirichlet distribution α = e + 1 can be obtained.

Within the proposed method, for the i th sample x i , the overall loss objective is

L overall (α i ) = L acc (α i ) + λL con (α i ) , (7)
where L acc (α i ) is the prediction loss term, L con (α i ) is the loss of the consistency regulation across views, λ is a weight parameter with the range of [0, 1] to control the weight of consistency loss function. The details of these two loss functions are shown in the following subsection.

Prediction Loss Term

For training example x i , let y i encodes the ground-true class label k by setting y ik = 1 and y ij = 0, ∀j = k. Let Cat y i = k |P i be the likelihood, where P i ∼ Dir (P i |α i ), P i = (P i1 , ..., P ik ) T and the parameters α i = e i + 1. The expected sum of squares loss after the aggregation of a set of opinions {ω m } M m=1 is defined as

L acc (α i ) = L acc α ♦(M ) i = E Pi∼Dir Pi α ♦(M ) i y i -P i 2 2 = K j=1 y 2 ij -2y ij E [P ij ] + E P 2 ij , (8)
where α

♦(M ) i = e ♦(M ) i + 1 and e ♦(M ) i = e 1 i + • • • + e M i
is the process of evidence accumulation, which can increase the amount of support in favor of sample x i to be classified into k th class and decrease the overall uncertainty. Intuitively,

E P 2 ij = E[P ij ]
2 + Var (P ij ), then we get the following easily interpretable form

L acc (α i ) = K j=1 (y ij -E [P ij ]) 2 + Var (P ij ) = K j=1 y ij -p ij 2 Lerr α ♦(M ) ij + p ij 1 -p ij S ♦(M ) i + 1 Lvar α ♦(M ) ij , (9)
where

S ♦(M ) i = K k=1 α ♦(M ) ik
is the Dirichlet strength,

p ij = α ♦(M ) ij S ♦(M ) i
is the expectation of the Dirichlet distribution. It is obvious that the loss aims to achieve the joint goals of minimizing the prediction error L err α ♦(M ) i and the variance L var α ♦(M ) i of the integrated opinions by decomposing the first and second terms. In addition, our loss objective has the following propositions. Proposition 1. By integrating the evidence of the correct label from different views in terms of opinion aggregation with evidence accumulation, the prediction error loss L err α ♦(M ) i will be smaller than the prediction error loss from a single view L err α m i , for m = 1, ..., M .

Proof 1. Let e m ij > 0 be the evidence of the j th class extracted from the m th view classifier for the i th sample with correct label j, e ♦(M ) ij > 0 be the integrated evidence from evidence accumulation of M views. After the opinion aggregation, L err α

♦(M ) i is updated as 1 - α ♦(M ) ij S ♦(M ) i 2 y ij =1 + k =j α ♦(M ) ik S ♦(M ) i 2 y ik =0 , (10)
which is equal with

  1 - α m ij + v =m e v ij S m i + v =m e v ij    2 + k =j    α m ik S m i + v =m e v ij    2 . (11)
Obviously, L err α

♦(M ) i is smaller than L err (α m i ) since   1 - α m ij + v =m e v ij S m i + v =m e v ij    2 < 1 - α m ij S m i 2 (12)
and

k =j    α m ik S m i + v =m e v ij    2 < k =j α m ik S m i 2 . ( 13
)
Proposition 2. By integrating the evidence from multi-view in terms of opinion aggregation with evidence accumulation, we guarantee the decrease of the overall uncertainty.

Proof 2. Let e m i be the evidence captured from the m th view classifier for the i th sample. e ♦(M ) i be the integrated evidence from the evidence accumulation of M views. After the opinion aggregation, the overall uncertainty u ♦(M ) is updated as

u ♦(M ) = 1 - K j=1 e ♦(M ) ij S ♦(M ) i = K S ♦(M ) i = K S m i + v =m K j=1 e v ij ( 14
)
which is smaller than the uncertainty of single view result

u m = K S m i since S m i + v =m K j=1 e v ij > S m i .
These two propositions theoretically guarantee the prediction error and uncertainty of multi-view learning results will decrease with increasing views, which can produce accurate and trusted learning result. Our experimental results can also verify these propositions to validate the effectiveness of proposed method.

Consistency Regulation

We further extend the proposed method by adding a consistency regulation loss which minimizes the opinion entropy across opinions to guarantee the consistency of results between different views (step (3) in Figure . 1). The consistency loss is computed as

Lcon (αi) = M m=1 M v =m E(ω m i , ω v i ) (M -1) , ( 15
)
where 1/(M -1) is used for normalization and E(ω m i , ω v i ) is the opinion entropy described in previous subsection.

Experiments

In this section, we evaluate the proposed method on realworld multi-view datasets and compare it with existing multi-view learning methods. Furthermore, we also provide the uncertainty estimation analysis on noisy data.

Datasets

We conduct experiments on six real-world multi-view datasets as follows: CUB

Compared Methods

We compare our method with several state-of-the-art multiview learning methods as follows:

• DCCA: Deep Canonically Correlated Analysis [START_REF] Andrew | Deep canonical correlation analysis[END_REF]

Implementation Details

For our algorithm, we conduct the fully connected networks for all datasets. The Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF] is used to train the network, where l 2 -norm regularization is set to 1e -5 . We then use 5-fold cross-validation to select the learning rate from 1e -4 , 3e -4 , 1e -3 , 3e -3 . For all datasets, 20% samples are used as test set. We run 10 times for each method to report the mean values and standard deviations. The model is implemented by PyTorch on one NVIDIA TITAN Xp with GPU of 12GB memory.

Performance Evaluation

In this subsection, we conduct two tests to evaluate the performance of our method. The first test is to verify the effectiveness of our method and the second is to overall evaluate the superiority of our method. Effectiveness evaluation. To validate the effectiveness of our multi-view learning method, we first compare the average prediction error for multi-view learning results (shown as red line, termed as V) with average prediction error for each single-view learning result (termed as V1-V6) on all datasets. The experimental results are shown in Figure 2, where the y-coordinate represents the average prediction error of data, the x-coordinate indicates the current epoch in training. On all datasets, the prediction error for multiview (red line) are always smaller than each single-view in proposed method, which proves our method can efficiently reduce the prediction error after integration of multiple views to produce more accurate results. We also theoretically prove this conclusion in the Proposition 1. Furthermore, Figure 2 also demonstrates the convergence of proposed method. Typically, the optimization process is stable, where the loss decrease quickly and converges within a number of iterations.

Comparison with the methods. Then we overall evaluate our algorithm by comparing it with state-of-the-art multiview learning methods in terms of accuracy metric. The detailed experimental results are shown in Table 1. We find that, on all datasets, our method consistently achieves better performance. Taking the results on HMDB as examples, our method improves the accuracy by about 20% compared to the second-best model (TMC) in terms of accuracy, which verifies the improved performance of the proposed method.

Uncertainty Estimation Analysis

Due to the property differences and inconsistency of data sources, the uncertainty estimation becomes more important for the multi-view learning. Therefore, in this subsection, we conduct qualitative experiments to provide some insights for the estimated uncertainty, which can evaluate the uncertainty estimation performance of our method.

Ability of capturing uncertainty. Due to the limitation of pages, in this part, we just show the uncertainty estimation ability of our method on Caltech101 dataset with two views. We first add noise to half of the test samples in one view. Similarly to the work of [START_REF] Geng | Uncertainty-Aware Multi-View Representation Learning[END_REF], the noise vectors (denoted by ) are sampled from Gaussian distribution N (0, I). Then we add these noise vectors multiplied with intensity η to pollute half of the original test samples, i.e., the i th sample xi = x i + η i . Then we obtain a Gaussian kernel density estimation [START_REF] Scott | Multivariate density estimation: theory, practice, and visualization[END_REF] of learned uncertainty shown in Figure 3. We find that the distribution curves of noisy samples (red curves) are nearly overlapped with the curves of clean samples (green curves) when the noise intensity is small (η = 0.1). Then uncertainty of noisy samples grows with increasing noise intensity. This means that the estimated uncertainty is associated with the sample quality, which verifies the uncertainty estimation ability of our method and further guarantees our method can obtain a trusted multi-view result with the decrease of the overall uncertainty after aggregation of multiple views. Overall uncertainty evaluation. To evaluate the overall uncertainty, we add Gaussian noise with the fixed value of noise intensity (η = 0.5) to 50% of the test samples and compare the average uncertainty of multi-view learning results with the minimal average uncertainty among different single-view learning results on all datasets to verify the decrease of the overall uncertainty with the increasing views. The results are shown in Table 2, where the U multi indicates the average uncertainty degree for multi-view results on all datasets,U smin represents the minimal average uncertainty degree among different single-view results. The results indicate the uncertainty for multi-view results are always smaller than each single-view result in proposed method, which proves that our method can produce more reliable multiview deep learning results. We also theoretically prove this conclusion in Proposition 2. Moreover, we also conducted a thorough ablation study to justify the effectiveness of our major technical component, including fusion strategy and related model parameters. Additional comparisons with existing uncertainty-based methods [START_REF] Gal | Bayesian convolutional neural networks with Bernoulli approximate variational inference[END_REF][START_REF] Lakshminarayanan | Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles[END_REF][START_REF] Heo | Uncertainty-aware attention for reliable interpretation and prediction[END_REF]) and comparisons with different types of noise and the analysis of real-world applications are also performed. All of these experiments validate the effectiveness and superiority of our model. Detailed results are provided in Technical Appendix B and C.

Conclusion

In this work, we propose an efficient trusted multi-view deep learning method with opinion aggregation, which can generate trusted classification results on multi-view data. Our method tries to represent the learning results from different data sources as the opinions in evidence theory, which can precisely measure the uncertainty of learning results. By the opinion aggregation with evidence accumulation, our method can reduce the uncertainty of aggregated opinion to generate more reliable multi-view deep learning results. Furthermore, we further extend our method by adding a consistency regulation loss to guarantee the consistency of results between different views. The experimental results validate the effectiveness, reliability and robustness of the proposed multi-view deep learning method.

Technical Appendix

A. Details of the Datasets

We conduct experiments on the following six real-world datasets to evaluate our method. Details of these datasets are as follows:

B. Ablation Study

We conduct a detailed ablation study to clearly demonstrate the effectiveness of our major technical components. We adopt the HMDB data with two views for illustration purpose.

Fusion strategy. In this part, we validate the effectiveness of proposed fusion strategy. Figure 4 compares the results of proposed fusion strategy without consistency regulation for multi-view (orange bar, termed as V) with the results from each single-view (termed as V1, V2). The experimental results confirm the effectiveness of our fusion strategy. Even if we remove the consistency regulation term, our accurate scores are also better than the scores from each single-view. Balance of consistency regulation. Figure 5 shows the results using different fixed consistency regulation ratios (λ in the equation 7) on original HMDB dataset, where λ = 0 means we just use the fusion strategy and do not consider the conflict among multiple views. We can find the consistency regulation has effect on the data when λ = 0.1 or λ = 0.3, but has slight negative effect with high values of λ. We guess there is some noise in one of views, therefore, emphasizing consistency too much during the training process may cause model performance degradation. Therefore, to consider the unknown of the multi-view dataset structure in real-world, we typically set the initial λ = 0, then λ slowly increases from 0 to 1.

C. Additional Comparison Results

In this part, we show four additional comparison results.

Impact analysis of noise type. We conduct an experiment on HMDB dataset to analyze the impact of different noise types. Besides the Gaussian noise, similarly to the work (Zhang et al. 2018a), we consider two types of noise, i.e., the global noise E g and the sample-specific noise E s . Formally, we have the noise E = E s + αE g . For the sample-specific noise, E s , we randomly generate a vector and then select randomly a few columns (50 in experiments), setting the other columns with zeros. While for the global noise, we randomly generate a vector and multiply it with a coefficient α to control the noise magnitude. Then, for the i th sample, we have the xi = x i + ηE i . The same as the previous experiments, we add the noise to half of the test samples in one view. Figure 6 shows the performance with different uncertainty degree of two types of noise, respectively. We can find, on all types of noise, the performance of both multi-view and single-view decreases with increasing of the noise intensity (η for gaussian noise , α for new noise E). The performance of single-view on noisy view data decrease rapidly. However, the performance of our method is quite stable, which means our method is more robust to all types of noise and can alleviate the influence of noisy samples.

Comparison with uncertainty-based deep learning method. To further evaluate the performance of our method, we compare proposed method with several existing [START_REF] Lakshminarayanan | Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles[END_REF], UA+ (uncertainty-aware attention) [START_REF] Heo | Uncertainty-aware attention for reliable interpretation and prediction[END_REF]. Due to some uncertainty-based methods use singleview data, we concatenate the original features of multiple views for all comparison methods. The results shown in Table 3 indicate our method outperforms other methods on all datasets. Taking the results on HMDB, Scene15 as examples, our method improves the accuracy by about 20% and 8% compared to the second-best model (DE) in terms of accuracy, which verifies the superiority of the proposed method.

Comparison with uncertainty-based deep learning method in an end-to-end test. The previous experiments are based on the multiple types of features, to further evaluate the performance of our method on the original multimodal dataset, we conduct an end-to-end test on the original multi-modal dataset Food-101, which consists of images and text descriptions. In this test, we use Inception-V3 network [START_REF] Szegedy | Rethinking the inception architecture for computer vision[END_REF]) pre-trained with ImageNet and Bert [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] as the backbones for image and text respectively. Our method can jointly optimize these two networks according to our loss function with opinion aggregation. For the comparison methods, we concatenate the outputs of these two backbone networks as the input of the classifier. For all algorithms, The Adam optimizer [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF]) is used to train the network, where l 2 -norm regularization is set to 1e -5 . The learning rate is set to 1e -4 . Moreover, 20% samples are used as test set. We run 10 times for each method to report the mean values and standard deviations. The model is implemented by PyTorch on two NVIDIA TITAN Xp with GPU of 12GB memory. The results shown in Table 4 indicate our method also has the good performance in the end-to-end manner.

Real-world application analysis. In our experimental dataset, Food-101 is a noisy large-scale multi-modal dataset. It contains many noise samples, i.e., wrong or missing text descriptions, mismatched images, wrong class labels, or the too many objects in one image. For these noisy samples, our method assigns high uncertainty to them, which means our method is more suitable for the real-world application. The examples with high uncertainty are shown in Figure 7. We can find, Figure (1) has a missing text description described by error code. Figure ( 2) is a mismatched image which should be put into the "apple juic" class not the "apple pie". Figure (3) has a wrong class label which should be the "dinner set" not the "baby back rips". Figure (4) has too many objects. Therefore, it is reasonable that these images are assigned to high uncertainty values by our model.

D. Source Code

The code for this work can be found in the Supplementary Material source code.zip file. Text: So why did I decide to call this an apple pie smoothie? Well for one, I thought it sounded pretty good. And secondly, I pretty much use all the ingredients from an apple pie in this delectable drink. We've got apples, spices, sweetness, and chia seeds. Wait, _ what _ ? Chia seeds aren't in apple pie! Yeah, I know.

Figure 1 :

 1 Figure 1: Illustration of proposed method. (1) The evidence extracted from a neural network is represented as an opinion. (2) Opinion aggregation with evidence accumulation. (3) Maximization of the consistency based on the opinion entropy across views. In this method, assume a 3-classification tasks, the opinion ω from this task is visualized as a point inside a tetrahedron, which in fact is a barycentric coordinate system of four axes. The vertical elevation of the opinion point inside the tetrahedron represents the uncertainty mass u; The distances from each of the three triangular side planes to the opinion point represent the respective belief masses b = (b 1 , b 2 , b 3 )T ; The base rate distribution a is indicated as a point on the base triangular plane. The line that joins the tetrahedron summit and the base rate point represents the director. The projected probability distribution P point is geometrically determined by tracing a projection from the opinion point, parallel to the director, onto the base plane.

  [START_REF] Wah | The caltech-ucsd birds-200-2011 dataset[END_REF]): Caltech-UCSD Birds dataset contains 11788 images and text descriptions from 200 categories of birds. Food-101(Wang et al. 2015b): UMPC Food-101 dataset consists of 86796 images and text descriptions from 101 classes of food. HMDB[START_REF] Kuehne | HMDB: a large video database for human motion recognition[END_REF]: This dataset is one of the largest human action recognition dataset, which consists of 6718 images of 51 categories of actions with two views. Handwritten (van[START_REF] Van Breukelen | Handwritten digit recognition by combined classifiers[END_REF]): This dataset consists of handwritten numerals ('0'-'9') from a collection of Dutch utility maps, the handwritten digits are represented with six views.Cal- tech101 (Fei-Fei, Fergus, and Perona 2004): This dataset consists of 8677 images from 101 classes, which contains two views. Scene15 (Fei-Fei and Perona 2005): Scene15 dataset contains 4485 images from 15 indoor and outdoor scene categories with three views. Details of each dataset are presented in the Technical Appendix A.

Figure 2 :

 2 Figure 2: Prediction error with different epoches.

Figure 3 :

 3 Figure 3: Investigation of our model in capturing data noise. The curves in green and red correspond to distributions of clean and noisy data respectively.

Figure 4 :

 4 Figure 4: Performance evaluation of fusion strategy.

Figure 5 :

 5 Figure 5: Accuracy with different consistency regulation ratios.

  Figure 6: Investigation of our model in capturing deferent types of data noise. The curves in red and blue correspond to accuracy of multi-view and noisy view respectively.

Table 1 :

 1 Comparison with state-of-the-art multi-view learning methods based on accuracy (%).

	Data	DCCA	DCCAE	CPM-NetS	DUA-Nets	TMC	Ours
	CUB	82.12±3.03	85.39±1.28	89.32±0.38	81.13±1.67	91.00±2.36	95.43±0.20
	HMDB	46.83±0.77	49.12±1.00	63.32±0.43	62.73±0.23	65.98±2.92	88.20±0.58
	Scene15	54.77±1.13	55.03±0.34	67.29±1.01	68.23±0.11	67.79±0.21	75.57±0.02
	Caltech101	89.00±0.15	90.11±0.21	90.35±2.12	93.83±0.34	92.93±0.20	94.63±0.04
	Handwritten	97.55±0.38	97.25±0.42	94.55±1.36	98.10±0.32	98.51±0.13	99.75±0.00
	Food-101	81.68±2.23	85.30±0.31	86.45±1.51	87.73±2.27	90.21±1.20	93.75±0.32

Table 2 :

 2 Overall uncertainty evaluation.

	Uncertainty	CUB	Caltech101	HMDB
	U smin	0.4896	0.5047	0.5995
	U multi	0.2255	0.4038	0.4577
	Uncertainty Scene15 Handwritten Food-101
	U smin	0.4652	0.4337	0.6352
	U multi	0.3433	0.2574	0.4378

Table 3 :

 3 Comparison with uncertainty-based deep learning methods based on accuracy (%).

	Data	MCDO	DE	UA+	Ours
	CUB	89.83±0.43	90.39±0.28	87.51±1.23	95.43±0.20
	HMDB	65.12±2.21	69.35±1.01	67.52±1.31	88.20±0.58
	Scene15	66.13±1.54	67.12±0.21	64.98±2.61	75.57±0.02
	Caltech101	90.04±0.12	93.11±0.10	88.96±1.24	94.63±0.04
	Handwritten	97.73±0.26	98.75±0.05	97.55±0.32	99.75±0.00
	Food-101	81.56±1.33	87.30±0.43	89.70±0.21	93.75±0.32
	uncertainty-based deep learning method, including MCDO
	(Monte Carlo Dropout) (Gal and Ghahramani 2015), DE
	(Deep Ensemble)

Table 4 :

 4 Comparison with uncertainty-based deep learning in an end-to-end test based on accuracy (%).

	Data	MCDO	DE	UA+	Ours
	Food-101	78.56±2.11	86.15±0.23	87.64±1.02	92.25±0.75

Acknowledgments

This work was supported by National Natural Science Foundation of China (Serial Nos. 61976134, 61991410, 62173252) and Natural Science Foundation of Shanghai (NO. 21ZR1423900).

(2)

(3) (4)