Asma Trabelsi
email: trabelsyasma@gmail.com

Zied Elouedi
email: zied.elouedi@gmx.fr

Eric Lefevre
email: eric.lefevre@univ-artois.fr

Zied Elouedi Larodec

Eric Lefevre Ea

Comparing dependent combination rules under the belief classifier fusion framework

Keywords: Belief function theory, combination rules, dependent information sources, multiclassifier fusion framework

Data fusion, within the evidence theory framework, consists of obtaining a unique belief function by the combination of several belief functions induced from various information sources. Considerable attention has been paid to combination rules dealing with beliefs induced from non-distinct information sources. The most popular fusion rule is the cautious conjunctive rule, proposed by Denoeux. This latter has the empty set, called also the conflict, as an absorbing element. In fact, the mass assigned to the conflict tends toward 1 when applying a high number of the cautious conjunctive operator, and consequently, the conflict loses its initial role as an alarm signal, announcing that there is a kind of disagreement between sources. This problem has led to the introduction of the normalized cautious rule which totally ignores the mass assigned to the conflict. An intermediate rule between the cautious conjunctive and the normalized cautious rules, named the cautious Combination With Adaptive Conflict (cautious CWAC), has been proposed to preserve the initial alarm role of the conflict. Despite this diversification, no great effort has been devoted until now to

Introduction

Pattern recognition is an exciting field in machine learning which has been widely studied over the past few years to solve classification problems. Among the proposed solutions, some have relied on an advanced method named multiple classifier systems or ensemble classifier [START_REF] Wolpert | Stacked generalization[END_REF][START_REF] Kittler | On combining classifiers[END_REF][START_REF] Dietterich | An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization[END_REF][START_REF] Reformat | Building ensemble classifiers using belief functions and OWA operators[END_REF]. The underlying idea behind the use of ensemble classifier is that different classifiers can provide higher classification accuracy about patterns to be classified [START_REF] Sharkey | Combining diverse neural nets[END_REF][START_REF] Kuncheva | An experimental study on diversity for bagging and boosting with linear classifiers[END_REF][START_REF] Kuncheva | Measures of diversity in classifier ensembles and their relationship with the ensemble accuracy[END_REF]. Ensemble classifier can take different forms: A such ensemble can be constructed by learning different machine learning classifiers using all the training examples (Trabelsi, Elouedi, & Lefevre, 2015a[START_REF] Trabelsi | Classifier fusion within the belief function framework using dependent combination rules[END_REF], by training the same classifier on different subsets of training instances [bagging [START_REF] Breiman | Bagging predictors[END_REF], boosting [START_REF] Freund | A decision-theoretic generalization of on-line learning and an application to boosting[END_REF] and cross-validation] or on different subsets of features [Random Subspaces [START_REF] Ho | The random subspace method for constructing decision forests[END_REF][START_REF] Bertoni | Biomolecular cancer prediction with random subspace ensembles of support vector machines[END_REF][START_REF] Kuncheva | Random subspace ensembles for FMRI classification[END_REF], Rotation Forests [START_REF] Rodriguez | Rotation forest: A new classifier ensemble method[END_REF]] and also by introducing randomness into a learning procedure [e.g., using different initial weights for the Nearest Neighbor classifier, picking different splits for the decision tree classifier [START_REF] Dietterich | An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization[END_REF][START_REF] Geurts | Extremely randomized trees[END_REF], etc]. Up to now, various combination approaches have been proposed to combine diverse classifiers such as voting-based systems [START_REF] Ruta | Classifier selection for majority voting[END_REF], plurality [START_REF] Hansen | Neural network ensembles[END_REF], fuzzy integrals [START_REF] Cho | Combining multiple neural networks by fuzzy integral for robust classification[END_REF][START_REF] Pizzi | Aggregating multiple classification results using fuzzy integration and stochastic feature selection[END_REF], classifier local accuracy [START_REF] Woods | Combination of Multiple Classifiers Using Local Accuracy Estimates[END_REF], Bayesian theory (L. [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF][START_REF] Kuncheva | An experimental study on diversity for bagging and boosting with linear classifiers[END_REF], multilayered perceptrons [START_REF] Huang | The combination of multiple classifiers by a neural network approach[END_REF] and belief function theory (L. [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF]; P. [START_REF] Xu | Evidential logistic regression for binary SVM classifier calibration[END_REF][START_REF] Xu | Evidential calibration of binary SVM classifiers[END_REF][START_REF] Mercier | Fusion of multi-level decision systems using the Transferable Belief Model[END_REF][START_REF] Le | Combining classifiers for word sense disambiguation based on Dempster-Shafer theory and OWA operators[END_REF][START_REF] Quost | Pairwise classifier combination using belief functions[END_REF][START_REF] Denoeux | Conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence[END_REF][START_REF] Bi | The combination of multiple classifiers using an evidential reasoning approach[END_REF][START_REF] Reformat | Building ensemble classifiers using belief functions and OWA operators[END_REF]. The belief function theory, originated from the work of [START_REF] Dempster | Upper and Lower probabilities induced by a multivalued mapping[END_REF] and [START_REF] Shafer | A mathematical theory of evidence[END_REF], is regarded as a convenient method for representing and managing different kinds of imperfect data and has proved to be an efficient tool for merging. Thanks to its combination rules, a member of classifiers such that their outputs are expressed in terms of mass functions (Al-Ani & Deriche, 2002) can be combined. The basic rule of combination is Dempster's rule also called orthogonal sum. This latter requires a normalization task in order to maintain the basic properties of the belief functions [START_REF] Lefevre | Belief function combination and conflict management[END_REF][START_REF] Yen | Generalizing the Dempster-Shafer Theory to Fuzzy Sets[END_REF]. [START_REF] Zadeh | A Simple View of the Dempster-Shafer Theory of Evidence and Its Implication for the Rule of Combination[END_REF] has proved that this normalization can involve counterintuitive behaviors in the case where the initial conditions of the Dempster rule are not respected (e.g., where the information sources to be combined are not reliable). Thus, several combination rules have been developed to address this problem such as Yager's rule (1987), Dubois and Parde's rule (1988), Smets's rule (1990a) and Murphy's rule (2000). However, these rules can only be used when the combined information sources are independent. A source is presumed to be independent on another one when its belief knowledge does not affect the belief of the other one. Quite recently, considerable attention has been paid to the combination of beliefs coming from dependent sources of information such as the cautious rule and its unnormalized version [START_REF] Denoeux | The cautious rule of combination for belief functions and some extension[END_REF], the bold rule [START_REF] Denoeux | Conjunctive and disjunctive combination of belief functions induced by nondistinct bodies of evidence[END_REF], the cautious Combination With Adaptive Conflict rule (cautious CWAC) [START_REF] Boubaker | Conflict management with dependent information sources in the belief function framework[END_REF], the t-norms-based rules (2008) and Cattaneo's rule ( 2003), etc. An information concerning the independence of sources or not guides the choice of the type of combination rules to be used. Nevertheless, the selection of the most convenient rule among several alternatives is crucial, but the criteria to be based on have not been deeply reported until now. Thus, in this paper, we suggest to evaluate and compare combination rules within the classifier fusion framework to find out the most adequate rule. More concretely, we evaluate and compare combination rules dealing with non-independent information sources, in particular the cautious conjunctive rule, the normalized cautious rule and the cautious CWAC rule. This choice is justified by the fact that these alternative rules mainly differ according to the way of managing the mass assigned to the empty set called also conflict. Indeed, the cautious conjunctive rule, which is the first proposed rule to deal with non-independent information sources, does not keep the initial alarm role of the conflict owing to the use of the conjunctive operator that has the empty set as an absorbing element (Lefevre & Elouedi, 2013). In view of this problem, the normalized cautious rule, obtained by the use of the Dempster operator, has been proposed to ignore completely the value of the conflict yielded by combining pieces of evidence. On the other hand, the cautious Combination With Adaptive Conflict rule (cautious CWAC), which is defined by an adaptive weighting between the cautious conjunctive and the normalized cautious rules, gives the conflict its initial role as an alarm signal indicating that there is a sort of disagreement between sources [START_REF] Boubaker | Conflict management with dependent information sources in the belief function framework[END_REF]. Therefore, in this investigation, we propose to compare the cautious CWAC rule with the cautious conjunctive and the normalized cautious rules according to some proposed assessment criteria. The remaining of this paper is organized as follows: We provide in Sect 2 a brief overview of the fundamental concepts of the belief function theory. We outline various combination rules in Sect 3. We present the procedure of combining multiple classifiers within the belief function framework in Sect 4. Sect 5 is dedicated to highlighting our comparative approach. Our experiments and results are given in Sect 6. The conclusion is reported in Sect 7.

Fundamental concepts of the belief function theory

The belief function theory is regarded as a practical and efficient framework for representing and managing uncertain knowledge. This section briefly recalls some basic concepts of this theory.

Information representation

Let Θ be a finite non-empty set of N elementary events related to a given problem. These events are assumed to be exhaustive and mutually exclusive. Such set Θ , called the frame of discernment, is defined as:

Θ = {θ 1 , θ 2 , . . . , θ N } (1)
The agent beliefs on 2 Θ are represented by the so-called basic belief assignment (bba), named originally by [START_REF] Shafer | A mathematical theory of evidence[END_REF] basic probability assignment (bpa) . The bba is de-fined as follows:

m : 2 Θ → [0, 1] m( / 0) = 0 and ∑ A⊆Θ m(A) = 1 (2)
The amount m(A), called basic belief mass (bbm), states the part of belief that exactly supports the event A. All the subsets A of Θ such that m(A) is strictly positive are refereed to the focal elements of m.

In Shafer's original work (1976), the mass assigned to the empty set should have a null value (i.e., impossible proposition). Such bba is called a normalized basic belief assignment. However, Smets (1990b) allows unnormalized bba and interprets m( / 0) as the amount of conflict between the pieces of evidence or as the part of belief given when the true value is not included in the frame of discernment.

As well, it is possible to transform any unnormalized bba, denoted by m , into a normalized one, denoted by m. The normalization process is defined by:

m(A) = 0 if A = / 0 k.m (A) otherwise (3)
where k -1 = 1-m( / 0) is called the normalization factor.

From the bba m, a belief function ,bel, a plausibility function pl and a communality function q are defined, respectively, as:

bel(A) = ∑ / 0 =B⊆A m(B) (4) pl(A) = ∑ A∩B = / 0 m(B) (5)
and

q(A) = ∑ B⊇A m(B) (6)
The quantity bel(A) represents the total mass of belief that supports the event A, while pl(A) quantifies the maximum possible support that could be given to a subset A of the frame of discernment Θ . On the other hand, the communality q(A) stands for the sum of masses allocated to the supersets of A . It is important to note that m, bel, pl and q are regarded as various expressions of the same information [START_REF] Denoeux | Reasoning with imprecise belief structures[END_REF].

Special bbas

In this subsection, we will focus on some special cases of bbas used for expressing particular situations of uncertainty. A vacuous bba is a normalized bba that has Θ as its unique focal element [START_REF] Shafer | A mathematical theory of evidence[END_REF]. Such bba, representing the state of total ignorance, is defined as follows:

m(Θ ) = 1 and m(A) = 0 ∀ A = Θ (7)
A categorical bba is a normalized bba that has a unique focal element A different from the frame of discernment Θ . It is defined as follows:

m(A) = 1, for some A ⊂ Θ (8) m(B) = 0, for B ⊆ Θ , B = A
A certain bba is a particular case of the categorical belief function where its unique focal element is a singleton. This bba represents the state of total certainty. It is defined as follows:

m(A) = 1 for some A ⊂ Θ and |A|=1 (9) m(B) = 0 for all B ⊆ Θ and B = A
A bba is called simple support function (ssf) if it has at most two focal elements: the frame of discernment Θ and a strict subset of Θ called the focus of the ssf [START_REF] Smets | The canonical decomposition of a weighted belief[END_REF]. A simple support function is defined as follows [START_REF] Smets | The canonical decomposition of a weighted belief[END_REF]:

m(X) =      w if X = Θ 1 -w if X = A for some A ⊆ Θ 0 otherwise (10)
In simple terms, such ssf can be written as A w where A is the focus and w ∈ [0,1].

A bba is named dogmatic only if the frame of discernment Θ is not a focal element (m(Θ ) = 0); otherwise it is called non-dogmatic (m(Θ ) > 0).

Discounting

The combination of bbas induced from not fully reliable information sources may lead to a non-informative bba. However, a method of discounting seems to be imperative before combining sources. Let m be a bba induced from an information source with a reliability rate (1α) [START_REF] Shafer | A mathematical theory of evidence[END_REF]. The discounted bba m α is obtained as follows:

m α (A) = (1 -α)m(A) ∀ A ⊂ Θ . ( 11
) m α (Θ ) = α + (1 -α)m(Θ )
where the coefficient α ∈ [0,1] is called the discounted factor.

Distance between two bbas

The distance measures of evidence represent the degree of dissimilarity between pieces of evidence. One of the wellknown distances is the one proposed by Jousselme. The [START_REF] Jousselme | A new distance between two bodies of evidence[END_REF] distance between two bbas m 1 and m 2 is found as follows:

d(m 1 , m 2 ) = 1 2 (m 1 -m 2 ) T D(m 1 -m 2 ) ( 12
)
where D is the Jaccard similarity measure defined by:

D(A, B) =      1 if A=B= / 0 |A ∩ B| |A ∪ B| ∀ A,B ∈ 2 Θ (13)

Pignistic transformation

To make a decision within the belief function framework, [START_REF] Smets | The Transferable Belief Model for quantified belief representation[END_REF] proposes to transform beliefs into a probability measure called the pignistic probability and denoted by BetP. It is defined as follows:

BetP(A) = ∑ B⊆Θ |A ∩ B| |B| m(B) 1 -m( / 0) ∀ A ∈ Θ (14)
3 Combining information sources Data fusion consists in obtaining a relevant information by the combination of several imperfect data (uncertain, imprecise and incomplete) [START_REF] Lefevre | Belief function combination and conflict management[END_REF]. The belief function framework provides numerous tools for ensuring fusion. Some of them treat the problem of independent information sources, while others assume sources of information combined to be dependent. Two information sources are dependent in the case where the knowledge of the belief induced by one source affects the belief of the other source; otherwise these sources are independent [START_REF] Denoeux | Belief Functions: Theory and Applications[END_REF].

Combining independent pieces of evidence

As mentioned earlier, there exist several combination rules assuming items of evidence combined to be independent.

In this subsection, we emphasize on the conjunctive rule [START_REF] Smets | The application of the Transferable Belief Model to diagnostic problems[END_REF], the Dempster's rule (1967) and the CWAC rule (Lefevre & Elouedi, 2013).

The conjunctive rule, proposed by [START_REF] Smets | The application of the Transferable Belief Model to diagnostic problems[END_REF], is used to combine two bbas provided by reliable and distinct information sources. The resulting bba, denoted m 1 ∩ m 2 , is defined by:

(m 1 ∩ m 2 )(A) = ∑ B,C⊆Θ :B∩C=A m 1 (B).m 2 (C) (15)
The mass assigned to the empty set (m 1 ∩ m 2 ( / 0)) quantifies the degree of disagreement between the two combined sources. In the case where the number of information sources increases, the conflict tends toward 1 and consequently loses its initial role as an alarm signal reflecting the existence of disagreement between sources.

The Dempster rule, based on the orthogonal sum, is a normalized version of the conjunctive rule where the mass of the empty set must be reallocated over all focal elements in the case where m 1 ∩ m 2 ( / 0) = 0 thanks to a normalization factor, denoted k [START_REF] Shafer | A mathematical theory of evidence[END_REF]. This rule, assuming pieces of evidence combined to be reliable and distinct, is defined as follows:

(m 1 ⊕ m 2 )(A) = k(m 1 ∩ m 2 )(A) (16)
and

(m 1 ⊕ m 2 )( / 0) = 0 (17)
where

k -1 = 1 -(m 1 ∩ m 2 )( / 0) (18)
The conjunctive and the Dempster rules are commutative, associative but not idempotent which make them unusable when sources are non-distinct.

Authors, in [START_REF] Lefevre | How to Preserve the Conflict As an Alarm in the Combination of Belief Functions[END_REF], have proposed the CWAC combination rule which is defined by an adaptive weighting between the conjunctive and the Dempster rules. This adaptive weighting offers an effective way to obtain the same behavior as the conjunctive rule when the bbas are contradictory and the same behavior as the Dempster rule when the bbas are similar. Assume we have M bbas, denoted as m 1 ,. . . , m M , the result of their combination using the CWAC operator is noted as m ↔ and is defined as follows:

m ↔ (A) = Dm ∩ (A) + (1 -D)m ⊕ (A) (19)
and

m ↔ ( / 0) = 1 when m ∩ ( / 0) = 1 (20) with m ∩ (A) = ( ∩ i m i )(A), m ⊕ (A) = (⊕ i m i )(A) and D = max i, j [d(m i , m j )] is the Jousselme distance between m i and m j ∀ i, j ∈ [1,M].
The CWAC rule is commutative, not associative and non-idempotent.

Combining dependent pieces of evidence

By against of the conjunctive operator, the cautious conjunctive rule, proposed by Denoeux and denoted by ∧ , is used to combine pieces of evidence induced from reliable dependent information sources based on the conjunctive canonical decomposition stated by [START_REF] Smets | The canonical decomposition of a weighted belief[END_REF]. Let m 1 and m 2 be two non-dogmatic bbas, the result of their combination, denoted by m 1 ∧ m 2 , is given as follows [START_REF] Denoeux | The cautious rule of combination for belief functions and some extension[END_REF]:

m 1 ∧ m 2 (A) = ∩ A⊂Θ A w 1 (A)∧w 2 (A) (21)
where w 1 (A) ∧ w 2 (A) corresponds to the weight function of a bba m 1 ∧ m 2 and ∧ represents the minimum operator. The weights w(A) for every A ⊂ Θ can be obtained from the commonalities as follows:

w(A) = ∏ B⊇A q(B) (-1) |B|-|A|-1 . ( 22
)
It is important to note that information sources may be dependent without being identical (e.g. experts may observe different but correlated variables). As the cautious conjunctive rule inherits the absorbing effect of the conjunctive rule, the conflict may lose its initial alarm role when we apply a significant number of cautious conjunctive combinations.

The normalized version of the cautious conjunctive rule, denoted by ∧ * , is inspired from the behavior of the Dempster operator in order to overcome the value of the conflictual mass [START_REF] Denoeux | The cautious rule of combination for belief functions and some extension[END_REF]. It is defined as:

m 1 ∧ * m 2 = ⊕ / 0 =A⊂Θ A w 1 (A)∧w 2 (A) (23)
We thus have:

m 1 ∧ * m 2 (A) = k.m 1 ∧ m 2 (A) (24) with k -1 = 1 -m 1 ∧ m 2 ( / 0) and m 1 ∧ * m 2 ( / 0) = 0.
This rule has the same properties as its unormalized version counterpart: it is commutative, associative and idempotent.

An intermediate rule between the cautious conjunctive and the normalized cautious rules has been developped by [START_REF] Boubaker | Conflict management with dependent information sources in the belief function framework[END_REF] under the name of cautious CWAC rule and denoted by m • . This proposed rule is defined by an adaptive weighting between the cautious conjunctive rule and its unnormalized version in order to manage the conflict induced from the combination of several information sources. As a consequence, this latter not only ensures the combination of dependent pieces of evidence but also keeps the conflict as an alarm signal announcing that there is a kind of disagreement between sources. Assuming there are M bbas, denoted as m 1 ,...,m i ,... , m M , the result of the cautious CWAC operator is set to:

∀ A ⊆ Θ , m ∧ ( / 0) = 1; m • (A) = Dm ∧ (A) + (1 -D)m ∧ * (A) (25) with D=max i, j [d(m i , m j )] is the Jousselme distance measure between m i and m j ∀ i, j ∈ [1,M].
The cautious CWAC rule is commutative but not associative. It is idempotent only in case where D is equal to 0.

The combination of multiple classifiers within the belief function framework

An increasing interest in the classification process has relied on the so-called ensemble classifier systems [START_REF] Dietterich | An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, boosting, and randomization[END_REF][START_REF] Kittler | On combining classifiers[END_REF][START_REF] Wolpert | Stacked generalization[END_REF]. These kinds of systems are based on the paradigm of combining the outputs of a member of individual classifiers in order to achieve the best classification accuracy. The belief function framework has proved to be an efficient tool for ensuring classifier fusion (Al-Ani & Deriche, 2002). Authors, in [START_REF] Xu | Methods of combining multiple classifiers and their applications to handwriting recognition[END_REF], have shown that the procedure of merging the outputs of numerous classifiers, when using the framework of belief functions, comprises two distinct levels. The first one consists of building mass functions from classifiers' outputs, whereas the second one is dedicated to combining these mass functions through a combination rule R to finally give an improved information decision D F (see Fig 1). One of the most important key points within the belief function framework concerns the assignment of masses. Therefore, several assignments have been discussed based on the performance of classifiers in terms of recognition rate (correct answer), substitution rate (wrong answer) and rejection rate which are computed from a confusion matrix. Let C = {C 1 , . . . ,C M } be a set of M classifiers, and let Θ = {θ 1 , . . . ,θ K } be a set of K class labels. Each classifier takes as input a pattern test x and outputs either a class label θ k or a rejection class θ k+1 . The confusion matrix M i relative to a classifier C i is given by: -The recognition rate:

M i = n i 11 • • • n i 1K n i 21 • • • n i 2K . . . . . . n i K1 • • • n i KK n i (K+1)1 • • • n i (
R i = ∑ K k=1 n i kk n i (26)
-The substitution rate:

S i = ∑ K k=1 ∑ K l=1;l =k n i kl n i (27)
-The rejection rate:

T i = 1 -(R i + S i ) (28)
There exist several assignments based on the performance rates of classifiers [START_REF] Mercier | Fusion of multi-level decision systems using the Transferable Belief Model[END_REF]) such as the Xu's assignment (1992), the reliability assignment [START_REF] Johansson | A Study on Class-Specifically Discounted Belief for Ensemble Classifiers[END_REF], the calibration method (P. [START_REF] Xu | Evidential logistic regression for binary SVM classifier calibration[END_REF]. In this paper, we propose to model each classifier output using the reliability assignment. Suppose that the classifier C i outputs a class label θ k , the basic belief assignment is computed as follows:

m i (θ k ) = R i (29) m i (Θ ) = 1 -R i
The output results associated with M classification methods are different; by the way, we will have M number of mass functions for each test pattern. In the case where the outputs of all classifiers are converted into bbas, we move on to the next step which consists of the classifier fusion through some combination rules provided by the belief function framework.

Classifier fusion

This level concerns the problem related to the question "How to aggregate the results from a set of classifiers such a way that we obtain the best result?". As there exist several combination rules within the belief function framework, the choice of the most efficient one becomes really a challenging task and it still requires non-trivial effort. In a previous work (Trabelsi, Elouedi, & Lefevre, 2015a), we have proved that the CWAC rule is the well-suited rule in the case of independent classifiers. In this paper, we propose to carry out a comparative study which allows to select the best fusion rule among the cautious conjunctive, the normalized cautious and the cautious CWAC ones.

Comparative approach

Let us remind that our purpose is to compare the cautious CWAC rule with the cautious conjunctive and the normalized cautious ones. To achieve our goal, we suggest to resort on the distance and the PCC criteria. The distance criterion, corresponding to the Jousselme distance between the testing instance's bba and its real class, is used to compare the cautious CWAC rule with the cautious conjunctive one, whereas the PCC criterion, representing the percent of the correctly classified instances, is used to compare the cautious CWAC rule with the normalized cautious one.

-For comparing the cautious CWAC rule with the cautious conjunctive one, we should proceed as follows:

1. The real class θ j of all test instances must be transformed into a mass function as follows:

m r ({θ j }) = 1 (30) m r (A) = 0 ∀ A ⊆ Θ and A = {θ j }
2. Subsequently, we compute for each instance the Jousselme distance between the mass function relative to its real class (m r ) and the mass function obtained by combining bbas induced from M classifiers. 3. At last, we sum the Jousselme distances produced by all test instances to get a total one. 4. The most adequate fusion rule is the one that has the minimum value of total distance. -The comparison of the cautious CWAC rule with the normalized cautious one requires the use of 3 variables n 1 , n 2 and n 3 which, respectively represent the number of well-classified, misclassified and rejected instances. Whence, for each combination rule, we suggest the following stages: 1. At first, we have set tolerance threshold values to be S = {0.1, 0.2, . . . , 1}. Then, for any test pattern, we verify the mass assigned to the empty set m( / 0). In the case where m( / 0) is greater than the threshold s, the classifier rejects instance in place of misclassifying it and consequently, the number of rejected instances n 3 must be incremented. On the contrary, if m( / 0) is less than the threshold s, we calculate the pignistic probability (BetP) to make a decision about the class to be chosen. Besides, the actual class will be compared to the real one: We increment n 1 if there are similar and n 2 if they are different. 2. Once we have calculated our well-classified, misclassified and rejected instances, we calculate then the PCC for each threshold s ∈ S as follows:

PCC = n 1 n 1 + n 2 * 100 (31)
3. Finally, we compute the average of all PCCs to obtain a mean PCC. 4. The best combination rule is the one that has the highest value of mean PCC.

Note that the use of two evaluation criteria is justified by the fact that the mass assigned to the empty set is almost high for the cautious conjunctive rule. In such a case, applying the PCC criterion may cause a high number of rejected instances leading to erroneous results. In addition, the mass attributed to the empty set, when using the normalized cautious rule, is null, and consequently, the Jousselme distance, which mostly tends toward 0, may adversely affect the results.

Implementation and simulation

In this section, we detail the different experimentation tests that we have employed to compare the cautious CWAC rule with both conjunctive and normalized cautious rules under the classifier fusion framework.

Experimental settings

To ensure the implementation of our approach, we have developed our program with MATLAB V7.13 (R2013a). We have further performed our experiments on several real databases obtained from the UCI. repository (P. [START_REF] Murphy | UCI repository databases[END_REF]. A brief description of these databases is presented in Table 1 where # Instances, #Attributes and #Classes denote, respectively, the total number of instances, the number of attributes and the number of classes. Although the classes number of all chosen databases is equal to either two or three, our alternative combination rules can also support a class number greater than three. We have carried out experiments using four popular machine learning algorithms implemented in Weka [START_REF] Hall | The WEKA data mining software: an update[END_REF]; Naive Bayes (NB), k-Nearest Neighbors (k-NN), Decision Tree (DT) and Neural Network (NN) algorithms. We have run these Weka classifiers on Matlab based on the principle of the leave-one-out cross validation method which divides a data set with N instances into N-1 parts for training and the remaining instance for testing. This process is repeated N times where each instance is used once as a test set. As a result, we get N test patterns with their predicted class labels from each classifier.

Experimental results

Within the framework of classifier fusion, several factors including the number of classifiers, the performance of individual classifiers and the dependency between classifiers may affect the combination rule behavior. The influence of classifier dependencies on the performance of various combination rules has been the interest of numerous researchers [START_REF] Quost | Classifier fusion in the Dempster-Shafer framework using optimized t-norm based combination rules[END_REF]. As a matter of fact, several correlation measures have been proposed in the literature. One of the most commonly used measures is the one proposed by Goebel and Yan (2004). This latter, denoted by ρ, allows to quantify the dependency between a member (r (≥2) of classifiers. Based on the outputs of the individual classifiers, the correlation coefficient is computed as follows:

ρ = m.N f N -N f -N t + m.N f ( 32
)
where m is the number of classifiers, N is the number of test patterns, N t is the number of test patterns for which all classifiers had a right answer, and N f is the number of test patterns for which all classifiers had a wrong answer. In the following, we will study the influence of classifier dependencies on the relative performance of the cautious conjunctive, the normalized cautious and the cautious CWAC rules. Thus, we propose to carry out different experimental tests where classifier dependency degrees will differ from one experimentation to another. The results carried out from the different tests will be presented and analyzed in order to pick out the most appropriate fusion rule.

Experimentation 1

As a first experimentation, we look forward to combine a set of classifiers with a low correlation through the cautious conjunctive, the normalized cautious and the cautious CWAC rules. Thus, we propose to combine heterogeneous sets of classifiers which are able to make different errors on a new pattern to be classified. Exactly, we have chosen to combine the 1-NN, the NB, the NN and the DT classifiers. The correlation coefficient between these classifiers is presented in Table 2. As seen in Table 2, the degree of correlation between the 1-NN, the NB, the NN and the DT classifiers is almost low for the different databases (e.g., the ρ-correlations are equal to 0 for Iris database and 0.1 for Parkinson's database). Accordingly, we will study the behavior of the three alternative operators, where classifiers to be combined have low correlation, in the purpose of finding out the most convenient one. Let us lead off by comparing the cautious conjunctive rule with the cautious CWAC one in terms of the distance criterion. The total distance results are given in Table 3 where we can notice that the total distance relative to the cautious CWAC rule is lower than the one obtained by applying the cautious conjunctive rule for all the databases. For instance, the total distance relative to the Car Evaluation database by the use of the cautious conjunctive operator is equal to 876.39, whereas the total distance obtained by applying the cautious CWAC operator is equal to 852.50. This interpretation is also available for all the other databases. Accordingly, we can say that the cautious CWAC rule is more efficient than the cautious conjunctive one in terms of the distance criterion. From this conclusion, we move on to the comparison of the cautious CWAC rule with the normalized cautious rule in terms of the PCC criterion. Figure 2 represents the mean PCC values for both the normalized cautious and the cautious CWAC rules relative to all the mentioned databases. This latter indicates that the cautious CWAC rule has the highest value of mean PCC compared with the normalized cautious one for all the databases. These results may be explained by the fact that the mean number of the rejected instances relative to the cautious CWAC rule is greater than 0, while that relative to the normalized cautious one is equal to 0. Accordingly, we can conclude that the cautious CWAC rule is more efficient than the normalized cautious rule in terms of the PCC criterion.

Experimentation 2

We recall that we aim to study the behavior of the cautious conjunctive, the normalized cautious and the cautious CWAC rules with different classifier dependency degrees. Thus, in this experimentation, we tend to increase the corre- Subsequently, the outputs obtained by C 1 , C 2 and C 3 will be combined in a same way to obtain a consensus result (see Figure 3 where R 1 corresponds to the conjunctive rule and R 2 will be either the cautious conjunctive rule, the normalized cautious rule or the cautious CWAC rule).

The correlation coefficient between C 1 , C 2 and C 3 is given in Table 4 for the different mentioned databases. It is clear from the results in Table 4 that the ρ-correlation coefficient remains high for the different databases (e.g., for the Parkinsons we have 1 as ρ-correlation value). As a result, we can conclude that C 1 , C 2 and C 3 are highly correlated. From this conclusion, we move on to analyze the performance of the three alternative combination rules. The total distance and the mean PCC results are presented in Table 5. From this latter, we can notice that the total distance of the cautious conjunctive rule is greater than that relative to the cautious CWAC one for the different databases (e.g. for the database Diabetes, the total distance of the cautious conjunctive and the cautious CWAC rules are respectively 334.87 and 295.23). Accordingly, it can be said that the cautious CWAC rule is better than the cautious conjunctive one in term of the distance criterion. Besides, we can note that the mean PCC of the cautious CWAC rule is greater than that relative to the normalized cautious one for the different databases. For instance, for Hepatitis database, we have 83.22% as normalized cautious mean PCC and 85.79% as cautious CWAC mean PCC. So far, we can assume that this interpretation is also available for the other databases. From this, there is no doubt that the cautious CWAC rule is the best fusion rule compared with the normalized cautious according to the PCC criterion.

Experimentation 3

In the previous experimentation, we have generated a set of correlated classifiers based on the output of mutiple distinct classifiers (the NB, the DT the NN and the 1-NN classifiers). However, there exist other ways for building highly dependent classifiers from a unique learning algorithm. One of correlation measure for the different databases in term of P value. This latter clearly show that the degree of dependency between the Naive Bayes classifiers remains high for all the mentioned databases. For instance, it equals 0.934 for Iris database and it is equal to 0.849 for Diabetes database. Therefore, it can be said that the NB 1 , NB 2 , NB 3 and NB 4 are highly correlated and consequently we try to combine them using the cautious conjunctive, the normalized cautious and the cautious CWAC rules. The total distance result for the different databases is given in Figure 4, where we can note that the total distance of the cautious conjunctive rule is almost greater than that relative to the cautious CWAC rule for the different databases. So, we can deduce the efficiency of the cautious CWAC rule compared with the cautious conjunctive one in term of the distance criterion. Let's move on now to compare the cautious CWAC rule with the normalized cautious rule in term of the PCC criterion. The mean PCC of the different databases is given in Figure 5 where we can notice that the cautious CWAC rule has the highest values of mean PCC compared with the normalized cautious rule for the different databases. So, it can be said that the cautious CWAC rule is better than the normalized cautious one in the context of the PCC criterion.

Conclusion

In this paper, we have introduced classifier fusion approach adapted to the framework of belief functions using dependent combination rules; the cautious conjunctive, the normalized cautious and the cautious CWAC combination rules.

Our objective is to pick out the most appropriate fusion rule among these three ones. To this end, we have proposed a comparative approach that aims to compare the cautious CWAC rule with the cautious conjunctive and the normalized cautious ones in terms of two evaluation criteria, namely the Jousselme distance and the PCC. Experimental results have shown the efficiency of the cautious CWAC rule compared with the cautious conjunctive one according to the distance criterion. Furthermore, we have proved that mostly the cautious CWAC rule is more efficient than the normalized cautious one in term of the PCC criterion. In sum, we can deduce that the cautious CWAC rule is the most adequate fusion rule compared with the two others. As future works, we look forward to using the correlation coefficient as a new assessment criterion to enhance our approach. We also intend to compare the cautious CWAC rule with other existing dependent rules. -Ani, A., & Deriche, M. (2002). A new technique for

Al

Fig. 1

 1 Fig. 1 Classifier fusion within the belief function framework

  K+1)Kwhere each row k corresponds to the predicted class θ k , each column l corresponds to the actual class θ l , an element n i kl represents the number of test patterns of class θ l which have been classified by C i in class θ k and n i denotes the total number of patterns classified by the classifier C i . The performance rates of C i are computed as follows:

Fig. 2

 2 Fig. 2 PCC results of the normalized cautious and the cautious CWAC rules

Fig. 3

 3 Fig. 3 Two fusion rules to combine classifiers

Fig. 4

 4 Fig. 4 Distances for both cautious conjunctive and cautious CWAC rules

Table 1

 1 Description of databases

	Databases	#Instances #Attributes #Classes
	Nursery	12960	8	5
	Chess	3196	36	2
	Tic-Tac-Toa	958	9	2
	Diabetes	768	2	2
	Car Evaluation	1728	6	4
	Hepatitis	155	19	2
	Iris	150	4	3
	Parkinson's	195	23	2
	Wine	178	13	3

Table 2

 2 Correlation coefficient between 1-NN, NB, NN and DT classifiers

	Databases	ρ-correlation
	Nursery	0.005
	Chess	0.01
	Tic-Tac-Toa	0
	Diabetes	0.535
	Car Evaluation	0.05
	Hepatitis	0.505
	Iris	0.2667
	Parkinsons	0.1
	Wine	0

Table 3

 3 Distance results

	Databases	Cautious conjunctive Cautious CWAC
	Nursery	8624.19	8481.42
	Chess	1731.05	1713.93
	Tic-Tac-Toa	280.17	252.50
	Diabetes	334.87	312.45
	Car Evaluation	876.39	852.50
	Hepatitis	54.90	51.56
	Iris	15.09	14.78
	Parkinsons	60.92	55.31
	Wine	24.47	23.83

Table 4

 4 PCCs for both cautious conjunctive and cautious CWAC rules Correlation coefficient between the classifiers C 1 ,C 2 and C 3 P 2 , P 3 and P 4 . These partitions have common instances which are dependent on the rate P. Besides, we suggest to learn four Naive Bayes classifiers NB 1 , NB 2 , NB 3 and NB 4 from respectively P 1 , P 2 , P 3 and P 4 . Subsequently, we should quantify the degree of dependency between the naive Bayes classifiers for the different values of P before any combination level takes place. As partitions will be randomly generated, the learning procedure should be executed several times (100 times) and consequently the 100 estimate results will be averaged to yield an overall estimate result. If classifiers are highly correlated, we move on to combine them through the cautious conjunctive, the normalized cautious and the cautious CWAC rules. Table6presents the ρ-

	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious
				Cautious CWAC				Cautious CWAC				Cautious CWAC
	100					100
											74
	95					98					72
						96					70
	90										68
	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
			P					P					P
	(a) Nursery database			(b) Chess database		(c) Tic-Tac-Toa database
	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious
				Cautious CWAC				Cautious CWAC				Cautious CWAC
	80					100					90
	78					80					88
	76
						60					86
	74
	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
			P					P					P
	(d) Diabetes database		(e) Car Evaluation database	(f) Hepatitis database
	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious	PCC(%))			Normalized cautious
				Cautious CWAC				Cautious CWAC				Cautious CWAC
						100					100
	74
						98					98
	72
						96					96
	70
	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9	0.1	0.3	0.5	0.7	0.9
			P					P					P
	(g) Parkinons database			(h) Iris database			(i) Wine database
	Fig. 5 Databases		ρ-correlation			ing set into four training partitions of equal size denoted P 1 ,
	Nursery		0.725
	Chess		0.938
	Tic-Tac-Toa	0.723
	Diabetes		0.900
	Car Evaluation	0.821
	Hepatitis		0.950
	Iris			0.981
	Parkinsons			1
	Wine		0.974
	the proposed solutions consists of training the same learn-
	ing algorithm with different learning subsets. Thus, in this
	experimentation, we propose to randomly divide the train-

Table 5

 5 Combination results of the classifiers C 1 ,C 2 and C 3

	Databases	Total distance criterion Cautious Cautious	Mean PCC criterion Normalized Cautious
		conjunctive	CWAC	cautious	CWAC
	Nursery	8624.19	8312.07	98.90 %	99.63 %
	Chess	1731.055	1653.72	99.52 %	99.53 %
	Tic-Tac-Toa	280.17	220.31	99.16 %	99.69 %
	Diabetes	334.87	295.23	75.65 %	77.44 %
	Car Evaluation	876.39	842.03	53.76%	88.53%
	Hepatitis	54.90	49.07	83.22%	85.79%
	Iris	15.51	14.52	96.66%	98.75%
	Parkinsons	60.92	46.99	96.41%	97.53%
	Wine	24.47	19.71	99.43%	99.88%

Table 6

 6

	ρ-correlation for the different databases
	P	Nursery Chess Tic-Tac-Toa Diabetes Car Evaluation Hepatitis	Iris	Parkinsons	Wine
	0.1	0.823	0.930	0.679	0.794	0.870	0.760	0.917	0.827	0.886
	0.2	0.835	0.936	0.696	0.804	0.890	0.769	0.920	0.831	0.890
	0.3	0.848	0.942	0.720	0.815	0.896	0.784	0.923	0.840	0.894
	0.4	0.852	0.946	0.738	0.827	0.903	0.798	0.927	0.851	0.901
	0.5	0.863	0.951	0.752	0.843	0.916	0.813	0.930	0.867	0.910
	0.6	0.871	0.954	0.782	0.858	0.924	0.836	0.933	0.877	0.919
	0.7	0.873	0.957	0.814	0.877	0.936	0.858	0.943	0.895	0.932
	0.8	0.884	0.986	0.847	0.900	0.941	0.880	0.950	0.912	0.945
	0.9	0.892	0.991	0.887	0.927	0.950	0.918	0.959	0.941	0.965
	Mean	0.860	0.957	0.768	0.849	0.914	0.824	0.934	0.871	0.913