Benjamin Quost
email: benjamin.quost@utc.fr

Thierry Denoeux
email: thierry.denoeux@utc.fr

Shoumei Li

Parametric Classification with Soft Labels using the Evidential EM Algorithm Linear Discriminant Analysis vs. Logistic Regression

Keywords: Partially supervised learning, Belief functions, Dempster-Shafer theory, Machine Learning, Uncertain data, Discriminant analysis, Logistic regression Mathematics Subject Classification (2000) MSC 62H30, MSC

Partially supervised learning extends both supervised and unsupervised learning, by considering situations in which only partial information about the response variable is available. In this paper, we consider partially supervised classification and we assume the learning instances to be labeled by Dempster-Shafer mass functions, called soft labels. Linear discriminant analysis and logistic regression are considered as special cases of generative and discriminative parametric models. We show that the evidential EM algorithm can be particularized to fit the parameters in each of these models. We describe experimental results with simulated data sets as well as with two real applications: K-complex detection in sleep EEGs signals and facial expression recognition. These results confirm the interest of using soft labels for classification as compared to potentially erroneous crisp labels, when the true class membership is partially unknown or ill-defined.

Introduction

Classically, a distinction is made in data analysis and machine learning between supervised and unsupervised learning. In supervised learning, the data set is composed of observations of the response variable and a list of input variables for n individuals of some population. The problem is then to predict the value of the response variable for a new individual. In unsupervised learning, no response variable is observed, and the task is to find some underlying structure in the data (such as a partition or a manifold).

In recent years, though, the distinction between supervised and unsupervised learning has been blurred by the introduction of new paradigms that lie between these two extremes. One of these paradigms is semi-supervised learning [START_REF]Semi-Supervised Learning[END_REF], in which the response variable is perfectly known for some individuals, and totally unknown for others. An even more general paradigm is partially supervised learning [START_REF] Denoeux | A k-nearest neighbor classification rule based on Dempster-Shafer theory[END_REF][START_REF] Denoeux | Handling possibilistic labels in pattern classification using evidential reasoning[END_REF][START_REF] Hüllermeier | Learning from ambiguously labeled examples[END_REF][START_REF] Nguyen | Classification with partial labels[END_REF][START_REF] Côme | Learning from partially supervised data using mixture models and belief functions[END_REF][START_REF] Cour | Learning from partial labels[END_REF], in which the values of the response variable for learning instances is only assumed to be partially known or uncertain, i.e., is subject to some hard or soft constraints. In [START_REF] Denoeux | A k-nearest neighbor classification rule based on Dempster-Shafer theory[END_REF], the author first considered a type of classification problem in which partial knowledge about class labels is expressed in the Dempster-Shafer (DS) framework by belief functions [START_REF] Dempster | Upper and lower probabilities induced by a multivalued mapping[END_REF][START_REF] Shafer | A mathematical theory of evidence[END_REF]: we can then speak of soft labels [START_REF] Côme | Learning from partially supervised data using mixture models and belief functions[END_REF]. The interest of DS theory in this context relies on the generality of belief functions, which encompass probabilities, sets and possibility measures as special cases: using belief functions to label training data thus allows us to encode various pieces of evidence about class labels. In particular, soft labels typically occur when no ground truth about class labels is available and data have to be labeled by experts or using some indirect method, or when the presence of noise casts doubt on the observations of the response variable. In these cases, using soft labels makes it possible to model the available side knowledge regarding the class information to be predicted. Due to the generality of the DS framework, the concept of soft label allows us to express uncertainty about class labels in a variety of ways, including sets, possibility distributions or probability distributions.

To learn from data with soft labels, nonparametric techniques such as the evidential k-nearest neighbor rule [START_REF] Denoeux | A k-nearest neighbor classification rule based on Dempster-Shafer theory[END_REF] and decision trees [START_REF] Denoeux | Induction of decision trees for partially classified data[END_REF][START_REF] Elouedi | Belief decision trees: Theoretical foundations[END_REF][START_REF] Trabelsi | Pruning belief decision tree methods in averaging and conjunctive approaches[END_REF] have first been proposed. A general mechanism for parametric inference in the presence of uncertain data (of which soft labels are a special case) was later introduced in [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF]. In this approach, uncertain class information is represented by mass functions. The notion of likelihood can be extended to such soft class labels: it can be shown to depend on the contour functions associated with the uncertain observations. Maximizing this generalized likelihood generally requires using an iterative procedure. Such a procedure, called the Evidential EM (E 2 M) algorithm, was introduced in [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF]; it extends the Expectation-Maximization (EM) algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] by allowing for maximum-likelihood estimation from partially missing data. The E 2 M algorithm has been applied to a variety of tasks and models, including partially supervised Independent Factor Analysis [START_REF] Cherfi | Partially supervised independent factor analysis using soft labels elicited from multiple experts: Application to railway track circuit diagnosis[END_REF], Hidden Markov Models with partially hidden states [START_REF] Ramasso | Making use of partial knowledge about hidden states in HMMs: an approach based on belief functions[END_REF], fuzzy data clustering using Gaussian mixture models [START_REF] Quost | Clustering and classification of fuzzy data using the fuzzy EM algorithm[END_REF], decision trees [START_REF] Sutton-Charani | Learning decision trees from uncertain data with an evidential EM approach[END_REF][START_REF] Sutton-Charani | Training and evaluating classifiers from evidential data: Application to E2M decision tree pruning[END_REF][START_REF] Ma | Online active learning of decision trees with evidential data[END_REF], and mixture models with progressively censored data [START_REF] Zhou | Evidential-EM algorithm applied to progressively censored observations[END_REF].

Two types of parametric models are commonly used in classification: generative models describe the joint distribution of the input vector W and the class label Z, while discriminative models represent the conditional distribution of Z given W . Generative models, such as Gaussian mixture models, are suitable both for unsupervised learning (clustering) and for supervised classification. In contrast, discriminative models (such as logistic regression) cannot be used in an unsupervised setting, because they need observations of the response variable Y . However, they rely on fewer adaptive parameters and less restrictive assumptions, which may result in better predictive performance in supervised learning tasks, especially when class-conditional distributions are misspecified [START_REF] Press | Choosing between logistic regression and discriminant analysis[END_REF] [3, p. 204]. The question then arises of the relative performances of generative and discriminative models in the presence of soft labels. To the best of our knowledge, this question has not been addressed until now. It will be investigated in this paper, with emphasis on linear classification. As representatives of generative and discriminative classifiers, we will consider, respectively, partially supervised Linear Discriminant Analysis (LDA), and Logistic Regression (LR) based on the E 2 M algorithm. These two techniques will be introduced and compared using simulated and real data with soft labels obtained both by simulation, and from real experts. The objective of this paper is to study the influence of label uncertainty on the performances of these two classification models and, if possible, to formulate prescriptions regarding the suitability of each of these two approaches in situations where class labels are uncertain. Another problem addressed in this paper concern the assessment of classifier performances based on partially labeled data, a problem which had not received attention before. We propose a solution based on the notions of lower and upper expected losses. This method is useful, in particular, for model (e.g., classifier or feature) selection in presence of data with soft labels.

The rest of this paper is organized as follows. Background information about the theory of belief functions and the E 2 M algorithm will first be recalled in Section 2. Partially supervised LDA and LR will then be introduced in Section 3. The issues of performance evaluation and model selection will be addressed in Section 4, and experimental results will be presented in Section 5. Section 6 will conclude the paper.

Background

In order to make the paper self-contained, the basic concepts of Dempster-Shafer theory will first be recalled in Section 2.1. The notion of evidential likelihood and the E 2 M algorithm will then be described in Sections 2.2 and 2.3, respectively.

Dempster-Shafer Theory

In Dempster-Shafer theory [START_REF] Shafer | A mathematical theory of evidence[END_REF], uncertain knowledge about some variable X taking values in a finite set X is described by a mass function, which is defined as a mapping m from the power set 2 X to [0, 1] verifying A⊆X m(A) = 1 and m(∅) = 0. The subsets A of X such that m(A) > 0 are called the focal sets of m. A mass function m with a single focal set is said to be logical. Logical mass functions are in one-to-one correspondence with the subsets of X . The mass function m ? such that m ? (X ) = 1 is said to be vacuous; it represents total ignorance.

From a mass function m, we can compute a belief function and a plausibility function, defined, respectively, as

Bel(A) = B⊆A m(B) and P l(A) = B∩A =∅ m(B) = 1 -Bel(A),
for all A ⊆ X . Functions Bel and P l are, respectively, completely monotone and completely alternating capacities [START_REF] Shafer | A mathematical theory of evidence[END_REF]. When all the focal sets of m are singletons, m is said to be Bayesian; Bel and P l then boil down to the same probability measure. When m is consonant, i.e., when its focal sets are nested, then P l is a possibility measure [START_REF] Zadeh | Fuzzy sets as a basis for a theory of possibility[END_REF]: it verifies P l(A ∪ B) = max(P l(A), P l(B))

for all A, B ⊆ X . The mapping pl : X → [0, 1] defined by pl(x) = P l({x}) for any x ∈ X is called the contour function of m. When m is consonant, the following equalities hold,

P l(A) = max x∈A pl(x)
for all A ⊆ X , and max x∈ pl(x) = 1. When m is Bayesian, pl is a probability mass function and x∈X pl(x) = 1. The discounting operation [START_REF] Shafer | A mathematical theory of evidence[END_REF] allows us to take into account the reliability of a source of information. Assume that a source provides us with a mass function m, and we have a degree of confidence 1 -α in the reliability of the source. We can then construct a new mass function α m defined as

α m = (1 -α)m + αm ? . ( 1
)
Parameter α is called the discount rate. If α = 0, the mass function is unchanged. If α = 0, the discounted mass function is vacuous.

Given two mass function m 1 and m 2 , their orthogonal sum is defined as the mass function

m 1 ⊕ m 2 such that (m 1 ⊕ m 2 )(∅) = 0 and (m 1 ⊕ m 2 )(A) = 1 1 -κ B∩C=A m 1 (B)m 2 (C) (2)
for all A ⊆ X , A = ∅, where

κ = B∩C=∅ m 1 (B)m 2 (C) (3)
is called the degree of conflict between m 1 and m 2 . The orthogonal sum of m 1 and m 2 is well-defined as long as they are not totally conflicting, i.e., their degree of conflict is strictly less than 1. Operation ⊕ is called Dempster's rule of combination. The contour function pl 1 ⊕ pl 2 of m 1 ⊕ m 2 is proportional to the product of the contour functions pl 1 and pl 2 associated with m 1 and m 2 : we have

(pl 1 ⊕ pl 2 )(x) = pl 1 (x)pl 2 (x) 1 -κ (4)
for all x ∈ X . The orthogonal sum of a Bayesian mass function m 1 and an arbitrary mass function m 2 is Bayesian; the degree of conflict between m 1 and m 2 can then be obtained from the contour functions as

κ = 1 - x∈X pl 1 (x)pl 2 (x). (5)

Evidential Likelihood

Let X be a discrete random vector with finite sample space X and probability mass function p X (x; θ) assumed to be known up to a parameter θ ∈ Θ. After a realization x of X has been observed, the (complete-data) likelihood function is the mapping from Θ to [0, 1] defined by

L c (θ) = p X (x; θ), ∀θ ∈ Θ. (6)
Let us now assume that x is not observed precisely, but we collect some evidence about x. This evidence induces partial knowledge of x described by a mass function m on X . The likelihood function [START_REF] Côme | Learning from partially supervised data using mixture models and belief functions[END_REF] can then be generalized [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF] to

L(θ) = A⊆X m(A) x∈A p X (x; θ), ∀θ ∈ Θ, (7)
Function L(θ) defined by ( 7) is called the evidential likelihood function induced by the uncertain data m. It must be emphasized that the notion of evidential likelihood is a new concept. This concept is distinct from the classical one of likelihood, because mass function m is not a realization of a random element. See [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF] for a detailed discussion on this issue, and [START_REF] Couso | Maximum likelihood under incomplete information: Toward a comparison of criteria[END_REF] for alternative definitions of likelihood in the case of imprecise observations. When m is logical with m(A) = 1 for some A ⊆ X , then L(θ) is simply the probability of the event X ∈ A. The evidential likelihood function then becomes identical to the likelihood function under the Coarsening at Random (CAR) assumption [START_REF] Heitjan | Ignorability and coarse data[END_REF]. However, we do not assume in our model that A is generated at random. Instead, the mass function characterizes epistemic uncertainty about x. This partial knowledge can be due to a indirect observation of the quantity of interest, to the subjectivity of the information source, or to the presence of noise with ill-known distribution. Whenever mass function m in ( 7) is certain, i.e., when m({x}) = 1, the evidential likelihood [START_REF] Cour | Learning from partial labels[END_REF] coincides with the classical likelihood [START_REF] Côme | Learning from partially supervised data using mixture models and belief functions[END_REF], which only depends on the pdf p X modeling the random data generating process. By permuting the two summations in [START_REF] Cour | Learning from partial labels[END_REF], we get another expression for L(θ) as

L(θ) = x∈X p X (x; θ) A x m(A) = x∈X p X (x; θ)pl(x), ( 8
)
where pl is the contour function associated to m. Comparing ( 8) and ( 5), we can see that 1-L(θ) equals the degree of conflict between the uncertain data m and the probability mass function p(x; θ). Maximizing L(θ; m) thus amounts to minimizing the conflict between the data and the model. Equation ( 8) also reveals that L(θ; pl) can alternatively be viewed as the expectation of pl(X),

L(θ) = E θ [pl(X)]. (9)
In the special case where X = (X 1 , . . . , X n ) is an independent sample, and assuming that the contour function pl can be decomposed as

pl(x) = pl 1 (x 1 ) . . . pl n (x n ), (10)
a property called cognitive independence by Shafer [START_REF] Shafer | A mathematical theory of evidence[END_REF], (9) simplifies to

L(θ) = n i=1 E θ [pl i (X i )]. (11)
Finally, we can remark that the normalized likelihood pl(θ) = L(θ)/L( θ), where θ is a maximizer of L(θ), and it is assumed that L( θ) < ∞, can be interpreted as the contour function of a consonant belief function on Θ (or, equivalently, as a possibility distribution on Θ) [START_REF] Denoeux | Likelihood-based belief function: justification and some extensions to low-quality data[END_REF]. Equations ( 9) and [START_REF] Denoeux | A k-nearest neighbor classification rule based on Dempster-Shafer theory[END_REF] can directly be extended to the case of continuous data.

E 2 M Algorithm

The E 2 M algorithm introduced in [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF] is a generalization of the EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], which allows one to maximize the evidential likelihood ( 7)- [START_REF] Dempster | Upper and lower probabilities induced by a multivalued mapping[END_REF]. Similarly to the EM algorithm, each iteration q of E 2 M is composed of two steps:

1. The E-step requires the combination of p X (x; θ (q) ), the probability mass function of X for the current estimate θ (q) of θ, with the contour function pl. The result of this combination is a probability mass function p X (•|pl; θ (q) ) = p X (•; θ (q) ) ⊕ pl, which can be computed as p X (x|pl; θ (q) ) = p X (x; θ (q) )pl(x) L(θ (q) ; pl) , for all x ∈ X . Then, the expectation of the complete-data log-likelihood c (θ) = log p X (X; θ) with respect to p(x|pl; θ (q) ) is calculated,

Q(θ, θ (q) ) = x∈X c (θ; x)p X (x; θ (q) )pl(x) L(θ (q) ; pl) . ( 12
)
2. The M-step then consists in maximizing function Q(θ, θ (q) ) with respect to θ, finding the new estimate θ (q+1) such that Q(θ (q+1) , θ (q) ) ≥ Q(θ, θ (q) ) for all θ ∈ Θ.

The E-and M-steps are iterated until L(θ (q+1) ) -L(θ (q) ) ≤ for some arbitrarily small . The E 2 M algorithm was shown in [START_REF] Denoeux | Maximum likelihood estimation from uncertain data in the belief function framework[END_REF] to increase the evidential likelihood as each iteration, i.e., to ensure that L(θ (q+1) ) ≥ L(θ (q) ) for all q. Consequently, it converges to a local maximum of the evidential likelihood if this function is bounded from above. When the contour function pl corresponds to a logical mass function, i.e., when pl(x) = I(x ∈ A) for some A ⊂ X , where I(•) is the indicator function, the evidential likelihood -and thus the expectation (12) -coincide with their certain counterparts: the E 2 M algorithm then boils down to the classical EM algorithm.

The E 2 M algorithm has been applied to several Machine Learning problems. A first category of application concerns problems in which we want to exploit some information about variables that are usually considered as latent: we can then speak of "partially latent" variables. For instance, in [START_REF] Cherfi | Partially supervised independent factor analysis using soft labels elicited from multiple experts: Application to railway track circuit diagnosis[END_REF], the authors consider an Independent Factor Analysis model in which the discrete latent variables represent the states of railway track circuits, about which partial knowledge is elicited from experts. In [START_REF] Ramasso | Making use of partial knowledge about hidden states in HMMs: an approach based on belief functions[END_REF], the authors apply the E 2 M algorithm to estimate the parameter of a Hidden Markov Model in which the hidden state variable is not totally "hidden", but partially observed; they describe an application to machine condition monitoring. A second category of applications concerns the problem of learning from uncertain data. For instance, Sutton-Charani et al. [START_REF] Sutton-Charani | Learning decision trees from uncertain data with an evidential EM approach[END_REF][START_REF] Sutton-Charani | Training and evaluating classifiers from evidential data: Application to E2M decision tree pruning[END_REF] and Ma et al. [START_REF] Ma | Online active learning of decision trees with evidential data[END_REF] applied the E 2 M algorithm to decision tree inference from data with uncertain attributes, while the problem of clustering data with fuzzy attributes was considered by Quost and Denoeux in [START_REF] Quost | Clustering and classification of fuzzy data using the fuzzy EM algorithm[END_REF]. In [START_REF] Zhou | Evidential-EM algorithm applied to progressively censored observations[END_REF], the authors considered the modeling of lifetime data using mixture models and progressively censored observations. The task of learning a classifier from partially labeled data also belongs to this second category. This problem will be addressed in the next section.

Application to Linear Classification Models

In this section, the complete data are assumed to consist in an i.i.d. sample

X = {(W i , Z i )} n i=1f from (W, Z)
, where W is a d-dimensional random input vector and Z is the class variable, taking values in a finite set Z = {1, . . . , K}. The complete dataset x = {(w i , z i )} n i=1 is a realization from X. The notion of soft label will first be introduced in Section 3.1. The application of the E 2 M algorithm to estimate the parameter of LDA and LR using uncertain class information will then be described, respectively, in Section 3.2 and 3.3. The complexity of partially supervised LDA and LR will then be discussed in Section 3.4.

Soft Labels

In some applications, class information is uncertain. This is the case, in particular, when the class labels cannot be observed directly and have to be inferred by an unsupervised learning algorithm (see, e.g., [START_REF] Denoeux | Evidential clustering of large dissimilarity data[END_REF][START_REF] Liu | Credal c-means clustering method based on belief functions[END_REF][START_REF] Liu | Hybrid classification system for uncertain data[END_REF]) or by any other indirect method. Sometimes, class labels are assessed subjectively by an expert, a group of experts (as in the two examples described in Section 5.2), or even by a large number of individuals through crowdsourcing (see, e.g. [START_REF] Abassi | Crowd label aggregation under a belief function framework[END_REF][START_REF] Rjab | Characterization of experts in crowdsourcing platforms[END_REF]). In the multiple-expert case, the more discordant the opinions are, the more doubt can be cast on the class information that would be obtained, for instance, by majority voting.

Rather than using crisp but possibly erroneous class information z i , soft class labels may be used, so as to reflect the degree of confidence in each of the possible classes. Figure 1 illustrates the advantage of soft labels over crisp, but potentially erroneous labels. In this toy example, two doubtful instances have a strong influence on the decision boundary. By expressing lack of confidence in the class membership of these instances using soft labels, we decrease their influence on the classifier. When information about the class labels of some instances is not reliable, it is preferable to take this uncertainty into account, in order to decrease the influence of the most doubtful patterns on the decision rule. In this paper, we consider the representation of uncertainty in the Dempster-Shafer framework. A soft label for instance i is defined as a mass function m i on the set Z of classes. An n-tuple (m 1 , . . . , m n ) of mass functions for n learning instances is called a credal partition [START_REF] Denoeux | EVCLUS: Evidential clustering of proximity data[END_REF]. As noted in [START_REF] Denoeux | Beyond fuzzy, possibilistic and rough: An investigation of belief functions in clustering[END_REF], the notion of credal partition subsumes most other soft clustering notions. For instance, if mass functions m i are consonant, they define possibility distributions and, equivalently, fuzzy subsets of Z. If mass functions are logical (i.e., if they verify m i (A i ) = 1 for some A i ⊆ Z), then we can define the lower and upper approximations of each class k as, respectively, the set of instances for which A i = {k}, and the set of instances for which k ∈ A i . We then recover notions from rough classification and clustering [START_REF] Peters | Soft clustering: fuzzy and rough approaches and their extensions and derivatives[END_REF].

Because the E 2 M algorithm described in Section 2.3 uses only the contour functions, we can equivalently represent soft labels by the plausibilities pl ik = pl i (k) of each class k, where pl i is the contour function of m i . Partially supervised learning is then defined as the task of learning a classifier from a learning set {(w i , pl i )} n i=1 with soft labels. We can remark that the partially supervised paradigm encompasses most of the classical learning frameworks. Fully supervised learning is recovered when there is no uncertainty on class labels, i.e., when we have, for all i, pl ik = 1 for some k ∈ {1, . . . , K} and pl i = 0 for all = k. Unsupervised learning corresponds to the situation where the soft labels are vacuous, i.e., pl ik = 1 for all i and all k. When some labels are crisp and some are vacuous, we get the semi-supervised learning paradigm. It is clear, however, that soft labelling goes much further than these classical formalisms, by allowing the user to specify numerical degrees of plausibility for each class and each learning instance. Induction strategies may then be adapted to such soft labels in order to avoid learning a biased model by attaching too much importance to doubtful information.

In the sequel, we will consider the application of the E 2 M algorithm to estimate the parameter of two models: LDA and LR using data with soft labels.

Linear Discriminant Analysis

LDA is based on the assumption that the conditional distribution of W given Z = k is multivariate normal with mean µ k and covariance matrix Σ independent on k:

W |(Z = k) ∼ N (µ k , Σ), k = 1, . . . , K.
Let π k be the marginal probability that Z = k, and

θ = (µ 1 , . . . , µ K , Σ, π 1 , . . . , π K-1 )
the parameter vector. The complete-data likelihood is

L c (θ) = n i=1 p(w i |Z i = z i )p(z i ) (13a) = n i=1 K k=1 φ(w i ; µ k , Σ) z ik π z ik k , (13b)
where φ(•; µ k , Σ) is the multivariate normal density,

φ(w; µ k , Σ) = 1 (2π) d/2 |Σ| 1/2 exp - 1 2 (w -µ) T Σ -1 (w -µ) ,
and z ik is a binary class indicator variable, such that z ik = 1 if z i = k and z ik = 0 otherwise. Under the assumption of cognitive independence [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF], the contour function on

X is pl(x) = n i=1 pl i (x i ), with pl i (x i ) = pl ik if x i = (w i , k) for some k = 1, . . . , K 0 otherwise.
The evidential likelihood ( 11) is thus

L(θ) = n i=1 K k=1 pl ik φ(w i ; µ k , Σ)π k . (14)
We can remark that, when there is no uncertainty, i.e., when pl ik = z ik for all (i, k), we have

K k=1 pl ik φ(w i ; µ k , Σ)π k = K k=1 φ(w i ; µ k , Σ) pl ik π pl ik k ,
and the evidential likelihood [START_REF] Denoeux | Beyond fuzzy, possibilistic and rough: An investigation of belief functions in clustering[END_REF] becomes identical to the complete-data likelihood (13b). When, on the other hand, uncertainty is maximal, i.e., class labels are completely unknown, then pl ik = 1 for all (i, k), and the evidential likelihood [START_REF] Denoeux | Beyond fuzzy, possibilistic and rough: An investigation of belief functions in clustering[END_REF] becomes

L(θ) = n i=1 K k=1 φ(w i ; µ k , Σ)π k ,
which is the likelihood function corresponding to the unsupervised case.

In the E-step of the E 2 M algorithm for this model, we compute the expectation of the complete-data log-likelihood

c (θ) = n i=1 K k=1 z ik [log φ(w i ; µ k , Σ) + log π k ]
with respect to the combined probability mass function

p X (x|pl; θ (q) ) = n i=1 p(x i |pl i ; θ (q) ), with p(x i |pl i ; θ (q) ) =      pl ik π (q) k φ(w i ; µ (q) k , Σ (q) ) pl i π (q) φ(w i ; µ (q) , Σ (q) ) if x i = (w i , k) for some k 0 otherwise.
We get

Q(θ, θ (q) ) = n i=1 K k=1 ζ (q) ik [log φ(w i ; µ k , Σ)π k + log π k ] , (15)
with

ζ (q) ik = E(Z ik |pl; θ (q) ) = pl ik π (q) k φ(w i ; µ (q) k , Σ (q) ) pl i π (q) φ(w i ; µ (q) , Σ (q) ) . ( 16
)
We can remark that the function Q(θ, θ (q) ) defined by [START_REF] Denoeux | EVCLUS: Evidential clustering of proximity data[END_REF] has exactly the same form as the function computed in the E-step of the EM algorithm applied to the normal mixture model in the unsupervised case (see, e.g., [33, pages 81-83]). When class labels are completely unknown, i.e., pl ik = 1 for all i and k, then ζ (q) ik defined by ( 16) becomes equal to the conditional expectation of Z ik given W i = w i , and E 2 M boils down to the classical EM algorithm.

Because of the formal similarity with the EM algorithm, the parameter values maximizing Q(θ, θ (q) ) can be readily obtained as

π (q+1) k = 1 n n i=1 ζ (q)
ik , µ

(q+1) k = n i=1 ζ (q) ik w i n i=1 ζ (q) ik , (17a)
Σ (q+1) = 1 n n i=1 K k=1 ζ (q) ik (w i -µ (q+1) k )(w i -µ (q+1) k ) T . ( 17b
)

Logistic Regression

In contrast with LDA, LR starts with a model of the conditional distribution of Z given W = w. Specifically, assume that the conditional probabilities of each class given W = w are given by

p k (w; θ) = exp(β T k w) 1 + K-1 =1 exp(β T w) , k = 1, . . . , K -1 (18a) p K (w; θ) = 1 1 + K-1 =1 exp(β T w) , (18b)
where p k (w; θ) is a notation for Pr(Z = k|W = w; θ), β k is a p + 1-dimensional vector of coefficients, θ = (β T 1 , . . . , β T K-1 ) T is the vector of all parameters in the model, and w = (1, w T ) T is an extended input vector. Logistic regression proceeds by maximizing the conditional likelihood

L c (θ) = n i=1 Pr(Z i = z i |w i ; θ) (19a) = n i=1 K k=1 p k (w; θ) z ik . ( 19b
)
After plugging the expression of the conditional class probabilities ( 18) into (19b), and taking the logarithm, we obtain the following expression for the complete-data conditional log-likelihood,

c (θ) = n i=1 K-1 k=1 z ik β T k w i -log 1 + K-1 k=1 β T k w i . ( 20
)
Under the same cognitive independence assumption [START_REF] Dempster | Maximum likelihood from incomplete data via the EM algorithm[END_REF] as before, the evidential likelihood is

L(θ) = n i=1 K k=1 pl ik p k (w i ; θ). (21)
We can easily check that L(θ) = L c (θ) whenever pl ik = z ik for all (i, k), i.e., when there is no label uncertainty. On the other hand, in case of maximal uncertainty, i.e., when pl ik = 1 for all (i, k), we have L(θ) = 1 for all θ, and the model parameters can no longer be estimated. The E 2 M algorithm for the maximization of the evidential likelihood (21) can be described as follows [START_REF] Quost | Logistic regression of soft labeled instances via the evidential EM algorithm[END_REF]. In the E-step, we compute the expectation of the complete-data log-likelihood [START_REF] Elouedi | Belief decision trees: Theoretical foundations[END_REF] with respect to the combined probability mass function

p Z (z|pl; θ (q) ) = n i=1 p Zi (z i |pl i ; θ (q) ), with p Zi (k|pl i ; θ (q) ) = pl ik p k (w i ; θ (q) ) pl i p (w i ; θ (q) ) , k = 1, . . . , K.
We get

Q(θ, θ (q) ) = n i=1 K-1 k=1 ζ (q) ik β T k w i -log 1 + K-1 k=1 β T k w i , (22)
with

ζ (q) ik = E(Z ik |pl; θ (q) ) = pl ik p k (w i ; θ (q) ) pl i p (w i ; θ (q) ) . ( 23
)
The maximization of ( 22) cannot be performed in one step and requires an iterative optimization procedure, such as the Newton-Raphson (NR) algorithm. It is actually not necessary to maximize function Q(θ, θ (q) ): we may simply make a step uphill, i.e., find some new estimate θ (q+1) such that Q(θ (q+1) , θ (q) ) > Q(θ (q) , θ (q) ). Such a procedure is classically called a Generalized EM algorithm [32, page 24]. An uphill step starting from the previous estimate θ (q) can be made by carrying out one iteration of the NR algorithm with line search, i.e., by using the following update rule,

θ (q+1) = θ (q) -η ∂ 2 Q(θ, θ (q) ) ∂θ∂θ T -1 θ=θ (q)
∂Q(θ, θ (q) ) ∂θ

θ=θ (q) , ( 24
)
where η is the step size. Typically, η is initially set to 1 and, if the objective function has not increased, it is repeatedly cut by two until an uphill step is achieved. The gradient vector and Hessian matrix can be computed using matrix operations as in standard LR (see, e.g., [START_REF] Hasan | Fast estimation of multinomial logit models: R package mnlogit[END_REF][START_REF] Li | Logistic regression[END_REF] for details).

Complexity

Since partially supervised learning is a more difficult problem than fully supervised learning, we can expect partially supervised LDA or LR to require more computational resources than their classical counterparts. Yet, partially supervised learning algorithm have roughly the same complexity as the corresponding supervised learning procedures.

In classical LDA the maximum likelihood estimates (MLEs) have a closedform expression, whereas in presence of soft labels they are determined after repeatedly computing the expected labels [START_REF] Denoeux | Induction of decision trees for partially classified data[END_REF] and the model parameters (17a)-(17b) [START_REF] Quost | Clustering and classification of fuzzy data using the fuzzy EM algorithm[END_REF]. However, each iteration of the E 2 M algorithm has the same complexity as that of the MLE calculation in classical LDA: consequently, the running time of partially supervised supervised LDA is roughly equal to that of classical LDA multiplied by the number of iterations of E 2 M.

Similarly, partially supervised logistic regression involves repeatedly estimating the expected class labels [START_REF] Hüllermeier | Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization[END_REF] and the model parameters [START_REF] Hüllermeier | Learning from ambiguously labeled examples[END_REF]. However, parameter estimation with classical LR already involves an iterative procedure (the NR algorithm). As stated in Section 3.3, we can perform only one iteration of the NR algorithm at each M-step of the E 2 M algorithm. Further, each iteration of the NR algorithm has the same complexity as one iteration of E 2 M. As a consequence, there is no significant difference between the running times of classical and partially supervised LR.

Performance evaluation and model selection

Performance evaluation and model selection are fundamental issues when designing classifiers. Typically, a loss function is defined, and the performance of a classifier is measured by its expected loss, or risk. If the loss is assumed to be equal to 1 for misclassification and 0 in case for correct classification, then the risk is the error probability. This probability may be estimated from an independent test set, or using cross-validation when the amount of data at hand is too low. In the case of data with soft labels, error (or risk) estimation becomes more difficult, because the true class is uncertain. This issue is addressed in this section.

Lower and upper expected loss

Assume that, for some test pattern i with soft label m i , we get the predicted class z i . Let λ( z i , z i ) denote the loss incurred if the true class is z i . The actual loss is unknown, because we only have partial knowledge of z i , expressed by mass function m i . However, we can compute lower and upper expected values [START_REF] Dempster | Upper and lower probabilities induced by a multivalued mapping[END_REF] as follows:

λ * ( z i , m i ) = A⊆Z m i (A) min k∈A λ( z i , k) (25a) λ * ( z i , m i ) = A⊆Z m i (A) max k∈A λ( z i , k). (25b)
The lower and upper expected losses can be seen as, respectively, optimistic and pessimistic assessments of the unknown loss λ( z i , z i ). A trade-off between these two assessments can be achieved by computing a convex sum

λ ρ ( z i , m i ) = ρλ * ( z i , m i ) + (1 -ρ)λ * ( z i , m i ), (26)
where ρ ∈ [0, 1] is a pessimism index [START_REF] Jaffray | Linear utility theory for belief functions[END_REF][START_REF] Strat | Decision analysis using belief functions[END_REF]. Although the particular form of Eq. ( 26) can be justified axiomatically [START_REF] Jaffray | Linear utility theory for belief functions[END_REF], the theory tells us nothing about the choice of ρ, which models the attitude of the decision maker. In the absence of a better argument, we propose to fix ρ = 0.5, based on a symmetry consideration.

In the special case where λ is the 0-1 loss function, defined by λ( z i , z i ) = I( z i = z i ), we have

min k∈A λ( z i , k) = 0 if z i ∈ A 1 otherwise. Consequently, λ * ( z i , m i ) = A⊆Z: zi ∈A m i (A) (27a) = 1 - A⊆Z: zi∈A m i (A) (27b) = 1 -pl i ( z i ). (27c)
Similarly,

max k∈A λ( z i , k) = 0 if A = { z i } 1 otherwise. Hence, λ * ( z i , m i ) = 1 -m i ({ z i }). ( 28
)
The width of the lower-upper expected loss interval reflects the imprecision of the mass function m i . It is null if m i is Bayesian, and it is equal to one if m i is vacuous.

We can also remark that the lower expected 0-1 loss criterion [START_REF] Li | Logistic regression[END_REF] has an interesting alternative interpretation in terms of degree of conflict. Let m i be the certain mass function defined by m i ({ z i }) = 1. The degree of conflict (3) between m i and m i is

κ i = B∩C=∅ m i (B)m i (C) (29a) = C zi m i (C) (29b) = 1 -pl i ( z i ) = λ * ( z i , m i ). ( 29c
)
The lower expected loss is thus equal to the degree of conflict between the prediction z i and the mass function m i representing one's knowledge about the true class.

Application to performance assessment

Given a test set of size n t , with soft labels {m i } nt i=1 , lower and upper test error rates can be defined by averaging, respectively, the lower and upper expected losses,

λ = 1 n t nt i=1 λ * ( z i , m i ) = 1 - 1 n t nt i=1 pl i ( z i ) (30a) λ = 1 n t nt i=1 λ * ( z i , m i ) = 1 - 1 n t nt i=1 m i ({ z i }). (30b)
These two quantities can be seen as, respectively, optimistic and pessimistic assessments of the test error rate. In the fully supervised case with certain mass function

m i verifying m i ({z i }) = 1, we have pl i ( z i ) = m i ({ z i }) = I(z i = z i );
both λ and λ then boil down to the usual test error rate. In the unsupervised case, we have λ = 0 and λ = 1, meaning that the error rate cannot be assessed. Given a set of models (such as, e.g., LDA and LR with different subsets of input variables), their performances can be assessed using cross-validation estimates of the error rates. Typically, the learning set is partitioned into B blocks of approximately equal size, and B classifiers are trained, leaving one of the B blocks aside. The classifier performance is then assessed from its predictions on the block that was left aside. Lower and upper cross-validation estimates λ CV and λ CV can be obtained in this way.

Given two classifiers C 1 and C 2 , and their lower and upper cross-validation error rates {λ

(i) CV , λ (i) CV }, i = 1, 2, maximality leads to considering classifier C 1 as better than C 2 if λ (1) CV < λ (2)
CV . This criterion yields only a partial order over classifiers. If a total order is needed, then we can base the comparison on the weighted means (1 -ρ)λ

(i) CV + ρλ (i)
CV , after fixing the value of the pessimism index ρ.

Experiments

In this section, we report on experiments with LDA and LR trained from data with soft labels. In Section 5.1, we will first consider simulated and real data in which noise has been artificially introduced in class labels. We will then compare the performances of classifiers trained using the noisy labels, with those of classifiers trained using soft labels encoding labeling uncertainty. In Section 5.2, we will describe two real applications in which soft labels naturally arise.

Simulations

Experimental procedure One synthetic dataset and seven real datasets are considered in this experiment. The synthetic dataset, synthetic with n = 200 instances was generated according to a Gaussian mixture model with g = 3 classes in proportions π 1 = 0.45, π 2 = 0.35 and π 3 = 0.2 in a p = 2-dimensional space, and

µ 1 = 1 -1 , µ 2 = 0 1 , µ 3 = -1 0 , Σ 1 = Σ 2 = Σ 3 = 1 0 0 1 .
The real datasets are classical datasets of the literature. Their main characteristics are reported in Table 1. In these data sets, the true class label is known for each learning instance. To study the influence of different levels of label uncertainty on classifier performance, we need to artificially introduce some uncertainty. For that purpose, we generated, for each instance, a probability p i at random according to a beta distribution with fixed variance (set to 0.04) and expectation p set to 0.1, 0.2, . . . , 0.9 for the synthetic dataset, and 0.1, 0.2, . . . , 0.7 for the others. Then, with probability p i , the label was replaced with a label picked at random. This process simulates the fact that, if the user only has partial information on class labels, but is forced to provide crisp labels, then some errors will inevitably be introduced.

We then generated sets of labels according to four different strategies. The first strategy is to ignore label uncertainty and consider the noisy labels as the true ones. Another approach, referred to hereafter as adaptive semi-supervised learning, is to discard the most uncertain labels; in our simulations, a label z i was kept if p i ≤ 0.5, and the instance was considered as unlabeled otherwise. The other two strategies use soft labels. In the first case, we consider that the user knows each noise level p i . We then discount (see Eq. ( 1)) the label information with a discount rate equal to p i . If k * is the observed label, we thus set pl ik * = 1 and pl i = p i for all = k * . The second strategy for generating soft labels is similar, except that each p i is replaced by the average noise level p. Thus, in this situation, only generic knowledge about the mislabeling process is used, instead of specific information related to each instance. This strategy uses less information, but we might argue that it is much easier to set up than the previous one, which requires an expert to assess each probability p i of a label being erroneous.

For the synthetic dataset, we used a test set of size 1000 from the same distribution as the training data. For the real datasets, the classifiers were trained using 2/3 of the instances picked at random, the remaining ones being used for testing. Two classifiers were considered: LDA and LR. When fitting LDA to the data, the "perfect" model (i.e., obtained from the data without corrupted labels) was first determined, by considering ten different starting points (the initial proportions were set to 1/g, the initial covariance matrices to the identity matrix, and the expectations vectors were sampled at random in the training set) and retaining the best solution (i.e., the one with the highest log-likelihood). The corresponding "actual" parameters were then used as starting values for the four LDA corresponding to the various labeling strategies mentioned above. For logistic regression, the initial parameter matrix was set to the (p + 1) × (g -1) null matrix, and we used quadratic penalization with coefficient λ = 0.01. For each dataset, this whole procedure (training and test instance selection, introduction of label noise, model training, and model testing) was repeated 50 times, and average error rates and 95% confidence intervals were computed. bels or the semi-supervised strategy. This is particularly true for LDA, where the benefit of using soft labels is critical when the level of label noise is high. For a majority of datasets, soft labels based on the average mislabeling rate are almost as effective as those based on individual mislabeling rates (see, e.g., Figures 2(a), 2(b), 3(a)). For some datasets, however, there is a significant difference between both types of soft labels, the more informative one allowing for better performances (see Figures 4(b), 5(a)).

Results and discussion

The benefit of using soft labels together with LR also exists but it is less important than it is with LDA. Generally, soft labels make it possible to improve classification accuracy, but the difference with the adaptive semi-supervised strategy -and sometimes even noisy labels -is less remarkable, especially when mislabeling probability is high. For most datasets, LR and LDA have similar performances with low label noise, but LDA becomes superior when mislabeling probability increases. This observation is consistent with our initial intuition exposed in Section 1, since a discriminative classifier such as LR cannot handle unsupervised learning, in contrast with LDA, which is based on a generative model. Two noticeable exceptions are the Satimage and Vowel datasets, for which LR performs critically better than LDA, with both sets of soft labels (Figures 4(f) and 5(e)). This result may be explained by a strong departure from normality for these two datasets. To check this assumption, we applied Mardia's test of multivariate normality [START_REF] Mardia | Measures of multivariate skewness and kurtosis with applications[END_REF] to both datasets. The results, reported in Tables 2 and3, show that the departure from normality is, indeed, highly significant in both cases. Overall, LDA tends to outperform LR for a majority of datasets, especially when soft labels are highly uncertain. However, LDA can also perform poorly in some cases. In real applications, it is thus necessary to run both classifiers and select the best model using, e.g., cross-validation and the performance measures proposed in Section 4.

Real Applications

In this section, we present two real applications where class labels are elicited from human subjects or "experts". The first application described in Section 5.2.1 concerns the detection of K-complexes in EEG data. In this application, the ground truth cannot be determined objectively and we need to have the data labeled by physicians. Uncertainty is reflected by disagreement between experts. The second application, described in Section 5.2.2, is about expression recognition in facial images. The expression on a face cannot always be determined unambiguously by human subjects. Five subjects were asked to rate the plausibility of each of the six basic expressions for each image, and the resulting credal assessments were combined by Dempster's rule. In each of the two applications, we will compare the performances of LDA and those of LR, both with soft labels and with crisp labels ignoring labeling uncertainty. The methods introduced in this paper were applied to the problem of detecting K-complexes in sleep EEG, using the data described in [START_REF] Richard | Une méthodologie pour la détection à structure imposée[END_REF][START_REF] Richard | Data driven design and complexity control of time-frequency detectors[END_REF]. The K-complex is a transient EEG pattern which plays a major role in sleep stage assessment. It has a duration of 500 to 1500 ms, and is characterized by a sharp upward wave followed by a downward one. Its amplitude is three times background activity [START_REF] Richard | Data driven design and complexity control of time-frequency detectors[END_REF]. The discrimination of K-complexes from background activity is generally considered as a very complex pattern recognition problem. The data used in this experiment consisted of 1178 EEG signals encoded as 64-dimensional patterns. Some of these signals were negative examples containing paroxysmal delta bursts, a phenomenon bearing some resemblance to K-complexes. The other signals consisted of patterns which, after visual in-spection by five physicians, had been classified as containing a K-complex by at least one of them. Among these examples, those categorized in the K-complex class by a majority of experts were considered as positive examples, the others as negative ones. Each example (positive or negative) was then assigned a soft label consisting of a Bayesian mass function m i such that

m i ({1}) = k i /5, m i ({0}) = 1 -k i /5, (31)
where 1 and 0 represent, respectively, the positive (K-complex) and negative (delta wave) class, and k i denotes the number of experts who classified the pattern as positive. The numbers of cases for each value of k i are shown in Both LR and LDA were applied to these data. To reduce the input dimension, Principal Component Analysis (PCA) was first used as a preprocessing step, and the number of components was varied between 1 and 20. The LR and LDA classifiers were trained using three different sets of labels:

1. Soft labels [START_REF] Mardia | Measures of multivariate skewness and kurtosis with applications[END_REF], taking into account the proportion of experts in favor of each class; 2. Crisp labels, corresponding to the majority decision; 3. "Semi-supervised labels": instances classified as positive by two or three experts were considered as ambiguous and were labeled by the vacuous mass function m ? ; the other instances were labeled unambiguously according to the majority class.

The results were evaluated using two measures: the expected loss [START_REF] Ma | Online active learning of decision trees with evidential data[END_REF], and the error rate, assuming the majority class to be the true class. We note that, as mass functions m i are Bayesian, the lower and upper expected losses are equal in this case. Again, since the data are randomly separated into training and test instances, 10-fold cross-validation was used in order to compute average error rates. The mean cross-validation expected loss and error rate with corresponding 95% confidence intervals are represented as functions of the number of principal components for the LR and LDA with the three sets of labels in Figure 7. For LR, training the classifier with soft labels clearly improves the performances, according to both criteria (Figures 7(a) and 7(b)). For LDA, this is true for 5 to 8 principal components, with non significant differences for smaller or larger numbers of components (Figures 7(c) and 7(d)). For both methods, the best results are obtained with 12 principal components. Figure 8 shows boxplots of expected losses and error rates for LR and LDA with 12 principal components, trained using with the three label sets. Logistic regression outperforms LDA for this dataset, and the best results are obtained when training the classifier with soft labels, whatever the performance measure.

For this dataset, LR was thus clearly able to exploit the information contained in soft labels to construct a better classifier. This phenomenon can be explained by the fact that ambiguous patterns, which have a great chance of being mislabeled, have less informative soft labels. Their influence on the final parameter estimates is thus downweighted by the E 2 M algorithm.

Facial Expression Recognition

As another application of partially supervised learning with soft labels, we considered the task of recognizing facial expressions from face images. Here, soft labels may arise from the difficulty of labeling faces with a single expression unambiguously. We considered 216 images data from the CMU-Pittsburgh image database [START_REF] Kanade | Comprehensive database for facial expression analysis[END_REF], with 36 images for each of the six basic expressions (see In each graph, the three curves corresponds to three sets of labels: soft labels, majority labels, and "semi-supervised" labels (see details in text).

Figure 9). Following [START_REF] Dubuisson | A solution for facial expression representation and recognition[END_REF], we used aligned and cropped versions of these images of size 60 × 70, corresponding to vectors of 4200 pixels.

The database was split into a learning set and a test set of 108 images each (with 18 images per expression). Images in the learning set were viewed by five subjects, who assigned a soft label to each image by assessing plausibilities for each of the six expressions. Figure 10 shows two examples of images with the corresponding plausibility assessments, for which there was some disagreement between assessors. For instance, for the image of Figure 10(a), one subject considered that it expresses disgust for sure (white horizontal bar), while others hesitated between anger and disgust, or between disgust and sadness. The assessments from the five experts were discounted [START_REF] Shafer | A mathematical theory of evidence[END_REF] and combined by Dempster's rule using [START_REF]Semi-Supervised Learning[END_REF]. For image i, the unnormalized contour function pl * i was thus

pl * i (k) = 5 j=1 [0.1 + 0.9 pl ij (ω k )] , (32)
where pl ij (k) is the plausibility of expression k for image i assessed by subject j. The plausibilities were then normalized to ensure that max k pl i (k) = 1.

Figure 11 shows two examples of individual and combined plausibility assessments. We can remark that combination rule [START_REF] Mclachlan | The EM Algorithm and Extensions[END_REF] rate obtained for around 30 principal components. For LDA, the best results are obtained with the soft labels, which can be explained by the smaller importance given to ambiguous cases. However, the LDA classifier trained with soft labels is more prone to overfitting, as shown by the sharp increase of error rate with the input dimension. In contrast, LR does not work too well for this data, with a lowest error rate almost twice as large as that obtained by LDA.

For LR, there also seems to be some advantage of using soft labels for input dimensions larger than 40, but this advantage is not so clear as it is for LDA. This second experiment using a real data set with soft labels provides additional evidence of the potential gain of using soft labels and confirms similar findings reported in [START_REF] Cherfi | Partially supervised independent factor analysis using soft labels elicited from multiple experts: Application to railway track circuit diagnosis[END_REF] or [START_REF] Ramasso | Making use of partial knowledge about hidden states in HMMs: an approach based on belief functions[END_REF], for instance. As with the sleep data analyzed in Section 5.2.1, this gain of performance can be explained by the smaller influence of potentially mislabeled data on the final classifier.

Conclusions

Whereas a lot of methods exist for supervised or unsupervised classification, we have seen in recent years a growing interest for more general forms of learning tasks, in which information about the class of learning instances is partial. In this paper, we have considered a very general framework for addressing such problems, which consists in representing partial class information by Dempster-Shafer mass functions constituting soft labels. Two types of linear classifiers have been considered: LDA based on a generative model, and LR based on a discriminative model. Both classifiers can be trained in a partially supervised mode by maximizing an evidential likelihood function, which can be done efficiently using the evidential EM algorithm.

As a generative model represents the joint distribution of the input vector and the response variable, it can be trained in an unsupervised setting; consequently, it can be expected to perform well with scant class information. In contrast, a discriminative model relies on weaker assumptions and can be expected to perform better than a generative model when the model is inadequate, provided sufficient class information is available. Our experimental results support these assumptions. Using artificial and real data for which class labels have been perturbed by noise, we have shown that soft labels taking into account labelling uncertainty can be successfully exploited to achieve significantly lower error rates, as compared to precise but noisy labels. Both LDA and LR classifiers are able to exploit partial information contained in soft labels. However, LDA usually performs better, especially when the mislabeling probability is high, in which case the soft labels become very imprecise and learning becomes almost unsupervised. However, LR outperforms LDA for some datasets. It is thus necessary to consider several models and select the best ones, which can also be done in a partially supervised setting using lower and upper expected losses.

We have also considered two real applications in which soft labels naturally arise from the absence of ground truth class information: detection of K-complex in EEG sleep data and recognition of facial expressions in images. In these two cases, soft labels have been shown to allow for better performances, as compared to using the most plausible label for each instance and ignoring labeling uncertainty. Whereas large amounts of unlabeled data (e.g., images) can be retrieved from the internet or can be generated by sensors, reliably labeled data remain relatively scarce because of the cost of visually inspecting the data to determine their class, when no ground truth is available. In this context, our results suggest that partial class information can be useful, provided labeling uncertainty is suitably represented in the form of soft labels.

Although linear classifiers have been exclusively considered in this study for simplicity, there is obviously no conceptual difficulty with extending our approach to non linear parametric classifiers such as, e.g., quadratic or mixture discriminant analysis, generalized additive models, kernel logistic regression, etc. More fundamentally, other likelihood-based approaches to inference from uncertain data, such has proposed in [START_REF] Hüllermeier | Learning from imprecise and fuzzy observations: Data disambiguation through generalized loss minimization[END_REF] or [START_REF] Couso | Maximum likelihood under incomplete information: Toward a comparison of criteria[END_REF], could be considered and compared with our method. Finally, the elicitation of soft labels from single or multiple experts is also an important topic that remains to be thoroughly investigated.

Davoine for providing the facial expression data. This research was supported by a grant from the Beijing Government as part of the Overseas Talents program.

Fig. 1 :

 1 Fig. 1: Effect of soft labels. Left: crisp labels; two wrongly labeled instances result in an incorrect decision boundary. Right: soft labels; the influence of doubtful instances is decreased.

Fig. 5 :

 5 Fig. 5: vowel (a, c, e) and wine (b, d, f) data. Top: LDA, Middle: LR, Bottom: LDA vs. LR. Black/o: noisy labels, blue/*: adaptive semi-supervised learning, red/x: soft labels based on individual noise level p i , green/+: soft labels based on mean noise level p; plain lines: LDA, dashed lines: logistic regression.

Fig. 6 :

 6 Fig. 6: Six examples of EEG signals. The figures above each plot indicates the classification given of the five experts (1 for K-complex and 0 for delta-wave).

Fig. 7 :

 7 Fig. 7: Performances of LR (a, b) and LDA (c, d) for the sleep data, as a function of the number of principal components. The criteria are the expected loss (a,c) and the error rate taking the majority labels as ground truth (b, d).In each graph, the three curves corresponds to three sets of labels: soft labels, majority labels, and "semi-supervised" labels (see details in text).

Fig. 8 :Fig. 9 :

 89 Fig. 8: Expected losses (a) and error rates (b) for classifiers trained with 12 principal components as input (sleep data). The classifiers are, from left to right: LR with soft labels, majority labels and semi-supervised labels; LDA with soft labels, majority labels and semi-supervised labels.

Fig. 12 :

 12 Fig.12: Facial expression data: test error rates vs. number of principal components for LDA and LR, with soft and crisp labels.

Table 1 :

 1 Real datasets used in Section 5.1.

	dataset	amount of instances number of classes dimension
	Breastcancer	569	2	30
	Ionosphere	350	2	34
	Iris	150	3	4
	Satimage	6435	6	36
	Sonar	208	2	60
	Vowel	990	11	10
	Wine	178	3	13

  Overall, taking into account label uncertainty almost always significantly improves classification accuracy, as compared to noisy la-Bottom: LDA vs. LR. Black/o: noisy labels, blue/*: adaptive semi-supervised learning, red/x: soft labels based on individual noise level p i , green/+: soft labels based on mean noise level p; plain lines: LDA, dashed lines: logistic regression.

		0.55 0.28 0.44 0.75					synthetic1-200-1000, LDA ionosphere, LDA sonar, LDA vowelini, LDA								0.22 0.35 0.25 0.35			breastcancer, LDA iris, LDA satimage, LDA wine, LDA
	empirical average error rate empirical average error rate empirical average error rate empirical average error rate	0.3 0.35 0.4 0.45 0.5 0.18 0.2 0.22 0.24 0.26 0.32 0.34 0.36 0.38 0.6 0.65 0.4 0.42 0.7	noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2							empirical average error rate empirical average error rate empirical average error rate empirical average error rate	0.08 0.1 0.12 0.14 0.16 0.18 0.2 0.1 0.15 0.2 0.25 0.3 0.19 0.2 0.21 0.22 0.1 0.15 0.2 0.25 0.23 0.24 0.3	noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2 noisy LDA adaptive semi-sup. LDA plaus. LDA 1 plaus. LDA 2
		0.55														0.06 0.18
		0.25 0.16 0.3													0.04 0.05 0.17 0.05
		0 0 0 0.2 0.14 0.28 0 0.5	0.1 0.1 0.1 0.1	0.2	0.2 0.2 0.2	0.3	expected label noise 0.4 0.5 0.6 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.7	0.6 0.6 0.6	0.8	0.7 0.7 0.7	0.9	1 0.8 0.8 0.8		0 0 0 0.16 0.02 0 0 0	0.1 0.1 0.1 0.1	0.2 0.2 0.2 0.2	expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.6 0.6 0.6 0.6	0.7 0.7 0.7 0.7	0.8 0.8 0.8 0.8
							(a) (a) (a) (a)													(b) (b) (b) (b)
	0.55 0.35 0.46 0.8			synthetic1-200-1000, logistic regression ionosphere, logistic regression sonar, logistic regression vowelini, logistic regression								0.3 0.4 0.28 0.4			breastcancer, logistic regression iris, logistic regression satimage, logistic regression wine, logistic regression
	0.5 0.42 0.44 0.75	noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2										0.25 0.35 0.26 0.35	noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2 noisy LRg adaptive semi-sup. LRg plaus. LRg 1 plaus. LRg 2
	empirical average error rate empirical average error rate empirical average error rate 0.32 0.3 0.35 0.4 0.45 0.2 0.25 0.3 0.34 0.36 0.38 0.4 empirical average error rate 0.55 0.6 0.65 0.7												empirical average error rate empirical average error rate empirical average error rate empirical average error rate	0.1 0.15 0.2 0.1 0.15 0.2 0.25 0.3 0.16 0.18 0.2 0.22 0.24 0.15 0.2 0.25 0.3
		0.3
	0.25 0.28 0.5															0.05 0.05 0.14 0.1
	0 0 0.2 0.15 0 0.26 0 0.45	0.1 0.1 0.1 0.1	0.2	0.2 0.2 0.2	0.3	expected label noise 0.4 0.5 0.6 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.7	0.6 0.6 0.6	0.8	0.9 0.7 0.7 0.7	1 0.8 0.8 0.8			0 0 0 0 0.05 0 0 0.12	0.1 0.1 0.1 0.1	0.2 0.2 0.2 0.2	expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.6 0.6 0.6 0.6	0.7 0.7 0.7 0.7	0.8 0.8 0.8 0.8
							(c) (c) (c) (c)													(d) (d) (d) (d)
	empirical average error rate empirical average error rate empirical average error rate empirical average error rate	0.3 0.35 0.2 0.25 0.3 0.3 0.32 0.34 0.36 0.38 0.6 0.65 0.7 0.4 0.4 0.45 0.35 0.44 0.8 0.42 0.75	synthetic1-200-1000, LDA vs logistic regression adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 ionosphere, LDA vs logistic regression sonar, LDA vs logistic regression vowelini, LDA vs logistic regression adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1				empirical average error rate empirical average error rate empirical average error rate empirical average error rate	0.1 0.15 0.2 0.1 0.15 0.2 0.25 0.3 0.18 0.2 0.22 0.24 0.1 0.15 0.2 0.25 0.3 0.25 0.4 0.28 0.4 0.3 0.35 0.26 0.35	breastcancer, LDA vs logistic regression iris, LDA vs logistic regression satimage, LDA vs logistic regression wine, LDA vs logistic regression adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1 adaptive semi-sup. LDA plaus. LDA 1 adaptive semi-sup. log. reg. plaus. log. reg. 1
		0.25 0.55 0.28														0.05 0.05 0.16 0.05
		0 0 0 0 0.26 0.2 0.15 0.5	0.1 0.1 0.1 0.1	0.2	0.2 0.2 0.2	0.3	expected label noise 0.4 0.5 0.6 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.7	0.6 0.6 0.6	0.8	0.7 0.7 0.7	0.9	1 0.8 0.8 0.8			0 0 0 0 0 0.14 0 0	0.1 0.1 0.1 0.1	0.2 0.2 0.2 0.2	expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5 expected label noise 0.3 0.4 0.5	0.6 0.6 0.6 0.6	0.7 0.7 0.7 0.7	0.8 0.8 0.8 0.8
							(e) (e) (e) (e)													(f) (f) (f) (f)

Fig. 2: synthetic (a, c, e) and breastcancer (b, d, f) data. Top: LDA, Middle: LR, Bottom: LDA vs. LR. noisy labels, blue/*: adaptive semi-supervised learning, red/x: soft labels based on individual noise level p i , green/+: soft labels based on mean noise level p; plain lines: LDA, dashed lines: logistic regression. Fig. 3: ionosphere (a, c, e) and iris (b, d, f) data. Top: LDA, Middle: LR, Bottom: LDA vs. Black/o: noisy labels, blue/*: adaptive semi-supervised learning, red/x: soft labels based on individual noise level p i , green/+: soft labels based on mean noise level p; plain lines: LDA, dashed lines: logistic regression. Fig. 4: sonar (a, c, e) and satimage (b, d, f) data. Top: LDA, Middle: LR,

Table 2 :

 2 Results of Mardia's test of multivariate normality to the each of the 11 classes in the Vowel dataset. The statistics are: Mardia's multivariate skewness statistic with its χ 2 and p-values, and Mardia's multivariate kurtosis statistic with its z and p-values.

	class skewness χ 2 -value p-value kurtosis z-value p-value
	1	45.9	688	1.11e-49	116	-1.153 2.49e-1
	2	42.0	631	2.70e-41	110	-3.182 1.46e-3
	3	46.0	690	6.06e-50	113	-2.103 3.55e-2
	4	42.5	638	2.30e-42	109	-3.267 1.09e-3
	5	44.3	665	3.48e-46	114	-1.979 4.78e-2
	6	47.2	707	1.53e-52	118	-0.588 5.56e-1
	7	44.0	660	1.65e-45	109	-3.517 4.36e-4
	8	36.0	539	7.91e-29	108	-3.608 3.09e-4
	9	29.1		1.76e-16	109	-3.256 1.13e-3
	10	44.4	666	2.52e-46	113	-2.296 2.17e-2
	11	36.8	552	1.89e-30	115	-1.529 1.26e-1

Table 3 :

 3 Results of Mardia's test of multivariate normality to the each of the 6 classes in the Satimage dataset. The statistics are: Mardia's multivariate skewness statistic with its χ 2 and p-values, and Mardia's multivariate kurtosis statistic with its z and p-values.

	class skewness χ 2 -value p-value kurtosis z-value p-value
	1	344	87861	0	2032	248.4	0
	2	227	26561	0	1644	69.9	0
	3	363	82257	0	2020	229.5	0
	4	321	33475	0	1769	96.0	0
	5	181	21357	0	1613	62.4	0
	6	210	52795	0	1774	150.5	0
	5.2.1 K-complex Detection in EEG Sleep Data

Table 4 .

 4 In total, 116 + 88 + 59 = 263 instances (22.33%) were classified as positive by a majority of experts, while the remaining 915 instances (77.67%) are almost certainly negative, or were not recognized as positive by a majority of experts. Examples of signals with their classification by the five physicians are shown in Figure 6.

Table 4 :

 4 Sleep data: number of instances n k classified in the positive class by k experts.

	k	0	1	2	3	4	5
	n k 397 340 178 116 88 59

  eliminates alternatives

		0.5				Facial expression data
		0.45
		0.4
	Test error rate	0.25 0.3 0.35
		0.2
										LDA Soft labels
		0.15								LDA Crisp labels LR Soft labels
										LR Crisp labels
		0.1
		10	15	20	25	30	35	40	45	50	55	60
					Number of principal components

Acknowledgements The author thanks Cédric Richard and Régis Lengellé for providing the EEG data, as well as the Foundation for Applied Neuroscience Research in Psychiatry (CHS de Rouffach, 68250 Rouffach, France) for authorizing their use. They also thank Franck

As in Section 5.2.1, PCA was used to reduce the dimension of the data. Logistic regression and LDA were then applied to the data with different numbers of principal components, and two sets of labels: the combined soft labels, and crisp labels corresponding to the most plausible expression. For logistic regression, we used quadratic penalization with coefficient λ = 10 -4 . The results are reported in Figure 12, which shows the test error rate as a function of the number of components, for each of the two classifiers and the two sets of labels. For this dataset, LDA clearly outperforms LR, with the lowest error