Sébastien Ramel
email: sebastien.ramel@univ-artois.fr

Frédéric Pichon
email: frederic.pichon@univ-artois.fr

François Delmotte
email: francois.delmotte@univ-artois.fr

A Reliable Version of Choquistic Regression based on Evidence Theory

Keywords: Belief functions, Choquistic regression, Choquet integral, Logistic regression, Monotonic classification, Reliable classification, Epistemic uncertainty, Nonlinear models

Choquistic regression is an elegant generalisation of logistic regression, which preserves its monotonicity whilst alleviating its linearity. However, much as logistic regression, it lacks self-awareness, that is, an ability to represent the ignorance (aka epistemic uncertainty) involved in its predictions, which is crucial in safety-critical classification problems. Recently, an extension of logistic regression was introduced to remedy this issue for this latter classifier. This extension is formalised within evidence theory and relies in particular on a sound method for statistical inference and prediction developed in this framework. In this paper, a similar extension is derived for choquistic regression. The usefulness of the obtained approach is confirmed empirically in classification problems where cautiousness in decisionmaking is allowed.

Introduction

In various practical classification problems such as corporate bond rating [START_REF] Sill | Monotonic networks[END_REF], teaching course evaluation [3] or breast cancer diagnosis [START_REF] Ryu | Breast cancer prediction using the isotonic separation technique[END_REF], it is common to have some prior knowledge that the relation between the input (predictor) variables and the output (target) variable is monotonic, i.e., everything else being equal, the increase of a particular input variable can only increase (or only decrease) the output variable. Accounting for this particular prior knowledge when developing a classification model is important for at least two reasons [START_REF] Sill | Monotonicity hints[END_REF][START_REF] Kotłowski | Rule learning with monotonicity constraints[END_REF][START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF][START_REF] Barile | Studies in Learning Monotonic Models from Data[END_REF][START_REF] Cano | Monotonic classification: An overview on algorithms, performance measures and data sets[END_REF]: [START_REF] Barile | Studies in Learning Monotonic Models from Data[END_REF] it may be beneficial for model induction and in particular to improve the accuracy of predictions; [START_REF] Calamai | Projected gradient methods for linearly constrained problems[END_REF] only models satisfying the monotonicity constraint (for short, monotonic models) may be considered acceptable by the end-users. This topic of so-called monotonic classification is not new, yet it has received an increasing interest in the last few years as shown recently in [START_REF] Cano | Monotonic classification: An overview on algorithms, performance measures and data sets[END_REF].

Focusing on (monotonic) binary classification, where the target variable takes one of two values (the positive class and the negative class), a well-established model for this task is logistic regression [START_REF] Hosmer | Applied logistic regression[END_REF]. It obtains the probability of the positive class (and thus of the negative class) by modelling the log-odds of the positive class as a linear function of the predictors. Its monotonicity together with its interpretability has contributed to its popularity as underlined in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]. Yet, logistic regression has at least two limitations, which impedes its use in some real classification problems.

First, the model for the log-odds of the positive class being linear in the predictors, there is a lack of flexibility from a learning point of view with respect to the possibility of interactions between the predictors. This issue can be overcome by replacing the linear model with more complex models that are nonlinear in the predictors [START_REF] Jaccard | Interaction Effects in Logistic Regression[END_REF]. However, this increased flexibility may come at the expense of losing monotonicity and of affecting interpretability. For instance, kernel logistic regression [START_REF] Zhu | Kernel logistic regression and the import vector machine[END_REF] with polynomial and Gaussian kernels are flexible extensions of logistic regression but they are not necessarily monotone [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]. A notable exception, though, is the proposal of Fallah Tehrani et al. [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF], which uses the Choquet integral [START_REF] Choquet | Theory of capacities[END_REF] for modelling the log-odds of the positive class. The obtained classification model, called choquistic regression, generalises the logistic regression, guarantees monotonicity and provides flexibility in terms of modelling nonlinear relationships between predictors and the log-odds of the positive class. Moreover, it benefits from measures defined for the Choquet integral making it possible to quantify the importance of each predictor as well as the interaction between predictors, and thus it has some level of interpretability. Needless to say, there is a price to pay for the greater flexibility of choquistic regression, which is a higher computational complexity than that of logistic regression. Yet, this complexity may still be acceptable for some real problems.

Secondly, the uncertainty in a given prediction being modelled by a probability measure (characterised by a single number, such as the probability of the positive class), there is a lack of flexibility from a representational point of view with respect to the different possible sources of the uncertainty [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF][START_REF] Xu | Evidential calibration of binary SVM classifiers[END_REF]. In particular, logistic regression fails to provide a quantification of the ignorance involved in its prediction, ignorance (aka epistemic or reducible uncertainty) being the part of the uncertainty caused by a lack of knowledge, such as a limited amount of training data [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF]. Basically, logistic regression lacks the ability of "knowing what it knows and what not" or, for short, "self-awareness" [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF]. Such an ability may be important in critical classification problems such as medical diagnosis, where it may be used to, e.g., postpone the ultimate decision until further data is acquired. This issue of so-called reliable classification [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF][START_REF] Corani | Learning reliable classifiers from small or incomplete data sets: The naive credal classifier 2[END_REF] can be addressed by replacing the probabilistic representation of uncertainty with richer uncertainty representations. Two noteworthy proposals in this respect, extending logistic regression, are those of Senge et al. [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF] and of Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF]. Both are grounded in probability and statistics, but the former is formalised within the framework of fuzzy preference modelling [START_REF] Fodor | Fuzzy Preference Modelling and Multicriteria Decision Support[END_REF] whereas the latter is formalised within evidence theory (aka theory of belief functions or Dempster-Shafer theory) [START_REF] Shafer | A mathematical theory of evidence[END_REF]. They were applied to two different problems: medical diagnosis and binary SVM classifier stacking, respectively. From a formal point of view, the uncertainty representation in each of these two methods is characterised by two numbers (instead of the single number of the conventional logistic regression) and makes it possible to quantify the ignorance in a given prediction. Obviously, these finer representations of the uncertainty involve higher computational costs than that of logistic regression, but these costs may still be tolerable in some cases.

Choquistic regression is a sound, elegant and useful generalisation of logistic regression, allowing flexibility in the modelling of interactions between predictors, while maintaining monotonicity and interpretability. However, it suffers from the same limitation as logistic regression with respect to selfawareness. Therefore, it seems interesting to try and extend to choquistic regression the approach of Senge et al. [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF] or of Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF]. In this paper, a first study in this direction is reported, where the approach of Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] is extended to choquistic regression. We focused on this approach (rather than the one of Senge et al. [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF]) because it relies on a recent method for statistical inference and prediction proposed in [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF], which has gained momentum and is already quite well developed on both the theoretical and practical sides (see, e.g., [START_REF] Min | Forecasting tourist arrivals to thailand using belief functions[END_REF][START_REF] Phochanachan | Estimating oil price value at risk using belief functions[END_REF][START_REF] Kanjanatarakul | Prediction of future observations using belief functions: A likelihood-based approach[END_REF][START_REF] Minary | Face pixel detection using evidential calibration and fusion[END_REF][START_REF] Denoeux | Quantifying predictive uncertainty using belief functions: different approaches and practical construction[END_REF][START_REF] Chakpitak | Comparing linear and nonlinear models in forecasting telephone subscriptions using likelihood based belief functions[END_REF][START_REF] Denoeux | Multistep prediction using point-cloud approximation of continuous belief functions[END_REF] for recent results around this method, including a few successful applications). Another reason for this focus is the study conducted in the context of active learning in [START_REF] Ramel | Active evidential calibration of binary SVM classifiers[END_REF], which provides an additional experimental argument that the quantification of ignorance permitted by the approach of [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] can be a useful measure of the lack of knowledge.

The rest of this paper is organised as follow. The logistic regression model and its generalisation based on the Choquet integral due to [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF], are recalled in Section 2. Then, in Section 3, necessary notions of evidence theory are provided and the extension of logistic regression based on this theory, introduced in [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF], is presented. This latter extension is carried over to the more general choquistic regression model in Section 4. The resulting evidential extension of choquistic regression is compared experimentally in Section 5 to choquistic regression and to the evidential extension of logistic regression. Finally, Section 6 concludes the paper.

Choquistic regression

In this section, logistic regression is first recalled. Then, necessary material on the Choquet integral is provided and an extension of logistic regression relying on this integral is presented.

Logistic regression

As already mentioned, logistic regression (LR) [START_REF] Hosmer | Applied logistic regression[END_REF] is a model commonly used in binary classification problems. For an object (instance) with unknown label (class or target) y ∈ Y = {0, 1} and observed feature (predictor) vector x = (x1, . . . , xm) ∈ X = R m , and given a training set D = {x (i) , y (i) } n i=1 of n objects supposed to be an i.i.d. sample from an underlying (unknown) probability measure PXY , LR yields a probability P (y = 1|x) for the positive class (and thus also of the negative class) given the predictors.

LR assumes a monotone relationship of the probability with respect to the predictors, in the sense that every thing else being equal in the predictors, an increase of the value of an individual predictor xj can either only increase or only decrease the probability.

More specifically, LR is obtained by assuming that the log-odds of y = 1 given the predictor vector x is a linear function of the predictors:

log P (y = 1|x) P (y = 0|x) = σ0 + σ m x, (1)
where σ0 ∈ R is a bias (the intercept) and σm = (σ1, . . . , σm) ∈ R m is a vector of regression coefficients. According to Eq. ( 1), a positive (resp. negative) coefficient σj > 0 implies that an increase of xj will necessarily increase (resp. decrease) the probability of the positive class. Besides, from (1), one easily obtains

π l := P (y = 1|x) = h(σ0 + σ m x), (2)
where

h(z) = (1 + exp(-z)) -1 , ∀z ∈ R,
is known as the logistic function.

Let σ = (σ0, σm) ∈ Σ = R m+1 denote the parameters of the LR model defined by [START_REF] Calamai | Projected gradient methods for linearly constrained problems[END_REF]. The actual values σ of the parameters σ to be used in [START_REF] Calamai | Projected gradient methods for linearly constrained problems[END_REF] are determined thanks to the training set D. More precisely, they are chosen as the ones maximising the conditional likelihood function:

LD(σ) = n i=1 P (y (i) |x (i) ) = n i=1 (π (i) l ) y (i) • (1 -π (i) l ) 1-y (i) , ∀σ ∈ Σ. (3)
In practice, the conditional log-likelihood function

D (σ) = log LD(σ) = n i=1 y (i) log π (i) l + (1 -y (i) ) log(1 -π (i) l ), ∀σ ∈ Σ,
is maximised to find σ since it has a more convenient form than (3) and its maximum is also attained for σ = σ. Formally, we have:

σ = arg max σ∈Σ D (σ), ( 4
)
and σ is called a maximum likelihood estimate (MLE) of σ. These instances are illustrated in Figure 1. Let us consider an object (also shown in Fig. 1) with observed feature vector x = (2, -1) and unknown label y ∈ Y = {0, 1}. The probability P (y = 1|x) of the positive class for this object given its predictors may then be obtained using LR, which essentially amounts to finding a MLE σ = ( σ0, σ1, σ2) of σ using (4) and plugging it in Eq. (2): we find σ ≈ (0.140, 1.222, 0.848), which induces P (y = 1|x) ≈ 0.85. Using the classical (Bayesian) decision strategy of minimising the expected loss [START_REF] Duda | Pattern recognition and scene analysis[END_REF], the decision reached for the label y of this object is y = 1 in the case where the 0/1 loss function (a wrong decision costs 1 and a correct decision costs nothing) is used. More generally, Figure 1 shows the decision boundary of LR in the case of 0/1 loss, which corresponds to those x ∈ X such that P (y = 1|x) = 0.5.

The Choquet integral

The Choquet integral [START_REF] Choquet | Theory of capacities[END_REF] allows the integration of functions with respect to non-additive measures, also known as capacities.

Definition 1 (Capacity

). Let C = {c1, • • • , cm} be a finite set. A capacity is a set function µ : 2 C → [0, 1]
satisfying monotonicity, i.e., µ(A) ≤ µ(B) for all A ⊆ B ⊆ C, and normalisation, i.e., µ(∅) = 0 and µ(C) = 1.

Cµ(g) = m i=1 g(c (i) ) -g(c (i-1) ) • µ(A (i) ), (5)
where

A (i) = {c (i) , . . . , c (m) }, g(c (0) ) = 0 and (•) is a permutation of {1, . . . , m} such that 0 ≤ g(c (1) ) ≤ g(c (2) ) ≤ . . . ≤ g(c (m) ).
Besides its use in classification (recalled in Section 2.3), the Choquet integral plays a role in other domains. For instance, computing the lower expectation of a function of an uncertain variable, when uncertainty is modelled by a belief function (which is a particular kind of non-additive measures for the representation of uncertainty) corresponds to computing the Choquet integral of this function with respect to the belief function [START_REF] Gilboa | Additive representations of non-additive measures and the Choquet integral[END_REF]. However, above all, its prominent field of application has been that of multiple criteria decision making (MCDM), where it is used as an aggregation operator (the nonadditivity allowing to account for interactions between criteria) [START_REF] Grabisch | A decade of application of the Choquet and Sugeno integrals in multi-criteria decision aid[END_REF] as briefly recalled in the following.

When using the Choquet integral in MCDM, C represents a set of relevant criteria to compare different alternatives, µ(A) is interpreted as the importance of the subset of criteria A ⊆ C, g(ci) ∈ [0, 1] is the utility of a given alternative with respect to criterion ci indicating the degree to which ci is satisfied for this alternative, and Cµ(g) represents the overall evaluation (satisfaction) of this alternative. For instance, if one is contemplating buying a house, the alternatives are the different houses on offer, the set of criteria might be C = {living area, garden size, number of rooms} and µ(A) is the evaluation of an alternative satisfying criteria A (and not satisfying C\A). Besides, interactions between criteria are accounted for through µ: if subsets of criteria A ⊆ C and B ⊆ C can be seen as complementary (e.g., A = {living area} and B = {garden size} since both areas are important on the final decision) then this is formally expressed by µ(A ∪ B) > µ(A) + µ(B) (positive interaction), whereas if A and B are redundant (e.g., A = {living area} and B = {number of rooms} since either of the two may suffice), this is represented by µ(A ∪ B) < µ(A) + µ(B) (negative interaction).

The Choquet integral admits a useful representation based on the so-called Möbius transform.

Definition 3 (Möbius transform). The Möbius transform mµ of a capacity µ is given by

mµ(A) = B⊆A (-1) |A|-|B| µ(B), ∀A ⊆ C.
Remarking that mµ(∅) = 0, we denote by mµ the vector of size 2 m-1 whose elements are the values mµ(A) ∈ R, ∅ = A ⊆ C, ordered according to some arbitrary order on the subsets of C.

A capacity µ can be recovered from its Möbius transform mµ by:

µ(A) = B⊆A mµ(B), ∀A ⊆ C. ( 6
)
Using [START_REF] Choquet | Theory of capacities[END_REF], it is easy to show that the Choquet integral (5) of a function g with respect to the capacity µ satisfies:

Cµ(g) = A⊆C mµ(A) • min c i ∈A g(ci). ( 7
)
An interesting property that a capacity µ may satisfy is k-order additivity (or, simply, k-additivity): if k is the smallest integer such that mµ(A) = 0 for all A ⊆ C with |A| > k, then µ is said to be kadditive. In MCDM, this property means an absence of interaction between subsets of criteria A, B ⊂ C such that |A| > k and |B| > k. It also implies that µ is then characterised by less than the 2 m-1 values required in the general case.

The choquistic regression model

In [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF], it is proposed to extend the LR model by replacing σ0 + σ m x in (1) by

γ (Cµ(gx) -β) , (8)
which yields

πc := P (y = 1|x) = h(γ (Cµ(gx) -β)), (9)
where γ and β are two parameters such that γ > 0 and β ∈ [0, 1], and where Cµ(gx) is the Choquet integral with respect to measure µ of the function

gx : C → [0, 1]
that maps, for a given object with feature value vector x = (x1, . . . , xm) , each feature ci (viewing C as the set of features describing objects) to a value xi = gx(ci) ∈ [0, 1] corresponding to a normalisation of the feature value xi.

Assuming that some prior knowledge on the monotonicity of the classification problem at hand is available and, more specifically, that the direction of the influence of each input feature xi on the probability of the positive class is known, then the following normalisation can be used [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]: if the influence is positive (increasing), then

xi = xi -mi Mi -mi , ( 10
)
with mi and Mi the lower and upper bounds for xi, which are either known or estimated from the learning set D as mi = min 1≤j≤n x (j) i

and Mi = max 1≤j≤n x (j) i ; and if the influence is negative (decreasing), then xi = (Mi -xi)/(Mi -mi). In the case where the direction of the influence of predictor xi is actually not known, then it is estimated from the data using LR [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]: if σi > 0 then the influence is assumed to be positive, otherwise it is considered to be negative.

In a nutshell, this approach, called choquistic regression (CR), proceeds in two steps. First, an aggregated value Cµ(gx) ∈ [0, 1] is obtained for an instance x. Then, this value is compared to the threshold β. If Cµ(gx) > β, then P (y = 1|x) > 0.5 ("the decision tends to be positive" [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]), whereas if Cµ(gx) < β, then P (y = 1|x) < 0.5 (the decision "tends to be negative" [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]). Furthermore, the parameter γ acts as a scaling factor. If Cµ(gx) > β, then as γ increases, P (y = 1|x) increases, whereas if Cµ(gx) < β, then as γ increases, P (y = 1|x) decreases. In addition, let us remark that this model is a proper generalisation of LR (we refer the interested reader to [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]Section 5.3]).

Let υ = (γ, mµ, β) denote the parameters of the CR model defined by [START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF], where mµ is the Möbius transform of µ. Similarly as in the case of LR, it is proposed in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF] to determine the actual values υ for the parameters υ using the maximum likelihood principle. The conditional likelihood of the CR parameters is given by

LD(υ) = n i=1 (π (i) c ) y (i) • (1 -π (i) c ) 1-y (i) . ( 11
)
Hence, the conditional log-likelihood function can be written as:

D (υ) = n i=1 y (i) log π (i) c + (1 -y (i) ) log(1 -π (i) c ). ( 12
)
Unlike the coefficients σ of LR, the parameters υ of CR must respect some constraints and in particular mµ must be the Möbius transform of some proper capacity µ. Therefore the maximisation of the conditional log-likelihood ( 12) is actually a constrained optimisation problem, which may be formally written as [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]:

υ = arg max υ∈Υ D (υ) (13)
with Υ the set composed of the vectors (γ, mµ, β) satisfying

β ∈ [0, 1], γ ∈ R>0, A⊆C mµ(A) = 1 (14)
and

B⊆A\{c i } mµ(B ∪ {ci}) ≥ 0, ∀A ⊆ C, ∀ci ∈ C. ( 15
)
Example 2. Continuing Example 1, the probability P (y = 1|x) of the positive class for the object with feature vector x = (2, -1) may also be obtained using CR, which amounts to finding a MLE υ of υ using [START_REF] Denoeux | Decision-making with belief functions: a review[END_REF] and plugging it in (9): we find mµ({1}) ≈ 0.020, mµ({2}) = 0, mµ({1, 2}) ≈ 0.980, γ ≈ 15.077, β ≈ 0.426, which induces P (y = 1|x) ≈ 0.59. Similarly as for LR, the decision reached for the label y of this object using CR is y = 1 in the case of 0/1 loss. More generally, Figure 2 shows the decision boundary of CR in the case of 0/1 loss.

Let us remark that CR is quite a flexible model and runs thus the risk of overfitting the data. A solution proposed in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF] to mitigate this issue is to restrict the capacity µ to be k-additive, for some k < m determined through cross-validation.

Evidential logistic regression

An extension of LR based on evidence theory is recalled in this section. For the paper to be self-contained, necessary notions -in particular an approach for statistical inference and prediction -of the theory of evidence on which this extension relies, are provided first.

Basic notions of the theory of evidence

Evidence theory is a framework for uncertainty modelling and reasoning. Let Y be a variable whose actual value y belongs to some finite set Y = {y1, . . . , yK } (called the frame of discernment). Uncertainty about y is represented in this theory by a mapping Several functions are in one-to-one correspondence with m Y and, in particular, the belief and plausibility functions defined, respectively, for all A ⊆ Y as

m Y : 2 Y → [0, 1], called mass function, such that A⊆Y m Y (A) = 1, (16)
Bel Y (A) = B⊆A m Y (B), and P l Y (A) = B∩A =∅ m Y (B). ( 17
)
The degree of belief Bel Y (A) is interpreted as the amount of evidence strictly supporting y ∈ A, while the plausibility P l Y (A) represents the amount of evidence not contradicting y ∈ A. Due to the one-to-one correspondence between these functions, m Y can be recovered from, e.g., Bel Y . We have

m Y (A) = B⊆A (-1) |A|-|B| Bel Y (B), ∀A ⊆ Y. (18)
The contour function pl Y is defined as the plausibility function restricted to the singletons, i.e., pl Y (yi) = P l Y ({yi}), for all yi ∈ Y. When its focal sets are nested, a mass function m Y is said to be consonant and it is fully characterised by its contour function. Its associated plausibility function can then be recovered as follows:

P l Y (A) = sup y i ∈A pl Y (yi), ∀A ⊆ Y. (19)
There exist several strategies to make a decision about the actual value y of Y given a mass function m Y on Y representing uncertainty about y [START_REF] Denoeux | Analysis of evidence-theoretic decision rules for pattern classification[END_REF][START_REF] Denoeux | Decision-making with belief functions: a review[END_REF][START_REF] Ma | Making set-valued predictions in evidential classification: A comparison of different approaches[END_REF]. Some yield systematically a precise decision, i.e., y = yi for some yi ∈ Y, while other may yield only an imprecise decision, i.e., y ∈ A for some A ⊆ Y, when there is too much uncertainty. In the particular case where the frame of discernment is binary (by convention, in this paper, Y = {0, 1}) and 0/1 loss is used, then the classical precise decision strategies (i.e., minimising the pignistic, or lower, or upper expected loss, see [START_REF] Denoeux | Analysis of evidence-theoretic decision rules for pattern classification[END_REF]) amount to the same decision rule:

y = 1 if m Y ({1}) > m Y ({0}), 0 otherwise. ( 20
)
Among the imprecise decision strategies, the interval dominance decision rule has recently shown its interest in exploiting evidential classifiers [START_REF] Denoeux | Logistic regression, neural networks and Dempster-Shafer theory: a new perspective[END_REF]. In the binary case and with 0/1 loss, this rule reads as follows:

y = 1 if Bel Y ({1}) > P l Y ({0}), 0 if Bel Y ({0}) > P l Y ({1}), (21)
and y ∈ {0, 1} otherwise.

Belief function-based approach to statistical inference and prediction

Shafer [START_REF] Shafer | A mathematical theory of evidence[END_REF] introduced a belief function-based approach to statistical inference, which was subsequently justified by Denoeux [START_REF] Denoeux | Likelihood-based belief function: justification and some extensions to low-quality data[END_REF]. Given an observed realisation z ∈ Z of some random quantity Z ∼ PZ (•; θ) with θ ∈ Θ the unknown parameter of interest, this approach represents knowledge about θ by a consonant belief function Bel Θ z whose contour function pl Θ z is the relative likelihood:

pl Θ z (θ) = Lz(θ) Lz( θ) , ∀ θ ∈ Θ,
where

Lz(θ) = PZ (z; θ), θ is a MLE of θ and it is assumed that Lz( θ) < ∞.
Bel Θ z is called the likelihood-based belief function. Its focal sets are the level-sets of pl Θ z defined as

Γz(u) = {θ ∈ Θ | pl Θ z (θ) ≥ u}, ( 22
)
for u ∈ [0, 1]. Moreover, Bel Θ z can be regarded as being induced by the random set [START_REF] Nguyen | On random sets and belief functions[END_REF] Γz(U ) with

U ∼ U ([0, 1]), in the sense that Bel Θ z (A) = PU ({u ∈ [0, 1]|Γz(u) ⊆ A}), ∀A ⊆ Θ.
In [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF][START_REF] Kanjanatarakul | Prediction of future observations using belief functions: A likelihood-based approach[END_REF], this likelihood-based approach to statistical inference was extended to the prediction problem, which consists in making statements about a not-yet-observed realisation y ∈ Y of some random quantity Y ∼ PY (•; θ) given knowledge about θ obtained by observing z (represented here by Bel Θ z ). The extension relies on Dempster's sampling model [START_REF] Dempster | New methods for reasoning towards posterior distributions based on sample data[END_REF], which expresses Y as a function ϕ of the parameter θ and some unobserved variable V with known probability distribution PV independent of θ:

Y = ϕ(θ, V ). ( 23
)
In [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF][START_REF] Kanjanatarakul | Prediction of future observations using belief functions: A likelihood-based approach[END_REF] (see, also, [START_REF] Denoeux | Quantifying predictive uncertainty using belief functions: different approaches and practical construction[END_REF]), the function ϕ is obtained by inverting the cdf of Y and V ∼ U ([0, 1]).

In this scheme, variables U and V are independent and thus the distribution PU,V of (U, V ) is the uniform distribution on [0, 1] 2 . Besides, for any (u, v) ∈ [0, 1] 2 , we can assert from ( 22) and ( 23) that Y ∈ ϕ(Γz(u), v). Accordingly, knowledge about the future realisation y given the observed data z may be represented by the belief function Bel Y z induced by the random set ϕ(Γz(U ), V ) and defined as [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF][START_REF] Kanjanatarakul | Prediction of future observations using belief functions: A likelihood-based approach[END_REF]]

Bel Y z (A) = PU,V {(u, v) ∈ [0, 1] 2 | ϕ(Γz(u), v) ⊆ A} , ∀A ⊆ Y. ( 24
)
Bel Y z is called the predictive belief function. Its associated plausibility function P l Y z is defined by

P l Y z (A) = PU,V {(u, v) ∈ [0, 1] 2 | ϕ(Γz(u), v) ∩ A = ∅} , ∀A ⊆ Y. (25)
This approach to statistical inference and prediction is illustrated by Example 3, where Y is a Bernoulli variable, as this is the case on which the evidential extension of LR recalled in the next section relies.

pl Θ z (θ) = θ z (1 -θ) n-z θz (1 -θ) n-z = θ θ n θ • 1 -θ 1 -θ n(1-θ) , ∀θ ∈ [0, 1], 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 θ pl Θ z (θ) m Y z ({1}) m Y z (Y) m Y z ({0}) (a) n = 15 0 0.2 0.4 0.6 0.8 1 0 0.2 0.4 0.6 0.8 1 θ m Y z ({1}) m Y z (Y) m Y z ({0}) (b) n = 30
Y = ϕ(θ, V ) = 1, if V ≤ θ, 0, otherwise, with V ∼ U ([0, 1]).
Using the fact that pl Θ z is continuous and unimodal (as illustrated by Fig. 3), it can be shown (see [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF]Example 2]) that the expressions of the predictive belief and predictive plausibility that y = 1, given by [START_REF] Kanjanatarakul | Prediction of future observations using belief functions: A likelihood-based approach[END_REF] and (25) for A = {1}, reduce respectively to

Bel Y z ({1}) = θ - θ 0 pl Θ z (θ)dθ, (26)
P l Y z ({1}) = θ + 1 θ pl Θ z (θ)dθ. ( 27
)
For instance, for n = 15 and θ =

m Y z ({1}) = Bel Y z ({1}), m Y z ({0}) = 1 -P l Y z ({1}), m Y z (Y) = P l Y z ({1}) -Bel Y z ({1}) = 1 0 pl Θ z (θ)dθ.
Each mass m Y z (A), A ⊆ Y, corresponds to a particular area with respect to the function pl Θ z , as illustrated by Fig. 3. In particular, m Y z (Y), which represents the amount of belief, called ignorance [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF], that cannot be committed to any specific hypothesis, is equal to the area under the contour function pl Θ z . Let us remark that the size of this area tends to 0 as n tends to infinity [START_REF] Kanjanatarakul | Forecasting using belief functions: An application to marketing econometrics[END_REF]. In other words, m Y z (Y) reflects the amount of data: the more data there are, the less ignorance there is.

The evidential logistic regression model

Following previous work from Xu et al. [START_REF] Xu | Evidential calibration of binary SVM classifiers[END_REF], Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] introduced a belief function-based extension of LR, called hereafter evidential logistic regression (ELR), which they used as a meta-classifier to combine classifiers' outputs. In ELR, the label y ∈ Y = {0, 1} of an object whose feature vector x has been observed, is seen as the realisation of a binary random variable Y following a Bernoulli distribution with parameter θ = h(σ0 + σ m x), where there is some uncertainty on σ = (σ0, σm) induced by the observation of the training data D. The representation of the uncertainty on σ and the subsequent prediction of Y are carried out using the approach for statistical inference and prediction recalled in the preceding section, therefore yielding a belief degree Bel Y D,x ({1}) and a plausibility degree P l Y D,x ({1}) (rather than a probability P (y = 1|x) as in LR) for the positive class given the predictors.

More precisely, Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] represent uncertainty on σ by the likelihood-based (consonant) belief function Bel Σ D whose contour function is defined by

pl Σ D (σ) = LD(σ) LD( σ) , ∀σ ∈ Σ, (28)
and whose corresponding plausibility function is obtained as

P l Σ D (A) = sup σ∈A pl Σ D (σ), ∀A ⊆ Σ.
Then, the uncertainty with respect to θ induced by the uncertainty on σ and the observed feature vector x, is represented by a consonant belief function Bel Θ D,x with contour function pl Θ D,x obtained as

pl Θ D,x (θ) = 0 if θ ∈ {0, 1}, P l Σ D ({σ ∈ Σ | θ = h(σ0 + σ m x)}) otherwise,
where

{σ ∈ Σ | θ = h(σ0 + σ m x)} = σ ∈ Σ | -(σ0 + σ m x) = ln(θ -1 -1) = σ ∈ Σ | σ0 = -ln(θ -1 -1) -σ m x , which yields pl Θ D,x (θ) = sup σm∈R m pl Σ D -ln(θ -1 -1) -σ m x, σm , ∀θ ∈ (0, 1). ( 29
)
The value pl Θ D,x (θ), θ ∈ (0, 1), can be obtained by an iterative maximisation algorithm. Finally, the predictive belief Bel Y D,x ({1}) and predictive plausibility P l Y D,x ({1}) of the positive class given the predictors are obtained from the contour function pl Θ D,x using Eqs. ( 26) and [START_REF] Lachaize | Evidential framework for error correcting output code classification[END_REF], respectively. Note that, using Eqs. ( 16)-( 18), the uncertainty with respect to the label y of an object whose feature vector x has been observed, can then be equivalently characterized by some other pairs of quantities than the belief degree Bel Y D,x ({1}) and the plausibility degree P l Y D,x ({1}), such as the mass m Y D,x ({1}) supporting the positive class together with the mass m Y D,x (Y) representing the amount of ignorance. Example 4. In order to be able to provide graphical representations of the inner workings of ELR, we will consider, only in this example, a slightly simplified version of ELR where a degree of freedom of the model is removed. Specifically, we consider the case where the LR extended by ELR is LR with the bias term σ0 fixed (arbitrarily) to σ0 = 0. In this case, considering the classification problem with training dataset D of Example 1, LR is defined with coefficients σm = (σ1, σ2) ∈ R 2 .

The steps to obtain the predictive belief Bel Y D,x ({1}) and predictive plausibility P l Y D,x ({1}) for the object with observed feature vector x = (2, -1) are now illustrated, where the restriction σ0 = 0 holds throughout the rest of the example. The levels sets of the contour function pl Σ D defined by [START_REF] Ma | Making set-valued predictions in evidential classification: A comparison of different approaches[END_REF] are shown in Figure 4a. From (29) (and σ0 = 0), the value of pl Θ D,x (θ) is defined as the maximum of pl Σ D along the line :

σ2 = (-ln(θ -1 -1) -σ1x1)/x2.
This line is shown for different values of θ in Figure 4a. Figure 4b shows the resulting contour function pl Θ D,x for feature vector x = (2, -1), from which Bel Y D,x ({1}) and P l Y D,x ({1}) are obtained using Eqs. ( 26) and ( 27) respectively. We find:

Bel Y D,x (
m Y D,x ({1}) = 0.5655, m Y D,x (Y) = 0.3923, m Y D,x ({0}) = 0.0422.
Using the interval dominance decision rule with m Y D,x and 0/1 loss, one obtains that the decision for the label y of the object having feature vector x = (2, -1) is y = 1. For this instance, the uncertainty is thus too low to yield an imprecise decision. A precise decision is reached, which is the same decision as the one obtained with LR. Nonetheless, let us remark that in general, decisions made using LR and ELR may differ as illustrated in Figure 5b, which shows the decision regions (in the case of 0/1 loss) of ELR and in particular the fact that it may yield imprecise decisions. For completeness, Figure 5c shows the degree of ignorance yielded by ELR for any x ∈ X ; we can remark that the areas where ignorance is the greatest correspond to the areas where imprecise decisions are made.

Evidential choquistic regression

In this section, a belief function-based extension of the CR model is proposed, following the approach of Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] to extend the LR model that was recalled in Section 3.3. This extension of CR, derived in Section 4.2, is presented using a reparameterisation of CR in a linear form provided in Section 4.1. As will be seen, this reparameterisation offers several advantages. Furthermore, in Section 4.3, the computational complexity of this extension of CR is analysed.

Reparameterised choquistic regression

The CR model may be rewritten in a linear form as follows. From [START_REF] Corani | Learning reliable classifiers from small or incomplete data sets: The naive credal classifier 2[END_REF], expression [START_REF] Dempster | New methods for reasoning towards posterior distributions based on sample data[END_REF] verifies

γ (Cµ(gx) -β) = γ A⊆C mµ(A) • min c i ∈A gx(ci) -β = -γβ + A⊆C γ • mµ(A) • min c i ∈A gx(ci) = -γβ + A⊆C,A =∅ γ • mµ(A) • min c i ∈A gx(ci). ( 30
)
Let ψ : 2 C → R and φx : 2 C → R be the mappings defined, respectively, as

ψ(∅) = -γβ, ψ(A) = γ • mµ(A), ∀A ⊆ C, A = ∅, and
φx(∅) = 1, φx(A) = min c i ∈A gx(ci), ∀A ⊆ C, A = ∅.
Then, from [START_REF] Minary | Face pixel detection using evidential calibration and fusion[END_REF], we obtain

γ (Cµ(gx) -β) = A⊆C ψ(A) • φx(A) = ψ φ x , (31)
with ψ and φ x the column vectors whose elements are, respectively, the values ψ(A) and φx(A), A ⊆ C, ordered according to the same (arbitrary) order on the subsets of C. Let us note that the original parameters υ = (γ, mµ, β) of CR can be recovered from the alternative parameters ψ as follows:

γ = A⊆C,A =∅ ψ(A), β = -ψ(∅) A⊆C,A =∅ ψ(A) , mµ(A) = ψ(A) B⊆C,B =∅ ψ(B) , ∀A ⊆ C, A = ∅, using the fact that µ(C) = 1 = B⊆C,B =∅ mµ(B).
Under this reparametrisation, fitting the CR model amounts to determining the actual values ψ of the parameters ψ. The conditional likelihood of these alternative parameters is given by

LD(ψ) = n i=1 (h(ψ φ x (i) )) y (i) • (1 -h(ψ φ x (i) )) (1-y (i) ) , ( 32
)
and the log-likelihood is then

D (ψ) = n i=1 y (i) log(h(ψ φ x (i) )) + (1 -y (i) ) log(1 -h(ψ φ x (i) )).
Given the constraints ( 14) and ( 15) as well as the relations between parameters υ and ψ, the values ψ are obtained by solving the following constrained optimisation problem:

ψ = arg max ψ∈Ψ D (ψ) (33)
with Ψ the set composed of the vectors ψ ∈ R 2 m satisfying

A⊆C,A =∅ ψ(A) ≥ -ψ(∅) ≥ 0, A⊆C,A =∅ ψ(A) > 0,
and

B⊆A\{c i } ψ(B ∪ {ci}) ≥ 0, ∀A ⊆ C, ∀ci ∈ C.
In other words, and as already remarked in [17, Section 6], CR can be seen as fitting a (constrained) linear function in the feature space spanned by the set of features {φx(A)|A ⊆ C}. From a formal point of view, this reparametrisation presents thus CR under a similar form to that of LR. This is particularly interesting as it makes it possible to extend straightforwardly to CR both the approach followed by Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] to derive ELR and some other useful results related to LR, as will be seen in the next section. Furthermore, from a practical of view, let us remark that we have observed (for instance, when running the experiments reported in Section 5) that solving the optimisation problem [START_REF] Nadeem | Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option[END_REF] is on average an order of magnitude faster than solving the optimisation problem (13) when using the fmincon function of Matlab (function reportedly used in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF] to solve ( 13)).

The evidential choquistic regression model

In this section, an evidential extension of CR, called evidential choquistic regression (ECR), is derived. This extension may be obtained equivalently using the CR model based on the expression [START_REF] Dempster | New methods for reasoning towards posterior distributions based on sample data[END_REF] or based on the linear form [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF]. Since this latter expression has the advantages mentioned in the preceding section, it is preferred.

Similarly to the ELR model recalled in Section 3.3, we propose to see the label y ∈ Y = {0, 1} of an object whose feature vector x has been observed, as the realisation of a binary random variable Y following a Bernoulli distribution with parameter θ = h(ψ φ x ), where there is some uncertainty on ψ induced by the observation of the training data D. This latter uncertainty is represented by the likelihood-based (consonant) belief function Bel Ψ D whose contour function is defined by

pl Ψ D (ψ) = LD(ψ) LD( ψ) , ∀ψ ∈ Ψ,
and whose corresponding plausibility function is obtained as

P l Ψ D (A) = sup ψ∈A pl ψ D (ψ).
Then, the uncertainty with respect to θ induced by the uncertainty on ψ and the observed feature vector x, is represented by the consonant belief function Bel Θ D,x whose contour function is given by

pl Θ D,x (θ) = 0 if θ ∈ {0, 1}, P l Ψ D ({ψ ∈ Ψ | θ = h(ψ φ x )}) otherwise,
where

{ψ ∈ Ψ | θ = h(ψ φ x )} = ψ ∈ Ψ | -ψ φ x = ln(θ -1 -1) = ψ ∈ Ψ | ψ(∅) = -ln(θ -1 -1) - A⊆C,A =∅ ψ(A) • φx(A) , which yields pl Θ D,x (θ) = sup ψ * ∈R 2 m -1 pl Ψ D -ln(θ -1 -1) - A⊆C,A =∅ ψ(A) • φx(A), ψ * , ∀θ ∈ (0, 1), (34)
with ψ * the vector whose elements are the values ψ(A), A ⊆ C, A = ∅. The value pl Θ D,x (θ), θ ∈ (0, 1), can be obtained by an iterative maximisation algorithm.

Similarly as for ELR, the prediction of Y given this uncertainty on θ is carried out using the approach for prediction recalled in Section 3.2. Specifically, the predictive belief Bel Y D,x ({1}) and predictive plausibility P l Y D,x ({1}) of the positive class given the predictors are obtained from the contour function pl Θ D,x defined by (34) using Eqs. ( 26) and ( 27), respectively.

Equivalently, we have

m Y D,x ({0}) = 0.1111, m Y D,x ({1}) = 0.3403, m Y D,x (Y) = 0.5485.
Using the interval dominance decision rule with m Y D,x and 0/1 loss, the decision reached for the label y of the object with feature vector x = (2, -1) is y = {0, 1}.

In other words, the uncertainty is too high and an imprecise decision is obtained when using the evidential extension of CR (CR yielded the decision y = 1, see Example 2). This contrasts with LR where the same precise decision (y = 1) was obtained with its evidential extension (see Examples 1 and 5, respectively).

For completeness, the decision regions (in the case of 0/1 loss) and the levels of ignorance of ECR are shown in Figures 6b and6c, respectively. Similarly as for ELR, regions where imprecise decisions are made correspond to areas with greatest levels of ignorance. [START_REF] Platt | Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods[END_REF] of the conditional likelihood LD of the LR parameters may be used: the label y (i) ∈ {0, 1} in (3) is then replaced by t (i) ∈ [0, 1] defined by

Remark 1. In order to prevent LR from over-fitting when the training examples in D are perfectly separable, a modified version due to Platt

t (i) = N + +1 N + +2 if y (i) = 1, 1 N -+2 if y (i) = 0, ( 35
)
where N+ and N-are the number of positive and negative samples, respectively, in D. This modification ensures LD to have a unique supremum σ. In their derivation of ELR -precisely, in their definition [START_REF] Ma | Making set-valued predictions in evidential classification: A comparison of different approaches[END_REF] of the contour function pl Σ D -Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] actually used this modified version of LD. As already mentioned in Section 2.3, CR may also be prone to over-fitting. Besides restricting µ to be k-additive for some k < m, it is proposed in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF] to address this issue by adding a L1-regulariser on the Möbius transform in the objective [START_REF] Denoeux | Decision-making with belief functions: a review[END_REF]. However, we may as well consider a modified version of the conditional likelihood LD of the CR parameters, where the label y (i) in [START_REF] Denoeux | Likelihood-based belief function: justification and some extensions to low-quality data[END_REF] (or in [START_REF] Minka | A comparison of numerical optimizers for logistic regression[END_REF] if the reparametrisation is used) has been replaced by t (i) defined by [START_REF] Nguyen | Epistemic uncertainty sampling[END_REF]. Let us remark that, thanks to the reparameterised form of CR and its formal similarity with LR, it is then straightforward to obtain that this latter modification ensures LD to have a unique supremum ψ. In the experiments reported in Section 5, the modified labels [START_REF] Nguyen | Epistemic uncertainty sampling[END_REF] (rather than the original labels y (i) ) are not only used in the conditional likelihood involved in ELR (as it is the actual definition of ELR given in [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF]), but they are also used in all the conditional likelihoods involved in LR, CR and ECR. This is done in order to ensure a fairer comparison between the approaches.

Complexity

In this section, we provide a brief analysis of the computational complexity involved in making a prediction using ECR. To derive this complexity, we start by recalling the complexity of training LR, then we proceed with the complexity of making a prediction using ELR and with that of training CR, before ending with that of making a prediction using ECR.

Given a training set D composed of n instances described by m-dimensional feature vectors, the computational complexity of finding the MLE of LR according to Eq. ( 4), using gradient ascent (GA), is [START_REF] Minka | A comparison of numerical optimizers for logistic regression[END_REF]:

O ((m + 1) • n • t) ,
with t the number of ascent iterations. Making a prediction using ELR involves essentially computing contour function pl Θ D,x (inducing the predictive belief function Bel Y D,x ), which in practice is approximated by: (1) selecting a finite number c of values of θ ∈ (0, 1); (2) evaluating pl Θ D,x , that is, solving the optimisation problem ( 29), for each selected value of θ; (3) interpolating pl Θ D,x linearly between the selected values of θ. Note that the same approach is followed for making a prediction using ECR (in step [START_REF] Calamai | Projected gradient methods for linearly constrained problems[END_REF], optimisation problem ( 34) is then solved instead of ( 29)).

The complexity of solving the optimisation problem (29), using GA, is similar to that of finding the MLE of LR: it is O (m • n • t) . Since this optimisation is done c times, the complexity of making a prediction using ELR is

O (c • m • n • t) .
Training CR amounts to solving the constrained optimisation problem [START_REF] Nadeem | Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option[END_REF], which can be done using projected gradient ascent [START_REF] Calamai | Projected gradient methods for linearly constrained problems[END_REF]. From the complexity of training LR, we obtain that the complexity of training CR is

O (2 m • n • t) .
We thus face an exponential complexity in the number of features. This complexity can nonetheless be significantly reduced by restricting the capacity underlying CR to be k-additive, for some k < m, which, we recall, is also useful to prevent overfitting. Using this restriction, the dimension of the parameter space Ψ of CR is reduced to

O k l=0 m l • n • t .
As already mentioned, making a prediction using ECR follows the same approach to that of making a prediction using ELR. We have in particular that ECR inherits the complexity of CR, in the same manner as ELR inherits the complexity of LR. Therefore, the complexity of making a prediction using ECR is

O (c • 2 m • n • t) .
However, using the restriction to k-additive capacities as done in the experiments reported in Section 5, the complexity comes down to

O c • k l=1 m l • n • t ,

Experiments

In this section, the results of some experiments that we conducted are reported. Their goal is twofold. First, we would like to show that by capturing uncertainty in a more subtle way, ECR is a useful reliable variant of CR. Secondly, we seek to show that by enabling more flexibility with respect to the possibility of interactions between the predictors, ECR is competitive with ELR. Our experiments rely on some (monotone) datasets, which are presented in the next section.

Datasets

We used nine datasets, summarised in Table 1, for which the assumption of monotonicity in the input variables seems reasonable. To our knowledge, at least some of them have also been used in previous works on monotonic classification, such as in [START_REF] Feelders | Monotone relabeling in ordinal classification[END_REF][START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF][START_REF] Cano | Monotonic classification: An overview on algorithms, performance measures and data sets[END_REF]. They are available from the UCI1 , StatLib2 and WEKA3 dataset repositories. Some of them have numerical or categorical outputs, which were thus binarised. Furthermore, all the input features were normalised using [START_REF] Denoeux | Analysis of evidence-theoretic decision rules for pattern classification[END_REF]. The following paragraphs provide some more details on each one of the datasets. The categorical output was binarised into a two-class target by distinguishing between those women that do not use a contraceptive method (category "no use") and those that do use one (categories "long-term methods" and "short-term methods").

Blood Transfusion Service Center (BLO)

Haberman's Survival (HAB)

Hamster (HAM)

This dataset contains the measurements of the weights of 6 organs (lung, heart, liver, spleen, kidney, testes) of 73 hamsters from a strain with a congenital heart problem. The numeric output is the weight measurement of testes, which was binarised as done in the OpenML dataset repository 4 , i.e., it was binarised to a two-class target by computing the mean and classifying all instances with a measurement lower than the mean as positive and all others as negative.

Lecturers Evaluation (LEV)

This dataset is composed of examples of anonymous lecturer evaluations sampled at the end of courses where students were asked to score lecturers according to four ordinal features including oral skills and contribution to their professional/general knowledge. The ordinal output is an overall evaluation of a lecturer's performance which was binarised by interpreting the scores between 3 and 4 as positive (good evaluation) and the scores between 0 and 2 as negative (bad evaluation), as done in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF].

Mammographic (MMG)

This dataset is composed of 961 instances of patients. The problem is to predict whether a mammographic mass lesion is benign or malignant, given 3 BI-RADS features (mass shape, mass margin, density) and the patient's age.

Auto MPG (MPG)

This dataset was designed for the American Statistical Association Exposition of 1983. The problem is about the prediction of the city-cycle fuel consumption (in miles per gallon) based on seven features of a car: cylinders, displacement, horsepower, weight, acceleration, model year, origin. The numerical consumption output was binarised by thresholding at the median, as done in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]. Furthermore, incomplete instances were removed.

Teacher Assistant Evaluation (TAE)

The data was collected at the Statistics Department of the University of Wisconsin-Madison. It represents a collection of 151 evaluations of teacher assistants based on 6 ordinal features: native speaker, instructor, course, semester, class size. The ordinal output represents 3 categories (low, medium, and high), which was binarised to a two-class target by distinguishing between great (category high) and not great evaluation (categories medium and low).

Yeast (YST)

This dataset is composed of 1484 instances describing the cellular localisation sites of proteins, given 8 features extracted from the amino acid contents. The protein localisation site is a nominal output encoded with 10 modalities ("cyt" for cytosolic or cytoskeletal, "nuc" for nuclear, "mit" for mitochondrial,...). It was converted to a two-class target by relabelling the majority localisation site modality as positive and all others as negative, which amounts to predicting whether the localisation site is "cyt".

Accuracy

As stressed in [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF], it is important to make sure that an improved representation of uncertainty permitted by a reliable variant of a classifier, does not come at the expense of a loss in accuracy. Therefore, we looked first at the predictive accuracy of our reliable version of CR, i.e., of ECR, before investigating its potential benefits with respect to CR in terms of uncertainty representation. This also allowed us to check whether the superior accuracy of CR in comparison to LR observed in [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF] when the amount of training data is sufficiently large5 , is carried over to their evidential extensions, as this seems to be another important requirement.

For each dataset, we followed a similar procedure to that of [START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF]. The dataset was randomly split into two parts: a training set and a test set. Each of the models (LR, CR, ELR, ECR) was learnt on the training set and its performance was measured using the standard misclassification rate (0/1 loss) estimated on the test set -for the reliable classifiers ELR and ECR, a (precise) prediction was obtained using the decision rule [START_REF] Gilboa | Additive representations of non-additive measures and the Choquet integral[END_REF]. This process was repeated 100 times and the results were averaged over these repetitions. Furthermore, we considered three proportions between training and test data, which were 20:80, 50:50 and 80:20, in order to study the influence of the amount of training data. In addition, to obtain the best accuracy, the underlying capacity of CR (and thus of ECR, as it extends CR) was restricted to be k-additive, with k determined by 10-fold cross validation on the training set.

The results for this experiment are given in Tables 2 and3. First, as can be seen from Table 2, ECR and ELR have essentially the same predictive accuracies as CR and LR, respectively. Second, ECR has a better predictive accuracy than ELR (this is all the more true as the training set size increases).

In order to further confirm these observed differences in the classifier performances, we resorted to the statistical tests recommended by Demšar [START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF]. First, a Friedman test (using the FF statistic, see [9, Section 3.2.2]) was used for comparing LR, CR, ELR and ECR over the nine datasets, with the null hypothesis that the classifiers are equivalent. The test statistic relies on the average rankings of the classifiers (provided in Table 2), where the rankings are based in this experiment on the misclassification rate. We found that the null hypothesis is rejected at significance level α = 0.05, for any of the three proportions considered between training and test data. When the null hypothesis is rejected, the classifiers can then be compared to each other using a post-hoc test such as the one of Bonferroni-Dunn [START_REF] Demšar | Statistical comparisons of classifiers over multiple data sets[END_REF], which relies also on the average rankings of the classifiers. We used this test to compare ECR to the other classifiers at significance level α = 0.05. We found that ECR is significantly better than both LR and ELR and that the difference with CR is not significant, for any of the three proportions considered between training and test data.

Overall, these results mean that the two requirements set above for ECR seem thus to be met: enabling the possibility to capture extra information on the reliability of predictions does not deteriorate the predictive accuracy, and enabling nonlinear dependencies between input attributes brings an improvement in predictive accuracy not only in the probabilistic setting but also in the evidential one. Moreover, this latter flexibility in terms of dependencies between input attributes is all the more exploited to improve accuracy as the learning set size increases, as can be seen from Table 3, which shows that the optimal (cross-validated) value of k is increasing in the learning set size.

Utility-discounted accuracy

As it was illustrated by Examples 5 and 6, an interest of the improved representation of uncertainty offered by the reliable variants of LR and CR is the ability to make imprecise predictions when there is too much uncertainty. The question of the measurement of the quality of classifiers issuing potentially imprecise predictions has received a few answers (see [START_REF] Zaffalon | Evaluating credal classifiers by utility-discounted predictive accuracy[END_REF][START_REF] Yang | The costs of indeterminacy: How to determine them[END_REF] and the references therein). In general, the idea is to have a measure taking both into account the accuracy and the precision of the predictions. A standard approach is to extend the classical accuracy measure to imprecise predictions, which makes it then possible to compare classifiers issuing imprecise predictions with classifiers issuing only precise predictions.

A first proposal in this vein, known as discounted accuracy, reads as follows. Let T = {x (i) , y (i) } T i=1 be a test set composed of T objects, with y (i) ∈ Y the known (and precise) label of object i. Furthermore, let A (i) ⊆ Y denote the (potentially imprecise) prediction for the label of object i, provided by a classifier having observed feature vector x (i) . Then, the discounted accuracy AccD of this classifier is:

AccD = 1 T T i=1 1 A (i) (y (i) ) • 1 |A (i) | , ( 36
)
with 1 A (i) the indicator function of subset A (i) . This measure clearly degenerates into the classical accuracy measure when all A (i) , i = 1, . . . , T , are singletons, i.e., all the predictions are precise. An issue with the discounted accuracy is that the value of an imprecise prediction A (i) is considered to be the same as that of a purely random choice within A (i) , as criticised by Zaffalon et al. [START_REF] Zaffalon | Evaluating credal classifiers by utility-discounted predictive accuracy[END_REF] other words, in the case of ambiguity, it does not reward caution. To address this problem, Zaffalon et al. [START_REF] Zaffalon | Evaluating credal classifiers by utility-discounted predictive accuracy[END_REF] justified another measure called utility-discounted accuracy, which also degenerates to the classical accuracy measure when all the predictions are precise. This measure is defined as follows:

AccU = 1 T T i=1 1 A (i) (y (i) ) • u( 1 |A (i) | ), ( 37
)
where u is a concave function on [0, 1], such that u(1) = 1 and u(0) = 0. Function u is interpreted as modelling the utility of a prediction according to its precision (the more precise the prediction, the higher the utility). In particular, it rewards an imprecise prediction A (i) ⊆ Y at least as much as the discounted accuracy since this latter accuracy corresponds to giving a utility 1 |A (i) | to this prediction, whereas prediction A (i) gets a utility u( 1

|A (i) | ) ≥ 1 |A (i) |
thanks to the concavity of u. In the case of binary classification (Y = {0, 1}), which is of interest in this paper, Zaffalon et al. [START_REF] Zaffalon | Evaluating credal classifiers by utility-discounted predictive accuracy[END_REF] proposed to reward the imprecise prediction A (i) = {0, 1} by a utility between 0.65 and 0.8, i.e., we should have u( 1|{0,1}| ) = u(0.5) ≥ 0.65 and u(0.5) ≤ 0.8 (in contrast to the discounted accuracy, which corresponds to giving a utility of only 0.5 to such prediction). This means that by setting u(0.5) = 0.65, one is modelling a moderate cautiousness-seeking attitude -imprecise predictions are valued (rewarded), yet to a small extent -whereas by setting u(0.5) = 0.80, imprecise predictions are rewarded to a greater extent, hence modelling a stronger cautiousness-seeking attitude.

With that in mind, we repeated the experimental procedure described in Section 5.2, but with two modifications. First, instead of using the decision rule [START_REF] Gilboa | Additive representations of non-additive measures and the Choquet integral[END_REF] for ELR and ECR, we use the interval dominance decision rule [START_REF] Grabisch | A decade of application of the Choquet and Sugeno integrals in multi-criteria decision aid[END_REF], therefore yielding potentially imprecise decisions for these classifiers. Second, instead of measuring the performances of LR, CR, ELR and ECR using the misclassification rate, we used the utility-discounted accuracy 6 with u(0.5) = 0.65 and also with u(0.5) = 0.8; the former performance measure is denoted by AccU 65 and the latter is denoted by AccU 80 .

The results of this second experiment are presented in Tables 4 and5, respectively, for the performance measures AccU 65 and AccU 80 , respectively. As can be seen from these tables, ECR and ELR have globally better performances than CR and LR, respectively. Furthermore, the difference in performance between a classifier and its reliable version is generally all the greater as there are fewer training data. Overall, this suggests that ECR and ELR are able to capture useful information on the reliability of predictions and, in particular, that the greater uncertainty induced by fewer training data is appropriately taken into account by these reliable variants of CR and LR. Comparing the performances of ECR and ELR, we can see that ECR is globally better than ELR. In addition, similarly as in the case of predictive accuracy (see Section 5.2), the difference in performance between ECR and ELR is typically all the greater as there are more training data. This means that the flexibility of ECR is beneficial also when making reliable predictions and is all the more so as the learning set size increases.

Similarly as in Section 5.2, Friedman tests based on the average rankings (provided in Tables 4 and5) induced by performance measures AccU 65 and AccU 80 , were used for comparing further the classifiers. We found for both of these performance measures that the null hypothesis (the classifiers are equivalent) is rejected at level α = 0.05, for any of the three proportions considered between training and test data. We proceeded with Bonferroni-Dunn tests (with level α = 0.05) based on the average rankings induced by AccU 65 and AccU 80 . We found that according to AccU 65 , ECR is significantly better than LR and that the differences with CR and ELR are not significant, whereas according to AccU 80 , ECR is significantly better than both LR and CR and the difference with ELR is not significant; all of these statements are obtained for any of the three proportions considered between training and test data. The non-significant differences with ELR according to these particular tests, mitigate the conclusion drawn above from Tables 4 and5 that ECR is globally better than ELR when making reliable predictions.

To sum up this section and the previous one, we can conclude that ECR is a useful extension of both CR and ELR. It combines the advantages of both in terms of modelling capabilities, which is reflected by the results of our experiments. More precisely, the ability of ECR to capture uncertainty in a subtler way makes it a useful variant of CR when cautious decision-making matters; ECR carries thus to CR the advantage of ELR over LR. In addition, the ability of ECR to capture nonlinear dependencies between input attributes makes it a useful alternative to ELR when accurate decision-making matters; ECR carries thus to ELR the advantage of CR over LR. Moreover, ECR compares also favorably with ELR in terms of reliable predictions, albeit to a lesser extent than in comparison to CR since the observed quantitative improvements were not confirmed by the statistical tests conducted. As will be seen, the experiment reported in the next section provides further evidence in favour of ECR over ELR in another setting related to cautious decision-making. Prior to that, we should nonetheless recall that these advantages come at a price: ECR combines also the drawbacks of CR and ELR in terms of computational complexity, as discussed in Section 4.3. Computing a prediction for ECR can indeed be intensive; this is typically the case for optimal (cross-validated) values of k such that k ≥ 8.

Datasets

Accuracy-rejection curves

In addition to the possibility of making imprecise predictions when there is too much uncertainty, ECR makes it possible to quantify the ignorance in a given prediction using the quantity m Y D,x (Y). In order to validate the relevance of this quantity as a measure of ignorance, we used accuracy-rejection curves [START_REF] Nadeem | Accuracy-rejection curves (ARCs) for comparing classification methods with a reject option[END_REF], similarly as done previously in the context of reliable classification in [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF]. This means that we rejected classification of instances for which m Y D,x (Y) is above some threshold, and evaluated the accuracy on the remaining instances. Then, we repeated this process for other thresholds. Finally, we plotted the obtained results as an accuracy-rejection curve, which is nothing but the accuracy on the non-rejected samples as a function of the rejection rate. The rationale for computing such curves is that, if m Y D,x (Y) is a valid measure of the ignorance associated with the classification of an object x, then we should observe a monotone dependency between rejection rate and accuracy.

We repeated the experimental procedure described in Section 5. The results of this third experiment are shown in Figure 7. As it can be seen, for ECR, the accuracy on non-rejected instances is increasing in the rejection rate. This is an empirical confirmation that ECR is self-aware, through the quantity m Y D,x (Y), of the reliability of its predictions. A similar observation can be made for ELR, although there are two noticeable differences. First, on the dataset TAE (see Fig. 7h), there is a drop in accuracy for ELR after the 50% rejection mark. This can perhaps be explained by the fact that this mark on this dataset corresponds to only 38 non-rejected instances, and thus the results past this mark may not be too meaningful. Second, on the dataset HAM (see Fig. 7d), ELR fail to do better than random rejection up to the 65% rejection mark. In contrast, ECR does not exhibit these undesirable behaviours. Moreover, its accuracy for any rejection rate is on average at least as good, or even better, than that of ELR. In sum, the flexibility of ECR with respect to ELR is also beneficial in terms of accuracy-rejection curves.

Conclusions

Choquistic regression is a generalisation of logistic regression, allowing flexibility in the modelling of interactions between predictors, while maintaining monotonicity. In this paper, we derived a reliable variant of choquistic regression, which makes it possible to represent the ignorance involved in a given prediction. This ability has been recognised as particularly important in critical classification problems such as medical diagnosis [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF]. Our approach follows the one in [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF], where a reliable version of logistic regression is proposed. It is formalised within evidence theory and relies in particular on a sound method for statistical inference and prediction developed in this framework. Through a series of experiments we showed that, by capturing uncertainty in a proper way and by enabling flexibility in the interactions between predictors, the reliable version of choquistic regression constitutes a useful alternative, when cautiousness in decision-making is allowed, both to choquistic regression and to the reliable logistic regression from [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF].

The ability to quantify the ignorance, aka epistemic uncertainty, in a prediction seems particularly interesting in an active learning [START_REF] Settles | Active learning[END_REF] setting, specifically in the context of the popular uncertainty sampling approach, as preliminary yet promising studies [START_REF] Ramel | Active evidential calibration of binary SVM classifiers[END_REF][START_REF] Nguyen | Epistemic uncertainty sampling[END_REF] in this direction show. The rationale is that since ignorance corresponds to the reducible part of the uncertainty, it seems reasonable to assume that the active learner should query the label of the instance for which its current prediction has maximum ignorance, as this should lead to the most effective reduction of the learner's uncertainty, which is the typical goal of active learning. This was coined as the "epistemic uncertainty principle" in [START_REF] Nguyen | Epistemic uncertainty sampling[END_REF]. Ongoing work consists in testing this principle with respect to our reliable variant of choquistic regression.

The reliable versions of logistic regression proposed by Senge et al. [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF] and by Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF] seem to have been developed independently from one another. In future works, we plan on comparing these two approaches, both theoretically and empirically. Moreover, a belief function-based analysis of logistic regression was proposed recently in [START_REF] Denoeux | Logistic regression, neural networks and Dempster-Shafer theory: a new perspective[END_REF]. From a formal point of view, this analysis makes it possible to derive from the logistic regression model, a belief function on the unknown label of an object, and bears thus some similarity with the approach of Minary et al. [START_REF] Minary | Evidential joint calibration of binary SVM classifiers[END_REF]. We are interested by conducting also a thorough comparison between these two approaches. Such comparisons are not only interesting in themselves but could also inform further research in the direction of this paper, that is, developing reliable variants of generalisations of logistic regression.

Other perspectives include investigating two generalisations of our proposal. First, we would like to go beyond binary classification and handle multi-class classification. This may be achieved by following two different paths: (1) using binary decomposition, similarly as done in [START_REF] Nguyen | Reliable multi-class classification based on pairwise epistemic and aleatoric uncertainty[END_REF] to extend the reliable (binary) classification approach of Senge et al. [START_REF] Senge | Reliable classification: Learning classifiers that distinguish aleatoric and epistemic uncertainty[END_REF], and in which case we could rely on previous works dealing with binary decomposition in the belief function framework [START_REF] Quost | Pairwise classifier combination using belief functions[END_REF][START_REF] Lachaize | Evidential framework for error correcting output code classification[END_REF]; (2) by defining a multinomial version of choquistic regression, in a similar manner as multinomial logistic regression is defined (see, e.g., [START_REF] Denoeux | Logistic regression, neural networks and Dempster-Shafer theory: a new perspective[END_REF]), and by deriving its belief function-based reliable variant using previous results for the multinomial logistic regression [START_REF] Xu | Evidential multinomial logistic regression for multiclass classifier calibration[END_REF]. A second worthy generalisation would be the ability to handle learning data having soft labels (aka partially supervised learning). The foundations for such endeavour could be found in promising results concerning logistic regression obtained recently in [START_REF] Quost | Parametric classification with soft labels using the evidential EM algorithm: linear discriminant analysis versus logistic regression[END_REF].

Example 1 .

 1 Let D be the training set composed of n = 30 training instances generated as follows: 15 instances for the positive class and 15 instances for the negative class have been drawn respectively from the bivariate normal distributions N (µ1, S1) and N (µ0, S0), with means µ1 = (1, 1) and µ0 = (-1, -1), and covariance matrices S1 and S0 such that

2 xFigure 1 :Definition 2 (

 212 Figure 1: Training set D: instances of the positive class (light grey triangles) and of the negative class (dark grey squares). Object of interest (⊕) with feature vector x = (2, -1). Decision boundary ( ) of LR in the case of 0/1 loss, with associated decision regions: y = 1 (light grey) and y = 0 (dark grey).

2 xFigure 2 :

 22 Figure 2: Decision boundary ( ) of CR in the case of 0/1 loss.

Example 3 .

 3 Let Z ∼ B(n, θ), i.e., Z is a random variable following a binomial distribution with parameters n ∈ N and θ ∈ [0, 1]. Assume we have observed z ∈ N successes out of the n trials. Knowledge about θ given z is then represented by the likelihood-based belief function Bel Θ z with contour function defined by:

Figure 3 :

 3 Figure 3: Contour functions for the binomial distribution with θ = 1 3 and n ∈ {15, 30}, and induced predictive mass functions for the next trial.

  Level sets of the contour function pl Σ D . Lines σ2 = (-ln(θ -1 -1) -σ1x1)/x2 for x = (x1, x2) = (2, -1) and θ ∈ {0.2, 0.4, 0.6, 0.837, 0.9, 0.995}, with the maximum on each line ( • ) .

  ({1}) (b) Contour function pl Θ D,x (θ) for x = (2, -1). Associated predictive mass function m Y D,x . Maximums (•) found for different values of θ in Fig. 4a.

Figure 4 :

 4 Figure 4: Computation of the contour function on Θ involved in ELR.

  {1}) = 0.5703, P l Y D,x ({1}) = 0.9517. Equivalently, we have m Y D,x ({1}) = 0.5703, m Y D,x (Y) = 0.3815 (and m Y D,x ({0}) = 0.0483 using (16)). Example 5. Relaxing the restriction on the bias parameter ( i.e., the restriction σ0 = 0) used in Example 4 solely for the sake of illustration, let us provide the actual results obtained when one applies ELR to the data from Example 1. Figure 5a shows the contour function pl Θ D,x for the object with observed feature vector x = (2, -1) from which Bel Y D,x ({1}) and P l Y D,x ({1}) are obtained using Eqs. (26) and (27) respectively. We find: Bel Y D,x ({1}) = 0.5655, P l Y D,x ({1}) = 0.9578, or, equivalently,

  Contour function pl Θ D,x (θ) for x = (2, -1). Associated predictive mass function m Y D,x . Training instances D. Object of interest (⊕). Decision boundary ( ) of LR. Decision regions of ELR according to the interval dominance rule: y = 1 (light grey), y = 0 (dark grey) and y ∈ Y = {0, 1} (white).

2 (

 2 c) Training instances D. Decision boundary ( ) of LR. Contour lines of the ignorance m Y D,x (Y) for any x ∈ X .

Figure 5 :

 5 Figure 5: Evidential logistic regression.

Example 6 .

 6 Let us illustrate ECR on the classification problem with training dataset D of Example 1. The contour function pl Θ D,x (θ) obtained from (34) for feature vector x = (2, -1) is illustrated in Figure 6a. This contour function induces Bel Y D,x ({1}) = 0.3403, P l Y D,x ({1}) = 0.8889.

  Contour function pl Θ D,x (θ) for x = (2, -1).

  Training instances D. Object of interest (⊕). Decision boundary ( ) of CR. Decision regions of ECR according to the interval dominance decision rule: y = 1 (light grey), y = 0 (dark grey) and y ∈ {0, 1} (white).

  Training instances D. Decision boundary ( ) of CR. Contour lines of the ignorance m Y D,x (Y) obtained with ECR.

Figure 6 :

 6 Figure 6: Evidential choquistic regression.

  the complexity of training CR becomes

2 ,

 2 except that instead of looking at the misclassification rate on a given test set, we computed on this set the accuracy-rejection curve based on m Y D,x (Y) issued by ECR. Furthermore, we only considered the proportion 50:50 between training and test data. For completeness, we also computed accuracy-rejection curves based on m Y D,x (Y) issued by ELR, and we plotted accuracy-rejection curves based on random rejection (instead of m Y D,x (Y)) of test instances classified by both ECR and ELR.

Table 1 :

 1 which remains acceptable for real applications having low dimensional features and for k small. Datasets used in the experiments

	Name Blood Transfusion Service Center (BLO) Contraceptive Method Choice (CMC) Haberman's Survival (HAB) Hamster (HAM) Lecturers Evaluation (LEV) Mammographic (MMG) Auto MPG (MPG) Teaching Assistant Evaluation (TAE) Yeast (YST)	#instances #features source 748 4 UCI 1473 9 UCI 306 3 UCI 73 5 StatLib 1000 4 WEKA 961 4 UCI 398 7 UCI 151 5 UCI 1484 8 UCI

  This dataset contains 306 instances on the survival of patients who had undergone surgery for breast cancer between 1958 and 1970 at the University of Chicago's Billings Hospital. Instances are described by 3 ordinal features: patient's age, year of operation, and amount of detected axillary nodes. The goal is to predict the dependent binary variable representing whether the patient died within 5 years.

Table 3 :

 3 . In Mean (± standard deviation) value of k determined by cross-validation over the 100 repetitions.

	Datasets BLO(20:80) 1.33 ± 0.49 k CMC(20:80) 2.90 ± 1.48 HAB(20:80) 1.52 ± 0.66 HAM(20:80) 2.16 ± 0.98 LEV(20:80) 2.29 ± 1.05 MMG(20:80) 1.96 ± 0.95 MPG(20:80) 2.17 ± 1.13 TAE(20:80) 2.07 ± 1.00 YST(20:80) 2.78 ± 1.28 BLO(50:50) 1.70 ± 0.46 CMC(50:50) 3.42 ± 1.51 HAB(50:50) 1.89 ± 0.75 HAM(50:50) 2.65 ± 0.89 LEV(50:50) 3.29 ± 0.57 MMG(50:50) 2.17 ± 0.91 MPG(50:50) 2.74 ± 1.19 TAE(50:50) 2.47 ± 1.12 YST(50:50) 3.34 ± 1.25 BLO(80:20) 1.97 ± 0.17 CMC(80:20) 3.85 ± 1.69 HAB(80:20) 2.00 ± 0.75 HAM(80:20) 3.11 ± 0.63 LEV(80:20) 3.55 ± 0.50 MMG(80:20) 2.16 ± 0.77 MPG(80:20) 3.00 ± 1.03 TAE(80:20) 3.04 ± 1.00 YST(80:20) 3.70 ± 0.96

Table 4 :

 4 Mean and standard deviation of Acc U65 (associated ranking) for CR, LR and their reliable variants, trained on 20%, 50%, and 80% of the datasets.

		CR	ECR	LR	ELR
	BLO(20:80) CMC(20:80) HAB(20:80) HAM(20:80) LEV(20:80) MMG(20:80) MPG(20:80) TAE(20:80) YST(20:80) Average ranking	.7744 ± .0125(3) .7848 ± .0117(1) .7729 ± .0128(4) .7838 ± .0108(2) .6810 ± .0151(3) .7115 ± .0113(1) .6650 ± .0128(4) .6966 ± .0094(2) .7393 ± .0184(3) .7528 ± .0163(1) .7389 ± .0210(4) .7513 ± .0163(2) .5502 ± .0812(3) .6318 ± .0387(2) .5328 ± .0683(4) .6342 ± .0362(1) .8457 ± .0116(3) .8555 ± .0086(1) .8372 ± .0098(4) .8542 ± .0089(2) .7926 ± .0126(3) .8026 ± .0073(2) .7913 ± .0116(4) .8028 ± .0080(1) .8907 ± .0168(1) .8722 ± .0224(4) .8898 ± .0173(2) .8866 ± .0118(3) .6654 ± .0450(3) .6850 ± .0288(1) .6627 ± .0463(4) .6837 ± .0243(2) .6902 ± .0101(3) .7202 ± .0144(1) .6819 ± .0107(4) .7170 ± .0089(2) 2.78 1.56 3.78 1.89
	BLO(50:50) CMC(50:50) HAB(50:50) HAM(50:50) LEV(50:50) MMG(50:50) MPG(50:50) TAE(50:50) YST(50:50) Average ranking	.7775 ± .0165(3) .7856 ± .0167(1) .7741 ± .0162(4) .7852 ± .0167(2) .6974 ± .0127(2) .7228 ± .0092(1) .6716 ± .0117(4) .6968 ± .0114(3) .7463 ± .0278(3) .7587 ± .0251(1) .7444 ± .0284(4) .7567 ± .0255(2) .6017 ± .0862(3) .6515 ± .0506(1) .5547 ± .0716(4) .6292 ± .0399(2) .8577 ± .0118(2) .8637 ± .0094(1) .8363 ± .0121(4) .8523 ± .0109(3) .8017 ± .0147(3) .8071 ± .0113(1) .8001 ± .0132(4) .8053 ± .0114(2) .9090 ± .0204(1) .9055 ± .0151(2) .9022 ± .0178(4) .9029 ± .0152(3) .6861 ± .0468(3) .7075 ± .0294(2) .6733 ± .0432(4) .7077 ± .0310(1) .6900 ± .0131(3) .7159 ± .0125(1) .6789 ± .0140(4) .7044 ± .0125(2) 2.56 1.22 4.00 2.22
	BLO(80:20) CMC(80:20) HAB(80:20) HAM(80:20) LEV(80:20) MMG(80:20) MPG(80:20) TAE(80:20) YST(80:20) Average ranking	.7805 ± .0291(3) .7827 ± .0277(2) .7693 ± .0283(4) .7834 ± .0295(1) .7032 ± .0215(2) .7283 ± .0179(1) .6755 ± .0250(4) .6968 ± .0227(3) .7452 ± .0508(3) .7567 ± .0502(1) .7418 ± .0500(4) .7512 ± .0488(2) .6587 ± .1110(2) .6828 ± .0721(1) .5640 ± .1061(4) .6208 ± .0677(3) .8571 ± .0220(2) .8651 ± .0191(1) .8343 ± .0205(4) .8496 ± .0182(3) .8090 ± .0243(2) .8112 ± .0217(1) .8051 ± .0236(4) .8067 ± .0222(3) .9162 ± .0288(1) .9156 ± .0235(2) .9095 ± .0295(4) .9114 ± .0269(3) .6950 ± .0814(3) .7134 ± .0587(2) .6663 ± .0788(4) .7140 ± .0660(1) .6905 ± .0221(3) .7128 ± .0190(1) .6800 ± .0230(4) .7018 ± .0209(2) 2.33 1.33 4.00 2.33

Table 5 :

 5 Mean and standard deviation of Acc U80 (associated ranking) for CR, LR and their reliable variants, trained on 20%, 50%, and 80% of the datasets.

https://archive.ics.uci.edu

http://lib.stat.cmu.edu/datasets

https://waikato.github.io/weka-wiki/datasets

See https://www.openml.org/d/893

In[START_REF] Fallah Tehrani | Learning monotone nonlinear models using the Choquet integral[END_REF], it was observed that CR outperforms LR when there is sufficiently extensive training data, but also that LR may be better if the amount of training data is small, which the authors explained by the fact that the flexibility of CR, i.e., its ability to capture nonlinear dependencies between input attributes, may lead to overfitting when there is few data and is becoming more and more advantageous as more data is available.

Obviously, for LR and CR, this measure is nothing but their predictive accuracy (already provided in Section 5.2 in the form of the misclassification rate), as these models yield only precise predictions.

Acknowledgements

This work is funded in part by the ELSAT2020 project, which is co-financed by the European Union with the European Regional Development Fund, the French state and the Hauts de France Region Council.