Benoît Lalloué

Jean-Marie Monnez

Donata Lucci

Eliane Albuisson

Construction of Parsimonious Event Risk Scores by an Ensemble Method. An Illustration for Short-Term Predictions in Chronic Heart Failure Patients from the GISSI-HF Trial

Keywords: Ensemble Score, Ensemble Methods, Scoring, Variable Selection, Heart Failure

Selecting which explanatory variables to include in a given score is a common difficulty, as a balance must be found between statistical fit and practical application. This article presents a methodology for constructing parsimonious event risk scores combining a stepwise selection of variables with ensemble scores obtained by aggregation of several scores, using several classifiers, bootstrap samples and various modalities of random selection of variables. Selection methods based on a probabilistic model can be used to achieve a stepwise selection for a given classifier such as logistic regression, but not directly for an ensemble classifier constructed by aggregation of several classifiers. Three selection methods are proposed in this framework, two involving a backward selection of the variables based on their coefficients in an ensemble score and the third involving a forward selection of the variables maximizing the AUC. The stepwise selection allows constructing a succession of scores, with the practitioner able to choose which score best fits his needs. These three methods are compared in an application to construct parsimonious short-term event risk scores in chronic HF patients, using as event the composite endpoint of death or hospitalization for worsening HF within 180 days of a visit. Focusing on the fastest method, four scores are constructed, yielding out-of-bag AUCs ranging from 0.81 (26 variables) to 0.76 (2 variables).

Introduction 1.Ensemble Scores

In [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF], we proposed a method for constructing an event risk-score based on the use of an ensemble method and applied the latter to construct a short-term event risk score in heart failure (HF) patients. The principle of an ensemble method is to build a collection of predictors and thereafter aggregate the predictions [START_REF] Hastie | The Elements of Statistical Learning. 2nd Edition[END_REF], a well-known example being the random forests method [START_REF] Breiman | Random Forests[END_REF]. An ensemble predictor is expected to be better than each of the individual predictors, provided that 1) each single predictor is relatively good and 2) single predictors are sufficiently different from each other [START_REF] Genuer | Arbres CART et Forêts aléatoires, Importance et sélection de variables[END_REF]. Our goal is now to define a methodology for constructing a parsimonious event risk score using an already-defined ensemble method. This methodology and the ensemble method defined in [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF] are then used to define a short-term event risk score in chronic HF patients in order to provide an example of concrete application (Section 1.3).

Selection of Variables

The more the variables contained in a model, the more complicated its use in particular in clinical practice. Therefore, a balance must be found between increasing the number of variables to allow for a better statistical fit and keeping this number sufficiently small to facilitate practical application. With the increased number of potential predictors in the medical field (through the use of "big data" from both electronic medical records and the increasing number of available biomarkers), the need for the statistical selection of variables also increases, particularly if the goal is to continue building parsimonious and effective models. For HF, variables can be selected using a literature review in order to assess which variables are the most clinically relevant [5] [6]. This often constitutes a preliminary step before using various methods of statistical selection. Among the statistical methods, a simple method is to retain only the significant variables derived from univariate analyses [START_REF] Bhandari | Plasma Growth Hormone Is a Strong Predictor of Risk at 1 Year in Acute Heart Failure[END_REF] [8] [START_REF] Xu | A Severity Index Study of Long-Term Prognosis in Patients with Chronic Heart Failure[END_REF] or from a full multivariate model [START_REF] Barlera | Predictors of Mortality in 6975 Patients with Chronic Heart Failure in the Gruppo Italiano per lo Studio della Streptochinasi nell'Infarto Miocardico-Heart Failure Trial: Proposal for a Nomogram[END_REF] [START_REF] Lupón | Development of a Novel Heart Failure Risk Tool: The Barcelona Bio-Heart Failure Risk Calculator (BCN Bio-HF Calculator)[END_REF]. Slightly more elaborate methods such as stepwise selection can also be used [START_REF] Levy | The Seattle Heart Failure Model: Prediction of Survival in Heart Failure[END_REF] [START_REF] Pocock | Predicting Survival in Heart Failure: A Risk Score Based on 39 372 Patients from 30 Studies[END_REF]. Finally, certain studies select variables with more complex methods, using bootstrapping [START_REF] Voors | Development and Validation of Multivariable Models to Predict Mortality and Hospitalization in Patients with Heart Failure[END_REF], random forests and decision trees [START_REF] Krumholz | Do Non-Clinical Factors Improve Prediction of Readmission Risk?: Results From the Tele-HF Study[END_REF] [START_REF] Desai | Applied Mathematics Comparison of Machine Learning Methods with Traditional Models for Use of Administrative Claims with Electronic Medical Records to Predict Heart Failure Outcomes[END_REF] or other selection methods [START_REF] Simpson | Prognostic Models Derived in PARADIGM-HF and Validated in ATMOSPHERE and the Swedish Heart Failure Registry to Predict Mortality and Morbidity in Chronic Heart Failure[END_REF].

Since the primary goal in the present study is to construct a score using an already-defined ensemble method, some of the above selection methods are not applicable in this setting. For example, the likelihood ratio test based on a probabilistic model can be used to achieve a stepwise selection for a given classifier such as logistic regression, but not directly for a classifier constructed by aggre-gation of several classifiers. Other selection criteria must therefore be defined in this framework. Given this context, this article presents in Section 2 a methodology for constructing parsimonious event scores combining a stepwise selection of variables and the use of ensemble scores. In particular, we define herein three methods, two of which involve a backward selection based on the variables' coefficients in an ensemble score, and the third involving the combination of a forward selection using the area under the ROC curve (AUC) as criterion and an ensemble score. Due to the stepwise selection, a succession of scores is constructed which allows the user to choose which of the latter yields the best balance between performance and the number of variables.

Application to Chronic Heart Failure Patients Scoring

As a concrete illustration, these three methods of construction of parsimonious scores are compared according to AUC and processing time in an application aimed at constructing short-term event risk scores in chronic heart failure (CHF) patients. Heart failure is a global and major cause of mortality and morbidity [START_REF] Metra | Heart Failure[END_REF] [START_REF] Orso | Epidemiology of Heart Failure[END_REF]. The association between HF outcomes (death, hospitalization, device implantation, transplantation, etc.) and a large number of variables (whether demographic, clinical, biochemical, biomarkers, etc.) has been widely highlighted in the medical literature. A common approach to usefully synthesize the information provided by this large number of predictor variables is to create a risk score aimed at predicting the probability of adverse events. Many predicting scores and models have already been published: in a recent literature review, Di Tanna et al. [START_REF] Di Tanna | Evaluating Risk Prediction Models for Adults with Heart Failure: A Systematic Literature Review[END_REF] identified 58 risk-prediction models for HF in 40 articles published between 2013 and 2018. Among these articles, 11 studies used logistic regressions (mostly binary and multivariate) and 22 Cox regressions (mostly multivariate and stepwise). A much larger number of these models have furthermore been published over the last three decades [21] [22] [23] [START_REF] Ferrero | Prognostic Scores in Heart Failure-Critical Appraisal and Practical Use[END_REF]. Scores using other methods, such as machine learning methods, are rarer although increasingly proposed nowadays [START_REF] Desai | Applied Mathematics Comparison of Machine Learning Methods with Traditional Models for Use of Administrative Claims with Electronic Medical Records to Predict Heart Failure Outcomes[END_REF] [25] [START_REF] Bazoukis | Machine Learning versus Conventional Clinical Methods in Guiding Management of Heart Failure Patients-A Systematic Review[END_REF]. Some studies aiming to predict HF events have used various forms of ensemble methods without designating the latter as such, for example by constructing multiple imputed datasets, drawing bootstrap samples on each of these datasets, and subsequently building models on each sample prior to their aggregation [START_REF] Pocock | Predicting Survival in Heart Failure: A Risk Score Based on 39 372 Patients from 30 Studies[END_REF] [14] [START_REF] O'connor | Factors Related to Morbidity and Mortality in Patients with Chronic Heart Failure with Systolic Dysfunction: The HF-ACTION Predictive Risk Score Model[END_REF].

In [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF], Duarte et al. used their proposed methodology to construct a short-term event risk score in HF patients, using an ensemble method involving two classification rules (logistic regression and linear discriminant analysis), bootstrap samples as well as introducing random selections of variables in the construction of predictors. We used herein this methodology and constructed parsimonious scores using the methods defined in Section 2. The application for short-term predictions in CHF patients is presented in Section 3, and a discussion in Section 4.

Methodology

In this section, a methodology for constructing parsimonious event risk scores combining a stepwise selection of variables with ensemble scores is presented. Each method consists of two phases, first a preselection of variables per classifier, second a stepwise construction of ensemble scores.

Preliminary Exclusion of Variables

Univariate tests (Wilcoxon test for continuous variables and Fisher's exact test for categorical variables) are first used to test the association between the response variable and each explanatory variable. Variables with a p-value greater than 0.2 are excluded.

Construction of an Ensemble Score for a Binary Outcome

The methodology detailed in Duarte et al. [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF] is adapted to construct the scores.

Basically, an ensemble method is used, where several models are built using various classification methods, different samples and different variable selections, and are subsequently aggregated in a unique score by weighted averaging. This method can be described in seven phases, as follows:

1) n 1 classifiers are chosen.

2) n 2 bootstrap samples are drawn from the working sample. Each bootstrap sample is used n 1 times (each sample is used by each classifier).

3) n 3 modalities of random selection of variables are chosen, "modality" representing a means to select the variables.

4

) n 1 n 2 n 3 models are built, each using a different combination of classifiers, bootstrap samples and modalities of selection of variables.

5) A first aggregation by classifiers is performed. The coefficients of the models are averaged to yield n 1 intermediate scores.

6) The coefficients of the intermediate scores are normalized such that the scores themselves are between 0 and 100, using the same method as in Duarte et al. ([1], Subsection 4.4.2).

7) The final score is constructed by taking a convex combination of the intermediate scores maximizing the AUC OOB (AUC on out-of-bag samples).

The AUC OOB (AUC on out-of-bag samples) is computed as follows: for a given statistical unit, the scores obtained from bootstrap samples that do not include this statistical unit are aggregated to obtain an OOB prediction. By applying this method for all statistical units, the OOB predictions for the entire sample are used to compute the AUC OOB.

The search of an optimal set of coefficients of the convex combination of the intermediate scores may be achieved in a discrete subset of the set

( ) { } 1 1 1 1
,..., : ... 1

n n A α α α α = + + = .
We used this method in the application. This search may take too much time due to the number of elements of A. Otherwise, the simplest way is to use

( ) ( ) { } 1
1, 0,..., 0 ,..., 0, 0,...,1 A = , thus to choose the classifier among the n 1 classifiers which maximizes the AUC OOB. Note that in Super Learner ( [START_REF] Polley | Super Learner in Prediction[END_REF] among others), the coefficients i α are determined using cross validation and a least square regression technique. Compared to the methodology presented in Duarte et al. [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF] the normalization of the coefficients is carried out before rather than after the final aggregation. The latter change is made to balance the intermediate scores in the event that their raw coefficients would have different orders of magnitude.

Preselection of Variables and Construction of Parsimonious Scores

As the number p of explanatory variables after the first exclusion of variables still remains too large to create a parsimonious score, a second phase is added in order to preselect a fewer number of variables. Three different methods with an additional preselection are proposed and their results compared. In Method 1, any adapted preselection of variables can be performed for each of the n 1 classifiers and the sets of preselected variables are united in one set; then, a backward construction of scores is performed. In Method 2, a backward construction of scores is performed with a random selection of variables at each step. In Method 3, a forward preselection of variables for one of the classifiers or for each of the classifiers using the AUC in resubstitution as criterion is performed followed by a forward construction of scores using the AUC OOB as criterion.

Method 1

Preselection of variables: For each of the n 1 classifiers, any adapted preselection of variables can be performed. Thus, n 1 sets of preselected variables are created. The union of these n 1 sets is used as initial preselection. Let s be the number of preselected variables.

Backward construction of scores: For 1, 2,..., i s = , at step i: an ensemble score is constructed from 1 j s i = -+ variables (i.e., for i = 1, j = s; for i = s, j = 1), using the method described in 2.2 with n 1 classifiers, n 2 bootstrap samples, n 3 modalities of random selection of variables. The variable with the lowest normalized and standardized coefficient in absolute value in this score is excluded for the step i + 1 (backward selection). This allowed determining the evolution of the AUC OOB according to the number of selected variables, as well as the order of removal of the variables.

Parsimonious scores with few variables can be chosen among this sequence of s scores.

Method 2

Preselection of variables: No initial preselection of variables is performed; all of the p explanatory variables are included.

Backward construction of scores: For 1, 2,..., i p = , at step i: an ensemble score is constructed from 1 j p i = -+ variables (i.e., for i = 1, j = p; for i = p, j = 1), using the method described in 2.2 with n 1 classifiers, n 2 bootstrap samples, n 3 modalities of random selection of variables. The variable with the lowest normalized and standardized coefficient in absolute value in this score is excluded for the step i + 1. Again, this process allows determining the evolution of the AUC OOB according to the number of selected variables, as well as the order of removal of the variables, and parsimonious scores with few variables can be chosen among this sequence of p scores.

Method 3

Forward preselection of variables: A forward preselection using AUC as criterion is performed for one of the classifiers or each of the classifiers. For a given classifier, let t denote a stopping time; for 1, 2,..., i t = , at step i: i -1 variables denoted 1 1 ,..., i V Vare available from step i -1, for every set of variables ,..., ,

i j V V V - with 1,..., 1 j i ≠
-, a classification is performed on the entire sample without bootstrapping; the variable, denoted V i , yielding the maximal AUC in resubstitution is included, provided that the AUC significantly increases using DeLong's test; otherwise, the inclusion of variables is stopped.

Note that the AUC can be computed as long as there is a prediction for each statistical unit, without assumption on the manner with which this prediction was obtained.

Forward construction of scores: For each classifier, for 1, 2,..., i t = , at step i: an intermediate score using the i preselected variables for this classifier, is constructed, using n 2 bootstrap samples (the same for all of the classifiers) and n 3 modalities of random selection of variables. The n 1 intermediate scores using the same number of preselected variables are aggregated in a final score by combining their predictions for each statistical unit as described in 2.2.

Comparison Criteria between the Methods

The area under the ROC curve (AUC) for the out-of-bag (OOB) estimations is used as internal validation and as the main criterion to compare the different scores. Several AUC OOB are studied: the AUC OOB for the intermediate scores and, mainly, the AUC OOB for the global score. Sensitivity (Se) and specificity (Sp) corresponding to the highest Youden index (Se + Sp -1), as well as the number of selected variables and processing time, are also taken into account.

Application for Short-Term Predictions in CHF Patients

Choices Made for the Construction of the Ensemble Scores

Herein, two classifiers (n 1 = 2), linear discriminant analysis (LDA), which is equivalent to linear regression on binary outcomes, and logistic regression (LR) were chosen to construct the ensemble scores. The number of bootstrap samples was, n 2 = 1000. Two modalities of random selection of variables were chosen, n 3 = 2: namely, one modality consisted in randomly drawing a defined number of variables; the other in randomly drawing a defined number of groups of related variables (correlated or linked by construction) and, for each selected group, randomly draw one variable. The groups of related variables used in the application are shown in the Supplementary Material A.1. 1). The optimal value of λ was determined by testing values from 0 to 1 using incremental 0.01 steps and selecting the value maximizing the AUC OOB.

1 LDA LR S S S λ λ = + - ( 0 1 λ ≤ ≤ ) (see Figure

Choices Made for the Preselection of Variables

For Method 1, a stepwise preselection using the Akaike Information Criterion (AIC) was performed on the working sample, without bootstrapping, both for LDA and for LR. Note that herein, the AIC can be used as criterion since both LDA and LR are probabilistic models.

For Method 3, the results presented used a forward preselection with LR (Method 3a). Results obtained using a forward preselection using both LR and LDA (Method 3b) or using only LDA (Method 3c) are available as Supplementary Material (Part B).

Description of the Data

Description of the Original Data

The data used in this study are derived from the GISSI-HF trial: a multicenter, randomized, double-blind, placebo-controlled trial designed to assess the effect of n -3 polyunsaturated fatty acids in patients with CHF. The detailed protocol and main results of this trial have already been described elsewhere [29] [30].

Eligible patients were adult men and women with clinical evidence of HF of any cause, with a New York Heart Association (NYHA) class II-IV, and having had a left ventricular ejection fraction (LVEF) measured within 3 months prior to enrolment. Patients with a LVEF greater than 40% had to have been admitted at least once to hospital for HF in the preceding year to meet the inclusion criteria. In addition to contraindications linked to the studied treatment, exclusion criteria included acute coronary syndrome or revascularization procedure within the preceding 1 month; and planned cardiac surgery expected to be performed within 3 months after randomization.

After randomization and the baseline visit, patients underwent scheduled visits at 1, 3, 6, 12 months and every 6 months thereafter until the end of the trial. Data collected at baseline included patient description, medical history, etiology of HF, LVEF measurements, electrocardiogram data, clinical and cardiovascular examination, blood chemistry tests, pharmacological treatments and dietary habits. During the follow-up visits, collected data consisted of patient description, clinical and cardiovascular examination, LVEF measurement, electrocardiogram data, blood chemistry tests (only at 1, 3, 6, 12, 24, 36 and 48 months), pharmacological treatment (including the study treatment) and dietary habits. Events of interest were also recorded. The entire GISSI-HF trial included 7046 eligible and randomized patients, with the final sample analyzed in [START_REF] Investigators | Effect of N-3 Polyunsaturated Fatty Acids in Patients with Chronic Heart Failure (the GISSI-HF Trial): A Randomised, Double-Blind, Placebo-Controlled Trial[END_REF] and comprised of 6975 patients.

The present study used a subsample of the GISSI-HF data containing 1231 patients with N-terminal prohormone brain natriuretic peptide (NT-proBNP) measurements. The dataset included baseline and follow-up visits for these patients, as well as their associated health events.

(Patient, visit) couples were used herein as statistical units, i.e. each observation was associated to a patient for a given visit. We assumed that the short-term future of a patient was only dependent on the most recent measurements. Thus, the links between several couples pertaining to the same patient were not taken into account, as in [1] [31]. This yields an initial sample of 12,882 (patient, visit) couples.

Variables Pre-Processing

Several variables were derived from the available data, either for the follow-up visits (when values were available at baseline but not for the follow-up) or for all visits: mean blood pressure (BP) (1/3 * systolic BP + 2/3 * diastolic BP); estimated plasma volume (ePVS) ((100-hematocrit)/hemoglobin as defined in [START_REF] Duarte | Prognostic Value of Estimated Plasma Volume in Heart Failure[END_REF]); estimated glomerular filtration rate (eGFR) (using the MDRD formula [START_REF] Levey | A More Accurate Method to Estimate Glomerular Filtration Rate from Serum Creatinine: A New Prediction Equation[END_REF]); age and body mass index (BMI). Binary variables for the therapeutic classes of drugs were also derived from detailed information pertaining to pharmacological treatments in order to indicate the consumption of ACE-inhibitors, beta-blockers, calcium antagonists or diuretics.

Categorical variables were recoded as binary dummy variables. In particular, in the case of ordinal variables (i.e. NYHA class and peripheral edema), an ordinal encoding was used, namely constructing the binary variables NYHA ≥ II, NYHA ≥ III and NYHA ≥ IV and, similarly, peripheral edema ≥ ankles, peripheral edema ≥ knee, peripheral edema ≥ above.

Since some variables were only available at baseline but were unlikely to change over time (e.g. sex), their values were copied for follow-up visits. Similarly, certain medical history variables available at baseline (such as previous acute myocardial infarction (AMI), previous stroke, angina pectoris, coronary artery bypass graft (CABG), previous hospitalization for worsening HF) were copied for follow-up visits and, when possible, updated using the information from the events. NT-proBNP values were only measured at baseline and at the 3-months follow-up. Due to the importance of this variable in the literature [START_REF] Simpson | Prognostic Models Derived in PARADIGM-HF and Validated in ATMOSPHERE and the Swedish Heart Failure Registry to Predict Mortality and Morbidity in Chronic Heart Failure[END_REF] [34] [START_REF] Khanam | Validation of the MAGGIC (Meta-Analysis Global Group in Chronic Heart Failure) Heart Failure Risk Score and the Effect of Adding Natriuretic Peptide for Predicting Mortality after Discharge in Hospitalized Patients with Heart Failure[END_REF] [36], it was decided to retain and interpolate its value for the other visits as follows: the value for the 1-month follow-up visit was computed as 2/3 * (baseline value) + 1/3 * (3-months value). Value of the 3-months visit was copied for the subsequent visits.

Lastly, the response variable was defined as the occurrence of a composite event (death for worsening HF or hospitalization for worsening HF) within 180 days of a visit.

Exclusion of Variables and Observations

Since the laboratory tests for measuring blood parameters were performed only at baseline, 1, 3, 6, 12, 24, 36 and 48 months, only the observations corresponding to these visits were retained. Incomplete observations (with missing values) were also excluded.

Several variables not relevant to this study were excluded (e.g. "technical variables", such as identification numbers or dates, or "intermediary variables" used to build other variables, such as the cause of death or drug doses), as well as variables with more than 1000 missing values. The remaining variables and the groups of related variables are shown in the Supplementary Material A.1.

Six binary variables with univariate p-value greater than 0.2 (Fisher's exact test) were excluded: gender being "female", main cause of HF being "hypertension" or "other", history of coronary angioplasty, left ventricular hypertrophy, pathological Q waves.

Winsorization and Transformation of the Variables

In order to eliminate outliers without excluding the associated observations, all continuous variables were winsorized: all values lower than the 1 st percentile (respectively greater than the 99 th percentile) were set to the value of the 1 st percentile (resp. the 99 th percentile). This method was used to avoid excluding more observations, since the number of cases was already small compared to the controls and to avoid reducing the number of patients with event.

Continuous variables were then transformed to satisfy the linearity assumption of logistic regression. For each continuous variable, a similar method to that described in Duarte et al. [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF] was used. First, the restricted cubic splines method with 3 knots was used to test the linearity assumption for each variable under the univariate logistic model: using a likelihood ratio test, the nullity of the coefficient associated with the cubic component of the spline was tested [START_REF] Voors | Development and Validation of Multivariable Models to Predict Mortality and Hospitalization in Patients with Heart Failure[END_REF] [START_REF] Simpson | Prognostic Models Derived in PARADIGM-HF and Validated in ATMOSPHERE and the Swedish Heart Failure Registry to Predict Mortality and Morbidity in Chronic Heart Failure[END_REF]. Then, for each variable with a significantly non-null coefficient with a 5% threshold, a graphical representation of the links between the variable and the logit was performed. If the relationship was monotonous, simple monotonic transformations of the form ( )

a f x x = with { } 2, 1, 1 2,1 2,1, 2 a ∈ --- or ( ) ( ) ln f x x =
were tested. If the relationship was not monotonous, quadratic transformations of the form ( ) ( )

2 f x x k = -
were tested, with k situated between the minimum and the maximum of the variable by incremental 0.1 steps.

To determine the values of a or k, all possible values were tested and the transformation which yielded a non-significant p-value for the linearity test and a minimal p-value for the test of nullity of the coefficient in univariate logistic regression was retained. Eleven of 23 continuous variables had a significantly non-null coefficient associated with the cubic component of the restricted cubic spline when tested. Among these eleven variables, six (mean blood pressure, eGFR, triglycerides, cholesterol HDL, total cholesterol and NT-proBNP) had a monotonic relationship with the logit. All except eGFR and NT-proBNP had x -2 for optimal transformation, while the optimal transformation for eGFR and NT-proBNP was 1/x and ln(x) respectively. The remaining five variables (BMI, systolic blood pressure, hematocrit, uricemia and LVEF) had a quadratic relationship with the logit and were transformed accordingly. After the transformation, the coefficient associated with the cubic component of the spline was non-significantly different from 0 for each of the transformed variables.

This transformed dataset was used for Methods 1, 2 and the LR intermediate score of Method 3a. A similar technique was used on a duplicate dataset for the LDA intermediate score of Method 3a, but with transformation of the variables in order to satisfy the linearity assumption for linear regression; fifteen variables were transformed: ten were transformed using a quadratic (x -k) 2 transformation (BMI, systolic blood pressure, diastolic blood pressure, mean blood pressure, hematocrit, hemoglobin, ePVS, serum sodium, uricemia, total cholesterol and LVEF); three using an inverse square x -2 transformation (eGFR, triglycerides, cholesterol HDL); one using a square transformation (serum creatinine);

and one using a square root transformation (NT-proBNP).

The p-values of the tests, before and after transformation, as well as the transformation functions applied to the variables both for the LR and for the LDA are available as Supplementary Material (Part C).

Working Sample

Given the large imbalance between cases and controls, the sample was balanced by duplicating each case 15 times. This is equivalent to giving each case fifteen times more weight than a control. Preliminary analyses (not shown) showed that using a sample that was rebalanced in this manner resulted in better performance compared to using the unbalanced sample.

After the exclusions, the working sample consisted in 11,411 observations of 62 explanatory variables, with 5595 (duplicated) events and 5816 non-events. Summary statistics of the sample prior to data management (winsorization, transformation of the variables and sample balancing) are available in Supplementary Material A.2. Summary statistics of the sample after winsorization and sample balancing, but before the transformation of the variables, are provided in Supplementary Material A.1.

Results for the Preselections of Variables by the Three Methods

The detailed preselections with their corresponding AUC are given in Table 1.

The numbers of variables needed to obtain a given AUC OOB for each of the three methods are provided in Table 2.

For Method 1, 50 variables were preselected during the stepwise selection phase, after which the maximum AUC OOB was obtained for the score using 49 variables. The total runtime for the first method was approximately 1h30 (5 min for the two stepwise preselections and 1h25 for the backward selection using scores).

Comparatively, for Method 2, the maximum AUC OOB corresponded to the score using 58 variables. The total runtime of the second method was approximately 1h35 (exclusively for the backward selection using scores).

For Method 3a, the logistic forward preselection yielded 26 variables, mostly clinical or biological, after which the AUC no longer increased significantly. The total runtime of the third method was approximately 1h05 minutes if all the scores were constructed (less than 5 min for the preselection and 30 min for each of the successions of scores). However, unlike the other two methods, it is not mandatory to construct all of the scores with Method 3a and one could construct only one score after the preselection of variables. In this case, the total runtime would be reduced to less than 10 min (less than 5 min for the preselection and 2 -5 min to construct one score).

Preselected variables were extremely similar between all 3 methods. For Methods 1 and 2, three variables were needed to obtain an AUC OOB greater than 0.75 (for Method 3a, only two were needed). Among these variables, two were common to all methods: NT-proBNP and NYHA ≥ III. In order to obtain an AUC OOB above 0.78, all methods necessitated eight variables, seven of which were common to the three methods: NT-proBNP, NYHA ≥ III, Glycemia, systolic blood pressure, beta-blockers, peripheral edema ≥ "above" and NYHA ≥ II. Lastly, for an AUC OOB threshold of 0.80, Methods 1 and 2 necessitated 17 variables, while Method 3a necessitated 15. In this case, 13 variables were common to the three methods: added to the six aforementioned variables were cholesterol HDL, heart rate, uricemia, third heart sound, bilirubin and paroxystic atrial fibrillation. Globally, the three selections were very similar.

For a fixed number of variables, the three methods yielded extremely similar AUC OOB, even when the selections of variables themselves were different. Since Method 3a generally yielded the best AUC OOB for a given number of selected variables and with a faster runtime, only the results for parsimonious scores constructed by this method are given at the end of this section.

Results for Parsimonious Scores Constructed by Method 3a

Four scores constructed by Method 3a were particularly studied: the score including all variables selected by the forward preselection, denoted S3.26 (the number of the method and the number of variables used), and three "parsimonious" scores, denoted S3.15, S3.8 and S3.2, which yielded an AUC OOB above certain thresholds (0.80, 0.78 and 0.75). To attain these thresholds, 15, 8 and 2

variables were respectively needed. The AUC OOB with λ = 0.5 and the optimal λ, as well as the optimal sensitivity and specificity according to the maximum Youden index of these four scores are given in Table 3.

Score S3.2 had an AUC OOB of 0.7523 with an optimal λ = 1 (i.e. only LDA Note: even if the methods necessitated the same number of variables to obtain a given AUC, the variables themselves may not be the same. was used). Score S3.8 had an AUC OOB of 0.7835 with an optimal λ = 0.06. Score S3.15 had an AUC OOB of 0.8001 with an optimal λ = 0.09. Finally, the full score including all preselected variables had an AUC OOB of 0.8137 with an optimal λ = 0 (i.e. only LR was used). It is interesting to note that for score S3.2, only LDA was used while for score S3.26 only LR was used. Thus, both classifiers are useful.

Discussion

Methodological Discussion

In this article, we presented and compared different methods of construction of parsimonious ensemble scores, with the construction of short-term event scores for CHF as a concrete illustration. Parsimonious scores were obtained by combining stepwise selections of variables and the use of an ensemble score. Since classic criteria of stepwise selection based on probabilistic models cannot be used in the case of an ensemble score, we proposed using a criterion based on the absolute values of the coefficients of variables in an ensemble score and a second criterion based on the AUC. An advantage of a stepwise selection of predictors is that it allows automatically building a succession of scores and therefore choosing which of the latter has the best balance between performance and the number of variables, according to the desired quality objectives. Once this choice is made, the selected score can be used as a "classic" score. The use of an ensemble method to construct this score also provides confidence in the stability and performance of the results. Indeed, ensemble methods generally yield better results than a single predictor, provided that the predictors constituting the ensemble perform sufficiently well individually and are sufficiently different from each other [START_REF] Genuer | Arbres CART et Forêts aléatoires, Importance et sélection de variables[END_REF]. The downside is that since the method relies on estimating a large number of models before their aggregation, this approach takes longer than estimating a single model. However, in the present context, it is only necessary to perform this procedure once to obtain the selection of variables and their associated coefficients, after which a simple linear combination is sufficient to obtain the score for any new observation.

Other selection methods could have been tested, for example by building all possible ensemble scores at each step with one more variable than in the previous step, keeping only the variable yielding the largest increase in AUC OOB. However, this would have entailed a lengthy processing time due to the large number of ensemble scores to construct and preliminary results (not shown) conclude that they would not have yielded a better performance than the presented methods. In the application, variants of Method 3 could also be used, e.g. preselecting variables using LDA as opposed to LR. Summarized results for these alternative methods are presented in the Supplementary Material.

Application Discussion

Regarding the variables used, when applying our method to the construction of a All variables included in the parsimonious scores S3.15, S3.8 and S3.2 are easily available from either the patient's medical history (paroxystic atrial fibrillation, previous AMI, implantable defibrillator, neoplasia), the patient's drug consumption (beta-blockers), a clinical examination (NYHA class, peripheral edema, heart rate, blood pressure, third heart sound), or laboratory blood tests (NT-proBNP, glycemia, cholesterol HDL, bilirubin, uricemia, triglycerides).

To our knowledge, no study has presented a score for short-term (180 days) events in CHF. Therefore, comparing the performance of our scores with others in the literature is difficult. Recent existing scores were generally constructed to predict long-term events for CHF patients, often at 1 or 2 years [10] [12] [17] [36] [START_REF] Canepa | Performance of Prognostic Risk Scores in Chronic Heart Failure Patients Enrolled in the European Society of Cardiology Heart Failure Long-Term Registry[END_REF] and sometimes longer [START_REF] Xu | A Severity Index Study of Long-Term Prognosis in Patients with Chronic Heart Failure[END_REF] [13], or to predict either short-or long-term events for acute HF patient [START_REF] Duarte | Methodology for Constructing a Short-Term Event Risk Score in Heart Failure Patients[END_REF] [START_REF] Khanam | Validation of the MAGGIC (Meta-Analysis Global Group in Chronic Heart Failure) Heart Failure Risk Score and the Effect of Adding Natriuretic Peptide for Predicting Mortality after Discharge in Hospitalized Patients with Heart Failure[END_REF]. For instance, regarding CHF:

• In Voors et al. [START_REF] Voors | Development and Validation of Multivariable Models to Predict Mortality and Hospitalization in Patients with Heart Failure[END_REF], several models were compared to predict different outcomes in CHF patients. Their models using 15 or 9 variables (including NT-proBNP) to predict a composite endpoint of all-cause mortality or HF hospitalization yielded an AUC of 0.71 or 0.69 in derivation, respectively. Herein, scores S3.15 and S3.8 obtained 0.80 and 0.78 AUC OOB values, respectively.

• The AUC of score S3.8 is similar to that of the score proposed by Spinar et al. [START_REF] Spinar | Prognostic Value of NT-proBNP Added to Clinical Parameters to Predict Two-Year Prognosis of Chronic Heart Failure Patients with Mid-Range and Reduced Ejection Fraction-A Report from FAR NHL Prospective Registry[END_REF] to assess the 2-year prognosis of CHF (all-cause mortality, heart transplantation, device implantation), which yielded an AUC of 0.79 without cross-validation nor external validation for a model using 7 variables.

• The MAGGIC risk score [START_REF] Pocock | Predicting Survival in Heart Failure: A Risk Score Based on 39 372 Patients from 30 Studies[END_REF], which has been shown to feature one of the best accuracies to predict 1-year mortality in CHF patients in Canepa et al. [37] using 13 variables and subsequently studied on many validation cohorts, had an AUC between 0.64 and 0.74 in the studies without NT-proBNP [START_REF] Kwon | Artificial Intelligence Algorithm for Predicting Mortality of Patients with Acute Heart Failure[END_REF] [35] [37] [START_REF] Rich | Meta-Analysis Global Group in Chronic (MAGGIC) Heart Failure Risk Score: Validation of a Simple Tool for the Prediction of Morbidity and Mortality in Heart Failure with Preserved Ejection Fraction[END_REF], and of 0.74 with NT-proBNP [START_REF] Khanam | Validation of the MAGGIC (Meta-Analysis Global Group in Chronic Heart Failure) Heart Failure Risk Score and the Effect of Adding Natriuretic Peptide for Predicting Mortality after Discharge in Hospitalized Patients with Heart Failure[END_REF]. Note that the AUC for the composite endpoint of death and hospitalization, as used in the current study, is generally lower than the AUC for all-cause death only. Score S3.8 achieved an AUC OOB of 0.78 using 8 variables.

The main limitation of our application study is that only one dataset was used in our tests. However, the present work is mostly a "proof of concept" of the usefulness of the presented methods of construction of parsimonious ensemble scores.

Conclusion

Variables selection methods based on a probabilistic model can be used to achieve a stepwise selection for a given classifier such as logistic regression, but not directly for a classifier constructed by aggregation of several classifiers. In this article, we have proposed to construct parsimonious ensemble scores using sample balancing, several classifiers, bootstrap samples and stepwise variable selection methods in this setting. As a concrete application, we constructed a short-term event (death or hospitalization for HF at 180 days) score for CHF patients, yielding satisfactory AUC values with respect to other scores in other HF patients' populations. The methods proposed and tested in this article can be reproduced on any delay, any set of variables and any other settings (other types of HF or other diseases) as long as there is a sufficient number of cases, i.e. a sufficiently large training dataset. Applications on other datasets and comparisons with other methods should be conducted in order to confirm the interest of the proposed methods.

Figure 1 .

 1 Figure 1. Methodology of construction of the ensemble score.

  ; b binary variable encoding; c ordinal encoding; d baseline value copied to follow-up visits; e baseline value copied to follow-up visits and updated when possible; f interpolated values. SD: standard deviation; BMI: body mass index; NYHA: New York Heart Association; HF: heart failure; CVP: central venous pressure; ePVS: estimated plasma volume; eGFR: estimated glomerular filtration rate; HDL, high-density lipoprotein; AMI: acute myocardial infarction; COPD: chronic obstructive pulmonary disease; CABG: coronary artery bypass graft; AF; atrial fibrillation; LVEF: left ventricular ejection fraction; ACE: angiotensin-converting enzyme.

Table 1 .

 1 Preselections of variables obtained with the three methods and corresponding AUC OOB of the associated scores. AUC OOB obtained for the score including the variable in the row as well as all previous variables. **The AUC OOB of these columns were obtained by building an intermediate score using only LDA (respectively LR) for the linear part (resp. logistic part) from the selected variables. ***The AUC OOB of this column was obtained by constructing a full ensemble score with the same number of variables for both LDA and LR, using the optimal λ for each score.

		Method 1		Method 2		Method 3a
		Variables	AUC OOB*	Variables	AUC OOB*	Variables	AUC OOB** (LR part)	AUC OOB** (LDA part)	AUC OOB*** (all)
	1	NT-proBNP	0.7246	NT-proBNP	0.7246	NT-proBNP	0.7246 0.7246	0.7246
	2	NYHA ≥ III	0.7482	NYHA ≥ III	0.7482	NYHA ≥ III	0.7482 0.7523	0.7523
	3	Periph. edema ≥ "above"	0.7547	Heart rate	0.7529	NYHA ≥ II	0.7550 0.7579	0.7579
	4	Glycemia	0.7620	Systolic BP	0.7591	Glycemia	0.7621 0.7642	0.7642
	5	Systolic BP	0.7671	NYHA ≥ II	0.7647 Periph. edema ≥ "above" 0.7687 0.7688	0.7694
	6	Beta-blockers	0.7730	Beta-blockers	0.7696	Beta-blockers	0.7731 0.7714	0.7736
	7	NYHA ≥ II	0.7787	Glycemia	0.7764	Systolic BP	0.7791 0.7761	0.7792
	8	Cholesterol HDL	0.7829	Periph. edema ≥ "above"	0.7810	Cholesterol HDL	0.7835 0.7796	0.7835
	9	Mean BP	0.7827	Cholesterol HDL	0.7852	Paroxystic AF	0.7864 0.7831	0.7867
		Diastolic BP	0.7840	Uricemia	0.7885	Uricemia	0.7902 0.7866	0.7904
		Heart rate	0.7861	Bilirubin	0.7912	Bilirubin	0.7925 0.7876	0.7926
		Uricemia	0.7897	Diuretics	0.7913 Implantable defibrillator	0.7948 0.7908	0.7950
		Third heart sound	0.7922	Previous AMI	0.7932	Neoplasia	0.7966 0.7924	0.7968
		Bilirubin	0.7950	Paroxystic AF	0.7953	Third heart sound	0.7984 0.7947	0.7985
		Previous AMI	0.7967	Third heart sound	0.7982	Heart rate	0.8001 0.7963	0.8002
		Paroxystic AF	0.7988	LVEF	0.7990	Previous AMI	0.8020 0.7977	0.8020
		Implantable defibrillator	0.8010	Triglycerides	0.8006	Triglycerides	0.8038 0.7993	0.8038
		Neoplasia	0.8027	Neoplasia	0.8028	LVEF	0.8052 0.8010	0.8052
		LVEF	0.8045	Ascitis	0.8038	Hypertension	0.8067 0.8021	0.8067
		Triglycerides	0.8064	Implantable defibrillator	0.8060	Mitral insufficiency	0.8080 0.8040	0.8080
		Diuretics	0.8070	Hemoglobin	0.8058	Smoker or ex-smoker	0.8091 0.8053	0.8091
		Ascitis	0.8085	ePVS	0.8061	Ascitis	0.8104 0.8060	0.8104
		Mid-apical pulmonary rales 0.8091	Hematocrit	0.8070 Periph. edema ≥ "ankles" 0.8116 0.8069	0.8116
		Smoker or ex-smoker	0.8099	Smoker or ex-smoker	0.8080	NYHA ≥ IV	0.8119 0.8071	0.8119
		Mitral insufficiency	0.8108	Mitral insufficiency	0.8086	BMI	0.8130 0.8084	0.8130
		Hypertension	0.8121	BMI	0.8103 Mid-apical pulmonary rales 0.8137 0.8084	0.8137
		BMI	0.8131	Hypertension	0.8119
		Periph. edema ≥ "ankles" 0.8144 Previous hosp. for worsening HF 0.8119
		Periph. edema ≥ "knee"	0.8151	Mid-apical pulmonary rales	0.8127
		CABG	0.8157	Diabetes	0.8127
		Calcium antagonists	0.8161	CABG	0.8133
		Previous hosp. for worsening HF	0.8166	Periph. edema ≥ "knee"	0.8136

B. Lalloué et al. DOI: 10.4236/am.2021.127045 638 Applied Mathematics *ACE: angiotensin-converting enzyme; AF: atrial fibrillation; AMI: acute myocardial infarction; AUC OOB: area under the ROC curve out-of-bag; BMI: body mass index; BP: blood pressure CABG: coronary artery bypass graft; COPD: chronic obstructive pulmonary disease; CVP: central venous pressure; eGFR: estimated glomerular filtration rate; ePVS: estimated plasma volume; HDL: high-density lipoprotein; HF: heart failure; LVEF: left ventricular ejection fraction; NT-proBNP: N-terminal prohormone brain natriuretic peptide; NYHA: New York Heart Association. B. Lalloué et al. DOI: 10.4236/am.2021.127045 640 Applied Mathematics

Table 2 .

 2 Number of variables needed to obtain an AUC above given thresholds.

	AUC OOB	Method 1	Method 2	Method 3a	Number of variables common to all methods
	≥0.750	3	3	2	2
	≥0.760	4	5	4	2
	≥0.770	6	7	6	4
	≥0.780	8	8	8	7
	≥0.790	13	11	10	9
	≥0.800	17	17	15	13
	≥0.810	25	26	22	21

Table 3 .

 3 Summary of the characteristics of the parsimonious scores constructed using Method 3a.

	Score designation	S3.26	S3.15	S3.8	S3.2
	Data	Working sample defined in Section 3.3. Variables transformed differently for the linear intermediate score and the logistic intermediate score.
	Number of bootstrap samples				1000
	Number of variables used	26		15		8		2
	Number of modalities				2
	λ value	λ = 0.5	λ = 0 (optimal)	λ = 0.5	λ = 0.09 (optimal)	λ = 0.5	λ = 0.06 (optimal)	λ = 0.5	λ = 1 (optimal)
	AUC OOB of the LDA	0.8084	0.7963	0.7796	0.7523
	AUC OOB of the LR	0.8137	0.8001	0.7835	0.7482
	AUC OOB of the final score	0.8121	0.8137	0.7996	0.8002	0.7830	0.7835	0.7502	0.7523
	Sensitivity*	0.861	0.823	0.759	0.724	0.713	0.748	0.810	0.826
	Specificity*	0.611	0.651	0.689	0.719	0.707	0.675	0.551	0.547
	Maximum Youden index	0.472	0.474	0.448	0.443	0.420	0.423	0.361	0.373

*Sensitivity and specificity associated with the maximum value of the Youden index. B. Lalloué et al. DOI: 10.4236/am.2021.127045 641 Applied Mathematics

  BMI and aortic stenosis. In the present study, certain variables used in a number of scores were included in the original set of variables but were not selected in the final scores, such as age, gender, diabetes, serum creatinine, eGFR, hemoglobin or serum sodium. Sex was not significant in univariate analysis. The remainder of these variables were not retained during the forward AUC preselection phase in Method 3a. However, it should be noted that these variables were selected in Methods 1 and 2, generally in the second half of the selection. Finally, the preselection of Method 3a also included less common variables such as glycemia, peripheral edema, cholesterol HDL, bilirubin, implantable defibrillator, neoplasia, triglycerides, mitral insufficiency, as well as history of AMI, hypertension or ascites.

	B. Lalloué et al.
	short-term score in patients with CHF, the most predictive variable was syste-
	matically NT-proBNP, which is a well-known predictor of HF [10] [17] [20] [34]
	[35] [36]. Other explanatory variables, such as NYHA class, systolic blood pres-
	sure, LVEF, BMI, beta-blocker medication, uricemia, atrial fibrillation, heart rate
	or smoking status, have also often been selected in other studies [11] [13] [17]
	[20] [22] [23] [36]. Note that in a previous study on the same 1231 patients from
	the GISSI-HF trial with NT-proBNP, Barlera et al. [10] constructed a mortality
	predictive score using a Cox model and 14 variables: NT-proBNP, hs-cTnT,
	NYHA class, age, COPD, systolic blood pressure, diabetes, eGFR, sex, uricemia,
	LVEF, hemoglobin,
	DOI: 10.4236/am.2021.127045	642	Applied Mathematics

Material for "Construction of Parsimonious Event Risk Scores by an Ensemble Method. An Illustration for Short-Term Predictions in Chronic Heart Failure Patients from the GISSI-HF Trial" A1. Summary Statistics of the Sample before Data ManagementTable A1 .

 A1 Descriptive statistics of the explanatory variables after winsorization and sample balancing performed before transformation of the variables. derived variable; b binary variable encoding; c ordinal encoding; d baseline value copied to follow-up visits; e baseline value copied to follow-up visits and updated when possible; f interpolated values;

	B. Lalloué et al.
		Continued Supplementary Variables Serum creatinine g eGFR a,g,h Serum potassium g Serum sodium g Uricemia g Triglycerides g Cholesterol HDL g Total Cholesterol g		Renal function ----Groups of related variables Mean (SD) or N (%) 1.27 (0.44) 64.08 (22.63) 4.48 (0.50) 139.49 (3.33) 6.43 (1.94) 137.92 (84.01) 47.58 (13.19) Cholesterol 175.10 (44.48)
		Female b,d Bilirubin g		--	2227 (19.5%) 0.84 (0.42)
		Age a,g Glycemia g		--	68.10 (10.20) 122.98 (46.60)
		Years of school education d,g NT-proBNP f,g	--	6.92 (3.65) 1856.60 (2194.91)
		Weight g Diabetes mellitus b,d BMI a,g Hypertension b,d		-Obesity -	75.87 (14.33) 3481 (30.5%) 26.96 (4.48) 6470 (56.7%)
		Smoker or ex-smoker b,d Previous AMI b,e	--	6645 (58.2%) 5421 (47.5%)
		Heart Rate g Previous stroke b,e		--	72.49 (13.38) 643 (5.6%)
		Diastolic blood pressure g Previous hosp. for worsening HF b,e	-	76.28 (10.17) 6526 (57.2%)
		Systolic blood pressure g Angina pectoris b,e	Blood pressure -	125.21 (19.41) 2060 (18.1%)
		Mean blood pressure a,g Coronary angioplasty b,d	-	92.58 (12.17) 1478 (13.0%)
		≥II Transient ischemic attack (TIA) b,d	-	10837 (95.0%) 1228 (10.8%)
		NYHA class c (ref: "NYHA I") COPD b,d	≥III	NYHA -	3061 (26.8%) 2348 (20.6%)
		CABG b,e	≥IV	-	242 (2.1%) 2847 (24.9%)
		≥Ankles Implantable defibrillator b,d	-	1768 (15.5%) 1020 (8.9%)
		Peripheral edema c,d Paroxystic AF b,d (ref: "No") Neoplasia b,d	≥Knee ≥Above	Peripheral edema --	316 (2.8%) 2756 (24.2%) 159 (1.4%) 592 (5.2%)
		Cardiomyopathy Definitive pace maker b,d	--	3126 (27.4%) 1944 (17.0%)
		Main cause of HF b Waiting for cardiac transplantation b,d Hypertension	--	1726 (15.3%) 122 (1.1%)
		(ref: "Ischemic") LVEF d,g	Other	--	346 (3.0%) 32.58 (10.05)
		Not known Bundle branch block b	--	175 (1.5%) 3883 (34.0%)
		Ascites b,d Atrial fibrillation b		--	147 (1.3%) 2087 (18.3%)
		Hepatomegaly b,d Left ventricular hypertrophy b	--	2188 (19.2%) 1885 (16.5%)
		Mitral insufficiency b,d Pathological Q waves b	--	5461 (47.9%) 2236 (19.6%)
		CVP > 6 cm H20 b,d Normal ECG evaluation b Basal pulmonary rales b,d ACE-inhibitors a,b Mid-apical pulmonary rales b,d Beta-blockers a,b Pulmonary rales b,d Calcium antagonists a,b Aortic stenosis b,d Diuretics a,b	----------	1139 (10.0%) 415 (3.6%) 8782 (77.0%) 1732 (15.2%) 7430 (65.1%) 79 (0.7%) 803 (7.0%) 599 (5.2%) 315 (2.8%) 10813 (94.8%)
		Third heart sound (S3) b,d	-	2177 (19.1%)
		Hematocrit g			40.16 (4.53)
		Hemoglobin g		Hematology	13.40 (1.60)
		ePVS a,g			4.57 (0.92)
	DOI: 10.4236/am.2021.127045 DOI: 10.4236/am.2021.127045	647 648			Applied Mathematics Applied Mathematics

a g winsorized variable. SD: standard deviation; BMI: body mass index; NYHA: New York Heart Association; HF: heart failure; CVP: central venous pressure; ePVS: estimated plasma volume; eGFR: estimated glomerular filtration rate; HDL, high-density lipoprotein; AMI: acute myocardial infarction; COPD: chronic obstructive pulmonary disease; CABG: coronary artery bypass graft; AF: atrial fibrillation; LVEF: left ventricular ejection fraction; ACE: angiotensin-converting enzyme. B. Lalloué et al. DOI: 10.4236/am.2021.127045 649 Applied Mathematics

Table A2 .

 A2 Descriptive statistics of the explanatory variables prior to the data management phase.

	Variables		Groups of related variables	Mean (SD) or N (%)
	Female b,d		-	1219 (19.7%)
	Age a		-	66.94 (10.76)
	Years of school education d	-	7.00 (3.77)
	Weight BMI a		Obesity	76.25 (14.76) 26.96 (4.46)
	Smoker or ex-smoker b,d	-	3425 (55.3%)
	Heart rate		-	70.22 (13.25)
	Diastolic blood pressure		77.37 (10.29)
	Systolic blood pressure	Blood pressure	127.16 (18.75)
	Mean blood pressure a		93.96 (12.07)
		≥II		5671 (91.6%)
	NYHA c (ref: "NYHA I")	≥III	NYHA	1017 (16.4%)
		≥IV		46 (0.74%)
		≥Ankles		732 (11.8%)
	Peripheral edema c,d (ref: "No")	≥Knee	Peripheral edema	106 (1.7%)
		≥Above		33 (0.5%)
		Cardiomyopathy	-	1866 (30.2%)
	Main cause of HF b	Hypertension	-	956 (15.4%)
	(ref: "Ischemic")	Other	-	178 (2.9%)
		Not known	-	133 (2.1%)
	Ascites b,d

Acknowledgements

The authors thank Mr. Pierre Pothier for editing this manuscript. Results incorporated in this article received funding from the investments for the Future program, France under grant agreement No ANR-15-RHU-0004.

Conflicts of Interest

The authors declare no conflicts of interest regarding the publication of this paper. DOI: 10.4236/am.2021.127045 651 Applied Mathematics

A2. Alternative Methods: Description and Results

A2.1. Method 3b

Preselection of variables: Two forward preselections using AUC as criterion were performed, one for logistic regression (LR) and the other for linear discriminant analysis (LDA). Let t denote a stopping time. For 1, 2,..., i t = : at step i:

Vwere available from step 1 i -. For every set of va- riables 1 1 ,..., ,

,..., 1 j i ≠ -, a logistic regression (respectively a li- near regression) was performed on the entire sample without bootstrap. The variable, denoted i V , yielding the maximal AUC in resubstitution was included for the step 1 i + , provided that the AUC significantly increased using DeLong's test; otherwise, the inclusion of variables was stopped ( t i = ). The preselection using logistic regression (respectively LDA) was used to build an intermediate LR score (respectively an intermediate LDA score). Note that the number of preselected variables and the preselected variables themselves may differ between the two preselections. Note that, contrary to the preselection phase of Method 1 with AIC, there is no need in this instance for a probabilistic model. Indeed, the AUC can be computed as long as there is a prediction, without assumption on the manner with which this prediction was obtained.

Construction of intermediate scores: For each classifier, intermediate scores using only the associated selected variables were constructed, using 1000 bootstrap samples (the same for both classifiers) and two modalities of selection of variables (all variables or all groups of related variables). Since the preselection was performed separately for both classifiers, intermediate scores may not use the same variables.

Construction of final scores: The two intermediate scores were aggregated in a final score by averaging their prediction for each statistical unit. Since the intermediate scores in this method were constructed independently from each other on two different sets of variables, there were multiple ways to combine the latter. In this instance, intermediate scores using the same number of preselected variables by classifier were aggregated in a final score.

A2.2. Method 3c

Preselection of variables: A forward preselection using AUC as criterion was performed using LDA. Let t denote a stopping time. For 1, 2,..., i t = : at step i:

-, a linear regression was performed on the entire sample without bootstrap. The variable, denoted i V , yielding the maximal AUC in resubstitution was included for the step 1 i + , provided that the AUC significantly increased using DeLong's test; otherwise, the inclusion of variables was stopped ( t i = ).

Note that, contrary to the preselection phase of Method 1 with AIC, there is no need in this instance for a probabilistic model. Indeed, the AUC can be computed as long as there is a prediction for each statistical unit, without assumption on the manner with which this prediction was obtained.