Koen W De Bock
email: kdebock@audencia.com

Arno De Caigny
email: a.de-caigny@ieseg.fr

Spline-Rule Ensemble Classifiers with Structured Sparsity Regularization for Interpretable Customer Churn Modeling

Keywords: customer churn prediction, predictive analytics, spline-rule ensemble, interpretable data science, sparse group lasso, regularized regression

Spline-rule ensemble classifiers with structured sparsity regularization for interpretable customer churn modeling

Introduction

An important application of data science is to drive and to support data-driven decision making.

Many decision makers are convinced that the use of customer-data capabilities allows to gain an unbeatable competitive advantage [START_REF] Hagius | When Data Creates Competitive Advantage[END_REF]. Therefore, modern companies have developed the analytical and technological capabilities that enable collection, storage and analysis of data. An important business domain that relies heavily on advanced statistical -and machine learning algorithms to support operational decision making is customer retention management [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF]. Customer churn prediction (CCP) is of crucial importance for managing customer retention as a tool to identify customers who are at risk to abandon the company and to better understand why customers are at risk [START_REF] Ascarza | Pursuit of Enhanced Customer Retention Management: Review, Key Issues, and Future Directions[END_REF]. In line with these managerial objectives of CCP models, previous research in CCP focused both on predictive performance (i.e. detecting who is at risk) [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF][START_REF] Óskarsdóttir | Social network analytics for churn prediction in telco: Model building, evaluation and network architecture[END_REF] and interpretability (i.e. understanding why a customer is at risk) [START_REF] Gustafsson | The Effects of Customer Satisfaction, Relationship Commitment Dimensions, and Triggers on Customer Retention[END_REF].

Accuracy in CCP is generally pursued due to its immediate impact on campaign profitability [START_REF] Neslin | Defection detection: Measuring and understanding the predictive accuracy of customer churn models[END_REF]. Model interpretability is crucial to facilitate management buy-in and organizational acceptance, to deliver insights into the drivers of churn and loyalty and consequently, to provide venues for formulating strategies to remedy customer churn and promote loyalty [START_REF] Masand | CHAMP: A prototype for automated cellular churn prediction[END_REF][START_REF] De Bock | Reconciling performance and interpretability in customer churn prediction using ensemble learning based on generalized additive models[END_REF].

Algorithms that combine good predictive performance and interpretable output, such as decision trees (DT) or logistic regression (LR), are preferred in CCP [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF][START_REF] Martens | Performance of classification models from a user perspective[END_REF]. Ensemble learners can achieve higher predictive performance, but often lack on the interpretability criterion [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF]. A notable exception are rule ensembles (RE), a technique that is designed to combine the merits of ensemble learners with a high degree of interpretability [START_REF] Friedman | Predictive learning via rule ensembles[END_REF]. Like many other ensemble learners, rule ensembles first generate a set of decision trees. However, unlike other ensemble learners, trees are decomposed into rules and only a dense set of the rules derived from these trees is retained though the application of lasso regression. The initial variables are also added to the lasso regression in the form of linear basis functions (i.e., variable transformations) to better account for linear variable effects. Rule ensembles thus combine terms rather than member classifiers. The simple nature of the constituent terms that form the model and their selection through lasso regression result in an easily interpretable model. Recently, spline-rule ensembles (SRE) are presented as an extension to rule ensembles that complement rules and linear terms with single-term spline functions in order to better accommodate univariate, nonlinear relationships between the dependent variable and individual explanatory variables [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF].

Whilst the promise of competitive predictive performance and model interpretability has attracted attention in several domains such as bioinformatics and computer science [e.g. [START_REF] Klepsch | Impact of the Recent Mouse P-Glycoprotein Structure for Structure-Based Ligand Design[END_REF][START_REF] Ouyang | A large-scale empirical analysis of email spam detection through network characteristics in a stand-alone enterprise[END_REF], applications of rule ensembles in management, and more specifically, decision support in business, remain scarce to date. In an application of corporate bankruptcy prediction, SRE demonstrated superior performance over conventional RE whilst the added value of the integration of spline functions was demonstrated [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF].

Despite their promising traits, other applications of RE and SRE in business decision-making problems are very scarce and to the best of our knowledge RE and SRE have not been empirically assessed for predicting customer churn thus far. This study's primary objective is to evaluate and compare both model architectures in the domain of CCP.

RE and SRE rely on lasso regression, which does not consider relatedness that exists between covariates. This is, nevertheless, very important to consider, because the building blocks of SRE (i.e. splines, linear base functions and rules) can share a dependence on the same variables, which can cause the model to become unnecessarily complex. Imagine for example the impact on a model's ease of interpretation if a variable enters the model in three terms: a linear base function, a spline and a rule. In such a case, an analyst would face difficulties to assess the isolated effect of that variable on the churn probability. These issues are aggravated when conflicting parameter estimate signs emerge. To tackle these issues and significantly improve the interpretability of SRE, the second objective of this study is to propose a new algorithm entitled spline-rule ensembles with sparse group lasso regularization (SRE-SGL). SRE-SGL groups rule, spline and linear terms according to the variables upon which they depend by applying a straightforward indexing function. This term grouping is followed by sparse group lasso (SGL) regularization [START_REF] Simon | A sparse-group lasso[END_REF] that accommodates this group structure by enforcing regularization between as well as within term groups. As such, the co-occurrence of terms that depend on the same variable or variable set is discouraged and the complexity of the resulting model is reduced in comparison to a conventional SRE model. The contributions of this paper are the following: (i) RE and SRE are evaluated and compared in the field of CCP and their ability to reconcile accuracy and model interpretability is assessed; and (ii) SRE-SGL, extending spline-rule ensembles with sparse group lasso regularization, is introduced as a natural extension of generic RE and SRE that simplifies model interpretation. To assess and compare predictive performance of RE, SRE and the new SRE-SGL, as well as a set of benchmark algorithms, experiments are conducted on a large set of 14 data sets containing real-world customer churn data sets in various sectors to compare RE and its extensions with a set of benchmark algorithms in terms of predictive performance. The added value offered by SRE-SGL in comparison to RE and SRE in terms of model interpretability is illustrated using an in-depth case study. This paper is structured as follows. In the next section related research is discussed. This involves three subsections: Section 2.1 discusses the concept of interpretability in data science. Section 2.2 discusses prior literature in customer churn prediction that focusses on the trade-off between accuracy and interpretability. Section 2.3 introduces rule-based ensemble classifiers and their applications. Section 3 presents the methodology. Section 4 handles the data and the experimental design. The results of our large benchmark experiment and a case study to demonstrate the interpretability of SRE-SGL are discussed in section 5. The study's conclusions, limitations and areas for future research are presented in section 6.

Related Literature

Interpretability in Data Science

Interpretability is an important topic in data science and various approaches have been proposed for explaining model predictions [START_REF] Ribeiro | Why should i trust you?" Explaining the predictions of any classifier[END_REF][START_REF] Martens | Explaining data-driven document classifications[END_REF]. Interpretability cannot be described in a pure mathematical formula, and depends on human interpretation. Hence, interpretability can be defined as the degree to which humans can understand the cause of a decision [START_REF] Biran | Explanation and Justification in Machine Learning: A Survey[END_REF][START_REF] Miller | Explanation in artificial intelligence: Insights from the social sciences[END_REF]. As the ability to understand the cause of a decision depends on the observer, interpretability is a subjective topic. Nevertheless, it is an important dimension to consider for model evaluation to ensure that predictions are unbiased, sensitive information is protected, the reliability and robustness of the model is checked and that humans can trust the model [START_REF] Molnar | Interpretable Machine Learning. A Guide for Making Black Box Models Explainable[END_REF].

Approaches to explain model predictions vary in scope and flexibility [START_REF] Martens | Explaining data-driven document classifications[END_REF]. The scope indicates the level of explanations and can either be on the global or on the instance level. Global explanations give an insight in the model's predictions over all observations and for all possible variables' values.

Instance-level explanations, on the other hand, are specific for a single prediction and help to understand why a certain instance received a specific prediction. Flexibility indicates whether the approach is specific to the model or model-agnostic. Flexibility is linked to the way interpretability is achieved.

Intrinsic interpretable models achieve interpretability by restricting the complexity of the machine learning algorithm and their interpretability is thus often model-specific. The model can also be analyzed after training using so called post-hoc methods, which are often model-agnostic approaches.

Our approach focuses on global, model-specific interpretability. The output of SRE-SGL is intrinsically interpretable, which allow to interpret the model's output directly as demonstrated in the case study in section 5.2.

Interpretable Customer Churn Prediction

CCP models serve a dual purpose to decision makers; detecting customers who are at risk of churning and helping to understand why customers are at risk of churning [START_REF] Ascarza | Pursuit of Enhanced Customer Retention Management: Review, Key Issues, and Future Directions[END_REF]. Therefore, CCP models require not only high predictive performance but also interpretable output. In this section, we review literature in the CCP domain that focuses on churn prediction modeling as tool for better decision making.

The predictive performance of CCP models is a well-researched topic because of its importance for decision makers to detect which customers are at risk of churning. There are many strategies to improve the predictive performance of CCP models such as intelligent data preprocessing [START_REF] Coussement | A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry[END_REF], data augmentation [START_REF] De Caigny | Incorporating textual information in customer churn prediction models based on a convolutional neural network[END_REF] or by the choice of algorithm [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF]. Given the motivation of the focal study, we focus on the latter. Researchers have experimented with a wide range of algorithms in extensive benchmarking studies [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF]. Such studies focused on the algorithms' ability to discriminate between churningand non-churning customers. Logistic regression is the standard benchmark algorithm in CCP because of its ability to produce decent and robust results [START_REF] Coussement | A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry[END_REF][START_REF] Neslin | Defection detection: Measuring and understanding the predictive accuracy of customer churn models[END_REF]. More complex algorithms, however, frequently perform significantly better in terms of predictive performance [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF][START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF][START_REF] De Caigny | Incorporating textual information in customer churn prediction models based on a convolutional neural network[END_REF][START_REF] Coussement | Churn prediction in subscription services: An application of support vector machines while comparing two parameter-selection techniques[END_REF]. The results in large benchmarking studies demonstrate that especially ensembles, such as random forests, perform well [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF].

Despite the beneficial traits in terms of predictive performance, interpretability of ensembles remains an issue which causes that they are not always the preferred option.

CCP models should be interpretable in order to assist decision makers in managing customer retention. Recent studies in CCP explicitly acknowledge the importance of interpretability of predictive models. Martens et al. [START_REF] Martens | Performance of classification models from a user perspective[END_REF] propose a complete framework to assess the overall performance of classification models from a user perspective in terms of accuracy, interpretability and justifiability. In their analysis, interpretability is based on the output type and output size. They state that some output types, such as rules or linear ones, and smaller output sizes are intuitively easier to understand for humans.

A first strategy to obtain interpretable models in CCP is by making non-interpretable output of so called "black-box models" more interpretable through additional analyses. On the one hand, several model-agnostic interpretation techniques exist that to reveal the magnitude and nature of the effect that variables exert on a model's predictions. Notable examples are permutation-based feature importance scores, and partial dependence functions and plots. Both techniques have witnessed widespread adoption in CCP literature. On the other hand, transparent surrogate classifiers can be created to complement, or replace, opaque models. An example of this approach is rule extraction. For example, Verbeke et al. [START_REF] Verbeke | Building comprehensible customer churn prediction models with advanced rule induction techniques[END_REF] experimented with new rule induction techniques, which induce accurate as well as interpretable classification rule-sets. Farquad et al. [START_REF] Farquad | Churn prediction using comprehensible support vector machine: An analytical CRM application[END_REF] propose a hybrid approach to render interpretable rule-based output for a support vector machine model. A drawback of such methods is that they only approximate the original model. The development of interpretable models is a second strategy. Miguéis et al. [START_REF] Miguéis | Customer attrition in retailing: An application of Multivariate Adaptive Regression Splines[END_REF] introduce multivariate adaptive regression splines (MARS [START_REF] Friedman | Multivariate Adaptive Regression Splines[END_REF]) to customer churn prediction and highlight its ability to uncover nonlinear effects. Coussement et al. [START_REF] Coussement | Improved marketing decision making in a customer churn prediction context using generalized additive models[END_REF] introduced Generalized Additive Models (GAM) as a highly interpretable model in CCP. Several extensions of GAM have been presented that improved the predictive performance while maintaining its interpretability [START_REF] Coussement | Customer churn prediction in the online gambling industry: The beneficial effect of ensemble learning[END_REF][START_REF] De Bock | Reconciling performance and interpretability in customer churn prediction modeling using ensemble learning based on generalized additive models[END_REF][START_REF] De Bock | Ensemble classification based on generalized additive models[END_REF]. Other interpretable models depend at least partly on a tree-based structure. Qi et al. [START_REF] Qi | ADTreesLogit model for customer churn prediction[END_REF] introduce ADTreesLogit, a model that integrates the advantage of ADTrees in the logistic regression model, to improve the predictive accuracy and interpretability of existing churn prediction models. De Caigny et al. [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF] introduce the logit leaf model, a highly interpretable hybrid model based on decision trees and logistic regression that delivers actionable insights.

A final strategy involves imposing the interpretability criterion in the feature engineering. Backiel et al. [START_REF] Backiel | Predicting time-to-churn of prepaid mobile telephone customers using social network analysis[END_REF] demonstrate how interpretable features can be extracted out of call records by using social network analysis. The use of these network features can improve the performance over local features while remaining highly interpretable. Verbraken et al. [START_REF] Verbraken | Profit optimizing customer churn prediction with Bayesian network classifiers[END_REF] stress the importance of compact networks derived from a Bayesian network classifier for the model interpretability. In the telecommunication industry, Lima et al. [START_REF] Lima | Domain Knowledge Integration in Data Mining Using Decision Tables: Case Studies in Churn Prediction[END_REF] show how domain knowledge can be incorporated in the data mining process for churn prediction.

The proposed SRE-SGL fits perfectly in the interpretable customer churn prediction literature, because SRE-SGL combines excellent predictive performance, associated with ensemble learning, and interpretability. The SRE-SGL algorithm has two main strategies to ensure interpretability. First, it returns inherently comprehensible output that can be directly analyzed by decision makers. Second, the model automatically introduces new, potentially insightful features through the combination of splines, linear base functions and rules. Such features can shed new light in the understanding of what is driving customer churn. To the best of our knowledge, RE and its extensions have never been used in CCP.

Rule-Based Ensemble Classifiers and Applications

Ensemble classification prescribes the training of multiple base learners, or member classifiers into one model and the use of a fusion rule to aggregate their individual outputs into an overall prediction [START_REF] Kuncheva | Combining pattern classifiers: methods and algorithms[END_REF]. The most well-known methods differ in the strategy deployed to transform the training data set into member training data for each base learner. Examples include iterative weighing of instances in adaboost [START_REF] Freund | Experiments with a new boosting algorithm[END_REF], bootstrap sampling in bagging [START_REF] Breiman | Bagging predictors[END_REF] , combining bagging and random feature selection at the node level in random forests [START_REF] Breiman | Random forests[END_REF], and feature extraction in rotation forests [START_REF] Rodríguez | Rotation forest: A new classifier ensemble method[END_REF]. While the subject of the algorithm choice for generating an ensemble's base learners is widely investigated, decision trees are still the most popular and the default option in the aforementioned ensemble strategies.

This study builds on previous work that has investigated the merits of deploying decision rules as base learners in ensemble classifiers. Decision rules can be interpreted as simple classifiers that take the form of logical expressions: if [conditions] then [decision]. The earliest surfacing of rule-based ensemble learning is, to the best of our knowledge, the SLIPPER algorithm [START_REF] Cohen | A Simple, Fast, and Effective Rule Learner[END_REF] that uses boosting to create an ensemble of decision rules. Subsequently, Rule ensembles were proposed by Friedman and Popescu [START_REF] Frieman | Predictive learning via rule ensembles[END_REF] to denote a class of ensemble learners for classification and regression that derive rules from decision trees and use them as base learners in a combination scheme based on regularized regression. A related approach was presented by Błaszczyński et al. [START_REF] Błaszczyński | Ensemble of decision rules[END_REF] and Dembczyński, Kotłowski and Słowiński [START_REF] Dembczyński | Maximum Likelihood Rule Ensembles[END_REF] who generate rules directly and use a different loss criterion. Since then, several variations, extensions and applications of rule ensembles have been presented, for example in cancer classification [START_REF] Yang | Mining diagnostic rules of breast tumor on ultrasound image using cost-sensitive RuleFit method[END_REF], sensor fault classification [START_REF] Mohapatra | Weighted majority rule ensemble classifier for sensor fault classification for plasma position control in Tokamaks[END_REF], analysis of start-up performance [START_REF] Debrulle | Configurations of Business Founder Resources, Strategy and Environment Determining New Venture Performance[END_REF] and streetscape satisfaction [START_REF] Shimokawa | Modified Rule Ensemble Method for Binary Data and Its Applications[END_REF]. The proposed SRE-SGL in this study extends the approach described in [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF], in which rule ensembles were applied in the field of bankruptcy prediction and an extension, spline-rule ensembles, demonstrated a significant improvement in predictive performance over conventional rule ensembles.

Methodology

Rule and Spline-Rule Ensembles

In contrast to many well-known ensemble learners that combine decision trees, RE [START_REF] Frieman | Predictive learning via rule ensembles[END_REF] initiates by deriving rules from decision trees and use them as base learners in a supervised, linear combination scheme. Consider a data set D with an input vector X summarizing n instances on p features 𝑥 𝑘 ; 𝑘 = 1 to 𝑝 and an outcome vector Y. Specifically, RE derive rules 𝑟 𝑗 (x); j=1 to q from a set of decision trees trained on X and Y for all internal and terminal nodes within every tree (interior and terminal). A rule 𝑟 𝑗 (x) is the product of the indicator functions that define whether input vector x meets certain criteria defined on one or more variables:

𝑟 𝑗 (𝑥) = ∏ 𝐼(𝑥 𝑘 ∈ 𝑠 𝑗𝑘 𝑠 𝑗𝑘 ≠𝑆 𝑘 ) ( 1
)
where 𝑠 𝑗𝑘 represents a range or subset of values of variable 𝑥 𝑘 and 𝑆 𝑘 denotes the full range or set of values of this variable. Variables upon which a rule 𝑟 𝑗 (x) depends (i.e., for which 𝑠 𝑗𝑘 ≠ 𝑆 𝑘 ) are called defining variables. The rules are complemented by linear basis functions 𝑙(𝑥 𝑘 ) ; 𝑘 = 1, … , 𝑝 which denote variables 𝑥 𝑘 subsequently subjected to winsorization and normalization as defined by:

𝑙(𝑥 𝑘 ) = 0.4 * 𝑤𝑖𝑛(𝑥 𝑘 ) 𝑠𝑑(𝑤𝑖𝑛(𝑥 𝑘 )) (2)
with 𝑤𝑖𝑛(𝑥 𝑘 ) = min (𝛿 𝑘 + , max (𝛿 𝑘 -, 𝑥 𝑘 )) denoting the winsorized version of variable 𝑥 𝑘 and where 𝛿 𝑘 and 𝛿 𝑘 + specify the β th and (1-β) th percentiles of 𝑥 𝑘 1 .

The final model takes the form of a linear regularized lasso-regression [START_REF] Tibshirani | Regression shrinkage and selection via the lasso[END_REF] applied to outcome vector Y and an intermediate term matrix T that contains values for p+q terms 𝑡(𝑥): p variable transformations and q binary rule outcomes for all instances in a training data set.

𝑎𝑟𝑔 𝑚𝑖𝑛 𝛽 1 2 ‖𝑦 -𝑋𝛽‖ 2 2 + 𝜆‖𝛽‖ 1 (3)
Where shrinkage parameter λ controls sparsity: increasing values decrease the proportion of non-zero parameter estimates. ‖. ‖ 2 and ‖. ‖ 1 are the ℓ 2 and ℓ 1 vector norms, respectively.

Several metrics allow an identification of the relative importance and nature of relationships of the terms selected in a rule ensemble model. The first are term coefficients, i.e. the regression parameter estimates that indicate the influence a term has upon the logit transformation of the probability to churn.

These deliver insight into whether a term influences churn positively or negatively, and to which extent.

Furthermore, rule support figures are specific to rule terms and indicate for which percentage of instances in the training data set a rule applies. Finally, term importance measures reflect the relative importance of the terms in the model and are obtained through 1 The constant of 0.4 is chosen so that linear terms in a subsequent regularized lasso-regression (equation ( 3)) receive the same a priori influence as a typical rule. Specifically, this constant reflects the average standard deviation that characterizes a population of rules characterized by a support that follows a uniform distribution; supp (r j (x)) ~𝑈(0,1). We kindly refer the reader to Friedman & Popescu [START_REF] Friedman | Predictive learning via rule ensembles[END_REF] for more details.

𝑇𝐼(𝑡) = { |𝛽 𝑡 |√𝑠𝑢𝑝𝑝 (𝑟 𝑗 (𝑥)) (1 -𝑠𝑢𝑝𝑝 (𝑟 𝑗 (𝑥))) if 𝑡 is a rule term; 𝑡 = 𝑟 𝑗 (𝑥) |𝛽 𝑡 |. 𝑠𝑑(𝑙(𝑥 𝑘 )) if 𝑡 is a linear term; 𝑡 = 𝑙(𝑥 𝑘 ) |𝛽 𝑡 |. 𝑠𝑑 (𝑠 𝑔 (𝑥 𝑔 )) if 𝑡 is a spline term; 𝑡 = 𝑠 𝑔 (𝑥 𝑔 ) (4)
where 𝑠upp (𝑟 𝑗 (𝑥)) represents the rule support for rule 𝑟 𝑗 (𝑥) [START_REF] Friedman | Predictive learning via rule ensembles[END_REF]. Hence, 𝑇𝐼(𝑡) represents the absolute value of the regression coefficient of a standardized term 𝑡.

Due to their nature, rule ensembles balance model flexibility and interpretability, which are classifier qualities that often conflict. Model interpretability in rule ensembles stems from the process of creating simplified, easily understandable base learners, while the regularization enforces model sparsity as many terms receive a parameter estimate equal to 0. Besides linear variable effects, variable interactions are naturally accommodated through the inclusion of rules. Moreover, these rules allow an identification of non-linear effects of individual variables on an outcome variable. However, this is only possible in an indirect manner when multiple rules, defined on the same variable are selected simultaneously.

Spline-rule ensembles aim for a more direct support of non-linear effects. To this end, a third term class was introduced in [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF]: smooth functions, and in particular penalized cubic regression splines [START_REF] Wood | Generalized additive models: an introduction with R[END_REF] of individual continuous variable. Penalized cubic regression splines determine a set of v knots ξ 1 , ξ 2 , … , ξ 𝑣 over a variable's range and estimate a function that is built up of cubic polynomials between every pair of adjacent knots. As such, they allow to model a non-linear relation between a variable and the customer churn probability. Specifically, 𝑠(𝑥) (the cubic regression spline function) takes the form

𝑠(𝑥) = 𝛽 0 + 𝛽 1 𝑥 + 𝛽 2 𝑥 2 + 𝛽 3 𝑥 3 + 𝛽 4 ℎ(𝑥, 𝜉 1 ) + ⋯ + 𝛽 𝑣+3 ℎ(𝑥, 𝜉 𝑣 ) with ℎ(𝑥, 𝜉) = { (𝑥 -𝜉) 3 if 𝑥 > 𝜉 0 otherwise . ( 5
)
Minimizing

∑ (𝑦 𝑖 -𝑠(𝑥 𝑖 )) 2 𝑛 𝑖=1 -𝜌 ∫(𝑠"(𝑥)) 2 𝑑𝑥 (6)
allows a determination of values for β 1 , β 2 , … , β 𝑣+3 , ξ 1 , ξ 2 , … , ξ 𝑣 as well as ρ which represents a smoothing parameter, i.e. a penalty term that is required to penalize excessive curvature in the function.

The inclusion of penalized cubic regression spline functions 𝑠(𝑥) for all continuous variables 𝑥 1 , … , 𝑥 𝑢 (𝑢 ≤ 𝑝) in term matrix T in equation ( 3) updates the lasso regularization that estimates the final model. Note that in [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF] experiments compared lasso regularization to ridge regression and elastic net regularization by generalizing equation [START_REF] Ascarza | Pursuit of Enhanced Customer Retention Management: Review, Key Issues, and Future Directions[END_REF]. Results of a model variant comparison in the field of bankruptcy prediction [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF] demonstrated no significant differences. Hence, in the current study these variants are not investigated further.

Sparse-Group Lasso (SGL) Regularized Regression

Rule and spline-rule ensembles rely on lasso regression to perform ensemble selection and improve interpretability through shrinkage. A limitation of lasso regression is that it does not take into account relatedness (a group structure) that exists between covariates. Variations of lasso regression enable structured regularization. Specifically, the group lasso [START_REF] Yuan | Model selection and estimation in regression with grouped variables[END_REF] and sparse-group lasso (SGL) [START_REF] Simon | A sparse-group lasso[END_REF] allow variable grouping. In the case of the former, sparsity is enforced on the group level so that all variables within a selected group receive non-zero parameter estimates when their group is selected and 0 otherwise. In the case of SGL, a dual goal of sparsity is pursued: both at the between-group as the withingroup level. In other words, the regression attempts to shrink the model to as few group as possible, and to as few variables within selected groups as possible. The SGL takes the following form:

𝑎𝑟𝑔 𝑚𝑖𝑛 𝛽 1 2𝑛 ‖𝑦 -∑ 𝑇 (𝑜) 𝛽 (𝑜) 𝑚 𝑜=1 ‖ 2 2 + (1 -𝛼)𝜆 ∑ √𝑝 𝑜 ‖𝛽 (𝑜) ‖ 2 + 𝛼𝜆‖𝛽‖ 1 𝑚 𝑜=1 (7)
In which m is the number of variable groups, 𝑇 (𝑜) is the partial term matrix reduced to variables that belong to group o, 𝑝 𝑜 is the number of variables in group o; and shrinkage is controlled by parameters λ (the shrinkage parameter) and α (0≤α≤1), the mixing parameter that controls the trade-off between between-and within-group level regularization.

Spline-Rule Ensembles with SGL Regularization (SRE-SGL)

A notable disadvantage of lasso regression is that it enforces shrinkage through a loss function that does not necessarily avoid a simultaneous selection of multiple terms that rely on the same underlying variable(s). Model interpretability becomes more challenging when a variable 𝑥 𝑘 enters a model simultaneously as a linear basis function 𝑙(𝑥 𝑘 ), a spline 𝑠 𝑘 (𝑥 𝑘 ) and a (univariate) rule 𝑟 𝑙 (𝑥 𝑘 ). Likewise, the occurrence of similar multivariate rules that share identical defining variables complicates interpretation. These issues are aggravated when conflicting parameter estimate signs emerge. To tackle these issues and significantly improve the interpretability of spline-rule ensembles we define SRE-SGL as spline-rule ensembles with term grouping and structured regularization using SGL. SRE-SGL allows a decision maker to obtain a more interpretable model by leveraging relatedness between splines, linear basis functions and rules in term matrix T that share a dependence on the same variables.

Figure 1 graphically depicts the core mechanisms of SRE-SGL. The training process comprises of three stages. The first stage is identical to regular spline-rule ensembles and involves the derivation of linear basis functions, tree rules and penalized cubic regression splines that will serve as candidate ensemble members. The second stage, proper to SRE-SGL, involves term grouping. This involves the grouping of terms in term matrix T according to the variables upon which they depend. Specifically, we propose the following indexing function:

𝑡𝑔(𝑡(x 𝑠 )) ∶= 𝑤: 𝑥 𝑠 = 𝑠 𝑤 ∈ 𝑆 (8)
Where t is a term in term matrix T, x 𝑠 is the set of variables upon which t depends; S is the indexed set of unique defining term variable sets that identifies |𝑆| = 𝑚 groups and s 𝑤 is the w th element of S.

For example, a multivariate term such as a rule 𝑟 𝑣 (𝑥 1 , 𝑥 3 ) that contains conditions on variables 𝑥 1 and 𝑥 3 would contribute the set {𝑥 1 , 𝑥 3 } to S while univariate terms such as smoothing spline 𝑠 2 (𝑥 2 ) and linear basis function 𝑙(𝑥 2 ) contributes singleton {𝑥 2 } to S. 𝑡𝑔(𝑡(x 𝑠 )) assigns a unique grouping index to every unique set of defining variables that emerges in the terms of matrix T. This encourages the term selection to choose between alternative terms that are defined on identical variables or variable sets. In the case of univariate terms, shrinkage involves selection within term sets (such as 𝑇 (1) and 𝑇 (𝑝) in Figure 1) consisting of univariate rules, linear basis functions, splines or any subset of these, each dependent on the same variable. In the case of multivariate terms (i.e., rules with multiple conditions) this involves shrinkage within term sets (such as 𝑇 (𝑝+1) and 𝑇 (𝑚) in Figure 1) consisting of rules that share the same sets of defining variables.

Term grouping

The final component is a logistic regression with SGL regularization applied to the grouped term matrix T′ = (𝑇 (1) , 𝑇 (2) , … , 𝑇 (𝑚) ) and outcome vector Y as identified by equation ( 7).

Experimental Validation

Data Sets

The experimental validation of SRE-SGL involves two dimensions: predictive accuracy and interpretability. For the former, the performance of SRE-SGL is compared to a set of benchmark algorithms over fourteen real-world customer churn data sets.

Experimental Set-Up

First, to assess the predictive performance of SRE-SGL, a comparison is made to a set of seven benchmark algorithms: two closely related algorithms upon which it builds: conventional RE and SRE;

and five algorithms that are characterized by a widespread adoption by practitioners, due to both high interpretability and strong predictive performance, on the one hand, and frequent adoption as benchmark algorithms in prior churn prediction literature on the other: regularized logistic regression, a CART decision tree [START_REF] Breiman | Classification and regression trees[END_REF], random forest [START_REF] Breiman | Random forests[END_REF] a generalized additive logistic regression model (GAM) [START_REF] Hastie | Generalized Additive Models[END_REF], and, finally, a multivariate adaptive regression splines (MARS) model [START_REF] Friedman | Multivariate Adaptive Regression Splines[END_REF]. The regularized logistic regression models are implemented with elastic net regularization. The GAM takes the form of a semiparametric logistic regression: a binary outcome variable is predicted using a combination of splines (for continuous variables) and linear terms (for dummy variables).

Data pre-processing can have an important impact on the predictive performance of classifiers in customer churn prediction [START_REF] Coussement | A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry[END_REF]. In this study, however, the impact of different data pre-processing techniques on the predictive performance of the classifiers is not one of the research objectives.

Therefore, all preprocessing steps related to handling missing values, categorical variables, outliers, class imbalance and variable selection are equal for all algorithms and chosen in line with previous benchmark studies in CCP [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF][START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF], which help to keep the study and presentation of results lean. First, missing values are imputed with the median and dummy variables are created to flag instances that are imputed for a certain variable [START_REF] Coussement | A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry[END_REF]. Next, categorical variables are dummy-encoded into a number (the number of categories minus one) of binary variables that indicate the presence or absence of a particular characteristic. Since high cardinality does not pose an issue in our datasets, we did not rely on strategies to reduce the number of categories to a manageable size, such as coarse classification using hierarchical agglomerative clustering with Euclidian distance [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF][START_REF] Tan | Introduction to Data Mining[END_REF]. Then, outliers, defined as unusual values that are more than three standard deviations from the variable's mean, are transformed using winsorization.

Class imbalance, a result of the number of churners being much lower than the number of non-churners, is handled by undersampling the majority class, i.e. non-churning customers, to the same as the churners [START_REF] De Caigny | Incorporating textual information in customer churn prediction models based on a convolutional neural network[END_REF]. Finally, Fisher score selection is applied as an input selection procedure to reduce the dimensionality of the initial feature space to twenty [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF]. This is justified because a classifier often yields equal, or even better, predictive performance on a small set of highly predictive variables than on an exhaustive set of mainly redundant variables.

A fair comparison of classifier performance requires a strategy to tune hyperparameters whilst reducing the variability in results due to sampling. To these ends, a 5x3 cross-validation experimental design, nowadays common in CCP literature [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF][START_REF] De Caigny | Incorporating textual information in customer churn prediction models based on a convolutional neural network[END_REF][START_REF] Burez | Handling class imbalance in customer churn prediction[END_REF], is deployed. This procedure involves a stratified Hyperparameter settings of the algorithms are optimized from broad ranges of values, similar as in previous CCP studies [START_REF] De Caigny | A new hybrid classification algorithm for customer churn prediction based on logistic regression and decision trees[END_REF][START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF][START_REF] Coussement | A comparative analysis of data preparation algorithms for customer churn prediction: A case study in the telecommunication industry[END_REF]. Appendix B provides an overview of the optimized hyperparameters and their candidate values for all considered algorithms. Following the approach in [START_REF] Bock | The best of two worlds: Balancing model strength and comprehensibility in business failure prediction using spline-rule ensembles[END_REF], SRE and SRE-SGL deploy penalized cubic regression splines to estimate spline terms 2 . The GAM benchmark model is configured with penalized cubic regression splines with shrinkage. Note that cubic regression splines in SRE, SRE-SGL and GAM depend on smoothness parameter ρ which is internally optimized using the generalized cross-validation (GCV) criterion [START_REF] Craven | Smoothing noisy data with spline functions[END_REF][START_REF] Wood | Stable and efficient multiple smoothing parameter estimation for generalized additive models[END_REF] during spline estimation.

To assess the predictive performance of SRE-SGL relative to the benchmark algorithms, a statistical framework based on the non-parametric Friedman test is used as described by Demšar [START_REF] Demšar | Statistical Comparisons of Classifiers over Multiple Data Sets[END_REF]. As 6 algorithms and 14 data sets are considered in the focal experiment, the Friedman statistic is defined as:

χ F 2 = 12 * 14 6(6+1] [∑ AR a 2 a - 6(6+1) 2 4 ] (9)
where AR a denotes the average rank of the performance measures of an algorithm a= 1,2,…,6 over our 14 data sets. The Friedman test is assumed to be distributed according to χ 𝐹 2 with k-1 degrees of freedom under the null hypothesis that states that the results of all algorithms do not differ and thus the ranks AR a 2 should be equal. Only if the null-hypothesis is rejected, SRE-SGL is pairwise compared with the benchmark algorithms using the Holm post-hoc test [START_REF] García | Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power[END_REF]. Predictive performance is measured in terms of area under the receiver operating characteristic curve (AUC) and top decile lift (TDL), both commonly reported in churn prediction literature.

The second part of the experimental validation of SRE-SGL involves the dimension of interpretability, assessed through an in-depth case study on one specific data set (ds9 in Table 1). Note that for reasons of consistency, the data set is preprocessed and variables are selected as described above.

However, deviating from the 5x3-fold cross-validation deployed for comparing classifiers' predictive performance, all models and derived insights reported in this section are based on a unique fold, i.e. a single data split of the Cell2cell data set. [START_REF] Martens | Performance of classification models from a user perspective[END_REF] which expresses the sum of term importances in which variable 𝑥 𝑘 occurs, each term divided by the cardinality of x j , the set of variables on which it depends. Hence, higher values are awarded to variables appearing (i) more frequently and (ii) in more influential terms than others. Second, we deploy partial dependence functions to identify the aggregated effects that variables or variable pairs exert in RE, SRE and SRE-SGL models. Partial dependence functions identify the isolated effect of one or more variables in a predictive model 𝐹(x) by taking into account an averaged effect taken over the other variables [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction[END_REF]:

𝐹 ̂𝑠(x 𝑠 ) = 1 𝑛 ∑ 𝐹(x 𝑠 , x 𝑖\𝑠 )
𝑛 𝑖=1 [START_REF] Friedman | Predictive learning via rule ensembles[END_REF] where x 𝑠 is the set of variables of interest, n is, in this context, the number of customers in the data set while x 𝑖\𝑠 represents the values of customer i for all variables not occurring in variable set x 𝑠 . When

x 𝑠 consists of one variable, the nature of the relationship between a single variable and the log odds of customer churn is revealed. Hence, partial dependence functions constitute a popular instrument to reveal variable effects in predictive customer scoring [e.g. 62]. Our analysis reports the corresponding partial dependence plots. Moreover, when x 𝑠 consists of multiple variables, equation ( 12) provides the basis for a quantity to analyze the presence and strength of interaction effects. Specifically, the strength of the interaction effect between variables 𝑥 𝑗 and 𝑥 𝑘 [START_REF] Hastie | The Elements of Statistical Learning: Data Mining, Inference, and Prediction[END_REF] can be expressed as

𝐻 𝑗𝑘 2 = ∑ [𝐹 ̂𝑗𝑘 (𝑥 𝑖𝑗 , 𝑥 𝑖𝑘 ) -𝐹 ̂𝑗(𝑥 𝑖𝑗 ) -𝐹 ̂𝑘(𝑥 𝑖𝑘 )] 2 / ∑ 𝐹 ̂𝑗𝑘 2 (𝑥 𝑖𝑗 , 𝑥 𝑖𝑘 ) 𝑛 𝑖=1 𝑛 𝑖=1 . ( 12
)
5 Results

Predictive Performance Benchmark

This section presents the results of our experiment in which the predictive performance of SRE-SGL is compared with 7 benchmark algorithms over 14 CCP data sets originating from different industries.

Appendix D presents the average cross-validated results, in terms of AUC and TDL respectively, for these fourteen data sets. The standard deviations over the different runs are indicated between brackets and the best performing algorithm is underlined for every data set.

These results serve as input to determine the average ranks of the classifiers, required for the Friedman test. Lower average ranks indicate better performance. The best performing algorithm is indicated in bold. *** , ** , * Indicates significance on 99%, 95%, 90% level respectively. Significant differences are indicated in italic.

The adjusted p-value for Holm post-hoc test is shown between brackets. Results show that SRE-SGL achieves significantly better predictive performance than LR,DT, MARS and GAM in terms of AUC, and it significantly outperforms DT, LR, MARS and GAM in terms of TDL. Figure 2 summarizes the post-hoc test results visually by means of critical difference plots [START_REF] Demšar | Statistical Comparisons of Classifiers over Multiple Data Sets[END_REF] for AUC and TDL.

Model Interpretability: A Case Study

In this section, the SRE-SGL model is assessed in terms of its ability to deliver model interpretability. To this end, a case study focusing on customer churn prediction in a telecom setting using the Cell2cell data set (ds9 in Table 1) is presented. This public data set is well documented and has been used in previous customer churn studies [START_REF] Verbeke | New insights into churn prediction in the telecommunication sector: A profit driven data mining approach[END_REF]. The three objectives of this case study are the following: (i) to illustrate how the spline-rule ensemble model with SGL regularization results in an interpretable model, (ii) to demonstrate how SRE-SGL offers a higher degree of interpretability in comparison to rule ensemble and spline-rule ensemble models with lasso regularization and (iii), to analyze whether the introduction of structured regularization in SRE-SGL substantially changes the role of variables in the model in contrast to conventional rule and spline-rule ensembles by investigating variable importance and isolated model variable effects. Table 3 provides an overview of the 20 selected variables through applying Fisher-score selection.

SRE-SGL Model Interpretation

The most direct way of gaining insights into a classifier's functioning is an interpretation of the model itself. In contrast to alternative homogenous ensemble methods, rule ensembles and spline-rule ensembles deliver a facilitated model interpretability thanks to three elements: (i) the nature of candidate ensemble members (i.e., rules, linear terms and splines), (ii) their simple linear combination and (iii) the shrinkage resulting from the selection procedure to which they are submitted. SRE-SGL delivers shrinkage through sparse group lasso regularization and thus enables a more intelligent selection of competing terms through structured sparsity regularization. Table 4 shows the selected terms of the SRE-SGL model while Figure 3 visualizes the penalized cubic regression splines selected by the model. The following observations emerge from Table 4. First, the model contains 21 terms: 6 rules, 7 linear terms and 8 splines. Hence, the model illustrates well how SRE-SGL is capable of revealing linear effects, nonlinear effects as well as interaction effects. Second, investigating the nature of the impact of terms on customer churn is straightforward. The 4 most important terms are univariate: linear basis functions for retcalls and revenue, and splines for changem and eqpdays. Positive linear effects exist for retcalls, revenue and changem_M; negative ones for creditde, recchrge and webcap (in order of importance). Splines, visualized in Figure 3, reveal varying non-linear relationships to the probability to churn. Second, the rules reveal the existence of interaction effects. Closer inspection reveals that eqpdays interacts with two variables: recchrge and retcalls. Both are represented by three rules each, that are consistent in terms of coefficient sign and rule conditions.

SRE-SGL Comparison to Rule and Spline-Rule Ensembles

Next, we wish to compare the SRE-SGL model to a conventional RE and SRE model fit to the same data set (Tables A. 2). Second, the absence of structured regularization in conventional lasso regularization in rule and spline-rule ensembles compromises interpretability due to the presence of conflicting rules. For example, consider the interaction effect between eqpdays and retcalls. Both the rule and spline-rule ensemble model also recognize their interaction by selection rules defined on both.

However, the nature of the interaction effect is much harder to disentangle due to (i) the number of rules that capture the interaction effect (7 for RE and SRE, versus A second comparison analyses the importance of individual variables in the models. Figure 4 presents variable importance measures for all variables in the SRE-SGL, SRE and RE models, rescaled so that the most important variable receives a score of 100. These results show consistency between SRE-SGL, SRE and RE models, despite their differing model structures. In total, 14 variables appear in the SRE-SGL and SRE models, while there are 10 in the RE model. SRE-SGL, SRE and RE agree on the five most important variables (with an importance level above 20) are retcalls, eqpdays, revenue, changem and recchrge. These are the variables that dominate both as univariate terms, and in multiple multivariate rules in the model.

A third dimension on which SRE-SGL is compared to SRE and RE is the analysis of isolated variables effects. Our intention is to analyze whether the introduction of structured regularization in SRE-SGL substantially alters the nature of the effects found in regular SRE and RE models. To this end, the isolated variable effects are visualized in Figure 5 that shows the corresponding partial dependence plots for the variables in the model, ordered by importance (defined in Figure 4) for SRE-SGL, SRE and RE models. A comparison of these dependence plots leads to the following observations. First, effects in the SRE-SGL and SRE models are highly similar. This provides evidence that the altered model structure due to the introduction of structured regularization does in fact not substantially alter the isolated effects of individual variables. The enforcing of a grouping structure on model terms in SGL leads to a model that is easier to understand, yet is similar in its functioning. Second, these plots highlight the added value of adding spline functions to rules and linear basis functions for churn prediction. Non-linear effects that revealed in SRE and SRE-SGL models are either not incorporated (e.g. mou, directas), or substituted by a linear effect (e.g. changem, incalls) or a piecewise linear effect (eqpdays) in the RE model. Note that such a piecewise linear function requires multiple rules to be simultaneously present in the model (as demonstrated by the many rules defined on eqpdays in the RE model in Table A.1). Third, it is useful to compare these partial dependence visualizations with the SRE-SGL model described earlier. The selected linear and spline terms can be easily recognized, the partial dependence function for the variable eqpdays is a composite of the spline term and the rules that feature the variable while the piecewise linear plots for retcalls and recchrge summarize the linear effects and rules in which they emerge.

An alternative use of partial dependence functions is the analysis of variable interactions. An analysis of the interaction effects in the SRE-SGL model is available in Appendix E.

Conclusions, limitations and directions for future research

Customer churn prediction is an important instrument in companies' retention management strategies. Such models ought to be as accurate as possible albeit not at the expense of decreased interpretability. Ensemble methods have been gaining critical acclaim since many years, mostly due to their association to strong predictive accuracy. However, in contexts where decision makers attribute high value to interpretability, their black-box nature compromises their potential deployment. RE and SRE constitute a family of classifiers tailored to reconcile these seemingly conflicting objectives. Based on rules extracted from decision trees, spline terms and simple linear terms, they offer increased flexibility over other established methods that are easy to understand, such as decision trees or logistic regression. Yet, this increased model complexity, that could be seen as the cost of this increased flexibility, is very limited thanks to regularization, i.e. the shrinkage of the full set of candidate model terms.

Since rules, splines and linear terms are essentially based on the same set of variables, interpretation of a rule-or a spline-rule ensemble model could become less straightforward when regularization shrinkage does not prevent such terms from being selected simultaneously. To remedy this, we propose SRE-SGL, spline-rule ensembles with structure regularization through sparse group lasso regularization.

We define a straightforward indexing function to group terms when they share the same set of defining variables. Through sparse group lasso regularization, term selection is driven by shrinkage at two levels:

the between-group level and the within-group level. The contributions of this study are thus the following: (i) we introduce spline-rule ensembles to the field of customer churn prediction and demonstrate their ability to deliver insightful yet accurate models;

(ii) we propose SRE-SGL as an extension to spline-rule ensembles that retains the qualities of splinerule ensembles yet avoids the pitfall of reduced model interpretability due to conflicting model term selection, (iii) an extensive benchmark study is conducted to determine how SRE-SGL performs in comparison to well-established competing algorithms balancing accuracy and interpretability, and (iv) a case study is conducted to illustrate how SRE-SGL avoids the issues described above and achieves a higher degree of interpretability in comparison to conventional rule and spline-rule ensembles.

Note that a number of limitations of this study could be identified. First, our experimental comparison evaluated two well-known metrics for assessing classifier performance in the domain of

Figure 1 :

 1 Figure 1: Visual representation of SRE-SGL model training stages

  split of the data set in three equal parts. In each fold, one of these data parts serves as test data sample used to determine a classifier's predictive accuracy. The other two parts serve as training and validation data samples for training and evaluating a number of alternative classifier configurations by varying their hyperparameters. The configuration that corresponds to the best performance on the validation sample is chosen to train a final model on a stacked data sample obtained by combining the training and validation samples. Three estimates of predictive performance are thus obtained per fold, and this procedure is repeated five times. All classifiers are thus trained, validated and tested on exactly the same data samples.

Figure 2 :

 2 Figure 2: Critical difference plots for AUC (subplot (a)) and TDL (subplot (b))

Figure 3 :

 3 Figure 3: Visual representation of the eight penalized cubic regression spline terms in the Cell2cell SRE-SGL model

Figure 4 :

 4 Figure 4: Cell2cell SRE-SGL, SRE and RE model variable importances

Figure 5 :

 5 Figure 5: Partial dependence plots for selected variables: a comparison of the SRE-SGL, SRE and RE Cell2cell models

Table 1 : Data set characteristics: data set identifier, industry, number of observations, number of attributes, customer churn percentage and source

 1 Table 1 presents the most important characteristics of these data sets such as the industry, number of observations, number of attributes and churn incidence. Most of the data sets are proprietary and were obtained through exclusive company collaborations which limits the level of detail that can be disclosed. Therefore, the dimension of interpretability is assessed by means of a case study on publicly available data set ds9 in Section 5.2. Center for Customer Relationship Management Duke University, February 2014. URL: http://www.fuqua.duke.edu/centers/ccrm

	Data	Industry	# observations	# attributes	% churn	Source
	set
	Ds1	Financial Services	631,627	>100	2.53%	European Financial Services provider
	Ds2	Financial Services	602,575	>100	3.16%	European Financial Services provider
	Ds3	Financial Services	573,895	>100	2.57%	European Financial Services provider
	Ds4	Newspaper	427,833	>100	11.14%	European newspaper company
	Ds5	Financial Services	398,087	>100	4.50%	European Financial Services provider
	Ds6	Financial Services	316,578	>100	6.45%	European Financial Services provider
	Ds7	Financial services	117,808	>100	3.55%	European Financial Services provider
	Ds8	Financial Services	102,279	>100	5.99%	European Financial Services provider
	Ds9	Telecom	71,047	87	29.00%	Duke 1
	Ds10	Telecom	50,000	>100	7.34%	European telecom operator
	Ds11	Telecom	47,761	43	3.69%	European telecom operator
	Ds12	Retail	32,371	47	25.15%	European supermarket retailer
	Ds13	Energy	20,000	33	10.00%	European energy company
	Ds14	Retail	3,827	16	28.14%	European DIY retailer
	1

  Interpretability of RE, SRE and SRE-SGL models is in first instance assessed by understanding selected model terms, as well as the metrics that allow an identification of the relative importance and nature of relationships of the terms selected in a rule ensemble model presented in Section 3.1:

	𝑉𝐼(𝑥 𝑘 ) = ∑ 𝑥 𝑘 ∈x 𝑗	𝑇𝐼(𝑡 𝑗 (x 𝑗 )) |x 𝑗 |

coefficient estimates, term importance measures and rule support values for rule terms. Since model terms and regularization procedures vary between RE, SRE and SRE-SGL, substantial differences amongst resulting models can be reasonably expected. Therefore, our comparison of SRE-SGL to RE and SRE models involves two additional analyses that enable a comparison of aggregated variable importance and effects. Our aim is to verify to what extent the introduction of a more stringent SGLregularization in SRE-SGL alters model effects in comparison to RE and SRE. First, since variables might emerge in multiple terms in (spline-) rule ensemble models and regularization does not prevent their simultaneous selection, the models' term importance measures are insufficient First, variable importance measures VI(𝑥 𝑘 ) allow to assess the relative importance of individual variables 𝑥 𝑘 in a model. This is usually of great importance to decision makers. They are obtained through:

Table 2 .

 2 Table2displays the average ranks for the different algorithms over the 14 data sets for The adjusted p-values, based on the Holm post-hoc test, are given between brackets in SRE-SGL has the lowest rank (i.e. best predictive performance) evaluated with TDL and second-best performance based on AUC. The results indicate that SRE-SGL performs always at least as well as the best performing algorithm in our benchmark. Compared with conventional RE and SRE, the results indicate that SRE-SGL performs significantly better than RE for both performance measures. There are no significant differences between SRE-SGL and SRE, indicating that reduction in model complexity of the compact SRE-SGL model over the extensive SRE model does not negatively impact the predictive performance. In comparison to traditional benchmark algorithms in CCP (i.e. DT, LR, RF, MARS and GAM), the SRE-SGL model demonstrates superior predictive performance in our benchmark study.

	both AUC and TDL. A lower average rank indicates better performance. The Friedman statistic is
	approximately chi-squared distributed with 7 degrees of freedom and equals 55.22 (p-value = 0.00)

based on AUC ranks and 58.84 (p-value= 0.00) based on TDL ranks, which indicates that there are significant differences in terms of ranks. In the post-hoc analysis, SRE-SGL serves as our control algorithm. *** (0.000) 7.857 *** (0.000) Regularized logistic regression (LR) 5.500 *** (0.002) 4.500 ** (0.013) Random forest (RF) 3.714 (0.179) 3.607 (0.118) Rule ensembles (RE) 4.321 * (0.056) 4.643 ** (0.011) Spline-rule ensemble (SRE) 2.036 (0.908) 2.500 (0.489) Generalized additive model (GAM) 5.500 *** (0.002) 5.697 *** (0.000) Multivariate additive regression splines (MARS) 5.429 *** (0.002) 5.357 *** (0.001)

Table 2 : Average classifier ranks across data sets for different performance measures

 2

Table 3 : Overview of selected variables in Cell2cell data set (ds9). Descriptive statistics (mean and standard deviation) are provided for preprocessed training data.

 3

	Variable	Definition	Mean	SD
	label
	callwait	Mean number of waiting calls	-0.0210	0.9759
	changem	% change in minutes of use	-0.0168	0.9992
	changem_M	Dummy that indicates whether changem (%	0.0071	0.0842
		change in minutes of use) is imputed
	creditde	Low credit rating -de	0.1190	0.3238
	custcare	Mean number of customer care calls	-0.0304	0.9684
	directas	Mean number of director-assisted calls	-0.0089	0.9888
	eqpdays	Number of days of the current equipment	0.0461	1.0045
	incalls	Mean number of inbound voice calls	-0.0264	0.9842
	models	Number of models issued	-0.0187	0.9849
	mou	Mean monthly minutes of use	-0.0306	0.9838
	mou_M	Dummy that indicates whether mou is imputed	0.0029	0.0539
	opeakvce	Mean number of in and out off-peak voice call	-0.0244	0.9859
	outcalls	Mean number of outbound voice calls	-0.0261	0.9844
	phones	Number of handsets issued	-0.0198	0.9869
	recchrge	Mean total recurring charge	-0.0335	0.9914
	retcalls	Number of calls previously made to the retention	0.0252	1.0596
		team
	revenue	Mean monthly revenue	-0.0151	0.9882
	revenue_M	Dummy that indicates whether revenue is	0.0029	0.0539
		imputed
	setprcm	Missing data on handset price	0.5742	0.4945
	webcap	Handset is web-capable	0.8908	0.3119

Table 4 : The Cell2cell SRE-SGL model: terms, term types, rule conditions, coefficients, rule support and term importance. Terms are sorted on the basis of their importance.

 4

	Term	Type	Term or rule specification	Coefficient	Rule	Term importance
	index				support
	1 Linear term	retcalls	0.2828	-	100
	2 Linear term	revenue	0.1982	-	65.3561
	3	Spline	s(changem)	0.5164	-	40.7220
	4	Spline	s(eqpdays)	0.2917	-	32.0586
	5	Rule	eqpdays ≥ -0.2954	0.1295	0.5529	21.4745
			recchrge < 1.3767
	6	Rule	retcalls < 2.3026	-0.1242	0.3901	20.2173
			eqpdays < -0.2996
	7 Linear term	creditde	-0.1805	-	19.5004
	8 Linear term	recchrge	-0.0581	-	19.2098
	9	Rule	retcalls < 2.3026	-0.1033	0.3918	16.8194
			eqpdays < -0.2954
	10	Spline	s(mou)	0.2993	-	13.7562
	11	Rule	eqpdays ≥ -0.2954	0.0768	0.5640	12.7105
			recchrge < 1.6328
	12	Rule	eqpdays ≥ -0.2954	0.0565	0.5483	9.3771
			recchrge < 1.1742
	13 Linear term	setprcm	-0.0445	-	7.3422
	14	Spline	s(incalls)	0.1465	-	4.5656
	15 Linear term	changem_M	0.1578	-	4.4317
	16	Spline	s(recchrge)	0.0588	-	1.2510
	17	Spline	s(directas)	0.0634	-	0.9684
	18	Spline	s(callwait)	0.0470	-	0.3502
	19 Linear term	webcap	-0.0029	-	0.3051
	20	Spline	s(custcare)	0.0219	-	0.2444
	21	Rule	retcalls < 2.3026	-0.0005	0.3947	0.0868
			eqpdays < -0.2913

  Table 4 also provides coefficient estimates, term importance measures and rule support values for rule terms. Values of term importance measures 𝑇𝐼(𝑡)are rescaled so that the most important term receives a value of 100.

	s(changem)	s(eqpdays)	s(mou)
	changem	eqpdays	mou
	s(incalls)	s(recchrge)	s(directas)
	incalls	recchrge	directas
	s(callwait)	s(custcare)
	callwait	custcare

  1 and A.2 in Appendix A, respectively). This comparison illustrates the beneficial impact of SGL regularization with regards to interpretability on multiple accounts. This comparison involves three dimensions: (i) model structure and interpretation, (ii) variable importance and (iii) isolated variable effects.In terms of model structure and interpretation, substantial differences emerge that favor SRE-SGL in terms of interpretability. First, in terms of model size, the SGL led to a more compact model for SRE-SGL in comparison to rule and spline-rule ensemble models. Both benchmark models contain more terms: 31 and 38, respectively, versus 21 for the SRE-SGL model. Despite this large difference in model size, the predictive performance of SRE-SGL is better than RE and similar to SRE (as shown in Table

  model, and for the interaction between recchrge and retcalls. While in the SRE-SGL model, the sparse group lasso did not prevent a selection of multiple rules defined on both pairs of variables, within-group level shrinkage resulted in a more stringent selection of consistent rules. Third, the SRE model shows how in the absence of structured regularization, univariate terms can also cause conflicts and jeopardize model interpretability. For example: changem and recchrge occur both as a linear term and as a spline in the model. Eqpdays occurs as a spline, and in 8 univarate rules with varying coefficient signs. These problems are lifted in the SRE-SGL model. The exception is the double occurrence of recchrge, which shows that the introduction of within-group shrinkage in SGL is not absolute.

	importance
	3 in SRE-SGL), and (ii) the presence of seemingly conflicting rules. For example, consider rules that depend on the variables retcalls and eqpdays. terms 7 and 25 in the rule ensemble model: (retcalls < 2.3026 * eqpdays < -0.2996; coefficient -0.0681) versus (retcalls ≥ 2.3026 * eqpdays < -0.2913; coefficient 0.0414). Such problems also emerge in the SRE Variable retcalls eqpdays revenue changem recchrge credited mou setprom incalls changem_M directas callwait webcap custcare

  SRE-SGL aims for accurate models that consist of as few terms of as few variable groups as possible. Extensive experiments on a large set of churn prediction data sets confirm the highly competitive nature of SRE-SGL in terms of two dimensions it aims to reconcile: predictive accuracy and interpretability. Specifically, results demonstrate how SRE-SGL outperforms a decision tree, logistic regression, GAM, MARS and a random forest model on most datasets. It also consistently outperforms conventional RE models, and the introduction of structured regularization does not result in a disadvantage over standard SRE. Model interpretability of SRE-SGL was assessed in detail and compared to RE and SRE by means of a case study, investigating churn prediction in the setting of a telecommunications company. An analysis by means of variable importance measures and partial dependence functions demonstrates that the effects captured by SRE-SGL are highly similar to those found in the RE and SRE models. However, the SRE-SGL model is simpler in nature and its interpretability, unlike conventional RE and SRE models, is not compromised by inconsistent or conflicting model terms or term effects based on the same variables.

Table A .1: The Cell2cell rule ensemble model: terms, term types, rule conditions, rule support and term importance. Terms are sorted by their importance.

 A

	Term	Type	Term or rule specification	Coefficient	Rule		Term
					support	importance
	1	Linear term	retcalls	0.2107	-			100
	2	Linear term	revenue	0.1523	-			67.4154
	3	Spline	s(changem)	0.3692	-			39.0854
	4	Linear term	recchrge	-0.0505	-			22.4276
	5	Linear term	creditde	-0.1467	-			21.2720
	6	Linear term	changem	-0.0438	-			19.6160
	7	Spline	s(eqpdays)	0.1242	-			18.3232
	8	Rule	recchrge < 1.3767	0.0785	0.5529		17.4713
			eqpdays ≥ -0.29547
	9	Linear term	setprcm	-0.0769	-			17.0326
	10	Rule	recchrge < 1.6328	0.0705	0.5640		15.6602
			eqpdays ≥ -0.2954
	11	Rule	recchrge < 1.1742	0.0693	0.5483		15.4544
			eqpdays ≥ -0.2954
	12	Rule	retcalls < 2.3026	-0.0697	0.3901		15.2346
			eqpdays < -0.2996
	13	Rule	retcalls < 2.3026	-0.0665	0.3918		14.5401
			eqpdays < -0.2954
	14	Linear term	webcap	-0.0987	-			13.7891
	15	Spline	s(mou)	0.1646	-			10.1537
	16 12 17 13 18 19 14 20 15 21 16 22 17 18 23 19 24 20 25 21 26	Rule Rule Linear term Rule Rule Rule Rule Rule Rule Rule Spline Rule Rule Rule Linear term Linear term Rule Rule Linear term Rule Rule Rule	retcalls < 2.3026 retcalls < 2.3026 eqpdays < -0.2913 eqpdays < -0.2913 webcap eqpdays ≥ -0.2996 retcalls < 2.3026 eqpdays < -0.2996 eqpdays ≥ -0.2954 eqpdays < -0.3615 eqpdays ≥ -0.2996 eqpdays < -0.2954 eqpdays < -0.2996 s(incalls) eqpdays ≥ -0.2954 retcalls < 2.3026 eqpdays < -0.2954 eqpdays < -0.3614 incalls changem_M eqpdays < -0.3161 eqpdays < -0.3160 setprcm eqpdays ≥ -0.3161 eqpdays ≥ -0.3160 eqpdays < -0.3615 retcalls ≥ 2.3026	-0.0552 -0.0462 -0.0765 0.0369 -0.0386 -0.0366 0.0343 0.0351 -0.0342 -0.0349 0.1299 0.0339 -0.0226 -0.0339 -0.0135 0.1264 -0.0238 -0.0166 -0.0234 0.0160 0.0233 0.0562	0.3947 0.3947 -0.5920 0.3654 0.4080 0.5903 0.5920 0.4097 0.4080 -0.5903 0.3654 0.4097 --0.4017 0.4017 -0.5983 0.5983 0.0173	10.1205 15.2900 8.1317 13.5115 8.0486 10.5356 7.5649 7.5306 9.7650 5.4361 9.7185 4.8747 9.4440 9.4374 4.7648 7.5088 3.6533 6.6227 3.5121 6.5475 6.4581 3.2754
	22 27	Rule Rule	eqpdays ≥ -0.2913 recchrge ≥ 1.3767	0.0215 -0.0369	0.5873 0.0374	5.9955 3.1351
	23	Rule	eqpdays < -0.2913 eqpdays ≥ -0.2954	-0.0214		0.4127	5.9731
	24 28	Linear term Rule	changem_M eqpdays < -0.2913	0.0770 -0.0142	0.4127	-	3.6709 3.1247
	25 29	Rule Rule	retcalls ≥ 2.3026 eqpdays ≥ -0.2913	0.0414 0.0136	0.0180 0.5873	3.1160 2.9880
	30	Rule	eqpdays < -0.2913 retcalls ≥ 2.3026	0.0469	0.0180		2.7890
	26	Rule	retcalls ≥ 2.3026 eqpdays < -0.2913	0.0415		0.0173	3.0616
	31	Rule	eqpdays < -0.3615 recchrge ≥ 1.6328	-0.0348	0.0264		2.4933
	27	Rule	retcalls ≥ 2.3026 eqpdays ≥ -0.2954	0.0288		0.0179	2.1605
	32	Linear term	eqpdays < -0.296 mou	-0.0054	-			2.3950
	28 33	Rule Spline	retcalls ≥ 2.3026 s(recchrge)	0.0280 0.0520	-	0.0179	2.1015 1.4844
	29 34	Rule Rule	eqpdays < -0.2954 eqpdays < -0.3615 eqpdays < -0.2996 retcalls ≥ 2.3026	-0.0049 0.0248	0.3827 0.0179	1.3619 1.4729
	30 35	Rule Rule	eqpdays ≥ -0.3615 retcalls ≥ 2.3026	0.0049 0.0231	0.6173 0.0179	1.3523 1.3708
	31	Rule	recchrge ≥ 1.3767 eqpdays < -0.2954	-0.0113		0.0374	1.2163
	36	Spline	eqpdays ≥ -0.2954 s(directas)	0.0605	-			1.2405
	37	Spline	s(custcare)	0.0354	-			0.5295
	38	Spline	s(callwait)	0.0373	-			0.3732

Table A .2: The Cell2cell spline-rule ensemble model: terms, term types, rule conditions, coefficients, rule support and term importance. Terms are sorted by their importance.

 A

	Appendices B -E

Appendices B-E are available as online supplementary materials available for download at https://github.com/koendebock/SRE-SGL/blob/master/SRE-SGL_supplementary_materials.pdf.

Experiments with alternative spline estimation methods in SRE and SRE-SGL did not reveal significant differences with respect to predictive performance. In particular, we compared penalized cubic regression splines to penalized cubic regression splines with shrinkage, penalized thin plate regression splines, P-splines and penalized thin plate regression splines with shrinkage. Detailed results of these experiments and statistical tests are available in Appendix C.

CCP with a widespread adoption and recognition in practice: top-decile lift and AUC. Recently, promising metrics that integrate cost and profit considerations in their evaluation gained popularity in the domain, such as the expected maximum profit criterion [START_REF] Verbraken | A novel profit maximizing metric for measuring classification performance of customer churn prediction models[END_REF]. Future research should evaluate SRE-SGL on the basis of such metrics, and we intend to extend SRE-SGL so that these metrics guide decision tree training as well as regularization. Second, this study is exclusively focused on customer churn prediction which is one of the most established applications in the broader field of marketing analytics.

Future research should explore the viability of SRE-SGL for other tasks. Third, regularized regression is one strategy for reducing a large set of terms and combining them. Other strategies that are common for the practice of ensemble selection of ensemble pruning such as optimization, could also be deployed for term selection as well as integrating variable group selectivity. Fourth, SRE-SGL, like RE and SRE, involves the initial creation of a large set of rules and terms and sometimes overly complex rules can still be retained in the final model. Follow-up research should explore strategies for the definition of preliminary term filters that enforce a priori selection. This could be a viable option to further constrain models in pursuit of enhanced model interpretability. Finally, data preprocessing practices such as feature selection, class imbalance and outlier treatment are known to impact results. While this study based its experimental set-up on prior literature, future research could revisit the impact of such practices on the novel algorithms presented in this study.

Appendices Appendix A: Cell2cell Rule Ensemble and Spline-Rule Ensemble Models