Ha-Thu Nguyen
email: hathunguyen12@gmail.com

Reject inference in application scorecards: evidence from France

Keywords: Reject inference, sample selection, selection bias, logistic regression, reweighting, parceling, fuzzy augmentation, Heckman's two-stage correction JEL Classification: C51, C52, C53, G21

Credit scoring models are commonly developed using only accepted Known Good/Bad (G/B) applications, called KGB model, because we only know the performance of those accepted in the past. Obviously, the KGB model is not indicative of the entire through-the-door population, and reject inference precisely attempts to address the bias by assigning an inferred G/B status to rejected applications. In this paper, we discuss the pros and cons of various reject inference techniques, and pitfalls to avoid when using them. We consider a real dataset of a major French consumer finance bank to assess the effectiveness of the practice of using reject inference. To do that, we rely on the logistic regression framework to model probabilities to become good/bad, and then validate the model performance with and without sample selection bias correction. Our main results can be summarized as follows. First, we show that the best reject inference technique is not necessarily the most complicated one: reweighting and parceling provide more accurate and relevant results than fuzzy augmentation and Heckman's two-stage correction. Second, disregarding rejected applications significantly impacts the forecast accuracy of the scorecard. Third, as the sum of standard errors dramatically reduces when the sample size increases, reject inference turns out to produce an improved representation of the population. Finally, reject inference appears to be an effective way to reduce overfitting in model selection.

INTRODUCTION

Nowadays, credit scoring is of paramount importance to evaluate credit risk when banks decide whether or not to approve a credit. Such models are traditionally developed based on the repayment performance of previous applicants in the portfolio, assuming that the sample used for the development represents the overall population [START_REF] Greene | Sample selection in credit-scoring models[END_REF]. In case of application scorecard, this assumption is not fully satisfied, because we only get information on the repayment behavior of those who have been accepted and finally booked for credit. The behavior of those who have been declined is unknown. Thus, by selecting only approved applicants and ignoring rejected ones, the modeling sample is intrinsically biased. If the model is to be used only to predict future bad rates of the approved sample (behavior scorecard), there will be no selection bias issue. However, since the model is typically designed to assess the whole through-the-door population to make new decisions on whether accept or reject an application (application scorecard), the bias turns to be a serious issue (see [START_REF] Chandler | Using credit scoring to improve the quality of consumer receivables: legal and statistical implications[END_REF][START_REF] Chandler | Using credit scoring to improve the quality of consumer receivables: legal and statistical implications[END_REF][START_REF] Avery | Credit scoring models with discriminant analysis and truncated samples[END_REF] for early discussions on this topic). [START_REF] Eisenbeis | Pitfalls in the application of discriminant analysis in business, finance and economics[END_REF], [START_REF] Reichert | An examination of the conceptual issues involved in developing credit scoring models[END_REF], [START_REF] Joanes | Reject inference applied to logistic regression for credit scoring[END_REF], [START_REF] Hand | Reject inference in credit operations[END_REF], [START_REF] Feelders | Credit scoring and reject inference with mixture models[END_REF] among others, also report that using a model based solely on accepted applicants often generate misleading results. Reject inference techniques is then a way to address this concern.

The importance of reject inference varies, depending on the current decision process, data, and approval/rejection rates. According to [START_REF] Siddiqi | Credit risk scorecards: developing and implementing intelligent credit scoring[END_REF], reject inference does not make much significant difference in the following situations: (i) a very high confidence level with a high approval rate assumes all rejects are equal to bads; (ii) a very low confidence level assumes that decisions are randomly made or using a mistaken adjudication tool. In cases with either low or medium approval rates, reject inference helps in identifying creditworthy applications that are currently rejected.

Previous research has proposed different techniques to reduce selection bias (see [START_REF] Banasik | Does reject inference really improve the performance of application scoring models?[END_REF] for example). While the performance of these techniques has largely been discussed at a theoretical level, very few studies have been done from an empirical point of view, as most datasets on which reject inference techniques have been tested are incomplete or simulated [START_REF] Åstebro | Bound and collapse Bayesian reject inference for credit scoring[END_REF]. We fill this gap in the present paper. Indeed, a significant contribution and novelty of our empirical analysis is to use a dataset coming from one of the biggest French consumer finance banks, which contains complete information on both rejected and accepted applications, and focus on the results that could be reached if reject inference would occur. The aim of the paper is to shed some new light on the relevance of reject inference procedures in the development of application credit scoring models.

The rest of the paper is structured as follows. Section 2 discusses the issue of selection bias and different reject inference methods in credit scoring: manual estimation, bureau data-based method, reweighting, parceling, fuzzy augmentation and Heckman's correction method. The pros and cons of each technique are also presented in this section, with a special focus on the Heckman's two-stage estimation method which has been widely used to correct selection bias.

Section 3 is devoted to the empirical part of the paper. Using a real dataset of a leading French consumer finance bank, we evaluate the effectiveness of the practice of using reject inference. Finally, Section 4 concludes the paper with a discussion on the key findings of the study.

REVIEW OF REJECT INFERENCE TECHNIQUES

In the sense of [START_REF] Rubin | Inference and missing data[END_REF] and [START_REF] Little | Statistical analysis with missing Data[END_REF], missing data can be classified in one of three groups: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). In the first two cases, the missing data mechanism is ignorable. Indeed, missing at random means that the probability of default, given all the exogenous variables of the model, is the same whether an application is accepted or rejected. On the contrary, if the data is MNAR, the missing data mechanism is non-ignorable because extra information on future default is added by human judgment, and then this will change the probability of default. In this case, sample selection bias happens. Missing data must be included in the model to get proper estimates of the outcome.

In this section, we discuss different applicable methods for non-ignorable missing data. [START_REF] Montrichard | Reject inference methodologies in credit risk modeling[END_REF] presents manual estimation, known as "expert rules", as one of applicable reject inference methods. It corresponds to the case where the analysts use their own experience to manually simulate the performance of the rejects. For example, they argue that if one variable is greater than a specific limit, over 50% of that group will turn to be "bad" after 12 months. They will then select this segment from the declined dataset and randomly assign 50% of the applicants as "good" and the remaining as "bad". For the remaining rejects, they assume an overall 30% bad rate. The use of the method is not statistically justified and it is consequently likely to be too subjective to be reliable. In fact, if one chooses different rates, the results will be completely different.

Manual estimation

Similar in-house or Bureau data-based method

One of the most commonly used practices in reject inference is to obtain external performance data from credit bureaus on rejected applicants, suggested by [START_REF] Ash | Best practices in reject inferencing[END_REF], [START_REF] Siddiqi | Credit risk scorecards: developing and implementing intelligent credit scoring[END_REF], [START_REF] Barkova | Sample selection bias in acquisition credit scoring models: an evaluation of the supplemental-data approach[END_REF][START_REF] Etimova | An empirical survey of reject inference techniques[END_REF]. Using performance data coming from other creditors over an observation period allows lender to infer how the rejects would have performed if they had been approved. It is assumed that default on other products is equivalent to default on the interested product; the new model is then created with consideration of the behavior of the reject. Thus, using real historical data and giving a more realistic perspective of the market and other clients' behaviors are the core advantages of this method.

However, it also contains several downsides. First, obtaining bureau data is costly and time consuming. It requires extra effort for data preparation and extraction. Second, the quality of the data is uncertain. Third, the main assumption of this approach requires a significant bureau match rate. Finally, the approach cannot completely eliminate bias, since rejects without performance information are probably considered to be non-random.

Reweighting

Reweighting is a well-used technique that involves weighting accepted applications in such a way as to scale up to the total population [START_REF] Hsia | Credit scoring and the equal credit opportunity act[END_REF][START_REF] Banasik | Does reject inference really improve the performance of application scoring models?[END_REF][START_REF] Montrichard | Reject inference methodologies in credit risk modeling[END_REF]. The technique uses the ratio between the number of approved accounts in a cluster and the number of declines. Based upon this ratio, the approved accounts will be reweighted such that the number of the weights will equal the total number of applications. As accepted scores are monotonically related to the probability of being accepted, we can replace scores by their probabilities, and consider each individual record in spite of clusters. Each accepted record has a probability to be approved (𝑝 𝐴 ), and correspondingly a sampling weight of 𝑤 = (1/𝑝 𝐴 ). A new model using weighted accepts is then estimated.

Simplicity is a major advantage of this technique. It contains, however, several drawbacks. [START_REF] Banasik | Does reject inference really improve the performance of application scoring models?[END_REF], using their data, demonstrate that reweighting does not really improve the performance of the good-bad model. First, the scope for improving on a model parameterized only on the approvals appears small. Second, reweighting applications within an approval sample and adopting a cut-off point based on those approvals do not seem to perform better than an unweighted estimation. Third, reweighting may undermine the ability to apply good/bad knowledge of the population, without giving any compensating advantage. Reweighting in Parnitske (2005) also appears unsuitable to completely get rid of selection bias. Lacking knowledge of the tendency to become delinquent of the rejects, this method can only lead to improvements by chance. Though, combining reweighting and supplementary data improves the results.

Extrapolation

Extrapolation involves assigning a default status to the rejects, based on the same model that is fitted to the approvals only, and then re-estimating the model [START_REF] Ash | Best practices in reject inferencing[END_REF]. To simulate outcomes for the rejects, we should follow three steps.

Step 1: Construct a G/B model on the approvals (KGB population) as usual to get the scorecard A.

Step 2: Estimate probabilities to be bad that can be assigned to rejects based on outcomes given by the scorecard A. These estimated bad rates will be then used to simulate outcomes on the rejects, either by Monte Carlo parceling methods or by fuzzy augmentation.

The Monte Carlo parceling method simulates a 0/1 outcome (𝑦 = 1 if default/bad, 𝑦 = 0 if non-default/good) for each reject. Next, we generate a random number 𝑟 ⋲ [0,1] from a uniform distribution. A reject with score 𝑠 𝐴 , calculated with scorecard A, is simulated with 𝑦 = 1 if and only if 𝑟 < 𝑝(𝑦 = 1 | 𝐴, 𝑠 𝐴 ). This option is realistic. However, since there is a random factor, the outcome may vary if one changes the random seed generator. Several repeated simulations should be done to get more accurate results.

Fuzzy augmentation, in a different way, considers rejects as being both partially "Good" and partially "Bad" (0/1 outcomes with probabilities for each reject). Two records are created to each reject: (i) 𝑦 = 1 with weight 𝑝(𝑦 = 1 | 𝐴, 𝑠 𝐴 ) , and (ii) 𝑦 = 0 with weight (1 -𝑝( 𝑦 = 1 | | 𝐴, 𝑠 𝐴 )). The overall weight of each reject is equal to 1. Accepted applications are given by real outcomes with weights equal to 1. This option is more complicated. However, it is more stable and repeatable, thanks to the deterministic nature of the augmentation [START_REF] Montrichard | Reject inference methodologies in credit risk modeling[END_REF].

Step 3: Construct a new model on the approvals and the rejects with their simulated outcomes to get the scorecard B. These extrapolation methods have been discussed with different points of view. [START_REF] Banasik | Does reject inference really improve the performance of application scoring models?[END_REF] argue that extrapolation tends to remain parameter estimates unchanged, but the good/bad rate it provides appears to be insufficiently appropriate. These methods may be quite arbitrary and result incorrectly or lead to a distortion of the actual default data (Kiefer and Larsen, 2006).

However, Parnitske (2005) shows that extrapolation needs no additional risk and leads to a significant improvement to the model. [START_REF] Zeng | A rule of thumb for reject inference in credit scoring[END_REF] also state that fuzzy augmentation technique appears more accurate among others. The authors introduce a rule of thumb, in which one can specify a factor to increase the bad rate of rejected applicants. [START_REF] Heckman | Sample selection bias as a specification error[END_REF] discusses the bias that results from using nonrandom selected samples to estimate behavioral relationships as an ordinary specification error or "omitted variables" bias.

Heckman's two-step correction method

The performance of the rejected applicants in credit scoring can then be inferred from Heckman's method [START_REF] Mok | Reject inference in credit Scoring[END_REF]. The model is composed of two mechanisms, which are the selection mechanism 𝑧 𝑖 and the outcome mechanism 𝑦 𝑖 , and are modeled by the latent variables 𝑧 𝑖 * and 𝑦 𝑖 * for observation 𝑖 (𝑖 = 1, … , 𝑁) respectively:

𝑧 𝑖 = { 1 𝑖𝑓 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 𝑖𝑠 𝑎𝑐𝑐𝑒𝑝𝑡𝑒𝑑, 𝑧 𝑖 * ≥ 0 0 𝑖𝑓 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 𝑖𝑠 𝑟𝑒𝑗𝑒𝑐𝑡𝑒𝑑, 𝑧 𝑖 * < 0 𝑦 𝑖 = { 1 𝑖𝑓 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 𝑖𝑠 𝑔𝑜𝑜𝑑, 𝑦 𝑖 * ≥ 0 0 𝑖𝑓 𝑎𝑝𝑝𝑙𝑖𝑐𝑎𝑛𝑡 𝑖𝑠 𝑏𝑎𝑑, 𝑦 𝑖 * < 0
The latent variables 𝑧 𝑖 * and 𝑦 𝑖 * depend on explanatory variables 𝑤 𝑖 with parameters 𝛾 and 𝛽, and random errors 𝑢 𝑖 and 𝑒 𝑖 :

𝑧 𝑖 * = 𝑤 𝑖 𝛾 + 𝑢 𝑖 𝑦 𝑖 * = 𝑤 𝑖 𝛽 + 𝑒 𝑖
The errors 𝑢 𝑖 and 𝑒 𝑖 are assumed to be bivariate normally distributed as follows:

[ 𝑢 𝑖 𝑒 𝑖 ] ~ Ν(𝜇, ∑) 𝜇 = [ 0 0 ] ∑ = [ 1 𝜌 𝜌 1 ]
A selectivity problem arises because the performance indicator 𝑦 𝑖 is observed only when the applicant is accepted (𝑧 𝑖 = 1) and the errors 𝑢 𝑖 and 𝑒 𝑖 are correlated (𝜌 ≠ 0). In such a situation, the usual least squares estimators of 𝛽 are biased and inconsistent.

Consistent estimators are based on the conditional expectation:

Ε[𝑦 𝑖 * |𝑧 𝑖 * > 0] = 𝑤 𝑖 𝛽 + Ε[𝑒 𝑖 * |𝑧 𝑖 * ≥ 0] = 𝑤 𝑖 𝛽 + Ε[𝑒 𝑖 * |𝑢 𝑖 * ≥ -𝑤 𝑖 𝛾] = 𝑤 𝑖 𝛽 + 𝜌𝐻 𝑖
Selection bias is corrected by formulating the conditional expectation with the correlation coefficient 𝜌 and the hazard function 𝐻 𝑖 , also called the inverse of Mills ratio:

𝐻 𝑖 = 𝜙(-𝑤 𝑖 𝛾) 1 -Φ(-𝑤 𝑖 𝛾)
where ∅ and Φ denote, respectively, the density and the cumulative distribution function for a standard normal random variable.

The parameters in Heckman's correction method can be estimated in two stages. First, the parameter 𝛾 is estimated by probit analysis to obtain an estimate 𝐻 ̂𝑖 of 𝐻 𝑖 . The maximum likelihood estimation (MLE) of 𝛾 can be calculated by means of probit analysis, where the data is assumed to be complete.

Ε[𝑧 𝑖

* ] = Φ(𝑤 𝑖 𝛾)

The corresponding log-likelihood function 𝐿 is derived as follows:

𝐿 = ∑ 𝑧 𝑖 𝑙𝑛 Ρ(𝑧 𝑖 = 1) = 𝑁 𝑖 ∑ 𝑧 𝑖 ln Ε (𝑧 𝑖 ) = ∑ 𝑧 𝑖 ln Φ (𝑤 𝑖 𝛾) 𝑁 𝑖 𝑁 𝑖
The MLE of 𝛾 is obtained by taking 𝛾 that sets the derivative of 𝐿 to zero. An estimate 𝐻 ̂𝑖 of 𝐻 𝑖 is also calculated by applying the inverse of Mills ratio.

Second, the parameters 𝛽 and 𝜌 are estimated by ordinary least squares (OLS) estimation. The OLS estimates of 𝛽 and 𝜌 are obtained by minimizing the sum of squared errors (SSE) defined as follows:

𝑆𝑆𝐸 = ∑ (𝑦 𝑖 -Ε[𝑦 𝑖 * |𝑧 𝑖 * ≥ 0]) 2 𝑁 𝑖=1
In case of no sample bias, the errors 𝑢 𝑖 and 𝑒 𝑖 are not correlated. The presence of sample bias can be then re-checked by testing the null hypothesis that 𝜌 = 0.

The Heckman's two-step correction method has been widely recommended by econometricians (see [START_REF] Poirier | Partial observability in bivariate probit models[END_REF][START_REF] Meng | On the cost of partial observability in the bivariate probit model[END_REF]and Boye et al., 1989, for early discussions). [START_REF] Greene | Sample selection in credit-scoring models[END_REF] analyzes the impact of Heckman's procedure on selection bias and points out that the coefficients of default change significantly compared to the results obtained without correcting bias. Using a proprietary data set, [START_REF] Banasik | Sample selection bias in credit scoring models[END_REF] find that the Heckman-type selection procedure can get an improvement in terms of performance, but the gain is relatively small. [START_REF] Banasik | Reject inference, augmentation, and sample selection[END_REF] 2012) also agree with this point of view. The authors believe that in practice, we cannot determine a true model specification and a strong sensitivity makes the usefulness of the approach questionable. [START_REF] Hand | Handling selection bias when choosing actions in retail credit applications[END_REF], using simulated data, point out that even if the normality assumption holds, when enough customers are rejected and accepted, or when the original accept/reject decision depends largely on the unobserved variables, correcting selection bias through Heckman's method cannot really help.

How well do these methods?

Literature raises a question of whether we should use reject inference methods, or in other words, how to validate a method and assess its potential gains. Little empirical research on reject inference has been performed on datasets which are incomplete or simulated. Different studies demonstrate that it is difficult to have a reliable model based on reject inference because the assumptions are often strong and easily violated. However, when the information loss due to selection bias is significant and it cannot consequently be neglected, it is recommended to perform reject inference and test different approaches to find out which one is better to reduce the bias.

Table 1 summarizes pros and cons of each reject inference approach.

Heckman's twostep correction

Consider the likelihood of approval and the likelihood of bad performance associated with the predictive attributes.

-Test bias -Have a quite solid basis in statistical theory -Complicated -Assume that the bivariate model is linear with errors that are normally distributed and homoscedastic. If the assumptions do not hold, the estimates will not be reliable.

REJECT INFERENCE IN PRACTICE

We aim to investigate the performance of reject inference to correct selection bias, using a recent dataset provided by a major French consumer finance bank (hereafter called bank A).

Our sample comprises 198,587 credit histories from January 2013 to December 2013 provided by bank A. For 56,016 applications, the repayment status is known; and all other 142,571 applications are rejected by score (the applications rejected due to a police rule are excluded from the scope of our analysis 1 ), then we do not possess any information on their potential repayment. These applications come from bank A's retail loan portfolio for new customers. With such a high-risk portfolio, the rejection rate is consequently high (in excess of 70%); and bias due to selection is typically not ignorable. Thus, reject inference needs to be accounted for to reduce selection bias.

Each of 56,016 applications is observed over a one-year performance window. Figure 1 describes twelve outcome windows selected for scorecard development. Applications are defined as "bad" if they become delinquent for thirty days or more within one year. 2 The worst-ever definition is used. An application is considered as "bad" if the condition holds true at any point over one year, and not at the end of the period. All other applications are defined as "good". 2) for our analysis: (i) "full" dataset contains all observations, both approval and declined applications, (ii) "approve" dataset includes approvals only, and (iii) "decline" dataset includes declines only.

1 The following rules are applied when selecting rejects to be incorporated into the study:

-If the applications are rejected due to a hard policy rule, which is still in force, then they should not be put into the model. -The applications rejected due to a soft policy rule and then overridden should be included in the model.

-The applications rejected due to a hard policy rule, which is no longer in force, should also be included. 2 The definition of bad used in this study is strictly chosen by bank A for this specific portfolio.

Jan '13

Jan'14

Feb '13

Feb'14

observation dates

-Approvals -Declines

Predict outcome status

Known Good/Bad (observed) Inferred Good/Bad (unobserved)

Dec '13

Dec '14 … Our KGB scorecard is developed on the "approve" dataset. As mentioned in Nguyen (2015a), when reviewing variables for possible inclusion in the scorecard, we need to consider the following primary factors: (i) the variables have a significant degree of predictive power (through fine and coarse classing); (ii) they are stable for use; (iii) they have a low correlation to each other; and (iv) they are reasonable enough to explain the business, and also compliant (no legal or ethical restrictions on their use). Only variables that passed this prescreening step are selected for modeling. Next, we fit a logistic regression model to the dataset. The logistic regression is still the most common method used in credit scoring since it is easy to implement, understand and interpret [START_REF] Thomas | A survey of credit and behavioral scoring: forecasting financial risk of lending to consumers[END_REF][START_REF] Kocenda | Default predictors in retail credit scoring: evidence from Czech banking data[END_REF]Vojtek, 2006). Moreover, the method works best for binary outcomes, and then it is adopted to build the scorecard since our outcome is typical binary good/bad.

After several trials (see [START_REF] Nguyen | Default predictors in credit scoring: evidence from France's retail banking institution[END_REF] for criteria allowing selecting the best model among candidates), we find out a scorecard which works best on the "approve" dataset. The final scorecard consists of the following characteristics: marital status, occupation, age of the client, time at job, time at present address, household income, residential status, loan duration, and type of credit. Table 3 on the next pages presents a detailed characteristic analysis.

Table 3 shows that the variables selected in the final scorecard have a good predictive power in overall. The highest predictive ability to separate goods from bads comes from marital status, household income, residential status, loan duration and type of credit. For example, it is obvious that the risk is lower for married than single or divorced clients, given the fact that marital status has an effect on the applicant's responsibility, reliability, and financial wealth. The Kolmogorov-Smirnov (KS) statistic and the Gini statistic are generally used to measure the discrimination of a scorecard [START_REF] Nguyen | Default predictors in credit scoring: evidence from France's retail banking institution[END_REF]. The KS measures the widest spread between cumulative goods and cumulative bads. The divergence between the two curves determines the strength of the scorecard to differentiate good customers from bad ones. The higher the KS, the better the model, since goods are more separated from bads. Like the KS, the Gini coefficient is a quantitative measure of how well the model discriminates between goods and bads but by looking at actual discrimination versus perfect discrimination. A good scorecard in general has a KS greater than 35% and Gini higher than 40%. However, in practice, these thresholds are applicable for scorecards having at least some behavioral variables, e.g. behavioral scorecard or application scorecard for a known-customers portfolio.

In our case study, as we have no behavioral variables or high discriminatory variables which can give us a high Gini like other scorecards, we expect then a Gini higher than 35% which could be considered to be satisfactory.

As a result, our scorecard shows a good performance with 𝐾𝑆 𝑑𝑒𝑣 of 35% and 𝐺𝑖𝑛𝑖 𝑑𝑒𝑣 of 46%.

The results turn out to be better than expected for such an application scorecard for new customers.

To validate the scorecard, we use a test sample which contains 90,928 credit histories from January to June 2014, which will be observed until June 2015. The choice of the sample is based on the last available repayment behavior information we possess (June 2015). The repayment status is known for 35,726 applications, which are finally financed; and unknown for the remaining 55,202 applications, which are rejected by score. Each of 35,726 applications is observed over a one-year performance window. A "test" dataset which includes these 35,726 approvals and their repayment status is then created for the sake of validation.

Validating the scorecard on the "test" dataset shows a quite significant decrease in terms of performance with 𝐾𝑆 𝑡𝑒𝑠𝑡 of 30% and 𝐺𝑖𝑛𝑖 𝑡𝑒𝑠𝑡 of 39% . We use ∆𝐺𝑖𝑛𝑖 , which is the gap between 𝐺𝑖𝑛𝑖 𝑑𝑒𝑣 and 𝐺𝑖𝑛𝑖 𝑡𝑒𝑠𝑡 , as a representative indicator to measure overfitting. In fact, overfitting occurs when the model is typically trained by maximizing its performance on training data, while its efficiency is defined by its ability to perform well on an unobserved data and not by its performance on training data. In this paper, we aim to test different reject inference techniques to adjust for sample selection bias and find out the one maximizing performance while minimizing overfitting effects.

The following reject inference techniques are incorporated into the study: reweighting, parceling, fuzzy augmentation and Heckman's two-step correction method. The bureau databased method is not discussed in our empirical analysis since there is no credit bureau in France. 5 The results are summarized in Table 4. The overall bad rate in Table 4 is defined as the ratio between the number of bad accounts (both known and inferred bad accounts, but only known bad accounts in case of KGB model) and the total number of good and bad accounts. Since (i) reject inference only provides an estimation of performance if an application had been accepted, and (ii) rejects are more probable to become bad than good, the bad rate with correcting bias should be greater than the one without correcting bias.

In practice, there is a convention that the ratio of the known GB odds to the inferred GB odds should be between two and four times [START_REF] Siddiqi | Credit risk scorecards: developing and implementing intelligent credit scoring[END_REF]. If the ratio is lower, we may infer that the rejects are too similar to the approvals, which is unlikely to be true. If it is higher, then we may be too harsh on the rejects, and may infer too many bad accounts. In this case, the rejects may have too much influence on the final model which is dangerous as their performance has been inferred rather than based on actual history.

Table 4 shows that fuzzy augmentation has a known-to-inferred GB odds ratio of 19.6, given its associated new bad rate is 43%, which is too high. The Heckman's correction method has a non-significant ratio because its associated new bad rate is only 1%. In our case study, the Heckman's method does not produce better results, as expected. Only reweighting and parceling give better results in terms of performance (38% and 39% respectively), while reducing overfitting effect (from 7 points to 6 points and 1 point respectively). The results of KGB model, reweighting and parceling are illustrated in Appendixes A, B and C.

Table 5 displays the estimates of the parameters of the model with and without correcting selection bias, using reweighting and parceling techniques. The results show that reject inference reduces dramatically the sum of standard errors (0.42 vs. 0.99, and 0.58 vs. 0.99).

14 Moreover, ignoring rejects may impact the forecast accuracy of the scorecard. In fact, the full model (with reweighting and parceling) yields parameter estimates, which are different, both statistically (when the sign of the estimate is reversed, e.g. time_at_job2) and economically (when the estimates change significantly, e.g. occupation2, age1, time_at_job4, household_income4, loan_duration4), from the case when rejects are disregarded (KGB model).

CONCLUSION

Reject inference is of great importance to correct sample selection bias which may affect credit scoring. Literature has mostly focused on presenting different reject inference techniques, and only few efforts have been made to quantify their potential gains. In this paper, we benefit from a real dataset from a known French consumer finance bank to assess the effectiveness of the reject inference techniques. In fact, when applied to a dataset of almost 200,000 new customers requesting credit with a high reject rate in excess of 70 %, our results turn out to be particularly interesting since they have not yet been found by previous researchers. First, we prove that the most suitable reject inference technique is not necessarily the most complicated one. In our case study, reweighting and parceling provide more accurate and relevant results than fuzzy augmentation and Heckman's two-step correction method. Second, we find that ignoring the rejected applicants may impact the forecast accuracy of the scorecard: the full model yields parameter estimates, which are different, both statistically and economically. Third, we demonstrate that reject inference turns out to produce an improved representation of the population since it reduces dramatically the sum of standard errors. Finally, and most importantly, reject inference appears to be an effective way to reduce overfitting in model selection.

On the whole, our results highlight that reject inference can reduce sample bias, and its effect is not modest as it is stated in the literature. Especially in case of a portfolio with a high rejection rate, we cannot ignore rejected applications. Since the techniques are complicated to be implemented, reject inference has been performed with care and caution to find out the most pertinent model developed on the entire through-the-door population.

Figure 1 :

 1 Figure 1: Observations and outcome windows

  also indicate that using a bivariate probit model to address selection bias can improve model accuracy. A recent study byBucker et al. (2013) shows that the model using Heckman's method yields parameter estimates significantly different, both statistically and economically, from the case where rejects are disregarded.Other researchers demonstrate that Heckman's bivariate two-stage model does not work well for reject inference.[START_REF] Puhani | The Heckman correction for sample selection and its critique[END_REF]'s findings show that the estimators of Heckman's model are inefficient. Since the bivariate model is assumed to be linear with errors which are normally distributed and homoscedastic, the estimations are then not reliable when the assumptions are broken.Åstebro and Chen (

Table 1 :

 1 Comparison of different reject inference methods

		Description	Advantages	Disadvantages
	Manual estimation Build a model on the total population:	-Easy and quick	-Require solid experience to manually
		-Use known performance for the	-May reduce selection bias	simulate the performance of the rejects
		accepts		-Too subjective to be reliable
		-Use experiences to manually
		simulate the performance of the
		rejects
	Bureau data-based	Build a model on the total population:	-Use real historical data	-Costly and time-consuming
	method	-On known performance for accepts	-Give a more realistic perspective of the	-Require extra effort for data preparation
		-On reference performance for rejects	market and other clients' behaviors	and extraction
			-May reduce selection bias	-Uncertain data quality
				-Require a significant bureau match rate
	Reweighting	-Create an "approve/decline" model	-Simplicity	-Require sufficient historical data, then
		on the entire dataset. Each record	-Relatively quick	may not work well on small samples with
		will have a probability to be	-May reduce selection bias	high reject rates
		approved associated with it (p).		-The assumption 𝑃(𝑏𝑎𝑑 | 𝑋, 𝑟𝑒𝑗𝑒𝑐𝑡𝑒𝑑) =
		-Apply a weight (1/p) to scale up to		𝑃(𝑏𝑎𝑑 | 𝑋, 𝑎𝑐𝑐𝑒𝑝𝑡𝑒𝑑) is strong and
		the total population		generally unrealistic.
	Extrapolation	Assign a default status to the rejects,	-Finely adjust probabilities to common	-Time-consuming
		based on the model built on the	expectations.	-Complicated
		accepted applicants only, and then re-	-May lead to a significant improvement to	-Simulations may lead to incorrect results
		estimate the model.	the model

Table 2 :

 2 Full/Approve/Decline datasets

	Dataset	#	%
	"full"	198,587	100%
	"approve"	56,016	28%
	"decline	142,571	72%

Table 3 :

 3 Characteristic analysisIf the weight is negative, this means there are more bads than goods. If it is positive, this means there are more goods than bads. If it is close to 0, there is a similar number of bads and goods.

	Variable	Attribute	# Good	# Bad	Bad rate	Weight of evidence 3	Information value 4
		Married	27 682	873	3.1%	0.48	9.5%
	1. Marital status	Divorced, Widowed, Cohabiting	16 314	946	5.5%	-0.13	0.5%
		Single, Separated	9 303	898	8.8%	-0.64	10.0%
			53 299	2 717				20.0%
		Managers and professionals	6 920	220	3.1%	0.47	2.3%
		Technicians and
	2. Occupation	professionals associated	17 602	701	3.8%	0.25	1.8%
		Service and sales workers	18 502	1 018	5.2%	-0.08	0.2%
		Other jobs	10 275	778	7.0%	-0.40	3.7%
			53 299	2 717				8.0%
	3. Age of the client (in years)	>= 64 50 -63 40 -49 20 -39	10 655 16 369 13 687 12 588	349 732 749 887	3.2% 4.3% 5.2% 6.6%	0.44 0.13 -0.07 -0.32	3.2% 0.5% 0.1% 2.9%
			53 299	2 717				6.7%
		>= 246	9 019	273	2.9%	0.52	3.6%
	4. Time at jobs	164 -245 & retired	20 169	808	3.9%	0.24	2.0%
	(in months)	63 -163	13 251	824	5.9%	-0.20	1.1%
		<= 62 & unemployed	10 860	812	7.0%	-0.38	3.6%
			53 299	2 717				10.3%
	5. Time at present address (in months)	>= 210 102 -209 28 -101 <= 27	13 572 13 399 15 840 10 488	432 563 954 768	3.1% 4.0% 5.7% 6.8%	0.47 0.19 -0.17 -0.36	4.5% 0.9% 0.9% 3.1%
			53 299	2 717				9.4%
	6. Household	> 3725	13 657	343	2.5%	0.71	9.2%
	income	2756 -3724	13 416	595	4.2%	0.14	0.5%
	(in euros per	2004 -2755	13 281	721	5.1%	-0.06	0.1%
	month)	<= 2003	12 945	1 058	7.6%	-0.47	6.9%
			53 299	2 717				16.7%
	7. Residential	House owner	33 114	934	2.7%	0.59	16.4%
	status	Non-house owner	20 185	1 783	8.1%	-0.55	15.3%
			53 299	2 717				31.7%
	3 Weight of evidence is a measure of how good or bad the accounts are within a particular attribute:
		𝑊𝑒𝑖𝑔ℎ𝑡 𝑜𝑓 𝑒𝑣𝑖𝑑𝑒𝑛𝑐𝑒 = 𝑙𝑜𝑔( %𝑔𝑜𝑜𝑑 %𝑏𝑎𝑑	)

Table 4 :

 4 Comparison of results of different reject inference methods

		Overall Bad rate	Known GB Odds/ Inferred GB Odds	𝑮𝒊𝒏𝒊 𝒅𝒆𝒗 𝑮𝒊𝒏𝒊 𝒕𝒆𝒔𝒕	Overfitting
	KGB model	5%	NA	46%	39%	7%
	Reweighting	10%	2.8	44%	38%	6%
	Parceling	11%	3.2	38%	39%	-1%
	Fuzzy augmentation	43%	19.6	21%	39%	-18%
	Heckman's bivariate two-stage model	1%	NS 6	NS	NS	NS

The Information value measures the predictive ability of a characteristic to separate between good and bad accounts.𝐼𝑛𝑓𝑜𝑟𝑚𝑎𝑡𝑖𝑜𝑛 𝑣𝑎𝑙𝑢𝑒 = 𝑤𝑒𝑖𝑔ℎ𝑡 × (%𝑔𝑜𝑜𝑑 -% 𝑏𝑎𝑑)

According to Viola de Azevedo Cunha (2013), the sole public register is the French Central Bank (Banque de France), which is responsible for supplying credit institutions with credit information.

NS stands for non-significant.

The following attributes are not presented in Table5 :-The reference attribute in each variable with its weight of evidence in absolute value closest to 0; -The attributes which are not statistically significant at the 0.05 level; -The attributes which are correlated with each other.

SE stands for standard error.

NS means non-significant.

ACKNOWLEDGEMENTS

I am heartily thankful to my supervisor, Professor Valérie Mignon, for her valuable comments on earlier drafts of this paper. I would also like to thank the providers of the data, who prefer to remain anonymous. All errors are my own.

Cumulative

Good

Cumulative Bad

Lorenz Curve/Gini

Gini coeff Random