C Bouveyron

J Jacques

Adaptive mixtures of regressions: Improving predictive inference when population has changed

Keywords: Transfer learning, Mixture of regressions, Switching regression, EM algorithm, Bayesian inference, MCMC algorithm

The present work investigates the estimation of regression mixtures when population has changed between the training and the prediction stages. Two approaches are proposed: a parametric approach modelling the relationship between dependent variables of both populations, and a Bayesian approach in which the priors on the prediction population depend on the mixture regression parameters of the training population. The relevance of both approaches is illustrated on simulations and on an environmental dataset.

Introduction

The mixture of regressions, introduced by [START_REF] Goldfeld | A markov model for switching regressions[END_REF] as the switching regression model, is a popular regression model in order to model complex system. In particular, the switching regression model is often used in Economics to model phenomena with different phases. This model assumes that the dependent variable Y ∈ R can be linked to a covariate x = (1, x 1 , ..., x p ) ∈ R p+1 by one of K possible regression models:

Y = x t β k + σ k ε, k = 1, ..., K (1.1)
with prior probabilities π 1 , . . ., π K (with the classical constraint ∑ K i=1 π k = 1), where ε ∼ N (0, 1), β k = (β k0 , ..., β kp ) ∈ {β 1 , . . . , β K } is the regression parameter vector in R p+1 and σ 2 k ∈ {σ 2 1 , . . ., σ 2 K } is the residual variance. The conditional density distribution of Y given x is therefore:

p(y|x) = K ∑ k=1 π k φ (y|x t β k , σ 2 k ), (1.2)
where φ (•|x t β k , σ 2 k ) is the univariate Gaussian density parametrized by its mean x t β k and vari- ance σ 2 k . Among the works which focused on this model, we can emphasize the following ones which have contributed to the popularity of this model: [START_REF] Hurn | Estimating mixtures of regressions[END_REF] proposes a Bayesian inference for the model estimation, [START_REF] Zhu | Hypothesis testing in mixture regression models[END_REF] studies the asymptotic theory of parameter estimators in order to define hypothesis tests, and [START_REF] Khalili | Variable selection in finite mixture of regression models[END_REF] considers variable selection for this specific regression model. The present paper focuses on the problem of using a mixture regression model for prediction when the modelled phenomenon has changed between the training stage, which has led to the parameter estimation, and the prediction stage. More precisely, we assume that model (1.1) has been estimated with a sample from a given training population, let us say P, (of size large enough to have a satisfying estimation quality), and we want to use it to predict the dependent variable Y for a new population, let us say P * , which could be different from the training one. For instance, the difference between both populations can be due to a switch in the covariate distribution or to a variation of the link between the covariates and the dependent variable. The goal is then to transfer the knowledge from the training (source) population to the prediction (target) population. This task is usually known as transfer learning (see [START_REF] Pan | A survey on transfer learning[END_REF] for a complete survey), and can be summarized by Figure 1 in the case of the regression mixture model.

We now give some application examples of transfer learning. In a biological context, [START_REF] Biernacki | A generalized discriminant rule when training population and test population differ on their descriptive parameters[END_REF] and [START_REF] Jacques | Extension of model-based classification for binary data when training and test populations differ[END_REF] proposed models for clustering male and female birds: the source population consists of birds from a common species whereas the target population is composed of birds from a rarer species. Another application concerns the problem of sentiment classification as considered by [START_REF] Blitzer | Biographies, bollywood, boomboxes and blenders: Domain adaptation for sentiment classification[END_REF]. As the review data can be very different among several type of products, there is a need to collect a large sample of labelled data for each product in order to train a specific review-classification model per product. The use of transfer learning techniques allows to adapt a sentiment classifier from one product to another one. In [START_REF] Bouveyron | Adaptive linear models for regression: improving prediction when population has changed[END_REF], the authors predict house prices from house features for a city of the USA West Coast (San Jose, California) by adapting a regression model learned with data issued from another city stated on the East Coast (Birmingham, Alabama). The use of a transfer learning model allows to spare an additional and expensive collect of training data for the target population (San Jose housing in this application). [START_REF] Pan | Domain adaptation via transfer component analysis[END_REF] and [START_REF] Xue | Topic-bridged plsa for cross-domain text classification[END_REF] consider applications in text classification and WiFi localisation. More examples can be found in [START_REF] Pan | A survey on transfer learning[END_REF].

Related work

Transfer learning is a particularly active research field since the NIPS-95 workshop "Learning to Learn", in which the need for machine learning methods reusing previously learned knowledge was exhibited. Contrary to previously cited works in the classification context ( [START_REF] Biernacki | A generalized discriminant rule when training population and test population differ on their descriptive parameters[END_REF][START_REF] Jacques | Extension of model-based classification for binary data when training and test populations differ[END_REF]), in which the data of the target population can be unlabelled, the regression purpose need to observe at least some couples (y i , x i ) in the target population. In this case, we speak of inductive transfer learning. Readers interested in a comprehensive review can refer to [START_REF] Pan | A survey on transfer learning[END_REF].

Most of the methods allowing to treat such a setting are especially designed for estimating simultaneously the parameters of both source and target populations (we speak of multi-task learning), but can easily be adapted for transfer learning. They consider either a Bayesian or a regularization framework. Typically, in the Bayesian approach, each task is assume to share the same prior (see [START_REF] Lawrence | Learning to learn with the informative vector machine[END_REF] for instance). In the regularization framework, parameters between models for source and target population are assumed to be linked (see [START_REF] Evgeniou | Regularized multi-task llearning[END_REF] in a SVM context for instance).

In the regression context, Covariate Shift is a specific transfer learning problem considering that the probability density of the covariates in the target population is different from the one of the source population. However, the relationship between covariates and dependent variable is assumed not to have changed ( [START_REF] Shimodaira | Improving predictive inference under covariate shift by weighting the log-likelihood function[END_REF][START_REF] Storkey | Mixture regression for covariate shift[END_REF][START_REF] Sugiyama | Active learning in approximately linear regression based on conditional expectation of generalization error[END_REF][START_REF] Sugiyama | Input-dependent estimation of generalization error under covariate shift[END_REF][START_REF] Sugiyama | Covariate shift adaptation by importance weighted cross validation[END_REF]). Thus, if the regression model is exactly known, a change in the probability distribution of the explanatory variables is not a problem. Unfortunately, this is never the case in practice and the regression model estimated with the training data could be very disappointing when applied to data with a different probability distribution.

The originality of our work consists in introducing parametric models allowing to link the source and target populations. A more conventional Bayesian approach is also investigated, and comparison of both approaches are carried out on simulation and real data.

Problem formulation

Assuming that the target population P * , for which we want to predict Y , is different from the source population P, the mixture regression model for P * can be written as follows:

Y * = x * t β * k + σ * k ε * , (1.3) p(y * |x * ) = K * ∑ k=1 π * k φ (y * |x * t β * k , σ * k 2 ), with ε * ∼ N (0, 1), β * k ∈ {β * 1 , . . . , β * K * } and σ * k ∈ {σ * 1 , . . . , σ * K * }.
Let us now precise the focus of this paper by making the three following assumptions: H 1 : the couples of variables (Y, x) and (Y * , x * ) are assumed to be the same but measured on two different populations.

H 2 : the size n * of the observation sample S * = (y * i , x * i ) i=1,n * of population P * is assumed to be small compared to the number of observations of the source population P. Otherwise, the mixture regression model could be estimated directly without using the source population.

H 3 : as both populations have the same nature, each mixture is assumed to have the same number of components (K * = K).

Under these assumptions, the goal is then to predict Y * for some new x * by using both samples S = (y i , x i ) i=1,n and S * . The challenge consists therefore in exhibiting a link between both populations.

Organization of the manuscript

The reminder of this work is organised as follows. Section 2 proposes a first solution to improve the predictive inference on the target population by defining parametric models for the link between mixture regression models of both populations. This approach has the advantage to lead to interpretable results, which should help the practitioner in analysing the differences between the source and target populations. An alternative Bayesian approach, most frequent in transfer learning, is presented in Section 3. The link between regression models is then formulated through prior densities on the target population. The advantage of this strategy is its flexibility which can fit into different situations, if the prior densities are well chosen. In Section 4, the performance of both the parametric and the Bayesian approaches is first illustrated on simulations. Then, the proposed strategies are compared to classical methods on an environmental dataset. Section 5 finally proposes some concluding remarks and directions for future works.

Parametric approach for adaptive mixture of regressions

This section presents a parametric approach which consists in modelling the link between training and test populations by a parametric relationship between the regression parameters.

Parametric models to link the reference and test populations

Let us first introduce a latent variable Z * ∈ {0, 1} K representing the belonging of observations to the K mixture components, i.e. z * ik = 1 indicates that the i-th observation (x * i , y * i ) comes from the k-th component and z * ik = 0 otherwise. Conditionally to an observation x of the covariates, we would like to exhibit a distributional relationship between the dependent variables of the same mixture component such that Y * |x,z * ik =1 and ψ k (Y |x,z ik =1 ) have the same probability distribution, with ψ k a function from R to R.

Let β k and β * k (1 ≤ k ≤ K) be respectively the parameters of the mixture regression models in the source and the target populations (Equations (1.1) and (1.3)). We assume in this section that the function ψ k , exhibiting the link between the source and target populations, is such that:

β * k = Λ k β k , where Λ k = diag(λ k0 , λ k1 , . . ., λ kp ) (2.1) σ * k is free,
where diag(λ k0 , λ k1 , . . . , λ kp ) denotes the diagonal matrix containing (λ k0 , λ k1 , . . . , λ kp ) on its diagonal. The interest of introducing such a link lies in the reduction of the number of parameters to estimate for the mixture regression model for P * . In the sequel, we go further by introducing some constraints on Λ k and σ * k in order to define a family of parsimonious models, which includes many of the situations that may be encountered in practice:

• M 1 assumes both populations are the same:

Λ k = I d is the identity matrix (σ * k = σ k ),
• M 2 models assume the link between both populations is covariate and mixture component independent:

-M 2a : λ k0 = 1, λ k j = λ and σ * k = λ σ k ∀1 ≤ j ≤ p, -M 2b : λ k0 = λ , λ k j = 1 and σ * k = σ k ∀1 ≤ j ≤ p, -M 2c : Λ k = λ I d and σ * k = λ σ k , -M 2d : λ k0 = λ 0 , λ k j = λ 1 and σ * k = λ 1 σ k ∀1 ≤ j ≤ p,
• M 3 models assume the link between both populations is covariate independent:

-M 3a : λ k0 = 1, λ k j = λ k and σ * k = λ k σ k ∀1 ≤ j ≤ p, -M 3b : λ k0 = λ k , λ k j = 1 and σ * k = σ k ∀1 ≤ j ≤ p, -M 3c : Λ k = λ k I d and σ * k = λ k σ k , -M 3d : λ k0 = λ k0 , λ k j = λ k1 and σ * k = λ k1 σ k ∀1 ≤ j ≤ p,
• M 4 models assume the link between both populations is mixture component independent (σ * k free):

-M 4a : λ k0 = 1 and λ k j = λ j ∀1 ≤ j ≤ p, -M 4b : Λ k = Λ with Λ a diagonal matrix,

• M 5 assumes Λ k is unconstrained, which leads to estimate the mixture regression model for P * by using only S * (σ * k free).

Let us remark that other transformation models could be defined, in particular by considering that only the variance component is different between the source and target populations. Even though, only the previous models are investigated in this paper, the practitioner can easily introduced other models if needed, by following the strategy presented here. Moreover, the mixing proportions are allowed to be the same in each population or to be different. In the latter case, they consequently have to be estimated using the sample S * . Corresponding notations for the models are respectively pM • when the mixing proportion of P * have to be estimated and M • when not. Table 1 gives the number of parameters to estimate for each model. If the mixing proportions are different from P to P * , K -1 parameters to estimate must be added to these values. The estimation of the models M 2 to M 4 are derived in the next subsection. Let us also remark that by only assuming that the function ψ k (defined at the beginning of this section) is C 1 , rather than assuming (2.1), [START_REF] Biernacki | A generalized discriminant rule when training population and test population differ on their descriptive parameters[END_REF] proves that ψ k is necessarily affine, and then , where (λ k1 , λ k2 ) ∈ R 2 . We therefore obtain the following relationship between the model parameters of P and P * :

Model M 1 M 2a M 2b M 2c M 2d M 3a M 3b M 3c M 3d M 4a M 4b M 5 Param. 0 1 1 1 2 K K K 2K p + K p + K + 1 K(p + 2)
β * k = (λ k1 + λ k2 β k0 , λ k2 β k1 , . . ., λ k2 β kp ) t , (2.2) σ * k = λ k2 σ k . (2.
3)

The model M 3d previously defined, which is the most general model among the M 2 and M 3 classes of models, is equivalent to the model defined by relations (2.2) and (2.3). M 4 -type models allow to introduce more flexibility in the proposed model.

Parameter estimation

In the situation under review in this paper, the mixture of regressions is assumed to be known (β k and σ k will be estimated in practice from a sample of sufficient size) for the source popu- lation P, and the goal is to estimate the mixture of regressions for P * . This will be done in two steps. In the first step, the link parameters Λ k and the mixing proportions π * k are estimated as well as the residual variances σ * 2 k when necessary (models M 4 ). In the second step, the estimation of the mixture regression parameters β * k and the residual variances σ * 2 k (for models M 2 and M 3 ) are deduced by plug-in through equations (2.1) and (2.3). In the sequel, only the situation where mixing proportions are different from those of population P is considered.

The estimation of the link parameters is carried out by maximum likelihood using a missing data approach via the EM algorithm [START_REF] Dempster | Maximum likelihood from incomplete data (with discussion)[END_REF]. This technique is certainly the most popular approach for inference in mixtures of regressions. Conditionally to a sample S * = (y * , x * ) of observations, where y * = (y * 1 , . . . , y * n ) and x * = (x * 1 , . . . , x * n ), the log-likelihood of model (1.3) is given by:

L(θ ; y * , x * ) = n * ∑ i=1 ln K ∑ k=1 π * k φ (y * i |x t i Λ k β k , σ * 2 k ) , (2.4
)

with θ = (π * 1 , . . ., π * K , Λ 1 , . . . , Λ K , σ * 1 , . . . , σ * K )
, and the complete log-likelihood is:

L c (θ ; y * , x * , z * ) = n * ∑ i=1 K ∑ k=1 z * ik ln π * k φ (y * i |x t i Λ k β k , σ * 2 k ) , (2.5)
where z * = (z * ik ) i=1,n * ,k=1,K is the unobserved latent variable, introduced in Section 2, and assumed to be distributed as a one order multinomial M (1, π * 1 , . . . , π * K ).

The E step From a current value θ (q) of the parameter θ , the E step of the EM algorithm consists in computing the conditional expectation of the complete log-likelihood:

Q(θ , θ (q) ) = E θ (q) [L c (θ ; y * , x * , z * )|y * , x * ] = n * ∑ i=1 K ∑ k=1 t (q) ik ln(π * k ) + ln(φ (y * i |x t i Λ k β k , σ * 2 k )) , (2.6)
where:

t (q) ik = E[z * ik |y * , x * ] = P(z * ik = 1|y * , x * ) = π * k (q) φ (y * i |x * t i Λ (q) k β k , σ * 2 k (q) ) ∑ K l=1 π * l (q) φ (y * i |x * t i Λ (q) l β l , σ * 2 l (q) ) , (2.7)
is the conditional probability for the observation i to belong to the k-th mixture component.

The M step

The M step of the EM algorithm consists in choosing the value θ (q+1) which maximizes the conditional expectation Q computed in the E step:

θ (q+1) = argmax θ ∈Θ Q(θ ; θ (q) ), (2.8)
where Θ is a parameter space depending on the model at hand. For the mixing proportions, the maximum is as usual reached for:

π (q+1) k = 1 n * n * ∑ i=1 t (q) ik . (2.9)
For the residual variances (models M 4 ), we have:

σ * 2(q+1) k = 1 ∑ n * i=1 t (q) ik n * ∑ i=1 t (q) ik (y * i -x * t i Λ (q) k β k ) 2 .
(2.10)

The reminder of this section details the maximisation of the link parameters:

• for model pM 2a : λ (q+1) is the positive solution of the quadratic equation

n * λ 2 + λ n * ∑ i=1 K ∑ k=1 t (q) ik (y * i -β k0 )x * t i∼0 β k∼0 σ 2 k - n * ∑ i=1 K ∑ k=1 t (q) ik (y * i -β k0 ) 2 σ 2 k = 0, where x * i∼0 = (x * i1 , . . . , x * ip ) is the vector x * i without its first component x * i0
, and similarly β k∼0 = (β k1 , . . . , β kp ),

• for model pM 3a : λ (q+1) k is the positive solution of the quadratic equation

n * k λ 2 k + λ k n * ∑ i=1 t (q) ik (y * i -β k0 )x * t i∼0 β k∼0 σ 2 k - n * ∑ i=1 t (q) ik (y * i -β k0 ) 2 σ 2 k = 0, where x * i∼0 = (x * i1 , . . . , x * ip ) is the vector x * i without its first component x * i0 , similarly β k∼0 = (β k1 , . . . , β kp ), and n * k = ∑ n * i=1 t (q)
ik ,

• for model pM 2b :

λ (q+1) = n * ∑ i=1 K ∑ k=1 t (q) ik σ 2 k β k0 2 -1 n * ∑ i=1 K ∑ k=1 t (q) ik σ 2 k (y * i -x * t i∼0 β k∼0 )β k0 ,
• for model pM 3b : λ

(q+1) k = n * ∑ i=1 t (q) ik σ 2 k β k0 2 -1 n * ∑ i=1 t (q) ik σ 2 k (y * i -x * t i∼0 β k∼0 )β k0 ,
• for model pM 2c : λ (q+1) is the positive solution of the quadratic equation

n * λ 2 + λ n * ∑ i=1 K ∑ k=1 t (q) ik y * i x * t i β k σ 2 k - n * ∑ i=1 K ∑ k=1 t (q) ik y * i 2 σ 2 k = 0, • for model pM 3c : λ (q+1) k
is the positive solution of the quadratic equation

n * k λ 2 k + λ k n * ∑ i=1 t (q) ik y * i x * t i β k σ 2 k - n * ∑ i=1 t (q) ik y * i 2 σ 2 k = 0,
For the model pM 2d , as two interdependent scalar parameters λ 0 and λ 1 are considered, no an- alytical formulae are available for the global maximum on both λ 0 and λ 1 . In such a situation, an easy way to carry out the maximization is to consider a descending algorithm in which λ 0 and λ 1 are alternatively maximized. Using such a strategy incorporated in a EM algorithm is very frequent and, in such a case, the algorithm is called a generalized EM algorithm (GEM, [START_REF] Dempster | Maximum likelihood from incomplete data (with discussion)[END_REF]). Update formulae for these two parameters are consequently:

λ (q+1) 0 = ∑ n * i=1 ∑ K k=1 t (q) ik β k0 (y * i -λ (q+1) 1 x * t i∼0 β k∼0 )σ -2 k ∑ n * i=1 ∑ K k=1 t (q) ik β 2 k0 σ -2 k , and λ (q+1) 1
is the positive solution of the quadratic system:

n * λ 2 1 + λ 1 n * ∑ i=1 K ∑ k=1 t (q) ik σ 2 k x * t i∼0 β k∼0 (y * i -λ (q+1) 0 β k0 ) - n * ∑ i=1 K ∑ k=1 t (q) ik σ 2 k (y * i -λ (q+1) 0 β k0 ) 2 = 0.
For the model pM 3d , the same algorithm is considered with the following update formulas:

λ (q+1) k0 = ∑ n * i=1 t (q) ik (y * i -λ (q+1) k1 x * t i∼0 β k∼0 ) ∑ n * i=1 t (q) ik β k0 , and λ (q+1) k1
is the positive solution of the quadratic system:

n * k λ 2 1 + λ 1 n * ∑ i=1 t (q) ik σ 2 k x * t i∼0 β k∼0 (y * i -λ (q+1) k0 β k0 ) - n * ∑ i=1 t (q) ik σ 2 k (y * i -λ (q+1) k0 β k0 ) 2 = 0.
Models M 4a and M 4b have respectively p and p + 1 scalar parameters plus the residual variance. A descending algorithm has to be used for alternatively maximizing the variances (by (2.10)) and each scalar link parameter. Update formulas for the link parameters are the following:

• model M 4a , ∀1 ≤ J ≤ p: λ (q+1) J = n ∑ i=1 K ∑ k=1 t (q) ik σ * k 2 (x * iJ β kJ ) 2 -1 n ∑ i=1 K ∑ k=1 t (q) ik σ * k 2 x * t iJ β kJ y * i -β k0 - p ∑ j=1, j =J λ (q+1) j x * i j β k j , • model M 4b , ∀0 ≤ J ≤ p: λ (q+1) J = n ∑ i=1 K ∑ k=1 t (q) ik σ * k 2 (x * iJ β kJ ) 2 -1 n ∑ i=1 K ∑ k=1 t (q) ik σ * k 2 x * iJ β kJ y * i - p ∑ j=0, j =J λ (q+1) j x * i j β k j . with x i0 = 1 for all 1 ≤ i ≤ n.
The EM algorithm stops when the difference of the likelihood value of two consecutive steps is lower than a given threshold ε (typically ε = 10 -6 ).

Convergence considerations

Since the parameter estimation is based on an EM algorithm which respects the classical conditions of the EM theory [START_REF] Dempster | Maximum likelihood from incomplete data (with discussion)[END_REF], its convergence to a local maximum of the likelihood is guaranteed. Several strategies have been proposed in the literature to initialize the EM algorithm in order to help the algorithm to reach the global optimum of the likelihood. A popular practice [START_REF] Biernacki | Initializing EM using the properties of its trajectories in Gaussian mixtures[END_REF] executes the EM algorithm several times from a random initialization and keeps only the set of parameters associated with the highest likelihood. In this specific work, initializing the link parameter with Λ k = I d could also be an interesting alternative since it corresponds to P = P * .

Model selection

In order to select the most appropriate model of transformation among the 24 transformation models defined in Section 2, we propose to use two well known criteria. The reader interested in a comparison of the respective performances of models selection criteria could refer to [START_REF] Hastie | The elements of statistical learning[END_REF] for instance. The first considered criterion is the PRESS criterion [START_REF] Allen | The relationship between variable selection and data augmentation and a method for prediction[END_REF], also known as the cross-validation criterion, which represents the mean squared prediction error computed on a cross-validation scheme, formally defined by:

PRESS = n * ∑ i=1 (y * i -ŷ * (i) ) 2 ,
where ŷ * (i) is the prediction of y * i obtained by the mixture regression model estimated without using the ith observation of the sample S * . This criterion is one of the most often used for model selection in regression analysis, and we encourage its use when it is computationally feasible. The second considered criterion is the Bayesian Information Criterion (BIC, [START_REF] Schwarz | Estimating the dimension of a model[END_REF]), which is a penalized likelihood criterion with a less computational cost. The BIC criterion is defined by:

BIC = -2 ln ℓ + ν ln n * ,
where ℓ is the maximum log-likelihood value and ν is the number of estimated parameters (see Table 1). It consists in selecting the models leading to the highest likelihood while penalizing models with a large number of parameters. Let us precise that, for both criteria, the most adapted model is the one with the smallest criterion value.

Bayesian approach for adaptive mixture of regressions

The previous section has considered the modelling and the estimation of parametric adaptive models for mixture of regressions with the classical frequentist point of view. This section adopts a Bayesian approach to infer adaptive mixture of regressions and Gibbs sampling is considered for the estimation of the posterior distribution.

A Bayesian view of the problem

The classical treatment of the mixture regression problem seeks a point estimate of the unknown regression parameters. By contrast, the Bayesian approach [START_REF] Hurn | Estimating mixtures of regressions[END_REF][START_REF] Robert | The Bayesian Choice[END_REF] characterizes the uncertainty on parameters through a probability distribution, called a prior distribution. Bayesian analysis combines the prior information on the parameters (carried out by the prior distribution) with information on the current sample (through the likelihood function) to provide estimates of the parameters using the posterior distribution. In the context of adaptive mixture of regressions, the Bayesian approach makes particularly sense since there is an actual prior on the model parameters of population P * . Indeed, even though source and target populations differ, they are here assumed to have a strong link and it is therefore natural to define the prior on parameters of population P * according to the ones of population P.

In the context of mixture of regressions, it is usual to assume the conditional independence between the mixing parameters π * and both component parameters

β * = {β * 1 , ...β * K } and σ * 2 = {σ * 2 1 , ..., σ * 2 K }. The independence between (β * k , σ * 2 k ) and (β * ℓ , σ * 2 ℓ
) is as well assumed for all k = ℓ, k, ℓ = 1, ..., K. For simplicity, only conjugate priors are considered in this work and, since model parameters of the reference population P are assumed to be known, prior distributions of the parameters of population P * will depend on model parameters of the population P. We therefore propose to assume that, for all k = 1, ..., K, the prior distribution for β * k is a normal distribution centred in β k :

β * k ∼ N (β k , σ * 2 k A k ),
where A k is a (p + 1) × (p + 1) covariance matrix. The prior distribution of σ * 2 k , for all k = 1, ..., K, is assumed to be an inverse-gamma distribution:

σ * 2 k ∼ I G (γ k , ν k ).
The prior distribution for parameters π * = {π * 1 , ..., π * K } is assumed to be a Dirichlet distribution centred in the mixing proportions (π 1 , ..., π K ) of population P:

π * ∼ D(π 1 , ..., π K ).
With such a modelling, the regression coefficients β * k and the mixing proportions π * = {π * 1 , ..., π * K } of population P * are naturally linked to the ones of population P. The variance terms σ * 2 k A k control how the regression coefficients β * k differ from the ones of the reference population P. In the experiments presented in Section 4, the prior parameters ν k , γ k and A k , k = 1, ..., K, were respectively set to 1, 2 and the identity matrix.

Finally, by combining the likelihood of the mixture of regressions model and the priors, we end up with the joint posterior distribution:

p(θ * |Y * ) ∝ n * ∏ i=1 K ∑ k=1 π * k φ (y * i |x * t β * k , σ * 2 k ) p(π * ) K ∏ k=1 p(β * k |σ * 2 k )p(σ * 2 k )
,

where θ * = (π * k , β * k , σ * k ) k=1,K .
However, since the posterior distribution p(θ * |Y * ) takes into account all possible partitions of the sample into K groups, the maximization of p(θ * |Y * ) is intractable even with moderately large sample size and Markov Chain Monte Carlo methods have to be used.

Gibbs sampler for adaptive mixture of regressions

Markov Chain Monte Carlo methods allow to approximate a complicated distribution by using samples drawn indirectly from this distribution. Among MCMC methods, the Gibbs sampler is the most commonly used approach when dealing with mixture distribution ( [START_REF] Diebolt | Estimation of finite mixture distributions through bayesian sampling[END_REF]). In Gibbs sampling, the vector parameter θ * is partitioned into s groups of parameters {θ * 1 , ..., θ * s } and a Markov chain is generated by iteratively sampling from the conditional posterior distributions. Once a Markov chain of length Q has been generated, sample values can be averaged on the last sampling iterations to provide consistent estimates of model parameters. In the context of inference for mixture distribution, the Gibbs sampler requires to add a latent variable Z * ∈ {0, 1} K representing the allocation of observations to the K mixture components (introduced in Section 2). Since the latent variable Z * is not observed, Z * can be viewed as unknown and should be estimated along with the other model parameters. Consequently, given estimates β and π of respectively regression parameters and mixing proportions of population P and starting from initial values π * (0) , β * (0) and σ * 2(0) , the Gibbs algorithm generates, at iteration q, parameter values from the conditional posterior distributions: Z * (q) ∼ p(Z|Y * , β , π, π * (q-1) , β * (q-1) , σ * 2(q-1) ), π * (q) ∼ p(π * |Y * , β , π, Z * (q) , β * (q-1) , σ * 2(q-1) ), q) , π * (q) , σ * 2(q-1) ).

σ * 2(q) k ∼ p(σ * 2 k |Y * , β , π, Z * (q) , π * (q) , β * (q-1) ), β * (q) k ∼ p(β * k |Y * , β , π, Z * (
According to the priors given in the previous paragraph, the conditional posterior distribution of Z * is a multinomial distribution:

z * i |Y * , β , π, π * , β * , σ * 2 ∼ M (1,t i1 , ...,t iK ), where t ik = π * k φ (y * i |x * t i β * k , σ * 2 k )/ ∑ K ℓ=1 π * ℓ φ (y * i |x * t i β * ℓ , σ * 2 ℓ )
, and the conditional posterior distribution of π * is a Dirichlet distribution:

π * |Y * , β , π, Z * , β * , σ * 2 ∼ D( π1 + n * 1 , ..., πK + n * K ), with n * k = ∑ n i=1 z * ik .
Once the component belongings of each observation are known, the observations of the same component k can be gathered into the matrices x * k and Y * k , for all k = 1, ..., K. With these notations, the conditional posterior distribution of σ * 2 k is an inverse gamma:

σ * 2 k |Y * , β , π, Z * , π * , β * k ∼ I G (γ k + n k /2, ν k + S k /2), where S k = (Y * k -x * t k β * k ) t (Y * k -x * t k β * k ) + ( βk -β * k ) t (A k + (x * t k x * k ) -1 ) -1 ( βk -β * k )
, and the conditional posterior distribution of β * k is a normal distribution:

β * k |Y * , β , π, Z * , π * , σ * 2 k ∼ N (m k , ∆ k ) , with m k = (A -1 k + x * t k x * k ) -1 (x * t k Y * k + A -1 k βk ), ∆ k = σ * 2 k (x * t k x * k + A -1 k ) -1 .
Finally, consistent estimates of model parameters π * , β * and σ * 2 are obtained by averaging on the last Qq 0 sampling iterations, where q 0 defines the number of iterations of the so called "burning phase" of the Gibbs sampler.

The label switching problem

When simulating a Markov chain to estimate parameters of a mixture model, the label switching problem frequently arises and is due to the multimodality of the likelihood. Indeed, if the prior distributions are symmetric, the posterior distribution inherits the multimodality of the likelihood. In such a case, the Markov chain can move from one mode to another and it is difficult to deduce consistent estimators of model parameters. The earliest solution, proposed by [START_REF] Richardson | On bayesian analysis of mixtures with an unknown number of components[END_REF], consists in adding identifiability constraints on model parameters such as an order relation in mixing proportions. Unfortunately, this approach does not work very well as showed by [START_REF] Celeux | Computational and inferential difficulties with mixture posterior distributions[END_REF]. By contrast, some authors like [START_REF] Celeux | Computational and inferential difficulties with mixture posterior distributions[END_REF] and [START_REF] Stephens | Dealing with label switching in mixture models[END_REF] propose to work a posteriori on the generated Markov chain in order to reorganize it according to a specific criterion. The Stephens' procedure reorganizes the Markov chain by searching the correct permutations of mixture component which minimizes a divergence criterion. The solutions proposed by Celeux et al. are in the same spirit and, among the different proposed criteria, they propose in particular to reorganize the Markov chain using a sequential k-means algorithm. Both the Stephens and Celeux's approaches are efficient to deal with the label switching problem. However, the sequential k-means algorithm has the advantage to be less memory consuming and, in the experiments presented in Section 4, this approach is used to overcome the label switching problem.

Experimental results

This section proposes experiments on simulated and real data in order to highlight the main features of the adaptive models proposed in the previous sections. After an introductory example, the ability of BIC criterion to select the best model is investigated on simulation in a second experiment. In a third experiment, the behaviour of adaptive mixtures of regressions (parametric and Bayesian) is compared to the one of classical mixtures of regressions on simulated data. The last experiment will demonstrate the interest of using adaptive mixtures of regressions on an illustrative real dataset, and where the size of the target population sample will be artificially moved from small to larger sizes.

An introductory example

This first experiment aims to compare the basic behaviours of adaptive mixtures of regressions (parametric and Bayesian), hereafter referred to as AMR (respectively AMRp and AMRb), and classical mixtures of regressions, referred to as MR. For this study, the reference population P is modelled by a 2 component mixture of quadratic polynomial regressions with parameters β 1 = (3, 0, -2), σ 1 = 1, β 2 = (-3, 0, 0.5) and σ 2 = 0.75. The covariate x is uni- formly distributed on [-3,3] and the sample size for P is n = 1 000. The left panel of Figure 2 shows the mixture regression of population P as well as some observations simulated from this model. The mixture model of population P * has then been obtained from the previous model by multiplying all regression parameters of population P by a factor 3. It follows that β * 1 = (9, 0, -6) and β * 2 = (-9, 0, 1.5). Finally, 20 observations of population P * have been simulated using the latter model on [0, 3], which therefore corresponds to a censured model. The right panel of Figure 2 shows the actual mixture regression model of population P * as well as the 20 simulated observations (red triangles). These 20 observations of P * were used by the three studied regression methods to estimate the regression model of P * and to predict the value of 5 000 validation observations of P * . The mean square error (MSE), computed on the validation sample, has been chosen to evaluate the predicting ability of each regressions method in this introductory example. Since mixture of regressions provides K predictions (one prediction per mixture component), the MSE values reported in the following experiments are, for each observation, the minimum of the K prediction errors.

Figure 3 illustrates the estimation procedure of the Bayesian approach on this toy dataset. The MCMC procedure was made of 1 000 sampling iterations including a burning phase of 100 iterations. The left panel of Figure 3 shows the sampled proportions over the MCMC iterations. As one can see, after the burning phase, the proportions of both mixture components stabilize in the neighbourhood of 0.5 which is the actual value of π 1 and π 2 . The central panel presents the sampled values for regression parameters β 1 and β 2 in the parameter space (restricted to β k1 and β k3 for k = 1, 2 because both β 12 and β 22 are both equal to 0). The blue and green dashed lines indicate at the intersections the actual values of regression parameters. It appears that the Bayesian approach succeeds in estimating the conditional distributions of regression parameters. Finally, the right panel exhibits some of the 1 000 regression models generated during the MCMC iterations which are then used to provide by averaging the final estimated regression model of P * .

Figure 4 presents the results obtained for the considered example with the classical mixture of regressions (MR), parametric adaptive mixture of regressions (AMRp) and Bayesian adaptive mixture of regressions (AMRb). The MR method used only the 20 observations sampled from P * whereas AMR and AMRb combines the informations carried by these observations with the knowledge on P to build their estimation of the mixture regression model of P * . In order not to favour the adaptive approaches, the actual number of components and dimension of the polynomial regression were also provided to the MR method. Nevertheless, the MR method provides a poor estimate of the regression model and its mean square error (MSE) value, computed on a independent validation set, is consequently high (3704.4). Conversely, the parametric (with the model pM 3c ) and Bayesian approaches of AMR give good estimations of the P * model (they should be compared to the red curves of Fig. 3). The associated MSE values are naturally much lower than the one of the classical MR method (26.4 for AMRp and 45.2 for AMRb). Nonetheless, the Bayesian approach performs less than the parametric AMRp. This could be due to the fact that AMRb favours too much the prior (the regression parameters of P) in this situation with only few observations of the new population. This introductory example has shown that adaptive regression models succeed in transferring the knowledge of a reference population to a new population.

Model selection

In this second experiment, we investigate the ability of BIC to select the best AMRp model, in the same condition as for the previous experiment. For this, we carry out simulations according to the different AMRp models and estimate a mixture of regressions on these simulated data with usual mixture of regression (MR) on both populations (P and P * ) and with the ten AMRp models (M 2a to M 4b ). The experimental setup is as follows:

• as in the previous experiment, the reference population P is modelled by a 2 component mixture of quadratic polynomial regressions with parameters β 1 = (3, 0, -2), σ 1 = 1 and β 2 = (-3, 0, 0.5), σ 2 = 0.75. The sample size for P is n = 500.

• 10 scenarios have been used to define the mixture model of P * , corresponding to the ten AMRp models M 2a to M 4b with the following link parameters (with the same notation as in Section 2): λ = 2 for the models M 2a , M 2b and M 2c ; (λ 0 , λ 1 ) = (4, 2) for for P * , a dataset of size n * = 20 is simulated. The simulation are repeated 100 times in order to average the results.

M 2d ; (λ 1 , λ 2 ) = (2, 4) for M 3a , M 3b and M 3c ; (λ 10 , λ 20 , λ 11 , λ 21 ) = (3, 5, 2, 1/4) for M 3d ; (λ 1 , λ 2 ) = (4, 2) for M 3a ; Λ = diag(3,
• for each scenario, MR on both populations (P and P * ) and the ten AMRp models M 2a to M 4b are estimated.

For each model of simulation and each estimated model, Table 2 presents: the average MSE, evaluated on a validation dataset of size 5 000 and averaged over the 100 simulations and the number of times the estimated model has been selected by the BIC criterion. The last column gives the average MSE for the model selected by BIC. These experiments show that the asymptotic BIC criterion found relatively well the true model (the one used for the simulations), even in this difficult situation in which the sample size for P * is small (n * = 20). Let us notice that, since in P the second coefficient of β 1 and β 2 is 0, the model M 4a is here equivalent to M 2a which is logically selected by BIC since it is less complex than M 4a . Moreover, the BIC criterion leads generally to choose models with lower MSE than MR on P or P * (a bad model choice by BIC could nevertheless drastically degrades the average MSE, as for instance for the simulations according to M 2d or M 3d ). Finally, this experiment confirms the good comportment of the EM algorithm for the estimation of AMRp models parameters, since MSE values are most of the time better for the model used for the simulation than for MR.

Influence of the size of S *

This experiment focuses now on the influence of the number of observations n * from the new population P * on the estimation quality of mixture regression models for the MR, AMRp and AMRb methods. The experimental setup is the same as for the experiment of Section 4.1, except that the number of observations n * from the new population P * varies from 6 to 200. For each value of n * , the regression model of P * has been estimated with the three studied methods and the associated MSE values have been computed again on a independent validation set of 5 000 observations. Finally, the experiment has been replicated 50 times in order to average the results. Figure 5 shows the evolution of the median logarithm of the MSE value according to the the size of S * for the classical mixture of regressions (MR), parametric adaptive mixture of regressions (AMRp) and Bayesian adaptive mixture of regressions (AMRb) methods.

For the classical mixture of regressions, the multiple initialization strategy discussed in Section 2.3 is used in order to avoid bad initializations of the EM algorithm. For the parametric approach of the AMR method, the model used is pM 3c . Associated boxplots are presented by Figure 6 on a logarithmic scale. On view of Figure 5, it can be first noticed that the performance of the classical MR method is, as expected, sensitive to the the size of S * . Indeed, for small sample sizes, the MR method provides poor estimates of the mixture regression model of population P * and this consequently yields poor prediction performances (large MSE values). Again, as one can expect, the model estimation and the prediction improve when the number of observations n * from the new population P * increases. More surprisingly, as it can be observed on the left panel of Figure 6, the variance of the prediction performance of the MR method remains large for sample sizes bigger than 100, even with the multiple initialization strategy. This remind us that the fitting of a mixture regression model is always a difficult and sensitive task. Conversely, the adaptive methods AMRp and AMRb which exploit their knowledge on the reference population obtain on average good prediction results (low MSE values) and this even for very small numbers of observations n * . In particular, the parametric approach AMRp provides very stable prediction results and its variance decreases quickly when n * increases. The Bayesian approach AMRb, even though it is much efficient and stable than the classical MR method, appears to be slightly less efficient than the parametric approach AMRp. To summarize, this study on simulations has shown that adaptive regression models greatly improve the prediction and reduce the predictor variance compared to the classical mixture regression approach when the number of observations of the new population is small.

Illustration on real data: CO 2 emissions vs gross national product

In this last experiment, the link between CO 2 emission and gross national product (GNP) of various countries is investigated. The sources of the data are The official United Nations site for the Millennium Development Goals Indicators and the World Development Indicators of the World Bank. Figure 7 plots the CO 2 emission per capita versus the logarithm of GNP per capita for 111 countries, in 1980 (left) and 1999 (right). A mixture of second order polynomial regressions seems to be particularly well adapted to fit these data and will be used in the sequel.

Let remark that regression model with heteroscedasticity could also be appropriated for such data, but these kind of models are out of the topic of the present work. For the 1980's data, two groups of countries are easily distinguishable: a first minority group (about 25% of the whole sample) is made of countries for which a grow in the GNP is linked to a high grow of the CO 2 emission, whereas the second group (about 75%) seems to have more environmental political orientations. As pointed out by [START_REF] Hurn | Estimating mixtures of regressions[END_REF], the study of such data could be particularly useful for countries with low GNP in order to clarify in which development path they are embarking. This country discrimination into two groups is more difficult to obtain on the 1999's data: it seems that countries which had high CO 2 emission in 1980 have adopted a more environmental development than in the past, and a two-component mixture regression model could be more difficult to exhibit.

In order to help this distinction, parametric adaptive mixture models are used to estimate the mixture regression model on the 1999's data. The ten AMRp models, with free component proportions π * k , pM 2a to pM 4b , AMRb model, classical mixture of second order polynomial regressions with two components (MR) and usual second order polynomial regression (UR) are considered. Different sample size of the 1999's data are tested: 30%, 50%, 70% and 100% of the S * size (n * = 111). The experiments have been repeated 20 times in order to average the results. Table 3 summarizes these results: MSE corresponds to the mean square error, whereas PRESS and BIC are the model selection criteria introduced in Section 2.4. In this application, the total number of available data in the 1999 population is not sufficiently large to separate them into two training and test samples. For this reason, MSE is computed on the whole S * sample, even though a part of it has been used for the training (from 30% for the first experiment to 100% for the last one). Consequently, MSE is a significant indicator of predictive ability of the model when 30% and 50% of the whole dataset are used as training set since 70% and 50% of the samples used to compute the MSE remain independent from the training stage. However, MSE is a less significant indicator of predictive ability for the two last experiments and the PRESS should be preferred in these situations as indicator of predictive ability.

Table 3 first allows to remark that the 1999's data are actually made of two components as in the 1980's data since both PRESS and MSE are better for MR (2 components) than UR (1 component) for all sizes n * of S * . This first result validates the assumption that both the reference population P and the new population P * have the same number K = 2 components, and consequently the use of adaptive mixture of regression makes sense for this data. Secondly, AMRp turns out to provide very satisfying predictions for all values of n * and particularly outperforms the other approaches when n * is relatively small (less than 77 here). Indeed, both BIC, PRESS and MSE testify that the models of AMRp provide better predictions than the other studied methods when n * is equal to 30%, 50% and 70% of the whole sample. Furthermore, it should be noticed that ARMp provide stable results according to variations on n * . In particular, the models pM 2 are those which appear the most efficient on this dataset and this means that the link between both populations P and P * is mixture component independent. On the other hand, the Bayesian approach AMRb appears to provide results as stable as the ones of AMRp but slightly less satisfying. The results of the Bayesian approach would probably be better with a more specific choice of the priors. This application illustrates well the interest of combining informations on both past (1980) and present (1999) situations in order to analyse the link between CO 2 emissions and gross national product for several countries in 1999, especially when the number of data for the present situation is not sufficiently large. Moreover, the competition between the parametric AMR models is also informative. Effectively, it seems that three models are particularly well adapted to model the link between the 1980's data and those of 1999's data: pM 2a , pM 2b and pM 2c . The specificity of these models is that they consider the same transformation for both classes of countries, which means, conversely to what one might prima facie have thought, that all the countries have made an effort to reduce their CO 2 emissions and not only those which had the higher ones.

Conclusion

We proposed in this paper adaptive models for mixture of regressions in order to improve the predictive inference when the studied population has changed between training and prediction phases. The first class of models considers a parsimonious and parametric link between the mixture of regressions of both populations, whereas the second approach adopts a Bayesian point a view in which the populations are linked by the prior information imposed on the mixture regression parameters. On both simulated and real data, models considering parametric link turn out to be the most powerful: all the interest of such adaptive methods consists in their sparsity, which leads to significantly decrease the number of observations of the target population required for the estimation. As the indispensable stage of data collecting is often expensive and time consuming, there is a real interest to consider adaptive mixture of regressions in practical applications. Moreover, as it has been showed in the illustration on real data, the competition between the parametric link models provides informations on the link between populations, which can be meaningful for the practitioner.

Regarding the further works, a first perspective concerns the Bayesian approach. In this paper, the prior hyperparameters for σ * 2 k were simply fixed to values seeming experimentally reasonable. The results of the Bayesian approach may be improved by working on the choice of these hyperparameters. One generic way to do this is to make similar assumptions as in the frequentist approach. For instance, the variance σ * 2 k A k of the regression parameters β * k could be assumed to be common between mixture components or to be equal to σ * 2 k I d . The selection between the considered assumptions could then be done by choosing those maximizing the integrated likelihood [START_REF] Raftery | Estimating the integrated likelihood via posterior simulation using the harmonic mean identity[END_REF]. A second working perspective is related to the joint estimation of the models of both populations P and P * . Indeed, the reference regression model being only estimated in practice, the quality of this estimation, depending on the size n of the available sample, is directly responsible of the estimation quality of the mixture regression model for P * . In some situations (typically when n is small compared to the model complexity), it could be interesting to consider a full likelihood estimation which consists in estimating simultaneously both mixture regression models. Such an approach has been recently considered in [START_REF] Lourme | Gaussian model-based classification when training and test population differ: Estimating jointly related parameters[END_REF] in the supervised classification context. It must be emphasized that such a full likelihood estimation of both mixtures of regression must consider the same estimation method (parametric or Bayesian) for both populations.

Figure 1 :

 1 Figure 1: Learning process of transfer learning for the regression mixture model.

Figure 2 :

 2 Figure 2: Populations P and P * used for the introductory example. Curves black (left) and red (right) indicates respectively the actual mixture regression of populations P and P * .

Figure 3 :

 3 Figure3: Results obtained for the introductory example with the Bayesian approach of adaptive mixture of regressions (AMRb). From left to right: mixing proportions over the MCMC iterations, Gibbs sampling in the parameter space and some of the generated regression curves. See text for details.

Figure 4 :

 4 Figure 4: Results obtained for the introductory example with classical mixture of regressions (MR), parametric adaptive mixture of regressions (AMRp) and Bayesian adaptive mixture of regressions (AMRb) methods. See text for details.

Figure 5 :

 5 Figure 5: Median logarithm of the MSE value according to the the size of S * for the classical mixture of regressions (MR), parametric adaptive mixture of regressions (AMRp) and Bayesian adaptive mixture of regressions (AMRb) methods.

Figure 6 :

 6 Figure 6: Boxplots of MSE values (on logarithmic scale) according to the the size of S * for the classical mixture of regressions (left), parametric adaptive mixture of regressions (centre) and Bayesian adaptive mixture of regressions (right) methods.

Figure 7 :

 7 Figure 7: Emission of CO 2 per capita versus GNP per capita in 1980 (left) and 1999 (right).

Table 1 :

 1 Number of parameters to estimate for each model of the proposed family. ik =1 have the same probability distribution λ k1 + λ k2 Y |x,z ik =1

	Y * |x,z *

Table 2 :

 2 Average MSE value (top number) and number of BIC selection (bottom number) for the 10 AMRp models, usual mixture of regression (MR) on P and P * , for 10 scenarios of simulation corresponding to each AMRp models.

	2, 3/2) for M 4b . According to the resulting model

Table 3 :

 3 MSE on the whole 1999's sample, PRESS and BIC criterion for the 10 parametric adaptive mixture models (AMRp pM 2a to pM 4b ), AMRb model, usual regression model (UR) and classical regressions mixture model (MR), for 4 sizes of the 1999's sample: 33, 55, 77 and 111 (whole sample). Lower BIC, PRESS and MSE values for each sample size are in bold character. ( †): Cross-validation on MCMC procedures is too computationally heavy to be computed in a reasonable time.