Pierre Alquier

Nicolas Chopin

Feng Liang

Keywords: prior

Je voudrais tout d'abord remercier vivement mon directeur de thèse, Nicolas Chopin, pour son engagement, sa disponibilité et son enthousiasme contagieux lors de nos discussions scientifiques. Pour toutes ces raisons, travailler sous sa direction a été une exprience très agréable et enrichissante. J'adresse également mes remerciements à Randal Douc, Erwan Le Pennec et Christian Robert d'avoir accepté de participer à mon jury de thèse ainsi qu'à Christophe Andrieu et Olivier Catoni pour avoir accepté de la rapporter et pour leur lecture attentive et leurs commentaires constructifs. Je remercie les doctorants ayant séjourné au bureau E28: Adélaïde, Edwin, JB et Mathieu ainsi que ceux du CREST et de Dauphine: Clara

Introduction

In this thesis we study some computational aspects of Bayesian statistics, as well as Gibbs posteriors. We describe both statistical approaches in the first two sections. We then give a brief overview of computational aspects linked to the implemention of those estimators in Section 1.3 of this chapter. This chapter is devoted to discussing the different issues that will arise in the manuscript; detailed accounts will be found in the following chapters.

Bayesian statistics

Statistical analysis starts with a collection of probability distributions P θ indexed by a parameter θ ∈ Θ, where Θ is an arbitrary set. In this thesis most examples will be taken from parametric statistics, with Θ ⊂ R d . The statistician then confronts the model to some observations on the probability space (X , A, {P θ , θ ∈ Θ}).

The goal of this thesis is not the choice of the collection of probability distribution but rather, given the model and the data, to discuss ways of improving the computation of estimators of θ.

We describe in the following the Bayesian paradigm that will be used in this manuscript. In particular we discuss the minimization of the integrated posterior risk. As we will see, this criterion leads to computational difficulties because it requires the evaluation of high dimensional integrals.

In Section 1.2, we will take another approach: we will not suppose a specific probability model but rather minimize an empirical risk over classes of classifiers and give theoretical guaranties for such an approach.

In all the following we suppose the model to be dominated by some measure and we define the likelihood as the probability density of the observation given the parameter.

Definition 1.1.1 We call likelihood the probability density of the collection (X 1:n ) ∈ X n conditioned on θ. We denote it by L(X 1:n |θ).

We give two examples of models that will be used in the rest of the thesis. Those models will be used in the introduction as examples and will be further studied in the different chapters of this thesis.

Example 1.1.1 Probit. This is a special case of generalized linear models where the probability of a binary random variable is expressed as a linear combination of some covariates. Conditionally on θ and a vector of deterministic covariates x the model can be expressed in a hierarchical form:

Y i = ✶ Z i >0 , Z i |β ∼ N (x i β, 1).
Each observation is sampled independently from this model. The likelihood is given by

L(y 1:n |θ) = n i=1 (Φ(x i θ)) y i (1 -Φ(x i θ)) 1-y i .
This model will be studied in details in Chapter 2. In Chapter 3 we give an example of a model where we allow correlations between the (Z i ) i and a way to compute the likelihood.

Example 1.1.2 State space model (SSM). SSM is another model that we will use in this thesis (see Chapter 6). This is a time series model with an unobserved Markov process (x t ) t≥0 with transition density x 0 ∼ g 0 (.; θ);

x t |x 1:t-1 ∼ g t (.|x t-1 ; θ)

and an observation y t at each time t depending on the current value of the chain,

y t ∼ f t (.|x t ; θ).
The likelihood of the observations up to time T is given by L(y 1:T ; θ) = ˆT t=1 f (y t |x t ; θ)g t (x t |x t-1 ; θ)µ 0 (x 0 )dx 0:T .

The fact that the likelihood takes the form of an integral with respect to a Markov process leads to computational issues. Those will be discussed in more details in Chapter 6.

2

Bayesian statistics

The main idea of the Bayesian paradigm is to endow Θ with the structure of a probability space (Θ, B, π) where the probability measure π is referred to as the prior. We do not discuss how to choose π; discussion of this subject can be found in [START_REF] Robert | The Bayesian choice: from decision-theoretic foundations to computational implementation[END_REF].

Once we have defined the prior, we can use the likelihood as a conditional probability and using Bayes identity one can define the posterior distribution: Definition 1.1.2 The posterior distribution is the distribution of the parameter given the observations,

π(dθ|D) = L(x 1:n |θ)π(dθ) ´Θ L(x 1:n |θ ′ )π(dθ ′ )
.

We call the integral m π (x 1:n ) := ´Θ L(x 1:n |θ ′ )π(dθ ′ ) the marginal likelihood.

The posterior is the distribution of the parameter given a fixed observed sample. We derive in the following section criteria to choose an estimator using this distribution.

Risk minimizer

Given a model {P θ , θ ∈ Θ}, the data and a prior π(dθ), with support Θ, the goal of the statistician is to take a decision δ minimizing a risk. Several types of risk can be considered and lead to different estimators. We now give examples that are used in this manuscript.

Once we have defined the risk one cannot simply minimize over value of ℓ because of the unknown θ. To deal with this issue within the Bayesian paradigm the statistician defines the posterior risk.

Definition 1.1.3 The Posterior risk is the loss function integrated with respect to the posterior distribution: ρ(π, δ|D) = ˆℓ(θ, δ)π(dθ|D).

A Bayesian estimator is a minimizer of the posterior risk. The most usual loss is the quadratic loss ℓ(δ, θ) = θδ 2 2 ; direct computation yields δ π = E π (θ|D). An other common loss is the following 0-1 loss,

ℓ(θ, δ) = ✶ Θ 1 (θ)✶ δ=1 + ✶ Θ 2 (θ)✶ δ=0 .
Direct computations yield:

δ π = 1 if P π (θ ∈ Θ 1 |D) > P π (θ ∈ Θ 2 |D) 0 otherwise
In the light of those examples we see the main computational issue appear. Those two examples show that the paradigm leads to non trivial integration problems. In both case one wants to compute: ´Θ h(θ)π(dθ)L(θ|D)

´Θ π(dθ)L(θ|D) , for some function h; i.e. an expectation with respect to a distribution that is known only up to the normalizing constant m π (D) = ´Θ π(dθ)L(θ|D).

Model choice

We can encode model uncertainty by defining the parameter space Θ = j Θ j ×{j} where {j} indexes the model and Θ j is the parameter space of models j. The standard approach in the Bayesian paradigm is to endow the space of model indexes with a prior distribution and to treat it as an additional parameter. This is easily encapsulated in the framework described in the previous section. Most of the time the likelihood itself will not be tractable in this case. We can still solve the problem using specially tailored MCMC algorithms [START_REF] Green | Reversible Jump Markov Chain Monte Carlo computation and Bayesian Model Determination[END_REF]. In Chapter 2 we will propose an alternative to this for the case of covariate uncertainty for probit models.

We have defined the normalizing constant of the posterior in Definition 1.1.2 as m π (x) = ´π(θ)L(x|θ)dθ. We write the likelihood of model j, m π (x|j). We assign a prior probability π j to model j. We end up with an integration problem with respect to the following distribution, π(j|D) = π j m π (x|j) j π j m π (x|j)

.

For covariate selection in a Gaussian linear model one can compute the marginal likelihood provided that the prior is conjugate to the Gaussian family (i.e. a Gaussian distribution). An example of the approach is given in [START_REF] Chopin | Sequential monte carlo on large binary sampling spaces[END_REF]. In general the marginal likelihood is not tractable. A way to get around this issue is to replace it by an estimator (see Chapter 2 for an application to the probit model).

PAC-Bayesian Bounds

When no clear mathematical model is available to the statistician or that existing models are too costly to evaluate or to build, one may still be able to construct a pseudo-posterior to replicate the behavior of the risk minimizer. We use for this matter a Gibbs posterior (defined bellow) for the chosen risk. For this posterior we prove nonasymptotic bound. The idea originated in machine learning [START_REF] Shawe-Taylor | A PAC analysis of a Bayesian estimator[END_REF], [START_REF] Mcallester | Some pac-bayesian theorems[END_REF]) as way to bound the theoretical risk in probability by a computable empirical quantity. We follow [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] and use the approach to get oracle inequalities on the risk integrated under our given pseudo-posterior.

We observe a sample (X 1 , Y 1 ), • • • , (X n , Y n ) taking values in X × Y where the pairs (X i , Y i ) have the same distribution P.

The statistician defines a set of predictor {f θ : X → Y}. We suppose we also have at our disposal a risk function R(θ) and its empirical counterpart r n (θ).

The approach is summarized for classification in this section. That is we take

Y = {-1, 1}, R(θ) = P (Y f θ (X) ≤ 0) and r n (θ) = 1 n n i=1 ✶ Y i f θ (X i )≤0 .
In the following we also suppose linear classifiers f θ (x) = 2✶ <x,θ>≥0 -1 ∈ {-1, 1}. More details on the derivation of the estimator and theoretical results are given in Chapter 4 and 5.

Definition 1.2.1 The Gibbs posterior for an empirical risk r n (θ) and a prior π ξ is given by

ρλ (dθ) = 1 Z λ,ξ exp {-λr n (θ)} π ξ (dθ)
where ξ is the vector of hyperparameters, and Z ξ,λ = ´Θ e -λrn(θ) π ξ (dθ).

Contrarily to the posterior defined in the preceding section we do not derive this probability density from Bayes' formula. The Gibbs posterior defined in definition 1.2.1 is the solution of the following variational problem:

-log Z λ,ξ = inf ρ∈M 1 + {λρ(r n (θ)) + K(ρ, π ξ )} ,
where M 1 + is the set of all probability measures. This is easily deduced from the following equality: for any ρ ∈ M 1 + log Z λ,ξ + K(ρ, ρλ ) = λρ(r n (θ)) + K(ρ, π ξ ).

Part of this thesis is concerned with proving oracle inequalities for those measures and providing approximations to the Gibbs posterior. We give a quick overview of an approach to derive bounds for the expected risk under a Gibbs posterior. We follow [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] and refer the reader to this reference for a complete account and tighter bounds.

We give a simplified proof for the classification loss, to give an idea of the tools used in Chapter 4 and 5. In Chapter 4 we will extend these results to the case of an AUC loss. The loss will be used to propose a method to rank instances from bipartite data.

Introduction

Using the above relation we derive the following results,

P sup ρ∈M 1 + λ (ρ(r n (θ)) -ρ(R(θ))) + K(ρ, π ξ ) + η ≥ 0 = P {log π ξ [exp {-λ(r n (θ) -R(θ)) + η}] ≥ 0} = P {π ξ [exp {-λ(r n (θ) -R(θ)) + η}] ≥ 1} ≤ Pπ ξ [exp {-λ(r n (θ) -R(θ)) + η}] = π ξ P [exp {-λ(r n (θ) -R(θ)) + η}] .
Since r n (θ) is bounded from above, Hoeffding's inequality allows us to write:

πP [exp {-λ(r n (θ) -R(θ)) + η}] ≤ π exp λ 2 2n + η
see Lemma 2.2 an Theorem 2.1 in [START_REF] Boucheron | Concentration Inequalities[END_REF]. A great part of the effort will be put on finding concentration inequality at this step. In Chapter 4 for instance we prove a Bernstein inequality for the AUC risk to this aim.

We put η = -λ 2 2nlog 2 ǫ and get ∀ρ ∈ M 1 + with probability less than ǫ/2:

λ (ρ(R(θ)) -ρ(r n (θ))) + K(ρ, π) - λ 2 2n -log 2 ǫ ≥ 0 (1.1)
such that with probability at least 1ǫ/2 we can upper bound the theoretical risk under the Gibbs posterior ρλ (Rθ) ≤ inf

ρ∈M 1 + ρ(r n (θ)) + 1 λ K(ρ, π) + λ 2n + 1 λ log 2 ǫ .
(1.2) Chapter 4 will be concerned with extensions of the framework to rank data, and with computational tools to compute approximations of the Gibbs posterior.

Notice that this intermediate result comes under the only hypothesis that the data is iid. In fact we can even weaken this assumption by taking a weakly dependent sample (see for instance [START_REF] Alquier | Prediction of quantiles by statistical learning and application to GDP forecasting[END_REF]).

Minimizing the bound

Another quantity of interest is the normalizing constant. In the framework described previously we can write inequality (1.2) using the dual formulation introduced in the beginning of the section w.p. at least 1ǫ

ρ(R(θ)) ≤ - 1 λ log Z λ,ξ + λ 2n + 1 λ log 2 ǫ .
Several authors have proposed to use this bound not only to give an empirical bound on the true risk but also for estimation. That is, choose the hyper-parameter of the prior or an approximation of the model, such that the empirical bound is the tightest. The log-normalizing constant is not always easy to compute, however we show in the following sections some tools to find an unbiased estimator of this integral. In Chapter 5 we show how to replace the optimal measure by the optimal measure in a smaller class of distribution for which the integral is tractable.

PAC Bayesian oracle inequality

We call oracle some estimator based on an unobservable quantity. In most of this thesis it will be given by the minimizer of the theoretical risk. Here we write θ = arg min θ∈Θ R(θ). From equation 1.1 we get simultaneously ∀ρ ∈ M 1 + (Θ) with probability 1ǫ,

ρ(R(θ)) ≤ ρ(r n (θ)) + 1 λ K(ρ, π ξ ) + λ 2n + 1 λ log 2 ǫ , ρ(r n (θ)) ≤ ρ(R(θ)) + 1 λ K(ρ, π ξ ) + λ 2n + 1 λ log 2 ǫ .
Specifying ρ as the Gibbs posterior in the first of the two equations and replacing ρ(r n ) by its upper bound one gets with probability 1ǫ,

ρλ (R(θ)) ≤ inf ρ∈M + 1 ρ(R(θ)) + 2 λ K(ρ, π ξ ) + λ n + 2 λ log 2 ǫ .
The inequality gives rise to a bound on the integrated theoretical risk. To obtain a specific bound and introduce the oracle risk we will introduce an additional assumption.

Definition 1.2.2 There exists a constant c > 0 such that ∀(θ, θ ′ ) ∈ Θ2 with θ = θ ′ = 1 we have P (< X, θ >< X, θ ′ >≤ 0) ≤ c θθ ′ .

The assumption implies that for any two given linear classifiers we can control the amount of points that are classified differently. More specifically, suppose that X admits a density, with respect to the d -1 spherical measure, upper bounded by B then we can show, The hypothesis imposes that for any two classifiers there is enough mass in the region where their conclusion differs.

P {< X, θ >< X, θ ′ >≤ 0} ≤ B 2π arccos(< θ, θ ′ >) ≤ B 2π θ1 X θ2 X • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 8
By specifying the prior to be a Gaussian distribution with variance ϑ we get an oracle inequality. That is we take π(θ) = d i=1 ϕ(θ i ; 0, ϑ 2 ). Several other prior can be considered (see Chapter 4 for an application to spike and slab to induce sparsity).

Theorem 1.2.1 Take λ = √ nd and ϑ = 1 √ d . Under the assumption of Definition 1.2.2, for any ε > 0, with probability at least 1ε we have,

ρλ (R(θ)) ≤ R( θ) + d n log 4ne 2 + c √ n + d 4n 3 + 2 log 2 ε √ nd .
The proof of this result is provided in Chapter 5. Note that we can also deduce bounds in expectation from bound of Theorem 1.2.1, using the fact that ´∞ 0 P {X > t} dt = E (X).

In Chapter 4 we specify other types of priors with the aim of dealing with different hypotheses i.e. a spike and slab for sparsity and a Gaussian process prior for non-linear classifier.

The issue with the result is that the estimator is intractable. We will give ways to approximate this measure in the subsequent chapters. In Chapter 5 in particular we give an approximation for which we can obtain oracle inequalities directly.

Computational aspects

The general idea behind Bayesian computational statistics is easily summed up by Minka [2001a]. " [. . . ] instantiate the things we know, and integrate over the things we don't know, to compute whatever expectation or probability we seek" We will therefore be interested in computing the integral ˆΘ h(θ)π(dθ|D).

We have seen that for a quadratic loss the optimal estimator is the posterior mean. But we can also take interest in other quantities such as moments, the mode, posterior probabilities etc. In all these cases we have to deal with the intractability of the integral with respect to the posterior as well as the intractability of the normalizing constant. We will divide the introduction of the different algorithms in two sections, the first dealing with Monte Carlo , i.e. based on random samples, the second on deterministic variational approximations.

Monte Carlo Principle of Monte Carlo

Monte Carlo is based on the use of the law of large numbers (LLN) to approximate integrals. We will be interested in computing integrals of the form I h := E P (h(X)). The LLN gives us, under the hypothesis that the (X i ) ≥0 are iid P and the existence of I h , that:

1

N N i=1 h(X i ) P ---→ N →∞ I h .
A stronger result is given by the central limit theorem under the hypothesis of finite second order moment,

√ N 1 N N i=1 h(X i ) -I h L ---→ N →∞ N (0, σ 2 ),
with σ 2 the variance of h(X) under P.

We need a way to sample from P in the most general way possible to apply these results. Several approaches exist for sampling, the two direct ones one may consider are:

• Inverting the c.d.f: for sampling from a distribution with c.d.f F one can sample a uniform U ∼ U [0,1] and compute Y = F -1 (U ). This requires knowledge of the c.d.f. which we have seen will not happen often in practice, and the ability to compute its inverse or pseudo-inverse. We give an example below for the truncated Gaussian distribution which we will use repeatedly in Chapter 3.

• Rejection sampling Accept reject (AR) is based on the fact that sampling uniformly in the epigraph of a density leads to points marginally distributed under it. The additional layer here is that we use a proposal distribution g such that uniformly on the support of the target f we have that f ≤ M g for a known constant M . Then AR consists in sampling from the joint distribution

(U, X) ∼ M ✶ [0≤U≤ f (X)
M g(X) ] g(X). The proposal g is a degree of freedom for the user and will impact the efficiency of the algorithm. Finally, note that for log concave densities there exists algorithms to automatically construct g, see Chapter 3 of Robert and Casella [2004b] for an example.

In the above we have left one question unanswered: all of the algorithms require the ability to generate i.i.d sequence of uniform distribution. In this thesis we will consider that such an algorithm is given. An account is given in [START_REF] Niederreiter | Quasi-Monte Carlo methods and pseudo-random numbers[END_REF].

We give an example in the following on how to apply the above methodology to a truncated Gaussian (see Chapter 3).

Y = m + σΦ -1 U Φ b -m σ -Φ a -m σ + Φ a -m σ
This particular sampler will be used extensively in Chapter 3.

An AR algorithm has also been developed for this example [START_REF] Robert | Simulation of truncated normal variables[END_REF], using a translated exponential where the parameters are chosen as to maximize the acceptance probability. Chopin [2011a] proposes a more efficient algorithm based on the Ziggurat algorithm.

Importance sampling

Importance sampling is based on the identity

E P (h(X)) = E Q h(X) dP dQ (X)
under the hypothesis that Q >> P (measure Q dominates P).

As previously importance sampling is based on replacing the integration with respect to Q(dx) with its empirical counterpart

Q n (dx) = 1 N N i=1 δ X i (dx) with X i ∼ Q(dx) hence the estimator I h ≈ 1 M M i=1 w i h(X i ),
where we write w i (X i ) := dP dQ (X i ).

It is often the case that the weights w are known only up to a normalizing constant (in particular in the case of posterior sampling). One can use a normalized estimator. That is we will replace the unknown normalizing constant by the corresponding IS estimator 1 M M i=1 w i (X i ). Because this is a convergent estimator bounded away from 0, one can show that the ratio

M i=1 w i (X i )h(X i ) M i=1 w i (X i )
converges towards the correct integral. One can get results for finite M through concentration inequalities [START_REF] Cappé | Inference in Hidden Markov Models[END_REF] chapter 9).

In the rest of the chapter we suppose that both measures are dominated with respect to a base measure and write p q (X) the ratio of densities with respect to this measure.

Remark 1.3.1 Unbiased target

It is interesting to note that importance sampling remains valid if one replaces the weights with an unbiased estimator. Suppose we have a quantity TZ (X) such that E S TZ (X) = p q (X). In this case we can build an algorithm consisting in sampling Z i ∼ S and X i ∼ Q and forming the estimator

Îh = 1 N N i=1 TZ i (X i )h(X i ),
one readily sees that integrating with respect to Z yields an IS estimator of I h .

In the following we give a running example that we use to illustrate the case of unbiased targets. In Chapter 6 we will use such results in the context of state space models extending the algorithm of Chopin et al. [2013b].

Example 1.3.2 Symmetric Skellam distribution

The difference of two random variables with Poisson distributions is known as the Skellam distribution. We can write its density as a convolution between a Poisson and a negative Poisson, p(z; λ) = ∞ i=0 ̟(i + z; λ)̟(i; λ).

(1.3)

where ̟(x; λ) is the Poisson pdf with parameter λ evaluated at x. This distribution can be written as the expectation of E̟(z + X) where the variable X is distributed as a Poisson of parameter λ. An unbiased estimator is obtained by averaging over M X samples of a Poisson distribution.

We can apply the methodology described above to this case (see Figure 1.2). We use a Gaussian proposal with moments calibrated from the method of moment estimator. In Figure 1.2 we show the effect of changing the value of M X and hence modifying the variance of our unbiased estimator.

On a side note, notice that the estimator of the normalizing constant introduced above is unbiased. Hence it can be used inside other algorithm to perform inference on an intractable posterior, we use this approach for instance in Chapter 2, for model selection.

Resampling

Another important alternative is to sample from the empirical mixture formed by the weighted sample obtained with importance sampling, μRIS (dx) = M i=1 w i j w j δ x i (dx), where (w i , x i ) 1≤•••≤M is a consistently weighted sample from the target distribution (i.e. the IS estimator converges to the correct distribution).

Several algorithms exist to perform resampling. In this thesis we will use systematic resampling (see [START_REF] Douc | Comparison of resampling schemes for particle filtering[END_REF] for a comparison of resampling algorithms for particle filters). The convergence of the measure to the target in ensured by portmanteau lemma. Finite sample results also exist: Chapter 7 of [START_REF] Cappé | Inference in Hidden Markov Models[END_REF] provides a Hoeffding type inequality.

Monte Carlo Markov Chain (MCMC)

MCMC relies on a different idea. It consists in building a Markov kernel with the target distribution as an invariant distribution. MCMC relies on the ergodic We provide the convergent sequence of estimator of the conditional expectation of the parameter for different value of M X . We show the IS estimator and 95% confidence bands (obtained by replication). We use a simulated dataset for illustration with n = 40 observations. The value of M X has as expected an influence on the variance.

theorem to approach integration with respect to the invariant distribution. That is, we rely on the following theorem.

Theorem 1.3.1 (Robert and Casella [2004b]) Let (X n ) n≥0 be a Harris recurrent Markov chain with invariant probability π for any

f such that f ∈ L 1 (π) we have that 1 N N i=1 f (X i ) ---→ N →∞ ˆf (x)π(dx).
As for the iid case we can get a CLT under additional assumptions. We refer the reader to [START_REF] Gareth | General state space Markov chains and MCMC algorithms[END_REF] for a review on Markov chain theory 1 Introduction applied to MCMC and to Robert and Casella [2004b] for some applications of the results.

Gibbs sampler

We start our very brief overview of MCMC methods with Gibbs sampling. The methodology consists in constructing a Markov Chain by sampling alternatively from conditional distributions. One can easily show that this results in a Markov Chain targeting the joint distribution.

We describe the algorithm in pseudo-code below Algorithm 1 Two stage Gibbs sampler Input: Conditional distributions p(X|Z) and p(Z|X), a starting point Z 0 .

Output: (X, Z) 1≤t≤T a Markov chain with invariant probability p(X, Z)

For t ∈ {1, . . . , T } a. Sample X t ∼ p(.|Z t-1 ). b. Sample Z t ∼ p(.|X t ).

End For

The Gibbs sampler needs conditional conjugacy in its simplest form to work. For our example 1.1.1 on the probit model we can see that putting a Gaussian prior and sampling alternatively from p(β|Z 1:n , Y 1:n ) and p(Z 1:n |β, Y 1:n ) leads to a sampler targeting the correct joint distribution π(β, Z 1:n |X 1:n ). It is direct to see that the joint distribution of β, Z is given by,

π(β, Z 1:n |X 1:n ) = n i=1 {✶ Y i =1 ✶ Z i ≥0 + ✶ Y i =0 ✶ Z i ≤0 } ϕ(Z i ; x i β, 1)ϕ(β; 0 p , ϑI p ).
Hence the two conditionals are given by: -1 . One iterates between the two distributions as shown in Algorithm 1. In Chapter 2 we discuss alternatives to the Gibbs sampler that outperform it on many datasets.

β|(Z, Y ) 1:n ∼ N (Q -1 x t Y, Q -1 ) Z i |β, Y ∼ N (x i β, 1) {✶ Y i =1 ✶ Z i ≥0 + ✶ Y i =0 ✶ Z i <0 } with Q = x T x + ϑI p
Unbiased target In the case where only an unbiased version of the density is available, one can still construct a Gibbs sampler. Suppose that we can write the following joint distribution:

p(θ, X 1:M X ) = 1 M X M X i=1 TX i (θ) M X i=1 q i (X i )
such that integrating with respect to X gives the correct target density (with q an auxiliary distribution). That is such that T (θ) is an unbiased estimator under q of our target.

We can augment the space with an index k chosen with probability TX k (θ) M X . The joint distribution is then given by:

p(k, θ, X 1:M X ) = TX k (θ) 1 M X M X i=1 q(X i )
Summing over k gives the correct distribution. Hence the Gibbs sampler constructed on this joint distribution, which samples iteratively from k|θ, X 1:M X , θ|k, X k and X 1:M X |θ, k gives the correct distribution. This idea will be the building block of the parameter update in Chapter 6; the algorithm is developed in the special case of state space models. We use again the Skellam distribution as a toy example to illustrate the principle.

Example 1.3.3 Symmetric Skellam distribution (continued)
We put a Γ(1, 1) prior on λ as was done previously. Using what was described above we can write an algorithm to sample from the posterior. The efficiency of the algorithm could be discussed but it is however given here just as an example to prepare the reader for more advanced use of this idea in Chapter 6.

Algorithm 2 A Gibbs sampler for the Estimation of the Skellam parameter. The experiment is performed on the same simulated dataset as for importance sampling. It can be compared to Figure 1.2. At first glance the scheme does not seem that efficient as we need a large M X to get a variance comparable to the IS version.

Input: λ 0 , M X and M θ Output: (λ i ) 0≤i≤M θ For i ∈ {1, • • • , M θ } a. Sample M X i.i.d samples X j 1:N ∼ P N (λ) for j = k b. Sample k|λ, X 1:M X 1:n ∼ M X i=1 j ̟(y i + X i j ; λ) δ i c. Sample λ|X k 1:n , y ∼ Γ(2 + n i=1 (2y i + X k i ), 2n +
Remark 1.3.2 Before we start explaining a more general class of algorithms,let us note a property of the autocorrelations of Gibbs samplers.

Assuming without loss of generality Eh(X) = 0, we have that

E (h(X)h(X ′ )) = ˆh(X)h(X ′ ) ˆp(X|Z)p(Z|X ′ )dZp(X ′ )dX ′ dX = ˆ ˆh(X)p(X|Z)dX 2 p(Z)dZ ≥ 0
This is not the case for Metropolis-Hastings. In fact, using a certain version of a MCMC algorithm, Hamiltonian Monte Carlo (HMC) (see Chapter 2) we can construct an example where the correlation is negative between samples therefore leading to a gain in efficiency as compared to iid sampling. Metropolis-Hasting A more general approach to building a Markov chain with the correct invariant distribution is the Metropolis-Hastings (MH) algorithm. The algorithm uses a proposal and acceptance mechanism. A new state is proposed according to a Markov kernel, with density q. The proposal is accepted with a given probability (see Algorithm 3). The degree of freedom of the algorithm is given by the choice of the proposal q. Several common choices have been adopted in the literature. We give a few. First one may choose to propose independently from the current state. The difficulty with the approach is that we need to ensure that the proposal dominates the target. By adopting this view we are back to the problem of AR and IS. A more useful approach is to move locally around the current point by using a random walk (i.e. a distribution such that q(x|y) = q(y|x)). The density of the proposal therefore cancels in the acceptance ratio.

Many other proposals will be considered in the sequel, in particular we will make use of the gradient of the target to explore more efficiently the state space.

Unbiased target As for IS one requirement of those methods is the availability of the density up to a multiplicative factor. Suppose we have access to a random variable TX (θ) such that E q TX (θ) = T (θ) is our target distribution. By taking Algorithm 3 Metropolis Hastings algorithm

Input: θ 0 , M Output: (θ t ) t≥0 For t ∈ {1, • • • , M } a. Sample θ prop ∼ q(.|θ t-1 ). b. Sample U ∼ U([0, 1]). c. If U ≤ π(θprop|D)q(θ t-1 |θprop) π(θ t-1 |D)q(θprop|θ t-1 ) , set θ t ← θ prop , otherwise set θ t ← θ t-1 .
End For for proposal on (θ, X), q(θ ′ |θ)q(X ′ ), and for target distribution TX (θ)q(X) one gets the correct invariant with the acceptance ratio:

α(θ, θ ′ ) = 1 ∧ TX ′ (θ ′ ) ✟ ✟ ✟ ✟ q(X ′ )q(θ|θ ′ ) ✟ ✟ ✟ q(X) TX (θ) ✟ ✟ ✟ q(X)q(θ ′ |θ) ✟ ✟ ✟ ✟ q(X ′ ) .
In addition after simplification the acceptance ratio writes:

α(θ, θ ′ ) = 1 ∧ TX ′ (θ ′ )q(θ|θ ′ ) TX (θ)q(θ ′ |θ) .
Hence applying the methodology of Algorithm 3 to an unbiased estimator of the target yields a Markov chain with the correct invariant probability.

Sequential Monte Carlo (SMC)

The first step to building the SMC algorithm is to define a sequence of densities (π n ) ∈T where T is an index set, and such that there is one N ∈ T such that π N is the target. The idea is to apply IS to move from one distribution to another. We will explain in the rest of the section how we can take profit of resampling and ideas from MCMC to improve upon this. SMC is in particular useful for moderate dimensions in the case where MCMC explores the space too slowly.

We give a brief overview of the idea of the algorithm and relay the full description to Chapter 2 for static models and to Chapter 3 and 6 for SSM.

Particle filter (PF) Particle filters originated in [START_REF] Neil J Gordon | Novel approach to nonlinear/non-gaussian bayesian state estimation[END_REF] for state estimation in SSM. We describe the algorithm using the notations of Example 1.1.2.

The index set in this case is time and we want to sample recursively from the predictive distribution p(x t+1 |y 1:t ) and the filtering distribution p(x t |y 1:t ). To this aim we introduce the forward backward recursion,

p(x t+1 |y 1:t ) = ˆX p(x t |y 1:t )g(x t |x t-1 )dx t p(x t+1 |y 1:t+1 ) ∝ p(x t+1 |y 1:t )f (y t+1 |x t+1 )
This suggests the following importance sampling based algorithm: start by sampling an array of size M independently according to the initial distribution x i ∼ g 0 (x i 0 ); we call the elements particles. Move the particles according to the Markov kernel g(x i 1 |x i 0 ) weight each particles according to w i 1 = f 0 (y 1 |x 1 ). The operation is iterated this way for every time steps t. In the end we sample recursively from the Markov chain T t=1 g(x t |x t-1 ) and compute the product of weights

w i t = T t=1 f (y t |x i t )
. This algorithm amounts to an importance sampling algorithm where the simulations and weight computation are done sequentially. It is referred in the literature as sequential importance sampling (SIS) [START_REF] Cappé | Inference in Hidden Markov Models[END_REF].

We can use the algorithm to get an unbiased estimator of the normalizing constant, for the case of SSM given a value of the parameters θ it corresponds to the likelihood, L(y 1:T ; θ) =

´ t p(y t |x t ; θ)g t (x t |x t-1 ; θ)µ 0 (dx 0 )dx 0:T . In chapter 3 we reinterpret the GHK algorithm as a SIS algorithm for a given state space model, we use it to build a more efficient estimator of the Gaussian orthant probability.

The SIS algorithm as described proposes the state of the Markov chain using the true process g t (x t |x t-1 ) at each time t. On can easily see, that as for IS, we can choose an auxiliary distribution as long as we correct accordingly in the weights. We will use this feature in Chapter 3 and in fact propose each state by the optimal proposal (minimizing the variance).

One issue of SIS is the fact that we have to compute a product of weights; this leads to weight degeneracy [START_REF] Cappé | Inference in Hidden Markov Models[END_REF]. As T grows the number of particles with small weights will grow, and the normalized weight will go to zero. One can measure the variance of the weigths by introducing the ESS (effective sample size) [START_REF] Kong | Sequential imputations and bayesian missing data problems[END_REF],

ESS t = M i=1 w i t 2 M i=1 (w i t ) 2 ∈ [0, M ].
As the time horizon grows it is common to observe that this quantity goes to zero. In Chapter 3 we show that for a specific model, when computing the orthant probability with GHK, some quantity closely related to the ESS goes to 0 exponentially fast.

The solution proposed in the literature is to introduce resampling at each step or in an adaptive manner whenever the ESS falls under a given value. The particles are resampled according to their current weights and the later are set to one. We do not describe the case of PF with unbiased estimator of the weights although the interested reader can refer to [START_REF] Fearnhead | Random weight particle filtering of continuous time processes[END_REF].

Algorithm 4 Particle Filter

Input: M the number of particles Sample: Sample

x i 0 ∼ g 0 (.) for t = 1 : T -1 do if ESS < η ⋆ then Z ← Z × { 1 M M i=1 w i t } Resample a j t ∼ i w i t j w j t δ i , set w j t ← 1 else a 1:M t = 1 : M end if Sample x i t+1 ∼ q t+1 (.|x a i t t ) Set w i t+1 ← w i t f t+1 (y t+1 |x i t+1 )g t+1 (x i t+1 |x a i t t ) q t+1 (x i t+1 |x a i t t ) end for return Z × 1 M M i=1 w i T and (x i T , w i T ) 1≤i≤M
Particle filters can also be used to compute the likelihood of the model. In fact it can be shown that the quantity defined by Z in Algorithm 4 is an unbiased estimator of the likelihood (Del Moral [1996a], Lemma 3). This is useful in particular in lights of the remarks made on the use of algorithms with unbiased estimator of the target. Andrieu et al. [2010a] suggested using a Gibbs sampler and a MH using the estimator of the likelihood given by a particle filter.

In the following we will show that this algorithm can be generalized to more general problems.

Static models

The idea of using this framework to sample from other types of distribution originated in Neal [2001a], Chopin [2002a], (see also Del Moral et al. [2006b]).

The user needs to define a sequence of distribution; several choices can be considered. We describe the two used in this thesis.

• The first one was proposed under the name IBIS (iterated batch importance sampling) [Chopin, 2002a].

π n (θ) ∝ π(θ)L(y 1:n |θ)
and consists in adding one or several data points in a sequential manner. It is used in particular in Chapter 6 for parameter estimation of state space models.

• The sequence builds a bridge between two distributions,

π n (θ) ∝ {π(θ)L(y 1:N |θ)} γn {q(θ)} 1-γn whith γ n a sequence such that 0 = γ 0 < γ 1 < • • • < γ N = 1.
The choice of the sequence is of paramount importance for the efficiency of the algorithm. In what follows we use an adaptive choice for γ such that the ESS does not fall under a given threshold (see [START_REF] Jasra | Inference for Lévy-Driven Stochastic Volatility Models via Adaptive Sequential Monte Carlo[END_REF] and the different chapters of this thesis).

After a few resampling steps all particles θ would tend to be equal. We introduce an additional step to increase the diversity of the particle system, at time n we move the particles according to a kernel K n that leaves π n invariant. The choice of K n is of paramount importance as it will condition the diversity of the particle system.

The main steps are described in Algorithm (12) bellow:

Algorithm 5 Sequential Monte Carlo Input Γ,a, M , α ⋆ , Set Z ← 1 Each computation involving m is done ∀m ∈ 1 : M Init η m 1 ∼ π 0 (.), and w m 1 = 1 for t ∈ 1 : T -1 do At time t the weighted system is distributed as (w m t , η m 1:t ) ∼ π t (.). if ESS(w 1:M t ) < α ⋆ then Z ← Z × { 1 M M i=1 w i t }. Resample: η ′m t ∼ M j=1 w j t δ η j t , w m t ← 1. Move: η m t ∼ K t (η ′m t , dη m t ) where K t leaves π t (η 1:t ) invariant. end if w m t+1 ← w m t × π t+1 (η m t ) πt(η m t ) . end for return Z × { 1 M M i=1 w i T }, and (w i T , η i T )
A nice feature of SMC is that at a given step one can use estimators based on the system of particles to build adaptive proposals and MCMC steps. We will make some use of this in the different chapters, see in particular Chapter 6 where we use SMC in the context of intractable targets.

Approximate Inference

In the previous section we justified the algorithms as being exact with the CPU effort growing to infinity. Another approach is to approximate the target by the closest distribution in a tractable family. We give a brief overview of the two most used algorithms of this type and relay a full description to Chapter 2 for expectation propagation (EP) and Chapter 4 for variational Bayes (VB).

Expectation Propagation

In most applications of EP we will consider a posterior that can be written as a product of a tractable distribution and intractable factors. We call tractable a distribution for which we have easy access to moments and is stable by multiplication (as are exponential family models). We write:

π(θ|Y ) ∝ π(θ) i t i (θ).
EP works by successively approximating each sites by a given parametric distribution. To be more specific let us describe the global approximation Q to the previous posterior as a product:

Q(θ) ∝ π(θ) i ti (θ),
where each ti (θ) approximates its corresponding site t i (θ). EP works by cycling through the sites and updating them. Once a site has been updated we can use this to update the global approximation. Let us describe the process in more detail. We start by computing the cavity distribution as the approximation removed of the influence of site j,

Q \j (θ) ∝ π(θ) i =j ti (θ).
This distribution is easily obtained for exponential models, and consists only in a modification of the natural parameters. We multiply the cavity distribution by the corresponding new site t j (θ) to obtain the hybrid distribution:

h j (θ) = Q \j (θ)t j (θ) ´Q\j (θ)t j (θ)dθ .
This new distribution would ideally be our new distribution updating information of site j, it is however intractable. The approach consists in defining the new approximation as the one minimizing the KL divergence of Q with respect to the hybrid over the considered exponential family. Because the approximating distribution is in an exponential family the minimization can be done by computing the moments of the distribution (see Seeger [2005a]). This is done in cyclic manner for each site until a convergence criterion is achieved. We give an example where an EP algorithm can be used, and an iteration of the algorithm for a simple case.

Example 1.3.4 PAC-Bayesian 0-1 Classification The pseudo posterior we want to approximate is given by:

π ξ,γ (θ|D) ∝ n i=1 exp - λ n ✶ Y i <X i ,θ>≤0 π(θ)
and we suppose that the prior π(θ) is Gaussian. We can use EP to get a Gaussian approximation of the posterior because the moments of the distribution

h i (θ) ∝ ϕ(θ; m \i , Σ \i ) exp -λ n ✶ Y i <X i ,θ>≤0 can be com- puted exactly.
To make this application clearer, in Figure 1.5 we show the iterations of an EP approximation for a site of the form t(θ) = ✶ θ<0 .

More details on the computation of each terms in specific cases are given in the chapters of this thesis. Note in particular the EP is used in Chapter 2 to compute an approximation of the posterior of a probit model, and in Chapter 4 to approximate the Gibbs posterior under a AUC loss.

Fractional Expectation Propagation

Fractional EP is similar to the algorithm described previously with the exception that only a fraction of each site is treated at each iteration. We will denote this fraction by α ≤ 1. The corresponding cavity distribution is now given by Q \α,j (θ) ∝ n i=1 ti (θ) tj (θ) 1-α . The hybrid distribution can be written has

h j (θ) ∝ Q \α,j (θ) (t j (θ)) α .
The rest of the algorithm follows the steps of standard EP.

The rationale behind this strategy is twofold [START_REF] Jylänki | Robust gaussian process regression with a student-t likelihood[END_REF]. First taking α < 1 for non log-concave sites flattens the distribution and therefore prevents numerical issues linked to multimodality. Second it prevents the cavity moment of becoming too small (or even negative). Taking specific values of α can also allow us to transform a intractable problem into a tractable one. We give such an example in the following.

Example 1.3.5 Consider the Gaussian regression model with Student prior distribution as advocated for instance by [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF]. where t ν,γ (β i ) is the Student density with degree of freedom ν and scale γ. This prior leads to an intractable problem.

π(β|Y ) ∝ exp - 1 2 Y -X t β 2
Standard EP algorithm also leads to intractable integrals using fractional EP we can get a tractable problem that leads to tractable sites.

We can write

t ν,γ (β i ) ∝ 1 1 + β i γ 2 ν+1 2
Taking a fractional parameter η = -2 ν+1 for a Gaussian approximation one gets a tractable EP approximation for an otherwise intractable case. This can also be used in the Gaussian process case with Student likelihood to prevent from using numerical integration [START_REF] Jylänki | Robust gaussian process regression with a student-t likelihood[END_REF]. Regression with student prior marginal of the 3 first coefficients on the Boston dataset. In green the EP approximation and in blue the marginal from the sampled posterior (RWMH).

Variational Bayes

The variational Bayes algorithm appears as a way to transform an integral form problem in a finite dimension optimization problem [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF],MacKay [2002] and Chap. 10 in Bishop [2006a]). The idea of the approach is to minimize the KL divergence of a measure ρ with respect to the posterior measure of interest. In most cases of interest it is an intractable problem because of the unknown normalizing constant, however we can write the following useful decomposition of the log marginal likelihood,

log m(D) = E ρ log π(θ)L(D|θ) ρ(θ) + K(ρ|π(.|D)).
Because the left handside does not depend on ρ, minimizing the first term of the right handside is equivalent to minimizing the KL divergence. This surrogate objective function can be seen as a lower bound of the marginal likelihood. The above definition gives a natural optimization algorithm, minimizing the objective on all probability distribution yields the posterior, we minimize the objective restricted to a set of tractable distributions. We now review two types of families popular in the VB literature.

• Mean field VB: for a certain decomposition Θ

= Θ 1 × • • • × Θ d , F is the set 1 Introduction
of product probability measures

F MF = ρ ∈ M 1 + (Θ) : ρ(dθ) = d i=1 ρ i (dθ i ), ∀i ∈ {1, . . . , d}, ρ i ∈ M 1 + (Θ i ) .
(1.4) The infimum of the KL divergence K(ρ, π(.|D)), relative to ρ = i ρ i satisfies the following fixed point condition [Bishop, 2006a;Parisi, 1988, Chap. 10]:

∀j ∈ {1, • • • , d} ρ j (dθ j ) ∝ exp ˆ{log L (D|θ) + log π(θ)} i =j ρ i (dθ i ) .
(1.5) This leads to a natural algorithm were we update successively every ρ j until stabilization.

• Parametric family:

F P = ρ ∈ M 1 + (Θ) : ρ(dθ) = f (θ; m)dθ, m ∈ M ;
and M is finite-dimensional; say F P is the family of Gaussian distributions (of dimension d). In this case, several methods may be used to compute the infimum. As above, one may use fixed-point iterations, provided an equation similar to (5.4) is available. Alternatively, one may directly maximize the bound with respect to parameter m, using numerical optimization routines.

Mean field VB has been shown to underestimate the variance of the distribution. This is partly due to the fact the KL divergence takes infinite values if the target does not dominate ρ. In particular the condition is violated if the tails of ρ are lighter than the target.

To illustrate this problematic behavior let us look at an experiment proposed in Bishop [2006a], namely the mean field approximation of a centered bivariate Gaussian with variance 1 and correlation ρ. One readily checks that the iterations are given by variance of 1ρ 2 and a sequence of means converging to 0. Figure 1.7 illustrates the two distributions, with the approximation constrained by the variance in the direction of the smallest eigenvalue. In chapter 5 we give conditions under which VB approximations of the Gibbs posterior do not deteriorate the rate of convergence of the approximation.

Overview

Overview

This thesis is divided into 5 chapters, we give a short descriprion of each of them in the following. Abstract We study the computation of Gaussian orthant probabilities, i.e. the probability that a Gaussian variable falls inside a quadrant. The Geweke-Hajivassiliou-Keane (GHK) algorithm [START_REF] Genz | Numerical Computation of Multivariate Normal Probabilities[END_REF][START_REF] Geweke | Efficient simulation from the multivariate normal and student-t distributions subject to linear constraints[END_REF][START_REF] Hajivassiliou | Simulation of multivariate normal rectangle probabilities and their derivatives theoretical and computational results[END_REF][START_REF] Keane | Simulation estimation for panel data models with limited dependent variables[END_REF], is currently used for integrals of dimension greater than 10. In this chapter we show that for Markovian covariances GHK can be interpreted as the estimator of the normalizing constant of a state space model using sequential importance sampling (SIS). We show for an AR(1) the variance of the GHK, properly normalized, diverges exponentially fast with the dimension. As an improvement we propose using a particle filter (PF). We then generalize this idea to arbitrary covariance matrices using Sequential Monte Carlo (SMC) with properly tailored MCMC moves. We show empirically that this can lead to drastic improvements on currently used algorithms. We also extend the framework to orthants of mixture of Gaussians (Student, Cauchy etc.), and to the simulation of truncated Gaussians. Abstract We develop a scoring and classification procedure based on the PAC-Bayesian approach and the AUC (Area Under Curve) criterion. We focus initially on the class of linear score functions. We derive PAC-Bayesian non-asymptotic bounds for two types of prior for the score parameters: a Gaussian prior, and a spike and slab prior; the latter makes it possible to perform feature selection. One important advantage of our approach is that it is amenable to powerful Bayesian computational tools. We derive in particular a Sequential Monte Carlo algorithm, as an efficient method which may be used as a gold standard, and an expectation propagation algorithm, as a much faster but approximate method. We also extend our method to a class of non-linear score functions, essentially leading to a nonparametric procedure, by considering a Gaussian process prior.

Properties of variational approximations of Gibbs posteriors

Resumé Abstract The PAC-Bayesian approach is a powerful set of techniques to derive non-asymptotic risk bounds for random estimators. The corresponding optimal distribution of estimators, usually called the Gibbs posterior, is unfortunately intractable. One may sample from it using Markov chain Monte Carlo, but this is often too slow for big datasets. We consider instead variational approximations of the Gibbs posterior, which are fast to compute. We undertake a general study of the properties of such approximations. Our main finding is that such a variational approximation has often the same rate of convergence as the original PAC-Bayesian procedure it approximates. We specialise our results to several learning tasks (classification, ranking, matrix completion), discuss how to implement a variational approximation in each case, and illustrate the good properties of said approximation on real datasets. Abstract SMC 2 (Chopin et al., 2013) is an efficient algorithm for sequential estimation and state inference of state-space models. It generates N θ parameter particles θ m , and, for each θ m , it runs a particle filter of size N x (i.e. at each time step, N x particles are generated in the state space X ). We discuss how to automatically calibrate N x in the course of the algorithm. Our approach relies on conditional Sequential Monte Carlo updates, monitoring the state of the pseudo random number generator and on an estimator of the variance of the unbiased estimate of the likelihood that is produced by the particle filters, which is obtained using nonparametric regression techniques. We observe that our approach is both less CPU intensive and with smaller Monte Carlo errors than the initial version of SMC 2 .

Introduction

The field of Bayesian computation seems hard to track these days, as it is blossoming in many directions. MCMC (Markov chain Monte Carlo) remains the main approach, but it is no longer restricted to Gibbs sampling and Hastings-Metropolis, as it includes more advanced, Physics-inspired methods, such as HMC [Hybrid Monte Carlo, Neal, 2010b] and its variants [START_REF] Girolami | Riemann manifold Langevin and Hamiltonian Monte Carlo methods[END_REF][START_REF] Hoffman | The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian monte carlo[END_REF][START_REF] Shahbaba | Split hamiltonian monte carlo[END_REF]. On the other hand, there is also a growing interest for alternatives to MCMC, such as SMC (Sequential Monte Carlo, e.g. Del Moral et al., 2006a), nested sampling [START_REF] Skilling | Nested sampling for general Bayesian computation[END_REF], or the fast approximations that originated from machine learning, such as Variational Bayes [e.g. Bishop, 2006b, Chap. 10], andEP [Expectation Propagation, Minka, 2001b]. Even Laplace approximation has resurfaced in particular thanks to the INLA methodology [START_REF] Rue | Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations[END_REF].

One thing however that all these approaches have in common is they are almost always illustrated by a binary regression example; see e.g. the aforementioned papers. In other words, binary regressions models, such as probit or logit, are a de facto benchmark for Bayesian computation.

This remark leads to several questions. Are binary regression models a reasonable benchmark for Bayesian computation? Should they be used then to develop a 'benchmark culture' in Bayesian computation, like in e.g. optimisation? And practically, which of these methods actually 'works best' for approximating the posterior distribution of a binary regression model?

The objective of this chapter is to answer these questions. As the ironic title suggests, our findings shall lead to us be critical of certain current practices. Specifically, most papers seem content with comparing some new algorithm with Gibbs sampling, on a few small datasets, such as the well-known Pima Indians diabetes dataset (8 covariates). But we shall see that, for such datasets, approaches that are even more basic than Gibbs sampling are actually hard to beat. In other words, datasets considered in the literature may be too toy-like to be used as a relevant benchmark. On the other hand, if ones considers larger datasets (with say 100 covariates), then not so many approaches seem to remain competitive.

We would also like to discuss how Bayesian computation algorithms should be compared. One obvious criterion is the error versus CPU time trade-off; this implies discussing which posterior quantities one may need to approximate. A related point is whether the considered method comes with a simple way to evaluate the numerical error. Other criteria of interest are: (a) how easy to implement is the considered method? (b) how generic is it? (does changing the prior or the link function requires a complete rewrite of the source code?) (c) to which extent does it require manual tuning to obtain good performance? (d) is it amenable to parallelisation? Points (a) and (b) are rarely discussed in Statistics, but relate to the important fact that, the simpler the program, the easier it is to maintain, and to make it bug-free. Regarding point (c), we warn beforehand that, as a matter of principle, we shall refuse to manually tune an algorithm on a per dataset basis. Rather, we will discuss, for each approach, some (hopefully reasonable) general recipe for how to choose the tuning parameters. This has two motivations. First, human time is far more valuable than computer time: [START_REF] Cook | Time exchange rate. The Endeavour[END_REF] mentions that one hour of CPU time is today three orders of magnitude less expensive than one hour of pay for a programmer (or similarly a scientist). Second, any method requiring too much manual tuning through trial and error may be practically of no use beyond a small number of experts.

Finally, we also hope this chapter may serve as an up to date review of the state of Bayesian computation. We believe this review to be timely for a number of reasons. First, as already mentioned, because Bayesian computation seems to develop currently in several different directions. Second, and this relates to criterion (d), the current interest in parallel computation [START_REF] Lee | On the utility of graphics cards to perform massively parallel simulation of advanced Monte Carlo methods[END_REF][START_REF] Suchard | Understanding GPU programming for statistical computation: Studies in massively parallel massive mixtures[END_REF] may require a re-assessment of Bayesian computational methods: method A may perform better than method B on a single core architecture, while performing much worse on a parallel architecture. Finally, although the phrase 'big data' seems to be a tired trope already, it is certainly true that datasets are getting bigger and bigger, which in return means that statistical methods needs to be evaluated on bigger and bigger datasets. To be fair, we will not really consider in this work the kind of huge datasets that pertain to 'big data', but we will at least strive to move away from the kind of 'ridiculously small' data encountered too often in Bayesian computation papers.

The chapter is structured as follows. Section 2.2 covers certain useful preliminaries on binary regression models. Section 2.3 discusses fast approximations, that is, deterministic algorithms that offer an approximation of the posterior, at a lower cost than sampling-based methods. Section 2.4 discusses 'exact', sampling-based methods. Section 2.5 is the most important part of the chapter, as it contains an extensive numerical comparison of all these methods. Section 2.6 discusses variable selection. Section 2.7 discusses our findings, and their implications for both end users and Bayesian computation experts.

Preliminaries: binary regression models

Likelihood, prior

The likelihood of a binary regression model have the generic expression

p(D|β) = n D i=1 F (y i β T x i ) (2.1)
where the data D consist of n responses y i ∈ {-1, 1} and n vectors x i of p covariates, and F is some CDF (cumulative distribution function) that transforms the linear form y i β T x i into a probability. Taking F = Φ, the standard normal CDF, gives the probit model, while taking F = L, the logistic CDF, L(x) = 1/ (1 + e -x ), leads to the logistic model. Other choices could be considered, such as e.g. the CDF of a Student distribution (robit model) to better accommodate outliers. We follow [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF]'s recommendation to standardise the predictors in a preliminary step: non-binary predictors have mean 0 and standard deviation 0.5, binary predictors have mean 0 and range 1, and the intercept (if present) is set to 1. This standardisation facilitates prior specification: one then may set up a "weakly informative" prior for β, that is a proper prior that assigns a low probability that the marginal effect of one predictor is outside a reasonable range. Specifically, we shall consider two priors p(β) in this work: (a) the default prior recommended by [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF], a product of independent Cauchys with centre 0 and scale 10 for the constant predictor, 2.5 for all the other predictors (henceforth, the Cauchy prior); and (b) a product of independent Gaussians with mean 0 and standard deviation equal to twice the scale of the Cauchy prior (henceforth the Gaussian prior).

Of course, other priors could be considered, such as e.g. Jeffreys' prior [START_REF] Firth | Bias reduction of maximum likelihood estimates[END_REF], or a Laplace prior [START_REF] Kabán | On bayesian classification with laplace priors[END_REF]. Our main point in considering the two priors above is to determine to which extent certain Bayesian computation methods may be prior-dependent, either in their implementation (e.g. Gibbs sampling) or in their performance, or both. In particular, one may expect the Cauchy prior to be more difficult to deal with, given its heavy tails.

Posterior maximisation (Gaussian prior)

We explain in this section how to quickly compute the mode, and the Hessian at the mode, of the posterior:

p(β|D) = p(β)p(D|β) p(D) , p(D) = ˆRd p(β)p(D|β) dβ,
where p(β) is one of the two priors presented in the previous section, and Z(D) is the marginal likelihood of the data (also known as the evidence). These quantities will prove useful later, in particular to tune certain of the considered methods.

The two first derivatives of the log-posterior density may be computed as:

∂ ∂β log p(β|D) = ∂ ∂β log p(β) + ∂ ∂β log p(D|β), ∂ 2 ∂β∂β T log p(β|D) = ∂ 2 ∂β∂β T log p(β) + ∂ 2 ∂β∂β T log p(D|β)
where

∂ ∂β log p(D|β) = n D i=1 (log F ) ′ (y i β T x i )y i x i ∂ 2 ∂β∂β T log p(D|β) = n D i=1 (log F ) ′′ (y i β T x i )x i x T i
and (log F ) ′ and (log F ) ′′ are the two first derivatives of log F . Provided that log F is concave, which is the case for probit and logit regressions, the Hessian of the log-likelihood is clearly a negative definite matrix. Moreover, if we consider the Gaussian prior, then the Hessian is of the log-posterior is also negative (as the sum of two negative matrices, as Gaussian densities are log-concave). We stick to the Gaussian prior for now.

This suggests the following standard approach to compute the MAP (maximum a posterior) estimator, that is the point β MAP that maximises the posterior density p(β|D): to use Newton-Raphson, that is, to iterate

β (new) = β (old) -H -1 ∂ ∂β log p(β (old) |D) (2.2)
until convergence is reached; here H is Hessian of the log posterior at β = β (old) , as computed above. The iteration above corresponds to finding the zero of a local, quadratic approximation of the log-posterior. Newton-Raphson typically works very well (converges in a small number of iterations) when the function to maximise is concave. We note two points in passing. First, one may obtain the MLE (maximum likelihood estimator) by simply taking p(β) = 1 above (i.e. a Gaussian with infinite variance). But the MLE is not properly defined when complete separation occurs, that is, there exists a hyperplane that separates perfectly the two outcomes: y i β T CS x i ≥ 0 for some β CS and all i ∈ 1 : N . This remark gives an extra incentive for performing Bayesian inference, or at least MAP estimation, in cases where complete separation may occur, in particular when the number of covariates is large [START_REF] Firth | Bias reduction of maximum likelihood estimates[END_REF][START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF].

Second, H in (2.2) is sometimes replaced by some approximation, leading to socalled quasi-Newton algorithms. Some of these algorithms such as IRLS (iterated reweighted least squares) have a nice statistical interpretation. For our purposes however, all these variants seem to show similar performance, so we will stick to the standard version of Newton-Raphson.

Posterior maximisation (Cauchy prior)

The log-density of the Cauchy prior is not concave:

log p(β) = - p j=1 log (πσ j ) - p j=1 log(1 + β 2 j /σ 2 j )
for scales σ j chosen as explained in Section 2.2.1. Hence, the corresponding logposterior is no longer guaranteed to be concave, which in turn means that Newton-Raphson algorithm might fail to converge. However, we shall observe that, for most of the datasets considered in this chapter, Newton-Raphson does converge quickly even for our Cauchy prior. In each case, we used as starting point for the Newton-Raphson iterations the OLS (ordinary least square) estimate. We suspect what happens is that, for most standard datasets, the posterior derived from a Cauchy prior remains log-concave, at least in a region that encloses the MAP estimator and our starting point.

Fast approximation methods

This section discusses fast approximation methods, that is methods that are deterministic, fast (compared to sampling-based methods), but which comes with an approximation error which is difficult to assess. These methods include the Laplace approximation, which was popular in Statistics before the advent of MCMC methods, but also recent Machine Learning methods, such as EP (Expectation Propagation, Minka, 2001b), and VB (Variational Bayes, e.g. Bishop, 2006b, Chap. 10). However, we will not discuss VB, as [START_REF] Consonni | Mean-field variational approximate Bayesian inference for latent variable models[END_REF] give convincing (formal and numerical) arguments that this type of approach does not work well for probit models.

Concretely, we will focus on the approximation of the following posterior quantities: the marginal likelihood p(D), as this may be used in model choice; and the marginal distributions p(β i |D) for each component β i of β. Clearly these are the most commonly used summaries of the posterior distribution, and other quantities, such as the posterior expectation of β, may be directly deduced from them.

Finally, one should bear in mind that such fast approximations may be used as a preliminary step to calibrate an exact, more expensive method, such as those described in Section 2.4.

Laplace approximation

The Laplace approximation is based on a Taylor expansion of the posterior logdensity around the mode β MAP :

log p(β|D) ≈ log p(β MAP |D) - 1 2 (β -β MAP ) T Q (β -β MAP ) ,
where Q = -H, i.e. minus the Hessian of log p(β|D) at β = β MAP ; recall that we explained how to compute these quantities in Section 2.2.2. One may deduce a Gaussian approximation of the posterior by simply exponentiating the equation above, and normalising:

q L (β) = N p β; β MAP , Q -1 := (2π) -p/2 |Q| 1/2 exp - 1 2 (β -β MAP ) T Q (β -β MAP ) . (2.3)
In addition, since for any β,

p(D) = p(β)p(D|β) p(β|D)
one obtains an approximation to the marginal likelihood p(D) as follows:

p(D) ≈ Z L (D) := p(β MAP )p(D|β MAP ) (2π) -p/2 |Q| 1/2 .
From now on, we will refer to this particular Gaussian approximation q L as the Laplace approximation, even if this phrase is sometimes used in Statistics for higher-order approximations, as discussed in the next Section. We defer to Section 2.3.5 the discussion of the advantages and drawbacks of this approximation scheme.

Improved Laplace, connection with INLA

Consider the marginal distributions p(β j |D) = ´p(β|D)dβ -j for each component β j of β, where β -j is β minus β j . A first approximation may be obtained by simply computing the marginals of the Laplace approximation q L . An improved (but more expensive) approximation may be obtained from:

p(β j |D) ∝ p(β)p(D|β) p(β -j |β j , D)
which suggests to choose a fine grid of β j values (deduced for instance from q L (β)), and for each β j value, compute a Laplace approximation of p(β -j |β j , D), by computing the mode β-j (β j ) and the Hessian Ĥ(β j ) of log p(β -j |β j , D), and then approximate (up to a constant)

p(β j |D) ≈ q IL (β j ) ∝ p β(β j ) p(D| β(β j )) Ĥ(β j ) 1/2
where β(β j ) is the vector obtained by inserting β i at position i in β-j (β j ), and IL stands for "Improved Laplace". One may also deduce posterior expectations of functions of β j in this way. See also [START_REF] Tierney | Accurate approximations for posterior moments and marginal densities[END_REF], [START_REF] Tierney | Fully exponential Laplace approximations to expectations and variances of non-positive functions[END_REF] for higher order approximations for posterior expectations. We note in passing the connection to the INLA scheme of [START_REF] Rue | Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations[END_REF]. INLA applies to posteriors p(θ, x|D) where x is a latent variable such that p(x|θ, D) is close to a Gaussian, and θ is a low-dimensional hyper-parameter. It constructs a grid of θ-values, and for each grid point θ j , it computes an improve Laplace approximation of the marginals of p(x|θ j , D). In our context, β may be identified to x, θ to an empty set, and INLA reduces to the improved Laplace approximation described above. [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF] (Cauchy prior) [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF] recommend against the Laplace approximation for a Student prior (of which our Cauchy prior is a special case), because, as explained in Section 2.2.3, the corresponding log-posterior is not guaranteed to be concave, and this might prevent Newton-Raphson to converge. In our simulations however, we found the Laplace approximation to work reasonably well for a Cauchy prior. We now briefly describe the alternative approximation scheme proposed by [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF] for Student priors, which we call for convenience Laplace-EM.

The EM algorithm of

Laplace-EM is based on the well-known representation of a Student distribution, β j |σ 2 j ∼ N 1 (0, σ 2 j ), σ 2 j ∼ Inv -Gamma(ν/2, s j ν/2); take ν = 1 to recover our Cauchy prior. Conditional on σ 2 = (σ 2 1 , . . . , σ 2 p ), the prior on β is Gaussian, hence, for a fixed σ 2 one may implement Newton-Raphson to maximise the logdensity of p(β|σ 2 , D), and deduce a Laplace (Gaussian) approximation of the same distribution.

Laplace-EM is an approximate EM [Expectation Maximisation, [START_REF] Dempster | Maximum likelihood from incomplete data via the em algorithm[END_REF] algorithm, which aims at maximising in σ 2 = (σ 2 1 , . . . , σ 2 p ) the marginal posterior distribution p(σ 2 |D) = ´p(σ 2 , β|D) dβ. Each iteration involves an expectation with respect to the intractable conditional distribution p(β|σ 2 , D), which is Laplace approximated, using a single Newton-Raphson iteration. When this approximate EM algorithm has converged to some value σ 2 ⋆ , one more Newton-Raphson iteration is performed to compute a final Laplace approximation of p(β|σ 2 ⋆ , D), which is then reported as a Gaussian approximation to the posterior. We refer the readers to [START_REF] Gelman | A weakly informative default prior distribution for logistic and other regression models[END_REF] for more details on Laplace-EM.

Expectation-Propagation

Like Laplace, Expectation Propagation [EP, Minka, 2001b] generates a Gaussian approximation of the posterior, but it is based on different ideas. The consensus in machine learning seems to be that EP provides a better approximation than Laplace [e.g. [START_REF] Nickisch | Approximations for Binary Gaussian Process Classification[END_REF]; the intuition being that Laplace is 'too local' (i.e. it fitted so at to match closely the posterior around the mode), while EP is able to provide a global approximation to the posterior.

Starting from the decomposition of the posterior as product of (n D + 1) factors:

p(β|D) = 1 p(D) n D i=0 l i (β), l i (β) = F (y i β T x i ) for i ≥ 1,
and l 0 is the prior, l 0 (β) = p(β), EP computes iteratively a parametric approxi-mation of the posterior with the same structure

q EP (β) = n D i=0 1 Z i q i (β).
(2.4)

Taking q i to be an unnormalised Gaussian densities written in natural exponential form

q i (β) = exp - 1 2 β T Q i β + β T r i ,
one obtains for q EP a Gaussian with natural parameters Q = n i=0 Q i and r i = n i=0 r i ; note that the more standard parametrisation of Gaussians may be recovered by taking

Σ = Q -1 , µ = Q -1 r.
Other exponential families could be considered for q and the q i 's, see e.g. Seeger [2005b], but Gaussian approximations seems the most natural choice here.

An EP iteration consists in updating one factor q i , or equivalently (Z i , Q i , r i ), while keeping the other factors as fixed, by moment matching between the hybrid distribution h(β) ∝ l i (β)

j =i q j (β)
and the global approximation q defined in (2.4): compute

Z h = ˆli (β) j =i q j (β) dβ µ h = 1 Z h ˆβl i (β) j =i q j (β) dβ Σ h = 1 Z h ˆββ T l i (β) j =i q j (β) dβ
and set

Q i = Σ -1 h -Q -i , r i = Σ -1 h µ h -r -i , log Z i = log Z h -Ψ(r, Q) + Ψ(r -i , Q -i )
where

r -i = j =i r j , Q -i = j =i Q j , and ψ(r, Q) is the normalising constant of a Gaussian distribution with natural parameters (r, Q), ψ(r, Q) = ˆRp exp - 1 2 β T Qβ + β T r dβ = - 1 2 log |Q/2π| + 1 2 r T Qr.
In practice, EP proceeds by looping over sites, updating each one in turn until convergence is achieved.

To implement EP for binary regression models, two points must be addressed. First, how to compute the hybrid moments? For the probit model, these moments may be computed exactly, see the supplement to the paper, while for the other links function (such as logistic), numerical (one-dimensional) quadrature may be used. Second, how to deal with the prior? If the prior is Gaussian, one may simply set q 0 to the prior, and never update q 0 in the course of the algorithm. For a Cauchy prior, q 0 is simply treated as an extra site.

EP being a fairly recent method, it is currently lacking in terms of supporting theory, both in terms of algorithmic convergence (does it converge in a finite number of iterations?), and statistical convergence (does the resulting approximation converges in some sense to the true posterior distribution as n D → +∞?). On the other hand, there is mounting evidence that EP works very well in many problems; again see e.g. Nickisch and Rasmussen [e.g. 2008].

Discussion of the different approximation schemes

Laplace (and its variants) have complexity O(n D + p 3 ), while EP has complexity O(n D p 3 ). Incidentally, one sees that the number of covariates p is more critical than the number of instances n D in determining how 'big' (how time-intensive to process) is a given dataset. This will be a recurring point in this chapter.

The p 3 term in both complexities is due to the p×p matrix operations performed by both algorithms; e.g. the Newton-Raphson update (2.2) requires solving a linear system of order p. EP requires to perform such p 3 operations at each site (i.e. for each single observation), hence the O(n D p 3 ) complexity, while Laplace perform such operations only once per iteration. EP is therefore expected to be more expensive than Laplace.

This remark may be mitigated as follows. First, one may modify EP so as to update the global approximation only at the end of each iteration (complete pass over the data). The resulting algorithm [START_REF] Van Gerven | Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior[END_REF]] may be easily implemented on parallel hardware: simply distribute the n D factors over the processors. Even without parallelisation, parallel EP requires only one single matrix inversion per iteration.

Second, the 'improved Laplace' approximation for the marginals described in Section 2.3.1 requires to perform quite a few basic Laplace approximations, so its speed advantage compared to standard EP essentially vanishes.

Points that remain in favour of Laplace is that it is simpler to implement than EP, and the resulting code is very generic: adapting to either a different prior, or a different link function (choice of F in 2.1), is simply a matter of writing a function that evaluates the corresponding function. We have seen that such an adaptation requires more work in EP, although to be fair the general structure of the algorithm is not model-dependent. On the other hand, we shall see that EP is often more accurate, and works in more examples, than Laplace; this is especially the case for the Cauchy prior.

Exact methods

We now turn to sampling-based methods, which are 'exact', at least in the limit: one may make the approximation error as small as desired, by running the corresponding algorithm for long enough. We will see that all of these algorithms requires some form of calibration that requires prior knowledge on the shape of the posterior distribution. Since the approximation methods covered in the previous section are faster by orders of magnitude than sampling-based methods, we will assume that a Gaussian approximation q(β) (say, obtained by Laplace or EP) has been computed in a preliminary step.

Our gold standard: Importance sampling

Let q(β) denote a generic approximation of the posterior p(β|D). Importance sampling (IS) is based on the trivial identity

p(D) = ˆp(β)p(D|β) dβ = ˆq(β) p(β)p(D|β) q(β) dβ
which leads to the following recipe: sample β 1 , . . . , β N ∼ q, then compute as an estimator of p(D)

Z N = 1 N N n=1 w(β n ), w(β) := p(β)p(D|β) q(β) .
(2.5)

In addition, since

ˆϕ(β)p(β|D) dβ = ´ϕ(β)q(β)w(β) dβ ´q(β)w(β) dβ
one may approximate any posterior moment as

ϕ N = N n=1 w(β n )ϕ(β n ) N n=1 w(β n ) . (2.6)
Approximating posterior marginals is also straightforward; one may instance use kernel density estimation on the weighted sample (β n , w(β n )) N n=1 .

Concerning the choice of q, we will restrict ourselves to the Gaussian approximations generated either from Laplace or EP algorithm. It is sometimes recommended to use a Student distribution instead, as a way to ensure that the variance of the above estimators is finite, but we did not observe any benefit for doing so in our simulations.

It is of course a bit provocative to call IS our gold standard, as it is sometimes perceived as an obsolete method. We would like to stress out however that IS is hard to beat relative to most of the criteria laid out in the introduction:

• because it is based on IID sampling, assessing the Monte Carlo error of the above estimators is trivial: e.g. the variance of Z N may be estimated as N -1 times the empirical variance of the weights w(β n ). The auto-normalised estimator 2.6 has asymptotic variance

E q w(β) 2 {ϕ(β) -µ(ϕ)} 2 , µ(ϕ) = ˆϕ(β)p(β|D) dβ
which is also trivial to approximate from the simulated β n 's.

• Other advantages brought by IID sampling are: (a) importance sampling is easy to parallelize; and (b) importance sampling is amenable to QMC (Quasi-Monte Carlo) integration, as explained in the following section.

• Importance sampling offers an approximation of the marginal likelihood p(D) at no extra cost.

• Code is simple and generic.

Of course, what remains to determine is whether importance sampling does well relative to our main criterion, i.e. error versus CPU trade-off. We do know that IS suffers from a curse of dimensionality: take both q and and the target density π to be the density of IID distributions: q(β) = p j=1 q 1 (β j ), π(β) = p j=1 π 1 (β j ); then it is easy to see that the variance of the weights grows exponentially with p. Thus we expect IS to collapse when p is too large; meaning that a large proportion of the β n gets a negligible weight. On the other hand, for small to moderate dimensions, we will observe surprising good results; see Section 2.5. We will also present below a SMC algorithm that automatically reduces to IS when IS performs well, while doing something more elaborate in more difficult scenarios.

The standard way to assess the weight degeneracy is to compute the effective sample size [START_REF] Kong | Sequential imputations and bayesian missing data problems[END_REF],

ESS = N n=1 w(β n ) 2 N n=1 w(β n ) 2 ∈ [1, N ],
which roughly approximates how many simulations from the target distribution would be required to produce the same level of error. In our simulations, we will compute instead the efficiency factor EF, which is simply the ratio EF = ESS/N .

Improving importance sampling by Quasi-Monte Carlo

Quasi-Monte Carlo may be seen as an elaborate variance reduction technique: starting from the Monte Carlo estimators Z N and ϕ N , see (2.5) and (2.6), one may re-express the simulated vectors as functions of uniform variates u n in [0, 1] d ; for instance:

β n = µ + Cζ n , ζ n = Φ -1 (u n )
where Φ -1 is Φ -1 , the N (0, 1) inverse CDF, applied component-wise. Then, one replaces the N vectors u n by a low-discrepancy sequence; that is a sequence of N vectors that spread more evenly over [0, 1] d ; e.g. a Halton or a Sobol' sequence. Under appropriate conditions, QMC error converges at rate O(N -1+ǫ ), for any ǫ > 0, to be compared with the standard Monte Carlo rate O P (N -1/2 ). We refer to [START_REF] Lemieux | Monte Carlo and Quasi-Monte Carlo Sampling[END_REF] for more background on QMC, as well as how to construct QMC sequences.

Oddly enough, the possibility to use QMC in conjunction with importance sampling is very rarely mentioned in the literature; see however [START_REF] Hörmann | Quasi importance sampling[END_REF]. More generally, QMC seems often overlooked in Statistics. We shall see however that this simple IS-QMC strategy often performs very well.

One drawback of IS-QMC is that we lose the ability to evaluate the approximation error in a simple manner. A partial remedy is to use randomised Quasi-Monte Carlo (RQMC), that is, the u n are generated in such a way that (a) with probability one, u 1:N is a QMC point set; and (b) each vector u n is marginally sampled from [0, 1] d . Then QMC estimators that are empirical averages, such as Z N = N -1 N n=1 w(β n ) become unbiased estimators, and their error may be assessed through the empirical variance over repeated runs. Technically, estimators that are ratios of QMC averages, such as ϕ N , are not unbiased, but for all practical purposes their bias is small enough that assessing error through empirical variances over repeated runs remains a reasonable approach.

MCMC

The general principle of MCMC (Markov chain Monte Carlo) is to simulate a Markov chain that leaves invariant the posterior distribution p(β|D); see Robert and Casella [2004a] for a general overview. Often mentioned drawbacks of MCMC simulation are (a) the difficulty to parallelize such algorithms (although see e.g. [START_REF] Jacob | Using parallel computation to improve independent metropolis-hastings based estimation[END_REF] for an attempt at this problem); (b) the need to specify a good starting point for the chain (or alternatively to determine the burn-in period, that is, the length of the initial part of the chain that should be discarded) and (c) the difficulty to assess the convergence of the chain (that is, to determine if the distribution of β t at iteration t is sufficiently close to the invariant distribution p(β|D)).

To be fair, these problems are not so critical for binary regression models. Regarding (b), one may simply start the chain from the posterior mode, or from a draw of one of the Gaussian approximations covered in the previous section. Regarding (c) for most standard datasets, MCMC converges reasonably fast, and convergence is easy to assess visually. The main issue in practice is that MCMC generates correlated random variables, and these correlations inflate the Monte Carlo variance.

Gibbs sampling

Consider the following data-augmentation formulation of binary regression:

z i = β T x i + ǫ i y i = sgn(z i )
where z = (z 1 , . . . , z n D ) T is a vector of latent variables, and assume for a start that ǫ i ∼ N (0, 1) (probit regression). One recognises p(β|z, D) as the posterior of a linear regression model, which is tractable (for an appropriate prior). This suggests to sample from p(β, z|D) using Gibbs sampling [START_REF] Albert | Bayesian analysis of binary and polychotomous response data[END_REF]: i.e. iterate the two following steps: (a) sample from z|β, D; and (b) sample from β|z, D.

For (a), the z i 's are conditionally independent, and follows a truncated Gaussian distribution

p(z i |β, D) ∝ N 1 z i ; β T x i , 1 ✶ {z i y i > 0}
which is easy to sample from [Chopin, 2011b]. For Step (b) and a Gaussian prior N p (0, Σ prior ), one has, thanks to standard conjugacy properties:

β|z, D ∼ N p (µ post (z), Σ post ) , Σ -1 post = Σ -1 prior + xx T , µ post (z) = Σ -1 post xz
where x is the n × p matrix obtained by stacking the x T i . Note that Σ post and its inverse need to be computed only once, hence the complexity of a Gibbs iteration is O(p 2 ), not O(p 3 ).

The main drawback of Gibbs sampling is that it is particularly not generic: its implementation depends very strongly on the prior and the model. Sticking to the probit case, switching to another prior requires deriving a new way to update β|z, D. For instance, for a prior which is a product of Students with scales σ j (e.g. our Cauchy prior), one may add extra latent variables, by resorting to the well-known representation: β j |s j ∼ N 1 (0, νσ 2 j /s j ), s j ∼ Chi 2 (ν); with ν = 1 for our Cauchy prior. Then the algorithm have three steps: (a) an update of the z i 's, exactly as above; (b) an update of β, as above but with Σ prior replaced by the diagonal matrix with elements νσ 2 j /s j , j = 1, . . . , p; and (c) an (independent) update of the p latent variables s j , with s j |β, z, D ∼ Gamma (1 + ν)/2, 1 + νβ 2 j /σ 2 j /2 . The complexity of Step (b) is now O(p 3 ), since Σ prior and Σ post must be recomputed at each iteration.

Of course, considering yet another type of prior would require deriving another strategy for sampling β. Then if one turns to logistic regression, things get rather complicated. In fact, deriving an efficient Gibbs sampler for logistic regression is a topic of current research; see [START_REF] Frühwirth | Data augmentation and mcmc for binary and multinomial logit models[END_REF]; [START_REF] Gramacy | Simulation-based regularized logistic regression[END_REF]; [START_REF] Holmes | Bayesian auxiliary variable models for binary and multinomial regression[END_REF]; [START_REF] Nicholas | Bayesian inference for logistic models using pólya-gamma latent variables[END_REF]. In a nutshell, the two first papers use the same data augmentation as above, but with ǫ i ∼ Logistic(1) written as a certain mixture of Gaussians (infinite for the first paper, finite but approximate for the second paper), while [START_REF] Nicholas | Bayesian inference for logistic models using pólya-gamma latent variables[END_REF] use instead a representation of a logistic likelihood as an infinite mixture of Gaussians, with a Polya-Gamma as the mixing distribution. Each representation leads to introducing extra latent variables, and discussing how to sample their conditional distributions.

Since their implementation is so model-dependent, the main justification for Gibbs samplers should be their greater performance relative to more generic algorithms. We will investigate if this is indeed the case in our numerical section.

Hastings-Metropolis

Hastings-Metropolis consists in iterating the step described as Algorithm 6. Much like importance sampling, Hastings-Metropolis is both simple and generic, that is, up to the choice of the proposal kernel κ(β ⋆ |β) (the distribution of the proposed point β ⋆ , given the current point β). A naive approach is to take κ(β ⋆ |β) independent of β, κ(β ⋆ |β) = q(β ⋆ ), where q is some approximation of the posterior. In practice, this usually does not work better than importance sampling based on the same proposal, hence this strategy is hardly used.

A more usual strategy is to set the proposal kernel to a random walk:

κ(β ⋆ |β) = N p (β, Σ prop ).
It is well known that the choice of Σ prop is critical for good performance. For instance, in the univariate case, if Σ prop is too small, the chain moves slowly, while if too large, proposed moves are rarely accepted.

A result from the optimal scaling literature [e.g. [START_REF] Gareth | Optimal Scaling for Various Metropolis-Hastings Algorithms[END_REF]] is that, for a N p (0, I p ) target, Σ prop = (λ 2 /p)I p with λ = 2.38 is asymptotically optimal, in the sense that as p → ∞, this choice leads to the fastest exploration. Since the posterior of a binary regression model is reasonably close to a Gaussian,

Algorithm 6 Hastings-Metropolis iteration Input β Output β ′ 1 Sample β ⋆ ∼ κ(β ⋆ |β). 2 With probability 1 ∧ r, r = p(β ⋆ )p(D|β ⋆ )κ(β|β ⋆ ) p(β)p(D|β)κ(β ⋆ |β) , set β ′ = β ⋆ ; otherwise set β ′ = β.
we adapt this result by taking Σ prop = (λ 2 /p)Σ q in our simulations, where Σ q is the covariance matrix of a (Laplace or EP) Gaussian approximation of the posterior. This strategy seems validated by the fact we obtain acceptance rates close to the optimal rate, as given by [START_REF] Gareth | Optimal Scaling for Various Metropolis-Hastings Algorithms[END_REF].

The bad news behind this optimality result is that the chain requires O(p) steps to move a O(1) distance. Thus random walk exploration tends to become slow for large p. This is usually cited as the main motivation to develop more elaborate MCMC strategies, such as HMC, which we cover in the following section.

HMC

Hamiltonian Monte Carlo (HMC, also known as Hybrid Monte Carlo, [START_REF] Duane | Hybrid monte carlo[END_REF] is a new type of MCMC algorithm, where one is able to perform several steps in the parameter space before determining if the new position is accepted or not. Consequently, HMC is able to make much bigger jumps in the parameter space than standard Metropolis algorithms. See Neal [2010b] for an excellent introduction.

Consider the pair (β, α), where β ∼ p(β|D), and α ∼ N p (0, M -1 ), thus with joint un-normalised density exp {-H(β, α)}, with

H(β, α) = E(β) + 1 2 α T M α, E(β) = -log {p(β)p(D|β)} .
The physical interpretation of HMC is that of a particle at position β, with velocity α, potential energy E(β), kinetic energy 1 2 α T M α, for some mass matrix M , and therefore total energy given by H(β, α). The particle is expected to follow a trajectory such that H(β, α) remains constant over time.

In practice, HMC proceeds as follows: first, sample a new velocity vector, α ∼ N p (0, M -1 ). Second, move the particle while keeping the Hamiltonian H constant; in practice, discretisation must be used, so L steps of step-size ǫ are performed through leap-frop steps; see Algorithm 7 which describes one such step. Third, the new position, obtained after L leap-frog steps is accepted or rejected according to probability 1 ∧ exp {H(β, α) -H(β ⋆ , α ⋆ )}; see Algorithm 8 for a summary. The validity of the algorithm relies on the fact that a leap-frog step is "volume preserving"; that is, the deterministic transformation (β, α) → (β 1 , α 1 ) has Jacobian one. This is why the acceptance probability admits this simple expression.

Algorithm 7 Leap-frog step Input (β, α) Output (β 1 , α 1 ) 1 α 1/2 ← α -ǫ 2 ∇ β E(β) 2 β 1 ← β + ǫα 1/2 3 α 1 ← α 1/2 -ǫ 2 ∇ β E(β 1 ) Algorithm 8 HMC iteration Input β Output β ′ 1 Sample momentum α ∼ N p (0, M ).
2 Perform L leap-frog steps (see Algorithm 7), starting from (β, α); call (β ⋆ , α ⋆ ) the final position.

3 With probability 1 ∧ r, r = exp {H(β, α) -H(β ⋆ , α ⋆ )} set β ′ = β ⋆ ; otherwise set β ′ = β.
The tuning parameters of HMC are M (the mass matrix), L (number of leapfrog steps), and ǫ (the stepsize). For M , we follow Neal [2010b]'s recommendation and take M -1 = Σ q , an approximation of the posterior variance (again obtained from either Laplace or EP). This is equivalent to rescaling the posterior so as to have a covariance matrix close to identity. In this way, we avoid the bad mixing typically incurred by strong correlations between components.

The difficulty to choose L and ǫ seems to be the main drawback of HMC. The performance of HMC seems very sensitive to these tuning parameters, yet clear guidelines on how to choose them seem currently lacking. A popular approach is to fix Lǫ to some value, and to use vanishing adaptation [START_REF] Andrieu | A tutorial on adaptive MCMC[END_REF] to adapt ǫ so as to target acceptance rate of 0.65 (the optimal rate according to the formal study of HMC by [START_REF] Beskos | Optimal tuning of the hybrid monte carlo algorithm[END_REF]

: i.e. at iteration t, take ǫ = ǫ t , with ǫ t = ǫ t-1 -η t (R t -0.65), η t = t -κ , κ ∈ (1/2, 1
) and R t the acceptance rate up to iteration t. The rationale for fixing Lǫ is that quantity may be interpreted as a 'simulation length', i.e. how much distance one moves at each step; if too small, the algorithm may exhibit random walk behaviour, while if too large, it may move a long distance before coming back close to its starting point. Since the spread of is already taken into account through M -1 = Σ q , we took ǫL = 1 in our simulations.

NUTS and other variants of HMC [START_REF] Girolami | Riemann manifold Langevin and Hamiltonian Monte Carlo methods[END_REF] proposed an interesting variation of HMC, where the mass matrix M is allowed to depends on β; e.g. M (β) is set to the Fisher information of the model. This allows the corresponding algorithm, called RHMC (Riemanian HMC), to adapt locally to the geometry of the target distribution. The main drawback of RHMC is that each iteration involves computing derivatives of M (β) with respect to β, which is very expensive, especially if p is large. For binary regression, we found RMHC to be too expensive relative to plain HMC, even when taking into account the better exploration brought by RHMC. This might be related to the fact that the posterior of a binary regression model is rather Gaussian-like and thus may not require such a local adaptation of the sampler.

We now focus on NUTS [No U-Turn sampler, Hoffman and Gelman, 2013], a variant of HMC which does not require to specify a priori L, the number of leapfrog steps. Instead, NUTS aims at keeping on doing such steps until the trajectory starts to loop back to its initial position. Of course, the difficulty in this exercise is to preserve the time reversibility of the simulated Markov chain. To that effect, NUTS constructs iteratively a binary tree whose leaves correspond to different velocity-position pairs (α, β) obtained after a certain number of leap-frog steps. The tree starts with two leaves, one at the current velocity-position pair, and another leaf that corresponds to one leap-frop step, either in the forward or backward direction (i.e. by reversing the sign of velocity); then it iteratively doubles the number of leaves, by taking twice more leap frog steps, again either in the forward or backward direction. The tree stops growing when at least one leaf corresponds to a "U-turn"; then NUTS chooses randomly one leaf, among those leaves that would have generated the current position with the same binary tree mechanism; in this way reversibility is preserved. Finally NUTS moves the new position that corresponds to the chosen leaf.

We refer the readers to [START_REF] Hoffman | The no-U-turn sampler: Adaptively setting path lengths in Hamiltonian monte carlo[END_REF] for a more precise description of NUTS. Given its complexity, implementing directly NUTS seems to require more efforts than the other algorithms covered in this chapter. Fortunately, the STAN package (http://mc-stan.org/) provides a C++ implementation of NUTS which is both efficient and user-friendly: the only required input is a description of the model in a probabilistic programming language similar to BUGS. In particular, STAN is able to automatically derive the log-likelihood and its gradient, and no tuning of any sort is required from the user. Thus, we will use STAN to assess NUTS in our numerical comparisons.

Sequential Monte Carlo

Sequential Monte Carlo (SMC) is a class of algorithms for approximating iteratively a sequence of distributions π t , t = 0, . . . , T , using importance sampling, resampling, and MCMC steps. We focus here on the non-sequential use of SMC [Chopin, 2002b;Del Moral et al., 2006a;Neal, 2001b], where one is only interested in approximating the final distribution π T (in our case, set to the posterior p(β|D)), and the previous π t 's are designed so as to allow for a smooth progression from some π 0 , which is easy to sample from, to π T .

At iteration t, SMC produces a set of weighted particles (simulations) (β n , w n ) N n=1 that approximates π t , in the sense that

1 N n=1 w n N n=1 w n ϕ(β n ) → E πt [ϕ(β)]
as N → +∞. At time 0, one samples β n ∼ π 0 , and set w n = 1. To progress from π t-1 to π t , one uses importance sampling: weights are multiplied by ratio π t (β n )/π t-1 (β n ). When the variance of the weights gets too large (which indicates that too few particles contribute significantly to the current approximation), one resamples the particles: each particle gets reproduced O n times, where O n ≥ 0 is random, and such that E(O n ) = N w n / N m=1 w m , and

N n=1
O n = N with probability one. In this way, particles with a low weights are likely to die, while particles with a large weight get reproduced many times. Finally, one may reintroduce diversity among the particles by applying one (or several) MCMC steps, using a MCMC kernel that leaves invariant the current distribution π t .

We focus in this chapter on tempering SMC, where the sequence

π t (β) ∝ q(β) 1-δt {p(β)p(D|β)} δt
corresponds to a linear interpolation (on the log-scale) between some distribution π 0 = q, and π T (β) = p(β|D), our posterior. This is a convenient choice in our case, as we have at our disposal some good approximation q (either from Laplace or EP) of our posterior. A second advantage of tempering SMC is that one can automatically adapt the "temperature ladder" δ t [Jasra et al., 2011a]. Algorithm 9 describes a tempering SMC algorithm based on such an adaptation scheme: at each iteration, the next distribution π t is chosen so that the efficiency factor (defined in Section 2.4.1) of the importance sampling step from π t-1 to π t equals a pre-defined level τ ∈ (0, 1); a default value is τ = 1/2.

Algorithm 9 tempering SMC Operations involving index n must be performed for all n ∈ 1 : N .

0 Sample β n ∼ q(β) and set δ ← 0.

1 Let, for δ ∈ [δ, 1], EF(δ) = 1 N N n=1 w γ (β n ) 2 N n=1 w γ (β n ) 2 , u δ (β) = p(β)p(D|β) q(β) δ .
If EF(1) ≥ τ , stop and return (β n , w n ) n=1:N with w n = u 1 (β n ); otherwise, use the bisection method [Press et al., 2007, Chap. 9] to solve numerically in δ the equation EF(γ) = τ.

2 Resample according to normalised weights W n = w n / N m=1 w m , with w n = u δ (β n ); see the supplement for one such resampling algorithm.

3 Update the β n 's through m MCMC steps that leaves invariant π t (β), using e.g. Algorithm 6 with κ(β ⋆ |β) = N p (β, Σ prop ), Σ prop = λ Σ, where Σ is the empirical covariance matrix of the resampled particles.

4 Set δ ← δ. Go to Step 1.
Another part of Algorithm 9 which is easily amenable to automatic calibration is the MCMC step. We use a random walk Metropolis step, i.e. Algorithm 6 with proposal kernel κ(β ⋆ |β) = N p (β, Σ prop ), but with Σ prop calibrated to the empirical variance of the particles Σ: Σ prop = λ Σ, for some λ. Finally, one may also automatically calibrate the number m of MCMC steps, as in Chapter 3, but in our simulations we simply took m = 3.

In the end, one obtains essentially a black-box algorithm. In practice, we shall often observe that, for simple datasets, our SMC algorithm automatically reduces to a single importance sampling step, because the efficiency factor of moving from the initial distribution q to the posterior is high enough. In that case, our SMC sampler performs exactly as standard importance sampling.

Numerical study

The point of this section is to compare numerically the different methods discussed in the previous sections, first on several datasets of standard size (that are representative of previous numerical studies), then in a second time on several bigger datasets.

We focus on the following quantities: the marginal likelihood of the data, p(D), and the p marginal posterior distributions of the regression coefficients β j . Regarding the latter, we follow [START_REF] Faes | Variational Bayesian Inference for Parametric and Nonparametric Regression With Missing Data[END_REF] in defining the 'marginal accuracy' of approximation q for component j to be

MA j = 1 - 1 2 ˆ+∞ -∞ |q(β j ) -p(β j |D)| dβ j .
This quantity lies in [0, 1], and is scale-invariant. Since the true marginals p(β j |D) are not available, we will approximate them through a Gibbs sampler run for a very long time. To give some scale to this criterion, assume q(

β j ) = N 1 (β j ; µ 1 , σ 2 ), p(β j |D) = N 1 (β j ; µ 2 , σ 2 ), then MA j is 2Φ(-δ/2) ≈ 1 -0.4 × δ for δ = |µ 1 -µ 2 |/σ
small enough; e.g. 0.996 for δ ≈ 0.01, 0.96 for δ ≈ 0.1.

In our results, we will refer to the following four prior/model 'scenarios': Gaussian/probit, Gaussian/logit, Cauchy/probit, Cauchy/logit, where Gaussian and Cauchy refer to the two priors discussed in Section 2.2.1. All the algorithms have been implemented in C++, using the Armadillo and Boost libraries, and run on a standard desktop computer (except when explicitly stated). Results for NUTS were obtained by running STAN (http://mc-stan.org/) version 2.4.0.

Datasets of moderate size

Table 2.1 lists the 7 datasets considered in this section (obtained from the UCI machine learning repository, except Elections, which is available on the web page of [START_REF] Gelman | Data analysis using regression and multilevel/hierarchical models[END_REF]'s book). These datasets are representative of the numerical studies found in the literature. In fact, it is a super-set of the real datasets considered in [START_REF] Girolami | Riemann manifold Langevin and Hamiltonian Monte Carlo methods[END_REF], [START_REF] Shahbaba | Split hamiltonian monte carlo[END_REF], [START_REF] Holmes | Bayesian auxiliary variable models for binary and multinomial regression[END_REF] and also (up to one dataset with 5 covariates) [START_REF] Nicholas | Bayesian inference for logistic models using pólya-gamma latent variables[END_REF]. In each case, an intercept have been included; i.e. p is the number of predictors plus one.

Dataset

Fast Approximations

We compare the four approximation schemes described in Section 2.3: Laplace, Improved Laplace, Laplace EM, and EP. We concentrate on the Cauchy/logit scenario for two reasons: (i) Laplace EM requires a Student prior; and (ii) Cauchy/logit seems the most challenging scenario for EP, as (a) a Cauchy prior is more difficult to deal with than a Gaussian prior in EP ; and (b) contrary to the probit case, the site update requires some approximation; see Section 2.3.4 for more details. Left panel of Fig. 2.1 plots the marginal accuracies of the four approximation schemes across all components and all datasets; Fig. 2.2 does the same, but separately for four selected datasets; results for the remaining datasets are available in the supplement.

EP seems to be the most accurate method on these datasets: marginal accuracy is about 0.99 across all components for EP, while marginal accuracy of the other approximation schemes tend to be lower, and may even drop to quite small values; see e.g. the German dataset, and the left tail in the left panel of Fig. 2.1.

EP also fared well in terms of CPU time: it was at most seven times as intensive as standard Laplace across the considered datasets, and about 10 to 20 times faster than Improved Laplace and Laplace EM. As expected (see Section 2.3.5), the largest running time for EP was observed for the dataset with the largest number of observations (German credit): i.e. 0.14s, while Laplace took 0.02s on the same data. Of course, the usual caveats apply regarding CPU time comparison, and how they may depend on the hardware, the implementation, and so on.

We also note in passing the disappointing performance of Laplace EM, which was supposed to replace standard Laplace when the prior is Student, but which actually performs not as well as standard Laplace on these datasets. We refer the reader to the supplement for similar results on the three other scenarios, which are consistent with those above. In addition, we also represent the approximation error of EP and Laplace for approximating the log-evidence in the right panel of Fig. 2.1. Again, EP is found to be more accurate than Laplace for most datasets (except for the Breast dataset).

To conclude, it seems that EP may be safely be used as a complete replacement of sampling-based methods on such datasets, as it produces nearly instant results, and the approximation error along all dimensions is essentially negligible.

Importance sampling, QMC

We now turn to importance sampling (IS), which we deemed our "gold standard" among sampling-based methods, because of its ease of use and other nice properties as discussed in Section 2.4.1. We use N = 5 × 10 5 samples, and a Gaussian EP proposal. (Results with a Laplace proposal are roughly similar.) We consider first the Gaussian/probit scenario, because this is particularly favorable to Gibbs sampling; see next section. Table 2.2 reports for each dataset the efficiency factor of IS (as defined in Section 2.4.1), the CPU time and two other quantities discussed We see that all these efficiency factors are all close to one, which means IS works almost as well as IID sampling would on such datasets. Further improvement may be obtained by using either parallelization, or QMC (Quasi-Monte Carlo, see Section 2.4.2). Table 2.2 reports the speed-up factor obtained when implementing multi-threading on our desktop computer which has a multi threading quad core CPU (hence 8 virtual cores). We also implemented IS on an Amazon EC2 instance with 32 virtual CPUs, and obtained speed-up factors about 20, and running times below 2s.

Finally, Table 2.2 also reports the MSE improvement (i.e. MSE ratio of IS relative to IS-QMC) obtained by using QMC, or more precisely RQMC (randomised QMC), based on a scrambled Sobol' sequence [see e.g. [START_REF] Lemieux | Monte Carlo and Quasi-Monte Carlo Sampling[END_REF]. Specifically, the table reports the median MSE improvement for the p posterior expectations (first column), and the MSE improvement for the evidence (second column). The improvement brought by RQMC varies strongly across datasets.

The efficiency gains brought by parallelization and QMC may be combined, because the bulk of the computation (as reported by a profiler) is the N likelihood evaluations, which are trivial to parallelize.

It is already clear that other sampling-based methods do not really have a fighting chance on such datasets, but we shall compare them in the next section for the sake of completeness. See also the supplement for results for other scenarios, which are very much in line with those above.

MCMC schemes

In order to compare the different sampling-based methods, we define the IRIS (Inefficiency Relative to Importance Sampling) criterion, for a given method M and a given posterior estimate, as follows:

MSE M M SE IS × CPU IS CPU M
where MSE M (resp. MSE IS ) is the mean square error of the posterior estimate obtained from method M (resp. from importance sampling), and CPU M the CPU time of method M (resp. importance sampling). The comparison is relative to importance sampling without parallelisation or quasi-Monte Carlo sampling. In terms of posterior estimates, we consider the expectation and variance of each posterior marginal p(β j |D). We observe that, in both cases, IRIS does not vary much across the p components, so we simply report the median of these p values. Fig 2 .3 reports the median IRIS across all datasets. We refer the reader to Section 2.4.3 for how we tuned these MCMC algorithms. The first observation is that all these MCMC schemes are significantly less efficient than importance sampling on such datasets. The source of inefficiency seems mostly due to the autocorrelations of the simulated chains (for Gibbs or random walk Metropolis), or, equivalently, the number of leap-frog steps performed at each iteration in HMC and NUTS. See the supplement for ACF's (Autocorrelation plots) to support this statement.

Second, HMC and NUTS do not perform significantly better than randomwalk Metropolis. As already discussed, HMC-type algorithms are expected to outperform random walk algorithms as p → +∞. But the considered datasets seem too small to give evidence to this phenomenon, and should not be considered as reasonable benchmarks for HMC-type algorithms (not to mention again that these algorithms are significantly outperformed by IS on such datasets). We note in passing that it might be possible to get better performance for HMC by finely tuning the quantities ǫ and L on per dataset basis. We have already explained in the introduction why we think this is bad practice, and we also add at this stage that the fact HMC requires so much more effort to obtain good performance (relative to other MCMC samplers) is a clear drawback.

Regarding Gibbs sampling, it seems a bit astonishing that an algorithm specialised to probit regression is not able to perform better than more generic approach on such simple datasets. Recall that the Gaussian/probit case is particularly favourable to Gibbs, as explained in Section 2.4.3. See the supplement for

Bigger datasets

Finally, we turn our attention to the bigger datasets summarised by Table 2.3. These datasets not only have more covariates (than those of the previous section), but also stronger correlations between these covariates (especially Sonar and Musk). We consider the probit/Gaussian scenario.

Regarding fast approximations, we observe again that EP performs very well, and better than Laplace; see Figure 2.5. It is only for DNA (180 covariates) that the EP approximation starts to suffer.

Regarding sampling-based methods, importance sampling may no longer be used as a reference, as the effective sample size collapses to a very small value for these datasets. We replace it by the tempering SMC algorithm described in Section 2.4.4. Moreover, we did not manage to calibrate HMC so as to obtain reasonable performance in this setting. Thus, among sampling-based algorithms, the four remaining contenders are: Gibbs sampling, NUTS, RWHM (random walk Hastings-Metropolis), and tempering SMC. Recall that the last two are calibrated with the approximation provided by EP.

Figure 2.5 reports the "effective sample size" of the output of these algorithms when run for the same fixed CPU time (corresponding to 5 × 10 5 iterations of RWHM), for the p posterior expectations (left panels), and the p posterior variances (right panels); here "effective sample size" is simply the posterior variance divided by the MSE of the estimate (across 50 independent runs of the same algorithm).

No algorithm seems to vastly outperform the others consistently across the three datasets. If anything, RWMH seems to show consistently best or second best performance.

Still, these results offer the following insights. Again, we see that Gibbs sam-

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0.
• • • • • • • • • • • • • • • • • • • • • • • • • • 0 2000 4000 6000 Gibbs NUTS RWMH SMC
Ess for fixed CPU time

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0 10000 20000 30000 40000 50000 Gibbs NUTS RWMH SMC
Ess for fixed CPU time

(a) Musk • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0 1000 2000 3000 Gibbs NUTS RWMH SMC
Ess for fixed CPU time

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0
• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0 500 1000 Gibbs NUTS RWMH SMC
Ess for fixed CPU time pling, despite being a specialised algorithm, does not outperform significantly more generic algorithms. Recall that the probit/Gaussian scenario is very favourable to Gibbs sampling; in other scenarios (results not shown), Gibbs is strongly dominated by other algorithms. More surprisingly, RWHM still performs well despite the high dimension. In addition, RHHM seems more robust than SMC to an imperfect calibration; see the DNA example, where the error of the EP approximation is greater.

• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0
On the other hand, SMC is more amenable to parallelisation, hence on a parallel architecture, SMC would be likely to outperform the other approaches.

Variable selection

We discuss in this section the implications of our findings on variable selection. The standard way to formalise variable selection is to introduce as a parameter the binary vector γ ∈ {0, 1} p , and to define the likelihood

p(D|β, γ) = n D i=1 F (y i β T γ x γ,i )
where β γ (resp. x γ,i ) is the vector of length |γ| that one obtains by excluding from β (resp. x i ) the components j such that γ j = 0. Several priors may be considered for this problem [START_REF] Chipman | The practical implementation of Bayesian model selection[END_REF], but for simplicity, we will take p(β, γ) = p(β)p(γ) where p(β) is either the Cauchy prior or the Gaussian prior discussed in Section 2.2.1, and p(γ) is the uniform distribution with respect to the set {0, 1} p , p(γ) = 2 -p . Computationally, variable selection is more challenging than parameter estimation, because the posterior p(β, γ|D) is a mixture of discrete and continuous components. If p is small, one may simply perform a complete enumeration: for all the 2 p possible values of γ, approximate p(D|γ) using e.g. importance sampling. If p is large, one may adapt the approach of [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF], as described in the next sections.

SMC algorithm of Schäfer and Chopin [2011]

In linear regression,

y i = β T γ x γ,i + ε i , ε i ∼ N 1 (0, σ 2
), the marginal likelihood p(D|γ) is available in close form (for a certain class of priors). [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF] use this property to construct a tempering SMC sampler, which transitions from the prior p(γ) to the posterior p(γ|D), through the tempering sequence π t (γ) ∝ p(γ)p(D|γ) δt , with δ t growing from 0 to 1. This algorithm has the same structure as Algorithm 9 (with the obvious replacements of the β's by γ's and so on.) The only difference is the MCMC step used to diversify the particles after resampling. Instead of a random walk step (which would be ill-defined on a discrete space), [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF] use a Metropolis step based on an independent proposal, constructed from a sequence of nested logistic regressions: proposal for first component γ 1 is Bernoulli, proposal for second component γ 2 , conditional on γ 1 , corresponds to a logistic regression with γ 1 and an intercept as covariates, and so on. The parameters of these p successive regressions are simply estimated from the current particle system. [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF] show that their algorithm significantly outperform several MCMC samplers on datasets with more than 100 covariates.

Adaptation to binary regression

For binary regression models, p(D|γ) is intractable, so the approach of [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF] cannot be applied directly. On the other hand, we have seen that (a) both Laplace and EP may provide a fast approximation of the evidence p(D|γ); and (b) both importance sampling and the tempering SMC algorithm may provide an unbiased estimator of p(D|γ).

Based on these remarks, [START_REF] Schäfer | Monte Carlo methods for sampling high-dimensional binary vectors[END_REF] in his PhD thesis considered the following extension of the SMC algorithm of [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF]: in the sequence π t (γ) ∝ p(γ)p(D|γ) δt , the intractable quantity p(D|γ) is simply replaced by an unbiased estimator (obtained with importance sampling and the Gaussian proposal corresponding to Laplace). The corresponding algorithm remains valid, thanks to pseudo-marginal arguments [see e.g. [START_REF] Andrieu | The pseudo-marginal approach for efficient Monte Carlo computations[END_REF]]. Specifically, one may re-interpret the resulting algorithm as a SMC algorithm for a sequence of distribution of an extended space, such that marginal in γ is exactly the posterior p(D|γ) at time t = T . In fact, it may be seen as a particular variant of the SMC 2 algorithm of Chopin et al. [2013a].

Numerical illustration

We now compare the proposed SMC approach with the Gibbs sampler of [START_REF] Holmes | Bayesian auxiliary variable models for binary and multinomial regression[END_REF] for sampling from p(β, γ|D), on the Musk dataset. Both algorithms were given the same CPU budget (15 minutes), and were run 50 times; see Figure 2.6. Clearly, the SMC sampler provides more reliable estimates of the inclusion probabilities p(γ j = 1|D) on such a big dataset. See also the PhD dissertation of [START_REF] Schäfer | Monte Carlo methods for sampling high-dimensional binary vectors[END_REF] for results consistent with those, on other datasets, and when comparing to the adaptive reversible jump sampler of [START_REF] Lamnisos | Adaptive Monte Carlo for Bayesian variable selection in regression models[END_REF].

Spike and slab

We also note in passing that a different approach to the variable selection problem is to assign a spike and slab prior to β [George and McCulloch, 1993a]:

p(β) = p j=1 λN 1 (β j ; 0, v 2 0 ) + (1 -λ)N 1 (β j ; 0, v 2 1 ) , v 2 0 ≪ v 2 1
where λ ∈ (0, 1), v 2 0 and v 2 1 are fixed hyper-parameters. This prior generates a continuous posterior (without point masses at β j = 0), which is easier to sample from than the discrete-continuous mixture obtained in the standard formulation of Bayesian variable selection. It would be interesting to see to which extent our discussion and findings extend to this particular type of posteriors; see for Chapter 4 for how to deal with such priors in EP.

Conclusion and extensions

Our main messages to users

Our first and perhaps most important message to end users is that Bayesian computation (for binary regression) is now sufficiently fast for routine use: if the right approach is used, results may be obtained near instantly on a standard computer, at least on simple datasets.

Concretely, as far as binary regression is concerned, our main recommendation is to always use EP. It is very fast, and its approximation error is negligible in most cases (for such models). EP requires some expertise to implement, but the second author will release shortly a R package that computes the EP approximation for any logit or probit model. The only drawback of EP is the current lack of theoretical support. We learnt however while finishing this manuscript that Simon Barthelmé and Guillaume Dehaene (personal communication) established that the error rate of EP is O(n -2 D ) in certain models (where n D is the sample size). This seems to explain why EP often performs so well.

In case one wishes to assess the EP error, by running in a second step some exact algorithm, we would recommend to use the SMC approach outlined in Section 2.4.4 (i.e. with initial particles simulated from the EP approximation). Often, this SMC sampler will reduce to a single importance sampling step, and will perform extremely well. Even when it does not, it should provide decent performance, especially if run on (and implemented for) a parallel architecture. Alternatively, on a single-core machine, random walk Metropolis is particularly simple to implement, and performs surprisingly well on high-dimensional data (when properly calibrated using EP). 64

Conclusion and extensions

Our main message to Bayesian computation experts

Our main message to Bayesian computation scientists was already in the title of this chapter: leave Pima Indians alone, and more generally, let's all refrain from now on from using datasets and models that are too simple to serve as a reasonable benchmark.

To elaborate, let's distinguish between specialised algorithms and generic algorithms.

For algorithms specialised to a given model and a given prior (i.e. Gibbs samplers), the choice of a "benchmark" reduces to the choice of a dataset. It seems unfortunate that such algorithms are often showcased on small datasets (20 covariates or less), for which simpler, more generic methods perform much better. As a matter of fact, we saw in our simulations that even for bigger datasets Gibbs sampling does not seem to offer better performance than generic methods.

For generic algorithms (Metropolis, HMC, and so on), the choice of a benchmark amounts to the choice of a target distribution. A common practice in papers proposing some novel algorithm for Bayesian computation is to compare that algorithm with a Gibbs sampler on a binary regression posterior for a small dataset. Again, we see from our numerical study that this benchmark is of of limited interest, and may not be more informative than a Gaussian target of the same dimension. If one wishes to stick with binary regression, then datasets with more than 100 covariates should be used, and numerical comparisons should include at least a properly calibrated random walk Metropolis sampler.

Big data and the p 3 frontier

Several recent papers [START_REF] Bardenet | On markov chain monte carlo methods for tall data[END_REF][START_REF] Scott | Bayes and big data: The consensus monte carlo algorithm[END_REF][START_REF] Wang | Parallelizing MCMC via Weierstrass sampler[END_REF] have approached the 'big data' problem in Bayesian computation by focussing on the big n D (many observations) scenario. In binary regression, and possibly in similar models, the big p problem (many covariates) seems more critical, as the complexity of most the algorithms we have discussed is O(n D p 3 ). Indeed, we do not believe that any of the methods discussed in this chapter is practical for p ≫ 1000. The large p problem may be therefore the current frontier of Bayesian computation for binary regression.

Perhaps one way to address the large p problem is to make stronger approximations; for instance by using EP with an approximation family of sparse Gaussians. Alternatively, one may use a variable selection prior that forbids that the number of active covariates is larger than a certain threshold.

Generalising to other models

We suspect some of our findings may apply more generally to other models (such as certain generalised linear models), but, of course, further study is required to assess this statement.

On the other hand, there are two aspects of our study which we recommend to consider more generally when studying other models: parallelisation, and taking into account the availability of fast approximations. The former has already been discussed. Regarding the latter, binary regression models are certainly not the only models such that some fast approximations may be obtained, whether through Laplace, INLA, Variational Bayes, or EP. And using this approximation to calibrate sampling-based algorithms (Hastings-Metropolis, HMC, SMC, and so on) will often have a dramatic impact on the relative performance of these algorithms. Alternatively, one may also discover in certain cases that these approximations are sufficiently accurate to be used directly.

3

Computation of Gaussian orthant probabilities in high dimension

Status: To appear in Statistics and Computing.

Introduction

There are many applications where computing an orthant probability in high dimension with respect to a Gaussian or Student distribution is an issue of interest. For instance it is common in statistics to compute the likelihood of models, where we observe only an event with respect to multivariate Gaussian random variables.

In Econometrics, the multivariate probit model [START_REF] Train | Discrete Choice methods with simulation[END_REF], where we observe a decision among J alternative choices each of them corresponding to a Gaussian utility, is commonly studied. It can be written as an orthant problem. Other such models are the spatial probit [START_REF] Lesage | New Orleans buisness recovery in the aftermath of huricane Katrina[END_REF] and Thurstonian models [START_REF] Yao | Bayesian estimation of Thurstonian ranking models based on the Gibbs sampler[END_REF]. Other applications than direct modelization can be found, such as multiple comparison tests [START_REF] Hochberg | Multiple comparison procedures[END_REF], where the integration is done with respect to a Student (see [START_REF] Bretz | On the numerical availability of multiple comparison procedures[END_REF] for an example). Orthant probabilities are also of interest in other fields than statistics, i.e. stochastic programming [START_REF] Prekopa | On probabilistic constrained programming[END_REF], structural system reliability [START_REF] Pandey | An effective approximation to evaluate multinormal integrals[END_REF], engineering, finance, etc. The problem at hand is the computation of the integral,

ˆ[a,b] (2π) -d 2 |Σ| -1 2 exp - 1 2 (y -m) t Σ -1 (y -m) dy. (3.1)
where a, b ∈ R d . The Student case will be written as a mixture of the above integral with an inverse Chi-square (see Section 3.5.1).

Many algorithms have been proposed to compute (3.1); for a review see [START_REF] Genz | Computation of Multivariate Normal and t Probabilities[END_REF]. They can be divided into two groups. The first are numerical algorithms to deal with small dimensional integrals. In dimension 3 there exist algorithms [START_REF] Genz | Computation of Multivariate Normal and t Probabilities[END_REF] where after sphericization, such that the Gaussian has an identity covariance matrix, one applies recursively numerical computations of the error function. For higher dimensions than three [START_REF] Minwa | The evaluation of general non-centered orthant pro[END_REF] propose to express orthant probabilities as differences of orthoscheme probabilities, where an orthoscheme is (3.1) with correlation matrix Ω = (ω ij ) satisfying ω ij = 0 ∀i, j |i -j| > 1. This can be easily computed by recursion. However the decomposition in orthoscheme probabilities has factorial complexity. The second group of algorithms is Monte Carlo based and may be used for dimensions higher than 10. In particular GHK due to [START_REF] Geweke | Efficient simulation from the multivariate normal and student-t distributions subject to linear constraints[END_REF], [START_REF] Keane | Simulation estimation for panel data models with limited dependent variables[END_REF] and [START_REF] Hajivassiliou | Simulation of multivariate normal rectangle probabilities and their derivatives theoretical and computational results[END_REF] and conjointly to [START_REF] Genz | Numerical Computation of Multivariate Normal Probabilities[END_REF], has been widely adopted for the applications described above.

In this chapter we show that in the case of Markovian covariances (i.e. covariances that can be written as those of Markovian processes), the GHK algorithm estimates the normalizing constant of a state space model (SSM), using sequential importance sampling (SIS) with optimal proposal. We show in addition for a first order autoregressive process (henceforth AR(1)) that the normalized variance diverges exponentially fast.

To avoid this behavior we propose to use a particle filter. We extend this methodology to the non Markovian case by using Sequential Monte Carlo (SMC). SMC allows additional gain in efficiency by considering different MCMC moves and proposals. In addition the algorithm is adaptive and simplifies automatically to the GHK if the integral is simple enough. In our numerical experiments we find a substantial improvement.

We start by reviewing the existing GHK algorithm (Section 3.2), we then discuss the algorithm's behavior for Markovian covariance matrices and propose an extension to higher dimensions (Section 3.3). In Section 3.4 we extend this proposal to arbitrary covariance matrices. We propose some extensions for the simulation of truncated distributions and for other distributions (3.5). Finally we present some numerical results and conclude (Sections 3.6 and 3.7).

Notations For any vector x ∈ R p for i ≤ p we write x <i ∈ R i for the vector of the i -1 first components, and we take a : b = {a, • • • , b}. We let x <1 = ∅, and also write x i:j for the vector (x i , x i+1 , • • • , x j ); Φ, ϕ are respectively the N (0, 1) Gaussian cdf and pdf, we write ϕ(x|A) for the pdf, ϕ(x) Φ(A) ✶ A (x), of a Gaussian truncated to the set A ⊂ R evaluated in x. We will also abuse notation and use Φ(A) to denote the probability of a set when

A ⊂ R. For instance Φ([a, b]) = Φ(b) -Φ(a).

Geweke-Hajivassiliou-Keane (GHK) simulator

From now on to simplify notations, and without loss of generality, we limit ourselves to the study of the following multidimensional integral:

F (a, b, Σ) = ˆ[a,b] (2π) -d 2 |Σ| -1 2 exp - 1 2 y t Σ -1 y dy (3.2)
with a, b ∈ R d . Note that the extension to integrals where some components of the vectors a, b are respectively -∞ and ∞ is direct.

Let Γ be the Cholesky decomposition of Σ, i.e. Σ = ΓΓ t with Γ = (γ ij ), γ ii > 0 and γ ij = 0 if j > i. We can write the previous equation after the change of variable η = Γ -1 y for which dη = |Γ| -1 dy:

F (a, b, Σ) = ˆb≥Γη≥a (2π) -d 2 exp - 1 2 η t η dη, the i-th truncation being such that 1 γ ii a i -i-1 j=1 γ ij η j ≤ η i ≤ 1 γ ii b i -i-1
j=1 γ ij η j , from the positivity of the (γ ii ). Thus we can write:

F (a, b, Σ) = ˆd i=1 ϕ(η i )1 {B i (η <i )} (η i )dη 1:d = ˆd i=1 Φ (B i (η <i )) ϕ(η i |B i (η <i )) dη 1:d , where the set B i (η <i ) = {η i : 1 γ ii a i -i-1 j=1 γ ij η j ≤ η i ≤ 1 γ ii a i -i-1 j=1 γ ij η j } is an interval.
The GHK algorithm is an importance sampling algorithm based on this structure. It proposes particles distributed under d i=1 ϕ(η i |B i (η <i )) and evaluates the average of the weights w n = d i=1 Φ (B i (η n <i )). The algorithm is described in pseudo-code in Alg. 10.

Algorithm 10 GHK simulator for m ∈ 1 : M do Sample: η m 1:d ∼ d i=1 ϕ(η i |B i (η <i )) Weights ⋆ : w m = d i=1 Φ (B i (η m <i )) end for return 1 M M i=1 w i ⋆ Recall that Φ(B i (η m
<i ) can be computed as a difference of two one dimensional cdf for the truncation defined above.

Algorithm 10 outputs an unbiased estimator of integral (3.1).

To generate truncated Gaussian variables the usual approach in the GHK simulator is to use the inverse cdf method. We follow this approach in the rest of the chapter except where stated otherwise. When the numerical stability of the inverse cdf is an issue we will use the algorithm proposed in Chopin [2011a].

In the next section we will study with more care the case where the covariance matrix of the underlying Gaussian vector has a Markovian structure.

The Markovian case

When the covariance matrix is Markovian, that is a matrix for which the inverse is tri-diagonal, the simulation step of Alg. 10 is the simulation of a Markov process (x 1:t ). At time t the weights depend on x t-1 only. Let us take a lag 1 autoregressive process (AR(1)) for the purpose of exposition, and study the probability of it being in some hyperrectangle

[a, b] = [a 1 , b 1 ] × • • • × [a T , b T ]. The integral of interest is therefore: ˆT t=1 1 {[at,bt]} (x t )ϕ(x t ; ̺ t x t-1 , σ 2 t )dx 1:T . (3.3)
The GHK algorithm consists in sampling from the Markov process:

x t |x t-1 ∼ ϕ ̺ t x t-1 , σ 2 t |B t (x t ) .
The matrix Σ -1 is tridiagonal, the weights at time t are therefore Φ( bt-̺tx t-1 σt ) -Φ( at-̺tx t-1 σt ). Eq. 3.3 can be seen as the likelihood of the state space model [START_REF] Cappé | Inference in Hidden Markov Models[END_REF]:

x t |x t-1 ∼ ϕ(x t ; ̺ t x t-1 , σ 2 t ) y t |x t ∼ 1 {[at,bt]} (x t )
where (y t ) t is observed. The GHK can be interpreted as a sequential importance sampler (SIS) using proposal ϕ(̺ t x t-1 , σ 2 t |B t (x t )).

Toy example

Let us specify a bit more the problem to simplify notation and show some properties of a thus defined algorithm.

Consider the problem of finding the probability that an AR(1),

X t = ̺X t-1 + ε t , |̺| < 1 is inside the hyper-cube [0, b] × • • • × [0, b],
for some b > 0. We have set σ = 1, a = 0 and ̺ a constant.

The GHK algorithm consists in this case in simulating the above Markov chain constrained to [0, b] and in computing under this distribution the products of the weights

T t=1 [Φ(b -̺X t ) -Φ(-̺X t )].
The simulations are therefore generated by the Markov probability kernel

P b (x, dy) = ϕ(y; ̺x, 1) Φ(b -̺x) -Φ(-̺x) ✶ [0,b] (y)dy, |̺| < 1. (3.4)
For this model we have the following proposition: Proposition 3.3.1 For the Markov model defined by (3.4), the normalized square product of weights of the normalizing constant has the following behavior:

lim inf T →∞ E T t=1 Φ (b -̺X t ) -Φ (-̺X t ) exp{2T E π log (Φ(b -̺X) -Φ(-̺X))} 1 √ T > exp {V π [ψ(X)] + τ } ,
(3.5) where subscript π denotes integration with respect to the invariant distribution of P b (x, dy), the other expectation is taken relatively to the Markov chain (X t ), and

ψ : x → log (Φ(b -̺x) -Φ(-̺x)), τ = 2 ∞ k=1 cov(X 0 , X k ).
Proof: A detailed proof is given in appendix 3.A. Under V -Uniform ergodicity, that follows from our proof, the denominator is the square of the limit of the product of weights and can be interpreted as a scaling factor. Thus the result above shows that this renormalized squared estimator diverges exponentially fast as the dimension of the integral increases.

Remark 3.3.1 In the course of the proof we showed that the normalizing constant has a log-normal limiting distribution, resulting in a skewed distribution. We expect that the distribution of the estimator will have its mode away from the expected value resulting in some apparent bias. In fact one can show that the normalized third order moment will also grow exponentially.

GHK has quadratic complexity however we can show that for at least one covariance structure the variance diverges exponentially fast. This fully justifies the use of an algorithm of higher computational complexity. In the following section we propose a natural extension to deal with this issue in the Markovian case.

Particle filter (PF)

PF is a common extension of SIS that corrects the weight degeneracy problem. The solution brought by particle filtering [START_REF] Neil J Gordon | Novel approach to nonlinear/non-gaussian bayesian state estimation[END_REF] is to use a resampling step, i.e. to kill those particles with low weights and to replicate those with high contribution. At time t one resamples the particles by sampling from the distribution M m=1 W m t δ x m t (dx) where W m t stands for the m-th renormalized weight at time t, and δ x (dx ′ ) the Dirac measure in x. All the weights are then set to one.

We use an adaptive version of this algorithm where the resampling step is triggered only when the ESS of the weight is lower than some threshold, where the ESS is defined as

N i=1 w i 2 i w 2 i ∈ [1, M ] ,
and indicates the number of draws from the independent distribution to obtain the same variance. Note that it is closely related to the inverse of equation (3.5), hence we expect that without resampling it goes to zero with exponential speed.

We define the state space model:

x t |x t-1 ∼ g t (x t |x t-1 ), y t |x t ∼ f t (y t |x t )
One can use a PF to compute the likelihood of such model,

L(y 1:T ) = ˆT t=1 g t (x t |x t-1 )f t (y t |x t )g 0 (x 0 )dx 0:T
A PF with proposal distribution q t (x t , x t-1 ) is described in Alg. 23. Our application corresponds to the special case where:

g t (x t |x t-1 ) = ϕ(x t ), f t (y t |x t , x t-1 ) = 1 {Bt(x<t)} (x t ), q t (x t |x t-1 ) = ϕ(x t |B t (x <t )),
where the set B t (x <t ) depends on x t-1 only.

The proposal thus defined corresponds to the optimal one [START_REF] Doucet | On sequential monte carlo sampling methods for bayesian filtering[END_REF], that is the distribution proportional to f t (y t |x t )g t (x t |x t-1 ) in our case proportional to ϕ(x t )1 {Bt(x<t)} (x t ) hence the truncated Gaussian. The weights are given by the normalizing constant ´ft (y t |x t )g t (x t |x t-1 )dx t , in our case Φ(B t (x <t )).

To resample we propose to use systematic resampling [START_REF] Carpenter | Improved particle filter for nonlinear problems[END_REF]] (for other approaches see [START_REF] Douc | Comparison of resampling schemes for particle filtering[END_REF]). Systematic resampling is described in Algorithm 13 (Appendix 3.B).

The particle filter thus defined outputs an unbiased estimator of the likelihood Del Moral [1996a], and thus the orthant probability in our case.

Note that the output of Algorithm 23 is of the form of a product of terms smaller than one, in our case those terms can be very small and lead to numerical issues. One way of dealing with this issue is to rewrite all the algorithm in log scale.

Algorithm 11 Particle Filter

Input: M the number of particles Sample: Sample x i 0 ∼ g 0 (.)

for t = 1 : T -1 do if ESS < η ⋆ then Z ← Z × { 1 M M i=1 w i t } Resample a j t ∼ i w i t j w j t δ i using algorithm 13, set w j t ← 1 else a 1:M t = 1 : M end if Sample x i t+1 ∼ q t+1 (.|x a i t t ) Set w i t+1 ← w i t f t+1 (y t+1 |x i t+1 )g t+1 (x i t+1 |x a i t t ) q t+1 (x i t+1 |x a i t t ) end for return Z × 1 M M i=1 w i T Remark 3.3.2
As we are here in the special case of being able to sample from the optimal distribution (as shown in Section 3.3) one could resort to the auxiliary particle filter (APF, [START_REF] Pitt | Filtering via simulation: Auxiliary particle filters[END_REF]). In fact in this special case the algorithm amounts to exchanging the resampling step and the move step of the particle filter. We tested this approach on some Markov processes and observed no improvements in term of variance on repeated draws.

Example 3.3.1 We can show that the previous process (Section 3.3.1) benefits from resampling when the ESS goes beneath a given level.

Figure (3.1)

shows that the GHK algorithm's variance increases more quickly as compared to the PF (that seem to have some stable variance on the considered dimension). In addition the distribution of the GHK estimator seem to be skewed towards smaller values as T increases. This results in some bias on the last boxplot. As described in remark 3.3.1 this behavior is due to the log-Normal limiting distribution of the output of the algorithm. The skewness coefficient increases exponentially with T .

Thurstonian Model

Thurstonian models arise in Psychology and Economics [START_REF] Yao | Bayesian estimation of Thurstonian ranking models based on the Gibbs sampler[END_REF] to describe the ranking of p alternatives by n individuals (referred to as judges). Suppose that we observe the rank r i = (k 1i , • • • , k p,i ) of some p independent Gaussian random variables, x i,j = β j + σε i,j , where ε i,j i.i.d

∼ N (0, 1). The likelihood of one observation is an orthant probability:

P θ {X p > • • • > X 1 } = ˆp i=1 1 {x i >x i-1 } ϕ(x i |β j , σ 2 )dx 1:p (3.6)
with the convention that X 0 = -∞. This model is similar to the previous one but with ρ = 1. We find that the likelihood is estimated with smaller variance. In addition, because of the heavy tail distribution of the GHK simulator's output we observe a bias (see Figure 3.2). Again one can explain the strong observed bias by remark 3.3.1 and the fact that we do not replicate enough the experiment to observe the tail of the distribution. In addition as suggested above the ESS of the GHK seems to decrease exponentially fast to zero.

From this observation we could apply this algorithm to perform inference by using Particle MCMC [Andrieu et al., 2010a], where this estimation of the likelihood can be plugged in a Random walk Metropolis Hastings and still target the appropriate distribution.

Non Markovian case

For more general covariances we propose to use Sequential Monte Carlo (SMC) [Del Moral et al., 2006b]. As previously we will base the algorithm on the proposal of GHK, increasing the dimension of the problem at each time step. However we now have an additional degree of freedom: the order in which we incorporate the variables. In the following section we study an approach to ordering the variables.

Variable ordering

We follow [START_REF] Gibson | Monte-carlo evalution of multivariate normal integrals and sensitivity to variate ordering[END_REF] in ordering the variables from the most difficult to the simplest, where difficult constraints are considered to be the one that impact the most the probability.

However we cannot evaluate exactly the probabilities as it is our final goal. Instead [START_REF] Gibson | Monte-carlo evalution of multivariate normal integrals and sensitivity to variate ordering[END_REF] propose to replace the simulations by the expected value of the truncated Gaussian.

The algorithm starts by choosing the first index i 1 , and defining η 1 as follows:

i 1 = arg min 1≤k≤T Φ a k γ kk , b k γ kk , η 1 = 1 Φ a i 1 γ i 1 i 1 , b i 1 γ i 1 i 1 ˆ a i 1 γ i 1 i 1 , b i 1 γ i 1 i 1 ηϕ(η)dη
i.e. the smallest possible probability that the Gaussian will be in [a k , b k ]. This enables an approximation of the next probability as a function of i 2 .

i 2 = arg min 2≤k≤T Φ 1 γkk (a k -γ1,k η 1 ), 1 γkk (b k -γ1,k η 1 ) .
where (γ ij ) = Γ is the Cholesky decomposition of the matrix after substituting the first and the i 1 th variable.

We end up with the desired vector (i 1 , • • • , i T ) that gives us the order in which to choose the covariances and truncation points. The algorithm is summed up by Alg. 14 in appendix 3.C. The algorithm has quadratic time complexity, however its cost is negligible as compared to the subsequent Monte Carlo algorithm.

We show the use of the reordering in moderate dimensions (50 and 60) on the GHK simulator. This is already a great improvement especially as the dimension increases. Figure (3.3), shows boxplots of 50 repetitions of the GHK for both ordered (white) and non-ordered (grey) inputs. Covariance matrices generated from random samples with heavy tails (see Section 3.6). In both case we use a GHK simulator with variable ordering (white), without (grey). The various dimension are simulated with the same algorithm and same seed, such that the small ones are subsets of the others. When the variables are not ordered we observe some outliers, and this phenomenon is reduced with [START_REF] Gibson | Monte-carlo evalution of multivariate normal integrals and sensitivity to variate ordering[END_REF]'s algorithm.

• • • • • • • • • • • • • • • • • • -5.030805e
• • • • • • • • • • • • • • • • • • • • • • • • • • -2.
From dimension 50 and upwards we start observing some skewed distributions for the GHK estimator as noted in remark 3.3.1. The phenomenon seems to be reduced by ordering.

This effect is relatively dangerous as some draws depart a lot from the mean value. The ordering will be used on all examples from now on to reduce the variance. In Section 3.6 we have empirical evidence that using an appropriate move step deals with this tail effect, in our examples.

We have shown that in the particular case of Markov processes we can strikingly benefit from the use of resampling. In the next section we attempt to generalize our finding to a broader range of problems.

A sequential Monte Carlo (SMC) algorithm

The algorithm discussed in Section 3.3 can be generalized to non Markovian Gaussian vectors by applying the SMC methodology. Define the following sequence of distribution:

π t (η 1:t ) = γ n (η 1:t ) Z t , γ t (η 1:t ) = t i=1 ϕ(η i )1 {B i (η <i )} , (3.7)
indexed by t, where the unnormalized quantity γ t (η 1:t ) is our target integrand. We thus want to compute an estimator of Z t for a given t,

Z t = ˆt i=1 ϕ(η i )1 {B i (η <i )} dη 1:t .
SMC samplers are a class of algorithms that generalize particle filters to non dynamic problems (Neal [2001a], Chopin [2002a], Del Moral et al. [2006b]). Their aim is to sample from a sequence of measures (π t ) t where π 0 is easy to sample from and π T is our target. The algorithm works by moving from one target to the other by importance sampling, and avoid degeneracy of the weights by resampling if the ESS falls bellow a threshold. In the case of the GHK the sequence of distribution consist of adding a dimension at each step. To ensure particle diversity after resampling the particles are moved according to a MCMC kernel targeting the current distribution. This is the most computationally expensive step. Different alternatives are described in the next section.

The main steps are described in Algorithm ( 12) bellow:

Algorithm 12 SMC for orthant probabilities Input Γ,a, M , α ⋆ , Set Z ← 1 Each computation involving m is done ∀m ∈ 1 : M Init η m 1 ∼ ϕ(.)1 B k (η <1 ) , and w m 1 = 1 for t ∈ 1 : T -1 do At time t the weighted system is distributed as (w m t , η m 1:t ) ∼ t k=1 ϕ(η k )1 B k (η <k ) ∝ π t (η 1:t ). if ESS(w 1:M t ) < α ⋆ then Z ← Z × { 1 M M i=1 w i t }. Resample: η ′m t ∼ M j=1 w j t δ η j t , w m t ← 1. Move: η m t ∼ K t (η ′m t , dη m t ) where K t leaves π t (η 1:t ) invariant. end if η m t+1 ∼ ϕ(.|B t+1 (η m <t+1 )), w m t+1 ← w m t × Φ(B t+1 (η <t+1 )). end for return Z × { 1 M M i=1 w i T }
An interesting feature of Algorithm 12 is that if the integral is simple enough the ESS will never fall under the threshold and the above algorithm breaks down to a GHK simulator. This allows the algorithm to adapt to simple cases at a minimal effort, that of computing the ESS.

Note also that the estimator is still unbiased (Del Moral [1996a]) and can therefore be used in more complex schemes such as PMCMC (Andrieu et al. [2010a]) or SMC 2 (Chopin et al. [2013b]).

Move steps

The moves step will have an important impact on the non-degeneracy of the particle system. We want to construct a Markov chain that moves the particles as far away from their initial position as possible. In addition this step will be the bulk of the added time complexity compared to GHK, so we want to make it as efficient as possible.

Gibbs sampler

The structure of our target (3.7), where the dependence of the Gaussian components lies within the truncation, does not allows a direct application of the Gibbs sampler of [START_REF] Robert | Simulation of truncated normal variables[END_REF], without a change of variable. In this section, to simplify notations we consider the special case of b = ∞. We write the conditional distribution at time t as proportional to t i=1 ϕ(η i )✶ (Γ<t,<tη<t)>b<t , where Γ <t,<t is the matrix built with the first t -1 lines and columns of Γ. The conditional is given by:

η i |η -i ∼ ϕ . t j≥i sign(γ ji )η i ≥ 1 |γ ji | a j - k =i η k γ jk .
We therefore have to compute those sets for each component up to t and simulate according to a truncated Gaussian. Computing the set can lead to one or two sided truncations depending on the sign of the γ ij .

The main drawback about having to compute this step each time we resample is its complexity. This operation has time complexity O(d 3 ) per time step. This is easily seen as the set in the above equation is just the result of some matrix inversion for a lower triangular system of dimension d. This leads to an SMC algorithm that seem to have a prohibitive complexity of O(d 4 ), where the GHK simulator had an O(d 2 ) complexity. However we have shown that GHK's variance diverges exponentially quickly on some examples suggesting that this complexity might be acceptable. In fact examples in high dimension show that even at constant computational cost the algorithm is able to out-perform GHK (see Section 3.6).

Hamiltonian Monte Carlo

An alternative to Gibbs sampler is to use Hamiltoninan Monte Carlo (HMC) (see [Neal, 2010a] for a survey), and the idea of [START_REF] Pakman | Exact Hamiltonian Monte Carlo for Truncated Multivariate Gaussians[END_REF] for truncated Gaussians.

HMC is based on interpreting the variables of interest as the position of a particle with potential the opposite of the log target and by simulating the momentum as a Gaussian with given mass matrix. The proposal of the Metropolis-Hastings is then constructed by applying the equations of motion up to a time horizon T HM C to the problem. This leads to an efficient algorithm that makes use of the gradient of the target to explore its support. We refer the reader to Neal [2010a] for more details on the algorithm and describe the approach proposed by [START_REF] Pakman | Exact Hamiltonian Monte Carlo for Truncated Multivariate Gaussians[END_REF] to adapt the algorithm to truncated Gaussians.

Based on the fact that the log density of a Gaussian random variable is a quadratic form, the movement equation can be dealt with explicitly. The scheme is written as an exact HMC (i.e. not resulting in numerical integration). Remains then to deal with the truncation. [START_REF] Pakman | Exact Hamiltonian Monte Carlo for Truncated Multivariate Gaussians[END_REF] show that they can be treated as "walls" for the given particle, a reflection principle can be applied for any particle hitting the constraint during the algorithm. In particular we must find the time at which occurs the first "hit". In our experiment the time horizon T HM C is set to a uniform draw on [0, π] as suggested in Neal [2010a]. The average value π/2 is advocated by [START_REF] Pakman | Exact Hamiltonian Monte Carlo for Truncated Multivariate Gaussians[END_REF].

The computation of the first hitting time dominates the cost of the algorithm. This is particularly true when the truncation are small as the number of hitting times will be high. Figure 3.D.1 in appendix 3.D shows a comparison of the SMC algorithm with the Gibbs sampler (grey) and exact HMC (white). Although this Markov chain algorithm seems to perform very well for a wide range of problems and has a neat formalism, we find that it does not outperform Gibbs sampling when used as a move. The specificity of the move step in SMC is that the particles are already distributed according to (3.7), therefore the move need not propagate each particle across all the support. In particular the strength of HMC in quickly exploring the target might be less useful in this context.

Overrelaxation

Overrelaxation for Gaussian random variables was proposed by [START_REF] Adler | Over-relaxation method for the Monte Carlo evaluation of the partition function for multiquadratic actions[END_REF] as a way of improving Gibbs sampling for a distribution with Gaussian conditionals.

For each component the proposal is

η ′ i |η -i ∼ N (µ i + α(η i -µ i ), σ 2 i (1 -α 2
)) for 0 ≤ α ≤ 1, and with µ i and σ 2 i the expectation and variance of η i |η -i . The case α = 0 is the classical Gibbs sampler, the case α = 1 is a special case of random walk Metropolis-Hasting proposal. One can check that if η i ∼ N (µ i , σ 2 i ) then η ′ i has the correct distribution. Given a particle η, we propose a new one according to:

η ′ |η ∼ N (αη, (1 -α 2 )I)
Setting aside the constraint for a moment the invariant distribution of such kernel is an independent (0,1)-Gaussian. If we add an acceptation step such that we accept if it satisfies the constraint at time t, the Markov kernel leaves the current distribution invariant (3.7).

We find that the fact that overrelaxation is close to a Metropolis adjusted Langevin algorithm (MALA) helps to calibrate the algorithm, log π(η) = -1 2 η T η, hence the proposal in MALA is N ((1 -ε 2 )η, ε 2 ). From [START_REF] Roberts | Optimal scaling of discrete approximations to langevin diffusions[END_REF] we have that ε should be O(d -1

3 ). To calibrate the algorithm we propose to match the two drifts. We find that α = O(1 -0. 5d -1 3 ), the constant should then be close from a problem to an other because locally we are always in the case of independent Gaussians (locally the constraints have less impact). We find that in our case taking α = 0.004 × (1d -1/3 ) gives the expected behavior and acceptance ratio.

Repeating the move step [START_REF] Dubarry | Particle approximation improvement of the joint smoothing distribution with on the fly variance estimation[END_REF] have shown, for particle filters, that applying some Metropolis Hastings kernel targeting the filtering distribution on the particles leads to a close to optimal variance (the variance is the same as one coming from an iid sample). This convergence results happens after O(log M ) iteration of the Markov kernel. These results suggest repeating the move step after each resampling step until some criterion of convergence is satisfied.

We compute the sum of absolute distances that the particles have moved after each step (a similar metric was used in [START_REF] Schäfer | Sequential monte carlo on large binary sampling spaces[END_REF] for the discrete case). We repeat the move until this scalar value stabilizes. The stabilization of the total metric should be associated with the cancellation of the dependence between the particles (leading to a close to independent system).

Block sampling

To diversify the particle system after each resampling we have relied until now on invariant kernels targeting the current distribution π t . An alternative to this approach is given by [START_REF] Doucet | Efiicient Block Sampling Strategies for Sequential Monte Carlo[END_REF], where importance sampling is done on the space of η t-L+1:t+1 with a given number L of previous time steps. This limits the behavior of the particles all stemming from one path after a few iterations. We briefly describe the idea in the following.

Suppose at time t-1 we have a weighted set of particles such that (w t-1 , η 1:t-1 ) ∼ π t-1 (η 1:t-1 ); instead of proposing a particle η ′ t , propose a block of size L, η ′ t-L+1:t ∼ q(.|η 1:t-1 ), and discard the particles η t-L+1:t-1 . The distribution of the resulting system is intractable because of the marginalization. However [START_REF] Doucet | Efiicient Block Sampling Strategies for Sequential Monte Carlo[END_REF] note that importance sampling is still possible on the extended set of particles (η 1:t-1 , η ′ t-L+1:t ) by introducing some auxiliary distribution λ t (η t-L+1:t-1 |η ′ 1:t ). This leads to the correct marginal whatever λ t and the algorithm has the following incremental weights:

π t (η 1:t-L , η ′ t-L+1:t )λ t (η t-L+1:t-1 |η ′ t-L+1:t , η 1:t-1 ) π t-1 (η 1:t-1 )q(η ′ t-L+1:t |η 1:t-1 )
.

The authors show that the optimal proposal and resulting weights are given by:

q opt (η ′ t-L+1:t |η 1:t-1 ) = π t (η ′ t-L+1:t |η 1:t-L ), w t = w t-1 π t (η ′ 1:t ) π t-L (η 1:t-L )
.

In our case the optimal proposal can then be shown to be:

q opt t (η ′ t-L+1:t |η 1:t-L ) = t i=1 ✶ B i (η <i ) t i=t-L+1 ϕ(η i ) ´ t i=1 ✶ B i (η <i ) t i=t-L+1 ϕ(η i )dη t-L+1:t .
Notice that this is the density of a truncated Gaussian distribution, yielding a weight depending on an orthant probability (denominator). In most cases this is not available and in our particular case it is the quantity of interest. We can however compute explicitly this integral for L = 1 and L = 2. The former is the usual case (block of size one). The case L = 2 did not bring any improvement in terms of variance in all our simulation. We concentrated on the extension to blocks of higher dimension.

In this case we have to resort to approximations of the proposal. The first idea would be to approximate it by a Gaussian using expectation propagation [Minka, 2001a]. However this approach did not perform better than the use of Gibbs sampler mentioned earlier. Another approach to approximate the distribution is to consider the Gibbs sampler on a block of size L with the GHK proposal.

Partial conclusion

We have shown that the proposed Gibbs sampler outperforms HMC. Concerning block sampling the different approaches were tested on several dimensions only to find that the best performing approach was to use partial Gibbs sampling, i.e. a Gibbs sampler on a block. In the numerical tests we provide in Section 3.6 we show only the latter.

In our simulations we propose to repeat each kernels as was explained in Section 3.4.3. We propose to test the Gibbs sampler and the overrelaxed random walk.

In addition we have studied other kernels based on the geometry of the problem; in particular, one can draw random walks on the line between the current particle and the basic solution of our constraint. Those approach did not however outperform the proposals discussed above.

Extentions

Student Orthant

We can easily extend our approach to the computation of orthant probabilities for other distributions, in particular for mixtures of Gaussians, that is probabilities that can be written as:

ˆfU (u) ˆfH|U (η|u)1 {b>η>a} dηdu, (3.8)
where f H|U is a Gaussian. Several distributions can be created as such. For instance, the Student distribution where the variance is marginally distributed as an inverse-χ 2 . Hence the distribution:

f H|U (η|u)1 {b>η>a} = n i=1 ϕ(η i )1 {B u i (η <i )} , where B u (η <i ) is B i (η <i )
where we multiply a by u ν and f U (u) = χ 2 ν (u). They are an interesting application to those algorithms because they come at a minimal additional cost and are of use in multiple comparison [START_REF] Bretz | On the numerical availability of multiple comparison procedures[END_REF].

Another example is the logistic distribution where f U (u) is some transformation of a Kolmogorov-Smirnov distribution (see [START_REF] Holmes | Bayesian auxiliary variable models for binary and multinomial regression[END_REF]). This could be used to perform Bayesian inference on multinomial logistic regression.

To deal with this integral we can extend the space on which the SMC is carried out at time t. Hence the move step is performed on the extended space f U (u)f H|U (η|u). In our Student example it amounts to taking as a target distribution

π n (η 1:n , u) ∝ n i=1 ϕ(η i )1 {B u i (η <i )} χ 2 ν (u).
The normalizing constant that the SMC algorithm approximates is

Z n = ˆn i=1 ϕ(η i )1 {B u i (η <i )} χ 2 ν (u)dη 1:n du.
At each move step we therefore move the particles using a Metropolis-Hastings algorithm targeting p(u|η 1:n ) and perform the remaining Gibbs sampler updates conditionally on U . This additional step allows for further mixing. Benefits from this step are already found in relatively low dimension as shown in Section 3.6.

SMC as a truncated distribution sampler

A natural extension is to use Alg. 12 to compute other integrals with respect to truncated Gaussians. At time t the output of the algorithm is a weighted sample

(w i t , η i 1:t ) i∈[1,M ] approximating π t (η 1:t ) ∝ t i=1 ϕ(η i )✶ B(η <i )
. Hence any integral of the form E πt (h(η)), where expectation is taken with respect to π t , can be approximated by M i=1 w i t M j=1 w j t h(η i 1:t ). The same argument goes for the truncated Student.

We test the idea for computing the expectation of truncated multivariate Student. We use a Gibbs sampler as a benchmark based on [START_REF] Robert | Simulation of truncated normal variables[END_REF]'s sampler by adding a MH step to deal with u (see previous section). The Gibbs update is done after a change of variable that leaves the truncations independent. This can be shown to be more efficient. We allocate 100 times more computational time to the Gibbs sampler than the SMC.

In Figure 3.4 we see that after thinning one out of 1000 points the ACF and trace plots point to bad exploration of the target's support. This behavior shows that the convergence is too slow for the algorithm to be of practical use. On the other hand the SMC is still stable as is shown in the next section.

In addition of outperforming the Gibbs sampler for fairly moderate dimension, the SMC algorithm was found to be stable for approximating the expectation in dimensions up to 100. The data are generated as explained in the "Numerical results section", for dimension 50. The left panels are trace plots for two components. The right panels are the ACFs. Both are shown after a thinning of 1/1000. Both show a slow convergence, whereas we observe that SMC is stable.

Numerical results

Covariance simulation, tunning parameters

To build the covariance matrices we propose to use draws from a Cauchy distribution. We start by sampling a matrix X and a vector a from an independent Cauchy distribution X ij ∼ C(0, 0.01) and a i ∼ C(0, 0.01), then construct the covariance matrix as Σ = X t X and the truncation as a = (a i ). Because of the heavy tails the resulting correlation matrix (figure 3.5a) has many close to zero entries and some high correlations. The truncations also have some very high levels (figure 3.5b).

We find that this approach leads to more challenging covariances than those built by sampling the spectrum of the covariance matrix as proposed for instance in [START_REF] Christen | On optimal direction Gibbs sampling[END_REF]. The tuning parameters of the algorithm are the threshold that tunes the number of steps of the MCMC kernel, and the targeted ESS under which we resample, α ⋆ . The former is set to some small value (0.01) and has not much influence. The latter gives us a trade off between variance and computational cost. In our example we have found that 0.5M allows good approximation, however this value should be increased with the difficulty of the problem.

GHK for moderate dimensions

All our results are shown at constant computational cost: we repeat the algorithm in a first time to get their execution time, and we then scale them accordingly. In the above example (dimension 40) for instance the number of draws associated with the GHK algorithm is 1, 065, 399. The number of particles of the SMC sampler with Gibbs Markov transition is 5217. The various dimensions are simulated as described in Section 3.6. Different moves are tested inside SMC. As we have already discussed GHK leads to a higher variance and outliers. Block sampling (BS) and Gibbs sampling (Gibbs) seem to outperform the overrelaxed random walk (RW). However all three stay stable until dimension 50.

We find that in moderate dimension (∼ 50) the GHK simulator breaks down in attempting to compute the probability of the orthant generated by our simulation scheme. This is all the more problematic as it gives an answer, and there is no way of checking its departure from the true value.

Another interesting aspect is the fact that the block sampling algorithm performs well in those dimensions. It is quicker than to move the particles in every dimension as is done with MCMC. The truncations that lead to a drop of probability are close together because of the ordering, hence once the difficult dimensions have been "absorbed" it is less and less paramount to visit the past truncations.

High dimension orthant probabilities

In dimensions higher than p = 70, the covariance we simulate lead to integrals that cannot be treated with the GHK algorithm. In our simulations GHK always returned NaN values due to the low values of the weights. For the SMC an indicator of the good behavior of the algorithm can be seen in either its reproducibility and the fact that we do not encounter asymmetry (see Remark 3.3.1) as for the GHK in the first two Sections. Furthermore the ESS does not fall very low along the particles' draw (Figure 3.7c). The various dimensions are simulated as described in Section 3.6. Different moves are tested inside SMC. As we have already discussed GHK leads to a higher variance and outliers. Gibbs sampling (Gibbs) seem to outperform the overrelaxed random walk (OR) and Block sampling (BS). However all three stay stable until dimension 180. The ESS for the Gibbs sampler is shown in panel c for a threshold of 0.5M and M = 3000. Despite some sudden drops it seems to be stable.

For those dimensions the Gibbs sampler performs best in terms of variance. However if one's goal is a fast algorithm, at the cost of higher variance the overrelaxation might be preferable at some point as the dimension of the target increases. The latter as a complexity smaller of one degree such that at constant computational cost it will have more and more particles allocated to it.

Student orthant probabilities

We use the same schemes as before to construct the covariance matrix and fix a degree of freedom of 3 in our experiments. As before we show an improvement as compared to previous algorithms. This improvements appears also for moderate dimensions. It seems that there is an important gain in considering the extended target.

As for the Gaussian case we find that the output of GHK is heavily skewed. It seems that it is not the case for our algorithm. Covariance matrices generated from random samples with heavy tails. We find that the SMC outperforms the GHK. As for the previous cases the GHK leads to some outliers.

Application to random utility models

Random utility models are an important area of research in Economics to model choice data [START_REF] Train | Discrete Choice methods with simulation[END_REF]. Consider an agent i confronted to J alternatives each giving utility

Y ⋆ ij ∀j ∈ {1, • • • , J} modeled by Y ⋆ ij = X i β + u ij with u ij a Gaussian noise. Individual i chooses alternative j if {∀k = j Y ⋆ ij > Y ⋆ ik }.
The likelihood is the probability of this set integrated over the unobserved alternatives. Hence the likelihood is given by:

L(Y i = j|Ω, β, X) = P k =j {Y ⋆ ij > Y ⋆ ik }} = P k =j {(X ij -X ik )β ⋆ > u ik -u ij }}
where integration is taken over u ∼ N (0, Ω), where u = (u ij ). The above integral is an orthant probability of dimension J -1. A yet more challenging case occurs in the presence of panel data. The latter corresponds to sequential choices of an individual in time. We denote those choices by the subscript t. We observe (j t ) t<T for every individual. Integration is now in dimension T (J -1) and takes the form:

L(Y i,1:T = j 1:T |Ω, β, X) = P T t=1 k =jt {(X ijtt -X ikt )β ⋆ > u ikt -u ijtt }}
We take the covariance structure studied in Bursh-Supan et al. [1992]. The noise term is u itk = α ik + η ikt where η ikt = ̺ i η ikt-1 + ν it , where (α ik ) are correlated amongst choices, so are ν it . The terms are all Gaussian.

The dataset is simulated to allow for examples that are more complex, and of variable size. In the model presented above individuals are independent so that we present results in computing the integral for n = 1, and have already a big advantage of using our methodology. Dataset: The Data is simulated using the covariance proposed in Bursh-Supan et al. [1992], the value of the parameter for which the likelihood is evaluated is taken at random.

Figure (

3.9) shows, for two problems of different size, the gain in precision at constant computational cost. The improvement is substantial and increases with the size of the problem. Taking n > 1 would only increase this effect as it would consist in taking products of such estimators.

The latter result suggests that we could use the likelihood for inference using either maximum likelihood or PMCMC. Computing the likelihood with lowest possible variance is also a key issue in finding the evidence in the most precise manner possible.

When comparing the two algorithms we set the number of particles of SMC to M = 1000, for the same computational cost we allocate M = 881031 to GHK. SMC has however still a lower variance. Regardless of computational time, the ability to compute precise integrals for a small number of particles can also be of importance. It is the case for instance in SMC 2 [Chopin et al., 2013b] where whole trajectories have to be kept in memory for several samplers.

One could extend these models to multivariate Probit with Student distribution and to multivariate logit models for more robustness. In this case we can use the algorithm based on the mixture representation built in Section 3.5.1.

Conclusion

We have shown empirically that the GHK algorithm collapses when the dimension of the problem increases (returning NaN values). In other cases, the distribution of estimates generated by GHK may have heavy tails (see also Remark 3.3.1). Theoretically for at least one covariance structure we have shown that the variance of the algorithm diverges exponentially fast with the dimension (Section 3.3). Our SMC algorithm seems to correct this behavior, and was found to be of practical use for many problems.

We have tested several kernels as part of the move step (Section 3.4.2). We advise the practitioners to use Gibbs sampling as the standard "go to" move step. However improvements in speed can be achieved for dimensions around 50 using only a partial update. In addition as the dimension increases one might want to use a method with lower complexity at the cost of having to repeat the move a bit more. In this case we recommend the use of an overrelaxed random walk Metropolis-Hastings.

We have shown that the same idea can be use for computing probabilities of mixtures of Gaussians. In addition we can use the weighted particles returned by the algorithm to compute other integrals (mean, variance, .etc). This approach The solution of the above equation is a truncated skew-Normal distribution,

π(dy) ∝ {Φ(b -̺y) -Φ(-̺y)} ϕ(y; 0, 1 -̺ 2 )✶ [0,b] (y)dy.
The moments of this distribution have been studied in [START_REF] Flecher | Truncated skew-normal distributions: Estimation by weighted moments and application to climatic data[END_REF], in particular note that 0 < V π (X) < ∞.

Define ψ : x → log [Φ(b -̺x) -Φ(-̺x)], by theorem 17.0.1 [START_REF] Meyn | Markov chains and Stochastic Stability[END_REF] to obtain a CLT for 1 √ T T t=1 ψ(X t ) we must ensure that there exist a constant e > 0 such that ψ 2 (x) < V e (x) on [0, b]. Such a constant can be found by noting that ψ(x) is bounded as long as b > 0 and that V e is strictly increasing of e > 0 with value on (0, ∞). The value of e depends on b. We obtain the following convergence result,

1 √ T T t=1 ψ(X t ) -T E π (ψ(X)) N (0, V π {ψ(X)} + τ ) ,
where the variance term is defined because ψ is bounded on [0, b], and τ = 2 ∞ k=1 cov(X 0 , X k ). By taking the exponential and using the continuous mapping theorem (p.7 [START_REF] Van Der | Asymptotic statistics[END_REF]) we get a log-normal limiting distribution

T t=1 (Φ(b -̺X t ) -Φ(-̺X t )) exp{T E π log (Φ(b -̺X) -Φ(-̺X))} 1 √ T EN (0, V π {ψ(X)} + τ ) .
By Portmanteau's Lemma (p.6 Van der Vaart [1998]) for x 2 as a continuous and positive function,

lim inf T →∞ E   T t=1 (Φ(b -̺X t ) -Φ(-̺X t )) 2 exp{2T E π log (Φ(b -̺X) -Φ(-̺X))} 1 √ T   > exp {V π [ψ(X)] + τ } lim inf T →∞ E T t=1 (Φ(b -̺X t ) -Φ(-̺X t )) 2 exp{2T E π log (Φ(b -̺X) -Φ(-̺X))} 1 √ T > exp {V π [ψ(X)] + τ }
where the last line is obtained by Jensen inequality. The denominator is the square of limit value of the normalizing constant under x 2 -Uniform ergodicity that follows from the above statement.

3.B Resampling

Algorithm 13 Systematic resampling, n particles Input: Vector of weights w and vector

x to sample from Set v ← nw, j ← 1, c = v 1 Sample: Sample U ∼ U [0,1] for k = 1, • • • , n do while c < u do Set j ← j + 1, c ← c + v j end while Set: xk ← x j , u ← u + 1 end for return x

3.C Variable Ordering

Algorithm 14 Variable Ordering

INIT: i 1 = arg min 1≤k≤T Φ a k γ kk , b k γ kk η 1 = 1 Φ a i 1 γ i 1 i 1 , b i 1 γ i 1 i 1 ´ a i 1 γ i 1 i 1 , b i 1 γ i 1 i 1 ηϕ(η)dη for i ∈ {2, • • • , d} do STEP 1 i j = arg min j≤k≤T Φ 1 γkk a k -j-1 l=1 γi l k η k , 1 γkk b k -j-1 l=1 γi l k η k STEP 2 η j = 1 Φ 1 γkk a k -j-1 l=1 γi l k η k , 1 γkk b k -j-1 l=1 γi l k η k × ˆ 1 γkk {ak-j-1 l=1 γi l k η k }, 1 γkk {bk-j-1 l=1 γi l k η k } ηϕ(η)dη end for return (i 1 , • • • , i d )
Where γ is updated accordingly when the order is changed. Covariance matrices generated from random samples with heavy tails. The various dimension are simulated with the same algorithm and same seed, such that the small ones are subsets of the others. The grey boxplot corresponds to the Gibbs sampler the white to the HMC. The Gibbs sampler seem to have smaller variance and no outliers.

3.D Hamiltonian Monte Carlo

95

Introduction

Bipartite ranking (scoring) amounts to rank (score) data from binary labels. An important problem in its own right, bipartite ranking is also an elegant way to formalise classification: once a score function has been estimated from the data, classification reduces to chooses a particular threshold, which determine to which class is assigned each data-point, according to whether its score is above or below that threshold. It is convenient to choose that threshold only once the score has been estimated, so as to get finer control of the false negative and false positive rates; this is easily achieved by plotting the ROC (Receiver operating characteristic) curve.

A standard optimality criterion for scoring is AUC (Area Under Curve), which measures the area under the ROC curve. AUC is appealing for at least two reasons. First, maximising AUC is equivalent to minimising the L 1 distance between the estimated score and the optimal score. Second, under mild conditions, [START_REF] Cortes | Auc optimization vs. error rate minimization[END_REF] show that AUC for a score s equals the probability that s(X -) < s(X + ) for X -(resp. X + ) a random draw from the negative (resp. positive class). [START_REF] Yan | Optimizing classifier performance via an approximation to the Wilcoxon-Mann-Whitney statistic[END_REF] observed AUC-based classification handles much better skewed classes (say the positive class is much larger than the other) than standard classifiers, because it enforces a small score for all members of the negative class (again assuming the negative class is the smaller one).

One practical issue with AUC maximisation is that the empirical version of AUC is not a continuous function. One way to address this problem is to "convexify" this function, and study the properties of so-obtained estimators [Clémençon et al., 2008a]. We follow instead the PAC-Bayesian approach in this chapter, which consists of using a random estimator sampled from a pseudo-posterior distribution that penalises exponentially the (in our case) AUC risk. It is well known [see e.g. the monograph of [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] that the PAC-Bayesian approach comes with a set of powerful technical tools to establish non-asymptotic bounds; the first part of the chapter derive such bounds. A second advantage however of this approach, as we show in the second part of the chapter, is that it is amenable to powerful Bayesian computational tools, such as Sequential Monte Carlo and Expectation Propagation.

Theoretical bounds from the PAC-Bayesian Approach

Notations

The data D consist in the realisation of n IID (independent and identically distributed) pairs (X i , Y i ) with distribution P , and taking values in

R d × {-1, 1}. Let n + = n i=1 ✶{Y i = +1}, n -= n -n + .
For a score function s : R d → R, the AUC risk and its empirical counter-part may be defined as:

R(s) = P (X,Y ),(X ′ ,Y ′ )∼P [{s(X) -s(X ′ )}(Y -Y ′ ) < 0] , R n (s) = 1 n(n -1) i =j ✶ [{s(X i ) -s(X j )}(Y i -Y j ) < 0] . Let σ(x) = E(Y |X = x), R = R(σ) and Rn = R n (σ). It is well known that σ is the score that minimise R(s), i.e. R(s) ≥ R = R(σ) for any score s.
The results of this section apply to the class of linear scores, s θ (x) = θ, x , where θ, x = θ T x denotes the inner product. Abusing notations, let R(θ) = R(s θ ), R n (θ) = R n (s θ ), and, for a given prior density π ξ (θ) that may depend on some hyperparameter ξ ∈ Ξ, define the Gibbs posterior density (or pseudo-posterior) as

π ξ,γ (θ|D) := π ξ (θ) exp {-γR n (θ)} Z ξ,γ (D) , Z ξ,γ (D) = ˆRd π ξ ( θ) exp -γR n ( θ) d θ
for γ > 0. Both the prior and posterior densities are defined with respect to the Lebesgue measure over R d .

Theoretical bounds from the PAC-Bayesian Approach

Assumptions and general results

Our general results require the following assumptions.

Definition 4.2.1 We say that Assumption Dens(c) is satisfied for c > 0 if

P( X 1 -X 2 , θ ≥ 0, X 1 -X 2 , θ ′ ≤ 0) ≤ c θ -θ ′ for any θ and θ ′ ∈ R d such that θ = θ ′ = 1.
This is a mild Assumption, which holds for instance as soon as (X 1 -X 2 )/ X 1 -X 2 admits a bounded probability density; see the appendix.

E (q θ 1,2 ) 2 ≤ C R(θ) -R 1 κ where q θ i,j = ✶{ θ, X i -X j (Y i -Y j ) < 0} -✶{[σ(X i ) -σ(X j )](Y i -Y j ) < 0} - R(θ) + R.
This assumption was introduced for classification by [START_REF] Mammen | Smooth discrimination analysis[END_REF], and used for ranking by Clémençon et al. [2008b] and [START_REF] Robbiano | Upper bounds and aggregation in bipartite ranking[END_REF] (see also a nice discussion in [START_REF] Lecué | Méthodes d'agrégation: optimalité et vitesses rapides[END_REF]). The larger κ, the less restrictive MA(κ, C). In fact, MA(∞, C) is always satisfied for C = 4. For a noiseless classification task (i.e. σ(X i )Y i ≥ 0 almost surely), R = 0,

E((q θ 1,2 ) 2 ) = Var(q θ 1,2 ) = E[✶{ θ, X 1 -X 2 (Y i -Y j ) < 0}] = R(θ) -R
and MA(1, 1) holds. More generally, MA(1, C) is satisfied as soon as the noise is small; see the discussion in Robiano 2013 (Proposition 5 p. 1256) for a formal statement. From now, we focus on either MA(1, C) or MA(∞, C), C ≥ 1. It is possible to prove convergence under MA(κ, 1) for a general κ ≥ 1, but at the price of complications regarding the choice of γ; see [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF], [START_REF] Alquier | Pac-bayesian bounds for randomized empirical risk minimizers[END_REF] and [START_REF] Robbiano | Upper bounds and aggregation in bipartite ranking[END_REF].

We use the classical PAC-Bayesian methodology initiated by [START_REF] Mcallester | Some pac-bayesian theorems[END_REF]; Shawe-Taylor and Williamson [1997] (see [START_REF] Alquier | Pac-bayesian bounds for randomized empirical risk minimizers[END_REF]; [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] for a complete survey and more recent advances) to get the following results. Proof of these and forthcoming results may be found in the appendix. Let K(ρ, π) denotes the Kullback-Liebler divergence, K(ρ, π) = ´ρ(dθ) log{ dρ dπ (θ)} if ρ << π, ∞ otherwise, and denote M 1 + the set of probability distributions ρ(dθ).

99

Lemma 4.2.1 Assume that MA(1, C) holds with C ≥ 1. For any fixed γ with 0 < γ ≤ (n -1)/(8C), for any ε > 0, with probability at least 1ε on the drawing of the data D,

ˆR(θ)π ξ,γ (θ|D)dθ -R ≤ 2 inf ρ∈M 1 + ˆR(θ)ρ(dθ) -R + 2 K(ρ, π ξ ) + log 4 ε γ . Lemma 4.2.2 Assume MA(∞, C) with C ≥ 1.
For any fixed γ with 0 < γ ≤ (n -1)/8, for any ǫ > 0 with probability 1ǫ on the drawing of D,

ˆR(θ)π ξ,γ (θ|D)dθ -R ≤ inf ρ∈M 1 + ˆR(θ)ρ(dθ) -R + 2 K(ρ, π ξ ) + log 2 ǫ γ + 16γ n -1 .
Both lemmas bound the expected risk excess, for a random estimator of θ generated from π ξ,γ (θ|D).

Independent Gaussian Prior

We now specialise these results to the prior density π ξ (θ) = d i=1 ϕ(θ i ; 0, ϑ), i.e. a product of independent Gaussian distributions N (0, ϑ); ξ = ϑ in this case.

ˆR(θ)π γ (θ|D)dθ -R ≤ 2 inf θ 0 R(θ 0 ) -R + α d log(n) + log 4 ǫ n -1 . Theorem 4.2.4 Assume MA(∞, C), C ≥ 1, Dens(c) c > 0, and take ϑ = 2 d (1 + 1 n 2 d ), γ = C dn log(n)
, there exists a constant α = α(c, C, d) such that for any ǫ > 0, with probability 1ǫ,

ˆR(θ)π γ (θ|D)dθ -R ≤ inf θ 0 R(θ 0 ) -R + α d log(n) + log 2 ǫ √ n .
The proof of these results is provided in the appendix. It is known that, under MA(κ, C), the rate (d/n) κ 2κ-1 is minimax-optimal for classification problems, see [START_REF] Lecué | Méthodes d'agrégation: optimalité et vitesses rapides[END_REF]. Following [START_REF] Robbiano | Upper bounds and aggregation in bipartite ranking[END_REF] we conjecture that this rate is also optimal for ranking problems.

Spike and slab prior for feature selection

The independent Gaussian prior considered in the previous section is a natural choice, but it does not accommodate sparsity, that is, the possibility that only a small subset of the components of X i actually determine the membership to either class. For sparse scenarios, one may use the spike and slab prior of [START_REF] Mitchell | Bayesian variable selection in linear regression[END_REF], George and McCulloch [1993b],

π ξ (θ) = d i=1 [pϕ(θ i ; 0, v 1 ) + (1 -p)ϕ(θ i ; 0, v 0 )] with ξ = (p, v 0 , v 1 ) ∈ [0, 1] × (R + ) 2 ,
and v 0 ≪ v 1 , for which we obtain the following result. Note θ 0 is the number of non-zero coordinates for θ ∈ R d .

Theorem 4.2.5 Assume MA(1, C) holds with C ≥ 1, Dens(c) holds with c > 0, and take p = 1exp(-1/d), v 0 ≤ 1/(2nd log(d)), and γ = (n -1)/(8C). Then there is a constant α = α(C, v 1 , c) such that for any ε > 0, with probability at least 1ε on the drawing of the data D,

ˆR(θ)π γ (dθ|D) -R ≤ 2 inf θ 0 R(θ 0 ) -R + α θ 0 0 log(nd) + log 4 ε 2(n -1) .
Compared to Theorem 4.2.3, the bound above increases logarithmically rather than linearly in d, and depends explicitly on θ 0 , the sparsity of θ. This suggests that the spike and slab prior should lead to better performance than the Gaussian prior in sparse scenarios. The rate θ 0 log(d)/n is the same as the one obtained in sparse regression, see e.g. [START_REF] Bühlmann | Statistics for High-Dimensionnal Data[END_REF].

Finally, note that if v 0 → 0, we recover the more standard prior which assigns a point mass at zero for every component. However this leads to a pseudo-posterior which is a mixture of 2 d components that mix Dirac masses and continuous distributions, and thus which is more difficult to approximate (although see the related remark in Section 4.3.4 for Expectation-Propagation).

Practical implementation of the PAC-Bayesian approach 4.3.1 Choice of hyper-parameters

Theorems 4.2.3, 4.2.4, and 4.2.5 propose specific values for hyper-parameters γ and ξ, but these values depend on some unknown constant C. Two data-driven ways to choose γ and ξ are (i) cross-validation (which we will use for γ), and (ii) (pseudo-)evidence maximisation (which we will use for ξ). The latter may be justified from intermediate results of our proofs in the appendix, which provide an empirical bound on the expected risk:

ˆR(θ)π ξ,γ (θ|D)dθ -R ≤ Ψ γ,n inf ρ∈M 1 + ˆRn (θ)ρ(dθ) -Rn + K(ρ, π) + log 2 ǫ γ with Ψ γ,n ≤ 2.
The right-hand side is minimised at ρ(dθ) = π ξ,γ (θ|D)dθ, and the so-obtained bound is -Ψ γ,n log(Z ξ,γ (D))/γ plus constants. Minimising the upper bound with respect to hyperparameter ξ is therefore equivalent to maximising log Z ξ,γ (D) with respect to ξ. This is of course akin to the empirical Bayes approach that is commonly used in probabilistic machine learning. Regarding γ the minimization is more cumbersome because the dependence with the log(2/ǫ) term and Ψ n,γ , which is why we recommend cross-validation instead. It seems noteworthy that, beside [START_REF] Alquier | Sparse single-index model[END_REF], very few papers discuss the practical implementation of PAC-Bayes, beyond some brief mention of MCMC (Markov chain Monte Carlo). However, estimating the normalising constant of a target density simulated with MCMC is notoriously difficult. In addition, even if one decides to fix the hyperparameters to some arbitrary value, MCMC may become slow and difficult to calibrate if the dimension of the sampling space becomes large. This is particularly true if the target does not (as in our case) have some specific structure that makes it possible to implement Gibbs sampling. The two next sections discuss two efficient approaches that make it possible to approximate both the pseudo-posterior π ξ,γ (θ|D) and its normalising constant, and also to perform cross-validation with little overhead.

Sequential Monte Carlo

Given the particular structure of the pseudo-posterior π ξ,γ (θ|D), a natural approach to simulate from π ξ,γ (θ|D) is to use tempering SMC [Sequential Monte Carlo Del Moral et al., 2006b] that is, define a certain sequence γ 0 = 0 < γ 1 < . . . < γ T , start by sampling from the prior π ξ (θ), then applies successive importance sampling steps, from π ξ,γ t-1 (θ|D) to π ξ,γt (θ|D), leading to importance weights proportional to:

π ξ,γt (θ|D) π ξ,γ t-1 (θ|D) ∝ exp {-(γ t -γ t-1 )R n (θ)} .
When the importance weights become too skewed, one rejuvenates the particles through a resampling step (draw particles randomly with replacement, with probability proportional to the weights) and a move step (move particles according to a certain MCMC kernel).

One big advantage of SMC is that it is very easy to make it fully adaptive. For the choice of the successive γ t , we follow [START_REF] Jasra | On population-based simulation for static inference[END_REF] in solving numerically (5.18) in order to impose that the Effective sample size has a fixed value. This ensures that the degeneracy of the weights always remain under a certain threshold. For the MCMC kernel, we use a Gaussian random walk Metropolis step, calibrated on the covariance matrix of the resampled particles. See Algorithm 19 for a summary.

Algorithm 15 Tempering SMC

Input N (number of particles), τ ∈ (0, 1) (ESS threshold), κ > 0 (random walk tuning parameter)

Init. Sample θ i 0 ∼ π ξ (θ) for i = 1 to N , set t ← 1, γ 0 = 0, Z 0 = 1.
Loop a. Solve in γ t the equation

{ N i=1 w t (θ i t-1 )} 2 N i=1 {w t (θ i t-1 )) 2 } = τ N, w t (θ) = exp[-(γ t -γ t-1 )R n (θ)] (4.1) using bisection search. If γ t ≥ γ T , set Z T = Z t-1 × 1 N N i=1 w t (θ i t-1
) , and stop. b. Resample: for i = 1 to N , draw A i t in 1, . . . , N so that P(A i t = j) = w t (θ j t-1 )/ N k=1 w t (θ k t-1 ); see Algorithm 1 in the appendix. c. Sample θ i t ∼ M t (θ

A i t
t-1 , dθ) for i = 1 to N where M t is a MCMC kernel that leaves invariant π t ; see Algorithm 3 in the appendix for an instance of such a MCMC kernel, which takes as an input S = κ Σ, where Σ is the covariance matrix of the θ

A i t t-1 . d. Set Z t = Z t-1 × 1 N N i=1 w t (θ i t-1 ) .
In our context, tempering SMC brings two extra advantages: it makes it possible to obtain samples from π ξ,γ (θ|D) for a whole range of values of γ, rather than a single value. And it provides an approximation of Z ξ,γ (D) for the same range of γ values, through the quantity Z t defined in Algorithm 19.

Expectation-Propagation (Gaussian prior)

The SMC sampler outlined in the previous section works fairly well, and we will use it as gold standard in our simulations. However, as any other Monte Carlo method, it may be too slow for large datasets. We now turn our attention to EP [Expectation-Propagation Minka, 2001a], a general framework to derive fast approximations to target distributions (and their normalising constants).

First note that the pseudo-posterior may be rewritten as:

π ξ,γ (θ|D) = 1 Z ξ,γ (D) π ξ (θ) × i,j f ij (θ), f ij (θ) = exp [-γ ′ ✶{ θ, X i -X j < 0}]
where γ ′ = γ/n + n -, and the product is over all (i, j) such that Y i = 1, Y j = -1. EP generates an approximation of this target distribution based on the same factorisation:

q(θ) ∝ q 0 (θ) i,j q ij (θ), q ij (θ) = exp{- 1 2 θ T Q ij θ + r T ij θ}.
We consider in the section the case where the prior is Gaussian, as in Section 4.2.3. Then one may set q 0 (θ) = π ξ (θ). The approximating factors are unnormalised Gaussian densities (under a natural parametrisation), leading to an overall approximation that is also Gaussian, but other types of exponential family parametrisations may be considered; see next section and Seeger [2005a]. EP updates iteratively each site q ij (that is, it updates the parameters Q ij and r ij ), conditional on all the sites, by matching the moments of q with those of the hybrid distribution

h ij (θ) ∝ q(θ) f ij (θ) q ij (θ) ∝ q 0 (θ)f ij (θ) (k,l) =(i,j) q kl (θ)
where again the product is over all (k, l) such that Y k = 1, Y l = -1, and (k, l) = (i, j).

We refer to the appendix for a precise algorithmic description of our EP implementation. We highlight the following points. First, the site update is particularly simple in our case:

h ij (θ) ∝ exp{θ T r h ij - 1 2 θ T Q h ij θ} exp [-γ ′ ✶{ θ, X i -X j < 0}] , with r h ij = (k,l) =(i,j) r kl , Q h ij = (k,l) =(i,j) Q kl ,
which may be interpreted as: θ conditional on T (θ) = θ, X i -X j has a d -1-dimensional Gaussian distribution, and the distribution of T (θ) is that of a one-dimensional Gaussian penalised by a step function. The two first moments of this particular hybrid may therefore be computed exactly, and in O(d 2 ) time, as explained in the appendix. The updates can be performed efficiently using the fact that the linear combination (X i -X j )θ is a one dimensional Gaussian. For our numerical experiment we used a parallel version of EP [START_REF] Van Gerven | Efficient Bayesian multivariate fMRI analysis using a sparsifying spatio-temporal prior[END_REF]. The complexity of our EP implementation is O(n + n -d 2 + d 3 ).

Second, EP offers at no extra cost an approximation of the normalising constant Z ξ,γ (D) of the target π ξ,γ (θ|D); in fact, one may even obtain derivatives of this approximated quantity with respect to hyper-parameters. See again the appendix for more details.

Third, in the EP framework, cross-validation may be interpreted as dropping all the factors q ij that depend on a given data-point X i in the global approximation q. This makes it possible to implement cross-validation at little extra cost [START_REF] Opper | Gaussian Processes for Classification: Mean-field Algorithms[END_REF].

Expectation-Propagation (spike and slab prior)

To adapt our EP algorithm to the spike and slab prior of Section 4.2.4, we introduce latent variables Z k = 0/1 which "choose" for each component θ k whether it comes from a slab, or from a spike, and we consider the joint target

π ξ,γ (θ, z|D) ∝ d k=1 B(z k ; p)N (θ k ; 0, v z k ) exp - γ n + n -ij ✶{ θ, X i -X j > 0} .
On top of the n + n -Gaussian sites defined in the previous section, we add a product of d sites to approximate the prior. Following Hernandez-Lobato et al.

[2013], we use

q k (θ k , z k ) = exp z k log p k 1 -p k - 1 2 θ 2 k u k + v k θ k
that is a (un-normalised) product of an independent Bernoulli distribution for z k , times a Gaussian distribution for θ k . Again that the site update is fairly straightforward, and may be implemented in O(d 2 ) time. See the appendix for more details. Another advantage of this formulation is that we obtain a Bernoulli approximation of the marginal pseudo-posterior π ξ,γ (z i = 1|D) to use in feature selection. Interestingly taking v 0 to be exactly zero also yield stable results corresponding to the case where the spike is a Dirac mass. 105

Extension to non-linear scores

To extend our methodology to non-linear score functions, we consider the pseudoposterior

π ξ,γ (ds|D) ∝ π ξ (ds) exp    - γ n + n -i∈D + , j∈D - ✶{s(X i ) -s(X j ) > 0}   
where π ξ (ds) is some prior probability measure with respect to an infinite-dimensional functional class. Let s i = s(X i ), s 1:n = (s 1 , . . . , s n ) ∈ R n , and assume that π ξ (ds) is a GP (Gaussian process) associated to some kernel k ξ (x, x ′ ), then using a standard trick in the GP literature [START_REF] Rasmussen | Gaussian processes for Machine Learning[END_REF], one may derive the marginal (posterior) density (with respect to the n-dimensional Lebesgue measure) of s 1:n as

π ξ,γ (s 1:n |D) ∝ N d (s 1:n ; 0, K ξ ) exp    - γ n + n -i∈D + , j∈D - ✶{s i -s j > 0}   
where N d (s 1:n ; 0, K ξ ) denotes the probability density of the N (0, K ξ ) distribution, and K ξ is the n × n matrix (k ξ (X i , X j )) n i,j=1 . This marginal pseudo-posterior retains essentially the structure of the pseudoposterior π ξ,γ (θ|D) for linear scores, except that the "parameter" s 1:n is now of dimension n. We can apply straightforwardly the SMC sampler of Section 4.B.1, and the EP algorithm of 4.B.2, to this new target distribution. In fact, for the EP implementation, the particular simple structure of a single site:

exp [-γ ′ ✶{s i -s j > 0}]
makes it possible to implement a site update in O(1) time, leading to an overall complexity O(n + n -+ n 3 ) for the EP algorithm.

Theoretical results for this approach could be obtained by applying lemmas from e.g. van der Vaart and van Zanten [2009], but we leave this for future study. We now compare our PAC-Bayesian approach (computed with EP) with Bayesian logistic regression (to deal with non-identifiable cases), and with the rankboost algorithm [START_REF] Freund | An efficient boosting algorithm for combining preferences[END_REF]] on different datasets1 ; note that [START_REF] Cortes | Auc optimization vs. error rate minimization[END_REF] showed that the function optimised by rankbook is AUC.

Numerical Illustration

As mentioned in Section 4.B, we set the prior hyperparameters by maximizing the evidence, and we use cross-validation to choose γ. To ensure convergence of EP, when dealing with difficult sites, we use damping [Seeger, 2005a]. The GP version of the algorithm is based on a squared exponential kernel. Table 5.2 summarises the results; balance refers to the size of the smaller class in the data (recall that the AUC criterion is particularly relevant for unbalanced classification tasks), EP-AUC (resp. GPEP-AUC) refers to the EP approximation of the pseudo-posterior based on our Gaussian prior (resp. Gaussian process prior). See also Figure 2 for ROC curve comparisons, and Table 2 in the appendix for a CPU time comparison.

Note how the GP approach performs better for the colon data, where the number of covariates ( 2000) is very large, but the number of observations is only 40. It seems also that EP gives a better approximation in this case because of the lower dimensionality of the pseudo-posterior (Figure 4.2b).

Finally, we also investigate feature selection for the DNA dataset (180 covariates) using a spike and slab prior. The regularization plot (4.3a) shows how certain coefficients shrink to zero as the spike's variance v 0 goes to zero, allowing for some The Glass dataset has originally more than two classes. We compare the "silicon" class against all others.

sparsity. The aim of a positive variance in the spike is to absorb negligible effects into it [START_REF] Ročková | Emvs: The EM approach to bayesian variable selection[END_REF]. We observe this effect on figure 4.3a where one of the covariates becomes positive when v 0 decreases. .2: Some ROC curves associated to the example described in a more systematic manner in table 5.2. In black is always the PAC version.

-0.3 -0.2 -0.1 0.0 0.1 1e-04 1e-02 v0 θ (a) Regularization plot • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • •••••••••• • • •••• • • ••• • •• • • •• • • •• •••• • • •• •• • • • • • • • • • • • • • • • • • • •• • • • •• • •• • • • • • • • • ••• • • • • • • • • • • • • •• • • •• • • • •• • • • • ••• • • ••• •• • ••• •••• •••••••• • •• • • • • •• • •• •••• • • • • • •••• •••••• • • •••••• • • ••• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • -0.1 0.0 0 50 100 150 V1 θ (b) Estimate

Conclusion

The combination of the PAC-Bayesian theory and Expectation-Propagation leads to fast and efficient AUC classification algorithms, as observed on a variety of datasets, some of them very unbalanced. Future work may include extending our approach to more general ranking problems (e.g. multi-class), establishing nonasymptotic bounds in the nonparametric case, and reducing the CPU time by considering only a subset of all the pairs of datapoints.

109

We will only upper bound the first term in the r.h.s., as the upper bound for the second term may be obtained exactly in the same way (just replace q θ i,j by -q θ i,j ). Now, use Hoeffding's decomposition [START_REF] Hoeffding | Probability Inequalities for Sums of Random Variables[END_REF]: this is the technique used by Hoeffding to prove inequalities on U-statistics. Hoeffding proved that

U n = 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)
where the sum is taken over all the permutations π of {1, . . . , n}. Jensen's inequality leads to

E exp[γU n ] = E exp   γ 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ 1 n! π E exp   γ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   .
We now use, for each of the terms in the sum, Massart's version of Bernstein's inequality [START_REF] Massart | Concentration Inequalities and Model Selection[END_REF] (ineq. (2.21) in Chapter 2, the assumption is checked by

q θ π(i),π(i+⌊ n 2 ⌋) ∈ [-2, 2] so E((q θ π(i),π(i+⌊ n 2 ⌋) ) k ) ≤ E((q θ π(i),π(i+⌊ n 2 ⌋) ) 2 )2 k-2
). We obtain:

E exp   γ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ exp   E((q θ π(1),π(1+⌊ n 2 ⌋) ) 2 ) γ 2 ⌊ n 2 ⌋ 2 1 -2 γ ⌊ n 2 ⌋   .
First, note that we have the inequality ⌊ n 2 ⌋ ≥ (n -1)/2. Then, remark that as the pairs (X i , Y i ) are iid, we have E((q θ π(1),π(1+⌊ n 2 ⌋) ) 2 ) = E((q θ 1,2 ) 2 ) so we have a simpler inequality

E exp   γ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ exp E((q θ 1,2 ) 2 ) γ 2 n-1 1 -4γ n-1 .
This ends the proof of the proposition.

The following proposition is also of use in the proof of lemma 2.1.

Proposition 4.A.2 For any measure ρ ∈ M 1 + (Θ) and any measurable function

h : θ → R such that ´exp(h(θ))π(dθ) < ∞, we have log ˆexp(h(θ))π(θ) = sup ρ∈M 1 + ˆh(θ)ρ(dθ) -K(ρ, π) .

111

In addition if h is bounded by above on the support of π the supremum is reached for the Gibbs distribution, ρ(dθ) ∝ exp (h(θ)) π(dθ).

Proof: e.g. [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF].

Proof of Lemma 2.1 From the proof of Proposition 4.A.1, and using the shorthand q θ for q θ 1,2 , we deduce

E exp{ρ γ(R n (θ) -Rn -R(θ) + R)} + η(θ) ≤ exp γ 2 n -1 ρ (Eq 2 θ ) (1 -4 γ n-1 ) + ρ (η(θ)) . (4.2)
Using proposition 4.A.2, and the fact that e x ≥ ✶{x ≥ 0} we have that

P{ sup ρ∈M 1 + (Θ) ρ γ(R n (θ) -Rn -R(θ) + R) -η(θ) -K(ρ, π) ≥ 0} ≤ E π{exp{ρ γ(R n (θ) -Rn -R(θ) + R) -η(θ)} } = π E{exp{ρ γ(R n (θ) -Rn -R(θ) + R) -η(θ)} } , by Fubini ≤ π exp γ 2 ρ(Eq 2 θ ) (n -1)(1 -4γ n-1 ) -ρ(η(θ))
, using (4.2).

In the following we take η(θ) = log 1 ǫ + γ 2 n-1 ρ(Eq 2 θ ) (1-4 γ n-1 ) leading to the following result with probability at least 1ǫ, ∀ρ ∈ M 1 + (Θ):

ρ(R n (θ)) -Rn ≤ ρ(R(θ)) -R + K(ρ, π) + log 1 ǫ γ + γ n -1 ρ(Eq 2 θ ) (1 -4 γ n-1 ) . (4.3)
Under MA(1, C) we can write:

ρ(R n (θ)) -Rn ≤ 1 + γC n -1 1 (1 -4 n-1 ) ρ(R(θ)) -R + K(ρ, π) + log 1 ǫ γ .
Using Bernstein's inequality in the symmetric case, with probability 1ǫ we can assert that:

1 - γC n -1 1 (1 -γ 4 n-1 ) ρ(R(θ)) -R ≤ ρ(R n (θ)) -Rn + K(ρ, π) + log 1 ǫ γ .
The latter is true in particular for ρ = π(θ|S), the Gibbs posterior:

1 - γC n -1 1 (1 -γ 4 n-1 ) ˆΘ R(θ)π γ (dθ|D) -R ≤ inf ρ∈M 1 + ρ(R n (θ)) -Rn + K(ρ, π) + log 1 ǫ γ .
Making use of equation ( 4.3) and the fact that γ ≤ (n -1)/8C we have with probability 1 -2ǫ:

ˆΘ R n (θ)π γ (dθ|D) -Rn ≤ 2 inf ρ∈M 1 + ρ(R(θ)) -R + 2 K(ρ, π) + log 1 ǫ γ .
Lemma 2.1 gives some approximately correct finite sample bound under hypothesis MA(1, C). It is easy to extend those results to the more general case of MA(∞, C). Note in particular that this assumption is always satisfied for C = 4.

Proof of Lemma 2.2 First consider in our case that, the margin assumption is always true for C = 4, E(q 2 θ ) ≤ 4, the rest of the proof is similar to that of lemma 2.1. From equation (4.3) with the above hypothesis:

ρ(R n (θ)) -Rn ≤ ρ(R(θ)) -R + K(ρ, π) + log 1 ǫ γ + 4γ n -1 1 (1 -4
n-1 ) From the Bernstein inequality with in the symmetric case we get with probability 1ǫ:

ρ(R(θ)) -R ≤ ρ(R n (θ)) -Rn + K(ρ, π) + log 1 ǫ γ + 4γ n -1 1 (1 -4
n-1 ) We get, after noting that the Gibbs posterior can be written as an infimum (Legendre transform), with probability 1 -2ǫ:

ˆ(R(θ)π γ (dθ|D) -R ≤ inf ρ∈M 1 + (Θ) ρ(R(θ)) -R + K(ρ, π) + log 1 ǫ γ + 16γ n -1 (we also used γ ≤ (n -1)/8).
The two above lemma depend on some class complexity K(ρ, π). The latter can be specialized to different choice of prior measure π. In the following we propose two specifications to a Gaussian prior and a spike and slab prior.

4.A.3 Proof of Theorem 2.3 (Independent Gaussian prior)

For any θ 0 ∈ R p with θ 0 = 1 and δ > 0 we put

ρ θ 0 ,δ (dθ) ∝ 1 θ-θ 0 ≤δ π(dθ).
Then we have, from Lemma 2.1, with probability at least 1ε,

ˆR(θ)π γ (dθ|D) -R ≤ 2 inf θ 0 ,δ ˆR(θ)ρ θ 0 ,δ (dθ) -R + 16C K(ρ θ 0 ,δ , π) + log 4 ε (n -1)
First, note that

R(θ) = E (✶{ θ, X -X ′ (Y -Y ′ ) < 0}) = E (✶{ θ 0 , X -X ′ (Y -Y ′ ) < 0}) + E (✶{ θ, X -X ′ (Y -Y ′ ) < 0} -✶{ θ 0 , X -X ′ (Y -Y ′ ) < 0}) ≤ R(θ 0 ) + P(sign θ, X -X ′ (Y -Y ′ ) = sign θ 0 , X -X ′ (Y -Y ′ )) = R(θ 0 ) + P(sign θ, X -X ′ = sign θ 0 , X -X ′ ) ≤ R(θ 0 ) + c θ θ -θ 0 ≤ R(θ 0 ) + 2c θ -θ 0 .
As a consequence ´R(θ)ρ θ 0 ,δ (dθ) ≤ R(θ 0 ) + 2cδ. The next step is to calculate K(ρ θ 0 ,δ , π). We have

K(ρ θ 0 ,δ , π) = log 1 π ({θ : θ -θ 0 ≤ δ}) .
Assuming that θ 0,1 > 0 (the proof is exactly symmetric in the other case)

-K(ρ θ 0 ,δ , π) = log π {θ : d i=1 (θ i -θ 0,i ) 2 ≤ δ 2 } ≥ d log π {θ : (θ 1 -θ 0,1 ) 2 ≤ δ 2 d } ≥ d log ˆθ0,1 √ ϑ + δ √ ϑd θ 0,1 √ ϑ -δ √ ϑd ϕ (0,1) (x)dx ≥ d log δ 2 √ ϑd ϕ θ 0,1 √ ϑ + δ √ ϑd ≥ d log δ 2 √ ϑd ϕ 1 √ ϑ + δ √ ϑd = d log δ 2 √ 2πϑd exp - 1 2 1 √ ϑ + δ √ ϑd 2 ≥ d log δ 2 √ 2πϑd exp - 1 ϑ - δ 2 ϑd K(ρ θ 0 ,δ , π) ≤ -d log{δ} + d 2 log{8πϑd} + 1 ϑ + δ 2 ϑd
And we can plug the equation above in the result of lemma 2.

1 with δ = 1 n ˆR(θ)π γ (θ|D) -R ≤ 2 inf θ 0 R(θ 0 ) -R + 2c 1 n + 2 γ d log{n} + d 2 log{8πϑd} + 1 ϑ + 1 n 2 ϑd + log 4 ǫ
Any γ = O(n) will lead to a convergence result. Taking γ = (n -1)/8C and optimizing in ϑ we obtain a variance of ϑ =

2(1+ 1 n 2 d ) d .

4.A.4 Proof of Theorem 2.4 (Independent Gaussian prior)

As was done for the previous lemmas we can lift the MA(∞, C) and use the lemma 2.2 instead, which gives rise to Theorem 2.4.

Use Lemma 2.2 and the same steps as in the proof of Theorem 4.2.3, optimize w.r.t. γ and ϑ to get the result.

We show the same kind of result in the following but for spike and slab priors.

4.

A.5 Proof of Theorem 2.5 (Spike and slab prior for feature selection)

As for the proof of theorem 4.2.3 we start by defining, for any θ 0 ∈ R p with θ 0 = 1 and δ > 0,

ρ θ 0 ,δ (dθ) ∝ 1 θ-θ 0 ≤δ π(dθ)
so that in the end, by a similar argument as previously it remains only to upper bound the following quantity,

K(ρ θ 0 ,δ , π) = log 1 π ({θ : θ -θ 0 ≤ δ}) .
Let π 0 denote the probability distribution such that the θ i are iid N (0, v 0 ). So:

-K(ρ θ 0 ,δ , π) = log π θ : d i=1 (θ i -θ 0,i ) 2 ≤ δ 2 ≥ log π θ : ∀i, (θ i -θ 0,i ) 2 ≤ δ 2 d = i:θ 0,i =0 log π (θ i -θ 0,i ) 2 ≤ δ 2 d + log π ∀i with θ 0,i = 0, θ 2 i < δ 2 d ≥ i:θ 0,i =0 log π (θ i -θ 0,i ) 2 ≤ δ 2 d + log π 0 ∀i with θ 0,i = 0, θ 2 i < δ 2 d + d log(1 -p) = i:θ 0,i =0 log π (θ i -θ 0,i ) 2 ≤ δ 2 d + log 1 -π 0 ∃i, θ 0,i = 0, θ 2 i > δ 2 d + d log(1 -p) ≥ i:θ 0,i =0 log π (θ i -θ 0,i ) 2 ≤ δ 2 d + log 1 - i:θ i =0 π 0 θ 2 i > δ 2 d + d log(1 -p).
Assume first that i is such that θ 0,i = 0. Then:

π 0 θ 2 i > δ 2 d = π 0 θ i √ v 0 > δ √ v 0 d ≤ exp - δ 2 2v 0 d ,
and so

i:θ 0,i =0 π 0 θ 2 i > δ 2 d ≤ d exp - δ 2 2v 0 d ≤ 1 2
as soon as v 0 ≤ δ 2 /(2d log(d)). Then, assume that i is such that θ 0,i = 0. Now assume that θ 0,i > 0 (the proof is exactly symmetric if θ 0,i < 0):

π θ : (θ i -θ 0,i ) 2 ≤ δ 2 d ≥ p ˆθ0,i √ v 1 + δ √ v 1 d θ 0,i √ v 1 -δ √ v 1 d ϕ (0,1) (x)dx ≥ pδ 2 √ v 1 d ϕ θ 0,i √ v 1 + δ √ v 1 d ≥ pδ 2 √ v 1 d ϕ 1 √ v 1 + δ √ v 1 d = pδ 2 √ 2πv 1 d exp - 1 2 1 √ v 1 + δ √ v 1 d 2 ≥ pδ 2 √ 2πv 1 d exp - 1 v 1 - δ 2 v 1 d .
Putting everything together:

K(ρ θ 0 ,δ , π) ≤ -θ 0 0 log pδ 2 √ 2πv 1 d exp - 1 v 1 - δ 2 v 1 d + log(2) + d log 1 1 -p = θ 0 0 log 2 √ 2πv 1 d pδ + 1 v 1 + δ 2 v 1 d + log(2) + d log 1 1 -p .
So, we have:

ˆR(θ)π γ (dθ|D) -R ≤ 2 inf θ 0 ,δ R(θ 0 ) -R + 2cδ + 16C θ 0 0 log 2 √ 2πv 1 d pδ + 1 v 1 + δ 2 v 1 d + log(2) + d log 1 1-p + log 4 ε (n -1)
4.B Practical implementation of the PAC-Bayesian approach

4.B.1 Sequential Monte Carlo

The resampling scheme we use in our SMC sampler is systematic resampling, see Algorithm 20.

To move the particles while leaving invariant the current target π ξ,γ (θ|D), we use the standard random walk Metropolis strategy, but scaled to the current set of particles, as outlined by Algorithm 17.

4.B.2 Expectation-Propagation (Gaussian prior)

EP aims at approximating posterior distributions of the form,

π(θ|D) = 1 Z π P 0 (θ) n i=1 t i (θ)
Algorithm 18 parallel EP for Gaussian Prior Input: ϑ, γ

Output: m and V

Init: V ← Σ, m ← 0 Untill Convergence Do
For all sites (i, j)Do in parallel a. Compute the cavity moments m \ij , V \ij b. Compute the 1st and 2nd order moments of q \ij (s ij )t ij (s i j)

c. Update K ij and h ij End For Update V = (Σ -1 + ij (X i -X j ) T (X i -X j )K ij ) -1 , m = V ( ij (X i -X j )h ij ) End While where q k (θ k , z k ) ∝ Ber(z k , p k )N (θ k ; m k , σ 2 k )
, and t ij (θ) is as in the previous section. The cavity moments are easy to compute as the approximation is Gaussian in θ and Bernoulli in z. In both cases we can deduce cavity moments because division is stable inside those classes of functions. We get some distribution Q \k (θ k ) ∝ Ber(z k ; p \k )N (θ k ; m \k , σ 2,\k ). We can compute the normalizing constant of the distribution Q \ij (θ)t k (θ k , z k ), namely,

Z k = p \k ˆN (θ k ; 0, v 0 )N (θ k ; m \k , σ 2,\k )dθ k +(1-p \k ) ˆN (θ k ; 0, v 0 )N (θ k ; m \k , σ 2,\k )dθ k
Where we can find the update by computing the derivatives of log Z k with respect to p \k , m \k and σ 2,\k Initialization for the Gaussian is done to a given Σ 0 that will be subtracted later on. The initial p k s are taken such that the approximation equals the prior p at the first iteration. see that the mode is well approximated however the variance is slightly underestimated.

4.C Numerical illustration

In Table 4.C.1 we show the CPU times in seconds, on all dataset studied. Experiments where run with a i7-3720QM CPU @ 2.60GHz intel processor with 6144 KB cache. Our linear model is overall faster on those datasets. A caveat is that Rankboost is implemented in Matlab, while our implementation is in C. The Gibbs posterior has also appeared in other places, and under different motivations: in Econometrics, as a way to avoid direct maximisation in moment estimation [START_REF] Chernozhukov | An MCMC approach to classical estimation[END_REF]; and in Bayesian decision theory, as as way to define a Bayesian posterior distribution when no likelihood has been specified [START_REF] Bissiri | A general framework for updating belief distributions[END_REF]. Another well-known connection, although less directly useful (for Statistics), is with thermodynamics, where r n is interpreted as an energy function, and λ as the inverse of a temperature.

Dataset Covariates

Whatever the perspective, estimators derived from Gibbs posteriors usually show excellent performance in diverse tasks, such as classification, regression, ranking, and so on, yet their actual implementation is still far from routine. The usual recommendation [START_REF] Alquier | Sparse single-index model[END_REF][START_REF] Dalalyan | Sparse regression learning by aggregation and Langevin Monte-Carlo[END_REF][START_REF] Guedj | PAC-Bayesian estimation and prevision in sparse additive models[END_REF] is to sample from a Gibbs posterior using MCMC [Markov chain Monte Carlo, see e.g. [START_REF] Green | Bayesian computation: a perspective on the current state, and sampling backwards and forwards[END_REF]; but constructing an efficient MCMC sampler is often difficult, and even efficient implementations are often too slow for practical uses when the dataset is very large.

In this chapter, we consider instead VB (Variational Bayes) approximations, which have been initially developed to provide fast approximations of 'true' posterior distributions (i.e. Bayesian posterior distributions for a given model); see [START_REF] Jordan | An introduction to variational methods for graphical models[END_REF]; [START_REF] Mackay | Information theory, inference and learning algorithms[END_REF] and Chap. 10 in Bishop [2006a].

Our main results are as follows: when PAC-Bayes bounds are available -mainly, when a strong concentration inequality holds -replacing the Gibbs posterior by a variational approximation does not affect the rate of convergence to the best possible prediction, on the condition that the Küllback-Leibler divergence between the posterior and the approximation is itself controlled in an appropriate way.

We also provide empirical bounds, which may be computed from the data so as to ascertain the actual performance of estimators obtained by variational approximation. All the results gives strong incentives, we believe, to recommend Variational Bayes as the default approach to approximate Gibbs posteriors.

The rest of the chapter is organized as follows. In Section 5.2 we introduce the notations and assumptions. In Section 5.3 we introduce variational approximations and the corresponding algorithms. The main results are provided in general form in Section 5.4: in Subsection 5.4.1, we give results under the assumption that a Hoeffding type inequality holds (slow rates) and in Subsection 5.4.2, we give results under the assumption that a Bernstein type inequality holds (fast rates). Note that for the sake of shortness, we will refer to these settings as "Hoeffding assumption" and "Bernstein assumption" even if this terminology is non standard. We then apply these results in various settings: classification (Section 5.5), convex classification (Section 5.6), ranking (Section 5.7), and matrix completion (Section 5.8). In each case, we show how to specialise the general results of Section 5.4 to the considered application, so as to obtain the properties of the VB 124 5.2 PAC-Bayesian framework approximation, and we also discuss its numerical implementation. All the proofs are collected in the Appendix.

PAC-Bayesian framework

We observe a sample (X 1 , Y 1 ), . . . , (X n , Y n ), taking values in X × Y, where the pairs (X i , Y i ) have the same distribution P . We will assume explicitly that the (X i , Y i )'s are independent in several of our specialised results, but we do not make this assumption at this stage, as some of our general results, and more generally the PAC-Bayesian theory, may be extended to dependent observations; see e.g. [START_REF] Alquier | Prediction of quantiles by statistical learning and application to GDP forecasting[END_REF]. The label set Y is always a subset of R. A set of predictors is chosen by the statistician: {f θ : X → R, θ ∈ Θ}. For example, in linear regression, we may have: f θ (x) = θ, x , the inner product of X = R d , while in classification, one may have f θ (x) = I θ,x >0 ∈ {0, 1}.

We assume we have at our disposal a risk function R(θ); typically R(θ) is a measure of the prevision error. We set R = R(θ), where θ ∈ arg min Θ R; i.e. f θ is an optimal predictor. We also assume that the risk function R(θ) has an empirical counterpart r n (θ), and set r n = r n (θ). Often, R and r n are based on a loss function

ℓ : R 2 → R; i.e. R(θ) = E[ℓ(Y, f θ (X))] and r n (θ) = 1 n n i=1 ℓ(Y i , f θ (X i ))
. (In this chapter, the symbol E will always denote the expectation with respect to the (unknown) law P of the (X i , Y i )'s.) There are situations however (e.g. ranking), where R and r n have a different form.

We define a prior probability measure π(•) on the set Θ (equipped with the standard σ-algebra for the considered context), and we let M 1 + (Θ) denote the set of all probability measures on Θ. The pseudo-posterior ρλ (also known as the Gibbs posterior, [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF][START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF], or the exponentially weighted aggregate, [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]) plays a central role in the PAC-Bayesian approach. It is obtained as the distribution that minimises the upper bound of a certain oracle inequality applied to random estimators. Practical estimators (predictors) may be derived from the pseudoposterior, by e.g. taking the expectation, or sampling from it. Of course, when exp[-λr n (θ)] may be interpreted as the likelihood of a certain model, ρλ becomes a Bayesian posterior distribution, but we will not restrict our attention to this particular case.

The following 'theoretical' counterpart of ρλ will prove useful to state results.

Definition 5.2.2 We define, for any λ > 0, π λ as π λ (dθ) = exp [-λR(θ)] ´exp[-λR]dπ π(dθ).

We will derive PAC-Bayesian bounds on predictions obtained by variational approximations of ρλ under two types of assumptions: a Hoeffding-type assumption, from which we may deduce slow rates of convergence (Subsection 5.4.1), and a Bernstein-type assumption, from which we may obtain fast rates of convergence (Subsection 5.4.2).

Definition 5.2.3 We say that a Hoeffding assumption is satisfied for prior π when there is a function f and an interval I ⊂ R * + such that, for any λ ∈ I,

π (E exp {λ[R(θ) -r n (θ)]}) π (E exp {λ[r n (θ) -R(θ)]}) ≤ exp [f (λ, n)] .
(5.1) Inequality (5.1) can be interpreted as an integrated version (with respect to π) of Hoeffding's inequality, for which f (λ, n) ≍ λ 2 /n. In many cases the loss will be bounded uniformly over θ; then Hoeffding's inequality will directly imply (5.1). The expectation with respect to π in (5.1) allows us to treat some cases where the loss is not upper bounded by specifying a prior with sufficiently light tails. Definition 5.2.4 We say that a Bernstein assumption is satisfied for prior π when there is a function g and an interval I ⊂ R * + such that, for any λ ∈ I,

π E exp λ[R(θ) -R] -λ[r n (θ) -r n ] π E exp λ[r n (θ) -r n ] -λ[R(θ) -R] ≤ π exp g(λ, n)[R(θ) -R] .
(5.2)

This assumption is satisfied for example by sums of i.i.d. sub-exponential random variables, see Subsection 2.4 p. 27 in [START_REF] Boucheron | Concentration Inequalities[END_REF], when a margin assumption on the function R(•) is satisfied [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF]. This is discussed in Section 5.4.2. Again, extensions beyond the i.i.d. case are possible, see e.g. [START_REF] Wintenberger | Deviation inequalities for sums of weakly dependent time series[END_REF] for a survey and new results. In all these examples, the important feature of the function g that we will use to derive rates of convergence is the fact that there is a constant c > 0 such that when λ = cn, g(λ, n) = g(cn, n) ≍ n.

As mentioned previously, we will often consider

r n (θ) = 1 n n i=1 ℓ(Y i , f θ (X i ))
, however, the previous assumptions can also be satisfied when r n (θ) is a U-statistic, using Hoeffding's decomposition of U-statistics combined with the corresponding inequality for sums of independent variables [START_REF] Hoeffding | Probability Inequalities for Sums of Random Variables[END_REF]. This idea comes from Clémençon et al. [2008a] and we will use it in our ranking application. 126

Numerical approximations of the pseudo-posterior

Remark 5.2.1 We could consider more generally inequalities of the form

π E exp λ[R(θ) -R] -λ[r n (θ) -r n ] π E exp λ[r n (θ) -r n ] -λ[R(θ) -R] ≤ π exp g(λ, n)[R(θ) -R] κ
that allow to use the more general form of the margin assumption of [START_REF] Mammen | Smooth discrimination analysis[END_REF]; [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF]. PAC-Bayes bounds in this context are provided by [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF]. However, the techniques involved would require many pages to be described so we decided to focus on the cases κ = 0 and κ = 1 to keep the exposition simple.

5.3 Numerical approximations of the pseudo-posterior

Monte Carlo

As already explained in the introduction, the usual approach to approximate ρλ is MCMC (Markov chain Monte Carlo) sampling. [START_REF] Ridgway | Computation of Gaussian orthant probabilities in high dimension[END_REF] proposed tempering SMC (Sequential Monte Carlo, e.g. Del Moral et al. [2006b]) as an alternative to MCMC to sample from Gibbs posteriors: one samples sequentially from ρλt , with 0 = λ 0 < • • • < λ T = λ where λ is the desired temperature. One advantage of this approach is that it makes it possible to contemplate different values of λ, and choose one by e.g. cross-validation. Another advantage is that such an algorithm requires little tuning; see Appendix B for more details on the implementation of tempering SMC. We will use tempering SMC as our gold standard in our numerical studies. SMC and related Monte Carlo algorithms tend to be too slow for practical use in situations where the sample size is large, the dimension of Θ is large, or f θ is expensive to compute. This motivates the use of fast, deterministic approximations, such as Variational Bayes, which we describe in the next section.

Variational Bayes

Various versions of VB (Variational Bayes) have appeared in the literature, but the main idea is as follows. We define a family F ⊂ M 1 + (Θ) of probability distributions that are considered as tractable. Then, we define the VB-approximation of ρλ : ρλ .

127

The difficulty is to find a family F (a) which is large enough, so that ρλ may be close to ρλ , and (b) such that computing ρλ is feasible. We now review two types of families popular in the VB literature.

• Mean field VB: for a certain decomposition Θ = Θ 1 × • • • × Θ d , F is the set of product probability measures

F MF = ρ ∈ M 1 + (Θ) : ρ(dθ) = d i=1 ρ i (dθ i ), ∀i ∈ {1, . . . , d}, ρ i ∈ M 1 + (Θ i ) .
(5.

3) The infimum of the KL divergence K(ρ, ρλ ), relative to ρ = i ρ i satisfies the following fixed point condition [Bishop, 2006a;Parisi, 1988, Chap. 10]:

∀j ∈ {1, • • • , d} ρ j (dθ j ) ∝ exp ˆ{-λr n (θ) + log π(θ)} i =j ρ i (dθ i ) π(dθ j ).
(5.4) This leads to a natural algorithm were we update successively every ρ j until stabilization.

• Parametric family:

F P = ρ ∈ M 1 + (Θ) : ρ(dθ) = f (θ; m)dθ, m ∈ M ;
and M is finite-dimensional; say F P is the family of Gaussian distributions (of dimension d). In this case, several methods may be used to compute the infimum. As above, one may used fixed-point iteration, provided an equation similar to (5.4) is available. Alternatively, one may directly maximize ´log[exp[-λr n (θ)] dπ dρ (θ)]ρ(dθ) with respect to paramater m, using numerical optimization routines. This approach was used for instance in Hoffman et al. [2013] with combination of some stochastic gradient descent to perform inference on a latent Dirichlet allocation model. See also e.g. Emtiyaz [START_REF] Emtiyaz Khan | Fast dual variational inference for non-conjugate latent gaussian models[END_REF]; [START_REF] Khan | Decoupled variational Gaussian inference[END_REF] for efficient algorithms for Gaussian variational approximation.

In what follows (Subsections 5.4.1 and 5.4.2) we provide tight bounds for the prevision risk of ρλ . This leads to the identification of a condition on F such that the risk of ρλ is not worse than the risk of ρλ . We will make this condition explicit in various examples, using either mean field VB or parametric approximations.

Remark 5.3.1 An useful identity, obtained by direct calculations, is: for any ρ ≪ π, log ˆexp [-λr n (θ)] π(dθ) = -λ ˆrn (θ)ρ(dθ) -K(ρ, π) + K(ρ, ρλ ).

(5.5)

Since the left hand side does not depend on ρ, one sees that ρλ , which minimises K(ρ, ρλ ) over F, is also the minimiser of:

ρλ = arg min ρ∈F ˆrn (θ)ρ(dθ) + 1 λ K(ρ, π)
This equation will appear frequently in the sequel in the form of an empirical upper bound.

General results

This section gives our general results, under either a Hoeffding Assumption (Definition 5.2.3) or a Bernstein Assumption (Definition 5.2.4), on risks bounds for the variational approximation, and how it relates to risks bounds for Gibbs posteriors. These results will be specialised to several learning problems in the following sections.

Bounds under the Hoeffding assumption Empirical bounds

Theorem 5.4.1 Under the Hoeffding assumption (Definition 5.2.3), for any ε > 0, with probability at least 1ε we have simultaneously for any ρ ∈ M 1 + (Θ),

ˆRdρ ≤ ˆrn dρ + f (λ, n) + K(ρ, π) + log 1 ε λ .
This result is a simple variant of a result in [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] but for the sake of completeness, its proof is given in Appendix 5.A. It gives us an upper bound on the risk of both the pseudo-posterior (take ρ = ρλ ) and its variational approximation (take ρ = ρλ ). These bounds may be be computed from the data, and therefore provide a simple way to evaluate the performance of the corresponding procedure, in the spirit of the first PAC-Bayesian inequalities [START_REF] Mcallester | PAC-Bayesian model averaging[END_REF][START_REF] Mcallester | Some pac-bayesian theorems[END_REF][START_REF] Shawe-Taylor | A PAC analysis of a Bayesian estimator[END_REF]]. However, this bound do not provide the rate of convergence of these estimators. For this reason, we also provide oracle-type inequalities.

Oracle-type inequalities

Another way to use PAC-Bayesian bounds is to compare ´Rdρ λ to the best possible risk, thus linking this approach to oracle inequalities. This is the point of view developed in [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF][START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF]; [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF].

129

Theorem 5.4.2 Assume that the Hoeffding assumption is satisfied (Definition 5.2.3). For any ε > 0, with probability at least 1ε we have simultaneously

ˆRdρ λ ≤ B λ (M 1 + (Θ)) := inf ρ∈M 1 + (Θ) ˆRdρ + 2 f (λ, n) + K(ρ, π) + log 2 ε λ and ˆRdρ λ ≤ B λ (F) := inf ρ∈F ˆRdρ + 2 f (λ, n) + K(ρ, π) + log 2 ε λ .
Moreover,

B λ (F) = B λ (M 1 + (Θ)) + 2 λ inf ρ∈F K(ρ, π λ 2 )
where we remind that π λ is defined in Definition 5.2.2.

In this way, we are able to compare ´Rdρ λ to the best possible aggregation procedure in M 1 + (Θ) and ´Rdρ λ to the best aggregation procedure in F. More importantly, we are able to obtain explicit expressions for the right-hand side of these inequalities in various models, and thus to obtain rates of convergence. This will be done in the remaining sections. This leads to the second interest of this result: if there is a λ = λ(n) that leads to B λ (M 1 + (Θ)) ≤ R + s n with s n → 0 for the pseudo-posterior ρλ , then we only have to prove that there is a ρ ∈ F such that K(ρ, π λ )/λ ≤ cs n for some constant c > 0 to ensure that the VB approximation ρλ also reaches the rate s n .

We will see in the following sections several examples where the approximation does not deteriorate the rate of convergence. But first let us show the equivalent oracle inequality under the Bernstein assumption.

Bounds under the Bernstein assumption

In this context the empirical bound on the risk would depend on the minimal achievable risk rn , and cannot be computed explicitly. We give the oracle inequality for both the Gibbs posterior and its VB approximation in the following theorem.

Theorem 5.4.3 Assume that the Bernstein assumption is satisfied (Definition 5.2.4). Assume that λ > 0 satisfies λg(λ, n) > 0. Then for any ε > 0, with probability at least 1ε we have simultaneously:

ˆRdρ λ -R ≤ B λ M 1 + (Θ) , ˆRdρ λ -R ≤ B λ (F),
where, for either

A = M 1 + (Θ) or A = F, B λ (A) = 1 λ -g(λ, n) inf ρ∈A [λ + g(λ, n)] ˆ(R -R)dρ + 2K(ρ, π) + 2 log 2 ε .
In addition,

B λ (F) = B λ M 1 + (Θ) + 2 λ -g(λ, n) inf ρ∈F K ρ, π λ+g(λ,n) 2 .
The main difference with Theorem 5.4.2 is that the function R(•) is replaced by R(•) -R. This is well known way to obtain better rates of convergence.

Application to classification

Preliminaries

In all this section, we assume that Y = {0, 1} and we consider linear classification:

Θ = X = R d , f θ (x) = 1 θ,x ≥0 . We put r n (θ) = 1 n n i=1 1 {f θ (X i ) =Y i } , R(θ) = P(Y = f θ (X)
) and assume that the [(X i , Y i )] n i=1 are i.i.d. In this setting, it is well-known that the Hoeffding assumption always holds. We state as a reminder the following lemma.

Lemma 5.5.1 Hoeffding assumption (5.1) is satisfied with f (λ, n) = λ 2 /(2n).

The proof is given in Appendix 5.A for the sake of completeness.

It is also possible to prove that Bernstein assumption (5.2) holds in the case where the so-called margin assumption of Mammen and Tsybakov is satisfied. This condition we use was introduced by [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF] in a classification setting, based on a related definition in [START_REF] Mammen | Smooth discrimination analysis[END_REF].

Lemma 5.5.2 Assume that Mammen and Tsybakov's margin assumption is satisfied: i.e. there is a constant C such that

E[(1 f θ (X) =Y -1 f θ (X) =Y ) 2 ] ≤ C[R(θ) -R].
Then Bernstein assumption (5.2) is satisfied with g(λ, n) = Cλ 2 2n-λ .

Remark 5.5.1 We refer the reader to [START_REF] Tsybakov | Optimal aggregation of classifiers in statistical learning[END_REF] for a proof that

P(0 < | θ, X | ≤ t) ≤ C ′ t
for some constant C ′ > 0 implies the margin assumption. In words, when X is not likely to be in the region θ, X ≃ 0, where points are hard to classify, then the problem becomes easier and the classification rate can be improved.

We propose in this context a Gaussian prior: π = N d (0, ϑ 2 I d ), and we consider a VB approach based on Gaussian families. The corresponding optimization problem is not convex, but remains feasible as we explain below.

Three sets of Variational Gaussian approximations

Consider the three following Gaussian families

F 1 = Φ m,σ 2 , m ∈ R d , σ 2 ∈ R * + , F 2 = Φ m,σ 2 , m ∈ R d , σ 2 ∈ (R * + ) 2 (mean field approximation), F 3 = Φ m,Σ , m ∈ R d , Σ ∈ S d+ (full covariance approximation), where Φ m,σ 2 is Gaussian distribution N d (m, σ 2 I d ), Φ m,σ 2 is N d (m, diag(σ 2 )), and Φ m,Σ is N d (m, Σ). Obviously, F 1 ⊂ F 2 ⊂ F 3 ⊂ M 1 + (Θ), and B λ (M 1 + (Θ)) ≤ B λ (F 3 ) ≤ B λ (F 2 ) ≤ B λ (F 1 ).
(5.6) Note that, for the sake of simplicity, we will use the following classical notations in the rest of the chapter: ϕ(•) is the density of N (0, 1) w.r.t. the Lebesgue measure, and Φ(•) the corresponding c.d.f. The rest of Section 5.5 is organized as follows. In Subsection 5.5.3, we calculate explicitly B λ (F 2 ) and B λ (F 1 ). Thanks to (5.6) this also gives an upper bound on B λ (F 3 ) and proves the validity of the three types of Gaussian approximations. Then, we give details on algorithms to compute the variational approximation based on F 2 and F 3 , and provide a numerical illustration on real data.

Theoretical analysis

We start with the empirical bound for F 2 (and F 1 as a consequence), which is a direct corollary of Theorem 5.4.1.

Corollary 5.5.3 For any ε > 0, with probability at least 1ε we have, for any

m ∈ R d , σ 2 ∈ (R + ) d , ˆRdΦ m,σ 2 ≤ ˆrn dΦ m,σ 2 + λ 2n + d i=1 1 2 log ϑ 2 σ 2 i + σ 2 i ϑ 2 + m 2 ϑ 2 -d 2 + log 1 ε λ .
We now want to apply Theorem 5.4.2 in this context. In order to do so, we introduce an additional assumption.

Definition 5.5.1 We say that Assumption A1 is satisfied when there is a constant c > 0 such that, for any (θ, θ ′ ) ∈ Θ 2 with θ = θ ′ = 1, P( X, θ X, θ ′ < 0) ≤ c θθ ′ .

Note that this is not a stringent assumption. For example, it is satisfied as soon as X/ X has a bounded density on the unit sphere.

Corollary 5.5.4 Assume that the VB approximation is done on either F 1 , F 2 or F 3 . Take λ = √ nd and ϑ = 1 √ d . Under Assumption A1, for any ε > 0, with probability at least 1ε we have simultaneously

´Rdρ λ ´Rdρ λ ≤ R + d n log 4ne 2 + c √ n + d 4n 3 + 2 log 2 ε √ nd .
See the appendix for a proof. Note also that the values λ = √ nd and ϑ = 1 √ d allow to derive this almost optimal rate of convergence, but are not necessarily the best choices in practice.

Remark 5.5.2 Note that Assumption A1 is not necessary to obtain oracle inequalities on the risk integrated under ρλ . We refer the reader to Chapter 1 in [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] for such assumption-free bounds. However, it is clear that without this assumption the shape of ρλ and ρλ might be very different. Thus, it seems reasonable to require that A1 is satisfied for the approximation of ρλ by ρλ to make sense.

We finally provide an application of Theorem 5.4.3. Under the additional constraint that the margin assumption is satisfied, we obtain a better rate.

Corollary 5.5.5 Assume that the VB approximation is done on either F 1 , F 2 or F 3 . Under Assumption A1 (Definition 5.5.1 page 133), and under Mammen and Tsybakov margin assumption, with λ = 2n C+2 and ϑ > 0, for any ε > 0, with probability at least 1ε,

´Rdρ λ ´Rdρ λ ≤ R+ (C + 2)(C + 1) 2 d log n ϑ n + 2dϑ n 2 + 2 ϑ - d ϑn + 2 n log 2 ε + √ d2c(2C + 1) n .
The prior variance optimizing the bound is ϑ = d/(d + 2 + 2d/n), this choice or any constant instead will lead to a rate in d log(n)/n. Note that the rate d/n is minimax-optimal in this context. This is, for example, a consequence of more general results in [START_REF] Lecué | Méthodes d'agrégation: optimalité et vitesses rapides[END_REF] under a general form of the the margin assumption. See the Appendix for a proof.

Implementation and numerical results

For family F 2 (mean field), the variational lower bound (5.5) equals

L λ,ϑ (m, σ) = - λ n n i=1 Φ -Y i X i m X i diag(σ 2 )X t i - m T m 2ϑ + 1 2 d k=1 log σ 2 k - σ 2 k ϑ ,
while for family F 3 (full covariance), it equals

L λ,ϑ (m, Σ) = - λ n n i=1 Φ -Y i X i m X i ΣX t i - m T m 2ϑ + 1 2 log |Σ| - 1 ϑ trΣ .
Both functions are non-convex, but the multimodality of the latter may be more severe due to the larger dimension of F 3 . To address this issue, we recommend to use the reparametrisation of [START_REF] Opper | The variational Gaussian approximation revisited[END_REF], which makes the dimension of the latter optimisation problem O(n); see [START_REF] Khan | Decoupled variational Gaussian inference[END_REF] for a related approach. In both cases, we found that deterministic annealing to be a good approach to optimise such non-convex functions. We refer to Appendix B for more details on deterministic annealing and on our particular implementation.

We now compare the numerical performance of the mean field and full covariance VB approximations to the Gibbs posterior (as approximated by SMC, see Section 5.3.1) for the classification of standard datasets; see Table 5.1. We also include results for a kernel SVM (support vector machine); this comparison is not entirely fair, since SVM is a non-linear classifier, while all the other classifiers are linear. Still, except for the Glass dataset, the full covariance VB approximation performs as well or better than both SMC and SVM (while being much faster to compute, especially compared to SMC).

Interestingly, VB outperforms SMC in certain cases. This might be due to the fact that a VB approximation tends to be more concentrated around the mode than the Gibbs posterior it approximates. Mean field VB does not perform so well on certain datasets (e.g. Indian). This may be due either to the approximation family being too small, or to the corresponding optmisation problem being strongly multi-modal.

Application to classification under convexified loss

Compared to the previous section, the advantage of convex classification is that the corresponding variational approximation will amount to minimising a convex Misclassification rates for different datasets and for the proposed approximations of the Gibbs posterior. The last column is the missclassification rate given by a kernel-SVM with radial kernel. The hyper-parameters are chosen by cross-validation.

function. This means that (a) the minimisation problem will be easier to deal with; and (b) we will be able to compute a bound for the integrated risk after a given number of steps of the minimisation procedure.

The setting is the same as in the previous section, except that for convenience we now take Y = {-1, 1}, and the risk is based on the hinge loss,

r H n (θ) = 1 n n i=1 max(0, 1 -Y i < θ, X i >).
We will write R H for the theoretical counterpart and RH for its minimum in θ. We keep the superscript H in order to allow comparison with the risk R under the 0 -1 loss. We assume in this section that the X i are uniformly bounded by a constant, |X i | < c x . Note that we do not require an assumption of the form (A1) to obtain the results of this section, as we rely directly on the Lipschitz continuity of the hinge risk.

Theoretical Results

Contrarily to the previous section, the risk is not bounded in θ, and we must specify a prior distribution for the Hoeffding assumption to hold.

Lemma 5.6.1 Under an independent Gaussian prior π such that each component is N (0, ϑ 2 ), and for λ < 1

(λ, n) = λ 2 /(4n) -1 2 log 1 -ϑ 2 λ 2 c 2 x 2n . 135
The main impact of such a bound is that the prior variance cannot be taken too big relative to λ.

Corollary 5.6.2 Assume that the VB approximation is done on either F 1 , F 2 or F 3 . Take λ = 1 cx n ϑ 2 and ϑ = 1 √ d . For any ε > 0, with probability at least 1ε we have simultaneously

´RH dρ λ ´RH dρ λ ≤ R H + c x 2 d n log n d + 2c x d n + 1 √ nd c 2 x + 1 2c x + 2c x log 2 ǫ
The oracle inequality in the above corollary enjoys the same rate of convergence as the equivalent result in the preceding section. In the following we link the two results.

Remark 5.6.1 As stated in the beginning of the section we can use the estimator specified under the hinge loss to bound the excess risk of the 0-1 loss. We write R ⋆ and R H⋆ the respective risk for their corresponding Bayes classifiers. From [START_REF] Zhang | Statistical behavior and consistency of classification methods based on convex risk minimization[END_REF] (section 3.3) we have the following inequality, linking the excess risk under the hinge loss and the 0 -1 loss,

R(θ) -R ⋆ ≤ R H (θ) -R H⋆
for every θ ∈ R d . By integrating with respect to ρH (the VB approximation on any F 1 , F 2 , F 3 of the Gibbs posterior for the hinge risk) and making use of Corollary 5.6.2 we have with high probability,

ρH (R(θ)) -R ⋆ ≤ inf θ∈R p R H (θ) -R H⋆ + O d n log n d .

Numerical application

We have motivated the introduction of the hinge loss as a convex upper bound.

In the sequel we show that the resulting VB approximation also leads to a convex optimization problem. This has the advantage of opening a range of possible optimization algorithms [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF]. In addition we are able to bound the error of the approximated measure after a fixed number of iterations (see Theorem 5.6.3). Under the model F 1 each individual risk is given by:

ρ m,σ (r i (θ)) = (1 -Γ i m) Φ 1 -Γ i m σ Γ i 2 + σ Γ i ϕ 1 -Γ i m σ Γ i 2 := Ξ i m σ , writting Γ i := Y i X i .
Hence the lower bound to be maximized is given by

L(m, σ) = - λ n n i=1 (1 -Γ i m) Φ 1 -Γ i m σ Γ i 2 + n i=1 σ Γ i ϕ 1 -Γ i m σ Γ i 2 - m 2 2 2ϑ + d 2 log σ 2 - ϑ σ 2 .
It is easy to see that the function is convex in (m, σ), first note that the map Misclassification rates for different datasets and for the proposed approximations of the Gibbs posterior. The hyperparameters are chosen by cross-validation. This is to be compared to Table 5.1.

Theorem 5.6.3 Assume that the VB approximation is done on F 1 , F 2 or F 3 . Denote by ρk (dθ) the VB approximated measure after the kth iteration of an optimal convex solver using the hinge loss. Take λ = 1 cx n d and ϑ = 1 √ d then under the hypothesis of Corollary 5.6.2 with probability 1ǫ

ˆRH dρ k ≤ R H + LM √ 1 + k + c x 2 d n log n d + 2c x d n + 1 √ nd c 2 x + 1 2c x + 2c x log 2 ǫ
where L is the Lipschitz coefficient on a ball of radius M of the objective function maximized in VB.

From Theorem 5.6.3 we can compute the number of iterations to get a given level of error at a given probability.

We find that on average the misclassification error (Table 5.2) is lower than for the 0-1 loss where we have no guaranties that the maximum is attained.

Application to ranking

Preliminaries

In this section we take Y = {0, 1} and consider again linear classifiers: Θ = X = R d , f θ (x) = 1 θ,x ≥0 . We consider however a different criterion: in ranking, not only we want to classify well an object x, but we want to make sure that given two different objects, the one that is more likely to correspond to a label 1 will be assigned a larger score through the function f θ . A usual way to measure this is to introduce the risk function

R(θ) = P[(Y 1 -Y 2 )(f θ (X 1 ) -f θ (X 2 )) < 0]
and the empirical risk

r n (θ) = 1 n(n -1) 1≤i =j≤n 1 {(Y i -Y j )(f θ (X i )-f θ (X j ))<0} .
Then, again, we recall classical results.

Lemma 5.7.1 The Hoeffding-type assumption is satisfied with f (λ, n) = λ 2 n-1 . The variant of the margin assumption adapted to ranking was established by [START_REF] Robbiano | Upper bounds and aggregation in bipartite ranking[END_REF] and [START_REF] Ridgway | Computation of Gaussian orthant probabilities in high dimension[END_REF].

Lemma 5.7.2 Assume the following margin assumption:

E[(1 [f θ (X 1 )-f θ (X 2 )][Y 1 -Y 2 ]<0 -1 [f θ (X 1 )-f θ (X 2 )][Y 1 -Y 2 ]<0 ) 2 ] ≤ C[R(θ) -R].
Then Bernstein assumption (5.2) is satisfied with g(λ, n) = Cλ 2 n-1-4λ .

138

Application to ranking

We still consider a Gaussian prior

π(dθ) = d i=1 ϕ(θ i ; 0, ϑ 2 )dθ i
and the approximation families will be the same as in Section 5.5:

F 1 = {Φ m,σ 2 , m ∈ R d , σ 2 ∈ R * + }, F 2 = {Φ m,σ 2 , m ∈ R d , σ 2 ∈ (R * + ) d } and F 3 = {Φ m,Σ , m ∈ R d , Σ ∈ S d+ }.

Theoretical study

Here again, we start with the empirical bound.

Corollary 5.7.3 For any ε > 0, with probability at least 1ε we have, for any

m ∈ R d , σ 2 ∈ (R + ) d , ˆRdΦ m,σ 2 ≤ ˆrn dΦ m,σ 2 + λ n -1 + d j=1 1 2 log ϑ 2 σ 2 i + σ 2 i ϑ 2 + m 2 ϑ 2 -d 2 + log 1 ε λ .
In order to derive a theoretical bound, we introduce the following variant of Assumption A1.

Definition 5.7.1 We say that Assumption A2 is satisfied when there is a constant c > 0 such that, for any (θ, θ

′ ) ∈ Θ 2 with θ = θ ′ = 1, P( X 1 -X 2 , θ X 1 -X 2 , θ ′ < 0) ≤ c θ -θ ′ .
Assumption A2 is satisfied as soon as (X 1 -X 2 )/ X 1 -X 2 has a bounded density on the unit sphere.

Corollary 5.7.4 Use either F 1 , F 2 or F 3 . Take λ = d(n-1) 2 and ϑ = 1. Under (A2), for any ε > 0, with probability at least 1ε,

´Rdρ λ ´Rdρ λ ≤ R + 2d n -1 1 + 1 2 log (2d(n -1)) + c √ 2 √ n -1 + 2 √ 2 log 2e ε (n -1)d .
Finally, under an additional margin assumption, we have:

Corollary 5.7.5 Under Assumption A2 and the margin assumption of Lemma (5.7.2), for λ = n-1 C+5 and ϑ > 0, for any ε > 0, with probability at least 1ε,

´Rdρ λ ´Rdρ λ ≤ R+ (C + 5)(C + 1) 2 d log n ϑ n -1 + 2dϑ n(n -1) + 2 ϑ - d ϑn -1 + 2 n -1 log 2 ε + √ d4c(C + 1) n .
The prior variance optimizing the bound is ϑ = d/(d + 2 + 2d/n). The proof is similar to the ones of Corollaries 5.5.4,5.5.5 and 5.7.4. As in the case of classification, ranking under an AUC loss can be done by replacing the indicator function by the corresponding upper bound given by an hinge loss. In this case we can derive similar results as for the convexified classification in particular we can get a convex minimization problem and obtain result without requiring assumption (A2).

Algorithms and numerical results

As an illustration we focus here on family F 2 (mean field). In this case the VB objective to maximize is given by:

L(m, σ 2 ) = - λ n + n -i:y i =1,j:y j =0 Φ   - Γ ij m d k=1 (γ k ij ) 2 σ 2 k   - m 2 2 2ϑ + 1 2 d k=1 log σ 2 k - σ 2 k ϑ ,
(5.7) where Γ ij = X i -X j , and where (γ k ij ) k are the elements of Γ. This function is expensive to compute, as it involves n + n -terms, the computation of which is O(p).

We propose to use a stochastic gradient descent in the spirit of Hoffman et al. [2013]. The model we consider is not in an exponential family, meaning we cannot use the trick developed by these authors. We propose instead to use a standard descent.

The idea is to replace the gradient by an unbiased version based on a batch of size B as described in Algorithm 22 in the Appendix. [START_REF] Robbins | A stochastic approximation method[END_REF] show that for a step-size (λ t ) t such that t λ 2 t < ∞ and t λ t = ∞ the algorithm converges to a local optimum.

In our case we propose to sample pairs of data with replacement and use the unbiased version of the derivative of the risk component. We use a simple gradient descent without any curvature information. One could also use recent research on stochastic quasi Newton-Raphson [START_REF] Byrd | A stochastic quasi-Newton method for large-scale optimization[END_REF].

For illustration, we consider a small dataset (Pima), and a larger one (Adult). The latter is already quite challenging with n + n -= 193, 829, 520 pairs to compare. In both cases with different size of batches convergence is obtained with a few iterations only and leads to acceptable bounds.

In Figure 5.1 we show the empirical bound on the AUC risk as a function of the iteration of the algorithm, for several batch sizes. The bound is taken for 95% probability, the batch sizes are taken to be B = 1, 10, 20, 50 for the Pima dataset, and 50 for the Adult dataset. The figure shows an additional feature of VB approximation in the context of Gibbs posterior: namely the possibility of computing the empirical upper bound given by Corollary 5.7.3. That is we can check the quality of the bound at each iteration of the algorithm, or for different values of the hyperparameters.

Application to matrix completion

The matrix completion problem has received increasing attention recently, partly due to spectacular theoretical results [START_REF] Candès | The power of convex relaxation: near-optimal matrix completion[END_REF], and to challenging applications like the Netflix challenge [START_REF] Bennett | The netflix prize[END_REF]. In the perspective of this chapter, the specific interest of this application is twofold. First, this is a case where the family of approximations is not parametric, but rather of the form (5.3), i.e. the family of products of independent components. Then, there is no known theoretical result for the Gibbs estimator in the considered model, yet we can still directly bound the loss induced by the variational approximation.

See the Appendix for a proof. For instance, in Theorem 5.4.3, in classification and ranking we had λ, λ-g(λ, n) and λ + g(λ, n) of order O(n). In this case we would have:

2 λ -g(λ, n) inf ρ∈F K ρ, π λ+g(λ,n) 2 = O C(a, C)r(m 1 + m 2 ) log [nb(m 1 + m 2 )K] n ,
and note that in this context it is known that the minimax rate is at least r(m 1 + m 2 )/n [START_REF] Koltchinskii | Nuclear-norm penalization and optimal rates for noisy low-rank matrix completion[END_REF].

Algorithm

As already mentioned, the approximation family is not parametric in this case, but rather of type mean field. The corresponding VB algorithm amounts to iterating equation (5.4), which takes the following form in this particular case:

u j (dU j,. ) ∝ exp - λ n i E V,U -j (Y X i -(U V T ) X i ) 2 - K k=1 E γ j 1 2γ k U 2 jk v j (dV j,. ) ∝ exp - λ n i E V -j ,U (Y X i -(U V T ) X i ) 2 - K k=1 E γ j 1 2γ k V 2 jk p(γ k ) ∝ exp - 1 2γ k j E U U 2 kj + i E V V 2 ik + (α + 1) log 1 γ k - β γ k
where the expectations are taken with respect to the thus defined variational approximations. One recognises Gaussian distributions for the first two, and an inverse Gamma distribution for the third. We refer to [START_REF] Lim | Variational Bayesian approach to movie rating prediction[END_REF] for more details on this algorithm and for a numerical illustration.

Discussion

We showed in several important scenarios that approximating a Gibbs posterior through VB (Variational Bayes) techniques does not deteriorate the rate of convergence of the corresponding procedure. We also described practical algorithms for fast computation of these VB approximations, and provided empirical bounds that may be computed from the data to evaluate the performance of the so-obtained VB-approximated procedure. We believe these results provide a strong incentive to recommend VB as the default approach to approximate Gibbs posteriors, in lieu of Monte Carlo methods. We hope to extend our results to other applications beyond those discussed in this chapter, such as regression. One technical difficulty with regression is that ρλ = arg min

ρ∈M 1 + (Θ)
λ ˆrn dρ + K(ρ, π) , (5.9)

log ˆexp(-λr n )dπ = min

ρ∈M 1 + (Θ)
λ ˆrn dρ + K(ρ, π) .

(5.10)

We will use (5.8), (5.9) and (5.10) several times in this appendix.

5.

A.2 Proof of the theorems in Subsection 5.4.1

Proof of Theorem 5.4.1. This proof follows the standard PAC-Bayesian approach (see [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF]). Apply Fubini's theorem to the first inequality of (5.1):

E ˆexp {λ[R(θ) -r n (θ)] -f (λ, n)} π(dθ) ≤ 1
then apply the preliminary remark with h(θ) = λ[r n (θ) -R(θ)]:

E exp sup ρ∈M 1 + (Θ) ˆλ[R(θ) -r n (θ)]ρ(dθ) -K(ρ, π) -f (λ, n) ≤ 1.
Multiply both sides by ε and use E[exp(U )] ≥ P(U > 0) for any U to obtain:

P sup ρ∈M 1 + (Θ) ˆλ[R(θ) -r n (θ)]ρ(dθ) -K(ρ, π) -f (λ, n) + log(ε) > 0 ≤ ε.
Then consider the complementary event:

P ∀ρ ∈ M 1 + (Θ), λ ˆRdρ ≤ λ ˆrn dρ + f (λ, n) + K(ρ, π) + log 1 ε ≥ 1 -ε.
Proof of Theorem 5.4.2. Using the same calculations as above, we have, with probability at least 1ε, simultaneously for all ρ ∈ M 1 + (Θ),

λ ˆRdρ ≤ λ ˆrn dρ + f (λ, n) + K(ρ, π) + log 2 ε (5.11) λ ˆrn dρ ≤ λ ˆRdρ + f (λ, n) + K(ρ, π) + log 2 ε .
(5.12)

We use (5.11) with ρ = ρλ and (5.9) to get

λ ˆRdρ λ ≤ inf ρ∈M 1 + (Θ) λ ˆrn dρ + f (λ, n) + K(ρ, π) + log 2 ε 146 5.A Proofs
and plugging (5.12) into the right-hand side, we obtain

λ ˆRdρ λ ≤ inf ρ∈M 1 + (Θ) λ ˆRdρ + 2f (λ, n) + 2K(ρ, π) + 2 log 2 ε .
Now, we work with ρλ = arg min ρ∈F K(ρ, ρλ ). Plugging (5.8) into (5.11) we get, for any ρ,

λ ˆRdρ ≤ f (λ, n) + K(ρ, ρλ ) -log ˆexp(-λr n )dπ + log 2 ε .
By definition of ρλ , we have:

λ ˆRdρ λ ≤ inf ρ∈F f (λ, n) + K(ρ, ρλ ) -log ˆexp(-λr n )dπ + log 2 ε
and, using (5.8) again, we obtain:

λ ˆRdρ λ ≤ inf ρ∈F λ ˆrn dρ + f (λ, n) + K(ρ, π) + log 2 ε .
We plug (5.12) into the right-hand side to obtain:

λ ˆRdρ λ ≤ inf ρ∈F λ ˆRdρ + 2f (λ, n) + 2K(ρ, π) + 2 log 2 ε .
This proves the second inequality of the theorem. In order to prove the claim

B λ (F) = B λ (M 1 + (Θ)) + 2 λ inf ρ∈F K(ρ, π λ 2 ), note that B λ (F) = inf ρ∈F ˆRdρ + 2f (λ, n) λ + 2K(ρ, π) λ + 2 log 2 ε λ = inf ρ∈F - 2 λ log ˆexp - λ 2 R dπ + 2f (λ, n) λ + 2K(ρ, π λ 2 ) λ + 2 log 2 ε λ = - 2 λ log ˆexp - λ 2 R dπ + 2f (λ, n) λ + 2 log 2 ε λ + 2 λ inf ρ∈F K(ρ, π λ 2 ) = B λ (M 1 + (Θ)) + 2 λ inf ρ∈F K(ρ, π λ 2 ).
This ends the proof.

5.

A.3 Proof of Theorem 5.4.3 (Subsection 5.4.2)

Proof of Theorem 5.4.3. As in the proof of Theorem 5.4.1, we apply Fubini, then (5.10) to the first inequality of (5.2) to obtain

E exp sup ρ ˆ λ[R(θ) -R] -λ[r n (θ) -r n ] -g(λ, n)[R(θ) -R] ρ(dθ) -K(ρ, π) ≤ 1
and we multiply both sides by ε/2 to get

P sup ρ [λ-g(λ, n)] ˆRdρ -R ≥ λ ˆrn dρ -r n +K(ρ, π)+log 2 ε ≤ ε 2 .
(5.13) We now consider the second inequality in (5.2):

E exp λ[r n (θ) -r n ] -λ[R(θ) -R] -g(λ, n)[R(θ) -R] ≤ 1.
The same derivation leads to

P sup ρ [λ-g(λ, n)] ˆrn dρ -r n ≥ λ ˆRdρ -R +K(ρ, π)+log 2 ε ≤ ε 2 .
(5.14) We combine (5.13) and (5.14) by a union bound argument, and we consider the complementary event: with probability at least 1ε, simultaneously for all ρ ∈

M 1 + (Θ), [λ -g(λ, n)] ˆRdρ -R ≤ λ ˆrn dρ -r n + K(ρ, π) + log 2 ε , (5.15) λ ˆrn dρ -r n ≤ [λ + g(λ, n)] ˆRdρ -R + K(ρ, π) + log 2 ε .
(5.16)

We now derive consequences of these two inequalities (in other words, we focus on the event where these two inequalities are satisfied). Using (5.9) in (5.15) yields

[λ -g(λ, n)] ˆRdρ λ -R ≤ inf ρ∈M 1 + (Θ) λ ˆrn dρ -r n + K(ρ, π) + log 2 ε .
We plug (5.16) into the right-hand side to obtain:

[λ -g(λ, n)] ˆRdρ λ -R ≤ inf ρ∈M 1 + (Θ) [λ + g(λ, n)] ˆRdρ -R + 2K(ρ, π) + 2 log 2 ε .

5.A Proofs

Now, we work with ρλ . Plugging (5.8) into (5.13) we get

[λ -g(λ, n)] ˆRdρ -R ≤ K(ρ, ρλ ) -log ˆexp[-λ(r n -r n )]dπ + log 2 ε .
By definition of ρλ , we have:

[λ -g(λ, n)] ˆRdρ λ -R ≤ inf ρ∈F K(ρ, ρλ ) -log ˆexp[-λ(r n -r n )]dπ + log 2 ε .
Then, apply (5.8) again to get:

[λ -g(λ, n)] ˆRdρ λ -R ≤ inf ρ∈F λ ˆ(r n -r n )dρ + K(ρ, π) + log 2 ε .
Plug (5.16) into the right-hand side to get

[λ -g(λ, n)] ˆRdρ λ -R ≤ inf ρ∈F [λ + g(λ, n)] ˆ(R -R)dρ + 2K(ρ, π) + 2 log 2 ε .

5.A.4 Proofs of Section 5.5

Proof of Lemma 5.5.1. Combine Theorem 2.1 p. 25 and Lemma 2.2 p. 27 in [START_REF] Boucheron | Concentration Inequalities[END_REF].

Proof of Lemma 5.5.2. Apply Theorem 2.10 in [START_REF] Boucheron | Concentration Inequalities[END_REF], and plug the margin assumption.

Proof of Corollary 5.5.4. We remind that thanks to (5.6) it is enough to prove the claim for F 1 . We apply Theorem 5.4.2 to get:

B λ (F 1 ) = inf (m,σ 2 ) ˆRdΦ m,σ 2 + λ n + 2 K(Φ m,σ 2 , π) + log 2 ε λ = inf (m,σ 2 )    ˆRdΦ m,σ 2 + λ n + 2 d 1 2 log ϑ 2 σ 2 + σ 2 ϑ 2 + m 2 ϑ 2 -d 2 + log 2 ε λ    .
Note that the minimizer of R, θ, is not unique (because f θ (x) does not depend on θ ) and we can chose it in such a way that θ = 1. Then

R(θ) -R = E 1 θ,X Y <0 -1 θ,X Y <0 ≤ E 1 θ,X θ,X <0 = P θ, X θ, X < 0 ≤ c θ θ -θ ≤ 2c θ -θ .
So:

B λ (F 1 ) ≤ R + inf (m,σ 2 ) 2c ˆ θ -θ Φ m,σ 2 (dθ) + λ n + 2 d 1 2 log ϑ 2 σ 2 + σ 2 ϑ 2 + m 2 ϑ 2 -d 2 + log 2 ε λ .
We now restrict the infimum to distributions ν such that m = θ:

B(F 1 ) ≤ R + inf σ 2    2c √ dσ + λ n + d log ϑ 2 σ 2 + 2dσ 2 ϑ 2 + 2 ϑ 2 -d + 2 log 2 ε λ    . We put σ = 1 2λ and substitute 1 √ d for ϑ to get B(F 1 ) ≤ R + λ n + c √ d + d log(4 λ 2 d ) + d 2 2λ 2 + d + 2 log 2 ε λ .
Substitute √ nd for λ to get the desired result. Proof of Corollary 5.5.5. We apply Theorem 5.4.3:

ˆ(R -R)dρ λ ≤ inf m,σ 2 λ + g(λ, n) λ -g(λ, n) ˆ(R -R)dΦ m,σ 2 + 1 λ -g(λ, n) 2K(Φ m,σ 2 , π) + 2 log 2 ǫ
where λ < 2n C+1 . Computations similar to those in the the proof of Corollary 5.5.4 lead to

ˆRdρ λ ≤ R + inf m,σ 2 2c λ + g(λ, n) λ -g(λ, n) ˆ θ -θ Φ m,σ 2 (dθ) + 2 d j=1 1 2 log ϑ 2 σ 2 + σ 2 ϑ 2 + m 2 ϑ 2 -d 2 + log 2 ε λ -g(λ, n) .
taking m = θ and λ = 2n C+2 , we get the result.

5.A.5 Proofs of Section 5.6

Proof of Lemma 5.6.1. For fixed θ we can upper bound the individual risk such that:

0 ≤ max(0, 1-< θ, X i > Y i ) ≤ 1 + | < θ, X i > |
such that we can apply Hoeffding's inequality conditionally on X i and fixed θ.

We get,

E exp λ(R H -r H n ) |X 1 , • • • , X n ≤ exp λ 2 8n 2 n i=1 (1 + | < θ, X i > |) 2 ≤ exp λ 2 4n + λ 2 c 2 x 4n θ 2
where the last inequality stems from the fact that (a + b) 2 ≤ 2 (a 2 + b 2 ) and the fact that we have supposed the X i to be bounded. We can take the expectation of this term with respect to the X i 's and with respect to our Gaussian prior.

π E exp λ(R H -r H n ) ≤ exp λ 2 4n (2π) d 2 √ ϑ 2 ˆexp λ 2 c 2 x 4n θ 2 - 1 2ϑ 2 θ 2 dθ ≤ exp λ 2 4n (2π) d 2 √ ϑ 2 ˆexp - 1 2 1 ϑ 2 - λ 2 c 2 x 2n θ 2 dθ
The integral is a properly defined Gaussian integral under the hypothesis that

1 ϑ 2 -λ 2 c 2 x 2n > 0 hence λ < 1 cx n ϑ
2 . The integral is proportional to a Gaussian and we can directly write:

π E exp λ(R H -r H n ) ≤ exp λ 2 4n 1 -ϑ 2 λ 2 c 2 x 2n
writing everything in the exponential gives the desired result. Proof of Corollary 5.6.2. We apply Theorem 5.4.2 to get:

B λ (F 1 ) = inf (m,σ 2 ) ˆRH dΦ m,σ 2 + λ 2n - 1 λ log 1 - ϑ 2 λ 2 c 2 x 2n + 2 K(Φ m,σ 2 , π) + log 2 ε λ = inf (m,σ 2 )    ˆRH dΦ m,σ 2 + 2 d j=1 1 2 log ϑ 2 σ 2 + σ 2 ϑ 2 + m 2 ϑ 2 -d 2 + log 2 ε λ    + 151 λ 2n - 1 λ log 1 - ϑλ 2 c 2 x 2n .
We use the fact that the hinge loss is Lipschitz and that the (X i ) are uniformly bounded X ∞ < c x . We get R H (θ) ≤ RH + c x √ d θ -θ and restrict the infimum to distributions ν such that m = θ:

B(F 1 ) ≤ R H + inf σ 2    c x dσ 2 + λ 2n - 1 λ log 1 - ϑ 2 λ 2 c 2 x 2n + d log ϑ 2 σ 2 + 2dσ 2 ϑ 2 + 2 ϑ 2 -d + 2 log 2 ε λ    .
We specify σ 2 = 1 √ dn and λ = 1 cx n ϑ 2 such that we get:

B(F 1 ) ≤ R H + c x d n + √ ϑ 2 2c x √ n -c x ϑ 2 n log 1 - 1 2 +d c x ϑ √ n log ϑ 2 √ nd +c x ϑ 2d nϑ 2 + 2 ϑ 2 -d + 2 log 2 ε √ n .
To get the correct rate we take the prior variance to be ϑ 2 = 1 d by replacing in the above equation we get the desired result.

Proof of Theorem 5.6.3. From [START_REF] Nesterov | Introductory lectures on convex optimization[END_REF] (th. 3.2.2) we have the following bound on the objective function minimized by VB, (the objective is not uniformly Lipschitz)

ρ k (r H n ) + 1 λ K(ρ k , π) -inf ρ∈F 1 ρ(r H n ) + 1 λ K(ρ, π) ≤ LM √ 1 + k .
(5.17)

where the initial point was taken in a ball of radius M around ρ.

We have from equation (5.11) specified for measures ρ k probability 1ε,

λ ˆRH dρ k ≤ λ ˆrH n dρ k + f (λ, n) + K(ρ k , π) + log 1 ε
Combining the two equations yields,

ˆRH dρ k ≤ LM √ 1 + k + 1 λ f (n, λ) + inf ρ∈F 1 ρ(r H n ) + 1 λ K(ρ, π) + 1 λ log 1 ε
We can therefore write for any ρ ∈ F 1 ,

ˆRH dρ k ≤ LM √ 1 + k + 1 λ f (n, λ) + ρ(r H n ) + 1 λ K(ρ, π) + 1 λ log 1 ε

5.A Proofs

Using equation (5.11) a second time we get with probability 1ε

ˆRH dρ k ≤ LM √ 1 + k + 2 λ f (n, λ) + ρ(R H ) + 2 λ K(ρ, π) + 2 λ log 2 ε
Because this is true for any ρ ∈ F 1 in 1ε we can write the bound for the smallest measure in F 1 .

ˆRH dρ k ≤ LM √ 1 + k + 2 λ f (n, λ) + inf ρ∈F 1 ρ(R H ) + 2 λ K(ρ, π) + 2 λ log 2 ε
By taking the Gaussian measure with variance 1 dn and mean θ in the infimum and taking λ = 1 cx √ nd and ϑ = 1 d , we can plug in the result of Corrolary 5.6.2 to get the result.

5.A.6 Proofs of Section 5.7

Proof of Lemma 5.7.1. The idea of the proof is to use Hoeffding's decomposition of U-statistics combined with Hoeffding's inequality for iid random variables. This was done in ranking by Clémençon et al. [2008a], and later in [START_REF] Ridgway | Computation of Gaussian orthant probabilities in high dimension[END_REF]; [START_REF] Robbiano | Upper bounds and aggregation in bipartite ranking[END_REF] for ranking via aggregation and Bayesian statistics. The proof is as follows: we define

q θ i,j = 1 (Y i -Y j )(f θ (X i )-f θ (X j ))<0 -R(θ) so that U n := 1 n(n -1) i,j q θ i,j = r n (θ) -R(Θ).
From [START_REF] Hoeffding | Probability Inequalities for Sums of Random Variables[END_REF] we have

U n = 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)
where the sum is taken over all the permutations π of {1, . . . , n}. Jensen's inequality leads to

E exp[λU n ] = E exp   λ 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ 1 n! π E exp   λ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   .
We now use, for each of the terms in the sum the same argument as in the proof of Lemma 5.5.1 to get

E exp[λU n ] ≤ 1 n! π exp λ 2 2⌊ n 2 ⌋ ≤ exp λ 2 n -1
(in the last step, we used ⌊ n 2 ⌋ ≥ (n -1)/2). We proceed in the same way to upper bound E exp[-λU n ]. Proof of Lemma 5.7.2. As already done above, we use Bernstein inequality and Hoeffding decomposition. Fix θ. We define this time

q θ i,j = 1{ θ, X i -X j (Y i -Y j ) < 0} -1{[σ(X i ) -σ(X j )](Y i -Y j ) < 0} -R(θ) + R so that U n := r n (θ) -r n -R(θ) + R = 1 n(n -1) i =j q θ i,j .
Then,

U n = 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋) .
Jensen's inequality:

E exp[λU n ] = E exp   λ 1 n! π 1 ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ 1 n! π E exp   λ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   .
Then, for each of the terms in the sum, use Bernstein's inequality:

E exp   λ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ exp   E((q θ π(1),π(1+⌊ n 2 ⌋) ) 2 ) λ 2 ⌊ n 2 ⌋ 2 1 -2 λ ⌊ n 2 ⌋   .
We use again ⌊ n 2 ⌋ ≥ (n -1)/2. Then, as the pairs (X i , Y i ) are iid, we have E((q θ π(1),π(1+⌊ n 2 ⌋) ) 2 ) = E((q θ 1,2 ) 2 ) and then E((q θ 1,2 ) 2 ) ≤ C[R(θ) -R] thanks to the margin assumption. So

E exp   λ ⌊ n 2 ⌋ ⌊ n 2 ⌋ i=1 q θ π(i),π(i+⌊ n 2 ⌋)   ≤ exp C[R(θ) -R] λ 2 n-1 1 -4λ n-1 .
This ends the proof of the proposition. Proof of Corollary 5.7.4. The calculations are similar to the ones in the proof of Corollary 5.5.4 so we don't give the details. Note that when we reach

B λ (F 1 ) ≤ R + 2λ n -1 + c √ d + d log(2λ) + 2 log 2e ε λ ,
an approximate minimization with respect to λ leads to the choice λ = d(n-1)

2

.

5.A.7 Proofs of Section 5.8

Proof. First, note that, for any ρ,

K(ρ, π β ) = β ˆ(R -R)dρ + K(ρ, π) + log ˆexp -β(R -R) dπ ≤ β ˆ(R -R)dρ + K(ρ, π).
Now, we define a subset of F that will be used for the calculation of the bound. We define for δ > 0 the probability distribution ρ U,V,δ (dθ) as π conditioned to θ = µν T with µ is uniform on {∀(i, ℓ), |µ i,ℓ -U i,ℓ | ≤ δ} and ν is uniform on {∀(j, ℓ), |ν i,ℓ -V j,ℓ | ≤ δ}. Note that ˆ(R -R)dρ M,N,δ = ˆE((θ X -M X ) 2 )ρ U,V,δ (dθ)

≤ ˆ3E(((U V T ) X -M X ) 2 )ρ U,V,δ (d(µ, ν))

+ 3 ˆE(((U ν T ) X -(U V T ) X ) 2 )ρ U,V,δ (d(µ, ν))

+ 3 ˆE(((µν T ) X -(U ν T ) X ) 2 )ρ U,V,δ (d(µ, ν)).
By definition, the first term is = 0. Moreover:

ˆE(((U ν T ) X -(U V T ) X ) 2 )ρ U,V,δ (d(µ, ν)) = ˆ1 m 1 m 2 i,j k U i,k (ν j,k -V j,k ) 2 ρ U,V,δ (d(µ, ν)) ≤ ˆ1 m 1 m 2 i,j k U 2 i,k k (ν j,k -V j,k ) 2 ρ U,V,δ (d(µ, ν)) 155 ≤ KrC 2 δ 2 .
In the same way, ˆE(((µν T ) X -(U ν T ) X ) 2 )ρ U,V,δ (d(µ, ν)) ≤ ˆ µ -U 2 F ν 2 F ρ U,V,δ (d(µ, ν))

≤ Kr(C + δ) 2 δ 2 .

So: ˆ(R -R)dρ M,N,δ ≤ 2Krδ 2 (C + δ 2 ). Now, let us consider the term K(ρ U,V,δ , π). An explicit calculation is possible but tedious. Instead, we might just introduce the set G δ = {θ = µν T , µ -U F ≤ δ, ν -V F ≤ δ} and note that K(ρ U,V,δ , π) ≤ log 1 π(G δ ) . An upper bound for G δ is calculated page 317-320 in [START_REF] Alquier | Bayesian methods for low-rank matrix estimation: short survey and theoretical study[END_REF] and the result is given by (10) in this reference: 2m 1 K log(2m 1 K) , δ 2 2m 2 K log(2m 2 K) is satisfied. So we obtain: Note that U 2 F ≤ C 2 rm 1 , V 2 F ≤ C 2 rm 2 and K ≤ m 1 +m 2 so it is clear that the choice δ = 1 β and b ≤ 1 2β(m 1 ∨m 2 ) log(2K(m 1 ∨m 2 )) leads to the existence of a constant C(a, C) such that K(ρ U,V,δ , π β ) ≤ C(a, C) r(m 1 + m 2 ) log [βb(m 1 + m 2 )K] + 1 β .

K(ρ U,V,δ , π) ≤ 4δ 2 + 2 U 2 F + 2 N 2 F +
K(ρ U,V,δ , π β ) ≤ β2Krδ 2 (C + δ 2 ) + 4δ 2 + 2 U 2 F + 2 N 2 F +

5.B Implementation details

5.B.1 Sequential Monte Carlo

Tempering SMC approximates iteratively a sequence of distribution ρ λt , with ρ λt (dθ) = 1 Z t exp (-λ t r n (θ)) π(dθ), and temperature ladder λ 0 = 0 < . . . < λ T = λ. The pseudo code below is given for an adaptive sequence of temperatures.

Algorithm 19 Tempering SMC

Input N (number of particles), τ ∈ (0, 1) (ESS threshold), κ > 0 (random walk tuning parameter)

Init. Sample θ i 0 ∼ π ξ (θ) for i = 1 to N , set t ← 1, λ 0 = 0, Z 0 = 1.

Loop a. Solve in λ t the equation The algorithm outputs a weighted sample (w i T , θ i T ) approximately distributed as target posterior, and an unbiased estimator of the normalizing constant Z λ T .

{ N i=1 w t (θ i t-
Step b. of algorithm 5.B.1 depends of a resampling algorithm. We choose to use Systematic resampling, described in Algorithm 20.

Algorithm 20 Systematic resampling

Input: Normalised weights W j t := w t (θ j t-1 )/ N i=1 w t (θ i t-1 ).

Output: indices A i ∈ {1, . . . , N }, for i = 1, . . . , N . A n ← m, and s ← s + 1.

End For

For the MCMC step, we used a Gaussian random-walk Metropolis kernel, with a covariance matrix for the random step that is proportional to the empirical covariance matrix of the current set of simulations.

5.B.2 Optimizing the bound

A natural idea to find a global optimum of the objective is to try to solve a sequence of local optimization problems with increasing inverse temperatures. For inverse temperature λ = 0 the problem can be solved exactly (as a KL divergence between two Gaussians). Then, for two consecutive temperatures, the corresponding solutions should be close enough. This idea has been coined under several names. It has a long history in variational Bayes litterature under the name deterministic annealing. Yuille uses it on mean field on Gibbs distribution for Markov random fields. In addition the intermediate results can be of interest in our case for selecting the temperature. One can compute the bound at almost no additional cost as a function of the current risk. In turns this can be used to monitor the bound. b. Break if the empirical bound increases.

End Loop We find that using a deterministic annealing algorithm with a limited amount of steps helps in finding a high enough optimum. On the left panel of we can see the one dimensional case where the initial problem γ = 0 corresponds to a convex minimization problem and where the increasing temperature gradually complexifies the optimization problem. Figure 5.B.1 shows that the solution given by DA is in average lower than randomly initialized optimization.

γ = 0 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • γ = 125 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • γ = 250 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • γ = 375 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • γ = 500 • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0
• • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • • 0.

5.C Stochastic gradient descent

The stochastic gradient descent algorithm used in Section 5.7 is described as Algorithm 22.

Algorithm 22 Stochastic Gradient Descent

Input B a batch size, an unbiased estimator of the gradient ∇B f , η ∈ (0, 1) and c While ¬converged

a. x t+1 = x t -λ t ∇B f (x t ) b. Update λ t+1 = 1 (t+c) η

End Loop

In all our experiment we take c = 1 and η = 0.9.

6

Towards the automatic calibration of the number of particles in SMC 2 This is joint work with Nicolas Chopin, Mathieu Gerber and Omiros Papaspiliopoulos Status: To be presented at SYSID 2015

Introduction

Consider a state-space model, with parameter θ ∈ Θ, latent Markov process (x t ) t≥0 , and observed process (y t ) t≥0 , taking values respectively in X and Y. The model is defined through the following probability densities: θ has prior p(θ), (x t ) t≥0 has initial law µ θ (x 0 ) and Markov transition f X θ (x t |x t-1 ), and the y t 's are conditionally independent, given the x t 's, with density f Y θ (y t |x t ). Sequential analysis of such a model amounts to computing recursively (in t) the posterior distributions p(θ, x 0:t |y 0:t ) = p(θ)µ θ (x 0 ) p(y 0:t )

t s=1 f X θ (x s |x s-1 ) t s=0 f Y θ (y s |x s )
or some of its marginals (e.g. p(θ|y 0:t )); the normalising constant p(y 0:t ) of the above density is the marginal likelihood (evidence) of the data observed up to time t. For a fixed θ, the standard approach to sequential analysis of state-space models is particle filtering: one propagates N x particles in X over time through mutation steps (based on proposal distribution q t,θ (x t |x t-1 ) at time t) and resampling steps; see Algorithm 23. Note the conventions: 1 : N x denotes the set of integers 161 {1, . . . , N x }, y 0:t is (y 0 , . . . , y t ), x 1:Nx t = (x 1 t , . . . , x Nx t ), x 1:Nx 0:t = (x 1:Nx 0 , . . . , x 1:Nx t ), and so on.

Algorithm 23 Particle filter (PF, for fixed θ)

Operations involving superscript n must be performed for all n ∈ 1 : N x . At time 0:

(a) Sample x n 0 ∼ q 0,θ (x 0 ).

(b) Compute weights w 0,θ (x n 0 ) = µ θ (x n 0 )f Y (y 0 |x n 0 ) q 0,θ (x n 0 ) normalised weights, W n 0,θ = w 0,θ (x n 0 )/ Nx i=1 w 0,θ (x i 0 ), and incremental likelihood estimate l0 (θ) = N -1

x Nx n=1 w n 0,θ .

Recursively, from time t = 1 to time t = T :

(a) Sample a n t ∼ M(W 1:Nx t-1,θ ), the multinomial distribution which generates value i ∈ 1 : N x with probability W i t-1,θ .

(b) Sample x n t ∼ q t,θ (•|x The output of Algorithm 23 may be used in different ways: at time t, the quantity Nx n=1 W n t,θ ϕ(x n t ) is a consistent (as N x → +∞) estimator of the filtering expectation E[ϕ(x t )|y 0:t , θ]; In addition, lt (θ) is an unbiased estimator of incremental likelihood p(y t |y 0:t-1 , θ), and t s=0 l(θ) is an unbiased estimator of the full likelihood p(y 0:t |θ) [Del Moral, 1996b, Lemma 3].

In order to perform joint inference on parameter θ and state variables, Chopin et al.

[2013a] derived the SMC 2 sampler, that is, a SMC (Sequential Monte Carlo) algorithm in θ-space, which generates and propagates N θ values θ m in Θ, and which, for each θ m , runs a particle filter (i.e. Algorithm 23) for θ = θ m , of size N x . One issue however is how to choose N x : if too big, then CPU time is wasted, while if taken too small, then the performance of the algorithm deteriorates. Chopin et al. [2013a] give formal results (adapted from Andrieu et al. [2010b]) that suggest that N x should grow at a linear rate during the course of the algorithm. They also propose a practical method for increasing N x adaptively, based on an importance sampling step where the N θ particle systems, of size N x , are replaced by new particle systems of size N new x . But this importance sampling step increases the degeneracy of the weights, which in return may lead to more frequent resampling steps, which are expensive. In this chapter, we derive an alternative way to increase N x adaptively, which is not based on importance sampling, but rather on a CSMC (conditional Sequential Monte Carlo) update, which is less CPU intensive.

6.2 Background on SMC 2

IBIS

To explain SMC 2 , we first recall the structure of the IBIS algorithm [Chopin, 2002b] as Algorithm 24. For a model with parameter θ ∈ Θ, prior p(θ), data y 0:T , and incremental likelihood p(y t |y 0:t-1 , θ), IBIS provides at each iteration t an approximation of partial posterior p(θ|y 0:t ). In practice, IBIS samples N θ particles θ m from the prior, then perfoms sequential importance sampling steps, from p(θ|y 0:t-1 ) to p(θ|y 0:t ) using incremental weight p(θ|y 0:t )/p(θ|y 0:t-1 ) ∝ p(y t |y 0:t-1 , θ).

To avoid weight degeneracy, one performs a resample-move step (described as Step (b) in Algorithm 24). When the ESS (effective sample size) of the weights, computed as:

ESS(ω 1:N θ ) = ( N θ m=1 ω m ) 2 N θ m=1 (ω m ) 2 ∈ [1, N ]
goes below some threshold ESS min (e.g. N/2), the θ m 's are resampled, then moved according to some Markov kernel K t that leaves invariant the current target of the algorithm, p(θ|y 0:t ). This resample-move step re-introduces diversity among the θ-particles.

A convenient default choice for K t is several iterations of random-walk Metropolis, with the random step calibrated to the spread of the current particle population (i.e. variance of random step equals some fraction of the covariance matrix of the resampled particles).

The main limitation of IBIS is that it requires evaluating the likelihood increment p(y t |y 0:t-1 , θ), which is typically intractable for state-space models. On the 163 Algorithm 24 IBIS Operations involving superscript m must be performed for all m ∈ 1 : N θ .

(Init) Sample θ m ∼ p(θ), set ω m ← 1.

From time t = 0 to time t = T , do where K t is a Markov kernel with invariant distribution p(θ|y 0:t ); finally reset particle system to θ 1:N θ ← θ1:N θ , ω 1:N θ ← (1, . . . , 1).

other hand, we have seen that this quantity may be estimated unbiasedly by particle filtering. This suggests combining IBIS (i.e. SMC in the θ-dimension) with particle filtering (i.e. SMC in the x t -dimension), as done in the SMC 2 algorithm.

SMC 2

The general structure of SMC 2 is recalled as Algorithm 25. Essentially, one recognises the IBIS algorithm, where the intractable incremental weight p(y t |y 0:t-1 , θ m ) has been replaced by the unbiased estimate lt (θ m ). This estimate is obtained from a PF run for θ = θ m ; thus N θ PFs are run in parallel. Denote (x 1:Nx,m 0:t , a 1:Nx,m 1:t ) the random variables generated by the PF associated to θ m . This 'double-layer' structure suggests that SMC 2 suffers from two levels of approximation, and as such that it requires both N x → +∞ and N θ → +∞ to converge. It turns out however that SMC 2 is valid for any fixed value of N x ; that is, for any fixed N x ≥ 1, it converges as N θ → +∞.

This property is intuitive in the simplified case when resampling-move steps are never triggered (i.e. take ESS min = 0). Then SMC 2 collapses to importance generated random variates is actually a deterministic sequence determined by the initial state (seed) of the PRNG. It is sufficient to store that initial state and z 0 in order to recover any z i in the future. The trade-off is an increase in CPU cost, as each access to z i require re-computing z 1 , . . . , z i .

We apply this idea to the variables (x 1:Nx,m 0:t , a 1:Nx,m 1:t

). By close inspection of Algorithm 25, we note that variables in a 'time slice' (x 1:Nx,m s , a 1:Nx,m s ), 0 < s ≤ t (or x 1:Nx,m 0 at time 0) are always generated jointly, either during Step (a), or during Step (c). In both cases, this time-slice is a deterministic function of the current PRNG state and the previous time slice. Thus, one may recover any time slice (when needed) by storing only (i) the PNRG state (immediately before the generation of the time slice); and (ii) in which Step (either (a) or (c)) the time slice was generated. This reduces the memory cost of SMC 2 from O(tN θ N x ) to O(N θ (t + N x )).

Compared to the path storage approach mentioned above, our PRNG recycling approach has a larger CPU cost, a smaller memory cost, and does not require any conditions on the mixing properties of process (x t ). Note that the CPU cost increase is within a factor of two, because each time a Particle Gibbs update is performed, the number of random variables that must be re-generated (i.e. the x n s and a n s in Algorithm 27) roughly equals the number of random variables that are generated for the first time (i.e. the xn s and ãn s in Algorithm 27).

Nonparametric estimation of N x

As seen in Algorithm 25, a Particle Gibbs step will be performed each time the ESS goes below some threshold. That the ESS is low may indicate that N x is also too low, and therefore that the variance of the likelihood estimates L t (θ m , x 1:Nx,m 0:t , a 1:Nx,m 1:t

) is too high. Our strategy is to update (each time a Particle Gibbs step is performed) the current value of N x to N new x = τ /σ 2 , where σ2 is some (possibly rough) estimate of the variance of the log likelihood estimates. This is motivated by results from [START_REF] Doucet | Efficient implementation of Markov chain Monte Carlo when using an unbiased likelihood estimator[END_REF], who also develop some theory that supports choosing τ ≈ 1 is optimal (although their optimality results do not extend straightforwardly to our settings).

Assume Θ ⊂ R d . To estimate σ 2 , we use backfitting to fit a GAM (generalized additive model) to the responses R m = log L t (θ m , x 1:Nx,m 0:t , a 1:Nx,m 1:t ):

R m = α + d j=1 f j (C m j ) + ε m ,
using as covariates C m j the d principal components of the resampled θ-particles. The estimate σ 2 is then the empirical variance of the residuals. See e.g. Chap. 9 of [START_REF] Hastie | The elements of statistical learning[END_REF] for more details on backfitting and GAM modelling.

We found this strategy to work well, with the caveat that choosing τ required some trial and error.

Additional considerations

Using Particle Gibbs as our PMCMC move within SMC 2 hast two advantages: (a) it makes it possible to change N x without changing the weights, as explained above; and (b) it also makes it possible to update the θ m according to Gibbs or Metropolis step that leaves θ|x 0:t , y 0:t invariant); see Step (3) of Algorithm 27. For models where sampling from θ|x 0:t , y 0:t is not convenient, one may instead update θ through several PMMH steps performed after the Particle Gibbs step.

Numerical example

We consider the following stochastic volatility model: x 0 ∼ N (µ, σ 2 /(1ρ 2 )), x tµ = ρ(x t-1µ) + σǫ t , ǫ t ∼ N (0, 1) and y t |x t ∼ N (0, e xt ); thus θ = (µ, ρ, σ), with ρ ∈ [-1, 1], σ > 0. We assign independent priors to the components of θ: µ ∼ N (0, 2 2 ), ρ ∼ N (0, 1) constrained to [-1, 1], and σ 2 ∼ IG(3, 0.5). The dataset consists in log-returns from the monthly SP500 index, observed from 29/05/2013 to 19/12/2014; T = 401. Figure 6.1 plots the marginal posterior p(ρ, σ 2 |y 0:15 ), as approximated by SMC 2 , run up to time 15. This figure illustrates the need for modelling nonparametrically the true likelihood as a function of θ, in order to estimate the variance of the estimated likelihood. runs are incomplete, as they were stopped when the CPU time exceeded 10 hours. On the other hand, the CPU cost of other versions is more stable across runs, and, more importantly, quite lower. (Recall that Algorithm (c) does not perform PMMH steps). Note the poor performance of Algorithm (b). Figure 6.5 compares the box-plots of posterior estimates of σ at final time T , obtained from several runs of Algorithms (c) and (d). Algorithm (c) shows slightly less variability, while being 30% faster on average. One sees that the improvement brought by ability to sample from θ|x 0:t , y 0:t is modest here for parameter estimation, but recall that in Figure 6.3, the improvement was more substantial.

Figure 1

 1 Figure 1.1: Illustration of the assumption

  Example 1.3.1 Truncated Normal distribution The aim is to sample Y ∼ N [a,b] (m, σ 2 ). The c.d.f of a (m, σ 2 ) Gaussian truncated to [a, b] is given by: F (x; m, σ, a, b) algorithm consisting in sampling a uniform distribution U on [0, 1] and taking

Figure 1 . 2 :

 12 Figure 1.2: Sequence of estimator of the parameter of a symmetric Skellam distribution

Figure 1

 1 Figure 1.3: ACF and trace plot of the Gibbs sampler for the Skellam distribution

Figure 1

 1 Figure 1.4: An example of an MCMC run with negative autocorrelationWe show the trace plot for the two first marginals of the posterior of a probit model on the Pima dataset, the second panel shows the ACF of the first marginal.

Figure 1

 1 Figure 1.5: Expectation Propagation in Action In panel (a) we show the global approximation Q (initial point), the site approximation is removed to get Q \j in panel (b). Panel (c) and (d) show the site (indicator function) and the hybrid h(θ) ∝ N (θ; m, s)✶ θ>0 (in purple). The closest Gaussian to the hybrid is the new approximation panel (e) (the one with the same mean and variance).

Figure 1 . 6 :

 16 Figure 1.6: Marginal of the Gaussian approximation of a regression with Student priors

Figure 1 . 7 :

 17 Figure 1.7: Mean field VB approximation (red) of bivariate Gaussian distribution with correlation ρ = 0.99 (blue)

  Figure 2.3: IRIS (Inefficiency relative to importance sampling) across all datasets for MCMC schemes and Gaussian/probit scenario; left (resp. right) panel shows median IRIS when estimating the p posterior expectations (resp. the p posterior variances).

Figure 2

 2 Figure 2.4: Marginal accuracies across the p dimensions of EP and Laplace, for datasets Musk, Sonar and DNA

Figure 2

 2 Figure 2.5: Effective sample size for a fixed CPU time for samplingbased algorithms: posterior expectations (left), and posterior variances (right) for datasets (from top to bottom): Musk, Sonar, and ADN

Figure 2 . 6 :

 26 Figure 2.6: Variation of estimated inclusion probabilities p(γ j = 1|D) over 50 runs for the p covariates of Musk dataset: median (red line), 80% confidence interval (white box); the blackbox extends until the maximum value.

Figure 3

 3 Figure 3.1: Estimates of Orthant probabilities by Particle FilterEstimation of the log probability that an AR(1) process (defined previously) with ̺ = 0.7 has all its component in[0, 15]. GHK sampler (grey) and PF (white) on various dimension from 100 to 200. On the right panel the two ESS for dimension 200. On both cases M is set to 1000.

Figure 3

 3 Figure 3.2: Estimates of the likelihood of a Thurstonian model by Particle Filter and by GHK Estimation of the likelihood of a Thurstoninan model with p = 10 and the number of observations is ranging from 1 to 5 the PF (white) and the GHK (grey). The threshold ESS is set to 0.5M and the number of particles is set to M = 1000. The right panel shows the ESS of both algorithms for T = 1 and p = 40.

Figure 3

 3 Figure 3.3: Estimates of Orthant probabilities with (white) and without (grey) variable ordering

Figure 3

 3 Figure 3.4: Truncated Student sampling after thinning one out of every 1000, using a Gibbs sampler

  Figure 3.5: Generated correlationsCovariance matrices generated from random samples with heavy tails. The various dimension are simulated with the same algorithm and same seed, such that the small ones are subsets of the others. The left panel shows a heatmap of the correlation, the right panel the left trucation of the integral.

Figure 3 . 6 :

 36 Figure 3.6: Estimates of Orthant probabilities for GHK compared to SMC with different moves.

Figure 3 . 7 :

 37 Figure 3.7: Estimates of Orthant probabilities p = 130 and p = 180

Figure 3 . 8 :

 38 Figure 3.8: Estimates of Student orthant probabilities SMC vs GHK

Figure 3 . 9 :

 39 Figure 3.9: Estimates of the likelihood of a multivariate Probit

Figure 3

 3 Figure 3.D.1: Estimates of Orthant probabilities Gibbs vs HMC

  Definition 4.2.2 (Mammen & Tsybakov margin assumption) We say that Assumption MA(κ, C) is satisfied for κ ∈ [1, +∞] and C ≥ 1 if

  Theorem 4.2.3 Assume MA(1, C), C ≥ 1, Dens(c), c > 0, and take ϑ = 2 d (1 + 1 n 2 d ), γ = (n -1)/8C, then there exists a constant α = α(c, C, d) such that for any ǫ > 0, with probability 1ǫ,

Figure 1 Figure 4

 14 Figure1compares the EP approximation with the output of our SMC sampler, on the well-known Pima Indians dataset and a Gaussian prior. Marginal first and second order moments essentially match; see the appendix for further details. The subsequent results are obtained with EP.

Figure 4 . 3 :

 43 Figure 4.3: Regularization plot for v 0 ∈ 10 -6 , 0.1 and estimation for v 0 = 10 -6 for DNA dataset; blue circles denote posterior probabilities ≥ 0.5.

Figure 4

 4 Figure4.2: Some ROC curves associated to the example described in a more systematic manner in table 5.2. In black is always the PAC version.

Figure 4 .

 4 Figure 4.C.1 shows the posterior marginals as given by EP and tempering SMC. The later is exact in the sense that the only error stems from Monte Carlo; we

Figure

  Figure 4.C.1: Comparison of the output of the two algorithms

  Definition 5.2.1 We define, for any λ > 0, the pseudo-posterior ρλ by ρλ (dθ) = exp[-λr n (θ)] ´exp[-λr n ]dπ π(dθ).

  ρ, ρλ ) denotes the KL (Küllback-Leibler) divergence of ρλ relative to ρ:K(m, µ) = ´log[ dm dµ ]dm if m ≪ µ (i.e. µ dominates m), K(m, µ) = +∞ otherwise.

  with bounded design |X ij | < c x , Hoeffding assumption (5.1) is satisfied with f

Figure 5

 5 Figure 5.1: Error bound at each iteration, stochastic descent, Pima and Adult datasets. Stochastic VB with fixed temperature λ = 100 for Pima and λ = 1000 for adult. The left panel shows several curves that correspond to different batch sizes; these curves are hard to distinguish. The right panel is for a batch size of 50. The adult dataset has n = 32556 observation and n + n -= 193829520 possible pairs. The convergence is obtained in order of seconds. The bounds are the empirical bounds obtained in Corollary 5.7.3 for a probability of 95%.

  2 log(2)+ (m 1 + m 2 )r log 1 δ 3π(m 1 ∨ m 2 )K 4 + 2K log Γ(a)3 a+1 exp(2) b a+1 2 aas soon as the restriction b ≤ δ 2

a.

  Sample U ∼ U([0, 1]). b. Compute cumulative weights as C n = n m=1 N W m . c. Set s ← U , m ← 1. d. For n = 1 : N While C m < s do m ← m + 1.

Algorithm 21

 21 Deterministic annealingInput (λ t ) t∈[0,T ] a sequence of inverse temperature Init. Set m = 0 and Σ = ϑI d , the values minimizing the KL-divergence for λ = 0 Loop t=1,. . . ,T a. m λt , Σ λt = Minimize L λt (m, Σ) using some local optimization routine with initial points m λ t-1 , Σ λ t-1

Figure 5

 5 Figure 5.B.1: Deterministic annealing on a Pima Indians with one covariate and full model resp. The right panel gives the empirical bound obtained for the DA method (in red) and the dot are direct global optimization based on L-BFGS algorithms from starting values drawn from the prior. Each optimization problem is repeated 20 times.

Figure

( a )

 a Update importance weightsω m ← ω m × p(y t |y 0:t-1 , θ). (b) If ESS(ω 1:N θ ) ≤ ESS min , sample (for all m) θm from mixture 1 N θ m=1 ω m N θ m=1 ω m K t (θ m , dθ),

Figure 6 . 2 :

 62 Figure 6.2: Evolution of N x over time for 5 runs of the four considered SMC 2 algorithms; red dotted line is Algorithm (a), blue dashed is (b), black solid is (c), green double-dashed is (d). Results of (c) and (d) are nearly undistinguishable.

Figure 6 .Figure 6

 66 Figure6.3 plots the empirical variance of the estimated marginal likelihood (evidence, p(y 0:t )), normalised with the running time up to time step t. One observes that version (c) does quite better than (d), and far much better than (a). Results from Algorithm (b) were to variable to be included.

Figure 6 .

 6 Figure 6.4 plots the acceptance rate of PMMH steps for Algorithms (a), (b) and (d). (Recall that Algorithm (c) does not perform PMMH steps). Note the poor performance of Algorithm (b).Figure 6.5 compares the box-plots of posterior estimates of σ at final time T , obtained from several runs of Algorithms (c) and (d). Algorithm (c) shows slightly less variability, while being 30% faster

Figure 6

 6 Figure 6.5: Box-plots of posterior estimate of parameter σ at final time T , over repeated runs of Algorithm (c) (left panel) and Algorithm (d) (right panel).

Table 2

 2

	1.00
		•
	•
		•
	0.95
	0.90
	value		•
	0.85
			•
	0.80
				•
	EP	Improved Laplace	Laplace	Laplace EM
		(d) German

.2: Performance of importance sampling (IS), and QMC importance sampling (IS-QMC), on all datasets, in Gaussian/probit scenario: efficiency factor (EF), CPU time (in seconds), speed gain when using multi-threading Intel hyper-threaded quad core CPU (Speed gain MT), and efficiency gain of QMC (see text).

  Dataset Covariates Balance EP-AUC GPEP-AUC Logit Rankboost

	Pima	7	34%	0.8617	0.8557	0.8646	0.8224
	Credit	60	28%	0.7952	0.7922	0.7561	0.788
	DNA	180	22%	0.9814	0.9812	0.9696	0.9814
	SPECTF	22	50%	0.8684	0.8545	0.8715	0.8684
	Colon	2000	40%	0.7034	0.75	0.73	0.5935
	Glass	10	1%	0.9843	0.9629	0.9029	0.9436
			Table 4.1: Comparison of AUC.

  Balance EP-AUC GPEP-AUC Rankboost

	Pima	7	34%	0.06	7.75	3.26
	Credit	60	28%	1.98	7.59	56.54
	DNA	180	22%	11.26	63.47	141.60
	SPECTF	22	50%	0.25	63.47	3.55
	Colon	2000	40%	636.63	60.99	156.85
	Glass	10	1%	0.23	1.33	2.36
		Table 4.C.1: Computation times in seconds
						121

  Dataset Covariates Mean Field (F 2 ) Full cov. (F 3 ) SMC SVM

	Pima	7	31.0	21.3	22.3	30.4
	Credit	60	32.0	33.6	32.0	32.0
	DNA	180	23.6	23.6	23.6	20.4
	SPECTF	22	08.0	06.9	08.5	10.1
	Glass	10	34.6	19.6	23.3	4.7
	Indian	11	48.0	25.5	26.2	26.8
	Breast	10	35.1	1.1	1.1	1.7

Table 5 .

 5 1: Comparison of misclassification rates (%).

  function with linear mappings we have the result. Similar reasoning could be held for the case F 2 and F 3 , where in later the parametrization should be done in C such that Σ = CC t . The bound is however not universally Lipschitz in σ, this impacts the optimization algorithms.On the class of functionF 0 = Φ m, 1 n , m ∈ R d, for which our Oracle inequalities still hold we could get faster numerical algorithms. The objective function has Lipschitz continuous derivatives and we would get a rate of L (1+k) 2 . Other convex loss could be considered which could lead to convex optimization problems. For instance one could consider the exponential loss.

	Ψ :	x y	→ xΦ	x y	+ yϕ	x y	,
	is convex and note that we can write Ξ i		m σ	= Ψ A	x y	+ b hence by
	composition of convex Dataset Covariates Hinge loss SMC
	Pima		7		21.8		22.3
	Credit		60		27.2		32.0
	DNA		180		4.2			23.6
	SPECTF		22		19.2		08.5
	Glass		10		26.12		23.3
	Indian		11		26.2		25.5
	Breast		10		0.5			1.1
	Table 5.2: Comparison of misclassification rates (%).

  1 )} 2 N i=1 {w t (θ i t-1 )) 2 } = τ N, w t (θ) = exp[-(λ tλ t-1 )r n (θ)] (5.18) using bisection search. If λ t ≥ λ T , set Z T = Z t-1 × 1 Resample: for i = 1 to N , draw A i t in 1, . . . , N so that P(A i t = j) = w t (θ j t-1 )/ N k=1 w t (θ k t-1 ); see Algorithm 20 in the appendix. c. Sample θ i t ∼ M t (θ , dθ) for i = 1 to N where M t is a MCMC kernel that leaves invariant π t ; see comments below. d. Set Z t = Z t-1 × 1

		N	N i=1 w t (θ i t-1 ) ,
	and stop.
	A i t t-1 N	N i=1 w t (θ i t-1 ) .

b.

θθ ′

d i=1 t ν (β i )

Computation of Gaussian orthant probabilities in high dimensioncan outperform a classical Gibbs sampler when the dimension exceeds 20.

All available at http://archive.ics.uci.edu/ml/

Remerciements

Appendix

3.A Proof of proposition 2.1

Proof: We have that for 0 < b < ∞ the transition density p b (x, y)dy, associated with the kernel P b (x, dy) with respect to the Lebesgue measure, is lower bounded by a constant and the transition is continuous. This Markov chain is a ψ-irreducible on a compact support. Hence we can show that the whole support [0, b] is small [START_REF] Meyn | Markov chains and Stochastic Stability[END_REF].

Hence by theorem 16.1.2 to show V-Uniform ergodicity the transition must satisfy the drift condition: ˆp(x, y)V e (y)dy ≤ (1β)V e (x) + c✶ [0,b] (x) for β > 0, c < ∞ and a certain V e (x) with value in [1, ∞). We take V e (x) = ex 2 +1, e > 0. In the following we check this condition.

The left hand side is given by E(X 2 t |X t-1 = x) for the above transition probability,

The ratio is continuous on the bounded set [0, b], and can be bounded by a constant, such that the by taking β = 1 -̺ 2 > 0 the drift condition is satisfied for a c(e) depending on e.

In addition we can compute exactly the invariant measure. It is unique and given by the solution of: A simple sufficient condition for Dens(c) to hold is that (X 1 -X 2 )/ X 1 -X 2 admits a probability density with respect to the spherical measure of dimension d -1 which is bounded above by B. Then

4.A.2 Proof of Lemma 2.1

In order to prove Lemma 2.1 we need the following Bernstein inequality.

Proposition 4.A.1 (Bernstein's inequality for U-statistics) For any γ > 0,

First, note that

Algorithm 16 Systematic resampling

Input: Normalised weights W j t := w t (θ j t-1 )/ N i=1 w t (θ i t-1 ).

Output:

b. Compute cumulative weights as

A n ← m, and s ← s + 1.

End For by approximating each site t i (θ) by a distribution from an exponential family q i (θ).

The algorithm cycles through each site, computes the cavity distribution Q \i (θ) ∝ Q(θ)q -1 i (θ) and minimizes the Kullback-Leibler divergence between Q \i (θ)t i (θ) and the global approximation Q(θ). This is efficiently done by using properties of the exponential family (e.g. Bishop [2006a]).

In the Gaussian case the EP approximation can be written as a product of some prior and a product of sites:

for which the sites are unnormalized Gaussians for the natural parametrization

going from one to the other is easily done by multiplying θ by (e ie j )X where ∀i ∈ {1, • • • , n}, e i is a vector of zeroes with one on the i-th line. Hence we keep in memory only (K ij ) ij and (h ij ) ij .

While computing the cavity moment we must compute (Q -(X i -X j )(X i -X j )K ij ) and its inverse. The latter can be computed efficiently using Woodbury formula. Equivalently one could use similar tricks where only the Cholesky factorisation is saved and updated as in Seeger [2005a]. By precomputing some matrix multiplication the later cavity moment computation can be done in complexity O(p 2 ).

Algorithm 17 Gaussian random walk Metropolis step

Input: θ, S (d × d positive matrix)

To update the sites we compute normalizing constant Z ij = ´N (s; m \ij , σ \ij )t ij (s)ds and use properties of exponential families.

Normalising Constant

The normalizing constant of the posterior can be computed using EP. We have that for each sites t ij (θ) = C ij q ij (θ) we replace those sites in integral we wish to approximate,

The integral on the right hand side is a Gaussian convolution and is therefore also Gaussian. The C ij s can be approximated by matching the zeroth order moment in the site update. As noted in the chapter we can also compute the derivatives with respect to some prior hyper-parameter (see Seeger [2005a]).

4.B.3 Expectation-Propagation (spike and slab prior)

The posterior can be written as

where z k ∈ {0, 1} codes the origin of θ k , spike/slab, and where

). The approximation given by EP is of the form,

5

Properties of variational approximations of Gibbs posteriors

This is joint work with Pierre Alquier and Nicolas Chopin Status: Submitted to Journal of Machine Learning Rechearch

Introduction

A Gibbs posterior, also known as a PAC-Bayesian or pseudo-posterior, is a probability distribution for random estimators of the form:

More precise definitions will follow, but for now, θ may be interpreted as a parameter (in a finite or infinite-dimensional space), r n (θ) as an empirical measure of risk (e.g. prediction error), and π(dθ) a prior distribution. We will follow in this chapter the PAC (Probably Approximatively Correct)-Bayesian approach, which originates from machine learning [START_REF] Catoni | Statistical learning theory and stochastic optimization[END_REF][START_REF] Mcallester | Some pac-bayesian theorems[END_REF][START_REF] Shawe-Taylor | A PAC analysis of a Bayesian estimator[END_REF]; see [START_REF] Catoni | PAC-Bayesian Supervised Classification[END_REF] for an exhaustive study, and [START_REF] Dalalyan | Aggregation by exponential weighting, sharp PAC-Bayesian bounds and sparsity[END_REF]; [START_REF] Jiang | Gibbs posterior for variable selection in highdimensional classification and data mining[END_REF]; [START_REF] Yang | Aggregating regression procedures to improve performance[END_REF]; [START_REF] Zhang | Information theoretical upper and lower bounds for statistical estimation[END_REF] for related perspectives (such as the aggregation of estimators in the last 3 papers). There, ρλ appears as the probability distribution that minimises the upper bound of an oracle inequality on the risk of random estimators. The PAC-Bayesian approach offers sharp theoretical guarantees on the properties of such estimators, without assuming a particular model for the data generating process.

123

We observe i.i.d. pairs ((X i , Y i )) n i=1 where X i ∈ {1, . . . , m 1 } × {1, . . . , m 2 }, and we assume that there is a

where • F stands for the Frobenius norm.

A common way to parametrise the problem is

where K is large; e.g. K = min(m 1 , m 2 ). Following [START_REF] Salakhutdinov | Bayesian probabilistic matrix factorization using Markov chain Monte Carlo[END_REF], we define the following prior distribution:

where the γ j 's are i.i.d. from an inverse gamma distribution, γ j ∼ IΓ(a, b).

Note that VB algorithms were used in this context by [START_REF] Lim | Variational Bayesian approach to movie rating prediction[END_REF] (with a slightly simpler prior however: the γ j 's are fixed rather than random). Since then, this prior and variants were used in several papers [e.g. [START_REF] Lawrence | Non-linear matrix factorization with Gaussian processes[END_REF][START_REF] Zhou | Nonparametric bayesian matrix completion[END_REF]. Until now, no theoretical results were proved up to our knowledge. Two papers prove minimax-optimal rates for slightly modified estimators (by truncation), for which efficient algorithms are unknown [START_REF] Mai | A Bayesian approach for matrix completion: optimal rate under general sampling distribution[END_REF][START_REF] Suzuki | Convergence rate of Bayesian tensor estimator: Optimal rate without restricted strong convexity[END_REF]. However, using Theorems 5.4.2 and 5.4.3 we are able to prove the following: if there is a PAC-Bayesian bound leading to a rate for ρλ in this context, then the same rate holds for ρλ . In other words: if someone proves the conjecture that the Gibbs estimator is minimax-optimal (up to log terms) in this context, then the VB approximation will enjoy automatically the same property.

We propose the following approximation:

note that the prior π does not depend on the knowledge of r though).

Choose the prior distribution on the hyper-parameters γ j as inverse gamma Inv-Γ(a, b)

Then there is a constant C(a, C) such that, for any β > 0,

the risk function is not bounded, which makes our approach a bit less direct to apply. In many papers on PAC-Bayesian bounds for regression, the noise can be unbounded (usually, it is assumed to be sub-exponential), but one assumes that the predictors are bounded, see e.g. [START_REF] Alquier | Sparse single-index model[END_REF]. However, using the robust loss function of Audibert and Catoni, it is possible to relax this assumption [START_REF] Audibert | Robust linear least squares regression[END_REF][START_REF] Catoni | Challenging the empirical mean and empirical variance: A deviation study[END_REF]. This requires a more technical analysis, which we leave for further work.

Appendix 5.A Proofs

5.A.1 Preliminary remarks

We start by a general remark. Let h be a function

Direct calculation yields, for any ρ ≪ π with ´hdρ < ∞,

Two well known consequences are

We will use these inequalities many times in the followings. The most frequent application will be with

, the first case leads to

Operations involving superscript m must be performed for all m ∈ 1 : N θ .

(Init) Sample θ m ∼ p(θ), set ω m ← 1.

From time t = 0 to time t = T , do (a) For each θ m , run iteration t of Algorithm 23, so as to obtain (x 1:Nx,m 0:t , a 1:Nx,m 1:t

), and lt (θ m ).

where K t is a PMCMC kernel with invariant distribution π t (θ, x 1:Nx 0:t , a 1:Nx 1:t ) (see text); finally reset particle system to (θ m , x 1:Nx,m 0:t sampling, with weights replaced by unbiased estimates, and it is easy to show convergence from first principles.

We now give a brief outline of the formal justification of SMC 2 for fixed N x , and refer to Chopin et al. [2013a] for more details. SMC 2 may be formalised as a SMC sampler for the sequence of extended distributions:

where ψ t,θ denotes the joint pdf of the random variables generated by a PF up to time t (for parameter θ), and ls (θ) denotes the unbiased estimate of the likelihood increment computed from that PF, l0 (θ) = N -1

x N n=1 w 0 (x n 0 ), ls (θ) = N -1

x N n=1 w s,θ (x a n t s-1 , x n s ) for s > 0; i.e. ls (θ) is actually a function of (θ, x 1:Nx 0:s , a 1:Nx 1:s ). One recognises in π t the type of extended target distribution simulated by PM-CMC (Particle MCMC, Andrieu et al. [2010b]) algorithms. Note π t is a proper probability density (it integrates to one), and that the marginal distribution of θ is p(θ|y 0:t ). These two properties are easily deduced from the unbiasedness of t s=0 ls (θ) (as an estimator of p(y 0:t |θ)). In addition, π t (θ, x 1:Nx 0:t , a 1:Nx 1:t ) = π t-1 (θ, x 1:Nx 0:t-1 , a 1:Nx 1:t-1 )

where one recognises in the second factor the distribution of the variables generated by a PF at time t, conditional on those variables generated up to time t -1. Thus, the equation above justifies both Step (a) of Algorithm 25, where the particle filters are extended from time t -1 to t, and Step (b), where the particles (θ m , x 1:Nx,m 0:t , a 1:Nx,m 1:t

) are reweighted by lt (θ m ). We describe in the following section PMCMC moves that may be used in Step (c). Before, we note that a naive implementation of SMC 2 has a O(tN x N θ ) memory cost at time t, as one must stores in memory (θ m , x 1:Nx,m 0:t , a 1:Nx,m 1:t

) for each m ∈ 1 : N θ . This memory cost may be substantial even on a modern computer.

PMCMC moves

To make more explicit the dependence of the unbiased estimate of the likelihood on the variables generated during the course of PF, define

The PMMH (Particle Markov Metropolis-Hastings) kernel, described as Algorithm 26, may be described informally as a Metropolis step in θ-space, where the likelihood of both the current value and the proposed value have been replaced by unbiased estimators. Formally, as proven in Andrieu et al. [2010b], it is in fact a standard Metropolis step with respect to the extended distribution π t (θ, x 1:Nx 0:t , a 1:Nx 1:t ); in particular it leaves invariant p(θ|y 0:t ). (For convenience, our description of PMMH assumes a random walk proposal, but PMMH is not restricted to this kind of proposal.)

In practice, we set Σ t , the covariance matrix of the proposal, to a fraction of the covariance matrix of the resampled θ-particles.

One advantage of using PMHMH within SMC 2 is that it does not require storing all the variables generated by the N θ PFs: operations at time t > 0 require only having access to, for each m, (θ m , x 1:Nx,m t-1 , a 1:Nx,m t-1

) and L t-1 (θ m , x 1:Nx,m 0:t-1 , a 1:Nx,m 1:t

), which is computed recursively. Memory cost then reduces to O(N θ N x ).

The Particle Gibbs approach is an alternative PMCMC step, based on the following property of target π t : if one extends π t with random index k, such that Algorithm 26 Random walk PMMH update Input: (θ, x 1:Nx 0:t , a 1:Nx 1:t ) Output: ( θ, x1:Nx 0:t , ã1:Nx

2. Generate PF (Algorithm 23) for parameter θ ⋆ ; let (x 1:Nx,⋆ 0:t , a 1:Nx,⋆ 1:t

) the output.

k ∈ 1 : N x , and k ∼ M(W 1:Nx T ), the normalised weights at the final iteration, then (a) the selected trajectory, together with θ, follow the posterior distribution p(θ, x 0:t |y 0:t ); and (b) the remaining arguments of π t follow a CSMC (conditional SMC) distribution, which corresponds to the distribution of the random variables generated by a PF, but conditional on one trajectory fixed to the selected trajectory; see Algorithm 27.

In contrast with PMMH, implementing particle Gibbs steps within SMC 2 requires having access to all the variables (θ m , x 1:Nx,m 0:t , a 1:Nx,m 1:t ) at time t, which as we have already discussed, might incur too big a memory cost. et al. [2010b] show that, in order to obtain reasonable performance for PMMH, one should take N x = O(t). [START_REF] Andrieu | Uniform Ergodicity of the Iterated Conditional SMC and Geometric Ergodicity of Particle Gibbs samplers[END_REF] show a similar result for Particle Gibbs.

Choosing N x

Andrieu

In the context of SMC 2 , this suggests that N x should be allowed to increase in the course of the algorithm. To that effect, Chopin et al. [2013a] devised an exchange step, which consists in exchanging the current particle systems, of size N x , with new particle systems, of size N new

x , through importance sampling. In Chopin et al. [2013a] ), for n ∈ 2 : N , and x n t ∼ q 1, θ(•|x ãn 1 t-1 ), and so on.

acy immediately after the resampling step. In particular, we will observe in our simulations that this prevents us from changing N x too frequently, as the ESS of the weights then becomes too low.

In this chapter, we discuss how to use a Particle Gibbs step in order to increase N x without changing the weights.

Proposed approach

Particle Gibbs and memory cost

We first remark that the Particle Gibbs step, Algorithm 27, offers a very simple way to change N x during the course of the algorithm: In Step (2), simply regenerate a particle system (conditional on selected trajectory x1 0:t ) of size N new x . But, as already discussed, such a strategy requires then to access past particle values x n s (and also a n s ), rather than only current particle values x n t . This problem may be addressed in two ways. First, one may remark that, to implement Particle Gibbs, one needs to store only those x n s (and a n s ) which have descendant among the N x current particles x n t . [START_REF] Jacob | Path storage in the particle filter[END_REF] developed such a path storage approach, and gave conditions on the mixing of Markov chain (x t ) under which this approach has memory cost O(t + N x log N x ) (for a single PF with N x particles, run until time t). Thus, an implementation of this approach within SMC 2 would lead to a O(N θ (t + N x log N x )) memory cost.

A second approach, developed here, exploits the deterministic nature of PRNGs (pseudo-random number generators): a sequence z 0 , z 1 , . . . , z i , . . . of computer- For this model, sampling jointly from θ|x 0:t , y 0:t is difficult, but it is easy to perform a Gibbs step that leaves invariant θ|x 0:t , y 0:t , as the full conditionals of each component (e.g. µ|σ, ρ, x 0:t , y 0:t and so on) are standard distributions. Let's call 'full PG' Algorithm 27, where Step 2 consists of this Gibbs step for θ|x 0:t , y 0:t ; and conversely let's call 'partial PG' Algorithm 27 with θ = θ in Step 2 (θ is not updated).

We compare four versions of SMC 2 : (a) the standard version, as proposed in Chopin et al. [2013a] (i.e. Step (c) of Algorithm 25 is a PMMH step, and that step is followed by an exchange step to double N x when the acceptance rate of PMMH is below 20%); (b) the same algorithm, except that an exchange step is systematically performed after Step (c), and N x is set to the value obtained with our non-parametric approach (see Section 6.3.2); (c) the version developed in this chapter, with full PG steps (and N x updated through the non-parametric procedure); (d) the same algorithm, but with partial PG steps, followed by 3 PMMH steps to update θ.

The point of Algorithm (b) is to show that adapting N x too often during the course of the algorithm is not desirable when using the exchange step, as this leads to too much variance. The point of Algorithm (d) is to see how our approach performs when sampling from θ|x 0:t , y 0:t (either independently or through MCMC) is not feasible. Figure 6.2 plots the evolution of N x over time for the four SMC 2 algorithms. One sees that, for these model and dataset, the CPU cost of the standard SMC 2 algorithm is quite volatile, as N x increases very quickly in certain runs. In fact certain