Houssam Zenati

Alberto Bietti

Matthieu Martin

Eustache Diemert

Julien Mairal

Counterfactual Learning of Stochastic Policies with Continuous Actions: from Models to Offline Evaluation

Keywords: counterfactual learning, offline contextual bandits, continuous actions

published or not. The documents may come

L'archive ouverte

Introduction

Logged interaction data is widely available in many applications such as drug dosage prescription [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], recommender systems [START_REF] Li | An unbiased offline evaluation of contextual bandit algorithms with generalized linear models[END_REF], or online auctions [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF]. An important task is to leverage past data in order to find a good policy for selecting actions (e.g., drug doses) from available features (or contexts), rather than relying on randomized trials or sequential exploration, which may be costly to obtain or subject to ethical concerns.

More precisely, we consider offline logged bandit feedback data, consisting of contexts and actions selected by a given logging policy, associated to observed rewards. This is known as bandit feedback, since the reward is only observed for the action chosen by the logging policy. The problem of finding a good policy thus requires a form of counterfactual reasoning to estimate what the rewards would have been, had we used a different policy. When the logging policy is stochastic, one may obtain unbiased reward estimates under a new policy through importance sampling with inverse propensity scoring [IPS,[START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF]. One may then use this estimator or its variants for optimizing new policies without the need for costly experiments [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF][START_REF] Dudik | Doubly robust policy evaluation and learning[END_REF][START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF], an approach also known as counterfactual risk minimization (CRM). While this setting is not sequential, we assume that learning a stochastic policy is required so that one may gather new exploration data after deployment.

In this paper, we focus on stochastic policies with continuous actions, which, unlike the discrete setting, have received little attention in the context of counterfactual policy optimization [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF][START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF][START_REF] Chen | Personalized dose finding using outcome weighted learning[END_REF]. As noted by Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] and as our experiments confirm, addressing the continuous case with naive discretization strategies performs poorly. Our first contribution is about data modeling: we introduce a joint embedding of actions and contexts relying on kernel methods, which takes into account the continuous nature of actions, leading to rich classes of estimators that prove to be effective in practice.

In the context of CRM, the problem of estimation is intrinsically related to the problem of optimization of a non-convex objective function. In our second contribution, we underline the role of optimization algorithms [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF]. We believe that this aspect was overlooked, as previous work has mostly studied the effectiveness of estimation methods regardless of the optimization procedure. In this paper, we show that appropriate tools can bring significant benefits. To that effect, we introduce differentiable estimators based on soft-clipping the importance weights, which are more amenable to gradient-based optimization than previous hard clipping procedures [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF][START_REF] Wang | Optimal and adaptive off-policy evaluation in contextual bandits[END_REF]. We also find that proximal point algorithms [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF] tend to dominate simpler off-the-shelf optimization approaches, while keeping a reasonable computation cost.

Finally, an open problem in counterfactual reasoning is the difficult question of reliable evaluation of new policies based on logged data only. Despite significant progress thanks to various IPS estimators, we believe that this issue is still acute, since we need to be able to estimate the quality of policies and possibly select among different candidate ones before being able to deploy them in practice. Our last contribution is a small step towards solving this challenge, and consists of a new offline evaluation benchmark along with a new large-scale dataset, which we call CoCoA, obtained from a real-world system. The key idea is to introduce importance sampling diagnostics [START_REF] Owen | Monte Carlo theory, methods and examples[END_REF] to discard unreliable solutions along with significance tests to assess improvements to a reference policy. We believe that this contribution will be useful for the research community; in particular, we are not aware of similar publicly available large-scale datasets for continuous actions.

Organization of the paper

We present below the different sections of this paper and summarize the contributions in each part:

• Modeling of continuous action policies: we review the CRM framework and introduce our counterfactual cost predictor (CCP) parametrization which uses a joint embedding of contexts and actions to parametrize stochastic policies.

• Optimization perspectives for CRM: we introduce a differentiable and smooth clipping strategy for importance sampling weights and explain how proximal point algorithms can be used to better optimize CRM objectives.

• Evaluation on real data: we release an open and real-world dataset for the logged bandit feedback problem with continuous actions. Moreover, we introduce our offline evaluation protocol that tackles the underlooked problem of offline evaluation in logged bandits problems.

• Experimental results: we provide extensive experiments to validate our proposed offline protocol and to discuss the relevance of continuous action strategies, the performance of our CCP model, as well as the optimization strategies.

We then discuss open questions for future research directions, such as the difficulty of developping a doubly robust estimator based on our CCP model, and on the links between optimization strategies and offline evaluation methods.

Related Work

A large effort has been devoted to designing CRM estimators that have less variance than the IPS method, through clipping importance weights [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF][START_REF] Wang | Optimal and adaptive off-policy evaluation in contextual bandits[END_REF], variance regularization [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF], or by leveraging reward estimators through doubly robust methods [START_REF] Dudik | Doubly robust policy evaluation and learning[END_REF][START_REF] Robins | Semiparametric efficiency in multivariate regression models with missing data[END_REF]. In order to tackle an overfitting phenomenon termed "propensity overfitting", Swaminathan and Joachims [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF] also consider self-normalized estimators [START_REF] Owen | Monte Carlo theory, methods and examples[END_REF]. Such estimation techniques also appear in the context of sequential learning in contextual bandits [START_REF] Agarwal | Taming the monster: A fast and simple algorithm for contextual bandits[END_REF][START_REF] Langford | The epoch-greedy algorithm for multi-armed bandits with side information[END_REF], as well as for off-policy evaluation in reinforcement learning [START_REF] Jiang | Doubly robust off-policy value evaluation for reinforcement learning[END_REF]. In contrast, the setting we consider is not sequential.

While most approaches for counterfactual policy optimization tend to focus on discrete actions, few works have tackled the continuous action case, again with a focus on estimation rather than optimization. In particular, propensity scores for continuous actions were considered by Hirano and Imbens [START_REF] Hirano | Applied Bayesian modeling and causal inference from incomplete-data perspectives[END_REF]. More recently, evaluation and optimization of continuous action policies were studied in a non-parametric context by Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], and by Demirer et al. [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF] in a semi-parametric setting.

In contrast to these previous methods, (i) we focus on stochastic policies while they consider deterministic ones, even though the kernel smoothing approach of Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] may be interpreted as learning a deterministic policy perturbed by Gaussian noise. (ii) The terminology of kernels used by Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] refers to a different mathematical tool than the kernel embedding used in our work. We use positive definite kernels to define a nonlinear representation of actions and contexts in order to model the reward function, whereas Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] use kernel density estimation to obtain good importance sampling estimates. Chen et al. [START_REF] Chen | Personalized dose finding using outcome weighted learning[END_REF] also use a kernel embedding of contexts in their policy parametrization, while our method jointly models contexts and actions. Moreover, their method requires computing an n × n Gram matrix, which does not scale with large datasets; in principle, it should be however possible to modify their method to handle kernel approximations such as the Nyström method [START_REF] Williams | Using the nyström method to speed up kernel machines[END_REF]. Besides, their learning formulation with a quadratic problem is not compatible with CRM regularizers introduced by [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF] which would change their optimization procedure. Eventually, we note that Krause and Ong [START_REF] Krause | Contextual gaussian process bandit optimization[END_REF] use similar kernels to ours for jointly modeling contexts and actions, but in the different setting of sequential decision making with upper confidence bound strategies. (iii) While Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] and Demirer et al. [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF] focus on policy estimation, our work introduces a new continuous-action data representation and encompasses optimization: in particular, we propose a new contextual policy parameterization, which leads to significant gains compared to baselines parametrized policies on the problems we consider, as well as further improvements related to the optimization strategy. We also note that, apart from [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF] that uses an internal offline cross-validation for model selection, previous works did not perform offline model selection nor evaluation protocols, which are crucial for deploying methods on real data. We provide a brief summary in Table 1 to summarize the key differences with our work. [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] and the application of [START_REF] Chen | Personalized dose finding using outcome weighted learning[END_REF] to large-scale data, see main text.

Optimization methods for learning stochastic policies have been mainly studied in the context of reinforcement learning through the policy gradient theorem [START_REF] Ahmed | Understanding the impact of entropy on policy optimization[END_REF][START_REF] Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF][START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF]. Such methods typically need to observe samples from the new policy at each optimization step, which is not possible in our setting. Other methods leverage a form of off-policy estimates during optimization [START_REF] Kakade | Approximately optimal approximate reinforcement learning[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF], but these approaches still require to deploy learned policies at each step, while we consider objective functions involving only a fixed dataset of collected data. In the context of CRM, Su et al. [START_REF] Su | Cab: Continuous adaptive blending for policy evaluation and learning[END_REF] introduce an estimator with a continuous clipping objective that achieves an improved bias-variance trade-off over the doubly-robust strategy. Nevertheless, this estimator is non-smooth, unlike our soft-clipping estimator.

Modeling of Continous Action Policies

We now review the CRM framework, and then present our modelling approach for policies with continuous actions.

Background

For a stochastic policy π over a set of actions A, a contextual bandit environment generates i.i.d. context features x ∼ PX in X for some probability distribution PX , actions a ∼ π(•|x) and costs y ∼ PY (•|x, a) for some conditional probability distribution PY . We denote the resulting distribution over triplets (x, a, y) as Pπ.

Then, we consider a logged dataset (xi, ai, yi)i=1,...,n, where we assume (xi, ai, yi) ∼ Pπ 0 i.i.d. for a given stochastic logging policy π0, and we assume the propensities π0,i := π0(ai|xi) to be known. In the continuous action case, we assume that the probability distributions admit densities and thus propensities denote the density function of π0 evaluated on the actions given a context. The expected loss or risk of a policy π is then given by

L(π) = E (x,a,y)∼Pπ [y] . (1)
For the logged bandit, the task is to determine a policy π in a set of stochastic policies Π with small risk. Note that this definition may also include deterministic policies by allowing Dirac measures, unless Π includes a specific constraint e.g., minimum variance, which may be desirable in order to gather data for future offline experiments.

In our setting, the expectation in (1) cannot be computed directly for any π, as the available interaction data comes from a different distribution Pπ 0 . Yet, multiple empirical estimators of the risk hereafter allow to derive an empirical optimal policy that is found by solving

π ∈ arg min π∈Π L(π) + Ω(π), (2)
where the objective consists of an empirical estimate L of the risk and of possible data-dependent regularizers, denoted by Ω. When using counterfactual estimators for L, this method has been called counterfactual risk minimization [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF].

The counterfactual approach tackles the distribution mismatch between the logging policy π0(•|x) and a policy π in Π via importance sampling. The IPS method [START_REF] Horvitz | A generalization of sampling without replacement from a finite universe[END_REF] relies on correcting the distribution mismatch using the well-known relation

L(π) = E (x,a,y)∼Pπ 0 y π(a|x) π0(a|x) , (3)
assuming π0 has non-zero mass on the support of π, which allows us to derive an unbiased empirical estimate

LIPS(π) = 1 n n i=1 yi π(ai|xi) π0,i . (4)
Clipped estimator. Since the empirical estimator LIPS(π) may suffer from large variance and is subject to various overfitting phenomena, regularization strategies have been proposed. In particular, this estimator may overfit negative feedback values yi for samples that are unlikely under π0 (see motivation for clipped estimators in Appendix 7), resulting in higher variances. Clipping the importance sampling weights in Eq. ( 5) as Bottou et al. [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF] mitigates this problem, leading to a clipped (cIPS) estimator

LcIPS(π) = 1 n n i=1 yi min π(ai|xi) π0,i , M . (5)
Smaller values of M reduce the variance of L(π) but induce a larger bias. Swaminathan and Joachims [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF] also propose adding an empirical variance penalty term controlled by a factor λ to the empirical risk L(π). Specifically, they write νi(π) = yi min π(a i |x i ) π 0 (a i |x i ) , M and consider the empirical variance for regularization:

V (π) = 1 n -1 n i=1 (νi(π) -ν(π)) 2 , with ν(π) = 1 n n i=1 νi(π),
which is subsequently used to obtain a regularized objective L with hyperparameters M and λ for clipping and variance penalization, respectively:

L(π) = LcIPS(π) + λ V (π). (6)
The self-normalized estimator. Swaminathan and Joachims [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF] also introduce a regularization mechanism for tackling the so-called propensity overfitting issue, occuring with rich policy classes, where the method would focus only on maximizing (resp. minimizing) the sum of ratios π(ai|xi)/π0,i for negative (resp. positive) costs. This effect is corrected through the following self-normalized importance sampling (SNIPS) estimator [24, see also], which is equivariant to additive shifts in cost values:

LSNIPS(π) = n i=1 yiw π i n i=1 w π i , with w π i = π(ai|xi) π0,i . (7)
The direct method DM and the optimal greedy policy. An important quantity is the expected cost given actions and context, denoted by η * (x, a) = E[y|x, a]. If this expected cost was known, an optimal (deterministic) greedy policy π * would simply select actions that minimize the expected cost

π * (x) = arg min a∈A η * (x, a). (8)
It is then tempting to use the available data to learn an estimator η(x, a) of the expected cost, for instance by using ridge regression to fit yi ≈ η(xi, ai) on the training data. Then, we may use the deterministic greedy policy πDM(x) = arg min a η(x, a). This approach, termed direct method (DM), has the benefit of avoiding the high-variance problems of IPS-based methods, but may suffer from large bias since it ignores the potential mismatch between πDM and π0. Specifically, the bias is problematic when the logging policy provides unbalanced data samples (e.g., only samples actions in a specific part of the action space) leading to overfitting [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF][START_REF] Dudik | Doubly robust policy evaluation and learning[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF]. Conversely, counterfactual methods re-balance these generated data samples with importance weights and mitigate the distribution mismatch to better estimate reward function on less explored actions. Nevertheless, such cost estimators can be sometimes effective in practice and may be used to improve IPS estimators in the so-called doubly robust (DR) estimator [START_REF] Dudik | Doubly robust policy evaluation and learning[END_REF] by applying IPS to the residuals yi -η(xi, ai), thus using η as a control variate to decrease the variance of IPS.

While such greedy deterministic policies may be sufficient for exploitation, stochastic policies may be needed in some situations, for instance when one wants to still encourage some exploration in a future round of data logs. Using a stochastic policy also allows us to obtain more accurate off-policy estimates when performing cross-validation on logged data. Then, it may be possible to define a stochastic version of the direct method by adding Gaussian noise with variance σ2 :

πSDM(•|x) = N (πDM(x), σ 2 ), (9)
In the context of offline evaluation on bandit data, such a smoothing procedure may also be seen as a form of kernel smoothing for better estimation [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF].

Our Model for Continuous Actions Policies: the Counterfactual Cost Predictor (CCP)

When considering continuous action spaces, the choice of policies is more involved than in the discrete case. One may indeed naively discretize the action space into buckets and leverage discrete action strategies, but then local information within each bucket gets lost and it is non-trivial to choose an appropriate bucketization of the action space based on logged data, which contains non-discrete actions.

In this paper, we thus focus on stochastic policies belonging to certain classes of continuous distributions, such as normal or log-normal. Specifically, we consider a set of context-dependent policies Π = {π θ , θ ∈ Θ} parameterized by θ = (µ(x), σ), with context-dependent mean µ(x) and standard deviation σ. For example, for normal distributions, π θ (•|x) = N (µ(x), σ 2 ). This class of distributions require learning a parametrized model of the mean µ(x). Before introducing a more flexible model, we will consider the following baselines, given a context x in R d :

• constant: µ(x) = b (context-independent); • linear : µ(x) = x, β + b with β in R d ; • poly: µ(x) = x Bx + b = xx , B + b with B in R d×d .
These baselines require learning the parameters b, β or B by using the CRM approach. Intuitively, the goal is to find a stochastic policy that is close to the optimal deterministic one from Eq. [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF]. While these approaches can be effective in simple problems, they may be limited in more difficult scenarios where the expected cost η * (x, a) has a complex behavior as a function of a. This motivates the need for classes of policies which can better capture such variability by considering a joint model η(x, a) of the cost.

The counterfactual cost predictor (CCP). Assuming that we are given such a model η(x, a), which we call cost predictor, we parametrize the mean of a stochastic policy by using a soft-argmin operator with temperature γ:

µCCP(x) = m i=1 ai exp(-γη(x, ai)) m j=1 exp(-γη(x, aj)) , (10)
where a1, . . . , am are anchor points (e.g., a regular grid or quantiles of the action space), and µCCP may be viewed as a smooth approximation of a greedy policy µ greedy (x) = arg mina η(x, a). The motivation for introducing a soft-argmin operator is to avoid the optimization over actions and to make the resulting CRM problem differentiable. This allows CCP policies to capture complex behavior of η * as a function of a.

Cost predictors based on kernels. The above CCP model is parameterized by a model η(x, a) that may be interpreted as a cost predictor. In a continuous action problem, a reasonable assumption is that costes vary smoothly as a function of actions. Then, a good choice is to take η in a space of smooth functions such as the reproducing kernel Hilbert space H (RKHS) defined by a positive definite kernel [START_REF] Schölkopf | Learning with kernels: support vector machines, regularization, optimization, and beyond[END_REF], so that one may control the smoothness of η through regularization with the RKHS norm. More precisely, we consider kernels of the form

K((x, a), (x , a )) = ψX (x), ψX (x ) e -α contexts x ∈ X actions a ∈ A K((x, a), (x , a )) Joint kernel embedding ψ(x, a) ∈ R p Nyström approximation η(x, a) = β CCP , ψ(x, a)
Cost predictor where, for simplicity, ψX (x) is either a linear embedding ψX (x) = x or a quadratic one ψX (x) = (xx T , x), while actions are compared via a Gaussian kernel, allowing to model complex interactions between contexts and actions.

Nyström method and explicit embedding. Since traditional kernel methods lack scalability, we rely on the classical Nyström approximation [START_REF] Williams | Using the nyström method to speed up kernel machines[END_REF] of the Gaussian kernel, which provides us a finite-dimensional approximate embedding ψA(a) in R m such that e -α 2 a-a 2

≈ ψA(a), ψA(a ) for all actions a, a . This allows us to build a finite-dimensional embedding

ψ(x, a) = ψX (x) ⊗ ψA(a),
where ⊗ denotes the tensorial product, such that

K((x, a), (x , a )) ≈ ψX (x), ψX (x ) ψA(a), ψA(a ) = ψ(x, a), ψ(x , a ) .
More precisely, Nyström's approximation consists of projecting each point from the RKHS to a m-dimensional subspace defined as the span of m anchor points, representing here the mapping to the RKHS of m actions ā1, ā2, . . . , ām. In practice, we may choose āi to be equal to the ai in [START_REF] Fukushima | A generalized proximal point algorithm for certain non-convex minimization problems[END_REF], since in both cases the goal is to choose a set of "representative" actions. For one-dimensional actions (A ⊆ R), it is reasonable to consider a uniform grid, or a non-uniform ones based on quantiles of the empirical distribution of actions in the dataset. In higher dimensions, one may simply use a K-means algorithms and assign anchor points to centroids.

From an implementation point of view, Nyström's approximation considers the embedding ψA(a

) = K -1/2
AA KA(a), where KAA = [KA(āi, āj)]ij and KA(a) = [KA(a, āi)]i and KA is the Gaussian kernel. Then, using the kernel K in (11) yields a model η parametrized by a vector βCCP in η(x, a) = βCCP, ψ(x, a) , and 2 regularization on βCCP corresponds to controlling the RKHS norm and the smoothness of η. The anchor points that we use can be seen as the parameters of an interpolation strategy defining a smooth function, similar to knots in spline interpolation. Naive discretization strategies would prevent us from exploiting such a smoothness assumption on the cost with respect to actions and from exploiting the structure of the action space. Note that Section 5 provides a comparison with naive discretization strategies, showing important benefits of the kernel approach. Our goal was to design a stochastic, computationally tractable, differentiable approximation of the optimal (but unknown) greedy policy [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF]. The model is illustrated in Figure 1.

On Optimization Perspectives for CRM

Because our models yield non-convex CRM problems, we believe that it is crucial to study optimization aspects. Here, we introduce a differentiable clipping strategy for importance weights, and discuss optimization algorithms.

Soft Clipping IPS

The classical hard clipping estimator

LcIPS(π) = 1 n n i=1 yi min {π(ai|xi)/π0,i, M } (12)
makes the objective function non-differentiable, and yields terms in the objective with clipped weights to have zero gradient. In other words, a trivial stationary point of the objective function is that of a stochastic policy that differs enough from the logging policy such that all importance weights are clipped. To alleviate this issue, we propose a differentiable logarithmic soft-clipping strategy. Given a threshold parameter M ≥ 0 and an importance weight wi = π(ai|xi)/π0,i, we consider the soft-clipped weights:

ζ(wi, M ) = wi if wi ≤ M α(M ) log (wi + α(M ) -M ) otherwise, (13)
where α(M ) is such that α(M ) log(α(M )) = M , which yields a differentiable operator. We illustrate the soft clipping expression in Figure 2 and give further explanations about the benefits of clipping strategies in Appendix 7. Then, the IPS estimator with soft clipping becomes

LscIPS(π) = 1 n n i=1 yiζ π(ai|xi) π0,i , M . (14)
We now provide a similar generalization bound to that of Swaminathan and Joachims [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF] (for the hard-clipped version) for the variance-regularized objective of this soft-clipped estimator, justifying its use as a good optimization objective for minimizing the expected risk. Writing χi(π) = yiζ π(a i |x i ) π 0 (a i |x i ) , M , we recall the empirical variance with scIPS that is used for regularization:

V (π) = 1 n -1 n i=1 (χi(π) -χ(π)) 2 , with χ(π) = 1 n n i=1 χi(π).
We assume that costs yi ∈ [-1, 0] almost surely, as in [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF], and make the additional assumption that the importance weights π(ai|xi)/π0(ai|xi) are upper bounded by a constant W almost surely for all π ∈ Π. This is satisfied, for instance, if all policies have a given compact support (e.g., actions are constrained to belong to a given interval) and π0 puts mass everywhere in this support.

L(π) ≤ LscIPS(π) + O   V (π)Cn(Π, δ) n + SCn(Π, δ) n   ,
where S = ζ(W, M ) = O(log W ), V denotes the empirical variance of the cost estimates, and Cn(Π, δ) is a complexity measure of the policy class.

We now prove the Proposition 3.1.

Proof. We begin by proving the result for a finite policy class and for which we use Cn(Π, δ) = log(2|Π|/δ). Let

fi(π) = 1 + yi S ζ π(ai|xi) π0(ai|xi) , M .
We have fi(π) ∈ [0, 1] and

E (x i ,a i ,y i )∼Dπ 0 [fi(π)] = 1 + R M (π)/S, with L M (π) = E (x,a,y)∼Dπ 0 yζ π(a|x) π0(a|x) , M .
We can now apply the concentration bound of Maurer and Pontil [START_REF] Maurer | Empirical bernstein bounds and sample variance penalization[END_REF]Corollary 5] to the fi and rescale appropriately by S to obtain that with probability 1 -δ, for all π ∈ Π,

L M (π) ≤ LscIPS(π) + 2 V (π) log(2|Π|/δ) n + S 7 log(2|Π|/δ) 3(n -1) .
We then use that L(π) ≤ L M (π), since (a) ≤ 0 and ζ(w, M ) ≤ w for all w.

We then conclude the proof by noting that the above result above can be extended to infinite policy classes by essentially replacing |Π| above with an ∞ covering number of the set {(f1(π), . . . , fn(π)) : π ∈ Π}, by leveraging Maurer and Pontil [START_REF] Maurer | Empirical bernstein bounds and sample variance penalization[END_REF]Theorem 6] as in [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF].

Note that while the bound requires importance weights bounded by a constant W , the bound only scales logarithmically with W when W M , compared to a linear dependence for IPS. However we gain significant benefits in terms of optimization by having a smooth objective. Remark 3.1. If costs are in the range [-c, 0], the constant S can be replaced by cS, making the bound homogeneous in the scale (indeed, the variance term is also scaled by c).

Remark 3.2. The scIPS estimator is less biased than the cIPS. We emphasize however that the main motivation for such a clipping strategy is to provide a differentiable estimator which is not the case for cIPS in areas where all point are clipped.

Proximal Point Algorithms

Non-convex CRM objectives have been optimized with classical gradient-based methods [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF] such as L-BFGS [START_REF] Liu | On the limited memory bfgs method for large scale optimization[END_REF], or the stochastic gradient descent approach [START_REF] Joachims | Deep learning with logged bandit feedback[END_REF]. Proximal point methods are classical approaches originally designed for convex optimization [START_REF] Rockafellar | Monotone operators and the proximal point algorithm[END_REF], which were then found to be useful for nonconvex functions [START_REF] Fukushima | A generalized proximal point algorithm for certain non-convex minimization problems[END_REF][START_REF] Paquette | Catalyst for gradient-based nonconvex optimization[END_REF]. In order to minimize a function L, the main idea is to approximately solve a sequence of subproblems that are better conditioned than L, such that the sequence of iterates converges towards a better stationary point of L. More precisely, the proximal point method consists of computing a sequence

θ k ≈ arg min θ L(θ) + κ 2 θ -θ k-1 2 2 , (15)
where L(θ) = L(π θ ) + Ω(π θ ) and κ > 0 is a constant parameter. The regularization term Ω often penalizes the variance [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF], see Appendix 9.2. The role of the quadratic function in [START_REF] Kakade | Approximately optimal approximate reinforcement learning[END_REF] is to make subproblems "less nonconvex" and for many machine learning formulations, it is even possible to obtain convex sub-problems with large enough κ [see 25].

In this paper, we consider such a strategy (15) with a parameter κ, which we set to zero only for the last iteration. Note that the effect of the proximal point algorithm differs from the proximal policy optimization (PPO) strategy used in reinforcement learning [START_REF] Schulman | Proximal policy optimization algorithms[END_REF], even though both approaches are related. PPO encourages a new stochastic policy to be close to a previous one in Kullback-Leibler distance. Whereas the term used in PPO modifies the objective function (and changes the set of stationary points), the proximal point algorithm optimizes and finds a stationary point of the original objective L, even with fixed κ.

The proximal point algorithm (PPA) introduces an additional computational cost as it leads to solving multiple sub-problems instead of a single learning problem. In practice for 10 PPA iterations and with the L-BFGS solver, the computational overhead was about 3× in comparison to L-BFGS without PPA. This overhead seems to be the price to pay to improve the test reward and obtain better local optima, as we show in the experimental section 5.2.2. Nevertheless, we would like to emphasize that computational time is often not critical for the applications we consider, since optimization is performed offline.

On Evaluation and Model Selection for Real World Data

The CRM framework helps finding solutions when online experiments are costly, dangerous or raising ethical concerns. As such it needs a reliable validation and evaluation procedure before rolling-out any solution in the real world. In the continuous action domain, previous work have mainly considered semi-simulated scenarios [START_REF] Bertsimas | Optimization over continuous and multi-dimensional decisions with observational data[END_REF][START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], where contexts are taken from supervised datasets but rewards are synthetically generated. To foster research on practical continuous policy optimization, we release a new large-scale dataset called CoCoA, which to our knowledge is the first to provide logged exploration data from a real-world system with continuous actions. Additionally, we introduce a benchmark protocol for reliably evaluating policies using off-policy evaluation.

The CoCoADataset

The CoCoAdataset comes from the Criteo online advertising platform which ran an experiment involving a randomized, continuous policy for real-time bidding. Data has been properly anonymized so as to not disclose any private information. Each sample represents a bidding opportunity for which a multi-dimensional context x in R d is observed and a continuous action a in R + has been chosen according to a stochastic policy π0 that is logged along with the reward -y (meaning cost y) in R. The reward represents an advertising objective such as sales or visits and is jointly caused by the action and context (a, x). Particular care has been taken to guarantee that each sample (xi, ai, π0(ai|xi), yi) is independent. The goal is to learn a contextual, continuous, stochastic policy π(a|x) that generates more reward in expectation than π0, evaluated offline, while keeping some exploration (stochastic part). As seen in Table 2, a typical feature of this dataset is the high variance of the cost (V[Y ]), motivating the scale of the dataset N to obtain precise counterfactual estimates. The link to download the dataset is available in the code repository: https://github.com/criteo-research/optimization-continuous-action-crm.

Evaluation Protocol for Logged Data

In order to estimate the test performance of a policy on real-world systems, off-policy evaluation is needed, as we only have access to logged exploration data. Yet, this involves in practice a number of choices and difficulties, the most documented being i) potentially infinite variance of IPS estimators [START_REF] Bottou | Counterfactual reasoning and learning systems: The example of computational advertising[END_REF] and ii) propensity over-fitting [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF][START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF]. The former implies that it can be difficult to accurately assess the performance of new policies due to large confidence intervals, while the latter may lead to estimates that reflect large importance weights rather than rewards. A proper evaluation protocol should therefore guard against such outcomes. A first, structuring choice is the IPS estimator. While variants of IPS exist to reduce variance, such as clipped IPS, we found Self-Normalized IPS [SNIPS, [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF][START_REF] Lefortier | Large-scale validation of counterfactual learning methods: A test-bed[END_REF][START_REF] Owen | Monte Carlo theory, methods and examples[END_REF][START_REF] Nedelec | A comparative study of counterfactual estimators[END_REF] to be more effective in practice. Indeed, it avoids the choice of a clipping threshold, generally reduces variance and is equivariant with respect to translation of the reward.

A second component is the use of importance sampling diagnostics to prevent propensity over-fitting. Lefortier et al. [START_REF] Lefortier | Large-scale validation of counterfactual learning methods: A test-bed[END_REF] propose to check if the empirical average of importance weights deviates from 1. However, there is no precise guideline based on this quantity to reject estimates. Instead, we recommend to use a diagnostic on the effective sample size n eff = ( n i=1 wi) 2 / n i=1 w 2 i , which measures how many samples are actually usable to perform estimation of the counterfactual estimate; we follow Owen [START_REF] Owen | Monte Carlo theory, methods and examples[END_REF], who recommends to reject the estimate when the relative effective sample size n eff /n is less than 1%. A third choice is a statistical decision procedure to check if L(π) < L(π0). In theory, any statistical test against a null hypothesis H0: L(π) ≥ L(π0) with confidence level 1 -δ can be used.

Finally, we present our protocol in Algorithm 1. Since we cannot evaluate such a protocol on purely offline data, we performed an empirical evaluation on synthetic setups where we could analytically design true positive (L(π) < L(π0)) else Keep H 0 , consider the estimate to be invalid. end and true negative policies. We discuss in Section 5 the concrete parameters of Alg. 1 and their influence on false (non-)discovery rates in practice.

Model Selection with the Offline Protocol In order to make realistic evaluations, hyperparameter selection is always conducted by estimating the loss of a new policy π in a counterfactual manner. This requires using a validation set (or cross-validation) with propensities obtained from the logging policy π0 of the training set. Such estimates are less accurate than online ones, which would require to gather new data obtained from π, which we assume is not feasible in real-world scenarios.

To solve this issue, we have chosen to discard unreliable estimates that do not pass the effective sample size test from Alg. 1. When doing cross-validation, it implies discarding folds that do not pass the test, and averaging estimates computed on the remaining folds. Although this induces a bias in the cross-validation procedure, we have found it to significantly reduce the variance and dramatically improve the quality of model selection when the number of samples is small, especially for the Warfarin dataset in Sec. 5.

Experimental Setup and Evaluation

We first provide an empirical evaluation of our offline protocol and proceed with a presentation of our empirical findings on synthetic datasets that allow evaluation for any test policy, before presenting our results including the CoCoAdataset.

Experimental Validation of the Protocol

In this section, we study the ability of Alg. 1 to accurately decide if a candidate policy π is better than a reference logging policy π0 (condition L(π) ≤ L(π0)) on synthetic data. Here we simulate logging policy π0 being a lognormal distribution of known mean and variance, and an optimal policy π * being a Gaussian distribution. We generate a logged dataset by sampling actions a ∼ π0 and trying to evaluate policies π with costs observed under the logging policy. We compare the costs predicted using IPS and SNIPS offline metrics to the online metric as the setup is synthetic, it is then easy to check that indeed they are better or worse than π0. We compare the IPS and SNIPS estimates along with their level of confidences and the influence of the effective sample size diagnostic. Offline evaluations of policies π illustrated in Figure 3 are estimated from logged data (xi, ai, yi, π0)i=1...n where ai ∼ π0(•|xi) and where the policy risk would be optimal under the oracle policy π * .

While the goal of counterfactual learning is to find a policy π which is as close a possible to the optimal policy π * , based on samples drawn from a logging policy π0, it is in practice hard to assess the statistical significance of a policy that is too "far" from the logging policy. Offline importance sampling estimates are indeed limited when the distribution mismatch between the evaluated policy and the logging policy (in terms of KL divergence DKL(π0||π)) is large. Therefore we create a setup where we evaluate the quality of offline estimates for policies (i) "close" to the logging policy (meaning the KL divergence DKL(π0||π) is low) and (ii) "close" to the oracle optimal policy (meaning the KL divergence DKL(π * ||π) is low). In this experiment, we focus on evaluating the ability of the offline protocol to correctly assess whether L(π) ≤ L(π0) or not by comparing to online truth estimates. Specifically, for both setups (i) and (ii), we compare the number of False Positives (FP) and False Negatives (FN) of the two offline protocols for N = 2000 initializations, by adding Gaussian noise to the parameters of the closed form policies. False negatives are generated when the offline protocol keeps H0 : L(π) ≥ L(π0) while the online evaluation reveals that L(π) ≤ L(π0), while false positives are generated in the opposite case when the protocol rejects H0 while it is true. We also show histograms of the differences between online and offline boundary decisions for (L(π) < L(π0)), using bootstrapped distribution of SNIPS estimates to build confidence intervals.

Validation of the use of SNIPS estimates for the offline protocol. To assess the performance of our evaluation protocol, we first compare the use of IPS and SNIPS estimates for the offline evaluation protocol and discard solutions with low importance sampling diagnostics n eff n < ν with the recommended value ν = 0.01 from [START_REF] Owen | Monte Carlo theory, methods and examples[END_REF]. In Table 3, we provide an analysis of false positives and false negatives in both setups. We first observe that for setup (i) the SNIPS estimates has both fewer false positives and false negatives. Note that is setup is probably more realistic for real-world applications where we want to ensure incremental gains over the logging policy. In setup (ii) where importance sampling is more likely to fail when the evaluated policy is too "far" from the logging policy, we observe that the SNIPS estimate has a drastically lower number of false negatives than the IPS estimate, though it slightly has more false positives, thus illustrating how conservative this estimator is.

Table 3: Comparison of false positives and false negatives: Perturbation to the logging policy π 0 (setup (i)) and perturbation to the optimal policy (setup (ii)). The SNIPS estimator yields less FN and FP on setup (i), while being more effective on setup (ii) as well by inducing a drastically lower FP rate than IPS and a low FN rate. The effective sample size threshold is fixed at ν = 0.01 We then provide in Fig. 4 histograms of the differences of the upper boundary decisions between online estimates and bootstrapped offline estimates over all samples for both setups (i, left) and (ii, right). Both histograms illustrate how the IPS estimate underestimates the value of the reward with regard to the online estimate, unlike the SNIPS estimates. In the setup (ii) in particular, the IPS estimate underestimates severely the reward, which may explain why IPS has lower number of false positives when the evaluated policy is far from the logging policy. However in both setups, IPS has a higher number of false negatives. We also observed that our SNIPS estimates were highly correlated to the true (online) reward (average correlation ρ = .968, 30% higher than IPS, see plots in Appendix 8.1) for the synthetic setups presented in section 5.2.1, which therefore confirms our findings. 5 how it influences the performance of the offline evaluation protocol for the two previously discussed setups where we consider evaluations of (i) perturbations of the logging policy (left) and (ii) perturbations of the optimal policy (right) in our synthetic setup. We compute precision, recall and F1 scores for each threshold values between 0 and 1. One can see that for low threshold values where no policies are filtered, precision, recall and F1 scores remain unchanged. Once the ESS raises above a certain threshold, undesirable policies start being filtered but more false negatives are created when the ESS is too high. Overall, ESS criterion is relevant for both setups. However, we observe that on simple synthetic setups the effective sample size criterion ν = n eff /n is seldom necessary for policies close to the logging policy (π ≈ π0). Conversely, for policies which are not close to the logging policy the standard statistical significance testing at 1 -δ level was by itself not enough to guarantee a low false discovery rate (FDR) which justified the use of ν. Adjusting the effective sample size can therefore influence the performance of the protocol (see Appendix

Offline Protocol Setup (i) Setup (ii) IPS SNIPS IPS SNIPS π π 0 Keep H 0 π π 0 Keep H 0 π π 0 Keep H 0 π π 0 Keep H 0 "Truth

Experimental Evaluation of the Continuous Modelling and the Optimization Perspectives

In this section we introduce our empirical settings for evaluation and present our proposed CCP policy parametrization, and the influence of optimization in counterfactural risk minimization problems.

Experimental Setup

We present the synthetic potential prediction setup, a semi-synthetic setup as well as our real-world setup.

Synthetic potential prediction. We introduce simple synthetic environments with the following generative process: an unobserved random group index g in G is drawn, which influences the drawing of a context x and of an unobserved "potential" p in R, according to a joint conditional distribution P X ,P |G . Intuitively, the potential p may be compared to users a priori responsiveness to a treatment. The observed reward -y is then a function of the context x, action a, and potential p. The causal graph corresponding to this process is given in Figure 6. Then, we generate three datasets ("noisymoons, noisycircles, and anisotropic", abbreviated respect. "moons, circles, and GMM" in Table 4 and illustrated in Figure 7, with two-dimensional contexts on 2 or 3 groups and different choices of P X ,P |G . Learned policies should vary with the context to adapt to the underlying potentials.

noisymoons noisycircles anisotropic

The goal is then to find a policy π(a|x) that maximizes reward by adapting to an unobserved potential. For our experiments, potentials are normally distributed conditionally on the group index, p|g ∼ N (µg, σ 2 ). As many real-world applications feature a reward function that increases first with the action up to a peak and finally drops, we have chosen a piecewise linear function peaked at a = p (see Appendix 9.1, Figure 16), that mimics reward over the CoCoAdataset presented in Section 4. In bidding applications, a potential may represent an unknown true value for an advertisement, and the reward is then maximized when the bid (action) matches this value. In medicine, increasing drug dosage may increase treatment effectiveness but if dosage exceeds a threshold, secondary effects may appear and eclipse benefits [START_REF] Barnes | Drug dosage in laboratory animals: a handbook[END_REF].

Semi-synthetic setting with medical data. We follow the setup of Kallus and Zhou [16] using a dataset on dosage of the Warfarin blood thinner drug [START_REF]Estimation of the Warfarin dose with clinical and pharmacogenetic data[END_REF]. The dataset consists of covariates about patients along with a dosage treatment prescription by a medical expert, which is a scalar value and thus makes the setting useful for continuous action modelling. While the dataset is supervised, we simulate a contextual bandit environment by using a hand-crafted reward function that is maximal for actions a that are within 10% of the expert's therapeutic drug dosage, following Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF].

Specifically, the semi-synthetic cost inputs prescriptions from medical experts to obtain y(a, x) = max(|a -t * | -0.1t * , 0), so as to mimic the expert prediction. The logging policy π0 samples actions a ∼ π0 contextually to a patient's body mass index (BMI) score ZBMI = x BMI -µ BMI σ BMI and can be analytically written with i.i.d noise e ∼ N (0, 1),

moments of the therapeutic dose distribution µ * T , σ * T such that a = µ * T + σ * T √ θZBMI + σ * T √ 1 -θ (θ = 0.
5 in the setup of [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF]). The logging probability density function thus is a continuous density of a standard normal distribution over the quantity

a-µ * T +σ * T √ θZ BM I σ * T √ 1-θ .
Evaluation methodology For synthetic datasets, we generate training, validation, and test sets of size 10 000 each. For the CoCoA dataset, we consider a 50%-25%-25% training-validation-test sets. We then run each method with 5 different random intializations such that the initial policy is close to the logging policy. Hyperparameters are selected on a validation set with logged bandit feedback as explained in Algorithm 1. We use an offline SNIPS estimate of the obtained policies, while discarding solutions deemed unsafe with the importance sampling diagnostic.

On the semi-synthetic Warfarin dataset we used a cross-validation procedure to improve model selection due to the low dataset size. For estimating the final test performance and confidence intervals on synthetic and on semi-synthetic datasets, we use an online estimate by leveraging the known reward function and taking a Monte Carlo average with 100 action samples per context: this accounts for the randomness of the policy itself over given fixed samples. For offline estimates we leverage the randomness across samples to build confidence intervals: we use a 100-fold bootstrap and take percentiles of the distribution of rewards. For the CoCoA dataset, we report SNIPS estimates for the test metrics.

Empirical Evaluation

We now evaluate our proposed CCP policy parametrization and the influence of optimization in counterfactural risk minimization problems.

Continuous action space requires more than naive discretization. In Figure 8, we compare our continuous parametrization to discretization strategies that bucketize the action space and consider stochastic discreteaction policies on the resulting buckets, using IPS, DM or DR estimators. On all synthetic datasets, continuous modelings such as CCP perform significantly better than discrete approaches (see also Appendix 10.1), across all choices considered for the number of anchor points/buckets. To achieve a reasonable performance, naive discretization strategies require a much finer grid, and are thus also more computationally costly. The plots also show that our (stochastic) direct method strategy, where we add a minimal amount of noise to the deterministic DM in order to pass the n eff /n > ν validation criterion, has similar benefits in terms of robustness to anchor points, thanks to our proposed Nyström parameterization. Nevertheless, it is overall outperformed by CCP, highlighting a benefit of using counterfactual methods compared to a direct fit to observed rewards.

Counterfactual cost predictor (CCP) provides a competitive parameterization for continuousaction policy learning. We compare our CCP modelling approach to other parameterized modelings (constant, linear and non-linear described in Section 2.2) on our synthetic and semi-synthetic setups described in Section 5.2.1 as well as the CoCoAdataset presented in Section 4.1. We also provide a baseline comparison to Chen et al. [START_REF] Chen | Personalized dose finding using outcome weighted learning[END_REF] using their surrogate loss formulation for continous actions and Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], who propose a counterfactual method using kernel density estimation. Their approach is based on an automatic kernel bandwidth selection procedure which did not perform well on our datasets except Warfarin; instead, we select the best bandwidth on a grid through cross-validation and selecting it through our offline protocol. We experimented using the generic doubly robust method from Demirer et al. [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF] but could not reach satisfactory results using the parameters and feature maps that were used in their empirical section and with the specific closed form estimators for their applications. Nevertheless, by adapting their method with more elaborated models and feature maps, we managed to obtain performances beating the logging policy; these modifications would make promising directions for future research venues.

In Table 4, we show a comparison of test rewards for several estimators. We report here the performances of scIPS and SNIPS strategies. For the Warfarin dataset, following Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], we only consider the linear context parametrization baseline, since the dataset has categorical features and higher-dimensional contexts. All results are obtained by using the offline model selection procedure for optimizing hyper-parameters (see Section 4.2). Test rewards on NoisyMoons dataset with varying numbers of anchor points for our continuous parametrization for CCP and DM, versus naive discretization. Note that few anchor points are sufficient to achieve good results on this dataset; this is not the case for more complicated ones (e.g., Warfarin requires at least 15 anchor points).

Overall, we find our CCP parameterization to improve over all baselines, which highlights the effectiveness of the reward predictor at exploiting the continuous action structure. Note that unlike synthetic setups, it is harder to obtain policies that beat the logging policy with large statistical significance on the CoCoAdataset where the logging policy already makes a satisfactory baseline for real-world deployment. Only CCP and Kallus and Zhou [START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF] are passing the significance test. Soft-clipping improves performance of the counterfactual policy learning. Figure 9 shows the improvements in test reward of our optimization-driven strategies for the soft-clipping estimator for the synthetic datasets (see also Appendix 10.2). The points correspond to different choices of the clipping parameter M , models and initialization, with the rest of the hyper-parameters optimized on the validation set using the offline evaluation protocol. This plot also shows that soft clipping provides benefits over hard clipping, perhaps thanks to a more favorable optimization landscape. Overall, these figures confirm that the optimization perspective is important when considering CRM problems.

Proximal point algorithm (PPA) influences optimization of non-convex CRM objective functions and policy learning performance. We illustrate in Figure 10 the improvements in test reward and in training objective of our optimization-driven strategies with the use of the proximal point algorithm (see also Appendix 10.2). Here, each point compares the test metric for fixed models as well as initialization seeds, while optimizing the remaining hyperparameters on the validation set with the offline evaluation protocol. Figure 10 (left) illustrates the benefits of the proximal point method when optimizing the (non-convex) CRM objective in a wide range of hyperparameter configurations, while Figure 10 (center) shows that in many cases this improves the test reward as well. In our experiments, we have chosen L-BFGS because it was performing best among the solvers we tried (nonlinear conjugate gradient (CG) and Newton) and used 10 PPA iterations. For further information, Figure 10 (right) presents a comparison between CG and L-BFGS for different parameters κ and number of iterations. As for computational time, for 10 PPA iterations, the computational overhead was about 3× in comparison to L-BFGS without PPA. This overhead seems to be the price to pay to improve the test reward and obtain better local optima.

Overall, these figures confirm that the proximal point algorithm improves performance in CRM optimization problems.

Conclusion and Discussion

In this paper, we address the problem of counterfactual learning of stochastic policies on real data with continuous actions. This raises several challenges about different steps of the CRM pipeline such as (i) modelization, (ii) optimization, and (iii) evaluation. First, we propose a new parametrization based on a joint kernel embedding of contexts and actions, showing competitive performance. Second, we underline the importance of optimization in CRM formulations with soft-clipping and proximal point methods. Third, we propose an offline evaluation protocol and a new large-scale dataset, which, to the best of our knowledge, is the first with real-world logged propensities and continuous actions. For future research directions, we would like to discuss the doubly robust estimator (which achieves the best results in the discrete action case) with the CCP parametrization of stochastic policies with continuous actions, as well as further optimization perspectives and the offline model selection.

Doubly-robust estimators for continuous action policies While Demirer et al. [START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF] provide a doubly robust (DR) estimator on continuous action using a semi-parametric model of the policy value function, we did not propose a doubly-robust estimator along with our CCP modelling. Indeed, their policy learning is performed in two stages (i) estimate a doubly robust parameter θ DR (x, a, r) in the semi-parametric model of the value function The doubly robust estimation is performed with respect to the first parameter learned in (i) for the value function, while we follow the CRM setting [START_REF] Swaminathan | Counterfactual risk minimization: Learning from logged bandit feedback[END_REF] and directly derive estimators of the policy value (risk) itself, which would correspond to the phase (ii). To derive an estimate a DR estimator of such policy values, we tried extending the standard DR approach for discrete actions from Dudik et al. [START_REF] Dudik | Doubly robust policy evaluation and learning[END_REF] to continuous actions by using our anchors points, but these worked poorly in practice. Actually, a proper DR method for estimating the expectation of a policy risk likely requires new techniques for dealing with integration over the training policy in the direct method term, which is non-trivial and goes beyond the scope of this work. We hope to be able to do this in the future.

E[y|a, x] = V (a, x) = θ * (x)
On further optimization perspectives and offline model selection As mentioned in Section 3, our use of the proximal point algorithm differs from approaches that enhance policies to stay close to the logging policies which modify the objective function as in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] in reinforcement learning. Another venue for future work would be to investigate distributionnally robust methods that do such modifications of the objective function or add constraints on the distribution being optimized. The policy thereof obtained would thus be closer to the logging policy in the CRM context. Moreover, as we showed in Section 4.2 with importance sampling estimates and diagnostics, the offline decision becomes less statistically significant as the evaluated policy is far from the logging policy. Investigating how the distributionally robust optimization would yield better CRM solutions with regards to the offline evaluation protocol would make an interesting future direction of research.

Correlation of Self-Normalized Importance Sampling with Online Rewards

We show in Figures 12,14,13 comparisons of IPS and SNIPS against an on-policy estimate of the reward for policies obtained from our experiments for linear and non-linear contextual modellings on the synthetic datasets, where policies can be directly evaluated online. Each point represents an experiment for a model and a hyperparameter combination. We measure the R 2 score to assess the quality of the estimation, and find that the SNIPS estimator is indeed more robust and gives a better fit to the on-policy estimate. Note also that overall the IPS estimates illustrate severe variance compared to their SNIPS estimate. While SNIPS indeed reduces the variance of the estimate, the bias it introduces does not deteriorate too much its (positive) correlation with the online evaluation. These figures further justify the choice of the self-normalized estimator SNIPS [START_REF] Swaminathan | The self-normalized estimator for counterfactual learning[END_REF] for offline evaluation and validation to estimate the reward on held-out logged bandit data. The SNIPS estimator is indeed more robust to the reward distribution thanks to its equivariance property to additive shifts and does not require hyperparameter tuning.

Importance Sampling Diagnostics in What-If simulations

Importance sampling estimators rely on weighted observations to address the distribution mismatch for offline evaluation, which may cause large variance of the estimator. Notably, when the evaluated policy differs too much from the logging policy, many importance weights are large and the estimator is inaccurate. We provide here a motivating example to illustrate the effect of importance sampling diagnostics in a simple scenario.

When evaluating with SNIPS, we consider an "effective sample size" quantity given in terms of the importance weights wi = π(ai|xi)/π0(ai|xi) by ne = ( n i=1 wi) 2 / n i=1 w 2 i . When this quantity is much smaller than the sample size n, this indicates that only few of the examples contribute to the estimate, so that the obtained value is likely a poor estimate. Apart from that, we note also that IPS weights have an expectation of 1 when summed over the logging policy distribution (that is E (x,a)∼π 0 [π(a|x)/π0(a|x)] = 1.). Therefore, another sanity check, which is valid for any estimator, is to look for the empirical mean 1/n n i=1 π(ai|xi)/π0,i and compare its deviation to 1. In the example below, we illustrate three diagnostics: (i) the one based on effective sample size described in Section 4; (ii) confidence intervals, and (iii) empirical mean of IPS weights. The three of them coincide and allow us to remove test estimates when the diagnostics fail. Note that in this example, the parameterized distribution that is learned (multivariate Gaussian) is not the same as the parameterized distribution of the logging policy (multivariate Lognormal) which skewness may explain the asymmetry of the plots. This points out another practical problem: even though different parametrization of policies is theoretically possible, the probability density masses overlap is in practice what is most important to ensure successful importance sampling. This observation is of utmost interest for real-life applications where the initialization of a policy to be learned needs to be "close" to the logging policy; otherwise importance sampling may fail from the very first iteration of an optimization in learning problems.

Details on the Experiment Setup and Reproducibility

In this section we give additional details on synthetic and semi-synthetic datasets, we provide details on the evaluation methodology and information for experiment reproducibility.

Synthetic and Semi-Synthetic setups

Synthetic setups As many real-world applications feature a reward function that increases first with the action, then plateaus and finally drops, we have chosen a piecewise linear function as shown in Fig. 16 that mimics reward buckets over the CoCoA dataset presented in Section 4. Semi-synthetic medical setup

Reproducibility

We provide code for reproducibility and all experiments were run on a cpu cluster, each node consisting on 24 CPU cores (2 x Intel(R) Xeon(R) Gold 6146 CPU@ 3.20GHz), with 500Go of RAM.

Policy parametrization.

In our experiments, we consider two forms of parametrizations: (i) a lognormal distribution with θ = (θµ, σ), π (µ,σ) = log N (m, s) with s = log (σ 2 /µ 2 + 1); m = log(µ)-s 2 /2, so that Ea∼π (µ,σ) [a] = µ and Vara∼π (µ,σ) [a] = σ 2 ; (ii) a normal distribution π (µ,σ) = N (µ, σ). In both cases, the mean µ may depend on the context (see Section 3), while the standard deviation σ is a learned constant. We add a positivity constraint for σ and add an entropy regularization term to the objective in order to encourage exploratory policies and avoid degenerate solutions.

Models.

For parametrized distributions, we experimented both with normal and lognormal distributions on all datasets, and different baseline parameterizations including constant, linear and quadratic feature maps. We also performed some of our experiments on low-dimensional datasets with a stratified piece-wise contextual parameterization, which partitions the space by bucketizing each feature by taking K (for e.g K = 4) quantiles, and taking the cross product of these partitions for each feature. However this baseline is not scalable for higher dimensional datasets such as the Warfarin dataset.

Hyperparameters. In Table 5 we show the hyperparameters considered to run the experiments to reproduce all the results. Note that the grid of hyperparameters is larger for synthetic data. For our experiments involving anchor points, we validated the number of anchor points and kernel bandwidths similarly to other hyperparameters.

Additional Results and Additional Evaluation Metrics

In this section we provided additional results on both contextual modeling and optimization driven approaches of CRM.

Continuous vs Discrete strategies in continuous-action space

We provide in Figure 17 additional plots for the continuous vs discrete strategies for the synthetic setups described in Section 5.2.1. Test rewards for CLP and (stochastic) direct method (DM) with Nyström parameterization, versus a discrete approach, with varying numbers of anchor points. We add a minimal amount of noise to the deterministic DM in order to pass the n eff validation criterion.

Optimization Driven Approaches of CRM

In this part we provide additional results on optimization driven approaches of CRM for the Noisycircles, Anisotropic, Warfarin and CoCoAdatasets. Both Noisycircles and Anisotropic datasets in Figure 18 show the improvements in test reward and in training objective of our optimization-driven strategies, namely the soft-clipping estimator and the use of the proximal point algorithm. Overall we see that for most configurations, the proximal point method better optimizes the objective function and provides better test performances, while the soft-clipping estimator performs better than its hardclipping variant, which may be attributed to the better optimization properties. For semi-synthetic Warfarin and real-world CoCoA datasets in Figure 18 we also show the improvements in test reward and in training objective of our optimization-driven strategies. More particularly we demonstrate the effectiveness of proximal point methods on the Warfarin dataset where most proximal configurations perform better than the base algorithm. Moreover, soft-clipping strategies perform better than its hard-clipping variant on real-world dataset with outliers and noises, which demonstrate the effectiveness of this smooth estimator for real-world setups.

Figure 1 :

 1 Figure 1: Illustration of the joint kernel embedding for the counterfactual cost predictor (CCP) and cost estimator.

MFigure 2 :

 2 Figure 2: Different clipping strategies c on the importance weights w. Weights are clipped for M = 3, the hard clipping c(w) = min(w, M ) provides no gradient for w > M , while the soft clipping c(w) = ζ(w, M ) and the unclipped estimators c(w) = w do.

Proposition 3 . 1 (

 31 Generalization bound for RM scIPS ). Assume costs yi in [-1, 0] and importance weights bounded by W . With probability 1 -δ, for all π in Π,

Algorithm 1 :

 1 Evaluation Protocol Input: 1 -δ: confidence of statistical test (def: 0.95); ν: a max deviance ratio for effective sample size (def: 0.01); Output: counter-factual estimation of L(π) and decision to reject the null hypothesis H 0 : L(π) ≥ L(π 0 ). 1. Split dataset D → D train , D valid , D test 2. Train new policy π on D train and tune hyper-parameters on D valid (can be replaced by internal cross-validation) 3. Estimate effective sample size n eff (π) on D valid if n eff n > ν then Estimate LSNIPS (π) on D test and test LSNIPS (π) < L(π 0 ) on D test with confidence 1 -δ.

Figure 3 :

 3 Figure3: Illustration of policies: logging policy π 0 , optimal π * and example policy π.

Figure 4 :

 4 Figure 4: Histogram of differences between online reward and offline lower confidence bound. Perturbation to the logging policy π 0 (left), perturbation to the optimal policy π * (right). Effective sample size threshold ν = 0.01

Figure 5 :

 5 Figure 5: Precision, recall and F1 score varying with the ESS threshold on synthetic setups (i) and (ii). Setup (i) perturbation of the logging policy (left) and setup (ii) perturbation to the optimal policy (right). The ESS threshold can maximize the F1 score.

Figure 6 :

 6 Figure 6: Causal Graph of the synthetic setting. A denotes action, X context, G unobserved group label, Y outcome and P unobserved potentials. Unobserved elements are dotted.

Figure 7 :

 7 Figure 7: Contexts (points in R 2 ), and potentials represented by a color map for the synthetic datasets. Learned policies should vary with the context to adapt to the underlying potentials.

Figure 8 :

 8 Figure8: Continuous vs discretization strategies. Test rewards on NoisyMoons dataset with varying numbers of anchor points for our continuous parametrization for CCP and DM, versus naive discretization. Note that few anchor points are sufficient to achieve good results on this dataset; this is not the case for more complicated ones (e.g., Warfarin requires at least 15 anchor points).

Figure 9 :

 9 Figure 9: Influence of soft-clipping. Relative improvements in the test performance for soft-vs hard-clipping on synthetic datasets. The points correspond to different choices of the clipping parameter, models and initialization.

Figure 10 :

 10 Figure 10: Influence of proximal point optimization. Relative improvements in the training objective w and w/o using the proximal point method (left), relative improvements in the test performance w and w/o using the proximal point method (center) comparison of test performance with different number of proximal steps and (right).

  , φ(a, x) and (ii) learn a policy in the empirical Monte Carlo estimate of the policy value by solving , ai, ri), φ(π(xi), xi) .

Figure 12 :

 12 Figure 12: Correlation between offline and online estimates on Anisotropic synthetic data. Linear (left) and non-linear (right) contextual modellings. Ideal fit would be y = x.

Figure 13 :

 13 Figure 13: Correlation between offline and online estimates on NoisyCircles synthetic data. Linear (left) and non-linear (right) contextual modellings. Ideal fit would be y = x.

Figure 14 :

 14 Figure 14: Correlation between offline and online estimates on NoisyMoons synthetic data. Linear (left) and non-linear (right) contextual modellings. Ideal fit would be y = x.

Example 8 . 1 .Figure 15 :

 8115 Figure 15: Importance sampling diagnostics. Ideal importance sampling: i) effective sample n e /n close to 1, ii) low confidence intervals (C.I.s) for m, iii) empirical mean 1 n i w i close to 1. Note that when µ differs too much from µ π0 , importance sampling fails.

Figure 16 :

 16 Figure 16: Synthetic reward engineering. The synthetic reward (left) is inspired from real-dataset reward buckets (right).

Figure 17 :

 17 Figure17: Continuous vs discrete. Test rewards for CLP and (stochastic) direct method (DM) with Nyström parameterization, versus a discrete approach, with varying numbers of anchor points. We add a minimal amount of noise to the deterministic DM in order to pass the n eff validation criterion.

Figure 18 :

 18 Figure 18: Optimization-driven approaches (NoisyCircles, Anisotropic, Warfarin and CoCoA datasets).Relative improvements in the training objective from using the proximal point method (left), comparison of test rewards for proximal point vs the simpler gradient-based method (center), and for soft-vs hard-clipping (right). See alsoFigures 9,[START_REF] Fukushima | A generalized proximal point algorithm for certain non-convex minimization problems[END_REF] for the NoisyMoons dataset.

Table 1 :

 1

	Method	Stochastic	Policy Parametrization	Kernels	CRM Regularizers	Offline evaluation protocol	Large-scale
	Chen et al. [7]		Linear	Embedding of contexts			\
	Kallus and Zhou [16]	\	Linear	Kernel Density Estimation
	Demirer et al. [8]		Any	Not used
	Ours		CCP	Joint embedding of contexts/actions

Comparison to

[START_REF] Chen | Personalized dose finding using outcome weighted learning[END_REF][START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF][START_REF] Demirer | Semi-parametric efficient policy learning with continuous actions[END_REF]

, CCP refers to our continuous action model, see section 2.2. For discussions on stochastic interpretation of

Table 2 :

 2 CoCoAdataset summary statistics.

	N	d E[-Y ] V[Y ] V[A] P(Y = 0)
	120.10 6 3 11.37	9455	.01	.07

  8.2 for further illustrations of importance sampling diagnostics in what-if simulations).

				Effective Sample Size influence on	0				Effective Sample Size influence on	0
		1.0								1.0
	Performance	0.8 0.9							Performance	0.8 0.9
		0.7		Precision						0.7		Precision
				Recall								Recall
				F1								F1
		0.6	0.0	0.1	0.2	0.3 ESS threshold 0.4	0.5	0.6	0.7	0.6	0.0	0.1	0.2	0.3 ESS threshold 0.4	0.5	0.6	0.7

Table 4 :

 4 Test rewards (higher the better) for several estimators (see main text for details). * refers to the original result from Kallus and Zhou[START_REF] Kallus | Policy evaluation and optimization with continuous treatments[END_REF], denoted by KZ.

			Noisycircles	NoisyMoons	Anisotropic	Warfarin	CoCoA
	Logging policy π 0	0.5301	0.5301	0.4533	-13.377	11.34
		Constant	0.6115 ± 0.0000	0.6116 ± 0.0000	0.6026 ± 0.0000	-8.964 ± 0.001	11.36 ± 0.13
	scIPS	Linear Poly	0.6113 ± 0.0001 0.6959 ± 0.0001	0.7326 ± 0.0001 0.7281 ± 0.0001	0.7638 ± 0.0005 -12.857 ± 0.002 0.7448 ± 0.0008 -	11.35 ± 0.02 10.36 ± 0.11
		CCP	0.7674 ± 0.0008	0.7805 ± 0.0004	0.7703 ± 0.0002	-8.720 ± 0.001	11.44 * ± 0.10
		Constant	0.6115 ± 0.0001	0.6115 ± 0.0001	0.5930 ± 0.0001	-9.511 ± 0.001	11.32 ± 0.13
	SNIPS	Linear Poly	0.6115 ± 0.0001 0.6969 ± 0.0001	0.7360 ± 0.0001 0.7370 ± 0.0001	0.7103 ± 0.0003 -10.583 ± 0.005 0.5801 ± 0.0002 -	10.34 ± 0.12 11.13 ± 0.08
		CCP	0.6972 ± 0.0001 0.74091 ± 0.0004 0.7899 ± 0.0002	-9.161 ± 0.001	11.48 * ± 0.14
	Chen et al. [7]	0.608 ± 0.0002	0.645 ± 0.0003	0.754 ± 0.0002	-9.407 ± 0.004	-
	Kallus and Zhou [16]	0.612 ± 0.0001	0.734 ± 0.0001	0.785 ± 0.0002	-10.19 *	11.38 ± 0.07

Table 5 :

 5 Table of hyperparameters for the Synthetic and CoCoA datasets

		Synthetic	Warfarin	CoCoA
	Variance reg. λ	{0., 0.001, 0.01, 0.1, 1, 10, 100}	{0.00010.0010.010.1}	{0., 0.001, 0.1}
	Clipping M	{1, 1.7, 2.8, 4.6, 7.7, 12.9, 21.5, 35.9, 59.9, 100.0}	{1, 2.1, 4.5, 9.5, 20}	{1, 2.1, 4.5, 9.5, 10, 20, 100}
	Prox. κ	{0.001, 0.01, 0.1, 1}	{0.001, 0.01, 0.1}	{0.001, 0.01, 0.1}
	Reg. param. C	{0.00001, 0.0001, 0.001, 0.01, 0.1}	{0.00001, 0.0001, 0.001, 0.01, 0.1} {0.00001, 0.0001, 0.001, 0.01, 0.1}
	Number of anchor points	{2, 3, 5, 7, 10}	{5, 7, 10, 12, 15, 20}	{2, 3, 5}
	Softmax γ	{1, 10, 100}	{1, 5, 10}	{0.1, 0.5, 1, 5}

The code with open-source implementations for experimental reproducibility is available at https://github.com/ criteo-research/optimization-continuous-action-crm.

a-a 2 ,(11)

Acknowledgments

The authors thank Christophe Renaudin (Criteo AI Lab) for his help in the data collection and engineering which was necessary for this project. JM was supported by the ERC grant number 714381 (SOLARIS project) and by ANR 3IA MIAI@Grenoble Alpes, (ANR-19-P3IA-0003). AB acknowledges support from the ERC grant number 724063 (SEQUOIA project).

Appendix

This appendix is organized as follows: in Appendix 7, we present discussions and toy experiments to motivate the need for clipping strategies on real datasets. Next, we provide motivations in Appendix 8 for the offline evaluation protocol with experiments justifying the need for appropriate diagnostics and statistical testing for importance sampling. Then, Appendix 9 is devoted to experimental details that were omitted from the main paper for space limitation reasons, and which are important for reproducing our results (see also the code provided with the submission). In Appendix 10, we present additional experimental results to those in the main paper.

Motivation for Clipped Estimators

In this section we provide a motivation example for clipping strategies in counterfactual systems in a toy example.

In Figure 11 we provide an example of unbounded variance and loss overfitting problem. We recall the data generation: a hidden group label g in G is drawn, and influences the associated context distribution x and of an unobserved potential p in R, according to a joint conditional distribution P X,P |G The observed reward r is then a function of the context x, action a, and potential p. Here, we design one outlier (big red dark dot on Figure 2 left). This point has a noisy reward r, higher than neighbors, and a potential p high as its neighbors have a low potential. We artificially added a noise in the reward function f that can be written as:

As explained in Section 5.2.1, the reward function is a linear function, with its maximum localized at the point x = p(x), i.e. at the potential sampled. The observability of the potential p is only through this reward function f . Hereafter, we compare the optimal policy computed, using different type of estimators.

The task is to predict the high potentials (red circles) and low potentials (blue circles) in the ground truth data (left). Unfortunately, a rare event sample with high potential is put in the low potential cluster (big dark red dot). The action taken by the logging policy is low while the reward is high: this sample is an outlier because it has a high reward while being a high potential that has been predicted with a low action. The resulting unclipped estimator is biased and overfits this high reward/low propensity sample. The rewards of the points around this outlier are low as the diameter of the points in the middle figure show. Inversely, clipped estimator with soft-clipping succeeds to learn the potential distributions, does not overfit the outlier, and has larger rewards than the clipping policies as the diameter of the points show.

Motivation for Offline Evaluation Protocol

In this part we demonstrate the offline/online correlation of the estimator we use for real-world systems and for validation of our methods even in synthetic and semi-synthetic setups. We provide further explanations of the necessity of importance sampling diagnostics and we perform experiments to empirically assess the rate of false discoveries of our protocol.