Yannis Flet-Berliac
email: yannis.flet-berliac@inria.fr

Reda Ouhamma
email: reda.ouhamma@inria.fr

Odalric-Ambrym Maillard

Philippe Preux

LEARNING VALUE FUNCTIONS IN DEEP POLICY GRA-DIENTS USING RESIDUAL VARIANCE

Policy gradient algorithms have proven to be successful in diverse decision making and control tasks. However, these methods suffer from high sample complexity and instability issues. In this paper, we address these challenges by providing a different approach for training the critic in the actor-critic framework. Our work builds on recent studies indicating that traditional actor-critic algorithms do not succeed in fitting the true value function, calling for the need to identify a better objective for the critic. In our method, the critic uses a new state-value (resp. state-actionvalue) function approximation that learns the value of the states (resp. state-action pairs) relative to their mean value rather than the absolute value as in conventional actor-critic. We prove the theoretical consistency of the new gradient estimator and observe dramatic empirical improvement across a variety of continuous control tasks and algorithms. Furthermore, we validate our method in tasks with sparse rewards, where we provide experimental evidence and theoretical insights.

INTRODUCTION

Model-free deep reinforcement learning (RL) has been successfully used in a wide range of problem domains, ranging from teaching computers to control robots to playing sophisticated strategy games [START_REF] Silver | Deterministic policy gradient algorithms[END_REF][START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF]Lillicrap et al., 2016;[START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. Stateof-the-art policy gradient algorithms currently combine ingenious learning schemes with neural networks as function approximators in the so-called actor-critic framework [START_REF] Richard S Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF][START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. While such methods demonstrate great performance in continuous control tasks, several discrepancies persist between what motivates the conceptual framework of these algorithms and what is implemented in practice to obtain maximum gains.

For instance, research aimed at improving the learning of value functions often restricts the class of function approximators through different assumptions, then propose a critic formulation that allows for a more stable policy gradient. However, new studies [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF][START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF] indicate that state-of-the-art policy gradient methods [START_REF] Schulman | Trust region policy optimization[END_REF]2017) fail to fit the true value function and that recently proposed state-action-dependent baselines [START_REF] Gu | Continuous deep q-learning with model-based acceleration[END_REF][START_REF] Liu | Action-dependent control variates for policy optimization via stein identity[END_REF][START_REF] Wu | Variance reduction for policy gradient with action-dependent factorized baselines[END_REF] do not reduce gradient variance more than state-dependent ones.

These findings leave the reader skeptical about actor-critic algorithms, suggesting that recent research tends to improve performance by introducing a bias rather than stabilizing the learning. Consequently, attempting to find a better baseline is questionable, as critics would typically fail to fit it [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF]. In [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF], the authors argue that "much larger gains could be achieved by instead improving the accuracy of the value function". Following this line of thought, we are interested in ways to better approximate the value function. One approach addressing this issue is to put more focus on relative state-action values, an idea introduced in the literature on advantage reinforcement learning (Harmon & Baird III) followed by works on dueling [START_REF] Wang | Dueling network architectures for deep reinforcement learning[END_REF] neural networks. More recent work [START_REF] Lin | Ranking policy gradient[END_REF]) also suggests that considering the relative action values, or more precisely the ranking of actions in a state leads to better policies. The main argument behind this intuition is that it suffices to identify the optimal actions to solve a task. We extend this principle of relative action value with respect to the mean value to cover both state and state-action-value functions with a new objective for the critic: minimizing the variance of residual errors.

In essence, this modified loss function puts more focus on the values of states (resp. state-actions) relative to their mean value rather than their absolute values, with the intuition that solving a task corresponds to identifying the optimal action(s) rather than estimating the exact value of each state. In summary, this paper:

• Introduces Actor with Variance Estimated Critic (AVEC), an actor-critic method providing a new training objective for the critic based on the residual variance.

• Provides evidence for the improvement of the value function approximation as well as theoretical consistency of the modified gradient estimator.

• Demonstrates experimentally that AVEC, when coupled with state-of-the-art policy gradient algorithms, yields a significant performance boost on a set of challenging tasks, including environments with sparse rewards.

• Provides empirical evidence supporting a better fit of the true value function and a substantial stabilization of the gradient.

RELATED WORK

Our approach builds on three lines of research, of which we give a quick overview: policy gradient algorithms, regularization in policy gradient methods, and exploration in RL.

Policy gradient methods use stochastic gradient ascent to compute a policy gradient estimator. This was originally formulated as the REINFORCE algorithm [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF]. [START_REF] Kakade | Approximately optimal approximate reinforcement learning[END_REF] later created conservative policy iteration and provided lower bounds for the minimum objective improvement. [START_REF] Peters | Relative entropy policy search[END_REF] replaced regularization by a trust region constraint to stabilize training. In addition, extensive research investigated methods to improve the stability of gradient updates, and although it is possible to obtain an unbiased estimate of the policy gradient from empirical trajectories, the corresponding variance can be extremely high. To improve stability, [START_REF] Weaver | The optimal reward baseline for gradient•based reinforcement learning[END_REF] show that subtracting a baseline [START_REF] Williams | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF] from the value function in the policy gradient can be very beneficial in reducing variance without damaging the bias. However, in practice, these modifications on the actor-critic framework usually result in improved performance without a significant variance reduction [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF][START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF]. Currently, one of the most dominant on-policy methods are proximal policy optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and trust region policy optimization (TRPO) [START_REF] Schulman | Trust region policy optimization[END_REF], both of which require new samples to be collected for each gradient step. Another direction of research that overcomes this limitation is off-policy algorithms, which therefore benefit from all sample transitions; soft actor-critic (SAC) [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] is one such approach achieving state-of-the-art performance.

Several works also investigate regularization effects on the policy gradient [START_REF] Jaderberg | Reinforcement learning with unsupervised auxiliary tasks[END_REF][START_REF] Namkoong | Variance-based regularization with convex objectives[END_REF][START_REF] Kartal | Terminal prediction as an auxiliary task for deep reinforcement learning[END_REF][START_REF] Flet | Merl: Multi-head reinforcement learning[END_REF]2020); it is often used to shift the bias-variance trade-off towards reducing the variance while introducing a small bias. In RL, regularization is often used to encourage exploration and takes the form of an entropy term [START_REF] Ronald | Function optimization using connectionist reinforcement learning algorithms[END_REF][START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. Moreover, while regularization in machine learning generally consists in smoothing over the observation space, in the RL setting, [START_REF] Thodoroff | Temporal regularization for markov decision process[END_REF] show that it is possible to smooth over the temporal dimension as well. Furthermore, [START_REF] Zhao | Regularized policy gradients: direct variance reduction in policy gradient estimation[END_REF] analyze the effects of a regularization using the variance of the policy gradient (the idea is reminiscent of SVRG descent [START_REF] Johnson | Accelerating stochastic gradient descent using predictive variance reduction[END_REF]) which proves to provide more consistent policy improvements at the expense of reduced performance. In contrast, as we will see later, AVEC does not change the policy network optimization procedure nor involves any additional computational cost.

Exploration has been studied under different angles in RL, one common strategy is -greedy, where the agent explores with probability by taking a random action. This method, just like entropy regularization, enforces uniform exploration and has achieved recent success in game playing en-vironments [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF][START_REF] Van Hasselt | Deep reinforcement learning with double qlearning[END_REF][START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. On the other hand, for most policy-based RL, exploration is a natural component of any algorithm following a stochastic policy, choosing sub-optimal actions with non-zero probability. Furthermore, policy gradient literature contains exploration methods based on uncertainty estimates of values (Kaelbling, 1993;[START_REF] Tokic | Adaptive ε-greedy exploration in reinforcement learning based on value differences[END_REF], and algorithms which provide intrinsic exploration or curiosity bonus to encourage exploration [START_REF] Schmidhuber | Developmental robotics, optimal artificial curiosity, creativity, music, and the fine arts[END_REF][START_REF] Bellemare | Unifying count-based exploration and intrinsic motivation[END_REF][START_REF] Flet-Berliac | Adversarially guided actor-critic[END_REF].

While existing research may share some motivations with our method, no previous work in RL applies the variance of residual errors as an objective loss function. In the context of linear regression, [START_REF] Brown | On small-sample estimation[END_REF] considers a median-unbiased estimator minimizing the risk with respect to the absolutedeviation loss function [START_REF] Pham | The mean and median absolute deviations[END_REF] (similar in spirit to the variance of residual errors), their motivation is nonetheless different to ours. Indeed, they seek to be robust to outliers whereas, when considering noiseless RL problems, one usually seeks to capture those (sometimes rare) signals corresponding to the rewards.

PRELIMINARIES

BACKGROUND AND NOTATIONS

We consider an infinite-horizon Markov Decision Problem (MDP) with continuous states s ∈ S, continuous actions a ∈ A, transition distribution s t+1 ∼ P(s t , a t ) and reward function r t ∼ R(s t , a t ). Let π θ (a|s) denote a stochastic policy with parameter θ, we restrict policies to being Gaussian distributions. In the following, π and π θ denote the same object. The agent repeatedly interacts with the environment by sampling action a t ∼ π(.|s t ), receives reward r t and transitions to a new state s t+1 . The objective is to maximize the expected sum of discounted rewards:

J(π) E τ ∼π ∞ t=0 γ t r (s t , a t ) , (1)
where γ ∈ [0, 1) is a discount factor [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], and τ = (s 0 , a 0 , r 0 , s 1 , a 1 , r 1 , . . . ) is a trajectory sampled from the environment using policy π. We denote the value of a state s in the MDP framework while following a policy π by V π (s) E τ ∼π [ ∞ t=0 γ t r (s t , a t ) |s 0 = s] and the value of a state-action pair of performing action a in state s and then following policy π by Q π (s, a) E τ ∼π [ ∞ t=0 γ t r (s t , a t ) |s 0 = s, a 0 = a]. Finally, the advantage function which quantifies how an action a is better than the average action in state s is denoted A π (s, a) Q π (s, a) -V π (s).

CRITICS IN DEEP POLICY GRADIENTS

In this section, we consider the case where the value functions are learned using function estimators and then used in an approximation of the gradient. Without loss of generality, we consider the algorithms that approximate the state-value function V . The analysis holds for algorithms that approximate the state-action-value function Q. Let f φ : S → R be an estimator of V π with φ its parameter. f φ is traditionally learned through minimizing the mean squared error (MSE) against V π . At iteration k, the critic minimizes:

L AC = E s f φ (s) -V π θ k (s) 2 , ( 2
)
where the states s are collected under policy π θ k , and V π θ k (s) is an empirical estimate of V (see Section 4.3 for details). Similarly, using f φ : S × A → R instead, one can fit an empirical target Qπ .

METHOD: ACTOR WITH VARIANCE ESTIMATED CRITIC

In this section, we introduce AVEC and discuss its correctness, motivations and implementation.

DEFINING AN ALTERNATIVE CRITIC

Recent work [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF] empirically demonstrates that while the value network succeeds in the supervised learning task of fitting V π (resp. Qπ ), it does not fit V π (resp. Q π ). We address this deficiency in the estimation of the critic by introducing an alternative value network loss. Following empirical evidence indicating that the problem is the approximation error and not the estimator per se, AVEC adopts a loss that can provide a better approximation error, and yields better estimators of the value function (as will be shown in Section 5.3). At update k:

L AVEC = E s (f φ (s) -V π θ k (s)) -E s f φ (s) -V π θ k (s) 2 , (3)
with states s collected using π θ k . Note that the gradient flows in f φ twice using Eq. 3. Then, we define our bias-corrected estimator:

g φ : S → R such that g φ (s) = f φ (s) + E s [ V π θ k (s) -f φ (s)].
Analogously to Eq. 3, we define an alternative critic for the estimation of Q π by replacing V π by Qπ and f φ (s) by f φ (s, a).

Proposition (AVEC Policy Gradient). If f φ : S × A → R satisfies the parameterization assumption [START_REF] Richard S Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] then g φ provides an unbiased policy gradient:

∇ θ J (π θ ) = E (s,a)∼π θ [∇ θ log(π θ (s, a))g φ (s, a)] .
Proof. See Appendix A. This result also holds for the estimation of V π θ with f φ : S → R.

BUILDING MOTIVATION

Here, we present the intuition behind using AVEC for actor-critic algorithms. [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF] and [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF] indicate that the approximation error V π -V π is problematic, suggesting that the variance of the empirical targets V π (s t ) is high. Using L AVEC , our approach reduces the variance term of the MSE (or distance to V π ) but mechanistically also increases the bias. Our intuition is that since the bias is already quite substantial [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF], it may be possible to reduce the variance enough so that even though the bias increases, the total MSE reduces.

State-value function estimation. In this case, optimizing the critic with L AVEC can be interpreted as fitting V π (s) = V π (s) -E s [ V π (s )] using the MSE. We show that the targets V π are better estimations of

V π (s) = V π (s) -E s [V π (s )] than V π are of V π .
To illustrate this, consider T independent random variables (X i ) i∈{1,...,T } . We denote

X i = X i -1 T T j=1 X j and V(X) the variance of X. Then, V (X i ) = V (X i ) -2 T V (X i ) + 1 T 2 T j=1 V (X j ) and V(X i ) < V(X i ) as long as ∀i 1 T T j=1 V(X j ) < 2V(X i ),
or more generally when state-values are not strongly negatively correlated1 and not very discordant. This entails that V π has a more compact span, and is consequently easier to fit. This analysis shows that the variance term of the MSE is reduced compared to traditional actor-critic algorithms, but does not guarantee it counterbalances the bias increase. Nevertheless, in practice, the bias is so high that the difference due to learning with AVEC is only marginal and the total MSE decreases. We empirically demonstrate this claim in Section 5.3. State-action-value function estimation. In this case, Eq. 3 translates into replacing V π (s) by Qπ (s, a) and f φ (s) by f φ (s, a) and the rationale for optimizing the residual variance of the value function instead of the full MSE becomes more straightforward: the practical use of the Q-function is to disentangle the relative values of actions for each state [START_REF] Richard S Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF]. AVEC's effect on relative values is illustrated in a didactic regression with one variable example in Fig. 1 where grey markers are observations and the blue line is our current estimation. Minimizing the MSE, the line is expected to move towards the orange one in order to reduce errors uniformly. Minimizing the residual variance, it is expected to move near the red one. In fact, L AVEC tends to further penalize observations that are far away from the mean, implying that AVEC allows a better recovery of the "shape" of the target near extrema. In particular, we see in the figure that the maximum and minimum observation values are quickly identified. Would the approximators be linear and the target state-values independent, the two losses become equivalent since ordinary least squares would provide minimum-variance mean-unbiased estimation.

It should be noted that, as in all the works related to ours, we consider noiseless tasks, i.e. the transition matrix is deterministic. As such, there are no outliers and extreme state-action values correspond to learning signals. In this context, high estimation errors indicate where (in the state or action-state space) the training of the value function should be improved.

IMPLEMENTATION

We apply this new formulation to three of the most dominant deep policy gradient methods to study whether it results in a better estimation of the value function. A better estimation of the value function implies better policy improvements. We now describe how AVEC incorporates its residual variance objective into the critics of PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF], TRPO [START_REF] Schulman | Trust region policy optimization[END_REF] and SAC [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. Let B be a batch of transitions. In PPO and TRPO, AVEC modifies the learning of V φ (line 12 of Algorithm 1) using:

L 1 AVEC (φ) = E s∼B (f φ (s) -V π (s)) -E s∼B f φ (s) -V π (s) 2 , then V φ = f φ (s) + E s∼B [ V π (s) -f φ (s)], where V π (s t ) = f φold (s t ) + A t such that f φold (s t )
are the estimates given by the last value function and A t is the advantage of the policy, i.e. the returns minus the expected values (A t is often estimated using generalized advantage estimation [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF]. In SAC, AVEC modifies the objective function of (Q φi ) i=1,2 (line 13 of Algorithm 2 in Appendix C) using:

L 2 AVEC (φ i ) = E (s,a)∼B (f φi (s, a) -Qπ (s, a)) -E (s,a)∼B f φi (s, a) -Qπ (s, a) 2 , then Q φi = f φi (s, a) + E (s,a)∼B [ Qπ (s, a) -f φi (s, a)]
, where Qπ (s, a) is estimated using temporal difference (see [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]): Qπ (s t , a t ) = r(s t , a t ) + γE st+1∼π [V ψ (s t+1 )] with ψ the value function parameter (see Algorithm 2). The reader may have noticed that L 1 AVEC and L 2 AVEC slightly differ from Eq. 3. The residual variance of the value function (L AVEC ) is not tractable since a priori state-values are dependent and their joint law is unknown. Consequently, in practice, we use the empirical variance proxy assuming independence (cf. Appendix D). [START_REF] Greensmith | Variance reduction techniques for gradient estimates in reinforcement learning[END_REF] provide some support for this approximation by showing that weakly dependent variables tend to concentrate more than independent ones. Finally, notice that AVEC does not modify any other part of the considered algorithms whatsoever, which makes its implementation straightforward and keeps the same computational complexity.

EXPERIMENTAL STUDY

In this section, we conduct experiments along four orthogonal directions. We point out that a comparison to variance-reduction methods is not considered in this paper: [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF] demonstrated that their implementations diverge from the unbiased methods presented in the respective papers and unveiled that not only do they fail to reduce the variance of the gradient, but that their unbiased versions do not improve performance either. Note that in all experiments we choose the hyperparameters providing the best performance for the considered methods which can only penalyze AVEC (cf. Appendix E). In all the figures hereafter (except Fig. 3c and3d), lines are average performances and shaded areas represent one standard deviation. (.%) is the change in performance due to AVEC.

CONTINUOUS CONTROL

Algorithm 1 AVEC coupled with PPO or TRPO.

J ALGO denotes the policy loss of either algorithm (described in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]2015)).

1: Input parameters: λ π ≥ 0, λ V ≥ 0 2: Initialize policy parameter θ and value function parameter φ 3: for each update step do for each environment step do 6:

a t ∼ π θ (s t ) 7: s t+1 ∼ P (s t , a t ) 8: B ← B ∪ {(s t , a t , r t , s t+1 )} 9:
end for 10:

for each gradient step do 11:

θ ← θ -λ π ∇θ J ALGO (π θ ) 12: φ ← φ -λ V ∇φ L 1 AVEC (φ) 13:
end for 14: end for For ease of comparison with other methods, we evaluate AVEC on the MuJoCo [START_REF] Todorov | Mujoco: A physics engine for model-based control[END_REF] and the PyBullet [START_REF] Coumans | Pybullet, a python module for physics simulation for games, robotics and machine learning[END_REF] continuous control benchmarks (see Appendix G for details) using OpenAI Gym [START_REF] Brockman | Openai gym[END_REF]. Note that the PyBullet versions of the locomotion tasks are harder than the MuJoCo equivalents2 . We choose a representative set of tasks for the experimental evaluation; their action and observation space dimensions are reported in Appendix H. We assess the benefits of AVEC when coupled with the most prominent policy gradient algorithms, currently state-of-the-art methods: PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and TRPO [START_REF] Schulman | Trust region policy optimization[END_REF], both on-policy methods, and SAC [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF], an off-policy maximum entropy deep RL algorithm. We provide the list of hyperparameters and further implementation details in Appendix D and E.

Table 1 reports the results while Fig. 2 and8 show the total average return for SAC and PPO. TRPO results are provided in Appendix F for readability. When coupled with SAC and PPO, AVEC brings very significant improvement (on average +26% for SAC and +39% for PPO) in the performance of the policy gradient algorithms, improvement which is consistent across tasks. As for TRPO, while the improvement in performance is less striking, AVEC still manages to be more efficient in terms of sampling in all tasks. Overall, AVEC improves TRPO, PPO and SAC in terms of performance and efficiency. This does not imply that our method would also improve other policy gradient methods that use the traditional actor-critic framework, but since we evaluate our method coupled with three of the best performing on-and off-policy algorithms, we believe that these experiments are sufficient to prove the relevance of AVEC. Furthermore, in our experiments we do not seek the best hyperparameters for the AVEC variants, we simply adopt the parameters allowing us to optimally reproduce the baselines. Alternatively, if one seeks to evaluate AVEC independently of a considered baseline, further hyperparameter tuning should produce better results. Notice that since no additional calculations are needed in AVEC's implementation, computational complexity remains unchanged.

SPARSE REWARD SIGNALS

Domains with sparse rewards are challenging to solve with uniform exploration as agents receive no feedback on their actions before starting to collect rewards. In such conditions AVEC performs better, suggesting that the shape of the value function is better approximated, encouraging exploration. The relative value estimate of an unseen state is more accurate: in Section 4.2, AVEC identifies extreme state-values (e.g., non-zero rewards in tasks with sparse rewards) faster. In Fig. 3a and 3b, we report the performance of AVEC in the Acrobot and MountainCar environments: both have sparse rewards.

AVEC enhances TRPO and PPO in both experiments. When PPO and AVEC-PPO both reach the best possible performance, AVEC-PPO exhibits better sample efficiency. Fig. 3c and 3d illustrate how the agent improves its exploration strategy in MountainCar: while the PPO agent remains stuck at the bottom of the hill (red), the graph suggest that AVEC-PPO learns the difficult locomotion principles in the absence of rewards and visits a much larger part of the state space (green).

This improved performance in sparse environments can be explained by the fact that AVEC is able to pick up on experienced positive reward more easily. Moreover, the reconstructed shape of the value function is more accurate around such rewarding states, which pushes the agent to explore further around experienced states with high values. We observe that PPO better fits the empirical target than when equipped with AVEC, which is to be expected since vanilla PPO optimizes the MSE directly. This result put aside the remarkable improvement in the performance of AVEC-PPO (Fig. 2) suggests that AVEC might be a better estimator of the true value function.

We examine this claim below because if true, it would indicate that it is indeed possible to simultaneously improve the performance of the agents and the stability of the method. Learning the True Target. A fundamental premise of policy gradient methods is that optimizing the objective based on an empirical estimation of the value function leads to a better policy. Which is why we investigate the quality of fit of the true target. To approximate the true value function, we fit the returns sampled from the current policy using a large number of transitions (3 • 10 5 ). Fig. 5 shows that g φ is far closer to the true value function half of the time (horizon is 10 6 ) than the estimator obtained with MSE, then as close to it. Comparing Fig. 5 with Fig. 4, we see that the distance to the true target is close to the estimation error for AVEC-PPO, while for PPO, it is at least two orders of magnitude higher at all times. We further investigate these results in Fig. 9 in Appendix B.2 where we study the variation of the squared bias and variance components of the MSE to the true target (MSE = Var + Bias 2 ). We find, as expected, that using AVEC reduces the variance term significantly while slightly increasing the bias term, which Fig. 5 confirms is negligible since the total MSE is substantially reduced (

g φ (AVEC) -V π 2 ≤ V φ (PPO) -V π 2 )
where V φ (PPO) is the value function estimator in PPO. For completeness, we also analyze the distance to the true target for the Q-function estimator in SAC and AVEC-SAC in AntBullet and HalfCheetahBullet in Appendix B.3, with similar results and interpretation. We conclude that AVEC improves the value function approximation and we expect that the gradient is more stable. Empirical Variance Reduction. We choose to study the gradient variance using the average pairwise cosine similarity metric as it allows a comparison with Ilyas et al. ( 2020), with which we share the same experimental setup and scales. Fig. 6 shows that AVEC yields a higher average (10 batches per iteration) pairwise cosine similarity, which means closer batch-estimates of the gradient and, in turn, indicates smaller gradient variance. Further analysis with additional tasks is included in Appendix B.4. The variance reduction effect observed in several environments suggests that AVEC is the first method since the introduction of the value function baseline to further reduce the variance of the gradient and improve performance.

ABLATION STUDY

In this section, we examine how changing the relative importance of the bias and the residual variance in the loss of the value network affects learning. For this study, we choose difficult tasks of PyBullet and use PPO because it is more efficient than TRPO and requires less computations than SAC. For an estimator ŷn of (y i ) i∈{1,...,n} , we write Bias = 1 n n i=1 (ŷ i -y i ) and Var = 1 n-1 n i=1 (ŷ i -y i -Bias) 2 . Consequently: MSE = Var + Bias 2 . We denote L α = Var + αBias 2 , with α ∈ R. In Fig. 7, Bias-α means that we use L α and Var-α means that we use L 1 α . We observe that while no consistent order on the choices of α is identified, AVEC seems to outperform all other weightings. Note that, for readability purposes, the graphs have been split and the curves of AVEC-PPO and PPO are the same in Fig. 7a and7c, and in Fig. 7b and7d. A more extensive hyperparameter study with more α values might provide even higher performances, nevertheless we believe that the stability of an algorithm is crucial for a reliable performance. As such, the tuning of hyperparameters to achieve good results should remain mild.

DISCUSSION

In this work, we introduce a training objective for the critic in actor-critic algorithms to better approximate the true value function. In addition to being well-motivated by recent studies on the behaviour of deep policy gradient algorithms, we demonstrate that this modification is both theoretically sound and intuitively supported by the need to improve the approximation error of the critic. The application of Actor with Variance Estimated Critic (AVEC) to state-of-the-art policy gradient methods produces considerable gains in performance (on average +26% for SAC and +39% for PPO) over the standard actor-critic training, without any additional hyperparameter tuning.

First, for SAC-like algorithms where the critic learns a state-action-value function, our results strongly suggest that state-actions with extreme values are identified more quickly. Second, for PPO-like methods where the critic learns the state-values, we show that the variance of the gradient is reduced and empirically demonstrate that this is due to a better approximation of the state-values. In sparse reward environments, the theoretical intuition behind a variance estimated critic is more explicit and is also supported by empirical evidence. In addition to corroborating the results in [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF] proving that the value estimator fails to fit V π , we propose a method that succeeds in improving both the sample complexity and the stability of prominent actor-critic algorithms. Furthermore, AVEC benefits from its simplicity of implementation since no further assumptions are required (such as horizon awareness [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF] to remedy the deficiency of existing variance-reduction methods) and the modification of current algorithms represents only a few lines of code.

In this paper, we have demonstrated the benefits of a more thorough analysis of the critic objective in policy gradient methods. Despite our strongly favourable results, we do not claim that the residual variance is the optimal loss for the state-value or the state-action-value functions, and we note that the design of comparably superior estimators for critics in deep policy gradient methods merits further study. In future work, further analysis of the bias-variance trade-off and extension of the results to stochastic environments is anticipated; we consider the problem of noise separation in the latter, as this is the first obstacle to accessing the variance and distinguishing extreme values from outliers.

A UNBIASED AVEC POLICY GRADIENT

In this section, we consider the case in which the state-action-value function of a policy π θ is approximated. We prove that given some assumptions on this estimator function, we can use it to yield a valid gradient direction, i.e., we are able to prove policy improvement when following this direction.

In this setting, the critic minimizes the following loss:

E (s,a)∼π ( Qπ θ (s, a) -f φ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f φ (s, a)]) 2 .
When a local optimum is reached, the gradient of the latter expression is zero:

∇ φ L AVEC = E (s,a)∼π ( Qπ θ (s, a) -f φ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f φ (s, a)])( ∂f φ (s, a) ∂φ -E (s,a)∼π [ ∂f φ (s, a) ∂φ ]) = 0.
In the expression above, the expected value of the partial derivative disappears because the term in the first bracket is centered:

E (s,a)∼π ( Qπ θ (s, a) -f φ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f φ (s, a)])E (s,a)∼π [ ∂f φ (s, a) ∂φ ] = E (s,a)∼π ∂f φ (s, a) ∂φ : = 0 E (s,a)∼π [ Qπ θ (s, a) -f φ (s, a) -E (s,a)∼π [ Qπ θ -f φ ]] = 0.
Simplifying the gradient at the local optimum becomes:

E (s,a)∼π ( Qπ θ (s, a) -f φ (s, a) -E (s,a)∼π [ Qπ θ (s, a) -f φ (s, a)])( ∂f φ (s, a) ∂φ ) = 0. (4)
Then, if we denote g φ = f φ (s, a)+E (s,a)∼π [ Qπ (s, a)-f φ (s, a)], and use the policy parameterization assumption: ∂f φ (s, a) ∂φ = ∂π θ (s, a) ∂θ

1 π θ (s, a) , (5)
we obtain:

∇ θ J = E (s,a)∼π θ [∇ θ log(π θ (s, a))g φ (s, a)] . (6)
Proof. By combining the parameterization assumption in Eq. 5 with Eq. 4, we have:

E (s,a)∼π θ ( Qπ θ (s, a) -g φ (s, a)) ∂π θ (s, a) ∂θ 1 π θ (s, a) = 0. (7)
Since the expression above is null, we have the following:

∇ θ J = E (s,a)∼π θ [∇ θ log(π θ (s, a)) Qπ θ (s, a)] = E (s,a)∼π θ [∇ θ log(π θ (s, a)) Qπ θ (s, a)] -E (s,a)∼π θ [( Qπ θ (s, a) -g φ (s, a)) ∂π θ (s, a) ∂θ 1 π θ (s, a) ] = E (s,a)∼π θ [∇ θ log(π θ (s, a))g φ (s, a)].
Remark. While the proof seems more or less generic, the assumption in Eq. 5 is extremely constraining to the possible approximators. [START_REF] Richard S Sutton | Policy gradient methods for reinforcement learning with function approximation[END_REF] quotes J. Tsitsiklis who believes that a linear g φ in the features of the policy may be the only feasible solution for this condition. Concretely, such an assumption cannot hold since neural networks are the standard approximators used in practice. Moreover, empirical analysis [START_REF] Mance | Multi-player residual advantage learning with general function approximation[END_REF] indicates that commonly used algorithms fail to fit the true value function. However, this does not rule out the usefulness of the approach but rather begs for more questioning of the true effect of such biased baselines.

B ADDITIONAL EXPERIMENTS

B.1 CONTINUOUS CONTROL: WALKER2D Fig. 8 shows the total average return for AVEC coupled with SAC and PPO on the Walker2d task. Similar to considered other continuous control tasks from MuJoCo and PyBullet, AVEC brings a significant performance improvement (+26% for SAC and +33% for PPO), confirming the generality of our approach.

B.4 VARIANCE REDUCTION

In Fig. 11, we study the empirical variance of the gradient in measuring the average pairwise cosine similarity (10 gradient measurements) in two additional tasks: HopperBullet and Walker2DBullet. We also vary the trajectory size used in the estimation of the gradient.

C IMPLEMENTATION OF AVEC COUPLED WITH SAC

In Algorithm 2, J V is the squared residual error objective to train the soft value function. See [START_REF] Haarnoja | Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] for further details and notations about SAC, not directly relevant here.

Algorithm 2 AVEC coupled with SAC.

1: Input parameters: β ∈ [0, 1], λ V ≥ 0, λ Q ≥ 0, λ π ≥ 0 2: Initialize policy parameter θ, value function parameter ψ and ψ and Q-functions parameters φ 1 and φ 2 3: D ← ∅ 4: for each iteration do 5:

for each step do 6: ψ ← ψ -λ V ∇ψ J V (ψ) 13:

a t ∼
φ i ← φ i -λ Q ∇φi L 2
AVEC (φ i ) for i ∈ {1, 2}

14:

θ ← θ -λ π ∇θ J(π θ )

15:

ψ ← βψ + (1 -β) ψ 16:
end for 17: end for

D IMPLEMENTATION DETAILS

Theoretically, L AVEC is defined as the residual variance of the value function (cf. Eq. 3). However, state-values for a non-optimal policy are dependent and the variance is not tractable without access to the joint law of state-values. Consequently, to implement AVEC in practice we use the best-known proxy at hand, which is the empirical variance formula assuming independence:

L AVEC = 1 T -1 T t=1 f φ (s t ) -V π (s t ) - 1 T T t=1 f φ (s t ) -V π (s t ) 2 ,
where T is the size of the sampled trajectory.

F COMPARATIVE EVALUATION OF AVEC WITH TRPO

In order to evaluate the performance gains in using AVEC instead of the usual actor-critic framework, we produce some additional experiments with the TRPO [START_REF] Schulman | Trust region policy optimization[END_REF] algorithm. Fig. 12 shows the learning curves while Table 5 reports the results.

Figure 1 :

 1 Figure 1: Comparison of simple models derived when L AVEC is used instead of the MSE.

  (a) We validate the superiority of AVEC compared to the traditional actor-critic training. (b) We evaluate AVEC in environments with sparse rewards. (c) We clarify the practical implications of using AVEC by examining the bias in both the empirical and true value function estimations as well as the variance in the empirical gradient. (d) We provide an ablation analysis and study the bias-variance trade-off in the critic by considering two continuous control tasks.

Figure 2 :

 2 Figure2: Comparative evaluation (6 seeds) of AVEC with SAC and PPO on PyBullet ("TaskBullet") and MuJoCo ("Task") tasks. X-axis: number of timesteps. Y-axis: average total reward.

Figure 3 :

 3 Figure 3: (a,b): Comparative evaluation (6 seeds) of AVEC in sparse reward tasks. X-axis: number of timesteps. Y-axis: average total reward. (c,d): Respectively state visitation frequency and phase portrait of visited states of AVEC-TRPO (green) and TRPO (red) in MountainCar.

Figure 4 :

 4 Figure 4: L 2 distance to V π .

Figure 5 :

 5 Figure 5: L 2 distance to V π . X-axis: we run PPO and AVEC-PPO and ∀t ∈ {1, 2, 4, 6, 9} • 10 5 we stop training, use the current policy to collect 3•10 5 transitions and estimate V π .

Figure 6 :

 6 Figure 6: Average gradient cosine-similarity.

Figure 7 :

 7 Figure 7: Sensitivity (6 seeds) of AVEC-PPO with respect to (a,b): the bias; (c,d): the variance. X-axis: number of timesteps. Y-axis: average total reward.

Figure 8 :

 8 Figure 8: Comparative evaluation (6 seeds) of AVEC with SAC (left) and PPO (right) on the Walker2d MuJoCo task. Lines are average performances and shaded areas represent one standard deviation.

Figure 11 :

 11 Figure11: Average cosine similarity between gradient measurements. AVEC empirically reduces the variance compared to PPO or PPO without a baseline (PPO-nobaseline). Trajectory size used in estimation of the gradient variance: 3000 (upper row), 6000 (middle row), 9000 (lower row). Lines are average performances and shaded areas represent one standard deviation.

Figure 12 :

 12 Figure 12: Comparative evaluation of AVEC with TRPO. We run with 6 different seeds: lines are average performances and shaded areas represent one standard deviation.

Table 1 :

 1 Average total reward of the last 100 episodes over 6 runs of 10 6 timesteps. Comparative evaluation of AVEC with SAC and PPO. ± corresponds to a single standard deviation over trials and

	Task	SAC	AVEC-SAC	PPO	AVEC-PPO
	Ant	3084	3650 ± 127 (+18%)	972	1202 ± 148 (+24%)
	AntBullet	1193	2252 ± 82 (+89%)	1174	2216 ± 99 (+89%)
	HalfCheetah	10028 11018 ± 102 (+10%) 1068	1403 ± 37 (+31%)
	HalfCheetahBullet	1255	1331 ± 184 (+6%)	1329	2223 ± 62 (+67%)
	Humanoid	4084	4472 ± 424 (+10%)	391	415 ± 4.6 (+6%)
	Reacher	-6.0	-5.0 ± 0.1 (+20%)	-7.4	-5.9 ± 0.3 (+25%)
	Walker2d	3452	4334 ± 128 (+26%)	2193 2923 ± 151 (+33%)

  π θ (a t |s t ) 7:s t+1 ∼ P (s t , a t ) D ← D ∪ {(s t , a t , r t , s t+1 )}

	8:
	9:	end for
	10:	for each gradient step do
	11:	sample batch B from D
	12:

Table 5 :

 5 Average total reward of the last 100 episodes over 6 runs of 10 6 timesteps. Comparative evaluation of AVEC with TRPO. ± corresponds to a single standard deviation over trials and (.%) is the change in performance due to AVEC.

	Task	TRPO	AVEC-TRPO
	Ant	-50.5	-43.5 ± 2.2 (+16%)
	AntBullet	564	970 ± 70 (+72%)
	HCheetah	346	466 ± 56 (+35%)
	HCBullet	1154	1281 ± 94 (+11%)
	Humanoid	352	344 ± 1.2 (-3%)
	Reacher	-8.5	-9.9 ± 1.3 (-16%)

[START_REF] Greensmith | Variance reduction techniques for gradient estimates in reinforcement learning[END_REF] analyze the dependent case: in general, weakly dependent variables tend to concentrate more than independent ones.

Bullet Physics SDK GitHub Issue.

B.2 VARIATION OF THE BIAS AND VARIANCE TERMS: PPO

In Fig. 9, we show the variation of the bias and variance terms in the MSE between the estimators (of AVEC-PPO and PPO) and the true target: E[ g φ -V π 2 2 ] = Bias(AVEC) 2 + Var(AVEC) and E[ V φ (PPO) -V π 2 2 ] = Bias(PPO) 2 + Var(PPO) where V φ (PPO) is the value function estimator in PPO. We observe that the variance reduction is more substantial than that of the bias. Using those results and Fig. 5 showing that the distance of the estimator to V π is lower when using AVEC confirms that the variance reduction effect counterbalances the bias increase. Note that the % Variation of the Var term is always negative in our experiments, and that the shaded areas that suggest otherwise are merely due to a false assumption of symmetrical deviations, itself due to the assumption of Gaussianity needed to construct confidence intervals. . X-axis: we run PPO and AVEC-PPO and for every t ∈ {1, 2, 4, 6, 9} • 10 5 , we stop training, use the current policy to interact with the environment for 3 • 10 5 transitions, and use these transitions to estimate the true value function. Lines are average variations and shaded areas represent one standard deviation (5 seeds).

B.3 LEARNING THE TRUE TARGET: SAC

In Fig. 10, we compare the error between the Q-function estimator and the true Q-function for SAC and AVEC-SAC in AntBullet and HalfCheetahBullet. We note a modest but consistent reduction in this error when using AVEC coupled with SAC, echoing the significant performance gains in Fig. 2.

E EXPERIMENT DETAILS

In all experiments we choose to use the same hyperparameter values for all tasks as the bestperforming ones reported in the literature or in their respective open source implementation documentation. We thus ensure the best performance for the conventional actor-critic framework. In other words, since we are interested in evaluating the impact of this new critic, everything else is kept as is. This experimental protocol may not benefit AVEC.

In Table 2, 3 and 4, we report the list of hyperparameters common to all continuous control experiments.

Environment Description

Ant-v2 Make a four-legged creature walk forward as fast as possible. AntBulletEnv-v0

Idem. Ant is heavier, encouraging it to typically have two or more legs on the ground (source: Py-Bullet Guide -url).

HalfCheetah-v2

Make a 2D cheetah robot run. HalfCheetahBulletEnv-v0 Idem.

Humanoid-v2

Make a three-dimensional bipedal robot walk forward as fast as possible, without falling over.

Reacher-v2

Make a 2D robot reach to a randomly located target.

Walker2d-v2

Make a 2D robot walk forward as fast as possible.

Acrobot-v1

Swing the end of a two-joint acrobot up to a given height.

MountainCar-v0

Get an under powered car to the top of a hill.

H DIMENSIONS OF STUDIED TASKS


Fabienne Méducin

Bruno Bresson

Nicolas Lequeux

Marie-Noëlle De Noirfontaine

Hélène Zanni

Calcium silicate hydrates investigated by solid-state high resolution 1 H and 29 Si nuclear magnetic resonance

Keywords: Hydration products, Temperature, Scanning Electron Microscopy, Spectroscopy, X-ray diffraction

This work focuses on phases formed during cement hydration under high pressure and temperature: portlandite Ca(OH) 2 (CH); hillebrandite Ca 2 (SiO 3 )(OH) 2 (β-dicalcium silicate hydrate); calcium silicate hydrate (C-S-H); jaffeite Ca 6 (Si 2 O 7 )(OH) 6 (tricalcium silicate hydrate); α-C 2 SH Ca 2 (SiO 3 )(OH) 2 (α-dicalcium silicate hydrate); xonotlite Ca 6 (Si 6 O 17 )(OH) 2 and kilchoanite Ca 6 (SiO 4 )(Si 3 O 10 ). Portlandite and hillebrandite were synthesized and characterised by high resolution solid-state 1 H and 29 Si Nuclear Magnetic Resonance. In addition, information from the literature concerning the last five phases was gathered. In certain cases, a schematic 3D-structure could be determined. These data allow identification of the other phases present in a mixture. Their morphology was also observed by Scanning Electron Microscopy.

Introduction

The study of cement hydration at high temperature and under high pressure (up to 200 °C and 1000 bar) is of great interest in forecasting the durability of oil well cements. During this process, a cement slurry is pumped into the steel casing of the well up the annular space between the casing and the surrounding rock to support and protect it. From the view point of oil and gas companies, the disappearance of the mechanical resistance of cement at high pressure and temperature is critical; it can lead to a loss of integrity of cemented annulus and, consequently, can stop production. It is therefore important to know the structure of the calcium silicate phases formed under such unusual conditions. Cement hydration has been studied widely under conditions of normal temperature and pressure, and several studies have been carried out on model systems at higher temperatures [1 2 3]. The studies at high pressure concern mainly thermodynamic equilibrium between high temperature and high pressure phases in the ternary H 2 O-CaO-SiO 2 system [START_REF] Harker | Dehydration series in the system CaSiO 3 -SiO 2 -H 2 O[END_REF][START_REF] Pistorius | Thermal decomposition of portlandite and xonotlite to high pressures and high temperatures[END_REF]. However, the results are not easily transposed to cement paste hydration since this system may not reach an equilibrium state (the hydrated phases formed are metastable) [START_REF] Taylor | Cement Chemistry[END_REF]. This paper focuses on the structure of calcium silicate hydrates which can be formed under high pressure and temperature: portlandite Ca(OH) 2 (CH), hillebrandite Ca 2 (SiO 3 )(OH) 2 (β-dicalcium silicate hydrate), calcium silicate hydrates (C-S-H), jaffeite Ca 6 (Si 2 O 7 )(OH) 6 (tricalcium silicate hydrate), α-C 2 SH Ca 2 (SiO 3 ) (OH) 2 (α-dicalcium silicate hydrate), xonotlite Ca 6 (Si 6 O 17 ) (OH) 2 , and kilchoanite Ca 6 (SiO 4 )(Si 3 O 10 ). Subsequent text uses the notation of cement chemistry: C = CaO, S = SiO 2 , H=H 2 O. The first two phases (portlandite CH and hillebrandite β-C 2 SH) were synthesized and then characterised by high resolution 1 H and 29 Si NMR. In addition, information contained in the literature concerning the last five phases was gathered, especially regarding crystallography which can be helpful to interpret the NMR spectra. In certain cases, a schematic 3Dstructure could easily be determined. The results obtained allow the resolution of phase mixtures as was shown in three mixtures formed during Ca 3 SiO 5 hydration under high pressure and temperature.

Experimental procedure

Synthesis

Portlandite Ca(OH) 2 (CH) and hillebrandite Ca 2 (SiO 3 )(OH) 2 (β-dicalcium silicate hydrate) were synthesized in the Laboratoire Céramiques et Matériaux Minéraux at the Ecole Supérieure de Physique et Chimie Industrielles de Paris.

Portlandite

Portlandite occurs both as a natural mineral and a synthetic product, obtained by the hydration of lime, CaO, under a nitrogen atmosphere to avoid formation of CaCO 3 , and then drying by acetone-ether.

Hillebrandite

Hillebrandite occurs as a natural mineral [START_REF] Wright | On three contact minerals from Velardena, Durango, Mexico (gehlenite, spurrite and hillebrandite)[END_REF] but is difficult to prepare synthetically in a pure state. A mixture of CaO and quartz (grain size 10 μm), with a molar CaO/SiO 2 (C/S) ratio = 2 and a water/solid (W/S) ratio = 15, was heated at 200 °C under steam pressure for 7 days, and then solvent dried by using acetone-ether.

C-S-H, jaffeite, α-C 2 SH, xonotlite and kilchoanite preparation

These five phases were not synthesized as pure products but can be formed (always in mixtures) during cement hydration under high pressure and temperature as follows. Synthetic tricalcium silicate C 3 S (triclinic T1 structure) has been mixed at room temperature with a water/cement (W/C) ratio of 0.44, the standard consistency requirement of class G cement, used in oil well cementing [START_REF] Noïk | Oilwell cement durability[END_REF]. The average grain size of C 3 S determined by laser granulometry is 10.1 μm and the specific surface reaches 3.15 g/cm 3 . Free lime CaO (∼ 1%) and C 2 S (∼ 0.5%) as impurities are respectively detected by X-ray diffraction and 29 Si NMR in the C 3 S sample. The C 3 S purity is therefore acceptable for hydration. Distilled water was used. In some cases, SiO 2 named "S8 Silica" (provided by B.J. Services) is added (W/C = 0.55) until the C/S ratio is 1.5. BET surface is 0.62 m 2 /g and half of the grains are smaller than 20.7 μm. S8 silica contains mainly crushed quartz but microcline KAlSi 3 O 8 is present as impurity (which is not detected either by X-ray diffraction or by 29 Si NMR).

A Teflon pot filled up with this cement paste was placed in a high temperature and pressure cell described elsewhere [START_REF] Méducin | Effect of high pressure on tricalcium silicate hydration at normal temperature and at 200[END_REF]. Argon was used as the pressure transmitting gas. The pressure was regulated first, and then the external heating began once the pressure was constant. Stable pressure and temperature conditions were reached after half an hour, and the silicate sample was left in the cell to react. After a determined curing time, the pressure and temperature were rapidly dropped. After removal and crushing, the sample was analyzed.

Analyses

Hydrates were studied by a set of methods. First, the X-ray Diffraction (XRD) was performed on a Philips PW 1820 device, using Cu K α radiation, to characterize the crystal structures. Measurements were performed on powdered samples. The diffraction patterns were obtained in the range 2Θ = 10 to 80°w ith a step of 0.03°and 4 s per step.

Both 29 Si and 1 H Nuclear Magnetic Resonance (NMR) were recorded to determine the characteristic peaks for each phase and to allow the quantification of mixtures [START_REF] Méducin | Complementary analyses of a tricalcium silicate hydrated at high pressure and temperature[END_REF]. The samples with 29 Si isotopic natural abundance (4.7%) were investigated using an ASX 300 Bruker spectrometer at a resonance frequency of 59.6 MHz. The pulse sequences used -MAS Single Pulse Experiment (SPE) and 1 H- 29 Si CPMAS (Cross-Polarized Magic Angle Spinning)are well known in solidstate NMR [START_REF] Lippmaa | A high resolution 29 Si NMR study of the hydration of tricalcium silicate[END_REF][START_REF] Rodger | Hydration of tricalcium silicate followed by 29 Si NMR with cross-polarization[END_REF]. The resolution is optimized with the signal of Q 8 M 8 , [Si(CH 3 ) 3 ] 8 Si 8 O 20 , which was used also as reference for the chemical shift measurement, giving a peak at 11.6 ppm. MAS Single Pulse and CPMAS experiments were performed with the following parameters: a π/2 pulse length of 3.3 μs, a spinning frequency of 5 kHz, a number of scans equal to 10,000, a recycle interval of 5 s and a contact time ranging from 1 to 10 ms in the CPMAS experiments. With the CPMAS experiments the dynamics of polarization of each phase are clearly seen to be in agreement with previous work [START_REF] Bresson | Pressure and temperature influence on tricalcium silicate hydration. A 1 H and a 29 Si NMR study[END_REF][START_REF] Bresson | Tricalcium silicate hydration at high temperature. A 29 Si and a 1 H investigation[END_REF]. 1 H NMR experiments were performed on an ASX 300 Bruker spectrometer at a resonance frequency of 300 MHz on a sample with 1 H isotopic natural abundance (∼ 100%). The pulse sequence used -CRAMPS (Combined Rotation And Multi Pulse Spectroscopy) [START_REF] Gerstein | Encyclopedia of Nuclear Magnetic Resonance[END_REF] is well known in solid-state 1 H-NMR [START_REF] Bresson | Contribution of 1 H combined rotation and multipulse spectroscopy nuclear magnetic resonance to the study of tricalcium silicate hydration[END_REF][START_REF] Heidemann | Characterization of protons in C-S-H phases by means of high-speed 1 H MAS NMR investigation[END_REF]. It allows removal of the homonuclear proton-proton dipolar interaction, which broadens the peaks in MAS 1 H-NMR spectra. Resolution and calibration are adjusted with the signal of adamantane, C 10 H 16 , which gives a peak at 1.7 ppm. All the chemical shifts are given with ± 0.4 ppm accuracy.

Finally, Scanning Electron Microscopy (SEM) was used to observe the morphology of the different phases. Secondary electron images of fractures were performed on a JEOL 6300F field emission electron microscope. The samples were first coated with a thin metallic layer of platinum to increase the poor conductivity. The observations were recorded under a 4 ⁎ 10 -6 Torr vacuum at 15 kV. The unit cell of portlandite is hexagonal, with the following parameters a = b = 3.545 Å and c = 4.895 Å, space group P-3m1 (trigonal) [START_REF] Megaw | The thermal expansions of certain crystals with layer lattices[END_REF][START_REF] Henderson | A nuclear magnetic resonance determination of the hydrogen positions in Ca(OH) 2 at T = 25 °C[END_REF]. Hydrogen atom occupies only one crystallographic site (1/3, 2/3, 0.437). The positions for calcium and oxygen are respectively (0,0,0) and (1/3, 2/3, 0.233) [START_REF] Winchell | Microscopic Character of Artificial Inorg[END_REF][START_REF] Petch | The hydrogen positions in portlandite, Ca(OH) 2 , as indicated by the electron distribution[END_REF]. Its performed diffraction pattern is in broad agreement with data in literature and is therefore not shown. In this phase, the CaO 6 octahedra share edge as can be seen in Fig. 1 (simulated structure by the software CaRIne).

Results and discussion

Moreover, this phase has been widely studied by 1 H-NMR [START_REF] Bresson | Pressure and temperature influence on tricalcium silicate hydration. A 1 H and a 29 Si NMR study[END_REF][START_REF] Heidemann | Proton high-resolution solid-state NMR spectroscopy using CRAMPS techniques for studies in silicate and cement science[END_REF]. Different pulse sequences (CRAMPS, MREV8) were used and a single peak (corresponding to the only crystallographic site) was found with an isotropic chemical shift equal to 0.7 ppm, which was obtained with the CRAMPS sequence in this study as well (Fig. 2). This peak corresponds to the hydrogen atom linked to the group Ca-O-, as specified in Heidemann's table of chemical shifts [START_REF] Heidemann | Proton high-resolution solid-state NMR spectroscopy using CRAMPS techniques for studies in silicate and cement science[END_REF].

3.1.2. Hillebrandite, Ca 2 (SiO 3 )(OH) 2 (β-dicalcium silicate hydrate)
This phase was synthesised according to the protocol given above and the analyses are then possible on the pure phase. The results obtained allow a comparison with literature values permitting the latter to be identified in phase mixtures.

The unit cell of the structure is orthorhombic with a = 3.6389 Å, b = 16.311 Å and c = 11.829 Å; Cmc21 [START_REF] Heller | X-Ray investigation of hillebrandite[END_REF][START_REF] Dai | Crystal structure of hillebrandite: a natural analogue of calcium silicate hydrate (CSH) phases in Portland cement[END_REF]. The hydrogen atom can be found in two fully occupied crystallographic sites; silicon has two crystallographic sites as well but the occupancy is half for both [START_REF] Dai | Crystal structure of hillebrandite: a natural analogue of calcium silicate hydrate (CSH) phases in Portland cement[END_REF]. The purity of the phase is checked by XRD and hillebrandite is the only phase Fig. 2. 1 H-NMR CRAMPS spectrum of portlandite with adamantane. Fig. 3. 29 Si-NMR spectrum (SPE) of hillebrandite. detected. The diffraction pattern performed brings no new scientific insights and is therefore not presented.

The recorded 29 Si-NMR spectrum (MAS SPE) is provided in Fig. 3. Two peaks are observed at -86.0 ppm (corresponding to Q 2 entities) and -77.8 ppm (corresponding to Q 1 entities, undetectable on the SPE but clearly observed on the CPMAS spectrum) in agreement with Michel and Engelhardt's table of chemical shifts where Q n stands for a SiO 4 tetrahedron linked to n other tetrahedra [START_REF] Engelhardt | High-Resolution Solid-State NMR of Silicates and Zeolites[END_REF]. This result is consistent with previous reports [START_REF] Bell | Study of calcium silicate hydrates by solid state high resolution 29 Si nuclear magnetic resonance[END_REF][START_REF] Cong | 29 Si and 17 O NMR investigation of the structure of some crystalline calcium silicate hydrates[END_REF].

In the 1 H-NMR spectrum obtained with the CRAMPS sequence, two peaks are found at 2.0 and 0.7 ppm (Fig. 4). The first peak is related to the H atom connected with Si-O-(or with H-O-) and the second to the H atom connected to Ca-Oin hillebrandite (no portlandite detected by XRD). Therefore, the two different crystallographic sites cannot be distinguished by CRAMPS. The spectrum given has been performed without the adamantaneadded later as a referenceto confirm the chemical shifts.

Crystallographic data concerning the other phases

The five other phases were not synthesised but crystallographic data from the literature provide useful information in understanding NMR spectra and in the identification of different phases in a mixture. [START_REF] Taylor | Cement Chemistry[END_REF][START_REF] Klur | C-S-H structure evolution with calcium content by multinuclear NMR[END_REF]. A structure has nevertheless been reported at room temperature [START_REF] Klur | C-S-H structure evolution with calcium content by multinuclear NMR[END_REF]. Fig. 5 shows the double Ca-layers and the SiO 4 tetrahedrons chains (Q 1 and Q 2 entities). Isotropic chemical shifts between -70 and -92 ppm for the 29 Si-NMR are then expected for this phase in a mixture, in agreement with Michel and Engelhardt's table of chemical shifts [START_REF] Engelhardt | High-Resolution Solid-State NMR of Silicates and Zeolites[END_REF]. 0.438, 0.110) and (0.406, 0.413, 0.339) [START_REF] Marsh | A revised structure for α-dicalcium silicate hydrate[END_REF]. The simulated structure (Fig. 7) shows calcium layers and isolated SiO 4 tetrahedrons (Q 0 entities). [START_REF] Hejny | Polytypism in xonotlite Ca 6 Si 6 O 17 (OH) 2[END_REF]. The simulated structure shows successive calcium and SiO 4 tetrahedrons layers (Fig. 8). The Q 2 entities are clearly identified here. We are then able to predict isotropic chemical shifts for the 29 Si-NMR between -75 and -92 ppm if this phase is part of a mixture [START_REF] Engelhardt | High-Resolution Solid-State NMR of Silicates and Zeolites[END_REF]. [START_REF] Kimata | The crystal structure of manganoan kilchoanite, Ca 2.33 Mn 0.67-Si 2 O 7 : a site-preference rule for the substitution of Mn for Ca[END_REF]. The simulated structure (by CaRIne) shows calcium and SiO 4 tetrahedrons layers (Fig. 9) where Q 0 entities and Q 2 entities could be identified.

Calcium silicate hydrates (C-S-H). C-S-H are poorly crystallized and non-stoichiometric

Identification of phases in mixtures

During oil well cementing, the hydration of C 3 S may take place under high pressure and temperature and a mixture of calcium silicate hydrates is formed. Through the identification of the two first phases, portlandite and hillebrandite, by 29 Si-NMR and 1 H-NMR CRAMPS and the crystallographic data gathered from the literature, the other phases can be identified with the following method. Three samples were investigated, they were formed respectively at 120 °C, 400 bar for 4 days; 200 °C, 600 bar for 14 days and 200 °C, 600 bar for 4 days with silica addition.

C 3 S hydration at 120 °C 400 bar for 4 days

X-Ray powder diffraction identifies as crystalline phases (jaffeite Ca 6 (Si 2 O 7 )(OH) 6 , portlandite Ca(OH) 2 and calcite CaCO 3 ). Fig. 10 shows the 29 Si-NMR SPE MAS spectrum for a C 3 S sample hydrated at 120 °C, under 400 bar for 4 days. Three peaks are observed at -85.6 and -82.3 ppm (corresponding to Q 2 entities, i.e. SiO 4 tetrahedra in the middle of silicate chains) and -79.3 ppm corresponds to Q 1 entities, tetrahedra at the ends of silicate chains [START_REF] Engelhardt | High-Resolution Solid-State NMR of Silicates and Zeolites[END_REF]. The peaks at -85.6 and -79.3 are very broad showing that the observed phase is poorly crystallized. Its Fig. 10. 29 Si-NMR spectrum (SPE) of the C 3 S hydrated sample at 120 °C under 400 bar for 4 days. full width at half maximum height is ∼ 4 ppm. The third peak, at -82.3 ppm, is not so broad and CPMAS experiments (given elsewhere [START_REF] Méducin | Complementary analyses of a tricalcium silicate hydrated at high pressure and temperature[END_REF]) at different contact times, 1, 5 and 10 ms, were undertaken to highlight the presence of two different phases. Previous work shows that the peak at -82.3 ppm is due to jaffeite [START_REF] Bresson | Pressure and temperature influence on tricalcium silicate hydration. A 1 H and a 29 Si NMR study[END_REF] and this is in agreement with its crystal structure in which Q 2 entities were clearly identified (Fig. 6). Fig. 11 shows the 1 H-NMR CRAMPS spectrum of this sample. Two peaks are detected at around 4.8 ppm (H linked to -OSi or -OH), 0.7 ppm and a shoulder at -0.2 ppm (H linked to -OCa). It confirms the presence of portlandite in the sample (also detected by X-ray diffraction) from the chemical shift at 0.7 ppm. The presence of C-S-H in the sample is highlighted through peaks at around 4.8 ppm and 0.7 ppm, as reported in the literature [START_REF] Bresson | Pressure and temperature influence on tricalcium silicate hydration. A 1 H and a 29 Si NMR study[END_REF][START_REF] Heidemann | Characterization of protons in C-S-H phases by means of high-speed 1 H MAS NMR investigation[END_REF]. The peak at 0.7 ppm (H linked to -OCa) is a contribution from CH and C-S-H. The only peak not yet attributed must be the CRAMPS signature of jaffeite, detected in the mixture by XRD and 29 Si-NMR: the chemical shift for jaffeite is then -0.2 ppm.

This sample has been investigated by SEM and the following morphologies have been determined. Fig. 12 gives a C-S-H morphology, which forms radiating clusters with numerous disordered needles [START_REF] Taylor | Cement Chemistry[END_REF][START_REF] Schlegel | Faserförmige Kalziumsilikathydrate[END_REF]. Fig. 13 shows blocks of portlandite, the composition of which was determined by dispersive X-ray analysis. This analysis clearly confirms that there was no silicon in these blocks, only calcium and oxygen (hydrogen was not detected). The only phase containing no silicon in the sample is CH. The morphology of jaffeite is said to be acicular [START_REF] Schlegel | Faserförmige Kalziumsilikathydrate[END_REF] and the only other morphology found in the considered sample were the organized needles (fasciae), as shown in Fig. 14.

C 3 S hydration at 200 °C 600 bar for 14 days

The phases formed during the hydration of a C 3 S sample at 200 °C under 600 bar for 14 days were examined. XRD detects portlandite, hillebrandite, jaffeite and α-dicalcium silicate hydrate. Fig. 15 gives the 29 Si-NMR MAS spectrum for this sample. Three peaks are observed at -86.0 ppm (hillebrandite); -82.3 ppm (jaffeite) and -72.4 ppm. The latter corresponds to Q 0 entities in α-dicalcium silicate hydrate. This assignment agrees with the structure of α-C 2 SH (Fig. 7).

The same sample, investigated by 1 H-NMR CRAMPS (Fig. 16), gives four peaks: at 2.3 ppm, 1.7 ppm (hillebrandite), 0.7 ppm (portlandite) and -0.2 ppm (jaffeite). A fifth peak for adsorbed water can be seen at around 5 ppm. The peak at 2.3 ppm is consequently the signature of α-dicalcium silicate hydrate. In the literature, this phase is said to give two 1 H-NMR CRAMPS peaks at 2.3 ppm and -9.6 ppm [START_REF] Heidemann | Characterization of protons in C-S-H phases by means of high-speed 1 H MAS NMR investigation[END_REF][START_REF] Heidemann | Proton high-resolution solid-state NMR spectroscopy using CRAMPS techniques for studies in silicate and cement science[END_REF], but our work only shows clearly the former.

Previous work shows that hillebrandite has an acicular disordered morphology [START_REF] Schlegel | Faserförmige Kalziumsilikathydrate[END_REF] and such needles are indeed found in the present sample (Fig. 17). The morphology of α-dicalcium silicate hydrate is said to be acicular [START_REF] Schlegel | Faserförmige Kalziumsilikathydrate[END_REF] but is not often observed owing to the damage by the microprobe electron beam [START_REF] Taylor | Cement Chemistry[END_REF]. However, a cloudy aspect could be seen for the studied sample, which corresponds to none of the other morphologies found in Fig. 18.

C 3 S hydration at 200 °C 600 bar for 4 days with silica addition

The phases formed after the hydration of tricalcium silicate under 200 °C and 600 bar for 4 days with silica addition (W/ C = 0.55; C/S = 1.5) have been investigated. The phases detected by XRD are kilchoanite, xonotlite and α-dicalcium silicate hydrate. The added silica was not detected and should have totally reacted.

The 29 Si-NMR spectrum of the sample is given Fig. 19. Five peaks are observed, at -72.4 ppm (α-dicalcium silicate hydrate) with a shoulder around -73.0 ppm, -75.3 ppm, -86.7 ppm (broad peak) and -97.8 ppm. The four latter peaks can be assigned to xonotlite and kilchoanite with the help of literature data. Spectra of xonotlite are found for single pulse (MAS) and crosspolarization experiments (CPMAS with a 3 ms contact time) in the literature [START_REF] Bell | Study of calcium silicate hydrates by solid state high resolution 29 Si nuclear magnetic resonance[END_REF]. Three peaks are observed for this phase at -86.7 and -97.7 ppm (corresponding to Q 2 and Q 3 entities respectively) and -79.8 ppm corresponding to Q 1 entities, undetectable on the single pulse spectrum. Consequently, the peaks at -86.7 ppm and -97.8 ppm observed in our work can be attributed to xonotlite, although the peak expected at -79.8 ppm was not clearly detected. The 29 Si-NMR MAS spectra of kilchoanite found in the literature are for single pulse (MAS) and cross-polarization experiments (CPMAS with a 3 ms contact time) [START_REF] Bell | Study of calcium silicate hydrates by solid state high resolution 29 Si nuclear magnetic resonance[END_REF]. Three peaks are observed at -73.1 and -75.3 ppm (corresponding to Q 0 entities identified on Fig. 13) and Q 2 entities at -85.6 ppm. Bell et al. synthesized this phase at low temperature which explains the presence of water on its surface and the possibility of observing a CPMAS spectrum [START_REF] Bell | Study of calcium silicate hydrates by solid state high resolution 29 Si nuclear magnetic resonance[END_REF]. In our work, the signatures of the kilchoanite are the following: the most intense peak of the literature (at -73.1 ppm) is certainly the shoulder detected at -73.0 ppm and the peak at -75.3 ppm is readily detected, in agreement with Bell's work. The third peak of the literature may not be differentiated from the broad peak we found at -86.7 ppm.

The 1 H-NMR CRAMPS spectrum of the sample is not given since the xonotlite position could not be clearly identified. The position for the α-dicalcium silicate hydrate peak is known from the previous sample studied (2.3 ppm). The amount of xonotlite in the sample is probably not sufficient to identify clearly its peak. Alternatively both phases could give coincident peaks. Only the synthesis of pure xonotlite would resolve this point. This sample was observed by SEM, previous work shows that xonotlite has an acicular morphology [START_REF] Schlegel | Faserförmige Kalziumsilikathydrate[END_REF]. Fig. 20 shows xonotlite fibres connected to kilchoanite globules (the most abundant phase identified through the others analysis methods excluding α-dicalcium silicate hydrate).

Conclusion

Table 1 gives the characterization of each phase reviewed in this paper. The morphology is obtained by SEM images of secondary electrons and confirms data from the literature. The isotropic chemical shifts for 29 Si-NMR as well as for 1 H-NMR are given with an accuracy of ± 0.4 ppm.

3. 1 .

 1 Pure phases 3.1.1. Portlandite Ca(OH) 2 (CH)

Fig. 4 .Fig. 5 .

 45 Fig. 4. 1 H-NMR CRAMPS spectrum of hillebrandite.

3. 1 . 3 . 2 .Fig. 6 .Fig. 7 .

 13267 Fig. 6. Crystal structure of jaffeite.

Fig. 8 .

 8 Fig. 8. Crystal structure of xonotlite.

3. 1 . 3 . 4 .

 134 Xonotlite Ca 6 (Si 6 O 17 )(OH) 2 . Xonotlite in monoclinic P2/a, with a = 17.032 Å, b = 7.363 Å and c = 7.012 Å; β = 90.36°

3. 1 . 3 . 5 .

 135 Kilchoanite Ca 6 (SiO 4 )(Si 3 O 10 ). Kilchoanite occurs in nature and also as a synthetic. Previous work shows a = 11.433 Å, b = 5.08 Å and c = 22.017 Å, orthorhombic Ima2

Fig. 11 .

 11 Fig. 11. 1 H-NMR CRAMPS spectrum of the C 3 S hydrated sample at 120 °C under 400 bar for 4 days.

Fig. 12 .

 12 Fig. 12. SEM image of secondary electrons (× 5000) of the C-S-H.

Fig. 14 .

 14 Fig. 14. SEM image of secondary electrons (× 10,000) of jaffeite.

Fig. 16 .

 16 Fig. 16. 1 H-NMR CRAMPS spectrum of the C 3 S hydrated sample at 200 °C under 600 bar for 14 days.

Fig. 18 .

 18 Fig. 18. SEM image of secondary electrons (× 3500) of α-dicalcium silicate hydrate.

Fig. 20 .

 20 Fig. 20. SEM image of secondary electrons (× 5000) of xonotlite (fibres) and kilchoanite (globules).

Acknowledgements

The authors are grateful to Pierre Tougne and Victor Moulanier for expert technical support during CRAMPS experiments. FM kindly acknowledges Dr Sharon Ashbrook and Dr Karl Whittle for helpful discussions and suggestions.