Marcin Andrychowicz

Anton Raichuk

Piotr Stańczyk

Manu Orsini

Sertan Girgin

Raphaël Marinier

Léonard Hussenot

Matthieu Geist

Olivier Pietquin

Marcin Michalski

Piotr Sta

Raphael Marinier

Sylvain Gelly

Olivier Bachem

What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

Keywords: AWR_0, 03 AWR_0, 1 AWR_0, 3 PG PPO_0, 1 PPO_0, 2 PPO_0, 3 PPO_0

établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Deep reinforcement learning (RL) has seen increased interest in recent years due to its ability to have neural-network-based agents learn to act in environments through interactions. For continuous control tasks, on-policy algorithms such as REINFORCE [START_REF] Ronald | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF], TRPO [START_REF] Schulman | Trust region policy optimization[END_REF], A3C [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF], PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] and off-policy algorithms such as DDPG [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] and SAC [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF] have enabled successful applications such as quadrupedal locomotion [START_REF] Haarnoja | Soft actor-critic algorithms and applications[END_REF], self-driving [START_REF] Kendall | Learning to drive in a day[END_REF] or dexterous in-hand manipulation [START_REF] Haarnoja | Soft actor-critic algorithms and applications[END_REF][START_REF] Andrychowicz | Learning dexterous in-hand manipulation[END_REF][START_REF] Openai | Solving Rubik's Cube with a Robot Hand[END_REF].

Many of these papers investigate in depth different loss functions and learning paradigms. Yet, it is less visible that behind successful experiments in deep RL there are complicated code bases that contain a large number of low-and high-level design decisions that are usually not discussed in research papers. While one may assume that such "choices" do not matter, there is some evidence that they are in fact crucial for or even driving good performance [START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF].

While there are open-source implementations available that can be used by practitioners, this is still unsatisfactory: In research publications, often different algorithms implemented in different code bases are compared one-to-one. This makes it impossible to assess whether improvements are due to the algorithms or due to their implementations. Furthermore, without an understanding of lower-level choices, it is hard to assess the performance of high-level algorithmic choices as performance may strongly depend on the tuning of hyperparameters and implementation-level details. Overall, this makes it hard to attribute progress in RL and slows down further research [START_REF] Islam | Reproducibility of benchmarked deep reinforcement learning tasks for continuous control[END_REF][START_REF] Henderson | Deep reinforcement learning that matters[END_REF][START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF].

Our contributions. Our key goal in this paper is to investigate such lower level choices in depth and to understand their impact on final agent performance. Hence, as our key contributions, we [START_REF] Peter | Robust estimation of a location parameter[END_REF] implement >50 choices in a unified on-policy algorithm implementation, (2) conducted a large-scale (more than 250'000 agents trained) experimental study that covers different aspects of the training 1. We researched prior work and popular code bases to make a list of commonly used choices, i.e., different loss functions (both for value functions and policies), architectural choices such as initialization methods, heuristic tricks such as gradient clipping and all their corresponding hyperparameters. 2. Based on this, we implemented a single, unified on-policy agent and corresponding training protocol starting from the SEED RL code base [START_REF] Espeholt | SEED RL: Scalable and Efficient Deep-RL with Accelerated Central Inference[END_REF]. Whenever we were faced with implementation decisions that required us to take decisions that could not be clearly motivated or had alternative solutions, we further added such decisions as additional choices. 3. We verified that when all choices are selected as in the PPO implementation from OpenAI baselines, we obtain similar performance as reported in the PPO paper [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. We chose PPO because it is probably the most commonly used on-policy RL algorithm at the moment.

The resulting agent implementation is detailed in Appendix B. The key property is that the implementation exposes all choices as configuration options in an unified manner. For convenience, we mark each of the choice in this paper with a number (e.g., C1) and a fixed name (e.g. num_envs (C1)) that can be easily used to find a description of the choice in Appendix B.

Difficulty of investigating choices. The primary goal of this paper is to understand how the different choices affect the final performance of an agent and to derive recommendations for these choices. There are two key reasons why this is challenging:

First, we are mainly interested in insights on choices for good hyperparameter configurations. Yet, if all choices are sampled randomly, the performance is very bad and little (if any) training progress is made. This may be explained by the presence of sub-optimal settings (e.g., hyperparameters of the wrong scale) that prohibit learning at all. If there are many choices, the probability of such failure increases exponentially.

Second, many choices may have strong interactions with other related choices, for example the learning rate and the minibatch size. This means that such choices need to be tuned together and experiments where only a single choice is varied but interacting choices are kept fixed may be misleading.

Basic experimental design. To address these issues, we design a series of experiments as follows:

We create groups of choices around thematic groups where we suspect interactions between different choices, for example we group together all choices related to neural network architecture. We also include Adam learning rate (C24) in all of the groups as we suspect that it may interact with many other choices.

Then, in each experiment, we train a large number of models where we randomly sample the choices within the corresponding group 2 . All other settings (for choices not in the group) are set to settings of a competitive base configuration (detailed in Appendix C) that is close to the default PPOv2 configuration 3 scaled up to 256 parallel environments. This has two effects: First, it ensures that our set of trained models contains good models (as verified by performance statistics in the corresponding results). Second, it guarantees that we have models that have different combinations of potentially interacting choices.

We then consider two different analyses for each choice (e.g, for advantage_estimator (C6)):

Conditional 95th percentile: For each potential value of that choice (e.g., advantage_estimator (C6) = N-Step), we look at the performance distribution of sampled configurations with that value. We report the 95th percentile of the performance as well as a confidence interval based on a binomial approximation 4 . Intuitively, this corresponds to a robust estimate of the performance one can expect if all other choices in the group were tuned with random search and a limited budget of roughly 20 hyperparameter configurations.

Distribution of choice within top 5% configurations. We further consider for each choice the distribution of values among the top 5% configurations trained in that experiment. The reasoning is as follows: By design of the experiment, values for each choice are distributed uniformly at random. Thus, if certain values are over-represented in the top models, this indicates that the specific choice is important in guaranteeing good performance.

Performance measures. We employ the following way to compute performance: For each hyperparameter configuration, we train 3 models with independent random seeds where each model is trained for one million (Hopper, HalfCheetah, Walker2d) or two million environment steps (Ant, Humanoid). We evaluate trained policies every hundred thousand steps by freezing the policy and computing the average undiscounted episode return of 100 episodes (with the stochastic policy). We then average these score to obtain a single performance score of the seed which is proportional to the area under the learning curve. This ensures we assign higher scores to agents that learn quickly. The performance score of a hyperparameter configuration is finally set to the median performance score across the 3 seeds. This reduces the impact of training noise, i.e., that certain seeds of the same configuration may train much better than others.

3 Experiments Interpretation. Fig. 1 shows the 95-th percentile of the average policy score during training for different policy losses (C14). We observe that PPO performs better than the other losses on 4 out of 5 environments and is one of the top performing losses on HalfCheetah. As we randomly sample the loss specific hyperparameters in this analysis, one might argue that our approach favours choices that are not too sensitive to hyperparameters. At the same time, there might be losses that are sensitive to their hyperparameters but for which good settings may be easily found. Fig. 5 shows that even if we condition on choosing the optimal loss hyperparameters for each loss 6 , PPO still outperforms the other losses on the two hardest tasks -Humanoid and Ant 7 and is one of the top performing losses on the other 3 tasks. Moreover, we show the empirical cumulative density functions of agent performance conditioned on the policy loss used in Fig. 4. Perhaps unsurprisingly, PG and V-trace perform worse on all tasks. This is likely caused by their inability to handle data that become off-policy in one iteration, either due to multiple passes (C3) over experience (which can be seen in Fig. 14) or a large experience buffer (C2) in relation to the batch size (C4). Overall, these results show that trust-region optimization (preventing the current policy from diverging too much from the behavioral one) which is present in all the other policy losses is crucial for good sample complexity.

For PPO and its clipping threshold (C16), we further observe that = 0.2 and = 0.3 perform reasonably well in all environments but that lower ( = 0.1) or higher ( = 0.5) values give better performance on some of the environments (See Fig. 10 and Fig. 32).

Recommendation. Use the PPO policy loss. Start with the clipping threshold set to 0.25 but also try lower and higher values if possible.

Networks architecture (based on the results in Appendix E)

Study description. We investigate the impact of differences in the policy and value function neural network architectures. We consider choices related to the network structure and size (C47, C48, C49, C50, C51, C52, C52), activation functions (C55), and initialization of network weights (C56, C57, C58). We further include choices related to the standard deviation of actions (C59, C60, C61, C62) and transformations of sampled actions (C63).

Interpretation. Separate value and policy networks (C47) appear to lead to better performance on four out of five environments (Fig. 15). To avoid analyzing the other choices based on bad models, we thus focus for the rest of this experiment only on agents with separate value and policy networks. Regarding network sizes, the optimal width of the policy MLP depends on the complexity of the environment (Fig. 18) and too low or too high values can cause significant drop in performance while for the value function there seems to be no downside in using wider networks (Fig. 21). Moreover, on some environments it is beneficial to make the value network wider than the policy one, e.g. on HalfCheetah the best results are achieved with 16 -32 units per layer in the policy network and 256 in the value network. Two hidden layers appear to work well for policy (Fig. 22) and value networks (Fig. 20) in all tested environments. As for activation functions, we observe that tanh activations perform best and relu worst. (Fig. 30).

Interestingly, the initial policy appears to have a surprisingly high impact on the training performance.

The key recipe appears is to initialize the policy at the beginning of training so that the action distribution is centered around 08 regardless of the observation and has a rather small standard deviation. This can be achieved by initializing the policy MLP with smaller weights in the last layer (C57, Fig. 24, this alone boosts the performance on Humanoid by 66%) so that the initial action distribution is almost independent of the observation and by introducing an offset in the standard deviation of actions (C61). Fig. 2 shows that the performance is very sensitive to the initial action standard deviation with 0.5 performing best on all environments except Hopper where higher values perform better. domain expected by the environment (C63): clipping and applying a tanh function. tanh performs slightly better overall (in particular it improves the performance on HalfCheetah by 30% ). Comparing Fig. 17 and Fig. 2 suggests that the difference might be mostly caused by the decreased magnitude of initial actions 9 .

Other choices appear to be less important: The scale of the last layer initialization matters much less for the value MLP (C58) than for the policy MLP (Fig. 19). Apart from the last layer scaling, the network initialization scheme (C56) does not matter too much (Fig. 27). Only he_normal and he_uniform [START_REF] He | Delving deep into rectifiers: Surpassing human-level performance on imagenet classification[END_REF] appear to be suboptimal choices with the other options performing very similarly. There also appears to be no clear benefits if the standard deviation of the policy is learned for each state (i.e. outputted by the policy network) or once globally for all states (C59, Fig. 23). For the transformation of policy output into action standard deviation (C60), softplus and exponentiation perform very similarly 10 (Fig. 25). Finally, the minimum action standard deviation (C62) seems to matter little, if it is not set too large (Fig. 30).

Recommendation. Initialize the last policy layer with 100× smaller weights. Use softplus to transform network output into action standard deviation and add a (negative) offset to its input to decrease the initial standard deviation of actions. Tune this offset if possible. Use tanh both as the activation function (if the networks are not too deep) and to transform the samples from the normal distribution to the bounded action space. Use a wide value MLP (no layers shared with the policy) but tune the policy width (it might need to be narrower than the value MLP).

Normalization and clipping (based on the results in Appendix F)

Study description. We investigate the impact of different normalization techniques: observation normalization (C64), value function normalization (C66), per-minibatch advantage normalization (C67), as well as gradient (C68) and observation (C65) clipping.

Interpretation. Input normalization (C64) is crucial for good performance on all environments apart from Hopper (Fig. 33). Quite surprisingly, value function normalization (C66) also influences the performance very strongly -it is crucial for good performance on HalfCheetah and Humanoid, helps slightly on Hopper and Ant and significantly hurts the performance on Walker2d (Fig. 37). We are not sure why the value function scale matters that much but suspect that it affects the performance by changing the speed of the value function fitting. 11 In contrast to observation and value function normalization, per-minibatch advantage normalization (C67) seems not to affect the performance too much (Fig. 35). Similarly, we have found little evidence that clipping normalized12 observations (C65) helps (Fig. 38) but it might be worth using if there is a risk of extremely high observations due to simulator divergence. Finally, gradient clipping (C68) provides a small performance boost with the exact clipping threshold making little difference (Fig. 34).

Recommendation. Always use observation normalization and check if value function normalization improves performance. Gradient clipping might slightly help but is of secondary importance.

Advantage Estimation (based on the results in Appendix G)

Study description. We compare the most commonly used advantage estimators (C6): N-step [START_REF] Richard S Sutton | Introduction to reinforcement learning[END_REF], GAE [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF] and V-trace [START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF] and their hyperparameters (C7, C8, C9, C10). We also experiment with applying PPO-style pessimistic clipping (C13) to the value loss (present in the original PPO implementation but not mentioned in the PPO paper [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]) and using Huber loss [START_REF] Peter | Robust estimation of a location parameter[END_REF] instead of MSE for value learning (C11, C12). Moreover, we varied the number of parallel environments used (C1) as it changes the length of the experience fragments collected in each step.

Interpretation. GAE and V-trace appear to perform better than N-step returns (Fig. 44 and40) which indicates that it is beneficial to combine the value estimators from multiple timesteps. We have not found a significant performance difference between GAE and V-trace in our experiments. λ = 0.9 (C8, C9) performed well regardless of whether GAE (Fig. 45) or V-trace (Fig. 49) was used on all tasks but tuning this value per environment may lead to modest performance gains. We have found that PPO-style value loss clipping (C13) hurts the performance regardless of the clipping threshold 13(Fig. 43). Similarly, the Huber loss (C11) performed worse than MSE in all environments (Fig. 42) regardless of the value of the threshold (C12) used (Fig. 48).

Recommendation. Use GAE with λ = 0.9 but neither Huber loss nor PPO-style value loss clipping.

Training setup (based on the results in Appendix H)

Study description. We investigate choices related to the data collection and minibatch handling: the number of parallel environments used (C1), the number of transitions gathered in each iteration (C2), the number of passes over the data (C3), minibatch size (C4) and how the data is split into minibatches (C5).

For the last choice, in addition to standard choices, we also consider a new small modification of the original PPO approach: The original PPO implementation splits the data in each policy iteration step into individual transitions and then randomly assigns them to minibatches (C5). This makes it impossible to compute advantages as the temporal structure is broken. Therefore, the advantages are computed once at the beginning of each policy iteration step and then used in minibatch policy and value function optimization. This results in higher diversity of data in each minibatch at the cost of using slightly stale advantage estimations. As a remedy to this problem, we propose to recompute the advantages at the beginning of each pass over the data instead of just once per iteration.

Results. Unsurprisingly, going over the experience multiple times appears to be crucial for good sample complexity (Fig. 54). Often, this is computationally cheap due to the simple models considered, in particular on machines with accelerators such as GPUs and TPUs. As we increase the number of parallel environments (C1), performance decreases sharply on some of the environments (Fig. 55). This is likely caused by shortened experience chunks (See Sec. B.1 for the detailed description of the data collection process) and earlier value bootstrapping. Despite that, training with more environments usually leads to faster training in wall-clock time if enough CPU cores are available. Increasing the batch size (C4) does not appear to hurt the sample complexity in the range we tested (Fig. 57) which suggests that it should be increased for faster iteration speed. On the other hand, the number of transitions gathered in each iteration (C2) influences the performance quite significantly (Fig. 52). Finally, we compare different ways to handle minibatches (See Sec. B.1 for the detailed description of different variants) in Fig. 53 and 58. The plots suggest that stale advantages can in fact hurt performance and that recomputing them at the beginning of each pass at least partially mitigates the problem and performs best among all variants.

Recommendation. Go over experience multiple times. Shuffle individual transitions before assigning them to minibatches and recompute advantages once per data pass (See App. B.1 for the details). For faster wall-clock time training use many parallel environments and increase the batch size (both might hurt the sample complexity). Tune the number of transitions in each iteration (C2) if possible.

Timesteps handling (based on the results in Appendix I)

Study description. We investigate choices related to the handling of timesteps: discount factor 14 (C20), frame skip (C21), and how episode termination due to timestep limits are handled (C22). The latter relates to a technical difficulty explained in App. B.4 where one assumes for the algorithm an infinite time horizon but then trains using a finite time horizon [START_REF] Pardo | Time limits in reinforcement learning[END_REF].

Interpretation. Fig. 60 shows that the performance depends heavily on the discount factor γ (C20) with γ = 0.99 performing reasonably well in all environments. Skipping every other frame (C21) improves the performance on 2 out of 5 environments (Fig. 61). Proper handling of episodes abandoned due to the timestep limit seems not to affect the performance (C22, Fig. 62) which is probably caused by the fact that the timestep limit is quite high (1000 transitions) in all the environments we considered.

Recommendation. Discount factor γ is one of the most important hyperparameters and should be tuned per environment (start with γ = 0.99). Try frame skip if possible. There is no need to handle environments step limits in a special way for large step limits.

Optimizers (based on the results in Appendix J)

Study description. We investigate two gradient-based optimizers commonly used in RL: (C23) -Adam [START_REF] Diederik | Adam: A Method for Stochastic Optimization[END_REF] and RMSprop -as well as their hyperparameters (C24, C25, C26, C27, C28, C29, C30) and a linear learning rate decay schedule (C31).

Interpretation. The differences in performance between the optimizers (C23) appear to be rather small with no optimizer consistently outperforming the other across environments (Fig. 66). Unsurprisingly, the learning rate influences the performance very strongly (Fig. 69) with the default value of 0.0003 for Adam (C24) performing well on all tasks. Fig. 67 shows that Adam works better with momentum (C26). For RMSprop, momentum (C27) makes less difference (Fig. 71) but our results suggest that it might slightly improve performance 15 . Whether the centered or uncentered version of RMSprop is used (C30) makes no difference (Fig. 70) and similarly we did not find any difference between different values of the coefficients (C28, C29, Fig. 68 and72). Linearly decaying the learning rate to 0 increases the performance on 4 out of 5 tasks but the gains are very small apart from Ant, where it leads to 15% higher scores (Fig. 65).

Recommendation. Use Adam [START_REF] Diederik | Adam: A Method for Stochastic Optimization[END_REF] optimizer with momentum β 1 = 0.9 and a tuned learning rate (0.0003 is a safe default). Linearly decaying the learning rate may slightly improve performance but is of secondary importance.

3.8 Regularization (based on the results in Appendix K) Study description. We investigate different policy regularizers (C32), which can have either the form of a penalty (C33, e.g. bonus for higher entropy) or a soft constraint (C34, e.g. entropy should not be lower than some threshold) which is enforced with a Lagrange multiplier. In particular, we consider the following regularization terms: entropy (C40, C46), the Kullback-Leibler divergence (KL) between a reference N (0, 1) action distribution and the current policy (C37, C43) and the KL divergence and reverse KL divergence between the current policy and the behavioral one (C35, C41, C36, C42), as well as the "decoupled" KL divergence from [START_REF] Abdolmaleki | Maximum a posteriori policy optimisation[END_REF][START_REF] Francis | V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control[END_REF] (C38, C39, C44, C45).

Interpretation. We do not find evidence that any of the investigated regularizers helps significantly on our environments with the exception of HalfCheetah on which all constraints (especially the entropy constraint) help (Fig. 76 and77). However, the performance boost is largely independent on the constraint threshold (Fig. 83, 84, 87, 89, 90 and 91) which suggests that the effect is caused by the initial high strength of the penalty (before it gets adjusted) and not by the desired constraint. While it is a bit surprising that regularization does not help at all (apart from HalfCheetah), we conjecture that regularization might be less important in our experiments because: (1) the PPO policy loss already enforces the trust region which makes KL penalties or constraints redundant; and (2) the careful policy initialization (See Sec. 3.2) is enough to guarantee good exploration and makes the entropy bonus or constraint redundant.

Related Work

Islam et al. [START_REF] Islam | Reproducibility of benchmarked deep reinforcement learning tasks for continuous control[END_REF] and Henderson et al. [START_REF] Henderson | Deep reinforcement learning that matters[END_REF] point out the reproducibility issues in RL including the performance differences between different code bases, the importance of hyperparameter tuning and the high level of stochasticity due to random seeds. Tucker et al. [START_REF] Tucker | The mirage of action-dependent baselines in reinforcement learning[END_REF] showed that the gains, which had been attributed to one of the recently proposed policy gradients improvements, were, in fact, caused by the implementation details. The most closely related work to ours is probably Engstrom et al. [START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF] where the authors investigate code-level improvements in the PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] code base and conclude that they are responsible for the most of the performance difference between PPO and TRPO [START_REF] Schulman | Trust region policy optimization[END_REF]. Our work is also similar to other large-scale studies done in other fields of Deep Learning, e.g. model-based RL [START_REF] Langlois | Benchmarking model-based reinforcement learning[END_REF], GANs [START_REF] Lucic | Are gans created equal? a large-scale study[END_REF], NLP [START_REF] Kaplan | Scaling laws for neural language models[END_REF], disentangled representations [START_REF] Locatello | Challenging common assumptions in the unsupervised learning of disentangled representations[END_REF] and convolution network architectures [START_REF] Radosavovic | Designing Network Design Spaces[END_REF].

A Reinforcement Learning Background

We consider the standard reinforcement learning formalism consisting of an agent interacting with an environment. To simplify the exposition we assume in this section that the environment is fully observable. An environment is described by a set of states S, a set of actions A, a distribution of initial states p(s0), a reward function r : S × A → R, transition probabilities p(st+1|st, at) (t is a timestep index explained later), termination probabilities T (st, at) and a discount factor γ ∈ [0, 1].

A policy π is a mapping from state to a distribution over actions. Every episode starts by sampling an initial state s0. At every timestep t the agent produces an action based on the current state: at ∼ π(•|st). In turn, the agent receives a reward rt = r(st, at) and the environment's state is updated. With probability T (st, at) the episode is terminated, and otherwise the new environments state st+1 is sampled from p(•|st, at). The discounted sum of future rewards, also referred to as the return, is defined as Rt = ∞ i=t γ i-t ri. The agent's goal is to find the policy π which maximizes the expected return Eπ[R0|s0], where the expectation is taken over the initial state distribution, the policy, and environment transitions accordingly to the dynamics specified above. The Q-function or action-value function of a given policy π is defined as Q π (st, at) = Eπ[Rt|st, at], while the V-function or state-value function is defined as

V π (st) = Eπ[Rt|st]. The value A π (st, at) = Q π (st, at) -V π (st)
is called the advantage and tells whether the action at is better or worse than an average action the policy π takes in the state st.

In practice, the policies and value functions are going to be represented as neural networks. In particular, RL algorithms we consider maintain two neural networks: one representing the current policy π and a value network which approximates the value function of the current policy V ≈ V π .

B List of Investigated Choices

In this section we list all algorithmic choices which we consider in our experiments. See Sec. A for a very brief introduction to RL and the notation we use in this section.

B.1 Data collection and optimization loop

RL algorithms interleave running the current policy in the environment with policy and value function networks optimization. In particular, we create num_envs (C1) environments [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF]. In each iteration, we run all environments synchronously sampling actions from the current policy until we have gathered iteration_size (C2) transitions total (this means that we have num_envs (C1) experience fragments, each consisting of iteration_size (C2) / num_envs (C1) transitions). Then, we perform num_epochs (C3) epochs of minibatch updates where in each epoch we split the data into minibatches of size batch_size (C4), and performing gradient-based optimization [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]. Going over collected experience multiple times means that it is not strictly an on-policy RL algorithm but it may increase the sample complexity of the algorithm at the cost of more computationally expensive optimization step.

We consider four different variants of the above scheme (choice C5):

• Fixed trajectories: Each minibatch consists of full experience fragments and in each epoch we go over exactly the same minibatches in the same order.

• Shuffle trajectories: Like Fixed trajectories but we randomly assign full experience fragments to minibatches in each epoch.

• Shuffle transitions: We break experience fragments into individual transitions and assign them randomly to minibatches in each epoch. This makes the estimation of advantages impossible in each minibatch (most of the advantage estimators use future states, See App. B.2) so we precompute all advantages at the beginning of each iteration using full experience fragments. This approach leads to higher diversity of data in each minibatch at the price of somewhat stale advantage estimations. The original PPO implementation from OpenAI Baselines 16 works this way but this is not mentioned in the PPO paper [START_REF] Schulman | Proximal policy optimization algorithms[END_REF].

• Shuffle transitions (recompute advantages): Like Shuffle transitions but we recompute advantages at the beginning of each epoch.

B.2 Advantage estimation

Let V be an approximator of the value function of some policy, i.e. V ≈ V π . We experimented with the three most commonly used advantage estimators in on-policy RL (choice C6):

• N-step return [START_REF] Richard S Sutton | Introduction to reinforcement learning[END_REF] is defined as

V (N ) t = t+N -1 i=t γ i-t ri + γ N V (st+N ) ≈ V π (st).
The parameter N (choice C7) controls the bias-variance tradeoff of the estimator with bigger values resulting in an estimator closer to empirical returns and having less bias and more variance. Given N-step returns we can estimate advantages as follows:

Â(N) t = V (N) t -V (st) ≈ A π (st, at).
• Generalized Advantage Estimator, GAE(λ) [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF] is a method that combines multi-step returns in the following way:

V GAE t = (1 -λ) N >0 λ N -1 V (N ) t ≈ V π (st),
where 0 < λ < 1 is a hyperparameter (choice C8) controlling the bias-variance trade-off. Using this, we can estimate advantages with:

ÂGAE t = V GAE t -V (st) ≈ A π (st, at).
It is possible to compute the values of this estimator for all states encountered in an episode in linear time [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF].

• V-trace(λ, c, ρ) [START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF] is an extension of GAE which introduces truncated importance sampling weights to account for the fact that the current policy might be slightly different from the policy which generated the experience. It is parameterized by λ (choice C9) which serves the same role as in GAE and two parameters c and ρ which are truncation thresholds for two different types of importance weights. See [START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF] for the detailed description of the V-trace estimator. All experiments in the original paper [START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF] use c = ρ = 1. Similarly, we only consider the case c = ρ, i.e., we consider a single choice V-Trace advantage c, ρ (C10).

The value network is trained by fitting one of the returns described above with an MSE (quadratic) or a Huber [START_REF] Peter | Robust estimation of a location parameter[END_REF] loss (choice C11). Huber loss is a quadratic around zero up to some threshold (choice C12) at which point it becomes a linear function.

The original PPO implementation [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] uses an additional pessimistic clipping in the value loss function. See [START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF] for the description of this technique. It is parameterized by a clipping threshold (choice C13).

B.3 Policy losses

Let π denote the policy being optimized, and µ the behavioral policy, i.e. the policy which generated the experience. Moreover, let Âπ t and µ t be some estimators of the advantage at timestep t for the policies π and µ. We consider optimizing the policy with the following policy losses (choice C14):

• Policy Gradients (PG) [START_REF] Ronald | Simple statistical gradient-following algorithms for connectionist reinforcement learning[END_REF] with advantages: LPG = -log π(at|st) Âπ t . It can be shown that if Âπ t estimators are unbiased, then ∇ θ LPG is an unbiased estimator of the gradient of the policy performance assuming that experience was generated by the current policy π.

• V-trace [START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF]: L ρ V-trace = sg(ρt)LPG, where ρt = min( π(a t |s t ) µ(a t |s t ) , ρ) is a truncated importance weight, sg is the stop_gradient operator 17 and ρ is a hyperparameter (choice C15). ∇ θ L ρ V-trace is an unbiased estimator of the gradient of the policy performance if ρ = ∞ regardless of the behavioural policyµ 18 .

• Proximal Policy Optimization (PPO) [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]:

L PPO = -min π(at|st) µ(at|st) Âπ t , clip π(at|st) µ(at|st) , 1 1 + , 1 + Âπ t ,
where is a hyperparameter 19 C16. This loss encourages the policy to take actions which are better than average (have positive advantage) while clipping discourages bigger changes to the policy by limiting how much can be gained by changing the policy on a particular data point.

• Advantage-Weighted Regression (AWR) [START_REF] Bin | Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning[END_REF]: 17 Identity function with gradient zero. 18 Assuming that advantage estimators are unbiased and µ(at|st) > 0 for all pairs (st, at) for which π(at|st) > 0. 19 The original PPO paper used 1instead 1/(1 + ) as the lower bound for the clipping. Both variants are used in practice and we have decided to use 1/(1 + ) as it is more symmetric.

L β, ωmax AWR = -log π(at|st) min (exp(A µ t /β), ωmax) .
It can be shown that for ωmax = ∞ (choice C17) it corresponds to an approximate optimization of the policy π under a constraint of the form KL(π||µ) < where the KL bound depends on the exponentiation temperature β (choice C18). Notice that in contrast to previous policy losses, AWR uses estimates of the advantages for the behavioral policy (A µ t ) and not the current one (A π t ). AWR was proposed mostly as an off-policy RL algorithm.

• On-Policy Maximum a Posteriori Policy Optimization (V-MPO) [START_REF] Francis | V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control[END_REF]: This policy loss is the same as AWR with the following differences: (1) exponentiation is replaced with the softmax operator and there is no clipping with ωmax;

(2) only samples with the top half advantages in each batch are used; (3) the temperature β is treated as a Lagrange multiplier and adjusted automatically to keep a constraint on how much the weights (i.e. softmax outputs) diverge from a uniform distribution with the constraint threshold being a hyperparameter (choice C19). (4) A soft constraint on KL(µ||π) is added. In our experiments, we did not treat this constraint as a part of the V-MPO policy loss as policy regularization is considered separately (See Sec. B.6).

• Repeat Positive Advantages (RPA): 20 . This is a new loss we introduce in this paper. We choose this loss because it is the limiting case of AWR and V-MPO. In particular, L β, ωmax AWR converges to ωmaxLRPA for β → 0 and for → 0 V-MPO converges to RPA with [At > 0] replaced by only taking the top half advantages in each batch 21 (the two conditions become even more similar if advantage normalization is used, See Sec. B.9).

LRPA = -log π(at|st)[At > 0]

B.4 Handling of timesteps

The most important hyperparameter controlling how timesteps are handled is the discount factor γ (choice C20). Moreover, we consider the so-called frame skip22 (choice C21). Frame skip equal to n means that we modify the environment by repeating each action outputted by the policy n times (unless the episode has terminated in the meantime) and sum the received rewards. When using frame skip, we also adjust the discount factor appropriately, i.e. we discount with γ n instead of γ.

Many reinforcement learning environments (including the ones we use in our experiments) have step limits which means that an episode is terminated after some fixed number of steps (assuming it was not terminated earlier for some other reason). Moreover, the number of remaining environment steps is not included in policy observations which makes the environments non-Markovian and can potentially make learning harder [START_REF] Pardo | Time limits in reinforcement learning[END_REF].

We consider two ways to handle such abandoned episodes. We either treat the final transition as any other terminal transition, e.g. the value target for the last state is equal to the final reward, or we take the fact that we do not know what would happen if the episode was not terminated into account. In the latter case, we set the advantage for the final state to zero and its value target to the current value function. This also influences the value targets for prior states as the value targets are computed recursively [START_REF] Schulman | High-dimensional continuous control using generalized advantage estimation[END_REF][START_REF] Espeholt | IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures[END_REF]. We denote this choice by Handle abandoned? (C22).

B.5 Optimizers

We experiment with two most commonly used gradient-based optimizers in RL (choice C23): Adam [START_REF] Diederik | Adam: A Method for Stochastic Optimization[END_REF] and RMSProp. 23 You can find the description of the optimizers and their hyperparameters in the original publications.

For both optimizers, we sweep the learning rate (choices C24 and C25), momentum (choices C26 and C27) and the parameters added for numerical stability (choice C28 and C29). Moreover, for RMSProp we consider both centered and uncentered versions (choice C30). For their remaining hyperparameters, we use the default values from TensorFlow [START_REF] Abadi | Tensorflow: A system for large-scale machine learning[END_REF], i.e. β2 = 0.999 for Adam and ρ = 0.1 for RMSProp. Finally, we allow a linear learning rate schedule via the hyperparameter Learning rate decay (C31) which defines the terminal learning rate as a fraction of the initial learning rate (i.e., 0.0 correspond to a decay to zero whereas 1.0 corresponds to no decay).

B.6 Policy regularization

We consider three different modes for regularization (choice C32):

• No regularization: We apply no regularization.

• Penalty: we apply a regularizer R with fixed strength, i.e., we add to the loss the term αR for some fixed coefficient α which is a hyperparameter.

• Constraint: We impose a soft constraint of the form R < on the value of the regularizer where is a hyperparameter. This can be achieved by treating α as a Lagrange multiplier. In this case, we optimize the value of α > 0 together with the networks parameter by adding to the loss the term α • sg( -R), where sg in the stop_gradient operator. In practice, we use the following parametrization α = exp(c • p) where p is a trainable parameter (initialized with 0) and c is a coefficient controlling how fast α is adjusted (we use c = 10 in all experiments). After each gradient step, we clip the value of p to the range [log(10 -6 )/10, log(10 6 )/10] to avoid extremely small and large values.

We consider a number of different policy regularizers both for penalty (choice C33) and constraint regularization (choice C34):

• Entropy H(π(•|s)) -it encourages the policy to try diverse actions [START_REF] Ziebart | Modeling purposeful adaptive behavior with the principle of maximum causal entropy[END_REF].

• KL(µ(•|s)||π(•|s)) -the Kullback-Leibler divergence between the behavioral and the current policy [START_REF] Schulman | Trust region policy optimization[END_REF] prevents the probability of taking a given action from decreasing too rapidly.

• KL(π(•|s)||µ(•|s)) -similar to the previous one but prevents too rapid increase of probabilities.

• While one could add any linear combination of the above terms to the loss, we have decided to only use a single regularizer in each experiment. Overall, all these combinations of regularization modes and different hyperparameters lead to the choices detailed in Table 1. We use multilayer perceptrons (MLPs) to represent policies and value functions. We either use separate networks for the policy and value function, or use a single network with two linear heads, one for the policy and one for the value function (choice C47). We consider different widths for the shared MLP (choice C48), the policy MLP (choice C49) and the value MLP (choice C50) as well as different depths for the shared MLP (choice C51), the policy MLP (choice C52) and the value MLP (choice C53). If we use the shared MLP, we further add a hyperparameter Baseline cost (shared) (C54) that rescales the contribution of the value loss to the full objective function. This is important in this case as the shared layers of the MLP affect the loss terms related to both the policy and the value function. We further consider different activation functions (choice C55) and different neural network initializers (choice C56). For the initialization of both the last layer in the policy MLP / the policy head (choice C57) and the last layer in the value MLP / the value head (choice C58), we further consider a hyperparameter that rescales the network weights of these layers after initialization.

B.8 Action distribution parameterization

A policy is a mapping from states to distributions of actions. In practice, a parametric distribution is chosen and the policy output is treated as the distribution parameters. The vast majority of RL applications in continuous control use a Gaussian distribution to represent the action distribution and this is also the approach we take.

This, however, still leaves a few decisions which need to be make in the implementation:

• Should the standard deviation of actions be a part of the network output (used e.g. in [START_REF] Haarnoja | Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor[END_REF]) or should it be independent of inputs like in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] (choice C59)? In the latter case, the standard deviation is still learnable but it is the same for each state.

• Gaussian distributions are parameterized with a mean and a standard deviation which has to be non-negative. What function should be used to transform network outputs which can be negative into the standard deviation (choice C60)? We consider exponentiation 25 (used e.g. in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]) and the softplus26 function (used e.g. in [START_REF] Janner | When to trust your model: Model-based policy optimization[END_REF]).

• What should be the initial standard deviation of the action distribution (choice C61)? We can control it by adding some fixed value to the input to the function computing the standard deviation (e.g. softplus).

• Should we add a small value to the standard deviation to avoid very low values (choice C62)?

• Most continuous control environments expect actions from a bounded range (often [-1, 1]) but the commonly used Gaussian distribution can produce values of an arbitrary magnitude. We consider two approaches to handle this (choice C63): The easiest solution is to just clip the action to the allowed range when sending it to the environment (used e.g. in [START_REF] Schulman | Proximal policy optimization algorithms[END_REF]). Another approach is to apply the tanh function to the distribution to bound the range of actions (used e.g. in [START_REF] Andrychowicz | Learning dexterous in-hand manipulation[END_REF]). This additional transformation changes the density of actions -if action u is parameterized as u = tanh(x), where x is a sample from a Gaussian distribution with probability density function p θ , than the density of u is log pu(u) = log p θ (x) -log tanh (x), where x = tanh -1 (u). This additional log tanh (x) term does not affect policy losses because they only use ∇ θ log pu(u) = ∇ θ log p θ (x). Similarly, this term does not affect the KL divergences which may be used for regularization (See Sec. B.6) because the KL divergence has a form of the difference of two log-probabilities on the same sample and the two log tanh (x) terms cancel out. 27 The only place where the log tanh (x) term affects the policy gradient computation and should be included is the entropy regularization as H(U ) = -Eu log pu(u) = Ex[-log p θ (x) + log tanh (x)]. This additional log tanh (x) term penalizies the policy for taking extreme actions which prevents tanh saturation and the loss of the gradient. Moreover, it prevents the action entropy from becoming unbounded.

To sum up, we parameterize the actions distribution as

Tu(N (xµ, Tρ(xρ + cρ) + ρ)),
where

• xµ is a part of the policy network output,

• xρ is either a part of the policy network output or a separate learnable parameter (one per action dimension),

• ρ (C62) is a hyperparameter controlling minimal standard deviation,

• Tρ (C60) is a standard deviation transformation (R → R ≥0 ), • Tu (C63) is an action transformation (R → [-1, 1]),
• cρ is a constant controlling the initial standard deviation and computed as cρ = T -1 ρ (iρρ) where iρ is the desired initial standard deviation (C61).

B.9 Data normalization and clipping

While it is not always mentioned in RL publications, many RL implementations perform different types of data normalization. In particular, we consider the following:

• Observation normalization (choice C64). If enabled, we keep the empirical mean oµ and standard deviation oρ of each observation coordinate (based on all observations seen so far) and normalize observations by subtracting the empirical mean and dividing by max(oρ, 10 -6 ). This results in all neural networks inputs having approximately zero mean and standard deviation equal to one. Moreover, we optionally clip the normalized observations to the range [-omax, omax] where omax is a hyperparameter (choice C65).

• Value function normalization (choice C66). Similarly to observations, we also maintain the empirical mean vµ and standard deviation vρ of value function targets (See Sec. B.2). The value function network predicts normalized targets ( V -vµ)/ max(vρ, 10 -6 ) and its outputs are denormalized accordingly to obtain predicted values: V = vµ + Vout max(vρ, 10 -6 ) where Vout is the value network output.

• Per minibatch advantage normalization (choice C67). We normalize advantages in each minibatch by subtracting their mean and dividing by their standard deviation for the policy loss.

• Gradient clipping (choice C68). We rescale the gradient before feeding it to the optimizer so that its L2 norm does not exceed the desired threshold.

C Default settings for experiments

Table 2 shows the default configuration used for all the experiments in this paper. We only list sub-choices that are active (e.g. we use the PPO loss so we do not list hyperparameters associated with different policy losses).

D Experiment Policy Losses

D.1 Design

For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• num_epochs (C3): {1, 3, 10}

• Policy loss (C14): {AWR, PG, PPO, RPA, V-MPO, V-Trace} -For the case "Policy loss (C14) = AWR", we further sampled the sub-choices: * AWR β (C18): {0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3} * AWR ωmax (C17): {1.1, 1.2, 1.3, 1.5} -For the case "Policy loss (C14) = PPO", we further sampled the sub-choices:

* PPO (C16): {0.1, 0.2, 0.3, 0.5} -For the case "Policy loss (C14) = V-MPO", we further sampled the sub-choices: * V-MPO n (C19): {0.0001, 0.0003, 0.001, 0.003, 0.01, 0.03, 0.1, 0.3, 1.0} -For the case "Policy loss (C14) = V-Trace", we further sampled the sub-choices: * V-Trace loss ρ (C15): {1.0, 1.2, 1.5, 2.0}

• Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001, 0.003}

All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

D.2 Results

We report aggregate statistics of the experiment in Table 3 as well as training curves in Figure 3. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 5678910111213.

E Experiment Networks architecture

E.1 Design

For each of the 5 environments, we sampled 4000 choice configurations where we sampled the following choices independently and uniformly from the following ranges: • Minimum standard deviation ρ (C62): {0.0, 0.01, 0.1}

• Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001}

• Activation (C55): {ELU, Leaky ReLU, ReLU, Sigmoid, Swish, Tanh}

All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

After running the experiment described above we noticed (Fig. 15) that separate policy and value function networks (C47) perform better and we have rerun the experiment with only this variant present.

E.2 Results

We report aggregate statistics of the experiment in Table 4 as well as training curves in Figure 16. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 1718192021222324252627282930.

F Experiment Normalization and clipping

F.1 Design

For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges: For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

• PPO (C16): {0.1, 0.

F.2 Results

We report aggregate statistics of the experiment in Table 5 as well as training curves in Figure 31. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 32333435363738.

G Experiment Advantage Estimation

G.1 Design

For each of the 5 environments, we sampled 4000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• num_envs (C1): {64, 128, 256}

• Value function loss (C11): {Huber, MSE} -For the case "Value function loss (C11) = Huber", we further sampled the sub-choices: * Huber delta (C12): {0.001, 0.01, 0.1, 1.0}

• PPO-style value clipping (C13): {0.001, 0.01, 0.1, 1.0, None}

• advantage_estimator (C6): {GAE, N-step, V-Trace} -For the case "advantage_estimator (C6) = GAE", we further sampled the sub-choices: * GAE λ (C8): {0.8, 0.9, 0.95, 0.99} -For the case "advantage_estimator (C6) = N-step", we further sampled the sub-choices:

* N-step N (C7): {1, 3, 10, 1000000} -For the case "advantage_estimator (C6) = V-Trace", we further sampled the sub-choices: * V-Trace advantage λ (C9): {0.8, 0.9, 0.95, 0.99, 1.0} * V-Trace advantage c, ρ (C10): {1.0, 1.2, 1.5, 2.0}

• Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001, 0.003}

All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

G.2 Results

We report aggregate statistics of the experiment in Table 6 as well as training curves in Figure 39. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 4041424344454647484950.

H Experiment Training setup

H.1 Design

For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• iteration_size (C2): {512, 1024, 2048, 4096}

• batch_mode (C5): {Fixed trajectories, Shuffle trajectories, Shuffle transitions, Shuffle transitions (recompute advantages)} • num_epochs (C3): {1, 3, 10}

• num_envs (C1): {64, 128, 256}

• Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001, 0.003}

• batch_size (C4): {64, 128, 256}

All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

H.2 Results

We report aggregate statistics of the experiment in Table 7 as well as training curves in Figure 51. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 52535455565758.

I Experiment Time

I.1 Design

For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• Discount factor γ (C20): {0.95, 0.97, 0.99, 0.999}

• Frame skip (C21): {1, 2, 5}

• Handle abandoned? (C22): {False, True}

• Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001}

All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

I.2 Results

We report aggregate statistics of the experiment in

K Experiment Regularizers

K.1 Design

For each of the 5 environments, we sampled 4000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• Regularization type (C32): {Constraint, No regularization, Penalty} -For the case "Regularization type (C32) = Constraint", we further sampled the sub-choices: * Regularizer (in case of constraint) (C34): {KL(µ||π), KL(π||µ), KL(ref||π), decoupled KL(µ||π), entropy} For the case "Regularizer (in case of constraint) (C34) = KL(µ||π)", we further sampled the sub-choices:

-Threshold for KL(µ||π) (C35): {0.005, 0.01, 0.02, 0.04, 0.08} For the case "Regularizer (in case of constraint) (C34) = KL(π||µ)", we further sampled the sub-choices:

-Threshold for KL(π||µ) (C36): {0.005, 0.01, 0.02, 0.04, 0.08} For the case "Regularizer (in case of constraint) (C34) = KL(ref||π)", we further sampled the sub-choices:

-Threshold for KL(ref||π) (C37): {10.0, 20.0, 40.0, 80.0, 160.0} For the case "Regularizer (in case of constraint) (C34) = decoupled KL(µ||π)", we further sampled the sub-choices:

-Threshold for mean in decoupled KL(µ||π) (C38): {0.005, 0.01, 0.02, 0.04, 0.08} -Threshold for std in decoupled KL(µ||π) (C39): {5e-05, 0.000125, 0.00025, 0.0005, 0.001, 0.002, 0.004} For the case "Regularizer (in case of constraint) (C34) = entropy", we further sampled the sub-choices:

-Threshold for entropy H(π) (C40): {0.0, -5.0, -10.0, -15.0} -For the case "Regularization type (C32) = Penalty", we further sampled the sub-choices: * Regularizer (in case of penalty) (C33): {KL(µ||π), KL(π||µ), KL(ref||π), decoupled KL(µ||π), entropy} For the case "Regularizer (in case of penalty) (C33) = KL(µ||π)", we further sampled the sub-choices:

-Regularizer coefficient for KL(µ||π) (C41): {0.003, 0.01, 0.03, 0.1, 0.3, 1.0} For the case "Regularizer (in case of penalty) (C33) = KL(π||µ)", we further sampled the sub-choices:

-Regularizer coefficient for KL(π||µ) (C42): {0.003, 0.01, 0.03, 0.1, 0.3, 1.0} For the case "Regularizer (in case of penalty) (C33) = KL(ref||π)", we further sampled the sub-choices:

-Regularizer coefficient for KL(ref||π) (C43): {3e-06, 1e-05, 3e-05, 0.0001, 0.0003, 0.001} For the case "Regularizer (in case of penalty) (C33) = decoupled KL(µ||π)", we further sampled the sub-choices:

-Regularizer coefficient for mean in decoupled KL(µ||π) (C44): {0.003, 0.01, 0.03, 0.1, 0.3, 1.0} -Regularizer coefficient for std in decoupled KL(µ||π) (C45): {0.1, 0.3, 1.0, 3.0, 10.0, 30.0, 100.0, 300.0} For the case "Regularizer (in case of penalty) (C33) = entropy", we further sampled the sub-choices:

-Regularizer coefficient for entropy (C46): {1e-05, 3e-05, 0.0001, 0.0003, 0.001, 0.003} • Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001, 0.003} All the other choices were set to the default values as described in Appendix C.

For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

K.2 Results

We report aggregate statistics of the experiment in Table 10 as well as training curves in Figure 75. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 7677787980818283848586878889909192

Figure 1 :

 1 Figure 1: Comparison of different policy losses (C14).

Figure 2 :

 2 Figure 2: Comparison of different initial standard deviations of actions (C61).

Fig. 17

 17 Fig.17compares two approaches to transform unbounded sampled actions into the bounded [-1, 1] domain expected by the environment (C63): clipping and applying a tanh function. tanh performs slightly better overall (in particular it improves the performance on HalfCheetah by 30% ). Comparing Fig.17and Fig.2suggests that the difference might be mostly caused by the decreased magnitude of initial actions9 .

  KL(ref(•|s)||π(•|s)) where ref is some reference distribution. We use ref = N (0, 1) in all experiments. This kind of regularization encourages the policy to try all possible actions. • Decoupled KL(µ(•|s)||π(•|s)). For Gaussian distributions we can split KL(µ(•|s)||π(•|s)) into a term which depends on the change in the mean of the distribution and another one which depends on the change in the standard deviation: KL(µ(•|s)||π(•|s)) = KL(µ(•|s)||ζ(•|s))+KL(ζ(•|s)||π(•|s)) where ζ(•|s) is a Gaussian distribution with same mean as µ(•|s) and the same standard deviation as π(•|s). Therefore, instead of using KL(µ(•|s)||π(•|s)) directly, we can use two separate regularizers, KL(µ(•|s)||ζ(•|s)) and KL(ζ(•|s)||π(•|s)), with different strengths. The soft constraint version of this regularizer in used in V-MPO 24 [34] with the threshold on KL(µ(•|s)||ζ(•|s)) being orders of magnitude lower than the one on KL(ζ(•|s)||π(•|s)).

Figure 3 :Figure 4 :

 34 Figure 3: Training curves.

Figure 5 :

 5 Figure 5: Comparison of 95th percentile of the performance of different policy losses conditioned on their hyperparameters.

Figure 6 :Figure 7 :

 67 Figure 6: Analysis of choice num_epochs (C3): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 8 :

 8 Figure 8: Analysis of choice V-Trace loss ρ (C15): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 9 :

 9 Figure 9: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 10 :

 10 Figure 10: Analysis of choice PPO (C16): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 12 :

 12 Figure 12: Analysis of choice AWR β (C18): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 13 :Figure 14 :

 1314 Figure 13: Analysis of choice AWR ω max (C17): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

•••

  Action transformation Tu (C63): {clip, tanh} • Last value layer scaling (C58): {0.001, 0.01, 0.1, 1.0} Global standard deviation? (C59): {False, True} • Last policy layer scaling (C57): {0.001, 0.01, 0.1, 1.0} Standard deviation transformation Tρ (C60): {exp, softplus} • Initial standard deviation iρ (C61): {0.1, 0.5, 1.0, 2.0}

Figure 15 :

 15 Figure 15: Analysis of choice Shared MLPs? (C47): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 16 :Figure 17 :

 1617 Figure 16: Training curves.

Figure 18 :

 18 Figure 18: Analysis of choice Policy MLP width (C49): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 19 :

 19 Figure 19: Analysis of choice Last value layer scaling (C58): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 20 :

 20 Figure 20: Analysis of choice Value MLP depth (C53): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 21 :

 21 Figure 21: Analysis of choice Value MLP width (C50): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 22 :

 22 Figure 22: Analysis of choice Policy MLP depth (C52): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 23 :

 23 Figure 23: Analysis of choice Global standard deviation? (C59): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 24 :

 24 Figure 24: Analysis of choice Last policy layer scaling (C57): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 25 :

 25 Figure 25: Analysis of choice Standard deviation transformation T ρ (C60): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 26 :

 26 Figure 26: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 27 :

 27 Figure 27: Analysis of choice Initializer (C56): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 28 :Figure 29 :

 2829 Figure 28: Analysis of choice Initial standard deviation i ρ (C61): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 30 :

 30 Figure 30: Analysis of choice Activation (C55): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

•

  2, 0.3, 0.5} • Input normalization (C64): {Average, None} -For the case "Input normalization (C64) = Average", we further sampled the sub-choices: * Input clipping (C65): {1.0, 2.0, 5.0, 10.0, None} • Gradient clipping (C68): {0.5, 1.0, 2.0, 5.0, None} • Per minibatch advantage normalization (C67): {False, True} • Adam learning rate (C24): {3e-05, 0.0001, 0.0003, 0.001} Value function normalization (C66): {Average, None} All the other choices were set to the default values as described in Appendix C.

Figure 31 :

 31 Figure 31: Training curves.

Figure 32 :

 32 Figure 32: Analysis of choice PPO (C16): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 33 :

 33 Figure 33: Analysis of choice Input normalization (C64): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 34 :

 34 Figure 34: Analysis of choice Gradient clipping (C68): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 35 :

 35 Figure 35: Analysis of choice Per minibatch advantage normalization (C67): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 36 :

 36 Figure 36: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 37 :

 37 Figure 37: Analysis of choice Value function normalization (C66): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 38 :

 38 Figure 38: Analysis of choice Input clipping (C65): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 39 :Figure 42 :

 3942 Figure 39: Training curves.

Figure 43 :

 43 Figure 43: Analysis of choice PPO-style value clipping (C13): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 44 :

 44 Figure 44: Analysis of choice advantage_estimator (C6): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 45 :

 45 Figure 45: Analysis of choice GAE λ (C8): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 46 :

 46 Figure 46: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 47 :Figure 48 :

 4748 Figure 47: Analysis of choice N-step N (C7): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 49 :

 49 Figure 49: Analysis of choice V-Trace advantage λ (C9): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 50 :

 50 Figure 50: Analysis of choice V-Trace advantage c, ρ (C10): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 51 :

 51 Figure 51: Training curves.

Figure 52 :

 52 Figure 52: Analysis of choice iteration_size (C2): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 58 :

 58 Figure 58: Analysis of choice batch_mode (C5): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations(right). In order to obtain narrower confidence intervals in this experiment we only sweep batch_mode (C5), num_envs (C1), Adam learning rate (C24).

Figure 59 :Figure 60 :Figure 62 :

 596062 Figure 59: Training curves.

Figure 63 :

 63 Figure 63: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 72 :

 72 Figure 72: Analysis of choice RMSProp (C29): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 73 :

 73 Figure 73: Analysis of choice RMSProp learning rate (C25): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 74 :

 74 Figure 74: 95th percentile of performance scores conditioned on RMSProp momentum (C27)(rows) and RMSProp learning rate (C25)(bars).

  .

Figure 78 :

 78 Figure 78: Analysis of choice Regularizer (in case of penalty) (C33): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 79 :

 79 Figure 79: Analysis of choice Regularizer coefficient for KL(ref||π) (C43): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Figure 80 :

 80 Figure 80: Analysis of choice Adam learning rate (C24): 95th percentile of performance scores conditioned on choice (left) and distribution of choices in top 5% of configurations (right).

Figure 81 :

 81 Figure 81: Analysis of choice Regularizer coefficient for entropy (C46): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

Table 1 :

 1 Choices pertaining to regularization.

	Choice Name
	C32	Regularization type
	C33	Regularizer (in case of penalty)
	C34	Regularizer (in case of constraint)
	C35	Threshold for KL(µ||π)
	C36	Threshold for KL(π||µ)
	C37	Threshold for KL(ref||π)
	C38	Threshold for mean in decoupled KL(µ||π)
	C39	Threshold for std in decoupled KL(µ||π)
	C40	Threshold for entropy H(π)
	C41	Regularizer coefficient for KL(µ||π)
	C42	Regularizer coefficient for KL(π||µ)
	C43	Regularizer coefficient for KL(ref||π)
	C44	Regularizer coefficient for mean in decoupled KL(µ||π)
	C45	Regularizer coefficient for std in decoupled KL(µ||π)
	C46	Regularizer coefficient for entropy
	B.7 Neural network architecture

Table 2 :

 2 Default settings used in experiments.

	Choice Name

Table 3 :

 3 Performance quantiles across choice configurations.

									Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
			90th percentile			1490			994			1103					1224		459
			95th percentile			1727			1080			1297					1630		565
			99th percentile			2290			1363			1621					2611		869
			Max						2862			2048			1901					3435		1351
	1500 2000 2500		Hopper-v1 Overall mean top 10% mean top 5% mean top 1% mean			3000 4000 5000		Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	1250 1500 1750 2000		Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean			1500 2000 2500		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	1000							2000			750 1000							1000						2000
	500							1000			250 500							0 500						0 1000
	0	0	200000	400000	600000	800000	1000000	0	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0	0	200000	400000	600000	800000	1000000	500	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

  Figure 11: Analysis of choice V-MPO n (C19): 95th percentile of performance scores conditioned on sub-choice (left) and distribution of sub-choices in top 5% of configurations (right).

	250 500 750 1000 1250 1500 1750	Hopper-v1	500 1000 1500 2000 2500 Humanoid-v1	200 400 600 800 1000	Walker2d-v1	250 500 750 1000 1250 1500 1750 HalfCheetah-v1	250 500 750 1000 1250 1500 1750 2000	Ant-v1	0.05 0.10 0.15 0.20 0.25 0.30	Hopper-v1	0.05 0.10 0.15 0.20 Humanoid-v1	0.05 0.10 0.15 0.20 Walker2d-v1	0.05 0.10 0.15 0.20 0.25 0.30 0.35 HalfCheetah-v1 0.05 0.10 0.15 0.20 0.25	Ant-v1	0.02 0.04 0.06 0.08 0.10 0.12 0.14 0.16	all
	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3 1.0
	200 400 600 800 1000 1200 1400 1600	Hopper-v1	250 500 750 1000 1250 1500 1750 Humanoid-v1	100 200 300 400 500 600 700 800	Walker2d-v1	200 400 600 800 1000 1200 1400 HalfCheetah-v1	500 1000 1500 2000 2500	Ant-v1	0.05 0.10 0.15 0.20 0.25 0.30	Hopper-v1	0.05 0.10 0.15 0.20 0.25 0.30 0.35 Humanoid-v1	0.05 0.10 0.15 0.20 0.25 0.30 0.35 Walker2d-v1	0.05 0.10 0.15 0.20 0.25 0.30 HalfCheetah-v1 0.05 0.10 0.15 0.20 0.25 0.30 0.35 Ant-v1
	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3	0.00	0.0001 0.0003 0.001 0.003 0.01 0.03 0.1 0.3

Table 4 :

 4 Performance quantiles across choice configurations.

							Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
		90th percentile		2098		1513			1133					1817		528
		95th percentile		2494		2120			1349					2382		637
		99th percentile		3138		3031			1582					3202		934
		Max					4112		4358			1875					3987		1265
	1500 2000 2500	Hopper-v1 Overall mean top 10% mean top 5% mean top 1% mean				Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	1500 2000	Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000 5000		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000 5000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	1000								1000					2000						2000
	500								500					1000						1000
	0								0					0						0
	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0	200000	400000	600000	800000	1000000	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

Table 5 :

 5 Performance quantiles across choice configurations.

							Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
		90th percentile		2058		1265			1533					1649		1143
		95th percentile		2287		1716			1662					2165		1564
		99th percentile		2662		2465			1809					3100		2031
		Max					3333		3515			2074					3482		2371
	2000 2500		Hopper-v1			4000 5000	Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	2500 3000 3500	Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	1500						3000		2000					2000
	0 500 1000			Overall mean top 10% mean top 5% mean top 1% mean	0 1000 2000		0 500 1000 1500					0 1000						0 1000 2000
	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0	200000	400000	600000	800000	1000000	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

Table 6 :

 6 Performance quantiles across choice configurations.

							Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
		90th percentile		1705		1128			1626					1922		947
		95th percentile		2114		1535			1777					2374		1185
		99th percentile		2781		2631			2001					3013		1697
		Max					3775		3613			2215					3564		2309
	2000 2500		Hopper-v1			4000 5000	Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	2000 2500 3000	Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean		3000 4000 5000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	0 500 1000 1500			top 1% mean top 5% mean top 10% mean Overall mean	0 1000 2000 3000		0 500 1000 1500					0 1000 2000						0 1000 2000
	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0	200000	400000	600000	800000	1000000	0	200000	400000	600000	800000	1000000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

Table 7 :

 7 Performance quantiles across choice configurations.

							Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
			90th percentile			2203			1316			1695					2310			1190
			95th percentile			2484			1673			1853					2655			1431
			99th percentile			2907			2665			2060					3014			1844
			Max				3563			3693			2434					3502			2426
	1500 2000 2500 3000			Hopper-v1	3000 4000 5000		Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	2000 2500 3000		Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean			2000 3000 4000		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean			3000 4000 5000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	0 500 1000	0	200000	400000	600000 Overall mean 800000 1000000 top 5% mean top 1% mean top 10% mean	0 1000 2000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0 500 1000 1500	0	200000	400000	600000	800000	1000000	0 1000	0	200000	400000	600000	800000	1000000	0 1000 2000	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

  Table 8 as well as training curves in Figure 59. For each of the investigated choices in this experiment, we further provide a per-choice analysis in Figures 60-63.

Table 8 :

 8 Performance quantiles across choice configurations.

									Ant-v1 HalfCheetah-v1 Hopper-v1 Humanoid-v1 Walker2d-v1
			90th percentile			1462			1063			1243					1431			761
			95th percentile			1654			1235			1675					2158			810
			99th percentile			2220			1423			2204					2769			974
			Max						2833			1918			2434					3106			1431
	2000 2500 3000 3500		Hopper-v1 Overall mean top 10% mean top 5% mean top 1% mean			3000 4000 5000		Humanoid-v1 Overall mean top 10% mean top 5% mean top 1% mean	1500 2000 2500		Walker2d-v1 Overall mean top 10% mean top 5% mean top 1% mean		1500 2000 2500		HalfCheetah-v1 Overall mean top 10% mean top 5% mean top 1% mean			3000 4000		Ant-v1 Overall mean top 10% mean top 5% mean top 1% mean
	1500							2000			1000						1000							2000
	500 1000							1000			500						0 500							1000
	0	0	200000	400000	600000	800000	1000000	0	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000	0	0	200000	400000	600000	800000	1000000	0	200000	400000	600000	800000	1000000	0	0	250000 500000 750000 1000000 1250000 1500000 1750000 2000000

It has been noticed that the version of the Mujoco physics simulator[START_REF] Todorov | Mujoco: A physics engine for model-based control[END_REF] can slightly influence the behaviour of some of the environmentshttps://github.com/openai/gym/issues/1541. We used Mujoco

2.0 in our experiments.

We used the V-MPO policy loss without the decoupled KL constraint as we investigate the effects of different policy regularizers separately in Sec.3.8.

AWR loss has two hyperparameters -the temperature β (C18) and the weight clipping coefficient ωmax (C17). We only condition on β which is more important.

These two tasks were not included in the original PPO paper[START_REF] Schulman | Proximal policy optimization algorithms[END_REF] so the hyperparameters we use were not tuned for them.

All environments expect normalized actions in [-1, 1].

tanh can also potentially perform better with entropy regularization (not used in this experiment) as it bounds the maximum possible policy entropy.

We noticed that some of the training runs with exponentiation resulted in NaNs but clipping the exponent solves this issue (See Sec. B.8 for the details).

Another explanation could be the interaction between the value function normalization and PPO-style value clipping (C13). We have, however, disable the value clipping in this experiment to avoid this interaction. The disabling of the value clipping could also explain why our conclusions are different from[START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF] where a form of value normalization improved the performance on Walker.

We only applied clipping if input normalization was enabled.

This is consistent with prior work[START_REF] Engstrom | Implementation Matters in Deep RL: A Case Study on PPO and TRPO[END_REF].

While the discount factor is sometimes treated as a part of the environment, we assume that the real goal is to maximize undiscounted returns and the discount factor is a part of the algorithm which makes learning easier.

Importantly, switching from no momentum to momentum 0.9 increases the RMSprop step size by approximately 10× and requires an appropriate adjustment to the learning rate (Fig.74).

https://github.com/openai/baselines/tree/master/baselines/ppo2

[P ] denotes the Iverson bracket, i.e. [P ] = 1 if P is true and [P ] = 0 otherwise.

For → 0, we have β → ∞ which results in softmax(A µ t /β) → 1.

While not too common in continuous control, this technique is standard in RL for Atari[START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF].

RMSProp was proposed by Geoffrey Hinton in one of his Coursera lectures: http://www.cs.toronto. edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf

The current arXiv version of the V-MPO paper[START_REF] Francis | V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control[END_REF] incorrectly uses the standard deviation of the old policy instead of the new one in the definition of KL(µ(•|s)||π(•|s)) which leads to a slightly different decomposition. We do not expect this to make any difference in practice.

For numerical stability, we clip the exponent to the range[-15, 15]. Notice that due to clipping this function has zero derivative outside of the range [-15, 15] which is undesirable. Therefore, we use a "custom gradient" for the clipping function, namely we assume that it has derivative equal 1 everywhere.

softplus(x) = log(e x + 1)

KL(U1, U2) = Eu←U 1 log U1(u) -log U2(u) = Ex←X 1 (log X1(x) -log tanh (x)) -(log X2(x)log tanh (x)) = Ex←X 1 log X1(x) -log X2(x) = KL(X1||X2).

For each of the 5 environments, we sampled 2000 choice configurations where we sampled the following choices independently and uniformly from the following ranges:

• Learning rate decay (C31): {0.0, 1.0} For each of the sampled choice configurations, we train 3 agents with different random seeds and compute the performance metric as described in Section 2.

J.2 Results

We report aggregate statistics of the experiment in