Paul Daoudi
email: paul.daoudi1@huawei.com

Dos Ludovic
email: ludovic.dos.santos@huawei.com

Merwan Santos

Aladin Barlier
email: merwan.barlier@huawei.com

Virmaux

DENSITY ESTIMATION FOR CONSERVATIVE Q-LEARNING

Batch Reinforcement Learning algorithms aim at learning the best policy from a batch of data without interacting with the environment. Within this setting, one difficulty is to correctly assess the value of state-action pairs far from the data set. Indeed, the lack of information may provoke an overestimation of the value function, leading to non-desirable behaviours. A compromise between enhancing the performance of the behaviour policy and staying close to it must be found. To alleviate this issue, most existing approaches introduce a regularization term to favor state-action pairs from the data set. In this paper, we refine this idea by estimating the density of these state-action pairs to distinguish neighbourhoods. The resulting regularization guides the policy toward meaningful unseen regions, improving the learning process. We hence introduce Density Conservative Q-Learning (D-CQL), a sound batch RL algorithm that carefully penalizes the value function based on the information collected in the state-action space. The performance of our approach is outlined on many classical benchmark in batch RL.

INTRODUCTION

Transposing recent successes of Reinforcement Learning (RL) from domains such as recommendation systems [START_REF] Rojanavasu | New recommendation system using reinforcement learning[END_REF][START_REF] Zheng | DRN: A deep reinforcement learning framework for news recommendation[END_REF], video games [START_REF] Mnih | Playing atari with deep reinforcement learning[END_REF], go [START_REF] Silver | Mastering the game of go without human knowledge[END_REF], to real-world systems is not possible without facing many challenges [START_REF] Dulac-Arnold | Challenges of real-world reinforcement learning: definitions, benchmarks and analysis[END_REF]. One of those being that in many real-world applications, direct access to the system is limited or even forbidden. A learning controller may incorrectly assess the implications of its actions and damage the system. Batch (or offline) Reinforcement Learning [START_REF] Lange | Batch reinforcement learning[END_REF][START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF] provides a framework to address those RL problems when no interaction with the system is allowed. In place, the learner is given a data set collected under a (possibly unknown) behavioural policy and has to derive the most efficient policy out of this data set.

One of the main issue in this setting is known as the value-function over-estimation problem [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF]Kumar et al., 2019a;[START_REF] Wu | Behavior regularized offline reinforcement learning[END_REF]. When the data set covers only a small subset of the state-action space, the agent typically wrongly extrapolates the value function related to out-of-distribution (OOD) pairs far from the data set. This incorrect value estimate is then used as a target in the learning process, leading to highly increasing estimates which may lead to a disastrous learned policy. This problem has been extensively studied in the traditional online Reinforcement Learning setting [START_REF] Sutton | Reinforcement learning -an introduction[END_REF]. However in this case, this issue is quite naturally alleviated: when a value function becomes high whether because the region is promising or because of an erroneous over-optimism, it will drive the agent to visit the related state-action pair. Therefore, the agent will have the chance to directly check the consequences of such pairs and its estimation can be corrected if needed. Incorporating uncertainty with Ensemble Learning (van Hasselt, 2010;[START_REF] Hado Van Hasselt | Deep reinforcement learning with double qlearning[END_REF][START_REF] Anschel | Averaged-dqn: Variance reduction and stabilization for deep reinforcement learning[END_REF] can also enhance learning.

In the offline case, since no inspection can be done to investigate the accuracy of the estimates, those methods cannot be used, and hence classical deep off-policy methods may dramatically fail [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF]. Additional approaches must thus be investigated to build robust and efficient agents. An important family of state-of-the-art algorithms addressing this problem focus on constraining the learned policy to stay close to the data set either by minimizing its distance to the behavioural policy (Siegel et al., 2020a;[START_REF] Wu | Behavior regularized offline reinforcement learning[END_REF]Kumar et al., 2019b) or by penalizing unseen state-action pairs [START_REF] Luo | Learning self-correctable policies and value functions from demonstrations with negative sampling[END_REF][START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF][START_REF] Yu | Combo: Conservative offline model-based policy optimization[END_REF]. In particular, [START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF] propose to use a lower bound on the value function as a proxy during the policy optimization process. This bound should be tight when state-action pairs are contained in the data set and loosened otherwise. Following this direction, they proposed Conservative Q-Learning (CQL) that introduces a penalization on the value functions associated to OOD actions. Nevertheless, given the lack of information regarding the behavioural policy, their regularization remains abrupt and might be problematic in practice. First, the resulting lower bound is loosened with the recommended distributions. Second, their approach does not consider neighbourhoods of actions and could therefore over-penalize interesting regions of the action space. This problem is particularly relevant when the data set has been gathered with a behavioural policy with medium performances: actions close to the data set distribution leading to better results are very likely to appear in the optimization process and they must not be too heavily penalized.

In this paper, we tackle these problems by introducing a novel penalization based on an estimation of the probability density function of the data set instead of the behavioural policy. Besides being easier to learn than the behavioural policy, this density is able to provide information on which regions are near the data set and thus safe to learn from, and which are the ones far from it and prone to over-estimation errors. We show how to integrate this penalization to refine CQL, leading to our proposed algorithm namely Density Conservative Q-Learning (D-CQL). On top of instigating those relevant information, we show that this new regularization leads to a more appropriate lower bound on the value function. We finally investigate empirically the relevance of our approach.

PRELIMINARIES AND MOTIVATIONS

NOTATIONS AND POLICY ITERATION

The agent-environment framework is modeled as a Markov Decision Process (MDP) (S, A, r, P, γ) defined by a state space S, an action space A, a transition kernel P : S × A → ∆(S), a reward function r : S × A → [R min , R max ] and a discount factor γ ∈]0, 1[. A policy π : S → ∆(A) is a decision rule mapping a state over a distribution of actions. The value of a policy is measured through the value function

V π (s) = E P [ t=0 γ t r(s t , a t ) | s 0 = s, a t ∼ π(•|s t )] and its associated Q-value function Q π (s, a) = E P [ t=0 γ t r(s t , a t ) | s 0 = s, a 0 = a, a t ∼ π(•|s t ) ∀t ≥ 1].
The goal is to find the policy maximizing these value functions.

Let r, Q be matrices associated to all state-action pairs and P π be the transition matrix induced by the policy π: P π (s, s ) = E a∼π(•|s) [P (s |s, a)]. Value functions can be learned by iterating the Bellman operator defined for the Q-function as , a )]. This operator is a γ-contraction [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF], hence having a unique fixed point Q π . Many algorithms rely on the Policy Iteration scheme [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF][START_REF] Fujimoto | Addressing function approximation error in actor-critic methods[END_REF][START_REF] Haarnoja | Soft actor-critic algorithms and applications[END_REF], where the agent alternates between Policy Evaluation (PE) with the computation of Q π and Policy Improvement (PI) by maximizing the learned Q-values:

B π Q = r + γP π Q where P π Q(s, a) = E s ∼P (•|s,a),a ∼π(•|s) [Q(s
Q k+1 ← arg min Q B π k Q k -Q, ( policy
evaluation)

π k+1 ← arg max π E a∼π Q k+1 (•, a) . (policy improvement)
This iterative process converges towards the optimal policy [START_REF] Sutton | Reinforcement learning -an introduction[END_REF][START_REF] Santos | Convergence properties of policy iteration[END_REF]. Since no knowledge is assumed on the environment, these steps are commonly solved using samples from a data set D = {(s i , a i , r i ) N i=1 }. The expectation under P (•|s, a) is now estimated using samples and leads to the empirical Bellman operator B. Actions that belong to the data set distribution given a state s are denoted as in-distribution (ID) and the ones far from it are out-ofdistribution (OOD). We slightly abuse notations and consider D is also the distribution on S ×A×S induced by the data set and propose:

Q k+1 ← arg min Q E s,a,s ∼D Bπ k Q k (s, a) -Q(s, a) 2 ,
(approximate policy evaluation)

π k+1 ← arg max π E s∼D,a∼π(•|s) Q k+1 (s, a) .
(approximate policy improvement)

The Q-values and the policy π are commonly estimated using Neural Networks that are trained with gradient optimization methods. In the approximate policy evaluation step, an important aspect is the expectation of actions from π(•|s ) in the bootstrapped target. This is where OOD actions may appear: extrapolation errors emerge then are back-propagated to eventually lead to highly overoptimistic estimates. Note that there are no OOD states in the Bellman update as the empirical operator only depends on a seen state s . They might appear in model-based algorithms [START_REF] Yu | Combo: Conservative offline model-based policy optimization[END_REF] but are, for now, out of the scope of model-free algorithms.

In this work, we focus on the batch setting where the agent cannot interact with the environment. A fixed data set has been gathered with an unknown behavioural policy π β . No knowledge is assumed on π β as the data set can come from different sources: optimal control, human or a mixture of them [START_REF] Fu | D4rl: Datasets for deep data-driven reinforcement learning[END_REF][START_REF] ¸aglar Gülc ¸ehre | RL unplugged: A collection of benchmarks for offline reinforcement learning[END_REF]. A major challenge is to find a good trade-off between staying close to the data set to avoid extrapolation errors and taking some liberty to overcome the suboptimality of the behaviour policy.

CONSERVATIVE Q-LEARNING

CQL addresses the problem of Q-value overestimation exacerbated in batch RL [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF][START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF], caused by the back-propagation of a possibly incorrect bootstrapped target in the policy evaluation step. The error associated to the the target may be important when the consequences of a given state-action pair are unknown, and could lead to too optimistic estimates.

In batch RL, the data set often describes a small subset of the state-action space thus OOD actions are likely to appear. Besides, the agent may never get the chance to visit the state-action pairs related to high (learned) Q-values and cannot correct its wrong estimations. [START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF] propose to penalize the Q-values associated to actions not described by the data set while keeping intact actions from the data set. This greatly stabilizes learning: the Q-estimates no longer take drastically high values, and it implicitly drives the agent to favor state-action pairs described in the data set. This penalization translates into minimizing the Q-values on an arbitrary distribution µ (e.g. uniform) and maximizing them on π β formally expressed as:

Qk+1 ← arg min Q α E s∼D E a∼µ(•|s) [Q(s, a)] -E a∼π β (•|s) [Q(s, a)] + E s,a,s ∼D Bπ k Qk (s, a) -Q(s, a) 2 .
(1)

Assuming no sampling errors and that supp µ ⊂ supp π β , this update provides a lower-bound on the expected Q-values on the distribution µ:

E a∼µ Q CQL (s, a) ≤ E a∼µ [Q π (s, a)] for all s ∈ S with α > 0.
When µ is the current policy π, the lower bound is in the value function, that is V CQL (s) ≤ V π (s). This property may be desirable on applications where we must check if the current value function is above a certain threshold such as Constrained Policy Optimization [START_REF] Achiam | Constrained policy optimization[END_REF] and Conservative Exploration [START_REF] Garcelon | Conservative exploration in reinforcement learning[END_REF]. However, for the common setting where µ is a uniform distribution over the action space, the lower bound is loosened to a uniform expectation, and the inequality in the value function no longer holds.

Another point of attention is the maximization term relying on the knowledge of the behavioural policy in equation 1. In the general setting, it is out of reach as the data set may be gathered in various ways. [START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF] propose to use the empirical data set distribution πβ (a|s) =

s ,a ∈D 1[s =s,a =a] s ∈D 1[s =s]
, a sum of Dirac over the data set actions. Not only does this further breaks the assumption supp µ ⊂ supp πβ (as in practice supp π β ⊂ supp πβ ), it also might be problematic on the penalization itself: the distance between the selected actions and the data set distribution is no longer considered explicitly. Actions not appearing in the data set will be equally penalized no matter their distance with D. Thus, the regularization drives the agent to stay close to the behavioural policy. We believe having a smoother regularization that considers which neighbourhoods of the data set are relevant is important to get the best agent as possible.

A NEW WEIGHTING SCHEME ON THE OUT-OF-DISTRIBUTION ACTIONS

The penalization of CQL pushes the learned policy of the agent to stay very close to the behavioural policy as it penalizes any action that does not belong to the data set and thereby prevents the agent from investigating on potential relevant areas. This problem is of high interest, for example, when the data set is gathered with a distinctly sub-optimal policy but more efficient actions exist in the neighborhood of the sub-optimal ones. The chances of finding a better policy would be reduced as any action that does not belong to the data set will be equally penalized. Nevertheless, in view of the fixed nature of the batch setting and the high sensibility of the Bellman update, we should be cautious with respect to which regions the agent should focus on.

Bearing these complications in mind, we propose to refine the original regularization by appropriately weighting the actions according to how uncertain their consequences are given D. To do so, we quantify the distance between the data set and the (sampled) penalized actions to soften the regularization. We propose a regularization based on the joined density of state-action pairs in the data set, denoted ρ β : S × A → R + . This density describes how likely a state-action pair belongs to the data set and, by extension, how uncertain its associated Q-function is. Various techniques can be used to learn such a distribution. In fact, the literature regarding density estimation is quite abundant [START_REF] Bishop | Pattern recognition and machine learning, 5th Edition. Information science and statistics[END_REF][START_REF] Uria | Rnade: The real-valued neural autoregressive densityestimator[END_REF][START_REF] Kobyzev | Normalizing flows: An introduction and review of current methods[END_REF], even when dealing with high dimensional inputs such as images [START_REF] Winkler | Learning likelihoods with conditional normalizing flows[END_REF][START_REF] Papamakarios | Normalizing flows for probabilistic modeling and inference[END_REF]. Our method can also benefit from future developments in this area.

Using this density, we build a weighting scheme ζ ν (a|s)1 based on how uncertain the consequences of an action a, given s, are w.r.t. D. It is integrated as follows:

Qk+1 ← arg min Q α E s∼D,a∼µ(•|s) [ζ ν (a|s)Q(s, a)] + E s,a,s ∼D Bπ k Q k (s, a) -Q(s, a) 2 .
(2)

The Theorem 1 For any µ satisfying supp µ ⊂ supp π β and assuming the absence of sampling errors, the Q-values obtained by iterating equation 2 are

∀s ∈ D, a ∈ supp π β (•|s), Q D-CQL (s, a) = Q π (s, a) -α (I -γP π ) -1 µ ζ ν π β (a|s) , (3)
and α > 0 leads to a point-wise lower bound on the true Q-values that becomes tight whenever ζ ν (a|s) = 0.

The proof of this theorem follows closely the one in (Kumar et al., 2020, Theorem 3.1), but we still include it in Appendix A for completeness.

CHOICE OF THE WEIGHTING SCHEME

The weighting scheme should capture the uncertainty of an action a given a state s. With a D the action present in the data set related to the state s, we propose the following indicator:

ζ ν (a|s) = max ν , 1 - ρ β (s, a) ρ β (s, a D ) . ( 4
)
Notice that the weighting scheme contains a few tricks on top of the density information. First, because the density may take high values in some areas, especially when dealing with narrow distributions, we normalize the weighting scheme by ρ β (s, a D ). The state-action pair (s, a D ) belongs to the data set, so its associated density is able to scale the weighting scheme well. It also frees its dependency on the state s. Second, considering the propensity of the Q-values to take extremely high values rapidly, the weighting scheme must be below 1. Last but not least, making a too harsh distinction between ID and OOD actions may result in an unstable learning, hence the introduction of the hyper-parameter ν.

Hyper-parameter ν We need ν = 0 for the lower bound of Theorem 1 to be tight. Indeed, when ν = 0, the Q-values of ID actions are not penalized as ζ ν = 0. It follows the intuition of penalizing only Q-values associated to OOD actions. However, since OOD actions do not appear in the Bellman loss term of Equation ( 2), the minimum of their Q-values is unbounded which eventually affects the performances of the algorithm. By setting ν > 0, the regularization term will also include ID actions and this minimization will be counterbalanced by the Bellman loss that tends to maximize their Qvalues. In other words, the lower ν, the greater the difference between OOD and ID Q-values to balance the objective. In practice, it is common to have knowledge on the quality of the behavioural policy and on the system, which can help setting the right ν. For instance, if we know that an expert has gathered the data set, we will want to stay close to it and set ν close to 0. Otherwise, if we know that the exploration has not been good and that there is no risk of harming the system, we can choose a bigger ν to allow the controller to consider uncertain actions. In Appendix C.2, we conduct a detailed analysis of the effects of the OOD penalization with varying ν.

PRACTICAL ALGORITHM

Following the ideas expressed in [START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF], Appendix A), we add an entropic regularization to Equation (2) that leads to the following update for Density Conservative Q-Learning (D-CQL):

(5)

Qk+1 ← arg min Q α E s∼D log a∈A exp (ζ ν (a|s)Q(s, a)) + 1 2 E s,a,s ∼D B π k Qk (s, a) -Q(s, a) 2 .
The log-sum-exp operator can be seen as a soft differential maximum operator. Equation ( 5) only penalizes the highest Q-values weighted by ζ ν . This expression exacerbate D-CQL's behaviour described in Section 3.2 when ν is close to zero. Same conclusion holds, the lower ν, the greater the difference between OOD and ID Q-values to balance the objective. In practice, this operator is computed using Importance Sampling (IS) using M action samples from both the current policy and the uniform distribution U(A). We describe D-CQL in Algorithm 1.

RELATED WORKS

Learning a policy from a fixed data set has been of high interest in Reinforcement Learning. A first way to address this problem is using Behavioural Cloning. When the data set is complete and comes from an expert policy, it may achieve great successes as shown in special cases of autonomous driving (Pomerleau, 1988) and flying (Sammut et al., 1992). However, when the agent finds itself in a situation not described by the data set, it may choose catastrophic actions and its performance can quickly degrade [START_REF] Codevilla | Exploring the limitations of behavior cloning for autonomous driving[END_REF]. Imitation Learning [START_REF] Hussein | Imitation learning: A survey of learning methods[END_REF][START_REF] Ho | Generative adversarial imitation learning[END_REF] aims to counter this problematic but still requires an expert policy. However, when the agent finds itself in a situation not described by the data set, it may choose catastrophic actions and its performance can quickly degrade [START_REF] Codevilla | Exploring the limitations of behavior cloning for autonomous driving[END_REF]. Imitation Learning [START_REF] Hussein | Imitation learning: A survey of learning methods[END_REF][START_REF] Ho | Generative adversarial imitation learning[END_REF] aims to counter this problematic but still requires an expert policy. In realworld applications, the data set may come from sub-optimal policies and instead of reproducing the behavioural policy, the goal of Batch RL is to extract a better one. Recently, many algorithms have been introduced, all relying on making sure the learned policy stays close to the behavioural one, avoiding the presence of Out-Of-Distribution (OOD) actions [START_REF] Levine | Offline reinforcement learning: Tutorial, review, and perspectives on open problems[END_REF]. It can take several forms.

Many recent algorithms cope with the distributional shift by directly avoiding the selection of OOD actions by the policy, preventing the agent to learn from highly extrapolated values. BCQ [START_REF] Fujimoto | Off-policy deep reinforcement learning without exploration[END_REF] explicitly parametrize the policy to stay within a parametric ball around the behavioural policy. Similarly, SPIBB [START_REF] Laroche | Safe policy improvement with baseline bootstrapping[END_REF] restricts the support of the learned policy to remain in the support of the behavioural policy. Other methods -such as BRAC (Kumar et al., 2019a) and BEAR [START_REF] Wu | Behavior regularized offline reinforcement learning[END_REF] -modify the RL objective with a measure of closeness between learned and behavioural policy. AWR [START_REF] Peng | Advantage-weighted regression: Simple and scalable off-policy reinforcement learning[END_REF] or ABM (Siegel et al., 2020b) build a Trust Region around the behavioural policy. Rather than a constraint between the policies, AlgaeDICE [START_REF] Nachum | Algaedice: Policy gradient from arbitrary experience[END_REF] and OptiDICE [START_REF] Lee | Optidice: Offline policy optimization via stationary distribution correction estimation[END_REF] focused on the state-action stationary distributions which provided competitive results. While successful on a variety of tasks, they do not explicitly counter the over-estimation problem that may prevent learning [START_REF] Kostrikov | Offline reinforcement learning with fisher divergence critic regularization[END_REF]. Another line of work is to enhance the robustness of the agent while leaving the objective unchanged. REM [START_REF] Agarwal | An optimistic perspective on offline reinforcement learning[END_REF] chooses to not modify the objectives, but uses Ensemble Learning to build a mixture of targets stabilizing the updates. In practice, it rarely competes with the above methods.

More related to our work is [START_REF] Dadashi | Offline reinforcement learning with pseudometric learning[END_REF][START_REF] Rezaeifar | Offline reinforcement learning as anti-exploration[END_REF] where a clever pessimistic bonus is introduced in the rewards acting as anti-exploration. It represents the distance of the selected action with the one who would be selected by the behavioural policy. The authors showed it acted similarly than a KL-penalty between the learned and behavioural policy, linking their work with [START_REF] Wu | Behavior regularized offline reinforcement learning[END_REF] and Kumar et al. (2019a). However, playing with the rewards might lead to a completely modified goal (Ng et al., 1999) and finding an appropriate reward modification might be complicated [START_REF] Harutyunyan | Expressing arbitrary reward functions as potential-based advice[END_REF]. We argue that modifying the Q-values lead to a better policy control as the whole objective is considered. In this line of search, [START_REF] Kostrikov | Offline reinforcement learning with fisher divergence critic regularization[END_REF] augmented the Q-values with the entropy of the behavioural policy and constrained them with as Fisher divergence term. In contrast, rather than modifying the RL objective with an explicit penalty on the data set distance, we use the density as an uncertainty surrogate in the value function estimation.

Note our data set density estimation can be combined with most presented methods. Especially, it could replace the "anti-exploration" bonus or be introduced to improve the Behavioural Cloning step in [START_REF] Wu | Behavior regularized offline reinforcement learning[END_REF] required to build the measures of closeness between policies.

EXPERIMENTS

In this section, we evaluate D-CQL on various settings. Our algorithm was designed to take the most of data sets collected with a distinctly sub-optimal policy, where the trade-off between staying close to the behavioural policy and discovering better actions is culminating. An ablation study on ν and on the density estimation technique used can be found in Appendix C.2.

DATA SETS

Our algorithm is first tested on four different data sets gathered on two classic control environments with continuous actions: Pendulum and MountainCarContinuous [START_REF] William | Efficient memory-based learning for robot control[END_REF]. For each environment, two data sets were gathered with a bad and medium policy, where the policies have been created by running a Soft-Actor-Critic agent [START_REF] Haarnoja | Soft actor-critic algorithms and applications[END_REF] and early-stopping the training.

Pendulum is an inverted pendulum swing up problem, where the reward is a smooth function contained in [-16, 0]. The goal is to is to swing it up so it stays upright for 200 time-steps. The expert policy induces an average return of -150, and the bad and medium policies respectively induce an average return of -1000 and -700. The data set possesses 20 thousand transitions. Mountain-CarContinuous is also a classic environment in Reinforcement Learning. A car must drive up a hill, but must first drive back and forth to be able to climb it. Unlike Pendulum, the reward is sporadic: it takes the value of +100 when the goal is reached, and -energy else. The bad data set is composed of 1 success out of 100 trajectories, and the medium data set 4 successes out of 50. We have found both CQL and D-CQL easily solve this environment when the number of successes is higher. The expert policy induces an average return of 95.

Finally, we test our agents on the medium data sets from D4RL which is closer to real-world performances. To further match a realistic setting we only consider a reduce proportion, X%, of the data sets. Indeed, huge data collection might not be that realistic in practice (e.g. low acquisition frequency) and learning on such huge data sets can be problematic w.r.t. computation time.

EXPERIMENTAL PROTOCOL

Learning the density D-CQL relies on an accurate density estimation step. Though any technique can be used, we propose to use a highly expressive method: Normalizing Flows (Rezende & Mohamed, 2015; [START_REF] Kobyzev | Normalizing flows: An introduction and review of current methods[END_REF]. This choice was fueled with their successes on high dimensional distributions [START_REF] Papamakarios | Normalizing flows for probabilistic modeling and inference[END_REF] and their elegance. Indeed, they are known to generalize and scale well from complex distributions [START_REF] Winkler | Learning likelihoods with conditional normalizing flows[END_REF] -including images [START_REF] Kobyzev | Normalizing flows: An introduction and review of current methods[END_REF] -while keeping a tractable learning. We optimize the model hyper-parameters with a grid search where each data set was split into a train and a test set. The density estimator was then trained on the whole data set using the best selected hyper-parameters. A comparison of different density estimation algorithms used for D-CQL can be found in Appendix C.2.

Learning the agents We compare our agents with CQL2 , a state-of-the-art method that also penalizes OOD actions in the approximate policy evaluation step. Experiments on the classic control data sets have been done with a grid search on all the hyper-parameters, including experiments with CQL. Considering the significant length of the experiments on D4RL, the hyper-parameter search was done on 1 million (M) gradient steps. Additional details are reported in Appendix C.1.

Metric We compare the agents' performances using the normalized average return (NAR): normalized score = 100 * scorerandom score expert scorerandom score However, it is common knowledge that these methods are unstable and a decrease in performance during learning (Aviral Kumar, 2021;[START_REF] Kumar | Implicit under-parameterization inhibits data-efficient deep reinforcement learning[END_REF]. We thus propose to evaluate the best agents after 10M gradient steps. Thus, once the best hyper-parameters selected on 1M gradient steps, we further continue the learning for 9M steps for the best agent. Regarding classic control data sets, we observed that 100k updates were sufficient to draw a relevant comparison. We report the overall best normalized average return, i.e. the best normalized average return considering all epochs, and the best final normalized average return in Table 1 andTable 2.

RESULTS

Results on the classic control data sets are shown in Table 1. We observe our refined regularization improve the agent's performance on both the overall and final metrics on 3 out of 4 tasks. The agent was able to take the most out of the neighbourhood of the data set and extract more efficient policies. Notice the greater variance on the MountainCarContinuous environment, explained by the sparsity of its reward function. In any case, D-CQL was able to better take advantage of the information provided by the data set.

M O U N T A I N C A R C-B A D
BEST OVERALL 40 ± 17.7 59.5 ± 36.6 48.7 BEST FINAL 34.9 ± 38.3 47 ± 29.1 34.9

M O U N T A I N C A R C-M E D I U M BEST OVERALL 92 ± 0.9 92 ± 0.3 0 BEST FINAL 60.3 ± 23.4 59.4 ± 27.4 -1.5 TO T A L ME A N BEST OVERALL 15.6 BEST FINAL 15.3
Regarding the more complex D4RL's data sets, the results are reported in Table 2. Our agent was able to beat CQL respectively 5 and 6 times out of 9 w.r.t. the overall and final metric. On average, it D-CQL outperforms CQL by 3% and 7.5% on the overall and final metric.

Table 2: Normalized scores between CQL and D-CQL on the D4RL MuJoCo medium data sets.

DATA SET METRIC (10M) CQL D-CQL IMPROVEMENT (%) H A L F C H E E T A H-M E D I U M 10% BEST OVERALL 14.7 ± 3.2 14.4 ± 3.8 -2 BEST FINAL 10.3 ± 4.7 12.6 ± 4.5 22.3 W A L K E R2D-M E D I U M 10% BEST OVERALL 48.4 ± 8.4 50.4 ± 3.5 4.1 BEST FINAL 44 ± 5.4 48.2 ± 6.7 9.5 H O P P E R-M E D I U M 10%
BEST OVERALL 81.5 ± 9.3 80.9 ± 7.7 -0.7 BEST FINAL 43.7 ± 9.3 37.2 ± 13.9 -14.9

H A L F C H E E T A H-M E D I U M 5%
BEST OVERALL 15 ± 3.9 15.8 ± 3.5 5.3 BEST FINAL 9.4 ± 5.5 11.1 ± 4.1 18.1

W A L K E R2D-M E D I U M 5%
BEST OVERALL 40.5 ± 6.6 37.4 ± 6.3 -7.7 BEST FINAL 33.5 ± 8.4 31.6 ± 6.4 -5.4

H O P P E R-M E D I U M 5%
BEST OVERALL 58 ± 13 67.1 ± 8 15.7 BEST FINAL 43.6 ± 14.9 48.7 ± 13.6 11.7

H A L F C H E E T A H-M E D I U M 1%

CONCLUSION

In this paper we propose to introduce the use of the data set density ρ β which provides insights to guide the agent towards meaningful areas, inducing better performances in practice. This work focused on model-free agents that do not consider OOD states. However, the density ρ β also yields information on OOD states which can be integrated on model-based algorithms.

In addition, our experiments highlighted two other points of interests: the decreasing effect and the relatively high variance of the batch agents. Despite some previous works aiming to cope with these problems, a complete understanding of these phenomenons is lacking which is crucial before batch RL can be applied on real-world systems.

A LOWER BOUND DETAILS

In this section, we re-state our main theorem and prove it. An additional insight is provided to handle the sampling errors.

Theorem 2 For any µ satisfying supp µ ⊂ supp π β and assuming there are no sampling errors, then the Q-values obtained by iterating equation 2 are

∀s ∈ D, a ∈ supp π β (•|s), Q D-CQL (s, a) = Q π (s, a) -α (I -γP π ) -1 µ ζ ν π β (a|s) . (6)
α > 0 leads to a point-wise lower bound on the true Q-values that becomes tight whenever ζ(a|s) = 0.

Proof 1 This proof is done assuming the true Bellman operator B has been used instead of B.

Setting the integrande of the derivative of 2 leads to

Q k+1 (s, a) = B π Q k (s, a) -α µ(a|s) ζ(a|s) π β (a|s) . ( 7
)
The interest of ζ is clearly exposed in this equation: the learned Q-values will be highly conservative when associated to OOD actions. On the other hand, when ζ(a|s) ≈ 0, the update leads to an application of the classic Bellman operator and drive ID Q-values to stay close to their real associated Q-value.

Let µ, π β and ζ the matrices associated to {µ(a|s)}, {π β (a|s)} and {ζ(a|s)} for all s ∈ D and a ∈ supp π β (•|s). The operator associated to this update:

T ζ : Q → B π Q k -α µ ζ
π β defined for any function Q on defined on states belonging to D and actions belonging to supp π β (•|s) is a γ-contraction. Its fixed point is

Q D-CQL = Q π -α (I -γP π ) -1 µ ζ π β . ( 8
)
Remark 1 The sampling error can be handled using concentration inequalities and would lead to the following lower bound, that holds with probability 1 -δ:

Q D-CQL ≤ Q π -α (I -γP π ) -1 µ ζ π β + (I -γP π ) -1 C r,P,δ (1 -γ) |D)| . (9)
We refer to Appendix C of [START_REF] Kumar | Conservative q-learning for offline reinforcement learning[END_REF] for additional details.

B NORMALIZING FLOWS

We now describe the used architecture to estimate ρ β based on Normalizing Flows. They are based on the change of variable for a diffeomorphism f : X → Y and two densities defined on X and Y ρ

X (x) = ρ Y (f (x)) det (Jac f (x)) . (10)
We then learn a diffeomorphism f that maps the unknown distribution on X to a known distribution on Y , for example a multivariate normal distribution so that the factor ρ Y f (X) is efficiently calculable.

In our case, let x = (s, a) and y its associated sample belonging to the Y distribution and let D their dimension. Let's assume there is only one differential mapping y = f (x), and let d < D. We follow the architecture from RealNVP [START_REF] Dinh | Density estimation using real NVP[END_REF] where f maps x to y as follows:

y 1:d = x 1:d , y d+1:D = x d+1:D exp (s(x 1:d )) + t(x 1:d ) , (11)
where s and t are functions defined in R d → R m-d and is the Hadamard product. This transformation results in a simple formula for the Jacobian determinant:

|det (Jac f ) (x)| = exp j s(x 1:d ) j .
This mapping is then repeated N times where each input is randomly permuted before applying equation 11 to ensure each dimension is rightfully processed.

In all datasets, s and t are parametrized by a Neural Network with 1 layer. The number of mappings N as well as the number of units in each layer were selected thanks to grid search on respectively (5, 10, 20) and (64, 128, 256). Y was chosen as a Multivariate Normal distribution N (0, I).

C D-CQL EXPERIMENTS

For reproducible purposes, we now present additional details regarding the experiments. CQL's code was extracted from https://github.com/young-geng/CQL. On all plots, the abscissa represents the number of epochs and the ordinate the cumulative average return over 5 seeds. For the experiments on the classic control data sets, one epoch represents 100 gradient updates and 1000 updates for the ones on the D4RL data sets.

C.1 HYPER-PARAMETERS

We first present all the hyper-parameters used in the experiments. For the experiments on the data sets of D4RL, most of the hyper-parameters -learning rate, size of the networks, optimizer, . . .were unchanged to have a proper comparison with CQL. For the experiments on the classic control environments, the size of the hidden layers of the neural networks were minimized to 64 with the same number of layers.

The main hyper-parameters of the algorithms are α and ν, on which we performed a grid search for each experiment. This grid search was done on (1, 5, 10) for α and on (0.0, 0.1, 0.9, 1.0) for ν. As mentioned in Section 6, the grid search for the classic control environments (Pendulum and MountainCarContinuous) was done on the complete experiments, that is on the 100K gradient updates. Regarding the experiments on the D4RL MuJoCo data sets, the grid search was performed on only 1M gradient updates. The best hyper-parameters were then chosen and we continued this experiment on 10M updates.

C.2 ABLATION STUDY

Study on ν First, we present an ablation study on the hyper-parameter ν on the classic control environments. For all environments, we set α to 5 and launched experiments with a varying ν.

Considering the length of the experiments on the D4RL data sets, we performed the ablation study on hopper-medium (1% and 5%) and on walker2d-medium (1% and 5%). The impact on the hyper-parameter ν varies according to the environment. Nevertheless, we observe the intuitive choice ν = 0 often never produce the best results which were generated with a higher ν. We attribute this effect to the unbounded minimization of the Q-values towards minus infinity discussed in Section 3.2. This can be notably seen on pendulum-medium and hopper-medium-1%, where the best ν is 0.9. This high number has to be put in perspective as ζ ν has been introduced as a multiplicative factor and is inside the soft differential maximum operator, both of which increase the importance of our weighting factor.

We also observe that having a distinction between ID and OOD actions in the penalization may not be useful in some environments. For instance, data sets from the walker2d environment do not need to make such distinction as illustrated by the results of ν = 1. Forcing the Q-values to have a reasonable scale during learning seems to be enough for the Bellman update to work on this environment.

Study on the density estimation technique used

We continue the ablation study by testing the robustness on the density estimation technique used. We compare the Normalizing Flows RealNVP with two other methods: the auto-regressive algorithm RNADE [START_REF] Uria | Rnade: The real-valued neural autoregressive densityestimator[END_REF] and a Mixture of Gaussians (MOG). To make fair comparisons, RNADE and MOG densities have also been created with a grid search on their hyper-parameters, that is on the number of their components, the number and size of the layers of the Neural Network(s) used. We performed the study on data sets from the Pendulum environment, where we can see our algorithm is robust with respect to the density estimation technique used. It remains to be confirmed on higher dimensional environments.

C.3 COMPARISON WITH CQL

We finally provide the plots for Table 1 andTable 2. The hyper-parameters of the experiments are the one chosen from the final normalized average return metric.

  weighting scheme ζ ν (a|s) is able to control how much the Q-value of an action a chosen under µ(•|s) should be penalized and therefore satisfies our goal. Note that CQL corresponds to a particular version of D-CQL where ζ ν (a|s) = 1 -δ a D (a), where δ a D denotes the Dirac over a D (the actions associated to state s in the data set). The new regularization leads to a point-wise lower bound on the Q-values that is tight on in-distribution (ID) actions with a well-chosen ζ ν . It thereby follows the classic Q-Learning procedure for ID actions, while only penalizing the Q-values associated to OOD actions depending on their distance with respect to the data set distribution, controlled by ζ ν .

  Figure 1: Performances of the algorithm w.r.t. to the hyper-parameter ν of the weighting scheme on the classic control data sets.

Figure 2 :

 2 Figure 2: Performances of the algorithm w.r.t. to the hyper-parameter ν of the weighting scheme for D4RL data sets.

Figure 3 :

 3 Figure 3: Performances of the algorithm w.r.t. to the density estimation technique used.

Figure 4 :Figure 5 :

 45 Figure 4: Performances of the algorithm compared to CQL on classic control data sets.

  Algorithm 1 D-CQL Learn ρ β with behavioural cloning Initialize Q θ and π Φ for k ∈ (1, . . . ) do Sample a batch B from D For each state s ∈ B sample M actions from U(A), and M actions from π Φ (•|s) Update Q θ with gradient descent on Equation 5 using Importance Sampling Update π Φ with gradient ascent on Equation policy improvement end for

Table 1 :

 1 Normalized scores between CQL and D-CQL on the classic control data sets.

	DATA SET	METRIC (100K)	CQL	D-CQL	IMPROVEMENT (%)
	P E N D U L U M-B A D	BEST OVERALL	64.2 ± 13.4 67.7 ± 11.9	5.4
		BEST FINAL	50.1 ± 6.6	58.2 ± 15.6	16.1
	P E N D U L U M-M E D I U M	BEST OVERALL	81.3 ± 9.4	88 ± 6.7	8.3
		BEST FINAL	74.7 ± 23.7	83.4 ± 9.5	11.7

  Overall, both Table1and Table2illustrate the fact that D-CQL outperforms or is comparable with CQL. Detailed plots can be found in Appendix C.3. Furthermore note the decrease of performance for both methods w.r.t. gradient steps emphasized by the differences between the best and final performances. While out of the scope of this paper, it remains an open question in off-policy algorithms(Aviral Kumar, 2021;[START_REF] Kumar | Implicit under-parameterization inhibits data-efficient deep reinforcement learning[END_REF]. Last but not least, our experiments shown that even though the methods might prevent the over-estimation of the Q-values, agents' performances have still non-negligible variances.

		BEST OVERALL	18.6 ± 3.2	17.6 ± 2.8	-5.4
		BEST FINAL	11.6 ± 5.1	15 ± 4.6	29.3
	W A L K E R2D-M E D I U M 1%	BEST OVERALL	24.1 ± 14.1 26.7 ± 15.4	10.8
		BEST FINAL	22.9 ± 16.5	20.2 ± 8.7	-11.8
	H O P P E R-M E D I U M 1%	BEST OVERALL	64.1 ± 9.7	68.4 ± 9.1	6.7
		BEST FINAL	63 ± 4.7	64.4 ± 7.9	3.6
	TO T A L ME A N	BEST OVERALL			3
		BEST FINAL			7.5

DENSITY CONSERVATIVE Q-LEARNINGIn this section, we introduce Density-CQL (D-CQL), our algorithm circumventing these issues.

ζν is a weighting scheme, not a density. However, we abuse notations by taking a conditional notation of ζ on the state s to highlight that it depends only on the action given a state.

The results for CQL have been generated with the code https://github.com/young-geng/CQL.

ACKNOWLEDGMENTS

We thank Christophe Prieur and Bogdan Robu from GIPSA Lab of Grenoble-Alpes University for the useful discussions we had regarding this work. We also thank the anonymous reviewers for their feedback which improved the paper.


Janine Rogalski
email: rogalski.muret@gmail.com

New questionings for interventions and research conducted based on stakeholders' activity

Introduction

It was from a double point of view that I read the publication coordinated by Simon Flandrin, Christine Vidal-Gomel and Raquel Becerril Ortega: the point of view of the French-speaking world's ergonomics psychology following in the footsteps of Jacques Leplat, and that of professional didactics in line with the works of Pierre Pastré, in which the analysis of the activity of a subject already on the job is used to organize the intervention of a subject in training. This point of view differs from the CHAT theory (Cultural-historical activity theory) developed by Engeström and used to analyse learning (Engeström & Sannino, 2009), since we focus on individual stakeholders instead of a more complex system of activity in which they would be integrated, as well as on situations seen as crucial determining factors of the activity, without getting centred on the instruments of action. Simulation situations appear to be good candidates to help trainees develop skills, as well as situations of action, which are also potential means to develop skills (Savoyant, 2010(Savoyant, /2005)).

The aforementioned work had me raise some questions about simulation used for training purposes, around which this paper is organized. A very important specificity of those situations is the fact that someone is intervening at once as the object of the action (for care, help, etc.) and as the active and interactive subject in relation to other participants. Models of activity designed in cognitive ergonomics, initially elaborated to analyse situations in the field of industry, must thus be subjected to a significant transposition process. The same can be said of the design and making of professional training sessions or sequences, and all the more so of the design and use of simulation situations as analysed by this school of thought. We will start with presenting the directions in which the categories of situations that could be grounds for simulation-based training have been broadened. We will outline the adaptation of Leplat's framework model (1981;1997) to the objectives of care training, acknowledging factors that are important for that transposition. They are mostly organisational factors: in what context do care activities occur -in an institution or at the "patient's" home -and temporal factors: is it a one-off intervention such as the accompaniment of a nursing act with appeasing words or gestures? Is it the care activity that a care professional or family member acting as a caregiver performs throughout their working day or week, or even their whole career? Finally, we will question the diverse modalities of transposition of the target situationswhich should be mastered by the end of the training course -into simulation situations for training, in order to take into account the care-related dimension of an activity that is directed towards another person (from hospital care to home support).

Evolution of the categories of simulation-based training situations

For very many years, simulation has been used as a double-edged tool: for researchers and for trainers who analyse the activity of professionals on the job with a view to design training courses (Béguin & Weill-Fassina, 1997). Regarding this particular use of simulation for training purposes, works in the field of ergonomics psychology (or cognitive ergonomics) have initially dealt with material situations, primarily in the aviation sector, and related to industrial processes and other similar dynamic processes (Bainbridge & Quintanilla, 1989). Process management analysis was also transposed into specific human interaction situations. It has been the case with the activity of teachers in class (Rogalski, 2003;Goigoux, 2005;Roditi, 2008). Clearly, this was not enough to go beyond the analysis of activity and think out a transposition in terms of simulation for teachers' training. We can point out in particular that in a "strict" -or blind!transposition, we are faced with the following problem: "Is it possible -and if so, howto transpose pupils, who are at once the objects of the teachers' actions, and subjects of their own learning processes?" This difficulty is somewhat subdued if we integrate the fact that the objective is not to simulate "the object of the action" but "the situation of action". This shift in perspective was developed in the special issue of the journal Le travail humain, coordinated by Samurçay and De Keyser (1998). It consists in questioning the extent to which the activity in a simulation situation represents the activity in the target situation and prepares trainees to handle it. In the field of teachers' training, the most accurate simulation situation seems to be the one in which teachers take on the role of pupils. It is the case, for instance, when a new content is integrated to the curriculum they are in charge of. Trainers help teachers familiarize themselves with this new content and the teachers -possibly with the trainers' help -must transpose this situation that they have experienced as students into the teaching they will provide to their own pupils. It is that very same perspective that was favoured in Lang's works (2001), whose objective was to introduce ergonomics in initial professional training courses, so that the students, as future operators, could become ergonomics-savvy stakeholders and thus contribute to the prevention of occupational risks. Teachers were put in the position of students: after being taught about the concepts by teachersresearchers, they had to conduct an analysis of the activity of operators in situation and report the results found. After this experience, they were required to prepare their future teachings. In didactical mathematics, this type of simulation has been called "homologybased training" (Robert & Vivier, 2013); it can thus also be used in professional training.

A similar simulation situation can exist, but only under specific conditions of activity: it is called "modeling-based" training. The trainer works "as closely as possible" with the student or apprentice and makes the content of his activity explicit; the student/apprentice follows and reproduces the trainer's actions, which models their own actions.

We can interpret in terms of "modeling" a situation where a trainer films a lecture given in front of simulated students/apprentices, so that teachers-in-training can later reproduce in their actual work situations what the "model" has done in the recording, which they get to watch and analyse. Here, notably, some properties of video recording are exploited: the possibility to freeze the image or go backwards allows for a detailed observation and a finer analysis. We personally participated in one of those training sessions, acting as a trainee (Janine Rogalski and Renan Samurçay). A video recording focusing on the trainer was taken in order to be used as a model by the next trainer. In this particular case, the trainer must mobilize their acting skills, since they must consider the simulation in front of them as an actual group of professionals attending a training course.

This modeling-based training can be seen as a derived form of "teach your twin" training (Oddone, Ré & Briante, 1981;Saujat, 2004). We chose to allude briefly to those examples, since care situations share an essential property with training situations: the object of the action is another human subject with their own autonomy, and simulation situations for training purposes thus pose partly similar problems.

We are now going to present a framework for analysis that will allow us to orient and specify the questions that can be raised in these types of simulation-based training courses: i.e., the double regulation of activity model proposed by Leplat (1981;1997). We will then propose a list of questions that should be answered in the study, design and use of such simulation situations.

A framework model for analysing care-related activity and designing simulation situations: Leplat's double regulation of activity model

In the collective work he published, "Apprendre par la simulation. De l'analyse du travail aux apprentissages professionnels" ("Learning through simulation. From work analysis to professional training", 2015), Pastré develops the idea that the link between the analysis of activity and the use of work situations for training purposes is at the core of professional didactics, these two dimensions walking hand in hand. Daniellou and Rabardel (Daniellou, 2005;Daniellou & Rabardel, 2005) restated the key principles of the French-speaking world's ergonomics tradition and specified the links between this tradition and the input contributed by the theories of activity developed in Russia in the footsteps of Vygotsky (1978) andLéontiev (1979). We will use here one of the components of this very same research tradition, initiated by Leplat: the double regulation of activity model (1997).

This model, introduced and developed in work psychology by Leplat (1997) is a conceptual tool which we are going to use to present the dynamics of activity in a carerelated situation. The analysis of activity, its conditions and the professional skills involved were first brought into play for studying the security of systems (Leplat, 1981) and human/machine interactions (Leplat, 1989;de Montmollin, 1991), then for designing computer systems. We will then propose a model for care-related activity, drawing inspiration from the work done in the studies related to teachers' activity.

Leplat's model is built around two main ideas: the double determination of the activity and the effects produced by its being carried out. This model can be directly transposed in the case of collective activity (Leplat, 1994, p. 213), which allows for the analysis to go beyond individual stakeholders. On the one hand, the activity is determined by factors that are related to both the situation which the activity responds to, and the subject who develops it; on the other hand, a subject's action has effects both on the object targeted by this action (the target-state) and on the subject themselves. Darses presents the state of the works relying on activity analysis: […]" from a psychological point of view, as the product of a coupling between a task (which provides external determinants), and a subject (who provides internal determinants)" (Darses, 2016, p. 193). The activity in itself is motivated by personal and social factors (it is always "addressed" to another person -or several other people), it aims at reaching goals and organizes operations to achieve them (Leontiev, 1979;Savoyant, 2010Savoyant, /1979)).

The regulation of the activity is due to the fact that it changes over time according to two types of effects produced depending on the subject's intentions. The first dimension of this regulation is taking into account the state in which the action has put the object, compared to what the goal of the action was. This regulation is well-known in learning psychology, but also as an effect of professional experience. The second dimension of the regulation is the adaptation of the activity according to its impact on the stakeholder themselves (their tiredness, their interest, their emotions, their social relationships at work, etc.). It can lead them, for instance, to carry out the most complicated operations first: this is called regulation by anticipation and helps prevent risks (see for example, Clot & Simonet, 2015).

The initial double regulation model does not specify over what time span the action is being considered, but it can be completed and adapted to various temporalities. In the field of work, we can distinguish three main time spans that are relevant; they correspond to three levels of "granularity" in the analysis of the subject's activity (be it individual or collective). At "micro" level, the activity is analysed over a "short" time span (depending on the object of the action and the dynamics at play: it can be only a few seconds long in the case of plane piloting). For people-oriented tasks, it can be a specific intervention for a patient in an institution or at home (helping the patient shower, doing the dishes, tidying up the room, etc.). For training situations, it can be a practical exercise that the trainee must carry out. At "meso" level, the whole duration of the activity in a specific work position is being considered: for a trainer, it corresponds to one day of training course or even a whole session; for care professionals, to their entire working period in one position (or one day with the professional's own "patients"). At "macro" level, we consider the longest time span associated with each kind of task: the school year of a teacher in charge of organizing the implementation of the curriculum in a certain class; a professional training course in the long run; the long-term activity -depending on the results -of therapists such as, for example, physiotherapists or speech pathologists.

The regulation of the care-related dimension in a people-oriented activity

As far as care is concerned, two different situations must a priori be distinguished. In the first type of situation, the care professionals act within the context of an organization such as that of a hospital or an establishment for dependent people. In the second case, they work with patients who are independent enough to live at home, but need help in their day-to-day life.

Two dimensions then take on different values: the temporal dimension of the organization of the activity and the social dimension of collective action. Indeed, within the context of an institution, on the one hand, a group of care professionals is in charge of a number of patients, all in the same location, for the duration of the working day. On the other hand, they have the possibility to help each other perform some care-related actions, and must "physically" cooperate with each other (to turn over a bedridden patient, change sheets, help a patient move from their bed to their wheelchair, etc.). The move from one patient to another happens over a short time span, since they all are in the same functional space.

In a work situation at the patient's home, the constraints are different: they are related to the organization and layout of the person's place, the number of at-home visits that must be carried out in a day, the trips that must be taken, the administrative follow-up of the patients, etc. In most cases, professionals who work at the patients' home have comparatively less people in their care than those who work in institutions, but the travel time needed to carry out the necessary visits is a limiting factor and can be stressinducing. Moreover, those stakeholders cannot get immediate help when an unplanned difficulty arises. The working collective is also more "distant" which makes it more complicated to discuss patients and their care routines as well as talk about various workrelated difficulties and exchange views on the way to deal with them. In this case too, albeit not for the same reasons than in institutions, the knowledge that "time is running out" is a factor that puts pressure on the relationship between the professional and the patient, and causes tensions in the caregivers' activity. Indeed, care is centred on the interpersonal dimension of the "patient's" well-being. The positive impact of affective social interactions on mental and physical health has been proven: such interactions may need time to develop and produce an effect. Furthermore, the cooperation with other stakeholders mostly takes place remotely, which increases the need for anticipation and, potentially, the caregivers' decision-taking responsibility (we can think of pain management, which depends on a medical decision but is implemented by nurses -in particular when the painkillers can have serious side effects).

What model(s) of activity apply for care stakeholders?

The question of simulation for training purposes is not limited to the determining elements that we have just gone back over. It calls for an analysis of the activity itself as it is being carried out. One of the goals of training is indeed to outline a reference activity for trainees. The analysis makes it possible to mark out the elements of the activity which care stakeholders are going to be trained for, or that will contribute to their development as professionals. The distinction between technical and non-technical skills is well-known in the field of collective work situations analysis: in this case, the non-technical skills are those that are directed towards human interaction during a specific task, as shown by Salas et al. (Bedwell, Fiore, & Salas, 2014). Here, we focus on those that are at stake in care-related situations that occur between the stakeholder and their "patient". This analysis can be conducted from two different perspectives: either it is centred on the activity of the "caregiver" stakeholder, who takes another human stakeholder has the object of his action, or it concerns the interaction between the two stakeholders, who are in an "request and answer" situation where the caregiver carries out actions that the person cared for cannot perform on their own. From the theoretical point of view of cognitive ergonomics, the first perspective is that of an individual action, whereas the second perspective takes into consideration the activity of a collective, in a strongly dissymmetrical relationship. We chose here to take the first approach in order to analyse the care stakeholders' activity (vis-à-vis those in need of care), taking into account their training goals. Several aspects of the activity can be integrated in the training course.

For instance, a review of questions about care for seriously ill people highlights the expectations related to care: "[it] encompasses the patient and family being respected, given complete information, involved in decision-making and supported in their physical, psychological, social and existential needs. The studies highlight the importance of [...] structuring service organization to enable care continuity" (authors ' abstract, Guisti et al., 2020).

We also keep in mind the works of Naweed, Stahlut, and Keeffe (2021) whose detailed analyses are consistent with Leplat's model (op. cit.) and allow us to identify a number of avenues for reflection about stakeholders' training. They bring to light determining factors of the situation that can have negative effects on the activity being carried out and its results on the patient's care, their well-being or even that of the care professionals themselves: "contextual factors in scenarios highlighted inadequate staffing and procedures, inadequate training, challenging residents, time poverty, and low support". From the perspective of the caregivers' activity, they underline the development of "close relationships with residents, influencing care provision but blurring personal boundaries" (which, by contrast, highlights the necessity to maintain professional detachment). The results of these works (op. cit.) bring to light the occurrence of adaptation processes in the course of the activity: "individually directed adaptive strategies [...] used to alleviate dissonance and maintain emotional resilience", and the place of diagnosis and prognosis processes, "including dynamic risk assessment involving rule breaking". Their analyses were conducted at the macro or meso levels of analysis of the activity (expected or carried out).

One element that seems to have been left out of the studies in this questions review is the management of the tension that exists between the various goals related to the same "patient", i.e. carrying out a care or help-related action for the patient's benefit while also interacting with them in a quality way. Achieving this type of joined objectives is an integral part of the short-term activity of caregivers (at the meso-level of granularity in the analysis of activity). This type of goal can give rise to the elaboration of collective strategies. For instance, in an ongoing research survey (Renaudin, in preparation), we can identify a strategy for managing aggressiveness in elderly patients suffering from cognitive impairments during their washing routine. This particularly complicated and hazardous moment relies on a specific distribution of tasks: one of the caregivers attentively interacts with the patient in order to divert their attention from the ongoing washing, while the other caregiver carries out the washing gestures. The interaction becomes more intensive at the trickiest moments. Strategies of that kind are precious for initial and ongoing training. They can be analysed, debated, improved. They participate at once in the prevention of occupational risks and the fostering of the patient's well-being. Those compromises that can be identified in the analysis of activity at micro-level can occur in every type of care or home-support organization.

In the paragraphs above, we have oriented our questioning towards the care-related dimension of an activity directed towards another person. This dimension can also be present in collective actions between co-workers (in "horizontal cooperation" as in the example mentioned) or between stakeholders in a hierarchical relationship (in "vertical cooperation"). Examples of this have been given in this book, about stakeholders in civil security or police forces, highlighting the possibility to design simulations for their training (see Dubois, Vandertrate & Van Daele's chapter). Furthermore, the hierarchical relationship can also be used for care-related purposes in order to circumvent defensive behaviours; for instance, a chief first responder systematically requests the services of a hospital psychiatrist for first-responder fire fighters under his command who are confronted to tragic situations affecting people, thus avoiding possible denial regarding the psychological impact of their intervention. In this case, it is not easy to see how the situation could be transposed and integrated in a relevant manner into the training of new chief first responders.

Transposing the care-related dimension from situations of action to training situations via simulation

The model of activity analysis we worked on shows the diversity of perspectives that can be considered and makes it possible to orient the transposition of those objectives and the characteristics of the target situations towards simulation-based training. It allows us to see the impact of the choice of components and the relationships at play in the target situations that will be reproduced in a simulation situation according to the goals of the training course.

Another dimension of the analysis of activity for training purposes, and its transposition into didactic simulations, is that of the three levels of functioning in a work situation, developed by J. Rasmussen (1983), i.e., respectively, the skill-based, rule-based and knowledge-based levels. At skill-based level, routines of professional functioning and automatisms are at play; at rule-based level, the stakeholder draws from a basis of action processes that can be expressed in terms of involvement: "in case of situation S1, carry out action A1". At knowledge-based level, the stakeholder is faced with a problem of which he must design the resolution (or to which he must find an acceptable enough solution). It is easy to see that the level that is most easily transposed into training is the one that aims at assimilating rules (rule-based level) by adapting into simulations a series of cases in which this or that rule will be applicable. It seems to us that it is this level that is at stake, for instance, in Naweed, Stahlut and 0'Keeffe's analyses (2021), which consider that "versatility as an adaptive response to challenges [...]" in elderly people' s care is the "essence of care".

The skill-based level is very strongly linked to the multiplicity of experiences that have thus become sedimented (Rogalski & Leplat, 2011): in initial training, it can thus be aimed at for professional development, relying on oriented simulations called "drills"i.e. learning through rehearsal, repetition and practice of a task -based on repeated interactions with a high-physical fidelity simulator.

A simulator with a high functional fidelity but a low physical one will only be appropriate for knowledge-based processes, while a simulation that is faithful on a conceptual level but modifies the conditions of information gathering and action in the target situation will pose problems with regard to the processes of representation of the situation (indispensable for functioning at rule-based level).

In the field of care, simulation can, for instance, aim at building up a representation of this "other, sick or elderly" who is the object of a trainees' action. Thus, Eost-Telling, Kingston, Taylor and Emmerson (2021) present "ageing simulation equipment" (impeding movements and walk, affecting sight and hearing, etc.) to enable caregivers in training "to undergo experiences which potentially affect older patients and by asking the learner to act the role of patient this may be more effective in developing greater empathy [...]." The authors, however, do not underestimate conceptual and theoretical training (which aims at reaching knowledge-based level), but use that simulation "to challenge ageist attitudes and behaviour".

The question of taking into account the three levels of functioning in the prospect of designing and performing simulation situations, which has already been raised in Rogalski (1997), brings up another, much trickier question: what role can simulations play in the process of acquiring a particular component of professional skill, i.e. the possibility to switch from one level of functioning to another over the course of the activity? It seems to us that this last question has hardly been explored by research in the field of professional activity analysis, and that it is particularly crucial in situations of action exerted "on" or directed "towards" another person, when it comes to training professionals to take into account the care-related dimension of such situations.

We hope to show that broad avenues are still open to research and action in the schools of thought that mobilize theories on activity, with a both an epistemic and a pragmatic dimension.

Renaudin, R. (in preparation). Caregivers' strategies for working with potentially aggressive cognitively impaired elderly patients. Master Education sciences. Nantes University, FR: Department of Education and Training Sciences. We talk of simulation, not experimentation of a new teaching or a new pedagogical method, given that the "pupils" in the training course are not the actual pupils targeted by the teaching but teachers in the position of pupils. Professional didactics is a school of thought that allies research and intervention in professional training, and relies at once on the contributions of Leplat's cognitive ergonomics and Vergnaud's model of conceptualization in action (1990). Mayen (2015) gives a detailed presentation of it. We use the term "situation" to refer to the task system, with a system of resources and constraints, within an organization of stakeholders and prescribers. We can see this term as a development of Leplat's definition of task (following Leontiev's, op. cit.): a goal to reach under certain conditions.

We nevertheless also mention here the specific situations created by long-term illnesses or pathologies, where the patients can become "expert patients" and interact with care professionals on an equal footing. Personal communication. Anonymity was requested.