Olivier Pietquin
email: olivier.pietquin@supelec.fr

Fabio Tango
email: fabio.tango@crf.it

Raghav Aras
email: raghav.aras@supelec.fr

Batch Reinforcement Learning for Optimizing Longitudinal Driving Assistance Strategies

Partially Autonomous Driver's Assistance Systems (PADAS) are systems aiming at providing a safer driving experience to people. Especially, one application of such systems is to assist the drivers in reacting optimally so as to prevent collisions with a leading vehicle. Several means can be used by a PADAS to reach this goal. For instance, warning signals can be sent to the driver or the PADAS can actually modify the speed of the car by braking automatically. An optimal combination of different warning signals together with assistive braking is expected to reduce the probability of collision. How to associate the right combination of PADAS actions to a given situation so as to achieve this aim remains an open problem.

In this paper, the use of a statistical machine learning method, namely the reinforcement learning paradigm, is proposed to automatically derive an optimal PADAS action selection strategy from a database of driving experiments. Experimental results conducted on actual car simulators with human drivers show that this method achieves a significant reduction of the risk of collision.

I. Introduction

A. Context

The work presented in this paper has been realized inside the co-funded European project ISi-PADAS 1 . This project aims at developing and implementing a Partially Autonomous Driver Assistance System (PADAS in short) in a static driving simulator, including the interfaces between the driver and the system (comprising haptic, visual and audio interfaces). The development of such a system was based on the in-depth analysis of the car-accidents, conducted in the project, about causes of driver's errors responsible for rear-end crashes [START_REF] Muhrer | Results from accident analysis[END_REF]. For this aim, the dataset comprising 4256 accidents from Braunschweig 2002 has been used [START_REF] Vollrath | Ableitung von anforderungen an ein fahrerassistenzsystem aus sicht der verkehrssicherheit[END_REF]. Additionally, a sample from the German National Accident Database (Statistisches Bundesamt) from 2002 was used including 185004 accidents [START_REF] Briest | In welchen situationen machen fahrer welche fehler? ableitung von anforderungen an fahrerassistenzsysteme durch[END_REF]. All rear-end crashes take part for 73.3% of analysed accidents and in particular 22.81% of the severe accidents (with major damage of more than 6000 Euro, injuries or fatalities). Following a vehicle too 1 www.isi-padas.eu Fig. 1.

architecture of the LOSS application, showing the two modalities and the constituting functions closely represents the most important cause of accident in 85,61% of all the cases analyzed. Moreover, 75,98% of the rear-end crashes occur in undisturbed/flow traffic (where the expectation of an event is probably low). In this context, many studies have shown the benefits of Forward Collision Warning (FCW) and of Adaptive Cruise Control (ACC) in reducing the number and severity of rear-end collisions [START_REF] Kiefer | Status of nhtsa's rear-end crash prevention research program[END_REF], especially in conditions where drivers have to cope with car following tasks in limited traffic flows or heavy -but not congested -traffic 2 . In this context, the PADAS developed in the project was focused in assisting drivers during their longitudinal driving task and it is called Longitudinal Support System (or LoSS, in short). In particular, it can prevent a collision with a leading vehicle by providing warnings to the driver, up to bringing the vehicle to a halt independently of driver's action, through the support for an assisted braking action. The system has two mechanisms at its disposal in order to realize this collision-avoidance capability:

• it can provide warning signals to the driver • it can decelerate the vehicle. Fig. 1 illustrates the functional scheme of PADAS application: Two types (or modes) of LOSS have been considered: the Advanced Forward Collision Warning (FCW+, in short) and the Advanced Adaptive Cruise Control (ACC+, in short) which are both constituted by 3 functions:

• Forward Collision Warning (FCW) or Adaptive Cruise Control (ACC), respectively for FCW+ and for ACC+

• Assisted Braking (AB), for both • Emergency Braking (EB), for both FCW represent the "traditional"forward collision warning system: if the driver travels too close (too short headway) or too fast with respect to the vehicle ahead on the same trajectory, s/he receive a warning of different severities, depending on the dynamic conditions between the two vehicles. This is the first function for the FCW+ system. Alternatively, the first function can be the "traditional" ACC, in which not only the speed is kept at the specific value (like in the Cruise Control), but also the distance is kept within pre-defined threshold of headway. ACC is used as first function in the ACC+ system. Then, the other two functions are common to both the systems. In particular, AB function provides the proper assistance to the driver: if the driver acts on the brake pedal, thus indicating the will to brake, but not performing in the most appropriated way, the system is able to modulate the braking action automatically. Finally, if the driver ignored warning and AB did not intervene (i.e. driver did not apply on the brake) EB function acts, in order to avoid accidents or at least to minimise the effects, in case it is not avoidable anymore.

B. Contribution

This paper addresses the problem of finding an optimal warning and intervention strategy for a partially autonomous driver's assistance system (PADAS). Here, an optimal strategy is regarded as the one minimizing the risk (probability) of collision with an obstacle ahead, while keeping the number of warnings and interventions as low as possible, in order to support the driver and avoid distraction or annoyance. A novel approach to this problem is proposed, based on the solution of a sequential decision making problem solved via the Reinforcement Learning (RL) [START_REF] Sutton | Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)[END_REF] paradigm and learning on a fixed dataset.

In the RL paradigm, a learning agent (the controller or decision maker) learns an optimal control strategy by either interacting with the system or observing interactions between another controller and the system. To do so, the system's dynamic as to be cast into the Markov Decision Processes (MDP) formalism [START_REF] Bellman | A markovian decision process[END_REF]. The system is then considered as made up of states (situations to which a control as to be associated) and the control is considered as being a sequence of actions applied to the system. In every situation, the controller has to choose an action to apply to the system. Once the action has been applied, the system steps to another state according to its own dynamics and a reward qualifying the quality of the transition is generated. The controller has thus to learn a mapping between situations and actions (named as strategy or policy) that maximises the long run reward.

The collision avoidance problem is thus seen as a sequential decision making problem to optimize given a set of {vehicle+driver}-PADAS interaction sequences in which each sequence {s 1 , a 1 , s 2 , a 2 , . . . , s T , a T } consists of the vehicle+driver's states {s i } and the PADAS' reactions {a i } to it. Each such sequence engenders a probability of collision, and an optimal strategy is one which gives rise to a subset of this set of sequences for which the average or expected probability of collision is the smallest. In our problem, we do not know the probability of collision associated with any given sequence. The only data that is available to us consists only of small subset of the set of sequences of {vehicle+driver}-system interactions. On the other hand, we must find a strategy that takes into account all possible vehicle-system interaction sequences.

Few attempts to use RL in the case of autonomous driving can be found in the literature and most of them are related to steering the vehicle [START_REF] Kuhnert | Autonomous vehicle steering based on evaluative feedback by reinforcement learning[END_REF], [START_REF] Martinez-Marin | A reinforcement learning algorithm for optimal motion of car-like vehicles[END_REF], [START_REF] Oh | A new reinforcement learning vehicle control architecture for vision-based road following[END_REF]. In this paper, we focus on RL for a continuous longitudinal support to the driver, which starts from the pre-collision and collision warnings for the driver and ends to the automatic emergency braking, through an assisted braking action. Therefore, we consider specifically the longitudinal driving task -car-following situation -and in that case, the driver is completely in the control-loop of the vehicle for the first two stages of the supporting function, while the system takes the full control of the vehicle only in the last phase of the emergency braking. In this context, the decision maker has to take into account driver's behavior in the dynamic of the controlled system which is not case in the steering task. The fact that the driver stays in the loop makes the system highly stochastic (and maybe non-stationary) because of inter-and intra-driver variability, that is to say that to similar situations from the point of view of the PADAS, different reactions can occur given that the driver is different or the perception of the situation is different even for a same driver.

The rest of this paper is organized as follows. Section II presents the theoretical background of Reinforcement Learning and Markov Decision Processes. Section III presents how the problem of PADAS strategy optimisation can be cast into the MDP paradigm. In Section IV, experimental results are provided. Finally, Section IV concludes and provides possible future investigations.

II. Markov Decision Processes

The problem of optimally controlling a stochastic dynamic system is often addressed by the machine learning community in the framework of the Markov Decision Processes (MDP) [START_REF] Bellman | A markovian decision process[END_REF]. An MDP is a stochastic finite state machine which is formally described in the next section.

A. Definition

An MDP is defined as a tuple {S, A, P, R, γ} where S is the set of states (the different configurations of the system), A is the set of actions (which cause a change of the system's state), P : s, a ∈ S × A → p(.|s, a) ∈ P(S) is a set of Markovian transition probabilities (the probability to transit from one state s to another s given that action a was taken in state s), a reward function R : s, a, s ∈ S × A × S → r = R(s, a, s ) ∈ R associating a scalar to each transition and a discounting factor γ which decreases long-term rewards' influence.

The Markov property means that the probability to step from one state s to another s given an action a depends only on s and a and not on the previously visited states and taken actions :

P (s i+1 |s i , a i , s i-1 , a i-1 , . . . , s 0 , a 0 ) = P (s i+1 |s i , a i ) (1)
The way the controller selects actions is modeled by a socalled policy, π : s ∈ S → π(.|s) ∈ P(A), which associates to each state a probability distribution over actions. The quality of such a policy is quantified by a so-called value function,

V π (s) = E[ ∞ i=0 γ i r i |s 0 = s, π],
which associates to each state the expected cumulative discounted reward from starting in the considered state and then following the given policy. An optimal policy π * is one of those which maximize the associated value function for each state:

π * ∈ argmax π V π .
One can also define the state-action value (or Q-) function which provides an additional degree of freedom on the first action to be chosen:

Q π (s, a) = E[ ∞ i=0 γ i r i |s 0 = s, a 0 = a, π].
(

An optimal deterministic policy can thus be obtained by maximizing the optimal Q-function over actions:

π * = argmax a∈A Q * (s, a). ( 3
)
The Q-function generalizes the value function in the sense that

V π (s) = E a|s,π [Q π (s, a)
]. This is why only the Qfunction will be considered in the rest of this paper. Notice that knowing the Q * (s, a) function for a given MDP is enough to derive an optimal policy. So most of algorithms aim at estimating this Q * function.

B. Dynamic Programming

Because of the Markovian property of the transition probabilities (see eq. ( 1)), the expression of the Q-function (eq. ( 2)) can be rewritten as:

Q π (s, a) = E s |s,a [R(s, a, s ) + γQ π (s , π(s ))] = T π Q π (s, a) (4)
This is the so-called Bellman evaluation equation and T π is the Bellman evaluation operator. The optimal Q-function Q * (s, a) can also be written as a Bellman equation (the Bellman optimality equation, where T * is the Bellman optimality operator):

Q * (s, a) = E s |s,a [R(s, a, s ) + γ max b∈A Q * (s , b)] = T * Q * (s, a) (5)
Dynamic programming (DP) [START_REF] Bellman | Dynamic Programming[END_REF] aims at computing the optimal policy π * using Bellman equations if the transition probabilities and the reward function are known. Two classes of algorithms can be distinguished. First, the policy iteration algorithm computes the optimal policy in an iterative way. The initial policy is arbitrary set to π 0 . At iteration k, the policy π k-1 is evaluated, that is the associated

Q-function Q π k-1 (s, a) is computed.
To do so, eq. ( 4) is used. Since T π is linear eq. ( 4) defines a linear system that can be solved by standard methods or by an iterative method using the fact that Q π is the unique fixed-point of the Bellman evaluation operator (T π being a contraction that is

T π V 1 -T π V 2 ≤ α V 1 -V 2 with α ∈]0, 1]): Qπ i = T π Qπ i-1 , ∀ Qπ 0 lim i→∞ Qπ i = Q π .
Roughly, this means any policy π can be evaluated by starting from an arbitrary initial Q-value Q π 0 and than iteratively applying eq. ( 4) to it. Once evaluated, the policy is improved, that is π k is greedy respectively to

Q π k-1 : π k (s) = argmax a∈A Q π k-1 (s, a).
Evaluation and improvement steps are iterated until convergence of π k to π * (which can be demonstrated to happen in a finite number of iterations when π k = π k-1 ). In practice, the learning is stopped when two consecutive policies are identical which happens when π k = π * since the policy which is greedy according to its own Q-function is the optimal policy by definition (eq. ( 3)).

The second algorithm is the value iteration algorithm. It aims at estimating directly the optimal state-action value function Q * which is the solution of eq. ( 5). The T * operator is not linear (because of the max operator), therefore computing Q * via standard system-solving methods is not possible. However, it can be shown that T * is also a contraction [START_REF] Puterman | Markov Decision Processes: Discrete Stochastic Dynamic Programming[END_REF]. Therefore, according to Banach fixedpoint theorem, Q * can be estimated using the following iterative way:

Q * i = T * Q * i-1 , ∀ Q * 0 lim i→∞ Q * i = Q * (6)
However, the convergence takes an infinite number of iterations. Practically speaking, iterations are stopped when some criterion is met, classically a small difference between two iterations: Q * i -Q * i-1 < ξ. The estimated optimal policy (which is what we are ultimately interested in) is greedy respectively to the estimated optimal Q-function: π * (s) = argmax a∈A Q * (s, a).

C. Approximate DP and Reinforcement learning

Dynamic programming makes two very strong assumptions. First it assumes that the dynamic of the system is known and therefore that transition probabilities and the reward function are available. Second, an exact representation of the Q-function is supposed to be computable. Both assumptions are rarely met. Instead of having access to the dynamics of the system two options are often possible: either a database of interaction examples is available, either the system itself can be tested through a trial-anderror process. In the former case, the class of algorithms aiming at estimating an optimal policy from a fixed data set is named Approximate Dynamic Programming (ADP). It forms a set of batch algorithms (off-line) to solve the optimal control problem at sight. In the later case, the learning agent has to improve its strategy online while interacting with the system. This is the general Reinforcement Learning problem. In any case, if the state-action space is too large to be enumerated so as to have an exact (tabular) representation of the Q-function Q(s, a), then it is common to learn a parametric approximation Q θ (s, a) where θ is a set of parameters to learn. The parameterization can be of any form. For example, if the Q-function is approximated by a neural network like in [START_REF] Tesauro | Temporal Difference Learning and TD-Gammon[END_REF], the set of parameters θ contains the synaptic weights of the network. However, a common choice is the linear parameterization like in [START_REF] Bradtke | Linear Least-Squares algorithms for temporal difference learning[END_REF]:

Q θ (s, a) = p i=1 φ i θ i = φ T θ (7)
Most renown ADP algorithms are Least Square Temporal Differences (LSTD) [START_REF] Bradtke | Linear Least-Squares algorithms for temporal difference learning[END_REF], Least Square Policy Iteration (LSPI) [START_REF] Lagoudakis | Least-squares policy iteration[END_REF] and Fitted Value Iteration (FVI) [START_REF] Gordon | Stable Function Approximation in Dynamic Programming[END_REF]. LSTD is an efficient batch algorithm for policy evaluation. LSPI uses LSTD in a policy iteration framework to learn an optimal policy from a database of interactions. FVI is more related to the value iteration algorithm and aims at directly discovering the optimal Q-function from a dataset. There are a lot of online reinforcement learning algorithms in the the literature. They are mostly based on the computation of a temporal difference which can be seen as the difference between the right and left members of one of the Bellman equations for one transition. For instance, in the case of the Bellman evaluation equation, the temporal difference δ E i for a given transition {s i , a i , r i , s i+1 , a i+1 } is given by:

δ E i = r i + γ Qθi-1 (s i+1 , a i+1 ) -Qθi-1 (s i , a i )
where θ i is the evaluation of the Q-function parameter set at time i. The temporal difference δ O i for the Bellman optimality equation is:

δ O i = r i + γ max b∈A Qθi-1 (s i+1 , b) -Qθi-1 (s i , a i )
Temporal differences algorithms use this quantity δ i to update the parameters estimation after each interaction using a Widrow-Hoff-like equation:

θ i = θ i-1 + K i δ i
where K i is a gain. Algorithms differ by the temporal difference considered and the gain computed. For instance, the SARSA algorithm [START_REF] Sutton | Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)[END_REF] uses the δ E i temporal difference and the gain K i is a simple learning rate. It is an online version of policy iteration. The Q-learning algorithm [START_REF] Watkins | Learning from delayed rewards[END_REF] uses the δ O i temporal difference also with a simple learning rate and is an online version of the value iteration algorithm. Other methods exist like the Kalman Temporal Differences framework (KTD) [START_REF] Geist | Kalman Temporal Differences[END_REF] where the gain is computed with a Kalman filter and either temporal difference can be used, Gaussian Processes Temporal Differences (GPTD) [START_REF] Engel | Reinforcement Learning with Gaussian Processes[END_REF] where the gain is computed with Gaussian Processes and where only the evaluation temporal difference can be used etc.

D. On-policy vs Off-policy

According to what has been said in subsections II-B and II-C, one can distinguish two different categories of algorithms: those based on the Bellman evaluation algorithm and those based on the Bellman optimality equation. The former algorithms are called on-policy algorithms because they evaluate the currently used policy and then improve it to finally obtain an optimal policy. The later are called off-policy algorithms since they aim at learning the Qfunction of the optimal policy from observations of an other policy. These algorithms are generally used when the system cannot be controlled during learning. It is the case when learning from a fixed set of data (the agent cannot generate new transitions than those contained in the dataset) or when there are constrains on the strategy that are admissible by the system to be controlled. It is the case in our application since there is a human driver in the loop and the learning decision maker cannot try any random sequence of action for example.

E. Model-based vs Model-free

The algorithms mentioned in subsections II-B and II-C are called model-free algorithms because they don't need a model of the dynamics of the system. That means that the transition probabilities and the reward function (the model of the MDP) are not known neither computed by the algorithm. It is also possible to evaluate these quantities either from a database or from online interactions. This gives rise to model-based algorithms. If the stateaction space is small enough to be enumerated, than the these quantities are discrete values that can be evaluated by a simple counting (maximum likelihood estimation). The DYNA-Q algorithm [START_REF] Sutton | Reinforcement Learning: An Introduction (Adaptive Computation and Machine Learning)[END_REF] proceeds this way. If not, these quantities have to be estimated as parametric functions like in the case of the Q-function in the previous section [START_REF] Strehl | An Analysis of Model-Based Interval Estimation for Markov Decision Processes[END_REF]. For example, the transition probabilities can be approximated as having a Gaussian distribution and the parameters (µ, σ) of each transition probability can be estimated through interactions. A comparison between both types of method can be found in [START_REF] Atkeson | A comparison of direct and model-based reinforcement learning[END_REF]

III. PADAS as an MDP

To cast the PADAS optimization problem described in Section I-B into the Markov Decision Process (MDP) framework [START_REF] Bellman | A markovian decision process[END_REF], one has to define state and action spaces and a reward function. The optimal policy will be the one that maximizes the expected discounted reward over the long run. Let's remind the problem we want to solve. The collision avoidance problem described in the introduction consists of devising a method of using the system's resources (capability to send warning signals to the driver and to decelerate the vehicle) such that the probability of a collision is minimized while introducing the minimum of disturbance for the driver.

We will consider that at any time t, a collision of the host vehicle with the leading vehicle occurs if the time to collision at time t drops below a threshold . Time to collision θ at time t is defined as

θ t = d t v t -v t
where d t is the distance to the next obstacle, v t is the velocity of the car and v t is the velocity of the obstacle. These variables can be measured with equipment available on cars with PADAS installed.

The state is thus made of a single continuous real value which is the estimated time to colision θ t . The elementary actions the PADAS can take are of two kind : (i) apply braking pressure equal to b times the maximum braking pressure of the vehicle (b ∈ [0, 1]), (ii) send a warning signal to the driver (warnings can be of audio,visual or haptic type). An actual PADAS action will be any combination of warnings and braking pressure application. The reward function is defined according to three different costs : a cost for the time to collision (the higher the time to collision the lower the cost), a cost for braking (again, the larger the braking, the larger is the cost) and finally a cost for sending signal (with each signal is associated a level of urgency; the more urgent the signal, the larger is the cost). Of course, a very high cost is associated to a time to collision of 0 (which means collision). Intempestive breaking and warnings are also judged to have a negative impact on the driving experience and negative rewards are associated.

IV. Experiments and Results

The experiments were divided into three phases: data collection, optimal strategy learning and testing. The learning is done in a batch manner, that is from collected data, and not online (during the driving). This choice is made so as not to disturb the drivers by changing the PADAS intervention strategy while they are driving the car. Indeed, these changes in the warning and braking strategy, meaning that into a similar situation different PADAS actions could occur, could induce unusual behaviors and stress of the drivers which will not be encountered in real scenarios.

A. Data Collection

A driving experiment has been conducted on a ScanerII3 car simulator, a fixed based system that comprises a mockup of a car with real driving controls (i.e. seat, steering wheel, pedals, gear, handbrake), a digital simulated dashboard displaying a traditional instrumental panel and a frontal projection screen where the simulated environment is displayed to the driver. This type of simulator can output data at a frequency of 20 Hz (= 0.05 s). Five subjects participated to the experiments, driving on an extra-urban and motorways scenarios for more than 1 hour each, using the ACC+ mode of the PADAS. The PADAS used an initial strategy π 0 which was based on handcrafted rules (the time to collision was discretized and to each bin was associated an action) and stochastic noise (that is random actions could sometimes be selected) was added to artificially explore the state-action space.

Trials of about 600 minutes involving different human drivers driving the simulator were conducted. The simulation trials data is organized in the form of a set of episodes. Each episode is sequence of state-action pairs of the MDP model of the collision avoidance problem described in Section III. Values of θ could be computed every 300ms which provided a large amount of transition to learn. An example of particular episode is provided on Fig. 2

B. Optimal Strategy Learning

As explained in Section II, several approaches can be envisioned to solve the optimal control problem at sight. Particularly, model-free or model-based algorithms could be investigated. In this paper is reported the result of the learning with a model-based algorithm (see Section II-E), that is a model that learns the transition probabilities of the {vehicle+driver} system. The {vehicle+driver} system is indeed a stochastic system since there are inter-and intra-driver variabilities inducing different reactions to similar situations given that the driver is different or even given that the perception of the situation is different for a same driver. Moreover, since we learn from a fixed set of data, an off policy algorithm (see Section II-D) is chosen.

A state in this MDP is a time to collision. The minimum value of the time to collision is 0 seconds and the maximum value can be considered to be 50 seconds (as far as collision avoidance is concerned, any value above 50 seconds can be considered equivalent to 50 seconds). So, a state in the MDP is a number in the interval [0, 50]. This is an infinite set. A model-based approach would require to learn transition probabilities between states. Either we learn parameters of continuous distributions, either we discretized the state space. In order to render it finite, we exhaust the interval into disjoint partitions of unequal sizes. As an example: [0, 0.5[, [0.5, 1[, [1, 1.5[, [1.5, 2[, [2, 3[, [3, 5[, [5, 7[, [7, 10[, [10, 15[, [15, 50]. These partitions (or bins) represent the states of a (finite) MDP. The portioning just described gives 10 states and was driven by expert knowledge. So, if the time to collision is say 0.73 seconds, the MDP is in the 2 nd state, if it is 2.15 seconds, the MDP is in 5 th state and so on.

The probabilities P (.|s, a) of stepping from one state (as defined hereabove) to another given that an action has been taken are computed for every tuple (s, a, s ) present in the database. The associated costs are then computed (a cost is regarded as a negative reward) and the value iteration algorithm (see Section II-B) is applied to the learnt MDP.

C. Results

Two optimal strategies were constructed based on two different partitions of the set of possible time to collisions. We refer to the two policies as π * 1 and π * 2 . For the sake of demonstration, one of both learnt strategies (π * 1 ) is explained in Table I: In the testing phase, π * 1 , π * 2 and the hand-coded strategy π 0 that served to generate the database (but without the stochastic action selection) were tested by using each of them in turn as the PADAS strategy in simulation trials involving different human drivers. Note that the hand-coded policy was based just on common sense. To assess the quality of the learnt strategies, two variables were monitored: the time to collision and the distance between the vehicles. A collision was said to have occurred if the time to collision dropped to less than 1.5 seconds or if the distance dropped to less than 1 meter. Table II gives the results of the learnt policies π * 1,2 and the handcrafted policy π 0 (this time without random actions). It is important to notice that the policies π * 1,2 have been learnt from samples generated with a policy π 0 which was acceptable for drivers (off-policy learning with an acceptable behavioral policy). Indeed, a completely random policy could theoretically provide better results since the whole state-action space could be sampled uniformly, yet it would have impacted the drivers' behavior. Table II lists the percentage of samples in which a collision occurred according to the two definitions given above. These results show that the strategies derived using the MDP approach render the driving experience safer by reducing the number of collisions and near collisions. V. Conclusions and perspectives In this paper, an approach for conceiving an optimal warning and intervention strategy for a PADAS is presented. The problem is modelled as an unknown Markov Decision Process, that is the transition probabilities are unknown because it depends on the drivers reactions to driving situation and PADAS interventions. Data collected from simulation trials with real human drivers were used to learn the unknown parts of this MDP. The optimal control problem is solved using the value iteration algorithm to obtain an optimal strategy. Since the approach consists in learning the model of the MDP, it can be regarded as a sort of model-based reinforcement learning.

Obtained results indicate that there exists a real benefit in adopting this approach: the percentage of collisions or near collisions drops down by a non-negligible amount while the optimized cost function ensure not to disturb the drivers with unnecessary warning signals. The other advantage of our approach is that the optimal strategy is constructed directly from the data. No hypothesis is thus made about driver's behavior. Moreover, the learning phase doesn't introduce bias effects on the driver's behavior neither. Finally, it shows that learning can be done from recorded information which can actually be obtained on real cars, which means that the learning could be improved and even personalized for each driver.

This work opens up interesting possibilities for conceiving intelligent systems for vehicles but also to model drivers. Indeed, the MDP approach provides the right framework to allow, as a by-product, to construct a driver behaviour model embodied in the state transition probabilities of the MDP. It could therefore be possible to simulate drivers or to infer some information about the user's perception of the situation.

In addition, the entire range of algorithms for solving MDPs, including those from the domain of reinforcement learning such as LSPI [START_REF] Lagoudakis | Least-squares policy iteration[END_REF] or FVI [START_REF] Gordon | Stable Function Approximation in Dynamic Programming[END_REF] (briefly described in Section II-C) can be used to constructing sequential decision making strategies for intelligent systems that are based entirely on observed data, and not on complex hypothesis. This would particularly be interesting since they are able to deal with continuous state-action spaces which is the case in the problem described here. Indeed, the partitioning of the state and action space described in Section IV-B introduces an aliasing problem that may lead to suboptimal strategies. The use of algorithms dealing with continuous state-action spaces could actually implicitly learn the optimal partitioning and thus produce more accurate policies.

Next steps involve the integration of this MDP model with the driver's distraction classifier, in order to improve the efficacy of the PADAS, also depending on the driver's states. In addition, we would like to integrate MDP model of PADAS with a model for driver's manoeuvres classification and prediction, based on Hidden Markov Models. In fact, the possibility to "understand" driver's intention can make the PADAS able to intervene and to support drivers in a more acceptable and appropriated way, anticipating wrong behaviors even before they can be a serious danger for drivers themselves and for the other road actors. In addition, we think to extend MDP approaches to other road traffic actors, so including not only the system "driver + vehicle", but also some specific categories of vulnerable road users, such as pedestrians, cyclists, etc.

Fig. 2 .

 2 Fig. 2. Episode example : time to collision vs time

TABLE I Learnt policy π * 1 Time to Collision In- terval Action to Take

 I

	[0, 0.5[	send emergency signal, apply maxi-
		mum brake
	[0.5, 0.5[	send danger signal, apply 80% brake
	[1, 2[	send danger signal, apply 40% brake
	[2, 3[	send danger signal, apply 20% brake
	[3, 10[	send collision warning signal, don't
		apply any brake
	[10, 50]	Send normal signal, don't apply any
		brake

see http://www.prevent-ip.org for several field test performed by Dutch Ministry of Transport or see[START_REF] Lee | Collision Warning Timing, Driver Distraction, and Driver Response to Imminent Rear-End Collisions in a High-Fidelity Driving Simulator[END_REF]

www.scaner2.com

Acknowledgment

The research leading to these results has received funding from the European Commission Seventh Framework Programme (FP7/2007-2013) under grant agreement no FP7 -218552, Project ISi-PADAS (Integrated Human Modelling and Simulation to support Human Error Risk Analysis of Partially Autonomous Driver Assistance Systems). The authors would like to specially thank the ISi-PADAS consortium that has supported the development of this research References


Joanna Norton
email: joanna.norton@inserm.fr

Guilhem De Roquefeuil

Michel David

Jean-Philippe Boulenger

Karen Ritchie

Anthony Mann

The mental health of doctor-shoppers: experience from a patient-led fee-for-service primary care setting

Keywords: general practice, mental illness, doctor-shopping, health service organisation general practice, mental illness, doctor-shopping, health service organisation

Background: doctor-shopping may reflect unmet patients needs and place a significant burden on health resources; however little is known about its underlying reasons and how it relates to mental illness and its recognition in an open access fee-for-service setting. Methods: in 2003-2004, consecutive patients of 46 French general practitioners (GPs), completed in the waiting room the self-report Patient Health Questionnaire and reported on service use in the past six months. During the consultation, GPs rated the severity of any physical and psychiatric disorder.

Results: of 1079 patients, those rated by the GP as having a common mental disorder were significantly more likely to be doctor-shoppers but only if visiting another GP because of dissatisfaction with previous care (OR=2.3, 95% CI: 1.4-3.7). A similar trend is observed for those identified as having a common mental disorder by the Patient Health Questionnaire. Among patients with a common mental disorder, 'dissatisfied' doctor-shoppers were significantly more likely to be recognised as cases by the GP, adjusting for covariates (OR=6, 95% CI: 2.1-17.2). This was not the case for those doctor-shopping for practical reasons alone.

Conclusion: doctor-shopping behaviour is associated with higher rates of mental illness and, among cases, an increased likelihood of recognition by the GP, only however for doctorshoppers dissatisfied with previous care. This suggests a benefit to letting patients choose their GP as is the case in France and change if not satisfied.

Introduction

Up until recently patients in France were able to access easily and rapidly any GP of their choice. GPs had no patient lists and no gate-keeping role, working mainly alone on a fee-forservice basis with patients reimbursed at a later stage by the state [START_REF] Bourgueil | Trois modèles types d'organisation des soins primaires en Europe, au Canada, en Australie et en Nouvelle-Zélande[END_REF].

Despite high patient satisfaction with the health care system [START_REF] Degos | Can France keep its patients happy?[END_REF], this flexibility can lead to a lack of care coordination and continuity, wasted time and expense for the State [START_REF] Von Korff | Disability and depression among high utilizers of health care. A longitudinal analysis[END_REF] [START_REF] Von Korff | Disability and depression among high utilizers of health care. A longitudinal analysis[END_REF]. Although changes were introduced in 2004 [START_REF] Bourgueil | Trois modèles types d'organisation des soins primaires en Europe, au Canada, en Australie et en Nouvelle-Zélande[END_REF][START_REF] Dourgnon | Les assurés et le médecin traitant: premier bilan après la réforme[END_REF], with lesser reimbursements for doctor-shopping (DS) patients, this service use behaviour is still possible.

DS is usually defined as frequent changes in doctor for the treatment of a same complaint or in a same illness episode [START_REF] Von Korff | Disability and depression among high utilizers of health care. A longitudinal analysis[END_REF]. Less is known about DS than other service use behaviours as it can only be studied in settings allowing for it. The reasons for DS are complex. Patient factors which have been suggested include dissatisfaction with or distrust of doctor [START_REF] Billinghurst | Why do patients change their general practitioner? A postal questionnaire study of patients in Avon[END_REF][START_REF] Guo | Illness behavior and patient satisfaction as correlates of self-referral in Japan[END_REF][START_REF] Harris | How do patients choose physicians? Evidence from a national survey of enrollees in employment-related health plans[END_REF], a lesser understanding of doctors' explanations and disbelief of diagnosis and treatment [START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF], educational background [START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF], gender [START_REF] Thomas | Assessing the outcome of making it easier for patients to change general practitioner: practice characteristics associated with patient movements[END_REF] and health status [START_REF] Harris | How do patients choose physicians? Evidence from a national survey of enrollees in employment-related health plans[END_REF]. DS patients have been found to be more chronically ill and have higher GHQ scores [START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF]. Factors related to the health care system include convenience, waiting time, fees and doctor's reputation [START_REF] Billinghurst | Why do patients change their general practitioner? A postal questionnaire study of patients in Avon[END_REF][START_REF] Bronstein | Choosing a doctor: an exploratory study of factors influencing patients' choice of a primary care doctor[END_REF][START_REF] Leung | Time costs of waiting, doctor-shopping and private-public sector imbalance: microdata evidence from Hong Kong[END_REF][START_REF] Lo | Doctorshopping in Hong Kong: implications for quality of care[END_REF][START_REF] Yeung | Waiting time and doctor shopping in a mixed medical economy[END_REF]. Studies on DS in France have only focused on the prescription of drugs, namely High Dosage Buprenorphine [START_REF] Feroni | French general practitioners' attitudes and prescription patterns toward buprenorphine maintenance treatment: does doctor shopping reflect buprenorphine misuse?[END_REF][START_REF] Pradel | Impact of a prescription monitoring program on doctor-shopping for high dosage buprenorphine[END_REF], DS acting as a mechanism for obtaining multiple prescriptions and accounting for 15% to 20% of all prescriptions [START_REF] Pradel | Impact of a prescription monitoring program on doctor-shopping for high dosage buprenorphine[END_REF].

The objective of this study is to explore the associations between DS, common mental disorders (CMD) and their recognition. Of particular interest is to discover whether DS patients have similar rates of CMDs when considering the reasons for DS and whether this influences GP case-recognition.

Methods

The study methods have been described elsewhere [START_REF] Norton | Use of the PRIME-MD Patient Health Questionnaire for estimating the prevalence of psychiatric disorders in French primary care: comparison with family practitioner estimates and relationship to psychotropic medication use[END_REF]. Briefly, the study (2003)(2004) was carried out in and around the city of Montpellier, in a sample of GPs practicing in an urban area (with 249 registered GPs for a population of 140.000) and a semirural area (with 73 registered GPs for a population of 80.000). For each GP, a research assistant invited all eligible consecutive patients entering the waiting room to complete questionnaires until 25 patients per GP had participated. Exclusion criteria were: age (< 18 years), not living in the study area and not consulting for one's self.

Instruments

During the consultation, GPs completed a short questionnaire with an estimation of severity of physical and psychiatric symptoms on a 5-point scale with a clear indication that ratings of three and above were considered as cases.

Patients completed self-administered questionnaires of which: the anxiety, depression and somatoform disorder modules of the DSM-IV criteria based Patient Health Questionnaire (PHQ) [START_REF] Spitzer | Validation and utility of a self-report version of PRIME-MD: the PHQ primary care study. Primary Care Evaluation of Mental Disorders. Patient Health Questionnaire[END_REF], the Brief Disability Questionnaire (BDQ) [START_REF] Ustun | Mental Illness in General Health Care. An International Study[END_REF], and adapted versions of the Client Service Receipt Inventory (CSRI) [START_REF] Beecham | Costing psychiatric interventions[END_REF] and the Short Explanatory Model Interview (SEMI) [START_REF] Lloyd | The development of the Short Explanatory Model Interview (SEMI) and its use among primarycare attenders with common mental disorders[END_REF].

Analysis

The analysis was carried out on the 1079 patients with full datasets for the variables selected for the study, which represents 94% of the total sample. Percentages are presented for categorical variables, medians and ranges for skewed continuous variables. Given the two-stage sampling process, marginal generalised estimating equation (GEE) models were used to test associations. Univariate and multivariate analyses were performed to test the associations between explanatory variables and DS as the dependent variable. Variables with p-values <0.10 in the univariate analysis were entered in the multivariate models. Odds ratios (OR) with 95% confidence intervals (CIs) are given. A second set of analyses was carried out on the subset of PHQ CMD cases with GP caserecognition as the dependent variable. Statistical analyses were performed using SAS version 9.1 (SAS Institute, Cary, NC, USA).

Results

Description of the GP sample

The acceptance rate was 32.8%. Participating GPs were significantly younger and acceptance significantly higher in the rural area. In all, 46 GPs participated, 31 in the urban area and 15 in the semi-rural area.

Of the sample, 56% were male. Median age was 45 (range: 32-59) and the same for both sexes; 60% had been practicing for at least 10 years and 80% declared having received some form of mental health training in the last 3 years. Two-thirds practiced alone.

Description of the patient sample

The response rate was 89.8%. Of the 1079 patients, 61.8% were female. Median age was 42 (range: 18-93), 49.4% were married and 33.6% single; 33.5% had a high (post-school) educational level. Regarding current occupation, 40.1% were working, 10.8% unemployed, 14.1% students and 35% retired or other [START_REF] Norton | Use of the PRIME-MD Patient Health Questionnaire for estimating the prevalence of psychiatric disorders in French primary care: comparison with family practitioner estimates and relationship to psychotropic medication use[END_REF]. Overall, 27.1% of patients reached criteria for a CMD (ie. major depression, other depressive disorders, panic disorder, other anxiety disorders or somatoform disorder) on the PHQ. GPs rated 28.4% as having a CMD (mild, moderate or severe psychiatric symptomatology). Of the 292 PHQ CMD cases, 51% were classified as having a CMD by the GP (case-recognition).

Of the sample, 27.7% were FAs (6 or more GPs visits over the past six months, including the survey-day visit). Regarding DS, 72% of the patients had always visited the same GP, 10% two, 15% three and 3% four or more different GPs. The main reason for visiting a different GP was geographical (patient or GP on holiday, newly moved, travelling for work, retired) (65% of DSs), followed by avoiding long waiting times (15.6%), looking for a convenient appointment (14.6%), not satisfied with previous GPs (9.3%), looking for advice from a GP with a speciality (8.9%), looking for a GP accredited to sign administrative papers or taking credit card (7.4%), wanting a GP with good listening skills (7%) and wanting a second advice (6.6%). In sum, 72% always saw the same GP (non-DSs), 18.4% doctor-shopped for practical reasons only ('practical DSs') and 9.6% because they were dissatisfied with care previously received ('dissatisfied DSs').

Association between CMDs and DS

Among dissatisfied DSs, significantly more (44.7%) met CMD criteria, compared to practical DSs (30.7%) and non-DSs (23.9%) (Table 1). GP ratings of psychiatric disorder are significantly lower for practical DSs (22.1%), than for dissatisfied DSs (51.5%) or non-DSs (26.9%) Patients always seeing the same GP and practical DSs have similar rates of FA (27.1% and 25.1%, respectively), compared to significantly higher rates among dissatisfied DSs (36.9%).

-insert Table 1 here-In the multivariate analysis, patients rated as cases by the GP were significantly more likely to be dissatisfied DSs (Table 2). Patients with a CMD according to the PHQ were significantly more likely to be dissatisfied DSs when adjusting for covariates in Model 1, but not when adjusting for psychological reason for visit and FA behaviour (Model 2). There were not interactions between CMDs and FA behaviour on DS.

-insert Table 2 here-

The effect of DS on GP case-recognition

Whatever the level of adjustment, dissatisfied DSs were significantly more likely to be recognised by the GP (Table 3).

-insert Table 3 here-

Discussion

Limits

The study limitations are described in detail elsewhere [START_REF] Norton | Use of the PRIME-MD Patient Health Questionnaire for estimating the prevalence of psychiatric disorders in French primary care: comparison with family practitioner estimates and relationship to psychotropic medication use[END_REF]. The main limitation is the low but common GP participation rate (33%), selecting GPs particularly interested in mental health issues with a high level of training in this area. This, along with the study effect, will no doubt have led to an overestimation of case-recognition [START_REF] Freeman | Evolving general practice consultation in Britain: issues of length and context[END_REF]. DS is recorded over the past six months which is likely to be affected by patient recall bias. Also mental illness influencing DS requires that the patient had the condition at the beginning of the 6-month period which we are not able to establish. Furthermore, DS behaviour is usually defined in terms of multiple visits for a same health condition [START_REF] Von Korff | Disability and depression among high utilizers of health care. A longitudinal analysis[END_REF], however reason for visit was only recorded for the survey-day visit. Despite these limits, the study included a large sample of GP attenders, with a high participation rate, assessed by internationally accepted measures.

Association between CMD and DS

As found elsewhere, DS patients tend to be younger, more often single, with a higher level of education [START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF]. The percentage of unemployed is considerably higher among dissatisfied DSs. With regard to disability, personal and social problems, mental illness as rated by the GP or using the PHQ, practical DSs and non-DSs have similar and lower rates than dissatisfied DSs. On the survey-day, dissatisfied DSs were more likely to be visiting for a psychological reason, according to the patient or the GP, and were more likely to be presenting with psychological symptoms.

In the multivariate analysis, patients rated as CMD cases by the GP were more likely to be dissatisfied DSs but not practical DSs. Although weakened by adjusting for covariates, a similar trend is shown for patients reaching CMD caseness criteria on the PHQ, for both types of DSs. It is difficult to compare these results with other studies as those that examine factors associated with DS do not take into account the reasons for this behaviour [START_REF] Billinghurst | Why do patients change their general practitioner? A postal questionnaire study of patients in Avon[END_REF][START_REF] Guo | Illness behavior and patient satisfaction as correlates of self-referral in Japan[END_REF][START_REF] Harris | How do patients choose physicians? Evidence from a national survey of enrollees in employment-related health plans[END_REF][START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF][START_REF] Thomas | Assessing the outcome of making it easier for patients to change general practitioner: practice characteristics associated with patient movements[END_REF]. Sato et al. for example found higher GHQ scores among undifferentiated DS patients [START_REF] Sato | Doctor-shopping patients and users of alternative medicine among Japanese primary care patients[END_REF].

GP ratings of CMD may be higher among dissatisfied DS because the latter present more often with psychological symptoms. In our study, the percentage of patients visiting the GP for a psychological reason on the survey-day was over twice as high among dissatisfied DSs than others. Even when adjusting for psychological reason for survey-day visit, patients rated as having a CMD by the GP were more likely to be dissatisfied DSs.

The effect of DS on GP case-recognition

Among CMD cases, dissatisfied DSs were more likely than non-DSs to have their disorder recognised by the GP. Despite large confidence intervals, this relationship remained significant after adjustment for covariates. The higher recognition rate among dissatisfied DSs could be seen as not fitting with the large body of evidence showing that GPs are better able to recognise psychiatric disorders in patients they know well [START_REF] Bushnell | Frequency of consultations and general practitioner recognition of psychological symptoms[END_REF][START_REF] Marks | Determinants of the ability of general practitioners to detect psychiatric illness[END_REF][START_REF] Rosenberg | Determinants of the diagnosis of psychological problems by primary care physicians in patients with normal GHQ-28 scores[END_REF]. However, firstly more than half of DSs declared they were visiting their usual GP on the survey-day. Secondly, dissatisfied DSs, regardless of the presence of a CMD, were more likely than others to be consulting for psychological reasons and presenting psychological symptoms, which have been shown to increase recognition [START_REF] Aragones | Somatized depression in primary care attenders[END_REF][START_REF] Maginn | The detection of psychological problems by General Practitioners--influence of ethnicity and other demographic variables[END_REF][START_REF] Tylee | The importance of somatic symptoms in depression in primary care[END_REF].

In conclusion, these findings from a large sample of French GP attenders suggest that it is important to consider the reasons for DS, the prevalence of CMDs being particularly high in dissatisfied DSs. Although we cannot firmly conclude that patients with CMDs are more likely than others to DS, we have shown that among CMD patients, dissatisfied DSs are more likely to have their symptoms recognised by the GP. This may be because they select the GP for this ability or because they have learnt to better present their needs, thereby alerting the GP to possible underlying psychiatric symptoms. If this leads to recognition and treatment, what can be regarded as undesirable patient behaviour in terms of continuity of care would turn out to be an effective means of obtaining care. This suggests it can be beneficial to let patients choose their GP as is the case in France and change if not satisfied.

Table 1 .

 1 Sociodemographic and clinical characteristics of the sample according to DS behaviour (N=1079)Model 1: adjusted for socio-demographic variables (age (for 10 year increase), sex, marital status, education, occupation), personal and social problems, disability (2 levels) Model 2: further adjusted for psychological reason for visit and FA

				Doctor-shopping
		No (1)	Yes, practical (2)	Yes, dissatisf. prev. care (3)	p-values
		(n=777)	(n=199)	(n=103)	(2 v. 1) (3 v. 1)
	Study zone (urban)	64.2	72.9	78.6	0.11, 0.06
	Gender, female (%)	60.0	65.8	68.9	0.06, 0.09
	Age (median, IQR) for 10 yr increase	46 (33)	30 (26)	33.5 (21.5)	<.0001, <.0001
	Marital Status (%)
	Married	52.6	42.7	38.8
	Single	28.6	48.2	41.8
	Div/Sep/Widowed	18.8	9.1	19.4	<.0001, 0.03
	Educational level (%)
	Low	30.9	17.6	26.2
	Medium	39.1	39.7	32.0
	High	30.0	42.7	41.8	0.001, 0.21
	Occupation (%)
	Employed	40.9	38.2	37.9
	Unemployed	9.0	11.0	23.3
	Student	8.8	32.7	18.4
	Retired, other	41.3	18.1	20.4	<.0001, <.0001
	Bothered by personal/social probs. (%)
	Not at all	59.4	47.7	34.0
	A little	31.3	42.2	46.6
	Alot	9.3	10.1	19.4	0.06, 0.0003
	Disability (%)
	None/mild	63.1	72.9	55.3
	Moderate	25.9	23.1	26.2
	Severe	11.0	4.0	18.5	0.01, 0.18
	Physical disorder (GP rating) (%)	61.4	61.8	63.1	0.99, 0.85
	Psychiatric disorder (GP rating) (%)	26.9	22.1	51.5	0.19, <.0001
	Common mental disorder (PHQ quest.) (%)	23.9	30.7	44.7	0.07, <.0001
	FA: No. of GP visits > 6 (%)	27.2	25.1	36.9	0.74, 0.02
	On the survey-day:
	Survey-day GP is usual GP (patient) (%)	89.8	54.6	54.4	<.0001, <.0001
	Reason for visit (patient): psychological (%)	6.1	7.0	17.5	0.66, 0.0002
	Reason for visit (GP): psychological (%)	5.8	7.1	16.5	0.39, 0.0008
	Presenting symptoms (GP)	23.3	20.1	47.6	0.38, <.0001

Table 3 .

 3 GP case-recognition according to GP service use: multivariate model (N=292) Model 1: adjusted for age, sex, occupation, personal and social problems, disability and physical health problem (GP rating), FA Model 2: further adjusted for seeing personal GP on survey-day Note: it was not possible to adjust for reason for visit: psychological, as this concerned only 2 CMD patients not recognised by the GP

			GP case-recognition
		% of cases	Model 1	Model 2
		recognised by GP
	Doctor-shopping
	No	51.1	1	1
	YES -practical reason	31.7	0.6 (0.3-1.2) 0.15	0.8 (0.40-1.74) 0.62
	YES -dissatisf. prev. care	76.1	3.0 (1.2-7.1) 0.014	6.0 (2.1-17.2) 0.0009