Simon Enjalbert

Frédéric Vanderhaegen

A hybrid reinforced learning system to estimate resilience indicators

Keywords: Resilience engineering, Learning, Man-Machine systems

This paper describes a learning system based on resilience indicators. It proposes a hybrid learning system to estimate Human-Machine System performance when facing unprecedented situations. Collected data from various criteria are compared with data estimated using the local and the global resilience indicators, to give both instantaneous and over-time Human-Machine System states. The learning system can be composed of two different, separate reinforcement functions; the first allowing reinforcement of its own system knowledge and the second allowing reinforcement of its estimation function. When used together in a hybrid approach, the resilience indicator estimation should be improved. The learning system is then applied in a simulated air transport context and the impact of each reinforcement function on resilience indicator estimation is assessed. The hypothesis on performance of hybrid reinforcement learning is confirmed and it provides better results than those obtained by the knowledge based reinforcement or the estimation based reinforcement alone.

1. Introduction [START_REF] Ouedraogo | How to learn from the resilience of human-machine systems?[END_REF] defined resilience as the positive ability of a Human-Machine System (HMS) to recover from or adapt to critical situations. The recovery function consists of getting back to the previous normal functioning state and the adaptation function aims to provide the system with a new stable functioning state. A large amount of research has been performed in research laboratories about system safety and security in transport or industry based on this concept [START_REF] Orwin | New indices for quantifying the resistance and resilience of soil biota to exogenous disturbances[END_REF][START_REF] Pérez-España | An inverse relationship between stability and maturity in models of aquatic ecosystems[END_REF][START_REF] Enjalbert | Validation of a unified model of driver behaviour for train domain[END_REF][START_REF] Cacciabue | Unified driver model simulation and its application to the automotive, rail and maritime domains[END_REF].

Some of this research involves assessment based on various criteria. These system evaluation criteria mainly concern human or machine behaviours or their effects, or the occurrence or consequences of external perturbations. These effects or perturbations relate for instance to human workload ( [START_REF] Vanderhaegen | Multilevel organization design: the case of the air traffic control[END_REF], to human errors [START_REF] Lin | Integrating human behavior modeling and data mining techniques to predict human errors in numerical typing[END_REF], to the quality or the production of services [START_REF] Polet | Human behaviour analysis of barrier deviations using a benefit-cost-deficit model[END_REF], and to the quality of cooperation or learning activities [START_REF] Vanderhaegen | Principles of cooperation and competition: application to car driver behavior analysis[END_REF]. Therefore, resilience emerges in a risk management process and relates to the system capacity to survive both planned and unexpected hazardous events [START_REF] Enjalbert | Assessment of transportation system resilience[END_REF]. Unprecedented situations are defined as events with a very low frequency of occurrence and/or which may have catastrophic consequences for HMS.

This paper focuses on the learning system developed to estimate resilience indicators. The reinforcement functions of the learning system concern reinforcement of the system knowledge and reinforcement of the estimation parameters. This hybrid approach has been developed and tested on a flight simulator during an in-flight refuelling activity involving a team composed of four people. Several unexpected events with potential catastrophic consequences are incorporated and data collected on HMS during unprecedented situations are used by the hybrid reinforced learning system to estimate the local and the global resilience indicators.

In the second section of the paper, the need for resilience indicators and the principles of the reinforced iterative learning approaches are presented in order to introduce the contribution of the present work. In the third section, the generic architecture of the learning system is detailed with specific focus on reinforcement functions. Finally, a validation example showing the impact of reinforcement functions on resilience indicator estimation is described and the effectiveness of hybrid reinforcement is demonstrated.

Learning approaches and resilience assessment

Several concepts of learning can be found in literature. For instance, learning by imitation or observation consists in copying a given behaviour, or a sequence or a repetition of behaviours [START_REF] Chella | A cognitive framework for imitation learning[END_REF][START_REF] Calinon | On learning, representing, and generalizing a task in a humanoid robot[END_REF]. When facing a new situation for which no knowledge is defined, trial-and-error based learning should be applied [START_REF] Rose | Selforganization in decentralized networks: A trial and error learning approach[END_REF]. A redundant learning system is another way to engage the learning capacity of the system [START_REF] Vanderhaegen | Reinforced learning systems based on merged and cumulative knowledge to predict human actions[END_REF]. Cooperative learning or co-learning are then useful for exchanging data between decision makers in order to understand the learning process or to share knowledge [START_REF] Doisy | The impact of human-robot interface design on the use of a learning robot system[END_REF]. Effective techniques, characterized by efficient self-learning and adaptivity abilities, have been employed to construct learning systems [START_REF] Xu | Iterative learning control design without a priori knowledge of the control direction[END_REF][START_REF] Liu | An iterative adaptive dynamic programming algorithm for optimal control of unknown discrete-time nonlinear systems with constrained inputs[END_REF][START_REF] Norrlöf | A note on causal and cite iterative learning control algorithms[END_REF][START_REF] Wiering | Ensemble algorithms in reinforcement learning[END_REF]. Many of these involve reinforcement learning or reinforced learning. Reinforcement learning is usually applied for repetitive tasks, in order to minimize tracking errors. If the error reduction is successful, the reinforcement is based on a reward for managing knowledge. Other authors prefer using the vocabulary of reinforced learning because their interest is not limited to repetitive tasks and error tracking reduction. [START_REF] Vanderhaegen | A benefit/cost/deficit (bcd) model for learning from human errors[END_REF] focused on the learning from human errors in order to provide human operators with decision support tools .

In this study, the learning approach objective is to estimate missing or immeasurable data from Human-Machine System facing unprecedented situations. In the first section, a theoretical analysis based on extended State of the Art of Iterative Learning Control (ILC) systems is proposed. Then, in a second section, indicators based on resilience criteria for HMS are developed.

Finally, these indicators are adapted to reinforced learning approaches.

Iterative learning control systems

The feedforward process aims at assessing the future possible decisions regarding the current system states and the management of the previous ones. The feedback aims at recovering possible erroneous knowledge, at refining knowledge or at creating new knowledge [START_REF] Vanderhaegen | Autonomy control of human-machine systems[END_REF]. So the feedforward-feedback mechanism that consists in using the current knowledge related to previous activities in order to calculate the future ones. A great number of research works have proposed feedback and/or feedforward controllers using different methods in order to reach the mentioned objectives.

There are frequency based approach (related to iteration frequency) or tem-poral based approach (related to timing process).

Iterative Learning Control (ILC) systems are used to benefit from the repetitive nature of the tasks as experience gained to compensate for the poor or incomplete knowledge of the plant model and disturbance. The repeatability of the task determines the learning ability of the ILC. Current (e i ) in Equation (1) and previous (e i-1 ) in Equation (2) tracking errors , and previous input u i-1 are used to assess the current input u i in Table 1.

The recursive process of ILC technique to assess the current characteristics and to improve tracking control performance in batch processes is given in Equation (3). The formalism can be seen as a generalization of the previous ones; the control is done regarding the previous errors at certain level because of limited memory capacity. A feedback-feedforward structure for the trajectory tracking of a linear Direct Current motor is given in Equation (4). The same structure for sharp tracking control of a manipulator robot, by employing a saturated input γ which limits the control input within a reasonable bound, was also proposed. The corresponding learning control updating law is given by Equation (5). The class of non-linear systems to which the proposed learning scheme can be applied is then extended. A combined feedback-feedforward controller and disturbance observer designed for a direct drive motion control was proposed in Equation ( 6). The digital disturbance observer is included in the proposed feedbackfeedforward control structure to compensate for disturbances (friction and cogging effects).

Finally, a framework for the assessment of the consequences of human errors based on learning and prediction of the actions of a human operator is given in Equation (7) in Table 1. These processes are modelled by using the itera-

u i = u i-1 +G f eedback (e i ) (2)
Radac and Precup (2016) [START_REF] Lee | Iterative learning control applied to batch processes: An overview[END_REF]

u i = u i-1 +G 1 (e i-1 )+• • •+G p (e i-p ) (3) Lee et al. (2000) u i = u i-1 +G f f e i-1 +G f b e i (4) Jang et al. (1995) u i = γν i = γ(u i-1 +G f f e i-1 +G f b e i ) (5)
Yan and Shiu ( 2008)

u i = u f f i + u f b i -u d i = G f f (e i-1 , u i-1 ) + G f b e i -G d (e i-1 , u i-1 ) (6)
Vanderhaegen et al. (2009) u i = e i +G((e i-1 , u i-1 ), . . . , (e 0 , u 0 )) (7) [START_REF] Polet | Iterative learning control based tools to learn from human error[END_REF] tive learning control concept and by integrating it in a feedforward-feedback approach.

ILC has become a competitive control method through the development of different learning controllers for many applications, essentially in robotic operations, chemical processes and motor drive machines. Initially the ILC input signal is formed using the error from previous iterations, i.e., the input u i is computed using the previous input u i-1 and e i-1 in so-called Previous Cycle Learning (PCL) in Equation ( 1) or recursively e i-1 , . . . , e i-p in Equation ( 3). Several authors have computed the input u i using the current tracking error e i in so-called Current Cycle Learning (CCL) in Equation ( 2).

Then, it has been proposed to combine the current error, e i with the previous one e i-1 , when forming u i in Equation ( 4), ( 5) and ( 6). This approach leads to a causal relationship between the current error and the input signal. It can be seen that PCL and CCL are functioning a complementary manner with the aim to improve the control performance through Previous and Current Cycle Learning (PCCL) structure, complementary role of feedback and feedforward structures.

The formalisms, summarised in Table 1, are used to deal with machines processes control (optimize robot or motor motion) during repetitive tasks mostly tracking errors performance control by managing a static knowledge.

These control processes are not applied to problems involving humans and do not manage knowledge in unexpected or unprecedented situations. An extended approach by using the previous couples ((e i-1 , u i-1 ), . . . , (e 0 , u 0 )) was proposed with feedforward-feedback learning control systems having their updating laws mostly depending on current and/or previous errors in Equa-tion ( 7). The originality of this model is that it is applied to HMS with the aim to predict human errors. It combines feedforward-feedback processes and use predefined knowledge that is reinforced or corrected regarding the observed previous couples.

A State of the Art has been realized to compare different structures of the feedforward and/or feedback Iterative Learning Control systems in order to select the more appropriate one or to build an efficient one, for improving knowledge on known situations and for creating knowledge related to new situations. Therefore, the proposed article extends the Iterative Learning Control concept by proposing a hybrid reinforced learning structure that reinforces the learning by controlling two criteria of learning errors: errors between knowledge and error between predictions by taking into account matrices of data instead of vectors of data. Moreover, this new structure is applied to predict resilience indicators.

Resilience indicators

The proposed learning contribution should be able to estimate instanta-

U ki =    u ki i i=1 u ki    (8)
An iteration i is a discrete event at the end of a time window, that enables synchronization between HMS to be compared depending on the situation.

The term U i is then a vector with k rows, representing the k resilience criteria for iteration i in Equation ( 9).

U i =         U 1i U 2i . . . U ki         (9)
Finally, in Equation ( 10), the matrix U I is incremented, by iteration 1 to i with i columns based on U ki vectors.

U I =         U 11 U 12 . . . U 1i U 21 U 22 . . . U 2i . . . . . . . . . . . . U k1 U k2 . . . U ki         (10)

Reinforced learning systems

Reinforcement or reinforced learning systems are mainly based on the iterative learning control principle illustrated in Figure 1 with K the content of a knowledge composed of K I matrices from precedent 'similar' situations.

Iterative learning control consists, for iteration i and for each of the k resilience criteria, of using the previous tracking errors ε ki-1 from a given reinforcement function G f eedback and the previous state of HMS in terms of resilience indicators u ki-1 from U i-1 in U I-1 to estimate the current state of HMS u * ki . This process is repeated for the k resilience criteria and can be generalized as in Equation (11).

U * i = U i-1 + G f eedback (ε i-1 ) (11)
The tracking error, denoted ε i for iteration i, is assessed in Equation ( 12) by evaluating the differences between the actual vector U i from matrix U I and the estimated vector U * i for the k resilience criteria.

ε i =         ε 1i ε 2i . . . ε ki         =         U * 1i -U 1i U * 2i -U 2i . . . U * ki -U ki         =                        u * 1i -u 1i i i=1 (u * 1i -u 1i )       u * 2i -u 2i i i=1 (u * 2i -u 2i )    . . .    u * ki -u ki i i=1 (u * ki -u ki )                        (12)
Other approaches integrate a feedforward-feedback process in order to take into account possible future tracking errors with another function called G f eedf oward in Equation ( 13):

U * i = U i-1 + G f eedf orward (ε i ) + G f eedback (ε i-1 ) (13)
A way to simplify this approach in Equation ( 14) considers the tracking errors as input data ε i , and the current result of the function (output) is defined by combining the previous pairs (ε i-1 , U i-1 ):

U * i = ε i + G f eedback (ε i-1 , U i-1 ) (14)
The current and previous errors ε i , ε i-1 , ... ε 1 , considered as inputs for estimation of HMS resilience indicators, can be composed from the benefits, costs and potential deficits or dangers of a given human action [START_REF] Vanderhaegen | A benefit/cost/deficit (bcd) model for learning from human errors[END_REF].

At a given iteration i, the reinforced learning system aims to estimate the local and global indicator values, termed U * i in Equation ( 15) relating to the previous actual change in U I-1 , based on the errors between the previous estimation, the actual value and the content of a knowledge K:

U * i = G f eedf orward (U I-1 ) + G f eedback (U I-1 , U * i-1 , K) (15)
The knowledge K is composed of K I matrices, each containing the local and the global indicator values for iterations 1 to i. The structure of K I is then the same as that of matrix U I for iteration i. 3. Principle of the hybrid learning system 3.1. Generic architecture of the learning system

The generic hybrid learning system architecture is an iterative process, depicted in Figure 2. Each iteration in the G f eedf oward function aims to determine the presence of a similar matrix U I-1 in the knowledge K with the objective of estimating local and global resilience indicators U * i . In terms of the inputs U I-1 , the G f eedf oward function searches for the K I-1 in K, denoted E I-1 , that is similar to U I-1 , in order to identify vector U * i that is assumed to be equal to E i . The initial estimation of U * i is realized in Equation ( 16) by applying this G f eedf oward (U I-1 ) function based on the Euclidean norm:

G f eedf orward (U I-1 ) = U * i , U * i = E i , ∀K I-1 ∈ K, ||E I-1 -U I-1 || = M in||K I-1 -U I-1 || (16)
After this, two kinds of reinforcement could be applied to improve estimation performance by using the input matrix U I-1 and previous error ε i-1 :

• Reinforcement of the current knowledge.

• Reinforcement of the current estimation.

To quantify the impact of the reinforcement functions, the difference between the Mean Absolute Errors (MAE), and the Mean Squared Error (MSE) are assessed. The reinforcement processes aim to refine this estimation by integrating the results of the G f eedback (U I-1 , U * i-1 , K) function. At a given iteration i, the first reinforcement integrates the knowledge K by considering the errors between the inputs U I-1 and the knowledge content, whereas the second reinforcement process takes into account the ε i-1 error between U i-1 and U * i-1 made at the previous iteration i -1. The corresponding G f eedback functions are then respectively renamed G f bk k in Equation( 17) for Knowledge reinforcement and G f bk e in Equation ( 18) for Estimation reinforcement, and are defined with the following inputs:

• Knowledge reinforcement:

G f eedback (U I-1 , U * i-1 , K) = G f bk k (U I-1 , K) (17)
• Estimation reinforcement:

G f eedback (U I-1 , U * i-1 , K) = G f bk e (U I-1 , U * i-1 ) (18)

Knowledge reinforcement

The first reinforcement principle aims to reinforce the knowledge content related to the gaps between the matrix content K I-1 and the inputs U I-1 .

The reinforcement consists, for iteration i, of searching for the winner vector,

denoted W i-1 in W I-1 in K I-1
, which is similar to the input vector U i-1 in U I-1 . The knowledge reinforcement then proceeds in two steps. The first step, G f bk node , aims to reinforce the W I-1 matrix content and the second step, G f bk base , consists of merging this new knowledge with the other knowledge matrices in Equation ( 19).

G f bk k (U I-1 , K) = G f bk base (G f bk node (U I-1 , K)) (19)
The G f bk node (U I-1 , K) function in Equation ( 20) aims to identify the W i-1 vector in W I-1 defined as follows:

G f bk node (U I-1 , K) = W I-1 , ∀K I-1 ∈ K, ||W I-1 -U I-1 || = M in||K I-1 -U I-1 || (20)
The reinforcement of W I-1 is achieved with the G f bk node (U I-1 , K) function in Equation ( 21):

G f bk node (U I-1 , K) = W reinf orced I-1 , W reinf orced I-1 = W I-1 + (W I-1 -U I-1 ) = 2 * W I-1 -U I-1 (21)
The reinforcement of the entire knowledge is performed using the G f bk base (U I-1 , K) function in Equation ( 22):

G f bk base (U I-1 , K) = K reinf orced I-1 , ∀K I-1 ∈ K, K reinf orced I-1 = W reinf orced I-1 , K reinf orced I-1 = K I-1 + (K I-1 -W reinf orced I-1 ) = 2 * K I-1 -W reinf orced I-1 (22)
The estimation of U * i becomes, in Equation ( 23):

U * i = G f eedf orward (U I-1 ) + G f bk base (G f bk node (U I-1 , K)) (23)

Estimation reinforcement

For all iterations, the estimation error between U i and U * i , denoted ε i , is assessed using the G f bk e (U I ) function in Equation ( 24) that is dedicated to this vector assessment:

G f bk e (U I , U * i ) = ε i (24)
If the error of the previous iteration noted ε i-1 is known, then, this assessment can be approximated in Equation ( 25) as follows:

G f bk e (U I , U * i ) = ε i = 1 2 * (ε i + ε i-1 ) = 1 2 * (ε i-1 + U i-1 -U * i-1 ) (25)
Therefore, the estimation of U * i is carried out using this error and by applying the G f eedf orward (U I-1 ) function in Equation ( 26):

U * i = G f eedf orward (U I-1 ) + ε i (26)

Hybrid reinforcement

The hybrid reinforcement integrates both the knowledge reinforcement and the estimation reinforcement. The G f eedback

(U I-1 , U * i-1 , K) function is composed of the G f bk e (U I-1 , U * i-1
) function and the G f bk k (U I-1 , K) function, in Equation ( 27):

G f eedback (U I-1 , U * i-1 , K) = G f bk e (U I-1 , U * i-1 ) + G f bk k (U I-1 , K) (27)
After integrating the component functions characteristics, this function becomes, in Equation ( 28):

G f eedback (U I-1 , U * i-1 , K) = ε i + G f bk base (G f bk node (U I-1 , K)) (28)
The global application for determining U * i is then given by Equation ( 29):

U * i = G f eedf orward (U I-1 ) + ε i + G f bk base (G f bk node (U I-1 ), K) (29)

Evaluation of hybrid reinforced learning system

The Hybrid reinforced learning architecture was tested using a military

Experimental protocol

The experimental scenario was inspired by a real incident. Initially, smoke accompanied by a burning smell is detected in the cabin. Then, a series of apparently unlinked faults occur (e.g., frost on the windows, loss of fuel indications, an overheating transformer, smoke). The aircraft is over the ocean and cannot land. The problem is an electrical failure and is located in a specific area of a generator. Its fuse, which is poorly visible, has blown. In fact, all the failed components have the same origin, but expert opinion is divided between two possible causes. Thus, the team has to face an ambiguous or uncertain situation. Facing these successive faults, the team has to make sense of the situation in order to apply the correct procedures. They are not expected to know the recovery rules, but they have all the manuals with which to identify them.

Despite teams differences, comparison between teams could be achieved in terms of the criteria selected for this study, aggregated into so-called local and global resilience indicators. Several criteria were defined in order to evaluate the general Human-Machine System development: criteria related to system safety, human workload, and the team mission. These criteria are the main factors concerning system performance in the event of a major mishap and are evaluated by experts. The goal of the experiment was to observe the reaction of teams faced with unprecedented situations and to estimate the resilience indicators. The specific hypothesis to be tested during the experiment mainly concerned the way teams reacted (procedures, etc.) but is not described in this paper, since here we are focusing on the estimation of resilience indicators. A more detailed discussion and description has been realised by [START_REF] Ouedraogo | How to learn from the resilience of human-machine systems?[END_REF].

U i =                     u m i i i=1 u m i       u s i i i=1 u s i       u w i i i=1 u w i                     (30)
For iteration i, the system state in terms of local and global resilience indicators must be estimated. The resulting matrix, denoted U I , is composed of vectors U i depicted in Equation ( 30), containing the same number of rows as there are k criteria of the resilience estimation:

• The team mission criterion u m i is the percentage success in achievement of this mission.

• The safety criterion u s i relates to the recovery efficiency to faults.

• The human workload criterion u w i is linked to the number of interactions between staff members (i.e., frequency of communication, actions) and between the staff and the technical system (e.g., standard procedures, applied actions).

All the u ki were given a score by experts between 0 and 1; with 1 indicating maximum resilience to the situation (success of mission or aircraft safe or 100% workload available) and 0 the minimum for the worst situations with respect to the resilience criteria (mission failed or aircraft crashed or overloaded operators). Then i i=0 u ki is assessed between 0 and i.

Reinforcement impact assessment

At iteration i, |E ki |, the absolute error for criteria k, is calculated using Equation(31).

|E ki | = 1 2 * |u * k i -u k i | + 1 i * i i=1 |u * k i -u k i | (31)
Then |E i | is calculated to provide a single value as the error of estimation by merging all criteria with the same relative weighting. The result is in the range [0; 1] in Equation (32).

|E i | = 1 k * k k=1 |E ki | (32)
The Mean Absolute Error (MAE) of estimation is summed for i iterations using Equation ( 33). local resilience global resilience Teams ∆ER ∆KHR ∆ER ∆KHR Team 1 -0,29 -0,53 -0,15 -0,15

M AE = 1 i * i i=1 |E i | (33)
Team 2 0,05 -0,11 0,19 -0,06 Team 3 0,06 -0,20 -0,50 -0,27

Team 4 -0,13 -0,34 0,06 -0,28

Team 5 -0,28 -0,24 -0,05 -0,12

Team 6 -0,43 -0,64 -0,33 -0,49

Average -0,17 -0,34 -0,13 -0,23

Next, the difference in MAE between two functions is calculated at the end of the experiment to evaluate the reinforcement impact on the estimation process with a result in the range [-1; 1].

local resilience global resilience

Teams KR ER HR KR ER HR Team 1 1,31 0,41 0,32 0,63 0,33 0,49 Team 2 0,84 0,46 0,67 0,66 1,14 0,55 Team 3 0,70 0,88 0,40 0,76 0,43 0,43 Team 4 0,71 0,53 0,21 0,55 0,28 0,23 Team 5 0,82 0,71 0,62 0,09 0,15 0,07 Team 6 0,83 0,51 0,11 0,75 0,44 0,20 Average 0,87 0,58 0,39 0,57 0,46 0,33

21

The results are reinforced by

M SE = 1 i -1 * i i=2 (E i -E i-1 ) 2 (34)
The function with the lowest MSE is assumed to be the most efficient, because the difference between the estimated and measured vector was minimized. A value of 0 indicates a perfect estimation of resilience indicators with no error in the three selected criteria and 4 indicates the maximum error. In because the team members apply the correct manoeuvres in an incorrect sequence. the team 2 results are by far the worst in the experiment, because the team members did not notice smoke in the early stage of the experiment and apply the correct manoeuvres with delay. There is nothing in particular to note regarding the team 3 results, except that ER in local resilience is slightly better than KR but worse than HR. Finally, teams 1 & 2 & 3 have HR results that are above the average whereas teams 4 & 5 & 6 are below the average. This can be explained by the fact these teams have been asked, at the beginning of the experiment, to follow the same procedures as they would usually perform, whereas last three teams have been asked to practice new procedures. The overall performance of the teams as given by average values shows that the Hybrid Reinforcement is the most efficient function, as it combines both ER and KR for both local and global resilience indicators.

Conclusion

For a Human-Machine System facing a critical situation due to unprecedented events, i.e. events with a very rare frequency of occurrence and which may have catastrophic consequences, the concept of resilience has been defined as the positive ability to recover or to adapt to this critical situation.

A brief discussion on the need to estimate a Human-Machine System state with resilience indicators has been presented. A solution employing estimation through a learning system has been selected. Therefore, two reinforced learning functions, using knowledge or estimation reinforcement, and a hybrid approach including both of these have been proposed and implemented to illustrate the feasibility of such estimations. The proposed iterative reinforced learning structure has the ability to estimate resilience indicators and to learn from experience. Because the knowledge reinforcement function requires substantial sets of representative data in order to train the network, it is less efficient than the estimation reinforcement function. The main advantage of the hybrid reinforcement function is that it combines both estimation reinforcement and knowledge reinforcement, so that it allows the most efficient estimation results. However, it is not possible at this stage to provide consistent statistics, since only six teams were included in the experiment of this project. The system designed for the estimation of resilience indicator is a first step with the potential goal of being combined with a model of Human-Machine System, to simulate more scenario and configurations and to consolidate the results.

In the future, we aim to refine the reinforcement function in order to reduce estimation errors. For example, the relative weighting of criteria during reinforcement impact assessment could be analysed to improve results. We would also like to explore the use of more advanced machine learning methods to implement promising practical estimation structure. An example involving scenarios applied to urban guided transport systems will be addressed to perform a more in-depth analysis and to consolidate the early results.

Acknowledgment

The present research work has been supported by:

The Nord-Pas-de-Calais Region,

The European Community,

The

  neous and over-time HMS states, called respectively the local and the global resilience indicators. These indicators are based on several criteria such as the success level of a given task, the safety level of this task or the human workload in terms of interactions with the technical systems. For an iteration i and k criteria of resilience, the vector denoted U ki in Equation (8) is based on two indicators, the local indicator, u ki , and the global indicator, i i=1 u ki .

Figure 1 :

 1 Figure 1: Estimation process in learning system architecture.

Finally, two reinforcement

  processes are combined to form the iterative learning function: the first consisting of reinforcing the associated knowledge, and the second aiming to reinforce the error reduction between the estimated and actual values. Each reinforcement impact will be individually assessed, then both reinforcements will be combined in order to produce a so-called hybrid reinforcement function taking into account both local and global previous resilience indicators values i.e. a multi objective hybrid reinforcement as suggested by[START_REF] Delgado | Multiobjective hybrid optimization and training of recurrent neural networks[END_REF].

Figure 2 :

 2 Figure 2: The generic hybrid learning system architecture.

  air transport system, involving a simulated cockpit with a four-person flight crew, illustrated in Figure3. The experiments were performed with the inflight refuelling group from the Istres air base (France). Six military teams, working in small four-person group, were trained together and required to take a large number of decisions in uncertain situations. Their activities were reproducible using a flight simulator of a BC-135 Boeing during inflight refuelling.

Figure 3 :

 3 Figure 3: The military air transport system cockpit.

  Regional Delegation for Research and Technology, The Ministry of Higher Education and Research, The Scientific Research Group on Supervisory, Safety and Security of Complex Systems, The International Research Network HAMASYTI (Human-Machine Systems in Transportation and Industry),

Table 1 :

 1 Different formalisms for feedforward and/or feedback based learning control.

	References	Formula
	Xu et al. (2004)	u i = u i-1 +G f eedf orward (e i-1 )
		(1)
	Ojha et al. (2017)
	Geng et al. (2017)
	Xu et al. (2004)

Table 2 :

 2 Difference in MAE on the resilience indicator estimation.

  Table2shows the differences in Mean Absolute Error (MAE) results obtained between no reinforcement and the Estimation reinforcement functions, denoted ∆ER, and between the Knowledge reinforcement and the Hybrid reinforcement functions, denoted ∆KHR. For the estimation of one particular team, the data from the other teams were integrated into the knowledge, because the comparisons between no reinforcement and the Knowledge reinforcement, and between no reinforcement and the Hybrid reinforcement functions could not be achieved.If the MAE difference is positive, then the first function is better than the second, i.e. no reinforcement gives better results than the Estimation reinforcement function for ∆ER and Knowledge reinforcement is better than Hybrid reinforcement for ∆KHR. Of course, if the MAE difference is negative, then the second tested function is better than the first. ∆ER was negative in each case except for team2. This demonstrates the effectiveness of Estimation reinforcement versus no reinforcement. For ∆KHR, the negative values, lower than ∆ER in 10/12 cases, demonstrate the value of Hybrid reinforcement. Finally, the lowest average values confirm the performance of Hybrid reinforcement for both local and global resilience indicators.

Table 3 :

 3 MSE on the resilience indicator estimation.

Table 2

 2

	, which presents the Mean Squared

Table 3

 3

	, the Estimation Reinforcement function provides a 9/12 improvement
	over Knowledge Reinforcement, mainly because there is a lack of expert data
	for network initialization and training. If each team is considered indepen-
	dently, the performance of each function can be assessed. It can be seen
	that the performance of the resilience indicators are not necessarily linked;
	for instance, team 1 has a very good local HR value (less than the average
	MSE value) and very poor global HR value (greater than the average MSE)

And the National Center for Scientific Research.

The authors gratefully acknowledge the support of these institutions.


Keywords: Human-Machine System, Minimization of energy consumption, Multi-model approach, Quadratic optimization

In the transport domain, the aim of operators and manufacturers is not only to perform a route as quickly as possible but also to take into account the energy consumption. The scientific advances in the field of solving optimization problems challenge driving habits. Eco-driving assistance systems are then designed to reduce energy consumption. In this paper, achieving an energy efficient driving profile for a tram-driver system is presented. The optimization problem is to achieve inter-station distance in a time t, while respecting the constraints imposed by tram and route models, to minimize energy consumption.

Eco-driving command for tram-driver system

Salvatore La Delfa * , * * Simon Enjalbert * * Philippe Polet * * Frédéric Vanderhaegen * *

INTRODUCTION

The work presented in this article is carried out in the context of the ECOVIGIDRIV project. Driving a tram involves the design of complex control panels that require from drivers more knowledge, performance and attention. On-time performance, passenger comfort and safety are the main tasks allocated to drivers. The operators want to increase traffic while reducing the energy consumption of the network. A major constraint is that the tram moves in an open world, i.e. it can be disturbed by pedestrians and cars. The tram driver must anticipate the behavior of drivers and pedestrians, while reducing its energy consumption. To date, the trams are not equipped with ecodriving assistance system. Our goal is to provide a support system for energy-efficient driving that should reduce energy consumption while maintaining the vigilance level of the driver (impacting the security level), respecting speed limits and time route, ensuring the arrival of the tram at intersections with throttle handle in neutral position (which is a safe operation procedure), and which must not be intrusive. The aim of the article is therefore to provide the specifications of a system minimizing energy consumption and its impact on the driver vigilance with compliance to the network procedures. In the railway industry, support systems for energy-efficient driving have been developed. The ADAS (Advanced Driver Advisory Systems) optimize control of the passengers or freight train. They calculate a velocity profile based on timetable of the operator, on vehicle model and on characteristics of tracks to minimize energy consumption. A first system, Energymiser

Authors would like to thank ECOVIGIDRIV project members, the Fonds Unique Interministriel and the Nord Pas de Calais region. [START_REF] Albrecht | Using simulation to assess the benefits of energy-efficient driving strategies[END_REF], is used in the fast transportation of passengers over long distances. The energy-efficient information is displayed to the driver who must implement it. It is an acceleration control that the driver results in a position of the throttle handle. This manipulator has three zones: a positive acceleration zone; a "neutral" zone (traction control is zero with vehicle ahead by inertia); and a braking area. Whereas the train is running over long distances with a time route which allows the speed control, the tram travels shorter distances. Therefore, the slow dynamics of a speed control available in train domain is not suitable for the tram. It is then necessary to apply an acceleration command to suggest eco-driving for the driver . A second system, Freightmiser [START_REF] Albrecht | Using simulation to assess the benefits of energy-efficient driving strategies[END_REF]; [START_REF] Coleman | Network optimisation using integrated scheduling and driver advice systems[END_REF]), is used in the freight transport. The special feature of this system is to calculate a suboptimal control for a freight train whose engine may be electric, diesel or hybrid, while optimizing the management of the rail network. Freights are less constrained in time that trains carrying passengers. A third system, Metromiser [START_REF] Howlett | Optimal strategies for the control of a train[END_REF]), is used in the metro but its complex interface makes it unusable for drivers in an open world. Finally, other systems dedicated to the management of the railway system [START_REF] Howlett | Local energy minimization in optimal train control[END_REF][START_REF] Li-Xing | A control simulation method of high-speed trains on railway network with irregular influence[END_REF]), the energy recovery [START_REF] De Martinis | Models and methods to optimise train speed profiles with and without energy recovery systems: a suburban test case[END_REF]) and the speed constraints [START_REF] Liu | Energy-efficient operation of rail vehicles[END_REF]; [START_REF] Feng | Optimization of target speeds of highspeed railway trains for traction energy saving and transport efficiency improvement[END_REF]) can be considered in the optimization control problem. However, these systems are effective during an automated operation of the vehicle in a closed environment excluding the driver state in the stage of calculating the velocity profile to be followed. The driver should be considered as part of the system and he sometimes may not take into account information provided by the assistance system. Moreover, driver state changes over-the-time. Therefore, we must realize a predictive controller that will calculate in real time operational acceleration trajectory including the impact of human behavior during the calculation phase. In this article, the eco-driving command for tram-driver system is presented. First, the dynamics are modeled for tram (motion and energy consumption) and a controller is designed. Then, the constraints of the tram model and the optimization problem to reduce energy consumption are discussed. The problem is then discretized to solve it numerically. The controller is then applied in a realistic simulation to test performances. Finally the perspectives for future work are proposed.

CONTROLLER DESIGN

The aim of the work is to achieve energy-efficient control of a tram-driver model. For this, a multi-model approach is proposed in Fig. 1 to take into account all the components and characteristics. Three components are modeled: the controller, the driver and the tram. The tram model input variable is the throttle handle position denoted γ(t) and the output variables are the position p(t), the velocity v(t) and the traction force u(t). The total mass m of the tram with passengers is a parameter that is updated at every stop station. The controller model calculates the instruction v i (t), the intended velocity, that the driver must reach to reduce the consumption of the tram from the remaining time route, the tram output variables, and controller internal models that are tram and vehicle consumption models. In this article, the controller permitting energy consumption reduction during a route integrating the tram model is presented. The motion model of the tram is given by the equation (1) to optimize the control.

ṗ(t) = v(t) v(t) = 1 m u(t) - 1 m (A + Bv(t) + Cv 2 (t)) -gi with i = i + k e r c & i = sin(α) (1)
The terms A, B and C are constants defined by the manufacturer and correspond to mechanical and aerodynamic characteristics of the tram. The term A + Bv(t) represents the resistance due to bearings and mechanical frictions and Cv 2 (t) the aerodynamic drag term with C that characterizes its coefficient of air penetration. The term gi represents the resistive force produced by the slope and the passage of a tram in turn, with the gravitational acceleration g and i the intensity of the turn added to the slope, and u(t) the traction or braking force applied at time t. The tram model is controlled by a throttle which, according to its γ(t) position, provides an entry into traction or braking effort u(t). Tram engine dynamics is not considered in this study. As the dynamics of the electric motor and the transmission are faster than the dynamics of the tram, only the mechanical dynamics will be considered. In addition, the traction or braking force is proportional to the throttle position as mentionned in [START_REF] Vial | Vers un dimensionnement optimal structure-commande de système multi-convertisseurs[END_REF]; [START_REF] Howlett | Energyefficient train control[END_REF]. The traction force is proportional to the fuel supply rate in the engine, i.e. the position of the throttle through a train diesel engine and/or hybrid. The position of the throttle γ(t) is bounded between -1 and 1 and u(t) the traction or braking force is limited between u min and u max are based on the velocity of the tram in equation ( 2).

∀γ(t) ∈ [-1; 1], u min ≤ u(t) ≤ u max , γ(t) = u(t) u max (2)
The resistive force against tram move can be simplified into a linear function v(t) in the velocity range [v min ; v max ]. In addition, the constant C is very low and the velocity of the tram is relatively low. In this velocity range, the term A + Bv(t) + Cv 2 (t) is approximated in a 1 + b 1 v(t)by using the least squares method that gives in equation ( 3):

ṗ(t) v(t) = 0 1 0 - b 1 m p(t) v(t) + 0 1 m u(t) + 0 - a 1 m + gi (3)
The controller model is now presented and the problem must be formulated in order to be optimized.

OPTIMIZATION PROBLEM

The objective is to minimize power consumption of a tram in equation ( 4), during its displacement from a station A to station B in a time route T which is defined by the equation ( 5). The tram starts from the station A to the position p A with a velocity equal to zero and arrives at the station B at the position p B with zero velocity. It starts at time T A and it arrives at the station B at the time T B . Either the expression of mechanical power confirmed by the operator and in the literature [START_REF] Vial | Vers un dimensionnement optimal structure-commande de système multi-convertisseurs[END_REF]) which corresponds to the force multiplied by the velocity of the tram in equation ( 4):

E = min u,p,v T B T A u(t)v(t) (4)
i.e.:

p(T A ) = p A , p(T B ) = p B v(T A ) = 0 , v(T B ) = 0 (5)
The influence of braking or jerking when driving as well as energy recovery, are not considered in this study. Nevertheless, we know that, according to De Martinis (De Martinis and Gallo (2013)), these factors included in the problem could be solved. Several authors consider tram control optimization as the basis for eco-driving efficiently [START_REF] Howlett | Energyefficient train control[END_REF]; [START_REF] Monastyrsky | Rapid computation of optimal control for vehicles[END_REF]; [START_REF] Delprat | Control of a parallel hybrid powertrain: optimal control[END_REF]; [START_REF] Rousseau | Véhicule hybride et commande optimale[END_REF]) i.e. minimizing traction effort or braking u(t), which corresponds to the input of the dynamic model of the tram. But the tram constrained by its operating range (maximum traction/braking) which depends on the vehicle velocity. Furthermore, acceleration as well as its dynamics (jerk) are constraints. However, these constraints are linear and are not considered in this approach for simplification reasons. In the next section, tram model using constraints are developed.

Tram constraints

The tram performances are defined according to its engine but also by the requirements of operators. These constraints confine the range of use of the electric tram efforts. Therefore, they must be integrated into the controller. The constraints of the model are discussed in this section. The maximum traction and maximum electrical braking depend on the velocity in Fig. 2. The dotted path represents the traction force allowing the tram to accelerate for a throttle position equal to 1. If the velocity is lower than v t , then the traction force is equal to u max , else if the traction force is decreasing up to v max . The curve represented by dashes corresponds to the force applied to brake function depending on the velocity of the tram with the throttle position equal to -1. The maximum electric braking force is constant between v f1 and v f2 . It is equal to u min and is not compensated by a mechanical force. The electrical brake force decreases when velocity is lower than v f1 or upper than v f2 . During these two stages, the brake force is mechanically compensated. The method of least squares is used to partially linearize the curves of traction and brake forces (in Fig. 2). So tram operating range will be expressed in the following form:

u(t) ≤ u max , v(t) ∈ [0; v t [ kv(t) + u(t) ≤ u max , v(t) ∈ [v t ; v max ] (6) k 1 v(t) -u(t) ≥ 0 , v(t) ∈ [0; v f1 [ u(t) ≥ u min , v(t) ∈ [v f1 ; v f2 [ k 2 v(t) ≥ u min , v(t) ∈ [v f2 ; v max ] (7)
Where k, k 1 , k 2 are constants. In equation ( 6) are represented the operating range of the tram traction and in equation ( 7) tram operating range when braking. The tram is also limited by its velocity in equation ( 8) which is between v min and v max .

v min ≤ v(t) ≤ v max (8)
In the following, the tram operating range (u(t); v(t)) is denoted D. In next section is given the method for minimizing energy consumption.

Tram control optimization

The objective is to minimize the energy consumption of the tram. The cost function in equation ( 4) is minimized within the constraints in equations (3,6,7,8) and those defined by the route. Either the notation in equations (9,10,11):

ẋ(t) = Ax(t) + Bu(t) + d (9) x(t) = p(t) v(t) (10)
Fig. 2. Total force depending on velocity.

A = 0 1 0 - b 1 m , B = 0 1 m & d = 0 - a 1 m + gi (11)
with d a matrix that represents the disturbances caused by the track profile which are slopes, bends, etc. The current cost function does not solve a linear quadratic control problem because it is not convex nor quadratic. We must change the model in order to calculate the command to be applied by the driver. To do this, simply to express the criterion to be minimized so as to solve a convex quadratic linear problem. For this purpose, Ru 2 (t), x T (t)Qx(t) are added to equation ( 4) where Q and R are symmetrical weighting matrices in equation ( 12) defined to obtain the convex quadratic objective function in equation ( 13). R is defined positive and Q is defined not negative [START_REF] Anderson | Optimal control: linear quadratic methods[END_REF]). Thus, it is possible to determine the solution to consume less energy along the tram route. Criterion ( 13) is then:

0 < R, 0 ≤ Q & S = 0 1 2 (12) J = 2x T (t)Su(t) + Ru 2 (t) + x T (t)Qx(t) (13)
The term v(t) is replaced by 2x T (t)S where S is a weighting matrix. The transpose of x(t), here x T (t), is the set of solutions of the problem on his condition and u(t) the set of solutions on the command. Now, for a convex quadratic linear problem the cost function is written in equation ( 14):

J = (u(t) + R -1 S T x(t)) T R(u(t) + R -1 S T x(t)) +x T (t)(Q -SR -1 S T )x(t) (14)
The expression of our modified tram model in equation ( 15) is denoted as follows:

ẋ(t) = (A -BR -1 S T )x(t) + B ũ(t) + d ũ(t) = u(t) + R -1 S T x(t) (15)
It is now possible to calculate the command to minimize the cost function of the convex quadratic linear problem in equation ( 16):

J = min ũ,p,v T B T A x T (t)(Q -SR -1 S T )x(t) + Rũ 2 (t) (16)
The cost function is quadratic if and only if (Q-SR -1 S) ≥ 0 according to [START_REF] Borne | Commande et optimisation des processus[END_REF]). The equation ( 13) allows to obtain the equation ( 17).

SR -1 S T = 0 0 0 1 4R (17)
The terms Q and R previously added are used to obtain a convex cost, it is necessary that these terms are as small as possible. However, in order to avoid any penalty on the tram position, the Q matrix is defined in equation ( 18).

Q = 0 0 0 q 2 (18)
To ensure the condition (Q -SR -1 S T ) ≥ 0, we must have q 2 > 0 and 4q 2 R ≥ 1. To obtain a convex cost, traction or brake force and velocity of the tram can not be heavily penalized. The criterion is defined to calculate the command of our tram to minimize energy consumption.

The discretization of the model permits the solving of optimization problem numerically.

Model discretization

To solve the optimization problem numerically, it must be discrete. The sampling period to avoid any data loss during the route should be determined. It must be chosen depending on the speed of the dynamic of the tram on the route. In this study, the data considered are those of Valenciennes tramway. Sampling period T s is one second, the mass m of 50000kg is used with R equal to 0.1, q 2 = 2.5 and 4q 2 R = 1 and where à = A -BR -1 S T and B = B. The tram model given in equation ( 15) becomes (equation ( 19):

x(t) = Ãx(t) + B ũ(t) + d ũ(t) = u(t) + R -1 S T x(t) (19)
The discrete state model in equation ( 20) is obtained for a sampling interval of one second for the model in equation ( 19):

x(k + 1) = A d x(k) + B d ũ(k) + d d ũ(k) = u(k) + R -1 S T x(k) (20)
The cost function J in equation ( 16) is also discretized in equation ( 21) and T the number of steps in equation ( 22) were calculated according to predefined constraints:

J = T k=0 Rũ(k) 2 (21) T = T B -T A T S (22)
Which constraints are expressed by equation (5,8). The control problem is a convex quadratic problem. Thus, the resolution time to set the driving profile is faster than the expected refreshing time for the information given to the driver. In addition, it is used in real time to calculate a new driving profile as the constraints and used models may not be accurate enough. This is why the optimization calculation is restarted every second. In this case, the interior point method is used to determine the optimum that reduces the energy consumption according to the constraints applied both on the vehicle and on the environment, and that need to be updated during the route.

CONTROLLER APPLICATION

The optimization algorithm is used in real time. Thus, at each iteration a new driving profile for solving the minimization of energy consumption for the route of the tram driving simulator is defined. The advantage of operating in real time is to adapt to the situation, to update the constraints and the model that may be wrong (for instance changing the maximum velocity on the section due to work in progress). In addition, changing the profile of the route corresponds to a disturbance of the command that is considered by changing the value of the matrix d of the equation ( 9) and is corrected during the next step. In this section the environment and the route of the tram are described. The presented data are resulting from the tram driving simulator "OKSimRail" developed by Oktal society and is particularly used to coach tram drivers. The tram travels a total distance of 1842.58m for a time of 357s not considering stops that the tram is respecting in stations. In this route, there are four subroutes that correspond to the inter-station distances. The first sub-route, P 1, is the distance made to go from the starting point to the station 1. This sub-route is a straight line of 167m to be performed in 43s. The second sub-route, P 2, with a distance of 625m is to be performed in 140s from station 1 to station 2. It consists of one intersection, four speed changes and two bends. The distance of 600m between station 2 and station 3 is to be completed in 110s. This is the third sub-route, P 3, where trams must cross two intersections, two speed changes and one turn. The last sub-route, P 4, is to complete the distance between the station 3 and the station 4 separated by 450m in 64s.

The driver must regulate the velocity of the tram three times and addresses two intersections and end on a turn before arriving at final station. During the experiment the traffic lights are not included. route is presented in Fig. 3.

The command from the controller is calculated on this route. The resolution model calculates a velocity profile that meets the velocity limits, the crossing of intersections and stop at stations in Fig. 4.

Initially, the command is determined by not taking into account the impact of bends on the tram and then, in a second time, bends of the track were included to assess their impact on energy consumption. The position of the throttle is rarely above 50% when the tram accelerates in Fig. 5, and to maintain velocity, throttle position is very close to the neutral position ( = 0). Finally, the only route with strong variations of the command is P 4, where the route time is barely small regarding the large distance. However, the command the driver has to follow in order to carry out P 2 and P 3 sub-routes is really different when considering or not the impact of bends in Fig. 5. The tram requires greater acceleration when traveling on a turn. This "over-acceleration" corrects the error that is produced by interference from the track bends and implies a position of throttle away from neutral.

For instance, the same route P 2, 625m in 140s, performed considering bends implies energy over-consumption increase of 55% in table 1. So bends can not be neglected in the optimization problem.

In this study, the energy consumption for driving "Pulse & Glide" (P &G which is to move as fast as possible to the speed limit and then let the vehicle advance by its own inertia) performed automatically is compared with an optimized driving calculated by the controller. The route time is of route identical in both cases. Table 2 shows the saved energy between the two driving modes. The controller can reduce energy consumption, up to 12% in P 2, on the entire route in Table 2. The slight overconsumption in P 1 is explained by the high precision of the controller that gives an energy consumption value for a throttle position range where it is normally zero. This does not prevent to reduce energy consumption by 7% for the entire route. The controller and its performance were presented in this section and some improvements are already considered and will be discussed as perspectives in next section.

CONCLUSION

In this paper, a method to calculate a driving profile for minimizing energy consumption is presented. The controller calculates the eco-driving command to apply. It allows, in the studied case, to reduce energy consumption up to 12%. To include changes in speed and slopes and bends in the controller, the study of a hybrid Model Predictive Controller (MPC) is considered. If the computation time becomes too important, a controller that includes both the advantages of a hybrid MPC and the current algorithm will be studied. The integration of the driver model in the controller is a perspective that we want to explore. The driving task can not be automatic because the driver main task is to ensure security and safety during route. The driver should apply the calculated driving profile but his response time and his ability to respect instructions can vary and is not stable over-the-time, for instance for security reasons if an object can be found on the tram tracks. Therefore, it is proposed to incorporate into the controller a model of Human Operator in Fig. 6. The measured variables of the driver may be incorrect or not observable. Thus, the estimation of the driver will set the model parameters and information mode that drivers will have to follow to minimize energy consumption. Some driver model can be found and developed based on literature review as for example in [START_REF] Cacciabue | Unified driver model simulation and its application to the automotive, rail and maritime domains[END_REF]). The refresh time of the ecodriving instruction to be displayed to the driver will be defined in later studies. To assist the driver in eco-driving, it is possible to achieve a cooperative eco-driving system [START_REF] Vanderhaegen | Multilevel organization design: the case of the air traffic control[END_REF]; [START_REF] Lemoine | Cooperation between humans and machines: first results of an experiment with a multi-level cooperative organisation in air traffic control[END_REF]) where the human operator allocates a task to the controller (followed by eco-driving route) and performs a parallel task (security management) [START_REF] Millot | Human-machine cooperation: a solution for life-critical systems?[END_REF]; [START_REF] Soualmi | Automation-driver cooperative driving in presence of undetected obstacles[END_REF]). Another controller can support the driver in its movement [START_REF] Masikos | Ecogem-cooperative advanced driver assistance system for green cars[END_REF]; [START_REF] Marouf | Control of an electric power assisted steering system using reference model[END_REF][START_REF] Sentouh | Cooperative steering assist control system[END_REF]) to help implement ecodriving instruction. External events (passing pedestrians, Fig. 6. Multi-model approach including Human in the controller. road vehicles on tracks, etc.) can disturb the following of driving profile and thus causing delay. The controller will then adjust to the new traffic conditions. If the route is no longer feasible in the originally allocated time, and in order not to jeopardize the safety and the comfort of passengers (overspeed, sudden acceleration, etc.), the controller must be able to relax the time constraint to ensure solving the optimization problem. The driver behavior can be considered as a perturbation which need to be minimized regarding energy consumption. Indicators focusing on time and distance traveled respecting the application of ecodriving command should be developed to demonstrate the impact of driver on the tram system.

Fig. 1 .

 1 Fig. 1. Eco-driving command for a tram-driver system.

Fig. 3 .

 3 Fig. 3. Simulated route.

Table 1 .

 1 Energy over consumption considering bends

	Route	P1	P2	P3	P4	Total
	Distance (m)	167 625 600 450	1842
	Duration (s)	43	140 110	64	357
	Over consumption (%)		55	8	3	16

Table 2 .

 2 Energy consumption comparison between P&G and controller

	Route	P1	P2	P3	P4	Total
	Distance (m)	167 625 600 450	1842
	Saved energy (%)	-12	12	5	10	7

ACKNOWLEDGEMENTS

The present research work has been supported by: The Nord-Pas-de-Calais Region, The European Community, The Regional Delegation for Research and Technology, The Ministry of Higher Education and Research, The International Research Network HAMASYTI (Human-Machine Systems in Transportation and Industry), And the National Center for Scientific Research. The authors gratefully acknowledge the support of these institutions.


Simon Enjalbert Kiswendsida

Abel Ouedraogo

Frédéric Vanderhaegen
email: frederic.vanderhaegen@univ-valenciennes.fr

Validation of a Unified Model of Driver behaviour for train domain

Keywords: Driver behaviour, Human factors, Train control, Railways simulation, European Train Control System, NMRA

In this paper, authors goal is to present part of the work carried out within the European project ITERATE. The objective of ITERATE is to develop a Unified Model of Driver (UMD) behaviour and driver interaction with innovative technologies in emergency situations. Such a UMD could be used when designing innovative technologies for assessment and tuning of the systems in a safe and controllable environment, when guiding designers in identifying potential problem areas, when adapting systems to the drivers before being available on the market and providing better support to the driver in emergency situations, or, for authorities, as a guide in assessing and approving innovative technologies. Authors will mainly focus on the UMD validation process on a miniature railway platform. An exhaustive presentation of the COR&GEST platform is given to understand how the comparison with a numerical simulation is possible. The command of the miniature train model is demonstrated and the visual interface for driver is depicted. Then, the data collected during experiments and the process to analyse the results are indicated. Finally, the numerical SiMUD tool and the experimental results are compared to assess the validity of the UMD.

INTRODUCTION

In this paper, authors want to present all developments carried out on the Railway Driving & Traffic Monitoring platform named COR&GEST to validate a Unified Model of Driver (UMD) behaviour. Experiments and results presented in next sections have been collected through the "IT for error remediation and trapping emergencies" ITERATE project from Seventh Framework Programme of the European Commission and are fully discussed by [START_REF] Hjälmdahl | Validation of the UMD model for cars and trains[END_REF]. In the early stage of ITERATE project, a theoretical Universal Model of Driver behaviour has been developed by [START_REF] Oppenheim | Critical review of models and parameters for driver models in different surface transport systems and in different safety critical situations[END_REF] and resulted in the ITERATE Unified Model of Driver behaviour presented by [START_REF] Shinar | Review of models of driver behaviour and development of a unified driver behaviour model for driving in safety critical situations[END_REF]; [START_REF] Oppenheim | Description of unified model of driver behaviour (UMD) and definition of key parameters for specific application to different surface transport domains of application[END_REF] in Figure 1. Drivers experience, driver state (fatigue or alert), attitude (whether sensation seeker or not), workload (different levels induced by secondary task, denoted WL) and culture (regional or company/organisational) are factors that will be considered together with influences from the environment and the vehicle. The next stage in ITERATE was then to collect data that could be used for testing the relationships of the factors within the model as well as provide input to development of the numerical model. This was done during portable experiments where train simulators were build and shipped between the five participating countries to make sure that identical experiments were carried out and that the cultural effect was studied in a correct way. In parallel to the experimental work, the numerical simulation tool was developed (SiMUD) using inputs from the initial theory and from the experiments [START_REF] Forsman | Results from the analysis and input to the development and validation of the statistical models[END_REF]. The UMD validation must be realized across the comparison between the model and the simulation, i.e. comparing SiMUD to the real drivers in a train simulator. For the simulation the criterion is that it should reproduce the same behaviour as is generated on group level in the simulator with real drivers and the main effects should be comparable. So, for example, if sensation seeking drivers (SS) are shown to drive at a higher speed than non SS, then should the modelled SS drive faster as well. For trains a new simulator with new train drivers on a new track was used. For practical reasons however the speed control by beacons system (French KVB) used differed somewhat from initial European Train Control System (ETCS) even though it deals with speed warning and control as well. In the first section, the railway simulation platform COR&GEST used for validation protocol will be detailed. All explanations to transform a miniature railway in an scientific experimental platform are given. In the second section, the behaviours of drivers experimented on our platform across collected data are presented. Finally, in Fig. 1. The ITERATE Unified Model of Driver behaviour the third section, the comparison between UMD tool and collected results is realized to validate the effectiveness of the approach.

THE RAILWAY SIMULATION PLATFORM COR&GEST

Many standards defined by the National Model Railroad Association (NMRA) are widely followed by the industry and modellers, including their HO scale track and their Digital Command Control (DCC) standard set. Concerning the train, the ZIMO MX6 decoder family is used to command the miniature train. For the environment, the COR&GEST platform is a miniature railway made up of about 33 meters of HO tracks and switches divided in 41 sections in Figure 2. HO is one of the most popular scale of model railway and his ratio is about 1:87. Nevertheless, another scale, the N scale (1:160), was used for landscape and all environment because it takes less space than HO scale. Switches and signal lights can be piloted from monitoring and scenario workstation. It allows commanding the set of elements constituting the railway tracks. To ensure a good communication between all elements, a minimum period of 110 ms (approximatively 9 Hz) is necessary to guarantee the digital interface and the miniature train to operate correctly. In next subsections, details will be given about the train model used on the platform and the train driver interface developed for the UMD validation.

The train model

Different methods exist using Configuration Variables (CV's) to program speed curves and optimize the driving characteristics of a miniature train. On COR&GEST platform, the train speed is calculated depending on the real observed behaviour of train on the platform. In Table 1 are given some of the NMRA codes from 0 (stop) to 126 (maximum speed) to command directly the miniature train with NMRA codes. A camera is located in the head of the miniature train, and, when a driver pilots the train, the perception of the landscape (which is N scale) should be reflected in displayed speed and coherent with For instance, maximum speed, acceleration and braking power are adapted to scenario to be tested and cannot be implemented through CV's if desired behaviours are too much complicated regarding what manufacturers have planned for their decoders. In the ITERATE validation scenario, maximum speed of the train has been limited to Fig. 2. The COR&GEST railway platform 160Km/h, i.e. 96 converted in NMRA code. The acceleration power increases with speed but is of course reduced as the train reaches his maximum speed. Many rules have been created to make the train behaviour as realistic as possible avoiding linear acceleration. Braking power has been set to guarantee that braking is always more powerful than acceleration. Finally, a natural decrement of speed has been simulated and so the train speed will decrease till train stop if no acceleration command is given by driver for a long period. One variable, stack, will be calculated and updated with sample rate of the experiment in Equation 1.

stack t = stack t-1 +T hrottleP osition t ÷16×∆÷1000 (1)
with T hrottleP osition t a numerical value included between [-16;16], respectively for full braking and full acceleration, and ∆ is the sample rate time in milliseconds (time elapsed between t-1 and t). stack t is calculated and used in the Algorithm 1 to command the miniature train in speed through NMRA codes. For instance, if the train is starting Algorithm 1 acceleration and braking algorithm if

T hrottleP osition t > 0 then if stack t ≥ 0.4 & N M RAcode t < 20 then N M RAcode t ← N M RAcode t-1 + 2 else if stack t ≥ 0.6 & 20 ≤ N M RAcode t < 33 then N M RAcode t ← N M RAcode t-1 + 2 else if stack t ≥ 0.6 & 33 ≤ N M RAcode t < 83 then N M RAcode t ← N M RAcode t-1 + 1 else if stack t ≥ 1.2 & 83 ≤ N M RAcode t < 90 then N M RAcode t ← N M RAcode t-1 + 1 else if stack t ≥ 1.8 & 90 ≤ N M RAcode t < 96 then N M RAcode t ← N M RAcode t-1 + 1 end if else if T hrottleP osition t < 0 & stack t ≥ 0.4 then N M RAcode t ← N M RAcode t-1 -2 else if stack t ≥ 5 then N M RAcode t ← N M RAcode t-1 -2 end if stack t ← 0
from a stop at station and if actual NMRA code is between 33 and 83, as soon as stack t will be greater than 0.6, NMRA code will be increase by 1. This will occur in 0.6 second if the train driver fully accelerate, 1.2 seconds if fifty percent of full acceleration (T hrottleP osition t = +8) and so on. On the other hand, if train actual speed is 60Km/h (NMRA code 46) and if driver want to stop the train with full braking ((T hrottleP osition t = -16), NMRA code will be decrease 2 by 2 every 0.4 second.

The driver interface

During initial experiments, the ETCS was used to collect data to configure SiMUD tool. ETCS was selected because it gathers all existing technologies and systems supporting the operator across countries [START_REF] Lai | Review of existing technologies and systems supporting the operator[END_REF][START_REF] Oppenheim | Critical review of models and parameters for driver models in different surface transport systems and in different safety critical situations[END_REF]. For validation stage, an adapted version of the KVB french system was selected to compare results. The video resulting from the camera located in the head of miniature train and the signal is encapsulated in a visual displayed to drivers in Figure 4. On the upper left corner of Figure 4, the yellow triangle on blue square flag give to the driver the information about the remaining distance before next section (the distance in meters is displayed one thousand meters before). The driver gets information on how to perform the speed reduction thanks to this information and the original KVB display (depicted just after). On the lower left corner, pressure gauges of the train give information of braking power applied. From left to right, the original KVB display panel, the current maximum permitted speed (120 km/h in green), the actual speed (0 km/h on the horizontal graph), the throttle position (100, which means no acceleration or braking power), the simulation time (11:03:52) and a timetable for drivers (with station name, arrival time and departure time) are included. Finally, the driver is able to pilot miniature train with "raildriver" joystick presented in the lower right corner. One of the throttle is configured and used to accelerate or brake the train.

BEHAVIOURAL RESPONSE OF TRAIN DRIVER

During the train driving scenario the drivers were asked to drive the train according to a given timetable. The train driving scenario included several changes in speed limit and stations at which the driver were to stop. The changes in speed limit and the stops included different track alignments, i.e. slopes, and levels of the permitted speeds. The Fig. 4. Train driving interface driver was assisted by a modified version of the ETCS that issued a continuous warning to the driver if the speed limit was exceeded. In the approach to changes in speed limit and stop at stations the system indicated the required braking through a permitted speed profile. A continuous warning was issued to the driver if the permitted speed profile was exceeded during the deceleration to new lower speed limit or to stop at a station. To understand how ETCS is functioning, in Figure 5 the driver is driving at 140 km/h and then gets a notification about a new lower speed ahead, in this case 90 km/h. The bar on the left says that he has 845 meters left to the new speed. Also note the small square in the upper left hand corner. This square will grow as we approach the point where braking has to start to be able to safely reach the new speed.

During validation experiments, a simulation of the onboard computer who generates two speed-thresholds based on the received signals from the beacons has been developed. If the train is over the speed limit, passing the first speed-threshold, an audible alarm sounds and the control panel indicates to the driver to adjust the train speed without delay. If the second speed threshold is passed, the KVB automatically engages emergency brakes on the train. This automatic braking has been shunted in our experiments to Fig. 5. The original ETCS display let operators the opportunity to drive as fast as possible without the risk to be "punished" by the advanced driver assistance system. To manipulate workload level in the train experiment the same count backwards secondary task with subtraction by 7s or by 1s that was used in the initial experiments was used. A full description of the task has been realized by [START_REF] Barnard | Specification of test procedures for the simulator experiments[END_REF]. Three levels of workload are still remaining (High, Medium and Low, low should be considered as baseline situation). Three types of events can be defined:

• Strong speed reduction is an event in which the train driver has to reduce speed in the approach to a potentially dangerous area. (in general the allowed speed is 30 km/h within the station area if there is a track change and a station stop). • Speed reduction while driving is an event in which the train driver has to reduce speed along the track.

The required speed reduction is lower in the speed reduction while driving events compared to the strong speed reduction events. • Braking for station stop are events in which the train driver is required to stop at a station. The speed reduction during these events is in general from 30 km/h to zero.

For the validation step of the ITERATE project, only the two first speed reduction events have been used in further developments because they were easy to compare directly to data from car simulation (same type of speed reduction events) in order to validate the UMD. In the train experiment, the following performance indicators were used in the analysis:

• Difference between permitted and measured speed (i.e. permitted speed -measured speed, km/h) at specific positions. • Mean difference between permitted and measured speed (km/h) between these positions.

Others indicators, like the resilience one, have been studied in the project by [START_REF] Enjalbert | Assessment of transportation system resilience[END_REF]; [START_REF] Ouedraogo | How to learn from the resilience of humanmachine systems[END_REF] but were not kept because it would have been too difficult at this stage to implement them in UMD numerical tool. An example of strong speed reduction that illustrates the points and sections is shown in Figure 6.

For each speed reduction event , the difference between measured and permitted speed behaviour indicators will be calculated at three points (PI, PII and PIII) and over the two sections connecting these three points (SI and SII). PI corresponds to the start of the event, PII is defined as the point at which the driver is supposed to reduce the speed event and PIII is the end point of the event (the point at which the new speed should have been reached, information given by ETCS to drivers). SI is the section in between points PI and PII. Similarly, SII is the section between points PII and PIII. For the sections, SI and SII, the performance indicator used is the mean difference between permitted and measured speed.

In this example, PI located 1000 m before new speed limit (KM 43148), PII at 41 m from new speed limit (KM 43907) and PIII is the new speed limit point (KM 44148). SI is defined as the section between PI and PII. Finally SII is the section between PII and PIII. Note that, in the example, the difference between permitted and measured speed will be negative at point PIII and over section SII for participant FR#2. If the train driver exceeds the permitted speed, the KVB system issues a continuous warning. These performance indicators will be

UMD BEHAVIOURAL VALIDATION ON COR&GEST PLATFORM

To validate the simulation, a simulator trial with 32 subjects distributed for two levels of experience and two levels of sensation seeking was carried out. For numerical simulation different parameters, presented in Table 2, have been selected to simulate driver behaviour in SiMUD tool. Two of them, culture and driver state, have been set to France and alert to make it possible to compare experimental results. Considering the factors experience and attitude, drivers were selected to include both novices and experienced and High/Low sensation seekers. Finally, three levels of Workload were manipulated during simulation. The difference between permitted and measured speed have been analysed for the reduction while driving and strong speed reduction events. P-values from the analysis of variance of the performance indicators of the train driving experiment are shown in Table 3 -Table 4. Factors that are found to be significant at the 0.05 level are highlighted in the tables. None of the factor turned out as significant in this validation experiment. This result is coherent with the one obtained during initial experiments. Only second order interactions can be observed with significant effect. For example, considering Experience & WorkLoad for the speed reduction while driving events, when workload is low or high the novices seem to loose their ability to control their speed and to anticipate deceleration like experimented do. Many cases have been studied and discussed in original work but results will be limited in this paper. If we consider only the mean differences of speed between permitted and measured for the speed reduction while driving events, when comparing the results generated from the simulator experiment in Table 5 and the numerical simulations in Table 6 a good overall coherence that needs to be explained was found1 . For instance, in Figure 7, the differences of speed between permitted and measured for the speed reduction while driving events for experienced and low sensation seeker drivers comparison are presented. It can be first observed that UMD tool predicts more or less the same behaviour whatever driver workload level, i.e. the three SiMUD experienced & low sensation seeker speed profiles may be surimposed. Then, during first part of curve (from PI to PII), UMD software keep speed few kilometres per hour under maximum permitted speed whereas real drivers anticipated deceleration. Then, during the second part (from PII to PIII), SiMUD initially decelerates then recovers to new to maximum permitted speed and real drivers, that had previously anticipated deceleration, only recover few kilometres per hour under new limitation. Observed differences in speed profiles are consistent regarding differences of platform design between initial experiments (data used for SiMUD parametrisation) and validation experiments. The results show that the numerical simulation manages to repeat the behaviour of the French drivers on a general level in that sense that it adopts approximately the same speed profile. The simulated driver reduces speed at last moment before deceleration point without any anticipation and the braking is smoother. This effect is most likely due to the fact that the portable simulators used to tune the model did not give the same sensation of speed to drivers and so they anticipated a lot and they brake harder during these experiments. Some specific limits on train simulation were here highlighted. Tuning of SiMUD tool for experimented support system and platform is needed for more accurate results:

• Train model of SiMUD should be updated to fit better with experimental platform. • Piloting is necessary to calibrate anticipation and confidence drivers have on the KVB/ETCS warning system.

CONCLUSION

The aim of this study was to show that the UMD model and the SiMUD tool developed within ITERATE project are valid and can be used to predict behaviour of real train drivers. A simulator experiment with 32 test drivers was performed. The numerical simulation developed and tuned by data from the initial experiments was put to the test in this experiment. The simulator experiment carried out at LAMIH from UVHC on COR&GEST platform was repeated using the SiMUD tool for trains. The same environment was implemented in the UMD and simulations were carried out for French drivers, experienced and novice, High & Low sensation seekers and workload was varied throughout the numerical simulation. Moreover, the description of the COR&GEST miniature railway platform has been realized and validated. Urban guided transportation are a new application domain that will be investigated in next projects with a new model of tramway to be implemented and a new visual display according to "on sight" driving and low speed material.

Fig. 3 .

 3 Fig. 3. The train speed commanded with NMRA codes braking are commanded depending to train experiment. For instance, maximum speed, acceleration and braking power are adapted to scenario to be tested and cannot be implemented through CV's if desired behaviours are too much complicated regarding what manufacturers have planned for their decoders. In the ITERATE validation scenario, maximum speed of the train has been limited to

Fig. 6 .

 6 Fig. 6. Example of strong speed reduction event including speed profiles for two participants

Fig. 7 .

 7 Fig. 7. Example of speed (in kilometres per hour) for the speed reduction while driving event(140 km/h to 120 km/h) depending on Workload (Experienced & Low Sensation Seeker)

Table 1 .

 1 From NMRA codes to observed speed (HO to N scale) conversion table

	NMRA Speed (Km/h)
	0	0
	20	12
	33	31
	46	60
	57	91
	83	139
	90	151
	96	160
	111	183
	126	203
	his sensation of speed. The full NMRA codes (HO train
	scale) to observed speed (N landscape scale) conversion
	table is illustrated in Figure 3. Moreover, acceleration and

Table 2 .

 2 Parameters selected for comparison with SiMUD tool

	Parameters Value
	CULT	FR
	DS	Alert
	EXP	Novice & Experienced
	ATT	Low Sensation Seeker & High Sensation Seeker
	WL	Low & Medium & High
	observed in validation experiments and compared to those
	of numerical UMD simulation in next section.

Table 3 .

 3 Analysis of variance: p-values, difference between permitted and measured speed for the strong speed reduction events

	Difference between permitted and measured speed	PI	SI	PII	SII	PIII
	Intercept	.000 .000	.000	.000	.000
	Experience	.264 .848	.439	.221	.130
	SensationSeeker	.983 .377	.563	.920	.763
	WorkLoad	.093 .554	.532	.417	.564
	ParticipantID (Experience * SensationSeeker * WorkLoad)
	Experience * WorkLoad	.738 .975	.879	.836	.716
	SensationSeeker * WorkLoad	.249 .913	.984	.862	.818
	SensationSeeker * Experience	.083 .115 .010	.002	.003

Table 4 .

 4 Analysis of variance: p-values, difference between permitted and measured speed for the speed reduction while driving events

	Difference between permitted and measured speed	PI	SI	PII	SII	PIII
	Intercept	.000	.000 .000 .000	.000
	Experience	.754	.080 .052 .371	.996
	SensationSeeker	.781	.858 .160 .294	.466
	WorkLoad	.242	.641 .311 .341	.702
	ParticipantID (Experience * SensationSeeker * WorkLoad)
	Experience * WorkLoad	.046	.260 .404 .425	.506
	SensationSeeker * WorkLoad	.998	.959 .643 .786	.623
	SensationSeeker * Experience	.533	.686 .369 .523	.708

Table 5 .

 5 Mean difference of speed between permitted and measured (in kilometres per hour) for the speed reduction while driving events during validation experiments

	Experience	Sensation Seeker	WorkLoad	PI	SI	PII	SII	PIII
	Novice	Low sensation seeker	Low	3.167	6.890	14.003	4.847	3.000
			Medium	2.583	8.990	17.667	8.007	4.000
			High	3.667	7.727	13.177	5.490	2.310
	Experienced Low sensation seeker	Low	4.875 14.760 22.250 10.385 6.500
			Medium	4.875 16.145 24.440 12.685 7.750
			High	3.375 14.995 22.750	9.940	6.000
		High sensation seeker Low	3.150 15.480 21.426	9.770	5.150
			Medium	4.100 13.998 21.376 10.220 5.600
			High	3.750 12.496 18.900	8.188	4.750

Table 6 .

 6 Mean difference of speed between permitted and measured (in kilometres per hour) for the speed reduction while driving events with SiMUD

	Experience	Sensation Seeker	WorkLoad	PI	SI	PII	SII	PIII
	Novice	Low sensation seeker	Low	6.190 6.280 6.410 14.750 1.500
			Medium	6.030 6.090 6.150 12.000 1.480
			High	6.020 6.060 6.080 14.380 1.420
	Experienced Low sensation seeker	Low	4.690 4.750 4.810 13.900 1.000
			Medium	4.740 4.820 4.940 15.700 0.890
			High	4.760 4.810 4.840 13.940 0.890
		High sensation seeker Low	4.030 4.130 4.220 13.310 0.800
			Medium	4.010 5.040 4.170 13.210 0.640
			High	4.060 4.100 4.320 13.290 0.630

For high sensation seekers, no experience level is needed because of SiMUD model, i.e. SiMUD will behave same if sensation seeker degree is set to high despite of the experience level

ACKNOWLEDGEMENTS

The present research work has been supported by The International Campus on Safety and Intermodality in Transportation, The Nord-Pas-de-Calais Region, The European Community, The Regional Delegation for Research and Technology, The Ministry of Higher Education and Research, The Scientific Research Group on Supervisory, Safety and Security of Complex Systems, The European Research Group on Human-Machine Systems in Transportation, And the National Center for Scientific Research. The authors gratefully acknowledge the support of these institutions.

The authors also cordially thanks Rudy Dahyot, development engineer, for all the work realized on the COR&GEST platform.

The research leading to these results has received funding from the European Commission Seventh Framework Programme (FP7/2007-2013) under grant agreement n218496 Project ITERATE.