Romain Billot

Tai-Yu Ma

Feng Chu

Georges Yannis

Maria Nadia Postorino

Edouard Leurent

Marc Nabhan

Jean Mercat

Sonia Assou

Matthias Couste

Benoit Laussat

Martine Sebastien

Soukaina

Jean-Luc Sopheaktra

Eric Sylvain

Gregory, Nelson, and many others for the good times we spent together over meals and coffee breaks. I also extend my sincere

Finally, I did some adaptation of proposed models and implemented them with different combinations in the SUMO traffic simulator. In these experiments, all private cars were controlled by our models, while other types of vehicles (trucks and motorcycles) were controlled by the default SUMO model. I compared the models and derived their performance for both the NGSIM 101 and the HighD traffic data-sets. I concluded that the AI-based models (a carfollowing model using neural network and a lane-changing model using Q-learning) provide better performance compared to the physical models. However, the simulation using those models was calibrated on the NGSIM data-set and was not found to be equally at representing HighD traffic. Finally, the NGSIM-calibrated IDM model is less sensitive to traffic data and can be used to simulate unknown traffic like the HighD data-set, but it requires re-calibration with the maximum speed of vehicles in the traffic.

Resumé des Travaux de Thèse

La conduite autonome soulève un intérêt croissant et l'on s'attend à ce que les véhicules entièrement autonomes puissent réduire les accidents de la route et améliorer la sécurité globale du trafic. Un véhicule autonome est un véhicule entièrement automatisé qui est capable de rouler sur la route dans toutes les conditions, sans intervention d'un conducteur. Cependant, le système de véhicule autonome est très complexe, et consiste en un système de perception, un système de décision et un système de contrôle. En outre, l'environnement routier est dynamique et implique de nombreuses interactions entre les usagers de la route. Par conséquent, les tests de conduite sont essentiels pour valider des fonctionnalités du véhicule autonome. Les tests de conduite dans le monde réel semblent être un grand défi, car des accidents mortels pourraient être provoqués par un véhicule autonome. Alternativement, la simulation numérique permet de réduire le temps et les coûts, et d'éviter les situations potentiellement dangereuses pour la validation des véhicules autonomes. L'intérêt croissant proté à l'utilisation de la simulation numérique du trafic routier souligne l'importance d'une bonne compréhension et d'une bonne modélisation du comportement de conduite humaine.

Dans cette thèse, le travail se concentre principalement sur la modélisation microscopique du trafic pour les modèles de conduite humaine, dans le but de créer des simulations numériques du trafic routier les plus réalistes possibles, à utiliser ensuite pour la validation et les tests des fonctionnalités de véhicules autonomes. Les principales contributions de cette thèse consistent en 1) une approche de modélisation microscopique du trafic pour la génération de collisions inter véhiculaires dans la simulation numérique, basée sur une base de données réelles, 2) un modèle par l'apprentissage par renforcement pour imiter le changement de voie humain sur une autoroute, et 3) un modèle de réseau neuronal, en utilisant le modèle LSTM, pour reproduire simultanément le comportement de poursuite et le comportement de changement de voies.

Génération de collisions par la simulation numérique de trafic Mon premier travail dans cette thèse est l'analyse du comportement du conducteur en ce qui concerne la loi de poursuite. En outre, dans le but de tester la réaction d'un véhicule autonome en cas d'accident dans le trafic, j'ai proposé une approche pour la génération de collisions entre véhicules. Après avoir étudié différents profils de conducteurs dans un ensemble de données de trafic réelles, j'ai classifié trois profils de conducteurs, en distinguant les profils agressif et inattentif, du profil normal. J'ai ensuite proposé d'augmenter la proportion des deux profils de conducteurs (agressif et inattentif) dans la population totale du trafic, et de simuler le trafic à l'aide du simulateur SUMO (Simulation of Urban Mobility), pour observer les occurrences éventuelles de collisions de voitures. Par cette méthode, j'ai caractérisé une relation entre les ratios de ces deux profils de conducteurs sur l'ensemble de la population de conducteurs dans le trafic, et le nombre de collisions de voitures. Cette approche est étudiée sur une partie de l'ensemble de données NGSIM 101, et ensuite, validée sur une autre partie de cette base de données. J'ai également étudié la gravité des collisions générées dans la simulation. J'ai constaté que les collisions impliquées par un conducteur leader inattentif suivi par un conducteur agressif, sont les plus fréquentes; tandis que les collisions impliquées entre deux conducteurs inattentifs sont les plus graves.

Modélisation du changement de voie par apprentissage par renforcement Après avoir étudié le comportement des conducteurs en termes de poursuite (car-following), le deuxième travail de ma thèse porte sur la modélisation des changements de voie, où un modèle d'apprentissage par renforcement a été développé. Le modèle permet d'imiter les décisions réelles de changement de voie, en se basant sur l'ensemble de données de trafic NGSIM. En conséquence, le modèle montre de bonnes performances dans l'imitation des décisions humaines de changement de voie, avec jusqu'à 95 % de précision. En outre, le modèle utilise la simulation numérique du trafic (SUMO) pour compléter les situations inconnues dans la base de données réelles, où nous avons observé que 13 % des conditions de trafic supplémentaires sont créées par l'environnement de simulation du trafic.

Modèle de comportement de conduite humaine par le réseau neuronal LSTM Dans le troisième travail, j'ai proposé un modèle pour prédire le comportement de poursuite et le comportement de changement de voies en même temps, en utilisant les réseaux neuronaux LSTM. Dans ce travail, j'ai proposé différents modèles avec différentes conceptions des variables d'entrée; puis je les ai comparés. Le meilleur modèle sélectionné montre de bonnes performances pour la prédiction de la vitesse longitudinale et de la position latérale des voitures. De plus, les résultats obtenus montrent que ce modèle sélectionné est plus performant que le modèle classique IDM (Intelligent Driver Model) dans la précision de la reproduction du comportement de poursuite. J'ai d'abord implémenté les modèles proposés sur la base de l'ensemble de données NGSIM 101, ensuite sur l'ensemble de données HighD.

Enfin, j'ai implémenté les modèles proposés avec différentes combinaisons et quelques adaptations dans le simulateur de trafic SUMO, afin de simuler l'ensemble du trafic. Dans ces x implémentations, toutes les voitures sont contrôlées par nos modèles proposés, alors que les autres types de véhicules, y compris les camions et les motos, sont contrôlés par le modèle SUMO par défaut. J'ai comparé les modèles et dérivé leurs performances dans la simulation numérique du trafic, pour les deux trafics présentés respectivement dans la base de données NGSIM 101 et la base de données HighD. J'ai trouvé que les modèles basés sur l'IA (un modèle de poursuite utilisant un réseau neuronal et un modèle de changement de voie utilisant l'apprentissage par renforcement) fournissent de meilleures performances, par rapport aux modèles physiquesCependant, la simulation utilisant ces modèles proposés et calibrés avec la base de données NGSIM n'arrive pas à reproduire correctement le trafic de la base de données HighD. Le modèle IDM calibré par NGSIM est moins sensible aux données de trafic et peut être utilisé pour simuler un trafic inconnu comme les données de HighD, mais il requiert une recalibration avec la vitesse maximale des véhicules qui correspond à la condition de trafic dans le base HighD.

Chapter 1 Introduction

Traffic has a significant importance in our daily life, and traffic problems attract researches' attentions, because of its complexity and the uncertainty that arises from human interaction. Researchers are interested in understanding the traffic flow dynamics and the human driving behavior for several reasons : 1) the increase in traffic demand and the limited capacity of road cause a lot of jams; 2) the construction of road infrastructure and driving rules limit human driving behavior; 3) traffic accidents occur every day, which may be caused by the inattention of human drivers; 4) finding useful methods to optimize traffic is essential to improve traffic conditions. This interest of understanding of road traffic dynamics and human driving behavior helps the improvement of road traffic modeling and numerical simulation. In addition, those research works lead to some further advances of some new applications in the future regarding to the intelligent traffic system, such as the novel driver assistance system, and some works on predicting the long-term evolution of traffic for optimizing traffic flow.

Problem statement

According to the UN, 68% of the world population is projected to live in urban areas by 2050 from 55% today (United Nations, 2018 n.d.). Mobility needs in cities are thus expected to grow, at least proportionally to this rate, leading to an increase of car traffic. On the other hand, urban space is scarce and new road infrastructure is not always a feasible solution. The increasing imbalance between demand and supply is expected to aggravate congestion phenomena and exacerbate all relevant externalities: delays and travel time variability, user discomfort, noise and emissions, car crashes. As a result, traffic management remains a hot topic as it allows a better usage of the existing capacity to counter for additional demand. Also, traffic analysis continues to attract research attention as new data become available, new tools Introduction become accessible (such as simulation), network optimization techniques are evolving, new technologies allow for the implementation of cutting-edge techniques. Some researchers and policy makers believe that the arrival of ICVs (Intelligent Connected Vehicles) and AVs (Autonomous Vehicles) will provide a viable solution to urban mobility [START_REF] Sjoberg | Cooperative intelligent transport systems in Europe: Current deployment status and outlook[END_REF][START_REF] Liu | Intelligent and connected vehicles: Current situation, future directions, and challenges[END_REF][START_REF] Laris | Waymo launches nation's first commercial self-driving taxi service in Arizona[END_REF]. Others argue that ICVs may prove highly beneficial in terms of safety and mobility extension of certain groups (seniors, etc), but can bring some unintended effect caused by this new class of travel, such as empty runs [START_REF] Pakusch | Unintended effects of autonomous driving: A study on mobility preferences in the future[END_REF]. In any case, all analysts and stakeholders agree on the fact that the expected benefits are not to be observed in the short-term.

The car industry, in particular, needs to validate and test ICVs in real driving situations both in the field and on simulators; the latter mainly for the case of accidents and other extreme-case scenarios. A transitional period will first occur with increasing levels of automation where the cohabitation of both human and non-human driving vehicles need to be considered. Highways are expected to be concerned first as they provide a better controlled environment compared to city centers. The transitional phase attracts significant research interest as it is expected to be critical for the future full-scale deployment of ICVs and their characteristics: legislative, technical and operational.

The first step towards describing the critical transitional period is to accurately describe driving behaviors. Driving behavior includes a number of decisions made by the driver under variable driving conditions (speed choice, position on the pavement, distance to preceding vehicle, lane-changing maneuver, acceptance gap), but also some rather intrinsic attributes such as reaction time. It is known and well established that driver behaviors are strongly heterogeneous across individuals and, even, across choices of the same individual under different conditions and states. In fact, human factors that are not easily quantified and assessed, such as fatigue, have an important impact upon driving behavior. Consequently, an individual's behavior is hard to be predicted and the collective behavior of a group of drivers is predictable at a certain level of certainty and error. Most behavioral studies use average values of the choice parameters for statistical aggregation. However, a more detailed analysis would provide additional insight and contribute to the better understanding of traffic dynamics over the transitional phase.

Secondly, the aspect of the safety outcome should be put forward after relevant quantitative analysis in order to validate ICVs and increase their acceptability rate as well as the confidence of the general public. Currently, traffic data related to incidents and accidents (both preand post-crash) remain scarce. Collisions are rare events and naturalistic experiments or fullcoverage CCTV (Closed-circuit television) data do not capture them in sufficient numbers to perform statistical analysis. As a result, crashes are poorly represented in traffic simulators also. Regarding pre-crash patterns, risky situations and traffic conflicts lead to collisions with a 1.2 Industrial context specific probability that is not always data-proven. Regarding to post-crash patterns, simulators fail to reproduce the ground-truth and, in most cases, they simply remove all implicated vehicles (SUMO safety, 2021 n.d.). Secondary accidents are thus not considered even though studies show that they represent half of total crashes. The problem is even greater when it comes to ICVs as crash data are almost non-available. In conclusion, a more sophisticated representation of crashes in traffic simulators proves to be necessary for the test and validation of ICVs.

Thirdly, nowadays many tools and knowledge bases become available and have not been fully exploited in the purpose of better understanding human driving and proposing highperformance solution of this cohabitation of human diving and non-human driving. Those tools include (i) new disaggregated data (especially from traffic cameras and drones), (ii) latest developments of traffic software integrating ICV functionalities, (iii) new powerful methods (such as deep learning, reinforcement learning) suitable for modelling from data, and (iv) latest research results from a constantly growing corpus of literature. As a result, we identify here an urgent need for integration and synthesis of existing tools and knowledge in order to propose a holistic approach for the representation and assessment of the cohabitation period, its possible characteristics and risks.

In response to previous remarks and observations, the present thesis focuses on traffic modelling in the context of validation and testing of ICVs especially during the critical period of cohabitation of human and non-human driving. The case studies target urban highways and heterogeneous human driving behaviors under variable traffic dynamic states. From a methodological standpoint, effort was made to integrate many new tools that had not been fully exploited before: real world traffic data, sophisticated modeling and simulation techniques and statistical methods.

Industrial context

The present PhD thesis was realized in an industrial context as it was partly financed by Renault, the leading car industrial stakeholder in France. This paragraph summarizes industrial challenges.

Introduction

in the Figure 1.1. The autonomous vehicles need to be equipped with a variety of sensors to perceive their surroundings, such as radar, lidar, sonar, GPS and inertial measurement units. The Perception module takes sensor data as input and produces a high-level reconstruction and recognition of the scene. The Decision module then chooses the desired behavior and trajectory of the vehicle, based on the current situation Finally, the Control module manipulates forces, by the way of steering and throttle controls, to follow the desired trajectory decided by the decision module. The decision and control is the most complex part, and in the context of Autonomous Driving, the decision and control module is often implemented with a hierarchical structure whose layers work at different timescales.

• Route Planning: The highest level is the route planning, in which, vehicle chooses a route through the given road network. Such a route is formulated as a problem of finding a cost-minimization path on a road network.

• Behavior Decision Making: After choosing the optimal path from route planning, the autonomous vehicle should be able to navigate the selected route and interact with other traffic components depending on the rules of road, road conditions, and signals of infrastructure. In this layer, an appropriate behavior can be selected based on respecting every constraint. So in this layer, it takes optimal route path decided from Route Planning, combining the information of other agents in the road (other vehicles around, obstacles), road signs (eg. speed limit), rules of road (eg. propriety of right) as inputs.

• Motion Planning: Once the Behavioral Layer decides on which driving scenario to be performed, the decision arrive into the motion planning to get a path and trajectory considering comfortable of passenger and avoidance of collision with obstacle.

• Vehicle Control: In order to execute the trajectory calculated in Motion Planning path, the vehicle controller is used to select appropriate inputs to figure out the motion and correct

Industrial context

errors. This layer aims to enhance and make the system more stable. It takes path and trajectory found in before layers and puts out the steering, throttle or brake commands.

The system of autonomous vehicle control is complex, each layer carries out an appreciate decision in each time step while taking the result of layer before and other environment and estimation information as inputs. Otherwise, a good decision making depends of the capacity of estimation environment information and decision model. Every decision need to be computed based on scenario (task of behavior decision making), road conditions (highway, urban, national way), surrounding vehicles types (moto, normal car, truck) and their behavior.

The final motion decision need consider about safety, comfort of passenger and capacity of the vehicle.

Validation of autonomous vehicles

As indicated above, the autonomous vehicle represents a highly complex system. A failure can occur during any stage of this process, which will result in a wrong behavior on the road. Nowadays, the self-driving software testing is realized by million lines of codes included in all the algorithms of perception, decision, and control for the autonomous vehicle. However, it is not enough to ensure the testing of the autonomous vehicle facing on all the various situations in the traffic [START_REF] Koopman | Challenges in autonomous vehicle testing and validation[END_REF]Koopman and Wagner, 2017;[START_REF] Hutchison | Robustness testing of autonomy software[END_REF]. According to the authors in [START_REF] Kalra | Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?[END_REF], to make a car 95% as safe as an experienced driver, autonomous vehicles would have to be driven hundreds of millions of miles and need to be failure-free. Even to make sure the autonomous vehicle is 10% or 20% safer than humans, hundreds of billions of miles autonomous driving are needed to demonstrate their reliability in terms of fatalities and injuries [START_REF] Kalra | Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability?[END_REF]. Alternatively, due to the increase of computer simulation techniques, traffic simulations can benefit the training and testing of autonomous vehicles [START_REF] Belbachir | Simulation-driven validation of advanced driving-assistance systems[END_REF]Chao et al., 2020). In simulation methods of the validation of autonomous vehicle, we can differentiate three principal methods:

1. Data-driven resimulation : Representing the collected real driving scene directly in numerical simulators of data recorded during real driving testing to resimulate what happens in the road numerically. In this way, we can insert autonomous vehicle in the traffic to see how it works in the traffic. The resimulation of driven data allows checking the performance of each components of autonomous vehicle during driving. That is the most simple way by numerical simulation to make sure if the perception system recognizes well the traffic scene, if the decision system makes a good decision, and the controller works well. Resimulation could be the first test step for some software updating.

Introduction

2.

Model-based simulation: This approach is entirely numerical and it generates numerical traffic data. We can easily parameterize the behavior of each vehicle and obtain various traffic conditions. In this approach, the test of autonomous vehicle is expected to be divided into two types : Scenario-Based Validation and Validation in a large-scale traffic.

• Scenario-based Validation: In this method, we define specific scenarios to test separately the functionalities of the autonomous vehicle. such as scenario for testing Adaptive Cruise Control (ACC), scenario for testing the Automatic Emergency Braking (AEB), scenario for testing lane changing and so on.

• Validation in a large-scale traffic: Finally, we expect the simulation of traffic can achieve as a high level of reality, that the autonomous vehicle can drive in simulation like driving in the real world for a long trajectory.

Virtual immersive simulator:

A person drives a real vehicle that is connected to the environment through simulator software using virtual reality glasses or a 180-degree high-resolution screen. This technique is useful in studying the ergonomics and behavior of drivers using autonomous driving modes, while the realism of the scenario remains always a challenge.

All these simulation approaches of the validation of autonomous vehicle need a reconstruction of a realist traffic in simulation environment. That encourages a hard research work on the road traffic modelling and human driving behavior modelling.

Some relative projects in Renault

Renault, as known as one of the biggest automobile manufacturers in the world, also has a highly interest of developing the autonomous vehicle and investing in autonomous vehicle simulation to complete tasks of the validation of autonomous vehicle, by the real-driving testing. Renault participates in some European projects on the validation of autonomous vehicle. The project PEGASUS [START_REF] Pütz | System validation of highly automated vehicles with a database of relevant traffic scenarios[END_REF] develops a validation framework with a database containing relevant traffic scenarios. Scenarios can then be generated for a single use case by choosing certain values for all inputs, and can be launched into a Renault in-house simulation software called SCANeR Studio [START_REF] Champion | Traffic generation with the SCANeR© II simulator: towards a multi-agent architecture[END_REF]. As well, Renault collaborates with other participants in the project UDRIVE [START_REF] Nes | The potential of naturalistic driving for in-depth understanding of driver behavior: UDRIVE results and beyond[END_REF], which has established a rich cross-European naturalistic driving database to study the human driving behavior. This databases provides extensive, reliable insights into driving behavior in real traffic as a foundation for improving the safety and sustainability of European road traffic. In addition, Renault has a long-term strategy on launching a massive simulation more and more accessible, which is useful to complement real test-driving. However, the models in SCANeR Studio for simulating the traffic have many limitations, which avoid to have human-like vehicle movements, that is

Scope and objectives

why the models used in SCANeR Studio need to be improved and to be updated. Moreover, the simulation of several accidents in the traffic via numerical simulation is interesting for the validation of the autonomous vehicle in dealing with traffic accidents. However, for the generation of traffic with accidents, this modelling mechanism is even not completely cited in the existent traffic simulators for the research work, such as SUMO, Vissim, Aimsum. Therefore, the proposition of simulating accidents is also an objective of this thesis.

Scope and objectives

Therefore, we are interested in this thesis in microscopic traffic models, focusing on the human driving maneuvers (longitudinal and lateral decision making in driving task), related to vehicular interactions. By this kind of models, we would like to build a realistic traffic environment for testing and validating some autonomous vehicle's features in the future. There are some interesting topics for this thesis:

Car-following models Car-following models describe the longitudinal behavior of the vehicle.

A car-following model is complete if it is able to describe all situations including acceleration and cruising in free traffic, following other vehicles in stationary and non-stationary situations, and approaching slow or standing vehicles, and red traffic lights. However, CF models have been well developed compared to lane changing models. It is important to well calibrate car-following models.

Lane changing models Lane changing models describe the lateral behavior of the vehicle.

Considering LC behavior, the modeling efforts are not as many as in the CF behavior. This is due to the complexity of the LC behavior, which is affected by the surrounding vehicles (of the considered vehicle) and by the traffic flow environment (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007). In some reviews of lane changing models (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007;[START_REF] Moridpour | Lane changing models: a critical review[END_REF]Z. Zheng, 2014), the authors present several rule-based lane changing models, but they tend to exhibit limited performance due to the uncertainty and to the complexity of the driving environment. For this reason, Machine Learning methods, and especially Reinforcement Learning (RL) ones, provide an alternative approach, which has shown a great success in many different domains, such as robotics, video game playing, dialogue chat bot, etc.

A simultaneous model for car-following and lane changing As well known, two driving behaviors are generally distinguished in the microscopic traffic modeling: Car-Following (CF) and Lane Change (LC), which describe respectively longitudinal and lateral vehicular Introduction movements. Researchers begin to consider the development of models which can describe the car-following as well as the lane changing behaviors, in the same time. This is important because these two behaviors are related each other.

Traffic accidents One of the major challenges of autonomous vehicles is communicating with other vehicles and accurately recognizing the patterns of human driving behavior in mixed traffic [START_REF] Schwarting | Planning and decision-making for autonomous vehicles[END_REF]. As well known, human driving behavior is not perfect and million deaths occur each year due to traffic accidents, among that, 90% of accidents are produced because of driver's mistakes and inattention [START_REF] Singh | Critical reasons for crashes investigated in the national motor vehicle crash causation survey[END_REF]. Therefore, the process of testing autonomous vehicles in dealing with traffic accidents is very important. However, the relationship between different driver profiles and the car-collision occurrences has not been understood clearly. Meanwhile, the procedure of car-collisions generation in traffic numerical simulators is not fully integrated. By this point of view, it is interesting to identify the driver profiles in a traffic data, and understanding their relationship with accident occurrence.

To clarify, the purpose of this thesis is to provide additional insight in the test and validation of ICVs by exploiting and further developing recently developed methods and tools. The specific thesis objectives can be summarized as follows:

1. Enhancing car-collision generation in numerical traffic simulation 2. Exploring the impact of the presence of different driving profiles in the same traffic stream in terms of both car-following behavior and collision severity 3. Enhancing the realism of lane-changing behaviors using reinforcement learning 4. Enhancing the realism of simultaneous car-following and lane-changing behaviors using neural networks

The level of traffic analysis adopted was microscopic. The disaggregated traffic databases used were the NGSIM 101 dataset and the HighD dataset. The main traffic simulator was SUMO (Simulation of Urban Mobility). All the data and tools are fully described in the following chapters.

Outline and Contributions

The remainder of the manuscript is organized as follows:

Chapter 2 reviews the state of the art for the traffic modelling, from classical models to the models using artificial intelligence.

Outline and Contributions

Chapter 3 presents our first research work on the analysis of driver behavior and proposition of a novel approach of vehicular-collision generation in numerical traffic simulator. This work is based on a real traffic dataset, the NGSIM dataset, and considered only car-following behavior.

Chapter 4 describes the second work in my PHD, where we proposed a lane changing model using Reinforcement Learning. This is a model with imitation of real lane change decision from the NGSIM dataset.

Introduction

List of publications

Publications in international conferences with proceedings [START_REF] Zhao | A data-based traffic modeling approach for inter-vehicular collision simulation[END_REF]. A data-based traffic modeling approach for inter-vehicular collision simulation. In Proceedings of 8th Transport Research Arena TRA 2020, April 27-30, 2020, Helsinki, Finland. TRA (used in Chapter 3)

• Lu Zhao, Nadir Farhi, Zoi Christoforou, and Nadia Haddadou (2021b)

Publications of journals

• Lu Zhao, Nadir Farhi, Zoi Christoforou, and Nadia Haddadou (2021a). Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach. Journal of advanced transportation (used in Chapter 3)

Submission of conferences

• Lu Zhao, Nadir Farhi, Zoi Christoforou, and Nadia Haddadou (2022)

Literature Review

In the recent years, many contributions have been made on road traffic modeling, based on several approaches and disciplines, including physics, mathematics, machine learning, behavioral psychology, etc. That leads to a better understanding of driver behavior and traffic flow dynamics. Mathematical models can be roughly classified into macroscopic, microscopic and mesoscopic models. Macroscopic models describe the collective state of traffic in terms of space-time fields for local density, speed, and flow; while microscopic models describe traffic flow from the perspective of individual drivers and vehicles. Mesoscopic models, on the other hand, combine the strengths of microscopic and macroscopic models to simulate traffic at different levels of detail. This chapter reviews the most recent approaches and methods of road traffic modeling and simulation.

Overview of road traffic modelling and simulation

As indicated above, road traffic models can be classified by distinguishing the scale of modeling.

In one of the most recent publication of the SUMO (Simulation of Urban Mobility) traffic simulator [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF], four scales have been considered:

• Macroscopic : presents traffic flow on the basis of continuous fluid dynamics theory, which handles the description of aggregated information such as density (number of vehicles per km), mean speed (km/h), and the vehicular flow (number of vehicles per hour). Macroscopic models describe vehicle's behavior and interactions at a low level of detail, focusing on reproducing aggregated behaviors measured with quantities such as flow density and traffic flux (Chao et al., 2020). At this level of modelling, the vehicles' behavior is homogeneous. Such models are limited for simulating street level traffic, which requires the representation of detailed interactions among individual vehicles.

• Microscopic : each vehicle and its dynamics are modeled individually. Microscopic models are developed for modelling heterogeneous behaviors of agents in the traffic, for diverse road typologies, and interactions among surrounding vehicles. In microscopic models, car-following models and lane changing models are two fundamental ones. They describe respectively the vehicle longitudinal and lateral movement.

• Mesoscopic: a mixture of macroscopic and microscopic. Mesoscopic models are an intermediate approach between macroscopic and microscopic approaches. The core idea of the mesoscopic models is to describe traffic flow dynamics in an aggregate manner while representing the behaviors of individual drivers using probability distribution functions (Chao et al., 2020). Herein, vehicle movement is mostly simulated using queue approaches and single vehicles are moved between such queues.

• Sub-microscopic : each vehicle and also functions inside the vehicle are explicitly simulated, for example, the engine's rotation speed in relation to the vehicle's speed or the driver's preferred gear switching actions. In such models, the vehicle dynamics are generally complex.

One of the advantages of macroscopic models is the computation and simulation time, which is very low compared to the ones needed in microscopic models. Consequently, large networks can be modeled and numerically simulated. However, the vehicle individual behavior is not considered in macroscopic models. For the validation of autonomous vehicle features, one needs to test and validate the safety aspects of the autonomous vehicle's decision making for its movements. It is obvious that the numerical simulation of the local interactions of vehicles is important in this case. Microscopic and/or sub-microscopic traffic models are then necessary. They are more accurate for the modeling of individual vehicle behavior. Sub-microscopic traffic models allow more detailed computations compared to simple microscopic simulations. Furthermore, sub-microscopic models require longer computation times, which restrains the size of the networks to be simulated.

Little words on macroscopic traffic modelling

We are interested in this thesis in microscopic traffic models, focusing on the human driving manoeuvres (longitudinal and lateral decision making in driving task), related to vehicular interactions. By this kind of models, we would like to build a realistic traffic environment for testing and validating some autonomous vehicle's features in the future.

My literature review on the traffic flow modeling is mainly based on one book and some research articles. The book (Treiber and Kesting, 2013b) offers a comprehensive and didactic account of the different aspects of vehicular traffic flow dynamics, and the calibration mechanism for the traffic flow models. In [START_REF] Kotusevski | A review of traffic simulation software[END_REF]Chao et al., 2020), the authors summarized the traffic simulation techniques including different simulation soft-wares, traffic models, traffic data-sets, with some new applications in Autonomous Vehicles. In this chapter, I will begin with a short talk on macroscopic models, then give a review on road traffic microscopic models, with more details.

Little words on macroscopic traffic modelling

Macroscopic traffic models are generally based on a flow-density (or equivalently speeddensity) relationship, traditionally called the fundamental diagram. The latter is obtained from collected traffic data, and presents generally at least two traffic phases: a free flow phase where the vehicular flow increases with the car-density, and where the vehicles run freely; and a

Literature Review

congestion phase where the vehicular flow decreases with the car-density, and where congestion appears. Macroscopic models are generally capable to reproduce important phenomena in road traffic such as the backward propagation of congestion, as well as the shock-wave propagation. One of the most known macroscopic model is the first order one, called the model of Lighthill Whitham and Richards (LWR) [START_REF] Khan | A macroscopic traffic model for traffic flow harmonization[END_REF]. This model combines simply the conservation equation of traffic with a fundamental diagram curve formula. Several other models, mainly second order ones have been developed, in particular to include the acceleration variable into the traffic modeling, such as Payne-Whitham (PW) model [START_REF] Grewal | Identification of parameters in a freeway traffic model[END_REF].

Classical microscopic traffic models

Car-Following (CF) and Lane-Changing (LC) maneuvers describe vehicular longitudinal and lateral interactions on the road respectively. They are two fundamental models for the microscopic traffic modeling and simulation. CF models describe the movements of a vehicle in response to the actions of its leading vehicle (LV), while LC models describe the choice of lane in a multi-lane roads. CF models generally compute the vehicle continuous longitudinal movement, e.g. speed or acceleration. Lane changing models are different from car-following models. The decision of LC is a discrete choice, i.e., performing a lane change, or not. Nevertheless, discrete decisions and actions can also pertain to the longitudinal dynamics; for example, when approaching a yellow traffic light, drivers have to decide whether it is safe to pass or not safe, need to stop. In a traffic simulation system, it is often essential to apply discrete choices, that involves several levels of decision:

1. Strategic level aims at choosing the destination, the mode, and the route.

2.

Tactical level includes changing lanes to enter a priority road. This includes also cooperative behavior such as allowing another vehicle to merge at a point of lane closure. However, modeling the tactical level is notoriously difficult and is only attempted in the most elaborate commercial simulators (Treiber and Kesting, 2013b).

3.

Operative level, the actual decision is made at this level; e.g. the decision of changing lane, stay on the same lane, stop for the yellow traffic light; etc.

4.

Post-decision level, the actions executing the decision are simulated, e.g. performing the lane change or keeping to one's lane, waiting or entering a priority road, or cruising versus stopping at the traffic light.

The CF and LC models we consider here are restricted to the operative and post-decision levels.

Classical microscopic traffic models

Car-following models

A car-following model describes the movements of vehicles in response to the actions of their respective leading vehicles (LV). In a CF model, variables on the individual movement of each vehicle are considered: x n (position), v n (speed), a n (acceleration); see Fig. 2.2, where n is the index of the considered (following) vehicle, and n -1 is the index of the leader one. Some relative variables are also used, such as ∆x = x n-1 -x n (the inter-vehicular distance, which is the rear-to-front bumper distance of two vehicles), ∆v = v n-1 -v n (the relative speed), and ∆a = a n-1 -a n (the relative acceleration). Car-following models are based on the principle of stimulus-response, assuming that every (following) car responds at time t + ∆t to a stimulus of its leader car at time t. The time step ∆t represents the discritization in time, including implicitly the reaction time of the following car.

Among the main car following models, we distinguish First order models (formula (2.1)) from second order ones (formula (2.2)), where ẋn (t) and ẍn (t) denote respectively the car speed and the car acceleration of vehicle n at time t.

ẋn (t + ∆t) = f (∆x(t), ẋn-1 (t))
(2.1)

ẍn (t + ∆t) = f (∆x(t), ∆v(t), ẋn (t), ẋn-1 (t), . . .) (2.2)
Several interesting reviews on car-following can be found in the literature [START_REF] Brackstone | Car-following: a historical review[END_REF]Mohammad and Z. Zheng, 2014;[START_REF] Aghabayk | A state-of-the-art review of carfollowing models with particular considerations of heavy vehicles[END_REF]. I briefly present here the most commonly used. The first car-following models were proposed in 1950s. Since then, a large number of models were developed; we cite the Gazis-Herman-Rothery model [START_REF] Gazis | Nonlinear follow-the-leader models of traffic flow[END_REF], the optimal velocity model [START_REF] Bando | Dynamical model of traffic congestion and numerical simulation[END_REF], the models proposed by Helly [START_REF] Helly | Simulation of bottlenecks in single-lane traffic flow[END_REF], Gipps [START_REF] Gipps | A behavioural car-following model for computer simulation[END_REF], and Wiedemann [START_REF] Wiedemann | Simulation des Strassenverkehrsflusses[END_REF], and the intelligent driver model [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF]. Car-following models can be classified into five categories (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF]: stimulus-based, safety distance, desired measures, optimal velocity, and psycho-physical models.

Literature Review

Gazis-Herman-Rothery (GHR) model

The GHR model [START_REF] Chandler | Traffic dynamics: studies in car following[END_REF]) is a stimulus-response model in which drivers perform their acceleration/deceleration depending on car-speeds, relative carspeeds with respect to leading vehicles, and inter-vehicle distances.

a n (t + ∆t) = αv n (t) β ∆v(t) ∆x(t) γ (2.3)
Where, α, β, γ are parameters of the model.

Gipps model

The Gipps model [START_REF] Gipps | A behavioural car-following model for computer simulation[END_REF] was developed based on the required safety-distance between two consecutive vehicles to avoid car-collision. The speed of the follower vehicle is chosen to always maintain a minimum distance with the leader vehicle, in order to prevent collision, even when the leader vehicle brakes suddenly. In this model, the safe gap with the leading vehicle needs to exceed a minimum value s 0 plus a stopping distance. The worst case scenario is when the leading vehicle suddenly decelerates and stops. In this case, the following vehicle needs to brake to stop too. To guarantee safety, the final distance to leading vehicle should not become smaller than the minimum gap s 0 . The safe distance can be formulated as follows :

s saf e = s 0 + v n T + v 2 n 2b - v 2 n-1 2b (2.4)
where v 2 n 2b and

v 2 n-1
2b describe the necessary stopping distance for the following vehicle n and for the leading vehicle n -1 respectively, b is the maximum deceleration, and T is a reaction time for the following driver. In this way, the safe speed and the real speed taken in t + ∆t of vehicle can be formulated as follows :

v saf e (∆x, v n-1 ) = -b∆t + b 2 ∆t 2 + v 2 n-1 + 2b(∆x -s 0 ) (2.5) v(t + ∆t) = min[v + a∆t, v 0 , v saf e (∆x, v n-1 )] (2.6)
where a is the vehicle maximum acceleration, v 0 is the desired speed for the driver; see Tab.

Krauss model

The Krauss model [START_REF] Krauß | Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics[END_REF] is also based on a safety distance and it is car-following model by default in SUMO (Simulation of Urban Mobility) traffic simulator. As for the Gipps model, the Krauss model calculates directly the follower speed v(t + ∆t) with a delay of reaction

Classical microscopic traffic models

v(t + ∆t) = max(0, v des (t) -µ) (2.7) v des (t) = min[v + a∆t, v 0 , v saf e (∆x, v n-1 )] (2.8) v saf e (∆x, v n-1 ) = v n-1 (t) + ∆x(t) -v n (t)T v n-1 (t)+vn(t) 2b + T (2.9)
In this model, b, a are the maximal deceleration and maximal acceleration, respectively, v 0 is the desired speed of the following driver, and T is its reaction time.

Wiedemann model

The Wiedemann model [START_REF] Wiedemann | Simulation des Strassenverkehrsflusses[END_REF] s, v, ∆v) apply. The boundaries between the regimes are given by nonlinear equations of the form f (s, v, ∆v) = 0 defining curved areas in three-dimensional state space (s, v, ∆v), spanned by the exogenous variables (M. Zhu, Xuesong Wang, and Y. Wang, 2018).

Full velocity difference(FVD) model

The FVD model (Jiang, Wu, and Z. Zhu, 2001) resembles to the optimal velocity model developed in [START_REF] Bando | Dynamical model of traffic congestion and numerical simulation[END_REF]. The acceleration function consists in a term proportional to an

Literature Review

optimal velocity v * (∆x n ) and a term that takes into account the velocity difference as a linear stimulus :

a(t + ∆t) = α[ v * n (∆x(t) -v n (t))] + λ∆v(t) (2.10) λ =    λ 0 if ∆x(t) ⩽ s c 0 if ∆x(t) > s c (2.11)
where α and λ 0 are constant sensitivity coefficients, s c is a threshold between car-following and free driving, and v * n (∆x(t)) is the optimal velocity depending on the headway to the lead vehicle, which can be defined as :

v * n (∆x(t)) = v 0 2 tanh( ∆x(t) b -β) -tanh(-β) (2.12)
where v 0 is the desired speed of the following vehicle, β and b are parameters.

Intelligent Driver Model (IDM)

The IDM [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF] output is the acceleration chosen by the follwing driver :

a(t + ∆t) = a 1 - v n v 0 δ - s * (v n , ∆v) ∆x 2 , ( 2.13)
where

s * (v n , ∆v) := s 0 + max 0, v n T + v n ∆v 2 √ ab , (2.14)
where T is a minimal time headway, a and b are maximal acceleration and deceleration respectively, s 0 is the minimal gap and v 0 is the desired speed which describe also the (maximal) desired speed. This model can describe vehicles' behaviors in a free road and the car-following dynamics.

On a free road, the acceleration respects a relation with the current speed :

a[1 -( vn v 0 ) δ ].
It decreases to zero when approaching the desired speed. The parameter δ controls the reduction of the acceleration value. The decision of decelerating depends on the desired distance s * and the actual gap. When the actual gap is close to s * , the acceleration is nearly zero. Meanwhile, when approaching a slower vehicle, s * increases and it decreases when the leader is faster. In addition, see Table 2.2 and Table 2.3 for the examples of IDM model parameters.

Classical microscopic traffic models

Lane change models

Lane change models determine lane change choices on multi-lane roads and speed adjustments related to lane changing. Considering lane changing behavior, the modeling efforts are not as many as in the CF behavior. This is due to the complexity of the LC behavior, which is affected by the surrounding vehicles and by the traffic flow environment (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007). Fig. 2.3 illustrates surrounding vehicles in LC models. Lane change models are combined with car-following ones in order to have a complete microscopic traffic model. When the car-following model is parameterized to simulate aggressive drivers, the lane-changing driver profiles should be aggressive as well, without introducing new parameters. In some reviews of lane changing models (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007;[START_REF] Moridpour | Lane changing models: a critical review[END_REF]Z. Zheng, 2014), the authors propose Literature Review several rule-based lane changing models. More specifically, LC includes two components : the decision to consider a lane change and the decision to execute the lane change. Lane change decisions are often categorized into mandatory (MLC) or discretionary (DLC). MLC is performed when the driver must leave the current lane to reach the planned destination. DLC is performed in order to gain a speed advantage or a better driving environment. In (Z. [START_REF] Zheng | Recent developments and research needs in modeling lane changing[END_REF], the authors mentioned that the major for LC models in the literature can be also categorized into two groups: 1) models that aim to capture the lane changing decision-making process, where the LC decision is taken only depending on the drivers motivation; 2) models that aim to quantify the impact of lane changing behavior on surrounding vehicles, where the LC decision is taken related to a predictive impact for the overall traffic agents. Furthermore, gap acceptance models are used to model the execution of lane changes and with the purpose of traffic safety. We are interested in this thesis in mostly DLC models that the decision of lane changing is motivated by a gain in speed, and in the safety of execution of lane changing, which is important. I summarize below an overview of classical LC models types; see 2.4.

Gipps' LC model:

In the Gipps' LC model (Gipps, 1986), the decision of lane change is based on the following factors:

• whether it is physically possible and safe to change lanes without an unacceptable risk of collision,

• the location of permanent obstructions,

• the presence of special purpose lanes such as transit lanes,

Classical microscopic traffic models

• the driver's intended turning movement,

• the presence of heavy vehicles,

• the possibility of gaining in speed

In the Gipps' model, which is deterministic, a driver decides to maintain the desired speed or prepare for the LC maneuver, depending on the distance to the intended LC. Moreover, this model assumes that a lane changing takes place only when it is safe, and the gap is enough in the target lane. This model has a serious limitation in congested and incident-affected condition. In the model of Yousil and Hunt [START_REF] Yousif | Modelling lane utilisation on British dual-carriageway roads: effects on lane-changing[END_REF], a similar logic is used to model the decision of lane changing. Instead of a deterministic process of each rule, the model of Yousil and hunt introduces randomness, to make the model more realistic. More specifically, the probability for a driver to start a mandatory LC at a distance x n from the downstream node (or incident, lane drop, etc.) is defined as follows :

p n =    exp[(x n -x 0 ) 2 /σ 2 n ] if x n > x 0 1 if x n ⩽ x 0 (2.15)

MOBIL (Minimizing Overall Braking Induced by Lane Changes)

MOBIL model [START_REF] Kesting | General lane-changing model MOBIL for car-following models[END_REF]) is a discrete decision of lane change maneuver, which is used by combining with a longitudinal acceleration model. According to this model, a vehicle n ∈ [1, N v ] decides to change lane when:

1. it is safe to cut-in:

ãn ≥ -b safe ;

2. there is an incentive, for the ego-vehicle and possibly its followers:

ãn -a n the vehicle +p    ãf -a f old follower + ãf ′ -a f ′ new follower    ≥ ∆a th ;
where f is the old follower in the current lane, f ′ is the new follower in the target lane. ãx and a x , x ∈ {n, f, f ′ } are the future and current acceleration if the lane change takes place and the future acceleration ãx is predicted by a longitudinal acceleration model (e.g. IDM).

The parameter p ∈ [0, 1] models the politeness. p = 0 means a very altruistic behavior, and p ∈ [0, 0.5] is realistic behavior, while p = 1 means very egoistic. We notice that the safety criterion is not ignored: p < 0: very malicious that the driver does not respect safety criterion. δa th is a threshold to respect an overall acceleration if lane change takes place. b safe is the

Literature Review

maximum braking imposed to a vehicle for the lane changing. See Tab. 2.4 for the example of MOBIL model's parameters. [START_REF] Kesting | General lane-changing model MOBIL for car-following models[END_REF]Treiber and Kesting, 2013b) Parameter's name value Politeness factor p [0,0.5] Maximum safe deceleration b saf e 4m/s2 Threshold a th 0.2 m/s2

By application of the right priority rule on most European highways, left lane should only be used for the purpose of overtaking, and passing in the right lane is forbidden unless the traffic is congested. In this case, the 'keep-right' directive of the lane-usage rule is implemented by a constant bias ∆a bias in addition to the threshold ∆a th . Thus, the MOBIL model is reformulate to :

• left to right : ãeur n -a n + p(ã f -a f ) > ∆a th -∆a bias • right to left : a n -ãeur n + p(a f ′ -ãf ′ ) > ∆a th + ∆a bias
where

ãeur n =    min(a n , ãn ) if v n > v lead > v crit a n otherwise
where ãeur n is the acceleration in the right lane, ãn is the acceleration in the left lane, v lead is the speed of the preceding vehcile in the left lane, v n is the speed of lane changer in the right lane and v crit is a critical speed below which the traffic is congested, e.g. v crit = 60km/h. ∆a bias can be set to 0.3m/s 2 .

Cellular automata based LCD models

A typical CA (CA = (ε, Σ, N, δ) model includes four key components : the physical environment (ε), the cells' states (σ), the cells' neighborhoods (N ), and local transition rules (δ), which uses integer variables to describe the dynamical properties of the system. In traffic CA models, the road is divided into cells of a certain length ∆x, and the time is discretized into steps of ∆t. Each road cell can be occupied by a vehicle or it can be empty. In [START_REF] Barlovic | Online traffic simulation with cellular automata[END_REF], the author proposes a microscopic traffic model (car-following and lane changing) based on CA, in which the motion of a vehicle performs with respect to following rules:

• Collision-free acceleration : v

(1)=min(v i ,vmax,g i ) i ,
• Randomisation : with a certain probability p do v

(2) i = max(v (1) i -1, 0) • Movement: x i = x i + v (2) i
Where g i is the gap (the number of empty cells) with the preceding vehicle, x i is the vehicle's position. If a driver wants to change lane, he has to take into consideration the gaps g s and g p (in Fig2.5) on the alternative lane in order to prevent crashes.

Modelling of driving behavior by artificial intelligence

Replicating vehicles' trajectories is one of the primary objectives of modeling vehicles' motion in traffic flow. We can find many classical models for CF and LC models. Although these traditional models have produced great achievements in microscopic traffic simulation, they have limitations :

• Limitation of trajectory accuracy. For example, most current car-following models are simplified, i.e., they contain only a small number of parameters (desired speed, minimal time headway, minimal safe distance...etc.). Simplification leads to suitable analytical properties and rapid simulations. However, it also renders the models limited in flexibility and accuracy, because using few parameters can hardly model the inherently complex driving process.

• Poor generalization capability. Classical models calibrated with empirical data try to emulate drivers' output (e.g., speed and spacing) by finding the model parameters that maximize the likelihood of the actions taken in the calibration dataset. As such, the calibrated car-following model cannot be generalized to traffic scenarios and drivers that were not represented in the calibration dataset.

• Absence of adaptive updating. For example, most parameters of car-following models are fixed to reflect the average driver's characteristics.

To address the above limitations, instead of handcraft manual models, for the driving behavior, machine learning provides an alternative approach based on data analysis that automates analytical model building. It is a branch of artificial intelligence based on the idea that systems can learn from data, identify patterns and make decisions with minimal human intervention.

Literature Review

Deep Neural networks

During the last decades, deep learning (deep neural networks) methods, a group of specific machine learning methods aiming at building data based, haven shown excellent performances in many fields (e.g. natural language processing, image processing, finance trading strategy, etc.) In the field of microscopic road traffic flow modeling, researchers are interested in developing models which can describe and reproduce the vehicle motion in the real road traffic. The use of deep learning approaches for the microscopic modeling of road traffic flow is also becoming increasingly popular, as the physical models are limited in the face to such complex systems.

Though first proposed in the 50's [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF], neural networks became utterly successful since only about a decade ago; thanks to i) the availability of huge amounts of data (mass data), ii) the amazing growth of the available computing power, and iii) the newly developed optimization algorithms (efficient variants of classical gradient algorithms). A neural network (NN) is an interconnected collection of artificial neurons. An artificial neuron transforms given input values into an output value by multiplying the inputs by weights and by running them through an activation function:

Ŷ = ϕ(w • X) (2.16)
where Ŷ is the output, ϕ is the activation function, w is the vector of weights, and X = (x 1 , x 2 , . . . , x n ) is the input vector with n input features. Examples of popular activation functions are the Sigmoid function, the Hyperbolic Tangent (tanh) and the Rectied Linear Unit (ReLU). Several Neurons can be arranged into one layer, then many layers can be arranged together and become a deep neural network. Formally, the output Ŷ of a multi-layer neurons with an input X and with k layers, is given as follows :

   f k (x) = ϕ k (w k • x) Ŷ = f k (f k-1 (..f 1 (X))) (2.17)
In order to learn the weights of a neural network in the context of supervised learning, a loss function must be defined, suitable to the problem to handle. This function should denote how close the predictions provided by the neural network are to the expected predictions called as actual labels (Y ). After choosing the loss function, the goal is to globally minimize this error (loss function L). One of the existing method in the learning process to update the weights, is the Stochastic Gradient Descent (SGD).

In practice, to efficiently train a neural network, epochs and mini-batches are defined for the weights updating process. A mini-batch refers to a subset (with m individual samples) of the whole training set, to compute a partial gradient and update the weights accordingly. An • update weights :

w i ← w i -α 1 m m j=1 ∇L j (w i )

Long short-Term Memory (LSTM) neural networks

Long short-Term Memory (LSTM) neural networks [START_REF] Hochreiter | Long short-term memory[END_REF] are a kind of recurrent neural networks (RNNs), which are able to learn time series with keeping a memory of previous sequences [START_REF] Gers | Learning to forget: Continual prediction with LSTM[END_REF]. The idea of RNNs is to be able to connect previous information to the present task, where previous information has long-term effect on the present behavior. On Fig. 2.6 we have a LSTM neural network, where a sequence of LSTM cells are used to connect the input signal (x 0 , x 1 , ...x t , x t+1 , . . .). A LSTM cell consists of 3 gates (forget gate, input gate, output gate), and each cell can give an output h t (as shown in Fig. 2.6). The forget gate decides how much to "forget" from the previous cell Literature Review state c t-1 . The input gate determines the amount of new information to be stored in memory, based on new input x t and the previous output information h t-1 . The output gate computes the new output by combining the previous cell state and the output of input gate. We give below the operation function of the 3 gates of a LSTM cell :

1. Forget Gate :

f t = σ(w f [h t-1 , x t ] + b f ) 2. Input gate : i t = σ(w i [h t-1 , x t ] + b i ]) c t = tanh(w c [h t-1 , x t ] + b c ) 3. Output gate: o t = σ(w o [h t-1 , x t ] + b o ]) 4. LSTM cell state : c t = f t * c t-1 + i t * c t 5. LSTM cell output: h t = o t * tanh(c t )
LSTM for driving modelling : Some Related works

Several studies have been performed on LSTM networks for human driving behavior modeling, and/or human driving forecasting. We cite [START_REF] Xie | A data-driven lane-changing model based on deep learning[END_REF], where authors developed a LC model (the LC decision prediction combining the LC execution trajectory) using neural networks. In [START_REF] Xie | A data-driven lane-changing model based on deep learning[END_REF], the execution of lane changing trajectory used LSTM neural network. In (Y. [START_REF] Zhang | Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[END_REF], the authors proposed a simultaneous CF and CL model using LSTM neural network, with a constraint of time headway condition to retrain the neural network when this condition is not satisfied. In [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], the authors used a LSTM network for vehicle trajectory prediction from 1s to 10s in the future. All these works show that LSTM is essentially useful for the prediction of human driving behavior. However, different studies used different model structures, and they used also different input designs for the neural networks.

Reinforcement Learning (RL)

Reinforcement Learning (RL) is a mathematical approach for learning by interaction in a complex and uncertain environment, where an agent tries to maximize the total amount of a reward received from the environment. Different from the supervised learning method, RL guides

Modelling of driving behavior by artificial intelligence

the agent to discover most rewarded actions by trial-and-error exploration in the environment, meanwhile agent's action affects the environment changes also. Many researchers have shown the great efficiency of using RL for car-driving modeling. Reinforcement learning(RL) is a general framework for decision-making under uncertainty. It provides an agent who intends to solve optimal control problems based on Markov Decision Processes(MDP).

Markov decision process

A Markov decision process (MDP) describes the sequential decision making process, denoted (S, A, P, R, γ), where S is the state space, A is the discrete action set, P denotes the transition probabilities,R(S, A) is the rewards, and γ is a discount factor. The objective is to seek a policy π(a|s) applied in every time step, that maximises the the future return (G t ), which is discounted by γ (the discount factor), with 0 ≤ γ ≤ 1. G t is defined as:

G t = R t + γR t+1 + γ 2 R t+2 + ... = ∞ k=0 γ k R t+k , (2.18)
where R t+k is the reward given at time step t + k. The return value G t should be a random variable along a trajectory τ = (s 0 , a 0 , s 1 , a 1 , . . .) which indicates the accumulated rewards induced by the policy π(a t |s t ) and the system dynamics P (s t+1 |s t , a t ). The discount factor γ essentially determines how much the distant rewards impacts the agent's decision of each action. If γ = 0, the agent will be completely short-sighted and takes care only of the immediate reward produced by the action. If γ = 1, the agent will evaluate each action based on the total of all future rewards; see [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] for a further comprehension of MDP and reinforcement learning.

State Value Function and Action-State Value function

The state value function, denoted V π (s), is defined as the expected return of the policy when starting with a given state s :

V π (s) = E[G|s 0 = s] (2.19)
The state-action value function, denoted Q π (s, a), is defined as the expected return when starting with a given state s and a given action a :

Q π (s, a) = E[G|s 0 = s, a 0 = a] (2.20)

Literature Review

State Value Function and Action-State Value function Optimality

As mentioned above, the objective of a reinforcement learning task is to obtain a policy which can achieve a great return for the agent. The policy with the greater expected return than all other policies consists in the optimal policy, denoted π * . Therefore, we can obtain the optimal policy by maximizing either the state-value function or the the action-value function. We also define the optimal state-value function V * (s) and the optimal action-value function Q * (s, a) as follows :

∀s ∈ S, V ⋆ (s) = ∆ Q π ⋆ (s) = max π V π (s); ∀(s, a) ∈ S × A, Q ⋆ (s, a) = ∆ Q π ⋆ (s, a) = max π Q π (s, a).
Based on the Bellman principle of dynamic programming (see for example in [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF]), the Bellman equation on V * (s) is written as follows :

V * (s) = max a q π * (s, a) = max a E π * [G t |s t = s, a t = a] = max a E π * [ ∞ k=0 γ k R t+k+1 |s t = s, a t = a] = max a E π * [R t+1 + γ ∞ k=0 γ k R t+k+2 |s t = s, a t = a] = max a E[R t+1 + γV * (s t+1 )|s t = s, a t = a] (2.21)
The Bellman equation on Q * (s, a) is written as follows :

Q * (s, a) = E[R t+1 + γ max a ′ Q * (s t+1 , a ′ )|s t = s, a t = a] (2.22)

Model-free and Model-based approaches

Reinforcement learning algorithms can be grouped into two main families, called Model-free and Model-based. Model Based algorithms attempt to estimate P (s t+1 |a t , s t ) and R(s t , a t ) of the MDP, based on a history of transitions D = {(s t , a t , r t , s t+1 )}, using for example Maximum Likelihood Estimation, with a hypothesis for dynamics function :

max P t P (s t+1 | s t , a t ) (2.23)

Modelling of driving behavior by artificial intelligence

and reward function :

min R t ∥R(s t , a t ) -R(s t , a t )∥ 2 (2.24)
Thus, the estimated MDP(S, A, P, R, γ) can be achieved and the optimal policy π * can be computed by dynamic programming. In the other side, model-free algorithms aim at getting the optimal policy π * directly, without knowing the P (s t+1 |a t , s t ) in the MDP.

Model-free methods

Value-based methods: Among the model-free algorithms, the Value-based methods optimizes the value function by evaluation and improvement. Then, from the optimal value function, the optimal policy is attained by selecting the action at a state with the largest optimal value.

Policy-based methods: Differently, in the Policy-based methods, the agent learns policies directly, where we define an objective function which could be the value function of the initial state s 0 for policy π θ :

J(θ) = V π θ (s 0 ) (2.25)
and aiming at finding : θ * = arg max θ J(θ) = arg max θ V π θ (s 0 )] It can be formulated by the gradient ascent :

∇J(θ) = ∇V π θ (s 0 )] (2.26)
Actor-Critic: Actor-Critic methods are a combination of value and policy based methods consisting of 1) the "Critic" to estimate the value function and 2) the "Actor" to update the policy distribution in the direction suggested by the Critic (e.g. with policy gradients). In Figure 2.7, a simple Actor-Critic algorithm is shown :

We give in Table 2.5 some famous reinforcement learning algorithms :

Literature Review Algorithm type Q-learning [START_REF] Watkins | Q-learning[END_REF] Value-based Sarsa [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] Value-based

Deep Double Q-learning [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF] Value-based REINFORCE [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF] Policy-based. A3C [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF] Actor-Critic DDPG [START_REF] Lillicrap | Continuous control with deep reinforcement learning[END_REF] Actor-Critic PPO [START_REF] Schulman | Proximal policy optimization algorithms[END_REF] Actor-Critic Table 2.5 -Summary of some famous reinforcement learning algorithms.

Modelling of driving behavior by artificial intelligence

Reinforcement learning for lane-changing model: some related works

Reinforcement Learning (RL) gains increasingly great performance among other methods for modelling and automating driving behaviour. Especially for lane-changing decisions, for which previously existed models were less efficient at describing the complexity of the driver's behaviour, RL methods perform as good as rule-based models, or in some cases, even better. Some works in RL approach the decision learning problem using discrete state and/or action spaces, while other works treat use continuous spaces. Depending on this decision regarding the description of the state and actions spaces, different RL algorithms have been applied either to describe both longitudinal and lateral vehicle movement or only one of them.

In (L. Wang et al., 2019), the authors developed an ego-efficient lane-change strategy using Q-learning, and also tested the effects of the strategy on the network level through Aimsum traffic simulation platform. The possible actions considered were three included a left lane change, a right lane change, and no change of lane. The state consisted of nine variables which included the ego-vehicle's acceleration gain from a lane change, the acceleration gain of the surrounding vehicles as well as the mean acceleration difference between the middle lane and the other lanes, for a number of vehicles downstream the ego-vehicle, and its standard deviation. All variables were discretized corresponding to six predefined intervals. The reward is defined as the acceleration gain, that allows to optimize the traffic condition by maximizing the controlled vehicle speed. In the experiment, the simulated traffic environment consists of two kinds of vehicles (ego-vehicles, controlled by teh IDM model (Intelligent Driver Model) for car-following, and RL for lane changing. Normal vehicles are controlled by the IDM model for car-following and the Gipps model for lane changing. Numerical simulation showed an improvement in speed of the vehicle adopting the lane-changing strategy developed with RL. The impact on the network was also positive, as more vehicles adopted this strategy, but only while the percentage of these vehicles remained below 60%.

Another approach was followed by (P. [START_REF] Wang | Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm[END_REF] where the authors used an actor-critic algorithm for the automation of lane changing decisions. Using the Deep Deterministic Policy Gradient (DDPG), they extended the problem to a continuous action space considering the yaw acceleration as the lateral control variable. The state representation included the ego-vehicle's speed, position, orientation and acceleration, the speed and position of vehicles in the target lane, the road geometry, etc. The reward function took into account the lateral dynamics, punishing large yaw rates and yaw acceleration, the maneuvering time, and the position deviation. The results from numerical simulation showed that, after sufficient training, the agent learned efficiently how to make a smooth lane-changing maneuver with a success rate of 100%. [START_REF] Ye | Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning[END_REF] also used an actor-critic algorithm to propose a method for automating lanechanging decisions. They proposed a model for mandatory lane-changes developed using the

Literature Review

Proximal Policy Optimization algorithm. In this work as well the state space was continuous, but the action space was discrete. The state representation included the ego-vehicle's longitudinal speed, position and acceleration, the lateral position and speed, as well as the relative distance of the surrounding vehicles, the longitudinal speed, acceleration and lateral position. Four surrounding vehicles were taken into account : the leading and following vehicle in the current lane and the leading and following vehicle in the target lane. The action space was designed in both lateral and longitudinal direction, and included 6 possible actions. The action was a combination of deciding whether to keep or change a lane, as well as abort a lane-changing maneuver, and selecting which leading vehicle to follow longitudinally. The reward function was designed so as to incorporate comfort, safety and efficiency of the lane change. It included a part for evaluation of the lateral and longitudinal jerk, another part for evaluation of the travel time and relative distance to the target lane, and a part for the evaluation of the collision risk. The numerical simulation results showed that the agent could learn successfully the lane changing strategy with a success rate of 95% and a collision rate of 2%, which outperformed the trained rule-based agent used as a reference.

With the aim to study both longitudinal and lateral movements, [START_REF] Hoel | Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning[END_REF] proposed a model for the automation of both speed and lane-changing decisions based on Deep Q-Learning. Using this algorithm, which aims to approximate the optimal Q-function with a deep neural network, they were able to treat the state space as continuous in contrast with simple Q-learning. In this work the state consisted of 27 variables which represented the ego-vehicle's normalised speed, the existence of lanes beside it, the normalised relative speed and position of the surrounding vehicles, and a value between (-1) and 1 depending on the lane of each surrounding vehicle with respect to the ego-vehicle. The action space was discrete but included also decisions for acceleration or deceleration along with keeping or changing the lane. In [START_REF] Hoel | Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning[END_REF], the reward was positive and based on the normalized driven distance during a time step, while also penalising with (-10) in the condition of collision or driving out of the road. A small negative reward of (-1) was also assigned to lane-changes in order to limit their number. Two agents were implemented: an agent which developed with DQN a strategy only for lane-changing, and used the IDM model for car-following, and an agent trained to control both speed and lane-changing. Using the IDM and MOBIL models as a reference, the results showed similar performance with the reference model for the first, but better performance for the second, indicating that maybe longitudinal and lateral decisions should not be studied separately. Finally, the application to both a highway driving case and an overtaking case with oncoming traffic illustrated the generality of their method.

In Table 3.2, I give out the overview of these related works using reinforcement learning based algorithms for the driving modelling, which I presented above. I indicate how state, action and reward variables have been defined.

Modelling of driving behavior by artificial intelligence

Source

Literature Review

Traffic safety and Accidents

Another topic of interest in this thesis is the car-collision occurrence modeling. We are interested in generating car-collisions by numerical simulation of the road traffic, in order to evaluate how the autonomous vehicle can deal with a traffic with car-collisions and accidents. In this field, a large body of literature focuses on the traffic safety, especially exploring the link between ambient traffic conditions and collision occurrences [START_REF] Shi | Key risk indicators for accident assessment conditioned on pre-crash vehicle trajectory[END_REF] or crash injury severity [START_REF] Savolainen | The statistical analysis of highway crash-injury severities: a review and assessment of methodological alternatives[END_REF]. On highways specifically, the authors of [START_REF] Salem | Risk index model building: application for field ramp metering safety evaluation on urban motorway traffic in Paris[END_REF] provide evidence that crashes have a specific relationship with traffic flow characteristics (volume, density, speed). In [START_REF] Christoforou | Identifying crash type propensity using real-time traffic data on freeways[END_REF], the authors identified the impact of general geometric parameters, weather, and traffic flow on different types of crashes (rear-end, sideswipe, multiple vehicle involved) based on a French traffic and crash data-set. Other studies employed statistical and machine learning approaches, such as Multivariate Probit models [START_REF] Christoforou | Identifying crash type propensity using real-time traffic data on freeways[END_REF] or Support Vector Machine (Dong, Huang, and L. Zheng, 2015), to investigate the relationship between influencing traffic factors and crash occurrences. Another approach consists in using traffic simulation for crash modeling. In (Azevedo, Cardoso, and M. E. Ben-Akiva, 2018), the authors proposed models for three types of crashes depending on vehicle interactions and maneuvers, and estimated the parameters of the models via simulation. In (J. Wang et al., 2017), the authors studied the impact of driving violations (driving over speed limit, slow driving, and abrupt hard braking) on crash occurrence, for different traffic conditions through simulation.

Road traffic simulators

There exists a wide variety of traffic simulators based on microscopic models, multi-agent or cellular automata. However, most of the simulators are for a business purpose, and therefore provide very few design details, such as Aimsun (Casas et al., 2010a), PTV Vissim (Fellendorf and Vortisch, 2010a), Paramics [START_REF] Cameron | PARAMICS-Parallel microscopic simulation of road traffic[END_REF]. Some open source alternatives exist also, such as SUMO [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF], MATSim [START_REF] Horni | The multi-agent transport simulation MATSim[END_REF] and MITSIMLab (M. [START_REF] Ben-Akiva | Traffic simulation with MITSIMLab[END_REF]. In a recent publication [START_REF] Luo | Simulating Autonomous Driving in Massive Mixed Urban Traffic[END_REF], the authors present their simulator SUMMIT and compare it with some other simulators in the context of testing autonomous vehicles in terms of realworld maps integration, possibility of simulating unregulated behaviors, possibility of simulating dense traffic, and realistic visual & sensors; see Table2.7 below.

In this thesis, we used principally SUMO traffic simulator, and we realised some works also using SCANeR studio driving simulator, which is a Renault internal simulator for the autonomous vehicle.

Road traffic simulators

Simulator real-world maps

unregulated behaviors Dense traffic realistic visual & sensors SimMobilityST ✓ x ✓ x SUMO ✓ x ✓ x TORCS x ✓ x ✓ Apollo x x x ✓ Sim4CV x x x ✓ GTAV x x x ✓ CARLA x x x ✓ Autono ViSim x ✓ ✓ ✓ Forced-based simulator x x ✓ ✓ SUMMIT ✓ ✓ ✓ ✓

SUMO traffic simulator

"Simulation of Urban MObility", or "SUMO", is an open source, microscopic, multi-modal traffic simulator [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF], including modeling of road vehicles, public transport and pedestrians. It is released since 2001 and is available for many state-of-the-art microscopic traffic models. SUMO provides traffic simulations, including synthetic road network creation, real road network import, traffic and emission computation, and simulation visualisation. SUMO allows to generate a large number of different measures. The outputs are collected into files or a socket connection following the common rules for writing files. In addition, SUMO can be enhanced with custom models, and it provides various APIs to remotely control the simulation.

Literature Review

In principal, in SUMO, the microscopic driving dynamics of road vehicles are determined by the interplay of several models briefly listed below [START_REF] Erdmann | Lane-changing model in SUMO[END_REF]:

• Car-following model: determines the speed of a vehicle in relation to the vehicle ahead of it.

• Intersection model: determines the behavior of vehicles at different types of intersections in regard to right-of-way rules, gap acceptance and avoiding junction blockage.

• Lane-changing model: determines lane choice on multi-lane roads and speed adjustments related to lane changing.

SCANeR studio driving simulator

Renault has been developing driving simulators since several years. SCANeR studio is the driving simulator software which includes vehicle dynamics, traffic generation, visual and kinaesthetic modules for driving simulators. SCANeR studio provides a default traffic vehicle model which controls the traffic vehicle's movement in real time [START_REF] Champion | Traffic generation with the SCANeR© II simulator: towards a multi-agent architecture[END_REF]. The architecture of SCANeR studio is shown in Figure 2.9.

Road traffic data-sets

Regarding to traffic data collection, the traffic can be directly observed by cameras on top of a tall building or by cameras fixed in a drone. Then, using some object-tracking algorithms, the 2.7 Road traffic data-sets trajectories, i.e. the positions of each vehicle over time, can be extracted from the video. By this process, vehicles within a given road section are captured in this way, a microscopic traffic trajectory data-set is achieved. These kinds of data-set can be used for studying both on the human driving behavior and the traffic flow. In this thesis, we have used two traffic trajectory data-sets : NGSIM [START_REF] Alexiadis | The next generation simulation program[END_REF] and HighD [START_REF] Krajewski | The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems[END_REF]. The work in the Chapter 3 and in the Chapter 4 are based on NGSIM dataset. The work in the Chapter 5 is first based on NGSIM dataset, then we have validated the model using HighD dataset. In addition, the work in the Chapter 6 is on NGSIM data-set and HighD data-set to understand and compare the sensitivity of each proposed models. I give in this section some descriptions of the NGSIM and the HighD data-sets.

NGSIM data-set

NGSIM 101 [START_REF] Alexiadis | The next generation simulation program[END_REF]) is an open data-set released by the US Federal Highway Administration (FHWA). The dataset acquisition camera and the road section map is shown in Figure 2.10. In NGSIM 101 dataset, all vehicle trajectories are provided with a rate of 10Hz, In Figure 2.11, we present traffic volumes and speeds during each of the three time periods. It can be observed that, in the beginning of the first 15 minutes, the mean speed is between 10 and 15 m/s. After that, traffic becomes denser resulting to a sudden fall of the mean speed. In the second and third 15-minute time periods, traffic is more congested and mean speed varies between 5 and 10 m/s.

HighD data-set

Similarly to the NGSIM dataset, the HighD dataset [START_REF] Krajewski | The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems[END_REF] is a dataset of vehicle trajectories recorded on German highways. The vehicular traffic was recorded at six different locations and includes more than 110 500 vehicles using the camera from a drone(see. Figure 2.12). Each vehicle's trajectory, including vehicle type, size and manoeuvres, is automatically extracted. Using state-of-the-art computer vision algorithms, the positioning error is typically less than ten centimeters. Although the dataset was created for the safety validation of highly automated vehicles, it is also suitable for many other tasks such as the analysis of traffic patterns or the parameterization of driver models.The HighD dataset provides totally 60th sub-datasets with different road sections and different time periods. Differently from the NGSIM dataset, the HighD dataset has a higher resolution and its data frequency is 25Hz, while the data frequency of the NGSIM dataset is 10Hz.

Road traffic data-sets

Figure 2.12 -HighD dataset recording setup includes a drone that hovers next to German highways and captures traffic from a bird's eye view on a road section with a length of about 420 m [START_REF] Krajewski | The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems[END_REF] Chapter 3

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

The emergence of Intelligent Connected Vehicles (ICVs) is expected to contribute to resolving traffic congestion and safety problems. However, it is inevitable that ICV safety issues in mixed traffic (involving ICVs and human driven vehicles) will be a critical challenge. The numerical simulation of scenarios involving a mix different driving profiles is expected to be an important safety assessment tool to use in the process of testing and validating ICVs. Numerical simulation should be specifically important regarding extreme scenarios, including car collisions, which are rarely captured in real-world data-sets and can be generated numerically. In this work, we propose a novel approach for car-collision generation in numerical simulations based on the assumption that car-collision occurrences are mostly associated with certain specific driver profiles. Using the NGSIM 101 data-set, we identify three different driver profiles: aggressive, inattentive and normal drivers. We then replicate car-collision occurrences by varying the percentages of these three driver profiles in the simulated environment, allowing us to establish a relationship between driver profiles and car-collision occurrences. We also investigate the severity of generated car-collisions and classify them with respect to the driver profiles of the cars involved in the collisions. Our approach of replicating car-collision occurrences in numerical simulations will facilitate the testing and validation of ICVs in the future, especially regarding the testing of ICV functionalities in dealing with traffic accidents.

Motivation

With the development of the information and the telecommunication technologies, along with the rapid growth of New Energy Vehicles (NEVs), Intelligent Connected Vehicles (ICV) have become an increasingly active research topic. ICVs are expected to reshape future mobility and contribute to mitigating road traffic congestion and safety problems [START_REF] Fagnant | Preparing a nation for autonomous vehicles: opportunities, barriers and policy recommendations[END_REF].

One of the major challenges of ICVs is communicating with other vehicles and accurately recognizing the patterns of human driving behavior in mixed traffic [START_REF] Schwarting | Planning and decision-making for autonomous vehicles[END_REF]. As we know, human driving behavior is not perfect and million deaths occur each year due to traffic accidents, among that, 90% of accidents are produced because of driver's mistake and inattention [START_REF] Singh | Critical reasons for crashes investigated in the national motor vehicle crash causation survey[END_REF]. Therefore, the process of testing ICVs in dealing with traffic accident is very important. Moreover, the validation and the testing of ICVs in mixed traffic consisting of ICVs and human driven vehicles, are necessary to ensure the security of ICVs functionalities. However, performing physical tests of ICVs is not only time-consuming, but it is sometimes dangerous as well, especially in the traffic with accidents. The numerical traffic simulation is an alternative tool to test and validation autonomous vehicle. One can generate easily massive scenarios even dangerous ones with vehicular collisions, which happen rarely in the real life. It is important to understand human driver behaviors in order to observe the collision related driver profiles. However, the relationship between different driver profiles and the car-collision occurrences has not been understood clearly. Meanwhile the procedure of car-collisions generation in traffic numerical simulators is not fully integrated. We propose in this chapter an approach for generating vehicular-collision in numerical traffic simulation with the purpose of facilitating the testing and validation of ICVs in the future.

Background

It is well known that traffic numerical simulation is established by using mathematical models to reconstruct road traffic [START_REF] Chao | A survey on visual traffic simulation: Models, evaluations, and applications in autonomous driving[END_REF], and can be performed on many scales (i.e. microscopic, mesoscopic, macroscopic, etc.). In particular, for the validation of the ICVs behavioral decision system, microscopic traffic simulators are particularly interesting, as they focus on replicating the vehicles' individual behaviors in details, in order to create an entire traffic environment. Various microscopic traffic simulators have been developed; we cite SUMO [START_REF] Krajzewicz | Recent development and applications of SUMO-Simulation of Urban MObility[END_REF], VISSIM (Fellendorf and Vortisch, 2010b), and AIMSUN (Casas et al., 2010b). However, the vehicle collision generation mechanism has not yet been described completely in any of those simulators.

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

As well known, drivers have different driving patterns. Therefore, several driver profiles can be distinguished and observed from real driven data, e.g. real vehicle trajectories. Next Generation simulation (NGSIM) published four vehicle trajectory data-sets (NGSIM [START_REF] Alexiadis | The next generation simulation program[END_REF])), which have been widely used by many researchers in transportation [START_REF] He | Research based on high-fidelity NGSIM vehicle trajectory datasets: A review[END_REF]. More recently, data-sets on motion trajectory of traffic objects detected by autonomous vehicle equipment were published (e.g. Waymo data [START_REF] Waymo | Waymo Open Dataset[END_REF], [START_REF] Argoverse | Argoverse[END_REF], nuScenes (nuscenes, 2019), etc.). These data-sets can be used to observe the single human driver behavior (Marina [START_REF] Martinez | Driving Style Recognition for Intelligent Vehicle Control and Advanced Driver Assistance: A Survey[END_REF]. However the whole traffic situation is not captured in these data-sets. The driving profile identification is mostly based on clustering approaches with real driven data [START_REF] Júnior | Driver behavior profiling: An investigation with different smartphone sensors and machine learning[END_REF]. Many open data-sets on vehicular traffic are nowadays available and could be applied for studying and analyzing human driving behavior, but driving profile patterns are not given directly in the data-sets. The authors of [START_REF] Meiring | A review of intelligent driving style analysis systems and related artificial intelligence algorithms[END_REF] reviewed various methods of classification of driving styles: 'normal', 'aggressive', 'inattentive' and so on. Aggressive drivers are often characterized by risky speeding (such as abrupt instantaneous speed change, over limit speed, excessive acceleration or deceleration) and frequent lane changing behavior [START_REF] Aljaafreh | Driving style recognition using fuzzy logic[END_REF]. Inattentive drivers can be characterized by long reaction time and forced sudden lane change with an instantaneous deviation from normal behavior [START_REF] Meiring | A review of intelligent driving style analysis systems and related artificial intelligence algorithms[END_REF]. Some studies used clustering approaches (K.-T. Chen and H.-Y. W. Chen, 2019) to classify the different drivers profiles, but this kind of methods have the uncertainty in the result of classification, the algorithm complexity, and the need of specific data-sets, which makes the method hard to generalize.

In the other hand, vehicle collisions are rarely observed. In [START_REF] Abdulhafedh | Road traffic crash data: an overview on sources, problems, and collection methods[END_REF], several vehicle collision related data-sets have been summarized. However, most of these data-sets provide only vehicle collision statistics, without involving car-trajectories. Nevertheless, the Strategic Highway Research Program 2-Naturalistic Driving Study (SHRP2-NDS, 2016) provides a data-set where several accident situations are captured, yet in terms of quantification of the vehicle collision causes and impacts, it seems to be unsatisfactory. Therefore, a captured vehicle collision study would be an untouched research area.

As what we can find, a large body of literature on road traffic safety explores the link between ambient traffic conditions and collision occurrences [START_REF] Shi | Key risk indicators for accident assessment conditioned on pre-crash vehicle trajectory[END_REF] and/or vehicle-collision injury severity [START_REF] Savolainen | The statistical analysis of highway crash-injury severities: a review and assessment of methodological alternatives[END_REF]. On highway specifically, the authors of [START_REF] Salem | Risk index model building: application for field ramp metering safety evaluation on urban motorway traffic in Paris[END_REF] provide evidence that vehicle collisions have a specific relationship with traffic flow characteristics (volume, density, speed). In [START_REF] Christoforou | Identifying crash type propensity using real-time traffic data on freeways[END_REF], the authors identified the impact of general geometric parameters, weather, and traffic flow on different types of collision (rear-end, sideswipe, multiple vehicle involved) based on French traffic and vehicular statistic data. Some other studies employed statistical and machine learning approaches, such as Multivariate Probit models (Christoforou, Cohen, and

Materials and Method

Karlaftis, 2011) or Support Vector Machine (Dong, Huang, and L. Zheng, 2015) to investigate the relationship between influencing traffic factors and vehicle collision occurrences. Several previous works consist in using traffic simulation for vehicle collision modeling. In (Azevedo, Cardoso, and M. E. Ben-Akiva, 2018), the authors proposed models for three types of vehicle collision depending on vehicle interactions and maneuvers, and estimated the parameters of the models via simulation. In (J. Wang et al., 2017), the authors studied the impact of driving violations (driving over speed limit, slow driving, and abrupt hard braking) on carcollision occurrence for different traffic conditions through simulation. Nevertheless most of the literature treats the road traffic safety, without considering driving profiles.

In this work, we propose a novel approach for car-collision (collision between cars) generation in numerical simulation by varying the percentages of different driver profiles in the traffic, aiming at establishing a relationship between driving profiles and car-collision occurrences. The profiles are extracted from the NGSIM database and integrated in the traffic simulator SUMO [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF], using the calibrated Intelligent Driver Model (IDM), which is one of the most human-like car-following model [START_REF] Vasconcelos | Calibration of the Gipps car-following model using trajectory data[END_REF]M. Zhu, Xuesong Wang, Tarko, et al., 2018).

Our purposes in this work are: 1) Extracting driver profiles from a real driven data, NGSIM 101 data-set; 2) Replicating different driver profiles using microscopic traffic model after a calibration; 3) Establishing a method of reproducing realistic traffic simulation based on microscopic road traffic modelling in SUMO simulator; 4) Generating car-collisions by appropriately varying the percentages of the driver profiles; 5) Characterizing the relationship between the car-collision occurrences with the driver profiles; and 6) Observing the severity of the generated car-collisions.

Materials and Method

In this section, the concerned materials and the proposed method will be presented. The approach structure is illustrated in Figure 3.1. We study the driver profiles from car-trajectories and we propose the definition of an 'aggressive' driver profile as the one who leaves always short time-headways with respect to the leader vehicle, and an 'inattentive' driver profile as the one with particularly long reaction time, compared to the other drivers. We classify all the drivers with intermediate values of reaction time and time-headway at the 'normal' driver profile. The thresholds for the three driver profiles are derived from the NGSIM 101 data-set [START_REF] Alexiadis | The next generation simulation program[END_REF]. After the classification of the driver profiles, the specific driver profiles (aggressive and inattentive) are represented using calibrated IDM model (the IDM model with an extension of reaction time), and the road traffic is then simulated using SUMO ("Simulation of Urban MObility") [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF]. The IDM model calibration is

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

performed by using a genetic algorithm, to find the optimal set of IDM parameters with an objective minimize the predefined error between real driver trajectory and the output of the IDM model. In essence, we artificially increase the percentage of drivers with 'extreme' profiles (aggressive or inattentive) and, then, count and analyse the car-collisions generated by traffic simulation. In the numerical simulation experiments, we proposed 4 different experiments. The remaining of the presentation of this work is structured as follows: In Section 3.4, we present the classification of driver profiles from the NGSIM 101 dataset, and this part includes 1) the presentation of the NGSIM 101 dataset 2) the proposition of different driver profiles, and 3) the profiles extracted from NGSIM 101 dataset. In Section 3.5, we present SUMO traffic simulator, the choice of car-following model, IDM model, and the calibration of the IDM model for aggressive driver profiles and inattentive driver profiles, as well as for the normal driver profiles. Section 3.7 shows in details for the 4 simulation experiments for the generation of vehicular collisions. Section 3.8 presents the results obtained from numerical simulations and the investigation of the relationship between generated car collisions and driver profiles. Section 3.9 presents the validation of the approach with a different database. Section 3.10 deals with the severity of simulated car-collisions with respect to different driving profiles. Finally, Section 3.11 summarizes major findings and provides recommendations for further research.

Classification of driver profiles from the NGSIM 101 dataset

A driver profile can be defined as the average driving behavior of a given driving class [START_REF] Itkonen | Trade-off between jerk and time headway as an indicator of driving style[END_REF]. Driving behavior is related to driver baseline skills, socio-demographics (age,

Classification of driver profiles from the NGSIM 101 dataset

gender, occupation... etc.) and current state (fatigue, distraction... etc.). From driven data, driver's characteristic can be illustrated by the speed, the acceleration, the jerk, and some other relevant traffic indicators, such as time-headway (THW) and time to collision (TTC) [START_REF] Itkonen | Trade-off between jerk and time headway as an indicator of driving style[END_REF]. THW and TTC are critical safety indicators for car-following behavior [START_REF] Suweda | Time headway analysis to determine the road capacity[END_REF]. THW measures the time passed for reaching the leader's position while running at current speed, while TTC is usually used for judging the moment to start braking, and in the control of braking [START_REF] Ayres | Preferred time-headway of highway drivers[END_REF]. Several studies show that the distribution of THW is related to driving speed and also to traffic flow conditions. According to [START_REF] Winsum | Choice of time-headway in car-following and the role of time-to-collision information in braking[END_REF], a negative correlation between the car-speed and THW can exist. In [START_REF] Maurya | Study on speed and time-headway distributions on two-lane bidirectional road in heterogeneous traffic condition[END_REF], it was shown that the speed and THW follow different distribution patterns under different traffic density levels. In this work, the classification of different driver profiles begins with the observations from the NGSIM data.

NGSIM 101 dataset description

A brief description of the NGSIM 101 data-set has been given in section 2.7.1. In this data-set, it provides totally 3 subsets of 15-minute traffic registration. In this work, we applied the proposed approach based on the first 15-minute data-set. The second 15-minute data-set is then used for the method validation. We note that we are interested here in understanding driving profiles in terms of the car-following behavior, without lane change or other behaviors. After a data processing of the first 15-minute part, we selected approximately 1500 car trajectories (drivers) from a total population of 1993 vehicle trajectories, to extract specific driver profiles. The selection was based on the following condition: each car must always have a leading car (downstream) whose trajectory is continuous at least over 40s. In addition, the same data processing was made for the second 15-minutes of data and resulted to the selection of 1300 car trajectories out of a total of 1495.

Proposition of driver profiles classification

In this work, the classification of different driver profiles begins with observing the vehicle time headway (THW) from the NGSIM data. Figure 3.2 shows the distribution of each car driver's maximum THW, mean THW, minimum THW, and standard deviation of THW, for the 1993 vehicle trajectories of the first 15 min NGSIM 101 data-set. Figure 3.2 provides evidence on the heterogeneity of human driving profiles as the mean THW ranges from near 0s to 5s and the minimum of THW ranges from near 0s to more than 3s. Based on this preliminary findings, we propose the definition of three profiles: (i) 'aggressive': shorter car time-headway, (ii) 'inattentive': longer reaction time, and (iii) 'normal' for intermediate values of reaction time and car time-headway. We notice that since the reaction time cannot be detected in the real driven data, we estimate and suppose that the reaction time is approximate to the minimum THW value, which is the minimum safe time gap that the driver estimate with the leading vehicle during his whole trajectory (longer than 400 time steps, explained in section 4. 3.4). The definitions of the the two driver profiles (aggressive and inattentive) are formalized below.

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

1. Aggressive driver profile: A driver i is considered to be aggressive with respect to a threshold t * on the time headway THW if

THW(i) := 1 T T t THW(i, t) < t * (3.1)
We will explain in section 3.4.3 how we fixed the threshold t * in our approach.

2. Inattentive driver profile (drivers with long reaction time): A driver i is considered to be inattentive (with a long reaction-time ) with respect to a threshold t on the time headway

THW if min THW(i) := min t T HW (i, t) > t (3.2)
We will explain in section 3.4.3 how we fixed the threshold t in our approach.

3. Normal driver profile : the drivers whose profiles are not aggressive or inattentive are called normal. They have intermediate values of reaction time and time headway.

Classification of driver profiles from the NGSIM 101 dataset

Specific driver profiles in the NGSIM data-set

In a traffic data-set with a large number of trajectories, it is always possible to distinguish 'good' drivers from 'risky' ones. Following the definition of the two 'extreme' driver profiles of section 3.4.2, we consider the following four driver profile groups in the NGSIM dataset: group 1: the 2.5% most aggressive drivers, group 2: the second 2.5% most aggressive drivers, i.e. situated between the 2.5 and 5-percentile of mean THWs, group 3: the 2.5% most inattentive drivers, and group 4: the second 2.5% most inattentive drivers (drivers ranked between 2.5% and 5% in terms of inattentiveness). The four groups are defined as follows: Let us denote the set I agg (t) of the drivers i whose T HW (i) is less than t:

I agg (t) := {i, T HW (i) < t}. (3.3)
Now, we denote by N the total number of drivers of the considered data-set, and by N agg (t) the cardinal (number of elements) of I agg (t):

N agg (t) := Card(I agg (t)). (3.4)
Finally, we denote by p agg (t) the proportion of the number of drivers in I agg (t) with respect to the total number of drivers (N ):

p agg (t) := N agg (t)/N. (3.5)
Let us now define t * 1 and t * 2 as follows:

t * 1 = p -1 agg (0.025) := max{t, p agg (t) ≤ 0.025}, (3.6
)

t * 2 = p -1 agg (0.05) := max{t, p agg (t) ≤ 0.05}. (3.7)
Groups 1 and 2 are then defined as follows:

Group 1 = I agg (t * 1 ) = I agg p -1 agg (0.025) , (3.8) Group 2 = I agg (t * 2 ) \ I agg (t * 1 ) = I agg p -1 agg (0.05) \ I agg p -1 agg (0.025) . (3.9)
The groups 3 and 4 are defined similarly, that is:

I inatt (t) := {i, min T HW (i) > t}, ( 3
Group 3 = I inatt (t * 3 ) = I inatt p -1 inatt (0.025) , (3.15) Group 4 = I inatt (t * 4 ) \ I inatt (t * 3 ) = I inatt p -1 inatt (0.05) \ I inatt p -1 inatt (0.025) . (3.16)
Applying the definitions of groups 1, 2, 3, and 4 above, we obtain the thresholds of Table 3.1 on T HW for groups 1 and 2, and on min T HW for groups 3 and 4; see also Figure 3.3. In Figure 3.4a and Figure 3.4b, we give the distribution of the average THW and minimum THW values respectively for the selected aggressive drivers and inattentive drivers. We can see that the value of minimun THW for two groups of aggressive drivers' ranges from about 0.1s to 0.8s. On the other side, the average THW of inattentive drivers is far larger than that of aggressive drivers, which ranges from about 2s to 5s.

SUMO traffic simulator and the choice of car-following model

Once we extracted the different driver profiles from NGSIM 101 data-set, we need a traffic simulator to simulate the NGSIM traffic and the microscopic traffic model for representing the driver profiles. We choose SUMO ("Simulation of Urban MObility") as the traffic numerical simulator to apply our proposition, which have beed already presented in the Section 2.6.1. As we are interested only in car-following behavior in this work, the generation of different driver profiles is performed by calibrating a car-following model. Many interesting reviews on carfollowing can be found in the literature [START_REF] Brackstone | Car-following: a historical review[END_REF]Mohammad and Z. Zheng, 2014;[START_REF] Aghabayk | A state-of-the-art review of carfollowing models with particular considerations of heavy vehicles[END_REF]. We briefly present here some of the most commonly used. GHR model [START_REF] Chandler | Traffic dynamics: studies in car following[END_REF]) is a stimulus-response model in which drivers perform their acceleration/deceleration depending on car-speeds, relative car-speeds with respect to leading vehicles, inter-vehicle distances, and drivers' reaction time. [START_REF] Wiedemann | Simulation des Strassenverkehrsflusses[END_REF] car-following model is used in Vissim simulator and describes the psychophysiological aspects of driving behavior in terms of four discrete driving regimes. This model considers different modes of operation, divided into 'no reaction zone' (free-road), 'closing in', 'must decelerate', and 'car-following' by human perceptual thresholds (Mohammad and Z. Zheng, 2014). Gipps model [START_REF] Gipps | A behavioural car-following model for computer simulation[END_REF] and Krauss model [START_REF] Krauß | Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics[END_REF] are based on safety distance. In the Gipps model, drivers update their car-speeds with respect to keep a minimum safety distance to the leader ones, in order to avoid collisions and while considering the extreme case of a leader car braking suddenly. Krauss model [START_REF] Krauß | Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics[END_REF] is the carfollowing model by default in SUMO simulator. It extends the Gipps model by modeling the imperfection of human driver with stochastic terms of car-speed. This propriety makes (or is supposed to make) the model more realistic. The IDM car-following model was first published in [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF] which improved the initial results produced by simulation approach the Gipps' model. The acceleration is calculated as a function of the desired speed and the desired space headway. The IDM model is suitable for both free flow and congestion phases (Mohammad and Z. Zheng, 2014). This attribute makes the model more efficient and facilitates its calibration with real driven data.

We chose here the IDM car-following model [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF], which is already implemented in SUMO, and it is shown by several research works that IDM is the most human-like car-following among certain compared models [START_REF] Vasconcelos | Calibration of the Gipps car-following model using trajectory data[END_REF]M. Zhu, Xuesong Wang, Tarko, et al., 2018). Let us briefly recall the IDM model. The model is described by the following formulas :

dV dt = a 1 - V V 0 δ - s * (V, ∆V ) s 2 , ( 3.17)
where The output of the IDM model is the acceleration dV /dt given as a function of the influencing factors (the inputs of the model): the own speed V, the inter-vehicular distance s and the relative speed ∆V with respect to the leading vehicle.

s * (V, ∆V ) := s 0 + max 0, V T + V ∆V 2 √ ab , ( 3

Car-following model calibration method

Reaction time in car-following models

We notice that the original IDM car-following model does not include reaction time as a parameter [START_REF] Aghabayk | A state-of-the-art review of carfollowing models with particular considerations of heavy vehicles[END_REF]. Nevertheless, in a recent development of SUMO simulator [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF], the authors improved the work on reaction time modeling. They indicated that reaction time could be introduced in the driving model (car-following and lane change models) as an additional parameter. In the work we present here, we introduced a reaction time parameter into the IDM car-following model. In SUMO, the simulation step can be set to ∆t, and the driver reaction time is a time delay to make a decision both for updating acceleration or lane changing depending on the present state of traffic. This delay of decision time (reaction time) can be set as (n * ∆t), where n is an additional parameter which we determine by calibration.

Car-following model calibration method

To achieve a realistic representation of traffic behavior, the calibration with real driven data for the traffic model has to be applied. Two main types of calibration of car-following models exist: 1) Estimation of driving model parameters accounting for the physical meanings of each parameter [START_REF] Lu | A video-based approach to calibrating car-following parameters in VISSIM for urban traffic[END_REF], in which, parameters can be extracted directly from vehicle trajectory data; 2) Calibration of driving models, which can be constructed as an optimization problem.

In the second approach, the objective function and the optimization method is selected so as to minimize the distance between the simulated vehicle trajectories by the model and real vehicle trajectories, and in order to find the optimal set of parameters. Several mathematical optimization methods and algorithms such as Newton, Gauss-Newton, gradient descent, and Levenberg-Marquardt are presented for car-following model parameters optimization in (Treiber and Kesting, 2013a). The authors of (Treiber and Kesting, 2013a) proved that genetic algorithms (GA) are also effective to solve optimization problems for car-following model calibration. Many of the recent works on car-following model calibration used GA methods [START_REF] Vasconcelos | Calibration of the Gipps car-following model using trajectory data[END_REF]M. Zhu, Xuesong Wang, Tarko, et al., 2018). Their works show that GA optimisation method is efficient for car-following model calibration. In Table 3.2, an overview is given regarding to the several recent works on car-following models calibration by optimisation methods.

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

Articles

Pre-definitions of IDM model calibration

In our approach, the identification of drivers with 'extreme' profiles is carried out by calibrating the IDM model parameters, using real driven data. We use here the same method used in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF], where a genetic algorithm (GA) is applied and a measure of error is defined as the objective function. The used GA method is the default model in Matlab 2020b (R2020b, 2020) and for the algorithm description, it is presented in the book [START_REF] Singiresu | Engineering Optimization: Theory and Practice[END_REF]. As what is presented, GA is inspired by biologic evolution carried out by mechanisms of inheritance, selection, mutation and recombination to find the optimal solution among all generations (sets of parameter groups).

In our work, the GA is implemented to find the optimal values of the IDM model parameters for each driver trajectory with the following procedure (same as what is presented in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF]):

1. A population of N individuals is initialized depending on the predefined parameter boundaries, each representing a set of parameters of the IDM model.

2. Computation of the fitness of each individual in the population, and the faintness is determined by a predetermined objective function ( RMSPE on the vehicle position).

Car-following model calibration method

3. Crossover between randomly selected pairs of individuals (parents) and mutation within randomly selected individuals to produce the next generation (children)

4. Repeat steps 2 and 3 until the termination criteria are met.

The goodness of each individual (a parameter group) is evaluated by the objective function. In general, the mean square error (MSE) is widely used for calibration of car-following models. Here, another error metric is used as presented in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF], the root mean square percentage errors (RMSPE).

RM SP E(θ) =

1 ) denotes the car trajectory (positions in time) from data (respectively from simulation), and where θ represents all the parameters of the considered model. The car longitudinal position is chosen as the target variable in the objective function.

P P i=0 (x simul i (θ) -x data i (θ)) 2 x data i (θ) 2 , ( 3
The IDM model calculates car acceleration, as described in equation (3.20), based on the drivers reaction time (n * ∆t). Car's speed and position are calculated as follows in equation ( 3.21) and (3.22), based on the Euler Method.

a simul (t + n∆t) = IDM (θ; variables(t)) (3.20) v simul (t + ∆t) = v simul (t) + a simul (t) * ∆t (3.21)
x simul (t + ∆t) = x simul (t) + v simul (t) * ∆t (3.22) In this work, the parameters which need to be calibrated are: a (maximal acceleration), b (maximal deceleration), V 0 (desired speed), s 0 (minimal inter-vehicular distance to leading vehicle) and T (desired THW), as well as the additional parameter n for the reaction time (n * ∆t). In addition, the parameter δ (exponential parameter of speed) is fixed at δ = 4 as proposed in [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF].

In the calibration process of the IDM model with the reaction time parameter for the extreme driver profiles, upper and lower bounds are needed to be set for all the parameters of IDM. For the reaction time and desired THW parameters, the bounds for these two parameters are different for the aggressive and inattentive driver profiles. For aggressive drivers, we chose the desired THW ranges from 0.1 to 4 s, while the reaction time is from 0.1 to 2 s. For inattentive drivers, we chose the desired THW ranges from 1 to 4 s while the reaction time goes from tr i -0.3 to tr i + 0.3 seconds, where tr i is the value of min THW(i); see section 3.4.2 and we approximate the driver's reaction time to be around tr i . For the other parameters, the bounds are the same for both aggressive and inattentive parameters. Thus, the desired speed (V 0 ) ranges from 10 to simulation approach 40 m/s, s 0 ranges from 0.1 to 10 m, maximum acceleration and maximum deceleration range from 0.1 and 5 m/s 2 as what is proposed in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF].

For drivers with 'normal' profile, we simulate them using an average profile. Their calibration of IDM parameters is extracted directly from the data-set by their physical meanings and combined with the default values in SUMO simulator.

IDM model calibration result for extreme driver profiles

As mentioned above, the calibration of extreme driver profiles is considered as the optimization problem with the aim at minimizing the error between the simulated car-trajectories by IDM model and the real ones available from the data-set using a genetic algorithm. Moreover, we repeated the genetic algorithm 10 times for every driver trajectory, in order to approach the global optimum solution for the parameters of the calibration. Firstly, we observed that the IDM model with the reaction time parameter is efficient for replicating the human driver behavior; see. A.1. The result by IDM model with the extension of reaction time is better than the original IDM model without reaction time parameter. Therefore, we use the IDM model with the reaction time parameter to present the driver profiles in numerical simulated traffic .

The results of calibration for the 4 specific groups of drivers are given in Tables A.1 -A.4 of Appendix A.2. For the calibration on the car-trajectories variable (car-positions in time), we obtained for all 4 driver profile groups an RMPSE ranging in (1% ,2%). From the calibration results, given in Tables A.1 -A.4, we notice that group 1 driver profiles have shorter reaction times and desired headway times compared to group 2 driver profiles. On the other hand, group 3 driver profiles have longer reaction times and desired headway times compared to group 4 driver profiles. Moreover, the desired THW (parameter T in formula (2.14)) of attentive drivers is much larger than the one of aggressive drivers.

Numerical simulation experiments setup

The reproduction of the NGSIM traffic data by numerical simulation consists of the creation of the road section, the selection of the microscopic traffic model (IDM car-following model, plus the default lane changing model in SUMO), and the simulation using SUMO. NETEDIT (SUMO, 2019a) tool of SUMO simulator allows to create manual map network of NGSIM, presented as a road of 640 meters with 5 straight lanes. The creation of each vehicle is provided by its longitudinal origin position, longitudinal destination position, entering lane, and entering time (in seconds). This necessary information is extracted from the car trajectory data. In

Numerical simulation experiments setup

addition, during the simulation, SUMO can detect physical collisions (front and back bumper meet or overlap) (SUMO, 2019b). We can get the collision counts at the end of each simulation.

Four experiments of car-collision simulation are proposed for different combinations of the two selected groups of drivers. Each experiment is carried out using different combinations of one group of aggressive drivers and one group of inattentive drivers. Driver profiles used in simulation are described in Table 3 As what we proposed in above, for example, for the experiment E1 (Table 3.3) which is the first experiment, we use 2.5% of first aggressive driver profiles, 2.5% of first inattentive driver profiles, and 95% of "normal drivers" as the initial scenario of the traffic. This corresponds to the original traffic condition of the first 15-minute sub-dataset in NGSIM 101 dataset. We simulate the traffic in SUMO for this first scenario, where all the driver profiles use their own calibrated parameters of the IDM model. The results are shown in Figure 3.5, where we give the mean traffic speed from the real data and the mean traffic speed presented from numerical simulations in SUMO. We can see that the traffic simulation represents well the state of traffic (car-speed) over time, compared to the real data.

We assume then the car-collisions can be generated in each simulation by increasing the number of aggressive and/or inattentive drivers in the traffic. The increasing of the percentages of extreme driver profiles is done artificially and randomly by replacing the same number of 'normal' drivers by the selected extreme driver profiles. Thus, we reset the simulation by increasing the rates of drivers associated to each extreme driver profile (2.5% in the origin data-set, 2.5%, 5%, 10%, 15%, 20%, 25%, 30%, 35%, 40%, 45% and 50% in simulation). For each simulation, once the rates of the extreme driver profiles are fixed, all the other drivers are taken as 'normal'. simulation approach Figure 3.5 -The average traffic flow speed from real data (blue) and the simulated average car speed (orange) in SUMO of the initial traffic of the first 15 minutes sample, where 2.5% are aggressive drivers simulated using their own IDM calibrated parameters, and 2.5% are inattentive drivers simulated using their own IDM calibrated parameters. The remaining driver profiles (95%) are normal drivers in the traffic.

The results for vehicular collision generation

In this section, we give the results of the numerical simulations for the car collision generation. In subsection 3.8.1, we provide the number of collisions generated by numerical simulation based on the first 15 minutes data. The results are presented for the 4 predefined scenarios. Moreover, in order to understand the relationship between the driver profiles and the generated car-collisions, we give in subsection 3.8.2 a deeper analysis of the collisions obtained in the numerical simulation experiment 1 (E1). In subsection 3.8.3, we show that the number of simulated car-collisions can be written as a function of the rates of aggressive and inattentive driver profiles in the traffic.

After the presentation of the obtained results on the first 15-minutes sub-dataset, we give in section 3.9 the results for the validation of the proposed approach using the second 15-minute time period of the NGSIM 101 dataset.

The results for vehicular collision generation

Number of collisions by numerical simulation

We start with the implementation of experiment E1. We observed that the number of collisions occurred in each simulation is various even when using the same rate of specific drivers. This is due to the random distribution of the specific (aggressive and inattentive) driver profiles over all the drivers. Nevertheless, we observed that the mean number of car-collisions converges in simulation time, for each case. Thus, the presented collision counts in each case shown in the following tables are the average value over 10 simulations for each case. The number of collisions obtained by numerical simulation is given in Tables 3.4-3 In Table 3.4, the simulated collision counts are shown along with different rates of specific profiles for E1. The results indicate clearly that the number of inter-vehicular collision occurrences increases with the considered number of aggressive and/or inattentive driver profiles. The results for E2, E3, and E4 are given respectively in Tables 3.5-3.7. The number of obtained collisions is lower compared to the ones of Table 3.4. The number of collisions obtained with E4 is the lowest one compared to the other three experiments. This result shows that a different risk level is associated to each profile. The first group of aggressive driver profiles could potentially cause more collisions than the second group of aggressive driver profiles (comparison of E1 and E2 results). Similarly, the first group of inattentive driver profiles could cause more collisions than the second group of inattentive driver profiles (comparison of E3 and E4 results). Interestingly, in the four experiments, no collisions occurred when the percentage of 'extreme' drivers was small. With the increase of the rate of each specific driver profile, the number of collisions increases. However, this increase seems to be different from one experiment to an other; see Figure 3.6, where each line presents the number of collisions in the simulation of each four experiments, under the condition that aggressive drivers and inattentive drivers have

The results for vehicular collision generation

the same rate in the traffic. The Experiment 1, where we generated collisions based on varying the percentage of the most aggressive and the most inattentive drivers, can generate more collisions than the other three experiments. The Experiment 4 has probably the lest number of collisions, where we generated collisions based on varying the percentage of the second aggressive and the second inattentive drivers.

Analysis of collisions obtained in Experiment 1 (E1)

We further analyze here the results obtained in E1 (Table 3.4) to investigate the relationship between generated car collisions and driver profiles. Figure 3.7 depicts the number of the simulated collisions as a function of the two rates of aggressive and inattentive driver profiles. More precisely, Figure 3.7 shows the contours of this function, which takes indeed the values of the result shown in Table 3.4. From Figure 3.7, it seems that the sum of the two rates of aggressive and inattentive driver profiles is important in determining the number of collisions. Indeed, we can see on Figure 3.7 that the contours resemble to decreasing straight lines with slope (-1). However, the contours are not linear, which means that the sum of the rates of aggressive and inattentive driver profiles, although important, is not the unique parameter for the determination of the number of collisions. This will be confirmed in the next section, simulation approach where, using statistical regression, we approximate the number of collisions with a non linear function of the two rates of aggressive and inattentive driver profiles. As mentioned above in section 3.7, a car-collision is observed in SUMO when a following vehicle (veh2) collides the rear-end of the proceeding vehicle (veh1). We can get information on the profiles of the two vehicles involved in each collision. The distribution of rear-end collisions produced by different driver profiles (aggressive, normal, and inattentive) in percentage is shown in Figure 3.8. Abbreviations ag, long, and nor are used to indicate aggressive, inattentive, and normal driver profiles respectively. In Figure 3.8, we give the percentages of the number of collisions caused by each pair of driver profiles rates. Thus, collisions caused by the pair inattentive(veh1)-aggressive(veh2) of driver profiles, in this order, account for the largest ratio, which is 28.99 % of the whole number of simulated collisions.

The results for vehicular collision generation

The number of collisions as a function of the rates of aggressive and inattentive driver profiles in traffic

According to the results given in Table 3.4, the relationship between the number of simulated collisions and the rates of aggressive and inattentive driver profiles is shown in Figure 3.7.

In order to better understand this relationship, we use statistical regression and assume that car-collision counts (Y) are proportional to the percentage of aggressive drivers (x 1 %) and the percentage of inattentive drivers (x 2 %), which can be formulated as Y = f (x 1 , x 2 ).

We have first tried linear regression and the result is shown in section A.3.1 of Appendix A.3, which concludes that the assumption of linearity in not convincing. Indeed, we observe from Table 3.4 (red cases), for example, that the number of collisions generated with 30 % of aggressive driver profiles and 30 % of inattentive driver profiles is 60. Holding on this 60%, however, with 25 % (also respectively 20 %, 15 %, 10 %, 35 %, 40 %, 45 %, and 50 %) of aggressive driver profiles, and 35 % (also respectively 40 %, 45 %, 50 %, 25 %, 20 %, 15 %, and 10 %) of inattentive driver profiles, the number of generated collisions is less substantial, although the sum of all the pairs of considered rates of aggressive and inattentive driver profiles is 60 %. Moreover, it seems that the number of generated collisions decreases with the increasing of the absolute difference between the rates of aggressive and inattentive driver profiles. In other terms, the number of generated collisions increases with the uniformity of mixing both profiles simulation approach (aggressive and inattentive). Based on this observation, we propose to add a quadratic term of the approximation of the relationship between the number of generated collisions and the rates of aggressive and inattentive driver profiles.

Ŷ = f (x 1 , x 2 ) = max 0, β 0 + β 1 * x 1 + β 2 * x 2 + β 3 * (x 1 -x 2 ) 2 , (3.23)
where Ŷ is the estimation of car-collision counts; x 1 % is the percentage of aggressive drivers and x 2 % is the percentage of inattentive drivers; We lower bound the function by zero, since the number of collisions cannot be negative. We obtained the following coefficients: We notice here that β 3 < 0. First, the fact that β 3 ̸ = 0 confirms the significance of the effect of the absolute difference between the two rates of aggressive and inattentive driver profiles on the number of generated collisions. Second, the fact that β 3 is negative confirms our hypotheses, that the number of generated collisions decreases with the increase of the absolute difference between the rates of aggressive and inattentive driver profiles.

β 0 = -23.35, β 1 = 1.
In Figure 3.9, the number of collisions for each percentage of the two drivers profiles is presented, as well as the surface obtained by the non-linear regression. simulation approach

Validation of the approach on the second 15-minute time period of data

In this section, the same approach (summarized in Figure 3.1) for car-collision generation is carried out based on the second 15-min time period of the NGSIM 101 data-set, with the purpose of validation of the approach, where 1495 vehicles are registered in this period on the 5 normal traffic lanes. Similarly to the work using the first 15-minute data, we begin by selecting the 4 driver profiles groups by the percentage of 2.5%. As the result, the thresholds on T HW (mean THW) and min T HW for each driver profile are shown in Table 3.8. The thresholds obtained for the 4 groups of drivers are similar to the ones obtained based on the first 15-minute period of data (comparison of Table 3.8 and Table 3.1). In addition, the simulation of the original traffic in SUMO for this period is shown in Figure 3.10. This Figure shows that the traffic simulation is well calibrated for the second period of data, with initial percentage of aggressive drivers and inattentive drivers (2.5% for each group) in the traffic, and the limit speed conditions on the two sides of the road.

Type of drivers Indicator Threshold 1st aggressive (group 1) mean THW ⩽ 0.86s 2nd aggressive (group 2) mean THW [0.86s, 0.98s] 1st inattentive (group 3) min THW ⩾ 1.99s 2nd inattentive (group 4) min THW [1.74s, 1.99s] Table 3.8 -Selected drivers and the associated threshold for the second 15-minute data. 3.9 Validation of the approach on the second 15-minute time period of data

Number of collisions by numerical simulation based on the second 15-minute data

The simulated resulting collisions, for the 4 experiments (E1-E4), are shown in Tables 3.9-3.12. Furthermore, Figure 3.11 shows the number of generated collisions for the four experiments. The result using the second 15-minute period data is close to the result using the first 15minute period. The simulated collisions using the second 15-minute data show the same trend as the first 15-minute data, where Experiment E1 simulates the greatest number of collisions, while E4 simulates the lowest number of collisions. simulation approach 1st aggressive drivers (horizontal) 2nd inattentive drivers (vertical) 2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag

The number of collisions and the rates of different driver profiles in traffic based on the second 15-minute data

The results of the number of collisions for experiment E1 based on the second 15-min data are given in Table 3.9 and Figure 3.12. Therefore, as for the first 15-min time period data, a non-linear regression is performed (Equation 3.23), where we get the following coefficients: β 0 = 0, -8.154, β 1 = 1.580, β 2 = 0.8502, and β 3 = -0.7575, with an RMSE = 3.761.

In Figure 3.13, the number of collisions for each percentage of two drivers profiles is presented, as well as the surface obtained by the non-linear regression by the proposition (Equation 3.23).

Analysis of car-collision severity

Analysis of car-collision severity

All the simulated car-collisions can be further used to explore car-collision severity. Several works have investigated the relationship of rear-end car-collision severity to car speed. In [START_REF] Jurewicz | Exploration of vehicle impact speed-injury severity relationships for application in safer road design[END_REF], the authors reported that the critical impact speed was approximately 55 km/h for rear-end collisions. In addition, in [START_REF] Elvik | A re-parameterisation of the Power Model of the relationship between the speed of traffic and the number of accidents and accident victims[END_REF], the authors claim that the change of speed before and after the car-collision is a critical indicator for car-collision severity outcomes.

In physics, the kinetic energy (KE) of an object is the energy that it possesses due to its speed (3.24). In the case of front-rear collision of two moving cars, the kinetic energy is related to the relative speed of the front vehicle and the rear car (3.25).

KE = 1 2 m * v 2 (3.24) KE ∝ ∆v (3.25)
Relative speed could thus be an important surrogate to indicate the severity of collision. In simulations with 50% drivers being aggressive and the other 50% being inattentive, 866 collisions were generated (with 5 simulations). Among all collisions, 220 (25.4%) concerned aggressive-aggressive couples of driver profiles, 408 (47.1%) concerned inattentive-aggressive, 107 (12.4%) concerned inattentive-inattentive, and 131 (15.1%) concerned aggressive-inattentive. In Figure 3.14, the distribution of relative speed for these 866 simulated collisions is shown. collisions implicating inattentive(veh1)-aggressive(veh2) drivers account for the largest rate of all simulated collisions. All generated collisions have a relative speed below 50 km/h. Furthermore, collisions are more severe when they implicate two inattentive drivers, since the mean relative speed is at 23.27 km/h, which is more critical than in the other cases. Car-collision severity is much lighter between two aggressive drivers who have an average relative speed at 15.7 km/h. SCANeR Studio (AVSimulation, 2019) [START_REF] Champion | Traffic generation with the SCANeR© II simulator: towards a multi-agent architecture[END_REF] is a driving simulation software, which also includes Bullet physic engine (a free and open-source software of simulation of collision detection, soft and rigid body dynamics). It provides the opportunity to simulate vehicle collisions with a vehicle physical model. For every simulated car-collision in SUMO, the involved two vehicles' trajectory can be registered. Then the car-collision scene can be replayed in SCANeR Studio. Thus, the result of a car-collision simulated in SUMO can be re-simulated in SCANeR Studio; see Figure 3.15. In Figure 3.16, the collision force for certain replayed collisions is shown. Each point corresponds to one re-simulated car-collision by SCANeR. It seems that the force is a super-linear function of the relative speed; see Figure 3.16. However, this fact needs more investigations and should be considered in our future research. The objective will for each sub-figure), Abbreviations "ag" and "inatt" denote "aggressive" and "inattentive" driver profile respectively. It shows that the most collisions are occurred between inattentive-aggressive drivers, while the most severe (dangerous) collisions are produced between inattentive-inattentive drivers.

be to couple SUMO traffic simulator with SCANeR studio (the immersive driving simulator) in order to build a high performance system to test and validate autonomous vehicles.

Conclusion and perspectives

In this work, the main contributions consist in 1) extracting three proposed driver profiles based on real driven data-set; 2) replicating them in simulated environment; and 3) establishing a relationship between car-collision occurrences and these different driver profiles by varying their percentages in the whole traffic via virtual simulation.

Based on the NGSIM 101 data-set, two specific driver profiles which are related to carcollisions on road networks have been characterized, defined as i) aggressive drivers who keep short time-headways with their leaders, and ii) inattentive drivers with long reaction-times. All the other driver profiles are considered as "normal" with intermediate values of reaction time and time-headway. These three driver profiles have been simulated by using the Intelligent Driver Model (IDM) car-following model, with an extension including driver reaction time.

In order to represent the real driver profiles, the IDM model is calibrated using a genetic algorithm. Finally, by increasing the percentage of the two extreme driver profiles (aggressive and inattentive) over the whole traffic in virtual traffic simulation, we investigated the effect of these driver profiles on the car-collision occurrences. The results of the numerical simulations show that the percentages of the aggressive and inattentive driver profiles over the whole driver population are determinant in the car-collision occurrences and in the resulting severity outcomes. One of the important results we obtained in this work is the characterization of the relationship between the ratios of these two driver profiles over the whole drivers population, and the car-collision occurrences counts. We have also classified the occurred car-collisions and analyzed their severity, in particular with respect

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

to the relative speed between the cars which cause the collisions. Another important result of this research is that, the generated car-collisions involved between an aggressive driver as a leader and an inattentive driver as a follower occupy the most frequent collision occurrences, meanwhile the generated car-collisions involved between two inattentive drivers are the most severe ones.

The safety validation of Intelligent Connected Vehicles is essential and could be critical for their deployment. In order to complete the demonstration of the reliability of the system's safety, autonomous vehicles need to be driven for hundreds of millions of miles. However, the huge cost of such physical tests, combining with its inherent danger of testing situations where collisions can happen, makes the numerical simulation of scenarios mixing different driver profiles as an important safety assessment tool for ICVs testing. Indeed, during the deployment phase of ICVs, the recognition of drivers' profiles should be considered to avoid collision generation. As shown in our experiments, based on our approach, different traffic scenarios can be generated with different driver profiles in traffic simulation. For future research, this work is expected to highly facilitate future ICVs testing and validation for the car manufacturing industry via numerical traffic simulation.

Chapter 4

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

Microscopic modeling of human driving consists generally in combining both car-following and lane-change models. While the human car-following process has been extensively developed and well modeled, the lane-change behavior is more complex to understand and still remains to be explored. Classical lane-change models are usually rule-based and handcrafted, that tend to exhibit limited performance. Machine Learning algorithms, particularly Reinforcement Learning (RL) ones, provide an alternative approach and have recently achieved high success in modeling difficult decision-making processes in many fields. We propose in this chapter a reinforcement learning based model for the human lane-change behavior, with an online calibration of real lane-change decisions, extracted from the NGSIM data-set. We use the traffic vehicular simulator SUMO ("Simulation of Urban MObility") to create a numerical simulation environment. The utilization of numerical traffic simulation allows us enriching the data-set, for training the agent to find an optimal policy for lane change. Thus, about 13% additional traffic situations, not present in the real data, are created by the traffic simulation environment. The trained agent is collision-free and human-like. It is able to reproduce well lane-change behaviors in real data and those in the additional simulated data. Indeed, our RL model can perform up to 95.37% of the real decisions observed in the data-set.

Motivation

Driving behaviour models describe the drivers manoeuvring decision in different traffic conditions, which are important for traffic researches and for microscopic traffic simulation. Carfollowing (CF) and lane-change (LC) models are the two fundamental models in human driving modelling. CF models have been well developed. They describe the longitudinal behavior when a given vehicle follows the leading one. Most of literature reviews focus on summarizing car-following models (Mohammad and Z. Zheng, 2014;[START_REF] Aghabayk | A state-of-the-art review of carfollowing models with particular considerations of heavy vehicles[END_REF]. Considering LC behavior, the modeling efforts are not as many as in the CF behavior. This is due to the complexity of the LC behavior, which is affected by the surrounding vehicles (of the considered vehicle) and by the traffic flow environment (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007). In some reviews of lane changing models (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007;[START_REF] Moridpour | Lane changing models: a critical review[END_REF]Z. Zheng, 2014), the authors present several rule-based lane changing models, but they tend to exhibit limited performance due to the uncertainty and to the complexity of the driving environment. For this reason, Machine Learning methods, and especially Reinforcement Learning (RL) ones, provide an alternative approach, which has shown a great success in many different domains, such as robotics, video game playing, dialogue chat bot, etc.

Reinforcement Learning (RL) is a general framework for learning-based sequential decision making. With reinforcement learning, an agent can learn to solve problems where the decision process is facing previously unknown situations through trial and error. Therefore, using reinforcement learning combined with simulated traffic environment, vehicles can be trained to develop a lane changing strategy, trying an action at a situation, and observing the result of this action. Being rewarded for good actions and penalised for bad ones, they learn how to act optimally through the experience, which is quite useful considering all the different situations that might exist, and how difficult it would be to program the vehicle's actions in all these situations.

Many researchers have shown the great efficiency of using RL for car-driving modeling. In (L. Wang et al., 2019), the authors developed an ego-efficient lane-change strategy using Q-learning and also tested the effects of the strategy on the network level through Aimsum traffic simulation platform. With the aim to study both longitudinal and lateral movements, the authors of [START_REF] Hoel | Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning[END_REF] proposed a model for the automation of both speed and lane-change decisions based on Deep Q Learning. Another approach was followed by (P. [START_REF] Wang | Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm[END_REF] who used an actor-critic algorithm for the automation of lane-change decisions, and the problem has been solved using the Deep Deterministic Policy Gradient. In [START_REF] Ye | Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning[END_REF], an actor-critic algorithm is also used to propose a method for automating lane-change decisions. The authors of [START_REF] Ye | Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning[END_REF] proposed a model for mandatory lane-changes, developed using the Proximal Policy Optimization algorithm. All

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

this relative works are using only simulation environments and propose intelligent lane-change models, but the calibration with real human driving behavior has not been mentioned in these relevant works. However, real human driving behavior needs to be considered to make the model more realistic. Even for autonomous vehicle, a human-like driving model could be more comfortable for the passengers.

Background of Imitation Human Driving Behavior

Imitation learning, also called behavior cloning or learning from demonstration, is a form of supervised learning that can learn a policy from off-line collected data D(S * , A * ), where S * indicates the real state of real world and A * is the corresponding action [START_REF] Schaal | Computational approaches to motor learning by imitation[END_REF][START_REF] Ross | A reduction of imitation learning and structured prediction to no-regret online learning[END_REF]. This method is recently used for car-driving modeling. In [START_REF] Codevilla | End-to-end driving via conditional imitation learning[END_REF], the authors presented an end-to-end vision-based driving model by supervised learning using deep neural networks, which is a model to predict the driving actions from camera input. However, using only real data is limited due to the lack of situation in the collected data-set. Indeed, RL methods allow the agent to learn on-line by taking mistakes through the reinforcement signal from the external environment. The agent then tries to reinforce its action to improve future performance. Therefore, a signal can be formulated to guide the agent to learn from real data. As presented in (Y. [START_REF] Zhang | Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[END_REF]M. Zhu, Xuesong Wang, and Y. Wang, 2018), the authors used real data to learn a policy of car-following through RL. They associate a positive reward when the action is the same as in the data-set, for a state present in the collected data. Otherwise, the action is penalised with a negative reward.

Deriving from this idea and aiming at learning from both the simulation environment and the real lane changing decisions, we propose in this chapter an approach for lane changing modeling using the Q-learning algorithm [START_REF] Watkins | Q-learning[END_REF] with an on-line imitation of the human lane-change decision from the NGSIM 101 data-set. In this work, our major contribution is the use of Q-learning algorithm, and the calibration of the lane changing decisions extracted from the NGSIM 101 data-set. In addition, we use the traffic vehicular simulator SUMO to create a numerical simulation environment, that allows us enriching the data-set for training the agent to find an optimal policy for lane change. The results show that the policy is sufficiently trained and closed to the real human behavior. In order to understand the learned policy, we use a color map visualisation to show the distribution of lane changes as functions of several couples of states.

A model for imitation of the real lane change behavior from the NGSIM dataset A model for imitation of the real lane change behavior from the NGSIM dataset

As mentioned above, we propose in this study a lane changing model based on a Q-learning algorithm with an on-line calibration with human lane changing decision. With the purpose of obtaining a human-like and collision-free lane changing strategy, we apply this study based on the NGSIM 101 data-set. We use the vehicular simulator SUMO to build a traffic dynamic environment. In this section, we first introduce in sub-section 4.3.1, the architecture of our method. We then give the definitions of the state and the action variables for the lane changing model, in sub-section 4.3.3. We describe the NGSIM data-set as well as the preparation of the lane changing decision data in sub-section 4.3.4. In sub-section 4.3.5, we give the discretization of the state variables. Finally, we give in sub-section 4.3.6, the definition we use here for the reward, with the concept of learning the real lane changing decision.

Method architecture

We Illustrate in Fig. 4.1 the architecture of our approach of lane changing modeling using Q-learning and online calibration with real data. We use the vehicular traffic simulator SUMO ("Simulation of Urban MObility") [START_REF] Lopez | Microscopic traffic simulation using sumo[END_REF] to create a simulation environment. The controlled vehicle (the agent) exchanges with the simulated traffic environment and is trained to develop a lane changing strategy through evolving the policy by the return from the environment. In addition, real lane changing decisions extracted from the NGSIM data-set are used for online calibration of the target vehicle lane changing actions. As presented in Fig. 4.1, once a current policy π k applies an action a

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

at a given state s, SUMO combining real data D(S * , A * ) returns a new state s ′ and a reward R. Then, with the data (s, a, s ′ , R), the policy can be updated by the Q-learning algorithm to get a new policy (π k+1 ). Then by iterating, the optimal policy can be found by trial-and-error with the environment. Referring to (Y. [START_REF] Zhang | Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[END_REF], where mimicking the real behavior has been studied, at a given state, which is present in the collected data, the reward could be positive when the action is the same as in the data; otherwise, the action needs to be penalised by a negative reward.

Q-learning

Q-learning [START_REF] Watkins | Q-learning[END_REF]) is an algorithm of RL, which is model-free and with the purpose to optimize the action-value function defined as Q(s, a). This Q-value function expresses the expected return of taking action a following a policy π and being at state s:

Q π (s t , a t ) = E π [G t |s 0 = s, a 0 = a] = E π ∞ k=0 γ k R t+k |s 0 = s, a 0 = a , ( 4.1)
where G t , s 0 , a 0 , k, γ k and R t denote respectively ....

According to the Bellman principle, the optimal action value function Q * (s, a) is the maximum action-value function over all policies, which can be formulated as :

Q * (s, a) = E R + γ max a ′ Q * (s ′ , a ′ ) = R t + γ E max a ′ Q * (s ′ , a ′ ) . (4.2)
In theory, the value of a state is relative to the values of its successor states. The value Q(s, a) of the current state must equal the discounted Q(s ′ , a ′ ) of the expected next state, plus the immediate reward. Therefore, for the optimal Q * (a, s), the optimal policy is to select an action a ′ , that maximizes the expected value of Q * (s ′ , a ′ ). At the end, the optimal policy is attained by selecting at each state the action with the largest optimal Q-value : choose a from s using policy derived from Q (e.g., ε-greedy or decaying-ε-greedy policy)

π * (s) = arg max a Q * (s, a) (4.
6 take action a, observe R, s ′ 7 Q(s, a) ← Q(s, a) + α[R + γ max a ′ Q(s ′ , a ′ ) -Q(s, a)] 8 S ← S ′

end

The Q-learning Algorithm 4.1 is an off-policy RL algorithm, in the sense that the learned policy can be different from the policy used to choose an action. The algorithm uses the Temporal Difference (TD) method, which learns by bootstrapping from current estimate, where R + γ max a ′ Q(s ′ , a ′ ) named TD target, is a target function for Q(s, a) in the current estimate. For the reinforcement learning, once we define the task that the agent needs to accomplish, the definition of the state, action and reward variables are important. The authors of (L. Wang et al., 2019;[START_REF] Hoel | Automated Speed and Lane Change Decision Making using Deep Reinforcement Learning[END_REF][START_REF] Wang | Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm[END_REF][START_REF] Ye | Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning[END_REF] have used different ways to represent the traffic including that of the target controlled vehicle and of its surrounding vehicles. We propose in our approach a new structure of the representation of the target vehicle (vehicle n) and of its surrounding vehicles (vehicles a, b, c, d, e, and f) as shown in Fig. 4.2. We consider here 7 state variables; see Figure 4 In the case where the target vehicle is in the leftest (respectively, rightest) lane, then Dis ab (respectively, Dis ef ) is set to zero, and the relative speed ∆V ab_n (respectively, ∆V ef _n ) is set to -V n .

Definition of the state and action variables

∆V ab_n : (V a + V b )/2 -V n ∆V ef _n : (V e + V f )/2 -V n l inf o :
The action is the lane change decision, which belongs to {0,1,2}: 0: do not change the lane, 1: change to the right lane and 2: change to the left lane.

The NGSIM 101 data-set

As we mentioned previously, a brief description of the NGSIM data-set has been given in the section 2.7.1. NGSIM 101 dataset consists of three files of sub-data-sets for 15 minutes traffic data and 1500-2000 vehicles in each 15-minute data. Therefore, we can extract all the lane changing trajectories in the three files of NGSIM 101 dataset. Indeed, we are interested in the trajectories of lane changing without a motivation of going to the off-ramp. With this objective, a pre-processing of data is done and we obtained all the available lane changing trajectories are presented in Fig. 4.3.

For imitating the real lane changing behavior, the state variables at the moment that the driver decides to change lane need to be extracted. We estimate approximately the beginning of lane changing from vehicle trajectories as the moment that the driver decides to change lane. We use the same method as the one used in [START_REF] Thiemann | Estimating acceleration and lane-changing dynamics from next generation simulation trajectory data[END_REF] to capture the beginning of lane changing behavior and get the set of lane changing data (S + , A + ). In the other side, other trajectories not in the lane-changing trajectories can be regarded as no change lane moments. We can get by that a set of no lane changing data (S -, A -). The data D(S * , A * ), where (S * , A * ) = (S + , A + ) ∪ (S -, A -), is prepared for learning real lane change decisions. In the collected data D(S * , A * ), we save the 7 state variables as well as the corresponding action, as defined above in section 4.3.3.

Discretization of the state Variables

The first 6 state variables are empirically discretized and the discretization is corresponding to the NGSIM data. We set each variable to 5 possible values, (0, 1,2,3,4), to represent five sub-intervals, divided according to the quantiles 20%, 40%, 60%, and 80%; see Tab. 4.1. The five sub-intervals are [0%, 20%[, [20%, 40%[, [40%, 60%[, [60%, 80%[ , [80%, 100%].

In addition, the data D(S * , A * ) is discretized by the same methods, and after removing all duplicates in the NGSIM data, we obtain 13199 distinct couples of (s * , a * ).

Learning lane change decision in NGSIM data-set

The reward estimates how good the agent performs an action in a current state. With the objective of imitating the real actions (those of the data-sets), we use the reward to define how close is the agent's behavior to the real driver behavior. We denote the collected data D(S * , A * ), which contains a set of groups of (s * , a * ). Regarding imitation learning, we look for a policy π θ (s) which takes the same action a * if (s, a * ) ∈ D. Therefore, the reward for an action a which follows the policy π θ (s) and s ∈ S * , (s, a * ) ∈ D is defined as follows.

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

reward =    1, if π θ (s) ≡ a * -1, if π θ (s) ̸ = a * (4.4)
However, we need to consider that the target vehicle can meet an unknown state s not present in the collected data. In this case, the action a * taken by the closest s * ∈ S * need to be applied. The distance between the SUMO simulation state and the real state in collected data is calculated using the Euclidean norm (∥•∥ 2 ), and the closest s * is taken as the arg min

s * ∥s * -s∥ 2 .
The reward is defined as in (4.4).

In the other side, we give a punishment (negative reward) for the collision appearances and for out of road situations. For any action that causes a collision or an out-of-road situation, the reward is taken equal to -c, and this reward is defined as follows: reward = -c if crash or out-of-road situation. (4.5) where c is a positive constant. We set the reward in this case to c = 50 during the simulation experiment.

Simulation Experiment

We use the simulator SUMO to reproduce the NGSIM traffic after calibration. The reproduction of the NGSIM traffic data by numerical simulation consists in the creation of the road section, the selection of the microscopic traffic model (IDM car-following model, plus the default lane changing model in SUMO), and the model's parameters. We chose here the IDM car-

A model for imitation of the real lane change behavior from the NGSIM dataset

following model [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF], which is already implemented in SUMO. The IDM parameters are calibrated using the NGSIM vehicle's real trajectory with a genetic algorithm [START_REF] Zhao | A data-based traffic modeling approach for inter-vehicular collision simulation[END_REF]. Besides that, the creation of each vehicle is provided by its longitudinal origin position, longitudinal destination position, entering lane, and entering time (in seconds). This necessary information is also extracted from NGSIM 101 data-set. All the details of the preparation of simulation of NGSIM traffic in SUMO are presented in the Section 3.7 of chapter 3.

Training setup

Using the SUMO's associated Python API, named traffic control interface (TraCI), we can access the vehicle sequential information in the environment, and execute the lane change decision with our model, in order to control the target vehicle lane changing dynamics. Regarding the training of our model, we choose only one target vehicle to control from its origin to its destination, in every episode of simulation. We deactivate completely the default lane changing model for the chosen target vehicle. Moreover, the target vehicle is controlled by the IDM for car-following, and by the Q-learning algorithm for lane changing. We control the target vehicle to make a decision every 1 second. For each lane change, the target vehicle can arrive the target lane in 1 second. The other vehicles are considered as traffic vehicles, and they are controlled by the IDM model for car-following and by the SUMO default lane changing model, for lane changing behavior. Model configuration for each vehicle is shown in Tab 4.2. Furthermore, The SUMO simulation frequency is set to 10Hz. The visualisation of traffic simulation in SUMO is shown in 4.4, where the red vehicle is the target one at each episode.

The setting up of Q-learning hyper-parameters is shown in Tab. 4.3. As we mentioned it in section 2.4.2, γ is a parameter which represents how important the future return for the immediate action. As we model here human behavior, which is not perfect to anticipate so far for the decision of an action, we take here γ = 0.55 that the impact of the decision making is approximate to zero after 3 steps (3 seconds) far away, because 0.55 3 ≈ 0.16 and 0.55 4 ≈ 0.09. Moreover, the epsilon-greedy policy provides the agent's exploration of all possible couples of (s, a). We use here a decaying-epsilon-greedy policy algorithm, where the ε is a decreasing sequence on the number of episodes (n): ε n = e -n/d , where d is a parameter, taken here d = 3000.

The results for a first tested model without imitation from NGSIM Data

In the first proposition, we consider that the motivation of lane changing behavior is simply the gain in speed for the target vehicle. In this context, the reward of lane changing decision is the 'Speed gain', where the agent maximizes always the speed to take a decision of lane change.

The states and actions have the same definitions as in section 4. 3.3. Unlike what is presented in 4.3.6, the reward is defined as :

reward =    Speed Gain (v t+1 -v t )
-10 if crash or out-of-road situation

The results for imitation of human lane change model

We have created a traffic simulation environment, based on the human driving data from the NGSIM 101 data-set, and have used the Q-learning algorithm proposed above in order to mimic the real lane changing decision of human drivers. The limitation of the real data and the lack of traffic situations prevents the supervisor learning approach to cover all driving situations. The simulation traffic environment completes the unknown situations in the collected data. In the NGSIM 101 data-set, we distinguish about 13000 different state-action couples, while by numerical simulation, the vehicles have visited more than 16000 different couples of (s, a). We choose different cars in traffic as the target one at each episode, among randomly 800 chosen cars. The result of the evolution of the total reward of each episode is shown in Fig. 4.5. The expectation of the total reward tends to converge after 12000 simulation episodes. Moreover, the crash and the out of road situations do not occur any more at the last episodes; see Fig. 4.6, where 1 means the presence of collisions or out-of-roads and 0 means no presence such events. ). Moreover, the distribution of the number of lane changes for the last 1000 vehicles is provided in Figure 4.7. From this result, the number of lane changes is most realistic. In order to measure the accuracy of lane change behavior imitation, see Tab. 4.4, we collected all visited states of vehicles in the last 800 episodes, and we found that 87.09% of the states are present in the real data, while 12.23% of the states have the closest sate in the collected data with a distance of 1, as defined in sub-section 4.3.6. In simulation states present in the data-set 87.09% simulation states with distance of 1 to the data-set 12.23% simulation states with distance of 2 to the data-set 0.80% simulation actions present in the data-set 95.37% simulation actions not present in the data-set 4.63%

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

In order to illustrate the learned policy, and to understand its logic, we plot it as functions of several couples of states: a = f (V n, ∆V preceding ), a = f (V n, ∆V ef _n ), a = f (V n, Dis ef ) and a = f (Dis ab , Dis ef ) illustrated with color maps in Fig. 4.8-(A), (B), (C), and (D) respectively.

We recall that we have 5 discrete values for each state variable. We use RGB color to represent the action (change left (in red), change right (in green) and stay on lane (in blue). The grey level show the probability of each action in each case. The black cases are the states never visited. In addition, we have 4 views in each figure: the three views on the right side show each action respectively for change left, change right, and stay on current lane; plus a main view with combination of all actions.

In Fig. 4.8-(A), the x-axis represents the speed of the target vehicle, while the y-axis represents the relative speed of the target vehicle, with respect to the preceding vehicle. We can see from this figure that the controlled vehicle tends to maneuver a lane change in the cases when

The results

(A) a = f (V n, ∆V preceding ) (B) a = f (V n, ∆V ef _n ) (C) a = f (V n, Dis ef ) (D) a = f (Dis ab , Dis ef )

Introduction

In the last decades, deep learning methods, a specific set of machine learning methods designed to build models based on sample data, have shown excellent performance in many fields (e.g. natural language processing, image processing and financial trading strategies). In the field of microscopic road traffic, where researchers are interested in modeling individual vehicle movements on road networks, the use of deep learning is also becoming increasingly popular.

As well known, two driving behaviors are generally distinguished in the microscopic traffic modeling: Car-Following (CF) and Lane Change (LC), which describe respectively longitudinal and lateral vehicular movements. In the literature review, one of the first car-following models is the GHR model [START_REF] Chandler | Traffic dynamics: studies in car following[END_REF]. Among the most famous models, we cite the Wiedemann model (Mohammad and Z. Zheng, 2014), Gipps model [START_REF] Gipps | A behavioural car-following model for computer simulation[END_REF], Krauss model [START_REF] Krauß | Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics[END_REF] and Intelligent driver model(IDM) [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF]. Some reviews of car-following models can be found in [START_REF] Brackstone | Car-following: a historical review[END_REF]Mohammad and Z. Zheng, 2014;[START_REF] Aghabayk | A state-of-the-art review of carfollowing models with particular considerations of heavy vehicles[END_REF]. In addition, in order to model the imperfect human driver, the authors of [START_REF] Treiber | Delays, inaccuracies and anticipation in microscopic traffic models[END_REF][START_REF] Lindorfer | Modeling the imperfect driver: Incorporating human factors in a microscopic traffic model[END_REF] carried out some extensions based on car-following models, to describe mathematically for the reaction time, the anticipation, and estimation errors of human drivers. For lane change models, we cite (Toledo, Koutsopoulos, and M. Ben-Akiva, 2007;[START_REF] Moridpour | Lane changing models: a critical review[END_REF]Z. Zheng, 2014) as reviews, authors have shown that the lane change decision is usually taken based on several sub-models of the driver's lane change motivation (e.g. mandatory LC and discretionary LC; or free, cooperative, and forced LC). We can observe that, compared to CF models, which are well developed, LC models are more complex and rule-based with many hand-crafted parameters. LC models generally show limited performance and are hard to calibrate, and to catch with human lane-changing behavior (Z. [START_REF] Zheng | Recent developments and research needs in modeling lane changing[END_REF].

On the other hand, a number of data-driven CF and LC models using machine learning methods have been developed (Q. Wang, Z. Li, and L. Li, 2014;[START_REF] Xie | A data-driven lane-changing model based on deep learning[END_REF][START_REF] Wang | Capturing car-following behaviors by deep learning[END_REF]. It has been demonstrated that, comparing with physical models, the machine learning based ones, including deep learning methods, provide a better performance for the accuracy of trajectories, and for the replication of traffic flow characteristics.

Long short-Term Memory (LSTM) neural networks [START_REF] Hochreiter | Long short-term memory[END_REF] are recurrent neural networks (RNNs), able to learn time series with keeping a memory of previous sequences [START_REF] Gers | Learning to forget: Continual prediction with LSTM[END_REF]. Several studies have been applied using LSTM for human driving behavior modeling or human driving forecasting, such as in [START_REF] Xie | A data-driven lane-changing model based on deep learning[END_REF], where the authors developed a LC model (the LC decision prediction combining the LC execution trajectory). In the latter model [START_REF] Xie | A data-driven lane-changing model based on deep learning[END_REF], the execution of the lane change decisions is formulated based on a LSTM neural network. In (Y. Zhang et al.,

Problem statement

2018), the authors proposed a simultaneous CF and CL model using a LSTM neural network, and with a constraint on the car time headway, which permits to retrain the neural network in the case where the time headway is not satisfied. Moreover, in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], the authors proposed a LSTM model for vehicle trajectory prediction from 1s to 10s in the future. All these works show that LSTM modeling is essentially useful for the prediction of human driving behavior. However, different studies used different model structures, with different input designs for the neural networks.

In this chapter, we propose a LSTM model to learn human driving CF and LC behavior simultaneously. Our model is based on the relevant works in (Y. [START_REF] Zhang | Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[END_REF][START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], where the sequential decision for longitudinal (speed and acceleration) and lateral (position or lane number) movements are taken simultaneously. We propose two different designs for the model architecture, using neural networks with LSTM layer, and three designs for the input variables. By studying different combinations of the model architectures and the input designs, we aim at discovering the best combination of architecture and input design. The models are trained and tested on the NGSIM 101 dataset [START_REF] Alexiadis | The next generation simulation program[END_REF]. The model performance is evaluated in terms of prediction accuracy on longitudinal speed and lateral position. The obtained result is compared with a physical car-following model, the Intelligent Driver Model (IDM) [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF], and with another model using classical neural networks.

Problem statement

In the state-of-the-art, we can find that Car-following and Lane Change behaviors are usually considered and modelled independently. However, in the real world, human drivers can hardly split the driving task into two independently tasks (Y. [START_REF] Zhang | Human-like autonomous vehicle speed control by deep reinforcement learning with double Q-learning[END_REF]. We propose in this work to model simultaneously CF and LC decisions using a LSTM neural network. The experiment is applied on the NGSIM 101 dataset. Our model is based on an existent work presented in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF].

The prediction of the driving decision making (e.g. speed, position, etc.) can be considered as a regression problem by LSTM architecture, where the vehicle's present behavior is related to current traffic conditions. The latter conditions consist of all the information of surrounding vehicles (presented by their own trajectory), and also of the vehicles' own previous behavior in the passed time. We denote by X the space of observable features, and by O the motion prediction outputs for the target vehicle. With this assumption, for LSTM neural networks, the input is a sequence of history observations (X = (x 1 , x 2 , ...x k )). x ∈ X denotes the information on the target vehicle history as well as the history information of its surrounding vehicles at each time step. The variable k is the length of time series in the input. The output of the neural Long Short-Time Memory Neural Networks for Human Driving Behavior Modelling network is the prediction of the target vehicle motion Ŷ ∈ O. We aim at using supervised learning approach and train a regression function f such that the predicted outputs Ŷ = f (X) match the actual values (real data) as closely as possible.

As in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], we use here the Keras deep learning framework, where an extended LSTM model has been implemented. The algorithm is described in [START_REF] Gers | Learning to forget: Continual prediction with LSTM[END_REF]. The models are trained and tested on the NGSIM 101 dataset [START_REF] Alexiadis | The next generation simulation program[END_REF], which is an open highway traffic dataset containing about 6000 vehicle trajectories. In this work, we evaluate totally two different model architectures combining three different input designs. Among the different combinations, we aim to discover the most efficient model.

In the following sections, we will present the NGSIM 101 dataset (section 4.3.4). Then, we will describe the model structure, including the models description, the proposition of the different designs for input variables, and the definition of output variables (section 5.4). In order to show the result with comparison with a physical traffic model (Intelligent driver model (IDM) [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF]), we will demonstrate the calibration method of the IDM model in the section 5.5. At the end, we will illustrate the obtained results in section 5.6.

NGSIM 101 data-set and Data preparation

As we mentioned previously, a brief description of the NGSIM data-set has been given in the section 2.7.1.

In our study, we focused only on car trajectories in the five main lanes. In addition, we are only interested in discretionary lane changes, which are independent on entering and leaving the road. The authors of [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF] observed that the data is noised for the vehicle's position, velocity, and acceleration signals. Therefore, in this work, we firstly pre-processed the data-set using the same method as in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], where we apply a first order Savitzky-Golay filter [START_REF] Savitzky | Smoothing and differentiation of data by simplified least squares procedures[END_REF] to smooth each vehicle longitudinal and lateral position, and then we recompute their corresponding velocities and accelerations.

Model structure

The model we propose in this chapter is based on an existent work presented in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF], where a LSTM neural network is used to predict the future car trajectories from 1s to 10s. We investigate here different neural network architectures with different input designs,

Input variables

Several previous works have been performed regarding the use of deep neural networks to model human driving behavior [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF]X. Zhang et al., 2019). Different designs of the input variables of the models have been proposed. In (Altché and La Fortelle,

Model structure

• longitudinal speed v i and longitudinal acceleration a i

• vehicle length l i and vehicle width w i

• relative speed ∆v i = v target -v i • longitudinal distance s long_i = x target -x i • lateral distance s lat_i = y target -y i
For the input design 1, we have a total of 60 input variables of the model.

2. Input design 2: in the second input design, we are interested in finding the performance influence by using different inputs. Comparing with the first design, we consider here that the relative variables with respect to the target vehicle are more important than the absolute ones. The acceleration is ignored in this design to avoid redundancy. In addition, we defined a new variable for noticing the presence of surrounding vehicle i, named as presence p i (0 for non presence, and 1 for presence). If vehicle i is not present, all the relevant value (l i , w i , ∆v i , s long_i , s lat_i ) for this vehicle are set to 0. For the input design 2, we have a total of 41 input variables of the model.

Input design 3:

In this design we take all the input variables of design 1 and design 2. Similarly, if a vehicle i is not present, then all the relevant value (x i , y i , v i , a i l i , w i , ∆v i , s long_i , s lat_i ) for this vehicle are set to zero. For the input design 3, we have a total of 66 input variables of the model.

x target y target v target a target l target w target x i y i v i a i l i w i ∆v i s long_i s lat_i p i input design 1 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ input design 2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ input design 3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
Table 5.1 -input designs

Output variables

As described in the sections before, our problem is set to learn the decision of human driving behavior using LSTM neural networks by prediction of car-following and lane changing behavior, with the condition of keeping a memory of vehicle previous action information and the previous motion information, as well as the current motion information of surrounding vehicles. We choose to use the longitudinal speed and lateral position as output, because for the longitudinal positions, the values can become quite large compared to the lateral positions. As mentioned in section 5.4.1, for the inputs, we use k hist sequences information in the history as inputs. Therefore, we define the two outputs of the model to control the vehicle in real

Long Short-Time Memory Neural Networks for Human Driving Behavior Modelling time, which are v target (k hist + 1) and x target (k hist + 1). The index k hist goes up to 50 in our experiments.

Training and Test experiments

The NGSIM 101 dataset is separated into 3 sub-datasets, and each sub-dataset corresponds to 15-minute traffic data. We train the neural network using the first 15 minutes data, and test separately with the second and third 15-min sub-datasets. In practice, to train the neural network, epochs and mini-batches are defined for the weights updating process. A mini-batch refers to a subset (m individual samples) of the whole training set, to compute a partial gradient and update the weights accordingly. An epoch refers to processing the whole training set once. We used the Adam algorithm [START_REF] Kingma | Adam: A method for stochastic optimization[END_REF], which is a stochastic gradient algorithm and has been implemented in Keras framework. For the loss function of neural network, we used the Mean Square Error (MSE). The learning rate is another parameter, which determines how big a step is taken toward the descent direction to the minimum of loss function when updating the weights in neural networks. Therefore, regarding hyper-parameters for training the neural networks, the configuration is shown in

Intelligent driver model (IDM)

We propose to compare our LSTM model with an existing physical car-following model. In this purpose, we choose the Intelligent driver model (IDM) [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF]. The IDM model is a classical car-following model used in various microscopic traffic simulators, such as SUMO [START_REF] Krajzewicz | Recent development and applications of SUMO-Simulation of Urban MObility[END_REF], VISSIM (Fellendorf and Vortisch, 2010b), and AIMSUN (Casas et al., 2010b). Authors in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF][START_REF] Pourabdollah | Calibration and evaluation of car following models using real-world driving data[END_REF] found that the IDM model is one of the most human-like car-following models. There are several methods for calibrating microscopic models. We use in this paper the same method as in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF], where a genetic algorithm is applied for each driver data to find the optimal IDM parameters. The objective function used to calibrate the IDM model is the root mean square error (RMSE) of the vehicle speed. The

Results

IDM model is written as in 3.18, and the parameters considered to be calibrated are pre-defined in 3.6.1. As the result, we give in Figure 5.6 the distribution of the RMSE on the car-speed variable, obtained from the calibration of the IDM model. The RMSE value is presented by every driver data in the first 15-minutes of the NGSIM 101 data-set (about 1900 vehicles). We notice that from Figure 5.6, the RMSE of calibrated IDM model for each driver ranges (mostly) from about 0.5 to 1.5 (m/s).

Results

We present in this section the results of our proposed LSTM model by comparison with the IDM model for speed decision, and with the model of classical neural networks, for both longitudinal speed and lateral position. Firstly, Fig. 5.7 gives the loss evolution of our proposed LSTM models in the training phase, for different input designs. The loss value presented here is the default Keras mean error (MSE) between label and prediction. In the case of having two outputs, the error is the average MSE of the two outputs. Fig. 5.7 shows the training loss during 20 epochs. We can see that for all the different models, the loss decreases and converges. We can also infer from Fig. 5.7 that the proposed LSTM models with input design 2 give the best results in terms of the MSE evolution in training phase.

In order to compare the results of all models, we use here a metric denoted by 'Mean driver RMSE' which calculates the average error between the real data and the predicted value from the outputs of each model, for every driver in the data-set. It is defined as follows : where D is the number of drivers in each sub-dataset; n i is the total number of sequences of the vehicle trajectory, in every sub-dataset i; y t is the value of the considered variable in the real data; and ŷt is the output of the model for the considered variable (car-position, car-speed, etc.). Here, the output of the IDM model is the longitudinal speed, while for LSTM model, the outputs are the longitudinal speed and the lateral position. The 'Mean driver RMSE' on the car-speed variable for the IDM model is 0.9263 (see the RMSE distribution in Fig. 5.6). This error is far from the accuracy provided by the LSTM models, as it is shown below. The 'Mean driver RMSE' for the different models in terms of longitudinal velocity and lateral position is given in Table 5.3. These results in Table 5.3 show that M1_design_2 provides a best performance among all presented models, thus, the accuracy both in longitudinal velocity and lateral position is the best, as well as for training and tests. Models using input design 2 have

M ean driver RM SE = 1 D D i=1 1 n i n i t=1 (y t -ŷt ) 2 (

Results

better performance than models using other input designs. In addition, models using LSTM are better than models using only classical neural networks (comparing M1_input2, M2_input2 and M N N _input2, M N N 2 _input2 in Table 5.3). In Fig. 5.8, we show some trajectory prediction results for the LSTM model (model 1 with input design 2: M1input2) during the training phase. We show the longitudinal velocity and lateral position predictions for two drivers. In Fig. 5.9, we show some trajectory prediction results of the LSTM model (model 1 with input design 2: M1input2) in the test phase for the other two drivers. Fig. 5.8 and Fig. 5.9 show the high accuracy achieved by the M1input2 model.

Results

Model validation using HighD dataset

As we presented above, the result of different models based on NGSIM 101 dataset show the M2_input2 has a best performance among all the tested models. In this section, I will present the result of all proposed models using another dataset, HighD [START_REF] Krajewski | The highD Dataset: A Drone Dataset of Naturalistic Vehicle Trajectories on German Highways for Validation of Highly Automated Driving Systems[END_REF].

HighD dataset

As what I presented in the previous chapter, in the Section 2.7.2, a brief presentation of HighD dataset is given. As the same application using NGSIM 101 dataset, we choose a part of HighD dataset and separated it into one part for training, one part for first test, and one part for the second test. HighD dataset is larger than NGSIM 101 dataset. It contains 60 files of registration available in the whole HighD dataset and about 2000 vehicle-trajectories in each file. The six recorded roads are all straight roads with two directions and each direction has two or three lanes.

In the experiment for the validation of the proposed models by HighD dataset, among 60 files availble in HighD dataset, we chose randomly 10 files within about 20000 vehicle trajectories as the training dataset, 5 different files within about 10000 vehicle trajectories as the dataset in test1 and another 5 files with about 10000 vehicle trajectories as the dataset in test2. First, we obtained the result of the evolution of loss value in the training phase (see. Figure 5.10). As the same, the loss value presented here is the default Keras mean error (MSE) between real data and prediction. In our case, the outputs contains the prediction of speed and the prediction of lateral position, the loss value is the average MSE of these two outputs. From the Figure 5.10, it shows the similar result that we trained the model using NGSIM dataset, where the models using second input design (M1_input2 and M2_input2) have better performance than other models using other type of input designs. Furthermore, they are better in terms of results in training result, Test1 and also Test2 (see. Long Short-Time Memory Neural Networks for Human Driving Behavior Modelling

Model validation using HighD dataset

Conclusion

We have presented in this chapter an LSTM neural network model for car-following and lanechanging on road networks. The experiments have been applied on the NGSIM 101 data-set for the training and tests. Furthermore, we applied the same experiment on another data-set HighD data-set, the result is similar to the result of that on NGISM 101 data-set. The obtained results show that the LSTM neural network model has outstanding performance in terms of accuracy for longitudinal speed and lateral position variables. In comparison with the physical microscopic traffic model IDM (Intelligent Driving Model), the LSTM model has significant improved performance on the car-following part. This can be explained by the fact that physical traffic models have generally a small number of parameters, rather than neural network models.

In addition, we have compared the LSTM model with a classical neural network model, and concluded that the model using LSTM has better performance than the classical neural network model.Our perspectives on this research direction are: 1) implementation the model on traffic simulators, such as SUMO (Simulation of Urban Mobility) to simulate a more realistic road traffic, and 2) extension of the model, in particular for considering several driver profiles.

Chapter 6

Highway Traffic simulation in SUMO using proposed models

We present in this chapter the implementation in SUMO of the car-following and lane-change models proposed and adapted in the preceding chapters. The objective is to simulate the whole highway traffic, where we control all the cars in the traffic by our proposed models. Meanwhile, the other types of vehicles, consisting of trucks and motorcycles, are controlled by the default models in SUMO. Different combinations of car-following models with lane changing models are tested and analyzed, aiming at understanding the performance of each model combination. Moreover, we used these different combinations of models to simulate the NGSIM traffic and the HighD traffic data-sets, in order to analyze the sensibility of all the proposed models on these two data-sets.

In this chapter, section 6.1 presents the analysis of the data we worked on, including the NGSIM data-set and the HighD data-set, with the purpose of understanding these two traffic data-sets, and therefore derive the main resemblances and differences. Section 6.2 gives the details on the implementation of the models in SUMO. Section 6.4 and section 6.5 give out respectively the results of simulation for the NGSIM traffic and the HighD traffic.

Microscopic traffic datasets : NGSIM and HighD

The traffic analysis began with some statistics of traffic (macroscopic indicators), including the traffic average speed, the car-density as well as the speed-density diagram (fundamental diagram).

NGSIM 101 dataset

As seen in the previous chapters (Chapter 3, 4, and 5), the works are mainly based on the NGSIM dataset. In Figure 6.1, a visualisation of the vehicle positions is shown. A congestion phenomenon can be illustrated by a shock of position deferential in the vehicle trajectories (a large reducing of the slope in vehicle position trajectory, see Figure 6.2). Similarly, a congestion can also be shown by a shock-wave in the car-speed trajectories, where a shock of speed shows the local minimum of travelling speed in consecutive vehicle speed trajectories (see Figure 6.3). We can observe the traffic of NGSIM has some congestion, which sometimes starts from the middle of the road section (Figure 6.2a). From Figure 6.3, we can observe that the vehicle-speed goes up and down.

To understand the traffic on each lane, we do some investigation for the first 15 minute data. In Figure 6.4a, we show the mean traffic speed of each lane for the first minutes data. We can see that the average traffic speed evolves similarly in each lane, where the speed oscillates up 6.1 Microscopic traffic datasets : NGSIM and HighD and down. In particular, we can observe that in the lane 4, which is the leftest lane and the highest speeding lane in general, the change of speed is the most one while the speed is slower than in the other lanes. Concerning the vehicle density, which is another macroscopic traffic indicator, it can be defined as the number of vehicles in each lane (here, the length of NGSIM road section is fixed (640m)). In Figure 6.4b, we show the evolution of the vehicle-density in each lane. From this figure, we can observe that the number of vehicles in the lane 4 is the biggest one, which has an effect on the traffic speed on lane 4, the slowest one (see Figure 6.4a).

After observing the traffic speed and traffic density separately, the traffic fundamental diagram (speed-density diagram) can be displayed. For each lane in the first 15-minute period data of the NGSIM 101 dataset, the speed-density diagram is shown in Figure 6.5, while the speed-density diagram for the 3 periods of the NGSIM 101 dataset is shown in Figure 6.6. We can see that there exists some similarity on the traffic speed-density diagrams between the lanes (Figure 6.5), and also some similarity of the traffic speed-density diagrams for the 3 periods in the NGSIM dataset (Figure 6.6). We can see that the traffic is congested: the vehicle speed decreases with the increasing of the traffic density. Highway Traffic simulation in SUMO using proposed models

Implementation of the models in SUMO

As well known, car-following and lane changing models are the two fundamental models in microscopic traffic. In this thesis, we used the car-following model IDM (intelligence driver model) in Chapter 3, which is a well known car-following model, and is already available in SUMO. In Chapter 5, we proposed a human-like driving model using LSTM neural network, simultaneously for car-following and lane-changing behavior control. The latter model controls the vehicle movement by predicting the decision on speed and lateral position, based on the vehicle own movements and on the states of its surrounding vehicles in the historic steps. As shown above, the neural network based model can be more efficient in predicting the human driving behavior, than the classical physical microscopic model, e.g. the intelligent driver model (IDM). Indeed, the neural network based model performs a good accuracy for the real vehicle trajectory data. In Chapter 4, we have shown that classical physical lane change models have limited performance, and the reinforcement learning provides an alternative method for lane changing modelling. Q-learning algorithms have been used with online calibration on human lane change, based on the NGSIM data. The results have also shown a good accuracy for human lane change imitation.

In this section, we study the use of different combinations of car-following models and lane changing models, including 1) the calibrated IDM model, 2) an adapted neural network based car-following model, 3) the SUMO default lane changing model and 4) our proposed reinforcement learning based lane changing model. We simulate the NGSIM and HighD traffics, and compare different combinations.

Calibration of the IDM model

The IDM model needs to be calibrated to present the real human behavior. For the car-following model calibration, we have presented previously in Section 3.6 the method of calibration for the selected extreme profiles (aggressive and inattentive), but with the purpose of generation of car collisions (in numerical simulation) by increasing the number of extreme drivers in the traffic (see. Chapter 3). In this section, with the objective of generalizing the calibration work, and in order to reuse the calibrated model for other use cases, we propose the use of the NGSIM data to calibrate the IDM model and to obtain a set of IDM parameters. This set of parameters is the average value of parameters calibrated through all drivers trajectories registered in the first 15 minute period NGSIM 101 dataset. We use here the same method as in the one in section 3.6 for calibrating all drivers presented in NGSIM data-set, and we choose the parameters' boundaries with an assumption to be suitable for each driver. Let us recall that the calibration is performed using a genetic algorithm to find the optimal parameters of the IDM model for each diver in the NGISM data-set, with the purpose of minimising a proposed

Implementation of the models in SUMO

error in position. The error is defined as the root mean square percentage errors (RMSPE) :

RM SP E(θ) = 1 P P i=0 (x simul i (θ) -x data i (θ)) 2 x data i (θ) 2 , ( 6.1)
The boundary of each parameter (upper and lower bounds) are given as an input to the genetic algorithm. In Table 6.1 we show the chosen boundary of each parameter of the IDM model, and we calculate the result for errors between the real data and the output of the calibrated IDM model, including RMSE for speed, and RMPSE for the position. We show in Table 6.1 the mean value, the standard deviation, the median, and the value of the quantile at 25% and 75% for each parameter and each error value. Table 6.1 shows the performance of IDM model and the calibration by the genetic algorithm. We have the RMSE on the car-speed at 1.004m/s for about 1900 drivers, and the RMSE on the car-position at about 9.1m, with only 4.4% of RMPSE. The performance of the IDM model calibration can be also observed in Figure 6.12, where we show the result of calibration of the IDM model for 4 selected drivers. We give in each sub-figure the driver's real speed and the speed computed by the calibrated IDM model. From Figure 6.11, we see that the parameters V 0 and T are well distributed in their respective intervals. However, the parameters a and b are rather close to the upper limit 6 m/s 2 . This result corresponds to what is found in (M. [START_REF] Zhu | Modeling car-following behavior on urban expressways in Shanghai: A naturalistic driving study[END_REF]. We use the column of the mean value in Table 6.1 as the calibrated parameters of IDM to simulate the traffic in SUMO.

Car-following model using Neural network

The model that we proposed in our research work in Chapter 5 is a model using LSTM neural network, where we compared three different input designs as shown in Table 6.2. Since this model has a lot of inputs, We propose a simplification of the model in order to implement it in SUMO and to simulate road traffic in SUMO.

x target y target v target a target l target w target x i y i v i a i l i w i ∆v i s long_i s lat_i p i input design 1 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ input design 2 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ input design 3 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 6.2 -input designs
In order to implement a neural network car-following model in SUMO to simulate the NGSIM and the HighD traffics, I propose to use a simpler car-following model with a classical neural network trained by the NGSIM data. The model uses only the features shown in Table 6.3. In addition, the target vehicle position is a variable depending on the traffic map, where the traffic road length is not constant. We don not use here the target absolute variables, but relative values with the surrounding vehicles, which are more representative. Moreover, for the surrounding vehicles, we considered only the leading vehicle and the following vehicle of the target vehicle, which are the most two important participants for the car-flowing speed controlling.

All the used features of target vehicles :

• longitudinal position x target (t)

• longitudinal speed v target (t)

• vehicle length l target and vehicle width w target All the features of the leading vehicle and following vehicle are :

• relative speed ∆v i (t) = v target (t) -v i (t) • longitudinal distance s long_i (t) = x target (t) -x i (t)
• vehicle length l i and vehicle width w i • presence p i : 0 for non presence, 1 for presence if vehicle v i is no present, all the relevant value (l i , w i , ∆v i , s long_i , s lat_i ) for this vehicle set to be 0.

In this input design, we have a total of 13 features as inputs of the neural network.

Highway Traffic simulation in SUMO using proposed models

x target y target v target a target l target w target x i y i v i a i l i w i ∆v i s long_i s lat_i p i input design2 SUMO ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 6.3 -Input design2 in SUMO implementation

Neural Network model

The neural network model is changed. We reduce the input features, and take only one output of the model, which is the longitudinal speed. In this context, we supposed using a classical fully connected neural networks (mathew2012neural), without the LSTM architecture; see Fig. 6.13. The neural network model for car-following is shown in Fig. 6.14. We give in Table 6.4 the RMSE for all models trained and tested using the NGSIM data-set. The car-following model proposed for SUMO implementation has a similar accuracy as the best model we obtained in Chapter 5. Therefore, we think that this car-following model using a fully connected neural network can replicate the human car-following driving behavior.

Implementation of the models in SUMO

Implementation of the neural network CF model

In SUMO, the car-following model is controlled by a parameter called 'speed mode', and this parameter is presented by a 5-bit value. The given integer is a bitset (bit0 is the least significant bit) with the following fields: We set the speed mode at 1 for controlled vehicle, in order guaranty safety (avoid car-collisions). For all the other situations, the speed is controlled by our neural network car-following model, proposed for the implementation in SUMO.

Lane change model using Reinforcement Learning

The proposed lane change model is the the model we presented in Chapter 4. This model is also calibrated using the NGSIM 101 data-set.

Implementation of the lane changing model based on reinforcement learning in SUMO

The lane change model discriminates four reasons to change lanes:

• strategic (change lanes to continue the route)

• cooperative (change in order to allow others to change)

• speed gain (the other lane allows for faster driving)

Highway Traffic simulation in SUMO using proposed models

• obligation to drive on the right This strategy of lane change control has a parameter named 'lane change mode' and to disable all autonomous changing but still handle safety checks in the simulation, either one of the modes 256 (collision avoidance) or 512 (collision avoidance and safety-gap enforcement) may be used. We set the lane change mode at 256 to disable all lane change strategy of SUMO, but we need to check for collision avoidance when our lane change model demands a change lane.

To be noticed, the lane change order by our lane change model can be prevented by SUMO if safety is not guaranteed.

Controlling one car in the traffic

The implementation began with testing in the NGSIM scenario for controlling only one car using the neural network based car-following model and the lane changing model using reinforcement learning. The result in simulation for the vehicle speed control is presented in Figure 6.15. It seems that the vehicle is well controlled by our proposed model, and has a different speed policy comparing to the sumo default model. We show in Table . 6.16a the Highway Traffic simulation in SUMO using proposed models

Different combinations of models for Highway traffic simulation in SUMO

Once we tested the control of one vehicle in simulation using the proposed CF and LC models, we begin the simulation of the whole traffic (NGSIM and HighD) aiming at testing all the combinations given in Table 6.

✓ ✓ Model1 ✓ ✓ Model2 ✓ ✓ Model3 ✓ ✓ Model4
✓ ✓ Table 6.5 -A summary of all the tested models in simulation

Results of the Simulation of NGSIM traffic

In Figure 6.17, we show a screenshot for the visualisation of NGSIM traffic in SUMO. We present the results in two parts consisting of traffic average speed evaluation and number of lane changes evaluation.

Traffic Speed evaluation

The comparison of the average traffic speed between real data and simulated traffic using different combination of models is shown in Figure 6.18. We give the RMSE between the NGSIM data and the simulated traffics using different combinations on traffic average speed, respectively for sumo_default model, model1, model2, model3 and model4 in Table.6.6. We observe that the model2, which uses CF model by neural network and LC model using reinforcement learning

Results of the Simulation of NGSIM traffic

has the best performance among all the tested models. The traffic speed-density diagram of the simulated traffic obtained from SUMO is shown in Figure 6.19. We show the speed-density diagram produced by each simulated model. We observe that the Model2, which uses the CF by neural network and LC by reinforcement learning has the best performance; see Figure 6.19.

We then focus on the two best models by the criteria shown in Table 6.6, which are the model2 and the model1. We show their diagram of speed-density in Figure 6.20. Therefore, by the result shown in Table 6.6 and the result shown in Figure 6.20, we observe that model2 has the best performance for simulating the NGSIM traffic compared with the real data.

Results of the Simulation of NGSIM traffic Vehicles Lane-changing number evaluation

The lane change distribution of each cars is shown in Figure 6.21. Comparing to the real distribution of number of lane changes in the NGSIM data-set, the distribution of lane changing number produced by all models in simulation is similar to that of the real data. However, we notice that the result of simulation for the lane changing are obtained under the SUMO collision avoidance check (when the LC is done by our model). Therefore, even the lane changing model using reinforcement learning is shown as a collision-free in Chapter 5. In order to exted the model for controlling all the cars' LC in the traffic instead of controlling only one car, we need to introduce safety and car-collision avoidance in the lane changing modeling. Highway Traffic simulation in SUMO using proposed models

Results of the Simulation of the HighD traffic

In the previous section, we have shows the simulation results for the NGSIM traffic. In this section, we present the results of simulations for the HighD traffic. In the HighD data-set, we choose the scenario of the 13th sub-dataset among from a total of 60 sub-datasets. The 13th sub-dataset contains traffic data on a 3-lane road map on the highway with 1217 vehicles in the traffic during 17 minutes. Similarly to the evaluation of simulations for the NGSIM traffic, we are also interested in the presentation of traffic speed evaluation and lane changing evaluation for the simulations of HighD traffic. In Figure 6.22, we show a screenshot of the HighD simulation in SUMO. As already indicated in section 6.1, the NGSIM and the HighD traffics are different in terms of traffic density speed. We reuse the proposed models studied for the NGSIM data-set (Table6.5), to simulate the HighD traffic and to observe the performances of each model.

Traffic Speed evaluation

For the traffic speed evaluation, we evaluate the simulation using different combinations of CF and LC models; see Table 6.5). In Figure 6.23, we show the results of HighD traffic simulations regarding the average traffic speed. We firstly tested the IDM model calibrated with the NGSIM data-set. The parameter maximum speed (V0 in Table 6.1) of IDM needs to be re-calibrated to simulate the HighD traffic. The average (on drivers) of the maximum speed in the NGSIM traffic is 25.07m/s (see V0 in Table 6.1), while for the HighD traffic, we rather take 40m/s for the maximum speed.

We show in Figure 6.24 the result of the HighD traffic simulations, for different models. for the SUMO default model with the maximum speed 40m/s, the model1 using calibrated IDM model with the maximum speed 40m/s and the SUMO default lane change model, the model2, the model3 and the model4 presented in Table 6.5. In Table 6.7, we compute the RMSE error between the simulation result and the real data of the HighD dataset, in terms of traffic average speed. We observe that the model1 using the calibrated IDM model with the maximum speed at 40m/s and the SUMO default lane change model is the best one; while the model4 using the CF by neural network and default SUMO LC model is the second best model for simulating the HighD traffic. The model2 and model3 give a traffic speed very different from the real data.

Conclusion

The NGSIM traffic is very heavy, while the HighD one is very light. The neural network and reinforcement learning models trained only on the NGSIM data-set are not suitable for simulating traffic as different as the HighD one, from the NGSIM one. The IDM model, which uses physical parameters, is easy to extend with human factors (such as the reaction time) and

Conclusion

it can be used to generate car-collisions. The NGSIM-calibrated IDM model is less sensitive to data, and can be used to simulate other traffics like the HighD one, but it requires a re-calibration with the maximum vehicle speed.

Chapter 7

Conclusion and Perspectives

Driving behavior models describe the driver's maneuvering decisions under different traffic conditions which are crucial for microscopic traffic simulation systems and important in several research areas such as traffic safety, capacity analysis, and assignment studies. On the other hand, there is a growing interest in autonomous driving, which is a rather challenging topic because it involves the automated operation of vehicles in many different situations. We expect that fully autonomous vehicles can reduce car accidents and improve overall traffic safety. However, the complex system of autonomous vehicles combining the perception, decision, the control system and the traffic environment is a dynamically changing environment which involves many interactions between road users. Driving tests are essential to validate and to ensure the functionalities of the vehicle. Driving tests in the real world are a big challenge because fatal accidents involving autonomous vehicles may happen during the test. Digital simulation can reduce the cost and time and avoid potentially dangerous situations for validation of Advanced Driver-Assistance Systems (ADAS) and autonomous driving. All those increasing uses of traffic simulation for many studies highlight the importance of a good understanding and modelling of human driving behavior.

Conclusion on our contributions

In this thesis, the works are focused principally on human driving models, consisting of car-following and lane change models. We have also proposed a model for generation of car-collisions in numerical simulation, for the validation of autonomous vehicle features in dealing with traffic accidents.

Classical models in microscopic traffic modelling started with concepts of physics based on the classical Newtonian mechanics, and then augmented by behavioral aspects and driving rules. This kind of physical models have already been completely integrated in many traffic

Conclusion and Perspectives

simulators, such as SUMO, Aimsun, Vissim, etc. We have firstly investigated the classical models in Chapter 3, aiming at proposing a novel approach of car-collision generation method in numerical simulation. We have derived and distinguished different driver profiles, including aggressive, inattentive and normal drivers, based on the real traffic from the NGSIM data-set.

Within this first work, we proposed a calibration mechanism for a chosen car-following model (IDM model), using a genetic algorithm. The objective of the calibration of the IDM model is for representing the different real driver profiles in numerical simulation. We then proposed to increase the percentage of 'extreme' profiles (aggressive and inattentive) in numerical simulation, in order to generate car-collisions. This work has been experimented based on the first period 15-minute data of the NGSIM 101 data-set. We have then validated our proposed approach using the second 15-minute period data. One of the important results we obtained in this work is the characterization of the relationship between the ratios of these two driver profiles over the whole drivers population, and the car-collision occurrences counts. We have also classified the occurred car-collisions and analyzed their severity, in particular with respect to the relative speed between the cars which cause the collisions. Furthermore, we have investigated the severity of the generated collisions, and find that the generated car-collisions involved between an aggressive driver as a leader and an inattentive driver as a follower occupy the most frequent collision occurrences, meanwhile the generated car-collisions involved between two inattentive drivers are the most severe ones.

In the other chapters of this thesis, we focused on artificial intelligence algorithms which have already demonstrated their efficiency in data-based modelling, during the last years. These algorithms provide alternative methods for modelling human driving behaviors. With this objective, we investigated and applied some machine learning algorithms for modelling human driving behavior, including a model for lane change using Reinforcement Learning (in Chapter 4) and a model combining car-following and lane changing behavior prediction using Long Short-Term Memory (LSTM) (in Chapter 5).

In Chapter 4, we proposed a Q-learning model for the modelling of human lane change decisions. The model demonstrates good performances in mimicking the human lane changing decisions (up to 95%). Moreover, the model uses numerical simulation to complete the data with unknown situations. We observed a percentage of 13% of additional traffic situations created by the traffic simulation environment. Training in a simulated traffic environment allows the improvement of the model performance. In Chapter 5, we presented a LSTM neural network model combining car-following and lane-changing on road networks. In this work, we proposed several models with different input designs to investigate the best one. The derived model has shown a good performance on both predicting the longitudinal speed and the lateral position. Moreover, the obtained results show that this model has a better performance than the classical physical model IDM in terms of the accuracy of replicating the car-following behavior. We experimented firstly the proposed models on the NGSIM 101 data-set, and then on the

Perspectives

HighD 101 data-set. We have observed that the use of relative variables between vehicles as inputs (input design 2) gives the best result compared to the other input designs. All our tested models have significant performances in the accuracy of the prediction of longitudinal speed and lateral position simultaneously.

At last, in Chapter 6, we implemented the proposed models with different combinations and some adaptations in SUMO, for simulating the whole traffic, where all the private cars are controlled by our models, while other kinds of vehicles including trucks and motorcycles are controlled by the default model in SUMO. The objective in this part was to compare the models and see the performance of traffic simulation in both the NGSIM 101 and the HighD data-sets. Traffic simulation with our proposed car-following model using neural networks, and with our lane changing model using reinforcement learning gives the best results in simulating the traffic of the NGSIM dataset. The models using artificial intelligence provide a better performance, even in the simulation of all the cars in the traffic for the NGSIM dataset. However, as indicated in section 6.1, there exists large differences between the NGSIM data-set and the HighD data-set. The simulation using the proposed models calibrated with the NGSIM dataset is not so good for replicating the HighD traffic. The NGSIM-calibrated IDM model is less sensitive to data, and it can be used to simulate unknown traffic like the HighD one. However, it requires a re-calibration with the maximum vehicle speed in traffic.

Perspectives

Modelling and simulation of realistic traffic is an essentially difficult problem. In this thesis, we investigated and proposed some microscopic traffic models, calibrated and validated with the NGSIM 101 and the HighD data-sets, and tested in numerical simulation with SUMO. Observation and analysis of real traffic data being the first work for the traffic modelling, the acquisition of big amounts of traffic data with different traffic conditions and scenarios, is to be considered in order to improve the efficiency of the models. It is then important to also improve the process of traffic data collection, under different traffic conditions, different networks, different driver profiles, etc. We note that the NGSIM traffic data have been collected with NGVIDEO (NGSIM, 2020), a customized software application developed for extracting the vehicle trajectory data from the video. The techniques of extracting vehicle trajectories from video are available by some methods using neural network models, consisting of camera calibration methods, and multi-objects tracking methods. We have tested some algorithms for automatic extraction of vehicle trajectories from videos, using a combination of an objectdetection algorithm, Yolo5 (Jocher, 2020) and an object-tracking algorithm, DeepSort [START_REF] Wojke | Simple online and realtime tracking with a deep association metric[END_REF], with the code referenced by Mikel Brostrom [START_REF] Brostrom | Yolov5,DeepSort with PyTorch[END_REF]. This application of extracting vehicle trajectory from video is presented in the Appendix C. Using this kind of methods, we can succeed extract more traffic data providing vehicle trajectories

Conclusion and Perspectives

data from video in order to enrich the traffic data-set with different traffic conditions, such as fluid dense, congested, full congested. Furthermore, one of the important questions for the developed models is: are they able to reproduce a traffic which is very different from the one used for their calibration ? (The NGSIM 101 data-set includes congested traffic where many driver interactions are present, while, the HighD data-set is recorded from a less congested and lighter traffic, where the interactions between drivers are not very frequent). Actually, in Chapter 6 we have seen that, for physical models like the IDM model, the response to the question can be positive, even though some macroscopic calibration with the new data-set is sometimes needed (calibration of the maximum speed in our case). We have also seen that machine learning models are in general more dependent on the data than the physical ones, which makes the reproduction of other traffics more difficult. The perspectives in this direction is to develop mixed models (i.e. physical and data-based) able to benefit from the advantages of both physical and data-based models, without suffering from their disadvantages. The question is then to find the good mix ! Let us now give some perspectives for the developed models. First, regarding our proposed car-collision generation method, we define three different driver profiles. Then the car-collision generation method is applied artificially by increasing the proportion of aggressive and inattentive drivers in traffic. This work is only an experiment based on the NGSIM 101 data-set. An interesting work would be to test the method with another data-set (the HighD one for example), derive different driver profiles, and try to generate car-collisions by numerical simulations. For our proposed lane change model using Q-learning and online calibration of the NGSIM data-set, we have obtained some good performances. However, as well known, with the development of reinforcement learning, deep reinforcement learning, allowing continuous approximations would further enhance model performance, and it provides the possibility to consider more variables in the input of the model. Some new deep reinforcement learning algorithms have been developed recently, such as the Deep Double Q-learning [START_REF] Van Hasselt | Deep reinforcement learning with double q-learning[END_REF], A3C [START_REF] Mnih | Asynchronous methods for deep reinforcement learning[END_REF], DDPG (P. [START_REF] Wang | Continuous Control for Automated Lane Change Behavior Based on Deep Deterministic Policy Gradient Algorithm[END_REF] and PPO [START_REF] Ye | Automated Lane Change Strategy using Proximal Policy Optimization-based Deep Reinforcement Learning[END_REF]. It is then interesting to test such new algorithms for modelling lane change for vehicular traffic. In the other side, for the model using LSTM neural network, we found that the inputs using relative variables between the target vehicle and its surroundings vehicles can improve the model prediction accuracy of target vehicle's speed and lateral position. It would be interesting to consider these variables in the model of lane changing using reinforcement learning. Finally, for the LSTM model, it performs well in replicating real human driving behavior, but the structure of the model, which requires consideration of 50 sequences in the vehicle history trajectory, makes the model difficult to implement in a traffic simulator, which also poses a challenge in terms of computation time. A simpler model structure with similar performance needs to be considered in the future.

C.3 Yolo5 + DeepSort for multi-object tracking

In multi-object tracking (MOT) problem, tracking-by-detection is a popular method where objects are firstly detected in each frame and represented as bounding boxes, then by the result of object detection, tracking is the assignment of the same objects from the previous frame to the current frame. In other words, we would like to detect all objects in the video, and during the video, one object need have only one identification from its first appearance to its disappearance in the video.

The object detection algorithms using deep learning can be classified into two groups, classification based algorithms and regression-based algorithms. You only look once (YOLO) is a state-of-the-art, real-time object detection algorithm and one of regression-based algorithms for predicting the classes and bounding boxes for the entire image at once. YOLO has already been evolved from YOLO [START_REF] Redmon | You only look once: Unified, real-time object detection[END_REF], YOLOV2 [START_REF] Redmon | YOLO9000: better, faster, stronger[END_REF], YOLOV3 (Redmon and Farhadi, 2018), YOLOV4 (Bochkovskiy, C.-Y. Wang, and Liao, 2020), to the most recent YOLOV5 (Jocher, 2020), which is the most performant and fastest version.

Simple Online and Realtime Tracking (SORT) [START_REF] Bewley | Simple online and realtime tracking[END_REF] is an efficient approach to multiple object tracking with a focus on simple, effective algorithms. In SORT, a Kalman filter is used to firstly give the estimation and prediction of objects in the current frame by knowing the previous positions. This prediction information can be used for the correction of the position given by the detection algorithm. Then, in assigning new detections in the current frame to existing targets observed in previous frames, each target's bounding box geometry is estimated by predicting its new location in the current frame. The assignment cost matrix is then computed as the intersection-over-union (IOU) distance between each detection and all predicted bounding boxes from the existing targets. The assignment is solved optimally using the Hungarian algorithm. The DeepSort [START_REF] Wojke | Simple online and realtime tracking with a deep association metric[END_REF] is an improved version of SORT including appearance information extracted by deep neural networks. This deep appearance descriptor is offline pre-trained on a large-scale person re-identification data-

C.5 Preliminary Results

We will give some preliminary results by our collected video. As a result, we extracted the vehicle trajectories from video and provided a .sqlite format table including all the vehicle positions by time. The vehicle trajectories are obtained in

Figure 1 . 1 -

 11 Figure 1.1 -Autonomous vehicle architecture

Figure 2 . 1 -

 21 Figure 2.1 -Four different approaches for traffic modelling and simulation

Figure 2 . 2 -

 22 Figure 2.2 -Car-following scenario description

  2.1 where examples for the values of the Gipps model parameters are given.

Figure 2 . 3 -

 23 Figure 2.3 -The considered surrounding vehicles in the LC models

Figure 2 . 4 -

 24 Figure 2.4 -An overview of LC models types

Figure 2 . 5 -

 25 Figure 2.5 -An example of CA model

2. 4 Algorithm 2 . 1 :

 421 Modelling of driving behavior by artificial intelligence epoch refers to processing the whole training set once. A neural network training algorithm can be summarized as follows : Neural network learning procedure (Stochastic Gradient Descent) 1 Data: input X, output Y, 2 Parameter : batch size = m, learning rate = α 3 for number of epochs do 4 for number of mini-batches do • sample a batch x m from the training data set • predict ŷm • compute the loss function L m = L(ŷ m , y m )

Figure 2 . 6 -

 26 Figure 2.6 -LSTM neural network architecture[START_REF] Gers | Learning to forget: Continual prediction with LSTM[END_REF]. A LSTM cell consists of 3 gates (forget gate, input gate and output gate), where sigm denotes the sigmoid function, tanh denotes the tangent function, [, ] denotes a concatenation.

Figure 2 . 7 -

 27 Figure 2.7 -Simple Actor-Critic algorithm(Zhou, 2020)

Figure 2 .

 2 Figure 2.8 -SUMO screenshot(SUMO, 2021)

Figure 2 . 9 -

 29 Figure 2.9 -Architecture of the SCANeR studio driving simulator.

  (a) NGSIM traffic camera installation (b) NGSIM 101 road section

Figure 2 .

 2 Figure 2.10 -NGSIM 101 dataset acquisition camera and road section map

Figure 2 .

 2 Figure 2.11 -NGSIM 101 dataset traffic flow (number of vehicles in traffic and traffic mean speed for each time frame) for three 15-minute periods respectively : from 7:50 a.m. to 8:05 a.m., from 8:05 a.m. to 8:15 a.m., and from 8:20 a.m. to 8:35 a.m..

Contents 3 . 1

 31 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach 3.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 3.3 Materials and Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.4 Classification of driver profiles from the NGSIM 101 dataset . . . . . . . . . 46 3.5 SUMO traffic simulator and the choice of car-following model . . . . . . . . 51 3.6 Car-following model calibration method . . . . . . . . . . . . . . . . . . . . . 53 3.7 Numerical simulation experiments setup . . . . . . . . . . . . . . . . . . . . . 56 3.8 The results for vehicular collision generation . . . . . . . . . . . . . . . . . . 58 3.9 Validation of the approach on the second 15-minute time period of data . . 66 3.10 Analysis of car-collision severity . . . . . . . . . . . . . . . . . . . . . . . . . . 71 3.11 Conclusion and perspectives . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 3 . 1 -

 31 Figure 3.1 -Steps of the approach

Figure 3 . 2 -

 32 Figure 3.2 -The distribution of the maximum THW, the mean THW, the minimum THW and the standard deviation of THW the first 15 minutes data of NGSIM 101 dataset. The unit on the x-axis is seconds.

  Type of driversIndicator Threshold 1st aggressive (group 1) mean THW ⩽ 0.82s 2nd aggressive(group 2) mean THW [0.82s, 0.94s] 1st inattentive (group 3) min THW ⩾ 1.84s 2nd inattentive (group 4) min THW[1.656s, 1.84s]

Figure 3 . 3 -

 33 Figure 3.3 -T HW (THW_mean) and min T HW (THW_min) distribution of all drivers in the first 15 mins period data, group 1 in orange, group 2 in green, group 3 in red and group 4 in purple. The remainder drivers (blue points) are considered as drivers with 'normal' driving profile.

Figure 3 . 4 -

 34 Figure 3.4 -T HW (THW_mean) and min T HW (THW_min) distribution for the selected aggressive drivers and inattentive drivers

  .18) with dV /dt: vehicle acceleration V : vehicle speed s: inter-vehicular distance ∆V : relative speed with leading vehicle a: maximum acceleration b: maximum deceleration V 0 : desired speed δ: exponential parameter of speed, we fixed it at 4 referring to[START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF] s 0 : minimal inter-vehicular distance with leading vehicle T : desired time headway (desired THW)

Figure 3 . 6 -

 36 Figure 3.6 -Simulated car-collision counts for the four experiments, where aggressive and inattentive drivers have same rate (axis-x) in traffic (based on the first 15-min data). For example, 2.5% (axis-x) means each profile (aggressive, inattentive) takes 2.5% in the traffic.

Figure 3 . 7 -

 37 Figure 3.7 -Counter curves of the number of generated car-collisions, function of the percentages of aggressive and inattentive driver profiles if the traffic, for scenario E1 on first 15-minute data. Abbreviations "ag" and "long" mean "aggressive" and "inattentive" respectively.

Figure 3 . 8 -

 38 Figure 3.8 -Distribution of involved driver profile types in simulated rear-end collisions (E1 based on first 15-minute data), veh1 indicates the leading vehicle, veh2 indicates the following vehicle. The suffix ag, long, nor are the aggressive, inattentive, normal driver profiles respectively.

Figure 3 .

 3 Figure 3.10 -Average traffic flow speed of real data (blue) and simulation (orange) in SUMO for the second 15-min data.

Figure 3 .

 3 Figure 3.11 -Simulated car-collision counts for the four experiments, where aggressive and inattentive drivers have same rate (axis-x) in traffic (based on the second 15-min data). For example, 2.5% (axis-x) means each profile (aggressive, inattentive) takes 2.5% in the traffic.

Figure 3 .

 3 Figure 3.12 -E1: Simulated car-collision counts for different percentages of aggressive and inattentive drivers based on second 15-minute data. The suffix ag, long are the aggressive, inattentive driver profile respectively.

Figure 3 .

 3 Figure3.14 -The distribution of relative speed for simulated collisions (µ indicates the mean value for each sub-figure), Abbreviations "ag" and "inatt" denote "aggressive" and "inattentive" driver profile respectively. It shows that the most collisions are occurred between inattentive-aggressive drivers, while the most severe (dangerous) collisions are produced between inattentive-inattentive drivers.

Figure 3 .

 3 Figure 3.15 -SCANeR screenshot for a collision.

Figure 3 .

 3 Figure 3.16 -The collision force by SCANeR as a function of the relative speed (Y-axis for collision force (Newton) and X-axis for relative speed of collision (km/h)).

  model for imitation of the real lane change behavior from the NGSIM dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Figure 4 . 1 -

 41 Figure 4.1 -Illustration of the lane changing model using reinforcement learning and online calibration with real data.

  s, a), S, A, discount factor γ, learning rate α 3 Repeat (for each episode): 4 while not converged do 5

Figure 4 . 2 -

 42 Figure 4.2 -The presentation of traffic environment for the control of the lane changing behavior of the target vehicle (in pink).

  .

2 :

 2 Imitation of Real Lane-Change Decisions Using Reinforcement LearningV n : target vehicle speed∆V preceding : V n -V dDis ab : inter-vehicular distance between a and b Dis ef : inter-vehicular distance between e and f

  the current lane of target vehicle. 0: in the leftest lane; 1: in the rightest lane; 2 in the other intermediate lanes; 3: out of road.

4. 3 AFigure 4 . 3 -

 343 Figure 4.3 -Vehicle trajectories of lane changing in the NGSIM 101 data-set. The point (0s,0m) is the moment when vehicle changes the index of lane.

Figure 4 . 4 -

 44 Figure 4.4 -Screenshots of SUMO simulation. The red car is the target car in two different episodes

(4. 6 )4. 4

 64 For this model, the results are shown in the Appendix B.2. The results of the evolution of the total reward of each episode are shown in Fig. B.1. The expectation of the total reward tends to converge after 40000 simulation episodes. Moreover, the crash and the out of road situations do not occur any more at the last episodes after about 41000; see Fig. B.2, where 1 means the presence of collisions or out-of-roads and 0 means no presence of the event. However, using this model, the number of lane changes for the vehicles in the last 200 episodes ranges from 1 The results to 20 times, which is shown in Figure B.3. This result seems unrealistic, since changing lane 20 times in a road of 600m is not really human-like.

Figure 4 . 5 -

 45 Figure 4.5 -Total reward evolution. Blue line is the total reward for each episode. Orange line presents the average for each 20 episodes

Figure 4 . 6 -

 46 Figure 4.6 -Crash and out of road occurrence evolution. 1 means presence of the event, 0 means no presence of the event

6

 6

Figure 4 . 7 -

 47 Figure 4.7 -Distribution of number of lane changes for the vehicles in the last 1000 episodes

Figure 4 . 8 -

 48 Figure 4.8 -Color maps for the learned policy. red: change left, green: change right, blue: stay in the current lane.

5. 4

 4 Model structurefor both longitudinal and lateral vehicle movements. We use the neural network structure in Altché and La Fortelle, 2017 as the reference network, as shown in Fig.5.1. Furthermore, we propose a second architecture using two layers of LSTM, instead of one layer; see Fig. 5.2.

Figure 5 . 1 -

 51 Figure 5.1 -M1: The reference architecture[START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF] uses a first layer of 256 LSTM cells, followed by two dense (fully connected) hidden layers of 256 and 128 neurons respectively and a final dense output layer containing as many neurons as the number of outputs.

Figure 5 . 2 -

 52 Figure 5.2 -M2: Neural network with two layers of LSTM

Figure 5 . 3 -Figure 5 . 4 -

 5354 Figure 5.3 -M N N : Neural network without LSTM

Figure 5 . 6 -

 56 Figure 5.6 -RMSE distribution on the car-speed variable from the calibration of the IDM model. The RMSE value is presented for each driver data in the first 15 minutes of the NGSIM 101 data-set.

5 . 1 )Figure 5 . 7 -

 5157 Figure 5.7 -The evolution of loss during the training for different models using different input designs.

Figure 5 . 9 -

 59 Figure 5.9 -Result of M1_input2 model for the speed and lateral position respectively for the drivers 275, 1358 in the test data-set and the driver 849 in test2 data-set.

Figure 5 .

 5 Figure 5.10 -The evolution of loss during the training for different models using different input designs by HighD dataset.

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

Figure 6 . 1 -

 61 Figure 6.1 -Screenshot of the visualisation of NGSIM 101 traffic, the figure axis-x show the distance in meter (eg. 100m, 200m) to the leftest point of the road section, and the index from 0 to 4 indicating each lane from right to left.

( a )Figure 6 . 2 -Figure 6 . 3 -Figure 6 . 5 -

 a626365 Figure 6.2 -NGSIM 101 dataset: Vehicle trajectories (longitudinal position in meter) by time (Fram ID in 0.1s)

Figure 6 . 6 -

 66 Figure 6.6 -NGSIM 101 dataset: The speed-density diagram of the whole traffic for the periods of first (NGSIM 0 in the figure), second (NGSIM 1 in the figure), third (NGSIM 3 in the figure) 15-nimute data in NGSIM 101 dataset.

( a )Figure 6 . 8 -

 a68 Figure 6.8 -The 13th sub-dataset in HighD dataset: Traffic speed evolution and traffic density evolution in each lane

Figure 6 Figure 6 .

 66 Figure 6.9 -Three subfigures Figure 6.10 -The comparison of NGSIM 101 dataset and HighD dataset : the first 15 minute period data in NGISM 101 dataset and the 13th sub-dataset in HighD dataset

Figure 6 .

 6 Figure 6.11 -The distribution of the calibrated IDM model parameters for all drivers in the first 15 minute period data in NGSIM 101 dataset

Figure 6 .

 6 Figure 6.12 -Some result examples for the IDM model calibration, each sub-figure shows the real speed in NGSIM and the predicted speed using calibrated IDM model

Figure 6 .Figure 6 .

 66 Figure 6.13 -Example of a fully connected artificial neural network

  Regard right of way at intersections• bit4: Brake hard to avoid passing a red light

Figure 6 .

 6 Figure 6.15 -Example of a controlled vehicle speed in SUMO

  6.16b the results by our proposed model. The controlled car changes only once its lane by using the default model in SUMO, meanwhile, it changes 4 times using our lane change model with reinforcement learning.

Figure 6 .

 6 Figure 6.17 -Screenshot for the simulation of NGSIM traffic

Figure 6 .

 6 Figure 6.18 -NGSIM simulations : Traffic average speed simulation for all cars controlled by different models, respectively for sumo_default model, model1, model2, model3 and model4 (see Table.6.5 for the presentation of all models)

  Figure 6.18 -NGSIM simulations : Traffic average speed simulation for all cars controlled by different models, respectively for sumo_default model, model1, model2, model3 and model4 (see Table.6.5 for the presentation of all models)

Figure 6 .

 6 Figure 6.19 -NGSIM simulations : The display of speed-density diagram for different models and the real NGSIM data

Figure 6 .

 6 Figure 6.20 -The display of speed-density diagram the real NGSIM data, and the result simulated by the model1 and model2, which are the best models regardingTable 6.6.

Figure 6 .

 6 Figure 6.21 -NGSIM simulations : Vehicles' lane changes number distribution for all models, including the NGSIM dataset

Figure 6 .

 6 Figure 6.22 -Screenshot for the simulation of HighD traffic

6. 5 Figure 6 .

 56 Figure 6.23 -HighD simulations: Simulate HighD using IDM model with parameters calibrated by NGSIM

Figure 6 . 7 -Figure 6 .

 676 Figure 6.24 -HighD simulations: Simulate HighD using all proposed models

Figure 6 .

 6 Figure 6.27 -HighD simulations : Vehicles' lane changes number distribution for all models, including the HighD dataset

C. 3

 3 Figure C.3 -Detection lines for the count of different type of vehicles

C. 5

 5 Figure C.6 -The reference points in the camera view image. The reference points in this image are indicated in the orthographic image

  Figure C.8. In Figure C.9, we show the vehicle speed distribution for different types of vehicles.

Figure C. 8 -

 8 Figure C.8 -Vehicle trajectories extraction presented in the orthographic Google Earth image

Table 2 .

 2

	1 -Parameters of the Gipps' model and typical values in different scenarios (Treiber and Kesting,
	2013b)
	Parameter's name	Highway	City
	Desired speed v 0	120 km/h 54 km/h
	reaction time T	1.1 s	1.1 s
	Minimum distance s 0	3.0 m	2.0 m
	Acceleration a	1.5 m/s2	1.5 m/s
	Deceleration b	1.0 m/s2	1.0 m/s
	time T . The speed is carried out from a desired speed with a perturbation of random human
	imperfection (µ). The Krauss model is an extension of the Gipps one, in which, the safe distance
	is taken into account to avoid car-collisions. So that the desired speed is also carried out from
	the comparison among: safe speed, maximal speed, and the maximal speed supposed using
	maximal acceleration. The safe speed is submitted to the condition of collision avoidance.

Table 2 .2 -Parameters

 2 of IDM for different vehicle types(Treiber and Kesting, 2013b)

	Parameter's name	Car	Truck
	Desired speed V desired	120 km/h 80 km/h
	Minimal Time headway T	1.5 s	1.7 s
	Minimal gap s 0	2.0 m	2.0 m
	Maximal Acceleration a	0.3 m/s2	0.3 m/s
	Maximal Deceleration b	3.0 m/s2	2.0 m/s
	Acceleration exponent δ	4	4

Table 2 .

 2

	3 -Parameters of IDM in different scenarios (Treiber and Kesting, 2013b)
	Parameter's name	Highway City traffic
	Desired speed V desired	120 km/h	54 km/h
	Minimal Time headway T	1.0 s	1.0 s
	Minimal gap s 0	2.0 m	2.0 m
	Maximal or comfortable Acceleration a 1.0 m/s2	1.0 m/s
	Maximal or comfortable Deceleration b 1.5 m/s2	1.5 m/s
	Acceleration exponent δ	4	4

Table 2 .4 -Parameters of MOBIL model

 2

Table 2 . 6

 26

-Overview of environment description in reinforcement learning based algorithms for the driving modeling.

Table 2 .

 2

7 -Comparison between SUMMIT and existing driving simulators.

[START_REF] Luo | Simulating Autonomous Driving in Massive Mixed Urban Traffic[END_REF]

Table 3 .

 3

1 -4 groups specific drivers and obtained threshold of profile definition

Table 3 .

 3 2 -Overview of calibration of car-following models.

		Car-flowing model	Optimization algorithm	Objective function	Data	Result
	(Pourabdollah, 2017)	Krauss, IDM, wiedemann	GA	Normalized MSE for energy demand	Microscopic data within 200 different trips	IDM is best to represent drivers behaviors and energy demand
			Nelder-Mead
			(NM),	Sum of		GA gives
	(Li,2016)	GHR, IDM	sequential quadratic programming	square error (SSE) for position	NGSIM data	the best result for the calibration, IDM is
			(SQP),	and velocity		better than GHR
			GA
	(Zhu,2018)	GHR, GIPPS, IDM, wiedemann FVD,	GA	RMSE velocity of spacing and	42 Chinese expressway drivers on urban	IDM is the best and more stable
	(Treiber,2014)	IDM	Levenberg -Marquardt algorithm	SSE of gaps, acceleration, and speed	NGSIM data	Calibration based on gaps error is more reliable

  .3, and the different groups of driver profiles have been described inSection 3.4.3.

	Name of experiment Aggressive drivers group	Inattentive drivers group
	Experiment 1 (E1)	2.5% 1st aggressive drivers	2.5% 1st inattentive drivers
	Experiment 2 (E2)	2.5% 2nd aggressive drivers 2.5% 1st inattentive drivers
	Experiment 3 (E3)	2.5% 1st aggressive drivers	2.5% 2nd inattentive drivers
	Experiment 4 (E4)	2.5% 2nd aggressive drivers 2.5% 2nd inattentive drivers

Table 3 . 3

 33

-Numerical simulation experiments

Table 3 .

 3

	4 -Experiment 1: Simulated collision counts for different percentages of 1st aggressive and
	1st inattentive drivers. For example, in the case that 25% for aggressive, and 25% for inattentive, the
	remaining 50% are normal drivers.
	2nd aggressive drivers
	(horizontal) 1st inattentive drivers	2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag
	(vertical)
	2.5%_inatt	0	0	1	3	4	5	7
	5%_inatt	0	1	2	3	5	7	8
	10%_inatt	4	5	5	9	13	16	17
	20%_inatt	10	11	15	22	25	35	43
	30%_inatt	12	14	17	24	32	39	52
	40%_inatt	21	21	34	36	47	53	63
	50%_inatt	27	29	37	48	61	69	80

Table 3 .

 3

	5 -Experiment 2: Simulated car-collision counts for different percentages of 2nd aggressive
	and 1st inattentive drivers.

Table 3 .

 3

	6 -Experiment 3: Simulated car-collision counts for different percentages of 1st aggressive and
	2nd inattentive drivers..
	2nd aggressive drivers
	(horizontal) 2nd inattentive drivers	2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag
	(vertical)
	2.5%_inatt	0	0	1	2	2	4	8
	5%_inatt	0	1	2	2	4	5	6
	10%_inatt	2	1	2	4	6	10	14
	20%_inatt	4	5	8	14	14	16	30
	30%_inatt	8	12	12	16	28	30	42
	40%_inatt	16	16	22	24	28	34	52
	50%_inatt	25	20	26	38	42	52	58

Table 3 . 7

 37

-Experiment 4: Simulated car-collision counts for different percentages of 2nd aggressive and 2nd inattentive drivers.

  4201, β 2 = 1.3602, and β 3 = -1.4377, with RMSE = 4.864; Let us note that another non-linear regression has been tested in section A.3.2 of Appendix A.3. However, the results are not better than the ones obtained by the regression(3.23).

Table 3 .

 3

	1st aggressive drivers
	(horizontal) 1st inattentive drivers	2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag
	(vertical)
	2.5%_inatt	0.0	2.0	4.0	9.0	11.0	17.0	20.0
	5%_inatt	3.0	5.0	7.0	12.0	15.0	22.0	26.0
	10%_inatt	11.0	14.0	16.0	21.0	30.0	34.0	39.0
	20%_inatt	25.0	28.0	34.0	42.0	50.0	65.0	70.0
	30%_inatt	34.0	39.0	45.0	51.0	56.0	68.0	79.0
	40%_inatt	47.0	52.0	57.0	70.0	79.0	91.0	105.0
	50%_inatt	58.0	60.0	68.0	79.0	82.0	102.0	115.0
	2nd aggressive drivers
	(horizontal) 1st inattentive drivers	2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag
	(vertical)
	2.5%_inatt	0.0	1.0	2.0	5.0	7.0	12.0	16.0
	5%_inatt	4.0	5.0	6.0	9.0	13.0	15.0	19.0
	10%_inatt	9.0	10.0	12.0	16.0	18.0	24.0	29.0
	20%_inatt	22.0	25.0	26.0	32.0	37.0	41.0	44.0
	30%_inatt	38.0	37.0	37.0	44.0	45.0	55.0	61.0
	40%_inatt	44.0	48.0	52.0	56.0	61.0	67.0	78.0
	50%_inatt	58.0	59.0	60.0	68.0	84.0	83.0	88.0

9 -Experiment 1: Simulated car-collision counts by varying respectively the percentage of 1st aggressive and 1st inattentive drivers for second 15-min period.

Table 3 .

 3

10 -Experiment 2: Simulated car-collision counts by varying respectively the percentage of 2nd aggressive and 1st inattentive drivers over the second 15-minute period.

Table 3 .

 3 11 -Experiment 3: Simulated car-collision counts by varying respectively the percentage of 1st aggressive and 2nd inattentive drivers over the second 15-min period.

	2nd aggressive drivers
	(horizontal) 2nd inattentive drivers	2.5%_ag 5%_ag 10%_ag 20%_ag 30%_ag 40%_ag 50%_ag
	(vertical)
	2.5%_inatt	0.0	1.0	2.0	7.0	9.0	10.0	15.0
	5%_inatt	2.0	4.0	6.0	8.0	11.0	16.0	16.0
	10%_inatt	5.0	6.0	7.0	10.0	12.0	17.0	19.0
	20%_inatt	13.0	15.0	18.0	20.0	22.0	27.0	30.0
	30%_inatt	20.0	22.0	23.0	26.0	31.0	35.0	38.0
	40%_inatt	28.0	28.0	31.0	32.0	37.0	42.0	43.0
	50%_inatt	30.0	32.0	32.0	37.0	40.0	45.0	46.0

Table 3 .

 3

12 -Experiment 4: Simulated collisions number by varying respectively the percentage of the 2nd aggressive drivers and the 2nd inattentive drivers of second 15-min data.

Table 4 .

 4 1 -Quantile values in the NGSIM data for the first 6 state variables.

	Variables	20%	40%	60%	80%
	V n (m/s)	8,43	10,75 12,51 15,08
	∆V preceding (m/s) -1.37 -0.5	0,06	0,94
	Dis ab (m)	6,61	17,04 24,11 33,84
	Dis ef (m)	10,65 19,23 26,31 36,78
	∆V ab (m/s)	-5,88 -1,86 -0,432 0,84
	∆V ef (m/s)	-3,49 -0,57 0,68	2,15

Table 4 .

 4

	2 -Model Configuration for each type vehicle
	Behavior	Target vehicle	Traffic vehicles
	CF model	IDM	IDM
	LC decision	Q-learning	SUMO default model
	Application of LC	1 second	1 second

of Real Lane-Change Decisions Using Reinforcement LearningTable 4 .

 4 3 -Configuration of Hyper-parameters of Q-learning

	Parameter name	Value
	ε decaying parameter (d)	3000
	Discount factor γ	0.55
	Learning-rate α	0.05

Table 4 . 4 -

 44 Comparison of the real and simulated states and actions

Table 5

 5

		.2.
	Parameter name	Value
	batch size	128
	number of epochs	20
	learning rate	0.01

Table 5 .

 5 2 -Configuration of some Hyper-parameters

Table 5 .

 5 4).

	Mean-driver-RMSE	Train Speed (m/s) lateral Position (m) Speed (m/s) lateral Position (m) Speed (m/s) lateral Position (m) Test1 Test2
	M1_input1	0.071583	0.044184	0.07242	0.043247	0.06659	0.046756
	M2_input1	0.037969	0.037736	0.037002	0.039664	0.040165	0.042509
	M1_input2	0.010865	0.016171	0.010465	0.016323	0.011431	0.015859
	M2_input2	0.033533	0.019365	0.031219	0.019708	0.036982	0.020328
	M1_input3	0.209486	0.038016	0.189182	0.034586	0.187857	0.035655
	M2_input3	0.034134	0.021585	0.032075	0.022203	0.035832	0.021433
	Table 5.4 -HIGHD dataset : Mean RMSE for each different model using different input design and for
	training, test1, test2 respectively.

Table 6 .

 6

1 -Calibration results for all drivers of the first 15-minute period data in NGSIM dataset.

Table 6 .

 6 4 -Mean RMSE for the different models with different input designs, and for training, test1 and test2 respectively.

	Mean-driver-RMSE	Train Speed (m/s) lateral Position (m) Speed (m/s) lateral Position (m) Speed (m/s) lateral Position (m) Test1 Test2
	M1_input1	0.2819	0.1308	0.27	0.14	0.294	0.1389
	M2_input1	0.1524	0.1608	0.1767	0.167	0.1964	0.1709
	M1_input2	0.056	0.024	0.05	0.024	0.051	0.023
	M2_input2	0.0686	0.0848	0.0688	0.0752	0.0695	0.0669
	M1_input3	0.4188	0.1185	0.4356	0.1353	0.4155	0.1494
	M2_input3	0.2465	0.1331	0.3248	0.1305	0.3021	0.1396
	M N N _input2	0.2495	0.1266	0.2565	0.1229	0.2749	0.1209
	M N N 2 _input2	0.1423	0.1338	0.1389	0.16415	0.1503	0.1802
	CF by classical NN for SUMO 0.09		0.0760		0.0764

Table 6 .

 6 6 -NGSIM simulations : RMSE between traffic average speed in simulation for all cars controlled respectively by sumo_default model, model1, model2, model3 and model4 (see Table.6.5 for the presentation of all models), with real NGSIM traffic average speed

Table 6

 6

.6.

4.4 The results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 4.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

Proceedings of ICITS002 2021: 15. International Conference on Intelligent Transportation Systems December 27-28, 2021 in Vienna, Austria.

Acknowledgements

I am deeply grateful my thesis director Nadir FARHI, and my academic co-supervisor Zoi CHRISTOFOROU, for their guidance, support, and patience in helping me with all my work. I am also very grateful to my industrial supervisors, Nadia HADDADOU, Yves TOURBIER, Rodolphe GELIN, for their insightful advice during these three years.

The results for vehicular collision generation

Figure 3.9 -E1: Non-linear regression of the number of simulated collisions. Abbreviations "ag" and "long" indicate "aggressive" and "inattentive" driver profiles respectively.

Analysis of driver behavior and inter-vehicular collision : a data-based traffic modeling and simulation approach

Figure 3.13 -Second 15-mints data E1: Non-linear regression of the number of simulated collisions. Abbreviations "ag" and "long" denote "aggressive" and "inattentive" driver profile respectively.

Imitation of Real Lane-Change Decisions Using Reinforcement Learning

the preceding vehicle obstructs it, i. e. when the controlled vehicle's speed is small and the relative speed with the preceding vehicle (∆V preceding ) is big (see cases (0,4), (1,4), (2,4) in . Lane changes are less observed for high speeds of the controlled vehicle.

From , we observe that the agent tends to change to the left lane for low values of ∆V ef _n , and to the right lane for high values of ∆V ef _n . In other terms, the agent decides to go to the lane with a high relative speed, i. e. where the controlled vehicle's speed will increase significantly. Moreover, it seems that there is a curve ∆V ef _n = 3 -V n , on Fig. 4.8-(B), under which the agent tends to change to the left lane, and over which it tends to stay on its current lane. We do not yet fully understood this observation. On Fig. 4.8-(C), we can see that the vehicle tends to change right when the inter-vehicular distance with respect to vehicles e and f, is sufficiently large. On Fig. 4.8-(D), it is obvious that lane changes are possible only when at least one of the inter-vehicular distances on the left and right lanes is sufficiently large. All these observations of learned strategy of lane changing are interesting to understand and to validate this lane changing model using RL.

Conclusion

In this work, we proposed a vehicular lane change model using Q-learning with on-line calibration on the real human lane change decision data. It is a data-based model, which aims to mimic the real lane change decisions taken by human drivers. We applied this study based on the NGSIM 101 data-set and used the vehicular simulator SUMO to complete the data and to cover all the state and action variable spaces. A big number of episodes have been considered for the training, during each of them the controlled vehicle (the agent) executes our Q-learning algorithm to take decisions on its lane change, where the other vehicles run under traffic models included in SUMO (car-following and lane-change). Moreover, our Q-learning algorithm permits to complete the unobserved situations, and to propose actions for the states not present in real data. Thus, about 13% additional traffic situations, not present in the real data, are created by the traffic simulation environment.

We have shown that with the approach proposed in this article we can mimic and reproduce real lane change decisions observed in traffic data, up to a rate of 95%. Finally, we have illustrated the obtained optimal policy by means of color maps, and discuss preliminary conclusions on the determinant states and thresholds on states in the decision process of lane changing. For future research, we aim at understanding more about this lane change model, and trying to test this model in other road traffic conditions.

Chapter 5

Long Short-Time Memory Neural Networks for Human Driving Behavior Modelling

In this chapter, a long short-term memory (LSTM) neural network model is proposed to replicate simultaneously car-following and lane-changing behaviors in road networks. By combining two kinds of LSTM layers and three input designs of the neural network, six variants of the LSTM model have been created. These models were trained and tested on the NGSIM 101 dataset, and the results were evaluated in terms of longitudinal speed and lateral position respectively. Then, we compared the LSTM model with a classical car-following model (the Intelligent Driving Model (IDM)) in the part of speed decision. In addition, the LSTM model is compared with a model using classical neural networks. After the comparison, the LSTM model demonstrates higher accuracy than the physical model IDM in terms of car-following behavior and displays better performance with regard to both car-following and lane-changing behavior compared to the classical neural network model. For the input variables, the authors of (X. [START_REF] Zhang | Simultaneous modeling of car-following and lane-changing behaviors using deep learning[END_REF] considered only one input variable which is the vehicle positions of the surrounding vehicles. In order to better understand the effect of different input variables on the human driving behavior using LSTM neural network, we consider in this article different variables. Moreover, we propose three different input designs, see Tab. 6.2. The considered input variables are defined as follows.

Long Short-Time Memory

1. Input design 1: in this design, we apply all possible input variables for the vehicle movements, including its longitudinal position, lateral position, speed, acceleration and vehicle size. For the surrounding vehicles, we consider position, speed, acceleration, as well as the relative distance and speed with respect to the target vehicle. We notice that the variable time-to-collision (TTC) has also been considered as an input variable in [START_REF] Altché | An LSTM network for highway trajectory prediction[END_REF]. which we do not consider here, because we think that this information is redundant with the two variables of relative speed and distance.

The following input variables are considered for the target vehicle (red vehicle in Fig. 5.5):

• longitudinal position x target and lateral position y target

• longitudinal speed v target and longitudinal acceleration a target

• vehicle length l target and vehicle width w target

The following input variables are considered for the surrounding vehicles ((a,b,c,d,e,f) in Fig. 5.5):

• longitudinal position x i and lateral position y i Long Short-Time Memory Neural Networks for Human Driving Behavior Modelling

Highway Traffic simulation in SUMO using proposed models

The HighD dataset

We choose randomly the 13th sub-dataset to compare with the NGSIM dataset. In this subdataset, the traffic is captured on a bi-direction highway road section of length 420m. There are 3 lanes in each direction. We selected all the drivers in one direction (direction id = 2 in the data), and there are about 14 minutes registration of traffic consisting of 1317 vehicles' trajectories.

There are about 10s of trajectory for each vehicle (see in Figure 6.7 where a screenshot for the visualisation of this sub-dataset is given). In Figure 6.8a, we show the vehicle position trajectory in the HighD dataset, and we can find the vehicle trajectory is smooth without any shocks, in contrary to the NGSIM dataset (Figure 6.2). In Figure 6.8b, we show the mean traffic speed on each lane, where we can see that it evolves similarly in each lane (at a high level), where the speed does not oscillate as in the NGSIM traffic. We can also observe that on lane 2, which is the leftest one, the average speed is the highest one among the three lanes. In Figure 6.8c, we show the evolution of the vehicle-density diagram for each lane, and from this figure, we can observe that the number of vehicles is almost the same in average on the three lanes, and it ranges from 2 to about 12 in 420 meter. This vehicular density is much lower than that of the NGSIM dataset, see Figure 6.4b).

Highway Traffic simulation in SUMO using proposed models

Comparison of NGISM dataset and HighD dataset

To show the difference between the NGSIM 101 and the HighD datasets, we give in Figure 6.9a, Figure 6.9b and Figure 6.9c, a comparison between the NGSIM 101 first 15 minute period data and the 13th sub-dataset of HighD dataset, in terms of traffic mean speed, traffic density, and the speed-density diagram. The HighD traffic is not as dense as the one in the NGSIM 101 dataset; see Figure 6.9b). This can also be seen in Figure 6.9a and Figure 6.9c.

In the following sections, we simulate in SUMO, the two traffics of NGSIM and HighD with different combinations of car-following and lane-change models. In section 6.2, we first give the adaptation and the implementation of our proposed models used in Chapter 3, Chapter 4, and Chapter 5. Then, in section 6.4 and section 6.5, we give the results of simulation of the first 15 minutes period traffic in the NGSIM dataset and the traffic of 13th sub-dataset in the HighD datset.

Implementation of the models in SUMO

A.2 Calibration Result for specific drivers

The results of calibration for 4 specific groups of driver in the 15-minute data of NGSIM 101 dataset are given as follows: in

A.3 Regression of the number of simulated car-collisions for first 15 minutes data A.3.1 Linear regression

In order to find the relationship between simulates crashes number and the rate of specific profile drivers for the result shown in Table 3.4, We obtain:

A.3.2 First test of non-linear regression based on the first 15-minute data

In this subsection, we suppose the number of generated collisions based on the first 15-minute data has a non-linear relation with the percentage of each driver profile, where the formula is as below :

The result of coefficients :

where the coefficient of determination R 2 = 0.9683. RMSE (root mean square error)= 5.630

A.4 Regression result of the number of simulated car-collisions for second 15 minutes data

A.4 Regression result of the number of simulated car-collisions for second 15 minutes data

As the same, we investigated the regression function for the generated collision counts based on the second 15-minute data. By the same way as that for the first 15-minute data, we test firstly the linear regression, and then, a polynomial regression is tested, consisting of two different functions.

Linear regression :

We obtained :

, with β 0 = -10.5262, β 1 = 1.537 and β 2 = 0.807. where RMSE (root mean square error) = 6.16

Polynomial regression:

By the proposition in the Equation A.1, we find for the generated collision number, it respects to :

where RMSE (root mean square error)= 9.478.

Appendix B

Complements on Chapter 4 B.1 Introduction

In this appendix, all other implementations of lane change models using reinforcement learning are presented.

B.2 Initial model using Q learning for lane change decision modelling

In the first proposition, we consider that the motivation of lane changing behavior usually is the speeding, that the driver wants to speed up. In this context, the reward of lane changing decision is the 'Speed gain', where the agent maximizes always the speed to take a decision of lane change. As presented in the section 4.3.3, the states and actions are used the same definitions. Unlike what is presented in 4.3.6, the reward is defined as :

The result of the evolution of the total reward of each episode is shown in Fig. B.1. The expectation of the total reward tends to converge after 40000 simulation episodes. Moreover, the crash and the out of road situations do not occur any more at the last episodes after about 41000; see

Automated counting of vehicles in the traffic and extraction of vehicle trajectories : vehicle detection and vehicle-trajectory tracking from video

Historically, traffic data has been sourced from traffic cameras and sensors, as well as on-board sensors on vehicles. In section 2.7, we have presented the data-sets which we worked on for my PHD works, NGSIM and HighD. This kind of data-sets are very important for human driving behavior research, which contain vehicle trajectories, and are usually extracted from videos by object-tracking methods. The methods of object-tracking are widely studied in the computer vision domain, and have many advances using deep learning methods in the recent years. Here, we proposed and tested a pipeline for automated extraction of vehicle trajectories from videos, using a combination of an object-detection algorithm, Yolo5 (Jocher, 2020) and an objecttracking algorithm, DeepSort [START_REF] Wojke | Simple online and realtime tracking with a deep association metric[END_REF], with the code referenced by Mikel Brostrom [START_REF] Brostrom | Yolov5,DeepSort with PyTorch[END_REF]. Furthermore, once we extract the objects trajectories in the video (in each image frame), the camera calibration is necessary to find the matrix, to convert the image coordinates to real world coordinates. We used an open-source software for the camera calibration, named as T-calibration, provided by an European project funded by Horizon 2020 (Lund University, 2018). As a result, we made a complete application for counting automatically different type of vehicles, including cars, trucks, motorcycles, bicycles in the traffic and extraction of vehicle trajectories from recorded traffic videos. This application can be used for the acquisition of a large-scale traffic data-set with different traffic conditions and for further research on human driving behavior. The pipeline has four steps:

C.2 Customization of detection zone and vehicle counting line

In the application we made, the customization of detection zone and vehicle counting line is implemented to facilitate the utilisation of the application. We need to choose a detection zone in order to ignore the objects in the other zone and the counting lines can be drawn by the user to choose the counting area. Here we give an example in

Automated counting of vehicles in the traffic and extraction of vehicle trajectories : vehicle detection and vehicle-trajectory tracking from video

set. During the tracking on real time, the deep appearance information is also an input for Hungarian algorithm to optimise the objects assignment.

For this step, we used the code referenced by Mikel Brostrom [START_REF] Brostrom | Yolov5,DeepSort with PyTorch[END_REF]. The YOLOV5 has been trained using a large object detection data-set, COCO data-set [START_REF] Lin | Microsoft coco: Common objects in context[END_REF], in which, all objects we want to detect in the traffic are included. The deep appearance descriptor in DeepSort is pre-trained using MOT challenge dataset [START_REF] Leal-Taixé | Motchallenge 2015: Towards a benchmark for multi-target tracking[END_REF]. We observed that this appearance descriptor pre-trained on person data-set is effective for other type of objects we want to detect. We show the result of YOLOV5 + DeepSort for the multi-object tracking in our collected video in

C.4 Camera calibration to find the matrix for converting the coordinates in the video to real world coordinates

Once we apply the YOLOV5 + DeepSort for the tracking for whole of our collected video, the vehicle trajectories in the image can be obtained by the position of the center of the bounding box in each frame. However, to convert the objects coordinates in the image to real world image, the calibration of the camera needs to be applied to find a matrix for the transformation of image coordinates to real world coordinates. We used an open-source software for the camera calibration, named as T-calibration, provided by an European project funded by Horizon 2020 (Lund University, 2018). By this application, we can get automatically the calibration result of camera by giving the reference points in the camera view image and the orthographic image of the detection zone. We show the reference points in the camera image in

C.6 Perspectives

This work is tested and implemented with another PHD student, Yeltsin VALERO, working on Microscopic modelling of Personal Displacement Equipment (PDE) under mixed traffic and shared space conditions, with the objective to prepare the data-set for further using in the projects for studying the human driving behavior in mixed traffic.

For this project, one challenge for measuring the performance of our method is the lack of ground-truth data. In future work, we will evaluate the learned models using human annotations as ground-truth. In the other hand, change the view of camera is important for better detection. We can use the view like in the HighD and NGSIM data-set. That will facilitate the camera calibration task and reduce the calibration error to make the transformation of vehicle coordinates in image to real world coordinates.

List of Figures

List of Figures

List of Algorithms

List of References

List of References