Deep Reinforcement Learning Based Decision Making to Improve Multi-Lane Highway Traffic Under Uncertain Driving Environments

Keywords: Automated driving, decision making, uncertain driving environment, reinforcement learning, multi-lane traffic, integrated risk assessment

Dealing with environmental uncertainties and improving vehicle decision-making ability are among the most important issues for autonomous highway driving. To this end, we propose a vehicle decision-making framework based on heuristic reinforcement learning while considering environmental uncertainties. In particular, a future integrated risk assessment model is used to solve the environmental uncertainty. First, this paper predicts the uncertain environment based on a long short-term memory model, including predicting driving intention and vehicle motion trajectory. A future integrated risk assessment model in uncertain environments is proposed in the reinforcement learning framework. Moreover, to solve the exploration and exploitation dilemma in reinforcement learning, a heuristic decaying state entropy (HDSE) deep reinforcement learning algorithm is proposed, which effectively shortens the training time period of the agent. A path tracking model and a rule-based vehicle decision model are built to deal with the path tracking problem and the interaction decision problem of surrounding vehicles. Finally, the vehicle decision framework is validated in both low-density and high-density traffic scenarios. The obtained results show that the proposed vehicle decision-making framework based on HDSE deep reinforcement learning considering environmental uncertainty improves the traffic efficiency while ensuring the vehicle safety.

Introduction

Over the past two decades, autonomous vehicles have rapidly developed and attracted much attention in academic and industry research [START_REF] Dong | The context of tactical control of lanechanging autonomous vehicles and connectivity range assessment[END_REF]. About 94% of traffic accidents are caused by drowsiness, distraction, and poor decision-making, which can be solved by autonomous driving [START_REF] Alvaro | Driver education.: Enhancing knowledge of sleep, fatigue and risky behaviour to improve decision making in young drivers[END_REF]. An autonomous driving system can be divided into four key parts: perception, decision-making, planning and control [START_REF] Li | Risk assessment based collision avoidance decision-making for autonomous vehicles in multiscenarios[END_REF]. A decision-making strategy is regarded as the human brain, which is crucial for autonomous driving [START_REF] Kiran | Deep Reinforcement Learning for Autonomous Driving[END_REF]. Under uncertain environments, an autonomous vehicle needs to understand the driving intentions of surrounding vehicles to cooperate with them via reasonable driving behaviors. The corresponding behavior decision-making ability largely determines the driving performance of autonomous vehicles.

Vehicle decision-making methods can generally be divided into rule-based and data-driven methods. The most famous rule-based decision-making method can be found in the defense advanced research projects agency (DARPA) Urban Challenge, where the Junior team won the 2005 DARPA championship with a finite state machine [START_REF] Montemerlo | The DARPA Urban Challenge[END_REF]; in 2007, the Knigh Rider team used a hierarchical state machine to solve parking task [START_REF] Patz | A practical approach to robotic design for the DARPA Urban Challenge[END_REF]. However, these methods ignore the dynamics and uncertainty of the environment. Moreover, the traditional rule-based systems cannot solve the decisionmaking problems with many characteristics of driving scenes.

Recently, with the rapid developments of artificial intelligence, using machine learning to solve the vehicle decision-making problem has become a hotspot research topic. The most common one is reinforcement learning, which is different from supervised learning and unsupervised learning. In reinforcement learning, the agent continuously interacts with the environment to improve its performance on specified tasks, so this learning scheme is very similar to the learning of human beings [START_REF] Jiao | Real-world ride-hailing vehicle repositioning using deep reinforcement learning[END_REF]. Since reinforcement learning has the advantages of active learning and the ability to react to changes in the environment, reinforcement learning-based vehicle decision making has applications in highways [START_REF] Xu | A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways[END_REF], roundabouts [START_REF] Zhang | Adaptive Decision-Making for Automated Vehicles Under Roundabout Scenarios Using Optimization Embedded Reinforcement Learning[END_REF], highway exits [START_REF] Cao | Highway Exiting Planner for Automated Vehicles Using Reinforcement Learning[END_REF], on-ramp merging [START_REF] Liu | An Efficient On-Ramp Merging Strategy for Connected and Automated Vehicles in Multi-Lane Traffic[END_REF], and other scenarios.

Image-based solutions have been used to obtain the information about the vehicle's surrounding environment [START_REF] Wang | Harmonious Lane Changing via Deep Reinforcement Learning[END_REF][START_REF] Chen | Stabilization Approaches for Reinforcement Learning-Based End-to-End Autonomous Driving[END_REF]. However, these observation models (image, radar, or grid-based) are all unstructured datasets and need to be processed by convolutional neural networks. This approach requires a large number of samples and time convergence [START_REF] Li | Reinforcement Learning and Deep Learning Based Lateral Control for Autonomous Driving[END_REF], resulting in a long learning process [START_REF] Jaritz | End-to-End Race Driving with Deep Reinforcement Learning[END_REF]. Therefore, it is necessary to preprocess the data and use structured datasets to speed up the convergence. For example, Fu et al. proposed an emergency braking strategy based on the deep deterministic policy gradient, which considered efficiency, accuracy, and passengers' comfort in the reward function [START_REF] Fu | A Decision-Making Strategy for Vehicle Autonomous Braking in Emergency via Deep Reinforcement Learning[END_REF]. Hoel et al. [START_REF] Hoel | Combining Planning and Deep Reinforcement Learning in Tactical Decision Making for Autonomous Driving[END_REF] extended the AlphaGo Zero algorithm to the continuous state-space domain and applied it to the field of autonomous driving decision-making. In [START_REF] Liao | Decision-Making Strategy on Highway for Autonomous Vehicles Using Deep Reinforcement Learning[END_REF], the dueling deep Q-network algorithm was used to solve the decision-making problem under the highway driving conditions.

There are multiple uncertainties in the driving process of autonomous vehicles. Most papers use the partially observable Markov decision process (POMDP) to solve the vehicle decision-making problem under an uncertain environment. Hubmann et al. [START_REF] Hubmann | Automated Driving in Uncertain Environments.: Planning With Interaction and Uncertain Maneuver Prediction[END_REF] adopted POMDP to solve the vehicle decision-making problem at the intersection by considering the uncertainty of the intention of surrounding vehicles and the uncertainty of sensors. However, the designed action space is only longitudinal action, which is not extended to the lateral motion of the vehicle, so it is not suitable for highway decision-making. Zhang et al. [START_REF] Pouya | Expandable-Partially Observable Markov Decision-Process Framework for Modeling and Analysis of Autonomous Vehicle Behavior[END_REF] combined the POMDP model with heuristics and proposed a probabilistic modeling framework to explain environmental uncertainty. Galceran et al. [START_REF] Galceran | Multipolicy decisionmaking for autonomous driving via changepoint-based behavior prediction.: Theory and experiment[END_REF] proposed a multi-strategy decision framework based on POMPD, which models the vehicle behavior of the self-vehicle and surrounding vehicles as a set of discrete closed-loop policies. However, due to insufficient samples of intentions and inaccurate initial behavior predictions, the risk-aware objective may not reflect the policy evaluation [START_REF] Mehta | Fast discovery of influential outcomes for risk-aware MPDM[END_REF]. POMPD provides a general mathematical framework to solve uncertainty problems. However, due to the "curse of dimensionality" issue, POMDP is challenging in terms of computation [START_REF] Ye | DESPOT.: online POMDP planning with regularization[END_REF]. Although various simplification and discretization methods have been adopted, the existing methods are not efficient enough to deal with highly dynamic driving scenarios [START_REF] Zhang | Efficient Uncertainty-aware Decisionmaking for Automated Driving Using Guided Branching[END_REF].

Autonomous vehicles are faced with many unavoidable uncertainties, such as the uncertainty of surrounding vehicles' behavior, the uncertainty of surrounding vehicles' motion, and the uncertainty of interaction between self-vehicle and surrounding vehicles [START_REF] Okumura | Challenges in Perception and Decision Making for Intelligent Automotive Vehicles.: A Case Study[END_REF]. These complex and uncertain factors are unavoidable and widespread in highway environments, bringing severe challenges to autonomous vehicles' behavior decision-making system. An unreasonable decision-making behavior lead to significant traffic accidents.

The vehicle observation space describes the information obtained by the autonomous vehicle. It generally includes the state of the vehicle (position, speed and heading angle), the topology information, and other traffic participants, e.g., surrounding vehicles, obstacles [START_REF] Aradi | Survey of Deep Reinforcement Learning for Motion Planning of Autonomous Vehicles[END_REF]. The selection of the vehicle state space is critical. x y v and s heading are the longitudinal displacement, lateral displacement, speed, and heading angle of the self-vehicle, respectively. Moreover, LK represents the lane keeping, LC is the change to the left lane, RC is the change to the right lane, FD is to drive faster, SD is to drive slower, EB represents the emergency braking, CL represents the action of changing the lane, and NONE corresponds to maintain the speed and the lane.

From Table 1, it can be seen that the state of the self-vehicle and the surrounding vehicles are chosen as the observation space in most of related works. Motivated by the above issues, we propose a new decision-making framework for autonomous driving considering the uncertainty of driving environments. Moreover, a new future integrated risk assessment state model is proposed as the observation space. How to obtain an efficient exploration is one of the main challenges in reinforcement learning [START_REF] Sledge | Balancing exploration and exploitation in reinforcement learning using a value of information criterion[END_REF]. Moreover, the essence of the exploration and exploitation dilemma is how to make the algorithm achieve better convergence in a limited time. Therefore, we propose a novel exploration method to balance the exploration and the exploitation. Specifically, the main contributions of this paper are summarized as follows.

  : , L De L sc et K i C r 1)
Differently from [START_REF] Liao | Decision-Making Strategy on Highway for Autonomous Vehicles Using Deep Reinforcement Learning[END_REF][START_REF] Mo | Safe Reinforcement Learning for Autonomous Vehicle Using Monte Carlo Tree Search[END_REF], the design of the new decisionmaking framework for autonomous driving takes into account the environmental uncertainty in terms of driving intention of surrounding vehicles, future driving risk in each lane, and interaction between surrounding vehicles and self-vehicle.

2) A heuristic decaying state entropy (HDSE) deep reinforcement learning algorithm is proposed to solve the well-known exploration and exploitation dilemma.

3) A future integrated risk assessment model is developed to effectively deal with environmental uncertainties. This paper is organized as follows. In Section 2, the algorithm framework is presented. Moreover, the uncertain driving environment is discussed, including the prediction of lane change intention, the vehicle motion trajectory, and the future integrated risk assessment model. In Section 3, the problem of vehicle decision-making in an uncertain environment is proposed, and the HDSE deep reinforcement learning algorithm is designed to solve the Markov problem. Section 4 shows the relevant results of a series of case studies. Section 5 concludes the paper.

System Framework and Prediction of Uncertain Driving Environment

System Framework

The proposed vehicle decision-making framework is illustrated in Fig. 1, which can be divided into uncertain environment prediction and vehicle decision system. The driving intention identification module identifies the lanechanging intention and the vehicle motion trajectory of the surrounding vehicles based on long short-term memory (LSTM). The future integrated risk assessment function and the future target lane of surrounding vehicles are calculated using a prediction of the driving environment and the integrated risk assessment function, see Section 2.4. The result is used as the input of the vehicle decision system together with the current state of the vehicle and the surrounding vehicles. In addition, the data obtained from the interaction with the environment is stored in the experience replay memory and are extracted to input the reinforcement learning model during the training process. The vehicle decision system adopts the HDSE deep reinforcement learning algorithm to learn the optimal driving behavior by interacting with the driving environment. In particular, the optimal action is heuristically generated by RULE a , which is determined by the intelligent driver model (IDM) in [START_REF] Treiber | Congested traffic states in empirical observations and microscopic simulations[END_REF] and minimizes the overall braking induced by lane changes (MOBIL) [START_REF] Kesting | General Lane-Changing Model MOBIL for Car-Following Models[END_REF]. By this way, the vehicle decision system obtains optimal decision actions considering environment uncertainties, which include different driving actions, e.g., change to the left lane, change to the right lane, drive faster, drive slower, maintain speed and lane.

Lane Change Intention Prediction

Driving intent prediction is one of the core technologies of intelligent vehicles, which can infer the future intent of the driver to predict the likelihood of a possible collision and also take measures to avoid accidents in advance [START_REF] Li | Analysis of Driver's Steering Behavior for Lane Change Prediction[END_REF]. This paper uses data from the Next Generation Simulation (NGSIM) program conducted by the Federal Highway Administration for lane change intention prediction and driving trajectory prediction [START_REF] Alexiadis | The next generation simulation program[END_REF]. The NGSIM project was undertaken to study microscopic vehicle driving behavior and included the US-101 and I-80 highway datasets. For this paper, 543 lane change scenarios and 870 lane-keeping scenarios were extracted to construct the dataset. We used 11 data items from the NGSIM data, including dataset id, Vehicle id, Frame index, Local X, Local Y, Lane id, v_length, v_Width, v_Class, Velocity, Acceleration. Moreover, the dataset is divided into a training set and a test set with a ratio of 8:2. Each dataset has a trajectory duration of 8 s, and here we use 4 s of track history and a 4 s prediction horizon. The inputs of both the lane change intention prediction and the vehicle motion trajectory prediction can be expressed as ( )

( ) ( ) ( ) ep , , , 1,
t t t si t T T T T  = = - -  I S E (1)
where 2. The structure of both the FCNN network and the LSTM network have the same number of network nodes. The structure of the LSTM network is shown in Fig. 2, where Relu and Sigmoid are the activation functions, and the FC means the fully connected layers. The Area Under Curve (AUC) is the area under the Receiver Operating Characteristic (ROC) curve, which is the evaluation index of the two-class classification. F1-score is the harmonic mean of precision and recall. As can be seen from Table 2, the accuracy of the LSTM method is higher than that of the neural-network-based method. Moreover, the accuracy and loss evolutions during the training process are shown in Fig. 3.

  ( ) ( ) ( ) ( ) ( ) e , ,

Vehicle Motion Trajectory Prediction

The vehicle motion trajectory is generally considered a time series. Hence, we built a vehicle motion trajectory prediction based on the LSTM model which is suitable for time series prediction. The network structure of the model is shown in Fig. 4, where R-dropout 0.2 means that the dropout probability. With the particular structure of LSTM, the cell states are transferred to the subsequent training with minimal time loss, which can avoid the gradual decay of the leading output. The output of the trajectory prediction module is defined as ( )

( ) ( ) ( ) , , 1, 2,
,

ego eg t o tt f x y t T T T T  = = + + +  O (2)
where f T is the predicted time horizon. After the vehicle motion trajectory prediction training is completed, we randomly selected 10 different initial positions and vehicle speeds for testing from the test set, and the results are shown in Table 3. As shown in Table 3, the proposed model can effectively predict the vehicle motion trajectory. The mean absolute error (MAE) of the lateral position is significantly better than that of the longitudinal position, and the prediction accuracy decreases as the prediction time horizon increases. The prediction results of both root mean square error (RMSE) and MAE can meet the requirements of motion trajectory prediction.

( ) ( ) ( ) (3)

Heuristic Decaying State Entropy Deep Reinforcement Learning Algorithm

This section proposes a vehicle decision-making algorithm based on HDSE deep reinforcement learning in uncertain environments. First, we establish commonly used vehicle decision models and uncertainty vehicle decision models that consider the future vehicle uncertainty based on the future integrated risk assessment function [START_REF] Xu | A Reinforcement Learning Approach to Autonomous Decision Making of Intelligent Vehicles on Highways[END_REF]. Then, based on these models, the decision-making problem are solved by the HDSE deep reinforcement learning algorithm.

Vehicle Decision Model

We describe the vehicle decision problem as a Markov decision process, which is generally defined as a tuple (S, A, P, R, γ). This tuple is composed of a state space S, an action space A, a state transition probability P, a reward function R, and a discount factor γ which is used to calculate the cumulative reward of the whole process.

3) Action Space: To improve the convergence speed, the action space of the reinforcement learning output is the vehicle's behavioral decisions. Then, the vehicle's action is defined as the action space that solves all highway driving tasks, that is

  , , , , A LC RC FD SD NONE = (11)
To ensure the predictability of the vehicle behavior decision, the interval between two behavior decisions of the vehicle will not be too short, the execution interval is 1s. 4) Reward Function: Complex reward functions, e.g., to guarantee the distance between the ego-vehicle and the vehicle ahead, may reduce the agent's exploration ability [START_REF] Kiran | Deep Reinforcement Learning for Autonomous Driving[END_REF].

Hence, the goal here is to use straightforward reward functions to solve the vehicle decision-making problem. To this end, we construct the following rewards to guarantee a high-speed driving with collision avoidance: ( , ) ( , ) ( , )

speed collision R s a R s a R s a =+ (12) min max min ( , ) speed speed vv R s a C vv - = - (13)
vehicle collision ( , ) 0 otherwise

collision collision C R s a  =   (14)
where s and a represent the current state and action, speed C is the reward constant for vehicle speed in the desired range, collision C is the penalty factor for a vehicle collision.

Heuristic Decaying State Entropy Dueling Double DQN

The agent interacts and explores with the environment in realtime and directly learns from the obtained experiential data to eventually maximize the cumulative returns or to achieve a specific goal. However, the high dimensionality of the input features and the computational complexity bring challenges to traditional reinforcement learning methods [START_REF] Arulkumaran | Deep Reinforcement Learning.: A Brief Survey[END_REF]. Mnih In reinforcement learning, the agent's goal is to maximize the expectation of the cumulative reward. The value function uses the expectation of return to evaluate the agent's performance under the current state or a specific state and action. To solve the reinforcement learning problem, we define the following optimal state action-value function ( )

, tt Q s a 
under optimal strategy π as the maximum expected return obtained for a specific action in a specific state:

( ) ( ) , max , a Q s a r Q s a       =+  (15)
where s, a and r are the state, action and reward of each period respectively, 𝑠 ′ is the state of the next period, 𝑎 ′ is the action of the next period,  is the discount factor, is the expected value. The optimal strategy is obtained by maximizing the optimal action-value function as

( ) ( ) arg max , a s Q s a  = (16)
The observation space is generally a continuous highdimensional state space for the autonomous vehicle decision problem. Hence, we use a parameterized continuous function

Q s a r s L y Q s a    =-  (17)
where

( ) max , , Q a y r Q s a  -   =+
, the parameter of the target Q-network  -is copied from the Q-network at a certain number of iterations during the training process. However, the target Q-network ,arg max , , , .

DDQN a y r Q s Q s a    -  =+  (18)
Here, the actions are selected based on the evaluate Qnetwork, and then the Q-value is determined based on the target Q-network. The double DQN uses two parameters  and  -, where the parameter 𝜃 is used to select the action, and the parameter  -is used to evaluate the state-action value function. The new loss function can be defined from [START_REF] Liao | Decision-Making Strategy on Highway for Autonomous Vehicles Using Deep Reinforcement Learning[END_REF] as

( ) ( ) ( ) ,, 2 , ( ) , ,
DDQN s a r s L y Q s a      =-  (19)
Minimizing the loss function [START_REF] Hubmann | Automated Driving in Uncertain Environments.: Planning With Interaction and Uncertain Maneuver Prediction[END_REF], we obtain the parameters of the evaluate Q-network. To estimate more accurately the Q values, we use dueling networks to divide the Q-network into two parts, i.e., the value function part and the advantage function part. The Q-value function is defined as ( , )

( ) ( , ) Q s a V s A s a =+ ( 20
)
where () Vs is the value function, ( , ) A s a is the advantage function. The detailed structure of the Q-network established in this paper is shown in Fig. 5. The network input is the future integrated risk assessment state IRAF S described in [START_REF] Cao | Highway Exiting Planner for Automated Vehicles Using Reinforcement Learning[END_REF], and the output is the state-action value function ( , ) Q s a . The traditional ε-greedy exploration has an equal probability for each exploration in the same episode, without considering that the weights of exploration should be varied for different values of ( , ) Q s a . Here, we define the action probability distribution ()

s pa based on ( , ) Q s a as ( ) ( ) exp ( , ) max ( , ) () exp ( , ) max ( , ) aA s bA aA Q s a Q s a pa Q s b Q s a    - = -  ( 21
)
where A is the action space. The entropy () Hs of a discrete random variable s with probability distribution ()

s pa is defined as ( ) ( ) ( ) || log . s A s aA H s p a p a  =- (22)
Differently from [START_REF] Usama | Learning-Driven Exploration for Reinforcement Learning[END_REF], to make the probability of entropy converge to 0 at the end of the training, we use the decaying method. The entropy processed by the decaying method can be expressed as ( ) For the above method, we established the HDSE reinforcement learning algorithm for dueling double DQN, to improve the agent learning efficiency and to predict the value function more accurately.

( ) ( ) 1 exp r de f f n H s H s t t    =  + - -     (

Case Studies

To validate the proposed decision-making algorithm for autonomous vehicles, we provide two different cases to analyze the characteristics of our proposed algorithm from different perspectives. In addition, comparative studies between the proposed vehicle decision-making method and surrounding vehicles decision-making methods in different scenarios are also presented.

Experiment Setting and Comparative Groups

We adopt Breadth-First Search in the graph description of the road network to generate the shortest path from the initial point to the destination and transfer it to the path tracking model. An MPC controller is used for path tracking whereas the speed tracking is performed by an PI controller.

Both the proposed algorithm and the compared algorithm are trained and evaluated in the highway-env on the python platform [START_REF] Leurent | An Environment for Autonomous Driving Decision-Making[END_REF]. The initial states of the algorithms during training and testing are random, including the initial position and velocity of the self and surrounding vehicles. And the random range of initial velocity is 23-25m/s. In order to compare the performance of different algorithms, we selected the same random seeds for testing to ensure the uniformity of the test results. After generating the random initial state, the behavior of the surrounding vehicles is decided based on the current state. Specifically, the longitudinal and lateral decision models are IDM and MOBIL models, respectively.

To increase the complexity of the case, we set the number of lanes as four, which is more complex than the three lanes and can further verify the reliability of the algorithm. And the initial speed of the self-vehicle is 25m/s. As for the hyperparameters of the dueling double DQN, the learning rate is 0.0005, and the reward discount factor γ is 0.99. The agent only acquires the signal of the vehicle ahead with a range of 180m. For the same scenario, the four following strategies are tested and compared.

1) UHDSE: HDSE reinforcement learning algorithm for dueling double DQN considering environment uncertainties.

The observation space is the future integrated risk assessment state model [START_REF] Cao | Highway Exiting Planner for Automated Vehicles Using Reinforcement Learning[END_REF], and the reinforcement learning model is the HDSE-dueling double DQN.

2) UDDDQN: Dueling double DQN reinforcement learning algorithm considering environment uncertainty. The observation space is the future integrated risk assessment state model [START_REF] Cao | Highway Exiting Planner for Automated Vehicles Using Reinforcement Learning[END_REF] in uncertain environments, and the reinforcement learning model is the dueling double DQN.

3) DDDQN: Dueling double DQN reinforcement learning algorithm without considering environmental uncertainties. The observation space is the vehicle actual motion state model [START_REF] Zhang | Adaptive Decision-Making for Automated Vehicles Under Roundabout Scenarios Using Optimization Embedded Reinforcement Learning[END_REF], and the reinforcement learning model is the dueling double DQN.

4) RULE:

Rule-based vehicle decision-making method. The longitudinal and lateral decision models are IDM and MOBIL models, respectively.

To show the effectiveness of the risk assessment model proposed in this paper on the vehicle decision problem, we compare the UDDQN algorithm with the DDDQN algorithm. The risk assessment model was used in the UDDQN algorithm, while it is not used in the DDDQN algorithm. Similarly, we show the difference before and after accounting the uncertainty by comparing the performance of UDDQN and DDDQN algorithms.

Case 1: Low-Density Traffic Flow

For this case, the four methods (UHDSE, UDDDQN, DDDQN and RULE) are trained and tested in a low-density scenario simulating a normal highway driving situation, which is depicted in Fig. 6. The low-density traffic flow means that the average headway is 28m. The corresponding simulation results are shown in Fig. 7. Fig. 7(a) shows the total rewards of the four methods under the considered low-density traffic flow. In the initial phase, the RULE strategy provides higher returns than the reinforcement learning-based vehicle decision strategy. However, after 400 training episodes, the reinforcement learning-based strategy leads to significantly higher rewards. Meanwhile, the agent obtains the highest rewards under the UHDSE method. The collision probability of the reinforcement learning-based vehicle decision strategy in Fig. 7(b) rapidly decreases from 1 and converges to 0.1~0.3 with training. In addition, the UHDSE method has the lowest collision rate after 2100 training episodes, which yields a higher total rewards as shown in Fig. 7(a). Fig. 7(c) shows the average speed of the self-vehicle under the four decisionmaking strategies. The reinforcement learning-based vehicle uses a relatively high speed (25m/s) in the initial phase for higher speed reward ( , ) speed R s a . However, after a training period, the agent learns from the historical data that this is not the optimal choice because it brings a higher collision probability, so the vehicle speed gradually converges to about 22.5km/h. The optimal probabilities in Fig. 7(d) all converge from 0 to 1. In particular, the variation of the optimal action probability based on the UHDSE strategy fluctuates significantly due to the state-entropy based exploration probability. In contrast, the optimal action probabilities of the UDDQN and DDDQN strategies equally and smoothly due to the ε-greedy exploration.

The test results for the four decision-making strategies are summarized in Table 5 It can be seen that the UHDSE-based vehicle decision method leads to the highest rewards and the lowest collision percentage, which is consistent with the training results. The UDDDQN strategy achieves the highest average vehicle speed, and it sacrifices part of the safety for the highest traffic efficiency, which is undesirable. Moreover, the UHDSE-based method achieves a balance between the safety and the traffic efficiency, which improves the vehicle traffic efficiency while ensuring the safety as much as possible. We find that the vehicles performed more lane-changing behaviors to ensure the traffic efficiency to drive on the appropriate road in the low-density traffic environment. The None action percentage of the reinforcement learningbased vehicle decision strategy is 0 because the agent learned that performing other actions brings higher gains.

Case 2: High-Density Traffic Flow

To analyze the obstacle avoidance capability of the proposed vehicle decision-making method, we perform a highway test scenario with a high density of vehicles as depicted in Fig. 8. Moreover, the high-density traffic flow means that the average headway is 14m. The considered test scenario is more dangerous than the real-world situations to test the performance of the vehicle decision strategies in extreme environments. The corresponding results are shown in Fig. 9.Fig. 9(a) shows the total rewards of the agent under the four decision-making methods. The agent can obtain the highest reward under the UHDSE method and reaches the convergence at 1300 episodes faster than the UDDQN (1950 episodes) and DDDQN (2150 episodes) methods. In addition, the total rewards of the converged UHDSE method (13.5) are higher than those of the UDDDQN (13.1) and DDDQN (12.3) methods. For the RULE method, the total rewards were better than the other methods when the agent was undertrained, but when the agent was trained for some time (about 800 episodes), the reinforcement learning-based UHDSE, UDDDQN, and DDDQN methods performed significantly better than the RULE-based methods (5.6). From Fig. 9(b), it can be obtained that the collision probability under the UHDSE method is significantly lower than the other three methods. Since safety is the primary concern in the practical application of autonomous driving, the decision model based on the UHDSE method is more promising to be applied in the real driving environment. We can see that, at the beginning of the training, the reinforcement learning-based method maintains a higher speed than the RULE-based method to obtain a higher speed reward ( , ) speed R s a . In particular, the average speed of the DDDQN-based method tends to increase at the beginning period. However, after a period of training, the reinforcement learning-based agent finds that higher speeds tend to bring negative collision rewards ( , ) collision R s a , so the speed decreases and eventually converges to the maximum reward. Compared with other methods, the UHDSE-based vehicle decision method can guarantee a higher vehicle speed while maintaining the lowest collision probability, improving traffic flow efficiency. Fig. 9(d) shows the variation of optimal action selection probability as the running times increase in the reinforcement learning algorithm. At the initial stage of training in reinforcement learning, due to the lack of historical information, the optimal action probability should be set low since the actual optimal action is not available. The optimal probabilities in Fig. 9(d) all converge from 0 to 1, which is coherent to the requirement of exploration and exploitation balance in reinforcement learning algorithms. Unlike UDDQN and DDDQN methods, the variation of the optimal action probabilities of the UHDSE-based agent significantly fluctuates. This is due to the state entropy-based exploration probability used by the UHDSE method, which obtains different action probabilities based on the current ( , ) Q s a . The test results of the four decision strategies are summarized in Table 6 Compared with the DDDQN and RULE strategies, the UHDSE and UDDQN vehicle decision strategies considering environmental uncertainty have a better performance. However, the average vehicle speed of the UDDDQN strategy is the lowest, and a part of the vehicle traffic efficiency is sacrificed to ensure the safety. Moreover, for the specific action percentage, compared with DDDQN, the UHDSE-based vehicle decision method has a higher average speed and a lower collision rate but also a lower lanechanging behavior, indicating that the DDDQN-based method has some invalid lane-changing. That is, lane changing may bring short-term rewards but not long-term rewards, and we will analyze the ineffective lane changing of the DDDQN strategy specifically in Figs. 10 and11. The lowest percentage of none actions is adopted with the UHDSE method, which indicates that it explores the environment more fully. In Fig. 10 at t=0s, the self-vehicle under both methods is moving in lane 3. At t=2s, the self-vehicle under DDDQN strategy changes lanes to lane 1 because vehicle A on lane 1 is further away than vehicle B. After 2s, vehicle A started braking due to the small distance in front of it, so there is a collision with vehicle A, which is an invalid lane change, only considering the short-term traffic efficiency. On the other hand, under the UHDSE strategy, the effect of an uncertain environment is considered, and the future motion state of vehicle A is predicted. Then, it chooses to continue driving in lane 3, which improves the efficiency of the passage with the premise of safety. In Fig. 11, at t = 3s, the self-vehicle moves in lane 1 under both strategies. Since the leading vehicle C in lane 1 is slow, which affects the passing efficiency, the selfvehicle under both strategies decides to perform a lane change operation. However, under the DDDQN strategy, the selfvehicle did not choose the right time to change lanes and started to change lanes before vehicle D. The collision occurred because there was no suitable space for a lane change. On the other hand, under the UHDSE strategy, the self-vehicle slows down in advance and chooses to start a lane change after the vehicle D. There is a sufficient lane change space to ensure the safety of lane change.

Conclusions

We have proposed a novel vehicle decision framework based on heuristic deep reinforcement learning for vehicle decision problems in uncertain environments. In view of the environmental uncertainty of highways, the lane change intention and vehicle motion trajectory of vehicles are predicted based on LSTM. Based on this model, a future risk assessment state model is proposed as a part of the reinforcement learning framework. Aiming at the previous literature's exploration and exploitation dilemma of reinforcement learning, we propose a heuristic decay state entropy-based reinforcement learning algorithm based on dueling double DQN, which adopts different exploration weights based on the entropy of current Q-values to improve the exploration efficiency.

The obtained results show that the proposed decisionmaking framework achieves a good performance in both lowdensity and high-density traffic flows, reducing the collision rate and improving the traffic efficiency. In the low-density traffic case, the UHDSE algorithm significantly reduces the collision probability while increasing the average speed compared to the DDDQN algorithm, because the UHDSE algorithm considers the future risk and completes the lane change behavior in advance before the risk appears. In addition, compared with the common DDDQN method in high-density traffic cases, the convergence time is reduced about 39.5%, the collision rate is reduced about 25%, and the average total return is improved about 17.1%. However, as shown in Table 6 there is also a 12% probability of collision, which is caused by a mismatch in the dataset. Moreover, the training of the intention recognition is done with the NGSIM real vehicle data, while the training vehicle decisions are done with simulation data. Therefore, building a training environment that better simulates the real world and testing in real vehicles is our future work. Moreover, game theory can be applied to the decision-making framework to represent the interaction between the self-vehicle and surrounding vehicles.

Fig. 1

 1 Fig. 1 System framework for vehicle decision system.

  p T is the history time horizon. Moreover, the output of the intention prediction module is the probability of changing lanes _ lane change P . Since the driving intention is influenced by various potential factors, we choose the LSTM model with a high nonlinear fitting ability and data feature extraction capability for its prediction. After the training periods, the lane change decision model summarizes the lane change decision behavior's pattern in the NGSIM data, predicting the driver's lane change or lane-keeping choice under current traffic environments. We selected the same training and test sets from the NGSIM dataset to test the intention recognition model under the fully connected neural network (FCNN), and LSTM network, respectively, and the results are shown in Table

Fig. 2 .

 2 Fig. 2. Network structure of the LSTM method in lane change intention prediction.

Fig. 3 .

 3 Fig. 3. Accuracy and loss evolutions during the training process.

Fig. 4 .

 4 Fig. 4. Network structure of the LSTM method in lane change trajectory prediction.

0 ca 1 

 01 of the self-vehicle and the surrounding vehicle Distance between the self-vehicle's distance sensor and the side of the vehicle e v Vehicle speed of the self-vehicle e Heading angle of the self-vehicle s v Vehicle speed of the surrounding vehicle s Heading angle of the surrounding vehicle x  Relative longitudinal position of the self-vehicle and the surrounding vehicle 0 Distance between the self-vehicle's distance sensor and the leading vehicle Reaction time of the driver s d Deceleration of the self-vehicle and surrounding vehicle next e v Self-vehicle speed at the next time next sx v Longitudinal vehicle speeds at the next time

Fig. 5 .

 5 Fig. 5. Structure of the Q-network.

Fig. 6 .

 6 Fig. 6. Diagram of traffic flow for low-density cases: self-vehicle (in green) and surrounding vehicles (in blue).

Fig. 7 .

 7 Fig. 7. Results of the low-density traffic case. (a) Total rewards. (b) Collision probability. (c) Average speed of self-vehicle. (d) Optimal action probabilities.

Fig. 8

 8 Fig. 8 Diagram of traffic flow for high-density cases: selfvehicle (in green) and surrounding vehicles (in blue).

Fig. 9 .

 9 Fig. 9. Results of the high-density traffic case. (a) Total rewards. (b) Collision probability. (c) Average speed of self-vehicle. (d) Optimal action probabilities.

Fig. 9 (

 9 Fig.9(c) describes the average vehicle speed of the selfvehicle under different methods. We can see that, at the beginning of the training, the reinforcement learning-based method maintains a higher speed than the RULE-based method to obtain a higher speed reward ( , ) speed R s a . In

Fig. 10 .

 10 Fig. 10. Vehicle driving conditions under Scenario 1: self-vehicle (in green), surrounding vehicles (in blue), crashed vehicle (in red) and the same markers (A, B, C and D) represent the same vehicles.

Fig. 11 .

 11 Fig. 11. Vehicle driving conditions under Scenario 2: self-vehicle (in green), surrounding vehicles (in blue), crashed vehicle (in red) and the same markers (A, B, C and D) represent the same vehicles.To further analyze the impact of considering environmental uncertainty on vehicle decision-making, two scenarios from

  were selected to analyze the difference in the decision-making of autonomous vehicles under UHDSE and DDDQN strategies. Figs. 10 and11 represent the vehicle positions in the two considered scenarios at different instants.

  Table1shows the state space and the corresponding action space in related works. In the observation space, ego v is the speed of the self-vehicle,

			dvy are the relative
			s
	longitudinal and lateral speeds of the surrounding vehicles
	and the self-vehicle,	,,
	s	s	s

ego lane is the lane index of the self-vehicle, des v is the ideal speed of the self-vehicle, ego behavior is the behavior of the self-vehicle, ego heading is the heading angle of the selfvehicle, s lane is the lane index of the surrounding vehicles, s dx and s dy are the relative longitudinal and lateral distances between the surrounding vehicles and the selfvehicle, respectively.

s dv and

Table 1 .

 1 Selection of the state space and the corresponding action space in related works.

	Method	Problem	Observation Space								Action Space
	Alizadeh et al. [28]	Highway	 surrounding: lane dx dy dv , , , s s s s   ego: , ego ego v lane		 Discrete ,, LK LC RC :	
	Ye et al.	Vehicle-	ego   ego: v									Continuo u s	:
	[29]	following	 surrounding: dx ss , dv							x   a
	Hoel et al. [30]	Highway	 surrounding: lane dx dv sss , ,  ego ego  ego: , v lane		 Discrete , , LC RC : FD ,	LK	,	, SD	 EB
	Xu et al. [8]	Highway	 surrounding: lane s  ego ego des , , s dx  ego: , , v lane v	s v			 Discrete LK LC ,, : RC	
	Wolf et al. [31]	Highway	 surrounding: lane s  ego ego , , , , s s s y x v ego  head ing ego: , , behavior v lane	s		 Discrete , , LC RC : FD ,	 SD
	Nageshrao et al. [32]	Highway	 surrounding: dx s	s , , , s dy dv	dvy	s		 Discrete , , , , LK LC RC FD :	NONE	SD , ,	 EB
	Aradi et al. [33]	Highway	 surrounding: dx s  ego ego ego: , , vy ea , , s dy g d in h	s dv ego  		 Discrete , , LC RC : FD ,	 SD
	Yu et al. [34]	Highway	 ego: v ego surrounding: , lane ego  dx  s , , s dy	s dv	

  ,

				S	t	=	t xy	t	v	t	lane	t	is the history
							ego ego	o eg	e go
	information		for			the	predicted	vehicle,
	E	si ( ) t	=	 dx si ( ) t	si ( ) ,, t dv	si ( ) t lane		is the history information for the
	surrounding vehicle,

Table 2 .

 2 Performance comparison for lane change intention.

	Method Actual classification	Predicted lane changing	Predicted lane keeping	Accuracy Total accuracy	F1	AUC
	LSTM	lane changing lane keeping	267 65	9 1978	96.7% 96.8%	96.8%	0.878 0.97
	FCNN	lane changing lane keeping	262 244	14 1799	94.9% 88.1%	91.5%	0.670 0.91

Table 3 .

 3 Vehicle nomenclature.

	Prediction	RMSE	MAE of x	MAE of y
	horizon	(m)	(m)	(m)
	1	0.70	0.45	0.15
	2	1.79	1.20	0.32
	3	3.22	2.37	0.40
	4	5.57	3.56	0.51
	2.4. Future Integrated Risk Assessment Model
	Time-to-collision (TTC) is commonly used for risk
	assessment, but it is inaccurate to only use TTC for risk
	assessment [40]. Hence, this paper defines a newly integrated
	risk assessment function based on the reference [41] to more
	comprehensively assess the future risk. Differently from TTC,
	the integrated risk assessment function considers both
	longitudinal and lateral risks of the vehicle. The vehicle
	nomenclature is given in

Table 4 .

 4 According to the vehicle kinematics model, the lateral risk and the longitudinal risk are expressed as

Table 4 .

 4 Vehicle nomenclature.

  Since the surrounding vehicles are not always present around the observable range when the vehicle is running. To ensure that the same number of features is input to the deep neural network, assume that the ith vehicle is at the observable range boundary when it is absent, i.e., =200 Future Integrated Risk Assessment State Model: The state space should be as efficient and straightforward as possible to effectively reduce the training difficulty and to improve the algorithm's performance. Compared with the actual vehicle motion state model, we propose an observation space model that considers the intention of surrounding vehicles and the future risk assessment of vehicles. Based on the future integrated risk assessment model, we define an observation space that takes into account the driver intention and the

	1) Vehicle Actual Motion State Model: To reflect its
	representativeness, the first observation space of this paper is
	selected as
	actual S	=	{ , ego x y lane d dv lane , , , , ego ego si si si	},	i =	1, 2,3, 4.	(9)
									d	si	and	si dv	=30.
	2) future risk assessment as
	S	IRAF	=	{ lane IRAF IRAF , , ego si si next	,	, lane lane s target i si	},	(10)
	for	i =	1, 2,3, 4	.	target si lane	is the driver's intention,
	next si IRAF	is the future integrated risk assessment function
	calculated from (8), and i represents the leading vehicle on
	the ith lane. Similarly, if there is no leading vehicle in on the
	ith lane, it is assumed at the boundary of the observation
	range, i.e.,	0 IRAF = ,	0 IRAF = and next si	targ si et lane	=	i s lane

si

IRAF

is the integrated risk assessment function defined in (5),

Table 5 .

 5 Test data set results in low traffic flow.

	Items

Table 6 .

 6 Test data set results in high traffic flow.

	Items	UHDSE	UDDDQN	DDDQN	RULE
	Average Total Reward	16.43	15.68	14.03	9.26
	Crash percentage	0.28	0.46	0.53	0.81
	Average vehicle speed	20.41	19.94	20.33	21.09
	Lane change maneuver percentage	18.6%	15.2%	21.4%	8.3%
	Longitudinal maneuver percentage	81.2%	80.7%	62.7%	84.0%
	None action percentage	0.3%	4.0%	15.9%	7.7%

LK Lane keeping LSTM changes MSE Mean square error NGSIM Next generation simulation NONE