Huifan Deng
email: denghuifan163@163.com

Youqun Qun Zhao
email: yqzhao@nuaa.edu.cn

Anh-Tu Nguyen
email: tnguyen@uphf.fr

Senior Member, IEEE Chao Huang
email: hchao.huang@polyu.edu.hk

Fault-Tolerant Predictive Control With Deep-Reinforcement-Learning-Based Torque Distribution for Four In-Wheel Motor Drive Electric Vehicles

Keywords: Electric ground vehicles, vehicle motion dynamics, fault-tolerant control, reinforcement learning, torque vectoring, in-wheel/hub motor

This paper proposes a fault-tolerant control (FTC) method for four in-wheel motor drive electric vehicles considering both vehicle stability and motor power consumption. First, a seven degrees-of-freedom vehicle nonlinear model integrating motor faults is built to design a hierarchical FTC control scheme. The control structure is composed of two levels: an upperlevel nonlinear model predictive controller and a lower-level fault-tolerant coordinated controller. The upper-level controller provides an appropriate reference in terms of additional yaw moment and vehicle longitudinal force, required for vehicle stability control, to the lower-level controller. This latter aims at distributing the four-wheel torques taking into account both vehicle stability and power consumption. Specifically, the weighting factor involved in the optimization-based design of the lower-level controller is determined online by the randomized ensembled double Q-learning reinforcement learning algorithm to achieve an optimal control strategy for the whole vehicle operating range. Moreover, the tradeoff between vehicle stability and power consumption is analyzed, and the necessity of using reinforcement learning is discussed. Numerical experiments are performed under various driving scenarios with a high-fidelity CarSim vehicle model to demonstrate the effectiveness of the proposed control method. Via a comparative study, we highlight the advantages of the new FTC control method over many related existing control results in terms of improving the vehicle stability and driver comfort, as well as reducing the power consumption.

I. INTRODUCTION

With the rapid development of automotive industry, energy issues have become more prominent. In addition, automobile exhaust pollution is another major concern of automotive industry. Carbon monoxide, nitrogen oxides and particulate matter are the main causes of environment pollution and global warming. Electric vehicles (EVs) have been considered as an effective solution for these issues [START_REF] Mao | Effects of rotor position error on longitudinal vibration of electric wheel system in in-wheel PMSM driven vehicle[END_REF], [START_REF] Hu | Robust yaw stability control for in-wheel motor electric vehicles[END_REF]. Four in-wheel motor drive (4IWMD) electric vehicles are the ultimate structure of future EVs, which have become a hot research application [START_REF] Karki | Status of pure electric vehicle power train technology and future prospects[END_REF]. The major advantage of 4IWMD electric vehicles is that each in-wheel motor can be independently controlled. Moreover, the motor torques and speeds can be measured in real-time, which makes the whole vehicle dynamics control more convenient and flexible to improve the vehicle stability [START_REF] Dalboni | Nonlinear model predictive control for integrated energy-efficient torque-vectoring and anti-roll moment distribution[END_REF]. Since the actuators of EVs are redundant, i.e., over-actuated systems, how to distribute the motor torques to achieve the best vehicle control performance is a crucial research problem. In particular, due to the presence of several actuators and the common use of wire-controlled technology, actuator faults may frequently occur, which can seriously affect the driving safety [START_REF] Zhao | Individual auxiliary and fault-tolerant control of steer-by-wire system considering different drivers steering characteristics[END_REF]. Hence, fault-tolerant control (FTC) is essential for 4IWMD EVs.

Numerous fault-tolerant control results have been reported in the literature for electric vehicles, including sliding mode control [START_REF] Zhang | Adaptive sliding mode fault-tolerant coordination control for four-wheel independently driven electric vehicles[END_REF]- [START_REF] Zhang | Robust fault-tolerant control for four-wheel individually actuated electric vehicle considering driver steering characteristics[END_REF], linear parameter-varying control [START_REF] Wang | Linear parameter-varying controller design for four-wheel independently actuated electric ground vehicles with active steering systems[END_REF], Takagi-Sugeno fuzzy model-based control [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF], [START_REF] Guo | Robust lateral control of autonomous four-wheel independent drive electric vehicles considering the roll effects and actuator faults[END_REF], observer-based non-fragile control [START_REF] Aravindh | Design of observer-based non-fragile load frequency control for power systems with electric vehicles[END_REF]. An adaptive sliding mode FTC considering mismatched nonlinear disturbances has been proposed in [START_REF] Tang | Actuator fault-tolerant control for four-wheel-drive-by-wire electric vehicle[END_REF] with hardwarein-the-loop validations. A nonlinear disturbance observer has been used to estimate non-matching disturbances and to attenuate the parameter perturbation. The authors in [START_REF] Zhang | Adaptive sliding mode fault-tolerant coordination control for four-wheel independently driven electric vehicles[END_REF] have developed an adaptive sliding-mode fault-tolerant coordinated control method, which allows to reduce the chattering by an adaptive variable reaching law. To increase the applicability range, the FTC control method in [START_REF] Guo | Robust adaptive fault-tolerant control of fourwheel independently actuated electric vehicles[END_REF] considers the loss-of-effectiveness fault and the bias fault in the vehicle modeling. A three-step FTC scheme taking into account uncertain parameters, external disturbances and actuator faults has been proposed in [START_REF] Wang | Fault-tolerant control of electric ground vehicles using a triple-step nonlinear approach[END_REF], where the issues on parametric uncertainty and faults are mitigated using Lyapunov techniques. Despite these great advances in EVs control, the power consumption of EVs has not been well addressed for the control design. Indeed, these control results have mainly focused on the vehicle stability in case of faulty motors. However, when the motor faults are not serious, FTC design with power consumption saving should be considered. The authors in [START_REF] Parra | On nonlinear model predictive control for energy-efficient torque-vectoring[END_REF]- [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF] have developed torque distribution algorithms for 4IWMD EVs with healthy motor actuators by taking into account the motor power in the objective function of the quadratic optimization-based design problem to reduce the vehicle power consumption. In particular, fuzzy logic has been used in [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF] to adjust the tradeoff between the power consumption and the vehicle stability. However, due to the strong coupling between stability and power consumption, the existing torque distribution methods can only provide the optimal solution for the current operating moment, which can be insignificant for the overall operating range of the vehicle. Model predictive control (MPC) technique can be considered as an alternative solution to deal with this tradeoff. Unfortunately, MPC prediction is usually available over a time window, not for the whole vehicle travel. In particular, if the MPC prediction horizon increases, then the computational burden of the involved online optimization problem becomes too heavy for real-time vehicle control purposes.

Reinforcement learning (RL) brings some new inspiration to control methods since it can interact with the environment and keep adapting to achieve the best control performance. RL has been applied to several fields, e.g., battery energy management [START_REF] Zhu | Rule-based reinforcement learning for efficient robot navigation with space reduction[END_REF], robotic control [START_REF] Wei | Deep reinforcement learning based direct torque control strategy for distributed drive electric vehicles considering active safety and energy saving performance[END_REF], motion planning of autonomous vehicles [START_REF] Zhang | Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles[END_REF]. However, due to the strict requirement on vehicle stability, very few studies have been reported on using RL for torque distribution of EVs. A direct torque distribution algorithm based on a deep RL algorithm has been proposed in [START_REF] Wei | Deep reinforcement learning based direct torque control strategy for distributed drive electric vehicles considering active safety and energy saving performance[END_REF] to improve the EVs stability while reducing the motor power consumption. However, the motor torques of EVs are directly output by the proposed RL algorithm, which may yield an unstable closed-loop vehicle system. Then, the RL control strategy in [START_REF] Wei | Deep reinforcement learning based direct torque control strategy for distributed drive electric vehicles considering active safety and energy saving performance[END_REF] may lead to some risky driving situations. RL algorithms have been also exploited for fault-tolerant control design for complex systems. Zhang et al. [START_REF] Zhang | Reinforcement learning-based fault-tolerant control with application to flux cored wire system[END_REF] have used RL to obtain implicit models as references for solving the FTC problem where explicit reference models are not available. An RL-based FTC method has been proposed in [START_REF]Fault tolerant control using reinforcement learning and particle swarm optimization[END_REF] to mitigate the impacts of modeling uncertainties and unexpected faults.

Motivated by the above technical issues, this paper investigates the FTC design for 4IWMD electric vehicles considering both power consumption and vehicle stability, which has not been addressed in the literature. The proposed fault-tolerant control structure is composed of two hierarchical levels: upper control level and lower control level. Nonlinear model predictive control (NMPC) technique is used to design the upper-level controller, which aims at providing appropriate references on additional yaw moment and longitudinal force to guarantee the vehicle stability. Using these vehicle input references, the lower-level controller distributes the four-wheel torques taking into account the tradeoff between vehicle stability and power consumption. The main contributions of this paper are summarized as follows.

• We propose an RL-based FTC control method for 4IWMD EVs, which allows to improve the vehicle stability while globally reducing the power consumption for the whole operating range. • The deep RL-based torque distribution algorithm can be adapted in function of vehicle operating conditions to effectively manage the stability-consumption tradeoff. • Exploiting the randomized ensembled double Q-learning (REDQ) algorithm to EVs control, the proposed strategy can improve the safety issue compared to existing methods using RL algorithms to directly control the motor torques. Several numerical experiments and comparative studies are performed with a high-fidelity CarSim vehicle model to demonstrate the effectiveness of the proposed control method. The necessity for a learning-based torque distribution to globally improve the vehicle stability and driver workload while reducing the power consumption is also highlighted.

II. PROBLEM FORMULATION

This section first introduces the overall hierarchical control structure for 4IWMD EVs. Then, after the vehicle modeling, the control problem is formulated. The vehicle nomenclature is given in Table I, where the indices ij ∈ {f l, f r, rl, rr} respectively represent the left front wheel, the right front wheel, the left rear wheel, and the right rear wheel.

A. Overall Hierarchical Control Architecture

Fig. 1 depicts the overall architecture of the proposed hierarchical control scheme for 4IWMD EVs. The main components include a vehicle reference model, an upper-level controller, a lower-level controller, and a deep RL algorithm. The vehicle reference model provides the desired yaw rate and sideslip angle to the upperlevel controller. Based on NMPC technique and a seven degreesof-freedom vehicle model, this latter is designed to generate appropriate references on yaw moment and longitudinal force to guarantee the vehicle stability. Using these information, the lowerlevel controller aims at distributing the four-wheel torques for faulttolerant control purposes. A fault diagnosis (FD) scheme is used to provide the estimates of the fault factors. To manage the vehicle actuator faults, these estimates are input to the REDQ algorithm as the deep RL states. The fault factor estimates are also used to design the lower-level fault-tolerant controller. Moreover, the REDQ algorithm based on the risk assessment dynamically adjusts the tradeoff between the electric power consumption and the vehicle stability. Then, fault-tolerant coordination controller provides torque inputs to EVs while achieving the best consumption-stability tradeoff.

Fault-tolerant coordination control

RL output

Upper-level controller

Lower-level controller

B. Vehicle Reference Model

To guarantee the vehicle stability, the tracking control of the vehicle yaw rate and the sideslip angle must be performed [START_REF] Xiong | Vehicle dynamics control of four in-wheel motor drive electric vehicle using gain scheduling based on tyre cornering stiffness estimation[END_REF]. To this end, the desired value of the sideslip angle can be conservatively set as β d = 0 as discussed in [START_REF] Hu | Should the desired heading in path following of autonomous vehicles be the tangent direction of the desired path?[END_REF], while that of the yaw rate γ d is determined from the following two degrees-offreedom (DoF) vehicle model [START_REF] Rajamani | Vehicle Dynamics and Control[END_REF]:

mvx( β + γ) = 2C f δ f -β - aγ vx + 2Cr bγ vx -β Iz γ = 2aC f δ f -β - a vx -2bCr b vx -β . (1)
Taking into account the road adhesion coefficient µ, the lateral acceleration ay = γvx is limited by γvx ≤ µg, where g is the acceleration of gravity. Then, the reference yaw rate can be deduced from (1) as

γ d = min vx (l f + lr)(1 + Kv 2 x ) δ , µg vx sign(δ), with K = m(aC f -bCr) 2C f Cr(a+b) 2 .

C. Vehicle Dynamics with Motor Actuator Faults

The vehicle model (1) only represents the basic lateral dynamics, without any information on the nonlinear tires characteristics. Moreover, the left wheels and the right wheels are assumed to be the same. Hence, motor actuator faults cannot be considered with model (1). To avoid these drawbacks, a seven DoF vehicle model, as depicted in Fig. 2, can be used. The corresponding vehicle dynamics can be described by [START_REF] Zhang | Stability research of distributed drive electric vehicle by adaptive direct yaw moment control[END_REF]

m vx = F xf l + F xf r cos δ f -F yf l + F yf r sin δ f + F xrl + Fxrr + mvyγ m vy = F xf l + F xf r sin δ f + F yf l + F yf r cos δ f + F yrl + Fyrr-mvxγ Iz γ = a F yf l + F yf r cos δ f -b F yrl -Fyrr + dw 2 F yf l -F yf r sin δ f + Mz I ωij ωij = -F xij Rω + T tij .
(

) 2
The vehicle control inputs, i.e., longitudinal force Fx and yaw moment Mz, are expressed as follows [START_REF] Peng | Torque coordinated control of four in-wheel motor independent-drive vehicles with consideration of the safety and economy[END_REF]:

Fx = F xf l + F xf r + F xrl + Fxrr. Mz = F xf l a sin δ f - dw 2 cos δ f + dw 2 (Fxrr -F xrl ) + F xf r a sin δ f + dw 2 cos δ f . (3)
The nonlinear characteristics of the tires are represented by the well-known Magic formula as [28]

F 0 xij λ ij = Dx sin{Cxatan[Bx(λ ij + S hx ) (1 -Ex) + Exatan(Bx(λ ij + S hx ))]} + Svx, F 0 yij α ij = Dy sin{Cy arctan[By(α ij + S hy ) (1 -Ey) + Eyatan(By(α ij + S hy ))]} + Svy, (4)
where F 0 xij and F 0 yij are respectively the longitudinal forces and lateral forces of the tires, for ij ∈ {f l, f r, rl, rr}, and Bx, By, Cx, Cy, Dx, Dy, Ex, Ey, S hx , S hy , Svx, Svy are the characteristic parameters of the Magic formula. The tire slip angles are expressed by

α f l =atan vy + aγ vx -1 2 dwγ -δ f , α rl =atan vy -bγ vx -1 2 dwγ α f r =atan vy + aγ vx + 1 2 dwγ -δ f , αrr=atan vy -bγ vx + 1 2 dwγ (5)
The tire longitudinal slip ratios are defined as

λ ij = v ij -ω ij Rω v ij , ij ∈ {f l, f r, rl, rr}, (6)
where the vehicle speeds at the four wheels are expressed by

v f l = (vy + aγ) sin δ f + vx - γdw 2 cos δ f , v f r = (vy + aγ) sin δ f + vx + γdw 2 cos δ f , v rl = vx - γdw 2 , vrr = vx + γdw 2 .
Since the tire forces must satisfy the adhesion ellipse, the longitudinal forces and the lateral forces of the tires need to be corrected as follows [START_REF] Peng | Torque coordinated control of four in-wheel motor independent-drive vehicles with consideration of the safety and economy[END_REF]:

F xij = F 0 xij ψ xij ψ ij , F yij = F 0 yij ψ xij ψ ij , (7)
with The fault diagnosis and fault factor estimation for the motors of 4IWMD electric vehicles have been widely studied in the literature, see for instance [START_REF] Wang | Fault-tolerant control with active fault diagnosis for four-wheel independently driven electric ground vehicles[END_REF], [START_REF] Wang | Actuator-redundancy-based fault diagnosis for four-wheel independently actuated electric vehicles[END_REF], [START_REF] Zhang | Active fault-tolerant control for electric vehicles with independently driven rear in-wheel motors against certain actuator faults[END_REF] and related references. Here, we mainly focus on the torque distribution for FTC purposes. Then, it is assumed that the fault diagnosis scheme proposed in [START_REF] Wang | Fault-tolerant control with active fault diagnosis for four-wheel independently driven electric ground vehicles[END_REF] is given to estimate the fault factors k ij , for ij ∈ {f l, f r, rl, rr}, for FTC design. Moreover, to better represent the fault factor estimation, the effects of time delay and disturbances are also taken into account as

ψ ij = ψ 2 xij + ψ 2 yij , ψ xij = - λij 1+λij and ψ yij = - tan(αij ) 1+λij , for ij ∈ {f l, f r, rl, rr}.
k ij = T aij T ij , kij = 1 tps + 1 k ij + ∆d, (8)
where T aij is the actual vehicle torque input, k ij is the fault factor, kij is its estimate, tp is the delay time, ∆d is the disturbance, and s is the Laplace variable. Considering the fault factors in [START_REF] Zhang | Robust fault-tolerant control for four-wheel individually actuated electric vehicle considering driver steering characteristics[END_REF], and the relationship between the motor torques and the longitudinal forces F xij = T ij , for ij ∈ {f l, f r, rl, rr}, the longitudinal force Fx and the yaw moment Mz in (3) can be represented by

Fx = A Fx uc, Mz = A Mz uc, (9)
where the vector of the four-wheel motor torques uc = T f l T f r T rl Trr is the vehicle control input, and

A Fx = 1 Rω kfl cos δ f kfr cos δ f krl krr , A Mz = 1 Rω ζ 1 kfl ζ 2 kfr -dw 2 krl dw 2 krr , with ζ 1 = l f sin δ f -dw 2 cos δ f and ζ 2 = dw 2 cos δ f + l f sin δ f .

D. Problem Formulation

The nonlinear seven DoF vehicle dynamics can be rewritten from ( 2), ( 4), ( 5), ( 6) and [START_REF] Wang | Fault-tolerant control with active fault diagnosis for four-wheel independently driven electric ground vehicles[END_REF], in form ẋ = f (x, u), [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF] where x = vx β γ is the vehicle state, and u = Fx Mz is the vehicle control input. The explicit form of the nonlinear function f (•) in ( 10) can be easily obtained, which is omitted for brevity. This paper develops a multiobjective control strategy for the 4IWMD electric vehicle [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF], which is typically an over-actuated system as shown in [START_REF] Wang | Linear parameter-varying controller design for four-wheel independently actuated electric ground vehicles with active steering systems[END_REF]. The motor torques are distributed to guarantee both safety and comfort issues while globally reducing the power consumption of the motors over the whole vehicle operating range. In particular, the electric vehicle can also safely operate despite the occurrence of motor faults. To this end, we propose a fault-tolerant control scheme with two hierarchical levels: upper-level NMPC control and lower-level fault-tolerant torque distribution, as detailed in Section III.

III. HIERARCHICAL FAULT-TOLERANT CONTROL DESIGN

This section provides the technical details on both upper and lower control levels of the proposed hierarchical FTC scheme.

A. Upper-Level NMPC Control

Based on the seven DoF vehicle model [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF] and NMPC technique, the upper control level aims at generating the ideal total longitudinal force F *

x and the additional yaw moment

M * z , i.e., u * = F * x M * z
, to track the vehicle state reference for stability guarantee while taking into account the comfort and the energy saving issues.

Considering the nonlinear tire characteristics, when the tires operate within an unstable region, the actual lateral tire force cannot be well represented by a linear tire force model [START_REF] Rajamani | Vehicle Dynamics and Control[END_REF]. Then, a controller, whose design is based on the linear tire force model, cannot meet the performance requirements under such limited operating conditions. To improve the control performance under limited driving scenarios, the NMPC technique is used together with a nonlinear vehicle model taking into account the nonlinear tire characteristics. As a result, the NMPC controller design for the vehicle model [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF] can be formulated as the following nonlinear optimization problem:

min X ,U Np-1 k=0 e k W Q e k + u k W R u k + ∆u k W S ∆u k s.t. x k+1 = f d (x k , u k ), k = 0, • • • , Np -2, x min ≤ x k ≤ xmax, k = 0, • • • , Np -1, u min ≤ u k ≤ umax, k = 0, • • • , Np -1, (11)
where [START_REF] Aouaouda | Robust fault tolerant tracking controller design for vehicle dynamics: A descriptor approach[END_REF], x min and xmax are the upper and lower bounds of the vehicle states, u min and umax are the upper and lower bounds of the vehicle inputs,

∆u = u k -u k-1 , e k = x k -x d k , x d = v xd β d γ d is the vehicle state reference, f d (x k , u k ) is the discrete-time counterpart of the nonlinear function f (x, u) in
X = [x 0 , • • • , x Np-1 ] and U = [u 0 , • • • , u Nc-1 ]
are the state and the input sequences in the prediction horizon Np and the control horizon Nc, respectively. The weighting matrices W Q , W R and W S are predefined to track the ideal vehicle state quickly and smoothly, and to guarantee that the 4IWMD electric vehicle can achieve a good dynamic performance.

The nonlinear optimization problem [START_REF] Guo | Robust lateral control of autonomous four-wheel independent drive electric vehicles considering the roll effects and actuator faults[END_REF] can be solved using the sequential quadratic programming algorithm [START_REF] Houska | An auto-generated real-time iteration algorithm for nonlinear MPC in the microsecond range[END_REF]. After obtaining the optimal control solution sequence U, only its first element is implemented into the lower control level and the predicted time horizon is moved backward by one step to continue solving the new nonlinear programming problem.

B. Lower-Level Fault-Tolerant Torque Distribution

The lower control level aims at distribute four-wheel torque considering motor fault, with torque control inputs uc as [START_REF] Wang | Linear parameter-varying controller design for four-wheel independently actuated electric ground vehicles with active steering systems[END_REF]. Such that the total longitudinal force F *

x and the additional yaw moment M * z , given by the upper control level, can be achieved. For faulttolerant torque distribution, based on a deep RL algorithm, the tradeoff between the vehicle stability and the power consumption can be dynamically adapted according to the current state of the vehicle. Moreover, to maintain the longitudinal velocity for lateral control purposes, the lower-level controller must satisfy the condition.

F * x = A Fx uc, (12)
where A Fx and uc are defined in [START_REF] Wang | Linear parameter-varying controller design for four-wheel independently actuated electric ground vehicles with active steering systems[END_REF]. 1) Integrated Cost Function: The fault-tolerant coordinated control design with energy saving consideration can be reformulated as a multiobjective optimization problem, whose three main objectives are described below.

a) Guarantee the Vehicle Stability despite of: Motor Faults Since the longitudinal velocity is controlled by [START_REF] Aravindh | Design of observer-based non-fragile load frequency control for power systems with electric vehicles[END_REF], the four-wheel torques should satisfy the ideal additional yaw moment input M * z from the upper-level controller. This can be represented by the following cost function:

J 1 = (A M z uc -M * z ) W M z (A M z uc -M * z ), (13)
where A M z and uc are defined in [START_REF] Wang | Linear parameter-varying controller design for four-wheel independently actuated electric ground vehicles with active steering systems[END_REF], and W M z is a weighting matrix. b) Minimize the Attachment Utilization Ratio: To describe the tire stability margin, the following adhesion utilization ratio, i.e., the ratio between the actual road adhesion and the maximum possible adhesion, is used:

λr = 4 i=1 F 2 xij + F 2 yij (µF zij ) 2 , ij ∈ {f l, f r, rl, rr},
where µ is the road adhesion coefficient. The expressions of the vertical loads of the four vehicle wheels F zij , for ij ∈ {f l, f r, rl, rr}, are detailed in [START_REF] Peng | Torque coordinated control of four in-wheel motor independent-drive vehicles with consideration of the safety and economy[END_REF]. Note that the tire lateral forces are uncontrollable, then we only consider the tire longitudinal forces for the torque control allocation. Hence, the following cost function is used to guarantee the maximum tire stability margin:

J 2 = u c Wauc, (14)
with Wa = diag

1 (µRωFzij ) 2 4×4
. c) Minimize the Motor Power Consumption: The motor power consumption is computed as

P ij = 1 η(T ij , ω ij ) T ij ω ij , ij ∈ {f l, f r, rl, rr}, (15)
where the motor efficiency η(T ij , ω ij ), depending on the motor torque and speed, is obtained from a look-up-table as depicted in Fig. 3. Then, considering that the values of the motor torques can be both negative and positive, the following cost function is used to minimize the motor power consumption:

J 3 = ij∈{f l,f r,rl,rr} P 2 ij = u c Wpuc, (16)
where the weighting matrix Wp is computed from [START_REF] Wang | Fault-tolerant control of electric ground vehicles using a triple-step nonlinear approach[END_REF] as

Wp = diag ω 2 ij η 2 (Tij ,ωij ) 4×4
, for ij ∈ {f l, f r, rl, rr}. The integrated cost function, taking into account the specifications on both vehicle stability and motor power consumption, can be reformulated from ( 13), ( 14) and ( 16) as

J = α 1 J 1 + α 2 J 2 + α 3 J 3 . (17)
The weighting factors α 1 , α 2 and α 3 are defined as

α 1 = α 0 + (1 -α 0 )α RL , α 2 = α RL , α 3 = 1 -α RL , (18)
where α RL ∈ [0, 1] is the preference factor representing the tradeoff between power consumption and vehicle stability for the motor torque distribution. This factor α RL is obtained from a deep RL algorithm, described in Section III-C.

Remark 1. Since safety is a primary requirement for vehicle driving in any situation, the parameter α 0 ∈ [0, 1] is introduced in (18) such that J 1 is always involved in the integrated cost function J defined in [START_REF] Peng | Torque coordinated control of four in-wheel motor independent-drive vehicles with consideration of the safety and economy[END_REF]. This allows avoiding the vehicle instability issue that may occur with an unstable RL agent obtained from a training process. After several tests, we select α 0 = 0.2 to design the RLbased torque distribution algorithm.

From ( 17) and ( 18), the integrated cost function can be rewritten as follows:

J = u c Huc + G uc + C, (19)
with

H = (α 0 + (1 -α 0 )α RL )A Mz WaA Mz + α RL Wa + (1 -α RL )Wp, G = (α 0 + (1 -α 0 )α RL )(-2M z W Mz A Mz ), C = (α 0 + (1 -α 0 )α RL )M z W Mz Mz.
Since uc is not involved in C, this term has no impact on the control optimization problem. Hence, it follows from [START_REF] Zhu | Rule-based reinforcement learning for efficient robot navigation with space reduction[END_REF] that the final integrated cost function can be defined as

J = u c Huc + G uc. ( 20
)
2) Optimization-Based Torque Distribution: Note that the actual motor torque outputs must be constrained by the maximum adhesion force of the road surface and the external characteristics of the motors as

-(µF zij ) 2 -F 2 yij Rω ≤ T ij ≤ µF zij 2 -F 2 yij Rω, T ij ≤ T * ij (ω ij ), (21)
where the peak torques T * ij (ω ij ), for ij ∈ {f l, f r, rl, rr}, of the electric motors are obtained from the motor external characteristics, e.g., the motor efficiency map.

From the definition of the integrated cost function J in [START_REF] Wei | Deep reinforcement learning based direct torque control strategy for distributed drive electric vehicles considering active safety and energy saving performance[END_REF], the torque equality constraint [START_REF] Aravindh | Design of observer-based non-fragile load frequency control for power systems with electric vehicles[END_REF] and the torque limitation constraint [START_REF] Zhang | Safe reinforcement learning with stability guarantee for motion planning of autonomous vehicles[END_REF], the torque distribution problem can be reformulated as the following quadratic optimization problem:

min uc J = u c Huc + G uc s.t. A Fx uc = F * x , u c ≤ uc ≤ uc, ( 22
)
where the inequality operator in ( 22) is element-wise, and the vectors u c and uc are respectively defined as

u ij = max -T * ij (ω ij ), -(µF zij ) 2 -F 2 yij Rω , u ij = min T * ij (ω ij ), (µF zij ) 2 -F 2 yij Rω ,
for ij ∈ {f l, f r, rl, rr}. Here, the quadratic optimization problem ( 22) is solved using the Gurobi solver [START_REF]Gurobi optimizer reference manual[END_REF] to obtain the four-wheel torques.

C. Deep Reinforcement Learning for Torque Distribution

The proposed torque distribution is based on the result of the optimization problem [START_REF] Zhang | Reinforcement learning-based fault-tolerant control with application to flux cored wire system[END_REF]. Hence, the weighting factor α RL involved in the integrated cost function J plays a key role to achieve the best tradeoff between vehicle stability and motor power consumption. It is challenging to adapt this factor in function of each specific driving situation to manage such a stabilityconsumption tradeoff, especially in the presence of motor faults. In contrast to traditional control methods, RL can take into account the rewards for the whole operating range of EVs. Hence, RL can offer a better performance when the future system states should be considered to appropriately execute tasks. Therefore, to achieve the best performance possible from a global viewpoint with the whole vehicle operating range, we propose to compute the value of α RL based on an RL algorithm. The proposed RL algorithm can constantly interact with the driving environment to deduce the most appropriate value of α RL for the whole range by maximizing a predefined reward. RL algorithms can be classified into onpolicy and off-policy algorithms. Off-policy learning allows the use of older samples during the training process, which can improve the sample efficiency. It has been shown that the soft actor-critic (SAC) algorithm is one of the best performing off-policy RL algorithms [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]. However, the REDQ algorithm can outperform the SAC algorithm, and can be even better than model-based RL algorithms for multiple tasks [START_REF] Chen | Randomized ensembled double Q-learning: Learning fast without a model[END_REF]. Therefore, here we select the integrated learning REDQ framework to develop our RL-based torque distribution algorithm.

The RL action space and state space are selected as follows:

A = {α RL }, S = {eγ , kij , ε}, (23)
where eγ = γ -γ d is the yaw rate deviation. The stability index ε in the state space S in ( 23) can be obtained from the vehicle phase portrait [START_REF] Wu | Intervention criterion and control research for active front steering with consideration of road adhesion[END_REF], which is defined as

ε = 1 B 2 β + B 1 B 2 β ,
where the parameters B 1 and B 2 are related to the road surface adhesion coefficient, and their values can be found in [START_REF] Zhai | Electronic stability control based on motor driving and braking torque distribution for a four in-wheel motor drive electric vehicle[END_REF].

Since we focus on ensuring the stability while improving the power consumption and the driving handling of 4IWMD electric vehicles, the reward function R is defined as

R = R stability + Reconomy + R driver , (24)
with

R stability = -Ws t 0 (β -β d ) 2 + (γ -γ d ) 2 dτ, Reconomy = -We t 0 T ij ω ij η(T ij , ω ij ) dτ, R driver = -W d t 0 δ2 sw + W aδ a 2 x dτ, (25)
where Ws, We and W d are respectively the penalty factors corresponding to the vehicle stability, the motor power consumption, and the driver workload. The weighting factor W aδ represents the tradeoff between the rate of the driver's steering angle δsw and the vehicle longitudinal acceleration ax.

The RL sample process is stated as follows. At step t, the state s t ∈ S receives a reward r t = R(s t ) as defined in [START_REF] Xiong | Vehicle dynamics control of four in-wheel motor drive electric vehicle using gain scheduling based on tyre cornering stiffness estimation[END_REF]. Then, the policy π generates an action a t ∈ A, which interacts with the environment to generate the next state s t+1 . As a measure of the randomness of a policy, the entropy is inclined in the objective of the REDQ algorithm to make the generated policy more robust and generalizable. Then, the optimal policy π * is expressed as

π * = arg max π E τ ∼π γ f (r t + ξH(π(•|s t+1 ))) , (26)
where

E τ ∼π [•]
is the expectation corresponding to the trajectory distribution τ of (s t , a t ) followed by the policy π, ξ is a hyperparameter to determine the weight of policy entropy in the optimization objective, γ f ∈ [0, 1] is a discount factor, and H(•) is the entropy corresponding to the policy π.

The optimal state action-value function Q π (s t , a t ) reflects the performance of the control system, i.e., a higher value of Q π (s t , a t ) represents a higher reward for the executing action a t in the state s t . According to the Bellman equation, the objective of the Q-value is expressed as

Q π (s t , a t ) = E at+1∼π [ψ t ], (27)
with

ψ t = r t + γ f max at+1 Q π (s t+1 , a t+1 ) + ξH(π(•|s t+1 )).
Due to the high dimensionality of the state space S, we use a deep neural network to represent the Q-value function [START_REF] Zhang | Stability research of distributed drive electric vehicle by adaptive direct yaw moment control[END_REF] and the policy π. Then, the evaluate Q-value function is represented as Q θ (s t , a t ) by the network with the hyperparameter θ, and the policy π is represented as π φ by the network with the hyperparameter φ. To reduce the estimation error of the Q-function, we include a target Q-network Qθ (s t , a t ) with θ as the network hyperparameter. For the REDQ algorithm, N double DQN networks are built, and we compute the target value of Q-function by randomly selecting M networks Qθ (s t , a t ), with M ≤ N , among these N DQN networks as

y = r t + γ f min m∈I M Qm(st+1, a t+1 , θt ) + ξH(π(•|s t+1 )) . ( 28
)
where the subset I M of M random elements is such that I M ⊆ {1, 2, . . . , N }. The hyperparameters of the evaluate Q-networks can be obtained by minimizing the loss function

L θ (θ i ) = E ζt y -Q θi (s t , a t ) 2 , i = 1, 2, . . . N, (29)
with ζ t = (s t , a t , r t , s t+1 ). During the training process, the evaluation network hyperparameters θ i are used to periodically update the target network hyperparameters θi as follows:

θi ← ρ θi + (1 -ρ)θ i , i = 1, 2, . . . N, ( 30
)
where ρ is an updating factor. Moreover, we can train the hyperparameter φ of the policy π φ by minimizing the following loss function:

L φ (φ) = E ζt 1 N N i=1 Q θi (s t , a t ) + ξH(π(a φ (s)|s t )) , ( 31
)
where a φ (s) ∼ π φ (•|s) represents the action according to the policy π φ . The REDQ algorithm generates a Gaussian distribution with a mean value µ φ (s t ) and a standard deviation value σ φ (s t ) such that the actual value of α RL (s t ) is computed as [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF]

α RL (s t ) = 1 2 tanh(µ φ (s t ) + σ φ (s t ) ϕ) + 1 2 , ( 32
)
where denotes the Hadamard product, and ϕ ∼ N (0, I) is an independent noise. The µ φ (s t ) network and the σ φ (s t ) network are initialized with random values when they are untrained.

To solve the problem of low data utilization, we introduce the update-to-data (UTD) ratio U to control the number of data reuse. During the training process, we periodically get multiple samples from the replay buffer D to improve the data utilization [START_REF] Chen | Randomized ensembled double Q-learning: Learning fast without a model[END_REF]. Algorithm 1 shows the update process of the proposed deep RL algorithm, for which the soft update method is used to update the Q-networks. The REDQ algorithm can provide the optimal policy π * in [START_REF] Rajamani | Vehicle Dynamics and Control[END_REF]. Hence, we can compute the most appropriate value of α RL for the optimization problem [START_REF] Zhang | Reinforcement learning-based fault-tolerant control with application to flux cored wire system[END_REF] according to the current state as in [START_REF]Gurobi optimizer reference manual[END_REF]. Moreover, note that the trained RL algorithm does not induce any additional computational burden to the 4IWMD electric vehicle control problem. Compute the target value y with expression [START_REF] Pacejka | The magic formula tyre model[END_REF] 10:

Algorithm 1 Randomized Ensembled Double

Q-Learning 1: Initialize N hyperparameters θ i of Q θ (s t , a t ) Set target hyperparameters θi ← θ i , i ∈ {1,
for i = 1, 2, • • • , N do 11:
Update θ i using L θ (θ i ) in ( 29) and gradient descent 12: Update the hyperparameters of Qθ (s t , a t ) with [START_REF] Zhang | Active fault-tolerant control for electric vehicles with independently driven rear in-wheel motors against certain actuator faults[END_REF] 13: end for 14:

end for 15: Update φ using L φ (φ) in [START_REF] Houska | An auto-generated real-time iteration algorithm for nonlinear MPC in the microsecond range[END_REF] and gradient descent 16: end while

IV. ILLUSTRATIVE RESULTS AND EVALUATIONS

This section first describes the test environment and some performance indicators. Then, several case studies are presented to demonstrate the effectiveness of the proposed control method. Due to the over-actuation feature, 4IWMD electric vehicles can normally perform the driving tasks if one single motor or two motors on different sides are damaged. However, driving with two simultaneously damaged electric motors is not common in practice. Hence, here we focus on the torque distribution in case of partial or complete damage of a single electric motor. Note that when the RL algorithm is trained, the computational burden of the proposed FTC method mainly depends on the online NMPC optimization problem [START_REF] Guo | Robust lateral control of autonomous four-wheel independent drive electric vehicles considering the roll effects and actuator faults[END_REF], which has been experimentally shown to be compatible with real-time control implementation for 4IWMD EVs in [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF].

A. Simulation Setting

For validation purposes, a simulation platform is set up as shown in Fig. 1. A high-fidelity 4IWMD electric vehicle model is built using CarSim platform, the control algorithms are implemented in Matlab/Simulink, and the RL algorithm is implemented in Python. The signals can be transmitted and communicated between CarSim, Matlab/Simulink and Python for co-simulations. For test scenarios, we set tp = 0.1, and ∆d is a uniformly distributed random number with a sampling time of 0.05 and an amplitude of 0.03 for the fault factor estimation [START_REF] Zhang | Robust fault-tolerant control for four-wheel individually actuated electric vehicle considering driver steering characteristics[END_REF]. The nonlinear vehicle model ( 10) is discretized using the multiple shooting discretization method with a 4th-order Runge Kutta solver [START_REF] Diehl | Fast direct multiple shooting algorithms for optimal robot control[END_REF]. The NMPC algorithm implemented at the upper control level is solved using the ACADO toolkit [START_REF] Houska | An auto-generated real-time iteration algorithm for nonlinear MPC in the microsecond range[END_REF]. The numerical efficiency of this NMPC control implementation for real-time EVs control has been shown in [START_REF] Dalboni | Nonlinear model predictive control for integrated energy-efficient torque-vectoring and anti-roll moment distribution[END_REF], [START_REF] Parra | On nonlinear model predictive control for energy-efficient torque-vectoring[END_REF], [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF]. To verify the effectiveness of the proposed control method, we select the following five representative control strategies for comparisons and validations.

1) RL-FTC: This is the proposed fault-tolerant hierarchical control scheme with the REDQ RL algorithm.

2) SAC-FTC: This control scheme has the same structure as RL-FTC. However, a commonly used Soft Actor-Critic RL algorithm [START_REF] Haarnoja | Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor[END_REF] is used in place of the REDQ RL algorithm as for RL-FTC control.

3) CO-FTC: To our knowledge, FTC for 4IWMD electric vehicles considering both power consumption and vehicle stability has not been available in the literature. Hence, the control method in [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF] is adapted taking into account the motor faults in the control design. The stability-consumption tradeoff of this control method is managed using fuzzy logic.

4) CO w/o FTC: The control strategy is similar as CO-FTC without taking into account the motor faults, i.e., the control results in [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF].

5) w/o control:

There is no vehicle lateral control, the steering is controlled by the driver, and the speed tracking is performed by a tuned PID controller available in CarSim.

The following evaluation indicators are proposed to analyze the quality of the torque distribution algorithms in terms of handling stability, driver workload, motor load, additional yaw moment, and velocity tracking:

• Handling stability indicator Es = t 0 (e 2 β +e 2 γ )dτ
, where e β = β -β d and eγ = γ -γ d .

• Driver workload indicator E d = -R driver , where R driver is defined in [START_REF] Hu | Should the desired heading in path following of autonomous vehicles be the tangent direction of the desired path?[END_REF].

• Motor load indicator Em = t 0 ij∈{f l,f r,rl,rr} ∆T 2 ij dτ. • Additional yaw moment indicator E Mz = t 0 M 2 z dτ . • Velocity tracking indicator Ev x = t 0 (v xd -vx) 2 dτ .

B. Test 1: Analysis of Stability-Consumption Tradeoff

For this test, we examine the role of different weights α RL to analyze the tradeoff between vehicle stability and power consumption. To this end, the electric vehicle performs a double lane change (DLC) with µ = 0.5 and v xd = 72 (km/h). Moreover, the motor fault occurs at 7s with k rl = 0.5.

The value of α RL is set as a constant for several experiments with the proposed FTC method and the corresponding results are shown in Table II. Following the definition of the integrated objective function J in [START_REF] Wei | Deep reinforcement learning based direct torque control strategy for distributed drive electric vehicles considering active safety and energy saving performance[END_REF], a smaller α RL leads to a less motor power consumption E (kJ). However, note from Table II that when α RL = 0 the motors consume 3.9% more energy than when α RL = 0.25. This is because the vehicle stability is degraded when the torque distribution focuses on the power consumption as shown by the values of Es. Specifically, when the power consumption becomes higher, the upper-level controller generates more additional yaw moment Mz to maintain the stability, which in turn requires an important torque to generate the additional yaw moment. The increasing torque affects the motor power consumption. At the lower control level, the torque distribution based on the quadratic optimization problem [START_REF] Zhang | Reinforcement learning-based fault-tolerant control with application to flux cored wire system[END_REF] can only get the optimal solution at the current moment. Then, the whole running state of the vehicle should be considered via RL to achieve the best stability-consumption tradeoff for the whole vehicle running state. Note that the weight allocation based on fuzzy logic [START_REF] Deng | Torque vectoring algorithm based on mechanical elastic electric wheels with consideration of the stability and economy[END_REF] cannot allow to get the optimal solution under the whole running condition.

TABLE II PERFORMANCE EVALUATION W.R.T. DIFFERENT RL WEIGHTS α RL . Weight Es E d Em [N 2 • m 2 • s] E Mz [N 2 • m 2 • s] E [KJ] α RL =

C. Test 2: Training with a Random Motor Fault

To train a well-adapted RL agent for multiple driving conditions, we select a DLC with straight and turning traffic conditions with µ = 0.3 and v xd = 72 (km/h), which can cover different levels of driving risk. Also, to simulate the motor faults, we set k ij = 0.5, for t ≥ 7s, where ij where is randomly selected from the set {f l, f r, rl, rr}.

For both RL-FTC and SAC-FTC controllers, the agents are trained under the same conditions. Observe from Fig. 4(a) that both control strategies converge to the highest rewards when increasing the number of training episodes. Also, Figs. 4(b) and (c) show that both controllers increase the vehicle stability while reducing the power consumption. Despite their effectiveness, we can see however that the RL-FTC controller achieves higher rewards and converges faster than the SAC-FTC controller, i.e., the rewards of the RL-FTC controller begin to increase at episode 180 in place of episode 250 for the SAC-FTC controller.

For comparisons, we select the trained agents at different episodes to describe the variation of the agents under test conditions with an increasing training. Fig. 5 depicts the vehicle performance in case of a motor fault factor k f r = 0.5 under 0-700 training episodes. As shown in Fig. 5(a), the vehicle can complete the DLC under all control results, which indicates that the In particular, the test performance improves more significantly with the 200-episodes and 500-episodes trained agents. This is because in these cases, the agent receives valid data from the replay buffer, which significantly decreases the loss of the estimated Q-network. This result is also consistent with the trend depicted in Fig. 4(b). Table III shows the results of the evaluation metrics according to the number of training episodes. Compared to the untrained agent, the stability indicator of the agent trained with 700 sets is increased by 93%, while the motor power consumption indicator is decreased by 77%. The reduction of power consumption is also confirmed by the result shown in Fig. 6(a). We can see that in addition to an improved vehicle stability and a reduced power consumption, the control performance in terms of driver workload, motor load, additional yaw moment, and speed tracking ability are also improved. The behavior of α RL according to the number of training episodes is presented in Fig. 6(b). Without training, the value of α RL is generated based on a randomly generated action network. When the number of training sets reaches 200, the RL algorithm starts learning the behavior of α RL to achieve a good reward. However, the action variation of α RL is not yet significant. After 500 training episodes, the RL training results allow to achieve the best reward with some local tuning. We now examine the effect of different fault estimation results on the control performance of the proposed RL-FTC controller. To this end, we compare the evaluation indicators obtained with different values of the time delay tp, which is the main parameter in the fault factor estimation expression [START_REF] Zhang | Robust fault-tolerant control for four-wheel individually actuated electric vehicle considering driver steering characteristics[END_REF]. Fig. 7 depicts the behaviors of α RL with respect to different values of tp. As expected, after the occurrence of the motor fault at 7s, the estimate kfr converges quickly to its reference with small values of tp, and the corresponding responses of the RL action α RL are faster. However, we can see in Table IV that the variations of the performance indicators in function of the time delays are not significant. This means that the proposed RL-FTC controller can robustly perform the driving task despite the delay effects.

D. Test 3: Acceleration on a Split-µ Surface with Motor Fault

This test is used to verify the effectiveness of the proposed control strategy under straight-line acceleration conditions on splitµ surfaces. The road adhesion coefficients on the left side and on the right side of the road are respectively set as 0.1 and 0.8. The initial longitudinal velocity is 80 km/h, and it accelerates uniformly to 120 km/h within 20s. For this test, the left front motor is subject to fault k f l = 0, i.e., this electric motor is completely damaged, for t ≥ 10s. Fig. 8 shows that from 10s, i.e., 250m, the vehicle stability starts to decrease due to the motor fault. We can see in Fig. 7(a) that the lateral offset of both FTC controllers, i.e., RL-FTC and CO-FTC, are smaller than those without FTC, but the difference is not significant. This is because of the controllers without considering the FTC sacrifice a part of the velocity tracking ability. As shown in Fig. 7(b), the maximum velocity deviations of the RL-FTC controller and the CO-FTC controller are -0.23 km/h and -0.25 km/h, respectively, while the maximum velocity deviations obtained with the CO w/o FTC controller and with the case without lateral control are -1.03 km/h and -1.36 km/h, respectively. Similarly, as shown in Fig. 7(c), these latter sacrifice a part of the speed tracking ability, the difference in the yaw rate between the four control strategies is not significant.

As depicted in Fig. 9, the phase trajectory portrait shows a small difference in vehicle stability. However, the RL-FTC controller can save about 53% of motor power consumption compared to the strategy without lateral control, and an improvement compared to both the CO-FTC controller and the CO w/o FTC controller. This is because the current driving conditions are less risky and the RL-FTC and CO-FTC controllers tend to adopt a torque distribution approach that minimizes the power consumption. Note that the CO-FTC controller does not save as much energy as the RL-FTC controller.

As shown in Fig. 10, from 10s, the left front wheel motor is completely damaged and the output torque rapidly turns to 0. After the fault occurrence, the torques of the other wheels quickly increase to ensure the speed tracking performance. Note from Fig. 10(c) that the left rear wheel torques of both RL-FTC and CO-FTC controllers quickly decrease after a short increase. This is because the left rear wheel is on the same side as the damaged left front wheel, and the left and right side torques need to be maintained in balance to ensure the vehicle stability. Table V shows the evaluation indicators of the four controllers under linear acceleration conditions. For the RL-FTC controller, the stability indicator is slightly reduced. This is because to maintain the speed tracking capability, the controller tends to consider saving the motor power consumption since the current condition is without high risk. We can see from Fig. 11(a) that RL-FTC, CO-FTC, and CO w/o FTC controllers can complete the considered DLC task, which is not the case of the strategy without lateral control. In Figs. 11(b) and (c), the performance of RL-FTC and CO-FTC controllers is better than that of CO w/o FTC controller. In particular, after 7s the CO w/o FTC controller shows a significant variation, which indicates the effectiveness of the FTC in maintaining the vehicle stability with faulty motors. There is no significant difference in terms of stability performance between both RL-FTC and CO-FTC controllers, which guarantee a stable vehicle driving. This is also explicitly confirmed with the phase trajectory portrait in Fig. 12(a). Both RL-FTC and CO-FTC controllers have similar trajectories in the phase plane with lower boundaries than CO w/o FTC and w/o control controllers. Note that without lateral control, the vehicle state exceeds the stability boundary and cannot return to a stable point. As shown in Fig. 12(b), the RL-FTC controller leads to the lowest motor power consumption, which confirms that the RL-FTC controller can reduce the power consumption while guaranteeing the vehicle stability even with extreme driving conditions. With the quantitative results in Table VI, we can also see the advantages of the RL-FTC controller over other compared ones in terms of power consumption, driver burden and vehicle stability. V. CONCLUDING REMARKS A new fault-tolerant model predictive control method considering both the vehicle stability and the motor power consumption has been proposed. To achieve the best tradeoff between vehicle stability and power consumption, the weighting factor of the optimization-based torque distribution problem is adjusted online based on the REDQ deep RL algorithm. Moreover, several tests and analysis are performed with different values of the weighting factor to highlight that fuzzy logic cannot allow for an optimal control strategy for the whole vehicle operating range. Different evaluation indicators are given to quantify the performance of various control strategies under multiple driving scenarios. The co-simulation results obtained with a CarSim electric vehicle model show that the proposed FTC scheme can reduce the power consumption and the driver workload while guaranteeing the vehicle stability despite the presence of motor faults. For future works, we consider FTC control design for different types of motor faults and different steering characteristics of drivers. Moreover, vehicle experiments should be conducted to verify the real-time performance of the proposed control results.

Fig. 1 .

 1 Fig. 1. Overall hierarchical control architecture for 4WID electric vehicles.

Fig. 2 .

 2 Fig. 2. Schematic of a seven degrees-of-freedom nonlinear vehicle model.

Fig. 3 .

 3 Fig. 3. Electric motor efficiency map.

Fig. 4 .

 4 Fig. 4. RL algorithm training results. (a) Total rewards, (b) Performance indicator Es, (c) Motor power consumption E.

Fig. 5 .

 5 Fig. 5. Test results with different training episodes. (a) Vehicle trajectory, (b) Yaw rate, (c) Phase trajectory portrait.

Fig. 6 .

 6 Fig. 6. Test results obtained with different numbers of training episodes. (a) Motor power consumption indicator E, (b) RL action of α RL .

Fig. 7 .

 7 Fig. 7. Test results with different values of time delay. (a) Estimated fault factor krl , (b) RL action of α RL .

Fig. 8 .Fig. 9 .

 89 Fig. 8. Performance comparison results obtained with straight-line acceleration conditions. (a) Vehicle trajectory, (b) Velocity error, (c) Yaw rate.

Fig. 10 .

 10 Fig. 10. Results of four-wheel torques in straight-line conditions. (a) Front left wheel torque, (b) Front right wheel torque, (c) Rear left wheel torque, (d) Rear right wheel torque.

  km/h. The electric vehicle performs a DLC task with the reference trajectory shown in Fig.11(a). A fault of the rear left motor occurs at 7s with k rl = 0.5.

Fig. 11 .

 11 Fig. 11. Performance comparison results obtained with DLC conditions. (a) Vehicle movement trajectory, (b) Yaw rate, (c) Sideslip angle rate.

Fig. 12 .

 12 Fig. 12. Performance comparison results obtained with DLC conditions. (a) Phase trajectory portrait. (b) Motor power consumption E.

TABLE I

 I

		VEHICLE NOMENCLATURE.
	Symbol Description
	m	Total mass of the vehicle
	vx	Longitudinal velocity
	vy	Lateral velocity
	β	Sideslip angle of the vehicle center of gravity (CG)
	γ	Vehicle yaw rate
	δ f	Steering angle of front wheels
	a	Distance from the front wheel axles to the CG
	b	Distance from the rear wheel axles to the CG
	dw	Track width
	Iz	Yaw mass moment of inertia
	Rω	Wheel rolling radius
	C f	Front tires cornering stiffness
	Cr	Rear tires cornering stiffness
	F xij	Longitudinal force of the wheel
	F yij	Lateral force of the wheel
	ω ij	Wheel rotation rate
	I ωij	Wheel moment of inertia
	T i j	Output torque of the in-wheel motor
	g	Acceleration of gravity
	ay	Lateral acceleration

  2, . . . , N } Empty the replay buffer D 2: while agent interacting with the environment do Select an action a t based on π φ (•|s t ) Collect r t and s t+1 after taking the action a t Store the data {s t , a t , r t , s t+1 } in the replay buffer D Sample a mini-batch {(s t , a t , r t , s t+1 )} from D

	3:
	4:
	5:
	6:	for U updates do
	7:
	8:	Select randomly a subset I M ⊆ {1, 2, . . . , N }
	9:

TABLE III PERFORMANCE

 III EVALUATION W.R.T. DIFFERENT TRAINING EPISODES.

	Episodes	Es	E d	Em		Ev x [m 2 /s]
	0	6.678 92.11	50370	1.40E+06	30.0
	200	5.306 85.62	39410	1.41E+06	12.1
	500	0.506 18.80	29570	1.02E+06	1.16E-06
	700	0.494 16.63	23530	9.56E+05	1.26E-06

[N 2 • m 2 • s] E Mz [N 2 • m 2 • s]

TABLE IV PERFORMANCE

 IV EVALUATION W.R.T. DIFFERENT VALUES OF TIME DELAY.

	tp	Es	E d	Em		Ev x [m 2 /s]
	0.01 0.4935 15.14	20670	865200	118.75 1.63E-05
	0.1	0.494	16.63	23530	956000	118.77 1.63E-05
	1	0.5068	16.2	18420	975600	119.47 1.92E-05
	2	0.5695 16.68	20400	871900	119.17 2.08E-05

[N 2 • m 2 • s] E Mz [N 2 • m 2 • s] E [KJ]

  This test aims to analyze the control performance under extreme driving conditions. To simulate an emergency lane change behavior under a low adhesion road surface, we set µ = 0.3 and v xd = 72

	Front left wheel torque (N×m)	-50 0 50 100	0	2	4	6	8	10	12	14 RL-FTC 16 18 CO-FTC CO w/o FTC 20 w/o control	0 50 100 150 200	2	4	6	8	10	12	14	16	18	20
							Time (s)
														200
		150
														150
		100
														100
		50
														50
			0	2	4	6	8	10	12	14	16	18	20	0	2	4	6	8	10	12	14	16	18	20

E. Test 4: Double Lane Change with Motor Fault

TABLE V PERFORMANCE

 V INDICATORS IN STRAIGHT-LINE CONDITIONS.

	Controller	Es	E d	Em		Ev x [m 2 /s]
	RL-FTC	0.0093 6.548	4575	2.35E+05	0.014
	CO-FTC	0.0089 6.579	4724	2.30E+05	0.016
	CO w/o FTC 0.0031 6.702	3102	2.36E+05	0.355
	w/o control	0.0053 6.719	3134	0.00E+00	0.658

[N 2 • m 2 • s] E Mz [N 2 • m 2 • s]

TABLE VI PERFORMANCE

 VI INDICATORS IN DLC CONDITIONS.

	Controller	Es	E d	Em		Ev x [m 2 /s]
	RL-FTC	0.4485 17.34	16300	747600	9.3E-07
	CO-FTC	0.4391 19.01	24270	869300	1.2E-06
	CO w/o FTC 0.9724 13.99	18510	458700	2.7E-06
	w/o control	10.010 88.28	356100	0	2.2E+01

[N 2 • m 2 • s] E Mz [N 2 • m 2 • s]