Biao Yin
email: biao.yin@enpc.fr

Mahjoub Dridi

Abdellah El Moudni

Recursive least-squares temporal difference learning for adaptive traffic signal control at intersection

Keywords: adaptive traffic signal control, recursive least-squares temporal difference, approximate dynamic programming, adaptive phase sequence

This paper presents a new method to solve the scheduling problem of adaptive traffic signal control at intersection. The method involves recursive least-squares temporal difference (RLS-TD(λ))

learning that is integrated into approximate dynamic programming. The learning mechanism of RLS-TD(λ) is to make an adaptation of linear function approximation by updating its parameters based on environmental feedback. This study investigates the method implementation after modelling a traffic dynamic system at intersection in discrete time. In the model, different traffic control schemes regarding signal phase sequence are considered, especially the defined adaptive phase sequence (APS). By simulating traffic scenarios, RLS-TD(λ) is superior to TD(λ) for updating functional parameters in the approximation, and APS outperforms other conventional control schemes on reducing traffic delay. By comparing with other traffic signal control algorithms, the proposed algorithm yields satisfying results in terms of traffic delay and computation time.

Introduction

Traffic congestion has critical impacts on people's daily life and environment. It leads to excess delays, reduced safety, and increased environmental pollution. Traffic signal timing aims to schedule flows at intersection efficiently to reduce congestion. In earlier research, a fixed-time (or pre-timed) cyclical signal plan is made for traffic management. Control parameters, such as cycle length, phase splits, and offset, are off-line optimized by using historical traffic data, and then are set in advance for signal controller. Fixedtime control is limited to low flow fluctuations, and is inefficient for traffic changes. Later, adaptive control occurs that it can adjust the control parameters in real time, responding to traffic demands. Adaptive control is more flexible in operation than fixed-time control. Adaptive traffic signal control systems have been available in practice, but most of them use conventional control methods. As the development of intelligent transportation systems, intelligent algorithm plays an important role to be implemented in signal controller.

One of the most well-known intelligent algorithms is reinforcement learning [START_REF] Khan | Reinforcement learning and optimal adaptive control: An overview and implementation examples[END_REF][START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF][START_REF] Xu | Reinforcement learning algorithms with function approximation: recent advances and applications[END_REF], which can make an agent (signal controller) do goal-directed learning and decision making by interacting with unknown environment. Over the past decade, many research has been worked on reinforcement learning with function approximation, which has been brought together with approximate dynamic programming (ADP) community [START_REF] Powell | Approximate Dynamic Programming: Solving the curses of dimensionality[END_REF][START_REF] Wang | Adaptive dynamic programming: an introduction[END_REF][START_REF] Werbos | Approximate dynamic programming for real-time control and neural modeling[END_REF]. Reinforcement learning mainly focuses on control problems based on Markov decision process (MDP) without model information, such as Q-learning. Differently, ADP usually obtains near-optimal solutions of MDP with some model information, which can be viewed as planning cases for sequential decision making [START_REF] Xu | Reinforcement learning algorithms with function approximation: recent advances and applications[END_REF]. From literature [START_REF] Cai | Adaptive traffic signal control using approximate dynamic programming[END_REF][START_REF] Haijema | An MDP decomposition approach for traffic control at isolated signalized intersections[END_REF][START_REF] Yu | Stochastic adaptive control model for traffic signal systems[END_REF], MDP can describe the traffic signal control problems well in a discrete-time dynamical way. However, conventional dynamic programming (DP) algorithm meets "curse of dimensionality" due to a large state space, according to the MDP description of this problem.

ADP offers a near-optimal solution and reduces computation cost tremendously using an approximate technique in its learning mechanism. In order to update the approximate function in ADP, there are several candidates for the learning mechanism, such as gradient descent [START_REF] Baird | Gradient descent for general reinforcement learning[END_REF], TD(λ) [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF], RLS-TD(λ) [START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF], and kernel-based learning [START_REF] Ormoneit | Kernel-based reinforcement learning[END_REF].

Here, we focus on RLS-TD(λ) learning. Bradtke and Barto [START_REF] Bradtke | Linear least-squares algorithms for temporal difference learning[END_REF] firstly proposed Least-Squares TD (LS-TD) and its recursive version RLS-TD in a linear regression. Afterward, in [START_REF] Boyan | Technical update: Least-squares temporal difference learning[END_REF] and [START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF], LS-TD(λ) and RLS-TD(λ) were presented, respectively, as the extensions of LS-TD and RLS-TD with trace-decay parameter λ (from λ=0 to general 01  ). As mentioned by Boyan [START_REF] Boyan | Technical update: Least-squares temporal difference learning[END_REF], LS-TD(λ) offers several significant advantages by comparing with TD(λ), such as fewer training samples, no step-size coefficient, and independence on arbitrary initial values. Similar advantages also appear in RLS-TD(λ). Moreover, RLS-TD(λ) has a recursive computation so that it is more suitable for online learning than LS-TD(λ).

In the other aspect, signal controller operates a traffic control scheme to schedule different directional flows, in order to make them evacuate from intersection without conflicts. Based on an MDP model framework, we study three kinds of traffic control schemes, namely fixed phase sequence (FPS), variable phase sequence (VPS), and adaptive phase sequence (APS), to know the effectiveness of the control strategies. All schemes take into account the properties of compatible traffic flows and signal phase sequences. In FPS, both of traffic flow combination and phase sequence are fixed. In VPS, traffic flow combination is fixed but phase sequence is variable. In APS, both of traffic flow combination and phase sequence are variable. In other words, compatible traffic movements can be grouped optionally in APS to occupy the intersection zone. But control performance is unknown when APS is integrated into a related algorithm.

Considering the advantages of RLS-TD(λ) learning and the flexibility of APS scheme, the objective of this study is to design an related algorithm to make some improvements in control performances regarding traffic delay and computing efficiency. Two main contributions are made in the paper. First, we propose an algorithm that integrates RLS-TD(λ) learning into ADP in an application to real-time adaptive traffic signal control at a typical intersection. Second, APS control scheme is implemented to compare with the other schemes based on the proposed algorithm, and its effectiveness for adaptive control is verified.

The rest of the paper is organized as follows. A literature review of related works is presented in Section 2. Main principles of ADP approach with RLS-TD(λ) learning in linear function approximation are described in Section 3. Section 4 presents an on-line control algorithm after modelling a typical signalized intersection with the three kinds of traffic control schemes. In Section 5, we do experiments and analyse simulation results. Finally, we draw conclusions and give the future work.

Literature review

Nowadays, adaptive traffic control systems are widely used around the world. From the view of system deployment, more and more adaptive traffic control systems are decentralized, although centralized systems (e.g., SCOOT [START_REF] Hunt | SCOOT -a traffic responsive method of coordinating signals[END_REF], SCATS [START_REF] Lowrie | The Sydney coordinated adaptive traffic system-principles, methodology, algorithms[END_REF]) have been higher degree of control logic on the intersection level [START_REF] Mladenovic | Adaptive Traffic Control Systems: Guidelines for Development of Functional Requirements[END_REF]. In methodology, many distributed systems, such as OPAC [START_REF] Gartner | Implementation of the OPAC adaptive control strategy in a traffic signal network[END_REF], PRODYN [START_REF] Henry | The PRODYN real time traffic algorithm[END_REF], and RHODES [START_REF] Mirchandani | A real-time traffic signal control system: architecture, algorithms, and analysis[END_REF],

uniformly recognized the importance of DP in solving the multi-stage decision-making problems. DP is not implemented directly for these complex systems because of its weakness in computation. There are two ways often taken into account for computation efficiency. One is to apply heuristic techniques and the other one is to simplify state variables, for example, state aggregation method. Control performances in these ways are not guaranteed. By reducing search states, forward DP algorithms are proposed to obtain exact solutions of traffic control problems [START_REF] Heung | Coordinated road-junction traffic control by dynamic programming[END_REF][START_REF] Wu | Discrete methods for urban intersection traffic controlling[END_REF]. However, related research has common limitations of the short-term planning and the requirement of perfect information about traffic arrivals.

There is considerable literature focusing on intelligent algorithms for adaptive traffic signal control systems [START_REF] Park | Realizing benefits of adaptive signal control at an isolated intersection[END_REF][START_REF] Abdulhai | Reinforcement learning for true adaptive traffic signal control[END_REF][START_REF] Lee | Real-time optimization for adaptive traffic signal control using genetic algorithms[END_REF][START_REF] Kergaye | Comparative evaluation of adaptive traffic control system assessments through field and microsimulation[END_REF][START_REF] Li | Traffic signal timing via deep reinforcement learning[END_REF]. A literature overview on this subject can be found in [START_REF] Araghi | A review on computational intelligence methods for controlling traffic signal timing[END_REF]. We introduce some influential references as follows. Park and Chang [START_REF] Park | Realizing benefits of adaptive signal control at an isolated intersection[END_REF] used genetic algorithm for preliminary study of adaptive signal control at isolated intersection, provided that perfect knowledge of individual vehicle arrivals is available. Lee et al. [START_REF] Lee | Real-time optimization for adaptive traffic signal control using genetic algorithms[END_REF] developed the work in [START_REF] Park | Realizing benefits of adaptive signal control at an isolated intersection[END_REF] using rolling horizon approach for realtime adaptive signal optimization. García-Nieto et al. [START_REF] García-Nieto | Swarm intelligence for traffic light scheduling: Application to real urban areas[END_REF] proposed a swarm intelligence approach to find successful cycle programs of traffic lights, using a microscopic traffic simulator. However, these evolutionary algorithms are generally limited by the explorations of micro-evolutionary processes and the planning models based on cellular processes, which cost much computation in simulation. Srinivasan et al. [START_REF] Srinivasan | Neural networks for real-time traffic signal control[END_REF] presented an approach about hybrid neural networks with fuzzy rules for real-time traffic signal control by introducing multi-agent system. The proposed system is proven to control a large-scale traffic network effectively. This approach is fine, but it is not sensitive to traffic environmental impacts. Recently, reinforcement learning and ADP methods obtain much attention in applications to traffic signal control problems, especially in micro-simulation models [START_REF] Cai | Adaptive traffic signal control using approximate dynamic programming[END_REF][START_REF] Arel | Reinforcement learning-based multi-agent system for network traffic signal control[END_REF][START_REF] Bazzan | Opportunities for multiagent systems and multiagent reinforcement learning in traffic control[END_REF][START_REF] Box | An automated signalized junction controller that learns strategies by temporal difference reinforcement learning[END_REF][START_REF] Prashanth | Reinforcement learning with function approximation for traffic signal control[END_REF]. These methods are sensitive to environment due to their learning mechanisms for updating values in time. With regard to the distinctions of approximation processes, there are several learning methods to update the value of approximate function.

In [START_REF] El-Tantawy | Multiagent reinforcement learning for integrated network of adaptive traffic signal controllers (MARLIN-ATSC): Methodology and large-scale application on downtown Toronto[END_REF], Q-learning is used to update the cooperative multi-agent, according to the best-response of Q-value at next state. In each intersection agent, the large state-space still remains. The likelihood of Q-value evaluated by the count of visit states need to take a long simulation time to converge. In the works of Arel et al. [START_REF] Arel | Reinforcement learning-based multi-agent system for network traffic signal control[END_REF], Box and Waterson [START_REF] Box | An automated signalized junction controller that learns strategies by temporal difference reinforcement learning[END_REF], and Li et al. [START_REF] Li | Adaptive dynamic programming for multi-intersections traffic signal intelligent control[END_REF], reinforcement learning with the approximation of neural networks, namely action network and critic network, are implemented to adaptive traffic signal control. The parameters in these neural networks are updated by gradient descent. Cai et al. [START_REF] Cai | Adaptive traffic signal control using approximate dynamic programming[END_REF] suggest that a simple linear approximation is sufficient for online operation. Because non-linear functions, such as neural networks, for exploring the complex approximation may not prove cost effective due to some difficulties in network training [START_REF] Zhao | Full-range adaptive cruise control based on supervised adaptive dynamic programming[END_REF], especially for the complex and discrete-time traffic control problem [START_REF] Huang | Modular design of urban traffic-light control systems based on synchronized timed Petri nets[END_REF]. In [7, Erreur ! Signet non défini.], linear approximations based on feature-extraction function were successfully employed, using TD(0) (λ=0) learning to update parameters. As mentioned before, RLS-TD(λ) is superior to TD(λ) in theory study. However, the effectiveness of RLS-TD(λ) has not been researched in field study, for example, in the application to adaptive traffic signal control problem.

In addition, to authors' knowledge, works on traffic control scheme at intersection are usually discussed on fixed flow combinations. It means that compatible flows always remain the same group, owning the right-of-way to occupy the conflict zone. On the other hand, signal phase sequence mostly operates in a fixed or variable way (see the survey in [START_REF] El-Tantawy | Design of reinforcement learning parameters for seamless application of adaptive traffic signal control[END_REF]). In the paper, FPS and VPS control schemes do in this way, too. Moreover, we will test the performance of the defined APS scheme, and make the comparisons with FPS and VPS in experiments.

As discussed above, the parameter learning techniques in ADP algorithm and the traffic control schemes of signal plan should be further studied, aiming to find an appropriate solution of traffic signal control problem.

Approximate dynamic programming with RLS-TD(λ)

Although DP can achieve an optimal control policy under MDP model framework. Powell [START_REF] Powell | Approximate Dynamic Programming: Solving the curses of dimensionality[END_REF] has mentioned that DP has computation burden as "three curses of dimensionality", which refers to the dimensions of state space, information space (space of random noise), and decision space. While, ADP approach with machine learning is computationally efficient to obtain a near-optimal solution. This section presents RLS-TD(λ) learning that is integrated into ADP approach due to the computational benefits of its learning efficiency.

Markov decision process

The environment of the decision problem we discuss is described by a finite MDP in [START_REF] Busoniu | A comprehensive survey of multiagent reinforcement learning[END_REF]

t tt t J s E r          (1)
where ( 01)  is a discount factor. The objective is to find an optimal policy by minimizing Eq. ( 1).

The following Bellman's equation offers a recursive solution to this goal. That is,

1 ** 11 () ( ) min ( , , ) ( ) tt t t t t t t t a A s sS J s r p s a s J s             (2)
for all t sS  .

ADP with linear function approximation

Traditional DP algorithms, such as value iteration and policy iteration, can solve the Bellman's equation given in Eq. ( 2) [START_REF] Bertsekas | Dynamic programming and optimal control[END_REF]. From practical experiences, there are two limitations of DP algorithms. First, the knowledge of transition probability 1 ( , , ) t t t p s a s  is required, and it is subject to the complete information from the environment. This makes DP in a short-term planning, which affects the performance over the whole horizon. Although some methods weaken this impact, such as rolling horizon approach and model predictive control, they will cost additional computation especially in large-scale DP problems. Second, it has to loop over the entire state space to evaluate the decisions such that an optimal stationary policy is finally obtained. The computation complexity is in an exponential order to the size of state variables. The computation requirement grows extremely even for a small size problem. For example, there is an isolated intersection with 8 lanes in 4-phase signal plan. Each lane can contain at most 19 queuing vehicles. Arrival information is either 0 or 1 for each lane and signal decision for each lane is either 0 remaining green or 1 switching current status. Thus, the computation order (without any simplification by control scheme) is Using linear function approximation, consequently, the objective value function in ADP is given by

1 () ˆ( ) min { ( , )} tt t t t t a A s J s E r J s     (4)
where ˆ() t Js is the observed value of current state according to environment.

In ADP, the estimated value J of visiting actual state by following a learning process will approximate the optimal value J  in convergence. Instead of computing the optimal J  with hugedimensional states, we compute the low-dimensional parameter vector t  . As a result, computation requirement is substantially reduced. For linear function approximation, it has been proven that there is only one optimum *  , which is achieved ultimately in convergence [11]. Due to limited state resources and solutions from this approximation, the parameter t  is updated to an ideal goal *

 by an incremental process to minimize some error metric between the estimated value J and observed value Ĵ . In the following part, we will use RLS-TD(λ) learning technique to find *  .

RLS-TD(λ)

Suppose that we observe a sequence of states

             (8)
With this notation, Eq. ( 7) can be rewritten as

1 t t t t t z       (9)
and the eligibility vectors can be updated recursively according to

11 t t t zz     (10)
initialized with 0 z  0 .

In the study of Tsitsiklis and Van Roy [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF], the above linear TD(λ) algorithm is proved to converge with probability 1 under certain assumptions. For special case λ=0, TD(0) is an equivalent to single-step TD algorithm where only the most recent observation matters to calculate the value function as well as the update of approximation.

To improve the efficiency of linear TD(λ) learning, LS-TD and RLS-TD learning methods are suggested in [START_REF] Bradtke | Linear least-squares algorithms for temporal difference learning[END_REF]. In LS-TD, the least-squares approximation to *  at time t ( 1) t  is the vector t  that minimizes the quadratic objective function

  2 T 1 1 1 ( ) ( ) . t ti i t i i Or t           (11)
By taking the partial derivative of ()

(

) .

t ii t ti i i i i r tt                       (12)
LS-TD method requires the computation of a matrix inverse at each time step. Thus, recursive least-squares technique is used to derive a modified algorithm, namely RLS-TD, to reduce the computational complexity of LS-TD. The weight update rules of RLS-TD are as follows:

T 11 ()

t t t t t r         (13)
Notice that Eq. ( 15) looks like the TD(0) learning rule for function approximation that is linear in the parameters, except that the scalar step-size parameter in TD(0) is replaced by a gain matrix. To use RLS-TD method, users must specify 0  and 0

P . 0
 is normally set to be vector 0. 0

P is typically to use 0 PI   ,
where I is the identity matrix and  is a small positive constant.

According to [START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF], RLS-TD(λ) can be viewed as the extension of RLS-TD with 01   . Besides Eq. ( 13), the weight update rules of standard RLS-TD(λ) are given by

          (17)
The detailed derivation of RLS-TD(λ) and convergence proof can be found in [START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF]. In the next section, we will focus on adaptive traffic signal control system, in which the ADP with RLS-TD(λ) learning is implemented.

Adaptive traffic signal control algorithm

In this section, firstly, we present modeling framework of traffic signal control at intersection based on MDP. Then, traffic control schemes and related decisions are discussed. Finally, we summarize the algorithm using a multi-step planning for online adaptive control.

Traffic dynamic system

Traffic signal control at intersection is considered as a stochastic discrete event system. Time intervals refer to identical divisions of a planning horizon. Assume that one interval is from t to t+1. At first, the following principal assumptions are given:

(1) Traffic signal durations are divided by discrete unit intervals. The size of one interval is 2 seconds, which is usually set to be a safe headway.

(2) Signal phases are composed of effective greens and reds only, excluding amber intervals. There is no pedestrian phase.

(3) Each phase contains at least mandatory intervals including inter-green intervals and minimum green intervals, during which no signal switching is admissible. The extension of green signal is one interval per step. There is no lost time for vehicle receiving green signal.

t t t t k n k n w n y n     (18) 1 ( ) ( ) ( ) ( )
where ()

t
According to Eq. ( 2), the objective of the control problem is to minimize the discounted total queue lengths corresponding to traffic delay. ,, G GG, and 4 G . In FPS scheme, signal controller organizes these combinations in a continuous loop, e.g., 1 (0,0,0,0,0,0,0,0) t a  and the action of signal switch is, for example, 1T (1,1,0,0,1,1,0,0) t a  for the next particular phase. The action 1 t a indicates that signals on lane 1 and lane 5 will change from green to red, and on lane 2 and lane 6 they do contrarily. As calculated by Eq. ( 18), the next transferred signal state is 1T 1

This is the final version published in

2 3 4 1 G G G G G     .

(

) mod 2 (0,1,0,0,0,1,0,0)

t t t x x a     .
The flow combinations for signal phases in VPS are the same with those in FPS. Differently, the phase in VPS may operate one after another in an uncertain or unordered way. For example, in Fig. 2 In APS scheme, the possibilities of phase choice are more than those in four-phase mechanism referring to FPS and VPS. It means that there are additional traffic flow combinations which can avoid flow conflicts, e.g., lane 1 can be combined with one of lanes 4, 5, and 6. We list all combination possibilities, as shown in Fig. 2 (c). For the relationship (combination coefficient) between two lanes, assign "1" if they are non-conflicting, and assign "0" otherwise. One possible phase sequence may operate like (1, 5)  (1, 6)  (3, 8)  (2, 6)  (4, 7)  …. In total, there are 12 combinations considering the upper triangular matrix as symmetry. APS groups all possible compatible traffic flows for signal phases. In other words, the action space APS () t As has 12 possible action vectors. This scheme is firstly considered in our study for the adaptive traffic signal control algorithm.

The control behaviors of traffic flows organized by FPS, VPS, and APS get sequentially more and more flexible. For numerous adaptive traffic signal control systems, signal phase duration is not fixed and phase sequence is cyclic, such as FPS scheme. When next alternative phase does not work in a particular phase order but in an acyclic way, it operates like VPS scheme. Moreover, in APS, any compatible traffic flows can be serviced at intersection. It does not have strict phase sequence requirement, which is significantly different from the conventional four-phase mechanism.

This is the final version published in Neural

Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

Control algorithm

Regarding the RLS-TD(λ) learning in ADP approach and our traffic system modeling above, some specifics are required to design the algorithm for traffic signal control.

With linear function approximation () J  in ADP, the above Eq. ( 24) can be modified by

1 () ˆ( ) min ( , ) . k tt tM k t M t w k t M t a A s kt J s E r J s              (25)
And signal controller can implement the following optimal action greedily 1 * () arg min ( , ) .

k tt tM k t M t w k t M t a A s kt a E r J s              (26)
According to Eqs. ( 13), [START_REF] Hunt | SCOOT -a traffic responsive method of coordinating signals[END_REF], and ( 17), the update of t  in RLS-TD(λ) learning rule by M-step planning is calculated as

1 T 1 () tM k t M t k t t M t kt r               (27)

This is the final version published in Neural

Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

Data preparation

In practice, traffic data is detected from inductive loops embedded upstream of each lane or other detecting techniques. In our experiments, random traffic arrival data is generated in each step by computer simulation.

A simple 0-1 Bernoulli distribution is adopted. By using inverse transformation method (ITM), random data consist of binary values 1 and 0, where value 1 represents vehicle arrival during one interval, and 0 otherwise. The probability of number "1" in distribution means an expected traffic arrival rate. Because vehicle arrivals generated by Monte Carlo simulation, traffic volumes referring to the sum of number "1" are a little different given the same traffic arrival rates.

For a finer planning, the value of either 1 or 0 in each unit increment, by which one interval can be divided integrally, is randomly chosen. The sum of values in all increments per interval is subject to the value of being either 1 or 0. For example, if there are 4 increments per interval, the situation of vehicle arrivals may be (0,0,1,0) or (0,0,0,0) , etc.

Simulator generates data according to the traffic arrival rate vector, which corresponds to the combinations 1 2 3 4

( , , , ) G G G G

. Traffic Scenario A and B that have asymmetric and symmetric rates, respectively, as well as the corresponding traffic volumes, are shown in Table 1. Traffic Scenario C gives symmetric time-varying arrival rates, which are processed smoothly ranging from 0.10 veh/int to 0.20 veh/int, as shown in Fig. 3. Notice that the highest traffic arrival rate in Traffic Scenario B has the intensity (calculated by the method in [START_REF] Gartner | Evaluation of optimized policies for adaptive control strategy[END_REF]) almost 0.9, which is already close to the road saturation.

Performance measure

The main performance measure refers to the term of average traffic delay denoted by D, which is expressed in seconds and calculated according to the following formulation in OPAC system [START_REF] Gartner | Evaluation of optimized policies for adaptive control strategy[END_REF].

D A 2 ( ) () NT t nt NT t nt kn T D T wn    (30)
where TD (veh•int) is to measure the total vehicle-intervals (in 2 s units) that is the sum of queue lengths of all lanes N during the period T. TA(veh) is the total number of vehicle arrivals at intersection.

Moreover, average queue length of waiting vehicles at intersection is concerned, and computation time is considered for the algorithm efficiency.

Implementation

For the simulation of traffic signal control system, system parameter setting is given in Table 2. Actually, learning parameter  in the comparative TD(λ) is step-size scheduling in a time-varying form [START_REF] Tsitsiklis | An analysis of temporal-difference learning with function approximation[END_REF]. From the experience of related study [START_REF] Cai | Adaptive traffic signal control using approximate dynamic programming[END_REF], we use a constant leaning rate at  =0.001. By numerical experiments, the functional parameters 0 () g n  and 0 () r n  are initially set to guarantee the control performance at the beginning of simulation. Regarding the performance of ADP with RLS-TD(λ) learning in FPS, VPS, and APS schemes, we compare the proposed algorithm with a fixed-time control (FC) method and certain adaptive control (AC) algorithms, i.e., Greedy algorithm, Heuristic algorithm, and Q-learning. They are introduced as follows.

 FC. In this control method, we consider the algorithm in [START_REF] Haijema | An MDP decomposition approach for traffic control at isolated signalized intersections[END_REF], called Haijema-MDP in this paper.

The optimal-fixed-cycle method starts with the minimum-cycle-length algorithm which increases the cycle length by unit increment based on the evaluation of Markov chains, according to the expected traffic arrival rates. Algorithm stops until no better cycle length can be found. In the literature, authors mentioned that Haijema-MDP was better than Webster's method, especially in unbalanced arrival rates.

 Greedy algorithm. In this method, the M-step planning evaluation function in Eq.( 24) lacks the part of exact state values and only has M steps of immediate rewards for searching forward. The decisions are greedily chosen by evaluating the immediate reward function.

 Heuristic algorithm. This is a forward search dynamic programming algorithm, shorten by FSDP [START_REF] Yin | Forward search algorithm based on dynamic programming for real-time adaptive traffic signal control[END_REF]. Because this method has a global optimization solution, computation complexity is very high by increasing the planning forward steps. We make this algorithm for two tests with planning horizon p T =16 intervals and p T =25 intervals, respectively.

 Q-learning. In this method, the normal rule for updating value function is used as follows,

Q s a Q s a r s a Q s a Q s a         , (31)
where t  is the learning rate, s is the transferred state from s taken action a . Entire traffic states need to be looped because their values need to be updated overall. Here, states are reduced by using three flow density levels, i.e., low, medium, and high. From numerical experiments, various settings of thresholds between the levels are confirmed, concerning the different traffic arrival rates.

All the experiments are implemented in computer simulation. The program is built by MATLAB 64 bits and executed by processor Intel Core i5, CPU 2.67GHz × 4. Assume that the detected traffic arrival information in the future can be received in M intervals. The information data is processed in rolling horizon approach, which can make the system decision per interval with the rolling M steps forward. In real-time operation, the executive time of algorithm should be guaranteed. That is to say, the run time per step while making a decision must be less than one interval in reality.

Results and analysis

To know the effectiveness of RLS-TD(λ) learning for the traffic control field, firstly, we analyse parameter evolutions in the algorithm. By implementing RLS-TD(λ) learning, the parameter evolutions of selected lanes are illustrated in Fig. 4. All of these parameters trend to relative steady levels after some steps.

Obvious differences appear in the parameters g Theoretically, RLS-TD(λ) is better than the conventional TD(λ) learning, according to studies in [START_REF] Boyan | Technical update: Least-squares temporal difference learning[END_REF][START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF].

We investigate their applications in the traffic control field and compare their experimental results, as shown in Fig. 5. Obviously, the parameters () From the analysis of parameter evolutions and the performance of average delay, RLS-TD(λ) is superior to TD(λ) learning in ADP approach for adaptive traffic signal control. By the following experiments, ADP with RLS-TD(λ) learning comparing with other control methods are discussed. flow organizations, such as APS, would likely come true.

In the near future, the proposed algorithm will be extended to two aspects. First, to make more reasonable and realistic, the model side need enrich the definitions of traffic state and control reward. As objective function with single control variable in learning methods often has limited performances, multiobjective signal timing control is a potential solution of this problem. Encouraged work could be found in references [START_REF] Khamis | Enhanced multiagent multi-objective reinforcement learning for urban traffic light control[END_REF][START_REF] Khamis | Adaptive multi-objective reinforcement learning with hybrid exploration for traffic signal control based on cooperative multi-agent framework[END_REF]. Second, the algorithm could also be promoted, aiming to find a solution of coordinated network control. By the reason of effects of upstream traffic, it is very important to consider the coordination among intersections so as to improve signal control efficiency in a global view.

Appendix: Derivation of RLS-TD(λ) in M-step planning

In M-step planning, the objective function Eq. ( 11) is modified as:

2 1 T 1 1 ( ) ( ) t i M k i M t k i M t i k i i Or t                  . ( 32
)
According to related theories [START_REF] Bradtke | Linear least-squares algorithms for temporal difference learning[END_REF][START_REF] Söderström | Instrumental variable methods for system identification[END_REF], we can rewrite Eq. ( 12) by using t  as the instrumental variable in LS-TD. That is,

t t i M M k i t i i M k i i k i ii r tt                               . ( 33
)
In LS-TD(λ), t  can be estimated as

t t i M M k i t i i i M k i i k i t t i M M k i i i i M k i k i i i i z z r tt z z r                                                        (34)
where using the eligibility vector t z in Eq. ( 10) substitutes the variable t  . to matrix inverse lemma and RLS-TD(λ) [START_REF] Xu | Efficient reinforcement learning using recursive least-squares methods[END_REF], the parameter vector t  updated by RLS-TD(λ) in M-step planning (in Eqs. ( 27), [START_REF] Li | Traffic signal timing via deep reinforcement learning[END_REF], and ( 29)) can be guaranteed.

Conflict of Interest:

The authors declare that they have no conflict of interest.

10 4. 2

 102 schemes, namely FPS, VPS, and APS are discussed.As shown in Fig.2(a), the intersection is controlled by typical four signal phases. The eight movements are divided into four flow combinations, denoted by 1 2 3

   . With VPS scheme, signal controller makes the phase sequence optimization and skips to another one considering its performances, such as queue length and vehicle waiting time. Therefore, the action space VPS () t As has 4 possible action vectors, one for remaining the same phase and three actions for switching to other phases.

20. end while 5

 5 Simulation and analysisIn this section, traffic signal control at isolated intersection depicted in Fig.1is simulated. Traffic scenarios in different traffic arrival rates are tested, using the three different traffic control schemes, i.e., FPS, VPS, and APS. To validate the proposed ADP_RLS-TD(λ) algorithm, its experimental results are compared with other different methods.

t

  , which associate to the feature-extraction function regarding vehicles receiving green signal. The values of steady levels related to (in Fig.4(c). This could be explained as follows. In Traffic Scenario A, the arrival rates of lane 1, 2, 6, and 7 are 0.1, 0.2, 0.2, and 0.1 veh/int, respectively. Lane 1 and 7 receive less green durations than lane 2 and 6, and go faster to reach the lower values at steady level. While the parameters r t  give almost the same values at steady levels. That means a little difference of queue lengths on the lanes where vehicles are waiting for red signal.

  TD(λ) achieve much less variances than those in TD(λ). Moreover, RLS-TD(λ) makes earlier stable trends in their evolutions. From the subfigures, different λ values in RLS-TD(λ) generate almost the same parameter evolutions. By contrary, big differences appear among the performances of TD(λ). Especially, in Fig.5(d), the parameter evolutions of TD(1) have the largest fluctuations. Similar aforementioned results can be obtained by other parameter settings of λ and n.In Fig.6, the results of average traffic delay are compared between RLS-TD(λ) and TD(λ) learning with different settings of λ. Traffic Scenario A, B, and C are tested. The performance of RLS-TD(λ) learning is better than TD(λ) learning, by reason of its advantages for updating parameters. In Traffic Scenario B and C, TD(1) even gets unreasonable values, which are removed.

Fig. 1

 1 Fig. 1 Illustration of a typical intersection with conflicts.

Fig. 2 9 20Fig. 3

 293 Fig. 2 Illustrations of traffic control scheme in (a) FPS, (b) VPS, and (c) APS.

Fig. 4

 4 Fig. 4 Evolutions of functional parameters by using RLS-TD(λ) learning (in Traffic Scenario A, APS with λ=0, n=1, 2, 6, and 7).

Fig. 5

 5 Fig. 5 Comparisons of functional parameters by using ADP approach between RLS-TD(λ) and TD(λ) learning (in Traffic Scenario C, APS with n=1): (a) λ=0, (b) λ=0.2, (c) λ=0.5, and (d) λ=1.

Fig. 6 Fig. 7

 67 Fig. 6 Comparisons of average delays by ADP between RLS-TD(λ) and TD(λ) learning in APS.

Fig. 8

 8 Fig. 8 Evolutions of average queue length by ADP with RLS-TD(λ) (in Traffic Scenario B, λ=0).

This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

   , where S is the finite set of environment states, A is the finite set of actions,

	: p S A S   	[0,1]	is the state transition probability function, and	: R S A S    is the reward
	function. In detail, the state	sS  describes the environment at each time t. The controller can choose the
							t
	state at each time by taking actions	aA  . As a result of the action	a , the environment changes its state
							t	t
	from	t s to some 1 t sS   , according to the state transition probability given by p which is represented as
	p	1 ( , , ) t t t s a s  . The controller receives immediately a scalar reward	t r  according to the reward function
	R	:	t r		1 ( , , ) t t t R s a s 	.
						For a given admissible policy π involving a sequence of decisions : a	S	A  , the expected total
							t
	reward is given by the following infinite-horizon discounted value function	J  , which is expressed as
	( )		,
						0
							. A finite MDP is
	a tuple	, , , S A p R
							5

This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

   is initialized to an arbitrary vector, t  is a sequence of scalar step-size, λ is a parameter in  

	where 0				0,1 ,
	and		 	( , ) tt Js 	is the vector of partial derivatives with respect to the components of	t  .
							t
								In the case of linear function approximation, a more convenient representation of TD(λ) is obtained
	by defining a sequence of eligibility vectors	z . That is,
												t
						t
	z							( )	tk		J	( , ) s
	t									t	t
												t
						k	0
						t
								( )	tk	.
											t
						k	0
	t w , i.e., 0 0 1 1 1 2 ( , , , , , , , , , T T s a w s a w s a w  . Temporal difference 1 ) T	t  (also called TD error) is defined
	corresponding to the transition from	t s to 1 t s  by
		t		t r	1 + ( , ) ( , ) t tt t J s J s     	(5)
	Since the linear case	( , ) Js		T  	, TD error in Eq. (5) at time t can be rewritten as
												t	t	t t
		t	1 () t t       t r	T		t	(6)
	Then, for t =0, 1, … ,T, multi-step TD (also called TD(λ)) updates	 according to the formula
												t
											t
		t	1 	t    t t  	0  k 	( ) 	tk 	 	t	( , ) t t Js 	(7)

t s based on the simulation with random input information

This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

  [START_REF] Powell | Approximate Dynamic Programming: Solving the curses of dimensionality[END_REF] The discharge rate (saturation flow) on each lane is one vehicle per interval. This rate is equivalent to 1800 vehicles per hour.Under the framework of MDP, as mentioned in Section 3.1, model formulation requires the characterizations of state, action, transition probability, and reward (cost) function. At time t, for an isolated intersection with total N lanes, traffic state can be expressed by

	t s	( , ), t t k x s S t  , where t k is a vehicle
	state vector with element () t kn representing actual number of queuing vehicles on lane n (	1,..., nN 	); t x
	is a signal state vector with element () t xn representing the signal state on lane n. Assign ( ) 1 t xn to green
	signal and ( ) 0 t xn to red signal. The decision or action of the system is	() tt a A s 	. Let	A	t s S  	() t A s
	denote action space. We define () t anto be a binary variable that ( ) 1 t an means signal switches on lane
	n, otherwise, ( ) 0 t an . During the mandatory intervals, we have ( ) 0 t an as well. Once the system makes
	a decision at time t, the traffic state t s will change. The transitions of signal state and vehicle state are
	expressed respectively by
	1 ( ) ( ( ) t x n t x n a n ( )) mod 2 t

 

  Because transition probabilities are hard to achieve for all state transitions, we can use the power of computer to generate random observations, which satisfy a specific distribution. The process is referred as Monte Carlo sampling. We have deterministic state transitions described in Eqs.[START_REF] Mladenovic | Adaptive Traffic Control Systems: Guidelines for Development of Functional Requirements[END_REF] and (19). Vehicle state at next time step is determined by the system state t s , information of vehicle arrival t w , and policy decision t a at the current step t. Since state transitions are influenced by random arriving vehicles, the process of vehicle state can be seen as a stochastic process with Markov property. As for the reward function

		t ynis also a binary variable
	restricted by
	() yn t	1, if ( ) 1 and ( ) 0, otherwise. t t x n k n w n ( ) 1 t       	(20)
	t r , in our study, it is defined by the sum of queue lengths at the next time 1 t  . That is,
		N
	11 1 ( , , ) t t t t n r R s a s t k n ( ).    

wn denotes traffic arrival satisfying a distribution according to traffic arrival rates. It adopts the value of either 0 or 1 vehicle/interval (veh/int). The traffic departure rate

()

  Generally, the inter-green interval or red clearance time represented by the barrier is used to separate phases for different flow directions. Adaptive signal control operates FPS to either extend or terminate current green phase, according to the detected traffic arrival information. In a particular state

			t s , the action space FPS () t As
	has only 2 possible action vectors such that	01 t t FPS ( ) { , } t A s a a 	. The action of signal unchanged is
	0T

This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

	1   tt PP	T 11 T 1 () 1 ( ) M t t t t M t M t t M t t P z P Pz            	(28)
	1   t   t	1 t    1 ( t M t P   T 1 ) M t Pz t 		t	t z	.	(29)
	Derivation of RLS-TD() in M-step planning is located in Appendix. Finally, on-line adaptive traffic signal
	control algorithm using ADP with RLS-TD(λ) learning is summarized in Algorithm 1.
	Algorithm 1: Adaptive traffic signal control algorithm by ADP_RLS-TD(λ)
	1. Choose an initial state 0 s	0 ( ) ( ( ), ( )) 0 n k n x n 	, parameter		0	0 ( ) ( ( ), ( )) 0 gr n n n   	T	for
	each lane n; set time	0 t  , planning step mM  ;
	2. Receive the traffic arrivals	w during M intervals from detected information;
								t
	3. Choose the action space APS A (or FPS VPS , AA);
	4. while tT  do
	5. if	0 m  then
	6. Signal unchanged with * 0 a  , and	max( 1,0) mm 	;
							t
	7. else
	8. for each	tt APS () a A s 	do
	9. Pre-calculate and store the accumulated rewards and estimated values;
	10. end for
	11. Find the optimal decision * a using Eq. (26);
								t
	12. if * 1 a  then
		t
	13. Change signal to all-red, and set	1 mM  ;
	14. end if
	15. end if
	16. Update functional parameter vector	 using Eqs. (27), (28), and (29);
									t
	17. Implement optimal decision * a at time interval t;
									t
	18. Transfer system state () sn, including signal state () xn and vehicle state () kn
							t		t	t
	by Eqs. (18) and (19), respectively;
	19.	1 tt ;

Table 1

 1 Traffic scenarios of asymmetric and symmetric average arrival rates.

		Arrival rate (veh/int)	Traffic volume by ITM (veh/h)
	Traffic Scenario A	(0.10, 0.20, 0.10, 0.20)	(350, 722, 365, 735)
	Traffic Scenario B	(0.20, 0.20, 0.20, 0.20)	(742, 725, 740, 735)

Table 2

 2 Details of system parameters.

	Parameters	Definitions	Value setting
	T			simulation period	40000 intervals
	N		total lanes at intersection	8
	g	min	minimum green time	3 intervals
	gmax	maximum green time	30 intervals
	g	int	inter-green (all-red) time	1 interval
	M	mandatory multi-step	4 intervals
		0 () g n	initial parameter to green signal 5 in Scenario A and B; 3 in Scenario C
		0 () r n	initial parameter to red signal	5
				discount factor	0.90
				learning rate constant	0.001
				parameter in matrix 0 P	0.01
	s			saturation (departure) flow	1 veh/int=1800 veh/h
	a

This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

	t	1 ( , )	t	( , )	( ( , ) t t	min ( , ) t aA	t	( , ))

Table 3

 3 Results of average delay (s) in Traffic Scenario A, B, and C.

	Traffic Scenario	A			B			C
	Haijema-MDP	23.68			49.02			30.69
	Phase sequence	FPS	VPS	APS	FPS	VPS	APS	FPS	VPS	APS
	Greedy		23.21 21.88 12.37	59.90 55.85 24.52	30.08 28.56 18.87
	FSDP ( p T =16)	18.65 14.80 8.74	45.30 40.53 12.58	22.12 20.25 9.23
	FSDP (	p T =25)	17.13 13.76 8.25	39.76 38.68 11.80	20.74 18.82 9.01
	Q-learning	20.03 18.23 12.02	49.32 48.90 22.10	28.14 26.88 15.42
	ADP_RLS-TD(0)	20.06 17.44 10.57	42.24 41.91 19.43	25.67 23.94 12.33
	ADP_RLS-TD(0)* 1	14.78 11.16 9.34	20.09 18.36 13.41	14.38 12.27 9.72
	Improvement (%)	0	15.43 45.96	0	5.04	57.34	0	8.11	46.95
	Note 1: simulation step is 0.25 interval (0.5 s) in a fine solution.

Table 4

 4 Comparisons of run time in Scenario A.

	Methods	FSDP (	p T =16)		FSDP (	p T =25)		ADP_RLS-TD(λ)
	Phase sequence	FPS	VPS	APS	FPS	VPS	APS	FPS	VPS	APS	APS* 1
	Run time (min)	1.3	6.3	99.6	3.2	23.9	582.5 0.3	0.7	1.1	4.4

Note 1: the same in Table

3

.

Table 3

 3 By comparing the learning methods, ADP_RLS-TD(λ) (λ=0) yields better results than Qlearning. Especially under the severe traffic density situation -Traffic Scenario B, ADP_RLS-TD(0) obtains delay reductions about 7 s in FPS and VPS, and 2.7 s in APS. On the other hand, FPS, VPS, and APS schemes have significant impacts on the performance of AC algorithms. Comparing with FPS, APS can obtain the 45.96%, 57.34%, and 46.95% improvements of average delay reduction in Traffic Scenario A, B, and C, respectively. Followed by VPS, average delay reductions are 15.43% and 8.11% in Traffic Scenario A and C, and 5.04% for Traffic Scenario B. The detailed improvements related to these control schemes in AC methods are shown in Fig.7. Integrated into appropriate algorithms, APS outperforms on average traffic delay than FPS and VPS, which are related to cyclic and acyclic signal control, respectively. APS can operate in a highly adaptive way. It means that not only the phase sequence is acyclic but also all the possible phases are evaluated by utility function.

	This

shows the results of average traffic delay after implementing different algorithms. The performances of FPS, VPS, and APS schemes are also involved. As a whole, AC methods that are able to adjust phase duration and phase sequence, are better than FC, except the results of Greedy algorithm for the FPS and VPS schemes in Traffic Scenario B. Greedy algorithm with limited information to guide decision making is not suitable for adaptive traffic signal control in these cases. With more model information (more planning steps), FSDP (Tp=16) and FSDP (Tp=25) are much better than Greedy algorithm.

is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

This is the final version published in

Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9 This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9 This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9 This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9

As shown in Table 3 and Fig. 7, FSDP (Tp=25) method looks like the appropriate approach with lower traffic delays and higher improvements. Actually, see Table 4, FSDP (Tp=25) costs much time during the whole simulation, especially in APS, where run time is 582.5 min. In VPS, it costs about 23.9 min.

Because the action space of APS is much larger than VPS, FSDP must search optimal values in a huge state-space when using APS scheme. Meanwhile, the computing time increases considerably as Tp increases.

By contrary, ADP_RLS-TD(λ) algorithm costs a little time in computation, even in APS where time consumption is about 1.1 min for the whole simulation (about 0.002 s per simulation step). In this algorithm, a forward process and an estimation by linear function approximation are the key points to reduce the computation complexity. Based on the simulation results, we argue that the optimal DP algorithms such as FSDP cannot conveniently apply to complex applications. For example, it is hard to work in the APS scheme, finer planning solution, and traffic network control. While, a near-optimal DP algorithm integrated with learning techniques, such as ADP_RLS-TD(λ) in this paper, indicates great potential to solve the highdimensional problem. The computational efficiency of ADP with RLS-TD(λ) learning is fully capable for online control at isolated intersection.

Additionally, in [START_REF] Sutton | Reinforcement Learning: An Introduction[END_REF], the authors presented the evidence that planning in very small steps may be the most efficient approach even on pure planning problems if the problem is too large to be solved exactly.

In the work of [START_REF] Cai | Adaptive traffic signal control using approximate dynamic programming[END_REF], the fine solution was achieved from perturbation learning which was better than the coarse solution. We investigate the finer step planning to know if this planning is suitable for ADP with RLS-TD(λ) learning in its application. Let the small step be 0.25 interval (0.5 s) in the experiments.

Considering ADP with RLS-TD(0) algorithm, consequently, the results of traffic delay are better than those by the original step of being one interval, as shown in Table 3. On the other hand, the performances of average queue length are shown in Fig. 8, which indicates the relevant effectiveness by the different control schemes as well. Obviously, APS works very well in less variance and lower queue length. Although these fine solutions cost more computation time (see in Table 4), it is enough to guarantee the online operation by using ADP with RLS-TD(λ) learning.

Conclusions

This study has investigated ADP with RLS-TD(λ) learning in an application to real-time adaptive traffic signal control at intersection. Regarding the evolutions of linear functional parameters in the tested scenarios, RLS-TD(λ) learning converged faster to the steady state, and the parameters fluctuated less than TD(λ) learning. Meanwhile, less traffic delays were observed by RLS-TD(λ) learning with different settings of λ. On the other hand, we compared ADP_RLS-TD(λ) algorithm with other control methods from experimental results. Although the heuristic algorithm FSDP achieved low traffic delays, it consumed too much time to operate in the fine scheduling case. While ADP with RLS-TD(λ) learning reduced largely the run time and yielded acceptable traffic delays, especially in the fine solution. Moreover, from the performances of the algorithms with FPS, VPS, and APS strategies, APS outperforms with 45%~58% delay reduction than FPS, especially under the high traffic density situation. However, it is impractical to implement the APS strategy in current real-world traffic operation due to the security factor when merging flows from different directions. If protocols among vehicles are able to guarantee the path security, flexible This is the final version published in Neural Computing and Applications, 2017 DOI10.1007/s00521-017-3066-9


Jacques Fontanel

By way of introduction Unbridled globalization, an aggravating factor in economic and social crises

Keywords: Pandemic, war, peace, misery, intolerance, political economy, economic war Pandémie, guerre, paix, misère, intolérance, économie politique, guerre économique

There are mainly four eternal problems of mankind (pandemics, war, religious intolerance, hunger), which man once thought he could control, that are now coming back to the fore. The appearance of new pandemics was not excluded, but confidence in modern logistical, scientific and medical progress was sufficiently high to create an irresponsible sense of security. With nuclear weapons and the opening of most countries to free trade, conflict between the great powers seemed highly unlikely, given the destructive consequences of nuclear weapons and the common interest in economic interdependencies. But economics is not the only subject of wars. Religious beliefs and differing notions of the concept of democracy do not lead to a peaceful world, not least because of the proselytizing and will to power expressed by some states. Philosophical and religious issues seem to lead to more secularism and less tolerance. Finally, the famous trickledown theory does not work and hunger is still present in many regions of the Earth and this phenomenon is increased in times of war and conflicts within or outside these territories.

Il y a principalement quatre problèmes éternels de l'humanité (les pandémies, la guerre, l'intolérance religieuse, la faim), que l'homme a cru un temps capable de contrôler, qui reviennent aujourd'hui sur le devant de la scène. L'apparition de nouvelles pandémies n'était pas exclue, mais la confiance dans les progrès logistiques, scientifiques et médicaux modernes était suffisamment élevée pour créer un sentiment de sécurité irresponsable. Avec les armes nucléaires et l'ouverture de la plupart des pays au libre-échange, les conflits entre les grandes puissances semblaient hautement improbables, compte tenu des conséquences destructrices des armes nucléaires et de l'intérêt commun aux interdépendances économiques. Mais l'économie n'est pas le seul sujet des guerres. Les croyances religieuses et les notions divergentes du concept de démocratie ne conduisent pas à un monde pacifique, notamment en raison du prosélytisme et de la volonté de puissance exprimés par certains États. Les questions philosophiques et religieuses semblent conduire à plus de laïcité et moins de tolérance. Enfin, la fameuse théorie du ruissellement ne fonctionne pas et la faim s'invite encore de nos jours dans de nombreuses régions de la Terre et ce phénomène est accru en temps de guerre et de conflits à l'intérieur ou à l'extérieur de ces territoires.

There are many factors of crisis in the daily life of the citizens of this world. Thus, the four great eternal problems of humanity (pandemics, war, religious intolerance, hunger), which for a time man believed capable, if not of eradicating, at least of controlling, have come back to challenge international chancelleries, philosophers or the whole of suffering humanity.

-The famous pandemics such as the plague or smallpox seemed to have been solved when Covid-19 appeared. Of course, the appearance of new pandemics was not excluded, but the modern world, imbued with its technicality, has always thought that it had the logistical, scientific and medical means to find a quick solution to the lethality of this threat.

-With nuclear weapons and the opening of most countries to free trade under the control of the World Trade Organization, conflicts between the great powers seemed highly unlikely, given the destructive consequences of nuclear weapons capable of eradicating life on Earth. Moreover, the economic interdependencies that lead to the respect of the "soft market" favor situations of negotiations for a lasting peace, considering the economic stakes of the partners, who have no interest in becoming belligerents.

-Philosophical and religious issues seemed to lead to more secularism and less intolerance. Today, the principles of life that bring people together diverge, between a Western world based on republican principles, but also on an inequitable consumer society, the world of the great contesting powers that are not very sensitive to democratic values and are concerned with establishing a less American-centric world that is more open to their own ideologies (China, Russia, India, Middle Eastern states, for example) and finally the countries of the rest of the world (the Third World in perpetuity) still sucked in by economic scarcity and hunger, continuing imbalances of state power, national and international wars and conflicts, and religious issues.

-Religious beliefs and divergent notions of the concept of democracy do not lead to a peaceful world, especially because of proselytizing and the will to power expressed by great powers.

For liberal economists, with the end of the Soviet Union, globalization is a step towards peace. However, war is still a threat for a humanity capable today of committing suicide, with nuclear and cybernetic weapons. The analyses of mercantilism do not disappear and the relations of force are still alive. The nature of "war" has changed. National security cannot be limited to the military, it also includes health, education, industrial risks or the protection of natural resources. The economy is both a cause of war and a means of armed conflict. Economic warfare uses weapons adapted to obtain a right or the exercise of a domination. As such, blockades, embargoes, boycotts or mercantilist-type power actions are techniques of war or conflict recognized by States. The international economic system is largely dominated and organized by the great powers and by multinational corporations. Moreover, the coercive power of states is mainly controlled by the largest multinationals. The current globalization is criticized, it is accused of developing inequalities and factors of conflicts, by giving power to the actors of international finance and to an unscrupulous commercial world. With the process of digitalization, a new type of colonialism in a dematerialized world deeply influenced by mercantilist behavior. There is a need for economic and social democratization of democracy.

The concept of international security has often been analyzed from a military perspective. Today, the concept of economic and human security broadens this conception, adding to strategic and military considerations, international pollution, global warming, environmental rights and the security of individuals and minority groups. The aim is to reduce the potential for military, police, economic, psychological and social violence against human beings, to reduce poverty, lack of care, precariousness and illiteracy. Security theories must take into account national and international economic, human and social relations, as well as the balance of power in the international system.

Wars and the threat of war remain constant realities in our societies, and the twentieth century has been particularly rich in deadly conflicts. However, the neoclassical current still assumes, never discussed as if it were self-evident, that international trade is a factor of peace; when states cannot and should not intervene in the national economy, they are no longer in a position to wage economic war. With the slogan "America first" inspired by mercantilism and government-controlled Chinese capitalism, the domination and demonstration effects of states obviously change the conditions of the international market economy. Moreover, short-term self-interest is not compatible with the need to collectively save the planet Earth in great danger. Capitalism reveals its flaws, especially in the fields of ecology, environment, climate, but also in the unbearable inequalities of income and power between countries or between citizens.

Today's economic globalization is not in itself a factor of peace. With its development, the economic war has taken other strategic forms imposed both by the most powerful countries and by the large multinational corporations, which ultimately account only for Washington or possibly for those of Chinese origin, in Beijing. It is a question of using the adapted weapons to obtain a right or the exercise of domination. Economic and military values become inseparable, which clearly expresses the close relationship between globalization and the balance of power in today's world. The Covid-19 pandemic has clearly highlighted the dangers of an economic globalization that has little respect for environmental standards and is always threatened by the belligerent actions of the great powers in the never-ending race for economic, strategic and political power. In this context, the Thucydides trap of world leadership between the United States and China cannot fail to worry the citizens of the world. At the same time, the threats of "rogue states" and terrorism have not disappeared. What then are the new forms of state power, but also what are the greatest threats shaking the world in the 21st century? Environmental and economic conflicts will be dangerous. The ecological and economic catastrophes (global warming, rare earths, transformation of production methods, excessive inequalities) are underway, close to the tipping point. The liberal economic system is not able to solve it, with the individual interest priority.

GDP is the most widely used economic indicator. It is a basis for classifying states in their obsession of development and power. The more the environmental and societal issues become worrying, the more the current indicator systems, especially GDP, become unsuitable. Short run economism neglects the scarcity of clean air but also democratic freedoms, citizen solidarity or cultural life. It is urgent to construct sets of indicators adapted to the questions posed for the understanding of the international relations and comparisons. GDP is the most widely used economic aggregate to compare, over time and space, the development of the national economy. International experts usually refer to this indicator to comment on the national economy. Its weight is important in determining the classification of countries according to their degree of development among developed countries. However, the limitations of GDP are both technical and conceptual. Real GDP (removing the effects of inflation) per capita does not refer to the productive and redistributive inequalities of economic actors. It does not include bad goods, pollution and it does not account for many free activities. The unique or dominant reference to GDP is dangerous. Economic actors should no longer be encouraged to go in the direction of an unsustainable form of growth. It is necessary to have indicators on democratic freedoms, citizen solidarity, cultural influence, the rise of the digital economy and artificial intelligence. It is also essential to count the negative effects of growth and their discounted costs, such as the definitive disappearance of certain raw materials, the influence of pollution on the people concerned and, in international relations, its effects on neighboring countries.

The crisis of globalisation and the rise of the Covid-19 pandemic pose new threats to the national security of all countries. States have been increasingly challenged on their role, particularly in the economic order. Their actions concerning ecology, climate or air pollution have been strongly inspired by the decisions of specialised international organisations, too often advised by dominant commercial and financial interests. In this context, states have no longer been able to assume the full range of national security components. Mercantilists considered that dependence on a State's foreign trade led to an unacceptable reduction to the power of the Prince. Today, all countries are dependent on others. Moreover, the pandemic has highlighted the inability of states to find quick solutions to the economic dependencies of states, creating a sense of insecurity among citizens. Tax havens contribute to the degradation of available statistical information, they aggravate the public debt of many States, they participate in the plunder of developing countries and they call into question the principle of equality of citizens and businesses before the tax. In addition, by their action and the coverup they allow on the "reality" of financial funds, they sometimes prevent developing countries from benefiting from the funds generated by the activities carried out on their territory.

The faith in the self-balancing of the markets is however contradicted by the facts, but its defenders always find in economic intervention the State the fundamental cause of crises. The role assigned to the State in economic and social life is mainly a philosophical, moral and of course political question. Market economy did not lead to peace, and globalisation is not a panacea. At the international level, American leadership is taking hold, but China is beginning to concurrence it. The struggle will be fierce in the years to come, because both countries want to rule the world to their benefit, thanks to their domination effects on the most powerful technologies for the future, such as artificial intelligence, space and especially information management. Europe is still in political, economic and philosophical crisis. BRICS do seek to define common solidarity policies, but their internal divergences often do not allow them to impose their objectives. Social inequalities have been growing for almost half a century. In the future, they will raise considerable economic problems that will undoubtedly lead to worrying social and societal conflicts. The robotic and digital revolutions call into question the importance and content of human labour. Finally, the looming environmental crisis, especially with global warming, may lead to the progressive collapse of the human race.

Scarcity is inscribed in human fibres; it concerns the limited time of life of the individual, but also the survival of man. Sometimes it is natural with the depletion of mineral resources; sometimes it is the result of an economic system of domination and the expression of a political power of power. The security of a community depends as much on the existence and appropriate exercise of weapons systems or expressions of power as on the sense of relative incapacity of potential adversaries to engage in aggression against it. It is, however, a costly service, especially in the context of economic globalisation that produces growing inequalities. Moreover, economics is sometimes a weapon in a system of nonself-governing nations, artificially creating pockets of poverty through predation or political coercion.

AROA proposes a philosophical, ideological and methodological reversal of the usual economic and statistical analyses. International experts still have knowledge based on the values they have been taught in the past and which they constantly reproduce. The statistical methods that underlie economic and political decisions only take into account market values, which are subject to manipulation that influences all economic agents. The famous catching up of development in stages is only a delusion at a time in history when great environmental and human damage has been committed by the economic actors of a greedy capitalism that is not interested in the living conditions of people or their survival. Under these conditions, Africa must build its own instruments to measure the progress of its national economies against the needs of its people and citizens.

Russia considers the conceptualization of a policy of "soft power" and "hard power " active, at least in the environment of Russia's proximity, mainly with the former members of the USSR. Russia wants to be once again a dominant actor on the international scene. Russia is increasingly worried about the progressive "encirclement" of its territory, with the enlargement of NATO and the reduction of its field of influence. In 2018, for United States, Russia is still perceived as the main geopolitical enemy, with China. However, economically, Russia has lost much ground to the United States, Europe and China. At the end, the military effort will be more difficult to grow in the case of a new arms race. The annexation of Crimea to Russia has been the subject of economic retaliation measures mainly by NATO members. Putin has used the lessons of Sun Tzu to achieve a result that, on the side of Russia, is globally positive; since the question of the return of Crimea to Ukraine is hardly mentioned anymore. In this context, we realize that Russia has suffered some negative effects on its growth, but it has considerably improved the conditions of its national security and its patrimony and heritage.

Economic science is an ideology that consecrates the omnipotence of the market economy, the police state and the management, often short-term, of an economy condemned to perpetual economic growth. With the Covid-19 pandemic, the analyses of liberal economists are losing their bearings, because the basic assumptions no longer have any concrete application in the face of a profound economic and social crisis. The economy now reveals its eminently political character. The international, national and local public sectors are organizing the fight against the crisis of the market economy. The State then becomes the central actor in the management of the national economy, in connection with the other States. The profoundly political and social character of a globalized economy highlights the violence of relations between states and citizens and often between states themselves. This situation of collective dependence is likely to create many tensions, which may lead to new conflicts or wars between states.

Public debt is always seen as the result of mismanagement of public funds and as a burden that future generations will have to bear. The morality defined by the political system of the market economy is based on the respect of debt repayment. However, public debt is different and has not to be analysed as a worse management of public resources. Contrary to private agents, State has not, in principle, a limited time horizon and it is possible to repay its debt over long periods by "rolling over its debt". Moreover, public debt helps to regulate the economy in times of crisis.This obviously depends on the type of debt and the economic circumstances of its development. In a crisis situation, the state is the economic agent capable of reviving national activity by investing and thus preparing the future of future generations. The weight of the debt, its spread over time, the nationality of the creditors, the investments effectively undertaken by the State or public administrations are all factors that deserve to be studied in order to analyse the interest of the public debt. Several methods exist which have been applied in economic history and which show that debt has often been the pragmatic and effective solution to combat national economic stagnation. When a Covid-19 arrives, public health becomes the national and international priority and the public debt is a solution to solve both the pandemic and the problems of unemployment, health costs and education gap that it causes. Public debt is an instrument for societal change, with less inequalities, poverty and precariousness.

Economic globalization has favored the rise of tax havens and offshore centers, which allow powerful economic actors to escape at the new tax levies necessary to reduce public debt. At the same time, criminal activities benefit from money laundering. Money laundering circuits were so opaque that very few banks knew whether or not they had dirty or terrorist money in their books. It thus favored the policies of "beggar-thy-neighbor" with impunity for countries. There is no consensual definition of tax havens, judicial, financial and judicial. The term tax haven is often used to define all "non-cooperative territories", with resources of unknown origin. Since 2014, the United States has enacted a Foreign Account Tax Compliance Act (FATCA), which requires financial institutions around the world to disclose the transactions of US nationals. Tax havens still cultivate the secret, they protect all their operations, and they distract most activities to make more complex reading from outside. Capitalism has become difficult to control, politicians no longer control the economic situation, and worst solutions are possible because greed and foolishness of men have no limit.

Due to economic globalization, tax havens allow owners to place their capital, even virtually, in financial centers that are particularly undemanding in terms of taxation and control over the origin of the funds thus placed. A distinction must be made between tax havens "stricto sensu", regulatory havens, judicial havens and offshore financial centers (OFC). The United Kingdom has the greatest weight in the world, ahead of Switzerland, Luxembourg, Hong Kong, the Cayman Islands, Singapore and the United States. There is even a certain specialization of "tax havens". Until 2010, the United Kingdom or Switzerland were highly recommended by banks, but today investments in Europe are increasingly risky for Europeans themselves. At the initiative of France, a tax of 3% of the turnover (and not the profits) generated by the exploitation of digital activities has been proposed on these companies. However, tax havens" are still very present in the world of financial markets. Most multinational companies use tax havens to optimize their profits. Democracy is clearly in danger.

When an economic crisis occurs, often caused by speculation by private actors, the state is called to the rescue and usually socialises the losses. Private debt then becomes public debt. For the liberal doxa, a debtor must pay his debts. In the case of public debt, this is not the case in practice, as the state has no time constraints. Therefore, it can "roll over" its debt and support a national economy in difficulty. Without its intervention, the economic crisis can lead to the collapse of societies and to particularly dangerous social and political problems, as economic history reminds us. It is usual to use particular indicators to underline the importance and dangerousness of public debt, but these indicators, too aggregated, do not give a satisfactory answer on the sustainability of the debt, whose temporal structure, the importance of the burden and the citizenship of the creditors modify its concrete interpretation.

GAFAMs dominate the world of information and knowledge highways. Their technological, economic and financial successes are remarkable. However, their quasi-monopoly situation constitutes a danger for the world, both by the power of their actions on the economic and political world, by their disinterest in financing public services, by their ability to use and reproduce information to their advantage and to produce important biases on political choices and freedom of expression.

Climate change poses a problem for the economic system that fostered the polluting industrial revolutions of coal and oil. Today, the digital economy revolution offers significant hope for reducing pollution and promoting decarbonisation. However, the economic interest struggles of the powerful lobbies of the polluting sectors seem to reduce the potential for transformation of an economic system driven by the search for short-term profit. The major powers want to preserve their economic gains and are undertaking this revolution at a pace that suits them, which is not without conflict, given the urgency of action in the face of the harmful transformations undergone by eco-systems. Furthermore, the GAFAMs, the powerful providers of digital services and instruments, together with their Chinese competitors, have a considerable economic and strategic force that could undermine freedoms and human and citizens' rights. Finally, for the production of digital tools, rare earths are likely to pose new problems, those relating to the pollution involved in their production and their relative scarcity compared to the stocks known today.