Marc Jungers

Emmanuel Trélat

Hisham Abou-Kandil

É Tr Élat

MIN-MAX AND MIN-MIN STACKELBERG STRATEGIES WITH CLOSED-LOOP INFORMATION STRUCTURE

Keywords: 1991 Mathematics Subject Classification. 91A65, 49N70, 49N90 Stackelberg strategy, game theory, multi-criteria optimization, closedloop information structure, bilevel optimization problem

Min-max and min-min Stackelberg strategies with

closed-loop information structure

Introduction

A Stackelberg game, named after Heinrich von Stackelberg in recognition of his pioneering work on static games [START_REF] Stackelberg | Marktform und Gleichgewicht[END_REF], designates a two-player noncooperative decision making problem formalized as a hierarchical combination of two optimization problems. The lower level decision maker, called the follower, selects a strategy optimizing his/her own objective function, depending on the strategy of the upper level decision maker, called the leader. The leader may decide his/her strategy, optimizing his/her objective function, relative to the decisions of both players by knowing the rational reaction of the follower. Such a problem may be viewed as a particular bilevel optimization problem [START_REF] Vicente | Bilevel and multilevel programming: a bibliography review[END_REF][START_REF] Dempe | Essays and Surveys in Global Optimization[END_REF][START_REF] Ye | Optimal strategies for bilevel dynamic problems[END_REF]. When the rational reaction set of the follower is not reduced to a singleton, the situation is more complex and several formulations exist and have been introduced by Leitmann [START_REF] Leitmann | On generalized Stackelberg strategies[END_REF] (see also [START_REF]Existence of solutions of two-level optimization problems with nonunique lower-level solutions[END_REF]) and called weak and strong Stackelberg strategies by Breton and al. [START_REF] Breton | Sequential Stackelberg equilibria in two-person games[END_REF] or pessimistic and optimistic ones in [START_REF] Dempe | Essays and Surveys in Global Optimization[END_REF][START_REF] Dempe | Second order optimality conditions for bilevel set optimization problems[END_REF]. The term strong reflects the fact that the leader and the follower are seeking to minimize the criterion of the leader. Such a strategy leads to several motivating properties. In particular, it is stable to perturbations and could thus be called a Stackelberg equilibrium and in addition the resulting criterion of the leader will be equal or better than the one obtained via a Nash solution. The term weak is applied to Stackelberg strategies for which the latter two properties do not hold anymore. Nonetheless they are interesting and adapted to model performance guarantees (or for instance, robustness in control theory), without assuming additionally that the follower, after having minimized his/her own criterion, tries to maximize the criterion of the leader as proposed in [START_REF] Leitmann | On generalized Stackelberg strategies[END_REF]. The strong and weak Stackelberg strategies will be referred to respectively as the minmin and min-max ones in the whole paper. The class of strong-weak Stackelberg strategies, introduced in [START_REF] Aboussoror | Strong-weak Stackelberg problems in finite dimensional spaces[END_REF] generalizes and gathers together strong and weak ones. Their computational aspects in static games have been studied in [START_REF]Weak via strong Stackelberg problem: New results[END_REF][START_REF] Loridan | A theoretical approximation scheme for Stackelberg problems[END_REF].

Game theory being a generic multiobjective optimization framework, the field of applications of Stackelberg strategies is large and includes, for example, economy [START_REF] Bagchi | Stackelberg Differential Games in Economic Models[END_REF], social behaviors, marketing [START_REF] He | A survey of Stackelberg differential game models in supply and marketing channels[END_REF], network communications [START_REF] Bas | A Stackelberg network game with a large number of followers[END_REF][START_REF] Lasaulce | Introducing hierarchy in energy games[END_REF], military intelligence [START_REF] Novak | A differential game related to terrorism: Nash and Stackelberg strategies[END_REF]. The Stackelberg strategy for differential games was introduced in [START_REF] Simaan | Additional aspects of the Stackelberg strategy in nonzero-sum games[END_REF][START_REF]On the Stackelberg strategy in nonzero-sum games[END_REF][START_REF] Chen | Stackelberg solution for two-person games with biased information patterns[END_REF]. We consider here two-player nonzero sum differential games with one leader and one follower.

The information structure [START_REF]Dynamic Noncooperative Game Theory[END_REF] in the game is the set of all available information for the players to make their decisions. The methods used to tackle such a Stackelberg optimization problem depend on the specific information structure.

When open-loop information structure is considered, no measurement of the state of the system is available and the players are committed to follow a predetermined strategy based on their knowledge of the initial state, the system's model and the cost functional to be minimized. Necessary conditions for obtaining a Stackelberg strategy with an open-loop information structure are well known [START_REF] Abou-Kandil | Analytical Solution for an Open-Loop Stackelberg Game[END_REF][START_REF] Simaan | Additional aspects of the Stackelberg strategy in nonzero-sum games[END_REF][START_REF]On the Stackelberg strategy in nonzero-sum games[END_REF][START_REF] Starr | Further properties of nonzero-sum differential games[END_REF][START_REF]Nonzero-sum differential games[END_REF][START_REF] Freiling | Existence and uniqueness of open-loop stackelberg equilibria in linear-quadratic differential games[END_REF][START_REF] Yashan | Stackelberg equilibirums of open-loop differential games[END_REF][START_REF] Nie | Dynamic Stackelberg games under open-loop complete information[END_REF][START_REF] Jungers | Anti-palindromic pencil formulation for open-loop Stackelberg strategy in discrete-time[END_REF] and are derived from the standard Pontryagin Minimum Principle [START_REF] Lee | Foundations of Optimal Control Theory[END_REF]. The obtained controls in this case are only functions of time.

The Stackelberg strategy is known to be inconsistent in time [START_REF] Dockner | Differential games in economics and management science[END_REF][START_REF] Cruz | Survey of Nash and Stackelberg equilibrium strategies in dynamic games[END_REF], and dynamic programming cannot help to derive the optimal controls. Note however that the concept of feedback Stackelberg control, not considered in the present paper, is defined as the limit of controls obtained by dynamic programming on infinitesimally small time subintervals (see [START_REF]Dynamic Noncooperative Game Theory[END_REF][START_REF] Bas | Feedback equilibria in differential games with structural and modal uncertainties[END_REF][START_REF] Mallozzi | Existence of a feedback equilibrium for two-stage Stackelberg games[END_REF][START_REF] Nie | Dynamic feedback Stackelberg games with non-unique solutions[END_REF][START_REF] Nie | Dynamic programming approach to discrete time dynamic feedback Stackelberg games with independent and dependent followers[END_REF][START_REF] Jungers | Matrix block formulation of closed-loop memoryless Stackelberg strategy for discrete-time games[END_REF]), that is in a multistage framework of repeated games. This concept differs from the concept of closed-loop control under consideration in this paper.

For the closed-loop information structure case (or more precisely the memoryless closed-loop information structure), each player has access to current state measurements and thus can adapt his strategy to the system's evolution. For the closed-loop information structure case, determining the Stackelberg strategy for differential games is much harder than the other information structures and has been an open problem for a long time. The main difficulty comes from the presence, in the expression of the rational reaction set of the follower, of the partial derivative of the leader's control with respect to the measurement of the state. Several attempts have been proposed in the literature to overcome this difficulty [START_REF]Dynamic Noncooperative Game Theory[END_REF]. Among such techniques, two main approaches could be distinguished.

The first method is dedicated to min-min Stackelberg strategies with a teamoptimal approach introduced in [START_REF] Bas | Closed-loop Stackelberg strategies with applications in the optimal control of multilevel systems[END_REF][START_REF] Bas | Team-optimal closed-loop Stackelberg strategies in hierarchical control problems[END_REF]. At the first step, the leader and the follower are aiming at optimizing the leader criterion as a team. Under some weak assumptions for linear-quadratic games [START_REF]A Stackelberg solution of dynamic games[END_REF], the optimal value of the leader criterion is attained for a parametrized family of controls for the leader and the follower. At the second step, the parameters of both controls are chosen such that the control of the follower lies in the rational reaction set in response to the control of the leader [START_REF] Tolwinski | Closed-loop Stackelberg solution to a multistage linear-quadratic game[END_REF]. This could be interpreted as a threat formulated by the leader towards the follower [START_REF]A Stackelberg solution of dynamic games[END_REF], that is the leader punishes the follower, if he/she does not comply with the leader's policy, like (grim-) trigger for repeated games [START_REF] Friedman | A non-cooperative Equilibrium for Supergames[END_REF][START_REF] Axelrod | The evolution of cooperation[END_REF].

The second approach consists in defining the whole rational reaction set of the follower for a given control of the leader. The resulting optimal control problem turns out to be nonclassical, not solvable a priori with the usual Pontryagin Minimum Principle. To solve this kind of nonclassical problem, a variational method is proposed in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF], assuming that this is a normal optimization problem (the possible occurence of an abnormal case is not mentioned). Moreover, in [START_REF] Medanic | Closed-loop Stackelberg strategies in linear-quadratic problems[END_REF] it is emphasized that this technique does not lead to a solution for all initial states, and the difficulty is bypassed by assuming that the initial state of the system is uniformly distributed over the unit sphere and replacing the optimization criterion with its mean value over the initial state.

In this paper, we investigate both min-max and min-min Stackelberg strategies with closed-loop information structure. The best response set of the follower is characterized. This allows a convenient reformulation of the constraints of the leader's optimization problem under an assumption that is weaker than considering the best response set reduced to a singleton. Nevertheless it is shown that under a standard strict Legendre condition, it is possible to solve both min-max and min-min Stackelberg strategies. Note that min-max and min-min Stackelberg strategies coincide whenever the best response set is reduced to a singleton, and this happens in the important Linear Quadratic case. The optimality necessary conditions for the leader are obtained, for both min-max and min-min Stackelberg strategies, along the associated trajectory, in the same spirit as in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF] for min-min Stackelberg strategy and by considering all cases. In addition, sufficient conditions of the optimization problem for Linear Quadratic differential games are established using focal times theory. Based on these necessary and/or sufficient conditions, we then characterize all initial states from which there emanates an optimal trajectory. Also, an extension is proposed to associate with every initial state an optimal trajectory by introducing the Jacobian of the leader's control in his own criterion. Note that in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF], although the final result (for Linear Quadratic games only) is correct, some of the arguments thereof used to derive the necessary conditions are either erroneous or missing.

The outline of the paper is as follows: in Section 2, the min-max and minmin Stackelberg strategies are mathematically formulated. Section 3 gathers the necessary conditions of existence for a strategy for the follower (Section 3.1) and for the leader (Section 3.2) for min-max and min-min Stackelberg strategies. A degeneration property of the Stackelberg strategy is emphasized in Section 3.3. These necessary conditions are detailed in the case of Linear Quadratic two-player differential games in Section 3.4. The sufficient conditions are provided for the Linear Quadratic case in Section 4. All these results lead to the two main results of this paper Theorem 1 and Theorem 2, which ensure the existence of optimal trajectories. Concluding remarks make up Section 5. The main proofs are gathered in Appendix A.

Preliminaries: Stackelberg strategy

A two-player differential game with finite horizon comprises • a set of two players K = {1, 2}, where here Player 1 is called the leader and Player 2 the follower. • a game duration T = [0, t f ], with t f > 0.

• a state-space X = R n which contains all the states x(t)

at time t ∈ T . • open sets U ⊂ L ∞ (T × X , R m1 ) and V ⊂ L ∞ (T × X , R m2 ) representing
the action sets (or control sets) respectively of the leader and the follower. They will be precise below. • a mapping f : T × X × U × V → X of class C 1 defining the evolution law of the state as

(2.1) ẋ(t) = f t, x(t), u |t , v |t , x(0) = x 0 ,
where u |t and v |t are the values at time t of the controls of both players u ∈ U and v ∈ V. • utility functions or criteria

J i : U × V → R of Player i ∈ K, defined as (2.2) J i (u, v) = g i (x(t f )) + t f 0 L i t, x(t), u |t , v |t dt.
The functions L 1 and L 2 are C 1 with respect to x, u and v are continuous with respect to time t. • The information structure of Player i ∈ K, which will be discussed below. The leader (Player 1), who chooses the control u, aims at minimizing the criterion J 1 and the follower (Player 2), who chooses the control v, aims at minimizing the criterion J 2 . We assume that the game is a single stage one, that is the choice of the controls should be done before the game begins at t = 0. It is assumed in this paper that there exists an information bias in the game which induces a hierarchy between the two players. We therefore have a Stackelberg differential game. The follower will rationally react to an observed control u of the leader. The leader is aware of this rational behavior and will use this bias of information to choose his/her control and to minimize his/her own criterion. Definition 1. The rational reaction set or best response set of the follower is defined by

(2.3) T : u ∈ U -→ T u ⊂ V where T u = {v | v minimizes J 2 (u, v), v ∈ V} . Definition 2. A min-min Stackelberg strategy (u * * , v * * ) is defined by the minimization problem (2.4) v * * ∈ T u * * , u * * minimizes min v∈T u J 1 (u, v).
The min-min Stackelberg strategy is stable to a deviation of the controls and allows the leader to reach a criterion value at least as small as that associated with a Nash equilibrium [START_REF] Leitmann | On generalized Stackelberg strategies[END_REF]. This definition, considered in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF] is suitable in team optimization problem [START_REF] Bas | Team-optimal closed-loop Stackelberg strategies in hierarchical control problems[END_REF], that is games where the follower selects in his/her rational reaction set the control which minimizes the utility function of the leader. As mentioned in [START_REF] Leitmann | On generalized Stackelberg strategies[END_REF], the follower could nonetheless be interested not only in minimizing his/her own cost (in response to the leader's control) but also in maximizing that of the leader. We thus consider the min-max Stackelberg strategy. Definition 3. A min-max Stackelberg strategy (u * , v * ) is defined by the minimization problem

(2.5) v * ∈ T u * , u * minimizes max v∈T u J 1 (u, v).
A min-max Stackelberg strategy can be interpreted as the minimization of the leader's criterion whenever the worst choice of the follower's control among the rational reaction set occurs. It can be viewed as a robust or risk-averse property with respect to the choice of the follower, which could be crucial in automatic control for robust control such as H 2 /H ∞ -control [START_REF] Zhou | Robust and Optimal Control[END_REF][START_REF] Limebeer | A Nash game approach to mixed H 2 /H∞ control[END_REF].

Remark 1. These two definitions coincide whenever T is a single-valued mapping. Considering that T is a single-valued mapping is a widespread assumption in the literature simplifying the arguments. Here we do not assume that and we make a weaker assumption (Assumption 1 in Section 3.2) that is related to the criterion J 1 and the set T . Assumption 1 will be discussed in Section 3.2. Note moreover that, in the Linear-Quadratic case (investigated in Section 3.4 and more specifically in Section 4), T is a single-valued mapping and hence both definitions min-min and min-max coincide in that case.

The structure of the controls has to be formalized to make precise the induced optimization problems:

• Whenever the controls are only functions of time t, that is u |t = u(t) and The controls are thus designed along the trajectory x(t) associated with the Stackelberg solution. • Considering u |t = u(t, x) and v |t = v(t, x), defined for every x ∈ R n , and not only along the trajectory x(t) corresponds to the concept of feedback Stackelberg solution, in the spirit of the dynamic programming approach [START_REF] Bas | Feedback equilibria in differential games with structural and modal uncertainties[END_REF][START_REF]On the Stackelberg strategy in nonzero-sum games[END_REF][START_REF] Simaan | Additional aspects of the Stackelberg strategy in nonzero-sum games[END_REF], even though dynamic programming does not apply rigorously to such a Stackelberg strategy, due to time inconsistency [START_REF] Cruz | Survey of Nash and Stackelberg equilibrium strategies in dynamic games[END_REF].

v |t = v(t)
As said above, in this paper we consider closed-loop Stackelberg strategies. Note that, in the linear quadratic case (see Section 3.4), the values of the closed-loop Stackelberg controls and feedback Stackelberg controls coincide along the associated trajectory.

Within the framework of a closed-loop information structure, the evolution law of the game state given by (2.1) is written as

(2.6) ẋ(t) = f (t, x(t), u(t, x(t)), v(t, x(t))) , x(0) = x 0 .
Moreover the sets U and V can be specified such that, for every couple (u, v) ∈ U ×V, the associated trajectory x(•), solution of (2.6), is well defined on T . Throughout the paper, for the sake of clarity, we use the notation u x = ∂u ∂x to denote the Jacobian of u(t, x) with respect to the second variable x. We thus have

U = {u(•, •) ∈ L ∞ (T × X , R m1 ),
V = {v(•, •) ∈ L ∞ (T × X , R m2 ), such v(t, x(t)) is continuous in x(t) and
piecewise continuous in t}

The main difficulty in a closed-loop Stackelberg strategy is the presence of the partial derivative ∂u * ∂x in the necessary conditions for the follower. Different alternatives, surveyed e.g. in [START_REF]Dynamic Noncooperative Game Theory[END_REF], have been proposed in the literature to overcome the difficulty raised by the presence of the partial derivative ∂u * ∂x in the necessary conditions for the follower. The first approach consists in finding an equivalent team problem leading to a global minimization of the leader's cost and obtaining a particular representation of the leader's control [START_REF] Bas | Team-optimal closed-loop Stackelberg strategies in hierarchical control problems[END_REF]. The second approach consists in determining the follower's rational reaction set and the necessary conditions for the leader optimizing a dynamical problem over an infinite dimensional strategy space subject to dynamical constraints (evolution of the state vector and follower's rational reaction set). In [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF], this problem is handled using a variational method, which however does not lead to all solutions. In this paper, based on the Pontryagin Minimum Principle, we derive necessary conditions for a min-max and min-min Stackelberg strategies, in the sense discussed formerly. Our study permits to compute the values of the controls u * (t, x(t)) and v * (t, x(t)) along the optimal trajectories. We do not provide an expression of the Stackelberg controls u * (t, x) and v * (t, x) for every x, except in the linear quadratic case (see Section 3.4) where our main result can be made more precise and more explicit. Finally, using the theory of focal points, we provide sufficient conditions for local optimality (which are global in the linear quadratic case).

Necessary conditions for a min-max and min-min Stackelberg strategies

Due to the hierarchy between the two players, necessary conditions are first established for the follower, and then for the leader.

3.1. For the follower. The best response set or rational reaction set T of the follower defined by Definition 1 does not depend on the choice of min-max or minmin Stackelberg strategy. The best response set of the follower T u * involved in the definition of Stackelberg strategy (2.4) or (2.5) (implies that, for a fixed control u * , the control v * of the follower solves the following optimization problem:

Problem 1. min v∈V J 2 (u * , v), subject to (3.7) ẋ(t) = f (t, x(t), u * (t, x(t)), v(t, x(t))), x(0) = x 0 .
Necessary conditions for the existence of an optimal solution of Problem 1 for the follower are derived in the next proposition, proved in Appendix.

Proposition 1. Consider a closed-loop min-max Stackelberg pair of controls (u * , v * ) (or respectively min-min Stackelberg pair of controls (u * * , v * * )) for the system (3.7), associated with the trajectory x(•), then there exists an absolutely continuous mapping p 2 : [0, t f ] → R n , being a non trivial line vector, such that

0 = ∂H 2 ∂v (t, x(t), u * (t, x(t)), v(t, x(t))) (3.8) = p 2 (t) ∂f ∂v (t, x(t), u * (t, x(t)), v(t, x(t))) + ∂L 2 ∂v (t, x(t), u * (t, x(t)), v(t, x(t))), ṗ2 (t) = - dH 2 dx (t, x(t), u * (t, x(t)), v(t, x(t)), p 2 (t)) (3.9) = -p 2 (t) ∂f ∂x (t, x(t), u * (t, x(t)), v(t, x(t))) - ∂L 2 ∂x (t, x(t), u * (t, x(t)), v(t, x(t))) -p 2 (t) ∂f ∂u (t, x(t), u * (t, x(t)), v(t, x(t))) ∂u * ∂x (t, x(t)) - ∂L 2 ∂u (t, x(t), u * (t, x(t)), v(t, x(t))) ∂u * ∂x (t, x(t)), p 2 (t f ) = ∂g 2 (x(t f )) ∂x , (3.10)
where H 2 denotes the Hamiltonian of the follower, The set valued mapping T , which is defined by equations (3.8)-(3.10) and satisfies T u ⊆ T u, will be used in the next subsection to derive necessary conditions for the leader, under a weak assumption on T , T and the criterion J 1 , as explained next.

H 2 (t, x, u, v, p 2 ) = p 2 f (t, x, u, v)+ L 2 (t, x, u, v).
3.2. For the leader. From the leader's point of view, unlike the follower's one, the optimization problems related to min-max and min-min Stackelberg strategies differ. We first consider the min-max Stackelberg strategy and we make the following assumption, needed to derive Proposition 2.

Assumption 1. (3.11) J 1 (u, v ) ≤ J 1 (u, v), ∀v ∈ T u, v ∈ T u, u ∈ U * nb , where U * nb denotes a neighborhood of u * in U. Proposition 2.
Consider a pair of controls (u * , v * ) associated with a min-max Stackelberg solution. The control u * is defined by Equation (2.5), that is

(3.12) u * ∈ arg min u∈U max v∈T u J 1 (u, v).
Under Assumption 1, there holds

(3.13) u * ∈ arg min u∈U max v∈T u J 1 (u, v).
This proposition is proved in Appendix.

In the same way, let us now consider the min-min Stackelberg strategy and make the following assumption necessary to derive Proposition 3.

Assumption 2. (3.14) J 1 (u, v ) ≥ J 1 (u, v), ∀v ∈ T u, v ∈ T u, u ∈ U * * nb , where U * *
nb denotes a neighborhood of u * * in U. Proposition 3. Consider a pair of controls (u * * , v * * ) associated with a min-min Stackelberg solution. The control u * * is defined by Equation (2.4), that is

(3.15) u * * ∈ arg min u∈U min v∈T u J 1 (u, v).
Under Assumption 2, there holds

(3.16) u * * ∈ arg min u∈U min v∈T u J 1 (u, v).
Remark 3. Assumption 1 and Assumption 2, dedicated respectively to min-max and min-min Stackelberg strategies, are weaker than assuming that T is reduced to a singleton and differ only by their sign. It preserves the diversity between these both frameworks. Note thus that the Assumption 2 has been already assumed in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF] to solve the min-min Stackelberg strategy.

Remark 4. Note that, under standard assumptions such as the convexity of J 2 [START_REF] Lee | Foundations of Optimal Control Theory[END_REF]Chap. 5], or the fact that the set T u * be reduced to a singleton (as in the linear quadratic case, see Section 3.4), or the fact that T u = T u (as assumed in [START_REF] Bas | Feedback equilibria in differential games with structural and modal uncertainties[END_REF][START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF] for example), the necessary conditions for the follower are also sufficient (see [START_REF] Lee | Foundations of Optimal Control Theory[END_REF]Chap. 5]) and Assumption 1 and Assumption 2 are fullfilled. In the linear quadratic case in particular, Assumption 1 and Assumption 2 are automatically satisfied.

Propositions 2 and 3 stress out that in the constraints of the leader optimization problem for min-max and min-min Stackelberg strategies, the set T u could be replaced by the set T u, without loss of generality. Assume that Equation (3.8) is solvable, but admits several (local) solutions. Then, assume that f and L 2 are C 2 with respect to the variable v, that is

∂H 2 ∂v is C 1 with respect to v. If the strict
Legendre condition holds at every (local) solution, i.e. ∂ 2 H 2 ∂v 2 is positive definite, then it follows from the Implicit Function Theorem that, locally, every solution v can be written as

(3.17) v(t, x) = S(t, x(t), p 2 (t), u * (t, x(t))) ∈ T u * ,
with S continuous with respect to t and C 1 with respect to x and p 2 . These solutions being isolated, the set of all these (local) solutions is discrete. Then, our main results apply for min-max (respectively for min-min) Stackelberg strategies by selecting among the discrete set of solutions (3.17) the one maximizing (respectively minimizing) the criterion J 1 of the leader. We stress again that, in the linear quadratic case (see Section 3.4), there exists a unique global solution. This allows to obtain the same solution for both min-max and min-min Stackelberg strategies. The leader, with his top hierarchical position with respect to the follower, can impose the control of the follower. The leader knows the reaction of the follower, i.e., he knows the function S. Then the leader seeks to minimize his own criterion where v is replaced by the function S. Using the notations L1 (t, x, p 2 , u) = L 1 (t, x, u, S(t, x, p 2 , u)) and

(3.18) J1 (u) = t f 0 L1 (t, x(t), p 2 (t), u(t, x(t)))dt + g 1 (x(t f )),
the following problem is considered:

(3.19) min u∈U J1 (u)
under two dynamical constraints:

ẋ(t) = f (t, x(t), u(t, x(t)), S(t, x(t), p 2 (t), u(t, x(t)))), (3.20) = F 1 (t, x(t), p 2 (t), u(t, x(t))), ṗ2 (t) = -p 2 (t) ∂f ∂x (t, x(t), u(t, x(t)), S(t, x(t), p 2 (t), u(t, x(t)))) (3.21) - ∂L 2 ∂x (t,
(0) = x 0 , p 2 (t f ) = ∂g 2 ∂x (x(t f )). Denote L2 (t, x, p 2 , u) = L 2 (t, x, u, S(t, x, p 2 , u)), F 21 (t, x, p 2 , u) = -p 2 ∂F 1 ∂x (t, x, p 2 , u) - ∂ L2 ∂x (t, x, p 2 , u) and finally F 22 (t, x, p 2 , u) = -p 2 ∂F 1 ∂u (t, x, p 2 , u) - ∂ L2 ∂u (t, x, p 2 , u).
Due to the nonclassical term u x , the usual Pontryagin Minimum Principle (see [START_REF] Lee | Foundations of Optimal Control Theory[END_REF]) cannot be applied. However it is possible to adapt its proof and derive a version of the Pontryagin Minimum Principle adapted to the system (3.

« T (3.24) -λ 1 (t) ∂F 1 ∂x (t, x(t), p 2 (t), u(t, x(t))) -λ • ∂ L1 ∂x (t, x(t), p 2 (t), u(t, x(t))), λ2 (t) = -λ 2 (t) " ∂F 21 ∂p 2 (t, x(t), p 2 (t), u(t, x(t))) + ∂F 22 ∂p 2 (t, x(t), p 2 (t), u(t, x(t)))ux « T (3.25) -λ 1 (t) ∂F 1 ∂p 2 (t, x(t), p 2 (t), u(t, x(t))) -λ • ∂ L1 ∂p 2 (t, x(t), p 2 (t), u(t, x(t))),
for almost every t ∈ [0, t f ]. Moreover, the following relations, called transversality conditions, hold:

(3.26) λ 2 (0) = 0, λ 1 (t f ) -λ • ∂g 1 ∂x (x(t f )) + λ 2 (t f ) ∂ 2 g 2 ∂x 2 (x(t f )) = 0.
F 22 = -p 2 ∂F 1 ∂u - ∂ L2 ∂u .
In the general case, the relation F 22 ≡ 0 is not obvious to analyze, however we will see that, in the linear quadratic case, this relation does not hold under a weak additional assumption of the Kalman type (see Proposition 7 further). In the general nonlinear case, based on the genericity strategies developed in [START_REF] Agrachev | On subanalyticity of Carnot-Carathéodory distances[END_REF][START_REF] Chitour | Propiétés génériques des trajectoires singulières[END_REF][START_REF]Genericity results for singular curves[END_REF][START_REF]Singular trajectories of control-affine systems[END_REF], we conjecture that the relation F 22 = 0 does not hold generically. The strategy would consist in derivating an infinite number of times the latter relation, to infer an infinite number of independent relations, and to use Thom's transversality Theorem. It is however not obvious to turn this fact into a proper theorem and we let this question open in the general nonlinear case. The next proposition, proved in Appendix, investigates the first case of that alternative, which is, in some sense, the generic one. Proposition 5. Under the additional assumption that the m 1 × m 1 matrix

(3.27) ∂ ∂u p 2 ∂f ∂u + ∂L 2 ∂u
T is invertible, there holds

(3.28) λ 2 ≡ 0.
Remark 5. The fact that λ 2 ≡ 0 means that the leader does not take into account the rational reaction set of the follower. It is actually not in contradiction with the hierarchical position between the leader and the follower; indeed, in this case the leader does not take into account the reaction of the follower, because he can impose his desired control to the follower. The leader is omnipotent with respect to the follower. The condition ∂F 22 ∂u invertible formalizes this privileged position of the leader.

Proposition 5, under a weak assumption, emphasizes the omnipotence of the leader leading to a degeneration of the min-max and min-min Stackelberg strategies. The hierarchical roles of the players seem to disappear. An omnipotent leader is able to impose his/her control to the other player without taking into account the rational reaction set of the follower.

These conditions happen to be more explicit in the linear quadratic case. In the next paragraph we focus on that case, and analyze more deeply the former necessary conditions. Our analysis finally leads to a more precise result on the Stackelberg controls in the linear quadratic case.

3.4. Linear Quadratic case. In this section, we focus on the linear quadratic case, due to its widespread presence in the literature [START_REF] Chen | An anticipative feedback solution for the infinite-horizon, linear-quadratic, dynamic, Stackelberg game[END_REF], and reformulate and make more explicit our previous results. Consider a linear dynamic constraint

(3.29) ẋ = Ax + B 1 u + B 2 v
and the quadratic criteria

J 1 (u, v) = t f 0 1 2 (x T Q 1 x + u T R 11 u + v T R 12 v) dt + 1 2 x(t f ) T K 1f x(t f ), (3.30) J 2 (u, v) = t f 0 1 2 (x T Q 2 x + u T R 21 u + v T R 22 v) dt + 1 2 x(t f ) T K 2f x(t f ), (3.31)
where the matrices Q i , R ij and K if are symmetric for i, j ∈ K, and Q i ≥ 0, R ii > 0, R 12 > 0, and R 21 invertible. In what follows, denote

S ij = B j R -1 jj R ij R -1 jj B T j and S i = B i R -1
ii B T i for i, j ∈ K.

(3.32) H 2 (t, x, p 2 , u, v) = p 2 (Ax + B 1 u + B 2 v) + 1 2 (x T Q 2 x + u T R 21 u + v T R 22 v).
Applying the relations (3.8), (3.9), we obtain

ṗ2 (t) = - dH 2 dx (t, x(t), p 2 (t), u(t, x(t))), (3.33) = -p 2 (t)A -x T (t)Q 2 -p 2 (t)B 1 ∂u * ∂x (t, x(t)) -u T (t, x(t))R 21 ∂u * ∂x (t, x(t)), p 2 (t f ) = x(t f ) T K 2f , (3.34) ∂H 2 ∂v = 0 = p 2 (t)B 2 + v T (t, x(t))R 22 . (3.35)
Since R 22 is invertible by assumption, the optimal control is

(3.36) v(t, x(t)) = -R -1 22 B T 2 p T 2 (t) = S(t, x(t), p 2 (t), u(t, x(t))).
3.4.2. Necessary conditions for the leader. In the case of quadratic criteria, there holds

F 1 (t, x, p 2 , u) = Ax + B 1 u -S 2 p T 2 , (3.37) F 21 (t, x, p 2 , u) = -p 2 A -x T Q 2 , (3.38) F 22 (t, x, p 2 , u) = -p 2 B 1 -u T R 21 . (3.39)
Using the expression of the optimal control of the follower (3.36), the instantaneous leader's criterion can be written as L1 (t, x, p 2 , u)

= 1 2 x T Q 1 x + u T R 11 u + p 2 S 12
∂H ∂u = 0 = λ 1 (t)B 1 -λ 2 (t) ∂u ∂x (t, x(t)) T R 21 + λ • u T (t, x(t))R 11 , (3.40) ∂H ∂u y = 0 = -λ T 2 (t) p 2 (t)B 1 + u T (t, x(t))R 21 , (3.41) λ1 (t) = -λ 1 (t)A + λ 2 (t)Q 2 -λ • x T (t)Q 1 , (3.42) λ2 (t) = λ 1 (t)S 2 + λ 2 (t) A + B 1 ∂u ∂x (t, x(t)) T -λ • p 2 (t)S 12 . (3.43)
with the transversality conditions We next prove by contradiction that λ 0 = 0. If λ 0 were equal to 0, then we would infer from (3.42)-(3.44) that λ 1 , like λ 2 is identically equal to zero by Cauchy uniqueness; thus, (λ 1 , λ 2 , λ 0 ) is trivial, and this is a contradiction with the Pontryagin Minimum Principle. From now on, we normalize the adjoint vector so that λ • = 1.

(3.44) λ 1 (t f ) = λ • x(t f ) T K 1f -λ 2 (t f )K 2f , λ 2 (0) = 0. From (3.41), as discussed in Section 3.3, either λ 2 ≡ 0 or p 2 (t)B 1 +u T (t, x(t))R 21 ≡ 0 or both along the interval [0, t f ]. Without a priori consideration about p 2 (t)B 1 + u T (t, x(t))R
From (3.40), we deduce with the invertibility of R 11 , that

(3.45) u(t, x(t)) = -R -1 11 B T 1 λ T 1 (t)
. Moreover, Equation (3.43) becomes, with λ 2 ≡ 0 along the interval [0, t f ], λ 1 (t)S 2p 2 (t)S 12 ≡ 0. Assuming that the rank of B 2 is maximal, that is, rank B 2 = m 2 (the number of the components of the control v), this relation yields

(3.46) λ 1 (t)B 2 = p 2 (t)B 2 R -1 22 R 12 . Substitution of v from (3.36) into (3.46) gives R 12 v(t, x(t)) = -B T 2 λ T 1 (t). If R 12 is invertible, then the control v admits two expressions: (3.47) v(t, x(t)) = -R -1 12 B T 2 λ T 1 (t) = -R -1 22 B T 2 p T 2 (t)
. We gather the previous necessary conditions for optimality in the following proposition. Proposition 6. For x 0 = 0, if the matrices Q i , R ij and K if are symmetric, if R 11 > 0, R 22 > 0, R 12 > 0, and R 21 invertible and if rank B 2 = m 2 (B 2 is of full rank), then the controls issued from a min-max or min-min Stackelberg strategy with a closed-loop information structure are

u(t, x(t)) = -R -1 11 B T 1 λ T 1 (t), (3.48) v(t, x(t)) = -R -1 22 B T 2 p T 2 (t) = -R -1 12 B T 2 λ T 1 (t), (3.49) with ẋ(t) = Ax(t) + B 1 u(t, x(t)) + B 2 v(t, x(t)), x(0) = x 0 , (3.50) ṗ2 (t) = -p 2 (t)A -x T (t)Q 2 -p 2 (t)B 1 + u T (t, x(t))R 21 ∂u ∂x (t, x(t)), (3.51) λ1 (t) = -λ 1 (t)A -x T (t)Q 1 , λ 1 (t f ) = x T f K 1f , p 2 (t f ) = x T f K 2f , (3.52) λ 1 (t)B 2 = p 2 (t)B 2 R -1 22 R 12 . (3.53)
Remark 6. As it will be justified below by Proposition 7, the case x 0 = 0 leads only to the trivial solution, which has only few interest. Thus in the sequel of the paper, x 0 is always considered non trivial to avoid the trivial optimal trajectory. From (3.41), even if λ 2 ≡ 0, two cases must be yet considered to precise necessary conditions: either p 2 B 1 + u T R 21 ≡ 0 or p 2 B 1 + u T R 21 ≡ 0. We next prove that the first case is irrelevant under some additional weak assumptions.

(3.54) x(t) ≡ λ T 1 (t) ≡ p T 2 (t) ≡ 0, ∀t ∈ [0, t f ]
. This means that there exists a unique optimal trajectory, which is trivial.

The proof of Proposition 7 (see Appendix) relies on the following lemma concerning the Kalman condition (also proved in Appendix).

Lemma 1. Assuming that the pair A T , Q 1 and one of the pairs (A, B 1 ) or (A, B 2 ) satisfy the Kalman condition, then the pair (A, B) satisfies also the Kalman condition, where (3.55)

A =   A T -Q 1 -Q 2 -S 1 -B 2 R -1 12 B T 2 -A 0 0 0 -A   ; B =   0 0 B 2 B 1 R -1 11 R 21 -B 2 R -1 22 R 12 -B 1   .
Remark 7. This means that the particular case p 2 B 1 +u T R 21 ≡ 0 can be discarded under weak assumptions on the system. The leader should be able to observe the system (pair (Q 1 , A) observable) and at least one player should be able to control the system ((A, B 1 ) or (A, B 2 ) controllable). Once again, it is emphasized that the roles of the players are not symmetric.

(3.56) λ 1 (t f )B 2 = x T (t f )K 1f B 2 = p 2 (t f )B 2 R -1 22 R 12 = x T (t f )K 2f B 2 R -1 22 R 12 , and λ1 (t)B 2 = ṗ2 (t)B 2 R -1 22 R 12 (3.57) = -λ 1 (t)A + x T (t)Q 1 B 2 (3.58) = -p 2 (t)A + x T (t)Q 2 B 2 R -1 22 R 12 -(p 2 (t)B 1 + u T (t, x(t))R 21 ) ∂u ∂x (t, x(t)) B 2 R -1 22 R 12 . (3.59) Hence along the interval T (3.60) (p 2 (t)B 1 + u T (t, x(t))R 21 ) ∂u ∂x (t, x(t)) B 2 = λ 1 (t)A + x T (t)Q 1 B 2 R -1 12 R 22 -(p 2 (t)A + x T (t)Q 2 )B 2 . Therefore, (3.53) is equivalent to (3.61)          B T 2 K 1f -R 12 R -1 22 B T 2 K 2f x(t f ) = 0, p 2 (t)B 1 + u T (t, x(t))R 21 ∂u ∂x (t, x(t)) B 2 ≡ λ 1 (t)A + x T (t)Q 1 B 2 R -1 12 R 22 -(p 2 (t)A + x T (t)Q 2 )B 2 .
Equation (3.60) permits to derive an expression of ∂u ∂x , since

p 2 B 1 + u T R 21 ≡ 0 (3.62) (p 2 B 1 + u T R 21 ) ∂u ∂x ≡ w 2 + w 2 , with (3.63) w 2 ≡ λ 1 (t)A + x T (t)Q 1 B 2 R -1 12 R 22 -(p 2 (t)A + x T (t)Q 2 )B 2 (B T 2 B 2 ) -1 B T 2
, and (w 2 ) T ∈ Ker B T 2 (arbitrary). The constraint (3.56) translates into a constraint on the set of initial points x 0 ∈ R n from which a solution starts.

Lemma 2. The optimal solutions must emanate from initial conditions x 0 lying in a subspace of R n of codimension m 2 (at most).

It should be noted that, given a starting point x 0 lying in the subspace of Lemma 2, there exists a unique trajectory starting from x 0 , but it is achieved by all controls which satisfy the relation (3.60). An optimal trajectory induces several possible ∂u ∂x .

This fact appears in [START_REF] Medanic | Closed-loop Stackelberg strategies in linear-quadratic problems[END_REF] where it is assumed that the initial state is uniformly distributed over the unit sphere and replacing the optimization criterion with its mean value over the initial state.

Remark 8. In the case of an optimization problem without terminal criteria, the relation (3.56) does not reduce the set of initial states x 0 associated with optimal trajectories. We gather all previous results in the following theorem.

Theorem 1. Under the following assumptions:

• x 0 = 0,

• Q i , R ij and K if are symmetric, with Q i ≥ 0, • R 11 > 0, R 22 > 0, R 12 > 0 and R 21 is invertible,
• the pair (A T , Q 1 ) and one of the pairs (A, B 1 ) or (A, B 2 ) satisfy the Kalman condition, • rank B 2 = m 2 (B 2 of full rank), the optimal trajectory satisfies the necessary conditions

(3.64) u(t, x(t)) = -R -1 11 B T 1 K 1 (t)x(t), v(t, x(t)) = -R -1 12 B T 2 K 1 (t)x(t), with (3.65) ẋ(t) = A -(B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 )K 1 (t) x(t), x(0) = x 0
, where K 1 is the unique solution of the matrix differential equation

(3.66) K1 (t) = -K 1 (t)A-A T K 1 (t)-Q 1 +K 1 (t)(B 1 R -1 11 B T 1 +B 2 R -1 12 B T 2 )K 1 (t), K 1 (t f ) = K 1f . Furthermore ∂u(t, x(t)) ∂x satisfies along the interval [0, t f ] (3.67) (p 2 (t)B 1 + u T (t, x(t)R 21 ) ∂u ∂x (t, x(t))B 2 = (λ 1 (t)A + x T (t)Q 1 )B 2 R -1 12 R 22 -(p 2 (t)A + x T (t)Q 2 )B 2 , where (3.68) ṗ2 (t) = -p 2 (t)A -x T (t)Q 2 -(p 2 (t)B 1 + u T (t, x(t))R 21 ) ∂u ∂x (t, x(t)), p 2 (t f ) = x T (t f )K 2f and (B T 2 K 1f -R 12 R -1 22 B T 2 K 2f )x(t f ) = 0.
Theorem 1 provides rigorous necessary conditions for closed-loop min-max and min-min Stackelberg solutions of generic linear quadratic games. Up to now this problem has remained open and was only partially solved in particular cases in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF]. It should be stressed again that the trajectory associated with closed-loop Stackelberg solution is unique, nevertheless it induces several possible u x , which satisfy Equations (3.67)-(3.68) and are completely characterized by (3.62). This degree of freedom in the choice of u x , leading to the same trajectory, requires an additional objective, e.g. arguments related to the robustness or the sensitivity of the min-max or min-min Stackelberg solution.

Sufficient conditions

In this section, using elements of focal point theory, we derive sufficient optimality conditions, first for the leader, and then for the follower in the case of Linear Quadratic games.

Preliminary comments, focal times. The optimization problem of the leader is min

u Ĵ1 (u) where (4.69)              Ĵ1 (u) = 1 2 t f 0 x T (t)Q 1 x(t) + u T (t, x(t))R 11 u(t, x(t)) + p 2 (t)S 12 p T 2 (t) dt + 1 2 x T (t f )K 1f x(t f ), ẋ(t) = Ax(t) -S 2 p T 2 (t) + B 1 u(t, x(t)), ṗT 2 (t) = -A T p T 2 (t) -Q 2 x(t) -w T p 2 (t)B 1 + u T (t, x(t))R 21 T ,
with x(0) = x 0 and p 2 (t f ) = x T (t f )K 1f . When p 2 (t)B 1 + u T (t, x(t))R 21 ≡ 0, the control w is cheap (see [START_REF] Bonnard | The role of singular trajectories in control theory[END_REF] for the concept of cheap control), since it only appears in the dynamics of p 2 , and nowhere else (it does also not appear in the cost); then, we rather consider p 2 as a control. Note that this is a particular case of the so-called Goh transformation (see [START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF][START_REF] Agrachev | Abnormal sub-Riemannian geodesics: Morse index and rigidity[END_REF][START_REF] Bonnard | The role of singular trajectories in control theory[END_REF][START_REF] Trélat | Asymptotics of accessibility sets along an abnormal trajectory[END_REF] for the definition and properties of the Goh transformation, related to singular trajectories or abnormal extremals). Actually, in what follows we consider ξ = B T 2 p T 2 as a control. Then the problem (4.69) can be rewritten as min (u,ξ) Ĵ1 (u, ξ) where (4.70)

       ẋ(t) = Ax(t) -B 2 R -1 22 ξ(t) + B 1 u(t, x(t)), Ĵ1 (u, ξ) = 1 2 x T (t f )K 1f x(t f ) + 1 2 t f 0 x T (t)Q 1 x(t) + u T (t, x(t))R 11 u(t, x(t)) + ξ T (t)R -1 22 R 12 R -1 22 ξ(t) dt.
Remark 9. Note that this Linear Quadratic problem with controls (u, ξ) is related to the Team optimal approach in [START_REF] Bas | Team-optimal closed-loop Stackelberg strategies in hierarchical control problems[END_REF]. In this reference, the first step in the research of min-max or min-min Stackelberg strategy is to obtain the minimum of the criterion of the leader, by a team cooperation between the leader and the follower. Then the follower control is modified to achieve the minimum of the criterion of the follower.

A necessary condition for the existence of an optimal control of the problem (4.70

) is R -1 22 R 12 R -1 22 ≥ 0. It is equivalent to R 12 ≥ 0, since R 22 is positive definite. When t f is small, R 12 >
0 is a sufficient condition for the existence of an optimal control (see e.g. [START_REF] Lee | Foundations of Optimal Control Theory[END_REF][START_REF]Contrôle optimal: théorie et applications[END_REF][START_REF] Cesari | Optimization -Theory and applications[END_REF]). In the following, it is assumed that R 12 > 0.

Under this assumption, the optimal controls u and ξ are given by (4.71)

u(t, x(t)) = -R -1 11 B T 1 λ T 1 (t), ξ(t) = R 22 R -1 12 B T 2 λ T 1 (t).
Recall that, in order to characterize focal points (for a definition and properties of focal points we refer the reader to [START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF][START_REF] Bonnard | The role of singular trajectories in control theory[END_REF][START_REF] Bonnard | Mécanique céleste et contrôle de systèmes spatiaux[END_REF]), we consider the variational system

(4.72) δ ẋ(t) = Aδx(t) -B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 δλ T 1 (t), δ λ1 (t) = -δλ 1 (t)A -δx T (t)Q 1 .
By definition, the first focal time t c > 0 along the trajectory x(t) associated with the controls (u, ξ) is the first positive time at which there exists a solution (δx, δλ 1 ) satisfying (recall that x(0) = x 0 is fixed)

(4.73) δx(0) = 0, δλ 1 (t c ) = δx T (t c )K 1f .
It is well known that this condition is equivalent to

(4.74) K(t) -→ t→tc; t<tc +∞,
where K(t) is the solution of the Riccati differential equation

(4.75) K(t) = K(t)A + A T K(t) + Q 1 -K(t) B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 K(t), K(0) = K 1f .
The first focal time t c is a finite escape time for the Riccati differential equation (4.75). Note that K(t) = K 1 (t f -t), where K 1 (t) is defined by (3.66). Lemma 3. Rigorously, since the first focal time is defined by an infimum, a remark is due to comment on its existence. It is well known (see [START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF][START_REF] Bonnard | The role of singular trajectories in control theory[END_REF][START_REF] Bonnard | Mécanique céleste et contrôle de systèmes spatiaux[END_REF]) that, if R 11 > 0 and R 12 > 0 then the first focal time t c is well defined, and t c is either a positive real number, or is equal to +∞. The optimization problem for the follower is min Ĵ2 where (4.76)

       ẋ(t) = Ax(t) + B 1 u(t, x(t)) + B 2 v(t, x(t)), x(0) = x 0 , Ĵ2 = 1 2 x T (t f )K 2f x(t f ) + 1 2 t f 0 x T (t)Q 2 x(t) + u T (t, x(t))R 21 u(t, x(t)) + v T (t, x(t))R 22 v(t, x(t)) dt with v(t, x(t)) = -R -1 22 B T 2 p T 2 (t) where p 2 (t f ) = x T (t f )K 2f and (4.77) ṗ2 (t) = -p 2 (t)A -x T (t)Q 2 -p 2 (t)B 1 + u T (t, x(t))R 21 ∂u ∂x (t, x(t)).
The variational system along the trajectory x(•) is

δ ẋ(t) = Aδx(t) + B 1 ∂u ∂x δx -S 2 p T 2 (t), (4.78) δ ṗ2 (t) = -δp 2 (t)A -δx T (t)Q 2 (4.79) -p 2 (t)B 1 + u T (t, x(t))R 21 ∂ 2 u ∂x 2 (t, x(t))δx -δp 2 (t)B 1 + ∂u ∂x (t, x(t))δx(t) T R 21 ∂u ∂x (t, x(t)).
Here, due to the freedom in the choice of w 2 to obtain u x by relation (3.62), we choose u(t, x(t)) affine with respect to x(t), thus ∂ 2 u ∂x 2 = 0. Equation (4.79) then rewrites (4.80)

δ ṗ2 (t) = -δp 2 (t)A-δx T (t)Q 2 -δp 2 (t)B 1 + ∂u ∂x (t, x(t))δx(t) T R 21 ∂u ∂x (t, x(t)).
By definition, the first focal time t c along the trajectory x(t) associated with the control v is the first time at which there exists a solution (δx, δp 2 ) of (4.78)-( 4.79) such that δx(0) = 0 and δp 2 (t c ) = δx T (t c )K 2f . For each choice of admissible term u x (that is choice of w 2 ) satisfying relation (3.67), there exists a first focal time t c . 4.2. Sufficient conditions for min-max and min-min Stackelberg Strategies. We gather the previous remarks in the following result.

(p 2 B 1 + u T R 21 )u x = w 2 + w 2 . Let (4.81) T * = min (t c , t c ) > 0,
where t c is the first focal time of the Riccati differential equation (3.66) and t c is the first focal time of the system (4.78)-(4.80), induced by u x , that is by w 2 . For every t f < T * , there exists a unique solution of the Riccati differential equation (3.66). Denoting x(t, x 0 ) the obtained trajectory, let

(4.82) H = x 0 ∈ R n B T 2 K 1f -R 12 R -1 22 B T 2 K 2f x(t f , x 0 ) = 0 .
Then, for every x 0 ∈ H, there exists a unique optimal solution of the optimization problem (4.76) on [0, t f ] associated with w 2 . The optimal controls (u * , v * ) associated with this unique optimal trajectory satisfy (p 2 B 1 + u T R 21 )u x = w 2 + w 2 and furthermore

(4.83) u(t, x(t)) = -R -1 11 B T 1 K 1 (t)x(t), v(t, x(t)) = -R -1 12 B T 2 K 1 (t)x(t).
In addition, for every x 0 / ∈ H, there exists no optimal trajectory starting from x 0 .

Remark 11. Theorem 2 is a result of existence of closed-loop min-max and minmin Stackelberg strategies for linear-quadratic differential games, which is new, to the best of our knowledge.

Remark 12. The sufficient conditions for optimality are developed in the Linear Quadratic case, and are global in that case. It is also, by the same argument, possible to express similar sufficient conditions in the general case of nonlinear criteria. However they are not developed here, because their expressions are more technical and because they lead only to local optimality results (see [START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF] or [16, chap. 9]).

Remark 13. The assumption R 12 > 0 is required to derive Theorem 2. This assumption is used in a crucial way in order to derive Lemma 3 (more precisely, to derive the inequality (1.150)). It is natural to make such an assumption when inspecting the minimization criterion Ĵ1 (u, ξ) defined by (4.70): indeed, as explained few lines above (4.70), the problem degenerates into a cheap control problem. In this sense, ξ = B T 2 p T 2 may then be considered as a control, and therefore it is clear that one has to assume that R 12 > 0 in order to ensure nice coercivity properties for the quadratic criterion Ĵ1 (u, ξ). Remark 14. From Remark 10, the assumption Q 1 ≥ 0 ensures that t c = +∞. A lower bound of T * = min(t c , t c ) corresponds to a lower bound of t c the first focal time of the non-linear variational system (4.78)-(4.80). It depends implicitly on the choice of u x , that is the choice of w 2 . The problem of determining a lower bound of t c is open. 4.3. Extension: weighting of u x in criteria. The problem is degenerated, because for each x 0 ∈ H, there may exist an infinite choice of terms ∂u * ∂x . A way to yield a unique ∂u * ∂x is to include a weight on the term ∂u * ∂x in the criterion J 1 of the leader, as in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF]. Then the leader takes into account a restriction on the Jacobian of its control. The leader is no more omnipotent.

The new criterion of the leader is then

J 1 (u, v) = 1 2 x(t f ) T K 1f x(t f ) + 1 2 t f 0 x T (t)Q 1 x(t) (4.84) +u T (t, x(t))R 11 u(t, x(t)) + v T (t, x(t))R 12 v(t, x(t)) + m1 j=1 ∂u j ∂x (t, x(t)) R j ∂u j ∂x (t, x(t)) T dt,
where u j are the m 1 components of the control u, and

R j ∈ R n×n , (∀j = 1, • • • , m 1 )
are symmetric positive definite matrices. Nothing changes for the follower. However the necessary conditions for the leader are modified as follows

∂H ∂u = 0 = λ 1 (t)B 1 -λ 2 (t) ∂u ∂x (t, x(t)) T R 21 + λ • u T (t, x(t))R 11 , (4.85) ∂H ∂u y = 0 = λ 2 (t) B T 1 p T 2 (t) + R 21 u(t, x(t)) j (4.86) + ∂u j ∂x (t, x(t))R j j=1,••• ,m1
.

The other necessary conditions (3.24) and (3.25) are the same. Equations (4.85) and (4.86) are easily solvable, without considering different cases. In this framework, λ 2 cannot be trivial. As in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF], we infer from (4.86) that (4.87)

∂u * j ∂x (t, x(t)) = B T 1 p T 2 (t) + R 21 u(t, x(t)) j λ 2 (t)R -1 j j=1,••• ,m1
.

For the sake of simplicity, we next assume, as in [START_REF] Papavassilopoulos | Nonclassical control problems and Stackelberg games[END_REF], that R j = R > 0, for every

j = 1, • • • , m 1 . Then (4.88) ∂u * ∂x (t, x(t)) = B T 1 p T 2 (t) + R 21 u(t, x(t)) λ 2 (t)R -1 .
Plugging this expression into (4.85), we get (4.89)

R 11 u(t, x(t)) = -B T 1 λ T 1 (t) + R 21 B T 1 p T 2 (t)λ 2 (t)R -1 λ T 2 (t) + R 2 21 uλ 2 (t)R -1 λ T 2 (t), or (4.90) R 11 -λ 2 (t)R -1 λ T 2 (t)R 2 21 u(t, x(t)) = -B T 1 λ T 1 (t) + R 21 B T 1 p T 2 (t)λ 2 (t)R -1 λ T 2 (t). Remark 15. For t = 0, λ 2 (0) = 0, then R 11 -λ 2 (0)R -1 λ T 2 (0)R 2 21 = R 11 > 0 is invertible. For t ≥ 0 small enough, the matrix R 11 -λ 2 (t)R -1 λ T 2 (t)R 2 21 is invertible. As long as R 11 -λ 2 (t)R -1 λ T 2 (t)R 2 21 is invertible, the optimal control is (4.91) u(t, x(t)) = R 11 -λ 2 (t)R -1 λ T 2 (t)R 2 21 -1 × -B T 1 λ T 1 (t) + R 21 B T 1 p T 2 (t)λ 2 (t)R -1 λ T 2 (t)
The nonlinear optimization problem becomes

ẋ(t) = Ax(t) -S 2 p T 2 (t) + B 1 R 11 -λ 2 (t)R -1 λ T 2 (t)R 2 21 -1 (4.92) × -B T 1 λ T 1 (t) + R 21 B T 1 p T 2 (t)λ 2 (t)R -1 λ T 2 (t) , ṗ2 (t) = -p 2 (t)A -x T (t)Q 2 -p 2 (t)B 1 + u T (t, x(t))R 21 2 λ 2 (t)R -1 , (4.93) λ1 (t) = -λ 1 (t)A + λ 2 (t)Q 2 -x T (t)Q 1 , (4.94) λ2 (t) = +λ 2 (t) A T + R -1 λ T 2 (t) p 2 (t)B 1 + u T (t, x(t))R 21 B T 1 (4.95) +λ 1 (t)S 2 -p 2 (t)S 12 .
with boundary conditions

x(0) = x 0 , p 2 (t f ) = x T (t f )K 2f , (4.96) λ 2 (0) = 0, λ 1 (t f ) = x T (t f )K 1f -λ 2 (t f )K 2f . (4.97)
Remark 16. For R = γId, if we let γ tend to +∞, then we recover the necessary conditions for the strategy of Stackelberg with an open-loop information structure. Note that this coincidence is obtained only by taking the limit γ → +∞ without modifying the criterion of the leader.

Remark 17. These conditions are necessary conditions. As previously, the theory of focal points leads to sufficient conditions associated with the min-max or minmin Stackelberg strategy with closed-loop information structure including a weight for u x in the criterion of the leader, namely, given x 0 ∈ R n . For t f less than the global focal time of the system, there exists only one trajectory starting from x 0 solution of (4.92)-(4.97) associated with the optimal control (u, u x ) (4.91)-(4.88).

Conclusion

In this paper the min-max and min-min Stackelberg strategies with a closedloop information structure are studied. The framework is restricted to two-player differential games. Necessary conditions for the existence of a closed-loop minmax and min-min Stackelberg strategies are derived by considering all cases. It is also shown that they may degenerate whenever the leader is omnipotent and can impose his control to the follower. The focal times theory provides sufficient conditions for the optimization problems of the two players. The Linear Quadratic case is used to illustrate the obtained necessary and sufficient conditions. Moreover in this linear quadratic case, the control u(t, x) is obtained for each state x. An extension is proposed to allow an optimal trajectory starting from any initial state by including, in the criterion, the Jacobian of his/her control in the criterion of the leader.

Appendix A. Proof of Theorems

Preliminaries

We start with preliminaries essentially borrowed from [START_REF] Lee | Foundations of Optimal Control Theory[END_REF][START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF][START_REF]Contrôle optimal: théorie et applications[END_REF][START_REF] Bonnard | The role of singular trajectories in control theory[END_REF]. First of all, consider a usual optimal control problem:

(1.98) min C(u) under ẋ(t) = f (t, x(t), u(t)), with C(u) = t f 0 f 0 (t, x(t), u(t))dt, where x(t) ∈ R n and u(t) ∈ R m . A usual way to derive the Pontryagin Minimum Principle for such an optimal control problem is to extend the control system with a new state variable representing the cost, in the following way. Define the extended state z = x x 0 ∈ R n+1 , with x 0 (0) = 0 and f = f f 0 . Consider the extended control system (1.99) ż(t) = f (t, z(t), u(t)).

The associated end-point mapping e z0,t f at time t f is defined by

(1.100) e z0,t f : U -→ R n+1 v -→ z u (t f ),
with z u the trajectory solution of (1.99) associated to the control u, and U is the set of admissible controls. The crucial remark which is at the basis of the proof of the Minimum Principle is the following: if a trajectory x(.), associated with a control u on [0, t f ] is optimal, then the end-point mapping e z0,t f is not locally surjective at u, then it follows from an implicit function argument that the first differential of the end-point mapping at u is not surjective (at least in the case where there is no constraint on the controls). This fact leads to a Lagrange multipliers type equation which finally leads to the well known Pontryagin Minimum Principle (see [START_REF]Contrôle optimal: théorie et applications[END_REF] for details).

In the present paper we are not dealing with such a classic optimal control problem, however the previous reasoning may be adapted, even though our controls now depend also on x(t), and we first derive a proof of Proposition 1.

Proof of Proposition 1.

Define the extended state z = x x ∈ R n+1 (and the projector q(z) = x), where x is the instantaneous cost associated with the criterion of the follower such that ẋ(t) = L 2 (t, x(t), u * (t, x(t)), v(t, x(t)), x(0) = 0, which leads to the dynamic of the extended state z:

(1.101) ż(t) = f (t, z(t), v(t, x(t))) = f L 2 .
It is pointed out that the function t → u * (t, x(t)) is fixed and the control of the follower v(t, x(t)) is the optimization variable. Definition 4. The end-point mapping at time t f of system (1.101) with the initial state z 0 = x 0 0 is the mapping

(1.102) e z0,t f : V -→ R n+1 v -→ z v (t f )
where z v is the solution of (1.101), associated to v, starting from z 0 , and V is the set of admissible controls.

To compute the Fréchet first derivative, consider a fixed control δv such that v and v + δv belong to V and denote z + δz the trajectory associated with the latter control [START_REF]Contrôle optimal: théorie et applications[END_REF][START_REF] Lee | Foundations of Optimal Control Theory[END_REF]. An expansion to the first order of f leads to d(z + δz) dt = f t, z + δz, u * (t, q(z + δz)), v(t, q(z + δz)) + δv(t, q(z + δz)) + o(δz), = f (t, z, u * (t, q(z)), v(t, q(z))) + fz (t, z, u * (t, q(z)), v(t, q(z)))δz + fu (t, z, u * (t, q(z)), v(t, q(z)))u * y + o(δz). Furthermore, to the first order, v(t, q(z + δz)) = v(t, q(z) + q z (z)δz + o(δz)) = v(t, q(z)) + v y (t, q(z))q z (z)δz + o(δz), u * (t, q(z + δz)) = u * (t, q(z)) + u * y (t, q(z))q z (z)δz + o(δz), δv(t, q(z + δz)) = δv(t, q(z)) + o(δz).

Plugging these Taylor series expansions into relation (1.103), we have, at the first order, d(δz) dt = fz δz + fu u * y q z δz + fv δv + fv v y q z (z)δz (1.103) = fz + fu u * y q z + fv v y q z (z)

a(t) δz + fv b(t)
δv. (1.104) Using the transition matrix Φ(t) satisfying Φ(t) = aΦ(t) and Φ(0) = Id (Id denoting the identity matrix), it follows that

(1.105) de z0,t f (v) • δv = δz(t f ) = Φ(t f ) t f 0 Φ -1 (s)b(s)δvds.
If a closed-loop Stackelberg control v * ∈ V of the follower is optimal, then the first derivative of the end-point mapping, de z0,t f (v * ), is not surjective, and hence there exists a line vector φ ∈ R n+1 , φ = 0 such that (1.106) φ • de z0,t f (v * )δv = 0, ∀ δv ∈ V.

Set φ(t) = φΦ(t f )Φ -1 (t), then the relation (1.106) is satisfied for every control δv, and thus Furthermore, derivating φ(t) = φΦ(t f )Φ -1 (t) with respect to t, we obtain (1.108) φ(t) = -φ(t)a = -φ(t) fz + fu u * y q z + fv v y q z (z) = -φ(t) fz + fu u * y q z , the last equality holds due to relation (1.107). Denoting φ(t) = p 2 (t) p • 2 (t) , we obtain that p • 2 is a constant scalar. Finally the initial condition z 0 being fixed and the final condition z(t f ) being free, the standard transversality condition associated with the system (1.101) implies that p • 2 = 0, because φ = p 2 (t f ) p • 2 = 0. We normalize the costate vector so that p

• 2 = 1, since (p 2 (t f ), p • 2 )
is defined up to a multiplicative scalar. The transversality condition leads to Equation (3.10) and p 2 satisfies Equation (3.9). In addition relation (1.107) could be reformulated into Equation (3.8).

Proof of Proposition 2. For each u ∈ U, the inclusion T u ⊆ T u implies that (1.109) max

v∈T u J 1 (u, v) ≤ max v ∈T u J 1 (u, v ). With Assumption 1, in a neighborhood U * nb of u * ∈ U, we have (1.110) max v∈T u J 1 (u, v) = max v ∈T u J 1 (u, v ), ∀u ∈ U * nb .
Thus u * , defined by (3.12) is also given by (3.13).

Proof of Proposition 4.

As previously, we define the extended state

(1.111) Z = x T p 2 x • T ∈ R 2n+1 ,
where x • is the instantaneous cost associated with the criterion of the leader satisfying ẋ• (t) = L1 (t, x, p 2 , u), x • (0) = 0. The extended system is subject to the dynamics

(1.112) Ż(t) = F (t, Z, u, u T y ) =   F 1 (t, x, p 2 , u) (F 21 (t, x, p 2 , u) + F 22 (t, x, p 2 , u)u y ) T L1 (t, x, p 2 , u)   ,
where u = u(t, h(Z)) is a function of time t and of the projection h(Z) = x.

The end-point mapping at time t f of system (1.112) with initial state Z 0 is the mapping

(1.113) E Z0,t f : U -→ R 2n+1 u -→ Z u (t f )
where Z u is the solution of (1.112), associated to u, starting from Z 0 . Here U denotes the open set of controls u ∈ L ∞ ([0, t f ] × R n , R m1 ) such that the solution Z u (•) of (1.112), associated with u and starting from Z 0 , is well defined on [0,

t f ]. Note that, if F is of class C p , p ≥ 1, then E Z0,t f is also of class C p .
To compute the Fréchet first derivative, we proceed as in [START_REF]Contrôle optimal: théorie et applications[END_REF][START_REF] Lee | Foundations of Optimal Control Theory[END_REF], consider a fixed control δu on U and note Z + δZ the trajectory associated with the control u + δu. An expansion to the first order of F leads to

d(Z + δZ) dt = F t, Z + δZ, u(t, h(Z + δZ)) + δu(t, h(Z + δZ)), (1.114) u x (t, h(Z + δZ)) T + δu x (t, h(Z + δZ)) T + o(δZ).
Furthermore an expansion to the first order of the control u gives

u(t, h(Z + δZ)) = u(t, h(Z) + h Z (Z)δZ + o(δZ)) = u(t, h(Z)) + u x (t, h(Z))h Z (Z)δZ + o(δZ) Therefore, at the first order, d(δZ) dt = FZ δZ + Fu u x h Z δZ + Fu δu + Fux u xx h Z δZ + Fux δu T x (1.115) = FZ + Fu u x h Z + Fux u xx h Z A δZ + Fu B δu + Fux C δu T x
Using the transition matrix M defined by Ṁ (t) = A(t)M (t) and M (0) = Id (Id denoting the identity matrix), it follows that (1.116)

dE Z0,t f (u) • δu = δZ(t f ) = M (t f ) t f 0 M -1 (s) B(s)δu(s) + C(s)δu T y (s) ds.
If u is the control of the leader in a closed-loop min-max or min-min Stackelberg solution, then there exists a vector ψ ∈ R 2n+1 , ψ = 0 such that

(1.117) ψ • dE Z0,t f (u)δu = 0, ∀ δu ∈ U. Set ψ(t) = ψM (t f )M -1 (t), then (
x(0) p T 2 (0) = x 0 p T 2 (0) ∈ M 0 , x(t f ) p T 2 (t f ) =   x(t f ) ∂g 2 ∂x (h(Z(t f )))   ∈ M 1 ,
where

M 0 = {x 0 } × R n = x p T 2 F 0 x p T 2 = x -x 0 = 0 , (1.121) M 1 = x p T 2 F 1 x p T 2 = ∂g 2 ∂x (h(Z(t f ))) -p 2 = 0 . (1.122)
The tangent manifolds T Z(0) M 0 and T Z(t f ) M 1 are defined by (1.123)

T Z(0) M 0 = (0, α) ∈ R 2n | α ∈ R n , T Z(t f ) M 1 = β, β ∂ 2 g 2 ∂x 2 | β ∈ R n .
The transversality conditions can be written as (see [START_REF] Agrachev | Control Theory from the Geometric Viewpoint[END_REF] or [53, p. 104])

(1.124) λ(0) ⊥ T Z0 M 0 , λ(t f ) -λ • ∂g 1 ∂Z (h(Z(t f ))) ⊥ T Z(t f ) M 1 ,
and lead to Equations (3.26).

Proof Since the dynamics and the criterion do not depend on u y , we can deduce that any control u y is extremal for the optimization problem. But the relation Derivating with respect to time the relation ( The k-order derivation of (1.139) with respect to time, at time t = t f , gives (1.141)

x T (t f ) x T (t f )K 1f x T (t f )K 2f A k B = 0, ∀k ∈ N.

The assumptions of Lemma 1 are checked. It leads to the controllability of the pair (A, B), which implies that x(t f ) = 0. Furthermore the autonomous linear system in x, λ 1 and p 2 with end value conditions x(t f ) = λ T 1 (t f ) = p T 2 (t f ) = 0 imposes, by a backward integration of (1.140) (1.142) x(t) ≡ λ T 1 (t) ≡ p T 2 (t) ≡ 0, ∀t ∈ [0, t f ].

The unique optimal trajectory in this case is the trivial one.

Proof of Lemma 2.

Similarly as in the classic linear quadratic problem, we seek a solution in the form λ T 1 (t) = K 1 (t)x(t). Then, the matrix K 1 (t) ∈ R n×n must satisfy (1.143) K1 (t)x(t) + K 1 (t) Ax(t) -

B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 K 1 (t)x(t) = -A T K 1 (t)x(t) -Q 1 x(t).
This relation should hold for every x, which leads to define K 1 (t) as the solution of the following Riccati differential equation (1.144)

K1 (t) = -K 1 (t)A -A T K 1 (t) -Q 1 + K 1 (t) B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 K 1 (t), K 1 (t f ) = K 1f .
The existence of a solution of the optimization problem is ensured in a standard way "a la Riccati" and by the uniqueness of an optimal trajectory. This is justified a posteriori in the following by using the theory of focal times.

Plugging λ T 1 (t) = K 1 (t)x(t) into (3.47) and (1.137), the state x(t) has the dynamical constraint (1.145) ẋ(t) = A -B 1 R -1 11 B T 1 + B 2 R -1 12 B T 2 K 1 (t) x(t) = Ãx(t), x(0) = x 0 .

Let M(t) be the transition matrix associated with (1.145). Then x(t) = M(t)x 0 . Then the constraint (3.56) becomes

(1.146) B T 2 K 1f -R 12 R -1 22 B T 2 K 2f M(t f )x 0 = 0.
This is a m 2 -codimension (at most) condition on the initial states x 0 .

Proof of Lemma 3.

From Equation (4.69), one gets x(t) = e tA x 0 -t 0 e (t-s)A B 2 R -1 22 ξ(s) -B 1 u(s) ds. There exist scalar constants C k ≥ 0 such that for a given t f > 0, for every t ∈ [0, t f ] For t f > 0 small enough,

x(t) ≤ C 1 x 0 + C 2 t f t f 0 ξ(s) 2 ds 1 2 + t f 0 u(s) 2
t f ≤ C 6 C 3 + C 5
, the criterion Ĵ1 (u, ξ) is finite, then 0 < t f < t c .

  All solutions v of Equations (3.8)-(3.10) are gathered in the set valued mapping T : U → V. Remark 2. Note that condition (3.9) may seem akin to open-loop control since it does not involve terms in ∂v ∂x . This dependency comes from condition (3.8) that implies that the open-loop and closed-loop control for the follower coincide.

3. 3 .

 3 Degeneration property. Equation (3.23) implies either that λ 2 ≡ 0 or F 22 ≡ 0 (or both) along the interval [0, t f ], where

3. 4 . 1 .

 41 Necessary conditions for the follower. The Hamiltonian associated with the follower (dynamic constraint (3.29) and criterion (3.31)) is

p 2 .

 2 The necessary conditions (3.22), (3.23), (3.24), (3.25) lead to

3. 4 . 3 .Proposition 7 .

 437 Case p 2 B 1 + u T R 21 ≡ 0. If the pair (A T , Q 1 )and one of the pairs (A, B 1 ) or (A, B 2 ) satisfy the Kalman condition, then

3. 4 . 4 .

 44 Case p 2 B 1 + u T R 21 ≡ 0. The relation (3.53) is equivalent to the following two relations:

Remark 10 .

 10 If Q 1 ≥ 0, then Equation (4.75) admits a solution on [0, +∞[. There is no finite escape time for this equation. Thus the first focal time is infinite (t c = +∞) [2, Corollary 3.6.7 and Example 3.6.8].

Theorem 2 . 2 ⊥

 22 Under the assumptions of Theorem 1, let w 2 be a function of time t such that w 2 ∈ B T . This choice of w 2 leads to design the Jacobian u x satisfying (3.62):

  t f 0 φ(t)b(t)δvdt = 0. It implies that almost everywhere on [0, t f ] (1.107) φ(t) b(t) = 0.

  ∂H ∂u = ψ(t) B(t) = 0, ∂H ∂u y = ψ(t) C(t) = 0. Denoting ψ = λ 1 λ 2 λ • , one obtains the necessary conditions (3.22)-(3.25) given by Proposition 4 for a closed-loop min-max or min-min Stackelberg strategy. Finally, some part of the initial and final values of the extended state Z, defined by (1.111) are imposed by the transversality condition for the follower optimization problem (3.10) and by the initial state x(0) = x 0 . We can formalize these conditions by defining two sets M 0 and M 1 (1.120)

3 A 2 B 1 R -1 11 RT 2 z 2

 321112 (3.22) is a constraint on u y . The relation(3.28) follows.Proof of Lemma 1. The proof uses the controllability Hautus test. The pair (A, B) satisfies the Kalman condition if and only if the matrix A -αI B is of full rank, for every α ∈ C. The proof consists in showing that all line vectors z T -αI B = 0, are trivial. Developing Equation (1.126), we have-z T 1 Q 1 = z T 2 (A -αI n ), (1.127) -z T 1 Q 2 = z T 3 (A -αI n ), (1.128) z T 1 (A T -αI n ) = z T 2 (S 1 + B 2 R 21 = z T 3 B 1 . (1.131)Multiplying Equation (1.127) by z 1 and Equation (1.129) by z 2 , we obtain(1.132) -z T 1 Q 1 z 1 = z T 2 (A -αI n )z 1 = z T 2 S 1 + B 2 R -1 12 BThe first term is nonpositive (Q 1 ≥ 0) and the last term is nonnegative, hence both are zero. It follows that z T 1 Q 1 = 0, z T 2 B 1 = 0 and z T 2 B 2 = 0. Plugging these relations into (1.127), (1.131), one getsz T 1 (A T -αI n ) = 0, z T 1 Q 1 = 0, (1.133) z T 2 (A -αI n ) = 0, z T 2 B 2 = 0, z T 2 B 1 = 0, (1.134) z T 3 (A -αI n ) = 0, z T 3 B 2 = 0, z T 3 B 1 = 0. (1.135) The relations (1.133) correspond to the observability Hautus test of the pair (Q 1 , A), the relations (1.134) and (1.135) to the controllability Hautus test of the pair (A, B 1 ) or (A, B 2 ). The assumptions of controllability and observability lead to z 1 , z 2 and z 3 trivial. Proof of Proposition 7. With the condition (1.136) p 2 B 1 + u T R 21 ≡ 0, the term ∂u ∂x does not appear anymore in the necessary conditions (3.50)-(3.53).

  ds

  and (4.70), we can compute a lower bound of the criterion Ĵ1 (u, ξ)2 Ĵ1 (u, ξ) ≥ C 6 -(C 3 + C 5 ) +x T (t f )K 1f x(t f ) -C 4 x 0 2 -C 5 t f .

  20)-(3.21) (see Appendix). The following proposition is proved in the Appendix.Proposition 4. If the trajectory x(•), associated with the pair (u * , v * ) of closedloop Stackelberg controls, is solution of the Stackelberg problem, then there exist absolutely continuous mappings λ 1 , λ 2 : [0, t f ] → R n , called costate vectors (written as line vectors by convention), and a scalar λ 0 ≥ 0, such that

	(3.22)	0 = λ 2 (t)	"	∂F 22 ∂u	(t, x(t), p 2 (t), u(t, x(t)))ux +	∂F 21 ∂u	(t, x(t), p 2 (t), u(t, x(t)))	« T
		+λ 1 (t)	∂F 1 ∂u	(t, x(t), p 2 (t), u(t, x(t))) + λ • ∂ L1 ∂u	(t, x(t), p 2 (t), u(t, x(t))),
	(3.23)	0 = λ T 2 (t)F 22 (t, x(t), p 2 (t), u(t, x(t))),
		= λ T 2 (t) p 2 (t)	∂F 1 ∂u	(t, x(t), p 2 (t), u(t, x(t))) +	∂ L2 ∂u	(t, x(t), p 2 (t), u(t, x(t)))	!	,
		λ1 (t) = -λ 2 (t)	"	∂F 21 ∂x	(t, x(t), p 2 (t), u(t, x(t))) +	∂F 22 ∂x	(t, x(t), p 2 (t), u(t, x(t)))ux

  FZ + Fu u y h Z + Fuy u yy h Z .This relation is satisfied for all controls u functions of t and x. In particular it is also satisfied for controls u functions of t only. For such a control, the relation (1.119) becomes

	1.118) ψ(t) = -ψ(t)A(t) = -ψ(t) Furthermore the relation (1.117) holds for every control δu, and thus
	t f
	(1.119)	ψ(t) B(t)δu(t, x) + C(t)δu T y (t, x) dt = 0.
	0
	t f 0 ψ(t) B(t)δu(t) dt = 0. It implies that almost everywhere on
	[0, t f ], ψ(t) B(t) = 0. Then, (1.119) leads to	t f 0 ψ(t) C(t)δu T y (t, x) dt = 0. Hence,
	almost everywhere on [0, t = -dH dZ (t, Z(t), u(t, h(Z(t))), u

f ], there holds ψ(t) C(t) = 0. Let H(t, Z, u, u y ) = ψ(t) F (t, Z, u, u y ) be the Hamiltonian associated with this optimization problem. The last equations can be rewritten almost everywhere on [0, t f ] as:

Ż(t) = F (t,

Z(t), u(t, h(Z(t))), u y (t, h(Z(t)))) = ∂H ∂ψ (t, Z(t), u(t, h(Z(t))), u y (t, h(Z(t)))), ψ(t) = -ψ(t) FZ (t, Z(t), u(t, h(Z(t))), u y (t, h(Z(t)))) + Fu (t, Z(t), u(t, h(Z(t))), u y (t, h(Z(t))))u y (t, h(Z(t)))h Z (Z(t)) + Fuy (t, Z(t), u(t, h(Z(t))), u y (t, h(Z(t))))u yy (t, h(Z(t)))h Z (Z(t)) , y (t, h(Z(t)))),

  of Proposition 5. The proof goes by contradiction. The term λ T 2 F 22 in (3.23) is the product of a column vector (λ T2 ) and a line vectorF 22 = p 2The triviality of this term induces that all components of λ 2 or all components of F 22 are trivial (or both). F 21 (t, x, p 2 , u(t, x, p 2 )), because F 22 = 0.

	∂f ∂u ≡ 0. If furthermore + ∂L 2 , since λ 2 is a line ∂u ∂F 22 = ∂u is invertible, then the Implicit Function Theorem applied to costate vector. Assume that λ 2 ≡ 0, then F 22 = p 2 ∂f ∂u + ∂L 2 ∂u ∂ ∂u p 2 ∂f ∂u + ∂L 2 ∂u
	the function F 22 with respect to the variable u permits to write locally along the
	trajectory the control u = u(t, x, p 2 ).
	The system in (x, p 2 ) is rewritten as
	(1.125)	ẋ(t) = F 1 (t, x, p 2 , u(t, x, p 2 )) ṗ2 (t) =

  From this relation and from(3.53), necessary conditions about x(t f ) are developed by successive derivations with respect to time,(1.138) λ 1 (t)B 2 -p 2 (t)B 2 R -1 22 R 12 = 0, λ 1 (t)B 1 R -1 11 R 21 -p 2 (t)B 1 =0. These two relations can be rewritten for every t ∈ [0, t f ] as (1.139) x(t) λ 1 (t) p 2 (t) B = 0. Plugging (1.136) into the dynamics of x, λ 1 and p 2 , we obtain the autonomous differential system (1.140) d dt x T (t) λ 1 (t) p 2 (t) = x T (t) λ 1 (t) p 2 (t) A

		1.136) does not induce necessary
	conditions for However assuming that R 21 is invertible, the control u admits two representations ∂u . ∂x
	(1.137)	u(t, x(t)) = -R -1 11 B T 1 λ T 1 (t) = -R -1 21 B T 1 p T 2 (t).