Charles Figuières
email: figuiere@ensam.inra.fr

Pierre Cartigny

Magaret Slade

Markov interactions in a class of dynamic games

Keywords: jeux dynamiques, équilibres markoviens parfaits, équilibres de Nash en boucle ouverte, interactions markoviennes, efficacité dynamic games, markov perfect equilibrium, open-loop Nash equilibrium, markov interactions, efficiency. JEL: C73, D92, L16

Cet article contribue à la compréhension des comportements économiques dans des contextes stratégiques inter-temporels. Par comparaison des équilibres markoviens parfaits et des équilibres de Nash en boucle ouverte d'une classe de jeux dynamiques très utilisée, je montre : i) en quoi les comportements diffèrent selon que la structure d'information qui prévaut est markovienne ou en boucle ouverte, ii) pour quelle raison une structure d'information donnée produit la concurrence la plus sévère dans le long terme. De plus, une contribution pragmatique de l'article, importante du point de vue du modélisateur, est d'établir que ces propriétés se déduisent directement de la structure des paiements, sans nécessité de calculer les équilibres au préalable.

Introduction

Economists often use static games, even when the strategic situation to be analyzed is made of a blend of stock and flow variables and would call for a dynamic formulation. For instance, if agents' decisions have persistent polluting effects, then it makes sense to consider ambient pollution as a stock, whereas emissions would better be formalized as flows. One may prefer static models for their greater simplicity; more satisfactorily, when dynamic outcomes converge to a steady state, and when the transition is not of prime interest, static models are meant to capture long run stable interactions, with an implicit dynamic game in the background. There is no objection against this approach if the equilibrium of the static game associated with the steady state coincide with the long run part of the dynamic equilibrium. Whether or not this coincidence holds depends closely on the information used by players when they devise their strategies in the real dynamic game. More precisely, it has been established in various dynamic games that an open-loop information pattern leads to equilibria that mimic, possibly exactly, static equilibria in the long run (Reynolds 1989, Driskill and McCafferty 1990[START_REF] Dockner | A dynamic theory of conjectural variations[END_REF]). This comes as no surprise, as I shall develop below, for open-loop Nash equilibria (OLNE hereafter) are akin to static Nash equilibria. But what happens when players do not use open-loop strategies? If we persist to use static models as shortcuts for steady states of unmodelled dynamic games, how inaccurate are static equilibria?

There are many different answers to the above questions, for inter-temporal frameworks open the possibility of various dynamic informational environments, therefore different alternative kinds of strategies, implying outcomes that can be very different from static equilibria, even in the long run. 1 But one may also ask what qualifications one particular kind of dynamic information pattern may produce to static (open-loop) findings? This paper deals with the markovian information pattern. Under such an information structure, behaviors at any point of time depend on a reduced set of variables, typically the current state of the system. Researchers have many reasons to focus on markovian behaviors, as documented in [START_REF] Maskin | Markov perfect equilibrium I. Observable actions[END_REF]. First of all they produce subgame perfect equilibria. There are also a few practical reasons: i) markov perfect equilibria (MPE) are widely used in applied dynamic game theory, a fact that justifies further theoretical attention, ii) their simplicity makes econometric estimation and inference easier, iii) they can readily be simulated 2 .

What modification(s) on long run behaviors are produced by a markovian information structure? For instance, are markovian behaviors systematically more aggressive than static behaviors? Partial answers can be found in the literature. [START_REF] Long | Comparing Open-loop With Markov Equilibria in a Class of Differential Games[END_REF] have analyzed this question for a class 2 of differential games satisfying homogeneity properties, and which is useful to modelize resource depletion problems. 3 In this class, markov strategies systematically bring about faster rates of extraction, as compared with openloop strategies.

But what for the differential games that do not possess the required homogeneity properties? Clearly, one cannot expect the same kind of systematic result, for outside the class of [START_REF] Long | Comparing Open-loop With Markov Equilibria in a Class of Differential Games[END_REF] one can find games where sometimes in turns out that markov strategies increase competition [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF][START_REF] Driskill | Dynamic duopoly with adjustment costs: A differential game approach[END_REF][START_REF] Dockner | A dynamic theory of conjectural variations[END_REF][START_REF] Driskill | Durable goods oligopoly[END_REF], and sometimes competition is softened (Van der Ploeg and De Zeew 1990, Melese and Michel 1991, [START_REF] Piga | A dynamic model of advertising and product differentiation[END_REF], Figuières 2001).

It is not clear what reasons explain those findings. The papers quoted above are too specific to capture important qualitative properties of markov strategies with the necessary degree of generality. What is needed actually is a framework that would be general enough to encompass a priori the two kinds of results, and to ascertain what component drives the answer. This is an ambitious stake. Luckily, as this paper show, this challenge is manageable for an important class of differential games with two state variables, which embodies most of the papers with two state variables quoted above, and even some papers with a single state variable. To some extent the present contribution is technical. It establishes some qualitative and comparative properties for the MPE and the OLNE; this will be useful for all the future applied analysis that will make use of the class of games here defined; and retrospectively it helps understanding and unifying the existing findings in the literature.

The simplicity of the results is in sharp contrast with the technical difficulties inherently associated to differential games. They summarize as follows: i) the payoff structure allows one to deduce whether the MPE behaviours are characterized by markov substitutability -meaning that decision rules are decreasing functions of the rival capital stock -or markov complementarity -where decision rules are increasing functions of the rival stock; ii) in games with markov substitutability (complementarity) markov strategies increase (soften) competition in the long run.

The paper is organized as follows. Section 2 gives more precisions about what can be expected from the comparison of OLNE and MPE, in relation with the goal of this paper. Section 3 presents a general class of differential games with two state variables. Section 4 is devoted to cooperative and non cooperative behavioral scenarii that have been used within this context: the corresponding outcomes are the Pareto efficient (utilitarian) solution, the OLNE and the MPE. Section 5 shows what information about the MPE can be deduced from the payoff structure and discusses the relationship between dynamic and static best responses. Section 6 compares the steady states of the three scenarii. Illustrations and further insights are provided in Section 3 More generally, differential games provide a suitable tool for the study of economic situations where some state variables evolve over time as a result of inter-temporal and strategic interactions among agents. Numerous fields have benefited from this tool, including industrial economics, macroeconomics, environmental economics, public economics and management. For a comprehensive introduction the reader can consult [START_REF] Dockner | Differential games in economics and management science[END_REF]; Amir (2003) provides a non technical extensive account of this literature.

7. As a by-product an existence result is given for the differential game used in [START_REF] Figuières | Complementarity, substitutability and the strategic accumulation of capital[END_REF]. Section 8 summarizes the contribution of this article and suggests possible extensions.

OLNE, MPE and markov interactions

The specificities of the OLNE and the MPE have been widely documented in the literature on differential games, but it is worth recalling their properties with the goal of this paper in mind.

Dynamic games with state variables involve subtle and important specificities as far as agents' behaviors are concerned. Two specificities absent in static frameworks are the following. First, a decision at any time has not only current consequences, but also future consequences via the law of motion of the state variables. Therefore a rational decision at time t reflects an inter-temporal trade-off that accounts for this natural dynamic interaction channel between the present and the future. Secondly, there is an additional and more subtle channel whereby a decision today can have persistent consequences, typically through rival players' strategies. The explanation is as follows. Presumably, agents can rethink their optimal plans at each decision stage in the future. In that case some restrictions must be imposed on the strategies to be used by players, in order to rule out empty or non credible threats, i.e. strategies made of future actions that may no longer be individually rational when they are to be actually implemented; in the language of Game Theory the good candidate strategies are required to be subgame perfect. Subgame perfection obtains when the strategies are determined by backward induction. By construction the resulting strategies at any time t are decision rules conditional on the levels achieved by the state variables at t. The very nature of those strategies produces indeed a second channel by which a player can influence the future course of interactions. More precisely, a change in a player's action today modifies the evolution of the state variables that in turn alters the actions to be undertaken by the rival players, as it can be deduced from their decision rules. This second channel will be referred to as the Markov Interactions channel.

The OLNE and the MPE are relevant with respect to the behavioral features just outlined, i.e. natural dynamic interactions and markov interactions. The OLNE typically captures the first feature, but not the second; the MPE captures both aspects. The OLNE involves strategies that are simply functions of time. In other words, at the beginning of the game the players decide upon the whole sequence of actions to be taken from then onwards. For this reason it is generally argued that the OLNE is a static interaction concept. Admittedly several authors have explored dynamic situations where the OLNE has a stationary state identical to the Nash equilibrium of the associated static "limit game" [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF][START_REF] Driskill | Dynamic duopoly with adjustment costs: A differential game approach[END_REF][START_REF] Dockner | A dynamic theory of conjectural variations[END_REF]). The associated limit game obtains when the parameters responsible for structural time-dependence tend to specific values, so that natural dynamic interactions disappear. In this paper I shall view the OLNE as an approximation for static interactions (bearing in mind that the approximation becomes exact when natural dynamic interactions vanish).

In contrast with the OLNE, even when natural dynamic interactions disappear, the MPE stationary state never coincides with the static equilibrium of the limit game, for markov interactions remain.

Clearly comparing the OLNE with the MPE is a way to assess what modifications on behaviors a proper account for the temporal structure brings about. This amounts to investigate what can be expected from markov interactions, which are solely responsible for any difference between the "static" equilibrium concept and the dynamic one.

A class of dynamic frameworks and their associated games

The class of economic situations under consideration includes most of the two-state differential games used in the economic literature, and some singlestate games as particular cases. Examples are capacity competition between firms (Reynolds 1986[START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF](Reynolds and 1991) ) [START_REF] Figuières | Complementarity, substitutability and the strategic accumulation of capital[END_REF], advertising games [START_REF] Fershtman | Goodwill and market share in oligopoly[END_REF][START_REF] Piga | A dynamic model of advertising and product differentiation[END_REF]) and dynamic voluntary contributions to a public good [START_REF] Fershtman | Goodwill and market share in oligopoly[END_REF]Nitzan 1991, Itaya and[START_REF] Itaya | A dynamic conjectural variation model in the private provision of public goods: a differential game approach[END_REF].

Two players invest over an infinite horizon to build their stocks. Time is continuous and the evolutions of the stocks obey:

(1) ½ Żi (t) = Q i (t) -bZ i (t), Z i (0) = Z 0i , i = 1, 2 ,
where Z i (t) ∈ R + and Q i (t) ∈ R represent, respectively, player i's capital stock and physical investment. Investment is reversible since Q i (t) is not restricted to be positive. The parameter b ≥ 0 is the physical depreciation rate.

At each point of time, the two capital stocks contribute to player i's instantaneous payoffs according to a function P i (Z i , Z j ). The functions P i (., . ), i = 1, 2 are assumed to be quadratic, and symmetric in the sense P j (Z j , Z i ) = P i (Z j , Z i ). Several economic examples of such revenue functions are given in Section 7 that are all particular cases of the general form:

P i (Z i , Z j ) = p 0 + p 1 Z i + p 2 Z j + p 11 2 Z 2 i + p 12 Z i Z j + p 22 2 Z 2 j ,
where p 0 ≥ 0, p 1 > 0, p 11 < 0, p 22 ≤ 0. The positive sign of p 1 ensures that, at least for some range of the capital stocks, the function P i (., .) increases with Z i . The other two assumptions on the parameters implies that the function P i (., . ) is strictly concave with respect to player i's stock and also concave with respect to the rival capital stock. There are no restrictions a priori on the scalars p 2 and p 12 , whose signs will depend on the context. 4 One should also mention P. R. [START_REF] Worthington | Strategic investment and Conjectural Variations[END_REF], who offers a discrete-time model of capacity competition, which is akin to Reynolds' continuous-time analyis (1987).

The instantaneous payoffs obtain by subtracting the costs of investment from P i (., .). Those costs are:

C(Q i ) = c 1 Q i + c 2 2 Q 2 i , c 1 > 0 , c 2 ≥ 0 .
Each player associates an inter-temporal payoff to every vector of outcomes (Z 1 (.), Q 1 (.), Z 2 (.), Q 2 (.)):

(2)

J i = Z ∞ 0 e -rt £ P i (Z i (t), Z j (t)) -C(Q i (t)) ¤ dt for i = 1, 2, j = 1, 2, i 6 = j.
The parameter r > 0 stands for the common discount rate.

The investments Q i (.) are endogenously determined: in non cooperative scenarii they are players' equilibrium strategies, selected in pre-specified strategy spaces.

The open-loop strategy space for player i, denoted S ol i , is the space of strategies that are piecewise continuous functions of time:

Q i : R + → R , t 7 → Q i (t) .
Let us call Γ ol the class of open-loop games defined by the two players, the open-loop strategy space S ol i , the dynamics (1) and the payoffs (2).

The markov (or feedback) stationary strategy space for player i, denoted S m i , is the space of functions of the stocks:

Q i : R + × R + → R , (Z i , Z j ) 7 → Q i (Z i , Z j ) .
Using this space of strategies S m i and the previous elements, one can define a class Γ m of markov differential games.

Whatever the strategy space to be used, the equilibrium concepts of interest are Nash equilibria, i.e. the profile of strategies in which each player optimizes his payoff, given the strategy of the other agent 5 .

A last remark is in order. The above class of games also describes actually two types of differential games with a single state variable. The first type occurs where the parameters of the functions P i (., .) are such that only the sum of the stocks matters for the players, i.e. when p 1 = p 2 , p 11 = p 12 = p 22 . The relevant state variable is then Z = Z 1 + Z 2 , and its law of motion is

Ż(t) = Ż1 (t) + Ż2 (t), i.e. Ż(t) = Q 1 (t) + Q 2 (t) -bZ (t) , Z(0) = Z 0 = Z 10 + Z 20 .
This single-state sub-class captures Fershtman and Nitzan (1991) and Itaya and Shimomura (2001)'s papers on dynamic voluntary contributions to a public good, and Dockner and Long (1992) analysis of international pollution.

The second type of single-state games is where only the difference of the stocks matters, like in Van der Ploeg and De Zeeuw (1990) and [START_REF] Melese | Reversing the arms race: a differential game model[END_REF]. The parameters are then such that p 1 = -p 2 , p 11 = p 22 = -p 12 , the relevant state variable is D = Z 1 -Z 2 , and the corresponding law of motion is:

.

D(t) = Q 1 (t) -Q 2 (t) -bD (t) , D(0) = D 0 = Z 10 -Z 20 .

Three behavioral scenarii

This section describes the three possible outcomes of particular interest in our framework, mainly by means of first order conditions that characterize behaviors. Those conditions are not sufficient to guaranty also existence. However as they are necessary, the analysis is valid whenever in the game at hand the outcomes do exist, which has been established in many examples that belong to the class of games I am considering 6 . Nevertheless I shall give an existence result for the example presented in section 6.3, for it has not already been done elsewhere.

Efficient behaviors

. A Pareto-efficient outcome is a pair of trajecto- ries (Q c 1 (.), Q c 2 (.)
) such that there exists no other pair that would make a player strictly better-off without making the other player worse-off. In our symmetric context it makes sense to focus on the centralized (utilitarian) criterion; then (Q c 1 (.), Q c 2 (.)) solves: max

Q 1 (.),Q 2 (.) J 1 + J 2 ,
subject to (1).

If λ 1 and λ 2 are the marginal values of the stocks, the current value Hamiltonian of the centralized problem is:

H = P 1 (Z 1 , Z 2 )+P 2 (Z 2 , Z 1 )-C(Q 1 )-C(Q 2 )+λ 1 (Q 1 -bZ 1 )+λ 2 (Q 2 -bZ 2 ) .
The necessary conditions for optimality read as:

         Ż1 = 1 c 2 λ 1 -bZ 1 -c 1 c 2 , Z 1 (0) = Z 10 , Ż2 = 1 c 2 λ 2 -bZ 2 -c 1 c 2 , Z 2 (0) = Z 20 , λ1 = (r + b) λ 1 -(p 1 + p 2 ) -(p 11 + p 22 )Z 1 -2p 12 Z 2 , λ2 = (r + b) λ 2 -(p 1 + p 2 ) -(p 11 + p 22 )Z 2 -2p 12 Z 1 , (SC i )
and the transversality condition is

lim t→∞ e -rt [λ 1 (t)(Z 1 (t) -Z c 1 (t)) + λ 2 (t)(Z 2 (t) -Z c 2 (t))] = 0 ,
where Z c i (.) is a candidate for optimization and Z i (.) is any other path. In specific examples some mild conditions are assumed in order to obtain a convergent centralized solution for this two-point boundary values problem, leading the stocks to the same positive steady state:

Z c = p 1 + p 2 -(r + b)c 1 (r + b)bc 2 -(p 11 + p 12 ) -(p 12 + p 22 )
.

produce conditions on the parameters and the stocks that, alone, are not that helpful in many applications. Specific applications often require additional restrictions, for instance over the range of admissible stocks, as it is the case in [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF], that may conflict, for some values of the parameters, with those required for existence.

4.2.

Open-loop Nash equilibrium. An open-loop Nash Equilibrium is a pair of trajectories (Q o 1 (.), Q o 2 (.)) defined as follows: the path Q o i (.) maximizes J i subject to (1), given the rival investment trajectory Q o j (.), and therefore the stock trajectory Z o j , of the other player. From this definition it is worthwhile noting that each player considers that his investment choice has no effect on the investment path of the other player.

Using the Maximum Principle to calculate best responses, it follows that the marginal value of player i's stock, denoted µ i , should equal the marginal cost of investment

C 0 (Q i ) = c 1 + c 2 Q i .
And the capital stocks and the marginal values must satisfy:

½ Żi (t) = 1 c 2 µ i (t) -bZ i -c 1 c 2 , Z i (0) = Z i0 , μi (t) = (r + b) µ i (t) -p 1 -p 11 Z i -p 12 Z o j , (SN i ) for i = 1, 2.
In addition the transversality condition are lim t→∞ e -rt µ i (t)(Z i (t)-

Z o i (t)) = 0, i = 1, 2, where Z o i (.
) is a candidate trajectory and Z i (.) is any other path.

Putting together (SN 1 ), (SN 2 ), the initial and transversality conditions, one finds again a two-point boundary value problem. At the steady state the two players share the same capital stock:

Z o = p 1 -(r + b)c 1 (r + b)c 2 b -(p 11 + p 12 )
.

Some conditions can easily be devised to ensure the existence of a convergent OLNE leading to this positive steady state.

Markov Perfect

Equilibrium. A MPE (Q * 1 , Q * 2
) can be characterized using a dynamic programming approach. The equilibrium markov strategies solve the following Hamilton-Jacobi-Bellman (HJB) equations:

r V i (Z i , Z j ) = max Q i {P i (Z i , Z j ) -C(Q i ) + V i 1 (Z i , Z j ) (Q i -b Z i ) +V i 2 (Z i , Z j ) (Q * j (Z i , Z j ) -b Z j ) ª , i = 1, 2 ,
where V i (Z i , Z j ) is player i's value function, for the subgame starting at (Z i , Z j ), given that the opponent is playing Q * j (Z i , Z j ). From the HJB equations it is clear that the value function, and therefore the behavior of each agent, is to be found using the information that the rival investment is a function of the stocks. At each point of time player i's optimal decision is an answer to the question: "how much do I invest today, given that this decision is costly, that it will affect current and future stocks, as can be deduced from the law of motion and the rival decision rule, and given that those stocks contribute to my overall payoffs according to V i (Z i , Z j )?". Optimal decisions are such that the marginal cost to invest, C 0 (Q i ), is equal to its marginal benefit V i 1 (Z i , Z j ). Since the symmetric class of games under consideration is linear quadratic, it is usual to look for MPE (Q

* 1 (Z 1 , Z 2 ), Q * 2 (Z 2 , Z 1
)) that are symmetric and linear functions of the stock variables. Besides, a MPE is said to be convergent if, when the equilibrium decisions are plugged into the system (1), this system is asymptotically stable: whatever the initial conditions, the capital stocks converge to their steady states.

The first order conditions 7 for the maximization of the HJB equations and the stability requirement mentioned above lead to : Theorem 4.1. Take a game γ m ∈ Γ m and let

Q * 1 (t) = v * 1 -c 1 c 2 + v * 11 c 2 Z 1 (t) + v * 12 c 2 Z 2 (t) (3) Q * 2 (t) = v * 1 -c 1 c 2 + v * 11 c 2 Z 2 (t) + v * 12 c 2 Z 1 (t) (4) V i (Z i (t), Z j (t)) = v * 0 + v * 1 Z i (t) + v * 2 Z j (t) + v * 11 2 Z 2 i (t) (5) +v * 12 Z i (t) Z j (t) + v * 22 2 Z 2 j (t) where v * 0 , v * 1 , v * 2 , v * 11 , v * 12 , v * 22 are solution to r v 0 = p 0 + (v 1 ) 2 2c 2 + (c 1 ) 2 2c 2 - c 1 c 2 (v 1 + v 2 ) + v 1 v 2 c 2 , r v 1 = p 1 + v 1 v 11 c 2 -c 1 v 11 + v 12 c 2 -bv 1 + v 1 v 12 c 2 + v 12 v 2 c 2 , r v 2 = p 2 + v 1 v 12 c 2 -c 1 v 12 + v 22 c 2 + v 1 v 22 c 2 + v 11 v 2 c 2 -bv 2 , r v 11 2 = p 11 2 + (v 11 ) 2 2c 2 -bv 11 + (v 12 ) 2 c 2 , r v 12 = p 12 + 2 v 11 v 12 c 2 + v 12 v 22 c 2 -2bv 12 , r v 22 2 = p 22 2 + (v 12 ) 2 2c 2 + v 11 v 22 c 2 -bv 22 , v 11 < bc 2 , (v 12 ) 2 < (v 11 -bc 2 ) 2 , then the pair (Q * 1 (Z 1 , Z 2 ), Q * 2 (Z 2 , Z 1 )
) is a convergent and symmetric linear MPE for the game γ m , and V i (Z i (t), Z j (t)) is player i's value function.

Proof : see appendix A

One recognizes in Theorem 4.1 the Riccati equations for the feedback coefficients of the value function. The sign of the feedback coefficient v * 12 /c 2 in the markovian strategies (3) and ( 4) conveys interesting information. Depending on the parameters, it can be positive or negative. Intuitively, when it is negative it corresponds to situations where an increase of player i's stock reduces the player j's incentive to invest. This case can be defined as one of markov substitutability (see also [START_REF] Figuières | Complementarity, substitutability and the strategic accumulation of capital[END_REF] for a related discussion). On the other hand when v * 12 /c 2 is positive an increase of Z i increases player j's incentive to invest, which can be defined as a case of markov complementarity. As already emphasized, the difference between the OLNE and 7 Those first order conditions are sufficient for maximisation since the expressions in curly brackets in the HJB equations are concave in Qi.

the MPE stems from the fact that, with the decision rules of the MPE, the players do take into account how a change in their current strategy affects the rival decision via the state variables; in the present class of games, this behavioral difference is qualitatively captured by the endogenous coefficients v * 11 and v * 12 . I shall come back in particular to the role of the coefficient v * 12 in Sections 5 and 6.

Payoff structure and markov interactions

Understanding markov interactions in a differential game is made difficult by the necessity to find out the MPE, which necessitates to solve the Riccati equations, a system of non linear equations. Greater insights would be achieved if it were possible, like for static games, to infer some of the qualitative properties of the equilibrium without having first to compute it, by the mere knowledge of the components of the game, and in particular the knowledge of the payoff structure. This is the purpose of the following theorem, one of the two main results of the paper, that relates the sign of one key endogenous variable to the sign of one key exogenous parameter : Theorem 5.1. Take a game γ m ∈ Γ m and let

Q * i (t) = v * 1 -c 1 c 2 + v * 11 c 2 Z i (t) + v * 12 c 2 Z j (t), i = 1, 2 j = 1, 2 i 6 = j
be a Linear Markov Perfect Equilibrium of γ m . Then sign(v * 12 ) = sign(P i 12 ) = sign(p 12 ).

Proof : appendix B

In the class of games Γ m the nature of markov interactions at a MPE can straightforwardly be deduced from the payoffs structure: when the function P i (Z i , Z j ) is such that P 12 < 0 (P 12 > 0), the game is characterized by markov substitutability (markov complementarity). As for the single state games that our two-state frameworks capture, it is left to the reader to check that both the public good model of [START_REF] Fershtman | Dynamic voluntary provision of public goods[END_REF], and the pollution game of Dockner and Long (1992), like any game in our class where only the sum of the two states matters, are games of markov substitutability; on the contrary, games where only the difference of the stocks matters (Van der Ploeg and De Zeeuw 1990, Melese and Michel 1991) are games of markov complementarity.

The distinction between markov substitutability and markov complementarity is akin to the usual distinction between strategic substitutability and strategic complementarity in static games. Is there a link between the dynamic and the static reaction functions? To answer this question one may consider intuitively what happens to the MPE when the cost parameters c 1 and c 2 tend to zero [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF] and Driskill and McCafferty 1989 also consider similar limit cases), for those parameters give the class of games Γ m its dynamic structure.

When c 2 → 0 (but remains positive), the marginal cost is reduced, which means that incentives to invest are increased. The MPE is still a linear function of the stocks, but it is characterized by a faster approach path towards the steady state; if the other cost parameter c 1 also tends to zero there is an initial infinite rate of investment, whereby agents almost "jump" directly to the markov steady state. In a sense the game is transformed into a static one, a "limit game" where the decision variables are the stocks and the payoffs are P i (Z i , Z j ). In static games the nature of strategic interactions is readily deduced from the payoffs structure. In particular, the sign of the second order cross derivative:

P i
12 (Z i , Z j ) = p 12 conveys useful information. When p 12 is positive (resp. negative) one speaks of strategic complementarity (substitutability) 8 . Let us recall that strategic substitutability (resp. complementarity) in a game means that the best reply of a player is a decreasing (resp. increasing) function of the opponent's actions.

When the parameter c 2 is strictly positive, the costs of investment are increasing and convex; as a result any change in the stocks takes place gradually over time. This stickiness restores the distinction between stocks and flows.

So, another way to express the result of this section is the following: if the "limit games" are characterized by strategic substitutability (complementarity) then the underlying dynamic games are characterized by markov substitutability (complementarity).

Assessing the impact of markov interactions

This section shows that the decision rules characterized by markov substitutability lead to less efficient levels of capital; intuitively markov information in this case is synonymous of more aggressive play. On the contrary, when the game features markov complementarity, decision rules entail a steady state that is closer to the efficient criterion; in this case, markov information is conducive of more cooperation.

Let us use the optimal control tools to determine the MPE steady state. When agents use markov strategies, player i's problem can be stated as follows. Given that player j's strategy is

Q * j (Z i , Z j ) = v 1 -q * c 2 + v * 11 c 2 Z j + v * 12 c 2 Z i ,
player i aims at maximizing (2) subject to [START_REF] Amir | Stochastic games in economics and related fields I: a general survey[END_REF]. Player i's current value Hamiltonian is Hi

= p 0 + p 1 Z i + p 2 Z j + p 11 2 Z 2 i + p 12 Z i Z j + p 22 2 Z 2 j -c 1 Q i - c 2 2 Q 2 i + µ 11 (Q i -bZ i ) + µ 12 ¡ Q * j (Z i , Z j ) -bZ j ¢ .
The necessary conditions for optimality are:

       Żi = 1 c 2 µ 11 -bZ i -c 1 c 2 , Z 1 (0) = Z 10 , μ11 = (r + b) µ 11 -p 1 -p 11 Z i -p 12 Z j -µ 12 ∂Q * j ∂Z i , μ12 = (r + b - ∂Q * j ∂Z j ) µ 12 -p 2 -p 12 Z i -p 22 Z j . (SN m i )
Note the difference with the OLNE necessary conditions, due to the terms c 2 respectively, the symmetric steady state Z i = Z j = Z m of (SN m i ) can be calculated to give:

Z m = p 1 -(r + b)c 1 + p 2 v * 12 /c 2 r+b-v * 11 /c 2 (r + b)c 2 b -(p 11 + p 12 ) -(p 12 + p 22 ) v * 12 /c 2 r+b-v * 11 /c 2 .
Lemma 6.1. In the expression for Z m the quantity

v * 12 /c 2 r + b -v * 11 /c 2 belongs to the interval ]-1; 1[ . Proof: appendix C.
We are now ready for the second main result: Theorem 6.1. When the markov game γ m ∈ Γ m is characterized by markov complementarity (P 12 > 0) the MPE steady state Z m is between the OLNE steady state Z o and the centralized steady state Z c .

When the markov game γ m ∈ Γ m is characterized by markov substitutability (P 12 < 0) the OLNE steady state Z o is between the MPE steady state Z m and the centralized steady state Z c . When P 12 = 0 the OLNE and the MPE coincide.

Proof : appendix D.

With the joint use of Theorems 5.1 and 6.1, understanding the effects of markov interactions for the class Γ m is straightforward. A first glance at the payoffs structure indicates whether the MPE is characterized by markov substitutability or markov complementarity (Theorem 5.1); then using this information and Theorem 6.1, one can deduce if Z m is closer to the efficient steady state than Z o , i.e. if competition is increased or reduced in the long run.

Further interesting insights require the comparison of the payoffs, after that of the steady states. In the numerical examples of Section 7 it turns out that, if commitment can serve as a substitute for cooperation in games with markov substitutability (because of less aggressive reactions), players are better-off if they use decision rules in games with markov complementarity (more collusive reactions). However one should take the usual precautions with numerical conclusions.

Illustrations and further results

7.1. Capacity competition between firms. In the industrial organization literature one of the best-known analyses with markov substitutability is that of [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF]. For other illustrations see the adjustment cost model of [START_REF] Driskill | Dynamic duopoly with adjustment costs: A differential game approach[END_REF] and Driskill (2001)'s analysis of the durable good duopoly.

Reynolds' model is a capital accumulation game of infinite duration between two industrial firms, whose gross profits at time t correspond to a choice of functions P i with parameters:

p 0 = 0, p 1 > 0, p 2 = 0, p 11 = -2, p 12 = -1, p 22 = 0,
that is:

P i (Z i , Z j ) = Z i (p 1 -Z i -Z j ), p 1 > 0, i = 1, 2, j = 1, 2, i 6 = j .
The profits of each firm depend on its own capital and on the capital stock of its rival. They are reduced-forms of profit functions in a two-stage game that takes place at each time t. In the first stage the firms decide upon the scale of production; the second stage is a quantity competition game, configured by the first stage capacity choices (see [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF] for details). Observe from the gross profit functions that we are dealing with markov substitutability interactions (p 12 < 0).

The costs of investment are:

C(Q i ) = c 1 Q i + c 2 2 Q 2 i , c 1 , c 2 > 0.
At each point of time, net profits are given by the difference between gross profits and cost functions. The payoffs of the players are the sums of the discounted net profits:

J i = Z ∞ 0 e -rt h Z i (t)(p 1 -Z i (t) -Z j (t)) - c 2 2 Q 2 i (t) -c 1 Q i (t)) i dt,
where r > 0 is the common discount rate. The existence of a unique linear and symmetric MPE for this game has been established under the following sufficient conditions on the model parameters:

b + r 2 ≤ 1 2 , c 2 ≤ 1, c 1 ≥ p 1 9 (r + b) .
Reynolds also showed that, at the linear symmetric MPE, the investment strategies:

Q i (t) = v * 1 -c 1 c 2 + v * 11 c 2 Z i (t) + v * 12 c 2 Z j (t) , v * 12 c 2 < 0 ,
are decreasing functions of the rival capital, and produce long run capacity above the open-loop steady state (which is itself above the centralized solution), as one could have expected from Theorems 5.1 and 6.1. The numerical example in tables 1, 2 and 3 illustrates these results. Table 1. Parameter values

p 0 = 0 p 1 = 60 p 2 = 0 p 11 = -3 p 12 = -1 p 22 = 0 c 1 = 100 c 2 = 1 r = 0.1 b = 0 Z 1 (0) = 10 Z 2 (0) = 5 Table 2. linear MPE v * 11 = -1.63 v * 12 = -0.3 v * 1 = 125.22
MPE Z m = 13.06 J m 1 = 1702.44 J m 2 = 1118.57 J m 1+2 = 2821.02 OLNE Z o = 12.5 J o 1 = 3320.99 J o 2 = 2736.89 J o 1+2 = 6057.88 Cooperation Z c = 10 J c 1 = 3484.63 J c 2 = 2869.89 J c 1+2 = 6354.53
In particular, it can be seen that the different stationary outcomes of the game, i.e. that of the open-loop Nash equilibrium Z o , of the MPE Z m and of the cooperative solution Z c , are ranked as follows:

Z c < Z o < Z m .
The cooperative solution in this setting corresponds to the optimal paths chosen by the cartel and must not be confused with a Pareto outcome: the investment choice (Q c 1 (.), Q c 2 (.)) provides the higher aggregated profits for the firms at the expense of the demand side of the economy. In this context of negative externalities it is not surprising to find that the long run cooperative outcome is lower than the non cooperative ones. And, since markovian interactions enhance competition, it is not puzzling either that the long run steady state is higher than the open loop (static Cournot) one. In fact the markovian industry capacity lies between Cournot and perfect competition (the same conclusion holds in Driskill and MacCafferty 1989) and yields a lower capacity price.

Numerically one can also check that the payoffs under the different outcomes verify :

J c i > J o i > J m i .
In this model, commitment is beneficial for the firms (although it is not pareto improving). 7.2. Arms race. Van der Ploeg and De Zeeuw (1990) have analyzed the question of arms accumulation as a differential game between two countries. This is a micro-founded model where the representative agent's welfare in each country depends on the level of security, which is perceived as an increasing function of the domestic stock of weapons but a decreasing function of the foreign stock. Actually what matters is the discrepancy between the two countries' arms stocks. A country's welfare also depends on the consumption of a private good and leisure. Because of the resource constraint the higher a country's investment in arms the higher the security but the lower the private good consumption and leisure: this is what the authors called the "butter versus guns" dilemma. The reduced form of their model boils down to the following expression for the function P i (Z i , Z j ) :

P i (Z i , Z j ) = p 1 (Z i -Z j ) - 1 2 p 12 (Z i -Z j ) 2 , p 1 > 0, p 12 > 0 .
In other words the coefficients of the function P i are:

p 0 = 0, p 1 = -p 2 > 0, p 11 = -p 12 = p 22 < 0.
Because the cross product term Z i Z j involves a positive factor p 12 > 0, the game is one of markov complementarity (Theorem 5.1).

The cost to increase the arms stock is:

C (Q i ) = c 2 2 Q 2 i , c 2 > 0,
and the differential equations for the evolution of the weapons stocks are given by [START_REF] Amir | Stochastic games in economics and related fields I: a general survey[END_REF]. The cooperative outcome leads to a moratorium on investment in weapons, whereby weapons are driven down over time to zero. Turning to non cooperative scenarii, the OLNE presumes that countries cannot condition their investment in arms on the rival's weapons stock (because for instance they cannot observe it) and lead to positive steady states. On the contrary the MPE presumes that countries can monitor the foreign weapons stock. The MPE leads to bilateral lower levels of arms and is therefore more efficient; the authors conclude that a unilateral arms treaty should allow countries to observe their rival's weapons stocks. In other words, observability leads to lower weapon stocks and higher long run welfare for both countries, i.e. [START_REF] Melese | Reversing the arms race: a differential game model[END_REF] have considered a slightly more general version of this game. In their resolution they use the reduced-form of this model with one single state variable, the difference of the weapons stocks. There are two jurisdictions indexed by i = 1, 2, building their public infrastructures Z 1 (t) and Z 2 (t) over time. Each jurisdiction is inhabited by a representative agent. His preferences are reflected by a linear utility function u i (x i , s i ) = x i + s i , defined over each possible pair of a consumption good x i and an index s i of the services that stem from the public infrastructures Z 1 and Z 2 . Put differently Z 1 and Z 2 are the inputs in the joined production process of s i

Z c < Z m < Z o , J c i > J m i > J o i .
s i = P i (Z i , Z j )
As a concrete illustration one can imagine two jurisdictions sharing the same waters (or the same international airspace); the environmental policies undertaken in one jurisdiction affect the welfare of the other.

Let us focus on quadratic technology P i such as:

P i (Z i , Z j ) = p 1 (Z i + Z j ) -(2 Z i -Z j ) 2 , p 1 > 0, i = 1, 2, j = 1, 2, i 6 = j,
i.e. the parameters are:

p 0 = 0, p 1 = p 2 > 0, p 11 = -8, p 12 = 4, p 22 = -2.
Note that the function P i (., .) is concave with respect to Z i ; for some range of the stocks the index of jurisdiction i is increasing with its own stock of infrastructure (P i 1 > 0), and the externalities between jurisdictions' infrastructures are positive ( P i 2 > 0). Finally, notice from the payoffs (P 12 = 4 > 0) and Theorem 5.1 that the MPE to follow will be characterized by markov complementarity between agents, i.e.

∂Q j ∂Z i > 0.
At each point of time the jurisdiction i can invest Q i to change her public stock of infrastructures, so that :

Żi = Q i , Z i (0) = Z i0 .
It is assumed that investment is costly and given by the increasing and convex function:

C(Q i ) = c 2 2 Q 2 i .
and it is measured in the same terms as the utility of the consumption good and s i .

Each jurisdiction also benefits from an exogenous revenue y i through lump sum taxes to finance the consumption good expenses and the cost of investment: [START_REF] Driskill | Durable goods oligopoly[END_REF] x

i + C(Q i ) = y i .
The substitution of the budget constraint [START_REF] Driskill | Durable goods oligopoly[END_REF] in the utility function yields:

W i (Z i , Z j ) = y i + P i (Z i , Z j ) -C(Q i ) , i= 1, 2.
At last, setting y i = 0, i = 1, 2 without loss of generality, one obtains a differential game where the dynamics are:

Żi = Q i , Z i (0) = Z i0 , i= 1, 2
and the payoffs are:

J i = Z ∞ 0 e -r t h p 1 (Z i (t) + Z j (t)) -(2 Z i (t) -Z j (t)) 2 - c 2 2 Q 2 i i dt
I shall call γ m c this game. It is possible to give an existence result for the MPE of this game.

Theorem 7.1. The game γ m c admits a convergent linear MPE if: p (cr 2 + 24)

• 1 2 cr 2 - 1 3 ¸-√ cr µ 1 2 cr 2 + 17 3 ¶ < 0.
Proof: appendix E.

From the condition given in this theorem, a more tractable sufficient condition for the existence of a convergent MPE is c 2 r 2 ≤ 2/3. Note that this result does not preclude the existence of multiple MPE. Consider the following numerical example. Table 4. Parameter values In contexts of positive externalities non cooperative equilibria generally result in under-investment. From Theorem 6.1 we can be more precise, and rank the outcomes as follows:

p 0 = 0 p 1 = 60 p 2 = 60 p 11 = -8 p 12 = 4 p 22 = -2 c 1 = 0 c 2 = 6 r = 0.1 b = 0 Z 1 (0) = 10 Z 2 (0) = 0
Z o < Z m < Z c .
And in the numerical example it turns out that :

J o i < J m i < J c i .
In the game γ m c it appears that markov behaviors induces outcomes that are nearer to the paretian criterion in the long run. In other words, for this model commitment is not pareto improving. Rather, if monitoring of the neighbor's infrastructures were possible, the local incentives to invest would reinforce each other.

Note also that under the following alternative properties for the production function of the index s i P i 1 (Z i , Z j ) < 0 , and P i 12 (Z i , Z j ) < 0 , commitment would have been pareto improving.

Conclusion

In the context of differential games with two state variables this paper assesses how markov interactions affect non cooperative (Nash) decisions.

First it is shown how the payoff structure indicates whether the economic situation at hand is characterized by markov substitutability or markov complementarity. Second, focusing on the steady states and comparing the MPE with the OLNE, it is proved that if competition is increased in games with markov substitutability, it is lessened when agents experience markov complementarity. These simple findings are illustrated by means of several well-known examples from the literature.

It is sometimes argued that the MPE is a fully dynamic concept, for it is made of strategies that would remain equilibrium strategies if they were to be reconsidered in the future, whatever the state of the system. On the contrary the OLNE does not generally possess this subgame perfection property; it involves only one decision date, at the beginning of the game, and it is presented as a somewhat static concept. Adding to this argument, many authors have exhibited dynamic situations where the OLNE has a stationary state identical to the Nash equilibrium of the associated static "limit game". The associated limit game obtains when key parameters tends to specific values so that the structural time-dependence is removed. On the contrary the MPE stationary state never coincides with the static equilibrium. Those supporting the idea that a static game should be thought of as a dynamic game in a stationary environment should then be warned that using the static Nash equilibrium concept amounts to make an approximation, and the extent of this approximation depends on the gap between the steady states of the OLNE and the MPE, which itself depends on whether the dynamic game is characterized by markov complementarity or markov substitutability.

To conclude I suggest three extensions. A first would be to ascertain that those results also hold with discrete-time versions of our differential games. A second would be to check the robustness of the conclusions when the non cooperative outcomes are Stackelberg equilibria, rather than Nash equilibria. A last direction for future research would be to explore the importance of complementarity and substitutability concepts in a class of dynamic games that goes beyond the linear quadratic specification. This can hardly be done without the recourse to numeric methods.

1 c 2 V i 2 V j 1 -b V i 2 Z j ,
For i = 1, 2. Because of the linear quadratic nature of the problem one can "guess" quadratic value functions :

V i (Z i , Z j ) = v 0 + v 1 Z i + v 2 Z j + v 11 2 Z 2 i + v 12 Z i Z j + v 22 2 Z 2 j ,
for which :

V i 1 = v 1 + v 11 Z i + v 12 Z j , V i 2 = v 2 + v 12 Z i + v 22 Z j .
Equivalently for agent j :

V j 1 = v 1 + v 11 Z j + v 12 Z i , V j 2 = v 2 + v 12 Z j + v 22 Z i .
Reporting those expressions in HJB one obtains :

r v 0 + r v 1 Z i + rv 2 Z j + r v 11 2 Z 2 i + r v 12 Z i Z j + r v 22 2 Z 2 j = p 0 + p 1 Z i + p 2 Z j + p 11 2 Z 2 i + p 12 Z i Z j + p 22 2 Z 2 j + 1 2c 2 (v 1 + v 11 Z i + v 12 Z j ) 2 + (c 1 ) 2 2c 2 - c 1 c 2 [v 1 + v 2 + (v 11 + v 12 )Z i + (v 12 + v 22 )Z j ] -b(v 1 Z i + v 11 Z 2 i + v 12 Z i Z j ) + 1 c 2 (v 2 + v 12 Z i + v 22 Z j )(v 1 + v 11 Z j + v 12 Z i ) -b (v 2 Z j + v 12 Z i Z j + v 22 Z 2 j ) .
Developing the r.h.s. yields a constant term and Z i , Z j , Z 2 i , Z i Z j , Z 2 j terms. Since this equation must be verified for all (Z i , Z j ), by identification we find the so-called Coupled Riccati equations for the coefficients in the value functions:

r v 0 = p 0 + (v 1 ) 2 2c 2 + (c 1 ) 2 2c 2 - c 1 c 2 (v 1 + v 2 ) + v 1 v 2 c 2 , (8)
that satisfy the coefficients of the value function at the equilibrium. Those equations correspond to that given in theorem 4.1. From [START_REF] Driskill | Dynamic duopoly with adjustment costs: A differential game approach[END_REF] the MPE strategies are then:

Q * i = v * 1 -c * 1 c 2 + v * 11 c 2 Z i + v * 12 c 2 Z j .
Plugging these strategies into the differential system (1) one can deduce the stability conditions: This is not sufficient however for a convergent MPE, since the stability conditions imposes that the intersection point in the relevant quadrant is also such that |v * 11 | > |v * 12 | (see figures G2a and G2b). To prove that consider first the point v11 such that v11 = -φ(v 11 ). To complete the proof it is necessary to show that ṽ 11 = Γ(v 11 ) < v11 (as can be seen from figure G2a). We need first to calculate v11 :

v * 11 -bc 2 < 0 , (14)

7. 3 .

 3 Public infrastructures competition. In Figuières (2002) one finds a dynamic extension of a public good problem proposed by Widasin (1991).

(v * 11 --r c 2 v 11 2 + p 11 c 2 2 + (v 11 ) 2 2 - 2 + p 22 c 2 2 + (v 12 ) 2 2 + v 11 v 22 -bc 2 v 22 .+ p 11 c 2 2 + (v 11 ) 2 2 - 12 v 11 v 11 = Γ(v 12 ) cr 2 Fig. G1 .

 112222222222221211112G1 Fig. G1. Existence of an intersection point in the relevant quadrant

2 + c 2 r v11 / 2 + 4 c 2 .-4c 2 = 0 . 3 .

 22203 This equation has two roots of opposite signs and only the negative one is relevantNext the inequality Γ(-v 11 ) < v11 can be explicitly stated to give-9v 3 11 + (c 2 2 r 2 + 34 c 2 ) v11 + 4 c 2 2 r 8 c 2 -v11 < 0 ,

Table 3 .

 3 Steady states and payoffs

Table 5

 5

	. linear MPE
	v * 11 = -6.13 v * 12 = 1.84 v * 1 = 103.6

Table 6 .

 6 Steady states and payoffs

	MPE OLNE Cooperation Z c = 120 J c Z m = 24.01 J m 1 = 22646.5 J m 2 = 22595.81 J m 1+2 = 45242.32 Z o = 15 J o 1 = 15885.18 J o 2 = 15324.63 J o 1+2 = 31209.81 1 = 51686.21 J c 2 = 46910.93 J c 1+2 = 98597.14

Stackelberg equilibria, sometimes encountered in the literature, will not be considered here.

In addition, establishing conditions of existence for the general class of games would

See for instance Bulow, Geanakoplos and Klemperer(1985), or Tirole (1988).

Appendix

Appendix A. Proof of theorem 4.1

In order to characterize the MPE (Q * 1 (Z 1 , Z 2 ), Q * 2 (Z 1 , Z 2 )) we will make use of dynamic programing in continuous time. The value function of agent i is defined a follows :

where the state of the system is governed by

). The Hamilton-Jacobi-Bellman (HJB) equations give necessary conditions to characterize the MPE (or feedback Nash equilibrium) :

The first order condition for the maximization of the r.h.s. of the HJB equation is :

If we report [START_REF] Driskill | Dynamic duopoly with adjustment costs: A differential game approach[END_REF] into HJB we obtains the following system with partial derivatives :

and v * 22 follows. Otherwise when p 12 6 = 0, making use of the stability conditions and the fact that p 22 ≤ 0,

From this inequality and the second equation

Therefore p 12 c 2 v 12 > 0, meaning that v 12 and P i 12 = p 12 have the same sign. QED.

Appendix C. Proof of lemma 1

From the first stability condition of the MPE, one can deduce

From the above inequality and the second stability condition, it follows

QED.

Appendix D. Proof of Theorem 6.1

Let us define the function

where ψ can take any value over the interval ]-1; 1[ . Note that when ψ = 0 the above expression boils down to the OLNE steady state; when ψ ' 1 this expression is arbitrarily close to the efficient steady state; finally ξ According to theorem 4.1 the Riccati equations lead in this example to the system for v 11 and v 12 : ( 16)

2 r , that satisfy the coefficients of the reaction functions at the equilibrium. This system, along with the stability conditions :

for a convergent equilibrium, will permit us to prove the theorem.

The stability condition requires that v * 11 < 0 and (v * 11 ) 2 -(v * 12 ) 2 > 0. Because here p 12 > 0 we also know from Theorem 5.1 that v * 12 > 0. So we look for solutions such that v * 11 < 0, v * 12 > 0 and |v * 11 | > |v * 12 |. The first equation of ( 16) can be viewed as a quadratic equation for v 12 . Its roots are of opposite sign (when not both null), and only the positive roots is admissible since v * 12 > 0. This first equation thus defines v * 12 as an explicit function of v * 11 :

It is straightforward to check that this function has the following properties : φ 0 > 0 , φ 00 < 0 . The second equation of ( 16) implicitly defines v * 11 as a function of v * 12 . Using the expression [START_REF] Reynolds | Capacity investment, preemption and commitment in an infinite horizon model[END_REF],

Plugging this expression into the second Riccati equation, after simplifications and rearrangements one obtains :

which simplifies to p (c 2 r 2 + 24)

The above inequality provides a sufficient condition for a convergent linear MPE and completes the proof. Two last remarks are worthwhile: i) a sufficient condition for this sufficient condition to hold is c 2 r 2 ≤ 2/3, ii) there might exist multiple convergent MPE. QED.