Anh Duy

Anh Nguyen

Duy

Alternative reversal variable

Keywords: Asset pricing models, short-term reversal, momentum, anomalies, market states. JEL classification: G12

come

Introduction

Financial literature proposes two investment strategies based on past returns. Specifically, one is based on firm's return of twelve months prior to the current month, called momentum strategy [START_REF] Jegadeesh | Returns to Buying Winners and Selling Losers : Implications for Stock Market Efficiency[END_REF]). The other is based on firm's previous month return, called short-term reversal strategy [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF]). For the momentum, it becomes conventional to construct it by taking past stock return from t-13 (or t-12) to t-2 as the criteria to classify the stock. The most recent month (t-1) is skipped in order to purge the negative effect of return reversal. Now, considering a simple situation that a stock, which is associated with both high prior month (t-1) and high past 12-month return (t-13 to t-2), how do we do? knowing that this stock will belong to the short side of reversal strategy, while belong to the long side of momentum strategy. If we implement [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal strategy (put this stock in the short side), the profit of short-term reversal profit is likely negatively affected by this stock, who exhibits a strong momentum.

In this paper, we begin by asking a simple question: does it exist a simple way to alleviate the momentum in individual stock return to construct the short-term reversal variable or forming the short-term reversal strategy.

In fact, it could be analytically shown by using the framework of [START_REF] Lo | When Are Contrarian Profits Due to Stock Market Overreaction[END_REF] that [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal strategy could be decomposed into two components. The first is the difference between realized of previous month return and the average of stock return from t-13 to t-2 (hereafer R 1-12 ). The second, which involves buying the stocks that have outperformed the market portfolio and selling the stocks that have underperformed the market portfolio, is equivalent to momentum strategy. This decomposition not only implies that the profit of [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal strategy is negative affected by stock return momentum, but also suggests that using R 1-12 instead of previous month return (hereafter R 1 ) to implement the reversal strategy would give higher profit since this variable allows us to isolate the short-term reversal from the momentum in stock return.

Consistent with this decomposition, we find that reversal strategy based on R 1-12 earns significantly higher risk-adjusted return than those of conventional reversal strategy, which is based on previous raw return. On average, the short-term reversal strategy based on R 1-12 provides 25 bps improvement from that of standard reversal strategy. This improvement is unaffected by adjustment for common risk factors. The evidences obtained from the direct comparison based on [START_REF] Fama | Risk, Return, and Equilibrium: Empirical Tests[END_REF] cross-sectional regressions corroborate with these findings.

Prior literature reports that states of market affect significantly the profitability of momentum strategy (e.g. [START_REF] Grundy | Understanding the Nature of the Risks and the Source of the Rewards to Momentum Investing[END_REF] and [START_REF] Cooper | Market States and Momentum[END_REF]). Recently, [START_REF] Hsu | Asymmetric effect of style comovement on momentum[END_REF] show that the variation of momentum profit across market states exits under style investing 2 . The above decomposition also suggests that the time-varying effect of momentum profit could have impact on the profit of reversal strategy. In other words, the states of market would have indirect impact on the profit of reversal strategy via the influence it has on the momentum. Consistent with this assertion, we find that [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal profit generated following 'up' market is about one third of the one realized following 'down' market. Similarly, we find that the profit of R 1-12 based reversal strategy following increase market is significantly lower than following decline market. Importantly, we observe that, following market decline, when momentum effect is weak or does not exist, the profits of these strategies are not significantly different. However, following market increase, when momentum effect is strong, R 1-12 based reversal strategy's return is considerably higher than conventional reversal strategy's one. The finding suggests that R 1-12 helps to reduce the strong momentum effect of stock price during 'up' market.

We check the robustness of our findings by conducting the out-of-sample tests. In particular, we replicate these above results for European stock market. In overall, the results obtained in E.U market corroborate with U.S evidences.

We deem that our study contributes to the existing literature in the following ways. First of all, we show that [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal effect is negatively affected by the momentum in stock return and propose a simple way to alleviate this problem. The idea is simple but certainly has important implication for the theoretical and practical purpose, given the number of studies, which employ the previous month return to control the reversal effect, and significantly improved reversal profit it offers. Secondly, our results also have important implications for several explanations that have been proposed in the literature to explain the short-term reversal. We show that market states also have a significant impact on short-term reversal profits, but in the inverse direction to what occurs for momentum profit. In particular, we find that the reversal profits are significantly higher in 'down' market than in 'up' market. The time-varying reversal profit across market states could be either due to the negative effect to the time-varying momentum profit and liquidity provision, which should be more pressing in the 'down' market. Our results are therefore complementary to the findings in [START_REF] Da | A Closer Look at the Short-Term Return Reversal[END_REF][START_REF] Avramov | Liquidity and Autocorrelation in Individual Stock Returns[END_REF] and [START_REF] Hameed | Stock market declines and liquidity[END_REF] and suggest that momentum could have significant impact on the strength of reversal. These evidences provide a different perspective of reversal profits, which could contribute further to the understanding of short-term reversal -an important phenomenon in stock price that is difficult to reconcile with risk-based model of expected return.

The rest of the paper is organized as follows. In Section 2, we decompose the conventional reversal strategy to the strategy based on R 1-12 and the momentum strategy. In Section 3, we describe the data using in this paper.

Section 4 presents our main findings on the performances of alternative reversal strategies. We verify the consistency of results by examining EU equity markets. In this section, we also investigate the influence of market states on reversal profits. We provide our concluding remarks in Section 5.

Analytical analysis

We follow [START_REF] Lo | When Are Contrarian Profits Due to Stock Market Overreaction[END_REF] and consider [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF] reversal strategy that assigns a portfolio weight to stock 'i' at time 't' of

W i,t = - 1 N (R i,t-1 ) -R M,t-1 (1)
Where R M,t-1 = N i=1 R i,t-1 and N is the number of securities in the market.

The weight of [START_REF] Jegadeesh | Evidence of predictable behavior of security returns[END_REF]'s strategy implies buying the previous month loser securities and selling short the previous month winner securities.

The profit for this strategy at time 't', P t , is given by:

P t = N i=1 W i,t R i,t or P t = - 1 N N i=1 (R i,t-1 -R M,t-1 )R i,t (2)
To show the effect of momentum, we add and subtract E12 i,t-1 , which is measured as 1/12 13 k=2 R t-k , to equation ( 2)

P t = - 1 N N i=1 (R i,t-1 -E12 i,t-1 -R M,t-1 + E12 i,t-1 )R i,t (3)
or we can rewrite equation (3) as:

P t = - 1 N N i=1 (R i,t-1 -E12 i,t-1 )R i,t - 1 N N i=1 (E12 i,t-1 -R M,t-1 )R i,t (4)
The decomposition from equation (4) tells that profit of conventional reversal strategy P t is likely negatively affected by the momentum return (the second component) and using the difference between previous month return and its 12-month average return (R 1-12 ) to create the reversal strategy would probably provide higher profit than the one generated from strategy based on previous return. 3 . In Graph 1, we plot the returns on reversal and momentum strategies. As could be seen in the Graph, the correlation between them is negative. In other words, reversal return tend to be higher in the period of momentum crash.

Data

The data includes NYSE, AMEX, and NASDAQ common stock monthly returns from July 1963 to December 2016. We exclude stocks belonging to 5 percent smallest market capitalization to alleviate the potential microstructure problem associated to these small stocks4 .The market return is the value-weighted index of all listed firms in CRSP and the risk-free rate is the one-month Treasury bill rate, both obtained from Ken French's data library5 .

Results

Reversal strategies

We begin our analysis by computing the principle variable, R 1-12 , as suggested by equation 4, which, is measured as the difference of previous return and the average of 12-month return from t-2 to t-13. However, we change a little bit to adapt for the empirical regularities. In fact, Novy-Marx (2012) shows that momentum return based on firm's performance twelve to seven months prior to the current month are stronger than the one based on firm's performance six to two months prior. To put more weight on the firm's performance twelve to seven months we compute R 1-12 as 13 1, we report the equal-weighted raw and riskadjusted monthly returns for the recent loser, recent winner and the conventional reversal strategy. Consistent with Jegadeesh (1990)'s findings, we find a significant profit of 1.26% (t-statistic = 6.96) per month. The risk-adjusted profits are still large 7 . The CAPM and three-factor alphas are 1.12% (tstatistic = 6.46) and 1.05% (t-statistic = 5.72) per month respectively. The 6 Our main results remained unchanged when we compute R 1-12 as suggested by equation ( 4), which is computed as [START_REF] Goyal | Is momentum an echo?[END_REF], in their investigation of 37 other major stock markets, points out that there is no such 'echo' in return and that Novy-Marx (2012)'s findings are likely driven by the effect of short-term reversals from month t-2, indeed. Though their studies entertain different hypothesis, but this weighting scheme is consistent with both.

k=2 k 12 (R t-1 -E12 t-1 ) 6 In Panel A of the Table
R 1-12 = 13 k=2 1 12 (R t-1 -E12 t-k ). Note that
7 The risk-adjusted profits are measured by regressing the monthly reversal profits on alternatively excess return of the market portfolio Sharpe (1964) and [START_REF] Fama | Common risk factors in the returns on stocks and bonds[END_REF] three factors, which add the size factor (SMB) and the value factor (HML) to the excess return of the market portfolio. R 1-12 based reversal profits are reported in the Panel B. The profit is 1.51% (t-value = 9.15) per month, which is higher than that generated by the standard reversal strategy. The CAPM and three-factor alphas are 1.38% (t = 7.98) and 1.43% (t = 7.56) respectively. The additional profit of 0.25% per month is highly significant (t-statistic = 3.53) and is unaffected by adjustment for common risk factors. *** Insert Table 1 about here *** We also employ the [START_REF] Fama | Risk, Return, and Equilibrium: Empirical Tests[END_REF] style cross-sectional regression to compare these strategies simultaneously 8 . From Table 2, we see that the return from the R 1-12 based reversal strategy is 1.12% (t-statistic = 7.30) per month versus the conventional strategy's 0.45% (t-statistic = 1.92) per month. Dominance of R 1-12 based reversal strategy becomes appearance when we look at the Fama and French risk-adjusted return. The risk-adjusted return for R 1-12 based reversal strategy is 1.20% (t-statistic = 8.76), while those for Jegadeesh's one become insignificant 0.18% (t-statistic = 0.73).

Overall, the results provide supports to our arguments that using R 1-12 , 8 In particular, we estimate the following cross-sectional regressions,

R t = a 0 + j l,t RL 1,t-1 + j h,t RH 1,t-1 + l t RL 1-12,t-1 + h t RH 1-12,t-1 + t (5)
Where R t is month 't' stock return. The independent variables are dummies that indicate whether the stock is held, either long or short in month 't' as part of one of the two strategies. In particular, RL 1,t-1 equals one if stock's previous month performance is in the bottom 20% and is zeros otherwise. RW 1,t-1 equals one if stock's previous month performance is in the top 20% and is zeros otherwise. RL 1-12,t-1 and RH 1-12,t-1 are defined similarly but stocks are ranked by R 1-12 measure.

which allows us to purge the strong momentum of individual stock return from the reversal, will improve the reversal profit. *** Insert Table 2 about here ***

Market states and reversals

To test the relation between market state and short-term reversal, we employ [START_REF] Cooper | Market States and Momentum[END_REF]'s approach. In particular, we first define the market state as 'up' ('down'), if the cumulative return of CRSP value weighted return, including dividends, over the formation period of momentum strategy (in our case is 12 months) is positive (negative). We also consider the alternative definition of market state. In particular, we define the market state as a 'up' ('down') if the cumulative CRSP value-weighted return in the past 24 months is positive (negative) (e.g [START_REF] Daniel | Momentum crashes[END_REF]. To test whether reversal profits in each market state are equal to zero, we regress the time series of average monthly reversal profits on two dummy variables for 'up' and 'down' market, with no intercept. To test if mean profit in 'down' (or low state) market are different from profits in 'up' (or high state) market, we regress average monthly reversal profits on 'down' market dummy, with a constant. This approach helps to preserve the full-time series of returns and allow us to estimate t-statistic that robust to auto-correlation and heteroscedasticity using Newey and West (1987) standard errors.

Since the results obtained from both definitions of market state are similar, we are focus only on the results for market states that are defined based on the cumulative return of 12 previous months. As can be seen in Panel A of Table 3 of the Table, from 1964 to 2016.12, following 'down' market, the raw, the CAPM and FF3 risk-adjusted reversal profits are 2.34% (t-statistic = 5.61), 2.28% (t-statistic = 5.49) and 2.34% (t-statistic = 5.61) respectively9 .

However, the reversal profits in the 'up' market are much lower. They are approximately one third of the ones realized in the 'down' market. Following the period of market increase, the raw, CAPM and Fama and French riskadjusted profits are 0.86% (t-statistic = 5.15), 0.61% (t-statistic = 3.73), and 0.66% (t-statistic = 3.92) per month respectively. The t-statistic for testing the equality of the profits across 'up' and 'down' market are reported in the last row of Panel A. These t-statistics suggest that the reversal (adjusted) profits are statistically distinguish between two states.

We also examine if the R 1-12 based reversal strategy return varies with market states. The results are also reported in Table 3 (the left column). We see that following 'down' market, the raw profit, CAPM and Fama and French risk-adjusted profits are 2.30% (t-statistic = 5.86), 2.28% (t-statistic = 5.93), and 2.40% (t-statistic = 5.72). Compare to the conventional reversal profit in the 'down' market, they are approximately the same. This result should not be surprise because the objective of R 1-12 is to reduce the effect of momentum.

In the 'down' market, when the momentum effect appears weak, the profit generated from R 1-12 should be not (or less) significantly different to the one generated from raw return. Following 'up' market, the R 1-12 based reversal strategy's (adjusted) returns are still statistically significant. Following 'up' market, compare to the conventional reversal profits, the R 1-12 based reversal strategy's profit are higher. We see that the additional profits (for both raw and adjusted ones) are consistent with those reported in Table 1 andTable 2, about 30bps per month. However, R 1-12 based reversal strategies profits obtained in the 'down' market is significantly higher than those in 'up' market. The results could be explained by liquidity provision. Because, in the 'down' market, it is also the period of liquidity dry-up, explaining higher reversal returns (see [START_REF] Campbell | Trading volume and Serial correlation in Stock Returns[END_REF], [START_REF] Avramov | Liquidity and Autocorrelation in Individual Stock Returns[END_REF]). However, this result could also be consistent with investor's overeaction. For example, [START_REF] Heyman | Investor attention and short-term return reversals[END_REF] argue that the investor tend to overreact in the 'down' market, leading to recent 'winner' stocks are more likely to revert 10 . They show that besides liquidity, overreaction is an important factor that drives price reversals, especially during times of high volatility.

*** Insert Table 3 We also investigate the implication of R 1-12 on expected return in the EU data. This non-US examination delivers a useful out-of-sample test on the implication of R 1-12 on expected returns. The obtained results confirm that the higher performance of R 1-12 based strategy relative to conventional reversal strategy presents not only in U.S. equity market but also in other equity markets in European areas 11 . Also, we see that R 1-12 based reversal strategy profits are significantly different conditioning on market states. The t-statistic for testing the equality of reversal profit across market states are highly significant.

Intra-industry reversals and Residual return reversals

Hameed and Mian (2014) find that compare to conventional reversal strategy, intra-industry reversals are stronger in magnitude and robust to market microstructure biases. We find that R 1-12 performance goes beyond the industry control.

We also compare the R 1-12 based reversal strategy to residual return reversals strategy proposed by [START_REF] Blitz | Short-term residual reversal[END_REF] and find that R 1-12 strategy generates higher return than the residual return based strategy. In addition, one problem with residual return is that it does not cover the whole sample of 11 The results are reported in Appendix A population of stocks, much of small stocks are set to missing in the process of residual return estimation. Consequently, when controlling for residual return in the cross-sectional regressions, the coefficient of lag return remains highly significant. This imply that lag one month return owns the information that does not belong to lagged residual return. In contrast, the coefficient of lag return become small and insignificant after controlling for R 1-1212 .

Conclusion

It becomes conventional to implement the momentum strategy by taking the past return from 't-12' (or 13) to 't-2', skipping 't-1' as the criteria to classify the stock in order to purge the negative effect of return reversal. However, in implementing the reversal strategy, we tend to ignore the strong momentum in individual stock return. A simple subtract the past 12-month average return from the recent return allows us to alleviate the return momentum.

Consequently, the reversals are significantly stronger. The additional profit of implementing the reversal strategy based on R 1-12 is statistically and economically significant. For example, on average the short-term reversal strategy based on R 1-12 yields returns that are higher than those of a conventional short-term reversal strategy about 25 bps per month. This improvement is unaffected by adjustment for common risk factors. The additional profit will be more impressive, about 45 bps per month, if we implement this strategy within industry.

Our results also contribute to the study of reversal anomaly by showing that market states have significantly impact on the profit of reversal strategy through momentum effect. We find that conditioning on the states of market has a significant impact on the profit of reversal strategies. In particular, following the 'down' market, when momentum effect appears weak, the profit of reversal strategy is significantly higher than following 'up' market, when momentum effect is strong.

(R t-1 -E12 t-k ).
'Loser -Winner' is the spread between loser and winner portfolio's return, while 'Low -High' is the spread between low and high R 1-12 portfolio's return. Diff is the difference between 'Low-High' and 'Loser -Winner' portfolio returns. Risk-adjusted returns are estimated by the capital asset pricing model (CAPM) and [START_REF] Fama | Common risk factors in the returns on stocks and bonds[END_REF] three-factor model (FF3). These factors are available at French's website. The reported t-statistic (in parentheses) are Newest West (1987) corrected. The sample period is from 1964.07 to 2016.12. Note: Each month between 1964.07 to 2016.12, a cross-sectional regression of the following form is estimated

R t = a 0 + j l,t RL 1,t-1 + j h,t RH 1,t-1 + l t RL 1-12,t-1 + h t RH 1-12,t-1 + t
Where RL 1 (RL 1-12 ) equals one if stock i's previous month return (R 1-12 ) is in the bottom 20% and is zeros otherwise. RH 1 (RH 1-12 ) equals one if stock i's previous month return (R 1-12 ) is in the top 20% and is zeros otherwise. The raw returns in the table are the time series average of these coefficients. Risk-adjusted return are the estimated intercepts from the time-series regressions of these averages on the contemporaneous market factor (CAPM) and Fama-French three factors (FF3). The reported t-statistic (in parentheses) are Newest West (1987) corrected. Note: The table present the conventional and R 1-12 based reversal returns in 'up' and 'down' market. We also reported the risk-adjusted returns across market states, where CAPM and Fama and French adjusted returns are defined as the return net of what is attributable to exposure to the market factor and Fama and French (1993) three factors respectively. Down = U p is the t-statistic of the test whether momentum profits in each state respectively equal to zeros. Panel A (Panel B) reports the results where market states are defined based on the cumulative return of the value weighted market index including dividends 12 (24) months priors to beginning of the holding period. The sample period is from 1964.07 to 2016.12. Note: To form the (R 1-12 based) reversal strategy within industries, we first sort stocks into industry groups based on Fama-French 10 industry classification and then rank stocks based on previous month returns (R 1-12 ) to form the equally-weighted lowest 20% and highest 20% portfolios within each industry. The lowest minus highest portfolio return in each industry is average across all industries to obtain intra-industry (R 1-12 based) reversal return. The Table reports the conventional reversal, intra-industry reversal return and intra-industry R 1-12 based reversal returns. Risk-adjusted returns are estimated by CAPM and Fama and French three-factor models. The reported t-statistic (in parentheses) are Newest West (1987) corrected. The sample period is from 1964.07 to 2016.12.

with residual return is that it does not cover the whole sample of population of stocks, much of small stocks are set to missing in the process of residual return estimation. Consequently, when controlling for residual return in the cross-sectional regressions, the coefficient of lag return remains highly significant. This imply that lag one month return owns the information that does not belong to lagged residual return. In contrast, the coefficient of lag return become small and insignificant after controlling for R 1-12 (Table B.4).

Figure 1 :

 1 Figure 1: Time-series of returns from momentum and reversal strategy. This plot shows the returns on reversal and momentum strategies for the period from 1964.07 to 2012.06

  about here ***

	4.3 Robustness checks
	4.3.1 E.U stock markets
	10 They measure the investor's overreaction by Google Search Volume Index

Table 1 :

 1 Portfolio returns

		Raw returns	Risk-adjusted Returns
			CAMP	FF3
	Panel A: Conventional reversal strategy
	Loser	1.871	0.786	0.564
		(5.70)	(4.19)	(4.20)
	Winner	0.609	-0.332	-0.485
		(2.13)	(-2.11)	(-4.62)
	Loser -Winner	1.262	1.118	1.050
		(6.96)	(6.46)	(5.72)
	Panel B: R 1-12 based reversal strategy
	Low	1.984	0.912	0.691
		(6.32)	(5.12)	(6.01)
	High	0.476	-0.463	-0.660
		(1.68)	(-2.98)	(-6.55)
	Low -High	1.508	1.376	1.351
		(9.94)	(9.15)	(8.96)
	Diff	0.246	0.257	0.301
		(3.53)	(4.04)	(4.47)
		13	k
		k=2	12

Note: Stocks are sorted alternatively by previous month return and by R 1-12 . Loser (winner) are the equally-weighted return of 20% of stocks with lowest (highest) previous month return.

Low (High)

, are the equally weighted return of 20% of stocks that have lowest (highest) R 1-12 , where R 1-12 =

Table 2 :

 2 Pairwise comparison

		Raw returns	Risk-adjusted Returns
			CAPM	FF3
	RL 1-12	0.442	0.383	0.380
		(5.35)	(4.75)	(4.91)
	RH 1-12	-0.679	-0.702	-0.818
		(-6.47)	(-7.24)	(-9.03)
	RL 1-12 -RH 1-12	1.121	1.086	1.198
		(7.31)	(7.63)	(8.76)
	RL 1	0.134	-0.023	-0.050
		(0.95)	(-0.18)	(-0.38)
	RW 1	-0.314	-0.349	-0.217
		(-2.25)	(-2.69)	(-1.72)
	RL 1 -RW 1	0.448	0.326	0.167
		(1.92)	(1.53)	(0.73)

Table 3 :

 3 Market states and reversal profits

		R 1-12 based reversal strategy	Conventional reversal strategy
	Panel A: Market states are defined based on 12-month cumulative market return
		nobs raw CAPM	FF3	raw CAPM	FF3
	Down	172 2.338	2.282	2.347	2.332	2.226	2.342
		(5.87) (5.94)	(5.61)	(5.02) (5.03)	(5.21)
	Up	457 1.195	1.000	1.009	0.859	0.606	0.657
		(8.63) (7.27)	(7.54)	(5.15) (3.73)	(3.92)
	Down = U p	(2.67) (3.15)	(2.91)	(2.94) (3.51)	(3.39)
	Panel B: Market states are defined based on 24-month cumulative market return
	Down	144 2.291	2.025	2.121	2.335	2.014	2.151
		(5.42) (5.27)	(4.89)	(4.83) (4.81)	(4.64)
	Up	474 1.268	1.144	1.146	0.932	0.751	0.799
		(8.34) (7.05)	(7.38)	(5.09) (3.86)	(4.26)
	Down = U p	(2.24) (2.22)	(1.91)	(2.66) (2.77)	(2.49)

Table A

 A Note: This table reports the average number of firms, average market equity (Size) and country's average percentage in term of total market equity for the countries included in the European sample. The data is from DataStream. The sample period is from 1990.01 to 2016.12.

	.1: Summary statistics for E.U sample
	Country	Firms Weight
	Austria	83	1,55
	Belgium	132	4,78
	Finland	91	2,64
	France	658	29,36
	Germany	639	28,52
	Greece	188	1,29
	Ireland	47	1,35
	Italy	211	11,15
	Netherlands 136	9,01
	Portugal	72	1,10
	Spain	129	9,24

Table A

 A Note: Each month between January 1990 to December 2016, a cross-sectional regression of the following form is estimated separately for European sample, Germany (GER), France (FR), ITALY, and Others which are group the remaining countriesR t = a 0 + j l,t RL 1,t-1 + j h,t RH 1,t-1 + l t RL 1-12,t-1 + h t RH 1-12,t-1 + tWhere RL 1 (RL 1-12 ) equals one if stock i's previous month (R 1-12 ) return is in the bottom 20% and is zeros otherwise. RH 1 (RH 1-12 ) equals one if stock i's previous month (R 1-12 ) return is in the top 20% and is zeros otherwise. The raw returns in the table are the time series average of these coefficients. Risk-adjusted return are the intercepts from the time-series regressions of these averages on the contemporaneous market factor (CAPM) and Fama-French three factors (FF3). The reported t-statistic (in parentheses) are Newest West (1987) corrected.

				.2: Reversal profits: E.U evidences
		EU		GER		FR		ITALY		OTHERS
		raw	adj	raw	adj	raw	adj	raw	adj	raw	adj
	a 0	0.944	0.584	0.843	0.578	1.186	0.678	0.522	0.235	0.904	0.448
		(3.53) (12.23)	(3.19) (8.40)	(4.26) (10.56)	(1.32) (3.03)	(3.10) (6.54)
	RL 1-12	0.385	0.497	0.381	0.333	0.255	0.418	0.549	0.644	0.346	0.527
		(2.93) (3.32)	(2.39) (2.02)	(1.62) (2.85)	(2.82) (2.96)	(1.61) (2.59)
	RH 1-12	-1.161 -1.535	-1.383 -1.831	-1.153 -1.520	-0.711 -0.871	-1.158 -1.500
		(-7.11) (-9.59)	(-5.65) (-7.43)	(-5.28) (-7.02)	(-3.15) (-3.64)	(-5.88) (-8.58)
	RL 1	0.198	-0.132	0.051 -0.184	0.861	0.486	-0.325 -0.520	0.106 -0.281
		(1.04) (-0.81)	(0.20) (-0.76)	(3.70) (2.26)	(-1.25) (-1.92)	(0.47) (-1.30)
	RW 1	0.687	1.131	0.643	1.088	0.072	0.571	0.429	0.596	1.071	1.408
		(2.58) (3.72)	(2.32) (4.06)	(0.24) (1.59)	(1.46) (1.94)	(2.79) (3.62)
	RL 1-12 -RH 1-12 1.546	2.032	1.764	2.165	1.408	1.938	1.259	1.515	1.504	2.027
		(6.56) (9.42)	(6.04) (7.40)	(5.00) (7.49)	(3.60) (3.91)	(4.15) (6.23)
	RL 1 -RW 1	-0.489 -1.263	-0.592 -1.271	0.789	-0.085	-0.754 -1.117	-0.965 -1.689
		(-1.36) (-3.44)	(-1.62) (-3.41)	(1.94) (-0.18)	(-1.58) (-2.18)	(-1.81) (-3.16)

  Table A.3: Market state and reversal profits: E.U evidencesNote: The table present the conventional and R 1-12 based reversal returns in 'up' and 'down' market for EU stock markets. We also reported the risk-adjusted returns across market states, where CAPM and Fama and French adjusted returns are defined as the return net of what is attributable to exposure to the market factor and Fama and French (1993) three factors respectively. Down = U p is the t-statistic of the test whether momentum profits in each state respectively equal to zeros. Panel A (Panel B) reports the results where market states are defined based on the cumulative return of the value weighted market index including dividends 12 (24) months priors to beginning of the holding period. The sample period is from 1990.01 to 2016.12

		Table B.1: Intra-industry reversals
					returns	Risk-adjusted Returns
		R 1-12 based reversal strategy	Conventional reversal strategy CAMP FF3
	Panel A: Market states are defined based on 12-month cumulative market return Intra-industry reversal return 1.591 1.470 1.396
		nobs raw	CAPM FF3	raw (9.86)	CAPM FF3 (9.45)	(8.72)
	Down Intra-industry R 1-12 based reversal return 119 1.765 1.640 1.810	1.485 0.933 1.709 1.600	1.556	1.568
		(5.71) (5.06)	(5.66)	(3.67) (2.35) (12.09) (11.49)	(4.05) (11.31)
	Up Intra-industry vs conventional reversal return 205 0.856 0.711 0.764	0.111 -0.215 -0.033 0.328 0.352 0.347
		(3.97) (3.29)	(3.49)	(0.35) (-0.68) (-0.10) (5.04) (5.48) (4.91)
	Down = U p Intra-industry R 1-12 based reversal return vs (2.31) (2.60) (2.30)	(2.74) (3.28) 0.447 0.482	(2.31) 0.519
	Panel B: Market states are defined based on 24-month cumulative market return conventional reversal return (4.68) (5.39) (5.28)
	Down Intra-industry R 1-12 based reversal return vs 107 1.682 1.507 1.664 (5.44) (4.43) (5.09) Intra-industry reversal return	1.581 0.981 0.118 0.130 (4.22) (2.61) (2.02) (2.41)	0.172 1.552 (4.31) (3.14)
	Up	217 0.947 0.828	0.894	0.140 -0.175 0.057
		(4.52) (3.96)	(4.10)	(0.41) (-0.53) (0.16)
	Down = U p	(1.96) (1.95)	(1.70)	(2.82) (2.93)	(2.29)

Table B .

 B 2: Residual return reversal strategiesNote: Stocks are sorted by residual terms obtained from the regression of R 1-12 on Fama and French three factors. Lower (higher) are the equally-weighted return of 20% of stocks with lowest (highest) previous month residual. The profit of residual reversal strategy using R 1-12 is the spread between the 1st and 5th quintile portfolio return. The Tablereportsthe profit of R 1-12 based reversal strategy and the[START_REF] Blitz | Short-term residual reversal[END_REF] reversal strategy using R 1-12 . The reported t-statistic (in parentheses) are NewestWest (1987) corrected. The sample period is from 1964.07 to 2016.12.

		Raw returns	Risk-adjusted Returns
			CAMP	FF3
	R 1-12 based reversal return	1.508	1.376	1.351
		(9.94)	(9.15)	(8.96)
	Residual reversal return using R 1-12	1.655	1.617	1.573
		(14.62)	(14.22)	(14.19)
	Diff	0.151	0.243	0.218
		(1.65)	(2.70)	(2.31)

Table B

 B Note: Table reports the estimated coefficients of the following regressionR t = a 0 + b l,t RRL t-1 + b h,t RRH t-1 + l t RL 1-12,t-1 + h t RH 1-12,t-1 + tWhere RRL equals one if stock i's previous month residual return, which is residual term obtained from the Fama and French three-factor regression scaled by the standard deviation of return over the estimated period, is in the bottom 20% and is zeros otherwise. RRH equals one if stock i's previous month residual is in the top 20% and is zeros otherwise. RL 1-12 ) and RH 1-12 indicate the bottom 20% and the top 20% stocks based on R 1-12 respectively. The results reported for the raw returns in the table are the time series average of these coefficients. Risk-adjusted return are the intercepts from the time-series regressions of these averages on the contemporaneous market factor (CAPM) and Fama-French three factors (FF3). The reported t-statistic (in parentheses) are Newest West (1987) corrected. The sample period is from 1964.07 to 2016.12.

		.3: Pairwise comparisons
		Raw returns	Risk-adjusted Returns
			CAPM	FF3
	RL 1-12	0.396	0.219	0.211
		(3.13)	(1.88)	(1.94)
	RH 1-12	-0.715	-0.777	-0.777
		(-5.61)	(-6.70)	(-7.23)
	RL 1-12 -RH 1-12	1.111	0.996	0.989
		(7.02)	(6.32)	(6.32)
	RRL	0.427	0.480	0.414
		(4.56)	(5.52)	(5.21)
	RRH	-0.383	-0.321	-0.349
		(-3.89)	(-3.58)	(-4.51)
	RRL -RRH	0.810	0.801	0.763
		(7.32)	(7.34)	(6.92)

Email adress: anh duy.NGUYEN@uca.fr ;CLeRMa -Université d'Auvergne-11 Boulevard Charles de Gaulle-63000 Clermont-Ferrand-France

In particular, they show that that the relation between momentum return and return comovement driven by style investing is significantly stronger in 'optimist' market state.

The analysis is similar to that of[START_REF] Hameed | industries and stock return reversal.pdf[END_REF]. However, they employ this for motivation of the intra-industry reversal. The main idea is that neutralizing the momentum effect will enhance the short-term reversal profit.

Note that none of paper's result is affected by whether we exclude these stocks.

http://mba.tuck.dartmouth.edu/pages/faculty/ken.french/datalibrary. html

To form the CAPM and Fama-French risk-adjusted profits, we first regress the timeseries of raw reversal profits on the correspondent factors and a constant in order to obtain the estimated factor loadings. Then, the risk-adjusted returns are measured as the reversal return net of what is attributable to exposure to the market factor and Fama and French three factors.

These results is reported in Appendix B

A E.U stock markets

In this Appendix, we verify the robustness of our findings by conducting the out-of-sample tests. In particular, we replicate these above results for European stock market. In particular, we examine: (i) whether the reversal strategy based on R 1-12 is significantly higher than that generated by conventional reversal strategy in European stock market; (ii) whether market states have significantly impact on European reversal strategy's profits. In Table 9, we presents summary statistic for the European countries included in our sample. Table 10 presents the results of the profit of reversal strategy formed following [START_REF] Fama | Risk, Return, and Equilibrium: Empirical Tests[END_REF] style for E.U stock markets.

Table 11, we examine our results about the relation between reversal profits and market states in EU stock market. reversals [START_REF] Hameed | industries and stock return reversal.pdf[END_REF] argue that implement the reversal strategy within industry will isolate the short-term reversal from an across-industry momentum (Moskowitz and Grinblatt (1999)). They find that compare to the conventional reversal, the intra-industry reversal is stronger in magnitude and robust to market micro-structure biases. We show that R 1-12 effect goes beyond the industry control. The R 1-12 based reversal strategy within-industry provides significantly higher return than the one generated by [START_REF] Hameed | industries and stock return reversal.pdf[END_REF] within-industry reversal strategy (see Table B.1) Recently, [START_REF] Blitz | Short-term residual reversal[END_REF] reports that reversal strategy based on residual return, which is obtained from [START_REF] Fama | Common risk factors in the returns on stocks and bonds[END_REF] three-factor regression, provides higher profit compare to the conventional strategy does. They argue that the conventional reversal strategy is negatively affected by the Fama and French (1993) three factors, making the profit lower. Therefore, forming the reversal strategy based on the return, which nets of Fama and French three-factor exposures, will improve the profit. In fact, the method proposed by [START_REF] Blitz | Short-term residual reversal[END_REF] is also related to the idea of neutralizing the momentum effect. In particular, we find that applying [START_REF] Blitz | Short-term residual reversal[END_REF]'s method for R 1-12 does not improve significantly the R 1-12 based reversal return (see Table B.2). Moreover, we find that R 1-12 strategy generates higher return than the residual return based strategy (Table B.3). An additional problem Note: Each month 't' we regress the cross-section of stock return on several explanatory variables. The Table reports the time-series average of these coefficients along with their Newey and West (1987) adjusted t-statistic in parentheses. The control variables are residual return (RR), which is the residual term from the Fama and French three-factor regression scaled by the standard deviation of return over the estimated period; lagged one-month return (LRet); the log of market capitalization (ME); book-to-market ratio (BM). R 1-12 is measured as 13 k=2 k 12 (R t-1 -E12 t-k ). The sample period is from 1964.07 to 2016.12.


Mario Garzón

Anne Spalanzani

Game theoretic decision making for autonomous vehicles' merge manoeuvre in high traffic scenarios

This paper presents a game theoretic decision making process for autonomous vehicles. Its goal is to provide a solution for a very challenging task: the merge manoeuvre in high traffic scenarios. Unlike previous approaches, the proposed solution does not rely on vehicle-to-vehicle communication or any specific coordination, moreover, it is capable of anticipating both the actions of other players and their reactions to the autonomous vehicle's movements.

The game used is an iterative, multi-player level-k model, which uses cognitive hierarchy reasoning for decision making and has been proved to correctly model human decisions in uncertain situations. This model uses reinforcement learning to obtain a near-optimal policy, and since it is an iterative model, it is possible to define a goal state so that the policy tries to reach it.

To test the decision making process, a kinematic simulation was implemented. The resulting policy was compared with a rule-based approach. The experiments show that the decision making system is capable of correctly performing the merge manoeuvre, by taking actions that require reactions of the other players to be successfully completed.

I. INTRODUCTION

The past few years have seen an increasing interest on the research and development of autonomous cars. As new developments are made, more complex tasks and situations are being tackled, therefore requiring either more complex or more creative solutions. Among the many challenges that have arise for autonomous cars, the interaction with human drivers is one of the most complex and interesting. This is due to the fact that involving human drivers require to model their behaviour and anticipate their intentions, which may not be clear and therefore can not be predicted with certitude.

This paper proposes a solution to a very challenging scenario, the on-ramp merging of an autonomous car into a road with high traffic flow, namely during a traffic jam. Furthermore, the scenario does not assume any shared control or intercommunication with neighbourhood vehicles, therefore, it requires a non-coordinated bi-directional interaction between the autonomous cars and human drivers. A real world situation, such as the one proposed in this scenario is shown in Figure 1.

In order to solve a situation as the one proposed here, it is necessary to solve many challenges, they can be broadly divided in four areas: situation awareness of the autonomous vehicle or ego-car, prediction of actions or movements by other vehicles, decision making of the ego-car and execution of the manoeuvre. The work presented on this paper focuses on two of those challenges, the anticipation of actions from other vehicles and the decision making of the ego-car. Those two task are strongly dependent on each other, because the decision making should consider both the actions of the other vehicles as well as their reaction to the movements of the ego-car. Moreover, the solution proposed solves these two tasks simultaneously by involving the estimated actions and reactions of other vehicles in the decision making model.

Commonly used decision making process are not suitable in this case. The reason for this is that most techniques relay on overly defensive strategies, usually assuming constant velocity or simplistic models for other vehicles and human drivers, applying a "stay out of their way" strategy. Although those approaches may work in simpler scenarios, they are not suitable in the presence of high traffic, because there it is impossible to merge without affecting the cars surrounding our vehicle [START_REF] Evestedt | Interaction aware trajectory planning for merge scenarios in congested traffic situations[END_REF].

The approach used in this paper is based on Game Theory, which is a tool that provides a good framework to model and solve interactions between multiple agents [START_REF] Kita | A merging-giveway interaction model of cars in a merging section: a game theoretic analysis[END_REF], therefore allowing the vehicle to make a decision that can be seen as a risky manoeuvre if seen by traditional techniques, and at the same time anticipating both the movements of other cars as well as their reaction to the movements of the ego-car.

There are many different techniques within game theory, some of them have already been applied to autonomous cars [START_REF] Meng | Dynamic decision making in lane change: Game theory with receding horizon[END_REF]. Furthermore, usually the game theoretical decision making seeks equilibrium by assuming mutual rationality and mutual consistency (i.e. every player will always make the rational decision and all the players' beliefs about other players are consistent and true). This however is not necessary true, and therefore new game models have been proposed.

The game theoretical model used in this paper, is based on the cognitive hierarchy model, proposed by Camerer et al. [START_REF] Camerer | A Cognitive Hierarchy Model of Games*[END_REF], which has been experimentally validated with cognition experiments [START_REF] Costa-Gomes | Cognition and behavior in two-person guessing games: An experimental study[END_REF]. Its main difference with most other game theory models, is that it assumes that some player's beliefs may be mistaken, and optimizes a policy based on that assumption, therefore yielding a higher level policy that is better than that of everyone else's. These games are also known as Level-k thinking, because starting from the assumption that every player will use a basic (level 0) policy, a player (level 1) will try to find a new optimal policy assuming every one else will use the previous one (level 0), and using the same logic, higher levels can be reached.

The main contribution of this paper, is the use of an iterative, multi-player level-k game model, in order to solve a very complex task for autonomous cars. This model, relies on the creation of an Iterative Semi Network-Form Game [START_REF] Lee | Counter-factual reinforcement learning: How to model decision-makers that anticipate the future[END_REF], which allows to obtain a near-optimal policy, taking into account the estimation of the possible actions of surrounding vehicles, as well as their reaction to the actions of the ego-car. Furthermore, the proposed technique not require any vehicle-to-vehicle communication or coordinated behaviour. Also, the observation space and the set of actions proposed, can be applied to many other scenarios and other complex situations. In addition to this, the iterative model proposed, makes it possible to set a goal state and therefore model long term intentions of the ego-car and the additional vehicles.

The remainder of this paper is structured as follows Section II presents a brief summary of related works and states the main differences of the work presented here. Then, Section III describes the proposed methodology and their main components. Section IV details the implementation procedure and Section V describes the scenarios and experiments used to test the capabilities of the system and, finally, Section VII presents the concluding remarks.

II. RELATED WORK

Several different techniques to perform the merging manoeuvre have been reported in the past few years. One of the first works using game theory for this tasks was proposed by Kita et at. [START_REF] Kita | A merging-giveway interaction model of cars in a merging section: a game theoretic analysis[END_REF], it models a two-person game where players can either give-way or not and merge or pass. Also it obtains the reward function by finding correlation with real world observations. However, its drawback, apart from only considering two players, is that it assumes perfect information and knowledge of the other player.

Some of the works found on the literature are based on a shared control of all the vehicles in the scenario. Merging using on slot-based driving [START_REF] Marinescu | On-ramp traffic merging using cooperative intelligent vehicles: A slot-based approach[END_REF], or relying on vehicle-tovehicle communication and strong cooperation, so as to yield space for the merge [START_REF] Milanes | Automated on-ramp merging system for congested traffic situations[END_REF] have been proposed. Similarly, Brechtel et al. [START_REF] Brechtel | Probabilistic decisionmaking under uncertainty for autonomous driving using continuous pomdps[END_REF] uses continuous POMDP in order to generate safe, efficient, and goal-directed driving behaviours for a two-vehicles interaction, although only two possible actions (change lane or stay) were considered. A different work includes non-cooperative road users [START_REF] Düring | Adaptive cooperative maneuver planning algorithm for conflict resolution in diverse traffic situations[END_REF], however, those additional road users are assumed to follow a single trajectory that is not modified by the ego-car's actions.

Considering the possible reactions of other vehicles (uncoordinated approaches), a multi-policy decision-making process was proposed by Cunningham et at. [START_REF] Cunningham | Mpdm: Multipolicy decision-making in dynamic, uncertain environments for autonomous driving[END_REF]. It uses driving models to create a set of policies that evaluate the consequences of the ego-car actions. The effects that the actions of an autonomous car can take on human drivers are studied in the work of Sadigh et al. [START_REF] Sadigh | Planning for autonomous cars that leverage effects on human actions[END_REF], this uses Inverse Reinforcement Learning to determine the cost functions for those possible actions. Another work uses a Receding Horizon Control and game theory to maximize mutual payoff of different players [START_REF] Meng | Dynamic decision making in lane change: Game theory with receding horizon[END_REF], However, all those works only consider one additional vehicle or player and two possible actions, rendering it invalid for high traffic scenarios.

Another approach generates many possible velocity profiles, and then it estimates an acceptable breaking or discomfort in other vehicles in reaction to each possible trajectory [START_REF] Evestedt | Interaction aware trajectory planning for merge scenarios in congested traffic situations[END_REF]. That estimation is based on the Intelligent Driver Model. A different approach, also based on a set of traffic rules and regulations, requires complex parameters and movements equations [START_REF] Rasekhipour | A potential field-based model predictive path-planning controller for autonomous road vehicles[END_REF]. The main drawback of these approaches is that, since they are model based, they will produce mostly over safe manoeuvres, that will cause longer waits in order wait for a safe space for the merge manoeuvre.

Finally, the work proposed by Li et al. [START_REF] Li | Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems[END_REF], which similarly to the work presented here, uses a level-k game theory approach to model and simulate traffic and it considers many players interacting and also many possible actions. However, that work is mainly focused on calibrating, testing and comparing decision and control systems for autonomous vehicles.

The main differences of the our proposal with respect to the related works is as follows. Firstly, it can consider the reactions of multiple players to the ego-car actions, furthermore, it differs from the work of Li et al. [START_REF] Li | Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems[END_REF] in that it can have different models or policies for each additional player, and it can also model end-of-roads or other variation of the road. Another important difference, is that it uses a time-iterative approach, that allows to define a goal state, and therefore it can be used to model many different scenarios, such as vehicles entering or leaving a highway. In addition to this, the assumptions of other players actions do not need to be perfect, and therefore it is possible to perform interactions with human drivers or other unconnected autonomous vehicles.

III. MERGE SCENARIO MODEL

The objective of the work presented here is to develop a decision making approach capable of solving a very complex task for autonomous vehicles, the on-ramp merge during a traffic jam. In order to solve this, it was necessary to define a world model, that allows to simulate the manoeuvre and the movements of all the vehicles. A model of the scenario, with its main components, is presented on Figure 2, which corresponds to a two lane road, that reduces to a single lane after a given distance.

A. World Model

In order to perform the simulated merge, a model of the world, which defines the movements of the vehicles, • v len Length of the vehicle.

• pos x, pos y Position (forward and lateral).

• accel x Linear acceleration.

• vel x, y Linear, (forward and lateral) velocity. Having this, the equations of motion are defined according to Equation 1pos x(t + 1) = pos x(t) + vel x(t) • t step vel x(t + 1) = vel x(t) + accel x(t) • t step pos y(t + 1) = pos y(t) + vel y(t) • t step (1) Some additional restrictions are also included on the world model, namely: The forward velocity is limited by a maximum speed parameter. The lateral position pos y is only controlled by the change lane actions, thus, the lateral position corresponds to the lane the vehicle is on, and the lateral velocity vel y will be either -1, 0 or 1 depending on the action taken. All the boundaries of the road, including the end of the merge ramp are ensured, as well as maximum acceleration and deceleration values. Furthermore, although the scope of this work does not include the use of a complex traffic simulator, the models of the world and vehicles, as well as the information available, have been selected so as to facilitate its future integration with a high traffic simulator previously developed [START_REF] Garzón | An hybrid simulation tool for autonomous cars in very high traffic scenarios[END_REF].

B. Observation Space

As in any real world driving scenario, the vehicles involved are not capable of having complete information about every other vehicle. Moreover, when involving many other vehicles in the decision making, it will be very complex to process such amount of information. The solution proposed here relies on a different observation space, which, as any human driver will do, only considers the surroundings of the vehicle. Moreover, by having a broad set of discrete states, the observations can handle uncertainties that may be found in the measurements. The objective of these observations is to be able to provide a sufficient amount of information, without requiring complete or exact measurements. The definition of the observation space is shown in Figure 3. As can be seen there, each vehicle handles five different measurements, corresponding to the status of the front F , front-left F L, front-right F R, rear-left RL and rearright RR. Observations of the back of the vehicle are not considered, because, as stated by Li et al. [START_REF] Li | Game theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems[END_REF], they don't provide relevant enough information for the decision making process and including the will highly increment the size of the observation space.

There are four possible states for each of those measurements, they can be: forbidden, there is no road available; free, if there is no vehicle and the road is clear; vehicle static, if a vehicle is stopped or moving slower that the ego-car; and finally vehicle moving, meaning that a vehicle is present and moving at the same speed or faster than the ego-car.

C. Possible Actions

A set of five possible actions has been defined for the vehicles. As with the observation space, they are designed for the merge scenario, but they can also be used in many other situations. The actions are described next:

• left: Change lane to the left (if available).

• right: Change lane to the right (if available).

• remain: Maintain the same speed, direction and lane.

• accel: Increase speed at given rate. (up to max. velocity) • brake: Decrease speed at given rate. (until velocity is zero)

D. Reward function

The final item in the formulation of the problem is the definition of a reward function, which defines the goals and/or behaviour of the vehicles. Since a time-iterative approach is used, it was possible to use a reward function where one or multiple states are defined as the goal, so as to obtain a policy that drives the ego-car towards this state. The reward function used is defined as per Equation 2.

Reward =          pos x, if pos x > ramp len
and pos y = goal lane -350, if constraint violation 0, otherwise

For the merge scenario, the main goal will be to safely pass the bottleneck and get incorporated in the main lane of the road, therefore, the goal state will be any position in the main (left) lane after the bottleneck. To ensure this goal, only three different values are used for the reward: If there is a constraint violation, such as a collision with other vehicles, or a movement towards a forbidden state (out of the road), a very low reward value is assigned. If the car is able to get to the goal lane, and pass the bottleneck, then its reward will be positive and it will increase as it moves forward so as to encourage faster movements. Finally, if the car not merged, but is still on a valid state, a reward of zero is assigned.

IV. PROPOSED SOLUTION

The solution of the decision making problem presented here is based on the use of an Iterative Semi Network-Form Game. This type of game, which was first introduced by Lee et al. [START_REF] Lee | Counter-factual reinforcement learning: How to model decision-makers that anticipate the future[END_REF], combines Bayesian networks and Game Theory in order to generate a model where multiple human and autonomous components can interact. These form of games have been used to model cyber-physical security systems [START_REF] Backhaus | Cyber-physical security: A game theory model of humans interacting over control systems[END_REF] and to predict pilot behaviours [START_REF] Yildiz | Predicting pilot behavior in medium-scale scenarios using game theory and reinforcement learning[END_REF].

An iterative semi network-form game uses random variables and a probabilistic framework to represent the different components of the system. There are two types of nodes in this network, firstly the chance or Estimated Action nodes, which use a pre-defined fixed conditional probability distribution. The second type are the Decision Nodes, they represent the human players and their behaviour is learned or optimized. Each decision node requires a reward function (See Equation 2) and then game theory approach is applied in order to optimize their policy. An overview of the proposed iterative semi network-form game is presented in Figure 4.

A multi-player game, with up to 6 players, is proposed, however additional vehicles, following simple rules can be added to the simulation. For each of the players, a set of Estimated Action nodes is used to estimate the likely actions that the vehicles in the surroundings of the player may take at any given time. Each player then uses those possible actions to obtain an estimation of the new state, which will represent its near future status. Having this, it is possible for each player to tune its Decision Node's policy accordingly. The process to optimize that decision making policy is briefly described in Section IV-A.

A. Level-K Reinforcement Learning

In order to determine which is the best action to take in each case, a hierarchical or level-k reasoning process was used. The objective is to make meaningful predictions of the outcomes of the games assuming that the other players will use a basic (level-0) strategy, and then optimize the policy (i.e. conditional probability distribution) using that estimation, so as to obtain a higher level policy and so on, as aforementioned, this type of level-k solutions have been proved to produce results coherent with human behaviour.

Finding the optimal equilibrium for multiple players, in an iterative game using methods such as Nash equilibrium or quantal response equilibrium, may not be feasible because those methods become extremely complicated when a high number of player and possible actions is involved. Therefore, it is necessary to use a different approach. In this case, the solution is based on reinforcement learning. The system does not consider every possible combination of actions, instead it assumes that the players define a single policy for all the network, and then execute this policy over all iterations of the network, thus making the computational complexity independent of the number of time-steps. Also, by defining a single policy, it is not necessary to sample for different actions of the players, but rather it is possible to sample for different policies.

Once a policy has been defined, its utility is obtained by adding the rewards of every time step. Using this utility value, an optimization problem can be defined by trying to find the policy that leads to the highest expected infinite sum of discounted rewards. This optimization problem is solved using a Monte Carlo reinforcement learning approach, which searches for a near-optimal policy and can be seeded with lower level policies to improve its results.

V. EXPERIMENTS DESCRIPTION

This section describes the test performed in order to validate the decision making process and to evaluate the performance of the system itself. Firstly, the scenario, its initialization and the overall procedure of the simulation is explained. Then, the basic behaviour, used by level 0 players is explained, and finally a comparison of the performance of different level players is carried out.

A. Scenario set-up

The scenario used for the experiments is based on the world model presented on Figure 2. It is composed of a twolane road, each lane has a width of 3.2 m. The left lane is continuous and unlimited in distance, whereas the right lane ends after a distance of 25 m. The vehicles are assumed to drive on the centre of the lane, and they only move laterally to perform a lane change manoeuvre.

Since the scenario modelled is a traffic jam, the maximum velocity of the cars is limited to 5 m s -1 . The acceleration of the vehicles is 2.5 m s -2 and the breaking action decelerates the vehicle at 5 m s -2 .

The initialization of the vehicles is as follows: the ego-car and one more vehicle are placed on the merging ramp and the rest of the vehicles on the main lane. The vehicles located on the same lane are separated by a distance of 2.5 m, and all vehicles are assigned an initial speed of 2.5 m s -1 .

B. Simulation work-flow

Once the vehicles are assigned their initial position and speed, the main loop starts. The simulation is executed for 10 s, and as aforementioned, the movements have discrete time steps of 1 s. At each time step, each vehicle obtains an estimation of the actions of the vehicles in its surroundings. Then, using those estimated actions, their observations are updated, and the best action is selected accordingly. Each vehicle performs only one action per time step. Finally all selected actions are applied simultaneously and the state of each car is updated according to Equation 1. After the new state has been obtained, the reward function (See Section III-D) is computed, and the process continues with a new timestep.

C. Level 0 Behaviour

A set of basic rules was defined as the Level 0 policy. These rules generate a behaviour that can be used as reference for training as well as a point of comparison with the higher level policies found by the reinforcement learning algorithm.

The main idea of that behaviour is that all vehicles will continue on their lane until it is not possible to move forward (i.e. when they reach the end of the ramp), and then they will wait to have a clear space, change lane and continue their movement. This Level 0 policy is achieved by following the next rules: Using this basic policy, all vehicles are capable of merging if enough time is given. For the case of the ego-vehicle, in the 10 s of the simulation, it will be able to place immediately after the bottleneck. Moreover, additional vehicles, not participating on the game, are controlled using this policy.

VI. RESULTS AND EVALUATION

In order to have a qualitative evaluation of the decision making process, an example of the trajectories obtained using the proposed decision making system, as well as the reference basic behaviour, are presented in Figure 5, the main difference, as shown in the image, is that using the rule based approach the ego-car will always wait for a safe space in order to merge, in this case at time t = 9 s, whereas using the game theoretical approach, the ego-car can anticipate that the orange vehicle will break if it merges, and as shown, it correctly manages to do so at a much earlier time, at t = 3 s.

A. Resulting policy evaluation

Since obtaining a Nash equilibria for the resulting policy (i.e. probability distribution), is not possible in this case. It is necessary to use a different technique to provide a formal evaluation of the quality of the obtained policies.

This evaluation is obtained by using a Monte Carlo Importance Sampling estimation of the Predictive Game Theory (PGT) model [START_REF] Wolpert | Distribution-valued solution concepts[END_REF]. This sampling provides a quantification of the level of rationality inherent in a player's behaviour [START_REF] Wolpert | A predictive theory of games[END_REF]. It uses as reference a welfare function, which in this case is the same as the reward function defined in Section III-D, where negative values represent collisions or undesired states, whereas, positive values show that the car was able to pass the bottleneck, moreover, larger positive rewards mean that the given car reached a further position, and since the simulation time is fixed, it also means that it was capable of crossing faster. The resulting values of the importance sampling are shown in Figure 6. For both, level 1 and level 2 players, the expected welfare can vary highly, this is because using the game theoretical approach, there may be risky situations, and thus it may lead to lower rewards. As can be seen in Fig. 6a, for the Level 1 player, most of the expected rewards are negative, meaning that there is a considerable probability of collisions. However, when training a higher level policy (Fig 6b), it can be seen that, although there may be still some collisions, most of the rewards is positive, and is overall higher than the reward obtained by the level 1 policy, thus proving the efficiency of the decision making to find a solution to this scenario.

VII. CONCLUSIONS

This paper presents a possible solution for a very complex task of autonomous vehicles: the merge manoeuvre in very high traffic scenarios. The solution does not require any specific cooperation or vehicle-to-vehicle communication, and it is capable of anticipate the reactions of other vehicles to specific actions of the ego-car.

The proposal uses an iterative, multi-player level-k game, and finds a near-optimal solution using Monte Carlo reinforcement learning techniques. Moreover, the set of observation, possible actions and reward proposed, can be applied to different situations, and it can yield goal-oriented behaviours.

A kinematic simulation were used in order to test the decision making algorithm and a pre-defined rule-based policy was used for comparison. The results show the capacity of the proposed methodology to solve the task by taking actions that may be consider unsafe unless they take into account the reactions of the other players.

Fig. 1 :

 1 Fig. 1: Example of the task and scenario: An autonomous vehicle (in the blue circle), needs to merge with the traffic.

Fig. 2 :

 2 Fig. 2: Modelling of the world. The ego-car is shown in blue, and the variables considered in the world model are marked.

Fig. 3 :

 3 Fig. 3: Observation space. The possible states for each measurement are: forbidden, free, vehicle static, vehicle moving.

Fig. 4 :

 4 Fig. 4: One time-step and two player example of the proposed semi network-form game. Each car generates an estimation of the movements of the other players and makes the decision accordingly.

  if vel x > 0 and F = forbidden if vel x > 0 and F = vehicle static if any collision accel = if vel x = 0 and F = free lef t = if vel x = 0, RL = free and F = forbidden remain = elsewhere

Fig. 5 :

 5 Fig.5: Sequence of movements using rule-based and game theoretic approaches. The ego-car is shown in blue and the instant of the merge manoeuvre is highlighted.

Fig. 6 :

 6 Fig. 6: Level 1 vs level 2 average welfare, from Importance Sampling estimation.

The authors are with Univ. Grenoble Alpes, Inria, Grenoble INP, 38000 Grenoble, Francemario.garzon-oviedo@inria.fr, anne.spalanzani@inria.fr

ACKNOWLEDGMENT

This work was funded under project CAMPUS (Connected Automated Mobilty Platform for Urban Sustainability) sponsored by Programme d'Investissements d'Avenir (PIA) of french Agence de l'Environnement et de la Maîtrise de l' Énergie (ADEME).