Thorsten Martin
email: thorsten.martin@hec.edu.

Florian Sonnenburg
email: sonnenburg@wiso.uni-koeln.de.

Laurent Calvet

Jean Eduoard Colliard

Thierry Foucault

Johan Hombert

Stefan Jaspersen

Alexander Kempf

Hugues Langlois

Peter Limbach

Stefano Lovo

Alina Rosu

Clemens Otto

Jacques Olivier

Alexei Ovtchinnikov

Daniel Schmidt

Paul Tetlock

David Thesmar

Managerial Ownership Changes and Mutual Fund Performance

Keywords: JEL classification: G11, G14, G20, G23 Mutual Funds, Fund Manager Ownership Changes, Fund Performance Predictability, Incentive Alignment, Superior Information

We study the dynamics of fund manager ownership for a sample of U.S. equity mutual funds from 2005 to 2011. We find that ownership changes positively predict changes in future risk-adjusted fund performance. A one-standarddeviation increase in ownership predicts a 1.6 percent increase in alpha in the following year. Fund managers who are required to increase their ownership by fund family policy show the strongest increase in alpha. They do so by increasing their trading activity in line with the view that higher ownership aligns interests of managers with those of shareholders and induces higher effort.

Introduction

Since March 2005, mutual funds have to report fund managers ownership within the Statement of Additional Information (SAI) using broad ownership ranges. By using one year of ownership data, [START_REF] Khorana | Portfolio Manager Ownership and Fund Performance[END_REF] and [START_REF] Evans | Portfolio Manager Ownership and Mutual Fund Performance[END_REF] show that the level of ownership predicts future risk-adjusted performance. Since then, more and more fund investors are paying attention to manager ownership (see, e.g., [START_REF] Ma | Portfolio Manager Ownership and Mutual Fund Risk Taking[END_REF]) and some mutual fund families adopted policies which require their managers to hold ownership in the funds they manage. 1 This raises the question if the increased attention to managerial ownership and the implementation of ownership requirements by fund families is warranted.

Does managerial ownership provide valuable information about future fund performance or is the observed cross-sectional correlation driven by unobserved fund characteristics? Does ownership align incentives and can ownership requirements therefore be used to increase performance? Or do managers have superior information about future fund performance and choose to invest in funds which they know will perform better in the future? Given the nature of their cross-sectional data, these early studies are unable to answer these questions.

We fill this gap in the literature by examining the relationship between ownership changes and changes in future risk-adjusted fund performance using a hand-collected panel data set on mutual fund manager ownership. Examining ownership changes has two advantages in our setting. First, we are able to eliminate any heterogeneity bias stemming from time-invariant unobserved fund characteristics. For instance, funds differ in the degree of managerial discretion in making investment decisions. This managerial discretion may lead to higher fund returns on average, but exposes the fund investor to greater risk of moral 1 in The Wall Street Journal MarketWatch (26/07/2006) and Bloomberg BusinessWeek (14/01/2010). Both articles report that some mutual fund companies have started requiring their managers to invest in the funds they manage including Franklin Templeton Investments, Janus Capital Group, and T. Rowe Price. [START_REF] Laise | Another Way to Assess a Mutual Fund: Evidence Mounts That Performance Improves When Managers Invest in Their Own Portfolios[END_REF] [START_REF] Bloomberg | Fund Managers: Betting Their Own Money[END_REF] hazard (see, e.g., [START_REF] Chen | Directors' Ownership in the U.S. Mutual Fund Industry[END_REF]) and thus increases the optimal level of ownership. 2 Therefore, unobservable differences in managerial discretion may bias the ownership coefficient upwards. If managerial discretion is relatively stable within funds over time, we are able to eliminate this bias by using a first-difference approach. Moreover, we control for changes in a host of fund, board and family characteristics as well as predictors of fund performance which have been recently proposed by the literature.

Second, the cross-sectional studies are unable to examine whether the positive relationship between ownership levels and performance reflects fund managers superior information about future fund performance or better alignment of fund managers and shareholders interests. We use changes in ownership mandated by family policy to disentangle the superior information and the incentive alignment hypotheses. The idea is that family mandated changes unlikely reflect a fund manager s information about future fund performance. If the positive relationship between manager ownership and fund performance reflects fund manager s superior information about future fund performance, we do not expect that ownership changes which are mandated by the fund family increase fund performance. If on the other hand manager ownership aligns the fund manager s interests with those of shareholders, we expect ownership changes to have a causal effect on performance even if the change is required by the fund family.

Using a hand-collected dataset on managerial ownership for a sample of single managed U.S. domestic equity mutual funds over the period from 2005 to 2011, we find that ownership changes are positively related to changes in future risk-adjusted fund performance no matter whether we measure performance as [START_REF] Fama | Common Risk Factors in the Returns on Stocks and Bonds[END_REF] three-factor alpha, 2 For example, [START_REF] Chen | A First Look at Mutual Funds That Use Short Sales[END_REF] show that short-selling mutual funds outperform benchmarks by 1.5% per year. Consider a fund manager running two identical funds, but one is short-selling restricted and the other is not. From an optimal contracting perspective the manager should own more in the unrestricted fund as his ability to take actions against the interests of shareholders is greater. If this is the case, this creates a positive correlation between ownership and performance in the cross-section if shortselling restrictions are not controlled for. [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four-factor alpha, or Pástor and Stambaugh (2003) five-factor alpha. The relation between increases in ownership and increases in future risk-adjusted fund performance is also economically significant: a one-standard-deviation increase in ownership leads to an increase in alpha between 1.1 percentage points for the [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four-factor alpha and 1.6 percentage points for the Pástor and Stambaugh (2003) five-factor alpha. This result stands several robustness tests regarding the construction of our ownership measure.

Next, we control for other predictors of fund performance. Recent studies show that the level of ownership [START_REF] Khorana | Portfolio Manager Ownership and Fund Performance[END_REF] and [START_REF] Evans | Portfolio Manager Ownership and Mutual Fund Performance[END_REF]), the industry concentration [START_REF] Kacperczyk | On the Industry Concentration of Actively Managed Equity Mutual Funds[END_REF]), return gap [START_REF] Kacperczyk | Unobserved Actions of Mutual Funds[END_REF]), active share (Cremers and[START_REF] Cremers | How Active Is Your Fund Manager? A New Measure That Predicts Performance[END_REF][START_REF] Petajisto | Active Share and Mutual Fund Performance[END_REF]) and a fund s R 2 with respect to its benchmark [START_REF] Amihud | Mutual Fund's R2 as Predictor of Performance[END_REF]) predict future fund performance. We find that ownership changes are highly significant predictors of changes in future risk-adjusted fund performance even after controlling for the lagged level of ownership as well as changes in industry concentration, active share, return gap and R 2 . A one-standarddeviation increase in ownership predicts an increase in risk-adjusted performance of up to 1.6 percentage points.

We then analyze whether the documented positive relation between ownership changes and changes in risk-adjusted performance is due to incentive alignment or due to superior information. To disentangle these two hypotheses, we use the adoption of fund family policies requiring managers to hold some ownership in all funds they manage. We proxy for such a policy by using the ownership information we observe. If in a given year and fund family at least one fund has zero ownership, we define that the fund family has no strict ownership requirement in place. Contrary, if all funds within a given fund family have ownership greater than zero in a given year, we define that the family has a strict ownership requirement in place.

We find that future risk-adjusted performance increases even stronger with ownership when managers increase their ownership simultaneously to the adoption of a new family policy which requires managers to hold some ownership in their funds. A one-standarddeviation ownership increase simultaneous to the adoption of an ownership requirement by the family increases alpha between 0.4 and 0.5 percentage points more than other ownership increases. As these changes are most likely not driven by fund managers superior information about future fund performance, these results support the view that ownership aligns interests of fund manager s and shareholders and causally affects fund performance.

One possibility is that mandatory ownership increases induce the manager to exert more effort to seek out profitable investment opportunities. In line with this view, we find that mandatory ownership changes strongly predict future changes in trading activity. Managers who increase ownership simultaneously to the adoption of a family wide ownership requirement increase their active share, turnover, unobserved actions and their equity holdings and decrease their cash holdings.

The paper contributes to three main strands of the literature. First, our paper relates to the growing literature on governance is significantly improved if more independent directors are on the board or if independent directors have a higher ownership in the fund and thus a higher motivation to effectively monitor the fund (see, e.g., [START_REF] Tufano | Board Structure and Fee-Setting in the U.S. Mutual Fund Industry[END_REF], [START_REF] Ferris | Do Independent Directors and Chairmen Really Matter? The Role of Boards of Directors in Mutual Fund Governance[END_REF], [START_REF] Khorana | Board Structure, Mergers and Shareholder Wealth: A Study of the Mutual Fund Industry[END_REF]Wedge (2007), Cremers, et al. (2009), and [START_REF] Ding | Mutual Fund Performance and Governance Structure: The Role of Portfolio Managers and Boards of Directors[END_REF].

More related to our study, [START_REF] Khorana | Portfolio Manager Ownership and Fund Performance[END_REF] and [START_REF] Evans | Portfolio Manager Ownership and Mutual Fund Performance[END_REF] show in their cross-sectional analysis that managers with higher levels of ownership have better future fund performance. Our study contributes to this literature in two ways. To begin with, we are the first who use panel data on managerial ownership. Therefore we are able to rule out that any unobservable fund, manager or family fixed effects lead to a spurious correlation between ownership and performance. Our second contribution to this literature is that we are able to disentangle the superior information and incentive alignment hypotheses and we find support for the latter.

Second, our paper is related to a growing body of literature that analyzes managerial incentives in the mutual fund industry. Several studies look at the relationship between fund managers incentives and their risk-taking behavior arising from the convex flowperformance relation (e.g., [START_REF] Brown | Of Tournaments and Temptations: An Analysis of Managerial Incentives in the Mutual Fund Industry[END_REF], [START_REF] Chevalier | Risk Taking by Mutual Funds as a Response to Incentives[END_REF], [START_REF] Koski | How Are Derivatives Used? Evidence from the Mutual Fund Industry[END_REF], [START_REF] Kempf | Tournaments in Mutual-Fund Families[END_REF] , [START_REF] Kempf | Employment Risk, Compensation Incentives, and Managerial Risk Taking: Evidence from the Mutual Fund Industry[END_REF], and [START_REF] Schwarz | Mutual Fund Tournaments: The Sorting Bias and New Evidence[END_REF]). Another strand of this literature analyzes the link between advisory fee contracts and performance (e.g., [START_REF] Elton | Incentive Fees and Mutual Funds[END_REF], [START_REF] Dass | Mutual Funds and Bubbles: The Surprising Role of Contractual Incentives[END_REF], and [START_REF] Massa | Incentives and Mutual Fund Performance: Higher Performance or Just Higher Risk Taking?[END_REF]). In a recent paper, [START_REF] Ma | Portfolio Manager Compensation in the U[END_REF] show that fund managers with explicit performance-based incentives perform better. We complement this literature by showing that fund manager ownership can act as an explicit incentive tool to align manager and shareholders Third, our paper relates to the vast literature of managerial ownership in corporations. We find that ownership changes which are mandated by the fund family increase fund performance and therefore contribute to the controversy if manager ownership can be used to change firm value (see, e.g. [START_REF] Himmelberg | Understanding the Determinants of Managerial Ownership and the Link Between Ownership and Performance[END_REF], [START_REF] Zhou | Understanding the Determinants of Managerial Ownership and the Link between Ownership and Performance: Comment[END_REF] , and [START_REF] Fahlenbrach | Managerial Ownership Dynamics and Firm Value[END_REF]).

The rest of the paper proceeds as follows. Section 2 describes the data and compares our sample to the CRSP universe of mutual funds. In Section 3, we analyze the relation between ownership changes and changes in future risk-adjusted performance. In section 4, we examine if the results are robust to controlling for changes in other predictors of fund performance. In section 5 we explore how ownership changes due to fund family policy rather than personal portfolio decisions affect performance and Section 6 concludes. From the CRSP Survivor-Bias Free Mutual Fund Database we gather information on mutual funds monthly returns, total net assets, and other fund characteristics. We focus on single actively managed, domestic equity funds with no manager replacements during our sample period and exclude bond funds and international funds as well as index funds. We use investment objective. We aggregate the Lipper segments into seven broad categories: Aggressive Growth, Growth and Income, Income, Growth, Sector Funds, Utility Funds, and Mid-Cap Funds. Many funds offer multiple share classes which are listed as separate entries in the CRSP database. As these share classes are backed up by the same portfolio, we aggregate all share classes at the fund level to avoid multiple counting.

We match the CRSP funds to the Thomson Financial Mutual Fund Holdings Database using MFLINKS tables. Our last data source is the mutual funds SAI (in SEC filings 485APOS and 485BPOS), which are Part B of the mutual fund s prospectus. The data from the SEC filings 485APOS and 485BPOS can be downloaded in text files from SEC EDGAR. that the fund name often differs from the filer name under which a mutual fund discloses its filings with the SEC or that the filings 485APOS and 485BPOS may contain SAI from multiple funds. The SAI reports detailed information on each portfolio manager s ownership in the fund. The ownership is reported in seven ranges: None; $1 $10,000; $10,001 $50,000; $50,001 $100,000, $100,001 $500,000, $500,001 $1,000,000, or over $1,000,000. We convert these ownership ranges into dollar amounts using the bottom of each range.

We build two ownership measures: (1) We construct a yearly percentage ownership measure by dividing the converted dollar amount by a fund's year end total net assets (TNA) as suggested by the existing literature studying the impact of ownership on firm value going back to [START_REF] Jensen | Theory of the Firm: Managerial Behavior, Agency Costs and Ownership Structure[END_REF]. ( 2) We define ownership changes as the difference between fund managers current and lagged percentage ownership. To control for commonly used board characteristics, we further collect the following board information from the SEC files: Name of director, and whether the director is interested or independent as defined in the Investment Company Act (ICA). Our data cover 2,196 fund-year observations over the period 2005-2011.

Table 1 compares the summary statistics of our sample to the CRSP mutual fund universe with respect to the (Family TNA) flows, the turnover ratio of the fund, and the Pástor and Stambaugh (2003) five-factor alpha. Table 1 also provides summary statistics on the board and ownership measures for our sample of funds, namely the directors on the board, the level of ownership and the change in ownership.

Please insert TABLE 1 approximately here

The sample comparison shows that our sample funds are larger and belong to larger families. They also have slightly higher turnover and attract more flows. These differences are likely due to our sample selection criteria as we exclude team-managed funds as well as funds with manager changes to prevent that group dynamics or manager replacements drive the results. Further, as we are interested in manager ownership changes, we exclude funds without at least two consecutive years in the sample. The average fund in our sample has managerial ownership of 0.52 percent of the fund's TNA and changes it on average by 0.05 percent per year.

3 Ownership changes and changes in future fund performance

In this section we analyze the relation between ownership changes and changes in future risk-adjusted performance (Section 3.1). We check the robustness of our results in Section 3.2.

Main results

To examine the relation between ownership changes and changes in future riskadjusted performance, we use three different performance measures: (1) Fama and French We conduct first-difference regressions and use the change in the annualized performance measures from t-1 to t as dependent variable Performance in these regressions:

i,t i,t-1 1 i,t-1 2 i,t-1 3 i,t-1 1 i,t-1 2 + dSize ) + + BoardSize + IndepDire i,t-1 i,t ctors + (1)
Our main independent variable is the change in managerial ownership form year t-2 to t-1 Ownership . We add further variables to control for changes in fund, fund family, and governance characteristics. All of these changes are also measured from t-2 to t-1. At the fund level, we add changes in FundSize , and turnover ratio Turnover as control variables to the regressions. At the fund family level, we control for changes in logarithm of the size

FamilySize . The governance controls include board size

BoardSize as well as changes in the fraction of independent directors on the board IndepDirectors . To control for any unobservable time or segment effects that could equally affect all funds in a given year or a particular market segment, respectively, we also include time and segment fixed effects in all regressions. We cluster the standard errors at the fund level. Table 2 presents the results.

Please insert TABLE 2 approximately here

The results clearly show that ownership changes are positively related to changes in future risk-adjusted performance. The coefficient of ownership changes is positive and significant at the 1%-level in all specifications. The effect is not only statistically but also economically significant: For example, when Pástor and Stambaugh (2003) five-factor alpha is used as performance measure, an increase in ownership by one-standard-deviation (0.02935) predicts a 1.6 percent higher alpha (= 0.02935 × 0.5311), after taking all control variables into account .The other control variables have no notable consistent impact on future risk-adjusted performance.

Overall, the results from this section provide evidence that the observed crosssectional correlation between ownership and future fund performance is not stemming from a heterogeneity bias due to unobserved time-invariant fund characteristics.

Robustness

In this section we conduct additional tests to check that the positive effect of ownership changes on changes in future risk-adjusted performance is robust. The results from these tests are reported in Table 3.

Please insert TABLE 3 approximately here

Thus far, we have converted the bottom of each reported dollar range into dollar amounts. We now convert the reported dollar ranges into dollar amounts by assuming that the midpoint of the reported interval is always invested, except for ownership levels above $1 million, where we employ the bottom of the range. The results shown in Panel A remain similar: Ownership changes are positively associated with changes in future risk-adjusted performance.

In Panel B, we conduct a test to see whether our documented effect is driven by changes in the denominator of our ownership change measure. We replace our ownership change measure by a placebo ownership change measure. The nominator of this placebo ownership change measure takes on the mean dollar ownership in the sample for all funds and years whereas the denominator remains the fund size. Thus, all variation in this ownership change measure stems from variation in the denominator. We employ this measure in our first-difference regressions and do not find a significant effect on changes in future riskadjusted performance. This implies that the positive relation between ownership changes and changes in future risk-adjusted performance is not simply driven by changes in the denominator of the ownership change measure.

In Panel C, we additionally control for changes in fund flows. Although we control for changes in fund size, one might argue that the denominator of our ownership change measure is driven by changes in fund flows. Panel C shows that our main results do not change when controlling for changes in fund flows: Ownership changes are positively related to changes in risk-adjusted performance even after controlling for fund flows.

Finally, we split our sample into small and large funds using the median fund size in the sample as cut-off. As we are using changes in percentage ownership as our main independent variable, the results could be driven by small funds as a given dollar ownership change leads to a larger percentage ownership change for small funds. The results from Panel D show that the observed effect of ownership on performance is not driven by small funds.

Ownership changes predict future changes in performance for both small and large funds. The economic magnitude of the effect is even bigger for large funds. Given that the standard deviation of ownership changes for large funds (small funds) is 0.00053 (0.03913), a onestandard-deviation increase in percentage ownership of a large fund leads to an increase in risk-adjusted performance up to 2.0 percentage points (= 0.00053 × 37.156), compared to an increase in risk-adjusted performance up to 1.5 percentage points (= 0.03913 × 0.3867) for small funds.

Taken together, the findings in this section show that the baseline result of a positive impact of ownership changes on changes in risk-adjusted performance is robust to (1) using the midrange of the reported ownership range, (2) is not driven by changes in the denominator of the ownership measure, (3) is not driven by changes in fund flows, and ( 4) is not solely driven by small funds.

4 Ownership changes and other predictors of future fund performance

Having identified a robust measure to predict changes in future risk-adjusted performance, we now examine whether our measure survives after controlling for other existing measures to predict future performance.

In this context, [START_REF] Khorana | Portfolio Manager Ownership and Fund Performance[END_REF] and [START_REF] Evans | Portfolio Manager Ownership and Mutual Fund Performance[END_REF], both provide evidence that the level of ownership is positively associated with superior future performance.

Besides that, a growing body of literature uses holdings data of mutual funds to create performance predictability measures. [START_REF] Kacperczyk | On the Industry Concentration of Actively Managed Equity Mutual Funds[END_REF] develop the industry concentration index (ICI) and show that a high industry concentration is positively associated with future fund performance. [START_REF] Kacperczyk | Unobserved Actions of Mutual Funds[END_REF] use the return gap (RG) defined as the difference between the reported fund return and the return predicted from the previously disclosed fund holdings to measure unobserved actions by mutual fund managers. They find a positive correlation between the return gap (RG) and future fund performance. [START_REF] Cremers | How Active Is Your Fund Manager? A New Measure That Predicts Performance[END_REF] and Petajisto ( 2013) measure a fund s active share (AS) as the extent to which a manager deviates from her benchmark and find that active share positively predicts future fund performance. Finally, without using holdings data, [START_REF] Amihud | Mutual Fund's R2 as Predictor of Performance[END_REF] show that a fund s R 2 with respect to its benchmark positively predicts performance.

To test if changes in ownership survive as performance predictability measure, we now additionally control for the lagged level of ownership and changes in industry concentration (ICI), return gap (RG), active share (AS) and R 2 in our regressions.

To compute the industry concentration index (ICI), we first sort all stocks into ten industries following [START_REF] Kacperczyk | On the Industry Concentration of Actively Managed Equity Mutual Funds[END_REF] and then calculate the weight for a specific industry in a portfolio by summing up the portfolio weights of all stocks belonging to that industry. The sum of the squared industry weights (averaged across the quarters of a year) is then used as a measure of industry concentration.

To calculate the return gap, we follow [START_REF] Kacperczyk | Unobserved Actions of Mutual Funds[END_REF] by comparing the realized fund returns with holding-based fund returns. The latter is a hypothetical portfolio that invests in the previously disclosed fund holdings. We then compound the monthly return gap observations to come up with a yearly measure.

We use the active share database of [START_REF] Cremers | How Active Is Your Fund Manager? A New Measure That Predicts Performance[END_REF] and Petajisto (2013). 4 The active share is calculated as the absolute difference between the portfolio weight the stock universe and divided by two.

To compute the R² measure of Amihud and Goyenko (2013), we first run yearly regressions of fund s monthly excess returns on the Carhart (1997) four factors. We then obtain the fund s R² from these regressions.

We add these other performance predictors to our baseline regressions from Section 3.1. The control variables are the same as in (1). We again control for time and segment fixed effects in the regressions. Standard errors are clustered at the fund level. For the sake of brevity we report only the results for the changes in ownership as well as for the lagged level of ownership and the changes in the other performance predictors. Results are provided in Table 4.

Please insert TABLE 4 approximately here

The ownership change measure is positive and significant at the 1% level for the threefactor and five-factor alpha (and at the 5%-level for the four-factor alpha). The effect is also economically significant: After controlling for the alternative predictors of fund performance, we still find an economically meaningful effect of ownership changes on changes in future risk-adjusted performance. A one-standard-deviation increase in ownership leads to an increase in four-factor alpha of 0.9 percentage points (= 0.02935 × 0.3055) and an increase in five-factor alpha of 1.6 percentage points (= 0.02935 × 0.5323). Of the alternative predictors 4

We downloaded the active share data from Antti Petajisto http://www.petajisto.net/data.html.

only changes in return gap are significantly negatively related to changes in future fund performance. The standard deviation of return gap is 0.03906. Thus, the estimated slope of -0.5725 based on five-factor alpha implies that a one-standard-deviation increase in return gap translates to a decrease of 2.2 percentage points (= 0.03906 × -0.5725) in five-factor alpha.

Overall, the results from this section clearly show that ownership changes predict future risk-adjusted performance even after controlling for existing predictors of fund performance.

Changes in ownership induced by family policies

The results so far indicate that the cross-sectional correlation between managerial ownership and fund performance is not merely driven by unobserved time-invariant fund, manager or family characteristics. We now turn to the question whether the positive correlation is driven by incentive alignment or superior information. Under the superior information hypothesis managers increase their ownership in funds because they know these funds will perform better in the future. Under the incentive alignment hypothesis, managerial ownership aligns the manager s interests with those of fund shareholders leading to better investment decisions resulting in better performance.

We use the adoption of fund family policies requiring managers to hold some ownership in all funds they manage to disentangle the superior information hypothesis from the incentive alignment hypothesis. The idea is to capture ownership changes which are mandated by the fund family and thus do not reflect the manager s information. Under the superior information hypothesis, we expect that these mandatory ownership changes are not related to future changes in fund performance. Under the incentive alignment hypothesis however, ownership leads to aligned incentives regardless if the change in ownership is mandatory or voluntary. Therefore we expect mandatory and voluntary ownership changes to be positively related to future changes in fund performance under the incentive alignment hypothesis.

We construct a dummy variable for fund family policy changes which takes on the value one if the fund family did not have an ownership requirement in place in the past year and has such a requirement in place this year, and zero otherwise. As we cannot directly observe if such a family policy is in place, we proxy for it by using the ownership information we observe. If in a given year and fund family at least one fund has zero ownership, we define that in this fund family and year no strict ownership requirement is in place. If on the other hand, in a given year and fund family all funds in the family have ownership greater than zero, we define that in this fund family and year a strict ownership requirement is in place.

We then interact the fund family policy change dummy with the ownership change measure and run first-difference regressions using the change in the respective performance measure as dependent variable. Other control variables are the same as in (1). We again control for time and segment fixed effects in the regressions and cluster standard errors at the fund level. For the sake of brevity we report only the results for the ownership change measure, the family policy change dummy and the interaction between both. The results are shown in Table 5.

Please insert TABLE 5 approximately here

We find that both the change in the ownership measure as well as the interaction of this change with the fund family policy change dummy strongly predict future changes in risk-adjusted performance. Given that the standard deviation of ownership changes within the family policy change group is 0.00054 , a one-standard-deviation increase of funds where the family adopted an ownership requirement increases future alpha by 0.4 percentage points (= 0.00054 × 7.2716 (estimated slope for the five-factor alpha)) to 0.6 percentage points ((= 0.00054 × 11.8349 (estimated slope for the three-factor alpha)) more than for all other funds.

This leads to an overall effect of up to 1.6 percentage points based on five-factor alpha (= 0.02935 × 0.4269 + 0.00054 × 7.2716). These results support the incentive alignment hypothesis: Ownership has an even stronger effect on performance if the changes are mandated by the fund family.

Next we examine where this performance effect stems from. One possible explanation is that fund managers who are required by their fund family to start holding some ownership in the funds they manage subsequently exert more effort in seeking out profitable investment opportunities. If this is the case, we expect to observe increased activity by these fund managers. In Table 6 we employ the same setup as in Table 5, using changes in the fund s active share, turnover, return gap, as well as cash and equity holdings as dependent variables. 5

Please insert TABLE 6 approximately here

Table 6 shows that the interaction terms have the expected sign: A one standard Taking all results of Section 5 together, we interpret them as supporting the view that managerial ownership aligns interests of managers with those of shareholders and induces managers to exert more effort.

5

We measure cash (equity) holdings as reported cash (equity) from CRSP.

Conclusion

In response to a number of scandals in the mutual fund industry, the Securities and Exchange Commission (SEC) adopted new disclosure requirements for mutual funds in 2004.

Since March 2005, mutual fund managers are required to report their ownership in the funds they manage.

In contrast to earlier studies, we investigate the relationship between managerial ownership and fund performance dynamically. We find that managerial ownership changes are positively related to future changes in risk-adjusted fund performance. We show that this effect is robust to using the mid range instead of the minimum range of the reported dollar range and the results are not driven by changes in the denominator of our ownership measure or by small funds. We further show that our results hold even after controlling for the lagged level of ownership and changes in holdings-based predictors of future fund performance and a fund s R 2 . Thus, the relationship between ownership changes and future risk-adjusted performance is robust to controlling for existing predictability measures.

Using family ownership requirements, we disentangle the superior information and incentive alignment hypotheses: Contrary to the superior information hypothesis and in line with incentive alignment we find that ownership changes which are induced by family policies predict changes in future fund performance even better. Funds that are required to increase their ownership are associated with an increase in alpha by up to 1.6 percent per standard deviation of ownership increase. They do so by increasing their active share, turnover, unobserved actions, equity holdings and by decreasing their cash holdings.

Altogether, this study provides evidence that managerial ownership is an important tool to align manager interests with those of shareholders.

Table 1 Descriptive statistics

This table reports summary statistics for our sample of single-managed US equity mutual funds between 2005 and 2011 compared to the CRSP universe. We report mean and median differences for both samples for the following variables: fund size as measured by the total net assets in million USD, the fund turnover (in %), the fund flows (in %), the fund family size calculated as the total net assets of all team-and single-managed mutual funds in the family, and the Pástor and Stambaugh ( 2003) five-factor alpha. As the board characteristics and ownership measures are not available for the CRSP universe (denoted as n/a), we only report means and medians for the following variables for our sample: the fund board size, the fraction of independent directors on the board, the fund manager ownership level (in % of TNA), and the fund manager ownership change (in %) defined as the difference between the current percentage ownership and the lagged percentage ownership. ***, **, and * denote statistical significance at the 1%-, 5%-, and 10%level, respectively, for the difference in means and medians between both samples (based on t-tests and Mann-Whitney-Wilcoxon rank-sum tests). This table reports results from first-difference regressions of performance on lagged percentage ownership using three different performance measures: (1) [START_REF] Fama | Common Risk Factors in the Returns on Stocks and Bonds[END_REF] three-factor alpha, (2) [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four-factor alpha, and (3) Pástor and Stambaugh (2003) five-factor alpha. The dependent variable is the change from year t-1 to year t of the respective performance measure. As fund control variables we and the lagged family size (calculated as the total net assets of all team-and single-managed mutual funds in the family). As board control variables we use fund's lagged board size as well as the lagged fraction of independent directors on the board. All independent variables are calculated as changes from year t-2 to year t-1. The regression specifications include time fixed effects and segment fixed effects. Robust t-statistics of the regression coefficients in parentheses are based on standard errors clustered by fund. ***, **, and * denote statistical significance at the 1%-, 5%-, and 10%-level, respectively.

Mean

Dependent variable:

Change in Alpha

Table 3 Robustness

This table reports results from first-difference regressions of performance on lagged percentage ownership using three different performance measures: (1) [START_REF] Fama | Common Risk Factors in the Returns on Stocks and Bonds[END_REF] three-factor alpha, (2) [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] fourfactor alpha, and (3) Pástor and Stambaugh (2003) five-factor alpha. The dependent variable is the change from year t-1 to year t of the respective performance measure. In Panel A, the percentage ownership measure is calculated using the midpoint of the reported ownership range instead of the bottom of each range. In Panel B, we replace our percentage ownership measure by a placebo percentage ownership measure. The nominator of this placebo ownership measure takes on the mean dollar ownership in the sample for all funds and years whereas the denominator remains the fund size. In Panel C, we additionally control for the change in fund flows.

Panel D shows results for the subsamples of small and large funds using the sample median of fund size as cutoff. For sake of brevity, we only report the coefficients for the change in ownership. Other independent variables are defined as in This table reports results from first-difference regressions of performance on lagged percentage ownership using three different performance measures: (1) [START_REF] Fama | Common Risk Factors in the Returns on Stocks and Bonds[END_REF] three-factor alpha, (2) [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] fourfactor alpha, and (3) Pástor and Stambaugh (2003) five-factor alpha. The dependent variable is the change from year t-1 to year t of the respective performance measure. The ownership level is calculated as percentage ownership measure by dividing the converted dollar amount by a fund's year end total net assets (TNA) and lagged by one year. We use the active share measure of [START_REF] Cremers | How Active Is Your Fund Manager? A New Measure That Predicts Performance[END_REF] and [START_REF] Petajisto | Active Share and Mutual Fund Performance[END_REF]. To measure the industry concentration, we follow [START_REF] Kacperczyk | On the Industry Concentration of Actively Managed Equity Mutual Funds[END_REF] and sort all stocks into ten industries and calculate the weight for a specific industry in a portfolio by summing up the portfolio weights of all stocks belonging to that industry. The sum of the squared industry weights (averaged across the quarters of a year) is then used as a measure of industry concentration. To calculate the return gap, we follow [START_REF] Kacperczyk | Unobserved Actions of Mutual Funds[END_REF] by comparing the realized fund returns with holding-based fund returns. The latter is a hypothetical portfolio that invests in the previously disclosed fund holdings. We then compound the monthly return gap observations to come up with a yearly measure. To compute the R² measure of [START_REF] Amihud | Mutual Fund's R2 as Predictor of Performance[END_REF], we first run yearly regressions of fund´s monthly excess returns on the [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four factors. We then obtain the fund´s R² from these regressions. For sake of brevity, we only report the coefficients for the change in ownership and the additional control variables. Other independent variables include those defined in Table 2 and fund flows. All independent variables are calculated as changes from year t-2 to year t-1 (except for the ownership level in t-1). This table reports results from first-difference regressions of performance on lagged percentage ownership and the interaction between the ownership change measure and a family policy change dummy using three different performance measures: (1) [START_REF] Fama | Common Risk Factors in the Returns on Stocks and Bonds[END_REF] three-factor alpha, (2) [START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four-factor alpha, and

(3) Pástor and Stambaugh (2003) five-factor alpha. The dependent variable is the change from year t-1 to year t of the respective performance measure. The family policy changes dummy takes on the value one if the fund family did not have an ownership requirement in place in the past year and has such a requirement in place this year, and zero otherwise. For sake of brevity, we only report the coefficients for the change in ownership, the family policy dummy and the interaction between ownership changes and family policy changes. Other independent variables include those defined in

2

  Data and summary statistics For our empirical analysis, we use data from three sources: (1) the Center for Research in Security Prices (CSRP) Survivor-Bias Free US Mutual Fund Database 3 , (2) Thomson Financial Mutual Fund Holdings Database and (3) mutual funds Statement of Additional Information (SAI) filed with the SEC.

3

  Source: CRSP, Center for Research in Security Prices. Graduate School of Business, The University of Chicago. Used with permission. All rights reserved.

(

  1993) three-factor alpha, (2)[START_REF] Carhart | On Persistence in Mutual Fund Performance[END_REF] four-factor alpha, and (3)Pástor and Stambaugh (2003) five-factor alpha. The alpha measures are determined based on a yearly estimation of the respective factor models. We calculate the performance measures based on gross returns as gross returns better reflect the quality of the investment decisions of the fund manager. To onthly net return observations.

  deviation increase in percentage ownership concurrent to the adoption of an ownership requirement by the fund family increases active share by 0.09 percentage points (= 0.00054 × 1.6869), a fund s turnover by 1.23 percentage points (= 0.00054 × 22.8426), return gap by 0.45 percentage points (= 0.00054 × 8.3254) and equity holdings by 0.48 percentage points (= 0.00054 × 8.8644), whereas cash holdings decrease by 0.63 percentage points (= 0.00054 × -11.3762). The results reinforce the incentive alignment hypothesis: Ownership increases by funds when their families adopt ownership requirements significantly predict higher trading activity.

Table 2

 2 Ownership changes and changes in fund performance

Table 2 .

 2 All independent variables are calculated as changes from year t-2 to year t-1. The regression specifications include time fixed effects and segment fixed effects. Robust t-statistics of the regression coefficients in parentheses are based on standard errors clustered by fund. ***, **, and * denote statistical significance at the 1%-, 5%-, and 10%-level, respectively.

	Panel A: Mid range
	Dependent variable:	Change in Alpha t-1 to t
		3F	4F	5F
		(1)	(2)	(3)
	Ownership change t-2 to t-1	0.4262**	0.4359*	0.5780**
		(2.088)	(1.862)	(2.439)
	Change in fund and family characteristics	Yes	Yes	Yes
	Change in board characteristics	Yes	Yes	Yes
	Segment fixed effects	Yes	Yes	Yes
	Time fixed effects	Yes	Yes	Yes
	Observations	606	606	606
	Adjusted R-squared	0.112	0.095	0.093
	Panel B: Placebo ownership change
	Dependent variable:	Change in Alpha t-1 to t
		3F	4F	5F
		(1)	(2)	(3)
	Ownership change t-2 to t-1	0.9719	1.1012	1.2241
		(1.147)	(1.133)	(1.257)
	Change in fund and family characteristics	Yes	Yes	Yes
	Change in board characteristics	Yes	Yes	Yes
	Segment fixed effects	Yes	Yes	Yes
	Time fixed effects	Yes	Yes	Yes
	Observations	606	606	606
	Adjusted R-squared	0.112	0.097	0.093

Table 4

 4 Ownership changes and alternative predictors of future fund performance

  The regression specifications include time fixed effects and segment fixed effects. Robust t-statistics of the regression coefficients in parentheses are based on standard errors clustered by fund. ***, **, and * denote statistical significance at the 1%-, 5%-, and 10%-level, respectively.

	Dependent variable:	Change in Alpha t-1 to t
			3F	4F	5F
			(1)	(2)	(3)
	Ownership change t-2 to t-1	0.4362***	0.3055**	0.5323***
			(2.895)	(2.311)	(4.183)
	Additional controls
	Ownership level t-1	0.0038	0.1581	0.0278
			(0.024)	(1.098)	(0.200)
	Change in active share t-2 to t-1	0.3378**	0.2198	0.2080
			(2.277)	(1.444)	(1.320)
	Change in industry concentration t-2 to t-1	-0.3426	-0.3353	-0.3768*
			(-1.592)	(-1.567)	(-1.708)
	Change in return gap t-2 to t-1	-0.4957*** -0.4674*** -0.5725***
			(-2.868)	(-3.042)	(-3.258)
	Change in R 2	t-2 to t-1	0.1344	0.1097	0.1719
			(0.890)	(0.725)	(1.080)
	Change in fund and family characteristics	Yes	Yes	Yes
	Change in board characteristics	Yes	Yes	Yes
	Segment fixed effects	Yes	Yes	Yes
	Time fixed effects	Yes	Yes	Yes
	Observations		321	321	321
	Adjusted R-squared	0.211	0.164	0.204

Table 5

 5 Family policy change: fund performance

  Table 2 and fund flows. All independent variables are calculated as changes from year t-2 to year t-1. The regression specifications include time fixed effects and segment fixed effects. Robust t-statistics of the regression coefficients in parentheses are based on standard errors clustered by fund. ***, **, and * denote statistical significance at the 1%-, 5%-, and 10%-level, respectively.

	Dependent variable:	Change in Alpha t-1 to t
		3F	4F	5F
		(1)	(2)	(3)
	Ownership change t-2 to t-1	0.2869**	0.2856**	0.4269***
		(2.460)	(2.304)	(3.442)
	Ownership change t-2 to t-1 * Family policy change t-2 to t-1	11.8349***	9.4690***	7.2716***
		(4.654)	(3.680)	(2.680)
	Family policy change t-2 to t-1	-0.0261	-0.0231	-0.0106
		(-1.460)	(-1.285)	(-0.581)
	Change in fund and family characteristics	Yes	Yes	Yes
	Change in board characteristics	Yes	Yes	Yes
	Segment fixed effects	Yes	Yes	Yes
	Time fixed effects	Yes	Yes	Yes
	Observations	569	569	569
	Adjusted R-squared	0.123	0.099	0.098

Table 6 Family policy change: channels

This table reports results from first-difference regressions of various trading activity measures as dependent variables on lagged percentage ownership and the interaction between the ownership change measure and a family policy change dummy. Active share and return gap are defined as in Table 4. To examine changes in cash and equity holdings and, we measure cash (equity) holdings as reported cash (equity) from CRSP. The family policy changes dummy takes on the value one if the fund family did not have an ownership requirement in place in the past year and has such a requirement in place this year, and zero otherwise. For sake of brevity, we only report the coefficients for the change in ownership, the family policy dummy and the interaction between ownership changes and family policy changes. Other independent variables include those defined in Table 2


Jilles S Dibangoye

On Markov Policies For Decentralized POMDPs

Keywords: Decentralized control, centralized planning, Markov policies, Markov decision processes Processes décisionnels de de Markov partiallement observables et décentralisés, Planification Multi-Agents

This paper formulates the optimal decentralized control problem for a class of mathematical models in which the system to be controlled is characterized by a finite-state discrete-time Markov process. The states of this internal process are not directly observable by the agents; rather, they have available a set of observable outputs that are only probabilistically related to the internal state of the system. The paper demonstrates that, if there are only a finite number of control intervals remaining, then the optimal payoff function of a Markov policy is a piecewise-linear, convex function of the current observation probabilities of the internal partially observable Markov process. In addition, algorithms for utilizing this property to calculate either the optimal or an error-bounded Markov policy and payoff function for any finite horizon is outlined.

Introduction

The two concepts of state and state transition are essential to the modeling of complex dynamic systems. The concept of state allows one to focus on the features system that are essential to the problem at hand, while the concept of state transition provides the mechanism for structuring the system's dynamic behavior. In most situations, there is an element of uncertainty in the transitions of the process from one state to another, and this leads naturally to the use of Markov processes as quantitative models of the system. Unfortunately, in many practical applications agents are not permitted exact observation of the state of the process; rather, each agent possesses only certain local, unshared observation outputs, and acts without full knowledge of what others observe or plan to do. For example, there are many situations in game theory in which we would like to model the dynamics of the game's internal state as a Markov process, but this state is not directly observable. In such cases, we can often model what is observable as probabilistically related to the true state of the system. Figure 1 presents a pictorial representation of such a model, termed a decentralized partially observable Markov process [START_REF] Hans S Witsenhausen | Separation of estimation and control for discrete time systems[END_REF][START_REF] Yoshikawa | Separation of estimation and control for decentralized stochastic control systems[END_REF][START_REF] Daniel | The Complexity of Decentralized Control of Markov Decision Processes[END_REF][START_REF] Nair | Taming Decentralized POMDPs: Towards Efficient Policy Computation for Multiagent Settings[END_REF][START_REF] Emery-Montemerlo | Approximate Solutions for Partially Observable Stochastic Games with Common Payoffs[END_REF]. In this paper we shall consider decentralized partially observable Markov processes for which the underlying Markov process is a discrete-time finite-state Markov process; in addition, we shall limit the discussion to processes for which the number of possible outputs at each observation is finite.

As an example of this system, consider a simple two-agent "tiger world" -see Figure 1 -where the optimal control policies require the agents to coordinate their control alternatives [START_REF] Nair | Taming Decentralized POMDPs: Towards Efficient Policy Computation for Multiagent Settings[END_REF]. In this world there are two doors: behind one randomly chosen door is a hungry tiger, and behind the other is a pile of gold. Each agent has unique abilities. Agent 1 (the tiger listener) can hear the tiger roare.g., z 1 -which is a noisy indication of its current location, but cannot open the doors. Agent 2 (the door opener) can open the doorse.g., a 2 -but cannot hear the roars. To facilitate communication, agent 1 has two control alternatives, signal left and signal righte.g., a 1 -which each produce a unique observation e.g., z 2 -for agent 2. When a door is opened, the world resets and the tiger is placed behind a randomly chosen door. To act optimally, agent 1 must listen to the tiger's roars until it is confident about the tiger's location and then send the appropriate signal to agent 2. Agent 2 must wait for this signal and then open the appropriate door.

This example illustrates the characteristics of the general optimal decentralized control problem for partially observable Markov processes. This paper formulates and solves this general optimal distributed problem for a process that is to operate for only a finite number of periods under the control of a Markov Inria policy. A later paper will examine this decentralized control problem for a process that is to operate into the indefinite future.

Over the past decade there has been extensive research into decentralized partially observable Markov processes. Earlier work on decentralized control problems relies on distributed planning [START_REF] Peshkin | Learning to Cooperate via Policy Search[END_REF]. By distributed planning we meant both:

• the planning process is distributed among a variety of agents;

• a control policy is formulated that can be distributed among a variety of agents; This is the most challenging version of distributed planning, that is when both the planning process and its results are intended to be distributed [START_REF] Durfee | Distributed Problem Solving and Planning[END_REF]. In this case, it might be unnecessary to ever have a multi-agent control policy represented in its entirely anywhere in the system, and yet the distributed pieces of the control policy should be compatible, which at a minimum means that agents should not conflict with each other when executing the control policies, and preferably should help each other achieve their control policies when it would be rational to do so (e.g., when a helping agent is no worse off for its efforts). The literature of this kind of distributed planning is relatively rich and varied. It ranges from ad-hoc distributed coordination strategies, e.g., logic formalization; theory of intentions [START_REF] Cohen | Intention is choice with commitment[END_REF][START_REF] Jennings | Using joint responsibility to coordinate collaborative problem solving in dynamic environments[END_REF], to more formal approaches, e.g., distributed constraint reasoning [START_REF] Yokoo | Distributed constraint satisfaction: foundations of cooperation in multi-agent systems[END_REF]. In decentralized control of partially observable Markov processes, the control problem is broken into separate and independent partially observable Markov processes, one for each agent. Figure 2 illustrates a pictorial representation of such a model. In order to avoid sequences of control alternatives that lead to conflicts, each agent i's internal partially observable Markov process incorporates the total available information -denoted (S × Π ¬i ) -of the other agents, that is, what they observe and plan to do. As such, the planning process for each agent is achieved independently and separately for almost all exact state-of-the-art methods. Such solution methods discussed in [START_REF] Hansen | Dynamic Programming for Partially Observable Stochastic Games[END_REF] and [START_REF] Szer | Point-based Dynamic Programming for DEC-POMDPs[END_REF] suggest to computing the optimal history-dependent policy for each agent over the continuum (S × Π ¬i ). That is, the space of all possible probability distributions over what the other agents observe and plan to do. Unfortunately, the space of history-dependent policies Π ¬i of the other agents can become prohibitively huge as time goes on. More precisely, it grows doubly exponentially with the number of agents and control intervals. While such approaches preserves the ability to eventually find an optimal control policy for each agent -which is a key property -yet it makes planning impractical even for small toy problems. In attempts to scale up, specific policy classes have been addressed. [START_REF] Amato | Optimizing fixed-size stochastic controllers for POMDPs and decentralized POMDPs[END_REF][START_REF] Szer | An Optimal Best-First Search Algorithm for Solving Infinite Horizon DEC-POMDPs[END_REF][START_REF] Frans A Oliehoek | Optimal and Approximate Q-value Functions for Decentralized POMDPs[END_REF][START_REF] Jilles | Toward error-bounded algorithms for infinite-horizon DEC-POMDPs[END_REF][START_REF] Frans A Oliehoek | Incremental Clustering and Expansion for Faster Optimal Planning in Dec-POMDPs[END_REF][START_REF] Jilles | Error-Bounded Approximations for Infinite-Horizon Discounted Decentralized POMDPs[END_REF][START_REF] Jilles | Exploiting Separability in Multiagent Planning with Continuous-State MDPs (Extended Abstract)[END_REF][START_REF] Jilles S Dibangoye | Optimally Solving Dec-POMDPs as Continuous-State MDPs[END_REF] focus the effort of exact techniques only over memorybounded history-dependent policies. [START_REF] Seuken | Formal models and algorithms for decentralized decision making under uncertainty[END_REF][START_REF] Jilles | Point-based incremental pruning heuristic for solving finite-horizon DEC-POMDPs[END_REF][START_REF] Kumar | Point-based backup for decentralized POMDPs: complexity and new algorithms[END_REF][START_REF] Kumar | Scalable Multiagent Planning Using Probabilistic Inference[END_REF][START_REF] Kumar | Probabilistic Inference Techniques for Scalable Multiagent Decision Making[END_REF] pushed a little bit further the envelop by investigating the approximate calculation of memory-bounded and history-dependent policies.

In this paper, we review the optimal Markov policy computation in the context of decentralized control of partially observable Markov decision processes [START_REF] Jilles | Scaling Up Decentralized MDPs Through Heuristic Search[END_REF][START_REF] Jilles | Producing Efficient Error-bounded Solutions for Transition Independent Decentralized MDPs[END_REF], providing new insights. From the perspective of applications, we find it comforting that by restricting attention to Markov policies, which are simple to implement and calculate, we may achieve as large expected total reward as if we used approximate memory-bounded history-dependent policies.

Of special significance for this paper is the work of [START_REF] Jilles S Dibangoye | Optimally Solving Dec-POMDPs as Continuous-State MDPs[END_REF], who considered the general problem from the standpoint of centralized planning for decentralized control. [START_REF] Jilles S Dibangoye | Optimally Solving Dec-POMDPs as Continuous-State MDPs[END_REF] argued that control policies that are to be executed in distributed fashion can nonetheless be formulated in centralized manner. A centralized coordinator agent with such a policy can break it into separate threads. These separate control policies can be passed to agents that can execute them. To do so, [START_REF] Jilles S Dibangoye | Optimally Solving Dec-POMDPs as Continuous-State MDPs[END_REF] formulated the general problem as a partially observable Markov problem with a single constraint. The constraint induced that the resulting control policy can be broken into separate control policies, one for each agent. This process is illustrated in Figure 3. However, this work is significantly different. Indeed, it is the first research to consider control policies from the class of Markov policies. This class of policy plays a central role in our results. 2 Properties of the Model

Basic Terminology

To begin the explicit formulation of the control problem for a decentralized partially observable Markov process, we assume that the internal dynamics of the system under control can be modeled as a |S |state discrete-time Markov process. If there are τ control periods remaining, the problem is to select an alternative from the available set A that will optimize the performance of the system during its remaining lifetime. If alternative a τ is selected, then the conditional probability that the internal process will make its next transition to state s τ-1 if it is presently in state s τ will be written as

T (s |s, a) = p(s = s τ-1 |s = s τ , a = a τ ) (1)
Inria An observation will follow each transition, with

O(z |s, a, s ) = p(z = z τ-1 |s = s τ , a = a τ , s = s τ-1 ) (2)
denoting the probability of observing output z τ-1 if the internal state of the process is s τ-1 and alternative a τ is controlling the system in state s τ . It will prove convenient to define the probability of transiting from internal state s and observation output z to the internal state s and observation output z when agents together execute action alternative a:

p(s , z |s, a) = T (s |s, a) • O(z |s, a, s ). ( 3
)
With this representation of the process, it is easy to see that, if O(z |s, a, s ) is independent of s , then the observation of the output will yield no additional information about the internal state of the process. This is the case of the nonobservable Markov process. The other extreme is the more usual case that has been studied extensively in the literature [START_REF] Ronald | Dynamic Programming and Markov Processes[END_REF][START_REF] Martin L Puterman | Markov Decision Processes, Discrete Stochastic Dynamic Programming[END_REF]. If there is one output for each internal state of the process and if for each alternative O(z |s, a, s ) > 0 if and only if s ≡ z , then the process is said to be completely observable. In the paper, we refer to the latter model as decentralized Markov decision process with full joint observability [START_REF] Daniel | The Complexity of Decentralized Control of Markov Decision Processes[END_REF].

The calculation of an optimal control policy requires a reward structure for the process. Thus, we define

R(s, a) = r(s = s τ , a = a τ ) (4)
as the immediate award accrued if, while under the control of the alternative a τ during one control interval, the process internal state is s τ . The analysis to follow assumes that the centralized coordinator agent has no direct observation of the accrued rewards; that is, it only observes the outputs of the observation part of the process. If this assumption is violated, then it is easy to redefine the observation outputs of the process to include the internal states that are immediately available to the centralized coordinator agent. These quantities (S , A, Z, R, T, O) define the probabilistic model that underlies each decentralized control problem of partially observable Markov decision process. When the agents operate over N control intervals and has a discount factor λ ∈ [0, 1), the model is referred as finite horizon case with discounted rewards.

Decentralized Control Policy

The central objective of decentralized control of Markov decision process planning is to compute a decentralized control policy ξ for selecting action alternatives in order to maximize the expected sum of reward, that it gets on the next N control intervals; it should maximize

E ξ        N-1 τ=0 λ τ • r τ (s τ , a τ ) + r N (s N )        (5)
where r τ is the reward received at control interval τ.

A decentralized control policy ξ ≡ {ξ i } i describes the behavior ξ i of the agents at the execution time. When there is only one control interval remaining, all the agents can do is to take a single action alternative. With two control intervals remaining, they can take an action, make an observation, then take another action. In this paper, we represent an agent's control policy ξ i by a sequence of decision rules ξ i := (ξ 0 , • • • , ξ N-1 ). A decision rule ξ τ at time τ is a mapping from histories of observation outputs to actions. In general decentralized decision-making, there are two different classes of policies of potential interest. In the more general class, an action can be chosen on the basis of the entire history of past observations. That is, the class of history-dependent decision rules and policies, which are often represented a a a using decision tree, as illustrated in Figure 4. In a more restricted class, the action at time τ is based only on the current observation. Agent i's policy

• • • • • • depth N branching |Z| horizon N states |Z| z |Z| z 1 z 0 Markov Policy History-dependent Policy
ξ i = (ξ i 0 , ξ i 1 , • • • , ξ i N-1
) is said to be Markovian if it depends on the past history only through the local observation. That is, (Markov) decision rule ξ i τ is a mapping from observations z i ∈ Z i to local actions a i ∈ A i . We note p ξ i τ (a i |z i ) the probability of taking local action alternative a i after perceiving local observation output z i when executing decision rule ξ i τ . Since we are mainly concerned with decentralized decision making, we need to introduce a specific class of policies that is appropriate for this purpose. We define the class of decentralized control policies as follows. Team policy ξ = (ξ 0 , • • • , ξ N-1 ) is said to be a decentralized control policy if decision rule ξ τ is defined as a n-tuple

(ξ 1 τ , • • • , ξ n τ ) of decision rules, one ξ i τ for each agent i = 1, • • • , n. Notice that p ξ τ (a|z) = i p ξ i τ (a i |z i
) is the probability of selecting joint action alternative a after perceiving joint observation z when following ξ τ .

Sufficient Statistic

The uncertainty in the dynamic of the internal process produce uncertainty about the internal state of the system. For our formulation of the decentralized control problem, the current state of information about the internal state of the system can be encoded as the occupancy state θ, where θ(s, z) is the probability of observing output z and the internal state of the process is s. In other words, if the centralized coordinator agent has available to him his past observations of the process's outputs, then at any time the vector θ is a sufficient statistic for these past sequences of observations. Appendix A presents a proof of this rather intuitive result.

From this result it follows that the dynamics behavior of the occupancy state θ is itself a discrete-time continuous-state deterministic Markov process. This dynamic behavior of the state information is crucial to the calculation of the optimal as well as approximate control policy. If our prior state information about the observation output of the system is denoted by θ, and if the current decision rule is ξ τ , then we must be able to calculate our updated state information. If θ (s , z ) is the updated probability that the observation output and internal state of the system are z and s respectively given the new information, then the application of simple probability operations yields the following equation (Appendix A contains Inria the complete derivation):

θ (s , z ) = s,z,a p ξ τ (a|z) • θ(s, z) • p(s , z |s, a) (6)
Equation ( 6) defines a transformation from the vector θ τ to the vector θ τ-1 . Since this transformation plays an important role in the succeeding development, it is useful to introduce the notation

θ τ-1 = χ(θ τ |ξ τ ) (7)
The space of all probability distributions over S and Z is known as a standard (|S ||Z|-1)-simplex, denoted for the sake of simplicity in the remainder of the paper.

The Optimality Equations

With this as a background, the remainder of this section will introduce the optimality equations. These equations and their solutions play a central role in the theory of decentralized control of Markov decision processes.

To this end, we define V τ (θ τ ) as the maximum expected reward that the system can accrue during the lifetime of the process if the current occupancy state is θ τ and there are τ control intervals remaining before the process terminates. Then, expanding over all possible next transitions yields the recursive equation ( 8), for any occupancy state θ τ ∈ .

V τ (θ τ ) = max ξ τ s,z,a p ξ τ (a|z) • θ τ (s, z) • R(s, a) + λV τ-1 (χ(θ τ |ξ τ )) (8)
We refer to this system of equations (8) as the optimality equations for decentralized control of partially observable Markov decision processes under the control of Markov policies. Equation ( 8) can be simplified somewhat by defining the expected immediate reward for occupancy state θ τ if decision rule ξ τ is used during the next control interval as

R(θ τ , ξ τ ) := s,z,a p ξ τ (a|z) • θ τ (s, z) • R(s, a) (9)
Equation ( 8) then becomes

V τ (θ τ ) = max ξ τ R(θ τ , ξ τ ) + λV τ-1 (χ(θ τ |ξ τ )) (10)
Equation ( 10) is valid for any control interval τ ≥ 1. The value of terminating the process with a final occupancy state θ 0 is just:

V 0 (θ 0 ) = max ξ 0 R(θ 0 , ξ 0 ) (11)
Equation ( 10) represents a dynamic-programming problem over a continuous state space, the space of occupancy states. This is consistent with the previous assertion that the occupancy state θ τ is itself the state of the discrete-time continuous state deterministic Markov process. Appendix A discusses this in more detail.

p(z) p(z ) V τ (θ τ ) χ(θ τ |ξ τ ) υ 1 υ 0 θ τ υ 2 χ(θ τ |ξ τ ) ξτ ξ τ

Convexity and Piece-Wise Linearity

We have shown that the solution of the decentralized control of Markov decision process is the solution of the optimality equations

V τ (θ τ ) = max ξ τ R(θ τ , ξ τ ) + V τ-1 (χ(θ τ |ξ τ )) V 0 (θ 0 ) = max ξ 0 R(θ 0 , ξ 0 ), θ 0 ∈ (12)
Although ( 12) appears rather formidable, its solution has a rather simple form. In particular, we shall show that V τ (θ τ ) is piecewise linear and convex, and can thus be written as

V τ (θ τ ) = max k (υ τ • θ τ ) (13)
for some set of vectors υ k τ , for k = 1, 2, • • • . We shall use the term hyperplane to refer to one of the vectors υ τ in Equation ( 13). Apart from being an amusing curiosity the convexity and piecewise linearity of the value function is particularly useful for establishing convergence properties as well as for simplification of numerical algorithms.

Before giving the main theorem we will establish some simple properties of convex functions. We have Lemma 1. Let f 1 (x) and f 2 (x) be convex functions. The function f (x) = max { f 1 (x), f 2 (x)} is then also convex.

We have further

Lemma 2. Let the function R(•, ξ τ ) : → R be convex and let χ(•|ξ τ ) be a convex transformation which maps into . The function f : → R defined by f (x) = R (χ(x|ξ τ ), ξ τ ) , ∀x ∈ is then also convex.

Inria

Let 0 ≤ λ ≤ 1 and µ = 1 -λ, take x ∈ and y ∈ , then using convexity of R ξ τ and χ(•|ξ τ ) we find

f (λx + µy) = R (λ • χ(x|ξ τ ) + µ • χ(y|ξ τ ), ξ τ ) (14) ≤ λ • R (χ(x|ξ τ ), ξ τ ) + µ • R ξ τ (χ(y|ξ τ ), ξ τ ) (15) = λ f (x) + µ f (y) (16)
and the result is established.

We can now state the main result.

Theorem 1. Let χ(•|ξ τ ) be mappings from into , the functions V τ : → R defined recursively by ( 12) are then convex.

The linear function R(θ τ , ξ τ ) is convex. By repeated application of Lemma 1 we now find that V 0 (θ 0 ) is convex. Now consider V 1 (θ 1 ). It follows from Lemma 2 that V 0 (χ(θ 1 |ξ 1 )) is convex. As a sum of convex functions is convex we find that both terms within the brackets of the right member of ( 12) are convex. Application of Lemma 1 now shows that V 1 (θ 1 ) is convex. Now proceeding by induction we can show that all functions V τ (θ τ ) are convex, and the theorem is proved.

There are two important practical points to keep in mind. First, if the set of hyperplanes for V τ-1 has been calculated, then it is possible to calculate the optimum decentralized control policy and the corresponding hyperplane for any specified occupancy state θ τ for the τ-horizon case. This property will be most useful when we derive an algorithm for calculating the value function in Section 3. Secondly, the calculation of a new hyperplane using (13) yields an optimal decentralized control policy associated with each new hyperplane. Thus, in storing the optimal distributed decision rule, it is not necessary to store the complete description of the policy regions as illustrated in Figure 5; we need only store the set of hyperplanes along with the appropriate distributed decision rule for each hyperplane. Then, to find the optimal distributed decision rule for some occupancy state θ τ , we merely carry out the maximization in (13) and then use the control alternative associated with the maximizing hyperplane. This represents a considerable practical saving over previous solutions to this problem.

3 Exact Algorithm for Computing V τ (θ τ )

Having discovered the relatively simple form of the solution to the optimal decentralized control problem, it only remains to construct an orderly practical procedure for computing the hyperplanes and the corresponding mapping of these vectors onto the set of distributed alternative controls. In the succeeding discussion, we shall assume that the hyperplanes υ k τ-1 for the case of (τ -1) control intervals have been calculated. The problem then is to find an algorithm for calculating the hyperplanes υ k τ from this information. Let P ξ τ be the matrix whose the rows are p(•, •|s, ξ τ (z)) and R ξ τ be the vector whose components are R(s, ξ τ (z)), for any internal state and observation output (s, z). To implement the exact calculation of exact value function V τ , we first generate hyperplanes υ ξ τ defined as follows:

υ ξ τ = R ξ τ + γP ξ τ • υ τ-1 , ∀ξ τ , ∀υ τ-1 (17)
Finally value function V τ is represented by the set of hyperplanes υ ξ τ . It is often the case that an hyperplane υ τ will be completely dominated by another hyperplane υτ . Similarly, an hyperplane may be fully dominated by a set of other hyperplanes. Those hyperplanes can be pruned away without affecting the solution. Checking whether a single hyperplane is dominated requires solving a linear program with |S ||Z| variables and |V τ | constraints. But, it can be time-effective to apply pruning after each control interval to prevent an explosion of the solution size. Formally, an hyperplane υ is dominated, if for each occupancy state θ there exists another hyperplane υ such that ( υ -υ) • θ ≥ 0. This leads directly to the linear program below:

maximize ζ subject to: ( υ -υ) • θ ≥ ζ, (∀ υ υ) with s,z θ(s, z) = 1 (∀θ)
If the result of that program is negative or equal to zero (ζ ≤ 0), then hyperplane υ can be pruned away without affecting the solution.

To better understand the complexity of the exact calculation of V τ , let |V τ-1 | be the number of hyperplanes required to represent value function V τ-1 , and D the space of deterministic decision rules. So, in the worst case, the new solution V τ has size |D||V τ-1 |. Given that this polynomial growth occurs for over times, the importance of pruning away unnecessary hyperplanes is clear. The most critical point of this exact algorithm for computing V τ is the linear program described above. While the theoretical complexity of solving such a problem is only polynomial, in practice this problem is computationally intensive since in the worst case the number of constraints increases with decreasing control intervals remaining. Notice that we consider only deterministic control policies, while the main body of the paper consider randomized control policies. We rely on the assumption that deterministic policies may achieve as large an expected discounted total reward as if we used randomized control policies.

4 An Algorithm for Approximating V τ (θ τ )

To begin the algorithm, we pick a set of representative occupancy states at control interval τ, say Θ τ , and then applies value function calculation on those occupancy states only. As shown in the previous section, given hyperplanes at control interval (τ -1), it is relatively straightforward to generate hyperplanes at control interval τ. In this algorithm, we apply this procedure to the entire set of occupancy states Θ τ such that we generate a full value function at control interval τ. Given value function V τ-1 , we compute the hyperplane υ θ τ that is optimum for each occupancy state θ ∈ Θ τ as follows:

υ θ τ := max ξ τ R ξ τ • θ τ + γV τ-1 (χ(θ τ , ξ τ )) ( 18
)
To better understand the complexity of information-based approximations, let |Θ τ | be the number of occupancy states at control interval τ, and |V τ-1 | the number of hyperplanes in value function one step earlier. A full information-based approximation takes only polynomial time, i.e., |S ||Z||D||V τ-1 ||Θ τ |, and even more crucially, the size of value function V τ remains constant, i.e., |Θ τ |, over control intervals.

Despite these outstanding results, it is likely that the information-state approximation does not scale up to realistic applications. That is mainly because, in such applications size |D| of the set of all possible decision rules may be too prohibitive. Moreover, a large number of hyperplanes generated when using the exact technique is pruned away since each information-state θ τ ∈ Θ τ keeps only a single hyperplane.

The Error-Bound

For any occupancy state set Θ τ and control interval τ, our approximation produces V τ . We now show that the error between V τ and the optimal value function V τ is bounded. The bound depends on how densely θ τ samples the occupancy state simplex ; with denser sampling, V τ converges to V τ . Cutting off the approximate algorithm iterations at any control interval, we know that the difference between V τ and the optimal V τ is not too large.

Inria p(z) p(z ) V τ (θ) θ 1 Vτ(θ1) Vτ(θ0) θ 0 Vτ(θ2) θ 2
Figure 6: Theoretical properties of the approximation of V τ .

We begin by defining the density β of a set of occupancy states θ to be the maximum distance from any legal occupancy state to θ [START_REF] Inria | Anytime Point-Based Approximations for Large POMDPs[END_REF]. More precisely:

β = max θ ∈ min θ∈Θ τ θ -θ 1 (19)
Then, we can prove that the error introduced by a single application of operator H, instead of τ , is bounded by ,

≤ 1 -γ N-τ 1 -γ r ∞ β (20)
To better understand this error bound, let θ ∈ be the occupancy state where our approximate algorithm makes its worst error in the value update, and θ ∈ Θ τ be the closest (1-norm) sampled occupancy state to θ . Let υ be the hyperplane that is maximal at θ, and υ be the hyperplane that would be maximal at θ . By failing to include υ in its solution set, our approximation makes an error σ of at most (υ • θυ • θ ).

On the other hand, since υ is maximal at θ, then (υ

• θ ≤ υ • θ). So, ≤ υ • θ -υ • θ ≤ υ • θ -υ • θ + (υ • θ -υ • θ) ≤ υ • θ -υ • θ + υ • θ -υ • θ ≤ (υ -υ) • (θ -θ) ≤ υ -υ ∞ θ -θ 1 ≤ 1 -γ N-τ 1 -γ r ∞ β
The last inequality holds for r ∞ = max s,a |R(s, a)|. Moreover, 1-γ N-τ 1-γ r ∞ represents the maximum reward achievable starting at any state and following some sequence of action alternatives and observation outputs during (N -τ) control intervals. When we apply operator H over the τ control intervals remaining the error

σ τ = (1-γ N-τ )(1-γ τ+1 ) (1-γ) 2
r ∞ β produced by our approximation is given by,

τ = V τ -V * τ ∞ (21) = HV τ-1 -HV * τ-1 ∞ (22) ≤ HV τ-1 -HV τ-1 ∞ + HV τ-1 -HV * τ-1 ∞ (23) ≤ 1 -γ N-τ 1 -γ r ∞ β + HV τ-1 -HV * τ-1 ∞ (24) ≤ 1 -γ N-τ 1 -γ r ∞ β + γ V τ-1 -V * τ-1 ∞ (25) = 1 -γ N-τ 1 -γ r ∞ β + γ τ-1 (26) = (1 -γ N-τ )(1 -γ τ+1 ) (1 -γ) 2 r ∞ β (27)
The bound described in this section depends on how densely θ samples the occupancy state simplex . In the case where not all occupancy states are reachable, we don't need to sample all of densely, but can replace by the set of reachable occupancy states ¯ (Figure 7).

Occupancy State Set Θ τ Expansion

There is a clear trade-off between including fewer occupancy states (which would favor fast planning over good performance), versus including many occupancy states (which would slow down planning, but ensure better performance). This brings up the question of how many occupancy states should be included. However, the number of occupancy states is not the only consideration. It is likely that some collections of occupancy states (for example those frequently encountered) are more likely to produce a good value function than others. This brings up the question of which occupancy states should be included.

The error bound in (20) suggests that the occupancy state-based approximation performs best when its occupancy state set is uniformly dense in the set of reachable occupancy states. As shown in Figure 7, we can build a tree of reachable occupancy states. In this representation, each path through the tree corresponds to a sequence in the occupancy state space, and increasing depth corresponds to an increasing plan horizon.

As shown in this figure, the set of reachable occupancy states at control interval τ -denoted ¯ τincreases exponentially with increasing time, i.e., | ¯ τ | = O(|D| τ-1 ). Including all reachable occupancy states would guarantee optimal performance, but at the expense of computational tractability. Therefore, we must select a subset Θ τ ⊂ ¯ τ , which is sufficiently small for computational tractability, but sufficiently large for good value function approximation.

The approach we propose consists of initializing set Θ 0 to contain the initial occupancy state θ 0 , and then gradually expanding θ τ by greedily choosing new reachable occupancy states that improve the worst-case density as rapidly as possible.

To choose new reachable occupancy states, we stochastically simulate single-step forward trajectories from those occupancy states already in θ τ . Simulating a single-step forward trajectory for a given θ τ ∈ Θ τ requires selecting a distributed decision rule ξ τ , and then computing the new occupancy state θ ξ τ τ = χ(θ τ |ξ τ ). Rather than selecting a single distributed decision rule to simulate the forward trajectory for a given θ τ ∈ Θ τ , we do so with each distributed decision rule, thus producing new occupancy state set Θ τ+1 = {θ ξ τ |∀ξ τ ∈ D, ∀θ ∈ Θ τ }. Rather than accepting all new occupancy states, we calculate the L 1 Inria θ distance between each θ ξ τ τ and its closest neighbor in Θ τ+1 . We then keep only the occupancy state θ ξ τ τ that is farthest away from any occupancy state already in Θ τ+1 .

θ ξ 0 θ ξ 0 ξ 0 θ ξ 0 ξ m • • • • • • θ ξ m θ ξ m ξ m θ ξ m ξ 0 • • • • • • • • • • • • • • • |D| |D| |D| ξ 0 ξ m

Conclusion and discussion

As stated earlier, previous authors have attacked the calculation of the optimal decentralized control policy for partially observable Markov processes. Most of these exact methods are based on a single agent occupancy state -that is a mixture over the internal state of the process and all possible control policies of its teammates. With this as a background, the authors derive different smart strategies to compute the optimal history-dependent control policy for each agent. While useful, this occupancy state increases exponentially with increasing control intervals -thus restricting the ability to scale up to realistic applications. As an attempt to scale up, multiple approximate solution methods have emerged. Among all the class of information-state based approximation has shown impressive performances on the twoagent small toy problems. These methods -known as point-based algorithms -perform the update of the value function based solely on the mixture over the internal state of the system. By planning over a bounded number of such occupancy states, they favor fast planning over good performance. Indeed, a recent analysis on the solution produced by such techniques show that the error increases with decreasing number of occupancy states and increasing planning horizon. This is mainly because the occupancy state based on the mixture over the internal states of the system does not provide the sufficient statistic for decision-making in decentralized control of partially observable Markov decision processes. The other major limitation of information-states based approximations is the selection of a small parameter that bounds the number of possible control policies. Unfortunately, choosing the right parameter is not obvious, even more dynamically adjusting it may raise non negligible computation costs. Our work provides a alternative to this approximation scheme.

In this paper, we provide both theoretical and algorithmic insights for optimal as well as error-bounded approximate decentralized control of partially observable Markov decision processes using Markov policies. On the one hand, we demonstrate that the mixture over internal states of the process and observation outputs is the sufficient statistic for decision-making in such a setting. Having discovered the relatively simple form of the sufficient statistic, we demonstrate the piecewise linear, and convex nature of the value function. On the other hand, we derive a method for automatic generation of these occupancy states. This opens a large range of possible applications. From the perspective of applications, we find it comforting that by restricting attention to Markov policies, which are simple to implement and calculate, we may achieve as large expected total reward as if we used approximate memory-bounded history-dependent policies. We hope this work lays the foundation for further work in applying Markov policies in decentralized decision-making problem.

Appendix A

In this appendix we show that a sufficient statistic for the past histories of a decentralized control problem of partially observable Markov processes under the control of a Markov policy is just the current observation-state vector θ τ . In demonstrating this property, we derive the rule for updating the observation-state vector from one control interval to the next. To make this explicit, we define ξ 0:t as the total available information about the process at the end of the control interval t. Notice that in this appendix the time variable t increases with increasing time, whereas in the main body of the paper the time variable τ, which is equal to the number of remaining control intervals, decreased with increasing time. For the process as defined in this paper, the only information that we obtain during a control interval is the fact that a particular distributed decision rule ξ t has been executed. If ξ 0:t-1 and ξ t denote the distributed Markov policy at the end of control interval (t -1), and the current decision rule, respectively, then we can write

ξ 0:t = ξ 0:t-1 , ξ t (28)
That is, ξ 0:t represents our state information prior to control interval t plus the additional information that a particular distributed decision rule was recorded. By the definition of the observation-state vector, θ t (s , z ) = p(s = s t , z = z t |ξ 0:t ), (29) where z t is the discrete-valued randomized variable equal to the observation output of the process at the conclusion of control interval t. Application of the definition of conditional probability yields θ t (s , z ) = p(s = s t , z = z t , ξ 0:t )/p(ξ 0:t ) (30)

The expansion of (30) over all possible internal states and observation outputs of the process at the end of (t -1) plus the expansion of the joint probability as a product of conditional probabilities produces θ t (s , z ) = s,z p(s , z |s, z, ξ 0:t ) • p(s, z, |ξ 0:t ) (31)

The substitution of ( 28) into (31) yields θ t (s , z ) = s,z p(s , z |s, z, ξ 0:t-1 , ξ t ) • p(s, z|ξ 0:t-1 , ξ t ) (32)

The first probability in (32) will be independent of ξ 0:t-1 , since the next internal state s and observation output z of the system depend on the past history only through the previous internal state s, observation output z and control alternative prescribed by decision rule ξ t in ξ 0:t . Moreover, the second probability in (32) will be independent of ξ t , since control alternatives do not impact past internal states or observation outputs of the system. Then, expression (32) becomes:

θ t (
where p ξ t (a|z) denotes the probability of taking control alternative a after perceiving observation output z when following decision rule ξ t . The remaining two probabilities in (34) are just transition probabilities and observation probabilities for the process, while the first probability is just the information-state vector defined as follows:

p(s = s t-1 , z = z t-1 |ξ 0:t-1 ) = θ t-1 (s, z)

The important feature of ( 35) is that the calculation of the observation-state vector after control interval t requires only θ t-1 , the observation-state vector after control interval t -1; thus, θ t-1 summarizes all the information gained prior to control interval t and represents a sufficient statistic for the complete past history of the process ξ 0:t-1 .

In fact, (35) describes the possible transformations for a continuous-state deterministic Markov process in which the state of the process is the occupancy state vector θ t . For this process, the transition probability from one state to another is deterministic p(θ |θ, ξ t ) = 1 if and only if θ = χ(θ|ξ t ) otherwise p(θ |θ, ξ t ) = 0. This is a rather special case of continuous-state Markov process, since the state is continuous but the transition function is both discrete and deterministic.

Figure 1 :

 1 Figure 1: Decentralized Control of partially observable Markov process.

Figure 2 :

 2 Figure 2: Agent 1's internal partially observable Markov process.

Figure 3 :

 3 Figure 3: The centralized planning for decentralized control.

Figure 4 :

 4 Figure 4: Exponential-size (left-hand side) vs. polynomial-size (right-hand size) deterministic control policies.

Figure 5 :

 5 Figure 5: Information state transformation over {z, z } of the multi-agent tiger problem.

Figure 7 :

 7 Figure 7: The reachable occupancy states.

  Expanding the first probability in (34) over all possible control alternatives a ∈ A yields θ t (s , z ) =

	p(s, z|ξ t )
	s∈S ,z∈Z

s , z ) = s,z p(s , z |s, z, ξ t ) • p(s, z|ξ 0:t-1 )

(33)

Inria a∈A p ξ t (a|z) • p(z |s, a, s ) • p(s |s, a)

  Thus, we haveθ t (s , z ) = p ξ t (a|z) • O(z |s, a, s ) • T (s |s, a)

	θ t-1 (s, z)
	s∈S ,z∈Z

a∈A

RR n°9202

Acknowledgments