Rania Hentati-Ka¤el

Philippe De Peretti
email: philippe.de-peretti@univ-paris1.fr

Detecting Performance Persistence of Hedge Funds : A Runs-Based Analysis

Keywords: Hedge Funds, Runs Tests, Persistence, Clustering JEL classi…cation: G1, G110, C1

In this paper, we use nonparametric runs-based tests to analyze the randomness of returns and the persistence of relative returns of hedge funds. Runs tests are implemented on a universe of hedge extracted from HFR database over the period spanning January 2000 to December 2012. Our …ndings suggest that i) For about 80% of the

Introduction

The hedge funds industry has long been, naively, seen as being able to generate "all weather" positive returns, no matter what the market conditions were. Nevertheless, the recent …nancial crisis has cast some doubts on this opinion, leading investors to question whether this industry was signi…cantly able to over-perform the traditional management [START_REF] Gupta | Evaluating hedge fund performance: Traditional versus conditional approaches[END_REF]. The question of over-performances, or equivalently of the persistence of relative returns, is of key importance for investors. Indeed, assessing persistence is a milestone in the decision making process. For instance, one of the main strategies used by investors, e.g. funds of hedge funds strategy, to pick-up top hedge funds, relies on realized relative returns (versus HFR representative strategy index or traditional indices) momentum. Thus, selecting a hedge fund for its ability to signi…cantly over-perform the market during large periods may be a very useful tool.

Persistence has been studied by many authors using various method-ologies 1 as the Cross-Product Ratio (CPR) (DeSouza and Gokcan, 2004), Chi-square tests [START_REF] Carpenter | Survivorship bias and attrition effects in measures of performance persistence[END_REF], regression models [START_REF] Fama | Risk, return and equilibrium: Empirical tests[END_REF]Agarwal and Naik, 2000a), or the test of Hurst [START_REF] Amenc | Predictability in hedge fund returns[END_REF][START_REF] De Souza | Hedge fund investing: A quantitative approach to hedge fund manager selection and de-selection[END_REF][START_REF] Edwards | Hedge fund performance and manager skill[END_REF][START_REF] Eling | Does hedge fund performance persist? Overview and new empirical evidence[END_REF].

Clearly, three conclusions are to be drawn: i) Results vary with both the database (TASS, HFR, Tremont, ...) and the methods ii) Most studies agree to …nd a persistence for a one to a six-month horizon (short-term) [START_REF] Barès | Performance in the hedge funds industry: An analysis of short and long-term persistence[END_REF][START_REF] Boyson | Do hedge funds exhibit performance persistence? A new approach[END_REF][START_REF] Brorsen | Performance persistence and the source of returns for hedge funds[END_REF][START_REF] Herzberg | The Persistence of Hedge Fund Risk: Evidence and Implications for Investors[END_REF], but results are contradictory for longer periods, iii) There is no agreement whether the persistence is related to the nature of the strategy of the hedge fund.

The goal of this paper is to re-examine the questions of persistence ,and randomness of returns for a given hedge fund …rstly in absolute term and then relatively to a set of indices. For both analyses, we use the HFR data base, with a universe spanning more than 4000 hedge over the period spanning January 2000 to December 2012. Relative returns are computed using a blend of traditional and alternative indices: i) The median of the returns of funds having a common primary strategy, ii) An HFRI index computed for each primary strategy, iii) An overall index for the hedge fund market, and iv) The S&P500 index. Performances of hedge funds are thus analyzed with regard to peer groups, the whole hedge fund universe, and an external market.

To extract information about randomness and persistence, we use tests based on runs [START_REF] Wald | On a test whether two samples are from the same population[END_REF][START_REF] Mendenhall | Mathematical Statistics with Applications[END_REF]; [START_REF] Gibbons | Nonparametric statistical inference[END_REF]. Runs tests are very versatile and powerful tools. Used as two-sided tests, they allow to check for randomness. Used as a one-sided test, they allow to test for randomness against a pre-speci…ed alternative: Either clustering, i.e. persistence, implying the ability for a fund to signi…cantly over (under)-perform a given market, or mixing, i.e. systematically alternating over and under performances.

Our main …ndings suggest that i) Using two-sided tests, about 80% of the studied universe has returns at random, ii) A similar outcome is obtained when relative returns are used, iii) Hedge fund strategies displaying the highest percentage of funds generating clusters are Event-Driven and Relative Value, emphasizing the link between the strategy and the persistence, iv) For the relative returns, results deeply vary with the benchmark. This paper is organized as follows. In Section 2, we details how our series are computed, and introduce runs-based tests. An empirical application is also presented. In Section 3, we implement the tests on the HFR database, and present results in contingency tables crossing the results on runs tests with strategies. On Section 4, we split our sample into two sub-samples, before and after the 2007 crisis, and re-run the tests. Finally Section 5 discusses our main results and concludes.

Runs-based tests

Let fr j it g T t=1 , be an observed track record of T observations of returns for fund i having a main strategy j, j 2 (1; 4); where j = 1 corresponds to Equity Hedge, j = 2 to Event-Driven, j = 3 to Macro, and j = 4 to Relative Value. Now, de…ne fd j it g T t=1 j 2 (1; 4) as follows:

d j it = 8 <
:

1 if r j it b j it ; 0 otherwise. (1)
where b j it is either de…ned as:

b j it = b j i = median (r j i1 ; r j i2 ; :::; r j iT ) 0 ; j 2 (1; 4):

or:

b j it = b j t = median (r j it ; r j kt ; :::; r j lt ) 0 ; j 2 (1; 4); t = 1; :::; T:

b j it = b j t = HF RI j t ; j 2 (1; 4); t = 1; :::; T:

b j it = b t = HF RGI t ; t = 1; :::; T:

b j it = b t = SP 500 t ; t = 1; :::; T:

where: r j it ; r j jt ; :::; r j lt are the returns of funds having a common main strategy j; HF RI j t is a performance index corresponding to the primary strategy j, HF RGI t is the HFRI global performance index at time t, t = 1; :::; T: SP 500 t is the S&P500 index at time t; t = 1; :::; T:; Remark 1 De…nition (2) allows us to analyze the randomness of the series, whereas de…nitions of b j it given by (3) to ( 6) return an information about the relative performance of the fund, i.e. the possible persistence of the returns with regard to a benchmark, indicated by large clusters of 1's or 0's. Using [START_REF] Agarwal | Multi-period performance persistence analysis of hedge funds[END_REF] to compute b j it returns a straightforward information about the location of the return of the fund i in the distribution of the returns of a main strategy,.i.e. if the returns are located in the right (left) tail of the distribution during large periods of times, or is randomly distributed on the right or left tail.

Remark 2 In our opinion, the de…nition of a skilled manager should emphasize its ability to outperform its peers (representative HFRI hedge index), as well as the overall sample (HFRI Global Hedge index). Comparing performance to the overall sample attenuates the selection bias e¤ects (Databases have their own classi…cation criteria which could di¤er from one provider to another).Thus, the second and third benchmarks, (3) and ( 4) are used to study how a fund performs compared to its peers (funds classi…ed in the same class), whereas the fourth, [START_REF] Baquero | Survival, look-ahead bias and the persistence in hedge funds performance[END_REF], is used to study the relative performance of the fund with regard to whole hedge fund sample. The last benchmark, [START_REF] Barès | Performance in the hedge funds industry: An analysis of short and long-term persistence[END_REF], is used as an external reference, to see if funds are able to outperform traditional market (equity in our case).

We next use runs-based tests to analyze the information returned by the d j it s. De…ne a run of one kind of element, say of 1 0 s; as a successions of 1 0 s immediately preceded or followed by at least one 0, or nothing. Let T 1 be the number of 1 0 s and T 0 be the 0 0 s with T 1 + T 0 = T , and let r 1j be the number of runs of 1 0 s of length j and r 0j be the number of runs of 0 0 s of length j:

Let r 1 = P j r 1j be the total number of runs of 1 0 s, and r 0 = P j r 0j the total number of runs of 0 0 s. At last let r = r 1 + r 0 be the total number of runs of both kinds. For instance assuming that fd it g 14 t=1 takes the following values:

1 0 1 0 0 1 1 1 0 1 0 0 0 1 we have r 11 = 4; r 01 = 2; r 12 = 0; r 02 = 1; r 13 = 1; r 03 = 1; r 1 = 5; r 0 = 4 and r = 9.

Testing for randomness amounts to testing if we have either too few runs or two many runs by using a two-sided test, whereas testing for the null of randomness against the alternative of clustering i.e. persistence, amounts to using a one-sided test (focusing on the left tail of the distribution), testing

for too low values of r 1 (or r).

Following [START_REF] Gibbons | Nonparametric statistical inference[END_REF], exact and approximate distributions can be used to test for the null. Concerning the former, using combinatorial, the marginal (exact) distribution function of r 1 is given by:

P (r 1 ) = T 1 1 r 1 1 T 0 +1 r 0 T T 1 (7)
where T 1 1 r 1 1 is a binomial coe¢cient. Similarly, the (exact) distribution function of r is given by:

P (r) = 8 > > < > > : 2 ( T 1 1 1 2 r 1 )( T 0 1 1 2 r 1 ) ( T T 1 )
if r is even,

( T 1 1 1 2 r 1 2 )( T 0 1 1 2 r 3 2 ) ( T T 1 ) + ( T 1 1 1 2 r 3 2 )( T 0 1 1 2 r 1 2 ) ( T T 1 ) if r is odd. ( 8
)
Among many other, Gibbons and Chakraborti (1992) provide tabulations for small values of T 0 and T 1 , i.e. for T 0 T 1 12, such that ( 7) and ( 8) can be used to build one or two-sided tests.

For 'large' values of T 1 and T 0 , i.e. for T 0 > 12 and T 1 > 12 a normal approximation can be used. De…ne the …rst two moments of r 1 and r as:

E[r 1 ] = T 1 (T 0 + 1) T (9)
V [r 1 ] = (T 0 + 1) [2] (T 1 ) [2] T (T ) [2] (10)

where x [a] = x(x 1)(x 2):::(x a + 1);

E[r] = E[r 1 ] + E[r 0 ] = 2T 1 T 0 T + 1 (11)
V [r] = V [r 1 ] + V [r 0 ] + 2cov[r 1 ; r 0 ] = 2T 1 T 0 (2T 1 T 0 T ) T 2 (T 1) (12)
Then, using a continuity correction, the corresponding Z-stats are de…ned as:

Z r 1 = r 1 + 0:5 T 1 (T 0 + 1)T 1 q
(T 0 +1) [2] (T 1 ) [2] T (T ) [2] (13) and:

Z r = r + 0:5 2T 1 T 0 T 1 1 q 2T 1 T 2 (2T 1 T 0 T ) T 2 (T 1) (14)
Thus, for a one-sided test of clustering (r E(r) and/or r 1 E(r 1 )) one is thus to compare ( 13) and ( 14) to a standard normal deviate at . Similarly, if one suspects the series to have a tendency to mix, i.e. having too many runs, the right-tail of the standard normal must be considered and the corresponding statistics are given by:

Z r 1 = r 1 0:5 T 1 (T 0 + 1)T 1 q
(T 0 +1) [2] (T 1 ) [2] T (T ) [2] (15)

and:

Z r = r 0:5 2T 1 T 0 T 1 1 q 2T 1 T 2 (2T 1 T 0 T ) T 2 (T 1) (16)
Two-tailed tests are a combination of the above statistics with =2 as threshold.

In this paper, for T 100, we re-tabulate ( 7) and ( 8), and build signi…cance tests on exact probabilities. For T > 100, we base our tests on the normal approximation. Note, that on simulations we performed, the normal approximation appeared to be quite accurate for T 25 (see also the three …lters we apply, in the next section).

To pick up an exemple, consider three funds (fund #1, fund #2 and fund #3), all having an Equity Hedge strategy, with relative monthly performance Obviously, the bias amplitude varies from one database to another. For example, data providers on hedge funds have their own criteria to include and exclude one fund from their databases. The more criteria come into play the higher is the bias amplitude. For illustration TASS database has a higher survivorship bias than the HFR database and a higher attrition rate, which in turn is due to di¤erent criteria for adding and removing funds. Also, the funds have selection criteria that can be very diverse, and the data provided will not be representative of the same management universe. This is referred to as "selection bias". For instance, HFR doesn't cover managed futures unlike TASS and CISDM.

HFR may be considered to su¤er the least from the biases mentioned below. HFR is a leading data provider on hedge funds. ii) Minimal number of occurrence of failures and successes: We select hedge funds with T 1 > 10 and T 0 > 10,

iii) Relative size : The last …lter concerns the proportion of 10s and 00s, and we apply the condition max(T 1 ; T 0 )=min(T 1 ; T 0 ) 1:5.

The three …lters ensure a minimal number of observations, as a well a balanced number of both events, since what we search is to test for random arrangements.

Although database biases are not directly addressed in this paper, these "rules of thumb" are taken into account to limit their impacts. Also, the use of HFRI sub-strategy indices, which take account of all funds existing at one date, attenuates some biases, principally the survivorship bias.

Testing for randomness of returns

We begin by testing the randomness of the series using two-sided runs tests based on P (r) and P (r 1 ). These runs tests return a crucial information about the predictability of returns, and maybe about the (temporary) e¢ciency of the market. The benchmark is computed as the median of each series. For small samples (T 100) we use exact statistics based on ( 8) and ( 7), and for T > 100 asymptotic ones. Table 3 returns the results of runs tests. Results are twofold:

i) For about 82% of the funds, we fail to reject the null of randomness.

Thus, the vast majority of funds have returns distributed at random.

ii) The row percents, returning the proportions of funds at random or not, within each primary strategy, show that the proportions of funds having non-random returns are highest for Event-Driven (39.15%) and

Relative Value (34.00%).

Re…ning the analysis, we also implement one-sided tests, i.e. randomness against clustering or mixing (one sided tests). We next focus on one-sided tests of randomness, when the alternative is either clustering or mixing.

Analysing relative returns 3.3.1 Relative returns with respect to peers group

We now perform the tests on relative returns calculated either with regard to peers group, using either (3) or (4), i.e. the median of returns at time t for a primary strategy, or the corresponding HFRI index, or globally using (5), i.e. the HFRI Global index.

Table 5 contains the results of runs tests when the benchmark is computed, each period, as the median of the returns of the funds having the same primary strategy. 2878 funds matched the …lters described previously.

Focusing on the number of runs of success based on P (r 1 ), (7), [START_REF] Carpenter | Survivorship bias and attrition effects in measures of performance persistence[END_REF] As a reminder, Relative Value strategy covers a very diverse set of strategies which have all as common goal to take advantage upon ine¢ciencies and opportunities of the market. Arbitrage may be conditioned in some cases by the directional evolution of the market or the realization of a particular event so that the expected convergence is achieved.

A second explanation for this persistence arises from the fact that for certain trading strategies such as convexity trading (gamma trading), arbi-trageurs choose not to systematically adjust their portfolios when the stock price changes and prefer to do rather according to their expectations based on the existence of cycles trends (upward or downward) of the prices. Thus, in this type of strategy, model the convertible bond as a portfolio of options implies less liquidity and therefore it forced the managers in some cases stick with their positions for a certain number of months (that is to say, to redeem the bond).

Moreover, Fixed Income-Sovereign strategy (as de…ned by HFR) includes strategies in which the investment thesis is predicated on realization of a spread between related instruments in which one or multiple components of the spread is a sovereign …xed income instrument. Our results are therefore in line with [START_REF] Eling | Does hedge fund performance persist? Overview and new empirical evidence[END_REF], Brown and Goetzmann (2003) and [START_REF] Harri | Performance persistence and the source of returns for hedge funds[END_REF], who concluded that persistence was related to the kind of hedge fund strategy, and therefore in sharp contrast with Agarwal and Naik (2000b), …nding out that persistence was not related to a primary strategy.

Remark 3 At some degree, HFR classi…cation of hedge funds by primary strategy is relevant and it gives hedge fund allocator with some guidelines on needed e¤ort to pick-up good candidate capable to replicate past returns.

Indeed, hedge fund selection is more challenging for Macro and Equity hedge strategies than Relative value and Event driven. In other words, "top in class" funds tend to maintain their 1 st half ranking when the main strategy is Event driven or Relative value. For an investor, for instance a fund of hedge funds manager targeting to outperform an index representing a certain hedge strategy, or the industry as a whole had better to consider di¤erently the four HFR strategies. Tools needed to select top performing hedge funds should be adapted to the strategy (Event driven/Relative value versus Equity Hedge/Macro).

Relative performance with regard to the S&P500 index

Here, randomness is assessed relative to a traditional equity index, the S&P 500 index. The tested universe is composed of 3378 funds.

Results are given by Table 8 Sector: Technology / Healthcare, Systematic Diversi…ed Discretionary Thematic). These strategies are directional, with a high leverage. Trading ideas of these strategies are generated from the economic environment in general (top-down approach) and the alpha of the manager hold on his real ability to choose the right moment to implement his views (market timing). Thus, the corrections related to imbalances in markets can last several months and thereby explains the presence of trend and persistence in returns.

Remark 5 When a representative index of the most liquid stock in the US is used, then the global picture changes completely. Here, the more liquid strategy (in terms of traded assets) become leading (the reverse of the previous section). This raised the question of "prices manipulation" by hedge fund managers deploying Event driven or Relative value strategies in their portfolio. In their e¤ort to smooth returns, those hedge fund managers use price to re ‡ect their valuation rather than market prices.

Fund of hedge fund manager targeting to beat traditional markets should therefore consider a di¤erent framework in his portfolio construction process.

Macro and Equity hedge strategies require less e¤ort to select good candidate than Event driven and Relative value ones. 9 and10.

At …rst glance, almost all proportions of funds (row percent) clustering on each sub-period are lower compared to the whole period. Also, the impact of the crisis on persistence deeply depends on the type of benchmark considered:

i) It is positive (increased clustering), in the case of a traditional external index (S&P 500), and independently of the strategy ii) It is negative (decreased clustering), when HFR indices are used, except

for Event Driven peer group

The positive e¤ect in i) could be mainly explained by a behavioral change in the industry priority to generate returns. After the …nancial crisis of 2007, hedge funds return objective was indexed in absolute term rather than relative to a traditional benchmark.

In ii), the discrepancy between Event Driven and the rest of strategies may be due to the heterogeneity of the former. The returns spread, between sub-strategies within Event Driven, have been ampli…ed through the crisis.

Apart from Event Driven, the negative impact of the crisis resulted from the returns pro…le similarity of the underlying hedge fund after the crisis.

Conclusion and discussion

In this paper, we have used runs based tests to test for randomness in both returns, and relative performances. Clearly, concerning the former test, more than 80% of the funds have returns distributed at random. For relative performances, few funds have a tendency to cluster. But the key information is that, focusing on row percent, about one third of the …rms having a Relative

Value strategy present clustering, and 30% for the Event-Driven strategy, when one uses the median or indices by classes as benchmarks. Using the HFRI global index as a benchmark, only 12.29 % of the …rms have a tendency to cluster in the their relative performances. Still, at the sub-class index level, hedge funds having a Relative Value strategy have the highest probability to cluster (19.33 %). At last, concerning an external market, given by the S&P500 index, only 11% do cluster, but here, the highest proportion is found within the Macro strategy. At last, the 2007 …nancial crisis has increased the proportion of fund with the Event Driven strategy that do cluster. Our results show that few funds are able to signi…cantly over-perform the market during given periods.

In addidion to that, "Smart money" e¤ect consists in investing in funds that will outperform in the future. We believe that our work could provide smart investors with robust tools to assess its challenging mandate.

There is an avenue for future researches in this area. Once selected the hedge clustering, a natural development would be …nding external factors explaining persistence, thus leading to a possible forecasting of relative returns. Also, of primary importance as the manageurial skills [START_REF] Caglayan | Hedge fund performance and manager skill[END_REF]. The table is to be interpreted as follows: The Frequency returns the number of funds having returns at random or not within each main strategy. The Percent returns the number of funds having returns at random or not, divided by the total number of funds. The Row Percent returns the number of funds having returns at random or not, within a main strategy divided by the total number of funds in this strategy. The Column Percent returns the number of funds having returns at random or not in a main strategy divided by the total number of funds having returns at random or not.

d 1 ;

 1 d 2 and d 3 computed using[START_REF] Amenc | Predictability in hedge fund returns[END_REF].Figure (??) plots the returns of the three funds together with the corresponding HFRI Equity Hedge index. A visual inspection reveals that fund #1 has too many runs, and alternates, too often, successes and failures. This is indicative of a tendency to mix. Conversely, fund #2 has a tendency to produce clusters of both successes and failures, especially after the end of 2007. Turning to statistical analysis, for fund #1, we have r 11 = 17, r 12 = 7, r 13 = 1 and r 15 = 1, r 01 = 15, r 02 = 7, r 03 = 3, r 04 = 1 and therefore r 1 = 26 and r 0 = 26 and r = 52. Since E[r 1 ] = 20:74 and E[r] = 41:44, we have too many runs of lengths 1 and 2 of 1 0 s. This suggests that the arrangements of the 1 0 s and of the 0 0 s are not at random. The (exact) one-sided statistics (right-tail) are P (r 52) = 0:0119 and P (r 1 26) = 0:0168 and the corresponding asymptotic p-values are respectively 0:0121 and 0:0177 . This leads to reject the null, concluding to a tendency to mix for fund #1. Concerning fund #2, we have r 11 = 5, r 12 = 4, r 13 = 2; r 14 = 2 and r 15 = 2. For the runs of 0 0 s we have r 01 = 6, r 02 = 3, r 03 = 2, r 04 = 1 and r 08 = 3, with r = 30 and r 1 = r 0 = 15. The expected values are E[r] = 42:01 and E[r 1 ] = 20:95 suggesting clustering, and the associated exact (one-sided) probabilities are P (r 30) = 0:0048 and P (r 1 15) = 0:0074 (corresponding asymptotic p-values are 0:005 and 0:0078). The null is therefore rejected in favor of clustering. Turning next to the analysis of fund #3, no clear pattern appears in the graph. Turning to the runs analysis, we have r 01 = 15, r 02 = 6, r 03 = 2, r 04 = 2 and r 11 = 13, r 12 = 3, r 13 = 5, r 14 = 1, r 15 = 2, and r 19 = 1 with r 0 = r 1 = 25 and r = 50. With E[r] = 48:69 and E[r 1 ] = 24:52, the exact probabilities are P (r 50) = 0:43 and P (r 1 25) = 0:48. The corresponding two-sided asymptotic tests return a p-value of 0:8663 and 0:9765. Clearly the arrangements are at random. Thus the performance of this fund is not signi…cantly di¤erent from the Equity Hedge index.

4

  Before and after the 2007 crisis Financial crisis started in 2007 has revealed the vulnerability of the hedge fund industry; at epicenter, Relative Value funds group had the highest blowup rate. Has the crisis changed the hedge fund relative returns pro…le? To answer this question, we split into two sub-periods by choosing August 2007 as a breaking point. The results of tests carried out over the 2 periods are displayed in Tables

  Each one has its own biases 2 , and therefore the choice of a database is likely to have some impacts on the results. In this study, we use monthly 3 data from HFR database, over a period spanning

	January 2000 to December 2012. HFR may be considered to su¤er the least
	from the biases mentioned below
	It's well known that database start date a¤ects its indices performance.
	Indeed returns calculation takes into account of only still alive (resulting

3 Implementing runs-based tests 3.1 Data description: The HFR database With the growth of the hedge fund industry, the number of publicly available hedge fund databases has increased. The main data providers are Hedge Fund Research (HFR), Managed Account Reports (MAR) and Tremont Advisory Shareholders Services (TASS).

in a survivorship bias). Thus anteriority of databases is key for statistics relevancy. This is the case of HFR and CISDM database (start in 1994) compared to CSFB (start in 2000).

[START_REF] Brown | O¤shore hedge funds: Survival and performance: 1989-95[END_REF]

valued the average impact of the survivorship bias at 2.6%, compared to 3% for Fung and Hsieh (2000) and 2.43% for

[START_REF] Liang | Hedge fund performance: 1990-1999[END_REF]

.

Table 2

 2

	It covers a higher per-

returns summary statistics for funds having a common primary strategy. At …rst glance, it appears that mean returns are quite insensitive to the HFR classi…cation. Indeed, the four strategies delivered close average return over the period. Di¤erentiation is notable only through moment greater than 1. All distributions are positively skewed with an unexpectedly high kurtosis for Equity Hedge. For a comparison purpose, summary statistics for the S&P500 index is also provided

4

.

Before implementing the tests, the database is …rst …ltered. In particular, three …lters are applied: i) Track record size: We focus on hedge funds having more than 24 months of track record on December 2012, i.e.

T 24,

  Table 4 returns the results.

Clearly, 48.88 % of the Event-Driven funds and 41.14% of the Relative Value have a tendency to cluster, i.e. have positive autocorrelations in their returns. Conversely, very few Macro funds do cluster. Finally, the percent of funds having a tendency to mix is very low. Results are in sharp contrast with

[START_REF] Brooks | The statistical properties of hedge fund returns and their implications for investors[END_REF]

who, working only on indices found out an autocorrelation in returns

[START_REF] Lo | Stock market prices do not follow random walks: Evidence from a simple speci…cation test[END_REF]

.

Table 1 :

 1 Repartition by secondary strategies for Equity Hedge, Event-Driven, Macro and the Relative Value primary strategies

	Main Strategy

Table 2 :

 2 Distribution of returns by kind of primary strategies. The distribution of the S&P500 index is also given.

		Equity Hedge Event-Driven Macro Relative Value S&P500
	Mean	0.72	0.80	0.78	0.77	0.69
	Variance	33.45	19.35	27.83	20.07	19.18
	Skweness	12.73	1.84	0.71	2.89	-0.57
	Kurtosis	1413.27	43.69	20.74	225.36	1.03

Table 3 :

 3 Two-sided tests for randomness of returns. Results based on P (r),

	given at the 5% threshold (Number of hedge funds: 3947)
	Equity Hedge Not Random Random	Total
	Frequency	242	1656	1898
	Percent	6.13	41.96	48.09
	Row percent	12.75	87.25
	Col. percent	33.89	51.22
	Event-driven
	Frequency	157	244	401
	Percent	3.98	6.18	10.16
	Row percent	39.15	60.85
	Col. percent	21.99	7.55
	Macro
	Frequency	77	871	948
	Percent	1.95	22.07	24.02
	Row percent	8.12	91.88
	Col. percent	10.78	26.94
	Relative value
	Frequency	238	462	700
	Percent	6.03	11.71	17.73
	Row percent	34.00	66.00
	Col. percent	33.33	14.29
	Total
	Frequency	714	3233	3947
	Percent	18.09	81.91	100.00

Table 4 :

 4 One-sided tests for randomness of returns against clustering or mixing. Results based on P (r) and P (r 1 ) given at the 5% threshold (Number of hedge funds: 3947)The table is to be interpreted as follows: The Frequency returns the number of funds clustering or mixing within each main strategy. The Percent is the number of funds clustering or mixing divided by the total number of funds. The Row Percent returns the number of funds clustering or mixing within a main strategy, divided by the total number of funds in this strategy. The Column percent returns the number of funds clustering or mixing in a main strategy divided by the total number of funds clustering or mixing.

		Results based on P (r)	Results based on P (r 1 )
	Equity Hedge Clustering	Mixing	Clustering	Mixing	Total 1
	Frequency	353	9	316	7	1898
	Percent	8.94	0.23	8.01	0.18	48.09
	Row percent	18.60	0.47	16.65	0.37
	Col. percent	38.08	19.57	37.00	25.93
	Event-driven
	Frequency	196	0	184	0	401
	Percent	4.97	0.00	4.66	0.00	10.16
	Row percent	48.88	0.00	45.89	0.00
	Col. percent	21.14	0.00	21.55	0.00
	Macro
	Frequency	90	30	76	20	948
	Percent	2.28	0.76	1.93	0.51	24.02
	Row percent	9.49	3.16	8.02	2.11
	Col. percent	9.71	65.22	8.90	74.07
	Relative value
	Frequency	288	7	278	0	700
	Percent	7.30	0.18	7.04	0.00	17.73
	Row percent	41.14	1.00	39.71	0.00
	Col. percent	31.07	15.22	32.55	0.00
	Total
	Frequency	927	46	854	27	3947
	Percent	23.49	1.17	21.64	0.68	100

Table 5

 5

	: One-sided tests for randomness of relative performances against
	clustering or mixing. Relative performances computed using the median of
	returns. Results given at the 5% threshold by primary strategies (Number
	of hedge funds: 2878)
		Results based on P (r)	Results based on P (r 1 )
	Equity Hedge Clustering	Mixing	Clustering	Mixing	Total
	Frequency	212	10	185	8	1473
	Percent	7.37	0.35	6.43	0.28	51.18
	Row percent	14.39	0.68	12.56	0.54
	Col. percent	41.33	38.46	42.33	36.36
	Event-driven
	Frequency	92	2	73	1	270
	Percent	3.20	0.07	2.54	0.03	9.38
	Row percent	34.07	0.74	27.04	0.37
	Col. percent	17.93	7.69	16.70	4.55
	Macro
	Frequency	56	14	42	13	746
	Percent	1.95	0.49	1.46	0.45	25.92
	Row percent	7.51	1.88	5.63	1.74
	Col. percent	10.92	53.85	9.61	59.09
	Relative value
	Frequency	153	0	137	0	389
	Percent	5.32	0.00	4.76	0.00	13.52
	Row percent	39.33	0.00	35.22	0.00
	Col. percent	29.82	0.00	31.35	0.00
	Total
	Frequency	513	26	437	22	2878
	Percent	17.82	0.90	15.18	0.76	100
			29

Table 7 :

 7 One-sided tests for randomness of relative performances against clustering or mixing. Relative performances computed using the HFRI global index. Results given at the 5% threshold by primary strategies (Number of hedge funds: 2985)

		Results based on P (r)	Results based on P (r 1 )
	Equity Hedge Clustering	Mixing	Clustering	Mixing	Total
	Frequency	210	13	175	13	1470
	Percent	7.04	0.44	5.86	0.44	49.25
	Row percent	14.29	0.88	11.90	0.88
	Col. percent	48.84	56.52	47.68	76.42
	Event-driven
	Frequency	38	2	35	0	303
	Percent	1.27	0.07	1.17	0.00	10.15
	Row percent	12.54	0.66	11.55	0.00
	Col. percent	8.84	8.70	9.54	0.00
	Macro
	Frequency	88	7	71	4	767
	Percent	2.95	0.23	2.38	0.13	25.70
	Row percent	11.47	0.91	9.26	0.52
	Col. percent	20.47	30.43	19.35	23.53
	Relative value
	Frequency	94	1	86	0	445
	Percent	3.15	0.03	2.88	0.00	14.91
	Row percent	21.12	0.22	19.33	0.00
	Col. Percent	21.86	4.35	23.43	0.00
	Total
	Frequency	430	23	367	17	2985
	Percent	14.41	0.77	12.29	0.57	100

Table 8 :

 8 One-sided tests for randomness of relative performances against clustering or mixing. Relative performances computed using the S&P500 index. Results given at the 5% threeshold by primary strategies (Number of hedge funds: 2985)

		Results based on P (r)	Results based on P (r 1 )
	Equity Hedge Clustering	Mixing	Clustering	Mixing	Total
	Frequency	233	12	207	13	1652
	Percent	6.90	0.36	6.13	0.38	48.90
	Row percent	14.10	0.73	12.53	0.79
	Col. percent	54.82	52.17	54.91	68.42
	Event-driven
	Frequency	30	4	25	2	366
	Percent	0.89	0.12	0.74	0.06	10.83
	Row percent	8.20	1.09	6.83	0.55
	Col. percent	7.06	17.39	6.63	10.53
	Macro
	Frequency	115	6	104	3	781
	Percent	3.40	0.18	3.08	0.09	23.12
	Row percent	14.72	0.77	13.32	0.38
	Col. percent	27.06	26.09	27.59	15.79
	Relative value
	Frequency	47	1	41	1	579
	Percent	1.39	0.03	1.21	0.03	17.14
	Row percent	8.12	0.17	7.08	0.17
	Col. Percent	11.06	4.35	10.88	5.26
	Total
	Frequency	425	23	377	19	3378
	Percent	12.58	0.68	11.16	0.56	100

Table 10 :

 10 One-sided tests for randomness of relative performances, against clustering and mixing. Tests are performed before and after the 2007 …nancial crisis. Relative returns computed using the S&P500 index. The Results are given at the 5% threshold by primary strategies (Number of hedge funds: 2985)The table is to be interpreted as follows: The Row Percent returns the number of funds having clustering or mixing within a main strategy divided by the total number of funds in this strategy. The Column Percent returns the number of funds clustering or mixing divided by the total number of funds clustering or mixing

		Before the 2007 crisis	After the 2007 crisis
	Equity Hedge Clustering Mixing	Clustering	Mixing
	Row percent	6.75	2.25	9.75	0.76
	Col. percent	50.53	59.26	41.29	50.00
	Event-driven
	Row percent	3.26	2.17	8.09	0.78
	Col. percent	6.32	14.81	7.71	11.54
	Macro
	Row percent	8.31	0.78	18.28	0.98
	Col. percent	33.68	11.11	37.06	30.77
	Relative value
	Row percent	3.86	1.71	8.82	0.31
	Col. percent	9.47	14.81	13.93	7.69
	Total
	Percent	6.27	1.78	11.37	0.74

See also[START_REF] Edwards | Hedge fund performance and manager skill[END_REF],[START_REF] Harri | Performance persistence and the source of returns for hedge funds[END_REF], Brown, Goetzmann and Ibbotson (1999), Kat and Menexe (2003), Koh, Koh and Teo (2003), Baquero, the Hurst and Verbeek (2005), Kouwenberg (2003), Jagannathan et al.(2006).

Some authors use the databases in combination, e.g. Ackermann et al. (1999) and Capocci et al. (2005).

For studies estimating the impact of the di¤erent frequencies on the results, see[START_REF] Harri | Performance persistence and the source of returns for hedge funds[END_REF],[START_REF] Henn | Performance Analysis of Hedge Funds[END_REF] and[START_REF] Koh | Asian hedge funds: Return persistence, style, and fund characteristics[END_REF].

It is well known that biases may cause the underlying indexes to be over-estimated. Some study (e.g.[START_REF] Malkiel | Hedge funds: Risk and return[END_REF]; Posthuma van der Slui, 2004), report a biais of about 4%. To analyze the robustness of the results we re-run the tests exluding the hedge with returns no di¤erent than a margin of plus/minus 4% of the indices. Conclusions are not altered. Results not reported, avalaible under request.


Mehdi El Amine
email: mehdi.el-amine@ensam.eu

Jérôme Pailhes

Nicolas Perry

Critical review of multi-criteria decision aid methods in conceptual design phases: application to the development of a solar collector structure

Keywords: Multi-criteria decision aid methods, Selection methods, Aggregative methods, Innovative design, Conceptual design, Consistency

. In this work, an analysis of six MCDA methods (weighed sum, weighted product, Kim & Lin, compromise programming, TOPSIS, and ELECTRE I) was conducted. Our analysis was performed via an industrial case of solar collector structure development. The objective is to define the most appropriate MCDA methods in term of three criteria: (i) the consistency of the results, (ii) the ease of understanding and, (iii) the adaptation of the decision type. The results show that TOPSIS is the most consistent MCDA method in our case.

Introduction

Decision making is an inherent task in the product development process. It can be broken into (i) conceptual design decisions, when decision makers have to choose between several design concepts, and (ii) embodiment design decisions, when designers have to optimize design variables for the selected concepts. In this paper, we treat only the problem of conceptual design decisions.

Nomenclature xi interpretation values PI performance index

Decision support was first proposed by Nobel Prize winner Herbert Simon in the nineteen sixties [START_REF] Simon | The new science of management decision[END_REF]. In a broad sense, decision making is conducted in four steps: (i) identifying the problem (ii) generating design alternatives, (iii) evaluating design alternatives via evaluation schemes, and (iv) selecting the best alternative. The research community in decision support methods usually recognize that the most critical step in decision making process is how to choose among a given number of design alternatives (step (iv)). The ability of decision makers to make the best choice is strongly conditioned by two factors: (i) having a clear definition of design objectives and requirements, and (ii) being able to evaluate or predict the performance of the proposed alternatives. However, uncertainties and vagueness are inherent in engineering design and they characterize both design objectives and alternatives evaluation schemes. This is particularly true in the first steps of conceptual design. The decisions taken at this design phase have the greatest influence on overall product life cycle cost. In order to minimize risk and reduce the cost of regret in later processes, companies often opt for a least commitment and late design decisions during product development process. This usually implies the development of several design concepts in parallel to finally settle down the optimal solution. Indeed this strategy is effective to minimize risk. It is very important to adopt appropriate methodologies and theories to structure and ease decision making process. This can significantly reduce product development lead-time as well as the amount of human and material resources involved in the development process.

As shown in Fig. 1, multi-criteria decision making process can be decomposed into three principal steps: (i) an observation step, when data are collected about each alternative, (ii) an interpretation step, when decision makers express their preferences for each design criterion on the basis on data collected in the observation step, and (iii) a results analysis step, when decision makers combine interpretation performance of the design alternatives.

In the next section, the MCDA methods analyzed in our study are presented. After that, the industrial case study is presented and examined. Then, in Section 4, the objectives of the study and the proposed approach are presented. Finally, the results and the interpretations are presented and recommendations are given at the conclusion.

Presentation of MCDA methods

In this work, five aggregative methods (weighed sum, weighted product, Kim & Lin, compromise programming, and TOPSIS) and one selection method (ELECTRE I) were considered.

The weighted sum is the most widely used aggregation method. The weights assignment reflects the proportional importance of the different aggregated variables. However, the major disadvantage of this method is that it doesn't satisfy the principle of annihilation, which is generally not acceptable in design decision problems. The index is expressed by: step outcomes for different design criteria in order to determine the best alternative(s).

Because of the great difficulty for decision makers to With [START_REF] Simon | The new science of management decision[END_REF] intuitively follow this process, especially to combine information in appropriate ways (step 3 of decision making process), MCDA methods can be used to aid in multi-criteria decision making.

The preference expression mode used in step (ii) can have an important influence on decision making results [START_REF] Guitouni | Tentative guidelines to help choosing an appropriate MCDA method[END_REF]. The different expression modes used by the MCDA methods are:

The weighted product (WP) provides a conservative aggregative method that satisfies the principle of annihilation. However, the meaning of weights is less intuitive than weighted sum. They reflect the exponential relative importance, and not proportional between variables. The index is expressed by: direct rating, pairwise comparisons, and lotteries. Each mode has its advantages and disadvantages. There are three factors to consider when choosing the preference expression mode: (i) it must be adapted to the type of information available on the With [START_REF] Nash | Non-cooperative games[END_REF] design alternatives (step 1 in decision making process), (ii) decision makers must feel comfortable when using it, and (iii) the type of yielded information must be adapted to the decision situation.

Various MCDA methods are proposed and adopted by researchers and engineers to support design decision-making in engineering design. Choosing the most suitable MCDA method to compare multiple alternatives is a critical issue because these methods may yield different results for the same problem [START_REF] Zanakis | Multi-attribute decision making: A simulation comparison of select methods[END_REF][START_REF] Guitouni | Tentative guidelines to help choosing an appropriate MCDA method[END_REF][START_REF] Triantaphyllou | Development and evaluation of five fuzzy multiattribute decision-making methods[END_REF]. There are two families of MCDA methods: (i) multi-criteria selection methods, where interpretation step results are taken into account simultaneously to compare alternatives, and (ii) aggregative methods, where interpretation step results are aggregated into a single variable called performance index (PI) that reflect the Kim & Lin [START_REF] Kim | Simultaneous optimization of mechanical properties of steel by maximizing exponential desirability functions[END_REF] is the most conservative aggregative method. It also satisfies the principle of annihilation. However, the biggest disadvantage is that it does not allow any kind of compensation. Its use is thus very limited in engineering design. The performance index is expressed by:

(3) TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) is a method that was developed by Hwang (Hwang and Yoon [START_REF] Chen | Fuzzy Multiple Attribute Decision Making: Methods and Applications[END_REF]) and widely applied by other researchers (Deng et al. [START_REF] Deng | Inter-company comparison using modified TOPSIS with objective weights[END_REF]; Tsaur et al. [START_REF] Tsaur | The evaluation of airline service quality by fuzzy MCDM[END_REF]; Opricovic and Tzeng [START_REF] Opricovic | Multicriteria planning of post-earthquake sustainable reconstruction[END_REF]; Cheng et al. [START_REF] Cheng | An integrated multi-criteria decision analysis and inexact mixed integer linear programming approach for solid waste management[END_REF]; Montanari [START_REF] Montanari | Environmental efficiency analysis for enel thermo-power plants[END_REF]; Tong et al. [START_REF] Tong | Dynamic multiple responses by ideal solution analysis[END_REF]; Tzeng et al. [START_REF] Tzeng | Multi-criteria analysis of alternative fuel buses for public transportation[END_REF], etc.). In conceptual design, TOPSIS has been used in a very limited way [START_REF] Lin | Using AHP and TOPSIS approaches in customer-driven product design process[END_REF] of TOPSIS is that the best alternative should have the shortest distance from the ideal solution and the farthest distance from the negative-ideal solution. The TOPSIS procedure consists of the following steps: (i) calculate the normalized decision matrix, (ii) calculate the weighted normalized decision matrix, (iii) determine the ideal negative-ideal solution, (iv) calculate the separation measures from the ideal and negativeideal solution (respectively and ) by using the ndimensional Euclidean distance, (v) calculate the relative closeness to the ideal solution, and (iii) calculate the performance index of each solution by the following formula:

Presentation of industrial case

A previous work has been performed within a design department developing a collector for a solar thermal power plant with Fresnel mirrors. The main function of the solar collector is to concentrate and redirect solar radiation into the absorber tubes to heat up the transfer medium in absorber tubes. The recovered heat is then used to generate high pressure steam which drives a turbine to produce electricity. The solar collector is composed of a reflecting glass and a metal structure, whose function is to give and maintain reflecting glass shape (Fig. 2). A support solution is performed [START_REF] Kim | Simultaneous optimization of mechanical properties of steel by maximizing exponential desirability functions[END_REF] between the reflecting surface and the metal structure to ensure the connection between the two. Solar collector is

The separation measures from the ideal and negative-ideal solution are calculated by the following formula: driven by a rotation movement in order to pursue the movement of the sun.

In order to maintain a good thermal efficiency of the plant, a high reflection performance is required from the collector. [START_REF] Yager | Generalized OWA aggregation operators[END_REF] This implies that the elastic deformation of the structure must remain as low as possible. A high level of precision must also be considered when manufacturing the collector. [START_REF] Scott | Formalizing Negotiation in Engineering Design[END_REF] The technology of solar thermal power plant with Fresnel mirrors is relatively recent and thus the historical background is limited. This induces a real difficulty when evaluating With : The minimum interpretation value for criterion j : The maximum interpretation value for criterion j design alternatives. The implementation site also has a strong influence on collector design: climatic data, market data, etc. The difficulty is that the choice of implementation sites is not made and there is a variety of possible implementation sites Compromise Programming (CP) was developed by Yu [START_REF] Benayoun | Manual de reference du Programme electre[END_REF] and Zeleny [START_REF] Yu | Multiple Criteria Decision-Making: Concepts, Techniques and Extensions[END_REF]. The basic principle of CP is that the best alternative should have the shortest distance from the ideal solution. The performance index is expressed by: where the conditions may vary significantly.

Reflecting surface (slightly curved) Mirror mounting device [START_REF] Quirante | Development of a trade-off function for robust optimization problems in design engineering[END_REF] Pivot With Supporting structure

Structure supporting mirrors

Where and have the same meaning as in TOPSIS. The parameter p defines the desired type of distance. In this study, the CP method has been applied twice by using the parameter p equal to 1 and 2.

The ELECTRE I is an outranking method that is widely used by decision makers in many fields. It was first developed by Benayoun et al. [START_REF] Benayoun | Manual de reference du Programme electre[END_REF]. The ELECTRE I procedure consists of the following steps: (i) define a concordance index for each pair of alternatives, it represents the sum of weights of attributes for which alternative A is better than B, (ii) define a discordance index for each pair of alternatives. It denotes the absolute difference of this pair of attributes divided by the maximum difference over all pairs, (iii) establish threshold values for the two indices, (iv) generate the set of alternatives that is not outranked by any other alternative. In order to obtain an overall ranking of the alternatives, many threshold values are used in our study and each alternative is ranked according to how many times it belong to the set of nonoutranked alternatives.

In this study, we treat only the preliminary design of the metal structure (see Fig. 2). Three main solutions have been proposed for the metal structure: truss structure (S1), sandwich structure (S2) and tube structure (S3). Compared to the tube and truss structure, the sandwich structure is the most mature solution. Its behavior is well characterized by the company due to the many prototypes that have been made. By contrast, truss and tube structures are new solutions for the company and there are much less mature than the sandwich solution. As the product development progresses, the behavior models used become more and more accurate especially due to prototypes testing. Rectifications are introduced to the three solutions throughout the development process.

The criteria taken into account in this study are: elastic deflection (C1), angular elastic deformation (C2), raw material cost (C3), durability (C4), development time (C5), ease of industrialization (C6), adaptation to the mirror mounting solution (C7), adaptation to the guidance solution (C8). The development process is composed of three major milestones. At each milestone, a set of tasks is planned in order to investigate the performance of each concept. These actions generally consist in: numerical analysis and assign interpretation values, the semantic scale in Table .

Objectives and approach

In the present study, various MCDA methods were tested and analyzed based on the conceptual design phase of a solar

Criteria

context.

The first step of our study is to evaluate weights for the different criteria. It is difficult and risky to directly assign weights to criteria because they are from different natures. In our study, pairwise comparison was used because it is more appropriate to this case. Using the semantic scale in Table .1, the judgment matrix was constructed. The results are given in Table . 2. The matrix normalization method [START_REF] Lebel | Presentation « Prise de décision Multi critères » dans Prise de décision Multi critères[END_REF] was then used to calculate criteria weights from the judgment matrix. In order to limit the inconsistencies that occur when performing pairwise comparisons, the Consistency Ratio (CR) proposed by Saaty [START_REF] Saaty | The analytic hierarchy process: planning, priority setting, resource allocation[END_REF] was used as a guidance to check for consistency. As recommended by Saaty [START_REF] Saaty | The analytic hierarchy process: planning, priority setting, resource allocation[END_REF], a value of 0,1 was used as threshold for CR. Based on the interpretation results (Table . 2) and criteria weights obtained by pairwise comparisons, different MCDA methods were used to rank alternatives. From this survey, MCDA methods were evaluated in terms of three criteria: (i) the consistency of results, (ii) the ease of understanding and, (iii) the adaptation the decision type. After the evaluation of criteria weights, the next step is the evaluation of alternatives against each criterion (the interpretation step). A direct rating was used to evaluate each alternative against each criterion. This preference expression mode was used because it is simple and it yields a cardinal rating. Pairwise comparison is more suitable to deal with qualitative criteria and uncertainty in data. However, it yields ordinal evaluation of alternatives which is not suitable to use with aggregative methods. In order to help decision makers to

The consistency of results

The consistency of results is very important criterion of MCDA methods. It measures the closeness of the result given by the MCDA method to what really corresponds to decision makers' preferences. In our study, this criterion was evaluated by comparing the ranking results given by MCDA methods to the intuitive ranking addressed by decision makers

The ease of understanding

This aspect measures the effort and the time required for the decision makers to understand the MCDA method (assumptions, tradeoffs, and procedures of method). The less time and effort required to understanding the method, the better decision makers will use the method and the more effective the method will be [START_REF] Mahmoud | Comparison of different multicriteria evaluation methods for the Red Bluff diversion dam[END_REF]. Experienced decision makers generally prefer simple, more transparent methods [START_REF] Hobbs | Does choice of multicriteria method matter? An experiment in water resource planning[END_REF]. The MCDA methods were tested by three persons participating in the development project. The methods were then ranked on a scale of 1 to 5 (1 if the method is very difficult to understand and 5 if the method is very easy to understand).

Adaptation to the decision type

It is important that the type of decision given by the MCDA method corresponds to the type of decision expected by decision makers. For example, if the decision makers want to get a cardinal ranking of alternatives, then an outranking method is not appropriate.

Results and interpretation

As shown in Fig. 4, clear discrepancies are observed in the ranking of alternatives between compensatory and non compensatory aggregation methods. All the compensatory aggregation methods (weighted sum, TOPSIS, compromise programming with p=1, 2) yield the following ranking: A1A3A4. On the other hand, the two non-compensatory strategies (Kim & Lin and weighed product) yield the following ranking: A3A1A4. ELECTRE I yields the same ranking as compensatory aggregative methods. decision makers and it was highlighted strongly by TOPSIS and little by the other compensatory aggregative methods. For the other two solutions, the ranking yielded by compensatory aggregative methods is also coherent with the preferences of decision makers. From this observation we can estimate the consistency of evaluated MCDA methods in our case.

It can be remarked that TOPSIS intensifies the relative importance between alternatives compared to other compensatory aggregative methods. This intensification effect can be attributed to two factors: (i) the two reference points (ideal and negative-ideal solutions) are obtained according to the alternatives tested (relation ( 5) and ( 6)), and (ii) this method uses an Euclidian distance to calculate the closeness of alternatives to the reference points.

For the criterion of 'ease of understanding', WS and Kim & Lin were the easiest methods to understand, followed by WP, CP, TOPSIS and finally ELECTRE I which was particularly difficult to understand by decision makers who participate in the survey. Table 4 gives more details about the evaluation of MCDA methods against this criterion.

For the criterion of 'adaptation of the decision type', the most adapted decision type in our case is the cardinal ranking. Ordinal ranking (yielded by ELECTRE I) is not suitable because decision makers need to know how much better an Apart from TOPSIS and Kim & Lin aggregation methods, the performance indexes obtained by the other aggregation methods are very close together. It is thus difficult and risky to draw a conclusion based on these results since a small error in interpretation values could reverse the ranking. Kim & Lin aggregation is not suitable for our case because it does not provide any kind of compensation.

According to the interpretation results (Fig. 3), the truss alternative highly satisfies the criterion of raw material cost which is the most important criterion. In contrast, because of its low maturity, it collects low interpretation values (Fig. 3) for other criteria, in particular for the criterion 'ease of industrialization' which also has a very strong importance in our case. Therefore truss alternative obtains a low performance index by conservative aggregative methods. But the important fact that should be considered here is that the truss structure is a novel and immature alternative compared with the others. The potential of improvement is thus high. That is why this alternative was particularly attractive for decision makers. This statement was intuitively felt by

Conclusion

A comparative study of different MCDA methods was performed in this paper based on an industrial case study of a solar structure development. The study focused on the two main steps of decision making: the interpretation step and result analysis step (Fig. 1). Even if pairwise comparison seems to be more suitable to deal with qualitative criteria, the direct rating was privileged in the interpretation step because it allows having cardinal interpretation values which are much more suitable to use with aggregative methods. Many MCDA methods were then used to rank alternatives. It was found that the ranking differs between conservative and non-conservative methods.

Generally it was found that the results yielded by compensatory aggregative methods are more consistent. TOPSIS was the most consistent with decision makers' preferences because the results yielded by this method respect very well the decision makers' preferences. However, in the final design phases, it could be more appropriate to use non compensatory aggregation methods because the potential of improvement is low and decision makers cannot afford to take risks.

A further investigation should be done in a future works in order to evaluate more efficiently the performance of the MCDA methods. For example, a sensitivity analysis could be performed in order to evaluate the robustness of these methods with respect to uncertainties in interpretation values or criteria weights. Further industrial cases of product development could also be considered.

Fig. 1 .

 1 Fig. 1. Schematization of the decision making process.

Fig. 2 .

 2 Fig. 2. Schematization of solar collector structure.

C1

  3 was used. The results of the interpretation step are presented in Fig. 2. prototyping/tests. At the end of each milestone, an assessment grid is established to synthesize investigation works of results are discussed by different actors participating in the development process. The development team decides then what concepts should be eliminated and what actions must be done for the next milestone. In this treat the decision making problem at the first milestone.

Fig. 3 .

 3 Fig. 3. Interpretation results

Table . 1

 . Semantic scales used (a) for direct rating; (b) for pairwise comparison.

Table 2 .

 2 Evaluation of criteria weights by pairwise comparisons

	C1	C2	C3	C4	C5	C6	C7	C8

  alternative is over the others in order to be able to make a decision. Aside from ELECTRE I, all the other methods respect this criterion since they yield a cardinal ranking.Another disadvantage of the ELECTRE I method is that it evaluates a criterion even if it has a weight equal to zero. This can mislead the decision maker, since the discordance

	Performance index		S1: Truss	S2: Sandwich	S3: Tube
	0,550
	0,450			0,429		0,471
	0,350	0,345 0,336	0,350 0,335	0,343 0,336 0,322	0,351 0,338 0,311	0,310	calculation does not consider the value of the weights. The use of ELECTRE I is clearly not suitable in our case.
		0,319	0,315	0,286
	0,250			0,286
						0,220

Table 5 .

 5 Evaluation of MCDA methods against the three criteria.

	0,150
	WS	WP	Kim &	CP	CP	TOPSIS
			Lin	(p=1)	(p=2)
	Fig. 4. Ranking of alternatives by each aggregative method.