Michel Berthélemy
email: michel.berthelemy@mines-paristech.fr

Lina Escobar Rangel

Nuclear reactors' construction costs: The role of lead-time, standardization and technological progress

Keywords: nuclear power cost, lead-time learning effect, standardization

This paper provides the first comparative analysis of nuclear reactor construction costs in France and the United States. Studying the cost of nuclear power has often been a challenge, owing to the lack of reliable data sources and heterogeneity between countries, as well as the long time horizon which requires controlling for input prices and structural changes. We build a simultaneous system of equations for overnight costs and construction time (lead-time) to control for endogeneity, using expected demand variation as an instrument. We argue that benefits from nuclear reactor program standardization can arise through short term coordination gains, when the diversity of nuclear reactors' technologies under construction is low, or through long term benefits from learning spillovers from past reactor construction experience, if those spillovers are limited to similar reactors. We find that overnight construction costs benefit directly from learning spillovers but that these spillovers are only significant for nuclear models built by the same Architect-Engineer (A-E). In addition, we show that the standardization of nuclear reactors under construction has an indirect and positive effect on construction costs through a reduction in lead-time, the latter being one of the main drivers of construction costs. Conversely, we also explore the possibility of learning by searching and find that, contrary to other energy technologies, innovation leads to construction costs increases

Introduction

Many countries have asserted their interest in building nuclear power plants either to ensure security of energy supply, meet CO2 emission reduction targets, or both. This is the case for China, the Czech Republic, India, Poland, Turkey, the United Kingdom (UK) and the United States (US) (IAEA, 2012). Nevertheless, uncertainties surrounding construction costs of new nuclear reactors, along with risks associated with changes in the regulatory framework, have raised doubts about the competitiveness of this technology in both developed and developing countries and led to difficulties for the financing of nuclear new-build projects [START_REF] Nuttall | Financing the nuclear renaissance?[END_REF].

The construction costs of nuclear reactors are particularly important for the competitiveness of nuclear power for two reasons. Firstly, nuclear power is a base-load electricity source with a construction time of, on average, 7.4 years in OECD countries (See Table 4 in Section 4). As such, construction costs can represent between 60 to 80% of the levelized cost of nuclear power (IEA, 2010). Secondly, there is still a sense of agreement that building the first reactor of a new design will come with specific fixed costs. In that respect, the rationale for building this first reactor -and sometimes the associated subsidies-is motivated by the assumption that construction costs will decrease as the industry benefits from learning effects These risks and uncertainties are reflected in recent construction experience in OECD countries. For instance, while the initial cost estimate made by the French nuclear utility Electricité de France (EDF) in 2009 for the EPR nuclear reactor in Flamanville (France) was close to e3 billion (i.e. 2000 Ke 2010 /MWe), the latest announcement indicates that the costs may have nearly tripled, up to e8.5 billion (i.e., 5100 Ke 2010 /MWe)1 , and similar costs are expected for the EPR construction in Finland. In turn, other countries such as the UK, which have been considering the adoption of this technology, are reluctant to do so precisely because the costs estimates have increased and there remain large uncertainties about the possibility for EDF to derive learning by doing benefits from its current reactor's construction for future projects.

Despite the significance of construction costs for nuclear power competitive margins, the economic literature has so far failed to provide clear empirical evidence of the determinants of these costs and the existence of learning effects, mainly due to the lack of comparable and reliable data. In particular, data on construction costs for the French nuclear program were only published in 2012 (Cour des Comptes (2012)). Before this date, existing estimates [START_REF] Grubler | The cost of the french nuclear scale-up: A case of negative learning by doing[END_REF], were based on extrapolations of annual investment expenditures of EDF.

Most of the existing econometric studies have used data on US construction costs and attribute the escalation in costs to the increase in complexity of new reactors. Many authors argue that the experience gained by nuclear vendors led to the design of bigger and more complex reactors that took longer leadtimes to construct and required closer regulatory monitoring [START_REF] Komanoff | Power Plant Cost Escalation Nuclear and Coal Cost, Regulation and Economics[END_REF], [START_REF] Zimmerman | Learning effects and the commercialization of new technologies: The case of nuclear power[END_REF], [START_REF] Rothwell | Steam-electric scale economies and construction lead times[END_REF], [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF] and [START_REF] Cooper | Nuclear safety and nuclear economics[END_REF]). In the case of the French nuclear program, [START_REF] Grubler | The cost of the french nuclear scale-up: A case of negative learning by doing[END_REF] argues in favour of a negative learning by doing effect, whereas Escobar-Rangel and Leveque (2012) find evidence of learning within specific reactor models.

It is also generally accepted that the heterogeneity in the nuclear fleet and the multiplicity of vendors and utilities did not create the gains of learning by doing. [START_REF] David | Standardization, diversity and learning: Strategies for the coevolution of technology and industrial capacity[END_REF]argue that the lack of standardization in the nuclear US fleet entailed ≪ ballooning≫ of construction costs, although some positive learning effects are found by [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF] and [START_REF] Mccabe | Principals, agents and the learning curve: The case of steam-electric power plant design and construction[END_REF] for construction projects managed by utilities.

In this paper, we propose the first empirical investigation of the role of standardization and learning opportunities on nuclear reactors' construction costs, using historical cost data from the US and France. This choice is motivated by the fact that these two countries have followed different paths in terms of industrial structure and technological diversity. For instance, while in the US several firms have acted as Architect-Engineer (A-E) and vendors of nuclear reactors, these roles have been the responsibility of the utility EDF and Areva (formerly Framatome) in France, respectively. Similarly, if the two countries have both built Pressurized Water Reactors (PWR), France has implemented fewer technological variations compared to the US. This means that by looking at French and US experience together one can benefit from more heterogeneity in the data in order to derive robust estimates.

Our empirical strategy follows those of [START_REF] Rothwell | Steam-electric scale economies and construction lead times[END_REF] and [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF], where a simultaneous equation model is estimated for construction costs and lead-time using US data. However, our analysis tackles a number of other empirical shortcomings. Firstly, our study allows direct comparison of nuclear reactors' overnight construction costs in the two countries using the access to data 2 on engineering and other related expenditures for French reactors, which are not detailed in the Cour des Comptes (2012) report. Secondly, we use an IV approach, using the expected demand as an instrument for lead-time [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF], and also test the hypothesis that some of the benefits of standardization may have an indirect impact on cost through a reduction in lead-time. Thirdly, we consider two potential benefits of nuclear programs standardization: (i) standardization 2 These data on overnight construction costs have been made available to the authors by EDF. As such they differ slightly from the data available in the Cour des Comptes' report where expenditure regarding engineering work, pre-operating expenses, etc. are presented at an aggregated level can have short term benefits on costs reductions through reduced diversity of designs for reactors under construction, leading to coordination gains; and (ii) standardization may also allow long term benefits through learning by doing spillovers from similar units built previously. In order to capture these spillover effects we differentiate spillovers based on reactor models and A-E firms.

In addition, the literature and policy debate has so far essentially focused on the role of experience through the study of learning by doing effects, in nuclear construction cost reduction. However, considering the importance of public R&D expenditures on nuclear power, an equally important policy question would be the influence of learning by searching. For instance, for many energy technologies, learning by searching has been highlighted as an important driver of energy cost reduction.

In the nuclear power sector, evidence of a positive learning by searching effect has only been found using cost and innovation data from energy economics modelling tools [START_REF] Jamasb | Technical change theory and learning curves: Patterns of progress in electricity generation technologies[END_REF]. In that respect, to the best of our knowledge, there is no existing literature that has looked at this effect using existing cost data. Hence, our study is the first to bring together data on nuclear power overnight construction costs and knowledge capabilities data, using original data from nuclear reactors' costs in the US and in France, and a measure of the stock of knowledge based on patent data.

Our results suggest that standardization of nuclear reactors programs is one of the main factors in limiting costs escalation and takes place at two levels. Firstly, standardization benefits are found to originate from coordination gains induced by the diversity of reactors under construction. However, this effect impacts costs indirectly through a reduction in lead-time, which has a strong and significant impact on costs. This result is also confirmed in other OECD countries with different market structure and technological paths.

Secondly, we find that learning by doing spillovers also relate to some long term benefits of standardization, considering that these spillovers are limited to nuclear models built by the same A-E firm. This highlights the importance of reactor design standardization and the role played by the A-E firm in reducing construction costs increases.

On the other hand, we show that contrary to other energy technologies [START_REF] Erickson | Technological learning and renewable energy costs: implications for us renewable energy policy[END_REF] there is a negative effect of learning by searching on reactors' overnight construction costs. This can be explained by the fact that innovation in nuclear power technologies has been driven by nuclear safety considerations [START_REF] Berthélemy | Innovation, learning by doing opportunities and nuclear reactors' performance[END_REF], leading to safer but more expensive nuclear reactors.

These results suggest paths for future cost reductions through greater standardization of reactor technologies and more emphasis on the role of A-E firms in improving the competitiveness of nuclear power. In parallel, from a policy perspective one may argue that lead-time will play a more important role under a liberalized electricity market, where higher discount rates may apply, meaning that the competitiveness of nuclear power will be more conditional on the standardization of nuclear programs.

The ensuing sections of this paper are organized as follows: in Section 2 we present our research hypotheses along with stylized facts on the development and cost experience of nuclear power in the US and France; Section 3 describes our empirical strategy and the results; Section 4 further investigates international experiences with nuclear power construction using a larger dataset on nuclear power lead-time; and finally, Section 5 concludes.

2 Main hypotheses, data and stylized facts

Main hypotheses on the relation between construction costs and lead-times

The construction of nuclear reactors is a complex process and requires the coordination of several firms, subject to monitoring and regulation from a nuclear safety regulator prior to, during and after the construction stage. Typically, following or not a tender, an electricity generation firm (hereafter the utility) places an order for the construction of a nuclear reactor and selects a specific reactor design offered by a nuclear vendor. This construction is then managed by an A-E firm which supervises the construction and coordinates the multiple firms involved in the project. This includes the constructor, the Nuclear Steam Supply System (NSSS) manufacturer, the turbine manufacturer, as well as a number of subcontractors. The allocation of firms' responsibilities may differ between projects and, for instance, the utility may decide to also be the A-E (as it is the case in France and sometimes in the US).

One consequence of the involvement of multiple firms in the project is that the objective functions of these firms may differ [START_REF] Rothwell | Steam-electric scale economies and construction lead times[END_REF]. In particular, the A-E firm will minimize costs, whereas the utility will aim to maximize the net present value of the project. This means that the lead-time of the project becomes a decision variable for the utility as, for instance, it can decide to spend more on construction costs in order to reduce the construction period and derive revenue sooner.

From an empirical point of view, the construction cost will be determined by these two objective functions and will be jointly determined with lead-time, leading to a simultaneity problem and lead-time to enter into the cost equation. The inclusion of lead-time in the cost equation can be further motivated by the fact that there exists additional fixed costs associated with longer construction periods, for instance, as utilities are generally in charge of project financing and due to immobilized construction equipment and labour force.

Consequently, OLS estimators will be biased. One solution consists of using an instrument variable approach in order to regain consistency. Our empirical strategy follows this approach with national expected demand (EDem i ) of electricity in country c as an instrument for lead-time [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF], considering that future demand impacts the net present value of the project but does not influence current construction costs.

Our baseline model specification follows equations ( 1) and ( 2) where X i is a vector of independent variables which can impact both cost (CT i ) and lead-time (LT i ) and will be further presented in the next sub-section:

CT i = α 0 + α 1 LT i + J j=2 α j X ij + υ i
(1)

LT i = β 0 + β 1 EDem i + J j=2 β j X ij + ǫ i (2)
2.2 Data and hypotheses regarding the effects of standardization and learning opportunities

Data have been collected from a variety of sources. As mentioned in the introduction, overnight construction costs are collected from the Cour des Comptes (2012) report, and adjustments have been made to account for engineering costs using additional data from EDF. For the US, overnight construction costs data have been published in the online Appendix of [START_REF] Koomey | A reactor-level analysis if busbar cost for the us nuclear plants 1970-2005[END_REF].

However, the US costs data are still more detailed as they take place at the reactor level, whereas the French data have been published for pairs of reactors. This can be explained by the fact that the French nuclear program has been organized with the joint construction of two reactors on the same site. We tackle this shortcoming of the data by treating each pair of French reactors as one reactor, with the average capacity of the two and the date when the latest reactor's construction is completed as corresponding variables.

In parallel, data on nuclear reactor technical characteristics are collected from the IAEA Power Reactor Information System (PRIS) database. This covers the size of the reactor in MWe (Cap i ), that can be used to test the existence of economies of scale, the year when the construction of the reactor starts and the year when the construction is completed. We also collect information on nuclear reactor cooling systems and containment structures3 , in order to define different reactor models. Additional data on A-E firms and initial operators are collected for the US from the US Nuclear Regulatory Commission (NRC). For France, these data do not require any specific access as EDF has always acted as the A-E firm and is the sole utility to operate nuclear reactors.

Based on these reactor level data on technology characteristics and industrial structure, we consider three main channels through which the organization of the nuclear industry can reduce cost and/or lead-time: (i) the standardization of reactor models under construction; (ii) learning by doing opportunities from past reactors' construction; and (iii) learning by searching based on the discounted stock of nuclear specific patents

Hypotheses on the role of nuclear programs standardization

A number of efficiency gains may be expected from the standardization of nuclear programs. One aspect of standardization explored in the literature [START_REF] David | Standardization, diversity and learning: Strategies for the coevolution of technology and industrial capacity[END_REF] relates to a trade-off between the ability to learn from diversity in nuclear reactors versus learning from similar models . This dimension of standardization will be covered by the learning opportunities hypotheses, where learning may be conditional on the level of standardization.

In parallel, one may also argue that standardization benefits can arise from the diversity of nuclear reactors under-construction. We expect that a country with low diversity in models of nuclear reactors under-construction could benefit from coordination gains during the construction period. This can be motivated by the fact that similar high-tech components will be built during the same period, such as steam generators or turbines, leading to economies of scale. Similarly, a low diversity in reactors under-construction also lowers technological uncertainty. To measure this potential benefit of standardization during the construction period, we compute a Herfindahl Hirschman Index (HHI) index based on the number of specific reactor models under-construction when the construction of reactor starts.

This index is defined according to equation (3) as the sum of the squares of the market shares of the M reactor models under-construction in country c and year t. To capture this expected benefit of standardization, we also control for the number of reactors under-construction (N P P.U C i ) as, for instance, a high HHI could either correspond to a situation where only one reactor is underconstruction or to a situation where multiple and similar reactors are being built.

HHI c,t = M m=1 s 2 mtc (3)

Hypotheses on learning by doing opportunities

Because nuclear reactors are complex units, the ability to derive learning effects may be conditional to the similarities between reactors models and the A-E which builds reactors. In that respect, we hypothesize that experience spillovers can take place through two main channels: nuclear reactors completed by the same A-E firm (ExpArq i ) and nuclear reactors of the same design completed (ExpM o i ).

Furthermore, spillover might also be conditional on the A-E firm experience with specific nuclear design (ExpArqM o i ). One could argue that this corresponds to the second potential benefit of standardization mentioned above, where the ability to derive learning by doing spillover is conditional on the standardization path followed by the nuclear industry, and this could be interpreted an indirect benefit of reactor standardization [START_REF] David | Standardization, diversity and learning: Strategies for the coevolution of technology and industrial capacity[END_REF]. This more restricted level of learning by doing opportunities is defined following the traditional definition used in the literature (Irwin and Klenow, 1994). For instance, in equation ( 4) we consider that country level experience (ExoC i ) can be disentangle into four level of learning spillovers: (i) The experience of the A-E with the reactor model (ExpArqM o i ), (ii) the experience of the A-E with other models (ExpArqN oM o i ), (iii), the experience of other A-Es with the same model (ExpN oArqM o i ) and (iv) the experience of other A-Es with other models (ExpN oArqN oM o i )

ExpC i = ExpArqM o i + ExpArqN oM o i + ExpN oArqM o i + ExpN oArqN oM o i (4)

Hypotheses on leaning by searching

In parallel to the learning by doing hypothesis, one standard hypothesis made in the energy economics literature is that learning by doing might not be the only source of learning. In particular, learning by searching is often found in many empirical studies dealing with the energy sector e.g., [START_REF] Larsen | The impact of r&d on innovation for wind energy in denmark, germany and the united kingdom[END_REF]; [START_REF] Erickson | Technological learning and renewable energy costs: implications for us renewable energy policy[END_REF] to contribute to cost reduction. In the case of nuclear power, the impact of innovation activity on cost remains an empirical question. On one hand, there exists evidence [START_REF] Jamasb | Technical change theory and learning curves: Patterns of progress in electricity generation technologies[END_REF] that innovation may contribute to cost reduction. On the other hand, innovation in nuclear power essentially deals with safety improvements because of the role of safety regulation. As such, innovation might lead to safer, but more complex and more expensive nuclear reactors.

As a proxy for nuclear innovation, we rely on a unique dataset on nuclear specific patents, using a discounted stock of priority patent applications. This discounted stock is set at the country level, reflecting the fact that innovation can originate both from R&D laboratories and nuclear vendors and can be understood to reflect national knowledge capabilities. We set the discount factor at 10%, a conservative parameter found in many studies on the dynamics of innovation [START_REF] Peri | Determinants of knowledge flows and their effect on innovation[END_REF].

Instrument and Control variables

As aforementioned, we also use the expected demand of electricity as an instrument for lead-time, using the three year trend in future electricity consumption, and we control for the possibility of structural breaks following two major nuclear accidents: Three Mile Island (TMI) in 1979 and Chernobyl in 1986. Because, the TMI accident took place in the US, we also investigate for the possibility that this accident has had a heterogeneous effect in the two countries, with country specific TMI dummy variables.

Finally, we collect data on prices for two major inputs to the construction of nuclear reactors: cement and labour force. These data are collected from the French National Statistics Institute (INSEE) and the US Census Bureau respectively for the two countries. All the definitions and descriptive statistics for our relevant variables are summarized in Table 1 below.

Stylized facts

Figure 1 below highlights the strong differences between the trend in overnight construction costs in France and in the US. In particular, we observe that over the entire time period the costs have more than doubled in France, from 920e/MWe in 1980 for the Tricastin 3 and 4 reactors up to 2200e/MWe in 2000 for the Chooz 1 and 2 reactors. In the US, this increase has been much more rapid with the cost almost increasing by a factor of 14 from 600e/MWe in 1972 for Turkey Point 3 up to 8500e/MWe in 1989 for the Shoreham reactor.

One can also note that costs have been much more dispersed in the US. For instance, if we look at nuclear reactors completed in 1986 in the US, the costs range from 2000e/MWe for Catawba 2 and 6250e/MWe for the Hope Creek reactor. Since France and the US have experienced important differences in terms of industrial structure choices during this period, with more vertical integration and standardization of nuclear reactor designs for France, this figure provides initial suggestions that the French experience has been more successful in containing the escalation of construction costs. However, to explain the determinants behind construction costs and establish causality one has to develop a structural econometric framework, which we do in the next section. One may also notice that both programs have experienced some important In France, reactors were built essentially during the 1970s and the 1990s. Similarly to the case of the US, only two reactors have been completed since the early 1990s. However, the explanation for the lack of construction of new nuclear reactors might be somewhat different as the share of nuclear power in the French electricity mix had reached 75% by 2000, leading to lower incentives to build new reactors.

3 Model specifications and results: France versus the US

Model specifications

The simultaneous system of equations used to study construction costs and lead-time follows a Cobb-Douglas functional form 4 , taking into account the endogeneity of lead-time using expected demand of electricity as an instrument and controlling for the effects of capacity and input prices. A set of explanatory variables to identify learning effects are included, as well as the HHI index for short term standardization and dummy variables to differentiate projects managed or not by the utility, to capture the effect of structural breaks due to major nuclear accidents and to control for temporal and country fixed-effects.

Based on equations ( 1) and ( 2), the equations for the baseline model specification are as follows:

ln(CT i ) = α 0 + α 1 ln(LT i ) + α 2 ln(Cap i ) + α 3 ln(Cement) + α 4 ln(Labour)+ α 5 ArqU tility i + α 6 ExpArqM o i + α 7 ExpArqN oM o i + α 8 ExpN oArqM o i + α 9 ExpN oArqN oM o i + α 10 HHI.M o i + α 11 N N P.U C+ α 12 T mi.U S + α 13 T mi.F R + α 14 Country + α 15 T rend + υ i (5) ln(LT i ) = β 0 + β 1 ln(Cap i ) + β 2 ArqU tility i + β 3 ExpArqM o i + β 4 ExpArqN oM o i + β 5 ExpN oArqM o i + β 6 ExpN oArqN oM o i + β 7 HHI.M o i + β 8 ln(EDem) + β 9 N P P.U C + β 10 T mi.U S + β 11 T mi.F R + β 12 Country + β 13 T rend + ǫ i (6)
As aforementioned, our empirical strategy follows [START_REF] Rothwell | Steam-electric scale economies and construction lead times[END_REF] structural model which justifies the inclusion of lead-time in the cost equation as an endogenous variable. In his model, the utility chooses the construction leadtime to maximize the net present value of the plant and then the constructor minimizes the costs within this constraint. Moreover, [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF] argue that there are unobserved factors captured in the lead-times that are likely to affect the costs, such as the risks of changes in safety regulations during construction, or potential rise of hiring expenses due to long delays.

Recall that the HHI.M o measures the technological diversity in each country at the moment at which the construction of each reactor began. If this index is close to one, it means that in that year only one type of reactor was being built.

Results

The estimated output for equations ( 5) and ( 6) are presented in Tables 2 and 3 below, using four different model specifications. Estimation output of the cost and lead-time equations in Model 1 represents our baseline estimate.

In this model, we make the hypothesis that learning by doing exists at the A-E firm level and for specific nuclear models. In Model 2, we consider the possibility of learning by searching in addition to standardization and learning by doing. Model 3 focuses on the learning effects at the A-E firm level, and we aggregate the experience of the A-E firm regardless of the reactor model. Finally, Model 4 considers the experience at the reactor model level, regardless of the A-E firm.

The first result of our analysis refers to the importance of model specification in identifying significant learning effects in the construction of nuclear power plants. Previous studies account for the experience at the firm level as in Model 3 or at the technological level as in Model 4. However as we can see in Tables 2 and3, the learning effects are positive and significant only when we take into account solely the experience of A-E firms with specific models of reactor (ExpArqM o i ). After taking into account the effect of ExpArqM o i on lead-time, the point estimate of our baseline model (1) indicates that when this specific experience increases by 1%, costs are reduced by -0.142+1.933*0.009 = 0.124%.

In other words we find that, everything being equal, one can expect on average a 12.4% reduction in construction costs for the second unit of a reactor model built by the same A-E firm. This result is in line with what is expected from the economic literature on learning effects and recent evidence (Escobar-Rangel and Leveque, 2012) on the French nuclear fleet and or confirmed using different model specifications as shown in Appendix A5 . From a policy perspective, it is important to highlight that the benefits that a firm can derive from standardization in terms of reduction in construction costs, after building the first reactor of a series, requires long term commitment precisely because the construction of a nuclear power plant is a lengthy project.

Regarding the learning effects in the lead-time equation, we find that experience in the construction of other models, either of the same firm (ExpArqN oM o i ) or of others (ExpN oArqN oM o i ), has a negative impact on the construction periods which translates into an increase in the construction costs. This result shows that, due to the complexity of a nuclear reactor and the importance of A-E firms in construction projects, it is not possible to directly transfer previous knowledge and experience gained on the construction of any type of reactor to the new projects This detrimental effect of model diversity also leads to short term benefits of standardization. The estimate for the HHI.M o i index suggests that an increase in the diversity of models under construction in a given year (i.e. HHI smaller than 1), increases construction costs indirectly through lead-time, considering that lead-time is found to have a strong and significant impact on costs.

This result can be explained by the fact that when the diversity of nuclear reactors is high, the nuclear safety authority has to assess the potential risks of different models of reactors which prevents rapid monitoring and licensing procedures, due to the heterogeneity in demand which could lead to supply chain constraints and construction delays. As such, it is rational to find that this short term effect impacts primarily the lead-time equation.

Given the nature of these three results, one may argue that the lack of standardization harms the competitiveness of nuclear power in two ways. Firstly, it reduces the potential gains in terms of costs savings in the long term, through learning by doing at the firm level. Secondly, it tends to increase the construction lead-times and therefore the construction costs in the short term.

In addition, the results highlight positive and significant economies of scale. Indeed, we find that larger nuclear reactors take longer to build but are also cheaper per MWe. The net effect on cost can be derived from offsetting the direct effect on cost with the indirect effect on lead-time. For instance, Model 1 indicates a net impact of -0.769+(1.933*0.125)= -0.527. This coefficient can be interpreted as an elasticity, meaning that a 10% increase in size reduces construction costs by 5.27%.

With respect to the role of A-E firms, we also show that when a utility takes the A-E firm responsibility, construction costs are lower than when a project is managed by another firm. This result has been identified in previous studies (e.g. [START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF] and [START_REF] Mccabe | Principals, agents and the learning curve: The case of steam-electric power plant design and construction[END_REF]) and it can be understood by the fact that a vertically integrated utility reduces potential asymmetric information problems between the utility and the firms involved in the construction of nuclear reactors, leading to cost reductions.

In Table 2 we also present the results for Model 2, which in addition to the variables in Model 1 includes the discounted stock of priority patent applications (Know i ) in order to capture the effect of innovation on construction costs. The positive estimate found is contradictory to the pattern observed in many energy technologies, such as other renewable energy sources [START_REF] Erickson | Technological learning and renewable energy costs: implications for us renewable energy policy[END_REF]. This can be explained by the fact that innovation has been driven by the requirements of nuclear safety authorities [START_REF] Berthélemy | Innovation, learning by doing opportunities and nuclear reactors' performance[END_REF], leading to improvements in the safety performance of existing reactors [START_REF] Berthélemy | What drives innovation in nuclear power technologies? an empirical study based in patent counts[END_REF].

In other words, this highlights the long term trade-off faced by the nuclear power sector: on the one hand innovation is needed to reduce the externalities associated with nuclear accident risks; on the other hand this innovation hampers the competitiveness of nuclear power through an increase in construction costs.

One can also note that this result is contrary to the initial findings of [START_REF] Jamasb | Technical change theory and learning curves: Patterns of progress in electricity generation technologies[END_REF] who relies on data extracted from energy modelling tools. Hence, from a methodological perspective this result stresses the necessity of looking at real cost data before drawing policy conclusions on energy technologies costs trajectories.

Finally, it is important to analyze the effect of the major nuclear accidents in our system of equations. As we can see in Table 3, the impact on the construction costs both in the US and in France due to TMI and Chernobyl (CH) came indirectly from an increase in lead-time. Logically, TMI primarily impacted the US where this reactor was located and had no significant impact on France. Chernobyl, which took place in the Ukraine, had a positive and significant impact, albeit at the 10% level, on lead-time in the two countries. This result suggests that closer monitoring from nuclear safety authorities following these accidents resulted in delays in the construction of the reactors installed afterwards.

Nuclear reactors lead-time: Insights from other OECD countries

Given the close relationship between the construction costs and leadtime and the importance of the latter in nuclear power future deployment, in this section we further investigate the impact of capacity, experience and standardization on lead-time using a larger dataset on nuclear reactors from 6 OECD countries. Our aim is to gain some insights into the construction of other nuclear fleets for which cost information is not available, and to identify if the results from the US and the French experience also apply to other OECD countries which have followed different paths in terms of technological choices and industrial structure.

The role of lead-time under liberalized electricity markets

As shown in the previous section, the increase in construction lead-times has been one of the main drivers of the capital costs escalation in nuclear power both in the US and France. However, in addition to the effect on construction costs, an increase in lead-times also means a delay in revenues for the investors and an increase in the interim interest rates. This gives lead-time a stronger role in the competitiveness of nuclear power in liberalized electricity markets where revenues for nuclear reactors' operators are derived from wholesale markets and not from rates of return regulation.

In addition, from a microeconomic perspective, private investors tend to employ higher discount rates than those used for public infrastructure. This implies that one might prefer to build a CCGT gas plant that can be planned and built in 2 years and be willing to face the fossil fuel and carbon price risk, instead of waiting more than 7 years (in the best case scenario) to start recovering their investments [START_REF] Mackerron | Nuclear power and the characteristics of 'ordinariness' -the case of uk energy policy[END_REF]. In addition, the uncertainty of longer construction periods increases the difficulty of financing new nuclear capacity in liberalized electricity market [START_REF] Nuttall | Financing the nuclear renaissance?[END_REF]; [START_REF] Kessides | The future of the nuclear industry reconsidered: Risk, uncertainties and continued promise[END_REF] due to possible cost overruns.

Furthermore, in liberalized electricity markets the private option value that carbon-free nuclear power generation could bring, as a hedge against changes in gas or CO2 emissions, will most likely disappear because of the strong correlation between electricity, gas and carbon prices (F Roques and de Neufville, 2006).

Nuclear reactors lead-time can also have some macroeconomic implications, in particular if one also considers the pre-construction period necessary to receive the authorization to build a reactor, which means that a nuclear project can take between 10 to 15 years between the start of the planning process and the end of construction. In this sense, the risk of delays in the construction process might encourage a wait-and-see policy, as by the time the new nuclear power plant starts generating electricity it would be reasonable to expect alternative technologies to have reduced their costs and become more competitive.

Data and Model

The data used are also extracted from the Power Reactor Information System (PRIS) database developed by the IAEA . Similarly to the previous section, lead-time is computed as the difference in years between the construction and grid dates, and commercial reactors in 6 OECD countries are considered: the US, France, Canada, South Korea, Japan and the UK. Note that contrary to the previous section, we can make full use of the French data as, contrary to costs data, lead-time is available at the reactor level.

Figure 4 and Table 4 below highlight that the heterogeneity across our sample is substantial. This feature is understandable given the differences between countries and across years, in terms of labour productivity, regulatory licensing process, stage of development of construction techniques, etc. To capture these effects, we have included in our model a fixed effect for each country as well as a time trend and a quadratic term for the time trend.

In the next table, we present the means for the explanatory variables that we have chosen to use in our regression model. As we can see, differences in the lead-time means between the Western and Asian countries are substantial. For Japan and South Korea the construction of a new reactor took only 4 years Table 4 presents the average of the explanatory variable used in the regression. Differences in the lead-time average between Western and Asian countries are substantial. For Japan and South Korea the construction of a new reactor took only approximately 4 years, whereas in the US or in the UK it took more than twice as long, even when the average size of the reactors was similar. Important heterogeneity can also be observed for short term HHI.M o i and long term ExpArqM o i average level of standardization.

In Table 5 below we present the estimates similar to equation (4.6) in Section 3. We have also included nameplate capacity, electricity demand and the structural break dummies as controls. Two model specifications are considered, the second one introduces time fixed effects. Other robustness tests can also be found in Appendix B6 . These estimates show that increasing the size of the reactor has a positive and significant effect on lead-time. On average we have found an increase of 3% when scaling up by 10%. This result confirms the importance of offsetting the scale effects in the cost equation, as although increasing the size of the reactor means lower costs per MWe, the net effect should take into account the increase in the lead-time.

This model also confirms the insights from the previous section in terms of our HHI diversity index. Recall that high values of this index mean more market concentration, which in our case corresponds to a more standardized nuclear fleet. On the basis of the analysis using the lead-time, there is strong and significant evidence that reducing the diversity of the nuclear fleet is one of the major differences between countries with longer lead-times and those with shorter construction periods.

One can also notice the negative effect of the two major nuclear accidents on the construction lead-time. Both TMI and Chernobyl were found to be significant structural breaks, showing that these events have an influence beyond borders. As expected, the effect of TMI is stronger on the US compared to other countries.

Conclusion

In this paper we study the short and long term benefits of nuclear reactor standardization on the construction costs of nuclear reactors in the US and France between 1966 and 2002 using overnight construction costs data. Short term benefits are defined as the gains based on the diversity of nuclear reactors under construction, whereas long term benefits represent learning by doing spillovers from similar reactors. We build a system of equations to control for endogeneity between costs and lead-time, using the expected demand of electricity as an instrument for lead-time and control for input prices and the possibility of structural breaks following major nuclear accidents.

We show that short term gains from standardization have a positive impact on construction costs through a reduction in lead-time, the latter being one of the main drivers of construction costs in France and the US. This result is also confirmed for a range of other OECD countries with heterogeneous nuclear programs, and can be explained by the fact that the diversity of nuclear reactor models can lead to delays owing to supply line constraints or delays due to increased workload for the nuclear safety regulator. From a policy perspective, as liberalized electricity markets will tend to apply higher discount rates to nuclear new-build projects appraisal, we further argue that standardization of nuclear reactors will be a key criterion for the economic competitiveness of merchant nuclear reactors.

At the same time, we demonstrate that learning by doing spillovers are also conditional on the standardization of nuclear programs, considering that learning by doing spillovers only take place through reactors of the same model built by the same Architect-Engineer (A-E) firm. Regarding the role of the A-E firm, we also show that vertical integration of the utility and the A-E firm reduces construction costs, which can be explained by a reduction in the asymmetric information of the utility regarding costs.

Conversely, if we stress that lead-time contributes to construction costs reduction and has a stronger impact under liberalized electricity markets, this result may change for other determinants of construction costs. For example, we find evidence of economies of scale for construction costs, whereas from an investment perspective C [START_REF] Gollier | Choice of nuclear power investments under price uncertainty: valuing modularity[END_REF] show that Small and Medium Reactors (SMRs) will generate a significant option value when electricity prices are uncertain. In other words, investors may have to trade-off the economic gains associated with economies of scale for large reactors with the option value of SMRs. This calls for more research on the optimal size of nuclear reactors.

In parallel, we also find that the discounted stock of patents in the nuclear industry increases construction costs, reflecting the fact that innovation increases the complexity of nuclear reactors. This result is in direct contrast to the pattern in other energy technologies where technical progress contributes to costs reductions, and can be explained by the importance of safety regulation in the nuclear power sector which improves safety performance [START_REF] Berthélemy | What drives innovation in nuclear power technologies? an empirical study based in patent counts[END_REF], at the expense of increases in construction costs. This result is not without implications for the design of energy economics modelling tools, as the existing literature has shown that certain models' calibration [START_REF] Jamasb | Technical change theory and learning curves: Patterns of progress in electricity generation technologies[END_REF] implicitly assumes that nuclear construction costs benefit from innovation effort. This result highlights the importance of building these models on evidence based on actual cost data.

However, even if our results highlight the benefit of nuclear programs standardization and suggest that innovation effort contributes to costs increases, they do not answer the question of the optimal pace of technological change in nuclear power technologies. In other words, there exists a tradeoff between reductions in costs permitted by standardization and potential gains from adopting new technologies with better operating and safety performance. In addition, using patent data as a measure of innovation captures incremental innovation but fails to consider the possibility of radical technological change. In that respect, nuclear power has been characterized by incremental innovations from initial reactor designs in the 1950s. Radical innovations such as 4th generation of nuclear reactors could, on the other hand, contribute to costs reductions.

It is also important to note that safety regulation can impact construction costs and lead-time through dimensions other than technological change. In particular, the scope of standardization partly depends on the evolution of the safety rules in each country. This is reported to be the case in the US where, according to [START_REF] Cooper | Policy challenges of nuclear reactor construction, cost escalation and crowding out alternatives[END_REF], the increase in safety regulations issued by the NRC grew substantially following TMI (e.g. from three safety guidelines in 1970 to 143 by 1978), limiting the ability of nuclear vendors to standardize nuclear reactors as they had to comply with changes in safety rules.

Hence, the US experience shows that safety regulation can have important consequences on the economic competitiveness of nuclear reactors. In that respect, standardization and safety regulation do not have to be per-se incompatible and one could argue that for a given level of safety effort, nuclear safety regulation should be designed in order to allow nuclear reactors to benefit more from standardization gains. This would necessitate institutional reforms. One possibility would be to reinforce nuclear reactor certification procedures through cooperation between national nuclear authorities, meaning that a nuclear reactor design can be certified jointly in several countries. This would represent a change from the current regulatory framework where each national safety authority issues design certification with its own specification requests and different timeframes.

Appendix A Robustness checks for the model of costs and lead-times

Figure 1 :

 1 Figure 1: Nuclear reactors' overnight construction costs in the US and France

Figure 2 :

 2 Figure 2: Nuclear reactors'construction lead-time in the US and France

Figure 3 :

 3 Figure 3: Nuclear power installed capacity in MWe

!

Figure 4 :

 4 Figure 4: Nuclear reactors' construction lead-time in OCDE countries

Table 1 :

 1 Descriptive statistics

	Variable	Definition		Mean Std. Dev.	Min	Max
	CT	Cost in e 2010 /MWe	2282	1.639	599	8571
	LT	Construction time	8.578	3.507	4.3	23.3
	Cap	Size in MWe		992.390	201.854	478	1472.5
	HHI.M o	Standardization of reac-	0.230	0.171	0.122	1
		tors under-construction
	Know	Discounted stock of nu-	582.51	103.96 326.48	903.39
		clear patents
	ExpArqM o	Experience A-E model	1.695	2.672	0	14
		(# reactors)
	ExpArqN oM o	Experience A-E diff	9.867	13.162	0	54
		model(# reactors)
	ExpN oArqM o	Experience diff A-E	2.921	4.073	0	18
		model (# reactors)
	ExpN oArqN oM o Experience diff A-E diff	27.414	25.731	0	87
		model (# reactors)
	Arq.U tility	Vertical integration A-E	0.382	0.487	0	1
		with utility
	Cement	Cement cost index	88.019	31.571	36.8 186.556
	Labour	Labour cost index	247.568	168.027 87.439 921.968
	EDem	Future electricity de-	.043	.010	.017	.061
		mand (3 year trend)
	N P P.U C	Reactors	under-	42.632	20.747	2	69
		construction

Table 2 :

 2 Estimation output of equations (1) and (2)

			Model 1	Model 2
		Cost	Lead-time	Cost	Lead-time
	ln(Cap i )	-0.769 *** 0.125 ** -0.624 *** 0.125 **
		(0.192)	(0.053)	(0.182)	(0.0531)
	ln(Cement i )	0.126		0.0882
		(0.469)		(0.424)
	ln(Labour i )	-1.375		-0.771
		(0.852)		(0.806)
	ln(LT i )	1.933 ***	1.064 *
		(0.580)		(0.622)
	ExpArqM o i	-0.142 *** 0.009	-0.149 *** 0.009
		(0.038)	(0.011)	(0.034)	(0.011)
	ExpArqN oM o i	0.025	0.026 *** 0.029	0.026 ***
		(0.034)	(0.009)	(0.031)	(0.009)
	ExpN oArqM o i	0.046	0.010	0.038	0.010
		(0.039)	(0.012)	(0.035)	(0.012)
	ExpN oArqN oM o i	-0.068	0.141 *** -0.039	0.141 ***
		(0.096)	(0.017)	(0.087)	(0.017)
	Know i			1.416 ***
				(0.522)
	HHI.M o i	0.454	-0.566 *** 0.374	-0.566 ***
		(0.537)	(0.160)	(0.485)	(0.160)
	N P P.U C i	0.313 *** -0.071 **	0.324 *** -0.071 **
		(0.117)	(0.034)	(0.105)	(0.034)
	ArqU tility i	-0.256 *** 0.009	-0.285 *** 0.009
		(0.093)	(0.028)	(0.085)	(0.028)
	ln(EDem i )		-1.235 ***		-1.235 ***
			(0.113)		(0.113)
	T mi.U S	-0.058	0.272 *** 0.115	0.272 ***
		(0.184)	(0.0431)	(0.179)	(0.043)
	T mi.F R	-0.015	-0.028	-0.064	-0.028
		(0.246)	(0.074)	(0.223)	(0.074)
	CH	-0.077	0.058 *	-0.030	0.058 *
		(0.123)	(0.031)	(0.113)	(0.031)
	Constant	6.420 ** -2.347 *** -4.182	-2.347 ***
		(2.915)	(0.448)	(4.767)	(0.448)
	Country FE	Yes	Yes	Yes	Yes
	Trend + trend 2	Yes	Yes	Yes	Yes
	Obs.	128	128	128	128
	Adj. R 2	0.833	0.955	0.866	0.955

Table 3 :

 3 Estimation output of equations (1) and (2)

			Model 3	Model 4
		Cost	Lead-time	Cost	Lead-time
	ln(Cap i )	-0.680 *** 0.121 ** -0.716 *** 0.102 **
		(0.196)	(0.052)	(0.191)	(0.048)
	ln(Cement i )	0.038		0.070
		(0.480)		(0.457)
	ln(Labour i )	-1.183		-1.071
		(0.883)		(0.722)
	ln(LT i )	1.910 ***	1.808 ***
		(0.605)		(0.499)
	ln(ExpArq)	-0.049	0.030 ***
		(0.037)	(0.010)
	ln.ExpN oArq	-0.068	0.139 ***
		(0.098)	(0.016)
	ln.ExpM o			-0.055	0.018 **
				(0.034)	(0.008)
	ln.ExpN oM o			-0.061	0.164 ***
				(0.093)	(0.016)
	HHI.M o	0.599	-0.575 *** 0.619	-0.231 *
		(0.556)	(0.158)	(0.560)	(0.139)
	ln.N P P.U C	0.332 ***-0.0751 **	0.337 *** -0.076 **
		(0.123)	(0.033)	(0.114)	(0.030)
	Arq.U tility	-0.319 *** 0.012	-0.253 *** -0.010
		(0.095)	(0.027)	(0.082)	(0.022)
	ln.Demand		-1.228 ***		-1.228 ***
			(0.111)		(0.100)
	T mi.U S	-0.102	0.285 *** -0.116	0.293 ***
		(0.193)	(0.041)	(0.198)	(0.038)
	T mi.F R	0.051	-0.020	0.085	-0.228 ***
		(0.255)	(0.073)	(0.276)	(0.069)
	CH	-0.060	0.065 **	0.004	0.046 *
		(0.129)	(0.030)	(0.124)	(0.027)
	Constant	5.281 * -2.295 *** 5.067 * -2.313 ***
		(3.015)	(0.443)	(2.799)	(0.396)
	Country FE	Yes	Yes	Yes	Yes
	Trend + trend 2	Yes	Yes	Yes	Yes
	Obs.	128	128	128	128
	Adj. R 2	0.815	0.955	0.820	0.962

Table 4 :

 4 Mean for the explanatory variables

		Obs.	LT ExpArqM o	HHI	Cap
			(in year) (#reactors) (index) (in MWe)
	France	58	6.45	9.13	0.472	1083
	Canada	22	7.07	2.29	0.361	687
	South Korea	23	4.90	4.04	0.672	895.3
	Japan	50	4.10	3	0.341	919.1
	U.K	17	8.63	4.80	0.861	645.1
	U.S	98	9.27	2.10	0.152	972.4
	All Countries	291	7.41	4.12	0.378	934.4

Table 5 :

 5 Regression results for lead-time with experience and the HHI index

		(1)		(2)
	Variables	(ln LT )		(ln LT )
	HHI.M o i	-0.291	**	-0.472 ***
		(0.135)		(0.182)
	ln Cap i	0.395 ***	0.254 ***
		(0.052)		(0.052)
	ExpArqM o i	0.019		-0.008
		(0.032)		(0.029)
	ln EDem i	-16.970 *** -21.219 ***
		(2.866)		(3.265)
	ln N P P.U C i	-0.020		-0.054
		(0.033)		(0.047)
	T mi.U S	0.432	**	0.439 ***
		(0.044)		(0.062)
	T mi.Abroad	0.139 ***	0.142	**
		(0.054)		(0.061)
	Cherno	0.188 ***	0.214 ***
		(0.029)		(0.027)
	Constant	1.105 ***	1.977
		(0.402)		(0.440)
	Country FE	Yes		Yes
	Time FE	No		Yes
	Trend + Trend 2	Yes		No
	Obs.	286		286
	Adj. R 2	0.840		0.869
	Note: Robust standard errors in parentheses

Table 6 :

 6 Alternative model specifications for Equations (5) and (6)

			Model 5	Model 6
		Cost	Lead-time	Cost	Lead-time
	HHI.M o	1.249 ** -0.400 *** 0.247	-0.444 **
		(0.490)	(0.141)	(0.481)	(0.185)
	ln ExpArqM o	-0.153 *** 0.005
		(0.041)	(0.011)
	ln ExpArqN oM o	0.028	0.024 **
		(0.035)	(0.009)
	ln ExpN oArqM o	0.0461	0.014
		(0.041)	(0.011)
	ln ExpN oArqN oM o	-0.095	0.152 ***
		(0.103)	(0.017)
	Inv.ExpArqM o			0.335 *** -0.025
				(0.067)	(0.027)
	Inv.ExpArqN oM o			-0.097	-0.007
				(0.080)	(0.032)
	Inv.ExpN oArqM o			-0.150 * -0.016
				(0.079)	(0.032)
	Inv.ExpN oArqN oM o			0.181	-0.337 ***
				(0.245)	(0.076)
	ln Know			1.291 **
				(0.598)
	ln Cap	-0.839 *** 0.117 ** -0.609 *** 0.174 ***
		(0.221)	(0.050)	(0.182)	(0.061)
	ln N P P.U C	0.498 *** -0.101 **	0.318 *** -0.040
		(0.182)	(0.043)	(0.105)	(0.040)
	Arq.U tility	-0.255 *** -0.008	-0.292 *** 0.024
		(0.096)	(0.027)	(0.087)	(0.034)
	ln EDem		-1.202 ***		-1.467 ***
			(0.108)		(0.125)
	ln LT	2.270 ***	1.133 *
		(0.820)		(0.686)
	ln Cement	0.392		0.003
		(0.538)		(0.359)
	ln Labour	-2.020		-0.710
		(1.365)		(0.808)
	T mi.U S	-0.055	0.292 *** 0.075	0.300 ***
		(0.197)	(0.041)	(0.177)	(0.049)
	T mi.F R	-0.001	-0.053	-0.184	-0.004
		(0.252)	(0.071)	(0.223)	(0.089)
	CH	-0.107	0.053 *	-0.051	0.053
		(0.140)	(0.030)	(0.122)	(0.036)
	Constant	7.841 * -2.220 *** -3.776	-3.056 ***
		(4.597)	(0.428)	(5.726)	(0.507)
	Country FE	Yes	Yes	Yes	Yes
	Country specific trend + trend 2	Yes	Yes	No	No
	Trend + trend 2	No	No	Yes	Yes
	Obs.	128	128	128	128
	Adj. R 2	0.823	0.960	0.873	0.940

Table 7 :

 7 Alternative model specifications for lead-time in OECD countries

		(1)		(2)
	Variables	(ln LT )		(ln LT )
	HHI.M o	-0.509 ***	-0.458	**
		(0.189)		(0.200)
	ln Cap	0.225 ***	0.240 ***
		(0.051)		(0.052)
	ln ExpArqM o	-0.010
		(0.031)
	ln ExpArqN oM o	0.0411 ***
		(0.013)
	ln ExpN oArqM o	0.0141
		(0.018)
	ln ExpN oArqN oM o	0.080	*
		(0.041)
	Inv.ExpArqM o			0.0514
				(0.111)
	Inv.ExpArqN oM o			0.003
				(0.032)
	Inv.ExpN oArqM o			0.009
				(0.048)
	Inv.ExpN oArqN oM o			-0.238
				(0.163)
	ln EDem	-17.010 *** -21.240 ***
		(3.857)		(3.387)
	T mi.Abroad	0.126	*	0.124	*
		(0.066)		(0.069)
	T mi.U S	0.432 ***	0.448 ***
		(0.060)		(0.062)
	CH	0.214 ***	0.214 ***
		(0.029)		(0.027)
	Constant	2.111 ***	2.542 ***
		(0.450)		(0.576)
	Country FE	Yes		Yes
	Time FE	Yes		Yes
	Obs.	286		286
	Adj. R 2	0.876		0.872
	Note: Robust standard errors in parentheses

EDF Press Release 12/03/2012 (last accessed 10 June

2013): http://press.edf.com/pressreleases/all-press-releases/2012/flamanville-epr-costs-revised-still-on-schedule-93875.html

Data about nuclear reactors cooling system and containment structure are detailed in the Appendix C

This functional form that has been extensively used in the literature on nuclear construction costs e.g.[START_REF] Komanoff | Power Plant Cost Escalation Nuclear and Coal Cost, Regulation and Economics[END_REF];[START_REF] Cantor | The economics of nuclear power: Further evidence on learning, economics of scale and regulatory effects[END_REF] Hewlett (1988), McCabe (1996)

For robustness tests we consider, in Appendix A, country specific time trends (with a quadratic term) in order to control, for instance, for time variant changes in safety regulation in France and the US. We also define the learning by doing variables as 1/(1+X) instead of ln(X) as both model specifications have been used in the literature(Joskow and Rose, 1985). Our results remain unchanged

In Appendix B we consider the four learning spillovers channels used in Section 3 and also define them both as 1/(1 + X) and ln(X). The results remain unchanged

Appendix C Technical characteristics


Bertrand Putigny
email: bertrand.putigny@inria.fr

Benoit Ruelle
email: benoit.ruelle@ipb.fr

Brice Goglin
email: brice.goglin@inria.fr

Analysis of MPI Shared-Memory Communication Performance from a Cache Coherence Perspective

Shared memory MPI communication is an important part of the overall performance of parallel applications. However understanding the behavior of these data transfers is difficult because of the combined complexity of modern memory architectures with multiple levels of caches and complex cache coherence protocols, of MPI implementations, and of application needs.

We analyze shared memory MPI communication from a cache coherence perspective through a new memory model. It captures the memory architecture characteristics with microbenchmarks that exhibit the limitations of the memory accesses involved in the data transfer. We model the performance of intra-node communication without requiring complex analytical models. The advantage of the approach consists in not requiring deep knowledge of rarely documented hardware features such as caching policies or prefetchers that make modeling modern memory subsystems hardly feasible.

Our qualitative analysis based on this result leads to a better understanding of shared memory communication performance for scientific computing. We then discuss some possible optimizations such as buffer reuse order, cache flushing, and nontemporal instructions that could be used by MPI implementers.

I. INTRODUCTION

The performance of MPI communication in parallel scientific applications is often a key criteria for the overall software performance. Communication tuning has often been investigated for achieving better performance. Indeed, most MPI implementations adapt their communication strategies to the underlying architecture and to the operation parameters. For instance processes running of the same node communicate through shared memory instead of through the network interface.

Communication inside nodes usually relies on two memory copies across a shared-memory buffer. These copies involve cache coherence mechanisms that have an important impact on the actual performance of memory transfers. Unfortunately, MPI implementations tune shared memory communication strategies based on metrics that rarely take caches into account, merely by considering their sizes. Tuning of shared memory communication actually requires understanding the performance implications of cache coherence. Apprehending this impact can be cumbersome because modern memory architectures are increasingly complex, with multiple hierarchical levels of shared caches.

We propose a method based on memory microbenchmarks to help understanding this impact through a qualitative analysis. Relying on benchmarks to provide memory hierarchy insights avoids building very complex analytical models that are platform dependent. Our model improves productivity by remaining the same across different platforms. It only captures the low-level (and hidden) details of the architecture during measurements. Based on this model, the knowledge of buffer states in caches, and micro-benchmarks, we show that tuning shared memory communication depending on the cache coherence protocol can be a source of optimization ideas. This can ease prototyping of communication strategies through shared memory.

The remainder of the paper is organized as follows. Section II presents the scope and context of our work. Section III describes how cache coherence is actually involved in shared-memory MPI communication and how our model lets us analyze it. The model is then evaluated in Section IV and it leads us to discuss several possible optimization ideas. Related work is finally described in Section V before concluding.

II. OPTIMIZING INTRA-NODE MPI COMMUNICATION

A. On the Importance of Intra-node Communication

Intra-node communication had been involved in scientific computing long before the rise of multicore processors and many-cores nodes. Dual-processor servers were already considered to have one of the best performance ratios ten years ago. For instance, Thunderbird, one of the last very large clusters based on single-core processors, reached the fifth rank of the Top500 (http://top500.org) in 2005 while using two single-core processors per node.

Since the advent of multicore processors, more than 75% of the Top500 are now clusters of dual-processor nodes with at least 4 cores per processor, making intra-node communication even more critical to the overall performance. Indeed, the increasing importance of locality in modern servers causes most applications to communicate more with their local neighbors. Users are advised to map processes according to their affinities so as to benefit from cache-sharing, intra-node communication or reduced network distance as much as possible.

Most modern parallel applications still communicate through the de facto standard, MPI. And it is expected that a wide amount of data transfer between processes happens inside a single node, or even inside a single processor socket under a shared cache. This makes shared-memory communication increasingly critical to the overall parallel application performance. Moreover, coprocessors such as the Intel Xeon Phi may again emphasize this trend since they can run tens of MPI processes each.

B. Too many Configuration Options

Intra-node communication has been the subject of many research works since the advent of the MPI standard twenty years ago. Multiple data transfers strategies have been proposed [START_REF] Buntinas | Data Transfers between Processes in an SMP System: Performance Study and Application to MPI[END_REF], including relying on the external network interface, on specific network drivers, on custom operating system features [START_REF] Brightwell | SMARTMAP: Operating System Support for Efficient Data Sharing Among Processes on a Multi-Core Processor[END_REF], or on user-level techniques such as shared buffers and pipelining. This was still an active research area recently through platform-independent directcopy mechanims such as LiMIC [START_REF] Chai | Designing An Efficient Kernel-level and User-level Hybrid Approach for MPI Intra-node Communication on Multi-core Systems[END_REF] and KNEM [START_REF] Goglin | KNEM: a Generic and Scalable Kernel-Assisted Intra-node MPI Communication Framework[END_REF], and the inclusion of Cross Memory Attach [START_REF] Yeoh | Cross Memory Attach[END_REF] in the Linux kernel.

One common issue with all these strategies is: How to select the right one? None of them is the best for all communication patterns because small messages, large messages and collective operations do not have the same performance behavior and needs. Hardware characteristics such as the cache size has been used as a basic way to infer message size thresholds for switching between strategies [START_REF] Buntinas | Cache-Efficient, Intranode Large-Message MPI Communication with MPICH2-Nemesis[END_REF], [START_REF] Chai | Designing An Efficient Kernel-level and User-level Hybrid Approach for MPI Intra-node Communication on Multi-core Systems[END_REF], [START_REF] Ma | Locality and Topology aware Intra-node Communication Among Multicore CPUs[END_REF]. However this appeared not to work properly for collective operations where memory contention becomes a critical factor [START_REF] Moreaud | Optimizing MPI Communication within large Multicore nodes with Kernel assistance[END_REF]. Moreover some of these strategies also require the tuning of their internal parameters such as the pipeline chunk size.

Some MPI implementations such as Open MPI offer many configuration options for tuning intra-node communication. However proper tuning requires deep understanding of the implementation (how it actually transfers data), of the application (how it manipulates buffers, whether it needs overlap, etc.), and of the hardware memory architecture behavior (how memory accesses are implemented). It means that most users cannot actually tune this software, and even many developers unless they have all this knowledge.

Most users therefore do not complain that some implementations such as MPICH2 do not offer many explicit tuning options. However users often assume that MPI is properly tuned internally. Unfortunately this is hardly feasible given the aforementioned software and hardware complexity. Most MPI implementations just use hardwired default thresholds that were chosen five years ago. With tens of cores and much larger caches in modern platforms, the old defaults are likely far from optimal on todays platforms. Users may therefore complain about the observed performance and competitors may easily find cases where a carefully-tuned case-specific change improves performance.

C. The Need for a Better Understanding of Intra-node Communication

Proper automatic tuning of intra-node MPI communication strategy is very difficult because it depends on many factors: Is the transfer running alone on the machine or is it part of a large parallel communication scheme causing contention? Does the application want overlap? Does the hardware efficiently support these needs? Depending on the answers to these questions, the performance of a communication strategy may vary significantly.

We believe that cache coherence is the key to understanding these behaviors. Cache effects are often used as the easy cause of complex behaviors in memory-bound codes, especially shared-memory communication, without actually explaining them for real. Indeed the characteristics of caches (and of the cache coherence protocols that assembles them) is hidden in the hardware and rarely fully documented. Therefore cache coherence causes effects that cannot be easily modeled or even explained. Indeed we show later in this article that even modeling basic data transfers such as memory copies is difficult.

We want to tackle this problem with a new innovative approach that uses micro-benchmarks as a way to capture the complex behavior of the memory architecture. This idea was initially developed for predicting the behavior of memorybound applications [?]. We present in the next sections how we use it for analyzing and better understanding sharedmemory-based intra-node MPI communication.

III. HOW CACHE-COHERENCE MATTERS TO

INTRA-NODE MPI COMMUNICATION Shared-memory MPI communication requires memory copies between the memory and caches of two cores. The cache-coherence protocol is therefore heavily involved. Most modern HPC platforms use a protocol that is a variant of MESI [START_REF] Papamarcos | A low-overhead coherence solution for multiprocessors with private cache memories[END_REF]. Each cache line is in one of the following states:

Modified: The cache line has been modified, and the data in memory is consequently stale. The cache holding this cache line is the only one to hold it.

Exclusive: Data in the cache line is not present in any other cache. This cache line is clean (i.e. the version in main memory is the same).

Shared: This cache line is shared with other caches: other caches can also hold the same data. The cache line is clean (memory and other caches hold the same value).

Invalid: This line is not present in this cache. Intel and AMD platforms actually use the MESIF 1 and MOESI2 variants. We detail in this section how the MESI protocol handles shared-memory MPI communication.

A. Anatomy of Intra-node Communication Memory Accesses

Shared-memory MPI communication uses an intermediate buffer that is shared between the sender and receiver processes. The sender process writes the message to the shared buffer before the receiver process reads it. As described on Figure 1, every byte in the transferred message therefore sees the following cache states: 3) The receiver reads the shared-buffer from the sender core. This is a Load Miss from a remote Modified cache-line. The remote cache line gets copied in the local caches and both copies switch to the Shared state (this explain the state before step 2). 4) Finally the receiver writes the data to its receive buffer.

M
If the target buffer was recently used, this is a Store Hit (usually to a local Modified cache-line). Otherwise it is a Store Miss to main memory. Most modern MPI implementations follow this model. MPICH2 [START_REF] Buntinas | Implementation and Shared-Memory Evaluation of MPICH2 over the Nemesis Communication Subsystem[END_REF] and Open MPI [START_REF] Gabriel | Open MPI: Goals, concept, and design of a next generation MPI implementation[END_REF] both allocate one large buffer shared between all local processes. It is then divided into one set of fixed-size buffers (chunks) per sender. It means that each process always reuses the same buffers for all transfers, even toward different destination processes. We will see in Section IV-E that this reuse behavior is actually relevant on recent AMD platforms. Other strategies exist for various kinds of communication (for instance dedicating one larger buffer to each directed connection, etc.), but we will focus on this one when describing our model.

When the message is larger than fixed-size buffers, multiple ones are used and a pipeline protocol makes sure the receiver can read previous buffers while the sender fills the next ones. MPICH2 uses 64 kB cells while Open MPI uses 32 kB fragments 3 by default. As depicted in Table I, this pipelined model means that there may be 4 concurrent mem- 3 Open MPI uses a smaller first fragment so that the receiver can prepare the receiving of the next fragments before they actually arrive. ory accesses during a single transfer: Sender and receiver cores can execute their own copy in parallel. Each copy involves loads and stores that can be executed in parallel by modern cores. We will analyze the actual parallelism in Section IV-A). Figure 2 presents the memory access throughput when the target buffer is not in the local caches (load or store miss) but may be in some other caches in the socket. It shows that the observed read throughput may vary by a factor of 2 while the write throughput may vary by a factor of 2.5 depending on the buffer size and on the caching state. This result reveals that caches and cache-coherence protocols have a deep impact on the application performance. Difference of similar magnitude can be observed in case of cache hits.

B. Impact of the Cache Coherence on Memory Access Performance

Counting the number of cache misses is a widely used technique for analyzing or even predicting performance, either for shared-memory communication [START_REF] Buntinas | Cache-Efficient, Intranode Large-Message MPI Communication with MPICH2-Nemesis[END_REF], [START_REF] Pellegrini | On the Effects of CPU Caches on MPI Point-to-Point Communications[END_REF] or in a more general profiling context [START_REF] Pesterev | Locating Cache Performance Bottlenecks Using Data Profiling[END_REF]. However, the above graphs show that this number cannot actually be used as a reliable source for accurate performance prediction because neglecting the caching state is not acceptable. We intend to take this information into account for a better model and understanding of MPI shared-memory communication.

We explained in the previous section that a transfer is made of 4 memory accesses that are executed in parallel. The figure shows that the performance of each of them is deeply related to the involved cache states. We are now going to explain how to combine them to model the overall transfer performance.

C. Modeling Communication by Combining Microbenchmarks

The idea behind our memory model is that modern memory architectures make accurate analytical models too difficult because many hardware features, such as prefetchers or cache coherence implementations, are too complex and often not documented. We therefore chose to hide them inside the output of micro-benchmarks that capture the actual behavior of the hardware without having to understand and describe it for real. These benchmarks are run using our mbench framework. 4 It offers easy ways to setup memory buffers to specific cache states and compute the corresponding memory access throughputs for different numbers of threads. Measuring the memory throughput for different buffer sizes also capture the performance and sizes of each level of cache, which allows us to explicitly ignore them in the model.

Benchmark outputs are then combined to rebuild the application memory access pattern and predict its behavior with respect to scalability, buffer sizes, etc. This rebuilding considers involved buffers in the MESI cache-coherence protocol so as to find out the right performance of each buffer access. Given that most HPC architectures use a variant of the MESI protocol, we expect the model to match a wide range of HPC platforms.

The model targets memory-bound applications, i.e. where memory access is the key performance criteria and cannot be overlapped significantly with computation. More details can be found in [?]. We use it here for a better understanding of MPI shared memory communication based on the memory pattern described in section III-A. Each step translates into a benchmark output as listed in Table II. Given a message of size M and a maximal pipeline chunk of size C, there are n = ⌊M/C⌋ chunks of size C i (usually the first and/or last chunks are smaller than C if M is not an exact multiple of C). The overall transfer time is estimated to

T = S(C 1 ) + n i=2 max S(C i ), R(C i-1 ) + R(C n ) (1)
where S and R are the times to copy a chunk on the sender and receiver side respectively. When there is a single chunk, the sender and receiver times are added: the overall time is a sequential aggregation of both sides. When there are many chunks 5 , the first and last terms can be neglected, and the overall duration is the maximum of the sender and receiver copy times.

Finally our model allows us to estimate S and R from our benchmark outputs. Since modern processors can execute one load and one store at the same time, the duration of a copy should be the maximum of the corresponding loads and stores. For instance, copying from a recently generated source buffer to the shared buffer takes the maximum of a Load Hit Modified and a Store Hit Shared duration. Given that the throughput of these operations vary with the buffer size (because of caches), we still have to discuss whether each single chunk is copied at the throughput predicted for its own size (C i ) or for the entire message size (M ). We chose the later because all chunk operations are executed consecutively and therefore cause memory and cache contention just like if the entire message was manipulated at once.

For instance if the source and destination buffers have been used recently, S and R are estimated to

S(C i ) = C i min(LHM (M ), SHS(M )) (2)
R(C j ) = C j min(LM M (M ), SHM (M )) (3)
where LHM , SHS, LM M and SHM are the benchmarkmeasured throughputs for a Load Hit Modified, Store Hit Shared, Load Miss Modified and Store Hit Modified respectively.

IV. PERFORMANCE EVALUATION AND OPTIMIZATION HINTS

We now evaluate our model and use it to discuss some optimization ideas based on the impact of cache-coherence protocols on shared-memory MPI communication. Our evaluation platform is summarized in Figure 3. It consists in two kinds of nodes. The first contains two 8core 2 GHz Intel Xeon E5-2650 processors (Sandy-Bridge micro-architecture, 16 cores total, a single Hyper-Thread used per core). The second kind is made of four 16-core 2.1 GHz AMD Opteron 6272 processors (Bulldozer microarchitecture, 64 cores total). CPU frequency scaling as well as Intel Turbo Boost and AMD Turbo CORE technologies were disabled during tests so that the CPU and memory absolute performance does not vary.

A. Evaluation of the Model

To evaluate the model presented in Section III-C we compare its prediction with the performance of an experiment. However our model only predicts the performance of the actual data transfer while MPI implementations add a lot of control code (such as eager message management, rendezvous messages, synchronization) around it. We therefore designed a synthetic experiment that only mimics the data transfer within the Open MPI 1.7 implementation (32 kB pipeline chunks). The performance behavior is similar, but the synthetic program gets higher performance thanks to the removal of the Open MPI control overhead. Figure 4 presents the performance prediction of the model between 2 cores inside the same Intel socket. The top line is the parallel prediction which means both sender and receiver copies are executed fully in parallel. This is the asymptotic prediction for large messages. The bottom line is the sequential prediction which means copies are performed sequentially by the cores. This is the behavior for small messages when there is a single chunk.

As explained in Section III-C, the prediction model is a mix of these two cases transitioning from one to the other between 32 kB (single chunk) and 4 MB (128 chunks) message sizes. We observed that our model accurately predicts the performance except between 256 kB and 16 MB where the actual experiment is slower. These sizes corresponds to buffers that go into the L3 cache. We explain our misprediction by the fact that the L3 is shared between the two involved cores. It causes contention and capacity misses that our benchmark-based memory model does not really take into account accurately. However, our model works well when the message fits in L1 and L2 cache and in main memory.

One thing that makes our model hard to apply is the difficulty to predict the performance of memory copies that are involved on both sides and accumulated in the analytic formula (S and R functions). Figure 5 presents the prediction of one of the individual memory copies involved in the data transfer. It questions the pre-supposed ability of the processor to perform one load and one store in parallel as explained at the end of Section III-A. Up to 128 kB messages (inside the L1 and L2 private caches), the observed throughput is the parallel bandwidth reduced by 20%. However, for larger messages, in L3 and in main memory, we only measure only 10% above the sequential throughput while the parallel one would be twice higher. Again, this is related to contention in the shared L3 cache and on the memory channels, which do not optimally support heavy parallel loads and stores.

To summarize, our memory model can predict the performance of data transfer, assuming memory copy performance is understood, except when the shared L3 and main memory disturb parallel access performance. This shows why understanding shared-memory communication performance is always difficult: current memory architectures cannot be easily modeled, too many hidden hardware parameters are involved. Overall, we predict the performance behavior but not the absolute value very accurately. Fortunately, this is enough to analyze that behavior and discuss possible optimization hints in the next sections.

B. Impact of Application Buffer Reuse

One common source of mis-understanding of sharedmemory MPI communication performance is the reuse (or not) of application buffers in multiple iterations. As explained in Section III, this changes the involved MESI cache states and causes individual memory access performance to vary significantly. It makes performance comparison meaningless when it is not clear whether the same buffers were reused multiple times. Some benchmarks [START_REF]OSU Micro-Benchmarks[END_REF] always reuse the same buffer while others such as IMB [START_REF]Intel MPI Benchmarks[END_REF] have options to configure/avoid this reuse. We now look deeper at the actual impact of buffer reuse on the overall transfer time. Figure 6 compares the throughput depending on buffer reuse on both sides. We observe that the receiver buffer state is much more important than the sender. Unfortunately this result is not convenient for application tuning because locality is easier to maintain on the send side: the application can usually send the data as soon as it is ready, while it often does not receive exactly when it needs it immediately.

The receiver buffer state is more important than the sender because the receiver-side is slower. Therefore improving the sender locality to improve its transfer side will not significantly improve the overall transfer time. Indeed our micro-benchmarks reveal that the receiver side memory accesses (steps 3 and 4 on Table II) hardly pass 15 GB/s for large messages, while the sender side (steps 1 and 2) often achieves close to 20 GB/s. This imbalance between send and receive side copy durations could be a reason to switch to variable-size pipeline chunks as previously proposed for InfiniBand communication [START_REF] Denis | A high performance superpipeline protocol for infiniband[END_REF]. Unfortunately existing MPI implementations require deep intrusive changes before we could experiment this idea. We now focus on one of the transfer step that matters to the overall performance: when the receiver loads data that was previously written by another core (Load Miss Modified, step 3). The usual problem with this transition in the MESI protocol is that the remotely-modified data has to be written back to memory before it can be shared by both cores (see Figure 7). If a cache is shared between the cores, the writeback is not actually required. If no cache is shared, for instance when processes run on different sockets, the writeback is required, and Figure 8 confirms that it is expensive: An explicit flush of the remote copy increases the local Load Miss Modified by 10% to 100%. One optimization would consist in moving this expensive remote write-back from the receiver load (step 3) back to the sender store (step 2), by anticipating it using one of the following ideas: 1) The sender could explicitly flush these cache-lines, e.g. with clflush x86 instructions. Unfortunately, this severely slows down the sender copy as depicted on Figure 9.

C. Load Miss of Sender-written data

2) The sender could use a larger number of buffers so that the first buffers are automatically evicted when last ones are used. Unfortunately, current processors have very large caches that would require hundreds of buffers for this to work 63) The sender could use non-temporal store instructions to directly reach main memory. This idea has often been considered in the past but very rarely used in production. Figure 9 shows that our custom copy with non-temporal writes is only about 30% slower than the usual copy, so the idea looks indeed interesting. Thus we modified Open MPI to perform a non-temporal store during step 2. However Figure 10 reveals that it actually divides the overall performance by a factor of 2. We could not explain this phenomenon. Unfortunately the behavior of non-temporal instructions with respect to cache-coherence protocol implementations is not widely documented. Still, one has to keep in mind that moving the write-back to the sender-side may have the undesirable effect of moving the bottleneck from the receiver to the sender. It is therefore important to make sure that we do not slow the sender down too much. One idea that we are looking at is to force the write-back only when the sender is waiting for the receiver to progress: Once the sender filled all shared-buffers, it may have to wait until the receiver gives some of them back, it may therefore start manually flushing with clflush in the meantime.

D. Rewrite of Receiver-read data

We now focus on the other critical transition, when the sender writes to a buffer that was previously used (Store Hit Shared). The remote copy has to be invalidated before the local copy can switch from Shared to Modified (see Figure 11). Fortunately some modern processors such as Intel Xeon E5 feature a directory in their L3 cache so as to filter such invalidation requests when it is known that there are no other copies. So a former remote flushing could reduce the overhead. Figure 12 shows that this idea could indeed improve performance by 5-10% for significant buffer size (larger than the 64 kB L1, we do not know why the graph is not smooth for smaller buffers). So one could think of adding some flushing on the receiver side. However, as discussed in the previous section, this would slow down the receiver bottleneck even more.

Another problem to consider here is that flushing instructions such as clflush may also flush lines out of other core caches that are below a higher-level inclusive shared cache, which would further degrade performance. For instance it would flush out all copies inside the entire Intel socket on our platform because the L3 is inclusive. On AMD, only the L2 is mostly-inclusive. This idea should therefore only be considered when the MPI implementation knows for sure that the involved cores do not shared an inclusive cache.

To summarize, optimizing the Store Hit Shared state ( 2) is hardly feasible in the context of the MESI protocol. However we have to revisit this result in next section due to certain characteristics of MESI variant implementations.

E. Shared-buffer Reuse Order and MOESI Protocol

AMD platforms use the MOESI protocol that was (notably) designed to ease sharing of modified data. This feature looks very interesting in our study because step 3 needs to read a remotely modified buffer. MOESI avoids the aforementioned write-back to memory by allowing immediate sharing of these dirty cache-lines with other cores. The original modified lines switch to the new Owned state (that is responsible for doing the write-back to memory eventually) while the shared copies go to Shared state. Unlike MESI where both sender and receiver copies are in the same Shared state after step 3, MOESI therefore introduces an asymmetry. When a new transfer occurs through this shared-buffer, one of these asymmetric copies switches to Modified again during step 2 while the other gets invalidated. Given that the Modified state is similar to Owned (and not to Shared), one would expect that transitioning from Owned to Modified would be at least as quick as transitioning from Shared to Modified. Surprisingly Figure 13 shows the contrary: It is much faster (3x inside a socket, and 4x between sockets) to write to the Shared copy rather than the Owned one. We assume that a write-back always occurs when a cacheline leaves the Owned state and raises a non-documented phenomenon in this MOESI implementation. This unexpected behavior leads to another unexpected result on Figure 14: On our AMD platform, data transfers are faster when shared-buffers are used in alternating direction (5 to 50% faster). This behavior seems very specific to AMD current micro-architecture Bulldozer. Intel nodes and some older AMD hosts (Barcelona micro-architecture) do not show such an asymmetric performance depending on buffer reuse direction 7 . This result confirms the interest of our idea of hiding hardware complexity inside micro-benchmark outputs: Extracting the performance behavior from this black-box is much easier on modern platform that trying to formally understand and model the hardware.

V. RELATED WORK It is well known that tuning MPI implementations is difficult. Many configuration options are available and some of them even target conflicting use cases with respect to point-to-point operations vs collectives, blocking vs nonblocking, caching for intra-node communication, etc. When predicting a good configuration is not feasible, auto-tuning may be used to adapt the software to specific application needs. The OPTO framework [START_REF] Chaarawi | A tool for optimizing runtime parameters of open mpi[END_REF] tests all possible configuration combinations so as to automatically find the best one. Machine learning was also proposed as an alternative method [START_REF] Pellegrini | Optimizing MPI Runtime Parameter Settings by Using Machine Learning[END_REF]. A training tool finds out important characteristics of the platform before matching them with specific application needs.

Our approach is rather a qualitative approach that tries to understand cache-related issues instead of blindly finding the best tuning for specific applications. One common way to evaluate intra-node communication performance is to look at cache misses [START_REF] Pellegrini | On the Effects of CPU Caches on MPI Point-to-Point Communications[END_REF]. However we explained in Section III-B that this is hardly a reliable piece of information. This paper gives some basic optimization hints to application developers while we rather focus on MPI implementers, those that are experts in the problems considered here.

The only work that is really close to our research mostly focuses on Xeon Phi accelerator cards [START_REF] Garea | Modeling Communication in Cache-Coherent SMP Systems -A Case-Study with Xeon Phi[END_REF]. However only 7 Intel nodes actually show a small performance difference as well, possibly because the MESIF protocol also breaks the symmetry between Shared copies (the Forward copy is the only one that replies to bus requests).

synchronization issues (concurrent polling on shared receive queues) and small messages (up to 8 kB) are modeled. Our feeling is that modern memory architectures have a performance that is far too complex for such analytical models because of heavy and hardly-understandable behaviors when switching from L1 to L2, L3 or even main memory, or when looking at parallel accesses. This is why we hide this complexity inside micro-benchmark outputs.

VI. CONCLUSION AND FUTURE WORK

As intra-node communication becomes increasingly important for scientific application performance, the need to better understand and tune these data transfers raises. We believe that modern memory subsystems are too complex to be modeled accurately because of many hardware characteristics such as prefetchers and coherence protocols. We presented a new approach that models shared-memory communication performance by combining the individual throughput of micro-benchmarks based on the knowledge of buffer states in caches and of hardware cache coherence protocols. It allows us to predict the behavior of these transfers with a satisfyingly accuracy. It also shows that memory access parallelism and contention in shared caches and memory buses have complex behavior that explain why shared-memory communication performance is hard to tune and to model analytically.

The model and the predicted behavior was then used as the input for discussing optimization hints for MPI implementations. We identified two individual memory accesses within the overall data transfer that could be modified by adding some preflushing of cache-lines or non-temporal stores. Their limiting factors were described in current hardware cache coherence protocols. We then showed that the MOESI cache-coherence protocol of AMD platforms exhibits unexpected constraints on buffer reuse. It led us to improving communication performance by forcing sharedbuffer reusing in alternating direction.

One drawback of our work is that we have not yet implemented all these ideas in MPI implementations. Some of then were only tested in synthetic benchmarks. The reason is that they require very intrusive changes in the way shared-buffers are managed internally in both Open MPI and MPICH2. We will then experiment these ideas on real applications. One may argue that cache-related optimizations may exhibit different behaviors on benchmarks and real applications. Fortunately our optimization hints target the accesses to the shared buffers without increasing the overall cache pollution, and they do not modify the way application buffers are actually involved. Additionally it should be noted that our memory model already works for several real-world application kernels [START_REF] Putigny | A Benchmark-based Performance Model for Memory-bound HPC Applications[END_REF].

We are looking at other architectures such as ARM processors that use MESI-based cache-coherence protocols. Our model should also be refined to better predict the absolute performance by better understanding memory access parallelism and contention as well as better modeling capacity misses. Then we are looking at more complex communication patterns such as collective operations to see if our parallel benchmarks can explain the behavior of these operations at scale on large nodes. Finally we are trying to model other intra-node communication schemes such as direct-copy between processes through the kernel or copy offload to better understand when each strategy should be enabled.

Figure 2 .

 2 Figure 2. Load miss and store miss access throughput depending on the target buffer size and cache state, inside a 2 GHz 8-core Intel Xeon E5-2650 socket.

  (a) Intel Xeon Sandy Bridge E5-2650. (b) AMD Opteron Bulldozer 6272.

Figure 3 .

 3 Figure 3. One socket of each kind of node in the evaluation platform.

Figure 4 .

 4 Figure 4. Comparison of the benchmark-based prediction model, the sequential model, the parallel model, and the actual shared-memory transfer. Intel platform.

Figure 5 .

 5 Figure 5. Benchmark-based prediction of the receiver-side memory copy performance. The source buffer was recently written by another core (Load Miss Modified) while the destination buffer was recently used locally (Store Hit Modified). Intel platform.

Figure 6 .

 6 Figure 6.Impact of buffer reuse on IMB Pingpong throughput with Open MPI 1.7.3. IMB was modified to support buffer reuse on one side without the other. Intel platform.

Figure 7 .

 7 Figure 7. Anatomy of a Load Miss Modified (step 3) in the MESI protocol.

Figure 8 .

 8 Figure 8. Impact of a flush of modified data on the performance of reading from another core, on the Intel platform.

  SSE Custom memcpy with SSE non-temporal writesCustom memcpy with SSE and clflush

Figure 9 .

 9 Figure 9. Impact of non-temporal stores and manually flushing on the performance of the sender write step 2, on the Intel platform.

Figure 10 .

 10 Figure 10. Impact of non-temporal stores in the sender write step 2 on the performance of IMB pingpong between 2 cores on different sockets, on the Intel platform, with a modified Open MPI 1.7.3.

Figure 11 .

 11 Figure 11. Anatomy of a Store Hit Shared (step 2) in the MESI protocol.

Figure 12 .

 12 Figure 12. Impact of remote flushing on the performance of a local Store Hit Shared on the Intel platform.

Figure 13 .

 13 Figure 13.Store Hit performance depending on Shared, Owned and Modified state, inside a shared L3 cache, on AMD platform.

  8 shared-buffers for both directions Using different sets of 8 shared-buffers in each direction

Figure 14 .

 14 Figure 14. Performance of shared-memory data transfer depending on buffer reuse direction, inside a shared L3 cache, on AMD platform.

  or I Load Hit from a local Modified cache-line. If not available in the local caches anymore (either because it was generated a long time ago or because something else filled the caches in the meantime), this is a Load Miss that goes up to main memory.2) The sender then writes the data to the shared buffer.

	Sender cache		Receiver cache
	Source buffer	Shared buffer	Shared buffer	Destination
			S M S (2) (3)	S I S (2) (3)	M or I
	Load (1)	Store (2)	Load (3)	Store (4)
		Sender Core	Receiver Core
	Figure 1. Cache state transitions for the source, shared and destination
	buffers of both sender and receiver cores during the memory accesses
	involved in a shared-memory-based data transfer.
	1) The sender reads the data from its memory. Tempo-
	ral locality implies that it may have been generated
	(written) recently. If so, this step is a

That buffer was used by prior transfers. It is therefore usually available in the local cache as well as in the cache of another core. This is a Store Hit to a local Shared cache-line. The cache-line gets evicted from the remote caches and goes to the Modified state in the local caches.

Table I MEMORY

 I ACCESS PARALLELISM DURING A PIPELINED TRANSFER WHEN THE MESSAGE IS DIVIDED INTO 3 CHUNKS AND THE PROCESSOR CAN EXECUTE ONE LOAD AND ONE STORE IN PARALLEL.

	Time step	Sender Core	Receiver Core
	1	Load + Store (chunk #1)
	2	Load + Store (chunk #2) Load + Store (chunk #1)
	3	Load + Store (chunk #3) Load + Store (chunk #2)
	4		Load + Store (chunk #3)

Table II TRANSITIONS

 II INVOLVED IN OUR MODEL FOR EACH TRANSFER STEP.

	Step	Core	State transition
	1	Sender	Load Hit Modified if recently generated,
			Load Miss Modified otherwise
	2	Sender	Store Hit Shared
	3	Receiver Load Miss Modified
	4	Receiver Store Hit Modified if recently used,
			Store Miss Modified otherwise

The Forward state reduces the traffic by having a single shared-copy reply to bus requests.

The Owned state allows direct sharing of dirty cache-lines instead of first requiring a write-back to memory. See Section IV-E.

The mbench framework is available for download from https://github. com/bputigny/mbench. Only the most relevant benchmark outputs are included in the paper.

A 1 MB message uses 32 chunks in Open MPI and 16 in MPICH2.

640 and 192 buffers of 32 kB are needed on our Intel and AMD platforms respectively.

VII. ACKNOWLEDGMENTS

This work is supported by the STIC-AmSud SEHLOC project, and by Inria as part of the internship of Benoit Ruelle.