Laurent Bach

Wolff Sandrine

The BETA-EvaRIO impact evaluation method: Towards a bridging approach ?

Keywords:

This paper presents a specific approach in the field of evaluating the effects of publicly supported R&D activities: the BETA approach. Initially developed for application to large technology procurement / agency driven public R&D programmes, it has been adapted to a larger range of programmes supporting science and technology. The BETA approach consists of identifying and retrospectively measuring, at the micro-level via direct interviews, the impact of different types of learning processes triggered by participation in R&D. A further adaptation to the specific case of research infrastructures was conducted in the frame of the EvaRIO project. It also dealt with learning processes from an ex post and micro-level perspective, but covered a wider variety of activities and actors over time, leading to a broader view of effects.

Based on 30 years of impact evaluation experience, we argue that the revised BETA approach, despite its focus on the economic dimension of impacts, is at the intersection between different strands of evaluation research. The dual nature of the BETA approach is examined along several dimensions including an output vs process perspective, project vs organization scope, qualitative vs quantitative investigation, contribution vs attribution issues, private profitability vs public value perspective, etc. We show that the BETA-EvaRIO method can be used to complement other evaluation methodologies and can be considered as bridging among them, or can be seen as a single, specific approach.

Introduction

The last few years has seen numerous proposals for methods to evaluate the various impacts (including economic and societal) of public action in Science and Technology (S&T). This paper presents the BETA approach, a specific method to evaluate the effects of publicly supported R&D activities, which recently has been adapted to additional fields. Despite its focus on the economic dimension of impacts, we argue that the revised approach is at the intersection among different strands of the evaluation research.

The original BETA method was designed to evaluate the economic impact of publicly funded R&D programmes, often broken down into projects. It stressed the importance of the knowledge creation and diffusion processes triggered by those projects, and how project participants reaped their economic advantages. Typical candidates were large technology procurement and/or agency driven R&D programmes, but other public programmes supporting S&T were also covered. In 2013, the method was adapted to the case of Research Infrastructures (RIs) within the frame of the EvaRIO project. EvaRIO was aimed at a comprehensive mapping of the different effects generated by RIs, taking account of some specific dimensions such as variety of actors and activities involved in RIs, infrastructure life cycle, the time dimension, etc. The revised approach still considers learning processes from an ex post and micro-level perspective, but takes account also of additional effects and uses additional types of -mainly quantitative -data.

We discuss some of the main features of the BETA approach and situate it in the evolving Research Impact Assessment (RIA) landscape, focusing particularly on its latest development through EvaRIO. Section 2 presents the history and evolution of the BETA approach and compares the initial (the 'original BETA method' hereafter) to the latest version (the 'BETA-EvaRIO method' hereafter) in terms of economic theory and of evaluation. It also specifies the methodological and empirical characteristics of BETA-EvaRIO. Section 3 examines in more detail what can be considered the dual nature of the BETA approach, that is, its ability to achieve tradeoffs from traditional evaluation dilemmas such as outcome versus process, project versus organization, qualitative versus quantitative method, attribution versus contribution, and issues related to aggregation and additionality. We try to show that, especially in the case of the BETA-EvaRIO method, the BETA approach takes account of former hidden areas of impact and provides a starting point for other approaches and impact pathways. We believe it can be seen linking between different methodologies and as a first step towards a bridging evaluation approach.

Otherwise it can be considered as a unique, singular approach.

2 Evolution, fundamentals and main characteristics of the BETA approach: a brief presentation 2.1 Evolution of the BETA approach: from original BETA method to BETA-EvaRIO

A brief history of the BETA approach

The original BETA method was created and developed by the Bureau d'Economie Théorique et Appliquée in the late 1970s and early 1980s, to evaluate the impact of very large technology procurement/agency driven R&D programmes in areas such as civil nuclear (e.g., CERN) and space (European Space Agency -ESA). During the 1990s, it was applied mainly to collaborative projects funded by the European Union within the EC Framework Programmes in areas such as new materials, industrial technologies and Information Technology (IT). During the late 1990s and in the 2000s, some evaluation studies conducted by other labs -sometimes in cooperation with BETA -relied partly on the same approach for the evaluation of various national programmes such as in Brazil, Canada and Japan, and to evaluate the Eureka programme. The original BETA method was slightly amended and was used extensively for a wide range of other public programmes supporting S&T with a common focus on R&D activities performed by companies (often in collaboration with public labs) and operating on a project basis, that is, according to objectives and resources and time constraints (see Bach and Matt, 2005a for an extensive survey of these different studies).

In 2013, as part of the EU FP7 EvaRIO project, the approach was revised substantially and applied to the special case of RIs, broadly defined as unique facilities, resources or services used by researchers to produce excellent science. EvaRIO was aimed more precisely at developing a method and implementing preliminary tests of its feasibility and interest through case studies.

Although the method is not (yet) fully operational since it has not been applied on a large scale, the methodological developments (including some after the EU project at the request of an RI operator), and the nine RI case studies carried out in biological and medical sciences, are sufficiently advanced to provide some fruitful lessons.

Before explaining why and how the specific features of RIs were not suited to the original BETA method, we need first to describe the main features of this method.

Main principles of the original BETA method

The BETA method of evaluation and its main applications have been described in various reports and scientific publications [START_REF] Bach | Evaluation of the BRITE/EURAM program[END_REF]Bach & Matt 2005a;Georghiou et al. 2002). It emphasizes the role of the indirect effects at the core of the knowledge creation and diffusion processes. Indirect versus direct delineation depends on consideration of the objectives of the different projects within the programme being evaluated. Beyond the direct effects of the R&D activities conducted as part of a project (i.e., its expected results and their application), the indirect effects stem from exploitation of the various types of knowledge acquired by a given actor during these R&D activities (S&T, methods, networks, team building, etc.) and their redeployment outside the scope of objectives of these activities. Figure 1 synthesizes the logical scheme behind effect generation.

Figure 1: The original BETA method

The evaluation approach consisted of retrospective identification via face-to-face interviews, at the micro-level of individual programme participants, of the effects of the different types of learning triggered by R&D participation. The effects considered here include the output as well as the impact for participants only, not the 'second level' impact on the rest of the economy or society. 1

These effects were quantified using, wherever possible, standard sales, added value and costsavings figures (or proxies) and always following low estimation rules. 'Fatherhood coefficients' were used to evaluate the specific influence of the evaluated project. 2 The final quantitative result comprised the sum of all the effects measured (scaled for representative sampling of participants 1 Here the term effects is employed in the context of the BETA approach, where the usual distinction between output, outcome and impact is not strictly defined. More precisely, 'effects' covers output, outcome and the part of impact that solely affects the different categories of the actors being evaluated.

2 In the case of complex situations, a two-step approach was used: first, the influence of a given learning dimension was assessed (e.g. S&T vs network), second, the influence of the project on this parameter was evaluated. when sampling was used instead of the whole population). Qualitative insights were provided as well.. Thus, the BETA method was appropriate for evaluating the economic impacts of S&T policy represented by publicly funded R&D projects and, especially, collaborative projects involving companies, for example.

Adaptation challenges raised by RI impact evaluation

The BETA-EvaRIO method was developed from a different perspective and in a different context.

RIs are specific S&T policy tools, and their evaluation required drastic changes to many features of the BETA method.

According to the EU (EC 2010), 'research infrastructures' refer to facilities, resources or services of a unique or rare nature that are needed by the scientific and technological communities to conduct basic or applied research. They generally require a sizeable investment and entail high operating costs. They include the associated human resources and cover major equipment or group(s) of inter-related instruments, knowledge-based resources (collections, archives, databases) and e-infrastructures (grid computing, communication networks). This diversity of types of resources combines with wide variety in RI missions, user disciplines, location (single-site facilities vs distributed resources) and access (face-to-face vs virtual/remote, open vs selective, etc.), making the building of a single consistent method of impact evaluation fairly problematic.

The many specificities of RIs as S&T policy tools include especially: i) the fact that they are not simply a financial support for R&D projects, but consist of the organizations providing scientific users with excellent and evolving resources, largely of a public good nature; ii) the fact that they have to be considered in the long run (given their long lifespan); and iii) the heterogeneity of the actors participating in the various RI activities of designing, building, operating, using, decommissioning, etc.

A major consequence of these three features is that the evaluation is not confined only to the project, but needs to encompass the organizations performing the activities involved in RI, some of which may be conducted as projects, while others might have longer and indefinite timespans.

Moreover the mix of actors (individuals, groups, networks and communities) and their various roles (builders, operators, users, etc.) are linked to the variety of timescales that need to be taken into account, and the quite complex interactions among them. All this made the learning dynamics difficult to understand and gave rise to the following adaptations to the method:

• reconsideration of the theory underlying the evaluation. More specifically, it meant explicit consideration of the organizational dimension, which led to a transition from simple learning mechanisms (sufficient to assess project impact) to the much deeper notion of actor capacity (required to assess organizational impact);

• since the evaluation boundary is not necessarily established based on pre-defined specifications and duration (as in the case of a funding programme), full implementation of the BETA-EvaRIO method would need to begin by delineating the boundaries to the analysis, that is, the resources, actors and period under consideration, which should comply with the evaluation objectives.

• to cope with the complexity of RIs, the scope of the BETA-EvaRIO method has been broadened to include other effects beyond the direct and indirect effects described above. First, it provides an overall framework which, although inevitably limited in its coverage and theoretical assumptions, has led to a global mapping of possible RI effects and impacts, and indications of methods to evaluate them. Second it proposes new ways to assess a given subset of effects using specific tools and metrics.

A detailed presentation of BETA-EvaRIO effects, methods and metrics is provided in sub-section 2.2, while issues related to the theoretical background and evaluation objectives are discussed below.

Evaluation fundamentals of the BETA-EvaRIO method: Theoretical background and evaluation objectives

The BETA approach is rooted theoretically in the innovation economics and knowledge management literatures and, specifically, in evolutionary theory of innovation and the resourcebased view of the firm. Accordingly, the evaluation focuses on the effects derived from the different forms of learning experienced in the performance of R&D activities, such as publicly sponsored projects (original BETA method), or RI-based activities (e.g. building, operating or using the RI in the BETA-EvaRIO method). The performance of these activities allows the actors to acquire and generate new S&T knowledge and skills, to create/reinforce ties with other actors, modify their organization and work methods, gain reputation, recruit new staff, etc., all dimensions contributing to an increase in the participants' capacity.3

While the original BETA method focused on indirect effects and did not deal explicitly with the notion of capacity, the revised BETA-EvaRIO method aims at measuring variations in capacity.

Exploring and identifying increases in a given actor's capacity is the first challenge in the BETA-EvaRIO evaluation process. To some extent, a change in capacity corresponds to the potential for future impact: when a capacity gain is exploited and valorized by the actor, it leads to an economic effect, which, most of the time, can be expressed in monetary terms.

Distinguishing this capacity dimension is enabled by various theoretical considerations. Firstly, the nature of the capacity must be defined. Following the tradition established by seminal works in knowledge-based approaches in economics and management science [START_REF] Penrose | The Theory of the Growth of the Firm[END_REF][START_REF] Prahalad | The Core Competence of the Corporation[END_REF], Teece et al. 1997, among others), it is possible to identify different levels, although this may appear to be an oversimplification. Capacity comprises: i) resources such as knowledge assets, specific know-how, reputation and relational assets; ii) individual and collective competences reflecting the ability to mobilize, combine and exploit these resources for a given purpose; and iii) dynamic capabilities denoting the ability to build resources/competences and allow them to evolve to adapt to a changing environment. Resources can be generic or not, but competences and dynamic capabilities are mostly specific to a given actor and confer competitive advantage over competitors, especially if they remain uncodified and do not become part of the public 'pool of knowledge'. Also, it can be assumed that the actor who created a particular knowledge asset is best placed to use it.

Secondly, capacity is not destroyed or consumed by its use [START_REF] Prahalad | The Core Competence of the Corporation[END_REF]. Moreover, exploiting a capacity usually helps to maintain and develop it. This is a crucial aspect, related to the very nature of what is generated: information, knowledge, competences, etc. cannot be considered as a stock that diminishes (accumulates) through a measurable out (in) flow. To some extent, capacity exhibits some of the features of a fund (rather than a stock) in Goergescu-Roegen's (1971) terminology: it needs time to be constituted, it requires a flow of maintenance and exhibits some features of indivisibility. Correspondingly, capacity cannot be seen as additions of measurable and strictly separable units.

In the case of RIs, evaluating capacity change is even more complex due to the diversity of the actors and stakeholders and their different time horizons and points of intervention in the RI life cycle. This aspect can be blurred by the evaluator's position, ex-ante versus ex-post. The first circle of RI stakeholders includes RI operators (management, but also RI scientists), RI users (academic vs private), RI suppliers (instrumentation producers among others) and different levels of government (European, national, local and regional). All have a clear interest in implementing RI impact assessment, although sometimes for different reasons. This leads to consideration of another fundamental issue in RIA: the objectives of evaluation.

In relation to evaluation objectives, we need a taxonomy that goes beyond the classical distinction between summative and formative evaluation. In line with [START_REF] Gaunand | How does public agricultural research impact society? A characterization of various patterns[END_REF], we consider that there are three broad families of evaluation objectives: advocacy of the programme or organization under evaluation; internal learning; and responding to beneficiaries' questions. In a complementary way, according to [START_REF] Pichler | Research and Technology Policy Evaluation -Evaluation Standards in Research and Technology Policy[END_REF], evaluation can fulfil multiple functions:

• legitimizing -e.g. justifying the use of public funds;

• informing-providing the public with information on how public funds are being used and to what effect;

• (information oriented) learning-for funding institutions, decision-makers in the fields of technology policy, intermediaries, universities, research institutions, companies, consultants, scientists, etc.;

• steering-to establish policy objectives, planning of measures, etc., for the future;

• mediating-between the competing interests of different players in research and technology policy.

The BETA evaluation methods can cope with any of these goal families. In the RI case, evaluation objectives often are mixed. Stakeholders' questions and needs usually encompass the dimension of advocacy. For instance RI managers can be asked about 'return on investment', quantification of the benefits for users (especially companies), etc. At the same time, implementing BETA-EvaRIO provides an opportunity to learn about impact pathways and ways to improve them.

In general, the BETA approach is retrospective as opposed to prospective (ex post vs ex ante); however, it includes some interim evaluation aspects since it can be implemented as policy is ongoing. It can also be included as part of a monitoring exercise (starting from the beginning of a policy intervention and performed in parallel), but, so far, has not been used in that way. In relation to the focus of evaluation, BETA methods are a type of impact evaluation rather than a type of concept evaluation. However, they include some dimensions of design evaluation and process evaluation, as discussed later.4

To sum up the main points in section 2.1, the BETA original and BETA-EvaRIO methods follow more or less the same objectives and use a similar retrospective and micro-analytical approach based on direct interviews. Both rely on the same quantification principles about the gains from different learning domains (S&T, Organization and Method (O&M), network, etc.). Because of the specificities of the organizational dimension of RI, the theoretical framework needs to take explicit account of the notion of capacity. The content of the methods -the types of actors and effects covered and the measurement tools -also differ and are evolving.

The methodological and operational aspects of the BETA-EvaRIO method are discussed in the second part of Section 2.

Presentation of BETA-EvaRIO:

Focus on specific groups of actors

As a micro-approach, the BETA-EvaRIO evaluation method considers the actors as the entry point, whether to analyse, identify and measure effect generation or to determine the entity(ies) they affect. It is important that only the actors performing the R&D activities can be selected for this approach, since it is those activities that engage them in the learning processes on which the BETA approach focuses. Therefore, the first task in the evaluation is to identify the key actors at different stages in the RI life cycle. Following a simple and recognized classification, we distinguish:

• RI builders, i.e. suppliers for the construction, upgrades, etc., of RI elements;

• RI operators, i.e. the hosting scientific team and support staff that support users;

• RI users, i.e. researchers who:

are temporary visitors to and users of the RI;

have remote access to the RI;

are affiliated to the RI (i.e. hosting scientific team when using the RI for its own research).

Other actors, such as RI designers and those involved in facilities decommissioning, were identified, but EvaRIO was implemented at the level of the three categories listed above. Of course, the borders between the three roles may be blurred, and a given organization may play different roles. These combinations, while making the evaluation exercise more difficult, are related strongly to the nature of the resources provided by the RI (instruments, collections, data, etc.).

Typology of effects

The framework designed within EvaRIO5 distinguishes between four types of effects, which are presented below and applied to the three categories of actors in section 2.2.3.

Increase in RI-based activities: Direct effects

Depending on the role of the actors analysed, the nature of the activities they carry out in relation to the RI (RI-based activities) varies. However, they have in common that they represent a given volume of economic activity for a given actor, whether a builder, operator or user. Moreover, for users, benefiting from the RI resources (equipment, data, services, etc.) to conduct scientific experiments is also valuable: comparative advantage over alternative resources. All these elements correspond to direct effects.

Learning, accumulation of knowledge and changes in actors: Capacity effects

Capacity effects are the core of the evaluation framework, as emphasized in the theoretical rationale (Section 2.1.4). In relation to more operational matters, we propose to adopt a quite simplified and general definition of capacity: knowledge-based assets and the capacity to use them and make them evolve. Five subcategories can be distinguished according to the nature of capacity acquired:

• Science & Technology capacity: S&T knowledge and competences;

• Organization & Methods capacity: knowledge and competences regarding management of activities and organizational change;

• Network capacity: knowledge and competences regarding relations and ties to other actors;

• Reputation capacity: reputation assets;

• Human capital capacity: enlargement, diversification of staff.

The first dimensions correspond to different learning domains: S&T, O&M, network and reputation. The fifth, human capital, is different: it constitutes the base of capacity generation (and of its further exploitation through 'actual effect'). The first four operational categories combine the first two theoretical categories of resources and competences; the fifth category (human capital)

tries to capture the ability to make the other capacities evolve. It relates to dynamic capabilities, but

does not encompass all the dimensions of this complex notion. This is a potential area for further improvement to the BETA-EvaRIO method. Note that the different types of capacity effects are inspired by those in the original BETA method and correspond to the indirect effects categories in the BETA-EvaRIO method (see below).

Exploitation of capacity in two directions, 'on RI' versus 'out of RI', are the two final types of effects in the EvaRIO framework.

Exploitation of change in capacity 1: Effects on performance

A first opportunity for exploiting a capacity gain is another run of the RI-based activity, at a higher level of performance of the actor 'on RI'. The notion of performance can be envisaged in several dimensions (see Section 2.2.4 on metrics), including scientific, technical, managerial, economic, etc.. What matters is that the actors have accumulated experience and are able to perform their actions in a better way in relation to supplying, operating or using the RI.

Exploitation of change in capacity 2: Indirect effects

This fourth type of effects encompasses economic flows resulting from redeployment of the capacity obtained through participation in the RI-based activities. The difference from the previous category is that exploitation of the capacity gain is achieved not via on RI but via out of RI activities. Based on the subcategories of capacity proposed, we obtain that:

• S&T indirect effects include use of scientific expertise, worker's know-how, technology translated into a blueprint, new theories, 'tricks of the trade', etc., developed during the RIbased activities, and applied outside of these activities. This could lead to the design of new or improved products, processes or services (i.e., further activities in the same field), or to new research activities in more or less related domains:

• O&M indirect effects occur when the experience gained through the RI-based activities allows the actor to modify its internal organization and/or to apply new methods within project management, quality management, industrial accounting, etc.;

• Network indirect effects refer to the impact of RI activities on subsequent cooperation among economic actors. Typical cases include the establishment of new business/technical links between RI actors after completion of an initial RI project; collaborations resulting from RI workshops; more generally, any type of collaboration resulting from RI actor intermediation.

• Reputation/label indirect effects refers to the reputation acquired by actors running RI-based activities. Their enhanced image can become a marketing tool.

Figure 1 depicts the articulation between the different types of effects discussed above.

The effects put in context: Application to the three basic actors

The scope of effects is better understood by looking at a simplified description of the process of interactions between RI actors and of related learning and valorization mechanisms. Table 1 below provides an overview of the whole typology of effects adopted in the EvaRIO project. It should be noted that the EvaRIO approach clearly departs from the classical input-output-outcome-impact pathway used in many evaluation approaches, by trying to take side effects into account, and adopt a comprehensive view more adapted to the specific case of RIs. Direct effects correspond to the volume of activity expressed in monetary terms. In the case of operators and suppliers, these effects are very similar to economic effects examined using a standard intersectoral input-output approaches. Here, we refer to the classical effect corresponding to the economic activity generated by the construction and operation of the RI, first at the level of the RI contractors and then at the level of the RI contractors' suppliers, the suppliers of those suppliers, and so on, along the supplier-client chain (e.g., purchase of a new IT system, new equipment, etc.). In the case of users, direct effects correspond to the economic size of the research activities relying on the RI, which provide some insight into the specific advantage provided by the RI (not just in monetary terms). The amount of money injected by the bodies funding the RI (government, EC, private partners, etc.) triggers these effects.

As emphasized above, capacity effects are at the heart of the process, whatever the type of actor considered. They concern different kinds of knowledge (S&T, O&M, reputation building, etc.), which cannot be expressed in monetary terms. What is important is that all are, at least temporarily, re-usable by the actors. They represent a space where the actors can interact and contribute -more or less collectively.

Performance effects are especially important for operators, and their accounting is linked to the standard evaluation of RI performance. EvaRIO points to the need to find ways to enrich the classical cost-quality-timeline approach by considering elements better adapted to the specificity of research activities and domains.

Indirect effects can be deployed in different directions, for example, in line with the initial research field, but not based on RI, in other of the actor's research fields, and downstream towards market/society applications. This last direction potentially can generate societal effects; it constitutes a link between the EvaRIO approach and other approaches focussed on societal effects.

Basically, indirect effects are generated by exploiting different kinds of capacity (sometimes in combination) and then the same typology applies (S&T, etc.). It is interesting that they correspond to the valorization of a given capacity at a point in time, but do not limit the possibilities for the focal actor to exploit this capacity again. They are expressed in monetary terms.

A tentative architecture of metrics 6

Direct effects

The metrics used are monetary (with the exception of some comparative advantages for users).

Specifically:

i) for the operators: organizational budget (or sub-set of the organization hosting the RI) over the period being evaluated. It should include all costs incurred for operating the RI (including staff and maintenance);

ii) for the suppliers: contracts awarded to suppliers for the supply of the goods/services required to build, maintain or upgrade the RI over the period considered;

iii) for the users: in this case we consider two sub-sets of effects.

The first sub-set uses the approach applied to operators and suppliers: evaluation of a volume of activity depending on the RI. Here, these activities correspond to research using the RI, in the form of projects, programmes, recurrent or one-off uses of the RI, etc. Budget and staff figures can be used as proxies (proven in a small follow-up evaluation study run on an RI case).

The other sub-set of effects corresponds to the direct comparative advantage of using the RI. It can be measured in terms of resource and, hence, cost savings (on investment, materials, man-months, etc.), time savings (time-to-R&D result) or gains in the quality/reliability of results.

Capacity effects

Capacity effects are difficult to measure directly and evaluating and measuring change in actors' capacities are a very problematic component of the evaluation work. The methods proposed by economic and management theory to evaluate these intangible assets are not fully adapted to the type of capacity at stake (e.g. R&D). It is generally proposed to: (i) evaluate the future economic flows that this change in capacity will help the actor create; (ii) use the value currently imposed by the market on this capacity change; iii) estimate the cost of building or maintaining the capacity; iv) adopt a real options approach [START_REF] Ghezzi | On the financial evaluation of some intangible assets[END_REF].

We adopt a different approach, which departs from monetary evaluation and combines two components. First, we use qualitative evidence derived from interviews and in-depth investigation of each individual actor, to explain and reveal some of the main features of capacity changes.

Second, we exploit a large variety of ad hoc metrics that rely on 'external' data or objective observation. Both analyses are structured following the five dimensions of capacity defined above.

Thus, our approach mixes qualitative assessment and quantitative data in order to provide a multifaceted, but structured image of the capacity change. EvaRIO proposes a long list of metrics. One primary source is direct interviews with the actors involved; others include publicly available databases of S&T indicators (e.g., publications, patents, collaborative R&D projects); official/acknowledged actor rankings (of labs or firms' market positions). Note that it is not possible to combine these indicators to obtain a unique synthetic figure.

We would emphasize that this part of the study on the potential of impact is exploratory and the list of indicators could be extended. It would be interesting to adapt some of the indicators proposed in the emerging resource-based view/knowledge-based approaches to the context of competing companies, to S&T research. [START_REF] Godin | Measuring the Impacts of Science: Beyond the Economic Dimension[END_REF] and [START_REF] Autio | A framework of industrial knowledge spillovers in big-science centers[END_REF] propose quite original lists of indicators for some of the capacity dimensions we explore.

Effects on performance

As the actors are involved in very different RI-based activities, their performance in these activities cannot be assessed using only one measure. The case studies conducted using EvaRIO demonstrated the need to develop specific metrics -at least for RI operators. In fact, most, if not all operators have developed scoreboards and Key Performance Indicator (KPI) systems for such dimensions as cost/pricing, quality/reliability/availability of the resources offered to users, time required to access them, number of users, provision of scientific, technical or administrative support, confidentiality, rate of openness compared to maintenance period, etc. These ad hoc metrics are based on either objective measurements or user satisfaction surveys. There was a recent proposal related to the development of 'facility metrics' for specific facilities such as synchrotrons [START_REF] Hallonsten | Introducing 'facilitymetrics': a first review and analysis of commonly used measures of scientific leadership among synchrotron radiation facilities worldwide[END_REF]).

However, more sophisticated metrics will be required to express the performance of RIs operators as contributors to the advancement of scientific knowledge and the building of scientific communities. In this context, it is relevant to measure: (i) evolution in the number, variety, and renewal of resources and services provided by an RI (including experiments, training, access modes), and (ii) evolution in the number, variety and renewal of its users and user communities.7

Indirect effects

Indirect effects refer to the economic importance of effects stemming from the new knowledge acquired during the RI-based activity and valorized outside the RI. This part of evaluation relies heavily on the initial BETA method and follows its general framework in terms of S&T, O&M, Moreover, the measurement of effects performed by the BETA group provides systematic conservative estimates.

Methodological approach for the case studies

Nine case studies were conducted to test the BETA-EvaRIO method: four RIs were subjected to indepth analysis and five RI cases led to smaller studies, intended to complement the former by covering the diversity of RIs in terms of resources, organizational status and spatial-temporal borders. RI age varied from five to fifteen years.

The general protocol for conducting case studies via the BETA-EvaRIO method is as follows.

Interviews are important for collecting information: 76 interviews (lasting 2 hours on average) were conducted by experienced evaluators, and were recorded and synthesized in a confidential report. Regular internal debriefing meetings were held to share and discuss the empirical material obtained. Contact with interviewees was maintained to obtain clarification or additional (especially quantitative) information on specific points not fully covered during the interviews. The first series of interviews was framed at the operator level, to obtain a detailed understanding of the origin (history, key persons, main steps, previous collaborations) and current RI operations (organization, access rules, user community, industrial relationships), and to assess the relevance and potential difficulties of implementation. In the in-depth case studies, these 'framing interviews' enabled selection of a relevant sample of users and suppliers. A total of 61 interviews were analysed in depth; 32 were with operators, 23 were with users and the remaining 6 were with suppliers. 8 The sets of interviews dedicated to the small case studies were less extensive and mainly involved operators.

Identification and evaluation of effects was based on interview guidelines that included a list of topics. They did not rely on fully structured, precise and closed questions because the interviews were exploratory and the information could be obtained only through discussion with the interviewees. The discussion refined the specification of indicators by testing them and adding to the list defined at the beginning of EvaRIO.

Parallel in-depth collection of quantitative data was conducted in order to complete evaluation of the specific metrics of capacity developed in the EvaRIO project.

Positioning of the method in the recent RIA literature

As underlined in the introduction to this Special Issue, the variety of the methods used to evaluate different impacts of S&T research has grown dramatically, leading to an increase in the research impact assessment literature (see Donovan 2011 and the related Special Issue on Research Evaluation, [START_REF] Edler | The practice of evaluation in innovation policy in Europe[END_REF][START_REF] Gaunand | How does public agricultural research impact society? A characterization of various patterns[END_REF] for surveys covering the last two decades). While receiving less attention from the evaluation community than classical R&D programmes, RI have recently been the subject of impact assessments, using different methods [START_REF] Simmonds | Big Science Facilities and Innovation[END_REF][START_REF] Pancotti | Appraisal of Research Infrastructures: approaches, methods and practical implications[END_REF]. Without going into detail, it is possible to distinguish among these methods based on the type of approach used (e.g. quantitative econometrics or qualitative case studies) and the type of impacts addressed (e.g. economic vs. societal impact).

The BETA approach and, especially the BETA-EvaRIO method, is less easy to classify using these criteria. We would claim that, despite their use of quantification and their focus on economic impact (albeit quite specific economic impact), BETA methods resemble more hybrid types of approaches and have some features common to different families of methods, in some cases bridging between them. Section 3.1 examines some dual properties of the BETA-EvaRIO method (referring to the RIA literature in general since the issues raised here are mostly generic), and explores potential complementarities with other approaches applied in the narrower context of RI impact assessment.

The dual nature of the method

Based on 30 years impact evaluation experience, we examine the dual nature of the BETA approach along different dimensions. The dualities discussed here are related to a series of classical challenges common to evaluation studies and the distinctions proposed to classify these studies (see Link & Vonortas 2013;[START_REF] Pichler | Research and Technology Policy Evaluation -Evaluation Standards in Research and Technology Policy[END_REF][START_REF] Ruegg | Evaluating Realized Impacts of DOE/EERE R&D Programs -Standard Impact Evaluation Method[END_REF]. We selected some of these topics to support our hypothesis that the BETA-EvaRIO method exhibits dual features and fulfils a bridging role. The dualities highlighted here are considered also in response to some of questions raised in the introduction to this Special Issue (see [START_REF] Gaunand | How does public agricultural research impact society? A characterization of various patterns[END_REF] for a related discussion).

Input-output vs inside the black box

This first issue is related to an old debate in the innovation economics literature, which was translated to analysis of S&T policy. We consider that the rationale used by most standard evaluation approaches is based on an Input-Output view. Here, we refer to the neo-classical view of economic activities revisited in the Arrow-Nelson tradition, where research is represented by a production function similar to other production activities (producing output by combining inputs).

Evaluating input and output is the main challenge for evaluators. For instance, in the extensive survey of evaluation studies in [START_REF] Edler | The practice of evaluation in innovation policy in Europe[END_REF], a large share of the methods aimed at evaluating 'overall effectiveness' fall into this first category.

In the heterodox economics tradition, a complementary approach would be to open the black box.

In the case of S&T public policy, this implies taking account not only of input and output but also the system that transforms input into output and the process of creating output and, hence, the changes affecting the system (whether a research team, a lab, a company, a consortium or some kind of network of actors) when it creates output. Changes here can refer to organization, structure, processes or behaviour, and the more or less persistent changes are over time is important [START_REF] Bach | From economic foundations to S&T policy tools: a comparative analysis of the dominant paradigms[END_REF]; see also the related discussion on additionality in Section 3.1.5).

Between these two traditions and as mentioned in the description in Section 2, the BETA approach operates at different levels: output (e.g., indirect effects as re-defined in the BETA-EvaRIO method) and system changes (e.g., capacity effects); occasional, recurrent and persistent effectsalthough the level of maturity of the metrics is uneven.

Programmes and projects vs. organizations

The BETA method devotes attention to the analysis of projects within a given programme.

However, what is important is the actors that perform those projects and their ability to successfully achieve them and to transfer what they have learned to another context, that is, to other activities. Actors are at the intersection between these three levels of evaluation: the actors'

activities, the projects run by these actors (sometimes the unique form assumed by their activities)

and the organizations to which they belong and/or for whom the actors are performing the activities.

In the case of RIs, in particular, these three levels are interconnected. Since the BETA-EvaRIO method focuses on actors, it can provide insights into all these levels, especially for the case of RI operators running complex organizations for the performance of various types of activities. They use RI for their own research, maintain and develop the RI as a resource for external users, etc.

These activities usually are performed within different types of projects: users' projects and internal projects.

Although it blurs the boundary to the evaluation, an activity lens makes it possible to combine the project/programme and organization levels of analysis. Thus, the method is located at the interface between two interconnected, but distinct literature streams: RIA research and science-industry relationships. It should provide useful insights for both evaluation traditions.

Quantitative vs qualitative

The BETA-EvaRIO evaluation approach combines qualitative and quantitative tools; we discuss different aspects below.

The metrics are diverse: some are expressed in monetary terms, some in other quantitative terms, some are qualitative, but are transformed into quantitative terms using diverse scaling techniques.

Here, apart from standard Lickert-type assessments, the method translates socio-economic effects into monetary terms as far as possible. This translation is feasible when capacity enhancement (learning) is actually exploited and valorized in economic terms: time savings and subsequent cost reductions, added value due to new sales or patent licensing, new R&D grants or research contracts, etc.. So we obtain a single unit measure when economic effects are detectable.

Sources of information are diverse: the BETA-EvaRIO method is based mainly on information originating from the RI actors, but data collection also entails statistics from existing large S&T indicator databases and dedicated/ad hoc systematic data. The corresponding collection methods are similarly diverse: case studies based on direct interviews with actors, specific data surveys if relevant and feasible, use of sampling techniques to select actors for interview or survey purposes, etc.

The dual nature of the method relates to the fact that looking for figures is not inconsistent with fine analysis of impact processes in the BETA-EvaRIO approach. The main figures are derived from interviews, which allow detailed analysis of learning mechanisms and impact (cf. the interview protocol). Of course, the figures obtained are declarative and could be biased, but the risk of overestimation is countered by requesting conservative numbers.

There are at least two advantages from this mixed approach: (i) mixing sources, data and metrics allows data triangulation and checks of the consistency of some results; (ii) mixing qualitative and quantitative approaches provides insights to explain results and formulate hypotheses about complex causalities, resulting in a richer and deeper analysis.

Attribution vs contribution

The original BETA method was oriented towards an attribution approach, although it is not aimed at attributing effects to a given actor (based on the usual interpretation of the term attribution), but rather to a given (publicly sponsored) project. The implicit hypothesis is that several kinds of learning contribute to the generation of socio-economic effects and that these learning processes are, in turn, due to diverse activities, among which the evaluated project/programme is a more or less important part (see the short discussion on fatherhood coefficients in Section 2.1.2). However, it is worth noting that, without going as far as some fuzzy logic modelling might, the BETA approach never considers project influence to be an exact figure, but rather as within a range in which only the lower bound is retained. Also, the issue of 'marginal but decisive' contribution was often raised. The underlying logic was to try to give an account of the existence of a contribution by providing the lowest possible but credible figure.

The complexity of effect generation in the RI case reinforced this tendency to account for a contribution in the BETA-EvaRIO method. Multiple interactions among diverse actors, groups and communities prevail. Hence, the collective dimension of RI-based learning needs to be within an evaluation approach; however, some more work is needed on this area.

It is interesting that the RI context promotes a reconsideration of the attribution versus contribution issue, and elevates it to a more abstract and general, logical level. It does not focus on the possibility or not -to disentangle the influences of different actors, but rather more generally on the influence of different factors, whether actors, projects or activities. First, evaluating knowledge creation involves not just the actors, but also the actors' activities, projects, communities, etc. All these perimeters need to be considered in RI evaluations; it would be misleading to focus on only one of them. Second, the attribution/contribution issue involves the time dimension, as suggested by the standard 'project fallacy' issue, which highlights the misleading focus on the last in a series of different successive projects. RI-based learning by different actors within different time periods led to our tackling the difficult issue of evaluating non-isolated projects, activities or actors.

Although, again, more work is needed in this area, this question has been explored in both the synchronic case of cumulative learning through successive projects and the asynchronic case of learning that occurs in parallel projects.

Implications on the aggregation and additionality issues

An obvious consequence of the dualities described above is the difficulty related to the aggregation issue. Opening the black box, considering multiple perimeters (projects, organizations, activities), mixing quantitative and qualitative data, trying to broaden the initial attribution approach by introducing contribution elements, all contribute to making the evaluation results difficult to aggregate and synthesize.

Aggregation of the effects is complex because of the multiple levels at which it can be conducted.

To put it simply, four broad categories of effects and three types of actors imply many different aggregation strategies, some of which are either impossible to conduct or not relevant. The highest level of aggregation, that is, the macro-level, clearly is not feasible, but the meso-level is a possibility for certain categories of actors (operators, suppliers, etc.) or groups (including fuzzy groups such as research communities). Whatever the actor grouping considered, aggregation of all the types of effects is impossible due to fundamental differences in the intrinsic nature of the effects. However, one can envisage various aggregation possibilities for a given type of effect across different types of actors.

In theory, direct effects can be aggregated to an extent, as can indirect effects since they can be identified and quantified for each actor interviewed. More precisely, they can be summed up for a sample of actors and, in some cases, across the different categories of actors, on the condition that possible overlaps, double counting or inconsistency are systematically checked for. Although effects on performance cannot be summed since they are specific to each actor class, capacity effects can be identified and measured for each actor interviewed. However, in this latter case, aggregation raises questions which are not fully addressed by EvaRIO. For instance, the different capacity effects do not use the same metrics; each indicator reflects one dimension of capacity.

Resolving this problem would probably require scoring or multi-attribution methods to obtain an overall view of capacity change. Another issue is the likely possibility that the capacity of one actor is intertwined with the capacities of other actors, and that this collective asset is indivisible (i.e., the capacities of different actors do not exist individually -or only to a limited extent). In this situation, overlaps, combination, redundancy, non-additivity, etc. greatly increase complexity.

The non-additivity of capacity along with the other aggregation problems referred to above, are connected directly to an issue that is central to evaluation, that is, additionality. which is critical to a discussion of the private profitability versus public value of publicly sponsored R&D activities.

Additionality has been extensively discussed, especially in relation to the differences between input, output and behavioural additionality [START_REF] Buisseret | What difference does it make? Additionality in the public support of R&D in large firms[END_REF][START_REF] Gök | An Evolutionary Approach to Innovation Policy Evaluation: Behavioural Additionality and Organisational Routines[END_REF][START_REF] Gök | The Use of Behavioural Additionality evaluation in innovation policy making[END_REF][START_REF] Bach | From economic foundations to S&T policy tools: a comparative analysis of the dominant paradigms[END_REF][START_REF] Georghiou | Evaluation of Behavioural Additionality[END_REF]among others). It is clear that behavioural additionality is at the centre of an analysis of the effects related to learning from RI-based activities. However, output additionality is also important, especially in relation to indirect effects. In contrast, input additionality is concerned only with some direct effects.

While acknowledging the problems specific to aggregation and additionality linked to the dual nature of the BETA-EvaRIO method, the duality allows the possibility to establish a bridge among the different approaches. This point is discussed further in the next sub-section.

Complementarity with other methods

The complementarity of the BETA-EvaRIO method can be identified in terms of the different methods/metrics/data used to study the same types of impacts, as suggested by the input-output versus systemic duality presented above, quantitative versus qualitative and/or attribution versus contribution impacts. However, the complementarity issue is more complex than might appear at first sight. It could also concern coverage of different broad domains of impact (e.g., economic vs societal) or the use of different frameworks to map certain impacts. Finally, it might mean juxtaposing different approaches covering different types of impacts, in a kind of unified mapping (or the same analytical framework).

Although we do not explore all the facets of such a complex issue, we discuss a series of points below.

Complementarity: conceptual framework

The framework proposed by the BETA-EvaRIO method complements existing methods by providing another rationale for building the impact architecture, that is, based on learning processes rather than knowledge domains or stakeholders' objectives.

Several alternative frameworks rely on broad dimensions such as science, technology and innovation, work and population (mainly jobs and training), quality of life, ecological environmental impact domains proposed in the RIFI project [START_REF] Curaj | RIFI -Research Infrastructures-Foresight and Impact FenRIAM Methodology[END_REF], or the scientific, economic, human capital and societal areas of impact proposed by [START_REF] Zuijdam | The role and added value of large-scale RI, Ministry of Education, Culture and Science[END_REF].

The slightly more elaborate framework proposed in [START_REF] Péro | Role of RI for a competitive knowledge economy -Lessons learnt from the EC RI actions[END_REF] rationale, but these are often related to specific RI contexts. The above-mentioned approaches adopt a broader view on impacts, including pre-defined and more general rationales.

In contrast to all of these approaches, the BETA-EvaRIO framework puts learning and capacity building at the heart of the architecture of effects, thus, providing another perspective. However, understanding how one logic feeds another, or how it might help to shed light on a phenomenon that another can only foreshadow need more investigation.

Complementarity: mapping of effects

The BETA EvaRIO framework allows use of the main existing RI evaluation methods to be situated within a coherent mapping. Despite the wide heterogeneity in the methods used and/or proposed, certain families seem to dominate.

First, there have been numerous attempts to compose a sort of scoreboard of various isolated indicators reflecting, on the one hand, the activity and performance of RIs, and on the other hand, their output and impact9 . These aspects are mixed together, which blurs the distinction between evaluation for monitoring purposes and evaluation of impacts. The most frequent indicators include the standard S&T indicators (publications, patents, spin-offs, etc.). Other widely used indicators deal, for instance, with staff, funding, costs, users, availability and reliability of RI resources, jobsactivities in the local economy, PhD numbers, lectures and training programmes, visitors, scientific events, participation in R&D contracts, and many others.

Second, the input-output approach is used frequently to assess the impact of the amount of money spent on building the RI and/or on its operation and maintenance (SFTC 2010; and many examples provided in Curaj 2011). This includes the impact, on the (mainly local) economy, of expenditure on RI and its suppliers and, in some studies, by RI staff and visitors (e.g., the case of the synchrotron). Using existing statistics and Leontief-type matrixes combined with Keynesian multipliers, different waves of economic effects are estimated along the supplier chain. This rather classical approach is often applied to other types of infrastructure -beyond the field of research and technology.

Third, we need to mention the methods used to evaluate the impact of scientific results on economic growth and welfare (e.g., Toole 2012). Measurements use ad hoc tools and methods that are specific to each domain, but rely on welfare gains, due, for instance, to extended life expectancy, cost savings, consumer surpluses and suppliers' profits. However, these methods have not been applied to the results of RI use. They have been applied only downstream, considering the research process leading to the research results as a black box.

Figure 3 shows that the BETA-EvaRIO framework helps to fill the gap between these evaluation families:

• classical assessment of the performance of RI operations (Approaches Type I in Figure 3) is generally conducted during the activity, but can be computed ex ante or ex post. Clearly, it is related to the effects on performance of the EvaRIO typology of effects described in Section 2.2;

• standard evaluation of the economic impact of building and operating an RI can be carried out ex ante or ex post. It is well suited to the case of single-sited RIs (but not distributed RIs); these evaluations correspond to Approaches Type II in Figure 3 and are linked to the direct effects of the EvaRIo framework;

• evaluation of some societal and economic impacts of scientific achievements obtained from use of an RI (as a contributor), is related to the long term effects of scientific advances in a given field (agriculture, health, environment, etc.), which generally are evaluated ex post.

These effects (Approaches type III in Figure 3) are related to the indirect effects in the BETA-EvaRIO framework.

Figure 3: EvaRIO typology of effects and effects from other approaches

From a more operational perspective, this means also that there is no real need to design new evaluation tools and metrics since existing approaches can be applied. Therefore, EvaRIO development work and the case studies, focus only on parts of the effects, specifically, those that, in our view, are insufficiently or only partially covered by the other approaches.

Some of the data collected via implementation of the EvaRIO evaluation method can be used to feed these other approaches and/or act starting points for their implementation. This applies especially to the Type III approaches aimed at evaluating the overall impact of scientific advances (here, those made possible by RIs). Capacity effects are the triggers for all other types of effects, including indirect effects such as societal impacts. After identification at the micro-level of the first circle of actors, S&T capacity effects can be traced through their deployment with increasingly distant groups of stakeholders. Their identification allows an early investigation of emerging societal impacts. A capacity effect signals the possible starting point of an impact pathway.

Conclusion

In this paper, we presented a recent development of the BETA method adapted to the specific context of RIs. Through its ex-post view and emphasis on the micro level, the BETA-EvaRIO evaluation framework puts learning and capacity at the heart of effect generation and proposes an appropriate effects typology. These conceptual and methodological choices allow for an enlarged scope and complexity of analysis via consideration of various actors, activities, periods and RI resources. This has highlighted certain limitations and difficulties, some of which are still being addressed. Difficulties include the selection of metrics adapted to some effects (e.g., performance effects), the amount of resources required to run the detailed case studies and gather data, the systemic interactions among actors, especially at the meta-level of R&D communities and user networks, the different time horizons relevant for different parts of the system (actors, effects, activities, effects to be generated, etc.), and the aggregation of effects and consistency of the overall framework.

The broader perspective raised by RIs has revealed and increased the dual nature of the BETA approach and its complementarity with other evaluation approaches: evaluating the performance of R&D organizations, assessing the economic impact of infrastructures using input-ouput matrixes, and evaluating the societal and economic impacts of scientific advances, each of which is linked respectively to performance, direct and indirect effects. Thus, the BETA-EvaRIO approach embodies the possibility to bridge between these distinct evaluation approaches. Further development (and testing) of the method, especially in the directions described above, remain to be done to ascertain its bridging power and/or to deem it sufficiently established. In the case that consolidation is not achieved, the BETA method will remain a distinctive and singular approach, a kind of UFO in the galaxy of evaluation approaches.

Figure 2 :

 2 Figure 2: Conceptual framework: relations between the different types of effects

  network and reputation dimensions. In relation to metrics, most indirect effects can be expressed in economic terms whether referring to the added value generated by income or cost reductions enabled by efficiency gains. Accordingly, S&T effects can lead to the design of new or improved products, processes or services, which allow the actors to increase their sales, protect their market shares, benefit from cost reductions or win new research contracts. Network and reputation effects correspond to new sales or research contracts, and O&M effects generate cost reductions.As in the original BETA method, fatherhood coefficients are used to evaluate the influence of the RI-based activity, thus, taking account of the attribution problem. Whenever possible, the counterfactual (what if without RI based research) is considered at the micro-level of the actors.

labs t+1 ≠ cies, labs t

			direct vs indirect:
	Public		objective of project
	(co-)fundings	cies, labs t
		Pol. targeted R&D activity of cies, labs	Techno, products, services… as expected Direct'effects'
	knowledge methods, procedures, routines Learning processes affecting capacity, structure, networks competence & team building organisation behaviour and organisation at micro and meso (eg consortia) levels ! cies, Exploitation … NOT as expected affecting other activities than those related to evaluated programmes/projects Indirect'effects'

Table 1 : Crossing categories of effects and categories of actors -Summary

 1

		RI OPERATOR(S)	RI SUPPLIERS	RI USERS
				• Volume of activities
		Volume of activities	Volume of activities	corresponding to research
	Direct effects	corresponding to building	corresponding to supplying	projects using RI
		and operating the RI	RI components	• Direct advantage from
				using RI
	Capacity effects	Change in capacity due to	Change in capacity due to	Change in capacity due to
	(capacity: assets +	operating the RI, including	supplying RI components,	using the RI, including
	capability to	S&T capacity, Network,	including S&T capacity,	S&T capacity, Network,
	mobilize and make	O&M, Reputation, Human	Network, O&M,	O&M, Reputation, Human
	them evolve)	Capital	Reputation, Human Capital	Capital
	Effects on	Exploitation of capacity for	Exploitation of capacity for	Exploitation of capacity
	performance of RI-	enhancing performance as	enhancing performance as	for enhancing performance
	related activities	an RI operator	an RI supplier	as an RI user
		Exploitation of capacity by	Exploitation of capacity by	Exploitation of capacity by
		the actor for generating	the actor for generating	the actor for generating
		economic benefit 'out of	economic benefit 'out of	economic benefit 'out of
		RI':	RI':	RI':
	Indirect effects	• same research field but	• same research field but	• same research field but
		not on RI	not on RI	not on RI
		• other research field	• other research field	• other research field
		• downstream market/	• downstream market/	• downstream market/
		society applications	society applications	society applications

  , combines impact level categories (societal level corresponding to behavioural changes, e.g., use of internet, interest in science, etc.; health and environmental level, e.g., to understand climate change issues; economic level, i.e. gains in scale and scope linked to increased industry competitiveness, new technologies/products) with a time horizon, and a distinction between direct and indirect effects.Several UK studies have been inspired by the seminal work conducted in SPRU -Science and Technology Policy Research, on the impact of science[START_REF] Martin | The Relationship between Publicly funded basic research and Economic Performance[END_REF][START_REF] Martin | The Benefits of publically funded research[END_REF], based on which the DIUS (2008), and especially the Science and Technology Facilities Council, laid the foundations of the evaluation system (the Economic Impact Reporting Framework) used for several UK RIs.

Also interesting is the link established by some authors

[START_REF] Elias | Developing Methodologies for the Impact Analysis of Research Infrastructures on Innovation[END_REF][START_REF] Brottier | Socio-economic return of RI projects[END_REF]

between the area of impact and the interest or 'mental backgrounds' of various stakeholders: for a given RI, different types of effects and indicators can be adapted. Of course, approaches based on a 'logic model', as proposed, for instance, by Technopolis (2010) or by the Science & Technology Research Council (Dougan-McCallie 2014), also offer a framework with an objective-based

See attempts to grasp some of these aspects in[START_REF] Autio | A framework of industrial knowledge spillovers in big-science centers[END_REF] in the case of CERN.

"Concept evaluations review the mission, assumptions, fundamental hypotheses and framework conditions of interventions[…] design evaluations deal with the effectiveness of an intervention and its organisational structure[…] evaluating in detail how a programme, above all, is carried out as part of a process evaluation […] impact analyses assess the extent to which the objectives of an intervention have been achieved (effectiveness analysis) and attempt to identify all relevant directly or indirectly caused effects by an intervention, and, as far as possible, they attempt to quantify them"(Pichler et al., 2013 : pages 6-7).

Details are provided in the EvaRIO Core Report (2013) available at: http://evario.u-strasbg.fr/final-report.

A table summarizing the metrics is provided in "List of BETA-EvaRIO metrics and indicators", available at: http://evario.u-strasbg.fr/final-report.

While not completely absent, the issue of attribution is probably less difficult in the case of performance effects, since the learning in a given RI is developed mainly in activities related to that RI.

Note that 6 users are also suppliers.

Many examples were presented in various workshops. See among others the EvaRIO Final Conference "Evaluation of economic impacts of research infrastructures", 16-17 Octobre 2013, BETA, Strasbourg, papers available at: http://evario.u-strasbg.fr/conference/programme; or the ERF Workshop "The Socio-Economic Relevance of Research Infrastructures", 31 May -01 June 2012, DESY, Hamburg at: http://erf.desy.de/workshop.

Acknowledgments

The following paper draws heavily on research conducted in the EvaRIO (Evaluation of Research Infrastructures in Open innovation and research systems) project funded by the European Union 7th Framework Programme (FP7/2007_2013) under Grant Agreement n° 262281. The authors are indebted to the European Commission Directorate-General for Research and Innovation , and the contributions of numerous colleagues in BETA.