Kirsi Hyytinen

Faïz Gallouj

Marja Toivonen

A multi-criteria and multi-actor perspective for the evaluation of sustainability services

Keywords: Sustainable services, service innovation, multi-criteria evaluation, multi-agent framework

The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the innovations nor the multifaceted dimensions of performance. This paper aims to contribute to the need for a more diverse evaluation approach. We focus on the context of service innovation in the environmental sector, in which we develop further multi-criteria and multi-actor perspectives. The multi-criteria framework describes the impacts of sustainability services and the multi-actor framework aims at analyzing the impacts from the perspective of various actors involved. Thus, our study provides a two dimensional approach to assess the impacts of services. The focus is on understanding the dynamics of service creation in the environmental sector, and on using new evaluation methods and indicators in that sector.

INTRODUCTION

Systematic evaluation is an activity that has been an essential part of strategic thinking for decades. It has been linked both to the general development of organizations and to more specific functions. It has played a particularly important role in the context of innovation and R&D. In public interventionsoften carried out in the form of policy programsevaluation has been an embedded practice whose purpose has been to support decision making [START_REF] Rip | Societal Challenges for R&D Evaluation[END_REF]. Since the early days of evaluation, impact assessment has been a typical way to implement evaluation, which has meant that evaluation has been understood in terms of performance-related steering and monitoring [START_REF] Chelimsky | The Coming Transformation in Evaluation[END_REF][START_REF] Rip | Societal Challenges for R&D Evaluation[END_REF]. The primary idea has been to produce indicator based information to prove accountability and to legitimate the role and existence of individual organizations and policy instruments. This kind of an approach to evaluation includes several problems. First, it is backward looking and does not work well as a guiding instrument in the current society that is characterized by rapid changes (Todd and Wolpin, 2010;Weijermars and Wesemann, 2013). Second, the approach is "atomistic": it focuses on individual organizations or policies and overlooks the fact that impacts are co-produced by several actors and emerge in a broader context [START_REF] Rip | Societal Challenges for R&D Evaluation[END_REF]. Third, the indicator based assessment simplifies phenomena that emerge in cyclic, complex and long-term processes. Indicators assume a simple causal relationship between intervention and impacts, which is incompatible with the modern view about the emergence of innovations [START_REF] Hansson | Organisational use of evaluation[END_REF][START_REF] Van Der Knaap | Performance Evaluation and Performance management. Owercoming the Dowensides of Policy Objectives and Performance Indicators[END_REF][START_REF] Cozzens | Use and Usefulness of Performance Measurement in State Science and Technology Programs[END_REF]. Summarizing, the linear input-output-outcome -thinking included in the traditional evaluation does not correspond to the complex development processes and the multiple relationships between the contributing actors [START_REF] Arnold | Evaluating research and innovation policy: a systems world needs systems evaluations[END_REF][START_REF] Patton | Developmental evaluation: Applying complexity concepts to enhance innovation and use[END_REF].

As an answer to these problems, researchers have suggested a more versatile view: "plurality of methods" [START_REF] Dyehouse | A comparison of linear and systems thinking approaches for program evaluation illustrated using Indiana Interdisciplinary GK-12[END_REF]Williams and Imam, 2007). Systems thinking have often been applied hand in hand with this view. It means highlighting interrelationships and feedbacks between the whole, its parts and the environment they are interacting with [START_REF] Cabrera | Systems thinking[END_REF].

2

Recently, the increasing "servitization" of society has put pressure to develop a more advanced approach to evaluation. Here, an additional problem linked to the traditional impact assessment has become apparent: the indicators have been one-sidedly technologically focused [START_REF] Toivonen | Different types of innovation processes in services and their organizational implications[END_REF][START_REF] Gallouj | Public-private innovation networks in services[END_REF]. These kinds of indicators do not capture the immaterial and interactive aspects that are central characteristics of services. Today services are also increasingly forming systemic wholesespecially the most urgent problems in the present society cannot be solved via the development of individual services, but service systems. The issues of sustainability are an illustrative example of this development. The current evaluation methods are unable to tackle the challenge of the systemic nature of innovation and service creation [START_REF] Smith | Innovation as a systemic phenomenon: Rethinking the role of policy[END_REF]Edqvist, 2005;[START_REF] Ahrweiler | Innovation in Complex Social Systems[END_REF].

This paper aims to contribute to the above described need for a more diverse evaluation approach. We focus on the context of service innovation, in which we develop further a multi-criteria and multi-actor perspective for evaluation. Our starting points are the multi-criteria framework developed by [START_REF] Djellal | The Innovation gap and the performance gap in the service economies: a problem for public policy[END_REF], and the multi-actor framework developed by Windrum and Garcia-Goñi (2008). The former strives for better visibility of the multi-faceted dimensions of service innovation and highlights the importance of quality and social value. The impacts of innovation are analyzed using the concept of performance which is also interpreted broadly. The purpose is to diminish the "double innovation and performance gap" that derives from a narrow view on these concepts and has resulted in a difference between the reality and the ability of the traditional measures to perceive it (Djellal andGallouj, 2010, 2013). The latter frameworkthe multi-actor analysisis built on the characteristics-based view on service innovation [START_REF] Gallouj | Innovation in services[END_REF]. It focuses on understanding the dynamics and impacts of service innovation from the perspective of the different actor groups participating in the service development.

In the conceptual part of our paper we open the "innovation and performance gap" in the service context and review the opportunities for new kinds of evaluation approaches. Then we discuss and concretize the multi-criteria and multi-actor frameworks by applying them in services that are emerging to support the strivings for sustainability. More specifically, we analyze service innovations and their assessment in the context of sustainable energy production. The multi-criteria framework that we develop further utilizes the categorization of different "worlds of services"equating to different sets of justificatory criteria (Gadrey, 2005;Djellal and Gallouj, 2012). Empirical data has been collected in Finland in the context of public-private development programs in the area of environmental sustainability. The following research questions are guiding our work:

-How and by what means should the outcomes and impacts of environmental services be evaluated in order to take into account the complex and integrative nature that characterizes innovation in these services?

-What are the impacts of environmental service innovations from the perspectives of different societal fields and of different actors?

The results of our study provide a two dimensional approach to assess the impacts of services by combining the multi-criteria and multi-actor frameworks. The focus is on understanding the dynamics of service creation in the environmental sector and the use of evaluation methods and indicators in that sector. At a more detailed level, our study provides analytical material about the complementarities and contradictions between different actors and indicators in service development. Because the links between services and environmental issues have been scarcely studied, our study also contributes to narrowing the gap between these two fieldsa point whose importance has been emphasized recently [START_REF] Rubalcaba | Shaping, Organizing and Rethinking Service innovation: A multidimensional Framework[END_REF][START_REF] Djellal | The Innovation gap and the performance gap in the service economies: a problem for public policy[END_REF][START_REF] Gadrey | The environmental crisis and the economics of services: the need for revolution[END_REF].

The paper is divided into six sections. The second section after this introduction is based on literature and discusses the challenges of measurement and evaluation, particularly in the context of services. The third section presents the two main frameworks that we apply in our study to evaluate the outputs and impacts of services: the multi-criteria and multi-agent frameworks. In the fourth section, our case-study context and methodology are described. The fifth section analyzes the results of the application of the frameworks in the assessment of environmental services.

Concluding remarks ends up the paper.

3

MEASUREMENT CHALLENGE IN SERVICE INNOVATION AND PERFORMANCE

The importance of invisible innovation and performance

For two decades, service studies and specifically the studies on service innovation have argued that the traditional tools, indicators and measures do not capture the performance, innovativeness and impacts in services [START_REF] Sundbo | The theory of innovation: entrepreneurs, technology and strategy[END_REF]Metcalfe & Miles 2000). A central background reason for the existence of the gap is the "assimilation" perspective adopted in the early service research [START_REF] Coombs | Innovation, measurement and services: the new problematique[END_REF][START_REF] Gallouj | Economie de l'innovation dans les services[END_REF]. This perspective analyzed services innovation as an imitation of technological and manufacturing innovations, i.e. it did not recognize independent innovation possibilities in services. The perspective was based on the traditional definition of innovation as an invention which results from an R&D project [START_REF] Howells | Innovation, consumption and services: encapsulation and the combinatorial role of services[END_REF]. The linear, stage-gate model of an innovation process, which was raised to the position of a norm and marketed as a prerequisite for success (e.g. [START_REF] Cooper | New industrial financial services: what distinguishes winners[END_REF], increased the bias.

These arguments and perspectives have been developed in the context of service innovations. However, they are strongly rooted in general innovation theories basing on the Schumpeter"s definition of innovation that has afterwards been regenerated to "the neo-Schumpeterian" theory of innovation or "the broad view of innovation" (Lundvall, 2007). These traditions are interlinked by several common aspects that affect both the definition of innovations and evaluation of their effects and impacts. Important cornerstones are complexity and uncertainty of innovation process, intangible nature of innovation (focus in new solutions and processes) and systemic view of innovation encompassing multiple sources and actors taking part into the innovation process. [START_REF] Kline | An Overview of Innovation[END_REF]Dosi,1988;[START_REF] Lundvall | National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning[END_REF][START_REF] Freeman | Networks of innovators: A synthesis of research issues[END_REF][START_REF] Nelson | Technical Innovation and National Systems[END_REF]. Also from the perspective of general innovation literature conclusion has been parallel: current practices in defining innovation and evaluating them do still follow the mainstream linear innovation thinking which simplifies too much the innovation process as well as the complex dynamics between actors contributing innovation (e.g. [START_REF] Arnold | Evaluating research and innovation policy: a systems world needs systems evaluations[END_REF][START_REF] Patton | Developmental evaluation: Applying complexity concepts to enhance innovation and use[END_REF]). Thus, it does not take into account the systemic view of innovation (e.g. [START_REF] Smith | Innovation as a systemic phenomenon: Rethinking the role of policy[END_REF]Edqvist, 2005;[START_REF] Ahrweiler | Innovation in Complex Social Systems[END_REF] and does not capture the multifaceted and diversified performance of innovations (e.g. dealing with societal issues such as health, education, sustainable development) (see e.g. [START_REF] Freeman | Developing science, technologic and innovation indicators. What can we learn from past[END_REF].

A broader view on innovation has increasingly gained ground, but the assimilationist origin of the perspective on innovation and services is still influential. In particular, the measurement and indicators of innovation and performance are still largely based on the industrialist and technologist definitions (e.g. [START_REF] Smith | Innovation as a systemic phenomenon: Rethinking the role of policy[END_REF]Edqvist, 2005;[START_REF] Smits | The rose of systemic instruments in innovation policy[END_REF][START_REF] Djellal | The Innovation gap and the performance gap in the service economies: a problem for public policy[END_REF][START_REF] Windrum | Multi-agent framework for understanding the success and failure of ServPPINs, in Public-private innovation networks in services[END_REF]. In addition to the narrow view on innovation, the dominating view on performance is also mechanical and narrow. It is usually linked to the concept of productivity which refers to the linear and mechanistic input-output function (e.g. Djellal andGallouj, 2010, 2013;Kellog foundation, 2004;[START_REF] Patton | Developmental evaluation: Applying complexity concepts to enhance innovation and use[END_REF]. Its" traditional definition is unable to recognize the "hidden performance" concerning the societal aspects of services and innovations like equality, ecological sustainability and societal wellbeing. It also often excludes the aspect of social innovation. (e.g. [START_REF] Rubalcaba | Shaping, Organizing and Rethinking Service innovation: A multidimensional Framework[END_REF] Djellal and [START_REF] Djellal | The Innovation gap and the performance gap in the service economies: a problem for public policy[END_REF] have described the interaction between performance and innovation by referring to the visible vs. invisible nature of these phenomena. Whereas technology-based innovations are visible, nontechnological innovations are invisible. As regards performance, the authors link the visible-invisible dichotomy to short-term vs. long-term influences. Both in scientific and the managerial discussions, short-term influences of performance are often analyzed in terms of productivity and growth. Longer-term influences are increasingly analyzed in terms of environmental or social sustainability. There are four possibilities in the relationship between innovation and performance as Figure 1 illustrates (ibid., 668). The most apparent relation is between visible innovation and visible performance, but visible innovation may also lead to invisible performance by promoting the long term ecological sustainability or societal well-being. Correspondingly, invisible innovation may be a source of visible performance, i.e. growth and productivity, or promote sustainability. 4

Opening the specific characteristics of service innovation and performance

After the realization of the narrowness of the assimilation view, two alternatives have gained ground. The first is the differentiation perspectivealso known as the demarcation perspective [START_REF] Coombs | Innovation, measurement and services: the new problematique[END_REF][START_REF] Gallouj | Economie de l'innovation dans les services[END_REF]Gallouj , 1998)). It focuses on the specific characteristics of service innovation and has highlighted the difficulty of recognizing "newness" and its creator in the service context [START_REF] Preissl | Service Innovation: What makes it different? Empirical Evidence from Germany[END_REF]. The second alternative is an integrative or synthesis perspective (Cooms and Miles, 2000;[START_REF] Gallouj | Innovation in the service economy: The new wealth of nations[END_REF]) which has become increasingly relevant due to the blurring lines between goods and services. It highlights the production and consumption that focus on integrated solutions and systems.

Recently, these approaches have also been applied when searching better indicators for innovation and performance in services. The peculiar characteristics of services that specifically have been pointed out in this context are intangibility and the central role of interaction; the latter refers to the central role of co-production between the provider, customers and partners. An important implication of intangibility is the difficulty of defining the "unit of output" and differentiating the product from the process. These aspects challenge the definition of innovation and quality improvements in it (ibid.). Interactivity increases the complexity of the development of services innovations [START_REF] Gallouj | Public-private innovation networks in services[END_REF]. Ignoring these complex and dynamic relationships (cf. Arnold 2004) is often linked to the traditional technologic measures and the linear innovation model [START_REF] Smith | Innovation as a systemic phenomenon: Rethinking the role of policy[END_REF]Edqvist, 2005;[START_REF] Ahrweiler | Innovation in Complex Social Systems[END_REF]. It may lead to the oversimplification of the reality and to the biased understandingnot only of the impacts of services and innovationsbut also of their drivers and dynamics [START_REF] Arnold | Evaluating research and innovation policy: a systems world needs systems evaluations[END_REF].

New aspects in the analysis are inclusion of the time factor and the social nature of services [START_REF] Djellal | The Productivity in services: measurement and strategic perspectives[END_REF]. Time highlights the dynamic nature of services, focusing to their evolution over time. Considering time as a dynamic factor leads to the differentiation of short-term outputs from medium and long-term effects. The social nature of services derives from the fact that the value and benefit of services is always defined by users. This means that different actors have different values, and it is just this multiplicity of values which makes it necessary to include various criteria in evaluation. In addition to immediate users, it is important to take into account the multiple values of indirect users as well as those of different actors participating in the development, (Djellal andGallouj, 2010, 2013). Thus, in service innovation boundary lines are blurringnot only between products and processesbut also organizations often change simultaneously [START_REF] Preissl | Service Innovation: What makes it different? Empirical Evidence from Germany[END_REF]. These kinds of combinatory innovations and their broader socio-economic impacts cannot be tackled on the basis of the traditional linear and industrialist models.

The double gap has been noticed to cause significant implications to public policies, which are still very technologist oriented and do not take into account the innovation and performance gap included. Thus, the invisible innovation and performance remain invisible in the policy making, causing problemsnot only in the performance measurement as suchbut also in target setting, and in steering and policy planning. In order to improve the situation, both the visible and invisible aspects in innovation and performance has to be included in an integrative way [START_REF] Djellal | The Innovation gap and the performance gap in the service economies: a problem for public policy[END_REF].

However, the analysis above does not mean that all existing innovation and performance measures and indicators should be abandoned. Instead of that, the need for a more diversified, multi-criteria framework can be identified.

The framework should also be flexible, and besides the traditional economic measures (such as productivity and performance), focus on analyzing the aspects relating to social innovation and use value linked to the performance. This kind of an approach also prefers the concept "evaluation" over the concept of "measurement" (Djellal andGallouj, 2010, 2013).

A NEW FRAMEWORK FOR EVALUATION

Multi-criteria approach to diversify the perspectives of evaluation

These above described challenges in measurement and evaluation of services intend to assess the performance and impacts on the basis of a multidimensional approach which takes into account the special characteristics of services as well as their aspects of quality and social value (Djellal andGallouj, 2010, 2013;[START_REF] Gallouj | Public-private innovation networks in services[END_REF]. Like Djellal andGallouj (2010, 2013) have argued one potential and diversified way of analyzing the various purposes and outcomes of services is to link to them to the idea of different "worlds", which are the industrial and technological word, the market and financial world, the relational and domestic world, the civic world, the world of 5 innovation and the word of reputation. Referring to the initial description of worlds (deriving for the economic tradition; Boltanski and Thévenot, 1991;[START_REF] Gallouj | Innovation in the service economy: The new wealth of nations[END_REF] aim is in giving an insight to different justificatory aspects in the society considering their different principles and values.

In that model the outcomes are evaluated from the perspective of different goals or target areas, encompassing both the traditional measures arising from the industrial and technical as well as market and financial world and the modern evaluation criteria taking into account the needs of knowledge society (e.g. relational aspects such as trust and aspects relating to the responsibility and societal wellbeing). In addition to the different target areas the model focuses on the long time-scale in the generation of impacts by dividing outcomes into the direct, short-term outputs and indirect, long term-outcomes.

We have made some modifications to the original model. Our changes concern especially the aspect of "the civic world" that originally was focused to social relations characterized by the ethical issues such as equal treatment and fairness. In this analysis we suggest to integrate the concept of societal value and responsibility into the idea of civic world and also rename it "responsible world". Our aim is to better take into the account the aspects of social innovations, environmental sustainability and societal well-being (see Rubalcaba et al., 2012, Djellal andGallouj, 2013). In the original model also the word of innovation is differentiated. Because in study focuses is on innovations the perspective of innovation world is excluded as a vertical dimension. Instead we clarify the different elements of innovation horizontally by analyzing the core aspect of innovation in the light of each world.

In our study innovations are analyzed based on the following different worlds given the specific justification criteria related to the each of the worlds:

Industrial

Multi-actor perspective to analyze the impacts of multiple actors involved in services creation

The interactive and subjective nature of services (and their implications to the evaluation) leads to emphasize the multi-actor nature of services. In the recent literature the multi-actor perspective has been proposed as an analytical framework to better understand the complex interaction between policy makers (society), public and private service providers as well as users of the services and to help to evaluate the benefits of the services from the different perspectives [START_REF] Gallouj | Public-private innovation networks in services[END_REF]. By integrating the actors" perspective to the evaluation we aim to emphasize the subjective nature of evaluation; evaluation criteria are always dependent on actors" specific values and interests in a specific situation. In addition we aim to pay attention to the interaction of various actors and their values creating an evaluation situation (cf. [START_REF] Giddens | Social Theory and Modern Sociology[END_REF].

Interaction may be operationalized based on the multi-agent framework that has been developed specifically to the context of public and private innovation networks to develop innovations and services [START_REF] Windrum | Multi-agent framework for understanding the success and failure of ServPPINs, in Public-private innovation networks in services[END_REF].The central idea in the framework is to study and understand the complex interactions between multiple actors in concerning service innovations. Framework is based on the former work of (Windrum and Garcia-Goñi, 2008;[START_REF] Gallouj | Innovation in services[END_REF]. When the primary work focuses to analyze services based on their characteristics in the later work the core is in actors. Particularly important are the characteristics and preferences of the actors. One of the values of this dimension is that it puts particular emphasis on the public sector"s role in the innovation process. Typically innovation research has been focused on the private sector innovation without putting too much effort to understand the role of public sector in creation of innovations. However, according to [START_REF] Windrum | Multi-agent framework for understanding the success and failure of ServPPINs, in Public-private innovation networks in services[END_REF], both political organizations and non-governmental organizations play an important role in innovation generation and dissemination and needs to be considered. Compared to the former model the value added in the recent one is in the recognition of public sector actors and thus the interaction between economic, social and political spheres (ibid.).

In our study we are analyzing the impacts from the perspective of multiple different actors taking part to the service and innovation development. In the analysis the preferences and thus the benefits are emphasized from the viewpoint of society (represented by policy makers), public sector service providers, private sector service providers and end users.

6

CASE CONTEXT AND METHODOLOGY

In our empirical analysis the focus is in the complex combinatory services in the area of environmental analysis and measurement. Paper studies outputs and outcomes from the viewpoint of different worlds and different actors focusing to the Environmental data platform that aims to be a comprehensive platform to support continuous data gathering and real-time environmental monitoring, analysis and reporting. Innovation has been developed in research and development program built on public-private innovation network that aim to promote systemic change in the field of environment in Finland.

We have applied the case study methodology and qualitative approach in our study. The main method in data gathering in our study was face-to-face interviews (30 in total). The main topics focused on the manifestations of new innovative solutions within the programs, factors that promote or slow down their generalization, impacts of the innovations and their evaluation. The interviews were gathered between February and June 2013. The final sample represented actors in the area of environment in a versatile way. It consisted of representatives of private companies (e.g. companies developing environmental and industrial measurement), and universities and other public research organizations.

To analyze and interpret empirical data we applied the modified multi-criteria and multi-actor frameworks. To start we analyzed what are the impacts of innovation in a short and long term from the perspective of different worlds. In the analysis we reflect impacts both from the traditional and modern perspectives. From the industrialist viewpoint typical evaluation criteria are for example technical operations and volumes. Similarly market world uses criteria like value (in financial terms), market share and business transaction. Relational and responsible worlds aim to broaden the focus to take into account the societal value as well as intangible, interactive, social and systemic nature of services. Moreover, world of reputation analyzes innovation from the perspective of brand and image effect. Finally we illustrate in very general level how the impacts vary from the perspective of different actors participating into the innovation creation. In the analysis the role of policymakers, public service providers, private service providers and end-users are taken into account.

RESEARCH RESULTS

Multi-criteria perspective for the analysis of new sustainable services

Environmental data platform (table 1) is from an industrial perspective a prototype platform to gather and share environmental data. As a short term output it integrates real-time data sources (e.g. measured data of water quality and satellite data concerning environment and atmosphere) and provides visualized maps based on the data. In a long-term goal is to integrate multiple data sources in it and provide "cloud-based comprehensive solution" to produce and share environmental data. Cloud-based solution would make the access to big data possible. In addition the architecture for real-time monitoring, analyzing and reporting will be created to improve the quality and reliability of environmental information, weather forecasts and warnings. New architecture helps in developing new end-user applications and thus accelerates business start-ups generation. Our respondents highlighted that services are in the central role in the development of comprehensive solution and new information architecture.

In the market world the main characteristic of the innovation is free access to the multiple data sources and especially to public sectors data. According to our interviewees free access to data is considered to be an important social goal and as a starting point for the development of data platform. In a long term goal is in opening new markets based on environmental analysis and in creation of new export possibilities.

Preferences and valuation principles in the relational world highlight the interaction between multiple actors and a role of end users is the development and use of service. Central innovative aspect here is the connectivity via one mediator to multiple data sources. As output goal trust in the public-private innovation networks is enhanced. From the performance viewpoint, that enhances actors" connectivity and consolidates the networks. Long-term goal is to integrate citizens to the data provision, to better take into account the specific users" needs and requirements and hence provide personalized environmental data. In a long term users will be integrated part of the environmental data generation. That development removes the clear distinction between the production and use of data (citizens are becoming data producers), which has been considered as a prerequisite for a systemic change in the sector.

From the perspective of responsible world overall sustainability is highlighted as a central value in development of new solution. The other value based aspects highlighted are transparency and the citizens" equal rights to participate to the data provision. Innovative aspect relates to the open and equal access to public sector data. As a short term goal platform enhances transparency, availability and multifaceted us of public data. As a long-term goal respondents have considered the possibilities to apply new end-user applications for example to the game and school worlds to support the environmental education and thus increase the awareness of environmental issues from the early age. From the performance viewpoint that may increase responsibility as a value in decision making in individual, firm and policy level.

Finally in the world of reputation innovation is analyzed from the perspective of brand and image effect. From a short and long term the aim is in analyzing how the innovation impact to the attractiveness of different actors. The aspects concerning the image and brand needs to be analyzed from the perspective of specific actors and are thus analyzed in details in the following sub-chapter.

To summarize the central findings from the perspective of different type of evaluation criteria following aspects have been recognized. According to our interviews services have important role in renewing the environment sector.

To solve the most urgent societal problems, like sustainability, there is need to develop technologies and services as systemic wholes instead of individual technologies or services. The emergence of comprehensive service solutions in renewing the environment and energy technologies can be perceived by traditional measures. However, the focus from the perspective of industrial and financial world is in technical characteristics of the concepts and solutions, not so much in their social and systemic nature.

To capture the interactive and social nature of service development as well as its" interlinks to the social goals and system level changes the other criteria are needed. Like our example show, relational and responsible worlds are particularly important to make these aspects visible. Like our respondents highlighted the changing roles of data users and producers ("from user to producer") plays a central role in the renewal and a system level change in a sector. That can be perceived only from the perspective of relational world. In addition our example shows that impacts generated in the different worlds are often interdependent and complementary to each other. Some factors in relational and responsible worlds can be seen as a prerequisite to effects generated from the viewpoint of technical and financial worlds. For example for the development of environmental data platform the transparency and open access to public sector data is seen as a precondition.

Multi-actor perspective in the analysis of new sustainable services

In the following discussion and table (2) we exemplify the impacts from the perspective of multiple actors. Central actors taking part into the service innovation development are society (policy makers setting the societal goals and preferences for the development), public service providers, private service providers and end-users. By integrating the actors" perspective to the evaluation we aim to emphasize the subjective nature of evaluation; evaluation criteria are always dependent on actors" specific values and preferences in a specific situation. In the analysis we zoom environmental data platform from the perspective of longer term outcome to exemplify the diversity of the impacts based on the values and preferences of different actors.

Based on the preferences and values of the private service providers it is natural the benefits of the innovation relates to the knowledge transfer, the development of new commercial applications and thus in the creation of new business opportunities. From the financial perspective, that brings new revenues and opens up the access to the international markets. New evaluation criteria enrich the analysis by highlighting the importance of mutual trust between multiple partners, which has been highlighted as a core aspect in development of new innovations and services. In addition it relational world makes visible the opportunity for the new and changing roles and responsibilities when developing and offering new services. Moreover sustainability as a value and as a competitive advantage is has also been highlighted from the perspective of private service providers: a reputation as a responsible company is a competitive advantage for the contemporary companies which may boost their success in international market.

A long term outcome for the public service providers relates primarily to the access and new opening opportunities based on big data. That creates opportunity for the profound and multifaceted knowledge production which enhances more quality environmental data production and based on that creation of new public services. From the perspective of public sector service providers the opening the new markets and research opportunities were highlighted. And like the private sector companies they also found that knowing new actors and creating the mutual trust between the multiple partners for deepening the collaboration are in the part of the main benefits from the process. In the other words, trust can be characterized as an enabling factor in the co-creation of innovation. From the responsibility viewpoint provision of predictive information of weather and environmental changes helps to manage potential environmental risks. Because of the specific role of public sector service providers this perspective is highlighted here. Capacity to provide environmental (e.g. safety environment) and societal impact (e.g. citizens health and good living conditions) belongs to the roles and functions of public service providers.

From the viewpoint of end-users receiving personalized real-time environmental data is the most self-evident benefit of the innovation. The data personalized to your own needs and received via you own mobile phone based on your location mayfor example real-time pollen reports for the allergic and asthmatic personhelps people to predict the pollen risks and thus have important health impacts. In a longer term the citizens" ability to provide data and act as an active and equal data provider and empowered citizen was examined as an important impact. From the relational viewpoint the change in end-users role was recognized as an important impact from individual but also societal point of view: their active and equal role as an environmental data provider creates preconditions for the new innovative and service oriented culture as well as demonstrates the potential for the systemic change in the sector.

Society is serving the multiple actors thus the impacts described here is combination of everything we have represented above. In enabling the creation of new business ecosystems society"s preference is to enhance the emergence of start-ups, help them to internationalize (export share of GDP) and to create the jobs. From the financial viewpoint that brings new revenues for the society and helps in developing societal services (which in turn causes impact in health and living conditions). In addition society has a role to promote and consider more soft and social aspects in the society. For example create new "programs" (meaning devices and educational programs) for the environmental education and thus raising the awareness and responsibility of citizens are the most important impacts from the viewpoint of responsible world. In addition transparency of the society relating to the openness of public data is a self-evident goal for the society.

To summarize, like our analysis show experience of impacts and benefits are strongly related to the preferences and values of actors. Both their goal and motivation to take part into co-creation is different and thus are the experienced impacts. From that perspective it is evident that there are contradictions regarding the benefits of different actors. However, different goals and preferences are in the most cases seen as complementaries which enhances the cocreation. When considering the impacts crosswise from the perspective of multiple actors and different worlds we create more diversified and analytical picture of the generated or potential impacts. In addition analysis makes 11 visible that in many cases the impacts are strongly interlinked and are systemic in nature.

CONCLUDING REMARKS

The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the innovations nor the multifaceted dimensions of performance in the sector. This paper aims to contribute to the need for a more diverse evaluation approach. We focus on the context of service innovation in the sector on environment and energy, in which we develop further multi-criteria and multi-actor perspectives. Multi-criteria framework describes the impacts of new sustainable services and multi-actor framework aims in analyzing the impacts from the perspective of multiple actors taking part into the service and innovation development.

As a result we identified that services have an important role in renewing the energy and environment sector. The emergence of comprehensive service solutions in renewing the environmental sector can be perceived partially the by traditional measures. However the focus from that angle is in technical and measurable characteristics of the concepts and solutions. From the traditional perspective their interactive nature and multifaceted performance (including the social and value based aspects) cannot be captured. To perceive and make visible the societal goals of the service innovations as well as their interactive nature relational and responsible worlds are particularly important. Moreover, our analysis also shows that impacts generated in the different worlds are often interdependent and complementary to each other. Some factors in relational and responsible worlds can be seen as a prerequisite to effects generated from the viewpoint of technical and financial worlds. In other words relational and responsible aspects are needed to generate long-term technological and economic impacts.

From the multi-actors perspective impacts and benefits are strongly related to the preferences and values of actors. Analysis make visible that there are evident (value and motivation based) contradictions between different actors. However, when there is a mutual trust these are seen as complementaries which enhances the co-creation. When considering the impacts crosswise from the perspective of multiple actors and different worlds we create more diversified and analytical picture of the generated or potential impacts. In addition analysis makes visible that in many cases the impacts are strongly interlinked and are systemic in nature.

Regarding to the further studies more studies would be useful to test the generalizability of our results. Furthermore to go deeper to the dynamic relations (systemic nature) of the different actors and impacts the application of systemic modelling methods could be the next step. In addition generating indicators based on these findings would be both interesting and useful.

Figure 1 .

 1 Figure 1. Innovation and performance gap in measurement of services (source Djellal and Gallouj 2010)

Table 1 .

 1 A multi-criteria framework to analyse the impacts of environmental data platform

		Industrial and technological	Market and financial	Relational	Responsible	Reputation
	Innovation/Input	A prototype platform for	Free access to public sector data	Connectivity via one mediator	Open and equal access to public	Image as an innovation to
		environmental data gathering and		to multiple data sources and	sector data	enhance sustainability
		sharing (integrating real-time data		end-users
		sources & data storage)
	Output (direct, short	Varied measured and satellite data	Economical of free access to	New connections and actor	Transparency of public data; Easy	Short term image
	term)	concerning environment and	multiple environmental data	networks created (via new	access
		atmosphere (e.g Weather radar	sources	solution)
		visualised on the map)
	Performance related to	Ability to process increasing amount	Reduced cost of sharing data	Increased connectivity between	Increased transparency of public	Improvement/change in
	output	of environmental data more quickly		multiple actors	data	reputation and image
		and more effectively	Reduced cost of providing data
					Increased availability and better
					use of public data
	Outcome (indirect,	"Cloud-based comprehensive	New market opening based on	Integrating citizens and	Equality in sharing information	Long-term reputation and
	longterm)	solution" for producing and sharing	environmental monitoring	citizens" requirements to the	(data)	image
		environmental data		data provision
			New opening export possibilities		Quality controlled data
		Architecture for real time		"Users as a data providers"
		environmental monitoring, analysing		(Social innovation)	Platforms applied e.g. in
		and reporting, bid data			education: environmental
				Personalised environmental	education and awareness
		New end-user applications created		data
		New start-ups crated; new jobs		Trust
	Performance related to	Improvements in weather forecasts,	Improvements in the generation	Increased usability of	Increased awareness of	Long-term
	outcome	environmental information and	of various types of revenues	environmental data /knowledge	environmental issues	improvement/change of
		warnings	based on environmental data			reputation and image
				Increased trust	Increased responsibility in
		Increased quality and reliability of			decision making (individual, firm
		environmental data			and policy level)
		Increasing number of business start-
		ups based on environmental data;
		Increasing number of new jobs

Table 2 .

 2 A multi-actor framework to analyse the impacts from the perspective of different actors

		Industrial and technical	Market and financial	Relational	Responsible	Reputation
	Private service provider	Knowledge transfer: Better use of	New revenues	Mutual trust and co-	Sustainable business	Image as a successful and
		academic knowledge as a		creation between multiple		international start-up
		business opportunity	Access to international markets	partners	Awareness: corporate social	company
					responsibility in firm level
		Use of multiple data sources to	Export incomes	Opportunity for new roles	decision making	Socially responsible company
		create new business		and responsibilities		(as a competitive advantage);
				(intermediator)	Predictive information of	corporate social responsibility
		Variety of commercial			environmental changes
		applications			(managing risks)
	Public service provider	Access to the big data	New (research)markets based	Mutual trust and co-	Predictive information of	Internationally high level
			on multifaceted environmental	creation between multiple	environmental changes	environmental knowledge
		Knowledge transfer between	data	partners		(publications)
		multiple partners			Respond to the social needs
				Opportunity for new roles		Responsible public
		New profound and multifaceted		and responsibilities	Safety, security and health	organisation
		knowledge of environment and		(enabler, intermediator)	benefits
		atmosphere
		New public services
	End user	Access of to the multiple data	Free access to public data	Change in role: users as	Usability of environmental	Role as an active and
		sources and big data		data providers	information	empowered citizen
			Expenditures (applications, new
		Receive and provide real-time	devices)	Influence; ability to	Awareness and sensitivity to	Influential citizen
		environmental data (e.g. via smart		participate	ecological issues
		phone)
				Personalised environmental	Empowered citizen
		Better (measures) health		information based on your
				location)	Better health (experienced)
		Job opportunities
	Society (Policy makers)	New environmental knowledge	Tax revenues (new companies,	Trust in innovative	Creation of innovative,	Responsibility in decision
			new taxpayers)	networks consolidates the	creative and entrepreneurial	making
		New start-ups created		innovation networks	culture
			Investments; public			Innovation system: enabler
		New jobs created; employment	expenditures (e.g. renewed		Environmental education	for creativity, services and
		rate	educational infra)			innovations
					Transparency in society
		Growth of services,	Export share of GDP			Democratic society
		"servitization" , service society			Social impacts e.g. health,
			Market share of services (%)		living conditions
		New business ecosystems created
			Economical growth and		Quality of societal services
			productivity

5th InternationalConference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen


Silvia Cruz
email: silviacruz@ige.unicamp.br.

Sônia Paulino
email: sonia.paulino@usp.br

Faïz Gallouj
email: faiz.gallouj@univ-lille1.fr

Services and new societal challenges: innovation for sustainable growth and welfare

Keywords: ServPPIN, public service innovation, clean development mechanism, solid waste sector

This paper is devoted to the discussion of public services innovation in the Brazilian municipal solid waste sector, with emphasis on multi-agent participation within Clean Development Mechanisms (CDM) projects. The empirical context is based on six landfill CDM projects located in the São Paulo Metropolitan Area, Brazil. CDM projects have a dual purpose: reducing greenhouse gas (GHG) emissions and promoting local sustainable development in host countriesthrough the promotion of local co-benefits. The discussion is based on the analytical model provided by the ServPPIN concept (public-private innovation networks in services). It focuses on the characterization of the landfills selected and on the identification of the stakeholders involved within these landfills, pointing out the participation gaps. The results indicate that the participation of associations and cooperatives surrounding landfills is still marginal. Pulling this theoretical (ServPPIN) and empirical research (landfill CDM project) together, one can identify the main factors affecting the establishment of basic conditions for service innovation: a) interactions and the building of social relations aimed at innovation among various stakeholders; b) the development of competences on several fronts; especially relational and organizational; c) the role of the public sector (coordination role) in supporting the development of successful public-private innovation networks in services.

Introduction

This paper is devoted to the discussion of public services innovation in the Brazilian municipal solid waste sector, with emphasis on multi-agent participation within Clean Development Mechanisms (CDM) projects. The empirical context is based on six landfill CDM projects located in the São Paulo Metropolitan Area, Brazil. It comprises 39 municipalities, and approximately 20 million inhabitants, generating around 16 thousand tonnes of solid waste per day [START_REF] Cetesb | Inventário Estadual de Resíduos Sólidos Urbanos[END_REF].

The discussion is based on the analytical model provided by the ServPPIN concept (public-private innovation networks in services) [START_REF] Gallouj | Innovation in Services[END_REF][START_REF] Bučar | Weak institutional framework as incentive for service innovation networks: focus on knowledge-intensive business services[END_REF]Djellal and Gallouj, 2013;[START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF][START_REF] Labarthe | Effects of institutions on the integration of end-users' knowledge in ServPPINs: lessons from two case studies in agro-environmental knowledge-intensive services[END_REF]. Beyond the relational aspect (interfaces and feedbacks) among various public and In Brazil, the specifications of a project in order to promote local sustainable development are based on information from Annex III of Resolution n.1 of the Inter-Ministerial Commission on Global Climate Change2 . This establishes both the benefits to the local area and how project activities contribute to each of the following aspects: environmental sustainability; improvement in working conditions and net job creation; income distribution; training and technological development; and finally, regional integration and working in conjunction with other sectors (Brazil, 2003). However, analyzes performed on the promotion of local development, based on the CDM projects in the solid waste sector, indicate that most of the projects do not contribute significantly to the achievement of local sustainability [START_REF] Sutter | Does the current CDM deliver its sustainable development claim? An analysis of officially registered CDM projects[END_REF][START_REF] Siebel | Clean Development Mechanism -A Way to Sustainable Waste Management in Developing Countries? Österreichische Wasser-und Abfallwirtschaft[END_REF]. In this direction, the importance of civil society's effective participation throughout the approval process of a CDM project is noticeable.

CDM projects are not implemented with the explicit goal of promoting service innovation; however, these projects can create leverage and open new possibilities to improve services related to urban solid waste, converging with the National Policy on Solid Waste (Brazilian law nº 12,305/2010) requirements and social and environmental co-benefits generation from carbon market projects.

In this manner, there is need for the development of activities which focus on the steps for generation and treatment of solid waste, preceding the final disposal of waste, searching for alternatives to reduce the generation of waste, encouraging actions for reuse, recycling, and treatment.

Besides these, landfill CDM projects can improve management practice during the operation of landfills (air pollution, groundwater contamination, biogas management, visual, freight vehicle access arrangements, etc) as well as improve on activities carried out after landfill closure, in order to fulfil with the closure plan of the landfill area (maintenance of the vegetation, landfill cover system etc.).

Activities carried out in the projects developed in landfills involve various agents: concession-holders responsible for managing the landfill and capturing landfill gas (LFG); municipal departments; and representatives of associations from the communities surrounding the landfills. The importance of civil society's effective participation throughout the approval process of a CDM project cannot be overstated. Indeed, project proponents should send letters of invitation to all project stakeholders. The following agents should be considered in the case of Brazil: local authorities and chamber of deputies for all municipalities involved; municipal and state environmental departments; NGOs and social movement forums; community associations both directly and indirectly involved in project activities; and State and Federal Public Prosecution Offices.

In this context, this paper intends to analyze how the promotion of local co-benefits is linked to the promotion of innovation in urban solid waste services, emphasizing the participation of civil society, highlighted in the Kyoto Protocol and in the ServPPIN concept. The paper is structured into four sections. Following the introduction, section 2 discusses the public services through the ServPPIN concept, focusing on the multi-faceted and heterogeneous nature of the public sector and on the role of the civil society in innovation in public services. Section 3 is devoted to the presentation of empirical cases, focusing on the characterization of the landfills selected and on the identification of the stakeholder involved with these landfills, pointing out participation gaps, taking into account the following criteria: consultation, benefits, and interaction/interface. Our conclusions are set out in section 4.

ServPPIN concept and the role of the civil society in innovation in public services

The purpose of this section is to present: 1) specific characteristics of public sector and public service innovation; 2) the concept of public-private innovation networks (ServPPIN), highlighting the importance of multiple links and feedbacks between the public and private sectors, and also with service users and; 3) the role that third sector organizations/end users (citizens) have to play in the ServPPIN context.

Innovation in Public Services

There is an increasing scientific literature on innovation in the public sector. However, the literature generally neglects or underestimates the role of the public sector in the innovation process (Djellal et al, 2013, Windrum andGarcía-Goñi, 2008;[START_REF] Brandão | Inovação no setor público: análise da produção científica em periódicos nacionais e internacionais da área de administração[END_REF][START_REF] Osborne | Public sector innovation research: what's next? Innovation: Management[END_REF].

The public sector is often portrayed as a facilitator of innovation activities, merely drafting the legal framework, among other actions which do not include a leading role in the innovation process [START_REF] Gallouj | Innovation in Services[END_REF][START_REF] Mulgan | Innovation in the Public Sector, Strategy Unit[END_REF][START_REF] Hartley | Innovation in Governance and Public Services: Past and Present, Public money & management[END_REF][START_REF] Halvorsen | On the differences between public and private sector innovation[END_REF][START_REF] Koch | Innovation in the public sector[END_REF]Windrum and García-Goñi, 2008;Potts and Kastelle, 2010;[START_REF] Fuglsang | Bricolage and invisible innovation in public service innovation[END_REF][START_REF] Sundbo | Public-private innovation networks in services: the crucial role of entrepreneurial fit[END_REF]Djellal and Gallouj, 2013;Djellal et al. 2013;[START_REF] Osborne | Public sector innovation research: what's next? Innovation: Management[END_REF].

Although the idea has long been prevalent that public services are produced, predominantly, by the public sector, more recently there is improved clarity that public service provision generally involves a more complex and broad range of actors and institutions: public services can be provided by public actors, private actors, or both [START_REF] Di Meglio | The place of ServPPINs in the range of public-private collaboration arrangements for services provision[END_REF]. In this manner, systematic efforts will be required to foster innovation in the public sector [START_REF] Bloch | Public sector innovation -From theory to measurement[END_REF], and, therefore, in public services. [START_REF] Djellal | Beyond Productivity Measurement and Strategies: Performance Evaluation and Strategies in Services[END_REF] highlight the importance of public services according to the output produced by these services, which can be called social or civic outputs. According to the authors -These ‗outputs' contribute to social cohesion, solidarity and collective and civic identity‖ (Djellal and Gallouj, 2012, p. 11).

Another aspect of public service provision is the difficulty of identifying and demarcating activities that should be attributed to the public sector as opposed to the private sector, as many services and activities in the public sector are integrated with the activities of the private sector and vice versa [START_REF] Bugge | The public sector in innovation systems, Project -Measuring Public Innovation in the Nordic Countries: Toward a common statistical approach[END_REF]Potts and Kastelle, 2010). This aspect is observed also for services related to municipal solid waste that generally is carried out through concessions to the private sector.

According to [START_REF] Bugge | The public sector in innovation systems, Project -Measuring Public Innovation in the Nordic Countries: Toward a common statistical approach[END_REF] and [START_REF] Bloch | Public sector innovation -From theory to measurement[END_REF] the multi-faceted and heterogeneous nature of the public sector is a result of the multiple interfaces which characterize public organizations: 1) the interface with the private sector; 2) the interface between the public sector and citizens; and 3) the internal interfaces within the public sector (between government levels and between different areas of activity). These various interfaces, illustrate public sector heterogeneity, and also the permeability between organizations (private, public, and third sector) [START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF].

Having outlined the characteristics of multiple interfaces and heterogeneity of the public sector, the publicprivate innovation networks in services (ServPPIN) concept can be useful in understanding the logic of public service innovation.

Public-Private Innovation Networks in Services (ServPPIN)

One of the factors encouraging interest in examining Public-Private Innovation Networks is the growing recognition of the important role played by public sector organisations in the innovation process. Public administrations are thus no longer restricted to playing a supporting role in the innovation process. They may be active participants in that process, particularly insofar as their own activity is concerned.

Just like public-private partnerships (PPPs), ServPPINs are networks of collaborative partnerships between public and private organizations. However they are more comprehensive, open and flexible than traditional PPPs, which entail relations between actors that are more rigid, with predefined functions, rules and formal procedures (particularly contracts) that can bureaucratize the process, and limit the potential for innovation.

The high number, and diversity, of participants in a ServPPIN can lead to a complex and intensive process of interaction in which a large amount of heterogeneous information and knowledge (tacit and non-tacit) are likely to be exchanged, since plenty of channels are opened for interaction. In other words, ServPPIN can be thought of as a multi-agent service relationship system (Djellal and Gallouj, 2013).

The innovation network 3 concept is biased towards manufacturing and technology. This concept, in the traditional sense, suffers from a certain number of shortcomings; namely: a technology bias (with largely predominance of tangible technological innovation), a manufacturing bias (linked to the previous one), and a market bias (the private sector is central to innovation dynamics). This means in short that innovation networks are mainly focused on technological innovation produced by the collaboration of private actors in the manufacturing sector.

The ServPPIN concept provides a way of overcoming these various biases (Djellal and Gallouj 2013;[START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF][START_REF] Labarthe | Effects of institutions on the integration of end-users' knowledge in ServPPINs: lessons from two case studies in agro-environmental knowledge-intensive services[END_REF][START_REF] Windrum | Innovation and entrepreneurship in public services[END_REF]. It goes beyond the technologist view of innovation4 . Its perspective is broader, incorporating non-technological types of innovation such as: organizational, ad hoc (defined by [START_REF] Gallouj | Innovation in Services[END_REF] as the interactive solution to the specific problems of particular clients), social, and bricolage innovation (defined as innovation through non-programmed activities, trial-anderror processes and adaptation to random events - [START_REF] Fuglsang | Bricolage and invisible innovation in public service innovation[END_REF]).

Through the ServPPIN concept, Di Meglio (2013), [START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF] and [START_REF] Langergaard | Specificities of public sector service innovation[END_REF], emphasizes the importance of multiple links and feedbacks (interfaces) between public and private sector, and also with users -in our case, representatives of associations from the communities' surrounding landfillsand policy makers for service innovation. As pointed out by Djellal and Gallouj (2013):

ServPPINs are multi-agent service relationship systems. The actors involved in interaction have to deal with the illdefined nature of their respective products, their non-stockability, a diversity of systems of interaction, the multiplicity of possibly competing value systems and the fact that their products are located in different spatial and temporal scales. ServPPINs introduce the traditional research questions of service economics into network-based analyses of innovation. (Djellal and Gallouj, 2013, p. 30 highlighted by the authors).

According to [START_REF] Bučar | Weak institutional framework as incentive for service innovation networks: focus on knowledge-intensive business services[END_REF], ServPPINs can be understood as a place for social interaction and the construction of social relations aimed at innovation. Nevertheless, of all case studies provided by the ServPPIN research project5 , several were not explicitly oriented towards the innovation target. For example, in some hospital case studies the main objective was to reduce costs in the use of technologies [START_REF] Farias | Technology adoption in service organizations: a framework proposal for studying ICT diffusion in healthcare and hospital services[END_REF][START_REF] Schartinger | An institutional analysis of innovation in healthcare services[END_REF][START_REF] Windrum | Innovation and entrepreneurship in public services[END_REF].

Similarly the CDM landfill projects we examined were implemented in order to reduce GHG emissions, rather than having the explicit goal of promoting service innovation. In fact, the technology and innovation outcomes appear to be additional results of the local co-benefits generation. Interesting changes and innovations are likely to arise out of such networks -which can then be retrospectively labelled ‗innovation networks'. Taking into account non-technological, incremental and non-programmed innovations (ad hoc, bricolage, rapid application etc.) it is also possible to consider even those networks which are not explicitly (or immediately) oriented towards innovation to be ServPPINs.

The ServPPIN contributes to opening up the traditional innovation network concept to new actors: all market services, as well as third-sector organisations (NGOs, associations etc.). It extends potential forms of participation for certain actors, for example, the involvement of civil society in the decision-making and consultation processes. This is particularly the case with regards to the role civil society plays in the ServPPIN contexthelping to translate social preferences which are not merely reflected by market prices [START_REF] Fuglsang | Collaboration and trust in a public-private innovation network: a case study of an emerging innovation model[END_REF].

Public Service Users and Their Involvement in Service Innovation

The innovation process involves several players who contribute in different ways in the development of service innovations. In this sense, the involvement of customers/service users as ‗partners' in the process of service innovation development is highlighted, which can be achieved through dialogue with service users, joint experimentation, panels and other communication tools [START_REF] Hertog | Capabilities for managing service innovation: towards a conceptual framework[END_REF]2010). It is noteworthy in recent years that the participatory governance mechanisms have been widely promoted in developing countries [START_REF] Speer | Participatory Governance Reform: A Good Strategy for Increasing Government Responsiveness and Improving Public Services?[END_REF]. These mechanisms seek to involve citizens in decision-making regarding the allocation of public resources among communities, shaping public policy, as well as involvement in the monitoring and evaluation of government expenditure.

According to [START_REF] Speer | Participatory Governance Reform: A Good Strategy for Increasing Government Responsiveness and Improving Public Services?[END_REF], one of the key reasons cited for participatory governance mechanism implementation is related to the improvement of public services. Given that when citizens are empowered and democracy strengthened there is a tendency to an increase in local government responsiveness and accountability. This process tends to improve the efficiency and sustainability of public service delivery, since it might have the ability to match public services to user preference.

Participatory governance mechanisms can improve information flows in two ways: from citizens to governments about citizen preferences/demands; and from governments to citizens about government decisions and actions, as well as about service provision outcomes. [START_REF] Lehtonen | Multiple Voices of the User in Public Sector Services[END_REF] emphasize the collective preferences of citizenship, with a broader vision of the citizen, not only as a public service receptor, but also active in the production, control and planning of these activities. The authors note that active dialogue is required to negotiate and mediate under different citizen preferences. In this context, the relevance of user integration in the service innovation process it is brought to attention.

However, ServPPIN research Project results [START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF] show the limitation of the third sector organizations' participation and that the involvement of service end users (citizens) is marginal. Thus, especially regarding to the role that civil society has to play in ServPPINassisting in the translation of social preferences there is a need to empower public users in order to foster successful cooperation.

Schilling (2011) points out that improvements in cooperation among the agents is coupled with the need to manage knowledge from different sources for successful service innovation, and also requires the capacity to sustain a constant flow of communication between the many agents [START_REF] Green | A life cycle-based taxonomy of innovation networkswith a focus on public-private collaboration[END_REF], which can be achieved through the establishment of routines and communication channels 6 . The important role of information technology in the communication and processing of information in the service innovation process is also emphasized [START_REF] Di Meglio | The place of ServPPINs in the range of public-private collaboration arrangements for services provision[END_REF].

Thus, to analyze the participation as well as the role of civil society in public-private service innovation networks it is important to sustain an adequate standard of public services, taking into account the quality of service and social needs, particularly with regards to municipal solid waste services from the case of landfills with CDM projects.

In order to explore participation in the empirical context studied, as well as proposed by [START_REF] Labarthe | Effects of institutions on the integration of end-users' knowledge in ServPPINs: lessons from two case studies in agro-environmental knowledge-intensive services[END_REF], the end-users are not individual users that are integrated within the networks. It is rather some collective organizations representing them (here representatives of associations from the communities surrounding landfills), which are members of the ServPPINs, as presented in the empirical results section.

Empirical results: the Brazilian landfill CDM projects

Of the 331 Brazilian CDM projects registered, 51 are developed on landfills, 23 of which are located in the State of São Paulo (UnepRisoe, December 2014). Our research focuses on the São Paulo Metropolitan Area, which is one of the five largest urban conurbations in the world, and the largest in Brazil, with the city of São Paulo as the main nucleus. It comprises 39 municipalities with, approximately, 20 million inhabitants, 55.4% in the city (municipality) of São Paulo.

The landfills for the empirical research were selected on the basis of the following criteria: a) carbon market Brazilian projects: regulated and voluntary carbon market; b) projects scope: landfill projects; c) Localization: São Paulo Metropolitan Area; d) Methodology used to measure GHG emission reduction: ACM0001flaring or use of landfill gas; e) Monitoring period verified: with at least one monitoring period verified until the beginning of the empirical research (February, 2014). Through the prior criteria mentioned, the landfills selected are: Bandeirantes, São João, Caieiras, Itapevi, Pedreira, and Lara.

The Particularly for the analysis of the participation of associations, representatives of associations from the communities surrounding landfills, and recycling cooperatives, data were collected through semi-structured interviews organised into three categories to discuss opportunities for service innovations: consultation, benefits, and interface/interaction (Table 1).

Tab. 1 -Categories and items addressed to discuss opportunities for service innovations

Category

Items Addressed

Consultation

Participation before and after project implementation; public consultation (public audiences, surveys, questionnaires, workshops, visitation, panels); PDD comments; language and clarity of documents; language used; community engagement Benefits Cooperatives benefiting from CDM revenues; contributions of CDM projects to environmental education programs Interaction/ Interface Proposed meetings; disclosure of the annex III activities to stakeholders; channels for recording complaints; the coordination role achieved by the public sector Source: The authors

The interviews were conducted, in the first semester of 2014, with 4 associations and 3 waste-picker cooperatives engaged in the solid waste issue.

Stakeholders Participation under the Carbon Market Projects

With regards to participation, specifically for carbon market projects, the CDM Executive Board requires stakeholders' participation in the whole process of the activity development, fulfilling: 1) Project proponents should send letters of invitation to all project stakeholders; 2) Summary of information should be presented by the stakeholders; 3) Confirmation by the host country that the project assists in the achievement of sustainable development; 4) Report made of DNA justifying how the comments provided were taken into account.

The project information must be available in an appropriate manner, ensuring data accessibility, in a language and framework that can be understood by all stakeholders.

Even though stakeholder participation is explicitly considered by the CDM Executive Board, the process is still considered insufficient, in terms of contemplation of the actors, and in terms of measuring participation throughout the project cycle.

The participation of stakeholders in CDM projects has been widely challenged at the international level also (CAN, 2011; [START_REF] Watch | Manual para o Mecanismo de Desenvolvimento Limpo (MDL)[END_REF][START_REF] Foronda | Análisis y mejora de la participación de comunidades locales em proyectos de mercado de carbono[END_REF][START_REF] Subbarao | Can the Clean Development Mechanism (CDM) deliver?[END_REF][START_REF] Kolmuss | Improving the Stakeholder Consultation Process: an NGO perspective[END_REF] indicating that there is still a gulf to be breached, due to the intense asymmetrical information among stakeholders and the lack of a clear understanding about how the resources from the sale of Certified Emission Reductions (CERs) should be applied, added to the unpreparedness of public bodies to clarify questions regarding the carbon market. Table 2 shows some elements related to public consultation, openness and transparency of information to support improvements in the participation of stakeholders.

Tab. 2 -Elements for stakeholder participation in carbon market projects

Topic Public Consultation Openness and transparency of information Elements

Quality and range of communication tools between the project proponent and stakeholders (public audiences, surveys, questionnaires, workshops, visitors, panels, among other); Quality and range of publicity needed to achieve stakeholders; Frequency and timing; Period for Consultation; Place and time selected Anticipation in the disclosure; means of divulgation; accessibility of language and language used Source: based on [START_REF] Monzoni | Critérios de sustentabilidade para projetos MDL no Brasil. Colaboração Silvia Llosa[END_REF] and CDM Watch7 (2010).

In 2011, through public consultation, the United Nations Framework Convention on Climate Change (UNFCCC) conducted a study aiming to contribute towards the inclusion of different actors in CDM projects.

The key findings of the public consultation were:

-The comments of the stakeholders should be considered during the design of the project; -The first meeting with the stakeholders should be taken prior to the submission of the PDD; -Validation and verification of the project to ensure the required benefits; and -The establishments of mechanisms for affected stakeholders to express their demands. The elements outlined may contribute towards the fulfillment of the requirements related to the complete and effective involvement of communities surrounding landfills (and other stakeholders) on the project, through access to information and participation in the decision making process.

According to Decision 3/CMP.1 (Marrakech accords), UNFCCC defines stakeholders as: -the public, or any individuals, groups or communities affected or likely to be affected, by the proposed CDM project activities‖. Therefore the Designated Operational Entity (EOD), independent audit, responsible for the validation and verification of projects, should revise the PDD and other relevant documents for the validation and verification of the project, in order to evaluate, beyond the technical issues, if the requirements outlined by the CDM executive board in relation to stakeholders' participation8 are being met.

Characterization of the landfills and identification of stakeholder

The characterization of the landfills selected for the empirical research is shown Table 3. Source: The authors * These data are based on the preliminary modelled/projected emission reductions from the PDD The PDD section named -Stakeholders consultation9 ‖ was investigated for each landfill selected as an empirical case, in order to map the stakeholders. A weakness was verified in the PDD, particularly in relation to the actors considered as "potentially affected populations", i.e, communities surrounding landfills.

In the PDD -Stakeholders consultation‖ section, the stakeholders representative of civil society largely cited was the Brazilian Forum of NGOs and Social Movements for the Environment and Development; in other words, although the Forum is widely recognized for actions and activities aimed at protecting the environment and promoting sustainable development, it is generally for the purpose of this research to consider it as a stakeholder.

Thus, specific information about representatives of associations from the communities surrounding landfills directly affected by the activities of the enterprises was researched beyond the project design documents and the UNFCCC website, as well as direct searches on Google being used (web search engine tool) with the keywords: landfills name; cooperatives; associations/communities surrounding landfills.

In addition, the following databases were checked: Figure 1 shows, specifically, the stakeholders' mapping of the representatives of associations from the communities surrounding landfills. From this mapping (addressed previously in Fig. 1), Table 4 shows the representatives of associations selected. For the public sector, stakeholders include the city halls and municipal sub districts, as well as the departments related to CDM projects issues. For the private sector, the stakeholders include the concessionholders responsible for landfill management in the case of public landfills, the company that owns the landfills in the case of private landfills; and the concession-holders responsible for biogas recovery and power generation.



Tab. 4 -Representatives of associations selected

Participation Gaps

The role and participation that civil society has to play in the ServPPINhelping to translate social preferencesare fundamental to promote service innovation. Besides that, within the CDM projects context, participation is considered a core issue for the project to comply with the goal of promoting local sustainable development.

According to data collected from the representatives of civil society, the communities surrounding landfills lack information regarding the existence and operation of the CDM landfill projects; in addition there is no knowledge about local benefits promotion that the projects must perform. The major gaps identified are addressed in the following topic.

a) Consultation

The period designated for stakeholders to manifest their opinion about the project (30 days, set by UNFCCC) is considered inappropriate by the interviewees' representatives of Bandeirantes, São João, and Itapevi landfills. The interviewees of the other landfills were unaware about the availability of documents related to the CDM projects to be accessed and commented on.

Other issues raised are related to: difficulties in obtaining documents in the UNFCCC and MCT&I online platforms; the lack of documents available in the native language; and the technical language used in the documents related to the projects. Without a common language, surrounding communities may remain unaware of any possible benefits arising out of project implementation. Furthermore, the establishment of a common language can improve exposure of the different interests at stake. The asymmetric information problem diminishes, or even hampers, an effective participation of the associations of residents surrounding landfills.

With the exception of the Bandeirantes and São João landfills, which both held public audiences and meetings, there is no information on the existence of meetings, workshops and other types of mechanisms to facilitate the participation at the other landfill sites. However, even in the Bandeirantes and São João landfills cases, it is emphasized that the respective communities suffer difficulties to voice their claims in an effective manner: The views of the municipality and the operating company's technicians usually prevail.

It is thus necessary to develop the competence of the associations (from the communities surrounding the landfills) to discuss technical indicators about landfill operation and closure plans, as well as other alternatives for final waste disposal. There is a need on the part of technicians at the municipal departments, concessionholders, and CDM project developers to ensure that data communicated about CDM projects can be easily understood by the communities surrounding the landfills.

With regards to the PDD section named -Stakeholder consultation‖, only the Bandeirantes landfill did not receive comments from stakeholders. For other landfills, the comments were generally performed by Municipal Departments of Environment, Public prosecution, and state and municipal environmental bodies.

Only the Itapevi landfill received comments from the Brazilian Forum of NGOs, pointing out the need for improvements in local stakeholder involvement and the importance of setting up transparency mechanisms related to the projects; and Itapevi Public Prosecution declaring that the PDD of the landfill was not submitted in a local consultation process. Therefore, it is observed that the comments received mostly do not regard the stakeholders as set out by the Kyoto Protocol: Individuals, groups or communities affected or likely to be affected, by the proposed CDM project activities.

b) Benefits

According to the preferences of representatives of civil society, the projects should contribute to reduce or to avoid negative externalities generated by landfills. That is, to reduce pollution; to reduce risks (related to the activities performed by landfills); to increase quality of life; to ensure cooperatives and environmental education programs benefit from the resources of CDM.

In the PDDs, only Caieiras and Lara landfills mentioned the proposition of future projects to be carried out from the CER resources. Despite this, the absence of means for monitoring/supervising, if these proposals are to be implemented, is brought to attention.

The cooperative of Itapevi, was awarded with equipment for compacting waste; and the Caieiras landfill transfers 1% of the CER resources [START_REF] Candiani | Análise Da Implantação Das Medidas Mitigadoras Estabelecidas No Estudo De Impacto Ambiental Da Central De Tratamento De Resíduos -Caieiras -São Paulo[END_REF] to the Municipal Social Fund to be invested in the areas of education, health and environment. For the other landfills, there is no information about cooperatives and associations that benefited with the resources of the projects.

Specifically for Bandeirantes and São João landfills, problems of access to revenue from the sale of Certified Emission Reductions are noteworthy. The access to CER resources intended for FEMA12 occurs through public tenders. However, sub-districts need to develop their competence in to be able to publicize good quality projects and thus access revenue from the sale of carbon credits.

ServPPINs require increased from the public sector. Therefore, particularly in relation to the means for accessing the resources from CER (public tenders), it is recommendable to review the public tenders, adapting the requirements to the real situation of the cooperatives; to propose ways to support the cooperatives (i.e. action plans to capacitate the cooperatives to be able to design good quality projects and thus access the revenues from the sale of carbon credits); to prevent that bureaucratic issues hinder access to CER resources.

c) Interaction/Interface

While evaluating the existence of communication channels for receiving complaints, inquiries and comments from stakeholders, it is pointed out that there are no effective channels for participation of civil society associations and cooperatives surrounding landfills throughout the stages of the project cycle. Some of the complaints mentioned by the interviewees affected by the São João, Bandeirantes, Itapevi, and Caieiras landfills are related mainly to: the odor from the landfills; the misdirection of the resources derived from the sale of CERs; and the lack of benefits generated from the projects for the community. Moreover, it is observed that there is a need to publicize activities related to CDM projects and implement an effective communication channel that would improve discussions, negotiations and disclosure to the various stakeholders via public consultations and dialogue mechanisms.

Largely, the interactions are restricted to public audiences and meetings concerning the application of the resources from the sale of the carbon credits generated by Bandeirantes and São João CDM projects. At the other landfills, the interface is considered only by the availability of documents in the UNFCCC and MCT&I online platforms to receive stakeholders' comments about the projects. As highlighted by the ServPPIN concept, it is essential to empower the public sector and civil society for cooperation. In addition, it is important for the public sector to perform network coordination in an appropriate manner.

Conclusion

The ServPPIN concept highlights the importance of interaction/interface among public and private organizations and mainly, third sector organizations, for developing innovative services.

In this way, through the application of the ServPPIN concept, it was possible to identify and analyze the factors related to the establishment of fundamental conditions for public service innovation in the municipal solid waste sector through the implementation of landfill CDM projects.

For the analysis, the following variables were considered: consultation; benefits; interaction/interface.

 Regarding consultation, the main gaps identified are related to the period for stakeholders to voice their opinion about the project -30 days, as set out by UNFCCC, is considered inappropriate. Other issues raised are related to: difficulties in obtaining documents in the UNFCCC and MCT&I online platforms; the lack of documents available in the native language; and the technical language used in the documents related to the projects (generating asymmetric information). The comments pointed out in PDD section -Stakeholder consultation‖ in general, do not regard the stakeholders as set out by the Kyoto Protocol (individuals, groups or communities affected or likely to be affected, by the proposed CDM project activities).

 Regarding the benefits provided by project implementation, the absence of means for monitoring/supervising is highlighted, if the proposals described in the documents related to the projects are to be implemented. Additionally, and specifically for Bandeirantes and São João landfills, problems are noteworthy of access to revenue from the sale of Certified Emission Reductions, which occurs through public tenders, and the need to develop competences is highlighted, in order to make the cooperatives able to design good quality projects and, thus, access the revenue from the sale of carbon credits.

 Regarding the interaction/interface, the absence of effective channels for receiving complaints, inquiries and comments from stakeholders and the need to publicize activities related to CDM projects throughout the whole crediting period are noteworthy. For the Bandeirantes and São João landfills, interactions are restricted to public audiences and meetings concerning the application of resources from the sale of carbon credits. For the other landfills, interface is only considered from the availability of the documents in the UNFCCC and MCT&I online platforms.

Overall, the results presented corroborate with the results shown in the international literature regarding stakeholder participation under the carbon market projects (CAN, 2011;[START_REF] Watch | Manual para o Mecanismo de Desenvolvimento Limpo (MDL)[END_REF][START_REF] Foronda | Análisis y mejora de la participación de comunidades locales em proyectos de mercado de carbono[END_REF][START_REF] Subbarao | Can the Clean Development Mechanism (CDM) deliver?[END_REF][START_REF] Kolmuss | Improving the Stakeholder Consultation Process: an NGO perspective[END_REF], and in the ServPPIN research Project results [START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF] pointing out that service end user (citizen) involvement is marginal.

ServPPINs enable potential complementarities and synergies to be explored in various modes of cooperation between public, private and civil society organizations. In this way, in the instances studied, it is essential to empower the public and private sectors and civil society. Besides this, it is important the public sector performs the coordination of the network in an appropriate way. The proactive role of the public sector as a network leader turns out to be a very important determinant of success in the CDM landfills cases, as instruments capable of contributing to public service innovation in the municipal solid waste sector.

Pulling this theoretical (ServPPIN) and empirical research (landfill CDM project) together, once can identify the main factors constraining and affecting the establishment of the basic conditions for service innovation. Based on the set of case-studies investigated, lessons have been drawn concerning:

a) The interactions and the building of social relations aimed at innovation among various stakeholders involved;

b) The development of competence on several fronts, especially, relational and organizational; and c) The role of the public sector (mainly, the coordination role) in supporting the development of successful public-private innovation networks in services.

  data on the CDM landfill projects, from 2003 to 2014, were obtained through documentary research in three databases: United Nations Environment Program (UNEPRisoe), and UNFCCC CDM Registry, for accessing monitoring reports and project design documents; and The Ministry of Science, Technology and Innovation (Ministério da Ciência e Tecnologia e Inovação-MCTI) database for accessing the project documentation, based on Annex III of Resolution No. 01/ 2003 of the Inter-ministerial Commission on Climate Change (CIMGC) (document that describes the promotion of social and environmental co-benefits).

  Recyclable material Cooperatives listing, available in the website Department of Environment of São Paulo State -SMA 10 ;  -Registration of civil society organizationsdatabase of 2009-2011‖ listing, available in the integrated Water Resources Management System (SIGERH) 11 , taking into account mainly the data related to the watershed committee of Alto Tietê, regarding the geographical area selected for study;  Websites of the municipalities at Itapevi, Mauá, and Caieiras;  Websites of the subdistricts São Mateus, Perus, and Tremembé (city of São Paulo);  Minutes of public audiences related to carbon credits;  FEMA (Environment and Sustainable Development Fund) and Confema (Special Fund for the Environment and Sustainable Development Council) resolutions related to CDM projects (in this case, only for Bandeirantes and São João CDM projects).

Fig. 1 -

 1 Fig. 1 -Stakeholders' Mappingassociations from the communities surrounding landfillsSource: The authors.

  Founded in 2003 with the purpose of organizing the popular participation and demand improvements for the region of Perus.Initiative of the residents of São Matheus district that have mobilized against the bad odours and the possibility of accidents occurring due to the activities performed by the landfill

			Founded in 2006 with the purpose to enable the selective waste
	Pedreira	Cantareira Viva Cooperative	collection, income generation, and stimulate improvements in Vila Albertina the region (a neighborhood in the Jaçanã/Tremembé
			district, São Paulo)
	Itapevi	Itapevi Cooperative	Founded in 2003 supported by the Municipal Department of Environment. Currently with 20 registered cooperative members
	Lara	Association of Literacy Movement and Adults (AMOVA)	Founded in 2002 in the city of Mauá; is focused on providing housing to low income people, through partnerships with Caixa Econômica Federal (Brazilian bank) and the Ministry of Cities
		Source: The authors.
	Landfill	Representatives of Associations	Description
	Bandeirantes	Perus and Anhanguera Forum for local sustainable development
		Chico Mendes Cooperative	Founded in 1999 with the purpose employment generation for people on low income in the East Zone of São Paulo
	São João	More Life, Less Rubbish
		Campaign (Campanha Mais Vida
		Menos Lixo)
	Caieiras	Pinheiros Philanthropic Association	Founded in 1993 with the purpose to assist the local residents of Caieiras region (around 450 families)

Inter-Ministerial Commission on Global Climate Change is the Designated National Authority (DNA) in Brazil.

Innovation networks and systems have been the subject of an extensive literature in economics, sociology and management(Callon, 1992; Edquist, 1997; Latour, 1999; Lundvall, 1992; Nelson, 1993 among others).

The technology issue is taken into account in the designs of the carbon market projects and regarding the promotion of social and environmental local co-benefits, in the cases studied, all Project Design Documents (PDD) and Validation Reports indicate the development and diffusion of technologies through the project's implementation, highlighting the following aspects: training, technology development and transfer.

ServPPIN is an EU-funded research project which focuses on the role of public and private services on growth and welfare and the particular role of public-private innovation networks. For more information: http://www.servppin.com/. The main results of the project are also published in[START_REF] Gallouj | Public Private Innovation Networks in Services[END_REF].

In the majority of ServPPINs cases analyzed byDjellal and Gallouj (2013) it is clear that organizational innovations are related to better sharing of information, establishment of communication channels, etc. Exemples: Capacity Planning (UK)in the health sector; ITS Vienna Region (AT1), DoRIS (AT3), VIATIC (FR1), and Compano (AT2)in the transport sector.

The CDM Watch is an organization created in 2009 with the purpose of supervising the actions performed under the CDM, particularly aimed at strengthening civil society participation processes.

The requirements are: Project proponents should send letters of invitation to all project stakeholders; Summary of information presented by the stakeholders; Confirmation by the host country that the project assists in the achievement of sustainable development; Report of DNA justifying how the comments provided were taken into account.

This section includes: A brief description of how comments by local stakeholders have been invited and compiled; a summary of the comments received; and a report on how due account was taken of any comments received.

<www.ambiente.sp.gov.br>

<www.sigrh.sp.gov.br>

50% of the revenues from the Certified Emissions Reductions (CER) of the Bandeirantes and São João landfills are allocated to municipal government, specifically to the Environment and Sustainable Development Fund -FEMA, which is administered by the Municipal Department for the Environment.

This work was supported by Grant 2011/00081-5, São Paulo Research Foundation (FAPESP); CAPES Foundation/Ministry of Education of Brazil; and Centre Lillois d'Études et de Recherches Sociologiques et Économiques (CLERSÉ).