Boriana Rukanova
email: b.d.rukanova@tudelft.nl

Helle Zinner Henriksen

Frank Heijmann
email: fha.heijmann@belastingdienst.nl

Siti Arna

Arifah Arman

Yao-Hua Tan
email: y.tan@tudelft.nl

Public funding in collective innovations for publicprivate activities

Keywords: Innovation, Public Concern, Public Funding, International Trade

Whereas in market-driven situations the private parties have an interest in driving innovations towards implementation, in the case of public concerns, it is often the public concern that initiates the innovation process. The issue for the public funding agencies is then to stimulate idea generation and the process towards implementation and impact. However, these innovation processes are complex, as they involve a multiplicity of public and private actors with different and sometimes conflicting concerns. Thus, the benefits and business cases are not immediately clear and this makes it hard to scale beyond the proof of concept. In this paper we examine and derive lessons learned based on a longitudinal case study of a series four EU-funded projects (ITAIDE, INTEGRITY, CASSANDRA and CORE) in the international trade domain that aimed to develop digital trade infrastructure solutions (data pipelines) to address security and trade facilitation challenges. For our case analysis, we adapt and extend Bryson et al.'s framework [1] on cross-sector collaborations. We show how each of these projects covered one part of the public-private innovation trajectory, moving the innovation from the Initial R&D stage, to the Showcasing and dissemination stage to attract critical mass, towards a Turning point stage when the business cases for further upscaling become visible. We identify continuities (i.e. continuity of network & vision, funding and process) as well as a number of alignments as important factors that drive collective innovation processes towards implementation and impact. Further research is needed to establish to what extent these findings are applicable in other contexts.

Introduction

When we asked a manager from the Customs Administration of the Netherlands about the role of public funding, his response was very straightforward, namely that public funding should only be used for innovations that address public concerns and where the private sector is not able to provide the solution due to a lack of an immediate return on investment, and that public funding should not be used for initiatives that aim to advance the interests of individual commercial companies. While it is not the goal here to argue whether this view is right or wrong, this points to one of the very important functions of public funding, that is, to help address public concerns. It is often not easy to draw a clear line between what lies within the realm of a public concern and what does not. Public concern can be broadly seen as something that reflects the needs of society, for example safety and security, health, safe environment (including flora and fauna) and economic prosperity. These needs are sometimes equal to private commercial opportunities, but not always. And when they are not, public funding would be beneficial to drive R&D and innovation. Public funding would be valuable to enable public concerns to be directly addressed, but it can also be valuable when these public concerns are indirectly addressed, for instance when innovation leads to commercial products, which in turn generate turnover that may lead to further economic growth. In addition, a business community that does not innovate will in the long term have negative impacts on the economy, thus also on society. Added to that complexity is that in the innovation process to address some public concern, there are a multiplicity of business and government actors with sometimes conflicting goals and business models [START_REF] Klievink | Enhancing visibility in international supply chains: The data pipeline concept[END_REF]. Furthermore, in the initial stage of developing innovative concepts and solutions, there are many unknowns and the parties that invest are not necessarily those that will reap the benefits. It is thus hard to take the step from proof of concept towards implementation and impact [START_REF] Rukanova | Digital Trade Infrastructures: A Framework for Analysis[END_REF]. And in this process it is no longer clear where the government needs to provide support in terms of funding or other incentives, such as legislation or knowledge dissemination [START_REF] Henriksen | Diffusion of E-Commerce in Denmark: An Analysis of Institutional Intervention[END_REF], and where to leave it to the market to do the job.

This paper is largely empirically driven. In it, we examine and derive lessons learned based on a longitudinal case study of a series of four EU-funded projects (ITAIDE1 , INTEGRITY2 , CASSANDRA3 , CORE 4 ) in the international trade domain that aimed to develop digital trade infrastructure solutions (data pipelines; see [START_REF] Hesketh | Weaknesses in the supply chain: Who packed the box[END_REF]) to address public concerns related to security and trade facilitation. For our case analysis, we adapt and extend the framework Bryson et al. [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] on cross-sector collaborations. Based on our case analysis we show how each of these projects covered one part of the innovation trajectory moving the collective innovation from the Initial R&D stage, to the Showcasing and dissemination stage to attract a critical mass, towards a Turning point stage when the business cases for further upscaling become visible. We furthermore identify continuities (i.e. continuity of network and vision, continuity of funding and continuity of process) and a number of alignments as im-portant factors that drive the collective innovation process towards implementation and impact.

The remainder of this paper is structured as follows. In section two, we present our conceptual framework, which is an adaptation and extension of Bryson et al.'s [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] framework of cross-sector collaboration. In section three, we present our case methodology. In section four, we present the summary of our case findings. We end the paper with a discussion and conclusions.

Conceptual framework

In this paper we examine innovation processes that are intended to address a public concern and the role of public funding in such processes. A widely accepted definition of innovation is that innovation can be seen as "… an idea, practice, or object that is perceived as new by an individual or other unit of adoption" [10, p. 12]. Although this definition is a useful starting point, it refers to innovations in general and not specifically to those that are intended to address a public concern; we will come back to this point later in this section. An innovation development can be seen as a process that consists of all activities and impacts that arise from the recognition of a need or a problem, through the research, development and commercialisation of an innovation, through the diffusion and adoption of the innovation by users, to its consequences [START_REF] Rogers | Diffusion of Innovations[END_REF]. The duration of an innovation process can vary and can be as much as 15 or more years [START_REF] Ortt | Understanding the pre-diffusion phases[END_REF]. Trialability is a key concept from innovation management research [START_REF] Rogers | Diffusion of Innovations[END_REF]. Trialability means that the adoption of a technology innovation critically depends on the level at which organisations can first try out the new innovation in a confident environment and with low investments. In our cases, public funding provided such a confident environment. More than half a century ago, [START_REF] Olson | The Logic of Collective Action[END_REF] introduced the term collective action and suggested that such collective action is necessary among organisations to increase the speed of innovation. Looking at institutional innovation, [START_REF] Hargrave | A collective action model of institutional innovation[END_REF] identified a number of processes related to mobilising collective action, namely framing contests, construction of networks, enactment of institutional arrangements, and collective action process. Building on the collective action view, [START_REF] Rukanova | Overcoming Blockages in Collective Innovation in Digital Infrastructures[END_REF] further suggest that collective innovation from initial R&D to implementation can be traced when looking at a continuum of projects. While some projects taken in isolation may appear to result in a collective action failure [START_REF] De Reuver | Collective action for mobile payment platforms: A case study on collaboration issues between banks and telecom operators[END_REF], by taking a longitudinal and cross-project perspective these projects are a necessary step in the innovation trajectory that leads these innovations from initiation to implementation [START_REF] Rukanova | Overcoming Blockages in Collective Innovation in Digital Infrastructures[END_REF]. In addition, the need to organise vision in the collective action for digital trade innovation is also highlighted [START_REF] Van Stijn | Moving an eInnovation from a Living Lab to the real world: Politically savvy framing in ITAIDE's Beer Living Lab[END_REF]. Regarding innovations in highly regulated domains such as international trade, multiple levels also need to be taken into account [START_REF] Rukanova | Understanding the influence of multiple levels of governments on the development of inter-organizational systems[END_REF] to reflect government influences and regulatory concerns.

[1] explain how cross-sector collaboration involving parties such as businesses, non-for-profit organisations, communities and government is becoming increasingly important for tackling complex societal challenges. Based on a thorough literature review, they develop a framework for understanding cross-sector collaborations. The framework builds on the following key concepts: initial conditions, processes, structures and governance, contingencies and constraints, and outcomes and accountabilities. 5 The framework presents a suitable starting point for our analysis as it explicitly captures concepts that help to trace the progression from initial conditions towards outcomes and accountabilities. This is in line with our goal to illustrate the progression of public-private innovations from initiation to implementation. Furthermore, the concept of initial conditions is particularly useful, as it will allow us to capture the public concern (e.g. safety and security) and public funding that triggers collective public-private innovation processes. Bryson et al.'s [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] framework is also limited, as it does not explicitly allow an analysis of a multiplicity of projects that taken together can achieve transition towards implementation [START_REF] Rukanova | Overcoming Blockages in Collective Innovation in Digital Infrastructures[END_REF]. Furthermore, the multilevel nature of government is also not explicitly included [START_REF] Rukanova | Understanding the influence of multiple levels of governments on the development of inter-organizational systems[END_REF]. The figure presented below shows our initial conceptual framework. The above framework builds on [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] in the following way. First, inspired by [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF], we look at the process from initial conditions/public concern (left-hand part of Figure 1) towards outcomes and accountabilities (right-hand part of Figure 1). We also incorporate the view that a succession of independent projects can together lead to the scaling up and implementation of the solutions (based on [START_REF] Rukanova | Overcoming Blockages in Collective Innovation in Digital Infrastructures[END_REF]). This is captured in Figure 1 by the block labelled public-private innovation trajectories, where the trajectory con-tains a number of projects (1...N). At a project level, we again utilise the concepts of Bryson et al. [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] to capture how each individual project (comprising a project-specific set of actors) moves towards specific outcomes. Finally, we explicitly add the government (role and level) in our framework. The levels (national, economic zone (e.g. EU), international) are added to capture the level from which the funding or other source of government influence comes with respect to the innovation trajectories under analysis. In addition, in the conceptual framework we explicitly define three government roles to show that the agency providing the funding is not necessarily the same as the agency that owns the problem (i.e. the agency responsible for addressing certain societal problems) and drafts the regulatory and legislative frameworks.

Method

We conducted our case study in an interpretative, processual tradition [START_REF] Walsham | Interpreting information systems in organizations[END_REF]. The context of this study is a series of four EU-funded projects (ITAIDE (2005-2010), INTEGRITY (2008INTEGRITY ( -2011)), CASSANDRA (2011-2014), and CORE(2014-2018)) that were initiated to look for innovative digital innovations solutions to address the safety and security concern in international trade. The Customs Administration of the Netherlands played a key role in all these projects driving eCustoms innovation. A number of university and business partners also remained involved in the continuum of projects, while the network grew.

In the ITAIDE project (17 partners), the idea of reusing business data for government control purposes (also referred to as piggy-backing) was introduced and four living labs with beer, pharmaceuticals, paper and food were used to develop initial ideas and pilot these in a real-life setting [START_REF] Tan | Accelerating Global Supply Chains with IT-Innovation[END_REF]. While the innovations that were developed and piloted in ITAIDE showed how they solve the trade facilitation and security concerns, the business cases were not clear enough for parties to pick up the ideas further and there were also legislative constraints.

INTEGRITY (15 Partners) was started as an independent project, but through the involvement of the Customs Administration of the Netherlands some of the lessons learned from ITAIDE were also considered. In INTEGRITY, the UK Customs authority was also a partner and the idea of the data pipeline was born [START_REF] Hesketh | Weaknesses in the supply chain: Who packed the box[END_REF]. The data pipeline is an "an IT innovation to enable capturing data at the source" ([6], p. 14). Governments can use data pipeline information for government control purposes. In INTEGRITY, terminal operators were involved in piloting the solutions, but further upscaling and implementation of the solution was not achieved.

In CASSANDRA (26 partners), the data pipeline idea was further piloted, showcases were developed and a lot of effort was spent on disseminating the data pipeline to a wider business and government audience and on awareness building. The development of the Customs Dashboard (a special interface that the Dutch Customs authority uses to access data from the data pipeline) was initiated to allowed Customs to view data pipeline information. CORE (81 partners) was initiated as a demonstration project with the goal to do large-scale demonstrations. Various data pipelines were piloted and business parties are now investing in developing their own pipeline solutions. One of the pipelines that is now being rolled out is a block-chain enabled global data pipeline. In addition, the Dutch Customs authority, based on its experiences in these earlier projects, has decided to invest in its own operational Customs Dashboard that can interface with data pipelines. As such, the CORE project can be seen as a Turning point where business cases are becoming clear and business and government organisations are investing their own resources to move these innovations towards implementation.

The data collection took place over a period of 12 years. All the authors were involved in at least one of the projects but with different levels of involvement and roles. The authors had access to rich data. This included participation in meetings, redesign sessions, extensive interviews and document analysis in the context of the various demos, workshops and events organised by these projects. The data analysis was performed through the conceptual lens of the framework presented in section two in a number of iterative loops. In order to deal with biases, we benefited from the fact that one of the senior researchers in the team was involved in a limited way in only the ITAIDE project and followed the other projects remotely. As such, this researcher was very instrumental in questioning the assumptions. This led to us to sharpen the analysis and the presentation of the findings.

Case analysis

Figure 2 below summarises the results of our case analysis by using the conceptual framework presented in section two. In our analysis, we also identified a number of continuities and alignments that were instrumental in moving the innovation from initiation to implementation. These have therefore been added to Figure 2 and will be explained later. It is important to highlight that unlike commercially driven ideas, where parties can come together, develop a solution and attract investors to bring an idea to the market, when it comes to innovations that are intended to address a public concern, the government has identified the concern but needs to search for ways to address it. In our case, the public concern relates to increasing safety and security, and the EU-funded projects that we discuss were intended to stimulate the development and upscaling of solutions to address safety and security challenges. Looking at the role of government in initiating the innovation trajectories under analysis in our case, this relates to the EU level (see top level of Figure 2). In Figure 2, we use dotted lines to indicate other levels such as national and international, as these can also be sources of public funding, but for this paper they are beyond the scope of our analysis. At the EU level, we explicitly capture that in our case while the funding was provided by DG Research, the problem ownership related to safety and security lies with other DGs (such as DG TAXUD and DG Home), which play an important role setting EU objectives related to societal goals and regulatory frameworks. We now look at the continuum of the four EU projects and derive a number of observations by looking at the perspectives of process (P), structures and governance (S&G), initial conditions (IC), contingencies and constraints (C&C), and outcomes and accountabilities (O&A) (see Figure 2).

Taking the process (P) perspective from the framework and looking at the continuum of projects, we see that the projects progressed through a number of stages, such as the Initial R&D stage in ITAIDE, where initial ideas were developed and piloted in real-life situations; the R&D refinement stage, where the ideas further crystallised (the data pipeline idea in INTEGRITY) and were piloted; the Showcasing and dissemination stage, where showcases were developed and ideas disseminated to attract further critical mass (CASSANDRA); and the Turning-point stage in CORE. Here, a 'Turning point' is the moment that individual parties take over the next step and use their own resources to further scale up parts of the solutions developed, rather than relying on public/private funding for this upscaling step. 6 Thus, we see that in the progression towards upscaling and implementation, it was important to have continuity of the process 7 in terms of proceeding from one stage of the innovation process to the next stage (i.e. from the Initial R&D stage towards Turning point stage). Although in our case we do see such continuities, these are not necessarily guaranteed. If funding efforts are focused only on supporting Initial R&D and no progress is made towards the next step, this may lead to the proliferation of ideas and only limited progress towards implementation.

Looking at the structure and governance (S&G) perspective, we can trace the evolution of the network. While the network was relatively small in the Initial R&D stage (ITAIDE, 17 partners) and the R&D refinement stage (INTEGRITY, 15 Partners), the network expanded further in CASSANDRA (25 partners), where further showcases were accumulated and the idea of the data pipeline was further disseminated to create awareness. This was crucial for making the ideas tangible and getting more and more parties on board, which resulted in the further growth in the number of parties interested in committing to the idea and engaging in large-scale piloting in CORE (81 partners).

What is also key in our is case that some organisations and key people from these organisations remained involved throughout all the projects and sustained the vision. In the continuum of projects that we analysed, there were a few key individuals and organisations that had a vision and were able to carry on the ideas from one phase to another and attract critical mass and funding along the way. That was especially important in the last project, where the network grew significantly and there was a danger of losing focus. As such we see continuity of network and vision as key elements 6 The R&D stage can be preceded by invention stage where some specific technology is invented and developed (e.g. the Smart container seal was developed by a commercial organisation before piloted in ITAIDE). If in the context of addressing a public concern it is discovered that there is a need to invent new technologies but there is no business drive for businesses to invest in this stage, public funding could be useful to stimulate the Invention stage as well. 7 We do not see the process as linear, as it can evolve in a number of iteration loops.

for the success of the innovation trajectory under analysis. Continuity of vision and network does not mean that the network remained static and was limited to the same parties; on the contrary, the network grew to include new people and organisations. This was crucial for accumulating a critical mass of parties that subscribed to the idea and were ready to co-invest in the further demos and piloting.

Looking at the initial conditions (IC) (Figure 2), securing funding was key for each individual project. Continuity of funding throughout the different projects is another continuity that was essential for bringing the innovation process from initial R&D towards implementation. This was crucial, as without this continuity of funding the process could have stopped at an earlier stage, before the business models were clear and the industry was ready to pick up and invest in the further implementation and upscaling of the solutions. By funding we mean both public and private funding. Had funding been stopped too early, all the earlier investments could have become sunk costs. In the continuum of projects that we discussed above, it was important that the government continued to invest, but it was even more important that in this continuum businesses became increasingly interested and co-invested millions in this process, and that the network grew indicating that more parties started to believe in the ideas. Thus, gaining commitment from businesses to participate and invest in the public-private innovation trajectory is as crucial as the funding agency that provides the public share of funding the initiative.

Regarding contingencies and constraints (C&C), the lack of clear business cases was also one of the key issues, as scaling up could not occur until some of the network partners saw clear value and were willing to further invest. The business cases started to become clear in CORE. It is still a challenge to engage more businesses and government organisations, but this process is currently being driven by pioneering businesses or pioneering governments as part of their own activities. A further challenge was how to perform data analytics on such large data volumes provided by data pipelines. This challenge still needs to be tackled.

Finally, looking at outcomes and accountabilities (O&A), the first three projects achieved project-specific outcomes. Although these outcomes in isolation were not enough to achieve the implementation of a solution to address the societal concern, they were important stepping stones towards implementation in CORE and beyond.

To summarise, we identified three types of continuitiesnamely continuity of network and vision, continuity of funding and continuity of process (from Initial R&D towards the Turning point)as important success factors. We also identified two alignment processes that appeared important and that we discuss below.

Alignment of innovation trajectories with problem sponsor and problem owner: at the EU level, there are two DGs that are crucial for the public-private innovation trajectories: DG Research & Innovation, which provides the public funding, and the DGs that are the problem owners in the problem domain. While it is clear that there will be a lot of pressure to align the project with the DG that provides the funding (through regular review cycles), the alignment with the DGs that own the problem is also crucial for moving towards implementation in the problem domain. And we would argue that not only alignment but tight alignment with the DGs that own the problem is needed. For example, during ITAIDE, links were established with the problem owning DG and there were regular interactions, but the alignment was not very tight, as the innovation agendas had different time frames and priorities. It was therefore difficult to absorb the results of ITAIDE and move towards implementation. The innovators learned from this experience and made explicit efforts in the subsequent projects to achieve tight alignment. Ultimately in CORE a tighter alignment was achieved, which improved close communication lines and tight alignment between the innovation agenda and concerns of the problem owner DG and the efforts in CORE. However, this link is still not sufficient to align the policy agenda with the outcome of research. That alignment is crucial, especially in highly-regulated domains, such as the area of Customs. The support of the regulator is crucial for ensuring the adoption and upscaling of the innovations and realising further impact. Without such support, even the best solutions on paper will be hard to implement.

Alignment with the public concern and with the innovation trajectory and vision: we found out that it was necessary to constantly match the innovation process steps covered in each project to the source (the reason, the initial conditions/public concern) of funding. We found out, in particular in CORE, due to the many new actors that entered the project it was initially difficult to keep them focused on the initial concept, which made it difficult to keep track with the initial idea. Thus, continuous checks and balances are crucial, especially as the network grows and more actors are involved in piloting. Related to alignments, it is our view that in all these steps it is possible to find a balance between public and private interest, shared benefits: trade facilitation and safety/security are two sides of the same coin. This, of course, will not always be the case in R&D related to public concerns, but in the case of the data pipeline concept this was a fruitful outcome.

Discussion and Conclusion

In market-driven situations, innovations can start with inventions where some party develops a new technology and then looks for investors who have an interest in this innovation and will enable it to grow. In the case of public concerns, it is not the private market but the public concern that initiates the innovation process. The task of public funding agencies is to stimulate the generation of ideas and then stimulate the process of moving these ideas towards implementation and impact.

In this paper, we examined and derived lessons learned based on a longitudinal case study of a series four EU-funded projects (ITAIDE, INTEGRITY, CASSANDRA, CORE) in the international trade domain that aimed to develop digital trade infrastructure solutions (data pipelines) to address safety and security challenges. In the approach followed in these projects, trade facilitation was included as an additional public concern that was taken into account in shaping the solutions. For our case analysis, we adapted and extended Bryson et al.'s [START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] framework on cross-sector collaboration. We extended this framework by explicitly acknowledging that the progression of innovations from Initial R&D towards implementation can span multiple independent projects that individually can achieve limited outcomes but together can be seen as part of innovation trajectories that move innovations towards implementation. Based on our case analysis, we identified continuities (i.e. continuity of network and vision, continuity of funding and continuity of process (from Initial R&D towards Turning point and implementation)), as well as two alignments of the innovation trajectory (i.e. alignment with the project owner and project sponsor, and with the public concern), as important success factors.

As discussed, trialability is a key concept from innovation management research [START_REF] Rogers | Diffusion of Innovations[END_REF]. Trialability means that the adoption of technology innovation critically depends on the level at which organisations can first try out new innovations in a confident environment and with low investments. In our cases, the public funding provided such a confident environment, where a complex network of public and private actors could experiment, develop solutions, test them in real life and learn. In this process, the uncertainties and ambiguities were gradually reduced, and the showcases attracted critical mass and further funding. This process made it easier to reach a Turning point where the business cases gradually become more clear, which made it easier for parties to take the next step and proceed with further implementation and upscaling with own investments.

Although in the projects that we analysed we saw continuity of network and vision, continuity of funding and continuity of process, such continuities are not guaranteed. These continuities need to be carefully managed as they may be endangered when moving from one project to another. As a result, the vision, as well as the earlier efforts and investments, may be lost or insufficiently utilised. For example, if the funding and efforts are mainly focused on the R&D stageor are devoted to more fundamental research or broadened to adjacent fieldsand no steps are taken towards the Showcasing and dissemination stage to build awareness, or towards the Turning point stage, then it is likely that a lot of ideas will be generated but limited impact in terms of scaling up and adoption will be achieved. Similarly, if there is continuity of funding but not of network and vision, this may lead to a situation in which know-how and expertise are not efficiently utilised when moving towards implementation.

In discussions it was suggested that after the Initial R&D stage, the government could step forward and assume the role of an investor by taking the risk and investing, but if the project is successful the government would be one of the parties realising the gains and with the gains, the financing of a next project can be assured. Whereas different scenarios may be possible, we consider that continuity of vision and network, continuity of funding and continuity of process from Initial R&D to Turning point, as well as the alignment processes that we identified, are crucial for moving towards implementation and impact.

This study was largely empirically based and the lessons learned are limited to the four projects under analysis. Further research is needed to establish the applicability of the findings in other context and domains. However, the lessons learned can be insightful also for other initiatives where public funding is used to facilitate the development and upscaling of innovative solutions to address some public concern.

Fig. 1 .

 1 Fig. 1. Initial conceptual framework for public-private collective innovations adapting and extending the framework of Bryson et al. [1].

Fig. 2 .

 2 Fig. 2. Empirical model based on adapted and extended conceptual framework of Bryson et al. [1].

See[START_REF] Tan | Accelerating Global Supply Chains with IT-Innovation[END_REF]

http://www.integrity-supplychain.eu/

http://www.cassandra-project.eu/

http://www.coreproject.eu/

In Bryson et al.'s[START_REF] Bryson | The design and implementation of Cross-Sector collaborations: Propositions from the literature[END_REF] framework, each of these categories is further detailed into subcategories. For the sake of simplicity, we utilise only the high-level categories for building the adapted conceptual framework that we use in our analysis.

Acknowledgement

This research was partially funded by the CORE Project (nr. 603993), which is funded by the FP7 Framework Program of the European Commission. Ideas and opinions expressed by the authors do not necessarily represent those of all partners.


Jonathan Crusoe
email: jonathan.crusoe@liu.se

Ulf Melin
email: ulf.melin@liu.se

Investigating of Open Government Data Barriers A Literature Review and Conceptualization

Keywords: Open data, Open government data, OGD, Barriers, Risks, Challenges, Impediments, Myths, Process, Literature review

When focusing on open government data (OGD) publishing and related barriers, there are several complexities present. Largely, current research is focused on publishing and usage of OGD; and we argue that there are a need to investigate and to systematise OGD barrier research in order to understand and outline an expanded scope of the phenomenon. We expand by clarifying barriers linked to the release decision and the data's organisational context. To investigate the OGD barriers, we conduct a systematic literature review, identifying 34 articles as a point of departure for our analysis. From these articles we create, present and discuss illustrations on historical development, barrier types, and different research focuses on OGD. When analysing the articles, we identify a focus on technical, organisational, and legal barrier types, while studies on open data usage and systems are less frequent. Our analysis also identifies some possible open data research barriers. In the article we also relate barriers to an expanded OGD process (Suitability, Release, Publish, Use, and Evaluation), identifying 46 barriers with possible linkages. The results is an expanded scope and a conceptual illustration of OGD barriers.

Introduction

Open government data (OGD) refer to data sets that government agencies make available for third-party usage. Putting OGD in contrast to history, where public organisations produce data enclosed in silos where its value is limited. One purpose is to make data accessible to a broader audience by stimulating the third-party development of new e-services (e.g. apps for weather forecasts and smarter travelling routes). By opening data for others to reuse, we gain OGD [START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF]. OGD is surrounded by many expectations of service innovation, increased transparency and inclusive government agencies [START_REF] Attard | A systematic review of open government data initiatives[END_REF].

As a concept and phenomenon OGD is promising, but at the same time challenging. OGD research and practice has shown several complexities when publishing data [e.g. [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Dawes | Planning and de-signing open government data programs: An ecosystem approach[END_REF][START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. Challenges, or barriers as we conceptualise them in this paper, are described under different conceptual labels in previous research, such as; barriers [e.g. [START_REF] Barry | Barriers to open data release: A view from the top[END_REF], impediments [e.g. [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF], myths [e.g. [START_REF] Hellberg | The story of the sixth myth of open data and open government[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF], and challenges [e.g . 28]. Multiple analytical perspectives are also present in previous research; focusing on barriers linked to release [e.g. 4], adoption [e.g. 20], cognitive [e.g. 37], agenda [e.g. [START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF], process [e.g. 9], and use [e.g. 42], use and publish [e.g . 5]. OGD is also placed in different larger context, by using, e.g. an ecosystem perspective [START_REF] Dawes | Planning and de-signing open government data programs: An ecosystem approach[END_REF]. In present OGD research, there is a tendency to study specific parts or specific phases of a larger system or process.

Another complexity is the nature of the OGD barriers and the OGD process. The barriers can e.g. be linked to different process phases, and there are several ways to divide the OGD process into phases [e.g. [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Charalabidis | A taxonomy of open government data research areas and topics[END_REF][START_REF] Dawes | Planning and de-signing open government data programs: An ecosystem approach[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. Which is also further complicated when different actors can participate in one or several phases [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF], and activities in one phase can create barriers in a later for someone else [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF]. The complexity can be visualised as a barrier network [START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF], which include a reversal of the cause-and-effect; a barrier can come from a later solution. We, therefore, see earlier OGD research giving room for specialised solutions for unique and challenging problems; where solutions can cause barriers in both directions of the OGD process. There is a need to understand and systematise existing OGD barrier research with an expanded scope.

We expand the scope by taking a step back and view other actors and activities surrounding the publisher and open data publication, which includes more OGD barriers. We investigate the OGD barriers in earlier research with a systematic literature review and expand on earlier works, such as Zuiderwijk and Janssen's OGD process [START_REF] Yang | To open or not to open? determinants of open government data[END_REF], through a systematisation. An expanded scope can serve as inspiration for a future research agenda and practice in the field. The result illustrates insights into the complexities and challenges encountered when publishing. We also outline some future OGD barrier research avenues. The following research questions are focused in this paper: ─ What OGD barriers have been identified in previous research? ─ Where are the OGD barriers encountered in the OGD process and how are they connected?

In the following sections, we will start by describing our research approach. Then we will discuss the identified barriers from the literature and chart illustrations, and in the next section presents the OGD barrier systematisation. We end the paper with a conclusion containing limitations, future research, and implication.

Research Approach

We conducted a systematic literature review [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Cronin | Undertaking a literature review: a step-by-step approach[END_REF][START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF] where we analysed the findings through coding [START_REF] Saldaña | The coding manual for qualitative researchers[END_REF] to make illustrations and a systematisation. The literature review [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Cronin | Undertaking a literature review: a step-by-step approach[END_REF][START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF] started by defining keywords, and then selected a database that was searched, results were filtered and summarised, and lastly, we analysed the findings. We created simple keywords by combining "open data" with "barrier", "risk", "challenge", or "impediment". For database, we selected Google Scholar as it has good coverage, with recall and precision that is above average for simple keyword searches [START_REF] Walters | Comparative recall and precision of simple and expert searches in google scholar and eight other databases[END_REF]. For each keyword search, we looked at the first 50 results. We first searched all years, but also wanted to be sure that we caught the latest and relevant articles, we therefore explicitly searched the years 2016 and 2017. This gave us a total of 600 articles; we discarded duplicates. We identified relevant articles by studying the title and Google Scholar summary for concepts relating to our keywords. Relevant articles were summarised with article id, authors, title, year, publish origin, topic, and important conclusions, if needed, purpose and method were included, such as the case of the systematic literature reviews.

Later when analysing the articles, we discovered interesting references and snowballed. We summarised their content in the same way as above. This resulted in a final set of 34 articles. We analysed the articles to present them as data (see section 3) and to systematise OGD barriers (see section 4). For the data presentation, we decided to analyse development, types, and focuses, making a distinct coding effort for each into a representative illustration. When coding for development, we sorted the articles based on publication year. We then used them to form a graph and coded based on what and possible why. For analyzing the types, we divided articles into two groups: specific articles (containing a thematic division or typology of barriers into types), while the general articles contain barriers (but not structured by type, e.g. [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF] structured by role, or [START_REF] Barry | Barriers to open data release: A view from the top[END_REF] structured by type). We coded the first group by following their thematic division or typologies. After that, as part of categorization, we integrated the second group into the first through coding. When analysing focus we assumed that the abstracts provided an overview of the work with information about the background, objective, methods, results, and conclusions [START_REF] Andrade | How to write a good abstract for a scientific paper or conference presentation[END_REF][START_REF] Jacinto | Doing science: Writing conference abstracts[END_REF]. We therefore initially coded [START_REF] Saldaña | The coding manual for qualitative researchers[END_REF] the abstract for research focus. If the abstract lacked the necessary information, the whole paper were explored. After codes were categorised into groups and we merged and split groups to achieve a pedagogic presentation. The product was a sketch that we as researchers discussed and analysed in a generative dialogue, and some final changes were made. The results were the three illustrations in section 3. For the systematisation, we first identified an OGD process, with different parts and activities. We then categorise coded [START_REF] Saldaña | The coding manual for qualitative researchers[END_REF] the 34 articles into it. Giving the barriers a chance to expand the OGD process by e.g. including concepts. After, we identified possible linkages between barriers following the network idea of Huang, et al. [START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF]. Linkages were identified when barriers shared common core phenomena or phenomena were dependent on each other. As a final step, we refined the categories by reflecting on their meaning.

The Identified OGD Barrier Literature

Below, we present the articles as data through illustrations following our literature analysis, in the following order: historical development of the OGD barrier discussion, barrier types, and research focuses. We note that some research only acted as enablers [START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF][START_REF] Kassen | A promising phenomenon of open da-ta: A case study of the chicago open data project[END_REF][START_REF] Susha | Open data research in the nordic region: Towards a scandinavian approach[END_REF] (used to snowball) and was, therefore, not used in the later systematisation.

Historical Development

In this section, we present the historical development of the OGD barrier discussion. In Figure 1, we have divided the number of articles published 2011-2017. In 2014, we identified an increase of published articles. We will use this peak as an important point and discuss the time-period before and after.

Before 2014, we have a total of 4 articles researching OGD barriers. In 2012, we identified what seems to be the start of a crystallisation that had its full expression in 2014. E.g. the literature review [START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] focused on the user OGD barriers. The authors pull together research from different research domains. Suggesting that OGD challenges encountered covers a variety of domains and around 2012 these started to crystallise into concepts such as open data barriers, challenges, myths, risks, and impediments giving way for a collective discussion. When analysing the articles, we identified a pattern. Myths, for example, [START_REF] Hellberg | The story of the sixth myth of open data and open government[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Melin | Challenges and benefits in an open data initiative -a local government case study of myths and realities[END_REF] are not reused or connected to other OGD barrier research. Research has not revisited the focus on implementation and use from a multi-level perspective [START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF]. A process perspective [e.g. 3, 39] seems to be useful and often comes with a focus on the publisher, and therefore, lacks the barriers existing on the "outside" between publishing and feedback. Such as barriers for hackathon apps to reach the market [START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF]. We see a focus to understand barriers and their linked parts, leading to rather limited attempts to solve complex issues from multiple directions; from the micro to the macro perspective. When looking at probable causes for the decline above, it can be that research is in an incubation period or have expanded. If an incubation period is present, we identify two OGD research barriers; (1) we are still searching for the abstract approach to handle barriers, e.g. like outlined in [START_REF] Barry | Barriers to open data release: A view from the top[END_REF] and [START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF]. [START_REF] Appelgren | Data journalism in sweden: Introducing new methods and genres of journalism into "old" organizations[END_REF] We have not crystallised our knowledge about barriers, meaning that we are still working towards comprehension; one attempt has been made by [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF]. Both fit with the pattern above. possibility that the research has moved over to solutions. For both, we see two accompanying OGD research barriers; (3) keeping the knowledge body together between domains and (4) researching solutions that will not lead to barriers in other parts of the OGD process. To conclude, we identify a complexity in research and practice covering the micro technical and social (formats, metadata, skills, knowledge, etc.) and the macro technical and social (internet, ecosystem, society, etc.) making open data barriers a complex and challenging research task, as it covers multiple domains where barriers exist outside a particular actor.

Barrier Types

In Figure 2, we present barrier types identified in the 34 articles. We have coloured the two groups mentioned in section 2. When studying the different barrier types, it was common to find technical, organisational, and legal barriers (identified in 21, 19, and respectively 20 articles), something that was not surprising as open data originates in the publisher organisation's social and technical systems [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF]. On the other hand, data barriers appeared in 15 articles, but were sometimes fused with technical barriers such as format and quality; they tended to be linked. Making data barriers implicate technical barriers. For open data, the value is generated from reuse [START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF] and we, therefore, expected use to be equally studied as technical, organisational, and legal barriers, it was identified in 10 articles. However, use was sometimes fused with technical barriers through access and findability. Use was also often fused with participation and seemed to be assumed to go together, as they are often integrated into the OGD processes [e.g. 3]. Making use through implicitly equally studied to technical, organisational, and legal barriers. Equally important, but less studied was the OGD process (found in 8 articles), where there were a gap between accessing data and feedback, where usage or application barriers were absent. We see that there is a lack of a wider scope when studying OGD barriers. In the OGD process for the publish-and-use relationship, we found a tendency for barriers to affect both providers and users while not stated as such. Use, participation, and process barriers need study beyond the direct interaction between providers and users. Largely, it remains unknown how organisations [e.g. 9] connect with publisher-user relation [e.g. 40] and affects the user [e.g. 6] and how this, in turn, affect participation and the organisation as a publisher. Turning our attention towards the more individual-oriented barriers, such as skills, perceived risks, knowledge, and awareness, they have been studied in 13 articles, e.g. in the role of the OGD user. These barriers differ from the above as they focus on how a person needs to adapt to open data, rather than changes in techniques and organisations. As individuals populate organisations, it can be challenging to separate the two, but we think that, based on the patterns above, it is relevant to conclude that we should aim to focus more on individual-oriented barriers in organisational settings. The final barrier type is economic, which were identified in 11 articles; we found discussions about costs, but no clear evidence that benefits would outweigh them. There was also a lack of connection between costs, organisations, and solutions (both implementation and maintenance) to barriers. As pointed out by [START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF] data generation and maintenance cost money. Articles tend to focus less on the cost associated with solving barriers and challenges. However, it is not the case that costs are ignored, e.g. [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF] brings up cost issues. Two major issues seem to be calculating implementation costs for overcoming barriers and maintenance costs versus benefits, e.g. how do we calculate the value of increased transparency, democratic accountability, and stimulation of innovation [START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF]. We see a need for more research about economic barriers in comparison to other barriers. In Figure 3, we see an ongoing focus on publishing, usage, and systems over the years. The strongest focus was on publishing, while both usage and a system approach came second. Together with frequency, technical, organisational, and legal barriers are mentioned in favour overuse and participation (see Figure 2), we have identified that challenges seem to be inside public organisations (or we need more research on use and usage). Looking at the figure, we see that interaction was studied under the years 2013 and 2017, with focuses on engagement, participation, and platform. In the earlier section, use barriers were often fused with participation and interaction barriers, where use seems to link to publishing through technical barriers. We, therefore, expected to see a balance amongst the barriers between users and publishers, with some focus on equalisation of expectations and division of labour [START_REF] Melin | Challenges and benefits in an open data initiative -a local government case study of myths and realities[END_REF], but instead found a focus on how publishers "imped" interaction where users were less problematized. For studies focusing on usage, they focused either on certain user groups and their special usage [e.g. [START_REF] Appelgren | Data journalism in sweden: Introducing new methods and genres of journalism into "old" organizations[END_REF][START_REF] Hogan | Governance, Transparency and the Collaborative Design of Open Data Collaboration Plat-forms: understanding barriers[END_REF] or specifically on the use of provided data from a user perspective [e.g. [START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]. Here we found a gap in how different use barriers connect to different usages and users (see Davies [START_REF] Davies | Open data , democracy and public sector[END_REF] for examples of usages). The relative low societal focus can reflect the general maturity of the OGD field, being rather early in its lifecycle. We have, in research and practice, not yet solved issues surrounding publishing, using, and interaction. In the section above, legal barriers were mentioned second to most, but here we see that it has been studied the least. Legal and privacy barriers seem to accompany other focuses. It can be that they are solved by practical work, rather than theoretical. There is a lack of research on assumptions and publish decision barriers, unpacking this could lead to further insights about the publisher.

Research Focuses

Systematization of Open Government Data Barriers

In the following section, we expand the scope surrounding the OGD process and systematise the OGD barriers. The OGD process consists of phases that involve data generation, data publication, data discovery, data usage, and user feedback [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF], which is the final process after data publishing. Too, understand the barriers behind publication we take a step back. Open data originates from an organisation where it starts as closed, which means that the data has an organisational context [START_REF] Conradie | On the barriers for local government releasing open data[END_REF], before and after publication. In between data being close and open, there needs to be a decision for data release [START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF]. Meaning that we have five distinct processes containing barriers; when (1) identifying data's suitability for publishing, (2) deciding to release, (3) publishing the data, (4) someone uses the data, and (5) evaluating the impact and collect feedback. The processes are further complicated when it is known that activities in earlier phases can cause barrier further down [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF] and connect with each other [START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF], which we argue also applies to our five distinct processes.

The relationship between the distinct processes is complex. Placing the processes beside the ecosystem [START_REF] Dawes | Planning and de-signing open government data programs: An ecosystem approach[END_REF], we find that they are connecting through ecosystem elements. We view elements as a complex network of participants with their challenges (some not related to OGD). The identification of suitable data starts when there is interest in some data. The suitable data is input into the other processes. Inside a decision network, we find decisions to release processes. Participants are the organisation's internal management and enthusiasts, but also outsiders, such as activists, researchers, and politicians [e.g. 28]. If the decision is release, the publishing process triggers inside an organisation. The publisher (people behind data publishing) will then work to publish the data into the data market (people behind data use and products [e.g. 24]). Where users can transform the data through use into societal impact (people benefiting, affected by, or using open data outside or at the edge of the data market) [START_REF] Lindman | Business roles in the emerging open-data ecosystem[END_REF]. Impacts could be of political and social, economic, and operational and technical nature [START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF]. The decision network then evaluates the impacts, which can concern the publisher, users, and citizens. Evaluation can happen through feedback, discussion, [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF] and some evaluation is internal or external to the OGD process [START_REF] Dawes | Planning and de-signing open government data programs: An ecosystem approach[END_REF]. The processes form a cycle starting and ending with the decision network. For an example of the cycle; low-quality data negatively affects usage that results in an impact on the perceived usefulness of open data [START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF], resulting in less reason to publish. Low usefulness may lead to a revaluation of the open data's suitability. In the following subsections, we will present barriers identified for each of the five distinct processes. Bold words are barrier categories, while cursive are coded; behind both, there are citations to show quantity and origin.

Identifying Data Suitability

In the process of identifying data's suitability as open data, barriers will be encountered in the relationship between the data and organisation. Here the first thing to study is the relationship towards the Core Task [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Jaakkola | Open data: Opportunities and challenges[END_REF][START_REF] Lundell | e-governance in public sector ict procurement: what is shaping practice in sweden[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] is produced by the organisation and will, therefore, have a meaning dependent on that context, with accompanying assumptions. We will have to study the data's properties (e.g. static or dynamic), context (e.g. language), and quality (e.g. completeness, fragmentation, accuracy). The production of the data happens in the Collection Process [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] of the organisation, which can be integrated into other activities. In it, data is collected, created, or generated. Tools, filters, data processing and methods are used, which can create barriers for later publishing and reuse. E.g. filtering before storage can result in "No access to the original data (only processed data)" [20, p. 262]. The Storage System [3, 5, 9, 18-20, 22, 25, 26, 38, 39, 42] stores data for the long term. The form and location of the storage can vary; with everything from paper-based to digital and in-house to the cloud. At the same time, the owner of the data and the storage can differ. Barriers can be an inability to convert data, improve storage system, or the share amount. Something that brings us over to Path Dependency [START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Lundell | e-governance in public sector ict procurement: what is shaping practice in sweden[END_REF][START_REF] Melin | Challenges and benefits in an open data initiative -a local government case study of myths and realities[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] as a source of barriers. E.g. municipalities seldom considered document formats when buying software, instead focusing more on specific office applications [START_REF] Lundell | e-governance in public sector ict procurement: what is shaping practice in sweden[END_REF]. Meaning that they might have bought a system based on closed formats that are hard to open. In relation to collection, storage, and core task we have the Internal Usage [START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Jaakkola | Open data: Opportunities and challenges[END_REF] that will form the data with or for a purpose. A forming that might make the data unsuitable for external reuse. The data's Suitability [START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF][START_REF] Yang | To open or not to open? determinants of open government data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF] as open data, comes from itself and relationship with its organisation. We can, therefore, talk about suitability barriers that complicate publication and reuse, where solutions can problematize the organisation's day-to-day activities.

Decisions to Release

Once suitable data have been identified, a release decision can be made. The organisation might have several datasets and no idea which to publish, which relate to Market barriers. Here Supply and Demand [START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF] can identify sought-after data; if demand is neither expressed or heard then it forms a barrier as an organisation cannot prioritise publishing. Another side is Competition [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF]. Outside the organisation there might be actors competing with or around the data. Releasing the data can dislodge their business models. On the other hand, if we turn our attention towards the publisher we have Organizational barriers. If publishing the data do not align with the organisation's Objectives [START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF], this may form a barrier as other tasks are prioritised. Prioritization can also be affected by the availability of resources. Which makes Cost and Income [3-5, 9, 16, 20, 26-29] interesting, as it is known that open data can lower costs, but also income [START_REF] Barry | Barriers to open data release: A view from the top[END_REF][START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF]. Meaning that open data has a risk to threaten resources used in core tasks. This also moves us to the Division of Labour and Alternatives [START_REF] Melin | Challenges and benefits in an open data initiative -a local government case study of myths and realities[END_REF][START_REF] Melin | Open government data in an e-service context -managerial and conceptual challenges[END_REF]. For Division of Labour it is a question about expectations on publisher and user; e.g., if there is a need, should publisher or user implement visualisation tools. Placing this alongside objectives and resources, it also becomes a question about alternatives.

An alternative to open data within a government organisation, can, e.g. be to develop and launch e-services [e.g. 29] that are prepacked and controlled in another way that is publishing OGD for any user and any purpose. Another potential for declining to publish are Consequence barriers. Here Regulation and Legislation [3-5, 19, 20, 22, 23, 27-29, 31, 37, 40, 42] can block or problematize open data publishing. As an example, [START_REF] Barry | Barriers to open data release: A view from the top[END_REF] identified "Irish law can be a considerable barrier to making data available." (p. 145). In relation to the laws we also have External Safety [START_REF] Barry | Barriers to open data release: A view from the top[END_REF][START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Melin | Open government data in an e-service context -managerial and conceptual challenges[END_REF], where [START_REF] Kucera | Benefits and risks of open government data[END_REF] tells us "Detailed data about infrastructure (power plants, dams, transmitters etc.) might be misused to cause damage to the infrastructure." (p. 35). If something does go wrong (abuse, misinformation, fraud, accidents, etc.) there will be the question of Liability [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Barry | Barriers to open data release: A view from the top[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Melin | Open government data in an e-service context -managerial and conceptual challenges[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF]; if this is unclear, both publisher and users might be at risk of unexpected and unwanted responsibilities. Laws and norms also make Privacy [1, 3, 5, 16, 18-20, 23, 26-31, 39-42] intrusions something that must be avoided (a barrier), where the solution might be to clean the data, which can also make it lose value. Another solution is to use Embargo Periods [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF]. The final source of barriers is a lack of Skills for Release [START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF]; the organisation might lack the skills, competencies, and tools to support release decisions. If pushed, they might publish data that is a threat to external safety, useless, or reveals private information about citizens.

4.3

Publishing the Data

Publishing data is no easy task as there will be internal barriers, but also a need to consider usage. One barrier that can cause future barriers is a lack of Skills for Publishing [1, 3-5, 15, 18, 20, 22, 27, 34, 38, 39, 42], as it can e.g. result in inadequate infrastructure performance, lacking metadata descriptions, or inaccessible data. Another hurdle is Service barriers. Ownership [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Conradie | On the barriers for local government releasing open data[END_REF][START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF], as Susha et al. [START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF] tells us "For Groningen municipality, the biggest problem was to convince the data owners to publish data which they considered were not of sufficient quality." (p. 193); can be a barrier when the organisation are unsure of service quality. Unclear data ownership can also make it hard to know what can be published. Sufficient Quality [1, 5, 15, 18-20, 22, 23, 26, 39-42] must also be sought and guaranteed; else it can lead to future use barriers. The data might be the wrong format, unreliable, inaccurate, and incomplete. Quality connects technical infrastructure, data's suitability, and the users' needs. For the technical infrastructure, we are interested in external and internal barrier sources. For external barriers, we must consider Opening barriers. Here Access [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF], Metadata [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Yang | To open or not to open? determinants of open government data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF], and Format [1, 3-5, 15, 18, 20, 22, 27, 34, 38, 39, 42] can cause problems for prospecting users. Paywalls, registrations, and API only downloadable data can form access barriers. While metadata barriers can be none explained context, language, or purpose for the data, which can make it hard to identify opportunities and usages. On the other hand, formats that are badly documented or proprietary can hinder data integration and longterm sustainable usage. For the publisher selecting a format when there is no standard can be a barrier. The original storage system format can be a problem too, as exampled by Kassen [START_REF] Kassen | Open data in kazakhstan: incentives, implementation and challenges[END_REF] "…some data are really challenging to publish in a computer-readable format due to the fact that they only exists as a paper document…" (p. 317). In addition, converting the storage systems format to one that is open, machine-readable format can also be a barrier. For the internal technical infrastructure, we have Administration barriers, which likely will be experienced once data is published. The first is Performance [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Barry | Barriers to open data release: A view from the top[END_REF][START_REF] Hogan | Governance, Transparency and the Collaborative Design of Open Data Collaboration Plat-forms: understanding barriers[END_REF][START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Melin | Challenges and benefits in an open data initiative -a local government case study of myths and realities[END_REF][START_REF] Melin | Open government data in an e-service context -managerial and conceptual challenges[END_REF] where infrastructure might not be able to handle the external requests, which can affect the organisations day-to-day activities, but also impact users' perception of data quality (e.g. overloads that leads to system crashes). The second is Maintenance [5, 16, 18-20, 27, 29, 39, 42], where a possible increase in requests might require upgrading the technical infrastructure but can also lead to increased "wear and tear" (an increase in hardware replacement). As publishing is about opening closed data to third-party reuse the organisation is also opening for the public, therefore, the final is Security [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Wirtz | Resistance of public personnel to open government: A cognitive theory view of implementation barriers towards open government data[END_REF]. Where lacking security can lead to a threat against the core task, but also the external safety, as devious users gain access to private information or have the ability to damage the organisation's systems.

Using the Data

While there are user barriers outside the control of the publisher, some can be mitigated. First users must be able to find the data. Findability [3, 5, 16, 18-20, 23, 39, 41, 42] refers to the ability of the user to discover and identify the data, which to a degree can be solved with an open data portal. Findability related barriers could come from different sources, such as a market flooded with similar datasets or diffuse names. Zuiderwijk and Janssen [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF] tells us "Making open data findable poses the challenge for different government organizations to collaborate." (p. 122). Meaning that findability barriers is in the relation between publisher and user, between publishers, and both actors relation to the open data portal. Once users have found the data, they want access and read the License [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Jaakkola | Open data: Opportunities and challenges[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF]. The license can cause barriers when not explained, incomprehensible, or complicated. Another part is if the license restricts usage to much or is incompatible with other licenses. There is also a question about user Responsibility [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] that connects back to liability. An example is "…the UK Open Government Licence for instance requires that the re-user does not mislead others or misrepresent the Information or its source [45]…" [31, p. 7]. The authors argue that such clauses should not be needed, but also admit that it seemed to make public agencies more comfortable with publishing. We see that an unclear license can be a barrier for users, while unclear user responsibilities can be a barrier for the publishers. At this stage, the user might want to read Documentation [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF] that explains how the data can be used, accessed through an API, or see processing examples. Documentation connects back to metadata, but rather than explaining the what, it supports the user in the endeavor to use. In common for all above we have Language [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] as a barrier. If the users cannot read or understand websites, licenses, documentation, or the data it will likely complicate usage. We also have the Paywall [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]; the user must pay for data. A paywall can stop the users, especially if there is no way to express feedback of or experiment with the data. This means that a paywall is a barrier that can worsen if there is also bad metadata or documentation. Users will also experience barriers if either license, format, IT-systems, or data content lacks Interoperability [START_REF] Jaakkola | Open data: Opportunities and challenges[END_REF][START_REF] Kucera | Benefits and risks of open government data[END_REF][START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] with other data. The ability to combine data is important for open data to reach value. Interoperability can also come with collaboration barriers for publishers as standards are implemented. Another barrier is Timely [START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | The Negative Effects of Open Government Data -Investigating the Dark Side of Open Data[END_REF][START_REF] Zuiderwijk | Towards decision support for disclosing data: Closed or open data?[END_REF]; the data needs to reach the user before it spoils. The barrier here is long embargo period or slow access. While mentioned barriers focuses on the experiences of the users, we can also look at user Diversity barriers. Different Users [START_REF] Appelgren | Data journalism in sweden: Introducing new methods and genres of journalism into "old" organizations[END_REF][START_REF] Brugger | Current barriers to open government data use and visualization by political intermediaries[END_REF][START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF][START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF][START_REF] Susha | Driving factors of service in-novation using open government data: An exploratory study of entrepreneurs in two countries[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] comes with diverse needs and goals. E.g. visualise data or integrate data into an app. If not meet, users might be unable to use the data. On the other hand, publisher can find it hard to identify and meet users' needs. Tools / Software [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] can help users, but then we stand in for the question of who should implement them and how far. We also have the other side where data might need special tools to use, which can be a barrier if they are unusable or cost. Users also have their own barriers, for example "… current challenge facing data journalism in Sweden is the lack of time to collect, analyze and present data. This may be a result of streamlined organizations as well as not having enough competence and skills to efficiently work with data." [2, p. 404]. If data require special Skill for Usage [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Brugger | Current barriers to open government data use and visualization by political intermediaries[END_REF][START_REF] Hellberg | The story of the sixth myth of open data and open government[END_REF][START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF][START_REF] Hogan | Governance, Transparency and the Collaborative Design of Open Data Collaboration Plat-forms: understanding barriers[END_REF][START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF], users without them are at a disadvantage and how to supply these will be a barrier for both publishers and users. This leads us to Usability [3, 5, 16, 18-20, 29, 34, 39] as a barrier. Data loses usability when users face some combination of the barriers above or cannot use the data, in addition high amounts of unusable data in the data market can lower the findability of high quality data. This means that a combination of usability, awareness, and findability can result in Low Usage [START_REF] Hjalmarsson | Beyond innovation contests: A framework of barriers to open innovation of digital services[END_REF][START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Barriers and development directions for the publication and usage of open data: A socio-technical view[END_REF].

Evaluation

Once data is published and, hopefully, in use, it is relevant to evaluate the impact and collect feedback to improve. Here we can study Community barriers, such as lacking Channels [START_REF] Beno | Open data hopes and fears: Determining the barriers of open data[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Kassen | Open data in kazakhstan: incentives, implementation and challenges[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF]

Conclusions and Future Work

In this paper, we have identified and systematised OGD barriers into identifying data suitability, release decisions, data publishing, data usage, and evaluation based on a systematic literature review. We have illustrated the attention given to different barriers in OGD research from a longitudinal perspective and illustrated different types of barriers and focuses in the research field, including relating 46 OGD barriers to different parts and phases of the OGD process. With the illustrations in section 3, have we positioned and contextualised our research questions, and with the systematisation (Section 4), we have integrated the research questions to contribute to a result that is valuable for researchers in the area, and in the long run for the practice. In the analysis, we moved away from a perspective on barriers as something to be "easily" reduced or managed (e.g. regarding deciding to release OGD), towards viewing barriers as a network in a cycle and part of a strategic decision in a broader context. This is a way to expand the scope of OGD, where we move away from the reported myths surrounding open data.

We have also identified a knowledge gap between use and evaluation for barriers. The government can not solve barriers here, only mitigate by, e.g. ease usage. Our study opens for research from both the individual perspective and systematic perspectives on OGD barriers and their relations. One important and unexplored dimension of this; is the responsibilities of and division of labour between the publisher and user. We have also identified a need for further insight into open data barriers and myths, especially on the economic aspects, assumptions, and the gap between use and evaluation.

This paper indicates two research implications. [START_REF] Albano | Open government data: facilitating and motivating factors for coping with potential barriers in the brazilian context[END_REF] The reviews [e.g. [START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Hossain | State-of-the-art in open data research: Insights from existing literature and a research agenda[END_REF][START_REF] Zuiderwijk | Socio-technical impediments of open data[END_REF] are essential to identify and understand OGD barriers. To hold the knowledge body together, we need recurrent systematic literature reviews both on a general and specific level (e.g. usage or publication), where authors offer insights, research avenues, and address research barriers. One such avenue for a future systematic literature review is to investigate the focuses in previous OGD barrier studies. Hopefully, this will open for successful research and practice. (2) Current research does not support informed decisions towards a certain kind of usage; current research is rather general or even abstract. We need a thorough understanding of what barriers a publisher will encounter, but we lack the how and why. There is time to move to answer those questions and combine different conclusions to support and guide practice.

We have identified some limitations in our study. We did not use "open government data" as a keyword of the search. Instead, we used the broader "open data". The broader term usage results in that not all the literature were studies of governmental nature but may be relevant for that sub-field. We also used singulars instead of plural. We looked at the top 600 hits from Google Scholar, filtered with the year and different keywords. To avoid being too limited, we used snowballing, but there might still be important literature out there. Another limitation is that the systematisation is not yet evaluated in practice. We also acknowledge that barrier severity varies [e.g. 5]. As the systematisation is based on a broad pool of barrier research, we claim that the coverage, and to a degree the usability, of the systematisation is high, but taking the limitations into account more work can be done to evaluate and validate it.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Historical Development. Fig. 2. Barrier Types found in the literature

  On the other hand, if research has expanded; it can have expanded into new domains. E.g. from government into innovation [e.g. 14]. With this, the label used to address the OGD barriers might have changed to fit the new domain. Fitting with what we identified in 2012. There is also a

Fig. 3 .

 3 Fig. 3. Research focuses found in the literature (Based on abstracts).

  used to collect and communicate and low open data Awareness[START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Huang | Proposing a framework of barriers to opening government data in china[END_REF][START_REF] Kassen | Open data in kazakhstan: incentives, implementation and challenges[END_REF][START_REF] De Rosnay | Legal and institution-al challenges for opening data across public sectors: Towards common policy solutions[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF]. If end-users are not aware of their open data usage, they cannot tell how it impacts. In the inverse, if users have feedback, but no channel, the process will be hard to improve (and we lose impact insight too). Users that are not aware of OGD and its possibilities will not show Interest and Need[START_REF] Albano | Open government data: facilitating and motivating factors for coping with potential barriers in the brazilian context[END_REF][START_REF] Kassen | Open data in kazakhstan: incentives, implementation and challenges[END_REF][START_REF] Martin | Barriers to the open government data agenda: Taking a multi-level perspective[END_REF][START_REF] Martin | Open Data: Barriers, risks and opportunities[END_REF]. Even if they have awareness and channels, they can express interest, to express need they require access to the organisation's metadata inventory. Here we have connected interest and needed to supply and demand, as without expressed interest and need it will be hard to evaluate demand for some closed data. As we argued above, different users have diverse needs, and it, therefore, becomes important to think about Participation[START_REF] Attard | A systematic review of open government data initiatives[END_REF][START_REF] Hellberg | The story of the sixth myth of open data and open government[END_REF][START_REF] Hogan | Governance, Transparency and the Collaborative Design of Open Data Collaboration Plat-forms: understanding barriers[END_REF][START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF] in evaluation and feedback. Not involving a diverse group of users can result in a specialised solution that creates complications for other users. There is also another barrier related to participation, as Hellberg and Hedström[START_REF] Hellberg | The story of the sixth myth of open data and open government[END_REF] say "We believe that not everyone is interested in using public data, even if they have the necessary resources and competences." (p. 47). We see a possibility that low awareness and uniform participation can lead to specialised solutions, where not all with capability are interested in participation, which lowers usability by exclusion. Let us say we have participants that want to give us feedback; then we need to think about Coordination barriers. Here Quality in Evaluation[START_REF] Hjalmarsson | Mind the gap: Exploring stakeholders' value with open data assessment[END_REF][START_REF] Janssen | Benefits, adoption barriers and myths of open data and open government[END_REF][START_REF] Kassen | Open data in kazakhstan: incentives, implementation and challenges[END_REF] can be a challenge, as feedback can be in multitude and varied or users are without supporting documentation and tools; the user might also not know what to look for. Once we have participants with feedback, it becomes a question of Feedback and Error Handling[START_REF] Susha | Organizational measures to stimulate user engagement with open data[END_REF], which connects back to administration. If errors and feedback are not handled problems will continue constraining the open data impact. Moreover, if changes are not communicated to the users, it might cause complications in their use of open data, where consequences can be hard to anticipate. Based on the identified literature, we see that evaluation and feedback need research, especially in relation to barriers and on both active and passive evaluation.