Anne Fleur Van Veenstra
email: annefleur.vanveenstra@tno.nl

Bas Kotterink
email: bas.kotterink@tno.nl

Data-Driven Policy Making: The Policy Lab Approach

Keywords: Data-driven policy making, data for policy, co-creation, Policy Lab

Societal challenges such as migration, poverty, and climate change can be considered 'wicked problems' for which no optimal solution exists. To address such problems, public administrations increasingly aim for data-driven policy making. Data-driven policy making aims to make optimal use of sensor data, and collaborate with citizens to co-create policy. However, few public administrations have realized this so far. Therefore, in this paper an approach for datadriven policy making is developed that can be used in the setting of a Policy Lab. A Policy Lab is an experimental environment in which stakeholders collaborate to develop and test policy. Based on literature, we first identify innovations in data-driven policy making. Subsequently, we map these innovations to the stages of the policy cycle. We found that most innovations are concerned with using new data sources in traditional statistics and that methodologies capturing the benefits of data-driven policy making are still under development. Further research should focus on policy experimentation while developing new methodologies for data-driven policy making at the same time.

Introduction

Today's society faces complex 'wicked problems', such as migration, poverty, and climate change, for which not one optimal solution exists [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. In order to address such problems, governments aim to realize public sector innovation that gears them towards becoming platforms of open governance, making optimal use of information and communication technologies (ICTs) to create public value [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF]. Increasingly, ICTs are not only used for improving the daily operations of government, but also for enhancing the process of policy making [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. Policies address societal problems by formulating and implementing laws, rules and guidelines, and policy making is the process of creating and monitoring these policies. Hence, it is often conceptualized as a policy cycle, consisting of several different phases, such as agenda setting, policy formulation, decisionmaking, implementation and evaluation [START_REF] Lasswell | The policy orientation[END_REF]. ICTs may be used to support and enhance different phases of the policy cycle and enable experimentation [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. Data-driven policy making uses ICTs to capture the benefits of new data sources [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF], and to support collaboration with relevant stakeholders and citizens [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Linders | From e-government to we-government: Defining a typology for citizen coproduction in the age of social media[END_REF]. It builds on the notion of evidence-based policy making [see, for instance, [START_REF] Solesbury | Evidence Based Policy: Where it Came and Where it's Going[END_REF][START_REF] Head | Three Lenses of Evidence-Based Policy[END_REF]. In the literature on evidence-based policy making three types of evidence are considered relevant: "systematic ('scientific') research, program management experience ('practice'), and political judgement" [9, p. 1]. Data-driven policy making acknowledges the importance of these types of evidence, but can be distinguished from evidence-based policy making, since it is mainly concerned with the inclusion of big and open data sources into policy making as well as with co-creation of policy by involving citizens. Datadriven policy making is not only expected to result in better policies, but also aims to create legitimacy [START_REF] Bijlsma | An Empirical Analysis of Stakeholders' Influence on Policy Development[END_REF]. Involvement of citizens in a data-driven policy making process is especially important since public data and statistics are increasingly met by citizens' distrust [START_REF] Davies | How statistics lost their power -and why we should fear what comes next[END_REF].

To allow for better collaboration and involve citizens, public administrations around the world have set up Policy Labs to allow for experimentation and facilitate the involvement of relevant stakeholders [START_REF] Williamson | Governing methods: policy innovation labs, design and data science in the digital governance of education[END_REF][START_REF] Fuller | Public policy labs in European Union Member States[END_REF]. They, thus, address the need for experimentation and design-thinking to deal with wicked policy issues [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. Therefore, in this paper we develop a Policy Lab approach for data-driven policy making. First, based on literature of public sector innovation, we identify innovations in the use of data for policy making and co-creation of policy. Secondly, we map these innovations to different phases of the policy cycle. And thirdly, we develop an approach that can be used to guide data-driven policy making in a Policy Lab setting. The next section presents the theoretical background of public sector innovation. Section 3 discusses data-driven policy making and identifies innovations. Subsequently, section 4 presents the development of the Policy Lab approach, followed by a discussion and recommendations for further research in section 5. Finally, section 6 presents the conclusion.

Public Sector Innovation

Public sector innovation holds that "[p]ublic policy and services need to become more open and innovative as well as being efficient and effective" [1, p. 2], making optimal use of ICTs [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF]. As such, it encompasses a myriad of aspects. Gil-Garcia, Zhang & Puron-Cid [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF] refer to as much as fourteen aspects of smartness in government, including evidence-based, technology savviness, openness, citizen engagement, and innovation. According to Millard [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF], public sector innovation means that public administrations operate as a platform [START_REF] Janssen | Lean government and platform-based governance -Doing more with less[END_REF][START_REF] Meijer | E-governance innovation: barriers and strategies[END_REF] and use ICT to collaborate across organizational borders [START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF] and to involve citizens and other relevant stakeholders [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Linders | From e-government to we-government: Defining a typology for citizen coproduction in the age of social media[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF] with the purpose of creating public value [START_REF] Moore | Creating Public Value Strategic Management in Government[END_REF][START_REF] Stoker | Public Value Management A New Narrative for Governance? The American Review of Public Administration[END_REF][START_REF] Bannister | ICT, public values and transformative government: A framework and programme for research[END_REF]. Over the past decades, ICTs have had a great impact on services delivery [START_REF] Dawes | The Evolution and Continuing Challenges of E-Governance[END_REF], opened up public datasets [START_REF] Mcdermott | Building open government[END_REF] and increased citizens' participation [START_REF] Chadwick | Bringing E-Democracy Back In Why it Matters for Future Research on E-Governance[END_REF]. The use of ICTs for policy making can, thus, be seen as a next step in public sector innovation [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. The use of ICTs benefits policy making in two ways. The first is the use of new data sources, such as (real-time) sensor data, either physical (e.g. traffic monitoring [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF]), or virtual (e.g. social media data [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]). "Data-driven decisions and intensive use of data, through ubiquitous sensing, advanced metering and integrated applications enable governments to make more informed decisions and improve the effectiveness of public policies and programs" [14, p. 527]. Secondly, it requires from governments to collaborate across organizational borders and with citizens and businesses to enable co-creation of policies [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Linders | From e-government to we-government: Defining a typology for citizen coproduction in the age of social media[END_REF][START_REF] Meijer | E-governance innovation: barriers and strategies[END_REF]. "Co-creation is understood as the active flow and exchange of ideas, information, components and products across society (academia, government, business, civil society and citizens) which allows for a better understanding of participation, engagement and empowerment in policy development" [1, p. 5].

Besides the deployment of ICTs to use new data sources and enable co-creation of policies, public sector innovation is concerned with the ability of public administrations to experiment, using innovative approaches such as gaming, simulation, and installing of sensors for do-it-yourself measurements, and deploy 'design-thinking' [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF]. In order to do so, many public administrations have set up Policy Labs [START_REF] Williamson | Governing methods: policy innovation labs, design and data science in the digital governance of education[END_REF][START_REF] Fuller | Public policy labs in European Union Member States[END_REF]. "Policy Labs are emerging structures that construct public policies in an innovative, designoriented fashion, in particular by engaging citizens and companies working within the public sector" [13, p. 2]. Policy Labs exist in all shapes and sizes and on different levels of government (national, regional and municipal) [START_REF] Fuller | Public policy labs in European Union Member States[END_REF]. The majority of Policy Labs do not focus on a specific type of policy or on a specific phase of the policy cycle, but they employ a design and experimentation based approach to policy making [START_REF] Fuller | Public policy labs in European Union Member States[END_REF]. As such, Policy Labs can be considered as a specific instance of Living Labs, which aim to "support public open innovation processes" [26, p. 90]. While Living Labs are concerned with the involvement of private sector organizations as well as citizens in public open innovation processes in general [START_REF] Gasco | Living Labs: Implementing open innovation in the public sector[END_REF], Policy Labs focus on the involvement of citizens (and also other stakeholders) into the policy making process specifically.

Innovations in Data-driven Policy Making

Data-driven policy making thus aims to use new data sources such as (real-time) sensor data and new techniques for processing these data and to realize co-creation of policies, involving citizens and other relevant stakeholders. However, realizing data-driven policy making is complex: many challenges related to the capturing, integration and re-use of data exist [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF], as well as to the involvement of citizens and other stakeholders in policy making [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Linders | From e-government to we-government: Defining a typology for citizen coproduction in the age of social media[END_REF]. This section identifies innovations of data-driven policy making based on literature.

Use of new data sources in policy making

The use of new data sources holds big promises: it is expected to offer organizations greater operational efficiency and effectiveness, and lead to the development of new products, services and business models [START_REF] Manyika | Big Data: the next frontier for innovation, competition, and productivity[END_REF][START_REF] Cukier | Big Data: a Revolution That Will Transform how we Live, Work, and Think[END_REF][START_REF] Chen | Business Intelligence and Analytics: From Big Data to Big Impact[END_REF]. In the context of governments, "we are faced with a deluge of data that, when combined with new technologies and analysis techniques, has the potential to inform decision and policy making in unprecedented ways" [4, p. 10]. Big data is often defined as "vast datasets that cannot be analyzed using conventional software and analytic tools" [4, p. 2]. Since many 'big data' sources can be stored on a USB-stick nowadays, in the context of public administration, important characteristics of big data are not so much that they require large processing power, but more the variety and the interoperability because of its different data sources and formats [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF]. The use of (sensor) data in policy making encompasses three steps: capturing data, integrating data from different sources, and applying these data [START_REF] Harrison | Foundations for smarter cities[END_REF]. Table 1 summarizes the main opportunities, challenges and innovations per step.

Table 1. Opportunities, challenges and innovations of new data sources for policy making.

Steps of data use Opportunities Challenges Innovations Capturing data Availability of (realtime) sensor data [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF], open data [START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF] and social media data [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

Variety in data [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF], data quality [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF], reliability of data [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF], and security of data [START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF].

Crowdsourcing [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF]; nowcasting [START_REF] Taylor | Emerging practices and perspectives on Big Data analysis in economics: Bigger and better or more of the same?[END_REF].

Integrating data Cross-organizational collaboration [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF]; linking new data sources to traditional statistics [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF][START_REF] Brakel | Social media as a data source for official statistics; the Dutch Consumer Confidence Index[END_REF].

Interoperability [START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF]; lack of standardization, architectures, and portals [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF]; legacy systems [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF].

Sentiment analysis [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF], location mapping [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF], advanced social network analysis [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

Application of data

Real-time monitoring of policy [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]; transparency and accountability [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF].

Sense-making and interpretation [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

Visualization techniques [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]; computer simulation [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF].

Table 1 shows that public administrations increasingly see opportunities for the use of new data sources, mainly (real-time) sensor data [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF]. These data can be physical, such as roadside monitoring, but also virtual, such as social media data. A study from 2015 finds that governments mainly make use of two types of data for data-driven policy making: "public datasets (administrative (open) data and statistics about populations, economic indicators, education, etc.) that typically contain descriptive statistics, which are now used on a larger scale, used more intensively, and linked [… and …] social media, sensors and mobile phones that are […] analyzed with novel methods such as sentiment analysis, location mapping or advanced social network analysis" [31, p. 3]. Main issues are whether the data are of sufficient quality [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF], and whether they are reliable and secure [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF][START_REF] Hui | Creating Public Value in E-Government: A Public-Private-Citizen Collaboration Framework in Web 2.0[END_REF]. Otherwise, they may undermine the policy making process [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF]. Innovations in capturing data are crowdsourcing [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF], and nowcasting, which is the capturing of search engine data [START_REF] Taylor | Emerging practices and perspectives on Big Data analysis in economics: Bigger and better or more of the same?[END_REF].

Regarding integration of data, to make successful use of big and open data in organizational processes, cross-boundary information integration (in between government agencies and between not-for-profit organizations and private firms and the public sector) is necessary [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF]. The integration of data is becoming more important: linking these data sources with data sources that are traditionally used for policy making such as statistics, surveys and organizational databases is becoming the norm [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF][START_REF] Brakel | Social media as a data source for official statistics; the Dutch Consumer Confidence Index[END_REF]. However, many challenges exist: interoperability of data and lack of standardization, architectures and portals [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF][START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF]. Another issue are legacy systems that may negatively influence this linking [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Janssen | Benefits, Adoption Barriers and Myths of Open Data and Open Government[END_REF]. Poel et al. [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF] conclude that currently privately held data is of less relevance, as they are still hardly shared. Opportunities for data integration include sentiment analysis, location mapping, and social network analysis [START_REF] Bertot | Big data and e-government: issues, policies, and recommendations[END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

The third step in the use of new data sources is application and sense-making. While social media analysis and network analysis can be seen as forms of data integration that can be used to support the policy making, we consider the use of visualization tools and computer simulations to be applications of data to the actual process of policy making [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF]. However, "[a]mong the initiatives examined, there is little use of advanced analytics or visualization techniques" [31, p. 4]. Another opportunity is to realize greater accountability [START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF]. Likely, the most innovative use of new datasets take place in the hidden spheres of fighting crime and terrorism [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

Co-creation of policy

Another essential element of smartness in government is co-creation of policy, as ICT not only allows for collaborating with other organizations (public or private), but also with citizens [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Gil-Garcia | Conceptualizing smartness in government: An integrative and multi-dimensional view[END_REF]. Co-creation is the exchange of ideas and information between relevant actors, such as governments, businesses, civil society and citizens that lead to the develop of policies [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. Involvement of citizens in policy making is especially important since public data and statistics are increasingly met by citizens' distrust [START_REF] Davies | How statistics lost their power -and why we should fear what comes next[END_REF]. This can take on different forms, depending on the level of involvement [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]: it may range from merely informing public administrations, for example by tapping discussion fora, opinion polls and using social media [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF], to participating in decision making and in policy implementation. Table 2, which is based on Janssen & Helbig [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF], summarizes the main innovations and challenges to co-creation of policies.

Table 2. Opportunities, challenges and innovations in co-creation of policies.

Levels of involvement

Opportunities Challenges Innovations

Informing and signaling

Citizens identify problems and set the agenda [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF].

Social inclusion and overcoming exclusion [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF][START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]; lack of stability of social media [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Crowdsourcing [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]; online petitions [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]; participatory sensing [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF].

Decision making

Citizens being involved in selecting options [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Citizens' skills and motivation [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Meijer | E-governance innovation: barriers and strategies[END_REF]; skills and culture of the government agency [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Computer simulation and serious games [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF]; cross-platform social media analysis [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Implementation

Co-creation between governments, citizen and businesses [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]; policy evaluation [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF]; transparency and accountability [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Privacy and security [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]; accuracy [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

Camera surveillance, smart phone data, use of sensors [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]; agile implementation [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF].

The most basic form of citizen involvement is informing and signaling, meaning that citizens' information is used for identifying problems and setting the agenda [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. Main challenges for this level are to make sure that different groups of citizens are represented, without excluding relevant groups [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. Examples of this happening can be found in literature on using social media data during disasters and disease outbreak. While nowcasting using search engine data for predicting flu outbreaks can be an accurate predictive methodology; for predicting Ebola, this method proved to be much less accurate since in the areas where the main outbreak was, internet access is still scarce [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]. Furthermore, the stability of social media is a challenge for its use in signaling problems [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. Innovations in using citizens' ideas include crowdsourcing [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF], online petitions [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF], and participatory sensing [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF].

The inclusion of citizens' opinions in decision making refers to a higher level of involvement. This means that citizens are involved in the evaluation of policy options [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. The most elaborate form of this is the organization of a referendum, but using social media or other online tools, this could be done more efficiently and effectively [START_REF] Millard | Open Governance Systems: Doing More with More[END_REF][START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF]. Important challenges are to ensure that both citizens' and skills and motivation [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Meijer | E-governance innovation: barriers and strategies[END_REF] and that civil servants' skills and culture [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF] are sufficient. Innovations in involving citizens in the choice for different policy options and decision making are computer simulations and serious games [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF], and cross-platform social media analysis [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF].

The third level of involvement is implementation of policies, which can be seen as the most immersive level of co-creation. Opportunities for co-creation include collaboration between public administrations, private companies and citizens in policy implementation [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF], policy evaluation [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF], and transparency and accountability [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. Challenges include privacy and security [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF] and accuracy [START_REF] Ferro | Policy making 2.0: From theory to practice[END_REF]. Innovations in this level of involvement include camera surveillance, the use of smart phone data and sensors [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF], and allowing for agile implementation, delivering faster and better innovations because of regular and short-cycle interactions [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF].

4

The Policy Lab Approach

In the previous section we identified opportunities, challenges, and innovations based on literature of new technologies and co-creation in policy making. This section aims to present a coherent Policy Lab approach to data-driven policy making based on the innovations in these fields. Since the framework is to be used for policy making, we mapped these innovations to phases of the policy cycle [START_REF] Lasswell | The policy orientation[END_REF]. Inspired by Janssen & Helbig [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF], we distinguish three phases: predictive and problem definition, design and experimentation, and evaluation and implementation. Table 3 elaborates innovations and impact per phase of the Policy Lab approach, and identifies challenges. The first phase of policy making -predictive and problem definition, (real-time) sensor data is used, comprising physical sensor data such as roadside traffic data, and virtual data such as social media data. Furthermore, innovative approaches such as crowdsourcing and nowcasting are also used to predict and identify problems. This leads to the availability of (real-time) information that allows more precise predictions than those that are merely expert based. However, experts are still important to provide context information to the trends spotted by the data. Main challenges are the availability, quality, reliability and security of the data as well as representativeness of the data that should include viewpoints of different groups of citizens without excluding relevant groups. In a study on the use of data for policy making from 2015, over half of the cases identified were used for this first phase of policy making [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]. Accuracy of data and data models, ensuring privacy and security. Citizens' skills and motivation and skills and culture of the government agency need to be sufficient.

The second phase of policy making -design and experimentation, should ensure collaboration between government, private organizations, and citizens in the decision making process and choice for policy options. This requires the use of more advanced analytical approaches such as sentiment analysis, location mapping, social network analysis, visualization techniques, computer simulation and serious games to allow for the involvement of other stakeholders in the decision making. A major challenge for the integration of different data sources, the performance of more advanced analyses, and ensuring involvement of citizens is setting up an infrastructure that allows for interoperability and integration of data [START_REF] Gil-Garcia | Government inter-organizational information sharing initiatives: Understanding the main determinants of success[END_REF]. Standards, architectures and portals can be instruments for this. Traditionally, governments more often involve citizens after this phase, in the implementation, rather than in the process of decision making. This is reflected in the lower number of best practices in this phase [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF].

Evaluation and implementation -the third phase of policy making, allows for joint policy implementation and co-creation of services by government, businesses and citizens. An advantage of the use of new data sets and technologies is the use of an agile approach [START_REF] Janssen | Lean government and platform-based governance -Doing more with less[END_REF] that allows for short cycles of decision making and implementation. The involvement of relevant stakeholders in the implementation and ongoing monitoring of policy creates public value [START_REF] Moore | Creating Public Value Strategic Management in Government[END_REF][START_REF] Stoker | Public Value Management A New Narrative for Governance? The American Review of Public Administration[END_REF][START_REF] Bannister | ICT, public values and transformative government: A framework and programme for research[END_REF]. More insight and collaboration may result in greater transparency and accountability, but also to more surveillance. Accuracy of data and data models and ensuring privacy and security are major challenges. Furthermore, cocreation of policy requires specific skills and motivation of citizens as well as specific skills and culture of the government agency [START_REF] Janssen | Innovating and changing the policy-cycle: Policy-makers be prepared![END_REF][START_REF] Meijer | E-governance innovation: barriers and strategies[END_REF]. While in traditional e-participation, citizens are involved in policy implementation, actual co-creation involving citizens in the production of services is less often found in practice [START_REF] Linders | From e-government to we-government: Defining a typology for citizen coproduction in the age of social media[END_REF].

These innovations are challenging and in practice most governments do use new technologies and data sets for policy making, but they use this to enrich traditional statistical data rather than achieving co-creation [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]. Therefore, besides allowing for experimentation with policy making, new methodologies need to be developed that are able to make use of these new data sources and technologies. Using a design science approach [START_REF] Hevner | Design science in information systems research[END_REF], we developed the Policy Lab approach that can be used to guide innovations in data-driven policy making, allowing for experimentation with new policies and developing new data-driven methodologies at the same time. To validate this approach we held five internal workshops with experts that took place over the course of 2016. Furthermore, throughout this process we consulted academic and governmental stakeholders: four representatives of three academic institutions and six representatives of the national and local levels of government were involved. The Policy Lab approach is graphically presented in Figure 1. The conceptualization of the Policy Lab approach presented in Figure 1 consist of two circles. The inner circle is represents the policy making process, consisting of several phases, such as agenda setting, policy formulation, decision-making, implementation and evaluation. The outer circle of the Policy Lab approach focuses on the development of data-driven methodologies and co-creation. This approach allows the two circles to mutually influence each other: policy experiments can be used to develop and test new methodologies, that, in turn can be used for developing and evaluation policies.

Discussion

Based on the literature review, we found that most applications of new data sources, such as (real-time) sensor data are to link them to traditional statistics and few innovative methodologies are used for policy making [START_REF] Poel | Data for Policy: A study of big data and other innovative data-driven approaches for evidence-informed policymaking[END_REF]. However, "utilizing such [social] channels for policy making purposes does not constitute an established approach yet" [START_REF] Koussouris | Accelerating Policy Making 2.0: Innovation directions and research perspectives as distilled from four standout cases[END_REF]. This means that first of all instruments and methodologies for the use of these new data sets in traditional statistical and econometric methodologies should be developed [START_REF] Varian | Big Data: New Tricks for Econometrics[END_REF]. Furthermore, in order for governments to become used to these methodology, they could very well use the use the 'design-thinking approach' of a Policy Lab that allows for experimentation. This means that the Policy Lab approach, effectively, has three pillars: using new technologies and data sources for policy making, enabling co-creation and allowing for experimentation. The use of new datasets in traditional statistical or econometric studies is widely regarded to have a large potential for policy making. Traditional data sources are often text based or have a strong qualitative character rather than a numerical or machine generated form. Newer data sources are often human generated (social media) data, or machine generated sensor data. This can also be seen as the main distinction between data-driven and evidence-based policy. Using these newer data sources means that not only new methodologies need to deal with the size of these new data sets, but also with the variety of data, that may range from traditional statistics, to (real-time) sensor data to human generated text based social media data to images, video streams or geo-data. Statisticians and econometrists aiming to deal with these new (big) data sets, need to learn ways to incorporate them into their traditional methodologies [START_REF] Varian | Big Data: New Tricks for Econometrics[END_REF].

Fundamentally, there are no contradictions between big data and traditional econometric approaches, but the two have developed independently. For example, the use of big data sets enhances statistics in prediction methods (out-of-sample), which is often not possible in traditional econometrics because data sets are not large enough [START_REF] Varian | Big Data: New Tricks for Econometrics[END_REF]. Furthermore, when using big data sets it makes more sense to focus on model uncertainty than on sampling uncertainty, which is often examined in traditional econometrics. Finally, machine learning techniques such as decision tree learning may give a better picture than logistic regression [START_REF] Varian | Big Data: New Tricks for Econometrics[END_REF]. Traditional statistics, in turn, provide useful methods to help variable selection in big data models such as stepwise regression penalized regressions and Bayesian techniques (including time series analysis) [START_REF] Varian | Big Data: New Tricks for Econometrics[END_REF].

However, while these new methodologies could benefit from the incorporation of big data and linking them with traditional methodologies, traditional policy models are far from obsolete. Big data mainly concerns the discovery of correlations, while policy models present causations that have been developed based on practical experience [START_REF] Cowls | Causation, Correlation, and Big Data in Social Science[END_REF]. Causation hypotheses can ultimately be confirmed using controlled or natural experiments, and, thus, cannot be replaced with big data analyses alone. The degree to which the outcomes of such combinations of big data and statistical models can be explained, thus, represents a major issue. Therefore, the involvement of citizens and experimentation become paramount. This is even more the case in this 'post-factual' era, in which citizens are critical of official statistics and data [START_REF] Davies | How statistics lost their power -and why we should fear what comes next[END_REF]. A Policy Lab setting can be used for controlled experimentation allowing people to 'buy into' data, statistical methods and data-driven policies.

Similar to the challenges that Living Labs face, the Policy Lab approach, as a specific instance of a Living Lab, presents the risk of becoming primarily focused the implementation of an open innovation approach, rather than with achieving specific results [START_REF] Gasco | Living Labs: Implementing open innovation in the public sector[END_REF]. While involvement of new data sources and citizens in the policy making process are important objectives, the primary aim should be to improve policy making. If this is not achieved, this may result in a limited application of data-driven policies outside of the Policy Lab environment. This also means, as is the case for Living Labs, that scaling and sustainability are major challenges [START_REF] Gasco | Living Labs: Implementing open innovation in the public sector[END_REF].

Further research should thus focus on the development of these new methodologies that allow for combination of new data sources with traditional statistical data and the combination of big data methodologies with econometrics. Furthermore, experiments with policy development that address wicked problems should be carried out both to involve citizens and increase legitimacy of these policies and to capture the benefits of these new approaches for policy makers. This means that the Policy Lab approach should be validated and expanded based on these experiments. Finally, the issue of scalability and sustainability should be further explored to capture the benefits of datadriven policy making outside of the Policy Lab setting.

Conclusion

New data sources and ICTs have great potential for improving policy making. However, data-driven policy initiatives are scarce and the existing initiatives are, often, cases linking (real-time) sensor data to traditional statistical analyses. Therefore, using a design science approach, this paper develops a Policy Lab approach. Based on literature, we identified innovations in the use of new data sources and in co-creation of policies. The involvement of citizens will likely become more important for the legitimacy of statistics and data and policies. Subsequently, we mapped these innovations to the different phases of the policy cycle. Based on this overview, the Policy Lab approach draws on three aspects: using new data sources, co-creation and experimentation with policy making focusing on real-life wicked problems. The experiments can be used to develop data-driven policies as well as to develop new data-driven methodologies. Further research should focus on the development of methodologies for incorporating big data analyses into traditional statistical analysis and on experimentation with policy issues, thereby validating the Policy Lab approach.

Figure 1 .

 1 Figure 1. The Policy Lab approach for data-driven policy making.

Table 3 .

 3 Innovations, impact and challenges of data-driven policy making.

	Policy Cycle Phase	Innovations	Impact		Challenges
	Predictive & prob-	Use of (real-time)	Problem definition	Capturing different
	lem definition	sensor data from cit-	based on (real-time)	data sources and en-
		izens (e.g. social me-	data from different	suring data quality,
		dia data, crowd-	actors, rather than	reliability and secu-
		sourcing), business	merely	expert	rity as well as repre-
		and government for	based.		sentativeness of the
		problem definition			data.
		and prediction.
	Design & experi-	Using	advanced	Cross-organizational	Creating an infra-
	mentation	analyses, such as	collaboration and in-	structure ensuring
		sentiment analysis,	volvement of citi-	interoperability and
		location mapping,	zens require more	allowing for integra-
		social network anal-	advanced analyses	tion of data, in the
		ysis, visualization,	to be able to select	form of standards,
		computer simula-	policy options.	architectures, and
		tion and serious			portals.
		games for decision
		making.
	Evaluation & imple-	Collaborative data-	Public value crea-
	mentation	driven policy imple-	tion,	improved
		mentation by gov-	transparency	and
		ernments, citizens	accountability, but it
		and businesses, al-	may also lead to
		lowing for agility of	more surveillance.
		processes.


Guoray Cai
email: cai@ist.psu.edu

Feng Sun

Jessica Kropczynski
email: jkropczynski@ist.psu.edu

Crystallizing Local Political Knowledge for Informed Public Participation

Keywords: Civic engagement, eParticipation, online deliberation

Municipal governments often struggle to inform and engage citizens around local issues. Due to complexities of local politics and the diverse expressions in public and private spheres, citizens face a huge information barrier towards meaningful participation. To overcome such barrier, we explore a solution to provide citizens with clear, useful, and trustworthy information. We describe a framework for accomplishing this goal through issue-based knowledge crystallization. In order to put this framework into test, we devised Community Issue Review (CIR) as a concrete process for crystallizing local political knowledge. CIR is a structured deliberative process that use a citizen panel to conduct analysis of data relevant to a pending issue. We describe CIR in three aspects of its functions: institutional design, deliberative process, and productive outcome. Three special characteristics of CIR are emphasized: (1) fully embedded within local decision-making context; (2) hybrid (face-toface and online) deliberation; (3) facilitation on collaborative decisionanalysis. We present the iterative design of the CIR process and the lessons learned from field practices in a local community.

Introduction

Democracy empowers the public through their influence on political decisions [START_REF] Gastil | The Deliberative Democracy Handbook: Strategies for effective civic engagement in the 21st century[END_REF]. Policy issues in local governments are complex and contentious. While electronic government applications provide opportunities for broad participation, meaningful participation of public decision-making requires the ability of the participants to produce reasonable, well-informed opinion in light of discussion, new information, and claims made by fellow participants [START_REF] Chambers | Deliberative Democratic Theory[END_REF]. Such ability is often hindered by the lack of exposure to a diverse marketplace of ideas [START_REF] Price | Opinion Quality in Public Opinion Research[END_REF]. Since most citizens are not experts on public issues, their ability to contribute to public decision making is vitally based on comprehending the necessary information from media and others [START_REF] Panagiotopoulos | Supporting public decision making in policy deliberations: an ontological approach[END_REF]. Informing the public with adequate knowledge about policy issues is hard for many reasons. Understanding complex policy issues requires synthesizing three types of knowledge. First, the public needs to be informed by good science that characterizes the potential consequences (benefits and risks) of any policy option [START_REF] Council | Understanding risk: informing decisions in a democratic society[END_REF]. Second, it must give all stakeholders an opportunity to express their social, economic, and ethical concerns. Third, it must be informed by understanding the institutional, political, legal, and operational contexts of decision-making. These knowledge can be buried in a plethora of on-line and off-line information sources.

Recent proliferation of on-line participation platforms have significantly enriched the channels of expressing opinions on public matters [START_REF] Friess | A systematic review of online Deliberation Research[END_REF], but at the same time, it creates information glut to citizens' use. Typically, information about a policy issue is buried in new media, community forums, government web pages, documents, and reports, community meeting notes, as well as in the minds of experts, residents, and other stakeholders. Messages in these data are poorly framed, piece-wise, difficult to connect, redundant, and inconsistent. Unfortunately, interpreting and synthesizing such data are challenging tasks that few citizens are prepared to deal with. This problem has by far received little attention, and no practical solution has been proposed. This problem is well recognized in the literature of deliberative democracy [START_REF] Gudowsky | The Role of Information in Public Participation[END_REF] and policy communication studies [START_REF] Richards | Informational Barriers to Effective Policy-Public Communication: A Case Study of Wind Energy Planning in Saskatchewan, Canada[END_REF]. Elliman et al. [START_REF] Elliman | A model Building Tool to support Group Deliberation (eDelib)[END_REF] emphasized that the most fundamental barrier in public deliberations is the large amount of heterogeneous knowledge that needs to be made explicit in different formats at different stages of public opinion formation.

To address this problem of the wide dispersion of local knowledge, various computer-mediated systems and data mining techniques have been developed to automatically discover and aggregate diverse sources. Kavanaugh et al [START_REF] Kavanaugh | Encouraging Civic Participation Through Local News Aggregation[END_REF] developed a Virtual Town Square (VTS), a local news aggregator, that affords civic interaction through tagging, commenting, and sharing insights. However, even data are aggregated, they may still be too large for the public to make sense of them. Automated textual analysis tools have been used to detect important messages and alert analysts. Hagen et al [START_REF] Hagen | Introducing Textual Analysis Tools for Policy Informatics: A Case Study of e-Petitions[END_REF] automatically analyzed thousands of petitions to generate more concise reports for decision makers. Topic modeling methods [START_REF] Yin | Latent Community Topic Analysis: Integration of Community Discovery with Topic Modeling[END_REF] are useful here because it summarizes the most popular topics that appear news articles and blogs and representing them together in an intuitive way. Automated methods (as mentioned above) can improve the accessibility of community information sources. However, they are far from providing actionable knowledge to citizens. Useful knowledge has to be discovered from the data and be contextualized for certain tasks [START_REF] Rowley | The wisdom hierarchy: Representations of the dikw hierarchy[END_REF]. It is cognitively difficult and timeconsuming for a person to make sense of large and complex data. This paper argues for the need to communicate policy relevant knowledge more effectively to the public in order to maximize the chance of their meaningful participation with the constraints of the limited cognitive capacity and attentional resources. This need can be met by incorporating an explicit phase of "knowledge crystallization" before engaging the broader public to elicit their policy preferences. We present a conceptual framework for structuring knowledge crystallization tasks (section 2). Following this framework, we propose a concrete process, community issue review (CIR), that can be practiced as a policy knowledge co-creation tool in a variety of policy-decision contexts (section 3). We implemented and used CIR in evaluating a real community proposal and observed the positive impact to the level of engagement (section 4 and section 5). In the same time, we received feedback on how the CIR process and the supporting technology can be improved and be made more flexible to support best practices (section 6).

Issue-Based Knowledge Crystallization for Democratic Deliberation

Our research addresses the need of crystallizing knowledge to overcome the information glut experienced by local communities when residents are called for participating public deliberation on a pending policy issue or proposal for action. Instead of pouring a large amount of messy data to all the members of a local community, our solution is to crystallize the knowledge about the policy issue into a set of clearly stated findings, called Citizen's Statements, that is amenable to human mental processing [START_REF] Resnikoff | Empirical Foundations of Information and Software Science IV, chap[END_REF] by the lay public. We define knowledge crystallization as a process that aims to produce a most insightful and compact description of the relevant content of a data set for a given task without removing crucial information. Examples of knowledge crystallization tasks include writing a business intelligence newsletter, reporting on the analysis of a business strategic management practice, or a scientist writing a literature review article [START_REF] Card | Readings in information visualization: using vision to think[END_REF]. Issue-based Knowledge crystallization (IBKC) takes all the data that we can collect about a particular public issue, and puts them through a systematic process of distilling relevant nuggets, purifying, abstracting, and compacting to create a best and most accessible "form" of knowledge for human consumption.

Crystallization is a metaphor borrowed from chemical engineering, where the goal of crystallization is to produce a highly purified and ordered crystal lattice from raw materials through the processes of purification and condensation. We use the concept of knowledge crystals to represent a form of knowledge that is highly purified, compact, succinct, structured, and solid.

An overview of the knowledge crystallization process is shown in Figure 1. The rectangular boxes represent entities involved in the process. The arrows represent flow relationships among them. This process has four small loops and has one set of loops that cycle around knowledge evaporation and another that cycles around knowledge condensation, with plenty of interaction between these. This process is guided by a knowledge schema that reflects the structure of inquiries used by the decision-makers. A bigger rectangular task wraps the entire process and serves as the context.

Community Issue Review

Community Issue Review (CIR) is a community-level panel-based deliberation process for crystallizing knowledge about a pending community issue [START_REF] Kropczynski | Understanding the Roles of Artifacts in Democratic Deliberation from the Citizens Initiative Review[END_REF]. It is specially tailored to the need of informing the public on local policy issues. CIR guides a group of panelists to review an issue relevant to the community indepth through a multi-day public review process. Panelists are either randomly or strategically selected from a community. As representatives of a community, panelists are given access to a large amount of data from various sources concerning a given issue. CIR aims to generate an informative briefing of the issue, called Citizens' Statements, to provide the community with insights concerning the issue so that everyone in the community is able to form opinions effectively and efficiently.

CIR can be conducted in a purely face to face environment. However, citizens have their daily work and can only allocate limited time (especially daytime) and effort for CIR. In our experimental studies we choose to blend online and faceto-face activities. There are two face-to-face meetings on the first and last days respectively, and the panel works online during the time in between. The whole process may last about ten days to two weeks depending on the complexity of the issue. The expected outcome of CIR is a set of Citizens' Statements, including 10 findings, 5 pros, and 5 cons. The slots associated with the 20 statements are allocated in advance and can be assigned with customized labels for easier reference. Each category is equipped with a progress bar that shows the current working process of categorization. In the rest of the section, we describe the phases and relevant activities in a CIR process. We will also identify the challenges users may face and show how technical support can contribute.

Preparation

Recruiting Citizen Panel. One important design question is: who should be the part of the CIR panel for crystallizing the community knowledge? We choose to use a small group of citizens as the panel. By bringing in diverse perspectives and skills, we can expect CIR to perform an in-depth analysis in a deliberative manner. Since the outcome of the CIR is to be used by citizens in a community, using peer citizens in CIR ensures a degree of trust by other citizens. We follow the work of a small group deliberative democracy process [START_REF] Gastil | Making Direct Democracy Deliberative through Random Assemblies[END_REF], which selects panelist from the relevant population through stratified random sampling as representatives.

Assemble information package for a pending issue. In an ideal world, input to CIR should be all the information that can be found about a community issue. In reality, we use a team of undergraduate researchers to collect documents from government websites, public media, experts and senior citizen advisors to compile an information package to be distributed to each panelist. We also identify and invite subject matter experts and government officials to review and supplement the package to ensure completeness. The contents usually include neutral descriptions of proposals/ordinances and evidences for/against a policy proposal. Other than published reports, websites, and news articles, the document collection also contains interviews with subject matter experts and their written statements. During the CIR process, panelists are allowed to add more materials through a request.

Issue Briefing. Before moving to the first phase, the panelists will get together and learn about the community issue through a face-to-face meeting.

During this meeting, the CIR panel members are charged with investigating a policy issue that is usually complex and controversial. In the same time, they will pick up an information package with all the details about CIR process, data to be analyzed and the intended outcome. They will have a chance to get to know each other as collaborators. Panelists will also receive a training on how to use the online deliberation system, GeoDeliberator. They can also communicate with experts directly to resolve quick questions.

Phase I: Extract Information Nugget

Nugget Extraction in CIR is aimed at reliably recognizing and collecting all data segments (or nuggets) relevant to the pending issue. Doing a good job in this phase is the prerequisite for subsequent tasks of knowledge crystal formation, refinement, and compaction. During this first phase of CIR, panelists work asynchronously online to gather information nuggets relevant to a policy issue. The document view contains a collection of documents, plus a table of contents for easy navigation. These documents are identical to those included in the information package. When a panelist recognizes a nugget in a document, and he/she can extract nuggets by selecting a piece of text judged as relevant to one or more theme. Once a segment of text is selected, it will be highlighted with yellow color and be prompted to assign this information nugget to a theme. All the themes are listed on the top and a detailed explanation will be provided when placing cursor over each theme icon.

All the extracted nuggets are collected into the nugget list. Nugget List view is actively linked to the Document View, allowing panelists to trace back to where a nugget originates in the document. Capturing the relationship between nugget and its origin in documents effectively makes it possible to replay and review the analytical process later on [START_REF] North | Analytic provenance: Process+interaction+insight[END_REF].

Phase II: Assemble and Improve Claims

The purpose of Claim Assembly in CIR is to transform collected information nuggets into claims, which should be relatively well-written, self-contained, and based on facts and evidence. There are two kinds of claims: findings (objective facts) and opinions (facts with implicit position). The opinions can be further decomposed into two categories: substantiate and refute, depending on the position. A claim is informed by one or more information nuggets. Claims can be further elaborated and improved through adding more information nuggets, removing irrelevant or unimportant contents, rewording, merging several claims or splitting a claim.

A new claim can be created by clicking + button on the top of the claim list. Once a new claim is created, the view automatically switches to the claim workspace where one can write the claim text and cite any nuggets (by clicking the "adopt" button next to a nugget) that contribute to the claim. By adopting nuggets to a claim, a semantic link is established between the selected nuggets and the current claim. In addition to editing the claim directly, panelists are also encouraged to leave comments to others and discuss the claim.

Phase III: Generate Statements

Phase III is to refine the statements by making them more compact, defensible, and understandable. This involves two types of interface operations: categorization and refinement. Categorization is to decide whether a claim is a finding statement, or a substantiate statement or a refute statement. The categorization is performed by dragging and dropping claims from claim list to a category slot.

Phase IV: Communicate Statements

In this phase, the claims are compiled into a set of statements manageable length. These statements must be presented in a way that is easily understood by local citizens. Special care is given to the use of language that state expert and professional knowledge in a form usable by the general public. The final Citizens' statements contain ten statements of Findings that summarize the issue and why it is important to the community. Another ten statements are allocated to the five strongest arguments in favor of the issue and five strongest arguments against the issue.

Facilitative Moderation

CIR requires expert facilitators to be coupled with system support. Facilitators serve on a number of roles. Firstly, they mediate the conflicts among views on issue-relevant information, and manage different understandings, values, and knowledge [START_REF] Moore | The mediation process: Practical strategies for resolving conflict[END_REF]. Second, facilitators coordinate with the panelists and experts.

The facilitator practices its function through a control panel in the online system. From this control panel, the facilitator can monitor panelists' activities, control the process through a sequence of phases, manage schema, and manage documents.

Supporting Collaboration

In addition to entity-eccentric discussion and editing, collaboration among panelists is supported mainly through a chat room. Communications among panelists and subject matter experts are supported through a question panel. Panelists can directly ask a question in the question panel. Some of the questions can be answered immediately by peers. Others may have to be forwarded to subject matter experts. Once answers are received, they will be incorporated into the document collections.

Case Study

In order to test our implementation of CIR and iteratively improve it, we conducted a case study to gather the feedback from the potential panelists.

Procedure

The community issue used in this study is inflationary tax indexing. The proposal is that real estate tax should be increased by at least inflation every year just to keep pace with the cost of providing services to the Borough. We recruited fourteen participants as the citizen panel. Most of them were recruited via mailings that were sent at random based on the addresses provided the borough office. Three of them from specific student organizations were recruited via targeted email. The three students rent in the borough. The rest are homeowners. There are also four subject matter experts involved. Two of them are proponents that support the inflationary tax indexing, and two are opponents against the issue. The study lasted ten days. On the first day, a short introduction and a training session were conducted in the face-to-face meeting. In the first session, the moderator introduced the community issue review in terms of its process, expected outcome and various roles. Then the community issue of inflationary tax indexing was introduced, followed by a question and answer period that allows the panelists to ask questions.

After the meeting, the panelists went home and began to work in the online environment. They were expected to follow CIR phases and collaboratively produce candidate statements for the final-day meeting to discuss. During this period, a facilitator kept monitoring the online activities and moderate the process when necessary. To ensure steady progress, the facilitator sent emails every morning, along with a summary of the progress by far. In the final day meeting, the panelists discussed and finalized the 20 citizen's statements.

We wanted to gather input from the participants as much as possible since this is still a preliminary study, thus an exploratory approach was employed to collect their feedback. During each of the work sessions, researchers conducted observations and asked the panelists to describe what they were doing and thinking aloud, especially the intentions behind behaviors. We only provided guidance and assistance whenever the participants had confusion.

Evaluation and Feedback

Based on the participants' feedback and our observations, we compiled a series of points and summarized them in this section.

Knowledge Crystallization Scheme Without knowing how to decide and choose the knowledge schema, we adopted a set of themes as the knowledge crystallization scheme in this study. Each theme represents one important aspect of the issue and is shown as a phrase with detailed explanations on demand. For example, "Affordability" was used in the study as a theme, which indicates how a tax increase affects the price of owning and renting properties.

All the participants reported that the themes were only useful for the first phase to organize extracted nuggets. It was difficult for them to continue to use the themes as the extracted nuggets were transformed into claims. Instead, some of the participants suggested that several guiding questions would be more helpful for all phases.

Phase Transition We organized CIR as an explicitly phase-based process following the IBKC framework (Figure 2). Phases switch only when all participants feel that they have completed the current phase and ready to move on to the next phase. By enforcing phase-based process, panelists were expected to work synchronously and thus their contributions could be evaluated and utilized collaboratively.

However, Some participants reported that it was difficult for them to divide phase clearly; it caused confusion to them as they had to understand exactly the design of each phase. To address this problem, we enhanced our system by allowing panelists to be able to do all the work in one integrated interface where phases are implicitly enforced. We presented the revised interface to two participants and received positive feedback.

Learnability and Accessibility

The most common issue pointed by the participants was the usability of the online system. The participants were unaware of many available features and sometimes used the system in an incorrect way. The targeted users are ordinary citizens, among which many have insufficient computer skills. Therefore, on the one hand, the system should be designed to be easier to learn and operate. On the other hand, a more sophisticated technical support and training session should be provided.

Collaboration, Coordination, and Communication Our observations show that participants communicated a lot in face-to-face meetings while they worked almost individually in the online environment, though a variety of communication channels were provided. One participant believed it is due to time delays in asynchronous communication while people do expect immediate responses or in-time notification. This was explained by [START_REF] Curtis | Exploring collaborative online learning[END_REF].

One improvement is to provide subscription/notification service: Once a participant makes a contribution to an entity, she is considered to subscribe to the related thread. Whenever there is an update, e.g., another participant leaves a comment, the participant will be notified. The idea of the private and public workspace was also mentioned by some participants, which allows the participants to work in their private workspace and share with others only when necessary.

Flexibility of Organizing and Retrieving Contents Currently all the entities involved in CIR, such as documents, extracted nuggets, assembled claims, and candidate statements, are structured in a linear fashion. Although some filters are provided that allow panelists to select, for example, the information nuggets tagged by a particular theme, it is still limiting the way of organizing them. Some participants would like to see the system provide more means of structuring the entities, for example, to cluster documents based on contents in advance.

Another feature the participants hope to have is a search function. We intentionally removed the search function as a way to enforce people to go through all the documents thoroughly rather than doing a keyword search when doing sense-making. However, the participants do have a need to revisit what they have read, and search function can support that. Bookmarking is also a solution to that.

Discussion and Conclusion

In this paper, we identified the challenges caused by information overload and knowledge deficit that prevent ordinary citizens from participating public life effectively. Drawing from observations of how local government decision-making works and theories of information, we proposed CIR, an instance of knowledge crystallization, as a solution to the above problem. We implemented CIR with the help online technologies and presented it to a group of citizens strategically selected. Based on lessons learned from their feedback, we developed a better understanding of CIR process and the need for improving this process.

Community Issue Review should be considered as a general framework that can be implemented in a variety of the contexts and processes. The current implementation of CIR (as described in this paper) is far from being perfect. On the technical side, many system-support features are to be further refined and optimized. We are incorporating the lessons learned from experimental observations and the feedback collected from the case study to enhance the support for collaboration and communication among panelists. On the social side, our design of the CIR process should consider the vulnerability of the process and outcomes to power influences. In our observation of CIR practices in State College Borough, the Borough's council members are clearly the power holders since they make the final decision. The council can influence the CIR process by framing the policy issues from the government point of view, imposing pressure on which issue to be reviewed and and limiting policy options to be considered. To balance such potentials of power influence, we have explicitly included a few mechanisms in CIR to empower citizens. First, the citizen panel of CIR is the only body to execute the creation of the citizens' statement, and other players (subject matter experts, municipal staff, researchers) are all playing a supporting role during the process. Second, we run a special session of "issue-framing" in Day-1 of CIR to allow the panel to generate its own way of framing the policy issue and propose alternative solutions. Second, we asked the panel to deliberate on their value propositions and challenge those from the experts and the government. For the above reasons, we argue that citizens' participation in the CIR redistributes more power from the council to the citizens, compared to the existing citizen consultation methods that do not garner much participation.

Fig. 1 .

 1 Fig. 1. The framework for issue-based knowledge crystallization

Fig. 2 .

 2 Fig. 2. The overview of community issue review as a knowledge crystallization implementation

Fig. 3 .

 3 Fig. 3. Face-to-face meeting on Day 1

Acknowledgments

We acknowledge the support of this research by NSF under award IIS-1211059. The work of the first author is also partially supported by a grant from the Chinese Natural Science Foundation under award 71373108.