Dalibor Stanimirovic
email: dalibor.stanimirovic@fu.uni-lj.si

Tina Jukic
email: tina.jukic@fu.uni-lj.si

Janja Nograsek
email: janja.nograsek@fu.uni-lj.si

Mirko Vintar
email: mirko.vintar@fu.uni-lj.si

Analysis of the Methodologies for Evaluation of E-Government Policies

Keywords: e-government policy, evaluation methodology, evaluation and development level, comparative analysis

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Despite extensive research in the recent years [START_REF] Silcock | What is E-government?[END_REF][START_REF]OECD. Organization for Economic Co-operation and Development: E-government: Analysis Framework and Methodology[END_REF][START_REF] Yildiz | Examining the Motivations for E-Government from an Institutional Theory Perspective: Evidence from Turkey[END_REF] and considerable investments in the field; EU countries are investing approximately 2.2% of GDP in public sector ICT [START_REF]Capgemini: Smarter, Faster, Better eGovernment, 8th Benchmark Measurement[END_REF][START_REF] Capgemini | Sogeti: Digitizing Public Services in Europe: Putting ambition into action[END_REF][START_REF]eGovernment Benchmark Pilot on Open Government and Transparency[END_REF], the phenomenon of e-government remains ambiguous and still lacks a unified definition. OECD studies indicate that further e-government development is one of the most important factors of public sector rationalization, as well as faster countries' development [START_REF]Organization for Economic Co-operation and Development: Focus on Citizens: Public Engagement for Better Policy and Services[END_REF][START_REF]OECD. Organization for Economic Co-operation and Development: Rethinking e-Government Services: User-centred approaches[END_REF][START_REF]Organization for Economic Co-operation and Development: E-Government Indicators: Proposal for a new Framework and Methodology[END_REF]. E-government development so far has been marked by a large gap between supply and demand of public e-services in most countries, which can be prevailingly attributed to "politically driven" development rather than evidence-based evaluation and selection of e-government policies [START_REF] Kunstelj | Analysing the demand side of e-government: what can we learn from Slovenian users?[END_REF][START_REF] Kim | A Cross-national Analysis of Global E-government[END_REF][START_REF] Vintar | How much can we trust different e-government surveys? The case of Slovenia[END_REF]. Some countries (e.g. Estonia) [START_REF] Dutta | Estonia: A Sustainable Success in Networked Readiness? The Global Information[END_REF][START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF][START_REF]World Economic Forum: The Global Information Technology Report 2010-2011[END_REF] have been accomplishing much better results in evaluation and implementation of e-government policies compared to several other countries with much higher investments. Past experience in the field and public finance trends evidently require the development of methodologies 1 for evaluation of e-government policies which could enable e-government decision-makers to conduct more qualified and quantified preparation, execution and evaluation of e-government policiesbe it before or after their implementation (ex-ante or ex-post).

Despite the increasing number of evaluation methodologies, the numerous aspects of their study and comparison have largely been disregarded. They are basically too diverse and lack a unified and clear theoretical framework [START_REF] Vintar | How much can we trust different e-government surveys? The case of Slovenia[END_REF], [START_REF] Bannister | The curse of the benchmark: an assessment of the validity and value of egovernment comparisons[END_REF][START_REF] Janssen | If you measure it, they will score: An assessment of international eGovernment benchmarking[END_REF], which would allow a comparison of differences between them. The latter arise from various reasons: different (EU, UN, Brown University, EIU etc.) and heterogeneous promoters (international, national, consulting, research institutions etc.) [START_REF] Vintar | How much can we trust different e-government surveys? The case of Slovenia[END_REF], diverse environments [START_REF]Australian Government Information Office: Demand and Value Assessment Methodology -DAM & VAM[END_REF][START_REF]eGEP -eGovernment Economics Project: Expenditure Study Final Version[END_REF], various rationales and contextual background as well as the number and selection of indicators [START_REF]Australian Government Information Office: Demand and Value Assessment Methodology -DAM & VAM[END_REF][START_REF]eGEP -eGovernment Economics Project: Expenditure Study Final Version[END_REF]. Significant differences between evaluation methodologies are reflected within their main evaluation focus and evolving stage as well.

The paper is trying to overcome these limitations and establish the rudiments for theoretical framework which could facilitate a comparative analysis of existing methodologies in the field. Deriving from the aforementioned research objectives the paper is focusing primarily on the following interrelated research questions:

1. Overview of the existing methodologies for evaluation of e-government policies. 2. Identification and characterization of the key evaluation levels within egovernment policies. 3. Analysis of existing methodologies for evaluation of e-government policies according to identified evaluation levels and development levels.

The research is based on the study of abundant literature, relatively scarce research reports available from the field, and an in-depth analysis of the methodologies which have been already implemented in practice. Paper essentially represents a review and comparative analysis of the methodologies for evaluation of e-government policies.

The research was conducted within the research project aiming to determine which methodologies could be applied for evaluation of e-government policies in Slovenia.

Methodologies for evaluation of e-government policiesstate of the art

According to the subject of evaluation, methodologies could be classified in typical groups presented below.

Front-office maturity and readiness

The best-known benchmark measurements in EU have been conducted by Capgemini [START_REF]Capgemini: Smarter, Faster, Better eGovernment, 8th Benchmark Measurement[END_REF][START_REF] Capgemini | Sogeti: Digitizing Public Services in Europe: Putting ambition into action[END_REF][START_REF]eGovernment Benchmark Pilot on Open Government and Transparency[END_REF], while the most renowned benchmarkings on the global scale have been carried

out by the UN [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF]United Nations: United Nations E-Government Survey, From E-Government to Connected Governance[END_REF], Accenture [START_REF]Accenture: From e-Government to e-Governance: Using new technologies to strengthen relationships with citizens[END_REF] and Brown University [START_REF] West | Improving Technology Utilization in Electronic Government around the World[END_REF]. While focusing primarily on web site analysis (front office), all these methodologies used completely different indicators, hardly ensuring comprehensive evaluation of e-government policies on the national level [START_REF] Vintar | How much can we trust different e-government surveys? The case of Slovenia[END_REF], [START_REF] Bannister | The curse of the benchmark: an assessment of the validity and value of egovernment comparisons[END_REF], [START_REF] Dadayan | Measuring Return on Government IT Investments[END_REF]. While other important benchmark measurements converging on e-readiness and information society in general are: The Global Information Technology Report [START_REF] Wef | World Economic Forum: The Global Information Technology Report 2009-2010, ICT for Sustainability[END_REF], Digital economy rankings [START_REF]Economist Intelligence Unit: Digital economy rankings[END_REF] and United Nations e-Government Survey [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF]United Nations: United Nations E-Government Survey, From E-Government to Connected Governance[END_REF].

Effects and impacts of e-government policies

Ex-ante and ex-post evaluations of e-government policies are subject of numerous methodologies, among which we could highlight: MAREVA [START_REF] Adae | Fourth High Level Seminar on Measuring and Evaluating E-Government[END_REF], eGEP [START_REF]eGEP -eGovernment Economics Project: Expenditure Study Final Version[END_REF], WiBe 4.0 [START_REF] Rothig | WiBe 4.0 Methodology. Economic Efficiency Assessments in Particular with Regard to the Use of Information & Communication Technology[END_REF] and Australian AGIMO [START_REF]Australian Government Information Office: Demand and Value Assessment Methodology -DAM & VAM[END_REF]. MAREVA and WiBe 4.0 are dealing with exante and ex-post evaluations of e-government policies on the basis of parameters such as profitability, risks, benefits to external users and civil servants, services and project necessity. eGEP and AGIMO similarly analyze costs, related risks, provision and maintenance of e-services, as well as evaluate their performance and impacts.

Implementation of e-government policies requires revision of the sourcing issues [START_REF] Scholl | Electronic Government: Information Management Capacity, Organizational Capabilities, and the Sourcing Mix[END_REF], careful scrutiny of the complex outsourcing implications [START_REF] Greaver | Strategic Outsourcing: A Structured Approach to Outsourcing Decisions and Initiatives[END_REF], [START_REF] Kern | The winner's curse in IT outsourcing: Strategies for avoiding relational trauma[END_REF] and provision of indicators for objective evaluation of outsourcing process [START_REF] Lacity | Information Systems and Outsourcing[END_REF].

Given the complex effects of e-government policies on public sector organizations, research is engaged in analysis of joined-up e-government model [START_REF] Klievink | Realizing joined-up government -Dynamic capabilities and stage models for transformation[END_REF], organizational changes in the direction of network government [START_REF] Van Veenstra | Towards an Understanding of E-Government Induced Change -Drawing on Organization and Structuration Theories[END_REF], management and external factors which affect e-government development [START_REF] Schedler | Managing the e-government organization[END_REF], business process change, information management capacity, organizational capabilities and culture [START_REF] Scholl | Electronic Government: Information Management Capacity, Organizational Capabilities, and the Sourcing Mix[END_REF], [START_REF] Scholl | E-government: A Special Case of ICT-enabled Business Process Change[END_REF], [START_REF] Leitner | Organisational Changes, Skills and the Role of Leadership Required by eGovernment[END_REF].

National-level development

National-level development is partially discussed in United Nations e-Government Survey [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF]United Nations: United Nations E-Government Survey, From E-Government to Connected Governance[END_REF] through indicators such as e-participation, e-inclusion and econsultation. Martin and Byrne [START_REF] Martin | Implementing e-Government: widening the lens[END_REF] focused on critical factors of information society development providing a set of indicators for evaluation of e-government such as accessibility, digital divide, human rights, social inclusion, economic sustainability and life-long learning. Economic activities on national level could significantly affect e-government development in individual country. Scarce research [START_REF] Kim | A Cross-national Analysis of Global E-government[END_REF], [START_REF] Bavec | What Matters in the Development of the E-Government in the EU?[END_REF], [START_REF] Singh | Country-Level Determinants of E-Government Maturity[END_REF] is specifically emphasizing correlation between national economic indicators (GDP per capita, competitiveness, use of ICT in the private sector, innovation index and internet access) and e-government development on the national level and on the EU level [START_REF] Bavec | What Matters in the Development of the E-Government in the EU?[END_REF].

Evaluation of e-government policiesissues and barriers

Evaluation of e-government policies is generally difficult [START_REF] Capgemini | Sogeti: Digitizing Public Services in Europe: Putting ambition into action[END_REF], [START_REF]Organization for Economic Co-operation and Development: E-Government Indicators: Proposal for a new Framework and Methodology[END_REF], [START_REF] Bannister | The curse of the benchmark: an assessment of the validity and value of egovernment comparisons[END_REF], [START_REF] Dadayan | Measuring Return on Government IT Investments[END_REF], given the frequent lack of clarity of objectives owing to the different and often competing views held by different stakeholders. Effective evaluation requires good metrics, regular monitoring and reporting, disciplined use of robust evaluation frameworks and longterm evaluation practice largely depending on overall evaluation culture [START_REF]Organization for Economic Co-operation and Development: E-government and Administrative Simplification: Good Governance for Development in Arab Countries[END_REF], [START_REF] Hamilton | Building a Methodology for Measuring the Value of E-Services[END_REF].

Key evaluation levels within e-government policies

Overview of evaluation methodologies revealed they are focused predominantly on service level, while there are only a few methodologies, which could be actually applied for evaluation of e-government policies and decision-making at higher levels.

Methodologies are generally partial and mostly focused on evaluating changes that occur in the "front-office" operation, while "back-office" changes caused by ICT have largely failed to gain significant attention. Existing research facilitates extraction and synthesis of the key evaluation levels (Fig. 1) which are described below. Pyramid structure of the model indicates the direction of policy-making process and assumes hierarchical relationships between individual levels.

Infrastructural level

Infrastructural level primarily refers to maturity or environmental readiness for egovernment and e-commerce. Research in this area is focused either on the internal or external aspect of e-government. Internal aspect research is primarily engaged in [START_REF] Kunstelj | Metrika na področju spremljanja razvoja e-uprave[END_REF][START_REF] Kunstelj | Evaluating the progress of e-government development: A critical analysis[END_REF]: development strategies, policies and action plans, legal frameworks, the existence and use of appropriate information infrastructure, training of human resources, knowledge management, financial issues, motives and obstacles for the development of e-government. Research on the external aspect of the environment maturity is particularly concerned with [START_REF] Kunstelj | Metrika na področju spremljanja razvoja e-uprave[END_REF][START_REF] Kunstelj | Evaluating the progress of e-government development: A critical analysis[END_REF]: ownership, user interest and degree of e-government service usage and issues related to the general development of e-government.

Project level

Research at project level is primarily engaged in: 1) ex-ante evaluations of projects aiming to establish priorities for further development, 2) ex-post evaluations of projects aiming to evaluate the effects of projects and 3) decisions on the insourcing and/or outsourcing of projects. Regarding the first two points, a review shows that methodologies of this type often underestimate public benefits (public value) and hidden costs, such as costs of organizational change. Research implies significant advances in outsourcing of ICT projects. Studies [START_REF] Lacity | Information Systems and Outsourcing[END_REF], [START_REF] Taylor | Critical risks in outsourced IT projects: the intractable and the unforeseen[END_REF][START_REF] Jensen | Public Sector Outsourcing Contracts -The Impact of Uncertainty, Incentives and Transaction Costs on Contractual Relationships[END_REF] often reveal the hidden costs, vendor-lock in and loss of competencies as the most problematic issues, rarely dealing with the other potential negative consequences of outsourcing [START_REF] Kern | The winner's curse in IT outsourcing: Strategies for avoiding relational trauma[END_REF], [START_REF] Linder | Outsourcing for Radical Change: A Bold Approach to Enterprise Transformation[END_REF][START_REF] Power | The Outsourcing Handbook: How to Implement a Successful Outsourcing Process[END_REF].

Organizational level

Studies dealing with changes in the organizational structure are focusing on the reduction of hierarchical levels, decentralization, standardization, coordination and transformation of the existing organizational relations [START_REF] Klievink | Realizing joined-up government -Dynamic capabilities and stage models for transformation[END_REF], [START_REF] O'donnell | Transformational aspects of E-government in Ireland: Issues to be addressed[END_REF][START_REF] Griffin | Joined-up E-government: an exploratory study of UK local government progress[END_REF]. Research dealing with business process reengineering is analyzing horizontal integration of functions and services, vertical integration of organizations, information exchange, changes in time and place of operation [START_REF] Scholl | E-government: A Special Case of ICT-enabled Business Process Change[END_REF], [START_REF] Layne | Developing fully functional E-government: A four stage model[END_REF][START_REF] Elnaghi | A Leadership Model for e-Government Transformation[END_REF]. Research exploring the changes in organizational culture is primarily dealing with: changes in the organizational philosophy and strengthening the sense of affiliation and confidence [START_REF] Schedler | Managing the e-government organization[END_REF], [START_REF] Ho | Reinventing Local Governments and the E-Government Initiative[END_REF]. Changes in human resources refer to the new skills, knowledge and specific managerial abilities [START_REF] Leitner | Organisational Changes, Skills and the Role of Leadership Required by eGovernment[END_REF].

Political-sociological level

Proliferation of ICT and development of e-government have changed the social structure and political-sociological paradigm of the social community [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF] Norris | Digital Divide: Civic Engagement, Information Poverty and the Internet[END_REF]. Complex political-sociological effects of ICT and e-government have a significant impact on the social environment; they are affecting old and creating new forms of work and changing perception of the world and social relations [START_REF] Slevin | The Internet and Society[END_REF][START_REF] Anttiroiko | Toward the European Information Society[END_REF][START_REF] Thurlow | Computer mediated communication: Social interaction and the internet[END_REF][START_REF] Shim | E-Government and Anti-Corruption: Empirical Analysis of International Data[END_REF]. Accordingly, existing methodologies are converging on the following aspects of e-government evaluation: accessibility [START_REF]Organization for Economic Co-operation and Development: Focus on Citizens: Public Engagement for Better Policy and Services[END_REF], [START_REF]United Nations: United Nations E-Government Survey, From E-Government to Connected Governance[END_REF], [START_REF] Ho | Reinventing Local Governments and the E-Government Initiative[END_REF], citizens' trust and confidence [START_REF]Accenture: From e-Government to e-Governance: Using new technologies to strengthen relationships with citizens[END_REF], [START_REF] Tolbert | The Effects of E-Government on Trust and Confidence in Government[END_REF][START_REF] Parent | Building Citizen Trust through e-Government[END_REF], digital divide [START_REF]Organization for Economic Co-operation and Development: Focus on Citizens: Public Engagement for Better Policy and Services[END_REF], [START_REF]Organization for Economic Co-operation and Development: E-government and Administrative Simplification: Good Governance for Development in Arab Countries[END_REF], [START_REF] Wef | World Economic Forum: The Global Information Technology Report 2009-2010, ICT for Sustainability[END_REF], [START_REF] Norris | Digital Divide: Civic Engagement, Information Poverty and the Internet[END_REF], social stratification and cohesion, human rights and democratic participation [START_REF]OECD. Organization for Economic Co-operation and Development: Rethinking e-Government Services: User-centred approaches[END_REF], [START_REF]World Economic Forum: The Global Information Technology Report 2010-2011[END_REF], [START_REF] Martin | Implementing e-Government: widening the lens[END_REF], openness, transparency and corruption [START_REF]eGovernment Benchmark Pilot on Open Government and Transparency[END_REF], [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF]United Nations: United Nations E-Government Survey, From E-Government to Connected Governance[END_REF].

National level

Research reveals that economic activities on the national level significantly affect egovernment development, exposing GDP per capita as the most influential economic indicator [START_REF] Bavec | What Matters in the Development of the E-Government in the EU?[END_REF][START_REF] Singh | Country-Level Determinants of E-Government Maturity[END_REF]. Sing et al. [START_REF] Singh | Country-Level Determinants of E-Government Maturity[END_REF] assume that GDP plays a crucial role in the development of e-government via three influential factors (technological infrastructure, human capital and management index). Other prospective indicators occasionally overlapping with political-sociological indicators are [START_REF] Bolgherini | The Technology Trap and the Role of Political and Cultural Variables: A Critical Analysis of the E-Government Policies[END_REF][START_REF] Lor | Is a knowledge society possible without freedom of access to information[END_REF]: competitiveness, use of ICT in the private sector, innovation index [START_REF] Bavec | What Matters in the Development of the E-Government in the EU?[END_REF], education and urbanization [START_REF] Kim | A Cross-national Analysis of Global E-government[END_REF].

Analysis of existing methodologies for evaluation of egovernment policies

The review of existing methodologies was conducted in the second half of 2011. During that time the research team scanned journals and conference proceedings, books, reports of international organizations and other institutions, policy papers, development strategies and other related documents containing e-government related research. Focusing particularly on measurement, assessment and evaluation of egovernment policies and their effects we identified more than 50 relevant references.

The frequency of references is becoming much higher in the second half of the last decade, proving the field is evolving rapidly and attracting more interest. Taking into account development level of evaluation methodologies, we have identified basically three types of references: 1) purely theoretical papers aiming to develop some kind of conceptual framework for evaluation of e-government policies, 2) research efforts developed up to the degree of pilot application and 3) methodologies developed in the practice for the practice (practical application). Further on, evaluation methodologies have been classified according to the identified evaluation levels, using the serial number of methodologies from the list of references (Fig. 2).

Fig. 2. Classification of methodologies according to the evaluation levels and development levels

Analysing the diverse variety of evaluation methodologies identified in this area, certain general characteristics were identified and summarised below:

 Majority of the identified methodologies for evaluation of e-government policies are presented in scholarly papers and books.  Small number of methodologies is appearing in the form of specific handbooks, some of which include a tool for evaluation of e-government policies, for example WiBe 4.0 or VAST (software packages, Excel spreadsheets etc.).  Certain methodologies are rather abstract containing speculatively selected indicators often encompassing non evidence-based theoretical platforms, while their utilization does not facilitate the acquirement of quantifiable evaluation results.  Methodologies are to a large extent narrowly focused assessing predominantly one of the evaluation levels presented in the five-level model.  Mature methodologies are consisted of a large number of indicators, normally aligned for evaluation of e-government policies in the originating countries.  Methodologies generally do not provide a comprehensive evaluation of complex egovernment policies impacts and their potential long-term public benefits.  Various groups of indicators evaluating the itemized evaluation levels are appearing in dozens of different methodologies, including a large number of overlapping. Definitions of indicators vary widely, while evaluations are based on completely different methodological platforms, their results are very difficult to compare.

Particular features of methodologies evaluating individual levels are outlined hereinafter:

 Methodologies evaluating infrastructural level are mainly focused on ICT infrastructure and interoperability, human resources, legal framework and standards, policies and strategies, horizontal building blocks and other, often technical aspects. While generally focusing on only some of the itemized aspects (evaluation of particular technical aspects is very complex, e.g. interoperability) and allowing only a narrow insight into the context of e-government, they fail to provide a credible picture of the overall state of e-government.  Methodologies evaluating project level are generally very exhaustive in terms of the large number of indicators; however they rarely address the concept of public benefits comprehensively, while the vast amount of data needed for applied indicators considerably complicates their utilization and transfer to other environments.  Methodologies evaluating organizational level often address various organizational dimensions at least indirectly; failing to provide a full insight into the matter, consensus and clear rationalization of e-government induced organizational changes.  Methodologies evaluating political-sociological level are mostly partial, focusing usually on policy aspect, accessibility and digital divide. Other methodologies in the area addressing particularly social aspect contain general and intangible indicators, since the concepts such as trust, confidence, social cohesion, social relations etc. are difficult to define unequivocally, while their understanding differs according to the cultural and institutional environment.  Methodologies evaluating national level mainly explore the national-economic categories and their relations with the various aspects of development and implementation of e-government. They hardly formulate a clear research framework, while interdependence, direction and way of influence between economic indicators and e-government are not sufficiently explored and adequately elaborated.

After general systemization of identified methodologies (Fig. 2), we focused more closely on methodologies which have already achieved practical implementation.

Based on these criteria we analysed 13 methodologies [START_REF]Leveraging e-government at a time of financial and economic crisis[END_REF], [START_REF]Australian Government Information Office: Demand and Value Assessment Methodology -DAM & VAM[END_REF], [START_REF]eGEP -eGovernment Economics Project: Expenditure Study Final Version[END_REF], [START_REF] West | Improving Technology Utilization in Electronic Government around the World[END_REF], [START_REF] Wef | World Economic Forum: The Global Information Technology Report 2009-2010, ICT for Sustainability[END_REF], [START_REF]Economist Intelligence Unit: Digital economy rankings[END_REF], [START_REF] Adae | Fourth High Level Seminar on Measuring and Evaluating E-Government[END_REF], [START_REF] Rothig | WiBe 4.0 Methodology. Economic Efficiency Assessments in Particular with Regard to the Use of Information & Communication Technology[END_REF], [START_REF]OECD: Guide to measuring the information society. Directorate for science, technology and industry, Working party on indicators for the information society[END_REF], [START_REF] Rama Rao | E-Governance Assessment Frameworks (EAF Version 2.0)[END_REF], [START_REF] Baudu | Value Assessment Tool -VAST[END_REF], [START_REF]eGovernment Benchmarking in 2011[END_REF], [START_REF] Holzer | Digital Governance in Municipalities Worldwide, A Longitudinal Assessment of Municipal Websites Throughout the World[END_REF] which have been enumerated in category "Practical application" (Fig. 2). Analysis revealed substantial limitations and deficiencies. Although they have achieved a high level of maturity, and are used for evaluation of e-government policies in practice, they fail to address the evaluation of egovernment policies in an all-encompassing manner. Most of the outlined methodologies are focused on only one level within the presented five-level model, preventing the comprehensive and quality evaluation of e-government policies. Development of a comprehensive and practically applicable methodology for evaluation of e-government policies is obviously a difficult task. This is demonstrated in Fig. 2, confirming that methodologies which have tried to cover several evaluation levels are developed only up to conceptual framework or maximum pilot application. The latter shows that covering larger number of evaluation levels usually means a lower development level and consequently reduces the potential of methodologies for their practical application. This is not unexpected, since the focus on several evaluation levels means more complex methodology structure and a larger number of indicators, which exacerbates the transparency and complicates the use of methodology.

Research results indicate that achievement of the highest development level and practical application of methodologies for evaluation of e-government policies is largely dependent on the number of evaluation levels the methodology is focused on, and vice versa, meaning that the comprehensiveness of evaluation methodologies is to a large extent conversely related to their development level.

Conclusion and future work

Growing number of evaluation methodologies and their substantial diversity regarding the evaluation focus and level of maturity significantly complicate the establishment of a theoretical framework that would allow a wide-ranging comparison and analysis of the differences between methodologies. Numerous difficulties were encountered trying to delineate the evaluation levels covered by particular methodology, since the contained indicators are not clearly defined, enabling their speculative use on different evaluation levels. Various dilemmas emerged in determining which evaluation methodology achieved higher development level, as well. Although, the development level of methodologies was defined primarily on the criterion of their use in practice, objective definition of development level raises some very important questions of principle. These issues should be properly resolved in further research and succeeding experiments trying to establish a balanced theoretical framework for comparative analysis of evaluation methodologies. Despite aforementioned limitations, conducted analysis provides a valuable insight into the current e-government evaluation practice and facilitates exposure of inade-quately evaluated areas in the domain of e-government policies. The analysis results represent an advance in research of evaluation metrics and may eventually provide a solid platform for establishment of comprehensive methodology for evaluation of egovernment policies and consequently initiate more user oriented, cost effective and performance-based development of e-government. Evidently, the problems in the development of e-government are strongly interrelated with the low quality and underdeveloped methodologies for evaluation of e-government policies and their effects. Extensive research and existing methodologies reveal that the past development of egovernment, and particularly e-services was based primarily on political preferences and only exceptionally on professionally verifiable and measurable impacts of these services. Addressed shortcomings will have to be resolved, in order to ensure quality evaluation and implementation of e-government policies and ultimately accelerate the development of appropriate e-services with added value for all stakeholders.

Fig. 1 .

 1 Fig. 1. Five-level model for evaluation of e-government policies

The collective term "methodologies" will be used hereinafter, denoting approaches, indicator models, measurement frameworks and similar undertakings for evaluation of e-government policies.


Anneke Zuiderwijk

Marijn Janssen

Ronald Meijer
email: r.f.meijer@minvenj.nl

Sunil Choenni
email: r.choenni@minvenj.nl

Yannis Charalabidis
email: yannisx@aegean.gr

Keith Jeffery
email: keith.jeffery@stfc.ac.uk

Issues and guiding principles for opening governmental judicial research data

Keywords: open data, guiding opening data, institutional theory, opening governmental data, judicial research data

The opening of data is considered to provide many benefits. However, opening up data by public bodies is a complex and ill-understood activity. Although many public bodies might be willing to open up their data, they lack any systematic guidance. In this paper, guidance is provided by investigating the publishing processes at the Dutch Research and Documentation Centre (WODC), which owns governmental judicial research data. We developed guidance by providing 1) a list of issues that play a role in deciding whether to open data, 2) an alternative to completely publishing data (i.e. restricted access) and 3) solutions for overcoming some of the issues. The latter include dealing with privacy-sensitive data, deletion policies, publishing after embargo periods instead of not publishing at all, adding related documents and adding information about the quality and completeness of datasets. The institutional context should be taken into account when using the guidance, as opening data requires considerable changes of organizations.

Introduction

To quote from the Obama Administration, establishing openness in governmental organizations is considered to increase transparency, public participation and collaboration and therefore to "strengthen our democracy and promote efficiency and effectiveness in government" [1, p. 1]. Over the last years, various studies have argued that opening up data by governments may provide considerable advantages [2][START_REF] Zhang | Exploring stakeholders' expectations of the benefits and barriers of e-government knowledge sharing[END_REF][START_REF] Charalabidis | An architectural framework for open governmental data for researchers and citizens[END_REF][START_REF] Blakemore | Access to Public-Sector Information in Europe: Policy, rights and obligations[END_REF]. The European Commission [6] states that "Public Sector Information is an important primary material for digital content products and services" [p. 1].

According to Geiger and Von Lucke [START_REF] Geiger | Open Government Data. Free accessible data of the public sector[END_REF], open governmental data can be defined as "all stored data of the public sector which could be made accessible by government in the public interest without any restrictions on usage and distribution" [p. 185]. We adopt this definition because it does not necessarily include the publication of all stored governmental data and it may exclude the publication of public sector data which must remain confidential or are privacy-sensitive.

Opening up data by data producers is a complex and ill-understood activity, because many barriers counteract these processes [START_REF] Boulton | Science as a public enterprise: the case for open data[END_REF]. An important barriers is the threat of privacy violation by opening data and of being legally liable when opened data are misused [START_REF] Kalidien | Crime statistics online: potentials and challenges[END_REF]. Although many governmental organizations might be willing to open up their data, they lack guiding principles derived from practical case studies that help them in doing this [START_REF] Kalampokis | Open government data: A stage model[END_REF]. Some helpful guidelines for opening up governmental data were published in the past [START_REF] Kalampokis | Open government data: A stage model[END_REF][START_REF] Pollard | Opening up government data: making the case[END_REF], but none of these guidelines were derived from and tested in practice.

The process of opening up public sector data demands considerable changes in the public sector, such as changes in the funding and reward systems of organizations. However, it is usually not possible to explain how those types of e-Government initiatives evolve over a certain period of time by the current e-Government linear progression models [START_REF] Wahid | Explaining History of eGovernment Implementation in Developing Countries: An Analytical Framework[END_REF] and the development of composite e-Government services is usually ad-hoc [START_REF] Lampathaki | E-government services composition using multi-faceted metadata classification structures[END_REF]. Avgerou and Wahid propose to use institutional theory to study the implementation of information systems (IS) within organizations [START_REF] Avgerou | IT and organizational change: an institutionalist perspective[END_REF] and to explain how collective awareness or isomorphic change occurs [START_REF] Wahid | Explaining History of eGovernment Implementation in Developing Countries: An Analytical Framework[END_REF][START_REF] Hassan | Institutional Theory and E-Government Research[END_REF]. "Institutional theory postulates that organizations are driven to incorporate the prevailing rules, values, practices and logics in the institutional environment in order to increase their legitimacy and survival prospects" [16, p. 103, 17]. The latter may also be applied to governmental organizations that want to open up their data. In line with the foregoing, Scott [START_REF] Scott | Institutional Theory: Contributing to a Theoretical Research Program[END_REF] states that institutional theory "considers the processes by which structures, including schemas, rules, norms, and routines, become established as authoritative guidelines for social behavior" [p. 2]. From this perspective, Scott argues that "the boundaries of organizational fields are often vague or weak, allowing alternative logics to penetrate and support divergent models of behavior" [18, p. 11]. "Suppressed groups and interests may mobilize and successfully promote new models of structure and repertories of acting." [18, p. 11-12]. In line with this, Avgerou points out that "IS innovation is to a large extent sustainable by its own institutional forces, irrespective of its contribution to the processes of organizational change" [14, p. 1]. Moreover, from the perspective of institutional theory Orlikowski and Barley argue that IS-research should take into account the institutional context where IS are developed and implemented [START_REF] Orlikowski | Technology and institutions: what can research on information technology and research on organizations learn from each other?[END_REF].Taking an institutional lens is considered to be useful for this research, as it shows that the current institutional context should be taken into account when focusing on organizational changes [START_REF] Orlikowski | Technology and institutions: what can research on information technology and research on organizations learn from each other?[END_REF]. The aim of this paper is to develop guidance for opening up governmental data. We focus in particular on judicial research data and we use an institutional lens to understand the issues at hand [START_REF] Wahid | Explaining History of eGovernment Implementation in Developing Countries: An Analytical Framework[END_REF][START_REF] Avgerou | IT and organizational change: an institutionalist perspective[END_REF].

The publishing process of datasets was investigated at the Research and Documentation Centre (Wetenschappelijk Onderzoek-en Documentatiecentrum; WODC) in the Netherlands. The WODC is a criminal justice knowledge center that is part of the Dutch Ministry of Security and Justice. In this organization data are mainly gathered to advise about and to define the current and future research agenda of the Dutch Ministry of Security and Justice, to answer policy-related questions and to indicate the possible implications of research findings for standing policy. For this purpose the WODC systematically collects, stores, enhances and provides criminal justice information produced by external organizations.

To get more insight in issues and guidance possibilities of the publishing process, we first thoroughly analysed 45 datasets of which 3 were opened and 42 were not opened. We inductively tried to identify issues that may be relevant for guidance for opening data. While doing this, the following aspects were taken into account:

a. The context. For instance, the WODC works with confidential judicial research data, so that confidentiality and privacy-sensitivity should be taken into account. b. Current situations, including norms, values and beliefs [START_REF] Avgerou | IT and organizational change: an institutionalist perspective[END_REF][START_REF] Scott | Institutional Theory: Contributing to a Theoretical Research Program[END_REF]. This means that the requirements and guiding principles should be embedded in the current situation, so that, for example, the limits of costs and timeconsumption for an organization and the practices related to privacy sensitive information should be taken into account. c. Dominant rules, values, practices and logics in the institutional environment in order to increase their legitimacy and survival prospects [START_REF] Phang | Investigating Organizational Learning in eGovernment Projects: A Multi-Theoretic Approach[END_REF][START_REF] Meyer | Institutionalized Organizations: Formal Structure as Myth and Ceremony[END_REF]. For instance, in the current practice of the WODC, data are not opened when the WODC wants to reuse the data in the future itself. d. The boundaries of organizational fields are often vague or weak, allowing alternative logics to penetrate and support divergent models of behavior. A new model of acting [START_REF] Scott | Institutional Theory: Contributing to a Theoretical Research Program[END_REF] that could be promoted at the WODC may be that certain types of descriptive, contextual and detailed metadata should be provided when data are published. The previous steps resulted in an account of the issues that should be considered when opening governmental data. The list of issues was validated by carrying out eight interviews with three researchers working at the WODC. The validated account of the issues, the interviews and the aspects of institutional theory resulted in solutions for overcoming some of the issues. Finally, the possible solutions were discussed with two WODC-employees.

Case study background

The WODC aims to facilitate the reuse of research data, as this may provide the organization with benefits, such as the possibility to scrutinize and validate the data and to decrease the workload of the WODC. From 1982 until 2000 the WODC has opened up 21 datasets. In 2001, the Dutch Personal Data Protection Act (Wet Bescherming Persoonsgegevens; WBP) was introduced, which aims to guarantee citizens the right to privacy protection [START_REF] Winter | Wat niet weet, wat niet deert. Een evaluatieonderzoek naar de werking van de Wet bescherming persoonsgegevens in de praktijk[END_REF]. In connection with this new act and an increase in attention for privacy protection in society, the WODC changed its open data policy in 2007. No WODC-datasets have been opened between 2000 and 2008.

Between 2008 and 2012, data that are considered by WODC-researchers to be qualified for public opening have partly been collected and stored in a so-called digital 'research data safe'. Over these 4 years, 45 datasets have been stored in the safe. Almost all the datasets contain crime-related research data that have been used to write reports on. The reports have been published between the years 2002 and 2009 and can be downloaded from www.wodc.nl. In 2008 and 2009, three WODC-datasets that were stored in the research data safe have been opened by means of publication by the Data Archiving and Networked Services (DANS, www.dans.knaw.nl).

In addition, the WODC receives individual requests for data (e.g. via e-mail). The WODC receives about 120 requests per year and most of them are being approved. The requests may be seen as a form of restricted access to data, since certain types of data users are excluded from access to certain data, such as students.

When we look at the WODC from an institutional perspective, we see that in line with the current norms WODC-data are usually not being opened. It was stated that the WODC considers opening up data to be risky when decisions about opening or not opening data are based only on random individual datasets, without taking a broader framework into account. It was also argued that the privacy act does not provide sufficient guiding principles for opening up data. Because of these risk avoiding norms, it becomes automatism not to open up the data and it becomes very difficult to change this culture. Therefore, we decided to develop guidance for opening up governmental data.

Guiding opening data

In section 4.1 guidance for opening data is provided in the form of a list of issues that should be taken into account when opening WODC-data. This list provides input for section 4.3, which presents ways in which these issues can be identified in organizations and by making a distinction between three ways of access.

Guidance by identifying issues for opening data

Institutional theory suggests to take into account current situations, including norms, values and beliefs when developing guiding principles for opening public sector data [START_REF] Orlikowski | Technology and institutions: what can research on information technology and research on organizations learn from each other?[END_REF]. In the current situation, considerable issues play a role in determining whether to open data. In this section, guidance is developed in the form of a list enumerating the most important issues. In the list of issues a distinction was made between two categories: 1) general topics, which concern the dataset as a whole, and 2) dataset related issues, which concern the content of the dataset (see Table 1). In the first place policy confidentiality was considered to be an important issue. For instance, opening certain data may be dangerous to the state . Furthermore, the access to data may be restricted because a deletion policy may apply to the data [START_REF] Buruma | Het recht op vergetelheid. Politiële en justitiële gegevens in de digitale wereld[END_REF] and/or they may be placed under an embargo period. Data can, for example, only be used for a year and should then be deleted in accordance with law or appointments that are made with data providers. When these data have been opened and should be deleted after a year, it is not possible to control whether all people that downloaded these data will also delete it. Other issues with regard to opening up data concern organizational changes. In this context, opening up data requires the creation of a policy for opening data and a focus on opportunities. With regard to individual datasets organizational changes concern changes in funding and reward systems and in time consumption structures [START_REF] Boulton | Science as a public enterprise: the case for open data[END_REF]. In line with institutional theory, governmental organizations have a limited amount of money and time to spend on opening their data. In addition, data may be owned by different organizations so that the interests of all organizations should be taken into account [START_REF] Faerman | Understanding interorganizational cooperation: public-private collaboration in regulating financial market innovation[END_REF].

Two very important aspects of opening governmental data are the right to privacy [START_REF] Zhang | Exploring stakeholders' expectations of the benefits and barriers of e-government knowledge sharing[END_REF][START_REF] Kalidien | Crime statistics online: potentials and challenges[END_REF] and the provision of metadata. In case that pending research is still using or will use certain datasets, it is not in the interest of the organization to open up these datasets. Besides, data may be policy sensitive. This issue is related to the unclearness of how data users are going to use open governmental data [START_REF] Boulton | Science as a public enterprise: the case for open data[END_REF]. Policy sensitive data are not privacy sensitive data, but these data may be easily prone to misuse, misinterpretation and triggering of spurious findings. In addition to this, the consequences of publishing misused, minsterpretated and spurious findings may create negative publicity for the data producer. For example, the WODC monitors crime statistics about business communities. Over the last year, the name of this monitor has become a brand name that is important for the WODC. Therefore, misuse of this brand name may result in negative publicity for the WODC. According to institutional theory, this institutional belief should be taken into account. Another issue that is important for the reputation of the organization and possible damage to the organization concerns the legal responsibility for opening data. Opening certain datasets may be unlawful. Multiple authors [START_REF] Zhang | Exploring stakeholders' expectations of the benefits and barriers of e-government knowledge sharing[END_REF][START_REF] Edmiston | State and local e-government: prospects and challenges[END_REF][START_REF] Dawes | Interagency information sharing: Expected benefits, manageable risks[END_REF] state that organizations encounter substantial uncertainty, which is partly caused by legal principles competing with other values, such as security and system integration.

Other issues that are related to the context of the datasets concern completeness and exhaustiveness, the representation of the data, the validity, the reliability, the clearness and comprehensiveness and the provision of reports about analyses of the data. In line with these content related issues, the overall data quality should be taken into account. Finally, a remaining issue was added, as there may be other issues or combinations of issues that have not been identified by the analysis of datasets, but that would have been identified when other WODC-datasets or datasets of other organizations would have been analyzed.

Issues that were identified frequently from the 45 datasets are privacy-sensitivity and anonymization, a lack of metadata, a lack of clearness and comprehensiveness of column, row, value, variable and other names and the overall data quality. Policysensitivity, deletion policy and unlawfulness were identified infrequently.

Guidance for the identification of issues

Figure 1 shows a systematic process for guiding the identification of all the issues that were enumerated in section 4.1. The questions that are expected to easily rule out opening up a certain dataset are placed on top of the list, whereas questions that require further examination are placed at the bottom of the list. This is done so that data that cannot be opened are quickly identified. Aspects of institutional theory were taken into account in Figure 1 by considering the risk avoiding governmental culture. For instance, due to the fear of wrongful interpretations of the data and the impact of wrongful interpretation on the organization, such as hitting the news with a damaged reputation, guidance is provided to make the chance on wrongful interpretations as small as possible. The latter is done by presenting a list of metadata aspects that should be provided together with the data themselves. Preferably, metadata would be derived directly from the source, although the interpretation of the data could still be difficult even with considerable metadata. Furthermore, the guidance takes into account the general legal framework that is already provided by the WBP, the Dutch Law for Openness of Administration (Wet Openbaarheid van Bestuur; WOB) and the general government conditions for distributing tasks for the performance of services (Algemene Rijksvoorwaarden voor het verstrekken van Opdrachten tot het verrichten van Diensten; ARVODI).

Guidance for dealing with certain issues

In accordance with the possibilities that DANS provides for opening up data and the WBP, WOB and ARVODI, we suggest as guidance the use of three directions when opening WODC-data: open access, restricted access and combined open and restricted access. These directions will be explained in further detail in the following sections.

Open access

When all the questions in Figure 1 are answered and none of them points at the red 'do not open'-line, data can be opened according to the open access method. This means that data can be opened without any access restrictions.

A decrease of the risk on privacy violation while opening data in accord with open access could be arranged by anonymisation of personal details [e.g. see 25]. Also when the dataset is linked to other datasets, it should be avoided that this results in the exposition of the identity of individuals or groups of users [START_REF] Kalidien | Crime statistics online: potentials and challenges[END_REF][START_REF] Braak | Trusted third parties for secure and privacy-preserving data integration and sharing in the public sector[END_REF]. An example of the latter is provided by Kalidien et al. [START_REF] Kalidien | Crime statistics online: potentials and challenges[END_REF], who argue that the publication of data about the mean age of sex offenders per year, gender and city combined with data from other datasets, might expose the full identity of such a person [p. 3].

In addition, considerable attention should be paid to the provision of metadata, as metadata can yield significant benefits including creating order in datasets, improving find ability, accessibility, storing and preservation of data, improving easily analyzing, comparing, reproducing, finding inconsistencies, assessing and ranking the quality of data and avoiding unnecessary duplication of data [START_REF] Zuiderwijk | The necessity of metadata for open linked data and its contribution to policy analyses[END_REF] and hereby encouraging linking open government data [27], so that public value can be created.

Moreover, attention should be paid to the representation (Do the data represent a sample selection of the population?), the validity (Did the research methods and definitions provide the researcher with the type of data that he/she wanted and expected to obtain?), the reliability (Would repeating the research provide the researcher with the same data as this research did?) and the clearness and comprehensiveness (Are the column, row, value, variable and other names easily understandable and do they describe all important aspects of the data?) of datasets. Furthermore, the provision of additional reports, accounts and other documents that are related to the dataset may be helpful in interpreting the data and these should therefore be published with the data and linked to them.

Restricted access

Under the circumstances that data cannot be opened by using the open access method, the data producer may consider opening up the data with restricted access. Data producers may, for example, fear opening postal codes on a street level or on a neighborhood level. Instead of not publishing the postal codes at all, these could be provided in accord with restricted access. We propose to consider restricted access when the questions in Figure 1 point at the orange 'restricted access'-line. When restricted access is provided, data users first have to ask the data producer for permission to obtain access to the dataset. The data producer may decide whether or not to provide access to the dataset, depending on the type of data, the type of user and the purpose of the use for these data, where the openness of data should be seen as a function with various parameters, such as:

-Retrievability of certain data. A dataset may contain personal details or show content restrictions and can therefore only be provided to the data user when he or she signs a contract with the data producer. Furthermore, certain datasets can be fragmented so that only by using a specific IT system it becomes possible to pull together the whole dataset. Similarly, it is possible to make data meaningless unless the end user (or the software the end-users is permitted to use) has a 'key' which transforms the data into something useable. Moreover, datasets may be put on a waiting list providing restricted access on the short term or opening the data after an embargo period. -Accessibility of data for certain users. Access to data is usually arranged through software which makes it possible to restrict openness in line with the purpose of the data user, although no control exists on how data are used once the end-user has brought the data of interest outside of the IT system. However, one can restrict certain types of search, make some parts of the dataset invisible and restrict the use of the data in combination with other data. -Purposes of certain data use. Data providers may only want to provide their data when they know for which purposes the data will be used. The data provider may then have a better idea of the possible outcomes of the data use. Signing a contract is also a possible solution for this parameter. -Users. Typically users may be classified in 'security rings' around a source;

furthest out are all people, then increasingly restricted are continental nationals, country nationals, organizational employees (e.g. government), departmental employees, project or team employees, trusted employees. This may be more sophisticated by characterizing individual users according to previous history and other characteristics. Dependent on the type of user, the data producer may provide him or her with suitable specific advice with regard to this use.

Combining open and restricted access

Finally, a dataset may be opened with both the open access method as well as the restricted access method. Consider for example a dataset that consists of both privacysensitive data as well as non-privacy-sensitive data. The non-privacy-sensitive data may be opened with the open access method, whereas the privacy-sensitive data may be opened with the restricted access method. Whether a data producer wants to provide both open and restricted access depends on the considerations of the data producer.

Discussion and conclusions

From an institutional perspective, we found that on the basis of the prevailing standards, at the WODC-data are usually not opened. In the current situation risks are avoided as much as possible, due to the fear of privacy violations and the impact of inadequate decisions (violating privacy, hitting the news). However, the WODC has shown that it intends to open up more data by making policies and contracts about openness, such as a contract with a Dutch data archiving organization (DANS). Nevertheless, issues with privacy, legal liability, resource intensiveness, data quality and confidentiality are mentioned as considerable barriers for opening WODC-data. Because of these barriers only few datasets are opened. Furthermore, institutional theory shows that organizations tend to have a risk averse culture and therefore views not-opening data as the default option. In addition, the opening up governmental data requires cultural changes in organizations.

An analysis of 45 datasets was performed of WODC-datasets that have and have not been opened in the past. The analysis resulted in a list of issues that should be taken into account when opening up a dataset. The general list of topics includes: confidentiality, deletion policies, embargo placement, cost and time consumption, ownership, privacy-sensitivity and anonymization, lack of metadata, reuse of data by the organization itself, policy-sensitivity and unlawfulness. Besides this general topic list, a list with content-related topics was created, which includes completeness and exhaustiveness, representation, validity, reliability , clearness and comprehensiveness of column, row, value, variable and other names, provision of additional reports, the overall data quality and a rest category. On the basis of these issues, guidance for opening up governmental data was developed.

Because thinking binary in terms of opening and closing is too narrow, our guidance suggests alternative options to avoid rigorously not publishing data that potentially might be opened with three ways of opening up governmental data, namely open access, restricted access or combined open and restricted access. However, it may be difficult to simply follow the guiding principles by giving a yes/no-answer to the questions, because people have different values and they work in different institutional contexts and they may interpret the guidance differently. More insights in these institutional contexts and interpretations is necessary.

This research is a first effort in this field and the guiding principles that are presented in this paper are based on a single case. Further research should focus on the extension of the guiding principles and their applicability in other organizations. Furthermore, the guiding principles should be expanded, extended, specified and extensively tested. Nevertheless, the list of guiding principles can be used as a general means to check which issues should be discussed when one wants to open up governmental data.

Fig. 1 .

 1 Fig. 1. Guidance for identifying issues for opening up governmental judicial research data.

Table 1 .

 1 List of issues that should be taken into account in developing guidance for opening up WODC-data.

	Category	Issue
	General	Policy confidentiality
		Deletion policy
		Embargo placement
		Organizational changes including time-consumption
		and changes in funding and reward systems
		Ownership
		Privacy-sensitivity and anonymization
		Lack of metadata
		(Re)use of data by WODC itself
		Policy-sensitivity
		Unlawfulness
	Dataset	Completeness and exhaustiveness
		Representation
		Validity
		Reliability
		Clearness and comprehensiveness of column, row,
		value, variable and other names
		Provision of additional reports
		Overall data quality
		Other/rest category

Acknowledgements. This paper is related to the ENGAGE FP7 Infrastructure Project (An Infrastructure for Open, Linked Governmental Data Provision Towards Research Communities and Citizens; http://www.engage-project.eu), that started in June 2011. The authors would like to thank their colleagues of the ENGAGE project and the WODC for their input for this paper although the views expressed are the views of the authors and not necessarily of the project or the WODC.


Fathul Wahid
email: fathul.wahid@uia.no

The Current State of Research on eGovernment in Developing Countries: A Literature Review

Keywords: eGovernment, developing countries, literature review

This paper reports a review of literature on eGovernment in the context of developing countries published between 2005 and 2010. The insights emerging from this review may guide researchers in their continued investigation of eGovernment implementation, especially in the context of developing countries. From a review of 108 papers, the study found some substantive changes in the field of eGovernment research. These included increased adoption of interpretive paradigm and increased use of theories in the research. Some future research direction from the methodological perspective were provided: paying more attention to research paradigm and methodology, preserving multiculturalism in eGovernment research, encouraging action research and longitudinal studies, and improving the research quality by grounding it on theories.

Introduction

In a review of mainstream eGovernment literature from 2001 to 2005 (84 papers), Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] unearthed several interesting findings. Here are some of them: no papers adopted interpretive paradigm, only one paper used theory, and more than one-quarter papers had no discernable research method. However, good practices were also identified [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF], include significant recognition of human and other contextual factors that influenced the impact of eGovernment, use of a diverse range of ideas from other research domains, use of a range of different research methods and broad use of primary data. Although the study was not specifically on eGovernment research in the context of developing countries, but rather in a general context, it provided insights that can serve well as a starting point. Has something substantive happened in the meantime? Answer to this question is expected to provide insights to guide researchers in their continued investigation of eGovernment in the context of developing countries.

This paper aims to provide a more recent picture of eGovernment studies in the context of developing countries 1 from 2005 to 2010. This paper focuses on state of the eGovernment research. 1 A list of developing countries drawn up by the International Monetary Fund in April 2010 was used as the reference base (http://www.imf.org/external/pubs/ft/weo/2010/01/weodata/groups.htm#oem). theoretical papers. A paper was considered to be empirical if it reported a real case, whether it used primary or secondary data or both.

Method of Analysis

Content analysis was used in this study. Several areas of categorization were used to analyse the papers. Selection of the areas was influenced by earlier research both in eGovernment and in general information systems [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF][START_REF] Heeks | E-government in Africa: promise and practice[END_REF][START_REF] Chen | A paradigmatic and methodological examination of information systems research from 1991 to 2001[END_REF][START_REF] Grönlund | Ten years of e-Government: The 'end of history' and new beginning[END_REF][START_REF] Sahraoui | E-inclusion as a further stage of e-government? Transforming Government: People[END_REF]. The following categorization was used to classify the papers.

Research paradigm. This was classified into three categories: (a) positivist; (b) interpretative; and (c) critical [START_REF] Chen | A paradigmatic and methodological examination of information systems research from 1991 to 2001[END_REF][START_REF] Myers | Qualitative research in information systems[END_REF]. A research is said to be positivist if it attempts to test theory to increase the predictive understanding of phenomena [START_REF] Myers | Qualitative research in information systems[END_REF]. In this type of research, generally there is evidence of formal propositions, quantifiable measures of variables and hypothesis testing [START_REF] Orlikowski | Studying information technology in organizations: Research approaches and assumptions[END_REF]. Interpretative studies generally attempt to understand phenomena through the meanings that people assign to them and interpretative methods are "aimed at producing an understanding of the context of the information system, and the process whereby the information system influences and is influenced by the context" [9:4-5]. Critical research deals with social critique by assuming that social reality is historically constituted and that it is produced and reproduced by people. Hence, critical research focuses on the oppositions, conflicts and contradictions in contemporary society [START_REF] Myers | Qualitative research in information systems[END_REF][START_REF] Myers | A set of principles for conducting critical research in information systems[END_REF]. Knowledge framework. A set of categorized frameworks of knowledge used in eGovernment research proposed by Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] Methodology. The papers were also examined in accordance with the research method and data collection method used. The research methods were classified as: (a) survey; (2) case study; (c) experiment; and (d) action research [START_REF] Chen | A paradigmatic and methodological examination of information systems research from 1991 to 2001[END_REF]. The data collection methods were grouped as: (a) questionnaire; (b) interview; (c) reflection on project experience; (d) document analysis; (e) literature review; and (f) observation [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF]. In addition, the papers were also examined in terms of the time dimension of the research undertaken, whether cross-sectional or longitudinal, and in terms of data type, i.e., qualitative or quantitative.

Application. Topics were also grouped: (a) eAdministration -initiatives dealing particularly with improving the internal workings of the public sector; (b) eService -initiatives dealing particularly with the relationship between government and citizens as voters/stakeholders or as customers of public services; (c) eSociety -initiatives dealing with the relationship between public agencies and other institutions and with the relationship between civil society institutions; and (d) general eGovernment [START_REF] Heeks | E-government in Africa: promise and practice[END_REF]. Examples of the last topic include studies that aim to identify eGovernment challenges, barriers, and opportunities in a general context.

Focus of research. This was divided into three categories: (a) technocentric/online service delivery; (b) government-centric/organizational change; and (c) citizen-centric/better government. This categorization was based on a review of ten years of eGovernment development, which was conducted by Grönlund [START_REF] Grönlund | Ten years of e-Government: The 'end of history' and new beginning[END_REF] and identified three main models of eGovernment development, namely (1) the service delivery model, (2) the organizational change model, and (3) the better government model. The first model focuses on online service delivery from provision of online information to full electronic case handling. eGovernment implementation in this model is seen from a techno-centric perspective [START_REF] Sahraoui | E-inclusion as a further stage of e-government? Transforming Government: People[END_REF]. On the other hand, the third model does not necessarily involve eService delivery. Introducing cyber laws protecting privacy and increasing accessibility of eGovernment services (i.e., eInclusion initiatives) to various societal groups, including the disadvantaged ones, are examples of the initiatives in the third model [START_REF] Grönlund | Ten years of e-Government: The 'end of history' and new beginning[END_REF][START_REF] Sahraoui | E-inclusion as a further stage of e-government? Transforming Government: People[END_REF]. The second model (i.e., the organizational change model of eGovernment) assumes that IT itself cannot offer significant benefits without organizational change.

Findings

Overview of the Papers

Forty-three countries were reported in the papers. When the countries were not clearly stated or the study covered a lot of countries (such as Arab countries, Africa, Asia), I classified the papers into 'other countries' (Table 1). India was found to be the most frequently (11 times) reported country in the papers under study, followed by South Africa, China, Kenya, and Nigeria.

Research Paradigm

I attempted to group the papers based on their research paradigms used in the studies, although any attempt at classifying the papers like this might raise a discussion. Hence, since many papers did not clearly state their epistemological stance, I used additional criteria to classify the papers as follows.

The papers on eGovernment architecture and other technical aspects of eGovernment infrastructure were classified as positivist research, since they focused on building information technology (IT) artefact [see e.g., 11]. More specifically, these papers reported design research whose epistemology was primarily positivistic [START_REF] Iivari | A paradigmatic analysis of information systems as a design science[END_REF]. Also in this group of positivist research were papers reported a research model with a set of dependent and independent variables and used a statistical analysis to test some hypothesis [see e.g., 8]. The studies that carried out website evaluation were also grouped as positivist research. Almost half (42.6%) of the papers belonged to this group (see Table 2). The papers whose purpose was to evaluate and criticize the reasons or values behind an eGovernment initiative in a specific context were considered as critical research. Out of the 108 papers under study, only three papers that used a critical realism research paradigm.

The definition of interpretive research developed by Walsham [START_REF] Walsham | Interpreting Information Systems in Organizations[END_REF] was adopted to classify the papers. Around one-quarter (24.1%) of the papers adopted interpretive stance.

Other papers that did not belong to those three groups were put in the 'other' group. Examples were the papers that used secondary data to summarize or to present some ideas or concepts related to eGovernment, e.g., scope and challenges of eGovernment in a specific context. Descriptive literature review papers were also put in this group. This group had 30.6% of the papers.

Methodology

Case study was the most frequent research method used, although the protocols for conducting a proper case study research, such as those suggested by Benbasat et al. [START_REF] Benbasat | The case research strategy in studies of information systems[END_REF] and Yin [START_REF] Yin | Case Study Research: Design and Methods[END_REF], were in many cases not adequately followed (Table 3). I found that in 22.2% of the papers, the research methods were not easy to identify; for instance, the papers that attempted to assess development of eGovernment in a specific context lacked sufficient information on how the data were to be collected and conclusions drawn. Distributing questionnaire was the most popular data collection method, followed by interviews (see Table 4). Eight papers reported that more than one data collection (mixed) method was employed. No paper used only observation as its data collection method. Eighteen papers even did not report how the data has been collected explicitly, in a research method section, or implicitly, somewhere else in the paper. Note: * "A review of some relevant sources but without the rigor that might allow the approach to be called a proper literature review" [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] As many as 28.7% of the papers (31) adopted an individual/personal level of analysis, whereas 35 papers preferred the group/organization level, and the rest (42) focused on societal or country level. Qualitative research was found to be the approach preferred (62 papers) over quantitative (31) and mixed approaches [START_REF] Myers | A set of principles for conducting critical research in information systems[END_REF]. This study found that only six papers employed longitudinal studies.

3.4

Application This study found that the most frequently researched area of application was eService (46 papers), whereas the eSociety domain received the least attention (see Table 5). Thirty-five papers focused on eGovernment issues in general such as identifying eGovernment challenges, barriers, and opportunities. According to Heeks [START_REF] Heeks | E-government in Africa: promise and practice[END_REF], the focus of eService is to talk to citizens about details of public sector activities, to listen to citizens to gain input into public sector decisions and action, and to improve public services, whereas eSociety initiatives are intended to work better with business, to develop communities, and to strengthen partnership. Both of these applications deal with the demand side of eGovernment, whereas eAdministration, whose objectives are to cut process cost, manage process performance, and make strategic decisions in government, treats the supply side [START_REF] Madon | Evaluating the developmental impact of e-governance initiatives: an exploratory framework[END_REF].

3.5

Knowledge Framework Sixteen (14.8%) papers under study explicitly used theory (Table 6). Diffusion and adoption theories (such as Rogers's diffusion of innovation theory) were the most dominant theory used (in seven papers). Other theories used were stakeholder theory (two papers), actor network theory (two papers), and Giddens's structuration theory (two papers). The other three papers used institutional theory, theory of development, and intellectual capital theory. Some of these theories have been popular in information system research in the context of developing countries [START_REF] Walsham | Research on information systems in developing countries: Current landscape and future prospects[END_REF]. A large proportion of the papers (37.0%) used one or more concepts to inform the research process and analysis. The concepts included awareness, trustworthiness, usability, pushing versus pulling systems, good governance, hospitality and drifting, and interoperability. Some (7.4%) studies were framework-based. Various frameworks explicitly derived from a body of theoretical work used to study, e.g., information system adoption and success. Examples of scheme-based studies included the papers that used infrastructure scheme or architecture (6.5%). Model-based research (8.3%) mostly adopted stage model of eGovernment development. Fifteen (13.9%) papers fell into category-based studies. The categories used included gapanalysis, eGovernment barriers, and challenges.

In addition to the fact that only a limited number of the papers used theory explicitly, we should note here that 13 papers did not use any knowledge frameworks. One could ask whether the absence of theory in eGovernment research is a problem. If it is a problem, then it has been around for some years, as Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] found a similar phenomenon. They found that the background of the researchers had influence in this regard. Researchers whose academic base was informatics or computer science generally made no use of theory or framework.

3.6

Focus of Research This study revealed that the papers focused on various aspect of eGovernment, ranging from online service delivery, through organizational change, to better government (Table 7). The focus of research was reflected in the area of concern or in a more observable form, in the definition of eGovernment adopted by the researchers. For instance, one paper [16:124] that was dealing with online service delivery defined eGovernment as "a web-based project to enhance communication between the government and citizens, business partners, employees and other agencies, and information publication from the authority". As another example, a paper [17:37] that focused on organizational change defined eGovernment as "a way of organizing public management in order to increase efficiency, transparency, accessibility and responsiveness to citizens through the intensive and strategic use of information and communication technologies in the inner management of the public sector (intra and inter governmental relations) as well as in its daily relations with citizens and users of public services".

Other papers that used a more citizen-centric approach paid attention to eGovernment initiatives that were intended, for instance, to provide citizens with greater access to eGovernment services or to IT, especially the Internet, in general [e.g. [START_REF] Alexander | Information flows for meaningful implementation of the promotion of administrative justice act of South Africa[END_REF][START_REF] Galpaya | Taking e-government to the bottom of the pyramid: dial-a-gov[END_REF]. The main goal was to empower citizens and to realize the advantages of IT in development, where citizens are the main beneficiaries.

Discussion

Has something substantive happened in the field of eGovernment research since 2005? Findings from previous literature reviews made by Heeks and Bailur's [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] and Grönlund and Andersson [START_REF] Grönlund | e-Gov research quality improvements since 2003: More rigor, but research (perhaps) redefined[END_REF] were used as bases for comparison. Although these studies dealt with the eGovernment research in a general context, and did not focus on the context of developing countries, the findings could provide useful insights. To put them into the context, another literature review made by Walsham and Sahay [START_REF] Walsham | Research on information systems in developing countries: Current landscape and future prospects[END_REF] on IS research in the context of developing countries were brought in.

This study revealed that research paradigms of 30.6% of the papers were not discernable. This finding indicates that many eGovernment researchers did not pay attention to research epistemology or philosophies. This finding is in line with the previous literature review conducted by Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF], who found that most eGovernment research contained no clear statement of research philosophy. It is possible that for many of the eGovernment researchers the relevance of the research was more important than rigour. As Heeks and Bailur [1:251] noted, this finding still left "an open question about the importance and role of research philosophy in eGovernment research".

In 2005, Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] did not found any single papers that adopted interpretive paradigm. If we believe that interpretive research can provide more insights about a phenomenon under study, we may rejoice the recent development in our field since 24.1% of the studies adopted this paradigm. However, this study found that only few (2.8%) critical studies. This finding echoed the previous study made by Walsham and Sahay [15] that made a call for critical studies. They argued that this type of studies was important since it "can open up the 'black box' as an aid to deeper understanding, and a stimulus to appropriate action" [15:19]. It seemed that this call did not get a sufficient attention from the researchers of eGovernment in developing countries.

Walsham and Sahay [START_REF] Walsham | Research on information systems in developing countries: Current landscape and future prospects[END_REF] also recommended more action research and longitudinal studies. Similarly, Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] also found that around 80.0% of the papers reported cross-sectional research. Again, not many papers in this current study addressed this recommendation. This study found that only six papers employed longitudinal studies and three that reported action research. According to Walsham and Sahay [15:19],

"action research would appear to be particularly relevant in contexts where resources are scarce, when it can be argued that outside researchers should not only go away with data for their own papers and academic careers, but also aim to make a specific contribution in the research setting itself."

The action research may be also carried out in longitudinal studies, when interventions in field sites taking place on several occasions spaced out over time [START_REF] Walsham | Research on information systems in developing countries: Current landscape and future prospects[END_REF]. Often, action research is done in close collaboration with real stakeholders.

This study found that the proportion of theory-based studies increased. Heeks and Bailur's study [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] found only 1.2% theory-based studies out of 84, while this study revealed that 14.8% studies were theory-based. As presented above, there were a variety of theories brought in into the eGovernment research, such as institutional theory, theory of development, actor network theory, structuration theory, diffusion of innovation theory, and intellectual capital theory. This is promising since the quality of research can increase if the studies are theorybased. Theory is important in eGovernment research, because it serves as: (a) a means for researchers to communicate with practitioners; (b) a means for researchers to communicate with each other; (c) a means for accumulation of knowledge; and (d) a means for legitimacy and recognition of the field as an academic discipline [START_REF] Sahay | Information technology in developing countries: A need for theory building[END_REF]. During the research process, theory can be used to guide data collection and analysis [START_REF] Walsham | Doing interpretive research[END_REF].

It their study, Heeks and Bailur [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF] revealed that 20 (23.8%) papers did not reported their data collection methods. They considered it as 'bad practice' in eGovernment research. Unfortunately, there was no substantive change in this regard. This current study found that research methodology of 22.2% of the papers was indiscernible. In addition, data collection methods of 18 (16.7%) papers were also difficult to identify.

In 2006, Grönlund and Andersson's study [START_REF] Grönlund | e-Gov research quality improvements since 2003: More rigor, but research (perhaps) redefined[END_REF] found that the eGovernment studies was increasingly focusing on IT. A different picture was offered by this study, which found significant recognition of human and other contextual factors in addition to merely technical actor, use of a range of different research methods and broad use of primary data, and attention to various eGovernment applications (i.e., eAdministration, eService, and eSociety). This was considered as good practices in Heeks and Bailur's study [START_REF] Heeks | Analyzing e-government research: Perspectives, philosophies, theories, methods, and practice[END_REF]. Moreover, these practices were suggested by Grönlund and Andersson [START_REF] Grönlund | e-Gov research quality improvements since 2003: More rigor, but research (perhaps) redefined[END_REF] in their study. Instead of focusing on IT itself, eGovernment studies should focus on the role of IT in the context of society and government organization [START_REF] Grönlund | e-Gov research quality improvements since 2003: More rigor, but research (perhaps) redefined[END_REF].

To sum up, from the methodological perspective, some future research directions can be drawn from this discussion. These include (a) paying more attention to research paradigm and methodology; (b) preserving multiculturalism in eGovernment research, by adopting appropriate research paradigms; (c) encouraging action research and longitudinal studies; and (d) improving the research quality by bringing theories in.

Concluding Remarks

This paper has painted a current state of eGovernment research published between 2005 and 2010. The review was based on five areas of categorization, i.e., research paradigm, knowledge framework, methodology, application, and focus of research. Several future directions from the methodological perspective have been also presented. The main contributions of this paper are the answers to two research questions set at the outset. This study provided (a) a more recent picture of eGovernment research in the context of developing countries, and (b) a set of future research directions. This study was then expected to provide insights to eGovernment researchers. Lastly, like any other studies, this study has some limitations. First, the paper focused on state of eGovernment research and not the research issue. I will present the latter in another future paper. Second, although I have tried to include most of the relevant papers in the review, it is certainly possible that some were omitted, especially those which were not indexed in the five portals and those which were not written in English. This might have caused some bias. Third, the validity of the study may be questionable and open for discussion, since I was the single coder for all the papers under study. Similar studies should if possible use more than one coder to improve validity.

  was adopted. They grouped research as follows: (a) theory-based -when the paper made use of an explicit well-established theory such as structuration theory or institutional theory; (b) framework-based -when the paper used a framework from a body of theoretical work; (c) model-based -when the paper used a model presented without reference to any deeper knowledge framework, such as a stage model; (d) schema-based -when the paper made use of schemas of technique or a technical architecture of eGovernment; (e) concept-based -when the paper used a certain concept such as good governance or usability; and (f) category-based -when the paper presented a set of categories or list of factors. Papers that did not belong to any of these categories were categorized as non-framework-based research.

Table 1 .

 1 List of countries reported in the papers

	No	Country	n	No	Country	n
	1	India	11	23	Chile	1
	2	South Africa	7	24	Ethiopia	1
	3	China	6	25	Jamaica	1
	4	Kenya	6	26	Kazakhstan	1
	5	Nigeria	6	27	Kuwait	1
	6	Bangladesh	5	28	Malaysia	1
	7	Indonesia	5	29	Maldives	1
	8	Jordan	5	30	Mauritius	1
	9	Morocco	4	31	Mexico	1
	10	Nepal	4	32	Mongolia	1
	11	Pakistan	4	33	Mozambique	1
	12	Sri Lanka	4	34	Philippines	1
	13	Argentina	3	35	Rwanda	1
	14	Brazil	3	36	Saudi Arabia	1
	15	Egypt	3	37	Senegal	1
	16	Iran	3	38	Serbia	1
	17	Turkey	2	39	Taiwan	1
	18	Colombia	2	40	Tanzania	1
	19	Ghana	2	41	UAE	1
	20	Thailand	2	42	Uzbekistan	1
	21	Uganda	2	43	Zambia	1
	22	Cape Verde	1	44	Other countries	11

Table 2 .

 2 Research paradigm used in eGovernment research

	Research paradigm	Frequency	Percentage
	Positivist	46	42.6
	Interpretative	26	24.1
	Critical	3	2.8
	Other	33	30.6

Table 3 .

 3 Research methods used in eGovernment research

	Research method	Frequency	Percentage
	Survey	20	18.5
	Case study	60	55.6
	Experiment	1	0.9
	Action research	3	2.8
	Other	24	22.2

Table 4 .

 4 Data collection methods used in eGovernment research

	Data collection method	Frequency	Percentage
	Questionnaire	21	19.4
	Interview	21	19.4
	Reflection on project experience	16	14.8
	Document analysis	11	10.2
	Literature review	1	0.9
	Web content analysis	6	5.6
	Observation	0	0.0
	Hunt and peck*	6	5.6
	Mixed	8	7.4
	No discernable method	18	16.7

Table 5 .

 5 Area of application in eGovernment research

	Area of application	Frequency	Percentage
	eAdministration	17	15.7
	eServices	46	42.6
	eSociety	10	9.3
	eGovernment (in general)	35	32.4

Table 6 .

 6 Frameworks of knowledge used in eGovernment research

	Knowledge framework	Frequency	Percentage
	Theory-based	16	14.8
	Framework-based	8	7.4
	Model-based	9	8.3
	Schema-based	7	6.5
	Concept-based	40	37.0
	Category-based	15	13.9
	Non-framework-based	13	12.0

Table 7 .

 7 Focus of eGovernment research

	Focus of research	Frequency	Percentage
	Techno-centric/online service delivery	57	52.8
	Government-centric/organizational change	27	25.0
	Citizen-centric/better government	24	22.2

See ICT4D Journal RankingTable compiled by Richard Heeks (http://ict4dblog.wordpress.com /2010/04/14/ict4d-journal-ranking-table).