Anna Georgiadou
email: georgiadou.anna@gmail.com

Panagiotis Kokkinakos
email: pkokkinakos@epu.ntua.gr

Dimitrios Panopoulos

Sotirios Koussouris

Dimitrios Askounis
email: askous@epu.ntua.gr

Sotirios Koussou- Ris

A Multicriteria Methodology for the Selection and Prioritisation of Public Services

Keywords: Public Services, Analytic Network Process (ANP), Analytic Hierarchy Process (AHP), Multicriteria Decision Analysis (MCDA)

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The rapid technological development of the last decades has radically transformed the way government information and services are delivered to the citizens. Web-based telecommunication technologies have been utilized in order to improve and/or enhance the efficiency and effectiveness of service delivery in the public sector [START_REF] Jeong | Fundamental of Development Administration[END_REF][START_REF] Beynon-Davies | Electronic Local Government and the Modernisation Agenda: Progress and Prospects for Public Service Improvement[END_REF].

The general term "e-Government" was introduced so as to describe this technological penetration and public transformation and, today, it has progressed to the point where it has proven to be a force for effective governance and citizen participation, both at international and local levels [3]. This upward trend in e-Government development [START_REF] Millard | Reorganisation of Government Back-Offices for Better Electronic Public Services[END_REF] has accelerated despite, or maybe in part also due to, the current financial crisis and the pressing need for governments to be competent, transparent, accessible and efficient. The compelling factor and the bitter early lesson learned from the crisis is that trust matters, and lack of confidence in government, as in markets, can lead to calamity. The capacity to respond under difficult conditions and deliver the expected results are cornerstones of effective government and a foundation of public trust.

In this light, due to the aforementioned global economic crisis [START_REF] Niemira | An analytic network process model for financial-crisis forecasting[END_REF], the probability of a wrong investment has to be minimized and the spending of public money needs to be targeted and with low risk. In other words, the spawned need of public services prioritization will soon become a pressing challenge which could decide the future of e-Government's embracement and effectualness. The purpose of the present paper is to propose the implementation of a general multicriteria decision making method in the prioritization of public services. The question that is which of the public service scenarios under consideration should be delivered first, partially or fully, in an electronic way [START_REF] Lenk | Electronic Service Delivery -A driver of public sector modernization[END_REF]. This approach will enable governments take into account, not only historical data, but also the current conditions, the trends and the tacit knowledge of the experts.

In general, the domain of Multiple Criteria Decision Analysis (MCDA) concerns the approach of explicitly taking into consideration the pros and cons of a plurality of points of view, in order to make a decision [START_REF] Guitouni | Tentative guidelines to help choosing an appropriate MCDA method[END_REF][START_REF] Laaribi | A spatial decision aid: a multicriterion evaluation approach[END_REF][START_REF] Hanne | Intelligent Strategies for Meta Multiple Criteria Decision Making[END_REF]. MCDA is an activity which helps making decisions mainly in terms of choosing, ranking or sorting the potentials [START_REF] Kirytopoulos | An effective Markov based approach for calculating the Limit Matrix in the analytic network process[END_REF]. The proposed model is based on the Analytic Network Process (ANP), an advanced MCDA method [START_REF] Saaty | Decision Making with Dependence and Feedback: The Analytic Network Process[END_REF], and is meant to prioritize public services by using both quantitative and qualitative variables. This paper's object is twofold. It emphasizes on outlining the criteria which need to be considered in public services selection, while, on the other hand, it offers a comprehensive approach on how ANP can be utilized to aid the decision making process in the aforementioned problem. Its practical value is illustrated via an example in the area of the current Greek reality.

The rest of this paper is organised as follows. The second section offers a brief literature review and corresponding analysis of the AHP and ANP methodologies and describes the service scenarios under consideration, while it also defines the criteria which were used for the multicriteria model introduced. Moreover, in this section, the entire methodology application is presented as well as the respective results. The paper concludes with the last section where basic findings are discussed, minor limitations of the approach proposed are underlined and possible future steps are recognized.

Proposed Approach and Application

The current financial crisis enforced in a way the commanding penetration of the web-based technologies into the public sector. In this rather restrictive financial reality, the sectors where e-Government would most benefit the government as well as the users of public services (meaning both citizens and businesses) should carefully be examined and scaled. Therefore, the authors have tried to summarize and pinpoint the most important criteria in evaluating the delivered services while proposing the use of a multicriteria decision making approach in this field.

Methodological Multicriteria Background for Ranking and Prioritising Services

The methodology employed for selecting and prioritising services is based on the Analytic Network Process (ANP), a multiple-criteria decision analysis method which, since its introduction by Thomas L. Saaty, has been used around the world in a wide variety of decision situations, in fields such as government, business, industry, healthcare, and education [START_REF] Figueira | Multiple Criteria Decision Analysis: State of the Art Surveys[END_REF]. The ANP is a generalization of the Analytic Hierarchy Process [START_REF] Saaty | The Analytic Hierarchy Process: Planning, Priority Setting, Resource Allocation[END_REF][START_REF] Saaty | The Analytic Hierarchy Process Series[END_REF]. The Analytic Hierarchy Process (AHP) is designed to structure a decision process in a scenario affected by multiple independent factors. Based on this methodology, a complex problem is divided into sub-problems organized according to hierarchical levels [START_REF] Zahir | Geometry of decision making and the vector space formulation of the analytic hierarchy process[END_REF], with each level denoting a set of criteria or attributes related to the specific case. The top level of the hierarchy denotes the main goal, the bottom level contains the alternative or actions considered when achieving the goal, and the intermediate levels denote the factors that influence the upper levels. In this context, the AHP methodology makes it possible to compare different factors, where each factor's importance influences the final solution and the rank of the alternative that is to be used for reaching this solution [START_REF] Saaty | Decision Making in Complex Environments: the Analytic Hierarchy Process (AHP) for Decision Making and the Analytic Network Process (ANP) for Decision Making with Dependence and Feedback[END_REF][START_REF] Saaty | Decision Making with the Analytic Network Process[END_REF].

Service Scenarios' Description

The proposed model applies the ANP to the problem of evaluating, prioritizing and selecting public services [START_REF] Weber | Planning models for research and development[END_REF]. The model has been validated through a real case implementation [START_REF] Whitaker | Validation examples of the analytic hierarchy process and analytic network process[END_REF] concerning the prioritization of five different public services delivered by the Greek public sector. These services have been selected with the contribution of the Ministry of Administrative Reform and e-Government which provided the public service scenarios which are under consideration for improvement and radical transformation of the way they are delivered. The five most important ones which are presented below:

1. Access extracts of insurance record of Social Security Institutes: According to almost all European social security systems, every employee (citizen) must be registered to a public security organization. For each working day, a part of employee's salary is paid to the social security institute so that the employee can receive social security services (e.g. hospitalization, allowances). Usually the employer (e.g. a company) keeps a part of employee's salary and adds the employer's contribution so as to pay a specific amount of money to the Social Security institute in charge.

The employee can verify that the employer has paid its contribution to the Social Security institute by accessing and getting an extract of his/her record regarding the working days that have been paid for him/her. Currently this service is provided after an electronic or conventional application depending on the each specific case's sophistication level. 2. Certificates of various civil acts: Citizens request from e.g. a Citizen Service Centre the indented certificates while providing the necessary supporting documents (e.g. photocopy of the identity card for a birth certificate). This procedure has a wide application since such certificates are prerequisite to many other services (e.g. changing municipality). The authorized employee of the Citizen Service Centre collects and checks the supporting documents. In case something is missing, the procedure halts and the citizen is informed about the derived issue, otherwise, the Citizen Service Centre employee completes the electronic application and creates a new folder in the Citizen Service Centre's informational system. Then, the employee sends the application along with the supporting documents to the proper authority (via fax, post office etc.). When the Citizen Service Centre receives the certificate from the authority, the citizen is notified to collect it from the desired Citizen Service Centre.

Monitoring compensation procedures of Hellenic Organisation of Agricultural Insurances:

The Hellenic Organisation of Agricultural Insurances -ELGA is a governmental body for supporting agricultural production. ELGA provides social insurance to farmers and agricultural workers while, on the other hand, provides insurance on the agricultural production and compensates the producers in cases of physical disasters destroying their harvest. Whenever the harvest of a farmer is destroyedon the condition that he/she had applied all the necessary measures to be able to grow his/her products with safehe/she may claim for compensation. The proposed scenario has to do exactly with the procedure that is being followed by a farmer in order to get compensated after such a disaster. The service has as a main purpose to provide real-time information to the farmers concerning the stage of their requests for compensation, the height of the compensation decided and the time of the deposit. In addition the service provides to the farmer all the relevant certificates needed for his/her tax declaration concerning the compensation. The service is currently operated by the administrative services of ELGA with the involvement of the citizen service centers. 4. Transferring citizen rights to another municipality: The proposed scenario refers to the case that a citizen has moved to another municipality and wants to transfer his/her rights in order to be able to vote or to participate to several activities in the municipality he/she has moved.

Monitoring an application submitted through e-Gov procedures: Citizen Service

Centers in Greece act as a mediator between the citizen and the competent authorities. Whenever a Citizen requests a service, Citizen Service Centres create a dossier of the case including the application forms and the supporting documents. This dossier is transferred to the competent authority. Citizen Service Centres are not responsible/authorized for the execution of the service. Usually an estimated time for the execution of the service is announced to the citizen. As long as the response of the competent authority is in the expected timeframe, the result of the case is transferred to the Citizen Service Centre. As a final step Citizen Service Centres contact the citizen announcing that his/her case has been completed and deliver the result to the citizen. A new service has to be designed in order so that the Citizen Service Centres and the Citizen have constant information regarding the status of their case. This service will allow immediate response to the Citizens for the status of their case. Moreover, this service will provide valuable information for the public administration regarding the steps of a process or problems that usually cause delays. This is a valuable tool for indenting areas for public administration reform and simplification of the procedures.

Criteria Definition

In our effort to develop a public service evaluation ANP model, the most critical point was to define the criteria which need to be taken into consideration [START_REF] Huang | Multidimensional data in multidimensional scaling using the analytic network process[END_REF]. Towards that direction, the authors performed an extended research work concerning evaluation criteria in the public sector and proceeded to selecting the common criteria that were common among the most relevant approaches [START_REF] Jackson | Public Service Performance Evaluation: A Strategic Perspective[END_REF][START_REF] Hock | J: What Does It Mean to Optimise Public Service Delivery?[END_REF][START_REF] Dignan | What does "good enough" mean in public sector IT? OVUM[END_REF][START_REF] Sharma | Evaluating public service delivery in emerging markets, Service Operations, Logistics, and Informatics (SOLI)[END_REF] and reflect in the best way the cases examined. These criteria, combined with the ones extracted from the Services Description Template, introduced in [START_REF] Koussouris | A Selection And Prioritisation Framework For Collaborative Public Services Design[END_REF], aim to document the various facts and details of every public service scenario analysed. The criteria were reviewed and categorized into clusters that were later on communicated and evaluated by the executive officers of the Greek Ministry of Administrative Reform and e-Government, hereafter referred as decision makers. This procedure concluded in the formation of the final model.

In the following table, a detailed description of the criteria used is presented combined with their categorization into clusters:

M Political dimension

High/Medium /Low

Indicates the compliance of the service to the Organisation's Mission and the Strategic Policies as well as the projected Public Perception of the service and its political impact [START_REF] Bellos | An MCDA Approach for Project Selection in Public Sector[END_REF].

SERVICE LOAD DATA

N Total Service Duration

Duration in hours/days

Indicates how many hours/days does it take for the service to completeno matter if the service is provided conventionally or electronically.

O Frequency Requests/ year

Indicates the number of the requests of the specific service in a given amount of time.

P No of Inputs Number

Defines the number of prerequisites of the service.

At this point, a reference should be made to the need to adjust the criteria proposed via the Services Description Template so as to generalize their application. More specifically, the criteria "Requirement for Personal Presence at the submission of the application" and "Requirement for Personal Presence at the delivery of the service" were unified to lead to the more generic one "Requirement for Personal Presence" mainly due to the different nature of each public service scenario under investigation. Moreover, the Services Description Template seems to totally overlook the political dimension of a public service [START_REF] Medaglia | A multiobjective model for the selection and timing for public enterprise projects[END_REF] downgrading its role to the decision making procedure in the public sector.

From a technical point of view, the structure of the ANP model is described by its clusters and elements and the connection between them. These connections indicate the flow of influence among the elements. The resulting model is illustrated in the following figure and consists of five clusters:

Fig. 1. The Public Service Evaluation ANP Model

For the ANP implementation, Super Decisions1 software was employed, developed by the ANP Team (working for the Creative Decisions Foundation), to alleviate the mathematical burden. The figure depicted above represents the Public Service Evaluation ANP Model as constructed with the usage of Super Decisions software [START_REF] Saaty | Super decisions software[END_REF] so as to implement our example case study for the Greek public sector.

Analysis approach and paired comparisons

Having decided to adopt the influencing analysis approach (recommended and broadly used in bibliography) and with the valuable contribution of the decision makers, the connections among the clusters and elements were defined (depicted in Figure 1 with arrows) to reflect the interaction of the different factors in the real environment. For instance, the bidirectional arrow from cluster "1. Service Flow" to cluster "3. Service

Importance" captures the notion of outer dependence on ANP and is interpreted in means of influence among their elements. On the other hand, the concept of inner dependence is depicted with a feedback loop such as the one present in cluster "2.

Service Complexity" and represents the influence among different elements of the same cluster. Following the final formation of the public service MCDA model, the collaborating officers of the Ministry of Administrative Reform and e-Government were asked to respond to a series of pair wise comparisons with respect to a control criterion. "Super Decisions" software is able to calculate the inconsistency ratio for each comparison matrix, so the most consistent value for the entries can be determined. The inconsistency measure is useful for identifying possible errors in judgments as well as actual inconsistencies in the judgments themselves. Afterwards, the Clusters Priority Matrix is constructed; whose columns consist of the weighted priorities derived during the pair wise comparisons. The next step of the method was the computation of the SuperMartix, the Weighed SuperMatrix and the Limiting Matrix. These matrixes represent the impact or influence of each elements on other elements of the problem and are computed by the Super Decision software. Indicatively, the Weighted SuperMatrix of the examined problem is presented in the following table :   Table 3. The Weighted SuperMatrix

Final Prioritization Results

The final results, prioritization (normalized values) of public services under evaluation as derived from the limit matrix are presented in Table 4 and, for a more representative overlook of the outcome, in Figure 2. As shown above, the priorities of the services which were derived after applying the multi-criteria analysis do not represent only their importance for the Ministry of Administrative Reform and e-Government but also how much appropriate they are based on the technical implementation requirements. It becomes obvious that services like the prevailing one (Access extracts of insurance record of Social Security Organisation), which affect a large number of end users but are also compatible with the public authorities' initiatives and frameworks, take precedence against other scenarios that do not meet these standards. Nevertheless, it needs to be noted that, in order to reach a justified and viable result, a time consuming and possibly tiring effort is needed both by the Public Sector (which has to provide all respective data) and by the supporting technical user (who operates the suitable software) [START_REF] Bekkers | Electronic Service Delivery in Public Administration: Some Trends and Issues[END_REF].

Sensitivity Analysis

In order to check the stability of our results, a sensitivity analysis is performed with a series of a "what if" kind of questions to verify that the final classification is stable when the inputs, whether judgments or priorities, are changed. Therefore, the authors have proceeded by radically altering the responses received during the pairwise comparisons.

The first and most important part of our sensitivity analysis was the one regarding the "Level of compliance with National Interoperability Frameworks", which is the dominant criterion based on our ANP analysis. The procedure showed that irrespective of its value, the rank of the public services scenarios is preserved.

The rest of our analysis consisted of similar sensitivity tests for the rest of the criteria used in our model and the results led to the conclusion that the outcome is sufficiently stable and does not change the overall rank [START_REF] Saaty | Principia Mathematica Decernendi: Mathematical Principles of Decision Making[END_REF].

Table 1 .

 1 Criteria Description

	Clus-ter	S/N	Criterion	Possible Values	Description
		A	Public (Final) Service	Yes / No	Indicates whether the service is public (final) or if it is an intermediate process.
					Defines whether the service could be called
	SERVICE FLOW	B C	Self-appointed call of Service Re-usability by other services	Yes / No High/Medium /Low	only with the acquiescence of service receiver or, alternatively, automatically by a public organization. Indicates whether this service, although final, is also prerequisite to other public services -so it can be reasonably considered as more important Indicates whether this service is executed
		D	Service Execu-tion Domain	Internal/ External	within the boundaries of an organization (e.g. a municipality), or it includes interaction with other, «external» organizations (e.g. the po-
					lice) as well.
		E	Legal Frame-work Implica-tion	High/Medium /Low	States in which degree legislation is affecting the operation of a service.
	SERVICE COMPLEXITY	F G	Level of com-pliance with National In-teroperability Frameworks Requirement for Personal	Yes / No / No Interoperabil-ity Frame-work existing Yes / No	Indicates whether the delivery of the service is compliant with the correspondent interopera-bility framework. Indicates whether physical presence at the submission of the application, the delivery or

Table 2 .

 2 Clusters Priority Matrix

		1. Ser-vice Flow	2. Service Complexity	3. Service Importance	4. Service Data Load	Alternatives
	1. Service Flow	0.002652	0.558299	0.487794	0000	0000
	2. Service Com-plexity	0.681786	0.121505	0.121957	0000	0000
	3. Service Im-portance	0.079742	0.110766	0.195125	0.500000	0000
	4. Service Load Data	0.233168	0.087925	0000	0000	0000
	Alternatives	0.002652	0.121505	0.195125	0.500000	0000

Table 4 .

 4 Final Public Services Prioritization Results

	Alternatives	Total	Normal	Ideal	Ranking
	1. Access, extracts of insurance record in Social Security Organisation	0.0700	0.2867	1.0000	1
	2. Certificates of various civil acts	0.0532	0.2177	0.7593	2
	3. Monitoring compensation procedures
	of Hellenic Organisation of Agricultural	0.0402	0.1647	0.5746	4
	Insurances
	4. Transferring citizen rights to another municipality	0.0446	0.1828	0.6375	3
	5. Monitoring an application submitted through e-Gov procedures	0.0362	0.1482	0.5168

Fig. 2. Prioritization of Proposed Public Services

http://www.superdecisions.com/

Acknowledgements

. Part of the work presented in this paper is sponsored by the European Commission in the frames of the FP7 project "COCKPIT -Citizens Collaboration and Co-Creation in Public Service Delivery" (Contract No: FP7-ICT-2009-4 COCKPIT, http://www.cockpit-project.eu/).

Conclusions

The prioritization of public services is a rather difficult endeavour as it considers many conflicting criteria, both tangible and intangible. The present paper introduced an MCDA approach, based on the ANP methodology, capable of capturing this complex decision environment. In more detail, the proposed model enables the decision maker to better comprehend the various factors that influence the final outcome while, at the same time, documents the evaluation results in an indisputable way. The case study used to evaluate the effectiveness of the proposed model referred to a limited number of scenarios; however, the presented methodology is also applicable to sets of numerous alternatives with the cost of complexity. In our example, the highest priority was assigned to the "Access extracts of insurance record of Social Security Organisation" while the most dominant criteria were proven to be the "Level of compliance with National Interoperability Frameworks" and the "Re-usability by other services" demonstrating the importance service complexity and flow factors bear in this field.

The authors feel the obligation to underline the fact that the ANP methodology proposed bears a few limitations which mainly derive from the subjectivity of the input by the decision makers and the time-consuming nature of the pairwise comparisons which may lead to inconsistencies and, consequently, to doubtful and wrong results. Yet these limitations are not considered to be restrictive mainly due to the complexity of the e-Government field.

The present paper raises several important issues that could spark further research concerning the evaluation of the model and its implementation at additional case studies in different European countries.

In the near future, it would be useful if a coordinated pan European initiative among Public Institutions took place so as to define common and mutually accepted criteria for the prioritization of public services in order to formulate a unified model for the sophistication of public service delivery.

Last but not least, an ICT tool [START_REF] Lee | Using analytic network process and goal programming for interdependent information system project selection[END_REF][START_REF] Lee | An ANP-based technology network for identification of core technologies: A case of telecommunication technologies[END_REF] could be developed in order to apply the aforementioned methodology efficiently and effectively to any kind of service providing useful results and guidelines to its final users.


Eleftherios Siskos
email: lsiskos@epu.ntua.gr

Michail Malafekas
email: mmalafekas@gmail.com

Dimitris Askounis
email: askous@epu.ntua.gr

John Psarras

E-government Benchmarking in European Union: A Multicriteria Extreme Ranking Approach

Keywords: E-government, Multicriteria analysis, Robustness, Benchmarking, Ordinal regression, Extreme ranking analysis

E-government benchmarking is being conducted by various organizations but its assessment is based on a limited number of indicators and does not highlight the multidimensional nature of the electronically provided services. This paper outlines a multicriteria evaluation system based on four points of view: (1) infrastructures, (2) investments, (3) e-processes, and (4) users' attitude in order to evaluate European Union countries. In this paper, twenty one European Union countries are evaluated and ranked over their e-government progress. Their ranking is obtained through an additive value model which is assessed by an ordinal regression method and the use of the decision support system MIIDAS. In order to obtain robust evaluations, given the incomplete determination of intercriteria model parameters, the extreme ranking analysis method, based on powerful mathematical programming techniques, has been applied to estimate each country's best and worst possible ranking position.

Introduction

E-government benchmarks are used to assess the progress made by an individual country over a period of time and compare its growth against other countries. A focused assessment of egovernment and other initiatives such as e-commerce, e-education, e-health, and e-science is essential if a country is to make substantial progress [START_REF] Ojo | Determining progress towards e-Government -What are the core indicators?[END_REF]. Benchmarks can have a significant practical impact, both political and potentially economic and can influence the development of e-government services [START_REF] Bannister | The curse of the benchmark: an assessment of the validity and value of egovernment comparisons[END_REF]. The results of benchmarking and ranking studies, particularly global projects conducted by international organizations, attract considerable interest from a variety of observers, including governments (ITU, 2009).

Indices and indicators used in benchmarks are generally quantitative in nature, and collectively form a framework for assessment and ranking. [START_REF] Rorissa | Benchmarking e-Government: A comparison of frameworks for computing e-Government index and ranking[END_REF] classified the frameworks to those based on measurable characteristics of the entities; those that use one or more subjective measures; and the rest few that employ a combination of both. They also state that frameworks based on grounded and broadly applicable measures tend to attract fewer criticisms. On the other hand, the frameworks based on subjective measures are liable to controversies and complaints, especially from those countries or institutions who consider their characterization as inaccurate. Therefore, rankings should be supported by well understood and clarified frameworks and indices as well as transparent computational procedures to maximize their acceptability by the governments and the scientific community. The whole approach consists of helping decision makers (DM), experts or potential evaluators in determining their own country evaluations, based on their own value systems and their own ways of preferring, in order to propose alternative evaluation solutions in contrast to standard published benchmarks.

The aim of this paper is to extend this decision support methodology to take into account the incomplete determination of inter-criteria model parameters and to obtain a robust evaluation of the countries. These targets were achieved with the aid of the extreme ranking analysis [START_REF] Kadzinski | Extreme ranking analysis in robust ordinal regression[END_REF]) that estimates each country's best and worst possible rank. This method is based on powerful mathematical programming techniques.

The paper is organized as follows: In section 2 the consistent family of criteria is briefly outlined. Section 3 presents the assessment of the multicriteria evaluation model for a single specific decision maker-evaluator while section 4 presents the extreme ranking analysis method and the obtained results. Finally section 5 concludes the paper.

Multicriteria Benchmarking Modeling

In order to achieve an overall assessment of global e-government, a consistent family of criteria was built according to the classical modeling methodology of [START_REF] Roy | Méthodologie multicritère d'aide à la décision[END_REF], in the following way: Users' attitude criteria g 6 : Citizens' online interaction with authorities. This criterion indicates the percentage of citizens that are already using the web to interact with the authorities.

g 7 .
Businesses' online interaction with authorities. It indicates the percentage of businesses that are using the web to interact with the authorities.

g 8 : Users' experience. This criterion expresses citizens' experience over the 20 e-services and the national portal. The data composing this criterion stem from the European Commission's 9th Benchmark Measurement. All details about the criteria construction techniques are thoroughly described in [START_REF] Siskos | Multicriteria decision support for global egovernment evaluation[END_REF]. Tables 1 and2 present the criteria evaluation scales and the criteria scores achieved by the twenty one European countries, respectively.

Assessing an Overall Evaluation Model

The main target of the proposed methodological frame is the assessment of a multicriteria additive value system, for a single decision maker, that is described by the following formulae:

(1) , , for

(3) , for (2)
where is the performance vector of a country on the n criteria; and are the least and most preferable levels of the criterion , respectively; , are non decreasing marginal value functions of the performances , and is the relative weight of the i-th function . Thus, for a given country a, and represent the multicriteria vector of performances and the global value of the country a respectively.

Both the marginal and the global value functions have the monotonicity property of the true criterion. For instance, in the case of the global value function, given two countries a and b the following properties hold: (Preference) [START_REF] Greco | Robust ordinal regression[END_REF] (Indifference) [START_REF]ITU: Measuring the information society: The ICT development index[END_REF] The necessary hypothesis to validate an additive value function for a given decision maker (DM) is the preferential independence of all the criteria (see Keenney and Raiffa, 1976, Keeney, 1992 for instance). A pair of criteria ( , ) is preferentially independent from the rest of the criteria when the trade-offs between the and criteria are not dependent on the values of the rest of the criteria. All criteria are supposed to be preferentially independent when the same condition holds for all pairs of criteria. When the functions in formula (1) are already assessed, the linear model ( 1)-( 4) exists if and only if the inter-criteria parameters (weighs) are constant substitution rates (value trade-offs) between .

This value system can be obtained utilizing various methods (see [START_REF] Keeney | Decisions with multiple objectives: Preferences and value trade-offs[END_REF][START_REF] Keeney | Value-focused thinking: A path to creative decision making[END_REF][START_REF] Figueira | State-of-Art of Multiple Criteria Decision Analysis[END_REF]. Because of the objective difficulties to convince decision makers in externalizing tradeoffs between heterogeneous criteria and verify the preferential conditions cited above, decision analysts usually prefer to infer the DM's additive value function from global preference structures, by applying disaggregation or ordinal regression methods (see Jacquet-Lagrèze and [START_REF] Jacquet-Lagrèze | Assessing a set of additive utility functions for multicriteria decision making[END_REF][START_REF] Jacquet-Lagrèze | Preference disaggregation: 20 years of MCDA experience[END_REF][START_REF] Greco | Ordinal regression revisited: Multiple criteria ranking using a set of additive value functions[END_REF], 2010).

In this study the disaggregation UTA II method is implemented by assessing the additive model ( 1)-( 4) in two phases: In the first phase the expert is asked to assign some value points of the corresponding evaluation scale for every criterion separately. Then, each marginal value function is optimally fitted (see Fig. 1) and accepted by the DM. In a second phase, the criteria weights are estimated using inference procedures (see next section).

Estimation of Weights and Extreme Ranking Analysis

Through this phase the UTA II disaggregation procedure is used to infer the inter-criteria parameters , Specifically, the DM-evaluator is asked to give a ranking (weak order) on a set of reference countries , that are fictitious country profiles Therefore, for every pair of consecutive countries (a m , a m+1 ) holds, either a m Pa m+1 (preference of a m ) or a m Ia m+1 (indifference). UTA II solves the linear program ( 7)-( 11) below that has k constraints because of the transitivity of the (P, I) preference system.

Subject to:

for m = 1, 2, …, k -1 (8) or (9) (10) (11)
where δ is a small positive number, equal to 0.001 for instance; is the evaluation of the country on the i-th criterion and its corresponding marginal value; are the over-estimation and the under-estimation errors concerning the j-th country's position, respectively. This technique was applied for a set of thirteen country profiles and a zero error sum was obtained (

The optimally most characteristic weighting factors are reported in formula (12) while the corresponding ranking of the European countries is presented in Table 3. More details about UTA II illustration is given in [START_REF] Siskos | Multicriteria decision support for global egovernment evaluation[END_REF]. [START_REF] Ojo | Determining progress towards e-Government -What are the core indicators?[END_REF] , is the performance vector of a country on the eight criteria. However, the above estimation procedure bears robustness issues. In fact, there exists an infinite number of weighting vectors that are optimally consistent with the whole set of constraints ( 8)- [START_REF] Keeney | Decisions with multiple objectives: Preferences and value trade-offs[END_REF]. In order to study the impact of this indetermination on the ranking of the countries the extreme ranking analysis of [START_REF] Kadzinski | Extreme ranking analysis in robust ordinal regression[END_REF] has been applied with the aid of the GAMS platform.

The extreme ranking analysis algorithm examines each country individually and estimates the best and worst possible rank it can achieve. The methodology leading to the estimation of the best possible rank of each country is outlined below.

In order to determine the best possible rank of a country A, taking into consideration all the possible combinations of the criteria weighting factors, the number *   of the countries that surpass country A in the ranking under any circumstances is calculated. The countries that surpass country A in the ranking for a limited number of combinations of the criteria weights are not included in the *   set. Therefore, the best possible ranking position that can be achieved by the country A is

*   +1.
Thus, the problem is reduced to the calculation of the *   set for each individual country.

This set is calculated through the modeling and the solution of the mixed integer programming problem presented below:

Subject to:

Constraints (8) - ( 11
) (14) U(α) ≥ U(b) -Μu b , b A\ {α} (15)
where M is an auxiliary variable equal to a big positive value, and u b is a binary variable associated with comparison of the country A to another country B. There exist  -1 such variables, each corresponding to b A\ {α}.  is the total number of the countries under evaluation, i.e. 21. The determination of the worst possible ranking of a country A, requires a similar procedure.

In this case, however, it is estimated the number of countries

(17) U(b) ≥ U(α) - Μu b , b A\ {α} (18) (11)
The extreme ranking positions of the 21 European countries assessed, are graphically presented in Fig. 2.

Conclusion

The paper addressed the robust assessment of global e-government based on multiple criteria and special extreme ranking procedures. The proposed approach focused on the evaluation of European countries according to the standards of a benchmark.

The e-government evaluation process is an independent procedure enabling each individual to specify his (her) own preferences on criteria value functions and weights, and results in a personalized ranking of the countries. In other words, each evaluator has control over his (her) set of criteria and the assessment of his (her) own evaluation model. The proposed multicriteria techniques offer the possibility to combine different preferences and considerations of multiple decision makers and merge them easily through interactive iterative processes.

The next research steps include the development of robustness control procedures based on cardinal and visualization measures as well as the development of a decision support system aiding anyone to form his own e-government benchmarking.

Acknowledgements. The authors wish to acknowledge financial support from the Ministry of Education, Religious Affairs, Culture and Sports of Greece and the European Social Fund, under Grant THALIS, MIS 377350: Methodological approaches for studying robustness in multiple criteria decision making projects.

g 5 :

 5 E-participation. It expresses the interaction achieved between governments and citizens in a manner of information sharing, e-consultation and e-decision making. The source of the eparticipation criterion is the United Nation's survey on e-government published in 2012.

4 Fig. 1 .

 41 Fig. 1. Marginal value functions of the eight criteria



  that achieve a worse ranking position for all possible combinations of the weighting factors. Therefore, the worst possible rank a country A can achieve is  -* A  . * A  set is calculated through the solution of the integer programming problem outlined below:

Fig. 2 .

 2 Fig. 2. Extreme ranking positions of twenty one European countries.

  Two among others well-known e-government benchmarks still being conducted are those of the United Nations and the European Commission (see United Nations, 2012 and European Commission, 2010 for more details). are modeled (see below) to evaluate twenty one European Union countries for which the related data are available. Their ranking is obtained through an additive value model which is assessed by an ordinal regression method.

	Recently, Siskos et al. (2012) proposed a multicriteria evaluation system based on four points
	of view: (1) infrastructures, (2) investments, (3) e-processes, and (4) users' attitude from
	which eight criteria

Table 1 .

 1 Criteria evaluation scales and sources.

	Criterion	Index	Worst level	Best level	Source
	g 1	% population	0	100	Eurostat
	g 2	% population	0	100	Eurostat
	g 3	% GDP	0	5	Eurostat and IMF
	g 4	%	0	100	European Commission
	g 5	index [0-1]	0	1	United Nations
	g 6	% citizens	0	100	Eurostat
	g 7	% businesses	0	100	Eurostat
	g 8	% index	0	100	European Commission

Table 2 .

 2 Multicriteria evaluation of twenty one European countries.

		g 1	g 2	g 3	g 4	g 5	g 6	g	g 8
	Belgium	86.5	81.5	3.3	92	0.59	28		68
	Czech Rep.	94	90	3.3	85	0.13	35		22
	Denmark	83.5	85.5	3.8	95	0.64	11		30
	Germany	71.5	80.5	3.8	99	0.61	16		55
	Estonia	86	78	4.0	97	0.69	35		21
	Ireland	77	97	3.5	100	0.44	40		49
	Greece	86.5	80.5	2.4	70	0.26	57		33
	Spain	76.5	85	3.1	98	0.83	75		86
	France	84	92	3.7	94	0.60	65		83
	Italy	93.5	94.5	2.7	99	0.21	84		85
	Hungary	90	80	3.4	80	0.31	57		80
	Netherlands	75	87.5	3.5	97	0.60	80		80
	Austria	83	78	3.3	100	0.50	85		90
	Poland	77.5	60.5	2.7	87	0.24	88		81
	Portugal	79	81	3.8	100	0.27	79	59.5	75.5
	Slovenia	68	72.5	3.4	99	0.51	91		68.5
	Slovakia	87	77	3.1	81	0.07	91		94.5
	Finland	67	70	4.3	96	0.41	51		93.5
	Sweden	79.5	65.5	4.1	99	0.49	70		92.5
	Norway	88.5	90	2.5	92	0.50	83.5		87
	Un. Kingdom	83.5	87.5	4,3	97	0,77	83.5	76.5	91

Table 3 .

 3 E-government ranking of the twenty one European countries.

	Rank	European	Global Value
	position	Country
	1	Sweden	0.825
	2	Denmark	0.821
	3-4	Finland	0.796
	3-4	Netherlands	0.796
	5	Norway	0.765
	6-7	Germany	0.745
	6-7	United Kingdom	0.744
	8	France	0.738
	9	Estonia	0.729
	10-11	Austria	0.701
	10-11	Slovenia	0.701
	12	Spain	0.693
	13	Belgium	0.686
	14	Ireland	0.679
	15	Portugal	0.633
	16	Czech Republic	0.582
	17	Slovakia	0.578
	18	Hungary	0.568
	19	Poland	0.548
	20	Italy	0.533
	21	Greece	0.467


Christian Flender
email: christian.flender@iig.uni-freiburg.de

Order Effects in Observations of Stated and Revealed Privacy Preferences

Keywords: Privacy Decision Making, Cognitive Dissonance, TET 2.0

Many Internet services rely on consumers disclosing their personal data. Despite heavy usage and wide acceptance of services like Online Social Networks, doubts about sustainability of trusted relationships remain. Surveying consumers about their preferences reveals severe concerns about the fate of their personal data. In stark contrast to privacy concerns (stated preferences), however, consumers generously disclose personal data in exchange for free Internet services (revealed preferences). It has been argued that individuals experience dissonant states in privacy decision making. The tension between stated and revealed preferences is eliminated with the decision made in order to reduce discomfort. This paper proposes a survey design to determine 1) order effects as indicators for dissonant states in privacy decision making, and 2) the degree of experienced tension between stated and revealed preferences. Observations of data valuation and disclosure behavior are dissonant if they do not commute, i.e. disclosing data prior to valuating privacy does not equal privacy valuation before data disclosure. Determining the degree of dissonance in privacy decision making is expected to inform the design of transparency mechanisms to influence experienced dissonance between stated and revealed privacy preferences.

Introduction

With the global acceptance of Internet services dealing with personal data, generous disclosure behavior of individuals has posed pressing research problems. One of the biggest issues at present is the challenge to establish trust among all the Web participants involved in order to sustain the benefits of digital life for individuals and society at large. Measuring trust or the lack of it, however, is a difficult exercise, last but not least due to paradoxical preferences stated and revealed by Web users. What came to be known as the privacy paradox measures the contradiction between high valuation of privacy and generous disclosure of personal data [START_REF] Acquisti | Privacy and Rationality in Individual Decision Making[END_REF]. Many attempts to explain this discrepancy have been put forth from psychological, sociological, and economic perspectives. Some authors argue that individuals attribute a higher value to the benefits of Web services arriving sooner in time compared to the associated costs and long-term risks in the far future [START_REF] Acquisti | Losses, Gains, and Hyperbolic Discounting: An Experimental Approach to Information Security Attitudes and Behavior[END_REF]. Others share the view that peer group pressure imprisons Web users in Online Social Networks effectively rendering opting out behavior and avoidance of data disclosure undesirable [START_REF] Cusumano | Platform wars come to social media[END_REF]. Economically, it has been argued that Web users have information deficits about the way their data is used and valued on data markets which makes them careless about privacy matters [START_REF] Müller | Vertrauensinfrastruktur und Privatheit als ökonomische Fragestellung[END_REF].

More recently, privacy decision making has been put under the light of cognitive dissonance theory [START_REF] Flender | Type Indeterminacy in Privacy Decisions: The Privacy Paradox Revisited[END_REF]. The theory of cognitive dissonance attempts to explain the behavioral phenomenon of a person experiencing two conflicting cognitions (behaviors, ideas, beliefs, or attitudes) [START_REF] Festinger | A Theory of Cognitive Dissonance[END_REF]. People reduce or eliminate the tension arising from the dissonance by either changing cognitions or adding new cognitive elements in order to reduce discomfort. Subjects in dissonant situations experience an inconsistent state of mind and with the decision made they rationalize their choice in order to re-establish a consistent state. For instance, a smoker who knows or learns about negative health effects, either stops smoking, changes his attitudes towards smoking, or adds explanations about smoking to reduce dissonance, e.g. car driving is more dangerous than smoking. Likewise, Web users are supposed to experience two conflicting situations. On the one hand, they appreciate the right to be left alone in private. But when confronted with attractive Web services they change their attitude and generously disclose sensitive information about their personality and private life. Cognitive dissonance seems to have an impact upon privacy decision making and raises the following research questions:

1. Do Web users experience cognitive dissonance in privacy decision making? 2. What is the degree of cognitive dissonance in privacy decision making? 3. Can Transparency Enhancing Technology (TET) influence cognitive dissonance and trigger change?

The objective of this paper is to propose a design to measure cognitive dissonance in privacy decision making and pave the way for further studies on the effectiveness of Transparency Enhancing Technology (TET). TETs like privacy statements or dashboards for personal data management are meant to enable Web users to exercise their right to informational self-determination, i.e. their right to control, edit, manage, and delete personal information and decide when, how, and to what extent it is communicated to others [START_REF] Westin | Privacy and Freedom[END_REF]. At present, TETs are hardly used. Offering to Web users incentives thereby influencing dissonant behavior may inform TET 2.0, a new generation of transparency mechanisms changing user behavior towards trusted relationships based upon their free selfdetermination.

The method of research presented in this paper follows the design science paradigm [START_REF] Hevner | Design science in information systems research[END_REF]. A survey design is developed and implemented on the basis of methodological considerations on how to operationalize the measurement of cognitive dissonance in privacy decision making. Drawing from [START_REF] Flender | Type Indeterminacy in Privacy Decisions: The Privacy Paradox Revisited[END_REF], privacy decision making is inconsistent or dissonant if observations of stated and revealed privacy preferences are not interchangeable, i.e. they do not commute [START_REF] Flender | Type Indeterminacy in Privacy Decisions: The Privacy Paradox Revisited[END_REF]. The design developed in the following accounts for order effects in measurements of privacy preferences.

Order Effects Indicate Dissonance in Privacy Decisions

The baseline of many privacy research problems is the paradoxical observation of generous data disclosure and high appreciation for privacy [START_REF] Deuker | Addressing the privacy paradox by expanded privacy awarenessthe example of context-aware services[END_REF][START_REF] Spiekermann | E-privacy in 2nd generation ecommerce: privacy preferences versus actual behavior[END_REF][START_REF] Awad | The personalization privacy paradox: An empirical evaluation of information transparency and the willingness to be profiled online for personalization[END_REF][START_REF] Norberg | The privacy paradox: Personal information disclosure intentions versus behaviors[END_REF]. When people reflect upon the value and sensitivity of their personal information most of them are deeply concerned about the fate of the traces they leave on the Internet. Contrary to stated preferences, however, most people reveal a low valuation of their personal data when it comes to eased communication and cooperation on the Internet, comfortable online shopping, or reciprocal attention spending in Online Social Networks. A recent explanation of the privacy paradox was developed in [START_REF] Flender | Type Indeterminacy in Privacy Decisions: The Privacy Paradox Revisited[END_REF] and draws from cognitive dissonance theory. According to preference (type) indeterminacy in privacy decisions, the paradoxical nature of privacy decisions stems from the fact that states of decision outcomes are constructed with the decision made and thus do not exist prior to their construction. Moreover, the state of a decision outcome depends on the sequential order of successive measurements of privacy preferences. If sequential observations of stated and revealed privacy preferences interfere, order effects apply and indicate dissonance.

Mathematically, two conflicting privacy decision situations are modeled as two observables, or properties. The outcome state of a sequence of successive privacy decisions is determined with the decision made but not prior to it. For instance, measuring "high valuation of privacy" and observing "generous data disclosure" are two states of two privacy decision situations A (stated preference) and B (revealed preference). By means of a numerical example representative of the privacy paradox, it was shown that the sequences AB and BA do not commute, i.e. ABp = BAp where p is a state after two sequential decision situations A and B with the potential states a1 ) high valuation and a2 ) low valuation for A, and b1 ) data disclosure and b2 ) data concealment for B. Prior to an actual decision, individuals reside in a dissonant state of potentiality, i.e. a linear superposition a1 + a2 and b1 + b2 respectively. With the decision made the tension between A and B reduces the dissonant state to p which marks an opinion change, a behavioral change, or rationalization. It was shown that A and B cannot be measured simultaneously, i.e. they are incompatible or do not commute, by proving the probability of data disclosure after data valuation to be lower than the probability of data disclosure before data valuation. In other words, order effects in sequences of successive measurements of stated and revealed preferences model the dissonant situation of individuals experiencing a tension between two conflicting though not yet determined cognitions.

Theoretically, cognitive dissonance offers a new explanation of why people disclose data they way they do by arguing that privacy preferences are indeterminate and dissonant up to the point in time a decision is made. Stating preferences and revealing preferences about personal data are not interchangeable. Instead observations of stated and revealed preferences interfere, i.e. order effects apply when stating preferences before revealing preferences and vice versa.

The next section discusses studies for determining privacy preferences empirically. There is a multitude of methods for measuring Willingness-to-Pay (WTP) for public and private goods. Each method only represents the attempt to come as close as possible to the truth [START_REF] Völckner | An empirical comparison of methods for measuring consumers willingness to pay[END_REF]. There is not one best method. Rather appropriateness of a method depends upon validity and feasibility with regard to a given research problem. Validity of a method for measuring cognitive dissonance in privacy decision making has to account for the context-dependent nature of privacy preferences. Preferences are not out there readily determined. In fact, preferences are never revealed but constructed with the decision made. Two basic methodological assumptions derive from the discussion so far. First, the design for measuring preferences abstains from naive realism, or the claim that true preferences are out there readily determined. Second, the design assumes that observing order effects is a necessary condition for dissonant privacy decisions.

Observations of Privacy Preferences

To determine preferences of individuals about products and services Willingnessto-Pay (WTP) measures the value someone attributes to a material or ideal good. Generally, direct and indirect WTP methods are distinguished [START_REF] Miller | How should consumers' willingness to pay be measured?: An empirical comparison of state-of-the-art approaches[END_REF]. The former ask for the price someone would be willing to pay for a good. This could be the maximum price someone is willing to pay or the minimum price someone would charge for a good. Indirect methods survey participants about the legitimacy of a given price for the product in question. The problem with direct methods is the variability of preferred prices as individuals may differ substantially in their valuation of goods. Asking for the legitimacy of a given price bears the problem that answers provide less information about the true value individuals attribute to the product. Generally, WTP methods strive for incentive compatibility, i.e. subjects ought to reveal truthfully any private information about their willingness to pay for a good without taking strategic considerations into account, e.g. concealing private information now may bring bargaining advantages in the future. Thus incentive compatibility presupposes preferences to be out there readily determined for the subject though concealed for strategic reasons.

Determining WTP for privacy necessitates special consideration. Privacy is not a tangible good traded on designed or pre-configured markets. Rather it is a derived human right and treated differently with dependence upon culture and political regime. For instance, in Germany, privacy is often associated with the right to informational self-determination, i.e. the right of individuals to control, edit, manage, and delete their personal data and decide when, how, and to what extent it is communicated to others [START_REF] Westin | Privacy and Freedom[END_REF]. Hence privacy is more of a public good like environmental prevention or protection from crime provided by the administration though, like property rights, it is tightly linked to each individual in a society. The value individuals attribute to their personal data is a good indicator for their perceived importance of privacy. Since people do not trade personal data like they buy cars or pay for clothes but rather disclose personal data in exchange for using free (Internet) services WTP determination is somewhat different compared to classical goods or resources. In the context of privacy research Willingness-to-Accept (WTA) payments for personal data and Willingness-to-Protect (WTPr) personal data are common variants of WTP. WTA observes the value someone would accept from a third party for making use of his personal data. The value someone would pay for data protection is measured by WTPr.

In the context of Online Social Networks, Krasnova et al. ( 2009) conducted a conjoint analysis to determine WTP of users being assured that the provider does not make use of their personal data [START_REF] Krasnova | Investigating the value of privacy in online social networks: Conjoint analysis[END_REF]. The authors abstain from incentive compatibility and ask indirectly about stated privacy preferences of network users. From the background of their knowledge that the provider does not make use of personal data, users' average willingness to pay a monthly fee for continuing using the network was found to lie between e 14.40 and e 17.24.

Beresford et al. ( 2010) conducted a WTA study and showed that participants were willing to accept even small amounts of money for the disclosure of their data [START_REF] Beresford | Unwillingness to pay for privacy: A field experiment[END_REF]. Participants were obliged to buy a product online and with disclosure of personal data they received a small discount. Receiving a discount for data disclosure determines WTP indirectly. Obligations to actually buy a product generate incentive compatibility.

Bauer et al. ( 2012) determined WTPr or the price someone would pay for data protection [START_REF] Bauer | On the value of information -What Facebook users are willing to pay[END_REF]. The authors conducted a study according to the BDM method [START_REF] Becker | Measuring utility by a single-response sequential method[END_REF]. Participants received information about a fictive scenario where all Facebook users were threatened by the loss of their profile and asked directly for their WTPr personal data. Incentive compatibility was implemented by means of a lottery drawing. [START_REF] Acquisti | What is privacy worth?[END_REF] examined participants receiving shopping vouchers where the value of each voucher depended on personal data disclosure [START_REF] Acquisti | What is privacy worth?[END_REF]. Participants were offered a price for their personal data. The more they were willing to disclose information the higher was the voucher value. According to this strategy participants were indirectly asked about their WTA payments for personal data and incentivized by the possibility to actually redeem vouchers. Interestingly, the authors were able to show that WTA payments for personal data is higher than WTPr personal data. Data protection and data sales are two different scenarios and with dependence upon contextual situation people value their data differently. Likewise, it appears reasonable that stated and revealed preferences are contingent upon observational context.

In summary, there are direct and indirect methods as well as incentive compatible and incentive incompatible means for determining privacy preferences. Direct methods are cognitively more demanding as personal data is not a tradable good. Participants may not have a clue about the monetary value of their data. Moreover, for measuring cognitive dissonance the monetary value is not of interest. What counts is the deviation between stated and revealed preferences. Obligations to buy products under consideration of discounts, vouchers, or other benefits offered in exchange for data disclosure create incentives to re-veal true preferences. However, in most economic transactions and even more in Online Social Networks or other data sharing scenarios based on reciprocity, data disclosure is a side effect and there is no explicit trade-off between product and data. Incentive compatible methods create a highly artificial observational context and it is questionable if obligations trigger the revelation of true preferences. From the assumption of preferences being constructed with the decision made incentive compatibility appears even more suspicious. Last but not least, some authors argued that auctions and lotteries bear understanding problems and, instead of avoiding strategic valuations, they cause strategic behaviour [START_REF] Schreier | Auktionen versus Lotterien: Ein empirischer Vergleich zur Messung von Zahlungsbereitschaften[END_REF].

Order Effects in Observations of Preferences

In many disciplines such as linguistics or cognitive psychology it is well known that the meaning of words or concepts depend on the context of their usage. Homonyms such as apple are good examples of context-dependent semantics. In the context of a eating a fruit apple has a different meaning compared to the context of a using a computing device. In his Philosophical Investigations Wittgenstein famously noticed that language has more than a denotative function. Speaking words or asking questions is always part of an activity, or form of life [21]. Likewise, determining WTP through surveying participants is a highly context-dependent activity. Here it was shown that the order of questions posed in a survey changes answering behavior of participants [START_REF] Bateman | Ordering effects in nested 'top-down' and 'bottom-up' contingent valuation designs[END_REF]. In other words, WTP is influenced by order effects which are expected to be a rule rather than an exception [START_REF] Atmanpacher | Order effects in sequential measurements of noncommuting psychological observables[END_REF].

In several WTP surveys observations of proximal and distal order effects have been made [START_REF] Cai | Distal order effects in stated preference surveys[END_REF]. Proximal order effects influence WTP according to the sequential measurement of successive questions. Distal order effects influence WTP due to information provided, or questions asked, outside of the WTP questions posed. For instance, asking participants about the price they would charge for disclosing their address details prior to asking them for the price they would charge for revealing their income level is supposed to reveal proximal order effects. WTP is expected to differ from the WTP where questions are ordered the other way round. If participants assume that income level will be associated with address details their valuation is expected to differ from the situation where they do not draw this association.

Distal order effects stem from information provided prior to WTP questions, or demographic questions determining participants' age and gender. For instance, values for WTA or WTPr of users in Online Social Networks are supposed to differ with respect to a scenario where the network provider is named, e.g. Facebook, or a scenario where the provider is kept incognito.

In the literature several attempts have been made to design surveys in a way that prevents order effects. [START_REF] Bateman | Visible choice sets and scope sensitivity: an experiment and field test of study design effects upon contingent values[END_REF] ask participants to value product bundles where each bundle is ranked according to its size [START_REF] Bateman | Visible choice sets and scope sensitivity: an experiment and field test of study design effects upon contingent values[END_REF]. Top-down rankings list large bundles on top, whereas bottom-up rankings show products descending from small to large bundles. The authors argued that WTP is higher for product bundles following up on smaller bundles. Vice versa, WTP is lower for bundles valued after the previous judgment of a larger bundle. A potential explanation of such proximal order effects is a successively changing visible choice set. A visible choice set is a complete list of goods participants have in mind during the whole surveying process. The direction of goods to be valued, i.e. top-down or bottom-up, but also changing number and type of goods, influences perceptions of visible choice sets. To prevent proximal order effects occurring with dependence upon changing visible choice sets (stepwise disclosure), participants should be informed about all products to be valued prior to revealing their WTP (advanced disclosure).

As a mean to avoid order effects a priori advanced disclosure substantiates the role of knowledge about the good to be valued. Several studies witness a decreasing likelihood of order effects when participants are used to the particular good in question [START_REF] Clark | The causes of order effects in contingent valuation surveys: An experimental investigation[END_REF]. For instance, [START_REF] Boyle | The role of question order and respondent experience in contingent-valuation studies[END_REF] found that WTP of people inexperienced in white-water rafting is prone to order effects while WTP of rafting professionals is not [START_REF] Boyle | The role of question order and respondent experience in contingent-valuation studies[END_REF]. Kartmann et al. (1996) proved the absence of order effects in surveys where patients had to state WTP for treatments they were familiar with [START_REF] Kartman | Valuation of Health Changes with the Contingent Valuation Method: A Test of Scope and Question Order Effects[END_REF]. Transferred to the privacy paradox, familiarity with one's own privacy preferences is likely to have an impact on cognitive dissonance. If advanced disclosure indeed influences privacy preferences remains an empirical question. [START_REF] Bateman | Ordering effects in nested 'top-down' and 'bottom-up' contingent valuation designs[END_REF] argued that in sequential answers to successive WTP questions order effects are likely to occur if participants state their willingness to pay for additional or extra features of previously judged goods [START_REF] Bateman | Ordering effects in nested 'top-down' and 'bottom-up' contingent valuation designs[END_REF][START_REF] Bateman | Visible choice sets and scope sensitivity: an experiment and field test of study design effects upon contingent values[END_REF]. Such inclusive lists present goods not as alternatives to be valued independently of each other, but as associated features where the value depends on previous judgments of related goods. Exclusive lists are countermeasures and present goods independently of each other. In other studies [START_REF] Kumar | Attribute order and product familiarity effects in decision tasks using conjoint analysis[END_REF] and [START_REF] Chrzan | Three kinds of order effects in choice-based conjoint analysis[END_REF] conducted a conjoint analysis where product profiles are presented with varying attributes [START_REF] Kumar | Attribute order and product familiarity effects in decision tasks using conjoint analysis[END_REF][START_REF] Chrzan | Three kinds of order effects in choice-based conjoint analysis[END_REF]. The authors showed that the order of product attributes influences WTP.

Batemann et al. (2001) and

As discussed by [START_REF] Cai | Distal order effects in stated preference surveys[END_REF] distal order effects have an impact on survey results [START_REF] Cai | Distal order effects in stated preference surveys[END_REF]. Information and descriptions provided prior to, or outside of, WTP questions is subject to varying interpretations. Therefore, the authors suggest randomizing all the information provided in the survey. Others found incentive compatible methods to be susceptive to distal order effects. [START_REF] Clark | The causes of order effects in contingent valuation surveys: An experimental investigation[END_REF] argued that despite the purpose of the BDM method to guide participants in revealing their true WTP, often, people are misguided in understanding that stating their true WTP is in their own best interest [START_REF] Clark | The causes of order effects in contingent valuation surveys: An experimental investigation[END_REF]. Enlightening participants about what is best for them is a difficult exercise as people are quite unique in their ability to learn and apprehend the purpose behind inquiry. Teaching participants to act compatibly with the incentive provided is hard to control. Similar issues arise for auction mechanisms. Avoiding distal order effects in Vickrey auctions is difficult since explaining incentives behind second-price sealed-bid, i.e. exhausting WTP, also requires participants to learn and apprehend the pur-pose behind inquiry. Moreover, if participants interact among each other during auction experiments distal order effects are even harder to avoid.

A Design for Observations of Dissonant Privacy Preferences

Based on the discussion of WTP studies and order effects in the previous sections a design for measuring preferences in privacy decision making is developed in the remainder 1 . Individuals experience dissonant states if data valuation (stated preferences) and disclosure behaviour (revealed preferences) do not commute. Two basic methodological assumptions derive from the discussion so far. First, the design for measuring preferences abstains from naive realism, i.e. the claim that true preferences are out readily determined. Second, the design assumes that observing order effects is a necessary condition for dissonant privacy decisions. There are four questions A, B, C, and D. Questions A, C determine stated preferences for two similar scenarios. Likewise, questions B, D measure revealed preferences. A and B (scenario Facebook) measure the same type of data, i.e. name, address, and phone number, whereas C and D (scenario Amazon) also ask for the same type of data, i.e. bank account, monthly income, and credit card information. All questions are assigned to two different tasks. Task 1 comprises questions A and B (scenario Facebook); task 2 is composed of questions C and D (scenario Amazon). Each task represents a sequential order of two successive questions. There are four treatment groups each of which is assigned two tasks where the order of the two questions within each task either starts with a stated preference question or a revealed preference question dependent upon the other tasks. For instance, participants in treatment group 1 start in task 1 with a stated preference question and in task 2 with a revealed preference question. The variation of questions and tasks is meant to counteract order effects. The reason for having four questions, or two scenarios (Facebook and Amazon), is that order effects can be determined for each individual of a treatment group. With only two questions, or one scenario (Facebook or Amazon), determining order effects would be possible solely between individuals of different groups. Individuals of one group would face only one context, i.e. either AB or BA (cf. section 2).

Drawing from other studies observing privacy decision making, stated privacy preferences are best determined by surveying people about the degree to which they agree or disagree with a third party making use of their personal data (agreement). The degree to which people are concerned or agree with a third party constraining their right to privacy by making use of their data accommodates the fact that privacy is not a tradable good. Surveying participants indirectly appears reasonable since the real monetary value someone would charge a third party for making use of his personal data is not of interest. Revealed preferences are determined by measuring the degree to which someone complies with data disclosure as a necessary requirement for continuing using a service after terms and conditions have been changed (compliance). The degree of compliance mirrors the attractiveness of the service and the willingness to disclose personal data in exchange for using the service. There are several reasons for not choosing an incentive compatible method to determine revealed preferences. As discussed in section 3 obligations to buy a product in line with discounts or vouchers received in exchange for data disclosure creates an artificial context. Moreover, as discussed in section 4, incentive compatible methods are prone to order effects due to their attempt to communicate the incentive behind inquiry. Since participants differ substantially in their ability to learn and comprehend the purpose of determining their true WTP, distal order effects would be introduced into the survey design. Apart from incentives, surveying participants indirectly appears also reasonable here. The real monetary value for continuing using the service is not of interest and variability is reduced. Question B: Suppose Facebook changes its terms and conditions. To continue using the service all members are required to disclose their personal data truthfully. In case of false information the provider will close the respective account. Would you disclose the following data truthfully in order to continue using the service? [1: No -7: Yes] 1. phone number 2. address details 3. name Question D: Suppose Amazon changes its terms and conditions. To continue shopping online on Amazon members are required to disclose financial data. In case of false information the provider will close the respective account. Would you disclose the following data truthfully in order to continue using the service? [1: No -7: Yes] 1. bank account 2. monthly income 3. credit card information

The following measures are applied to deal with order effects. First, participants accessing the survey are randomly assigned to one of the four treatment groups.

According to the order of tasks as shown in Table 1 participants state and reveal their privacy preferences. Randomization of group assignments is meant to avoid order effects built into the design. Order effects can be determined for each individual by comparing WTPs for both scenarios, or they are observed on an aggregate level for each group. Additionally, WTP is compared among the four treatment groups. Second, before stating and revealing privacy preferences, participants receive information about the independence of each of the questions. Avoiding inclusive lists and semantic interdependence of questions explicitly counteracts order effects. Third, prior to WTP questions participants receive information about the process of the survey, i.e. information about the number of questions and what the questions are actually about. Advanced disclosure counteracts proximal order effects which are due to diverging visible choice sets. Finally, before the WTP questionnaire starts, participants' experience with the services is determined by asking for their familiarity with the subject. In the analysis phase familiarity as well as demographics will help clustering data according to their impact on order effects.

Conclusion and Future Work

Disclosure and dissemination of data has gained momentum. Incentives for using Internet services are tremendous and most Web users would not like to live without being online anymore. Nevertheless, most Internet users are deeply concerned about their digital future and the fate of their personal data. Being dissonant with regard to attitudes, opinions, and behaviors is omnipresent and not unusual for digital citizens. Here it was argued that cognitive dissonance accounts for the privacy paradox and that it can be measured in terms of order effects in sequential observations of successive privacy decisions. Existing WTP studies for determining the value of privacy have been discussed as well as design principles to avoid order effects built into the survey design. Eventually, a design artifact for measuring order effects in sequential privacy decisions has been proposed and implemented. Conducting the survey and presenting analysis results is planned for the near future.

The implications for follow-up studies are striking. Determining the degree of dissonance opens the arena for studies on the effectiveness of Transparency Enhancing Technology (TET). The guiding design principle for TET 2.0 to influence cognitive dissonance may become incentives, e.g. recommendations or search results, which do not fit putative preferences derived inferentially based upon individual and collective online behavior [START_REF] Celko | Die Echokammer. Abstrakt: Die grosse Gemeinschaft[END_REF]. Large incentives keep cognitive dissonance low and thus may hinder change instead of reinforcing it. Accordingly, effectiveness of TET 2.0 hinges on the right magnitude of incentives imposed upon individual online behavior.

  Question A: I am concerned about the following data being used by Facebook for commercial purposes. [1: I disagree -7: I strongly agree] 1. phone number 2. address details 3. name Question C: I am concerned about the following data being used by Amazon for commercial purposes. [1: I disagree -7: I strongly agree] 1. bank account 2. monthly income 3. credit card information

Table 1 .

 1 Assignment of WTP Questions and Tasks to Treatment Groups

	Treatment 1	Task 1	Question A Question B	Treatment 2	Task 1	Question B Question A
		Task 2	Question D Question C		Task 2	Question C Question D
	Treatment 3	Task 2	Question C Question D	Treatment 4	Task 2	Question D Question C
		Task 1	Question B Question A		Task 1	Question A Question B

A first version of the survey implementation is available at https://www.elearning. uni-freiburg.de/ofb/iig/, password: abba