Stefan Berthold
email: stefan.berthold@kau.se

Towards a Formal Language for Privacy Options

Describing complex ideas requires clear and concise languages. Many domains have developed their specific languages for describing problem instances independently from solutions and thus making a reference model of the domain available to solution developers. We contribute to the zoo of domain-specific languages within the privacy area with a language for describing data disclosure and usage contracts. Our Privacy Options Language is defined by a small number of primitives which can be composed to describe complex contracts. Our major contribution is the notion of contract rights which is based on the notion of obligations and therefore establishes both concepts as first-class language citizens in a new coherent model for privacy policy languages. Our model overcomes the traditional separation of the right and obligation notions known from access control based policy language approaches. We compare our language to the PrimeLife Policy Language and provide rules for the translation from our language to PrimeLife's language. Then, we present a canonical form of our contracts. It is used to ensure that contracts with equal semantics have the same syntax, thus eliminating the possibility of a covert channel in the syntax revealing information about the originator. Finally, we show different ways of how to extend our language.

Introduction

Informational self-determination means that individuals have control over their personal data. Control in this context particularly means that individuals can access, correct, and possibly delete their personal data stored in, e. g., customer databases of companies or institutions, and object to disclosure of their data to third parties. Control, however, can also mean the possibility to agree on data disclosure in exchange for a reasonable compensation.

An interesting idea has been proposed by Laudon [START_REF] Laudon | Markets and privacy[END_REF]: in his scenario, a regulated "national information market" is the only place where personal information is traded between institutions, their customers, and third parties. The market thus becomes a single point of control where individuals and institutions can exercise control and claim their rights on an equal level. This approach is particularly interesting since it uses the strengths of three different fields in order to obtain privacy: regulation for creating a safe environment where crime can be sued after the fact, technology for authorisation of data usage, and a market for determining fair prices for the data. Laudon assumes that customer data is stored by companies and institutions which may wish to use it for different purposes later. Taylor [START_REF] Taylor | Consumer privacy and the market for customer information[END_REF] names individual pricing and targeted advertising, among others, as reasons for letting stored customer (and consumer) data become highly valuable for companies and institutions. In order to keep in line with the legislation, institutions that intend to re-use data for another purpose have to return to the individual who will have to give consent and will claim compensation for the new usage of his data. Again, an ideal market would determine a fair price for the data.

These scenarios share the notion that personal data may be used long after its disclosure. Taking the view of Laudon [START_REF] Laudon | Markets and privacy[END_REF], individuals should receive a compensation depending on the benefit an institution gains by using the individual's personal data. Determining the value of a fair compensation is, however, not necessarily easy [START_REF] Acquisti | Protecting privacy with economics: Economic incentives for preventive technologies in ubiquitous computing environments[END_REF]. In particular, if we cannot assume that individuals can directly control the use of their data after the disclosure, they have to anticipate the consequences of the data disclosure at the time of the disclosure. Part of this problem has been discussed by Berthold and Böhme in [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF].

An important means for anticipating the consequences of data disclosure is an unambiguous language for describing all rights and obligations connected to the data disclosure. Such a language can be used by both, the individual that discloses data and the institution that receives the data. The individual will use the language for determining clear conditions of the data usage, e. g., limiting it to a specific purpose and possibly to a time frame. The institution can use the language as a management reference that determines under which conditions the data may be used for specific purposes and when data may not be used anymore. Given an appropriate legal framework (e. g., the one suggested in [START_REF] Laudon | Markets and privacy[END_REF]), statements of this language would even form contracts with legal rights and obligations for the individual and the institution. This idea has been extensively explored by Laudon [START_REF] Laudon | Markets and privacy[END_REF] and backed up by the results of Hann et al. [START_REF] Hann | Online information privacy: Measuring the cost-benefit trade-off[END_REF]. In more recent work, Berthold and Böhme [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF] elaborate on the similarities of contracts and data disclosure. The concrete specification of a suitable contract language for this purpose, however, is to the best of our knowledge still an open research question.

In order to fill this gap, we will turn our attention to specify a formal language for data disclosure contracts. The following requirements shall be met by our language (hence referred to as POL):

expressive power to capture the notion of Privacy Options [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF] expressive power to capture existing approaches such as "Sticky Policies" as used in the PrimeLife Policy Language (PPL) [6] easy extensibility and scalability (of syntax and semantics) Our approach is driven by the privacy measurement perspective which is mainly presented in [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF] while the notion of Sticky Policies in PrimeLife's PPL [START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF] has mainly developed with the (identity) management perspective in mind. Both approaches thus target similar goals. The major difference between both approaches is that PPL is an access control language, thus specifying which ac-cesses are allowed and which not, and POL is a contract language, thus specifying commitments to data disclosure and usage. Moreover, the strict separation on a semantical as well as a syntactical level between rights and obligations known from PPL does not exist in our language. POL is rather building the notion of rights on the notion of obligations.

The rest of this paper is structured as follows. Section 2 defines the language primitives and demonstrates how these primitives can be combined to contracts. In Section 3, we define an operational semantics for POL which describes how the language can be translated into specific data management actions in order to satisfy the contract. In Section 4, we discuss how and to which extent it is possible to translate POL to PPL. A dialect of POL, POL -, is defined and translation rules are given. In Section 5, we present a canonical form of POL contracts. An abstract rewriting system is defined which translates any POL contract to its canonical form. In Section 6, we show how to extend POL in different ways. Section 7 concludes this paper.

Privacy Option Language

We are indeed not the first specifying a formal language. Tool support for language specification has been grown large in recent years and so has the number of domain-specific languages. In order to avoid redundancy with previous work, we build upon an existing language for describing financial contracts, proposed by Peyton Jones and Eber [START_REF] Jones | How to write a financial contract[END_REF], and adapt it for our purposes. The domain of financial contracts turns out to be quite similar to the one of Privacy Options [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF]. Thus, we refer to our language as Privacy Option Language (POL).

Like the language in [START_REF] Jones | How to write a financial contract[END_REF], our language consists of a small number of primitives1 with basic semantics. For instance, a contract c 1 that settles the immediate usage of personal data a 1 for purpose p 1 can be written as c 1 = data a 1 p 1 . This describes the rights and obligations of one contract party and let us assume that this is the institution that receives the data. The contract c 2 written by the individual who discloses the data is the 'negation' of c 1 ,2 thus, we can write it as a function application with another primitive, give, c 2 = give c 1 . We define give such that c 2 itself is a contract as well as c 1 , thus, we can understand give as a primitive that transforms one contract to another one.

The other primitives of POL are syntactically equivalent to the language defined by Peyton Jones and Eber, i. e., -(c 3 = c 1 'and' c 2 ) and (c 4 = c 1 'or' c 2 ) 3each transforms two contracts into a single one, and requires that both con-tracts are executed and or requires that, no matter which of the two contracts will be executed, the other one may not be executed, gives the holder the right to acquire c 1 once (but not necessarily the first time) until the condition b is satisfied the first time, and c 10 = zero is a contract without rights and obligations.

-c 5 = if b c 1 is equivalent to c 1 if
All transformed contracts c i with i = 2, . . . , 9 are contracts like c 1 and c 10 and can thus be part of new transformations. A simple Privacy Option in which an institution acquires the right (but not the obligation) to use the data a for purpose p at time t can thus be stated in POL as c 11 = when (at t) (data a p 'or' zero). A strict time constraint as in c 11 is relevant when an institution that obtained data from an individual cannot use the data without interacting with the individual. Consider, for instance, a scenario where a shop can only use the purchase history of a customer when the customer returns to the shop [START_REF] Acquisti | Conditioning prices on purchase history[END_REF][START_REF] Böhme | Pricing strategies in electronic marketplaces with privacyenhancing technologies[END_REF].

The right to choose the zero contract gives the shop the option not to use the data, e. g., if the costs connected to the usage are higher than the expected benefits. Other contract combinations are conceivable. For instance, c 12 = until (at t) (data a p 'or' zero) models the obligation to delete data after a deadline t. 5

POL Contract Management

Peyton Jones and Eber demonstrate how various semantics can be defined upon their language syntax. They elaborate on a valuation process E, a denotational semantics which assigns a value to each contract. Valuation is better known as privacy measurement in the privacy domain. For many of the language primitives in POL, defining the valuation process would be a straight application of [START_REF] Jones | How to write a financial contract[END_REF] and [START_REF] Berthold | Valuating privacy with option pricing theory[END_REF]. Due to that and space constraints, we focus here on an operational semantics for managing Privacy Options specified in POL. In this context, management means what is usually referred to as back-office management of contracts, i. e., timely execution, simplification, and possibly even deletion of outdated contracts.

Definition 1 (Contract management).

A contract management function is a (total) function from time to a sequence of I/O operations and a possibly simplified contract, thus satisfying the (informal) type definition

ACT ION = Date → IO Contract . (1)
Note that in the type of contract management functions, IO Contract, is a monad [START_REF] Wadler | The essence of functional programming[END_REF] which allows returning a sequence of I/O operations as well as a contract as function results.

The management semantics defined in Figure 2 describes which operations take place depending on a given contract in POL and relies on the definition of a couple of auxiliary functions: -

OE give c = send(c) (O3) OE c1 'and' c2 = OE c1 OE c2 (O4) OE c1 'or' c2 = greedy E (OE c2 , OE c1 ) (O5) OE if o c = ifthenelse(V o , OE c , return(zero)) (O6) OE ifnot o c = ifthenelse(V o , return(zero), OE c ) (O7) OE when o c = when E (V o , OE c ) (O8) OE anytime o c = stopping E (V o , OE c ) (O9) OE until o c = absorb E (V o , OE c ) (O10)

Converting POL Contracts to PPL Sticky Policies

POL and PPL are both expressive languages, but of different nature, i. e., PPL is an access control language, whereas POL is a contract language, and none of them subsumes the semantics of the other one to its full extent. We therefore refrain from translating the full-featured POL to PPL, but instead define a subset of our language, POL -, and provide a mapping from this subset to PPL. PPL Sticky Policies are an extension of XACML 3.0 [START_REF] Rissanen | OASIS extensible access control language (XACML) version 3.0. OASIS working draft 10[END_REF], a powerful XML dialect for defining access control policies. While the complete XML Schema for PPL is yet not settled, the general language model is specified in [START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF]. Figure 3 displays the PPL elements which are relevant in this section. In POL, we know three primitives that change the availability of data over time. PPL can simulate one of them, until, i. e., it is possible to delete data by means of triggering an obligation after a certain time. The other two timedependent POL primitives, when and anytime, require to make previously unaccessible data available. In the current version of PPL [START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF], this can neither be achieved by triggering (a sequence of) obligations nor by defining Sticky Policies. The POL primitives when and anytime will therefore not be part of POL -.

Three further assumptions are made for POL -and its translation to PPL:

A1. The conditions of the POL -primitives until, if, and ifnot are limited to points in time and can therefore be used as triggers of PPL Obligations (Trigger at Time [START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF]). A2. We define a new Action, immediate usage, for PPL Obligations which causes the data to be used instantly. A3. PPL Actions triggered by Obligations are always permitted.

Let us start with three simple examples:

-The POL contract c 10 = zero is a contract without rights or obligations. It therefore translates to an empty PolicySet in PPL. -The POL contract c 1 = data a 1 p 1 requires the immediate use of the data item a 1 for purpose p 1 . After that usage, the data may not be used again.

After the translation, the purpose p 1 will be enforced by a StickyPolicy (nested in a Policy within a PolicySet). The immediate usage of the data can only be enforced by an Obligation attached to the StickyPolicy. The

Obligation will be triggered when data is stored 6 and cause the Action immediate usage. Another Obligation will make sure that the data is used only once. It will be triggered on data usage7 and will cause the data to be deleted instantly. -For translating a complex POL contract, e. g., c 13 = data a 1 p 1 'and' data a 2 p 2 , we first translate the data statements to PolicySets, wrap them in a new PolicySet and add two Obligations, both triggering on data usage of either a 1 or a 2 . 8 The idea is to cause the Action immediate usage of one data item as soon as the other one is used.

While the first two examples already provide general translation rules, the third is a special case of and in which the left and the right hand side are data statements. A general translation rule can, however, easily be derived by replacing the two Obligations in our example, which were tailored to the data primitive, with obligations that trigger other primitives accordingly.

The key idea outlined in these three examples is that every primitive of POL - can be translated to a PolicySet in PPL which may contain nested PolicySets. A translation of POL -to PPL will therefore form a tree of PolicySets which reflects the POL language structure as outlined in Figure 1, with one exception: the primitive give simply vanishes in the translation to PPL, since policies in PPL have the same appearance, no matter whether stored on the data subject's side or on the institution's side.

The translation of the remaining POL -primitives is straight-forward. Just like and, c 4 = c 1 'or' c 2 can be translated by first translating c 1 and c 2 to PolicySets, placing them into a new PolicySet, and complementing them with an Obligation which in case of or will delete the data in c 1 and c 2 as soon as an on data usage event occurs with regard to data of either c 1 or c 2 .

The translation rule for a contract c 9 = until b c 1 similarly first translates c 1 to a PolicySet, wraps it in a new PolicySet, and complements it with an Obligation which triggers when the condition b becomes true. Due to our assumption A1, this can be realised by the PPL Trigger at Time. As soon as the Obligation is triggered, it will delete the data.

Finally, translating c 5 = if b c 1 or c 6 = ifnot b c 1 respectively requires to translate c 1 to its corresponding PolicySet, placing it into a new one, and complementing it with Obligations such that the data items in c 1 are deleted if b is false (if) or true (ifnot), respectively.

We used data deletion in order to enforce the semantics of POL primitives in their PPL translation. This works well as long as the data items are not equal in different branches of the PolicySet tree. If this is not the case, PPL needs a method to (de)activate Policies depending on a (global) state. Note that if we use anonymous credentials [START_REF] Chaum | Security without identification: Transaction systems to make big brother obsolete[END_REF][START_REF] Camenisch | An efficient system for non-transferable anonymous credentials with optional anonymity revocation[END_REF] for representing data item, we do not need any of these assumptions. Then even credentials representing the same data would not be linkable and deletion would only affect one occurrence of the data item.

Canonical Form Contracts in POL

POL is not only an effort in language design, but primarily one in the privacy research domain, and it is important that we do not create new problems in one domain when solving problems in the other. A problem that therefore deserves our attention is that of (unintentionally) creating covert channels by using freedom in the POL syntax for expressing semantically equivalent contracts. Such a covert channel may be used to transport any information, but even if not used intentionally it will most likely convey information about the originator of a contract. In this section, we show that POL has freedom in its syntax and how to eliminate it. We show that each POL contract can be transformed to a canonical form by using the abstract rewriting system we define.

An example illustrates the syntactical freedom which exists in POL. The contracts on the left and right sides of the Equations ( 2) and ( 3) are semantically equivalent (we use the symbol ≡ POL ), however the syntax is different,

give zero ≡ POL zero , (2)
give (give (data a p)) ≡ POL data a p .

The degree of freedom increases when contracts become more advanced. We see that transformations to equivalent contracts follow generic transformation rules, e. g.,

(
a 3 p 3 )))) . (4)
In fact, the contract c 10 = zero has an infinite number of incarnations, thus giving each contract originator the option to use a unique version of c 10 . For our canonical form it would be best, if all incarnations of c 10 could be reduced to the simplest version, zero.

A standard method for reducing a language term to its canonical form is applying an abstract rewriting system (ARS). An ARS consists of a number of reduction rules which can be applied in any order to a given language term. Terms that cannot be reduced any further are in canonical form.

For POL, a suitable ARS is defined by the reduction rules in Figure 4. In Equation ( 4), we use the same set of rules to transform the initial contract to its equivalents. Two properties of ARS' are interesting for our canonical form of POL and we will look at them in the rest of this section:

-Termination: Given a POL contract, all applicable reduction rule sequences are finite, i. e., finish after a finite number of reduction steps with a canonical form contract which cannot be reduced any further. -Confluence: Given a POL contract, all applicable reduction rule sequences produce the same canonical form contract.

It is easy to see that for termination, an ARS may not contain reduction rules supporting commutativity. Let us assume that we have a rule c 1 'and' c 2 → c 2 'and' c 1 and c matches the rule's left hand side. Then, c can be reduced to the right hand side c , thus c → c , but then we see that c matches the left hand side of the same rule again and we can create an infinite sequence of reductions c → c → c → c → • • • . This would violate the termination property.

The non-commutativity of and and or, however, may interfere with the application of the Rules (R6) and (R7). We learn, for instance, from Rule (R7) that a contract (c 1 'or' c 1 'or' c 2 ) can be reduced to (c 1 'or' c 2 ), whereas the contract (c 1 'or' c 2 'or' c 1 ) is not reducible in the same way, though being equivalent except for the commutation of the contracts c 1 and c 2 . This dilemma can be solved by assuming an appropriate order of contracts in the function arguments of and and or such that Rule (R6) and (R7) matches if matching is possible with any permutation of the function arguments.

Termination can be proved by showing that a well-founded order POL exists such that POL r holds for all reductions where denotes the left hand side of the rule and r denotes the right hand side. For POL, this order relation can be defined by means of the order ≥ on the natural numbers,

c POL c ⇐⇒ m(c) ≥ m(c ) (∀c, c ∈ POL) , (5)
with the function m : POL → N defined as in Figure 5. It is easy to see that for any contract c it holds that m(c) > 1 and for any instantiation of a reduction rule in Figure 4 it holds that m( ) > m(r) with being the left hand side of the reduction and r being the right hand side. Thus a reduction sequence for a POL contract c finishes with a canonical form at latest after m(c) -2 reduction steps which proves that our ARS has the termination property.

Confluence of a terminating ARS can be proved by showing local confluence for all contracts, i. e., showing that all two different (one-step) reductions of one contract always terminate in the same canonical form. We will not explicitly prove confluence in this paper and postpone this task to future work. In careful checks, however, we were not able to produce a contract which violates the conditions for confluence. m :

if o (c1 'or' c2) → (if o c1) 'or' (if o c2) (R17) ifnot o (c1 'or' c2) → (ifnot o c1) 'or' (ifnot o c2) (R18) if o (if o c) → if o c (R19) if o (ifnot o → zero (R20) ifnot o (if o c) → zero (R21) ifnot o (ifnot o c) → ifnot o c (R22) if o (when o c) → if o c (R23) if o (until o c) → zero (R24) ifnot o (until o c) → until o c (
                                       zero → 2 data a p → 2 give c → m(c) + 1 c 'and' c → m(c) • m(c ) c 'or' c → m(c) • m(c ) if o c → m(c) 3 ifnot o c → m(c)

Extensions

There are two possible ways of extending the language: the first option is adding new primitives to the language and defining their semantics. While this is an obvious way of extending a language, it should be reserved to experts only. Adding primitives represents the risk of creating redundancy in the semantics which can lead to ambiguous contracts. The other way of extending the language is to define combinators. A combinator creates a contract by applying primitives rather than defining new primitives. In particular, if POL is implemented as embedded domain-specific language within a more expressive host language, creating a combinator library becomes easy even for end-users. Consider, for instance, a combinator that allows to execute a given contract twice. We can define this combinator as follows: ) This definition allows us to note a new contract, c 14 = until (at t) 'twice' c 1 , which appears like native English and provides intuitive semantics without further definitions.

twice f c = (f c) 'and' (f c) . ( C1
Assuming the full expressiveness of a multi-purpose language, here Haskell, we can even generalise twice to times, times n c = foldr1 (and) (replicate n c) , (C2) which immediately allows us to note c 15 = 5 'times' c 14 as a new contract and again with intuitive meaning. Another easy extension is the cond combinator which was a primitive in an earlier version of POL. It combines the functionality of the new primitives if and ifnot, but was discarded as a first class citizen of the language in order to keep the abstract rewriting system (Figure 4) small, cond o c 1 c 2 = (if o c 1 ) 'and' (ifnot o c 2 ) .

(C3)

While these three combinators make the language handy, they do not reflect the specific vocabulary used in the privacy research domain. One of these concepts which is easily translated to POL is the definition of a retention period for data usage. We assume that the host language provides a function now which returns the current time and define, retain t c = until (at (now + t)) (c 'or' zero) , (C4) a contract combinator which allows the holder of the contract to use the data in c (once) within the time frame t, e. g., c 16 = retain "6 months" c 1 .

Even applications of our language may look like extensions at a first glance. While POL provides, for instance, syntax for rights and obligations of contract partners, we have intentionally excluded the specification of contract partners from the scope of POL. This allowed us to focus on rights and obligations as one subject and postpone the specification of contract partners as a subject for another language which could wrap round POL. Approaching the specification of contract partners in an independent language is particularly interesting when accounting for complex contract partner relations, e. g., when two parties write a contract about the data of a third party.

Conclusions

We have specified a formal language POL for Privacy Options and added a semantics for managing Privacy Options. The management semantics allows to execute and simplify contracts. A simple modification of this semantics would switch off the simplification and retain the original contracts, e. g., for PrimeLife's Data Track.

We compare POL with PPL, a language with similar purpose, by elaborating on translation rules from POL to PPL. The translation shows that POL has its strengths in defining contracts that evolve over time. While in PPL data, once it became inaccessible, stays inaccessible forever (data deletion), we can define contracts in POL which flexibly allow data usage depending on time or events. POL is, however, not meant to be a drop-in replacement for PPL and rather focuses on a fraction of the functionality provided by PPL, i. e., Sticky Policies.

In contrast to most other approaches in the privacy policy domain, POL contracts can be transformed to a canonical form. This allows to eliminate information about the contract originator hidden in the freedom of the POL syntax. We deem that this is a particularly interesting feature in privacy negotiations when the contract proposals are evaluated by negotiation partners and none of them wants to reveal more information than the terms and conditions under which they could accept a contract.

Moreover, we have outlined how POL can be extended. Experts may add new language primitives and benefit from the inductive structure of the language, i. e., in most cases it will be sufficient to define the semantics of the added primitive in order to provide an extended language with all semantics. End-users can extract standard contract patterns and make them available to a larger audience by defining combinators, i. e., functions that combine primitives in a standardised way. Combinators may be defined in a high-level programming language or even in a visual environment, depending on the capabilities of the end-user. Neither the choice of a specific programming language, nor the choice of a suitable visual environment is determined by the work in this paper.

  the condition b is satisfied at the time of evaluation and equivalent to zero otherwise (the primitive zero is introduced in c 10 ), c 6 = ifnot b c 1 is equivalent to c 1 if the condition b is not satisfied at the time of evaluation and equivalent to zero otherwise, c 7 = when b c 1 postpones c 1 to the first time when the condition b becomes true, c 8 = anytime b c 1 gives the holder of the contract the right to acquire c 1 once (but not necessarily the first time) when the condition b is satisfied, c 9 = until b c 1

4

 4

a 1 p 1 c 1 = zero = c 10 Figure 1 .

 1101 Figure1. POL language primitives in type graph notation (left) and the contract c11 as function evaluation tree (right). The inductive definition of POL contracts is a major strength of the language.

-

  return : Contract → ACT ION The function return(c) propagates c without any further actions as a (possibly simplified) replacement of the original contract. use : PersonalData × Purpose → ACT ION The function use(a, p) lets the data controller immediately use the data item a for purpose p and returns zero, i. e., nullifies the original contract. send : Contract → ACT ION The function send(c) transmits attribute values contained in c and stores the contract c (on the individual's side) for later inspection, e. g., recording a Data Track as in PrimeLife. OE • : Contract → ACT ION OE zero = return(zero) (O1) OE data a p = use(a, p) (O2)

Figure 2 .

 2 Figure 2. Management of Privacy Options as operational semantics of POL.

Figure 3 .

 3 Figure 3. Model of the Sticky Policy and the obligation language as defined in PrimeLife [6, Figure 12]. Only elements being relevant for the conversion from POL to PPL are displayed.

  R25) when o zero → zero (R26) when o (give c) → give (when o c) (R27) when o (c1 'and' c2) → (when o c1) 'and' (when o c2) (R28) when o (if o c) → when o c (R29) when o (ifnot o c) → zero (R30) when o (when o c) → when o c (R31) when o (anytime o c) → when o c (R32) when o (until o c) → zero (R33) anytime o zero → zero (R34) anytime o (give c) → give (anytime o c) (R35) anytime o (if o c) → anytime o c (R36) anytime o (ifnot o c) → zero (R37) anytime o (when o c) → anytime o c (R38) anytime o (anytime o c) → anytime o c (R39) anytime o (until o c) → zero (R40) until o zero → zero (R41) until o (give c) → give (until o c) (R42) until o (if o c) → zero (R43) until o (ifnot o c) → until o c (R44) until o (when o c) → zero (R45) until o (anytime o c) → zero (R46) until o (until o c) → until o c (R47)

Figure 4 .

 4 Figure 4. Abstract term rewriting system for POL. The application of the rewriting rules terminates in a canonical form of the language.

3 .Figure 5 .

 35 Figure 5. The 'mass' function m : POL → N. It holds that for any reduction c1 → c2 with regard to a rule in Figure 4, m(c1) > m(c2) (c1, c2 ∈ POL).

Like Peyton Jones and Eber, we use Haskell syntax[START_REF]Haskell 98 Language and Libraries -The Revised Report[END_REF] to specify our POL. This allows to translate the ideas of this paper directly into program code.

'Negation' here means the exchange of obligation and rights, i. e., the rights of the one contract party become the obligations of the other party and vice versa.

The backticks around the functions and and or are Haskell syntax and let us use and and or in infix notation.

We assume that the function at transforms a time t into a condition suitable for the evaluation in contracts.

Here we assume that not using data after the deadline and deleting the data when the deadline occurs has the same consequences.

This trigger is can be realised by Trigger at Time or Trigger Personal Data Sent.[START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF]

In[START_REF] Raggett | Draft 2nd design for policy languages and protocols[END_REF], this trigger is called Trigger Personal Data Accessed for Purpose.

To be precise, each of these Obligations is embedded in a StickyPolicy which needs to be targeted to data a1 and purpose p1 or a2 and p2, respectively.

Acknowledgements

The author was partially funded by the Research Council of Norway through the PETweb II project. Thanks to Rainer Böhme, Simone Fischer-Hübner, Martin Günther, Stefan Lindskog, Tobias Pulls, and the anonymous reviewers of the PrimeLife Summer School 2010 for useful comments and suggestions.


FCIPS, I.S Thomas P P Keenan
email: keenan@ucalgary.ca

Oops -We Didn't Mean to Do That! --How Unintended Consequences Can Hijack Good Privacy and Security Polices

Keywords: Unintended consequences, technology policy, privacy, security, hacking, strategic foresight, policy analysis

All privacy laws, security policies, and even individual actions are subject to an often-forgotten factor -the "Law of Unintended Consequences" (LUC.) Yet LUC is not a "law" in the sense of appearing in the Criminal Code, nor is it a Law of Nature like gravity. It is actually a manifestation of our inadequate efforts at foresight, and there are things we can do to counteract it. This paper identifies classes of factors which have lead to unintended consequences in the privacy and computer security domains, though the list is by no means exhaustive. It is primarily intended to inspire further thinking and research. We clearly need to make a stronger effort to "foresee the unforeseeable" or at least "expect the unexpected" to maintain public confidence in technological systems. The disciplines of strategic foresight and automated policy analysis may prove useful in attaining this goal.

Each of these cases demonstrates one or more class of unintended consequences. Some are technical, like the web browser exploit; others are corporate and government policy decisions that were not thoroughly considered.

There is a rich body of theoretical literature on the LUC as it applies to technology, much of it from an engineering perspective. Healy discusses unanticipated consequences, i.e. "consequences which are not foreseen and dealt with in advance of their appearance. " [START_REF] Healy | The Unanticipated Consequences of Technology[END_REF] He cautions us not to confuse unanticipated consequences with undesirable or improbable ones. In an example about a nuclear power plant near an ocean, he notes that "the anticipated and intended goal or consequence is the production of electric power. The undesired but common and expected consequence is the heating of the ocean water near the plant. An undesired and improbable consequence would be a major explosion…" Examples from Chernobyl to Three Mile Island tell us that the improbable consequences should indeed be considered in a risk analysis of such a project.

In a book on this subject [START_REF] Perrow | Normal Accidents: Living with High Risk Technologies[END_REF] Perrow also discusses LUC at nuclear power plants, noting that "typical precautions, by adding to complexity may help create new categories of accidents." So, ironically, "at Chernobyl tests of a new safety system helped produce the meltdown and the subsequent fire."

This type of occurrence is what Tenner calls a "revenge effect." In his book on the subject [START_REF] Tenner | Why Things Bite Back: Technology and the Revenge of Unintended Consequences[END_REF] he uses the example of car alarms which are intended to protect vehicles from theft and vandalism. Of course they sometimes malfunction, triggering annoying false alarms with flashing lights and blaring horns. "In cities where alarms are most needed," Tenner writes, "neighbors silence malfunctioning systems by trashing cars." In other words, the technology intended to prevent car vandalism can lead to precisely that.

In the updated edition of his book on highly improbable events, Taleb [START_REF] Taleb | The Black Swan, The Impact of the Highly Improbable[END_REF] develops the concept of Black Swans, asserting that all really important discoveries and advances (e.g. the Internet) have come from events that eluded the normal prognostication techniques. So, he writes, "Black Swans being unpredictable, we need to adjust to their existence (rather than naively trying to predict them). There are so many things that we can do if we focus on antiknowledge, or what we do not know." Dörner [START_REF] Dörner | The Logic of Failure: Why Things Go Wrong and What We Can Do To Make Them Right[END_REF] provides a helpful framework for understanding why outcomes are often difficult and sometimes impossible to foresee. He notes that complexity, (system) dynamics, intransparence, ignorance and mistaken hypotheses can all play a role in preventing us from correctly foreseeing consequences. A system may simply be too complex or opaque for us to really understand it; it may be changing on its own; or we may simply have inaccurate mental models of reality which make our predictions incorrect.

It is clear that anticipating consequences is hard work, and, sometimes even impossible. Yet it is important work, because we are seeing more and more examples of negative outcomes of bad design at both the technical and policy levels. Without falling into the logical trap of trying to predict what is truly unpredictable, it does appear that there are some common factors that lead to unintended consequences in the domains of privacy, identity and security. It also seems to be helpful to draw on concepts from other disciplines to better understand these issues.

Factors That Can Lead to Unintended Consequences

One tool for understanding a complex phenomenon is to identify classes of factors which tend to contribute to it. A related technique is to reason by analogy with other concepts in other fields of human endeavor. The factors listed below, while certainly not an exhaustive list, are intended to shed some light on this type of analysis and to inspire further thinking. They draw on fields as diverse as accounting, information science, economics, and even human psychology to generate models that can be helpful in this research.

Materiality -Does This Matter to Me?

In accounting, a sum of money is considered "material" if it could, by its omission or misstatement, cause the reader of financial statements to form incorrect conclusions about the financial health of the entity. So, a few coins stuck under the cash register drawer are not material, but an unreported commitment to make a large purchase might well be. For the purposes of this paper, policies can be considered material if they have sufficient positive or negative consequences to affect the behavior of a reasonable person.

For example, in the Canadian blank media levy example noted above, the government's stated goal was to collect revenue to compensate musicians for music that was being copied onto blank media. Some argue that reducing media piracy, thereby pleasing the US government, and punishing the Canadian public for audacious copying were secondary goals. There appears to have been no serious contemplation of cross-border shopping in the legislative debate that led to the 1997 changes to Canada's Copyright Act. After the fact, the Retail Council of Canada did indeed note the damage to Canadian retailers and is now calling for the abolition of the blank media levy [6]. A similar levy in Australia was struck down by their Supreme Court. Some European countries have blank media levies, as permitted under the EU Copyright Directive of 2001 [7].

Canada's blank media levy is currently 29 cents Canadian per unit for recordable media such as CD-R, CD-RW, MiniDisc, etc. This money is placed into a fund to compensate copyright owners for their losses due to private copying of digital media such as music CDs.

The levy often exceeds the purchase price of the blank CD. Comparing US and Canadian vendors for blank Memorex CD-Rs, a US firm, Best Buy, offers a 50 CD spindle for $16.99 US. The Canadian branch of the same store sells a comparable 50 pack for $39.99. These are regular prices, not temporary sale ones. Since the currencies are currently close to par, the difference in unit price (34 cents vs. 80 cents) is largely due to the blank media levy. Because most Canadians live close to the US border, the temptation to cross-border shop is high and the risk of getting caught is very low. So consumer behavior has changed and the blank media levy is indeed material, in a way that it would not be if it were set at, say, two cents per unit.

Technology Substitution -Is There Another Way to Do This?

Consumers usually have choices, and will take their business to the vendor who gives them what they perceive as the best deal. For example, many drivers will avoid toll roads, seeking toll-free alternatives, unless the perceived value in terms of time and fuel savings exceeds the toll cost. In a similar fashion, Canadian consumers have "voted with their pocketbooks" by largely shunning the overpriced CDs in favor of other ways of storing digital content. In fact, it is difficult to find blank CDs in many general retail stores. Where did that business go? DVDs are not subject to the Canadian blank media levy, illustrating the fact that legislation almost always lags behind technology. A spindle of 50 DVD-Rs is available in Canada for $9.99, pushing the unit cost down to 20 cents each, and that's for media with almost six times the capacity of a CD! While it may be annoying to store your music on a disc that will not play in your car's CD player, people are coping by simply transferring their files from DVD to their mp3 player, or even directly to the mp3 player, thereby finding a way around the artificially high cost of CDs in Canada. The advent of cheap flash memory has altered the landscape here, with the very real possibility of storing significant amounts of content in semi-permanent solid state memory.

Of course, this has led the recording industry (the main force behind the blank media tax) to demand a new "iPod tax" to recoup their perceived losses and there is some legislative support for this [START_REF] Flatley | Is Canada's iPod Tax Back?[END_REF]. This illustrates the iterative nature of policy in which a cycle (some would say an endless and futile one) of measures are proposed, each trying to "plug the leak" discovered in the previous "solution."

The ability to substitute a functionally equivalent technology is certainly an important consideration is searching for unintended consequences, especially in the information technology domain where "a bit is a bit."

Scope conflict -My Law is Better than Your Law

An interesting situation arose after the passage of the US Health Insurance Portability and Accountability Act of 1996 (HIPPA.) It illustrates a problem when laws (or policies) are created in a vacuum, without paying proper attention to the other laws and policies that will be affected. HIPPA provides much-needed safeguards to give patients control over the use and disclosure of their medical information. However, as a US Federal law, it takes precedence over state laws including the ones that govern lawsuits brought by patients against health care providers [START_REF] Antognini | The law of unintended consequences: HIPAA and liability insurers; at first glance, the Privacy Regulations appear to be adverse to insurers and defense counsel, but McCarran-Ferguson and exceptions may save the day[END_REF]. Since the definition of a health care provider in HIPPA is very broad, the net effect of its passage was to allow patients to sue their health care provider and also demand that their information be withheld from the opposing side. The lawyers for the health care providers were thereby denied access to the very information they required to properly build their cases. For a period of time, this caused difficulty in the legal process and had to be addressed by further legislative changes [START_REF] Kapushian | Hungry, Hungry HIPPA: When Privacy Regulations Go Too Far[END_REF].

In a similar fashion, online gambling is currently illegal in the United States by virtue of the enactment of the Unlawful Internet Gambling Enforcement Act (UIGEA) in 2006. Banks and credit card issuers are banned from paying money to online gambling sites. Despite this, offshore online casinos are thriving, and have created an indirect technique called e-wallets to take money from U.S. gamblers. However, if a gambling site fails to pay a winning gambler, there is no way to sue the operators under US law since this activity is explicitly illegal under UIGEA. As Vogel points out [START_REF] Vogel | US Law Against Online Gambling Makes it the Biggest Loser[END_REF], if a person wanted to track down their winnings at an offshore casino, "the winner could go to the UK, Aruba or Bermuda, or to the locale stated in the terms of service. It may sound romantic, but it could cost more than it's worth to make the trip --not to mention payment of litigation fees." Effectively, US-based gamblers using foreign gaming sites are dependent on the honesty and integrity of the site operators as they lack legal protection under the laws of their own country.

Combinations of Information -The Devil is in the (Very Minor) Details

Increasingly, the concept of a stateless, memoryless interaction between computers on the Internet is being replaced by a complex and somewhat opaque web of identifications and quasiidentifications. Websites leave cookies, collect personal data, and services like Spokeo (www.spokeo.com) provide easy cross-correlation based on something as simple as an email address. The new web standard, HTML 5, will increase the number of places where cookies can be stored to make them even more persistent and harder to eradicate. Even apparently innocuous leftovers on your computer like the bits that govern what color a link is displayed in can provide clues as to where you've been online. This information is made available to websites via the a:visited pseudo-class, part of the CSS style sheet language that controls how text is presented.

Gilbert Wondracek and Thorsten Holz of the International Secure System Lab at the Technical University of Vienna demonstrated a way to steal the history of a user by exploiting the information that browsers use to render sites that have been previously visited in a different color than other sites [START_REF] Cameron | More Unintended Consequences of Browser Leakage[END_REF][START_REF] Wondracek | A Practical Attack to De-Anonymize Social Network Users[END_REF]. They then combine this with social network sites visited, on the theory that very few people belong to exactly the same combination of social networks. Another researcher, Joerg Resch of analyst company Kuppinger Cole used this technique to uniquely identify himself on the social network Xing and has provided an online experiment to allow others to try to do this [14]. Because of a fix applied to Xing, this technique is no longer functional, but it certainly made a point. One might also argue that if a small set of social networks become dominant, this type of uniqueness will decrease. However, there will always be some class of sites visited that is fairly unique to a specific user and could potentially be exploited.

In a similar spirit, Peter Eckersley of the Electronic Frontier Foundation (EFF) has developed this concept using the precise versions of software installed on a computer (browser, Flash version, plugins etc.) to create a "browser fingerprint." All of the information necessary to create this fingerprint is available upon request to websites visited from the browser [START_REF] Eckersley | How Unique is Your Web Browser?[END_REF]. He ran a fingerprinting algorithm on 470,161 informed participants visiting a particular EFF website. Eckersley concluded that, for this sample, "the distribution of our fingerprint contains at least 18.1 bits of entropy, meaning that if we pick a browser at random, at best we expect that only one in 286,777 other browsers will share its fingerprint." He concludes that "there are implications both for privacy policy and technical design." With an estimated 1.5 billion people using the Internet worldwide, browser fingerprinting may not identify a specific person. However, it can certainly be combined with other information to narrow the search.

Clearly the designers of browsers never intended to allow this type of de-anonymization, which is why information such as full browsing history is not passed to websites. However, they failed to realize that through the style information and the a:visited list they were essentially giving out a similar capability to anyone who was clever and devious enough to use it.

It is important to note that combinations of unintended consequence factors may come into play in a specific situation. For example, in the browser exploit example, materiality is also relevant since it would only be worthwhile to bother tracking a user if there was a purpose, such as sending targeted advertising or something more nefarious, like identity theft or blackmail.

Pricing Failure -What is the Price of a Human Life?

It is a basic tenet of economics that "everything has a cost, everything has a price." Yet, the actual implementation of that principle is often flawed. As Schumacher points out "to press non-economic values into the framework of the economic calculus, economists use the method of cost/benefit analysis…it is a procedure by which the higher is reduced to the level of the lower and the priceless is given a price. It can therefore never serve to clarify the situation and lead to an enlightened decision. All it can do is lead to self-deception or the deception of others; all one has to do to obtain the desired results is to impute suitable values to the immeasurable costs and benefits. The logical absurdity, however, is not the greatest fault of the undertaking: what is worse, and destructive of civilisation, is the pretence that everything has a price or, in other words, that money is the highest of all values." [START_REF] Schumacher | Small is Beautiful -Economics as if People Mattered[END_REF].

The 2010 oil spill off the US Gulf Coast is a stark example of the difficult of mixing monetary values (getting oil efficiently) with non-monetary ones. Long term ecological damage, injury to British Petroleum's reputation, and loss of human life clearly have symbolic and absolute values that defy translation into dollars and cents. While many observers argue that the spill was predictable, and perhaps even inevitable, few high level decision makers had considered the consequences such as possible health effects from the dispersants being used, as well as the psychological trauma on area residents.

In the realm of privacy and security policies, a dollar cost is often attributed to the compromise of private information, either by citing the amount spent by the party committing the breach to "remedy" it, (e.g. through paying for customers' credit insurance,) or the cost (in time and out of pocket expense) to the victim to reestablish their identity credentials. As one example, TJX Companies, Inc., the US-based parent of retail stores such as TJ Maxx and Winners, estimated the cost of its 18-month long privacy breach at $17 million US [START_REF] Gaudin | TJ Maxx Breach Costs Hit $17 Million[END_REF]. This line of thinking makes the implicit assumption that a victim can be "made whole" by financial compensation. Disturbing examples are arising that demonstrate that is not always possible. According to the media reports, a 24 year old teacher named Emma Jones committed suicide in February 2010 "'after (her) ex-boyfriend posted naked pictures of her on Facebook" [18]. While the man involved denies this action, relatives and acquaintances have stuck to the story that her embarrassment, combined with the fact that she feared legal action as she was living in Abu Dhabi, drove her to take her life. Clearly, for Jones and her family, this privacy breach falls into the realm of "priceless." Further confirmation that some things are considered "beyond price" comes from the outcry over online tributes to UK gunman Raoul Moat, who attacked a police officer. A Member of the UK Parliament called for the page to be taken down and "took the step of ordering Downing Street officials to contact Facebook, which has allowed 30,000 people to join a bizarre 'tribute group' glorifying Moat's crimes, to lodge a formal protest" [19]. In cases like this, financial compensation seems, as Schumacher argues, to be at best irrelevant and even crass.

Malicious and Semi-malicious Actors -An Unlocked Door Invites Intruders

No discussion of unintended consequences would be complete without acknowledging the important role of human beings in the ultimate outcome. The term "hacking" has come to refer to the malicious and often criminal activity of breaking into computer systems. However, in its original sense, hacking was a noble calling that saw highly talented people, often at universities, exploring the corners of technology for the sheer joy of it. Levy documented this very well 25 years ago and his book on the subject has recently been updated and re-released [START_REF] Levy | Hackers: Heroes of the Computer Revolution -25 th Anniversary Edition[END_REF]. Proof that this playfulness is still with us comes from the case of prankster/journalist John Hargrave [START_REF] Metzger | Prank Uses Obama in Attempt to Obtain Centurion Bling[END_REF] who successfully obtained a credit card in the name "Barack Obama" simply by phoning American Express and asking for a "supplementary card" in that name. It was duly issued and arrived in the mail.

Another example relates to the way in which airlines alter fare prices, often in the middle of the night. Clever travel agents wrote automated scripts that queried the airline computers on a continuous basis asking for specific fares on behalf of clients. This put such an unexpected load on the airline systems that they had to put limits on the number of queries made by travel agents.

Many jurisdictions have online systems to allow citizens to check their own, and their neighbors' property assessments for tax fairness purposes. Of course, these are often used for totally unrelated purposes (how much is my boss' house worth?) and this was enough of an issue that the City of Calgary had to take both legal and technical countermeasures to control it [22]. While most people who mis-used this system were, at best, "semi-malicious" in that they were driven by curiosity, at least one person objected when a newspaper published a photograph of his house as one of the most expensive in the city, citing privacy concerns and fears that he would be burglarized.

The best policy in terms of human behavior is to simply assume that if a feature exists in a system, someone will try to figure out a way to exploit it, and will often succeed.

Future technologies -Expect the Unexpected

Well beyond the scope of this paper, but impossible to ignore, is the ongoing impact of "not yet invented" future technologies. As one example, petabytes of data are being self-generated by users on Facebook, MySpace, Twitter, etc. and, we can assume, being archived somewhere. Many people even go to the trouble of identifying faces on Facebook through "tagging." Then of course there is the growing presence of surveillance cameras, biometric identifications, etc. Improved facial recognition and data-matching technologies, already under development, will mean that a curious (or malicious) person (or government) in the future may well be able to retrospectively track our movements and perhaps, even, accuse us of committing crimes that aren't even crimes yet! Sitting between current reality and things not yet invented, there is a whole class of technologies that are already existing, or known to be possible, but which have not yet emerged on a mass commercial basis. Some can be discovered by looking at the patent filings of high tech companies. For example, Apple was assigned a broad patent US patent 7814163 issued Oct. 12, 2010 [START_REF]&Sect2=HITOFF&d=PALL&p=1&u=%2Fnetahtml%2FPTO%2Fsrchn[END_REF] under the title "Text-based communication control for personal communication device." It provides an automated mechanism for controlling objectionable language in text messages, and also enforcing certain educational goals. As noted in the patent document, "These embodiments might, for example, require that a certain number of Spanish words per day be included in e-mails for a child learning Spanish." Pundits are already speculating that teenagers will find new euphemisms for the dirty words that are banned, and that pre-written Spanish emails will be readily downloadable to meet the quota requirements.

Those introducing new technologies, whether they are technology providers, network operators, companies, or governments would be well advised to stop and think about possible misuse before releasing new capabilities into the world. Simply stating that "we didn't think of that" will become less acceptable as disgruntled users demand compensation like that exacted from TJX and other firms that have breached consumer privacy.

Some Techniques for Anticipating Consequences

Whole organizations have been created to deal with the field of "strategic foresight," an attempt to gain insight into the future without making the assumption that it can be predicted by extrapolation of current situations. While still bedeviled by Taleb's Black Swans, which seemingly come out of nowhere, they have provided useful insights for many organizations. Then again, William Gibson, author of numerous science fictions novels and the man who coined the phrase "cyberspace," has acknowledged "most futurists are charlatans…when I made up that word (in 1982) I had no idea what it meant but it seems to have stuck." [START_REF] Gibson | Wordfest speech at the University of Calgary[END_REF] A specific technique within strategic foresight is the use of "scenario planning." Indeed this is the basis of a popular executive seminar taught at many business schools. The one taught at the University of Oxford holds out the promise of "using the future to improve our understanding of today" [25] and traces scenario planning back to the ancient Greeks.

Since scenarios are, in essence, stories about the future, it may well be productive for those faced with privacy and security decisions to adapt sagas like those identified here (and the future will provide a continuous supply.) To illustrate this, consider the case, investigated by the author, of a large US University that compromised the personal data on a large group of students. It happened because a student employee, wanting to test out cloud computing based file storage, needed a large file. He thoughtlessly chose a file containing personal data on all students currently living in the University's residence halls, and posted it on what turned out to be a publicly accessible server at Google. The unencrypted file contained the names, addresses, and social security numbers of a large number of students who had trusted the University to keep them confidential.

The case was further complicated by the fact that the person responsible was a casual student employee and had left the job by the time the matter was discovered. It wound up costing the University a substantial amount of money (to pay for credit insurance for students, etc.) as well as a public relations nightmare that involved writing apologetic letters to affected students.

While the perpetrator's superiors could not have anticipated that he would try to do ill-advised cloud computing experiments, system designers should craft systems, especially those used by casual employees, to restrict file access and sharing. As another example, also investigated by the author, a major Canadian hospital suffered breaches in patient confidentiality because a curious part time nurse made queries on patients all over the hospital with no proper controls.

Automated Policy Analysis -A Potentially Useful Tool to Minimize Unintended Consequences

Dan Lin et al. of Purdue University and Jorge Lobo of IBM T.J. Watson Research Center have developed a system called "EXAM -a Comprehensive Environment for the Analysis of Access Control Policies" [START_REF] Lin | EXAM -a Comprehensive Environment for the Analysis of Access Control Policies[END_REF]. It handles policies which can be expressed in XACML, a standard access control language that supports reasonably complex rules. They have recently started applying EXAM to privacy policies and are able to provide quantitative measures of the similarities of two privacy policies as well as identifying contradictions between them and flagging areas that are covered by one but not by the other. They have also developed graphical interfaces to display this information.

Damianou et. al. [START_REF] Damianou | Tools for Domain-Based Management of Distributed Systems[END_REF] have created a policy analysis tool called PONDER which provides a declarative language for specifying both security and management policies, as well as a hyperbolic tree viewer, a policy compiler and a policy editor. Acknowledging that "the refinement process is not expected to be fully automated," the authors make provisions for interactive editing, and graphical display. In the future, they hope to add animation.

In the specific domain of privacy, Sirin has applied a policy analysis tool called OWL to the US HIPPA legislation, which governs the use of medical information. As noted in a presentation at the APQC conference in Houston [START_REF] Siren | Automated Policy Analysis: HIPAA, XACML and OWL[END_REF] doing a deep analysis of policies does indeed turn up "holes in our policy" (in this case a nurse having "read and re-write" access to medical history files). As policies become more complex, tools like this will become even more necessary, and may help to identify contradictions, inconsistencies, and omissions.

While EXAM, PONDER, OWL, and similar approaches provide excellent technical vehicles for exploring policies, they may be of limited use in finding truly novel unanticipated consequences, simply because the "rule-makers" do not even think about them. How, for example, would browser designers have thought to worry about the color coding of already visited websites? Bertino has expressed enthusiasm for broadening the application of EXAM to other situations such as cloud computing policies, and work on that is ongoing [START_REF] Bertino | [END_REF].

Conclusions

Collecting stories about unintended consequences is certainly entertaining and instructive, and there is every indication that an endless supply of them is forthcoming. Indeed, Peter Neumann of SRI has been hard at work documenting the foibles of technology in the Risks Forum [30] since 1985, and that list of techno-human foibles shows no signs of slowing down. The hard, but really important work, is to derive from these anecdotes some principles that can help technology developers and policy makers to at least minimize the consequences of the LUC on their work.

Applying concepts from other fields (e.g. materiality from accounting) seems to be a useful tool, and worthy of further exploration. Scenario planning also has a place, especially since it forces people to think about unlikely events and sensitizes them to the inherent unpredictability and inaccuracy of the future.

Black Swans come into play too, but mainly as something to be enjoyed (or not) in retrospect.

Taleb makes it clear that attempting to predict them is futile, though we often try to convince ourselves when looking back that developments like penicillin and the Internet were actually logical outgrowths of previous events.

Automated policy analysis is another tool that may be useful. However, it is mainly confined to clarifying the underlying logic of policies. EXAM-like approaches may have some utility in finding subtle contradictions in privacy and security policies, but the really big issues will still require human thought and creativity.

Most of the examples in this paper were gleaned from publicly available sources such as newspapers and Internet postings. There are undoubtedly other rich sources of information about unintended consequences such as the files of national and local privacy commissioners, accident reports filed with government agencies, and internal corporate documents. The advent of "whistle blowing sites" such as the now-famous WikiLeaks can also provide a treasure trove of inspiration, especially as diplomatic cables which were intended for a very restricted audience are read and interpreted by a much broader community.

The real challenge is to understand the underlying patterns of thought and action that lead to unintended consequences, and, where possible, anticipate and prepare for them. This paper has identified seven proposed classes of factors driving LUC. These classes were developed by considering real life instances of unintended consequences and seeking similarities. Ideas from other disciplines such as economics, computer science and accounting were incorporated. The naming of the classes was an important part of the exercise, accomplished iteratively in conjunction with the input of participants in the PrimeLife/IFIP Summer School in August, 2010.

Armed with this model, researchers can take further examples of LUC and decide which category, or categories they fall into, what new categories should be defined, and how the existing ones should be refined. It is also worth noting that the focus of these examples was on undesirable unintended consequences. There is certainly a whole universe of unintended but positive "silver linings" to be explored.

As technology becomes more complex, and makes an even larger impact on our lives, it will be increasingly important to track unintended consequences and to learn from them. Unintended consequences will always be with us -they will just get more subtle and harder to predict as we think harder about them. Still, every instance of an undesirable consequence that is anticipated and avoided is a small victory that should improve our interaction with technology.

1 Introduction: The Unintended Consequences Problem Consider

  these fictitious news headlines, which are based on real-life cases: • Canada's Levy on Blank Media Triggers Surge in Cross-Border Shopping • US Patient Privacy Law Makes it Impossible for Hospitals to Defend Themselves • Web Browser Flaw Allows Sneaky People to Guess Your Identity • New Gambling Law Allows Americans to Be Cheated • Friendly Credit Card Company Sends Barack Obama Card to Someone Else

Acknowledgements

This work was funded in part by a Research and Study Travel Grant from the University of Calgary. The helpful comments of Prime Life project reviewers and other participants at the August 2010 Summer School in Helsingborg, Sweden are also gratefully acknowledged.