Meredydd Williams
email: meredydd.williams@cs.ox.ac.uk

Jason R C Nurse

Sadie Creese

Privacy Salience: Taxonomies and Research Opportunities

Keywords: Privacy Salience, Privacy Awareness, Taxonomy, IoT

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Privacy is a well-understood concept in the physical world. We all need some respite from the public gaze to enjoy our lives; indeed, it is essential to natural human development [START_REF] Berscheid | Privacy: A hidden variable in experimental social psychology[END_REF]. However, whereas individuals might consider a locked door as protecting one's privacy, they have difficulty practising equivalent actions online [START_REF] Creese | Can cognitive science help us make information risk more tangible online?[END_REF]. This can create a number of risks as users might be unaware of the digital dangers they face. Combining the definition of 'salience' [START_REF]Oxford English Dictionary: Salience[END_REF] with informational privacy [START_REF] Clarke | Introduction to dataveillance and information privacy, and definitions of terms[END_REF], we define 'privacy salience' as whether "informational privacy is prominent in a person's awareness or their memory of past experience". This differs slightly from 'privacy awareness', which we take to reflect long-term awareness of privacy, such as that which can be improved through educational campaigns. Risk in cyberspace is often intangible [START_REF] Jackson | Perceptions of risk in cyberspace[END_REF] and research [START_REF] Adjerid | Framing and the malleability of privacy choices[END_REF] suggests reduced salience can lead to unwise decisions. Some [START_REF] Williams | The perfect storm: The privacy paradox and the Internet-of-Things[END_REF] have claimed this intangibility could even contribute to the 'Privacy Paradox' [START_REF] Barnes | A privacy paradox: Social networking in the United States[END_REF], the disparity between what individuals claim about privacy and how they act. As technology permeates our society and we begin to live our lives 'online', privacy salience gains critical importance.

Previous research has considered the topic from a number of angles. For example, John et al. [START_REF] John | The best of strangers: Context dependent willingness to divulge personal information[END_REF] conducted several field experiments: two seeking to highlight privacy and one looking to hide the issue. They found that when privacy concerns were primed, participants were less likely to disclose their data. In contrast, Tsai et al. [START_REF] Tsai | The impact of privacy indicators on search engine browsing patterns[END_REF] modified search engine interfaces to promote privacyrespecting results. Their analysis of 15,000 queries found that their alterations encouraged prudent selections. Adjerid et al. [START_REF] Adjerid | Sleights of privacy: Framing, disclosures, and the limits of transparency[END_REF] studied how the provision of privacy information could influence user actions. They discovered that a delay of only 15 seconds between notice and decision could lead to less-private behaviour. Although previous studies concern a range of platforms and themes, the field has seen little systemisation of knowledge. Neither an extensive literature review nor a taxonomy have yet been produced: instruments which can both structure existing work and highlight future opportunities. Such developments are crucial to ensure that new studies do not overlook the varied findings of past research.

Therefore, we develop three extensive taxonomies of privacy salience literature, classifying studies by the themes they concern, the methodologies they apply and the platforms on which they are based. We select these factors as we believe they best encapsulate the content of the articles. Through a datadriven process of inductive coding [START_REF] Thomas | A general inductive approach for analyzing qualitative evaluation data[END_REF], we formulate categories ranging from social networks to smartphones, privacy seals to permissions. We classify our surveyed articles within these groups and discuss the literature most relevant to each section. We move on to investigate those category combinations, whether (Methodology,Platform), (Methodology,Theme) or (Platform,Theme), which are both feasible and underexplored. Our frequency matrices, populated by this series of category tuples, enable identification of both popular research areas and potential lacunas. For example, while privacy documents were often studied during web browsing, salience is rarely explored in the Internet-of-Things (IoT). This is of concern as unfamiliar devices could potentially mask the topic of privacy. We conclude by recommending both future work and potential extensions to our taxonomies.

The remainder of our paper is structured as follows. Section 2 discusses our methodology, including literature selection, exclusion criteria, coding processes and taxonomy construction. Section 3 then explores our three taxonomies in detail, highlighting relevant previous literature. In Section 4 we present our distribution matrices and identify opportunities for future research. Finally, we conclude the paper in Section 5 and reflect on possible extensions to this work.

We first outline our definitions, before describing our processes to select and exclude existing work. We continue by discussing our inductive coding processes [START_REF] Thomas | A general inductive approach for analyzing qualitative evaluation data[END_REF] and how our privacy salience taxonomies were constructed.

Definitions

To ensure our taxonomies are representative of the literature, we should precisely define our terms. The Oxford English Dictionary defines salience to be "[t]he quality or fact of being more prominent in a person's awareness or in his memory of past experience" [START_REF]Oxford English Dictionary: Salience[END_REF]. As privacy can be a nebulous topic, we scope our definition to encompass informational privacy. Clarke [START_REF] Clarke | Introduction to dataveillance and information privacy, and definitions of terms[END_REF] described this concept as "the interest an individual has in controlling, or at least significantly influencing, the handling of data about themselves". Therefore, to reiterate, we define privacy salience as as whether "informational privacy is prominent in a person's awareness or their memory of past experience". This differs slightly from 'privacy awareness', which we take to reflect long-term awareness of privacy, such as that which can be improved through educational campaigns.

We also explicitly specify what we consider to be a taxonomy. The Oxford English Dictionary [START_REF]Oxford English Dictionary: Taxonomy[END_REF] defines a taxonomy as a "particular system of classification", and in this work we classify privacy salience research. De Hoog [START_REF] De Hoog | Methodology of taxonomy[END_REF] explains how construction consists of three parts: ordering, representation and nomenclature. In terms of ordering, categories should be arranged in a certain order and this order expressed through "character correlation". For representation, the elements should be "maximally simple" and atomic in character. Finally, the categories should be named formally to ensure the structure is usable. We incorporated these key principles into the development of our taxonomies.

Literature Selection

We first surveyed existing research to identify those works which concerned privacy salience. We did not explicitly constrain ourselves to particular disciplines, as we sought to explore the topic from multiple angles. Accordingly, we conducted our literature search on a wide range of databases from a variety of fields. These consisted of Google Scholar, Scopus, Web of Science, SpringerLink, JSTOR and Mendeley (general); IEEE Xplore, ACM Digital Library, CiteSeerX and DBLP Computer Science (computer science, Human-Computer Interaction (HCI) and cyber security); ScienceDirect (sciences); the Social Science Research Network (SSRN) (social sciences) and HeinOnline (law). These databases index those fields from which privacy salience research frequently originates, such as HCI and psychology. With engines such as Google Scholar searching broader academia and SSRN considering the social sciences, we retrieved work from a wide range of disciplines. Since we frequently located the same articles in multiple search results, we are confident the literature was well surveyed.

We also used a variety of search terms to ensure all works considering privacy salience were identified. We began with the synonymous terms 'privacy salience' and 'privacy saliency', in addition to 'privacy tangibility' due to its similar definition. Being cognisant that the 'privacy salience' term only gained popularity in the late 2000s, we also searched for 'privacy awareness'. Frequently salience was not mentioned in articles, even though studies considered the effects of highlighting policies and notices. For this reason, we also used 'privacy policies', 'privacy seals', 'privacy notices', 'privacy warnings', 'privacy indicators' and 'privacy nudges'. By expanding our list of phrases, we successfully identified articles which might have been otherwise overlooked.

To further survey this topic, we undertook literature snowballing through the references and citations of identified works. This was performed in the systematic method of Wohlin [START_REF] Wohlin | Guidelines for snowballing in systematic literature studies and a replication in software engineering[END_REF], with extensive backwards and forwards snowballing following our database search. This collection was then manually-filtered to verify that selected works concerned the topic. This ensured that we did not sacrifice quantity for quality by expanding our search terms. Although terms can never be fully exhaustive, our broad selection concerns topics frequently associated with privacy salience. Since many search results were sorted by both relevance and citation count, it is unlikely we overlooked articles of significance.

Exclusion Criteria

To complement our search term expansion, we strengthened our exclusion criteria. Firstly, we only analysed articles in the English language to ensure works could be judged fairly. Secondly, we verified whether our search results actually concerned privacy salience, of which a majority did not. This is simply an artefact of database searches, where works can refer to 'privacy' or 'salience' but not both in combination. Furthermore, through our use of associated terms such as 'privacy policies', we retrieved many articles which considered these documents but not their effect on salience. These works were filtered out at this stage.

Thirdly, we excluded papers which directly duplicated research. For example, two articles by Hughes-Roberts [START_REF] Hughes-Roberts | On-line privacy behavior: Using user interfaces for salient factors[END_REF][START_REF] Hughes-Roberts | Reminding users of their privacy at the point of interaction: The effect of privacy salience on disclosure behaviour[END_REF] concerned the development of the same social networking interface. In these cases, we included the most recent paper as would be more likely to possess additional findings. For a similar reason, when multiple databases returned different versions of a work, we selected the most recent instance. Finally, as a means of ensuring our research was of a high quality, we excluded articles which were not peer-reviewed. Although this approach might have reduced the breadth of our survey, it is important that taxonomies are constructed on credible works.

Coding Process

Inductive reasoning can be beneficial when conducting research which has not been previously attempted. Since we are the first to either survey privacy salience or construct taxonomies on the topic, inductive coding appeared most appropriate. We conformed to the popular approach of Thomas [START_REF] Thomas | A general inductive approach for analyzing qualitative evaluation data[END_REF] which consists of data cleaning, text analysis, category creation, overlapping coding and category refinement. We defined our coding units physically [START_REF] Stemler | An overview of content analysis[END_REF], based on the natural boundaries of each article. We began coding by ensuring all text was legible, accessible and downloaded in a persistent format. We then analysed the documents to familiarise ourselves with the main topics.

After studying our articles on multiple occasions, we created initial groups based on their main concepts. As inductive coding progressed, we recognised that our categories clustered around methodologies, platforms and themes. We believe these factors best encapsulate the paper content, since they concern the research technique, the target of research and the research topic. Methodologies can influence how an issue is approached: while literature reviews reflect on an issue, field experiments conduct empirical studies. Although article frequency does not directly indicate which techniques are most fruitful, it acts as a useful proxy. While certain methodologies might be more appropriate for privacy salience research than others, we still expect a general trend between frequency of use and utility. We define platform as the domain on which salience is analysed, with instances ranging from social networks [START_REF] Stutzman | Friends only: Examining a privacy-enhancing behavior in Facebook[END_REF] to smartphones [START_REF] Rajivan | Influence of privacy attitude and privacy cue framing on android app choices[END_REF]. As privacy is inherently contextual, findings on one platform might differ from those on others. Although each article concerns privacy salience, the topic is explored through a range of themes. For example, while some researchers study the effect of privacy policies [START_REF] Vail | An empirical study of consumer perceptions and comprehension of web site privacy policies[END_REF], others analyse the influence of framing [START_REF] Johnson | Defaults, framing and privacy: Why opting in-opting out[END_REF].

It was at this stage we decided to develop three distinct taxonomies rather than a combined chart. While our factors each offered interesting insights, they differed excessively for a single structure. Although we considered a taxonomy with methodology, platform and theme top-level categories, this introduced unnecessary complexity. By developing three separate taxonomies and classifying our works, we can identify those combinations which might be underexplored.

Thomas' fourth coding procedure [START_REF] Thomas | A general inductive approach for analyzing qualitative evaluation data[END_REF] accepts that texts can be coded into multiple categories, or indeed no categories at all. This is useful in the case of themes, as a study might concern a number of topics. For example, Yang et al. [START_REF] Yang | Do social media privacy policies matter? Evaluating the effects of familiarity and privacy seals on cognitive processing[END_REF] investigated the influence of both privacy policies and trust seals, and their work should not be excluded from either group. As articles predominantly used one methodology and one platform, these factors support single categories. Finally, we refined our groups to ensure clarity and consistency. For example, although one work might analyse social media disclosures [START_REF] Hughes-Roberts | Reminding users of their privacy at the point of interaction: The effect of privacy salience on disclosure behaviour[END_REF] and another might study Facebook behaviour [START_REF] Stutzman | Friends only: Examining a privacy-enhancing behavior in Facebook[END_REF], both concern social networks. This was crucial for de Hoog's taxonomy 'ordering' principle [START_REF] De Hoog | Methodology of taxonomy[END_REF], as we ensured similar elements were grouped consistently.

Taxonomy Completion

We continued our construction processes in compliance with the 'representation' and 'nomenclature' principles [START_REF] De Hoog | Methodology of taxonomy[END_REF]. Where categories contained otherwise differing elements, further subdivisions were made. For example, although policies and seals both highlight privacy, their approaches are far from identical. We next created representative names for our categories, complying with the 'nomenclature' principle [START_REF] De Hoog | Methodology of taxonomy[END_REF]. Naming was undertaken iteratively, refining definitions as categories evolved. Titles aimed to encapsulate commonalities in a group; for example, privacy policies and notices could both be considered types of privacy document.

We begin by outlining category metrics before discussing our three taxonomies in detail. Through our database search and snowballing process, we received over 1000 potential results. After manual filtering, we found only 76 articles actually concerned privacy salience, with the other works just matching on search terms. This list was further reduced to 73 papers based on our aforementioned exclusion criteria. While our search dates were not constrained, our selected literature ranged from 1977 to June 2016. Research was conducted through a range of methodologies, with Field Experiments and Tool Development appearing most prevalent. In terms of platform, the Web was found to be most popular, with Social Networks also frequently explored. Interfaces were the most commonlyidentified theme, followed by Framing and privacy Documents. Table 1 below presents our categories and the quantitative distribution of works. As articles could concern multiple topics, this is reflected in our larger theme totals.

Methodology Taxonomy

As shown below in Figure 1, we subdivided methodologies into four approaches: Literature Reviews, Lab Experiments, Field Experiments and Tool Development. Methodologies were identified by comparing research techniques with standard definitions. Although a minority of articles possessed multiple approaches, we classified based on the predominant methodology. For example, when an application is created and then evaluated through a field study, it would be categorised within the Tool Development group [START_REF] Larose | Promoting i-safety: Effects of privacy warnings and privacy seals on risk assessment and online privacy behavior[END_REF]. Although we considered supporting multiple methodologies, this approach would have introduced complexity not warranted by our literature.

Literature Reviews study an existing body of work to derive novel findings. For example, Aguirre et al. [START_REF] Aguirre | The personalization-privacy paradox: Implications for new media[END_REF] drew on prior research to discuss how firms can best manage consumer relationships. They found that although service personalisation offers benefits, it can increase the salience of privacy risk. Cichy and Salge [START_REF] Cichy | The evolution of privacy norms: Mapping 35 years of technology-related privacy discourse[END_REF] also analysed previous work, studying [START_REF] Lipford | Understanding privacy settings in Facebook with an audience view[END_REF] years of privacy discourse in The New York Times. Through considering social norms and topic salience, they found perceptions to be susceptible to myopia and manipulation. Lab Experiments are empirical studies conducted in well-controlled environments. For example, a 24-person study was used to evaluate three privacyenhancing extensions [START_REF] Schaub | Watching them watching me: Browser extensions' impact on user privacy awareness and concern[END_REF]. The researchers found that although the plug-ins highlighted data collection, concerns were mitigated by the applications themselves. Vemou et al. [START_REF] Vemou | Directions for raising privacy awareness in SNS platforms[END_REF] established social network accounts to analyse profile registration and privacy policies. After exploring third-party access and audience management, they concluded that simplified settings might improve salience.

Field Experiments are undertaken in realistic environments, benefiting from greater ecological validity than lab studies. To explore the effects of framing, 280 participants were tasked to create a social networking profile [START_REF] Adjerid | Sleights of privacy: Framing, disclosures, and the limits of transparency[END_REF]. It was found that when privacy notices were followed by time delays, data disclosure increased. John et al. [START_REF] John | The best of strangers: Context dependent willingness to divulge personal information[END_REF] conducted three user studies, with the former increasing privacy salience and the latter two disguising the topic. They saw that even when risks are low, people refuse to disclose when their concerns are primed.

Tool Development concerns research which develops interfaces or applications to increase privacy salience. For example, PrivAware was a social networking tool which highlighted information loss [START_REF] Becker | Measuring privacy risk in online social networks[END_REF]. The system could infer personal details with 60% accuracy and gave recommendations for friend deletion. Lipford et al. [START_REF] Lipford | Understanding privacy settings in Facebook with an audience view[END_REF] developed an 'audience view', allowing users to observe their profiles as others do. After adding this tab to the Facebook interface, they found individuals better-understood the consequences of their actions.

Platform Taxonomy

Platforms were defined based on the domain in which privacy salience research was undertaken. As presented below in Figure 2, we distinguished between six categories: General, Web, Social Networks, Mobile, Software and Internet-of-Things. We found the General class to be beneficial, as several works [START_REF] Acquisti | Nudging privacy: The behavioral economics of personal information[END_REF][START_REF] Frey | An experiment with a confidentiality reminder in a telephone survey[END_REF][START_REF] Johnson | Defaults, framing and privacy: Why opting in-opting out[END_REF] consider salience without reference to a particular platform. As Social Networks are distinctive portals which may be accessed via either web browsers or smartphones, we deemed these to define a separate category.

General works study privacy salience without direct consideration of a specific environment. For example, Acquisti [START_REF] Acquisti | Nudging privacy: The behavioral economics of personal information[END_REF] discussed the importance of 'nudging' for realigning user behaviour. He emphasised the benefits of soft paternalism, but did not constrain privacy concepts to a particular domain. In a 1979 study, Reamer [START_REF] Reamer | Protecting research subjects and unintended consequences: The effect of guarantees of confidentiality[END_REF] analysed whether guarantees of confidentiality paradoxically reduced disclosure. He found survey participants which were assured anonymity were less likely to respond, suggesting salience had an effect. The Web category concerns works which study privacy salience during web browsing. Tsai et al. [START_REF] Tsai | The impact of privacy indicators on search engine browsing patterns[END_REF] found that privacy indicators on search engines can encourage prudent behaviour. Through their use of the Privacy Finder tool, they found sites were more popular if annotated as privacy-respecting. Plug-ins can also analyse online behaviour, such as the Privacy Fox browser extension [START_REF] Arshad | Privacy fox -A JavaScript-based P3P agent for Mozilla Firefox[END_REF]. This application both translated policies into short notices and highlighted website practices which might cause concern.

Since Social Networks support the interaction of online individuals, they are of great interest to privacy researchers. During a 6-week trial, Facebook users were nudged to remember their post audiences [START_REF] Wang | A field trial of privacy nudges for Facebook[END_REF]. By illustrating the potential consequences of their actions, unintended disclosures were reduced. Bonneau and Preibusch [START_REF] Bonneau | The privacy jungle: On the market for data protection in social networks[END_REF] evaluated 45 sites in a comprehensive analysis of social network protections. They saw that since data disclosure can be reduced by salient privacy, this influences interface design.

The Smartphones category concerns those works which study mobile phones and their apps. For example, the AppOps tool was used to highlight the data shared between smartphone applications [START_REF] Almuhimedi | Your location has been shared 5,398 times!: A field study on mobile app privacy nudging[END_REF]. When the consequences of lax privacy were illustrated, over half the participants changed their permissions. Balebako et al. [START_REF] Balebako | The impact of timing on the salience of smartphone app privacy notices[END_REF] explored the timing of privacy notices through an Android field experiment. They found salience was increased more by in-app dialogs than those shown before installation.

The Software section concerns works which analyse desktop applications, rather than online portals or mobile apps. In a similar manner to Balebako et al. [START_REF] Balebako | The impact of timing on the salience of smartphone app privacy notices[END_REF], a 222-person study explored how notice timing affects user behaviour [START_REF] Good | Noticing notice: A large-scale experiment on the timing of software license agreements[END_REF]. The researchers found that risky installations were reduced by summarising license agreements. Bravo-Lillo et al. [START_REF] Bravo-Lillo | Your attention please: Designing security-decision UIs to make genuine risks harder to ignore[END_REF] modified user interfaces to highlight security and privacy threats. In their study of dialog messages, they discovered salient warnings could reduce dangerous installations.

The Internet-of-Things category concerns the privacy analyses of smart devices. For example, it was proposed that RFID privacy salience could be increased by personal privacy assistants [START_REF] Konomi | Personal privacy assistants for RFID users[END_REF]. This gadget would process tag data and display risks on a mobile interface. Gisch et al. [START_REF] Gisch | The privacy badge: A privacy-awareness user interface for small devices[END_REF] developed the Privacy Badge, an awareness tool specifically designed for small devices. Their data loss visualisations were evaluated through a user study, which found their application to be usable and informative.

Theme Taxonomy

Themes refer to the range of topics considered in the privacy salience literature. We identified 17 themes through inductive coding, ranging from Nudging [START_REF] Acquisti | Nudging privacy: The behavioral economics of personal information[END_REF] to Adverts [START_REF] Wang | Investigating effects of control and ads awareness on android users' privacy behaviors and perceptions[END_REF] to privacy Settings [START_REF] Stutzman | Friends only: Examining a privacy-enhancing behavior in Facebook[END_REF]. Through our process of taxonomy construction, these classes were grouped under 7 top-level categories. This structure is presented below in Figure 3. This design both highlights theme commonalities and assists the lacuna identification discussed in Section 4. While we considered limiting works to a single theme, such an approach would underrepresent the secondary topics found in articles. Recognising that the vast majority of papers did concern several topics, we decided each work could support multiple themes. Documents refer to a range of privacy statements, such as policies, notices and seals. In one study, documents were presented which either concerned tech company protections or their negative activities [START_REF] Marreiros | Now that you mention it': A survey experiment on information, salience and online privacy[END_REF]. The authors found disclosure decreased even when they highlighted positive behaviour, suggesting privacy salience influenced action. Hui et al. [START_REF] Hui | The value of privacy assurance: An exploratory field experiment[END_REF] analysed how privacy statements and TRUSTe seals affect user behaviour. They discovered that seals had little impact, suggesting increased salience does not always translate into action.

Framing relates to the way in which information or choices are presented. For example, two online surveys explored whether default responses affect privacy decisions [START_REF] Johnson | Defaults, framing and privacy: Why opting in-opting out[END_REF]. When choices were made opt-in, the researchers found that agreement increased by 30%. Joinson et al. [START_REF] Joinson | Measuring self-disclosure online: Blurring and non-response to sensitive items in web-based surveys[END_REF] discovered that 'prefer not to say' options were used more frequently when privacy was salient. After priming the topic through a privacy questionnaire, they also found that sensitive questions were commonly avoided.

The Interfaces category refers to both visualisations and user interface developments. Kani-Zabihi and Helmhout [START_REF] Kani-Zabihi | Increasing service users' privacy awareness by introducing on-line interactive privacy features[END_REF] discussed online interactive privacy features; tools designed to support user decision-making. They described an enquiry system which helps individuals discuss their privacy concerns with service providers. In another study, graphical warnings were evaluated on mobile interfaces [START_REF] Christin | Raising user awareness about privacy threats in participatory sensing applications through graphical warnings[END_REF]. The researchers discovered that after their dialogs were presented, 70% of the participants claimed they would change their permissions.

Design guidelines can inform the development of technologies which make privacy salient. One article [START_REF] Lederer | Personal privacy through understanding and action: Five pitfalls for designers[END_REF] presented five pitfalls for privacy design, including obscuring information flow and inhibiting existing practice. The authors explained how "users can make informed use of a system only when they understand the scope of its privacy implications". Schaub et al. [START_REF] Schaub | A design space for effective privacy notices[END_REF] outlined requirements and best practices for privacy notice design. They went on to discuss how dialog messages could be challenged by Internet-of-Things interfaces.

Privacy is a Social construct and therefore many studies concern user interactions. For example, eight Facebook users were interviewed on the topic of online friendship [START_REF] Houghton | Privacy, social network sites, and social relations[END_REF]. The researchers discovered that unwise disclosures often originated from a lack of privacy salience. Ziegeldorf et al. [START_REF] Ziegeldorf | Comparison-based privacy: Nudging privacy in social media (position paper)[END_REF] observed that comparisons are a natural behaviour, with individuals evaluating their actions against those of their peers. Their nudging system aimed to incentivise privacy by highlighting the behaviour of others.

With Marketing frequently raising privacy concerns, this category concerns adverts and tracking. A study of 447 users analysed how smartphone ad awareness affects privacy perceptions [START_REF] Wang | Investigating effects of control and ads awareness on android users' privacy behaviors and perceptions[END_REF]. Individuals were found to make better privacy decisions when informed of data use procedures. Goldfarb and Tucker [START_REF] Goldfarb | Online display advertising: Targeting and obtrusiveness[END_REF] also analysed marketing, studying how relevance and obtrusiveness influence purchasing intent. They discovered that while ads are effective when either salient or targeted, combined approaches trigger privacy concerns.

Controls refer to the permissions and configurations which affect user privacy. 444 students were surveyed to gauge the popularity of 'friends-only' Facebook settings [START_REF] Stutzman | Friends only: Examining a privacy-enhancing behavior in Facebook[END_REF]. Although respondents were not aware of their exposure, privacy discussions were found to increase salience. Malandrino et al. [START_REF] Malandrino | Privacy awareness about information leakage: Who knows what about me?[END_REF] analysed data leakage through a browser plug-in. By highlighting the information collected by third-party services, they increased the salience of privacy violations.

Research Gaps and Opportunities

By highlighting those areas not frequently explored, we sought to identify potential research opportunities. We first considered the prevalence of our methodologies, platforms and themes. Although our surveyed works concern a wide variety of themes, we constrained our analyses to the top-level categories. While this simplification sacrificed a degree of depth, it assisted the identification of sparse research areas.

We next considered all possible 2-tuples for (Methodology,Platform), (Methodology,Theme) and (Platform,Theme) combinations. Again, use of all 17 lowlevel themes would have introduced significant complexity to this process. Furthermore, as the vast majority of combinations would not have been previously explored, we would have no direction in selecting future opportunities. While we considered the use of triples, this approach challenges tabular visualisation (as explained shortly). By analysing our 2-tuples individually, we could identify combinations with greater flexibility.

Through analysing the tuple frequency in our literature, we populated the entries in three matrices. Figure 4 presents (Methodology,Platform) works, Figure 5 concerns (Methodology,Theme) and Figure 6 relates to (Platform,Theme) combinations. As we supported one methodology and platform per paper (due to the predominance of this distribution), the first matrix was simple to complete. Since articles could concern several themes, each work could conform to multiple (Methodology,Theme) and (Platform,Theme) tuples. While this could skew the frequency of non-theme categories, it assisted the identification of less-populated research areas.

By comparing the frequency of matrix entries, we discerned which combinations are most popular. For example, Web Field Experiments were prevalent (14 instances), as were Tools with privacy Interfaces [START_REF] Federrath | Designing privacy enhancing technologies[END_REF]. Privacy Documents were commonly analysed during Web browsing [START_REF] Bravo-Lillo | Your attention please: Designing security-decision UIs to make genuine risks harder to ignore[END_REF], reflecting the research interest in policies and seals. Finally, these tuple frequencies were colour-coded based on their magnitude. This assisted identification of those areas least investigated in existing work. Although many combinations have been infrequently explored, we continue by exploring those sparse intersections with the greatest viability. A low frequency alone does not imply an opportunity, as some combinations are less feasible than others. However, whereas Social interactions might not be best analysed through Lab Experiments, several combinations do appear viable.

Research Opportunities

Through inspection of our frequency matrices, we observed three main areas with feasible opportunities. This is not to say that other alternatives are not valuable, such as much-needed Literature Reviews and Software studies, but the below topics appear of particular interest.

Due to the novelty of the platform, the Internet-of-Things (IoT) appears underexplored. As presented in Figure 4, few Lab or Field Experiments have analysed IoT salience. Although privacy risks can obscured by our current technologies, unfamiliar devices might exacerbate this issue. Researchers could use controlled lab studies to investigate how much data is leaked by novel products. This could be complemented by field experiments exploring how these tools are actually used. We could analyse how salient privacy is in smart home environments; locations which are personal and increasingly popular. Figure 6 suggests that while Social Network Controls are frequently studied, several platforms do not receive similar attention. IoT settings might be unfamiliar, hidden or challenging to adjust, resulting in unintentional data disclosure. Future work could explore whether privacy salience can enhanced by simplifying these interfaces.

Figure 5, the (Methodology,Theme) matrix, suggests that while Tool Developments are popular, principles of their Design are rarely considered. While guidelines have been constructed for Privacy Enhancing Technologies (PETs) [START_REF] Federrath | Designing privacy enhancing technologies[END_REF], the topic of salience has not been approached. Social network modifications [START_REF] Hughes-Roberts | Reminding users of their privacy at the point of interaction: The effect of privacy salience on disclosure behaviour[END_REF] and smartphone interfaces [START_REF] Almuhimedi | Your location has been shared 5,398 times!: A field study on mobile app privacy nudging[END_REF] have successfully highlighted privacy, resulting in improved user behaviour. For these achievements to be replicated generally, researchers should develop design principles for what could be considered 'Salience-Enhancing Technologies'.

Much of the web is supported by affiliate networks and targeted advertising. This has led many individuals to decry the increasingly-personalised nature of online ads. Despite this fact, Figure 6 suggests that few have explored the relationship between Marketing and privacy salience. Companies face the challenge of advertising their services without priming concerns. Future research could study whether online portals can respect user privacy while delivering targeted advertising. Alternatively, with the increasing popularity of ad-blocking software, researchers could explore how salience is affected when adverts are removed. It is of particular surprise these Marketing studies have not analysed Social Networks, especially considering Facebook's advertising strength. Researchers could investigate how privacy concerns are suppressed and suggest approaches for highlighting these ads.

In this work we have considered at length the important topic of privacy salience. We begun by surveying a wide range of existing literature before identifying common themes and topics. Through iteratively refining these categories we developed the first taxonomies on privacy salience. We structured these divisions on methodologies, platforms and themes; factors which encapsulate the content of existing research. By classifying prior work within these taxonomies, we systematise the knowledge in the literature. We proceeded by analysing category frequency and which factor combinations are most popular. We found that field studies of the web were prevalent, as were tools with privacy interfaces. Finally, through exploring our colour-coded matrices, we identified opportunities for future research. These include IoT field experiments and investigations of online marketing. We also recommend further analyses of privacy settings, whether in ubiquitous environments or desktop computers.

While we believe our taxonomies support the study of privacy salience, we accept several limitations to our work. A minority of articles possessed multiple methodologies or studied a variety of platforms. Due to the sparsity of these instances, we constrained our analyses in the interest of simplicity. An expanded future work would reflect the diversity inherent in many privacy studies. While we chose methodologies, platforms and themes to structure our taxonomies, we accept that there are several alternatives. We could have divided research by discipline as a means of highlighting interesting fields. Although this was considered, we felt it would be challenging to divide disciplines in an objective fashion. Alternatively, we could have categorised articles based on the metrics they analysed. For example, while many works studied disclosure [START_REF] John | The best of strangers: Context dependent willingness to divulge personal information[END_REF][START_REF] Reamer | Protecting research subjects and unintended consequences: The effect of guarantees of confidentiality[END_REF], others explored settings alteration [START_REF] Almuhimedi | Your location has been shared 5,398 times!: A field study on mobile app privacy nudging[END_REF][START_REF] Christin | Raising user awareness about privacy threats in participatory sensing applications through graphical warnings[END_REF]. In future analyses, metrics could be incorporated to consider how salience can be best investigated.

While our research presents an initial analysis, we envisage many instances of future work. As a means of validating our taxonomies, we could invite privacy experts to categorise the literature. In this approach, we would solicit a panel to analyse and classify each of our surveyed works. Through exploring their range of classifications, we could refine the consistency of our structures. Although research has been conducted in a range of platforms, there have been few comparative studies. Whereas privacy might be salient on a familiar web browser, the topic could be obscured by smartphone interfaces. Through conducting user studies on multiple platforms, we could explore how contexts affect privacy concerns. Finally, metrics should be studied to enable us to deconstruct the concept of privacy salience. While disclosure might relate to notions of confidentiality, privacy settings might be concerned with user control. As novel technologies proliferate, we believe privacy salience will only become increasingly important.

Fig. 1

 1 Fig. 1 Methodology taxonomy

Fig. 2

 2 Fig. 2 Platform taxonomy

Fig. 3

 3 Fig. 3 Theme taxonomy

Fig. 4 (

 4 Fig. 4 (Methodology,Platform) frequency matrix

Fig. 6 (

 6 Fig. 6 (Platform,Theme) frequency matrix

Table 1

 1 Literature distribution


Kovila P L Coopamootoo

Thomas Groß

Evidence-Based Methods for Privacy and Identity Management

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The Encyclopaedia Britannica defined science as a "system of knowledge that is concerned with the physical world and its phenomena and that entails unbiased observations and systematic experimentation." In general, it is a purpose of science to advance human knowledge. The scientific method is evidence-based, includes principles such as falsification or reproducibility as well as statistical tools to decide between hypotheses.

To what extent is security/privacy research a science? How does research in this field advance human knowledge? In the recent years, funding bodies have sought to strengthen evidence-based research in security and privacy and, arguably, those methods have seen adoption in the field, especially under the flag of "science of security" [START_REF] Peisert | How to design computer security experiments[END_REF][START_REF] Maxion | Making experiments dependable[END_REF][START_REF] Carroll | Realizing scientific methods for cyber security[END_REF][START_REF] Balenson | Cybersecurity Experimentation of the Future (CEF): Catalyzing a new generation of experimental cybersecurity research -final report[END_REF].

Challenges. Whereas the tenets of the scientific method are often demanded, they are easily subverted by methodological mistakes or insufficient power under the all too polished surface. Ioannidis [START_REF] Ioannidis | Why most published research findings are false[END_REF] gave a harsh account of the situation, arguing "why most published research findings are false."

To make matters worse, there is a replication crisis in science. For example, 270 researchers of the Open Science Collaboration [START_REF]Open Science Collaboration: An open, large-scale, collaborative effort to estimate the reproducibility of psychological science[END_REF] have reported on a dire Major contributions were made while the author was at Newcastle University.

situation after having sought to reproduce 100 well-known results published in three major psychology journals [START_REF]Open Science Collaboration: Estimating the reproducibility of psychological science[END_REF]. They could only reproduce 39% of the results. It is deemed likely that the replication crisis also pertains to other fields, including security or privacy and identity management.

Even down to the nitty-gritty of statistical inference, many misconceptions and controversies have been observed, including, for instance, a comprehensive account of null hypothesis testing by Nickerson [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF].

All that glitters is not gold. While evidence-based methods hold a promise to support the pursuit of knowledge in security and privacy, they ask of us great diligence to live up to their tenets. This IFIP workshop sought to sensitize participants to the hallmarks and inference methods of evidence-based research in privacy and identity management. It includes examples for true experiments as well as systematic literature reviews as two classes of evidence that are considered as most reliable.

Scope. Research methodology for evidence-based methods is a vast topic, filling tomes in the sciences. Consequently, this workshop summary will only offer a primer-an introduction to hallmarks, experiment design and statistical inference. Given that the workshop aimed at sensitizing for evidence-based methods and its requirements, we make a number of simplifications. We only focus on (a) true experiments (inducing an experiment condition), (b) hypothesis testing (rejecting a null hypothesis), (c) two conditions (control and experiment), (d) simple statistics (difference between means, t-test). Explicitly out of scope are: qualitative methods, observational studies and complex statistical models.

Outline. This workshop summary contains two theory sections on hallmarks of empirical research and statistical inference, where each of the theory section concludes with a concise checklist of quality criteria. Section 2 contains the hallmarks discussion, leading up to hypothesis testing. Then, we interleave a section on practical experiment design in Sect. 3 which reflects a round-table discussion of the workshop. From this intermezzo, we continue our theoretical inquiry with Sect. 4 on statistical inference and power. Section 5 reports on participants responses to the workshop questionnaire. We detail areas of privacy research with interest in experimental methodology, methodological issues encountered and their personal learning objective from the workshop.

Hallmarks of Empirical Research

Definition 1 (True Experiment [START_REF]Oxford Dictionary of Statistical Terms[END_REF]). An investigation in which the investigators have sufficient control of the system under study, in particular to be able to determine the assignment of different units of study to different conditions.

A true experiment follows requirements contributing towards rigorous science. The requirements include (a) definition of a falsifiable hypothesis, (b) defining and controlling variables, (c) assessing internal and external validity, (d) repeatability and reproducibility of the method and analysis [START_REF] Maxion | Making experiments dependable[END_REF][START_REF] Carroll | Realizing scientific methods for cyber security[END_REF].

Falsifiable Hypotheses

Definition 2 (Hypothesis [START_REF] Everitt | Cambridge dictionary of statistics[END_REF]). Specific testable predictions made generally about the response and explanatory variables in a study.

Testing hypotheses is one of the tenets behind scientific discovery. Popper [START_REF] Popper | The logic of scientific discovery[END_REF] coined the theory of falsification, whereby the researcher formulates a hypothesis such that the experiment can show it to be false. According to Popper, hypotheses cannot be inductively verified, but only empirically falsified. Falsifiable hypotheses are formulated such that they can be measured and observed.

Example 1 (Falsifiable Hypotheses).

-All swans are white. [START_REF] Popper | The logic of scientific discovery[END_REF] (Falsifying observation: a black swan) -Higher cognitive workload implies more click-throughs on phishing links.

(Falsifying observation: experiment showing equal phishing click-troughs across workloads)

Controlled Variables

In experiments, we distinguish between three types of variables: manipulated, controlled or measured. A variable that is manipulated, the independent variable, is to predict or explain the dependent or response variable.

Definition 3 (Variable).

A variable is some characteristic that differs from subject to subject or from time to time [START_REF] Everitt | Cambridge dictionary of statistics[END_REF].

(a) The independent variable IV is a variable that is induced/manipulated [START_REF] Miller | Experimental design and statistics[END_REF].

(b) The dependent variable DV is a variable that is observed/measured [START_REF] Miller | Experimental design and statistics[END_REF]. A systematic change in the IV causes a change in the DV. (c) A confounding variable (short: confounder) is an extraneous variable whose presence affects the variables being studied, so that the results do not reflect the actual relationship between the IV and DV.

Methods to actively control confounding variables include random assignment of subjects to conditions, restricting variation in confounders (e.g., selecting subjects of the same age eliminates confounding by age) and matching potential confounders across conditions. Statistical models can also be used to adjust for the bias introduced by a confounder during analysis.

Validity.

Validity refers to the extent to which a measuring instrument is measuring what was intended [START_REF] Everitt | Cambridge dictionary of statistics[END_REF], where a change in the IV entails a change in the DV.

Definition 4 (Validity [START_REF] Cook | Quasi-experimentation: Design and analysis for field settings[END_REF]). The best available approximation to the truth and falsity of propositions.

What we seek to validate are the statements, inferences and conclusions that we draw from results of empirical research [START_REF] Brewer | Research design and issues of validity. Handbook of research methods in social and personality psychology[END_REF]. We differentiate between internal and external validity.

Internal Validity. In most experiments, researchers are aiming to find out if IV A has an effect on DV B. If the experiment does not offer any alternative causes nor explanations on the outcome on B, then the experiment is internally valid.

Definition 5 (Internal Validity [START_REF] Brewer | Research design and issues of validity. Handbook of research methods in social and personality psychology[END_REF]). The truth that can be assigned to the conclusion that a cause-effect relationship between an IV and a DV has been established within the context of the particular research setting.

External Validity refers to the extent to which the study findings are generalizable from a laboratory setting to other settings.

Definition 6 (External Validity [START_REF] Brewer | Research design and issues of validity. Handbook of research methods in social and personality psychology[END_REF]). The question of whether an effect (and its underlying processes) that has been demonstrated in one research setting would be obtained in other settings, with different research participants and different research procedures.

Not all experiments can be both internally and externally valid. Depending on the purpose of the experiment, researchers need to make a trade-off.

Repeatability and Reproducibility

Replication is the attempt to recreate the conditions sufficient to obtaining a previous observed finding [START_REF]Open Science Collaboration: Estimating the reproducibility of psychological science[END_REF]. Scientific claims gain credence when their supporting evidence can be replicated [START_REF]Open Science Collaboration: Estimating the reproducibility of psychological science[END_REF]. Replication has been highlighted as a problem in scientific research. For example, the Open Science Collaboration [START_REF]Open Science Collaboration: Estimating the reproducibility of psychological science[END_REF] conducted a large-scale replication study (N = 100) of psychological journals and found that replication effects were on average half the magnitude of original effects.

We distinguish between repeatability and reproducibility as two conceptual frames for replication.

Definition 7 (Repeatability [START_REF] Everitt | Cambridge dictionary of statistics[END_REF]). The closeness of the results obtained in the same test material by the same observer or technician using the same equipment, apparatus and/or reagents over reasonably short intervals of time.

Definition 8 (Reproducibility [START_REF] Everitt | Cambridge dictionary of statistics[END_REF]). The closeness of results obtained on the same test material under changes of reagents, conditions, technicians, apparatus, laboratories and so on.

Remark 1 (Repeatability vs. Reproducibility).

While repeatability refers to replicating the experiment by keeping everything same (including the experimenter), reproducibility refers to altering specific components while keeping the design consistent, especially when the experiment is reproduced by an independent experimenter.

Hypothesis Testing

In this workshop summary, we limit the scope of our inquiry to hypothesis testing [START_REF] Fisher | Statistical methods for research workers[END_REF][START_REF] Neyman | On the use and interpretation of certain test criteria for purposes of statistical inference: Part i[END_REF][START_REF] Lehmann | The Fisher, Neyman-Pearson theories of testing hypotheses: One theory or two?[END_REF], a particular method of statistical inference that seeks to distinguish between hypotheses. We focus on making a decision between a null hypothesis and an alternative hypothesis.

Definition 9 (Hypothesis Testing).

(a) A statistical hypothesis test is a method of statistical inference in which a hypothesis of a proposed statistical relationship is compared to an idealized null hypothesis that claims there is no relationship. (b) The null hypothesis H 0 is the statistical hypothesis that there is no effect, no difference between conditions. (c) The alternative hypothesis H 1 is the statistical hypothesis that there is an effect, a difference between conditions.

Hypotheses are expressed on the population statistics, not the sample statistics.

Example 2 (Difference of Means). When considering the means across two conditions, the two hypotheses are:

-Null hypothesis H0: µ1 = µ2, -Alternative hypothesis H1: µ1 = µ2.

A sound procedure for hypothesis testing will proceed as illustrated in Fig.

Remark 2 (Controversy and Criticism

). There has been much controversy about hypothesis testing. Among its most vocal critics is Jacob Cohen [START_REF] Cohen | Things i have learned (so far)[END_REF]. First, we need to note that a statistically significant result only means that the effect is deemed not nil, nothing more. Slavishly following the "sanctification by significance" has been considered as one of great ailments in scientific reasoning. Nickerson [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF] offers a comprehensive overview of the controversies around Null Hypothesis Significance Testing (NHST), of which we highlight misconceptions on p-values in Remark 3 (Sect. 4.1).

Multiple proponents have argued to deprioritize hypothesis testing in favor of robust estimation with confidence intervals, e.g., as expressed by Gardner and Altman [START_REF] Gardner | Confidence intervals rather than p values: estimation rather than hypothesis testing[END_REF]. The American Psychology Association (APA) [START_REF]Publication Manual of the American Psychological Association. American Psychological Association (APA)[END_REF] has consequently made the reporting of confidence intervals a minimum standard.

While this workshop summary adheres to hypothesis testing, we advocate a cautious and diligent interpretation: Consider the size of effects investigated, the required power and sample size to detect those effects (cf. Sect. 4.2).

Checklist: Hallmarks

Make hypotheses falsifiable, i.e., construct them such that experiments or observations can show them to be false. Specify independent variables (IVs) and their manipulation. Operationalise dependent variables (DVs) and specify validated measurements. Explicitly declare null hypothesis and alternative hypothesis a priori. If feasible, prepare a randomized controlled testing the hypotheses. Control for confounders, e.g., by restricting variation or matching subjects. Establish to what extent a change in IV entails a change in DV. Report biases and assumptions that impact this entailment. Make it clear whether the study is repeating or reproducing existing research. Document recruitment, sampling, procedure, experiment design, manipulations, measurements, analyses clearly for forward reproducibility.

An Exercise in Experiment Design

We have prepared the ground by introducing hypothesis testing. Before we proceed with statistical inference in Sect. 4, we discuss an exercise in experiment design based on a hypothetical scenario.

Example 3 (Scenario "When the cat's away, the mice do play").

A security company observes that in the evenings-when the boss-is away more dangerous sites are accessed than during day times.

Developing Research Questions.

The participants are asked to answer the following questions:

1. What is an interesting research question (RQ) for the scenario of clicks to dangerous sites? 2. Independent Variable (IV): What factor influences the number of clicks on dangerous sites? 3. Dependent Variable (DV): How can we measure the outcome reliably? 4. What is a testable null hypothesis (H 0 )? 5. What is the alternative hypothesis (H 1 )?

We advocate for the simple example research questions to create a core experiment design to nail down how the IV is manipulated, how he success of this manipulation is checked, and how the DV is reliably measured. We offer a template in Fig. 2. Cf. Field and Hole [START_REF] Field | How to design and report experiments[END_REF] or Montgomery [START_REF] Montgomery | Design and analysis of experiments[END_REF] for experiment designs.

Manipulation (IV)

Control Condition Experiment Condition

Manipulation Check

Measurement (DV)

Fig. 2: A core template for a two-condition experiment with manipulation check.

The following examples were designed by workshop participants in roundtable discussions in response to the questions above.

Example 4 (Design Group 1).

RQ. How does the presence of the boss impact clicking dangerous links? IV. Presence of the boss. DV. #mistakes clicking dangerous links. H0. The mean number of mistakes is equal between the conditions "boss present" and "boss absent." H1. The mean number of mistakes is greater when the boss is absent.

Example 5 (Design Group 2).

RQ. How does cognitive workload impact clicking phishing links? IV. Cognitive workload. DV. #mistakes clicking phishing links. H0. The mean number of phishing mistakes is equal between the depleted experiment condition and the non-depleted control condition. H1. The mean number of phishing mistakes is greater in the depleted condition than in the control condition.

Example 6 (Design Group 3).

RQ. How does down-time impact clicking of dangerous sites? IV. Down-time without customer. DV. #mistakes accessing dangerous sites. H0. The mean number of accesses to dangerous sites is independent from the measured downtime. H1. An increased down-time implies an increased mean number of accesses to dangerous sites. We see from this example that for a given scenario a variety of relevant research questions and operationalizations in statistical hypotheses is possible. Consequently, it is crucial to write down precisely what is being investigated before the experiment is designed. The key points here are to commit to the independent and dependent variables, to settle the manipulation and measurement methods used, and to express the null and alternative hypotheses in the exact terms of these variables.

Structured Abstract.

We recommend a structured abstract as a concise tool of stating the intention of an experiment (in less than one page). The structured abstract covers 1. Background. The motivation and theoretical context of the experiment. Background. Psychology research predicts an impact of tiredness on decision making. Aim. We investigate the impact of tiredness on mistakes on phishing clickthroughs.

Statistical Inference and Power

Method. Two groups of 20 participants each were asked to evaluate 50 mixed e-mails (25 phishing), one group was tired, the other was not. We compared the number of mistakes across groups.

Results. The mean number of mistakes of the tired group (M E = 13.9, SD E = 5.77) was significantly greater than that of the control group (M C = 10.75, SD C = 3.75), two-tailed t(38) = -2.047, p = .049, 95% CI[0.18, 6.28].

We observed a medium effect size (d = 0.68). a The experiment achieved a power of 55%.

Impact. Tired users succumb to phishing.

a Reporting confidence intervals (CI) and the effect size as mandated by the APA guidelines [START_REF]Publication Manual of the American Psychological Association. American Psychological Association (APA)[END_REF].

Statistical Inference

As we have seen in Sect. 2.5, we seek to decide between the null hypothesis H 0 and the alternative hypothesis H 1 . We do not know what the situation in reality is: whether H 0 is true or false. All we can do is making an observation (in an experiment) and base a decision to reject or accept H 0 on the likelihood of that observation. Because H 0 and H 1 are meant to be complements, we end up with four decision outcomes summarized in Tab. 1.

Let us consider the left-hand column of Tab. 1 first: In reality, the null hypothesis H 0 is true. We specify in advance a significance criterion α, which quantifies the likelihood of mistakenly rejecting the null hypothesis H 0 . As Cohen formulates it [START_REF] Cohen | A power primer[END_REF], "α represents a policy: the maximum risk of attending such a rejection." If we reject H 0 even though H 0 is true, we commit a Type I Error.

If we correctly reject the alternative hypothesis H 1 and hence accept the null hypothesis H 0 , we do so at a confidence level 1 -α.

Test Statistics and p-Value. We conduct statistical tests to evaluate how likely the observation is, assuming the null hypothesis to be true.

Definition 10 (p-Value [22]).

A p-Value is the probability of data as extreme or more extreme as that obtained, computed under the presumption of the truth of the null hypothesis H 0 . In symbols, if we let D stand for data as or more extreme as that obtained, then a p-value is the conditional probability p = Pr (D|H 0 ).

Hypothesis testing with significant p-values attempts a statistical proof by contradiction indirectly. If the p-value is smaller than the specified level of significance α, we call a test statistic statistically significant and are entitled to reject the null hypothesis H 0 .

Remark 3 (p-Value Misconceptions).

Unfortunately, p-values are often misinterpreted, even in text books. Maxwell and Delaney [22, p.48] as well as Nickerson [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF] offer some pointers for typical misinterpretations. (a) We emphasize that in almost all cases, it holds that

p = Pr (D|H 0 ) = Pr (H 0 |D).
Considering these two conditional probabilities equivalent is a fallacy, called "the confusion of the inverse." (b) It is also a grave mistake to believe that p is the probability of the null hypothesis being true. (c) The likelihood of the alternative hypothesis H 1 is only indirectly related to the p-value [START_REF] Wacholder | Assessing the probability that a positive report is false: an approach for molecular epidemiology studies[END_REF][START_REF] Ioannidis | Why most published research findings are false[END_REF]. 4 Cohen [START_REF] Cohen | Things i have learned (so far)[END_REF] is vocal that the p-value "cannot tell us anything about the probability that the [alternative] hypothesis is true." (d) Note especially that p = Pr (D|H 0 ) is not a complement of Pr (D|H 1 ) [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF].

Effect Size and Power

Cohen [START_REF] Cohen | A power primer[END_REF] exhorts that an effect that is statistically significant is not necessarily scientifically significant or important. The importance of an effect is largely linked to the magnitude of the effect. For the example of the difference between two means, we are interested how large the difference between the two populations is, and whether it constitutes a non-trivial difference.

Effect Size. We seek to quantify of the magnitude of an effect. Definition 11 (Effect Size [START_REF] Cohen | A power primer[END_REF]). The effect size (ES) is the degree to which H 0 is false. It is indexed by the discrepancy between H 1 and H 0 . Each statistical test has its own ES index. All the indexes are scale free and continuous, ranging upward from zero, and for all, the H 0 is that ES = 0.

The importance of a significant effect with effect sizes is considered that crucial in the science, that the American Psychology Association (APA) [START_REF]Publication Manual of the American Psychological Association. American Psychological Association (APA)[END_REF] states that "estimates of appropriate effect sizes [. . . ] are the minimum expectations."

There are two main families of effect sizes [START_REF] Ellis | The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results[END_REF]: (a) the d family, assessing the differences between groups, and (b) the r family, measuring the strength of a relationship. Effect sizes can be further specified by, for instance, regression 4 Conditional probabilities follow Bayes' Theorem, Pr (H1|D) = Pr (D|H1)Pr (H1) Pr (D|H1)Pr (H1) + Pr (D|H0)Pr (H0) .

Nickerson [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF] discusses the links and caveats in depth.

coefficients or odds ratios. In this workshop summary, we focus on the d-family of effect sizes, especially on the difference between two means, measured with Cohen's d. We refer to Cohen [START_REF] Cohen | A power primer[END_REF][START_REF] Cohen | Statistical power analysis for the behavioral sciences[END_REF], Ellis [START_REF] Ellis | The essential guide to effect sizes: Statistical power, meta-analysis, and the interpretation of research results[END_REF] and Fritz et al. [START_REF] Fritz | Effect size estimates: current use, calculations, and interpretation[END_REF] for overviews of different effect size types and their calculations.

Power. Now we are prepared to consider the right-hand side of Tab. 1: How do we fare in a situation in which the null hypothesis H 0 is actually false? If we accept the null hypothesis H 0 mistakenly even though the alternative hypothesis H 1 is true, then we have committed a Type II Error. The likelihood of committing such an error is called β.

Consequently, if we are in the case that the alternative hypothesis H 1 is actually true, and we make a correct decision to reject the null hypothesis H 0 we do so at the likelihood of the power of our test.

Definition 12 (Power [START_REF] Cohen | A power primer[END_REF]). The statistical power of a significance test is the long-term probability, given the population ES, α, and N of rejecting H 0 . Power is 1 -β, the probability of rejecting a false H 0 .

The four quantities sample size N , effect size (ES), significance level α and power 1 -β are mathematically connected; given three of them, the fourth quantity can be computed. We recommend G*Power [START_REF] Faul | G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences[END_REF] for this computation.

In authoritative experimentation, we seek to create experiments with sufficient power (as a commonly used rule-of-thumb, 1 -β > .8) to have a sufficient likelihood of correctly rejecting the null hypothesis H 0 . Cohen [START_REF] Cohen | The statistical power of abnormal-social psychological research: a review[END_REF] and others have observed time and again an abysmal lack of power in scientific experiments. Fig. 3: Hypothesis testing with null hypothesis H 0 test distribution on the left and alternative hypothesis H 1 test distribution on the right. The null hypothesis is rejected if the a critical value is passed. The graph marks the critical areas for α and β, that is, the likelihoods to make Type I and Type II errors.

Underpowered Experiments. For research in privacy and identity management, we anticipate that the power of experiments is often too low, below 1 -β = .5, and the likelihood of correctly rejecting the null hypothesis basically a coin toss. We believe the experimenters underestimate the sample size, because of a missing understanding of effect sizes and a priori power analysis. Fig. 4 illustrates the sample sizes needed for different levels of power. Desired Power 1-beta > .8

Fig. 4: Power achieved for different effect sizes and sample sizes. It is apparent that a desired power of more than 1 -β = .8 needs large sample sizes N for smaller effect sizes d. (Here for an one-tailed independent samples t-test, α = .05)

Remark 4 (N = 30 debunked). There was a myth of a "rule-of-thumb" to run experiments with a per-group sample size of N = 30. This was debunked by Jacob Cohen [START_REF] Cohen | Things i have learned (so far)[END_REF], Some Things You Learn Aren't So. The sensitivity of two-tailed independent samples t-test for significance level α = .05 and power 1 -β = .95 implies required effect size: d = 0.94 (large). Smaller effect sizes d will not be detected at this power. At a medium effect size (d = .5), such an experiment will only achieve a power (1 -β) = .48, a coin toss.

High-Powered Experiments. As Cohen argues [START_REF] Cohen | Things i have learned (so far)[END_REF], the null hypothesis-that there is no effect whatsoever-is never actually true in reality. With a large enough sample-and thereby large enough power-even infinitesimal effects can still be detected with statistical significance (cf. Fig. 5). Consequently, it is crucial to put the hypothesis testing and the rejection of the null hypothesis with statistical significance into context of the sample size and achieved power.

Checklist: Statistical Inference

Table 2 contains further reading. Specify the exact contents of the statistical inference precisely, e.g., in a structured abstract naming IVs, DVs and hypotheses. Choose relevant test statistics and evaluate their assumptions carefully. Conduct an a priori power analysis to determine the required sample size for a committed significance level α and an appropriate power 1 -β > 80%.

Exercise diligence in interpreting p-values and significance, putting them into context with effect sizes and the post-hoc power the experiment achieved. Fig. 5: α-probability for different sample sizes N and effect sizes d (one-tailed independent-samples t-test at 1-β = .95), illustrating that-with a large enough sample size-even smallest effects can be detected with statistical significance.

Table 2: Further reading on statistical inference and power.

Reference Title Comment

Montgomery 2012 [START_REF] Montgomery | Design and analysis of experiments[END_REF] Design and Analysis of Experiments Detailed treatment of design and analysis of experiments. Howell 2012 [START_REF] Howell | Statistical methods for psychology[END_REF] Statistical Methods for Psychology

Statistics for experiments with human factors. Cohen 1992 [START_REF] Cohen | A power primer[END_REF] A Power Primer The quintessential concise introduction to effect size and power. Fritz et al. 2012 [START_REF] Fritz | Effect size estimates: current use, calculations, and interpretation[END_REF] Effect size estimates Survey of the use of effect size types incl. best practices for their computation and transformation.

Report the results following the APA Guidelines [START_REF]Publication Manual of the American Psychological Association. American Psychological Association (APA)[END_REF], especially by reporting appropriate effect sizes and confidence intervals. Include all data needed to recompute the results and their effect sizes (test statistics, standard deviations, coefficients, etc.).

Participant Feedback

We asked participants to fill in a questionnaire just before starting the workshop.

We summarize the outcomes of the 12 respondents in Fig. 6.

Area of Privacy Research

Second we asked participants "What area(s) of privacy do you research/interests you?" Of the 12 participants, six reported the human dimension of privacy, three reported technological aspects (geolocation and transparency), one reported the societal dimension and personal data (P3: "personal data, social exclusion, effect on the use of personal data"), one reported legal dimension in the context of health-care (P12: "Privacy for health-care systems, compliance with legal frameworks of privacy and data protection"), one philosophical dimension. The human dimension responses covered aspects of P1: "attitude, behaviour, decision-making;" P2: "HCI, usability;" P8: "views of privacy among 'normal' people;" P9: "perceptions of privacy, how to make people more aware;" P10: "genomic privacy;" P11: "corruption and human behaviour."

Methodological Issues

Third, we asked participants "What methodological issues have you encountered?" From the 12 response sheets, five were excluded as they did not answer the question. Validity was the most recurring response (six), with three participants pointing towards internal validity: P11 "measuring what is intended;" P4 "software can have errors and it is unclear if experimental results can be caused by error programs." One participant stated on validity and confounders, P9 "not understanding behavioural issues [. . . ] may ruin months of data gathering." Three responses were about external validity. P4 and P9: "generalisability;" P4 and P5: "representativeness." Two responses were about data: P1: "difficulty investigating categorical data" or large-scale data gathering (P3) The other responses included P1: "sample size;" P8 mentions running timely rigorous experiments: "experiments that can be done rigorously yet in a timely manner."

Opinion on Experimental Research in Privacy

Fourth, we asked "What is the state of experimental research in privacy?" From the 12 participants, five were new to privacy or reported they did not know. Two participants pointed to weaknesses, with P9 suggesting the state to be "dubious" and P11: "fairly poor." Two participants suggested an early state, P10, suggested experimental research in privacy is in "early development;" P5 stated it is "limited." Two participants suggested the state is progressing, with P1: "there are many longitudinal studies on privacy behaviour" and P6: "in progress." One participant, P4, suggested that it depends on the subject.

Requirements on Workshop

Fifth, we asked "What would you like to learn in this workshop today?" Of the 12 participants, six either did not provide a response or understand the question; six other thought the workshop would improve their skills in one way or another. Two participants were concerned with experiment design; P8 thought of "Good ways of running experiments;" P9 expected "better ideas to design experiments." P11 sought to learn about: "frameworks." P2 mentioned "tips and trades (sic) in ways forward;" and P1 expected help on "how to perform research with less mistakes."

Conclusion

From the participants' feedback in Sect. 5 we see that, on the one hand, participants are interested in research that lend themselves to evidence-based methods, such as human dimensions of privacy and identity. On the other hand, they report a dire situation of the state of play in the field and a need to learn more on research methodology.

"[Perhaps]

, we should simply study our Mr. Pritchard and learn our rhyme and meter and go quietly about the business of achieving other ambitions."

-John Keating, Dead Poets Society Should we simply run studies that receive "pass" marks in our communityignoring the depths of evidence-based methods-and go quietly about the business of achieving other ambitions? Mastering evidence-based methods is a challenging prospect, daunting at times. However, what is at stake here is our capacity as a community to truly learn from our research and advance our field's body of knowledge. Consequently, we certainly advocate going deep in research methodology. How? From the workshop experience, we believe there are three key ingredients that are reinforcing each other. (a) First, we would focus on the tenets of reaching clarity on research questions, hypotheses and variables, ideally specified in concise structured abstracts. (b) Second, we advocate the specification of sound experiment designs that not only replicate validated methods but also specify their components in such detail that they propagate forward reproducibility by other investigators. (c) Finally, we stress the importance of the quantitative tools from correct statistical inference, over effect sizes and power, to interval estimation, all strengthening the reliability of the reported results. We are convinced that these three ingredients are essential to advance the body of knowledge of our field.

Fig. 1 :

 1 Fig. 1: Simplified process of hypothesis testing.

Fig. 6 :

 6 Fig. 6: Feedback of 12 participants

  We will discuss statistical inference and p-values in Sect. 4.1.

	Research Question	Hypotheses	Stats Setup	Statistical Inference
	Relevant?	Testable? Falsifiable?	Assumptions? Test Statistic?	Evaluate Test Statistic
		Null	Significance	Critical Region
		Hypothesis	Level?	p-Value
		Alternative Hypothesis	Power? Sample Size?	Reject H 0 /H 1

1. 1. State null hypothesis H 0 and alternative hypothesis H 1 explicitly, first. 2. Evaluate the statistical assumptions made, select a relevant test statistic, and select a significance level α, a probability threshold below which the null hypothesis will be rejected (cf. Sect. 4.1). 3. Evaluate the statistical inference by calculating the test statistic. Reject the null hypothesis if and only if the p-value is less than the specified significance level α.

  2. Aim. The goal of the experiment expressed in one concise sentence. 3. Method. The concise method of the experiment, including sample size, group design, what is manipulated (IV) and what is measured (DV). 4. Expected Results. The factual outcomes expected from the experiment. 5. Expected Impact. So what? What does the experiment mean? A structured abstract is a superb tool in reporting findings soundly and endorsed by specialist venues, such as Learning from Authoritative Security Experiment Results (LASER) 3 . Example 7 reports the outcome of the scenario study.

Table 1 :

 1 Statistical inference decision matrix.

	Reality	H0 is TRUE.	H0 is FALSE.
	Decision	H1 is FALSE.	H1 is TRUE.
	We reject H1.	Confidence Level TYPE II ERROR 1 -α β
	We reject H0.	TYPE I ERROR α	POWER 1 -β

http://www.laser-workshop.org

Acknowledgments

We are grateful for the contributions and feedback from participants of the workshop. We are grateful for the discussions with Roy Maxion on evidencebased methods for cyber security. The preparation of the evidence-based methods workshop was in parts funded by the EPSRC Research Institute in Science of Cyber Security (RISCS), grant EP/K006568/1. This work was supported by a Newcastle-sponsored International Research Collaboration Award (IRCA) for work with Carnegie-Mellon University (CMU).