Kovila P L Coopamootoo
email: kovila.coopamootoo|thomas.gross@newcastle.ac.uk

Thomas Groß

Cyber Security & Privacy Experiments: A Design & Reporting Toolkit

Keywords:

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Cyber security and privacy are both exciting fields that weave together methodologies, theories and perspectives from various disciplines: mathematics, engineering, law, psychology and social sciences. As consequence, it gains a collective tapestry, a definite strength that exemplifies inter-disciplinary fields. However, the sharing of expertise and drawing on best practices of each discipline is a challenge. For example, the research area of human factors of cyber security and privacy is inter-disciplinary research area. It clearly benefits from the systematic design and reporting standards characteristic of the rigorous methodology of experimental psychology at its best.

Outline. In the rest of the paper, we discuss our choice for the set of indicators before detailing each of the nine completeness indicators. For each indicator we proceed with theoretical background, benefits for fulfilling the indicators, outcome of not achieving them, practical steps to take in design and reporting, together with best practice examples. We then provide the lessons learnt from a first connection with the community before providing the discussion and conclusion.

Choice of Completeness Indicators

We chose indicators that contribute and build-up towards sound statistical inference. As a consequence, we addressed reproducibility, internal validity, correct statistical reporting and parameter estimation. We deliberately excluded criteria on external validity and ethics, but may consider them in future versions of the toolkit.

Benefits of this first toolkit. The indicators are designed as a toolkit mainly for researchers, providing both a theoretical and a practical component. First, it acts as support for the design phase of a user study/experiment and aims to be one-stop resource. We provide a theoretical background with each indicator, substantiated with reasoning in the form of benefits for having the indicators and the outcome of not catering for them. Second, it acts as a companion for reporting via the practical steps to take, typical locations in articles and examples of good practice. Further, an additional benefit, is a clear list that can enable program committees to evaluate the reporting of research studies.

CI1: Upstream Replication

Theoretical Background

Similar to Coopamootoo & Groß [START_REF] Coopamootoo | Evidence-based methods for privacy and identity management[END_REF], we call replication the attempt to recreate the conditions sufficient to obtaining a previously observed finding, a definition adapted from the Open Science Collaboration [START_REF] Collaboration | Estimating the reproducibility of psychological science[END_REF]. We refer to upstream replication when a study replicates existing studies or previously validated methods/instruments. We note that replication of studies is an important research practice that provides confidence in the findings, where Cumming & Calin-Jageman [START_REF] Cumming | Introduction to the new statistics: Estimation, open science, and beyond[END_REF] point out that rarely, if ever can a single finding give definitive answer to a research question, while the Open Science Collaboration notes the alarming discovery that a number of widely known and accepted research findings cannot be replicated [START_REF] Cumming | Introduction to the new statistics: Estimation, open science, and beyond[END_REF].

We ask: 'Is the study reporting existing studies or methods?'

Benefits of fulfilling CI1. Researchers engaging in upstream replication are gaining sound foundations for their studies in employing methods whose exact properties are known and well-tested. For instance, for a measurement instrument we expect it to be known, which parameters of the population are measured. We expect of the instrument itself internal validity, repeatability and reproducibility. In the logic of the statistical inference of the given experiment we are then entitled to assume that the properties of the instrument are a given and will be the same for other researchers in the future. As a completeness indicator, CI1 thereby yields evidence whether the foundations of the given reported study are sound.

Outcome for not fulfilling CI1. Should evidence towards CI1 be missing, we would need to assume that the study did not pay attention to its sound foundations. This, in turn, means that the study is on uncertain footing. For the manipulation instruments, it is not assured that they cause the intended change in the participants reliably. For the measurement instruments, it is not assured that they measure the intended property. Consequently, instruments without evidence of sound a priori validation yield sources of errors that can well confound the main experiment and thereby put the overall inference in question.

Steps to Take

How to achieve CI1. The key principle towards gaining evidence for CI1 is the use of validated tools. We recommend to select manipulation and measurement instruments that come with strong evidence of their validation and properties.

In the experiment design and execution, researchers will employ instruments exactly as validated, for example, using the defined scale and scoring sheet as provided. If adaptations are made to the instrument instructions, these will be documented.

In the reporting of the study, researcher will then document the evidence for their exact replication, for instance, by including the exact materials used and by citing the validation study they rely upon.

We note that documentation of instruments also apply to those employed for manipulation checks.

Typical location in articles. CI is reported in the methods section with subsections on measurement apparatus and manipulation apparatus or experimental conditions.

Reporting Example.

Example 1 (CI1 -Manipulation Apparatus, with amendments from Nwadike et al. [START_REF] Nwadike | Evaluating users' affect states: Towards a study on privacy concerns[END_REF]).

We induce a happy and sad affect via video stimulus, a mood induction protocol recommended by Westermann's critical review of different methods [START_REF] Westermann | Relative effectiveness and validity of mood induction procedures: analysis[END_REF]. For happiness affect we used the restaurant scene from the movie When Harry meets Sally [clip length 155 seconds] while for sadness affect we used the dying scene from the movie The Champ [clip length 171 seconds]. We refer to Rottenberg et al. [START_REF] Rottenberg | Emotion elicitation using films. handbook of emotion elicitation and assessment[END_REF] to start and end the clips at the exact frames as previously validated.

Example 2 (CI1 -Manipulation Check, with amendments from [START_REF] Nwadike | Evaluating users' affect states: Towards a study on privacy concerns[END_REF]). We used the 60-item full PANAS-X questionnaire [START_REF] Watson | The panas-x: Manual for the positive and negative affect schedule-expanded form[END_REF] as manipulation check on the induced affect state. We focus on sadness and joviality as equivalent of happiness. The PANAS-X is scale is based on 5-point Likert-items anchored on 1 -very slightly or not at all, 2 -a little, 3 -moderately, 4quite a bit, and 5 -extremely. We anchored PANAS-X for affect "at the present moment."

Example 3 (CI1 -Validated Measurement Apparatus).

"The State-Trait Anxiety Inventory for Adults (STAI-AD) [START_REF] Spielberger | Manual for the state-trait anxiety inventory[END_REF] is a 40-question self-report questionnaire. We use the temporary construct of state anxiety, that is, "how you feel right now." It employs 4-point Likert items anchored on 1 -Not At All, 2 -Somewhat, 3 -Moderately So, and 4 -Very Much So."

Further Sources

Across sciences, a replication crisis has been observed. Prominently in psychology, a large scale replication endeavor by the Open Science Collaboration [START_REF] Collaboration | Estimating the reproducibility of psychological science[END_REF] of N = 100 studies across 3 psychology journals found that only 47% of the original effect sizes were in the 95% confidence interval of the replication effect size.

The Open Science Collaboration makes a case that research claims gain credibility when the supporting evidence undergoes sound replication [START_REF] Collaboration | Estimating the reproducibility of psychological science[END_REF]. We note that the replication needs to be done deliberately to increase the overall Positive Predictive Value of the results [START_REF] Ioannidis | Why most published research findings are false[END_REF][START_REF] Moonesinghe | Most published research findings are false-but a little replication goes a long way[END_REF].

In security literature, Maxion [START_REF] Maxion | Making experiments dependable[END_REF] postulates that repeatability, reproducibility and validity are the main criteria differentiating a well designed experiment from those that are not.

CI2: Reproducibility

Theoretical Background

CI2 considers the enablement of downstream replication. While downstream replication includes repeatability, that is, whether a study can be replicated by the same researchers, CI2 considers especially, whether the study is sufficiently reported to be reproducible by other researchers. We refer to Maxion [START_REF] Maxion | Making experiments dependable[END_REF] for further discussion on repeatability and reproducibility.

CI2 establishes whether the reporting supports reproducibility, defined as the closeness of results obtained on the same test material under "changes of [. . . ] conditions, technicians, apparatus, laboratories and so on" [START_REF] Everitt | Cambridge dictionary of statistics[END_REF]. A key requirement of replicating existing studies is the availability of clear documentation.which ideally would entail a detailed step-by-step experimental protocol, which makes provisions for reproducibility.

The principle for reproducibility is diligent documentation of all variables of the study's lifecycle. We ask 'Is there correct reporting of manipulation apparatus, measurement apparatus, detailed procedure, sample size, demographics, sampling and recruitment method, contributing towards reproducibility?' Benefits of fulfilling CI2. Offering sound reporting for reproducibility allows for downstream replication and contributes to the enablement of research synthesis in a field. This is crucial to enable falsification and hence empirical progress. Having a reproducible study at hand means that other researchers can test the theories evaluated in the given study and establish independent evidence on the theories, possibly falsifying the earlier result. Furthermore, replication studies inform the overall positive predictive value for the considered relations and allow for a meta analysis on the effect sizes and their confidence intervals.

Hence, as completeness indicator, CI2 checks whether evaluates whether the theories named in the given study can be empirically scrutinized in subsequent experimentation from the given reporting, and thereby whether the given study makes a sound contribution to empirical sciences.

Outcome for not fulfilling CI2. Should the evaluation for CI2 not offer evidence towards reproducibility, we need to assume that the given study cannot be replicated downstream. First, the lack of reproducibility leaves other researchers with a great ambiguity what was actually done. Second, following Poppers discussion on falsifiability [START_REF] Popper | The logic of scientific discovery[END_REF], a study that cannot be reproduced does not actually yield strong empirical evidence because other researchers cannot execute the offered experiment to falsify the reported theory, which in turn casts doubt on the study advancing empirical knowledge.

Steps to Take

How to achieve CI2. Researchers will provide detailed description of experiment design, including the all choices made, possibly supplemented by an experiment diagram, as well as the procedure executed in the experiment itself.

We note that documentation towards reproducibility will often also include planned analyses, which we consider under other CIs. A recommended practice in this case is to pre-commit the experiment and analysis plan at organizations such as the Open Science Framework1 or AsPredicted2 . As example, committed analysis plan and analysis report [START_REF] Groß | Analysis report -investigation of the effect of fear and stress on password choice[END_REF] published for password research [START_REF] Fordyce | Investigation of the effect of fear and stress on password choice[END_REF].

Typical location in articles. C2 covers the whole method section including a detailed procedure, sample recruitment and demographics, manipulation and measurement instruments. Planned analysis will be in the analysis or the results section.

Reporting Example. Example 4 (CI2 -Demographics).

We refer to Table 3 of Kluever and Zanibbi [START_REF] Kluever | Balancing usability and security in a video captcha[END_REF] for a detailed demographics report that is relevant to the context of the study reported.

Example 5 (CI2 -Measurement Apparatus precisely referencing sources).

"We administered the NASA Task Load Index in an online form. The form exactly replicated the full NASA TLX questionnaire as specified on in NASA Task Load Index (TLX), v. 1.0, Appendix, pp. 13. [START_REF] Hart | Development of nasa-tlx (task load index): Results of empirical and theoretical research[END_REF]" Example 6 (CI2 -Procedure).

"The procedure consisted of (i) pre-task questionnaires for demographics and personality traits, (ii) a manipulation to induce cognitive depletion, (iii) a manipulation check on the level of depletion, (iv) a password entry for a mock-up GMail registration, and (v) a debriefing and memorability check one week after the task with a GMail login mockup. " This was followed with a details of each section.

Further Sources

First, for reproducibility of the experiment design, which is what this CI mainly focuses on, we refer to experiment design methodology [START_REF] Field | How to design and report experiments[END_REF][START_REF] Maxwell | Designing experiments and analyzing data: A model comparison perspective[END_REF][START_REF] Montgomery | Design and analysis of experiments[END_REF].

Second, for reproducibility of the planned analyses, which involves the documentation of the plan as well as the recording of all the analyses done, we suggest inspiration from reproducibility principles from general computing science research [START_REF] Sandve | Ten simple rules for reproducible computational research[END_REF] or more specific sources with focus on computation-supported scientific practice [START_REF] Stodden | Implementing reproducible research[END_REF]. To render all computations, statistical analyses and graphs reproducible, we suggest the R framework knitr [START_REF] Xie | knitr: a comprehensive tool for reproducible research in r[END_REF] as demonstrated within the analysis report [START_REF] Groß | Analysis report -investigation of the effect of fear and stress on password choice[END_REF].

CI3: Internal Validity

Theoretical Background

CI3 addresses internal validity of the experiment, which refers to the truth that can be ascribed to cause-effect relationships between independent variables (IV) and dependent variables (DV) [START_REF] Brewer | Research design and issues of validity[END_REF], where the IV is a variable that is induced/manipulated and the DV is the variable that is observed/measured [START_REF] Miller | Experimental design and statistics[END_REF].

This CI asks for research questions and hypotheses that provide the foundations for null hypothesis significance testing (NHST) [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF]. Operationalization enables systematic and explicit clarification of the predictors or independent variables, and hence the cause and manipulation, while the target variable or dependent variables clarify the effect, hence the measurements. Subject assignment points to whether and how participants were randomly assigned and balanced across experimental conditions.

Manipulation check refers to verification that the manipulation has actually taken effect, hence assuring systematic effects.

We ask 'Is there an explicit and operational specification of the RQs, null and alternative hypotheses, IVs, DVs, subject assignment method and manipulation checks?' Benefits of fulfilling CI3. CI3 ensures internal validity and a solid statement of intention for Null Hypothesis Significance Testing (NHST) [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF].

Outcome for not fulfilling CI3. Should evidence for CI3 be missing, we would need to assume other possible explanations for the cause-effect relationship investigated, that is that the reported design could involve variables contributing unsystematic effects. This in turn would mean that other researchers could not rely on the results reported.

Steps to Take

How to achieve CI3. We propose in the first instance that following the step by step exercise we previously detailed [START_REF] Coopamootoo | Evidence-based methods for privacy and identity management[END_REF] on 'An Exercise in Experiment Design' to be beneficial for internal validity. In particular, developing research questions, defining testable hypotheses, operationalizing hypotheses into IVs and DVs. For IVs, reseachers will answer 'What factor is being manipulated and influences the outcome?' For DVs, 'What is being measured?' and how can we measure the outcome of manipulation reliably.

Typical location in articles. The aims section can detail the research questions and hypotheses where as the method to include sub-sections on operationalizing the variables into measures and experimental conditions. The method section will also include subject assignment information.

Reporting Example.

Example 7 (CI3 -Research Question, from Cherapau et al. [START_REF] Cherapau | On the impact of touch id on iphone passcodes[END_REF]).

"How availability of Touch ID sensor impacts users' selection of unlocking authentication secrets?".

Example 8 (CI3 -Hypotheses, from Cherapau et al. [START_REF] Cherapau | On the impact of touch id on iphone passcodes[END_REF]).

For null hypotheses H0: "Use of Touch ID has no effect on the entropy of passcodes used for iPhone locking." or "Availability of Touch ID has no effect on ratio of users who lock their iPhones." For corresponding alternative hypotheses H1: "Use of Touch ID affects the entropy of passcodes used for iPhone locking." or "Availability of Touch ID increases the ratio of users who lock their iPhones" [START_REF] Cherapau | On the impact of touch id on iphone passcodes[END_REF].

Example 9 (CI3 -Subject Assignment, amended from Bursztein et al. [START_REF] Bursztein | How good are humans at solving captchas? a large scale evaluation[END_REF]).

"Our task scheduler presented the CAPTCHAs to Turkers in the following way . . . Random Order -fully random, where any captcha from any scheme could follow any other."

We also refer to Example 2 for manipulation checks.

CI4: Limitations

Theoretical Background

CI4 establishes what other factors could affect the cause and effect relationship under investigation and hence limit validity including both internal and external validity. This CI is related to the requirement of controlled variables for experiment design, that is the assurance that an observed change in the dependent variable is a result of a systematic change in the independent variable [START_REF] Miller | Experimental design and statistics[END_REF].

We ask 'Was there a discussion on the limitations, possible confounders, biases and assumptions made?' Benefits of fulfilling CI4. CI4 provides transparency of validity and assurance that other possible explanations for the stated causal relations, have been considered. This in turn provides confidence in the reported results.

Outcome for not fulfilling CI4. Should the limitations not have been discussed in the experiment report, we would need to assume that the researchers might have failed to control variables that impact the internal validity of the experiment. This puts the reported effects into question.

Steps to Take

How to achieve CI4. Researchers are (1) to evaluate experimental designs for alternative explanations that could influence the observed effects, such as identifying confounding and controlling for variables, (2) to make explicit the boundaries and of the design, such as whether a convenient sample was used, and (3) acknowledge the limits in interpretations that can be inferred from the findings, such as whether the results are a correct reflection of estimates for the general population.

A discussion of the limits and boundaries of the study, identification of possible confounding variables whose presence affect the relationship under study, and possible assumptions made in setup, are all valuable inputs that strengthen the validity of the experiment.

Typical location in articles. While researchers may report and discuss limitations throughout the article, it is preferred to define a dedicated limitations section, that shows clarity and researcher awareness of the limits of their design.

Reporting Example.

Example 10 (CI4 -Sampling bias, from Akhawe & Felt [START_REF] Akhawe | Alice in warningland: A large-scale field study of browser security warning effectiveness[END_REF]).

"The participants in our field study are not a random population sample. Our study only represents users who opt in to browser telemetry programs. This might present a bias. The users who volunteered might be more likely to click through dialogs and less concerned about privacy. Thus, the clickthrough rates we measure could be higher than population-wide rates."

CI5: Reporting Standard

Theoretical Background

Statistical reporting guidelines helps the reader, reviewer, policy maker to gain confidence in the reported statistical analysis and results. As example, we propose reporting recommendations of the American Psychology Association (APA) [2] as quality standard.

We ask 'Was the result reported in the APA style?'

Benefits of fulfilling CI5. Reporting standards provide a degree of comprehensiveness in the information that is reported for empirical investigations. Uniform reporting standards make it easier to generalize within and across fields, to understand implications of individual studies and supports research synthesis.

Comprehensive reporting also supports decision makers in policy and practice towards understanding how the research was conducted [2].

Outcome for not fulfilling CI5. The impact of not fulfilling CI5 opens gaps and lead to questioning research quality, reuse and reproducibility.

Steps to Take

How to achieve CI5. Researchers are to closely adhere to statistical reporting standards such as the APA [2] and reporting statistical inference as recommended whether in paragraphs, tables or figures. This include reporting actual p-values, that is not only whether the p-value is less that α, and effect sizes and confidence intervals.

Typical location in articles. Reporting standards usually focus on the specification of the results section, yet can also indicate the format of a structured abstract or the structure of the overall publication.

Reporting Example.

Example 11 (CI5 -with amendments from Coopamootoo et al. [START_REF] Coopamootoo | An empirical investigation of security fatigue -the case of password choice after solving a captcha[END_REF]).

We computed a one-way ANOVA. "There was a statistically significant effect of the experiment condition on the password strength score, F (2, 63) = 6.716, p = .002 < .05. We measure the effect size . . . η 2 = .176, 95% CI [0.043, 0.296] [. . . ]."

8 CI6: Test Statistic

Theoretical Background

The reporting on the test statistic offers a precise interface on the result of the computed statistical analysis. This data allows for a future analysis of a posteriori likelihoods, such as in a Positive Predictive Value (PPV) [START_REF] Ioannidis | Why most published research findings are false[END_REF]. Simply put, this data helps other researchers to ascertain whether the result could be a false positive or not. We consider the precise documentation of the outcome of the statistical test. For instance, for a t-test we would expect to learn the t-value as well as the degrees of freedom, along with the exact p-value computed for this t.

We ask 'Did the result statement include test statistic and p-value?'

Benefits of fulfilling CI6. If the test statistic is fully specified, we gain important data for the subsequent analysis of the result. From the consistency of the reported test statistic and the p-value, we gain confidence in the correct reporting.

In addition, the data includes sufficient redundancy that others can validate the presented p-values or use the reporting of the test statistic to compute standardized effect sizes for subsequent meta-analysis.

Outcome for not fulfilling CI6. Should the test static or the p-values not be reported, e.g., by just stating that the result "is statistically significant, p < .05, we lose a lot of information. We could neither ascertain the confidence level of the significance nor the internal consistency of the reported test. Hence, the reported result will lack internal credibility and not be particularly trustworthy.

Steps to Take

How to achieve CI6. The key principle is to report sufficient data, such that others can cross-check the reported values and use them in further research synthesis. Usually, this involves reporting the test statistic itself, the degrees of freedom vis-à-vis of the sample size, and the exact p-value. When comparisons between conditions are made, then the descriptive statistics for the relevant conditions should be provided (e.g., mean and standard deviation for conditions of a t-test).

Typical location in articles. The test statistics will be specified in the results section of the paper. As a rule of thumb, for each result we claim as being statistically significant, we will provide the test statistic supporting that claim as suffix.

Reporting Example.

We refer to the Example 11 for test statists and p-value reporting.

9 CI7: Assumptions

Theoretical Background

Statistical tests can easily lead us astray if their assumptions are not fulfilled: they may produce spurious results. Even though some tests have been shown to be somewhat robust against borderline violations of their underlying assumptions, the burden of proof that the assumptions were sufficiently fulfilled is on the researchers who conducted the test.

In general, the exact type of test in a family needs to be specified to inform which assumptions come to bear. For instance, the assumptions of an independent-samples t-test will be different from a dependent-samples t-test. Similarly, it needs to specified whether the test is "one-tailed" or "two-tailed" to put the reported p-values into perspective.

To ascertain whether the statistical analyses were correctly employed on the data, statistical assumptions need to be made explicit in reporting. For example, the assumptions for parametric tests, in general, are normally distributed data, homogeneity of variance, interval data and independence [START_REF] Field | Discovering statistics using IBM SPSS statistics[END_REF]. Parametric statistical tests often require a systematic treatment of outliers.

We ask 'Were significance level α and test statistics properties and assumptions appropriately stated?' Benefits of fulfilling CI7. A precise specification of the test used and explicit documentation of the assumptions checked gives the reader confidence that the statistical tools were appropriately chosen and employed diligently.

Outcome for not fulfilling CI7. Should test properties and assumptions not be documented, we need to assume that researchers did not establish that they could reliably employ the statistical test. Consequently, the reported test statistics and p-values could be off and not be relied upon.

Steps to Take

How to achieve CI7. One would choose the designated significance level a priori and state it explicitly. Similarly, the researchers need to establish whether the test will be one-or two-tailed in advance. Researchers check whether the data meets the assumptions of the planned statistical test and explicitly report whether and how the data met the test assumptions. Decisions on how the data was treated (e.g., outlier management) need to be reported explicitly.

We emphasize that complex statistical models (such as regressions) usually require comprehensive post-hoc model diagnostics to evaluate whether the model is sound.

Typical location in articles. The treatment of assumptions is documented in the results section, either close to the report of the statistical test or in a separate subsection. Often it will support the confidence in the reported results, if a comprehensive analysis report is published alongside the research paper that documents all checks of assumptions and diagnostics, transformations of the data, and decisions made.

Reporting Example.

Example 12 (CI7 -Significance level α & test statistics properties, from Groß et al. [START_REF] Groß | Effect of cognitive depletion on password choice[END_REF]).

"All inferential statistics are computed with two-tailed tests and at an α level of .05" Example 13 (CI7 -Test statistics assumptions, from Groß et al. [START_REF] Groß | Effect of cognitive depletion on password choice[END_REF]).

"The distribution of the Passwordmeter password strength score is measured on interval level and is not significantly different from a normal distribution, Saphiro-Wilk, D(100) = .99, p = .652 > .05" a . "We computed Levene's test for the homogeneity of variances. For the password meter scores, the variances were not significantly unequal."

a We note here that numerical normality tests, such as Saphiro-Wilk may have too little sensitivity for small sample sizes and too much sensitivity for large sample sizes. [START_REF] Statistics | Testing for normality[END_REF] 10 CI8: Confidence Intervals on Effects

Theoretical Background

An effect size estimates the magnitude of an effect, an unknown parameter of the population, given the observed data of an experiment. Confidence interval procedures on the effect estimate the range of plausible values for the population parameter, if the experiment were repeated independently infinitely many times. We note that this is a frequentist view, in which the confidence level applies to the procedure. For instance, a series of 95% confidence intervals will tend to contain the population parameter on average 95% of the intervals. Effect sizes and their confidence interval offer an informative view on an experiment's observed effect magnitudes. Consequently, the APA guidelines [2] state that "estimates of appropriate effect sizes and confidence intervals are the minimum expectations." QI8 includes that the effect sizes are reported in a easily human-interpretable form.

We ask 'Were the appropriate the effect sizes and confidence intervals (CI) reported?'

An effect that is statistically significant is not necessarily scientifically significant or important. To draw conclusions on an effect's importance or practical implications, we consult the magnitude of the effect, its effect size. [START_REF] Cohen | A power primer[END_REF].

In estimation theory, the effect size (ES) provides a point estimate of effect in the population, while the confidence interval (CI) provides the interval estimate. While we endorse the use of estimation theory [START_REF] Cumming | Introduction to the new statistics: Estimation, open science, and beyond[END_REF][START_REF] Gardner | Confidence intervals rather than p values: estimation rather than hypothesis testing[END_REF], we note that interpreting confidence intervals correctly requires diligence [START_REF] Morey | The fallacy of placing confidence in confidence intervals[END_REF]. Notably, it is a fallacy to interpret a post-data X% confidence interval to have a X% probability to include the true population parameter.

Benefits of fulfilling CI8. CI8 evaluates the robust reporting of effect magnitudes through parameter and interval estimation, which yields, in turn, the foundation for future meta-analysis and research synthesis.

Outcome for not fulfilling CI8. Without effect size estimate, we only have the significance of the results and p-values to go on. However, we will miss out on information on the magnitude of the claimed effects. For example a significant p-value does not say how important the observed effect is: it could well be trivial, and neither contribute much to research nor vouch for changes to practice.

Steps to Take

How to achieve CI8. To compute effect sizes in experiments together with their confidence intervals and to report these in publications. Literature already provides a number of manuals and research articles on computing the different families of effect sizes [START_REF] Fritz | Effect size estimates: current use, calculations, and interpretation[END_REF][START_REF] Lakens | Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and anovas[END_REF] To also refer to the New Statistics [START_REF] Cumming | Introduction to the new statistics: Estimation, open science, and beyond[END_REF] for the estimation approach, effect-size and confidence intervals.

Typical location in articles. Effect sizes and their confidence intervals are documented in the results section, either stated as a suffix after the p-value of the corresponding statistical inference or provided in dedicated tables.

Reporting Example.

We refer to the Example 11 for effect size and confidence interval reporting.

Further Sources

Kirk [START_REF] Kirk | The importance of effect magnitude[END_REF] and Cumming [START_REF] Cumming | The new statistics: Why and how[END_REF] debated that the current research practice of exclusive focusing on a dichotomous reject-nonreject decision strategy of null hypothesis testing that can impeded scientific progress. Rather, they posit, the focus should be on the magnitude of effects, that is the practical significance of effects and the steady accumulation of knowledge. They advise to switch from the much disputed NHST to effect sizes, estimation and cumulation of evidence. We ask 'Was the significance and hypothesis testing decision interpreted correctly and put in context of effect size and sample size/power?'

Nickerson [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF] offers a comprehensive overview of the controversies around Null Hypothesis Significance Testing (NHST), while Maxwell and Delaney [31, p.48] and Goodman [START_REF] Goodman | A dirty dozen: twelve p-value misconceptions[END_REF] point to p-Value misconceptions, Morey et al. [START_REF] Morey | The fallacy of placing confidence in confidence intervals[END_REF] analyze confidence interval fallacies and Ioannidis [START_REF] Ioannidis | Why most published research findings are false[END_REF] argues "why most published research findings are false."

The evaluation in our work is founded on Nickerson's review [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF] on misconceptions around NHST, which include:

p misperceived as the probability that the hypothesis be true and 1 -p misperceived as the probability that the alternative hypothesis be true, a small p considered as evidence that the results be replicable, a small value of p misinterpreted as a treatment effect of large magnitude, statistical significance considered as theoretical or practical significance, significance level α misinterpreted as the probability that a Type I error will be made, -Type II error rate β considered to mean the probability that the null hypothesis be false, failing to reject the null hypothesis misrepresented as equivalent to demonstrating it to be true, failure to reject the null hypothesis misinterpreted as evidence of a failed experiment. While Nickerson's observations are concerned with the correct interpretation of NHST, for us CI9 also includes preparing the ground with population and sampling as well as a priori hypothesis specification, and post-hoc concerns such as multiple-comparison corrections.

Benefits of fulfilling CI9. Evidence towards CI9 convinces us of the robustness and diligence of the statistical inference made, because common pitfalls and fallacies have been avoided. The result statement will offer a sound starting point for the interpretation of the findings.

Outcome for not fulfilling CI9. Should there be evidence of incorrect statistical inference or the presence of fallacies, we would need to assume that the researchers interpretation of said results be tainted by the misinterpretations and misrepresentations made. Hence, the overall conclusion of the study would be put into question. We perceive reviews on misconceptions and fallacies as important guard rails [START_REF] Nickerson | Null hypothesis significance testing: a review of an old and continuing controversy[END_REF][START_REF] Maxwell | Designing experiments and analyzing data: A model comparison perspective[END_REF][START_REF] Goodman | A dirty dozen: twelve p-value misconceptions[END_REF][START_REF] Morey | The fallacy of placing confidence in confidence intervals[END_REF].

Steps to Take

How to achieve CI9. To achieve CI9, we recommend to investigate how p-values and confidence intervals can and cannot be interpreted. The key principle here is diligence: The devil is in the details.

Typical location in articles. The correctness of the statistical inference is prepared by the documentation of the a priori elements of a study in the methods section, supported by the correct reporting of statistical tests in the results and finally completed by the interpretation of the outcomes in the discussion.

Reporting Example.

Example 14 (CI9 -Type I error correction).

"Given the number of comparative t-tests computed on the data set, we compute a multiple comparisons correction, where differences marked with a dagger † in Table 1 are statistically significant under Bonferroni-Holm correction for all comparisons made."

12 Lessons Learnt from the workshop

Aim

To assess whether and how the set of nine indicators could be applied in practice.

Method

Procedure. We gave a small presentation of the hallmarks of experiment design (following our 2016 workshop at the same venue) and then presented the nine indicators as a set of 'Quality Indicators', where quality assessment is a stage employed within Systematic Literature Review procedures [START_REF] Ebse | Guidelines for performing systematic literature reviews in software engineering[END_REF]. These nine indicators were developed as a checklist of factors to be evaluated within experimental studies, as part of a UK Research Institute in Science of Cyber Security (RISCS) funded project, which had the overall aim to evaluate the state of the art in evidence-based methods in cyber security and privacy.

Prior to the workshop we developed a first version of a codebook which specified each of the indicators in terms of sub-criteria and examples and a codesheet providing a marking scheme.

Next, we facilitated open coding with the aim to extract concepts from the free-form text. We provided participants with (1) two example research articles reporting user experiments in the context of privacy [START_REF] Gideon | Power strips, prophylactics, and privacy, oh my![END_REF][START_REF] Korff | Too much choice: End-user privacy decisions in the context of choice proliferation[END_REF], (2) the CI specification as a codebook [START_REF] Coopamootoo | A codebook for experimental research: The nifty nine indicators v1.0[END_REF] and (3) the marking as a codesheet [START_REF] Coopamootoo | A codebook for experimental research: The nifty nine indicators v1.0[END_REF].

Participants. Participants worked in two groups to review the two articles. N = 9 participants attended the workshop, 6 female, 3 male. The 7 participants who provided their age had mean age 31.86 years (SD = 8.28). 5 participants were from a usable privacy and security background, while others were from other areas of privacy and security. Participants' first language varied (4 German, 2 English and 1 Tamil, 2 did not answer). With the sole aim to gauge participants' expertise, we offered participants three Likert questions to rate their frequency of use of evidence-based methods (from 1 -'Never' to 5 -'A great deal'), their skills (from 1 -'Poor' to 5 -'Excellent') and their familiarity (from 1 -'Not at all familiar' to 5 -'Extremely familiar') in designing experiments. Participants reported using evidence-based methods such as experiments with a median value of 3, to have a median skill level of 2 and median familiarity in designing experiments of 2.

Results

We provide results in the form of participant feedback and recommendations.

Practical Requirement. Participants recommended shaping of the indicators as a toolkit that can readily be employed by the community. This involves designing clear sections in the tool set that researchers and committee members can pick up. As a result, following the workshop, we have revised the indicators to match these requirements, as presented through sections CI1 to CI9. In this paper we provided the theoretical underpinnings for each CI together with 'Steps to Take' and 'Examples'.

Design Requirement. Participants noted the time commitment required if one does not know what to look for when applying the toolkit in a reviewing exercise. To address this, we provide clear examples for each CI together with typical sections in research papers that provide support for criteria fulfilling each CI.

Ethical Considerations. Participants suggested to factor in ethical considerations, as aspect of experimental reporting we omitted but foresee its benefits for completeness of reporting.

Discussion

Community progress. A toolkit, such as the one we provide here, contributes to a standard to aspire to. It supports the community in developing the skills to design, run and report rigorous experiments in cyber security. At the same time, while the lack of defined best practices requires individual researchers to determine what the standards they adhere to, our toolkit offers a common ground.

It also supports the reviewing process and program committee decisions, by offering syntactic criteria to check for the completeness of scientific reporting. In addition, it contributes to a culture of well designed and reported experiments that can serve as notable examples to follow in the field.

Added value for researchers. We believe this toolkit can be a valuable ingredient for inter-disciplinary security and privacy research. It combines theoretical background and practical guidelines to support foundations in experiment design and reporting. By following the requirements of participants as voiced during the workshop, we provided clear sections that can be picked up by researchers and committee members. It supports both novice and experienced usable security and privacy researchers. While learning a methodology takes time for any novice, we believe that this toolkit may support the learning the nitty-gritty of experimental methodology by being designed as a one-stop resource. For more advanced researchers, it presents itself as a checklist and offers some good practices to follow.

Not exhaustive. We observe that our current toolkit is not exhaustive and foresee that it will grow as discussions advance within the community. In line with this, we plan to facilitate further discussion exercises within the community and to seek ways for engagement.

Conclusion

This paper provides a first toolkit for experimental research in cyber security and privacy with a sampler of theoretical foundations and practical guidelines. I can support a study's lifecycle from conception, design, analysis and reporting to replication. It provides a companion for novice researchers as well as reviewers needing a structured checklist. Although the toolkit is certainly not exhaustive, it may still grow with discussions and evidence-based projects within the community. We believe that already in the current form, it can support a culture of robustly designed and reported experiments, thereby contributing to empirical research in the field.

  11 CI9: Statistical Inference 11.1 Theoretical Background CI9 evaluates the overall correctness of the statistical inference, that is, how statements on statistical significance are expressed and what conclusions are drawn from the statement. As such, CI9 relies to some extent on observations made with respect to preceding completeness indicators.

https://osf.io

https://aspredicted.org

Acknowledgment

We are indebted to the the participants of the "Workshop on Evidence-Based Methods" at the 2017 IFIP Summerschool on Privacy and Identity Management for their generous feedback. This work was supported by the UK Research Institute in Science of Cyber Security (RISCS II) project "Scientific Methods in Cyber Security: Systematic Evaluation and Community Knowledge Base for Evidence-Based Methods in Cyber Security." It was in parts funded by the ERC Starting Grant CASCAde (GA n o 716980).


Alexandr Railean
email: arailean@datenschutzzentrum.de

Delphine Reinhardt
email: delphine.reinhardt@cs.uni-bonn.de

Life-Long Privacy in the IoT? Measuring Privacy Attitudes Throughout the Life-Cycle of IoT Devices

Keywords: Internet of Things, IoT, privacy, usability

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The IoT is composed of devices, sensors or actuators, that connect, communicate or transmit information with or between each other through the Internet (adapted from [START_REF]Internet of things: Privacy & Security in a Connected World[END_REF]). It is rapidly growing, as the number of connected devices per person has increased from 1.84 to 3.3 between 2010 and 2016 [START_REF] Evans | The Internet of Things -How the Next Evolution of the Internet Is Changing Everything[END_REF][START_REF][END_REF]. Many IoT devices, such as light bulbs, power switches, air quality monitors, or fitness trackers, are widely available. There is also strong support in the "do it yourself" community: there are 21,714 hits on Github.com, and 49,000 hits on Instructables.com when searching for the term "IoT". Moreover, some appliance manufacturers aim at increasing the share of their connected products. For instance, Samsung's CEO stated that all their products will be part of the IoT by 2020 [START_REF] Samsung | All of Our Products Will Be Connected to the Web[END_REF]. Governments have also expressed interest in the IoT. For example, the Federal Trade Commission (FTC) issued a privacy and security guide [START_REF]Careful Connections: Building Security in the Internet of Things[END_REF] for businesses involved in IoT development, while the European Commission is working on regulations that have provisions for IoT communications [START_REF]Regulation of the European Parliament and of the Council Concerning the Respect for Private Life and the Protection of Personal Data in Electronic Communications and Repealing[END_REF]. This indicates that IoT is on the path of becoming an indispensable part of our daily lives, based on the current attention of all involved parties, i.e., enterprises, governments, and end users.

However, such products may expose end users and product owners to privacy risks that can occur at the interplay of factors like resource-constrained hardware, poor usability, ubiquitous deployment or the availability of many pools of data. These factors can make the implementation of well-established privacy and security mechanisms difficult. Additionally, users may get little or no feedback about the data collected while interacting with an environment that lacks an interface (e.g. when sensors are seamlessly embedded into walls or furniture). A ubiquitous deployment means that insights about the users can be gathered in locations where they are not expecting data collection. Moreover, linking different data pools having information about the users can facilitate their identification, and hence lead to their deanonymization. For example, studies show that information about a person can be derived by correlating data from disparate sources, such as smartphone sensors [START_REF] Christin | Privacy in Mobile Participatory Sensing: Current Trends and Future Challenges[END_REF][START_REF] Lane | On the Feasibility of User De-anonymization From Shared Mobile Sensor Data[END_REF], social media [START_REF] Kosinski | Private Traits and Attributes Are Predictable From Digital Records of Human Behavior[END_REF] or online reviews [START_REF] Narayanan | How to Break Anonymity of the Netflix Prize Dataset[END_REF]. At the same time, most people are not technically proficient [START_REF]OECD Skills Studies[END_REF], and even those who are often subvert their privacy [START_REF] Kang | My Data Just Goes Everywhere" User Mental Models of the Internet and Implications for Privacy and Security[END_REF]. This has been shown in the use of social media [START_REF] Barnes | A Privacy Paradox: Social Networking in the United States[END_REF] or instant messengers [START_REF] Luca | Expert and Non-Expert Attitudes Towards (Secure) Instant Messaging[END_REF].

This paper starts with a review of related work in Sec. 2. We then investigate whether the aforementioned patterns apply to IoT in Sec. [START_REF]Android API Versions[END_REF], by means of an online questionnaire introduced in Sec. [START_REF] Atzori | The Internet of Things: A Survey[END_REF]. The results, based on the answers of 110 participants, are shared in Sec. [START_REF] Barnes | A Privacy Paradox: Social Networking in the United States[END_REF]. The answers show that most participants are aware of privacy risks, though they are inclined to keep using a device that infringes on their privacy. Moreover, our results provide an understanding of the reasons behind the adoption of IoT devices by end users, and give a clearer picture of the attention our participants pay to privacy throughout the life-cycle of their IoT devices. We then test our hypotheses in Sec. 6. In Sec. 7 we discuss the results and limitations of our survey, as well as provide recommendations for IoT vendors. Sec. 8 concludes the paper and summarizes our findings. All the materials needed to replicate the survey are given in Appendix A.

Related Work

Naeini et al. explore people's preferences regarding IoT data collection and notifications of data collection in [START_REF] Naeini | Privacy Expectations and Preferences in an IoT World[END_REF]. They found that the participants of their study were more open towards data collection in public settings, and less so when data collection occurs in a private environment, if it involves biometric data, or if the data will be stored for long periods of time. They also develop a model that can predict one's data-collection preferences based on three data-points. Other works examine IoT from a legal perspective, a definition of IoT privacy is given in [START_REF] Ziegeldorf | Privacy in the Internet of Things: Threats and Challenges[END_REF], the paper identifies the possible privacy risks related to IoT. Peppet conducts another legal analysis in [START_REF] Peppet | Regulating the Internet of Things: First Steps Toward Managing Discrimination, Privacy, Security and Consent[END_REF] and discusses how privacy is affected by the difficulty of sensor data de-identification, thus questioning the distinction between personal data and other data. Another raised concern is that some IoT device vendors conflate the notion of "notice" with that of "consent", assuming that informing users about what a technology does is sufficient to indicate that use of technology implies consent (S 0 , please note that the statements marked with S n will be referred to in Sec. 7.2). The analysis also includes a comparison of the packages of several IoT devices with respect to privacy-related information, as well as their privacy policies. An extensive literature review and summary of IoT privacy issues is provided in [START_REF] Atzori | The Internet of Things: A Survey[END_REF][START_REF] Caron | The Internet of Things (IoT) and its Impact on Individual Privacy: An Australian Perspective[END_REF][START_REF] Mendez | Internet of Things: Survey on Security and Privacy[END_REF]. Other works are focused on location privacy [START_REF] Elkhodr | A Review of Mobile Location Privacy in the Internet of Things[END_REF][START_REF] Minch | Location Privacy in the Era of the Internet of Things and Big Data Analytics[END_REF], while [START_REF] Zhou | Security/Privacy of Wearable Fitness Tracking IoT Devices[END_REF] focuses on fitness trackers. Volkamer et al. discusses the importance of mental models formed by end-users and the role these models play in the trust and acceptance of new technologies in [START_REF] Volkamer | Mental Models -General Introduction and Review of Their Application to Human-Centred Security[END_REF]. There are other papers that present IoT life-cycle models, however they take a data-centered approach, examining what happens to the personal data acquired and transmitted by IoT devices [START_REF] Minch | Location Privacy in the Era of the Internet of Things and Big Data Analytics[END_REF][START_REF] Ziegeldorf | Privacy in the Internet of Things: Threats and Challenges[END_REF]. Our work, on the other hand, takes a user-centered approach, focusing on the different stages of the relationship between users and their IoT devices.

Research Goals

To examine the participants' privacy attitudes and user experience in the context of IoT device ownership, we focus on the following Research Questions (RQ):

-RQ 1 : What motivates potential users to acquire IoT devices? -RQ 2 : Would they continue using a device that infringes on their privacy? -RQ 3 : Are users aware of the extent to which IoT devices can interact with other equipment they own?

We then map the answers to the corresponding phases of the IoT device life-cycle (defined in Sec. 4), and look for user interface friction points that can potentially affect the privacy of end-users. This, in turn, enables us to suggest usability improvements and creates new research questions for the future.

The answers to the research questions help us test the following hypotheses (referred to as H), which are formulated on the basis of autoethnographic observations:

-H 1 : When dealing with IoT devices, most users treat them as appliances, rather than computers. -H 2 : Users are inclined to keep IoT devices that infringe on their privacy, if those devices have a high monetary value. -H 3 : Users are inclined to keep IoT devices that infringe on their privacy, if those devices were a gift from a close person.

Methodology

To answer the questions and test the hypotheses, we designed an online questionnaire, which covers the phases of the IoT device life-cycle we consider to have an impact on privacy: pre-acquisition, set-up, usage, maintenance, and decommissioning, as illustrated in Fig. 1. Note that we are not concerned with the factors that lead to decommissioning (e.g. resale, recycling, etc), we only focus on the privacy implications due to removal of IoT devices from service, regardless of the cause. In our questionnaire, we take a human-centered perspective and focus on what a person does with the device, rather than on what the device does with the data, in contrast to [START_REF] Minch | Location Privacy in the Era of the Internet of Things and Big Data Analytics[END_REF][START_REF] Ziegeldorf | Privacy in the Internet of Things: Threats and Challenges[END_REF]. We have especially phrased our questions in a way that should elicit what participants think about the device and what their beliefs about its behaviour are.

Distribution and audience

We have invited our participants via word of mouth, mailing lists, social media, and survey sharing platforms. Because it appeals to a wide audience, we have particularly taken care that non-experts could understand the goal of our questionnaire. To this end, we have defined and detailed the terminology used and given concrete examples. The introduction also provided key details about how the collected data would be handled, i.e., full anonymity and no disclosure of individual answers. In total, 193 participants have answered our online questionnaire. Among them, 110 participants have fully filled it out. We have therefore discarded the incomplete ones for computing the following results. The majority of our participants are male (57%), 5% preferred not to disclose their gender. The most represented age category is between 21 and 30 (52%), followed by 31 and 40 (28%), then by 41 and 50 (8%). 45% of the participants have a bachelor degree, 33% have a master degree, 8% have a secondary school level of education, 5% preferred not to disclose information about their education, while 3% have earned a doctorate degree. Geographically, most of our participants are from Eastern Europe (45%), followed by 31% from Western Europe and 14% from North America.

Self-selection bias

Since we have initiated the distribution of the survey, it is possible that the recruited participants fit a similar profile, thus biasing the sample. We have therefore asked the participants to indicate the different computer-related skills they have in question Q 30 (see Appendix A). We then assign to each skill a number of points according to the distribution presented in Tab. 1. The total number of points obtained by a participant finally determines the category they belong to. We categorize participants with a total number of points below 8 as novice, between 8 and 20 as medium, and greater than 20 as expert. Our sample counts 55% rated as expert, 37% are medium and 7% are novice.

Priming concerns

To avoid priming participants into a privacy-oriented mindset, the topic of the survey has been announced as "IoT usability". There was no mention of the term "privacy" in the call for participation, e.g. "You're invited to participate in an IoT usability survey". Additionally, privacy-themed questions and answer choices were uniformly distributed among other topics.

Results

Our results are based on the responses of 110 participants and are mapped to phases of our IoT lifecycle model. The first set of questions is aimed at all the participants, whether they own an IoT device or not. We have found that 41% of them do not own IoT devices, whereas the others own smart TVs (38%), smart watches (23%), fitness bracelets (18%), thermostats (12%) and voice assistants (12%) (multiple choices possible). 39% of the participants are planning to purchase new IoT devices in the next 6 months (74% of them already own an IoT device), 30% have no such plans (33% of them own an IoT device), while 27% are not sure about it (47% of them own an IoT device).

Pre-acquisition

We have then asked the participants to indicate, in a non-prioritized way, the "reasons to buy Internet-connected appliances" (Q 21 ). They have indicated 86 reasons in a free-text field, which we have clustered as follows: automation of routine tasks (38%), better remote control (31%), and new capabilities (31%).

Being socially connected (16%) and health improvements (12%) were selected by fewer participants. On the other hand, the participants have given 109 reasons why they would not buy such appliances. The most represented concerns are privacy (34%), security (30%) and cost (12%). Some of the arguments supporting the latter concern being (a) interaction with IoT devices will consume their data plan and inflate the bill, (b) an insecure IoT device that can make purchases can be taken over, allowing hackers to order items for free, (c) the cost of IoT devices is usually greater, due to their novelty, not due to their actual benefits, and (d) these devices become obsolete very fast.

Tab. 2 shows what participants would be looking for, if they were purchasing an IoT device. The responses indicate that convenience plays a key role. 72% look for ease of use, while 66% seek compatibility with existing devices. We have also seen that privacy is not of particular importance, it ranked 46%, close to "good brand reputation" (48%) and "low price" (47%). Another important highlight is that certifications from organizations like Technischer Überwachungsverein (T ÜV) or Federal Communications Commission (FCC) play little role in the choice of IoT devices. Such an attitude may be explained by a greater level of trust in product reviews published on the Internet, or by the fact that brand reputation is sufficient to decide which device to purchase.

Other features mentioned in a free-text field by participants were (a) guaranteed updates period (2 mentions), (b) open hardware/software and firmware access (2 mentions), (c) good security record (3 mentions), (d) wide functionality and customizability (3 mentions). One participant specifically indicated that the privacy policy should be "SHORT and clear" (S 1 ).

To learn the reasons why our participants chose to acquire their IoT devices, we have asked them to "[...] indicate the benefits of connected devices that appeal to [them] personally" (Q 23 ). Although this question is similar to Q 21 , it enables us to differentiate between benefits participants have heard of in principle, and benefits that they themselves are looking for. The results in Tab. 3 show that the responses are similar, the most common and least common reasons follow the same distribution, with a difference in health improvements. 12% chose it as a reason to buy IoT devices, 30% indicated that it is what appealed to them in particular. This observation leads us to the conclusion that in our sample, participants acquire IoT hardware for practical reasons, rather than because it is fashionable to do so.

Set up

In this and subsequent sections, we provide the results related to questions that involved participants who own IoT devices. Note that these questions were not displayed to those who indicated that they do not own an IoT device. Therefore the percentages shown are relative to a total of 65 participants. In Q 6 , we have asked participants "how satisfied [they] are with the process of using the device 'brand' ?", the answers are expressed on a 5-point Likert scale, ranging from "very dissatisfied" (1) to "very satisfied" (5), based on several criteria in Fig. 2.

We have found that "satisfied" and "very satisfied" are the most common answers to all the questions, except when it comes to the level of satisfaction with the accompanying documentation, where 42% chose the "neutral" option. A possible explanation is that the manual was never consulted due to lack of need, preference, or lack of interest. Lack of need can be the result of a successful configuration based solely on the clarity of the interface, or the technical experience of the end user. It can also be explained by the fact that the majority of participants rated "online materials (e.g. site, support services)" as "satisfying", Fig. 3. Extrema and quartiles of the valid participants' answers to Q9 based on the following criteria: configuring the device is easy (H, valid answers: 55), configuring it via a smartphone app is easy (I, 54), configuring it via a web-interface is easy (J, 54), set it up without reading the manual (K, 53). which could indicate that whatever questions they had were addressed online, as such materials are easier or faster to search.

We have further probed this matter by asking participants "when it comes to configuring [the IoT device], how much do [they] agree with the following statements" in Q 9 , and find that 71% agreed and strongly agreed to being able to set up and configure their device without reading the manual (Fig. 3). This supports the assumption that lack of need is what leads to the documentation being neglected. Such a level of success can have an undesired effect: satisfied end-users can stop tinkering with the device as soon as they accomplish their primary goals, thus missing potentially critical security and privacy tips the documentation could offer. We conclude that important privacy-related controls should be incorporated into the initial setup procedure, to ensure that end-users make informed privacy-related decisions (S 2 ).

Usage

When asked about continued use of an IoT device that infringes on the owner's privacy (Q 24 ), two of the top three reasons are related to the monetary value of the product, "it was an expensive purchase" and "it is difficult to return it or get a refund" got a combined score of 53%. In contrast, options related to family values are the least convincing reasons to keep it (14%). Other mentioned reasons were: (a) if it provides a unique function, (b) if it is crucial for daily use, or (c) if the infringement is negligible. Convenience is a major factor and its importance is often expressed throughout the collected answers. We have found that entertainment scores as high as health-related benefits (20%). This attitude resonates with the "dancing pigs" adage in computer security: "The user's going to pick dancing pigs over security every time" [START_REF] Schneier | Secrets and Lies: Digital Security in a Networked World[END_REF]. While studies [START_REF] Akhawe | Alice in Warningland: A Large-Scale Field Study of Browser Security Warning Effectiveness[END_REF] concluded that a better user interface helps people make wiser security-related decisions, those findings are not necessarily applicable in our context. Our question asks about a participant's choice in principle, which implies that this is a conscious decision they would make, no matter what the interface looked like. When it comes to discarding an IoT device that infringes on the owner's privacy (Q 25 ), the reasons chosen by participants were: "ethical and moral convictions" (46%), "it is easy to get a refund" (45%), "installing custom firmware voids the warranty" (38%), and "it is easy to re-sell" (32%). Among the reasons indicated in the free-text field, 2 participants mentioned that the decision depends on the magnitude of the infringement.

To get a better understanding of what IoT device owners think about the capabilities of their hardware, we have asked them to indicate "the resources [they] think are exposed to the IoT device" in Q 7 . The distribution of the answers is shown in Tab. [START_REF] Atzori | The Internet of Things: A Survey[END_REF]. In 69% of the responses, it is expected that an IoT device can interact with a smartphone, presumably because that is how it is configured and controlled. Other options have been chosen by fewer than 40% of the participants.

We have asked participants "who, in [their] opinion, can use, or otherwise interact with IoT [devices] installed in your home?" in Q 8 . The responses show that 35% of participants consider that hackers are capable of doing so, while 13% think the government can do that as well. These numbers indicate that the efforts of IoT device vendors are insufficient to establish trust and convince the participants that their product is secure (S 3 ), as it has been argued in [START_REF] Volkamer | Mental Models -General Introduction and Review of Their Application to Human-Centred Security[END_REF]. We have also found, by means of a Kruskal-Wallis test, that expert participants are more likely (χ 2 = 6.857, p = 0.032) 3 to consider that the government can access their IoT hardware. Note that they do not hold the same opinion about hackers. This may be explained by an expert's confidence in their own ability to secure a system from typical attackers. On the other hand, their awareness of the fact that state-level actors have much more resources may justify the belief that governments could conduct successful attacks, if they choose so. We have finally asked our participants whether they have "examined the privacy policy" of their IoT device in Q 12 , and find that 22% have done so. To understand whether IoT device adoption is a conscious decision, rather than a forced one (i.e. the IoTenabled device was purchased because there was no "dumb" analog), we have asked our participants if they "own any appliances, the IoT capabilities of which are not used" (Q 17 ). 22% of the participants who own IoT devices always use the IoT features, 5% turn them off explicitly, 5% are aware of the features but are ignoring them, while 2% use various external means to disable them. Among the recorded means, we have found stickers over cameras (two mentions), positioning the device with the camera pointing down (one mention) and using a network router to limit the traffic of particular devices (one mention).

Maintenance

To understand the participants' attitudes towards software updates, we have asked them "do [they] think IoT devices require software updates?" (Q 4 ). 92% consider that IoT devices require software updates, 5% do not know if that is the case, while 3% believe that updates are not necessary. In Tab. 6, we present the answers to the question "who should be responsible for updating the IoT device, in your opinion?" (Q 5 ). Although 60% of the participants consider that the manufacturer should be responsible for pushing updates to IoT devices (S 4 ), two participants indicated that they want to be the ones who decide whether an update is installed or not. This could be the result of prior experience with unwanted updates, that disabled useful features or added undesired ones (S 5 ). This could explain why some are aware of the availability of newer versions, but are not installing them (Tab. 7).

The results indicate that our participants see IoT devices as computer-like systems that require software updates, rather than "plug in and forget" devices. We emphasize that the most common expectation is for the updates to be rolled out by the manufacturer. This is an important point to be considered by IoT device designers, because if this expectation will not be met, it is possible that the devices will run outdated firmware, potentially exposing owners to security and privacy risks. The data also reveal a gap between those who expect updates to be automatically installed by the manufacturer (60%) and those who are aware that updates are automatic and are certain that their IoT device uses the latest version (27%). This difference could be explained in different ways, e.g. the IoT devices do not adequately reflect their update availability status (if at all) (S 6 ) or end users did not bother to check that. We measure that, using a 5-point Likert scale, by asking participants "How well does the device [...] express what it is currently doing?", listing several use cases, of which one is "installing an update" (Q 10 ). We have found that participants consider this to be expressed clearly (20%) to very clearly (35%), while another 20% have not experienced this use case. Sec. 5.5 discuses other implications related to update policies.

Decommissioning

To determine whether participants have gone through this procedure and measure their level of satisfaction with it, we have asked them "how satisfied are you with the process of [...] resetting [...] to default settings and wiping all data?" (Q 6 ) and "how well does the device express [...] that it is currently resetting itself to default settings and wiping the data?" (Q 10 ). We have found that the many of our participants have not had the experience of wiping the data off their IoT device (31%) or have not had the chance to see how this process is reflected in the interface (45%). It should be noted that some of the participants could have chosen the "N/A" option because their IoT device does not provide such a feature or it is not relevant for its function, the survey does not distinguish between these possibilities. Since this use case has been less explored by end users, manufacturers have fewer opportunities to receive feedback about this procedure. Thus, any existing usability shortcomings can possibly remain in the product for a longer period of time. In contrast, use cases related to set up and usage are likely to attract far more attention. We conclude that IoT device manufacturers should not perceive the lack of customer complaints as an indicator of good usability of their product in the decommissioning phase. Instead, they ought to conduct tests targeting this particular scenario (S 7 ).

Testing the hypotheses

In what follows, we successively test the hypotheses defined in Sec. 3, based on the answers given by participants.

H 1 : When dealing with IoT devices, most users treat them as appliances, rather than computers. On one hand, the arguments detailed in Sec. 5.4 suggest that most of the participants consider IoT devices to be computers, rather than appliances, based on their awareness of the fact that such devices require regular updates and have to be secured. However, the analysis in Sec. 5.3 indicates that this awareness is limited. For example a smart TV that runs an operating system with network capabilities is exposed to all of the resources listed in Q 7 , yet the participants' responses failed to reflect that. This could mean that some participants' level of confidence exceeds their actual understanding, which can lead to the false belief that the measures taken to protect their privacy are sufficient, when they are not. We cannot definitively support or refute H 1 , because the premise appears to be wrong. It is possible that there exists another model in the spectrum between computer and appliance, which describes more accurately how IoT devices are perceived. For example, participants may be used to smartphones and tablets, which require updates, but are nevertheless not treated as computers.

H 2 : Users are inclined to keep IoT devices that infringe on their privacy, if those devices have a high monetary value. The sampled population perceives privacy as a major concern in IoT adoption, but the concern can be overridden if the purchased IoT hardware was expensive, if it has an entertainment or utility value. In these circumstances, a substantial number of participants would continue using an IoT device, even if they are certain that it infringes on their privacy (Q 24 , Q 25 ). This can be partially explained by loss aversion, thus what matters is whether the owner can get reimbursed easily, regardless of the cost of the IoT device. When a refund is not possible, or if it is a tedious process, an inexpensive device is more likely to be discarded than an expensive one. Thus H 2 is supported, although we have to emphasize that other factors are at play. H 3 : Users are inclined to keep IoT devices that infringe on their privacy, if those devices were a gift from a close person. We have also found, by means of a Mann-Whitney U test, that females are more likely to keep using a rogue IoT device (U = 1066, n = 42, p = 0.012) 4 if it was a gift from a close person, thus H 3 is partially supported. It is possible that such attitudes are caused by emotional attachment to a person, however there may be other conditions too, e.g. the device has a likeable design, or it stores valuable content, like photographs. These additional factors were not checked by the questionnaire, so they should be investigated separately.

Discussion

The answers to Q 7 , "Which of these resources you think are exposed to the IoT device?" discussed in Sec. 5.3 could be a reason of concern. For example, in the case of a smart TV, a typical feature is to stream videos from remote sources, which requires some form of communication over networks, such as the Internet. This, in turn, implies that the device has to have an implementation of a network stack and software that leverages it. However, only two participants (rated at a medium skill level) indicated that their smart TV can access both, computers on their home network as well as other computers on the Internet. The same reasoning applies to voice-activated assistants (e.g. "Amazon Echo"). Only one participant correctly identified that their "Echo" can interact with local and remote hosts, which means that some participants are unaware of the fact that this device can transmit information via the Internet. While it is possible that some IoT devices are deliberately constrained by their owners (e.g. using firewalls), this should not be the case for assistants like "Echo", because they rely on an Internet connection for their basic features. Moreover, configuring Internet access is a required step in the setup phase, which the participants had to go through. This could be explained by the fact that they have an incomplete understanding of the capabilities of their device, or that someone else configured it for them (S 8 ). Product designers should consider this, because some of the user categories who could benefit from IoT, such as the elderly, may not be digitally literate, yet they must be aware of the implications of using the IoT device. Either the set-up procedure should be easy enough for anyone, or there should be a separate privacy summary that does not use technical or legal jargon and is easy to understand. We did not anticipate such results, therefore our survey was not crafted in a way that would enable us to determine whether this is a deliberate decision made by manufacturers, or an oversight, thus this matter has to be investigated separately.

Another important aspect is obsolescence, which we examine by analogy with smartphones. For example, the most common version of Android today has a market share of 31%, it was released two years ago [START_REF]Android API Versions[END_REF]. The two latest versions, 8.0 and 7.1, have a combined market share of 3.3%. Thus, a substantial number of smartphones are running outdated software. This is one of the reasons why the American Civil Liberties Union (ACLU) filed an FTC complaint over Android security issues [START_REF]Over Android Smartphone Security[END_REF]. If the same pattern arises in IoT, end-users will be stuck with outdated devices which, at best, can only be secured by applying external technical means (e.g. firewalls) or custom firmware. Neither of these options is novice-friendly. A strategy consumers can adapt is to decommission the device before the support period ends. While this solves their problem, the obsolete device will become someone else's problem. This creates the premises for a "tragedy of the commons" [START_REF] Hardin | The Tragedy of the Commons[END_REF], where the cost of security and privacy risks is distributed among all Internet users, instead of affecting IoT vendors or users specifically. Thus, the incentives to continue supporting and updating these devices is weak. This problem should be resolved in the future, otherwise it could hinder IoT adoption (S 9 ).

We have found some variation in attitudes, based on technical skills. Experts are more likely to indicate that they use a firewall, encrypted volumes and ad-blockers. They are also better-informed about IoT-related privacy and security news such as those about the Mirai botnet or the German steel factory incident. Note that we chose these topics because they were also covered by the international mainstream press, so non-experts could have heard about them. More surprisingly, the expert participants in our sample are also more likely to consider that manufacturers should be responsible for deploying IoT updates.

Note that our tests show that gender, age, and location do not have a significant impact on the participants' answers, unless otherwise stated.

Limitations

We encountered several limitations while running the survey. Firstly, people below the age of 18 were excluded, because of strict EU regulations concerning data collection from minors. However, this population segment could represent a significant portion of IoT technology consumers, thus their opinions should be accounted for. Secondly, we reached out to a technologically proficient audience (only 7% fell into the "novice" category), which is not representative of society in general. The modest number of participants finally gave us some hints about questions worth pursuing, but a study of a larger scale is required to make definitive claims about privacy attitudes.

Recommendations for IoT vendors

Based on the different statements S 0 to S 9 we highlighted in the paper, we would like to make the following recommendations to IoT manufacturers, to improve their privacy practices:

-S 0 Do not conflate "notice" with "consent" (based on [START_REF] Peppet | Regulating the Internet of Things: First Steps Toward Managing Discrimination, Privacy, Security and Consent[END_REF]) -S 1 Write concise privacy policies -S 2 Make privacy-related settings a mandatory part of the set-up phase -S 3 Find ways to address people's security and privacy concerns -S 4 Provide an automatic update feature -S 5 Make the list of version changes public -S 6 Reflect the update availability status clearly -S 7 Include decommissioning in usability tests -S 8 Consider that someone other than the end-user can set up the IoT device -S 9 Planned obsolescence should be more future-oriented

Conclusions

We have organized an online survey with 110 participants, to explore their privacy attitudes towards IoT devices. The results reveal a generally positive opinion about IoT, despite the awareness of existing privacy and security risks. The challenge is to address these issues before the end-users' skepticism creates a barrier in IoT adoption.

We have found a potential void in the user experience related to the decommissioning of such devices. Most participants have not gone through such a use case and there is a possibility that they will run into issues when they do so. Device manufacturers should consider this before releasing their products to the market. We have also found that the expected norm is that IoT devices are updated automatically and that it is the responsibility of the manufacturer to ensure the smoothness of the process. IoT device designers should implement such a capability in their product and provide clear information to end users when automatic updates are not available, and it is the user's responsibility to keep the device up to date.

Fig. 1 .

 1 Fig. 1. IoT device lifecycle pre-acquisition set up use maintenance decommissioning another owner

Table 1 .

 1 Distribution of points for each considered computer-related skill (Q30)

	Points Skills	Points Skills
	2	play video games	5	type complex documents in word
	2	view photos and watch videos		processors (e.g. macros, automatic
	2	browse the Internet and send		indexes, dynamic fields)
		emails	10 assemble computers or other elec-
	2	use a word-processor to type doc-		tronics from components
		uments	15 I know at least one programming
	5	set up email sorting filters		language

Table 2 .

 2 Desired IoT features (Q20)

	Feature	%	Feature	%
	ease of use	72	recommendations from friends and others 39
	compatibility with my existing devices	66	stylish design availability of technical documentation	35 35
	good brand reputation 48 low price 47	certifications by authorities (e.g. T ÜV, FCC)	20
	clear privacy policy	46	other (please specify)	8

Table 3 .

 3 IoT benefits that appeal to you personally (Q23) Extrema and quartiles of the valid participants' answers to Q6 based on the following criteria: plugging it in and connecting the cables (A, valid answers: 49), connecting it to [a] network or the Internet (B, 48), configuring the device settings (C, 50), accompanying documentation (D, 46), online materials (e.g. product site, support services) (E, 45), accompanying smartphone application (F, 43), resetting to default settings and wiping all data (G, 37). Invalid answers correspond to participants who skipped the questions or chose not to answer.

	Option	%	Option	%
	automation of routine tasks 59	health improvements	30
	better remote control	55	being connected to friends or family in a new way 26
	new capabilities	52	being connected to strangers or society in general 10
	energy saving	49	I don't know	10
	easier data management	34
	Fig. 2.

Table 4 .

 4 Which of these resources you think are exposed to the IoT device? (Q7)

	Option	%
	my smartphone	69
	other computers on my home network	40
	communications between other devices in my home and the Internet	31
	purpose-specific data (e.g. temp., humidity)	25
	other devices on my home network (e.g. printer) 24
	communications between devices in my home	22
	other computers on the Internet	15
	I don't know	11

Table 5 .

 5 Who can interact with the IoT device? (Q8)

	Option	%
	me	84
	others in my household (e.g. family)	65
	the manufacturer	38
	hackers	35
	the government	13
	my neighbors	4

Table 6 .

 6 Who should be responsible for updating IoT devices? (Q5)

	Option	%
	the manufacturer	60
	me, as the device owner	44
	the seller of the device	15
	a government agency	1
	I don't know	1

Table 7 .

 7 Is your IoT device running fully upto-date firmware/software? (Q3)

	Option	%
	N/A, I do not own any IoT device	41
	yes, it updates itself automatically	27
	yes, I update it manually	11
	I don't know	10
	no, but newer firmware is available	5

When p ≤ 0.5, it indicates that the results are not likely to be caused by chance, and that another set of participants would provide similar answers.

This indicates that the results are not likely to be caused by chance, and that if the same questions were given to other participants, the results would be similar.

Acknowledgments This research is funded by H2020 MSCA ITN Privacy&Us (project no 675730). We would like to thank the survey participants, Harald Zwingelberg and the anonymous peer reviewers for their helpful comments.

Appendix A Survey questions

The questions that featured in the survey are shown in Tab. 8. The list does not include the provided choices or other accompanying materials, they are available at https://www.datenschutzzentrum.de/projekte/privacy-us/. The site also provides the source code needed to replicate the survey and analyze the data.

Note that not all questions were shown to all participants (e.g. those who do not own IoT devices were not asked about their experience with such products). The label 'brand' was replaced with the IoT device name provided by participants in Q 2 . The table also mentions the type of each question, FT: free-text, MS: questions that allowed several options to be selected at the same time, MC: questions for which participants had to choose only one option out of several, L: Likert scale questions.