Inma Carrión

Jose L Fernández-Alemán

Ambrosio Toval
email: atoval@um.es

Usable privacy and security in Personal Health Records

Keywords: Usable privacy, usable security, PHRs, healthcare

PHRs (Personal Health Records) store individuals' personal health information. Access to this data is controlled by the patient, rather than by the health care provider. Companies such as Google and Microsoft are establishing a leadership position in this emerging market. In this context, the need for psychological acceptability in privacy and security protection mechanisms is essential. Any privacy and security mechanism must be acceptable from a usability perspective. This paper presents a study of the privacy policies of 22 free web-based PHRs. Security and privacy characteristics have been extracted according to the ISO/TS 13606-4 standard. In general, quite a good level was observed in the characteristics analyzed. Nevertheless, some improvements could be made to current PHR privacy policies to enhance the management of other users' data, the notification of changes to the privacy policy to users and the audit of accesses to users' PHRs.

Introduction

In recent years, governments around the world have shown an increasing interest in computerizing health-care records [START_REF] Sood | Electronic Medical Records: A Review Comparing the Challenges in Developed and Developing Countries[END_REF]. The growing use of Web 2.0 technologies signifies that patients can now access their own health information via tools such as Personal Health Records (PHRs). The Markle Foundation defines a PHR as "An electronic application through which individuals can access, manage and share their health information, and that of others for whom they are authorized, in a private, secure, and confidential environment" [START_REF]Connecting for health personal health working group. connecting for health. the personal health working group final report[END_REF]. The following benefits can be attained with PHRs [START_REF]myMediConnect Personal Health Records[END_REF] [START_REF] Tang | Personal health records: Definitions, benefits, and strategies for overcoming barriers to adoption[END_REF]: (i) they provide a unified summary of users' entire health histories; (ii) they are easy to understand and use; (iii) 24/7 access to all users' healthcare data from anywhere in the world; (iv) collaborative disease tracking; and (v) continuous communication between patient and physicians.

At present, a number of cutting-edge companies such as Google and Microsoft attempt to deliver healthcare services with their own PHRs: Google Health and Microsoft HealthVault. However, a number of new security and privacy threats hang over patients' health data in this context [START_REF] Farzandipour | Security requirements and solutions in electronic health records: Lessons learned from a comparative study[END_REF]. Information might be fragmented and accessible from several sites (by visiting different doctors' offices, hospitals, providers, etc). Safety defects in some of these systems could cause the disclosure of information to unauthorized people or companies, and health data therefore need protection against manipulations, unauthorized access and abuses. Data needs careful protection, thus leading to the necessity for extreme strictness in storage and information exchange activities. These threats are arguably more challenging than those found in most other industry sectors owing to [START_REF]Health informatics Electronic health record communication part 4: Security[END_REF]: (1) the amount of patient health record entries; (2) the number of healthcare personnel and organizations that might come into contact with a patient at any one time; (3) the difficulty involved in classifying the sensibility of a patient record entry; (4) the provision of very rapid appropriate access in a distributed computing environment; (5) the need for reviews of access permissions and for the PHR entries to be rigorously managed.

This research aims to study the privacy and security of PHRs from a usability perspective. The privacy policies of 22 free web-based PHRs were analyzed to extract their main characteristics. Our study on usable privacy and security is classified as a security feature study, according to the five-category framework presented by Birge [START_REF] Birge | Enhancing research into usable privacy and security[END_REF]. The remainder of the paper is organized as follows. Section 2 justifies the importance of this research by reviewing the related literature. Section 3 introduces the method used in the experiment: instrumentation, experimental procedure and analysis of characteristics. Section 4 shows the principal results obtained from the data collected. Finally, Section 5 presents some concluding remarks.

Related work

In recent years, relevant research has been carried out into PHRs, particularly with regard to the most popular PHRs, Google Health and Microsoft HealthVault. Martino and Ahuja [START_REF] Martino | Privacy policies of personal health records: an evaluation of their effectiveness in protecting patient information[END_REF] assessed these platforms to highlight those vulnerabilities existing in PHR privacy policy coverage and gaps in privacy policy notification mechanisms. The authors used well-researched evaluation criteria reported in literature [START_REF]Review of the Personal Health Record (PHR) service provider market: Privacy and security[END_REF][START_REF]Detailed PHR privacy report cards[END_REF]11].

Kotz et al. [START_REF] Kotz | A privacy framework for mobile health and home-care systems[END_REF] compare existing privacy frameworks and identify a privacy framework for mobile healthcare and home-care systems. Mohan and Blough [START_REF] Mohan | An attribute-based authorization policy framework with dynamic conflict resolution[END_REF] propose a framework which supports the need to change the rule and policy combination algorithms dynamically based on contextual information. Williams [START_REF] Williams | Social networking applications in health care: threats to the privacy and security of health information[END_REF] presents a survey of the research literature related to the security and privacy of personal health information in social networking applications, and provides a snapshot of privacy and security safeguards for social network websites. Huang et al. [START_REF] Huang | Privacy preservation and information security protection for patients' portable electronic health records[END_REF] design a method according to HIPAA guidelines to preserve the privacy and security of patients' portable medical records in portable storage media. Carrión et al. evaluate the privacy policies of 20 PHRs to check that the privacy of patients' data was preserved in accordance with HIPAA guidelines [START_REF] Carrión | Assessing HIPAA standard in practice: PHRs Privacy Policies[END_REF]. Sunyaev et al. develop criteria to assess the PHR systems from the users' viewpoint. The criteria were classified into three categories: patient information, personal control and additional services [START_REF] Sunyaev | Evaluation framework for personal health records: Microsoft healthvault vs. google health[END_REF]. To illustrate their proposal, these authors applied the criteria to Google Health and Microsoft HealthVault. Sunyaev et al. [START_REF] Sunyaev | Comparative evaluation of Google Health API vs. Microsoft Healthvault API[END_REF] also performed an evaluation of Google Health API and Microsoft HealthVault API. An evaluation of the ethical, legal, and social issues (ELSI) was presented by Cushman et al. [START_REF] Cushman | Ethical, legal and social issues for personal health records and applications[END_REF], who group this evaluation in four areas: privacy and confidentiality, data security, decision support, and the legal-regulatory regime for health data.

In this work, we analyze 22 free web-based PHRs, including Google Health and Microsoft HealthVault. To the best of our knowledge, no other studies have dealt with the security and privacy features of so many PHRs.

Classification Framework

Instrumentation

The problem of selecting the set of PHRs to be included in the study was confronted by consulting the web site of the American Health Information Management Association (AHIMA): myPHR. Its url is www.myphr.com and contains large amount of information on the PHRs. To the best of our knowledge, this web site provides the most comprehensive list of PHRs that a user can find, and has also been used to select PHRs in multi-source sampling [START_REF] Hulse | Deriving consumer-facing disease concepts for family health histories using multi-source sampling[END_REF]. It includes a section called "Choose a PHR" which was used to obtain a free web-based PHRs list. At the moment of accessing it on March 2011, a total of 29 free web-based PHRs (specified in Table 1) were retrieved. Note that AHIMA classifies the PHRs according to their format and cost. In this respect, some PHRs have premium accounts, and can thus also be classified as "for purchase''.

Experimental procedure

The privacy policy of each PHR selected was reviewed by one author. This review was carried out between February and March 2011. Difficulties were encountered in seeking the PHRs' privacy policy because some of them were not on the PHR home page and others were fragmented in several documents. While the review was conducted, the formulation criteria for the enquiries was carefully discussed and agreed in an attempt to obtain a comparative framework that would be as comprehensive and clear as possible. Moreover, a number of the listed PHRs were not in force, or had no privacy policy or an equivalent document. In conclusion, the original amount of PHRs for participation (29) was reduced to 22.

The ISO/TS 13606-4 [START_REF]Health informatics Electronic health record communication part 4: Security[END_REF] is a Technical Specification (TS) which provides a basic framework that can be used as a minimum specification for an EHR (Electronic Health Records) access policy, and a generic representation for the communication of policy information. We have adopted ISO/TS 13606-4 as the basis of a classification framework for the evaluation of the set of relevant PHR features. According to the ISO/TS 13606-4: (P1) "health records should be created, processed and managed in ways that guarantee the confidentiality of their contents and legitimate control by patients in how they are used"; (P2) "the subject of care has the right to play a pivotal role in decisions on the content and distribution of his or her electronic health record, as well as rights to be informed of its contents"; (P3) "the communication of health record information to third parties should take place only with patient consent". A number of PHR security and privacy features have been defined according to these principles. 3. Information related to the accesses and de-identified user information. 4. Information related to the accesses and identified user information. Security measures. This characteristic is related to the question What security measures are used in PHR systems? There are two types of security measures: physical measures and electronic measures. The physical security measures are related to the protection of the servers in which the data are stored. The electronic security measures are related to how stored and transmitted data are protected, for example, by using a Secure Sockets Layer (SSL) scheme. The values that this characteristic may take are: 0. Not indicated. 1. Physical security measures. 2. Electronic security measures.

Analysis of Characteristics

3. Physical security measures and electronic security measures.

Changes in Privacy Policy. This characteristic is related to the question Are changes in privacy policy notified to users? Changes in Privacy Policy should be notified to users in order to make them aware of how their data are managed by the PHR system. The values that this characteristic may take are: 0. Not indicated.

1. Changes are notified to users.

2. Changes are announced on home page.

3. Changes are notified to users and changes are announced on home page. 4. Changes may not be notified.

Standards. This characteristic is related to the question Are PHR systems based on privacy and security standards? The PHR systems analyzed use or are based on two standards: the Health Insurance Portability and Accountability Act (HIPAA) [21] and the Health On the Net Code of Conduct (HONcode) [START_REF] Boyer | The health on the net code of conduct for medical and health websites[END_REF]. The values that this characteristic may take are: 0. Not indicated. 1. HIPAA is mentioned. 2. System is covered by HONcode. 3. HIPAA is mentioned and system is covered by HONcode.

Results

Table 2 shows the results obtained for each PHR system included in the review. For example, Google Health takes the value 1 (according to the numbering in Section 3) for the PL characteristic. The scores were calculated according to the security and privacy characteristics of the PHRs. The following variables are considered in this study: The security level was quantified by employing the characteristics: Access audit and Security measures. The privacy level was quantified by using the characteristics: Privacy policy location, Cookies, Changes in privacy policy and Data accessed without the user's permission. Fig. 1 shows these scores as two overlapping histograms. In general, quite a good level can be observed in the characteristics analyzed. Nevertheless, some improvements could be made to current PHR privacy policies to enhance specific capabilities such as: the management of other users' data, the notification of changes in the privacy policy to users and the audit of accesses to users' PHRs, as shown in Table 2.

Conclusions

In this paper, a survey of 22 free web-based PHRs has been carried out. Most PHRs (86.36%) access different kinds of information without the users' explicit consent, but access to their identified information without their consent is never acceptable. Only thirteen PHRs (59.09%) notify or announce changes in their privacy policy to users. A number of PHRs (40.90%) do not provide any information about how the users provide access to their PHR data. In three PHRs (13.64%), physical security measures and electronic security measures are not reported. The Privacy Policies must include this aspect in order for users to perceive that their information will be protected by the PHR system. In a large number of PHRs (63.64%) the users cannot see an audit of accesses to their PHRs. This aspect should be improved because the users should be aware of how their information has been shared. A number of the PHRs reviewed (36.36%) are not regulated by a standard setting body. In future work, we intend to extend our analysis to a wider sample of PHRs, and compare the security and privacy features of free web-based PHRs with proprietary web-based PHRs.

Fig. 1 .

 1 Fig. 1. PHRs usability score distribution.

Table 1 .

 1 PHRs lists hosted in AHIMA.

	Format	Cost	Amount
	Web-based	Free	29
	Software-based	Free	0
	Paper-based	Free	3
	Web-based	For Purchase	63
	Software-based	For Purchase	1
	Paper-based	For Purchase	13

  Nine security characteristics were defined to analyze the PHRs: Privacy policy location, Data source, Data managed, Access management, Access audit, Data accessed without the user's permission, Security measures, Changes in privacy policy and Standards:Privacy Policy Location. This characteristic is related to the question Where is the Privacy Policy on the PHR web site? PHRs should provide a Privacy Policy which describes how users' data are used in order for users to be informed. The Privacy Policy should be easily accessible by users. The difficulty of Privacy Policy access is assessed by counting the number of links clicked. The values that this characteristic may take are: 0. The Privacy Policy is not visible or not accessible. Generally, the user is his/her data source, but there are PHRs which do not only use this source. Some contact the users' healthcare providers, others allow other users and different programs to enter users' data and others use self-monitoring devices to obtain users' data. The values that this characteristic may This characteristic is related to the question Who do the data managed by the users belong to? The users can manage their own data, but they can sometimes manage other users' data, such as that of their family. This characteristic is related to the question What data are accessed without the user's explicit consent? The PHR systems typically access certain data related to the users in order to verify that everything is correct. The values that this characteristic may take are: 0. Not indicated.

	Access management. This characteristic is related to the question Who can obtain
	access granted by the users? The users decide who can access their PHR data. The
	PHR systems analyzed allow access to be given to different roles. The values that this
	characteristic may take are:
	0. Not indicated.
	1. Other users and services/programs.
	2. Healthcare professionals.
	3. Other users.
	4. Other users, healthcare professionals and services/programs.
	Access audit. This characteristic is related to the question Can users see an audit
	of accesses to their PHRs? The values that this characteristic may take are:
	0. No.
	1. Yes.
	Data accessed without the user's permission.
	take are:
	0. Not indicated.
	1. User.
	2. User healthcare provider.
	3. User and his/her healthcare providers.
	4. User, other authorized users and other services/programs.
	5. Self-monitoring devices connected with the user.
	Data Managed. The values that this
	characteristic may take are:
	0. Not indicated.
	1. Data user.
	2. Data user and his/her family data.

[START_REF] Sood | Electronic Medical Records: A Review Comparing the Challenges in Developed and Developing Countries[END_REF]

. The Privacy Policy is accessed by clicking one link. 2. The Privacy Policy is accessed by clicking two or more links. Data Source. This characteristic is related to the question Where do users' PHR data proceed from? 1. Information related to the accesses.

2. De-identified user information.

Table 2 .

 2 Characteristics of each PHR system. PL: Privacy policy location; DS: Data source; DM Data managed; AM: Access management; AA; Access audit; DA: Data accessed without the user's permission; SM: Security measures; CP: Changes in privacy policy; S: Standards.

	Tool	PL	DS	DM AM	AA	DA	SM	CP	S
	1. Google Health	1	4	1	1	1	3	3	2	1
	2. ZebraHealth	2	1	0	0	0	1	3	4	1
	3. myHealthFolders	1	1	2	2	1	1	3	1	0
	4. Keas	1	4	1	0	0	2	3	3	0
	5. EMRy Stick Personal Health Record	2	1	1	0	1	1	0	0	0
	6. My HealthVet	2	1	1	2	0	1	2	0	1
	7. myMediConnect	0	3	1	2	0	0	3	0	1
	8. MyChart	1	2	1	0	1	4	0	0	1
	9. MedicAlert	1	1	1	3	0	2	3	2	0
	10. Microsoft HealthVault	1	4	1	4	1	1	3	2	3
	11. MediCompass	1	5	1	2	0	2	3	0	3
	12. TeleMedical	1	1	2	0	0	0	2	2	2
	13. Health Butler	1	1	1	2	0	2	0	4	0
	14. NoMoreClipBoard.com	1	3	2	2	1	2	2	2	1
	15. MiVIA	1	0	1	2	0	3	3	2	1
	16. iHealthRecord	1	0	0	0	0	1	2	4	0
	17. Dr. I-Net	1	3	1	2	0	0	3	0	0
	18. My Doclopedia PHR	1	2	1	2	0	3	2	2	1
	19. dLife	1	0	0	0	0	4	2	2	0
	20. RememberItNow!	1	4	1	4	1	3	2	3	0
	21. MedsFile	1	1	1	0	1	4	1	1	0
	22.Juniper Health	1	1	2	0	0	2	3	2	0

Acknowledgments. This work has been partially financed by the Spanish Ministry of Science and Technology, project PANGEA, TIN2009-13718-C02-02.


Sidney Fels
email: ssfels@ece.ubc.ca

Michael Lyons
email: lyons@im.ritsumei.ac.jp

Interaction and Music Technology

Keywords: Music, Technology, Interactive Techniques, HCI

This SIG intends to investigate the ongoing dialogue between music technology and the field of human-computer interaction. Our specific aims are to explore major findings of specialized musical interface research over recent years and convey these to HCI researchers who may be interested but not yet active in this area, as well as to consider how to stimulate closer cooperation between music technology and HCI research communities.

Introduction

Advances in digital audio technologies have led to a situation where computers now play a role in most music production and performance. Digital technologies offer unprecedented opportunities for the creation and manipulation of sound, however the flexibility of these new technologies implies a confusing array of choices for composers and performers. Some musicians have embraced this challenge by using computers directly to create music, leading to an ongoing development of new musical forms. However, most would agree that the computer is not a musical instrument, in the same sense as traditional instruments, and it is natural to seek to design interfaces which allow the computer to be 'played' in fashion suited for human bodies and brains. A decade ago we organized a workshop on "New Interfaces for Musical Expression" [START_REF] Poupyrev | New interfaces for musical expression[END_REF] at the ACM CHI 2001 conference, with the aim of addressing this issue by exploring connections with the better established field of human-computer interaction. The workshop soon developed into an annual international conference (NIME), which attracts several hundred participants every year. NIME is now considered to be a major venue for the publication of advances in music technology, with an emphasis on methodology and technology from the broader field of human-computer interaction.

The authors recently began a series of outreach activities [START_REF] Fels | Creating new interfaces for musical expression: introduction to NIME[END_REF] to bring a summary of findings of the first ten years of the NIME conference to the attention of the broader research community in human-computer interaction. While some of the members of the music technology research community continue to participate in HCI-related conferences such as CHI and INTERACT, presence of music technology related studies at these conferences is relatively sparse in view of the broad interest in music and its importance for emerging technologies. Moreover we feel that some of the important findings being presented at NIME could be of general interest to researchers who may not have a specific focus on music-related projects.

We therefore propose to invite any participants of INTERACT 2011 who may be interested in music as a potential domain for the study of human-computer interaction to join us for a discussion exploring the past, present, and future dialogue between interaction and music technology.

Format of the Discussion

This SIG meeting will consist of a structured discussion addressing several topics of shared interest in the music technology and HCI research communities. Topics for discussion include, but are not expressly limited to the following:

•

  Design and Aesthetics of musical interface design • Tools for prototyping new music technology • Methodologies for evaluating musical interfaces • Models of 'Playability' of new musical interfaces • Theories of human action and perception in musical performance • Mobile music making • Collaborative music making • Networked music making • Music technology for novices • Musical interfaces in the HCI Curriculum • Future of Music, Technology, and HCI • How to better represent music themed research in the HCI community


Celeste Lyn Paul
email: cpaul2@umbc.edu

Emile Morse
email: emile.morse@nist.gov

Aiping Zhang
email: aiping.zhang@nist.gov

Yee-Yin Choong
email: yee-yin.choong@nist.gov

Mary Theofanos
email: mary.theofanos@nist.gov

A Field Study of User Behavior and Perceptions in Smartcard Authentication

Keywords: Human factors, multi-factor authentication, security, smartcard

A field study of 24 participants over 10 weeks explored user behavior and perceptions in a smartcard authentication system. Ethnographic methods used to collect data included diaries, surveys, interviews, and field observations. We observed a number of issues users experienced while they integrated smartcards into their work processes, including forgetting smartcards in readers, forgetting to use smartcards to authenticate, and difficulty understanding digital signatures and encryption. The greatest perceived benefit was the use of an easy-to-remember PIN in replacement of complicated passwords. The greatest perceived drawback was the lack of smartcardsupported applications. Overall, most participants had a positive experience using smartcards for authentication. Perceptions were influenced by personal benefits experienced by participants rather than an increase in security.

Introduction

In an attempt to protect their valuable data, organizations such as banking and financial institutions, corporations, and governments spend a large fraction of their information technology budgets on security [START_REF]Response to White House Cyber Review Questions[END_REF]. Because of security, users of these systems must adhere to new business rules, information security policies, and training sessions -which takes time and effort away from their primary jobs.

There is a constant tension between users who are trying to get work done and security specialists who lock down systems to make them safer and more secure. Despite not being the security specialists, the burden of maintaining a secure work environment often falls on users. While the efficacy of such security practices can be debated [START_REF] Herley | So Long and No Thanks for the Externalities: The Rational Rejection of Security Advice by Users[END_REF], it is clear that there is a role for HCI researchers to play in understanding how users and security can be optimized to serve both. Users have often been described as the "weakest link" in information security, but they lack the proper motivation and tools to effectively contribute to the information security ecosystem [START_REF] Sasse | Transforming the 'weakest link' -a human/computer interaction approach to usable and effective security[END_REF]. Many approaches designed to increase security have placed additional burdens on users. Users now have longer, more complex, and more frequently changing passwords [START_REF] Summers | Password policy: the good, the bad, and the ugly[END_REF]; more restrictions on what they can do on the web [START_REF] Schechter | The Emperor's New Security Indicators[END_REF], and they must use additional keys, tokens, or cards in order access information systems [START_REF] Aussel | Smartcards and Digital Security[END_REF].

Homeland Security Presidential Directive 12 (HSPD12) [START_REF]Policy for a common identification standard for federal employees and contractors[END_REF] defines requirements for a standardized, U.S. Government-wide identification mechanism for a reliable identification credential. Personal Identity Verification (PIV) [START_REF]Personal identity verification (PIV) for federal employees and contractors[END_REF] is a smartcardbased multi-factor authentication (MFA) mechanism to increase security of government resources. The PIV smartcard contains personally identifiable information (PII) about the card holder such as his full name and agency, PKI certificates for authentication, encryption, and digital signature, and biometrics such as fingerprints and photo. It can be used for physical building access, for information system authentication, to support PKI, and as an identity card.

The PIV smartcard affects hundreds of thousands of U.S. Government employees and contractors. While the use of the smartcard is mandated by policy, we are concerned with how introducing this additional authentication factor will impact the perceptions, behaviors, and work processes of so many users. The PIV smartcards are meant to be used throughout the day as often as passwords would be used. This makes smartcard use very different from other types of card scenarios users may have experience with, e.g. weekly use of an Automatic Teller Machine (ATM) card. This is one of several reasons why we studied user behavior and perceptions using smartcards.

Research Goals

The purpose of this study was to understand factors that affect user behavior and perceptions in the use of smartcards for authentication and to examine factors that affect user behavior and perceptions of security in general. We had three main goals for the study. First, we wanted to learn how users would use the smartcards in their everyday work processes. Second, we wanted to learn how users' work processes might change to accommodate smartcard use. Third, we were interested in the user benefits and drawbacks of using smartcards in authentication.

Related Work

Multi-factor authentication (MFA) is the use of two or more independent security factors to authenticate to an information system. There are three factors commonly used in MFA [START_REF] O'gorman | Comparing Passwords, Tokens, and Biometrics for User Authentication[END_REF]. The first factor is "something you know" such as a password, passphrase, or personal identification number (PIN). While passwords are perhaps the most common "something you know" authentication factor, challenges to password usability include the cognitive limits related to the number of passwords [START_REF] Florencio | A large-scale study of web password habits[END_REF] and length of passwords [START_REF] Summers | Password policy: the good, the bad, and the ugly[END_REF] users must remember. A second factor is "something you are", i.e. using biometrics such as fingerprints or a facial image. A benefit of using a biometric as an authentication factor is that it does not depend on secrecy [START_REF] O'gorman | Comparing Passwords, Tokens, and Biometrics for User Authentication[END_REF]. It is something users will always have with them and will never need to remember. The third factor is "something you have" such as a key, token, or card. ATMs are a classic example of MFA using a card in combination with a PIN [START_REF] Weir | Usable security: User preferences for authentication methods in eBanking and the effects of experience[END_REF]. There are also additional authentication factors not as commonly used. For example, "someone you know" considers social network characteristics such as who you went to school with [START_REF] Brainard | Fourth-Factor Authentication: Somebody You Know[END_REF][START_REF] Schechter | It's not what you know, but who you know: a social approach to last-resort authentication[END_REF]. Another factor, "something you do" considers behavioral characteristics such as online shopping habits [START_REF] Jakobsson | Implicit Authentication for Mobile Devices[END_REF].

In this paper, we focus on the "something you have" authentication factor, specifically the use of smartcards in MFA. Smartcards are tokens that include embedded chips that can store information. Smartcards are different from magnetic strip cards in that while both can store data, the chip in smartcards makes them more secure and provides more features. A smartcard can be used for MFA in several ways. The smartcard can be "something you have" by acting as a token. It can be used along with "something you know" such as a password or PIN. Smartcards can also support the "something you are" factor by storing biometric data on the embedded chip that can later be matched to the user.

Examples of smartcards used as authentication tokens similar to PIV include the European national electronic identity (e-ID) card [START_REF] Arora | National e-ID card schemes: A European overview[END_REF] that stores different types of PII depending on the requirements of the issuing country. The U.S. Department of State uses the Biometric Logical Access Development and Execution (PKI/BLADE) card as an employee identity card and authentication token [START_REF]Identity, Credential and Access Management Subcommittee[END_REF][START_REF]Cost/Benefit Comparison between PKI/BLADE and Passwordbased Authentication v1.0[END_REF]. The U.S. Department of Defense uses the Common Access Card as a military identity card, Geneva Conventions Identification Card, and authentication token [START_REF]Identity, Credential and Access Management Subcommittee[END_REF].

While work has focused on smartcard security weaknesses such as problems with the embedded chip and PIN mechanism [START_REF] Murdoch | Chip and PIN is Broken[END_REF] and PIV implementation standard [START_REF] Irwin | Identity, Credential, and Access Management at NASA, from Zachman to Attributes[END_REF][START_REF] Karger | Privacy and Security Threat Analysis of the Federal Employee Personal Identity Verification (PIV) Program[END_REF], very little work has looked at smartcard usability [START_REF] Sasse | Usability and Trust in Information Systems[END_REF]. Proctor et al. [START_REF] Proctor | A Task Analysis of Usability in Third-Party Authentication[END_REF] compared multiple authentication methods through formal task analysis. They warn that the physical manipulation of a smartcard in the authentication process can add complexity to the authentication task and reduce ease of use compared to other authentication methods.

Braz and Robert [START_REF] Braz | Security and Usability: The Case of the User Authentication Methods[END_REF] conducted a comparative analysis of different authentication methods. They compared methods such as passwords, smartcards, fingerprints, and keystroke patterns, on qualities such as benefits, drawbacks, security, and usability. Overall, they found that the smartcard rated as one of the most secure and usable methods for authentication.

Baldwin and Malone [START_REF] Baldwin | Utilizing Smart Cards for Authentication and Compliance Tracking in a Diabetes Case Management System[END_REF] described the use of smartcards in a health management system. The ability to store information, such as PII, increased the usefulness of smartcards for authentication beyond being a token. Patients identified themselves by presenting the smartcard and providing a PIN. Visits for care and therapy were recorded on the smartcard, creating a case history. Patients paid for services and filed claims using the smartcard. The smartcard provided an easy way to identify patients and help the patients manage their health care accounts.

The U.S. Department of State analyzed the impact of the PKI/BLADE smartcard with PIN on their userbase [START_REF]Cost/Benefit Comparison between PKI/BLADE and Passwordbased Authentication v1.0[END_REF]. Their smartcard system allowed users to replace multiple username/password authentication credentials with a single smartcard/PIN credential. They analyzed their technical support logs to understand how the smartcard system affected user support requests. Before smartcard deployment, password reset support requests averaged 25.8% of all technical support requests per year. After smartcard deployment, password reset support requests dropped to an average of 10.6% of all technical support requests per year.

Strouble et al. [START_REF] Strouble | Productivity and Usability Effects of Using a Two-Factor Security System[END_REF] conducted a survey that looked at issues concerning security, usability, and productivity of smartcards. They found the use of a PIN instead of a password improved the security of the authentication mechanism, but did not necessarily increase usability of the smartcard system. Sixty-seven percent of the participants forgot their smartcard in a smartcard reader at least once, resulting in potential security risks. Six percent of those participants had their smartcard lost or stolen, resulting in security risks, replacement costs, and productivity loss.

Methodology

Participants

We studied 24 participants from a U.S. Government research institute over a period of approximately 10 weeks. There were 10 males, an average age of 47 (SEM ± 2), and and a distribution of education that was representative of the organization (8 high school degrees; 10 college degrees; 6 post-graduate degrees). Ten participants were engaged in technical work; five were support staff, e.g. secretarial work; and nine worked in an administrative specialty, e.g. finance. Two participants reported having experience with smartcards before the study.

Our study participants were recruited from an institution-wide technology pilot of 100 users testing the PIV smartcard technology. The technology pilot used the same smartcard system as our study; the only difference was that our study included additional research methodologies to assess the smartcard system. Recruitment in the technology pilot was designed to include test users from a representative sample of users in the institution, except for research scientists. Even without scientists, the technology pilot had a sampling of job roles and education similar to an industry corporation. Our study sample reflected the same demographics as the overall groups recruited in the technology pilot; except in our study there was a higher proportion of females. Participation in both the technology pilot and our study was voluntary. However, all participants were aware that the smartcard system would soon be mandatory for all users.

Study Environment

Participants were given access to a fully functional PIV smartcard authentication system. Although we recruited from a "technology pilot," it was the system intended for institute-wide implementation. Participants had both an identity card and a smartcard. Participants also had both smartcards/PINs and usernames/passwords. The smartcard PINs were 6 to 8 numbers long and selected by participants at the time of smartcard issuance. The password length policy was 10 characters at the beginning of the study and changed to 12 characters one week after the study began. Passwords expired every 90 days and every participant changed passwords at least once during the study. Participants used their own computers and workspaces. Participants used either a laptop or desktop computer. Those with laptop computers docked the laptops at their workspaces and used external monitors, keyboards, and mice. Several participants with laptops occasionally worked from home, and several participants without laptops occasionally worked from home using their personal computers. All computers were running Windows XP with ActivClient smartcard middleware 1 . One of three types of smartcard readers was used by each participant (Fig. 1): an external USB reader (n=13), an internal laptop reader (n=6), or an integrated keyboard reader (n=5). Participant tasks were limited to those supported by the smartcard implementation. Supported tasks included using the smartcard to login/logout and to lock/unlock a computer, encrypt/decrypt and digitally sign an email or document, and authenticate to several smartcard-enabled web applications. In each case, participants could authenticate with either their smartcards/PINs or usernames/passwords.

Data Collection Methods

Several ethnographic research methods were used to obtain a breadth of coverage as well as a depth of understanding of our participants over the course of the study.

1 Specific hardware and software products identified in this report were used in order to perform the evaluations described. In no case does such identification imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the products and equipment identified are necessarily the best available for the purpose. Participants were asked to respond to the following statements in a standardized survey two or more times over the course of the study in order to evaluate their experiences with the smartcard system.

1. I am confident I know how the smartcard works and what it does.

2. I take the smartcard with me every time I leave my computer.

3. Using the PIN for the smartcard is easier than using a password. 4. The smartcard makes the login process easier than the current password-based login system. 5. The smartcard makes the login process faster than the current password-based login system. 6. Compared to using passwords, using the smartcard is more secure. 7. I [plan/will continue] to use the smartcard. 8. I would encourage my colleagues to switch to the smartcard. 9. I am [looking forward to/have] enjoyed using the smartcard Survey statements were rated on a 5-point scale from 'Strongly Disagree' (1) to 'Strongly Agree' (5) with 'Neither Agree nor Disagree' (3) as neutral. Participants were also given space for several questions to provide additional comments. While the statements were framed positively and may bias responses towards the positive, we analyzed the results in terms of relative change rather than absolute value.

Daily Diaries.

Participants were asked to keep daily written diaries of notable smartcard events. We provided notebooks for them to write in and a guide on the types of events to note, such as forgetting to login using the smartcard. If participants were not comfortable writing in the provided notebook, they were encouraged to keep notes in an electronic document.

Daily Email Surveys. The daily email surveys asked participants to report about specific smartcard usage in a Yes/No format, such as "Did you use your password today?", and also provided an area for additional comments. The purpose of the daily email surveys was to supplement the daily diaries as a way of reporting critical events, and not as a quantitatively evaluated questionnaire. Site Visits and Interviews. Site visits allowed us to observe smartcard use in the participant's natural environment. Interviews provided an opportunity to discuss participant's smartcard experience since the previous visit and review any critical events that were reported in the daily diaries or daily email surveys. Field notes from the site visit observations and interviews were recorded.

Procedure

The study was conducted in three phases: Smartcard Installation, Smartcard Use, and Study Wrap-Up. See Table 1 for a summary of research activities.

Smartcard Installation. Before we met with participants, we sent the Pre-Install survey and a technical support person provided a brief training document via email.

Participants were asked to complete the Pre-Install survey before the first site visit.

The training document contained a step-by-step scenario guide on how to complete smartcard-related activities. The technical support person then installed the smartcard hardware and software, demonstrated how to use the smartcard, and guided participants through several smartcard use scenarios. The support person also asked participants to perform tasks, such as locking and unlocking their computers. After installation was complete, we conducted the Post-Install site visit. We observed participants using their smartcards, interviewed them about their first time experiences with the smartcards, administered the Post-Install surveys, and provided them the Daily Diaries and guide.

Smartcard Use. During smartcard use, participants were asked to keep diaries of notable smartcard usage and events. The daily email survey was sent near the end of the day for participants to complete and return. Researchers would review the daily diaries and email surveys before each site visit in order to discuss any critical events during the interview, if necessary. Site visits were conducted two and six weeks after Smartcard Installation. During these site visits the researchers observed participants using their smartcards, interviewed them about their smartcard experiences to date, and administered the 2-week or 6-week surveys.

Study Wrap-up. Study wrap-up activities took place between 10 to 12 weeks after Smartcard Installation. Extended business travel or paid time off was not counted towards participants' total study time. This resulted in slightly longer periods of participation for a few participants. During the last site visit, we observed participants using their smartcards, interviewed them about their overall smartcard experiences, administered the Exit surveys, and collected the Daily Diaries.

We report the analysis of our study results in three sections. First, we provide descriptive statistics for the quantitative data collection methods. Second, we discuss our analysis of the qualitative data, including in-context discussion of the quantitative results. Third, we reflect on the methodology and provide lessons learned.

Descriptive Statistics of Quantitative Methods

Periodic Surveys. Results from the periodic surveys are reported as mean values with 95% confidence intervals in Table 2. Pairwise comparisons are discussed alongside the qualitative results. The Wilcoxon signed-rank test is used for pairwise comparisons and Kendall's correlation is used for measuring relationships.

Daily Email Surveys. There were a total of 682 daily email surveys collected; the average number of emails collected from each participant during the study for each

Analysis of Observations

Table 3 provides a summary of reported issues with discussion in the following section.

User Confidence. Participant confidence in using the new smartcard authentication system increased immediately after installation (Pre-install/Post-install: W=3.87, p=0.001) and remained high throughout the study (Post-install/Exit: Kendall's W=0.04, p=0.41).

Seventeen participants reported reading the provided training materials before the smartcard hardware and software were installed. Several participants indicated that they preferred in-person training rather than reading the training documentation. One participant noted that he preferred a "hand holding demo" (P3) when using a new system for the first time. Another participant indicated that she "learn[s] better by hands on training" (P12). The personalized training could be a factor in the significant increase of confidence post-installation. However, not all participants felt the need for attended training. As one participant explained, "I prefer to jump in and just start using any new product, referring to the documentation only when I get stuck or find features I'm curious about" (P16).

Smartcard Readers. Fifteen participants used USB readers that were placed on their desks. The location of a reader on a desk varied as did the number of items that might obscure it from view. The reader itself added to clutter on the desk, as one participant commented, "I know I'm going to dislike the wire connecting the smartcard reader to the computer -makes for a messy desk!" (P7). This participant attempted to clean her desk by moving the reader out of the way, but later attributed the position of the reader to why she may have forgotten her smartcard, "I tidied [

the] smartcard reader cord -made [the] reader less intrusive, but moved it further out of [the] workspace. It may be a reason for forgetting to remove the smartcard to lock my computer."

Another participant also blamed his USB reader for why he forgot his card, "I walked away at one point and forgot my smartcard. This has happened once or twice, and it makes me think that smartcard readers should probably be fairly visible" (P14). Later that week, he tried a keyboard reader with success, "Switched out my [USB] reader & keyboard for a new keyboard that included a smartcard reader. I like the setup much better. Less clunky, and the smartcard is more visible." Besides adding to clutter on the desk, an additional problem with the USB reader was that it was not attached to a stable object and required participants to use two hands when removing and inserting the smartcard. One participant remedied this problem by attaching her USB reader to her computer with rubber bands.

Not all participants could use a keyboard reader because their keyboards were in keyboard trays attached to their desks. Several of these participants who used keyboard trays also kept their keyboards partially hidden under their desks while they typed. Unless the keyboard was completely pulled from under the desk, it would not fit under the desk with the smartcard in the keyboard reader.

Using Smartcards. Remembering to remove their smartcards from their readers was a commonly reported incident by Thirteen participants (54%) forgot their smartcards in their readers at least once during the study. The most common scenario for participants to forget their smartcards was during short trips out of their work areas, such as down the hall to visit a colleague or visit the restroom. Three participants forgot their smartcards in their readers after leaving an access-controlled area, and had to rely on their non-smartcard identity cards to gain access to their buildings. Six of 24 participants forgot smartcards in their readers overnight. One participant forgot her smartcard in the reader overnight and drove back to campus to retrieve the card. Three participants reported forgetting their smartcards at home and had to use their passwords to login. Incidents where participants left their smartcards in the readers overnight or at home only occurred once or twice per participant. Even though half of the participants reported in interviews that they forgot their smartcards in the readers at some point during the study, most participants reported remembering their smartcards most of the time by the end of the study.

Many participants who forgot their smartcards in readers early in the study reported that they forgot their smartcards less often as time went on. A few of these participants pointed out that it seemed to take them about one month before they developed a habit for using and remembering their smartcards; however, this change is not indicated in the periodic survey results. One participant described a system she developed to help her remember her smartcard; when she removed her smartcard from her badge holder, she would place the badge holder in front of her keyboard. It served as a reminder for her to take her smartcard before she left the office. This participant did not report forgetting her smartcard at any time during the study. There were several reasons why participants did not use the smartcard to login or lock their computers. In the beginning of the study, most participants simply forgot to use their smartcards because it was not yet a habit. This was especially true for participants who had good security habits, such as those who consistently locked their computers with the keyboard when they left their workspace. As one participant stated, "This is going to take some getting used to -I have been using the keyboard to lock my machine for 10 years -hard habit to break" (P21). Other reasons participants did not use the smartcard to login or unlock their computers included because they forgot their smartcard at home, were using multiple computers at once, or were prompted to enter a username and password by the software. The design of the login dialog may have contributed to whether participants used their usernames and passwords to login. By default, the computer displayed a username and password dialog instead of a PIN dialog. A few participants discovered if they pressed Escape on the keyboard with their smartcards in their readers, a PIN dialog would display. This information was shared with the other participants. If participants did not use the smartcard to login to a session, they would not be able to remove the smartcard from the reader in order to quickly lock the computer. This caused some confusion in the beginning of the study when participants were not yet consistently using the smartcard to login, "After logging in with my keyboard, I locked the machine but the smartcard could not unlock it until I logged in and locked the machine again" (P14).

Unlocking computers also caused confusion for several participants in the beginning of the study. When returning to their locked computers, out of habit these participants would use CTRL+ALT+DEL in order to cancel the screensaver and unlock their computers. Using this key combination displayed the username and password dialog. Since participants were prompted with username and password dialogs, they entered their usernames and passwords and created sessions that could not be locked by removing their smartcards. It took time and practice for these participants to get used to using their smartcards to unlock their computers without pressing Participants were neutral whether smartcard authentication was faster than passwords (Exit: 3.64 ± 0.49). Nine participants noticed that using smartcards took longer to authenticate and login than using usernames and passwords. Smartcard login was observed by participants to take 10-30 seconds longer than their password logins. The physical act of inserting the card also added time to the login process. While the smartcard is slower in some cases, most participants considered the overall system tradeoffs and still felt smartcards were faster and easier to use, "Unlocking when I returned to my desk was simple and no harder or time consuming than username and password -maybe easier" (P7).

Participants who worked both at their computers and elsewhere in their work areas often experienced automatic computer screen locking after 15 minutes of inactivity. When the screen automatically locked with a smartcard in the reader, a message describing an unattended smartcard appeared. Participants who frequently worked at their work areas found these error messages frustrating, "It's not unattended, I'm right here!" (P17). However, participants who accidentally locked their computers using their keyboards felt the unattended message was useful, "The message helped me not forget my smartcard when I accidentally locked using the keyboard" (P2).

Passwords vs. PINs. Overall, participants found using the smartcard easier than using passwords (Exit, 4.24 ± 0.31). Participants also found that logging in with the PIN was easier than using their passwords (Exit, 4.32 ± 0.39). Many participants noted that the PIN was an important feature of their positive smartcard experience, particularly for its ease-of-use. The PIN was numeric-only while the system password consisted of numbers, upper-and lower-case letters, and special characters. The PIN never expired compared to the system password that expired every 90 days. The PIN was six to eight characters in length compared to the system password requirements of 10 or 12 characters in length. One participant noted the importance of the PIN not changing, "If the PIN has to be changed as often as the password, there would be a reduced benefit to having the PIN" (P21). Password length had a noticeable effect on participants' perceptions. Several participants complained how the new 12-character password requirement made it more difficult to remember their passwords.

The password length requirement was not the only burden passwords placed on participants. Smartcard use prevented participants from practicing their passwords as often as before the study. Several participants felt they were at risk for forgetting their passwords, "It is an effort remembering my system password" (P15). Some participants needed their primary network password to login to computers that were not smartcard-enabled, providing an opportunity to practice their passwords if they were synced. However, participants who did not sync passwords or used their passwords to only login to their computers were at a greater risk for forgetting.

Participants also described various ways they managed passwords before their experience with smartcards. Nine participants reported managing their passwords by recording them on paper and storing them in their wallets, purses, or drawer in their offices. Some participants also attempted syncing passwords for multiple applications so they had fewer passwords to remember. However, not all password requirements were the same and it was easy for their to get out of sync. It was also a hassle to retrieve a password for every account. Two participants reported using software to save and manage passwords. Some participants anticipated the smartcard moving towards a SSO solution, "The idea of having one "pin" for all applications is a dream come true! Also -less work for both user and the IT help desks for resetting passwords!" (P18).

Certificates. Authentication was supported for several web applications. Participants authenticated to web applications by visiting the login page. A browser dialog appeared asking participants to select a certificate to use for authentication. For web application authentication, the certificate used for authentication did not matter. However, the authentication process was different depending on which certificate participants choose. If participants chose the non-repudiation certificate used for digital signatures, they were asked to enter a PIN. If participants chose the certificate used for encryption, they were not asked to enter a PIN. However, without expert knowledge in certificates it was difficult for participants to identify the encryption certificate from the digital signature certificate.

The smartcard authentication system also does not allow certificates to be backed up or saved/transferred. Three participants expressed concerns about the lifetime of certificates used to encrypt email and documents. If a smartcard is lost, stolen, expires, or is replaced, the certificates are lost forever. Previously encrypted email and documents could no longer be decrypted and the information would no longer be accessible. One of the participants worked with financial information and was concerned with not being able to access old encrypted data because his data needs to be available for auditing.

Digital Signatures and Encryption. The most infrequently used smartcard features were digital signatures and encryption. Once familiar with the functionality, participants were comfortable with digitally signing and encrypting email and documents and found both easy to use. However, most participants did not know when they would have a need to sign or encrypt an email or document, even after reading the sample use cases in the training document and after discussions with researchers during interviews. The training documentation explained how to sign and encrypt, but not why a participant would want to do so. No participants indicated a need for signing email or documents; although, several participants routinely tested the features. Several participants stated they would not consider using signing and encrypting unless it became policy, "I see no value in a digitally signed email and would do so only if I was required to" (P23). Two participants tried encryption to send passwords through email, and found it useful. The few participants who considered using signing and encrypting were those who already used some type of signing and encrypting in other applications.

All participants unintentionally signed email at some point in the study due to a technical problem that temporarily changed an option in their email clients. While most participants immediately noticed a difference in behavior and found typing a PIN for sent email inconvenient, one participant decided to experiment with email signing for the rest of the study. This participant did not find digital signatures a huge burden, but he was also a technical user who understood the purpose of digital signatures. He also acknowledged the functionality might not be for everyone, "I send a small number of emails on a typical day, so it isn't a big deal for me, but if I had to enter [the PIN] 50 or 100 times a day, it would become bothersome" (P21). Few participants could describe a practical use for digital signatures. As one participant expressed his doubts about the usefulness of digital signatures, "I don't see the point. People are going to know who I am based on what I say in the email" (P2).

Participants who worked regularly with PII considered the possibility of signing and encrypting email or documents, but in practice found it impossible to use with the current smartcard implementation. PII in the workplace was shared between coworkers through secure applications, shared files on a shared remote storage location, or paper. Sharing PII out of the workplace with external contacts was the most common scenario where encryption would be useful. However, a participant could not encrypt a message to another user or verify that user's signature unless they had the user's public certificate. There was no infrastructure to easily obtain, share, and verify certificates from contacts outside the institute. Several participants explained how they thought encryption would be more useful once they knew their colleagues outside the institution were setup and supported for sharing. Security Behavior. Many participants commented that they felt the smartcard was more secure; however, reasons why they felt the smartcard was more secure varied. For some participants, using the smartcard "enforces good habits" (P14) and encouraged participants to lock their computers. As one participant described, "I felt my computer was more secure than ever before because I was forced to secure my computer each time I left my office by taking my smartcard with me each time" (P15). At the same time, one participant who had already developed good computer locking habits was afraid that the smartcard had negatively impacted how often she locked her computer in the beginning of the study.

While participants felt the smartcards were more secure, few could articulate how or why. Three participants explained the smartcard was an additional security factor. Two participants noted the smartcards increased security because they would be difficult to crack or copy. There were mixed feelings about the use of a PIN instead of a password. Some participants felt that the shorter PIN was a benefit because the PIN is easy to remember and security would increase because it would not need to be written down. However, one participant was concerned that the PIN was not complex or long enough and might pose a security risk, "The PIN for the smartcard is all numeric & 6-8 digits. Not sure if the multi-factor aspect makes it more secure than 1 more complex password alone" (P23).

At the beginning of the study, several participants described themselves as being very diligent about security. Participants who seemed to have the best security habits, such as consistently locking their computer screens when leaving the office, were those who worked with PII or financial data. These participants were very aware of the sensitivity of the information they worked with, and felt that most security measures were justified. Participants who did not these job roles had very different attitudes toward security. There seemed to be a high amount of inter-office trust, i.e. coworkers were not the threat. Two participants indicated they left their smartcards near their readers when they temporarily left their workspaces. One of these participants attempted to justify this behavior, "It is OK since no one can get to my computer without the PIN and my other card can get me in the building." (P18). Although this participant was warned that her non-smartcard identity card would be phased out, she did not consider this when she developed this behavior.

Overall Experience. Even though each participant reported at least one problem or issue regarding smartcard use during the study the overall satisfaction of participants at the end of the study seemed positive. All but 3 participants (88%) indicated during the exit interview that they would recommend the smartcard to their colleagues. In general, participants were positive about using the smartcard, especially those in administrative job roles. These participants used the smartcards to access multiple applications and described a noticeable benefit.

Overall, participants were very positive about continuing their smartcard use after the study (Exit, 4.48 ± 0.32). However, not all participants had a consistently positive experience with the smartcards. Bad experiences and general frustration with the smartcards seemed to have an effect on some participants' behavior and perceptions. While minor problems, especially at the beginning of the study, were expected and accepted by most participants, issues that were persistent and affected work productivity were not acceptable. Early frustration with the smartcard had noticeable effects, "Off to a bad start today and never fully recovered. I didn't use the smartcard for most of the day" (P7). Although this participant had a particularly frustrating day, she resumed using the smartcard the next day and reported positive comments about her smartcard experience during the exit interview.

Another participant could not find a way to fit smartcard use into her existing work process. The smartcard authentication was noticeably slow to her and she described being "always in a hurry to login" (P24). She explained in the exit interview that if the smartcard became policy, she would use it; however, there were not enough benefits to encourage her to continue using the smartcard voluntarily. This benefit tradeoff was discussed by another participant who shared the same sentiments about recommending the technology, "I can't really recommend it, as it has few clear benefits to offset the downsides" (P23).

Methodology Lessons

This study utilized a number of qualitative field methods including daily diaries, daily surveys, periodic surveys, periodic user interviews, and field observations. The lessons learned in this study will help improve the design and methods of future work.

Most participants kept the written diary until the 2-week interview, but few participants wrote much after that. Several participants kept a written journal in a text file on their computers because they felt it was easier to keep than a diary. Most participants summarized daily diary events in the daily surveys. Despite the lack of participation in keeping the daily diaries, we feel that providing the diaries helped participants understand the purpose and type of information we were interested in. Although the study designs were different, our methodology experiences are consistent with those reported in a diary study of password use conducted by Inglesant and Sasse [START_REF] Inglesant | The true cost of unusable password policies: password use in the wild[END_REF].

As previously discussed, daily surveys were answered more frequently by participants in the beginning of the study than at the end of the study. When asked during interviews, several participants reported not responding to daily surveys because they did not have new events or comments to report and felt their responses would not add value to their participation. Modifying daily survey questions to fit with participants' evolving user experience could help encourage continued daily survey participation.

We observed participants' smartcard use from installation through 10 weeks of use. We observed new behaviors and perceptions up to six weeks after installation. After six weeks up until 10 weeks of observation, the rate of new observations was infrequent. At a minimum, we recommend users be studied for six weeks to be able to observe any transitions from learning to everyday use. Observations for less than two weeks after installation would not be sufficient to fully understand the impact the new technology has on participant behavior and perceptions.

Summary

Over the course of 10 weeks, we studied how 24 participants used smartcards in multi-factor authentication. We had three main goals for the study. First, we wanted to learn how users would use the smartcards in their everyday work processes. All participants quickly became confident using the smartcards and reported them easier to use than password authentication. One consistent problem was that many participants forgot to use their smartcards to login, lock, or unlock their computers during the beginning of the study. One reason was because they had not yet developed a habit for using their smartcards; another reason was that the form factor of the smartcard reader also had an effect on participants forgetting their smartcards.

Second, we wanted to learn how users' work processes might change to accommodate smartcard use. We found two ways smartcards changed the way some of our participants worked. First, the physical presence of new hardware to interact with had an effect on user behavior with the smartcard system. Some participants felt the location of their smartcard readers was the cause of forgetting their smartcards in the readers. They experimented with the location and form factor of their smartcard readers until they were satisfied with their setup. We also describe one participant who developed a novel strategy to help her consistently remember her smartcard.

Second, digital signatures and encryption added an extra element to participants' document and email usage. Several participants experimented with digital signatures and encryption to see how they could incorporate the functionality into their work processes. However, few participants had a need for digital signatures and encryption.

Third, we were interested in the user benefits and drawbacks of using smartcards in authentication. The greatest perceived benefit of using smartcards was the use of a PIN in replacement of a password. Participants immediately noticed the difference between a shorter numeric PIN compared to the longer alpha-numeric-special character-password they were used to. Participants also appreciated that the same PIN was used for authenticating to multiple accounts, alleviating the need for complex password management practices. The greatest drawback to using smartcards was the lack of authentication support for more applications. The few participants who had trouble integrating the smartcard into their work processes used the smartcard only to authenticate to their computers. These participants experienced more overhead for smartcard use with fewer benefits than the participants who used the smartcards to authenticate to multiple systems.

Overall, participants had a positive experience with the smartcards and most indicated they would continue using the smartcards voluntarily and would even recommend using smartcards to their colleagues. It was interesting to see that participants' perceptions of the value of smartcards were related to personal benefits gained, such as an alternative to managing passwords, over the institutional benefit from increased security of a multi-factor authentication method.

Our multi-method approach provided us a richer understanding of smartcard use that could not be attained through traditional laboratory testing. While each method contributed to the understanding of our participants' behavior and perceptions, our experience provided lessons to further improve the methods for future or similar work.

Fig. 1 .

 1 Fig. 1. From left to right: USB reader, laptop reader, integrated keyboard reader

Table 1 :

 1 Summary of study phases and data collection methods

	Installation		Smartcard Use		Wrap-up
	Pre-Install	Post-Install	Daily	2-Week	6-Week	Exit
	Survey	Site visit	Daily diary	Site visit	Site visit	Site visit
		Interview	Daily email	Interview	Interview	Interview
		Survey		Survey	Survey	Survey

Table 2 :

 2 Survey statement with mean values and 95% confidence intervals.

	Periodic Survey Statement
	Pre-Install	Post-Install	2-Week	6-Week	Exit
	1. User confidence with smartcard
	3.50 ± 0.38	4.42 ± 0.19	4.33 ± 0.30	4.12 ± 0.40	4.20 ± 0.34
	2. Remembering smartcard
	-	-	3.67 ± 0.47	4.04 ± 0.43	3.88 ± 0.41
	3. PIN is easier than password
	3.5 ± 0.29	3.92 ± 0.38	4.17 ± 0.28	4.24 ± 0.38	4.32 ± 0.39
	4. Smartcard easier than password
	3.40 ± 0.34	3.73 ± 0.39	4.08 ± 0.35	4.24 ± 0.31	4.24 ± 0.31
	5. Smartcard faster than password
	3.33 ± 0.33	3.50 ± 0.41	3.83 ± 0.40	3.68 ± 0.52	3.64 ± 0.49
	6. Smartcard more secure than password
	3.73 ± 0.34	-	-	3.60 ± 0.32	3.84 ± 0.29
	7. Smartcard adoption
	4.19 ± 0.19	4.35 ± 0.19	4.29 ± 0.38	-	4.48 ± 0.32
	8. Recommend smartcard to colleagues
	-	-	4.08 ± 0.33	4.04 ± 0.26	4.32 ± 0.31
	9. Smartcard satisfaction
	3.92 ± 0.26	-	-	-	4.32 ± 0.29

Table 3 :

 3 Summary of study observations by issue topic.

	Issue Topic	Observation
	User Confidence	• Confidence in using smartcards increased after Installation
	Smartcard Reader	• New object in environment to get used to • Reason for forgetting smartcard in reader
		• Form factor may matter
	Using Smartcards	• Smartcards easier for authentication than passwords
		• Forgot to remove smartcards from readers • Forgot to use smartcards to login
		• Forgot to use smartcards to lock screens • Forgot to use smartcards to unlock screens
		• Smartcard slower to login than password but faster otherwise • Unattended smartcard message is sometimes useful
	Password vs. PIN	• PINs easier to use than passwords • Password requirements were burdensome
		• Passwords became difficult to remember because of smartcard use • Various password management strategies
	Certificates	• Selecting certificate for web application authentication was confusing • Certificates could not be backed up or transferred
	Digital Signatures
	and Encryption

• Digital signatures and encryption were easy to use • Did not understand digital signatures and none would use them • Understood encryption but few would use it • Implementation does not support inter-institutional use Security Behavior • Smartcards gave perceived increase in security • Low understanding of how or why security works • PII users were the most security conscious Overall Experience • Overall positive experience with smartcards • Most would recommend smartcards to colleagues • Most would continue using it voluntarily • Some had problems fitting smartcards into work processes

Acknowledgements. We would like to thank Serge Egelman for his comments.


Anders Bruun
email: bruun@cs.aau.dk

Prof Jan Stage

Barefooted Usability Evaluation: Addressing the Mindset, Resources and Competences *

Keywords: Usability evaluation, Training, Software Developers, End Users

Lack of usability specialists, high resource requirements and developer mindsets are three considerable barriers for introducing usability engineering into software companies. This Ph.D. project explores the effect of letting software developers and end users apply usability engineering methods instead of a specialist, a solution which may reduce the barriers.

Introduction

During the last decade software companies have increased their focus on introducing usability engineering into their development processes. However, one challenge experienced by companies is the sheer lack of usability specialists in the industry, which leads to missing competences and ultimately problems of incorporating usability [START_REF] Nielsen | Finding usability problems through heuristic evaluation[END_REF]. However, small companies do not even have the privilege of staffing usability specialists as these have to cope with the constraint of low budgets. In practice this means that small software companies do not have the funds to pay for comprehensive consultancy or staffing of usability specialists [START_REF] Häkli | Introducing user-centered design in a small-size software development organization[END_REF] as they are expensive to hire [START_REF] Nielsen | Usability inspection methods[END_REF]. Another known problem, denoted the "developer mindset" [START_REF] Bak | Obstacles to usability evaluation in practice: a survey of software development organizations[END_REF], regards attitudinal aspects such as missing acceptance or low prioritization from the side of the developers. This is also a hindrance to future usability work within a company. Thus, small software companies wanting to focus on usability engineering face these three barriers: Missing competences, high resource demands and developer mindset.

If usability specialists are in scarce supply and small companies cannot afford to hire these, who, then, could undertake the usability work? This project explores the solution of letting the developers themselves and the end users apply usability engineering methods instead of a specialist, which in turn may reduce the three barriers.

Theoretical Foundation and Hypotheses

Before moving on, the following question should be answered: Why is it that, by letting developers and end users apply usability engineering methods, the barriers of missing competences, high resource demands and the developer mindset will be reduced? To answer this we may turn to Mao Zedongs cultural revolution in China in the 1960's. One vision behind this revolution was to bring better healthcare services to farmers living in rural areas. The problem at the time was the existence of an urban bias of doctors leading to scarce medical competences outside the cities. To counter this problem, Mao transferred medical knowledge from urban doctors to rural areas by letting farmers receive training in basic medicine such as the delivery of babies, how to ensure better sanitation etc. [START_REF] Daqing | China's barefoot doctor: Past, present, and future[END_REF]. Thus, some of the farmers would work part time in the rice fields walking around barefooted and part time as doctors in the local area. According to the World Health Organization these so called "barefoot doctors" reduced the amount of resources required, which was observed as reductions in healthcare budgets. These savings were caused by the preventive and tertiary care provided [START_REF] Daqing | China's barefoot doctor: Past, present, and future[END_REF]. An additional advantage was that rural farmers perceived the barefoot doctors as peers hereby respecting their advice [START_REF] Markle | Understanding Global Health[END_REF].

Although finding ourselves in a domain differing considerably from Chinese healthcare, we today face a similar challenge as before Mao's Cultural Revolution. Caused by missing competences, high resource requirements and developer mindset, small software companies, in a usability sense, have become the modern rural areas not being able to focus on usability work. This parallel challenge may be met by a parallel solution, i.e. by providing basic training in usability engineering methods, developers and end users could potentially act as barefooted usability personnel. This obviously could reduce the barrier of missing competences and could potentially ease the problem of scarce supply of specialists. Additionally, resource requirements may be brought down as usability engineering is preventive in avoiding expensive projects failures [START_REF] Bias | Cost-Justifying Usability: An update for the Internet Age[END_REF] and because the need to employ usability specialists is lessened. Also, other software developers may perceive barefooted usability personnel as peers, hereby accepting the advice given and reducing the barrier of developer mindset.

Thus, based on the notion of barefoot doctors the following hypotheses are derived:

• H1: Barefooted usability engineering reduces the barrier of missing competences.

• H2: Barefooted usability engineering reduces the barrier of resource requirements.

• H3: Barefooted usability engineering reduces the barrier of developer mindset.

Up until this point the term "usability engineering" has been applied, which covers several methods for analysis, design and evaluation activities. During my Ph.D. I do not have time to train developers and end users in all methods, which is why I focus on training these in evaluation methods, and more specifically in conducting user based usability evaluations (as opposed to inspection based). User based evaluations have proven effective in convincing developers of the importance of usability work as they provide first hand insights in difficulties experienced by end users [START_REF] Høegh | The Impact of Usability Reports and User Test Observations on Developers' Understanding of Usability Data: An Exploratory Study[END_REF] and may, thus, ease the introduction of usability engineering into software companies.

The following two research questions are addressed in the thesis:

• RQ1: What are the previous experiences and future research needs on training developers and end users in conducting usability evaluations?

• RQ2: How do usability evaluations conducted by developers and end users help in reducing the barriers of missing competences, high resource demands and developer mindset? How do developers and end users perform in comparison?

No previous studies have examined how usability evaluations conducted by developers and end users help in reducing all of the three barriers. A recent literature review identifies future research needs regarding training of software developers. The review shows that empirical studies of training developers in user based evaluation methods in general are needed [START_REF] Bruun | Training software developers in usability engineering: a literature review[END_REF].

Additionally, previous studies have shown that evaluations conducted by end users are possible, e.g. by applying remote asynchronous methods without the presence of an evaluator, see e.g. [START_REF] Castillo | Remote usability evaluation: Can users report their own critical incidents?[END_REF]. However, these studies focus on users' performance versus specialists', which leaves room for comparing user performance with that of developers.

Research Method

In this project my aim is to intervene by training developers and end users in applying usability evaluation methods and to study how these subjects apply methods in a natural setting. These characteristics call for the action case as a research method [START_REF] Braa | Interpretation, intervention, and reduction in the organizational laboratory: a framework for in-context information system research[END_REF]. To ensure data validity, multiple data collection methods will be employed:

• Observation: E.g. observations on correctness in conducting evaluations, amount of resources required (hours, money etc.) and how usability problems are prioritized and observations on product improvements.

• Surveys and interviews: For instance asking developers and other stakeholders how results from usability evaluations are used, how evaluations should be prioritized, what they perceive as barriers for doing more usability engineering etc.

• Document analysis: Analysis of usability reports with respect to no. of problems identified, quality of problem descriptions, reflections on the usability evaluation procedure etc.

Research Contribution

This project will contribute to the HCI research community by providing understanding of the barefooted theory as a driver for reducing the most significant barriers in incorporating usability engineering. This understanding emerges through reference studies in which the theory is applied to create changes in industrial practice. Thus, understanding and change are the types of outcomes this Ph.D. will bring to the research community and industrial practice respectively [START_REF] Braa | Interpretation, intervention, and reduction in the organizational laboratory: a framework for in-context information system research[END_REF].


Luca Longo
email: llongo@cs.tcd.ie

Human-Computer Interaction and Human Mental Workload: Assessing Cognitive Engagement in the World Wide Web

Keywords: Human-Computer Interaction, Human Mental Workload, User Engagement, Artificial Intelligence, Web-mining, Human Factors

Assessing the cognitive engagement of a user while seeking and consuming information over the World Wide Web is a key challenge for studying the quality of interactions. Indicators of cognitive engagement are useful for enhancing usability of interfaces, designing adaptable systems but also for analysing user behaviour and performance. For this purpose, we aim to adopt the multifaceted concept of Human Mental Workload, mainly applied in psychology and cognitive sciences, to study individual performance and user engagement in the context of Web. We aim to design a framework in which mental workload can be measured, analysed and explained. This will lead to analysis of individual and mass behaviour, guidelines and recommendation for interaction design, usability of front-end web applications and proposal of adaptive systems.

Introduction

The proliferation of web applications and the explosion of the Internet as an accessible data resource is rapidly changing the way users access, publish, seek and consume information, and knowledge working in general. Interacting with Internet technologies and surfing the WWW is a complex cognitive activity that involves visual, auditory and tactile human modalities rather than physical channels: in these activities human cognition is engaged at multiple stages. Analysing user behaviour while seeking and consuming information over the Web and designing user-centered web applications are becoming key topics in the last few years in the field of HCI. As a consequence, indicators of cognitive engagement are increasingly needed for studying the quality of the user experience while interacting with web resources.

Each action a person performs over the Web can be routinely monitored and stored in web-interaction logs. Existing methodologies for knowledge discovery from such sources are mainly focused on statistical and quantitative analysis. They mainly deal with data such as requested URLs, time properties, the provenance, the type of operating system and browser of a web-user. However, a more fine-grained set of web activities such as clicking, scrolling, keyboard usage, mouse movements, that better describe the user's physical interaction with web interfaces, can now be gathered using new Internet technologies.

Although logs turn out to be detailed sources of information, prior research, both academic and industrial, did not prove useful in interpreting logs from a cognitive perspective that links objective web-actions to cognitive theories of task execution. Reading and interpreting interaction logs, from this perspective, is a non-trivial problem: it is extremely difficult to identify users' original intention, goal, degree of attention and interest from these footprints.

Our proposal is to develop an executable framework, inspired by the concept of Human Mental Workload, large-scale applicable in the context of Web, to measure, assess, study and explain users' attention and cognitive engagement [START_REF] Kantowitz | Development of Nasa-TLX (Task Load Index): Results of Empirical and Theoretical Research[END_REF]. Models of human mental workload, in other fields, have been proven useful in the design of adaptive interfaces and in the study of individual performance. However, the explicit and invasive way of conducting experiments towards indexes of human mental workload, generally laboratory-driven, have not had a large-scale success.

Research Question

To what extent can the Human-Computer(Web) interaction be automatically captured and interpreted towards automatic assessments of Human Mental Workload? What are the factors that contribute to define human mental workload in the World Wide Web.

Approach

Our aim is to define a framework in which the concept of human mental workload can be measured, analysed and explained. The adopted approach is holistic and convergent, and it follows a general progression:

• analytical/empirical exploration of human mental workload characteristics;

• resolving and correlating these characteristics to reliable objective measures;

• definition of an extensible context-aware computational model for the Web;

• validation of the designed prototype with reliable subjective measures.

The research agenda includes the following stages:

1. implementation of a technique that allows non-invasive monitoring of webusers' activity without influencing their experience while consuming information over the World Wide Web; 2. implementation of a technology, large-scale applicable, capable of gathering a more fine-grained corpus of interaction fingerprints such as clicking, scrolling, mouse movements and keyboard usage, closer to the physical human-computer interaction; 3. identification of interaction patterns and development of human mental workload characteristics for the Web; 4. designing an extensible computational model for aggregating human mental workload characteristics; 5. validation and justification of generated indexes of human mental workload by comparison with existing psychology and cognitive measures [1, 2]; 6. application of generated indexes for designing engaging technologies and web applications.

The first 2 stages are aimed at developing a transparent non-invasive technology for gathering detailed human-computer interaction patterns while surfing the Web. The third aims to identify and objectively assess the factors that influence human mental workload in the context of Web. The fourth includes the design of a computational prototype, scenarios-aware, able to aggregate the factors individuated in 3 for a certain web-scenario. The fifth stage is aimed at validating the computational model via comparisons with subjective existing measures, successfully applied in psychology or cognitive science. The last gives practical means to deploying metrics, definitions and techniques for designing adaptable and engaging web applications. The research stages are not independent, rather they are related aspects that inform and influence the entire approach towards the measurement and definition of human mental workload The ongoing work is at stage 3 and 4. The monitoring technology has been deployed and the human mental workload characteristics along with the parallel definition of the computational model, are investigated.

Sketch of the solution

Our framework is composed by 5 functional-based macro-levels. The humancomputer interaction level underlines the process in which users interact with webinterfaces seeking and consuming information. Their interaction is monitored at the technology level: a designed technology aims to mine detailed signs of interaction such as mouse movements, keyboard usage and web-browser events over time. Here a raw interaction log is produced and subsequently consumed at the pre-processing level. Row data is pre-processed using machine learning techniques towards the identification of web-interaction sessions and tasks within each session. Each task is then recognised to be of a certain typology, thanks to an abstract high-level tasktaxonomy. This carries information about task demands in terms of cognitive resources and difficulty. At the computational heuristic level characteristics of human mental workload are individuated, computed and aggregated towards a unique index for a given task. Example of characteristics that may influence the mental workload are task interference, cognitive resources overlapping between multiple tasks, task concurrency, time sharing as suggested by the multiple resource theory of attention proposed by Wickens [START_REF] Wickens | Applied Attention Theory[END_REF][START_REF] Wickens | Multiple Resource and Performance Prediction[END_REF]. Individual skills and traits are important characteristics that undoubtedly moderate levels of workload as well as time and richness/control of online activity. The human mental workload characteristics relate to specific scenarios and how they can be calculated by adopting state-of-the-art metrics in psychology, IR or novel metrics. The last application level is represented by the practical use of indexes of human mental workload towards the design of user-centered technologies and user engagement aware web-applications.

Expected contribution of the research

The research will contribute to the field of Human-Computer Interaction within several areas, proposing a novel large-scale executable paradigm capable of human mental workload automatic assessment in the context of Web. The expected outcomes of the line of research are to produce an applicable framework able to lead to generalisable design recommendations and strategies along with examples and guidelines employable by other researchers or designers. Our contribution will provide reference and case studies for the application of a theory useful for promoting user-engagement aware applications and interactive design contributing to the appreciation and support of design practice in HCI.

Fig. 1 .

 1 Fig. 1. Anatomy of a framework based on Human Mental Workload assessments.