Romaric Marcilly
email: romaric.marcilly@univ-lille.fr

PhD Blake Lesselroth

MD, MBI Sandra Guerlinger

MPsy Annick Pigot

PhD Jessica Schiro

PhD Sylvia Pelayo

Active involvement of end-users in an EHR procurement process: a usability walkthrough feasibility case study

Keywords: electronic health records, method, procurement process, usability, patient safety

Introduction. Despite federal incentives promoting user-centered design processes during electronic health record (EHR) development, poor EHR usability remains a workflow and safety problem. Usability depends on alignment between EHR characteristics, user attributes, and context of use. Therefore, end-users should participate in EHR procurement to select a product that best fits business needs. Instead, end-users are often relegated to the organizational fringes during technology demonstrations and selection. We hypothesized that a usability walkthrough could enable end-users to evaluate EHR usability -even in resourceconstrained settings. This article assesses the feasibility of this method using a real-world case study of an EHR procurement. Method. As part of the hospital's transition to a new EHR, we organized a usability walkthrough for clinical staff. Nine representative end-users identified and categorized a range of usability issues. Our team's experts gathered user feedback and perceptions about the walkthrough using a mixed-methods approach. Finally, we tracked deviations from the planned protocol to identify implementation challenges and root causes.

Results. Participants detected 258 usability problems, and usability experts identified 7 additional usability problems. Both groups generally agreed on usability criteria and severity scores (Krippendorff's α = 0.66 and = 0.75, respectively). Thirty-two problems required clinical domain expertise to be identified. Participants indicated they liked the method and would use it for future technology purchases. Vendor preparation oversights caused protocol and timeline deviations; the EHRs were not always properly populated or configured for the walkthroughs. Discussion. The usability walkthrough is a feasible method to involve end-users in EHR evaluation during a procurement process. Engaging clinical domain experts is crucial to identify usability issues that might otherwise be missed.

INTRODUCTION

Background

Government usability standards -including those from the US Office of the National Coordinator (ONC) of Health Information Technology (HIT) -emphasize the importance of user-centered design to maximize electronic health record (EHR) usability. [START_REF]Standards, Implementation Specifications, and Certification Criteria for Electronic Health Record Technology[END_REF] Published studies [START_REF] Ratwani | A usability and safety analysis of electronic health records: a multi-center study[END_REF][START_REF] Ratwani | Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings[END_REF] and best-practice recommendations [START_REF]Taking Action against Clinician Burnout: A Systems Approach to Professional Well-Being[END_REF][START_REF]Health IT and Patient Safety: Building Safer Systems for Better Care[END_REF] show that improving user experience can promote technology adoption, increase system efficiency [START_REF] Linder | Barriers to electronic health record use during patient visits[END_REF][START_REF] Gans | Medical groups' adoption of electronic health records and information systems[END_REF] , improve patient safety [START_REF] Han | Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system[END_REF][START_REF] Wawrzyniak | EHR Usage Problems: A Preliminary Study[END_REF][START_REF] Koppel | Role of Computerized Physician Order Entry Systems in Facilitating Medication Errors[END_REF] , and reduce clinician burnout. [START_REF] Melnick | The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians[END_REF] Nevertheless, clinicians cite EHR design as one of the most common causes of clinical errors, long work hours, and dissatisfaction. [START_REF] Marcilly | Building Usability Knowledge for Health Information Technology: A Usability-Oriented Analysis of Incident Reports[END_REF][START_REF] Kaipio | Usability problems do not heal by themselves: National survey on physicians' experiences with EHRs in Finland[END_REF] Furthermore, not all vendors -including those headquartered in the US -integrate user-centered design principles into the product lifecycle. [START_REF] Ratwani | A usability and safety analysis of electronic health records: a multi-center study[END_REF][START_REF] Ratwani | Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors[END_REF] It is incumbent upon healthcare organizational leaders and clinical champions to be actively engaged throughout implementation from technology procurement, through system configuration, to eventual deployment and training.

How a clinician uses an EHR depends on the system's work context and socio-technical characteristics (i.e., technology, organizational climate, and user). [START_REF] Carayon | Work system design for patient safety: the SEIPS model[END_REF][START_REF] Sittig | A new sociotechnical model for studying health information technology in complex adaptive healthcare systems[END_REF][START_REF] Holahan | Beyond technology acceptance to effective technology use: a parsimonious and actionable model[END_REF] The specialty and practice settings are critically important to consider. For example, an emergency physician will not use the EHR like a geriatrician -each will have specific needs within their workflow.

It is, therefore, a best practice to include end-users in the EHR procurement process. [START_REF] Lobato | Investigation of Human Factors Engineering Methods Used in Medical Device Procurement Process[END_REF][START_REF] Cassano-Piché | Choosing Safer Medical Technologies: How Human Factors Methods Can Help in the Procurement Process[END_REF][START_REF] Seto | Usability Evaluations as Part of the Procurement Process: Case Study of Hospital Point of Care Carts[END_REF][START_REF] Garmer | User participation in requirements elicitation comparing focus group interviews and usability tests for eliciting usability requirements for medical equipment: a case study[END_REF] The methods organizational leaders use to engage end-users can influence the selection process outcome. [START_REF] Kannry | Using an evidence-based approach for system selection at a large academic medical center: lessons learned in selecting an ambulatory EMR at Mount Sinai Hospital[END_REF][START_REF] Priestman | Lessons learned from a comprehensive electronic patient record procurement process-implications for healthcare organisations[END_REF][START_REF] Mcdowell | Herding cats: the challenges of EMR vendor selection[END_REF][START_REF] Gray | Computerized prescriber order-entry systems: evaluation, selection, and implementation[END_REF] While executives often only invite end-users to EHR demonstrations, [START_REF] Mcdowell | Herding cats: the challenges of EMR vendor selection[END_REF] there are practical ways for users to evaluate products during demonstrations. Project managers can administer the Usability Questionnaire for Demonstrations in Procurement (DPUQ) [START_REF] Tyllinen | End-Users' Voice in EHR Selection: Development of a Usability Questionnaire for Demonstrations in Procurement (DPUQ)[END_REF] to gather end-user perceptions. Usability experts can then supplement this with a heuristic evaluation during the demonstration (HED). [START_REF] Tyllinen | We Need Numbers!: Heuristic Evaluation during Demonstrations (HED) for Measuring Usability in IT System Procurement[END_REF] The HED is a low-cost method where an expert rates a product against a set of usability heuristics. [START_REF] Nielsen | Usability Engineering[END_REF][START_REF] Scapin | Ergonomic criteria for evaluating the ergonomic quality of interactive systems[END_REF][START_REF] Zhang | Using usability heuristics to evaluate patient safety of medical devices[END_REF] These methods notwithstanding, data gathered during vendor demonstrations may not be the best quality since users rarely interact with the EHR, and sales representatives can hide product weaknesses. [START_REF] Kushniruk | Increasing the safety of healthcare information systems through improved procurement: toward a framework for selection of safe healthcare systems[END_REF] Prospective customers can gather more predictive data by conducting usability tests with clinical information processing scenarios (CLIPS). CLIPS are scripts representing clinical situations with tasks for users to complete while experts observe and collect data. [START_REF] Priestman | Lessons learned from a comprehensive electronic patient record procurement process-implications for healthcare organisations[END_REF][START_REF] Kushniruk | Increasing the safety of healthcare information systems through improved procurement: toward a framework for selection of safe healthcare systems[END_REF][START_REF] Kushniruk | Selecting electronic health record systems: development of a framework for testing candidate systems[END_REF] Stakeholders, however, may be apprehensive about the time and cost of simulation testing of multiple EHRs with CLIPS. [START_REF] Carvalho | Ensuring the Safety of Health Information Systems: Using Heuristics for Patient Safety[END_REF][START_REF] Schumacher | How to Select an Elctronic Health Record System That Healthcare Professionals Can Use[END_REF] Heuristic evaluations are typically more cost-effective but do not involve end-users and can miss important issues. [START_REF] Kannry | Using an evidence-based approach for system selection at a large academic medical center: lessons learned in selecting an ambulatory EMR at Mount Sinai Hospital[END_REF][START_REF] Priestman | Lessons learned from a comprehensive electronic patient record procurement process-implications for healthcare organisations[END_REF][START_REF] Nielsen | Usability Engineering[END_REF] We see a need to combine these methods into a single protocol.

Problem Statement and Research Questions

Schumacher and colleagues described a pragmatic approach to involve end-users in the procurement process: the usability walkthrough. [START_REF] Schumacher | How to Select an Elctronic Health Record System That Healthcare Professionals Can Use[END_REF] During a usability walkthrough, end-users complete CLIPS with the EHR and classify usability problems using a set of heuristics. After the walkthrough, participants rate effectiveness, efficiency, and satisfaction.

A usability walkthrough is like simulation testing in that it involves end-users. In user simulation testing, experts identify problems while watching users test the technology, whereas, in a usability walkthrough, users identify the problems. [START_REF] Nielsen | Usability Engineering[END_REF][START_REF] Schumacher | How to Select an Elctronic Health Record System That Healthcare Professionals Can Use[END_REF][START_REF] Jaspers | A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence[END_REF] The usability walkthrough is also like a cognitive walkthrough -both methods require users to think and talk through a clinical scenario. However, the cognitive walkthrough evaluates product learnability by asking standardized questions about interface intuitiveness and the ability to guide users through tasks. [START_REF] Jaspers | A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence[END_REF][START_REF] Lewis | Cognitive Walkthroughs. In: Handbook of Human-Computer Interaction[END_REF] The usability walkthrough, by contrast, measures multiple usability dimensions by replacing standardized questions with heuristics. To conduct a usability walkthrough correctly, usability professionals train the end-users to apply the same heuristics experts use.

To summarize, usability walkthroughs permit users to compare EHRs without vendor interference and allow clinicians to champion their needs and preferences. [START_REF] Schumacher | How to Select an Elctronic Health Record System That Healthcare Professionals Can Use[END_REF] Yet, no studies have formally investigated the feasibility of conducting a usability walkthrough during an EHR procurement. We sought to close this knowledge gap by asking four questions: (1) Are endusers able to detect, describe, and prioritize usability problems? (2) Does the usability walkthrough method help identify problems only detectable by clinical experts? (3) How satisfied are end-users with the usability walkthrough process? (4) What are the challenges of implementing a usability walkthrough during EHR procurement? To answer these questions, we conducted an implementation study of a usability walkthrough as a hospital transitioned to a new commercial EHR. In this article, we report on the method's feasibility.

The results of the EHR evaluation are published elsewhere. [START_REF] Marcilly | Competitive Usability Evaluation of Electronic Health Records: Preliminary Results of a Case Study[END_REF]

METHODS

Study Context

Leadership at a private, non-profit, 1,000-bed teaching hospital in Lille, France, issued a request for proposals (RFP) and organized a procurement process to select a replacement for their current commercial EHR. The process included three steps: (1) a vendor demonstration;

(2) a usability walkthrough with each candidate EHR; and (3) technical and economic comparisons of EHRs selected during the walkthrough. The comparisons focused on "backend" functionality (e.g., data interoperability). In this manuscript, we describe the second step of the process (i.e., usability walkthrough).

To conserve resources and adhere to a timeline, it was necessary to quickly thin the pool of EHR candidates for later technical evaluation. We did not set out to exhaustively safety test products or generate summative statistics during the second phase. Therefore, we limited the number of users recruited.

Usability Workshop Implementation

The project manager (AP) and four usability experts (RM, SG, JS, SP) designed a usability instructional session and a usability workshop that included structured exercises and evaluation instruments. We planned to conduct two instructional sessions and five workshops over three weeks from September to October 2020 (Figure 1).

Usability Instructional Sessions

We hosted two face-to-face instructional sessions. We held the first session two weeks before the workshops and a refresher the day of the first workshop. In both sessions, we explained the dimensions of usability, demonstrated the walkthrough method, reviewed usability assessment criteria (adapted from Scapin and Bastien) [START_REF] Scapin | Ergonomic criteria for evaluating the ergonomic quality of interactive systems[END_REF] , and introduced a usability issue severity rating scale. We answered questions and furnished the participants with a written summary of all content.

Workshop Design

Preparation and participants

We received proposals from five vendors and scheduled five 3-hour workshops over one week: one per candidate (Figure 1). During each workshop, the vendor presented their EHR to stakeholders. We then excused vendors from the proceedings; they were not permitted to interact with end-users during any other evaluation stage -including the usability walkthrough and end-user debriefing session (outlined below).

A multidisciplinary team of clinical representatives, the procurement manager, and a usability expert (JS) identified common inpatient EHR use scenarios and concerned end-users.

We designed seven CLIPS simulating 59 EHR tasks (Appendix A), targeting frequently used or critical functionalities (Table 1). We also created a clinical dataset for the EHR. We sent the CLIPS and dataset to the EHR vendors two weeks before the usability sessions.

We recruited nine end-users: three physicians (an emergency physician from the emergency unit, a cardiologist from the cardiology unit, and a neurologist from the geriatrics unit), three nurses (from emergency, cardiology and geriatrics units), one pharmacist (from the central pharmacy), one medical clerk, and one admission officer (i.e., non-clinical staff member trained to manage administrative and logistic duties). Participants volunteered for the workshops; they were not compensated for their participation. None had been trained to use the candidate EHRs.

Usability walkthrough

We organized participants into four evaluation groups -each supervised by a usability expert.

Each group completed CLIPS at a computer workstation. Three groups included a physician and a nurse, whereas the fourth included the pharmacist, admission officer, and clerk. The usability experts facilitated the walkthrough, tracked time, and gathered field observations. We first gave each group written CLIPS and a patient summary. We then instructed groups to use the EHR to complete tasks, describe issues encountered, and assign each issue a usability criterion (i.e., "guidance," "workload," "compatibility," "significance of codes," "adaptability," "error management," "consistency," or "explicit control"). Participants also assigned a severity level (i.e., "light," "minor," or "major") to each issue. Please refer to Appendices B and C for criterion and severity definitions. [START_REF] Scapin | Ergonomic criteria for evaluating the ergonomic quality of interactive systems[END_REF] We audio-recorded comments.

The usability experts documented direct quotes during the sessions, issues reported by endusers, and criteria and severity scores. After each task, end-users completed a 4-item questionnaire adapted from Schumacher et al. with 5-point Likert-type items evaluating EHR features availability, completeness, ease of use, and efficiency. [START_REF] Schumacher | How to Select an Elctronic Health Record System That Healthcare Professionals Can Use[END_REF] We published our questionnaire findings in a companion article. [START_REF] Marcilly | Competitive Usability Evaluation of Electronic Health Records: Preliminary Results of a Case Study[END_REF] The walkthrough ended once participants completed all CLIPS or after two hours had elapsed. Afterward, users completed the System Usability Scale (SUS) [START_REF] Brooke | SUS: a "quick and dirty" usability scale[END_REF] -a validated 10-item questionnaire with Likert-type statements and performance benchmarks. We published the results of this questionnaire in a companion article. [START_REF] Marcilly | Competitive Usability Evaluation of Electronic Health Records: Preliminary Results of a Case Study[END_REF] We organized end-users into groups according to professional roles. A usability expert then debriefed each group using a semistructured interview script exploring EHR strengths and weaknesses (Appendix D).

Data collection and analysis

Question 1: Can end-users detect, describe, and prioritize usability problems?

Our usability experts first read problems identified by the participants and excluded (1) those unrelated to specific EHR characteristics (e.g., opinions without descriptions), (2) those concerned with the technology platform (e.g., connection failures), or (3) those rooted in data upload problems. They then combined multiple descriptions of the same problem (i.e., deduplication) to reach a final list of usability problems.

Next, we created a usability expert "comparison set". We combined lists of problems identified by participants and problems detected by usability experts. For each problem, we assigned a usability criterion and a severity level. Experts independently categorized problems using our a priori usability criteria and severity levels. [START_REF] Scapin | Ergonomic criteria for evaluating the ergonomic quality of interactive systems[END_REF] Disagreements were resolved through consensus. [START_REF] Nielsen | Usability Engineering[END_REF] We then compared end-users' lists and assignments to the "comparison set".

We calculated concordance between end-users and experts using percent agreement and Krippendorf's α. [START_REF] Krippendorff | Content Analysis: An Introduction to Its Methodology[END_REF] We also calculated the average issue detection rate within user profiles when there were multiple representatives (i.e., nurses and physicians).

Question 2: Does the usability walkthrough method identify problems that require clinical domain expertise to be detected?

Two usability experts screened problem descriptions to identify those requiring clinical expertise to detect. Since these represented new types of problems only clinicians recognized, our usability experts categorized each problem inductively.

Question 3: How satisfied are end-users that participate in a usability walkthrough?

After the last walkthrough, we asked participants to provide feedback on the method. To measure user satisfaction, we developed an eight-item questionnaire with 5-point Likert scales anchored by strongly disagree and strongly agree (Table 5).

Participants also answered open-ended questions about the strengths and weaknesses of the method. We compared each rating to 3 (i.e., the median value) using the Wilcoxon sample signed-rank test with a significance threshold of 0.05. Two usability experts analyzed the qualitative data inductively to identify important or recurrent themes.

Question 4: What are the challenges associated with implementing a usability walkthrough during an EHR procurement process?

We recorded all deviations from the workshop agenda and evaluation protocol. Two usability experts categorized each deviation according to the root cause.

RESULTS

In this section, we report on the feasibility and utility of our usability evaluation strategy. We reported EHR usability findings in a companion article. [START_REF] Marcilly | Competitive Usability Evaluation of Electronic Health Records: Preliminary Results of a Case Study[END_REF] Two vendors withdrew their applications during the procurement process, leaving only three candidate EHRs for analysis.

Are end-users able to detect, describe, and prioritize usability problems?

Participants reported 361 usability problems. We excluded 21 issues (5.82%) using our eligibility criteria. After deduplication, the final list consisted of 265 problems: 258 detected by end-users (97.36%) and 7 detected only by usability experts (2.64%) (Table 2). Each enduser within a professional role detected between 26.82% to 70.37% of all problems identified by that group (mean = 42.92%; SD = 14.11). On average, 59.83% of the problems were detected by one participant, 23.73% by two, and 12.75% by three.

End-users assigned a criterion to 218 of the 258 problems (84.49%); 157 matched those assigned by usability experts (72%; Krippendorff's α = 0.66 [0.59;0.74], Table 3). The largest discrepancies were in three categories: "guidance", "workload", and "compatibility".

Participants assigned a severity level to 217 of the 258 problems identified (84.10%): 165 matched those assigned by experts (76%; Krippendorff's α = 0.75 [0.68 ;0.82], Table 4). Endusers more often rated problems as "light" or "major," whereas usability experts more often rated problems as "minor."

Does the usability walkthrough method identify problems that require clinical domain expertise to be detected?

Thirty-two of the 258 problems (12%) required clinical expertise to detect (Appendix E). In one instance, a pharmacist using the medication review module could not determine how to make a drug substitution, and in another, a physician did not receive an error message after entering the same treatment order twice. We include a complete list of issues in Appendix E.

We classified these problems into 5 novel categories: patient identifiers (n=4), information availability and visibility (n=12), EHR configurations (n=9), access to EHR functions by users' roles (n=2) and work and cognitive load problems (n=5).

There were instances when patient identifiers were not visible while performing tasks, increasing the risk of wrong patient selection errors. End-users encountered screens missing critical information, including orders, decision support, care plan actions, and medication changes. There were readability issues associated with typography and inappropriate "hard stops" (e.g., health insurance information queries blocking access to other functions). We found a mismatch between user EHR permissions and real-world scope of practice (e.g., nurses were granted access to prescribe when not permitted in clinical practice). Finally, the EHRs tended to increase workload and cognitive load. For example, some features increased the number of actions per task, failed to provide the user with feedback, or made selections confusing (e.g., a mismatch between medication type and dosing units).

How satisfied are end-users with the usability walkthrough?

Overall, end-users valued the usability walkthrough. The average score for each questionnaire item was at least four on a five-point scale (Table 5). Participants, however, struggled to assign usability criteria (n= 4 of 9), and all reported some challenges learning the scoring and categorization system. One respondent said, "…understanding the criteria takes time," and another would have appreciated "even more training upstream of the evaluations."

Nevertheless, most respondents (n= 8 of 9) said the method was "easy to learn." Most participants liked the ability to quantify subjective impressions of the technology and indicated they would be willing to use this method again (n=8 of 9). They all agreed that the method "clearly distinguished [EHRs] strengths and weaknesses," and the data permitted a "detailed comparison of EHRs."

What challenges exist when implementing a usability walkthrough during an EHR procurement process?

Two vendors withdrew their applications. One had concerns about their product's usability; the other did not disclose a reason. The remaining vendors were not prepared for the evaluation. We could not complete all CLIPS due to technical issues (e.g., features not working) and database "locks" that prevented multiple users from opening the same test record simultaneously (Table 6). In some cases, the EHRs were missing patient data, whereas, in others, the data that participants were expected to enter were already in the chart.

DISCUSSION

Principal Findings

Initial selection or transition to a new EHR can have seismic consequences on health system outcomes, patient safety, and clinician well-being. [START_REF] Alami | Rethinking the electronic health record through the quadruple aim: time to align its value with the health system[END_REF] Choosing one that does not meet organizational needs can destabilize the work system and undermine patient safety. To our knowledge, this is the first study to evaluate the feasibility of a usability walkthrough involving end-users in EHR procurement. After a short training session, clinicians could identify, and risk stratify real EHR usability problems. The procurement team used our results to guide selection decisions and screen candidate EHRs for the third phase of the evaluation.

Our findings match studies of usability testing showing that testing products with as few as 3-5 users can identify up to 85% of interface problems affecting similar users. [START_REF] Sauro | Quantifying the User Experience: Practical Statistics for User Research[END_REF] In our study, end-users within the same profile did not detect all the same problems. It is, therefore, crucial to recruit at least 3-5 evaluators per profile. [START_REF] Nielsen | Heuristic evaluation of user interfaces[END_REF] While the assignment of criteria and scores by end-users and usability experts were consistent, end-users sometimes struggled to disambiguate "guidance", "workload", and "compatibility" criteria. [START_REF] Følstad | The usability inspection performance of work-domain experts: An empirical study[END_REF] Clinicians also tended to assign higher severity scores compared to usability experts. We hypothesize that the consensus method our experts used when assigning severity levels had a moderating effect on scores. Overall, our findings suggest that the usability walkthrough method -and the inclusion of end-users -is valid for identifying usability issues, is generalizable across settings, and may be extensible to other technologies.

End-users identified 32 usability problems that required clinical expertise to detect. This is consistent with studies showing that end-users with domain expertise identify problems that usability specialists may overlook. [START_REF] Fu | Who Finds what in Usability Evaluation[END_REF] Some of the problems identified by clinicians could have had severe patient safety consequences. Patient identifier issues are known to increase the risk of wrong-patient prescribing [START_REF] Sopan | Reducing wrong patient selection errors: exploring the design space of user interface techniques[END_REF] , missing patient data can affect medical decision-making, and inappropriate "hard stops" can delay patient care. [START_REF] Han | Unexpected increased mortality after implementation of a commercially sold computerized physician order entry system[END_REF][START_REF] Graber | The impact of electronic health records on diagnosis[END_REF] Therefore, our findings reinforce the axiom that end-users must be engaged throughout the HIT lifecycle -including the procurement process.

Feedback on Methods and Implications for Practice

Participants said they would use the walkthrough and scoring system for similar projects despite the learning curve. The information helped them objectively evaluate features and compare products systematically, and argue preferences and advocate for their constituency.

Data collected without vendor interference may generate more reliable information and more safety concerns than traditional product demonstrations, enabling business leaders to base decisions on technology elements or objective measures of human-computer interaction. [START_REF] Kushniruk | Increasing the safety of healthcare information systems through improved procurement: toward a framework for selection of safe healthcare systems[END_REF]

Lessons Learned and Limitations

While usability walkthrough methods do not require extensive training, there are opportunities to improve orientation and testing activities. [START_REF] Sears | Heuristic Walkthroughs: Finding the Problems Without the Noise[END_REF] Sending the CLIPS and sample patient data to the vendors could have given them an unfair advantage and influenced how they configured their EHRs. In practice, however, the vendors were unprepared for testing.

We could not complete several test scripts because vendors did not include the minimum data required for each simulation. This may have biased our selection decisions. Before implementing the usability walkthrough, usability experts should complete a "cross-check" of the EHR setup. We hypothesize that France's lack of incentives for usability work and evaluation could explain vendors' lack of awareness surrounding prerequisites for usability evaluation (e.g., missing mock patient data in a test patient record). Usability requirements included in the ONC certification program and other federal research funding programs may have incentivized vendors to collaborate more effectively with US healthcare organizations and create better test environments. [START_REF] Hettinger | Advancing electronic health record vendor usability maturity: Progress and next steps[END_REF] Readers should interpret the findings from our case study with caution. Our results are based on the behaviors and observations of a small sample of users recruited at a French hospital during an EHR procurement. The findings could differ across users, settings, products, or time points. Nevertheless, this case study shows how to gather usability data and stakeholder sentiments at the pace of healthcare operations.

Usability research adds up-front costs for the customer. [START_REF] Sauro | How Much Does a Usability Test Cost? Measuring U[END_REF] The main costs include time spent creating test scenarios, recruiting participants, conducting usability evaluation, and retaining usability experts to proctor sessions, facilitate debriefs and analyze the results. [START_REF] Nielsen | Return on Investment (ROI) Ofr Usability[END_REF] Executives must be convinced that method generates a return on investment if they are to budget evaluation resources during procurement. Cost of testing must be weighed against the downstream costs of a poorly designed EHR related to training, workflow, efficiency losses, clinician burnout, staff disenfranchisement, patient endangerment, and legal actions. [START_REF] Cassano-Piché | Choosing Safer Medical Technologies: How Human Factors Methods Can Help in the Procurement Process[END_REF]

CONCLUSIONS

The usability walkthrough is a feasible method to engage end-users in usability evaluation, compare EHRs during a procurement process, and galvanize buy-in for change among stakeholders. Nevertheless, efforts are needed to raise usability awareness and incentivize vendors to optimize product performance. Without strong federal policies or technical regulations, the procurement process represents an important lever for change.

ETHICS APPROVAL, PERMISSIONS, SUBJECT PROTECTIONS

According to French law, studies in human sciences in the field of health do not require approval by an ethics committee (decree n°2017-884, article 2). Nevertheless, this study was performed in accordance with the 1964 Helsinki declaration and later amendments. Written informed consent was obtained from all study participants.

Legends for the figure

Evaluation process.

TABLES Table 1.

List of end-user profiles and summary of end-user tasks during clinical information processing scenarios (CLIPS). When prescribing a medication, the list of units is long and includes units that are not relevant to the selected medication. (EHR1) During the medication review, pharmacists do not have the possibility to refuse a prescription line easily: they can easily accept or refuse the whole prescription. But it makes them perform several actions to question and make a proposal for a single prescription line which adds an additional workload. (EHR1) After the prescription entry, the physician does not know if the prescription is sent and seen by the nurses. (EHR1) Blood culture is listed twice in the prescription list: one corresponds to care, the other to a blood test. Depending on the choice, the information given to the nurse is very different. The doctor does not see the difference between these two options in the list. (EHR2) On the conclusion window, the "save" and "validate" buttons are semantically ambiguous and do not have the same consequences (possibility or not to change the entered conclusion). However, when the conclusion is written by an intern, it must be reread by a senior before being definitively validated. (EHR3)

Profile Summary of CLIPS and tasks Pharmacist (n = 1 )

 1 Validation of a prescription, entry of medication dispensation Admissions officer (n = 1) Entry of new patient's administrative data, of administrative discharge, of consultation and day hospital pre-admissions Secretary (n = 1) Dictation transcription, search and entry of appointments bookings for consultation and hospitalization Nurses (n = 3) Patient's arrival in the emergency room: entry of reason for admission, vital parameters, patient prioritization, medication administration, patient orientation, transfer to another service. Patient hospitalization: entry of the bed installation, of the entry synthesis, of the initial clinical synthesis, of the autonomy assessment, of the Braden score, of the blood test, of the care planning, of the therapeutic administration, of the care delivery, of the traceability of a bandage, of the urinary catheter insertion, of the department exit, visualization of the care plan, correction of an error, update of a targeted transmission Physicians (n =3)Patient's arrival in the emergency room: entry of medical observation, prescription, medical decision, and coding Patient hospitalization: entry of medical history and disease, of the usual treatment, of allergies, of the initial prescription, of the progress note, of the discharge order and coding, visualization of laboratory results, of the care plan, of the prescriptions, of the medical observations and of the vital parameters, change of prescription, request for infectious advice, dictation of discharge letter.

Table 2 .

 2 Frequency of problems detected per user profile, and number of end-users identifying the problem ("0" indicates only the usability expert identified a problem).

	User profile	Number of	Number of problems detected (% per profile where	Sum
		end-users	relevant)
		detecting the	EHR1	EHR 2	EHR3
		problem
	Physician	0 (UE)	1 (2.3%)	1 (2.4%)	1 (6.25%)	3
		1	31 (72%)	25 (61%)	8 (50%)	64
		2	8 (18.6%)	11 (26.8%)	4 (25%)	23
		3	3 (7%)	4 (9.7%)	3 (18.75%)	10
	Nurse	0 (UE)	1 (3.8%)	0	2 (6.9%)	3
		1	15 (53.8%)	14 (63.6%)	17 (58.6%)	46
		2	8 (30.8%)	3 (13.6%)	8 (27.6%)	19
		3	3 (11.5)	5 (22.7%)	2 (6.9%)	10
	Pharmacist	0 (UE)	0	0	0	0
		1	14 (100%)	23 (100%)	9 (100%)	46
	Medical clerk	0 (UE)	0	0	0	0
		1	10 (100%)	9 (100%)	6 (100%)	25
	Admission officer	0 (UE)	1 (20%)	0	0	1
		1	4 (80%)	5(100%)	6 (100%)	15
	Sum		99	100	66	265

Footnote. Acronyms: UE -Usability expert; EHR -Electronic health record.

Table 3 .

 3 Number and concordance of usability problems per criterion according to end-users and usability experts. The diagonal (light grey) indicates agreement between raters. This table does not include emergent issues and categories requiring clinical expertise to identify (see Appendix E for this data).

	FIGURES
	Figure 1.

ACKNOWLEDGMENTS

The authors thank the participants along with Lille Catholic Hospital's EHR procurement process management group. This case study was funded by the Group of hospitals of the Catholic Institute of Lille. Some usability problems identified during the reported procurement process were published in a conference proceeding. 38

CONFLICTS OF INTEREST DISCLOSURES

Four authors report that their laboratory received a grant to evaluate EHRs' usability.

End-users

Usability criterion Definition Guidance

Refers to the means available to advise, orient, inform, instruct, and guide the users throughout their interactions with a computer (messages, alarms, labels, etc.), including from a lexical point of view Workload Concerns all interface elements that play a role in the reduction of the users' perceptual or cognitive load, and in the increase of the dialogue efficiency Explicit control Refers to the system processing of explicit user actions, and to the control users have on the processing of their actions by the system Adaptability Refers to system's capacity to behave contextually and according to the users' needs and preferences Error management

Refers to the means available to prevent or reduce errors and to recover from them when they occur. Errors are defined in this context as invalid data entry, invalid format for data entry, incorrect command syntax, etc. Consistency Refers to the way interface design choices (codes, naming, formats, procedures, etc.) are maintained in similar contexts, and are different when applied to different contexts Significance of codes Qualifies the relationship between a term and/or a sign and its reference. Codes and names are significant to the users when there is a strong semantic relationship between such codes and the items or actions they refer to Compatibility Refers to the match between users' characteristics (memory, perceptions, customs, skills, age, expectations, etc.) and task characteristics on the one hand, and the organization of the output, input, and dialogue for a given application, on the other hand APPENDIX C. Usability issue severity levels and descriptions.

Severity level Definition Light

At worst, the problem causes mild and easily overcome inconvenience Moderate

The problem slows down or hinders task performance Severe

The problem prevents task performance and may cause the user to make mistakes Appendix D. Semi-structured interview script for the post-session debrief.

1. For the difficulties/hesitations encountered but not commented by the participants but noted by the usability experts: we noticed that you had difficulties to perform On the care plan, the samples to be taken are not visible, which means that there is a risk of not seeing them and not taking them (EHR3).

The pharmacist does not notice very small notifications when there is a change in a prescription, which prevents them from identifying prescriptions that need to be analyzed (EHR1)

The typography of the prescriptions is misleading: 1D sachet is easily confused with 10 sachets. (EHR 1)

The care plan is not legible and too small, there is a strong risk that nurses will miss information. (EHR1)

In the medication review module, pharmacists cannot find out how to make a drug substitution. Instead, they only find out how to make a proposal without a dose. (EHR1) Some prescriptions are not visible to the physician (e.g., blood culture prescription), which can cause represcribing and prescription duplication problems. (EHR 2)

The pharmacist cannot enter a proposal for pharmaceutical intervention (EHR2) After prescribing a 1L infusion, a 500mL

The pharmacist cannot substitute medications