Emeline Brulé

Brianna J Tomlinson

Oussama Metatla

Christophe Jouffrais
email: christophe.jouffrais@irit.fr

Marcos Serrano
email: marcos.serrano@irit.fr

Review of Quantitative Empirical Evaluations of Technology for People with Visual Impairments

Keywords: Assistive Technology, Visual Impairments, Literature Review, Evaluation Methods, Experiments, Education CCS Concepts, Human-centered computing → Usability testing, Accessibility systems and tools, Laboratory experiments, Empirical studies in HCI, •Social

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Research in Human-Computer Interaction (HCI) on assistive technologies for people with visual impairments (PVI) 1 is an 1 Participants can be referred to using person-first or disability-first language. There are many debates regarding which one is best practice [START_REF] Morton | Editorial Perspective: The use of person-first language in scholarly writing may accentuate stigma[END_REF][START_REF] Cavender | Accessible writing guide[END_REF]. We use people with visual impairments which was preferred by a majority of authors and PVI as an abbreviation.

CHI '20, April 25-30, 2020, Honolulu, HI, USA ACM ISBN XXX. DOI: https://doi.org/10.1145/XXXXX ever growing domain with the general goal of improving accessibility as a means for addressing broader societal issues of equality, diversity and inclusion. Research on technologies for PVI spans a variety of application areas, such as education, entertainment and mobility [START_REF] Bhowmick | An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends[END_REF], and investigates the potential of novel interaction techniques, including tactile display (haptics and vibratory feedback) [START_REF] Tekli | Evaluating Touch-screen Vibration Modality for Blind Users to Access Simple Shapes and Graphics[END_REF], auditory display (verbal and non-verbal feedback) [START_REF] Bardot | From Tactile to Virtual: Using a Smartwatch to Improve Spatial Map Exploration for Visually Impaired Users[END_REF][START_REF] Metatla | Inclusive Education Technologies: Emerging Opportunities for People with Visual Impairments[END_REF][START_REF] Tomlinson | Exploring auditory graphing software in the classroom: The effect of auditory graphs on the classroom environment[END_REF], tangible interaction [START_REF] Ducasse | Tangible Reels: Construction and Exploration of Tangible Maps by Visually Impaired Users[END_REF][START_REF] Brulé | Representing children living with visual impairments in the design process: A case study with personae[END_REF], and sensory substitution [START_REF] Maidenbaum | Sensory substitution: closing the gap between basic research and widespread practical visual rehabilitation[END_REF][START_REF] Gori | Devices for visually impaired people: High technological devices with low user acceptance and no adaptability for children[END_REF]. However, designing and conducting adequate quantitative empirical remains a challenge, an issue also noted in the field of education sciences [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF]. For instance, a literature review by Kelly and Smith [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF] found that less than 10% of educational technologies for visually impaired children underwent some form of quantitative empirical evaluation. A 2018 workshop on inclusive educational technologies for PVI [START_REF] Metatla | Toward Classroom Experiences Inclusive of Students with Disabilities[END_REF] identified the lack of overview and guidelines on evaluation practices as a barrier for developing novel technologies. To address this challenge, it is important for the HCI community to understand existing practices. Yet, to date, no survey addresses the evaluation of technologies and interaction techniques for PVI in HCI.

Quantitative empirical evaluation methodologies are popular, if not considered a gold standard, in the field of HCI [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF][START_REF] Lam | Seven guiding scenarios for information visualization evaluation[END_REF]. However, these are usually difficult to implement when working on technologies for PVI. Common challenges include heterogeneous participant profiles, limited availability of participants, and deciding how and when to include sighted participants in evaluations [START_REF] Bardot | Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision[END_REF]. Hence, we address the following research question: What is, at present, best practice in quantitative empirical evaluations of technologies for people with visual impairments in HCI? Providing answers to this question will not only guide researchers in this area to devise and carry out appropriate quantitative evaluations, but will also allow for a more informed and uniform approach to reviewing them in scientific publications.

To answer this question, we developed three secondary research questions: RQ1: Are there common standards for conducting quantitative empirical evaluations of technologies for PVI; if so, what are they?; RQ2: Overall, are participants in these evaluations representative of the target population?; and RQ3: What are the difficulties encountered by researchers in this type of evaluation?

We focused on papers reporting practices of quantitative evaluations in top-tier venues (i.e., CHI, ASSETS, TOCHI, and TACCESS). Our final corpus includes 178 papers published since 1988, each involving one or more quantitative empirical evaluations (243 experiments in total). We tagged these papers before analysing: [START_REF]Human Computer Interaction -Google Scholar Metrics[END_REF] the design of these evaluations and how they are used; [START_REF] Iyad | Making Microsoft Excelâ Ďć: multimodal presentation of charts[END_REF] the representativeness of the participants involved in the evaluations in this corpus; and (3) the difficulties reported by authors.

Overall, we found a large variety of accepted practices for evaluation, which could lessen integration of findings over time and interdisciplinary application. We highlight concerns regarding the quality and details in reporting evaluations and the representativeness of the participant sample. We propose a set of recommendations for addressing these concerns, including using shared standards for describing participants with visual impairment and borrowing other quantitative approaches to evaluation from connected fields such as rehabilitation and special education research.

BACKGROUND

Surveying evaluation approaches supports the establishment of best practices in research, helps maintain consistency in the review processes, and builds a strong and coherent body of scholarship [START_REF] Lam | Seven guiding scenarios for information visualization evaluation[END_REF]. However, to date, there are no surveys focused on quantitative empirical evaluations of technologies and interaction techniques for PVI, despite the large dominance of this type of evaluations in HCI venues such as the field's flagship conference, ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF]. This is especially crucial as previous work has long reported challenges specific to evaluation with this user group (e.g., [START_REF] Robert | An approach to the evaluation of assistive technology[END_REF][START_REF] Friedlander | Bullseye! when Fitts' law doesn't fit[END_REF][START_REF] Ticianne Gr Darin | Investigating the Mode in Multimodal Video Games: Usability Issues for Learners who are Blind[END_REF]).

Quantitative empirical evaluations in HCI

By quantitative empirical evaluation, we refer to Barkhuus and Rode [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF]'s classification of evaluation methods at CHI: quantitative refers to the type of measures taken (e.g., number of errors or time to accomplish tasks) and empirical means the evaluation involves users directly (instead of experts, as in the case in analytical evaluation such as heuristic evaluation). Quantitative empirical evaluations are largely considered a gold standard in HCI [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF].

A quantitative empirical evaluation is the study, generally task-based, of a given device, technique, or system. It aims at objectively measuring its usability and usage by users, or exploring the extent to which a technology is beneficial compared to previous similar propositions (independent variables) [START_REF] Lam | Seven guiding scenarios for information visualization evaluation[END_REF][START_REF] Petrie | The evaluation of accessibility, usability, and user experience[END_REF][START_REF] Mcgrath | Methodology matters: Doing research in the behavioral and social sciences[END_REF]. These measures (dependent variables) are proxy for performance (e.g., speed) or subjective experience (e.g., preference). Within research, they can be formative (i.e., inform an improved or final iteration of a system design); or summative (i.e., used as a proof of validity and adequacy of a proposed system), and either can become the basis for creating an improved design [START_REF] Scholtz | Usability evaluation[END_REF]. Completion of statistical tests can confirm the trustworthiness of the results for a larger population. These evaluations often include qualitative questions.

Scholars have raised three main concerns about this type of evaluation: (1) there needs to be more transparency in the reporting and validity of experiments in light of practices such as 'p-hacking' (the lack of consideration of effect sizes and confidence interval or similar practices) [START_REF] Dragicevic | Fair statistical communication in HCI[END_REF]; (2) they are not necessarily representative of the larger user population [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF][START_REF] Sears | Representing Users in Accessibility Research[END_REF]; and (3) they do not provide a holistic picture of peoples' understanding of a system in context [START_REF] Greenberg | Usability evaluation considered harmful (some of the time)[END_REF][START_REF] Spiel | When empathy is not enough: Assessing the experiences of autistic children with technologies[END_REF][START_REF] Kjeldskov | Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations[END_REF]. This might, for instance, lead to discarding forward-thinking technology containing usability issues that could be resolved later [START_REF] Greenberg | Usability evaluation considered harmful (some of the time)[END_REF], or to missing what is important to marginalised users because they focus on restrictive measures of subjective experience [START_REF] Spiel | When empathy is not enough: Assessing the experiences of autistic children with technologies[END_REF][START_REF] Kjeldskov | Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations[END_REF]. The above concerns suggest that a review of quantitative evaluation in HCI should be concerned with (1) assessing current practices and standards for reporting evaluations, to highlight areas for improvement; (2) assessing the extent to which participants are representative of the target population; and (3) examining the challenges and limitations mentioned, to reflect on potential ways forward. It is in light of this characterization that we formulated the three previously-stated secondary research questions that we used to focus our survey of quantitative evaluation methods: RQ1: Are there common standards for conducting quantitative empirical evaluations of technologies for PVI; if so, what are they?; RQ2: Overall, are participants in these evaluations representative of the target population?; and RQ3: What are the difficulties encountered by researchers in this type of evaluation?

Evaluating with and for People with Vision Impairment

There are rising concerns in the accessibility community that few technologies are thoroughly evaluated, let alone in useful contexts or with the relevant user group, such as children (e.g., [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF][START_REF] Gori | Devices for visually impaired people: High technological devices with low user acceptance and no adaptability for children[END_REF][START_REF] Brulé | Evaluating Technologies with and for Disabled Children[END_REF][START_REF] Gallagher | Indoor navigation for people who are blind or vision impaired: where are we and where are we going[END_REF]). Recent initiatives to address this issue were aimed at building community resources on this topic [START_REF] Brulé | Evaluating Technologies with and for Disabled Children[END_REF] or developing alternative quantitative evaluation approaches that are easier for researchers to conduct [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF]. This is a two-way problem: if not properly evaluated and iterated on, technologies risk being rejected when commercialized [START_REF] Gori | Devices for visually impaired people: High technological devices with low user acceptance and no adaptability for children[END_REF]. If few prototypes are publicly available, it is difficult to assess their impact on practices in the field.

A first difficulty for evaluating technologies with PVI are the low numbers of participants, particularly children. According to the World Health Organisation (WHO) [START_REF] Pascolini | Global estimates of visual impairment: 2010[END_REF], prevalence of visual impairments is estimated between 0.2 and 3% in Western Europe and North American countries, with 90% of visually impaired people worldwide living in low-to-middle income countries [START_REF] Khashim | Global prevalence and causes of visual impairment with special reference to the general population of Saudi Arabia[END_REF]. Blindness is a small (7%) subset of visual impairments, and the vast majority of cases of visual impairments (65 to 82%) occur after the age of 50. Moreover, PVI may be solicited often by researchers, limiting their availability. Consequently, many experiments use blindfolded sighted participants instead. Bardot et al. [START_REF] Bardot | Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision[END_REF] list three experimental evaluation approaches involving participants who are sighted and visually impaired: (1) mixed group for all experiments, which has to account for differences in performance;

(2) formative experiments with a group of sighted blindfolded participants to reduce the number of alternatives to test, and a summative experiment with PVI; and (3) a single evaluation with a small group of PVI. The authors note that this third option is generally perceived as inadequate for evaluating performance due to the small sample size, making it necessary to include sighted participants (i.e., option (1) or ( 2)). The adequacy of this practice is debated, both on validity [START_REF] Sears | Representing Users in Accessibility Research[END_REF] and political grounds, with concerns that disability simulation negatively affects the perception and representation of disabled people [START_REF] Bennett | The Promise of Empathy: Design, Disability, and Knowing the "Other[END_REF].

A second difficulty is the lack of shared evaluation frameworks or adapted evaluation tools [START_REF] Sánchez | Software Usability for Blind Children Questionnaire (SUBC). Usability evaluation test[END_REF][START_REF] Brulé | Evaluating Technologies with and for Disabled Children[END_REF][START_REF] Friedlander | Bullseye! when Fitts' law doesn't fit[END_REF]. Previous research suggests interaction laws [START_REF] Friedlander | Bullseye! when Fitts' law doesn't fit[END_REF], usability questionnaires [START_REF] Sánchez | Software Usability for Blind Children Questionnaire (SUBC). Usability evaluation test[END_REF], and procedures (e.g., length [START_REF] Bardot | Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision[END_REF]) need to be adapted to PVI. It also suggests that we need more homogenised experimental designs for evaluating typical technologies (e.g., for common contexts, like indoor navigation [START_REF] Gallagher | Indoor navigation for people who are blind or vision impaired: where are we and where are we going[END_REF][START_REF] Kacorri | Environmental factors in indoor navigation based on real-world trajectories of blind users[END_REF]).

A final difficulty comes from the sample of participants: heterogeneity in terms of visual abilities, age of onset, associated impairments, access to education, type of assistive technology used [START_REF] Cavender | Accessible writing guide[END_REF] and previous experience with visual concepts [START_REF] Robert | An approach to the evaluation of assistive technology[END_REF]. This heterogeneity makes it difficult to have an adequate control group [START_REF] Metatla | Toward Classroom Experiences Inclusive of Students with Disabilities[END_REF][START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF][START_REF] Brulé | Representing children living with visual impairments in the design process: A case study with personae[END_REF]. To address this issue, some scholars call for alternative approaches to experimental design, such as single-subject experiments [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF]. Even though the above challenges have been acknowledged in the community, to our knowledge, there are no systematic surveys that provide a detailed overview of reported difficulties, accepted best practices in quantitative empirical evaluations, nor guidelines for the way they are, or should be used in practice.

What can we learn from a literature review?

There are two approaches to completing literature review papers: systematic and narrative [START_REF] John | Balancing the strengths of systematic and narrative reviews[END_REF]. Systematic literature reviews aim at achieving unbiased, complete and reproducible overviews of a given research field [START_REF] Kitchenham | Procedures for performing systematic reviews[END_REF]. They attempt to eliminate biases in the selection of papers included by meticulously describing the search and selection process, and they provide a quantitative summary of the papers found. However, the extent to which any literature review can be unbiased and complete is unclear due, for instance, to technical limitations of database searches [START_REF] Sebastian | Literature reviews and the hermeneutic circle[END_REF]. Narrative literature review focus on developing a critical theoretical understanding of a field at a given time (e.g., [START_REF] Boehner | How HCI interprets the probes[END_REF]). We took a systematic approach, on which we reflect critically.

LITERATURE REVIEW APPROACH

The purpose of this systematic literature review is to identify best practices for quantitative empirical evaluations with PVI (RQ). We structured our process following recommended practices for systematic reviews: search, study selection, data extraction, data analysis, and interpretation [START_REF] Kitchenham | Procedures for performing systematic reviews[END_REF]. We then aimed to understand these best practices by (1) studying the main characteristics of these experiments and their participants; [START_REF] Iyad | Making Microsoft Excelâ Ďć: multimodal presentation of charts[END_REF] identifying potential gaps in their design or reporting; and (3) discussing ways forward.

Search strategy and query

We first conducted a Scopus search across different publishers (IEEE, ACM, etc). This is described in supplementary material. However, this corpus missed many publications that did not explicitly discuss evaluation in the abstract. Moreover, it was very heterogeneous in terms of writing styles, length and structure. We thus decided to focus on ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) and the ACM SIGACCESS Conference on Computers and Accessibility (ASSETS), as well as their associated journals: ACM Transactions on Computer-Human Interaction (TOCHI) and ACM Transactions on Accessible Computing (TACCESS). These are largely considered premier venues for HCI publications [START_REF]Human Computer Interaction -Google Scholar Metrics[END_REF], and focusing on a limited number of well-recognised publications in which this type of evaluation is popular [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF] and is a strategy previously used by similar literature reviews (e.g., [START_REF] Lam | Seven guiding scenarios for information visualization evaluation[END_REF][START_REF] Perer | Integrating statistics and visualization for exploratory power: From long-term case studies to design guidelines[END_REF][START_REF] Halskov | The diversity of participatory design research practice at PDC 2002-2012[END_REF]). Our aim is not to derive best practices for all types of evaluation from this corpus: many other wellrecognised venues may favour different evaluation approaches and epistemologies [START_REF] Spiel | When empathy is not enough: Assessing the experiences of autistic children with technologies[END_REF]. However, given empirical quantitative evaluations are a well-recognised approach to evaluation across the field of HCI [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF], standards for their conduct and publication would benefit all publication venues.

We searched the proceedings of the publications listed above, using a request consisting only of the keywords for "visual impairments" on Scopus. The search yielded 821 results. In contrast with the preliminary study, we did not include technology keywords since they are implicit to HCI. Nor did we include the evaluation keywords as we filtered out papers manually. The main search string is below; the full string restricting the search to the publications listed above can be found in supplementary material: TITLE-ABS-KEY(blind OR "deaf blind" OR deafblind OR deaf-blind OR "eye disorder*" OR "partially sighted" OR "vision disorder*" OR "visual disabilit*" OR blindness OR "visual impairment*" OR "visually impaired" OR "partial vision")

Study Selection

We performed two rounds of study selection: (1) the first round included papers based on their abstracts, and (2) the second round included papers based on whether they reported a quantitative empirical evaluation.

First exclusion process: One researcher (five years of experience in this area) read the abstracts of the 821 papers to select studies likely to include a quantitative empirical evaluation. She included 298 papers and excluded 523 papers. In cases where the type of evaluation is not specified within the abstract, the papers were included.

Papers were excluded for the following reasons. The first set were out of scope (n=198), and either included systems or technology not intended for PVI, or papers reporting using only non-quantitative types of evaluation (e.g., qualitative, technical evaluation of algorithm accuracy). Papers that were not full papers were also excluded (n=324): we did not include posters, workshop papers, late-breaking work, or short papers. This is because the reporting of experiments have too much discrepancy (often involving a lack of detail compared to full papers). Finally, duplicate papers were excluded (n=1). See Supplementary material for the corpus at this stage.

Second exclusion process:

The second exclusion was shared among the authors (between 5 and 20 years of experience of working on technologies for PVI): each was assigned between 40 and 80 papers to review and code. During this process, authors excluded 120 papers. Papers were excluded because they were out of scope (n=63); they only reported qualitative evaluation(s) (see 3.4 for a discussion of mixed quantitative and qualitative evaluations); they were short papers (n=46); they did not meet requirements for reported information (n=10) (e.g., papers in which there was too much missing data about the evaluation); or they were duplicate papers (n=1). At the end of this second exclusion process, there were 178 papers remaining in the corpus.

Data coding and cleaning

The corpus was reviewed by the five authors. We first coded data describing the participants and evaluations from five papers. This enabled verifying expectations were aligned and that the data extracted was suitable for addressing the research questions. We could not find a model for the type of data to extract for this analysis. Instead, we developed the following reporting form through discussion and iteration:

• Mode of interaction (e.g., visual enhancement, tactile, tangible, gestures); • Application area (e.g., navigation and walking, techniques to access GUIs, education); • Number of quantitative empirical evaluations and of studies:

to identify whether evaluations are used in concert with other evaluation methods or in isolation, as well as the number of evaluations per paper; • Per evaluation: the description of its purpose, number and reported characteristics of participants, length and context, measures (dependent variables), aim of the evaluation (measuring: primarily performance; performance with subjective experience; or primarily subjective experience), the statistical tests used, presence of qualitative questions, difficulties mentioned by the authors and whether the coder estimates being able to reproduce the evaluation. These data enabled understanding of whether participants were representative of the larger visually impaired community, uncovered shared standards (such as the statistical tests used), and identified encountered difficulties; • Whether and how sighted participants were involved.

We carried out a verification process, during which an author read and extracted data from one paper for each year of publication in the corpus. She found no issue with the data extracted, but homogenized the way missing data was reported. For instance, coders had used "n/a," "0" or "not reported" alternatively to indicate absence of data. We also produced two documents for data processing and analysis: one with one paper per line and every quantitative empirical evaluations it includes, and one with one evaluation per line.

Challenges in the selection and coding process

The structured description was initially more extensive, including the type of p-values used (when used), number of tasks, precise description of visual impairments measurements, independent variables, etc. However, we found that even recent papers did not provide all these details, which we discuss below. Moreover, the coding process revealed that the difference quantitative and qualitative evaluations can be thin. One recurrent debate regarding the inclusion of papers in the corpus was how to handle papers using an experimental setup for evaluation, reporting some quantitative measures but focusing primarily on qualitative results. We decided to include them, since there are no guidelines describing the minimum number of quantitative measures a paper should include, and using a narrow definition of quantitative experiments would have excluded many older studies. They also share evaluation constructs: experience and performance, whereas qualitative evaluations may look at different aspects. For instance, we included [START_REF] Götzelmann | Autonomous Selection and Printing of 3D Models for People Who Are Blind[END_REF] which measured time and task success, but otherwise evaluated usability qualitatively; but we did not include Linespace because it presented only subjective usability feedback, reported thematically [START_REF] Swaminathan | Linespace: A sensemaking platform for the blind[END_REF]. Two of the five authors also included qualitative evaluations reporting Cohen's kappa for inter-rater reliability and included a coding sheet, because the results were quantifiable. We decided not to include them, since the evaluation constructs were different. A last discrepancy between coders was the inclusion of pilot studies if they are mentioned but not described. We included them, as this is precisely a practice that we would benefit from understanding. In summary, this approach favors homogeneity of evaluation constructs.

Analysis and Interpretation

The analysis was conducted using Excel and SPSS. We first calculated descriptive statistics to summarize each of the tagged aspects of evaluations for the entire corpus. We analysed values over the whole corpus before looking at trends by publication year to assess whether our findings were still representative of practices. When high heterogeneity was found (high standard deviation), we examined outlying papers to provide an explanation. We also propose a typology of papers and of how sighted participants are included in these evaluations based on the data. To interpret these results, we leveraged the works from the Background section to make recommendations for future work. The aim is to contextualise this work within current concerns about evaluation in HCI.

OVERVIEW OF THE CORPUS

Before reporting on evaluations, we describe the corpus in terms of number of papers per venue and year of publication, areas of application, and interaction modalities involved.

Paper origin

Among the 178 tagged papers, 50 had been published at CHI,

Area of application

In the initial coding sample, we included activities that papers stated they aimed to enable or ease (such as navigation or education, following Hersh's classification [START_REF] Hersh | Assistive technology for visually impaired and blind people[END_REF]). We expanded these categories when needed (e.g., gaming) and added technical categories for papers about systems and techniques that could be used in a variety of activities (e.g., GUIs in nonmobile contexts, mobile devices, or XR environments). Some papers belong to several categories, which is why percentages do not add to 100%. Reported percentages are rounded up to the nearest tenth decimal place.

The primary investigated application area is web browsing (n=43, 24.2%). Second was education and mobility (n=26, 14.6% each). The next two were technical categories: interaction with mobile screens (n=21, 11.8%) and access to traditional GUIs (n=16, 9%). This confirms a recent systematic literature review on the type of technologies currently being designed for PVI [START_REF] Bhowmick | An insight into assistive technology for the visually impaired and blind people: state-of-the-art and future trends[END_REF]. Rarer application areas include: data-visualisation and graphs (n=8, 4.5%); gaming, reading and writing, and daily life tasks (n=6, 3.4% each); access to shape and images, and object-finding; research on XR environments (n=5, 2.8% each); photography and sports (n=4, 2.3% each); rehabilitation and supporting collaboration (n=3, 1.7% each); coding or learning to code, access to videos, and smart home control (n=2, 1.1% each); and drawing images, producing accessible content, and in-person communication (n=1, 0.6% each).

Interaction modalities

Similar to the coding of the system, we initially considered 8 types of interaction modalities: speech (n=102, 57.3% of papers), non-speech audio (n=57, 32%), tactile (n=41, 23%), changes to the type of data or its structure (e.g. generating captions automatically, n=25, 14%), vibratory haptics (n=18, 10.1%), tangible interaction (n=17, 9.6%), force feedback haptics (n=14, 7.9%), and others. Papers in the "others" category were grouped initially; we later added the following categories: visual enhancement (highlight, zoom, etc., n=10, 5.6% of papers), keyboard input (n=5, 2.8%), visual (some evaluation aiming at understanding the suitability of visual interfaces for people with remaining visual abilities) and gestures (n=2, 1.1% each). Audio-based interaction clearly remains the primary approach for accessibility to this group, potentially due to its availability on most digital devices.

ARE THERE STANDARDS FOR CONDUCTING QUANTITA-TIVE EMPIRICAL EVALUATIONS?

In this section, we provide an overview of the paper structures and of the evaluation designs to answer RQ1: Are there common standards for conducting quantitative empirical evaluations of technologies for PVI; if so, what are they?

Number of experiments and studies per paper

Within the 178 tagged papers, 125 (70%) papers reported only one quantitative empirical evaluation, 41 (23%) papers reported two, and 12 (7%) reported three (i.e., we analysed 243 evaluations in total). There is a large variety in the evaluations being conducted and their use. We look more closely at the type of papers in which they are used. Papers can present a novel technology and its evaluation, which Wobbrock:2016:RCH:2930854.2907069 [START_REF] Wobbrock | Research Contributions in Human-computer Interaction[END_REF] describe as an Artifact contribution to HCI. Or they focus on evaluation, making an Empirical Research contribution.

Within papers presenting an Artifact contribution:

• 80 papers (44.9%) presented only one summative quantitative evaluation, while 6 papers presented several summative evaluations. This was generally because they distinguished between two groups (sighted and visually impaired, e.g., [START_REF] Reichinger | Pictures in your mind: using interactive gesture-controlled reliefs to explore art[END_REF]) or settings (lab and field, e.g., [START_REF] Sato | Navcog3: An evaluation of a smartphone-based blind indoor navigation assistant with semantic features in a large-scale environment[END_REF]); • 30 papers (16.9%) presented at least one formative and one summative quantitative evaluation. This appears to be more prevalent with time (sparse before 2008, at least one per year after 2008). These formative experiments often focused on choosing between several interaction techniques (e.g., [START_REF] Ducasse | BotMap: Non-Visual Panning and Zooming with an Actuated Tabletop Tangible Interface[END_REF]) or on ranking ways to present information (e.g., crowdsourced comparison in [START_REF] Gadde | From screen reading to aural glancing: towards instant access to key page sections[END_REF]); • 5 papers (2.8%) reported a summative quantitative evaluation with a pilot study (e.g., [START_REF] Albouys-Perrois | Towards a multisensory augmented reality map for blind and low vision people: A participatory design approach[END_REF]) but were not described in much detail past informing the final design. • 21 papers (11.8%) presented qualitative formative studies or evaluation and summative experiment(s) (e.g., [START_REF] Wang | EarTouch: Facilitating Smartphone Use for Visually Impaired People in Mobile and Public Scenarios[END_REF]). This is also a fairly recent type of paper (only two were published before 2008).

Taken together, the use of multiple studies and evaluations could show a trend towards a more iterative, user-centered, design processes. This could also suggest an adaptation to critiques about conducting evaluations with sighted participants, criticism of this practice are fairly recent. Finally, it could also suggest a rise in interdisciplinary and hybrid approaches:

• 11 papers (6.2%) presented only formative quantitative evaluations. In some cases they provided a second design iteration within the same paper (e.g., [START_REF] Vashistha | Sangeet swara: A community-moderated voice forum in rural india[END_REF]) or in a separate evaluation published later (e.g., [START_REF] Bruce N Walker | The audio abacus: representing numerical values with nonspeech sound for the visually impaired[END_REF]). This small number could be due to the community focus for including evaluation as part of the design process, instead of publishing entirely design-focused papers; • 4 papers (2.2%) presented a qualitative and a quantitative summative evaluation (e.g., [START_REF] Vacher | Evaluation of a context-aware voice interface for Ambient Assisted Living: qualitative user study vs. quantitative system evaluation[END_REF]). They were all published after 2015. In addition, 118 experiments (48.5%) were tagged as including qualitative questions. This could suggest a rise in hybrid approaches to evaluation;

• 6 papers (3.4%) presented analytical formative studies and evaluations (e.g., benchmark or technical evaluation not involving human participants).

Finally, an empirical quantitative evaluation can be the focus of a paper, and inform us as much about technologies as they do about users (n=15). For instance, [START_REF] Trewin | Usage of subjective scales in accessibility research[END_REF] used such an evaluation to make a methodological contribution, and [START_REF] Pal | Agency in assistive technology adoption: visual impairment and smartphone use in Bangalore[END_REF] examined the usability of phones with a theoretical aim.

Study length and context

Study length was not consistently reported. We were able to report it in 102, or 42% of, evaluations. When a time range was reported, we calculated the average. Four evaluations (1.6% of all evaluations) were conducted over periods superior to four days and up to two weeks long. For lab-based evaluations, the average length of en experiment is 89 minutes (median=66, min=10, max=360, stdev=70). For context, we found advice in the literature that evaluations should be shorter to avoid fatigue (e.g., [START_REF] Bardot | Investigating Feedback for Two-Handed Exploration of Digital Maps Without Vision[END_REF] recommends to keep them under 60 min). There is a high heterogeneity in the length of lab-based evaluations, and a very small percentage of long-term evaluation.

Context is not often explicitly reported and discussed, unless the experiment took place outside of a lab study. During data extraction, we assumed that when it was not reported, the experiment was conducted in an academic office or lab. In 45 cases (18.5% of evaluations), the experiments were conducted in a different type of space: online or remotely (n=14, 5.8% of all evaluations), everyday life settings (n=13, 5.3%), and in relevant public settings such as airports (n=8, 3.3%), schools (n=6, 2.5%), community spaces (e.g., the NY Lighthouse; n=3, 1.2%), and libraries (n=1, 0.4%). We can conclude quantitative empirical evaluations are primarily conducted in non-realistic settings.

Validated questionnaires

We reported whether evaluations used standardized questionnaires to investigate usability: 23 evaluations did so (9.5%). These validated questionnaires are primarily for measuring subjective experience. This includes the System Usability Scale (SUS, n=9, 3.7% of all evaluations, [START_REF] Bangor | An empirical evaluation of the system usability scale[END_REF]) and the User Experience questionnaire (UEQ, n=2, 0.8%, [START_REF] Schrepp | Applying the user experience questionnaire (UEQ) in different evaluation scenarios[END_REF]). Used a single time (or in 0.4% of evaluations each) were the Design Oriented Evaluation of Perceived Usability (DEEP, [START_REF] Yang | DEEP: design-oriented evaluation of perceived usability[END_REF]), the Software Usability for Blind Children Questionnaire (SUBC, [START_REF] Sánchez | Software Usability for Blind Children Questionnaire (SUBC). Usability evaluation test[END_REF]), the Single Use Question (SUQ, [START_REF] Sauro | Comparison of three one-question, post-task usability questionnaires[END_REF]), and the Positive and Negative Affect Schedule (PANAS, [START_REF] Watson | Development and validation of brief measures of positive and negative affect: the PANAS scales[END_REF]). Domain specific questionnaires were used, including the Robotic Social Attributes Scale (RoSAS, n=1, 0.4%, [START_REF] Carpinella | The robotic social attributes scale (rosas): Development and validation[END_REF]) and the Igroup Presence Questionnaire (n=1, 0.4%, [START_REF] Bob | Measuring presence in virtual environments: A presence questionnaire[END_REF]). Some papers reported adapting Nielsen's heuristics [START_REF] Nielsen | Heuristic evaluation of user interfaces[END_REF] or Shneiderman's golden rules [START_REF] Shneiderman | Designing the User Interface: Strategies for Effective Human-computer Interaction[END_REF] as the basis for subjective experience questions and Likert scales. Finally, eight studies (3.3%) used the NASA-TLX test for cognitive workload [START_REF] Sandra | NASA-task load index (NASA-TLX); 20 years later[END_REF]. Overall, there does not seem to be a standard shared set of tools and measures.

Measures

We coded for three types of measures: primarily task performance (n=145, 59.7%), task performance and subjective experience (n=69, 28.4%), and primarily subjective experience (n=29, 11.9%). Although the majority of papers focus on performance, a significant number of usability experiments focus on subjective experience, through measurements such as preference and perceived difficulty.

Statistical tests

We found that 143 evaluations (59.7%) used a statistical test to validate their results (twelve, or 4.9%, used descriptive statistics). The most common tests are from the ANOVA (n=64, 26.3%) or t-tests (n=33, 13.6%) families, followed by Wilcoxon (n=26, 10.7%). Many of the experiments without statistical tests are pilot studies, and others focus on subjective experience with quantitative measures. If we look at papers with only summative experiments, the number of papers using a statistical test raises to 67%. Hence, although it is a practice used by a majority of the papers, many others only report only numerical values. One reason provided is that the sample size of participants is too small (e.g., [START_REF] Syed | Accessible Gesture Typing for Non-Visual Text Entry on Smartphones[END_REF]).

Quality metrics

In addition to the exclusion of papers with inconsistent reporting or very little detail, we tagged (0 or 1) whether experiments were missing information that would make duplicating the experiment difficult. This code was used at the discretion of the coders. The coders looked for incomplete description of tasks, experiment design, and measures. Coders expressed this lack of detail for 39 evaluations (16%). This does not appear to have decreased significantly through time (it represents 15% of evaluations between 2015 and 2019). This is concerning, especially as many more papers could prove difficult to replicate in practice.

Answering RQ1

Overall, these findings suggest that the most common type of paper is one that presents a new system followed by a summative quantitative empirical evaluation (lab-based, measuring primarily performance, using a statistical test). It could be described as a standard. However, papers with formative studies and evaluations are becoming ever more common, with a large variety in the type of methods used. This could suggest an evolution towards more user-centered and iterative work (which might not be specific to assistive technologies for PVI), as well as better recognition of qualitative perspectives. We note, however, that very few papers were evaluated quantitatively in real settings and that there are concerns regarding reproducibility, which could weaken validity [START_REF] Kjeldskov | Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations[END_REF].

REPRESENTATIVENESS OF PARTICIPANTS

In this section, we analyse the characteristics of the participants involved to answer RQ2: Overall, are participants in these evaluations representative of the target population? We also propose a typology of how sighted participants are involved in evaluations.

Number of participants per evaluation

The average number of participants per evaluation was 22.3 (median=12, min=1, max=453, stdev=41.7). The number of participants varied with the type of evaluation. In formative evaluations, the average number of participants was 32.8 (median=12.4, stdev=64.1), as opposed to summative studies, where the number is 17.1 (median=11.8, stdev=23.1). The number of participants decreases according to the position of the experiment in the paper (as usually formative experiments are presented before summative ones): 24.5 for the first experiment (median=12, stdev=46.9), 17.3 participants for the second experiment (median=10, stdev=23.5), and 10.9 participants the third experiment (median=7, stdev=22).

Visual status of participants

Over the 243 experiments of our corpus, 208 (85.5%) included people with VI, 35 (14.4%) included blindfolded people, and 63 (25.9%) included sighted people (not blindfolded). Surprisingly, 3 papers did not report whether participants had visual impairment or not. Among the 208 studies with PVI, only 144 of them (59.2% of the corpus) did exclusively involve PVI. For experiments that only involved PVI, the average number of participants was 12.66 (median=9.67, stdev=11.8), while for studies that did not involve PVI, the average number was 14.9 (median=13, stdev=44.5). Those that used only blinfolded participants had 23.1 (median=14, stedv=24.6) participants on average; when participants were a combination of blindfolded and non-blindfolded sighted people (which only occurred for two studies) evaluations had 16 (median=16, stedv=5.6) participants on average. This is illustrated in Figure 2.

When looking at the distribution of the visual status of participants according to the type of study (see Figure 3), we observed that formative experiments more often involved blindfolded and sighted participants (n=44, 56% of formative studies), while summative experiments more frequently involved exclusively PVI (n=112, 69% of summative studies).

Concerning the type of visual impairment, 104 evaluations involved participants with non-specified visual impairments or severe visual impairments, 41 involve people who are legally blind, and 156 completely blind. Note that due to the absence of shared standards for reporting (here, we used the WHO typology [START_REF] Pascolini | Global estimates of visual impairment: 2010[END_REF]), some subjects may have been misclassified. types of impairment such as hearing impairment, intellectual disability, or limited mobility.

Age and gender

The age of participants was not consistently reported, but the majority of studies provided a median estimation. When data about every participant was available, we determined the median ourselves. When only a range was provided, we used the average.

The median age of participants was 37 years old (min=9, max=80). This median age was clearly higher for participants with vision impairment (38.3 years old) compared to sighted participants (29.6 years old).

Regarding gender: 65 experiments did not report this demographic. In papers that did report on it, men were more frequently involved in research than women (12 vs. 9.3 participants/evaluation). Across the entire corpus, the gender of three participants was reported as "other"2 , which we have to presume means non-binary and two did not disclose their gender.

Additional information

Beyond basic demographics, we identified four additional categories of information provided by authors (in 140 evaluations or 57.6%): technology usage (smartphone, computer, screen reader, etc.); expertise (braille, mathematics, musical, etc.); health-related information (visual and hearing acuity, health problems, etc.); and education (background). This suggests researchers are overall aware of the heterogeneity of participants' profiles, although there is no standard model for what to report [START_REF] Cavender | Accessible writing guide[END_REF].

Use of sighted participants

As noted above, sighted participants are often involved in evaluations (30% of the experiments in our corpus). We systematically tagged how sighted users participated in experiments:

• Comparison. When the performance of the two groups are evaluated, and then compared. Sometimes this is reported as one experiment, sometimes as two. This was the case of 34 evaluations in our corpus. Comparisons can be justified by fundamental research questions, such as in [START_REF] Mascetti | On the evaluation of novel sonification techniques for non-visual shape exploration[END_REF] "the comparison between the results of the two groups can help identifying similarities and differences in the cognitive process". • Replacement. In some evaluations, PVI are intended as the end-users, but are replaced by sighted participants. This is the case in 19 evaluations. Generally, sighted participants are used in formative or pilot studies (n=16), to choose the system that will be evaluated by PVI later. The analysis of the corpus confirms that replacement needs to be done carefully, as performance of both group of users vary depending on the task and technology (e.g., [START_REF] Iyad | Making Microsoft Excelâ Ďć: multimodal presentation of charts[END_REF]). • Baseline. Eight evaluations reported using sighted participants as a baseline for performance for a given system. For instance, a study of the accessibility of Augmented Reality glasses while walking required measuring impact of walking on sighted participants to identify the specific difficulty experienced by PVI [START_REF] Zhao | Understanding Low Vision People's Visual Perception on Commercial Augmented Reality Glasses[END_REF]. • Improve statistical significance. Two evaluations indicated that recruiting sighted users enabled them to verify statistical significance, especially in cases where they could also be a target user (e.g., [START_REF] Mascetti | On the evaluation of novel sonification techniques for non-visual shape exploration[END_REF]). In contrast, some recent studies (see below) state they did not use a statistical test because the number of participants was too small. It is hence unclear whether this practice will continue or will be replaced by other means of validating evaluations. • Two target user groups. Four evaluations explained their use of sighted participants by their goal to develop non-visual systems accessible by both sighted and PVI (e.g., [START_REF] Vacher | Evaluation of a context-aware voice interface for Ambient Assisted Living: qualitative user study vs. quantitative system evaluation[END_REF]). • Collaboration: Four evaluations cite collaboration as the main reason for involving sighted participants (one cites both collaboration and considering two target-user groups). • Understand impact on environment. In one case, sighted participants were involved as researchers wanted to evaluate the impact on the behavior of sighted pedestrians in public spaces. [START_REF] Kayukawa | BBeep: A Sonic Collision Avoidance System for Blind Travellers and Nearby Pedestrians[END_REF].

In summary, many evaluations involve sighted people; there are many legitimate cases in which to do so. Regarding the involvement of sighted participants as a replacement for PVI in formative evaluations, which is a more controversial approach, it might be possible to use qualitative studies to accomplish this instead, as the analysis of the corpus papers above suggests they are well accepted.

Answering RQ2

To contextualise these findings in terms of representativeness, we need to compare them to the population with VI in general. We note that blind and legally blind users are a minority within PVI: 7% according to the World Health Organisation (WHO) [START_REF] Pascolini | Global estimates of visual impairment: 2010[END_REF]. However, they are involved in the majority of experiments. Moreover, the WHO estimates that 65% of PVI and 82% of blind people are over the age of 50, hence likely to experience other impairments (2.4% in our corpus, the median age of PVI participants was 38.5 years-old). Overall, the design and therefore the evaluation of assistive technologies do not reflect the global population of PVI. We speculate the under-representation of blind participants can be explained by the fact that they may have more considerable accessibility issues than people with other visual impairments. Moreover, participants with additional impairments may instead be involved in studies about people with multiple disabilities. Another area of concern is that few experiments report the age of onset or cause of visual impairments, although this impacts cognition and technology use [START_REF] Victor R Schinazi | Spatial navigation by congenitally blind individuals[END_REF].

DIFFICULTIES REPORTED

During the tagging, we noted whether authors reported any particular difficulty concerning the study. With this, we aim to answer RQ3: What are the difficulties encountered by researchers in this type of evaluation?. We classified these comments, reported in 31 papers (17.3%), into seven categories. They are reported here from the most frequently occurring to the least, with the inclusion of representative examples from the corpus:

• Small available pool of PVI. Authors commented on recruitment difficulties (30%): "Obtaining a large number of blind participants was difficult due to their relatively low representation in the general population" [START_REF] Balik | Including blind people in computing through access to graphs[END_REF]. This was used to motivate the recruitment of sighted people for formative studies: "To avoid over-use of the relatively small pool of non-sighted people available for evaluations such as this, all but one of the participants [...] were sighted" [START_REF] Brown | Evaluation of a Non-visual Molecule Browser[END_REF]. Others reported their methods for recruiting participants, mostly through email lists and local organizations. This difficulty is even more common when it comes to recruiting children: "As access to blind children is very limited we first used blind and visually impaired young adults" [START_REF] Wilson | Using Dynamic Audio Feedback to Support Peripersonal Reaching in Young Visually Impaired People[END_REF]. • Sample size. As a direct consequence from the recruitment problem, 12% reported that the sample size for the study was small, explaining the choice of reporting only qualitative data: "the limited number of participants in our initial study makes it difficult for us to perform a statistical analysis of the quantitative data" [START_REF] Parente | Audio Enriched Links: Web Page Previews for Blind Users[END_REF]. • Expertise of the participants. Beyond the recruitment of PVI in general, finding participants with a specific expertise is very difficult (12%):"the number of people qualified to test the workstation and its components are small" [START_REF] Robert | An approach to the evaluation of assistive technology[END_REF]; "In Pakistan, finding blind computer users highly proficient with LaTeX was not an easy task" [START_REF] Manzoor | ALAP: Accessible LaTeX Based Mathematical Document Authoring and Presentation[END_REF]. • Consistency between participants. Again as a consequence of the small pool of available PVI, 10% reported that the resulting group of participants is too heterogeneous: "It would also be impossible to match participants; their backgrounds were simply too diverse: level of sight. aetiology of disability. level and form of education and so on." [START_REF] Robert | An approach to the evaluation of assistive technology[END_REF]. • Inclusion of sighted people. Authors also discussed (9%) the inclusion of sighted participants in the study as a limitation ("One limitation of this study is that the sighted participants' performance may not be representative of blind users" [START_REF] Hong | Evaluating Wrist-Based Haptic Feedback for Non-Visual Target Finding and Path Tracing on a 2D Surface[END_REF]). • Exclusion of participants. In some papers, authors reported (6%) that they had to exclude participants from the study results due to several reasons: "We excluded from this analysis one participant with low vision, since she could see, with some effort, the route on the screen" [START_REF] Guerreiro | Virtual Navigation for Blind People: Building Sequential Representations of the Real-World[END_REF] or "we had to exclude 4 of them; the first one did not have an understanding of what geometric figures are, [...] the second one had an unusually small hand [...] the third one kept falling asleep [...] and with the fourth participant, we experienced hardware failure" [START_REF] Soviak | Tactile Accessibility: Does Anyone Need a Haptic Glove?[END_REF]. • Other comments. Authors reported other difficulties, such as scheduling problems (e.g., "requiring participants to travel to the study location" [START_REF] Bragg | A Large Inclusive Study of Human Listening Rates[END_REF]), technical issues, the length of the experiment being too long for a certain modality, the difficulty to create an inclusive experimental environment, the generalization of the results, or the definition of the study measures.

Answering RQ3

Researchers do express difficulty to find and involve PVI in their studies. By extension, it is difficult to design and conduct experiments similar to how they would be done with sighted participants. Taken together with the results of RQ1 and RQ2, this suggests that the standards for quantitative empirical evaluations are hard to apply in this area of HCI, which could explain researchers turning to other approaches as discussed earlier.

DISCUSSION

Answering our research question

We addressed the research question What is, at present, best practice in quantitative empirical evaluations of technologies for people with visual impairment in HCI? through an analysis of published evaluations. We adopted a broad definition of quantitative empirical evaluations (instead of one restricted to controlled lab studies) to be able to identify the full range of approaches and what we can learn from them going forward. We found a wide range of approaches. There is one type of paper more common than others, a description of a design with one summative quantitative experiment, followed by papers with either a pilot or a formative quantitative empirical evaluation with one summative experiment. However, [START_REF]Human Computer Interaction -Google Scholar Metrics[END_REF] there are concerns about the quality of reporting of evaluations; (2) most design focus on a subset of PVI; and (3) researchers face difficulties in running evaluations (especially with PVI) while reaching the statistical significance expected in these venues. Practices appear to adapt to this issue in various ways, such as turning to qualitative evaluations or replacing visually impaired participants with blindfolded sighted ones.

At present, involving sighted participants instead of visually impaired participants is accepted practice, despite the many critiques leveraged against it. We thus need to discuss and build standards for empirical quantitative evaluations that both take into account the difficulties researchers are running into and this critique. For this, we propose to learn from other fields that have addressed the same issues.

Limitations and future work

We identify two areas regarding limitations and future work: evaluations in other fields, and evaluating beyond empirical quantitative evaluations.

Learning from Other Fields

We have limited our inquiry to publications about technologies in the field of HCI. This potentially limits our understanding of possible approaches to evaluation. During this research, we found similar literature review in learning sciences [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF] but not others. As pointed out in our answer to RQ2, many evaluations in our corpus do not follow standards for evaluating technologies with PVI used in related research fields. This could restrict interdisciplinary exchanges. Moreover, other disciplines concerned with assistive technologies can provide inspiration for the evaluations conducted in HCI. For instance, Kelly and Smith [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF] acknowledge the difficulty of conducting traditional experiments with a control group in special education, recommending instead the use of single-subject design. This is a standard practice in the field of orientation and mobility as well [START_REF] Catherine | Case Studies, Single-Subject Research, and N of 1 Randomized Trials: Comparisons and Contrasts1[END_REF][START_REF] Parker | Orientation and mobility with persons who are deaf-blind: An initial examination of single-subject design research[END_REF][START_REF] Steven | The use of the Sonic Pathfinder as a secondary mobility aid for travel in business environments: a single-subject design[END_REF]. Both areas of application are prominent in our corpus. Technologies could be evaluated with a low number of representative participants, with new experiments over time adding to the credibility of the evaluation. The question however is which HCI publications would allow this.

Other assessment approaches

We focus here on quantitative empirical evaluations. They pose a number of concerns, from their validity to their appropriateness for technology that do not aim at compensating an impairment but rather to foster experience or community. Several articles in the corpus raise concerns about the validity of quantitative evaluations for user preference, due to users attempting to please the designer [START_REF] Schrepp | Applying the user experience questionnaire (UEQ) in different evaluation scenarios[END_REF][START_REF] Trewin | Usage of subjective scales in accessibility research[END_REF], which seems more pronounced for participants with visual impairments [START_REF] Trewin | Usage of subjective scales in accessibility research[END_REF]. Further leveraging qualitative studies could be a way to understand more nuanced experiences with technologies [START_REF] Spiel | When empathy is not enough: Assessing the experiences of autistic children with technologies[END_REF]. We should also account for the novelty effect [START_REF] Kjeldskov | Was it worth the hassle?: ten years of mobile HCI research discussions on lab and field evaluations[END_REF]. Some papers in our corpus conduct evaluations in the field and/or over relatively long periods (up to two weeks), but this is difficult to do with large groups of participants. Single-subject experimental designs could also be a way to address this issue going forward.

Analytical evaluation includes heuristic evaluation, expert comparisons with other products, or expert-led analysis to outline a design space and design implications. These forms of expert-led evaluations have become increasingly rare in the 2000s at venues such as CHI [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF], although they are still used as formative studies (6 papers in our corpus). Heuristic evaluations [START_REF] Nielsen | Heuristic evaluation of user interfaces[END_REF] are based on expert users identifying issues and suggesting improvements, reporting the number of errors or rating functions. This could be interesting in the case of evaluations with PVI, as they can be considered as experts of non-visual uses of technologies. These results would have to be considered carefully and optimally, so several evaluation approaches should be used [START_REF] Wayne | Damaged merchandise? A review of experiments that compare usability evaluation methods[END_REF]. These other approaches to evaluation confirm the wide range of evaluation practices in this community. It suggests potential ways of complementing quantitative empirical evaluation, especially in the formative stages, to lessen the involvement of sighted participants [START_REF] Sears | Representing Users in Accessibility Research[END_REF]. However, the variety of evaluation approaches could also be seen negatively, as preventing the accumulation of knowledge due to generalization difficulty. We would argue that the community benefits from shared standards in the area of quantitative evaluation, particularly as these methods are key for securing funding for new interventions [START_REF] Stacy | The impact of assistive technology on the educational performance of students with visual impairments: A synthesis of the research[END_REF]. These standards should evolve as they are challenged by new approaches to evaluation and the historical methods are not always appropriate.

RECOMMENDATIONS

The exploration of the three initial research questions led us to a set of recommendations on how to conduct, report, and review experimental evaluations of technology for PVI.

Recommendations for conducting evaluations

In this section we discuss how the involvement of participants with visual impairments and approaches potentially enabling their wider participation.

• While it may be acceptable to include sighted participants for testing interaction techniques at the level of "actions," it is not for testing a device designed to do a more cognitive task. Indeed, visual impairments have consequences on education, knowledge and mental representations, which, in turn, have an impact on the execution of cognitive tasks. • This also suggests that if involving a few PVI in the early cycles of a user-centered design is considered beneficial, validating the final design with that same population is considered crucial. • Visual impairments other than (isolated) blindness seem under-represented across many of the studies. Involving other users could offer a different perspective on the usability of technologies, and may provide insights to the technology's benefit to a diverse population. • The variety of evaluation methods in the corpus should encourage researchers to use the full range of evaluation approaches, echoing Barkhuus and Rode [START_REF] Barkhuus | From Mice to Men -24 Years of Evaluation in CHI[END_REF]. We argue this is not opposed to a shared standard for quantitative evaluation, but that instead, it contributes to its evolution and improvement through time. • Evaluations conducted in the field, on participants' own spaces and schedules, and within a context of high involvement in the community such as participatory design could be easier to conduct. • Single subject design for quantitative evaluations could be an interesting approach to address the current difficulties encountered by researchers.

Recommendations for reporting evaluation

Within the corpus there were inconsistencies in reporting. Here are a few key points all authors can address to help situate study results and understand their generalizability:

• We encourage authors to report participant characteristics individually through a description table, using the user characteristics listed above, as long as this does not threaten their privacy and security. • We also encourage authors to use standardized metrics when giving an overview of the group, including median age, gender, education level, and experience with relevant technologies or subjects. • It would be beneficial to systematically use the WHO classification for visual impairments so experiments can be compared. This considers that technologies evaluated qualitatively have the aim to help access to fairly narrow tasks, for which this medical classification is relevant. In other cases, functional definitions (e.g., ability to read at a certain distance most of the time) might be better suited. • We provide a typology of the reasons for including sighted users as participants. These could be used as references to describe and motivate their inclusion. • For general reporting, an even greater transparency on the issues of designing and analysing results would be beneficial to this research community as it enables continuous review of standards.

Recommendations for reviewing evaluation

In addition to changes researchers can make to their own research and reporting practices, they can also consider the following points when reviewing work for others:

• We highlight the difficulties reported by researchers during their evaluations. Being aware and considerate of these difficulties could help promote more inclusive practices, instead of adaptations to quality metrics used for other technologies (e.g., adding sighted participants to reach validity). • Rejecting a paper based on low sample size may hinder research in the field; the community should support authors and new researchers in reporting results appropriate for the sample size, i.e., using in-depth qualitative analysis if quantitative is not possible. • Reviewers should pay special attention to how authors report their evaluations and particularly to comprehensiveness of the description of participants. • Reviewers can refer to the previous recommendations for reporting evaluations when suggesting changes for the cameraready version of a paper.

CONCLUSION

Motivated by a lack of systematic surveys on evaluation methodologies for technologies for PVI, we set out to review a corpus of 178 papers on assistive HCI published over the past three decades. We analysed types and forms of quantitative empirical evaluations used, sample representation, and reported difficulties in conducting evaluations in this space. Our findings confirm there is an implicit standard (a novel technique with a summative quantitative evaluation), which co-exists with a wide range of other types of article. We noted concerns about the reporting of evaluations and the representation of PVI in these evaluations. We also report on the difficulties experienced by researchers and discussed alternative and complementary approaches to quantitative empirical evaluations as they are currently run. We frame our findings in the form of recommendations for conducting, reporting, and reviewing empirical evaluations of technologies for PVI, that we hope will help improve evaluation practices in this domain.

Figure 1 .

 1 Figure 1. Number of evaluations in our corpus per year according to their type (formative or summative).

Figure 2 .

 2 Figure 2. Mean number of participants according to the participants' visual status.

  Number of experiments according to the participants visual status and type of experiment (formative or summative).

Among the 243 experiments, only 53 (21.8%) reported (at different levels of precision) the age of onset, 12 (4.9%) reported the causes of blindness, and only 6 (2.4%) reported other

3.

at ASSETS, 6 at TOCHI, and 24 at TACCESS. The publication year ranges from 1988 to 2019, with an increasing number of papers per year. This parallels the overall increase in the number of papers published in these venues, but also includes the higher rate of excluded papers in the 1990s, as quantitative empirical evaluations were less of a standard (Figure1).

Current writing guidelines on writing gender advise against using this word. See: morgan-klaus.com/sigchi-gender-guidelines

ACKNOWLEDGEMENTS

The authors would like to thank the workshop and SIG participants who contributed to map these issues (inclusiveeducation.tech). This work has received funding from the University of Sussex Research and Development Fund, from the French National Research Agency (AccessiMap ANR-14-CE17-0018) and the EPSRC (CRITICAL Fellowship Project, EP/N00616X/2).