Damien Masson
email: dmasson@uwaterloo.ca

Sylvain Malacria
email: sylvain.malacria@inria.fr

Géry Casiez
email: gery.casiez@univ-lille.fr

Daniel Vogel
email: dvogel@uwaterloo.ca

Statslator: Interactive Translation of NHST and Estimation Statistics Reporting Styles in Scientific Documents

Keywords: centered computing → Interactive systems and tools;, Mathematics of computing → Probability and statistics statistics, interactive system, reading interface, estimation, nhst, transparent statistics, explorable explanation

An independent Student's t-test did not show that the overall effect of gesture compared to tap on WPM was significant (t(36) = 1.26, p = 0.21). It did show that the effect on CER and KSPC was significant (t(36) = 2.12, p = 0.04 and t(36) = 15.77, p < 0.001).

ABSTRACT

Inferential statistics are typically reported using p-values (NHST) or confidence intervals on effect sizes (estimation). This is done using a range of styles, but some readers have preferences about how statistics should be presented and others have limited familiarity with alternatives. We propose a system to interactively translate statistical reporting styles in existing documents, allowing readers to switch between interval estimates, p-values, and standardized effect sizes, all using textual and graphical reports that are dynamic and user customizable. Forty years of CHI papers are examined. Using only the information reported in scientific documents, equations are derived and validated on simulated datasets to show that conversions between p-values and confidence intervals are accurate. The system helps readers interpret statistics in a familiar style, compare reports that use different styles, and even validate the correctness of reports. Code and data: https://osf.io/x4ue7

INTRODUCTION

When studying a population, a common approach is to collect data from a random sample of that population (e.g., participants) and use inferential statistics to generalize the findings. A detailed report of this analysis in scientific documents enables readers to evaluate the strength of the findings. However, the presentation of these reports as imposed by authors might be hard to understand [START_REF] Cumming | The New Statistics: Why and How[END_REF][START_REF] Krzywinski | Error Bars[END_REF], incomplete [START_REF] Sullivan | Using Effect Size-or Why the P Value Is Not Enough[END_REF], even misleading [START_REF] Belia | Researchers Misunderstand Confidence Intervals and Standard Error Bars[END_REF][START_REF] Goodman | A Dirty Dozen: Twelve P-Value Misconceptions[END_REF][START_REF] Lecoutre | Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests[END_REF].

Within the HCI community and other empirical fields, statistical reports commonly include p-values obtained from null hypothesis significance testing (NHST). However, p-values and their associated reporting language might mislead even trained scientists [START_REF] Lecoutre | Even statisticians are not immune to misinterpretations of Null Hypothesis Significance Tests[END_REF][START_REF] Mcshane | Blinding Us to the Obvious? The Effect of Statistical Training on the Evaluation of Evidence[END_REF], and reporting p-values alone is insufficient to draw meaningful and nuanced conclusions, capture uncertainty of results, and answer quantitative questions about the effect [START_REF] Cumming | Introduction to the New Statistics: Estimation[END_REF][START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF][START_REF] Goodman | A Dirty Dozen: Twelve P-Value Misconceptions[END_REF][START_REF] Lakens | The Practical Alternative to the P-Value Is the Correctly Used p-Value[END_REF]. For these reasons, some recommend reporting additional information such as effect sizes and interval estimates [START_REF]Publication Manual of the American Psychological Association[END_REF][START_REF] Sullivan | Using Effect Size-or Why the P Value Is Not Enough[END_REF], while others advocate for avoiding p-values and NHST altogether [START_REF] Robert | The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known[END_REF][START_REF] Cumming | The New Statistics: Why and How[END_REF]. This debate is particularly visible in HCI where all positions cohabit. P-values are prevalent in publications [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF][START_REF] Vornhagen | Statistical Significance Testing at CHI PLAY: Challenges and Opportunities for More Transparency[END_REF] and more robust NHST approaches are being developed [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF][START_REF] Wobbrock | The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures[END_REF]. Yet, workshops and articles are also promoting estimation [START_REF] Robert | The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known[END_REF][START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF][START_REF]Transparent Statistics Guidelines[END_REF] and bayesian approaches [START_REF] Kay | Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI[END_REF][START_REF] Phelan | Some Prior(s) Experience Necessary: Templates for Getting Started With Bayesian Analysis[END_REF].

While much of the ongoing debate is about what to report, it also extends to how inferential statistics should be reported. For example, what symbols to use [5, sec 6.44], what words to communicate findings [START_REF] Besançon | Definitely Maybe: Hedges And Boosters in the HCI Literature[END_REF][START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF], and what numeric precision to use [START_REF] Boos | P-Value Precision and Reproducibility[END_REF]. How authors report statistics continuously evolves as the field matures: for example, the practise of reporting p-values using inequalities has noticeably decreased since 2010 [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF]. New reporting styles like graphical presentations of effect sizes can effectively convey the information [START_REF] Robert | The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known[END_REF][START_REF]Transparent Statistics Guidelines[END_REF][START_REF] Helske | Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect[END_REF], but such figures are seldom found in documents because they can be difficult to create [START_REF] Cleveland | The Elements of Graphing Data[END_REF][START_REF] Kastellec | Using Graphs Instead of Tables in Political Science[END_REF] and take additional space [START_REF] Altman | Statistics Notes: Presentation of Numerical Data[END_REF][START_REF]Transparent Statistics Guidelines[END_REF]. Even graphical representations can be misunderstood, for example, error bars may depict standard deviations, standard errors, and confidence intervals [START_REF] Cumming | Error Bars in Experimental Biology[END_REF]. At best, the meaning of error bars is clear and a reader can recover a quantity of interest by "eyeballing it" [21, ch 5]. At worst, a reader can form the wrong conclusion about the depicted results [START_REF] Krzywinski | Error Bars[END_REF].

It is virtually impossible to please all readers with one style of report for inferential analysis. For this reason, some recommend including multiple reporting styles [5, sec 3.7], or even multiple alternative analysis ("multiverse analysis") [START_REF] Hall | A Survey of Tasks and Visualizations in Multiverse Analysis Reports[END_REF][START_REF] Steegen | Increasing Transparency Through a Multiverse Analysis[END_REF] which can be made interactive to give readers control over the presentation [START_REF] Dragicevic | Increasing the Transparency of Research Papers with Explorable Multiverse Analyses[END_REF]. However, they are difficult to create [START_REF] Gu | Understanding and Supporting Debugging Workflows in Multiverse Analysis[END_REF][START_REF] Liu | Boba: Authoring and Visualizing Multiverse Analyses[END_REF][START_REF] Sarma | Multiverse: Multiplexing Alternative Data Analyses in R Notebooks[END_REF] and often incompatible with publishing workflows that impose page limits and digital formats with no support for interactivity [START_REF] Masson | Chameleon: Bringing Interactivity to Static Digital Documents[END_REF]. Even in the unlikely case that all authors adopt reporting styles akin to interactive multiverse analyses, it is unclear how previously published documents could be supported, especially when their raw data is unavailable.

We argue for reader-centred statistics: readers should have the final word on statistical inference presentation because what matters is correct interpretation, be it through estimation or NHST 1 , textual or graphical content, or static or interactive documents. Moreover, this should be possible with old and new documents without substantial efforts. Our key insight is that most scientific articles report enough information about at least one type of inferential statistical analysis which can be translated into a different reporting style. For example, a confidence interval (CI) can be calculated given a p-value and means. The difficulty lies in three aspects:

(1) obtaining the statistical information-we analyzed statistical reports at CHI and propose a semi-automatic pipeline to extract common statistical reports; (2) converting the information into a target statistic-we demonstrate how to do these conversions and thoroughly evaluate their accuracy given common reporting practices; and (3) presenting the converted information back to readers 1 Some argue p-values cannot be properly interpreted even when their pitfalls are understood [START_REF] Robert | The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known[END_REF]. We believe readers should still decide if they wish to use p-values.

in their preferred style-we design a document reader that allows readers to customize textual, graphical, and interactive statistical reports. We implement our solutions to these aspects into Statslator, a tool for readers to retro-actively encode best practises in terms of statistical reporting with no author involvement. This supports documents that were already disseminated and bypasses limitations due to outdated publishing practises that discourage the use of figures because of page limits and prevent interactive documents due to reliance on the PDF format or inflexible publishing workflows.

BACKGROUND AND RELATED WORK

We first review the challenges associated with different ways of reporting statistics and then detail recommendations made by the community to tackle these challenges. We then detail how previous systems allowed readers to personalize documents without involving authors.

Challenges with Statistical Reporting

Prevalent practices in scientific documents can make statistical reports hard to understand.

First, there is much confusion about the meaning of statistical values. For example, the potential for misinterpretating p-values [START_REF] Cumming | Introduction to the New Statistics: Estimation[END_REF][START_REF] Goodman | A Dirty Dozen: Twelve P-Value Misconceptions[END_REF]: a p-value might lead a reader to believe the result is more certain than it is (dichotomous thinking); that it conveys effect size (ambiguous use of "significant"); and that there is no difference when p>.05 or that p is the probability that the null hypothesis is true (inverse probability fallacy). But misinterpretations are not limited to p-values. Any statistical report not fully understood by readers might be misinterpreted, including standard errors, confidence intervals [START_REF] Belia | Researchers Misunderstand Confidence Intervals and Standard Error Bars[END_REF], and effect sizes [START_REF] Kim | Putting Scientific Results in Perspective: Improving the Communication of Standardized Effect Sizes[END_REF]. Thus, it is important to give readers access to statistics they are familiar with, or default to ones that are easily understood.

Second, there are many equivalent ways to report the same statistical values, all requiring a different interpretation. Consider two groups 𝑀 1 = 5, 𝑆𝐷 = 1 and 𝑀 2 = 7, 𝑆𝐷 2 = 3. Now consider the multitude of valid ways to convey the effect size (calculated value in parenthesis): the mean difference (2), Cohen's d (.89), Glass' 𝛿 (.67), the rank-biserial correlation (.41), odds ratio (5.1), 𝜂 2 (.17), Cohen's f (.45), and the common language effect size (.74). Worse, there is a lack of consensus on how to calculate some specific effect sizes such as Cohen's d and all variations are found in documents, with no indication of what formula was used (like we just did) [START_REF] Lakens | Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs[END_REF]. Similarly, a confidence interval can be reported at various confidence level, and for different estimates [21, p 118]. These inconsistencies defeat most of the purpose of standardized effect sizes to convey an effect size that is comparable across studies.

Third, there are many ways to report and represent statistical values. In textual reports, values are often introduced with letters and Greek symbols [5, sec 6.44]. Besides being confusing, these symbols are inconsistent across documents and sometimes collide. For example, 𝑑 can refer to any of the four ways of calculating Cohen's d [START_REF] Lakens | Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs[END_REF]. Similarly, 𝑟 is the symbol for both the rank-biseral correlation and the Pearson correlation coefficient (which are equivalent only in specific situations). The reported values themselves can be rounded with arbitrary precision, or they can be given as inequalities, often the case for p-values [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF][START_REF] Boos | P-Value Precision and Reproducibility[END_REF]. Similar inconsistencies are in graphical reports of statistics. There is a wide diversity of chart types, and much has been written about how charts can deceive or confuse an inattentive reader [START_REF] Cairo | How Charts Lie: Getting Smarter about Visual Information[END_REF][START_REF] Vikram Pandey | How Deceptive Are Deceptive Visualizations? An Empirical Analysis of Common Distortion Techniques[END_REF]. Additionally, standard graphical marks may not have consistent meanings. For example, the visual style of an "error bar" is easily recognizable, but whether they represent standard error, standard deviation or a confidence interval may be misinterpreted or unknown [START_REF] Belia | Researchers Misunderstand Confidence Intervals and Standard Error Bars[END_REF][START_REF] Cumming | Error Bars in Experimental Biology[END_REF][START_REF] Hofman | How Visualizing Inferential Uncertainty Can Mislead Readers About Treatment Effects in Scientific Results[END_REF][START_REF] Krzywinski | Error Bars[END_REF]. Even the simple bars or points could be representing separate means or mean differences [START_REF] Cumming | The New Statistics: Why and How[END_REF]. With adequate detail in a caption, these kinds of issues can be mitigated and some recommend graphical reports instead of text [START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF][START_REF] Gelman | Let's Practice What We Preach[END_REF][START_REF]Transparent Statistics Guidelines[END_REF][START_REF] Helske | Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect[END_REF]. But, authors are reluctant to include many graphs since they can be difficult and time consuming to create and take up additional space [START_REF] Altman | Statistics Notes: Presentation of Numerical Data[END_REF][START_REF] Cleveland | The Elements of Graphing Data[END_REF][START_REF] Kastellec | Using Graphs Instead of Tables in Political Science[END_REF], which is unwanted for publications that limit page size or for peer review guidelines that equate contribution size to paper length.

Statistical reports are already difficult to read and understand for readers, and the diversity and inconsistency in how statistics are reported only makes it more challenging. Our approach enables a customizable presentation of statistical results controlled by readers, with embedded context, consistent calculations, and connections across reporting styles.

Recommendations to Report Statistics

The primary response to challenges with statistical reporting is to encourage authors to present results using a consistent, detailed, and clear style. For example, the American Psychological Association (APA) states that "complete reporting of all tested hypotheses and estimates of appropriate effect sizes and confidence intervals are the minimum expectations for all APA journals" [5, sec 3.7]. To make these reports consistent, APA also recommends specific phrasings and symbols. Within the HCI community, similar recommendations have been proposed [START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF][START_REF]Transparent Statistics Guidelines[END_REF], and tools have been created to assist authors in the process [START_REF] Jun | Tea: A High-level Language and Runtime System for Automating Statistical Analysis[END_REF][START_REF] Wacharamanotham | Statsplorer: Guiding Novices in Statistical Analysis[END_REF]. But these guidelines have yet to become standard practice [START_REF] Giofrè | The Influence of Journal Submission Guidelines on Authors' Reporting of Statistics and Use of Open Research Practices: Five Years Later[END_REF][START_REF] Giofrè | The Influence of Journal Submission Guidelines on Authors' Reporting of Statistics and Use of Open Research Practices[END_REF]: as of 2018, about 15% of CHI papers included confidence interval whereas 50% reported p-values [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF].

In parallel, these recommendations keep evolving. For example, the APA started recommending the use of CIs in its fifth edition following a push from the community [START_REF]What If There Were No Significance Tests[END_REF][START_REF] Kline | Beyond Significance Testing: Reforming Data Analysis Methods in Behavioral Research[END_REF]. Alternative presentations of statistics have been proposed and are often supported by experimental evidence. For example, figures showing effect size and confidence intervals can help readers [START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF][START_REF]Transparent Statistics Guidelines[END_REF][START_REF] Helske | Can Visualization Alleviate Dichotomous Thinking? Effects of Visual Representations on the Cliff Effect[END_REF]. Hypothetical outcome plots [START_REF] Hullman | Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences about Reliability of Variable Ordering[END_REF] (HOPs) rely on animations and help convey uncertainty [START_REF] Kale | Hypothetical Outcome Plots Help Untrained Observers Judge Trends in Ambiguous Data[END_REF]. Multiverse analysis reports [START_REF] Hall | A Survey of Tasks and Visualizations in Multiverse Analysis Reports[END_REF][START_REF] Steegen | Increasing Transparency Through a Multiverse Analysis[END_REF] that might be explorable [START_REF] Dragicevic | Increasing the Transparency of Research Papers with Explorable Multiverse Analyses[END_REF], can highlight how fragile or strong the results are. Analogies and some more natural effect sizes such as the common language effect size are often better understood [START_REF] Kim | Putting Scientific Results in Perspective: Improving the Communication of Standardized Effect Sizes[END_REF].

While recommendations exist, the bottleneck seems to be in their implementation, either because of slow adoption, difficulty of creation, or publishing format limitations. And of course, even if all these issues were to be solved, the problem would remain for existing documents.

Personalized Reading Experience

When authors' adoption of new guidelines is slow or unlikely to happen, a possible solution is to offer tools for readers. Several systems have been proposed that take as input a document and augment it in various ways.

For example, reported measurements such as distances might be difficult to interpret if readers cannot relate to them. Thus, systems have been proposed to automatically generate analogies and relatable explanations and visualizations of the measurements reported in a document [START_REF] Hullman | Improving Comprehension of Measurements Using Concrete Re-expression Strategies[END_REF][START_REF] Kim | Generating Personalized Spatial Analogies for Distances and Areas[END_REF]. Similarly, there is a wealth of research on how to generate visualizations to accompany documents, either to give more contexts while reading by leveraging external databases of relevant information [START_REF] Gao | NewsViews: An Automated Pipeline for Creating Custom Geovisualizations for News[END_REF][START_REF] Hullman | Contextifier: Automatic Generation of Annotated Stock Visualizations[END_REF], to generate visualization in-context for data that is already in the document but scattered in textual tables [START_REF] Sriram | Elastic Documents: Coupling Text and Tables through Contextual Visualizations for Enhanced Document Reading[END_REF], or simply to give readers a way to annotate documents using charts so that they can make sense of numbers in-text [START_REF] Masson | Charagraph: Interactive Generation of Charts for Realtime Annotation of Data-Rich Paragraphs[END_REF].

These approaches have the advantage of being immediately applicable and to all documents, including those already disseminated. In this work, we adapt this approach to the context of statistical reports, and adjust the generated presentations to fit the numerous recommendations made by the community.

WHAT IS REPORTED AT CHI?

We analyzed the proceedings of ACM CHI conferences to better understand what inferential statistics are reported and how they are presented. The goal is to gauge the feasibility of generating different statistical representations from the data reported in text. We examine CHI papers because HCI is a multidisciplinary field and CHI is very large and diverse. Statistical practises likely vary among CHI authors depending on their field, background, and exposure to inferential statistics.

Our analysis focuses on text, not values included within figures or tables, and our approach does not consider complex sentence structures. Therefore, our results should be viewed as lower bounds rather than absolute proportions. The code and data of our analysis is accessible online: https://osf.io/x4ue7/

Corpus of Papers and Analysis

We scraped 9,611 PDFs from 1982 to 20222 . For each paper, text was extracted using a Python port 3 of MuPDF4 and a set of caseinsensitive regular expressions as identified statistical reports:

• One way to identify reported values is to look for numbers preceded by the relationship symbol '=', '<', '>', '≤', '≥', '<<', or '>>'. When found, the number and preceding word or symbol were extracted. • Another way to identify values reported in text without a relationship symbol is to look for names of common descriptive and inferential statistics such as 'mean', 'median', 'standard deviation' and variations such as 'M', 'Mdn', and 'µ'. We only extracted those followed by the verb 'be' in any form such as 'is', 'was', and 'were', and then followed by a number. • To count the mention of confidence intervals, we looked for the terms '%CI', '% CI', and 'confidence intervals'. • To detect statistical tests, we checked for the name of a specific test among a list of 15 popular ones, including 't-test', 'ANOVA' and possible variations such as 'ANCOVA' and 'MANOVA'. The results were harmonized by reviewing the top candidates and grouping the ones referring to identical values. For example, 'mean', 'average', 'M', and 'µ' were all grouped as 'mean'. Statistical tests known under different names were also grouped, such as 'Wilcoxon Rank Sum Test' and 'Mann-Whitney U Test'.

Results

As a sanity check, we compared our results to Besançon and Dragicevic [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF] who also used regular expressions to examine CHI proceedings although their analysis was limited to p-values and CIs from 2010 and 2018. We found 5.9% to 14.2% of CI for 2010-2018, they report "from 6% to 15%". We found 48.4% of p-values, they report "around 50%".

Reported

Values. 6,266 papers (65%) mentioned at least one value. P-values (66%) were most common, followed by standard deviations (39%), means (35%) , F-values (28%), t-scores (19%), and confidence intervals (13%). Standard errors were found in only 2% of papers. Standardized effect sizes were also seldom found, such as Pearson's correlation coefficient (11%) and Cohen's d (5%).

Figure 2 shows the number of times a CI, mean, standard deviation, or p-value, or combination of these values, are reported in a paper. Of particular interest are papers reporting both p-values and means; means and standard deviations; and CI and means. For most of these 2,012 papers reporting p-values and means, we will show translating to a different reporting style is possible (section 4). Although papers reporting only p-values (1,612) cannot be translated to CIs, standardized effect sizes can still be calculated to complete the report (see section 4.3.1). For the remaining 2,642 papers, additional values will be needed. Overall, a large proportion of CHI papers have useful statistical reports, and recall that our conservative analysis likely underestimate actual occurrences and it is possible that values like means can be estimated from figures [START_REF] Jung | ChartSense: Interactive Data Extraction from Chart Images[END_REF][START_REF] Masson | ChartDetective: Easy and Accurate Interactive Data Extraction from Complex Vector Charts[END_REF] 3.2.2 Number of Decimals. Consistent with APA recommendations [5, sec 6.36], standard deviations, means, F-values, t-scores, and CIs were reported with a median of two decimals. The exception was for p-values, which were reported with a median of three decimals. However, 17% of these p-values were reported as inequalities with values '0.05' or '0.01'.

Statistical

Tests. 4,800 papers (50%) mentioned at least one statistical test. Among these papers, t-tests were mentioned in most (61%), followed by ANOVA (49%), Wilcoxon signed-rank test (10%), Mann-Whitney U test (9%), and Chi-squared test (9%). Other tests, such as Friedman, were found in less than 5% of these papers.

CONVERTING STATISTICAL REPORTS

Based on the information reported in papers, we present a set of equations to perform bidirectional conversions between NHSTbased reports and estimation-based reports. To our knowledge, these derivations were not or superficially covered, especially in the context of practices common in HCI studies. For example, some readers might have been taught the "conversion rule" that 95% CI is about twice as large as the standard error; the "overlap rule" that if two independent 95% CIs on the separate mean just touch, p is about 0.01, and no overlap means p<0.01; and the "difference rule" that if p>.05 then the 95% CI on the effect size will extend slightly past 0 [21, p 183]. What is sometimes omitted, however, is how these methods were derived, how precise conversions can be obtained, and when they do not work. For example, the overlap rule for a CI fails when the study follows a within-subject design [21, p 200], and the other two assume large sample sizes. Similarly, a widely cited article in the medical science community described how to convert p-values into CIs [START_REF] Altman | How to Obtain the Confidence Interval from a P Value[END_REF], but it assumes more than 60 participants: if applied to smaller studies typical in HCI [START_REF] Caine | Local Standards for Sample Size at CHI[END_REF], the converted estimations would be overoptimistic.

For all conversions, we presume the size of the groups 𝑁 1 and 𝑁 2 and the study design are known (i.e., within or between-subject) since this information would be reported in any rigorous scientific report. By extension, the degrees of freedom 𝑑 𝑓 for a t-test can be calculated if not already reported. For within designs 𝑑 𝑓 = 𝑁 -1, otherwise 𝑑 𝑓 = 𝑁 1 + 𝑁 2 -2. Unless specified otherwise, equations are valid for all variations of t-test, within and between subject, groups of various sizes, and with equal or unequal variances.

Converting to Confidence Intervals

When comparing two groups, the CI of interest is the CI for the effect size, where the effect size is usually the difference between the two group means. This CI can replace a p-value as it conveys the estimate of effect size and the uncertainty around it.

The CI is calculated as [Δ𝑀 -𝑀𝑜𝐸, Δ𝑀 + 𝑀𝑜𝐸] where Δ𝑀 = 𝑀 2 -𝑀 1 is the difference of the group means, or "unstandardized" effect size, and 𝑀𝑜𝐸 is the margin of error (corresponding to half the CI). Below, we show different ways to calculate 𝑀𝑜𝐸 needed to obtain the CI. Adopting this terminology, the calculation of a t-score can be expressed as follows.

𝑡𝑠𝑐𝑜𝑟𝑒 = 𝑀 2 -𝑀 1 𝐶 𝑣 × 𝐶 𝑠 (1)
Note this formulation is an abstraction since the calculation of 𝐶 𝑣 and 𝐶 𝑠 depends on the choice of t-test and the study design. But by referring to these three components, our equations are compatible with all standard t-tests. As such, the equation above can be rearranged to recover 𝐶 𝑣 ×𝐶 𝑠 given 𝑀 1 and 𝑀 2 and a t-score. The remaining 𝐶 𝑡 term is calculated using the t-distribution for a given degree of freedom (noted 𝑡 (𝑑 𝑓 )). As a result, given a t-score 𝑡𝑠𝑐𝑜𝑟𝑒, the means of both groups 𝑀 1 and 𝑀 2 , the 𝑀𝑜𝐸 at confidence level 𝛼 is calculated as follows

𝑀𝑜𝐸 𝛼 = 𝑡 𝛼 (𝑑 𝑓 ) × 𝑀 2 -𝑀 1 𝑡𝑠𝑐𝑜𝑟𝑒 (2)
A simplification that is often made is to use the normal distribution instead of the t-distribution; a large sample size results in large degrees of freedom, in which case the t-distribution approximates a normal distribution, so knowing 𝑑 𝑓 can be relaxed. For example, for a large 𝑑 𝑓 and a 95% confidence level, 𝑡 .95 ≈ 𝑧 .95 = 1.96 which simplifies to the equation presented by Altman and Bland [START_REF] Altman | How to Obtain the Confidence Interval from a P Value[END_REF]. However, in many HCI studies, the sample size is small, frequently around 12 participants [START_REF] Caine | Local Standards for Sample Size at CHI[END_REF]. In these cases, this assumption leads to narrower "overoptimistic" CIs and should be avoided.

4.1.2 From means and p-value. The p-value of a t-test is measured by the area under the curve of the t-distribution corresponding to the t-score. It refers to the probability of obtaining the t-score or a more extreme one when assuming the null hypothesis is true. Thus, to recover the t-score from a two-tailed p-value, we can use the inverse cumulative distribution function to recover the t-score that gives an area under the curve matching the p-value. As such, only the p-value and a degree of freedom are needed.

Once the t-score is recovered, Equation 2 is used to calculate the CI. One caveat of this approach is when p-values are reported using inequalities, although this practise is declining [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF]. In these cases, the CI will be larger and more likely to cause type 2 errors. Thus, it is often preferable to use the t-score when reported (section 4.1.1).

4.1.3

From independent means and standard deviations. Depending on whether the study design is within or between subject, the group means (and t-test) will be dependent or independent. Given the means and standard deviations of two between subject groups, an independent t-test can be calculated to obtain a t-score. Equation 2can then be used to obtain the CI. Note that the same does not apply to a dependent t-test for a within subject condition because the required standard deviation of the paired differences cannot be estimated from means and standard deviations alone. In this case, one of the solutions above should be used.

4.1.4

From a CI at a different confidence level. Given a confidence interval at confidence level 𝛼0, we can adapt equation 2 to calculate the CI at 𝛼1:

𝑀𝑜𝐸 𝛼1 = 𝑀𝑜𝐸 𝛼0 𝑡 𝛼0 (𝑑 𝑓 ) × 𝑡 𝛼1 (𝑑 𝑓 ) (3)
4.1.5 From CIs on separate means. Given two CIs on means of two independent groups, the CI of the difference can be recovered by first calculating the mean and standard deviation of each group, and then using the method from section 4.1.3. Assuming the t or normal distribution were used, the CI is symmetrical such that its centre is the mean of the sample. For the standard deviation 𝑠𝑑, it can be recovered by rearranging the equation that calculates the 𝑀𝑜𝐸 (half of the CI),

𝑠𝑑 = 𝑀𝑜𝐸 𝛼 𝑡 𝛼 (𝑁 -1) × √ 𝑁 (4)
In case another method such as bootstrapping was used to calculate the CI, the recovered 𝑠𝑑 will be approximate, although it is reasonable to calculate the CI on the difference of means (section 5.5). Note also that, for two dependent groups, the separate CIs are not enough and the t-score or p-value will be needed to recover the standard deviation of the paired differences.

Converting to p-values

Given a CI on the mean difference of two groups, and the means of both groups 𝑀 1 and 𝑀 2 , equation 2 can be rearranged to recover the t-score,

𝑡𝑠𝑐𝑜𝑟𝑒 = 𝑡 𝛼 (𝑑 𝑓 ) 𝑀𝑜𝐸 𝛼 × (𝑀 2 -𝑀 1 ) (5)
Then, the t-score is converted into a two-tailed p-value using the the cumulative distribution function of the t-distribution. Note that this assumes the t distribution was used to calculate the CI. The validity of this conversion with bootstrapped CIs is evaluated in section 5.5.

Converting to Standardized Effect Sizes

Whereas section 4.1 used the mean difference as an "unstandardized" effect size, standardized effect sizes such as Cohen's d might be of interest to readers when comparing results that are on different scales and from different experiments [START_REF] Cumming | The New Statistics: Why and How[END_REF]. Many effect sizes have been proposed, but most can be converted from one to another. Thus, below we show how to obtain Cohen's d, and how to convert it to a different effect size. These equations can be trivially extended to the case where only CIs are available by first converting the CI to a p-value (section 4.2).

From standard deviations.

Cohen's d is calculated by dividing the mean difference by a 'standardizer' which differs depending on the study design. For a between-subject study, the recommended standardizer is the pooled standard deviation [START_REF] Cumming | Introduction to the New Statistics: Estimation[END_REF][START_REF] Lakens | Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs[END_REF]. Given the standard deviations of two groups 𝑆𝐷 1 and 𝑆𝐷 2 ,

𝑑 between = 𝑀 2 -𝑀 1 √ (𝑁 1 -1)𝑆𝐷 2 1 +(𝑁 2 -1)𝑆𝐷 2 2 𝑑 𝑓 (6)
For a within subject design, the standardizer is usually a pooled average of the standard deviations [21, p 204],

𝑑 within = 𝑀 2 -𝑀 1 √ 𝑆𝐷 2 1 +𝑆𝐷 2 2 2 (7)
4.3.2 From a t-score (or p-value). Cohen's d can be obtained from just a t-score (and, by extension, a p-value). We report the equations obtained from Daniël Lakens [START_REF] Lakens | Calculating and Reporting Effect Sizes to Facilitate Cumulative Science: A Practical Primer for t-Tests and ANOVAs[END_REF]:

𝑑 between = 𝑡𝑠𝑐𝑜𝑟𝑒 √ 1 𝑁 1 + 1 𝑁 2 (8)
𝑑 within = 𝑡𝑠𝑐𝑜𝑟𝑒 √ 𝑁 (9)
4.3.3 Converting between effect sizes. The conversions between effect sizes have been well covered by previous work, especially in the context of meta-analyses. From Cohen's d, it is possible to calculate the point-biserial correlation and Hedges' g [START_REF]The Handbook of Research Synthesis and Meta-Analysis[END_REF]; odds ratios [START_REF] Borenstein | Introduction to Meta-Analysis[END_REF]; and the "common language effect size", also referred to as the "probability of superiority" [START_REF] William | Generalizing the Common Language Effect Size Indicator to Bivariate Normal Correlations[END_REF]. If p-values are corrected, then a converted CI will also be adjusted which may not reflect the expected confidence level. Corrected p-values often appear in reports using exploratory contrasts to account for multiple comparisons (e.g., post-hoc analysis after an ANOVA). Methods for estimation-based statistics traditionally avoid this issue by planning the analysis a priori to only focus on a few comparisons (typically no more than the degrees of freedom) [21, p 414], or to adjust CIs using different corrections [START_REF] Jean Dunn | Multiple Comparisons among Means[END_REF]. When the original "uncorrected" CI is desired, it can be obtained either by not relying on the p-value (e.g., using the t-score or standard deviation), or by "unadjusting" the p-value, assuming the original correction method is known. The goal is to recover the uncorrected p-value 𝑝 given the corrected p-value 𝑝 * . For a Bonferroni correction [START_REF] Rupert | Simultaneous Statistical Inference[END_REF], 𝑝 = 𝑝 * /𝑛 where 𝑛 is the number of pairwise comparisons done. For a Šidák correction [START_REF] Šidák | Rectangular Confidence Regions for the Means of Multivariate Normal Distributions[END_REF], 𝑝 = 1-(1-𝑝 * ) 1/𝑛 . For a Holm-Bonferroni correction [START_REF] Holm | A Simple Sequentially Rejective Multiple Test Procedure[END_REF], 𝑝 = 𝑝 * /(𝑛 -𝑖) where 𝑖 is the position of the p-value in the list of sorted p-values of all pairwise comparisons. In some cases, the recovered uncorrected p-values might be inexact: first, a correction for multiplicity typically increases p-values and might make them exceed 1 in which case the p-value is often rounded to 1 and some precision is lost. Second, for Holm-Bonferroni, the rank cannot be recovered when the correction changed the ordering of the p-values. These issues are investigated in detail in section 5.4.

Other Considerations

t-test Variations.

The equations above apply to common variations of t-tests including those for unequal variances, and unequal sample sizes. For example, for Yuen and Welch's t-test, the degrees of freedom will be different, but the equation to convert the p-value to CI will remain the same, given the correct 𝑑 𝑓 is used. Additionally, in some situations, different tests are equivalent to t-tests. For example, the result of an ANOVA on a condition with two-levels will be identical to a t-test, and the t-score is the square root of the F-score [START_REF] Henson | Analysis of Variance (ANOVA)[END_REF][START_REF] Michael | ANOVA. In Understanding Statistics and Experimental Design : How to Not Lie with Statistics[END_REF].

Non-parametric statistics.

Tests such as Wilcoxon signed rank and Mann-Whitney are typically reported when data does not follow the assumptions of a t-test. The equations above cannot be used with these tests and it is unclear how CIs could be recovered without the underlying data to find its distribution, and without the test giving an indication of what that distribution might be.

Conversions With Incorrect

Reports. One might also wonder what the conversion would do in cases where a t-test was applied on data that clearly breaks t-test assumptions. First, it is important to recognize that a t-test might still be a reasonable choice: for example, the central limit theorem states that, for some data distribution, the mean of the data will be normally distributed given a large enough number of samples, and thus, a t-test could be used. This might explain why t-tests are so prevalent in CHI papers (61% of papers that mentioned a statistical test). But more importantly, the equations are meant to convert the results, not to fix them. Little can be done if a statistical report uses the wrong test and obtains potentially erroneous results. 4.4.5 Chaining Equations. Most equations can be rearranged to calculate the measurements they involve. For the sake of brevity, we presented each equation only once. However, we provide an open source JavaScript library with more than 50 equations and possible rearrangements 5 . Given a set of measurements, the library will iteratively calculate all possible values. For a given value of interest, the library can also describe the possible ways to calculate it, and what values would be needed. It can also identify inconsistencies when there are multiple ways of calculating a value, but they yield different results (using a relative error threshold, currently 0.1).

CONVERSION ACCURACY

While the equations in section 4 are exact, written reports often round numbers, may use small samples, and calculate values using methods that could impact the conversion accuracy. Consistent with validation approaches used in the statistics literature [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF][START_REF] Zhu | Assessing and Comparing the Accuracy of Various Bootstrap Methods[END_REF], we conduct Monte-Carlo simulations of common statistical reports. Our three experiments use the conversion equations above to test the accuracy of: (1) converting reports of t-tests to CI; (2) converting reports of post-hoc pairwise comparisons with corrections to CI; and (3) converting reports of confidence intervals calculated via t-distribution and bootstrap methods to p-values.

The simulations are in python using numpy [START_REF] Harris | Array Programming with NumPy[END_REF] and pingouin [95] for statistical tests and distributions. The bootstrapped confidence intervals are calculated using arch [START_REF] Sheppard | bashtage/arch: Release 5[END_REF]. Code: https://osf.io/x4ue7.

Data Generation

As is standard with statistical simulations [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF][START_REF] Zhu | Assessing and Comparing the Accuracy of Various Bootstrap Methods[END_REF], the datasets used in our fictional reports are automatically generated to test a wide range of study designs. All three experiments use generated designs that are prevalent in HCI with the following shared conditions:

• design: Either between-subject or within-subject.

• size: The size of each group. We choose to use 8, 12, 24, or 40 because they are the most frequent sample sizes found in HCI studies [START_REF] Caine | Local Standards for Sample Size at CHI[END_REF]. However, this does not mean we endorse these small sample sizes since they might result in underpowered studies. • decimals: The number of decimals used to round all values (mean, standard deviation, t-score, and bounds of CI). For p-values, the rounding is done using the number of significant digits to more closely match what would be reported in a paper. For example, with 1 decimal, a p=0.048 is rounded to 0.05. Although some experiments might add conditions, they are all performed on at least 4 sizes × 2 designs × 10,000 repetitions. We use 10,000 repetitions as it has been shown to provide precise approximations with designs typical of HCI studies [START_REF] Richard | Within or Between? Comparing Experimental Designs for Virtual Embodiment Studies[END_REF]. The same datasets are re-used when varying decimals. We note cases where experiments add other conditions as appropriate.

Generation Process.

The data generation process follows standard practices [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF][START_REF] Zhu | Assessing and Comparing the Accuracy of Various Bootstrap Methods[END_REF]. For two groups 𝐴 and 𝐵, their samples are randomly drawn from two normal distributions N A (𝜇 𝐴 , 2) and N B (𝜇 𝐵 , 2). When simulating populations with equal means, then 𝜇 𝐴 = 𝜇 𝐵 = 0. Otherwise, when simulating populations with different means, 𝜇 𝐴 ≠ 𝜇 𝐵 and they are randomly drawn from a standard normal distribution N (0, 1).

When simulating within-subject designs, a random intercept unique to each subject is added. For example, for a subject 𝑋 , an intercept 𝛼 𝑋 is calculated and then added to the values of 𝑋 in group A and B. To calculate the intercept, we follow the procedure from Elkin et al. [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF]. The intercept is randomly drawn from N (0, 𝜎 2 ) where 𝜎 2 is randomly chosen to be either 0.1, 0.5 or 0.9 to represent a "reasonable ratio between within-subject variance and between-subject variance" [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF].

To calculate the ground truth statistical power, we calculate ttests using the data generated with full numerical precision. We then measure the proportion of obtained p-values that are inferior to 0.05 when the simulated populations are truly different. Similarly, to calculate the ground truth coverage, we obtain CIs using the t-distribution of the data will full numerical precision. We then measure the proportion of these CIs that include the true difference in population means.

Metrics

We report standard metrics used in statistics literature to validate CIs (coverage probability [START_REF] Sidik | A Simple Confidence Interval for Meta-Analysis[END_REF]) and p-values (type 1 error rate and power [START_REF] Elkin | An Aligned Rank Transform Procedure for Multifactor Contrast Tests[END_REF]). Recall that our goal is to validate the equations presented in section 4. Thus, we consider a conversion as "correct" if it recovers a p-value or CI that yields a similar score with these metrics as the corresponding p-value or CI calculated from the raw data. Below, we clarify the meaning of each metric.

Coverage of CI.

The coverage corresponds to the proportion of calculated CIs that contain the true population mean. For a confidence level of 95%, this proportion should be as close as possible to 95%. This means that over a large number of repetitions, we expect 95% of the 95% CIs to contain the true population mean.

Type I error rate.

Given a significance level 𝛼, the type 1 error rate is the proportion of false positives where a true null hypothesis is rejected (𝑝 < 𝛼). We use a significance level of 0.05 that is common in HCI, so this proportion should be as close as possible to 5%. Over a large number of repetitions, we expect 5% of p-values to be below 0.05 (even though they are type 1 errors).

Power. Statistical power is the proportion of true positives

where a false null hypothesis is correctly rejected (𝑝 < 𝛼). The closer the power is to 100%, the more statistically powerful the test. In practice, power can be much lower, especially when the sample size is small.

Experiment 1: t-test reports to CIs

This first experiment simulates conversions to obtain 95% CIs from reports comparing two groups with a t-test. In total, 240,000 reports are simulated (4 sizes × 2 designs × decimals × 10,000 repetitions). For each report, the 95% CI is calculated using either t-score and means (section 4.1.1); p-value and means (section 4.1.2); or means and standard deviations (section 4.1.3). As baseline, we report the coverage of the 95% CI calculated using the t-distribution of the sample raw data.

Results

. On average, three decimals is enough, two decimals give reasonable estimates, but one decimal yields narrower and overoptimistic CIs that do not capture 95% coverage. Table 1 shows coverage of CIs depending on conversion method, the study design, and the numeric precision of the values used. Perhaps because rounding errors propagated, "means + stds" produced the least accurate results with a coverage of 93.8% at two decimals and 90.4% at one decimal.

Experiment 2: Corrected p-values to CIs

This experiment simulates reports of multiple pairwise t-test comparisons as would be done post-hoc after an omnibus test, such as ANOVA. It differs from experiment 1 in that one report may contain 3, 6, or 10 comparisons (corresponding to an independent variable with either 3, 4, or 5 levels) and the p-values are adjusted to counteract the multiple comparisons problem. The goal is to evaluate the impact of these corrections on the calculated CI, and test the approximations to "unadjust" them. In total, 2,160,000 reports are simulated (3 comparisons × 3 corrections × 4 sizes × 2 designs × 3 decimals × 10,000 repetitions). The correction applied to the p-values is either Bonferroni [START_REF] Rupert | Simultaneous Statistical Inference[END_REF], Holm-Bonferroni [START_REF] Holm | A Simple Sequentially Rejective Multiple Test Procedure[END_REF] (that we refer to as Holm to avoid confusion), or Šidák [START_REF] Šidák | Rectangular Confidence Regions for the Means of Multivariate Normal Distributions[END_REF]. As baseline comparison, we report coverage for 95% CI calculated using the t-distribution of the sample raw data.

Results

. On average, corrected p-values tend to increase the confidence level of the CI to match a 99% CI. For reasons mentioned in section 4.4.1, reversing a correction is approximate and works best for Šidák. For Holm, the reversed correction results in recovering lower CIs (down to 92% confidence) and recovering the CI from the t-score should be preferred. Table 2 reports the breakdown of coverage for the different corrections and uncorrected p-values.

Experiment 3: CIs to p-values

This experiment simulates reports that include 95% CI and for which we would like to recover p-values. In total, 720,000 reports are simulated (3 CI Method × 4 size × 2 design × 3 decimals × 10,000 repetitions). The CI Method to calculate the 95% CI is either the tdistribution (referred to as t-CI), or popular bootstrapping methods such as percentile CI [START_REF] Efron | The Jackknife, the Bootstrap, and Other Resampling Plans[END_REF] (percentile-CI), the studentized CI [START_REF] Diciccio | Bootstrap Confidence Intervals[END_REF] (studentized-CI), or the bias-corrected and accelerated CI [START_REF] Efron | Better Bootstrap Confidence Intervals[END_REF] (BCa-CI). Bootstrapping methods use 2,000 resamples. As baseline, the results for the p-value obtained from appropriate Student's t-tests are reported.

Results

. Overall, p-values recovered from BCa and percentile bootstrapped CIs tend to inflate the type 1 error rate, but are statistically more powerful. This increased number of type 1 errors might be explained by these methods generating CIs with a coverage that does not match 95%. This finding is consistent with previous work that found the percentile and BCa method to perform poorly given small samples (N<50), whereas at this size the t-distrubtion or the studentized bootstrap is usually best [START_REF] Efron | The Bootstrap and Modern Statistics[END_REF][START_REF] Zhu | Assessing and Comparing the Accuracy of Various Bootstrap Methods[END_REF]. For comparison, our experiments use sample sizes between 8 and 40. P-values recovered from CI calculated from a t-distribution and studentized bootstrap tend to match the p-values that would have been obtained had a t-test been run, even at low one-decimal numeric precision. Table 3 shows the breakdown of type 1 error rate given the different CI methods and number of decimals. Table 4 shows the same breakdown for statistical power. Note that power may appear low, but this is consistent with what is expected, and has been shown before for such small sample sizes [START_REF] Delacre | Why Psychologists Should by Default Use Welch's t-Test Instead of Student's t-Test[END_REF].

STATSLATOR PDF VIEWER

We developed the Statslator PDF viewer for readers to interactively translate between statistical reporting presentation styles in existing documents and generate statistical complementary presentations like effect sizes and graphical charts. The tool can help readers interpret statistical reports. The user interface was designed to make the capabilities of the conversion equations in our library transparent to the reader, so they are aware of the provenance of the data, what calculations are done, and the quality of conversions.

Readers use Statslator to open and view a PDF document, then select content with statistical reporting they wish to translate into a new presentation style or complementary presentation. The new presentations appear in a sidebar, are highly configurable, and present related statistical values to explore and validate, or values that are easier to interpret correctly such as the common language effect size [START_REF] Kim | Putting Scientific Results in Perspective: Improving the Communication of Standardized Effect Sizes[END_REF] and S-values [START_REF] Rafi | Semantic and Cognitive Tools to Aid Statistical Science: Replace Confidence and Significance by Compatibility and Surprise[END_REF]. The reader can also choose from different representations, such as animated hypothetical outcome plots [START_REF] Hullman | Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences about Reliability of Variable Ordering[END_REF] and interactive plots, which, despite being powerful representations, are unlikely to be found in existing documents because of publishing formats and workflows. We describe the tool and its features in more detail using three use case scenarios.

Changing the Style of Statistical Report

Sam got a new smartwatch and decides to review the literature to find the best text entry method for this device. Sam first stumbles across "WatchWriter" [START_REF] Gordon | WatchWriter: Tap and Gesture Typing on a Smartwatch Miniature Keyboard with Statistical Decoding[END_REF], an article describing a keyboard for smartwatches. The article reports a user study that compares two ways of operating the keyboard either through taps or gestures. At first glance, it appears that the gesture version is preferable. The article reports "A one-way between-subjects ANOVA did not show that the overall effect of gesture compared to tap on WPM was significant (F(1, 36) = 1.59, p = 0.21). It did show that the effect on CER and KSPC was significant (F(1, 36) = 4.49, p = 0.04 and F(1, 36) = 248.60, p < 0.001). "6 . However, Sam is not very familiar with NHST, and the article does not report effect sizes making it difficult to know if it is worth investing time to learn the gesture technique. Thus, Sam selects the text mentioning the statistical results and the table that reports the means for each condition.

Automatic Extraction & Verification. After the selection, the panel on the right is updated to display two tables filled with statistical values from the selection or calculated, as well as a figure of the comparison (figure 3). The background of the cells are coloured based on the provenance of the information: green indicates that the value was obtained from the text or entered manually; and blue indicates that the value was converted. Sam decides to verify that the extracted data is correct by hovering over each value to obtain detailed information. When hovering over a value obtained from the text, a tooltip shows the sentence where the value was extracted from and the corresponding value in the PDF is underlined.

Measures per comparisons

When hovering over a value that was converted by our equations, a tooltip shows details of the calculation. The most accurate conversions are prioritized. For example, for the CI the t-score and means are used even though the standard deviations, p-values are also available.

Switching between Dependent Variables. By default, the generated report shows the results for words per minute (WPM). Sam clicks the blue underlined text in the caption of the first table in the panel to change the dependent variable and show the KSPC. Sam realizes that Cohen's d for the comparison of KSPC is quite large (=5.5) compared to the other dependent variables compared. Sam cannot recall the interpretation of Cohen's d, but the tool shows the common language effect size (=.99) that Sam knows to interpret as "when gesture was compared to tap, in 99 of 100 pairs gesture had a lower KSPC than tap. "

Configuration of the Graphical Statistical Report. Similarly, the figure generated by the system is interactive and shows the 95% confidence interval on the mean difference.

Sam knows about the overlapping rule for the 95% CIs of independent means and decides to change the figure by clicking the blue underlined text in the caption from "mean difference" to "means". However, after some tinkering, Sam is not sure anymore of the correct interpretation of a 95% confidence interval. Instead, Sam switches the chart to an animated hypothetical outcome plot [START_REF] Hullman | Hypothetical Outcome Plots Outperform Error Bars and Violin Plots for Inferences about Reliability of Variable Ordering[END_REF] which, after a few seconds, helps Sam gain intuition for the distribution of the data.

Comparing Two Reports

After further inspection, Sam realizes that WatchWriter relies on a statistical decoder which makes it difficult to enter words that are out-of-vocabulary. Instead, Sam investigates two alternative techniques that support OOV: SplitBoard [START_REF] Hong | Split-Board: A Simple Split Soft Keyboard for Wristwatch-sized Touch Screens[END_REF] and Swipeboard [START_REF] Xiang 'anthony' Chen | Swipeboard: A Text Entry Technique for Ultra-Small Interfaces That Supports Novice to Expert Transitions[END_REF]. While both articles present user studies, the two techniques are not compared with each other. Worse, the two studies use different protocols and different study durations, and Sam decides that judgment solely based on the reported means might be misleading. However, both studies include a comparison to ZoomBoard [START_REF] Oney | ZoomBoard: A Diminutive Qwerty Soft Keyboard Using Iterative Zooming for Ultra-Small Devices[END_REF], a third text entry technique. Thus, while keeping in mind that the two studies differ in many ways, Sam decides to calculate the standardized effect sizes for the two techniques compared to ZoomBoard. Sam had already opened the papers in Statslator, so Sam begins by selecting the paragraphs containing the statistical results in both papers.

Helping the System with Missing Values. When extracting data from SplitBoard, Sam notices that links for some values are red indicating that they are not available.

Sam hovers over the missing values in the table to get an indication

of what information would be needed for that value to be calculated.

When hovering over Cohen's d, the system indicates that the means and standard deviation are needed to perform the conversion. However, the SplitBoard paper does not report aggregated means with standard deviations. Sam decides to retrieve these values from the line chart of the WPM using an accurate chart data extraction tool [START_REF] Jung | ChartSense: Interactive Data Extraction from Chart Images[END_REF][START_REF] Masson | ChartDetective: Easy and Accurate Interactive Data Extraction from Complex Vector Charts[END_REF]. For SplitBoard, the t-score was directly obtained enabling the calculation of Cohen's d. Once the missing values are added in the table, the other values are calculated and Sam can review the effect sizes: Cohen's 𝑑 = 0.64 for SwipeBoard, and 𝑑 = 7.27 for SplitBoard.

Checking Correctness

While browsing recent preprints, Sam finds a brand new text entry method that looks promising. As always, Sam starts by selecting the statistical report to display the results in a different style. This time, the p-value and t-score have a red background: the system detected an inconsistency after cross-checking different ways values can be obtained.

It appears that the t-score does not match with the reported pvalues. Sam knows how these mistakes can easily occur when writing papers [START_REF] Liang | Analysis of T-Test Misuses and SPSS Operations in Medical Research Papers[END_REF][START_REF] Michèle | The Validity of the Tool[END_REF] and decides to send an email to the authors to warn them.

Implementation

Statslator is implemented using TypeScript using React [START_REF]React -A JavaScript Library for Building User Interfaces[END_REF] with PrimeReact [START_REF]PrimeReact | React UI Component Library[END_REF] for the interface; PDF.js [START_REF] Mozilla | [END_REF] for the PDF viewer; and ECharts [START_REF] Apache | Apache ECharts. Apache[END_REF] to generate dynamic visualizations. The source code is available 7 . Below, we detail the implementation of the text extraction to recover statistical values.

Extracting Statistical Information.

Upon selecting a text in a document, the text is extracted and sent to OpenAI's GPT-3.5 through the official Chat API8 . GPT-3.5 takes care of extracting the statistical information contained in the selection, even if the text contains tables and complex sentence structures. GPT-3.5 is a large language model based on a transformer architecture [START_REF] Vaswani | Attention Is All You Need[END_REF] and powers ChatGPT. The GPT models are state-of-the-art in many natural language processing tasks, especially for complex sentence structures [START_REF] Radford | Improving Language Understanding by Generative Pre-Training[END_REF]. The task of extracting statistical information is no exception: in our tests, GPT-3.5 outperformed all alternatives. GPT-3.5 requires a hand-crafted prompt that explains the task. We engineered the following prompt through repeated experiments: We use "gpt-3.5-turbo" which has an input limit of about 3,000 words, forbidding long text selections 9For privacy reasons or because it is a paid service, we also support alternatives to GPT-3.5. First, readers can always input the numbers manually. Second, we also provide an extraction algorithm that relies on regular expressions. The algorithm searches for APA symbols such as "M=" to extract statistical values (similar to section 3). We group values based on their order of appearance. For example, the first mentioned mean and standard deviation are grouped as a single condition. Similarly, the first p-value is associated with the first comparison of the two conditions mentioned.

DISCUSSION

Our work highlights three aspects: (1) even though most papers report statistics in a specific way, they usually contain enough information to convert them into a different statistical reporting style; (2) most reporting practices are compatible with accurate conversions; and (3) a PDF viewer that embeds these conversions can enable readers to control the presentation of statistics in existing documents to better understand the results, compare documents, and verify correctness. Before outlining the limitations and future work, it is important to clarify what this work is not.

In no circumstances does this work replace the proper practice of statistics by authors, nor does it weaken arguments for statistical reforms such as using CIs instead of p-values [START_REF] Cumming | The New Statistics: Why and How[END_REF]. The differences between an NHST and estimation approach are more than just the presentation of the results. For example, estimation-based thinking also implies a different way of formulating research questions and drawing conclusions which can hardly be an afterthought [START_REF] Robert | Estimation for Better Inference in Neuroscience[END_REF]. And best practices involve planning studies and following open science procedures [START_REF] Cockburn | HARK No More: On the Preregistration of CHI Experiments[END_REF][START_REF] Cumming | Introduction to the New Statistics: Estimation[END_REF][START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF]. Instead, our work helps readers desiring a different presentation of statistical results, perhaps to draw their own conclusions. If anything, our hope is that showing that conversions are possible will motivate authors to choose the method most appropriate to them and their research questions [START_REF] Lakens | The Practical Alternative to the P-Value Is the Correctly Used p-Value[END_REF] without worrying about possible push-back. As expressed by Andy Cockburn in response to an alt.chi article encouraging authors to use estimation and avoid dichotomous reports: "Sometimes, however, the author would prefer to NOT report dichotomous outcomes (for good reasons), but is compelled to do so by their fear/knowledge that if not included, reviewers will expect it and criticise its absence" [START_REF] Besançon | The Continued Prevalence of Dichotomous Inferences at CHI[END_REF]. With large-scale experiments, it is difficult to consider all cases. In section 3, when quantifying the proportions of each report at CHI, some papers might have fallen through the cracks, either because they use complex sentence structures, or because the reports are done in figures and tables. Our analysis was meant to motivate the feasibility of conversions and have reference proportions to decide the most appropriate input values for the equations.

Limitations

Similarly, there are an infinite number of parameters and study designs that could be tested in the experiment section 5, but we chose to focus on those most prevalent in HCI. This had the sideeffect of steering our experiments and equations towards the use of small samples which have been covered relatively poorly in the past. However, as the sample size increases, some of these considerations are no more relevant considering the central limit theorem [START_REF] Fischer | A History of the Central Limit Theorem: From Classical to Modern Probability Theory[END_REF].

7.1.2

The conversions might fail in some situations. As a corollary of the infinite space of study designs, we cannot guarantee that the conversion will be correct in all cases. Most statistical tests have underlying assumptions that might in fact be violated by the data and yield unexpected results. Generally, our conversion equations are based on the assumption that the authors respected the assumptions of the methods they used. However, it is not uncommon for scientists to use an inappropriate test [START_REF] Liang | Analysis of T-Test Misuses and SPSS Operations in Medical Research Papers[END_REF][START_REF] Wobbrock | The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures[END_REF] or make mistakes [START_REF] Michèle | Statcheck": Automatically Detect Statistical Reporting Inconsistencies to Increase Reproducibility of Meta-Analyses[END_REF][START_REF] Michèle | The Validity of the Tool[END_REF], especially considering the challenges associated with statistics in HCI [START_REF] Kay | Researcher-Centered Design of Statistics: Why Bayesian Statistics Better Fit the Culture and Incentives of HCI[END_REF]. In these cases, the results will most likely be as incorrect as their original presentation.

7.1.3

The text extraction of statistical values might fail. We extract statistical values using a large language model (GPT-3.5). While we found it to work well in our limited tests, we did not formally assess its accuracy. Instead, we focused on making sure readers could verify the extraction was successful through provenance verification features such as highlighting the value in the document and showing the sentence that contained the value on hover.

Future Work

7.2.1 Support for more papers through access to more data. The main hurdle in converting a statistical report is the lack of data. We focused on t-tests and papers reporting means and p-values or t-scores because we found these to be the most common at CHI. Conversion between any statistical report is theoretically possible given access to the raw data. Thus, one extension of our work could be to leverage that data when available. For example, the system could automatically pull data from a repository such as https://osf.io/. However, "Open Data" has still a long way to go in communities such as CHI where less than 1% of papers make data available [START_REF] Abbott | Local Standards for Anonymization Practices in Health, Wellness, Accessibility, and Aging Research at CHI[END_REF][START_REF] Masson | ChartDetective: Easy and Accurate Interactive Data Extraction from Complex Vector Charts[END_REF]. Other times, the data is in the document but buried inside data visualizations. In these cases, tools to "reverse-engineer" data visualizations (e.g. [START_REF] Jung | ChartSense: Interactive Data Extraction from Chart Images[END_REF][START_REF] Masson | ChartDetective: Easy and Accurate Interactive Data Extraction from Complex Vector Charts[END_REF]) could make the system work with a broader set of scientific articles. 7.2.2 Support for other conversions. Future work could support more tests and conversions towards a broader set of reporting styles. Tests such as Wilcoxon signed-rank and Mann-Whitney U would be a natural extension given their prevalence. The challenge is to infer the underlying distribution of the data. When the data is available, this could be done through visual inspection or by finding the best-fitting distribution. Otherwise, the distribution could be inferred from the type of measure. Additionally, effect sizes could be further supported for different tests, including 𝜂 2 for ANOVAs.

And a CI on these effect sizes could be calculated given limited data using approaches such as the noncentrality parameter [START_REF] Cumming | A Primer on the Understanding, Use, and Calculation of Confidence Intervals That Are Based on Central and Noncentral Distributions[END_REF][START_REF] James | Beyond the F Test: Effect Size Confidence Intervals and Tests of Close Fit in the Analysis of Variance and Contrast Analysis[END_REF].

In terms of statistical reports, we focused on NHST and estimation because they are fuelling many debates within the scientific community [START_REF] Robert | The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else Is Known[END_REF][START_REF] Dragicevic | Fair Statistical Communication in HCI[END_REF]. However, other approaches such as Bayesian statistics could be supported. For example, a Bayesian t-test has been proposed and can be calculated given a t-score and a prior [START_REF] Rouder | Bayesian t Tests for Accepting and Rejecting the Null Hypothesis[END_REF]. The prior could be controlled by readers to reflect their optimism and knowledge, similar to what Dragicevic et al. proposed [START_REF] Dragicevic | Increasing the Transparency of Research Papers with Explorable Multiverse Analyses[END_REF].

Statistical linting, meta-analyses, and statistical education.

There are many use cases that could be derived from our system. First, our focus was on readers, but the mechanisms leveraged to detect inconsistencies could power a statistical linter for authors. Similar to statscheck [START_REF] Michèle | Statcheck": Automatically Detect Statistical Reporting Inconsistencies to Increase Reproducibility of Meta-Analyses[END_REF] that detects inconsistencies between the reported p-value and reported test statistic, a statistical linter could leverage our system to cross-check the different ways of obtaining a value. This would allow the detection of serious problems such as using a t-test that does not match the study design.

Similarly, many of our equations could be useful in meta-analyses, especially within fields like HCI that deal with small samples for which typical meta-analyses practices are overoptimistic [START_REF] Altman | How to Obtain the Confidence Interval from a P Value[END_REF][START_REF] Altman | How to Obtain the P Value from a Confidence Interval[END_REF]. Our system could help scientists recover accurate data into a customizable and standardized statistical measure. Finally, we assumed readers have some experience and preferences regarding statistics, but our PDF viewer could also be used as an educational tool. Transitioning between statistical representations is particularly useful to develop an intuition [START_REF] Cumming | Introduction to the New Statistics: Estimation[END_REF][START_REF] Cumming | Designing Software for Cognitive Change: StatPlay and Understanding Statistics[END_REF]. Akin to explorable multiverse analyses reports, some options could be educational to give readers a better grasp of certain concepts [START_REF] Dragicevic | Increasing the Transparency of Research Papers with Explorable Multiverse Analyses[END_REF].

CONCLUSION

While much of the debate around statistics has been focused on how authors should practise and report them, little has been done to support readers and the thousands of documents already published. Through theoretical and empirical evidence, we showed that a majority of CHI papers report enough information to be converted to different statistical reporting styles and that the conversions remain mostly accurate under common reporting practices. We also describe the design and implementation of a PDF viewer to turn existing papers into the statistical reporting style readers prefer. Our hope is to provide an immediate solution to reconciliate readers with statistical reports, all while unburdening authors to let them focus on proper statistical practices.

Figure 1 :

 1 Figure 1: Statslator takes existing statistical reports (a) using NHST or estimation; (b) calculates all possible statistical values using accurate conversion equations; (c) shows the report using graphical and interactive figures configurable by readers.

Figure 2 :

 2 Figure 2: Upset plot showing the number of papers that report different sets of CIs, means, standard deviations, and p-values. Lines connecting dots across rows indicate what values are in the intersecting set.

4. 1 . 1

 11 From means and t-score. Cumming[21, p 163] explains that the calculation of the 𝑀𝑜𝐸 depends on the t component 𝐶 𝑡 = 𝑡 𝛼 (𝑑 𝑓 ), the variability component 𝐶 𝑣 , and the sample size component 𝐶 𝑠 .

4. 4 . 1

 41 Multiple comparisons and corrected p-values.

  Such as effect sizes and p-valuesMeasures per conditionsSuch as means, standard deviations Green indicates values from the document Blue indicates calculated valuesGraphical ReportConfigurable by clicking the text in blue

Figure 3 :

 3 Figure 3: Statslator user interface after selecting a paragraph with statistical information from WatchWriter [42]. The panel on the right shows the statistical measures extracted and calculated, as well as a configurable graphical report.

7. 1 . 1

 11 The analyses might have missed some papers and study designs.

Table 1 :

 1 Mean coverage (and standard deviations) of the 95% CIs calculated from different conversion equations and for different designs, and decimals. The closest the values are to the one obtained from raw data, the better.

			decimals
	Values Used	design
		1	2	3
	t-score + means	within .918 (.27) .947 (.22) .950 (.22)
	p-value + means within .927 (.26) .948 (.22) .950 (.22)
	raw data	within .950 (.22) .950 (.22) .950 (.22)
	t-score + means between .917 (.28) .947 (.22) .950 (.22)
	p-value + means between .927 (.26) .948 (.22) .950 (.22)
	means + stds	between .904 (.29) .938 (.24) .942 (.23)
	raw data	between .950 (.22) .950 (.22) .950 (.22)

Table 2 :

 2 Mean coverage (and standard deviations) of the 95% CIs calculated from p-values adjusted using a different correction. For uncorrected values, the closest the values to the the one obtained from raw data, the better.

			decimals
	correction
		1	2	3
	Bonferroni	.968 (.18) .988 (.11) .990 (.10)
	Šidák	.968 (.18) .988 (.11) .990 (.10)
	Holm	.964 (.19) .984 (.12) .986 (.12)
	Bonferroni uncorrected .938 (.24) .957 (.20) .959 (.20)
	Šidák uncorrected	.931 (.25) .951 (.22) .952 (.21)
	Holm uncorrected	.916 (.28) .925 (.26) .925 (.26)
	raw data	.950 (.22) .950 (.22) .950 (.22)

Table 3 :

 3 Mean type 1 error rate (and standard deviations) of the p-values calculated from the 95% CI obtained using different CI Methods and varying decimals precision. The closest the values to the the one obtained from raw data, the better. Coverage of the 95% CI provided for reference.

				decimals
	ci-method	Coverage
			1	2	3
	t-CI	.948 (.22) .047 (.21) .051 (.22) .051 (.22)
	percentile-CI	.935 (.25) .078 (.27) .083 (.28) .083 (.28)
	studentized-CI .944 (.23) .049 (.22) .055 (.23) .055 (.23)
	bCA-CI	.935 (.25) .075 (.26) .08 (.27) .081 (.27)
	raw data	-	.051 (.22) .051 (.22) .051 (.22)

Table 4 :

 4 Mean power (and standard deviations) of the p-values calculated from the 95% CI obtained using different CI Methods and varying decimals precision. The closest the values to the the one obtained from raw data, the better. Coverage of the 95% CI provided for reference.

				decimals
	ci-method	Coverage
			1	2	3
	t-CI	.948 (.22) .300 (.46) .307 (.46) .307 (.46)
	percentile-CI	.935 (.25) .353 (.48) .36 (.48) .361 (.48)
	studentized-CI .944 (.23) .295 (.46) .306 (.46) .307 (.46)
	bCA-CI	.935 (.25) .349 (.48) .356 (.48) .357 (.48)
	raw data	-	.307 (.46) .307 (.46) .307 (.46)

  <excerpt from the paper> Answer with this JSON structure: {"conditions": [/ * reported numbers that refer to a condition following this JSON format: [<number>, <type> / * example: mean, sd, upper CI * /, <condition >] * /], "comparisons": [/ * reported numbers that refer to a t-test comparison following this JSON format: [<number>, <type> / * example: p-value, t-score * /, < condition1>, <condition2> * /]]}

Note CHI was not held in 1984

https://github.com/pymupdf/PyMuPDF

https://mupdf.com/

http://ns.inria.fr/loki/statslator

The article mentions a one-way ANOVA with two levels which is strictly equivalent to a t-test and thus is supported by our tool.

http://ns.inria.fr/loki/statslator

https://platform.openai.com/docs/api-reference

The limit has been increased with newer versions of the GPT models. Full papers could now be parsed directly.

ACKNOWLEDGMENTS

This work was made possible by NSERC Discovery Grant 2018-05187 and the LAI Réapp.


T Pauporté

L Gollino

N Mercier

Iodoethylammonium (IEA + )-based lead-and iodide-deficient halide perovskites (d-HPs) for solar cells

Keywords: Perovskite solar cells, Lead-and iodide-Deficient perovskite, Iodoethylammonium, MAPbI3, toxicity

Perovskite materials, despite displaying remarkable properties, suffer from the use of toxic lead and poor stability towards moisture and thermal stresses. These strongly hinder their future development. Recently, a new family of 3D perovskites, deficient in lead and iodide, the d-HPs, has been discovered. They involve large organic cations that circumvent the Goldschmidt tolerance factor. In the present study, we report that a new cation, the iodoethylammonium I-(CH2)2-NH3 + (IEA + ), allows the synthesis of d-HP thin films based on the 3D MAPbI3 perovskite. This new compound has been tested for solar cell application. A final power conversion efficiency of 9.54 % was achieved.

INTRODUCTION

Hybrid halide perovskites (PVK) are promising semiconductors studied for numerous applications due to their various interesting optoelectronic properties such as high charge carrier mobility, tunable bandgap, high absorption coefficient, long carriers diffusion lengths and low recombination losses [START_REF] Lee | Efficient Hybrid Solar Cells Based on Meso-Superstructured Organometal Halide Perovskites[END_REF][START_REF] Chung | All-solid-state dye-sensitized solar cells with high efficiency[END_REF][START_REF] Kim | Lead Iodide Perovskite Sensitized All-Solid-State Submicron Thin Film Mesoscopic Solar Cell with Efficiency Exceeding 9%[END_REF][START_REF] Chen | Schottky solar cells based on CsSnI3 thin-films[END_REF][START_REF] Wang | Understanding degradation mechanisms of perovskite solar cells due to electrochemical metallization effect[END_REF][START_REF] Sutton | Bandgap-Tunable Cesium Lead Halide Perovskites with High Thermal Stability for Efficient Solar Cells[END_REF] . Their possible applications encompass notably lasers [START_REF] Leyden | Distributed Feedback Lasers and Light-Emitting Diodes Using 1-Naphthylmethylamnonium Low-Dimensional Perovskite[END_REF] , LEDs [START_REF] Wei | An overview on enhancing the stability of lead halide perovskite quantum dots and their applications in phosphor-converted LEDs[END_REF] , photodetectors 9,10 scintillation [START_REF] Mykhaylyk | Bright and fast scintillation of organolead perovskite MAPbBr 3 at low temperatures[END_REF] and perovskite solar cells (PSCs) [START_REF] Viana | Oxide hole blocking selective contacts in perovskite solar cells[END_REF][START_REF] Zhu | PEAI-Based Interfacial Layer for High-Efficiency and Stable Solar Cells Based on a MACl-Mediated Grown FA0.94MA0.06PbI3 Perovskite[END_REF][START_REF] Zheng | What are Methylammonium and Solvent Fates upon Halide Perovskite Thin-Film Preparation and Thermal Aging?[END_REF][START_REF] Zheng | Controlling the Formation Process of Methylammonium-Free Halide Perovskite Films for a Homogeneous Incorporation of Alkali Metal Cations Beneficial to Solar Cell Performance[END_REF][START_REF] Cao | High-Performance Tin-Lead Mixed-Perovskite Solar Cells with Vertical Compositional Gradient[END_REF][START_REF] Liu | Efficient and Stable FA-Rich Perovskite Photovoltaics: From Material Properties to Device Optimization[END_REF] . PSCs have become more and more attractive and investigated, with nowadays a record power conversion efficiency (PCE) of 25.7% [START_REF] Min | Perovskite solar cells with atomically coherent interlayers on SnO2 electrodes[END_REF] that competes directly with the commercialized silicon photovoltaic solar cells [START_REF] Haase | Laser contact openings for local poly-Si-metal contacts enabling 26.1%-efficient POLO-IBC solar cells[END_REF] . Nevertheless, perovskites faces two major issues: a rather poor stability [START_REF] Yang | Facile fabrication of large-grain CH3NH3PbI3-xBrx films for high-efficiency solar cells via CH3NH3Br-selective Ostwald ripening[END_REF][START_REF] Saliba | Incorporation of rubidium cations into perovskite solar cells improves photovoltaic performance[END_REF] as well as a high toxicity caused by the presence of lead [START_REF] Turren-Cruz | Methylammonium-free, high-performance, and stable perovskite solar cells on a planar architecture[END_REF][START_REF] Abate | Perovskite Solar Cells Go Lead Free[END_REF] . To solve them, one method is to replace lead totally or partially by other elements [START_REF] Zhu | Smooth and Compact FASnI3 Films for Lead-Free Perovskite Solar Cells with over 14% Efficiency[END_REF][START_REF] Lin | All-perovskite tandem solar cells with improved grain surface passivation[END_REF][START_REF] Gollino | Lead-Less Halide Perovskite Solar Cells[END_REF] . In recent years, we reported the d-HPs (Lead -and iodide-Deficient Halide Perovskites), a new family of 3D perovskites with an original structure presenting PbI + vacancies inside the perovskite lattice caused by the insertion of large organic cations that do not respect the Goldschmidt tolerance factor. Two large organic cations have been reported so far: hydroxyethylammonium, HO-(CH2)2-NH3 + (noted HEA + ) [START_REF] Leblanc | Leadand Iodide-Deficient (CH3NH3)PbI3 (d-MAPI): The Bridge between 2D and 3D Hybrid Perovskites[END_REF] and thioethylammonium, HS-(CH2)2-NH3 + (noted TEA + ) [START_REF] Leblanc | Enhanced Stability and Band Gap Tuning of α-[HC(NH2)2]PbI3 Hybrid Perovskite by Large Cation Integration[END_REF] . By inserting these cations, lead-and iodide-deficient MAPbI3 (MAPI) and FAPbI3 (FAPI) have been obtained with 3D corned-shared Pb1-xI3-x network and which display an impressively enhanced stability alongside a reduced toxicity.

In the present work, we unveil the use of the large organic cation iodoethylammonium I-(CH2)2-NH3 + (noted IEA + ) to successfully synthetize the lead-and iodide-deficient MAPbI3, noted d-MAPI-IEA, as thin films. They have been characterized by X-Ray Diffraction (XRD) and UV-Visible absorbance measurements. The resulting d-HP films have been additionally characterized by Scanning Electron Microscopy (SEM) and time-resolved photoluminescence (TRPL) to assess the impact of PbI + units octahedra substitution by IEA + on the optoelectronic and morphological properties of the perovskite layer. Finally, d-MAPI-IEA thin films have been incorporated in operational FTO/c-TiO2/m-Please cite this paper as : T. Pauporté, L. Gollino, N. Mercier, Ethylammonium (IEA+)-based leadand iodide-deficient halide perovskites (d-HPs) for solar cells. Proc. SPIE, 12422 (2023) 124220M. https://doi.org/10.1117/12.2647859 TiO2/PVK/spiro-OMeTAD/Au PSC structure (with c-TiO2 a compact TiO2 layer and m-TiO2 a mesoporous TiO2 layer) to prove the feasibility of such photovoltaic devices and a final efficiency of 9.54 % was achieved.

EXPERIMENTAL SECTION

Fluorine-doped SnO2 (FTO) substrates (TEC 7 from Pilkington) were etched pattern by zinc oxide powder and 10 % HCl solution prior to be cleaned with soap and water. The substrates were subsequently plunged for 20 min in a concentrated 2.2 M NaOH in ethanol/water (10:1 volume ratio) and cleaned in acetone using an ultrasonic bath for 12 min. They were then rinsed with deionized water in an ultrasonic bath for 15 min. [START_REF] Zheng | Control of perovskite film crystallization and growth direction to target homogeneous monolithic structures[END_REF] The substrates were subsequently heated at 500 °C for 15 min. The compact TiO2 electron transporting layer (ETL), noted c-TiO2, was prepared by aerosol spray pyrolysis as detailed in Ref. [START_REF] Zheng | Control of the quality and homogeneity of halide perovskites by mixed-chloride additives upon the film formation process[END_REF] . The anatase TiO2 NR30-D paste (from Greatcell Solar Materials) was diluted in ethanol with a 1:7 w/w ratio. 45 μL of the solution was dropped onto the compact TiO2 layer and spin-coated at 2000 rpm for 15 s. The layer was then dried on a hotplate at 70 °C for at least 10 min and finally heated at 500 °C under an air flux for 30 min, cooled down to 200 °C and removed from the hotplate before being transferred immediately to a N2-filled glovebox for perovskite layer deposition.

The perovskite layers were deposited using a one-step spin-coating method. For the MAPbI3 layer, a perovskite precursor solution with a concentration of 1.35 M was prepared by mixing 214.6 mg of MAI (Greatcell Solar Materials) and 622.3 mg of PbI2 (TCI) in 1 mL of DMSO. The solution, placed in a tightly closed bottle, was stirred for 2 h at 100 °C in a N2-filled glovebox. 40 µL of this solution was placed on top of the substrate before starting the spin-coating in a N2-filled glovebox. The spinning routine used was: 1000 rpm with an acceleration of 200 rpm/s for 10 s followed by a second spinning at 6000 rpm for 30 s with an acceleration of 4000 rpm/s. 100 µL of chlorobenzene was dripped 30 s after the start of the spinning routine at a slow speed. The films were then annealed on a hotplate at 105 °C for 1h inside a N2 filled glovebox.

For the d-MAPI-IEA, a perovskite precursor solution with a concentration of 1.0 M was prepared by mixing 132.5 mg of MAI (Greatcell Solar Materials), 384.2 mg of PbI2 (TCI) and 176.3 mg of (IEA)2PbI4 (n=1 MAPbI3-IEAI precursor) in 1 mL of DMF. The solution, placed in a tightly closed bottle, was stirred for 3 h at 50 °C in a N2-filled glovebox. 40 µL of this solution was placed on the top of the substrate before starting the spin-coating in a N2-filled glovebox. The spinning routine used was: 1000 rpm with an acceleration of 500 rpm/s for 10 s followed by a second spinning at 5000 rpm for 30 s with an acceleration of 1500 rpm/s. 100 µL of chlorobenzene was dripped 15 s after the start of the spinning routine at slow speed. The films were finally annealed at 150 °C for 1 hour. To finish the cells, a Spiro-OMeTAD layer was spincoated and a gold back contact electrode was thermally evaporated as described in Ref. [START_REF] Zheng | Control of the quality and homogeneity of halide perovskites by mixed-chloride additives upon the film formation process[END_REF] .

Perovskite film structure was characterized using a PANanalytical X-Pert high-resolution XRD operated at 40 kV and 45 mA and using the Cu Kα radiation with λ = 1.5406 Å. The film specular absorbance was measured by a Cary 5000 UVvis-NIR spectrophotometer using an integrating sphere. A glass/FTO/c-TiO2/m-TiO2 sample was used for the baseline. The time-resolved photoluminescence (TRPL) measurements were performed under a microscope lens (numerical aperture 0.7). The perovskites layers were spin-coated onto a glass/FTO/c-TiO2/m-TiO2 substrate. The top of the PVK layer was excited by a 470 nm diode laser (Picoquant) filtered by a 488 nm long pass filter. The emission was analyzed for TRPL decay by a PerkinElmer SPCM avalanche photodiode combined with a Picoharp acquisition card (500 ps characteristic time of the total system response function) used with the laser in a pulsed mode at a 10 nW excitation power (pulse duration 70 ps). The solar cells J-V curves were recorded by a Keithley 2410 digital source meter, using a 0.1 V.s -1 voltage scan rate. The solar cells were illuminated with a solar simulator (Abet Technology Sun 2000) filtered to mimic AM 1.5G conditions (100 mW cm -2 ). The illuminated surface was delimited by a black mask with an aperture diameter of 3 mm. The power density was calibrated at 100 mW cm -2 by the use of a reference silicon solar cell.

RESULTS AND DISCUSSION

All the perovskite solar cells depicted here were fabricated using a direct mesoporous architecture consisting of an FTOcoated glass as substrates, a combination c-TiO2 and m-TiO2 layers as electron transporting layer (ETL), perovskite (PVK) as the absorber, Spiro-OMeTAD as hole transporting layer (HTL) and gold as counter-electrode (Figure 1a). The d-HP presently studied consists in a classic 3D MAPbI3 in which we substituted PbI + units by iodoethylammonium Top-view SEM images were obtained to assess the morphology of the d-MAPI-IEA film. The film presents important pinholes (Figure 3b) and PbI2 flakes (white flakes) which are not observed for the MAPbI3 film (Figure 3a). Furthermore, the d-MAPI-IEA film is constituted of rather small grains. These facts agree with the reduced crystallinity and the PbI2 peak observed previously on the XRD patterns (Figure 2). The presence of IEA + thus causes a consequent downgrading of the perovskite morphology that will be shown to greatly hinder the devices efficiency and the potential of this d-HP for photovoltaic applications. The UV-Visible absorbance spectrum of the optimized d-MAPI-IEA film was measured and compared to the reference MAPbI3 (Figure 4a). The two films presented an absorbance edge in the 760-770 nm range. An additional edge at 505 nm was clearly observed after IEA + insertion, causing the d-MAPI-IEA to have a better absorbance below 505 nm. For a similar d-HP, the d-MAPI-HEA, a comparable edge was found at 555 nm and was attributed to the presence of domains with a high substitution level of PbI + by HEA + which cause the perovskite film to become slightly reddish. [START_REF] Leblanc | Leadand Iodide-Deficient (CH3NH3)PbI3 (d-MAPI): The Bridge between 2D and 3D Hybrid Perovskites[END_REF] In the case of d-MAPI-IEA, the same trend is observed. Indeed, as the content of IEA + within the lattice increases, the perovskite layer becomes red (Figure 5). In our case, despite the low substitution level of PbI + by IEA + used, domains with important substitution rate could exist and explain the presence of the absorbance edge at 505 nm. All over the rest of the UV-Visible region, the d-MAPI-IEA displays an inferior absorbance, which counteracts the beneficial aforementioned effect. The derivative absorbance highlights a slight blue-shift of the absorption edge around 760-770 nm, caused by Please cite this paper as : T. Pauporté, L. Gollino, N. Mercier, Ethylammonium (IEA+)-based leadand iodide-deficient halide perovskites (d-HPs) for solar cells. Proc. SPIE, 12422 (2023) 124220M. https://doi.org/10.1117/12.2647859 direct optical transition in the metal-organic perovskite absorber. We confirmed this blue-shift by plotting the Tauc plot of the absorbance spectra (Figure 4b). This allowed us to extract the optical bandgap values of MAPbI3 and d-MAPI-IEA perovskites at 1.60 eV and 1.61 eV, respectively. Such an increase of the bandgap value was expected with the expansion of the unit cell volume caused by IEA + cations insertion within the perovskite lattice and conclude the proof of formation of the d-HP d-MAPI-IEA. We also conducted TRPL measurements on the films deposited on FTO/c-TiO2/m-TiO2 substrates. Decay curves are presented in Figure 6. All the curves were fitted with a triple exponential function in order to obtain the best coefficient R² parameter. The function used and the fitting parameters extracted are gathered in Table 1. This allowed us to extract τfast and τslow, which represent the non-radiative recombination at the ETL/PVK interface (reflecting the speed of charge injection in the oxide transport layer) and the bimolecular recombination in the bulk of the perovskite (reflecting the bulk trap density), respectively. A higher τslow means a better bulk structural quality of the perovskite.

CONCLUSION

In the present paper, we report the successful synthesis of a new d-HP based on the use of the iodoethylammonium (IEA + ) organic cation. Lead -and iodide-deficient d-MAPI-IEA has been fabricated. The use of this perovskite in PSCs has been proven feasible. However, after diverse characterizations, d-MAPI-IEA has been found to display a downgraded structural quality along with a lower quality of the oxide ETL/PVK interface which limit its potential. Nevertheless, we believe that the replacement of the TiO2 oxide layers on which was deposited d-MAPI-IEA by other oxide ETL such as SnO2 or ZnO maybe be useful and revive the interest of this perovskite.

  Please cite this paper as : T. Pauporté, L. Gollino, N. Mercier, Ethylammonium (IEA+)-based leadand iodide-deficient halide perovskites (d-HPs) for solar cells. Proc. SPIE, 12422 (2023) 124220M. https://doi.org/10.1117/12.2647859 (IEA + ) large organic cation. A schematic representation of the crystallographic structure of the classic 3D MAPbI3 and the resulting d-MAPI-IEA is shown in Figure 1b. Both systems were thoroughly investigated and optimized.

Figure 1 :

 1 Figure 1: (a) Exploded schematic view of the perovskite solar cell structure. (b) Schematic representation of classic 3D MAPbI3 and of 3D d-MAPI-IEA perovskite formed after partial substitution of PbI + units by IEA + cations.

Figure 2 :

 2 Figure 2: (a) XRD patterns of MAPbI3 and d-MAPI-IEA perovskite films (FTO peaks are indicated by *. # symbol represents PbI2) (b) Same as (a) zoomed between 28-29°.

Figure 3 :

 3 Figure 3: Top-view SEM of (a) MAPbI3 and (b) d-MAPI-IEA perovskite films. Scale bar: 500 nm.

Figure 4 :

 4 Figure 4: (a) UV-Visible spectra of MAPbI3 and d-MAPI-IEA perovskite films. Inset: the associated derivative absorbance. (b) Tauc Plot associated to the UV-Visible absorbance spectra.

Figure 5 :Figure 6 :

 56 Figure 5: Optical microscope pictures of d-MAPI-IEA films for Pb/MA/IEA molar ratios of (a) 1/0.5/1 (b) 1/0.66/0.66 (c) 1/0.75/0.5 and (d) 1/0.8/0.4 (IEA + content decreases from a) to d))

Table 1 :

 1 Exponential function for TRPL fitting and extracted parameters ExpDec3 : y = y0+A1 exp(-t/τ1 )+ A2 exp(-t/τ2 )+A3 exp(-t/τ3 )

  Once d-MAPI-IEA was proved to be a d-HP and fully characterized as thin films, we explored its potential for an application in PSCs by conducting J-V measurements. d-MAPI-IEA devices reached a maximum power conversion efficiency (PCE) of 9.54 % on reverse scan, with VOC = 1.01 V, JSC = 17.21 mA/cm², FF = 54.80 % and HI = 51.0 % (PCE = 4.67 % in forward scan). As a comparison, its analogue classic MAPbI3 delivered a maximum efficiency of 18.94 %, with VOC = 1.04 V, JSC = 22.29 mA/cm², FF = 81.44 % and a hysteresis of 14.5 % (PCE = 16.20 % in forward scan) (Table 2). Associated J-V curves are shown in Figure 7a. Box charts of statistical J-V parameters values of MAPbI3 and d-MAPI-IEA are displayed in Figure 7b-f. Such efficiencies were expected and can be explained by the less ETL/PVK interface quality as well as less XRD characteristics and morphology. The system also suffers from an important hysteresis. However, this issue is shared by all the d-HPs studied. Despite low PCE values, the feasibility of d-MAPI-IEA-based PSCs has been proved.

Figure 7 :

 7 Figure 7: (a) J-V curves of best MAPbI3 and d-MAPI-IEA devices with measured reverse scan PCEs. (b-f) Box charts of MAPbI3 and d-MAPI-IEA PSCs J-V parameters, PCE and HI.

  In the meantime, a lower τfast value indicates a better charge extraction at ETL/PVK layer, therefore lower interface recombination centers Please cite this paper as : T. Pauporté, L. Gollino, N. Mercier, Ethylammonium (IEA+)-based leadand iodide-deficient halide perovskites (d-HPs) for solar cells. Proc. SPIE, 12422 (2023) 124220M. https://doi.org/10.1117/12.2647859 and defect density. In one hand, d-MAPI-IEA film exibited a higher τfast value than MAPbI3 (6.20 ns vs 0.72 ns) which indicates a lower quality of the TiO2/PVK interface caused by the presence of IEA + cations and implies a higher recombination centres density that will limit the short-circuit current density (JSC) and Fill Factor (FF) of the final PSCs. It also implies a slower electron injection that will aggravate the hysteresis (HI) phenomenon. On the other hand, d-MAPI-IEA exhibits a slightly lower τslow value than MAPbI3 (15.14 ns versus 15.62 ns), emphasizing a slight downgrading of the structural quality of the perovskite caused by IEA + insertion into the lattice. This less quality bulk perovskite is consistent with the morphology observed by SEM and the diffractograms obtained by XRD previously.

Table 2 :

 2 Best MAPbI3 and d-MAPI-IEA cells J-V curve parameters, PCE, and HI. Please cite this paper as : T. Pauporté, L. Gollino, N. Mercier, Ethylammonium (IEA+)-based leadand iodide-deficient halide perovskites (d-HPs) for solar cells. Proc. SPIE, 12422 (2023) 124220M. https://doi.org/10.1117/12.2647859

	Cell	Scan direction	VOC (V)	JSC (mA.cm -2 )	FF (%)	PCE (%)	HI (%)
		Reverse	1.04	22.29	81.44	18.94
	MAPbI3						14.5
		Forward	1.02	22.33	71.29	16.20