S Pinet

& C Zielinski

& S Mathôt

& S Dufau

F.-X Alario

& M Longcamp

Measuring sequences of keystrokes with jsPsych: Reliability of response times and interkeystroke intervals

Keywords: Interkeystroke intervals, Online experiment, Motor sequence

Although the precision and reliability of response time (RT) measurements performed via Web-based interfaces have been evaluated, sequences of keystrokes have never been investigated in detail. Keystrokes often occur in much more rapid succession than RTs, and operating systems may interpret successive or concomitant keyboard events according to both automatic and user-based settings. Sequence keystroke timing could thus be more sensitive than single RTs to noise in online measurements. Here, we quantified the precision and reliability of timing measures performed during sequences of keystrokes. We used the JavaScript jsPsych library to create an experiment involving finger-movement sequences, and ran it online with 633 participants. We manipulated the structure of three keystroke motor sequences, targeting a replication of previous findings regarding both RTs and interkeystroke intervals (IKIs). Our online data accurately reproduced the original results and allowed for a novel assessment of demographic variables such as age and gender. In parallel, we also measured the objective timing accuracy of the jsPsych interface by using specialized hardware and software, showing a constant 60-ms delay for RTs and a 0-ms delay for IKIs across the sequences. The distribution of IKIs revealed quantizing for a majority of participants, most likely due to the sampling frequency of their USB keyboards. Overall, these findings indicate that JsPsych provides good reliability and accuracy in sequence keystroke timings for mental chronometry purposes, through online recordings.

Online research via Web interfaces is becoming increasingly important in the field of cognitive psychology [START_REF] Gosling | Internet research in psychology[END_REF]. Collecting large amounts of data over hundreds of participants in a short amount of time holds the promise of overcoming the statistical power limitations of typical laboratory samples [START_REF] Reips | Standards for Internet-based experimenting[END_REF]. However, online experiments imply a trade-off between what is gained by dramatic increases in sample size and better sampling of the whole population, and what is lost to uncontrolled factors such as distracting environments and diversity in equipment configurations. The latter is acutely relevant when the experiments rely on mental chronometry [START_REF] Posner | Chronometric explorations of mind[END_REF].

A number of studies have evaluated the accuracy and reliability of response time measurements performed through various Web-based interfaces. Such empirical evaluations (reviewed in [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF] include direct comparisons of experimental results from Web and lab implementations [START_REF] Reimers | Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities[END_REF][START_REF] Schubert | ScriptingRT: A software library for collecting response latencies in online studies of cognition[END_REF], attempts to replicate classical experimental effects on response times (RTs) with online measures [START_REF] Crump | Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research[END_REF][START_REF] Enochson | Collecting psycholinguistic response time data using Amazon Mechanical Turk[END_REF][START_REF] Reimers | Task switching across the life span: Effects of age on general and specific switch costs[END_REF], and measures of the timing performance of Web-based testing setups using specialist software or hardware [START_REF] Keller | Timing accuracy of Web experiments: A case study using the WebExp software package[END_REF][START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF][START_REF] Simcox | Collecting response times using Amazon Mechanical Turk and Adobe Flash[END_REF]. In general, these studies have agreed that online RTs Electronic supplementary material The online version of this article (doi:10.3758/s13428-016-0776-3) contains supplementary material, which is available to authorized users. are reliable, if slightly overestimated (in the range of tens of milliseconds; de [START_REF] De Leeuw | Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task[END_REF][START_REF] Reimers | Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities[END_REF][START_REF] Schubert | ScriptingRT: A software library for collecting response latencies in online studies of cognition[END_REF].

The studies above have concerned the chronometry of single keypresses, with RTs being defined as the time elapsed between the onset of a stimulus and the unitary response. In contrast, tasks involving rapid sequences of keystrokes have never been investigated with online setups. Sequences of keystrokes are important for researchers interested in behaviors such as typing, musical performance, motor-sequence learning, serial RT tasks, rhythm production, and so forth. This line of research is especially interested in the structure of sequence programming and how temporal and ordinal forms of information are acquired during learning, and it relates to the general problem of serial order in behavior (for a review, see [START_REF] Rhodes | Learning and production of movement sequences: Behavioral, neurophysiological, and modeling perspectives[END_REF]. Two dependent variables can be derived from sequences of keystrokes: RTs, defined above, and interkeystroke intervals (IKIs), the time elapsing between two successive keystrokes. There are two broad reasons why IKIs might be more sensitive than RTs to the noise induced by online measurements.

First, IKIs are typically much shorter than RTs. For example, they can last a few tens of milliseconds for expert participants in typing studies [START_REF] Rumelhart | Simulating a skilled typist: A study of skilled cognitive-motor performance[END_REF], and less than 200 ms for well-trained participants in serial-RT tasks [START_REF] Nissen | Attentional requirements of learning: Evidence from performance measures[END_REF]. A given level of (unavoidable) chronometric imprecision could have a larger impact on the accuracy of such shorter durations than on longer RTs. Standard keyboards are connected through USB ports sampled at a rate of 125 Hz (i.e., every 8 ms). Such quantization could distort small differences that happen to be near that range. [START_REF] Crump | Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research[END_REF] and [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF] previously highlighted the difficulties inherent to shorter timings.

Second, we were also concerned that operating system (OS) settings, such as accessibility features or keyboard hotkeys, might potentially interfere in a detectable way with the recordings and the expected pattern of results. OSs interpret successive or concomitant keyboard events according to both automatic and user-based settings (multiple key presses interpreted as one, combinations of key presses triggering a particular event, etc.). The impact of this intermediate layer of software on keyboard chronometry is not known.

There are preliminary indications of how reliably the timing of successive responses can be recorded. [START_REF] Simcox | Collecting response times using Amazon Mechanical Turk and Adobe Flash[END_REF] and [START_REF] Keller | Timing accuracy of Web experiments: A case study using the WebExp software package[END_REF] used specialized equipment to generate a stream of keystrokes with a fixed known interval. They measured the recovery of such interval through a Web-based software, showing good timing accuracy. A limitation of this approach is that their manipulation involved fixed intervals and a single button. A crucial and original feature of the present study is the use of three distinct keys from the keyboard, and the generation of variable delays between keys, just as it happens in actual experiments involving sequence production.

To assess the accuracy of IKIs measured online, we adopted a two-fold strategy. First, we measured the timing accuracy of the jsPsych interface using a specialized hardware (Black Box Toolkit) that was modified such that three response switches from the keyboard could be alternatively and variably triggered without human intervention. Second, we ran an actual experiment online and performed a complete quality check of the data using descriptive and inferential statistics.

The experiment we designed involved finger movement sequences for which effects on the recorded RT and the IKIs are well-established (conditions 4 vs. 6 of Rosenbaum, Inhoff, & Gordon, 1984, Exp. 3). In a given block, participants produced two pretrained sequences of three consecutive finger responses. Across the two sequences, two of the responses were identical, whereas the third was different, which generates an uncertainty. In the original study, RTs decreased when the uncertain response occurred later in the sequence. In addition, the IKI preceding the uncertain response was lengthened. These effects of uncertainty on sequence programming and execution have been replicated [START_REF] Rosenbaum | Scheduling and programming of rapid finger sequences: Tests and elaborations of the hierarchical editor model[END_REF].

Stimulus display and response recording were controlled by the JavaScript library jsPsych, combined with HTML and CSS (de Leeuw, 2015). JavaScript offers technical advantages over its alternatives, such as Java or Adobe Flash [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF]: It is natively supported by all modern browsers, such as Firefox or Chrome, and it does not require any installation or updating of browser plugins. This ensures a responsive experimental design and an accurate measurement of RTs, because it is not affected by remote server and network latencies. This configuration has been successfully used to record single responses (e.g., de Leeuw & Motz, 2016, and the other references above).

Hardware assessment of timing accuracy via the Black Box Toolkit

Method Materials and procedure To assess the reliability of the RTs generated by multiple consecutive keystrokes, we resorted to specialized hardware, the Black Box Toolkit (BBTK; Black Box ToolKit Ltd, Sheffield, UK), which can automatically generate and record triggers with submillisecond precision. Three mechanical keys (corresponding to the letters S, D, and F) of a Dell standard USB keyboard were wired to the Black Box Toolkit. With this wiring, the BBTK was able to close the three key switches on demand and generate keyboard response sequences. In our tests, the BBTK was programmed to detect a visual stimulus through its optodetector and then automatically generate a sequence of three responses. To handle the display of visual stimuli and the keyboard response collection, we used a jsPsych procedure similar to that used in the actual experimental task (described in the next section and available online: https://github. com/blri/Online_experiments_jsPsych). This procedure was run on an iMac 27 using the Safari Web browser (version 9.0). Three tests were run, all of which consisted in displaying a white @ character on a black background 40 times, responded to with three keystrokes, thus generating 120 automated keyboard responses. The key identities activated by the BBTK were randomized across and within trials. The programmed RTs in the first test were randomly chosen within the interval [100, 250] ms. In the second test, they were fixed at 150 ms. In the third test, the range was [350,500] ms. The data are available at the following repository: https://osf.io/r5dfg/.

Results

The programmed keystroke times were compared to those recorded through the jsPsych procedure. In each of the three tests, the recorded keystroke times (from stimulus to actual keystroke) for each of the three keystroke conditions (first, second, and third keystrokes) were all overestimated by about 60 ms (SD = 8 ms; Fig. 1, upper panel). This observation was substantiated by an analysis of variance (ANOVA) with the crossed factors Keystroke Position and Test Block, in which none of the tests was significant (all ps > .1). When IKIs were computed (second minus first and third minus second keystroke times), the difference between the programmed and recorded values had a mean value of 0 ms (SD = 10 ms; Fig. 1, lower panel). Again, an ANOVA showed no significant effect of position nor of test block (all ps > .1). In short, the first keystroke was recorded with a delay of ~60 ms (probably due to the computation of the stimuli and the physics of LCD screens),1 and this delay was passed on reliably to subsequent keystrokes. Our measure of primary interest, IKIs, was unbiased, showing a mean error of 0 ms.

The variability in the difference between the programmed and recorded values, SD = 8 ms, was in the range of the sampling frequency of the USB keyboard (frequency of 125 Hz, thus a period of 8 ms). This prompted a final analysis to assess the likely quantization of the measure. To show this, we divided each IKI by 8 and took the remainder of the division. If sampling was somewhat biased toward 8, the distribution of the remainder values would not be homogeneous. On the contrary, if sampling happened every millisecond, the distribution should be homogeneous. The vast majority of IKIs appeared to be multiples of 8, which indeed corresponding to the sampling of the USB connection at 125 Hz (see Fig. 2). This quantizing cannot be related to the display of the visual stimulus, since only the onset of the stimulus, and therefore the activation of the opto-detector, depended on the screen parameters.

Discussion

The automatic assessment of sequence keystroke timings generated by the BBTK and recorded by a jsPsych procedure revealed that IKIs are unbiased (mean deviation 0 ms) but largely quantized (at 125 Hz, the USB port sampling rate). We now examined how these objective recording conditions fared when attempting to replicate a classic effect on human sequence production.

Online experiment

The jsPsych platform was used to collect data from a large sample of participants. To evaluate the reliability of the jsPysch platform to accurately measure the timing of sequences of keystrokes, we designed a task involving rapid sequences of keystrokes. The overall duration of the task had to be short (less that 20 min), paired with easy-tounderstand instructions to be adapted to an online format. Therefore, we aimed to reproduce two classic effects on motor sequence performance with a design adapted from Experiment 3, conditions 4 versus 6, of [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]. From the original design, we selected only two conditions, to show differences between RTs and IKIs related to the structure of the sequence performed while keeping the duration of the experiment as short as possible. The conditions were the most different in terms of the sequences used (different hands and fingers; see below) and yielded the largest difference in the RT and IKI measures. The original study explored the motor representations used to perform sequences in a choice-reaction-time task design, in which participants had to select a sequence of motor responses to a visual stimulus (X or O). In the original experiment, the sequences had three finger responses that differed by one element placed at either Position 2 or Position 3. The variable element involved both a change of hand and the choice of a nonhomologous finger (e.g., R-Ring to L-Index). [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF] found that the position of the uncertain response had an effect on RTs, with longer RTs for uncertain responses at Position 2 than at Position 3 (with means of about 460 and 380 ms, respectively; data from [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]. In addition, the IKI preceding the uncertain response was found to be longer, as was attested by a significant interaction between the position of the uncertainty and the position of the required response (with mean IKI1 uncertain = 193 ms; mean IKI1 certain = 177 ms; mean IKI2 uncertain = 233 ms; mean IKI2 certain = 163 ms; data from [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]. The original experiment involved six participants.

Method

Participants Members of our university staff were invited by e-mail to participate in the online experiment. After 31 days, we had collected data from over 600 participants (100 times more than in the original study), and data collection was discontinued. Ninety-two participants were excluded, mainly on the basis of self-reported technical or concentration issues, leaving a usable sample of 541 participants. Reported issues ranged from interruption during the task (17), difficulty understanding the instructions or staying focused (20), discomfort during the task (13), troubles with their specific devices (e.g., keyboard; 11), and various other issues (26). Any participant who reported an issue (whether specific or not) or was below 18 years of age was excluded from the final participant pool. Table 1 outlines the demographics of the final sample. We also collected information about the OSs and Web browsers used (Table 2). The most frequent combination was Windows and Firefox.

Stimuli and design

The design is summarized in Table 3. The two constant sequences were unimanual and consisted of the sequence [Index, Ring, Middle fingers], performed with either the left or the right hand (factor Hand). Four varying sequences comprised a variation from the constant sequences on either the second or the third position (factor Uncertainty). A block contained two interleaved sequences, one constant and one varying. Each participant was assigned randomly to one constant sequence (thus, to one level of the factor Hand, either IRM or irm; see Table 3) and performed the two uncertainty conditions (2nd and 3rd) for his or her assigned hand in consecutive blocks. The association of a given sequence with a visual stimulus (X or O) and the order of presentation of the uncertainty conditions were randomized across participants.

Procedure The online experiment consisted of a set of HTML, JavaScript, CSS, and PHP files. These files were stored on a server that was already set up and managed by S.M. (co-author of this study), and associated with the cogsci.nl domain name. The collected data were saved in the mySQL database that was configured on this server machine.

The experiment itself was mainly developed using the open-source jsPsych library (version 4.3, 2015; de Leeuw, ). This library contains predefined methods to manage experimental timelines, collect RTs and user actions (mouse or keyboard events), randomize stimuli, store the data, and prepare data for the backup. A plugin was devoted to the main task of the study, involving the typing of a three-key sequence following a visual stimulus (i.e., a character appearing on the screen). This plugin, called jspsych-key-sequence, allowed for recording the typing times of each pressed key through an adaptation of an available jsPsych plugin (jspsych-multistim-multi-response). Collection of each keystroke timing relied on the jsPsych method getKeyboardResponse(), which uses the general JavaScript object Date(). The visual stimulus, the corresponding expected typing sequences, the total number of stimulus, and the interstimulus duration were configurable as input parameters and were adapted for the blackbox testing. When given several types of stimuli and the associated sequences, the stimuli were first randomly shuffled to be displayed successively. The code for this experiment is available in the following GitHub repository: https://github. com/blri/Online_experiments_jsPsych. The online experiment started with a welcome screen containing concise explanations. Within this Web page, a push button only visible to a computer-based browser allowed the participant to go further; in this way, participants using tablets or smartphones were automatically excluded from the experiment. A second Web page was then loaded in a full-size window, which embedded the JavaScripts (jsPsych core script and plugins, jQuery) necessary to launch the experiment.

The experiment was divided into two parts, one for each uncertainty condition; each part comprised a training phase and a test phase. Written instructions described the task and introduced the pairing between visual stimuli (X and O) and sequences for each part. During the test phase, one of the two visual stimuli was displayed and stayed on until the participant hit three keys, followed by an interval of 500 ms. The test phase comprised two blocks of 20 trials each, in which ten trials of each sequence were intermixed. Participants were familiarized with the sequences during a training phase, which ended when each sequence was correctly performed four times. Before the test, the plugin was set in a training mode that could be switched on from the input settings, and allowed to provide feedback by changing the color of the stimulus before it disappeared (the initially black symbol became green if the key-sequence was correct, red otherwise). After the experiment, participants were asked to answer a few questions (handedness, gender, age, employed or not employed by the university) and had the opportunity to report whether any problem occurred during the experiment. No monetary compensation was offered.

Before launching the large-scale study, the online experiment was pretested on a small number of participants (N = 54), to assess the clarity of the instructions and the functioning of the program in various material configurations. A spy library was included in the experiment file, sending us the JavaScript error message encountered by any user 's browser (https://github.com/posabsolute/jQuery-Error-Handler-Plugin). Five errors were reported due to four JavaScript methods that were not interpreted by old browser versions. To avoid subsequent errors, a polyfill was added to the experiment file (see the code in the dedicated GitHub repository).

The data related to the experimental task comprised strings of sequences and the associated typing times and survey answers. They were recorded as a JavaScript object during the experiment, which was subsequently transformed to plain text before being sent to the server. At the end of each experiment, the data were transferred to the server as character arrays, including the participant's anonymous identifier, OS/browser information, and the jsPsych data. The transferred dataset was finally stored in the MySQL database with the help of the PHP files. Access to the database and the writing of the data inside the data table were done using the PHP Data Object extension, which ensured database portability and security.

All data are available at the following repository: https://osf.io/r5dfg/.

Statistical analyses

To replicate the original study, the data were first assessed via ANOVAs performed on the RTs and IKIs averaged per participant and cell design. Then, mixed linear regressions were used to estimate the actual effect sizes, and also the effects of additional variables: trial number, gender, age, handedness, and OS and Web browser. These variables were tested as linear predictors, except for age. The relation between age and performance has been reported as nonlinear [START_REF] Baltes | Emergence of a powerful connection between sensory and cognitive functions across the adult life span: A new window to the study of cognitive aging[END_REF], and spline [START_REF] Fozard | Age differences and changes in reaction time: The Baltimore Longitudinal Study of Aging[END_REF]. Visual inspection of our data suggested that the effects of age on performance (RTs) could be nonlinear. Therefore, we used restricted cubic splines with three knots, which allowed the effect to be modeled separately in two intervals, without a priori knowledge of the point of separation between these intervals. In the model, we included random intercepts for participants and items (i.e., the different finger sequences). Since how to compute p values in this kind of analysis has been debated [START_REF] Bates | Fitting linear mixed-effects models using lme4[END_REF], we took t values to approximate z values, and considered any value above 1.96 significant.

To characterize the data reliability and the added value of increasing the number of observations, we calculated the means and confidence intervals of RTs and IKIs over random samples of increasing size, then ran the same regression models on samples of increasing size.

Finally, on the basis of the results of the BBTK assessment, we searched for quantization in our data. We focused mainly on the sampling bias that could result from USB keyboards' sampling rate (125 Hz; i.e., a value being sampled every 8 ms), and used the same methodology as reported above.

Results

Replication of original study

RT and IKI distributions Twelve out of the sample of 541 participants were excluded because they did not reach 85% accuracy on the task, leaving a final sample of 529 participants. Only correct trials were included in the following analysis. We also excluded trials in which any IKI was equal to zero, since the order of key pressing then could not be determined.

Figure 3 presents the RT and IKI distributions of correct responses, which were right-skewed as expected (RT skewness = 117.1, IKI skewness = 5.46). The shape of the IKI distribution was also consistent with the typically described shapes of IKI distributions from laboratory studies of typing, which are less right-skewed than typical RT distributions (e.g., [START_REF] Gentner | The acquisition of typewriting skill[END_REF]. No deadline was imposed during the course of a trial, and a few very extreme values were recorded. To study the processes of interest, we choose a 3,000-ms high cutoff for RTs, to identify trials on which participants were actively engaged in the task. On the basis of rough approximations of neural conduction time, we considered RTs below 200 ms to be anticipations that were not directly triggered by to stimulus presentation, and excluded those trials. These cutoffs for RTs had very little effect on the IKI distribution (0.57% of the data were removed). Similarly, to study trials in which participants were engaged in the task for the whole course of the trial, we kept those with IKIs lower than 1,000 ms. This whole procedure left 92.5% of the data.

Effects of uncertainty on RT and IKI For RTs, the design included the following factors: Position of Uncertainty (on 2nd or 3rd keystroke), Type of Sequence (constant or varying), and Hand of the constant sequence (left or right). We did not include any interaction between Hand and the other factors in the design. An ANOVA revealed a main effect of uncertainty [F(1, 528) = 373, p < .001]; RTs were shorter for sequences varying on the 3rd rather than on the 2nd key (M 2nd = 677 ms, M 3rd = 560 ms). This is in good agreement with the original results of [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]. The main effect of sequence also reached significance [F(1, 528) = 187, p < .001]: RTs were shorter for constant sequences (M c = 597 ms, M v = 640 ms). Finally, the Sequence × Uncertainty interaction was also significant [F(1, 528) = 7.3, p < .01]. The main effect of hand was not significant (F < 1).

For IKIs, the design was similar, with the additional factor Position of the interval within the sequence (1st or 2nd IKI). An ANOVA revealed main effects of sequence .001] were also significant. Crucially, the Uncertainty × Position interaction was highly significant [F(1, 528) = 191, p < .001], indicating that IKIs were longer at the position of uncertainty (Fig. 4). The Uncertainty × Position interaction was robust and was also observed on subsets of the data (see the supplementary material for the OSs Windows and OS X and the Web browsers Chrome and Firefox). In short, the ANOVA revealed that the main results from the original study were replicated.

Effects of participant characteristics on RTs and IKIs

We also used regression to better estimate the effect sizes over and above the individual characteristics (participants and computer configurations; see Tables 4 and5). The contrasts in categorical variables were assessed against the most frequent category (Windows operating system and Firefox Web browser).

Summarizing the results on RTs (see Table 4), we obtained the same significant effects as with the original ANOVA, although the Sequence × Uncertainty interaction did not reach significance and presented a small estimate. Importantly, the estimates for the main effects of sequence and uncertainty were the largest. Regarding IKIs (see Table 5), all main effects and interactions reached significance, except the main effect of hand. Our interaction of interest, Position × Uncertainty, presented one of the largest estimates.

Introducing personal characteristics in the model yielded a significant main effect of gender, with male participants being faster than female participants, in terms of both RTs and IKIs. We also observed a slight slowing down of both RTs and IKIs with age, but no significant effect of handedness (Tables 4 and5). Regarding computer configurations, only the contrast of Chrome against Firefox on RTs approached significance, with responses collected from Chrome being faster (see the supplementary material). This contrast did not reach significance for IKIs.

Additional analyses

Having reproduced the result from [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF], we aimed to further characterize the data collected via the online platform.

Estimation of data quantization Similar to the analyses performed on the hardware test data above, we evaluated the extent of data quantization in the online experiment. Toward this end, the IKIs were sorted in increasing order. This sampling bias was assessed on the transformed IKI distributions, generated by taking the remainder values from the division by 8. The homogeneity of these distributions was quantified over the whole sample by means of independent chi-square tests for each participant (see the Method section), which revealed that 449 participants (83% of the sample) presented a sampling bias (as indexed by significant chi-square tests, FDR-corrected). This confirms that we should not expect a precision higher than 8 ms on actual IKI measurements. The reason why some of the data from some participants did not show this quantization could not be meaningfully traced to the specific features of their computer configurations that were available to us.

Data reliability Rosenbaum's original study had included only six participants. Our final sample size (N = 541) allowed us to assess the relationship between sample size and effect reliability. From the data, we randomly selected samples that ranged from six to 100 participants (with replacement). For each sample size, 20 -54.32, 57.57] samples were drawn from the original distribution, and we calculated the mean (RT or IKI) and estimated the confidence interval of each sample. As sample size increased, the sample means became less dispersed around the mean of the whole distribution, and the range of the confidence intervals decreased: That is, the sample means became more stable estimates of the mean of the whole sample (Fig. 6A).

RT & IKI distributions

To evaluate the effect of sample size on the experimental effects, we followed the same procedure, taking random samples of increasing size from the whole distribution and running the mixed regression model described above on each sample. Figure 6B presents the evolution of the beta estimate for the Uncertainty × Position interaction across samples. It shows that the effects are well estimated from a sample size of 50, because from this value and above, all of the confidence intervals include the mean of the whole distribution.

General discussion

We aimed at quantifying the precision and reliability of timing measures performed during sequences of keystrokes. We used the JavaScript jsPsych library to create an experiment involving finger-movement sequences. Black Box Toolkit hardware tests showed a systematic delay centered around 60 ms for RTs, and an unbiased measure with an average 0-ms delay for IKIs. Thus, the delay in RT measurements did not increase when several keystrokes were collected in rapid succession. Then, online experimental data that we collected accurately reproduced the original results [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]. Random subsampling of the data revealed that, in paradigms such as this, samples of at least 50 participants are necessary for accurate estimation of the data distributions and experimental effects. Finally, both the BBTK tests and online experiment revealed substantial quantizing of the IKI data, most likely due to the sampling frequency of the USB keyboards (125 Hz). This did not prevent the assessment of experimental effects with reasonable sample sizes.

The finding of a null difference between the programmed and recorded IKIs with the BBTK is a good indication that online testing can be used for assessing differences in timing between the elements of motor sequences performed through the keyboard. It generalizes the findings of previous experiments that used a stream of identical keystrokes separated by fixed intervals in comparable conditions [START_REF] Keller | Timing accuracy of Web experiments: A case study using the WebExp software package[END_REF][START_REF] Simcox | Collecting response times using Amazon Mechanical Turk and Adobe Flash[END_REF]. Here, we measured a sequence of 5), with confidence intervals as a surrounding shaded area responses to a visual signal, with variable interresponse intervals. In addition, we showed that even for short time intervals, the online system performed with very good timing precision. The standard deviations for the IKIs were very small (around 10 ms), a value that is considered accurate in RT measurements (see [START_REF] Reimers | Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities[END_REF][START_REF] De Leeuw | jsPsych: A JavaScript library for creating behavioral experiments in a Web browser[END_REF]. The BBTK tests therefore also indicate a good reliability of the measures.

The BBTK tests revealed a constant lag of about 60 ms on the RT measurements using our jsPsych configuration. The finding of a lag is consistent with another study that compared jsPsych with programs designed for in-lab recordings [START_REF] De Leeuw | Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task[END_REF]. Comparable amounts of RT overestimation were also observed by [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF] in a test with the BBTK and JavaScript in a design very similar to ours, except for the motor sequences feature and the use of predefined jsPsych functions. Those authors tested various browsers and OSs with the BBTK, using JavaScript and html5, and showed a general overestimation of RTs that varied from 30 to nearly 100 ms (on a Dell Optiplex machine). Some OSs seem to introduce longer lags; for instance, in [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF] experiment, the two Windows 7 machines measured RTs that were 30-40 ms longer than those on XP machines. In our case, the delay between programmed and recorded responses is probably a mixed result of the load of ongoing programs running in the OS, hardware constraints (of the Safari browser and Macintosh computer), and the jsPsych program itself. In addition, a standard deviation of 8 ms is a small value, similar to those found in Reimers andStewart (2007, 2015) for measures of RTs. On the basis of those SD values, those authors, by comparing online and traditional measures, acknowledged that online measures overestimate RTs but do not add extra noise to the data. Relative to the recording of keystroke timings using a JavaScript implementation relying on the BDate^object, the present use of the getKeyboardResponse() jsPsych function could possibly add a small processing overhead.2 If this overhead impacts the recording of keystroke timings, this impact will likely be very small, since our values are very close to those of [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF]. This supports the generalizability of the present findings to any JavaScript experiment, not just using jsPsych. In addition, this indicates that jsPsych can be used by researchers in experimental psychology who need to implement experiments with high timing precision.

The quantizing of IKIs occurred for all three of the BBTK tests and for the online measurements. A similar phenomenon has also been shown by [START_REF] Neath | Response time accuracy in Apple Macintosh computers[END_REF], when testing various configurations of program/ computers and keyboards. In their study, quantizing occurred only occasionally and seemed to be related to one particular configuration of a given Macintosh computer and a particular type of keyboard, and it was not discussed further. Quantizing of responses was also observed by [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF] on the basis of cumulative frequency distributions. Quantizing occurred more frequently under certain configurations, but the authors could not find any systematic predictor of quantizing in the data. Here, in addition to showing quantizing, as usual, with the cumulative frequency distributions, we used a procedure that allowed for the quantification of the intervals between which data were sampled. In the BBTK tests, data were Bpacked^by multiples of 8 ms, a value that corresponds well to the sampling rate of a USB port (125 Hz). This was also the case in data recorded online with actual participants and variable computer settings. A great majority of the sample displayed the same quantizing by steps in multiples of 8 ms. This is a strong suggestion that quantizing is due to the sampling of the keyboard by the system. Such noncontinuous sampling of the data should be acknowledged, and some researchers for whom quantizing matters should be aware of it and choose non-USB input devices. However, a specific study has shown that under typical RT measurement conditions, the variability in human performance outweighs the imprecision in response devices [START_REF] Damian | Does variability in human performance outweigh imprecision in response devices such as computer keyboards[END_REF].

Our online study on a large sample of participants also replicated the original results of [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF]:

The position of the uncertain response had an effect on RTs, with longer RTs for uncertain responses at Position 2 than to those at Position 3. In addition, we found the same interaction between the position of the uncertainty and the position of the required response on IKI measurements. The main effect of uncertainty reported in the original experiment did not reach significance in our sample, whereas the other effects we reported had not been significant in the original study. This discrepancy is probably related to the instability of the effects estimated by [START_REF] Rosenbaum | Choosing between movement sequences: A hierarchical editor model[END_REF] in the original study. The interaction of interest was replicated later on [START_REF] Rosenbaum | Scheduling and programming of rapid finger sequences: Tests and elaborations of the hierarchical editor model[END_REF] and can be considered reliable. It should nonetheless be kept in mind that the original effects were reported with a sample of only six participants. With regard to our specific conditions (e.g., the same number of trials as in the original study, but only a subset of the original experimental conditions) and the experimental context (online measurements), we found that a minimum sample size of 50 participants was necessary to provide a relatively good and reliable estimation of the effect of interest. This threshold is nonetheless specific to the present experiment, and clearly is not a recommendation for a minimum sample size in any online experiment, since its value is likely to depend on a number of factors, such as the number of trials and the effect sizes. Our analysis nonetheless provides an illustration of the trade-off between sample size and the precision of estimates of the effects of interest, which depends on the constraints of a given experiment. Systematic tests of this type in methodological or experimental online studies would be useful to get a better overview of the minimally required sample sizes in various contexts.

Our measured RTs were longer than those from Rosenbaum et al.'s (1984) study: This could be due to the time lag (as measured with the BBTK) introduced by the operating system, computer, screen, and keyboard used (see [START_REF] Neath | Response time accuracy in Apple Macintosh computers[END_REF], for an example for the keyboard), as well as by the online configuration of the browser and jsPsych. This finding is in agreement with previous studies that had compared in-lab and Web-based experiments and typically found delays from 25 to 100 ms [START_REF] Crump | Evaluating Amazon's Mechanical Turk as a tool for experimental behavioral research[END_REF][START_REF] De Leeuw | Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task[END_REF][START_REF] Reimers | Adobe Flash as a medium for online experimentation: A test of reaction time measurement capabilities[END_REF][START_REF] Schubert | ScriptingRT: A software library for collecting response latencies in online studies of cognition[END_REF].

Regression analyses also indicated that the material configuration had no measurable effect on IKIs, although an effect of browser was evidenced for RTs. Moreover, our online interface could capture some variability linked to demographic variables: For instance, it allowed for extra assessments of the effects of variables such as age or gender. We found slight influences of age and gender on both RTs and IKIs. Previously, no effects of age have been reported on typing rates [START_REF] Salthouse | Effects of age and skill in typing[END_REF] or on indirect measures of motor-sequence learning [START_REF] Howard | Age differences in learning serial patterns: Direct versus indirect measures[END_REF]. However, a male advantage on motor speeds has been reported [START_REF] Nicholson | Sex differences for speech and manual skill[END_REF]. Such effects should be taken cautiously, because predictors such as age, gender, and computer configurations might be correlated, as was suggested by [START_REF] Reimers | Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments[END_REF], and might lead to spurious effects of the covariates. Mixed regression analyses then present the advantage of accounting for all predictors and their specific variabilities at the same time.

In conclusion, online measurement using jsPsych appears to be an accurate way to test for fine differences between the IKIs in various conditions. It offers a promising tool for researchers interested in motor-sequence learning and execution.

Fig. 1

 1 Fig. 1 Boxplots (median, 1st and 3rd quartiles, lowest and highest values within 1.5 times the interquartile range) showing the differences between the response times (RTs) recorded by jsPsych and those programmed into the Black Box Toolkit. (Top) For each of the three tests, the three consecutive RTs are RT1, RT2, and RT3. All of the conditions had similar

Fig. 2

 2 Fig. 2 Interkeystroke intervals (IKIs) for the three tests. (Upper panels) For each test, IKIs sorted in order of increasing length are displayed. (Lower panels) IKI values were divided by 8 (in relation to the 125-Hz

  [F(1, 528) = 37, p < .001] and position [F(1, 528) = 7.4, p < .01]; IKIs were overall shorter in varying than in constant sequences (M v = 221, M c = 228 ms), and were also shorter at the second than at the first position (M 2nd = 222 ms, M 1st = 227 ms). The interactions Sequence × Position [F(1, 528) = 68.8, p < .001], Sequence × Uncertainty [F(1, 528) = 5.0, p < .05], and Sequence × Uncertainty × Position [F(1, 528) = 27.6, p <

  Figure 5 (upper panel) displays two representative participants: The IKIs for the participant in the left panel take almost all possible values between 130 and 190 ms; in contrast, the IKIs for the participant in the right panel are concentrated around multiples of 8.

Fig. 3

 3 Fig. 3 Distributions of response times and interkeystroke intervals for correct trials. The y-axis in each panel is log-transformed. Three very

Fig. 5

 5 Fig. 5 Data sampling of two representative participants (left and right columns). The upper row represents interkeystroke intervals (IKIs), ranked in increasing order. The lower row represents the distribution of

Fig. 6

 6 Fig. 6 (A) Means and confidence intervals of response times (RTs; top) and interkeystroke intervals (IKIs; bottom) over random samples of increasing size (see text for details). The horizontal line represents the mean of the whole sample with confidence intervals as a surrounding shaded area. Each point represents a random sample, sample size is indicated above each panel. (B) Beta estimates and confidence intervals

Table 1

 1

	Sample characteristics
	Handedness	471 right 69 left
	Gender	324 women 214 male
	Age	mean = 39.64, median = 39, Q1-Q3 = 30-47, range = 21-69
	Affiliation	228 work at AMU, 21 study at AMU, 19 unrelated, 273 not specified

Table 2

 2 Web browsers and OS sample characteristics

		Linux	OS X	Windows
	Firefox	27	54	254
	Chrome	4	18	80
	Explorer	0	0	51
	Safari	0	45	0
	Other	6	0	1

Table 4

 4 Mixed model regression coefficient for RTs

		ß	S E	t	95% CI	(sig.)
	(Intercept)	592.55	57.09	10.38
	Experimental variables
	Hand	-5.87	13.66	-0.43	[-32.69, 20.96]
	Sequence	46.71	5.43	8.61	[36.05, 57.37]	*
	Uncertainty	-116.48	3.34	-34.90	[-123.04, -109.92]	*
	Sequence × Uncertainty	-11.75	6.36	-1.85	[-24.24, 0.74]
	Trial number	-1.05	0.03	-31.65	[-1.11, -0.98]	*
	Participants' variables
	Gender	-54.32	13.78	-3.94	[-81.38, -27.26]	*
	Age (spline 1)	3.50	1.64	2.13	[0.27, 6.73]	*
	Age (spline 2)	3.94	2.07	1.90	[-0.13, 8.01]
	Handedness	25.51	19.53	1.31	[-12.85, 63.87]
	Computers' variables
	OS: OS X/Windows	4.19	19.95	0.21	[-35, 43.37]
	OS: Linux/Windows	-45.02	28.72	-1.57	[-101.43, 11.39]
	Web Browser: Chrome/Firefox	-55.77	17.51	-3.19	[-90.16, -21.38]	*
	Web Browser: IE/Firefox	27.45	23.23	1.18	[-18.19, 73.09]
	Web Browser: Safari/Firefox	-50.50	29.71	-1.70	[-108.86, 7.86]
	Web Browser: Autres/Firefox	36.51	62.11	0.59	[-85.5, 158.53]

Table 5

 5 Mixed model regression coefficients for IKIs

		ß	S E	t	95% CI	(sig.)
	(Intercept)	205.14	26.11	7.86
	Experimental variables
	Hand	-8.16	6.13	-1.33	[-20.21, 3.89]
	Sequence	-21.18	1.65	-12.84	[-24.42, -17.94]	*
	Uncertainty	-23.13	1.06	-21.91	[-25.2, -21.05]	*
	Position	-30.81	1.06	-29.18	[-32.89, -28.74]	*
	Sequence × Uncertainty	12.93	1.95	6.62	[9.09, 16.77]	*
	Sequence × Position	22.71	1.49	15.22	[19.78, 25.64]	*
	Uncertainty × Position	39.30	1.49	26.37	[36.38, 42.23]	*
	Sequence × Uncertainty × Position	-17.94	2.10	-8.53	[-22.07, -13.81]	*
	Trial number	-0.28	0.01	-37.89	[-0.3, -0.27]	*
	Participants' variables
	Gender	-35.56	6.32	-5.63	[-47.97, -23.16]	*
	Age (spline 1)	1.46	0.75	1.94	[-0.02, 2.94]
	Age (spline 2)	2.44	0.95	2.57	[0.58, 4.31]	*
	Handedness	9.41	8.95	1.05	[-8.18, 26.99]
	Computers' variables
	OS: OS X/Windows	-1.50	9.14	-0.16	[-19.46, 16.47]
	OS: Linux/Windows	-5.40	13.16	-0.41	[-31.26, 20.46]
	Web browser: Chrome/Firefox	-0.68	8.02	-0.08	[-16.44, 15.09]
	Web browser: IE/Firefox	1.78	10.65	0.17	[-19.14, 22.7]
	Web browser: Safari/Firefox	0.18	13.62	0.01	[-26.58, 26.93]
	Web browser: Autres/Firefox	1.63	28.48	0.06	[

The delay could occur between the jsPsych pseudocommand Bdisplay stim^(with which the jsPsych initial clock read would occur) and the actual physical display on the screen (detected by the opto-detector; the BBTK initial clock read). This delay is probably due to the computation of the display screen, the communication between graphic electronic components and the physics of LCD screens, and the timing inaccuracies of jsPsych.

Due to the addition of two if statements linked to the parameterization of the function. Time measurements themselves are done basically by instantiating JavaScript Date objects. In addition, profilers have indicated that the overall execution time of getKeyboardResponse(), including the definition of start_time, key_time, and the associated conditions, ranges

between 3.4 and 5.5 ms (using Chrome DevTools and Firebug profiling, respectively).


Introduction

Afin de disposer d'éléments concrets pour éclairer la décision publique, le groupe de travail BSN4 a demandé à l'Inist la réalisation d'une étude bibliométrique sur les publications françaises en Open Access (OA) en science technique médecine (STM) sur une période de 5 ans, de 2010 à 2014, afin d'en connaître le poids dans la production scientifique française et tenter d'estimer le montant des APC.

Ce document présente des remarques et préconisations méthodologiques suite à la réalisation de cette étude. L'étude fait l'objet de deux rapports, l'un traitant de la méthodologie, l'autre des résultats :

 OpenAccess_BSN4_Méthodologie_30septembre2016.docx  OpenAccess_BSN4_EtudeBibliométrique_30septembre2016.docx Un certain nombre de résultats sont consultables via le tableau de bord dynamique créé à l'aide de l'outil libre ezVIS 1 , dont l'accès est réservé.

Les limites de chaque étape de la méthode sont évoquées et des préconisations émises :

 Source pour la constitution du corpus : justification du choix du WoS comme la source unique de collecte des publications,  Sources pour le marquage des périodiques : examen des listes de périodiques utilisées (DOAJ, JCR, ROAD, liste des « Grands Editeurs ») et évaluation des risques d'erreur liés à l'utilisation de ces listes (avec un calcul de taux d'erreur) et à la méthode d'alignement,  Source pour le repérage des publications OA issues de périodiques non OA : explicitation des difficultés rencontrées avec notamment les limites de CrossRef,  Sources pour le montant des APCs : défauts de son attribution,  Répartition du montant des APCs par organisme : limites de la répartition des publications pour 5 organismes et des critères retenus pour déterminer l'organisme payeur.

Source constitutive du corpus : le WoS

Web of Science

La base produite par Thomson Reuters, Web of Science (WoS) Core Collection, a été retenue pour cette étude pour les raisons suivantes :  source multidisciplinaire, connue par le plus grand nombre, globalement représentative de la production scientifique mondiale et couvrant la majorité du domaine STM retenu dans l'étude,  source maîtrisée par le prestataire.

Autres sources envisageables

Périodiques avec un sceau de certification

Ne retenir que les périodiques auxquels le DOAJ attribue un sceau de certification d'OA est trop limitatif puisque l'on arrive à 352 titres. https://doaj.org/faq#seal

Périodiques selon les critères 2014

Ne conserver que les périodiques qui ont été retenus suite à la création de nouveaux critères de sélection en 2014 est envisageable. On passe alors de plus de 8 800 à plus de 4 000 titres. Ces nouveaux critères sont décrits à l'adresse suivante : https://doaj.org/publishers#advice.

Périodiques avec licences « CC by »

Ne retenir que les titres pour lesquels une mention de licence CC by est attribuée est également envisageable. On passe alors de plus de 8 800 à plus de 5 900 titres.

Appui au Pilotage Scientifique

Voici les résultats obtenus en fonction des critères retenus : Pour de futures études, le taux de marquage pourrait être augmenté en effectuant des listes pour des éditeurs supplémentaires. En effet, s'il s'avère qu'il y a plus de 1 300 éditeurs, 12 d'entre eux produisent la moitié des publications issues de périodiques « SansStatut »5 . Pour chacun de ces 12 éditeurs, le statut des périodiques représentant plus de 4% pourrait être obtenu manuellement : une soixantaine de titres seraient alors à considérer.

JCR -ROAD -« Grands Editeurs »

Méthode d'alignement des titres et ISSNs

Bilan du marquage des périodiques

Appui au Pilotage Scientifique

Pour réduire ces traitements manuels fastidieux, d'autres sources pourraient également être exploitées.

Autres sources envisageables

SHERPA/RoMEOhttp://www.sherpa.ac.uk/romeo/index.php SHERPA/RoMEO, qui recense les politiques des éditeurs de revues scientifiques en matière de copyright et d'auto-archivage, propose une liste de périodiques qui fournit l'information sur le statut OA ou non du périodique. Ce site est alimenté par le DOAJ, le service Zetoc de la British Library, et la liste « Entrez » hébergée par le NCBI (National Center for Biotechnology Information).

Source pour repérer les publications OA issues de périodiques non OA

Si compléter le marquage des périodiques OA et « Hybride » et celui des publications OA issues de périodiques OA est réalisable, la difficulté demeure de repérer les publications OA au sein de périodiques « Hybride ». Même s'il progresse, le taux de ces documents est extrêmement faible sur la période avec moins de 2% (584 publications). Ces documents sont repérés grâce à l'application CrossRef.

CrossRef

Sur près de 300 000 DOIs des publications issues de périodiques non OA interrogés via l'API de CrossRef, 584 documents ont répondu aux critères définis dans la méthodologie (0,2% de réponses « positives ») et sont donc considérés comme OA, soit moins de 2% des 34 660 publications du corpus France OA.

Selon un représentant de CrossRef, lors de l'utilisation de la ressource, la mention des licences est renseignée pour environ 5% des DOI. En attendant l'amélioration de l'alimentation de ce réservoir et afin de mieux repérer les publications OA issues de périodiques « Hybride », de nouvelles sources peuvent être exploitées.

Autres sources envisageables

PubMed Central

En interrogeant PMC à partir des DOIs présents dans le WoS, on peut récupérer l'information concernant le statut de l'article. On combine alors cette donnée au statut du périodique repéré à l'aide du fichier de PubMed Central mentionné ci-dessus6 . De plus, PubMed Central offre un certain nombre

Répartition des publications OA et du montant des APCs par organisme

Comme le document sur la méthodologie le précise, un point de la demande initiale du groupe BSN4 concerne le calcul de la part des publications OA et de leur coût par organisme. Des traitements, au départ non prévus, ont été réalisés à partir des champs « Affiliations » et « Reprint Author » car l'OST n'a pu fournir les marquages qu'il récupère lors des activités de repérages effectués par les organismes de l'ESR.

Répartition des publications OA par organisme

Le choix a priori de 5 organismes (CEA, CNRS, Inra, Inserm, ParisTech) et la recherche sur le seul sigle de l'organisme ne sont pas satisfaisants. Mais, cette première approche a permis de montrer que la méthode de repérage est pertinente car les chiffres concernant les publications OA de l'Inra et de l'Inserm correspondent à ce que ces organismes ont calculé de leur côté.

Pour élargir la répartition aux autres organismes français et affiner les résultats, il est indispensable que l'OST communique les marquages obtenus grâce au travail des différentes institutions ou de récupérer, dans le futur réservoir de Conditor, les marquages créés automatiquement.

Répartition du montant des APCs par organisme

Si la répartition des publications OA par organisme est satisfaisante, il n'en va pas de même pour la répartition des coûts pour les organismes retenus. En effet, l'Inra et l'Inserm ne retrouvent pas les chiffres escomptés. Une comparaison avec les coûts réels payés par l'Inserm à BioMed Central pour l'année 2014 met en évidence un surcoût de 165 000€ : alors que l'étude attribue un montant d'APCs de près de 250 000€ pour l'Inserm, le coût réel est de près de 85 000€ soit presque le tiers (Pour plus de détails voir l'Annexe 1).

A cela plusieurs raisons possibles, le champ « Reprint Author » utilisé pour connaître l'organisme payeur de l'APC :  n'est peut-être pas l'information la plus appropriée pour connaître l'organisme payeur. Une autre limite de l'étude actuelle est la répartition des publications par organisme qui s'est restreinte à 5 organismes. Cette étape serait indéniablement facilitée par la récupération des marquages de l'OST. Sans l'aide de ce dernier, de nouveaux traitements sont à envisager et pourquoi pas, à moyen terme (2-3 ans) la récupération des marquages automatiques effectués dans la plate-forme Conditor.

Conclusions et perspectives

Concernant le calcul du montant des APCs actuel, les résultats ne sont pas satisfaisants :

 une forte surestimation des montants dans la répartition des coûts par organisme,  la limite de l'utilisation du seul champ « Reprint Author »,  des données APC disparates qui entraînent le besoin d'une source d'information officielle pour une mise à jour des données. Le fichier de l'Inserm a permis de combler les manques du DOAJ et des listes des « Grands Editeurs » constituées pour l'occasion, Même si la proposition de prendre en compte les données comptables des organismes semble difficile à mettre en oeuvre (information peu précise, parcellaire, nécessitant des identifiants périodique ou article …), elle est à approfondir.

Dans le cadre d'une mise à jour de l'étude, si la méthodologie pour repérer l'OA peut être conservée, l'estimation des montants APC nécessite vraisemblablement l'exploitation de données comptables.

Pour le champ des STM, le WoS source multidisciplinaire est satisfaisant d'un point de vue couverture. D'autres sources devront être utilisées pour étendre l'étude aux SHS. A terme, Conditor semble la ressource incontournable qui pourrait d'ailleurs proposer les informations liées à l'Open Access et aux APCs. Sources pour repérer et marquer les périodiques OA et « Hybride » DOAJ (Directory of Open Access Journals)

  L'exploitation d'autres sources est envisageable soit pour compléter le repérage des publications françaises en science technique médecine (STM), soit pour initialiser une étude concernant les sciences humaines et sociales (SHS) :  Scopus : avant son utilisation, il est utile d'étudier son contenu et ses métadonnées. Un rapprochement avec les résultats de l'étude menée -dans le cadre de Conditor -d'une comparaison Scopus-WoS est donc souhaitable.  PubMed Central (PMC) : si cette base présente l'avantage de fournir une information concernant le statut de l'article, elle concerne essentiellement le domaine biomédical. Son utilisation risque donc de biaiser les résultats des autres domaines. Elle peut être utilisée pour des études spécifiques concernant des organismes ou des thématiques en lien avec cette source.  Conditor : cette future ressource doit recenser de manière plus exhaustive la production scientifique française. Sa constitution peut prévoir des métadonnées exploitables pour répondre plus facilement à des questions concernant l'Open Access. Dans le cadre de cette étude, le fichier DOAJ, source d'information reconnue qui propose un inventaire des périodiques OA, est exploité dans sa totalité, soit plus de 8 800 titres au 30 mai 2016. D'autres approches plus restrictives auraient pu être envisagées.

	Préconisations

Périodiques OA à écarter Publications à écarter Part des publications (Corpus 34 660)

	DOAJ -sceau de certification		Solution non retenue
	DOAJ -critères 2014	472	10 504	30%
	DOAJ -licences « CC by »	212	5 450	15,7%

Tableau 1 : Part des publications à écarter en fonction de sélection de différents critères à partir du DOAJ Préconisations Le DOAJ marque plus de 80% du corpus (785 périodiques, 28 214 publications). 9% du corpus est marqué uniquement par lui (173 périodiques, 3 097 publications). L'utilisation de la liste complète est préconisée pour ne pas diminuer ce pourcentage de 16 à 30% selon les critères retenus. Le DOAJ propose depuis fin 2015 une API. Elle serait à explorer pour récupérer de l'information au niveau de l'article grâce au DOI et non plus seulement au niveau du périodique.

Qualité des sources utilisées : taux d'erreur Au

		Périodiques	Publis	Part des publis	Publis de	Publis non OA	Taux
		OA*	OA**	(Corpus 34 660)	l'échantillon	de l'échantillon	d'erreur 2
	JCR	220	2 441	7%	1 372	214 3	1%
	ROAD	76	1 836	5%	1 314	230 4	1%
	« Grands Editeurs »	28	1 113	3%	805	0	0%

-delà du DOAJ, plusieurs autres sources sont utilisées pour repérer les périodiques OA ou « Hybride ». Le JCR (Journal Citation Reports) est choisi pour son lien avec le WoS, les listes des « Grands éditeurs » pour tenter de couvrir au maximum le corpus, ROAD (répertoire des ressources scientifiques et universitaires en libre accès) pour tester ce nouvel inventaire. Pour les listes ROAD et « Grands Editeurs », la date à laquelle le périodique est devenu OA n'est pas mentionnée. Si ce point est important pour l'étude longitudinale 2010-2014, il n'est pas un inconvénient en cas de mises à jour annuelles ultérieures. De plus, l'absence de nombreux ISSNs dans les listes des « Grands Editeurs » a pu empêcher de repérer des périodiques OA ou « Hybride » (voir plus loin).

Pour estimer un taux d'erreur pour l'ensemble du corpus France de 34 660 publications OA, on s'intéresse aux périodiques marqués par une seule des sources, respectivement : le JCR, ROAD, les listes des « Grands Editeurs ».

Pour chacune des sources, un échantillon comprenant les périodiques les plus productifs est constitué (56% des publications pour le JCR, 72% pour ROAD et les « Grands Editeurs ») et leur statut est validé manuellement. Un taux d'erreur est calculé à partir de cet échantillon. Tableau 2 : Taux d'erreur pour les listes JCR -ROAD -« Grands Editeurs » * Périodiques OA : périodiques OA marqués uniquement par la source concernée ** Publis OA : publications OA marquées uniquement par la source concernée Préconisations Si

la seule liste du DOAJ est utilisée, le corpus final est constitué de 15% (+/-2%) de publications en moins. Ceci étant, la multiplication des sources complexifie les traitements. Conserver le JCR est nécessaire car il est lié au WoS, source de collecte, et qu'il propose un statut de périodique par année. La liste ROAD est à surveiller car elle est encore en phase beta et ne propose pas de statut de périodique par année. Quant aux listes des « Grands Editeurs », la qualité des données accessibles sur leurs sites n'est pas toujours satisfaisante. La solution serait la constitution d'un réservoir unique d'une liste de périodiques auxquels seraient associées leurs différentes caractéristiques (titres, ISSNs, statut et montant des APC par année, éditeur …) en fonction des années.

Exemple envisageable d'impossibilité d'alignement non rencontré dans l'étude Préconisations Utiliser le fichier d'issn.org pour enrichir les listes où les ISSNs sont absents. L'alignement sur les titres conforte la nécessité d'une source « référentielle » pour éviter d'être confronté aux multiples formes d'écriture et pour avoir les différents ISSNs associés à un titre.

  ISSNs dans la liste des « Grands Editeurs » : l'alignement est plus difficile sur le titre seul même si, comme on l'a vu, certains traitements d'homogénéisation sont réalisés.-des alignements sur des titres identiques mais des ISSNs différents ne représentant pas le même périodique (cf. Tableau 3). -des ISSN différents, ISSN print dans la notice Wos et ISSN électronique dans les listes de périodiques, ou inversement un même ISSN avec des titres présentant des graphies différentes (cf. Tableau 3). Bien que ces types d'erreur possibles aient été identifiés, ils sont difficilement quantifiables.

	Appui au Pilotage Scientifique
	-l'absence d'WoS		DOAJ
	Nucleus 1949-1034		Nucleus 1678-6602 1982-2278
	Tableau 3 : Exemple de titres identiques correspondant à 2 périodiques distincts
			Titre DOAJ ou	ISSN DOAJ ou
	Titre WoS	ISSN WoS	ROAD ou Grands	DOAJ ou ROAD ou
			Editeurs	« Grands Editeurs »
	POLIMEROS-CIENCIA E TECNOLOGIA	1806-9282	Polímeros	0104-4230
	Tableau 4 :
	Risques d'erreur
	Les risques d'erreur peuvent être liés à :
	-la date de constitution des listes (ROAD et « Grands Editeurs ») : un périodique OA en 2014
	sera considéré comme OA quelle que soit l'année traitée (sauf si ce périodique est présent
	dans la liste du DOAJ qui précise la date à laquelle le périodique est devenu OA).
	2 Taux d'erreur = (Publis OA x Publis non OA de l'échantillon / Publis de l'échantillon) / 34 660

Présentation de la méthode Le marquage des périodiques est effectué à l'aide d'une comparaison de chaînes de caractères d'une part sur les titres (après une homogénéisation minimale) et d'autre part sur les ISSNs, de manière successive. Un alignement combiné, et non pas successif, Titre + ISSN, risque en effet d'être trop restrictif pour deux raisons :  Hétérogénéité des titres entre les différentes listes.  Absence de la donnée ISSN dans certaines listes de « Grands Editeurs » trouvées sur leurs sites. 3 Ces publications concernent : Santé Publique qui n'est pas OA, mais dont les articles concernant la période sont accessibles sur internet (à la date du 18 août 2016), et Inra Productions Animales (40 publications sans DOI) dont le statut n'est pas clair. Aucun DOI n'étant attribué, nous n'avons pas vérifié l'accessibilité de ces documents. La question est de savoir pourquoi ils sont considérés comme OA par Thomson Reuters. 4 Ces publications concernent le Bulletin De L Academie Nationale De Medecine qui n'est effectivement pas un périodique OA mais « Hybride ». Aucun DOI n'étant attribué, nous n'avons pas vérifié l'accessibilité de ces 230 documents.

  La liste de périodiques n'étant pas déchargeable et les données récupérables présentes sur le site ne présentant pas la mention d'OA (http://www.sherpa.ac.uk/romeo/journalbrowse.php?la=en&fIDnum=&mode=simple), il serait intéressant d'étudier l'API proposée : http://www.sherpa.ac.uk/romeo/apimanual.php?la=en&fIDnum=|&mode=simple. //www.ncbi.nlm.nih.gov/pmc/journals/#csvfile) peut être exploité. Même si les périodiques biomédicaux sont favorisés, c'est une piste envisageable.

	PubMed Central (PMC)
	Le fichier de PubMed Central (PMC) qui liste les périodiques présents dans la base avec la mention
	OA	(immédiat	ou	suite	à	un	embargo)	ou	« Hybride »
	(http:

Appui au Pilotage Scientifique Préconisations La solution est la même que celle envisagée pour repérer et marquer les périodiques, à savoir : la constitution d'un réservoir unique d'une liste de périodiques auxquels seraient associées leurs différentes caractéristiques (titres, ISSNs, statut et montant des APC par année, éditeur …) pour chaque année.

Préconisations En plus de la constitution d'un réservoir unique d'une liste de périodiques auxquels seraient associées leurs différentes caractéristiques (titres, ISSNs, statut et montant des APC par année, éditeur …) pour chaque année, exploiter les données comptables car l'utilisation du champ « RP » n'est pas pertinente.

  Dans ce cas, quelle donnée exploiter ? Nous avons fait une expérimentation sur le champ consacré au mail pour l'année 2014. Elle n'a pas été appliquée à la période. En effet, ce champ : o peut donner une autre information organisationnelle autre que celle du champ « Reprint Author ». Quelle donnée retenir dans ce cas ? o peut également présenter plusieurs organismes, o n'est pas toujours renseigné, o propose pour les 2/3 des adresses mails des données inexploitables car elles n'ont pas l'extension « .fr », o propose des adresses avec une extension « .fr » mais dont le préfixe ne définit pas l'organisme (« yahoo.fr » par exemple), o peut présenter une adresse qui ne reflète pas l'organisme payeur mais l'organisme « hébergeur » : les laboratoires CNRS ont souvent une adresse universitaire.  peut contenir plusieurs organismes. Dans ce cas, le montant est attribué à chaque organisme, ce qui signifie qu'il est comptabilisé plusieurs fois. Si le montant des APCs est fortement surestimé, il faut rappeler qu'en plus les publications OA issues de périodiques « hybride » n'ont pas été repérées de manière adéquate.

	La réflexion sur la répartition des coûts est donc à poursuivre. Une piste pourrait être l'exploitation
	des données comptables des organismes s'ils détiennent une information du type DOI ou identifiant
	d'article (clé UT pour le WoS, PMID (PubMed identifier) pour PubMed …) associé à un montant
	d'APC. Elle serait à l'image de l'initiative allemande OPEN@PC qui propose de recenser les
	dépenses APC des organismes de recherche (http://treemaps.intact-project.org/)

  Malgré la complexité de l'étude liée à la manipulation de nombreuses informations, éparses, partielles, provenant de différents organismes, présentées sous divers formats, il s'avère qu'en l'état actuel des choses on peut valider globalement la méthodologie pour le repérage des publications OA puisque l'Inra et l'Inserm retrouvent les données qu'ils obtiennent par ailleurs et qu'un taux d'erreur (dans le choix des sources retenues en dehors du DOAJ) estimé à 2% est acceptable.Ceci dit, avant la répercussion du marquage des périodiques sur les publications, 36% des périodiques sont considérés comme « SansStatut » 10 et 30% comme « Hybride ». Moins de 2% de ces publications ont été repérées par CrossRef comme étant des publications OA, ce qui est vraiment très peu même si ces données recoupent les résultats d'autres études.Si une mise à jour annuelle est envisagée, les listes des périodiques OA et « Hybride » initialement utilisées seront actualisées et enrichies. Le fichier d'issn.org pourra également être exploité pour améliorer les alignements des titres et des ISSNs.La proposition de rechercher manuellement le statut d'une soixantaine de périodiques est une solution envisageable pour réduire la part des périodiques « SansStatut ». doai.io fournit actuellement des résultats insatisfaisants et le mode de recherche sans API est inenvisageable eu égard au volume des données à traiter.Actuellement, rien n'existe pour repérer de manière satisfaisante les publications OA issues de périodiques « Hybride ». Une alimentation régulière et fiable de CrossRef concernant les mentions de licence par article serait donc un plus non négligeable. Il serait souhaitable de présenter aux éditeurs l'intérêt d'un tel outil pour qu'ils le renseignent systématiquement. Un contact avec les auteurs finlandais de l'étude longitudinale dédiée aux publications OA issues de périodiques « Hybride » est à envisager 11 . Quant à l'utilisation de PubMed Central, elle entrainera des biais disciplinaires.

	Appui au Pilotage Scientifique
	La proposition d'explorer d'autres applications (PubMed Central, SHERPA/RoMEO, doai.io) ne
	garantit pas nécessairement l'obtention de meilleurs résultats conséquents :
	-PubMed Central privilégie le biomédical,
	-SHERPA/RoMEO s'appuie beaucoup sur le DOAJ et reste au niveau du périodique,

-

https://github.com/madec-project/ezvis/

Amer Physical Soc ; Iop Publishing Ltd ; Royal Soc Chemistry ; Oxford Univ Press ; Lippincott Williams & Wilkins ; Edp Sciences S A ; Masson Editeur ; Spie-Int Soc Optical Engineering ; Cambridge Univ Press ; Amer Inst Physics ; Amer Geophysical Union ; Amer Soc Microbiology

http://www.ncbi.nlm.nih.gov/pmc/journals/#csvfile

Un périodique « Sans statut » est un périodique dont le statut n'est pas défini car il :n'appartient à aucun des 7 « Grands Editeurs » retenus ET -est absent des listes JCR, DOAJ et ROAD.

http://dx.doi.org/10.1016/j.joi.2016.08.002

Publications et montant des APCs attribués à l'Inserm d'après l'étude bibliométrique

Appui au Pilotage Scientifique

de filtres qui permettent de repérer les articles hybrides (ACS Author Choice, Elsevier Sponsored Documents, etc…) L'étude des APIs (http://www.ncbi.nlm.nih.gov/pmc/tools/developers/) pourrait être intéressante.

Comme déjà évoqué plus haut, les publications biomédicales seraient favorisées. Reste alors la distinction à effectuer entre l'article « purement » OA et l'article accessible après une période d'embargo.

doai.io

Cette application (http://doai.io/), développée par une association française 7 permet de retrouver, à partir d'un DOI, un article OA grâce essentiellement aux données du moteur Bielefeld Academic Search Engine (BASE 8 ). Des tests à partir d'une dizaine de DOIs ont été effectués par l'équipe Inist. On peut mentionner que d'une part des articles non OA ont été retrouvés 9 et que d'autre part, des articles OA n'ont pas été repérés. Mais le faible nombre de tests ne permet pas de conclure sur l'efficacité ou non de l'outil. La difficulté est également de connaître la raison de l'accessibilité de l'article : article « réellement » OA ou article devenu OA après une période d'embargo. L'étude de cet outil peut se poursuivre. Cependant l'absence d'API oblige à faire des tests DOI par DOI, ce qui semble difficile à réaliser lorsque, pour une année, plus de 75 000 articles français sont publiés.

Etude de M. Laakso et BC Björk

Sources pour le montant des APCs

Une source centralisée d'information sur les APCs n'existant pas, les données APCs sont récupérées à partir de la liste du DOAJ et d'une liste de périodiques fournie par l'Inserm. Cette dernière complète celle du DOAJ où près des trois quarts des périodiques du corpus France OA n'ont pas de donnée APC. Comme le document concernant la méthodologie le précise, quand un titre de périodique a un APC dans les 2 listes, le montant de la liste proposée par l'Inserm est retenu même s'il y a des écarts conséquents entre les 2 montants proposés.

Malgré ces 2 listes, 237 périodiques (2 605 publications, 7,5%, dont 1 200 concernent un périodique : Journal Of Physics Conference Series) restent sans donnée APC. Ces chiffres diminuent si l'on s'intéresse uniquement aux publications pour lesquelles la France (métropole et outre-mer) est mentionnée dans le champ « Reprint Author » : 2 089 publications sans donnée APC (6%), 4 898 publications avec 0 pour APC (14%). La seule solution pour améliorer ces 2 points est de traiter individuellement les périodiques concernés.

Il est à noter qu'un seul montant APC pour un périodique ayant été appliqué pour toutes les années, une progression annuelle ne peut être mise en évidence dans cette étude. Elle peut l'être si des mises à jour sont faites ultérieurement.