Keywords: Data-Driven Learning, DDL, Corpus Linguistics, Concordancing, Corpus-Based Language Learning subject 37 96.71 0.0035

Thirty years of data-driven learning: Taking stock and charting new directions over time

INTRODUCTION

Since corpora emerged in their modern form in the 1960s as large collections of electronic texts designed to represent an area of language use, numerous applications have been found for them in language teaching. These applications have been both indirect, with specialists using corpusderived information for improved language descriptions leading to new dictionaries and other resources, and direct, with language teachers and learners searching and using corpora themselves. The latter type of application was given prominence in work by Tim Johns at Birmingham University who used the term Data-Driven Learning (DDL) as early as 1990. Many researchers associate DDL with a central hard-core involving "the hands-on use of authentic corpus data (concordances) by advanced, sophisticated foreign or second language learners in higher education for inductive, self-directed language learning of advanced usage" (Boulton, 2011, p. 572). However, it can extend in many directions away from this prototypical core so that some activities may be more or less 'DDL-like', a broader definition with fuzzy boundaries encompassing many different ways of "using the tools and techniques of corpus linguistics for pedagogical purposes" (Gilquin & Granger, 2010, p. 359). This is the definition adopted here, with the addition that the pedagogical purposes are for foreign or second language (L2) learning, teaching or use.

In line with the usual evolution of new techniques in applied linguistics [START_REF] Shintani | Comprehension-based versus production-based grammar instruction: A meta-analysis of comparative studies[END_REF], early DDL publications were mostly descriptive or speculative, containing suggestions for corpus-based teaching activities; less than a dozen empirical studies had appeared by the end of the century. Since then, the body of empirical DDL research has been growing rapidly, including two special LLT issues in 2001 and 2017. The general topic of 'corpora' was fifth of 43 themes [START_REF] Gillespie | CALL research: Where are we now?[END_REF] detected in a selection of journals in CALL (Computer-Assisted Language Learning). The aim of this paper is to survey the evolution of DDL over 30 years, with a focus on empirical studies. Specifically, the integrated double analysis -manually coding the scope of the papers and semi-automatically analyzing them as a corpus for recurrent keywords -allowed us to examine not only what has been done, but also what future directions were suggested by the researchers, how (if at all) these suggestions changed over time, and how (if at all) they were realized in subsequently conducted and published studies.

A number of DDL researchers have provided position papers on DDL, recently including two plenary speeches by key DDL names in Language Teaching. [START_REF] Chambers | Towards the corpus revolution? Bridging the research-practice gap[END_REF] seeks to "bridge the research-practice gap", noting that DDL is mainly the practice of aficionados, and underlining the need for research among 'regular' teachers (i.e., non-researchers) -a request that could apply to countless other areas of CALL research. O'Keeffe (2020) echoes [START_REF] Flowerdew | Data-driven learning and language learning theories: Whither the twain shall meet[END_REF] in noting the lack of theoretical underpinnings in DDL and calling for a "broader research gaze" beyond constructivism and sociocultural theory to position it more firmly within the field of Second Language Acquisition (SLA). [START_REF] Tribble | Teaching and language corpora: Perspectives from a personal journey[END_REF] is yet another example of such position papers that take stock of the field, though they often tend towards a personal or theoretical stance rather than a meticulous review as such.

A number of scholars have already provided different types of syntheses for DDL, beginning with [START_REF] Chambers | Popularising corpus consultation by language learners and teachers[END_REF]. Her in-depth analysis of 12 studies reached the primary conclusion that the approach was worthwhile, but that there was a serious lack of stringent quantitative evaluation, a challenge taken up in numerous studies that followed. Currently, however, such complaints have reversed, with Pérez-Paredes (2019) now lamenting that "the fact that the pool of papers examined is fundamentally empirical may have contributed to a lack of theoretical positioning" (p. 17). As the first synthesis, it is not surprising that Chambers' sample is fairly small, though it is by no means exhaustive. A later attempt by [START_REF] Boulton | Learning outcomes from corpus consultation[END_REF] to focus on learning outcomes from 27 papers also found grounds for optimism. Some syntheses restrict the field: Boulton (2012a) to ESP (English for Specific Purposes) through 20 DDL papers; [START_REF] Luo | Data-driven learning in second language writing class: A survey of empirical studies[END_REF] to writing in 18 papers published between 2010-2016; [START_REF] Chen | A critical review of research and practice in data-driven learning (DDL) in the academic writing classroom[END_REF] to DDL in EAP (English for Academic Purposes) from 37 papers, mainly from 2010-2017. Godwin-Jones (2017) organized his survey by the type of DDL technology used. These studies tend to highlight the tremendous diversity of objectives and instruments, tools and uses, while at the same time noting areas that are in need of further research. As with the position papers discussed earlier, such narrative syntheses provide valuable insights, though they are often limited in scope, with fairly serendipitous collection procedures and manual analysis. This may be deliberate as in [START_REF] Boulton | Research timeline: Corpora in language teaching and learning[END_REF], which provided a timeline of personally selected papers for individual comment, but may at times be due to a lack of methodological rigor. An original perspective is provided in [START_REF] He | Study of corpus' influences in EAP research (2009-2018): A bibliometric analysis in CiteSpace[END_REF], a mainly bibliometric analysis of co-citation clusters (i.e., groups of articles which feature the same citations) in 328 papers collected using the keywords EAP and corpus from journals in the Social Science Citation Index between 2009 and 2018. Most recently, Pérez-Paredes' (2019) systematic review converted the papers he examined into a corpus, allowing the identification of frequent key clusters alongside a coding scheme that focuses particularly on normalization [START_REF] Chambers | Making CALL work: Towards normalisation[END_REF] of DDL within CALL. The limitations here are the focus on just five major CALL journals from 2011-2015, for a total of 32 papers. The most inclusive study to date is Boulton (in press), finding 351 empirical studies up until 2018. The methodology is used as a foundation for the present paper, with the trawls and coding now extended and analyzed historically, combined with a corpus keyword analysis of the conclusion sections.

Finally, there have been a number of meta-analyses of DDL. One advantage of this type of synthesis is the attempted rigor in collecting and analyzing the data; however, only quantitative data can be included, so valuable qualitative research is neglected. The first to meta-analyze DDL were [START_REF] Mizumoto | A meta-analysis of data-driven learning approach in the Japanese EFL classroom[END_REF], who surveyed 14 studies, all in Japan and featuring the second author, mainly for lexicogrammar among lower-level learners. Their pre/post-test comparison found medium effect sizes overall. This was followed by [START_REF] Cobb | Classroom applications of corpus analysis[END_REF] with 21 studies, updated in [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF] for 64 publications and over 3000 participants. The rigorous trawls and broad sweep to include potentially any application of DDL result in the most comprehensive meta-analysis to date, finding large effect sizes in both pre/post-test and control/experimental designs. Their overall finding was that "DDL works pretty well in almost any context where it has been extensively tried" (p. 386), with medium or large effect sizes recorded for almost all moderator variables based on at least 10 samples. The most recent meta-analysis can be found in [START_REF] Lee | The effects of corpus use on second language vocabulary learning: A multi-level analysis[END_REF], who focused exclusively on vocabulary in 29 studies. Their more sophisticated multi-level analysis provides valuable insights in this area, with medium effect sizes the norm. These last two papers both provide the complete data for further analysis, but the conclusion is clear: it is not now so much a question of whether DDL works, but how it may best be used with different learners in different contexts for different purposes.

Against this backdrop of reviews and syntheses of DDL, the present paper describes rigorous trawls conducted to uncover a near-exhaustive collection of 489 empirical evaluations of some aspect of DDL over three decades , published in English in academic journals, book chapters and conference proceedings; this makes it the largest such collection to date. The texts were converted into a corpus of over 2.5 million words to aid the analysis using corpus tools. The papers were read and coded for context, participants, research design and methodology, learning tasks and other variables, all leading up to the final theme of the technology used, i.e. the corpora and tools. Rather than simply treating the collection as a uniform whole, the papers are divided into five time periods for a historical perspective: as new research appears, it is to be expected that new areas will be addressed, whether pedagogical, technical or methodological. The main part of the analysis thus focuses on the conclusion sections for recommendations for future work, and assesses to what extent this has been converted into new research strands. Specifically, two kinds of data will be compared in the analysis: (1) the scope of the papers, as captured by the coding scheme of publication types and research designs; (2) the themes for future directions identified in the Conclusion sections, particularly via the grouping of key words and phrases. The aim here is to map the field, identify gaps in the existing research, and determine areas in need of further exploration.

Research questions

The following research questions guided our study:

1. How did the publication scope of empirical DDL research change from 1989 through 2019? 2. What themes for future research directions were identified in these studies and how did the frequency of these themes change over time? 3. How were these suggested future directions realized in the study designs in subsequent periods?

Questions (1) and (2) were explored first, and the results were used to inform the analysis conducted in response to question (3), which is the main research question of our study.

METHODOLOGY Collection

The objective was to analyze empirical DDL studies, with DDL defined as the use of corpus tools and techniques for pedagogical purposes in a foreign/second language; studies with native speakers only were excluded, with the exception of initial or in-service teacher training. Note that this definition is deliberately neutral on both the 'text' to be explored (which is not necessarily a corpus per se) and the procedures (whether the learners work 'hands-on' with the electronic corpora and tools, or 'hands-off' with print-outs or other pre-processed data derived from corpora, typically concordances, frequency lists, collocations, etc.). Empirical studies are also broadly defined, as any publication that provides some type of evaluation of DDL, from experimental designs exploring learning outcomes to observational studies of learning processes to feedback questionnaire studies; this may be the entire focus of the paper, or just a minor section. Papers were excluded if they only described corpora or tools, analyzed corpora (notably learner corpora), talked about potential or actual examples of DDL but with no evaluation other than the teacher-researcher's impressions, and so on.

Since one objective was to create a searchable corpus of published research, only full texts in English were included, notably journal articles, book chapters and conference proceedings.

Conference posters and oral presentations, slides, notes and other text types were excluded, as were PhD and master's dissertations since their length would potentially bias the corpus; they are also unsystematically available, though extracts are often reworked in other formats. On the other hand, short write-ups of conference papers or even in some cases long conference abstracts are included where they meet the other inclusion criteria. It is also clear that dozens or hundreds of papers in other languages do exist, at least in the European languages the authors are familiar with, and in several Asian languages.

The collection was started over 10 years ago, with recent versions being described in [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF] and Boulton (in press). New trawls were conducted for the present study by a graduate research assistant under the authors' guidance. These used combinations of key words including DDL, data-driven, corpus/corpora, concordanc*, and Johns; contextualizers were often needed to reduce noise, notably language and learning. The main databases searched were LLBA, MLA, ERIC, JSTOR, DOAJ, Web of Science, Academia, ResearchGate and Google Scholar (filetype:pdf to reduce noise); a number of new resources such as 1findr, SemanticScholar and ORCiD produced few or no additional items and were quickly abandoned. When 50 consecutive results produced no new hits, that source was discontinued. When a potential hit was identified, the abstract was read and the paper located and downloaded; borderline cases were discussed between the two authors. All the reference lists were scoured for further leads, and any source (journal, publisher or conference series) that provided two or more hits was individually followed up. Sometimes the process of locating the full text produced yet more results, as searching the title online could lead to the author's homepage with other studies, or publications that cited the original target.

Our objective was to include all and any papers that fit the criteria, regardless of other considerations. For example, some studies are reported in more than one paper, often with substantial overlap of the text itself (e.g., short conference papers later developed in journals); we also identified four cases of apparent plagiarism, but kept the papers here as our aim is to look at what is available, regardless of its potentially severe demerits. Quality is a tricky topic and largely subjective [START_REF] Lipsey | Practical meta-analysis[END_REF]: on reading a paper, one might be impressed by the freshness of the research questions or the elegance of the design; or, conversely, bored by the lack of originality or shocked by apparent flaws in the methodology. To sidestep this issue, many systematic reviews include articles from certain journals only (e.g., [START_REF] Pham | A scoping review of scoping reviews: Advancing the approach and enhancing the consistency[END_REF][START_REF] Pérez-Paredes | A systematic review of the uses and spread of corpora and data-driven learning in CALL research during 2011-2015[END_REF], the downside being that large quantities of research go unacknowledged. Our approach was to ignore quality itself, but to conduct parallel analyses of just those journals that feature in the top 100 of linguistics in the JCR rankings (Journal Citation Reports; Clarivate Web of Science, 2019). Such bibliometric systems have been roundly criticized, and a journal's impact factor can in no way be taken as a guarantee of the quality (or lack of it) in individual papers; nonetheless, it does provide a rough-and-ready indicator of the likely impact and readership a paper may have [START_REF] Schöpfel | Le JCR facteur d'impact (IF) et le SCImago Journal Rank Indicator (SJR) des revues françaises: Une étude comparative [The JCR impact factor (IF) and the SCImago[END_REF]. To avoid doubling the wordcount by systematically reporting for JCR and other publications, the comparisons will be brief and only reported where substantial differences are identified. Full data can be found in the online supplement so that others can sort and process the data in other ways. The next two sections explain how we came up with the taxonomies for the two types of data used in our analysispublication scope (captured in the article coding sheet) and future direction themes (captured via corpus analysis of keywords in article conclusions).

Publication scope: coding sheet and procedure

The collection was first coded as a form of scoping review to "map the existing literature in a field of interest in terms of the volume, nature, and characteristics of the primary research", especially useful when the topic is "of a complex or heterogeneous nature" (Pham et al., 2014, p. 371). The original manual was drawn up by [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF], informed by other metaanalyses and syntheses in applied linguistics. The spreadsheet essentially divides into four sections:

• Publication: date and source of publication, length.

• Population: L1 and L2, country and second/foreign language context, proficiency, university or other situation, specialization and comparing language for general, specific or academic purposes (LGP, LSP, LAP).

• Procedures: duration, corpora and tools (if hands-on), interaction type, language focus, and whether it has DDL as a learning aid or a reference resource, or looked at learner attitudes or behavior during use. • Design: number of participants in control and experimental groups, the instruments used and whether the data were analyzed statistically or qualitatively.

The first author had read and coded the first batch (401 papers). The second author independently read and coded a random subset of 10, and disagreements were resolved; this process was repeated until no new problems arose, subsequent to which she read and coded the 88 new papers found. The way the collection has grown over time does however mean that results should be taken as indicative rather than absolute. This is inevitable in a study of this scale as a complete re-read of all 489 papers would be prohibitively time-consuming. Some categories are entirely factual and very rarely problematic (e.g., date and source of publication), while others involve "best guesses due to insufficient information given in the primary studies" (Lin, 2014, p. 135). This particularly concerns the level of proficiency; most often, we used the labels given by the authors of each paper despite noted problems [START_REF] Burston | Looking for a needle in a haystack: CALL and advanced language proficiency[END_REF].

Duration is another sticking point, since it is variously given in minutes, hours, sessions, weeks, months, semesters, etc.: a semester may be fairly intensive with several long classes over 15 weeks entirely devoted to DDL, or it may be half a dozen classes where only a few minutes are dedicated to corpus work.

Identification of future direction themes: corpus keyword analysis

With a large collection of publications as we have here, corpus tools can be useful in helping to explore specific questions rather than relying on regular reading alone; however, we wanted to retain the human analysis and not rely entirely on automatic black-box document clustering, summarization or other procedures. For this, we used the freeware AntConc [START_REF] Anthony | AntConc (3.5.8m)[END_REF].

The first author converted the complete set of texts to txt format, cleaned and checked to ensure that only the original text remained. In particular, meta-data (affiliations, contact, acknowledgements, etc.) were removed along with headers and footers, figures and tables, primary and secondary data extracts (with the exception of academic quotations), references and appendices. The texts were checked (e.g. for ligatures, hyphens, mathematical symbols, diacritics, etc.) but not edited in any other way; any errors in the original were left in place. This gave a full corpus of 2,563,589 tokens which could be explored in its entirety to help with the analyses.

To identify themes suggested by article authors for future research directions, we used a combination of automated and manual analysis. First, we created a Conclusions corpus of final sections comprising 253,569 tokens. Next, we used AntConc tools to identify words and n-grams that were significantly more frequent in Conclusions compared to the full corpus. The Keyword tool analysis yielded 140 types and 66,699 tokens with positive keyness. Many of the Conclusions keywords were metalinguistic text structuring devices, including subheadings, typical of the sub-genre 'research article conclusions' (e.g., conclusion(s), limitations, implications), which were ignored for the current analysis. Words that could potentially be relevant to our main theme of interest (future research directions), were explored individually with the AntConc Concordance tool. A final list of 11 words was identified that primarily appeared in relevant contexts and in immediate clusters with other relevant words ( APPENDIX A). For the key n-gram comparison, the Conclusions corpus and full corpus n-gram lists were created first. The search parameters were set to 3-4 words, with the minimum occurrence of 3 times in at least 3 articles. Next, the Keyword tool was used to search the Conclusions corpus list for key n-grams in comparison to the full corpus list. This comparison yielded 436 key n-gram types for 11,890 n-gram tokens. Many of the Conclusions key n-grams (similar to keywords, see above) were metalinguistic text structuring devices (e.g., the findings of, the results of, the present study, discussion and conclusion), which were again ignored for the current analysis. The remaining n-grams (41 types, 646 tokens) were combined in clusters by common theme ( Table 1. Taxonomy of future directions themes from the Conclusions corpus.

RESULTS

Our analysis was divided into two steps as reported in the two sections below. The first describes the scope of the articles in our collection vis-à-vis the coding sheet rubrics; the second discusses changes in all identified themes in relation to the scope of the studies across time periods. The first section is shorter as it reports on the analyses conducted to enable the main analysis reported in the second section.

Scope of the articles in the collection

The initial pool in [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF], collected using similar procedures as here, consisted of 181 DDL papers (not counting 17 PhD dissertations and 7 papers in other languages); Boulton (in press) unearthed 351. The present study uncovered 489 in total, an increase of 170%. This rise is partly due to the large numbers of papers being published, partly to better indexing and the use of additional databases and improved procedures. This underlines, first, the general health of DDL as a research field; and second, that varied and rigorous methods are essential to tap the large numbers of studies that often fly under the radar, but that even now it would be naïve to claim we have an exhaustive collection.

The first study to meet our criteria is Baten et al. from 1989; the end cut-off point was set at 2019, the last full year prior to writing. While early publications are essential for a historical overview, it takes time to build momentum and for a substantial body to appear, and we grouped these from 1989-2003 for a total of 44 papers. This enabled us to settle upon four equal time periods of four years each for the rest as numbers increase: 2004-2007, 2008-2011, 2012-2015, 2016-2019. As each has different numbers of publications, main results will be given as rounded percentages for comparison purposes.

Of the 489 papers, 361 (74%) are journal articles from 185 different sources, 135 of them featuring only one paper. This suggests that a focus only on DDL-friendly sources will lead to major omissions in coverage. The most popular journals for DDL publications are ReCALL (28 papers), CALL (25), LLT (23) and System (11); no other journals reach double figures. It is worth noting that all 4 of these journals are in the JCR100 list, which in total accounts for 117 papers in the corpus (24%) from 15 different journals. This subset of nearly a quarter of the corpus is used as a comparison for the rest of the collection -not as a guide to quality in itself, but as an indicator of whether surveys limited to just top journals reflect the entire body of research we have identified. Figure 1 shows the evolution of publication, with the dotted line representing a three-year smoothing (i.e., averaged over a three-year period). Figure 2 shows the papers grouped into periods as outlined above, for the JCR100 list and others.

Changes in the identified themes in relation to the scope of the studies across time periods

A first group of themes concerns the nature of the research and how it is conducted. DDL research has primarily been experimental throughout its history, to the extent that the lack of theorization has been pinpointed as a potential problem. The keywords associated with the theme theory in article Conclusions ( APPENDIX B) show how references to this theme have changed over time, clearly in line with specific SLA theories adopted at each time in DDL. The concept of transfer associated with the generativist theory is used throughout the timeline but is key only in the earliest period. A number of terms associated with constructivist SLA theory appear as key during three of the later periods: hypertext (2004-2007), intelligences, encoding (2012-2015), and enhancement (2016-2019), while sociocultural theory comes to the fore in 2008-2011 with the concepts social, materializations, scaffolding. It must be noted that some of these terms are associated with a single author and appear only in Conclusions of the few theoretically oriented articles. The fact that the authors draw attention to these theories shows a desire to inspire more theory-grounded research in the future. However, few papers in our empirical collection actually do this (cf. Pérez-Paredes, 2019, discussed above).

Methodology themes

Methodology as a theme in its own right only appeared in 2016-2019 Conclusions through the keywords experimental, questionnaire, scores (41 occurrences, APPENDIX B). In general, prior to 2004, 43% of the studies were purely qualitative, giving no indication of numerical data whatsoever -a possibly ironic finding given the nature of corpus linguistics and DDL, though it is difficult to generalize given the range of objectives covered in different papers. On the other hand, 25% give raw numbers, percentages or descriptive statistics, and 32% performed some kind of statistical analysis. However, the figures stabilize after that for overall rates of 15%, 35% and 50% respectively. The JCR100 have a slightly greater portion of statistical analysis, but the difference is small (54% vs 49%).

Sample size is often acknowledged as a limitation in Conclusions, with calls for a larger number of participants in future studies (Figure 3). While not a big concern in the earliest period, it has regularly been mentioned in 18-22% of Conclusions since 2004. Despite these persistent requests, the average group size has remained remarkably stable between 38 and 42, presumably as this reflects convenience sampling and typical (university) class sizes. The JCR100 papers involve substantially larger groups than others (45 vs 36); however, the standard deviation is very high at 54. The median is 26, with 178 studies involving fewer than 20 DDL students, including 61 with fewer than 10, and 12 just one or two. At the other end of the scale, 30 had 100 or more experimental participants, the largest study involving 526. Clearly larger sample sizes increase power, but in DDL as in other areas of applied linguistics, researchers are limited by institutional considerations. The choice of instrument clearly depends on what the researchers want to focus on, though a general finding is that most studies use more than one instrument (mean = 1.9). The single most popular instrument is the questionnaire, used in 53% of papers, though only in 49 cases is it the sole tool. These papers are mainly interested in learners' perceptions of DDL and, in some cases, what they did. Other perception instruments include individual interviews (21%) or group discussions / focus interviews (8%), while behavior is also monitored by a wide variety of selfreports (journals, diaries, logs, think-aloud, stimulated recall: 17%), teacher observation or field notes (8%), tracking or screen recording (7% There have been numerous and increasing calls for more longitudinal research or longer DDL treatments, explicit or implied by mentioning that their own study was too short. In the three latest periods, 19-22% of the Conclusions mentioned this theme. In reality though, there is no obvious tendency towards longer or more ecological research, although this is difficult to assess given differing reporting practices (hours, weeks, sessions, semesters, etc.). Sometimes it is inherent in laboratory designs, with 19 studies lasting less than an hour, 34 a single session.

From 2008 on, at least, the studies seem to have similar duration regardless of how they are measured: 13 hours, 9 sessions, 8 weeks, or slightly over 1 semester for the longer ones (Figure 4). Nor is there any major difference between the JCR100 and other papers, although the former do show somewhat higher means for hours (15 vs 13) and sessions (12 vs 8). Only 9% included a longitudinal analysis via delayed post-tests, with another 5% including delayed questionnaires distributed after the end of the experiment. Therefore, despite the recognized need, little is being done to investigate extended DDL or follow up after use, likely due to planning considerations and the number of variables involved, as well as pressure to publish.

Figure 4. Mean duration with the measures available

Learning / contextual themes

A second group of themes concerns the learners themselves and their particular situations. Language proficiency has been cited frequently in the Conclusions, between 22% and 31% of articles mentioning it in each time period. This generally calls for a wider variety of proficiency levels to be explored, especially lower levels (Figure 5). Actual research seems to have responded to these calls, at least to some extent: in the first two periods, only 22% were with lower levels compared to 37% today. It may be that having established the credentials of DDL at higher levels, researchers are now experimenting to see how far they can go. More prosaically, it may be that the tools are more amenable, that there are more relevant corpora for lower levels, or that the techniques have been refined and adapted to new contexts. Nonetheless, more work would seem to be needed with lower levels of proficiency, which is not necessarily synonymous with age.

Figure 5. Sample concordances to the theme proficiency from the Conclusions corpus

The theme of institutions reflects the recently growing interest in expanding DDL applications to new learner populations and learning environments. The evidence is especially solid in regard to such keywords as elementary, young, mobile, each appearing in a range of 6 to 7 article Conclusions published in 2016-2019 ( APPENDIX B). The coding sheet however shows that calls for DDL extension outside tertiary education still remains a minority interest. Among the 477 studies that give some indication of the institutional context, just 9% are with younger learners, in schools (elementary, middle, junior high or high) or other pre-university courses, but the university constitutes the mass of participants, accounting for 85% overall. This may indicate that DDL is most appropriate at these levels, or that researchers simply follow suit, or, prosaically, that these are the participants that they have access to in their own teaching. Of these, 39% are undergraduates compared to just 16% of graduates, including doctoral students; a further 30% combine undergraduates and graduates or merely note that they were conducted at university. Only 7% are in other contexts, mainly language institutes (which may cover a multitude of situations), or on rare occasion with various types of professionals. There seems to be little change over time and no major differences between JCR100 and other papers. There is clearly much to be done to investigate how DDL may be used outside class, after the end of a course, or in non-university contexts.

Specialization comes up in other Conclusions keywords: translation, translators, engineering, legal, law ( APPENDIX B). Disciplinary or other specialization is indicated in 363 papers: 47% feature students majoring in languages, linguistics, translation, language teaching, or some other language-related degree: 19% are in other humanities and social sciences, 16% in STEM subjects (science, technology, engineering, medicine), and 18% mixed. These figures are fairly stable, although STEM subjects increased from 8% in the first two periods to 17% in the last two; the difference between JCR100 and other papers is negligible. However, DDL is often used for 'general' English purposes even in highly specialized disciplines; indeed, general language accounts for twice as many studies (66%) as LSP and LAP combined (15% and 16% respectively), with mixed studies relatively rare (about 2%, usually an LSP/LAP crossover); no notable differences were found over time or between JCR100 and other papers. This may seem unintuitive: since general language can be found in generic resources (dictionaries, textbooks, etc.), it might be thought that DDL is most appropriate for specific needs in narrow fields.

Geographic distribution is important for individual researchers, as highlighted by national keywords (e.g., Thai) calling for more work in their own country. Figure 6 shows the evolution by region in the 5 time periods. The suspicion that DDL is a largely European phenomenon (e.g., [START_REF] Hadley | Sensing the winds of change: An introduction to data-driven learning[END_REF] may have some basis in the earliest periods; also that it has been slow to take off in North America. The last two periods since 2012 have seen large increases in the main regions, with sudden huge increases in Asia and the Middle East in particular, even though these regions may be associated more with a teacher-fronted knowledge-dispensing educational model rather than with the kind of autonomous discovery-based processes involved in DDL. This may imply that DDL as a largely inductive process comes as a welcome change from usual practice.

Figure 6. Evolution by region

The target language was overwhelmingly English in 432 papers (89% of those stated), of which 95 (22%) were JCR100. Eleven other languages are listed, notably including 6 in French, 5 German and 3 Italian in the JCR100 studies. The country of origin generally indicates the first language, with the exception of studies in inner-circle English-speaking countries where they tend to be mixed with English as the L1: only 45% of papers are foreign-as opposed to secondlanguage learning there, as opposed to 85% overall.

Implementation themes

Interaction with DDL has remained stable, with the main procedures being introduced right from the start: keywords such as concordancing, concordance, concordancer, collocations, collocate change only slightly over time ( APPENDIX B). Nonetheless, researchers continuously call for more studies using these terms, i.e. more of the same. It is also clear that DDL has established its footing as the top key term in the latest batch of article Conclusions. Computer-based (hands-on) DDL has been by far the most popular theme here, attracting attention in at least 38% of Conclusions in each period, peaking in 2004-2007 at 54% (107 and 177 occurrences in the last two periods). The authors call for more hands-on DDL applications in which learners explore corpora autonomously or on their own (Figure 7). In contrast, paper-based was barely mentioned as a theme at the beginning and occurs in only about 9% of Conclusions in the three latest periods. This distribution is somewhat more balanced in what has actually been done as nearly two thirds of papers (63%) have participants working directly with a concordance and a quarter (25%) with prepared materials, usually paper but occasionally slide projections; 11% have used corpora integrated into a wider CALL program. However, 21% use two approaches, usually paper leading up to hands-on concordancing, but rarely comparing them directly. While participants are often encouraged to collaborate in pairs or small groups, with collabo* as a search term appearing in 88 of the 489 papers, it is rarely a main objective (only 11 use it 5 times or more). No convincing patterns emerge over time or between JCR100 and other publications. While specific tasks were not coded, it seems likely that new tools provide new affordances, though the basics were in place decades ago and may not have changed that much. 15% in 1989-2003 to about 30% in each subsequent period. In fact, just over a quarter (26%) overall use a corpus as a reference resource, compared to 42% looking at learning outcomes). It is noteworthy that the JCR100 papers are less focused on outcomes than the others (34% vs 44%), perhaps as pre/post-test designs are fairly obvious and easy to administer, and are relatively more interested in learners' behavior (32% vs 20%), which is more difficult to track.

Other objects of study include the processes involved (23%) and, especially, the participants' attitudes and perceptions in both JCR100 and other papers (56%). Typically collected via questionnaires or interviews as we saw earlier, this emic perspective often occurs as a secondary feature alongside the main focus and is almost de rigueur. With a large body of supportive research, the question now is less 'does it work' than 'how well does it work' with different learners with different needs in different contexts, using different resources, etc.

Guidance or training has consistently been mentioned as a theme in 22-26% of the Conclusions for each period with a spike of attention in 2004-2007 (more than 50%). Overwhelmingly, the authors have concluded that DDL requires extensive, planned, ongoing, meticulous, etc. teacher guidance (Figure 8). Although this did not figure as a category in our top-down coding sheet, a search for this theme in article abstracts yielded only 47 hits in total (in contrast to 182 hits in Conclusions), with only a handful of articles stating the amount of guidance needed for successful DDL as their research goal. This shows a disconnect between researchers' claims about the importance of and the need for more guidance and the small number of attempts to empirically support these claims. Language skills as mentioned in article Conclusions have been changing from period to period. Syntax and oral language skills were more prominent in the earliest period (1989)(1990)(1991)(1992)(1993)(1994)(1995)(1996)(1997)(1998)(1999)(2000)(2001)(2002)(2003), then attention shifted to pragmatics and discourse in 2004-2007, collocations, colligations, and lexical and prosodic targets in 2008-2011, and Despite these temporal differences in Conclusions, there is little obvious change over time in the collection as a whole. Language skills are mentioned as a focus in 125; of these, writing is far the most frequent (98), followed by translation and reading (17 each), with speaking (8) and listening (4) far behind. The figures are relatively stable over time, and there is little difference between JCR100 and other papers. Vocabulary, variously defined, is clearly a recurrent objective, listed in 123 papers, though it fades into lexicogrammar (including collocates, chunks, grammar-function words, error-correction, etc.), which occurs in 256 papers, with considerable overlap. Wider areas such as discourse, pragmatics and rhetorical functions, discourse markers and cohesion, etc. are highlighted in only 41 cases (nearly all non-JCR100), presumably as they are less amenable to the basic surface queries of corpus tools. This leaves 23 that are interested in corpus linguistics per se (e.g., how learners go about building or using a corpus, corpus literacy, etc.), and a diverse 18 which range from literary analysis and cultural awareness to systemicfunctional linguistics, language awareness, interference, cognitive skills and critical thinkingthough again, none of these are in the JCR100. While vocabulary and lexicogrammar are the most obvious focus, then, especially for writing, DDL has moved into wider areas of language.

Technological themes

Technology is an obvious candidate for change over time, and indeed some hits for this theme in the earliest period Conclusions clearly referred to now superannuated media, tools or corpora (e.g., cards, PET), while others became key in the latest period -possibilities like mobile or telecollaborative, and newer and currently developing corpora like COCA and FLAX ( APPENDIX B). Most data about technology, however, come from our top-down analysis of actual research.

In terms of software or interface, the BYU suite at English-Corpora.org is the most commonly used in 31% of all papers, though with a slight reduction in the final period. Other integrated interfaces include SketchEngine and the BNCweb in 11 cases each, VLC1 in 9 and MICASE/MICUSP in 8; no others feature more than 5 times. Some older stand-alone software is tending to disappear: MicroConcord, MonoConc and ParaConc (in 5 or 6 papers overall) do not feature at all in 2016-2019; similarly, WordSmith Tools, despite appearing in 33 papers overall, is down from a high of 30% of papers in 2004-2007 to just under 2% in 2016-2019, perhaps due to the availability of free rivals that are arguably more user-friendly for pedagogical purposes.

LexTutor has its ups and downs, being used a total of 23 times, while the most popular downloadable tool overall is AntConc with 43 recorded uses. This first appeared in 2002 but has no recorded use in our DDL collection until 2009. Other tools such as FLAX are starting to make an appearance (5 papers so far), though use of the researchers' own tools seems also to be in decline (20 cases, accounting for just 4% or 5% of papers in the last 3 periods). Other tools, though excellent for research, are virtually absent here (e.g. CQPweb). There are essentially no differences between JCR100 and other papers. This then is one area of real change over time as new technology is developed and made available, often linking corpora and other resources. For example, COCA has recently integrated links to outside resources in its interface, from dictionaries and automatic translation to videos, images, Google searches, and full texts.

Corpus is an unconvincing keyword in this context, with the specific corpora often but not always named -354 times among the 391 papers that feature hands-on work. Overall, 27% used more than one corpus, a figure increasing over the 5 time periods from 8% to 28%. Selfcompiled corpora were used as at least one source in 42% of studies, though the figure is decreasing (56% to 29%), presumably as reliable outside corpora become more readily available.

In line with this, just a third of papers named the corpus in 1989-2003 compared to over 70% in the last two periods. A significant number (19%) use more or less specialized corpora, whether disciplinary or genre, particularly academic texts, and more among the JCR100 papers (24% vs 17% for the rest). Some of these are ready-made corpora or subcorpora (e.g. MICASE or COCA_academic), though a large number are compiled from research articles as an obvious and relatively 'clean' source of texts, often but not always for disciplinary academic writing. Learner corpora remain difficult to come by or create, with fewer than 10% of studies using them in any time period; even fewer require learners to create their own corpora, rarely going above 5%, and virtually none using graded corpora specially written or compiled for the target proficiency level (under 3%, and in the final period only). Corpora of literary texts make a small contribution (3-6%) but disappear entirely in the latest period. Manuals were a popular source in 1989-2007 (17%) but have never since risen above 3%. The single most popular corpus is the BNC in one form or another in 24% of papers, ahead of COCA at 20%. COCA was first made available in 2008 (see [START_REF] Davies | The 385+ million word Corpus of Contemporary American English (1990-2008+): Design, architecture, and linguistic insights[END_REF] and by 2016-2019 is being used in nearly a third of all studies (32%), while the BNC has remained relatively steady at around 25-30% since 2008. Some of the older corpora are fading in popularity (the COBUILD / Bank of English and Brown families are down to under 3% and 2% respectively in the latest period), while others have yet to break on to the scene in a large way: MICASE/MICUSP and BASE / BAWE together account for just 4% of studies in 2016-2019. Despite their obvious appeal, the potential of corpora and tools such as WordAndPhrase or SkELL remain largely untapped here, reported in just 2 and 3 papers respectively. Surprisingly, perhaps, only 19 used the web directly as a 'corpus', hovering at around 5% for the entire span, despite anecdotal evidence that many learners are simply Googling words and phrases for answers to their language questions. Parallel corpora (often English paired with Chinese or Japanese) have been popular at times, featuring in 9% of papers, while multi-modal corpora are almost invisible -just 9 studies, with MICASE and Backbone the only two to occur more than once. New, larger, rigorous and specialized corpora are thus making an appearance, though there seems to be a dearth or underuse of substantially original types of corpus, especially spoken corpora, learner corpora, graded corpora, parallel corpora, and so on, all of which may be highly relevant for pedagogical purposes.

Of the 250 hands-on studies that note the size of at least one corpus in terms of tokens (as opposed to texts, sentences, n-grams, or characters), and disregarding the web, the smallest is 1.2k words of student texts (clearly not a 'corpus' in the usual sense, but amenable nonetheless to exploration with corpus tools and techniques), the largest 2.8bn; exactly half are under 100m words, very often self-compiled corpora. Figure 9 gives the percentage of corpora in different bands (≥1k, ≥10k, etc.) for the five time periods, where such information is explicitly given.

There is a striking decline in popularity for smaller corpora over time, while corpora of 100m words or more take the lion's share in the last three periods; corpora of over 1bn words only make an appearance in 2016-2019. The mean corpus size increases apace, from 15m tokens in early days to over a quarter of a billion in the last period, skewed by the mega-corpora (the median and mode are both 100m). Though none of the JCR100 papers feature corpora under 10k tokens, the other papers have a higher mean corpus size (199.6m vs 142.6m tokens), partly as they flirt more with very large corpora (5 of the 6 uses), partly also due to the increasing numbers of such publications in later periods when large corpora are more readily available. If "there's no data like more data", this may not be true for pedagogical purposes, where both small and large have their own advantages. In what follows, we bring together our main findings and outline our suggestions for future research.

The lack of a theoretical-ideological base may have helped DDL to get as far as it has, but after 30 years it would seem necessary to push further and expect theories to drive continued development. Some researchers have tried to pin DDL to underlying theories such as constructivism [START_REF] Che Cobb | Applying constructivism: A test for the learner as scientist[END_REF] or noticing and sociocultural theory [START_REF] Flowerdew | Data-driven learning and language learning theories: Whither the twain shall meet[END_REF], but these attempt to justify the approach after the event. Pérez-Paredes ( 2019) and O'Keeffe (2020) are among those calling for research to be more theory-led in future. Of course, an approach survives on its merits and works because it works (or doesn't), regardless of theory; but this does leave us rather hungry for more.

Methodologically, researchers have been employing a variety of data collection and analysis instruments and methods. Nevertheless, although research quality was not our goal, we could not help but notice that statistical analyses have frequently been insufficiently robust and reporting practices non-standard. Next, we found persistent and growing calls for greater sample sizes and duration but no growth trend in actual studies. Further, the field has been growing more diverse in terms of geographical regions and first languages spoken by DDL participants. However, English still heavily dominates as the target language, although the number of studies exploring DDL for other languages has increased recently (Vyatkina, 2020a). Finally, research has overwhelmingly been conducted with university students in language-for-general-purposes classes. Our findings thus confirm Gillespie's (2020) wider observations about empirical CALL studies: "they are conducted over a few weeks or a semester at most, or infrequently over one academic year; are preliminary, and usually not followed up; and involve a small number of students (sometimes fewer than 10), more often at beginners or intermediate level, in a single institution" (p. 138). There are clear pragmatic reasons behind using convenience samples and short interventions at institutions where researchers teach. Nevertheless, there is clearly much to be done with longer, ecological settings to investigate how DDL may be used outside class, after the end of a course, or in non-university (professional, primary and secondary schools) contexts. Greater collaboration, both within and between institutions, might help to not only overcome these limitations but also explore contextual differences. More outreach to language teachers in form of open access corpora, accessibly written DDL guides (e.g., [START_REF] Poole | A guide to using corpora for English language learners[END_REF][START_REF] Vyatkina | Incorporating corpora: Using corpora to teach German to Englishspeaking learners[END_REF], and training opportunities should also broaden the DDL research base.

Another area where we noticed little change in DDL research was types of learner interaction with technology. The development of corpora and corpus tools has been driven by research but also by pedagogical and other needs. These tools, in turn, have also been the driving force behind pedagogical innovation inasmuch as they allow previously time-consuming, difficult or impossible activities. The simple early tools were mainly used for the concordances they produced, quickly adding frequency lists (lemmatized or not), collocations, clusters (bundles or n-grams), keywords, distributions, and so on. Each of these allows a new way to interact with the language. However, our overview shows that the main procedures -examining concordances and frequency information -have remained largely unchanged since the early research days. Another area worth noting is the development of long-term, higher-level, non-language skills. Such skills as critical thinking, independent learning, and learner autonomy feature prominently in article Conclusions as alleged benefits from DDL, but we found virtually no direct exploration of these concepts as research objectives. Therefore, Boulton (2012b) observation still holds: "it is notable that much of the research to date focuses on targets that are easy to measure in a highly controlled experimental environment -short-term learning outcomes in vocabulary and lexicogrammar, as well as error-correction and Likert-scale questionnaires of learner attitudes, etc., [yet] there is a notable dearth of studies looking at the major advantages that are generally attributed to DDL" (p. 86). While operationally defining these abstract constructs certainly is not straightforward, we hope that future innovative research will pursue this challenging yet most promising direction that would go hand in hand with much needed theorization of DDL and its cross-fertilization with broader SLA and sister fields like educational psychology.

A final comment: although not specifically targeted in our analysis and largely impressionistic, it is our conviction from reading the papers that quantitative researchers should take advantage of contemporary multifactorial methods that allow for reliably exploring multiple variables simultaneously. More mixed-methods studies are needed that provide both rigorous quantitative accounts and ecologically valid qualitative analyses of DDL processes and outcomes. Studies that set out to test previous research findings should be clearly situated as replication studies, which have been lacking to date. Partial replication studies are especially welcome that would explore specific variables such as proficiency level or target language while keeping other design elements constant. Finally, a promising direction would be more integration of DDL and SLA.

Employing such SLA constructs as implicit and explicit knowledge, receptive and productive knowledge, knowledge breadth and depth, controlled and free production, and testing DDL effectiveness for the development of these knowledge types and language skills would undoubtedly bring both fields forward.

SUPPLEMENTARY MATERIALS

The entire coding sheet is available online as a simple Excel spreadsheet for others to sort and search at will. This contains the full references and abstracts which are usually freely available, but the rest of the corpus consists largely of copyright material which cannot be openly shared.

Figure 1 .Figure 2 .

 12 Figure 1. Article counts by publication date

Figure 3 .

 3 Figure 3. Sample concordances for the theme sample size from the Conclusions corpus

Figure 7 .

 7 Figure 7. Sample concordances to the theme autonomy from the Conclusions corpus

Figure 8 .

 8 Figure 8. Sample concordances to the guidance theme from the Conclusions corpus

Figure 9 .

 9 Figure 9. Corpus sizes reported over time

  ). A few (2%) analyzed project work or reports of some kind, 23% analyzed productions (usually writing or self-correction, occasionally translation or, in 3 cases, spoken production), with more controlled tests featuring in 45%. The JCR100 are more likely to use tracking (15% vs 5%) and questionnaires (60% vs 51%), but actually slightly less likely use 'tests' (43% vs 46%). This is perhaps because their aim is not (or no longer) to see if DDL 'works' but more subtle phenomena. Less formal instruments (reports, discussions, observations) have generally decreased over time (together, they were used in 43% of publications prior to 2003, declining in each period to 22% in 2016-2019), but so has the use of tracking, from a high of 13% in2004-2007 to 5% in 2016-2019. Other changes over time are minimal or not obviously directional, an interesting finding in itself.

  verbs in 2016-2019 ( APPENDIX B). Further, interest in chunks has become prominent in the two latest periods (keywords chunks, formulaic, MWE). While some of this keyness (e.g., pragmatics in 2004-2007) can be attributed to prominence of a few specific authors, other targets have attracted a wider interest (e.g., verbs being mentioned in Conclusions of 15 studies from 2016-2019).

  There are virtually no studies that compare different types of learner-corpus interaction or different types of corpora (specialized corpora, open-access corpora, web as corpus). Other underexplored possibilities include parallel corpora of translated texts in two or more languages, and multimodal corpora -written text aligned with sound and/or video recordings. Though these are among the most difficult to produce and access, they are potentially of tremendous interest for teaching and research. Some readily-available tools have managed to harvest online data to provide a DDL-like interface for multiple languages such as Linguee or DeepL, or videos such as PlayPhrase with TV and film extracts, or extracts from YouTube in YouGlish -despite the name, again available for different languages. Similarly, a promising area is combining corpus tools with other CALL practices such as multimedia glossing (e.g.,[START_REF] Frankenberg-Garcia | Developing a writing assistant to help EAP writers with collocations in real time[END_REF]. We now have far greater choice of tools and corpora, which are faster, more aesthetic, and user-friendly (Godwin-Jones, 2017), and our hope is that they will attract the attention of future DDL researchers.Overall, we found that much of research covers the same ground, with many studies confirming that DDL 'works'. Going forward, more nuanced, comparative research is needed. What works in different contexts and for different learner profiles? Are particular procedures, corpora or tools more appropriate in some settings than others? The field can now move away from experimental/control (or DDL/non-DDL) designs toward designs with two or more experimental groups.

Here and in some other cases, the original links may have changed.

ACKNOWLEDGEMENTS

The authors would like to thank Tom Cobb and James Thomas, as the present study draws on previous collaboration and inspiration. Particular thanks go to A. Jakob Johnson who conducted most of the latest round of trawls, listing the papers which the authors then assessed for inclusion; he also conducted bottom-up manual coding for future research directions themes. The project was supported in part by the College of Liberal Arts and Sciences, University of Kansas.

As the next step, we looked for keywords that characterized each time period in comparison with all other time periods. After using the Keyword tool and weeding out irrelevant hits from the total of 133, we identified 18 keywords in the [START_REF] Baten | The use of concordances in vocabulary acquisition[END_REF]-2003Conclusions subcorpus, 23 in 2004[START_REF] Chambers | Popularising corpus consultation by language learners and teachers[END_REF], 11 in 2008[START_REF] Boulton | Data-driven learning: The perpetual enigma[END_REF], 19 in 2012-2015, and 30 in 2016-2019 (APPENDIX C). The key n-gram comparison per time period yielded no obviously meaningful results.

In the final theme identification step, we grouped the resulting 15 themes (inferred from the 11 keywords and 41 key n-grams from the whole Conclusions corpus and 101 period-specific keywords) into 4 overarching categories (Table 1).

Thematic categories Themes Keywords and n-grams Theory and methodology

• sample size


Expanding methodological approaches in DDL research

This paper analyses the methodologies in 148 empirical data-driven learning studies for L2 English in prestige journals to examine best practice. Manual coding and corpus analysis of key words and n-grams from the past five years (2018-22) explore the field as a whole and how methodologies have evolved, suggesting improvements and future avenues for research.

INTRODUCTION

Data-Driven Learning (DDL) involves the use of corpus tools and techniques for foreign/second (L2) language learning and use. Hundreds of empirical studies have led to several research syntheses, from qualitative narrative reviews (e.g., [START_REF] Chambers | Popularising corpus consultation by language learners and teachers[END_REF] to quantitative meta-analyses (e.g., [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF], bibliometric analyses (e.g., [START_REF] Dong | The intellectual structure of the data-driven learning field: A bibliometric analysis[END_REF] and other systematic reviews (e.g., [START_REF] Pérez-Paredes | A systematic review of the uses and spread of corpora and data-driven learning in CALL research during 2011-2015[END_REF]. In the most comprehensive to date, [START_REF] Boulton | Thirty years of data-driven learning: Taking stock and charting new directions[END_REF] focused on how proposals for future research directions were taken up for exploration in subsequent studies: several methodological practices remained remarkably constant over thirty years despite repeated calls for change from DDL researchers themselves. This study builds on the methods and results of earlier DDL syntheses, with a particular focus on methodology in empirical DDL studies targeting English, since research results depend on the methodologies employed (cf. [START_REF] Chong | A typology of secondary research in Applied Linguistics[END_REF]. Specifically, manual coding alongside corpus analysis of the methodology sections allows a two-pronged overview of the field to identify methodologies used, especially in the last five years (2018)(2019)(2020)(2021)(2022), and to suggest avenues for future research.

METHOD

The article sample reflects our primary interest in empirical DDL research, defined as studies that evaluate some aspect of the explicit use of corpus tools and techniques for L2 learning or use. Scientific "quality" is clearly a major methodological issue here: while all researchers know intuitively what quality means, it is difficult to establish specific criteria to measure it empirically: relevance, research questions, rigorous design with appropriate materials and instruments, analysis and interpretation, etc. -all of these entail an element of subjective appreciation. Many studies of quality in applied linguistics, such as Xu et al. (in press), have combined quantifiable aspects (e.g., journal impact factor) with qualitative judgements (researchers' ratings of various journals), finding correlation between them (see also [START_REF] Choubsaz | A longitudinal analysis of highly cited papers in four CALL journals[END_REF]. Drawing on this, we limited our survey to research articles (RAs) published in journals ranked in the latest Journal Citation Reports (2023) for Linguistics and Education. While journal ranking is no guarantee of the scientific quality of any individual paper, taken as a group, papers published in high-ranking journals are likely to be better received and, by definition, have more impact overall. It additionally allows confidence in a near-exhaustive collection up to and including 2022. Keyword searches (DDL, data-driven, corpus/corpora, concordanc*) resulted in a total of 148 papers for English L2. The timeline with RA counts by year is presented in Figure 1: minor fluctuations in output are inevitable with relatively small numbers; the minor decrease between 2019 (13 RAs) and 2022 (7 RAs) can be compared with the fact that, at the time of writing, we are aware of 14 RAs in press in their final form, complete with DOI. Table 1 gives details of journals with at least three entries including one in each period, followed by less frequent sources. The manual stage of data analysis involved reading new papers to update the coding sheet1 in [START_REF] Boulton | Thirty years of data-driven learning: Taking stock and charting new directions[END_REF]. In the semi-automated stage, the 148 RAs included were converted to txt format and 'cleaned' to retain the authors' own text; a subcorpus of only the Methodology sections totals 252,326 words. The corpus was lemmatized via TagAnt, and analysis conducted using AntConc, with a stoplist to exclude the 200 most frequent items (e.g., the, of) from the BNC. Keyword and key n-gram lists were used for analysis of recurring methodological themes. This dual approach allowed us to triangulate the results and pinpoint specificities which one alone might miss.

FIGURE 1. Annual and cumulative totals for included RAs

CODING RESULTS

This section outlines the main methodological characteristics of DDL research to date and, where substantial differences are noted, specific to the last five years (2018)(2019)(2020)(2021)(2022). First, an overview of the background will provide minimum context. Overall, Asia accounted for 44% of all studies and is on the rise; Europe remained stable at around 27% of studies, with contributions from 14 countries; the Middle East rose from 7% to 13% (total 9%); North America represented 17% of all studies, mainly from the USA ( 16), but from 22% to 7%. In total, 'inner-circle' Englishspeaking countries represented 24% of RAs (down from 28% to 15%); these accounted for most second-language contexts, though the majority (72%) were in countries where English is not spoken as an everyday or official language. The participants are predominantly at upperintermediate or advanced levels of English proficiency: 57% compared to 20% lower-intermediate or below (there were no 'beginners'); the others are labeled 'intermediate' with no further qualification. 90% of studies were conducted at university, with just 11 in earlier education and 2 in other contexts. Where specified, nearly half of all courses were language related (45%); most were for English for general purposes (EGP, 60%), though in the final period, academic purposes (EAP) rose from 25% to 45%, while specific purposes (ESP) halved to just 5%. Vocabulary and lexicogrammar have dominated the linguistic goals with around a third of studies each, the others consisting of grammar, discourse and error-correction. Writing was already the main language skill in early papers but has risen further, from 58% to 69%, to the detriment in recent years of spoken (17%) or receptive skills (10%), and translation (down from 18% to just 3%).

The duration of intervention has been reported variously, making accurate comparison all but impossible. However, for most studies (100) reporting in hours/minutes, the mean duration dropped by a third, from 16 hours 48 minutes to 11 hours 09 minutes. A wide variety of corpora has been used hands-on, with a preference for large national corpora with built-in tools: COCA rose from 12% to 48% while the BNC dropped from 29% to 15%; between them they feature in 46% of all studies, followed by custom-designed corpora at 38%. Variety is decreasing, with no instances in 2018-22 of the web-as-corpus, graded texts, literary corpora, multimodal or parallel corpora. Overall, corpus size is increasing, from a median of 26 million to 100 million tokens. Concordancers remain the tool of choice for learner-corpus interaction in 65% of the studies, compared to prepared materials or corpora integrated to a wider CALL package.

Sample sizes vary considerably; the median has increased from 29 to 55 participants, although only 31% included a non-DDL control group. The main focus has been on the exploration of language gains (70% of the studies), using corpora either as a learning aid (39%) or a reference resource (32%). Gains are generally measured via tests (43%) or learner productions (27%), though questionnaires are the most popular data-collection instrument overall, rising from 57% to 67% of the studies. This is mainly for learner attitudes (54%); other emic tools include interviews (20%) and diaries (18%), though these are sometimes for assessing what the participants do with the tools (28%) -computerized logs to track behavior are relatively rare (16%). 64% used more than one instrument; most employed quantitative designs with only 10% being purely qualitative, 32% using raw numbers, percentages or at most descriptive statistics, and 58% at least some inferential statistics; these figures remain largely unchanged.

CORPUS RESULTS

To triangulate these results, AntConc was used for a semi-automated analysis of our lemmatized Methodology subcorpus. These results are largely corroborated by key n-grams -clusters that are significantly more frequent in the recent studies. The cut-off point was set at 10% of RAs (5 of the 46 in 2018-22), resulting in 100 key 2-grams, 77 3-grams, 18 4-grams and 5 5-grams; the top 5 of each are presented in Table 3. As with keywords, some are not particularly revealing (the teacher, the participants were), but most of the others relate to corpus tools (Corpus of Contemporary American English) and features (the concordance lines), and in particular to research design: the comparison (group), by the researchers, of the target (word, etc.), before and after the (instruction, etc.), in the pre-and / post-(tests). Continuing down the lists supports these general findings.

DISCUSSION AND CONCLUSION

No synthesis should be treated as the last word: each has its own strengths and limitations in terms of scope, inclusion, research questions and analysis. Our definition of DDL may be too broad for some, too narrow for others; non-empirical studies are excluded, as are non-ranked journals and other publications, and texts in languages other than English; different coding or analysis might uncover additional methodological items. However, these criteria do provide greater focus on a near-exhaustive collection of generally high-quality papers that meet transparent criteria.

On the positive side, there is a substantial body of empirical DDL research in prestige journals, especially over the last 10 years. Further, quantitative meta-analyses have shown medium or large effect sizes for DDL (e.g., [START_REF] Boulton | Corpus use in language learning: A meta-analysis[END_REF]. Our analysis originally divided the collection into several periods, but showed remarkably little difference in proportions of methodological features at that level of granularity. This is an interesting finding in itself: either the research has always been solid right from the early days or, alternatively, there is a strong degree of inertianot learning from past difficulties or adopting better methodological practices.

In general terms of background, several comparisons can be drawn with Plonsky's (2023) review of applied linguistics. We too find a dominance of L2 English (see [START_REF] Forti | Corpus use in Italian language pedagogy: Exploring the effects of data-driven learning[END_REF][START_REF] Vyatkina | Corpus-based applications in language teaching and research: The case of data-driven learning of German[END_REF], for DDL of other languages) and regional hubs, but our East Asia hub is more prominent and North America less so. Most DDL research has featured relatively advanced-level learners compared to applied linguistics more widely, though numbers should be interpreted as approximate since "proficiency is notoriously difficult to define and measure" (Plonsky, 2023, p. 11), and is reported inconsistently. Nonetheless, the overall higher proficiency level of participants in DDL research may be an indication of perceived challenges and, thus, reluctance to explore it more widely with lower-proficiency learners. University contexts also dominate DDL research, though at a staggering 90% compared to 39% in Plonsky. Clearly DDL has similar methodological strengths and weaknesses as other applied linguistics research. As elsewhere, there is an acute need for more research in other contexts and with other profiles. More specifically, DDL may have already established its default niche, painting itself into a corner with relatively advanced learners in language-related courses at university in certain regions of the world. It may of course be that this is the optimum setting for DDL, or simply that researchers default to the convention of existing studies (Larsson et al., 2023); work is under way to examine this.

The aim is generally EGP, with vocabulary and lexicogrammar for writing including errorcorrection. One possibility would be to consider domain-specific corpora to help with both ESP and EAP, and with content: even a small corpus of relevant texts (e.g., research articles or textbooks) could help learners explore terminology in their field and how it is put together in academic writing, and to find information about their own area of specialization. Second, traditional corpus tools favor lexical searches: new tools might help at other levels, for example through texts annotated for discourse features. Third, there is no obvious reason why a DDL approach cannot apply to skills such as translation, which is all but invisible in our recent collection.

Such moves would involve rethinking DDL not as corpus + concordancer but, as we defined it, as using the tools and techniques of corpus linguistics: harnessing software to explore large quantities of text, discovering patterns in a constructivist approach to language learning. This shift involves creative imagining in using familiar tools, even Google or Linguee or ChatGPT, to bring DDL to the learners rather than asking them to make the leap to corpus linguistics -a kind of "DDL-lite". For spoken skills, free tools like YouGlish allow users to query a particular word or phrase and then to watch short chunks of videos where the item occurs, hearing it in context and reading it in the subtitles. Despite the qualitative difference (target items can only be accessed sequentially), the principles are essentially the same. These points all highlight the need for greater variety in the texts used. Large corpora dominate much DDL for good reason, but smaller ones potentially have a significant role to play: if learners are shown how to create their own collections for their individual interests, they might look at news articles about their favorite sport, a novel they are reading, scripts from a much-loved TV series, etc. Personal involvement would provide greater relevance and potentially familiarity with the contents. In addition to traditional concordancers, software such as SKELL and Linggle are simpler and more user-friendly, and with English-Corpora.org it is possible to input text for immediate frequency analysis (an indication of difficulty, or items worth learning), visualization of register distribution, topics, collocates by part of speech, clusters and concordance lines -all on the same page.

Returning to more mainstream methodological issues, one surprising find is that the duration of the target instruction has decreased; this however may be due in part to poor reporting, an area in need of improvement in much applied linguistics. At the same time, sample size has increased, though this is partly due to the use of more than one experimental group -the intact group is still the norm. An incidental problem of convenience sampling is that it encourages pre/post-test designs: only a third of our studies involved a control group, though some did compare two groups doing DDL in different ways. Increased collaborative research would provide larger samples and greater statistical power (cf. [START_REF] Plonsky | Sampling and generalizability in Lx research: A second-order synthesis[END_REF], and could provide more options for comparative designs with different learner profiles (e.g., L1s, disciplines, proficiency levels), corpora, software, types of scaffolding, etc. Direct comparison would provide greater confidence than by comparing different studies. A final problem concerns the instruments used: questionnaires are becoming ubiquitous, though they often feel as if they are tacked on at the end in lip-service to some idea of triangulation or mixed methods research. This can be valuable in some cases, but other instruments (e.g., trackers) should be considered for monitoring learners' behavior. Another pressing need is for more research on learners' use of DDL outside class and, indeed, after the end of the course: since DDL has been claimed to foster autonomy and life-long learning, we need to actually research this and not just take it as given. The same goes for other underpinnings of the approach, inferred but rarely explored first-hand as the topic of a specific research question: whether DDL really does foster noticing, language awareness, greater cognitive depth, etc. There is certainly room for DDL to take greater inspiration from SLA theories (O'Keeffe, 2021), and greater consideration of best practice. Lack of research here no doubt reflects the difficulty of implementing such studies -we need greater creativity and imagination in designing studies to test the foundations of DDL.

To finish on a positive note: there is considerable empirical research in DDL for L2 English, much of it of relatively high quality. It is always possible to improve on research designs, but many of the criticisms apply to applied linguistics in general, and it is encouraging that the latest period shows an increase in research-related terms. More fundamentally, it is important for methodologies to go hand-in-hand with research questions right from the start of study design. The question is not 'what can we do?', or 'what is easy to do?', but 'what do we want to find out?' and, following that, 'what is the best way to go about it?'.

TABLE 1 Journals, Journal Impact Factor (JIF) and RA counts

 1 Australasian Journal of Educational Technology, British Journal of Educational Technology, Education and Information Technologies, Educational Technology & Society, English in Education, Ibérica, IEEE Transactions on Learning Technologies, International Journal of Corpus Linguistics, International Review of Applied Linguistics in Language Teaching , Interpreter and Translator Trainer, Journal of Computer Assisted Learning, Journal of Computing in Higher Education, Language Awareness, Language Learning, Language Teaching Research, Lingua, Modern Language Journal, Perspectives: Studies in Translation Theory and Practice, RELC Journal, TESOL

	JIF 2022	Journal

  Table2shows keywords (items that are statistically significantly more frequent in the recent vs early RAs); all appear in at least 20% of papers (10/46), ranked by log likelihood. All items were analyzed in context to identify evolutions in terminology and themes related to methodology compared to the earlier publications.One group becoming more frequent in the recent period points to a greater focus on research:

intervention, instruction and participants. Other instances include test, which generally co-occurs with pre and post or t-; min and score, condition (group comparisons), indicate (reporting results), change (following instruction), author (citations), question (either research questions, or in questionnaires or interviews) and response, and open (questions). Another group reflects specific technology or aspects of corpora: COCA and AntConc; sub (corpus), file (in corpus creation) and (concordance) line.

Target (word, collocation, etc.)

shows different framing; meaning also highlights a move away from form alone, with which it is often contrasted. Other keywords suggest lexical preferences (e.g., learner vs student), a range of meanings (e.g., interactions between students, or with the software or corpus, or between different data sets, etc.), or quite general concepts (e.g., follow), and are harder to interpret.

TABLE 2 Keywords in 2018-2022

 2

	Rank Type	f Tar	f Ref	R Tar	R Ref	Keyness (LL)	Rank Type	f Tar	f Ref	R Tar	R Ref	Keyness (LL)
	1 post	154 122 30 31	68.28	15 response 109 108 21 43	32.15
	2 intervention 58 18 11 11	63.92	16 indicate	78 66 21 40	31.05
	3 min	62 25 15 11	57.30	17 pre	133 146 27 39	30.94
	4 coca	69 34 20 14	53.96	18 score	140 164 26 49	27.38
	5 teacher	274 328 35 63	50.39	19 author	76 72 21 34	24.58
	6 learner	404 549 38 72	48.45	20 meaning	98 110 23 45	21.52
	7 file	47 19 16 12	43.35	21 question	212 301 39 80	21.20
	8 participant 467 675 41 80	43.24	22 follow	198 277 44 87	21.08
	9 t	55 27 12 15	43.16	23 change	49 40 15 26	20.70
	10 test	452 652 34 66	42.23	24 open	52 44 24 27	20.70
	11 condition	75 54 12 24	38.24	25 sub	42 31 10 10	20.68
	12 theme	25	5 12	5	34.28	26 target	174 238 29 56	20.29
	13 interaction	42 21 12 13	32.40	27 instruction 111 134 27 44	19.88
	14 line	141 156 28 53	32.16	28 antconc	31 19 10 10	19.37
	Note: f=frequency; R=range; Tar=target corpus (2018-22); Ref=reference corpus

TABLE 3 Key n-grams in 2018-2022

 3

		Rank Frequency Range Keyness Effect Key n-gram
		1	110	24	88.97 0.0089 concordance lines
		2	83	18	73.14 0.0067 post test
	2-grams	3	116	23	67.07 0.0093 the teacher
		4	22	7	60.36 0.0018 the comparison
		5	201	33	54.17 0.0160 the participants
		1	12	5	36.63 0.0058 by the researchers
		2	25	13	33.90 0.0120 the concordance lines
	3-grams	3	29	14	32.11 0.0139 of the target
		4	53	21	30.74 0.0250 the participants were
		5	10	6	30.52 0.0048 english language teaching
		1	20	17	29.23 0.0749 corpus of contemporary american
		2	20	17	29.23 0.0749 of contemporary american english
	4-grams	3	8	6	27.19 0.0314 before and after the
		4	7	5	23.78 0.0276 in the pre and
		5	7	7	23.78 0.0276 pre test and post
		1	20	17	22.67 0.2759 corpus of contemporary american english
		2	6	6	17.67 0.1017 pre test and post test
	5-grams	3	11	6	13.66 0.1705 the pre and post tests
		4	14	12	13.04 0.2044 the corpus of contemporary american
		5	12	12	11.57 0.1805 of contemporary american english coca
	Note: Range refers to the latest period

Available in the IRIS repository; the full text corpus cannot be made public due to copyright restrictions.


M Aucejo

N Totaro

J.-L Guyader
email: jean-louis.guyader@insa-lyon.fr

IDENTIFICATION OF SOURCE VELOCITIES IN PRESENCE OF EXTERNAL CORRELATED SOURCES WITH THE INVERSE PATCH TRANSFER FUNCTIONS (IPTF) METHOD

The identification of source velocities has been developed through several methods such as Nearfield Acoustical Holography (NAH) or inverse Boundary Element Method (iBEM). However, these methods fail to identify source velocities in presence of disturbing sources and need free field conditions. The inverse Patch Transfer Functions (iPTF) method proposes to overcome this difficulty through the double measurement of pressure and particle velocity fields on the open surfaces of a virtual cavity arbitrarily defined around the source. In the method, measurement and identification surfaces are divided into elementary areas called patches. Source velocities are then obtained from the acoustic field and the Patch Transfer Functions computed by Finite Element Method. The iPTF method can also be applied on complex 3D geometries and measurements can be performed in situ. In the present paper, the basic equations of the method are reminded and an experimental validation on simple source geometry (two baffled pistons) is presented and discussed.

INTRODUCTION

Nearfield Acoustical Holography (NAH) and inverse Boundary Method (iBEM) are holographic methods developed to identify acoustic field on a structure when direct measurements are unavailable.

The NAH firsly introduced by Williams et al. [START_REF] Williams | Sound source reconstructions using a microphone array[END_REF] is based on the spatial Fourier Transform of the pressure field measured on a hologram near the source at a given frequency. This method allows determining the three-dimensional source velocity field. However, NAH remains mainly applicable to simple geometries.

The iBEM belongs to inverse Frequency Response Function methods (iFRF), which are based on experimental or numerical evaluation and inversion of transfer matrices. In iBEM, the BEM is used to compute the transfer matrices. Although iBEM can be applied on complex structures, the inversion of transfer matrices remains a delicate issue because of their ill-conditioning. Similarly, iBEM needs a fine mesh definition (6 nodes per wavelength) to identify acoustic field, wich can lead to an excessive amount of measurements.

Neverthless, hybrid methods like hybrid NAH [START_REF] Wu | Hybrid nearfield acoustical holography[END_REF] were developed to overcome the inherent limitations of NAH and iBEM. But like previous methods, it requires a regularization of matrices to limit the effects of measuring uncertainty and incompleteness of acoustic pressure field.

In this paper, the inverse Patch Transfer Functions method (iPTF) is introduced to identify source velocities. This formulation is derived from the PTF method [START_REF] Ouisse | Patch transfer functions ad tool to couple linear acoustic problems[END_REF], which is a prediction tool of acoustic pressure inside and outside a cavity containing sources and apertures based on substructuring and acoustic impedance concepts. In this method, the acoustical medium is indeed divided into subdomains, which coupling surfaces are divided into elementary areas called patches. The iPTF method proposes to identify source velocities on complex structure thanks to the double measurement of patch pressure and patch particle velocity on a virtual acoustic cavity surrounding the source. By this way, this method is applicable to non anechoic environments and in presence of a disturbing source, which allows performing measurements in situ.

THEORETICAL BACKGROUND OF THE IPTF METHOD

The iPTF method is based on the definition of an arbitrary virtual acoustic cavity surrounding the source to identify (see Figure 1). The acoustic problem is governed by the Helmholtz equation in the virtual cavity Ω i . On boundaries, a normal harmonic particle velocity Vn e jωt is imposed on the source surface S v , while the normal harmonic particle velocity V rad n e jωt on S m results from the radiation of the source in the acoustic medium. Consequently, the virtual cavity problem to solve is given by Equation 1.

         ∆p (M ) + k 2 p (M ) = 0 ∀ M ∈ Ω i ∂p ∂n (M ) = -jρω Vn (M ) ∀ M ∈ S v ∂p ∂n (M ) = -jρω Vn rad (M ) ∀ M ∈ S m (1)
Where ∆ is the Laplacian operator, ∂ ∂n the normal derivative, ω the angular frequency and k the acoustic wavenumber.

Patch Transfer Functions of the virtual cavity are firstly computed. For this purpose, the virtual cavity surface S is then divided into elementary areas S p and the problem to solve for each patch is given by Equation 2.

         ∆p (M ) + k 2 p (M ) = 0 ∀ M ∈ Ω i ∂p ∂n (M ) = -jρω Vni ∀ S p ∂p ∂n (M ) = 0 ∀ M ∈ S \ S p (2)
The mean pressure p j , respecting Equation 2, is defined as the superposition of mean pressures due to normal velocities on patches of the virtual cavity surfaces S v and S m (see Equation 3).

p j = P k=1 Z jk v k + N i=1 Z ji v i (3)
For a sake of simplicity, Equation 3 is rewritten under matrix form (see Equation 4).

{P j } = [Z jk ]{V k } + [Z ji ]{V i } (4)
In this equation, {V k } is the patch source velocity on surface S v and {V i } is the patch radiated velocity by the source on S m , while [Z jk ] (resp. [Z ji ]) represents the patch impedance matrix between an excited patch k (resp. i) and a receiving patch j. These patch impedance matrices are obtained from the modal expansion of the virtual rigid wall cavity as expressed in Equation 5.

Z jk = p j v k = - n jωρc 2 S k Λ n (ω 2 n -ω 2 + jη n ω n ω) φ n k φ n j (5)
Where ω n and φ n are respectively the natural frequencies and the modal shapes of the virtual rigid wall cavity. One has to notice that this modal basis can be obtained either analytically if the geometry of the cavity is simple or using numerical methods like FEM otherwise.

The aim of the iPTF method is to identify source velocities {V k }. For that purpose, one uses Equation 4, which after simple matrix manipulation allows computing source velocities as expressed in Equation 6.

{V k } = [Z jk ] -1 ({P j } -[Z ji ]{V i }) (6)
Equation 6 shows that only patch pressure P j and patch velocity V i on surface S m are required in the formulation to determine source velocities. These two last quantities are actually measured with a PU probe. Moreover, thanks to this integral formulation the method can be used in a non anechoic environment and in presence of sources located outside the virtual cavity. Like other source identification methods, the inversion of patch impedance matrices remains indeed a delicate issue because of ill-conditioning. However, a criterion has to be followed to avoid illconditioning. Indeed, the number of virtual cavity modes must at least be equal to the number of patches in order to describe properly the modal shapes and avoid under-determination of the problem.

EXPERIMENTAL VALIDATION OF THE IPTF METHOD

Experimental set-up

The experimental set-up consists of two baffled pistons exited in phase opposition. The rigid baffle is a wooden thick plate of dimension 700 × 600 × 40 mm, in which two loudspeaker are inseted as presented in Figure 2. Furthermore, the identification area of dimension 450 × 350 mm is divided into 30 patches and the reference measurement is performed by measuring the particle velocity at the center of each patch in the very nearfield of the source. One can also notice that PU measurements is performed in a non anechoic room, which means that reflected sound from room boundaries influences PU probe measurement.

Finally, a parallelepiped virtual cavity of dimensions 450 × 350 × 20 mm is defined around the source and divided into 139 patches, where pressure and particle velocity are measured at the center of each patch.

Validation of the method principle

To validate experimentally the principle of the method, Equation 6 is applied on the configuration defined in section 3.1. The application of the iPTF method from experimental data shows a agreement in both magnitude and distribution as presented in Figures 3 and4.

Robustness of the method

As exposed in section 2, the iPTF method is theoretically independent of the presence a stationary disturbing source outside the virtual cavity. To prove this theoretical fact, a third correlated louspeaker is added (see Figure 5). This loudspeaker modifies the acoustic field in the measurement area, including the source surface. Consequently, the reference measurement is performed again to take into account the presence of the disturbing loudspeaker. As presented in Figure 6, the influence of the third loudspeaker is non negligible since the difference between pressure power spectrum measured with or without the disturbing source reaches 5 dB at certain frequencies.

In this example too, the iTPF method allows proper identifications in magnitude and spatial distribution since in Figure 7 identification differs from the reference by 3 dB at most on the considered frequency range. The comparison between reference map and the identified one in Figure 8 gives very satisfying results. That proves the efficiency of the method in presence of correlated disturbing source, which is an importante issue to identify source velocities in situ.

CONCLUSION

The iPTF method allows identifying source velocity field. Through a simple experimental validation, the ability of the method to identify and reconstruct the source velocity field with or without the presence of a stationary correlated disturbing source have been underlined. The main advantage of the method is the combined use of the integral formulation, FEM and PU measurements, which allows overcoming inherent limitations of classical methods. By this way, the method is independent of the environment, provided it is stationary, and can be applied on complex source geometry.

Figure 1 .

 1 Figure 1. Definition of the virtual acoustic cavity surrounding the source

Figure 2 .

 2 Figure 2. Definition of the experimental set-up

Figure 3 :

 3 Figure 3: Comparison of quadratic transfer function Velocity/Pressure on the first excited patch for a 450 × 350 × 20 mm virtual cavity, (-) Reference, (--) Identification

Figure 4 :

 4 Figure 4: Comparison between (a) the reference map measured at 5 mm from the source and (b) the identification map obtained experimentally with the iPTF method at 240 Hz for a 450 × 350 × 20 mm virtual cavity

Figure 6 :

 6 Figure 6: Comparison between the pressure power spectrum obtained on a patch of the measurement surface (-) without and (--) with the disturbing source versus frequency

Figure 7 :Figure 8 :

 78 Figure 7: Comparison of quadratic transfer function Velocity/Pressure on the first excited patch in presence of a correlated disturbing source, (-) Reference, (--) Identification