David Alfter

Matthew Shardlow
email: m.shardlow@mmu.ac.uk

Aline Villavicencio

Arne Jönsson
email: arne.jonsson@liu.se

Ingibjörg Elham Akhlaghi
email: elhamakhlaghi80@gmail.com

Iða Auðunardóttir

Branislav Bédi
email: branislav.bedi@arnastofnun.is

Hakeem Beedar
email: hbeedar@hotmail.com.au

Harald Berthelsen
email: berthelh@tcd.ie

Ingibjörg Iða Auðunardóttir

Cathy Chua
email: cathyc@pioneerbooks.com.au

Catia Cucchiarini
email: c.cucchiarini@let.ru.nl

Brynjarr Eyjólfsson

Nedelina Ivanova
email: nedelina@shh.is

Christèle Maizonniaux
email: christele.maizonniaux@flinders.edu.au

Neasa Ní Chiaráin
email: neasa.nichiarain@tcd.ie

Manny Rayner
email: emmanuel.rayner@unige.ch

John Sloan
email: sloanjo@tcd.ie

Sigurður Vigfússon

Ghil ' Ad Zuckermann
email: ghilad.zuckermann@adelaide.edu.au

Rémi Cardon

Thomas Franc

Renate Hauser
email: renateines.hauser@uzh.ch@volkcl.uzh.ch

Jannis Vamvas
email: vamvas@volkcl.uzh.ch

Sarah Ebling
email: ebling@volkcl.uzh.ch

Martin Volk

Evelina Rennes
email: evelina.rennes@liu.se

Marina Santini
email: marina.santini@ri.se

Ludivine Javourey-Drevet

Stéphane Dufau

Johannes C Ziegler

Núria Gala
email: nuria.gala@univ-amu.fr

Camille Pirali
email: camille.pirali@student.uclouvain.be

Thomas Franc ¸ois
email: thomas.francois@uclouvain.be

N Úria Gala

Nicolas Hernandez
email: nicolas.hernandez@ls2n.fr

Nabil Oulbaz

Tristan Faine
email: tristan.faine@etu.univ-nantes.fr

Reading Assistance through LARA, the Learning And Reading Assistant

Keywords: CALL, multimodality, reading, open source, evaluation Complex Word Identification, Lexical Complexity Prediction, Text Simplification word difficulty, readability, lexicography Corpus, Multilinguality, Simplified Language, Accessibility, Text Simplification Provider automatic text simplification, easy-to-read, automatic text adaptation reading practice, educational device, French L1, text adaptation, text comprehension activities Text Simplification, Complex Word Identification, Lexical Difficulty, Lexical Complexity Prediction, Annotation Tool open-source, free, corpus, toolkit, readability assessment, French

We present an overview of LARA, the Learning And Reading Assistant, an open source platform for easy creation and use of multimedia annotated texts designed to support the improvement of reading skills. The paper is divided into three parts. In the first, we give a brief summary of LARA's processing. In the second, we describe some generic functionality specially relevant for reading assistance: support for phonetically annotated texts, support for image-based texts, and integrated production of text-to-speech (TTS) generated audio. In the third, we outline some of the larger projects so far carried out with LARA, involving development of content for learning second and foreign (L2) languages such as Icelandic, Farsi, Irish, Old Norse and the Australian Aboriginal language Barngarla, where the issues involved overlap with those that arise when trying to help students improve first-language (L1) reading skills. All software and almost all content is freely available.

Message from the General Chair

Recent studies show that the number of children and adults facing difficulties in reading and understanding written texts is steadily growing. Reading challenges can show up early on and may include reading accuracy, speed, or comprehension to the extent that the impairment interferes with academic achievement or activities of daily life. Various technologies (text customization, text simplification, text to speech devices, screening for readers through games and web applications, to name a few) have been developed to help poor readers to get better access to information as well as to support reading development. Among those technologies, text adaptations are a powerful way to leverage document accessibility by using NLP techniques.

The "Second Workshop on Tools and Resources for REAding DIfficulties" (READI), collocated with the "International Conference on Language Resources and Evaluation" (LREC 2022), aims at presenting current state-of-the-art techniques and achievements for text adaptations together with existing reading aids and resources for lifelong learning. The materials can be addressed to children struggling with difficulties in learning to read, to the community of teachers, speech-language pathologists and parents seeking solutions, but also to adults and professionals involved with adults struggling with reading (illiterates, aphasic readers, low vision readers, etc.).

In the second edition of READI, 14 propositions have been submitted from which 10 were accepted. Thus, the total rate of accepted papers is 71%. This is similar to the first edition which had an acceptance rate of 66%. The accepted papers came from 48 different authors from 14 different countries (Belgium 9, France 9, Switzerland 6, Iceland 5, Poland 4, Australia 3, Ireland 3, Spain 3, Sweden 3, Germany 2, Iran 1, Netherlands 1, United Kingdom 1, and Independent scholar 1). READI also features two invited speakers, Carolina Scarton (University of Sheffield, UK) and Arne Jönsson (Linköping University, Sweden). Moreover, we have decided to include in the programm a one hour slot for three papers from the 1st READI Workshop in 2020 (as the workshop could not take place due to the covid crisis).

We are thankful to the authors who submitted their work to this workshop, to our Program Committee members for their contributions, to the reviewers and the additional reviewers who did a thorough job evaluating submissions, to Carolina Scarton and Arne Jönsson who kindly accepted to be our invited speakers, and to LREC committee for including this workshop into their program. The workshop has been supported by Aix Marseille Université, Laboratoire Parole et Langage (CNRS UMR 7309) and the Institute Language, Communication and the Brain (ILCB), funded by the French National Agency for Research (ANR, ANR-16-CONV-0002) and the Excellence Initiative of Aix-Marseille University A*MIDEX (ANR-11-IDEX-0001-02).

Introduction and overview

LARA

(https://www.unige.ch/callector/ lara/) is an open source learning-by-reading platform under development by an international consortium since 2018. Starting at the University of Geneva, the user base has grown quickly and now includes groups in over a dozen countries. LARA supports easy construction of annotated multimodal texts using open source tools which can either be invoked from the command-line or, more commonly, through an online portal. These texts typically include various features for reading assistance such as integrated audio, translations, and an automatically generated concordance. A screenshot showing a page from a LARA text is shown in Figure 1. The basic idea of adding multimedia annotations to texts in order to help non-L1 language learners is natural, and has been implemented in some form in many other platforms; prominent examples include LingQ1 , Learning With Texts2 , the Perseus Digital Library's Scaife viewer3 and Clilstore4 . What primarily distinguishes LARA from these is the project's open source nature, where new features are added in a bottom-up process driven by the demands of a diverse community involved in many different kinds of language-related projects.

As noted, LARA has originally been developed to help people learn and read non-L1 languages. However, the boundary between non-L1 and L1 turns out to be less clear than we had initially expected. For example, looking ahead to §4.1., a major hurdle for beginner learners of Irish is the opaque writing system, which makes it unusually difficult to acquire a good understanding of the letter/sound rules.

The problems these people face are not dissimilar to those experienced by people with L1 reading difficulties. In addition, recent extensions to the LARA functionality are moving in a direction that could make it more directly useful as a tool for providing assistance to people with reading difficulties. This paper is intended to provide a self-contained overview of the reading assistance facilities in LARA that may be relevant to the reading difficulties community. We start by presenting a brief summary of the core LARA functionality ( §2.), then describe recently added functionality particularly relevant in the present context ( §3.). In §4. we describe some LARA projects where the issues would appear to overlap with those arising when helping people with reading difficulties. The final section summarises and looks ahead.

Core LARA functionality

The core of LARA is a set of tools that make it easy to convert text into the multimodal annotated form illustrated in Figure 1. The conversion process consists of the following Figure 1: Example of Irish LARA content, Fairceallach Fhinn Mhic Cumhaill, ('Fionn's burly friend'), reproduced from [START_REF] Zuckerman | Lara in the service of revivalistics and documentary linguistics: Community engagement and endangered languages[END_REF]. A 'play all' audio button function is included at the top of the page to enable the listener to hear the entire story in one go (1). The text and images are in the pane on the left hand side. Clicking on a word displays information about it in the right hand pane. Here, the user has clicked on bhí = "to be (past tense)" (2), showing an automatically generated concordance; the lemma bí; and every variation of bí that is in this text (3). Hovering the mouse over a word plays audio and shows a popup translation at word-level. Clicking on a loudspeaker plays audio for the entire sentence as well as showing a popup translation (4). The back-arrows (5) link each line in the concordance to its context of occurrence. A link to the document can be found on the LARA examples page. 6 steps:

Segment: Segment the text. For European languages, this means splitting up lists of words into sentence-length segments using a sentence tokeniser. The result is then in general manually post-edited.

Tag: Tag the text, to mark each word with its lemma form and (optionally) part of speech. This is needed in order to build the lemma-oriented concordance. When a tagger/lemmatiser is available, this is first used to perform the tagging automatically, after which the result is again manually post-edited. Several tagger/lemmatisers are now integrated into LARA, covering over 20 languages.

Identify resources needed: Process the text to create a set of resource files which specify other annotation data that needs to be added. The most important are i) associations of words and segments with audio files, ii) associations of words and segments with translations, and iii) potential occurrences of multiword expressions (MWEs) taken from an MWE lexicon for the language in question.

Instantiate resources: Upload the resource files to tools which support easy entry of the missing information. Audio files are created through a user-friendly online recording tool. Translations are entered through a spreadsheet-like interface. Candidate MWEs are confirmed or rejected through another interactive tool.

Create pages: Combine all the information to create the multimedia pages.

These operation can either be carried out directly using command-line tools, or can be invoked through the LARA portal, a free online service that provides a user-friendly wizard-style interface. Full details and examples can be found in the online documentation 7 .

Functionality

We describe three pieces of LARA functionality introduced over the last year that are potentially relevant to helping people with reading difficulties: phonetic texts, annotated images and picture lexica, and integrated TTS.

Reading assistance through phonetic texts

LARA documents were originally conceived as texts with a hierarchical structure consisting of pages, segments and words, where the words are associated with lemmas. In order to address the needs of students who are uncertain Clicking on a letter group, here "ear" in "tear" at the end of the penultimate line, highlights the group, plays the sound, and shows a phonetic concordance on the right; clicking on a speaker icon plays audio for whole of the preceding word. A link to the text is posted on the LARA examples page.

about letter/sound correspondences, we have recently extended the framework to allow the option of creating texts annotated at the phonetic level. A "phonetic" LARA text is hierarchically divided into pages, words and letter-groups, where each letter-group is associated with a phonetic value. The same notation is used for both types of text, and nearly all of the processing associated with normal (wordoriented) LARA texts carries over to phonetic texts; in particular, a compiled phonetic text contains a phonetic concordance, giving examples of contexts where each phonetic value occurs. A playful example illustrating the "phonetic text" functionality is shown in Figure 2. It would be extremely laborious to construct phonetic LARA texts by hand, and there is a script that converts a normal text into the corresponding phonetic version. This post-processes the internalised text to convert each word into a corresponding phonetic version, while keeping formatting unchanged. For languages which are written completely phonetically, this only requires the annotator to supply the list of phonetically meaningful letter groups defining the orthography of the language. An example for Barngarla (cf. §4.2.) is Mangiri Yarda; this also uses the annotated image functionality described in §3.2.. For languages where online phonetic lexica exist, phonetic versions of most words can be read off the lexicon; free phonetic lexica for many languages are for example available from the IPA-dict project. 8 The challenge is to align the letters with the phonetic symbols. At the moment, the conversion script helps the annotator compile an aligned phonetic lexicon, where typical entries are as illustrated in 8 https://github.com/open-dict-data/ ipa-dict Figure 4. The script creates new entries automatically using a simple dynamic programming method which maximises the number of alignments already seen in the lexicon (this idea is partly inspired by the one from [START_REF] Jiampojamarn | Letterphoneme alignment: An exploration[END_REF]), after which a human annotator cleans up the result. Further details are given in the online documentation. 9Once a reasonable number of examples of aligned words have been collected, error rates become low and the cleaning-up process is quick. We present the results of a preliminary evaluation for English and French to support this claim. In English, we began by constructing an initial aligned word lexicon for a few small texts, the largest of which was "The Chaos" (cf. Figure 2). This produced a total of 990 aligned words, which included 264 unique primitive grapheme-sequence/phonemesequence correspondences; phonetic transcriptions were taken from the UK English IPA-dict resource. We then ran the alignment guessing script on the text of an English translation of Saint-Exupèry's Le Petit Prince. This contained 1833 unique words, of which 309 were already in the aligned-word lexicon. Of the remaining 1524, 78 were not in the IPA-dict lexicon, most of them either because they were heteronyms ("close", "live", "wind") or proper names ("africa", "antoine", "siberia"). Editing the aligned lexicon took an expert annotator, one of the authors, about three hours. Comparing the edited and raw versions, we found that the script had correctly aligned 1410/1524 of the new words (92.5%) and 9041/9580 of the (a) (b) Figure 3: Toy example of a piece of image-based text based on a simple cartoon, taken from [START_REF] Bédi | Using lara to create image-based and phonetically annotated multimodal texts for endangered languages[END_REF]. The LARA source (a) is given above. The screenshot below (b) shows the tool used to create the word locations file. The top control allows the annotator to choose the text to annotate, after which the slider with the series of thumbnails allows them to choose a page by its image. The bottom left pane presents the selected image, and the bottom right pane the associated words. The annotator can draw a polygon on the left and save it to a word, or select a word on the right to show the current polygon. Here, the annotator has just selected the word "man" on the right, showing the polygon for the picture of the man on the left. The speaker and pencil icons optionally associate audio or text with a whole line. A link to the LARA document is posted on the LARA examples page under "Two cartoons". graphemes (94.4%). Looking only at the subset for which IPA-dict entries were available, the figures were 1410/1446 (97.5%) for words and 9041/9102 (99.3%) for characters. Audio for the phonetic content was recorded by one of the authors, a native speaker of English. The French experiment was similar, though the initial version of the aligned lexicon was based on a smaller sample of language. This time, we used the original French edition of Le Petit Prince as the evaluation text. This contained 2583 unique words, of which 559 were already in the alignedword lexicon. Of the remaining 2024, 32 were not in the IPA-dict lexicon. Again, editing the guessed aligned lexicon took about three hours. The script correctly aligned 1876/2024 of the new words (92.7%) and 13722/14191 of the graphemes (96.7%). For the subset where IPA-dict entries were available, the figures were 1876/1992 (94.2%) and 13722/13966 (98.3%) for graphemes. This time, we experimented with a different strategy for creating the phonetic audio, and generated it using one of the French voices on the IPA-reader site. 10 . Links to the English and French versions are posted on the LARA examples page under the titles "The Little Prince" and "Le petit prince". "admirateur" "a|d|m|i|r|a|t|eu|r" "a|d|m|i|K|a|t|oe|K" "ainsi" "ain|s|i" "Ẽ|s|i|" "alors" "a|l|o|r|s" "a|l|O|K|" Figure 4: Examples of entries from French aligned pronunciation lexicon. Several letters can map into one (beginning of "ainsi"), and letters can map into the empty string (end of "alors").

Annotated images and picture lexica

Another extension to the original text-based LARA document structure concerns images. LARA has always supported inclusion of images using the HTML <img> tag, but these were represented internally as atomic constituents without internal structure. Layout was added using other HTML tags. Although this model works well for many documents, it ignores the fact that a written text is not just a collection of strings but also a visual object. For some kinds of documents, for example picture-books and posters, the visual content can be as important as the words. In order to address these issues, which are particularly relevant to helping students with reading difficulties, we have recently added new functionality to allow "annotated images" as components of a text. A component of this kind is delimited using the <annotated_image> tag. It must contain exactly one <img> tag and some text, where the text is interpreted as being associated with locations in the image. During the processing phase which identifies resources needed for a text, the images and associated words are extracted, after which they are uploaded to an online graphical tool where the annotator can draw the outline for the location of each word in an image. Figure 3 illustrates. The graphical correlate of a word can, but does not need to be a graphical representation of the word; it can equally well be a part of the image associated with the word. So for example the word "apple" could be associated in the image with the handwritten text A PPL E , but it could also be associated with an area of the image showing an apple. In the final LARA text, the locations in the image marked as associated with words react to clicking or hovering actions. Another piece of image-related functionality is the provision of support adding a "picture lexicon" to a text, which associates some lemmas with graphical images. This has already been used for the Barngarla project (cf. §4.2.).

The initial text "Welcome to Country with picture lexicon" linked from the examples page was warmly approved by the Barngarla elders guiding the language revival process.

Reading assistance through integrated TTS

Integration of TTS was primarily implemented to support the Irish group (cf. §4.1.), who have from the start used it exclusively to create Irish language audio. Initially, other groups were sceptical about creating LARA content that used TTS audio, believing that the quality would be insufficient compared to human-recorded audio. Two collaborative evaluation exercises have however demonstrated that, for many languages, TTS works much better than was generally expected. The first of these exercises was carried out during Q1 2021 and involved the Australian, Icelandic, Iranian, Irish, Dutch, Polish, Slovak and Swiss groups. About twenty LARA documents, in various languages, were produced in both TTS and human audio form and compared by 130 evaluators using an anonymous web form. One expects TTS audio to be much quicker to produce, but of lower quality: the goal was to obtain quantitative and qualitative data exploring the issues. The results were presented at EUROCALL 2021 [START_REF] Akhlaghi | Assessing the quality of TTS audio in the LARA learning-byreading platform[END_REF]. To our surprise, TTS audio was in fact rated equal to or better than human audio in three of the ten languages. A follow-on study was carried out in Q1 2022 and will be presented at the LREC 2022 conference. Since it used data taken from a uniform text, different translations of Saint-Exupèry's Le petit prince, comparisons between languages were more obviously meaningful, and the number of evaluators was approximately doubled. The results were similar to those of the first study. Although the quality of TTS varied widely between languages, the best TTS voices were of a quality comparable with non-professional human voices and again were in some cases preferred.

Example projects

LARA was originally designed for creating annotated texts that would improve learners' reading and listening skills in L2 languages. After three years of experience in using the tool, it turns out that the dividing line between L2 and L1 is less clear than we had realised, and that the issues appear to overlap to a considerable extent. We briefly describe some substantial projects exemplifying this observation. In §4.1. we consider the paradoxical case of Irish, where a country's official first language is simultaneously an endangered language. §4.2. describes use of LARA with Barngarla, an Australian Aboriginal language which for several decades was considered dead, but which is now being revived by ethnic Barngarla people. In §4.3. we look at texts designed to help Deaf Icelandic children improve their reading skills in Icelandic, and in §4.4. at Old Norse, the archaic form of Icelandic taught as an obligatory subject in Icelandic schools. §4.5. reviews a project carried out in Iran, where a series of Farsi readers have been converted into LARA form.

Online resources for reading assistance in Irish

Irish is in the possibly unique position of being both the official first language of a sovereign state and also an endangered language. It is a community language only in relatively small regions (Gaeltacht regions) in the West of Ireland, with daily speaker numbers of about 20,000, or less than 0.5% of the Irish population [START_REF] Cso | Census 2016 Profile 10 -Education, Skills and the Irish Language[END_REF]. At the same time, it is an obligatory subject in schools, with 700,000 learners in the education system in the Republic of Ireland [START_REF] Ní Chiaráin | The potential of text-to-speech synthesis in computer-assisted language learning: A minority language perspective[END_REF].

Teaching and learning Irish presents multiple challenges, and learning to read Irish is one of them. The first language of most learners will be English, a Germanic language whose structure diverges substantially from that of the Celtic language Irish; the basic word-order of Irish is different (VSO as opposed to English SVO), and it is highly inflected, with up to 42 inflected forms for verbs. A striking feature of the sound system is the contrast of palatalised and velarised consonants, with a very large inventory, relative to English. This feature is partially obscured, and complicated for learners, by the notoriously opaque writing system, and the link of the sounds to written forms is often poorly understood. The initial sounds of lexical items 'mutate' in certain grammatical contexts, so that e.g., in a word like bord 'table' it may be [b], [w], [v] or [m]. Although there is an agreed standardised written form, there is no single spoken standard, but rather three major dialects. Teachers are typically second language learners themselves, and their own confidence in the language can be problematic. They often feel overburdened with the major responsibility placed on them in the revitalisation and maintenance initiative, but report inadequate resources and training to fulfil it [START_REF] Dunne | Primary teachers' experiences in preparing to teach Irish: Views on promoting the language and language proficiency[END_REF].

In this context, it turns out that LARA has much to offer in terms of reading assistance. Using the synthetic voices developed for the main Irish dialects by the ABAIR project (ABAIR, 2020) and integrated into LARA, it is easy to link text to audio in any of the three dialects, bringing a native speaker model directly into the classroom; the lemmabased concordance similarly allows the learner to access the dictionary form of any word with a single click. Starting with pilot LARA adaptations of traditional Irish folkstories, the team at Trinity College Dublin have created a substantial set of Irish reading material in LARA form, posted on the An Scéalaí ("The Storyteller") CALL platform. 11 . User feedback has been extremely positive. In a recent survey, for example, 92% of 494 adult respondents reported that using An Scéalaí had a positive impact on their language learning journey. 90% of same stated they would like to continue using the platform into the future. Many users commented that LARA made complex texts accessible -learners felt they engaged more deeply and spent more time on 'difficult' reading materials than they would otherwise have done if presented to them in a more traditional format. Barngarla is an Australian Aboriginal language belonging to the Thura-Yura language group, a subgroup of the large Pama-Nyungan language family. Typically for a Pama-Nyungan language, Barngarla has a phonemic inventory featuring three vowels ([a], [i], [u]) and retroflex consonants, an ergative grammar with many cases, and a complex pronominal system. Unusual features include a number system with singular, dual, plural and superplural and matrilineal and patrilineal distinction in the dual.

During the twentieth century, Barngarla was intentionally eradicated under Australian 'stolen generation' policies, the last original native speaker dying in 1960. Language reclamation efforts were launched in 2011 [START_REF] Zuckermann | Revivalistics: From the Genesis of Israeli to Language Reclamation in Australia and Beyond[END_REF]. Since then, a series of language reclamation workshops have been held in which about 120 Barngarla people have participated. The primary resource used has been a dictionary, including a brief grammar, written by the German Lutheran missionary Clamor Wilhelm Schürmann [START_REF] Schürmann | A Vocabulary of the Parnkalla Language. Spoken by the natives inhabiting the western shore of Spencer's Gulf. To which is prefixed a collection of grammatical rules[END_REF][START_REF] Clendon | Clamor Schürmann's Barngarla grammar: A commentary on the first section of A vocabulary of the Parnkalla language[END_REF].

Other published resources for Barngarla, non-existent ten years ago, are now emerging. The most visible example to date is Barngarlidhi Manoo (Zuckermann and the Barngarla, 2019), a Barngarla alphabet book/primer compiled by Ghil'ad Zuckermann in collaboration with the nascent Barngarla revivalistic community. A first step in evaluating the possible relevance of LARA to Barngarla was to convert this book into LARA form [START_REF] Butterweck | Easy construction of multimedia online language textbooks and linguistics papers with LARA[END_REF]. The LARA reading assistance functionality is primarily used to attach audio recordings to Barngarla language: words and phrases marked in red can be played by hovering the mouse over them.

A second resource was produced as part of the "Fifty Words Project" 12 , which collects together fifty basic words such as "fire", "water", "sun" and "moon" for several dozen Aboriginal languages. The Barngarla version, recorded by ethnic Barngarla language custodian Jenna Richards from Galinyala (= Port Lincoln), is available on the Fifty Words page. A third Barngarla text, Mangiri Yarda ("Healthy Country") [START_REF] Zuckermann | Mangiri Yarda (Healthy Country: Barngarla Wellbeing and Nature)[END_REF] has been designed as a teaching resource. In contrast to Barngarlidhi Manoo, which is almost exclusively focused on vocabulary, Mangiri Yarda introduces some grammar. Links to all of these texts are posted on the LARA examples page.

Helping Deaf Icelanders improve their reading skills

Although Icelandic is the primary language of Iceland, Deaf children usually grow up learning a signed language as their first language. In practice, written Icelandic is not perceived as an L1 for these children, so tools that can help them make progress in reading are potentially very useful.

It turned out to be quite easy to extend LARA so that it can support this kind of scenario: basically, all that was necessary was to arrange things so that recorded signed video can systematically be used as an alternative to recorded au-dio. Thus a LARA text of this type is written in Icelandic, but words and sentences are associated with Icelandic Sign Language (ÍTM) signed videos. The signed video for a word is accessed by clicking on the word; the signed video for a sentence is accessed by clicking on a camera icon inserted at the end of the sentence. (In 'video mode', the camera icon replaces the usual loudspeaker icon). Videos are recorded using the same third-party recording tool as is used for recording audio content; the tool had already been adapted for this purpose in a previous project (Ahmed et al., 2016). The workflow for recording is modality-independent. The LARA portal creates the recording script from the text and uploads it to the recording tool; the voice talent/signer records the audio/video from the script; at the end, the portal downloads the recorded multimedia and inserts it into the LARA document.

A link to an initial example of a LARA document of this kind, a children's story about 2.7K words long, is posted on the LARA examples page. The student who created the signed content turned to two members of staff at the Center for feedback. One is a native ÍTM signer and the other has worked as an sign language interpreter for over two decades. There were many things to consider, as ÍTM is not a standardised language, even to the extent that the basic word order is unclear: research [START_REF] Brynjólfsdóttir | Hvað gerðir þú við peningana sem frúin í Hamborg gaf þér? Myndun hvspurninga í íslenska táknmálinu[END_REF] shows that subjects accept both SOV and SVO word orders. The central issue was the question of whether the signed translation of the text should be true to the Icelandic original or re-expressed in ÍTM. One argument is that, as a tool to learn written Icelandic, the translation should be faithful to the source so that ÍTM signs corresponding to the Icelandic words appearing there. The argument in the opposite direction is that a free re-interpretation is better suited to helping Deaf children understand the signed content. In the end an interpreting strategy was preferred for three reasons. Comprehension of the signed text is crucial for Deaf children; the interpreting strategy seemed to be a better fit to the content of a children's book; and in LARA learners can click on a word in the Icelandic text to see the ÍTM sign, if the corresponding sign did not appear in the freely translated segments.

Assistance in reading Old Norse epic poems

Old Norse, the language spoken in what is now Scandinavia from the 7th to the 15th century, is an important part of Icelandic culture. The linguistic evolution of Icelandic has proceeded more slowly than that of the mainland languages (Danish, Norwegian and Swedish), and it is close enough to Old Norse that Old Norse literary works are still more or less comprehensible; a reasonable comparison point for anglophones might be Chaucerian English. Old Norse language and literature is an obligatory subject in Icelandic secondary schools. It is however clear that many students find it challenging. They are particularly challenged by the Poetic Edda, a classic poem-cycle first written down in the late 13th century, which occupies a central place in the Old Norse canon. The dense, allusive language is much harder to understand than that in prose works, and the less motivated students often experience it as close to incomprehensible.

Particularly as a tagged version of the Poetic Edda already existed, the group at the Árni Magnússon Institute for Icelandic Studies felt that this combination of circumstances made it a good target for conversion into LARA form. Three of the best-known poems from the cycle -the Völuspá, Hávamál and Lokasenna -have now been completed, and several more are in preparation. The Völuspá project is presented in [START_REF] Bédi | Constructing an interactive old norse text with lara. CALL for widening participation: short papers from[END_REF]; as described there, initial feedback from Icelandic users has been very positive. All three of the Eddaic poems so far released have also been used as the basis of reading groups on the popular Goodreads review site. 13 . They attracted a small but enthusiastic audience, with a total of 185 posts for the three groups.

Online resources for reading assistance in Farsi

The Ferdowsi University of Mashhad (FUM; third highest ranked university in Iran) has used LARA since shortly after the inception of the LARA project. FUM began by developing short LARA texts in Farsi, for use in a Farsi course for Arabic-speaking students at FUM. Early results are reported in [START_REF] Akhlaghi | Overview of LARA: A learning and reading assistant[END_REF]. This pilot exercise was successful enough that Iranian funding was granted to convert an five-volume series of Farsi textbooks, developed at FUM by Professor Ehsan Ghabool, into LARA form. The project was completed during Q1 2021, and the result is now being used at FUM's International Center for Teaching Persian to Non-Persian People14 .

Summary and further directions

We have presented a brief overview of the LARA community and platform, focusing on issues that overlap with those relevant to supporting people with reading difficulties and illustrating with some practical use cases. Work in several of these areas is under active development. We are particularly interested in exploring the possibilities opened up by the new "phonetic text" and "annotated image" functionalities, and welcome suggestions from the reading difficulties community about ways to repurpose the LARA technology to this new domain.

Bibliographical References

ABAIR, (2020). ABAIR: An Sintéiseoir Gaeilge -The Irish Language Synthesiser ABAIR. http://www.abair.ie. As of 8 September 2020. Ahmed, F., Bouillon, P., Destefano, C., Gerlach, J., Hooper, A., Rayner, M., Strasly, I., Tsourakis, N., and Weiss, C. (2016). Rapid construction of a web-enabled medical speech to sign language translator using recorded video.

In Proceedings of FETLT 2016, Seville, Spain.

Introduction

Lexical Complexity Prediction (LCP) has applications in Text Simplification [START_REF] Zampieri | Complex word identification: Challenges in data annotation and system performance[END_REF], as well as Readability Assessment [START_REF] Ehara | Interpreting neural CWI classifiers' weights as vocabulary size[END_REF]. It is the task of identifying how complex a word is likely to be for an end user. Similarly, Complex Word Identification (CWI) is the task of identifying whether a word is complex or not. In both these tasks, disagreements naturally arise between annotators seeking to faithfully give their subjective opinions on the difficulty of the words in question. Take, for example the following sentence, taken from the CWI2018 shared task data (Yimam et al., 2017):

"A man and a woman questioned on suspicion of assisting an offender have been released."

The marked token (offender) may be considered complex by some and simple by others. In fact this example split the pool of annotators, being marked complex by 50% of the annotators and simple by the rest. This is not always the case though, and there are also words that are consistently annotated. For example, in the LCP2021 data [START_REF] Shardlow | Predicting lexical complexity in english texts: the complex 2.0 dataset. Language Resources and Evaluation[END_REF], the following example is given:

"Similarly, changes in synaptic plasticity due to Ca2+-permeable AMPARs [51,52,60], e.g., in piriform cortex, might alter odor memorization processes."

Clearly, here the entire context is very hard to understand, and the term in that context (synaptic plasticity) is inaccessible to a non-domain expert. As such, the term was annotated as the highest level of difficulty by all but one annotator. Similarly, in the following context, also taken from LCP2021 all annotators chose the easiest level of difficulty for the token hand:

"But he, beckoning to them with his hand to be silent, declared to them how the Lord had brought him out of the prison."

We can draw from these few examples that there are clear cases where annotators agree, and clear cases where annotators do not agree. These exist across multiple datasets and are not merely a factor of the token's complexity (i.e., we may naïvely assume that everyone agrees on simple words, but differs on complex words, or vice versa). For sake of ease, we will refer to subjectivity in the remainder of this paper in the context of the subjectivity of complexity. These initial insights allow us to form the following research hypotheses and questions:

RQ1: Can we distinguish words with subjective or consistent complexity? Are they the same across different datasets? RH1.1: We can identify from existing datasets clear patterns of subjective and nonsubjective complexity annotations.

RH1.2

The subjective and non-subjective complex words will be the same across datasets.

RQ2: What factors model subjectivity? RH2.1: Lexical ambiguity will correlate to subjectivity.

RH2.2: Lexical frequency will correlate to subjectivity.

RH2.3: Psycholinguistic norms will correlate to subjectivity.

RQ3: Can we reliably predict which words are likely to be consistently annotated as complex or simple, and which words are likely to be subjectively complex? RH3.1: Classical machine learning classifiers can predict subjectivity based on the lexical factors identified.

To answer these questions, the remainder of the paper is structured as follows: We define the notions of complexity and subjectivity in Section 2 and explore this in a concrete manner in Sections 3 and 4, which cover datasets from two shared tasks. We also discuss the internal mechanisms that were used during annotation and demonstrate the subjectivity that is present, which addresses RH1.1. Section 5 compares the two datasets in terms of the words that are found to be consistent or subjective and addresses RH1.2 accordingly. Section 6 identifies a number of pertinent features taken from the CWI/LCP literature and uses statistical methods to determine their relation to the subjectivity, addressing RH2.1-3. We build various classifiers to predict subjectivity in Section 7, which allows us to answer RH3.1. The paper concludes with a discussion of the work (Section 8) and a short discussion of the limited related works that exist (Section 9).

Definitions

We make an initial definition of the notion of subjectivity as follows. We build on this definition in the context of two datasets in Sections 3 and 4.

The complexity of a word is considered subjective if the returned complexity labels for that word span a range of complexity values.

More formally, we can define a complexity annotation scheme as taking vocabulary items v i from some vocabulary V and presenting them to a discrete set of n human annotators h 1 , ..., h n , drawn from a pool H of size at least n who each return some label l drawn from a discrete ordinal integer label set L. An annotation a i can be defined as a point in the relation A = H × L and each v i receives n annotations which can be represented as a vector -→ a (with indices a 1 ...a n ). Given these conditions, we can define 2 properties, complexity and subjectivity as follows. The complexity of a vocabulary item v i is the mean of the ordinal values of the labels in the annotations:

Complexity(v i ) = N j=1 a j n (1)
Similarly, we can use these definitions to define a formal measure of subjectivity modelled on the average absolute deviation of -→ a :

Subjectivity(v i ) = n j=1 |Complexity(v i ) -a j | n (2)
We may also define thresholds for complexity T c and subjectivity T s by which we define a vocabulary item as holding the property of complex or subjective:

Complex(v i ) → Complexity(v i ) > T c
(3)

Subjective(v i ) → Subjectivity(v i ) > T s (4)
T c may be sensibly set at 0.5 in complexity research, although this could be varied depending on the requirements of an application. T s will be some function of the magnitude of L (i.e., the more categories to choose from, the wider deviation is acceptable before crossing the subjectivity threshold) and also of N (i.e., the more annotators that we have, the more potential for subjectivity). We propose the following definition for determining a subjectivity threshold as follows:

T s = α × |L| × n (5)
where α is a normalising constant set to some small value between 0 and 1. We report on empirical values of α in the next two sections.

Subjectivity in CWI2018 Annotations

The CWI2018 data covered Wiki text and Newswire data. Annotators were asked to identify any word or span that they found to be complex in a context. Each context was presented to 20 annotators, of which 10 were native speakers of English and 10 were not. This resulted in 20 binary annotations for each identified term which indicated whether an annotator considered that term complex. These binary annotations were represented by the ordinal labels 0 and 1 such that if every annotator agreed a word was complex it would have 20 positive annotations and get a score of 1. If no annotator considered a word complex it would have 20 zeroes and be given an overall score of 0. Interestingly from the point of view of subjectivity, annotator disagreement is directly modelled in the complexity labels. As in the initial example given in the introduction, if 10 annotators found a word to be complex, whereas 10 found it to be simple, the word would be given a score of 0.5, according to the formulae given in Section 2. We investigate the nature of subjectivity in the English portion of CWI2018 data through the 3 plots in Figure 1. We firstly show in Figure 1a the distribution of subjectivity values in the CWI2018 dataset. These values were calculated using the formula for subjectivity given above. It is clear that most items in the dataset fall in the lower end of the subjectivity -coming in the 0.0-0.1 bin. These represent both complex and simple words, although the majority are simple words due to the nature of the dataset. There are a number of words in the subsequent bands, with the highest bin (0.4-0.5) having just under 3000 examples. Figure 1b shows the relationship between subjectivity and complexity in the binary annotation setting of CWI2018. The bell curve that arises represents the fact that the lowest-subjective elements are those with high or low complexity (everyone agreed either way), whereas the most subjective elements are those with a mid-level complexity (half the annotators said simple, the other half said complex). Finally, Figure 1c shows the effect of varying alpha (And hence the threshold) on the proportion of words that are considered subjective. A subjectivity threshold above 0.5 (α = 0.0125) leads to no words being considered subjective. Figure 1c demonstrates that the subjectivity threshold can be empirically set to determine the words that are determined as subjective. A subjectivity threshold of 0.4 (α = 0.01) would result in 5% of instances being considered subjective, whereas a lower threshold of 0.2 (α = 0.005) would result in 15% of instances considered subjective. A few examples of words across subjectivity values are described in Table 1.

Subj

Terms 0.0 back, bomb, censorship, death, instilled 0.25 assets, cushion, launches, previously 0.5 approaching, credence, overspending, slash

Subjectivity in LCP2021 Annotations

Whereas the CWI2018 data used binary annotation (L = {0, 1}) the LCP2021 task used a 5-point Likert scale (L = {0, 1, 2, 3, 4}). This allows annotators to agree on points in the Likert scale that do not represent the poles of the scale. For example, annotators may all agree that an instance is of medium complexity with a subjectivity of 0. Equally, annotators may nearly agree, centering around a given point, but disagreeing (within varying margins) from that point. Finally, it is possible that an instance might polarise the annotator pool. For example, if an instance is ambiguous one set of annotators may interpret in one way, whereas another take another interpretation. The first interpretation might lead to annotations of simplicity, whereas the latter leads to annotations of difficulty -creating a multi-modal distribution in the returned annotations. This has some negative ramifications for the definition of complexity used in this work, as the mean implicitly assumes a normal distribution. The complexity is still reflective of a central point in the annotations, but not a maximal point in this scenario. However, for our definition of subjectivity, the case of multi-modal distributions will still lead to high subjectivity values as the multiple modes will be separated from the centralised complexity value. In any case, this may become more of an issue with continuous annotations, as opposed to a 5-point Likert scale, where the few points in the scale force annotator decisions around common poles.

To investigate the phenomenon of subjectivity in the LCP2021 data, we applied the same transform following the equations from Section 2 to the original annotations to give a complexity and subjectivity value for each instance. The number of annotations for each instance in the LCP2021 data is 10. We demonstrate the subjectivity of these annotations by creating the same figures as for the CWI2018 data, as shown in Figure 2. not capped at 1 as it is a function of the ordinal labels which range from 0-4). The distribution appears to be Gaussian, with a left skew. The range of the values can be interpreted in the context of the number of labels available. A subjectivity of 0.6 means that the annotations were within 0.6 points of a label of each other. The maximum possible subjectivity in a 5-label anno-tation setting would be where an equal number of annotators have selected polarised values. (i.e., 0 and 4). In this case, the complexity of the annotations for any N would be 2, as would the subjectivity. So a subjectivity of 1.0 in this setting is half of the theoretical maximum possible subjectivity. Almost all of the annotations fall below this mark.

To further investigate the effect that the number of annotators has on the distribution of the annotation with respect to complexity, we first recast the 5-point scale as a binary annotation. We further recast the problem as a 3 and 4 point annotation problem by relabelling in the manner described in Table 2, where the top row describes the original annotation point and the subsequent describe the transformed point. 1 Applying the transform allowed us to produce the graphs in Figures 2b-2e.

Original 0 1 2 3 4 2-label 0 0 1 1 1 3-label 0 0 1 2 2 4-label 0 1 2 2 3
Table 2: Label transforms used.

To describe the boundaries of the graphs in Figures 2b-2e, we can consider that the y-axis is determined by the complexity and the x-axis is determined by the subjecctivity. In the simplest case (Figure 2b) a parabola is formed as the subjectivity requires the complexity to be calculated, summing the number of instances once to calculate the mean and then again to calculate the subjectivity (hence x 2 ). It is logical to consider that when we have only labels 0 and 1, the subjectivity will be 0 when the complexity is 0 and the subjectivity will be 1 when the complexity is 1 (as these cases can both only arise when the vector is all zeroes, or all ones). Similarly, when the complexity is 0.5, the subjectivity is also 0.5 as this arises when the annotator pool is perfectly polarised (i.e., half have chosen zero and half have chosen one).

Let us then consider the more complex case of Figure 2c. In this graph there are 3 labels available to the annotators. We can see that the annotations fall in a space that can be described by three boundaries. The upper boundary is described as above, the case where the annotation vector contains only instances of 0 or 2 (2 being the largest possible annotation). It is the same curve as in Figure 2b, but is twice as high and twice as wide.

There are also two clear lower bounds in the graph. The first, between 0 and 1 on the x-axis is described by the curve in Figure 2b as it is the case of annotations which contain only zeroes and ones (i.e., no twos). The second, falling between one and two on the x-axis is described by a new curve, which is the same shape as the other two, but similarly described by the annotation vectors containing only 1's and 2's.

Given the description of Figures 2b and2c above, it should be clear what is happening in the more complex Figures 2d and2e. In these, the upper bound is similarly described by the polarised case between the first and last labels, whereas the lower bounds are described by the polarised cases between successive labels. This gives rise to the effect that subjectivity minima appear at each ordinal label (i.e., when all annotators selected that label) and that a single maxima appears at complexity = 0.5, when half the annotators selected the lowest possible annotation and the other half selected the highest.

Considering Figure 2e, which represents the original labels in the LCP2021 data, we see that the spread of annotations covers almost the entire possible space. We can observe that lower subjectivity occurs at the two ends of the scale (0.0 and 4.0), with similarly lower values for subjectivity appearing at 1.0 and 3.0. Interestingly, where there should be a minima at 2.0, this is missing, indicating that annotators were unlikely to agree on the 'Neutral' category in the annotation scheme. The observed maxima is around 1.75, indicating that the top portion of potential subjectivity values is missing as the maximum possible subjectivity would be 2.0. We also analysed the threshold for subjectivity prediction and report our results in Figure 2f. This follows an inverse-S curve, in line with the normal distribution of subjectivity shown in Figure 2a. Again, we are not seeking to give a specific value for the subjectivity threshold here, but rather attempting to expose the behaviour of the thresholded values. We can see that a threshold of 0.25 (α = 0.005) will result in around 95% of terms being considered subjective, whereas a threshold of 0.5 (α = 0.01) will result in around 60% of the terms being considered subjective.

CWI2018 vs. LCP2021

Using the data above we can draw several comparisons between the two prominent existing datasets for CWI/LCP annotation. First of all it is clear from Figures 1a and 2a that the underlying distribution of subjectivity in CWI2018 and LCP2021 is fundamentally different. This is due to the existence of many more agreed upon simple terms in CWI2018. By comparison, the LCP2021 data contains much more subjectivity than the CWI2018 data. Whereas the majority of instances in the latter dataset have a subjectivity close to 0, the subjectivity in the LCP data is centered around 0.4-0.6 (i.e., around half a point on the Likert scale). This is a factor of the way in which each dataset was annotated. In the CWI2018 data, annotators were presented with a context and asked to identify any complex terms. If a term was identified by at least one annotator, it was included in the dataset. This leads to the case where many terms were annotated by only a single annotator, having an annotation vector with a single 1 and the rest 0's. In our definition of complexity/subjectivity this is labelled as low-complexity, low-subjectivity. But it may be the case that the nonannotations of the term are really just the other annotators neglecting to annotate that term, rather than a confirmation of the term's simplicity. Contrastingly, the LCP2021 data presented annotators with specific terms and requested an annotation decision for every given term. This means that every annotation in the dataset is representative of a meaningful decision by the annotator. Clearly, this has led to more subjectivity in the range of annotations that are returned for LCP2021 than CWI2018.

The range of subjectivity with respect to complexity values is also larger in the LCP2021 data as a result of the labels on a 5-point Likert scale that were employed. This can be seen when comparing Figure 1b to Figure 2e. Whereas for the CWI2018 data, the subjectivity values are linked directly to the complexity, the LCP2021 data has a range of subjectivity values for each complexity value. This is because each possible complexity value could be made up of many different annotation vectors. E.g., a complexity value of 2 could be made up of 10 annotations of 2 or 5 annotation of 1 and 5 annotations of 3, as well as many other ways. Whereas the former would have a low subjectivity value, the latter would have a higher subjectivity as the annotators agreed less. The subjectivity threshold behaves in a similar way between the two datasets. Both produce an inverse Scurve in Figures 1c and2f. The α value was used to determine common thresholds and across our 2 datasets it allows for a similar threshold to be set given different values of α. Further work on datasets with different values of n and L is needed to determine the robustness of α to these values. Both curves follow a stepped curve, due to the different values that could be produced by the formula for subjectivity operating on a fixed size vector of integers. The LCP2021 data has more levels, producing a smoother curve as it has more labels in the annotation scheme -allowing for a wider range of final values. We further compared the subjectivity values for common words between the CWI2018 and LCP2021 datasets. To do this, we took the subset of instances containing tokens that occurred in both datasets (n = 26166) and calculated Pearson's correlation between the subjectivity values in both datasets. The correlation was low at 0.189, indicating that the subjectivity for specific words in the two datasets is not wellaligned. This may seem surprising, as we would expect subjective words in one dataset to also be subjective in another dataset, however given the findings presented so far on the nature of subjectivity in each dataset and the description of the differing annotation protocols employed, it is conceviable that the discrep-ancy is in fact due to the differences in the datasets' construction and that future datasets following either protocol would have higher correlation.

Factors Affecting Subjectivity

To investigate our second research question, we adopt the LCP2021 data and perform a correlation analysis with a number of features which are used elsewhere in the literature to determine the complexity of a word. The feature categories and specific features, with identifiers are listed below:

Lexical Ambiguity: Lexical Frequency:

Web1T Frequency (LF1): The frequency of the term in the Google Web1T unigram dataset [START_REF] Brants | Linguistic Data Consortium[END_REF]. Subtlex Frequency (LF2): The frequency of the term in the Subtlex dataset [START_REF] Van Heuven | Subtlex-uk: A new and improved word frequency database for british english[END_REF]. log Web1T Frequency (LF3): log(LF1) log Subtlex Frequency (LF4): log(LF2) MRC Psycholinguistic Norms: Familiarity (PN1): How likely the word is to be known.

Concreteness (PN2):

The degree to which the word represents a grounded concept. Imageability (PN3): The degree to which the referent of a term can be visualised.

We use Pearson's correlation to determine the relationship between the subjectivity values for LCP2021 and the features we have determined above. These are presented in Table 3, where we also include the correlation with complexity for reference. The correlation between the complexity and subjectivity values was 0.641. Table 3 shows that the features we tried have a weak negative correlation with subjectivity. The correlation with the lowest magnitude (LA2, WordNet Tree Depth) is -0.094 and the highest (LF4, Log SUBTLEX Frequency) is -0.412. The correlation values for subjectivity are typically in line with, although slightly lower than those for complexity, except in the case of LA2 and LF3, which both show a larger discrepancy, although the reason for this is unclear.

Predicting Subjectivity

Finally, we train several models to predict subjectivity in the LCP2021 dataset. This could enable future applications to not only determine which words are complex or simple, but also determine whether a word is likely to split the opinions of users. This may be useful for determining simplification and personalisation strategies, or for better understanding the nature of a complexity value that is returned by a system. For example, if a system returns a neutral complexity, it is helpful to know if that value is likely to be agreed upon, or if some users will find the word difficult, whereas others will find it easy (giving an average of neutral).

We first select a subjectivity threshold of 0.68, which splits the data into 50% subjective and 50% nonsubjective. Duplicate tokens in the dataset were removed, leaving 5,617 instances. We did not take contexts into account, as our features are context-free. We then created a training (70%) and testing (30%) set for our experiments. We selected a Support Vector Machine (SVM), Random Forest (RF) and AdaBoost (AB) classifier from SciKitLearn and trained each one on our dataset. We did not tune the hyperparameters. We used all features described previously in Section 6. All results are reported on a single final run on the test set. We report the Precision, Recall and F1 score for both the Subjective and Non-Subjective classes in Table 4.

Our results are intended to demonstrate that subjectivity can be predicted using the features we have identified, as well as to give some simple baseline results for performance on this task. The scores indicate a reasonable predictive power, with AdaBoost giving an F1 score of 0.713 on the subjective class and 0.645 on the non-subjective class.

Discussion

Answers to Research Hypotheses

The answers to our initial research hypotheses stated earlier are given below: RH1.1: We demonstrated that we could identify subjective and non-subjective annotations through the use of an equation for determining a subjectivity value and setting a threshold. We investigated the nature of subjectivity in the CWI2018 and LCP2021 datasets and demonstrated that both datasets contain a range of subjective and non-subjective annotations. RH1.2: We found a low correlation between the subjectivity values for common terms in the two datasets we studied. Our analysis showed that the nature of subjectivity in these datasets is different, leading to the discrepancy.

RH2.1-2.3: We demonstrated that all of our feature categories had a low, but meaningful correlation with subjectivity. The features that we selected are also correlative with complexity and, as subjectivity and complexity are correlative with each other, we were able to use these features for subjectivity too.

LA1 LA2 LA3 LF1 LF2 LF3 LF4 PN1 PN2 PN3

Threats to Validity

One deficiency in our work is that we have not taken context into account. In the LCP2021 and CWI2018 annotations words were presented in context and the labels were given for the word in context, not for the word itself. This meant that repeated instances of a word had different annotation vectors and hence complexity labels (as different word senses, etc. affected the complexity). In our work, we have selected a single instance of each token, reducing the dataset size and ignoring the context. We expect to be able to address this in future work by investigating the context sensitivity of lexical subjectivity, in relation to complexity as well as other tasks.

The definition of complexity was formalised for this paper. Whilst this is reflective of the processes undertaken in previous papers to the best of the authors knowledge and given the reporting in previous work, it is possible that some unreported factors of the process are missing from our definitions. The measure of subjectivity was also determined within the scope of this work and is not adopted widely by the community. We hope that this work will introduce the notion of subjectivity and allow researchers working on lexical complexity to consider their annotations in the context of subjectivity.

Finally, a threat to the validity is that the work is done on secondary datasets. In the scope of this work, we have no control over the quality of the annotations that have been undertaken. Each dataset is reported on extensively in its own paper which detail the quality control mechanisms used to ensure that the annotators were doing the task expected of them.

Related Work

Complex Word Identification was first proposed as an initial step in the lexical simplification pipeline (De-vlin, 1998). Efforts to automatically predict complex words [START_REF] Shardlow | A comparison of techniques to automatically identify complex words[END_REF] using machine learning techniques showed this to be possible. The task was popularised by shared tasks (Paetzold and Specia, 2016;Yimam et al., 2018), where winning systems typically used feature based approaches (Gooding and Kochmar, 2018). Recently, the LCP2021 shared task (Shardlow et al., 2021) introduced continuous complexity prediction, as opposed to the binary or probabilistic prediction seen prior. High-ranking systems used either transformer based models [START_REF] Yaseen | Just-blue at semeval-2021 task 1: Predicting lexical complexity using bert and roberta pre-trained language models[END_REF] or feature engineering approaches [START_REF] Mosquera | Alejandro mosquera at semeval-2021 task 1: Exploring sentence and word features for lexical complexity prediction[END_REF]. Further work in CWI/LCP has sought to adapt the problem to a personalising task [START_REF] Lee | Personalizing lexical simplification[END_REF] in which the specific needs of a user are modelled and reflected in individualised complexity predictions. Recent work demonstrated that lexical complexity differs due to annotator background, such as native speakers vs. non-native speakers (Gooding et al., 2021). The distribution of lexical complexity shown in this work is backed up by previous works from the literature analysing the CWI-2018 dataset [START_REF] Quijada | HMC at SemEval-2016 task 11: Identifying complex words using depth-limited decision trees[END_REF]. This data, and by association the concept of lexical complexity, has been considered subjective previously by other authors [START_REF] Finnimore | Strong baselines for complex word identification across multiple languages[END_REF].

In the field of sentiment analysis the term subjectivity is used to refer to the degree to which a user is drawing on their own personal opinion vs. stating objective fact [START_REF] Maks | A lexicon model for deep sentiment analysis and opinion mining applications[END_REF][START_REF] Hill | Concreteness and subjectivity as dimensions of lexical meaning[END_REF]. That is a subtly different notion of subjectivity to the one used here. In the context of Lexical Complexity Prediction, we are assuming that a user's annotations are inherently drawn from personal experience, and instead our measure is whether those personal experiences converge or diverge.

Conclusion

We have investigated the nature of Lexical subjectivity within the scope of lexical complexity. We show that this exists across two prominent datasets and outline how it differs between them. We have also shown that subjectivity is not a stochastic phenomenon, but is correlated to several well-known features for lexical complexity and that we are able to predict subjectivity with simple machine learning classifiers in an unseen setting. We expect that transformer based methodologies will also provide strong scores on this task, and leave these experiments to future work. We release the datasets with subjectivity values, and the code used to create them via GitHub2 .

Introduction

Dictionaries are used by native speakers of a language and by language learners when they want to learn or check the meaning of a word. Even though there are dictionaries that are specifically targeted at learners, other widely known online dictionaries such as Wiktionary 1 may be one of the first resources that comes to mind. Along with definitions, those resources provide the users with examples of use for words.

In this article, we want to check how useful these examples are in helping users with understanding words. The hypothesis for this research question is that nonnative speakers can assess the difficulty of the meaning of a word when they see it used in a sentence, in the same way readers can infer the meaning of words based on the context [START_REF] Miller | Contextuality. Mental models in cognitive science: Essays in honour of Phil Johnson-Laird[END_REF][START_REF] Miller | On knowing a word[END_REF]. We observe how word difficulty is rated both by native and non-native speakers. We also assess whether a single word sense is rated differently based on the example of usage.

We also want to check whether dictionary definitions are more difficult when the word meanings themselves are more difficult. On the terminological side, it should be noted that complexity can be seen as an inherent property of a word or text, invariant and independent of the context [START_REF] Pallotti | A simple view of linguistic complexity[END_REF], whereas difficulty can be seen as a construct that arises when a given reader interacts with a given word or text; difficulty varies from reader to reader. However, the distinction between these two categories is not always very clear, even in the literature on the topic.

In an effort to clarify the terminological question, we detail the operationalizations of the concepts used in this paper: (1) in the data annotation, we look at lexical difficulty to choose words, since our selection is based on proficiency levels from a learner-targeted vocabu-1 en.wiktionary.org/ lary resource;

(2) we also assess words in terms of lexical difficulty, since the ratings we collect are based on the annotators' intuition;

(3) in the second experiment, the complexity of definitions is approximated through the lens of readability research, i.e. by characterizing the definitions using a number of linguistic variables.

The hypothesis here is that it is more difficult to explain difficult words, thus leading to a potentially more verbose explanation (that might nonetheless still be easy to understand); this is parallel to the idea that "languages encode conceptually more complex meanings with longer linguistic forms" [START_REF] Lewis | The length of words reflects their conceptual complexity[END_REF].

In order to assess these hypotheses, we performed various experiments with resources in French.

For the first hypothesis, we asked native speakers of French and non-native speakers of French to rate word difficulty based on their use in dictionary examples. To do so, seven annotators rated word difficulty by only having access to dictionary examples. The annotated data will be made available for research purposes. We surmise that the data might be useful for the evaluation of complex word identification systems. One contribution of this part of the work is that the resource is annotated at the sense level, and not at the word level, which -to the best of our knowledge -is something that has not been done before for French.

For the second hypothesis, we explore the correlation between the ranking of the words difficulty produced by the annotators and the readability of the definitions.

In section 3, we describe the data that we used as a source for our linguistic material. Section 4 describes the experimental protocol that we put in place. Results are presented in section 5 and discussed in section 6.

Related Work

Word lists are often used in second language learning scenarios (e.g. [START_REF] Laufer | A vocabulary-size test of controlled productive ability[END_REF][START_REF] O'dell | Assessing vocabulary[END_REF][START_REF] Meara | The rediscovery of vocabulary[END_REF][START_REF] Gu | Vocabulary learning in a second language: Person, task, context and strategies[END_REF][START_REF] Nation | Learning Vocabulary in Another Language[END_REF]). However, word lists compiled from L1 material are rarely suitable for L2 purposes (Richards (1974, p.72);Franc ¸ois et al. (2014, p.3767)). There are some resources such as the CEFRLex family2 that are based on L2 textbooks, thus directly targeting second language learners. However, even such resources generally use lemmas as primary entries, conflating different word senses. Especially from a language learning perspective, it is to be argued that not all word senses are learned at once, and thus basing vocabulary knowledge on lists where word senses are conflated is potentially misleading. Further, more frequent words (that are generally taught early, since frequency is often taken as a proxy for complexity, e.g. [START_REF] Rayner | Lexical complexity and fixation times in reading: Effects of word frequency, verb complexity, and lexical ambiguity[END_REF]) also tend to have more senses than less frequently used words [START_REF] Crossley | The development of polysemy and frequency use in English second language speakers[END_REF].

For English, there exists a dataset that is annotated both for lexical complexity and word senses, SeCoDa [START_REF] Strohmaier | SeCoDa: Sense complexity dataset[END_REF], leveraging the Cambridge Advanced Learner's Dictionary3 . However, even the shared tasks organized on the topic of Complex Word Identification (CWI; cf. (Paetzold and Specia, 2016;Yimam et al., 2018)) and Lexical Complexity Prediction (LCP; cf. (Shardlow et al., 2021)) do not explicitly distinguish between the complexity of different word senses; while some data sets do indeed present words in context, thus disambiguation of the words. That said, this information is not directly operationalized.

To the best of our knowledge, there is no work on using dictionaries to disambiguate vocabulary lists for French. Regarding the investigation of the complexity of definitions with regards to the complexity of their head words, we did not find any systematic study. We only found one article working on the complexity of dictionary definitions [START_REF] Gross | Complexité lexicale: le substantif débat (s)[END_REF]. However, the author states that a true semantic definition of a word (especially nouns) should not be conceptual but contain the whole set of appropriate predicates for this noun. This is not directly in line with our approach, as we work with conceptual definitions.

Data

Source

The data for our experiment comes from two different resources.

As we want to relate the outcome of the experiment to second language learning, we base ourselves on the French textbook-derived vocabulary list FLELex (Franc ¸ois et al., 2014). FLELex lists words as well as their frequencies observed across different proficiency levels. In order to divide FLELex into six discrete levels, we use the machine learning based level assignment proposed by Pintard and Franc ¸ois (2020) which is freely available through the CEFRLex webpage4 . It contains 14,236 rated words.

As we work at the word sense level, we rely on a dictionary resource. The resource we use is GLAWI [START_REF] Sajous | GLAWI, a free XML-encoded Machine-Readable Dictionary built from the French Wiktionary[END_REF]. GLAWI is an XML version of the French Wiktionary5 . GLAWI's senses are strongly fine-grained : in the list we extracted, the average number of definitions per lemma is 13. We also calculated the median value, which is 2. We filter GLAWI to extract the words that are found in FLELex. Every lemma, sense, definition or example that we mention throughout the article is extracted from the resulting subset.

Anchor Words

In order to "anchor" the relative difficulty rankings obtained with the methodology (cf. Section 4), we included "anchors", i.e. words that have a reliable fixed difficulty level. Anchor words were chosen among monosemous words that show a strong centrality for their respective level, i.e. words that are likely to be representative of a given level, based on the continuous numerical score N c introduced in Gala et al. ( 2013):

N c = N i + e -r , r = Σ i k=1 U k Σ N k=i+1 U k (1)
N c is calculated for a given level N i which corresponds to the level of first occurrence, i.e. the first level at which a word is observed with a frequency greater than 0, and modifies the level score by a score e -r ∈ [0, 1[. r is calculated as the ratio of frequencies U k , with U k the frequency at level k ∈ [1, N ]. In other words, r indicates the cumulative frequency up to level i divided by the remaining cumulative frequencies after level i.

High values of e -r indicate that there exists a nonnegligible frequency mass outside of level N i , while low values of e -r indicate that the main frequency mass is located at level N i .

For the selection of anchor words, we calculated N c for all words in FLELex, excluded all words that did not fullfil the criterion e -r < 0.1, and manually selected 5 words per level for a total of 30 anchor words.6

4. Experiments

Data Combination

For this experiment, we use a similar setup as in [START_REF] Alfter | Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts[END_REF], i.e. we arrange the example sentences into sets of four and ask annotators to select the easiest and the hardest of the words, a technique called bestword scaling [START_REF] Louviere | Best-worst scaling: Theory, methods and applications[END_REF]. A set of four items constitutes one task (see figure 1). We use the combinatorial redundancy reducing algorithm [START_REF] Alfter | Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts[END_REF] for calculating the optimal number of tasks with minimal redundancy. This number, for 120 examples, comes to 1300

Data Selection

For the experiment, we work at the definition (i.e. sense) level of words. We use example sentences from definitions as illustrations of a certain word sense.

In order to explore the different hypotheses, data was automatically selected according to the following criteria:

• 5 anchor words per level (5 * 6 = 30)

• 4 examples (= 1 task) per level that illustrate the same word but different senses (4 * 6 = 24)

• 4 examples (= 1 task) per level that illustrate the same word and same sense but with different examples (4 * 6 = 24)

• 3 paired examples per level, i.e. two examples of the same word but with different senses, chosen among words with at least two senses (3 * 2 * 6 = 36)

• 6 randomly chosen examples that have at least two senses Thus, the total number of examples in the experiment is 30 + 24 + 24 + 36 + 6 = 120. While this may seem like a relatively small number of items, we surmise that it is a sufficient amount for an exploratory work with an acceptable trade-off between quantity of items and annotation time.

Annotation

For the experimental design, we use a custom graphical user interface shown in Figure 1. 7 The user interface presents four sentences with one or more word(s) marked in bold and in purple, which is the word to be judged. After each sentence, we also display the lemma of the word. On the left side and right side of the examples are buttons to choose the easiest and hardest expressions. After selecting a word as being the easiest or hardest, the color of the lemma respectively changes to green and red in order to also reflect the choice visually.

Figure 1: Graphical user interface with 'dog' being selected as easiest sense.

The user interface is designed to be simple and intuitive to use. It is possible to stop annotation at any time and resume later at the point where one left off. Further, the interface can be accessed from different devices such as laptops, computers, or smartphones, and one can freely switch between devices. The interface automatically registers the time elapsed between the completion of two tasks.

The user interface also attempts to enforce valid answers by disallowing clicking next if no choice has been made or if only one side contains a choice.

Internally, the example chosen as easiest is assigned a score of 1, the example chosen as hardest a score of 3 and the two examples not selected a score of 2. In the end, all votes v ij for item i are aggregated into a single score s i , with i ∈ [1, 120] and j ∈ [1, n], n being the number of votes for item i:

s i = Σ n j=1 v ij n (2)
In order to see whether annotators are consistent in their annotation, we duplicated one task as control task.

After shuffling of the data, the control task was inserted at positions 4 and 1299 (of 1300).

For this experiment, we recruited two student helpers who were paid 12C per hour as well as three colleagues. In total, including two of the authors, seven people contributed to the experiment. Each time they access the interface (unless requested otherwise, using an opt-out option in the form of a checkbox), users see a page displaying the instructions.

Here is a translation (from French) of the detailed instructions displayed for the task (we leave out the instructions related to the interface):

You will see sets with 4 sentences followed by the lemma of the word in bold. We ask you, for each set, to indicate which of the emphasized word's meanings seems to be the most difficult to understand for you, and which one seems to be the easiest. Don't think for too long, use your intuition.

Judge only the item in bold, with reference to its dictionary form (e.g., verbs in the subjunctive mood should be judged as equal to verbs in the indicative mood).

Context is given to indicate the sense of the word and should not have an influence on the judgment.

The annotators were orally asked to avoid discussing the tasks between themselves before completion. Table 1 gives an overview over the demographic information of the participants.

Gender Male 4 Female 3 Mother tongue French 4 Japanese 1 Spanish 1 Luxembourgish 1
Table 1: Demographic information of participants Each annotator was asked to complete all 1300 tasks.

Definition Complexity Evaluation

To assess definition complexity, we randomly selected 30,000 definitions from words that appear in FLELex. We processed them through FABRA, the French Aggregator-Based Readability Assessment Toolkit (Wilkens et al., 2022) and performed clustering on the data. FABRA calculates in excess of 4,000 features for each sentence. We restricted FABRA features to surface (e.g. word length) and lexical (e.g. frequency in different word lists) features, as the other classes of features (syntactic and discourse features) are more suitable to full text, and definition texts are not necessarily complete sentences. Before clustering, we perform dimensionality reduction (to 100 dimension) using Principal Component Analysis (PCA) [START_REF] Pearson | On lines and planes of closest fit to systems of points in space[END_REF] as implemented in scikit-learn [START_REF] Pedregosa | Scikitlearn: Machine learning in Python[END_REF]. For clustering, we use the KMeans [START_REF] Lloyd | Least square quantization in pcm. bell telephone laboratories paper[END_REF][START_REF] Macqueen | Some methods for classification and analysis of multivariate observations[END_REF] implementation also available in scikit-learn, with the number of clusters set to 6 in order to match the number of CEFR levels, which are used by the FLELEX resource.

We then look at the cluster to which every definition linked to word senses that were annotated correspond, to check whether we can observe a correlation between the clusters and the difficulty level.

Results

Time per Annotator

Using a preliminary (randomly chosen) set of items, the authors tested the platform in order to estimate the time investment necessary. We found that it took about 12 seconds to complete one task. Based on this estimation and adding some margin (20 seconds per task), we estimated the total time needed to complete the experiment at around 7 hours.

Rankings

In order to compute a ranking, we calculate the score of each item according to equation 2 in three different ways: taking into account all annotators, only native speakers, and only non-native speakers. This gives us three rankings: the global ranking, the native ranking, and the non-native ranking. Due to space limitations, the full results are not included here but can be retrieved at https://github.com/daalft/ dicomplex.

Overall, the three rankings are very similar, the most dissimilar being the ranking between native and nonnative speakers. However, even the most dissimilar rankings are highly correlated (Pearson's rank correlation coefficient of 0.90) as detailed in Section 5.3. A qualitative analysis reveals that there are mainly differences in rank for the words haltérophilie 'weight lifting', on rank 73 in the native speaker ranking and rank 115 in the non-native speaker ranking (a difference of 42 ranks), and chèvrefeuille 'honeysuckle', rank 78 vs 101 (difference of 23 ranks). These words seem to be relatively well known by native speakers, but introduced very late for non-native speakers. The biggest rank span is found for joindre 'to join'.

Upon closer inspection, we can see that the example sentence at rank 67 is Ces planches, cette porte, ces fenêtres ne joignent pas bien. 'These planks, this door, these windows do not join well', which is indeed a rather rare use of joindre.

We can see that the different senses of a word end up at quite different ranks, as illustrated in Table 4. The maximum observed span is 66 for 'point', with an average span of about 34, a significantly higher span than for examples of the same sense. An exception to the wider spread is the word 'old'. Upon closer inspection, we can see that the example sentences that were automatically selected were very short and thus did not convey the fine-grained meaning distinctions ('old' as pertaining to a certain age of a person, ancient, a derogatory term, a term of veneration). On the other hand, point 'point/dot/stitch pattern' ranged from point 'dot' (e.g., a dot ends a sentence) to point 'stitch pattern', and the meaning of stitch pattern was ranked as hardest of the senses by a large margin (rank 74, the closest rank of other meanings being rank 28). Again, one can also see that the ranking order follows the CEFR levels from FLELex in broad terms. non-native speaker ranking. As can be observed, there is a clear progression from A1 to C2, with expected overlaps between adjacent levels. Further, the rankings are quite similar, although the non-native ranking seems to follow FLELex levels a bit more closely, which is to be expected, since FLELex is a second language learner oriented resource.

Intra-and Inter-Annotator Agreement

Based on the control task that was annotated twice by each annotator, we can see that five out of seven annotators were completely consistent in their annotation.

For the remaining two annotators, one person chose a different "easiest" word while the other person chose a different "most difficult" word. As the control task was randomly chosen, there was no expectation regarding which word should be considered the easiest or the most difficult. Furthermore, the two annotators in question still remained consistent in their other choice, hence we neither discard the these annotators nor proceed in any kind of remediation.

Inter-annotator agreement (Pearson's rank correlation coefficient) shows a high agreement of 0.90 between the ranking of native speakers and the ranking of nonnative speakers. This seems to confirm that non-native speakers can produce native-like rankings. This is consistent with what has been found in a similar study (cf. [START_REF] Alfter | Crowdsourcing Relative Rankings of Multi-Word Expressions: Experts versus Non-Experts[END_REF]).

Clustering

Figure 2a shows a visualization of the clustering using The 102 definitions corresponding to the 120 annotated examples are only found in three clusters out of six. Using the cluster numbers from the t-sne visualization, 48 were found in cluster 0 (red), 16 in cluster 2 (green) and 38 in cluster 4 (navy blue). The average scores by cluster, based on the global annotation ranking, are respectively 1.97 (that would be rank 59/120), 2.05 (that would be rank 65/120), and 2.01 (that would be rank 60/120). What we can draw from this observation is that no correlation between the difficulty of a word and the readability of its definition is found in the data we used for our experiments.

B1 festif A2 clignotant B1 festif A2 clignotant B1 festif A2 bricoleur B1 corridor B1 corridor A2 clignotant B1 vouvoyer B2 enquêter B2 enquêter B2 enquêter B1 vouvoyer B2 revolver B1 vacarme C2 haltérophilie C1 arrachage C1 arrachage C2 chèvrefeuille C1 discriminatoire C2 chèvrefeuille B1 vacarme B1 vacarme C1 discriminatoire C1 surcoût C2 chèvrefeuille C1 surcoût C1 discriminatoire C1 surcoût C2 haltérophilie C1 arrachage B2 lugubre B2 perspicacité B2 perspicacité B2 perspicacité B2 lugubre B2 lugubre C1 affligeant C1 affligeant C1 affligeant C2 haltérophilie C2 inexorable C2 inexorable C1 achoppement C2 enhardir C2 protéiforme C2 inexorable C1 achoppement C2 enhardir C2 enhardir C2 protéiforme C1 achoppement C2 protéiforme

Discussion and Future Work

While we found the clustering not to correlate with the ranking, we still see a clear delineation of clusters in both visualizations. Further studies should investigate these clusters in more detail to find out what they represent and whether this information might be useful in future studies. From our observations we can get insights about the writing process of a dictionary. For example, the definition of the easy word pêche with the meaning of 'peach' is Fruit du pêcher, parfumé et d'un goût savoureux, dont le très dur noyau est enrobé par une chair jaune ou blanche et une fine peau veloutée de teinte jaune et rouge-orange. ("Fruit of the peach tree, fragrant and tasty, whose very hard pit is coated by a yellow or white flesh and a thin skin mixed with yellow and red-orange"), while the difficult word perspicacité (perceptiveness) is defined with Pénétration d'esprit ("Spirit penetration"). Peach is defined in a quite detailed way, while the definition of perceptiveness is abstract and vague. Those are two extreme examples, but it contributes to illustrate why we did not find correlations between the readability of definitions and the difficulty of word senses. We believe it would be beneficial to perform more studies on this very aspect, namely with comparing learners' dictionaries and more traditional dictionaries, so as to identify gaps between the phrasing of the definitions and the need of the targeted audience and systematically prevent them.

It would be beneficial to fine-tune the methodology of annotation; the current methodology covers all relations between examples. However, it is possible to drastically reduce the number of comparisons needed by inferring relations based on annotated relations. Thus, for example, -and for simplicity's sake with a simple comparison between two items -if one finds that A is easier than B and that B is easier than C, one could infer that A is easier than C. Thus, one would not need to annotate the relation between A and C. Preliminary experiments have shown that for a binary classification, i.e. choosing the "easiest" of two items, with 100 items, this would require about 700 comparisons between two items. In contrast, using the current methodology and adapting it to the case of binary classification, one would need 4950 comparisons. It would also be interesting to further explore the differences between annotators, since the question of rater subjectivity has recently become a topic of interest in research on lexical complexity (Gooding et al., 2021;[START_REF] Shardlow | Agree to Disagree: Exploring Subjectivity in Lexical Complexity[END_REF].

Conclusion

In this article, we have presented a study on word sense complexity using a graded lexicon with CEFR levels for French and linguistic material (definitions and examples) extracted from the French Wiktionary. We asked seven annotators to rate the complexity of word senses based on their usage in a sentence. The resulting dataset will be made available upon publication. It consists 1,300 sets of four dictionary examples, along with the annotation of which one is the most difficult and which one is the easiest. Those 1,300 sets are found seven times, produced by four French native speakers and three non-native speakers. We have found that native speakers and non-native speakers agree to a quite large extent. However, the clustering was found not to correlate with rankings.

We compared the word senses ranking information to the corresponding definitions in order look for a correlation between a definition's readability and the difficulty of a word. We found no such correlation. Though, by examining closely the data we may argue that assessing the readability of definitions when writing a dictionary could improve its effectiveness.

Introduction

Simplified languages 1 are language varieties that have the purpose of enabling inclusion and social participation of people with low reading competence by making information easier to read and comprehend (Bredel and [START_REF] Bredel | Leichte Sprache. Theoretische Grundlagen. Orientierung für die Praxis[END_REF]. A typical application domain of simplified language is news articles (Saggion, 2017).

In recent years, web portals for simplified news have been created in several languages, and computational approaches to simplified language such as automatic readability assessment or text simplification are gaining interest. We present SNIML, a corpus of simple news in many languages. The corpus contains simplified news articles in Finnish, French, Italian, Swedish, English, and German. It comprises a total of 13,400 articles published between 2003 and 2022. All texts in SNIML are shared under an open license that allows for academic research use. We plan that future news articles are automatically collected and added to the corpus in a temporally stratified way. The corpus and various subcorpora are available for download. 2 While some resources for simplified language align simplified versions of texts to standard-language versions, our corpus currently does not contain any news in standard language. Furthermore, the sub-corpora involve texts created according to different simplification guidelines and for different target audiences (Section 3.3). As such, SNIML is particularly useful for automatic readability assessment and for unsupervised, 1 The term "simplified language" is used to denote the sum of all "comprehensibility-enhanced varieties of natural languages" (Maaß, 2020, p. 52), i.e., what is commonly termed "Easy Language" (German leichte Sprache) and "Plain Language" (German einfache Sprache). Maaß (2020, p. 52) mentions "easy-to-understand language" as an umbrella term subsuming these varieties. However, in this contribution, we prefer the term "simplified language" to emphasize the notion of the result of a simplification process.

2 https://pub.cl.uzh.ch/projects/sniml/

Standard language

One possible difference is that Omicron may be less likely than earlier variants to cause a loss of taste and smell.

Research suggests that 48 percent of patients with the original SARS-CoV-2 strain reported loss of smell and 41 percent reported loss of taste, but an analysis of a small Omicron outbreak among vaccinated people in Norway found that only 23 percent of patients reported loss of taste, and only 12 percent reported loss of smell.

Simplified language

Omicron may be less likely to cause a loss of taste and smell. The original virus caused such losses in almost half the sick people, and a study showed less than half that number losing taste and smell with Omicron. self-supervised or cross-lingual learning. In addition, the simple news articles could be combined with related news articles in standard language, yielding a large-scale multilingual comparable corpus of simple news.

Related Work

Many previous resources for the computational processing of simplified language are parallel, combining a simplified version of a text with its original version.

Parallel corpora have been used to train sequence-tosequence systems for text simplification (Wubben et [START_REF] Bott | Text simplification resources for Spanish[END_REF], the Newsela corpus of English and Spanish news [START_REF] Xu | Problems in Current Text Simplification Research: New Data Can Help[END_REF], the Alector corpus of French educational texts (Gala et al., 2020), as well as German parallel corpora compiled from various web sources [START_REF] Klaper | Building a German/simple German parallel corpus for automatic text simplification[END_REF][START_REF] Battisti | A corpus for automatic readability assessment and text simplification of German[END_REF]. Other parallel news datasets that have been used for automatic simplification in German are the APA dataset [START_REF] Säuberli | Benchmarking data-driven automatic text simplification for German[END_REF] and the 20m dataset [START_REF] Rios | A new dataset and efficient baselines for document-level text simplification in German[END_REF]. Some other previous resources for simplified language are comparable corpora, i.e, collections of simplified documents and standard-language documents that share the same topic but are not guaranteed to correlate on the sentence level. For example, [START_REF] Barzilay | Sentence alignment for monolingual comparable corpora[END_REF] combined Encyclopedia Britannica and Britannica Elementary to form a comparable corpus, and they proposed to use alignment techniques to extract parallel sentence pairs. Similarly, the combination of English Wikipedia and Simple English Wikipedia [START_REF] Zhu | A monolingual tree-based translation model for sentence simplification[END_REF], though often treated as a parallel corpus, can be characterized as a comparable corpus, since Simple English entries are not necessarily simplified versions of the standard-language entries. [START_REF] Hwang | Aligning sentences from standard Wikipedia to Simple Wikipedia[END_REF] used parallel corpus mining methods to create a more parallel version of this dataset. An alternative approach is to extract parallel sentences manually from comparable corpora [START_REF] Grabar | CLEAR -simple corpus for medical French[END_REF].

The SNIML corpus differs from previous resources with regard to its scale and its rich multilinguality. However, it is neither a parallel nor a comparable corpus, since it currently does not contain standard-language news articles. Thus, the structure of the corpus is best compared to datasets of raw text, such as the CC-News crawl (Common Crawl, 2016) or the Oscar corpus (Centre Inria de Paris, Équipe ALMAnaCH, 2019), even though SNIML is much smaller in size.

Corpus

Data

Language Variety The corpus is a collection of news articles that are written in simplified language. The articles originate from news providers in the USA, Finland (Finnish and Swedish), Belgium (French), Italy, and Switzerland (German). The level of simplification varies between the providers. Also, diverse target groups are addressed by the articles, including people with intellectual disabilities, people with low education, immigrants, emigrants, language learners in general, older adults, and children. The news providers are described in more detail in Section 3.2. Section 3.3 discusses the simplification guidelines involved.

Machine Translations

We have created English and German machine translations of most articles. These are mainly intended to be an aid for the users of the web reader interface (Section 4). In addition, the translations could be useful for data augmentation. We provide translations only in cases where the license permits derivative work, namely for articles provided by Informazione Facile, Journal Essentiel and The Times in Plain English.3

News Providers

We collected the articles of six news providers: Selkosanomat, Lätta Bladet, Journal Essentiel, Informazione Facile, The Times in Plain English and Infoeasy. Table 2 gives an overview of the providers and the licenses these providers apply to the text content. Text samples for each provider are listed in the Appendix (Table 3). In what follows, each provider is described in more detail.

Selkosanomat

Selkosanomat is a news platform in Finland that is published by the association Selkokeskus which is part of Kehitysvammaliitto (Developmental Disability Association). It offers a printed magazine as well as a free online newspaper. News on the topics Finland, world, sports, culture, and everyday life are published. The articles are written in Finnish.

Journal Essentiel

Journal Essentiel is an online journal in Belgium that is published by the non-profit association FUNOC (Formation pour l'Unversité ouverte de Charleroi) and primarily aims to be an educational information tool.

The Times in Plain English

The Times in Plain English is located in the USA and is published by the News in Plain English Inc. A wide range of topics are covered, including international news, news about New York, politics, health and education, law, and economy. To test readability, the publishers use Flesch-Kincaid Grade level (Kincaid et al., 1975).

Infoeasy

Infoeasy is a private initiative in Switzerland that provides an online magazine in easy language. The articles are mainly written in German but an increasing number of articles are translated into French. However, we only used the German articles for this corpus.

Infoeasy addresses a variety of topics including international news, news about Switzerland, current topics, society, culture, health, sports, economy, and science.

Simplification Guidelines

In this work, the focus was on creating a corpus of simplified news that is as diverse and comprehensive as possible. The aim was to include texts from sources in several languages and on a broad variety of topics. As a result, also the level of simplification and the target group vary among the different news providers.

Informazione Facile and The Times in Plain English assess the complexity of their texts with standardized, length-based readability indices. For assessment, In-formazione Facile uses the service of corrige.it that analyzes texts according to the GULPEASE Index [START_REF] Lucisano | Gulpease: una formula per la predizione della difficoltà dei testi in lingua italiana[END_REF]. The GULPEASE Index is tailored to the Italian language and includes a scale that associates the index with a level of education. The platform states that their texts are suitable for people with reading skills at the level of basic school education. Additionally, Informazione Facile uses a basic vocabulary reference [START_REF] Chiari | The new basic vocabulary of Italian as a linguistic resource[END_REF] to decide which words need further explanation. The Times in Plain English, on the other hand, uses the Flesch-Kincaid Grade Level, which assigns a school grade of the U.S. education system that is needed in order to be able to read the text at hand (Kincaid et al., 1975). However, the platform does not state the specific grade level used.

The magazines Selkosanomat and Lätta Bladet follow the guidelines for plain language that are listed on the website of Selkokeskus.4 . They include instructions for the vocabulary, such as preferring well-known vocabulary and explaining difficult words, and for the language structure, such as writing short sentences and using active voice. Additionally, specific guidelines for different text types, including media texts, exist. These guidelines are developed and maintained by Selkokeskus.

No information is given by Infoeasy and Journal Essentiel as to which guidelines they use. Future work could empirically analyze the subcorpora and compare their usefulness for different target audiences.

Data Collection

To obtain all published articles, we developed web scrapers to scrape the archive pages of the providers.

For this, all URLs of the articles were collected. By performing requests to the collected URLs, we obtained the HTML files of the article pages, which we then parsed. In the case of Informazione Facile, we received a database export in RSS format of the editors, therefore, no web scraping was needed.

To collect the newly published data, we use an RSSbased web scraping approach. Requests to the RSS feeds of the providers are made daily. The RSS files are parsed to extract the textual data and the metadata.

In cases where not all information is contained in the RSS file, the web pages of the articles are scraped to obtain the missing data. For each provider, we developed a specialized parser to parse the RSS and HTML files.

Dataset Format

The corpus is structured in XML. Each article is represented in an article element and can be identified with a unique ID. The textual data consist of the title, a short description and the complete text body of the news article. Additionally, metadata about the article are provided: the category or categories that the article was published in, keywords, the language, the URL, the publication date, the author, and the provider. Providers are further characterized with their name, a link leading to the website, the license, and a link to the license if one exists. As for now, the XML structure does not conform to the TEI format. However, we consider changing the format in a future version of the corpus.

Besides the complete corpus, we provide several subcorpora. To enable work on only one of the languages, a separate XML file is available for each provider. Also, files containing all articles published in a specific month are provided. For each month, additionally, a sub-corpus for each provider is compiled.

Web Reader Interface

In order to make the collected news articles not only available to researchers but also to the target groups of simplified language, we created a web-based reader interface in the style of a news aggregator. The user interface is available in German and English, while the news articles are provided both in their original language and (as machine translations) in the language of the user interface.

The articles are listed with their title, the original news provider, the publication date, and a short description of the content. They are ordered by publication date and are summarized under the week they were published in. The detail view of an article shows its complete text content.

Conclusion

The SNIML corpus compiles more than 13k simplified news articles in six languages. The articles are shared under an open license that permits academic use, and are planned to be continually updated.

The SNIML corpus is capable of serving as a useful complement to other resources for simplified language.

For example, Simple English Wikipedia has grown very large but is restricted to a single language, text style, and simplification level. Complementing Simple English Wikipedia, the SNIML corpus could thus improve the diversity and multilinguality of language models for simplified language, as well as identification systems for simplified language. Furthermore, the multilingual composition of SNIML opens up possibilities for the evaluation of crosslingual transfer. For example, a system for the identification of simplified language could be trained on the English portion and evaluated on the five other languages in the corpus. Moreover, the temporal stratification of the corpus makes it possible to evaluate a model for simplified language on concepts and topics unseen during training.

It has been shown on the example of standard English that temporal generalization is a challenge for language models [START_REF] Lazaridou | Mind the gap: Assessing temporal generalization in neural language models[END_REF], and we believe that it could even more so be a challenge for simplified language.

Future work may consist of aligning the articles to related articles in standard language. Such an extended version of SNIML could offer new opportunities for parallel corpus mining, or even for experiments towards an unsupervised text simplification system.

Acknowledgements

We thank the providers of the news articles in the corpus, namely The Times in Plain English, Selkosanomat and Lätta Bladet, Journal Essentiel, Informazione Facile, and Infoeasy, for agreeing to make their work available under an open license. Recovery is another word for: healthy again.

Bibliographical References

That's why we now need a Covid certificate in many places.

We are only allowed in with the certificate. And this certificate is only given to people • those who are vaccinated.

• that have recovered.

• who have taken a corona test. And the test must be negative.

Introduction

Poor readers come in many forms and include those affected by cognitive disabilities (e.g. individuals with dyslexia), but also those who have not yet developed the skills to master the language (e.g. children and second language learners). Poor readers from these different target groups have more or less widespread cognitive and language difficulties which selectively impair different aspects of reading comprehension. To meet the demands of accessible text of poor readers, a number of initiatives have attempted to adapt texts and make them more comprehensible. Examples of such initiatives in Sweden are the recommendations issued by The Swedish Agency for Accessible Media (Swedish: Myndigheten för tillgängliga medier, (MTM, 2021) MTM and the initiative "Comprehensible text" (Swedish: Begriplig text) (Begriplig Text, 2019). Internationally, the most influential set of guidelines on Easy Language is Plain text (PLAIN, 2011). Such recommendations and guidelines commonly have a one-size-fits-all approach, which means that they have a generalist approach and sometimes overlook the factors underlying different types of reading difficulties. It has been already been pointed out that recommendations and guidelines are often based on common sense assumptions rather than empirical testing on groups of poor readers [START_REF] Wengelin | Mot en evidensbaserad språkvård? en kritisk granskning av några svenska klarspråksråd i ljuset av forskning om läsbarhet och språkbearbetning[END_REF].

Although the one-size-fits-all approach is an important first step, individuals who struggle with reading have deficits in cognitive and language skills which make their reading process qualitatively different. Therefore, it is essential to consider the different target audiences when developing recommendations and guidelines, but also when implementing Automatic Text Simplification (ATS) systems. The readers' needs cannot, and should not, be taken out of the equation, rather their needs should be the cornerstone of ATS.

The rationale of the Swedish Simplification Toolkit is then to start addressing the one-fits-all bias that still exists in Automatic Text Simplification by putting the linguistic needs of the target audience in the limelight.

The set of computational methods underpinning the toolkit address some of these needs and at the same time reflects the current state-of-the-art in ATS for the Swedish language.

The toolkit is the concrete answer to the two research questions that drive our work, namely:

1. What types of linguistic simplification are needed, and which ones are implementable for the Swedish language?

2. Can ATS be conceived, designed and implemented to meet the needs of different target audiences?

In the next sections, we provide the background and illustrate our approach that we have implemented for the Swedish language.

Profiling Selected Target Audiences

In the short description of the target audiences provided below, we single out different linguistic phenomena that can be used to characterise audience-specific simplification needs. Simplification may be needed at lexical level (e.g. for individuals with dyslexia), at syntactic levels (e.g. for the individuals affected by aphasia) or at discourse levels (e.g. for people with Autism Spectrum Disorder). -11), developmental dyslexia is categorised under F81.0 Specific reading disorder. An article discussing the major findings of the research on dyslexia during the last decades [START_REF] Vellutino | Specific reading disability (dyslexia): What have we learned in the past four decades?[END_REF] showed that word identification inadequacies were the most basic cause of reading difficulties. Individuals with dyslexia experience a wide range of difficulties, such as problems with long words and less frequent words [START_REF] Hyönä | Eye fixation patterns among dyslexic and normal readers: Effects of word length and word frequency[END_REF][START_REF] Rello | Frequent words improve readability and short words improve understandability for people with dyslexia[END_REF]. Except for long and unfamiliar words, other issues that individuals with dyslexia may encounter have been listed, such as homophones, words that are orthographically similar, new words, and nonwords [START_REF] Rello | Frequent words improve readability and short words improve understandability for people with dyslexia[END_REF].

Aphasia is a language impairment caused by brain damage acquired by for example stroke, trauma to the head, neuro-degenerative diseases or brain surgery.

Common difficulties experience by individuals with aphasia include high information density, long sentences, long sequences of adjectives, passive voice and noun compounds [START_REF] Carroll | Simplifying Text for Language-Impaired Readers[END_REF]. Other difficulties described in the literature are sentences with object relative clauses and comparisons of word meaning ("is x larger than y?") [START_REF] Hillis | Aphasia: Progress in the last quarter of a century[END_REF].

Intellectual disability (ID) is characterised by low IQ and limitations in many cognitive abilities, such as working memory and executive functions [START_REF] Danielsson | Strengths and weaknesses in executive functioning in children with intellectual disability[END_REF]. Individuals with ID have a delay in reading as compared to typical readers which is manifested in capabilities concerning decoding and reading comprehension [START_REF] Nilsson | Investigating reading comprehension in adolescents with intellectual disabilites: Evaluating the simple view of reading[END_REF]Nilsson et al., 2021a). Using simple texts in order to enhance reading skills is a common strategy in education targeting individuals with ID. Due to the limited amount of textual resources, the teachers face a challenge when choosing accurate educational material, and they often adapt the texts themselves, for example by the use of readability metrics and metrics for reading level estimation, or by writing completely new texts.

Deaf and Hard-of-Hearing. One group of people that may struggle with reading is the deaf or hard-ofhearing. It is established that childhood hearing loss deeply affects language development, and the language deficits may also affect other cognitive developmental areas related to language negatively, such as the development of literacy [START_REF] Lederberg | Language and literacy development of deaf and hard-of-hearing children: successes and challenges[END_REF]. Children that are deaf and hard-of-hearing especially struggle with grammar (ibid.), most prominently syntactically complex sentences [START_REF] Siddharthan | Syntactic Simplification and Text Cohesion[END_REF] and grammatical morphology, as well as a limited vocabulary [START_REF] Fabbretti | Written language abilities in deaf italians[END_REF].

Autism Spectrum Disorder (ASD). With respect to reading comprehension skills, the ASD audience is diverse. The reading difficulties are less straightforward to describe than those exhibited in most of the other target audiences, due to the large variety of symptoms included in the diagnosis. Difficulties understanding figurative language is one of the most prominent problems. According to a meta-analysis of the research on figurative language for individuals with ASD [START_REF] Kalandadze | Figurative language comprehension in individuals with autism spectrum disorder: A meta-analytic review[END_REF], the difficulties seem to be related to basic language skills, and that enhanced general language skills might improve the comprehension of figurative language. The authors highlighted that it is important for individuals with ASD to be exposed to figurative language, and that it is beneficial to provide explanations to such constructions instead of avoiding them. A meta-analysis of reading comprehension skills of individuals with ASD found that the performance on reading comprehension of individuals with ASD depend on text type [START_REF] Brown | A meta-analysis of the reading comprehension skills of individuals on the autism spectrum[END_REF]. Generally, individuals with ASD perform better when reading texts that require little social knowledge. However, they also highlighted the fact that ASD covers a variety of symptoms and deficits, and that the diagnosis in itself does not imply any reading comprehension difficulties. Second Language Learners. This audience differs from many of the other groups, since learners of a new language do not necessarily have any impairment that hinders reading or understanding, but may rather experience difficulties related to a poor vocabulary, unfamiliarity of specific cultural phenomena, or a lack of knowledge about the grammar of the language that is being learnt. Knowing a language's vocabulary has proved to be an important factor for learning a new language. Knowing a language's vocabulary has proved to be an important factor for learning a new language.

Children. Although not having any physical or cognitive disability, a possible target audience for text adaptation or simplification is children. As the Internet is becoming the dominating source of textual information, there is a growing need for text adapted to the reading level of children of different ages. There is a developmental aspect of children's reading that should not be disregarded. The text should not be too simple, since reading encourages learning of new words, and the reading level should thus be adapted to the reading level of the certain reader (De Belder and Moens, 2010).

Simplification techniques

In this section we briefly present techniques that address the different levels of simplification.

Lexical simplification refers to the automatic simplification at word level. It aims at identifying and replacing complex words (or phrases) by an easier-toread alternative. Regardless of how we define simple words, the substitution of words that are more complex to simpler words with the same meaning is a rather well-studied area in ATS (e.g. [START_REF] Paetzold | A survey on lexical simplification[END_REF] for an overview), and although it is a challenging task with many non-trivial subtasks (identifying complex words, disambiguating word senses, etc.), the guidelines of substituting complex words and compounds can be considered possible to automate. The avoidance of jargon and technical terms can be solved with specialised term lists, such as the black list (Stadsrådsberedningen) used by the Swedish public authorities. To not split words on two lines, and how to write (or not write) numerical expressions are other guidelines that are relatively easy to automate. Abbreviations should be avoided and this is also a task that can be automated.

Syntactic simplification refers to the automatic simplification at text level. It aims at simplifying the text by restructuring the words of the sentences, and/or rewriting it into smaller sentences. The issue of keeping the text brief can be addressed through different kinds of ATS. For example, superfluous words and phrases can be recognised and deleted. Such simplification operations have been previously identified for Swedish Easy Language text [START_REF] Decker | Towards automatic grammatical simplification of Swedish text[END_REF], and while operations like these are relatively simple to implement from a technical point of view, one must be aware of the risk that relevant text information might be deleted in the process, which could cause confusion or impair the experienced reading flow of the reader. One guideline is to keep one proposition per sentence. To follow this guideline is slightly more complicated, as it requires some semantic parsing. One possible solution to this could be event-based simplification (see for example ( Štajner et al., 2016)), that identifies mentions of factual events and delete sentences or parts of sentences that are irrelevant to these event mentions. Such simplification approaches could enhance text comprehension by deleting irrelevant information and highlighting the main information, but will naturally also result in some loss of information. It is clear that the deletion of words, phrases or information could result in a more readable text, but there is also a risk that that the resulting text is, in fact, less readable. This could be due to loss of core information, as described above, but it could also be due to more typographic reasons, i.e. that features of the text layout makes the text less appealing to read. One guideline suggests to mix long and short sentences. This could be considered as a parameter when applying guidelines that intend to write as brief as possible. Guidelines that change negative statements to positive statements are possible to automate, but require a mechanism for identifying such structures, as well as a set of rewriting suggestions. For relatively simple cases (PLAIN, 2011), the task is more or less analogous to lexical simplification, but for more complicated cases with, for example, double negations, the task is slightly more complex. Some work has been done on identifying and substituting negations within the medical domain [START_REF] Burgers | How (not) to inform patients about drug use: use and effects of negations in dutch patient information leaflets[END_REF][START_REF] Mukherjee | Negait: A new parser for medical text simplification using morphological, sentential and double negation[END_REF]. It is generally recommended to use personal pronouns, and to address the reader directly. Such linguistic simplification strategies have previously been, at least partially, implemented in a rule-based simplifi-cation system [START_REF] Rennes | A tool for automatic simplification of swedish texts[END_REF]. In the same system, a rule for reordering sentences so that they keep a straight word order, with subject, verb and object kept close to each other, was implemented.

Discourse-level simplification. Syntactic skills facilitate access of meaning from grammatical structures, which is a fundamental process in gaining text meaning at any level of reading comprehension. Discourse skills allow readers to understand the cohesive interlinks within and between sentences and is important for a macro level of passage understanding. The macro level simplification could be easily implemented in an ATS tool with existing techniques. For example, to make the main information easy to find could include the automatic extraction of keywords and present them in clear ways (boldface, headlines, bullet lists, etc.), as well as providing an automatic summary of the text [START_REF] Hahn | The Challenges of Automatic Summarization[END_REF]. Keyword extraction and extractive summarisation (to extract the most important sentences of a text) are techniques that could be relatively easily implemented, whereas abstractive summarisation (to rewrite the summary from scratch) require more sophisticated methods and data for training [START_REF] Monsen | A method for building non-english corpora for abstractive text summarization[END_REF]. The guideline to let the general and the most important information be presented in the beginning could possibly be approached using the same keyword extraction and summarisation techniques.

The Swedish Simplification Toolkit

The Swedish Simplification Toolkit is a modular text simplification system covering a wide range of simplification and summarization functions. It is important to point out that in this context summarization is used as simplification tools as explained in Smith and Jönsson (2011a).

The Swedish Simplification Toolkit can be used via two interfaces that offer complementary functions, namely FRIENDLYREADER (more targeted to reading) and TECST (more targeted to writing), see Figure 1.

Both FRIENDLYREADER and TECST leverage on four modules: STILLETT [START_REF] Rennes | A tool for automatic simplification of swedish texts[END_REF], TEXTCOMP [START_REF] Falkenjack | Towards a model of general text complexity for Swedish[END_REF][START_REF] Falkenjack | Features Indicating Readability in Swedish Text[END_REF], COGSUM (Smith and Jönsson, 2011a;[START_REF] Smith | Enhancing extrac-tion based summarization with outside word space[END_REF] and JULIUSUM [START_REF] Monsen | A method for building non-english corpora for abstractive text summarization[END_REF]. SAPIS [START_REF] Fahlborg | Introducing SAPIS -an API service for text analysis and simplification[END_REF][START_REF] Falkenjack | Services for text simplification and analysis[END_REF] is a REST API aiming to make the services readily available and is used by both FRIENDLYREADER and TECST. All the core modules have been evaluated according to the criteria of intrinsic evaluation and all results can be found in the references given below.

In the rest of the section, first we describe the core modules and than the user interfaces through which the core modules are deployed. STILLETT [START_REF] Rennes | A tool for automatic simplification of swedish texts[END_REF] is a rulebased automatic text simplification tool for Swedish.

It started off as a Java application, partly built on COGFLUX [START_REF] Rybing | Towards a Rule Based System for Automatic Simplification of Texts[END_REF], with a dynamic structure of processes and modules, where each process runs a number of modules. In its original implementation, STILLETT included rules for rewriting to passive-to-active, quotation inversion, rearranging to straight word order, sentence split, and synonym replacement, in addition to the original rule sets proposed by [START_REF] Decker | Towards automatic grammatical simplification of Swedish text[END_REF]. The synonym replacement module implemented in STILLETT originally combined the word pairs from the Synlex lexicon [START_REF] Kann | Folkets användning av Lexin -en resurs[END_REF]) and frequency information. The Synlex lexicon includes 82,000 word pairs including an annotation of level of synonymity. This score was calculated by ratings made by voluntary Internet users, who graded the synonym pairs based on how synonymous they were. In addition to these strategies, we developed and evaluated other methods for finding more comprehensible synonyms.

The first method was based on a corpus of texts in simple Swedish, and the other method was based on theories from the field of cognitive linguistics where hypernyms with characteristics of basic-level words were found to be useful for the task of lexical simplification [START_REF] Rennes | Synonym replacement based on a study of basic-level nouns in swedish texts of different complexity[END_REF]. STILLETT has undergone several improvements (see for instance [START_REF] Johansson | Automatic extraction of synonyms from an easy-to-read corpus[END_REF]) since the first implementation and is today built on Python3, and uses dep tregex 1 [START_REF] Dvorkovich | Yandex School of Data Analysis approach to English-Turkish translation at WMT16 News Translation Task[END_REF], for reordering the dependency trees using rules inspired by Tregex [START_REF] Levy | Tregex and Tsurgeon: tools for querying and manipulating tree data structures[END_REF]. The included rule set still contains the original rules for rewriting to passiveto-active, quotation inversion, rearranging to straight word order, and sentence split, but the rules are further refined. The preprocessor is accessed through the REST API SAPIS and runs the Swedish pipeline with EFSELAB ( Östling, 2018) and MaltParser [START_REF] Nivre | Malt-Parser: A language-independent system for datadriven dependency parsing[END_REF] version 1.9.0. TEXTCOMP is a collection of text complexity measures. The main part of the included measures consists of the SCREAM (Swedish Compound REAdability Metric) features [START_REF] Sjöholm | Probability as readability: A new machine learning approach to readability assessment for written Swedish[END_REF]. SCREAM features include surface, lexical and structural features (the complete list of SCREAM features can be found in [START_REF] Falkenjack | Towards a model of general text complexity for Swedish[END_REF] and [START_REF] Falkenjack | Features Indicating Readability in Swedish Text[END_REF]). More recently, cohesion-related measures have been included, namely the Coh-Metrix measurements, translated to Swedish from the orginal English version (Graesser et al., 2004). COGSUM (Smith and Jönsson, 2011a;[START_REF] Smith | Enhancing extrac-tion based summarization with outside word space[END_REF]) is an automatic extractive summariser, which means that it extracts the most important sentences in order to create a shorter version of the text. COGSUM uses the Random Indexing (RI) [START_REF] Hassel | Java Random Indexing toolkit[END_REF][START_REF] Hassel | Resource Lean and Portable Automatic Text Summarization[END_REF] word space model with pre-trained word vectors, and a modified version of the PageRank algorithm to rank the sentences [START_REF] Chatterjee | Extraction-Based Single-Document Summarization Using Random Indexing[END_REF]. Evaluations have shown that COGSUM performs at an average ROUGE-1 score of 0.6. JULIUSUM [START_REF] Monsen | A method for building non-english corpora for abstractive text summarization[END_REF]) is an auto-matic abstractive summariser. An abstractive summarisation differs from an extractive summarisation in that the words and sentences are not directly extracted from the text, but instead generated based on a pre-trained model. This means that abstractive summaries can contain completely novel words and sentences not present in the source text, while maintaining the key content of the text. JULIUSUM was trained utilising the methodology proposed by [START_REF] Rothe | Leveraging pre-trained checkpoints for sequence generation tasks[END_REF], using a pretrained Swedish BERT model [START_REF] Malmsten | Playing with words at the national library of Sweden-making a Swedish BERT[END_REF] to warm-start an encoder-decoder model. The data used for training consisted of news articles published in Sweden's largest morning newspaper Dagens Nyheter (DN) during the years 2000-2020.

FRIENDLYREADER, Figure 1 left, is a customizable interface that can be adjusted to the specific needs of different target audiences. The idea is that the interface should contain the entire palette of simplification techniques, including both linguistic and layout simplifications, and that the user can adapt the text completely to their individual needs. FRIENDLYREADER is under constant development. In addition to the modules for simplification and summarization, FRIEND-LYREADER also contains text-to-speech functionality, which lets the reader listen to the text. The simplification related to text layout is the possibility to change font size, line spacing, font and line length. In its current state, the user pastes the text into a large text field and presses Run. The view in the left of Figure 1 is then presented to the user. The layout consists of three parts. The main field is the middle field, where the the text is presented to the reader. The left-hand side contains a menu with various types of text simplification. The user is presented with a number of options: 1. Summarise: The user can summarise the text using a slider that outputs summaries of different lengths. 2. Simplify: The user can simplify the text using the syntactic simplification operations of STILLETT. There are check boxes that lets the user choose what operations to make, and the rules are applied directly to the text. 3. Synonyms: By clicking Synonyms, the user can activate the exhibition of synonyms of words in the text. Words with available synonyms are highlighted in the text, and by clicking any such word, the user is presented to a list of possible synonyms. 4. Text-tospeech: The user can have the text read out loud by activating the text-to-speech functionality. 5. Text complexity: The user can see basic text complexity measures, such as LIX and OVIX. The user is also presented to a visualisation of the complexity of the text presented in a radar chart. The right-hand side contains a menu with various simplification options related to text visualization.

TECST (Text Complexity and Simplification Toolkit), Figure 1 right, is a a tool developed for web editors and writers of easy language texts, but could be used by anyone interested in calculating the complexity of a text, as well as applying various text simpli-fication techniques. The intuition behind this tool is that providing the easy language text writers with advanced techniques for measuring and visualising complexity, identifying complex linguistic structures, and give advice on how such structures should be adapted to suit the needs of various target audiences, is one way of making the text simplification process quicker and cheaper, without overlooking the expertise and unique competence provided by the human writer.

The TECST layout, presented in the right of Figure 1, consists of two fields: the editor, which makes up the main part of the tool layout, and the simplification and visualisation field. The editor allows the writer to customise the text using different fonts, font sizes, bold face, bullet point lists, and similar features often included in text editing tools. The simplification and visualisation field, on the right-hand side, presents information regarding the current complexity and simplification suggestions of the text. It has three tabs: visualisation, text information and text simplification. In the visualisaton tab, a text complexity visualisation in the form of a radar chart is presented. In the text information tab, the writer can choose to see a summary of the text, as well as some general information about the current text, such as the text length in words and sentences, as well as a subset of the text complexity measures.

Similarly to the features presented in the visualisation, the subset of text complexity features shown under the text information tab is customisable. The third tab, text simplification, allows the writer to make adaptations to the text. There are four options here. Summarisation: The user can summarise the text, by the use of a slider that regulates the length of the resulting summary. 2. Synonyms: The user can use a check box to highlight the words of the text that have available synonyms, and customise the synonym replacement functionality to mark long words, i.e. words longer than some length chosen by the user. 3. Markings: The user can use check boxes to let the tool identify and highlight different features of the text, such as long words, long sentences, and numbers. The number of characters that make up a long word is customisable, as well as the number of words that make up a long sentence. 4. Text simplification suggestions: The user can get suggested simplifications of complex sentences.

Discussion

The Swedish Simplification Toolkit meets the needs of the target audiences described in Section 2. There is a close match between the characterization of target audience presented in Table 1 and the options provided in the user interfaces. For example, in FRIEND-LYREADER, people affected by ID, as well as nonnative speakers and children, can click the button "Visa Synonymer" (Show synonyms) to fill up the gap of their limited vocabulary. People affected by aphasia have the possibility to convert passive voice into active voice by checking the box "passiv till aktiv form" (pas- At the time of this publication, no options have been implemented to convert figurative language into demetaphorized language. Also, no functionality has been created yet to compensate for the social knowledge that autistic people might miss from a standard text.

As mentioned above, the core modules have been evaluated, but the usability of the toolkit has not yet been tested on target audiences. Such evaluations include two activities, to assess the interface and interaction with the various simplification tools, and assessment of the various simplification techniques and text complexity measures. The latter is currently done in three studies, one involving students with dyslexia, one involving students having an intellectual disability and one with teachers for students with reading difficulties. The first two studies use texts that are adapted using the toolkit. The students read them on paper and assess the usability from various perspectives. The study with teachers investigate the use of text complexity measures and visualisation of text complexity and is conducted in focus groups where the teachers are presented a variety of text complexity measures and visualisations. The reason for using paper and not the interface is, of course, that we want to focus on the usability of the techniques for text simplification and complexity measure, not the usability of the interface. The answers to the research questions are:

1. What types of linguistic simplification are needed, and which ones are implementable for the Swedish language?

Answer: lexical, syntactic and discourse level simplification are needed for the target audiences that we have explored in this paper. However for the Swedish language, many areas are unexplored, as seen from the empty cells in Table 1, especially for discourse-level simplification.

2. Can ATS be conceived, designed and implemented to meet the needs of different target audiences?

Answer: Absolutely yes. We have presented an approach (core modules + interfaces) that shows how target audiences can adapt standard text to their needs.

Conclusion and Future Work

In this paper we presented the Swedish Simplification Toolkit, conceived and designed with the target audiences in mind. The toolkit is the outcome of years of theoretical and computational research. However, much remains to be done. Our next step will be the validation of the adaptations on readers from the target audiences. We are currently testing the effects of specific simplification operations on individuals with dyslexia and intellectual disabilities, and the results of this study will provide a starting point for further development of more customized text simplification. Future work includes the validation of the usability of the interfaces directly by target audiences.

Acknowledgements

This work has been funded by The Swedish Research Council (VR) and Sweden's innovation agency (VI-NOVA).

Begriplig Text. ( 2019). 19 råd för att skriva begripligt. Dyslexiförbundet.

Introduction

Becoming a fluent reader who easily understands a written text is a major societal issue. Comprehension of a text requires rapid progress from word recognition to the development of a mental representation of the text, based on linguistic analyses and ideas from the text, in connection with knowledge of the world.

Acquiring reading skills at the early years

Studying the development of children's general reading skills between the ages of 7 and 9 is of outmost importance because it is a decisive phase for the acquisition of automatic word recognition and text comprehension skills. Indeed, it is during this period that children enter the phase of self-teaching [START_REF] Share | Phonological recoding and selfteaching: Sine qua non of reading acquisition[END_REF][START_REF] Ziegler | Modelling reading development through phonological decoding and self-teaching: Implications for dyslexia[END_REF][START_REF] Ziegler | Learning to Read and Dyslexia: From Theory to Intervention Through Personalized Computational Models[END_REF] and develop automatic reading and text comprehension strategies [START_REF] Bianco | Comment enseigner la compréhension en lecture ? Hatier[END_REF][START_REF] Willingham | The usefulness of brief instruction in reading comprehension strategies[END_REF]. The virtuous circle of self-teaching is initiated when the child starts to use basic spelling-to-sound correspondences to decode novel words. Successful decoding of words (i.e., finding a match in phonological and semantic memory) reinforces the decoding mechanism, and generates an orthographic representation of the word (see [START_REF] Ziegler | Modelling reading development through phonological decoding and self-teaching: Implications for dyslexia[END_REF][START_REF] Ziegler | Learning to Read and Dyslexia: From Theory to Intervention Through Personalized Computational Models[END_REF] for a computational account of this process). Over time, thanks to repetition, the decoding process becomes more and more efficient, allowing a more rapid transformation of the orthographic to the phonological form of words, thus reducing the cognitive cost of decoding. The child will be able to progressively free cognitive resources for understanding what he or she reads rather than for decoding (Sprenger-Charolles & Ziegler, 2019). Comprehension requires many resources such as language knowledge, cultural knowledge, and cognitive efficiencies (memory, attention, reasoning, executive functions, visual abilities) [START_REF] Bianco | Du langage oral à la compréhension de l'écrit[END_REF].

Reading difficulties

Although most children learn to read during early elementary school, some of them fail to benefit from regular classroom instruction. Some reasons have been put forward in the literature. Vernon-Feagans and collaborators (2010, p. 183) distinguish two groups of struggling readers: "The first group comes to school with adequate oral language skills but has trouble with the processes involved in the relationship between oral language and the printed word. The second, larger group is characterized by problems in both oral language/vocabulary and print related/phonological knowledge. This latter group is composed mostly of low-income children who come to school without the prerequisite experiences in emergent literacy to allow them to profit from most whole class instructional practices".

Failing at the early years of reading instruction creates a gap between struggling readers and their peers, a gap that grows over the years. Three-quarters of students who are poor readers in grade 3 will remain poor readers in high school [START_REF] Foorman | The case for early reading intervention[END_REF].

In this context, the main contribution of our work is the proposition of an application for beginning readers of French, based on reading practice through simplified versions of literary and documentary texts (see section 4 for more details on text simplification). The tool can be easily used by children during classroom instruction or in autonomy at home (it has already been tested in six French schools from 2017 to 2019, see section 3.1 for details).

The paper is organized as follows. Section 2 reports on research studies related to reading difficulties of beginning readers in France. In the following sections, 3 to 5, we present the eBook HIBOU, the corpus, and the learning activities. To conclude, in section 6 we discuss on developments and evolutions of the eBook.

Beginning readers in France: an overview

In France, the CEDRE report (2015) that evaluates if the national educational curriculum at primary school has been achieved, shows that 11 % of children at the end of grade 5 fail to understand a text, i.e. they are unable to extract and analyze explicit and even less implicit information 1 . The international program PIRLS (Progress in International Reading Literacy) suggests that 10 out of 60 countries, in which reading comprehension has been tested with fourth-grade children, show alarmingly low average scores [START_REF] Mullis | PIRLS 2016 International Results in Reading. Retrieved from Boston College[END_REF]. The situation seems particularly problematic in France: 6% of the students (4% in Europe) have a standard score below 400, which is taken as an indicator that they do not master elementary reading and comprehension skills [START_REF] Mullis | PIRLS 2016 International Results in Reading. Retrieved from Boston College[END_REF]. Since the first PIRLS evaluation in 2001, the performance in reading comprehension of French students has dropped year after year, especially for complex reading comprehension skills (e.g., inference) and informative texts (e.g., scientific or documentary texts).

In recent national assessments, in which 4 837 students at grade 2 in France were tested on reading fluency and comprehension [START_REF] Andreu | Evaluations 2021 Repères CP, CE1 : premiers résultats » série études[END_REF], it was found that 15.3% had problems in comprehending a text when it was read on their own and 26.9% had difficulties reading aloud. This situation highlights the importance of providing early intervention programs and resources. One of such intervention programs might be developing reading activities with a resource with simplified texts and playful exercises. As showed by Javourey-Drevet and collaborators (2022), reading training through adapted (i.e. simplified) versions of texts improves fluency and comprehension: children find it easier to get to the meaning of a text, which in turn, might encourage exposure to more written texts. Reading simplified texts helps poor readers and children with weaker cognitive skills (nonverbal intelligence, memory). This solution might also be relevant as a motivational and inclusive strategy in a classroom with all kinds of reader profiles: all children read the same text (in terms of contents), but while regular readers have the original text, struggling readers read an adapted version to achieve the same objectives. The eBook that we present in this paper has been developed to address this issue. In addition to a variety of texts in original or simplified versions, the tool provides several lexical games to allow children to enhance their vocabulary skills.

HIBOU: eBook for reading and learning

HIBOU aims at training reading comprehension and fluency.

Genesis of the project

The project was initiated by the Laboratoire de Psychologie Cognitive (UMR 7290) and the Laboratoire Parole et Langage (UMR 7309) in collaboration with a school district in the South of France (Var) and the regional authority (Communauté d'Agglomération Sud Sainte Baume). HIBOU is an electronic book in Apple format (eBook). This kind of solution was suited for a research project on studying the development of children's general

1 Grade 5 is called 'Cours Moyen' 2 (CM2) in France.

2 https://lpc.univ-amu.fr/fr/hibou-livre-interactif reading skills between the ages of 7 and 9. For several years, we followed the evolution of the reading skills of children in grades 2 to 4.

The research project has now come to an end, but we have decided to make the eBook live beyond the project. We have improved it, we gave it a name, HIBOU, we enriched it with some iconography (see Figure 1) and we made it available for download 2 (right now only for the Apple ecosystem, but an open-access web platform is currently being developed, cf. section 6.2).

Figure 1: HIBOU Logo.

HIBOU has been developed on iPads to initially assess the effects of text simplification in beginning readers of French all profiles combined: regular readers, low-readers, dyslexic readers, etc. Since the eBook has been well received by the teachers, who could use it at different moments in the classrooms, we would like now to enhance the use of HIBOU through an open access web-platform, and to enrich the eBook with more advanced devices (i.e. adaptive learning to consider individual differences and profiles, see section 6.2).

Assessing the effects of text simplification for improving reading fluency and comprehension

HIBOU has been used in six primary schools in the south of France for three years (2017)(2018)(2019) to analyze the effects of text simplification in beginning readers of French [START_REF] Javourey-Drevet | La simplification de textes comme outil pour améliorer la fluidité et la compréhension de lecture chez les enfants à l'école primaire: une étude en longitudinal avec des textes littéraires et scientifiques chez des enfants entre 7 et 9 ans[END_REF]Javourey-Drevet et al., 2022). It allowed to collect information on reading times during the three years for about 165 children per year. Each participant in each grade (from 2 to 4) read 20 texts (10 literary and 10 documentary, for each category 5 original versions and 5 simplified versions were available). The choice of an adapted version was randomly proposed. After reading a given text, the participant had to answer to a multiplechoice quiz. Besides the reading and comprehension task, ten tests were used to assess the individual cognitive and language profile, proposed in the same order to each participant.

The results of the reading tests showed that text simplification was beneficial for fluency (F = 81.327, P = .000, ηp 2 = .373) and comprehension (F = 32.020, p = .000, ηp 2 = .189) over the three years for most students and for both types of texts [START_REF] Javourey-Drevet | La simplification de textes comme outil pour améliorer la fluidité et la compréhension de lecture chez les enfants à l'école primaire: une étude en longitudinal avec des textes littéraires et scientifiques chez des enfants entre 7 et 9 ans[END_REF]. This suggests that simplification can be effective across the elementary school curriculum. Text simplification does not directly affect text comprehension but rather impacts factors that influence comprehension, such as word recognition, vocabulary, inferences or morphosyntax.

The results of the reading tests for children of grade 2 showed that simplified texts were read faster than original texts (b=-0.03, SD=0.009, t=-3.5), and that scientific texts (b=0.02, SD=0.008, t=2.73) on reaction times were read slower than literary texts. The type of text did not reach significance (t=1.16). We also obtained a significant effect of simplification showing that comprehension was better for simplified texts than for original versions (b=0.55, SD=0.09, z=6.23). The gains in simplification (difference between simplified and original texts) were greater for poor readers than for good readers. For comprehension in scientific documentaries, simplification gains were stronger for children with poor non-verbal intelligence and low working memory (Javourey-Drevet et al., 2022).

Corpus and reading settings

The texts available in HIBOU are part of the ALECTOR corpus, a collection of 79 original texts in French with their simplified versions [START_REF] Gala | Alector: A parallel corpus of simplified French texts with alignments of misreadings by poor and dyslexic readers[END_REF]. The corpus is already available online3 through a platform that proposes different options for visualizing the texts. It also provides the comprehension questions for each text in the form of multiple-choice questions. Most of the texts from ALECTOR ( 69) have been integrated in HIBOU. On average, the original versions have about 300 words and the adapted versions 275 (both versions are longer in higher levels, going from grade 2 to grade 4). We focused on grades 2 to 4 because the reading programs and activities are part of the curriculum (we left aside grade 1 where children acquire basic decoding skills).

The corpus presents two kind of texts, narrative (literary) and scientific (documentary). While literary texts often reflect the world view and the sensitivity of its author with a language that emphasizes the aesthetics, the rhythm, etc., documentary texts aim at explaining or describing a scientific or technological causality. Scientific texts are descriptive and explanatory with a logical structure based on scientific reasoning. The texts are extracts taken from websites with stories4 or documentaries5 , selected pieces of magazines (BTJ6 , WAPITI 7 for example), and excerpts of books for children (Chichois de la rue des Mauvestis, Ciravégna, 1995, to give an example) 8 . The choice of these genres was motivated by our initial project, keeping in mind that the type of text influences comprehension: narrative texts are more easily understood than informative texts whose topics depend on specific world knowledge [START_REF] Best | Differential competencies contributing to children's comprehension of narrative and expository texts[END_REF][START_REF] Mcnamara | Comprehension challenges in the fourth grade: The roles of text cohesion, text genre, and readers' prior knowledge[END_REF], see section 4.1.

Text simplification

Simplifying a text renders word recognition and decoding more efficient for poor readers and children with weaker cognitive abilities, which has direct effects in reading fluency and text comprehension (Javourey-Drevet et al., 2022). Decoding, i.e. linking graphemes and phonemes, must be automated to read words so that he or she can extract the meaning of a sentence and construct an interpretation. Recognizing a written word does not automatically mean comprehending the meaning [START_REF] Ziegler | Modelling reading development through phonological decoding and self-teaching: Implications for dyslexia[END_REF]. Comprehension is practiced for both oral and written language. The Simple View of Reading (SVR) model (Gough & Tumner, 1986) describes reading comprehension as the product of word recognition and oral comprehension. Word identification constrains comprehension, but it is also the cognitive component of the comprehension process that determines the successful development of a mental representation from the text. Language processing requires complex operations in which memory and different processes of control and manipulation of the information are involved. In this context, text simplification can be a temporary aid for pupils with reading difficulties to continue decoding, reading, and enhancing text comprehension. The idea is to reduce the complexity of a text while preserving its original content (Saggion, 2017). By doing this, the text may be more easily read and understood. Text simplifications in the ALECTOR corpus were manually carried out by a group of researchers in educational sciences, cognitive psychology, linguistics, and speech therapy. It was decided to keep the simplified text as close as possible to the original version to improve readability and understandability while maintaining the original information content. As the corpus was created having in mind the development of a first automatic text simplification system for French [START_REF] Todirascu | HECTOR: a Hybrid Text Simplification Tool for Raw Texts in French[END_REF], we considered only linguistic transformations that could be later implemented. For instance, we gave priority to lexical substitutions and coreference chains substitutions (providing the referent to a pronoun), and straightforward reformulations (e.g. deletion of subordinate conjunctions to split a long sentence into two, see [START_REF] Gala | Recommandations pour des transformations de textes français afin d'améliorer leur lisibilité et leur compréhension[END_REF]). We deliberately avoided reformulations that would have changed the original texts rather drastically (i.e. summarization). Within the field of Natural Language Processing (NLP), automatic text simplification (ATS) has been explored from different angles since the late 90s, and especially very recently with the rapid growth of statistical and deep learning techniques. However, although being an active research area (see Al-Thanyyan and Azmi (2021) for a complete survey), when we started the project there was not yet a sufficiently satisfactory state-of-the-art tool for automatically simplifying texts in French. Therefore, manual simplifications, although very time-consuming, were the only option to provide adapted reading materials for struggling readers in elementary schools. The adaptations were made at four linguistic levels: vocabulary and morphology (lexical adaptations), sentence structures (syntactic adaptations) and pronouns (discourse adaptations). Lexical adaptations, i.e. substitutions of words with simpler synonyms (shorter, more frequent, simpler syllable structure, etc.) were performed based on two standard lexical resources for French: Manulex [START_REF] Lété | MANULEX: A grade-level lexical database from French elementary school readers[END_REF] and ReSyf9 [START_REF] Billami | ReSyf: a French lexicon with ranked synonyms[END_REF]. Syntactic and discursive simplifications followed a set of guidelines that we defined to address the specificities of poor and dyslexic children's needs [START_REF] Gala | Recommandations pour des transformations de textes français afin d'améliorer leur lisibilité et leur compréhension[END_REF][START_REF] Wilkens | Simplifying Coreference Chains for Dyslexic Children[END_REF]. The guidelines include a total of 29 recommendations: six for typography (e.g., police size, interlinear and character spacing, etc.), five for lexical substitutions (thus characterizing simpler synonym candidates), five for morphology (e.g. frequencies in morphological families, verb-tenses, etc.), nine for syntax (e.g. keeping the SVO order when possible, split complex sentences, avoid negation, etc.), and five for discourse related difficulties (e.g., pronoun substitution with a referent, replacement of certain determinants, etc.). The guidelines are free available on-line in the web-site of the ALECTOR project10 . The simplification effort was different according to the type of text. The specific vocabulary of documentary texts, often unknown to young readers, was difficult to adapt because of its specialization [START_REF] Marin | Lire un texte documentaire scientifique : quels obstacles, quelles aides à la compréhension ? Revue française de pédagogie[END_REF]. In these cases, we preferred to keep the original word (e.g., although being words with a complex structure -graphemes, complex syllables, etc.-we kept "atmosphere" and "scaphandre", respectively 'atmosphere' and 'diving suit', instead of trying to provide semantic equivalents). In literary texts, vocabulary adaptations were easier to perform thanks to synonymy (e.g. "mousquetaire" replaced by "soldat", 'musketeer' by 'soldier'). We choose the closest semantic synonym, knowing that full synonymy is rare, and unstable where it does exist (Murphy, 2003, p. 165).

Reading settings in the eBook

HIBOU displays ten different pages. The first one contains a web application integrated into the book, the other pages display pedagogical information and the first sentences of the texts to be read. The heart of HIBOU is the web application.

The reader can choose a text depending on his/her level at primary school (beginner = grade 2, intermediate = grade 3, advanced = grades 4 and 5). The complexity of the texts increases according to the level (longer texts, more complex structures, and more specialized vocabulary). There is currently an average of 20 texts per level. The texts are presented sentence by sentence (it has been shown that presenting texts line by line on electronic devices can improve reading speed and comprehension for struggling readers [START_REF] Schneps | E-Readers Are More Effective than Paper for Some with Dyslexia[END_REF]). The reader can also set up between-character and between-line spacing [START_REF] Zorzi | Extra-large letter spacing improves reading in dyslexia[END_REF]. The spacing of letters and words allows learners to be less influenced by the presence of neighboring letters and words when reading a word (i.e., crowding). After choosing the level of a text, from "Beginner" to "Advanced", the child (or the teacher) chooses among the category "Défi" for texts in the simplified version or "Challenge" for texts in the original version. On the page displaying the texts, an icon corresponds to the theme of the text (see Figure 2). A different color indicates the type of text: green frame for literary texts and red for documentaries.

The application was designed to allow researchers to collect reading times from the readers. By using these data, it was possible to implement an alert message ("you read too fast!", see Figure 3) when the reading time was below the minimum threshold according to the number of words in the sentence.

Learning activities

In addition to the reading activities, using HIBOU the child can improve his/her comprehension of the texts and work on his/her vocabulary through different word-games.

Multiple choice questions

After reading a text, a multiple-choice questionnaire is presented: five questions are asked to the child and for each of them three or four possible answers are proposed. The five questions are based on essential elements or events that are present within the whole text. They are displayed following the chronological order in which they appear in the text. The questions allow the child to consider the overall understanding of the text, sometimes by doing inferences. The proposed answers to each question are all plausible and related to the subject of the text.

For levels 2 and 3, a gap-fill summary is also provided (see Figure 4). Five words must be put in the right place in the text. For each text, we proposed the same grammatical category for statements, e.g. only nouns or only verbs. We also balanced the different grammatical categories between literary and scientific texts. . If the answer is totally incorrect, no reward is offered. For the gap-fill summary activity, each word or group of words correctly positioned provides a gold star (see Figure 5). Children can read a text several times before doing the questionnaire. They can also redo the assessment if they are not satisfied with their performance, but only after rereading the text.

Finally, a page allows the child to quickly see the number of texts already read, the score obtained in the questions and summaries (see Figure 6). The number of activities allows the child to win a cup: bronze trophy for 5 texts read, silver trophy for 10 texts read and gold trophy for 15 texts read.

Word identification

HIBOU proposes three word-games, two on spelling recognition and one focused on syntax and semantics. The games are based on a choice task between two alternatives, which is called lexical decision. The player chooses whether the word or sequence of words, presented in the center of the screen, is correct or not. A sound feedback allows the child to know whether his or her answer is right or wrong, so that he or she only memorizes a correct spelling.

The first word game, Mots Rigolos (Funny Words), is intended for readers of all levels. It is based on words from textbooks. The nonwords are constructed so that they do not look like words, for example "pamelut", "mias", "placiter", "nediter", etc.

The second word game, Pièges piégeux (Trapping Traps), is intended for good readers. The traps are constructed mainly by changing the spelling of words, for example "exemqle", "brôle" ("drôle" funny), "ensemfle" ("ensemble" together), "doucemen" ("doucement" slowly).

The third word game, Mots Mêlés (Mixed Words), is a game on sentences or phrases. Here, the trap sequences are utterances where the words have been scrambled (the SVO or syntagmatic order is not maintained). These utterances do not make sense and are not grammatically or syntactically correct, for instance "inquiets se très montrent" for "se montrent très inquiets" (they are very worried), "dresse sur il se" for "il se dresse sur" (he stands upright), "pas ne copie on" for "on ne copie pas" (do not copy), "bateau de son l'arrière" for "l'arrière de son bateau" (the back of his boat). The child must reorder the sequence.

The players earn coins if they answer quickly and correctly. An overall score is also given at the end of each game, along with a reading speed in words per minute. The score can be improved by playing several times.

Implementation and future work 6.1 From iPads to an online application

As previously mentioned, the heart of eBook is the interactive embedded web application: the reading application is programmed in HTML5 with some API such as Canvas and Web Storage, user interface and interactions are managed by Javascript routines and content of texts, questions and answers are stored in JSON format. HIBOU application can run on all versions of the iPad family, although sounds do not play correctly on iPad2 and earlier versions.

Originally developed as an eBook (Apple standard), the technical specifications of the iPad application make it suitable to be transferred to similar environments such as computers and tablets displaying a standard e-book (EPUB3 international standard). Such a development has recently been carried out by an industrial partner of the original HIBOU project (ISI Inc., France) that publishes electronic books for schools. The new e-book will be available by the end of 2022 with a tier policy that will make the access to the content free of charge for pupils.

Future work

Right now HIBOU has been used in six French schools and all the readers performances in the period 2017-2019 have been logged. There is currently work in progress on the analysis on the performances on the different kind of texts (literary vs documentary, original vs simplified) on the tested school levels (2-4).

In future work, we plan to enrich the corpus with more literary and documentary texts, along with their comprehension tasks, particularly for grade 5. As a result, the platform will provide texts addressed to children of all primary levels (grade 2 to 5). While in the long run, part of the simplifications will become semi-automatized by using a text simplification system for French (work in progress), in the short-term, we will train a group of teachers on text simplification to be able to increase the corpus and the associated learning activities.

Finally, we are planning the possibility to include adaptive learning devices thanks to IA, to propose specific texts and vocabulary activities adapted to each individual profile. By collecting individual feedback on how the child browses the texts and activities, the platform will be able to guide the learners by proposing them adapted materials. By doing so, HIBOU will enable weaker readers to progress at their rhythm while reading the same texts than normal readers.

It will be able to propose new adapted tasks for training, thus encouraging them to read more.

Conclusion

Adapting texts to the needs of struggling readers might be a solution to enhance reading practices and to boost reading comprehension. In this paper, we have proposed HIBOU, and eBook enabling to read and play with words to enhance reading practices and vocabulary skills. While it has been initially developed in the Apple ecosystem, we are now preparing an open access platform that will be freely available to teachers and learners of French.

In future work, we plan to enrich HIBOU with more texts by recruiting and training teachers on text simplification. We are also interested in applying recent advances in automatic text simplification in French to assist the teachers in the work of adapting the texts. Finally, we are foreseeing the possibility to include adaptive learning devices thanks to AI, to propose specific texts and vocabulary activities adapted to each individual profile.

Introduction

The importance of reading for language development, whether in an L1 or an L2, has been argued many times. However, in order for incidental learning of new vocabulary through reading to take place, it is necessary for the reader to already be familiar with the majority of the words they encounter [START_REF] Huckin | Incidental vocabulary acquisition in a second language: A review[END_REF][START_REF] Coady | L2 vocabulary acquisition through extensive reading[END_REF]. Presenting readers with texts of an adequate difficulty level is thus essential to foster their reading skills and vocabulary development. This can be achieved either by comparing reading materials and choosing one of the desired level, or by simplifying elements of a text that are too complex. Both cases require to identify potential sources of difficulty for readers, notably on the lexical level.

Predicting how difficult a word will be for a reader requires large amounts of data, which should ideally be collected directly from the target population. Italianspeaking learners of French, for instance, are likely to struggle with different aspects of the language than Japanese speakers, who in turn will not have the same needs as French-speaking readers with dyslexia. Despite this fact, most of the literature devoted to predicting lexical complexity on the basis of difficulty annotations disregards demographic information and produces reader-independent measures of complexity. This one-size-fits-all approach is a first issue that we wish to address in this article.

A second issue is that there is a lack of tools and resources to collect such annotations of lexical difficulty. Indeed, researchers are typically faced with two options: they can either use crowdsourcing websites and create a batch of Human Intelligence Tasks (HITs) to be completed by workers, or create a custom-made plat-form from scratch. Both of these approaches present shortcomings that can be prohibitive: the first option tends to be expensive and may impose unwanted constraints on the format of the study, while the second requires web programming knowledge and can be very time-consuming. This is a shame, as it may render such studies inaccessible for some, despite them providing valuable insights for the scientific community.

The tool presented in this paper aims to make the process of collecting annotations simpler and accessible for other languages than English, as well as to encourage researchers to collect and account for demographic data. Designed primarily in order to analyse lexical difficulty for learners of French as a foreign language (FFL), it could easily be adapted to different target groups as well. Moreover, the tool strives to involve foreign language teachers in the data collection process, by allowing them to view their students' answers in real time and gain insights into the needs of their class.

The following section (2) will give an overview of previous methodologies employed when collecting similar data, in order to define key features that need to be taken into account. Section 3 will then describe the online platform PADDLe, highlighting the ways it responds to those observations and giving a few pointers on possible use cases. In Section 4, some results from a pilot study conducted through the platform will be presented. Finally, concluding remarks and some future areas of improvement will be proposed in Section 5.

Related Work

The task of identifying words which might pose a problem to readers has been referred to as Complex Word Identification (CWI) or Lexical Complexity Prediction (LCP), depending on whether complexity is conceptualised on a binary or on a continuous scale.

Complexity Annotation Datasets

The success of a model attempting to predict lexical complexity is impacted by its architecture and the relevance of the selected features, but also by the quality of the data with which it is trained. This was made evident during the 2016 SemEval workshop (Paetzold and Specia, 2016), when the shortcomings of the dataset provided to participating teams for the CWI shared task were such that all teams performed rather poorly [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF].

The dataset collected for the shared task contained sentences extracted from corpora based on the standard and simplified versions of Wikipedia. Those sentences were annotated by non-native speakers of English, who were asked to assign a binary complexity label to each lexical word of a given sentence. A value of 1 indicated that the annotator could not understand the target word, regardless of whether they understood the meaning of the sentence as a whole. Sentences destined to make up the training set were annotated by 20 people each, while those forming the test set only received one annotation. Furthermore, the corpus was split in a rather unconventional way, with the test set being over forty times larger than the training set (Paetzold and Specia, 2016). This contributed to the complexity of the task according to [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF], and is probably largely responsible for the poor results obtained by the submitted systems.

To build their predictive models, participating teams were presented with two versions of the training set.

One version provided all individual annotations for each word, and was used by some teams to fine-tune their model. The other attributed a single tag to each word based on whether at least one reader had found it difficult (Paetzold and Specia, 2016). As a result, a word being marked as complex by only one of the annotators was considered just as complex as another to which all 20 annotators attributed a score of 1. Moreover, as pointed out by [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF], binary complexity judgements rely on an arbitrary threshold decided upon by each annotator. A value of 1 in the training set might therefore have represented very different levels of complexity, which added to the overall difficulty of the challenge. The subsequent edition of the task, organised in 2018, refined the collection and presentation of the data, which seems to have had a positive impact on the performance of submitted systems (Yimam et al., 2018).

The second CWI shared task made use of a multilingual corpus, with languages being represented either in both the training and the test set (English, German, Spanish), or only in the test set (French). Annotations were collected through crowdsourcing, using Amazon Mechanical Turk (MTurk). This time, the data were split so that there would be a larger amount of train-ing sentences than test sentences, and words in the test set received several annotations. Complexity was once again represented as a binary value, with a threshold of only one annotation required for a word to be considered complex. Interestingly, however, participants also received probabilistic values based on the proportion of annotators who did not understand a given word (Yimam et al., 2018). Unfortunately, such a probabilistic annotation system was not enough to make up for the shortcomings of binary annotations, as suggested by [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF], who observed that a value of 0.5 only meant that a word was found complex by half of the annotators -and thus simple by the other half. As such, no direct conclusions could be drawn about its level of complexity.

The organisers of the shared task made several other methodological decisions that differed from the previous edition. While in 2016 the words to be annotated were predefined and presented in a sentence, this time annotators were given a paragraph of five to ten lines in which they were free to select up to ten complex items. This constraint might have had an impact on how complete the data were: indeed, it is possible that annotators sometimes had to make a choice when they had identified more than ten words they thought were complex. A second difference with the first edition of the task was that annotators were asked to identify complex multi-word expressions (MWEs) as well as complex words. This, combined with the fact that words were not preselected, might have impacted the data negatively as well. Indeed, Gooding and Kochmar (2018) reported that certain sequences of words were interpreted as single words by some of the annotators, and as MWEs by others. As for the annotators themselves, they were no longer non-native speakers selecting complex words based on their own understanding of them, but a mix of natives and non-natives who were asked to identify items that could be difficult for learners or people with a reading impairment (Yimam et al., 2018). This is an important distinction to make, as it is likely that some annotators selected words that they themselves understood, but assumed other people might not. The predictions obtained from those annotations might therefore not be equally reliable for all target profiles, and perhaps especially so for readers with a learning or reading disability.

Based on the limitations of those two shared tasks, [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF] formulated a list of guidelines for future CWI datasets. These guidelines are:

1. The annotations should be continuous rather than binary;

2. The items to be annotated should be presented in context;

3. Multiple instances of a same item should be included in the dataset;

4. Each item should receive several annotations; 5. Annotators should represent a variety of profiles in terms of fluency and background;

6. Texts included in the corpus should represent different genres;

7. Both single words and multi-word expressions should be considered in the annotation process.

Recent Refinements

It is with these recommendations in mind that Shardlow et al. (2021a) compiled their own dataset for the 2021 shared task on Lexical Complexity Prediction. Similarly to Yimam et al. (2018), they collected annotations through crowdsourcing, using the Figure Eight (previously Crowdflower, now Appen) and Amazon Mechanical Turk platforms. They asked annotators to label the complexity of a word using a 5-point Likert scale, and took the mean of all annotations for an item as its goldstandard complexity. This system allowed them to obtain continuous values ranging from 0 to 1, thus moving from a binary classification task (complex or not complex) towards a prediction task estimating how complex a given word is. The items to annotate were preselected and presented in context, and each token occurred at least twice. Teams were therefore encouraged to take context into account when predicting complexity. Finally, the texts used to produce the dataset were taken from three diverse genres, and multi-word expressions were considered alongside single words in the annotation process. Systems submitted for the task obtained encouraging results, with the highest-ranking teams being very close together in score. This implies that different approaches succeeded in modelling the data almost equally well, which can be interpreted as being a testament to the dataset's quality just as much as to the ingenuity of the participating teams. The recommendations laid out by [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF] thus seem to be genuinely helpful when compiling a dataset for Lexical Complexity Prediction. It is why the tool presented in this paper was devised in a way that would allow researchers to adhere to each of the guidelines when gathering lexical difficulty annotation data.

A fundamental specificity of the three tasks presented above is that they aim to obtain a singular complexity value (whether binary or continuous), conceptualised as being intrinsic to the word itself and not dependent on the reader. What this means is that they assume a word to be somewhat universally complex or non-complex because of the characteristics it possesses, such as its frequency or its length. Inter-rater variability is expected, but seen almost as "noise" in the data resulting from subjective judgements rather than as a phenomenon of interest in itself. By contrast, when building a Text Simplification tool with a specific public in mind, the focus is likely to benefit from being shifted to lexical difficulty (i.e. how difficult someone perceives a word to be, based on their particular language knowledge) instead of complexity. Indeed, people are likely to have different simplification needs based on characteristics such as their proficiency level, reading disability or mother tongue. As a result, trying to predict their needs from those of a heterogeneous group might not always yield satisfactory results. This idea is confirmed by the very low mean inter-rater agreement obtained in the 2016 edition of the CWI task (Krippendorff's α of 0.244) (Paetzold and Specia, 2016). Similarly, Yimam et al. (2017) reported lower agreement scores between non-native speakers than between native speakers of English, most likely due to the fact that the first group is more diverse in terms of language background and proficiency level. Agreement between the two groups was also low, which according to the authors indicates that their simplification needs might differ. As for the third edition of the task, (Shardlow et al., 2021a) didn't report any inter-rater agreement scores.

Since the three tasks did not aim to predict complexity for different groups of readers, no demographic data were provided to the participating teams, even though they had been collected for the first two editions. Nevertheless, Paetzold and Specia (2016) observed that the number of words deemed complex by an annotator was correlated with their age as well as level of proficiency in English. This goes to show that, as suggested by Gooding and Kochmar (2018), including demographic information at both the annotation and the prediction steps should increase the performance of models. In a study whose methodology was inspired by the shared tasks, Tack (2021) addressed their shortcoming by taking individual differences into account. Via a custom-made online reading interface, L2 learners of French were presented with a set of texts (as opposed to sentences or paragraphs) based on their fluency level, so that all readers of the same level would annotate the same texts. They were asked to highlight any word that they personally found difficult, hence providing annotations of a binary nature. Two trials were organised to gather data: one with a smaller pool of participants (n = 9) with diverse L1 backgrounds, and one with a much bigger pool of participants (n = 47) all sharing the same mother tongue. An inter-rater analysis carried out for all annotators in the first trial yielded very similar results to those of Paetzold and Specia (2016), with a Krippendorff's α of 0.26. Grouping the annotators by proficiency level did not seem to have a clear impact on the metric: A2 readers got an α of 0.23, and B1 readers one of 0.30. Interestingly, the agreement rate between participants in the second trial, once grouped by proficiency level, was much higher: between 0.36 (B1 level) and 0.51 (B2 level). These results suggest that annotators with a similar profile (same mother tongue, proficiency level, education level and age, in this case) tend to agree more in their difficulty judgements than annotators with diverse profiles, which confirms the value of including demographic information in a CWI dataset.

It also follows that reading aids targeting a specific profile, such as dyslexic readers, adults with low literacy or L2 learners with a specific L1, would be likely to benefit from gathering data directly from that target population.

Summary and goals of our study

This brief overview of previous approaches to CWI/LCP dataset collection has shown the process to be a complex one, requiring many methodological decisions to be made. As the quality of the dataset appears to have a strong impact on the performance of models trained on it, making the right choices is of critical importance. This is why we created a tool that makes it possible for users to almost effortlessly design their own data collection process, and that encourages them to follow the recommendations formulated above. This tool will be further described in the next section.

Presentation of the Tool

PADDLe (Plateforme d'Annotation De la Difficulté LExicale) is an online platform 1 hosted by CENTAL, whose aim is to make CWI data collection easier. It currently only supports French, but should include a variety of other languages in the future. It allows researchers to create highly customisable webbased reading tasks and download the data in an easily parsable .csv format.

The tool sets out to make following the guidelines proposed by [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF] easy. Researchers are encouraged to define a continuous annotation scale and to include multi-word expressions, as well as to gather demographic information from participants (which would make them aware of how diverse their annotator pool is). The platform plans for words to be presented in context, and for several participants to annotate the same texts. Finally, researchers are free to add as many texts as they want, and can thus easily include several instances of a word as well as texts of various genres in their corpus.

Interest

PADDLe was conceived to offer an alternative to other online survey builders. It is completely free of use, customisable, designed specifically for CWI data collection and does not require any web development knowledge. It also allows teachers who ask their students to participate in a reading task to view the results of their class afterwards, to thank them for their contribution.

Functionalities and Options

The design decisions made when developing PADDLe were based on the conclusions drawn from the literature presented in section 2. The reading tasks created through the platform have the following format:

1. Demographic form: Participants answer a series of questions selected by the researchers;

1 It is available at this address.

2. Text annotation: Participants annotate a text by clicking on words and MWEs they find difficult, according to an annotation scale. The scale, as well as the boundaries of clickable units, are defined by the researchers.

3. Reading comprehension questions: Participants' global comprehension of the text is tested using multiple choice questions. Once they submit their answers, participants are given feedback on whether they answered correctly.

Step 2 and 3 are repeated as many times as decided by the researcher before the study ends. All three steps of the task can be customised as follows:

1. Demographic form: Researchers can select any of the following: participants' identifier (if they don't want the data to be anonymous), age, country of origin, education level, target language proficiency level, other languages known and proficiency level in each of those languages, time spent learning the target language in a non-native context / in a native context, learning or readings disabilities, and "other" (which gives them the option to add an open question).

Text annotation:

The task can include as many texts as necessary, and researchers can decide whether annotators will read all texts or only a subset of them. In the second case, participants can be presented with a) a set of texts chosen at random, b) all texts of a pre-defined and randomly selected group or c) one text drawn at random from each predetermined group. For each text, researchers provide a title, the id to use in the .csv files and the text itself, which must be formatted as one word or punctuation sign by line. This allows multi-word expressions (or other groups of words that are to be annotated as a unit) to be defined, by simply grouping them on the same line. Punctuation is not made clickable for annotators.

Researchers are also asked to provide an annotation scheme to be used in the reading task: each annotation level is given a colour and a label. Currently, the interface only allows to have between 2 and 5 levels (in addition to the "no annotation" level). This is to encourage users to choose a nonbinary scale of annotation.

Reading comprehension questions:

For each text, users are asked to provide between 1 and 6 comprehension questions, each with 2 to 5 possible answers. They must also indicate which of the answers is correct.

Other parts of the study are customisable as well, such as the consent form to be read by participants before beginning the task or the text presenting the study on its home page.

Possible Uses

All the options presented above aim to make the generated studies as malleable as possible. Instead of whole texts, users of the platform could decide to only include paragraphs, or even sentences. Similarly, they could decide to ask for annotations at the phrase or at the sentence level, by grouping words in a way that suits their research purposes. As a result, PADDLe could be used to answer a variety of research questions. One could for instance investigate the link between the proportion of words perceived as difficult and the quality of the general comprehension, based on the questions asked after each text. It could also serve to rate different simplifications of a text, in order to find the one readers understand best, or to compare a system's lexical complexity predictions with empirical difficulty judgements.

Outside of academic research, it could also prove an interesting tool for teachers who would like to pinpoint their students' difficulties. By asking all members of a class to annotate the same text and answer a few questions at the end, teachers could then refer to the visual representation of the results provided by the platform to immediately identify the words or aspects of the text found most difficult by the group. Figure 1 provides an example of said representation.

Pilot Study

To test the proper functioning and scientific interest of the interface before it could be used for larger research projects, a preliminary study was carried out in November 2021 with a small group of 16 L2 learners of French. It yielded some interesting first results, a selection of which will be presented in what follows.

The group was vastly homogeneous, as participants were all 18 year old students from Malaysia who shared the same mother tongue, all belonged to the B1 level in French and were currently following French classes at the Service Universitaire de Langues (SUL, Aix-Marseille university, France). The study used a total of three informative texts, and each participant was asked to annotate two of them -one seen by all participants (B1 level), and one randomly drawn from the remaining two texts (B2 level). As a result, the number of annotations is not balanced between the texts. Participants were asked to answer five comprehension questions after each text, in order to test their global understanding.

The annotation scale employed in the study, inspired by the Vocabulary Knowledge Scale [START_REF] Wesche | Assessing second language vocabulary knowledge: Depth versus breadth[END_REF], a revised version of it [START_REF] Sugiyama | Analyse de la compétence lexicale dans la compréhension écrite des apprenants japonais en franc ¸ais. Revue japonaise de didactique du franc ¸ais[END_REF] and the scale used in the 2021 LCP task [START_REF] Shardlow | Predicting lexical complexity in english texts[END_REF], was the following: 0. Easy word, no annotation;

1. Transparent word (Unknown, but can guess the meaning in context);

2. Vague word (Unsure of the meaning);

3. Opaque word (Cannot understand the word at all).

Each difficulty level was represented by a colour and outline, indicated in a legend above the text. All words started on 0, and participants could click on a word to cyclically increase its difficulty level (once for "transparent", twice for "vague", three times for "opaque" and four to go back to "easy"). Figure 2 shows an example of what the annotation process looked like.

The scale aimed to capture increasing levels of difficulty, from familiar to entirely opaque. All words of the text could be annotated, regardless of their part of speech. As a result, it was expected that most words would receive a score of 0. By contrast with the scales mentioned above, ours only used one level for easy words. This was to avoid asking participants to annotate too many words, as including two different levels for familiar words would have required annotators to consider every single word in a text.

Overview of the Results

Initially, the number of annotators per text was as follows: 13 for text B1 A, 5 for B2 A and 10 for B2 B (a few participants only annotated one text instead of two). However, we decided to discard any participation for which the total time spent on annotating a text and answering the questions was less than 60 seconds. As texts were between 432 and 564 words long and the average reading speed is about 250 words per minute, this seemed a more than reasonable threshold to enforce. Two annotations were thus discarded, from participants who spent 9.5 and 20.5 seconds on texts B1 A and B2 B respectively.

Table 1 provides some descriptive information about each text. On average, participants spent between 8 and 9 minutes on a task (annotation + questions), with a rather high level of variability between annotators. Every participant whose contribution was kept spent at least 2 minutes on a single task. The percentages provided for each level of difficulty were calculated based on the mean score attributed to each word. As possible values ranged from 0 (no annotation) to 3 (opaque word), we chose the following thresholds for each level: 0-0.74 (easy), 0.75-1.49 (transparent), 1.5-2.24 (vague) and 2.25-3 (opaque). Those ranges were selected to separate the space into four equal parts, and were only used to provide an idea of the distribution of difficult words. There does not seem to be any noticeable difference between the three texts, although one could have expected the two B2 texts to have a lower proportion of easy words. However, the average difficulty value of words that received a label other than 0 from at least one participant is slightly higher in the more advanced texts: 0.59 (sd: 0.56) for B1 A, 0.76 (sd: 0.71) for B2 A, and 0.84 (sd: 0.76) for B2 B.

Inter-Rater Agreement

For each text, an inter-rater agreement analysis was carried out using Krippendorff's α for ordinal values [START_REF] Krippendorff | Computing krippendorff's alpha-reliability[END_REF]. The results are presented in table 2.

The values obtained in this study are significantly higher than the one reported for the 2016 edition of the CWI task (0.244, (Paetzold and Specia, 2016)), which could be due to the fact that the group of annotators was more homogeneous. However, major differences 2021) is much more similar to this one and should therefore allow comparisons to be made: inter-rater agreement scores were computed for groups of 8 to 17 participants, and all words of the texts could be annotated.

As mentioned in section 2, a similar score to the one reported by Paetzold and Specia (2016) was achieved by the group of 9 participants with diverse L1 backgrounds, while agreement rates ranging from 0.36 to 0.51 were obtained for the four groups made up of more homogeneous profiles. The agreement rates computed for the present pilot study confirm the finding that annotators with a similar profile produce congruent difficulty judgements. Converting the difficulty levels to binary labels (any value higher than 0 is set to 1) in our data to more closely match the settings of Tack's study had almost no impact on the agreement scores, as shown in table 2. This can be explained by the fact that having fewer possible labels makes agreement by chance between annotators more likely, and goes to show that, as suggested by [START_REF] Antoine | Weighted Krippendorff's alpha is a more reliable metrics for multi-coders ordinal annotations: experimental studies on emotion, opinion and coreference annotation[END_REF], the weighted nature of Krippendorff's α makes it less sensitive to the number of coding categories than other metrics.

Link Between Proportion of Difficult Words and Global Text Comprehension

The question of whether there was a correlation between the annotation provided by a participant and their performance when answering comprehension questions was explored using Spearman's rank correlation coefficient. This non-parametric measure was used as the data did not follow a normal distribution. ber of questions answered correctly by a participant: the proportion of words annotated as difficult (score of 1, 2 or 3) and the time spent on the task. The hypotheses were that 1) annotators who had found more words difficult would have a harder time answering the comprehension questions and 2) participants who spent more time on the annotation task would answer more questions correctly. In other words, we expected to find a negative correlation between percentage of annotated words and number of correct answers, and a positive correlation between time spent and number of correct answers.

Most of the results were inconclusive, and did not seem to support our hypotheses. The seemingly high positive correlation between the amount of time spent doing the task and the number of correct answers for text B2 A was not statistically significant (p = 0.086), probably due to the annotator pool being too small. Interestingly, a significant positive correlation was found between the proportion of difficult words and the number of correct answers for text B1 A (p < 0.01 with a one-tailed test going against our initial hypothesis). The same trend was found when aggregating all data, with a smaller but still significant positive correlation between the two variables (Spearman's ρ: 0.35, p = 0.042). This perhaps surprising result could be due to the fact that participants who completed the task more rigorously found a higher number of words to annotate. Indeed, a Spearman test between the time spent annotating a text and the proportion of words annotated as difficult found a small positive correlation between the two -however, it was not significant (Spearman's ρ: 0.23, p = 0.134).

It is worth noting that none of the annotators found more than 5% of the words of each text difficult (max: 4.63%). This implies that all participants were over the vocabulary coverage threshold of 95% that Laufer and Ravenhorst-Kalovski (2010) argue is required in order to understand a text properly. It is therefore likely that the positive correlation only holds true past a certain threshold, and that the trend would have been reversed had some participants found a higher proportion of the words of a given text difficult.

Conclusion and Future Work

The results presented in section 4 only scratched the surface as regards the types of exploratory analyses which could be undertaken using the tool. Data collected with it could also be fed to a predictive model, as was done during the shared tasks mentioned in section 2. Researchers could then make use of the demo-graphic information that they are encouraged to gather in order to produce personalised predictions of difficulty.

The design of the tool aimed to address the shortcomings of previous approaches, namely the use of binary annotation data, the focus almost solely on English, and the low inter-rater agreement due to great heterogeneity in the pool of annotators. PADDLe is currently being used to gather data for a master's dissertation, which should further demonstrate the value of the interface.

Introduction

Text readability refers to the difficulty in understanding a given text. The difficulty depends on the reader's language ability and knowledge background as well as the linguistic complexity of the written object. Measuring the linguistic complexity or assessing the readability of spoken or written productions has been the concern of several researchers in pedagogy and (foreign) language teaching for decades. Children's language development [START_REF] Blandin | Recommandation d'âge pour des textes (age recommendation for texts). In Actes de la 6e conférence conjointe Journées d' Études sur la Parole (JEP, 33e édition)[END_REF] or second language (L2) learning [START_REF] Yancey | Investigating readability of french as a foreign language with deep learning and cognitive and pedagogical features[END_REF] are mainly in focus with tasks such as age or reader's level recommendation [START_REF] Rahman | Mama/papa, is this text for me?[END_REF]Pintard and Franc ¸ois, 2020), or text simplification (Javourey-Drevet et al., 2022). Works on readability assessment can be classified into three approaches: (1) the statistical formulas, (2) the language model (LM)-based measures, and (3) the supervised approaches. The latter can be categorised further into two types: (3a) the (linguistic) feature-based and (3b) the deep learning-based approaches.

The formulas (1) are often called traditional because they correspond to early works in the field [START_REF] Gunning | The Technique of Clear Writing[END_REF][START_REF] Smith | Automated readability index[END_REF]Kincaid et al., 1975;[START_REF] Mc Laughlin | Smog grading-a new readability formula[END_REF]. Despite the fact they do not capture all the linguistic complexity of the discourse, they have the advantage to be easily implementable. The LM-based approaches (2) benefit from being unsupervised. With the advent of deep learning in especially Natural Language Processing (NLP), the LMs switch from statistical to neural ones [START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF]. They can be considered as formulas' evolution. The feature-based approaches (3a) were the standard approaches before deep learning became the new reference of doing machine learning [START_REF] Balakrishna | Analyzing TextComplexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications[END_REF]Wilkens et al., 2022;[START_REF] Crossley | A largescaled corpus for assessing text readability[END_REF]. In practice, they remain quite competitive for readability tasks with end-users because they offer explicability and concrete (linguistic) objects that humans can discuss and under-stand. Deep neural architectures have been proposed to support the prediction of readability classes (Azpiazu and Pera, 2019b;[START_REF] Deutsch | Linguistic features for readability assessment[END_REF][START_REF] Rahman | Mama/papa, is this text for me?[END_REF][START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF][START_REF] Yancey | Investigating readability of french as a foreign language with deep learning and cognitive and pedagogical features[END_REF].

Works at the edge attempt to combine the advantage of a feature-based approach with a deep learning one [START_REF] Deutsch | Linguistic features for readability assessment[END_REF][START_REF] Qiu | Learning syntactic dense embedding with correlation graph for automatic readability assessment[END_REF]. Despite the interest for the field, resources for processing French are scarce, while open datasets and toolkits exist in other languages. Free implementations of the readability formulas exist for processing English1 . Linguistic feature-based approaches are also available as open source libraries for computing readability metrics in English2 [START_REF] Balakrishna | Analyzing TextComplexity and Text Simplification: Connecting Linguistics, Processing and Educational Applications[END_REF] and in Portuguese. 3The implementation of [START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF]'s neural approaches have been proposed for German readability assessment4 while [START_REF] Deutsch | Linguistic features for readability assessment[END_REF] and [START_REF] Qiu | Learning syntactic dense embedding with correlation graph for automatic readability assessment[END_REF] released their code with the paper respectively for processing English and Chinese. The study of English is also supported by the availability of several corpora [START_REF] Vajjala | On improving the accuracy of readability classification using insights from second language acquisition[END_REF][START_REF] Vajjala | OneStopEnglish corpus: A new corpus for automatic readability assessment and text simplification[END_REF]. Recently [START_REF] Crossley | A largescaled corpus for assessing text readability[END_REF] initiated the creation of an open corpus in English.

In terms of toolkit for processing French, the CEN-TAL Lab. offers AMesure,5 an on-line demonstration application to analyse lexical, syntactic and textual difficulties of French administrative texts and rate the readability with a scale from 1 to 5 [START_REF] Franc ¸ois | Amesure : une plateforme web d'assistance à la rédaction simple de textes administratifs[END_REF]. Recently, the CENTAL has deployed another web service called FABRA6 to assess reading difficulty in French. The toolkit is based on the aggregation of several linguistic features (Wilkens et al., 2022). Based on fine-tuning BERT on texts from French as a Foreign Language (FFL) course material following the Common European Framework of Reference for Languages (CEFR), [START_REF] Yancey | Investigating readability of french as a foreign language with deep learning and cognitive and pedagogical features[END_REF] will offer a web interface7 for readability evaluation. Without discussing the performance of these deployed analysers, the quality of a toolkit as a service will depend on both the bandwidth availability and the power of the server. In addition, it will act as a blackbox and will not allow modification.

Although there are nice projects funded by the National French Agency such as texttokids 8 , there are little corpora freely available yet. We can mention the works of (Gala et al., 2020) and (Azpiazu and Pera, 2019a) who make available French corpora with aligned original and simplified texts. Our contributions are:

1. ( 1) three open corpora for supporting research on readability assessment in French, 2.

(2) a dataset analysis with traditional formulas and an unsupervised measure, 3.

(3) a toolkit dedicated for French processing which includes the implementation of statistical formulas, a pseudo-perplexity measure, and stateof-the-art classifiers based on multi-layer perceptron (MLP), Support Vector Machine (SVM), fast-Text and fine-tuned BERT for predicting readability levels, 4. and (4) an evaluation of the toolkit on the three data sets.

The library and corpora will be made available under open license in a repository later on. The rest of the paper is structured as follows: Section 2 introduces the related work on readability measures and prediction techniques. We also say a few words on the grades system in France. Section 3 presents the corpora we collected for supporting readability studies and recommendation or prediction tasks. Section 4 presents a thorough analysis of our corpora as well as the report of the results of state-of-the art prediction systems.

Related Work

The readability assessment issue has been addressed by several researchers trying to find pertinent factors to take into account in order to automate this task. [START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF] offer a consolidated review of the major approaches.

Traditional formulas

Readability measures mentioned in this section refer to methods based on mathematical functions linking text structural characteristics to a simple value of readability as perceived by humans. The structural characteristics are statistical measures on each text such as total words, total sentences, number of long words and number of syllables.

The Gunning fog index (GFI) formula [START_REF] Gunning | The Technique of Clear Writing[END_REF] takes into consideration the total number of words and sentences and the number of long words (long words are defined as words longer than 7 characters). GFI value and readability are negatively correlated meaning that a high GFI value indicates a higher readability measure. The Automated readability index (ARI) formula [START_REF] Smith | Automated readability index[END_REF] corresponds to the number of study years needed to understand a text. It uses as features, similar to GFI, the total number of words and sentences in a text with the addition of the total number of characters. The Flesch reading ease (FRE) formula (Kincaid et al., 1975) brings an addition to the already mentioned formulas. It uses total number of syllables in a text to compute a score that increases with more readable documents. The Flesch-Kincaid grade level (FKGL) (Kincaid et al., 1975) is a similar formula to FRE, it corresponds to the number of years of education needed to understand a certain text.

The Simple Measure of Gobbledygook (SMOG) formula [START_REF] Mc Laughlin | Smog grading-a new readability formula[END_REF] similar to FKGL and ARI returns the number of years of education required to understand a text. It uses the number of polysyllables -the number of words containing three or more syllables in a text. Flesch's reading ease has been adapted to French language by [START_REF] Kandel | Application de l'indice de flesch à la langue francaise[END_REF]. They made changes to the coefficients of FRE to take into account the length difference between French and English Words. Their formula is named Reading Ease Level (REL).

Language model-based measures

Perplexity (ppl) is a common intrinsic metric for evaluating language models. It is defined as the exponential average negative log-likelihood of a sequence. For masked language models like BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF], [START_REF] Salazar | Masked language model scoring[END_REF] proposed an adaptation called the pseudo-perplexity (pppl). The lower the score is the better the language model is able to "predict" a given text. [START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF] also proposed a ranked sentence readability score (RSRS) which exploits language models to estimate a readability score for each word in a specific context.

Supervised approaches

Many traditional machine learning algorithms were experimented for the readability prediction task [START_REF] Schwarm | Reading level assessment using support vector machines and statistical language models[END_REF][START_REF] Vajjala | On improving the accuracy of readability classification using insights from second language acquisition[END_REF]. These methods used various kind of features: traditional formulas scores, discourse cohesion measures, lexico-semantic features, syntactic and language model measures. The literature reveals that Support Vector Machine (SVM) classifier was giving the best results for [START_REF] Martinc | Supervised and unsupervised neural approaches to text readability[END_REF]. Feature-based approaches are language and genredependent. With the success encountered by Deep Learning methods for tackling numerous NLP tasks, end-to-end neural architectures were also proposed for difficulty estimation or readability classification. [START_REF] Filighera | Automatic text difficulty estimation using embeddings and neural networks[END_REF] designed architectures comprising three global layers: an input layer made of contextual and non-contextual word embeddings (word2vec [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF], BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF] Since 2014, the French primary school (primaire) has been split into four learning cycles9 . To erase any maturity differences, the learner has 3 years to acquire the required skills before the next stage: cycle 1 "first learning" (under 6, PS-GS), cycle 2 "fundamental learning" (6-8, CP-CE2), cycle 3 "consolidation" (9-11, CM1-6e) and cycle 4 "enhancement" (12-14, 5e-3e). At the primary school, the reading levels follows this development.

In order to provide a basis for recognising language qualifications, the Council of Europe proposed to "organise language proficiency in six levels, which can be regrouped into three broad levels: Basic User (beginner A1, intermediate A2), Independent User (B1, B2) and Proficient User (C1, C2)" called the The Common European Framework of Reference for Languages (CEFR).10 A1 corresponds to beginner at primary school, A2 to intermediate at secondary school, B1 to newly independent at the end of the compulsory education (collège), B2 to advanced at high school (baccalauréat), C1 to autonomous learner, C2 to master.

Table 1 attempts to provide an overview of the alignment between the ages, grades and the education syllabus.

Datasets

Our datasets result from the compilation of various sources releasing children's and young adult's books under open licences (mainly in CC BY). These include the following projects: littérature de jeunesse libre, StoryWeaver, Bibebook, Je Lis Libre, WikiSource and Gutenberg. Some of these sources are collecting and packaging books coming from other sources. For more convenience, we will refer here to three distinct packages: littérature de jeunesse libre (ljl), Bibebook (bb) and Je Lis Libre (jll). Books belong to the literary genre (children story, adventure novel, poetry, theatre play...).

The littérature de jeunesse libre (ljl)11 corpus compiles children's books acquired from the StoryWeaver platform which defines four reading levels: 12 (lv1) beginning to read (easy words with repetition, short sentences, up to 250 words), (lv2) learning to read (simple concepts, from 250 to 600 words), (lv3) reading independently (popular topics with well sketched-out characters, 600 to 1500), (lv4) reading proficiently (rich vocabulary, word play, more than 1500 words). In our interpretation, we consider lv1 and lv2 covering the second learning cycle (lc2), and lv3 and lv4 covering the third one (lc3). Books are mainly children stories translated from Hindi or African literature. The 746 books were written by 460 distinct authors.

With the bibebook (bb) project, the Association de Promotion de l'Ecriture et de la Lecture (APEL) aims at promoting writing and reading activities for young adults. The corpus references books 13 that are in the public domain (i.e. with authors who died more than 70 years ago), and which are known as classic masterpieces that young adults read in French secondary school (such as La Fontaine's tales, Molière's plays, Vernes's adventure novels, Zola's novels, Racine's plays). Books are organised in three levels of difficulty: easy reading (age 10-12), intermediate reading (12-15), and advanced reading (15-18). The 208 books are written by 72 distinct authors.

The je lis libre14 project is a small database which refers to a subset of books present in bibebook database. The organisation is different and follows the reading recommendation from the Ministry of Education for a given secondary school grade: grades from 6 to 3 (3 being higher than 6 in the French education system).

To collect the books, we scrapped each website (while respecting the robots.txt restrictions) to get the pdf or epub files of each document, and used common tools, such as the pdftotext python library15 to convert them into text format. Thanks to adhoc filters or manual operations, we were able to clean them as much as possible by removing meta-data descriptions (header and footer). Dataset statistics are presented in Table 2. Sentence splitting and word tokenization were performed thanks to the NLP spaCy library and its fr core news sm16 model. When looking at the number of tokens or the number of documents for each readability class, we clearly see that the corpora are unbalanced. We can also note that the corpora are small in terms of number of documents while being big in terms of number of sentences and tokens. We do not report here the average number of tokens per document but we can easily infer from the

Datasets analysis and class prediction

In this section, we report:

• First the readability analysis of our corpora thanks to the traditional formulas and the pseudoperplexity measure (cf. Section 4.1) ;

• Then we evaluate baseline approaches over the corpora and provide preliminary results for the class prediction task (cf. Section 4.2).

In both studies, we did not use the raw versions of the corpora. For each corpus, due to the imbalance between the classes, the size of the documents and the small number of documents we have at our disposal for #d'). The readability classes follow an increasing order: lv1 < lv2 < lv3 < lv4, easy < interm. < advan and 6e < 5e < 4e < 3e.

each class, we decided to artificially generate new documents (d ′ ) from the big ones. New documents were generated to be between 140 and 200 words, with all beginning and ending not starting or ending in the middle of sentences. In [START_REF] Crossley | A largescaled corpus for assessing text readability[END_REF], the authors did the same to build up their corpus. The distinction is that our generation is automatic and consequently our generated documents may not correspond to an idea unit. For the ljl corpus, the strategy was to split the big documents into smaller pieces while for bb and jll, which comes with much larger documents, the strategy was to select text excerpts. We could not get smaller pieces with the ljl corpus. For the bb and jll corpora, we generated documents to obtain about 1k of documents per class. The number of generated documents remains proportional to the number of actual documents. Last column of Table 2 indicates the number of generated documents.

Dataset analysis

Table 3 reports the scores given by the traditional formulas and the pseudo-perplexity measure presented respectively in Section 2.1 and 2.2. The scores were averaged over all the documents of a given class. The pppl measure was computed by using the generative GPT model gpt-fr-cased-small.17 For each measure, we calculated the Pearson coefficient (pscore) in order to estimate the linear correlation between these values and the levels labeled in each corpus.

Regarding the ljl corpus, the computed scores of each measure match the classes: The higher a readability class is, the higher the scores are. This is translated into a positive We observe also that despite a positive increment, the lv3 and lv4 classes are closer than each of the other class pairs. This can indicate some difficulties to differentiate between them.

Looking at the bb and jll corpora, there is no significant correlation between the scores and their respective classes. We note, however, that for both corpora, the measures depict a peak in difficulty for the intermediate classes (namely the "intermediate" class in bb and the "5e" class in jll. In addition, the small deviation between the scores of the "4e" and the "3e" classes in the jll corpus seems to indicate there is no clear difference between the classes. Concerning the pseudo-perplexity scores, the Pearson coefficient does not detect any correlation with the readability classes. But the pppl seems to confirm the closeness in the language of the lv3 and lv4 classes of the ljl corpus. It also confirms that the intermediate classes of the bb and jll corpora seem to follow an unexpected behaviour. While in primary school the guideline is to pursue the children's development and to increase iteratively the linguistic complexity of the text, it seems that the reading recommendations in secondary school does not follow the same objective. Indeed the pedagogical choices are often to follow an historical progression, from old written texts to more contemporary ones. Further observations of the corpus are necessary to clarify these numbers.

Readability class prediction

The current section reports the results obtained with four baselines over the three corpora for a class prediction task. The baselines differ from the text repre-sentation and the learning and classification algorithm. Two baselines are feature-based approaches and rely directly on words. One is based on non-contextual subword embeddings; it is fastText [START_REF] Joulin | Bag of tricks for efficient text classification[END_REF].

And the last one is based on contextual embeddings; it is BERT [START_REF] Devlin | Bert: Pre-training of deep bidirectional transformers for language understanding[END_REF].

Classifiers

In practice, thanks to the scikit-learn18 library, we experimented several traditional machine learning algorithms (SVM, Random Forest, Logistic regression, multinomial Naive Bayes and multi-layer perceptron (MLP)) with normalised (or not) bag-of-words and TF-IDF text representations. We report only the very best of these approaches, namely the SVM and the MLP classifiers with a TF-IDF representation without any text normalisation.

FastText is a word embedding method that is an extension of the word2vec model [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF]. Instead of learning vectors for words directly, fastText represents each word as sub-word character n-grams. This offers more robustness to deal with previously unseen words. A document vector is obtained by averaging the subword embeddings. For the classification task, a multinomial logistic regression is used, where the document vector corresponds to the features. task. For our experiments, we used CamemBERT, a state-of-the-art language model for French [START_REF] Martin | Camembert: a tasty french language model[END_REF]. The implementations of the fastText and BERT classifiers were supported by the ktrain library [START_REF] Maiya | ktrain: A low-code library for augmented machine learning[END_REF]. The evaluation of the algorithms is based on the precision, recall, F1-score, accuracy, macro average F1score metrics. The reported results for MLP and SVM were obtained by cross validation by splitting each dataset into five folds. For fastText and CamemBERT, the scores were obtained by averaging the scores over five runs, eachone with a randomly selected dataset with 90% for training and 10% for validating. Optimal learning rate (lr) and number of epochs hyperparameters were set up by utilizing the following learning rate schedules: the triangular policy [START_REF] Smith | Cyclical learning rates for training neural networks[END_REF], the 1cycle policy [START_REF] Smith | A disciplined approach to neural network hyper-parameters: Part 1 -learning rate, batch size, momentum, and weight decay[END_REF], and SGDR Warm Restart [START_REF] Loshchilov | Sgdr: Stochastic gradient descent with warm restarts[END_REF]. We began training with a maximum value for lr. This was set to 0.0001 for fastText and 2e -5 for CamemBERT.

Results

Table 4 presents the results respectively for the corpora ljl, bb and jll. The best models are fastText and CamemBERT. Both are competing with each other over the three corpora but CamemBERT slightly outperforms fastText. FastText remains competitive probably by taking advantage of of a vocabulary made of subwords. MLP and SVM achieve similar performance; SVM being better on the ljl and bb corpora. For all the models we note that results are higher in the jll corpus than in the bb corpus. This may come from the fact that the task may be harder for the bb corpus since there is a larger number of documents and fewer number of classes to differentiate the documents. The lowest performance scores were obtained for the ljl corpus, but this may due to the size of the corpus which remains relatively small. The difference of performance between the classes of a same corpus seem to match the imbalance in number of instances between the classes. This suggests that future experiments should benefit from taking into consideration class weights. In general, the results are not bad but there is room for improvement in particular on the prediction task on a very small corpus (i.e. the ljl corpus). Despite the fact that the corpus and the number of classes were different, the results are consistent with the results of [START_REF] Yancey | Investigating readability of french as a foreign language with deep learning and cognitive and pedagogical features[END_REF] who observed that best results were obtained with a fine-tuned Camem-BERT model.

Conclusion

Supporting primary and secondary education and developing effective learning environments are part of the Unesco's open science recommendations and its Sustainable Development Goal 4 (SDG4). 19 What is noticeable about the modern age is the efforts for researchers to enable other peers to access to the data and tools they develop [START_REF] Crossley | A largescaled corpus for assessing text readability[END_REF]Wilkens et al., 2022). With this paper, we aim at contributing to the efforts. Our material contributions are three corpora and a library for assessing readability in French available under open licences20 . There are prospects for improving and extending the current work. One major direction will be to deepen the data analysis and the assessment of the data quality. Indeed, the low correlation coefficients question the quality of the bb and jll corpora. We plan to use the distribution of the current measures to filter out the outliers and observe whether the correlation scores improve. These measures attempt to capture the lexical complexity as well the syntax complexity (with the pppl). In order to verify the reliability of these measures to distinguish the different classes, we will compute correlations with additional lexical complexity measures (for instance by computing the distribution of the Dubois-Buyse school lexicon [START_REF] Ters | L'échelle dubois-buyse d'orthographe usuelle franc ¸aise[END_REF] over the classes of each corpus) as well as complementary measures designed for capturing the semantic complexity and the discourse cohesion of the texts. One appealing aspect with such linguistic features is that they can support the implementation of readability measures which allow to build self-explanable systems. Eventually we will also manually annotate a sample of the corpus to confirm there is no issues in the way the texts have been categorised.

The study of the classification errors may also allow to understand how to improve our datasets. Since the process of building documents is partially artificial, it is important to ensure that classifiers actually learn to distinguish between readability levels and not from hidden variables (such as authors, topics...). Attention will be paid to other datasets configurations to verify the independence of the classifiers to the variables. Last, we plan to extend the corpora. Since the data annotated by [START_REF] Crossley | A largescaled corpus for assessing text readability[END_REF] is available in numerous languages, we can study the possibility of transferring to French their manual annotation. New genres such as encyclopaedic textbooks21 will be considered, this could allow us to compare texts written by children and texts written by adults for children.

Introduction

Mastering a foreign language has become increasingly important in everyday life. English proficiency, for example, is correlated to higher salaries (e.g., [START_REF] Boyd | Immigrant language proficiency, earnings, and language policies[END_REF][START_REF] Pendakur | Colour my world: Have earnings gaps for canadianborn ethnic minorities changed over time?[END_REF][START_REF] Adamchik | Wage returns to english proficiency in poland[END_REF]). The increase of foreign language learners also implies an increasing number of participants in the proficiency tests, such as TOEFL and IELTS, which may impact the test cost (e.g. including the need for training new evaluators). Automated scoring makes assessing language proficiency more viable for largescale tests, which may be mandatory if one wants to study abroad [START_REF] Weigle | English language learners and automated scoring of essays: Critical considerations[END_REF]. In addition, the feedback provided by automated scoring based on linguistic features can also provide valuable insights to facilitate language learning [START_REF] Srichanyachon | Teacher written feedback for l2 learners' writing development[END_REF]. For English, various tools have been used to support the development of research on foreign language writing development. Some examples are Coh-Metrix (Graesser et al., 2004), L2 Syntactic Complexity Analyzer [START_REF] Lu | Automatic analysis of syntactic complexity in second language writing[END_REF], CTAP [START_REF] Chen | Ctap: A webbased tool supporting automatic complexity analysis[END_REF] and TAASSC [START_REF] Kyle | Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication[END_REF]. Although these tools provide a myriad of functional language descriptors, they are hardly extensible. Also, they are usually based on token units or n-grams as words to build features. However, multiwords expressions raise numerous challenges in natural language processing, descriptive linguistics and foreign language acquisition due to their formulaic structure [START_REF] Wray | Formulaic language in learners and native speakers[END_REF][START_REF] Wray | Formulaic language and the lexicon[END_REF], unit at some level of description [START_REF] Calzolari | Towards best practice for multiword expressions in computational lexicons[END_REF], and interpretation crossing word boundaries [START_REF] Sag | Multiword expressions: A pain in the neck for nlp[END_REF]. MWEs include several subcategories, such as verb-noun combinations (e.g. rock the boat and see stars), verb-particle constructions (e.g. take off and clear up), lexical bundles (e.g. I don't know whether) and compound nouns (e.g. cheese knife and rocket science). Targeting English as a foreign language, MWE's importance is undeniable when considering its ubiquity in the discourse produced by native speakers. Moreover, a learner may be considered handicapped in a language without knowledge about MWE [START_REF] Muraki | Concreteness ratings for 62 thousand english multiword expressions[END_REF]. [START_REF] Glucksberg | Metaphors in conversation: How are they understood? why are they used?[END_REF] estimated that English native speakers produce about four multiwords per minute and Jackendoff (1997) identified that they likely have the same order of magnitude as a single word in the mental lexicon of native speakers.

Given the prevalence of MWEs in native speakers' speech, we investigate their impact on learners' proficiency prediction. We compare MWE metrics with classic linguistic ones commonly used to identify learner proficiency to achieve this goal. In particular, we focus on MWEs and their concreteness (i.e., degree of concreteness/abstraction of an MWE). The main contributions of this paper are the following: (1) profile of MWE concreteness usage across the different levels of the Common European Framework of Reference for Languages (CEFR); (2) analysis of the capacity of MWE scores to individually identify the level; and (3) comparison of these scores with classic scores used to predict learners' level. This work is organized as follows: first, we shortly review the literature concerning the essay scoring focusing on English and linguistic descriptors in Section 2.

In Section 3, we present the linguistic descriptors and corpus used in this work. Next, in Section 4, we evaluate the impact of MWE descriptors on the prediction of learners' proficiency. Finally, we conclude by discussing the results in Section 5.

Related Work

Approaches for automatic prediction of language proficiency are mostly based on machine learning. These can be broadly divided into deep learning-based and feature-based, the latter being more interpretable. We thus focus on feature-based approaches for facilitating the comparison with the MWE descriptors.

The features have been drawn from explorations of linguistic patterns in corpora. For example, [START_REF] Lan | A corpus-based investigation on noun phrase complexity in l1 and l2 english writing[END_REF] showed that there is an association between the use of noun phrases and whether the author is an L1 or L2 user of English. The first language plays a vital role in the developmental trajectories, characterizing behavior, as discussed by [START_REF] Chen | Automatic extraction of subordinate clauses and its application in second language acquisition research[END_REF], who observed different developmental trajectories in learners whose L1 has clause subordination structures distinct from English. They may overuse or underuse certain grammatical structures depending on their CEFR level [START_REF] Zilio | Investigating productive and receptive knowledge: A profile for second language learning[END_REF]. Errors, such as punctuation, spelling and verb tense, are significant in predicting specific CEFR levels [START_REF] Ballier | A supervised learning model for the automatic assessment of language levels based on learner errors[END_REF]. [START_REF] Jung | Predicting second language writing proficiency in learner texts using computational tools[END_REF] demonstrate relevance regarding the conceptual similarity between paragraphs when comparing with the lexical diversity, familiarity and abstractness of the word. Some works also combined properties such as part-of-speech and n-grams [START_REF] Yannakoudakis | A new dataset and method for automatically grading esol texts[END_REF], the edit distance between errors and their corresponding target hypothesis [START_REF] Tono | Automatic extraction of l2 criterial lexico-grammatical features across pseudolongitudinal learner corpora: using edit distance and variability-based neighbour clustering. L2 vocabulary acquisition[END_REF], and syntactic, lexical, discourse and error features [START_REF] Vajjala | Automated assessment of nonnative learner essays: Investigating the role of linguistic features[END_REF]. [START_REF] Jung | Predicting second language writing proficiency in learner texts using computational tools[END_REF] showed that length-based features, specifically the number of words, are stronger predictors than the cohesion and syntactic complexity. However, they also emphasize that text length alone cannot be considered a good predictor of writing quality. Moreover, despite the variety of language-based features studied, only a few studies have tried to test multidimensional models with several features to investigate how they are comparable (e.g. [START_REF] Tack | Human and automated cefr-based grading of short answers[END_REF]). Corpus specificities may also bias studies. In EFCAM-DAT [START_REF] Geertzen | Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat)[END_REF], the task (i.e., the prompt 1 ) presented in the test might drive the learner to use different skills, as discussed by [START_REF] Alexopoulou | Task effects on linguistic complexity and accuracy: A large-scale learner corpus analysis employing natural language processing techniques[END_REF] and by [START_REF] Michel | Effects of task type on morphosyntactic complexity across proficiency: Evidence from a large learner corpus of a1 to c2 writings[END_REF], who identified task influence by exploring lexical and syntactic features.

Despite the amount of work on language assessment, there is still a comparability gap in the results. In this sense, [START_REF] Ballier | Machine learning for learner english: A plea for creating learner data challenges[END_REF] called for solutions for predicting CEFR levels for written productions using only the French part of the EFCAMDAT. Competitors used a variety of machine learning approaches with different processes including feature engineering, data representation and classification. The winner, Balikas (2018), used Gradient Boosted Trees and compared the use of language models, part-of-speech, bag-ofwords (BoW) and Latent Dirichlet Allocation (LDA) as features. Interestingly, their results of both BoW and LDA models were close. [START_REF] Arnold | Predicting cefrl levels in learner english on the basis of metrics and full texts[END_REF] use a multi-dimensional feature representation of written essays exploring LSTM and dense layers achieving an accuracy of 70%. Using EFCAMDAT texts written by French and Spanish learners, [START_REF] Gaillat | Predicting cefr levels in learners of english: the use of microsystem criterial features in a machine learning approach[END_REF] achieved an accuracy of 82% when exploring microsystems, identifying lexical and syntactic features as the more significant.

1 Prompts are the proposed topics for the writing.

Focusing on MWE, the literature has reported different effects depending on their type. [START_REF] Römer | A corpus perspective on the development of verb constructions in second language learners[END_REF] and [START_REF] Römer | Observing the emergence of constructional knowledge: Verb patterns in german and spanish learners of english at different proficiency levels[END_REF] studied the verb-argument construction (VCP) repertoire of English learners, remarking an increase in vocabulary, productivity and complexity according to learners' level. [START_REF] Du | Collocation use in efl learners' writing across multiple language proficiencies: A corpus-driven study[END_REF] studied collocation usage by English learners, using a list of 2,501 make/take+noun (the direct object). They observed that proficient learners tend to use collocations containing more semantically complicated and abstract nouns. [START_REF] Garner | A phrase-frame approach to investigating phraseology in learner writing across proficiency levels[END_REF] examined the use of p-frames2 by L1 German learners of English as a foreign language, observing that p-frames in texts from higher proficiency learners are more variable, less predictable, and more functionally complex. [START_REF] Arnon | More than words: Frequency effects for multi-word phrases[END_REF] explored the perceived transparency affected by multiword phrases (MWP; the specific combinations of words that occur together more than would be predicted by chance). For that, they compared verb+object phrase3 knowledge among intermediate and advanced L2 English learners in comparison to monolingual L1 speakers, observing that intermediate learners performed less accurately and advanced learners performed comparably with native English on transparent and semi-transparent items but were less accurate for non-transparent items. Moreover, both intermediate and advanced learners answered non-transparent items less accurately than transparent items. Exploring MWE validity, [START_REF] Dahlmann | Pauses as an indicator of psycholinguistically valid multi-word expressions (mwes)?[END_REF] studied pauses in various instances of very frequent extracted MWE candidates (i.g. n-grams) from a learner corpus. [START_REF] Arnon | More than words: Frequency effects for multi-word phrases[END_REF] studied the frequency of four-word phrases using the distributional information, identifying an association between frequency and the identification as a valid MWE. Based on n-grams statistics, [START_REF] Jung | Predicting second language writing proficiency in learner texts using computational tools[END_REF] identified a correlation between their frequency and essay score.

Methodology

Considering the goal of investigating the impact of MWE usage on the prediction of learners' proficiency, we annotated a corpus of essays written by English learners with features describing MWE occurrence and its concreteness. We also annotate the corpus with additional features aiming to assess the importance of MWE features. After we have the annotated corpus, we run the tests described in Section 4. We used EFCAMDAT [START_REF] Geertzen | Automatic linguistic annotation of large scale l2 databases: The ef-cambridge open language database (efcamdat)[END_REF], created by the University of Cambridge and Education First (EF) to supply the lack of data for numerous speakers across the proficiency spectrum and the amounts of annotated data. In total, it consists of +1M of essays across the 6 CEFR levels written by learners of 198 nationalities. Levels and nationalities are not balanced (e.g. 40% of all texts are from Brazilians, and 53.04% and 0.16% of the texts are at levels A1 and C2, respectively). Therefore, we selected only the 10 most common nationalities and joined levels C1 and C2 due to their low representation in the corpus. We also truncated the number of essays using the level with the least essays by nationality. For studying the impact of MWE on text produced by English learners, we explored 2 features:

1. MWE usage (MWE cnt ) a list-based [START_REF] Muraki | Concreteness ratings for 62 thousand english multiword expressions[END_REF] feature that consists of 62 thousand expressions from recommended expressions for learners, stimuli expressions used in language studies, dictionaries and n-grams frequency lists.

2. Concreteness of MWE [START_REF] Muraki | Concreteness ratings for 62 thousand english multiword expressions[END_REF] MWE conc . In other words, how the 62 thousand MWE are perceived as concrete/abstract according to 2,825 participants (all English native speakers). 4 The provided annotation was cleaned by removing participants with less than 33% of the ratings and with low correlation with others. On average, each MWE received 10.4 valid scores (minimum of 10).

Aiming to compare these 2 features with others reported in the literature, we also employed 337 features. As some of them are close in terms of definition and represented phenomenon, we grouped them into 14 families of features.

Length-based features (LEN) count the word length (i.e., number of letters in a token and its stem, and the number of syllables) and the number of words per sentence. In total, 4 length-based features. Graded resource features (GRD) contain normalized frequencies of word lemmas divided by level from EFLLex [START_REF] Dürlich | Efllex: A graded lexical resource for learners of english as a foreign language[END_REF]. We use a total of 6 features based on graded resources.

4 Unfamiliar MWE were not annotated.

Frequency features (FRQ) consider the frequency of words in a reference corpus. In this work, we consider the frequency of all words in a text, only content words (i.e., nouns, proper nouns, verbs, adjectives and adverbs in the text), only functional words, only common nouns, only verbs and only adjective. As the reference corpus, we explored the total normalized frequency (ignoring levels) in EFLLex [START_REF] Dürlich | Efllex: A graded lexical resource for learners of english as a foreign language[END_REF] and contextual diversity on SUBTLEX [START_REF] Brysbaert | Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english[END_REF]. In sum, 18 frequency-based features.

Features based on orthographic neighbor (NGH) measure orthographic or phonetic similarity between words. In this work, we use the mean orthographic and phonologic Levenstein distances [START_REF] Bartlett | On the syllabification of phonemes[END_REF] and the absolute and average number of neighbors and their frequency [START_REF] Brysbaert | Moving beyond kučera and francis: A critical evaluation of current word frequency norms and the introduction of a new and improved word frequency measure for american english[END_REF]. Also, the occurrence and cumulative frequency of neighbors with higher frequency than the words in the text are used. In total, 8 features. Lexical norms (NRM) features resort to the MRC database [START_REF] Coltheart | The mrc psycholinguistic database[END_REF] to annotate age of acquisition, concreteness, familiarity and imageability of each word. In addition, we also identify the percentage of out-of-vocabulary in each of the four features. Lexical sophistication (SOP) features identify the number of sophisticated tokens and types considering all words, content words, and verbs considering the surface form in [START_REF] Dale | A formula for predicting readability: Instructions[END_REF]. In sum, 6 features. Moreover, we use syntactic annotation automatically extracted from the Stanza parser [START_REF] Qi | Stanza: A python natural language processing toolkit for many human languages[END_REF]. 5Part-of-speech tags (POS) are counted using. 17 tags described in the Universal POS tags are considered. Morphology features (MOR) target the morphological components of the words. As they operate in a lower level of the POS, we also use the Stanza parser for annotating the 56 features. Dependency relations (DEP) employ the 37 functions proposed by Universal Dependencies6 . In addition, verb tense (TNS) features put together POS and morphology relations to identify the verb tenses as they are commonly taught. We use 19 verb tenses: simple tenses, perfect, continuous, emphatic and conditional tenses, and also the imperative, the tenses. All based on Stanza parser and identified through handcrafted rules. We also explore constituency parser [START_REF] Kitaev | Multilingual constituency parsing with self-attention and pre-training[END_REF] for extracting phrase (PRH) usage, differentiating 25 phrase types. In addition, we also count the number of phrases. Language development (DEV) features include the Yngve index constituency parser [START_REF] Yngve | A model and an hypothesis for language structure[END_REF], number of words before and after the main verb, and the average phrase and sentence depth in the text. In total, 5 features related to language development. Lexical diversity features (DVR) explore variations of type-token-ratio (TTR) that have been widely used for measuring language proficiency. In this work explored the Moving Average TTR (MATTR; [START_REF] Covington | Cutting the gordian knot: The moving-average typetoken ratio (mattr)[END_REF]) with a window size of 100 words; Corrected TTR (CTTR; [START_REF] Carroll | Language and thought[END_REF]); Root TTR (RTTR; [START_REF] Guiraud | Problèmes et méthodes de la statistique linguistique[END_REF]); Bilogarithmic TTR (LogTTR; [START_REF] Herdan | Type-token mathematics[END_REF][START_REF] Herdan | The advanced theory of language as choice and chance[END_REF]); SquaredTTR [START_REF] Chaudron | Discourse markedness and structural markedness: The acquisition of English noun phrases[END_REF]; and UberIndex [START_REF] Arnaud | Vocabulary and applied linguistics[END_REF]. For those, we distinguish between the ratios of lemmas and surface forms as well as all words, content words (i.e. nouns, proper nouns, verbs, adjectives, and adverbs in the text), adjective, adverb, adjective and adverb, nouns and pronouns, and verb. In addition, we specialized the verb features normalizing by the content words and verbs. In sum, we use 112 DVR features.

Coherence features (COH) use language models to compare the input text with the language's reference usage. We used ukWaC [START_REF] Baroni | The WaCky wide web: A collection of very large linguistically processed webcrawled corpora[END_REF], a 2 billion word corpus that covers a great range of themes, to train our models. Our first model, LSA, has 250 dimensions with stopwords and punctuations being removed and the 100,000 most frequent tokens/lemmas were kept. For the second model, PPMI, the dimension and window size were set to 500 and 2 without removing stopwords [START_REF] Bullinaria | Extracting semantic representations from word co-occurrence statistics: A computational study[END_REF]. For these models, we calculate the cosine similarity of all pairs of adjacent sentences and the cosine similarity of each sentence with all the other sentences are computed (for the PPMI case, all the word vectors of a sentence are averaged). In total that makes 8 features. We also estimate the probability and perplexity of each sentence by training two 4-gram models on ukWaC (uncased tokens and lemmas) in the third model. This was created using KenLM [START_REF] Heafield | Scalable modified Kneser-Ney language model estimation[END_REF], a language modeling toolkit based on modified Kneser-Ney smoothing [START_REF] Kneser | Improved backingoff for m-gram language modeling[END_REF]. The n-gram model added 4 features. Finally, the fourth model, 3 features, is a simple n-gram frequency varying n between 2 and 4 on uncased and lemmatized ukWaC using SRILM (Stolcke, 2002), a language modeling toolkit.

Results

Following our goal, we analyze the MWE usage on the annotated corpus. We start by describing the MWE usage and concreteness in the corpus. This analysis allowed to draw a general profile of MWE in learners' essays (Section 4.1). Then, we focus on the applicability of MWE features for automatic essay scoring by investigating their correlation with the CEFR level (Section 4.2) and their applicability as features for a machine learning model (Section 4.3). We also compared the proposed features with the classic ones in the last two studies to evaluate their capacity to discriminate the levels.

Profiling MWE usage

The analysis of MWE usage by learners showed that 5.78% of the essays do not contain MWEs. In A1, A2 and B1 levels, there is an increase in the MWE usage, but they are similarly used at B2 and C.

The use of MWEs along the levels and the 128 prompts were also analysed. Prompts are specific per level, varying between 23 and 31 prompts. Only in the higher levels there are few occurrences of the same prompt shared in different levels (3% of the prompts). The quantity of essays is not the same for each prompt. A normalization considering the average of the prompts that had fewer documents was made to get a reliable result. Considering 2 standard deviations to the prompt to be an outlier, we observe two outlier prompts at A1, none at A2, one at B1 and B2, and three at C. For all levels, it corresponds to less than 10%.

The MWE' concreteness have a correlation of -0.11 with their usage per level. We observe that beginners are more familiar with more concrete MWEs and get used to more abstracted expressions as they go through the levels (concreteness average scores for A1-C are 3.1603, 3.0151, 2.7119, 2.5263 and 2.6087, respectively). Moreover, C level contains MWE present in the list but without annotated scores. It suggests that these MWEs are truly specific and indicative of a learner's high proficiency.

The skewness and kurtosis of the concreteness were also analysed per level (kurtosis is summarized in Figure 1). The concreteness distribution for A1 is flattened. As the level increases, the distribution approaches a normal distribution. The skewness, on the other hand, has low values for A1 and they increase across the levels, going from 0.0514 (A1) until 0.3966 (C) 7 . This suggests that the data has a positive deviation as the level increase, it means that the weight happens in the direction of the low scores of concreteness.

Correlation

To study the relationship between the MWE and CEFR levels, we compared the Spearman correlation between MWE features and the level as well as all features described in Section 3. Those are summarized in Table 3 which shows the score most correlated with the level for each family of features presenting their rank and correlation considering the entire corpus and distinguishing by nationality. The table also shows the average rank and correlation of the features by family, considering the entire corpus, and by nationality; all correlations with p-value<0.05. The top 40 features are predominantly related to lexical diversity. This result goes in the same direction as [START_REF] Jung | Predicting second language writing proficiency in learner texts using computational tools[END_REF]. We also observed that the top 6 features have different ranks when nationality is considered. However, they are always in the top 6. Moreover, the top 1-3 are based on ratios considering all tokens, while and the top 4-6 are based on ratios of content words only. We also observed a band of features that alternate values between the top 7 and 16. Contrary to the pattern observed in the top 16 features, the features between 17 and 25 have almost constant rank across the nationalities. Below rank 25, we observed a considerable fluctuation in rank. This fluctuation can be seen in the standard deviation of the rank columns in Table 3.

We also analyzed the relation between the feature with the highest correlation and the average correlation for each family. As shown in Table 3, a higher correlated feature does not indicate that most of the features in their family are also highly correlated. For example, the SquaredTTR based on all tokens presented a correlation of 0.81 with the CEFR level, but in average the DVR features presented 0.42 as correlation. This indicates that only a few features are broadly meaningful for level identification. However, it does not mean that the other features may be ignored. Targeting on MWE, their average concreteness is more correlated with the level than their usage (0.36 v. 0.21).

In other words, the use of less concrete MWE is a better indication of a CEFR level than a higher number of MWE, although both features showed weak relationships with the level. Furthermore, we explored 18 statistics descriptors 8 to better describe the MWE usage and concreteness. The correlations between those and the CEFR levels are shown in Table 2 (absolute values lower than 0.26 and those with p-value>0.05 are not shown in Table 2). We also highlight that some separation statistical measures, such as minimum (Min) and first quartile (Q1), are better descriptors than the average one for MWEs concreteness. Moreover, we identified that the correlation between the levels and the number of words corrected by the MWE occurrence is 0.82. 8 Average, sum, minimum, maximum, length and mode as measures of range and tendency. Median, variance, standard deviation, relative standard deviation, dolch, first and third quartile, eighth and ninth percentiles and interquartile range as measures of dispersion and separation. Skewness and kurtosis for description of the curve.

MWE

Classification

For exploring the relationship between the scores, we resort to feature-based machine learning. We explored the relation inter-families by combining the different scores that compose each of the 14 families (see Section 3) as features for predicting the CEFR level of an essay. Since some families are strongly related, we also explore the combination of them as features. In other words, we combined parser (MOR, POS, DEP, PRH and TNS), and lexical norms-based (NRM and MWE conc ) features (NRM all ). In addition, for the sake of comparison, we considered the occurrence of MWE and their concreteness as individual features. Finally, we combined all features (all) to identify the full prediction capacity of a model trained using all features described in this work. For comparing the impact of the MWE features in this set of all features, we removed the MWE features from the training. Aiming to avoid bias of a specific model, we explored two machine learning models, one based on classification (Random Forest; RF) and the other on regression (Simple Logistic; SL). All these models were trained using stratified cross-validation 10 folds. The average9 and standard deviation results of these models using the different feature sets are shown in Table 4. For the SL, the results by feature family indicate that the best results are obtained when using the DVR features, in line with the results of the correlation study (Section 4.2). However, the MOR features seem to be more informative when using the FR. This difference is probably related to the search strategy employed by the RF, which can better divide the search space. The combination of different families had a remarkable positive effect on the parser-based features (increasing the F1 from 77% to 83% in the RF and the RMSE from 1.065 to 0.857 in the LR). The combination of lexical norms with the MWE concreteness showed a small improvement (p-value<0.05). Despite all these improvements by combining new features, the use of only DVR features achieved the best result in the regression. This again points to the need for an intricate search space strategy. Lastly, we did not observe a significant difference between the use of all features and all except the MWE-related features.

Conclusions

In this work, we study MWE features to predict essay scores. Concreteness of the MWEs found per level leads us to believe that MWE concreteness has an impact to predict essay scores. However, the correlation and machine learning results do not confirm it. MWE has been studied in other languages, such as French Franc ¸ois and Watrin (2011) who observed similar results. In future work, the approach proposed by Wilkens et al. (2022)

Figure 2 :

 2 Figure 2: "Phonetic" LARA version of Charivarius's 1920 pronunciation poem The Chaos (screenshot of first page).Clicking on a letter group, here "ear" in "tear" at the end of the penultimate line, highlights the group, plays the sound, and shows a phonetic concordance on the right; clicking on a speaker icon plays audio for whole of the preceding word. A link to the text is posted on the LARA examples page.

  A histogram showing the distribution of the subjectivity values in the CWI2018 data. Whilst most data is of lowsubjectivity. There are clear examples on the right of the graph where annotators disagreed. Subjectivity vs. complexity. The bell curve arises due to the binary annotation scheme as described in Section 3. The result of varying the subjectivity threshold according to α. Around 40% of the instances are considered subjective at low values of α.

Figure 1 :

 1 Figure 1: Analysis of the subjectivity values in the CWI2018 dataset annotations.

  Figure2ademonstrates the distribution of subjectivity values in our dataset, with a mean around 0.6 and subjectivity ranging from 0 to 1.5. (N.b., subjectivity is result of varying the subjectivity threshold according to α.

Figure 2 :

 2 Figure 2: Analysis of the subjectivity values in the LCP2018 dataset annotations.

  (a) Visualization of clusters using t-SNE (b) Visualization of clusters using UMAP

Figure 1 :

 1 Figure 1: Examples of news text in standard language and simplified language.

Figure 2 :

 2 Figure 2: Screenshot of the web reader interface.

Figure 1 :

 1 Figure 1: FRIENDLYREADER, left and TECST, right

Figure 2 :

 2 Figure 2: HIBOU interface for choosing the texts.

Figure 3 :

 3 Figure 3: HIBOU interface for reading sentence after sentence, and error message.

Figure 4 :

 4 Figure 4: HIBOU interface for gap-fill summary.The comprehension activity has been imagined as a game enabling to get awards. Correct answers to the quiz and the gap-fill summary give the child gold or silver stars which are displayed on the text selection page. Golden stars are proposed for correct answers; silver stars for inaccurate answers (one or more elements are present in the text). If the answer is totally incorrect, no reward is offered. For the gap-fill summary activity, each word or group of words correctly positioned provides a gold star (see Figure5).

Figure 5 :

 5 Figure 5: HIBOU interface for individual rewards after reading a text.

Figure 6 :

 6 Figure 6: HIBOU interface for overall rewards.

Figure 1 :

 1 Figure 1: Visual representation of participants' answers to the comprehension questions.

Figure 2 :

 2 Figure 2: Example of annotation.

Figure 1 :

 1 Figure 1: Concreteness kurtosis per level

Table 1 :

 1 Terms by subjectivity for CWI2018We can see from Table1that both simple (back, town) and complex (censorship, instilled) terms were agreed upon by all annotators. The most controversial words are typically longer words that may require some subjective or domain knowledge to fully understand.

Table 3 :

 3 Correlation analysis between common lexical features and complexity/subjectivity in the LCP2021 dataset

	Method RF SVM AB	Subjective R 0.658 0.681 0.669 0.656 0.632 0.644 Non-Subjective P F1 P R F1 0.623 0.836 0.714 0.736 0.476 0.579 0.660 0.775 0.713 0.715 0.586 0.645
	Table 4: Results of predicting which instances in the dataset will be subjective
	RH3.1: We were able to predict the subjective label of the words in the LCP2021 dataset with an F1 score of 0.71. This demonstrates that subjectivity is a pre-dictable phenomenon and we hope that in light of this finding future researchers will consider complexity in light of subjectivity.

  . Each example is shown between 40 and 49 times in different combinations with other examples. Care was taken to arrange the examples in such a manner that the four examples illustrating the same word but different senses end up as one task each.

Table 2 :

 2 Table2shows the average time taken for a single task, per annotator, in seconds. As the interface counts the number of seconds between the completion of two tasks, as long as annotators leave the interface open before continuing, time is being counted. By excluding outlier values -outliers being values of more than 90 seconds (an arbitrarily chosen threshold corresponding to an implausible time for a single task completion)we obtain the average time per task as shown on the right side in Table2(Avg time excl. outl.), which is much closer to the originally predicted time per task. Average time per task per annotator

	Annotator Avg time (s) Avg time excl. outl.
	1 2 3 4 5 6 7	15 101 12 45 39 78 11	12 9 9 18 16 8 11

Table 3 :

 3 rank 28 in the native ranking versus rank 43 in the non-native ranking, and clignotant 'indicator/turn signal', rank 33 versus rank 49. As regards the influence of context on the perceived difficulty of a word sense, we can see that the different examples of the same word sense end up rather close together on the ranking, with a maximum span (i.e. the difference between the maximum and minimum rank) of 19 ranks and an average span of about 14, as illustrated in Table3. Furthermore, one can see that the words follow the progression of CEFR levels, except for guérison 'recovery' which ends up closer to B1 level yet shows a very narrow clustering. Ranks of examples of the same word sense

	Word	Level	Ranks Span
	connaître 'to know'	A1	13, 17, 23, 27	14
	fixer 'to fix'	A2	31, 39, 40, 46	15
	joindre 'to join'	B1	48, 52, 56, 67	19
	prétention 'pretention'	B2	92, 96, 100, 105	13
	guérison 'recovery'	C1	47, 49, 50, 54	7
	attirail nalia' 'parapher-	C2	95, 99, 107, 110	15

Two other notable differences can be found between bricoleur 'tin-kerer',

Table 4 :

 4 Ranks of examples of different word senses

	Word	Level	Ranks Span
	vieux 'old'	A1	3, 4, 5, 6	3
	point tern' stitch pat-'point/dot/	A2	8, 25, 28, 74	66
	repasser again/redo' iron/pass 'to	B1	30, 58, 65, 79	49
	perte 'loss/ruin'	B2	32, 33, 53, 61	29
	pétiller 'to crackle' fizz/sparkle/	C1	73, 101, 104, 111	38
	fausser 'to fake' sify/forge/ fal-	C2	76, 83, 85, 97	21

Table 5 shows the rank positions of all anchor words for the global ranking, the native speaker ranking and the

Table 5 :

 5 Anchor words: comparison of ranking order positions for global, native and non-native rankings

	nique for dimensionality reduction and visualization of high-dimensional data, and Figure 2b shows a visual-ization of the clustering using Uniform Manifold Ap-proximation and Projection for Dimension Reduction (UMAP; McInnes et al. (2018)). UMAP is a fast and scalable dimensionality reduction algorithm that is said to be "better at preserving some aspects of global struc-ture of the data than most implementations of t-SNE" (McInnes et al., 2018).

Table 1 :

 1 Corpus statistics. and are added to new versions of the corpus. We plan to release a new version of SNIML every month.

	Dataset Statistics Table 1 provides statistics of the corpus and its sub-corpora. The corpus consists of 13,447 articles that were published by the news providers listed below. The lengths of the articles vary greatly, within one provider as well as among the dif-ferent platforms.

Journal Essentiel and Infoeasy tend to publish longer articles, averaging above 600 tokens per article, while the average for the other providers generally lies below 300 tokens per article. Temporal Stratification It is planned that news articles continue to be automatically fetched from the web

Table 2 :

 2 List of providers of the news articles that constitute the corpus.

	The articles address current topics and are written in French.

3.2.4. Lätta Bladet

Lätta Bladet is a sister magazine of Selkosanomat that is also situated in Finland and offers articles in Swedish. It is published by Selkokeskus and LL-Center, which is part of the interest organization of Swedish-speaking people with intellectual disabilities in Finland FDUV. Parallel to Selkosanomat, a printed magazine as well as a free online newspaper is offered. The same topics are covered by Lätta Bladet as by Selkosanomat. The articles are written in Swedish.

  Aluísio, S. and Gasperin, C. (2010). Fostering digital inclusion and accessibility: The PorSimples project for simplification of Portuguese texts. In Proceedings of the NAACL HLT 2010 Young Investigators Workshop on Computational Approaches to Languages of the Americas, pages 46-53, Los Angeles,

		Appendix
	Provider	Example	English machine translation
	The Times in Plain English	With Omicron, you may get a scratchy in the Omicron symptoms." still too early to say there is any difference coronavirus outbreak. An expert said, "It is and they are also the symptoms of the first the same symptoms as the Delta variant, muscle pain in the lower back. These are throat, nasal congestion, a dry cough, and	-
	Selkosanomat Rokotuksia halutaan vauhdittaa, koska ko-laan uusia tiukkoja rajoituksia. kuin Suomessa. Näissä maissa suunnitel-ja Norjassa omikron on levinnyt laajemmin yhä nopeammin. Britanniassa, Tanskassa ronaviruksen uusi muunnos omikron leviää	-
	Journal Essentiel	Je comprends les personnes qui disent: "Nous ne savons pas ce que ces vaccins pourraient nous causer à l'avenir." J'ai en-vie de leur répondre: "Il y a sans doute certains effets à long terme qui sont in-pandémie." notre meilleur moyen pour sortir de cette connus mais aujourd'hui, le vaccin est	I understand people who say, "We don't know what these vaccines might do to us in the future." I want to answer them: "There are probably some long-term effects that are unknown but today, the vaccine is our best way out of this pandemic."
	Informazione Facile	• I bambini fino agli 11 anni ricever-anno un terzo della dose prevista so-pra i 12 anni. • La sperimentazione del vaccino è bambini: 2.300. stata fatta su un piccolo numero di	• Children up to the age of 11 will re-ceive one third of the expected dose over the age of 12. • The vaccine trial was done on a small number of children: 2,300,
	Lätta Bladet	Regeringen vill få fart på vaccineringen. I börjar vaccinera barn över 5 år mot corona. Regeringen har också bestämt att man nu 000 vuxna som inte har fått coronavaccin. Finland finns det fortfarande ungefär 800	-
	Infoeasy	Genesen ist ein anderes Wort für: wieder gesund. Wir brauchen darum jetzt an vielen Orten ein Covid-Zertifikat. Nur mit dem Zertifikat dürfen wir hinein. Und dieses Zertifikat bekommen nur Per-sonen, • die geimpft sind. • die genesen sind. • die einen Corona-Test gemacht haben. Und der Test muss negativ sein. 5

Table 3 :

 3 Text examples for each news provider. Machine translations are provided if the license permits derivative work.

Table 1 :

 1 Target audiences and their simplification needs

	Dyslexia	Lexical Simpification Long words Less frequent words Homophones Words that are ortho-graphically similar New words Non-words	Syntactic Simplifica-tion	Discourse-level Sim-plification
	Aphasia Intellectual Disability (ID) Limited vocabulary Information density Noun compounds Deaf and Hard-of-Hearing Limited vocabulary	Long sentences Long sequences of ad-jectives Passive voice Object relative clauses Comparison of word meaning Complex sentences Morphology Syntax
	Austism Spectrum Disorder Second Language Learn-ers Children	Words related to emo-tions Limited vocabulary New words Limited vocabulary		Figurative language Texts that require little social knowledge Tight text structure
	Revision (ICD

Table 1 summarizes the target audiences and their main simplification needs. Dyslexia. In the International Statistical Classification of Diseases and Related Health Problems -Eleventh

Table 1 :

 1 Descriptive statistics for annotated texts.

	Texts	B1 A B2 A B2 B
	Annotators Number of words	12 432	5 564	9 448
	% Easy % Transparent % Vague % Opaque	97.22 98.23 97.77 1.85 1.06 1.34 0.93 0.35 0.22 0 0.35 0.67
	Average task time (s) 521.9 557.4 Standard deviation (s) 180 156	508 234

Table 3 :

 3 Spearman's correlation tests (ρ).

	The re-Two variables were tested for correlation with the num-sults are presented in table 3.

  Table that the document size in the ljl corpus goes from 150 to 1,500 words approximately, and to tens of thousands of words in the bb and jll corpora. The vocabulary size for ljl corpus is 23,123 words, 36,011 for jll and 38,503 for bb. The latter two are somewhat comparable, however the ljl corpus is lacking diversity in its words.

Table 3 :

 3 Pearson correlation score except for the Traditional formulas and pseudo-perplexity scores for all the readability class (R class ) of each corpus. The Pearson coefficient shows the correlation between the scores and the classes.

	R class	GFI	ARI littérature de jeunesse libre (ljl) FRE FKGL SMOG	REL	PPPL
	lv1 lv2 lv3 lv4 p-score	44.61 14.12 66.88 19.8 67.61 78.6 91.21 25.66 59.04 105.52 27.87 54.92 0.48 0.49 -0.40	4.28 6.32 8.06 8.81 0.45	15.97 94.38 18.65 84.55 21.11 76.81 22.15 73.81 0.49 -0.40	54.59 57.79 63.80 62.87 0.04
			Bibebook (bb)
	easy interm. 128.93 36.71 56.04 122.6 35.56 57.04 advan. 122.6 36.26 58.03 p-score -0.003 0.012 0.021	9.42 9.67 9.38 -0.006	23.85 74.49 152.33 24.06 73.56 414.00 23.95 75.30 161.62 0.005 0.019 -0.007
			Je Lis Libre (jll)
	6e 5e 4e 3e p-score	119.82 46.38 77.38 132.39 40.75 60.49 102.42 36.12 81.63 104.06 34.36 79.84 -0.11 -0.19 0.12	7.96 9.53 6.27 6.24 -0.17	23.74 91.45 177.68 24.38 77.17 114.06 21.69 95.73 172.71 21.12 94.32 169.45 -0.19 0.13 0.02

FRE measure since lower scores indicate that a text is less readable (negative p-score).

Table 4 :

 4 Results on 'littérature de jeunesse libre' (ljl), 'Bibebook' (bb) and 'Je Lis Libre' (jll) corpora for the class prediction task. Best Accuracy, F1-score and Macro average F1-score values are in bold.

	Unlike word2vec-like models, BERT provides contex-tual embeddings to represent the meaning of words in context. BERT benefits from a bidirectional architec-ture based on Transformers and their attention mecha-nism. BERT can easily be used for classification task by adding a supplement dense layer. Training BERT for a classification task results in fine-tuning a pre-trained BERT model with an additional layer for the

Table 1 :

 1 Table 3 presents the corpus size employed in this work, identifying the number of essays considered in each level for each nationality. Number of used texts for each nationality and its percentage in corpus used in this study.

	Nationality Brazil Germany Italy Russia France Mexico China Saudi Arabia Japan Taiwan	Usage per level Corpora (%) 2469 22.99 2469 22.99 1238 11.53 1195 11.13 818 7.62 762 7.09 555 5.17 468 4.36 420 3.91 347 3.23

Table 2 :

 2 Correlation of MWE features aggregators

		Kurt	Q3	Median	Q1	Min
	CONC 0.40 -0.29	-0.35	-0.37 -0.50
	CNT	-	-0.02	-	-	-

Table 3 :

 3 can be included in the feature's Correlation of different features and families of features considering the entire corpus and the learners' nationalities

	Family DVR	best score STTR (all surface tks)	general family rank best 1 138.9 (120.8) 0.81 0.42 (0.25) corr best family	by nationality rank best family 2.0 (0.8) 137.9 (117.4)
	DEV	depth	25	95.1 (77.5) 0.70 0.48 (0.17)	25.0 (0.0)	97.3 (78.7)
	DEP	mark	26 194.5 (126.5) 0.62 0.29 (0.20)	29.7 (4.7) 198.7 (125.1)
	POS	punct	35	204.3 (90.0) 0.59 0.27 (0.14)	35.1 (7.0)	214.6 (92.9)
	LEN	word per sent.	36	66.8 (28.3) 0.58 0.50 (0.07)	39.5 (12.3)	66.9 (25.6)
	NRM	AOA	42	105.6 (66.9) 0.58 0.43 (0.13)	41.2 (5.9)	107.7 (68.1)
	FRQ	content words subtlex	44 198.0 (107.8) 0.57 0.28 (0.18)	42.1 (5.1)	199 (107.5)
	PRH	SBAR	52 254.1 (103.8) 0.54 0.20 (0.16)	52.5 (6.9) 252.0 (100.8)
	TNS	use past	63	266.0 (85.0) 0.51 0.18 (0.12)	64.1 (3.6)	267.7 (84.4)
	MOR	finite verb	69	204.4 (94.4) 0.47 0.26 (0.14)	77.5 (14.8)	215.3 (97.1)
	NGH	phonologic dist	71 254.3 (118.1) 0.47 0.20 (0.17)	71.5 (8.7) 247.8 (113.2)
	SOP	verbs	75	163.8 (88.8) 0.46 0.32 (0.14)	78.7 (12.2)	166.6 (93.5)
	MWE COH	MWE conc PPMI (lemma)	142 183	-291.5 (68.3) 0.29 0.14 (0.09) 188.9 (25.4) 0.36 -136.7 (18.9)	-288.7 (59.4)
	GRD	C1		213	235.6 (28.5) 0.24 0.21 (0.04) 212.2 (14.4)	237.6 (28.8)
	MWE	MWE cnt	233	-	0.21	-	239.9 (18.4)	-
	Feature set LEN	RandForest ACC F1 0.553 0.553 0.897 SLogistic MAE RMSE 1.364
	FRQ		0.682 0.682 0.739	1.200
	GRD		0.490 0.490 1.014	1.487
	NGH		0.561 0.560 1.053	1.520
	NRM		0.624 0.624 0.744	1.158
	SOP		0.498 0.498 0.869	1.294
	DVR		0.745 0.745 0.410	0.789
	DEP		0.736 0.736 0.630	1.065
	PRH		0.645 0.645 0.941	1.406
	DEV		0.726 0.726 0.694	1.075
	POS		0.745 0.744 0.772	1.235
	MOR		0.775 0.775 0.682	1.126
	TNS		0.565 0.559 0.731	1.161
	COH		0.519 0.519 1.170	1.628
	MWE		0.428 0.425 1.455	1.916
	MWE cnt MWE conc	0.454 0.447 1.660 0.418 0.413 1.499	2.121 1.946
	Parser		0.835 0.835 0.425	0.857
	NRM all		0.640 0.640 0.734	1.153
	All		0.843 0.843 0.535	0.697
	All-MWE 0.844 0.844 0.534	0.699

Table 4 :

 4 Results of the machine learning models using different feature sets creation since we observed different behavior per level that are identified by statistical descriptors other than average. Therefore, it might lead to a better understanding of the learner's usage of MWE and its applicability for essay scoring.

https://www.lingq.com/

https://sourceforge.net/projects/lwt/

https://scaife.perseus.org/

http://multidict.net/clilstore/

https://www.issco.unige.ch/en/research/ projects/callector/LARADoc/build/html/ index.html

https://www.issco.unige.ch/en/research/ projects/callector/LARADoc/build/html/ phonetic_texts.html

http://ipa-reader.xyz/; "Celine" voice.

https://50words.online/

https://www.goodreads.com/topic/show/ 21221144-v-lusp-reading-group-verses-1-6; https://www.goodreads.com/topic/ show/22046862-h-vam-l-reading-group; https://www.goodreads.com/topic/show/ 22089678-lokasenna-reading-group

2 100 students; https://ctpl.um.ac.ir/index. php?lang=en

The relabelling is done in arbitrary manner to enable us to investigate the same effect in the

2, 3 and 4 point setting. It is not intended as a robust means of reducing the number of labels in an annotation setting.

https://github.com/MMU-TDMLab/LCP_ Subjectivity

https://cental.uclouvain.be/cefrlex

https://dictionary.cambridge.org/ dictionary/english/

https://cental.uclouvain.be/cefrlex/ flelex/

https://fr.wiktionary.org/

While this methodology of level assignment differs from the methodology by(Pintard and Franc ¸ois, 2020), it allows for a more fine-grained assessment of "centrality", and given that we work on a restricted subset of items where e -r < 0.1, the items under scrutiny are of comparable quality with regards to automatic level assignment.

The interface shows a translated mockup. Note that the English Wiktionary indicates years (plural) as a lemma for the second example: https://en.wiktionary.org/ wiki/years.

We used the commercial machine translation system of Microsoft Azure Cognitive Services (version 3.0). A manual investigation of the MT quality revealed typical machine translation errors such as wrong pronomina or incorrect gender. While we did not spot translation problems specific to simplified language, future work could investigate this question in more depth.

https://selkokeskus.fi/selkokieli/

https://corpusalector.huma-num.fr/

https://www.storyplayr.com/blog/le-conte-africain

https://kidiscience.cafe-sciences.org/articles/comment-leshuitres-fabriquent-elles-des-perles/

6 https://www.icem-vente-en-ligne.org/btj

https://www.wapiti-magazine.com/magazine

https://www.babelio.com/livres/Ciravegna-Chichois-de-la-ruedes-Mauvestis/459091

https://cental.uclouvain.be/resyf/

https://alectorsite.files.wordpress.com/2020/11/guidelines-linguistiques_alector_final.pdf

https://github.com/cdimascio/ py-readability-metrics

https://bitbucket.org/ nishkalavallabhi/complexity-features

https://github.com/vwoloszyn/ pylinguistics

https://github.com/kinimod23/GRANT

https://cental.uclouvain.be/amesure

https://cental.uclouvain.be/fabra

https://cental.uclouvain.be/amesure

https://texttokids.irisa.fr/project

Loi d'orientation sur l'éducation de 1989, modifiée en 2014 par un décret de 2013 https: //www.education.gouv.fr/bo/13/Hebdo32/ MENE1318869D.htm?cid_bo=73449

https://www.coe.int/en/web/ common-european-framework-reference-languages

litterature-jeunesse-libre.fr/bbs/

storyweaver.org.in/reading_levels

www.bibebook.com/visual-search?f%5B0% 5D=field_genre%3A1267

www.crdp-strasbourg.fr/je_lis_libre

https://github.com/jalan/pdftotext

https://spacy.io/models/fr

Sourced by https://huggingface.co/asi

https://scikit-learn.org

https://unesdoc.unesco.org/ark: /48223/pf0000259784

https://github.com/nicolashernandez/ READI-LREC22

https://fr.wikimini.org (written by children) and https://fr.vikidia.org

P-frames are a type of semi-fixed word sequence in which fixed words surround an open slot[START_REF] Stubbs | An example of frequent english phraseology: distributions, structures and functions[END_REF].

For example, break a bone (Transparent); break the silence (Semi-transparent); break the ice (Non-transparent).

We do not assess parser instabilities stability caused by learner errors, but[START_REF] Berzak | Universal dependencies for learner english[END_REF] addressed the subject.

https://universaldependencies.org/

A2 = 0.1572, B1 = 0.2607, B2 = 0.3519

The standard deviation RMSE is below 0.02 and for the other scores below 0.01.

Acknowledgments

We would like to thank the reviewers for their appreciation of the paper and their numerous detailed comments and suggestions. We also would like to thank our co-raters Nils Bouckaert, Alba Garcia Prades, Angela Kasparian, Hubert Naets and Nami Yamaguchi. David Alfter is supported by the Fonds de la Recherche Scientifique -FNRS under the grant MIS/PGY F.4518.21. Rémi Cardon is supported by the FSR Incoming Postdoc Fellowship program of the FSR -Université Catholique de Louvain.

Acknowledgements

This work has been supported by the pilot center on teacher training and research for education (AMPIRIC), funded by the Future Investment Program (PIA3), and the center of excellence on Language, Communication and Brain (ILCB), funded by the French National Agency for Research (ANR, ANR-16-CONV-0002) and the Excellence Initiative of Aix-Marseille University A*MIDEX (ANR-11-IDEX-0001-02). The research was directly funded through grants from the ANR, ALECTOR (ANR-16-CE28-0005) and MORPHEME (ANR-15-FRAL-0003-01) and the ERC (Advanced grant 742141).

The authors thank all the participants, the teachers who accepted to use the eBooks in their classrooms, and the school authorities who supported this study, especially Olivier Millangue, Nathalie Greppo-Chaignon, and Géraldine Gaudino. We also thank the Communauté d'Agglomération Sud Sainte Baume for providing access to the iPads used in the schools and for their technical support.

Acknowledgements

We are grateful to the the anonymous reviewers for their valuable comments which will help us to pursue this work in a more "high quality" direction. This work was partially supported by the French Agence Nationale de la Recherche, within its Programme d'Investissements d'Avenir, with grant ANR-16-IDEX-0007.

Acknowledgements

This research has been partially funded by a research convention with France Éducation International. Computational resources have been provided by the Consortium des Équipements de Calcul Intensif (C ÉCI), funded by the Fonds de la Recherche Scientifique de Belgique (F.R.S.-FNRS) under Grant No. 2.5020.11 and by the Walloon Region.

Author Index