Victoria Shure
email: victoria.shure@hu-berlin.de

Christin Laschke
email: laschke@leibniz-ipn.de

Bettina Rösken-Winter
email: bettina.roesken-winter@hu-berlin.de

Birte Pöhler

Utilizing a scriptwriting task as a tool to examine facilitator practices in response to teacher orientations for at-risk students

Keywords: Scriptwriting tasks, facilitator practices, teacher orientations, professional development

Teacher orientations influence instructional prioritizations and how teachers attend to students, particularly those at risk of being left behind. In PD programs, facilitators' practices to recognize and respond to such teacher orientations for supporting at-risk students are thus an important aspect of content-related facilitator expertise. Extending the use of scriptwriting tasks to the PD facilitator level, we present two contrasting cases of how facilitators employ practices to respond to teachers' orientations in a PD simulation. One facilitator avoids direct opposition with conflicting teacher orientations, while the second facilitator challenges the teachers' orientations that do not contribute to supporting at-risk students' learning. By discussing the contrasting practices in the facilitators' written scripts, we demonstrate how the scriptwriting task can be used to investigate content-related facilitator expertise in terms of practices in response to teachers' orientations.

Introduction

Mathematics represents a cumulative content area with basic concepts laying the foundation for later understanding. It is crucial that students can be supported in gaining knowledge from previous years or units, so they can develop deeper conceptual understanding as they move forward in later grades. The orientations that teachers have in response to less-privileged students who are at risk of being left behind are important to understand and necessary for PD facilitators to be able to identify and respond to. Such facilitator practices are part of their content-related expertise as conceptualized by [START_REF] Prediger | Conceptualizing content-related PD facilitator expertise[END_REF]. In this study, we draw on the [START_REF] Prediger | Conceptualizing content-related PD facilitator expertise[END_REF] framework and utilize a scriptwriting task to examine facilitators' practices for responding to teachers' orientations for supporting at-risk students' learning processes. We examine these practices in the context of a PD for teachers to monitor and enhance at-risk students' conceptual understanding of basic concepts. In the theory section, we first address teacher orientations as part of their teaching expertise. Particularly, we elaborate on four sets of orientations pertinent to the context of fostering at-risks students' mathematics performance. Second, we describe the framework of facilitators' content-related expertise, with a focus on the facets relevant for our study. Third, we expound upon the tool of a scriptwriting task, the instrument we applied. Thereafter, we present and discuss the data we gained in the context of the teacher PD project Mastering Math, which is built around supporting at-risk students' mathematics learning. Especially, we report how two facilitators were immersed in a PD simulation by the scriptwriting task and what practices they applied to deal with different teacher orientations.

Theoretical background

Teacher orientations as a part of teacher expertise

Various studies showed how teachers' orientations, respectively beliefs, regarding the nature and meaning of mathematics are manifested in their teaching practices. [START_REF] Schoenfeld | How we think. A theory of goal-oriented decision making and its educational approaches[END_REF] examined the components that impact how a teacher acts in the classroom and developed the theory of goal-oriented decision making, which considers resources, goals, and orientations as essential components. Such orientations and accompanying practices can have consequences for supporting students, especially those in need of additional assistance, if teachers prioritize, for example, procedural fluency over conceptual understanding. In expanding upon this conceptualization of orientations, Prediger (2019) created a framework of mathematics teacher expertise that includes orientations as one of five facets: "Orientations refer to content-related and more general beliefs that implicitly or explicitly guide the teachers' perceptions and prioritisations of jobs (e.g. beliefs about the content or students' learning processes)" (p. 370). Findings from empirical studies revealed four sets of contrasting orientations. First, teachers often focus on the development of procedural knowledge as opposed to conceptual knowledge [START_REF] Wilhelm | Examining relations between teachers' explanations of sources of students' difficulty in mathematics and students' opportunities to learn[END_REF]. Second, in terms of diagnosing individual student challenges, teachers often follow the school syllabus regardless if students fall behind, and, third, maintain shortterm mastery in contrast to emphasizing long-term goal mastery [START_REF] Prediger | Design research with a focus on content-specific professionalization processes: The case of noticing students' potentials[END_REF]. Fourth, teachers concentrate on individual student issues [START_REF] Krähenmann | Inclusive mathematics instruction: A conceptual framework and selected research results of a video study[END_REF], instead of facilitating rich discourse with all students. These four sets of orientations are addressed in the PD course Mastering Math that constitutes the context of our study.

Facilitator practices as part of their content-related expertise

In terms of what PD facilitators need to know and do to be able to provide successful PD programs for mathematics teachers, researchers have increasingly examined what constitutes necessary PD facilitator knowledge and practices [START_REF] Borko | Learning to lead: An approach to mathematics teacher leader development[END_REF][START_REF] Lesseig | Leader noticing of facilitation in videocases of mathematics professional development[END_REF]. In putting different knowledge domains in action, facilitators need to employ practices that are conducive to working productively with adult learners and construct environments in which teachers can collaborate about relevant topics and can feel safe and supported to share information [START_REF] Borko | Examining novice teacher leaders' facilitation of mathematics professional development[END_REF]. Accordingly, [START_REF] Prediger | Conceptualizing content-related PD facilitator expertise[END_REF] provided a framework for content-related facilitator expertise. The framework consists of jobs, which are conceptualized as typical and often complex situational demands of facilitating a specific PD, and practices as recurrent patterns of facilitators' utterances and actions for managing the jobs. These practices can be described and analyzed by revealing the underlying facilitator orientations and situative goals on which the facilitator implicitly or explicitly draws:

Orientations: Generic or content-related beliefs and pedagogical attitudes (e.g., about teachers' thinking or about the PD content) that implicitly or explicitly guide the facilitators' perceptions and prioritization of jobs (e.g., participant orientation).

Situative goals:

The goals that the facilitators pursue in a respective situation can directly refer to PD content learning goals (in brief, PD learning goals), can address process qualities (e.g., cognitive activation, briefly, process goals), or can be of an atmospheric nature (briefly atmospheric goals). (Prediger et al., 2021, p. 8) Thus, goals and orientations determine how facilitators act in a specific situation, and it is important to make them visible in facilitators' practices. For that purpose, scriptwriting tasks are a useful tool, which allow for creating a fictional context in which facilitators need to simulate their practice.

Scriptwriting tasks as approximation of PD practices

So far, scriptwriting tasks have been employed on the teacher level as a tool that functions as a bridge between planning and the actual course of action in the classroom by providing classroom situations for teachers to react to learners' utterances and provide possible explanations [START_REF] Zazkis | Imagining mathematics teaching via scripting tasks[END_REF]. By using scriptwriting tasks, one can draw on a situated approach to enable the approximation of the actual act of supporting students, but instead in a fictionalized situation. Scriptwriting tasks have been accordingly implemented as a means of investigating pre-service teachers' understanding of content and facilitating of learning [START_REF] Lim | Preservice teachers' learning paths of classroom discourse through scripting[END_REF]; and with in-service teachers [START_REF] Kontorovich | Turn vs. shape: teachers cope with incompatible perspectives on angle[END_REF] to assess how teachers deal with student alternate conceptions. Scriptwriting tasks, however, have not yet been utilized as a tool for assessing facilitators practices in response to teachers' orientations in a situated fictional PD context.

Aims and research questions

To examine PD facilitators' practices in response to teachers' orientations for supporting at-risk students, we employed a scriptwriting task to first provide facilitators with a fictional situation in which teachers in a PD are provided with a student dialogue of three students working on a task and then with a discussion between three teachers concerning how they would continue the conversation with the students. The facilitators were then given the task to first complete the dialogue with the fictional teachers and guide them to in discussing how to support the fictional students and, second, to provide rationale as to why they ended the dialogue at the chosen moment. We pursued the following research question: RQ: What practices do facilitators apply to respond to teachers' orientations for supporting at-risk students in completing the scriptwriting task, and what situative goals are behind these practices?

Methodology Instrument: scriptwriting task

The scriptwriting task was developed based on a classroom-level task concerning filling in the tens between 0 and 100 on a number line that contained the following prompt: How can you plot and label numbers on the number line? What can help you? The scriptwriting task included a fictional dialogue of three students with alternate conceptions of the task discussing their solutions followed by a fictional teacher dialogue of three teachers discussing how they could support the students in completing the task (Figure 1). After reading the fictional dialogues of the three students and the three teachers, the facilitators were asked to complete the dialogue with the three teachers (Karin, Sabine and Jana), thereby demonstrating how they will address the teachers' orientations for supporting at-risk students and help the teachers to foster student understanding of the basic concepts in the task. The four contrasting sets of teacher orientations for supporting at-risk students were embedded in the scriptwriting task.

Participants, data collection and data analysis

In total, a group of 14 PD facilitators from the federal state of Berlin were asked to participate in this study and were provided with the number line scriptwriting task described in the previous section. Six facilitators agreed to participate. All of the facilitators led PD courses in the program Mastering Math for primary school mathematics teachers, which focuses on means of supporting students at risk of being left behind. The Mastering Math PD program centers on the four sets of principles: (1) diagnostic vs. syllabus-bound orientation; (2) conceptual vs. procedural orientation; (3) long-term vs. short-term orientation; and (4) communicative vs. individualistic orientation. We selected these unique written scripts from two of the facilitators, in particular, as these two contrasting cases of facilitator practices illustrate how two facilitators react differently in the same fictive PD situation to enhance teachers' learning processes.

Results

The two written scripts demonstrate the facilitators' contrasting approaches in the extent to which they attempt to convince the teachers of the guiding principles of the PD program for supporting atrisk students. While the first facilitator poses questions and encourages the three teachers in the fictional PD situation to discuss the student misconceptions from the task without visibly advancing the guiding principles of the PD, the second facilitator more directly challenges the teachers who are not convinced by the guiding principles and engages them in a discussion surrounding long-term mastery achievement and diagnosing individual student capabilities in order to help move students forward. The two transcripts of the written scripts and the contrasting practices are presented in the following sections.

Case one: A hands-off approach to conveying PD guiding ideas

The facilitator in the first transcript encourages the teachers to provide their opinions concerning the challenges the students in the fictional dialogue encountered by posing questions as to the students' different approaches and different problems they faced. The facilitator responds to the teachers either by demonstrating agreement with the comments concerning Martin's problem that interferes with his understanding of the task and adding to the teachers' ideas with a short commentary that reflects the guiding ideas of the PD, or by remaining neutral by closing the discussion without further comment.

1

Facilitator:

The Well, with some students it's really hopeless. I wouldn't waste so much time, but rather concentrate on the better-performing students. 12 Jana: I believe that children have only really understood a subject when they can explain it in their own words. 13 Facilitator:

Thank you very much for your contributions and your assessment. At this point I would like to end the discussion for now and show you a short film.

The facilitator continues the dialogue in such a manner echoing the main elements of the dialogue beginning such that Karin expresses a distinct opinion concerning how to support at-risk students, reiterating that short-term, quick approaches are necessary, especially for such "hopeless" students when support can instead be focused on "better-performing students." It is evident that the facilitator identified the teacher orientations from the scriptwriting task dialogue and extended these to the continuation of the dialogue. The facilitator does not respond to Karin's comments, however, but makes it clear that such orientations can be expressed in the PD. In response to the follow-up question regarding why the facilitator ended the dialogue at that specific moment, the facilitator explained:

Facilitator 1: Karin is obviously still of the old school, according to the motto: Explain quickly and then continue with the material, don't waste any time. It is more important for her to support the better-performing students. The other two participants are more oriented towards the principles of MSK. In the first round of discussion, I would not try to persuade them. It is not about persuasion, but especially such teachers like Karin should be able to make their own discoveries and not be discouraged […] Thus, the facilitator reiterates her strategy of first assessing the orientations of the participating teachers, before engaging them or encouraging them to think about their viewpoints and their actions in the classroom. The facilitator creates an atmosphere in which the teachers are encouraged to express their opinions, at this stage, however, the facilitator does not push the teachers to reflect on their positions and reasoning behind such viewpoints or have the teachers focus more deeply on the mathematics behind the students' different representations from the student dialogue.

Case two: A direct approach to conveying PD guiding ideas

The second facilitator takes a different approach in the continuation of the dialogue with the three teachers in the written script and provides evaluative comments (see turn 24) of one of the teacher's comments concerning spending too much time on the process of remediation for students considered not capable of doing math. The second facilitator responds directly to Karin and even states that the idea that some children cannot do mathematics is "still a widespread belief," and then explains how to support children who struggle with more time and visualizations. While the facilitator does provide an evaluative response to Karin's statements, the facilitator asks Karin if she can try "to live with" the principle of long-term mastery for the time being, and moreover, the facilitator suggests that once Karin has completed a few remedial lessons, she will better understand the process of remediation. Thereby, the facilitator creates an atmosphere in which teachers with opposing orientations to the guiding principles of the PD are not completely discounted. In response to ending the continuation of the dialogue, the PD facilitator explains that such beliefs or orientations cannot be changed in one discussion as they are deeply rooted and will hopefully change with experience and observation:

Facilitator 2: Jana and Sabine are on the right track already and the essentials for conducting support discussions have been said. Karin still has doubts. But they can't be dispelled with a single conversation. These are deeply anchored beliefs about learning that -hopefully -will gradually disappear once Karin has gained her own experience and realizes that her colleagues are more convinced.

The facilitator thus acknowledges that changing a teacher's orientation to reflect the PD principles will not happen immediately and notes the role of the other members of the PD as relevant for influencing orientations. While the facilitator did respond to Karin's comments concerning some children who just cannot do mathematics, the facilitator does not press Karin further and instead provides an opportunity for another teacher in the PD to express support for long-term mastery.

Discussion and conclusion

The exemplifying cases show how facilitators could react differently in a PD in response to teachers' orientations. Facilitator 1 creates a situation when completing the scriptwriting task in which the facilitator pursues the content goal by directing the teachers' focus to students' thinking and challenges in solving the task, to then diagnosing students' learning processes in light of PD guiding principles. Facilitator 1 therefore purses the content goal by illustrating one of the PD guiding principles by analyzing the students' potential and challenges. The assertion of one of the teachers that it is important to avoid wasting time on students who are not able to understand is left uncommented. The facilitator thus seems to demonstrate the need to show esteem for participants and therefore, especially at the beginning of the PD, purses atmospheric goals instead of the content goal. In contrast, facilitator 2 discusses and confronts the orientation the teacher shows who insists on avoiding wasting time. Moreover, the facilitator consequently defends the PD guiding principles, thus focusing on the content goal, despite repeated objections of the teacher. Moreover, the facilitator seems to try to provoke the teacher to reinterpret or invalidate her own orientation. At the same time, the facilitator engages the teacher in the discussion process and shows esteem for her, presumably to maintain a good atmosphere. The practices of facilitator 1 match facilitator 2 practices to some respect, as both seem to recognize and take teachers' orientations into account. Furthermore, both facilitators refer to the PD guiding principles, with a focus on the content goal, albeit to different extents. In addition, both facilitators are anxious to maintain a good atmosphere.

As there have been calls for ways to expand research on expanding the professional growth of facilitators [START_REF] Borko | Examining novice teacher leaders' facilitation of mathematics professional development[END_REF][START_REF] Lesseig | Leader noticing of facilitation in videocases of mathematics professional development[END_REF], this research responds with a research tool that can be utilized to fill this gap. The scriptwriting task as a tool for professional development of PD facilitators of mathematics represents a situated form of practice. First, as a tool for facilitator educators to examine facilitator practices in responding to teachers' orientations, and second, as a means of discussing such responses to help facilitators continue on their path of professional growth.

Figure 1 :

 1 Figure 1: The scriptwriting task student dialogue (on the left) and teacher dialogue (on the right)

7

  Karin:But that can take a long time until Martin understands it. What if he doesn't realize that 50 has to be in the middle...then we'll still be sitting together the day after tomorrow. 8Facilitator: You're really making an important point here. There are children for whom MSK is ultimately not suitable either. And this would be a good time to find out what Martin's situation is like. If it's not clear to him that 50 is half of 100, and if he's not able to think in tens, then he needs a different kind of support. We would have to take a closer look at that in any case […] 21 Facilitator: …But why (looking at Karin) is that so important anyway, that we take so much time for this whole process? […] 23 Karin: But that takes an infinitely long time. And there are simply children where I don't know if that really helps. Math is also a bit like that -either you can do it or you can't. 24 Facilitator:Yes, that's still a widespread belief. And there really are kids for whom it's very difficult to achieve. But there are also many who just need a little more time and visualization and action. And time for visualization....so...they need to be able to link what they see to their thinking. And when they get that, it clicks pretty quickly. Then they replace their misconceptions with more appropriate ones. If you have experienced this yourself a few times, then you can comprehend it better. You just have to do a few remedial lessons, and then you realize that. But it also takes a bit of time to develop a feeling for which children they really make a difference and for which ones you still have to look for other forms of support. But when you get the children further along, it's totally satisfying for everyone. Can you live with that for now,

		Karin?
	25 Karin:	[nods]
	26 Jana:	And luckily we don't have any time pressure in remedial lessons. It just
		doesn't make sense to keep going if the kids haven't understood it yet.
	27 Facilitator:	Exactly, that's the great thing about remedial lessons! But unfortunately we
		do have a bit of time pressure...we're going to get on with our program now.
		Anyway, I'm really looking forward to your reports from your first remedial
		lessons.


Mika Hämäläinen

Khalid Alnajjar

Thierry Poibeau

Modern French Poetry Generation with RoBERTa and GPT-2

We present a novel neural model for modern poetry generation in French. The model consists of two pretrained neural models that are fine-tuned for the poem generation task. The encoder of the model is a RoBERTa based one while the decoder is based on GPT-2. This way the model can benefit from the superior natural language understanding performance of RoBERTa and the good natural language generation performance of GPT-2. Our evaluation shows that the model can create French poetry successfully. On a 5 point scale, the lowest score of 3.57 was given by human judges to typicality and emotionality of the output poetry while the best score of 3.79 was given to understandability.

Introduction

Poem generation is a challenging creative natural language generation task. As a form of art, it has undergone several changes in the history. Classical poetry incorporates typically meter and rhyme as their function was to help people recall poems, especially when poetic tradition was still mostly oral rather than written.

In the modern era, the role of the poetry has changed from an art form that has to follow a fixed structure that defines its meter and rhyming such as iamb, haiku or anapest. Modern poetry is more concerned about creating something new by breaking any strict structural rules and by continuously questioning what poetry is, what it can be and what it should be (see [START_REF] Kantokorpi | Runousopin perusteet[END_REF].

In the field of poem generation, meter is a feature that is very often considered in generated poetry [START_REF] Colton | Full-face poetry generation[END_REF][START_REF] Lau | Deep-speare: A joint neural model of poetic language, meter and rhyme[END_REF]Hämäläinen and Alnajjar 2019b;[START_REF] Zugarini | Neural poetry: Learning to generate poems using syllables[END_REF][START_REF] Lewis | Syllable neural language models for english poem generation[END_REF]. By incorporating meter, people can be more forgiving when evaluating the output of the system as it is known that people are ready to interpret more into the content of the output of a computationally creative system if the form is correct [START_REF] Veale | The shape of tweets to come: Automating language play in social networks[END_REF]. In other words, a poem that looks like a poem, as in that it follows a certain meter, must be a poem. A truly competent computational poet should be capable of generating something that is recognizable as a poem even if its output was modern free-form poetry.

In this paper, we explore the topic of modern poetry generation in French. We fine-tune a novel encoder-decoder architecture which consists of a RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] based model as the encoder and a GPT-2 [START_REF] Radford | Language models are unsupervised multitask learners[END_REF] based model as the decoder. Because RoBERTa is very good at natural language understanding tasks but poor at generation tasks and GPT-2 is good at generation, but bad at understanding, it makes sense to incorporate both of the models. The task of RoBERTa is to encode the input (i.e. to understand poetry) and the task of GPT-2 is to decode the output (i.e. to generate poetry).

Related work

Poem generation has sparked a lot of interest in the past as we can see in a recent survey on the field (Gonc ¸alo Oliveira 2017). There is also some work on generating French poetry in particular (Van de Cruys 2019; [START_REF] Poibeau | Sonnet combinatorics with OuPoCo[END_REF]. In this section, we will provide a quick overview on some of the related work.

Poetry has been generated by using rules. This can, for instance, be done by modeling certain poetic features to find suitable verses [START_REF] Lamb | TwitSong 3.0: towards semantic revisions in computational poetry[END_REF], by writing different rules to generate verses with different poetic features [START_REF] Hämäläinen | Harnessing nlg to create finnish poetry automatically[END_REF] or by using templates [START_REF] Colton | Full-face poetry generation[END_REF].

Genetic algorithms have been popular in poem generation as well [START_REF] Manurung | Using genetic algorithms to create meaningful poetic text[END_REF][START_REF] Yang | Research on community competition and adaptive genetic algorithm for automatic generation of tang poetry[END_REF]2021). In many of these approaches, the generation itself is not governed by rules, but rules are used to assess the quality of each generated poem. These approaches can be seen as a step away from fully relying on rules in the generative process.

Recently, different neural approaches have become common place in poetry generation such as LSTM models [START_REF] Zhang | Automatic generation method of ancient poetry based on lstm[END_REF][START_REF] Atassi | Comparison and generation of a poem in arabic language using the lstm, bilstm and gru[END_REF] and GPT-2 (Hu and Sun 2020;[START_REF] Beheitt | Automatic arabic poem generation with gpt-2[END_REF]. Our approach follows this line of work. How we set ourselves apart from the existing models is that we propose a novel poem generator model that can benefit from both RoBERTa and GPT-2 instead of relying only on one model. In addition, we condition the beam search on rhyme instead of solely relying on the output of a vanilla model.

Data

As machine learning requires data, we need a poem corpus.

For this reason, we crawl all the French poems that are available on Wikisource 1 . The poems are not free of noise as some of the poems include verses in Greek alphabets, multiple different types of quotation marks, hyphens and spaces of different lengths etc. We clean the data from all of these inconsistencies by manually inspecting odd characters and either by replacing them (e.g. only one type of a hyphen) or removing them (e.g. Greek letters). The corpus contains 7553 poems. In addition, we use the French sonnet corpus introduced by [START_REF] Poibeau | Sonnet combinatorics with OuPoCo[END_REF]. This corpus has 1039 sonnets.

Because these poems and sonnets are of different lengths, we split all of them into stanzas. From this point on, we treat a stanza as a poem so that all poems in our corpus are of a similar length. This gives us altogether 25,215 French poems and sonnets. For the purposes of our models, we do not make a distinction between poems and sonnets.

Poem generator

In this section, we describe our poem generation model. The model follows an encoder-decoder architecture where the encoder is a RoBERTa model and the decoder is a GPT-2 model. Rather than training these models from scratch, we use pretrained language models and fine-tune them for the task of poem generation using a transfer learning approach. We chose a RoBERTa-based model as the encoder given their great ability in capturing contextual semantics. GPT-2 is well-known for modeling a language; hence, making an optimal decoder for text-generation tasks.

First we have to pick the suitable pretrained models. As we use Transformers library [START_REF] Wolf | Transformers: State-of-the-art natural language processing[END_REF], we select our models from their repository. The current state-of-theart French RoBERTa model is CamemBERT 2 [START_REF] Martin | CamemBERT: a tasty French language model[END_REF] which is based on the RoBERTa [START_REF] Liu | Roberta: A robustly optimized bert pretraining approach[END_REF] architecture and trained on the large OSCAR corpus [START_REF] Abadji | Towards a Cleaner Document-Oriented Multilingual Crawled Corpus[END_REF] in French. We use CamemBERT as our encoder.

As for the selection of the GPT-2 model, there were several alternatives. By trying the models out, we could see that all of them except for Belgian GPT-2 3 (Louis 2020) predicted rather poor output. The model was trained on a variety of genres (such as news, Wikipedia, novels, European parliament text etc.) on a relatively big, around 60 GB, corpus. For this reason, we opted for Belgian GPT-2 as our decoder model.

1 https://fr.wikisource.org/wiki/Catégorie:Poèmes 2 https://huggingface.co/camembert-base 3 https://huggingface.co/antoiloui/belgpt2

We use Spacy4 [START_REF] Honnibal | spacy: Industrial-strength natural language processing in python[END_REF] to extract up to 4 keywords from each poem in the corpus. We train our encoder-decoder architecture for sequence to sequence generation, where it predicts the next verse in a poem given a previous verse. In the absence of a previous verse, we train the model to predict the first verse of a poem from the up to 4 keywords extracted from the poem. An example of input and output in the training data for one poem can be seen in Table 1. The first input consists of the keywords amant (lover), galamment (gallantly), escrime (fencing) and dame (lady), which are used to predict the first verse of the poem.

The poem corpus is split randomly to 80% for training and 20% for validation. The model is trained for 10 epochs. We use the Adam algorithm (Kingma and Ba 2014) with decoupled weight decay regularization [START_REF] Loshchilov | Decoupled weight decay regularization[END_REF] and learning rate of 5e-05 to optimize the parameters of the model, with cross entropy loss as the loss function to reduce the difference between gold standard token and predicted tokens.

Rhyming is taken into account during the generation phase. The model is requested to generate a sequence between 4 to 20 tokens, with a length penalty of 1.0 using a greedy approach. At each step of generating the output sequence (i.e., when predicting the next token), we use the model to predict the top 10 possible tokens instead of just one highest scoring output. We then sort these candidate tokens based on their probabilities and rhyming scores. The rhyming score is calculated by counting the number of tokens in the output that rhyme (full rhyme, consonance or assonance) with the input (i.e., the previous verse and any subsequent words generated during the run).

Because it is not easy to know whether two French words rhyme or not based on the orthography (similarly to English), we use eSpeak-ng5 to produce an IPA (international phonetic alphabet) representation for each token in the model's vocabulary. IPA alphabets are designed to represent how words are pronounced by writing out the actual phonemes. We use a simple set of rules to compare the IPA strings of two tokens with each other to determine whether they rhyme or not.

In practice, we first ensure that both of the IPA strings are equally long, if this is not the case, we remove characters from the beginning of the longer string until the IPA strings are equally long. If the strings are identical, no rhyme is considered, because a word does not make a good rhyme with itself. For full rhyme, the two IPA strings rhyme if they are identical from the first vowel onward. For assonance, we replace all consonants with a placeholder character C, if the D'un beau travail, d'une bonne pose, De la paix, de la beauté. Que je plains la beauté De la femme, qui m'inspire From a beautiful work, from a good pose. From the peace, from the beauty Oh, I lament the beauty Of the woman, who inspires me C'est ici que s'éveille le soleil, C'est ici que repose le grand créateur, Dont la ruine, hélas! se renouvelle De l'Enfant du Progrès It is here where the sun wakes It is here where the great creator rests Whose ruin, alas! renews itself From the Child of Progress C'est un des mois les plus beaux de l'année, C'est le printemps, c'est l'été, c'est Le ciel où mon printemps se joue. À mon jardin qui s'effondrit.

It is one of the most beautiful months of the year, It is the spring, it is the summer, it is The sky where my spring plays. In my garden that collapses Table 2: Examples of generated poetry and their translations. IPA strings are identical, i.e. they share the same vowels in the same positions, they are considered to have assonance rhyme. For consonance, we do the same as with assonance, but by replacing all vowels with a placeholder V.

Results and evaluation

For evaluation purposes, we generate 20 different poems consisting of 4 verses each. For each poem, we use a set of four randomly selected keywords among all the keywords extracted from the poem corpus. None of the keyword combinations is identical to what the model saw during the training. We generate the poems similarly to the example shown in Table 1. This means, that the keywords were used to generate the first verse, which was then used to generate the second verse and so on. Some of the generated poems and their translations can be seen in Table 2. As we can see, the generated output is cohesive and quite grammatical. We can, however, see that sometimes the verb conjugation might be wrong such as in the case of effondrit which is a non-existing inflectional form of effondrer (to collapse). Also, the model has a tendency of starting every verse with a capital letter even if it was a continuation to the sentence started in the previous verse.

We conduct a crowd-sourced evaluation on Appen 6 . We set French as a language requirement for the crowd-workers so that we know that they actually speak French and are able to assess French poetry. Each poem is evaluated by 20 different crowd-workers. An individual worker can evaluate all 20 different poems or just some of them, in which case the remaining unevaluated poems are shown to a different crowd-worker. An individual crowd-worker cannot evaluate the same poem multiple times.

For evaluation, we use the same parameters as used by several authors for evaluating poetry [START_REF] Toivanen | Corpus-Based Generation of Content and Form in Poetry[END_REF]Hämäläinen and Alnajjar 2019a;[START_REF] Shihadeh | Emily: An emily dickinson machine[END_REF]: (1) The poem is typical (2) The poem is understandable (3) The poem is grammatical (4) The poem evokes imagination (5) The poem evokes emotions (6) I like the poem. These statements are evaluated in a 5 point Likert scale, where 1 represents the worst and 5 the best grade. The results can be seen in Table 3. All in all, the results are good and show that the system can generate poetry successfully. The lowest scores were obtained for typicality and emotionality. The highest score was given to understandability. In the future, more robust human evaluation methods need to be applied to understand why these parameters scored high and low (Hämäläinen and Alnajjar 2021a;2021b).

Conclusions

In this paper, we have presented a novel approach to French poem generation. We have presented an architecture that consists of RoBERTa and GPT-2 models that are fine-tuned on a poem corpus. In addition, we have modeled rhyme as a part of the prediction pipeline of the model.

The results obtained in human evaluation are promising and they indicate that the model performs well in the task it was designed to do. In order to make the evaluation results more transparent, we have released them in full on Zenodo 7 together with the generated poems that were used in the evaluation.

Pretrained neural language models have been proven to be useful in poem generation. In the future, it would be interesting to study them in a multilingual setting, where a pretrained multilingual model is fine-tuned to generate poetry using the corpora of some languages other than the desired target language.

Table 1 :

 1 des dames, tu t'escrimes In vain, paladin of the ladies, you fight me with a sword 2 Vainement, paladin des dames, tu t'escrimes À crier que j'ai fait, ici, peu galamment. Shouting what I have done, here, little gallantly. 3 À crier que j'ai fait, ici, peu galamment. si de Laure Pétrarque avait été l'amant, if Pétrarque had been the lover of Laure 4 si de Laure Pétrarque avait été l'amant, Qui sait ce qu'il eût dit de Laure dans ses Rimes? Who knows what he said about Laure in his Rhymes? Example of the training data for one poem

	Verse Input	Output	Output in English
	1	amant galamment escrime dame	Vainement, paladin

Table 3 :

 3 The evaluation results and standard deviation

	6 https://appen.com/

The fr core news sm model

https://github.com/espeak-ng/espeak-ng

Acknowledgments

This work was partially financed by the Society of Swedish Literature in Finland with funding from Enhancing Conversational AI with Computational Creativity. This work was conducted during a mobility period supported by Nokia Foundation under grant number 20220193. This work was supported in part by the French government under management of Agence Nationale de la Recherche as part of the "Investissements d'avenir" program, reference ANR19-P3IA-0001 (PRAIRIE 3IA Institute). The work was also supported by the CNRS funded International Research Network Cyclades (Corpora and Computational Linguistics for Digital Humanities).

Author Contributions

The first two authors contributed to the work presented in this paper equally. The third author was involved in planning the methods and writing the paper.

7 https://zenodo.org/record/6558357