Kenneth Clark

Ella Beaucamp

A joint venture from Venice: Medieval rock-crystal miniatures

The paper will explore the production of rock-crystal miniatures in medieval Venice. This art form relied not only on the city's unique access to raw materials through trade and colonial territories, but also on collaboration between the city's craftsmen and guilds. Crystal cutters, glass workshops, wood sculptors and miniature painters worked together to produce artefacts with miniatures on vellum behind foils of rock-crystal and semi-precious stones or glass paste. Far from being a lowpriced 'imitation ware' of Byzantine enamel, this specific 'Venetian' form of decoration was used on altarpieces, candlesticks, frames and chessboards, which were commissioned by both sacred and secular clients, and exported to distant regions. How was the manufacture and distribution of rockcrystal miniatures organized in the lagoon and why did this craft come to an end in the fourteenth century? As we will see, the 'collaborative' structure of the Venetian guild system was one important reason for the success of this medieval industry.

Ella is a doctoral candidate in the Junior Research Group 'Premodern Objects. An Archaeology of Experience' at the Ludwig-Maximilians-University Munich under the direction of Dr. Philippe Cordez. She completed her study of Art History at the LMU in June 2016 with a master thesis on the so-called 'Patere and Formelle'marble reliefs from the palaces of medieval Venice. From 2013 to 2014 she spent one year as student assistant at the Kunsthistorisches Institut in Florence in the department of Prof. Dr. Gerhard Wolf.

Costanza Beltrami, The Courtauld Institute of Art Building and rebuilding the cloister of Segovia Cathedral (1436-1530): collaborations across space and time

In the early fifteenth century, the late-gothic master Juan Guas added a new cloister to the Old Cathedral of Segovia. Commissioned in 1471, the cloister was completed in just eight years, several months earlier than required by its contract. Guas' speed can be explained by his reliance on skilled stonecutters that followed him from one site to another. As is often noted, such network of 'employees' determined the master's success as a sophisticated late-gothic 'architect.' Nevertheless, non-hierarchical collaborations in space and time also shaped Guas' work. In fact, the cloister was begun in the 1460s by Juan de Toro, and then significantly altered in the 1520s, when Juan Campero disassembled and moved it to a new site following the Old Cathedral's destruction. Rather than focusing on Guas' innovative workshop structure, this paper analyses his 'collaboration' with his predecessors and followers, masters of similar expertise whose contributions dilute the concept of individual authorship.

Costanza is a second-year PhD candidate. Her thesis, supervised by Dr Tom Nickson, is titled 'A reconsideration of Juan Guas (active 1453-1496) and the Hispano-Flemish style.' It discusses the leading master mason of late medieval Castile, using this figure to question modern assumptions on collaboration, artistic exchange, individual creativity, identity, and stylistic hybridity. She is also interested in architectural drawings, and has recently authored Building a Crossing Tower: a design for Rouen Cathedral of 1516 (Paul Holberton Publishing, 2016).

Eleonora Cagol, Technische Universität Dresden

The workshop of Jörg Arzt and Jörg Feiss: winged altarpieces as evidence of artistic collaboration in South Tyrol at the end of the Middle Ages

The wooden altar of Saint Juliana, realised for a Church dedicated to the martyr in Vigo di Fassa in Trentino-South Tyrol, is a perfect example of artistic collaboration. Sculptures, reliefs, paintings, ornaments and architecture merge in this liturgical altar creating a complex artwork. At least five different artists were involved in its realisation, including the inventor, painter and "Fassmaler" Jörg Arzt and the sculptor Jörg Feiss. According to the few remaining archival sources, Arzt and Feiss led a very active workshop in Bolzano in the first twenty years of the cinquecento. The analysis of the production and the organization of this workshop enables us to understand how different artists worked together to create a "Gesamtkunstwerk", a synthesis of arts. Moreover, the examination of the workshop of Arzt and Feiss shows how artistic collaboration was planned and executed in the workshop and how crucial this cooperation was to realise a winged altarpiece.

Eleonora has been a PhD student at the Technische Universität Dresden since April 2015. She studied in Italy and in Belgium before moving to Dresden, where she has worked at the Gemäldegalerie Alte Meister and obtained a Masters in Art History. She is currently working on her PhD project about winged altars from the south Tyrolean workshop of Arzt and Feiss under the supervision of Prof. Jürgen Müller (TUD) and Prof. Massimo Ferretti (SNS).

Aude Chevalier, Paris Nanterre Université

Collaborations in medieval copperware craftsmanship: the case study of French copper alloy censers (11th-17th centuries)

In a craft such as medieval copperware each artisan was limited to specific parts. However, the sum of their collaborations was essential to the creation of these objects. The case study of French copper alloy censer manufacture is a revealing example of the variety of collaborations involved in such craftsmanship. From miners extracting ores and charcoal, to merchants selling the final object, censer production involved many different stakeholders through each step of the process. Close and organized cooperation between these stakeholders was therefore mandatory to the finished product. This overview of the censer-making process aims to show the weight of collaboration in such a production, by focusing on three main steps of these objects' conception: their aesthetic development, their manufacture and their sale.

Aude is a PhD candidate at Paris Nanterre Université (France) and a former student at Ecole du Louvre. Her research deals with the production, uses and iconographies of non-precious censers produced in France between the 11th and the 17th centuries, especially in the Auvergne region. Her research proposal was awarded a French Ministry of Culture grant in 2016. Aside from her research, she lectures on early Christian and High Middle Ages art at her university.

Maggie S. Crosland, The Courtauld Institute of Art Commission as Collaboration: Untangling Agency in the Book of Hours of Philip the Bold

The Grandes Heures of Philip the Bold, the only known book of hours made for Philip, presents an ideal case study through which to explore the complicated process of commissioning a manuscript. The hours is well documented in the Burgundian archives, from 1376, when the duke's confessor Guillaume de Valen was charged with its making, to 1379 when the scribe Jean L'Avenant was paid for its completion. However, no contract for the manuscript survives, leaving questions regarding whothe duke, the confessor, the scribe, or the unnamed artistswas responsible for the extensively illuminated manuscript. To what degree can we understand the role that each man played in choosing the contents of the manuscript? To what extent does this differentiation in agency even matter in our understanding of the functionality of this object? This paper aims to consider these questions alongside both the manuscript's contents and the broader context of royal manuscript production and use in late fourteenth-century Paris.

Maggie is a second-year PhD student at The Courtauld. Supervised by Dr Alixe Bovey and Dr Susie Nash, she studies usage and function of illuminated prayer books in fourteenth and fifteenth-century France, using Philip the Bold's book of hours as a case study. Maggie completed her BA at Bowdoin College and her MA at The Courtauld, and has held research positions at the J. Paul Getty Museum and the Art Institute of Chicago. She is an editor of The Courtauld postgraduate journal immediations.

Esther Dorado-Ladera, Independent Scholar

Reuse of Hispanic Muslim architecture during the Middle Ages: Christian interventions in the Aljaferia Palace

The appropriation of Hispanic Muslim architecture and the requirements resulting from the continuous movement of the borders during the Reconquista gave rise to the Christian reuse of Andalusian constructions. This paper will focus on the architectural interventions made in the Aljaferia Palace by Christian royalty. There is evidence of successive extensions and maintenance works in the palace, but the purpose and importance of the interventions by Peter IV and Ferdinand II were crucial. The different constructional approaches and the changing attitude towards the Islamic building throughout the Middle Ages will be discussed. Furthermore, the Christian works at the Aljaferia and the artistic activity of the whole region were deeply influenced by the remains of the Muslim palace. I will argue that the preservation of a large part of the initial construction led not just to the reuse of building materials and the acquisition of new techniques, but more importantly to the transfer of an entire visual culture.

Esther is an architect. She completed her PhD in Theory and History of Architecture in the Universitat Politècnica de Catalunya (Barcelona) in April 2015 with a thesis on the symbolic value of geometric ornamentation in medieval Iberian architecture. Since then her research interests have turned to the artistic interchange among the several ethnic communities that coexisted in the Iberian Peninsula during the Middle Ages and the transferring of visual culture elements between different architectural traditions.

Amanda Hilliam, National Gallery / Oxford Brookes University Carlo Crivelli and the Goldsmith's Art: shared aesthetics and technologies

The affinities between Carlo Crivelli's treatment of gold and the art of precious metalwork have been noted on several occasions in the literature on the Venetian painter. However, no attempt has been made to unpick the complex visual relationship between Crivelli's resplendent, mixed-media polyptychs and the reliquaries, processional crucifixes, religious regalia and secular accessories that were being expertly produced contemporaneously in the Marches, where Crivelli was active from c.1468c.1494. This paper will take the document in which the goldsmith Mariano di Venanzio Meneconi, who had partnered with the Marchigian goldsmith and sculptor, Pietro Vannini, is paid in relation to the 1482 triptych for San Domenico, Camerino, as a starting point for discussion. What role might Meneconi have played in the making of this object, which combines painted surfaces with raised, gilded gesso ornamentation, elements in wooden relief, attached precious stones and an elaborately tooled gold ground? Amanda Hilliam is a collaborative doctoral researcher at the National Gallery, London and Oxford Brookes University. Her research explores the interaction between painting and sculpture in the work of Carlo Crivelli. She received her BA from the University of Bristol and her MSt from the University of Oxford in History of Art and Visual Culture.

Bryan C. Keene, The J. Paul Getty Museum / The Courtauld Institute of Art Pacino / Pacinesque: Collaborative Choir Book Commissions in Early Trecento Florence

Scholarship on early Trecento Florentine manuscript illumination centres on numerous anonymous "masters"-like the Master of the Dominican Effigies or the Master of the Codex of Saint Georgewho appear to have collaborated at times with artists generally associated with Pacino di Bonaguida, an individual known from a single signed altarpiece and from two archival sources, yet considered today to have been the head of a prolific workshop. The term "Pacinesque" is an attribution often used for broadly associated works on parchment and on panel, including over thirty choir books made for every major religious community and for various parish churches in the region. This paper considers new methodological approaches for understanding collaborative choir book commissions, including a look at marginal instructions for illuminators, evidence from technical analysis about colour palette, codicological features, and an examination of related contemporaneous commissions for altar furnishings by panel painters, glass workers, and weavers.

Bryan is a specialist in Italian manuscript illumination and painting and is interested in the global Middle Ages, especially the nexus of Afro-Eurasian book culture (the subject of his forthcoming book). He has curated exhibitions and published on painting and illumination at the time of Giotto, Renaissance gardens and landscape painting, art of the Northern Italian courts, and on Giovanni di Paolo. He is completing his PhD at The Courtauld Institute of Art.

Eowyn Kerr-Di Carlo, The Courtauld Institute of Art Santa Maria degli Angeli or not? Considering Florentine artistic networks and the painter-illuminators of Fitzwilliam MS 30

This paper examines evidence of an artistic network connecting the scriptorium of Santa Maria degli Angeli to the Badia, Florence's Benedictine Abbey, in the early years of the fifteenth century. While both ecclesiastical institutions were known as centres of manuscript production in their own right neither has been considered in relation to the other. Indication of this cooperative working arrangement can be found in the commission and decoration of Fitzwilliam MS 30, a luxury Missal produced for Cardinal Angelo Acciaiuoli between 1402-1405 while acting as commendatory abbot of the Badia. The four artists responsible for MS 30's illuminations are all directly associated with Santa Maria degli Angeli yet they also regularly rented workspaces from the Badia. The close proximity of the Badia's commercial workshops, and supporting documentary material indicating the activities of their tenants, suggests a more collaborative relationship between the illuminators, binders and stationers than previously considered.

Eowyn is currently a PhD candidate at The Courtauld. Her research focuses on the techniques of late fourteenth-century painters and illuminators working within the circle of Lorenzo Monaco. She holds an MA in Art History from The Courtauld and an MA in Art Conservation from SUNY Buffalo. Eowyn worked for over ten years as a paintings conservator, and came to London as a Kress Conservation Fellow for the V&A's Medieval and Renaissance Galleries.

Oliver Mitchell, The Courtauld Institute of Art

Collaboration and conflict in Hugo de Folieto's Liber de rotae religionis et simulationis Soon after the composition of Hugo de Folieto's Liber de rotae religionis et simulationis, illustrators began to embellish the author's simple wheel diagrams with figures and captions in the manner of a Wheel of Fortune. This collaboration is not without conflict, asin their likeness to the Wheel of Fortune -these novel illustrations diverge from Hugo's own conception of the images. His individual authorship is thus contested by later collaborators, but this reworking of the original makes De rotae the fascinating work that it is. This paper will explore image-text relations in De rotae, tracing the changing wheel diagrams in its early manuscripts and showing how these differed from the wheels described in the text. It will be shown that this reworking, though far from seamless, contributed to what popularity De rotae achieved by engaging with the vast interpretative potential of the Wheel of Fortune at this early point in its history.

Oliver studied for a BA (2015) and MA (2016) The furnishing and funding of the medieval altar was equally affected by its liturgical function, the wishes of generous donors and the abilities of artists and craftsmen. Indeed, many facets of the altar's appearance required collaboration between parties with different concerns. Collaboration can be observed in individual objects destined for the altar, such as the chalice of Queen Urraca (eleventh century), and in the furnishing and design of individual chapels and their altar ensembles, such as the altarpiece and frontal created in-situ for Pedro López de Ayala's family chapel (1396). As a central gathering or reference point for the associated community at large, both lay and clerical, it is important to explore the ways in which it "took a village" to furnish, use and maintain the medieval altar.

Maeve gained her BA in Art History from NYU and her Masters from Hunter College. She is in the last year of her PhD, under the direction of Dr Tom Nickson and with the support of the Ochs Scholarship. Her dissertation aims to present a social history of altars and altar furnishings in 13th and 14th century Castile.

Marie Quillent, University of Picardy Jules Verne Artistic Collaboration in Medieval Funeral Sculpture in the North of France: the Tomb of Adrien de Hénencourt in the Cathedral of Amiens

This contribution deals with artistic collaboration in the conception and realisation of medieval funeral monuments in the North of France. Focusing on the particular case of the tomb of Adrien de Hénencourt, dated to the beginning of the sixteenth century in the cathedral of Amiens, it examines the documentary evidence for this tomb's execution. These documents reveal the names and specialisms of the artists involved, including painters and sculptors working in different media. Thanks to this evidence we are also able to understand the process through which the tomb was realised. As such, this case invites questions regarding the links between artists of differing specialisms, and in a more general sense, the methodologies of attributing sculpted artworks, which often focus on the sculptor and neglect the other artistic collaborators.

Marie is a French doctoral student affiliated to the University of Picardy Jules Verne. She is interested in medieval History of Art, Museology and Archaeology. In addition to her first degree in History of Art at the University of Picardy Jules Verne, she also holds a Museology diploma from the Ecole du Louvre and has taken part in archaeological excavations in the South of France. She is currently working on medieval funeral monuments in Northern France.

  at The Courtauld Institute of Art and is now undertaking a CHASE-funded doctoral research project on the Wheel of Fortune in medieval art c. 1100-1500 supervised by Prof John Lowden and Dr Alixe Bovey. Maeve O'Donnell-Morales, The Courtauld Institute of Art It Took a Village: Collaborations at the Medieval Altar between Donors, Artists and Clerics


Chloé Braud

Janet Yang

Eleni Liu

Philippe Metheniti

Laura Muller

Attapol T Rivière
email: attapol.t@chula.ac.th4yl879

Amir Rutherford

Zeldes

Yang Janet Liu

Eleni Metheniti

Philippe Muller

Laura Rivière

Attapol T Rutherford

Amir Zeldes
email: amir.zeldes@georgetown.edu

The DISRPT 2023 Shared Task on Elementary Discourse Unit Segmentation, Connective Detection, and Relation Classification

published or not. The documents may come

Introduction

Discourse parsing aims to uncover the underlying structure of monologues or dialogues, where spans of texts are linked together by semantic-pragmatic discourse relations such as EXPLANATION, CON-TRAST, TEMPORAL-ASYNCHRONOUS, or GOAL. Examples of such structures in different representations are given in Figures 1 and2. Several theoretical frameworks have been proposed for discourse analysis and have subsequently been used in many annotation projects. Common ones include the Rhetorical Structure Theory (RST, [START_REF] Mann | Rhetorical Structure Theory: Toward a functional theory of text organization[END_REF], where discourse structures are hierarchical constituent trees (Figure 1), and relation definitions are based on authors' or speakers' intents; the Segmented Discourse Representation Theory (SDRT, [START_REF] Asher | Logics of Conversation[END_REF], where structures are graphs with non-terminal nodes, and * Discourse Relation Parsing and Treebanking (DISRPT 2023) was held in conjunction with CODI at ACL 2023 in Toronto, Canada and Online (https://sites.google.com/ view/disrpt2023/). [START_REF] Yang | SciDTB: Discourse dependency TreeBank for scientific abstracts[END_REF]. relations are defined using formal logics; and the Penn Discourse Treebank (PDTB, [START_REF] Prasad | The Penn Discourse TreeBank as a resource for natural language generation[END_REF] with relations between isolated pairs of argument spans, possibly marked by a connective (e.g. but, because) which is then annotated a sense label. In addition, building upon several studies proposing to encode discourse structures as dependencies (Hirao et al., 2013;[START_REF] Muller | Constrained decoding for textlevel discourse parsing[END_REF], it has been proposed to annotate discourse graphs using pure dependency structures, with no nonterminal nodes (Figure 2), while keeping relations and segmentation rules from RST-which is abbreviated here as the DEP framework [START_REF] Yang | SciDTB: Discourse dependency TreeBank for scientific abstracts[END_REF].

Within each framework, numerous corpora of a variety of languages and domains have been annotated. However, the differences between the annotation projects hinder the evaluation of the progress made and to develop systems that should ideally perform well on the broadest possible range of data. [START_REF] Zeldes | The DIS-RPT 2019 shared task on elementary discourse unit segmentation and connective detection[END_REF] proposed the first iteration of the shared task of Discourse Relation Parsing and 1 Treebanking (DISRPT)1 in order to broaden the scope of discourse studies by including datasets and inviting researchers from different discourse theories, to facilitate cross-framework studies.

The first edition of the DISRPT shared task was limited to Task 1: discourse segmentationidentifying the elementary discourse units (EDUs) that may be linked by discourse relations; and Task 2: discourse connective detection-identifying specific lexical items, called connectives, that can signal a discourse relation (e.g. while, because, since, as long as etc.). In 2021, for the second edition, [START_REF] Zeldes | The DISRPT 2021 shared task on elementary discourse unit segmentation, connective detection, and relation classification[END_REF] added a third task, Task 3: discourse relation classification-identifying a relation label between a pair of attached discourse units. 2 This year, for the third edition, we maintained the three tasks but expanded the benchmark with 10 new corpora, including datasets from the DEP framework: in total, 26 corpora were made available across 4 frameworks and 13 languages in a unified format. In the last phase of the shared task, we released 6 surprise datasets including data for a new language (Thai), as well as 4 out-of-domain (OOD) corpora for which only dev and test partitions were available.

Three teams participated in the shared task, with one team including half of the organizers of the shared task. Overall, two systems were proposed for Tasks 1 and 2, and three systems for Task 3. Two systems are based on fine-tuning Transformer masked language model encoders, while the third one relies on a generative transformer model for relation classification. Only one team presented results for all tasks and tracks (MELODI), and another team (HITS) reported results for all tasks but were limited to the Treebanked track (i.e. parsed and gold sentence-split data) for Tasks 1 and 2. The third team (DiscoFLAN) focused on relation classification only. For the Treebanked track, MELODI ranked first on the EDU segmentation task, and HITS ranked first on the connective detection task with very similar mean scores. For relation classification, HITS ranked first.

Related Work

Automatic discourse analysis is an active domain of research, with increasing interests for the past few years as tools become increasingly capable of handling such tasks, and discourse information can be helpful for many applications, for example for authorship attribution [START_REF] Ferracane | Leveraging discourse information effectively for authorship attribution[END_REF][START_REF] Wei | RST-Style Discourse Parsing and Its Applications in Discourse Analysis[END_REF], fake news or political bias detection (Karimi and Tang, 2019;[START_REF] Devatine | Predicting political orientation in news with latent discourse structure to improve bias understanding[END_REF], sentiment analysis [START_REF] Bhatia | Better document-level sentiment analysis from RST discourse parsing[END_REF]Huber and Carenini, 2020), or for generation, with uses in machine translation [START_REF] Tu | A novel translation framework based on Rhetorical Structure Theory[END_REF]Joty et al., 2017;Webber et al., 2013) or summarization [START_REF] Louis | Discourse indicators for content selection in summarization[END_REF]Hirao et al., 2013;[START_REF] Yoshida | Dependency-based discourse parser for single-document summarization[END_REF][START_REF] Liu | Single document summarization as tree induction[END_REF][START_REF] Chen | Structure-aware abstractive conversation summarization via discourse and action graphs[END_REF]Hewett and Stede, 2022;[START_REF] Pu | Incorporating distributions of discourse structure for long document abstractive summarization[END_REF].

Discourse parsing is the full task of recovering a discourse structure of a document, either constituent trees in the RST-based framework, dependency trees for DEP, or graphs for SDRT. Performance is still far from perfect for full discourse parsing, and systems are mostly developed for English, monologues, and the newswire domain, using the largest news corpus available, the RST-DT (Carlson et al., 2001), with an F1 score of 55.4 at best for full trees [START_REF] Kobayashi | A simple and strong baseline for end-to-end neural RST-style discourse parsing[END_REF].

Recent studies also sometimes report results on other datasets, especially on GUM (e.g. [START_REF] Atwell | The change that matters in discourse parsing: Estimating the impact of domain shift on parser error[END_REF]Yu et al. 2022b), the largest RST English corpus to date, which is composed of multiple spoken and written genres [START_REF] Zeldes | The GUM Corpus: Creating Multilayer Resources in the Classroom[END_REF]. Very recently, [START_REF] Liu | Why can't discourse parsing generalize? a thorough investigation of the impact of data diversity[END_REF] showed the lack of generalization of existing SOTA RST discourse parsers through a series of experiments, with a significant performance drop when applied to unseen genres, and also demonstrated the importance of heterogeneous training data for robust discourse parsing.

A few attempts have also been made to develop systems for dialogues, especially using the SDRT STAC corpus [START_REF] Asher | Discourse structure and dialogue acts in multiparty dialogue: the STAC corpus[END_REF] with either supervised methods (Liu and Chen, 2021;[START_REF] Chi | Structured dialogue discourse parsing[END_REF]Yu et al., 2022a) or transfer learning strategies given the small size of the dataset (e.g. [START_REF] Fan | A distance-aware multi-task framework for conversational discourse parsing[END_REF].

Finally, multilingual RST discourse parsing has been the topic of a few work (Braud et al., 2017a;[START_REF] Liu | Multilingual neural RST discourse parsing[END_REF]Liu et al., , 2021) ) involving transfer to tackle data scarcity. Liu et al. (2021) in particular demonstrated that cross-lingual strategies could even help for English, and also that good segmentation is crucial for full discourse parsing, with a loss of up to 8% when using predicted EDUs.

As a matter of fact, an option to better understand the difficulty and low performance of discourse parsing is to examine its constituent subtasks, such as discourse segmentation, but also relation classification, and attachment ('naked' or unlabeled tree building). Many studies have been dedicated to these subtasks, with a specific focus on the first two. The aim of the DISRPT shared task is precisely to provide benchmarks for these critical steps toward full discourse parsing, to the extent possible in a formalism-neutral way, allowing participants to demonstrate the generalizability of their systems across languages, domains, and frameworks.

Discourse segmentation in particular has been seen as a solved task with performance as high as 94% on RST-DT as early as over 10 years ago [START_REF] Ngo | A reranking model for discourse segmentation 13 using subtree features[END_REF]. However, systems at the time were trained only on English newswires data with gold information about sentence boundaries and morpho-syntactic features. When facing realistic data in other languages and even in English, with systems based on predicted information, performance drops very substantially (Braud et al., 2017b). Disparities across languages and datasets were later emphasized within the DISRPT shared tasks [START_REF] Zeldes | The DIS-RPT 2019 shared task on elementary discourse unit segmentation and connective detection[END_REF][START_REF] Zeldes | The DISRPT 2021 shared task on elementary discourse unit segmentation, connective detection, and relation classification[END_REF] under realistic settings (with predicted sentence splits), with performance above 95% for some corpora, but scores in the 80s for the Spanish SCTB (82.5% at best), the Chinese SCTB (83.3), or the Russian RRT (86.2%). The best-performing system in 2019 [START_REF] Muller | ToNy: Contextual embeddings for accurate multilingual discourse segmentation of full documents[END_REF]) used a single multilingual BERT [START_REF] Devlin | BERT: Pre-training of deep bidirectional transformers for language understanding[END_REF] based model for every corpus, while in the second edition of DISRPT [START_REF] Zeldes | The DISRPT 2021 shared task on elementary discourse unit segmentation, connective detection, and relation classification[END_REF], the winning system (Gessler et al., 2021) achieved the best performance with an accuracy of around 91.5% on average, which relied on varied language models, either mono-or multilingual, as well as hand-crafted features. A loss of about 2% was observed when gold sentence boundaries are not given.

For Task 3, discourse relation classification is often further decomposed into different types of relations: explicit relations-ones that are triggered by a discourse connective (e.g. while, because), and implicit relations -ones that do not contain a discourse marker. The latter is considered a harder task, since no explicit cues are present, and has thus been studied more extensively (e.g. [START_REF] Kim | Implicit discourse relation classification: We need to talk about evaluation[END_REF][START_REF] Liang | Extending implicit discourse relation recognition to the PDTB-3[END_REF][START_REF] Long | Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations[END_REF].

For explicit relations, the task generally reduces to identifying the connectives, that is deciding whether a token such as 'and' is being used as a discourse marker, and then identifying the rela-tion, with the connective constraining the possible labels (e.g. 'and' can signal EXPANSION or RE-SULT, but not PURPOSE). Connective detection and explicit discourse relation classification have been considered easy tasks, with high performance [START_REF] Pitler | Easily identifiable discourse relations[END_REF], but it was later shown that performance drops drastically on non-news domains, or in languages with small datasets [START_REF] Xue | CoNLL 2016 shared task on multilingual shallow discourse parsing[END_REF][START_REF] Scholman | Comparison of methods for explicit discourse connective identification across various domains[END_REF]Johannsen and Søgaard, 2013).

For the first two editions of DISRPT, rather high performance was reported for connective detection: between an F1 score of 92-94 for the English and Turkish corpora, and an F1 score of 87 for the Chinese one, with only a small drop when gold sentence splits are not provided. However, this may be due to the relatively large and homogeneous datasets used in the evaluation. This year's new edition introduces 6 new corpora for Task 2, as well as OOD datasets for which no training data is available: this has made the task more challenging, with the mean scores now under 80% (see Section 5 below for details). We also report scores for implicit vs explicit relation classification for some corpora, which were not available in DISRPT 2021, and demonstrate low scores for implicit relations as well when data is scarce.

The DISRPT Shared Task is among the very few studies to report scores for both implicit and explicit (also including other types such as AltLex markers, see [START_REF] Prasad | Reflections on the Penn Discourse TreeBank, comparable corpora, and complementary annotation[END_REF]) relation classification, thus making it more practical for models to be able to recognize any types of relations. Task 3 was introduced in 2021, and the winning system was Transformers-based language-specific models for each target language and a set of hand-crafted features: overall the average performance was nevertheless still rather low (61.8%), showing room for substantial improvement.

Tasks and Tracks

Three tasks were proposed for DISRPT 2023:

[1] Discourse Unit Segmentation-the task consists of identifying each token as the start of an EDU or not (BO scheme at the token level).

[2] Discourse Connective Detection-the task consists of identifying each token as starting, being inside, or outside a discourse connective (i.e. BIO scheme at the token level).

[3] Discourse Relation Classification-the task consists of assigning a label to a pair of textual segments, given that a relation holds between the two units (i.e. multi-class classification).

While all corpora have data annotated for Task 3, note that they are not all relevant for Tasks 1 and 2:

• For corpora within the PDTB framework: the connectives are annotated, but no discourse segments are identified (i.e. Task 2 but not Task 1).

• Corpora within RST, SDRT, and DEP: the EDUs are identified, but not connectives (Task 1 but not Task 2).

The shared task also proposes two tracks for Tasks 1 and 2:

• Treebanked: data is tokenized, split into sentences, and parsed (morpho-syntactic information is given). When gold information was available in the original corpus, it is provided as is.

Otherwise, we provided predicted annotations done with Stanza [START_REF] Qi | Stanza: A Python natural language processing toolkit for many human languages[END_REF].

• Plain: data is tokenized and split into documents.

4 Shared Task Data

DISRPT Format

The goal of the Shared Task is to provide a unified format across corpora annotated in different frameworks.

Data Format Three types of formats are provided for each partition (train / dev / test) of each corpus. The .conllu and .tok files are the data for Tasks 1 and 2, and they correspond to the Treebanked and Plain tracks respectively. Metainformation is provided for documents in both formats. The .conllu files also have sentence annotation,3 part-of-speech (POS), and syntactic parse information, obtained either from Stanza or from gold standard treebanking. Some corpora have multi-word annotations: that is, both the contracted forms (indicated with specific IDs such as '2-3 can't') and the sub-forms ('2 can' and '3 not') appear in the files. Segmentation is indicated with a single label at the beginning of an EDU, at the position of the first token. The connective labels correspond to 2 labels: one indicating the beginning ('B') of a connective, and the other for tokens inside ('I') a connective. Note that discontinuous connectives (e.g. 'either ... or') are annotated as separate single connectives. 4The .rels files are for Task 3: each line corresponds to a pair of attached discourse units, with the annotation of the original relation from the corpus, and the label used for the shared task: in particular, PDTB-style relations are truncated at level 2 (e.g. CONTINGENCY.CAUSE.RESULT > CONTINGENCY.CAUSE, and RST-DT relations are grouped into 17 classes as done in Carlson and Marcu (2001). In addition, for the 2023 edition, some mappings were performed in order to make the data more homogeneous: only minimal modifications were done including correcting misspelling and non-significant merging such as E-ELAB > E-ELABORATION, SOLUTION-HOOD > SOLUTION-HOOD, and TOPICOMMENT > TOPIC-COMMENT. The full mapping is given in Table 7 in Appendix A. The files also contain sentence contexts for each span, indicate discontinuities in the spans, and provide the direction of the relation (unit 1 to unit 2: 1>2; or the reverse 1<2).

Changes since DISRPT 2021

The major change is the newly added 10 corpora, including datasets without training data with the aim of better testing models' generalizability. In addition to the minimal relations mappings described above, we also add the annotation of multi-word expressions in more corpora, for consistency.

Summary of the Datasets

In total, this year's DISRPT shared task included 26 corpora annotated across 4 frameworks and 13 languages. We provide general statistics of each dataset in Table 8 in Appendix B. For more information, please consult the relevant publications provided in the last column of the table.

The corpora vary not only in terms of languages and sizes, but also their genres and domains, including news, wiki, scientific documents, conversations, and so on. Corpora vary tremendously in extent: as shown in the upper part of the smallest have about 50 documents and 15k tokens. We note that the Russian corpus has twice the amount of tokens compared to corpora of the same size in terms of documents, which indicates longer documents (scientific papers). The English SciDTB, on the other hand, is very large in document count, with almost 800 documents that seem very short: it is composed of scientific abstracts. In the lower part of Table 8 (PDTB framework), the difference is even more obvious: the English PDTB dataset contains 2, 162 documents, while the English portion of the TED-Multilingual corpus only contains 6 documents (Zeyrek et al., 2018(Zeyrek et al., , 2019)).

In general, performance is lower for small datasets, and one way to improve performance when facing data scarcity is to take advantage of larger datasets, as attempted by some participants, notably for Task 3, relation classification.

The statistics also give insights into the differences between genres or languages and annotation guidelines across different corpora. The number of EDUs varies a lot: for example, the English STAC corpus contains a lot of EDUs relative to its size, likely due to the 'conversational' and abbreviated nature of online chatting. We can also see that the size of the label set differs between corpora, even within the same framework: between 9 and 23 for PDTB, 14 and 32 for RST, and 12 or 24 for DEP (SDRT has only 2 corpora with a more stable set of 16-18 labels). Label sets are not identical, even within the same framework, due to different relation definitions or granularity as well as variations in naming formats, e.g. with a single or a 2-level convention, as in CONCESSION vs COMPARISON.CONCESSION, or even more minor change such as the use of capital letters or not. Some datasets provide much more fine-grained relations (for example over 70 originally for RST-DT, or over 30 for GUM), but we follow the common practice of collapsing these to fewer coarse classes used in most parsing research (however, original fine-grained labels were retained in an additional column in the .rels files where available).

We count a total of 163 different relation names in the targeted level of granularity, which led one team to propose some mappings to reduce the label space. This situation is an important challenge when trying to experiment with joint learning across corpora, and points to an open research direction in increasing convergence of discourse relation labels in the field.

Finally, Table 1 provides the statistics for the splits of training, validation, and evaluation sets for each corpus. We indicate the size of the label set in each partition: unfortunately, in some corpora, some relations present in the training set are not available in the evaluation set (e.g. 2 relations missing in deu.rst.pcc, 3 in eus.rst.ert, and 4 in nld.rst.nldt); even more crucially, in a few corpora, some relations are present in the test set but not in the dev set, preventing a good learning of these labels (fas.rst.prstc, nld.rst.nldt, and por.rst.cstn). This is another motivation for joint learning over different corpora; it could also be interesting to think about new splits of the data that would better preserve the label distribution.

Sentence and EDU Segmentation

Sentences are the basic unit for grouping words in NLP. They correspond to EDU boundaries: in most RST, SDRT, and DEP datasets each sentence starts a new EDU. With sentences given, the segmentation task corresponds to finding intra-sentential EDU boundaries, and corpora in general include these boundaries to some extent, depending possibly on the genre or the annotation scheme: for some corpora with a low rate of intra-sentential EDU boundaries, the task could thus be easier if the sentence splitter already gives good results. As a baseline and an indication of the complexity of the task, we thus report results for a sentence-based baseline, where each sentence is predicted to correspond to one EDU (see Table 2). We also report performance using another tool for sentence splitting, namely Trankit [START_REF] Van Nguyen | Trankit: A lightweight transformer-based toolkit for multilingual natural language processing[END_REF], used by the team MELODI for the Plain track.

Sentence boundaries are gold for some corpora (English RST-DT and GUM, and German PCC), for the others, Stanza [START_REF] Qi | Stanza: A Python natural language processing toolkit for many human languages[END_REF] was used to provide sentence splits in the .conllu format (the Treebanked track). A .tok format is also provided, without information about sentences (the Plain track). Our baseline results are computed on the Treebanked data and shown in Table 2.

Error Rate: bad performance of sentence splitters We compute the error rate by looking at the tokens that are supposed to start a sentence but are not annotated as beginning an EDU: they thus correspond to errors in sentence segmentation. The error rate is not 0 for RST-DT (gold sentences), because of alignments errors with the Penn Treebank [START_REF] Marcus | Building a large annotated corpus of english: The penn treebank. Special Issue on Using Large Corpora[END_REF]. In addition, error rate is very high for the Russian corpus: 15% of the sentences do not correspond to a new EDU and thus are considered errors. The Russian RRT is composed of scientific papers containing lists of references annotated as one (very) large EDU while the tools tend to segment each reference as a separate sentence. We also have 'non-standard' sentences of the form "sci.comp_49-61" which might be figures. The error rate is also rather high for the e.g. French, Basque, Chinese, and Spanish corpora. The sentence splitter is clearly suboptimal for the French corpus, with errors due to e.g. lists or other uses of punctuations within sentences and also specific quotations marks, as shown in the examples below where curly brackets indicate predicted sentence boundaries:

• {Mais avec un Leica M7 , il est encore possible de dire : « Je fais de la photo !} {»} -But with a Leica M7, it is still possible to say "I'm taking pictures!"

• {En 1866 , le cartographe britannique Charles W.} {Wilson identifia les ruines de la synagogue (...)} -In 1866, British cartographer Charles W.

Wilson identified the ruins of the synagogue (...)

High error rates could affect performance since sentences are generally the units fed to the systems, especially when the documents are too long for even large contextualized language models.

Baseline F-Score An F1 score gives an idea of one aspect of the complexity of the task: if F1 is high, it means that the corpus does not contain many intra-sentential EDU boundaries, which are arguably harder to detect. The STAC corpus mostly contains EDUs corresponding to a 'sentence', even if the definition of this unit is less clear for dialogues. The task should also be easier, with a good sentence splitter, for several RST corpora (nld, deu, zho, and spa). On the other hand, many corpora contain a high rate of intra-sentential EDUs, making the task harder, e.g. the Chinese SciDTB, the Farsi PRSTC, or the Chinese RST GCDT.

Participating Systems

Three teams submitted systems in time for participation: overall there were two systems for Tasks 1-2 and three systems for Task 3. All scores reported below come from our reproduction of these systems.

System Descriptions

HITS The HITS team participated in Tasks 1-2 and Tasks 3, with two separate systems.

Their approach for Tasks 1 and 2 was languagespecific, by fine-tuning monolingual or multilingual transformer-based models per corpus-for corpora with a training set. Their classifier architecture was based on pretrained models (various BERT or RoBERTa based for the monolingual models, XLM-RoBERTa-base for the multilingual), fine-tuned with a bidirectional LSTM network with a CRF layer (BiLSTM-CRF, Huang et al. 2015).

They implemented an adversarial training strategy, which introduced small perturbations to the original inputs in order to help the trained model generalize better. For corpora without a training set (the surprise and OOD ones), they used their previously fine-tuned models of the same language and framework.

For Task 3, the team submitted a system composed of two fine-tuned transformer-based models (as in Tasks 1-2, BERT or RoBERTa based for the monolingual models, XLM-RoBERTa-base/large for the multilingual). For large corpora, a corpusspecific fine-tuned classifier was used, based on monolingual or multilingual models. However, they aggregated smaller corpora in a joint training approach based on their frameworks, and then fine-tuned a multilingual model for classificationand also used those for corpora without a training set. They also implemented the adversarial training strategy for this task, for specific datasets.

MELODI: DisCut and DiscReT

The MELODI team submitted two systems to handle Tasks 1-2 and Task 3 respectively: DisCut and DiscReT. The former system is a revised version of the team's 2021 submission (Kamaladdini Ezzabady et al., 2021). The main modifications to DisCut included a shift to a single multilingual language model to accommodate all languages (XLM-RoBERTa-large was chosen, [START_REF] Conneau | Unsupervised cross-lingual representation learning at scale[END_REF], and the use of a simple linear layer for classification, replacing the character-level CNN and token-level LSTM used in the 2021 version. Additionally, the team experimented with layer freezing, finding an overall optimum for the large language model when layers 0-5 of 24 were frozen. Both Tasks 1 and 2 were handled as BIO-encoded sequence labeling, and no additional features beyond sentence splits were used (for the plain text scenario, Trankit was used to preprocess the data, see [START_REF] Van Nguyen | Trankit: A lightweight transformer-based toolkit for multilingual natural language processing[END_REF].

For Task 3, MELODI submitted DiscReT, which was unique in not only using a multilingual language model for all languages (this time choosing mBERT-base-cased) but also training jointly on all datasets after performing label lower-casing and selective merging to reduce the total of possible labels from 163 to 135 across datasets. Their models are fine-tuned and fitted with a fine-tuned Adapter. Adapters (Houlsby et al., 2019) offer a lightweight alternative built on transformers that expose only a subset of parameters to fine-tuning, reaching comparable results to fully fine-tuned transformers. The system did not use additional features, except for encoding the relation direction information by permuting the order of input sequences to always begin with the source argument of the relation (meaning sequences were transposed from their natural order for relations of the form 1<2).

DiscoFLAN DiscoFlan is based on the Flan-T5 generative language model, itself a fine-tuning of the T5 model on a large set of additional tasks (Chung et al., 2022). The basic principle of this family of models is to encode an instruction in natural language input to resolve a given NLP task, and to learn to decode it as the answer. In the case of discourse relation classification within DIS-RPT, this is implemented in DiscoFlan by finetuning Flan-T5 and encoding the instruction "what discourse relation holds between sent1 and sent2: sent1 <text> sent2 <text>" in various languages, and learning to decode the discourse relation label. A post-processing step tried to match an output token to an existing label, or select the majority class if the output cannot be mapped. The majority class is computed on the training set, or the dev set for the OOD corpora that do not have training sets.

Results

Task 1: EDU Segmentation Table 3 shows the EDU Segmentation scores of the two submitted systems. The comparison between the two systems for the Treebanked track indicates very similar results, with the winner being DisCut (a mean F1 score of 91.87) from the MELODI team. Both systems used rather similar architectures, and the main difference was the language model used as backbone: always XLM-RoBERTa large for MELODI, and for HITS a language model was specifically chosen for the target language. As illustrated here, it seems that the hyper-parameter tuning including freezing layers and/or the use of a large version of RoBERTa allows performance to be on par with the specific base models. Major improvements were observed for nld.rst.nldt (MELODI +4 points), spa.rst.rststb (+2), spa.rst.sctb (+3), and zho.rst.sctb (+2). However, these variations should be taken with precaution as we noticed an important variance of the scores when reproducing the results, especially for small-sized corpora.

In general, scores are high, and the performance of DisCut is better than the ones obtained by the winning system DisCoDisCo in 2021 (Gessler et al., 2021), with a mean score of 91.77 when only considering the corpora used in 2021 against 91.48 for DisCoDisCo (for the Treebanked track). See the paper describing the MELODI results for a full comparison. Additionally, this year's mean scores are not far from the 2021 ones, despite the addition of the new corpora and one OOD dataset (eng.dep.covdtb). This demonstrates some ro-bustness of the approaches as well as the consistencies of the new annotations. We note that a few corpora are still challenging, with performance below 90, in particular rus.rst.rrt, which is likely due to the issue with the bibliographic parts; and spa.rst.sctb and zho.rst.sct, which are parallel corpora and correspond to a rather high rate of sentence segmentation errors (4-7%), which should be investigated further.

The Plain track gives the opportunity to test EDU segmentation in a more realistic setting, i.e. no sentence splits are provided. However, since LLMs have severe limitations on input size, the DisCut system relies on another sentence segmentation, done with Trankit (Nguyen et al., 2021), but using the same tokenization as required for the evaluation for the shared task (which means that the results do not exactly reflect the performance of Trankit). Results show the mean performance is similar to the Treebanked track while, this time, no corpus contains gold sentence splits which is encouraging for future use of this kind of system on new data. Task 2: Connective Detection Table 4 shows the connective detection results of the two submitted systems, which remain the same as for Task 1. We also observe similar scores between MELODI and HITS, but this time HITS is the winner (a mean F1 score of 80.47). Contrary to EDU segmentation, the new corpora added for this task are very challenging, especially the OOD ones coming from the TED multilingual corpus and the LUNA corpus, that are small and consist of documents from very specific genres (TED talks and speech transcriptions of dialogues). As a comparison, mean score of DisCoDisCo in 2021 was 91.22, while now the mean is around 80's. For this task, sentence segmentation seems less a crucial factor; however, the comparison between the two tracks demonstrate huge differences for some corpora, e.g. -5.75 for Luna and -15.74 for the Thai corpus when using Trankit vs Stanza. These differences should be investigated further to better assess the role of sentence splitting in connective detection.

Task 3: Relation Classification For the relation classification task, three systems were submitted: DiscReT, HITS, and DiscoFlan. The winning system is HITS, with a mean accuracy score of 62.36.

The proposed strategy, with single models for large corpora and merging for small ones within each framework, seems more effective than the joint learning over all corpora proposed in DiscReT. Interestingly, the second system is still on par with or even better for a few corpora, meaning that merging across corpora to some extent could also help.

The scores indicate that some corpora are very challenging: the German PCC, the Turkish TDB, and the Dutch NLDT, with the accuracy score lower than 52. The new corpora do not seem more challenging than the others, except for the Turkish TEDm. We note that scores are very high for the Thai corpus (95.83), which could be due to the fact that only explicit relations are annotated in the current version. Compared to 2021, HITS has lower performance, with a mean accuracy score of 58.18 when only considering the corpora available in 2021 against 61.82 for DisCoDisCo, which indicates that the merging strategy including the new corpora could lead to drop in performance compared to single models, but more analysis is needed to investigate the impact of the hand-crafted features used in DisCoDisCo.

In order to provide more insights into the results, we also provide scores for implicit/explicit relations for some corpora, as shown in Table 6. Unexpectedly, we observe large differences in performance between explicit and implicit relations, with the latter having scores in the 40s against around 85 for the former. Some exceptions are high scores for implicit in the Portuguese CRPC and low scores for explicit in the Turkish TEDm. We also provide scores for each relation label for all corpora in Appendix C.

Conclusion

The DISRPT 2023 shared task was very challenging, with the addition of datasets from a new framework, in new languages, and 4 OOD surprise datasets without training partitions. The submitted systems still demonstrated rather high performance for EDU segmentation, with room for improvement for some corpora / languages / domains. However, further research and error analysis are needed to better understand not only what could be missing in the current models, but also what could be improved in some annotation projects, especially for example when EDU boundaries do not match sentence segmentation. Connective detection has been shown to be far from a solved task, with specific challenges for speech or dialogue data and generalizability to new domains. Finally, challenges are still significant for discourse relation classification.

Competitors proposed original and attractive strategies to combine corpora due to data scarcity, but the label set explosion is a major obstacle as well as for analyzing the results. We hope that this work will bring new research and discussion in increasing convergence and cohesion of frameworks and annotation projects. We encourage researchers in the field to use the DISRPT data as a benchmark to evaluate their systems in the future in order to provide a realistic view of the robustness and generalization ability of their approaches.

A Relation Mapping Details

Table 7 provides the mapping done for the relation labels in addition to translation to English when needed. A few cases of labels were also removed when they did not correspond to a discourse relation.

B DISRPT 2023 Corpora Statistics

Figure 1 :

 1 Figure 1: An RST Tree Example (Iruskieta et al., 2015).

Figure 2 :

 2 Figure 2: A Dependency Example (Yang and Li, 2018).

Table 8

 8

	(RST, SDRT

Table 1 :

 1

Train / Dev / Test Statistics of DISRPT 2023 Datasets: boldface indicates a new corpus compared to DISRPT 2021, * indicates a surprise or an OOD dataset. '#Docs' and '#EDUs' correspond to the total number of documents and EDUs respectively. #Conn is the number of tokens starting a connective. '#Labels' corresponds to the size of the respective label set and '#Rels' to the total number of pairs annotated.

Table 2 :

 2 Sentence Segmentation Performance: each sentence beginning is annotated as an EDU boundary, baseline for segmentation (Task 1) computed on treebanked data (left) and .tok automatically split with Trankit (right), F1 scores on the dev and test partitions. Errors are the percentage of sentence beginnings not annotated as the beginning of an EDU (so an error of the sentence splitting). Corpora with gold sentences for the treebanked track are marked with a G .

	Corpus	Treebanked: Gold / Stanza %Error F1 dev test dev test	Plain: Trankit %Error F1 dev test dev	test
	deu.rst.pcc G eng.dep.covdtb eng.dep.scidtb eng.rst.gum G eng.rst.rstdt G eng.sdrt.stac G eus.rst.ert fas.rst.prstc fra.sdrt.annodis nld.rst.nldt por.rst.cstn rus.rst.rrt spa.rst.rststb spa.rst.sctb zho.dep.scidtb zho.rst.gcdt zho.rst.sctb	0.97 0.00 0.00 0.00 0.14 0.30 3.01 0.00 6.12 12.81 57.43 49.07 0.00 85.06 84.02 0.00 59.35 57.16 0.00 55.35 55.71 0.00 60.88 60.95 0.00 56.96 56.73 0.30 92.12 92.63 4.58 68.07 68.57 0.00 51.93 56.53 0.00 0.00 85.28 83.04 0.39 0.00 57.72 62.47 21.53 19.74 59.11 59.89 27.97 25.44 57.46 58.30 0.53 0.00 81.03 79.51 0.27 0.35 58.03 55.71 0.12 0.24 55.43 55.92 0.25 1.19 60.11 59.31 1.08 1.12 57.65 58.23 3.95 3.36 62.54 57.49 7.75 7.37 69.91 69.87 1.40 2.51 53.60 57.32 1.32 1.41 57.40 50.54 2.22 4.91 86.13 83.58 0.39 0.72 57.88 61.71 0.00 0.70 75.48 76.31 0.00 0.37 72.64 73.35 3.95 3.51 81.56 78.01 4.29 3.92 77.46 72.59 0.00 0.00 50.99 54.94 0.00 0.00 50.99 54.94 0.00 0.00 44.88 46.95 0.35 0.35 39.52 41.48 6.98 7.52 84.66 81.73 8.33 9.82 75.43 72.14

  .22 96.01 97.58 95.92 96.74 96.77 91.84 94.24 **eng.dep.covdtb 94.04 90.31 92.13 90.22 90.38 90.30 94.04 90.31 92.13 eng.dep.scidtb 94.96 95.18 95.07 94.77 95.09 94.93 94.94 94.05 94.49 eng.rst.gum 94.59 96.42 95.50 95.08 95.29 95.19 94.95 93.98 94.46 eng.rst.rstdt 97.21 98.04 97.62 96.46 97.66 97.06 96.70 98.81 97.74 eng.sdrt.stac 95.75 94.70 95.22 96.71 95.09 95.89 87.92 93.60 90.67 eus.rst.ert 88.18 91.76 89.93 90.14 90.14 90.14 89.66 92.57 91.

	corpus		track: Treebanked DisCut*	HITS			track: Plain DisCut*
		P	R	F1	P	R	F1	P	R	F1
	deu.rst.pcc fas.rst.prstc fra.sdrt.annodis nld.rst.nldt por.rst.cstn rus.rst.rrt spa.rst.rststb spa.rst.sctb zho.dep.scidtb zho.rst.gcdt zho.rst.sctb	97.88 9409 94.92 91.94 93.40 92.95 92.54 92.74 93.29 93.43 93.36 88.06 88.35 88.21 88.82 87.38 88.09 91.34 90.45 90.89 98.17 94.97 96.54 93.62 91.12 92.35 97.05 97.34 97.19 93.53 94.44 93.98 93.73 92.81 93.27 93.02 95.75 94.36 84.02 87.20 85.58 83.08 87.88 85.41 83.23 87.71 85.41 92.74 94.35 93.53 91.14 91.74 91.44 92.03 95.43 93.70 86.14 85.12 85.63 84.38 80.36 82.32 82.76 85.71 84.21 83.58 95.32 89.07 84.00 98.3 90.59 84.64 96.17 90.04 91.80 93.32 92.55 89.09 92.77 90.89 90.47 93.04 91.74 79.33 84.52 81.84 78.95 80.36 79.65 73.82 83.93 78.55
	mean	91.46 92.36 91.87 90.63 91.46 91.00 90.39 92.60 91.43

Table 3 :

 3 EDU Segmentation Results on Treebanked and Plain tracks: boldface indicates a new corpus compared to DISRPT 2021, and * * a surprise and OOD dataset. Disclosure: System marked with * was submitted by a team containing organizers and annotators of shared task datasets.

Table 4 :

 4 Connective Detection Results.

	corpus		track: Treebanked DisCut*	HITS			track: Plain DisCut*
		P	R	F1	P	R	F1	P	R	F1
	eng.pdtb.pdtb **eng.pdtb.tedm 82.69 74.46 78.36 81.74 77.49 79.56 83.77 69.26 75.83 95.49 91.89 93.66 93.61 94.06 93.83 94.08 89.32 91.64 ita.pdtb.luna 60.65 72.03 65.85 62.23 66.28 64.19 66.34 77.78 71.60 por.pdtb.crpc 80.81 80.51 80.66 80.59 80.88 80.73 78.49 80.51 79.49 **por.pdtb.tedm 77.52 83.25 80.29 73.71 84.24 78.62 74.78 84.73 79.45 tha.pdtb.tdtb 84.24 87.13 85.66 85.74 87.2 86.46 85.32 59.23 69.92 tur.pdtb.tdb 92.34 93.21 92.77 92.3 95.43 93.84 90.33 91.92 91.12 **tur.pdtb.tedm 87.41 50.61 64.10 91.49 52.23 66.49 51.01 88.73 64.78 zho.pdtb.cdtb 91.25 86.86 89.00 89.26 85.26 87.21 92.03 88.78 90.38
	mean	82.64 79.14 80.17 82.68 79.73 80.47 79.57 81.14 79.36

Table 5 :

 5 Relation Classification Results on the Test Set.

	corpus	DiscRet HITS DiscoFlan
	deu.rst.pcc **eng.dep.covdtb 41.30 26.92 eng.dep.scidtb 67.56 eng.pdtb.pdtb 69.25 **eng.pdtb.tedm 19.94 eng.rst.gum 55.34 eng.rst.rstdt 49.98 eng.sdrt.stac 56.89 eus.rst.ert 51.77 fas.rst.prstc 50.34 fra.sdrt.annodis 44.96 ita.pdtb.luna 58.42 nld.rst.nldt 43.69 por.pdtb.crpc 72.76 **por.pdtb.tedm 54.95 por.rst.cstn 62.87 rus.rst.rrt 61.52 spa.rst.rststb 58.22 spa.rst.sctb 33.33 tha.pdtb.tdtb 95.24 tur.pdtb.tdb 49.05 **tur.pdtb.tedm 49.73 zho.dep.scidtb 67.44 zho.pdtb.cdtb 69.13 zho.rst.gcdt 55.72 zho.rst.sctb 49.06	31.92 13.08 69.33 50.15 74.15 34.12 74.30 24.41 64.96 33.05 68.19 22.33 65.71 36.94 60.79 22.65 56.19 28.02 56.08 25.84 51.84 19.36 65.00 22.37 51.69 29.23 78.53 43.83 64.84 29.95 68.75 38.60 60.99 23.60 57.28 26.76 61.64 44.65 95.83 34.67 45.50 25.83 54.12 25.83 67.44 33.49 59.63 59.37 56.35 20.46 60.38 43.40
	mean	54.44	62.36 31.21
	corpus	DiscReT impl expl	HITS impl expl	#impl #expl
	eng.pdtb.pdtb 42.66 75.32 57.94 87.23 1008 1159 eng.pdtb.tedm 4.80 28.06 39.20 83.16 125 196 ita.pdtb.luna 17.21 62.02 49.18 72.48 122 258 por.pdtb.crpc 18.00 72.92 71.87 88.20 711 517 por.pdtb.tedm 15.85 69.95 42.68 85.25 164 183 tur.pdtb.tedm 22.95 52.40 44.26 59.62 122 208

Table 6 :

 6 Implicit/Explicit Classification Results.

Table 7 :

 7 Relation Mapping used in DISRPT 2023.

	Corpus	Original label	Mapped label
	eus.rst.ert spa.rst.rststb fas.rst.prstc por.rst.cstn deu.rst.pcc fra.sdrt.annodis e-elab anthitesis motibation solution-hood backgroun topicomment topichange topidrift non-volitional-cause non-volitional-cause-e nonvolitional-cause-e antithesis motivation solutionhood background topic-comment topic-change topic-drift nonvolitional-cause non-volitional-result nonvolitional-result non-volitional-result-e nonvolitional-result-e e-elab e-elaboration e-elaboration
	nld.rst.nldt eng.dep.scidtb ita.pdtb.luna	span null null	relation removed relation removed relation removed

  Table8provides detailed statistics on all DISRPT 2023 corpora regarding their sizes and properties.

	Corpus	Domain	mwt #Docs #Sents	#Tokens Vocab #EDUs #Conn #Labels #Rels References
		Tasks 1 and 3: EDU Segmentation and Relation Classification
	deu.rst.pcc	newspaper commentaries	n	176 2, 193	33, 222 8, 359 3, 018	-	26	2, 665 Potsdam Commentary Corpus
	**eng.dep.covdtb	scholarly paper abstracts on COVID-19 and related coronaviruses	y	300 2, 343	60, 849 8, 293 5, 705	-	12	(Stede and Neumann, 2014) 4, 985 COVID-19 Discourse Depen-dency Treebank (COVID19-
										DTB) (Nishida and Matsumoto,
	eng.dep.scidtb	scientific articles	y	798 4, 202 102, 493 8, 700 10, 986	-	24	2022) 9, 904 Discourse Dependency TreeBank for Scientific Abstracts (SciDTB)
	eng.rst.gum	multi-genre	y	213 11, 656 203, 879 19, 404 26, 252	-	14	(Yang and Li, 2018) 24, 688 Georgetown University Multi-
	eng.rst.rstdt	news	y	385 8, 318 205, 829 19, 160 21, 789	-	17	layer corpus V9 (Zeldes, 2017) 19, 778 RST Discourse Treebank (Carl-
	eng.sdrt.stac	dialogues	y	45 11, 087	52, 354 3, 967 12, 588	-	16	son et al., 2001) 12, 235 Strategic Conversations corpus
	eus.rst.ert	medical, terminological and scientific	n	164 2, 380	45, 780 13, 662 4, 202	-	29	(Asher et al., 2016) 3, 825 Basque RST Treebank (Iruskieta
	fas.rst.prstc	journalistic texts	y	150 2, 179	66, 694 7, 880 5, 853	-	17	et al., 2013) 5, 191 Persian RST Corpus (Shahmo-
	fra.sdrt.annodis	news, wiki	n	86 1, 507	32, 699 7, 513 3, 429	-	18	hammadi et al., 2021) 3, 338 ANNOtation DIScursive (Afan-
	nld.rst.nldt	expository texts and persuasive genres	n	80 1, 651	24, 898 4, 935 2, 343	-	32	tenos et al., 2012). 2, 264 Dutch Discourse Treebank (Re-
	por.rst.cstn	news	y	140 2, 221	58, 793 7, 786 5, 537	-	32	deker et al., 2012) 4, 993 Cross-document Structure The-
										ory News Corpus (Cardoso et al.,
	rus.rst.rrt	blog and news	n	332 23, 044 473, 005 75, 285 41, 542	-	22	2011) 34, 566 Russian RST Treebank (Toldova
	spa.rst.rststb	multi-genre	n	267 2, 089	58, 717 9, 444 3, 351	-	28	et al., 2017) 3, 049 RST Spanish	Treebank
	spa.rst.sctb	multi-genre	n	50	516	16, 515 3, 735	744	-	25	(da Cunha et al., 2011) 692 RST Spanish-Chinese Treebank
	zho.dep.scidtb	scientific	n	109	609	18, 761 2, 427 1, 407	-	23	(Spanish) (Cao et al., 2018) 1, 298 Discourse Dependency TreeBank for Scientific Abstracts (SciDTB)
										(Yi et al., 2021; Cheng and Li,
	zho.rst.gcdt	multi-genre	n	50 2, 692	62, 905 9, 818 9, 706	-	31	2019) 8, 413 Georgetown Chinese Discourse Treebank (GCDT) (Peng et al.,
	zho.rst.sctb	multi-genre	n	50	580	15, 496 2, 973	744	-	26	2022b,a) 692 RST Spanish-Chinese Treebank
										(Chinese) (Cao et al., 2018)
		Tasks 2 and 3: Connective Detection and Relation Classification
	eng.pdtb.pdtb	news	y 2, 162 48, 630 1, 156, 657 48, 937	-26, 048	23	47, 851 Penn	Discourse	Treebank
	**eng.pdtb.tedm TED talks	y	6	381	8, 048 1, 881	-341	20	(Prasad et al., 2014) 529 TED-Multilingual Discourse Bank (English) (Zeyrek et al.,
	ita.pdtb.luna	speech	y	60 3, 753	26, 114 2, 392	-1, 071	16	2018, 2019) 1, 547 LUNA Corpus Discourse Data Set (Tonelli et al., 2010; Riccardi
	por.pdtb.crpc 5	news, fiction, and didactic/scientific texts n	302 5, 194 186, 849 22, 208	-5, 159	22	et al., 2016) 11, 330 Portuguese Discourse Bank (CRPC) (Mendes and Lejeune,
	**por.pdtb.tedm TED talks	n	6	394	8, 190 2, 162	-305	20	2022; Généreux et al., 2012) 554 TED-Multilingual Discourse Bank (Portuguese) (Zeyrek et al.,
	*tha.pdtb.tdtb tur.pdtb.tdb	news multi-genre	n y	180 6, 534 256, 523 11, 789 197 31, 196 487, 389 88, 923	-10, 864 -8, 748	21 23	2018, 2019) 10, 865 Thai Discourse (TDTB) 3, 185 Turkish Discourse Bank (Zeyrek Treebank
										and Webber, 2008; Zeyrek and
	**tur.pdtb.tedm TED talks	y	6	410	6, 143 2, 771	-382	23	Kurfalı, 2017) 577 TED-Multilingual Discourse Bank (Turkish) (Zeyrek et al.,
	zho.pdtb.cdtb	news	n	164 2, 891	73, 314 9, 085	-1, 660	9	2018, 2019) 5, 270 Chinese Discourse Treebank
										(Zhou et al., 2014)

Table 8 :

 8 General Statistics of DISRPT 2023 Datasets: boldface indicates a new corpus compared to DISRPT 2021, * indicates a surprise dataset and * * a surprise and OOD dataset. 'mwt' corresponds to the annotation ('y') or not ('n') of multi-word expressions. '#Docs', '#Sents', '#Tokens' and '#EDUs' correspond to the total number of documents, sentences (the Treebanked track), tokens, and EDUs respectively. #Conn is the number of tokens starting a connective. 'Vocab' is the number of unique tokens. '#Labels' corresponds to the size of the respective label set and '#Rels' to the total number of pairs annotated.

	C Relation Scores Per Label	eng.pdtb.pdtb	P	HITS R	F1	P	DiscReT R	F1 Num.
	Tables below provide a detailed breakdown of the accuracy scores for each corpus and each label for the discourse relation classification task (i.e. Task 3). The results of the HITS and the DiscReT sys-tems are presented. HITS DiscReT deu.rst.pcc P R F1 P R F1 Num. antithesis 36.36 22.22 27.59 16.67 5.56 8.33 1800 background 16.67 11.76 13.79 0.00 0.00 0.00 1700 cause 25.00 100 40.00 0.00 0.00 0.00 200 circumstance 33.33 6.67 11.11 42.86 20.00 27.27 1500 concession 31.25 38.46 34.48 30.77 30.77 30.77 1300 condition 58.33 77.78 66.67 0.00 0.00 0.00 900 conjunction 33.33 57.14 42.11 0.00 0.00 0.00 700 contrast 16.67 12.50 14.29 0.00 0.00 0.00 800 e-elaboration 69.23 81.82 75.00 60.00 54.55 57.14 1100 elaboration 24.00 60.00 34.29 8.33 20.00 11.76 1000 evaluation-n 0.00 0.00 0.00 0.00 0.00 0.00 300 evaluation-s 0.00 0.00 0.00 0.00 0.00 0.00 1700 evidence 50.00 20.00 28.57 0.00 0.00 0.00 1000 interpretation 0.00 0.00 0.00 11.11 41.67 17.54 1200 joint 14.29 13.79 14.04 8.33 6.90 7.55 2900 list 42.42 53.85 47.46 59.09 50.00 54.17 2600 means 100 50.00 66.67 0.00 0.00 0.00 200 preparation 28.57 50.00 36.36 14.29 25.00 18.18 400 purpose 100 100 100 50.00 66.67 57.14 300 reason 52.00 38.24 44.07 43.33 38.24 40.62 3400 restatement 0.00 0.00 0.00 0.00 0.00 0.00 100 sequence 75.00 42.86 54.55 0.00 0.00 0.00 700 solutionhood 0.00 0.00 0.00 0.00 0.00 0.00 100 summary 0.00 0.00 0.00 0.00 0.00 0.00 100 macro avg 33.60 34.88 31.29 14.37 14.97 13.77 26000 weighted avg 33.51 31.92 30.72 21.84 20.00 19.88 26000 HITS DiscReT eng.dep.covdtb P R F1 P R F1 Num.	Comparison. Concession Comparison. Concession+ SpeechAct Comparison. Contrast Comparison. Similarity Contingency.Cause Contingency.Cause+ Belief Contingency.Cause+ SpeechAct Contingency. Condition Contingency. Condition+ SpeechAct Contingency. Negative-cause Contingency. Negative-condition Contingency.Purpose 73.33 71.74 72.53 78.95 65.22 71.43 4600 85.71 75.22 80.13 74.78 76.12 75.44 33500 0.00 0.00 0.00 0.00 0.00 0.00 200 65.62 62.69 64.12 68.48 47.01 55.75 13400 90.00 81.82 85.71 66.67 72.73 69.57 1100 76.96 68.27 72.36 67.59 70.19 68.87 41600 11.11 13.33 12.12 0.00 0.00 0.00 1500 0.00 0.00 0.00 0.00 0.00 0.00 300 81.93 89.47 85.53 84.85 73.68 78.87 7600 25.00 12.50 16.67 0.00 0.00 0.00 800 100 100 100 0.00 0.00 0.00 100 100 100 100 100 50.00 66.67 200 Expansion. 74.50 88.57 80.93 79.96 78.95 79.45 55100 Conjunction Expansion. 64.29 100 78.26 0.00 0.00 0.00 900 Disjunction Expansion. 30.77 16.67 21.62 0.00 0.00 0.00 2400 Equivalence Expansion.Exception 0.00 0.00 0.00 0.00 0.00 0.00 100 Expansion. 71.84 74.75 73.27 72.92 70.71 71.79 9900 Instantiation Expansion.Level-of-63.33 56.72 59.84 0.00 0.00 0.00 20100 detail Expansion.Manner 69.23 90.00 78.26 71.15 92.50 80.43 4000 Expansion. 86.67 66.67 75.36 85.71 46.15 60.00 3900 Substitution Hypophora 72.73 100 84.21 0.00 0.00 0.00 800 Temporal. 81.89 74.29 77.90 81.51 69.29 74.90 14000 Asynchronous Temporal. Synchronous 67.24 81.25 73.58 0.00 0.00 0.00 9600
	ATTRIBUTION 92.52 96.12 94.29 95.28 98.06 96.65 10300 BACKGROUND 61.02 82.44 70.13 0.00 0.00 0.00 13100 CAUSE-RESULT 57.73 41.48 48.28 0.00 0.00 0.00 13500 COMPARISON 83.33 13.07 22.60 84.00 13.73 23.60 15300 CONDITION 65.00 59.09 61.90 0.00 0.00 0.00 2200 ELABORATION 80.27 81.26 80.77 85.25 46.34 60.04 129700 ENABLEMENT 93.17 86.43 89.67 97.03 44.34 60.87 22100 FINDINGS 0.00 0.00 0.00 0.00 0.00 0.00 15400 JOINT 58.96 90.29 71.33 33.15 67.43 44.44 17500 MANNER-MEANS 80.43 64.35 71.50 83.64 40.00 54.12 11500 TEMPORAL 64.52 80.00 71.43 75.00 12.00 20.69 2500 TEXTUAL-ORGANIZATION 0.00 0.00 0.00 0.00 0.00 0.00 5500 macro avg 61.41 57.88 56.82 46.11 26.82 30.03 258600 weighted avg 71.69 69.33 68.56 66.50 38.21 46.17 258600 HITS DiscReT eng.dep.scidtb P R F1 P R F1 Num. attribution 94.78 96.95 95.85 95.38 94.66 95.02 13100 bg-compare 82.35 60.87 70.00 60.00 39.13 47.37 4600 bg-general 71.74 89.19 79.52 87.18 91.89 89.47 3700 bg-goal 52.11 66.07 58.27 33.33 62.50 43.48 5600 cause 33.33 36.36 34.78 0.00 0.00 0.00 1100 comparison 57.89 52.38 55.00 92.86 61.90 74.29 2100 condition 83.33 60.61 70.18 0.00 0.00 0.00 3300 contrast 72.29 84.51 77.92 0.00 0.00 0.00 7100 elab-addition 77.65 75.15 76.38 78.76 69.94 74.09 65200 elab-aspect 18.67 31.11 23.33 11.20 31.11 16.47 4500 elab-definition 20.00 25.00 22.22 0.00 0.00 0.00 400 elab-enumember 85.71 62.07 72.00 71.43 68.97 70.18 2900 elab-example 78.26 52.94 63.16 88.89 47.06 61.54 3400 elab-process_step 52.00 44.83 48.15 40.00 48.28 43.75 2900 enablement 77.04 81.89 79.39 79.67 77.17 78.40 12700 evaluation 81.62 84.83 83.20 71.26 69.66 70.45 17800 exp-evidence 70.00 53.85 60.87 0.00 0.00 0.00 1300 exp-reason 91.67 78.57 84.62 77.78 50.00 60.87 1400 joint 83.77 82.69 83.23 74.05 87.82 80.35 15600 manner-means progression result summary temporal 55.56 86.96 67.80 60.87 60.87 60.87 2300 0.00 0.00 0.00 0.00 0.00 0.00 100 30.77 25.81 28.07 0.00 0.00 0.00 3100 42.11 33.33 37.21 12.50 2.08 3.57 4800 86.61 80.17 83.26 90.57 79.34 84.58 12100	weighted avg macro avg eng.pdtb.tedm Comparison. Concession Comparison. Contrast Comparison. Similarity Contingency.Cause Contingency.Cause+ Belief Contingency.Cause+ SpeechAct Contingency. Condition Contingency.Purpose 46.15 75.00 57.14 100 12.50 22.22 800 74.28 74.30 73.88 63.02 60.35 61.37 225700 60.53 61.91 60.54 40.55 35.33 37.09 225700 HITS DiscReT P R F1 P R F1 Num. 62.16 88.46 73.02 52.38 42.31 46.81 2600 80.00 30.77 44.44 50.00 15.38 23.53 1300 50.00 28.57 36.36 0.00 0.00 0.00 700 65.71 43.40 52.27 46.67 13.21 20.59 5300 0.00 0.00 0.00 0.00 0.00 0.00 600 0.00 0.00 0.00 0.00 0.00 0.00 200 76.47 81.25 78.79 100 12.50 22.22 1600 Expansion. 65.96 80.17 72.37 76.00 32.76 45.78 11600 Conjunction Expansion. 100 100 100 0.00 0.00 0.00 200 Disjunction Expansion. 0.00 0.00 0.00 0.00 0.00 0.00 600 Equivalence Expansion. 100 33.33 50.00 0.00 0.00 0.00 900 Instantiation Expansion.Level-of-47.37 62.07 53.73 0.00 0.00 0.00 2900 detail Expansion.Manner 100 66.67 80.00 100 33.33 50.00 600 Expansion. 75.00 90.00 81.82 100 10.00 18.18 1000 Substitution Hypophora 100 66.67 80.00 0.00 0.00 0.00 600 Temporal. 66.67 63.64 65.12 33.33 4.55 8.00 2200 Asynchronous Temporal. Synchronous 83.33 71.43 76.92 0.00 0.00 0.00 1400 weighted avg 64.93 64.96 62.99 51.38 18.52 25.97 35100 macro avg 62.16 54.52 55.67 36.58 9.81 14.30 35100
	weighted avg macro avg	75.22 74.15 74.35 67.40 63.89 64.92 191100 62.47 60.26 60.60 46.91 43.43 43.95 191100

https://sites.google.com/view/disrpt2019

https://sites.google.com/georgetown.edu/ disrpt2021

For the ita.pdtb.luna, containing dialogue transcription, we rather use an 'utterance' unit that corresponds to a sequence of speech between two silences.

We found that the annotation was faulty in the Thai corpus, where discontinuous connectives were annotated as one single chain of 'B/I' labels, thus allowing 'I' labels to appear with no immediately preceding 'B'. This annotation will be corrected in the GitHub repository (https://github. com/disrpt/sharedtask2023/) for future use, and we will indicate the possible impact on the scores.

In this version of the corpus, 15 documents are missing compared to the original dataset due to pre-processing issues.

Acknowledgements

This work is partially supported by the AnDiaMO project (ANR-21-CE23-0020) and the ANR (ANR-19-PI3A-0004) through the AI Interdisciplinary Institute, ANITI, as part of France's "Investing for the Future -PIA3" program.

This work is also partially supported by the SLANT project (ANR-19-CE23-0022) and the ANR grant SUMM-RE (ANR-20-CE23-0017). Chloé Braud and Philippe Muller are part of the programme DesCartes and are also supported by the National Research Foundation, Prime Minister's Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme.