Alexis Grélois

Séverine Delahaye-Grélois

Krijn

lost, Martène contains numerous misspellings and offers only a limited number of statutes. 9 Canivez, like most editors of his time, made critical choices between the various manuscript versions, so that some manuscripts are under-used and others entirely overlooked. 10 In many cases, the version given by the later manuscript was preferred over the medieval one, 11 an error already committed by Leopold Janauschek with his list of Cistercian monasteries. 12 Moreover, Canivez's desire to provide a complete collection of the statuta was so strong that he quoted Angel Manrique's Annals 13 and other printed sources not only to assign a year to undated statutes (the so-called 1134 statutes collection), but also to offer knowledge on sessions that had left no material evidence. 14 Canivez also quoted Winter's book, which published decisions of the general chapters extracted from German archives and chronicles. 15 The result is a pot-pourri that is incoherent, especially where statutes of chapters held in Cîteaux or in Germany, during the Western Schism when the order was divided, are printed together. 16 For all these reasons, Canivez's edition gives a false appearance of completeness and unity. 17 10 Twelfth-Century Statutes, ed. Waddell, p. 7, 23. Canivez's palaeographic skills seem questionable since he placed in the fifteenth century a seventeenth-century collection (today Lille, Archives Départementales du Nord, 28 H 70) (Statuta 1, ed. Canivez, p. xvii; Twelfth-Century Statutes, ed. Waddell, p. 165). See also Cygler, Das Generalkapitel, pp. 86-87. 11 For instance, in a 1221 statute, Canivez's edition (Statuta 2, ed. Canivez, p. 1) follows Martène's and six seventeenth-and eighteenth-century manuscripts saying that each father-abbot is to determine the number of people allowed to stay in his daughter houses according to the quality of their possessions (Committur patribus abbatibus ut in filiabus suis secundum modum et qualitatem possessionum ordinent, auctoritate Capituli, de numero personarum [bold AG]), whereas two early thirteenth-century manuscripts (Dijon, Bibliothèque Municipale, 601, and Mons, Bibliothèque de l'Université de Mons-Hainaut, 31/192) indicate that the quantity of wealth is to be taken in account (secundum modum et quantitatem possessionum). The General Chapter's preparatory commission took the same decision in 1190 (Twelfth-Century Statutes, ed. Waddell, p. 188 #16). 12 Leopold Janauschek, Originum Cisterciensium 1 (Vienna: A. Hoelder, 1877). 13 Ángel Manrique, Cisterciensium seu verius ecclesiasticorum annalium a condito Cistercio… (Lyon: G. Boissat and L. Anisson, 1642-1659). 14 Statuta, ed. Canivez, pp. 1-45, 53-56, 59, 73-81, 85-86. 15 For instance Statuta 3, ed. Canivez, p. 502 #5 (year 1345) and 544 #8 (1366). 16 Statuta 3, ed. Canivez, 3, p. 637 #10 (1393). 17 Moreover many identifications of monasteries in Canivez's edition are erroneous, especially where he was dealing with facts that did not answer to his conceptions of the order's history. For example, because he did not accept that a Cistercian double monastery could exist, he understood a 1212 statute regarding a female community in Brolium as a mention of the La Brayelle nunnery in northern France, but, since the case was committed to the abbot of Savigny, it was really the abbey of Breuil-Benoît in Normandy (Statuta 1, ed. Canivez, p. 403 #62). Claude Evans also remarks that Canivez (Statuta 1, ed. Canivez, p. 344 #57) misinterpreted the newly-incorporated Begardum quoted in a 1207 statute as the Moravian abbey of Welehrad, because he read in Janauschek's catalogue that the Breton monastery of Bégard entered the order in 1130 (Claude Evans, L'abbaye cistercienne de Bégard des origines à 1478. Histoire et chartes (Turnhout: Brepols, 2012), p. 59).

These imperfections, and the discovery of many materials regarding the early Cistercian legislation, led the Trappist Father Chrysogonus Waddell to provide a new edition of the twelfth-century statutes. 18 His method was entirely new, in the sense that he separated the manuscript traditions into three categories according to their contents: annual statutes, compilations, and "local collections." Waddell gave a separate edition of each. One cannot help but be impressed by the sum of his erudition. Still, as Waddell recognized in his introduction, it is not the definitive edition: some conclusions need further discussion. 19 Above all, Waddell had no time to collect the decisions that are kept in the archives of individual houses but were not adopted into the statuta collections. 20 In short, a complete and definitive collection of the general chapter appears as a Cistercian Grail, always yearned for, sometimes approached, but never truly possessed. We all know that one learns much on such a quest, but the question is: what does one learn from it? Regarding the Cistercian statutes, we should face various issues: What are the materials available for such an edition? Are they sufficient for a good edition? If not, what does it teach us about Cistercian practices and about the directions in which Cistercian studies should turn?

A complete collection?

A classification of the manuscripts and early printed editions used by Canivez and Waddell can be based on their contents, their dates, and their authors. The organization of manuscripts allows us to distinguish three categories; the first two consisting of more or less coherent compilations. Some of these are official collections: the undated Statuta, 21 A first problem is that many years of this "complete collection" are missing. In the early 1730s, Cîteaux was still in possession of various manuscripts containing the statutes.

The first volume covered the period 1190-1282, but the second began only in 1387. 49 This gap can be partially filled with the copy from Valloire for years 1285-1298 and by two fourteenthcentury manuscripts from Clairvaux that offer the complete 1344 statutes and the first ones of 1352 in a very organized form. 50 This means that before 1190, in 1283 and 1284, between 1299 and 1343 and from 1345 to 1388, only abstracts are available. [START_REF]The only known statute for 1363 is[END_REF] The worst period goes from 1345 to 1386: the very scarce manuscripts offer only general decisions, if any, and there is no record at all for the chapters of 1364, 1376, 1378 to 1386, 1414 and 1415. One can answer that since continental monastic historians usually forget the fourteenth century ever existed, the loss is not tragic. The widespread belief that the statuta form a uniform collection has led many scholars to overestimate their meaning. Like Winter, the majority of scholars have argued (as some still do) that the order was closed to nuns until the early thirteenth century. 54 How can we explain these gaps in the documentation? We must investigate the motivations of the scribes, their patrons and scholars involved in the manuscript and printed tradition.

Regarding the "complete corpus," the gap between archival realities and collections may be explained by the definitory's process: it met after the general sessions and the scribes could decide to overlook definitive decisions (that would later be transmitted to fathers immediate 59 various individual statutes into a single, general one. 63 Quite often, their formulations do not match those of the "complete collection," as is the case with this 1195 statute prohibiting Cistercians and Carthusians to receive a member of the other order without its authorisation: 64 59 Among Cistercians, the father immediate of a community is the abbot of its mother-house. The transmission process may explain those differences. Around 1179, twelve abbots were commissioned to compile the 1157-1179 statutes collection. In 1185, the chapter elected a board of four abbots to "make" (fiunt) the definitions, and the definitorium was instituted in 1190. In 1224, the general chapter decided that its statutes must be known in every house of the order with the very wording the definitors had written in their book. 65 This is another expression of the famous Cistercian unanimity of spirit, but the "local collections" show that this claim was never fulfilled. This should not come as a surprise: even Clairvaux had its own lay brothers' customs 66 and, according to a thirteenth-century manuscript, Cîteaux had liturgical customs that were slightly different from the official Ecclesiastica officia! 67

The question remains how the various houses received notice of the new statutes.

Father Waddell expressed his disagreement with Father Lekai's opinion that, at least in the twelfth century, the statutes were passed orally, 68 but his demonstration is not entirely convincing. It is true that, according to the 1212 statutes he quotes, each house was supposed to have the year's general definitions at its disposal. Since the visitors were to check this, it means that each community was intended to have a written collection of general statutes in their official formulation. [START_REF]Quoniam aliqui abbates dicuntur minus esse solliciti quam deberent definitiones Capituli generalis secum deferre ad domos proprias, districte praecipitur, ut singuli abbates ad Capitulum venientes definitiones generales habere satagant, in redditu suo in capitulis recitandis. Visitatores vero, tempore visitationis suae, definitiones illius anni sibi exigant praesentari. Illis autem abbatibus qui, illo anno, non venerint ad Capitulum, patres abbates per se vel[END_REF] The real collections prove, however, that this was seldom, if ever, the case. In the thirteenth century, the definitory finished its work after most of the abbots had left Cîteaux; [START_REF] Mahn | L'ordre cistercien[END_REF] consequently, these abbots were unable to get the official collection right away and they had to wait for visitors to bring them. The hypothesis of an oral transmission is, therefore, not to be rejected entirely. The same 1212 statute attests that abbots had to recite (recitandis) the statutes to their community and to their daughter houses; many of them probably relied on the medieval art of memory, long since forgotten. In any case, a careful study of the Bujedo collection shows that it probably depends on a lost manuscript: one cannot imagine, otherwise, why a Spanish abbot would have dictated to his scribe a definition regarding some Burgundian mills. [START_REF]Twelfth-Century Statutes[END_REF] The most reliable hypothesis is that at least some abbots took notes during the chapter's sessions and the definitory's meetings. This can explain why the Clairvaux archives still contain outstanding transcriptions of the 1344 and 1352 chapters and why the Clairvaux version of the 1195 statute quoted above is more precise than the Cîteaux one, adding a reference to the commendatory letters mentioned by Benedict's Rule.

With this example, we are now facing the centuries long rivalry between Cîteaux and its second daughter which began at the time of Bernard 72 and ended only with the dissolution of both houses. In the early 1260s, an open conflict broke out regarding the nomination of the definitors, the abbot of Cîteaux trying to marginalize Clairvaux and its daughter houses. [START_REF] Mahn | L'ordre cistercien[END_REF] The careful organization of general and particular decisions in the 1344 parchment shows with what care Clairvaux fulfilled its role in the definition of Cistercian rules; it is also a testimony of its strong will to occupy the second place in the order, if not the first.

The preservation of the "complete" collection in later times can also be explained by rivalries within the Cistercian family or efforts to restore peace. One of the best collections we have at our disposal comes from the Feuillants order; its founder had left the Cistercian order to fulfil his reformation desire, but his successors stressed their bernardine identity. 74 In more recent times, editing the complete collection, which was intended to replace Martène's was a subject of rivalry between the two main Cistercian orders, the Common observance and the Trappist. The latter won the race with Canivez, but the efforts of this author would not have been necessary if Janauschek had not died before having achieved the same task, and while Father Trilhe was beginning the Belgium edition. Gregor Müller, the great historian of the Common observance, undertook the same project and refused to share the Wettingen manuscript with the Trappist Fathers. 75 No wonder the German Cistercians were the first to point out Canivez's errors! 76

Statutes and Codifications

The manuscript tradition shows a trend for annual statutes to enter into codifications, these being either official or local. It proves, therefore, that annual collections were not intended to be kept by every monastery, but rather that they were provisional documents designed to complete the existing codification before the promulgation of a new one. More than definitive Cistercian rules (which the codifications provided), the Statuta offer Cistercian rules in the making. The abbots often changed their minds because decisions proved inefficient or because they experienced a strong pressure to return to previous dispositions. In some cases, the general chapter faced a resistance and had to pull back: in 1242, the chapter ordered that nuns were to pronounce the name of their father immediate instead of their abbess's name in their vow of obedience during the profession. 77 This was clearly an attempt to lower the abbesses' motherhood and to strengthen the abbots' authority, seventeen years after the general chapter had begun the dissolution of filiation networks between female monasteries 78 .

But when the visitors (inspectors) announced the 1242 statute to the nuns, they faced an immediate uprising: in one particular nunnery the sisters organized a demonstration in the cloister; in other houses, visitors found the doors closed and were even attacked by armed nuns. 79 During their next chapter, the abbots decided to mitigate their decision, adding the abbess' name to the abbots' in the profession. 80 But that was not enough and the 1244 meeting had to capitulate; the nuns would promise obedience solely to their abbess. 81 The same chapter pronounced exclusion of rebel nunneries but this decision had no consequences. [START_REF]But the abbots of L'Aumône and Cîteaux confirmed transactions lead by Romorantin's nuns in 1251, 1252 and 1261 (Cartulaire de l'abbaye royale du Lieu-Notre-Dame-lès-Romorantin[END_REF] It is interesting to observe that in some cases codifications are more liberal than the statutes. A statute promulgated between 1162 and 1179 forbade male houses to have women milk the monks' cows, [START_REF]Twelfth-Century Statutes[END_REF] but the 1237 and 1257 codifications allowed this practice in countries where lay brothers refused to milk, if it took place outside the granges' precincts. [START_REF] Codifications | Lucet, p. 321. Men refused to milk in seventeenth-century England (Michael Roberts, "Sickles and Scythes: Women's Work and Men's Work at Harvest Time[END_REF] These examples show that historians should not base research upon isolated statutes;

on the contrary, they need to place each statutum in a wider context that involves not only the complete statuta corpus, but also codifications and other materials taken from individual houses' archives. Though a new edition of the thirteenth and fourteenth-century statutes would be a valuable contribution to the Cistercian studies, it should not be forgotten that the general chapter's statutes do not embrace the full reality of the white monks' history.

9

  Twelfth-Century Statutes From the Cistercian General Chapter. Latin Text with English Notes and Commentary, ed. by Chrysogonus Waddell. Cîteaux. Commentarii Cistercienses. Studia et documenta 12 (Cîteaux: Brecht, 2002), p. 22.

  abbeys out of the Cîteaux obedience, the order clearly took the opportunity offered by the 1385 Leulinghem treaty to begin a general reform in France and Spain.61 If we now turn back to "local collections," they only deal, as stated before, with general affairs. No wonder the monks of Vauclair or Alcobaça were not interested in the bad habits of the Irish or the Welsh. Even Cîteaux kept abridged compilations of annual statutes, in which it was easier and faster to find general decisions. The text offered by the example from Cîteaux is very close to the "complete collection," 62 but abridged series from Sept-Fons, Clairvaux, Vauclair, Alcobaça and especially Bujedo, have very different versions. They are obviously compilations, since their internal organizations are diverse (they ignore the annual classification) and the formulation of individual statutes often differs from the Cîteaux version. They are sometimes shorter and in some cases the Bujedo manuscript summarizes

the Instituta

  With the Exordium Cistercii and the Summa Cartae caritatis, the Capitula forms an abridged version of the earlier Cistercian customary, which is lost; I agree with Christopher Holdsworth that it was probably never actually written ("Narrative and Legislative Texts from Early Cîteaux: a Review Article", Cîteaux. Commentarii Cistercienses, 52 (2001), pp. 157-166 (159)). The date of the Exordium Cistercii is still a disputed matter: most scholars suggested 1123-1124 while Waddell thought 1136, but both hypothesis are fragile, since the first relies on a calculation of the number of Cistercian abbots at the time of the writing based generalis capituli, 22 the so-called 1157 collection (in fact a compilation for years 1157-1190)23 and the later codifications published by the general chapter itself in 1202, 1220, 24 1237, 1257, 25 1289, 1316 and 1350. 26 Some other compilations can be found only in one manuscript, a fact that led Waddell to consider them "local collections" (he published the twelfth-century statutes from Clairvaux, Igny, Signy, Vauclair, Tre Fontane, Bujedo and Alcobaça manuscripts 27 ). A third group contains statutes dated and written annually in the order of their publication by the chapter. It should be noted that the only annual statutes surviving in medieval manuscripts are additions to former compilations:

	18 Twelfth-Century Statutes, ed. Waddell. For an introduction to this edition, see also Chrysogonus Waddell,
	"Toward a New Provisional Edition of the Statutes of the Cistercian General Chapter, c. 1119-1189", in
	Studiosorum Speculum: Studies in Honor of Louis J. Lekai, O. Cist., ed. by Francis R. Swietek & John R.
	Sommerfeldt (Kalamazoo: Cistercian Publications, 1993), pp. 389-419.
	19 I do not follow Waddell (Twelfth-Century Statutes, pp. 53-64) when he describes Paris, BnF, latin 12169, fol.
	115 as "the earliest recoverable statuta" although undated; it seems more likely that it is the earliest compilation,
	prior to the Capitula, because it deals only with general matters in a very organized form (General Chapter,
	relationships between abbeys, community life).
	20 One exception is a statute from Winter. See Twelfth-Century Statutes, ed. Waddell, p. 501 #53 (year 1201).
	21 Father Waddell published twice his edition of the main twelfth-century statutes collections. For the Capitula,
	see Narrative and Legislative Texts from Early Cîteaux, ed. by Chrysogonus Waddell. Cîteaux. Commentarii
	Cistercienses. Studia et documenta 9 (Cîteaux: Brecht, 1999), pp. 186-191, and Twelfth-Century Statutes, ed.
	Waddell, pp. 512-516.

Table 1 :

 1 Year by year statutes in medieval manuscripts The reason is that the medieval witnesses, even when they mention years, deal mostly with general decisions. The expression "local collections", used by Waddell, is misleading: these collections are only "local" in the sense that they were written far from Cîteaux and that each provides particular formulations and singular collections of the Cîteaux statutes; they do not deal with local issues.29 The main point of interest for the authors

	Manuscript	Origin	Statutes years	Add-on to
	Montpellier, École de	Clairvaux	1157-1161	Capitula
	médecine, H 322 28
	Dijon, Bibliothèque	Sept-Fons or	1180-1220	"1157" Collection
	Municipale, 601	Cîteaux?
	San Isidro Abbey, 1	Bujedo	1174-1196, 1209-	Instituta Capituli
			1227	Generalis
	Mons, Bibliothèque de Cambron	1204-1231	1202 Codification

on the erroneous list of Janauschek (which includes abbeys that were not incorporated at that time, notably Cadouin in Périgord, see Alexis Grélois, "L'expansion cistercienne en France: la part des affiliations et des moniales", in Norm und Realität. Kontinuität und Wandel der Zisterzienser im Mittelalter, ed. by Franz J. Felten & Werner Rösener. Vita regularis. Abhandlungen 42 (Münster: Lit Verlag, 2009), pp. 287-324 (289), and Alexis Grélois, "Au-delà des catalogues: pour une étude à frais nouveaux de l'expansion cistercienne dans la France de l'Ouest", Annales de Bretagne et des pays de l'Ouest, 120 (2013), pp. 171-187), while Waddell's demonstration is based on stylistic approximations (Holdsworth "Narrative", pp. 162-165).

22

Narrative and Legislative Texts, ed. Waddell, pp. 319-368, and Twelfth-Century Statutes, ed. Waddell, pp. 532-565. In fact, there are several versions of this compilation which was begun before 1152 and finished in the early 1180s (Narrative and Legislative Texts, ed. Waddell, pp. 310-318). It is a general assumption that this new collection was set up to provide customaries to the numerous 1147 incorporated abbeys (Holdsworth "Narrative", p. 160).

23

Twelfth-Century Statutes, ed. Waddell, pp. 572-606. One can wonder why two statute compilations (Instituta generali capitulis and "1157") existed for the same period.

24

La codification cistercienne de 1202 et son évolution ultérieure, ed. by Bernard Lucet (Rome: Editiones cistercienses, 1964). 25 Les codifications cisterciennes de 1237 et de 1257, ed. by Bernard Lucet (Paris: C.N.R.S., 1977). 26 We still rely on the early modern edition by Julien Paris reprint in Nomasticon Cisterciense seu antiquiores ordinis Cisterciensis Constitutiones…, ed. by Hugues Séjalon (Solesmes: typographia Sancti Petri, 1892), pp. 366-471, 497-536. 27 Twelfth-Century Statutes, ed. Waddell, pp. 609-750. The Igny collection (Twelfth-Century Statutes, ed. Waddell, pp. 609-613) contained in a since lost manuscript was first published by Martène; the same compilation can be read in Dijon, Bibliothèque municipale, 601, whose origins (Sept-Fons or Cîteaux) is uncertain (Twelfth-Century Statutes, ed. Waddell, p. 79; Narrative and Legislative Texts, ed. Waddell, pp. 35-37). 28 La législation cistercienne abrégée du manuscrit de Montpellier H 322, ed. by Louis Duval-Arnould (Paris: Champion, 1997); Twelfth-Century Statutes, ed. Waddell, pp. 65-75.

of these manuscripts were changes in liturgy, followed by visitation process, filiation, ranks and discipline.

In fact, Canivez found local affairs almost exclusively in seventeenth-and eighteenthcentury copies. Some of these collections are excerpta (extracts) very close to the medieval collections, 30 but other, early modern, copies are much richer. Canivez called them collectiones completissimae.

31

Table 2 :

 2 Early modern statutes collections With very few exceptions like a statute on the consequences of war in Gascony in the Portuguese compilation of Alcobaça (Twelfth-Century Statutes, ed. Waddell, p. 693 #108) and another prohibiting the sale of Saracens in a customary from the Spanish abbey of Bujedo (Twelfth-Century Statutes, ed. Waddell, p. 711 #18). These exceptions are easily explained by the localization of both monasteries. 30 Paris, Bibliothèque de l'Arsenal 783 from Les Feuillants and Troyes, Médiathèque, 1285 from Clairvaux are most likely copies of the thirteenth-century Dijon, Bibliothèque Municipale, 601 (Twelfth-Century Statutes, ed. Waddell, pp. 83-84).Since we know that the Sankt Urban collection was copied in Cîteaux,47 and since the general chapter of 1224 48 mentioned the existence of an originalum Diffinitorum, i.e., an official collection of statutes compiled by the definitors appointed for the chapter, it is likely that the copies above quoted are based on the Definitory's collection. Are we getting closer to the

	Origins	Dates	Extracts	Complete collections
	Loos	Early 17 th Century	-	1190-1282 and
				1387 32 -1405 33
	Feuillants (Paris)	1660's	1190-1220, 1419,	1190-1282 37 and
			1601-1651 34 , 1157-	1387-1557 38
			1201, 1207 35 and
			1190-1282 36

29 31 Canivez (Statuta 1, p. xvi) distinguished "very complete" (collectiones plenissimae), abridged (breves collectiones) and very abdridged collections (brevissimae collectiones).

32

In fact, there are only general decisions for 1387 and 1388.

33

Lille, Archives Départementales du Nord, 28 H 70.

34

Paris, Bibliothèque de l'Arsenal, 783.

35

Paris, Bibliothèque de l'Arsenal, 785.

36

Paris, Bibliothèque Mazarine, 1758.

Table 3 :

 3 Degrees of completeness of the Cistercian general chapter's statutes (late 12 th -early 15 th Century) we have the complete collection of statutes from 1190 to 1282 and from 1285 to 1298? Again, the answer is no; Canivez himself offered definitive proof by quoting Winter's book.52 Charters or chronicles, however, occasionally record general chapter decisions that do not appear in the statutes collections.53

	"Complete"	Only general	Very few	"Dark years"	Compilations
	collection	Decisions	general
			decisions
					Capitula
		1157-1161			"1157"
					collection
		1180-1190			Instituta
					Generalis
					Capituli
	1191-1282				1202 codification
					1220 codification
					1237 codification
					1257 codification
		1282-1284
	1285-1298				1290 codification
		1299-1343			1316 codification
	1344
			1345-1351		1350 codification
		1352
			1353-1363
				1364
			1365-1375
				1376
			1377
				1378-1386
			1387-1388
	1389-1413

If we now turn back to the more glorious twelfth and thirteenth centuries, we face another problem: with very few exceptions, we have no individual decisions before 1190. The question is: do

  Their main proof was the fact that the Martène and Canivez editions did not mention Cistercian nuns until 1213, with the exception of an 1191 statute regarding the famous Burgos abbey of Las Huelgas. 55 But the charters of Las Huelgas contain two of the 1187 general chapter's decisions that do not exist in the collection and which show a very different attitude toward Cistercian nuns. 56 Similarly, in the archives of the Paris nunnery of St. Antoine-des-Champs there is a small parchment carta notifying the incorporation of St. Antoine and Port-Royal during the general chapter of 1206.

	by Harald Schwillus & Andreas Hölscher. Studien zur Geschichte, Kunst und Kultur der Zisterzienser 10
	(Berlin: Lukas, 2000), pp. 34-135; Franz Felten, "Waren die Zisterzienser frauenfeindlich? Die Zisterzienser und
	die religiöse Frauenbewegung im 12. Jahrhundert und frühen 13. Jahrhundert. Versuch einer Bestandsaufnahme
	der Forschung seit 1980", in Norm und Realität, ed. Felten & Rösener, pp. 179-223.
	55 Statuta 1, ed. Canivez, p. 139 #27.
	56 Documentación del monasterio de Las Huelgas de Burgos (1116-1230), ed. by José Manuel Lizoain Garrido
	(Burgos: Garrido Garrido, 1985), pp. 25-26 #13, 30-32 #16. There is also a 1188 letter of the abbot of Cîteaux

57

It is, therefore, inappropriate to decide that a nunnery was not "truly" Cistercian, i.e., formally incorporated, during the late twelfth and thirteenth centuries because its name does not appear in Canivez's index.

58

The transmission of the statuta 52 For instance, Statuta 2, ed. Canivez, pp. 473-474 #62-64 (in 1260) or Statuta 3, ed. Canivez, p. 287 #23 (in 1296).

53

Chartes et documents concernant l'abbaye de Cîteaux. 1098-1182, ed. by Jean Marilier (Rome: Editiones cistercienses, 1961), p. 1.

54

Ernst Günther Krenig, "Mittelalterliche Frauenklöster nach den Konstitutionen von Cîteaux", Analecta sacri ordinis Cisterciensis, 10 (1954), pp. 1-105; Sally Thompson, "The Problem of the Cistercian Nuns in the Twelfth and Early Thirteenth Centuries", in Medieval Women 1, ed. by Derek Baker (Oxford: Basil Blackwell, 1978), pp. 227-252; Franz Felten, "Die Zisterzienserorden und die Frauen", in Weltverachtung und Dynamik, ed. and the General Chapter to the abbesses of Castilla and León (Documentación, ed. Garrido, pp. 35-36 #19), but it seems to be a forgery since it mentions Guy as abbot of Cîteaux whereas William was in charge at that time.

57

Paris, Archives Nationales, L 1015 #2.

58

Moreover, the General Chapter's prerogatives increased during the twelfth and thirteenth centuries (Ghislain Baury, "Émules puis sujettes de l'Ordre cistercien. Les cisterciennes de Castille et d'ailleurs face au chapitre général aux XII e et XIII e siècles", Cîteaux. Commentarii Cistercienses, 52 (2001), pp. 27-58).

Table 4 :

 4 Statute prohibiting Cistercians and Carthusians to receive a member of the other order without its authorisation

	Manuscripts	Texts
	Copies of Cîteaux	De Cartusiensibus pacis charitatisque gratia, statuimus ut nullum de
	complete collection	eorum ordine sine ipsorum licentia recipiamus, et ipsi de nostris
		sine assensu nostro nullum recipiant.
	Dijon, Bibliothèque	De cartusiensibus constituitur ut nec ipsi sine licentia nostra nostros
	Municipale, 601	recipiant; nec nos suos sine sua licentia.
	Troyes, Médiathèque,	Cartusiensibus conceditur ut nullus eorum in ordine nostro
	1285 from Clairvaux	recipiatur sine litteris commendaticiis; nec aliquis de ordine nostro
		in ordine ipsorum.
	Paris, Arsenal 785	Fratres chartusiae de cetero non licet nobis recipere sine licentia
	from Les Feuillants	eorum, nec ipsi nostros sine licentia nostra.

  74 Benoist Pierre, "La réforme des Feuillants: une relecture cistercienne à la fin du XVI e siècle?", in Unanimité et diversité cisterciennes. Filiations-Réseaux-Relectures du XII e au XVII e siècle (Saint-Étienne: Publications de l'Université de Saint-Étienne, 2000), pp. 647-663.

75

Twelfth-Century Statutes, ed. Waddell, p. 169.

76

Twelfth-Century Statutes, ed. Waddell, pp. 23-25.

77

Statuta 2, ed. Canivez, p. 248 #16.

78

Statuta 2, ed. Canivez, p. 36-37 #8.


Emmanuelle Esperança- Rodier
email: emmanuelle.esperanca-rodier@univ-grenoble-alpes.fr

Francis Brunet-Manquat
email: francis.brunet-manquat@univ-grenoble-alpes.fr

Sophia Eady
email: sophia.eady@univ-grenoble-alpes.fr

ACCOLÉ: A Collaborative Platform of Error Annotation for Aligned Corpora

This article presents a platform, named ACCOLÉ, for the collaborative annotation of translation errors. ACCOLÉ offers a range of services that allow simplified management of corpora and error typologies, annotation of effective errors, collaborative discussion during annotation, and finally different kinds of search in corpora. ACCOLÉ allows the annotation of translation errors according to built-in error typologies, Vilar's typology or DQF-MQM or uploaded ones, on several annotated corpora of different texts, translated by different Statistical or Neural MT systems. It also help to process the annotated corpora created in order to look for typical error models and patterns, related to a specific MT system. ACCOLÉ currently provides 15 corpora, 19 projects of 134,273 source words and 114,511 target words, and 23,525 annotations. Eventually, we will implement the semi-automatic propagation of found patterns on other corpora to compare the behaviour of the MT systems on different domains, thus providing to the community a wide range of error-annotated bilingual parallel corpora.

State of the Art

Translation error analysis is a task that has been developed in recent years in order to improve the machine translation (MT) systems and to qualitatively evaluate these same systems. In addition to the field of MT, translation error analysis is used in particular to improve translation teaching methodologies [START_REF] Kübler | Origines des erreurs en Traduction Spécialisée : différentiation textométrique grâce aux corpus de textes cibles annotés[END_REF]. The integration of MT in translation masters brings a new questioning about the evaluation of MT systems. How do translators feel about different MT systems [START_REF] Schwab | Evaluation of NMT and SMT systems: A study on uses and perceptions[END_REF]? What tools to use for nonspecialists? The answer to these questions requires a linguistic analysis of MT errors in order to be able, firstly, to reveal the typical errors of each type of MT systems, secondly, to correlate these typical errors with the post-editing capability of the users, and thirdly to create corpora. Performed manually, the translation error analysis is a very time-consuming task.

Research is therefore concerned with the automatic analysis of translation errors [START_REF] Popović | Morpho-syntactic Information for Automatic Error Analysis of Statistical Machine Translation Output[END_REF]. Manual or automatic, this task requires error analysis criteria, often related to the particular characteristics of the tool used.

The work of [START_REF] Vilar | Error analysis of statistical machine translation output[END_REF], the MeLLANGE project [START_REF] Castagnoli | Designing a Learner Translator Corpus for Training Purposes[END_REF], [START_REF] Felice | Linguistic Features for Quality Estimation[END_REF], Wisniewski et al. (2013a) and the work of [START_REF] Lommel | Tutorial: MQM-DQF: A Good Marriage (Translation Quality for the 21st Century)[END_REF] focuses on the definition of typologies of linguistic errors. An error typology consists of a list of errors, categorized according to the corrective actions to be performed to improve the translation. For example, the error type "missing word" indicates that a word of the source segment has not been translated in the translation hypothesis and therefore does not appear there. In order to improve the translation hypothesis, corrective actions must be taken to translate this "missing word". ACCOLÉ proposes these different typologies of error annotation, but also makes it possible to propose new typologies, better adapted to the correlation of the errors of the systems to the non-correctible errors made by the users. Other work deals with the creation of corpora of error translations and translation error correction, such as Potet et al. (2012) and Wisniewski et al. (2013b). Other research focuses on the creation of tools for either quality estimation such as Quest ++ [START_REF] Specia | Multi-level Translation Quality Prediction with QuEst++[END_REF], or quality assessment with Kantan's MT LQR and PET [START_REF] Aziz | PET: a Tool for Post-editing and Assessing Machine Translation[END_REF]. ) or the error analysis such as BLAST [START_REF] Stymne | Blast: A tool for error analysis of machine translation output[END_REF], focused on the annotation, Coreference Annotator (Tsoumari and Petasis, 2011) more parallel corpora oriented, YAWAT [START_REF] Germann | Yawat: Yet Another Word Alignment Tool[END_REF] a tool of alignment that allows labeling, and TranslationQ [START_REF] Steurs | Translation Expert (TranslationQ & RevisionQ): Automated translation process with real-time feedback & evaluation/ revision with PIE[END_REF] translation and revision oriented.

It is in this last vein that we position ourselves. We want to analyze translation errors on different corpora, in collaboration with people from various backgrounds. Since the task of analyzing errors is already tedious, we have created an online platform that can be accessed at http://lig-accole.imag.fr. In the rest of this article, we will present our platform ACCOLÉ, before presenting the annotated corpora obtained so far.

ACCOLÉ, a platform to annotate translation errors

Purpose

The initial goal that guided the development of ACCOLÉ is the manual annotation of translation errors according to linguistic criteria. The underlying idea is to be able to provide a user with help in choosing a MT system to use, depending on the context (language and computer skills of the user, knowledge of the domain of the source document to be translated and the task for which s/he needs to translate the source document.) To do this, ACCOLÉ must be able to detect which linguistic phenomena are not handled correctly by the MT system studied. We offer on the same platform a range of services to meet the needs of analysis of translation errors. Thus, the main features of the platform ACCOLÉ are the simplified management of corpora, types of errors, annotators, etc. ; annotation of errors; collaboration and/or supervision during the annotation; the search for error models (type of errors at first, later morphosyntactic patterns) in the annotations and finally corpora creation.

Since the task of analyzing errors is already tedious, it is important that the people carrying it out have a simple access to the tool as well as the corpus they wish to analyze. This is why the platform is available online on a browser and does not require any specific set up.

The task consists in labelling translation errors by locating manually their occurrences in the target as well as in the source sentence.

Annotation Project Management

An annotation task is described on the platform thanks to the notion of project. A project is a couple consisting of a corpus and an error typology. Thus, as shown on Figure 1, a corpus can be associated with several error typologies in the form of several annotation projects. The corpus will only be loaded once on the platform.

The project leader will also provide the list of annotators and supervisors. The error typologies are handled by the platform managers.

This notion of project allows not to duplicate corpora. As presented in table 1, two typologies of errors i.e. [START_REF] Vilar | Error analysis of statistical machine translation output[END_REF] and DQF-MQM [START_REF] Lommel | Tutorial: MQM-DQF: A Good Marriage (Translation Quality for the 21st Century)[END_REF]

Error Typology Management

As well as permitting to upload new corpora, the platform offers the possibility to enter other error typologies.

As illustrated in figure 2, an error type will consist of a name, a category (optional), a subcategory (optional) and a code (keyboard shortcut that can be used when annotating). ACCOLÉ offers Vilar and DQF-MQM typologies. We are planning to add MeLLange.

Error Annotation

Once in a project, one can visualize the source and target sentences and identify the error manually by clicking on the source occurrence that has not been correctly translated, and on the corresponding target error occurrence. Figure 3 shows the interface proposed to the annotator.

Figure 3. Error Annotation Process.

The annotation is done in two steps. The first step is to select, with the mouse, words in the source sentence, and their equivalent in the target sentence, presenting a translation error. Single or Multiple selection can be done. The selection of non-adjacent words is possible. One can also select a word and its position in the translation hypothesis.

The second step is to choose the type of errors either with the help of the mouse, or with the aid of keyboard shortcuts, to be associated to the pair of previously selected source/target words. It is important to associate the type of errors with both the source and the target in order to identify error models that will ultimately propagate annotations to other corpora.

A collaborative platform

To answer the inter-annotator agreement issue [START_REF] Popović | Error Classification and Analysis for Machine Translation Quality Assessment[END_REF], ACCOLÉ offers two mechanisms to help the annotator in her/his task: a supervisory mechanism allowing a manager to control the progress of the task, this mechanism encourages especially the communication between supervisor and annotator by the possibility of creating threads for a specific source/target phrase pair (asking for clarification on a error type, pointing out an annotation error, etc.); and a collaborative mechanism allowing annotators to communicate about a precise source/target phrase pair among themselves. This latest mechanism is an option to be activated in the project while creating it.

SSTC representation of errors

The platform uses a data representation based on the Structured String-Tree Correspondences (SSTC) [START_REF] Boitet | Representation trees and string-tree correspondences[END_REF]). An error consists of a label and a set of SNODEs (range representing the substring in the corresponding source or target phrase). For example, in Figure 4, the error concerning "all" and "any" is described by its label Poor lexical choice (cat. Object word, sub-cat. Sense), by its position in the source sentence substring between the 49th and 56th characters) and the target phrase ). The advantage of using SNODEs in this way is to avoid the use of a syntactic structure to describe the error. SNODEs also permit to spot the position of a missing word in the target, as we can see in Figure 1 with "un des" not translated, thus represented by SNODE [16][17][18] in the target. They reversely permit to associate a position in the source to extra words appearing in the target.

The other advantage is that it permits to add a posteriori morphosyntactic analysis on the source and target sentences. One or more analyzes (Stanford Tagger, Xerox Incremental Parser, etc.) can be attached using SNODE to phrases. At the end of the annotation task, its analyzes will be used to search for models of errors (morphosyntactic patterns for example). The idea is to use errors as a pivotal representation in the search mechanism.

Looking for Errors

To study the behaviour of MT systems, morphosyntactic analyses or error types (as illustrated in figure 5) are used to search for error models (morphosyntactic patterns for example) within a project. The errors are therefore used as a pivotal representation in the search mechanism. It will help looking for any consistency in the errors of MT systems.

Corpora

At the time of writing this article, ACCOLÉ proposes 2 typologies of errors (those of [START_REF] Vilar | Error analysis of statistical machine translation output[END_REF] and DQF-MQM [START_REF] Lommel | Tutorial: MQM-DQF: A Good Marriage (Translation Quality for the 21st Century)[END_REF] as well as 15 French-English corpora (ranging from journalistic news, technical documents, patents, French monolingual texts, excerpts from the Basic Travel Expression Corpus (BTEC), up to climate documents), which led to the creation of 19 projects. These correspond to 6,817 sentences, 134,273 source words, 114,511 target words, for 23,525 annotations made by native English annotators, as shown in table 3. Some annotated corpora available have already been used for a linguistic comparison of the translation quality of different MT systems [START_REF] Esperança-Rodier | Comparaison de systèmes de traduction automatique, probabiliste et neuronal, par analyse d'erreurs[END_REF]. A study is planned to use ACCOLÉ for the bilingual annotation of MWEs. The error model (morphosyntactic patterns) search in the annotations, essential asset of ACCOLÉ, will allow us in a future version, to use the models of errors in order to propagate, thanks to neural network, the annotations of errors in a semi-automatic way on new corpora. This will allow to build more rapidly annotated corpora and consequently assessing almost in real-time new MT systems. We will then be able to compare the behaviour of MT systems on different domains, thus providing to the community error-annotated bilingual parallel corpora. We also want to improve it by also allowing the association of trees of dependencies. Further work evaluating inter-annotator agreement is still being conducted at the time of writing this paper.

Figure 1 .

 1 Figure 1. Project management.

Figure 2 .

 2 Figure 2. Error Typology Creation.

Figure 4 .

 4 Figure 4. Annotation Representation Example.

Figure 5 .

 5 Figure 5. Example of an error type search.

Table 1 :

 1 are available on ACCOLÉ. Fifteen French-English corpora, journalistic news, technical documents, patents, extracts from BTEC (Basic Travel Expression Corpus) climate documents are part of the nineteen existing projects. Project composition

	Projects	Error Typology
	Corpus	Vilar	DQF-MQM
	Citi1	Citi1-Vilar/Demonstration
	Citi2	Citi2-Vilar
	News 156 Moses	News 156 Moses -Vilar
	News 156 Google	News 156 Google -Vilar
	News 2000	News 2000 -Vilar
	BTEC1-LIGMoses	BTEC1-LIGMoses -Vilar BTEC1-LIGMoses -DQF-MQM
	BTEC1-NMTLIG	BTEC1-NMTLIG -Vilar	BTEC1-NMTLIG -DQF-MQM
	WIPOTranslateN	Hawaii	Brad Pitt
	WIPOTranslateS	Madagascar	George Clooney
	CANADA_eTranslation	Nouvelle Calédonie	Jude Law
	CANADA_LegacyMT@EC New Zealand	Robert
	Doc1_Fujitsu_180118	Altica test 1
	(monolingue)
	Doc2_Fujitsu_20180109	Altica test 2
	(monolingue)
	TOTAL	13	6
			19

Table 2 :

 2 Corpora List These corpora are structured according to SNODEs and are available on request in XML or JSON format. ACCOLÉ allows the annotation of translation errors according to built-in error typologies or uploaded ones, on several annotated corpora of different texts, translated by different Statistical or Neural MT systems. ACCOLÉ offers 15 corpora, 19 projects consisting in 248,784 words and 23,525 annotations.

	3 Conclusion and further work

Acknowledgements

The work reported above has been granted by LIG/Emergence funding in 2017 and 2019.


Emmanuelle Esperança-Rodier
email: emmanuelle.esperanca-rodier@univ-grenoble-alpes.fr

Caroline Rossi
email: caroline.rossi@univ-grenoble-alpes.fr

Time is Everything: A Comparative Study of Human Evaluation of SMT vs. NMT

Translation process research has developed tools to gather and analyse empirical data, but while a variety of measures have proved useful and reliable to assess machine translation post-editing effort (see e.g. Vieira 2016: 42), translation processes are seldom considered when assessing the relevance of a given Machine translation post-editing (MTPE) scenario.

Our study seeks to determine the impact of including MTPE in the evaluation process. We selected adequacy and fluency ratings. Based on two distinct experimental conditions, we then compared the ratings produced without performing PE and those produced immediately after a light PE process.

Inter-rater reliability was assessed for each segment in each text (N=55) using Fleiss' kappa for adequacy and fluency scores, and an intra class correlation coefficient (Vieira 2016: 52) for temporal measures. While the reliability of the measures collected without PE was low, the measures collected in PET were for the most part homogeneous. Qualitative analyses of the problematic segments, as evidenced by both kappa and intra class correlation coefficients, showed strong Spearman's correlations, whether positive or negative, between temporal measures and all the other metrics for NMT but weakest ones for SMT. Based on these results, we discuss the advantages and risks of NMTPE.

Introduction

Machine translation evaluation (MTE) is performed differently and with different goals in academia and industry (Drugan 2013, in Castilho et al. 2018: 11). However, with the current integration of neural machine translation into human translation workflows, reliable measures of the amount of effort needed to post-edit machine translation (PEMT) outputs have become a common goal for researchers, language service providers and machine translation vendors (ibid., p. 29). Translation process research has developed tools to gather and analyse empirical data, but while a variety of measures have proved useful and reliable to measure PEMT effort (see e.g. Vieira 2016: 42), translation processes are seldom considered when assessing the relevance of a given MTPE scenario.

Against this background, our study seeks to determine the impact of including MTPE in the evaluation process. We selected two of the most commonly used scales for the "declarative evaluation" of MT (Humphreys et al. 1991, in Way 2018b: 164): adequacy and fluency ratings. Based on two distinct experimental conditions, we then compared the ratings produced without performing PE and those produced immediately after a light PE process.

Methodology

Data was collected with a group of 14 trainee translators, using two different text types and two different tools. Based on the requirements of our French curriculum in specialised translation, we selected only translation into French, and for the sake of comparison we kept only one source language: English. A first series of assessments was conducted with KantanMT's language quality review system (LQR), which allows for a simple comparative evaluation of two systems without post-editing the outputs. The second series was done a few weeks later, in Post-Editing Tool (PET, [START_REF] Aziz | PET: a Tool for Post-editing and Assessing Machine Translation[END_REF]. Each experimental condition includes two source texts from two different domains (environmental discourse and patents). We generated usable SMT and NMT outputs using the European Commission's MT (eTranslation) with environmental texts and WIPO translate with patent extracts. In both conditions, the students were given a realistic scenario --i.e. they performed the evaluation, with a view to determining whether the MT output was relevant to a particular order.

Data

We have selected four documents dealing with two different subjects. Two documents were Patent extracts while the two other two were environmental texts.

Each document has been translated with Machine Translation (MT) systems trained on the domain, thus using WIPO Translate and eTranslation respectively for Patents and environmental texts. As those two MT systems offer a Neural version (NMT) as well as a Statistical one (SMT), we used them both to translate our documents.

Consequently, we obtained two translation versions of the same source document, which are one WIPO SMT translation version and one WIPO NMT version for a patent document. We therefore got four translations for the two Patent extracts: two SMT translations and two NMT for each text.

In the same way, we performed both NMT and SMT translations on the two environmental texts, having thus one eTranslation SMT translation version (called "LegacyMT@EC" in the eTranslation portal) and one eTranslation NMT version (called "cutting edge") for each environmental document, again resulting in four translations.

As a result, we have got 8 documents, as shown in Those documents were used in the two assessments conducted with our translator trainees. The first assessments have been conducted using the Kantan tool in order to assess adequacy and fluency of one patent NMT and one SMT translation and of one environmental text based on both NMT and SMT translations.

The second assessment has been fulfilled using PET in order to post-edit translations before assessing adequacy and fluency. For this last assessment, as required in PET [START_REF] Aziz | PET: a Tool for Post-editing and Assessing Machine Translation[END_REF], we have had to mix the NMT and SMT translations in the same document so that the translator trainees did not know which translation was the NMT one, nor the SMT one. We have indeed, created from the NMT version translation and the SMT version translation, two mixed-version translation documents as stated in Table 2 hereafter. To do so, we mixed the odd sentences from the NMT version with the even sentences from the SMT version and conversely the even sentences from the NMT version with the odd sentences from the SMT version, to create the second document. Table 2: structure of mixed translation texts.

Once created, we have started the series of assessments with our translator trainees as described in the following two sections.

First series of Assessment

As far as the first experiment is concerned, translator trainees were given three texts for each domain. The source document, patent and environmental discourse ones, its NTM version translation and its SMT version translation. The three texts were uploaded in Kantan, as shown in Figure 1 below, for the translator trainees to assess the adequacy and the fluency of each translation from a rank ranging from 1 to 5, with indeed a middle point As already mentioned in section 2.1, no post-editing was possible in Kantan's comparison tool (called "A-B test"). Furthermore, translator trainees did know which translation was the result of the NMT version or of the SMT one. This first series of assessment were done to familiarise translator trainees to adequacy and fluency ranking. At the end of their course, trainees had to do the second series of assessment. As said previously in section 2.1, for these series of assessment, the NMT and SMT versions were mixed into a new text, for the two domains, so that we would be sure the SMT and NMT versions would not be discovered by the translator trainees.

Second Series of Assessment

Four documents, one source document and one mixed-version translation document for each domain, were uploaded into PET. Before assessing the adequacy and fluency, the translator trainees had to post-edit the proposed translations.

After the post-edition, they were offered to score the accuracy and fluency with a rank ranging from 1 to 4, with currently no middle point, as can be seen in Figure 2. In order to ensure the quality of the work, the translator trainees were presented this assessment as the final exam of their curriculum, thus accounting for their credits.

Results

Inter-rater reliability was assessed for each segment in each text using Fleiss' kappa for adequacy and fluency scores, and an intra class correlation coefficient (Vieira 2016: 52) for temporal measures. While the reliability of the measures collected without PE was low, the measures collected in PET were for the most part homogeneous.

Inter-rater reliability

In table 3, we compare overall agreement in percentage, together with Fleiss's Kappa. The latter does not appear to be fully relevant to our experimental setting. Indeed, we do not think that chance scoring is likely to happen at all in the context of a course on MT with MA students. The first piece of evidence for their involvement is the time spent assessing (see below table 4). Besides, the students had been trained to use adequacy and fluency scoring, and the two sessions considered for the present study were a training session for the final test, and the final test itself (in which students were given a mark for the course).

Overall, adequacy ratings in PET are the most reliable from the lowest 41.11% to the highest 60% opposed to Kantan adequacy rating ranging from 26.84% to 39.93%, which is much lower. Fluency ratings in PET are less reliable as they ranged from 30% to 50% (20 point raise), but still on average higher than Kantan results -the latter ranging from 27.29% to 39.65%. It appears that our students had more trouble assessing fluency with the patent extracts. Those results show that a post-editing task before assessing the adequacy of the translation provides a better agreement between raters. According to domains, the post-edition of the environmental texts before assessing adequacy and fluency implied a better inter-rater agreement, respectively 26.84% vs. 41.11% (14 point raise) for SMT, 28.89% vs. 60% (31 point raise) for NMT and 32.65% vs. 50% (18 point raise) for SMT and 32.50% vs. 45.56% (13 point raise) for NMT. Nevertheless, it was less clear-cut for the post-edition of the patents before assessing adequacy, 39.93% vs. 46.67% (6.5 point raise) for SMT and 35.81% vs. 59.17% (23 point raise) for NMT, and fluency with 39.65% vs. 37.5% (2 point loss) for SMT and 27.29% vs. 30% (almost 3 point raise) for NMT. We could even notice with a loss of agreement of 2 points for the SMT version. This result can reinforce the fact that the translator trainees had more difficulties assessing fluency and Patents.

Concerning the different MT systems used, when comparing NMT vs. SMT, on the whole, we obtained a better agreement on adequacy for NMT than for SMT, 35.51% vs. 39.93% (4 point loss), 28.89% vs. 26.84% (2 point raise), 59.17% vs. 46.67% (12.5 point raise) and 60% vs. 41.11% (19 point raise). Again, the 4-point loss of agreement for NMT vs. SMT on Patents shows the difficulties of student to assess Patents. Moreover, we can see that the results are much better for the agreement on the adequacy for NMT when a post-edition had been performed prior to quality assessment. Contrastingly, the agreement on fluency, we obtained a better agreement for SMT than for NMT, 39.65% vs. 27.29% (12 point raise), 32.65% vs. 32.50% (equality), 37.6% vs. 30% (7 point raise), 50% vs 45.56% (4.5 point raise). We can see that the difference among agreements as regards to fluency are less clear-cut, reinforcing the evidence that translator trainees had more difficulties to assess fluency. Once again, we can see that agreement is stronger when a post-edition had taken place before the quality assessment even if it is less obvious.

Intra class correlation (ICC) coefficients

Similarly, and even though there was more variation in temporal measures, homogeneity in ICC coefficients was stronger in PET data.

In Kantan, we could only get the overall time spent assessing both the SMT and NMT version translations for one source segment. We thus had to work on the duration means of assessing SMT and NMT in PET in order to be able to compare the results.

With Kantan, we can see in table 4 that there is remarkable variation from the Patent domain compared to the environmental domain. In contrast, PET has more homogeneous means, even though the standard deviation is very high and that the intra class correlation (ICC) also shows a lot of variation among raters. We can see that the post-editing time impacts the mean duration. We can also notice the the ICC seems to increase when less time is spent on assessing. We finally sought to determine what went wrong by performing qualitative analyses of the problematic segments, as evidenced by both kappa and intra class correlation coefficients.

Mean

Qualitative Analyses

Qualitative analyses have been processed using the ACCOLÉ (Brunet-Manquat and Esperança-Rodier, 2018) annotation platform, as illustrated in Figure 3. Translation errors have been annotated according to DQF-MQM [START_REF] Lommel | Tutorial: MQM-DQF: A Good Marriage (Translation Quality for the 21st Century)[END_REF] error typology and correlations between the different metrics have been calculated using Spearman's correlations. Figure 3. ACCOLÉ Interface.

We ended with the annotations of each sentence assessed by the trainee translators, according to the DQF-MQM typology. For easier understanding, we have counted for each sentence, on the one hand, the number of annotated errors corresponding to the accuracy type and on the other hand, the number of annotated errors corresponding to the fluency type.

Table 5 shows the Spearman's correlation calculation done for each measure obtained for the NMT translation version we had for the environmental domain. Time is strongly correlated to HTER (r s =0.6103 with p-value=2.2525 -7 ) as well as the assessing time (r s =0.4973 with p-value=5.3038 -5 ) which leads us to say that time spent is dedicated to the post-edition. This is confirmed by the fact that HTER is correlated to the Assessing time (r s =0.3885 with p-value=0.0021).

Furthermore, the Assessing time is highly to the presence of Fluency errors (r s =0.8123 with p-value=3.3627 -15 ). Consequently, we can say that when there are Fluency errors the assessing time is longer.

As regards the correlation between Adequacy and Fluency (r s =0.4167 with p-value=0.0009), we can see that the scores given to the Adequacy of a translation have an influence on the score given to Fluency. In the same way, looking at the correlation between Accuracy errors and Fluency errors (r s =0.7830 with p-value=1.3894 -13 ), we can see that there is also a link leading us to say that the scores given to the Accuracy errors of a translation have an influence on the score given to the Fluency errors.

Logically enough, the correlation between the Total errors and the Accuracy errors and the Fluency Errors is really high.

It is also interesting to focus on the strong negative correlations from Adequacy and Fluency as regards Time, Hter and Assessing Time. Those negative correlations mean that, when there is a lot of time spent, and that there are a lot of changes between the translation and the reference, then the Adequacy and Fluency scores are much lower, and conversely. Table 6 shows Spearman's correlation for measures taken for the SMT translation on the same domain as the one for NMT. We almost find the same correlations, nevertheless, the correlations between Time and Hter (r s =0.3423 with p-value=0.0074) is weaker for the SMT translation opposed to NMT translations. The lower value suggests that there is more variation and that there might be more effort, as captured by variations in Time measurements, while Hter scores remain relatively similar. It is also the case for the correlations as regards Hter and Adequacy as well as Fluency, which are still strong negative correlations. As was the case for NMT, but in a smaller proportion, when the Hter score is high, the scores for Adequacy and Fluency are low. The lower proportion suggests that students might be less good at using Adequacy and Fluency measures for assessing effort with SMT.

Again, and as expected, the correlations between Total errors and Accuracy errors and Fluency errors are really strong.

Finally, for the SMT translation, the Assessing time is strongly correlated to Adequacy scores while for the NMT translation it was correlated to Fluency Scores. This statement matches previous NMT vs. SMT quality assessments showing that NMT translations are more fluent than SMT ouputs, while SMT translations are more adequate (see e.g. [START_REF] Koehn | Six challenges for neural machine translation[END_REF].

Discussion

Results showed strong correlations, whether positive or negative, between time spent postediting and all the other metrics for NMT but weakest correlations as far as SMT was concerned. Our results thus point to much more homogeneity in post-editing NMT outputs, with more variation in the treatment of SMT errors. The consequences of these differences for professional post-editors include both lighter cognitive effort and improved cognitive ergonomics when dealing with NMT. It is also worth noting that estimated effort (as expressed by Fluency and Adequacy ratings) was on the whole more realistic with NMT outputs. However, it remains to be seen how post-editors will address the higher risks induced by more homogeneous and fluent NMT outputs, notably that of meaning errors going unnoticed (Forcada, 2017: 303). Experimental designs including hidden errors and allowing for a measure of cognitive effort would help in determining whether the necessary attention to details with NMT outputs ends up being more demanding than the very regular, and somewhat teadious, post-editing tasks required with SMT.

  siège (3) positionné devant la presse de frein (2) pour un opérateur de ladite presse de frein (2), C-Un siège (3) positionnée devant la presse et frein pour un opérateur dudit frein, presse (2) c-un siège (3) positionné devant la presse de frein (2) pour un opérateur de ladite presse de frein (2), C-Un siège (3) positionnée devant la presse et frein pour un opérateur dudit frein, presse (2) d-Et-un système de support réglable (10) qui supporte le siège (3) et qui est fixé à un point de fixation (9), la presse de frein (2) ayant un cadre (4) qui supporte un premier porte-outil (5) et un second porte-outil (6 ). D-Et un système de support réglable (10) qui supporte le siège (3) et est fixée au niveau d'un point de fixation (9) (2), le frein de presse ayant un cadre (4) qui supporte un premier porte-outil (5) et un second porte-outil (6 ). D-Et un système de support réglable (10) qui supporte le siège (3) et est fixée au niveau d'un point de fixation (9) (2), le frein de presse ayant un cadre (4) qui supporte un premier porte-outil (5) et un second porte-outil (6 ). d-Et-un système de support réglable (10) qui supporte le siège (3) et qui est fixé à un point de fixation (9), la presse de frein (2) ayant un cadre (4) qui supporte un premier porte-outil (5) et un second porte-outil (6 ). e-Le premier porte-outil (5) peut être déplacé par rapport au deuxième porteoutil (6) pour exécuter un mouvement de travail. E-Le premier porte-outil (5) peut être déplacé par rapport au second support d'outil (6) pour effectuer un mouvement de travail. e-Le premier porte-outil (5) peut être déplacé par rapport au deuxième porteoutil (6) pour exécuter un mouvement de travail. E-Le premier porte-outil (5) peut être déplacé par rapport au second support d'outil (6) pour effectuer un mouvement de travail. … … … …

Figure 1 .

 1 Figure 1. Kantan interface.

Figure 2 .

 2 Figure 2. PET interface.

  Table 1 below.

	Documents	WIPO NMT	WIPO SMT	eTranslate NMT eTranslate SMT Total
	Patent 1	1	1			2
	Patent 2	1	1			2
	Environmental			1	1	2
	discourse Text 1
	Environmental			1	1	2
	discourse Text 2
	Total	2	2	2	2	8
			Table 1: data.

Table 3 :

 3 Inter-rater Agreements

duration (ms) per sentence Standard Deviation ICC

	Kantan Patents (WIPO	96.20558608 82.17102284 0.185130901
	Translate)
	Kantan Climate	249.3956044 271.804465 0.048161741
	(eTranslation)
	PET Patents (WIPO	32193.90625 31266.08828 0.094569923
	translate)
	PET Climate	36006.725	45218.36249	0.05
	(eTranslation

Table 4 :

 4 Intra class correlation coefficients

Table 5 :

 5 Spearman's correlation -NMT translation -environmental domain.

			Time
	Time	r s	1
		p %	0	Hter
	hter	r s p %	0.6103 2.2525 -7 0 1	AssessingT
	Assessing T r s p %	0.4973 5.3038 -5 0.0021 0.3885	1 0	Total errors
	Total errors r s	0.3599	0.2104	0.056	1
		p %	0.0047	0.1066	0.6707	0	Accuracy E
	Accuracy E r s p %	0.3991 0.0016	0.1971 0.1311	0.1501 0.2523	0.9852 4.0900 -46	1 0	Fluency E
	Fluency E	r s p %	0.2634 0.0420	0.0915 0.4867	0.8123 3.3627 -15 2.2014 -18 0.8575	0.7833 1.3894 -13	1 0	Adequacy
	Adequacy	r s p %	-0.481 0.0001	-0.5693 -0.3785 2.0636 -6 0.0029	-0.2904 0.0244	-0.2879 0.0257	-0.2091 1 0.1088 0	Fluency
	Fluency	r s p %	-0.5208 -0.6071 -0.4775 1.9916 -5 2.7045 -7 0.0001	0.0237 0.8574	0.0021 0.9873	0.1659 0.2052	0.4167 0.0009	1 0

Table 6 :

 6 Spearman's correlation -SMT translation -environmental domain.

			Time
	Time	r s	1
		p %	0	Hter
	hter	r s	0.288	1
		p %	0.0256	0	AssessingT
	Assessing T r s	0.2568	0.3423	1
		p %	0.0479	0.0074	0	Total errors
	Total errors r s	0.2715	0.0841	0.0761	1
		p %	0.0358	0.5228	0.5634	0	Accuracy E
	Accuracy E r s p %	0.3698 0.0036	0.0474 0.7193	0.186 0.1547	0.687 1.3528 -9	1 0	Fluency E
	Fluency E	r s p %	0.0965 0.4635	-0.0009 -0.09 0.9946 0.49442	0.7656 1.0415 -12	0.0955 0.4681	1 0	Adequacy
	Adequacy	r s	-0.2082 -0.4452 -0.4565	-0.2578	-0.1758	-0.097	1
		p %	0.1103	0.0003	0.0002	0.0467	0.1791	0.4608	0	Fluency
	Fluency	r s	-0.5046 -0.4432 -0.3071	-0.1798	-0.1025	-0.1553 0.4428	1
		p %	3.9331	0.0004	0.0170	0.1691	0.4359	0.2360	0.0004	0

Acknowledgements

The work reported above has been granted by NeuroCoG/Pôle Grenoble Cognition funding as well as LIG/Emergence funding.