Frederique Bordignon
email: frederique.bordignon@enpc.fr

Self-correction of science: a comparative study of negative citations and post-publication peer review

This study investigates whether negative citations in articles and comments posted on post-publication peer review platforms are both equally contributing to the correction of science. These 2 types of written evidence of disputes are compared by analyzing their occurrence in relation to articles that have already been retracted or corrected. We identified retracted or corrected articles in a corpus of 72,069 articles coming from the Engineering field, from 3 journals (Science, Tumor Biology, Cancer Research) and from 3 authors with many retractions to their credit (Sarkar, Schön, Voinnet). We used Scite to retrieve contradicting citations and PubPeer to retrieve the number of comments for each article, and then we considered them as traces left by scientists to contest published results.

Our study shows that contradicting citations are very uncommon and that retracted or corrected articles are not more contradicted in scholarly articles than those that are neither retracted nor corrected but they do generate more comments on Pubpeer, presumably because of the possibility for contributors to remain anonymous. Moreover, post-publication peer review platforms, although external to the scientific publication process contribute more to the correction of science than negative citations. Consequently, post-publication peer review venues, and more specifically the comments found on it, although not contributing to the scientific literature, are a mechanism for correcting science. Lastly, we introduced the idea of strengthening the role of contradicting citations to rehabilitate the clear expression of judgment in scientific papers.

Introduction

Science needs both innovation and self-correction. This has been ensured for centuries by academic controversy (disputatio 1 ), in other words, free and contradictory discussion within scientific peers. Scientists should both be sceptical of any new claim and provide statements, hypotheses and theories that can be falsified (refuted) [START_REF] Popper | The Logic of Scientific Discovery[END_REF]. Science self-correction can result in the production of new knowledge with publications that rely on previous statements, oppose to them or just ignore them. It can also take a more drastic way with the official purge of the scientific literature and the withdrawal of what is recognized as false science. Retraction notices or errata are then published to correct the scientific record. Our study focuses on the process of challenging past works and investigates 2 mechanisms: negative citations and post-publication peer review. Negative or contradicting citations can be found in scientific writing but postpublication peer review comments do not contribute to the scientific literature as such. Nevertheless, both target and cite a preexisting work by expressing a disagreement. We have made the methodological choice to compare them by analyzing their occurrence in relation to articles that have already been retracted or corrected, in other words, publications whose contentious nature has already been acknowledged by a correction of the scientific record. This work is therefore at the boundary of 3 research topics: retracted or corrected papers, negative in-text citations, and post-publication peer review. It will address the following research question: are negative citations and post-publication peer review comments both equally contributing to the correction of science?

Background

Retracted and corrected papers Retractions are a way to alert the scientific community (and beyond) that flawed research has been published. The problematic pieces of work are therefore supposed to be purged from the literature or at least flagged because they are partially or fully inaccurate. They should not be used to build new research. Retractions are reserved for circumstances in which significant portions of an article are incorrect or cannot be substantiated whereas errata are published when isolated inaccuracies have been identified [START_REF] Furman | Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine[END_REF]. In both cases, the reasons why a paper is retracted or corrected are various: fabrication or falsification of data and plagiarism are commonly agreed to be scientific misconduct, but honest errors can also lead to the correction of published literature. The frontier is sometimes hard to establish but the intention to deceive is the key aspect [START_REF] Bar-Ilan | Temporal characteristics of retracted articles[END_REF][START_REF] Fanelli | How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data[END_REF]. (Bar-Ilan and Halevi 2018) coined the term "scientific distortion" to describe a category of articles including intentional and unintentional errors, both being considered hurdles for the advancement of science.

Although [START_REF] Ioannidis | Why Most Published Research Findings Are False[END_REF] claims that "most research findings are false for most research designs and most fields", retracted and corrected papers are very scarce. Data from the RetractionWatch 2 database show that the absolute number of annual retractions has grown over the past decade (from 100 annual cases before 2000 to nearly 1,000 in 2014) but even if the rate doubled from 2003 to 2009, it has remained stable since 2012 and only 4 of every 10,000 papers are now retracted [START_REF] Brainard | Rethinking retractions[END_REF]. Although different from the ordinary process of self-correction of science (when new knowledge updates the old), retractions and errata are also a way to correct the scholarly record by changing the status of previously published works (Dougherty 2019). 1 Formalized procedure of debate dating from the Middle Ages. 2 RetractionWatch is a blog launched in 2010 by two science journalists and is dedicated to the tracking of retracted papers. In 2018, they released an online database of all the cases they have recorded: of all the cases they have recorded http://retractiondatabase.org.

The identification of retracted and corrected papers is difficult and despite COPE guidelines3 and the Crossmark initiative4 , publishers and bibliographic databases do not adopt any standards and are not consistent in their reporting of cases.

Contradicting or negative citations [START_REF] De Solla Price | Little science, big science[END_REF] discusses the phenomenon of cumulation of papers ("the way in which each paper is built on a foundation of previous papers then, in turn, is one of several points of departure for the next") and stated that "the most obvious manifestation of this scholarly bricklaying is the citation of references". A citation is, therefore, the explicit reference to another scientific piece of work within the full-text of a scholarly paper. And as [START_REF] Gross | Communicating science : the scientific article from the 17th century to the present[END_REF]) stated, from the late 20th century it has become a routine to conclude scientific articles with the list of references to past literature cited in the preceding text. Even if it has been done widely for decades, merely counting citations is controversial [START_REF] Cano | Citation behavior: Classification, utility, and location[END_REF][START_REF] Chubin | Content Analysis of References: Adjunct or Alternative to Citation Counting[END_REF][START_REF] Kaplan | The norms of citation behavior: Prolegomena to the footnote[END_REF]) and it is partly because "citation-in-context analysis" should also be taken into account [START_REF] Small | Co-citation context analysis and the structure of paradigms[END_REF]. There is indeed a wide variety of reasons for a researcher to cite a previous piece of work, and it is obvious that citing is not always performed in a positive or supportive manner. In the early 1980s, [START_REF] Garfield | Can Citation Indexing Be Automated? In Statistical Association Methods for Mechanized Documentation: Symposium Proceedings[END_REF] already discussed adding useful markers such as "critique", "data spurious" or "conclusions wrong" to describe the kind of relationship between the citing and cited documents. It is the same idea that is taken up by [START_REF] Peroni | FaBiO and CiTO: Ontologies for describing bibliographic resources and citations[END_REF]) by proposing the CiTO ontology (Citation Typing Ontology): it provides the authors with the ability to capture their intent when citing previous works, as it allows them to add specific metadata to annotate a citation with its reasons. Table 1 shows that whereas negative citations of all types are always mentioned as one or several specific categories in citation classification schemes, the studies that rely on them show consensually that they are very uncommon, most of the times representing less than 5% of all references cited in a document. This very low rate is quite surprising since, as (MacRoberts and MacRoberts 1984) stated, "criticism is the life blood of science". [START_REF] Cano | Citation behavior: Classification, utility, and location[END_REF] suggest that criticisms expressed through citations could also be part of the "falsification" process defined by [START_REF] Popper | The Logic of Scientific Discovery[END_REF]. But making a negative criticism towards another researcher requires a greater effort than making a perfunctory citation; it implies creating a context in which to justify an attack and explain why previous studies fell short [START_REF] Rousseau | Publication and Citation Analysis[END_REF][START_REF] White | Authors as citers over time[END_REF]. It also means to take a risk by openly stating one's disagreement. Moreover (Catalini et al. 2015) have shown that the authors were more willing to criticize researchers located further away geographically and explain it may be socially costly to negatively cite the work of a local colleague.

Among those studies presented in Table 1, early ones aimed at describing the motives for citations and demonstrating that they were not equal. The corpora they analysed were quite small. But recently with the help of Natural Language Processing tools and Machine Learning, the automatic analysis of in-text citations relies on sentiment analysis techniques with schemas based upon basic polarity sentiment: positive, neutral and negative. ). There is variance in the percentage of negative citations they identify: from 0 to 15% [START_REF] Gabehart | An analysis of citations to retracted articles in the scientific literature[END_REF][START_REF] Kochan | The persistence of fraud in the literature: The Darsee case[END_REF], and even up to 32% in [START_REF] Garfield | The Impact of Fraudulent Research on the Scientific Literature[END_REF], the unique study (Breuning case) not focusing on post-retraction period. When revisiting this study, [START_REF] Korpela | How long does it take for the scientific literature to purge itself of fraudulent material?: the Breuning case revisited[END_REF]) discovered negative citations were overrepresented but also signalled an upsurge of positive citations after several years. For these authors, the continuous citations of retracted articles after the publication of the retraction notice is a serious problem. But we believe that citations of these articles are problematic both before and after the retraction notice. And of course, it is even worse if the citation is supportive. The problem is indeed the dissemination of false statements and results at any time. What is more, some studies acknowledge that it is hard to determine a retraction date (Bar-Ilan and Halevi 2017; [START_REF] Neale | Analysis of Citations to Biomedical Articles Affected by Scientific Misconduct[END_REF], so comparative studies before and after retraction might not be accurate.

Post-publication peer review

Our study also investigates post-publication peer review comments as another type of traces left by scientists to contest published results. Traditional peer review is necessary to a sound process of publication and can be one of the mechanisms involved in the self-correcting nature of science. But peer review is unable to detect all problems, neither scientific flaws nor misconduct.

Comments or suggestions can be posted on academic social networks (e.g. ResearchGate, ScienceOpen) or with the help of dedicated tools (e.g. Hypothes.is). Some publishers also allow commenting on each publication on their website (e.g. PLoS, F1000) but most journals do not welcome correspondence or comments that criticize their publications [START_REF] Barbour | PubPeer: Scientific assessment without metrics[END_REF]. That is why post-publication peer review platforms have recently developed (Dubois and Guaspare 2019; Teixeira da Silva and Bornemann-Cimenti 2017). The most famous is PubPeer where comments can be made anonymously.

Although positive and praising comments can be made, they are in practice mainly negative or critical. Postpublication peer review venues are to be considered as tools to identify erroneous works that went through the traditional peer review process (Teixeira da Silva et al. 2017).

Methods and data

We introduce the idea to consider comments left on PubPeer as contradicting citations. Indeed, commenting on Pubpeer is the same principle as making a citation, with the consequence of creating a link between the comment and the target publication. However, what radically differentiates a comment on PubPeer from a citation in an article is that the former can be anonymous.

In order to evaluate negative citations in articles and comments on PubPeer as mechanisms for correcting science, we have built up a corpus of articles associated with retraction or correction notices. It is a way of having a material that has already been officially contested and corrected. From a methodological point of view, this saves us from having to individuate false science, and, therefore, it allows us to build a corpus of contested articles in an objective manner.

We used Scopus and the Web of Science to constitute a corpus of 72,069 articles (metadata), focusing on a particular discipline, on some particular journals and even individuals. We used 3 different criteria (discipline, journal, author) and constituted 7 datasets for comparison (Table 2):

-Discipline: we chose the Engineering field which seems to have not been the focus of as much attention as biomedical sciences. We retrieved the ISSN of all journals classified in the Engineering subject area according to the ASJC classification scheme5 and made two queries to retrieve the corresponding articles from the Web of Science and Scopus databases, limiting the results to 2012-2015 publication years. -Journal:

-we chose 2 journals in biomedical field, Cancer Research and Tumor Biology, which are journals with high rates of retractions according to the RetractionWatch database, -we chose a multidisciplinary high-impact journal, Science, because some studies hypothesized that articles in high-profile journals are more prone to retractions [START_REF] Fang | Retracted Science and the Retraction Index[END_REF][START_REF] Furman | Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine[END_REF]) and also because the most prestigious journals publish the least reliable science according to [START_REF] Brembs | Prestigious science journals struggle to reach even average reliability[END_REF]). We used their ISSN to retrieve articles from the Web of Science and Scopus databases, limiting the results to 2012-2015 publication years.

-Author: with 3 high-profile researchers whose substantial part of their work has been officially retracted or corrected: Sarkar, Schön and Voinnet. We made 3 queries and retrieved all published articles of these authors with no publication date limit.

The corpus compilation is long and painstaking because retractions and errata are uncommon. It is therefore, necessary to have a very important initial database to obtain a sufficiently representative corpus. And they are also difficult to identify and locate online [START_REF] Hesselmann | The visibility of scientific misconduct: A review of the literature on retracted journal articles[END_REF][START_REF] Poworoznek | Linking of errata: Current practices in online physical sciences journals[END_REF]; Teixeira da Silva and Bornemann-Cimenti 2017) and in bibliographic databases. That is the reason why we used both Scopus and Web of Science to identify retraction notices and cross-checked with RetractionWatch database. To be as exhaustive as possible, we retrieved the erratum notices for the corpus we delineated, extracted the DOI or the article title from each notice title (see example), and then tagged the original articles. When the same article is associated to one or several erratum notices and is also retracted, we tagged it as retracted.

Example of a title erratum notice: "Erratum: Recruitment of RNA polymerase II by the pioneer transcription factor PHA4 (Science (2015) 350 (aad5928) DOI: 10.1126/science.aad5928)" Example of a title retraction notice: "Retraction notice to "Preparation and characterization of vertically aligned ZnO microrods on glass substrate" [Mater. Lett. 97 (2013) 71-73]"

Then we used data from two websites to retrieve two kinds of critiques and enrich the corpus at article level:

-Scite6 to retrieve the number of contradicting citations, -PubPeer7 to retrieve the number of comments. Scite was launched in 2019. According to their founders, it is a platform that allows finding if a scientific article has been supported, contradicted, or mentioned by subsequent studies. Scite automatically extracts citations from papers and classifies them using deep learning models and a network of experts. PubPeer was founded in 2012, it is a commenting website for centralized post-publication discussions; some of them have led to retractions or corrections.

We assumed that the number of contradicting citations retrieved by Scite and the number of comments posted on PubPeer are indicators of controversy and attempts to make a correction. As we are interested in citations, we focus here on articles with at least 1 citation according to Scite database, that is 45,811 articles (the corpus is avalaible for download [START_REF] Bordignon | Data for: "Self-correction of science: a comparative study of negative citations and postpublication peer review[END_REF]).

The corpus contains 1.28% of retracted articles (see Table 2 for distribution by sub-corpus).

Scite returns 3 values (number of mentioning, contradicting, and supporting citations); we reduce each of them to the total number of citations found by Scite (not the number of citations identified by Scopus or the Web of Science). PubPeer returns a number of comments, we process them as contradicting citations and thus also reduce them to the number of citations identified by Scite. Our objective here is not to test Scite's accuracy and we must assume that its detection of citation polarity, even if it is perfectible, gives a trend on which we can rely in this study. We also decided to group the retracted and corrected articles under a single label.

Corpus

Results and discussion

Negative citations

Consistently with previous studies, negative citations are very uncommon in the whole corpus: 0.29% (n=1904) on average citations per article. 97.02% are mere mentions and only 2.71% are supporting citations (Table 3).

There are slight differences according to the corpus tested. In the Engineering domain, authors seem to be accustomed to citations of mere mentions, which is consistent with the rate found for Schön's papers, whose work is close to this domain. On focusing on retracted or corrected articles, we aimed at checking whether papers that became officially known to be contentious generate more negative citations, either before or after they were flagged. But in fact, no such trend has been detected.

Contradicting

Figure 1. Average contradicting citation rates according to corpus and article status

Figure 1 shows the average contradicting citation rate for the 7 corpora and the difference between retracted/corrected articles and the others. There is no significant emerging trend. Nevertheless, here is what we can say:

-Aside from the Voinnet corpus (and the Schön corpus as well, but there is no reference point), retracted or corrected articles are less contradicted than "normal" articles. But for both Engineering and Science corpora, the rates are too close to each other to be interpreted as different. -Corpora from individual cases are those for which the contradicting citation rate of retracted/corrected articles is the highest; however, the profiles of these 3 cases are completely different.

On the whole, these results show that negative citations are not more frequent towards retracted or corrected articles. Generally speaking, as they are too under-used by the authors, they do not have the impact they could have in the process of correcting science. Incidentally, the total absence of citations of papers that are supposed to be cited (because they are closely related to the topic) must be at least as significant an indicator.

Comments on PubPeer

Assuming that comments on PubPeer are a form of contestation very close to a negative citation, we tested the articles of our corpora in PubPeer to check if questioning the content of a publication is more frequent outside the usual publication process, and more particularly on a dedicated platform that allows anonymous comments. Table 4 shows the percentage of PubPeer comments received in respect to the total number of citations (in Scite database) and also displays the difference between retracted/corrected articles and the others.

Retracted

Comparison between negative citations and comments on Pubpeer

Lastly, Figure 2 shows no correlation between the rate of negative citations and comments on PubPeer and confirms PubPeer as a possible place where the debate and contestation of findings can be carried out.

Figure 2. No correlation between contradicting citation rate and PubPeer comment rate

Our results show that Pubpeer, although external to the scientific publication process, contributes more to the correction of science than negative citations. The shallow analysis of some retraction notices also attests to this, since they often explicitly mention PubPeer as the place where the debate took place (e.g. "As raised on PubPeer, the article was found to contain images with signs of duplication and manipulation..."), thus feeding the decision to correct the record with a retraction or an official corrigendum. If negative citations are so uncommon, it is because it is difficult for an author to disagree publicly for fear of consequences. By allowing anonymity, Pubpeer overcomes this limitation, facilitating criticism and stimulating debate to improve the soundness of science. In some extreme cases, the scientific record is corrected through retraction or correction, but in other contexts, negative critiques at least provide a signal to the community that "scientific distortion" has been reported. It is essential that comments remain rigorously moderated by Pubpeer and that they stay "factual and verifiable", as stated in the platform guidelines. This is to avoid anonymous criticisms done with malicious intent in an attempt to weaken a competitor, as may be seen on non-scientific platforms, for example on consumer review websites [START_REF] Wu | Fake online reviews: Literature review, synthesis, and directions for future research[END_REF].

Since our data show that negative citations are extremely scarce and supportive citations are hardly more frequent, this could lead us to question the interest in tagging the polarity of in-text citations, as suggested by [START_REF] Garfield | Can Citation Indexing Be Automated? In Statistical Association Methods for Mechanized Documentation: Symposium Proceedings[END_REF]) and made possible by the CiTO ontology [START_REF] Peroni | FaBiO and CiTO: Ontologies for describing bibliographic resources and citations[END_REF]. But when we consider the growing popularity of Pubpeer and the motivation of contributors to elaborate on the contradiction they provide, we could imagine on the contrary that journals should require a minimum ratio of contradicting citations to more easily see the benefit of an article in relation to previous works and to more easily identify "scientific distortions". This idea is compliant with the Leiden Manifesto [START_REF] Hicks | Bibliometrics: The Leiden Manifesto for research metrics[END_REF] decrying that performance evaluation is now led by the data rather than by judgment. It is also in the line of (Griesemer 2020)'s proposition to "reimagine a role for judgment in the face of the data-driven metrics". Contradicting citations could be seen as a first step or a mere contribution to the rehabilitation of judgment. Further work is needed to explore the implications of that suggestion so that it will not fuel new opportunities to game the metrics.

Conclusion and future work

Our study is consistent with previous works and shows that contradicting citations are very uncommon: 0.29% of citations in the whole corpus we tested. Retracted or corrected articles are not more contradicted in scholarly articles than those that are neither retracted nor corrected but they do generate more comments on Pubpeer, presumably because of the possibility for contributors to remain anonymous. Consequently, post-publication peer review venues, and more specifically the comments found on it, although not contributing to the scientific literature, are a mechanism for correcting science. Future work, including surveys involving authors, may confirm that it is the fear of expressing their disagreement formally that motivates researchers to contradict their peers anonymously on Pubpeer.

Our study deals with a twofold scarce material (retracted/corrected articles and contradicting citations), it is thus also necessary to confirm our results with other corpora and to take into consideration the reason for the retraction or erratum. Indeed, plagiarism will not trigger criticism like data manipulation or scientific mistakes would, and errata sometimes refer to errors that do not affect the soundness of science (e.g., an erratum published for an error in the bibliography). We, therefore, plan to use the classification of reasons developed by RetractionWatch in their database and to carry out automatic analysis of the texts of the retraction and erratum notices. Lastly, our study is dependent on the accuracy of Scite, a recent tool whose reliability was not the subject of our study but which will have to be measured to go further. In a future project, we aim to re-evaluate Scite results and also to explore the possibility of further refining its classification by proposing nuances in contradiction. The text-mining of the comments on Pubpeer will also enable us to better understand the expression of contradiction and to fuel future tools to detect it.

We have also cautiously introduced the idea of strengthening the role of contradicting citations to rehabilitate the clear expression of judgment in scientific papers. We will explore the implications of that suggestion in further studies so that it will not be used as a new way to game the metrics. In the meantime, the tools that allow leaving a comment straight at the article level or those that alert an article has been commented on any platform, are undoubtedly a means of increasing the expression of contradiction on the one hand and the openness to criticism on the other.

Table 1 .

 1 Results of citing article disprove, put into question the data as interpretation of cited source; Results of citing article furnish a new interpretation/explanation to the data of the cited source. Summary of literature on citation classification schemes and their focus on contradicting or negative citationsCitations of retracted and corrected papersThe studies about citations of retracted papers have mainly explored temporal characteristics, focusing on postretraction rate of citations, and concluded that citations continue long after retraction date, mostly because authors are unaware of the new status of the document they once read and now cite[START_REF] Bornemann-Cimenti | Perpetuation of Retracted Publications Using the Example of the Scott S. Reuben Case: Incidences, Reasons and Possible Improvements[END_REF][START_REF] Garfield | The Impact of Fraudulent Research on the Scientific Literature[END_REF][START_REF] Kochan | The persistence of fraud in the literature: The Darsee case[END_REF][START_REF] Korpela | How long does it take for the scientific literature to purge itself of fraudulent material?: the Breuning case revisited[END_REF][START_REF] Mott | Assessing the impact of retraction on the citation of randomized controlled trial reports: an interrupted time-series analysis[END_REF][START_REF] Oppenheim | Highly cited old papers and the reasons why they continue to be cited[END_REF][START_REF] Teixeira Da Silva | Why do some retracted papers continue to be cited?[END_REF][START_REF] Wouters | The Schön case: Analyzing in-text citations to papers before and after retraction[END_REF]. Some studies went further and classified citations received by retracted papers, but they examined only a few papers, sometimes focusing on particular high-profile misconduct cases[START_REF] Bornemann-Cimenti | Perpetuation of Retracted Publications Using the Example of the Scott S. Reuben Case: Incidences, Reasons and Possible Improvements[END_REF][START_REF] Gabehart | An analysis of citations to retracted articles in the scientific literature[END_REF][START_REF] Garfield | The Impact of Fraudulent Research on the Scientific Literature[END_REF][START_REF] Kochan | The persistence of fraud in the literature: The Darsee case[END_REF][START_REF] Korpela | How long does it take for the scientific literature to purge itself of fraudulent material?: the Breuning case revisited[END_REF]

		Categories dedicated to negative or contradicting citations	Corpus description	Negative citation rate
	(Garfield 1964)	Correcting one's own work; correcting the work of others; criticizing previous work;
		substantiating claims; disclaiming the work or ideas of others (negative claims); disputing	-	-
		priority claims of others (negative homage)
	(Moravcsik and Murugesan 1975) Confirmative-negational (and also evolutionary-juxtapositional)	30 articles (Phys. Rev. Jour.) 14%
	(Chubin and Moitra 1975)	Partially negated, totally negated	443 papers (phys.)	5%
	(Spiegel-Rosing 1977)	Cited source is negatively evaluated; 2,309 citations	0.11%
	(Oppenheim and Renn 1978)	Theory or method not applicable or not the best one	23	1.5%
	(Small 1980)	Negative or refuted	-	-
	(Brooks 1985)	Negative credit	26 researchers interviewed	-
			about their motivations for
			citing references
	(Cano 1989)	Negational (according to (Moravcsik and Murugesan 1975) classification)	42 articles	2%
	(Shadish et al. 1995)	Negative citations	283 articles	5%
	(Wouters 1998)	Negational vs Confirmative	25,617 citations (soc. sci.)	2.83%
			360 conference articles	4.1%
			(comp. ling.)
	(Peroni and Shotton 2012)	Negative (corrects, qualifies, disagrees with, disputes, refutes, critiques, parodies,	-	-
		ridicules)
	(Stremersch et al. 2015)	Negation (according to (Baumgartner and Pieters 2003) classification)	24,632 papers	0.6%
	(Catalini et al. 2015)	Negative citations	15,731 articles	2.40%
			(Journal of Immunology)
	(Bertin and Atanassova 2016)	Negational citations	3,601,842 sentences with in-	Ratio of negative
			text reference extracted	citations through
			(from 80,000 PLoS articles)	the IMRaD
				article sections

[START_REF] Garzone | Towards an Automated Citation Classifier[END_REF]

Citing work totally disputes some aspect of cited work; Citing work partially disputes some aspect of cited work; Citing work is totally not supported by cited work; Citing work is partially not supported by cited work; Citing work disputes priority claims; Citing work corrects cited work; Citing work questions cited work 20 articles (phys. & biochem.) -(Teufel et al. 2006) Weakness of cited approach; Contrast/Comparison in Goals or Methods (neutral); Author's work is stated to be superior to cited work; Contrast/Comparison in Results (neutral); Contrast between 2 cited methods

Table 2 .

 2 Corpus description (number of articles per corpus only considering those with at least one citation in Scite database)

		Number of articles		Number of articles with at least one citation
		retrieved from WoS	Publication		in Scite database
		and Scopus (deduplicated)	years	Non-retracted/ corrected	Retracted/ corrected	Total
	Engineering	60,919	2013-2015	38,232	278	38,510
	Cancer Research	1,753	2013-2015	1,678	59	1,737
	Tumor biology	2,860	2013-2015	2,518	98	2,616
	Science	2,706	2013-2015	2,403	51	2,454
	Sarkar	454	all	290	47	337
	Schön	107	all	22	20	42
	Voinnet	141	all	82	35	117
	Total	68,936		45,225	586	45,811

Table 3 .

 3 Average citation rate for each corpus and according to Scite citation type

	Mentioning	Supporting

Table 4 .

 4 Average ratio of PubPeer comments to citationsOur data show that in general, corrected and retracted papers receive more comments on PubPeer than others. Furthermore, individual and mediatized cases (Voinnet and Sarkar corpora) generate the most comments on average in terms of the number of citations.

		/corrected (n=586)	Non-retracted/corrected (n=45,225)	Total	Standard deviation
	Engineering	4.79%	0.03%	0.06%	4.16%
	Cancer Research	4.75%	0.16%	0.31%	3.14%
	Tumor biology	0.00%	0.22%	0.21%	4.25%
	Science	5.17%	0.34%	0.44%	7.28%
	Sarkar	15.60%	1.81%	3.73%	13.40%
	Schön	-	-	-	-
	Voinnet	19.32%	3.13%	7.98%	23.49%
	Total	5.80%	0.07%	0.15%	4.67%

Committee on Publication Ethics, http://publicationethics.org/files/retraction%20guidelines.pdf

"Crossmark gives readers quick and easy access to the current status of an item of content. With one click, you can see if content has been updated, corrected or retracted and access valuable additional metadata provided by the publisher" (source: https://www.crossref.org/services/crossmark)

All Science Journal Classification, which is used to classify journals and conferences in Scopus

https://scite.ai

https://pubpeer.com

Acknowledgments

We are indebted to Joshua Nicholson from Scite for making citation polarity data available.


Bénédicte Pincemin

Serge Heiden

Matthieu Decorde

[TEXTOMETRY ON AUDIOVISUAL CORPORA]

Keywords: manual transcription, speech-to-text software, non-verbal communication, semi-directed interviews, audio recordings, video archive, Actualités françaises newsreels, XML, multimedia, TXM

Textometry is applied to audiovisual corpora, such as transcripts from semi-directed interviews, or the Actualités françaises newsreels archive. A workflow using an assisted or automatic transcription software is efficient to get a rich encoding. New features are added to the TXM software: a specialized import module based on Transcriber XML format, a utility to convert text transcripts to XML, and the MediaPlayer extension to watch the video segment corresponding to a word context selection. Methodological thoughts arise from this experience. It is highly relevant that textometry takes into account internal text structures (such as speech turns) and other metainformation (such as timecodes). Meta-information has to be displayed and available for processing without being mixed with contents. Another challenge is to integrate multiple interrelated representations. A back-tomedia feature is as fundamental as the back-to-text one to provide context to interpretation work.

Background and context

Since its foundations in the 80s, textometry aims at processing corpora of textual documents. More than simply counting words, it is interested in every dimension of language. This has been stimulated by the availability of natural language processing software (NLP) such as lemmatizers, and by the advent of international standards for data and text structuring (XML, TEI).

Few textometric tools still focus on plain text and deal with words as character strings only, sticking to the form they show in the primary data, so as to account for the text "as it stands", with as little recoding as possible (Lexico). Most tools have integrated a lemmatizer component (usually TreeTagger) to get a three-fold representation for every word: as a graphical word form, as a lemma (to group the various inflections of a same word), and as a part-of-speech category (Hyperbase, DtmVic, Iramuteq). This offered new insights and induced developments towards possible ways to combine these three levels of word description, from multi-level queries in the search engine facility to parameters implied in different steps of a calculus [START_REF] Pincemin | Lexicométrie sur corpus étiquetés[END_REF].

Regarding XML and TEI text encoding, the challenge is to integrate structural information and various other encodings into the textometric workflow (Trameur, TXM), in contrast to discarding tags or extracting plain text only (Hyperbase, DtmVic). A first level of processing consists in managing a tree-view of the text, so that it can be divided according to multiple partition levels. But XML encoding can also be used, inter alia, to integrate multimedia content: images, time-coding to relate to an audio or video recording, etc. TXM may still be the only one that has developed textometric solutions for such corpus modalities. Specialized software for audiovisual data, such as CLAN (MacWhinney, 2000) for "list of turns" transcription type or ELAN (ELAN, 2018) for "partition of timelines" transcription type, focus first on corpus annotating, then browsing and searching the annotated information, with few analytical functions. The TEITOK web platform [START_REF] Janssen | TEITOK: Text-Faithful Annotated Corpora[END_REF] has developed advanced features for creating, viewing and annotating TEI (or XML) corpora, with possible alignment to facsimile images or to audio files, but analytical tools are mainly visualization tools such as frequency graphs and result plots on a geographic map. A textometric approach can then really complement the investigation tool kit for such corpora.

In this paper, we would like to introduce to TXM 10-year experience in re-thinking textometry in the context of multiple textual modalities, and to present software components that have been produced for such corpora. This experience can be divided into two main steps, corresponding to the two main sections of the article.

Recording transcripts: data modeling and new textometric features (2010-2015)

Context and application cases

In 2010, fellow teachers from the EVS geography Lab (UMR5600) invited us to contribute to a Master's Course in Geography about methodology and tools for fieldwork. Students had to conceive a survey and collect data via semi-directed interviews they recorded. Then, students transcribed the interviews they had conducted. These textual data were all gathered in a corpus, and qualified with socio-demographic information. Then students were introduced to TXM and could systematically analyze the survey data with the help of the textometric tool, in order to address their geographic research questions.

In the first academic year 2010-2011, the survey was conducted in partnership with a local association, CeDRE (Cellule Développement Rural Emploi), that supported women seeking employment or leading a project. The geographic investigation dealt with relationships between territory, work, and personal skills. 33 women were interviewed, which produced a CEDRE corpus of 300,000 words (24 hours of speech). In the following year 2011-2012, the geographic study focused on the French fishermen working on Lac Léman. The geographic question was related to their relationship to their environment and their perception of how it had evolved along various dimensions (economic, natural, social,…). Nearly all fishermen could be interviewed (38 men or women in a total of 46), so that the LEMAN corpus included nearly 400,000 words for 34 hours of speech (Le [START_REF] Lay | Retour de pêche. Le métier de pêcheur à travers le discours des professionnels français du lac Léman[END_REF].

We met the same kind of corpus in a linguistic and educational research context. Verbal interactions in a class were filmed, then students and teacher communications were transcribed to be analyzed accurately [START_REF] Blanc | Tracer la procéduralisation dans le contexte interactionnel et multimodal d'une classe d'immersion[END_REF][START_REF] Heiden | P1S8 Corpus[END_REF].

Video corpora were also used for research in History. The Matrice project organized an exhibition about the liberation from extermination camps in 1945. One of the devices presented was a textometric terminal allowing simple browsing and analysis of the transcripts of 108 video testimonies edited by INA French national archive (https://entretiens.ina.fr/memoires-de-la-shoah) (more than 3 million words).

Using a transcription software to interactively edit richly-encoded textual data

Basic case: the starting point is an audio recording

In the case of the fieldword done by geographers, based on semi-directed interviews, the primary data are audio recordings. A relevant solution for a workflow is then to rely on the transcription community's experience and tools, by using a specialized software for the transcription task. Thus, one can get a richly encoded corpus through an ergonomically efficient user interface. We chose Transcriber software: it managed the transcription elements we needed in our corpora, it ran on Windows, Mac and Linux, it had a widespread diffusion in the scientific community, it offered a sound XML output format for audio transcription, and it was free and open-source, like TXM (Figure 1). The critical importance of applying rules to normalize word representations is well-known for any statistical approach on linguistic data [START_REF] Muller Ch | Principes et méthodes de statistique lexicale[END_REF]. In the same way, interpreting speech as text implies a tremendous number of choices,1 concerning the way of writing words and managing very common phenomena such as silences, intonation, onomatopoeia, repetition, incomplete or peculiar pronunciation, unknown words, external events, etc. These choices must be directed by explicit and shared conventions in order to get a homogeneous and controllable transcript (one knows how a phenomenon is annotated or if it has been left out of scope). In this respect, we adapted the Transcriber transcription guide to our context [START_REF] Heiden | Guide de transcription d'entretien avec Transcriber pour TXM[END_REF], taking into account what annotation could be relevant and available in TXM analysis, next step.

In a transcription software such as Transcriber, the basic transcript unit is the utterance, which corresponds to an audio segment, typically matching a sentence or the part of a sentence between two breaks. Through the graphical interface, the user plays the audio recording and sets its borders: then this audio segment can easily be played and repeated again when the user is typing the written form of what she hears. Utterances are embedded in speech turns, to which a speaker is assigned. If needed, it is possible to encode the fact that several people are speaking at the same time. Speech turns are placed in sections that organize the entire transcript. In a semi-directed interview typically, sections can be used to match the topics of the interview grid.

Facilities are also provided to log non-verbal communication (such as hesitations, laughs, emphasis on a word, gestures), or to describe events which interfere with the current communication and should be known at the analysis stage.

With such an interface, transcript is easily and systematically synchronized with audio recording, at the utterance level. The text representation is equipped with full timecodes, that embed in the text the alignment information to the media. Furthermore, variables characterizing the interviewee can be associated with each transcript, in the usual way for TXM: they are entered as a spreadsheet, in which a line is an interview, and columns are characteristics for the interviewee.

Nowadays, Transcriber is no longer maintained. Nevertheless, the Transcriber XML format is still an efficient mean to represent transcription data: either other software programs provide a Transcriber XML export of the data, or one can translate another format into the Transcriber one. Even if TXM already converts the Transcriber XML format into XML TEI internally for processing, we would still be interested in a general XML TEI based standard format for transcripts [START_REF] Schmidt | A TEI-based Approach to Standardising Spoken Language Transcription[END_REF].

Extension to the case of existing transcripts

The case stated above may not however be the most common one. Researchers often already hold transcripts they made previously. Most of these colleagues are not acquainted with specialized work on verbal interaction, so they used basic word-processing software to get a written version of their interview data. They may have implemented more or less formal transcription conventions, for example with consistent notations to introduce speech turns and speaker identifiers, or to insert timecodes. These notations may be associated with colors or typographic styles for ease of reading.

If these existing transcripts have been annotated following strictly constant rules, an automatic program can be developed to translate this kind of markdown document to a Transcriber XML document (as regards to information that this format can encode). Then, the Transcriber XML document can be imported in TXM (cf. 2.2.1). Such a utility program was released as a TXM macro named TextTranscription2TRS. It processes initial transcripts following specific conventions. A tutorial presents the transcription conventions and the way to call the utility program [START_REF] Heiden | Tutoriel d'import de transcriptions d'enregistrements dans TXM[END_REF].

Textometric enhancements thanks to transcription structuring and encoding

Some pre-processing may be required on Transcriber XML files: concatenation of transcripts when an interview was interrupted, normalization of speaker codes or of topical categories, speaker anonymization. Then, information encoded in the Transcriber XML representation can be used at various stages of the analytical process.

For the import procedure, that inputs the corpus in TXM and makes it available for textometric analysis, one can choose if the interviewer's words are indexed, and contribute to the corpus, or if only the words of the interviewed people are taken into account when searching, counting, and computing statistics. In both cases anyway, the interviewer's questions are shown in the transcript edition which is provided by TXM to see words in context: indeed, the full context of both interviewer's and interviewee's statements are necessary to interpret textometric outputs.

In the same vein, various contextual information are rendered in the transcript displayed in TXM, such as stage directions, with a characteristic typographical layout (Figure 2). All this information that was considered significant by the transcriber, is made available to contextualize interpretation, however they are not included in corpus words, they don't weigh in textometric operations. This concerns speakers' names and timecodes, section indications, comments about concomitant events, information about pronunciation or intonation, silences, etc. [START_REF] Heiden | Guide de transcription d'entretien avec Transcriber pour TXM[END_REF].

Of course, several information are also useful for analytical processes. Some are available as structures defining contexts (topical sections, speech turns, utterances). These TXM structures are qualified with properties that can be queried: the topic for a section, the speaker for a speech turn or an utterance. Every word is fitted with several features: not only its lemma and part-of-speech as provided by TreeTagger software, but also transcription features (word partially pronounced and how, uncertain spelling or unknown word, quoted speech). These information items are available in any analytical processing: for instance, one can partition the corpus into speakers, or kind of speakers (Figure 3); one can look for words that were annotated as truncated.

Timecodes give the opportunity to get an accurate access to the recording. Thus, the transcript does not replace the original media.

During textometric analysis, one may need to check not only the textual context, but also the audio or video context which brings complementary information. Actually, transcription implies choices and reduction of information, if not errors, because everything cannot be annotated, and on top of that, what is understood depends on the transcriber's perception, interpretation and experience. A transcription can neither be complete, nor neutral. Thanks to the availability of the source media, the necessary reduction applied by the transcription operation becomes more acceptable and less crucial. Nothing from the source media is definitively lost. [START_REF] Lay | Retour de pêche. Le métier de pêcheur à travers le discours des professionnels français du lac Léman[END_REF] To facilitate direct access to the source media, a special feature was implemented in TXM: as the "back to text" feature is necessary to correctly interpret textometric outputs, a "back to media" feature was developed. It provides hyperlinks from any concordance line to an embedded media player. An advanced user, Bertrand Gaiffe, extended this feature to access the media not only from concordance view but also from the full text edition within TXM. He shared2 a groovy script for TXM, linking utterances in the transcript text edition to the media. This states the centrality of a back-to-media feature, as fundamental as the back-to-text feature in any textometric approach.

Main outcomes

These first experiments with audio and video corpora, which recorded communication interactions or interviews, emphasized several important features for the textometric method. In every corpus, we had to deal with several speakers, so it was relevant to use a software program that could manage categorized divisions inside a text, and model speech turns with an assigned speaker. A second feature shows highly relevant, whereas it is still rarely implemented in textometric software: dealing separately with what is displayed for text reading, and what is processed in analytical operations. Speakers' names, timecodes, events interfering with speech or non-verbal communication should be available at some steps of the analysis (corpus partitioning, word contextualization and text reading) without being mixed to the lexical content submitted to statistical computations. As a third important feature, we note the link to the source media, so as to gain a fuller context and to overcome the unavoidable lacks and warps of the transcript.

Over the same period, we observed that these features, which were obvious in the case of multimedia corpora, turned out to be fundamental and should be generalized from various kinds of corpora. For instance, for a linguistic and philological study on medieval manuscripts too, one wants to separate critical notes from the original text, to display the text with notes but process the text without note contents; and a view to the source media (the manuscript scan) restores a more complete perception of the original data. We developed the XML TEI zero import module (XML-XTZ) to meet these new requirements in a generic manner. We enlarged the text edition feature to a synoptic edition feature, allowing aligned views of textometric data on the one hand, and a digital copy of the original source on the other hand. Yet, the Transcriber XML import still manages specialized corpora with speech turns, timecodes, and all specific information conveyed by the Transcriber format.

As regards software innovation, several new components augmented TXM software and were made publicly available: the XML-TRS import module, the TextTranscription2TRS utility with an associated tutorial, and a beta version of the MediaPlayer extension.

Exploration of a large corpus of videos (since 2018)

Context: historical analysis of French newsreels 1945-1969 (ANTRACT ANR project)

The ANTRACT project is dedicated to the transdisciplinary analysis of the Actualités Françaises newsreels. From 1945 to 1969, a national weekly edition was broadcast in movie theaters as a preamble to films. Within 10 to 20 minutes, it exposed a dozen of topics, as various as political, economic, social or cultural subjects as well as sports events. As a textual corpus, this corresponds to a few million words (we will see that diverse textual representations are possible). Sponsored by the ANR research agency, this collaborative project brings together the Center of social history of contemporary worlds (CHS, Paris 1 & CNRS), the French National Audiovisual Institute (INA), Eurecom Graduate school and research center in digital sciences (specialized in image processing), the computer science laboratory from Le Mans University (LIUM) (specialized in speech-to-text technologies), and our team in IHRIM for the text analysis component (Carrive et al., in press).

In a previous research project in 2014-2015, we had met an even bigger video corpus, composed of the automatic transcription of news bulletins and reports about World War II (WWII) that were broadcast on French national television from 1980 to 2010. In the context of the Matrice project, dedicated to interaction between individual and social memory, these news were used to account for collective representations of WWII events, in comparison with outputs from neurolinguistic investigations [START_REF] Gagnepain | Collective memory shapes the organization of individual memories in the medial prefrontal cortex[END_REF]. The transcripts were encoded in the Vocapia XML format, synchronized at the word level. A dedicated TXM import module was developed. The resulting TXM corpus of 3,766 texts containing 125 million of lemmatized words was optimized to only build lexical tables, that were then processed by external topic modeling software such as MALLET [START_REF] Mccallum | MALLET: A Machine Learning for Language Toolkit[END_REF]. At that time, no specific processing was applied to this corpus in relation with its audiovisual source.

With these corpora, we shifted from manual transcription of audio or video recordings to automatic transcription of video recordings, scaling up the size and multiplicity of data representation.

Structural considerations: a multi-layered representation with various segmentations, various modalities; and archival material redundancy and overlaps

The Actualités françaises case presented a complex set of data: various modalities are available in archives, with non homothetic physical units, and new secondary representations can be added through automatic audio or image processing (Carrive et al., in press). The corpus can be thought of as a compound and multimodal view on a common object to be observed. Data derive from three main materials: (a) the video recordings themselves, (b) their archival description in INA documentation system, and (c) paper documents related to the production process. a) Video recordings associate both audio and video modalities; the primary unit is the film reel, which does not match a logical unit such as the newsreel program or a topical news report. These data are digitized so as to provide files which can be handled by software. Speech-to-text applied to the audio track provides text transcripts of the voice-over commentary (LIUM). Automatic image analysis was run to detect and track the presence of some famous people (Eurecom) or to extract written data displayed on the screen (INA). Manual annotation from historian researchers can be added within the INA Okapi web platform, and modeled as layers aligned with the video timeline. b) A documentary database provides for every news report a scope and content note written by a librarian. The information unit here is the report. Its archival description is composed of numerous fields: some are categorical data (such as the broadcast date), some are sets of descriptors (for instance keywords about topics, places, people shown), and there are also several textual data (title, abstract, sequence description). These data were exported from the INA database as spreadsheets, one report on a line and one descriptive field in a column. The full documentary scheme includes 69 fields, among which we selected the ones that could be relevant for our corpus and questions, which made a focus on about twenty fields. c) A third source of data was the original paper typescripts which archive a written version of the voice-over commentary. These may not reflect exactly what is said, they may have been prepared before the report capture, or finalized later. INA scanned these papers, then ran an optical character recognition (OCR) application to obtain a digital text representation. The information unit here is neither the film reel (as for videos), nor the news report (as for archival description), but the page, whose boundaries don't match any of the previous units. In particular, a news report can run on several pages, and a typescript page can also include several news reports. INA works on the automatic pairing of image zones and news reports, based on textual similarity between OCR output and speech-to-text transcription.

From these data we have generated two different TXM corpora so far. The first one, AFNOTICES, is based on documentary descriptions. As these are professional and natively digital productions, textual data are very clear and reliable. They can be represented in TXM in a linguistically-aware and structured manner (words are identified and lemmatized, lists are encoded as such, keyword types are distinguished, etc.) so that queries can be very accurate (Figure 4). However, from an historian point of view, one works here on secondary data: librarians are mediators between the researcher and the original video data. Moreover, librarians themselves cannot escape time passing, what they notice in films and the words they use in free description fields are time dependent.

A second corpus was generated, AFVOIXOFF, that was based on the automatic transcription of the voice-over commentary recorded in the videos. As a first step, INA achieved an automatic recomposition of film reel digitized files so as to get one video file per news report (instead of one reel): this was necessary to connect to each news report film its date and other relevant metadata. A second step consisted in selecting a consistent set of news reports: if all the archive was taken as a corpus, concordance views showed significant overlapping in data. Focusing on national broadcast editions (and removing regional or international editions as well as unused supplementary material) appeared as the best way to get an overview of Actualités françaises without the redundancy that was difficult to manage in textometric analysis.

In its current version, the AFVOIXOFF corpus is more comprehensive than the AFNOTICES one, because it associates both the video transcription output as main text and the documentary description as metadata. However, querying textual documentary fields is less efficient in AFVOIXOFF, because these fields are processed as character strings, not as textual content. A new corpus design is to be elaborated, to integrate multiple full text representations. TXM already deals with aligned corpora, but the current dedicated import module only reads corpora in TMX format. The new textometric corpus modeling will have to be designed to allow the analysis of aligned, structured and multimedia representations. In such a corpus, new modalities could be added, for instance text and images from typescripts.

Strengthening and expanding the back-to-media feature

As the Actualités françaises videos are at the heart of the ANTRACT project, we invested in bringing to maturity the MediaPlayer TXM extension. Moreover, when working on automatic transcriptions which had to accommodate archival material, we could expect more errors than in manual transcriptions, so a convenient means to watch the source video is essential.

Considering the digital data scale of such a video corpus, four different access strategies were developed: media files can be stored within the TXM corpus, or in a local directory; they can alternately be accessed on a remote repository, with or without authentication control.

The definition of the timespan to be played for a given word focus was refined: either it is a window of n words before and m words after the selected words; or the segment played corresponds to the words of a transcript structure in which the word focus occurs (speech turn, or whole section for instance).

In addition to linking concordance lines to the media player, a new feature was developed to access the media from any text selection in the full-text display window. Preference parameters were defined to manage the display layout when several windows are to be shown together (Figure 5) : typically, a concordance window with the output of a word search, a text window providing the transcript context of a given concordance line, and a videoplayer window providing a targeted access to the media to get a full context for any word or text segment. The user sets her default preference according to the data characteristics, to her digital equipment, and to her analytical patterns: does the mediaplayer window open above, under, or on the right-or left-hand side of the text to which it is linked? Another parameter manages the position of the text opened with a back-to-text hyperlink from a concordance line. Afterwards, every window position may be adjusted by drag-and-drop if needed.

Furthermore, the setup process was simplified, and its compatibility was extended, thanks to the use of JavaFX technology instead of relying on the external VLC component.

New analytical possibilities with an aligned multimodal corpus

Innovation comes from dealing with several representations together. A first benefit from this multiple representation is that one representation contextualizes another. In a search query, one can cross criteria from several representations: for instance, one can select news reports which show ruins (according to the documentary description) and whose comments mention any word related to building and novelty (in the voice-over commentary) (Figure 5). 3 The synoptic view of video and transcript, or of words in context and dates, are invaluable for an informed interpretation.

A second benefit is related to getting a better control on observations: combining representations allows cross-checking, which is especially useful in a framework where some data are secondary descriptions (here, documentary descriptions, that are mediated by librarians) or automatic productions (here, speech-to-text transcriptions). For instance, if one finds an amazingly high frequency of a given word in the voice-over comments, one can check if it was actually pronounced in the videos, or if it is due to a transcription error. This checking activity leads to correction needs. In the Antract framework, edition of transcripts or annotation data can be provided either within TXM through its new annotation tools, or with the help of the INA Okapi platform, which is dedicated to video corpora creation and annotation.

A third benefit is that some statistical induction can be made from the aligned data, applying the resonance principle [START_REF] Salem | Introduction à la résonance textuelle[END_REF]. For instance, when a crowd is shown, without any mention of a president, what is it told about?4 In TXM, this kind of calculus is achieved via specificity analysis on a subcorpus; and new corpus operators (intersection, union) have been implemented to allow complex definition of subcorpora. The resonance approach is quite promising for our Actualités françaises data, to study implicit relationships, or unexpected deviations, between what is told and what is shown; or between what has been noted in documentary descriptions and what occurs in the film itself; or even between the written archived commentary and what was actually recorded and displayed. This could be a inspirational entry point for considerations about textual genres and semiotic genres.

Main outcomes

The context of a large and complex video archive raised new questions about the implementation of a textometric approach. We had to develop and precise the definition of textual units, the various kinds of contexts involved, and the alignment possibilities, thanks to which various representations could contextualize one another. The textometric corpus could be modeled as multiple views on an object which has to be defined, because it is not given as a unique and obvious reality. Relevant data include both source data and derived data, especially textual representations automatically computed from audio data or image data (analysis of the video part of the films as well as scans of paper documents). This experience emphasizes the corpus design phase, the need for accessing various types of source data (searching, visualizing), and the relevance of a powerful query language and of the resonance statistical feature in order to analyze such multi-representation corpora.

These thoughts about fundamental aspects of data modeling and processing were implemented in different parts of TXM. The Transcriber XML import module was extended in order to manage synchronization at the word level, and to organize metadata with a better rendering for textual metadata conveying parallel textual representations. The MediaPlayer extension matured and gained robustness, flexibility and usability, thanks to renewed technologies, parameters, and configuration options. In particular, parameters were introduced to manage an optimal arrangement of windows for complex synoptic analyses.

Conclusion

These experiments and innovations in the textometric processing of audiovisual corpora show that the special requirements of this kind of corpora are an opportunity to both extend software capabilities and generalize a few elements of textometric methodology.

Our first experiments with manual transcription of audio or video recordings, were extended to large multimedia data with automatic processing for text extraction. In every case, data structuring and structure-aware processing were key features. Textual data may be divided into speech turns; speakers' names or timecodes have to be encoded and available for reading and analysis without being mixed with speakers' statements. Several textual or audiovisual representations may be relevant, and have to be organized so that cross-searches are available for augmented context, for better control of errors in data, and for new transmedia investigation possibilities.

As regards to software, TXM was augmented with a new import module based on the Transcriber XML format, the TextTranscription2TRS TXM utility was provided and documented to help the automatic conversion of existing transcripts towards the Transcriber format for the XML-TRS import module, and the MediaPlayer extension was developed to implement a "back-to-media" feature, providing a hypertextual efficient and targeted access to the source recording. At first glance, playing the video corresponding to a textometric corpus might seem like a technology gimmick, something nice and brilliant but simply added aside. Quite the contrary, this "back-to-media" feature is as essential for analysis and interpretation as the undisputed "back-to-text" function.

The textometric perspective on audiovisual corpora still gives a central position to textual data. Several textual representations, following rules from different text genres, compose a complex representation of a research object and connect media sources. Contextualization is also still confirmed as a key concept for semantic analysis, as its encoding get diversified: situational information may be recorded not only with metadata, but also with comments annotated in the text and with a multiplicity of parallel representations.

This research has benefited from the ANTRACT ANR project (ANR-17-CE38-0010) and the Matrice Equipex project (IA 10-EQPX-0021).

Figure 1 .

 1 Figure 1. Transcriber graphical user interface: example of a transcription for the CEDRE corpus.

Figure 2 .

 2 Figure 2. LEMAN corpus: Rendering of a transcript in TXM after an XML-TRS import: speaker names, timecodes, section titles, comments, etc. are displayed and styled, but they are not searched and counted with corpus words.

Figure 3 .

 3 Figure 3. LEMAN Corpus (fishermen interviews): 10 top-specific verbs according to the size of the fishing company (from left to right: large, medium-sized, small) (cf. Le[START_REF] Lay | Retour de pêche. Le métier de pêcheur à travers le discours des professionnels français du lac Léman[END_REF]

Figure 4 .

 4 Figure 4. AFNOTICES Corpus: an index (upper left) lists the location descriptors; one of them is selected and viewed in a concordance (lower left); a double-click on a concordance line displays the corresponding text notice.

Figure 5 .

 5 Figure 5. Corpus AFVOIXOFFV02: concordance with back-to-text and back-to-media hyperlinks and synoptic view.

Similar considerations happen in the edition of medieval manuscripts. This is even an overall philologic question: text is not unique and clear, decisions are necessary to establish it. Digital humanities reveal it again through encoding choices, so that a digital philology has to be developed(Guillot et al.,

2017).

Mail sent on txm-users list, November 2, 2017, 2:42 PM.

JADT 2020 : 15 es Journées internationales d'Analyse statistique des Données Textuelles

Query for AFVOIXOFFV02 corpus: [_.div_sequences=".*ruines.*" & word=".*constr.*|.*nouv.*"]

This example is illustrated in(Carrive et al., in press).