domain: shs.litt shs info

text:
Liana Ermakova Hcti -Ea

Jean Valère Cossu

Josiane Mothe

A Survey on Evaluation of Summarization Methods

Keywords: automatic summarization, text compression, evaluation campaigns, assessment metrics, extraction, extractive summarization, ROUGE Highlights

The increasing volume of textual information on any topic requires its compression to allow humans to digest it. This implies detecting the most important information and condensing it. These challenges have led to new developments in the area of Natural Language Processing (NLP) and Information Retrieval (IR) such as narrative summarization and evaluation methodologies for narrative extraction. Despite some progress over recent years with several solutions for information extraction and text summarization, the problems of generating consistent narrative summaries and evaluating them are still unresolved. With regard to evaluation, manual assessment is expensive, subjective and not applicable in real time or to large collections. Moreover, it does not provide re-usable benchmarks. Nevertheless, commonly used metrics for summary evaluation still imply substantial human effort since they require a comparison of candidate summaries with a set of reference summaries. The contributions of this paper are three-fold. First, we provide a comprehensive overview of existing metrics for summary evaluation. We discuss several limitations of existing frameworks for summary evaluation. Second, we introduce an automatic framework for the evaluation of metrics that does not require any human annotation. Finally, we evaluate the existing assessment metrics on a Wikipedia data set and a collection of scientific articles using this framework. Our findings show that the majority of existing metrics based on vocabulary overlap are not suitable for assessment based on comparison with a full text and we discuss this outcome.

Introduction

In recent decades, data explosion has obliged people to manage the constantly growing amount of information. Thus, more data has been created in the past two years than in the entire history of human existence [START_REF] Marr | Big Data: 20 Mind-Boggling Facts Everyone Must Read[END_REF]. The increasing volume of textual information in its various forms such as news articles, comments or posts on social networks poses new challenges for those who aim to understand the storyline of an event [START_REF] Jorge | ECIR 2018: Text2story Workshop -Narrative Extraction from Texts[END_REF]. Despite recent significant progress in the domains of information extraction and text mining, the problem of constructing consistent narratives has not yet been solved. New trends in the area of Natural Language Processing (NLP) and Information Retrieval (IR) have emerged such as narrative summarization, story evolution and shift detection, and evaluation methodologies for narrative extraction. They aim to find methods for detecting the most important information and condensing it for users.

Summarization is, by far, the most concrete and most recognized kind of text condensation [START_REF] Ansi | American National Standard for Writing Abstracts[END_REF][START_REF] Torres-Moreno | Resume automatique de documents : une approche statistique[END_REF]. A summary is a "condensed version of a source document having a recognizable genre and a very specific purpose: to give the reader an exact and concise idea of the contents of the source" [START_REF] Saggion | Generating Indicative-Informative Summaries with SumUM[END_REF].

Despite some progress over the last years in information extraction and text summarization, two major challenges remain: the problem of generating consistent narrative summaries and consequently how to evaluate summarization methods and obtained summaries. In order to compare the algorithms for automatic summarization, it is crucial to have a reliable metric for summary quality evaluation.

Summaries can be evaluated considering different facets, but primarily in terms of informativeness and readability [START_REF] Owczarzak | Overview of the TAC 2011 summarization track: Guided task and AESOP task[END_REF][START_REF] Bellot | INEX tweet contextualization task: Evaluation, results and lesson learned[END_REF]. In this paper, we focus mainly on informativeness assessment since this aspect is the most automatized in contrast to readability that is still usually evaluated manually. Manual assessment is expensive, subjective and not applicable to real time scenarios or large collections. Commonly-used metrics for summary evaluation involve substantial human effort in any case. Indeed, they require the comparison of the candidate summaries with a set of reference summaries, although there have been some attempts to use the full text as a reference [START_REF] Louis | Automatically assessing machine summary content without a gold standard[END_REF][START_REF] Ermakova | Is the Abstract a Mere Teaser? Evaluating Generosity of Article Abstracts in the Environmental Sciences[END_REF][START_REF] Ermakova | GRAD: A Metric for Evaluating Summaries[END_REF].

In this paper we focus on evaluation methodologies for narrative extraction. Our first contribution is to propose a comprehensive overview of existing metrics and data sets for summary evaluation. An assessment metric allows us to compare different approaches. However, in turn, the assessment metric should be meticulously chosen. Hence, a second contribution of this paper is to introduce an automatic framework to evaluate assessment metrics that does not require any human annotation. Finally, as a third contribution, we evaluate the main existing metrics from the literature on two different data sets: a Wikipedia data set and a collection of scientific articles. Our experiments show that the majority of existing metrics based on vocabulary overlap are not suitable to assess the quality of produced summaries when compared to the original full text.

The rest of the paper is organized as follows. Firstly, in Section 2, we provide a description of the most important international evaluation programs and collections for text summarization. Then, in Section 3 we present an overview of existing assessment metrics. The framework for the evaluation of assessment metrics is described in Section 4. Section 5 discusses the main challenges in the domain of summary evaluation revealed in the previous section and concludes this paper.

Evaluation Programs and Collections

An acute need for automatic text compression has led to the emergence of a number of evaluation programs sponsored by research and governmental institutions. These programs include reference data and metrics.

Shortly after the Internet explosion in the late 90s, the first annual summary evaluation campaign DUC (Document Understanding Conference) appeared. The first evaluation programs were general-purpose, while more recent ones cover specific domains, e.g. biomedical literature, search engine snippets, or are targeting specific applications, e.g. tweet contextualization within the Initiative for the Evaluation of XML Retrieval (INEX).

These assessment programs provide evaluation frameworks containing text collections, evaluation metrics, and gold standards (also called reference summaries or ground truth). In some cases, the evaluation frameworks are automatic (not requiring a gold standard) or semi-automatic (exploiting ground truth) and, thus, can be re-used by non-participants to the evaluation campaign to assess their approaches.

In this section, we discuss the major international evaluation campaigns in detail, namely Document Understanding Conference, Text Analysis Conference, Tweet Contextualization, Snippet retrieval, Sentence Ordering Assessment, and Stream Summarization.

Document Understanding Conference (DUC)

In 2000-2007, the Document Understanding Conferences (DUC) 1 1991-1994, Federal Register 1994, FBIS 1996, Los Angeles Times 1989-1990). 4 tasks were proposed:

1. very short summaries ( 10 words, no specific format other than linear) manually evaluated by coverage and usefulness; 2. short summaries focused by events ( 100 words) manually evaluated by quality and length-adjusted coverage;

3. short summaries focused by viewpoints ( 100 words) manually evaluated by quality and lengthadjusted coverage; 4. short summaries in response to a question ( 100 words) manually evaluated by quality, length-adjusted coverage, and responsiveness.

In 2004 DUC organized 5 tasks:

1. very short single-document summaries; 2. short multi-document summaries focused by TDT events; 3. very short cross-lingual single-document summaries; 4. short cross-lingual multi-document summaries focused by TDT events; 5. short summaries focused by questions.

The summaries in tasks 1-4 were evaluated by ROUGE metric [START_REF] Lin | Automatic evaluation of summaries using n-gram co-occurrence statistics[END_REF]. The summaries of the task 5 were manually evaluated intrinsically for quality and coverage and by their responsiveness to the question.

In 2006-2007, the main task was real-world complex question answering, namely, given a topic and a set of 25 relevant documents, to produce a 250-word summary answering the question in the topic statement [START_REF] Dang | Overview of duc 2006[END_REF]. There were 45 topics in the test data. The documents for summarization were taken from the AQUAINT corpus, made of news articles from the Associated Press and New York Times (1998)(1999)(2000) and Xinhua News Agency (1996)(1997)(1998)(1999)(2000). Reference summaries were written by 4 different NIST assessors. In 2005, the task was also answering complex questions, but the documents were chosen from Financial Times of London and Los Angeles Times [START_REF] Dang | Evaluation of question-focused summarization systems[END_REF].

The update task was introduced in 2007 and aimed to produce 100-words multi-document update summaries of news articles under the assumption that the user has already read earlier articles. This task was kept in TAC campaign.

The DUC campaign saw large improvements in participants' automatic systems with regards to annotators' proposals. However, there were limitations to participants' summaries since they were topic-dependent and extractive. The DUC data set contains: documents, manually created summaries, automatic baselines, summaries submitted by the participants, evaluation results, additional supporting data and software. The In 2008, DUC became a Summarization track in the Text Analysis Conference (TAC).

Summarization track in the Text Analysis Conference (TAC)

The Text Analysis Conference (TAC)2 was a series of evaluation campaigns organized to promote research in Natural Language Processing (NLP) by providing a large test collection, standard evaluation framework, and a forum to share results. Being a multi-task workshop, TAC was made of tracks each focusing on a particular NLP problem. TAC was interested in end-user tasks, as well as evaluation methods within the context of end-user tasks.

The Summarization Track evolved during the TAC campaigns [START_REF] Owczarzak | Overview of the TAC 2011 summarization track: Guided task and AESOP task[END_REF]. In 2008, Summarization Track included:

• Update Task. The goal was to write a short summary of a set of news articles, assuming that the user is already familiar with a given set of earlier articles. The summaries were evaluated for readability and content by the Pyramid Method (see Section 3 for details). Documents were issued from the AQUAINT-2 collection.

• Opinion Pilot. This task aimed at writing short coherent summaries of opinions from blogs. Summaries should be based either on the text snippet output created by Question Answering (QA) systems as a response to questions from the TAC QA Track, or on the associated documents. Informativeness evaluation was based on the Nugget Pyramid Method. Documents were taken from the Blog06 collection [START_REF] Macdonald | The trec blogs06 collection: Creating and analysing a blog test collection[END_REF].

The Update Summarization task continued in 2009.

In 2010, the summarization task focused on Guided Summarization: the goal was to write a short summary of a set of newswire articles for a given topic from predefined categories. A summary had to include all aspects found for a category, e.g. "Accidents and Natural Disasters: what happened; date; location; reasons for accident/disaster; casualties; damages; rescue efforts/countermeasures". Summaries were evaluated for readability, informativeness, and overall responsiveness (see Section 3) .

The 2011 Summarization Track was composed of three tasks:

• Guided Summarization. In contrast to 2010, redundancy in the update summaries was evaluated by the Pyramid score over initial summaries. Test documents were taken from the TAC 2010 KBP Source Data, rather than AQUAINT and AQUAINT-2. Source documents for summarization came from the newswire subset of the TAC 2010 Knowledge Base Population (KBP) Source Data. The collection consists of articles from the New York Times, the Associated Press, and the Xinhua News Agency newswires in 2007-2008.

• Automatically Evaluating Summaries Of Peers (AESOP). In addition to metrics summary informativeness (Pyramid, Responsiveness), AESOP targeted readability. Correlations at the summary level (within each topic) was reported.

• Multiling Pilot Task. This task aimed at promoting the use of multi-lingual algorithms for summarization, e.g. transforming an algorithm or a set of resources from a mono-lingual to a multi-lingual version.

Participants of the Guided Summarization Task had to write 100-word summaries of a set of 10 news articles for a given topic from a predefined category:

• accidents and natural disasters;

• attacks;

• health and safety;

• endangered resources;

• investigations and trials.

Summaries should cover all given important aspects for a category. The data set was composed of 44 topics. A topic had ID, category, title, and 20 relevant documents divided into 2 sets: Set A and Set B. All the documents in Set A chronologically preceded the documents in Set B. No topic narrative were provided, but the category and its aspects expressing the information need of a reader.

An "update" Guided Summarization Task was aimed at written a 100-word "update" summary of a subsequent 10 news articles for the topic, under the assumption that the user has already read the earlier articles. Thus, the goal of the "update" Guided Summarization Task was to recognize new (non-redundant) information in the second set of documents on the same topic.

TAC AESOP task was introduced in 2009. The AESOP task focused on metrics for content evaluation, such as overall responsiveness and Pyramid scores. Within the AESOP task, a collection of automatic evaluation tools was built. AESOP was running until 2011. In 2009 and 2010, AESOP task was focused on developing automatic metrics for content summary evaluation, more precisely measuring the average quality of summarization systems. In 2011, participating metrics should also be applicable on the level of individual summaries. Besides content evaluation, in 2011 the readability measuring should be also considered. Participants obtained the test data from the TAC Guided Summarization task, the humanauthored and automatic summaries from that task, namely:

• topic statements;

• two sets of 10 documents for each topic;

• four human written reference summaries for each document set;

• summaries evaluated in the TAC Guided Summarization task;

• list of topic categories and target aspects.

The output of participants automatic metrics was compared against manual metrics: (1) the Pyramid score for content evaluation; (2) Overall Readability for linguistic quality evaluation; and (3) Overall Responsiveness, which is a combined measure for content and linguistic quality evaluation. Pearson's, Spearman's, and Kendall's correlation coefficients between scores assigned by each automatic metric and the three manual metrics were computed. The assumption under the evaluation framework was that a good automatic metric should make the same significant difference between automatic summarizing systems as the manual metrics, but should not provide a ranking to two summarizing systems contradicting the manual metric. Thus, each run was evaluated by (1) the correlation with the manual metric and (2) its discriminative power compared with the manual metric. TAC Summarization Track was not running in 2012 and 2013. In 2014, TAC restricted the summarization task with only biomedical literature in the Biomedical Summarization track. Nowadays, this task is extremely important since the integration of big data into health care could save around $1000 a year per person according to recent studies [START_REF] Marr | Big Data: 20 Mind-Boggling Facts Everyone Must Read[END_REF].

The TAC test data are distributed by the Linguistic Data Consortium (LDC) 3 , an open consortium of universities, libraries, corporations and government research laboratories.

The majority of campaigns are focused on evaluation of summarization methods, while AESOP track aims at evaluating assessment metrics by measuring the correlation with the manual metrics and the discriminative power compared with the manual metric. In contrast to that, our framework is automatic and does not require human assigned scores.

Issue of Streams Summarization

The idea of summarizing streams of information related to a given event is not a recent research issue.

However, the rise of social medias and user generated content like micro-blogs gives a new opportunity to tackle the lack of framework.

It recently attracted several research teams in Europe used to focus on automatic summarization of Events or in TREC where it has its own track http://trecrts.github.io/ [START_REF] Allan | Trec 2017 common core track overview[END_REF].

From scheduled events

Indeed, to deal with the overflow of less relevant information, especially in the real-time news streams when human reference is hardly available within a short period (sports events, awards ceremonies or during elections early results phase), some researchers tried to use social media extraction to build so-called summaries.

For instance, the work presented by [START_REF] Zubiaga | Towards real-time summarization of scheduled events from twitter streams[END_REF] can be extended to any similar event where relatedness of social media extracts can be compared to official reports. These works as well as [START_REF] Shen | A participant-based approach for event summarization using twitter streams[END_REF] intended to be representative but their evaluation was event-detection centered since their purpose was to find the most relevant tweet with regards to the given event. By the way, they not intended to generate a story from the events.

The main drawback of these data sets is that although they are reusable, it may be difficult to compare with existing work since there is no given reference and each researcher can manually select her/his evaluation criterion (e.g. events detection and coverage). Moreover, it could be difficult to replicate such experiments since the context evolved through time.

Real-Time Summarization

From 2016 to 2018, TREC Real-Time Summarization (RTS) focused on different data set and considered, users with multiple information needs. Systems had to help users to keep up to date on their topics of interest. RTS can be considered as an extended version of real-time filtering task in the TREC 2015 Microblog and Temporal Summarization tracks which were on their last edition while both were running from 2010 and 2013.

Two scenarios had been proposed within RTS:

• the first one focuses on push notifications (immediate selection of relevant information, for instance).

For instance: notify me when an accident involving autonomous vehicle occurs;

• while the second one was based on email (daily recap of what happened yesterday). For instance, build a digest of all tweets published yesterday related to Apple products bug fixes.

TREC proposed experiments and evaluation using crowdsourced assessments. However, the assessment process evolved through years going through push and fully controlled system to semi-blind, pull-based process [START_REF] Lin | Overview of the trec 2016 real-time summarization track[END_REF]. The topics considered for each edition are still available as well as the "Evaluation Broker" API's code. Except tweets from the evaluation periods that have been removed by their authors, this framework could be re-usable.

INEX/CLEF Tweet Contextualization and CLEF Microblog Contextualization

The rise of user generated content and short texts like micro-blogs or tweets has led to new trends in summarization. Recently, the idea of contextualizing this content to improve tweet understanding was introduced into challenges such as INEX/CLEF Tweet Contextualization (TC) and CLEF Microblog Contextualization [START_REF] Bellot | INEX tweet contextualization task: Evaluation, results and lesson learned[END_REF].

Contextualization Track has grown as an evaluation forum at the intersection of text summarization and information retrieval. TC aims at automatically summarizing large text resources to contextualize (i.e. help to understand) short text passages (e.g. tweets).

Several systems, such as Linguamatics [START_REF]Text mining software, text analytics software, big data management[END_REF], proposed to automatically retrieve the wide range of vocabulary used in tweets, including topic tags, and use linguistic processing to collect and summarize the thousands of ways people have of saying the same thing. Recently, [START_REF] Meij | Adding semantics to microblog posts[END_REF] mapped a tweet into a set of Wikipedia articles, but instead of a summary they provided users with a set of related links [START_REF] Meij | Adding semantics to microblog posts[END_REF]. [START_REF] Sanjuan | Overview of the inex 2011 question answering track (qainex)[END_REF] .

SanJuan et al. took another step by introducing Tweet Contextualization (TC) Track

In 2011, the Question Answering Track aimed to evaluate TC in terms of (1) relevance of the retrieved information to tweets, and (2) readability of the presented results [START_REF] Sanjuan | Overview of the inex 2011 question answering track (qainex)[END_REF]. In 2012, this track was renamed Tweet Contextualization. In 2012, answers needed to be a concatenation of textual passages from an external textual resource providing background information that helped to understand the tweet content. Providing an answer to these kinds of task (retrieval and concatenation of relevant textual passages), thus, implies using text summarization.

Associated test collections are described in the table 1.

In 2011, the query data set included 132 tweets. Each tweet consisted of the identifier (id), the title (title), and the first sentence (txt) of a New York Times article published in July 2011.

For each tweet, participants had to provide a summary of up to 500 words in TREC format that is contextualizing the tweet, i.e. answering the question "what is this tweet about?". The summary had to contain as much relevant information as possible and not include irrelevant or redundant passages.

The summary had to be made solely of extracts from the XML dump of English Wikipedia articles (April 2011): 3,217,015 non-empty pages in total. All notes, history and bibliographic references were removed. Thus, a page was composed of a title (title), an abstract (a) and sections (s). Each section had a header (h)). Each abstract and all sections contained paragraphs (p) and entities (t) referring to other pages.

The summaries submitted by participants were compared with each other, with the baseline summary made of sentences (BaselineSum) and with the key terms (BaselineMWT). The baseline system was based on the Indri index with stemming and without stop words (language model). Part of speech tagging was performed by TreeTagger. The summarization algorithm was TermWatch [START_REF] Sanjuan | Overview of the inex 2011 question answering track (qainex)[END_REF].

In 2012, the text corpus was presented by an updated Wikipedia dump from November 2011. The query set was dramatically changed. It consisted of approximately 1000 real tweets written in English collected from informative accounts such as @CNN, @TennisTweets, @PeopleMag, @science etc. However, the task remained the same: to provide a summary up to 500 words in the TREC format. In 2013 there were 598 tweets in English to be contextualized from the Wikipedia dump of November 2012.

For all test collections, 50 tweets4 were selected to evaluate the informativeness of the summaries [START_REF] Sanjuan | Overview of the inex 2011 question answering track (qainex)[END_REF].

For each of those topics, all submitted passages were merged into a pool. Passages were sorted in alphabetic order and therefore each passage was judged whether it was relevant independently from others. Submitted summaries were compared with the corresponding pools of relevant passages. In 2011 summaries were also evaluated according to the overlap with the original New York Times articles. In 2013 the informativeness was estimated as the overlap of a summary with 3 pools of relevant passages [START_REF] Bellot | Overview of INEX[END_REF]:

• prior set (PRIOR) of relevant pages selected by organizers (40 tweets, 380 passages);

• pool selection (POOL) of the most relevant passages (1,760) from participant submissions for 45 selected tweets;

• all relevant texts (ALL) merged together with extra passages from a random pool of 10 tweets (70 tweets, 2,378 relevant passages).

In 2014 there were 240 tweets in English collected by the organizers of CLEF RepLab 2013. In 2014 participants should provide a context to tweets from the perspective of the related entities. Tweets were at least 80 characters long and did not contain URLs. A tweet had the following annotation types: the category (4 distinct), an entity name from the Wikipedia (64 distinct) and a manual topic label (235 distinct). The context had to explain the relationship between a tweet and an entity. As in previous years it should be a summary extracted from a Wikipedia dump.

In 2014, 2 gold standards (1/5 of the topics/tweets) were used:

• pool of relevant sentences per topic/tweet (SENT);

• pool of noun phrases (NOUN) extracted from these sentences together with the corresponding Wikipedia entry.

The pool of relevant passages was constructed through the summaries submitted by the participants of the track. Only these passages were judged as relevant or not. All other passages are considered as irrelevant. Thus, a brand new system which is able to catch relevant but not judged passages can be potentially underscored due to this bias. The same remark can be applied to traditional ad-hoc information retrieval campaigns since measures like MAP (mean average precision) consider all non-judged results as irrelevant (see for example TREC Robust data5 ).

INEX Snippet Retrieval Task

Search engines return to a user an immense volume of results that it is impossible to read. Therefore, to define whether a web page is relevant to a query without clicking on a link, a search engine provides a user with snippets. Snippets are small text passages appearing under every search result.

For the Snippet Retrieval Track 2012, the data collection consisted of the dump of the Wikipedia of October 2008 annotated with YAGO [START_REF] Schenkel | YAWN: A Semantically Annotated Wikipedia XML Corpus[END_REF] and 35 topics. Participants should provide 20 snippets per topic limited to 180 characters [START_REF] Trappett | Overview of the INEX 2011 Snippet Retrieval Track[END_REF]. In 2013, the Snippet Retrieval track was using the same document collection as the Tweet Contextualisation track, based on a dump of the English Wikipedia from November 2012. The description, and narrative (intent) provided the idea of the user information need.

The main drawback of this data set is that it is not re-usable after the track since the entire evaluation was manual.

Collections for Sentence Ordering Assessment

Automatic text generation systems, particularly multi-document extractive summarization, face sentence ordering problem [START_REF] Bollegala | A bottom-up approach to sentence ordering for multidocument summarization[END_REF][START_REF] Bellot | Overview of INEX[END_REF]. While evaluating informativeness of automatically produced content without involving manual process interests numerous research teams, the evaluation of readability is still mainly an expert work. However, there exist frameworks that allow the assessment of sentence ordering, i.e. readability, since sentence ordering is the only way to influence the readability of an extractive summary.

Two corpora are widely used for sentence ordering assessment:

• airplane Accidents from the National Transportation Safety Board,

• articles about Earthquakes from the North American News Corpus [START_REF] Lin | Automatically evaluating text coherence using discourse relations[END_REF][START_REF] Elsner | A Unified Local and Global Model for Discourse Coherence[END_REF][START_REF] Louis | A coherence model based on syntactic patterns[END_REF].

Each of these corpora has 100 original texts and for each document 20 permutations (2,000 in total).

Evaluation Measures

In this section, we present an overview of the metrics used to evaluate summaries that have been generated automatically. When applicable, we mention the evaluation program or collection from Section 2 the metric has been used for. Of course, performing an exhaustive survey seems hardly possible due to the number of related works. However, we consider a wide range of studies and adopt a high-level approach, focusing on the relevance of the measures depending on the context in which the summaries are generated.

We organize them in a typology with regard to the evaluation use case and resources available.

Informativeness Evaluation

Questionnaire-based Metrics

Summaries may be evaluated according to compression rate, i.e. proportion of summary length over full text length, or retention rate, i.e. proportion of the retained information [START_REF] Gholamrezazadeh | A comprehensive survey on text summarization systems[END_REF].

A good summary should have low compression rate and high retention rate. Compression rate is welldefined and can be easily computed while retention rate estimation is more problematic since it involves less formalized concepts. To measure the retained information, one assessor team develops a set of questions based on the input texts, while another group answers these questions reading only summaries [START_REF] Seki | Automatic summarization focusing on document genre and text structure[END_REF]. An assessor may be asked to evaluate the importance of each sentence/passage and the obtained annotation of importance allows summaries with a predefined compression rate to be generated [START_REF] Saggion | Developing Infrastructure for the Evaluation of Single and Multi-document Summarization Systems in a Cross-lingual Environment[END_REF] or serve as expert extracts which may be used as a gold standard.

In [START_REF] Owczarzak | An assessment of the accuracy of automatic evaluation in summarization[END_REF], the Responsiveness metric is proposed. It was designed for query-focused summarization and shows how well a summary satisfies the user's information need expressed by the query. The Responsiveness metric was applied at DUC. The expert nature of this metric makes it impossible for further re-use.

At INEX Snippet Retrieval Track 2012-2013 [START_REF] Bellot | Overview of INEX[END_REF], the relevance of the documents was judged independently from the relevance of the snippets. It was done in order to determine the effectiveness of a snippet to provide sufficient information about the corresponding document. Thus, assessors should evaluate results in two ways: (a) relevance evaluation of documents; (b) relevance evaluation of snippets.

350

Assessors had to go through the snippets, and decide whether the underlying document seemed relevant to the topic by reading only the snippet. They put 1 if it seemed to be relevant and 0 otherwise. After that, assessors had to read all of the documents and judge their relevance. Then, snippet-based relevance judgments were compared with the document-based relevance judgments (ground truth), i.e. a good snippet should be judged the same as the corresponding document. Then, these judgments were integrated by the 355 following measures [START_REF] Bellot | Overview of INEX[END_REF]:

• Mean prediction accuracy (MPA) -the average percentage of results the assessor correctly assessed:

MPA = T P + T N T P + FN + T N + FP (1)
where T P refers to true positive, T N to true negative, FN to false negative, and FP to false positive.

• Mean normalized prediction accuracy (MNPA) is the average of the relevant results correctly assessed and the irrelevant results correctly assessed:

MNPA = 0.5 × T P T P + FN + 0.5 × T N T N + FP (2)
• Recall is the average percentage of relevant documents correctly assessed:

R = T P T P + FN (3)
• Negative recall (NR) is the average percentage of irrelevant documents correctly assessed:

NR = T N T N + FP (4)
• Positive agreement (PA) is the conditional probability of agreement between snippet assessor and document assessor, given that one of the two is judged relevant:

PA = 2 × T P 2 * T P + FP + FN (5)
• Negative agreement (NA) is the conditional probability of agreement between snippet assessor and document assessor, given that one of the two judged irrelevant:

NA = 2 × T N 2 * T N + FP + FN (6)
• Geometric mean (GM) of recall and negative recall:

GM = √ R × NR (7)
Precision = × relevant relevant + redundant + notrelevant (8)
and

Utility = relevant -redundant -notrelevant (9)
However, the interleaving of outputs from several systems was responsible for introducing the redundancy and may disadvantaged some systems.

Overlap-based Measures

Reference summaries allow the metrics commonly used in information retrieval to be calculated: recall (R) and precision (P) over the number of terms/sentences appearing in reference and candidate summaries [START_REF] Gholamrezazadeh | A comprehensive survey on text summarization systems[END_REF]:

R = |S ∩C| |S| (10)
P = |S ∩C| |C| ( 11
)
where S is a set of terms/sentences in the set of reference summaries, C is a set of terms/sentences in the candidate summary.

Recall and precision may be integrated into the Fmeasure [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF]:

F = (β 2 + 1) × R × P β 2 × P + R (12)
β 2 = 1 -α α , α ∈ [0, 1] (13)
The F-measure is widely used in ad-hoc information retrieval but it is less useful in summary evaluation since a search engine result is potentially infinite while a summary is limited. Besides, the sentence-based measures of information retrieval types cannot be applied to abstract assessment since abstracting implies the reformulation of initial sentences.

Similarity between reference and candidate summaries may be estimated as cosine, dice or Jaccard coefficient, as well as the number of shared n-grams or longest common subsequence etc.

The cosine similarity between a summary and a reference hereinafter referred to as COS is estimated as follows:

cos(C, S) = S Ċ ∑ n i=1 S 2 i ∑ n i=1 C 2 i (14) cos(S,C) ∈ [0, 1] ( 15
)
where C is the summary under evaluation and S is the reference text, S i and C i are TF-IDF of the i-th term in the vector representation S and C respectively. In our experiments, we used IDFs learned on the entire Wikipedia collection (see Section 4).

One of the most efficient metrics of summary evaluation is ROUGE (Recall-Oriented Understudy for Gisting Evaluation) used at the Document Understanding Conference (DUC) [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF]. ROUGE is also based on comparison with a set of reference summaries. Proposed by Chin-Yew Lin in 2004, ROUGE aims to evaluate summaries but it is also used to assess the quality of machine translation. Several variants of the ROUGE metrics exist, e.g. ROUGE N (n-grams recall):

ROUGE N (S,C) = |S ∩C| |S| ( 16
)
where S is a multi-set of n-grams in the set of reference summaries, C is a a multi-set of n-grams in the candidate summary, N refers to the n-gram length. Thus, the main difference of ROUGE N from the recall is the use of multi-set instead of a simple set. Another dissimilarity is n-gram application. ROUGE N implies that a summary gets a higher score as it contains more n-grams co-occurring with reference summaries.

375

This metric is called further as ROUGE.

ROUGE N_multi computes pairwise n-gram recall with each reference summary S i and takes the maximal value:

ROUGE N_multi = arg max i ROUGE N (S i ,C) (17)
Another method ROUGE L is based on the search for the longest common substring (LCS) shared by two sentences:

ROUGE L = F lcs = (β 2 + 1) × R lcs × P lcs β 2 × P lcs + R lcs (18)
R lcs = |LCS(X,Y )| |X| (19)
P lcs = |LCS(X,Y )| |Y | (20)
where LCS(X,Y ) is the longest common substring of the sentences X and Y . If there is no shared subsequence ROUGE L = 0. ROUGE L includes the longest common n-gram and there is no need to compute its length in advance. ROUGE L allows to compare the sentence structure but only with respect to the longest shared part. For the whole texts ROUGE L can be estimated by the formulas:

ROUGE L = F lcs = (β 2 + 1) × R lcs × P lcs β 2 × P lcs + R lcs (21)
R lcs = ∑ Si∈S |LCS ∪ (S i ,C)| |S| (22)
P lcs = ∑ Si∈S |LCS ∪ (S i ,C)| |C| ( 23
)
where |LCS ∪ (S i ,C)| is the LCS score of the union of the longest common subsequences between each reference sentence s i and candidate summary C. For example, let s i = w 1 w 2 w 3 w 4 w 5 , and C = c 1 c 2 , c 1 = w 1 w 2 w 6 w 7 w 8 , c 2 = w 1 w 3 w 8 w 9 w 5 , then LCS for s i , c 1 is w 1 w 2 , and for s i , c 2 LCS is w 1 w 3 w 5 . The union is

w 1 w 2 w 3 w 5 . |LCS ∪ (s i ,C)| = 4/5.

380

Normalized pairwise comparison LCS MEAD (S,C) [START_REF] Radev | Evaluation of text summarization in a cross-lingual information retrieval framework[END_REF] is similar to ROUGE L when β = 1 [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF], but LCS MEAD takes the maximal value of LCS, while ROUGE L deals with the union of LCS [START_REF] Hovy | Summarization evaluation using transformed basic elements[END_REF].

LCS MEAD = (β 2 + 1) × R MEAD × P MEAD β 2 × P MEAD + R MEAD (24) R MEAD = ∑ s i ∈S max s j ∈C LCS(s i , s j ) m (25)
P MEAD = ∑ s j ∈C max s i ∈S LCS(s i , s j ) n (26)
One of the serious shortcomings of LCS is the fact that it does not consider the distance between words. Let consider an example. Let S = ABCDEFG be a reference, C1 = AZBYCZD a candidate, C2 = ABCDXY Z another candidate. The longest common substring is ABC and therefore, ROUGE L attributes the same score to both candidates. However, the second candidate is better. Weighted LCS, called WLCS, takes into account the length of consecutive matches by using dynamic programming approach memorizing the length of the current consecutive matches ending at words s i and c j . For more details about the algorithm see [START_REF] Lin | Automatic evaluation of summaries using n-gram co-occurrence statistics[END_REF].

The weighting function f is a function of consecutive matches at the table position (i, j) and should satisfy the following constraint: (∀x, y ∈ N) : f (x + y) > f (x) + f (y). That is to say consecutive matches should have higher score than non-consecutive ones. f may be the linear

f (k) = αk -β , α > 0, β > 0, polynomial or quadratic function f (k) = k 2 . Let f -1 be the inverse function of f , for example, f = k 2 , f -1 = k 1/2 .
In this case F-measure is estimated as follows:

F wlcs = (β 2 + 1) × R wlcs × P wlcs β 2 × P wlcs + R wlcs (27)
R wlcs = f -1 W LCS(X,Y ) f (m) (28)
P wlcs = f -1 W LCS(X,Y ) f (n) (29)
LCS based algorithms are a special case of edit distance [START_REF] Bangalore | Evaluation metrics for generation[END_REF].

The metric ROUGE-S is based on the counting of shared bi-grams the elements of which may be separated by arbitrary number of other words. The distance may be limited by d skip . Sometimes uni-gram smoothing is applied (ROUGE-SU).

To compute ROUGE-S the following formulas are applied:

F skip2 = (β 2 + 1) × R skip2 × P skip2 β 2 × P skip2 + R skip2 (30) R skip2 = SKIP2(X,Y ) C(m, 2) (31)
P skip2 = SKIP2(X,Y ) C(n, 2) (32)
where C(n, k) is the binomial coefficient n k , and SKIP2(X,Y ) is the number of common bi-grams with arbitrary distance in the texts X and Y respectively. The distance may be limited by d skip . Sometimes uni-gram smoothing is applied (ROUGE-SU).

The experiments conducted by the organizers of Automatically Evaluating Summaries of Peers (AE-SOP) task within Text Analysis Conference (TAC) showed that the metrics proposed by TAC participants, such as ROUGE-BE, DemokritosGR2, catholicasc1, and CLASSY1 significantly outperform ROUGE-2 which is the best from all ROUGE variants [START_REF] Owczarzak | An assessment of the accuracy of automatic evaluation in summarization[END_REF].

405

The metric BLEU (BiLingual Evaluation Understudy) commonly used for machine translation evaluation is also suitable for assessment of any generated text [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF]. As ROUGE, BLEU also estimates as the number of shared n-grams. In contrast to ROUGE which is recall oriented, BLEU uses a modified form of precision to compare a candidate against multiple references [START_REF] Papineni | BLEU: a method for automatic evaluation of machine translation[END_REF]:

p BLEU = |S ∩C| |C| ( 33
)
where S is a multi-set of n-grams in the reference summary, C is a a multi-set of n-grams in the candidate summary. Note that intersection of multi-sets captures the notion of clipped counts, i.e. upper bounding of the total count of each candidate word by its maximum reference count, introduced in [START_REF] Papineni | BLEU: a method for automatic evaluation of machine translation[END_REF].

Thus, BLEU measures how much n-grams in the candidate summary appear in the reference summaries, while ROUGE estimates how much n-grams in the reference summaries appear in the candidate one. Precision-based metrics have tendency to prefer short texts making them less appropriate to summary evaluation. However, a good summary should provide as much important information as possible. Besides, one can imagine an extreme case when a summarization system provides only one word which may be relevant and a precision-based metric would attribute the maximal score to it. Therefore, brevity penalty was introduced to penalizes texts shorter than the length of a reference [START_REF] Lin | Automatic evaluation of summaries using n-gram co-occurrence statistics[END_REF]:

BP =      1 if |C| > |S| avg exp 1- |S|avg |C| if |C| ≤ |S| avg (34)
Thus, the final score of BLEU is calculated as [START_REF] Papineni | BLEU: a method for automatic evaluation of machine translation[END_REF]:

BLEU = BP exp ∑ N n=1 w n log p BLEU n ( 35
)
or

log BLEU = min 1 - |S| avg |C| , 0 + N ∑ n=1 w n log p BLEU n ( 36
)
where N is the maximal n-gram order, w n is n-gram weight.

Further improvement of BLEU was done by US National Institute of Standards and Technology within 410 DUC conference. They introduced the metric NIST, which does not only calculates n-gram precision, but also weights the informativeness of a particular n-gram, for example by IDF [START_REF] Lin | Automatic evaluation of summaries using n-gram co-occurrence statistics[END_REF].

In contrast to BLEU, the METEOR (Metric for Evaluation of Translation with Explicit ORdering) metric is able to treat spelling variants, WordNet synsets and paraphrase tables and distinguishes function and content words [START_REF] Denkowski | Meteor 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems[END_REF]. METEOR is calculated based on the F-measure between recall and precision and a penalty p for fragmentation:

p = 0.5 |chunks| |S ∩C| 3 ( 37
)
MET EOR = F mean (1 -p) ( 38
)
where |chunks| is the total number of chunks, i.e. a set of uni-grams that are adjacent in the candidate and in the reference. Basic Elements (BE) which can be considered as paraphrases where proposed by Tratz and Hovy [START_REF] Hovy | Summarization evaluation using transformed basic elements[END_REF]. A BE is a syntactic unit up to 3 words with associated tags such as named entities and parts of speech. BE can deal lemmas, synonyms, hyponyms and hyperonyms, identical prepositional phrases, spelling variants, nominalization and denominalization (derivation in WordNet), transformations like prenominal noun -prepositional phrase, noun swapping for IS-A type rules, pronoun transformations, pertainym adjective transformation.

One of the derivatives of the edit distance is Word Error Rate (WER) [START_REF] Klakow | Testing the correlation of word error rate and perplexity[END_REF]:

W ER = Sub + Del + Ins |S| ( 39
)
where Sub is the number of substitutions, Del is the number of deletions, Ins is the number of insertions, and |S| is the number of words in the reference. In fact, WER is the length normalized edit distance.

The organizers of INEX Tweet Contextualization Track 2011-2014 evaluated extractive summaries by comparing them with the pool of passages judged as relevant. As the distance they used the Kullback-Leibler divergence or simple log difference [START_REF] Bellot | Overview of inex[END_REF]. They state that the Kullback-Leibler divergence is very sensitive to smoothing in case of small number of relevant passages in contrast to the absolute log-diff between frequencies [START_REF] Bellot | Overview of inex[END_REF].

Until 2011, the informativeness was evaluated by the Kullback-Leibler (KL) divergence [START_REF] Sanjuan | Overview of the inex 2010 question answering track (qa@inex)[END_REF]:

KL = ∑ w∈S∪C p C (w) log p C (w) p S (w) (40)
where p C (w) and p S (w) refer to the probability to see w in C and S respectively. Dirichlet smoothing was applied.

In 2011 the informativeness was estimated as the log difference:

Div = ∑ w∈S log |S(w)| |S| + 1 -log |C(w)| 500 + 1 ( 41
)
where S is the set of terms in the pool of relevant passages, |S(w)| is the frequency of a term w in the pool, |S| is the total number of terms in the pool, |C(w)| is the frequency of a term w in a summary, |C| is the total number of terms in a summary. A term may refer to a uni-gram, a bi-gram (two consecutive lemmas in the same sentence) or a bi-gram allowing a gap up to two lemmas between its component (with 2-gap). The lower values of Div corresponds to higher matching of tokens in a pool and a summary. The evaluation was carried out by FRESA [START_REF] Saggion | Multilingual summarization evaluation without human models[END_REF] package which includes a special lemmatizer.

Since 2012 the informativeness was evaluated by the following formula:

Dis = ∑ w∈S |S(w)| |S| × 1 - min (log P, log Q) max (log P, log Q) ( 42
)
where P and Q are computed as:

P = |S(w)| |S| + 1 (43) Q = |C(w)| |C| + 1 ( 44
)
Since |C(w)| |C| ∈ (0, 1] and |S(w)| |S| ∈ (0, 1], P > 1 and Q > 1. Therefore, max (log P, log Q) > 1. The logarithm allows dealing with highly frequent words. The evaluation toolkit was based on Porter stemmer. The lower values of Dis correspond to the higher informativeness. The complement of this dissimilarity measure 1 -Dis has similar properties than usual information retrieval Interpolate Precision measures. Thus, we used this complement as a competing metric hereafter referred to as INEX.

[44] introduced a trivergent model that outperformed the divergence score. In [START_REF] Campr | Comparing Semantic Models for Evaluating Automatic Document Summarization[END_REF], the authors proposed to use the similarity within semantic representation such as LSA, LDA, Word2Vec and Doc2Vec. However, ROUGE-1 outperformed all these metrics. In [START_REF] Ng | Better summarization evaluation with word embeddings for rouge[END_REF], ROUGE metric was modified by word embeddings but this variant showed lower results than the standard one.

Other Informativeness Metrics

A Pyramid score is based on the number of repetitions of Summary Content Units (SCU) in the goldstandards [START_REF] Nenkova | The pyramid method: Incorporating human content selection variation in summarization evaluation[END_REF]. SCUs are information units of variable length inside a sentence labeled by experts in their own words. SCUs are weighted according to the number of summaries they occur in and, thus, are organized into a pyramid. The final Pyramid score is estimated as follows:

Pyramid = ∑ n i=1 i × D i ∑ n i= j+1 i × |T i | + j × X -∑ n i= j+1 |T i | (45) j = max i n ∑ t=i |T t | ≥ X ( 46
)
where n is the number of tiers, |T i | denotes the number of SCUs at the level T i , D i is the number of SCUs in the summary appearing in T i , X is summary size in SCUs. Pyramid score was used at the DUC conference. The main drawback of the Pyramid score is that it is based on manual assessment not only of the reference summaries, but also of the candidate ones, and therefore it can not be re-used for evaluation of new candidates.

GRAD (GRAph Distance) metric aims to estimate how well summary terms are connected to full text terms and is based on the assumption that a good summary is made of the terms that refer to the central vertices in the semantic graph, i.e. the terms that are connected to the maximal number of other terms in a full text [START_REF] Ermakova | GRAD: A Metric for Evaluating Summaries[END_REF]. According to this metric, the score of a summary is estimated as a normalized inverted sum of distances from every term in the text to its closest term appearing in the summary S:

score(S) = 1 |S| ∑ v i min v j ∈V ∩S d(v j , v i ) ( 47
)
where d(v j , v i ) is the shortest path between v i and v j . To calculate minimal distances from every term in the text to its closest term from the summary, a modified Dijkstra's algorithm is used [START_REF] Dijkstra | A note on two problems in connexion with graphs[END_REF].

The metric GEM (GEnerosity Measure) attributes an absolute score [0,1] to a summary [START_REF] Ermakova | Is the Abstract a Mere Teaser? Evaluating Generosity of Article Abstracts in the Environmental Sciences[END_REF]. GEM relies on the importance of the different sections of a scientific paper. The GEM score is calculated as the sum of the weights of the section classes retrieved both in a summary and a full text normalized over the total sum of weights of section classes in a full text. For each sentence, the section class in a summary is assigned according to the class of the sentence from the full text with the maximal cosine similarity. As GRAD, GEM fails to distinguish native abstracts from other human summaries.

Readability Evaluation

Readability, coherence, conciseness, content, grammar, recall, pithiness etc. are usually assessed manually [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF][START_REF] Saggion | Developing Infrastructure for the Evaluation of Single and Multi-document Summarization Systems in a Cross-lingual Environment[END_REF] since often these parameters are not numerically expressed [START_REF] Gholamrezazadeh | A comprehensive survey on text summarization systems[END_REF].

The first metrics of readability were proposed outside the scope of automatic summarization. For example, Gunning fog index estimates the years of formal education a person needs to understand the text on the first reading and is computed based on the average sentence length and percentage of complex words [START_REF] Gunning | The technique of clear writing[END_REF]. The Flesch-Kincaid readability tests is designed to indicate how difficult a passage in English is to understand based on word length and sentence length [START_REF] Flesch | A new readability yardstick[END_REF]. Despite the easiness of these two metrics, they are not reliable [START_REF] Si | A statistical model for scientific readability[END_REF].

Language models were also used for estimate the readability [START_REF] Si | A statistical model for scientific readability[END_REF][START_REF] Collins-Thompson | A language modeling approach to predicting reading difficulty[END_REF][START_REF] Heilman | An analysis of statistical models and features for reading difficulty prediction[END_REF]. Machine learning techniques were proposed in [START_REF] Petersen | A machine learning approach to reading level assessment[END_REF][START_REF] Feng | A comparison of features for automatic readability assessment[END_REF]. Feng et al. proposed to use discourse, language modelling, parsing, POS, and quantitative features [START_REF] Feng | A comparison of features for automatic readability assessment[END_REF]. However, all these metrics are not applicable to the readability evaluation of extractive summaries since summary phrases are taken from source documents as they are.

Traditional methods of readability evaluation are based on familiarity of terms [START_REF] Stenner | The Lexile Framework[END_REF][START_REF] Chall | Readability revisited: The new Dale-Chall readability[END_REF][START_REF] Fry | A readability formula for short passages[END_REF] or their length [START_REF] Tavernier | Combining relevance and readability for INEX 2011 question-answering track[END_REF] and syntax complexity [START_REF] Collins-Thompson | A Language Modeling Approach to Predicting Reading Difficulty[END_REF]. Another set of methods is based on syntax analysis [START_REF] Mutton | Gleu: Automatic evaluation of sentence-level fluency[END_REF][START_REF] Wan | Searching for grammaticality: Propagating dependencies in the viterbi algorithm[END_REF][START_REF] Zwarts | Choosing the right translation: A syntactically informed classification approach[END_REF].

Syntactical methods may be combined with statistics (e.g. sentence length, the depth of a parse tree, omission of personal verb, rate of prepositional phrases, noun and verb groups etc.) [START_REF] Chae | Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text[END_REF]. The latter methods are suitable only for the readability evaluation of a particular sentence and therefore they cannot be used for extracts assessment. Researches also proposed to use language models for prediction of readability difficulties, i.e. predicting word difficulty based on the language model [START_REF] Collins-Thompson | A Language Modeling Approach to Predicting Reading Difficulty[END_REF][START_REF] Si | A statistical model for scientific readability[END_REF]. Usually assessors assign score to the readability of text in some range [START_REF] Barzilay | Inferring strategies for sentence ordering in multidocument news summarization[END_REF].

Syntactical errors, unresolved anaphora, redundant information and coherence influence readability and therefore the score may depend on the number of these mistakes [START_REF] Bellot | Overview of inex[END_REF].

At INEX Tweet Contextualization

Track 2011-2014, assessors were not asked to evaluate the relevance of the summaries. There were two metrics:

• Relaxed metric: a passage was considered valid if it was not marked as trash,

• Strict metric: a passage was considered valid if it did not have any problems mentioned above.

In 2011, the readability of summaries was estimated as the number of words (up to 500) in valid passages [START_REF] Sanjuan | Overview of the inex 2011 question answering track (qainex)[END_REF]. Since 2012, the score of a summary was the average normalized number of words in valid passages [START_REF] Bellot | Overview of INEX[END_REF]. Sentence ordering was not judged by conference organizers, however several authors affirm that it is quite important for text understanding [START_REF] Barzilay | Inferring strategies for sentence ordering in multidocument news summarization[END_REF].

Different non-parametric rank correlation coefficients (e.g. Kendall, Spearman or Pearson coefficients) may be used to find the dependence [START_REF] Lebanon | Cranking: Combining rankings using conditional probability models on permutations[END_REF]. However, as shown in [START_REF] Lapata | Probabilistic text structuring: Experiments with sentence ordering[END_REF], Kendall coefficient is the most suitable for sentence ordering assessment.

Current methods to evaluate readability are based on the familiarity of terms and syntax complexity [START_REF] Collins-Thompson | A language modeling approach to predicting reading difficulty[END_REF].

Word complexity may be estimated by humans [START_REF] Stenner | The lexile framework[END_REF][START_REF] Chall | Readability revisited: The new Dale-Chall readability[END_REF][START_REF] Fry | A readability formula for short passages[END_REF] or according to its length [START_REF] Tavernier | Combining relevance and readability for INEX 2011 question-answering track[END_REF]. Researches also propose to use language models [START_REF] Collins-Thompson | A language modeling approach to predicting reading difficulty[END_REF][START_REF] Si | A statistical model for scientific readability[END_REF]. Usually assessors assign a score to the readability of a text in some range [START_REF] Barzilay | Inferring strategies for sentence ordering in multidocument news summarization[END_REF]. BLEU with high-order n-grams can be used for readability evaluation. Edit distance may be applied as well for readability evaluation to assess word order or sentence order. These metrics are semi-automatic because they require a gold standard. Another set of methods is based on syntax analysis which may be combined with statistics (e.g. sentence length, depth of a parse tree, omission of personal verbs, rate of prepositional phrases, noun and verb groups) [START_REF] Mutton | Gleu: Automatic evaluation of sentence-level fluency[END_REF][START_REF] Wan | Searching for grammaticality: Propagating dependencies in the viterbi algorithm[END_REF][START_REF] Zwarts | Choosing the right translation: A syntactically informed classification approach[END_REF][START_REF] Chae | Predicting the fluency of text with shallow structural features: case studies of machine translation and human-written text[END_REF], but they remain suitable only for the readability evaluation of a particular sentence and, therefore, cannot be used for assessing extracts.

As shown, evaluating informativeness of automatically produced contents without involving manual process interest numerous research teams. However, the evaluation of readability is still mainly manual which leads to the need of designing self-sufficient metric that could measure sentence ordering hence readability. Ermakova [START_REF] Ermakova | Automatic Sentence Ordering Assessment Based on Similarity[END_REF] proposes an automatic approach for sentence order assessment where the similarity between adjacent sentences is used as a measure of text coherence. However, it assigns equal scores to initial and inverse sentence order due to the symmetric similarity measure. In contrast, the topic-comment based method proposed in [START_REF] Ermakova | A Metric for Sentence Ordering Assessment Based on Topic-Comment Structure[END_REF] can deal with this problem.

4 Limitations of the Existing Assessment Metrics

Framework to Evaluate Assessment Metrics

An assessment metric allows comparing various methods in order to choose the most appropriate one.

However, this assessment metric should also be carefully selected. In the previous section we described the main metrics used for summary evaluation. In this section we present the experimental results that revealed some limitations of existing metrics.

Traditionally, the quality of assessment metrics is evaluated as a correlation between expert results and candidate metrics (e.g. Kendall, Spearman, or Pearson coefficients) [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF]. A good metric should give low score to summaries which have low score according to human judgment and high score otherwise.

In the case where a reference summary is compared to other references, re-sampling methods are often used, e.g. jackknifing (use of the subsets of the reference summaries, each of which is missing one reference) or bootstrapping (random replacement of points in the data set). This is important, since comparing a reference to itself leads to the maximal score, but ignoring it results into different number of references. In case of re-sampling, the final score is the mean of all computed values [START_REF] Hovy | Summarization evaluation using transformed basic elements[END_REF].

In [START_REF] Louis | Automatically assessing machine summary content without a gold standard[END_REF], Louis and Nenkova suggested an automatic approach for summary evaluation without a gold standard. Instead of a set of reference summaries, a full text is used. The authors suggested to estimate summary scores by Kullback-Leibler divergence, Jensen Shannon divergence, and cosine similarity measure. Although these metrics have some correlation with traditional gold-standard based ROUGE scores, ROUGE-1 demonstrated better results.

In this section, we will compare the main overlap-based metrics ROUGE and INEX as defined in 42.

We took into account only automatic metrics that is why we did not consider the Pyramid score nor Responsiveness. ROUGE and INEX metrics seems to be the most widely used automatic metrics. They were applied during several years in different evaluation campaigns. We also compared the results with a simple cosine similarity. We will show that they have similar drawbacks since they are based on vocabulary overlap. Moreover, we compared the results with those of less classical metrics GRAD and GEM since they are not overlap based.

We show the limitations of the use of the full text as a gold standard for vocabulary overlap based metrics. In contrast to [START_REF] Louis | Automatically assessing machine summary content without a gold standard[END_REF], besides cosine similarity measure, we also compared INEX and ROUGE metrics that use the full text as a gold standard.

The evaluation framework we defined is two folds: on the one hand, we assume a human written summary should be considered as better than a automatically generated one, on the other hand we consider that the measure should assign higher score to the summary coming from the text under consideration than to the summaries of other texts.

The intuition underlying the first part of the evaluation is that a good assessment metric should assign a high score to a good summary and a low score to a bad one. In contrast to [START_REF] Lin | ROUGE: a package for automatic evaluation of summaries, Text Summarization Branches Out[END_REF], rather than calculating the correlation between the scores assigned to summaries by assessors and metrics, we propose to compare the percentage of times when a good summary of a text is scored lower than a worse one of the same text (accuracy).

This approach requires at least two summaries for every article. The data sets we used contain one human provided summary and another two summaries we generated automatically using poor methods described below. We assume that human written summaries are better than the generated ones and a good metric should reflect that. It is important to notice that we deliberately chose very simple methods for automatic summary creation to ensure that human written abstracts are much better than the generated ones. This evaluation framework does not require explicit human assigned scores and may be performed on very large collections. However, rank correlation coefficients are not applicable for these data since there is no ground truth beyond a pair of summaries and, thus, it is impossible to compare summaries of different full texts.

The second part of the evaluation aims at evaluating the capacity of an automatic measure to identify the appropriate abstract among human provided summaries of different texts (the text under consideration and other texts). We performed pair-wise comparison between each of 31 documents and abstracts coming from different texts using the automatic assessment metrics. A good metric should assign higher score to the abstract initially associated to the document by its authors not the abstract from another document.

Data Sets

We conducted experiments on two data sets with existing abstracts: scientific and Wikipedia articles.

Scientific Articles.

For our experiments we used the data from the ISTEX (Excellence Initiative of Scientific and Technical Information) platform 6 The second data set we used was a cleaned recent English Wikipedia XML dump created by the INEX organizers for Tweet Contextualization Track [START_REF] Bellot | Overview of inex[END_REF]. All notes, history and bibliographic references were removed. Thus, a page was composed of a title (title), an abstract (a) and sections (s). A section had a header (h). Abstract and sections contained paragraphs (p) and entities (t) referring to other pages. We selected the 100,000 longest articles. However, only 43,611 articles had both abstracts and not-empty sections and were kept for experiments. This data set is referred as Wikipedia.

Methods for Summary Generation

Generated summaries have low readability because of abrupt topic changes and unresolved anaphoras.

However from informativeness standpoint we assume they will be competitive with human provided summaries. Since our goal is mainly to evaluate evaluation metrics, we deliberately applied very simple methods to generate summaries since these simple methods should produce in average summaries of lower quality than summaries created by using more sophisticated methods. We believe that simple methods for summary generation are sufficient since (1) even in this case vocabulary-based metrics fail and ( 2) there is no need to check manually if they are less good than the human written ones. The two methods are presented below.

Random Extractive Summary.

The first method we used is extractive summarization that randomly selects the sentences from the full texts while the total number of words does not exceed a predefined threshold. To avoid bias because of the different summary sizes, the size of the generated summaries was set to the average size of human provided ones (200). For evaluation purpose, we generated a summary for each document only once. The random bias, however, should be vanished due to the average score over a high number of documents. We call this method RandSum.

Cosine Similarity Based Summary.

The second approach we applied for summary generation is based on the cosine similarity measure between bag-of-words representations of a candidate sentence C and a full texts A:

cos(C, A) = ∑ n i=1 C i A i ∑ n i=1 C 2 i ∑ n i=1 A 2 i ( 48
)
A i and C i refer to term TF-IDF from the corresponding texts. The sentences with the highest scores are selected until the total number of words in a summary is less than a predefined threshold. This method is later denoted by CosSum.

Competing Metrics

In our experiments we compared three vocabulary-based metrics (INEX, ROUGE-N, and COS) with a graph-based measure GRAD.

Since vocabulary overlap based metrics have very low performance when a full text is used as a gold standard, we also compared results with a random score assigned to a summary in the range [0, 1] (hereafter called RAND).

Additional details regarding metrics comparison including Jensen-Shannon and Kullback-Leibler divergence [START_REF] Louis | Automatically assessing machine summary content without a gold standard[END_REF] can be found in [START_REF] Deveaud | Informativeness for adhoc ir evaluation: A measure that prevents assessing individual documents[END_REF].

Revealed Limitations of the Use of the Full Text as a Gold Standard

Table 2 provides the evaluation results of competing metrics as the percentage of times when a human provided abstract is scored higher/lower/equally than/to a generated summary. We assume that human written summaries are better than the generated ones. Thus, we use the accuracy as a performance metric;

here, the accuracy means the percentage of times when human provided summary has higher score than the generated ones. The higher accuracy is, the better is the metric. We performed Student's t-test to check whether the results are significantly different from the random ones at the level p < 0.05. + / -indicates the accuracy of the metrics that is significantly higher/lower than the accuracy of RAND.

ROUGE and COS accuracy is less than 25% for all data sets. INEX has a high accuracy on ISTEX data but very poor accuracy on other collections. In all cases, the difference with the random baseline is significant. The table testifies that GRAD significantly outperforms the baseline assigning score randomly as well as the state-of-the-art vocabulary overlap based metrics (INEX, ROUGE).

Since the used methods for summary generation are the extractive ones, the metrics based on the overlap between terms should have a low performance. ROUGE and COS are in average twice as worse as the random baseline on both test collections. Although, the INEX measure showed quite competitive results on the ISTEX data set, on the Wikipedia and environmental science articles its results are much lower than the results obtained by RAND baseline. Thus, we can conclude that full texts can not serve as a gold standard especially in case of extractive summaries since in this case the measures are mainly reduced to the ratio of the size of a summary over the one of a full text.

One of the possible explanation why GRAD significantly outperforms the vocabulary overlap based metrics is that other measures are dealing with pure term frequencies regardless of their context. Under the circumstances, a term may be frequent in a particular context or a specific document part but not very informative for the entire text. In contrast, the GRAD metric takes into account how strongly a term is connected to all other terms in the text.

The major conclusion we can draw is that vocabulary overlap based metrics should be not used with full texts as references.

Ability to Find the Native Summary among Summaries of Different Texts

The second experiment aimed at evaluating the capacity of metrics to identify the appropriate abstract among human written summaries and follows the second part of the framework we suggested. We performed pair-wise comparison between each of 31 documents and abstracts coming from different texts (31 × 31 = 961 comparisons, 496 different text pairs). These documents on environmental sciences were selected from the ISTEX database. We used the same documents as in [START_REF] Ermakova | Is the Abstract a Mere Teaser? Evaluating Generosity of Article Abstracts in the Environmental Sciences[END_REF].

In Figures 1 to 4 we present the results of the subtraction of the score assigned to original abstracts and and the score of abstracts coming from the other texts when considering these 961 cross-text comparisons and various metrics. In these figures, if an abstract coming from another text is preferred by a metric over the native abstract, the point is below the zero line.

It is evident from the figure 1 that in most cases ROUGE is able to find the appropriate abstract from a set of abstracts summarizing different texts. This fact does not conflict with the conclusion from the previous section. It is important to notice that ROUGE is based on vocabulary overlap. A human-authored abstract can be paraphrased with regard to the source text, i.e. it may have different words leading lower ROUGE score. An extractive summary is coming from the source text, i.e. summary words are a subset of words from the full text leading higher ROUGE score. In contrast, when comparing human-written abstracts for different texts, the vocabulary overlap of the native abstract is higher than the one of the foreign summary because it presents another text.

In [START_REF] Ermakova | GRAD: A Metric for Evaluating Summaries[END_REF], it was reported that despite GRAD significantly outperforming vocabulary overlap based metrics, it fails to distinguish native abstracts from other human written summaries. Only in 15% of cases the score assigned to the native abstract was higher than the scores assigned to foreign abstracts. However, our new experiments demonstrated that GRAD is able to distinguish native abstracts from abstracts coming from other texts (see Figure 3). The difference is that here we do not normalize over the summary length In contrast, GEM produced a lot of errors (see Figure 4). This inability to find the right abstract is due to an implementation shortcoming of GEM since for each sentence in a summary, the section class is assigned according to the class of the sentence from the full text with the maximal cosine similarity. The section class would be assigned in any case, even if no section match at all the sentence under consideration (the 660 first section class would be assigned). GEM was not designed to search for right abstract. Thus, in contrast to traditional metrics, GEM compares overlap scores of sentences within a particular abstract rather than different summaries. Assuming that the summary is the native one, GEM aims to detect the presence of sections from a predefined list based on a set of rules (see [START_REF] Ermakova | Is the Abstract a Mere Teaser? Evaluating Generosity of Article Abstracts in the Environmental Sciences[END_REF] for more details). Vocabulary overlap-based metrics did not faced this issue: the trend is completely opposite (see Figures 1 and2). Thus, we can conclude that vocabulary overlap-based metrics and GRAD are able to find the native abstracts, while GEM can not capture this difference.

Discussion and Conclusion 670

In this paper, we discuss a wide range of evaluation frameworks for automatic summarization, including data sets and metrics used, as well as main competitions in the field, for the last 18 years. Manual assessment is time and money consuming and, therefore, not applicable to real time scenarios or large collections. The expert nature of metrics, such as responsiveness, retention rate, metrics used in INEX Snippet

Retrieval track, makes it impossible for re-use, since manual assessment requires expert intervention for 675 each summary under evaluation. Moreover, on average, assessment agreement is about 70% due to the fact that judges may have different opinions about summary quality and evaluation metrics [START_REF] Gholamrezazadeh | A comprehensive survey on text summarization systems[END_REF]. Inter-rater agreement is traditionally measured by Cohen's kappa [START_REF] Carletta | Assessing agreement on classification tasks: The kappa statistic[END_REF] or Krippendorff's alpha in the case of an arbitrary number of coders [START_REF] Krippendorff | Content Analysis: An Introduction to Its Methodology[END_REF]. This average 70% agreement between judges is also a big issue for expert metrics since it means that for an objective evaluation at least, all summaries of the same text should be evaluated by the same annotator.

Otherwise, there is a possibility that different methods are treated in different ways by different annotators.

In other words, one judge can be very strict, another very generous.

In contrast to manual assessment, gold-standard based evaluation is less subjective and theoretically re-usable for new methods.

Traditional evaluation frameworks (data sets and metrics) have several drawbacks. Sometimes the reuse of the gold standard by non-participants is problematic since often the ground truth is constructed only from the results initially submitted by the participants, i.e. some relevant results may not occur in a gold standard.

The use of the full text rather than a set of reference summaries for summary evaluation provides poor results [START_REF] Louis | Automatically assessing machine summary content without a gold standard[END_REF] since traditional overlap-based metrics are mainly reduced to the ratio of the size of a summary to the size of a full text since they are designed to be compared with summaries created by humans.

To provide evidence, we proposed a completely automatic framework for the evaluation of metrics to assess automatically produced summaries which does not require any human annotation. We conducted experiments on Wikipedia data set and a collection of scientific articles from the ISTEX database. The obtained results show that metrics based on vocabulary overlap are not suitable for measuring the quality of a summary with regard to a full text. However, the GRAD measure significantly outperformed overlapbased baselines on both test collections in distinguishing human written abstracts from generated summaries of poor quality.

The second problem with the full text as a gold standard is that every completely automatic metric without hidden data can be optimized, i.e. transformed into an automatic method that maximizes this metric. This optimization does not necessarily correlate with the quality of the produced summary.

  was a continuing evaluation campaign in the area of text summarization. DUC was an emerging response to the requirement of an evaluation framework for summarization systems within campaigns such as TREC (Text Retrieval Conferences) organized by NIST (National Institute of Standards and Technology), TIDES (Translingual Information Detection Extraction and Summarization) organized by DARPA (Defense Advanced Research Projects Agency), and the ARDA (Advanced Research and Development Activity) Advanced Question & Answering Program. DUC started out as a workshop in the TIDES program and evolved into a text summarization evaluation forum. Sponsored by ARDA, DUC was organized by NIST with the objective of further progress in automatic summarization by allowing researchers to participate in large-scale experiments. No data were distributed in 2000. In 2001, single-and multi-document summaries (around 10 documents per set) were evaluated by their coverage and readability by 10 human judges. In 2002, NIST provided 60 reference sets, 30 for training and 30 for testing. Each collection contained documents, perdocument summaries, and multi-document summaries, with sets defined by different types of criteria. In 2003, 3 collections were used: • 30 TREC document clusters of 10 documents on average (AP 1998-2000, New York Times 1998-2000, Xinhua News Agency 1996-2000); s/events/timespans and a subset of corresponding documents); • 30 TREC Novelty track document clusters of 22 documents on average (Financial Times of London

  data are distributed according to the Agreement Concerning Dissemination of DUC Results and the User TREC and TIPSTER user agreements. DUC 2006-2007 data are under the AQUAINT agreement.

Figure 1 :

 1 Figure 1: The difference between the ROUGE scores of native abstracts and abstracts associated with the other texts

Figure 2 :

 2 Figure 2: The difference between the INEX scores of native abstracts and abstracts associated with other the texts

Figures 1 and 3 25 -Figure 3 :Figure 4 :

 32534 Figures1 and 3show that ROUGE and GRAD perform equally well in their ability to assign a higher

Table 1 :

 1 Test collections (INEX/CLEF Tweet Contextualization 2011-2014)

		INEX 2011	INEX 2012	INEX 2013	INEX 2014
			XML dump of English Wikipedia
	Corpus
		April 2011	November 2011	November 2012	November 2012
		132 tweets
		(tweet = title +	1000 tweets	598 tweets	240 topics from RepLab 2013
	Queries	1-st snt of a NYT article)	from informative accounts	from informative accounts	(tweet + entity + category)
	Evaluation
	(informativeness/			50 tweets / 10 tweets
	readability)	50 tweets / 53 tweets	50 tweets / 18 tweets	with the largest text references	50 tweets/12 summaries per run
				Prior set of relevant pages
	Gold	New York Times articles		Pool selection of submitted passages	Pool of relevant sentences
	standards	Pool of relevant passages Pool of relevant passages	All relevant texts+10 random tweets	Pool of noun phrases

  that contains collections of scientific literature in all disciplines covering journal archives, digital books, databases, texts corpora etc. from the following publishers: Elsevier, Wiley, Springer, Oxford University Press, British Medical Journal, IOP Publishing, Nature, Royal Society of Chemistry, De Gruyter, Ecco Press, Emerald, Brill, Early English Books Online. We collected articles that contain authors abstracts, editorial or/and web summaries. We selected 4,234 full texts in TXT format with corresponding summaries provided as XML descriptions. We call this data set ISTEX.We also selected 4,535 articles with abstracts about environmental sciences from ISTEX (referred as Environmental). ISTEX documents are labeled by Web of Science categories. So, we retrieved documents labeled Environmental Studies or Environmental Science.

	4.2.2 Wikipedia.

Table 2 :

 2 Percentage of times when a human written abstract (H) is scored higher/lower/equally than/to a generated summary (S). The best values of H > S are in bold. + / -indicates accuracy significantly higher/lower than the accuracy of RAND.

				RandSum		CosSum
	Data Method
			H > S	H < S H = S	H > S	H < S H = S
		COS	16.60% -83.40% 0.00% 16.33% -83.66% 0.01%
	ISTEX	ROUGE 14.49% -85.31% 0.19% 17.61% -82.13% 0.27% INEX 61.39% + 38.61% 0.00% 62.04% + 37.88% 0.08%
		RAND 49.67% 50.33% 0.00% 50.01% 49.99% 0.00%
		GRAD 62.97% + 35.54% 1.48% 79.63% + 19.24% 1.13%
		COS	3.2% -	96.2% 0.6%	0.4% -	99.0% 0.6%
	Environmental	ROUGE 8.0% -INEX 5.8% -RAND 48.9%	91.2% 0.8% 14.9% -74.2% 0.9% 93.5% 0.7% 6.2% -93.1% 0.7% 51.1% 0.0% 49.3% 50.7% 0.00%
		GRAD 89.8% + 10.2% 0.0% 84.2% + 15.8% 0.0%
		COS	23.13% -76.85% 0.02% 17.17% -82.80% 0.02%
	Wikipedia	ROUGE 14.48% -85.31% 0.21% 22.21% -77.37% 0.43% INEX 14.08% -85.75% 0.17% 12.66% -87.17% 0.17%
		RAND 49.49% 50.51% 0.00% 49.00% 51.00% 0.00%
		GRAD 71.60% + 21.80% 6.60% 92.91% + 1.84% 5.25%

Table 3 :

 3 Descriptive statistics on Wikipedia data

		INEX ROUGE	COS	GRAD
	Mean	0.0671 0.0666 0.2788	0.0002
	Standard error	0.0004 0.0003 0.0004	0.0000
	Median	0.0297 0.0501 0.2825	0.0001
	Standard deviation 0.0838 0.0603 0.0860	0.0050
	Variance	0.0070 0.0036 0.0074	0.0000
	Kurstosis	1.8581 10.6040 -0.1688 16482.4494
	Skewness	1.5985 2.3678 -0.1224	120.1888
	Minimum	0.0000 0.0000 0.0000	0.0000
	Maximum	0.5255 0.8750 0.6173	0.7757

http://duc.nist.gov/

https://tac.nist.gov/

https://www.ldc.upenn.edu

Although this amount can be considered as limited, the INEX organizers shown than competitors ranking is not affected by variating the amount of tweets selected for evaluation.

https://trec.nist.gov/data/robust.html

http://www.istex.fr/

Acknowledgements

The authors would like to thanks Marianne Noël, Frédérique Bordignon, Nicolas Turenne, Anton Firsov, and Université Paris Descartes, Frontières du vivant.

All existing metrics, except GEM, have relative values allowing candidate summaries to be ranked, but they are not applicable for comparison of an isolated summary with the full text nor for comparison of metric scores for summaries of different documents. Thus, they enable us to answer the question Which summary is better? but not Is this summary a good one?

Another problem with the existing metrics (except GEM) is their output values. Theoretically, the majority of metrics are normalized, but in practice, the values tend to be quite small, usually ROUGE score is less than 0.2 (see Table 3). This means that the isolated values produced by these metrics cannot be interpreted, i.e. it is impossible to say, for example, whether the score of ROUGE-1 equal to 0.17 is good or not.

Structured abstracts tend to be informative [START_REF] Fontelo | Comparing data accuracy between structured abstracts and full-text journal articles: implications in their use for informing clinical decisions[END_REF]. One of the metrics taking into account document structure is BM25F [START_REF] Robertson | Simple BM25 Extension to Multiple Weighted Fields[END_REF], a field-based extension of Okapi's BM25 from the information retrieval. However, it is not suitable for summary scoring since it also gives a relative score allowing search result ranking. GEM considers the document structure but its major problem is that it is unable to distinguish summaries coming from different texts. This is an implementation drawback of the metric GEM explained in the previous section.

The analysis of the existing evaluation metrics shows the following future perspectives in the area:

• absolute values of metrics would allow us to extend their application, for example, to the domain of education or to the automatic evaluation of scientific abstract quality and even automatic reviewing;

• the interpretability of metrics remains problematic (i.e. how to interpret the score of an individual summary and answer the question Is this summary a good one?);

• vocabulary overlap-based metrics should be not used with full texts as references. Metrics not based on vocabulary overlap should be applied in the case where a full text is used as a reference.