Rémi Auguste
email: remi.auguste@univ-lille1.fr

Pierre Tirilly
email: pierre.tirilly@univ-lille1.fr

Jean Martinet
email: jean.martinet@univ-lille1.fr

Introducing FoxPersonTracks: a Benchmark for Person Re-Identification from TV Broadcast Shows

This paper introduces a novel person track dataset dedicated to person re-identification. The dataset is built from a set of real life TV shows broadcasted from BFMTV and LCP TV french channels, provided during REPERE challenge. It contains a total 4,604 persontracks (short video sequences featuring an individual with no background) from 266 persons. The dataset has been built from the REPERE dataset by following several automated processing and manual selection/filtering steps. It is meant to serve as a benchmark in person re-identification from images/videos. The dataset also provides re-identifications results using space-time histograms as a baseline, together with an evaluation tool in order to ease the comparison to other reidentification methods.

I. INTRODUCTION

Person re-identification from video is a fundamental task consisting in identifying an individual across shots of a single video or across various videos. It is an essential component of multi-camera tracking and has applications in automated videosurveillance, robotics and human computer-interaction, multimedia retrieval, and forensics.

Re-identifying individuals from real-life videos requires algorithms that are robust to changes in light, viewpoint, scale, and target appearance (due to gestures and, in some use cases, clothes, hair style, etc.). Developing and evaluating such algorithms requires benchmarks that offer data and evaluation protocols that encompass all the features of reallife data and all application domains. Current benchmarks for person re-identification from videos are limited in their range of applications (videosurveillance, mostly), size and settings. Additionally, recent work has shown the usefulness of reidentification for multimedia data understanding [START_REF] Favre | Percoli: A person identification system for the 2013 repere challenge[END_REF], calling for more research in this direction, and, consequently, dedicated benchmarks. This paper presents a new dataset dedicated to the training and evaluation of methods for person re-identification in TV broadcast shows. This dataset is built from a set of real-life TV shows broadcasted from BFMTV and LCP TV french channels and provided during the REPERE challenge [START_REF] Galibert | The first official repere evaluation[END_REF], and is available for download on ELDA website (URL elda.f r).

The paper is organized as follows : Section II describes existing re-identification datasets. Section III introduces the proposed dataset by describing its detailed content. Section IV presents experimental results on re-identification using spacetime histograms, that are meant to serve as a baseline for this dataset. Section V concludes our work.

II. RELATED WORK

To our knowledge, all the existing datasets dedicated to reidentification bench-marking are based on videosurveillance data [START_REF] Bedagkar-Gala | A survey of approaches and trends in person re-identification[END_REF], with the exception of the Rope dataset [START_REF] Gandhi | Detecting and naming actors in movies using generative appearance models[END_REF], based on the movie Rope by A. Hitchock.

Most datasets [START_REF] Gray | Evaluating appearance models for recognition, reacquisition, and tracking[END_REF] do not provide actual video data but only frames (eventually cropped) extracted from videos. In these datasets, the re-identification problem is set as follows: given a set of images, find the images that depict a single given individual. Based on this statement, the datasets only differ by their acquisition conditions (illumination, occlusions, viewpoint, etc.

-see [START_REF] Bedagkar-Gala | A survey of approaches and trends in person re-identification[END_REF] for details) and their statistics (provided in Table I): number of individuals to be re-identified, number of images per individual, presence or not of distractor images (images that do not contain re-identifiable individuals). Although it focuses on the problem at hand, this definition of the re-identification problem:

• does not allow to benchmark re-identification algorithms based on motion features;

• offers a limited range of acquisition conditions: illumination, occlusions, poses, etc.;

• prevents the evaluation of the impact of potential preprocessing steps (e.g. body/face detection, background subtraction) on the re-identification performance.

Besides, as shown in Table I The datasets that are the closest to our work [START_REF] Cinbis | Unsupervised metric learning for face identification in tv video[END_REF], [START_REF] Everingham | Hello! My name is... Buffy" -automatic naming of characters in TV video[END_REF] are both dedicated to face recognition in videos. They are based on different episodes from the TV series Buffy the Vampire Slayer. These datasets provide face tracks, i.e. video sequences featuring a single face, labeled by the name of the corresponding character. Their statistics are detailed in Table II: although they provide real-world data, the number of persons to be (re-)identified is small, allowing to estimate the performance of the algorithms on a limited scale only.

Finally, one should also mention the Hannah dataset [START_REF] Ozerov | On evaluating face tracks in movies[END_REF], dedicated to face tracking. Like the two Buffy datasets, it provides facetracks annotated by persons' names. Although it includes a fairly high amount of facetracks and persons (see Table II), the fact that it is based on a single movie makes the facetracks poorly representative of the complexity of its use case.

Dataset

# Persons # Face tracks Buffy (INRIA) [START_REF] Cinbis | Unsupervised metric learning for face identification in tv video[END_REF] 8 (+1) 639 Buffy (Oxford) [START_REF] Everingham | Hello! My name is... Buffy" -automatic naming of characters in TV video[END_REF] 12 (+2) 1,526 Hannah [START_REF] Ozerov | On evaluating face tracks in movies[END_REF] 238 (+16) 2,002 As compared to existing benchmarks, our dataset:

• covers an original yet significant use case of reidentification: TV broadcast shows;

• provides a large number of tracks (over 4,600) and a large number of individuals to be re-identified (266);

• provides full video shots rather than individual images, allowing to evaluate motion-based methods in addition to visual-based ones;

• is based on real-world data: excerpts from actual French TV broadcast shows.

III. DATASET DESCRIPTION

The main objective of this dataset is to evaluate reidentification algorithms, i.e. algorithms that aim at grouping shots that feature the same person (without explicitly naming this person). Baseline for face and upper body detection and segmentation provide a common ground to compare the absolute performance of various algorithms. Related tasks that can be evaluated using this dataset includes: person identification from video, face detection from video, relative impact of external components on the final re-identification performance (face detection, upper body detection, face/upper body segmentation).

A. Dataset contents

The proposed dataset is composed of:

• 4,604 video sequences from various TV shows, each featuring one of 266 different persons; such a sequence is called persontrack in the remainder of the paper;

• ground truth data providing the full name of the person for each shot;

• ground truth data providing the face position for a subset of 2081 keyframes extracted from the shots;

• baseline face detections for all frames of the shots;

• baseline detections of faces;

• baseline background subtraction data based on the detected face;

• evaluation software to compute the metrics presented in Section IV.

All data referred to as ground truth have been obtained manually. Data referred to as baseline as been obtained automatically.

In the remainder of this section, the source of the data and the processes used to produce the ground truth and baseline data are described, then statistics about the dataset are provided.

B. Data source

The original data used in this work was distributed as part of the REPERE challenge [START_REF] Galibert | The first official repere evaluation[END_REF], which aimed at providing a benchmark for person identification from broadcast TV shows. During the challenge, 299 videos were released, that covered 9 TV shows originally broadcasted by the French TV channels LCP and BFMTV. This data came together with manually obtained ground truth for speech recognition and speaker recognition over the whole data, and OCR and face detection over a number of keyframes.

The dataset described here has been produced based on 134 videos from the REPERE dataset (videos available during phases 0 and 1 of the challenge) and their associated ground truth.

REPERE

C. Persontrack extraction and filtering

The process used to generate the persontracks from the original videos is illustrated in Figure 1. First, the REPERE ground truth is used to extract sequences of consecutive frames annotated with names. The ground truth directly provides the first and last frames of such sequences together with the name of the person appearing in it. Two or more of these sequences may overlap, meaning that more than one person is present in them. Also, some persons present in the sequence may not be annotated (e.g. persons in the audience) according to REPERE's annotation guidelines. The remaining steps of this process aim at removing these sequences to provide "pure" persontracks containing only a single person.

Overlapping sequences are then removed from the dataset, based on the ground truth information. After this second step, the only sequences that may contain more than one person are the ones that were not fully annotated. To deal with such sequences, faces are counted using a face detector based on the Viola and Jones algorithm [START_REF] Viola | Robust real-time object detection[END_REF] and a post-processing step that filters out false positives based on color. Every frame in which more than one face was detected is manually inspected: if the frame actually contains more than one face, the whole corresponding sequence is discarded. After this third step, the only remaining sequences that may contain more than one person are those on which the ground truth was incomplete and the detector failed. All these sequences are manually inspected and every sequence containing more than one person is discarded. This last filtering step ensures the accuracy of the dataset. At the end of the filtering, we obtain 4,604 persontracks of 266 different persons.

D. Baseline data generation

The face detections provided in our dataset are obtained using the Viola and Jones detector [START_REF] Viola | Robust real-time object detection[END_REF], and an additional postprocessing step that computes the ratio of skin color in the detection window to filter out false positives and to resize the detection windows so that they contain as little background pixel as possible. Figure 2 shows an example of a detection window before and after this post-processing step.

After faces have been detected, the upper bodies are extracted based on a simple extension of the detection window. Then, the Grabcut algorithm is used to subtracts the background from the resulting windows (containing both the face and upper body). Figure 3 shows an example of these person-centered, background-free facetracks that are provided as baseline data in our dataset. Figure 4 shows the occurrence distribution in a log-scale. Since the original videos are broadcast TV shows, the journalists (anchorpersons) appear more frequently than any other person. Some journalists appear in more than 50 shows, while many other persons appear only once.

Figure 5 shows the persontrack length distribution in the dataset. It shows that most persontracks are short in length, with an average of 55 frames. Note that as the peak shows, one third of the persontracks has a length between 31 and 38 frames.

IV. BASELINE RESULTS

The different steps leading towards our dataset have been evaluated. The face detection and face extraction algorithm were evaluated during the REPERE challenge using the official metrics [START_REF] Kahn | A presentation of the repere challenge[END_REF]. We evaluated the results of our re-identification approach ourselves [START_REF] Auguste | Space-time histograms and their application to person re-identification in tv shows[END_REF]. The first step of our re-identification approach is to build a space-time histogram (RGB, 1,500 bins) for each persontrack and then calculate a similarity matrix per show. The similarity measure for space-time histograms is a combination of the χ 2 and the mahalanobis distance (used as a similarity). A precision at n (P@N) [START_REF] V. Raghavan | A critical investigation of recall and precision as measures of retrieval system performance[END_REF] is then computed for each persontrack and the weighted average P is used as an evaluation measure:

P = |M | i=1 P i n i |M | i=1 n i (1)
where M is the similarity matrix, i are the persontracks, n i the number of persontracks bearing the same identity than i and P i is the precision at n i .

V. CONCLUSION

We have introduced a new re-identification dataset named FoxPersonTracks. This dataset contains 4,604 persontracks showing 266 individuals, and compared to other existing datasets, it provides a large number of persontracks and a large number of individuals, with a wide range of acquisition conditions from several TV shows; also, it provides full video shots rather than individual images, allowing to use the motion. Finally, it provides an evaluation tool making it easy to compare algorithms in a unified way according to the presented metrics. We believe that this dataset is useful as a benchmark for person re-identification from images/videos.

Fig. 1 .

 1 Fig. 1. Steps for building the presented dataset from the REPERE dataset.

Fig. 2 .

 2 Fig. 2. Example of face detection before and after our optimization based on the skin color proportion.

Fig. 3 .

 3 Fig. 3. Example of extracted persontrack from a BFMTV TV show.

Fig. 4 .

 4 Fig. 4. Number of occurrences per identity distribution in the dataset ranging from 1 to 278 (x-axis) with an average of 17. The y-axis is logarithmic.

Fig. 6 .

 6 Fig. 6. Screenshots of representative persontracks featured in our dataset.

Fig. 5 .

 5 Fig. 5. Duration in frames distribution of our datasets ranging from 7 frames to 896 frames (x-axis) with an average of 55 frames. The y-axis is logarithmic.

TABLE II .

 II PROPERTIES OF EXISTING BENCHMARKS FOR FACE RECOGNITION IN VIDEOS. NOTE THAT "(+N)" DENOTES ADDITIONALNON-CHARACTER-SPECIFIC CLASSES ("OTHERS", "CROWDS", "KIDS", "UNIDENTIFIED", AND "FALSE POSITIVES").

TABLE III .

 III COMPARISON OF THE PRECISION OF THE RE-IDENTIFICATION ALGORITHM.


This article presents Apex, a system that can automatically assess a student essay based on its content. It relies on Latent Semantic Analysis, a tool which represents the meaning of words as vectors in a high-dimensional space. By comparing an essay and the text of a given course on a semantic basis, our system can measure how well the essay matches the text. Various assessments are presented to the student regarding the topic, the outline and the coherence of the essay. Our experiments yield promising results.

INTRODUCTION

This paper describes Apex (for an Assistant for Preparing EXams), a tool for evaluating student essays based on their content. It relies on a semantic text analysis method called Latent Semantic Analysis (LSA) [START_REF] Landauer | A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge[END_REF]. Basically, LSA represents each word of a text as a vector in a high-dimensional space such that the proximity between two vectors is closely related to the semantic similarity between the two corresponding words. Since a text is composed of words, this measure of similarity can be easily extended to compare texts. Thus, it is possible to compare two student essays, or a student essay and the text of a course, on a semantic basis.

Our goal is not only to grade essays per se but more generally to provide students with an environment to help them learn a domain by writing an essay, getting feedback, revising the text, etc. This process is compatible with the Hayes and Flower model of text composition [2]. However, our environment differs from other composition tools or tutors that provide lexical or grammatical information because it decreases the cognitive effort of the writer. Instead of building an intrusive tool, we aim to develop an openended learning environment [START_REF] Land | A Conceptual Framework for the Development of Theories in Action with Open-Ended Learning Environments[END_REF] in which the user is viewed as an active constructor of knowledge.

The paper is organized as follows: first, we present LSA and the various intelligent assessors or tutors that are based on this tool; second, we describe our tool, Apex, as well as two experiments we performed.

LATENT SEMANTIC ANALYSIS

Researchers in information retrieval (IR) have tried to identify lexical-semantic relations between words of text corpora. Evens, Vandendorpe and Wang [START_REF] Evens | Lexical-semantic relations for information retrieval[END_REF] analyzed lexical-semantic relations of words to extract both syntactic and semantic information of bibliographic queries. The authors developed IRS (Information Retrieval System), an IR automated procedure for revealing semantic relations from dictionary definitions. IRS is a system in which each bibliographic reference is an automatically extracted list of keywords used to identify documents. The system views each list as a documentvector. The user writes a paragraph, which is first transformed into a query-vector, and it is then compared with the document-vectors by computing the cosine between the former and the latter. The closer to 1 is the cosine, the closer to the query is the document retrieved. This method adequately expresses both syntactic and semantic information without complex data preprocessing.

Following these ideas, a new method, called Latent Semantic Analysis, gives promising results for semantic text processing. Before presenting LSA-based intelligent tutors or assessors, we need to describe LSA.

Description of LSA

In order to perform a semantic matching of pieces of text, LSA relies on large corpora of texts to build a semantic high-dimensional space containing all words and texts, by means of a statistical analysis [START_REF] Deerwester | Indexing by Latent Semantic Analysis[END_REF][START_REF] Landauer | An Introduction to Latent Semantic Analysis[END_REF].

Basically, the semantics of a word is determined from all of the contexts (namely paragraphs) in which that word occurs. For instance, the word bike occurs generally in the context of handlebars, pedal, ride, etc. Therefore, if a word like bicycle occurs in a similar context, the two words will be considered close to each other from a semantic point of view. Their corresponding vectors in the semantic space will be also close to each other. This semantic space is built by considering the number of occurrences of each word in each piece of text (basically paragraphs). For instance, with 300 paragraphs and a total of 2,000 words, we get a 300 x 2,000 matrix. Each word is then represented by a 300dimensional vector and each paragraph by a 2,000-dimensional vector. So far, it is just straightforward occurrence processing. The power of LSA, however, lies in the reduction of these dimensions, and in so doing induces semantic similarities between words. All vectors are reduced by a method close to eigenvector decomposition to, for instance, 100 dimensions. The matrix X is decomposed as a unique product of three matrices: X = T 0 S 0 D' 0 such that T 0 and D 0 have orthonormal columns and S 0 is diagonal. This method is called singular value decomposition. Then only the 100 columns of T 0 and D 0 corresponding to the 100 largest values of S 0 are kept, to obtain T, S, and D. The reduced matrix X such that: X = TSD' permits all words and pieces of texts to be represented as 100-dimensional vectors. It is this reduction which is the heart of the method because it allows the representation of the meaning of words, by means of the context in which they occur.

If the number of dimensions is too small, too much information is lost. If it is too big, not enough dependencies are drawn between vectors. A size of 100 to 300 gives the best results in the domain of language [7]. This method is quite robust: a word could be considered semantically close to another one although they do not co-occur in texts. In the same way, two documents could be considered similar although they share no words. An interesting feature of this method is that the semantic information is derived only from the co-occurrence of words in a large corpus of texts. There is no need to code semantic knowledge by means of a semantic network or logic formulas.

Various tests of LSA

We now present several experiments that test LSA. One experiment [START_REF] Landauer | A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge[END_REF] consisted in building a general semantic space from a large corpus of English texts, then testing it with the synonymy tests of the TOEFL (Test Of English as a Foreign Language). One word is given, and the task is to identify which word among a choice of four possible answers is semantically closest to it. LSA performed the TOEFL synonymy test by choosing the word with the highest similarity between its vector and the vector of the given word. LSA results compare favorably with the level of foreign students admitted to American universities.

In another study [START_REF] Foltz | Latent Semantic Analysis for text-based research[END_REF], subjects were asked to write essays from a number of texts. These essays were then ranked by human graders. Their task was to assess the adequacy between the essay and the texts. In parallel, LSA was "trained" with the texts and ranked the essays according to the semantic proximity between each of them and the texts. LSA results compare favorably again with the human results.

LSA also proved to be successful in checking hypertext links from a semantic point of view [START_REF] Dessus | Semantic Checking of Hypertext Links[END_REF]. In other words, LSA can test whether a link between two pages is appropriate based on their semantic content.

Landauer and Dumais considered LSA a good model of human learning [START_REF] Landauer | A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge[END_REF]. Their assumption is that humans construct knowledge by deriving word meaning from context while reading [START_REF] Fukkink | Effects of Instruction in Deriving Word Meaning from Context: A Meta-Analysis[END_REF]. Actually, Rehder et al. show that LSA is an appropriate tool to assess human knowledge in a specific domain [START_REF] Rehder | Using Latent Semantic Analysis to assess knowledge: Some technical considerations[END_REF].

Incorporating LSA into instructional software: tutors and assessors

LSA was also used in the field of tutoring systems and intelligent assessors. This section is devoted to the presentation of various applications of LSA in that domain.

Although LSA has been used for over ten years in several domains like information retrieval, machine learning, text analysis and so on, few LSA-based studies have been performed to improve instructional software. To our knowledge, the earliest attempt at implementing LSA in an educational tutor was carried out in 1997 by Stahl [START_REF] Stahl | Allowing learners to be articulate: incorporating automated text evaluation into collaborative software environments[END_REF]. Stahl suggests that LSA may be appropriate in education in the following ways:

1. to find the optimal text for learning; LSA can match student essays with texts in the same domain, in order to provide the student with the right amount of new information; 2. to connect students with each other and with relevant experts; LSA can assess the areas of interest or the level of knowledge of the users from their productions and then suggest matches; 3. to automatically assess writing, as we mentioned earlier.

According to these ideas and the previous psychological experiments, we can consider LSA an appropriate model of knowledge representation for designing instructional software. The following sections review various applications.

Application one-Autotutor

Wiemer-Hastings et al. [START_REF] Wiemer-Hastings | Improving an Intelligent Tutor's Comprehension of Students with Latent Semantic Analysis[END_REF][START_REF] Wiemer-Hastings | Approximate Natural Language Understanding for an Intelligent Tutor[END_REF] are developing Autotutor, an ITS with multimedia and human-like interaction (the authors plan to rely on speech synthesis and simulated facial movements) in the domain of computer literacy. One of the purposes of Autotutor, for which LSA is required, is to answer unrestricted student questions. It does so by selecting the closest piece of text to the question. The main interest is that the student is not limited to single word questions. Moreover, the system performance degrades gracefully which is usually not the case in natural language systems. However, since the user questions are quite short (10 to 20 words), it is likely that Autotutor does not use all of the power of LSA. Indeed the authors show quite similar performances when relying on keyword matching rather than LSA [START_REF] Wiemer-Hastings | How Latent is Latent Semantic Analysis?[END_REF] which is usually not the case in other LSA experiments [START_REF] Lemaire | Models of High-dimensional Semantic Spaces[END_REF]. Autotutor is a large project in progress, for which further evaluation in context is planned.

Application two-Language learning and game learning

LSA is not only a tool for matching texts on a semantic basis, but also a model of human learning, since LSA learns semantic similarities between words from the reading of texts, just as humans do. Actually, it was shown [START_REF] Landauer | A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge[END_REF] that the rate of vocabulary learning with LSA compares with the human rate between the ages of 2 and 20: when provided with 3,500 words a day, LSA learns 10 new words per day which is roughly close to the human data. This model of learning was used in a tutoring system to select the best stimulus to which to expose the student in order for learning to be optimal [START_REF] Lemaire | Tutoring Systems based on Latent Semantic Analysis[END_REF]. LSA is used to represent the domain model and the user model in the same semantic space. Two applications were designed. In the first one, the system selects English texts for French learners to read, according to their understanding of previous texts. The second one extends LSA to knowledge domains other than language. It helps students learn a game by playing with them. It selects a move not to win but in order that the next state of the game is the optimal stimulus for the student.

Application three-Intelligent Essay Assessor

Much work has been conducted in the field of automatic grading but the systems are mainly based on multiple choice exams [START_REF] Blank | Interactive Gradebook: the Missing (Hyper)Link[END_REF][START_REF] Stockburger | Automated Grading of Homework Assignments and Tests in Introductory and Intermediate Statistics Courses Using Active Server Pages[END_REF]. These grading programs are not hard to make. The difficulty lies rather in the design of the propositions which should be close enough to the right answer but still wrong. An alternative to multiple choice tests is to ask the student to write an essay about what he or she knows about a domain and then to compare that text to pre-graded texts.

As Wresch [START_REF] Wresch | The imminence of grading essays by computer-25 years later[END_REF] pointed out, automated grading of essays by computer began thirty five years ago with Page's work [START_REF] Page | The imminence of grading essays by computer[END_REF]. Page looked for correlations between simple features of student texts and the corresponding teacher grades. The most statistically significant features were: average word length (r = .51), number of common words (r = -.48), number of commas (r = .34), number of words (r = .32), number of prepositions (r = .25). Although these rough measures have often been criticized, some recent systems are closely related to Page's work (e.g. the Grammatik proofreading software). This approach has a major drawback: it only focuses on surface text features. Neither microstructure nor semantic textual characteristics are taken into account by these programs.

LSA can address these drawbacks because of its ability to capture semantic information. This is the foundation of the Intelligent Essay Assessor (IEA) [START_REF] Foltz | Automated Essay Scoring: applications to Educational Technology[END_REF]. IEA is first "trained" on several texts related to the domain. The student essay is then compared with pre-graded essays by means of two kinds of scores:

1. the holistic score, which returns the score of the closest pre-graded essay; 2. the gold standard, which returns the LSA proximity between the student essay and a standard essay.

An experiment using the holistic score was performed on 188 essays on biology. A correlation of .80 was shown between IEA grades and human graders.

One main interest of IEA is that the student can submit the essay again and again in order to improve the score. Foltz et al. note that the average grade increases from 85/100 (first submission) to 92/100. However, one problem is that the teacher needs to grade essays beforehand.

Application four-Summary Street

Summary Street and State of Essence [START_REF] Kintsch | and the LSA Research Group, Developing Summarization Skills through the Use of LSA-based Feedback[END_REF][START_REF] Stahl | and the LSA Research Group, Evolution of an LSA-Based Interactive Environment for Learning to Write Summaries[END_REF] are built on top of LSA. Both systems help students write good summaries. First of all, the student is provided with general advice on how to write a summary: select the most important information, find two or three ideas, substitute a general term for lists of items, not include trivial information, etc. Then the student selects a topic, reads the text and writes out a summary. LSA procedures are then applied to give a holistic grade to the summary. The system also warns the student if the summary is too long or too short.

Application five-Select-a-Kibitzer

In the same way, one of the designer of Autotutor has recently built Select-a-Kibitzer [START_REF] Wiemer-Hastings | Select-a-Kibitzer: A computer tool that gives meaningful feedback on student compositions, Interactive Learning Environments[END_REF], an agent-based computer tool that assesses student compositions. Each agent is responsible for providing a kind of advice: coherence, purpose, topic, and overall quality. It is worth noting that these assessments are quite similar to those provided by the previous applications. However, this application emphasizes the negotiated construction of the text by associating each agent to a character. Now we will present Apex, a similar system that we built on top of LSA, to help students learn a domain. Apex differs from IEA in that it does not rely on pre-graded essays but rather on various semantic comparisons with the course. It also differs from Summary Street and Select-a-Kibitzer since it takes into account the structure of the course to grade an essay.

APEX

As we indicated earlier, Apex can be used to grade a student essay with respect to the text of a course; however it can also provide detailed assessments on the content, the outline and the coherence of a student essay. The environment is designed so that the student can select a topic, write an essay on that topic, get various assessments, then rewrite the text, submit it again, etc. The only information Apex needs to process a student essay is the text of the course. This text has to be marked up by the teacher, so that it is divided into topics, and so that each topic is divided into notions. Basically, this structure corresponds to an outline view of a word processor, so it is quite straightforward to mark up such a document. First level titles should begin with #T (for Topic) whereas second level titles should begin with #N (for Notion). Only second level titles are followed by a paragraph. A notion can belong to several topics. However, the text of such a notion is written only once. In case of cross-references, the notion title should begin with #S. Figure 1 gives an example of such a marking-up. The notion title "Introduction to the solar system" is defined in the topic "The solar system" and referred to in the topic "Eclipses".

-Insert Figure 1 here-It is worth noting that we added a very large set of French texts (290,000 words) to the course, in order to improve the semantic knowledge of the system. These texts (three French novels) were not related to the course but they were useful for the system to better "understand" the non domain-dependent terms of the student essay.

Once connected to the system, the student selects a topic that he or she wishes to work on. The student then types a text about this topic in a text editor. At any time, he or she can get an evaluation of the essay.

Different types of evaluation

Three kinds of evaluation are available: content-based, outline-based or coherencebased. After reading any of the three evaluation texts, the student goes back to the essay, modifies the text and submits it again.

Content-based assessment

At the content level, the system identifies how well the notions are covered by requesting LSA to measure a semantic similarity (in the range [-1, 1]) between the student text and each notion of the selected topic. For instance, if the student selects the topic "Eclipses", he or she has to write an essay about that topic. There are three notions in that topic; therefore the student text will be successively compared with each of them. If the similarity is high, it means that the student has covered well the corresponding notion. Actually, the system provides a message to the student according to the value of the similarity. The current version of Apex relies on four categories:

1. if the similarity ∈ [-1; 0.1[, the message is "the topic was covered very poorly"; 2. if the similarity ∈ [0.1; 0.5[ the message is "the topic was covered poorly"; 3. if the similarity ∈ [0.5; 0.7[ the message is "the topic was covered well"; 4. if the similarity ∈ [0.7; 1] the message is "the topic was covered very well".

However, the number of categories as well as the corresponding values and the text of the message can be fully parametrized depending on the content of the course. To do so, the teacher has to edit and modify a text file which is shown in Figure 2.

-Insert Figure 2 here-In addition to this assessment, Apex provides a general grade of the student text. This grade is the average grade for each notion, multiplied by 20 in order to get a grade between 0 and 20 as is usual in the French school system. Figure 3 shows an example of a content-based evaluation of a student essay in the domain of cognitive ergonomics.

-Insert Figure 3 here-One problem we had to tackle concerns very short student texts. When the student has written out just a few words, the content-based assessment could yield a high score, although one might want the assessment to be considered low. Therefore, if the number of words of the student text is below 300 words (that value can be easily changed in the parameter file), all the content thresholds described earlier are raised so that the assessment is more severe: the shorter is the text, the closer to 1 becomes the thresholds. Basically, with N being the number of words of the essay, all content thresholds T are changed to:

T + (1-T )( 3 0 0-N 3 0 0 )
We also designed Apex on-line, a version which runs continuously in order for the student to get a real time assessment. While the student is typing out texts, the Apex window constantly refreshes the assessment. Instead of displaying the quite long natural language message about which notion is well or poorly covered, the system displays a scale on which all the notions of a topic are represented as numbers. The far right part of that scale corresponds to a similarity of 1, the far left to a similarity of 0. Therefore, the student can immediately identify the notions that are covered badly. The scale looks like this: In this example, the student knows that he or she has to work mainly on notions 2 and 6 to improve the text.

Outline-based assessment

At the outline level, the system displays the most similar notion of the course for each paragraph of the essay. The goal is to provide the student with an outline view of the essay. If the similarity computed by LSA is too low, the system prints that there is no predicted notion. This threshold is currently set to 0.4 but this can be changed easily by means of the parameter file we mentioned earlier. Figure 4 shows an example of an outline-based assessment. For instance, the first paragraph of the student text has been recognized by Apex as being concerned with the definition of activity. If this is not what was intended, the student should probably rework this paragraph.

-Insert Figure 4 here-

Coherence-based assessment

LSA has been used already to measure text coherence and has proven to be successful [START_REF] Foltz | The Measurement of Textual Coherence with Latent Semantic Analysis[END_REF]. At the coherence level, the system relies on LSA to measure semantic proximities between adjacent sentences. Therefore Apex can detect coherence breaks. Then it gives an average measure of coherence and if necessary an example of an important conceptual break between two sentences. Figure 5 shows an example of a coherencebased assessment: the student is required to work on the text again, and in particular to correct the linking of ideas between sentence 2 and sentence 3.

-Insert Figure 5 here-

Software information

LSA was designed by Bellcore Labs from which we obtained the source code, written in C under Unix. Apex is also written in C under Unix. The approximate duration for the assessments, with a 192 Ko course text, and a 1.9 Mo additional French text, on a workstation are:

• 300 ms per notion for content-based assessment;

• 3 seconds per paragraph of the student text for outline-based assessment;

• 150 ms per sentence of the student text for coherence-based assessment.

EXPERIMENTS

This set of experiments is based on a graduate course on the sociology of education at our university.

Correlation with human grades

In this experiment, we took 31 essays that were written a year ago by students at the university of Grenoble. We typed them out and ran Apex in order to get the general grade between 0 and 20 for each student essay. We first compared these grades with the grades the teacher had given a year ago. We got a significant correlation (r = .59, p < .001).

Then all essays were ranked by two judges who were teachers in similar domains. They had studied the relevant part of the course several times before the task. They graded each of the 31 texts in the same way that Apex does: each text was given 14 grades corresponding to the estimated adequacy between each of the 14 notions. Then an average grade was computed. We got significant correlations with Apex grades (r 1 = .59, p < .001; r 2 = .68, p < .0001). However these aggregate measures could not be refined due to a lack of variability in the 14 notions.

All of these results are in the range of correlations found in the literature. Wiemer-Hastings et al. [START_REF] Wiemer-Hastings | How Latent is Latent Semantic Analysis?[END_REF] had also compared LSA grades and human grades and none of the correlations were above .5. Foltz's similar correlations [START_REF] Foltz | Latent Semantic Analysis for text-based research[END_REF] .86] were obtained by Foltz et al. [START_REF] Foltz | Automated Essay Scoring: applications to Educational Technology[END_REF]. These repeated experiments show that LSA is a adequate tool for assessing knowledge, one that rivals the ability of humans.

Effects of Apex feedback on the quality of student essays

We designed a second experiment to assess the effect of Apex on the holistic quality of students essays. Three groups of students were given 45 minutes to write an essay:

1. a control group (N = 11) in which subjects wrote out an essay on a word processor;

2. an Apex-demand group (N = 11), where students wrote out their essays with the same word processor as the control group and were provided with assessments upon request;

3. an Apex on-line group (N = 9) in which the only difference with the latter group is the on-line help provided by Apex without explicit requests from the student.

Our hypothesis was that students in the Apex on-line group would write out better essays than students of the two other groups. The reason is twofold: first, the solicitation of Apex help in the Apex-demand group is supposed to increase the writer's cognitive effort, and second, the help provided is supposed to improve the essay quality as opposed to the control group. This is coherent with the results of Zellermayer et al. [START_REF] Zellermayer | Enhancing Writing-Related Metacognitions through a Computerized Writing Partner[END_REF].

A question was selected that covered 14 "notions" of the course according to the teacher. The 31 students wrote their essays. All essays were ranked by three judges: two of them (graders 1 and 2) were teachers in similar domains and the last one was a student who had passed the course. Each judge gave 14 grades for each text by the same procedure used in the first experiment.

Results are twofold (see Table 1). First, correlations between the human scores and the Apex ones are in a similar range to those obtained elsewhere (see supra). However, these correlations are smaller than the correlations between humans grades (around .8).

-Insert Table 1 here-Second, results (see Table 2) show no significant differences between groups which is coherent with a similar experiment [START_REF] Kintsch | and the LSA Research Group, Developing Summarization Skills through the Use of LSA-based Feedback[END_REF]. We plan to design another experiment to study the student writing process instead of just the performance. Methods such as thinkaloud protocols [START_REF] Kozma | The impact of computer-based tools and embedded prompts on writing processes and products of novice and advanced college writers[END_REF] or retrospective interviews [START_REF] Fatton | How to support in-process planning in a computer-based writing environment[END_REF] would be appropriate.

-Insert Table 2 here-

CONCLUSION

This paper describes a system that can assess student essays based on their content from different points of view. Our first experiment provide promising results. In particular, we show a significant interesting correlation between human grades and Apex grades. This can lead to the development of automatic grading systems not only based on multiple choice exams or surface features of essays, but rather on semantic features of unrestricted essays. This approach can also be used in a distance learning context since students can connect to the system and freely submit essays. The system evaluates the essays as many times as required by the students, without getting bored. Moreover, there is no need for the teacher to code any domain knowledge. The text of the course is all that is required.

Apex is only concerned with the content: the teacher is essential for evaluating spelling, syntax and the general structure of the essay. However, Apex evaluation of the content has some limits. Actually, LSA (which is the heart of Apex) has no way of detecting when a sentence has syntactic errors or when some common words are missing. Therefore, a student could fool LSA by writing an essay with only keywords. However, we think that if a student can provide the right keywords to fool LSA, then this student should have a good knowledge of the domain and that is exactly what we want to measure. The following notions were covered very poorly:

-Automatism-based errors (0.26) -Baille and Testu's experiment on problem solving (0.25)

The following notions were covered poorly:

-Errors based on declarative knowledge (0.40) -Testu's model (1993) (0.34)

The following notions were covered well:

-The definition of activity (0.59) -Rasmussen's model (0.70) -Errors based on procedural knowledge (0.57)

The following notions were covered very well:

-Applications of Rasmussen's model (0.77) 2: The activity is a set of unobservable behaviors that... 3: There are generally two kinds of procedures in the ...

Figure 1 .

 1 Figure 1. Marked-up text of a course to be processed by Apex

Figure 2 .

 2 Figure 2. Excerpt of the parameters to be modified by the teacher

Figure 3 .Paragraph 1 : 2 : 3 :Paragraph 4 : 5 :

 312345 Figure 3. Example of a content-based assessment

Figure 4 .

 4 Figure 4. Example of an outline-based assessment

Figure 5 .

 5 Figure 5. Example of a coherence-based assessment

  were in the range [.3; .55]. Wolfe et al. [27] indicate correlations around [.6; .7]. E. Kintsch et al. [23] mention a value of .64. The highest correlations ever found [.8;

  When the moon is between the earth and the sun, .... ................... #N Moon eclipses When the earth is between the sun and the moon,.... ..................

	#T The solar system
	#N Introduction to the solar system
	The solar system is composed of....
	................
	#N The sun
	The sun is a star which is....
	................
	#N The planets
	There are 9 planets in the solar system: ....
	.................
	#T Eclipses
	#S Introduction to the solar system
	#N Sun eclipses

Table 1 .

 1 Experiment 2: Correlations of grades between graders and Apex : p < .05; ** : p < .0001

		2	3	Apex
	1	0.83 **	0.72 **	0.38
	2		0.78 **	0.45 *
	3			0.46 *
	Note.

*

ACKNOWLEDGEMENTS

We would like to thank Susan Dumais and the Bellcore Labs to have allowed us to use and hack the code of the basic LSA programs. We are also grateful to Pascal Bressoux who provided us with the text from one of his courses and who helped us in formatting this course for our system, and to Diana Amadis, Ira Noveck, and Erica de Vries for their editorial inputs. Many thanks also to our typists and graders, Severine Parsoud and Dorothée Tourniaire.