Kari-Jouko Räihä
email: kari-jouko.raiha@uta.fi

Life in the Fast Lane: Effect of Language and Calibration Accuracy on the Speed of Text Entry by Gaze

Keywords: entry, gaze input, performance, entry speed, error rate, comparative evaluation, longitudinal study

Numerous techniques have been developed for text entry by gaze, and similarly, a number of evaluations have been carried out to determine the efficiency of the solutions. However, the results of the published experiments are inconclusive, and it is unclear what causes the difference in their findings.

Here we look particularly at the effect of the language used in the experiment. A study where participants entered text both in English and in Finnish does not show an effect of language structure: the entry rates were reasonably close to each other. The role of other explaining factors, such as calibration accuracy and experimental procedure, are discussed.

Introduction

Augmentative and alternative communication techniques are crucial for a considerable share of the world's population. Without such techniques they would, for instance, not be able to use many of the services that are increasingly being offered through the internet. Techniques that have been developed include speech synthesis and Braille printers for the vision impaired, and speech-to-text solutions for the hearing impaired. A particularly challenging user group is those with severe motor-neuron diseases, such as the locked-in syndrome or ALS. In such situations the cognitive abilities are intact, but the ability to control the muscles deteriorates and eventually disappears. The eyes are often the last muscle that works. Techniques that allow communication using just the eyes have been developed and studied extensively [START_REF]Gaze Interaction and Applications of Eye Tracking: Advances in Assistive Technologies[END_REF].

Here we focus on using the eyes for text entry. This, too, is an actively studied field; see [START_REF] Majaranta | Text Entry by Eye Gaze[END_REF][START_REF] Majaranta | Text Entry by Gaze: Utilizing Eye-tracking[END_REF] for reviews. However, as will be discussed in the next section, the results concerning the text entry speed that can be reached by using the eyes vary a lot. We will tackle one possible explanation: the difference in the language used in the experiments, and study it systematically. In addition, we will look at a number of other factors that might explain the results.

Why is it important to know the limits of text entry by gaze? By understanding the subtle differences in the implementation details of a technique we can produce better tools for the people that need the software. Sufficiently fast text entry can, for instance, in the future enable the development of applications where one partner enters text on a mobile phone with the eyes, the phone uses text-to-speech to transmit it to a hearing partner at the other end, who can then respond. To carry out a real conversation the process cannot take too long. Therefore "How fast can one type with the eyes?" is an equally interesting and important research question as "How fast can one type with the typewriter" was for decades.

Previous work

Results from past experiments with text entry by gaze have been collected in Tables 1 (experiments with a soft keyboard) and 2 (experiments with other techniques). We have included only longitudinal experiments where the participants came back to the lab on several days and thus had a chance to improve their performance through experience. The data in the tables is teased out from the publications. For [START_REF] Wobbrock | Longitudinal Evaluation of Discrete Consecutive Gaze Gestures for Text Entry[END_REF] the exact numbers were not reported and are therefore estimated from the graphs in the paper.

The same holds for the MSD rate in [START_REF] Pedrosa | Filteryedping: Design Challenges and User Performance of Dwell-Free Eye Typing[END_REF]. Several papers reported on more than one studies. For [START_REF] Räihä | An Exploratory Study of Eye Typing Fundamentals: Dwell Time, Text Entry Rate, Errors, and Workload[END_REF], there were results for the learning phase (denoted by lp in Table 1) and the advanced phase (denoted by ap). In [START_REF] Rough | An Evaluation of Dasher with a High-Performance Language Model as a Gaze Communication Method[END_REF] there were two experiments, denoted by e1 and e2. Pedrosa et al. [START_REF] Pedrosa | Filteryedping: Design Challenges and User Performance of Dwell-Free Eye Typing[END_REF] carried out a number of experiments; two longitudinal studies with able-bodied participants, followed by individual trials with users with motor disabilities. Here we report the numbers from their second longitudinal study that produced the fastest text entry rates. The WPM column shows the standard metric for measuring text entry speed: words per minute (WPM), where "word" means five letters (including spaces and punctuation). In other words, WPM is the number of characters entered in a minute divided by 5. MSD indicates uncorrected errors, i.e. errors that remain in the text that was entered. It is computed as the mean string distance of the entered phrase and the model phrase [START_REF] Wobbrock | Measures of Text Entry Performance[END_REF]. 1 and2 consisted of several sessions, the numbers reported indicate the averages of all participants for the best session in the series (typically the last one). Individual participants could reach better performance in some sessions. The best reported single session text entry rate was 23.10 WPM for soft keyboards [START_REF] Räihä | An Exploratory Study of Eye Typing Fundamentals: Dwell Time, Text Entry Rate, Errors, and Workload[END_REF] and 23.11 WPM for Dasher [START_REF] Tuisku | Now Dasher! Dash Away! Longitudinal Study of Fast Text Entry by Eye Gaze[END_REF].

It is understandable that for different interaction techniques there is high variation in the performance, but it is surprising that for the same basic technique (soft keyboards in Table 1 and Dasher in Table 2) the results vary so much. There are a number of factors that are not shown in the tables and that do affect the efficiency; we will discuss them extensively later. However, one aspect that stands out is that all the fastest results ( [START_REF] Majaranta | Fast Gaze Typing with an Adjustable Dwell Time[END_REF][START_REF] Räihä | An Exploratory Study of Eye Typing Fundamentals: Dwell Time, Text Entry Rate, Errors, and Workload[END_REF][START_REF] Tuisku | Now Dasher! Dash Away! Longitudinal Study of Fast Text Entry by Eye Gaze[END_REF]) were achieved with Finnish as the language of the phrases that were entered. In all the other studies the language was probably English (the language is not reported in [START_REF] Urbina | Alternatives to Single Character Entry and Dwell Time Selection on Eye Typing[END_REF] and it could also be German).

Could it be that language has such a drastic effect on the text entry speed? We cannot really say based on the results quoted above, because, e.g., different participants took part in each experiment. To shed light on this issue, we carried out an experiment where the same participants entered text using gaze both in English and in Finnish. Longitudinal studies are laborious and we were only able to do the experiment for soft keyboards, but there the difference in past results is anyhow the largest.

Method

We followed a similar experimental design as has become the norm (with small variations) in studies of text entry by gaze. The details are presented next.

Participants

Eight participants (5 male, 3 female) were recruited for the experiment. Seven of them had participated in a previous experiment where eye tracking was used passively just to follow participants' eye movements while viewing a set of images. None had previously used gaze as an input technique. Two participants wore eyeglasses. Selection of participants was based on their demonstrated ability to be tracked by the eye tracker, and on their self-reported knowledge of written English. Six participants reported their skill level as excellent ("mistakes in grammar and vocabulary are rare") and two reported it as good ("I can communicate but I make mistakes"). All had Finnish as their mother tongue. Each participant was paid €15 for each one hour test session. In addition, a bonus of €70 was paid both to the participant that reached the highest text entry rate, and to the participant that showed the biggest improvement in text entry rate over the sessions.

Apparatus

A Tobii T60 eye tracker with a 17-inch TFT color monitor with 1280  1024 resolution was used to track the gaze. A PC running Windows XP was used for the experiment. The software for text entry was an in-house application based on a soft keyboard. The application window filled the screen, as shown in Fig. 1. The top frame of the window is used to display the model phrase that the participant must enter. The text entered is shown in the next frame in red. These are later referred to as the source and target frames, respectively. The layout of the keys follows the layout on a standard Scandinavian qwerty keyboard. The letters on the keys are displayed in 18 pt Arial bold. There is no shift key, since the phrases that had to be entered did not have upper case letters. Similarly, the punctuation marks are missing for the same reason. On the row below there are five space keys and two backspace keys. The motivation for multiple keys with the same function was to provide a short path to these frequently used keys, as suggested in [START_REF] Räihä | An Exploratory Study of Eye Typing Fundamentals: Dwell Time, Text Entry Rate, Errors, and Workload[END_REF].

When a key is gazed at, a red curve starts to grow on the key, as shown for the letter "m" in Fig. 1. Once the curve has made a full circle, the letter gets selected: a sharp clicking sound is played and the letter appears in the target frame.

The bottom left and right corners hold "ready" keys (smileys) to be selected when the participant has finished entering the model phrase. This causes the next phrase to appear in the source frame and the target frame to be cleared.

The time needed for a key to be selected can be adjusted by the participant at any point during the experiment. This is done using the two gaze-controlled keys on the bottom row; the speedometer between them is an indicator of the selected threshold. Clicking on the "+" key increases speed, i.e., shortens the dwell time. The decrement depends on the current level of the dwell threshold. At 1000 ms (the initial setting), clicking on the "+" lowers the dwell to 900 ms. Moving to faster settings the decrement decreases, so that at 260 ms and below, clicking on "+" shortens the required dwell by 20 ms. The "-" key works with similar increments in the opposite direction.

In addition to the experimental text entry software, the TraQuMe application [START_REF] Akkil | TraQuMe: A Tool for Measuring the Gaze Tracking Quality[END_REF] was used to measure calibration accuracy in the beginning and end of each test session. In our experiment TraQuMe was used in a mode where it displayed five targets (much like in a typical calibration procedure), each for 1.5 s, and collected the gaze coordinates produced by the eye tracker during that time. The average x and y coordinates of the data points and their distance to the center of the target were computed to produce the offset, a measure for the accuracy of the calibration. Similarly, precision was measured by computing the dispersion (the standard deviations) of the gaze points during the 1.5 s span.

Of the two measures, small offset is more important for our eye typing software than small dispersion, because of the way that gaze is mapped to the keys on the keyboard. Since the algorithm is of crucial importance for obtaining good entry rates, we describe it here briefly. First, the expansion algorithm makes objects larger than their visual appearance. Therefore there are actually no gaps between the keys. The mapping algorithm further works with this expanded layout, where the effective size of each key is 100  100 pixels. From a distance of 60 cm the side of each key was thus about 2.5 degrees.

Each key has a time counter that is increased (by the amount of time between two successive gaze points) when a new gaze point hits the key, and decreased (by the same amount of time) otherwise. With each sample only one key increases its counter, all others decrease their counters or keep them at 0. Once a counter exceeds the dwell time, the corresponding key becomes selected, and all counters are reset. The dwell time progress circle is shown on the key that has the greatest counter value at the moment.

Procedure

Each participant first came to the lab for a briefing of the software and the experiment. The functionality of the software was explained and the procedure to be followed in the test sessions was described. The participant then had a chance to try out the software. Each participant first entered in Finnish two phrases with a 1000 ms dwell threshold. They were then instructed to try decrementing the threshold with one or two notches and to enter two phrases in English. They were further offered the chance to experiment with two more phrases, but only two participants made use of that opportunity; in general everyone got the hang of it during the first four phrases.

Finally the experimenter stepped in the user's role and showed with two phrases how text entry looks like when the dwell threshold is in the order of 300 ms. The participants were instructed that the goal of the experiment was to find out how fast they can enter correct text. They were advised to correct errors that they spotted immediately, but not to back up long stretches of already entered text if they noticed an error in the beginning.

The actual sessions took place on consecutive working days. Each participant was requested to carry out the test on at least five and at most eight consecutive days. All eight participants came back for at least six days, meaning that everybody had a weekend in between the sessions at some point. One participant did the experiment for seven days and four participants for eight days.

Each day of the experiment consisted of two 15-minute sessions with a 10-minute break in between. One session was in English and the other in Finnish. The order of the languages was swapped each day. Half of the participants started with Finnish, the other with English in the first day.

In the first session the participants were asked to use the initial dwell threshold setting of 1000 ms for the first phrase, and after that to adjust the threshold at will. At the end of the first day (Sessions 1 and 2), the rules of how to get the bonus were revealed. In particular, the metric to be used (entered words per minute minus error percentage) was explained.

The participants were seated at a distance of about 60 cm from the screen. In each of the two sessions the eye tracker was first calibrated for the participant using a 9point calibration. The calibration procedure was repeated until the result looked adequate in the calibration window of the ETU driver [START_REF] Špakov | iComponent -Device-Independent Platform for Analyzing Eye Movement Data and Developing Eye-Based Applications[END_REF] that was used by the software. Then TraQuMe [START_REF] Akkil | TraQuMe: A Tool for Measuring the Gaze Tracking Quality[END_REF] was used to quantitatively measure the quality of the calibration. The measurement was repeated at the end of the 15-minute session.

Task

The participant's task was to transcribe the model phrases as quickly and accurately as possible. The English phrases came from the 500-phrase collection [START_REF] Mackenzie | Phrase Sets for Evaluating Text Entry Techniques[END_REF] that is cus-tomarily used in such experiments. The upper case letters in the phrases were replaced by lower case letters for this experiment. The Finnish phrases mostly came from the Finnish translation [START_REF] Isokoski | Effect of Foreign Language on Text Transcription Performance: Finns Writing English[END_REF] of the English corpus. However, some English phrases that were lines in song lyrics (e.g., "starlight and dewdrops" from Beautiful Dreamer) did not make sense as direct translations, since the lyrics were not translated literally for the Finnish version of the song. In such cases popular Finnish songs were used instead as a source of phrases.

The text entry application chose the model phrases at random. It logged the text entered and full gaze data. It also kept time, so that when 15 minutes had passed, it allowed the participant to finish entering the current phrase but at the click of the ready key a save dialog would pop up. The experimenter then stepped up to an adjacent display, saved the log and carried out the calibration quality measurement.

Results

Altogether 114 text entry sessions took place. In them a total of 5 338 phrases and 146 607 letters were entered. In addition, in 38 cases the participant unintentionally clicked the ready key while entering the phrase, and these (0.7% of all cases) were removed from the data. Fig. 2 plots the average text entry speed over the sessions. The graph shows the data from all participants over six days (12 sessions) and separately the data from those four participants that continued for the full eight days.

Fig. 2. Text entry rate as a function of session number

For easier comparison with previous research, the entry rates in Fig. 2 and elsewhere in this paper are expressed in WPM without taking into account the errors in the remaining text. The participants were extremely conscientious in correcting the errors they spotted. On average, the error rate (measured as string edit distance, MSD) was 0.39%that is, approximately one incorrect character for every 260 correct charac- ters. The errors decreased from 0.58% in the beginning to 0.28% in the last session. Their effect is so negligible that the speed metric that takes errors into account follows very closely the same pattern as the WPM metric in Fig. 2.

The best average session rate, 25.6 WPM, was achieved by participant P5 in Session 12. He stopped then and did not continue for the final two days. In that session the resulting text had 4 errors, meaning an error rate of 0.06%. His data is shown in Fig. 3 as an example of how experience affected the measures. Fig. 3 also plots the keystrokes per character (KSPC), a common metric to measure how many errors were made and corrected during a session [START_REF] Wobbrock | Measures of Text Entry Performance[END_REF]. P5 was one of the participants who moved quickly to very short dwell times. That backfired in Session 3, where poor calibration created a need to backspace often, as Fig. 3 shows. High KSPC correlates strongly with lower WPM rates.

Fig. 3 indicates that for P5, results for Finnish were somewhat better than for English. Fig. 4 shows for the first six days, and for all participants, the text entry speed separately for Finnish and English. Each day each participant carried out one session in each language.

A two-way ANOVA with language and day as within-subject factors revealed a significant effect of day on WPM (F 7,5 = 17.48, p < .0001). Although the graph for Finnish stays above the one for English for every day, the difference is not big enough to be statistically significant (F 7,1 = 5.25, ns). On average, calibration accuracy remained on the same level throughout the sessions. The average offset at the beginning of sessions was 0.6 degrees (with SD of 0.06 degrees), and the average dispersion was 0.3 degrees (SD 0.05 degrees). The figures are low and as good as can be expected from the eye tracker used in the experiment. However, although the variation between sessions, taken over all participants, was thus low, there was higher variation for individual participants. The WPM rate for each session is plotted in Fig. 5 for each participant against the calibration quality at the beginning of the session. For simplicity, the sum of offset and dispersion was used as the combined quality measure. We also measured the calibration accuracy at the end of each session. However, those measurements are less reliable, since some participants had the tendency to relax at the end of the session and change their posture considerably. Therefore the measurements are not reported here. Another factor that directly affects text entry rate is the dwell time threshold. We recorded it at the start and end of each session. Fig. 6 plots text entry rate against the dwell time threshold at the end of each session; note that a faster or slower pace can have been used at some point during the session. However, with the exception of the first day, the within session adjustments of the threshold were moderate. The time taken to transcribe the model phrases is affected by how easy it is to recall them and their spelling. The number of glances back to the source and target frames at the top of the window are plotted in Fig. 7 for each participant. The first graph shows the overall number for the two frames, whereas the second graph focuses on the glances to the source frame only and shows the numbers for English and Finnish phrases separately.

A two-way ANOVA with language and day as within-subject factors revealed a significant effect of language on glances both to the source (F 7,1 = 15.24, p < .01) and target (F 7,1 = 7.62, p < .05) frames, with English giving rise to more glances than Finnish per phrase (4.0 vs. 2.7 for source, 2.7 vs. 2.2 for target). Day did not have a significant effect, nor was there interaction between language and day.

Discussion

Speed of Text Entry

This study did not provide with convincing evidence of the effect of language on the speed of text entry by gaze one way or the other. The trends in Fig. 4 appear clear on the surface, but the ANOVA analysis did not show a statistically significant difference (although it was fairly close with p = .056).

How much difference can one expect between the participants' mother tongue and another language that they know well? In a similar study that used a regular keyboard instead of eye gaze as the input technique, Isokoski and Linden [START_REF] Isokoski | Effect of Foreign Language on Text Transcription Performance: Finns Writing English[END_REF] found a bigger difference: 49.7 WPM for Finnish vs. 41.8 WPM for English, a difference of about 16%. In this light the difference of the average rates in the participants' best sessions, 22.8 WPM for Finnish and 20.9 WPM for English, i.e. a difference of about 8%, is considerably smaller than expected.

Text entry by gaze is in many ways different from using a keyboard, even if the layout of the keys is similar. With a hard keyboard the typist can use at least two hands, if not all ten fingers. In text entry by gaze the process is strictly sequential, with the same "device" (eyes) used to select in succession all the keys one by one. Then the distance to be covered when moving from one key to the next can come into play. Could it be different for the two languages?

To analyze this, we computed the bigram frequency matrices for both corpora. A bigram is a pair of letters that appear in succession in the text, and the frequency matrix gives the number of occurrences of each bigram in the corpus. A similar matrix was created for the distance between the letters in each bigram on the soft keyboard. Multiplying these matrices gives us a bigram distance matrix weighted with the frequencies, and taking the average over all bigrams then tells us whether we can expect an effect that is caused by the bigram frequencies being different.

Table 3 summarizes the key characteristics of the corpora. There is a difference of 304.4 pixels vs. 324.9 pixels (about 6.5%) in average weighted bigram distance in favor of English, which can be expected to affect the text entry speed. Adding this to the observed speed difference of 8% brings us closer to the findings in [START_REF] Isokoski | Effect of Foreign Language on Text Transcription Performance: Finns Writing English[END_REF]. Another factor that, however, works in the opposite direction, is the proportion of double letters (the same letter appearing twice in succession) in the two languages. Double letters are more frequent in Finnish than in English. To enter the double letters with our text entry software the user can simply keep looking at the letter. After the first of the pair of letters is entered, there is a fixed threshold of 150 ms before the dwell time counter is restarted. This is both quick and convenient, making the entry of double letter bigrams faster than entering the other bigrams: the average time to move the gaze from one character to the next was typically more than 200 ms. The share of double letter bigrams in the corpora is 7.4% for Finnish and 2.3% for English.

Finally, it should be noted that Fig. 2 indicates that the entry speeds were still improving after the 6 th day (12 th session). Our analysis was based on the first six days only to have a sufficient number of participants.

In addition to learning from day to day, one may ask if there was difference between the two daily sessions. Indeed, the overall grand mean of text entry speed was slightly higher for the second daily session (18.5 WPM vs. 17.4 WPM). However, the difference was not statistically significant (F 7,1 = 5.39, ns).

In summary, there are factors that have an adverse influence on the expected text entry speed: the mother tongue (other than English) puts English to a disadvantaged position, but the characteristics of the language can have features that affect the entry speed in both directions.

Errors

The extremely low level of errors in the text entered, 0.28% in the last sessions, may come as a surprise. It is largely explained by the feel of text entry with a low dwell threshold. Typing then becomes essentially an activity of moving the gaze from one letter to the next, with a very short time for waiting for the letter to get selected. The typist hears the click and realizes if the gaze was on the intended letter or not. If not, it feels natural to click on the backspace and correct the error immediately. In fact, a large portion of the remaining errors are not caused by entering a letter that the typist did not intend, but rather by recalling the model phrase incorrectly. With a low dwell threshold it becomes easier to get into a regular typing rhythm. The importance of rhythm has been noted before [START_REF] Majaranta | Text Entry by Eye Gaze[END_REF]. In our study one participant (P8) voluntarily commented on this; even if correcting the errors is an almost automatic activity, it still interferes with the typing rhythm, which she considered crucial for fast text entry.

Compared to typing with a conventional keyboard, the sequential nature of text entry by gaze has a positive effect on the level of errors. The two hands can be in action at the same time and may arrive at their target keys in the wrong order without the typist noticing this before looking at the text entered. With gaze such a competition situation is not possible and the typist can trust the order of letters to be correct.

This becomes evident also in analyzing the glances to the source and target frames. Fig. 7 (left) shows that most participants looked more at the source frame than the target frame. The two participants for which this does not hold (P1 and P7) had to struggle in many sessions with calibration that deteriorated during some sessions. The number of glances is particularly low for the overall fastest typist, P5. The formula for fast text entry speed seems to be a combination of a suitably small dwell threshold combined with trust in the correct letters getting entered, and thus no need to check the text frames frequently.

Trust is, of course, affected by calibration accuracy and thereby the software behaving "as expected". Nevertheless, Fig. 5 indicates that people can learn to adjust their gaze position to compensate for poorer calibrations as well. Participant P7, in particular, who was often struggling with a lower quality calibration, still put up respectable entry speed numbers, and reached even the second best result of all in spite of the calibration accuracy in that session being as poor as 1.2 degrees.

Getting back to the glances, Fig. 7 (right) shows that the source phrase is glanced at more often for English than for Finnish. This is understandable for people who don't have English as first language. In addition to some rare words with difficult spelling that might give problems even to native speakers (e.g., "conscience", "exasperation", "dewdrop"), the problem is compounded by the different spellings in dialects of English (e.g., "favourite" vs. "favorite"). No such problems exist with Finnish, which is written as pronounced. It is understandable that participants rechecked the source phrase in such cases to make sure that they got the spelling right.

In summary, the participants took many means to ensure the correctness of their entered text: correcting errors immediately and checking the spelling of words when in doubt. Excluding from the analysis errors that were the result of memory lapses would have further improved the correctness rate substantially.

Why Are the Findings Across Studies So Different?

If the language of the phrases does not explain the difference in the results obtained in different experiments, what does? Perhaps most importantly, even if the basic techniques are the same, they may differ significantly in important implementation details. This holds for the experiments with Dasher [START_REF] Rough | An Evaluation of Dasher with a High-Performance Language Model as a Gaze Communication Method[END_REF][START_REF] Tuisku | Now Dasher! Dash Away! Longitudinal Study of Fast Text Entry by Eye Gaze[END_REF], and the same is true for soft keyboards. For instance, feedback modes have a significant effect on text entry speed [START_REF] Majaranta | Effects of Feedback and Dwell Time on Eye Typing Speed and Accuracy[END_REF]. The dwell time threshold similarly affects both the efficiency and the experienced workload [START_REF] Räihä | An Exploratory Study of Eye Typing Fundamentals: Dwell Time, Text Entry Rate, Errors, and Workload[END_REF].

The fact that the user is in control of adjusting the dwell time threshold during a session, as pioneered in [START_REF] Majaranta | Fast Gaze Typing with an Adjustable Dwell Time[END_REF], is of crucial importance. In [START_REF] Rough | An Evaluation of Dasher with a High-Performance Language Model as a Gaze Communication Method[END_REF], the dwell time was also adjustable, but the paper does not give details on how this was done. Judging from the screenshot of the interface in [START_REF] Rough | An Evaluation of Dasher with a High-Performance Language Model as a Gaze Communication Method[END_REF], this was only possible between the sessions. It is understandable that users are not eager to jump to big decrements in the dwell threshold if they are unsure of how it affects them. When they have the tool to increase or decrease the threshold at will, they can experiment with shorter dwell times to see if it works for them.

The briefing given at the beginning of the experiment is also important. We showed ourselves what text entry with a short dwell time threshold looks like before the participants were engaged in the first session. This presumably caused them to move to short thresholds much faster than in any previous experiments. As Fig. 6 shows, a clear majority (71%) of the sessions ended with the threshold being in the 200-300 ms range. In 8% of the sessions it was below 200 ms and in 21% of the sessions above 300 ms. Only one participant used higher than 300 ms thresholds after the third day of the experiment.

It should be emphasized again that a short threshold not only enables fast text entry, but it gives a very different feeling to the user. Text entry changes from interacting with icons that represent keys to selecting letters that one wants to enter. In other words, interacting with the interface as an artefact fades to the background and the user can focus on the task at hand.

One obvious factor affecting the results is the corpus used [START_REF] Sanchis-Trilles | A Systematic Comparison of 3 Phrase Sampling Methods for Text Entry Experiments in 10 Languages[END_REF]. Especially in our case, when we decided to use only lower case letters to limit any confounding factors, it can be expected that the results are better than in previous studies. Comparing our best single session average (25.61 WPM) and overall last session average (20.36 for the 12 th session with all participants, 22.66 WPM for the 16 th session with final four participants) to the results quoted in Section 2 (23.10 WPM and 19.89 WPM, respectively), we see that this is indeed the case.

The apparatus used in the experiments is another significant factor. Most experiments reported in the literature have been carried out with Tobii eye trackers. Trackers with a higher sampling rate would provide an interesting comparison. Importantly, the algorithm for mapping gaze points to keys of the keyboard plays a central role. We suspect that the algorithm described in Section 3.2 contributes greatly to achieving high text entry rates, and it may explain much of the differences in the results of other soft keyboard experiments. This, of course, requires further experimentation.

Finally, the choice of participants and their motivation are also important factors. In most studies cited here the participants were compensated for their participation and often rewarded for best performance, so this should not be a major cause of variation. Our participants were native speakers of Finnish and fluent with English. The participants in [START_REF] Rough | An Evaluation of Dasher with a High-Performance Language Model as a Gaze Communication Method[END_REF] were also fluent in English, but the paper does not tell if this means that they were natives. Participants having English as their mother tongue can be expected to get even better results than those in our study, even if the spelling controversies discussed in Section 5.2 still remain to some extent.

Conclusion

We have carried out a controlled experiment where the participants entered text using a soft keyboard with their gaze from three to four hours (12 to 16 sessions of 15 minutes each), half of it in English and the other half in Finnish. No statistically significant difference was found between the languages. For reasons discussed in Section 5.3, our participants reached somewhat higher entry rates than has been reported in the literature before. In particular, the speed of text entry in English (best single session rate of 22.31 WPM, and overall average rate of 19.95 WPM for the sixth day and 21.07 WPM for the eighth day) is much higher than has been reported before.

We don't intend to claim the superiority of one technique over the other. Instead, we want to underline the importance of the many factors that affect the results in such experiments. To compare two designs with each other it is important to control as many of the experimental variables as possible: same participants, same briefing, same goal setting, same compensation, comparable calibration quality and same tracker. Without control over such factors the results are not fully commensurate.

Finally, it should be noted that our experiment (as most similar experiments) was carried out with able bodied users. The users that really need the technology of text entry by gaze can develop into very fast eye typists, as evidenced e.g. by some videos on the Internet for English1 and Finnish2 . To trace the world record speed in text entry by gaze, the experiment should involve such users who use the technique not for hours, but for days, weeks and months.

Fig. 1 .

 1 Fig. 1. A screenshot of the application used in the experiment. The QWERTY keyboard shows the Scandinavian layout used

Fig. 3 .

 3 Fig. 3. Development of text entry rate (as bars) and keystrokes per character (as line graph) for participant P5. Blue bars denote sessions with Finnish, striped red bars with English.

Fig. 4 .

 4 Fig. 4. Text entry rates for Finnish and English as a function of day of experiment

Fig. 5 .

 5 Fig. 5. WPM vs. quality of calibration at start of session for each participant

Fig. 6 .

 6 Fig. 6. WPM vs. dwell time threshold at end of session for each participant

Fig. 7 .

 7 Fig. 7. On the left the average number of glances to the source and target frames per phrase and participant. On the right the average number of glances to the source frame for the English and Finnish phrases.

Table 1 .

 1 Data from experiments on text entry by gaze with soft keyboards

	Refe-	Tracker	Partici-	Session	Sessions	Dwell	WPM	MSD
	rence		pants	length		threshold
	[5]	Tobii 1750	11	15 min	10	adjustable 19.89	0.36%
	[17]	Tobii 1750	8	8 phrases	14	330 ms	> 8 < 0.01%
	[10] lp Tobii T60	10	15 min	10	adjustable 18.45	0.58%
	[10] ap Tobii T60	10	15 min	5 420-250 ms 18.98	0.46%
	[11] e1 Tobii P10	9	10 min	9	1000 ms	6.0	2.20%
	[11] e2 Tobii P10	12	15 min	9	adjustable	7.0	4.00%

Table 2 .

 2 Data from experiments on text entry with novel techniques

	Refe-	Tracker	Partici-	Session	Sessions Technique WPM	MSD
	rence		pants	length
	[17]	Tobii 1750	8 8 phrases	14	EyeWrite	> 6 < 0.01%
	[15]	SR-Research Eyelink2	9 3 sayings	20	pEYEwrite-WoDyn	13.47	0.01%
	[14]	Tobii 1750	12	15 min	10	Dasher 17.26	0.57%
	[11] e1 Tobii P10	9	10 min	9	Dasher 12.4	1.70%
	[11] e2 Tobii P10	12	15 min	9	Dasher 14.2	3.30%
	[9]	Tobii REX	6	20 min	6	Filter-yedping	15.95 < 0.3%
	Since all the experiments included in Tables

Table 3 .

 3 Characteristics of the corpora.

	Finnish	English

http://www.eyegaze.com/eye-tracking-assistive-technology-device/

http://www.iltalehti.fi/iltv-doc/201411070115600_dc.shtml (text entry e.g. at 01:00 and 03:48)

http://www.yorku.ca/mack/RN-Anova.html

Acknowledgments

I am grateful to Per Ola Kristensson for suggesting this experiment and to Päivi Majaranta, Poika Isokoski, Saila Ovaska and Oleg Špakov for discussions and comments on the manuscript. In particular, Oleg Špakov implemented the soft keyboard and the gaze tracking algorithm used in the experiment. The statistical analysis was done using I. Scott MacKenzie's Java tools 3 . Finally, the participants in the experiment were committed and responsiblethank you very much! This work was supported by the Academy of Finland (project MIPI).


Imran Sehikh
email: imran.as@tcs.com

Dominique Fohr
email: dominique.fohr@loria.fr

Irina Illina
email: irina.illina@loria.fr

TOPIC SEGMENTATION IN ASR TRANSCRIPTS USING BIDIRECTIONAL RNNS FOR CHANGE DETECTION

Keywords:

Topic segmentation methods are mostly based on the idea of lexical cohesion, in which lexical distributions are analysed across the document and segment boundaries are marked in areas of low cohesion. We propose a novel approach for topic segmentation in speech recognition transcripts by measuring lexical cohesion using bidirectional Recurrent Neural Networks (RNN). The bidirectional RNNs capture context in the past and the following set of words. The past and following contexts are compared to perform topic change detection. In contrast to existing works based on sequence and discriminative models for topic segmentation, our approach does not use a segmented corpus nor (pseudo) topic labels for training. Our model is trained using news articles obtained from the internet. Evaluation on ASR transcripts of French TV broadcast news programs demonstrates the effectiveness of our proposed approach.

INTRODUCTION

The problem of topic segmentation, to automatically breakdown a text document into topically coherent segments, has been studied for a long time. With the increase in multimedia content on the internet, there has been an interest to extend topic segmentation to audio-video documents. Multimedia documents like broadcast news programs, meeting recordings, telephone conversations and lectures commonly consist of information on more than one topic. For example, broadcast news present events related to politics, economy, sports, weather and so on. Automatic segmentation of such documents, into coherent segments, is required by several down stream tasks such as topic detection and tracking [START_REF] Allan | Topic Detection and Tracking: Event-Based Information Organization[END_REF], summarisation, named entity extraction and for multimedia indexing and organisation [START_REF] Shan | Spoken document understanding and organization[END_REF].

Approaches to topic segmentation are based on the idea of lexical cohesion [START_REF] Purver | Spoken Language Understanding, chapter Topic Segmentation[END_REF]. Some of these methods analyse the lexical distribution across the document and mark segment boundaries in areas of low cohesion. This includes the original TextTiling algorithm [START_REF] Hearst | Texttiling: Segmenting text into multi-paragraph subtopic passages[END_REF] and its extensions using lexical chains [START_REF] Galley | Discourse segmentation of multiparty conversation[END_REF], semantic/topic space representations [START_REF] Olney | An orthonormal basis for topic segmentation in tutorial dialogue[END_REF][START_REF] Riedl | Text segmentation with topic models[END_REF] and Laplacian Eigenmaps [START_REF] Xie | Laplacian eigenmaps for automatic story segmentation of broadcast news[END_REF]. Another set of segmentation methods try to cluster together neighbouring areas instead of directly looking for topic boundaries. This includes the prominent C99 algorithm [START_REF] Freddy | Advances in domain independent linear text segmentation[END_REF] and its extensions using semantic/topic representations [START_REF] Freddy | Latent semantic analysis for text segmentation[END_REF][START_REF] Bestgen | Improving text segmentation using latent semantic analysis: A reanalysis of choi, wiemerhastings, and moore[END_REF][START_REF] Riedl | Text segmentation with topic models[END_REF]. Alternative methods based on generative probabilistic models have also been proposed for topic segmentation. These include extensions of classical probabilistic topic models to incorporate topic changes and boundaries [START_REF] Blei | Topic segmentation with an aspect hidden markov model[END_REF][START_REF] Purver | Unsupervised topic modelling for multi-party spoken discourse[END_REF][START_REF] Du | Topic segmentation with a structured topic model[END_REF][START_REF] Du | Topic segmentation with an ordering-based topic model[END_REF], and approaches which directly model words in each topic segment as draws from a corresponding multinomial language model [START_REF] Yamron | A hidden markov model approach to text segmentation and event tracking[END_REF][START_REF] Utiyama | A statistical model for domain-independent text segmentation[END_REF][START_REF] Eisenstein | Bayesian unsupervised topic segmentation[END_REF].

Most topic segmentation approaches were originally tried on textual resources or manual transcriptions of spoken resources. Unlike on text documents, topic segmentation on transcriptions obtained from an Automatic Speech Recognition (ASR) system cannot readily exploit sentence boundaries. In this regard, some works have used a fixed block of words as an unit [START_REF] Xie | Laplacian eigenmaps for automatic story segmentation of broadcast news[END_REF][START_REF] Xie | Multi-scale texttiling for automatic story segmentation in chinese broadcast news[END_REF] while others have relied on pauses in speech [START_REF] Bouchekif | Diachronic semantic cohesion for topic segmentation of tv broadcast news[END_REF]. Interestingly, prosodic cues and automatic sentence segmentation techniques have been leveraged in other works [START_REF] Tür | Integrating prosodic and lexical cues for automatic topic segmentation[END_REF][START_REF] Levow | Prosody-based topic segmentation for mandarin broadcast news[END_REF][START_REF] Rosenberg | Varying input segmentation for story boundary detection in english, arabic and mandarin broadcast news[END_REF]. Performance of topic segmentation on ASR transcripts is also affected by word errors from the ASR. To reduce the effect of word errors, the use of ASR confidence measures and lattices have been proposed [START_REF] Guinaudeau | Enhancing lexical cohesion measure with confidence measures, semantic relations and language model interpolation for multimedia spoken content topic segmentation[END_REF][START_REF] Mohri | Discriminative topic segmentation of text and speech[END_REF]. Discriminative features from the speech signal, speaker patterns and news structure have shown to improve performance on spoken documents [START_REF] Purver | Spoken Language Understanding, chapter Topic Segmentation[END_REF][START_REF] Galley | Discourse segmentation of multiparty conversation[END_REF][START_REF] Bouchekif | Diachronic semantic cohesion for topic segmentation of tv broadcast news[END_REF].

In this paper, we focus on lexical cohesion based topic segmentation on the ASR 1-best hypothesis. In contrast to the previous works, we propose a novel approach based on Recurrent Neural Networks (RNN). Recently RNNs have been shown to effectively model sequences like text, speech, music and videos, and have given state-of-the-art results in several sequence and temporal classification tasks [START_REF] Lipton | A Critical Review of Recurrent Neural Networks for Sequence Learning[END_REF]. Their ability to model long term context and the potential to train them discriminatively motivates us to try RNNs for the task of topic segmentation. More specifically, we aim to capture lexical cohesion using a bi-directional RNN with Long Short-Term Memory (LSTM) cells [START_REF] Gers | Learning to forget: Continual prediction with lstm[END_REF][START_REF] Lipton | A Critical Review of Recurrent Neural Networks for Sequence Learning[END_REF]. The bi-directional model would capture context from the past and the following set of words. These past and following contexts can then be compared to perform topic change detection.

Previous works on topic segmentation have adopted sequence modelling approaches based on Hidden Markov Models (HMM) [START_REF] Blei | Topic segmentation with an aspect hidden markov model[END_REF][START_REF] Yamron | A hidden markov model approach to text segmentation and event tracking[END_REF][START_REF] Tür | Integrating prosodic and lexical cues for automatic topic segmentation[END_REF][START_REF] Yu | A dnn-hmm approach to story segmentation[END_REF], and also discriminatively trained models including Conditional Random Fields (CRF) [START_REF] Wang | Broadcast news story segmentation using conditional random fields and multimodal features[END_REF], deep feed forward neural network with HMM [START_REF] Yu | A dnn-hmm approach to story segmentation[END_REF] and Support Vector Machines with sliding windows [START_REF] Mohri | Discriminative topic segmentation of text and speech[END_REF]. However, RNNs are better than fixed size windows and HMMs at exploiting contextual information [START_REF] Lipton | A Critical Review of Recurrent Neural Networks for Sequence Learning[END_REF]. Moreover, these works required a segmented training corpus and used (pseudo) topic labels of these segments for training their models (although pseudo labels were obtained in an unsupervised manner). As opposed to this, our proposed model is trained using news articles obtained from the internet.

The main contributions of this paper are (a) bi-directional RNNs for topic segmentation in ASR transcripts, which eliminate the need for an initial segmentation based on heuristics like fixed size windows or silence, (b) methods for discriminative training of bi-directional RNNs for topic segmentation without the need of a segmented training corpus and/or (pseudo) topic labels. We consider the application of segmentation of ASR transcripts of French broadcast news. The proposed model is evaluated on concatenated broadcast news videos as well as on real television news programs, and is compared to classical methods. The rest of the paper is organised as follows. Section 2 presents the idea behind our model and its architecture. It further includes a discussion on how our model is trained. Section 3 presents our experiment setup including the corpora, model configurations and evaluation methodology. This is followed by the segmentation results and a discussion in Section 3.4 and the conclusion in Section 4.

TOPIC CHANGE DETECTION WITH BI-DIRECTIONAL RNN

Topic segmentation can also be seen as a problem of topic change detection. A typical approach to mark a change point in a sequence is to take a window (of points) on either sides of a supposed change point and compare the adjacent windows using suitable features and representations. The comparison of adjacent windows gives a measure of whether the two windows belong to same the class or not. Following such a computation for all the points in the sequence, a similarity graph is obtained in which most of the peaks/valleys corre-spond to actual change points in the sequence. TextTiling [START_REF] Hearst | Texttiling: Segmenting text into multi-paragraph subtopic passages[END_REF] based algorithms for topic segmentation closely resemble this methodology.

Our proposed topic segmentation model is also based on a similar idea. The input is a sequence of words hypothesised by an ASR system and topic segmentation is to be performed on this 1-best ASR hypothesis. Words in the ASR hypothesis are represented using word embeddings. Instead of using fixed windows of words, RNNs are used to model the long term topic context and to perform change detection. More specifically we choose a bi-directional RNN [START_REF] Schuster | Bidirectional recurrent neural networks[END_REF] with LSTM cells. The bi-directional RNN allows to model context from the past and the following set of words, and it could be trained to measure topic cohesion between these contexts.

Model architecture

Our model consists of a layer of bi-directional RNN with LSTM cells. To understand the functioning of our model, a bi-directional LSTM-RNN is depicted in Figure 1.

x t-1 x t x t+1 h F t-1 h B t-1 h F t h B t h F t+1 h B t+1 f F LSTM () f F LSTM () f F LSTM () f B LSTM () f B LSTM () f B LSTM () Fig. 1. Bi-directional RNN with LSTM cells
The inputs to our model are word embeddings [START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF] corresponding to words in the ASR 1-best hypothesis. These embeddings are denoted as (x 1 , ...x t-1 , x t , x t+1 ..., x N ), where x t represents the current word and N is the length of the ASR hypothesis. Each word embedding is input to the forward and backward LSTM-RNNs, represented in Figure 1 by a chain of blocks of f F LSTM () and f B LSTM (). The hidden layer activations for the forward LSTM-RNN at time t are denoted as h F t and those for the backward LSTM-RNN are denoted as h B t . Thus,

h F t = f F LSTM (x t , h F t-1 ) (1)
h B t = f B LSTM (x t , h B t+1 ) (2)
where, f F LSTM () and f B LSTM () represent the operations inside an LSTM cell with forget gates (refer [START_REF] Gers | Learning to forget: Continual prediction with lstm[END_REF][START_REF] Lipton | A Critical Review of Recurrent Neural Networks for Sequence Learning[END_REF] for details). Hidden layer activations of the forward and backward LSTM-RNNs are transformed using a fully connected feed forward layer as follows:

c F t = h F t .W seg + b seg (3) c B t = h B t .W seg + b seg (4)
where W seg and b seg are weight and bias parameters of the feed forward layer. Then the outputs corresponding to forward and backward LSTM-RNN at each t are compared as:

s t = g(c F t .c B t ) (5)
where . denotes a dot product. This dot product compares the similarity between the topic context until t, as captured in c F t , and the topic context following t, which is captured in c B t . Thus {s t } t=1:N represents the topic context similarity across the ASR hypothesis and this similarity should be minimum at the segment boundaries. The similarity calculation also involves an output function g(). We present two different possibilities for the output function g(), in Section 2.3.

Training without topic labels

Approaches based on sequence models [START_REF] Blei | Topic segmentation with an aspect hidden markov model[END_REF][START_REF] Yamron | A hidden markov model approach to text segmentation and event tracking[END_REF][START_REF] Tür | Integrating prosodic and lexical cues for automatic topic segmentation[END_REF] and discriminatively trained models [START_REF] Mohri | Discriminative topic segmentation of text and speech[END_REF][START_REF] Yu | A dnn-hmm approach to story segmentation[END_REF] require a topic segmented corpus, as well as (pseudo) topic labels for these segments, during training. These previous works have tackled this issue by automatically generating topic labels for segments of text in the training corpus. To obtain topic labels, unsupervised clustering is performed on the text segments and each resulting cluster is marked with a different topic. A similar technique can be employed for training our model with bi-directional RNNs. However, in this work we present an alternative approach which does not rely on a segmented corpus and topic labels.

Our training set consists of news articles crawled from a news website, as detailed in Section 3.1. Each article contains text around a particular news event. To train our model, two or more news articles are randomly chosen and concatenated. Then the training objective is to mark the boundary between the concatenated articles as a topic change point. The assumption is that the training set has a significantly large number of news articles and from a wide time period (ranging over months). With such a training set, it is very less likely for a generated training sample to contain two or more news segments which are on the same news event and adjacent to each other. Even otherwise an unsupervised topic model, like Latent Dirichlet Allocation (LDA) [START_REF] David M Blei | Latent dirichlet allocation[END_REF], can be built on the training set and topic distributions of adjacent segments can be compared to avoid concatenation of articles on same news events. However, a detailed analysis on selection of training set samples is not in the scope of this paper and we form our training set using news articles spread over a period of 6 months.

Output layer and training criteria

As discussed in Section 2.2, the training objective is to mark the boundary between the concatenated training set samples. Recall that in the last layer of our model, in [START_REF] Galley | Discourse segmentation of multiparty conversation[END_REF], the similarity value should be minimum at the boundary. Accordingly, we present two possibilities for the choice of the output function g(), which can achieve this training objective.

Softmin Function

g(d t * ) = e -dt * N t=1 e -dt (6)
The softmin function will give a high output probability to a low dot product similarity value (

d t = c F t .c B t ).
For training with softmin, a cross-entropy cost function can be used. The cost is calculated as:

L = -log(s t ) ( 7
)
where t is the position of the topic change in input text. The softmin function followed by the cross-entropy cost will bring a discriminative capability to the segmentation model by maximising the likelihood of the true segmentation point (t * , known at the time of training), as compared to all the points (t = 1, 2, ...N ). During test the high output values will correspond to the hypothesised topic change points.

Flipped sigmoid function

g(d t ) = 1 -sigmoid(d t ) = 1 1 + e dt (8)
This function assigns a value close to zero to a low dot product similarity value (d t = c F t .c B t ) and a value close to 1 to a high dot product similarity. During training a binary cross-entropy cost function will be used, which is given as:

L = - t=1:N t =t log(s t ) - t log(1 -s t ) (9)
where t are positions of topic change in the input text. During test the output value at each t is regraded as the probability of change point at the corresponding word in the input text. This output function, unlike the softmin function, is (a) independent of the length (N ) of the input text and (b) allows training using input samples with more than one change point (for e.g. by concatenating more than two articles during training).

It should be noted that a softmax function and a standard sigmoid function can be used in place of the softmin and flipped sigmoid functions. However, our choice is motivated by the fact that dot product values in (5) represent similarity values and should be minimum at the topic change point. Results from our initial experiments, using different output functions, also supported this argument.

EXPERIMENTS SETUP

Experiment corpora

We use two set of corpora in our experiments. The first set consists of 24,000 news articles from French newspaper L'Express, and about 3000 news articles and 3000 news videos from the French website of the Euronews TV channel. These news articles and videos appeared during the period January -June 2014. More details about these datasets can be found in [START_REF] Sheikh | How diachronic text corpora affect context based retrieval of oov proper names for audio news[END_REF]. Each article/video in this dataset contains news on a particular event.

For training our model we concatenate randomly chosen articles from the L'Express dataset. Our validation set consists of 3000 samples formed by concatenating 2 articles at a time from Euronews text articles. Our first test set, referred as the Euronews Test Set, consists of 3000 samples with each sample obtained by concatenating 2 to 7 videos from Euronews. (News videos from Euronews are 2 to 5 min in duration.) For training and test, the punctuation marks are removed and all words are converted to lower case, as it would be with ASR transcripts.

Apart from the Euronews Test Set, we evaluate our models on a real test set consisting of 20 news programs that appeared on the French TV channel TV5 during February 2017. This test set is referred as the TV5 Test Set. On average a program is 12 minutes long and consists of 3 to 10 segments.

Model configurations and training

The model with the softmin function in the output layer, as discussed in Section 2.3.1, will be denoted as 'RNN-SMIN'. It is trained using samples obtained by concatenating 2 articles at a time from the L'Express corpus. Model with the flipped sigmoid function in the output layer, as discussed in Section 2.3.2, is trained with samples obtained by concatenating 2 to 4 articles at a time from the L'Express corpus. This model configuration will be denoted as 'RNN-FSIG'.

As inputs to the RNN-SMIN and RNN-FSIG, model we use pre-trained 200 dimensional Skip-gram word embeddings1 , which were trained on '.fr' domain websites. These word embeddings were not updated during training so that the model generalises to unseen words in the test set. In addition, we applied 50% dropout at the output of the word embedding layer to achieve generalisation and avoid overfitting. The word dropout also adds robustness to ASR errors, as we have shown earlier in [START_REF] Sheikh | Modelling semantic context of oov words in large vocabulary continuous speech recognition[END_REF]. For model training, we used mini-batch stochastic gradient descent with ADADELTA [START_REF] Zeiler | ADADELTA: An Adaptive Learning Rate Method[END_REF]. An early stopping criteria was used, which stops the model training when the error on the validation set starts increasing.

Evaluation setup

The topic segmentation models will be evaluated on the reference and ASR transcriptions of the Euronews Test Set, and on the ASR transcriptions of the TV5 Test Set. The ASR transcriptions are obtained from our French ASR system based on DNN-HMM acoustic models. Our ASR gives a word error rate of 16.4% on the Euronews Test Set.

We compare the performance of our models with two baseline topic segmentation methods. Our first baseline is the TopicTiling algorithm proposed in [START_REF] Riedl | Topictiling: A text segmentation algorithm based on lda[END_REF]. TopicTiling is based on the classical TextTiling algorithm for topic segmentation [START_REF] Hearst | Texttiling: Segmenting text into multi-paragraph subtopic passages[END_REF] and uses topic assignments from a LDA topic model. Our second baseline is the classical C99 algorithm for topic segmentation [START_REF] Freddy | Advances in domain independent linear text segmentation[END_REF], improved with representations learnt using Latent Semantic Analysis (LSA) [START_REF] Freddy | Latent semantic analysis for text segmentation[END_REF][START_REF] Bestgen | Improving text segmentation using latent semantic analysis: A reanalysis of choi, wiemerhastings, and moore[END_REF]. This baseline will be denoted as 'C99-LSA'.

For the C99-LSA and TopicTitling baseline methods, 200 dimensional LSA and LDA models were trained on the same training corpus that was used to train our RNN models. Stopwords were removed while training these baseline models, as well as the proposed models. Additionally lemmatisation was applied in case of baseline models. C99-LSA operates on sentence level representations. Since ASR transcripts do not have punctuations we used fixed blocks of words instead. Similarly, TopicTitling was used with fixed blocks of words as basic units. The block size was tuned using the validation set.

For comparison of the proposed and baseline methods, we use the standard topic segmentation evaluation measures P k and W D [START_REF] Purver | Spoken Language Understanding, chapter Topic Segmentation[END_REF]. They indicate the probability of segmentation error, with a lower value indicating a better performance. As compared to P k , W D penalises false alarms. We also calculate the Precision (P ) and Recall (R) [START_REF] Purver | Spoken Language Understanding, chapter Topic Segmentation[END_REF] on the segmentation results, and report the F 1 score which was calculated as F 1 = 2(P ×R)/(P +R). A window of 25 words on each side of a true segmentation point was used to label a hypothesised segmentation point as true positive or false positive.

Topic segmentation results and discussion

Table 1 presents the topic segmentation results on the Euronews Test Set. Results on both reference and ASR transcriptions are shown. Note that the training corpus, used for the baseline as well as the proposed models, and the Euronews Test Set consist of news from the same time period (see Section 3.1). So it is likely that they contain similar topics and hence it is a matched train-test condition. Results in Table 1 show that the C99-LSA method gives the lowest P k and W D errors. However, our proposed model RNN-FSIG gives the best F 1 score and its P k and W D is quite close to that of C99-LSA. For the Euronews Test Set our RNN-SMIN model performs only better than TopicTitling. Further analysis of the results on this test set revealed that both our proposed models have high precision (about 0.8) as compared to that of C99-LSA (at about 0.65). RNN-SMIN model showed a relatively lower recall.

Topic segmentation results on the TV5 Test Set are presented in Table 2. Only ASR transcription results are shown (due to lack of reliable reference transcriptions). Note that this test set is from a different time period than our training set (see Section 3.1) and hence there is a mis-matched train-test conditions in terms of contents and topics. Results in Table 2 show that our proposed models perform better than the baseline models, which seem to be affected by the mis-matched train-test conditions. On contrary to results on the Euronews Test Set, RNN-SMIN gives the best results and RNN-FSIG has a performance similar to that of RNN-SMIN. To further demonstrate the effectiveness of our proposed approach and to highlight the learning in our models we present patterns (appearing as coloured horizontal lines) on certain dimensions. For example, activations in 2(b) clearly higlight segments 2, 4, 5, 7 and 8 (from left to right). Similarly, projections in Figure 2(c) show activations for segments 5, 6, 7 and 8. These activations and their patterns demonstrate that the LSTM-RNNs have learned about topic coherence and acquired the ability to perform topic segmentation.

CONCLUSIONS

We proposed a novel approach for topic segmentation which measures lexical and topic cohesion using bidirectional Recurrent Neural Networks. The bi-directional RNNs captured context in the past and the following set of words, and performed topic change detection by comparing the past and following contexts. These models were trained discriminatively by concatenating news articles from the internet. Evaluation on ASR transcripts of French TV news programs showed that our RNN models can perform better than the C99-LSA and TopicTiling baseline methods. Our models achieved best F 1 scores throughout, due to higher precision rates, and performed well in mis-matched conditions where the baselines started lacking. With the use of standard word embeddings our model can be readily adapted to topic segmentation in different domains.

Figure 2 .Fig. 2 .

 22 Fig. 2. Visualisations of hidden layer activations in the RNN-FSIG model for the ASR transcript of a news program from TV5 French TV channel. (a) input word embeddings (b) LSTM output activations from the forward RNN (c) LSTM output activations from the backward RNN.

Table 1 .

 1 Topic segmentation error (P k , W D) and F 1 score on the Euronews Test Set (with matched train-test conditions). Best results are highlighted in bold.

		Reference	ASR
	P k	W D F 1	P

k W D F 1 TopicTitling 0.44 0.49 0.55 0.45 0.49 0.54 C99-LSA 0.21 0.25 0.75 0.22 0.26 0.73 RNN-SMIN 0.24 0.32 0.69 0.25 0.32 0.66 RNN-FSIG 0.21 0.26 0.79 0.24 0.28 0.78

Table 2 .

 2 Topic segmentation error (P

		P k	WD	F 1
	TopicTitling 0.38	0.45	0.53
	C99-LSA	0.29	0.35	0.60
	RNN-SMIN 0.26	0.34	0.66
	RNN-FSIG	0.26	0.34	0.64

k , W D) and F 1 score on ASR transcripts of the TV5 Test Set (with mis-matched train-test conditions). Best results are highlighted in bold.

obtained from http://fauconnier.github.io

ACKNOWLEDGEMENTS

This work is funded by the ContNomina project supported by the French National Research Agency (ANR) under contract ANR-12-BS02-0009. Experiments presented in this paper were carried out using the Grid'5000 testbed, supported by a scientific interest group hosted by Inria, CNRS, RENATER and other Universities and organisations (https://www.grid5000.fr).