Knechtlee Beat

Christoph Alexander Rüst

Thomas Rosemann

Romuald Lepers

Beat Knechtle
email: beat.knechtle@hispeed.ch

Dr Med Pd

Beat Knechtle

Age-related changes in 100-km ultra-marathon running performance

Keywords:

published or not. The documents may come

Introduction

In recent years there has been an increased interest in investigating the effects of aging on endurance performance in runners [START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF][START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF][START_REF] Lepers | Do older athletes reach limits in their performance during marathon running? Age[END_REF][START_REF] Tanaka | Invited Review: Dynamic exercise performance in masters athletes: Insight into effects of primary human aging on physiological functional capacity[END_REF][START_REF] Tanaka | Endurance exercise performance in masters athletes: Ageassociated changes and underlying physiological mechanisms[END_REF][START_REF] Wright | Age-related rates of decline in performance among elite senior athletes[END_REF]. Over the last decade the participation of master athletes (> 40 years old) in these events has increased, especially in the longer running distances such as half-marathons [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF], marathons [START_REF] Burfoot | The history of the marathon: 1976-present[END_REF][START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF][START_REF] Lepers | Do older athletes reach limits in their performance during marathon running? Age[END_REF][START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF], and ultra-marathons [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF][START_REF] Hoffman | The Western States 100-Mile Endurance Run: participation and performance trends[END_REF][START_REF] Hoffman | Historical analysis of participation in 161 km ultramarathons in North America[END_REF]. A recent analysis of the participation trend at the 'New York City Marathon' from 1980 to 2009 showed that over the three decades, the percent of finishers at the 'New York City Marathon' younger than 40 years decreased, while the percent of master finishers increased for both males and females [START_REF] Lepers | Do older athletes reach limits in their performance during marathon running? Age[END_REF].

Over the three decades 1980-1989, 1990-1999, and 2000-2009, male master athletes represented 36%, 45%, and 53% of total male finishers, respectively, while female master athletes represented 24%, 34%, and 40% of total female finishers, respectively.

Generally, the peak endurance performance is maintained until the age of 30 to 35 years, followed by a moderate decrease until the age of 50 to 60 years, and then a progressively steeper decline after the age of 70 to 75 years, independent of the distance and the sport's discipline [START_REF] Baker | Percentage decline in masters superathlete track and field performance with aging[END_REF][START_REF] Balmer | Indoor 16.1-km time-trial performance in cyclists aged 25-63 years[END_REF][START_REF] Donato | Declines in physiological functional capacity with age: A longitudinal study in peak swimming performance[END_REF][START_REF] Fleg | Role of muscle loss in the age-associated reduction in VO 2 max[END_REF][START_REF] Fuchi | Cardiovascular changes associated with decreased aerobic capacity and aging in long-distance runners[END_REF]Hunter et al. 2011;[START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF][START_REF] Tanaka | Invited Review: Dynamic exercise performance in masters athletes: Insight into effects of primary human aging on physiological functional capacity[END_REF][START_REF] Tanaka | Endurance exercise performance in masters athletes: Ageassociated changes and underlying physiological mechanisms[END_REF][START_REF] Wright | Age-related rates of decline in performance among elite senior athletes[END_REF]. Even if it is possible for a 70-yearold male athlete to finish a marathon in less than three hours [START_REF] Trappe | Marathon runners: How do they age?[END_REF], the gradient in declining performances increased notably after the age of 55 years for both sexes, and female performances tended to decline faster than those of males, especially in running [START_REF] Ransdell | Master athletes: An analysis of running, swimming and cycling performance by age and gender[END_REF].

Most studies examining endurance running performances in master athletes have focused on marathoners [START_REF] Burfoot | The history of the marathon: 1976-present[END_REF][START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF][START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF][START_REF] Trappe | Marathon runners: How do they age?[END_REF].

Marathon running performance for males and females is generally fastest, as indicated by world record performances, when individuals are 25 to 35 years old. A significant age-related decrease in marathon performance begins at the age of 35 years [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Wright | Age-related rates of decline in performance among elite senior athletes[END_REF]. The time taken to complete a marathon gradually increases with age, with a substantial decrease in performance after the age of 70 years [START_REF] Trappe | Marathon runners: How do they age?[END_REF]. In recreational marathoners there was no relevant time difference, in running, between the ages of 20 and 55 years, however, after the age of 55 years mean running times significantly increased [START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF]). The mean marathon performance of the age groups 35 to 44 years and 45 to 54 years tends to be better than the corresponding performance in the age groups 25 to 34 years [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF]. Regarding the marathon running performance for master runners older than 50 years, an improvement in performance has even been observed in recent years. It has also been suggested that master runners of 50 years and older improved their performance at a significantly greater rate than younger runners [START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF]). An analysis of the performance trend at the 'New York City Marathon' from 1989 to 2009, showed that during that period, the running times of master runners significantly decreased for males older than 64 years and for females older than 44 years, respectively [START_REF] Lepers | Do older athletes reach limits in their performance during marathon running? Age[END_REF].

To date, the age-related decline in running performance has been investigated in halfmarathoners [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF], marathoners [START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF][START_REF] Lepers | Do older athletes reach limits in their performance during marathon running? Age[END_REF][START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF][START_REF] Leyk | Performance, training and lifestyle parameters of marathon runners aged 20-80 years: results of the PACE-study[END_REF][START_REF] Trappe | Marathon runners: How do they age?[END_REF]) and 161-km ultramarathoners [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF][START_REF] Hoffman | The Western States 100-Mile Endurance Run: participation and performance trends[END_REF][START_REF] Hoffman | Historical analysis of participation in 161 km ultramarathons in North America[END_REF]. However, no study has analyzed the age-related changes in 100-km road-running performance. Also, nowadays, elderly runners in ultra-marathons are able to finish within the time limit.

Recently, it has been reported that an 81-year-old athlete was able to finish a 100-km ultramarathon within 19h 44min [START_REF] Knechtle | Change of body composition in an 81 year old runner in a 100 km run[END_REF]. A 100-km ultra-marathon takes about three time longer than a 42 km conventional marathon for the best runners (World male record 6h 13min for 100 km versus 2h 04min for 42 km). The energetic demands and mechanical stresses are much greater during a 100-km ultra-marathon compared with a conventional 42 km run. For example, it has been demonstrated that long-distance running such as completing an ultra-marathon induced more impact-stress on the skeletal muscle than a marathon [START_REF] Kim | Changes in serum cartilage oligomeric matrix protein (COMP), plasma CPK and plasma hs-CRP in relation to running distance in a marathon (42.195 km) and an ultra-marathon (200 km) race[END_REF]). Therefore, it may be interesting to analyse the age and gender interactions in 100-km road-running performance.

The aims of the present study were (i) to investigate the participation and performance trends in 100-km ultra-marathon from 1998 to 2010 and (ii) to compare the male versus female agerelated change in 100-km running performance. Respecting existing literature regarding halfmarathon, marathon and 161-km ultra-marathon, we hypothesized (i) an increase in participation and an improvement of performance for both male and female master athletes and (ii) a more pronounced age-related decline in the 100-km running performance for female compared with male runners.

Methods

This study was approved by the Institutional Review Board of St. Gallen, Switzerland, with a waiver for the requirement of an informed consent, given that the study involved the analysis of publicly available data. For each race the age of the athletes and the run performance times of both the male and female finishers at the Biel 100-km ultra-marathon (Switzerland) were analysed from 1998 to 2010. The data set from this study was obtained from the race website (www.100km.ch) and from the Race Director. No race results were available before 1998 as this was the year when electronic registration of race results started and paper results were not available from the races held before this date. The Biel 100-km ultra-marathon generally takes place on the first weekend in June. The runners start the race at 10:00 p.m. The Biel 100-km ultra-marathon involves a total altitude change of 645 m. Two thirds of the course is on asphalt, and one third is on unpaved roads. Throughout the 100 km there are 17 aid stations at intervals of ~5 to ~10 km supplying a variety of food and beverages. The organiser offers hypotonic sport drinks, tea, soup, caffeinated drinks, water, bananas, oranges, energy bars, cakes and bread. The athletes are allowed to be supported by a cyclist in order to have additional food and clothing, if necessary.

Data analysis

The age and the race time (converted to minutes) of all finishers were analysed from 1998 to 2010 for both male and female runners.

Age and performance of the winners and the top 10 overall

Firstly, the age and the race time of both male and female winners were analyzed during the 13-years period. Secondly, the age and race time data was averaged across the first ten male and female finishers for each of these years. The magnitudes of the gender difference were examined by calculating the percent difference for the race times of the male versus female winners and of the top ten male versus female finishers.

Age-related changes in performance

In order to analyse the age-related change in running performances, we pooled data from 1998 to 2010 and distinguished the age groups as follows: 18-24 years, 25-29 years, 30-34 years, 35-39 years, 40-44 years, 45-49 years, 50-54 years, 55-59 years, 60-64 years, 65-69 years, 70-74 years and 75 years and older. Because of the the small participation of athletes older than 65 years, especially for females, we therefore considered only the performances of the best 20 male and female runners per age group during the 13-years period. In addition, the 100-km running time performance of each runner finishing in the top 20 of each age group was normalized to the mean time performance of the top 20 of the best performing age group for both males and females. Thus, the age-related declines in performance were expressed using a ratio calculated between the individual and the mean time performances of the best performing age group.

Statistical analysis

Data is reported as means  SD in the text. Linear regressions were used for estimating the changes of the selected variables in each year of the race. Pearson's correlation coefficients were used to assess the association between various variables (Statsoft, Version 6.1, Statistica, Tulsa, OK, USA). The ages of the female versus male winners were compared with an unpaired student's t-test. To determine if the age of the top ten male and female finishers differed over the years, a separate ANOVA with repeated measures for each year, with gender as the between-subject factor, was performed. One-way ANOVA was used to compare the race times between the different age groups for both male and female runners. A two way ANOVA (age x sex) was used to compare performance ratios between males versus females across age. Tukey's post hoc analyses were used to test differences within the ANOVA when appropriate. Statistical significance was accepted at P < 0.05.

Results

Participation trends

From 1998 to 2010, there was a total of 19,650 finishers (2,615 females and 17,035 males) at the Biel 100-km ultra-marathon. The number of finishers each year over the history of the event is shown in Figure 1. In these years, the average number of finishers per year was 1,312

 237 [range: 1,060-1,999] for male runners and 200  47 [range: 167-349] for females.

Females accounted on average for 13.3  1.4% of the field over the 13-years period. The greatest participation, in 2008, corresponded with the 50 th anniversary of the race.

Age of the 100-km ultra-marathoners

The mean age of the finishers during this period was 46.5  10.8 years for males, and 46.8  10.9 years for females. The age distribution of both male and female finishers during the 13year period is shown in

Winner and top 10 overall age trends

Figure 4 shows the historical age trends of the female and male winners (Panel A) and top ten male and female top ten overall finishers (Panel B) between 1998 and 2010. The female winners were significantly (P < 0.01) younger than the male winners with 33.2  6.4 years versus 38.2  4.5 years, respectively. In contrast, ANOVA revealed no difference in the mean age of the top ten males and females over the years (P = 0.19). The mean age of the top ten female and male runners was 39.4  2.3 years and 40.4  1.9 years, respectively. During the period studied, both female and male winners' age remained stable across the years. In contrast, the mean age of the top ten overall increased significantly by 0.36 year per annum for female and by 0.37 year per annum for male, respectively. During this period, the top ten running time remained stable for females while it significantly increased by 2.4 min per annum for males.The average time difference between the top ten males and females was equal to 22  3 % [range: 18-27%].

Winner and top ten overall performance trends

Age-related changes in performance

The mean age-related changes in both male and female race times throughout 1998-2010 are shown in Figure 6. The race time increased in a curvilinear manner with advancing age.

There was a significant (P < 0.0001) age effect for both male and female race times. No significant difference in time was observed for the four age groups between 30 to 34 years and 45 to 49 years for males, and for the five age groups between 30 to 34 years and 50 to 54 years for females. In males, the race time was significantly (P < 0.01) longer for the age groups 50 to 54 years and older, 18 to 24 years and 25 to 29 years compared with the age groups between 30 to 34 years and 45 to 49 years. For females, the running times were significantly (P < 0.01) longer for the age groups 55 to 59 years and older, 18 to 24 years and 25 to 29 years compared with the age groups between 30 to 34 years and 50 to 54 years.

For both males and females, the performance ratios for the 100-km running decreased in a curvilinear manner with advancing age (see Figure 7). There was a significant interaction between age and sex for performance ratio (F = 18.2, P < 0.01). For the males, the performance ratio for age groups 45 to 49 years and above was significantly different from the best age group (i.e. 35 to 39 years). For the females, the performance ratio for age groups 50 to 54 years and above was significantly different from the best age group (i.e. 35 to 39 years). The performance ratio was significantly different (P < 0.01) between males and females for the age groups 18 to 24 years, 65 to 69 years, and 70 to 74 years.

Discussion

The first aim of the study was to investigate the participation and performance trends at the Biel 100-km ultra-marathon from 1998 to 2010. Regarding the participation trend over the studied period, the percentage of both male and female finishers decreased for the age groups comprised between 18 and 39 years, while it increased for the age groups comprised between 40 and 59 years. For the age group 60-69 years, the percentage of finishers slightly increased for male while it remained stable for female.

The age group with the largest participation in these 100-km ultra-marathoners was 40-49 years for both males (~36%) and females (~42%). By comparison, [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF] described that in marathoners, the age groups from 35 to 54 years had the greatest percent of finishers with ~36%, surpassing the age segments of 25 to 34 years with ~13% and 55 to 64 years with ~12% by more than a factor of two, irrespective of the gender. [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF] analysed the participation and performance trends in 161-km ultra-marathoners from 1977 to 2008 in the USA. The annual finish rates increased initially and then remained unchanged to the early 1990s. The age group of 40 to 49 years showed the largest participation for both males and females [START_REF] Hoffman | The Western States 100-Mile Endurance Run: participation and performance trends[END_REF]). It appears therefore that the age group with the largest participation in 100-km ultra-marathoners was quite similar from marathoners and 161-km ultra-marathoners. A possible explanation for the relative decrease in 100-km finishers of the younger age groups might be that younger athletes are more and more attracted by other sport's disciplines with more technical abilities and higher intensities [START_REF] Bernard | Age-Related Decline in Olympic Triathlon Performance: Effect of Locomotion Mode[END_REF][START_REF] Baker | Aging performance for masters records in athletics, swimming, rowing, cycling, triathlon, and weightlifting[END_REF]. In contrast, the relative increase in 100-km finishers of the older age groups might be explained by the rapid increase in the older population in Switzerland [START_REF] Robine | Nonagenarians and centenarians in Switzerland, 1860-2001: a demographic analysis[END_REF], as well as the considerable increase in life expectancy over the last 20 years in Switzerland [START_REF] Savidan | Premature deaths in Switzerland from 1995-2006: causes and trends[END_REF][START_REF] Wanner | Demographic study of ages and causes of death which contribute to gender disparities in life expectancy--case of Switzerland (1969-1993)[END_REF].

Factors such as enjoyment, health and fitness benefits, social and competition appear the main drivers for involvement of master athletes [START_REF] Shaw | Motivation and psychological skills in the senior athlete[END_REF].

Regarding the performance trend of elite (i.e. top ten) 100-km runners during the 1998-2010 period, the results showed that both the male and the female winner times remained unchanged during the period studied. From 1998 to 2010, the fastest winner time was 409 min for males (110% of the male world record set in 1998 by Takahiro Sunada from Japan with 373 min) and 471 min (119% of the female world record set in 2000 by Tomoe Abe from Japan with 393 min) for females, respectively. The 100-km run at Biel involves a total altitude change of 645 m that may significantly increase the running times compared with a flat course. Surprisingly, the top ten female running times remained unchanged while the top ten male running times significantly increased during the 13-years period. The decrease in elite male running performance while elite female performance stabilized is difficult to explain; lower competitive elite male field across the years could be a possible explanation. The average gender difference in 100-km running performance between the top ten males and females over the 1998-2010 period was ~22%. This was consistent with previous studies. In marathoners and half-marathoners, the running times of the best female athletes were ~20% for marathons and ~22.5% for half-marathons, greater compared with males [START_REF] Leyk | Aged-related changes in marathon and halfmarathon performances[END_REF]. Similarly, in 161-km ultra-marathoners, the fastest females were ~20% slower than the fastest males [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF].

The second of the study aim was to compare the age-related change in 100-km ultramarathoners between males and females. We hypothesized a more pronounced age-related decline in 100-km running performance for female compared with male runners. The first interesting finding was that both male and female 100-km ultra-marathoners younger than 30 years were slower compared with male runners between 30 and 49 years, and female runners between 30 and 54 years. These findings were different from reports on marathoners. Finish time data for the top 50 finishers in the 'New York City Marathon' has demonstrated that, regardless of gender, the marathon times were either comparable in the age groups 20 to 29 years and 30 to 39 years, or were slightly slower in the younger age group; followed by a curvilinear increase in times for each advancing 10-year age group [START_REF] Jokl | Master's performance in the New York City Marathon 1983-1999[END_REF]). It has been previously suggested that the age of the peak performance increased with the length of the race (Schulz et al. 1987). The present data confirms this assumption since it appears that high-level performances in a 100-km ultra-marathon can be achieved at a higher age compared with a 42-km marathon. Our results suggest that 100-km ultra-marathoners can succeed until 50 years of age for males and until 54 years of age for females. Regarding endurance performance, a cumulative number of training years is required for peak performance but in contrast age-related changes in physiological (and psychological) factors lead to the decline in performance. For 100-km ultra-marathoners, increasing training experience and age-related declines in physical function may balance each other between the age of 30 and 50 to 54 years. [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF] showed, in 161-km ultra-marathoners, that beyond the age of 30-39 years the average finish times increased linearly with age. The female age group 40-49 years and the male age group 30-39 years had the fastest race times, where runners in younger and older age groups were slower. In females, the age group 40-49 years was ~5.5 % faster than the 30-39 years one; among the males, the age group 40-49 years was ~4.0 % slower than the 30-39 years one [START_REF] Hoffman | Performance trends in 161-km ultramarathons[END_REF]. In Hoffman and Wegelin's study relating to 161-km ultra-marathoners, the best times among females were observed for ultra-runners aged between 30 and 39 years. The examination of the best times among the males revealed that the performance of the 40-49 years age group was no different from agegroups 20-29 and 30-39 years [START_REF] Hoffman | The Western States 100-Mile Endurance Run: participation and performance trends[END_REF]. Both the moderate decline in running performance and the large number of successful master athletes suggest that master runners are able to maintain a high degree of physiological performance late into life [START_REF] Young | Career-span analyses of track performance: Longitudinal data present a more optimistic view of age-related performance decline[END_REF].

It has been previously observed a greater rate of decline in endurance performance (e.g. swimming, running and triathlon) in females compared to males [START_REF] Donato | Declines in physiological functional capacity with age: A longitudinal study in peak swimming performance[END_REF][START_REF] Lepers | Age and gender interactions in ultraendurance performance: insight from the triathlon[END_REF][START_REF] Ransdell | Master athletes: An analysis of running, swimming and cycling performance by age and gender[END_REF]. For example, Wright and Pericelli (2008) investigated 1,351 male and 1,248 female finishers in the 2001 National Senior Olympic Games. Between 30 and 50 years old, there were minimal changes in performance per annum.

Over the next 25 years, from 50 to 75 years, the percentage change in performance per annum was three times the rate seen from 30 to 50 years (males, 1.46% per annum; females, 2.52% per annum, respectively). After the age of 75 years, the rate of performance decline per annum increased dramatically, with 10.29% for females and 4.1% for males, respectively. In the present study, the age-related decline in performance was similar between males and females until the age of 64 years but was significantly greater after for females compared with males. A first reason for this gender difference may partly be explained by selection bias. Indeed, there were a smaller number of female 100-km runners in the older age groups with females accounted for ~13% of all finishers from 1998 to 2010 ; which was lower than the female participation of ~20% in the 161-km ultra-marathons in the USA [START_REF] Hoffman | Historical analysis of participation in 161 km ultramarathons in North America[END_REF]. Social and psychological factors that leading to an improved performance in master runners should also be considered. It is possible that males are gaining more competitive opportunities as they age, or that they are seeking out competition later in life or later in their career [START_REF] Ransdell | Master athletes: An analysis of running, swimming and cycling performance by age and gender[END_REF]). In addition, as age increases master athletes do not have the same intrinsic drive to train as hard as they did when younger [START_REF] Korhonen | Biomechanical and skeletal muscle determinants of maximal running speed with aging[END_REF][START_REF] Reaburn | Endurance performance in masters athletes[END_REF][START_REF] Spirduso | Physical Dimensions of Aging, 2nd edition[END_REF]. [START_REF] Reaburn | Endurance performance in masters athletes[END_REF] reported genderbased differences in motivation with increasing age. Initially, males were more motivated by achievement and females by health, social interaction, and enjoyment. Over time, both male and female athletes ranked social interaction as the most important motivator for participation in master athletic events. Greater decrease in training volume and intensities in elderly female runners compared with their male counterpart may be the result of a change in intrinsic drive to train hard between male and female [START_REF] Korhonen | Biomechanical and skeletal muscle determinants of maximal running speed with aging[END_REF][START_REF] Okonek | Longitudinal analysis of change in sports performance of women between the ages of 30 and 75: a comparison between peak and leisure sports participation[END_REF][START_REF] Reaburn | Endurance performance in masters athletes[END_REF][START_REF] Spirduso | Physical Dimensions of Aging, 2nd edition[END_REF]. Apart from these socio-psychological reasons, physiological factors may also explain the gender difference in 100-km running performance with advancing age.

According to [START_REF] Reaburn | Endurance performance in masters athletes[END_REF], the physiological factors affecting endurance performance with increasing age are maximum oxygen consumption (VO 2 max), maximal heart rate, stroke volume, lactate threshold, economy of movement, muscle fiber type, activity of aerobic enzymes, blood volume and skeletal muscle mass. We should first address the finding that the females were able to achieve the same 100-km running performance time until the age of 54 years. At approximately 50 years old, females are generally peri-or postmenopausal [START_REF] Bernis | Environmental contexts of menopause in Spain: comparative results from recent research[END_REF][START_REF] Dratva | Is age at menopause increasing across Europe? Results on age at menopause and determinants from two population-based studies[END_REF]). In recent years, the age at menopause has shifted higher [START_REF] Dratva | Is age at menopause increasing across Europe? Results on age at menopause and determinants from two population-based studies[END_REF]). This might be a reason why the female peak performance has been maintained in the older age groups. Increased physical activity may also have increased the menopausal-related changes in bone mass. Physically active postmenopausal females show an increase in bone mineral density [START_REF] Hagberg | Moderate physical activity is associated with higher bone mineral density in postmenopausal women[END_REF] which may lead to a shift of the postmenopausal osteoporosis to higher ages [START_REF] Gibson | Determinants of bone density and prevalence of osteopenia among female runners in their second to seventh decades of age[END_REF], thus related to a reduced physical activity with increased morbidity and mortality [START_REF] Compston | Clinical and therapeutic aspects of osteoporosis[END_REF]. At the age of ~50 years, body weight and waist circumference increase in females. In addition, lean body mass and skeletal muscle mass decrease and fat mass increases [START_REF] Ho | Menopausal transition and changes of body composition: a prospective study in Chinese perimenopausal women[END_REF]Showers et al. 2007;[START_REF] Sternfeld | Physical activity and changes in weight and waist circumference in midlife women: findings from the Study of Women's Health Across the Nation[END_REF]). As the time after the menopause increases, the increases in fat mass and body weight become greater [START_REF] Guo | Aging, body composition, and lifestyle: the Fels Longitudinal Study[END_REF]. It has been shown that maintaining or increasing participation in regular physical activity contributes to prevention or attenuation of those gains [START_REF] Sternfeld | Physical activity and changes in weight and waist circumference in midlife women: findings from the Study of Women's Health Across the Nation[END_REF]).

Other factors leading to an improved performance in female master runners until the age of 54 years should also be considered. [START_REF] Ransdell | Master athletes: An analysis of running, swimming and cycling performance by age and gender[END_REF] described the smallest gender differences for running over 100 m, 800 m, 1,500 m, 5,000m, 10,000 m and the marathon in the age group 45 to 49 years. It is possible that females are gaining more competitive opportunities as they age, or that they are seeking out competition later in life, after childbirth, or later in their career [START_REF] Ransdell | Master athletes: An analysis of running, swimming and cycling performance by age and gender[END_REF]. Further reasons for the maintenance of running performance with age in 100 km female ultra-marathoners might be improved nutrition with an increased use of ergogenic supplements [START_REF] Striegel | The use of nutritional supplements among master athletes[END_REF], less over-use injuries despite increased age [START_REF] Knobloch | Acute and overuse injuries correlated to hours of training in master running athletes[END_REF], and less age-related diseases with increased morbidity and mortality [START_REF] Chevalier | Sports-related acute cardiovascular events in a general population: a French prospective study[END_REF]). However, [START_REF] Taunton | A retrospective case-control analysis of 2002 running injuries[END_REF] reported that females accounted for significantly more over-use injuries of the lower limbs, depending upon age and training history.

The age-related decline in 100-km running performance was greater for females compared with males after the age of 64 years. The decrease in running performance after the age of ~50 years can be explained in part by a decrease in skeletal muscle mass. A decreased muscle mass plays a role in the age-related decrease in VO 2 max in master endurance athletes [START_REF] Proctor | Skeletal muscle mass and the reduction of VO 2 max in trained older subjects[END_REF] thus leading to an impaired performance with increasing age. The decline in endurance performance appears primarily due to an age-related decrease in VO 2 max [START_REF] Reaburn | Endurance performance in masters athletes[END_REF] where gender differences in the rate of the age-related decline in VO 2 max are common [START_REF] Weiss | Gender differences in the decline in aerobic capacity and its physiological determinants during the later decades of life[END_REF]. Between 20 and 80 years old, the muscle area decreases by ~40% [START_REF] Lemmer | Age and gender responses to strength training and detraining[END_REF]. Muscle tissue analyses showed that both slow-and fast-twitch fibres decline with increasing age, although the loss of fast-twitch fibres is greater [START_REF] Korhonen | Biomechanical and skeletal muscle determinants of maximal running speed with aging[END_REF]. This is probably because of fibre atrophy as opposed to loss of inherent force production of the myofibrils [START_REF] Lemmer | Age and gender responses to strength training and detraining[END_REF]. Because muscle crosssection is related to overall muscular force production, strength declines with age. Females have a lower skeletal muscle mass compared to males, independent of the age and aging causes a similar decline in aerobic capacity in both genders, when expressed per kilogram of appendicular muscle mass [START_REF] Proctor | Skeletal muscle mass and the reduction of VO 2 max in trained older subjects[END_REF]. With advancing age, females tend to lose skeletal muscle mass more rapidly than males [START_REF] Spirduso | Physical Dimensions of Aging, 2nd edition[END_REF]. This may explain partly the greater age-related decline in 100-km running performance for females compared with males after the age of 64 years.

Conclusion

Although this study lacks some data about factors of endurance performance such as physiological and anthropometric parameters, training characteristics, previous experience (Knechtle et al. 2010a;Knechtle et al. 2010b;Knechtle et al., 2010c;[START_REF] Knechtle | What is associated with race performance in male 100 km ultra-marathoners: Anthropometry, training or marathon best time?[END_REF] and environmental conditions of the race [START_REF] Ely | Impact of weather on marathonrunning performance[END_REF][START_REF] Wegelin | Variables associated with odds of finishing and finish time in a 161-km ultramarathon[END_REF][START_REF] Vihma | Effects of weather on the performance of marathon runners[END_REF], it provides valuable data because master runners present a unique model to study the effects of high levels of physical training into older age [START_REF] Tanaka | Invited Review: Dynamic exercise performance in masters athletes: Insight into effects of primary human aging on physiological functional capacity[END_REF]. The percentage of both male and female 100-km finishers for the age groups comprised between 40 and 59 years increased during the 1998-2010 period. According to the top ten elite performances, the best age for a 100-km ultra-marathon performance is about 39-40 years, for both males and females. The age-related decline in performance was similar between males and females until the age of 64 years, but after 64 years it was significantly greater for females compared to males. The reasons for such a greater age-related decline in performance of female master runners compared with male master runners in the higher age groups are not clear. Differences in training volume and motivation, aerobic capacity, orthopaedic, nutrition, bone density, hormonal status, muscle mass, body composition, muscle typology for elderly female compared with male counterpart require further investigations.

Figure legends

Figure 3

Changes in the percentage of finishers per age group from 1998 to 2010, males (upper panel) and females (lower panel). In both males and females, the percentage of finishers decreased over this 13 year period for the age groups 18 to 29 (males: r = -0.69, P = 0.009; females: r = -0.71, P = 0.007) and 30 to 39 (males: r = -0.98, P = 0.0001; females: r = -0.83, P = 0.0005), while it increased in the age groups 40 to 49 (males: r = 0.79, P = 0.0013; females: r = 0.83, P = 0.0004) and 50 to 59 (males: r = 0.82, P = 0.0006; females: r = 0.78, P = 0.0018). The percentage of male finishers in the age groups 60-69 slightly increased (r = 0.56, P = 0.047), but remained stable in females (r = -0.28, P = 0.35).

Figure 4

Age of the overall male and female winners (Panel A) and mean ( SE) age of the overall top ten male and female finishers (Panel B) in the Biel 100-km ultra-marathon from 1998 to 2010.

The dotted line and solid line represent the linear regressions for the females and the males, respectively.

Figure 5

Performance times of the overall male and female winners in Biel 100-km ultra-marathon from 1998 to 2010 (see Panel A). Mean ( SD) performance times of the top ten overall male and female finishers over the same period (see Panel B). The dotted line and solid line represent the linear regressions for the females and the males, respectively.

Figure 6

Age-related changes in running performance in the Biel 100-km ultra-marathon. Data was pooled from 1998 to 2010. Values are means  SD. For males (n=240), race time was significantly (P<0.01) longer for the age groups 50-54 and older, 18 to 24 and 25-29, compared with the age groups between 30-34 and 45-49. For females (n=220), race time was significantly (P < 0.01) longer for the age groups 55-59 and older, 18-24 and 25-29, compared with age groups between 30-34 and 50-54. NS : Non significant difference between age groups from 30-34 and 45-49 for males, and from 30-34 and 50-54 for females.

Figure 7

Age-related declines in 100-km running performances for male (n=240) and female (n=220) runners expressed using a ratio calculated between the individual and the mean time performances of the best performing age group (mean  SD). For male, the performance ratio for age groups 45-49 years (full arrow) and above was significantly different from the best age group (i.e. 35-39 years). For female, the performance ratio for age groups 50-54 years (dotted arrow) and above was significantly different from the best age group (i.e. 35-39 years). *: P < 0.01 : Significantly different from female for the same age group.

Figure 2 .

 2 The 10 year age bracket with the largest participation was 40 to 49 for both sexes. Master runners older than 40 years represented 73% of the total finishers. The change in the percentage of finishers per age group from 1998 to 2010 for both males and females are presented in Figure3. Over this 13-year period, the percentage of finishers decreased for the age groups 18 to 29 years and 30 to 39 years, while it increased for the age groups 40 to 49 years and 50 to 59 years for both males and females. The percentage of finishers slightly increased for males in the age group 60 to 69 years and remained stable for females in the age group 60 to 69 years.

Figure 5

 5 Figure5shows the historical performance trends of the female and male winners (Panel A)

Figure 1

 1 Figure 1

Figure 2

 2 Figure2

  Figure1

Acknowledgements

We thank Mary Miller for her help in translation.

 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008


Mohammed Attia
email: mattia@computing.dcu.ie

Jennifer Foster
email: jfoster@computing.dcu.ie

Deirdre Hogan
email: dhogan@computing.dcu.ie

Joseph Le Roux
email: jleroux@computing.dcu.ie

Lamia Tounsi
email: ltounsi@computing.dcu.ie

Josef Van Genabith

Handling Unknown Words in Statistical Latent-Variable Parsing Models for Arabic, English and French

This paper presents a study of the impact of using simple and complex morphological clues to improve the classification of rare and unknown words for parsing. We compare this approach to a language-independent technique often used in parsers which is based solely on word frequencies. This study is applied to three languages that exhibit different levels of morphological expressiveness: Arabic, French and English. We integrate information about Arabic affixes and morphotactics into a PCFG-LA parser and obtain stateof-the-art accuracy. We also show that these morphological clues can be learnt automatically from an annotated corpus.

Introduction

For a parser to do a reasonable job of analysing free text, it must have a strategy for assigning part-ofspeech tags to words which are not in its lexicon. This problem, also known as the problem of unknown words, has received relatively little attention in the vast literature on Wall-Street-Journal (WSJ) statistical parsing. This is likely due to the fact that the proportion of unknown words in the standard English test set, Section 23 of the WSJ section of Penn Treebank, is quite small. The problem manifests itself when the text to be analysed comes from a different domain to the text upon which the parser has been trained, when the treebank upon which the parser has been trained is limited in size and when the language to be parsed is heavily inflected. We concentrate on the latter case, and examine the problem of unknown words for two languages which lie on opposite ends of the spectrum of morphological expressiveness and for one language which lies somewhere in between: Arabic, English and French.

In our experiments we use a Berkeley-style latentvariable PCFG parser and we contrast two techniques for handling unknown words within the generative parsing model: one in which no languagespecific information is employed and one in which morphological clues (or signatures) are exploited. We find that the improvement accrued from looking at a word's morphology is greater for Arabic and French than for English. The morphological clues we use for English are taken directly from the Berkeley parser [START_REF] Petrov | Learning accurate, compact and interpretable tree annotation[END_REF] and those for French from recent work on French statistical parsing with the Berkeley parser [START_REF] Crabbé | Expériences d'analyse syntaxique statistique du franc ¸ais[END_REF]Candito et al., 2009). For Arabic, we present our own set of heuristics to extract these signatures and demonstrate a statistically significant improvement of 3.25% over the baseline model which does not employ morphological information.

We next try to establish to what extent these clues can be learnt automatically by extracting affixes from the words in the training data and ranking these using information gain. We show that this automatic method performs quite well for all three languages.

The paper is organised as follows: In Section 2 we describe latent variable PCFG parsing models. This is followed in Section 3 by a description of our three datasets, including statistics on the extent of the unknown word problem in each. In Section 4, we present results on applying a version of the parser which uses a simple, language-agnostic, unknownword handling technique to our three languages. In Section 5, we show how this technique is extended to include morphological information and present parsing results for English and French. In Section 6, we describe the Arabic morphological system and explain how we used heuristic rules to cluster words into word-classes or signatures. We present parsing results for the version of the parser which uses this information. In Section 7, we describe our attempts to automatically determine the signatures for a language and present parsing results for the three languages. Finally, in Section 8, we discuss how this work might be fruitfully extended.

2 Latent Variable PCFG Parsing [START_REF] Johnson | PCFG models of linguistic tree representations[END_REF] showed that refining treebank categories with parent information leads to more accurate grammars. This was followed by a collection of linguistically motivated propositions for manual or semi-automatic modifications of categories in treebanks [START_REF] Klein | Accurate unlexicalised parsing[END_REF]. In PCFG-LAs, first introduced by [START_REF] Matsuzaki | Probabilistic CFG with latent annotations[END_REF], the refined categories are learnt from the treebank using unsupervised techniques. Each base category -and this includes part-of-speech tags -is augmented with an annotation that refines its distributional properties.

Following [START_REF] Petrov | Learning accurate, compact and interpretable tree annotation[END_REF] latent annotations and probabilities for the associated rules are learnt incrementally following an iterative process consisting of the repetition of three steps.

1. Split each annotation of each symbol into n (usually 2) new annotations and create rules with the new annotated symbols. Estimate 1 the probabilities of the newly created rules.

2. Evaluate the impact of the newly created annotations and discard the least useful ones. Reestimate probabilities with the new set of annotations.

3. Smooth the probabilities to prevent overfitting.

We use our own parser which trains a PCFG-LA using the above procedure and parses using the max-1 Estimation of the parameters is performed by running Expectation/Maximisation on the training corpus. rule parsing algorithm [START_REF] Petrov | Learning accurate, compact and interpretable tree annotation[END_REF][START_REF] Petrov | Improved inference for unlexicalized parsing[END_REF]. PCFG-LA parsing is relatively language-independent but has been shown to be very effective on several languages [START_REF] Petrov | Coarse-to-Fine Natural Language Processing[END_REF]. For our experiments, we set the number of iterations to be 5 and we test on sentences less than or equal to 40 words in length. All our experiments, apart from the final one, are carried out on the development sets of our three languages.

The Datasets

Arabic We use the the Penn Arabic Treebank (ATB) [START_REF] Bies | Penn Arabic Treebank guidelines[END_REF][START_REF] Maamouri | Developing an Arabic Treebank: Methods, guidelines, procedures, and tools[END_REF]. The ATB describes written Modern Standard Arabic newswire and follows the style and guidelines of the English Penn-II treebank. We use the part-of-speech tagset defined by Bikel and Bies [START_REF] Bikel | On the Parameter Space of Generative Lexicalized Parsing Models[END_REF]. We employ the usual treebank split (80% training, 10% development and 10% test).

English We use the Wall Street Journal section of the Penn-II Treebank [START_REF] Marcus | The Penn Treebank: Annotating predicate argument structure[END_REF]. We train our parser on sections 2-21 and use section 22 concatenated with section 24 as our development set. Final testing is carried out on Section 23.

French We use the French Treebank [START_REF] Abeillé | Treebanks: Building and Using Parsed Corpora, chapter Building a Treebank for French[END_REF] and divide it into 80% for training, 10% for development and 10% for final results. We follow the methodology defined by [START_REF] Crabbé | Expériences d'analyse syntaxique statistique du franc ¸ais[END_REF]: compound words are merged and the tagset consists of base categories augmented with morphological information in some cases2 .

Table 1 gives basic unknown word statistics for our three datasets. We calculate the proportion of words in our development sets which are unknown or rare (specified by the cutoff value) in the corresponding training set. To control for training set size, we also provide statistics when the English training set is reduced to the size of the Arabic and French training sets and when the Arabic training set is reduced to the size of the French training set. In an ideal world where training set sizes are the same for all languages, the problem of unknown words will be greatest for Arabic and smallest for English. It is reasonable to assume that the levels of inflectional richness have a role to play in these differences.

A Simple Lexical Probability Model

The simplest method for handling unknown words within a generative probabilistic parsing/tagging model is to reserve a proportion of the lexical rule probability mass for such cases. This is done by mapping rare words in the training data to a special UNKNOWN terminal symbol and estimating rule probabilities in the usual way. We illustrate the process with the toy unannotated PCFG in Figures 1 and2. The lexical rules in Fig. 1 are the original rules and the ones in Fig. 2 are the result of applying the rare-word-to-unknown-symbol transformation. Given the input sentence The shares recovered, the word recovered is mapped to the UNKNOWN token and the three edges corresponding to the rules N N S → UNKNOWN, V BD → UNKNOWN and JJ → UNKNOWN are added to the chart at this position. The disadvantage of this simple approach is obvious: all unknown words are treated equally and the tag whose probability distribution is most dominated by rare words in the training will be deemed the most likely (JJ for this example), regardless of the characteristics of the individual word. Apart from its ease of implementation, its main advantage is its language-independence -it can be used off-the-shelf for any language for which a PCFG is available. 3 One parameter along which the simple lexical 3 Our simple lexical model is equivalent to the Berkeley sim-pleLexicon option.

probability model can vary is the threshold used to decide whether a word in the training data is rare or "unknown". When the threshold is set to n, a word in the training data is considered to be unknown if it occurs n or fewer times. We experiment with three thresholds: 1, 5 and 10. The result of this experiment for our three languages is shown in Table 2.

The general trend we see in Table 2 is that the number of training set words considered to be unknown should be minimized. For all three languages, the worst performing grammar is the one obtained when the threshold is increased to 10. This result is not unexpected. With this simple lexical probability model, there is a trade-off between obtaining good guesses for words which do not occur in the training data and obtaining reliable statistics for words which do. The greater the proportion of the probability mass that we reserve for the unknown word section of the grammar, the more performance suffers on the known yet rare words since these are the words which are mapped to the UNKNOWN symbol. For example, assume the word restructuring occurs 10 times in the training data, always tagged as a VBG. If the unknown threshold is less than ten and if the word occurs in the sentence to be parsed, a VBG edge will be added to the chart at this word's position with the probability 10/#VBG. If, however, the threshold is set to 10, the word (in the training set and the input sentence) will be mapped to UNKNOWN and more possibilities will be explored (an edge for each T AG → UNKNOWN rule in the grammar). We can see from

Making use of Morphology

Unknown words are not all the same. We exploit this fact by examining the effect on parsing accuracy of clustering rare training set words using cues from the word's morphological structure. Affixes have been shown to be useful in part-of-speech tagging [START_REF] Schmid | Probabilistic part-of-speech tagging using decision trees[END_REF][START_REF] Tseng | Morphological features help POS tagging of unknown words across language varieties[END_REF] and have been used in the Charniak [START_REF] Charniak | A maximum-entropy-inspired parser[END_REF], Stanford [START_REF] Klein | Accurate unlexicalised parsing[END_REF] and Berkeley [START_REF] Petrov | Learning accurate, compact and interpretable tree annotation[END_REF] parsers. In this section, we contrast the effect on parsing accuracy of making use of such information for our three languages of interest.

Returning to our toy English example in Figures 1 and2, and given the input sentence The shares recovered, we would like to use the fact that the un-known word recovered ends with the past tense suffix -ed to boost the probability of the lexical rule V BD → UNKNOWN. If we specialise the UNKNOWN terminal using information from English morphology, we can do just that, resulting in the grammar in Figure 3. Now the word recovered is mapped to the symbol UNK-ed and the only edge which is added to the chart at this position is the one corresponding to the rule V BD → UNK-ed.

For our English experiments we use the unknown word classes (or signatures) which are used in the Berkeley parser. A signature indicates whether a words contains a digit or a hyphen, if a word starts with a capital letter or ends with one of the following English suffixes (both derivational and inflectional): -s, -ed, -ing, -ion, -er, -est, -ly, -ity, -y and -al.

For our French experiments we employ the same signature list as [START_REF] Crabbé | Expériences d'analyse syntaxique statistique du franc ¸ais[END_REF], which itself was adapted from [START_REF] Arun | Lexicalization in crosslinguistic probabilistic parsing: The case of French[END_REF]. This list consists of (a) conjugation suffixes of regu-lar verbs for common tenses (eg. -ons, -ez, -ent. . . ) and (b) derivational suffixes for nouns, adverbs and adjectives (eg. -tion, -ment, -able. . . ).

The result of employing signature information for French and English is shown in Table 3. Beside each f-score the absolute improvement over the UNKNOWN baseline (Table 2) is given. For both languages there is an improvement at all unknown thresholds. The improvement for English is statistically significant at unknown thresholds 1 and 10. 4 The improvement is more marked for French and is statistically significant at all levels.

In the next section, we experiment with signature lists for Arabic. 5

Arabic Signatures

In order to use morphological clues for Arabic we go further than just looking at suffixes. We exploit all the richness of the morphology of this language which can be expressed through morphotactics.

Handling Arabic Morphotactics

Morphotactics refers to the way morphemes combine together to form words [START_REF] Kenneth | Arabic morphology using only finite-state operations[END_REF][START_REF] Beesley | Finite State Morphology[END_REF]. Generally speaking, morphotactics can be concatenative, with morphemes either prefixed or suffixed to stems, or non-concatenative, with stems undergoing internal alternations to convey morphosyntactic information. Arabic is considered a typical example of a language that employs non-concatenative morphotactics.

Arabic words are traditionally classified into three types: verbs, nouns and particles. Adjectives take almost all the morphological forms of, and share the same templatic structures with, nouns. Adjectives, for example, can be definite, and are inflected for case, number and gender.

There are a number of indicators that tell us whether the word is a verb or a noun. Among 4 Statistical significance was determined using the stratified shuffling method. The software used to perform the test was downloaded from http://www.cis.upenn.edu/ ˜dbikel/software.html.

5 An inspection of the Berkeley Arabic grammar (available at http://code.google.com/p/berkeleyparser/ downloads/list) shows that no Arabic-specific signatures were employed. The Stanford parser uses 9 signatures for Arabic, designed for use with unvocalised text. An immediate future goal is to test this signature list with our parser. these indicators are prefixes, suffixes and word templates. A template [START_REF] Beesley | Finite State Morphology[END_REF] is a kind of vocalization mould in which a word fits. In derivational morphology Arabic words are formed through the amalgamation of two tiers, namely, root and template. A root is a sequence of three (rarely two or four) consonants which are called radicals, and the template is a pattern of vowels, or a combination of consonants and vowels, with slots into which the radicals of the root are inserted.

For the purpose of detection we use the reverse of this information. Given that we have a word, we try to extract the stem, by removing prefixes and suffixes, and match the word against a number of verbal and nominal templates. We found that most Arabic templatic structures are in complementary distribution, i.e. they are either restricted to nominal or verbal usage, and with simple regular expression matching we can decide whether a word form is a noun or a verb.

Noun Indicators

In order to detect that a word form is a noun (or adjective), we employ heuristic rules related to Arabic prefixes/suffixes and if none of these rules apply we attempt to match the word against templatic structures. Using this methodology, we are able to detect 95% of ATB nouns. 6 We define a list of 42 noun templates which are used to indicate active/passive participle nouns, verbal nouns, nouns of instrument and broken plural nouns (see Table 4 for some examples). Note that templates ending with taa marboutah "ap" or starting with meem madmoumah "mu" are not considered since they are covered by our suffix/prefix rules, which are as follows: 1-The definite article prefix or in Buckwalter transliteration "Al". 2-The tanween suffix , , or "N", "F", "K", "AF". 3-The feminine plural suffix À , or "+At". 4-The taa marboutah ending or "ap" whether as a 6 The heuristics we developed are designed to work on diacritized texts. Although diacritics are generally ignored in modern writing, the issue of restoring diacritics has been satisfactorily addressed by different researchers. For example, [START_REF] Nelken | Arabic diacritization using weighted finite-state transducers[END_REF] presented an algorithm for restoring diacritics to undiacritized MSA texts with an accuracy of over 90% and Habasah et al. (2009) reported on a freely-available toolkit (MADA-TOKAN) an accuracy of over 96%. feminine marker suffix or part of the word. 5-The genitive case marking kasrah « , or "+i". 6-Words of length of at least five characters ending with doubled yaa ¯ ª or "y˜". 7-Words of length of at least six characters ending with alif mamdoudah and hamzah or "A'". 8-Words of length of at least seven characters starting with meem madmoumah ¬ or "mu".

Unknown Threshold

Recall

Verb Indicators

In the same way, we define a list of 16 templates and we combine them with heuristic rules related to Arabic prefixes/suffixes to detect whether a word form is exclusively a verb. The prefix/suffix heuristics are as follows: 9-The plural marker suffix « or "uwA" indicates a verb. 10-The prefixes

À ¸ ª ¸© ¸« ¸«
or "sa", ">a", ">u", "na", "nu", "ya", "yu", "ta", "tu" indicate im-prefective verb. The verbal templates are less in number than the noun templates yet they are no less effective in detecting the word class (see Table 4 for examples). Using these heuristics we are able to detect 85% of ATB verbs.

Arabic Signatures

We map the 72 noun/verb classes that are identified using our hand-crafted heuristics into sets of signatures of varying sizes: 4, 6, 14, 21, 25, 28 and 72. The very coarse-grained set considers just 4 signatures UNK-noun, UNK-verb, UNK-num, and UNK and the most fine-grained set of 72 signatures associates one signature per heuristic. In addition, we have evaluated the effect of reordering rules and templates and also the effect of collating all signatures satisfying an unknown word. The results of using these various signatures sets in parsing 5. We achieve our best labeled bracketing f-score using 28 signatures with an unknown threshold of five. In fact we get an improvement of 3.25% over using no signatures at all (see Table 2). Table 3 describes in more detail the scores obtained using the 28 signatures present in Table 6. Apart from the set containing 72 signatures, all of the baseline signature sets in Table 5 yield a statistically significant improvement over the generic UNKNOWN results (p < 10 -4 ).

Using Information Gain to Determine Signatures

It is clear that dividing the UNKNOWN terminal into more fine-grained categories based on morphological information helps parsing for our three languages. In this section we explore whether useful morphological clues can be learnt automatically. If they can, it means that a latent-variable PCFG parser can be adapted to any language without knowledge of the language in question since the only languagespecific component in such a parser is the unknownsignature specification.

In a nutshell, we extract affix features from train-ing set words 7 and then use information gain to rank these features in terms of their predictive power in a POS-tagging task. The features deemed most discriminative are then used as signatures, replacing our baseline signatures described in Sections 5 and 6. We are not going as far as actual POS-tagging, but rather seeing whether the affixes that make good features for a part-of-speech tagger also make good unknown word signatures.

We experiment with English and French suffixes of length 1-3 and Arabic prefixes and suffixes of various lengths as well as stem prefixes and suffixes of length 2, 4 and 6. For each of our languages we experiment with several information gain thresholds on our development sets and we fix on an English signature list containing 24 suffixes, a French list containing 48 suffixes and an Arabic list containing 38 prefixes and suffixes.

Our development set results are presented in Table 7. For all three languages, the information gain signatures perform at a comparable level to the baseline hand-crafted signatures (Table 3). For each of the three unknown-word handling techniques, no signature (UNKNOWN), hand-crafted signatures and information gain signatures, we select the best unknown threshold for each language's development set and apply these grammars to our test sets. The f-scores are presented in Table 8, along with the upper bounds obtained by parsing with these grammars in gold-tag mode. For French, the effect of tagging accuracy on overall parse accuracy is striking. The improvements that we get from using morphological signatures are greatest for Arabic 8 and smallest for 7 We omit all function words and high frequency words because we are interested in the behaviour of words which are likely to be similar to rare words. 8 Bikel's parser trained on the same Arabic data and tested on the same input achieves an f-score of 76.50%. We trained a 5-split-merge-iteration Berkeley grammar and parsed with the English. The results for the information gain signatures are promising and warrant further exploration.

Conclusion

We experiment with two unknown-word-handling techniques in a statistical generative parsing model, applying them to Arabic, French and English. One technique is language-agnostic and the other makes use of some morphological information (signatures) in assigning part-of-speech tags to unknown words. The performance differences from the two techniques are smallest for English, the language with the sparsest morphology of the three and the smallest proportion of unknown words in its development set. As a result of carrying out these experiments, we have developed a list of Arabic signatures which can be used with any statistical parser which does Berkeley parser, achieving an f-score of 75.28%. We trained the Berkeley parser with the -treebank SINGLEFILE option so that English signatures were not employed.

its own tagging. We also present results which show that signatures can be learnt automatically.

Our experiments have been carried out using gold tokens. Tokenisation is an issue particularly for Arabic, but also for French (since the treebank contains merged compounds) and to a much lesser extent for English (unedited text with missing apostrophes). It is important that the experiments in this paper are repeated on untokenised text using automatic tokenisation methods (e.g. MADA-TOKAN).

The performance improvements that we demonstrate for Arabic unknown-word handling are obviously just the tip of the iceberg in terms of what can be done to improve performance on a morphologically rich language. The simple generative lexical probability model we use can be improved by adopting a more sophisticated approach in which known and unknown word counts are combined when estimating lexical rule probabilities for rare words (see [START_REF] Huang | Selftraining pcfg grammars with latent annotations across languages[END_REF] and the Berkeley sophis-ticatedLexicon training option). Further work will also include making use of a lexical resource external to the treebank [START_REF] Goldberg | Enhancing unlexicalized parsing performance using a wide coverage lexicon, fuzzy tag-set mapping, and EM-HMM-based lexical probabilities[END_REF][START_REF] Habash | Four techniques for online handling of out-of-vocabulary words in arabic-english statistical machine translation[END_REF] and investigating clustering techniques to reduce data sparseness (Candito and Crabbé, 2009).

Table 1 :

 1 Basic Unknown Word Statistics for Arabic, French and English

	language cutoff	#train	#dev	#unk	%unk	language	#train	#dev	#unk	%unk
	Arabic	0	594,683 70,188	3794	5.40	Reduced English 597,999 72,970	2627	3.60
	-	1	-	-	6023	8.58	(Arabic Size)	-	-	3849	5.27
	-	5	-	-	11,347 16.17	-	-	-	6700	9.18
	-	10	-	-	15,035 21.42	-	-	-	9083	12.45
	English	0	950,028 72,970	2062	2.83	Reduced Arabic 266,132 70,188	7027	10.01
	-	1	-	-	2983	4.09	(French Size)	-	-	10,208 14.54
	-	5	-	-	5306	7.27	-	-	-	16,977 24.19
	-	10	-	-	7230	9.91	-	-	-	21,434 30.54
	French	0	268,842 35,374	2116	5.98	Reduced English 265,464 72,970	4188	5.74
	-	1	-	-	3136	8.89	(French Size)	-	-	5894	8.08
	-	5	-	-	5697	16.11	-	-	-	10,105 13.85
	-	10	-	-	7584	21.44	-	-	-	13,053 17.89

Table 2 :

 2 Table 1 that at threshold 10, one fifth Varying the Unknown Threshold with the Simple Lexical Probability Model of the words in the Arabic and French development sets are unknown, and this is reflected in the drop in parsing performance at these thresholds.

	VBD -> fell 50/153 VBD -> reoriented 2/153 VBD -> went 100/153 VBD -> latched 1/153 NNS -> photofinishers 1/201 NNS -> shares 200/201 JJ -> financial 20/24 JJ -> centrist 4/24 DT -> the 170/170	VBD -> fell 50/153 VBD -> UNKNOWN 3/153 VBD -> went 100/153 NNS -> UNKNOWN 1/201 NNS -> shares 200/201 JJ -> financial 20/24 JJ -> UNKNOWN 4/24 DT -> the 170/170	VBD -> fell 50/153 VBD -> UNK-ed 3/153 VBD -> went 100/153 NNS -> UNK-s 1/201 NNS -> shares 200/201 JJ -> financial 20/24 JJ -> UNK-ist 4/24 DT -> the 170/170
	Figure 1: The original toy PCFG	Figure 2: Rare → UNKNOWN	Figure 3: KNOWN+SIGNATURE Rare → UN-
	Unknown Threshold Recall Precision F-Score Tagging Accuracy
				Arabic
	1	78.60	80.49	79.53	94.03
	5	77.17	79.81	78.47	91.16
	10	75.32	78.69	76.97	89.06
				English
	1	89.20	89.73	89.47	95.60
	5	88.91	89.74	89.33	94.66
	10	88.00	88.97	88.48	93.61
				French
	1	83.60	84.17	83.88	94.90
	5	82.31	83.10	82.70	92.99
	10	80.87	82.05	81.45	91.56

Table 3 :

 3 Baseline Signatures for Arabic, French and English statistically significant with *:p < 10 -4 , **: p < 10 -3 , ***: p < 0.004,

						Precision	F-Score	Tagging Accuracy
							Arabic
	1				80.67	82.19	*81.42 (+ 1.89)	96.32
	5				80.66	82.81	*81.72 (+ 3.25)	95.15
	10				79.86	82.49	*81.15 (+ 4.18)	94.38
							English
	1				***89.64	89.95	89.79 (+ 0.32)	96.44
	5				89.16	89.80	89.48 (+ 0.15)	96.32
	10				89.14	89.78	**89.46 (+ 0.98)	96.21
							French
	1				85.15	85.77	*85.46 (+ 1.58)	96.13
	5				84.08	84.80	*84.44 (+ 1.74)	95.54
	10				84.21	84.78	*84.49 (+ 3.04)	94.68
				Template Name	Regular	Specification
	Arabic	Buckwalter	Expression
	« © «	© Ã	«	{inofiEAl	{ino.i.A.	verbal noun (masdar)
	« ©	«		mifoEAl	mi.o.A.	noun instrument
	«	© « Â	¬	musotafoEil musota.o.i. noun participle
	Â ª «		« © «	mafAEiyl	ma.A.iy.	noun plural
	« © « Â	«	{isotafoEal {isota.o.a. verb
			¬
	«		©		fuwEil	.uw.i.	verb passive

Table 4 :

 4 Sample Arabic Templatic Structures for Nouns and Verbs

Table 5 :

 5 Baseline Signatures for Arabic our Arabic development set are presented in Table

	UNK

Table 7 :

 7 Information Gain Signature Resultsstatistically significant with *:p < 10 -4 , **: p < 2 • 10 -4 , ***: p < 0.005

	Unknown Threshold Recall Precision	F-Score	Tagging Accuracy
					Arabic IG
		1	80.10	82.15	*81.11 (+ 1.58)	96.53
		5	80.03	82.49	*81.32 (+ 2.85)	95.30
		10	80.17	82.40	*81.27 (+ 4.3)	94.66
					English IG
		1	89.38	89.87	89.63 (+ 0.16)	96.45
		5	89.54	90.22	***89.88 (+ 0.55)	96.41
		10	89.22	90.05	*89.63 (+ 1.15)	96.19
					French IG
		1	84.78	85.36	*85.07 (+ 1.19)	96.17
		5	84.63	85.24	**84.93 (+ 2.23)	95.30
		10	84.18	84.80	*84.49 (+ 3.09)	94.68
	Language	No Sig Baseline Sig IG Sig
	Arabic	78.34	*81.59	*81.33
	Arabic Gold Tag	81.46	82.43	81.90
	English	89.48	89.65	89.77
	English Gold Tag	89.94	90.10	90.23
	French	83.74	*85.77	**85.55
	French Gold Tag	88.82	88.41	88.86

statistically significant with *: p < 10 -4 , **: p < 10 -3 Table 8: F-Scores on Test Sets

This is called the CC tagset: base categories with verbal moods and extraction features

Acknowledgements

This research is funded by Enterprise Ireland (CFTD/07/229 and PC/09/037) and the Irish Research Council for Science Engineering and Technology (IRCSET). We thank Marie Candito and our three reviewers for their very helpful suggestions.


Jennifer Foster

Özlem Çetinoglu

Joachim Wagner
email: jwagner@computing.dcu.ie

Joseph Le Roux
email: joseph.le-roux@lif.univ-mrs.fr

Stephen Hogan
email: shogan@computing.dcu.ie

Joakim Nivre
email: joakim.nivre@lingfil.uu.se

Deirdre Hogan
email: dhogan@computing.dcu.ie

Josef Van Genabith

Özlem C ¸etinoglu

#hardtoparse: POS Tagging and Parsing the Twitterverse

come

Introduction

While much progress has been made on supervised approaches to common natural language processing tasks such as part-of-speech tagging and syntactic parsing, many obstacles still remain before these problems can be said to be solved. The problem of domain adaptation is a well known one within the NLP and the machine learning community. How can a tool trained on one linguistic genre be adapted to another without access to substantial amounts of labelled data? The challenge becomes yet more daunting when we face, not just a new target domain, but the rapidly evolving, linguistically diverse mix of domains that is Web 2.0. In this paper, we examine the problem of adapting a pipeline dependency parsing system, trained on edited newswire, to the language of Twitter.

A dependency-based representation of syntactic structure is appealing because it captures people's notions of grammatical relations more intuitively than phrase structure, because it is a natural mode of representation for languages with a free word order and because parsing algorithms exist, which, when combined with enough training data and an adequate probability model, can produce dependency trees reasonably accurately in linear time. Labelled dependency representations are particularly useful since they serve as a basis Copyright c 2011, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.

for recovery of predicate argument structure and an answer to the question of who did what to whom. The Stanford labelled dependency scheme (de Marneffe and Manning 2008) has been used in many NLP applications including question answering, information extraction and sentiment analysis. The Stanford dependencies were originally designed to be produced from the output of phrase structure parsers but they have been used recently in the context of direct parsing into dependency trees using parsers such as Malt [START_REF] Nivre | Maltparser: A datadriven parser-generator for dependency parsing[END_REF] in research described in [START_REF] Cer | Parsing to stanford dependencies: Trade-offs between speed and accuracy[END_REF], [START_REF] Petrov | Uptraining for accurate deterministic question parsing[END_REF][START_REF] Foster | Lfg without c-structures[END_REF].

We train Malt to produce basic Stanford dependencies. We first train a model on the Wall Street Journal (WSJ) section of the Penn Treebank [START_REF] Marcus | The penn treebank: Annotating predicate argument structure[END_REF]), and we examine the parser's performance on a small treebank of sentences taken from microblogs (tweets). We find that the parser's performance drops 20 percentage points in labelled attachment accuracy. Because Malt accepts as input a sequence of part-of-speech (POS) tags rather than a sequence of words we also evaluate the accuracy of SVMTool [START_REF] Giménez | Svmtool: A general pos tagger generator based on support vector machines[END_REF], the POS tagger that we use to supply the input to Malt. A substantial proportion of the parsing errors can be attributed to POS tagging errors.

Unsupervised approaches to parser domain adaptation have met with moderate success in the last five years. Mc-Closky et al. (2006) showed that the performance of the Charniak and Johnson two-stage reranking parser [START_REF] Charniak | Course-to-fine n-bestparsing and maxent discriminative reranking[END_REF] could be improved on out-of-domain text by retraining the first stage generative parser on trees produced by the two-stage parser. [START_REF] Petrov | Uptraining for accurate deterministic question parsing[END_REF] demonstrated that the performance of a deterministic dependency parser on question data could be greatly improved upon by retraining it on a combination of its original material and trees produced by a slower, yet more accurate phrase structure parser. We combine these two ideas by retraining Malt on trees produced by a self-trained version of the Charniak and Johnson parser, achieving an LAS improvement of 4.67% on our Twitter test set. We examine in detail the effect of this uptraining on individual dependency relations.

Twitter

Twitter is a service that combines microblogging and social networking: through it, users can send short messages of up to 140 characters called tweets to their followers, i. e. other users who previously connected with the sender (social networking). The system is open as the connections are established without confirmation of the followee. Furthermore, the messages are publicly shown on the senders's profile page (microblogging). The service started as a "mobile status update service" but soon was widely used to report on events.1 Alternative routes to content are provided by a search function and lists of trending topics for the past minute, day and week.

The service provider, Twitter Inc., reports a quickly growing user base (460,000 new users daily as of March 2011) and an average of 140 million tweets per day. The Twitter interface makes it easy to forward a tweet to all followers with a retweet (RT). The @-sign before a user name creates a link to the user's profile page. A tweet containing such a link is called a mention. Mentions are also used for replies which, by convention, start with "@user". Another special symbol in Twitter is the hashtag which marks keywords that categorise the tweet. To use more than a single word for a category label, users have to omit word spaces, e. g. #Book-Week. 2Wu et al. ( 2011) classify Twitter users as celebrities, media, organisations, bloggers and ordinary users and find limited interaction between the first four groups, except for bloggers who retweet 85 times more than ordinary users. A study by Pear Analytics classifies over 40% of tweets as "pointless babble".3 Yet, breaking news often appears on Twitter before it reaches mainstream media.

.

Dataset

We create a small treebank of 519 syntactically annotated sentences taken from tweets. The source for these sentences is a corpus of 60 million tweets on 50 themes including politics, business, sport and entertainment, collected using the public Twitter API between February and May 2009 [START_REF] Bermingham | Classifying sentiment in microblogs: Is brevity an advantage?[END_REF]. Some Twitter-specific characteristics of this corpus are provided in Table 1. The tweets in our treebank were split by hand into sentences, usernames were replaced by the generic string Username and urls were replaced by Urlname. We use 269 sentences as a development set, which we refer to as TwitterDev, and the remaining 250 as a testset, which we refer to as TwitterTest. [START_REF] Bikel | Intricacies of collins parsing model[END_REF]. They were then corrected by hand by one annotator, using as a reference the Penn Treebank bracketing guidelines [START_REF] Bies | Bracketing guidelines for Treebank II style, Penn Treebank Project[END_REF] and the Penn Treebank trees themselves. Twitter-specific structures obviously do not appear in the Penn Treebank and a decision had to be made on how these should be annotated. Links, usernames and hash tags are all annotated as proper nouns inside a single word noun phrase. Links at the end of a tweet are attached to the verb in the same way an adverb occurring at the end of a sentence would be. The symbol RT is annotated as a noun within a single word noun phrase. The annotator went through the dataset twice, and a second annotator then annotated 10% of the sentences. Agreement on labelled bracketing between the two annotators is 95.8%. The disagreements involve fragments, interjections and multi-word expressions (see Table 3).

Evaluation of WSJ-Trained Resources

In this section, we evaluate the accuracy of the POS tagger, SVMTool [START_REF] Giménez | Svmtool: A general pos tagger generator based on support vector machines[END_REF], and the dependency parser, Malt [START_REF] Nivre | Maltparser: A datadriven parser-generator for dependency parsing[END_REF], on the sentences in TwitterDev. As well as reporting tagging and parsing accuracy for TwitterDev, we also report performance on Section 22, WSJ22, as our in-domain reference test set. We also carry out a qualitative evaluation using those sentences from TwitterDev that are listed in Table 4.

Accuracy of POS Tagging

SVMTool [START_REF] Giménez | Svmtool: A general pos tagger generator based on support vector machines[END_REF] uses support vector machine learning to induce taggers for various languages. We use the WSJ-trained model supplied with the software. The accuracy of SVMTool on TwitterDev is 84.1% compared to an accuracy of 96.3% on WSJ22. The most common POS confusions for TwitterDev are listed in Table 5.

A substantial proportion of the errors are mistaggings of proper nouns. Some of these cases relate to the generic names Username and Urlname, which were used to replace usernames and links and which should both be tagged as 4). Hash tags should also be tagged as proper nouns -7 of the 14 hash tags in TwitterDev have been mistagged. Apart from proper nouns beginning with lowercase characters, there are other capitalisation conventions that are worth mentioning because they are likely to be contributing towards the POS mistagging rate. One of these is the use of uppercase characters in an entire word or even sentence (see Examples 1 and 3 in Table 4). [START_REF] Foster | cba to check the spelling" Investigating parser performance on discussion forum posts[END_REF] identifies the use of uppercase characters as one of the factors in the relatively poor performance of parsers on sentences from a discussion forum. Inspecting the words mistagged by SVMTool, it seems that this is also a problem for SVMTool. We can see from Table 5 that some of the errors involve words that are not proper nouns being tagged as such. A possible reason for this, is that, in some tweets, news headlines in particular, the first character in every word is capitalised (see example 10 in Table 4). A tagger's job is made more difficult if the word to be tagged is not in its lexicon. In this situation, the tagger can use clues based on the word's morphology, its position within the sentence and properties of words occurring very infrequently in its lexicon. Unsurprisingly, the unknown token rate in TwitterDev is much higher than in WSJ22: 16.6% compared to 2.8%.

Accuracy of WSJ-trained Malt

Malt [START_REF] Nivre | Maltparser: A datadriven parser-generator for dependency parsing[END_REF]) is a widely used multilingual parsing system. During training, a classifier is induced to predict a parsing action at a particular parsing configuration using information from the parse history and the remaining input string. During parsing, the classifier is used to drive the deterministic construction of a dependency tree. Malt can be used with several parsing algorithms including variants of shift-reduce parsing. We use the stackeager algorithm described in [START_REF] Nivre | An improved oracle for dependency parsing with online reordering[END_REF] and we train a linear classifier where the feature interactions are modelled explicitly. We train Malt on a version of Sections 2-21 of the WSJ treebank that has been converted to labelled dependency trees using the Stanford constituency to dependency converter. We use SVMTool to supply the POS tagged input to Malt.

Table 7 shows the labelled attachment accuracy (LAS) and unlabelled attachment accuracy (UAS) of a WSJ-trained Malt model on both TwitterDev and WSJ22. There is a very large difference in accuracy between the in-domain and outof-domain test sets -an absolute difference of 20.34% in LAS. POS tagging errors account for a substantial proportion of this as the difference between automatic and gold tag input on TwitterDev is 11.03%. The dependency trees for two of the example sentences (sentences 1 and 6 from Table 4) are shown in Figures 1 and2. The POS tagging error that occurs in Sentence 1 (ppl tagged as NN rather than NNS) does not prevent Malt from attaching ppl to the correct head using the correct dependency type. However, there is another misparse which is not caused by a POS tagging error, namely, the attachment of the last word that to the word him rather than call. The dependency tree in Figure 2 is completely misparsed due to the errors in POS tagging. beltran is incorrectly analysed as the head of the sentence, with i and heart incorrectly identified as nominal modifiers.

Improving Parser Performance

Malt Uptraining [START_REF] Petrov | Uptraining for accurate deterministic question parsing[END_REF] demonstrate that the Malt's performance on question data can be substantially improved by training it on trees produced by the Berkeley parser, which has the advantage of producing slightly more accurate Stanford depen- The additional training sentences come not from Twitter but from a sports discussion forum. 4 An important point to note is that the POS tagger SVMTool also needs to be retrained on the same sentences as Malt.

The uptraining LAS results for TwitterDev are shown in Figure 3. The x-axis shows the amount of additional Charniak and Johnson parse trees that were added to either one or two copies of WSJ2-21. We can see from these results that both types of uptraining improve over the baseline but that domain-adapted uptraining is more successful than vanilla uptraining. Using the best model for TwitterDev, we parse the sentences in TwitterTest and achieve an LAS improvement of 4.67% and a UAS improvement of 3.74%. Both improvements are statistically significant.

Uptraining Error Analysis

The performance on dependency types is detailed below: amod The scores for adjectival modifiers are around 70%, following the baseline < vanilla uptraining < domainadapted uptraining trend (69.4, 70.3, 72). Even the best score is lower than the gold-POS-tagged baseline (87%). 4 cc and conj Coordination is represented using two dependency relations. The first conjunct is the head of the coordination and the other conjuncts are dependent on the head via a conj relation. The coordinating item (e.g., and, or) is dependent on the head via the cc relation. Figure 1 contains such a coordinated phrase. Our experiments show that it is hard for parsers to recover the conj relation: the f-score starts at 54.8%, goes up to 67.5% in vanilla uptraining, then decreases to 65% in domain-adapted uptraining. cc is easier to recover. The baseline is 70.3%. vanilla uptraining results in a big jump to 81.6%. Malt benefits from the domainadapted parse trees a lot and the f-score goes up to 85.3%. Both vanilla uptraining and domain-adapted uptraining are better than the gold-POS-tagged baseline, which is 79%.

ccomp Using domain-adapted trees in training also helps recover clausal complements. The baseline is 56.5%. There is a 1% absolute increase in vanilla uptraining, and a 7% absolute jump in domain-adapted uptraining. cop Similar to ccomp, there is a 1 point increase in vanilla uptraining and a 6 point increase in domain-adapted uptraining. Still the best score is almost 6 points lower than the gold-POS-tagged baseline.

dep This relation is used rather when the converter cannot determine the dependency type and consequently it is very hard for parsers to correctly identify it. For the baseline system, the f-score is 15.8%, for vanilla uptraining it goes down to 13.9%, and for domain-adapted uptraining it is 17%. dobj For direct objects, vanilla uptraining (75.2%) outperforms the baseline (65.2%) and domain-adapted uptraining (72%). The best system is 5% absolute lower than the gold-POS-tagged baseline. nsubj Nominal subjects have an f-score of 69.4% in the baseline system. Vanilla uptraining adds a 4% absolute increment and domain-adapted uptraining uptraining adds a further 2.2% with a final f-score of 75.7%.

prep Uptraining helps find prepositional modifers by 3% absolute but there is only a slight increase from vanilla uptraining to domain-adapted uptraining.

xcomp Open clausal complements have a baseline score of 65.9%, which goes up to 81.1% with vanilla uptraining, but drops to 78.4% with domain-adapted uptraining. Both grammars outperform the gold POS-tagged baseline.

For sentences 1 and 6 from Table 4, the dependency trees produced by domain-adapted training of Malt are given in Figures 4 and5. ppl in Sentence 1 is now correctly identified as NNS by the retrained tagger. The correct dependency relation of ppl in the baseline system remains the same. In the uptrained version, that is dependent on the correct head call, however with the wrong deprel dobj. The dobj correctly attached in the baseline is now mislabelled as iobj in the uptrained Malt. Another difference between the baseline grammar and the grammar uptrained on domain adapted trees is in the conjuncts of the coordination. This is a genuinely ambiguous sentence. Although the baseline grammar interpretation in Figure 1 seems more probable, the parse in Figure 4 is also reasonable. In Sentence 6, the correct POS tagging of i as PRP leads to a better parse tree in domainadapted uptraining. The nominal modifier (nn) in Figure 2 is correctly replaced with a nominal subject (nsubj) in Figure 5 although the other parsing errors remain.

Conclusions

We have examined the consequences of applying an offthe-shelf WSJ-trained POS-tagging and dependency parsing model to the language of Twitter. Encouragingly, unsupervised techniques go some of the way towards improving performance over the off-the-shelf baseline . However, much work remains to be done, given the noisy, diverse and constantly changing nature of Twitter. Our next step is to experiment with the Twitter-specific POS tagset and tagger described by [START_REF] Gimpel | Part-of-speech Tagging for Twitter: Annotation, Features and Experiments[END_REF].

(

  FRAG (INTJ (UH congrats)) (NP (NNP Tiger)) (. !) (. !)) versus (FRAG (NP (NNS congrats)) (NP (NNP Tiger)) (. !) (. !)) (FRAG (INTJ (IN Of) (NN course)) (. !)) versus (FRAG (PP (IN Of) (NP (NN course))) (! !)) (S (VP (VBG picking) (PRT (RP up)) (NP (PRP$ my) (NN truck)) (PP (IN from) (NP (NNP toyota))) (PRN (NP (JJ nice) (NNS folks))))) versus (S (VP (VBG picking) (PRT (RP up)) (NP (PRP$ my) (NN truck)) (PP (IN from) (NP (NNP toyota)))) (NP (JJ nice) (NNS folks))) (FRAG (NP (NNP USA)) (: -) (NP (NNP USA)) (: -) (NP (NNP USA)) (. !) (. !) (. !) (. !)) versus (X (NP (NNP USA)) (: -) (NP (NNP USA)) (: -) (NP (NNP USA)) (. !) (. !) (. !) (. !)) (FRAG (NP (NNP Username)) (INTJ (UH Okay) (, ,) (UH okay)) (. .)) versus (X (NP (NNP Username)) (ADJP (JJ Okay)) (, ,) (ADJP (JJ okay)) (. .))

Figure 1 :

 1 Figure 1: The baseline Malt analysis for Sentence 1 from Table 4

Figure 2 :

 2 Figure 2: The baseline Malt analysis for Sentence 6 from Table 4

  4 A series of retraining experiments with a variety of parsers demonstrated that the discussion forum data (http:// news.bbc.co.uk/sport2/hi/606/default.stm) was more useful than Twitter data as a source of additional unlabelled training data.

Figure 3 :

 3 Figure 3: Malt uptraining results on Twitterdev

Figure 5 :

 5 Figure 5: The domain adapted Malt analysis for Sentence 6 from Table 4

Figure 4 :

 4 Figure 4: The domain adapted uptrained Malt analysis for Sentence 1 from Table 4

neg

  Negation benefits highly from training data with domain-adapted trees. Training with vanilla trees does not affect the baseline f-score, both are 78.6%. When domainadapted trees are used, the f-score jumps to 89.3%. nn The relation nn represents the dependency of nouns to a head noun in an NP. The baseline 64.8% goes down slightly in vanilla uptraining and goes up slightly in domainadapted uptraining. They are quite low when compared to the gold POS-tagged baseline which is 83.9%. Nominal compound Penn Treebank NPs are flat and this makes it hard for Stanford dependencies to correctly represent the dependency relation. Hence, the parsers trained on data converted by Stanford dependencies propagate the error.

Table 1 :

 1 Table 2 contains statistics on the treebank sentences. The treebank sentences were first parsed automatically using an implementation of the Collins Model 2 generative Twitter-specific characteristics of the full Twitter corpus

	Mean characters per tweet	80.9
	Mean words per tweet	14.7
	Proportion containing link	23.0%
	Proportion containing hashtag 5.4%
	Proportion mentions	38.9%
	Proportion replies		31.6%
	Proportion retweets	2.6%
	Corpus Name #Sen SL Mean SL Med. σ
	TwitterDev	269	11.14	10	6.43
	TwitterTest	250	11.36	10	6.80

Table 2 :

 2 Basic Statistics on the Twitter treebank: number of sentences, average sentence length, median sentence length and standard deviation statistical parser

Table 3 :

 3 Inter-annotator disagreements on a subset of TwitterDev trees 1. I just think he looks like a big baby , and ppl USED to call him that. I PRP just RB think VBP he PRP looks VBZ like IN a DT big JJ baby NN , , and CC ppl NN USED VBD to TO call VB him PRP that DT . . 2. been playing with the new Canon EOS 500d and the Nikon D5000 over the weekend . been VBN playing VBG with IN the DT new JJ Canon NNP EOS NNP 500d JJ and CC the DT Nikon NNP D5000 NN over IN the DT weekend NN . . 3. On Fox : RNC chair sends letter to GOP calling Obama " ARROGANT " #tcot #sgp #hhrs On IN Fox NNP : : RNC NNP chair NN sends VBZ letter NN to TO GOP NNP calling VBG Obama NNP '' '' ARROGANT NNP '' '' #tcot NN #sgp NN #hhrs NNS 4. FF > S4 FF NN > NN S4 NN 5.LOL ! LOL NNP ! . 6. i heart beltran . i FW heart NN beltran NN . . 7. Man Utd through to the last 8 ... Man NNP Utd NNP through IN to TO the DT last JJ 8 CD ... : 8. Bed soon . Bed VBN soon RB . . 9. twas okay . twas NNS okay JJ . . 10. Obama Loses Two More Appointees : Sanjay Gupta , Annette Nazareth Urlname Obama NNP Loses VBZ Two CD More JJR Appointees NNPS : : Sanjay NNP Gupta NNP , , Annette NNP Nazareth NNP Urlname NNP

Table 6 :

 6 Output of SVMTool for examples from Table41. I just think he looks like a big baby , and ppl USED to call him that . 2. been playing with the new Canon EOS 500d and the Nikon D5000 over the weekend . 3. On Fox : RNC chair sends letter to GOP calling Obama " ARROGANT " #tcot #sgp #hhrs

	4. FF > S4
	5. LOL !
	6. i heart beltran .
	7. Man Utd through to the last 8 ...
	8. Bed soon .
	9. twas okay .
	10. Obama Loses Two More Appointees
	: Sanjay Gupta , Annette Nazareth Urlname

Table 4 :

 4 Examples from TwitterDev

	Gold/System Freq.	Gold/System Freq.
	NNP/NN	59	VBZ/NNS	8
	NN/NNP	54	UH/NNP	7
	NNP/JJ	29	RB/NN	7
	NNP/VB	10	NNP/CD	7
	JJ/NN	10	NN/VB	6
	UH/NN	8	VB/NN	6
	JJ/NNP	8	VB/NNP	6
	NNP/NNS	8	VBP/VB	6
	NNPS/NNS	8	RP/IN	6

Table 5 :

 5

SVMTool Mistaggings on TwitterDev NNP. 19 of the 43 occurrences of Username are tagged incorrectly compared to just 3 of the 37 occurrences of Urlname. Another reason for the low recall of NNP tags is that tweeters, unlike Wall Street Journal editors, often do not capitalise proper nouns (see example 6 in Table

Table 7 :

 7 Excluding instances of Username and Malt Labelled Attachment and Unlabelled Attachment with SVMTool-tagged input and gold-tag input Urlname, the proportion of unknown tokens in TwitterDev is 14.0%. Of the words mistagged by SVMTool, 53.2% are words that are unknown to the tagger.We end this section by presenting, in Table6, the output of SVMTool for our example sentences in Table4. Tagging errors are highlighted in bold.

	Parser	LAS	UAS	LAS	UAS
		WSJ22	TwitterDev
	Malt Predicted Tag	87.98 90.61	67.64 73.75
	Malt Gold Tag	89.95 91.61	78.67 81.86

Most information in this section is compiled from articles from http://blog.twitter.com/ and http:// support.twitter.com/.

2 http://www.newyorker.com/online/blogs/ susanorlean/2010/06/hash.

html 3 http://www.pearanalytics.com/ blog/wp-content/uploads/2010/05/ Twitter-Study-August-2009.pdf

Acknowledgements

This research has been supported by the Enterprise Ireland Commercialisation Fund (CFTD/2007/229) and the Science Foundation Ireland (Grant 07/CE/ I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University, School of Computing, and by the French Agence Nationale pour la Recherche, through the SEQUOIA project (ANR-08-EMER-013). We thank the reviewers for their helpful comments.