Eva Navas
email: eva.navas@ehu.es

Inmaculada Hernáez
email: inma.hernaez@ehu.es

Iñaki Sainz
email: inaki.sainz@ehu.es

Evaluation of automatic break insertion for an agglutinative and inflected lan ACCEPTED MANUSCRIPT

Keywords: break prediction, prosodic phrasing, speech synthesis

This paper presents the evaluation of automatic break insertion for standard Basque. Basque is an agglutinative and inflected language and POS features, widely used for other languages, are not enough to accurately predict the insertion of breaks in the text. Other morpho-syntactic features, like grammatical case and information about syntagms have also been taken into account. With a textual corpus specially gathered for this study where the sentence internal punctuation marks have been removed, CARTs have been used to predict break locations. After applying parameter selection to the whole morpho-syntactic feature set, the best features were employed to build two CARTs, one that gives the same importance to deletion and insertion errors, T1, and another one, T2, that tries to minimise insertion errors. The objective evaluation of the break insertion algorithms gives a κ statistic of 0.518 and an F of 0.757 for T1 tree. The algorithms have also been subjectively evaluated and although T1 had better objective measures, the number of serious errors made by this tree is larger than the number of serious errors made by T2.

1

Introduction TTS systems must produce a natural sounding speech signal. Correctly placed breaks contribute greatly to this naturalness: they have influence in the grapheme to phoneme transcription due to the occurrence of external sandhi effects (such as the liaison in French), in the process of assigning durations to phones and in the generation of the intonation curve. Breaks give a rhythmic structure to the speech signal and in some cases they can even change the meaning of a sentence [START_REF] Hirschberg | Training Intonational Phrasing Rules Automatically for English and Spanish Text-to-Speech[END_REF]. However, even in error free texts, only part of the breaks is indicated in the text. Usually, punctuation marks divide the text and indicate how the elements are distributed. Although it is common to make a break when finding some punctuation marks such as full stops, exclamation marks, question marks, colon, semicolon and suspension points, other punctuation marks are not always realised with a break. This is the case of commas, parentheses, quotation marks and hyphens. In spite of having some logical indications of where to place the breaks, the reader usually places them where he/she thinks it is more convenient, often in places where no punctuation mark is present. This is the reason why a TTS system needs an algorithm to insert breaks in suitable places.

The task of automatically placing breaks in texts is undoubtedly a very difficult one. Due to the great influence in the quality of the produced synthetic speech that correct break positions have, usually TTS systems allow the use of the XML <break> tag to insert precise pauses. This is the case of the systems from AT&T Natural Voices™ 2 , Festival 3 , IBM 4 and VERBIO Speech Technologies 5 . <break> is an element defined in most of markup languages for speech synthesis, like Java Speech API Markup Language Specification (JSML), Speech Synthesis Markup Language (SSML), Virtual Human Markup Language (VHML) and Voice XML.

A particular framework where a break insertion algorithm is critical is a Speech to Speech translation (S2ST) task. The goal of the S2ST research is to enable interpersonal communication via natural spoken language for people who do not share a common language. A S2ST system makes use of very different technologies: automatic speech recognition (ASR) to transcribe the source speech, machine translation (MT) to convert the text from the source language to the target language and text to speech (TTS) conversion to produce the final speech signal in the target language. The TTS system must deal with text produced by the translation system, which in turn, uses the output text from the recognition system. Errors made by the two systems will reach the input of the TTS system which will have to cope with them. One important aspect prone to errors in this context is punctuation. Recognition and translation of punctuation is a difficult task.

Speakers divide their utterances into prosodic phrases separated by weak or strong prosodic boundaries. These boundaries can be perceptually classified according to the ToBI break indices [START_REF] Silverman | TOBI: A standard for Labeling English Prosody[END_REF]. Prosodic phrase break prediction can be decomposed into two tasks: prediction of break location and prediction of its acoustic realization. This paper deals with the former one, i.e. the building of an automatic model for the prediction of the positions of breaks.

Early models of break insertion used rules that searched for the best place to insert a break taking into account the word context [START_REF] Bachenko | A Computational Grammar of Discourse-Neutral Prosodic Parsing in English[END_REF]. [START_REF] Liberman | Text analysis and word pronunciation in text-to-speech synthesis[END_REF] decide the insertion of a break using the punctuation marks and the presence of a function word following a content word. This simple algorithm has been widely used in TTS systems, like in the MITalk TTS system [START_REF] Allen | From Text to Speech: the MITalk System[END_REF] and the multilingual one from Telefónica [START_REF] Castejón | Un Conversor Texto-Voz para Español[END_REF]. These are simple models that have good performance in most of the cases. But they are not suitable for agglutinative and inflected languages, where very few function words are used.

The goal of this study is to build a good break prediction algorithm for standard Basque that could be used in a S2ST application using available tools for automatic morpho-syntactic analysis and to check the validity of morphological and syntactic information for this task in Basque. Since Basque is an agglutinative and inflected language morpho-syntactic features not commonly used for break location prediction, like grammatical case and information about the grouping of words into syntagms, will be used.

The following sections are organised as follows: Section 2 presents the particularities of Basque related to this work. Section 3 describes the main characteristics of the corpus gathered for this study and presents the shallow analysis of the breaks in relation to the morpho-syntactic labels. Section 4 presents the statistical analysis carried out over the different predicting features considered and the predicting models built. It also describes the parameters used for the objective evaluation of the system performance. The objective and subjective evaluation of the developed algorithms is described in section 5. Finally, section 6 presents the discussion of the results.

2

Main features of Basque Basque language is official in the Autonomous Basque Community (Spanish area), so it is possible to study in Basque in all levels of education, although graduate level is limited to a small number of curricula, and post-graduate studies are almost non-existent. The language has a very high dialectal fragmentation with 7 main dialects and more than 50 varieties according to modern commonly accepted assumptions. A standardisation process of written Basque started on 1975, unifying and regulating the lexicon, the grammar, the syntax, the morphology and every other aspect of the written language. This process is still under way, particularly concerning lexica.

Basque is not an Indo-European language and differs considerably in grammar from the languages spoken in surroundings regions. It is an inflectional language in which grammatical relations between components within a clause are represented by suffixes. This is a distinguishing feature since the morphological information of the words is richer than in surrounding languages. Basque is an agglutinative language; that is, most words are formed by joining morphemes together. In particular, the affixes corresponding to the determination, number and grammatical case are taken in this order and independently of each other. One of the main characteristics of Basque is its declension system with numerous cases (more than 20), which differentiates it from the languages of the nearby countries. The inflections of determination, number and case appear only after the last element in the noun phrase. It is an extremely inflected language, inflecting both nouns and verbs. For an extensive review on Basque grammar, see Hualde and Ortiz de Urbina (2003).

As a minority language, language technology development for Basque differs in several aspects from the development for widely used languages. On the one hand, the size of the speakers' community is small. Consequently it has to face up to the scarcity of the resources and tools that could make possible this development at a reasonable and competitive rate. On the other hand, there are language-specific problems, related to language typology. It is not always possible to use or to adopt the language technologies developed for other languages. This is especially relevant in rule-based approaches, but also in corpus-based approaches, because truly efficient exploitation of corpus demands annotation, and this process is in most cases based on rule-based procedures, like morphological and syntactic analysis. For example, romance languages like Galician, Catalan or Occitan can take advantage of NLP developments for French or Spanish, but these developments are not so applicable to some languages, for example Basque. This applicability (or portability) depends largely on language similarity. As stated before, Basque is an agglutinative language, with a rich flexional morphology, and this requires specific procedures for language analysis and generation. For a review of resources and tools available for Basque see Díaz de [START_REF] Díaz De Ilarraza | HIZKING21: Integrating language engineering resources and tools into systems with linguistic capabilities[END_REF].

3

Corpus Since there was no suitable corpus in standard Basque for the study of the locations of breaks, a new corpus had to be collected. A textual database is considered appropriate to study break insertion because it is easy to collect and it provides with all the information needed. We have gathered a corpus introducing break labels in a written text, following the methodology proposed by [START_REF] Hirschberg | Training intonational phrasing rules automatically for English and Spanish text-to-speech[END_REF]. Texts were collected from one electronic newspaper and one electronic magazine, as well as from other Internet pages. The coverage of this corpus was analyzed using a large Basque Corpus of 1 100 000 words as a reference for the Basque language. With the corpus selected for this study the 64.4% of the large Basque Corpus is covered.

Our study makes the assumption that a break is always made when finding the following punctuation marks associated with a sentence boundary: full stops, exclamation marks, question marks, colon, semicolon and suspension points. In the framework of a S2ST system these breaks will be inserted in the TTS input text according to the pauses detected by a voice activity detector (VAD) working on the source speech. The correlation between pause durations and sentence boundary marks was studied for broadcast news data [START_REF] Gotoh | Sentence boundary detection in broadcast speech transcripts[END_REF] and it was observed that the longer the pause duration, the greater the chance of a sentence boundary existing. There have been several studies that have been able to detect sentence boundaries from speech and recognized text with success [START_REF] Hakkani-Tür | Combining words and prosody for information extraction from speech[END_REF][START_REF] Kim | The use of prosody in a combined system for punctuation generation and speech recognition[END_REF].

However, for an speech recognition system it is very difficult to place the sentence internal punctuation marks, like commas, quotation marks and parentheses, even taking into account acoustic information from the source speech [START_REF] Chen | Speech recognition with automatic punctuation[END_REF][START_REF] Christensen | Punctuation Annotation using Statistical Prosody Models[END_REF]. In the context of S2ST systems, these sentence internal punctuation marks will not be indicated or at least will not be reliable. So for our purposes, punctuation marks not indicating end of utterance were removed from the corpus, and the prediction of breaks will be made without this information.

In this work, both sentence internal and sentence boundary breaks will be predicted. Sentence internal breaks have to be predicted because they can be placed at points where no punctuation mark gives an indication, and because in our work all the information about internal punctuation marks has been removed as stated above. Sentence boundaries must be predicted because not all punctuation marks are related with a sentence boundary: full stops can be found in abbreviations, as decimal separator in numbers and in WEB addresses, question marks can be found mid-sentence…

3.1

Annotating the breaks The breaks were manually labelled by a Basque linguist, who hand labelled the corpus for likely prosodic boundaries. He was instructed to insert a break symbol wherever he thought a prosodic boundary was necessary when reading aloud the text. Only one level of break was considered in this study, hence a word juncture is considered to be either a break or a non break. As an example of the sentences from the corpus and the break annotation made, the two following sentences are presented, together with their closest translation to English6 (The sign | has been used to indicate the annotation of a break by the labeller):

• Sentence 1:

Errusiar honek | bost urte eman zituen | egunean hiru litro coca-cola edaten. | (Errusiar) (honek) (bost) (urte) (eman zituen) (egunean) (hiru) (litro) (coca-cola) (edaten) (Russian person) (this) (five) (years) (spent) (a day) (three) (litres) (coca-cola) (drinking)

This Russian person spent five years drinking three litres of coca-cola a day.

• Sentence 2: Askok esperoko ez duten arren | Garcia da Euskal Herriko biztanleen artean | gehien errepikatzen den abizena.

| (Askok) (esperoko) (ez) (duten) (arren) (Garcia) (da) (Euskal) (Herriko) (biztanleen) (artean) (gehien) (errepikatzen) (den) (abizena) (A lot of people) (expect) (no) (would) (although) (Garcia) (is) (Basque) (Country) (inhabitants) (among) (more) (repeated) (that is) (surname)
Although not a lot of people would expect it Garcia is the surname that is more repeated among the inhabitants of the Basque Country.

The main features of the labelled corpus are presented in table 1. As this table shows there is only 17.5% function words in this corpus. This small proportion implies that a simple algorithm based in content/function word discrimination cannot be used. In the corpus, only 9.4% of the annotated breaks followed a function word, and only 5.5% of the breaks were preceded by a function word. Therefore, most of the breaks occur between content words.

The number of syllables between breaks was also analyzed and compared with the distribution of the number of syllables in the sentences. These distributions are presented in figure 1. The mean number of syllables between breaks is 13.8, while the mean number of syllables in the sentences is 33.1.

3.2

Labelling the corpus Traditionally part of speech (POS) information has proven to be very useful for the prediction of breaks in several languages like English [START_REF] Black | Assigning phrase breaks from part-of-speech sequences[END_REF], French (Boula de [START_REF] Boula De Mareüil | Text Chunking for Prosodic Phrasing in French[END_REF], Thai [START_REF] Tesprasit | Learning phrase break detection in Thai text-to-speech[END_REF], Spanish [START_REF] Bonafonte | Phrase break prediction using a finite state transducer[END_REF], German [START_REF] Apel | Have a break! Modelling pauses in German speech[END_REF], Dutch [START_REF] Marsi | Learning to predict pitch accents and prosodic boundaries in Dutch[END_REF], Greek [START_REF] Maragoudakis | A Data-Driven Framework for Intonational Phrase Break Prediction[END_REF], Mandarin [START_REF] Zheng | Using multiple linguistic features for Mandarin phrase break prediction in maximum-entropy classification framework[END_REF] and Korean [START_REF] Kim | Incorporating second-order information into two-step major phrase break prediction for Korean[END_REF]. This information has also been used in standard Basque [START_REF] Navas | Assigning Phrase Breaks using CART's in Basque TTS[END_REF]. But, since Basque is a highly inflected language, other grammatical information that to our knowledge has never been used for the prediction of breaks, like grammatical case, can also be taken into account.

The corpus used in this study was morphologically and syntactically annotated, using some tools developed by the IXA research group7 . For the POS tagging, the morphological analyzer for Basque MORFEUS [START_REF] Ezeiza | Combining Stochastic and Rule-Based Methods for Disambiguation in Agglutinative Languages[END_REF] has been used. This program provides the lemma of each word and a set of main tags, corresponding to the POS or category of the word. Most of them have two more levels of subcategorization, according to grammatical considerations: the subcategory and the grammatical case. The accuracy of MORFEUS is 90% when category, subcategory and case are taken into account [START_REF] Ezeiza | Combining Stochastic and Rule-Based Methods for Disambiguation in Agglutinative Languages[END_REF]. The number of different tags supplied by MORFEUS was too large and detailed for our purposes. Besides, giving non significant features to a classifier introduces noise that may degrade its performance [START_REF] Read | Using part-of-speech for predicting phrase breaks[END_REF]. So, the full tag set was reduced, especially at the level of subcategory. The tag sets used for the category, subcategory and case, together with the corresponding description and an example in Basque with the closest translation to English are listed in tables 2, 3 and 4. These tables also give the percentage of words with a specific label that were followed by a hand annotated break in the corpus.

Table 2 shows that category PUNT, corresponding to punctuation marks, is followed by a break in the 99.2% of the cases. Taking into account that sentence internal punctuation marks (like commas) have been removed from the corpus, a 100% has to be expected. Actually, the cases with PUNT category not followed by a break are due to dots following abbreviations that have been misinterpreted as punctuation marks by the automatic labelling system. Category LAB (abbreviations) is always followed by a break in the corpus. In fact, only three abbreviations appear in the corpus and they happen to be followed by a break. This datum is not statistically meaningful, but most abbreviations are placed after the noun (like mister, department, street) and would fulfil the same conditions. In this table we can also see that the probability of having a break after categories SIG (acronyms) and PRT (particles) is very small. Finally categories BST (others) and IOR (pronouns) are almost never found before a break.

Concerning subcategory labels, shown in table 3, the following tags are never followed by a break in the corpus: ELK (reciprocal pronouns), FAK (factitive verbs), IGB (indeterminate pronouns) and ZIU (security adverbs). On the other hand, the subcategory LOK (connectives), a subcategorization of category LOT (conjunctions) is followed by a break in more than 70% of cases.

When applying this same analysis to grammatical case, we can see in table 4 that both genitive cases GEL (for places) and GEN (for people) are almost never followed by a break. Case DESK (descriptive) is never followed by a break while case DES (benefactive) is followed by a break in more than 70% of cases.

The tag NONE was added to the subcategory and case sets, for the categories that have no subcategory or grammatical case. In the same way the tag MISSING was used when the word should have subcategory or grammatical case, but the automatic labelling system did not assign any tag.

Syntactic information has also been used in the prediction of breaks [START_REF] Ingulfsen | Influence of syntax on prosodic boundary prediction[END_REF][START_REF] Koehn | Improving Intonational Phrasing with Syntactic Information[END_REF][START_REF] Ostendorf | A hierarchical stochastic model for automatic prediction of prosodic boundary location[END_REF]. Although there is not a direct mapping between syntactic and prosodic structures [START_REF] Frazier | Don't break, or do: prosodic boundary preferences[END_REF], there is undoubtedly a relation between them. Syntactic information for standard Basque was obtained with an automatic labeller developed also by the IXA research group [START_REF] Aduriz | A Cascaded Syntactic Analyser for Basque Lecture Notes on Computer Science[END_REF]. This tool provides us with the syntactic function of each word and also groups the words into syntagms. Breaks will likely occur between syntagms, so information related with the grouping of words might be very valuable for the break insertion algorithm. The performance of this tool is 83% precision rate (correctly selected groups / number of groups returned) and 81.4% recall rate (correctly selected groups / actual groups in the sentence).

The set of syntactic labels provided by this tool was reduced in the same way as POS, subcategory and case tag sets. The values considered in this study are listed in table 5. As table 5 shows, label MP (subordinate sentence) is always followed by a hand annotated break, while the probability of making a break after JADNAG_MP_KM (subordinate main verb as noun), JADNAG_MP_IZLG (subordinate main verb as adjectival complement), IZLG (adjectival complement) and PJ (coordinating conjunction) is very small.

Table 6 shows the labels used for the grouping of words. In this table we can see that words in the beginning or inside the syntagms have very small probability of having a break after them. In fact, the cases where this happens are due to labelling errors.

For all types of information obtained from the morphological and syntactic labellers, tags within a window of five words have been considered. This window size has been chosen taking into account that the average length of each syntagm is 2.17 words, therefore to include (in average) all the words in the current word syntagm the following and preceding two words must be considered. This window has been centred in the word preceding the break to predict.

4

Experiments The prediction of break locations is a classification problem, where each inter-word space or juncture is a potential phrase break and the classifier task is to decide whether the juncture is a phrase break. Many techniques have been applied to this problem, such as Classification and Regression Trees (CART) [START_REF] Apel | Have a break! Modelling pauses in German speech[END_REF][START_REF] Koehn | Improving Intonational Phrasing with Syntactic Information[END_REF][START_REF] Sangho | Tree-based Modeling of Prosodic Phrasing and Segmental Duration for Korean TTS Systems[END_REF][START_REF] Wang | Automatic Classification of Intonational Phrase Boundaries[END_REF], Memory Based Learning (MLB) [START_REF] Busser | Predicting phrase breaks with memory-based learning[END_REF][START_REF] Zhao | Prosodic Phrasing with Inductive Learning[END_REF], Transformational Rule Based Learning [START_REF] Fordyce | Prosody prediction for speech synthesis using transformational rulebased learning[END_REF], Hidden Markov Models (HMM) [START_REF] Bell | Adaptation of prosodic phrasing models[END_REF][START_REF] Schmid | New statistical methods for phrase break prediction[END_REF][START_REF] Taylor | Assigning phrase breaks from part-of-speech sequences[END_REF], N-grams combined with CARTs [START_REF] Sun | Intonational Phrase Break Prediction Using Decision Tree and N-Gram Model[END_REF], Finite State Transducers (FST) [START_REF] Veilleux | Markov Modeling of Prosodic Phrase Structure[END_REF], Support Vector Machines (SVM) [START_REF] Zhao | Prosodic Phrasing with Inductive Learning[END_REF], Bayesian induction [START_REF] Zervas | Bayesian Induction of intonational phrase breaks[END_REF], Maximum Entropy Models (ME) [START_REF] Li | Chinese prosody phrase break prediction based on maximum entropy model[END_REF] and Logistic Generalized Linear Models (Logistic GLM) [START_REF] Yi | Phrase break prediction using logistic generalized linear model[END_REF].

In this work CART [START_REF] Breiman | Classification and Regression Trees[END_REF] has been used as classification technique. The trees have been built using 10-fold cross validation [START_REF] Stone | Cross-validation choice and assessment of statistical predictions[END_REF] and taking into account the priors in the corpus (20.7% breaks vs. 79.3% non breaks).

4.1

Error measures The evaluation of the performance of the break insertion algorithms is not trivial. Several efficiency measures, like percentage of junctures correct, percentage of breaks correctly predicted and ratio of false insertions to the total number of junctures, have been traditionally proposed for the objective evaluation of the break insertion algorithms [START_REF] Taylor | Assigning phrase breaks from part-of-speech sequences[END_REF]. To calculate all these measures a juncture wrongly classified is considered an error. In our classification problem there are two types of possible errors: insertions (I), when the algorithm makes a break that was not in the reference database and deletions (D), when the algorithm does not make a break that was indicated in the reference database.

The overall score achieved by the tree is calculated as the number of junctures correctly classified divided by the total number of inter-word spaces. It is calculated according to equation 1.

100 * (%) N I D N S - - = (1)
where N stands for total number of junctures D stands for total number of deletions I stands for total number of insertions This datum has to be interpreted carefully, because it depends on the proportion of breaks in the original corpus. In our test database 81.05% of the inter-word spaces were labelled as non-breaks, therefore an algorithm that does not place any break at all would achieve an overall score of 81%. To avoid this problem another measure to compute the score of the algorithms, the kappa statistic, has been proposed. This statistic was first suggested for linguistic classification tasks by [START_REF] Carletta | Assessing agreement on classification tasks: the kappa statistic[END_REF] and has since been used by others to avoid the dependency of the score on the proportion of non-breaks in the text [START_REF] Navas | Assigning Phrase Breaks using CART's in Basque TTS[END_REF][START_REF] Sanders | Using probabilistic methods to predict phrase boundaries for a text-to-speech system[END_REF][START_REF] Torres | Automatic determination of phrase breaks for Argentine Spanish[END_REF].

The kappa statistic is calculated as indicated by equation 2.

N NB N NB S - - = 1 κ (2)
where S stands for the overall score NB stands for the total number of non-breaks in the data N stands for the total number of junctures In expression (2), the overall score achieved by the tree is compared with the probability of having a non-break label in the data, eliminating the dependency on the structure of the data. If the algorithm does not insert any break, the value of the kappa statistic will be 0. If the method predicts every inter-word space correctly, κ equals 1. Values lower than 0 indicate that the breaks placed by the algorithm are in the wrong places, so it would be better not to use it.

Other appropriate measures for this problem are precision (P) and recall (R). They are widely used in the field of information retrieval to evaluate the search performance [START_REF] Salton | The "Generality" Effect and the Retrieval Evaluation for Large Collections[END_REF]. These values and their weighted harmonic mean, known as balanced F-score [START_REF] Van Rijsbergen | Information Retrieval[END_REF], have also been applied to evaluate the performance of break prediction algorithms [START_REF] Busser | Predicting phrase breaks with memory-based learning[END_REF][START_REF] Li | Chinese prosody phrase break prediction based on maximum entropy model[END_REF][START_REF] Read | Stochastic and syntactic techniques for predicting phrase breaks[END_REF][START_REF] Sun | Intonational Phrase Break Prediction Using Decision Tree and N-Gram Model[END_REF]. In this context, precision evaluates the proportion of breaks correctly identified to all the predicted breaks and recall measures the proportion of breaks correctly identified, out of all manually annotated breaks in the corpus.

The F-score is also known as the F1 measure, because recall and precision are evenly weighted. It is calculated according to equation 3. where P stands for precision and R for recall. If no breaks are inserted P and F are undefined and R is zero. If the algorithm inserts breaks, but places them in incorrect places, P and R are zero and F is undefined.

4.2

Feature selection Choosing which features to keep and which to discard is not a trivial question in prediction problems. The full set of features used to predict the location of breaks may not be optimal. Certain attributes might be noisy and could compromise the accuracy of the prediction. The challenge is to remove the noisy attributes from the database, without losing the usefulness of the data. The feature selection methods are often performed according to the nature of the data, and therefore they are not generally applicable to all kinds of data analyses [START_REF] Blum | Selection of relevant features and examples in machine learning[END_REF][START_REF] Langley | Selection of relevant features in machine learning[END_REF].

Another alternative for parameter selection is the use of methods that rely on identifying potentially discriminating attributes by using some statistical measures of the information content of attributes, such as their correlation coefficients. The motivation behind these methods is that two well-correlated attributes contain much the same information, and therefore one of them could be eliminated from the analysis. In doing so, we would reduce some noise in the data, thus hopefully improving the prediction accuracy.

In our database all the features are categorical. To study the correlation among them a Pearson's Chi-Square test [START_REF] Siegel | Nonparametric Statistics for the Behavioral Sciences, Second Edition[END_REF] was applied. With this purpose the necessary contingency tables were obtained and the Contingency Coefficient C was calculated according to equation 4:

N C + = 2 2 χ χ (4)
where N stands for the number of observations and χ 2 is Chi square value. This C coefficient was then corrected to adjust its variation to the range 0-1, obtaining the Sakoda's coefficient C * [START_REF] Agresti | Introduction to Categorical Data Analysis[END_REF]. The value of Sakoda's coefficient measures the association between two categorical variables. A value of 0 for this coefficient means that both categorical features are independent. It is calculated according to equation 5.

k k C C 1 * - = ( 5
)
where k is the minimum between r (number of rows of the corresponding contingency table) and c (number of columns of the corresponding contingency table).

Sakoda's coefficient between all the predictor variables and break class has been calculated to know the theoretical power of prediction of each of them. Likewise Sakoda's coefficient between each pair of predicting features has also been obtained, to estimate the degree of correlation between them. The results of this study are presented in table 7. According to this analysis the variable more related to break is syntactic label (with a C * of 0.368), followed by POS (C * =0.362) and subcategory (C * =0.350). The predicting features that are more related are POS and subcategory, and the more independent ones case and syntagm.

To know which kind of information has actually more influence in the prediction of the placing of breaks, the predicting information was combined in different trees and their performance was evaluated using the test corpus. A set of trees trained with only one type of feature was built. Their results are shown in table 8. Among them the best tree is the one that uses syntactic labels with a κ of 0.498 and an F of 0.713. This is the feature that shows the bigger correlation with break class according to Sakoda's contingency coefficient calculated.

Then all the possible trees using two types of features were built. In this case the tree using syntactic and POS labels is the one with better results among all the possible combinations. In accordance with the statistical analysis, POS is the feature with the second best correlation with break class.

When training the set of trees that use three types on information, the next relevant feature turns out to be grammatical case. According to Sakoda's contingency coefficient, the next relevant feature should be subcategory. But looking to the coefficient between subcategory and POS (0.613) and subcategory and syntactic information (0.600), we can see that subcategory is correlated with both of them. This may be the reason why this is not the next best feature to consider in the prediction of breaks using the tree. Although it is correlated with break class, it does not add enough information to the system.

Information about syntagms is the next significant type of information to take into account in the prediction of breaks. Adding the subcategory makes the results worse. The best results for each number of parameters used are presented in figure 2.

Taking these results into account the break prediction trees have been built without subcategory information.

Results

For the prediction experiments, the corpus is divided in 75% for training and 25% for testing purposes. Two different prediction trees have been built. The first one, T1, gives the same importance to both types of predicting errors (insertions and deletions). In the second one, T2, insertion errors are considered more harmful and therefore, these insertion errors have been given a weight of 1.33. This coefficient has been chosen by testing a few different weights on specific test examples.

5.1

Objective evaluation To allow an objective comparison of the trees' performance, we consider a fictitious break insertion system that offers the minimum necessary performance. This minimum performance would be the correct placement of all the breaks corresponding to sentence boundaries. This fictitious system is used as a baseline to evaluate the performance of the break insertion algorithms developed. With this baseline system, no intra sentence break would be marked; it would only place breaks associated with the punctuation marks corresponding to sentences boundaries. The results of this baseline system for our test database are shown in table 9, together with the ones achieved by the two trained trees.

Baseline system has a precision of 100%, because all the breaks it indicates are correct. However, its recall is very low, because it misses a lot of breaks (all the sentence-internal breaks). Both trees built have worse precision but better recall than the baseline system, achieving a value of F above 70%. Objective results for T1 are slightly better (F 75.7%, κ 0.518) than the ones obtained for T2 (F 72.5%, κ 0.511). T1 has better recall (it places more breaks from the breaks to be placed) but worse precision (from the breaks it places, fewer are correct) than T2.

5.2

Subjective evaluation Measuring the efficiency of a break insertion algorithm by comparing its results with the text labelling has the disadvantage that the subjective importance of the error is not considered. Moreover, all the errors have the same influence in the final measure, although some of them are more serious than others. To have an estimation of the subjective performance of the algorithms a subjective evaluation was carried out. For a perceptual evaluation either a TTS system or a real speaker must be used to read aloud the text. However, we thought that this would introduce more variability in the evaluation results. Since there are many different ways to realize the breaks, the results would have a strong dependence on the acoustic realization of the breaks.

30 sentences from the test corpus were randomly selected for the evaluation. The shortest one has 9 words and the longest one 29. 15 people took part in the subjective evaluation. All of them are fluent in standard Basque.

Evaluators were asked to insert the appropriate breaks in the sentences. They had to annotate the text introducing the likely prosodic boundaries they considered missing. The occurrence of pauses is strongly speaker dependent [START_REF] Zellner | Pauses and the temporal structure of speech[END_REF] and each evaluator will insert the breaks he/she considers appropriate. Their labelling was compared with the reference labelling and the prediction made by the two trained trees. After this comparison the breaks were classified into these three classes:

• Correct breaks: breaks labelled by the algorithm which have also been labelled by any of the evaluators.

• Omitted breaks: breaks labelled by any of the evaluators, but not predicted by the algorithms.

• Wrong breaks: breaks predicted by the algorithms that were not labelled by any of the evaluators.

The following data are used as a measure of the performance:

• The absolute number of correct, omitted and wrong breaks for each algorithm. The larger the quantity of correct breaks and the smaller the quantity of the omitted and wrong breaks are the better.

• Mean score of the breaks, i.e., the mean number of evaluators that selected the breaks. This quantity has sense only for correct and omitted breaks. In the case of correct breaks, the bigger this mean score is the better, because this means that more evaluators agree with the breaks inserted by the algorithm. For omitted breaks, the smaller the mean score the better the result is. In this case, small mean scores indicate that few evaluators selected these breaks, therefore they are not very important.

• Number of breaks selected by at least 80% of the evaluators. This measure is calculated for correct and omitted breaks and gives an idea of the number of breaks that are important for the majority of the evaluators.

Results of each of the algorithms are presented in table 10.

As expected, the reference labeller achieves the best scores. His number of correct breaks is 79, selected by 68% of the evaluators. Among these 79 breaks, 39 have been selected by at least 80% of the evaluators. Therefore, the evaluators strongly agree on these 39 breaks. T1 and T2 have fewer correct breaks, 62 and 45 respectively, but the correct breaks they insert are the very important ones (with scores of 0.71 and 0.74). Considering omitted breaks, the reference labeller also has the best results. He has omitted only 52 breaks of the ones indicated by the evaluators, with less than a third of the evaluators having selected these 52 breaks on average. T2 omits more breaks than T1 (86 vs. 69), as it tries to minimize insertion errors. The breaks it omits are judged necessary by 42% of the evaluators. The reference labeller has only 3 wrong breaks which no evaluator has selected. This confirms that there is not only one correct way to insert breaks in a text, and that the agreement with our reference information is not complete. T2 has fewer wrong breaks than T1: it has been built to minimize insertion errors, so it inserts fewer correct breaks, but also fewer wrong breaks.

6

Discussion After analysing the results of these break location prediction experiments, a first conclusion is that placing breaks is a speaker dependent task. There is only partial agreement among speakers about the places where breaks have to be inserted. Some break positions are very clear and the agreement is almost complete (in our test 21% of the breaks were marked by more than 80% of the evaluators), but some others have a strong dependency on the evaluator (19% of the breaks marked were selected by less than 20% of the evaluators).

Another important conclusion is that there are some places where a break cannot be made: insertion errors are more critical than deletion errors. The T2 algorithm, which places fewer breaks than the T1 algorithm, produces fewer wrong breaks. One break wrongly placed causes a bad subjective impression that degrades notably the quality of the synthetic speech signal. Although T1 has better objective results, taking into account the number of wrong breaks, in practice it may be better to use T2.

Among the considered morpho-syntactical features, information about the syntactic function of the word is the more determinant one. In standard Basque prepositional functions are realized by case suffixes inside word-forms and there are fewer function words than in other languages. POS information is not enough for the prediction of break locations and grammatical case has also to be taken into account.

It is extremely difficult to compare the results achieved in break location prediction with other similar works. On the one hand the databases used are very different in most cases. The treatment of punctuation information is also very different in each study: Some works include all the punctuation marks which have a great correlation with breaks and make the prediction task easier. Other works completely eliminate punctuation marks. In our case, only sentence internal punctuation marks have been eliminated, due to the considerations about the S2ST framework where our TTS system must work. Another issue that makes the comparison of the results more difficult are the measures presented to evaluate the performance of the algorithms. However, just for reference, we briefly state below some recent results in the prediction of break location. F score has been calculated when possible, if not directly provided by the authors. [START_REF] Read | Stochastic and syntactic techniques for predicting phrase breaks[END_REF] achieve a best F of 80.19% with reduced POS tag set and syntactic information over MARSEC corpus. The predicting technique applied in this case was decision trees. [START_REF] Bell | Adaptation of prosodic phrasing models[END_REF] working on the same MARSEC corpus got an F of 83.3% and an F of 82.1% on the Boston Radio Corpus, applying HMM in both cases. [START_REF] Pfitzinger | Text-based and Signal-based Prediction of Break Indices and Pause Durations[END_REF] dealt with this problem in German. They used the IMS Radio News Corpus and predicted the ToBI break indices with a global F of 87.72%. [START_REF] Yi | Phrase break prediction using logistic generalized linear model[END_REF] applied a Logistic Generalised Linear Model to Mandarin assuming that phrase breaks follow a Bernoulli distribution. They got an F of 62.32% without any information about punctuation. [START_REF] Kim | Incorporating second-order information into two-step major phrase break prediction for Korean[END_REF] worked for Korean which is an agglutinative language. They used a General Maximum Entropy Model with POS and global and second order features such as distance between current junction and previous or next phrase break. Their model has an F of 79.6%, without using information about punctuation. Another work dealing with break prediction for Korean is the one presented in [START_REF] Yoon | A prosodic phrasing model for a Korean text-to-speech synthesis system[END_REF]. He built a CART with POS and syntactic features and achieved an F of 71.3% for intonational break prediction, using information about punctuation. [START_REF] Zervas | Experimental Evaluation of Tree-Based Algorithms for Intonational Breaks Representation[END_REF] use also decision trees and POS information and have a κ > 0.75 using a set of 11 POS labels. They worked for Modern Greek which is an inflected language. Oparin (2005) uses a rule based approach for the prediction of break location in Russian (an inflected language) with an F of 77.8%. [START_REF] Ingulfsen | Influence of syntax on prosodic boundary prediction[END_REF] using punctuation marks, POS and link grammar achieved an F of 79.4% over Boston Radio Corpus. [START_REF] Cox | A discriminative approach to phrase break modelling[END_REF] attained an F of 81.2% in the same Boston Radio Corpus, considering POS information and some other attributes derived with a sentence parser. In this work, Generalised Probabilistic Descent (GPD) was applied to a maximum entropy classifier.

Only most recent published works had been referred, but of course there have been many more works, which reflects the interest and importance of the problem.

Among the continuing lines of this work is the consideration of different window lengths and centrings for the labels considered in the prediction. Although most studies use a window centred in the word under study, some works indicate that considering more tags before the current word than after it can improve the results [START_REF] Maragoudakis | A Data-Driven Framework for Intonational Phrase Break Prediction[END_REF][START_REF] Schmid | New statistical methods for phrase break prediction[END_REF][START_REF] Sun | Intonational Phrase Break Prediction Using Decision Tree and N-Gram Model[END_REF].

Finally, the subjective evaluation process could be improved by allowing the quantisation of the correctness of the inserted breaks. This would simplify the interpretation of the results. 5. Syntactic labels considered Table 6. Labels about the grouping of words in syntagms Table 7. Sakoda's contingency coefficient between the feature indicated in the row title and the feature indicated in the column title Table 8. Results for the trees using only one type of information Table 9. Results for the baseline system and the prediction trees built Table 10. Result of the subjective evaluation

FIGURE LIST Fig 1 .

 LIST1 FIGURE LIST Fig 1. Histogram of the number of syllables between breaks compared with the histogram of the number of syllables in the sentence. Fig 2. Best κ and F results of the trees built with different number of types of information.

Fig 1 .

 1 Fig 1. Histogram of the number of syllables between breaks compared with the histogram of the number of syllables in the sentence.

TABLE LIST Table 1 .

 LIST1 Characteristics of the corpus Table 2. Category or Part of Speech (POS) tags Table 3. Subcategories considered Table 4. Grammatical cases considered Table

  Fig 2. Best and F results of the trees built with different number of types of information.

	Table 1 Characteristics of the corpus		kappa
	Feature		Value
	0.52 # of words			15 867
	# of breaks			3 589
	# of sentences # of intra-sentence breaks		1 470 + SYNTAGM 2 119
	0.51 % of function words % of breaks	+ CASE	17.5% 20.7%		+ SUBCATEGORY
		+ POS
	0.50				0.518
				0.509
			0.504
	0.49	0.498
						0.493
		SYNTACTIC
	0.48	FUNCTION
		1	2	3	4	5	# of types
							of features
				F
	0.77
	0.76		+ CASE	+ SYNTAGM
	0.75					+ SUBCAT
	0.74
	0.73
	0.72	+ POS	0.749	0.753	0.757
	0.71					0.738
	0.70	0.713 SYNTACTIC
	0.69	FUNCTION
		1	2	3	4	5	# of types
							of features

Table 2

 2 Category or Part of Speech (POS) tags

	Label	Description	Example		%followed by break
	ADB	Adverb	geroago	(later)	27.8%
	ADI	Main verb	eman	(give)	17.0%
	ADJ	Adjective	txiki	(little)	18.5%
	ADL	Auxiliary verb	du	(has)	49.4%
	ADT	Synthetic verb	dago	(is)	44.5%
	BST	Others	ohi	(use to)	0.0%
	DET	Determiner	hau	(this)	16.5%
	ERL	Relation suffixes	badirudi	(it seems)	54.8%
	IOR	Pronoun	gu	(we)	3.7%
	ITJ	Interjection	kaixo	(hi)	66.7%
	IZE	Noun	ordu	(hour)	18.6%
	LAB	Abbreviation	etab	(etc)	100.0%
	LOT	Conjunction	eta	(and)	18.0%
	PRT	Particle	omen	(it seems)	11.4%
	PUNT	Punctuation mark	.	(.)	99.2%
	SIG	Acronym	AEB	(USA)	8.3%

Table 3

 3

	Subcategories considered
	Label	Description	Example		%followed by break
	ARR	Common noun	liburu	(book)	19.8%
	GAL	Question pronoun/adjective	noiz	(when)	9.8%
	ADP	Periphrastic verb	behar	(have to)	6.3%
	DZG	Indeterminate quantifier	zenbait	(some)	11.4%
	DZH	Cardinal	hiru	(three)	13.6%
	ELK	Reciprocal pronoun	elkar	(one to another)	0.0%
	FAK	Factitive verb	arazi	(make somebody do something)	0.0%
	IGB	Indeterminate pronoun	norbait	(someone)	0.0%
	IZB	Person proper noun	Jon	(John)	16.5%
	IZO	Qualifying adjective	berria	(new)	21.0%
	JNT	Coordinate	eta	(and)	2.4%
	LIB	Place proper noun	Bilbao	(Bilbao)	20.3%
	LOK	Connective	hala ere	(however)	71.7%
	ORD	Ordinal	laugarren	(fourth)	6.9%
	ORO	General numeral	guzti	(all)	26.8%
	SIN	Simple verb	ekarri	(take)	17.1%
	ZIU	Security adverb	ote	(maybe)	0.0%

Table 6

 6 Labels about the grouping of words in syntagms

	LABEL	Description	%followed by break
	BEG	Beginning of the group	1.2%
	CEN	Inside a group	3.9%
	END	End of the group	24.5%
	UNI	Group of only one word	18.5%

Table 8

 8 Results for the trees using only one type of information

	TYPE OF INFORMATION	S	κ	P	R	F
	POS	89.2%	0.429	0.948	0.454	0.614
	SUBCATEGORY	89.8%	0.461	0.872	0.540	0.667
	CASE	85.9%	0.253	0.795	0.341	0.478
	SYNTACTIC FUNCTION		0.498	0.833	0.623	0.713
	SYNTAGM	87.6%	0.346	0.830	0.435	0.571

Table 9

 9 Results for the baseline system and the prediction trees built

	SYSTEM	S	κ	P	R	F
		89.6%	0.402	1.000	0.402	0.574
	T1	90.9%	0.518	0.764	0.750	0.757
	T2	90.7%	0.511	0.828	0.644	0.725

The first line shows the sentence in Basque. The second and third lines group the words in Basque and English in order to have a one-to-one correspondence between each group in the two languages. The forth line reorders the words in English to obtain a correct English sentence. The breaks are not marked in the English version, due to the differences in word ordering between both languages.

IXA research group of the University of the Basque Country (http://ixa.si.ehu.es/)

Acknowledgements

We are grateful to Nerea Ezeiza and the IXA research group for the automatic morphological and syntactic labelling of the corpus and to Iñaki Gaminde for the manual labelling of the breaks.

The authors would also like to thank all the people that took part in the subjective evaluation. This work has been partially founded by Spanish Ministry of Education and Sciences under grant TEC2006-13694-C03-02 (AVIVAVOZ project, http://www.avivavoz.es/) and by Basque Government under grant IE06-185 (ANHITZ project, http://www.anhitz.com/).