Alicia Pérez
email: alicia.perez@ehu.es

M Inés Torres

Francisco Casacuberta

Joining linguistic and statistical methods for Spanish-to-Basque speech translation

Keywords: spoken language translation, stochastic transducer, phrase-based translation model, morpho-syntactic knowledge modeling, bilingual resources

The goal of this work is to develop a text and speech translation system from Spanish to Basque. This pair of languages shows quite odd characteristics as they differ extraordinarily in both morphology and syntax, thus, attractive challenges in machine translation are involved. Nevertheless, since both languages share official status in the Basque Country, the underlying motivation is not only academic but also practical.

Finite-state transducers were adopted as basic translation models. The main contribution of this work involves the study of several techniques to improve probabilistic finite-state transducers by means of additional linguistic knowledge. Two methods to cope with both linguistics and statistics were proposed. The first one performed a morphological analysis in an attempt to benefit from atomic meaningful units when it comes to rendering the meaning from one language to the other. The second approach aimed at clustering words according to their syntactic role and used such phrases as translation unit. From the latter approach phrase-based finite-state transducers arose as a natural extension of classical ones.

The models were assessed under a restricted domain task, very repetitive and with a small vocabulary. Experimental results shown that both morphological and syntactical approaches outperformed the baseline under different test sets and architectures for speech translation.

Introduction

Speech translation represents nowadays a challenge in natural language processing due to the difficulties of combining speech and translation technologies. The so called statistical framework is, without doubt, a very promising approach for speech and translation modeling. Nevertheless, this approach requires large training material. Furthermore, the scarcity of available linguistic resources associated with minority languages as Basque, Catalan or Galician, has to be faced in advance. The work presented in this paper focuses on Spanish to Basque speech and text translation. Thus, the first stage of this work consisted of the generation of linguistic resources (corpus and tools) for Basque (Pérez et al., 2006b).

Spanish and Basque languages differ significantly in both morphology and syntax (for further details see appendix A). Hence, specific problems have to be faced: on the one hand, Basque is a highly inflected language with 17 cases (for a matter of comparison, in German there are 4 grammatical cases, 7 in Czech and 15 in Finnish); on the other hand, the typical syntactic construction leads to long distance relationships between Spanish and Basque. These characteristics are, somehow, depicted in Fig. 1. The mentioned differences do not occur, to this extent, between Spanish and other Iberian languages, such as Catalan, Galician or Portuguese. Therefore, translation from Spanish into Basque not only does it exhibit academic interest, but also represents a real necessity since both languages are co-official for the 2.5 million inhabitants of the Basque Country. Other tools that aimed at translating text from Spanish into Basque have been previously implemented in the literature such as Matxin [START_REF] Alegria | Transfer-based mt from spanish into basque: Reusability, standardization and open source[END_REF] within the project Opentrad [START_REF] Corbí-Bellot | An open-source shallow-transfer machine translation engine for the romance languages of spain[END_REF]. It makes use of a transfer approach to cope with text translation. Alternatively, this work deals with statistical speech and text translation and, to the authors' knowledge, it is the first approach in the literature related to speech translation between this pair of languages.

Within the statistical framework, we deal with finite-state models, which have been extensively applied to many fields of natural language processing such as language or phonology modeling. They have also been successfully introduced for speech translation within restricted domains [START_REF] Vidal | Finite-state speech-to-speech translation[END_REF][START_REF] Bangalore | Stochastic finite-state models for spoken language machine translation[END_REF]. There are different approaches that cope with machine translation using finite state models. On the one hand, [START_REF] Bangalore | A finite-state approach to machine translation[END_REF] proposed a two step translation process where first, a mapping from source to target language allowed the lexical selection, and then, a lexical reordering based on a tree-structure arranged the output so that it was grammatically correct. Other works [START_REF] Knight | Translation with finite-state devices[END_REF][START_REF] Kumar | A weighted finite state transducer translation template model for statistical machine translation[END_REF] formulated a generative source-channel model influenced by the IBM-Models [START_REF] Brown | The mathematics of statistical machine translation: Parameter estimation[END_REF] but implemented with weighted finite-state transducers. Our work, instead, focuses on a unique model that copes, at the same time, with both transference and, to some extent reordering. It defines a joint probability distribution over all possible translations. This approach leads to the integration of acoustic and translation models in a unique model that can be inferred from samples allowing a single and fast decoding. This integration presents certain similarity with the composition of several finitestate models proposed for speech recognition purposes [START_REF] Caseiro | Transducer composition for onthe-fly lexicon and language model integration[END_REF].

Taking the aforementioned finite-state models as a baseline, we aimed at improving their performance. Current trends in machine translation suggest that statistical methods could be benefited from linguistic knowledge (Och et al., 2003). Even though there have already been some attempts at exploiting linguistic resources within statistical methods [START_REF] Nießen | Morpho-syntactic analysis for reordering in statistical machine translation[END_REF][START_REF] Collins | Clause restructuring for statistical machine translation[END_REF][START_REF] De Gispert | Linguistic knowledge in statistical phrase-based word alignment[END_REF], this issue is still an open problem. In this work, morphologic and syntactic knowledge sources have been tightly introduced within statistical translation models. In this framework, two different approaches were proposed and assessed with a series of experimental results in a limited domain but highly practical application task. On the one hand, a morphological approach attempted at implementing the meaning-transfer models using a categorized target language and then completing the target lexical choice. On the other hand, a syntactical approach tried to build the transducer taking syntactic phrases as translation unit instead of running words. In fact, the state of the art in machine translation suggests the use of phrases as translation unit instead of words [START_REF] Koehn | Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding[END_REF]. As it will be shown, the proposed phrase-based SFST approach is related with a monotonic approach of the commonly used phrase-based models.

All in all, the aim of this paper is to make progress within the field of speech technologies for under-resourced languages as it is the case of Basque. The adopted strategy consists of reinforcing statistical methods by including linguistic knowledge within the finite-state framework. Since the pair of languages under study differ extraordinarily in what word ordering and agglutination concerns (as mentioned in appendix A), new strategies were to be explored. As a by-product, a new phrase-based approach has been formulated and implemented.

The organization of this paper is as follows: the statistical framework is faced in section 2, where speech translation problem is tackled with two different architectures. Stochastic finite-state transducers are developed in section 3, where decoding and learning problems are addressed, in addition, as an extension of those models a phrase-based approach for finite-state transducers is presented. Then, we propose to enrich the mentioned statistical models making use of linguistic features. In particular two approaches are explored in section 4. These approaches have been evaluated within the task described in section 5. Experimental results are given in section 6 and finally, some conclusions and guidelines for future work. For further details with regard to Basque and Spanish languages turn to appendix A.

Speech translation

The goal of the statistical speech translation is to search for the likeliest target language string t, given the acoustic representation x of some source language string.

t = arg max t P (t|x) (1)
The transcription of the speech into text, is an unknown string s in the source language that might be considered a hidden variable.

Let us assume that the probability of the acoustic signal related to the utterance in the source language has no dependency on the target string once the source string is known i.e. P (x|t, s) is independent of t. Hence, eq. ( 3) can be rewritten as: t = arg max t s P (t, s)P (x|s) (4)

In practice, the sum over all possible source strings in eq. ( 4) can be approximated by the maximum term involved. Typically, eq. ( 5) is implemented in a sub-optimal way, by means of a speech recognizer and a text-to-text translation system in a decoupled architecture. Taking into account that P (t, s) = P (t|s)P (s), this approach offers the translation of the speech transcription as follows:

(1) Given the acoustic representation x, find its expected text transcription: The serial architecture is the most widely used approach due to the fact that it is independent of the sort of translation paradigm used as both the speech recognition and the translation system are decoupled. Unfortunately, translation models are quite sensitive to input-errors [START_REF] Sarikaya | The impact of ASR on speech-to-speech translation performance[END_REF], thus the translation of two slightly distinct source strings may differ significantly. In short, the approach of eq. ( 5) by eq. ( 7) recurs to a strong assumption.

In order to achieve a tighter integration between speech recognition and translation stages, some effort have been made in the literature by using n-best lists [START_REF] Quan | Integrated n-best re-ranking for spoken language translation[END_REF], word-lattices [START_REF] Saleem | Using Word Lattice Information for a Tighter Coupling in Speech Translation Systems[END_REF] or confusion networks [START_REF] Bertoldi | Speech translation by confusion network decoding[END_REF]. Furthermore, joint probability in eq. ( 5) can be naturally implemented with finite-state transducers as it is well known. In addition, acoustic and translation finite-state models can be efficiently composed in an integrated architecture (Casacuberta et al., 2004). In practice (as will be shown in section 6.1), integrated architecture works as a speech recognition system that makes use of a translation model instead of the usual language model. The same acoustic models, typically hidden Markov models, can be used for either speech recognition or speech translation. Our attention is thus focussed on language modeling. In fact, the translation model (under finite-state transducer methodology) involves two language models: the source language model, is the input projection of the transducer, and the target language model the output projection. On the whole, the composition seems to be a robust technique to hierarchically integrate knowledge-sources of different complexity or depth level in either speech recognition [START_REF] Pereira | Speech Recognition by Composition of Weighted Finite Automata[END_REF][START_REF] Caseiro | Transducer composition for onthe-fly lexicon and language model integration[END_REF] or speech translation (Casacuberta et al., 2004).

In addition, there are efficient algorithms to carry out on the fly integration of these sort of models at decoding time [START_REF] Caseiro | A specialized on-the-fly algorithm for lexicon and language model composition[END_REF].

Stochastic Finite-State Transducers

Finite-state transducers are versatile models that count on thoroughly studied efficient implementations for training [START_REF] Casacuberta | Learning finite-state models for machine translation[END_REF] and decoding [START_REF] Mehryar Mohri | AT&T FSM LibraryTM Finite-State Machine Library[END_REF]. Definition and layout for probabilistic finite-state machines (automata and transducers) were comprehensively described in (Vidal et al., 2005a,b), and so we are going to follow that formalism. Though the formal definition is reported next, as an introductory notion a stochastic finite-state transducer (SFST) might be roughly described as a finite-state machine where each transition, labeled with an input/output pair, has associated a probability to occur. That is, a Mealy machine with a set of probability distributions involved.

Definition

An SFST is tuple T = Σ, ∆, Q, q 0 , R, F, P , where:

Σ is a finite set of input symbols (source vocabulary); ∆ is a finite set of output symbols (target vocabulary); Q is a finite set of states;

q 0 ∈ Q is the initial state; R ⊆ Q × Σ × ∆ * × Q is a set
of transitions such as (q, s, t, q ), which is a transition from the state q to the state q , with the source symbol s and producing the substring t; P : R → [0, 1] is a transition probability distribution;

F : Q → [0, 1] is a final state probability distribution;
The probability distributions satisfy the stochastic constraint:

∀q ∈ Q F (q) + s, t,q P (q, s, t, q ) = 1 (8)

Decoding

The goal of statistical machine translation is to find the target language string t that better matches the source string s. Within SFSTs, the analysis of a source string is carried out by exploring all the possible translation forms. A translation form, d(s, t), is a sequence of transitions in an SFST, T . That is, a path compatible with both the source and the target strings:

d(s, t) : (q 0 , s 1 , t1 , q 1 )(q 1 , s 2 , t2 , q 2 ) • • • (q J-1 , s J , tJ , q J ) where s = s 1 s 2 . . . s J is a sequence of source symbols and t = t 1 t 2 . . .

t I = t1 t2 . . . tJ is a sequence of target substrings (|s| = J, |t| = J j=1 | tj | = I). Ac-
cording to the SFST model, T , the probability associated with a translation form is the following one:

P T (d(s, t)) = F (q J ) J j=1
P (q j-1 , s j , tj , q j ) (9)

Therefore, the probability of (s, t) to be a translation pair, is the sum of the probability of all the possible translation forms compatible with that pair.

P T (s, t) = d(s,t) P T (d(s, t)) (10)
As a result, P (s, t), the distribution involved in eq. ( 5) can be estimated with an SFST model:

P (s, t) ≈ P T (s, t) = d(s,t) P T (d(s, t)) (11)
The resolution of the eq. ( 11) has proved to be a hard computational problem [START_REF] Casacuberta | Linguistic decoding is a difficult computational problem[END_REF], but it can be efficiently computed by the maximum approximation, which replaces the sum by the maximum.

P T (s, t) ≈ max d(s,t) P T (d(s, t)) (12)
Under this maximum approach Viterbi algorithm can be used to find the best sequence of states through the SFST given the input string. The translation is obtained by concatenating the output substrings through the optimal path.

Learning

Stochastic finite state transducers can be automatically learnt from bilingual corpora by efficient algorithms [START_REF] Casacuberta | Learning finite-state models for machine translation[END_REF]. The goal is to determine the topological parameters and probabilistic distributions defining a specific translation model for the task under consideration. Most common inference approaches [START_REF] Casacuberta | Inference of finite-state transducers by using regular grammars and morphisms[END_REF][START_REF] Bangalore | Stochastic finite-state models for spoken language machine translation[END_REF][START_REF] Matusov | Efficient statistical machine translation with constrained reordering[END_REF] lead up to an intermediate bilingual representation of each pair of sentences in the training data. This representation is obtained by segmenting, under different constraints, the pair of sentences into a string of bilingual phrases:

(s 1 , t1 )(s 2 , t2 ) • • • (s m , tm ).
In this work GIATI (Grammar Inference and Alignments for Transducers Inference) methodology is used to learn the translation models. This approach restricts the length of the source substrings to one and allows the length of the target substrings to be zero. GIATI was exhaustively described by (Casacuberta and Vidal, 2004) and it can be summarized as follows:

(1) For each bilingual pair (s, t) = (s J 1 , t I 1 ) from the training corpus, find a monotonic segmentation (s J 1 , tJ 1 ). Thereby, assign an output sequence of zero or more words to each input word, leading to the so called extended corpus (the intermediate bilingual representation of each training pair in terms of a single sequence of bilingual tokens, as previously mentioned). To do so, direct and inverse statistical alignments extracted with Giza++ free toolkit [START_REF] Och | A systematic comparison of various statistical alignment models[END_REF] were considered in this work. Each extended string, z ∈ (Σ × ∆ * ) * , satisfies:

z = (s 1 , t1 )(s 2 , t2 ) . . . (s J , tJ ) where J j=1 | tj | = I
(2) Then, infer a stochastic regular grammar. We promote the use of ktestable in the strict sense (k-TSS) grammars, which are a subset of regular grammars. Hence, the corresponding k-TSS stochastic finite-state automaton (SFSA) can be automatically learnt from positive samples making use of efficient algorithms based on a maximum likelihood approach [START_REF] García | Inference of k-testable languages in the strict sense and application to syntactic pattern recognition[END_REF]. k-TSS language models are considered to be the syntactic approach of the well known n-gram models. However, the syntactic approach allows to integrate K k-TSS models (ranging K from 1 to k) along with smoothing in a unique SFSA under a hierarchical back-off structure [START_REF] Torres | k-TSS language models in speech recognition systems[END_REF]. Smoothing is of interest in order to prevent the model from assigning null probabilities due to data sparseness at training stage [START_REF] Jelinek | Statistical Methods for Speech Recognition[END_REF].

(3) Finally, split the output sequence from the input word on each edge of the automaton, getting, in this way, the finite state transducer. That is, if a transition in the inferred automaton is like (q i , (s, t), q j ), then, there is an equivalent edge in the transducer with the input symbol s and the output substring t. The probability distributions and the topology learnt in the 2nd step remain unchanged.

A novel implementation of monotonic phrase-based models

As shown in this section, one of the contributions of this article is the alternative formulation of a monotonic phrase-based (PB) approach turning to the finite-state (FS) framework. This approach puts together the improvements in translation quality related to PB approach and the flexibility and speed associated with FS models [START_REF] González | Phrase-based finite state models[END_REF][START_REF] Pérez | Speech translation with phrase based stochastic finite-state transducers[END_REF]. Furthermore, this approach makes it possible a tight integration of both translation and acoustic models following the methodology proposed for automatic speech recognition in [START_REF] Caseiro | A specialized on-the-fly algorithm for lexicon and language model composition[END_REF]. In addition, this formulation allows to include morphologic information within the translation model as it will be shown latter.

It is usually difficult to capture relationships between languages when longdistance alignments are involved. These distances decrease taking as alignment unit the phrase instead of the word. Furthermore, segmentations might become monotonic, and these are, in fact, the sort of relationships that stochastic finite-state transducers are good at. The mentioned issues are depicted in the Fig. 2 through a pair of sentences in Spanish and Basque whose translation into English is "during the morning skies will remain mostly cloudy".

A monotonic approach usually involves a faster decoding than a non-monotonic one, and indeed time efficiency is an essential issue in speech translation. In what performance concerns, word-based monotonic approach has shown to be a good approach to deal with similar languages. Furthermore, phrase-based monotonic approach shows a wider application scope, thus, it seems to be useful even between languages with long-distance alignments as is the case of Spanish and Basque (at least for tasks such as the one considered in this work). Therefore, phrase-based SFSTs could improve the performance of previous SFSTs when long-distance reorderings are involved.

The phrase-based approach for finite-state transducer was recently introduced in the literature. Let us note that it was the work (Casacuberta and Vidal, 2004) which set a precedent in this investigation line. Alternatively, in (Ku- In [START_REF] Zhou | Constrained Phrase-based Translation Using Weighted Finite State Transducer[END_REF] the translation process was decomposed in terms of several probability distributions (specifically: output language model, permutation model, fertility model, NULL insertion model) and all of them were implemented as WFSTs. Both approaches were carried out by a composition of constituent transducers. Some assumptions were taken in order to infer several components from parallel training data. For instance, given a source sentence, uniform probability distribution over all the compatible segmentations was assumed, as well as for the number of segments to be generated. Alternatively, in this work, for a given training pair only one segmentation is taken into account, and thereby a single SFST is inferred. There is no need of other intermediate models since this model is self contained and copes with both the meaning transference and the arrangement of the output string at the same time. An intrinsic restriction on our approach, however, is its monotonic nature (as earlier mentioned).

Here a natural extension of the transducers described in previous section is proposed so that they cope with phrases instead of running words. To do so, the set of transitions (see section 3.1) is redefined as: R ⊆ Q × Σ + × ∆ * × Q, where such a transition, (q, s, t, q ), links the state q to the q , with the source sub-string s and producing the target substring t. In what the inference of such a model regards, first of all, let us extract the set of all the possible segmentations S for a given pair (s, t). The segmentation may be related to either statistically or linguistically motivated phrases without any restriction. A specific segmentation for a given pair, denoted as σ ∈ S(s, t), can be defined by the number of segments to be made along with the limits of the segments in the source and the target sentences respectively. In this context, segmentations can be introduced as hidden variables within the joint probability model.

P (s, t) = σ P (s, t, σ) = σ P (σ) • P (s, t|σ) (13)
Two probability distributions have to be modeled. On the one hand, let us assume to be uniform the segmentation distribution. On the one hand, let us assume a uniform distribution over all possible segmentations. In this context a given segmentation σ ∈ S(s, t), splits the source and target strings into a number of substrings (m σ ) and indeed it states the limits of each substring:

σ = {s 1 s2 • • • smσ ; t1 t2 • • • tmσ } where | si | > 0 and | ti | ≥ 0.
Thus, the probability of a given bilingual pair along with a specific segmentation could be expressed under the common approach for n-gram models [START_REF] Jelinek | Statistical Methods for Speech Recognition[END_REF] taking bilingual phrases as language-unit (instead of the typical monolingual running words):

P (s, t|σ) mσ i=1 P ((s i , ti )|(s i-n+1 , ti-n+1 ), . . . , (s i-1 , ti-1 )) ( 14
)
Once the decoding technique under phrase-based SFST approach has been described, we are going to proceed to describe the learning procedure. This methodology requires to have the training corpus segmented in advance (the segmentation technique used in this work will be presented in section 4.2). Then an SFST is learnt taking the segments as vocabulary units instead of running words. This transducer would be an acceptor of segmented strings, that is, strings built by means of sequences of those segments. Nevertheless, at decoding time the source sentence is not built in terms of segments but in terms of running words. Therefore, the previous model has to be generalized so that the analysis is performed in terms of words. As a segment can be unambiguously converted into words, it is possible to define an equivalence relationship between each segment and a left-to-right finite-state model at word level. Thus, we just proceeded to integrate those word-by-word models within each edge of the phrase based transducer. In brief, the phrase-based transducers we put forward in [START_REF] González | Phrase-based finite state models[END_REF][START_REF] Pérez | Speech translation with phrase based stochastic finite-state transducers[END_REF], kept up with a monotonic approach of the widely used statistical approaches [START_REF] Koehn | Statistical phrase-based translation[END_REF].

The final transducer is more restrictive than the one built taking words as units. Bear in mind that with a given history and any input symbol there always exists at least one path in the smoothed transducer with non-zero probability compatible with that symbol. In a smoothed SFST based on words, given a history any word is likely to appear, and therefore, any sequence of words will have a non-zero probability. In the same way, in a smoothed phrasebased SFST any phrase amongst those in the vocabulary is likely to appear, but not any string of words, only the strings compatible with the phrase-based model will have a non-zero probability.

4 On the use of linguistics within statistical models

Translation between Spanish and Basque presents specific problems, mainly due to the high differences in both morphology and syntax (as mentioned in appendix A). In this work two methods based on linguistic knowledge integration within finite-state transducers were explored.

Morphological approach

This method attempts at analyzing the morphology of the morphologically richer language (Basque, in this case) and split the words into basic lexically meaningful units (lexemes and morphemes). On account of this, each unit could have a closer counterpart in the other language (Spanish, in this case), and then, a more accurate translations could be obtained. In [START_REF] Hirsimäki | Unlimited vocabulary speech recognition with morph language models applied to Finnish[END_REF] an unsupervised algorithm for discovering word fragments was presented and evaluated in speech recognition for Finnish. The language model based on those units offered remarkable improvements over the traditional word-based language model. In [START_REF] Goldwater | Improving statistical mt through morphological analysis[END_REF] an attempt was made at homogenizing Czech and English on the basis of morphological characteristics in order to improve statistical translation models. Bearing in mind that Finish and Czech are highly inflected languages, it may be of interest to explore similar approaches for speech translation into Basque. In fact, previous works [START_REF] Agirre | Uso de información morfológica en el alineamiento español-euskara[END_REF][START_REF] Labaka | Comparing rulebased and data-driven approaches to spanish-to-basque machine translation[END_REF] have studied the benefits of including morphological information in order to improve statistical alignments between Spanish and Basque.

Since the morphemes are the atomic meaningful units no further divisible into smaller units, the statistical translation model could be learnt in terms of such simple units instead of running words. In fact, the number of different morphemes in the studied Basque corpus is around 40% smaller than the number of running words. Then it might be expected that more significant statistics could be collected amongst morphemes rather than amongst words.

To be more specific, the approach was conceived to work in two sequential steps, being each one implemented with an SFST. The first one would render the meaning from the source to the target without taking the grammatical cases into account. That is, this first step would provide a rough representation of the target string. The second step would select the appropriate word-forms in such a way that the previous string made sense in the target grammar. The aim of this approach is to decompose the translation process into two problems simpler than the original one. In this case, morphology comes to the aid of meaning rendering.

For the first step, the SFST was built taking source (Spanish) words as input vocabulary and restricting the target (Basque) vocabulary to basic morphological units with lexical meaning. That is, first of all we aimed at transferring the meaning by assembling a translation model from Spanish into lemmatized Basque. In short, all inflected variants of a word-form share the same lemma.

Since lemmatization plays the same role as categorization, other kind categories might also be useful in what comes to rendering the meaning. This process is expressed through eq. ( 15), where c stands for the involved sequence of lemmas (or categories, in general). In this work the joint probability involved is modelled with an SFST T 1 , that is, P (c, s)

≈ P T 1 (c, s). c = arg max c P (c|s) = arg max c P (c, s) (15)
The first SFST was experimentally proved to be an accurate statistically motivated transfer model, but it still needs for another stage to seek the proper declension cases of the given lemmas.

For the second step, a lexical choice has to be made on the basis of both the source string and the categorized target, as expressed through eq. ( 16).

t = arg max t P (t|s, c) (16)
Given the lemmatized representation obtained in the fist step, we did not make further use of the source string. Thus, eq. ( 16) was approximated by the following one. Once again, the joint probability was modeled with an SFST T 2 , that is,

P (t, c) ≈ P T 2 (t, c) t ≈ arg max t P (t| c) = arg max t P (t, c) (17)
It has to be noted that the underlying morphologic analysis was performed by Ametzagaiña group1 a non-profit organization working on I+D. For further details on this approach and exhaustive experimental results turn to (Pérez et al., 2006a). On the following, this morphology-based approach will be referred to as MB.

Example: the steps involved in the MB approach are here summarized with a sentence extracted from the corpus.

• Given the source sentence (in Spanish): "por la mañana los cielos permanecerán muy nubosos" whose counterpart in English is "during the morning skies will remain mostly cloudy" • The first transducer, T 1 , gives as a result the translation of the source sentence into lemmatised Basque: "goizalde zeru oso hodei mantendu izan" word by word, an equivalent in English would be "morning sky remain most cloud " • The second stage has to find the proper word-forms for the given lemmas.

For the task under consideration there are 4 words in the training corpus that share the lemma "goizalde" (the first token within the lemmatised Basque string), to be precise: goizalde, goizaldean, goizaldera and goizaldez.

That is, there are 4 words compatible with that lemma. The selection of the most likely word-form is carried out taking into account both the whole lemmatised sentence and an output language model (modelled by T 2 ). "goizaldean zerua oso hodeitsu mantenduko da"

Approach based on syntactic phrases

The general framework over phrase-based transducers (described in section 3.4) entails a segmentation technique. In our case the segmentation was syntactically motivated and accomplished by Ametzagaiña group. In this section the phrase extraction methodology is briefly described. The procedure involves a parser for each language under study. That is, the phrases are selected according to monolingual criteria. Even though for Spanish there are open-source analysers [START_REF] Carreras | Freeling: An open-source suite of language analyzers[END_REF], for Basque, however, there is none and it had to be developed.

The phrase identification was linguistically motivated and automatically accomplished following the steps listed below. Let us note that this is a domain independent procedure:

• First, a morphologic parsing allows to assign either one or more tags to each word within the corpus. These tags include information about linguistic categories such as declension case, number, definiteness, tense, etc. Besides, the stem and the morphemes are identified. In both Spanish and Basque they were defined around 45 classes with several sub-classes each. Both languages share the majority of them. Nevertheless, there are some tags that are just related to one of the two languages, ergative 2 case for instance, which does not exist in Spanish. Apart from this, let us emphasize that in this step ambiguity is not removed. As a result, more than one tag-set can be assigned to each word. • Next, ambiguities have to be removed. For each word, the most likely category-set is chosen so that the words of the sentence shared syntactically compatible categories according to predefined rules for each language. Around 15 rules were defined for each language over their corresponding category set so as to get a high coverage of word-forms. The order in which these items might appear within a specific syntactic function is described as well.

• Finally, linguistic phrases can be identified under an elementary criteria: recursively group all the words that share the same syntactic function whenever the frequency of that group (phrase) in the corpus exceeds a threshold.

On the first iteration of this algorithm just noun and verb phrases are distinguished, that is the parsing is quite shallow. Then, regular expressions (such as dates) are also taken into account so as to help the selection of an error-free category. As the iterations go ahead, more and more precise tokens are identified, such as composed stems, periphrastic verbs, etc. along with their syntactic role within the sentence they belong to.

Example: Throughout this example the aforementioned steps are going to be developed taking the following Spanish sentence as input: "las rachas de viento pueden superar una velocidad de 90 Km/h" (note that this is the example shown in Fig. 1). In the first step the words are analyzed separately without taking either left or right context into account. This analysis entails a source of ambiguities. For instance, the word "viento" in the sentence was given two tag-sets as summarized in Fig. 3. According to these results the word "viento" has two different lemmas, "viento" and "ventar". The grammar category associated to the former is noun, gender male and number singular. While the latter, "ventar", is a verb in indicative mode and present tense associated with the first person. The second step takes as input all the category-sets related to all the words within a sentence and by a set of rules defined over the tags, a segmentation is obtained. The output of this procedure over the sentences shown in Fig. 1 leads to the segmentation of Fig. 4. The phrases in one language usually have their counterpart in the other language as a phrase as well, not in a monotonic fashion, however. In addition, it might help to the alignments, since usually, the phrases playing the same role represent related entities in both languages. In this example, the verb phrase as a whole, even being in different places for each language, are tightly connected.

A manual inspection shown that more realistic correspondences could be described between these linguistically motivated phrases rather than between running words. For instance, the phrases "las rachas de viento" and "haize boladek", in Spanish and Basque respectively, are both noun phrases and mean the strong gusts. The same happens with the verb phrase, both the Spanish and the Basque phrases are translation pairs (meaning may exceed ). While in the last case, a couple of phrases in Spanish refers to a single one in Basque, being "una velocidad de 90 Km/h" the counterpart of "90 Km/h-ko abiadura" (meaning a speed of 90 Km/h).

As a consequence, the alignment unit was not word-form any longer, but linguistic token. Thus, Fig. 5 shows the statistical alignments at phrase level over the example of the Fig. 1. On the following, this phrase-based approach will be referred to as PB. METeus is a weather forecast corpus (Pérez et al., 2006b) composed by 28 months of daily weather forecast reports in Spanish and Basque. These reports were picked from those published in Internet by the Basque Institute of Meteorology3 . Initially, the bilingual corpus was aligned at paragraph level and it consisted of 3 865 paragraphs of 54 words on average. RECalign, a non-supervised statistical technique [START_REF] Nevado | Bilingual corpora segmentation using bilingual recursive alignments[END_REF] made the alignments at sentence level possible. A hundred paragraphs, randomly chosen, were evaluated by experts and agreed on the correctness of all of them.

From then onwards we worked with the corpus aligned at sentence level.

After the segmentation process, the corpus was randomly divided into two disjoint sets, referred to as Training and Test-1 in Table 1. The former consisted of 14 615 sentences (summing up 191 156 running words) and the latter of 1 500 sentences. In addition, a sub-test of 500 different sentences was extracted from Test-1 under one restriction: it was not allowed to be any coincidence with the training set. The latter test, Test-2, was recorded for speech translation purposes by 36 bilingual speakers. Each sentence was uttered by at least 3 speakers (male and female) resulting in 1 800 utterances, or equivalently in terms of time, 3.5 hours of audio signal recorded at 16 KHz.

Due to the nature of the task, the Test-1 set is quite repetitive, and so is the training set. Test-1 is statistically representative of the task whereas Test-2 has shown to be more difficult as can be derived from Table 1. On average, the length of the sentences in both the Training and Test-1 sets is around 13 words per sentence, whereas it is around 17 for the Test-2 set. Perplexity is another indicative feature that shows the bias in Test-2, which is relatively much higher than the one for Test-1. As a result, Test-2 might help to test the robustness of the models under situations biased against the training set.

The most remarkable feature of the corpus may lie in what the size of the vocabulary concerns. Notice that the size of the Basque vocabulary is 38% bigger than the Spanish one due to its inflected nature. Even though this is a medium difficulty task, there is a high data sparseness. As a matter of fact, notice that 26% of the words in the Basque vocabulary have just seen once all over the corpus (see singletons in Table 1). In addition, the 66% of the singletons have to do with different lemmas. 1 Main features of METeus corpus. There is a training set and two test sets. Test-1 was randomly selected from the task, while Test-2 consists of 500 different and training independent sentences with both text and speech representations.

6 Experimental results

Implementation issues

For speech translation purposes, the underlying recognizer used in this work is our own continuous-speech recognition system, which implements stochastic finite-state models at all levels: syntactic, lexical and acoustic-phonetic. These models are integrated on the fly at decoding time within the aforementioned integrated architecture, which is illustrated through Fig. 6. The integration on the fly has shown to be an efficient technique in speech-recognition framework [START_REF] Caseiro | A specialized on-the-fly algorithm for lexicon and language model composition[END_REF] and it can be implemented in the same way for speech translation.

Instead of the usual language model, we made use of the SFST itself (Fig. 6(a)), which had the syntactic structure provided by a k-testable in the strict sense model, with k=3 and back-off smoothing. Let us remark that our aim in this work is to develop different approaches of this model and thus the remaining models were not changed.

The lexical model consisted of the extended tokens (s k , tk ) of the SFST instead of the running words involved in a typical language model. The phonetic transcription for each extended token was automatically obtained on the basis of the input projection of each unit (s k ), that is, the Spanish vocabulary (or sub-strings) in this case. The model itself consists of the concatenation of the phonemes involved in a left-to-right finite-state model, as shown in Fig. 6(b).

Each phone-like unit was modeled by a typical left to right non-skipping selfloop three-state continuous hidden Markov model (referred to as HMM in Fig. 6(c)), with 32 Gaussians per state and acoustic representation. The speech signal database was parametrized into 12 Mel-frequency cepstral coefficients (MFCCs) with delta (∆MFCC) and acceleration (∆ 2 MFCC) coefficients, energy and delta-energy (E, ∆E), so four acoustic representations were defined. A phonetically-balanced Spanish database, called Albayzin [START_REF] Moreno | Albayzin speech database: Design of the phonetic corpus[END_REF], was used to train these models.

Evaluation

Three models were trained from the training data: the classical word-based (WB) SFST (introduced in section 3), the morphology-based (MB) approach (see section 4.1) and the phrase-based (PB) approach (see sections 3.4 and 4.2).

All the models were built under the same k=3 k-TSS topology, where the maximum length of the history is 2, as in a 3-gram language model [START_REF] Jelinek | Statistical Methods for Speech Recognition[END_REF].

No pre-or post-edit processing was carried out (neither on idioms or numbers or names etc.). Our purpose was to compare the three models under the same circumstances and study their performance for both text and speech translation. The systems were assessed under the commonly used automatic evaluation measures: bilingual evaluation understudy (BLEU) [START_REF] Papineni | BLEU: a method for automatic evaluation of machine translation[END_REF], NIST [START_REF] Doddington | Automatic evaluation of machine translation quality using n-gram co-occurrence statistics[END_REF], word error rate (WER) and position-independent error rate (PER).

Text translation results associated with the two test-sets described in section 5 are shown in Table 2. As it was expected, the training independent test set (Test-2) reports worse results than Test-1. in all, phrase-based (PB) approach outperformed both morpho-syntactic (MB) and word-based (WB) approach for both test sets. An additional experiment was carried out with Moses free toolkit [START_REF] Koehn | Open source toolkit for statistical machine translation: Factored translation models and confusion network decoding[END_REF], the state-of-the art in phrase-based translation models, taking at random 14 000 pairs from the training set for training purposes and the remaining 615 for tuning. The system was tested with Test-1 set. The obtained results made no statistical difference with respect to those obtained with the phrase-based SFST approach proposed here (specifically a BLEU score of 55.9 was obtained). 2 Text translation results with the three models described, namely, word-to-word model (WB), morphology-based approach (MB) and syntactically motivated phrasebased approach (PB).

WB MB PB

Speech translation results, shown in Table 3, were carried out with both decoupled and integrated architectures using the three models under study. For speech translation, as well as for text translation, syntactically motivated phrase-based SFST approach (PB) turned out to report the best performance. Morphologic knowledge in MB SFST approach, however, just provided slightly improvements over the classical approach (WB).

Comparing the two architectures studied for speech translation, in these experiments the integrated architecture has proved to offer slightly better performance than the decoupled under all the approaches explored (WB, MB and PB). In addition, the integrated architecture is even faster than the decoupled one, since the former occurs in a single decoding stage. These results obtained for a real task and natural corpus agreed with the conclusions extracted in [START_REF] Casacuberta | Learning finite-state models for machine translation[END_REF].

Even though our interest focuses on comparing the performance of different translation models, speech recognition rate is also reported since it plays an important role. Notice that with the decoupled architecture the output of the speech recognition system was the input for the text-translation systems. The speech recognition system was in the same circumstances (3-TSS language model instead of the translation model). Under the integrated architecture, however, speech transcription and translation is simultaneously produced, for this reason each SFST approach offered different recognition results. Slightly better recognition results are obtained with the MB-SFST than with the typical speech recognition system. Another odd fact extracted from the Table 3 Taking text and speech translation results into account (compare Test-2 of Table 2 with Table 3), it is remarkable the small differences in performance.

It might be concluded that the errors reflected in speech recognition were not directly propagated into translation errors. Bear in mind that the translation model is by itself an important source of noise. Its error rate is one magnitude order higher than in the case of the ASR. Thus, the SFST might be insensitive to small deviations in the input. In addition, the model is smoothed, which allows either correct or incorrect input strings while trying to give as a translation only well formed outputs according to the training data.

On the whole, both morphologic and syntactic knowledge introduced within statistical transducers improves the performance for both text and speech translation. Both MB and PB approaches outperform the classical one (WB).

The major improvement is associated with the PB approach, were syntactic phrases were taken as translation unit.

Concluding remarks and future work

This work deals with Spanish into Basque speech translation, a task that, to the authors' knowledge, had never been previously faced in other works.

To do so, stochastic finite-state transducers are considered. The goal of this work is to improve their performance by including linguistic knowledge. Two new approaches were proposed taking into account the specific the languages involved, namely high inflection of Basque and strong differences in syntax. On the one hand morphology was exploited and syntax on the other. The approaches focused on modeling rather than on decoding and represent a general methodology that may be applied to any pair of languages.

Morphologically rich languages present data sparsity on the statistics collected over running word n-grams. A lemmatized representation of the language, however, helped to render essential information about the meaning from one language to the other in the MB framework. This rough approach allowed to deal with rather reliable statistics. In the PB approach, linguistically motivated phrases were proposed as translation unit. Thus, the underlying formulation for phrase-based SFST is defined in this work as an extension of previous SFSTs. The phrases considered in the experimental results consisted of a sequence of words that played the same syntactic role within a given sentence.

The long-distance alignments between Basque and Spanish were cut down with the phrase-based approach. Therefore, the SFST was able to capture those sort of relationships with more accuracy than using running-words.

Experiments were carried out under a narrow domain framework with small and repetitive vocabulary. Results over different architectures for speech translation show that linguistic knowledge helps to improve the performance of statistical translation models. The improvement of the morphological approach was not as successful as the syntactically motivated phrase-based model, where the relative improvement on BLEU score was over the 6%. Let us mention the development of linguistic resources, tools and corpus, for Basque language as another contribution of this work.

For future work, we consider that rather accurate MB approach could be reached since, in practice, the second stage involved a strong assumption as it discarded the source string. The second stage could be exploited along with a statistical dictionary from source words into lemmatized targets. In addition, other kind of categories, instead of lemmas, could be explored as well. With regard to the phrase based SFST approach, we intend to explore not only syntactically motivated phrases and categories but also statistical ones. Finally, it could be of interest to explore the combination of both morphological and syntactic approaches

A Spanish and Basque languages

Machine translation between Spanish and Basque presents multiple challenges as they differ extraordinarily. They do not have the same origin, in fact, Basque is a pre-Indo-European language of still unknown origin, whereas Spanish is amongst the Romance group of the Indo-European family of languages. Nowadays, Spanish and Basque share official status for the 2.5 million inhabitants of the autonomous community of the Basque Country (Spain). Basque is spoken beyond that region, in some areas of Navarre (Spain), Atlantic Pyrenees (France), Reno (United States of America) etc. It has to bear in mind, however, that Basque is a minority language, while Spanish holds the 4th place in the world in what the number of native speakers concerns (after Mandarin Chinese, Hindi and English). Needless to say, there are significant differences with regard to the amount of available linguistic resources, but in these last years several groups from the university and industry are playing a valuable role in developing linguistic supports for Basque language.

Basque, in contrast to Spanish, is a highly inflected language (both in nouns and verbs). For noun phrases there are 17 cases which are at the same time modified by determiners related to number alternation, and indeed, they are susceptible to throwing together in several recurrence levels. For a matter of comparison, in German there are 4 grammatical cases, 7 in Czech, and 15 in Finnish. With regard to the verbs, apart from the tense, both subject (person and number), direct object and indirect object marks (if any) are part and parcel of verbs. This issue has a significant effect on the size of the vocabulary and the repetition ratio. Typically, given a text in Basque and Spanish (or English), the first one has usually fewer running words on the whole than the latter one but the number of different words (that is, the size of the vocabulary) is normally much higher, having direct repercussions on the statistics collected over word utterances.

As depicted in Fig. 1 long distance alignments are likely to occur between Spanish and Basque. This is due to the fact that typical syntactic construction in Basque is Subject-Objects-Verb (like Japanese) unlike Spanish (or English) where Subject-Verb-Objects construction is more common. The order of the phrases within a sentence can be changed with thematic purposes. As a matter of fact, the order of the phrases in Basque is topic-focus, meaning that the topic is stated first and then the focus immediately before the verb phrase. This characteristics have incidence on the statistical alignment modeling.

Fig. 1 .

 1 Fig.1. Manual alignments of a sentence show the different word order between Basque and Spanish; for a matter of clarity, Spanish-English alignment is also given. That is, both Basque and English sentences (in the abscissa) are the translation of the same Spanish sentence (in the ordinate).

  s) is the probability of the string s according to a language model of the source language.(2) Translation of s (the expected transcription of x): t ≈ arg max t P (t| s) (7)

  Phrase-to-phrase alignment. permanecerán muy nubosos los cielos por la mañana oso hodeitsu mantenduko da zerua goizaldean (b) Monotonic segmentation at phrase level.

Fig. 2 .

 2 Fig. 2. Phrase-to-phrase alignment and the subsequent monotonic segmentation between Basque (on the bottom), Spanish (on the top).

  mar et al., 2005) alignment template translation model was implemented making use of weighted finite-state transducers (WFSTs).

T1:Fig. 3 .

 3 Fig.3. The first step for the syntactically motivated segmentation resorts to a morphological parsing of individual words in order to categorize them. This step over the word "viento" gave as a result two different tag-sets.

Fig. 4 .

 4 Fig.4. Example of segmentation and related information. The classification of phrases provides a natural segmentation. In addition, it might help to the alignments, since usually, the phrases playing the same role represent related entities in both languages. In this example, the verb phrase as a whole, even being in different places for each language, are tightly connected.

Fig. 5 .

 5 Fig. 5. Alignments at phrase level between Basque and Spanish.

Fig. 6 .

 6 Fig. 6. Speech translation with integrated architecture involves on the fly Integration of several knowledge sources within a single finite-state network. (a) Finite-state transducer built on the basis of the segmentation of Fig. 2. (b) The lexical model consists on the phonetic transcription of the input substring by means of a left-to-right topology (SAMPA is used here). (c) Phone-like units are modeled by typical three-state continuous hidden Markov models (HMMs).

  is that the PB model offers the best translation results but the worst recognition rate.

			Integrated Arch.	Decoupled Arch.
			WB MB PB WB MB PB
	Speech Recognition WER	8.3	7.3	9.6		7.9
		BLEU 38.5 38.9 40.9 37.4 37.8 40.8
	Speech Translation	NIST	5.7	5.8	6.0	5.6	5.6	6.0
	Test-2	WER	51.3 50.5 49.6 51.2 51.2 50.3
		PER	42.5 41.8 40.4 42.2 42.6 40.3
	Table 3
	Speech translation results of Test-2 with different approaches, namely, word-to-word
	model (WB), morphological approach (MB) and syntactically motivated phrase-
	based approach (PB). Integrated architecture give as a result both the transcrip-
	tion and the translation of speech in a unique decoding step. Decoupled architecture
	entails two independent steps: first speech recognition and then text-to-text trans-
	lation of recognized utterances.

http://ametza.com

Ergative: a case of nouns in a few languages (e.g., Basque and Eskimo) that identifies the subject of a transitive verb.

http://www.euskalmet.net

Acknowledgments

We would like to thank anonymous reviewers for their criticisms and suggestions.

We would also like to thank Ametzagaiña group (http://www.ametza.com), and Josu Landa, in particular, for providing us with the morpho-syntactic parse which made possible this work. This work has been partially supported by the University of the Basque Country under grant 9/UPV 00224.310-15900/2004, by the Spanish CICYT under grant TIN2005-08660-C04-03 and by the research programme Consolider Ingenio-2010 MIPRCV (CSD2007-00018).