Zanon Marcely

Antonios Boito

Marika Anastasopoulos

Aline Lekakou

Laurent Villavicencio

Besacier

Marcely Zanon Boito
email: marcely.zanon-boito@univ-grenoble-alpes.fr

Antonios Anastasopoulos

Marika Lekakou

Aline Villavicencio

Laurent Besacier

A small Griko-Italian speech translation corpus

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

For many low-resource and endangered languages, speech data is easier to obtain than textual data. Oral tradition has historically been the main medium for passing cultural knowledge from one generation to the next, and at least 43% of the world's languages are still unwritten [START_REF] Lewis | Ethnologue: Languages of the world[END_REF]. Traditionally, documentary records of endangered languages are created by highly trained linguists in the field. However, modern technology has the potential to enable creation of much larger-scale (but lowerquality) resources. Recently proposed frameworks [START_REF] Bird | Collecting bilingual audio in remote indigenous communities[END_REF][START_REF] Stüker | Innovative technologies for under-resourced language documentation: The Bulb project[END_REF] propose collection of bilingual audio, rendering the resource interpretable through translations.

New technologies have been developed to facilitate collection of spoken translations [START_REF] Bird | Aikuma: A mobile app for collaborative language documentation[END_REF] along with speech in an endangered language, and there already exist recent examples of parallel speech collection efforts focused on endangered languages [START_REF] Blachon | Parallel speech collection for under-resourced language studies using the LIG-Aikuma mobile device app[END_REF][START_REF] Adda | Breaking the unwritten language barrier: The bulb project[END_REF][START_REF] Rialland | Parallel Corpora in Mboshi (Bantu C25, Congo-Brazzaville)[END_REF]. The translation is usually in a high-resource language that functions as a lingua franca of the area, as it is common for members of an endangered-language community to be bilingual.

Tackling the issue of the possible vanishing of more than 50% of the current spoken languages by the year 2100 [START_REF] Austin | The Cambridge handbook of endangered languages[END_REF], the Computational Language Documentation (CLD) field assembles two different communities: linguistics and informatics, proposing challenges [START_REF] Versteegh | The zero resource speech challenge 2015: Proposed approaches and results[END_REF][START_REF] Jansen | A summary of the 2012 jhu clsp workshop on zero resource speech technologies and models of early language acquisition[END_REF][START_REF] Dunbar | The zero resource speech challenge 2017[END_REF] and frameworks from speech signal [START_REF] Besacier | Towards speech translation of non written languages[END_REF][START_REF] Bartels | Toward human-assisted lexical unit discovery without text resources[END_REF][START_REF] Lignos | Recession segmentation: simpler online word segmentation using limited resources[END_REF]. However, as the interest on CLD approaches grows, it becomes clear the urgent need of more publicly available low-resource corpora to provide replicable evaluation of the proposed methods. We are aware of only a few endangered languages whose corpora are publicly available [START_REF] Godard | A very low resource language speech corpus for computational language documentation experiments[END_REF][START_REF] Hamlaoui | BULBasaa: A bilingual Bàsàá-French speech corpus for the evaluation of language documentation tools[END_REF].

Our work is part of this effort to share resources, and with this paper we present a corpus on a truly endangered dialect from south Italy, Griko. The corpus has several levels of information (speech, machine extracted pseudo-phones, transcriptions, translations and sentence alignment), and we believe it can be an interesting resource for evaluating documentation techniques on (very) low-resource settings.

In addition, we provide baseline results for two tasks: speech-to-translation alignment and unsupervised word discovery. We encourage the community to challenge these results by using their own techniques. For word discovery, we also provide the gold standard for evaluation following the Track 2 of the Zero Resource Challenge (ZRC) 2017 [START_REF] Dunbar | The zero resource speech challenge 2017[END_REF]. These metrics were extensively described in [START_REF] Ludusan | Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems[END_REF][START_REF] Dunbar | The zero resource speech challenge 2017[END_REF], and are another important community effort for increasing reproducibility.

This paper is organized as follows: after a quick related work (section 2), the Griko language is presented (section 3). Data processing methodology (section 4) and dataset are then presented. Our baseline systems and results for two tasks (sections 5 and 6) are finally described.

Related Work

Unsupervised Word Discovery (UWD) systems operate on unsegmented speech utterances and their goal is to output timestamps delimiting stretches of speech, associated with class labels, corresponding to real words in the language. This task is already considered in the Zero Resource Speech Challenge1 in a fully unsupervised setting: systems must learn to segment from a collection of raw speech signals only. Here, we investigate a slightly more favorable case where speech utterances are multilingually grounded (using cross-lingual supervision, where a written translation is available for each utterance). In CLD scenarios, this task helps to attenuate the heavy charge on field linguists: the output vocabulary can be used as a first clue of the lexicon present in the language of interest. As a monolingual setup, UWD was previously investigated from text input [START_REF] Goldwater | A Bayesian framework for word segmentation: Exploring the effects of context[END_REF] and from speech [START_REF] Jansen | Efficient spoken term discovery using randomized algorithms[END_REF][START_REF] Lee | Unsupervised lexicon discovery from acoustic input[END_REF][START_REF] Bartels | Toward human-assisted lexical unit discovery without text resources[END_REF][START_REF] Elsner | A joint learning model of word segmentation, lexical acquisition, and phonetic variability[END_REF].

The speech translation problem has been traditionally approached by feeding the output of a speech recognition system into a Machine Translation (MT) system. Speech recognition uncertainty was integrated with MT by using speech output lattices as input to translation models [START_REF] Ney | Speech translation: Coupling of recognition and translation[END_REF][START_REF] Matusov | On the integration of speech recognition and statistical machine translation[END_REF]. A sequence-tosequence model for speech translation without transcriptions has been introduced [START_REF] Duong | An attentional model for speech translation without transcription[END_REF], but was only evaluated on align-ment. Synthesized speech data were translated in [START_REF] Bérard | Listen and translate: A proof of concept for end-to-end speech-to-text translation[END_REF] using a model similar to the Listen Attend and Spell model [START_REF] Chan | Listen, attend and spell: A neural network for large vocabulary conversational speech recognition[END_REF], while a larger-scale study [START_REF] Bérard | End-to-end automatic speech translation of audiobooks[END_REF] used an end-to-end system for translating audio books between French and English. Sequence-tosequence models to both transcribe Spanish speech and translate it in English have also been proposed [START_REF] Weiss | Sequence-to-sequence models can directly transcribe foreign speech[END_REF], by jointly training the two tasks in a multitask scenario with two decoders sharing the speech encoder. This model was further extended [START_REF] Anastasopoulos | Tied multitask learning for neural speech translation[END_REF] with the translation decoder receiving information both from the speech encoder and the transcription decoder.

For endangered languages (extremely low-resource settings) the lack of training data leads to the problem being framed as a sparse translation problem. This semi-supervised task lies between speech translation and keyword spotting, with cross-lingual supervision being used for word segmentation [START_REF] Bansal | Towards speech-to-text translation without speech recognition[END_REF][START_REF] Bansal | Weakly supervised spoken term discovery using cross-lingual side information[END_REF][START_REF] Anastasopoulos | Spoken term discovery for language documentation using translations[END_REF][START_REF] Kamper | Visually grounded learning of keyword prediction from untranscribed speech[END_REF]. Bilingual setups for word segmentation were discussed by [START_REF] Stüker | Towards human translations guided language discovery for ASR systems[END_REF][START_REF] Stüker | Human Translations Guided Language Discovery for ASR Systems[END_REF][START_REF] Stahlberg | Word segmentation through cross-lingual word-to-phoneme alignment[END_REF][START_REF] Boito | Unwritten languages demand attention too! word discovery with encoder-decoder models[END_REF], but applied to speech transcripts (true phones). Among the most relevant to our approach are the works of [START_REF] Duong | An attentional model for speech translation without transcription[END_REF] on speech-to-translation alignment using attentional Neural Machine Translation (NMT) and of [START_REF] Bansal | Weakly supervised spoken term discovery using cross-lingual side information[END_REF][START_REF] Anastasopoulos | Spoken term discovery for language documentation using translations[END_REF] for language documentation. However, the former does not address word segmentation and is not applied to a language documentation scenario, while the latter methods do not provide a full coverage of the speech corpus analyzed. A neural approach for word segmentation in documentation scenarios using the soft attention matrices (which we also use for our baseline experiments) was investigated in [START_REF] Boito | Unwritten languages demand attention too! word discovery with encoder-decoder models[END_REF].

The Griko Language

Griko is a Greek dialect spoken in southern Italy, in the Greca Salentina area southeast of Lecce. 2 There is another endangered Italo-Greek variety in southern Italy spoken in the region of Calabria, known as Grecanico or Greco. Both languages, jointly referred to as Italiot Greek, were included as seriously endangered in the UNESCO Red Book of Endangered Languages in 1999. Griko is only partially intelligible with modern Greek, and unlike other Greek dialects, it uses the Latin alphabet. In addition, it is rare among the Greek dialects, due to its retention of the infinitive in particular syntactic contexts. Less than 20,000 people (mostly people over 60 years old) are believed to be native speakers [START_REF] Horrocks | Greek: A History of the Language and its Speakers[END_REF][START_REF] Douri | Griko and modern Greek in Grecìa Salentina: an overview[END_REF]; unfortunately, this number is quite likely an overestimation [START_REF] Chatzikyriakidis | Clitics in four dialects of modern Greek: A dynamic account[END_REF].

Resources in Griko are very scarce, with almost no corpora available for linguistic research. The first grammar of the language was composed by the German scholar Gerhard Rohlfs [START_REF] Rohlfs | Grammatica storica dei dialetti italogreci (Calabria, Salento) dt. Original[END_REF] to be followed by others [START_REF] Karanastasis | Grammatiki ton ellinikon idiomaton tis Kato Italias [Grammar of the Greek dialects of south Italy[END_REF].

Recently, a corpus of Griko narratives was released [START_REF] Anastasopoulos | Part-of-speech tagging on an endangered language: a parallel Griko-Italian resource[END_REF]: it contains 114 narratives originally collected by Vito Domenico Palumbo (1854Palumbo ( -1928) ) the most noted Griko scholar [START_REF] Palumbo | Io' mia fora' -Fiabe e Racconti della Greca Salentina [Once upon a time -Fairy Tales and Stories from Greca Salentina[END_REF][START_REF] Palumbo | Itela na su pó -Canti popolari della Greca Salentina [I wanted to tell you -Folk songs of Grecía Salentina[END_REF]. The narratives were further annotated with translations in Italian, and partly annotated with gold Part-of-Speech information.

Here, we present and extend the only Griko speech corpus available online 3 [START_REF] Lekakou | Documentation and analysis of an endangered language: aspects of the grammar of Griko[END_REF], consisting of about 20 minutes of speech in Griko, along with text translations into Italian. The original corpus (henceforth UoI corpus, as it is hosted at the University of Ioannina, Greece) consists of 330 mostly elicited utterances by nine native speakers, annotated with transcriptions, morphosyntactic tags, and glossed in Italian. 2 A discussion on the possible origins of Griko can be found in [START_REF] Manolessou | The greek dialects of southern Italy: an overview[END_REF]. 3 http://griko.project.uoi.gr

Data Processing

The original UoI corpus was collected during a field trip in Puglia, Italy by two linguists, with a particular focus on the use of infinitive and verbal morphosyntax. The corpus contains utterances from 9 different speakers (5 male, 4 female) from the 4 villages (Calimera, Sternatia, Martano, Corigliano) where native speakers could still be found. The digitally collected audio files (16-bit PCM, 44.1kHz, stereo) were manually segmented into utterances, transcribed, glossed in Italian, and annotated with extensive morphosyntactic tags by a trained linguist.

Annotation Extensions

In order to render the UoI corpus useful for speech-related computational research on Griko, we extend the corpus with the following annotations:

1. Free-form Italian translations for every utterance, created by a bilingual speaker, 2. gold-standard word-level alignment information for every utterance, including annotated silences, 3. gold-standard speech-to-translation alignments, 4. pseudo-phones representation, obtained by using the acoustic unit discovery (AUD) method presented in [START_REF] Ondel | Variational inference for acoustic unit discovery[END_REF],

5. ZRC gold standard for standard evaluation, described in the next section.

Figure 1 shows an example of sentence pairs from our collection, and Table 1 presents some statistics on these aligned transcriptions and translations. We observe that both sides of the parallel corpus are considerably similar with respect to the metrics presented here (sentence structure and vocabulary). This is reasonable: the two languages belong to the same family and have been in contact for centuries.

A reference compatible with the ZRC metrics

In addition to the word-level annotations, we built and make available a reference (in the format of the ZRC challenge) in order to allow evaluation of different word discovery approaches using this corpus. We had a manual alignment between speech and words, but no possibility to obtain an accurate automatic alignment between speech and phones (or graphemes) due to the very small amount of data available (not possible to train an acoustic model using a Kaldi pipeline on 330 signals, for instance).

Thus, we used the word-level alignment information between speech and transcription, and the silence annotation available in our corpus, to approximate a speech-to-grapheme alignment. For each word present in the corpus, we retrieve its time window and segment this time window into smaller ones, giving to each existing grapheme an equal portion of its word time window. We manually corrected some of the silence and word annotations to ensure that we had no overlap between silence and words time windows. This approximation was necessary to make the ZRC metrics work.

The final reference can be considered correct for evaluation of word discovery tasks (which do not take into account subword annotation), but should be consider with caution for evaluation of subword discovery tasks. Finally, we created two ZRC versions, one removing the silence tokens, used for grapheme evaluation, and a second one with all the information, used for pseudo-phones evaluation.

Speech-to-Translation Alignment

The task of speech-to-translation alignment is the problem of identifying portions in an audio segment that should be aligned to words in (text) translation, without access to transcriptions [START_REF] Duong | An attentional model for speech translation without transcription[END_REF]. Our speech-to-translation alignment annotations allow us to evaluate such methods on our corpus. Evaluation is performed by computing standard precision, recall, and F-score on the links between speech frames and translation words. Providing a baseline for future work, Table 2 reiterates previous results on speech-to-translation alignment. We present results with three methods: a naive proportional baseline (proportional), a neural alignment model [START_REF] Duong | An attentional model for speech translation without transcription[END_REF] (neural), and an unsupervised model (DTW-EM) [START_REF] Anastasopoulos | An unsupervised probability model for speech-to-translation alignment of lowresource languages[END_REF]. The naive baseline assumes no reordering and simply segments the audio to as many segments as the translation words, each with a length proportional to the word's length in characters. The neural alignment model trains a speech-to-translation end-to-end sequence-tosequence system with attention on all the data, and then the soft attention matrices are converted to hard alignments between audio segments and translation words. DTW-EM is an unsupervised model that extends the IBM Model 2 alignment model [START_REF] Och | Statistical multi-source translation[END_REF] to work on speech segments, combining it with a Dynamic Time-Warping-based clustering approach [START_REF] Petitjean | A global averaging method for dynamic time warping, with applications to clustering[END_REF].

Since the two languages have several similar characteristics, the naive proportional baseline is already very competitive; its recall is better than both other evaluated methods. The unsupervised model, however, achieves much higher Precision and F-score than the rest. Unsurprisingly, the neural approach performs significantly worse in this setting: 330 sentences are clearly not enough to train a robust word-level model.

Unsupervised Word Discovery Experiments

In this section we illustrate the use of our corpus for the task of unsupervised word discovery. We use three different baselines, one monolingual and two bilingual, and two different representation levels, graphemes (from text) and pseudo-phones (automatically extracted from speech). Evaluation is performed using the Boundary metric from the Zero Resource Challenge 2017 (Track 2) [START_REF] Dunbar | The zero resource speech challenge 2017[END_REF]. We compute recall, precision and F-score. Below, we describe the three baselines evaluated in this work.

• Dpseg (monolingual): dpseg4 is the non-parametric bayesian model introduced in [START_REF] Goldwater | A bayesian framework for word segmentation: Exploring the effects of context[END_REF]. On this setup, words are generated by a bigram model over a non-finity inventory, through the use of a Dirichlet-Process. Estimation is performed through Gibbs sampling. This approach is known as being very robust on low-resource scenarios.

The hyper-parameters used here are the same from [START_REF] Godard | Preliminary experiments on unsupervised word discovery in mboshi[END_REF].

• Proportional Segmentation (bilingual): this baseline uses the word boundaries in the translation to segment the input proportionally. We can expect considerable good results for proportional segmentation when applied on language pairs similar on sentence structure and average token length, and therefore, we expect good results for this baseline when applied to the Griko-Italian corpus (see Table 1).

• Neural Segmentation (bilingual): the method applied in this paper was presented in [START_REF] Boito | Unwritten languages demand attention too! word discovery with encoder-decoder models[END_REF]. It post-processes a NMT system's soft-alignment probability matrices to generate hard segmentation. Due to the length discrepancy between the symbols (graphemes and pseudophones) and the translations, our post-processing included alignment smoothing. This procedure, proposed by [START_REF] Duong | An attentional model for speech translation without transcription[END_REF], consists of adding temperature T to the softmax function used by the attention mechanism. Resulting soft-alignments matrices are further smoothed by averaging each probability by its right and left neighborhood. However, in this work we use T = 1 for all setups, and only the alignment matrices smoothing (averaging with the right and left neighbors) is used here. Also, for stability reasons, we report the averaged scores over 5 different trained models.

• Merged Neural Segmentation (bilingual): the same methodology from the previous baseline, with the difference of averaging the soft-alignment probability matrices before post-processing, instead of averaging only the scores. We use the same 5 runs from the previous setup to generate an averaged (merged) segmentation. For the pseudo-phones segmentation, all methods had a considerable drop in performance, specially dpseg. They all achieved similar F-scores, with the merged neural baseline being slightly more effective. Table 4 presents some numbers for the generated segmentation of all methods presented in this section. We observe that, for pseudo-phones, dpseg seems to oversegment the input (average tokens per sentence), while the neural baselines segmented the input considerably less.

Lastly, pseudo-phones were obtained through an unsupervised unit discovery system, which inevitably adds noise to the representation. This noise is then propagated to the word discovery system. We believe the achieved results for pseudophones illustrate the difficulty of the task of word discovery on extreme low-resource setups.

Conclusion

In this paper we presented an extension of a very small parallel corpus on an endangered language called Griko. We make this corpus, with all its different levels of representation, freely available to the community as an effort in the direction of research replicability for low-resource approaches. 5We illustrated the potential of this parallel corpus by performing the tasks of speech-to-text alignment and unsupervised word discovery. We encourage the community to challenge the baselines presented here.

Future work includes comparing the tasks results from this extreme case of language documentation with other lowresource corpora, such as the one presented in [START_REF] Godard | A very low resource language speech corpus for computational language documentation experiments[END_REF].

Acknowledgements

This work was partly funded by French ANR and German DFG under grant ANR-14-CE35-0002 (BULB project). An-

Table 1 :

 1 Statistics of the 330 sentences in our parallel Griko-Italian corpus.

		Griko		jatì ìche polemìsonta òli tin addomàda
		Italian		perché aveva lavorato tutta la settimana
	Figure 1: A tokenized and lower-cased sentence pair example in our Griko-Italian corpus.
		# tokens Vocabulary size	Average tokens length	Average # tokens per sentence	Shortest token	Largest token
	Griko	2,374		691	5.68	7.19	1	16
	Italian	2,384		456	5.76	7.22	1	13
	Method	P	R	F
	proportional 42.2 52.2 46.7
	neural	24.6 30.0 27.0
	DTW-EM	56.6 51.2 53.8

Table 2 :

 2 On speech-to-translation alignment, the unsupervised model outperforms the neural attentional model and the naive baseline in terms of Precision and F-score.

Table 3

 3 presents the achieved results. Even on this very low-resource scenario, dpseg has a remarkable performance for .10 71.60 44.70 44.80 44.70 42.66 51.84 46.72 50.20 54.00 52.10 pseudo-phones 23.30 36.90 28.50 28.50 29.90 29.20 32.00 27.68 29.56 34.30 26.70 30.00

			dpseg			proportional			neural			merged neural
		P	R	F	P	R	F	P	R	F	P	R	F
	grapheme	68.50 75

Table 3 :

 3 Boundary scores for the task of unsupervised word segmentation. Results for neural segmentation are the average over 5 runs. Best results for each metric are presented in bold.

			grapheme			pseudo-phones
		# tokens Vocabulary Size	Average # tokens per sentence	# tokens Vocabulary Size	Average # tokens per sentence
	proportional	2,370	1,715	7.18	2,366	1,431	7.17
	dpseg	2,629	567	7.97	3,912	520	11.85
	neural (average)	2,972	1,462	9.01	1,929	1,066	5.84
	merged neural	2,573	1,476	7.80	1,676	967	5.08

Table 4 :

 4 A comparison between the generated segmentation by the four baselines. For the neural baseline, results are the arithmetic mean between the statistics for the 5 runs.the task of word segmentation working with graphemes. It retrieved 75.10% of the correct boundaries (recall). The second best method from the baselines for grapheme segmentation was the merged version of the neural segmentation. The remaining two baselines (proportional and neural) had close performance, achieving retrieval between 44 and 52%.

http://zerospeech.com/2017

Available at http://homepages.inf.ed.ac.uk/sgwater/resources.html.

Available at goo.gl/EWa15G tonis Anastasopoulos was generously supported by NSF Award 1464553. Marika Lekakou was supported by the John S. Latsis Public Benefit Foundation under the project 'Documentation and analysis of an endangered language: aspects of the grammar of Griko'.