M Jean-

Pierre Chevrot

Marcel Cohen

Letra

Antonio Saura

Raphael Rubinstein

Keywords: acquisition du langage visuellement contextualisé, modèle de traitement de la parole visuellement contextualisée, acquisition lexicale, acquisition du langage grounded language learning, visually grounded speech model, lexical acquisition, language acquisition

La lingua maternal: asi se dize de lo ke se entendya enkaza

Introduction

I Background

Introduction Context and Motivation

In a relatively short period of time, children are able to acquire their native language allowing them to understand it and speak it effortlessly. This is an amazing feat, as they are able to do so without much supervision. They indeed do not need adults around them to explicitly teach them new words or correct them when they speak. Instead, the perceptible surrounding context seems to provide them with all the necessary information they need in order to acquire their native language.

According to Landau & Gleitman (1985, p. 1), " [n]o disagreement arises about the necessity for extralinguistic experience. Their [the theorists'] disagreements have to do with the sufficiency of experience for learning a language". Indeed, while some language acquisition theories [START_REF] Chomsky | Aspects of the Theory of Syntax[END_REF][START_REF] Pinker | Language Learnability and Language Development, With New Commentary by the Author[END_REF]) postulate strong innate knowledge where the environment only plays a minor role, other [START_REF] Skinner | Verbal Behavior, Century psychology series, Appleton-Century-Crofts[END_REF][START_REF] Tomasello | Constructing a Language[END_REF]) postulate a much larger influence of the environment. However, all these theories agree -to some extenton the fact that a certain amount of extralinguistic experience is necessary. Indeed, even in the strictest inneist theories, where the child would be born with a knowledge of grammatical and conceptual entities, a mapping still has to operate between the abstract entities of the human mind and their realisation in the physicial world, which is only perceptively accessible. Landau & Gleitman (1985, p. 7) state that "the child's input consists of sound/situation pairs, but his final output is a set of form/meaning pairs, appropriate to an infinite set of novel but well-circumscribed situations". The question is how do children transition from sound/situation pairs to form/meaning pairs? One step children have to go through is to understand that the sounds they perceive constitute conventional signs used for communication purposes, and not random sounds. This step might not be the first step children go through, but it is a critical step to go from sound to form. Transitioning from sound/situation to form/meaning pairs also implies a segmentation step. The first segmentation step we could think of is the segmentation of the speech stream into forms. Indeed, the speech stream contains a sequence of forms, which are however connected and not neatly separated from one another. Children thus have to learn how to segment the speech stream appropriately so as to discover the conventional word forms used in their native language. The second segmentation step we could think of consists in analysing the environment so as to identify and extract the stakeholders: who is talking, to whom, about what, etc. While the first step involves identifying patterns in the spoken modality, the other step involves identifying patterns in other modalities (visual, haptic, etc.). We presented these two steps as separate or as if they occurred sequentially but it is in fact not the case, as they do occur simultaneously. Moreover, these two steps seem to be fueled by similar routines in the child's mind (e.g. statistical sensitivity, see [START_REF] Kirkham | Visual statistical learning in infancy: evidence for a domain general learning mechanism[END_REF].

Once forms have been extracted from the speech stream, and referents extracted from the a set of form/meaning pairs (other processes include the acquisition of syntax and syntactic frames, the acquisition of phonology, etc.).

The form/meaning pairs the child builds should be stored in such a way they can be easily accessed so as to parse the speech stream and to build its own utterances. Hence, the stored word forms should be specific enough so as to not allow for similar sounding wordforms to be recognised, while not being overly specific so as to accomodate for variation and mispronounciations. This step is commonly referred to as word recognition.

In this thesis, we propose to study an neural model of visually grounded speech. Visually grounded speech models are models that are trained to solve a speech/image retrieval task. That is, given a spoken description of an image, they should find the closest matching image among a collection of images (or vice-versa, find a spoken description given an image). To do so, such networks have to learn how to appropriately transform the input image and the input spoken description so that it is easy to find one given the other. The task such networks have to solve is thus very close to that of a child acquiring her native language. Indeed, such networks are presented with sound/situation pairs (i.e. a spoken description and its matching image), and in order to find the correct matching image, we hypothetise that the network should learn to transform this pair into a form/meaning pair. Indeed, in order to find the matching image among a collection of images given a spoken utterance, it should somehow segment the speech stream into sub-units, so that the resulting subunits refer to objects in the image. The representation of the extracted units should be specific enough so that when prompted with these units, the network only retrieves images featuring instances that the spoken unit refers to. Similarly to humans, on the image side, the network should be able to abstract the referent, so that the target image can be retrieved, even if the object is presented in a non-canonical manner or among a cluttered environment. Consequently, the task neural models of visually grounded speech are trained to solve is a task which is very close to the one of children learning their mother tongue, and more specifically is very close to the task of lexical acquisition. They have to go through the same steps than a child, which are: segmentation, mapping, and recognition.

Contributions

In this thesis, we study a recurrent-based model of visually grounded speech. As such models solve a task similar to that of children, who discover the set form/meaning pairs of their native language, our analyses focus on understanding if neural models do so in the same way.

In this manuscript, we present the following contributions:

(i) We introduce a spoken extension of an image captioning data set that is suitable to train visually grounded speech models. Unlike most data sets of visually grounded speech which are in English, this data set is in Japanese which allows us to study in a contrastive approach the impact of the input language (English or Japanese) on the representations learnt by the model.

(ii) We study the patterns of the attention weights of the attention mechanisms of our models, so as to understand what parts of the speech signal are highlighted, and to what extent they differ from what randomness would predict. We also propose a longitudinal investigation, where we study how these attention weights evolve during the training phase.

(iii) We introduce a methodology stemming from the psycholiguistic litterature, the gating paradigm [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF], that allows us to easily investigate spoken word activation and competition in our model. To the best of our knowledge, this is the first time such methodology is used to investigate the representations learnt by neural models of speech processing.

(iv) We propose a method to simply introduce prior linguistic information in the form of segment boundaries (phone, syllable, or word boundaries) in a neural model of speech processing. The method we propose allows to integrate several types of boundaries, at different levels of the neural architecture, which allows to take into account the hierarchical nature of the spoken input. This allows us to study if segmenting the speech signal into sub-units allows the models to learn to better map images to their spoken descriptions.

Thesis Outline

This thesis is divided into two parts: Part I: Background consists of two background chapters and Part II: Contributions consists of four chapters presenting our contributions. The chapters of this manuscript are organised as follows:

Chapter 1: Background on Child Lexical Acquisition: This chapter is structured into three parts. In the first part, we present the strategies infants and children use in order to segment the speech stream into sub-units. In the second part, we review how children are able to map the segmented units they have extracted from the speech stream to their referent, so as to ultimately acquire their meaning. Finally, in the third part, we present several psycholinguistic models of word activation and recognition. We argue that language acquisition, and more specifically lexical acquisition, is only possible because of the multi-modal nature of language, and in particular, we show how the visual modality particularly helps children in this task. This review allows us to understand how lexical acquisition takes place in humans, and consequently, allows us to formulate hypothesis on how the neural model we study carries out its task.

Chapter 2: Background on Speech Processing and Term Discovery: This chapter presents the notion of unsupervised speech processing as well as the main approaches to unsupervised speech segmentation and term discovery. We argue that in order to be most successful, unsupervised approaches to speech segmentation and term discovery should be grounded to another source of information. We then present the basics of machine learning and artificial neural networks, and then review the existing models of visually grounded speech. We end this chapter by examining the difference between simulation and modelling and by discussing to what extent visually grounded models are indeed grounded.

Introduction

featuring real human speech. Our analysis mainly focuses on which parts of speech signals are highlighted by the models, and we make connections with child language acquisition. As our models are trained on two languages, English and Japanese, this enables us to observe which behaviours are language-general and which behaviours are language-specific. Finally, we analyse how the attention weights evolve during training.

Chapter 5: Word Activation, Competition, and Recognition: The experiments conducted in this chapter are directly inspired by prior psycholinguistic experiments on word recognition in humans. Using the gating paradigm [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF], we analyse the neural representations of a visually grounded speech model trained on an English data set. More specifically, we investigate if such models are able to recognise isolated words and explore how isolated word activation takes place. In the last part of this chapter, we investigate if word activation and recognition takes place through a process of competition, such as what has been postulated for human word recognition.

Chapter 6: Impact of Prior Linguistic Information: In this final chapter, we investigate if neural models of visually grounded speech have better performances if, instead of being given full utterances, they are given pre-segmented utterances. We investigate if this is the case with different levels of segmentation as well as with a random segmentation. We explore at which layers of the architecture such information should be introduced in order to be most effective. In the last part of this chapter, we explore if using a hierarchical model, where several levels of segmentation are simultaneously given to the model, improve the results, and which combination of segmentation levels proves the most effective.

Conclusion:

In this last chapter we summarise the contributions of this thesis, and Chapter 1. Background on Child Lexical Acquisition

Introduction

Language acquisition refers to the process by which a child learns her native language. In order to acquire her native language, one step the child has to perform is to build a mental lexicon [START_REF] Emmorey | The mental lexicon[END_REF]. A mental lexicon is a set of phonological word-forms associated to a meaning. It also contains information about the morphology of each form (e.g. plural formation, conjugations, etc.), syntactic information (e.g. part of speech, valency, expected semantic roles), as well as semantic and pragmatic information (e.g. connotation). This mental lexicon is then used by the child to parse the speech stream and understand what is said; or later on, to build her own utterances.

However, building this mental lexicon is far from easy. Indeed, the speech stream does not contain neatly separated words that the child would only have to assign a meaning to. The child first has to segment the speech stream into sub-units and ultimately pair a meaning to each of the resulting sub-units. These two processes may not necessarily occur simultaneously, and for a time being, the mental lexicon can contain word-forms that were not assigned any meaning yet (see [START_REF] Jusczyk | Infants' Detection of the Sound Patterns of Words in Fluent Speech[END_REF]. The meaning of each form itself is not a ready-to-use object, but has to be inferred and constructed by the child using perceptual information. Finally, the mental lexicon has to be structured in such a way it can be used by the child, notably when parsing the speech stream, in order to activate the meaning of the recognised words so as to interpret what is being said.

These basic processing steps to speech processing and comprehension that children should acquire in order to be able to parse the speech stream correctly -and ultimately acquire their native language -are identified by Di Cristo (2013, pp. 51-52) as segmentation, recognition and interpretation. He also adds another step, which takes place before the segmentation step, which is decoding and which is concerned with "extracting the speech signal from its acoustic environment and convert some of the information it contains into linguistic representations".

In this chapter, we will review the strategies that children use in order to parse and segment the spoken input into sub-units (Section 1.2). We will then present several models which account for how humans retrieve words from the mental lexicon (Secion 1.4). Finally, we will show the tools used by the child in order to infer the meaning of a given word, particularly focussing on how vision provides the essential tools to do so (Section 1.3).

Speech Segmentation

Speech segmentation is generally regarded as a typical example of chicken-and-egg problem. Indeed, in order to correctly parse the speech stream, one should know what a word is, but in order to know what a word is, one should already be able to parse the spoken input. Nonetheless, children are able to solve this riddle, as they all manage to learn their native language at some point. In a relatively short period of time, children are able to identify words in the speech stream without much supervision. In the following sections, we will review some of the cues children use in order to discover words and segment the speech stream.

Speech perception

Language acquisition does not start from the moment the child is born, but rather as soon as the foetus is able to hear, that is, when its body develops a functional proto-auditory system at about 19 weeks of gestational age (Hepper & Shahidullah 1994). Perception of speech evolves as the foetus grows and its auditory systems matures, enabling the brain to tune to the language environment prior to birth [START_REF] May | Language and the Newborn Brain: Does Prenatal Language Experience Shape the Neonate Neural Response to Speech?[END_REF]. As soon as 35 weeks of gestational age, the foetus is already "capable of discriminating different sounds" (Shahidullah & Hepper 1994). [START_REF] May | Language and the Newborn Brain: Does Prenatal Language Experience Shape the Neonate Neural Response to Speech?[END_REF] showed that neonates' brain responses were different for familiar languages and unfamiliar languages, effectively showing the foetus has learnt a robust enough representation of the language it was exposed to prior to birth so as to recognise it once born. [START_REF] Decasper | Prenatal maternal speech influences newborns' perception of speech sounds[END_REF] also demonstrated that newborns are able to recognise samples of stories that were read out loud by their mother during pregnancy. This lets the authors suspect that "foetuses had learned and remembered something about the acoustic cues which specified their particular target passage". These experiments therefore suggest that the exposure to language foetuses have is far from trivial and that they are already accustomed to their future mother tongue.

This pre-natal exposure also develops biases the child will be able to use once born. Hepper & Shahidullah (1994) showed that foetuses are more sensitive to low frequencies (500 Hz and lower) than to higher frequencies. Thus, human voices "form a salient auditory stimulus" as "the fundamental frequency of the human voice is around 225 Hz for females and 128 Hz for males". Newborns indeed attend more to human voices than to other acoustic stimuli [START_REF] Vouloumanos | Listening to language at birth: evidence for a bias for speech in neonates[END_REF].

Hence, as reported by [START_REF] Decasper | Prenatal maternal speech influences newborns' perception of speech sounds[END_REF], "newborns do not act like passive and neutral listeners". The newborn child should not be considered as a tabula rasa, but is on the contrary already attuned to its mother tongue. These biases will help the child segment the speech signal into sub-units.

Suprasegmental Cues

Suprasegmental cues are cues that are present in the suprasegmental features of speech and that are used by children to discover word boundaries. Suprasegmental features are modulations of the speech signal that may span over more than one segment (i.e. phone or syllable). These suprasegmental features, grouped under the umbrella term or prosody, include rhythm (tempo and pauses), intonation, and stress. As prosodic features are mainly linked to modulation of the fundamental frequency (F0) and that children are particularly sensitive to this frequency, they form salient cues the child can use to find words. Some authors even state that "at the beginning was prosody" (Di Cristo 2013, p. X). Indeed, prosody assumes many functional roles, the main one that may be of use for the infant being its demarcation function, that groups the spoken units together and that children can use to infer word boundaries.

The prosodic bootstrapping hypothesis -initially proposed by Lila R. [START_REF] Gleitman | Language Acquisition: The State of the Art[END_REF] -states that language acquisition is bootstrapped by the prosodic features the child is able to perceive, both from a lexical point of view than from a syntaxtical point of view (see [START_REF] Jusczyk | The Discovery of Spoken Language, A Bradford Book[END_REF] 38 for a review). An evidence that prosodic features are used by children to segment the speech stream was made by [START_REF] Cristia | Sensitivity to prosody at 6 months predicts vocabulary at 24 months[END_REF]. Their study shows that 6-month-old English-speaking infants' prosodic sensitivity 1 is a good predictor for their productive vocabulary at 24 months. Hence, if children indeed use prosodic cues to segment the speech stream, it appears natural that those that have the greatest sensitivity to prosodical information manage to segment more words than the others. We will see in the next section more precisely which prosodic cues children use in order to bootstrap language acquisition. 10 Chapter 1. Background on Child Lexical Acquisition

Rhythm

Languages may be divided into three rythmic classes depending on the unit of isochrony used: stress in stress-timed languages (e.g. English, Dutch) where the interval between two stressed syllables is equal, mora 2 in mora-timed languages (e.g. Japanese, Tamil) where the duration of each mora is equal, and syllables in syllable-timed languages (e.g. French, Spanish) where the duration of each syllable is equal. The rythmic segmentation bootstrapping hypothesis [START_REF] Nazzi | Perception and acquisition of linguistic rhythm by infants[END_REF][START_REF] Nazzi | Segmentation précoce de la parole continue en mots : évaluation inter-linguistique de l'hypothèse d'initialisation rythmique[END_REF] postulates that rhythm plays a major role in language acquisition, and that the segmentation units used by children to parse the speech stream depends on the rhythmic class their language belongs to.

A good indication that newborns are indeed sensitive to rhythmic information was made by [START_REF] Nazzi | Language discrimination by newborns: Toward an understanding of the role of rhythm[END_REF]. They showed that children are able to discriminate two languages (English and Japanese) that belong to two different rhythmic classes but failed to distinguish two languages if they belong to the same rhythmic class (such as English and Dutch, or Italian and Spanish). Infants are however able to discriminate two languages belonging to the same rhythmical class if one of them is their native language [START_REF] Nazzi | Language Discrimination by English-Learning 5-Month-Olds: Effects of Rhythm and Familiarity[END_REF][START_REF] Bosch | Native-language recognition abilities in 4-monthold infants from monolingual and bilingual environments[END_REF]. However, when the prosodic information is manipulated -either by shuffling the words of a sentence [START_REF] Dehaene-Lambertz | Language Discrimination Response Latencies In Two-Month-Old Infants[END_REF], or by manipulating the pitch [START_REF] Chong | Intonation Plays a Role in Language Discrimination by Infants[END_REF]) -children fail to discriminate two languages, even if one of them is their native language. These studies thus effectively show that infants are sensitive to the rythmic properties of their native language, which they can use to discover word boundaries.

Cutler [START_REF] Cutler | The role of strong syllables in segmentation for lexical access[END_REF] showed that English native adults tend to use rythmic information such as stressed syllables as evidence for a probable word boundary. 3 Jusczyk, Cutler & Redanz (1993) investigated if English-speaking infants had a preference for trochaic words -i.e. words that have a Strong-Weak (SW) stress pattern such as "GARden"and showed it was the case, hinting that they could use this information to segment the speech signal, such as what was observed for adults. Yet, this preference was only found for 9-month-old infants but not for 6-month-old infants suggesting such preference might only be a result of the linguistic environment, and hence takes time to develop. In a later study, [START_REF] Jusczyk | The Beginnings of Word Segmentation in English-Learning Infants[END_REF] tested if 7.5-month-old children use trochaic patterns to segment speech and "treat strong syllables as markers of word onsets". To do so, in a familiarisation phase, they embedded words with a Weak-Strong (WS) pattern in a sentence (such as "[...] guiTAR is [...]" or " [...] deVIce to [...]") and later tested if children showed a preference for SW or WS words: "guiTAR" vs. "TARis" and "deVIce" vs. "VIceto". Their results indeed show that 7.5-month-old children prefer "TARis" and "VIceto". This demonstrates, first, that 7.5-month-old infants are sensitive to SW patterns contrary to 6-month-old infants, and second, that they segment the input so that the resulting words have a SW pattern. Hence, 7.5-month-old English-speaking infants use stressed syllables as evidence for word boundaries, such as what was observed for adults.

Similarly to English, Dutch is a language which has more SW-patterned words than WS words. [START_REF] Houston | Crosslanguage word segmentation by 9-month-olds[END_REF] showed that 9-month-old Dutch children are able to 2 A mora is an intermediary unit between the phone and the syllable that determines the syllable's weight. They might be a perfect overlap between the number of morae and the number of syllables as in 建物 ("building") → たてもの /ta.te.mo.no/ with 4 morae and 4 syllables; but it is most often not the case such as in 漢字 ("kanji") → かんじ with 2 syllables (/kan.ýi/) and 3 morae (/ka-n-ýi/) or 東京 ("Tokyo") → とうきょう with 2 syllables (/to:.kjo:/) and 4 morae (/to-o-kjo-o/). The final ん of a syllable and the long vowel う count as single morae.

3 They do so by showing that their test subjects take longer to notice the presence of the word "mint" in MIN-TAYVE (Strong-Strong (SS) pattern) than in MIN-tesh (Strong-Weak (SW) pattern), as in the former, detecting "mint" supposes to reassemble word parts that the listener considered to be two different words.

recognise -and hence segment -words with a SW pattern from fluent speech after having been familiarised to individual words, such as what was observed for English-speaking infants. The same study also shows that English-speaking infants are equally successful in segmenting SW Dutch words, suggesting they apply the SW segmenting strategy to segment other languages than their native language.

Jusczyk, [START_REF] Jusczyk | The Beginnings of Word Segmentation in English-Learning Infants[END_REF] revealed that, at 10 months old, children are able to segment both SW and WS words correctly -words such as "guiTAR" and "deVIce" -showing that even though children initally rely on SW patterns to segment the speech stream, they shift to other cues later on, enabling them to segment WS-patterned words. [START_REF] Thiessen | Learning to Learn: Infants' Acquisition of Stress-Based Strategies for Word Segmentation[END_REF] further showed that English-speaking infants can be trained to segment WS disyllabic words after an habituation phase, revealing the SW pattern only emerges as a result of the linguistic environment, which uses more SW words than WS words. [START_REF] Nazzi | Early segmentation of fluent speech by infants acquiring French: Emerging evidence for crosslinguistic differences[END_REF] explored word segmentation in 8-month-old, 12-month-old, and 16month-old French-speaking infants to see if infants with different languages develop the same abilities at the same age. To do so, infants were familiarised with individual units (e.g. toucan) and in the test phase the infants had the choice to listen to two different passages: one that contained the target word embedded in fluent speech (e.g. [...]toucan [...]) or another one that contained another disyllabic (e.g. putois, polecat) word also embedded in fluent speech ([...]putois [...]). If infants chose to listen more often to the passage containing the target word, it can be concluded they recognised, and hence segmented, the target word embedded in the passage. No evidence of segmentation was found in 8-month-old infants as they did not favour one passage over another, while they seem to segment bisyllabic words at 16 months, as they listened preferably to the toucan passage than the other passage. In between, at 12 months old, infants are able to segment the final syllable of word (can from toucan) only if they were familiarised with can beforehand, but not the whole word. They are able to recognise the initial syllable of a bisyllabic word (tou from toucan) if and only if it is an exact match to the word heard in the familiarisation phase (tou spliced from toucan), but are not able to recognise toucan as a whole. This let [START_REF] Nazzi | Early segmentation of fluent speech by infants acquiring French: Emerging evidence for crosslinguistic differences[END_REF] conclude that French-speaking infants display a systematic pattern of segmentation later than English-or Dutch-speaking infants. The authors also conclude French-speaking infants segment the speech stream using syllables as the basic processing unit. Nonetheless, they also seem to be sensitive to stress as they should otherwise have been able to segment tou without requiring an exact match. Such syllabic segmentation was also observed in children whose native language was Castilian and/or Catalan [START_REF] Bosch | Rapid gains in segmenting fluent speech when words match the rhythmic unit: evidence from infants acquiring syllable-timed languages[END_REF]) which also are syllable-timed languages.

As shown by [START_REF] Nazzi | Segmentation précoce de la parole continue en mots : évaluation inter-linguistique de l'hypothèse d'initialisation rythmique[END_REF] and [START_REF] Nazzi | Early segmentation of fluent speech by infants acquiring French: Emerging evidence for crosslinguistic differences[END_REF], it seems the basic computation unit used by children to segment the speech stream depends on the rhythmic class of their native language: English-or Dutch-speaking infants segment units based on their stress as their language is a stressed-time language, while French-speaking infants' segmentation is based on syllables and not on stress patterns as their language is a syllable-timed language. Hence, while English-or Dutch-speaking infants extract SW patterns, French-speaking infants extract syllable-sized units.

For Japanese, it is less clear what the basic segmentation unit used by children is. Indeed, as Japanese is a mora-time language, the rhythmic segmentation hypothesis would imply that Japanese-speaking infants start by segmenting mora-like units. Even if it is in fact what is observed for adults [START_REF] Otake | Mora or Syllable? Speech Segmentation in Japanese[END_REF], recent works [START_REF] Inagaki | The Effect of Kana Literacy Acquisition on the Speech Segmentation Unit Used by Japanese Young Children[END_REF] suggest that pre-literate children segmentation is part mora-based, part syllable-based; but as literacy increases, segmentation gradually becomes solely mora-based. 4

Phonological Phrases & Utterance Boundaries

Prosody groups words together into units that are prosodically marked which are called phonological phrases (which correspond more-or-less to syntactic-based chunks [START_REF] Abney | Parsing By Chunks[END_REF]. [START_REF] Soderstrom | Six-month-olds recognize clauses embedded in different passages of fluent speech[END_REF] showed that children prefer to listen to clauses that form a single prosodic unit (e.g. "Leafy vegetables taste so good") rather than to the same clause (in terms of the sequence of words) that straddles over two sentences (e.g. "They must buy leafy vegetables. Taste so good helps their families.") and which consequently does not constitute a phonological unit anymore. Hence, children are able to detect phonological phrase boundaries and might use this information to infer word boundaries. [START_REF] Christophe | Discovering words in the continuous speech stream: the role of prosody[END_REF] showed that word recognition in 13-month-old infants is indeed constrained by phonological phrase boundaries. Children effectively recognise a familiarised target word (e.g. paper) if it is part of phonological phrase (e.g. " [The college] [with the biggest paper forms] [is best]"), but not if it straddles over two phonologial phrases (e.g. " [The butler] [with the highest pay] [performs the most]") thus indicating that 10-monthold infants know how to detect phonological phrase boundaries, and that a word may not straddle two phonological phrases. Prosodical phrase boundaries may be signalled by a pause as it is (theoretically) the only legal place where one can pause to catch one's breath (Di Cristo 2013, p. 15). However, most of the time, such boundaries are signalled by other acoustic cues, such as pre-boundary vowel lengthening [START_REF] Vaissière | Language-Independent Prosodic Features[END_REF] which could be a universal cue to detect prosodical phrase boundaries, or other allophonic cues which will be presented in the following section. Therefore, by paying attention to phonological phrases -whose "primary purpose is to segment statements and discourse into groups of meanings" Di Cristo (2013, p. 65) -children are able to posit word boundaries.

Another prosodic cue infants use to segment the speech stream is to pay specific attention to utterances-edges, 5 a strategy called "Edge Hypothesis". In their study, [START_REF] Johnson | The Edge Factor in Early Word Segmentation: Utterance-Level Prosody Enables Word Form Extraction by 6-Month-Olds[END_REF] showed that children as young as 6 months old that hear a novel word placed either at the beginning and/or at the end of the familiarisation utterance recognised the target word when heard in isolation, but not if this word appeared sentence medially. This shows that infants consider utterance boundaries as evidence for word boundaries and are able to effectively use this information afterwards.

Conclusion on Suprasegmental Cues

All in all, by paying attention to prosodic features, the child is able to posit boundaries in the speech signal. According to Di Cristo (2013, p. 63) children's sensitivity to rythmic cues "[is] likely to favour the establishment of routines, which will help identify word boundaries". Similarly, he argues that even though the main function of prosody is not to give direct cues to word boundaries, it gives a "metric framework" that the listener can use in order to posit word boundaries, or at least, posit boundaries for large linguistic units such as chunks, which necessarily correspond to word boundaries.

Segmental Cues

Contrary to the previous section where we presented suprasegmental cues, this section is concerned with segmental cues, that is, cues directly linked to the segments (i.e. phone[me]s) 1.2. Speech Segmentation 13 of a language. As [START_REF] Mattys | Integration of Multiple Speech Segmentation Cues: A hierarchical Framework[END_REF], we group under the term of segmental cues both phonotatic cues and acoustico-phonetical cues.

Phonotactics

Phonotactics refers to the study of the sound sequences of a language. Some sound sequences are said to be legal (i.e. allowed) while some are illegal (i.e. not allowed) in a given language. What constitutes (or not) a legal sequence depends on the language, and sequences that are legal in one language might be illegal in another language. For example, English allows very complex sound sequences within one syllable (such as CCCVCCCC 6 as in "strangles" /stôaeNglz/7 ) while such a sequence is not allowed in French. Moreover, some sequences might only be allowed at certain positions, but not at others (for example, /N/ is only allowed word-finally in French). All these constraints provide the child with potential cues to infer word boundaries.

First, let us consider phonotactics at word level only. [START_REF] Jusczyk | Infants' Sensitivity to the Sound Patterns of Native Language Words[END_REF] showed that 9-month-old children preferred listening to word lists whose words matched the phonotactic regularities of their native language than to a word list that violated these regularities, hence showing that infants are sensitive to the phonotactics of their language and intuitively know what sequences are legal. [START_REF] Jusczyk | Infants' Sensitivity to Phonotactic Patterns in the Native Language[END_REF] further showed that within the same language, 9-month-old children prefer a word list that contains words with very frequent sound patterns over a list consisting of words with infrequent sound patterns.8 Thus, by knowing which sound patterns are legal or not, the child can immediately know if the candidate sound sequence she has isolated may or may not constitute a word. If it does contain an illegal sequence, then it should not be regarded as a single sequence, but should be further segmented or discarded. This is in fact what was observed by [START_REF] Mackenzie | 12-Month-Olds' Phonotactic Knowledge Guides Their Word-Object Mappings[END_REF]: 12-month-old infants are able to associate English CVCV (e.g. panu), Japanese CVCV (e.g. hashi ) sequences to novel objects. They are also able to do so with English CCVC non word (e.g. plok ) but not Czech CCVC (e.g. ptak ) sequences as the /pt/ sequence is illegal as a word onset in English, while /pl/ is not.

The phonotactic regularities the child uncovers are not restricted to adjacent units, but may also include non-adjacent units. Also, the regularities the child is able to discover is not only limited to legal/illegal patterns but also includes frequent/infrequent phonotactic patterns. For example, [START_REF] Bibliography Nazzi | A perceptual equivalent of the labialcoronal effect in the first year of life[END_REF] put forward a Labial-Coronal (LC) bias in speech perception in French infants: they prefer to listen to pseudo-words in which a labial consonant comes before a coronal consonant -LC words such as /bude/ -than the reverse pattern, that is, CL words such as /deby/; as LC words are more frequent in French than CL words. This bias toward LC words has been shown to also constrain word learning: [START_REF] Gonzalez-Gomez | A "Bat" Is Easier to Learn than a "Tab": Effects of Relative Phonotactic Frequency on Infant Word Learning[END_REF] indeed showed that 14-month-old children are able to learn a mapping between a novel object and a word with a LC pattern (e.g. "bat"), but are unable to do so when the label of the word has a CL pattern (e.g. "tab"). It is only later on, at 16 months that they are able to learn both. Hence, not only children detect which phonotactic patterns are the most frequent, but the frequent patterns condition which words are learnt first.

Once children have enough information about the canonical shape of the individual words of their language, they can apply this information to segment the speech stream. Friederici & Wessels (1993) showed that 9-month-old Dutch-speaking infant could detect Chapter 1. Background on Child Lexical Acquisition legal and illegal sound sequences in connected speech (i.e. they show a preference towards the passage that has legal phonotactic patterns) provided certain conditions are met.9 Mattys et al. (1999) showed that English-speaking infants know what sequences are legal (or more probable) between words and within words. Infants presented with CVC•CVC10 disyllabic words accented with a SW pattern prefer sequences where the internal C•C is more probable to appear within words than between words.

A real evidence that infants do use phonotactic patterns to segment speech into sub-units was made by Mattys & Jusczyk (2001b). In their experiment they show that 9-month-old children are able to recognise a CVC word spoken in isolation if it was previously heard within a sentence (i.e. [...]C•CVC•C[...]) so that the phonotactic patterns provide good indications as to where the edges of the words are (i.e. both initial and final C•C have consonants that have a low probability to co-occur within the same word). When the word is embedded in a sentence where the phonotactic patterns do not make this obvious (i.e. initial and final C•C have a high probability to occur within a word), infants fail to show preference for the target word. Children are also able to detect the target word even if it is only the onset or the offset of the word that displays obvious phonotactic patterns signalling a boundary. Hence, word boundaries need not be phonotactically signaled from both ends in order to be correctly segmented.

Consequently, the child is able to detect the phonotactic regularities of its languages and uses this information to segment the speech stream into words and evaluate if the segmented pattern constitutes a good word candidate or not.

Vowel Harmony

Vowel harmony is found in many languages (e.g. Turkic languages, Finno-Ugric languages, etc.) and places a hard constraint on the form of the words. It may be considered as a special type of non-adjacent phonotactic constraint that only affects vowels within one word. Vowel harmony consists in an assimilation by which all the vowels of a word share a common feature (either rounded/unrounded, front/back, etc.).11 Therefore, if the child succeeds in discovering that its mother tongue has vowel harmony, it could use such cue to segment the spoken input. [START_REF] Ketrez | Harmonic cues for speech segmentation: a cross-linguistic corpus study on child-directed speech[END_REF] indeed showed that when vowel harmony is broken, it most likely signals a word boundary, and suggest that children might use this property so infer word boundaries. [START_REF] Mintz | Infants' sensitivity to vowel harmony and its role in segmenting speech[END_REF] showed in an artificial language learning (ALL) experiment that children indeed segment words based on vowel harmony. 6-to 7-month-old English-speaking infants where familiarised with the sequence pidigitokobogetepedubuku composed of four different words: pidigi, tokobo, getepe, and dubuku 12 which was repeated 45 times on a loop. Their results show that infants indeed segmented sequences corresponding to words (e.g. toboko) and not to part-words (e.g. bukupi, or gitoko). In a further experiment, they observe that infants can also segment words with a more complex harmony pattern. After hearing a sequence ditepubobidetupo consisting of four words with either front (dite or bide) or back harmony (pubo or tupo), children in the test phrase listen longer to words (e.g. dite) than to part-words (e.g. detu) showing the children extracted disyllabic pseudo-words based on their vowel harmony pattern. This result is coherent with other research [START_REF] Hohenberger | Sensitivity of Turkish infants to vowel harmony, in 'The Acquisition of Turkish in Childhood[END_REF] indeed showed that Turkish children as young as 6 months old prefer to listen to harmonic words over non-harmonic words. However, it seems that in their study children were only sensitive to the front/back harmony and not to the rounded/unrounded harmony.

All in all, these studies show that from a fairly early age, infants are sensitive to subtle differences such as vowel harmony and are able to use this information in order to segment the speech stream. More generally, it shows that infants are able to detect non-adjescent regularities "where X and Y are separated by intervening, unpredictable elements -such that listeners might be exposed to XAY, XBY, XCY -participants are able to learn that X predicts a following Y" [START_REF] Thiessen | Statistical learning[END_REF]; vowel harmony being only a special case of non-adjescent pattern. [START_REF] Newport | Learning at a distance I. Statistical learning of non-adjacent dependencies[END_REF], however, showed that children are able to keep track of non-adjacent regularities at phoneme level, but not a syllable level.

Allophones & Coarticulation

Allophones are sounds that correspond to different acoustic realisations of a given phoneme. Context usually governs the choice of one variant over another (i.e. in some cases the variant is systematically realised). For example, English /p/ may be realised [p] (after /s/ as in "spark"), [p h ] (word-initially as in "park"), or [p^] (word-finally as in "pop music"). Hence, if the child has learnt the rules that govern the appearance of one variant over another, it can use this information to segment the speech stream (e.g. boundary before [p h ], boundary after [p^], etc.).

A first indication that infants are sensitive to allophonic variations was made by [START_REF] Christophe | Do infants perceive word boundaries? An empirical study of the bootstrapping of lexical acquisition[END_REF]. They showed that 3-or 4-day-old children are able to detect a difference between a CVCV disyllabic sequence spliced from a word (e.g. /mÃta/ in sédimentation) and the same CVCV sequence straddling over a boundary (e.g. /mÃ#ta/ in déguisement talentueux ). Therefore, they are able to differentiate two different realisations of the same phonological sequence. For English, [START_REF] Hohne | Two-month-old infants' sensitivity to allophonic differences[END_REF] showed that while 2-month-old English infants are able to distinguish two sequences based on allophonic cues (e.g. "nitrate" [naIú h ô ˚eIt^] and "night rate" [naIt^ôeIt^]), it is only at 10.5 months they are able to effectively use this information to segment the speech stream [START_REF] Jusczyk | Infants' sensitivity to allophonic cues for word segmentation[END_REF]. Mattys & Jusczyk (2001a) showed how sensitivity to allophonic cues helps children in segmenting words and are not simply responding to acoustic patterns. First, 8.5-month-old children were familiarised with a set of isolated words (e.g. dice). In the test phase, infants were presented with passages that either contain the target word (e.g. Two dice can be rolled without difficulty.) and other passages that do not contain the target word. Results show that children listen significantly longer to the passage containing the target word, hence showing they did recognise the word they had only seen in isolation in connected speech. In a similar experiment, children are once again familiarised with isolated words (e.g. dice [daIs]). However, this time in the test phrase, infants are presented with passages that either contain a sequence in which the target word is split over two words (e.g. d#ice [d^PaIs] as in The rink had been sprinkled with spread ice) and two passages that do not contain the target sequence. If infants listen longer to the passage that contains the target sequence split over two words, it means they would have incorrectly recognised dice in weird ice. If not, it would mean they have identified an allophonic variation that signals a word boundary. Results confirm it is not the case, hence showing that they noticed the word boundary in the similar sounding sequence (weird#ice). These set of experiments thus confirm that allophonic cues are used by children in order to segment the speech stream into words.

Chapter 1. Background on Child Lexical Acquisition

Coarticulation refers to the fact that a segment is never exactly pronounced the same way given the preceding and following sounds. Indeed, sounds tend to be blended together. Nonetheless, the "level" of blending depends on several factors. [START_REF] Johnson | Word Segmentation by 8-Month-Olds: When Speech Cues Count More Than Statistics[END_REF] indeed note that there is "less overlap of adjacent sound segments [...] in word-final and word-initial consonant articulations belonging to different prosodic domains". Consequently, if two phonemes display a low level of coarticulation, it might be they belong to two different words. Thus, such information might help the child segment its input. [START_REF] Curtin | Coarticulatory Cues Enhance Infants"Recognition of Syllable Sequences in Speech[END_REF] showed in an ALL task that 7.5-month-old, are able to recognise isolated word-like patterns if these isolated patterns had coherent coarticulation effects between syllables, but could not if the syllables comprising the patterns where spliced in such a way that the coarticulation patterns where incoherent. This thus shows that children encode coarticulation information when remembering a new word and re-use this information later on to recognise words in the speech stream.

Transitional Probabilities

Children may also use statistics in order to decide if there is a word boundary between two adjacent syllables or not. If two syllables frequently co-occur, it seems reasonable to treat them as belonging to a single unit than to two different units.

In an ALL task, [START_REF] Saffran | Statistical Learning by 8-Month-Old Infants[END_REF] tested if 8-month-old infants were sensitive to statistical (ir)regularities, and more specifically transitional probabilities (TP) 13 in order to segment the speech stream into words. To do so, they assembled a set of four artificial words (pabiku, tibudo, golatu, and daropi ) so that the whole sequence would be two minutes long. The sequence was synthesised so that only statistical cues might be used (i.e. monotone voice, no stress, no pause, etc.). Because the words were repeated in a random order, the TP between two syllables of the same word is always 1 (e.g. bi is always followed by ku) while the TP of two syllables belonging to different words is 0.25 (e.g. ku might be either followed by ti [budo], go[latu], da [ropi] or pa [biku]). Hence, if children are indeed sensitive to the transitional probabilities of the syllables, they should consider the aforementioned words as valid units, but not tudaro, pigola, etc. It is in fact what the authors observe, letting them conclude that children do use transitional probabilities between syllables as a cue for word segmentation.

Even though the previous experiment does show that children are able to use TP to segment speech in a relatively short time, the input does not reflect the actual difficulty of real life language. Pelucchi et al. (2009b) hence tested if 8-month-old English-speaking infants were able to segment speech based on TP when presented with Italian words (e.g. fuga, getaway) embedded in grammatically and idiomatically correct Italian sentences. The sentences were constructed in such a way that the TP between fu and ga was always 1 (i.e. there were no other occurrences of the syllable fu except in the word fuga). Another real Italian word was used in the sentences (bici, bike), however this time the syllables bi was also used to construct other words (bira, bigia, bivio, etc.); hence, the TP between bi and ci is lower than 1. 14 At test time, if children indeed used TP to segment speech into sub-units, they should consider fuga as a real unit while bici should be discarded. It is in fact what is experimentally observed, therefore suggesting that using TP is not only used by children in ALL task, but also when confronted to real speech.

Not only are children able to keep track of forward transition probabilities (FTP), but

13 Probability that the next syllable is Y given the fact that the current syllable is X is given by T P = P (Y |X) = f req.XY f req.X . 14 Such as in this sentence: "La cavia Bida è in fuga da casa per aver giocato con le bilie blu". (The guinea pig Bida is on the run from home for playing with the blue balls) research suggest they can also keep track of backward transition probabilities (BTP). 15 To study this, Pelucchi et al. (2009a) embedded four target words into sentences in Italian (such as Pelucchi et al. 2009b): fuga, melo, bici, and casa. In all cases, FTP(Y|X) = P(ga|fu) = P(lo|me) = P(ci |bi ) = P(sa|ca) = 1 (i.e. no other words than the target words start by fu, me, bi, or ca). However, BTP probability was different for half of the words: in one case BTP(X|Y) = P(fu|ga) = P(me|lo) = 1 (i.e. no other words than fuga or melo have the final syllable ga or lo), while in the other case BTP(X|Y) = P(bi |ci ) = P(ca|sa) < 1 (i.e. other words than bici and casa end with ci or sa). If infants also pay attention to BTP, they should consider fuga and melo as better word candidates than bici and casa, as the former have both FTP and BTP = 1, while the latter have a BTP < 1. It is what is observed experimentally, letting the author conclude that infants do keep track of both FTP and BTP, and that such knowledge is used for multiple tasks, one of them being word segmentation. [START_REF] Estes | Can Infants Map Meaning to Newly Segmented Words?[END_REF] showed that children may segment a continuous speech stream into words using TP without necessarily immediately assigning a meaning to the segmented units. First, 17-month-old children were exposed to an artificial language of four words (e.g. timay#dobu#gapi#moku) for which the only cues for word segmentation were TP. Then, the infants had to learn a mapping between a novel object and a novel word that was either one of the words used to build the initial sound sequence (e.g. timay), or a novel word that did not occur in the sequence (e.g. nomay), or a part word that was made up of two word parts (e.g. pimo). The only children that managed learning a novel word-object mapping were those for which the novel word was one used to construct the sequence. Hence, the authors conclude that not only children used TP to segment the speech stream, but that prior exposure to word facilitates subsequent word-referent mapping. This result is coherent with previous results: [START_REF] Jusczyk | Infants' Detection of the Sound Patterns of Words in Fluent Speech[END_REF] indeed showed the child's proto-lexicon might also only contain word-forms that are not associated to any meaning at first.

Lexical Cues

Children may use lexical cues in order to parse the spoken input and segment it. Indeed, as soon as the child has isolated a certain number of word-forms, she can use them in order to isolate the words that precedes and/or follows the known word. [START_REF] White | Segmentation by lexical subtraction in Hungarian speakers of second-language English[END_REF] coined the term "segmentation by lexical subtraction" and defines it as "the use lexical knowledge to impose a segmentation structure on the speech input". 16Jusczyk & Aslin (1995) showed that 6-month-old children are able to detect known words in the speech stream if they already heard the same word in isolation. Further study by [START_REF] Jusczyk | Infants' Memory for Spoken Words[END_REF] observe that infants are able to remember words for relatively long time spans (as long as two weeks in their study). Thus, isolated words may be used by children to segment the speech stream. [START_REF] Brent | The role of exposure to isolated words in early vocabulary development[END_REF] estimate that 9% of infant directed speech are isolated words. [START_REF] Bortfeld | Mommy and Me: familiar names help launch babies into speech-stream segmentation[END_REF] later showed that 6-month-old infants already used familiar words ("mommy", "daddy", or the child's own name) to segment the following word in the speech stream. Indeed, as soon as the familiar word is recognised, its end necessarily constitutes the beginning of the next word. [START_REF] Bortfeld | Mommy and Me: familiar names help launch babies into speech-stream segmentation[END_REF] state (without experimentally testing) that the familiar words might not be restricted to those already mentioned, but might also include other familiar words such as "diaper" or "bottle" that are part of the child's everyday life. [START_REF] Bergelson | At 6-9 months, human infants know the meanings of many common nouns[END_REF] later showed that other common words known by children included words refering to body parts ("feet", "hands", "mouth", or "eyes") or refering to food ("yogurt", "banana", or "milk"). Hence, all these words might serve as support for the child to segment the speech stream into sub-units.

Other types of words such as functional words (i.e. closed-class words such as determiners, conjunctions, prepositions, etc.) are also used by the child to segment the speech stream. For example, [START_REF] Shi | Frequency and form as determinants of functor sensitivity in English-acquiring infants[END_REF] showed that the determiner "the" facilitates the recognitions of a test pseudo-word while non-sense words ("kuh" and "ler") or a least frequent determiner ("her") do not. A similar effect was found for 8-month-old French-speaking infants by [START_REF] Shi | The effect of functional morphemes on word segmentation in preverbal infants[END_REF] where infants better recognised a novel target word in isolation if it was heard during the familiarisation phase either with a highly frequent determiner before ("des") or less frequent determiner ("mes"), but not if it was presented with a non-sense similar-sounding word ("kes"). They however note that this effect is not observed with all determiners as they do not observe any difference when the target word is presented with "vos" or with "kos", thus showing that in order to be used, the determiners should be frequent. [START_REF] Höhle | Functional Elements in Infants' Speech Processing: The Role of Determiners in the Syntactic Categorization of Lexical Elements[END_REF] found a similar behaviour for 14-month-old German infants. Finally, [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF] ran a similar analysis on Japanese and the use of particles by children. Contrary to Indo-European languages, particles are postpositions and are used to indicate the function of the preceding word (subject, object, etc.). They found that 15-month-old Japanese toddlers use particles, and more specifically the "ga" particle which is the subject particle, in order to segment speech. Thus, the pattern of using functional words as cues for word segmentation seems to be valid cross-linguistically.

Finally, [START_REF] Johnson | The Edge Factor in Early Word Segmentation: Utterance-Level Prosody Enables Word Form Extraction by 6-Month-Olds[END_REF] report in their study that interjections (such as "oh", "no", "wow") account for 80% of the words heard in isolation in child directed speech. Such words, that do not serve any grammatical function but act merely as phatic and conative interjections [START_REF] Ameka | Interjections', Concise encyclopedia of grammatical categories[END_REF]), might help the child segment the speech stream.

Therefore, children use the lexical knowledge they have to parse the speech stream and acquire novel word. However, as noted by [START_REF] Curtin | Speech perception[END_REF], "familiar words are not enough for infants to excel at segmentation". Indeed, in order to fully segment the speech stream, the child should develop other routines such as those already discussed thereafter.

Other cues

Another cue that children could use in order to infer the lexicon of their language derives from the nature of the spoken input they hear. Indeed, Child Directed Speech (CDS) 17 has certain properties (at various levels: acoustics, syntax, etc.) which are not found in Adult Directed Speech (ADS). [START_REF] Fernald | A cross-language study of prosodic modifications in mothers' and fathers' speech to preverbal infants[END_REF] reports for example that both mothers and fathers have a higher fundamental frequency (F0) with exaggerated pitch contours and long pauses between utterances. CDS is also reported to have a slower rate (syllable/seconds). Nevertheless, [START_REF] Church | Infant-directed speech: Final syllable lengthening and rate of speech[END_REF] reports that this is to be explained by the elongated preboundary final vowel lengthening that is greater in CDS than in ADS. If this final syllable is excluded in the speech rate calculation, CDS and ADS speech rates are similar. [START_REF] Johnson | The Edge Factor in Early Word Segmentation: Utterance-Level Prosody Enables Word Form Extraction by 6-Month-Olds[END_REF] also reports a higher proportion of isolated words in infant-directed speech (IDS, up to 90-days-old infants) than in CDS (i.e. from 90 day old up to 2.5 years old) and the proportion decreases again in ADS. Their study also reveals that nouns are more likely to appear at utterance-edges in IDS than in CDS or ADS.

It should be noted that IDS/CDS is not universal, as in some cultures children are not (or barely) addressed to until they are able to speak [START_REF] Cristia | Child-Directed Speech Is Infrequent in a Forager-Farmer Population: A time Allocation Study[END_REF] or are addressed to like adults. Thus, this type of speech is not mandatory for children to learn their language, but seems to help the child when it is used. [START_REF] Nelson | How the prosodic cues in motherese might assist language learning[END_REF] indeed showed that 8-month-old infants prefer listening to speech excerpts in CDS that preserved clausal boundaries over excerpt in CDS that do not preserve clausal boundaries. They showed no such preference for ADS, whether clausal boundaries where respected are not. Infants thus seem to be able to better detect what a legal prosodical unit is when CDS was used but not when ADS was used. Therefore, CDS seem to provide useful cues to the child in order to segment the spoken input, at least at clausal level. [START_REF] Thiessen | Infant-Directed Speech Facilitates Word Segmentation[END_REF] further tested in a ALL task if CDS was easier to segment for 7.5month-old infants than ADS. To do so, they had infants listen to a sequence of words (dibo, kuda, lagoti, nifopa) whose only cue to word boundaries was the probability of co-occurence of each syllable. 18 The sentences were read both as CDS and ADS by the same speaker. In the test phase, children payed more attention to well-formed word (e.g. lagoti ) than to part-words (i.e. gotini or dalago) when having been familiarised with CDS, but show no difference when familiarised with ADS. Hence, the authors conclude that CDS indeed improves word segmentation for young children. [START_REF] Ma | Word Learning in Infant-and Adult-Directed Speech[END_REF] explored if CDS facilitated word learning compared to ADS. In their experiment, 21-month-old children were presented with novel objects (displayed on a screen) and were told the name of the novel object in a familiarisation passage (e.g. modi in the passage "Look here! It's a modi ! See the modi. That's the modi [...]" ) either in CDS or ADS. During the test phase, [START_REF] Ma | Word Learning in Infant-and Adult-Directed Speech[END_REF] checked if the children looked longer to the object that was presented as a modi than to the distractor object when hearing the sentence "Modi. Where's the modi ". Children that were familiarised with a CDS passage showed such a pattern, but not those that were familiarised with the ADS passage.

Hence, this body of work suggests that, even though CDS is not found in every culture, it enhances the ability of children to extract words from the speech stream. Di Cristo (2013, pp. 172-173) indeed states "the repetitive use of lexical stress in the speech addressed to the child helps to fix the child's attention and at the same time helps the child to identify word boundaries". Not only IDS/CDS help children identify word boundaries, they also help them map the extracted word-form to their referents. For Di Cristo (2013, p. 175), it may be because "the effect of the accentual function is to emphasise the element that is the subject of this prosodic distinction [... making] it possible to establish a link between a phenomenon of physical or perceptual salience and the promotion of a mental or cognitive salience."

Multiple Cues: from Mutual Exclusion to Combination

The previous sections presented the numerous cues children may use to segment the speech stream. Yet, not all cues are available nor used at the same time. Some cues are only available from a certain age, such as phonotactic cues as they require some amount of exposure to the target language, while some are available since the child was born, such as the ability to detect allophones. Similarly, even if multiple cues may be used at the same time, research shows that children might favour some cues over others at a certain point of their development. [START_REF] Johnson | Word Segmentation by 8-Month-Olds: When Speech Cues Count More Than Statistics[END_REF] tested in an ALL task if 8-month-old infants relied more on coarticulation or statistical cues to decide whether there was a boundary or not. The same Chapter 1. Background on Child Lexical Acquisition sequence of words as [START_REF] Saffran | Statistical Learning by 8-Month-Old Infants[END_REF] was used (...pabiku#tibudo#golatu#daropi..., see §1. 2.3.4) unless this time, instead of splicing a part-word (e.g. ku#tibu) from the original sequence, it was resynthetised. Contrary to the previous experiments of [START_REF] Saffran | Statistical Learning by 8-Month-Old Infants[END_REF], the new part-word had a coherent coarticulation pattern at word boundary -or what should theoretically be considered a word boundary from statistics alone (i.e. between ku and ti ). Hence, according to the coarticulation pattern, there should not be a word boundary while according to statistical cues there should be. If children favour the part-word this time, it means that they discarded the TP information and only considered the acoustic cues. They results show that infants consider ku#tibu as a valid word and hence favoured coarticulation information over statistical information. [START_REF] Mattys | Phonotactic and Prosodic Effects on Word Segmentation in Infants[END_REF] similarly showed that when phonotactic cues and prosodic cues provide conflicting information about the word boundary, 9-month-old infants only rely on the information provided by the prosodic cues. They favour a disyllabic CVC•CVC SW sequence with a frequent between-word C•C internal cluster over a disyllabic WS sequence with frequent within-word C•C internal cluster. Hence, more weight is given to prosodic cues than to phonotactic cues. [START_REF] Jusczyk | The Beginnings of Word Segmentation in English-Learning Infants[END_REF] revealed that while 7.5-month-old children are unable to segment WS words, 10-month-old children are able to do so. They cannot therefore do it using SW pattern as a cue, as the word that is extracted does not have this accentuation pattern. Consequently, at some point, infants need to discard the information provided by the primary (i.e. first used) cues and integrate other cues to segment the speech stream. Such pressure could arise from what is known as the Possible Word Constraint (PWC, see [START_REF] Norris | The Possible-Word Constraint in the Segmentation of Continuous Speech[END_REF]) which states that segmentation should leave a low number of un-analysed chunks (ideally none) and that the chunks resulting from the segmentation process should be existing words. [START_REF] Johnson | Lexical viability constraints on speech segmentation by infants[END_REF] showed that segmentation in 12-month-old infants is indeed conditioned by such PWC. Hence, in the "guiTAR is" experiment, the segmentation that conforms the best to the PWC is "guiTAR#is" despite the WS pattern and not "gui#TARis" as the latter leaves a non attested word "gui".

Finally, segmentation cues may also be used in conjunction rather than in isolation. There are infinitely many ways of segmenting a speech signal into sub-units, but the multiple cues children have access to constrain the possible segmentations. Indeed, "constraints should make a 'combinatorial explosion' less likely." (Thiessen & Erickson 2009, p. 44).

In the previously described experiments of Pelucchi et al. (2009b,a), where children found disyllabic words embedded in fluent Italian sentences, the Italian speaker was asked to read the sentences with a "lively voice, pretending to be in front of a baby". Hence, it might also be the combined effect of TP and CDS that helped children segment the spoken input, and not TP alone.

The use of TP to segment the speech stream also seems effective when combined with another cue. [START_REF] Johnson | Testing the limits of statistical learning for word segmentation[END_REF] showed that 5.5-and 8-month-old children do manage to extract words from the speech stream using TP if the words are uniform in length. However, when the words are not uniform in length (three-syllable and two-syllable long), infants do not seem to be able to segment the speech stream correctly. Therefore, this suggests that in order to be used effectively, TP should be combined with other cues, as real life language has words with a different number of syllables.

A similar result as [START_REF] Johnson | Testing the limits of statistical learning for word segmentation[END_REF] was observed by [START_REF] Mersad | When Mommy Comes to the Rescue of Statistics: Infants Combine Top-Down and Bottom-Up Cues to Segment Speech[END_REF]. In their experiments, they tested if 8-month-old French-speaking infants could segment an artificial language consisting of words of non uniform length (e.g. respectively bisyllabic words pabi, tibu, and mãma and trisyllabic words golatu, daropi ) where TP were the only cues to properly segment the speech stream. As in [START_REF] Johnson | Testing the limits of statistical learning for word segmentation[END_REF], French infants were not able to segment the speech stream into words. Yet, when the pseudo word mãma was replaced with mamã ("mum"), the infants succeeded in segmenting the speech stream, even if it consists of non-uniform words. Consequently, the authors suggest that children were able to segment the speech stream using these two cues in conjunction. Indeed, if TP alone had been enough, the infants should have managed to segment the speech stream in the mãma condition, which is not the case. The authors thus suggest that infants first perform a segmentation by lexical subtraction (see Section 1.2.4) -therefore discovering also the beginning and end of the other words -and then used TP to segment the rest: with the help of these two cues, the child managed to segment the speech stream, when if only one cue or the other had been present, the child would not have been able to do so.

Conclusion on Speech Segmentation

In this section, we presented the multiple cues children use in order to segment the speech stream. We showed that children are sensitive to suprasegmental cues (e.g. prosody), segmental cues (e.g. phonotactics), and subsegmental cues (e.g. coarticulation). Once segments have been isolated, they can then use statistical cues (e.g. TP) to further segment the speech stream. Additionally, we showed that children also make use of isolated words in order to segment the speech stream (segmentation by lexical subtraction) and we highlighted the benefits of using CDS over ADS.

It thus appears that children use a combination of top-down (e.g. isolated words) and bottom-up cues (e.g. phonotactics, TP, etc.) in order to segment the speech stream. Both approaches are necessary -as, for example, discovering the regular phonotactic patterns within words requires the knowledge of a set of individual words -and complementary -as in return, the inferred phonotactic patterns enable the learner to segment even more words.

We will conclude this section by introducing the Hierarchical Framework by [START_REF] Mattys | Integration of Multiple Speech Segmentation Cues: A hierarchical Framework[END_REF] which shows how the various cues we presented are organised in a mature segmentation system (Figure 1.1). As can be seen, the cues that are the most important are lexical cues. That is, lexicality places such a hard constraint on word segmentation (with the Possible Word Constraint), that it is the cue that is used primarily by adults to segment the speech stream. If, for some reason, the lexical level is not directly accessible, when for example the speech signal is noisy, then adults fall back on non-lexical cues to parse the speech stream (segmental cues, word stress).

Contrary to adults, speech segmentation in children is a developing system, and seems to be bootstrapped by the cues that adults tend to use the less favourably (i.e. prosodic cues). However, we believe that this diagram should also include an arrow at the top, as we have seen that children also hear a significant proportion of words in isolation (up to 9% such as mommy, bottle, etc.) and even more if interjections are counted. 19 Johnson et al. (2014) reports that 29% of the words in CDS are interjections and 80% of the words pronounced in isolation are interjections. These words are extremely frequent and can be used by the child to bootstrap speech segmentation. Hence, speech segmentation in a developping system uses, as in a mature system, both top-down and bottom-up cues, the only difference being the relative importance given to each approach. Given the little vocabulary children have, bottom-up cues are used more favorably than top-down cues.

This review allowed us to observe the different strategies used by children to segment the spoken input into sub-units. We will explore in this thesis if a neural model of visually grounded speech also segments its input into subunits as children do. [START_REF] Mattys | Speech segmentation[END_REF] presenting the cues used in a mature speech segmentation (i.e. adult) system and in a developing system (i.e. child) with the black arrows illustrating the back and forth interactions between top-level cues (semantics, lexical) and low-level cues (prosody). The grey triangle indicates the importance of each set of cues (with the wider end signalling the greater importance of such cues).

Word Mapping

In the previous sections, we presented the cues children use to segment the speech signal into sub-units. However, we somehow considered language in vacuo as if words existed per se, disconnected from any context. This is of course not the case:

The important point in all this is that language only has existence with respect to the physical and mental order of things. Language is not any sort of independent system, to be tapped into as required. Aspect of language have significance only as they relate to aspects of the world. (Dixon 2012, p. 434) This point, though obvious, should be mentioned as the NLP community often tends to consider language as having an existence of its own, disconnected from the real world. We will further discuss this point in the following chapter (see Section 2.1.3).

In this section, we highlight how children assign a meaning to the extracted word-forms they have isolated in the speech signal, and show why context is necessary to acquire a language. Recent research show, that the more often a word is heard in different contexts, the better it is learnt by the child [START_REF] Roy | Predicting the birth of a spoken word[END_REF]. Hence, contextual information plays a decisive role in lexical acquisition. We will mainly focus on the visual modality, as it is the modality that is used by our model, and show how it influences language acquisition.

Prerequisites

Once children have isolated a certain number of word-forms, they should assign them a meaning. Meaning is not a ready-to-be-used object that the child would only have to map to a given word-form; but rather should be constructed by the child on the basis of its experience with the physical world.

Hence, assigning each word-form a meaning is only the last step. The first step consists in finding the referent of each word-form. Finding the right referent can only be done in context, as a given word-form relates to a given object in the physical world, or a given percept. Potentially, the referent may change from one occurence of a word-form to another. For example, the word-form "dog" might be uttered when refering to different dogs, each one being a referent of the word-form "dog" when it was uttered. Once the child has infered the referents of a given word-form, it should find the commonalities shared by the referents in order to abstract a potential meaning.

However, finding which is the referent of a word-form is far from an easy task. Indeed, there are an infinite number of possibilities the child has to choose from (see Quine's famous gavagai example, [START_REF] Quine | Word and object / Willard van Orman Quine[END_REF]). The problem is even trickier if one thinks of abstract words such as pride, or love for which the referent is not perceptible and refers to inner sensations. A purely associanistic approach, where word-form/referent co-occurence alone would solve the problem, seems unrealistic. Indeed, if word learning were only done using cross-situational statistics, then children would necessary have to hear a word several times in order to learn it, which is not what is observed in practice. Indeed, children are able to do fast-mapping [START_REF] Carey | Acquiring a Single New Word[END_REF], that is, learn to quickly map a word-form to its referent and derive a meaning with a very few number of examples (in some cases only one example). This led researchers such as Bloom to argue that "statistical covariation between word and percept is neither necessary nor sufficient for word learning." (Bloom 2002, p. 59). Therefore, word/referent mapping and ultimately the acquisition of meaning seems to require something more.

Theory of Mind and Shared Attention

For Tomasello (2009, p.3), acquiring a language involves two skills: pattern-finding, and intention reading. Pattern-finding involves all the statistical computations the child makes in order to segment the speech stream, as well as recurrent sequence detection and extraction in the speech stream (which we presented in the past sections). Intention reading is, according to Tomasello, the necessary ingredient for children to acquire their language, and particularly when it comes to inferring what is the referent of a given word-form. [START_REF] Bloom | How Children Learn the Meanings of Words, A Bradford Book[END_REF] similarly argues that "children use their naive psychology or theory of mind to figure out what people are referring to when they use words". Theory of mind could be defined as the capacity humans have to infer the mental states of others and relate them with their own. This allows them to interpret and understand the emotions, intentions, beliefs, and behaviours people around them have [START_REF] Astington | Theory of Mind[END_REF]. Specifically for language, having a theory of mind is necessary to appropriately contextualise what is said. Hence, Bloom argues that children are able to do fast-mapping only because they have a theory of mind.

Having a theory of mind enables children to enter in shared attention frames (also called joint attention frames). A shared attention frame may be defined as "a triadic episode of interaction involving a caregiver, an infant/toddler and an object" [START_REF] Rudd | Joint Focus of Attention, in 'Encyclopedia of Child Behavior and Development[END_REF]. To be more precise, there is shared attention when both the child and the caregiver are attending to the same object, while being mutually aware of what the other is attending to (see Figure 1.2).

According to (Bloom 2002, p. 46), if having a theory of mind is a necessary prerequisite for building a lexicon, there should be a correlation between the onset of word learning and the first evidence of the child having a theory of mind. [START_REF] Morales | Following the direction of gaze and language development in 6-month-olds[END_REF], cited by Bloom 2002) observe that in six-month-old infants, the ability to follow the caregiver's gaze -and hence initiate a shared attention state -is a good predictor of their receptive vocabulary at 12 months. This result is coherent with previous findings: [START_REF] Tomasello | Joint attention and lexical acquisition style[END_REF] for example observed that infants that have long periods of shared attention with their caregiver at 6 months have overall a larger perceptive vocabulary than those for which shared attention moments are infrequent. [START_REF] Moore | Word learning in the context of referential and salience cues[END_REF] showed that shared attention was a cue that outweighted saliency when 24-month-old children learn a new word. That is, children are better able to learn a novel word if this word was part of a shared attention frame, even if it was not visually salient. On the contrary, children did not learn words when the object was not part of a shared attention frame. Hence, having a theory of mind, and being able to enter in shared attention frames seems crucial for lexical acquisition.

Assumption and Biases

Children's ability to assign a meaning to a word-form seems to obey a certain number of assumptions and biases. [START_REF] Markman | Constraints Children Place on Word Meanings[END_REF] lists several biases children seem to have when acquiring novel words: the whole object assumption, the taxonomic assumption, and the mutual exclusivity assumptions.

The whole object assumption states that children treat word labels as refering to an object as whole (e.g. a tree) and not a sub-part of it (e.g. trunk, leaf, branch). [START_REF] Hollich | Young children associate novel words with complex objects rather than salient parts[END_REF] recently gathered evidence in favour of this hypothesis. When 12 and 19month-old children are presented with novel objects with detachable parts, they associate the novel word to the whole object rather to one or the other detachable part. The taxonimic assumption states that the usage of novel words is extended to similar objects, that is, object with similar characteritics (e.g. dog label for a German Shepherd and a Bulldog), and not to objects thematically related (e.g. dog and leash) (see [START_REF] Markman | Children's sensitivity to constraints on word meaning: Taxonomic versus thematic relations[END_REF]. Finally, the mutual exclusivity assumption states there is a bijection between a wordform and its referents: that is, for a given label there is only one associated (conceptual) referent (e.g. /dOg/ for dogs), and a single referent can only be referred to by a single label. According to [START_REF] Markman | Constraints Children Place on Word Meanings[END_REF], this assumption balances the previously presented assumptions. Indeed, children necessarily have to acquire meaning for object parts and other properties of objects (e.g. color, texture, etc.). Such constraint would thus "lead them to analyze the object for some other property to label", and more generally, analyse in greater depth the world around them in the search for new referents.

Other constraints were postulated, such as the principle of constrast and the principle of conventionality [START_REF] Clark | The principle of contrast: A constraint on language acquisition[END_REF]. The principle of constrast says that "every two [word] forms constrast in meaning". That is, the child would consider that no two word-forms are exact synonyms. Such principle would also encourage the child to explore its environment so as to find a possible referent. The principle of conventionality states that "for certain meanings, there is a conventional form that speakers expect to be used in the language community". That is, children should expect stability in the linguistic system they are learning, and hence that word-forms are regularly associated to a given meaning. This is also what is expected with the mutual exclusitivity assumption.

Mismappings

Even though lexical acquisition seems to be guided by a certain number of biases, children often make mistakes. When they fail to correctly map a word-form to its referent, they usually overextend its usage (e.g. dog for every furry animal with four legs) or underextend it (e.g. dog for German Shepherds only). Even though over-and underextensions are most appearant in production as a suppletive mechanism to refer to something the child does not know the word for yet [START_REF] Clark | Strategies for Communicating[END_REF], they also occur in perception [START_REF] Behrend | Overextensions in early language comprehension: evidence from a signal detection approach[END_REF]. [START_REF] Rescorla | Overextension in early language development[END_REF] provides a typology of the main causes of overextensions: categorical overinclusions where a label is used for a referent close to the real referent (e.g. baby for children), analogical overextensions where a label is used for a referent which bears a similarity with the true referent (e.g. ball for marbles or apples), and predicate statements which are holophrastic (e.g. dog when pointing at a basket to show the dog is not here). [START_REF] Rescorla | Overextension in early language development[END_REF] estimates that one third of the child's first words are overextended, and about three quarters of the extentions being either categorical overinclusions or analogical overextensions, most of them being done on the basis of "perceptual similarity" (as opposed to functional similarity).

Regarding overextensions, most of them seem to be explainable by a shape-bias [START_REF] Landau | The importance of shape in early lexical learning[END_REF]; that is, most words are overextended based on the global shape of their referent (e.g. dog for any four-legged animal). Even though most authors state that children overextend words based on "perceptual" features (i.e. shape, size, texture), in most experiments, the visual modality is usually the only modality available to the subjects.20 Nonetheless, this seems reasonable to some extent, as "[s]hapes of certain things (those too large, distant, or gossamer to explore by hand, such as mountains, birds, and fog) are visible only in practice" (Landau & Gleitman 1985, p. 13).

Hence word acquisition and world perception seem to evolve jointly: first, word-forms are rudimentarily associated to referents, and second, the aforementioned assumptions and principles encourage the learner to further analyse its environement, so as to discover new referents that can be associated to new word-forms. Association between a word-form and its referent is done on the account of perceptual characteristics, for which vision seems to play a major role.

Visual Modality

In order to map the segmented word-forms to their referent, vision plays a central role. Indeed, "it is largely vision that directs the infant's attention to persons, objects and events; vision is important for the co-ordination of attention between parent and child; and it often constrains (especially in Anglo-American culture) the content of language addressed to the child" [START_REF] Andersen | The impact of input: language acquisition in the visually impaired[END_REF]. Vision hence provides the necessary interpretative context, which is necessary to get "language learning off the ground floor" [START_REF] Landau | Language and Experience: Evidence from the Blind Child, Cognitive Science Series[END_REF] and seems to be the modality used the most by children in order to map word-forms to their referent (this will be discussed more extensively in the following section 1.3.5.2).

Chapter 1. Background on Child Lexical Acquisition

Word Mapping and Vision

It seems that infants' ability to systematically associate a visual referent to a word-form only appears at around 12 months of age. [START_REF] Thiessen | Effects of Visual Information on Adults' and Infants' Auditory Statistical Learning[END_REF] tested adults and 8-month-old infants on their ability to segment words from fluent speech, and associate the target word with a visual referent. During the familiarisation phase, an image depicting the target word appeared on a screen at the same time the word was uttered, and disappeared from the screen once the word's offset was reached. Therefore, in this experiment there is a regular association between the boundaries of the target word and its visual appearance, which should facilitate both segmentation and mapping. The author observes that while adults do benefit from the synchronicty between both modalities, 8-month-old children do not. This result is in line with previous results, [START_REF] Werker | Acquisition of word-object associations by 14-month-old infants[END_REF] indeed showed that 8-to 12-month-old infants are not able to learn word-object pairing. They however found that 13-or 14-month-old infants are able to do so. [START_REF] Friedrich | Neurophysiological correlates of picture-word priming in one-yearolds, in 'Early Language Development[END_REF] made similiar observations by studying the event-related brain potential of children between 12 and 19 months. The subjects were primed with a visual stimulus which was followed by either its matching audio word-form (e.g. "cat" for the picture of a cat) or a mismatching one (e.g. "dog" for the picture of a cat) and observed if there is a N400 priming effect. The N400 potential is linked to semantic processing in the brain. The expectation is that if the two modalities match (e.g. "cat" with cat) then an N400 effect should be observed, and no effect should be observed if the modalities do not match. While the N400 effect was observed for infants from 14 months on, it was not for 12-month-old infants, showing they did not seem to map the word-form to its visual referent. [START_REF] Smith | Infants rapidly learn word-referent mappings via crosssituational statistics[END_REF] tested if 12-and 14-month-old children are able to quickly acquire the meaning of a novel word and effectively resolve the ambiguities that might arise if several referents are possible by simply using cross-situational statistics. To test this, they present two novel objects to infants, each paired with a novel noun (e.g. a picture of a ball and a baseball bat, paired with their respective name). Hence, the child is unable at this stage to tell which object is paired with which noun. Nonetheless, by seeing other occurences of one of the two objects paired with another noun (e.g. picture of a ball and a dog, paired with their respective name), the child is able to learn the correct word-object mapping. While this is not suprising for 14-month-old infants, [START_REF] Smith | Infants rapidly learn word-referent mappings via crosssituational statistics[END_REF] also found it was the case for 12-month-old infants, which seems contradictory with the result of [START_REF] Friedrich | Neurophysiological correlates of picture-word priming in one-yearolds, in 'Early Language Development[END_REF] and [START_REF] Werker | Acquisition of word-object associations by 14-month-old infants[END_REF]. However, as this ability seems to develop in a very short time frame (between the 12th and 13th month of age), it is reasonable to observe slight differences from an experiment to another, especially if the experimental settings are different. [START_REF] Vouloumanos | Infants' learning of novel words in a stochastic environment[END_REF] tested if 18-month-old infants are able to map a wordform to its referent when a given word-form is not always uttered at the time its referent is simultaneously perceptible. Their study reveal that children are able to map the word-form to its referent, even if the word-form is used in contexts where the referent is not perceptible. It shows that children are able to maintain several hypotheses of the meaning associated to a given word-form, and seem to select the most frequent one.

The visual modality and the intention reading skills of infants are combined so as to make sense of what is said. [START_REF] Goldin-Meadow | From gesture to word[END_REF] argues that "referring to an object in gesture could facilitate learning the word for that object in toddlers at the early stages of language learning" as the referent of a given word-form would be unambiguously signaled to the child. Similarly, being able to interpret other gestures, and more specifically gaze direction helps children learning novel words, by noticing what the caregiver is looking at [START_REF] Law | Using Gaze Direction to Learn Words at 18 Months: Relationships with Later Vocabulary[END_REF]). Computational simulations [START_REF] Smith | Infants rapidly learn word-referent mappings via crosssituational statistics[END_REF][START_REF] Frank | Using Speakers' Referential Intentions to Model Early Cross-Situational Word Learning[END_REF]) find that word learning by cross-situational statistics is significativelly enhanced by speakers' intention information.

Language Acquisition and Blindness

Even though vision seems to be the most prominent sense children would use to learn a language, lexical acquisition is still possible when vision is lacking: indeed, blind people are as proficient in their native language as sighted native speakers. 21 However, being congenitally blind might slightly slow the learning process and affect the quality of what is being learnt -mainly phonetics and semantics -at least in the early stages. We will thus review in this section what is known of language acquisition in blind children. This will enable us to better understand what role vision plays in sighted children when they acquire their native language.

Most authors observe a tendency in blind children to talk about past events instead of the "here and now". This tendency to favour past events over what is occurring in the "here and now" is to be explained by an absence of obvious shared attention between the child and the caregiver. Such absence of shared attention is mainly explained by an inability of the caregiver to interpret what the child does and an inability to understand what the child is interested in [START_REF] Andersen | The impact of input: language acquisition in the visually impaired[END_REF]. By focusing on past events, the child tries to recreate a situation where she knows the caregiver and herself have shared a common experience they can talk about. 22 Research has also shown that blind children also suffer from a lack of decentration, 23 that is, they fail to consider anything else other than themselves and their immediate surroundings. Thus, the main topics that are initiated by blind children refer to action or states involving the child herself (see [START_REF] Andersen | Blind children's language: resolving some differences[END_REF]Peltzer-Karpf 1994, p. 41) and not to the outside world. It is easily understandable why blind children do so, as it is impossible for them to discuss or get interested in persons or objetcs that are not directly noticeable (either by touch, hearing, or smell) contrary to sighted children who can get interested in objects that are not in their immediate surroundings. Consequently, by being deprived of sight, blind children can only initiate interactions that involve persons or objects that are in their immediate surrounding. Unsurprisingly, blind children mainly rely on haptic input (i.e. touch) to compensate for the lack of visual input (see Dunlea 1989, p. 160;or Peltzer-Karpf 1994, p. 58), but it can only compensate for so much.

Consequently, this lack of decentration generates interactions with the caregivers that are different from the interactions the same caregiver would have with sighted children. As caregivers usually fail to interact properly with blind children -at least at a very early stage -the nature of the conversations are affected. For example, [START_REF] Andersen | The impact of input: language acquisition in the visually impaired[END_REF] reports that caregivers repeatedly "bombard" the child with labels instead of providing a description of their environment as they would usually do. She also reports that caregivers tend to be more directive so as to encourage children to explore their environment. She finally notes that caregivers tend to "restrict their verbal input". Naturally, both the child's comprehension and production is affected by the nature of the interactions. [START_REF] Andersen | Blind children's language: resolving some differences[END_REF] reports in her study that deictic terms were only 21 Blind people also make the same conceptual differences as sighted people for verbs such as "look" and "see", the only difference in the way of looking or seeing being the modality. See [START_REF] Landau | Language and Experience: Evidence from the Blind Child, Cognitive Science Series[END_REF].

22 "Talking about shared past events allows the blind children to maximize the probability that they and their addressees have a common focus of attention, just as the use of visual cues aids sighted children in talking about the here and now." (Dunlea 1989, p. 157).

23 "The gradual progression of a child away from egocentrism toward a reality shared with others. [...] It can also be extended to the ability to consider many aspects of a situation, problem, or object, as reflected, for example, in the child's grasp of the concept of conservation." in https://dictionary.apa. org/decentration consulted on 16/11/2020. Chapter 1. Background on Child Lexical Acquisition used by sighted children and not by blind children. On the other side, [START_REF] Peltzer-Karpf | Spracherwerb bei hörenden, sehenden, hörgeschädigten, gehörlosen und blinden Kindern, number 403[END_REF] note that blind children tend to use twice as much interrogative pronouns as sighted children which seems natural as it is the only way for them to explore the world. 24Blindness also seems to affect the generalisation capacities of the child. Indeed, the learning paradigm in sighted children and blind children is different. As noted by Dunlea (1989, p.10 ) "the act of touching and feeling is a search for information; it implies a conscious effort to obtain sensory stimulation" while sighted children do not need to do a conscious effort in order to interpret and understand their environment. This has also a more insidious effect that blind children are only able to fully grasp (or even notice) some concepts once they are able to physically explore their environment, as noted by Peltzer-Karpf (1994, p. 40): "While sighted children formulate concepts such as ego, others, agent, action and object through observation even before the crawling age, the blind child lacks decisive possibilities here".25 Thus, the "computational load" blind children face once they are able to freely explore their environment is higher than for sighted children. 26Because blind children are only able, to some extent, to conceptualise a large part of their environment later than sighted children, [START_REF] Dunlea | Vision and the emergence of meaning: blind and sighted children's early language[END_REF] postulates that they fail to grasp the symbolic nature of language and fail to apply appropriately the words or chunks of words they learn to new situations, whereas sighted children typically display a pattern of overgeneralisation. Indeed, [START_REF] Andersen | Blind children's language: resolving some differences[END_REF] notes that blind children tend to parrot what they hear without further analysis (i.e. segmentation). Mills (1983, p. 145) also has the same observations where he notes that the blind "child's language may remain tied to familiar social routines [...] and a reliance on phrases acquired wholesale is often couple with a marked propensity for imitation and echolalia." Concerning the lack of generalisation, [START_REF] Andersen | Blind children's language: resolving some differences[END_REF] observe "word-referent isomorphism, where children treat words as if they are proper names". Vision thus seems to act as a facilitator for abstracting and conceptualising the world, as it is easier to perceive what makes a set of objects similar when all the objects can be perceived simultaneously, such as noted by (Dunlea 1989, p. 87): "Vision is unique in that it provides instant simultaneous access to information which is otherwise segmented by space and time. Without vision, most information is necessarily perceived sequentially through haptic exploration". When overgeneralisations do actually occur, [START_REF] Dunlea | Vision and the emergence of meaning: blind and sighted children's early language[END_REF] reports that they are mainly based on tactile or auditory information, but not on olfactory or taste information.

Finally, regarding phonetics and phonology, Dunlea (1989, p. 15) reports that blind children tend to confuse only acoustically similar sounds while sighted children also confuse sounds because of their visual similarity (i.e. mouth movements). Peltzer-Karpf (1994, p. 24) notes that even though blind children succeed in acquiring the phonological system of their native languages, it generally takes longer for them to do so than for sighted children.

Hence, it appears that the lack of visual input slightly hinders language learning at an early stage. As we mentioned, this is to be explained by the fact that vision helps children conceptualise the world their live in. They can do so very early on as they do not need to physically explore their environment, but simply have to watch and see. Contrary to sighted children, blind children have to be able to walk (or crawl) in order to discover their environment and make sense of it. Even though blind children and sighted children have similar lexemes, the meanings they associate to them are different (Dunlea 1989, p. 59). Only when they have conceptualise their world can lexical acquisition take place successfully. Vision also helps children make sense of what is said to them, and help the child segment the spoken input into smaller units.

Conclusion on Word Mapping

As we put forward in this section, language acquisition is a multimodal phenomenon, that relies both on sensory perciptible cues, such as vision, and social cues, by using, for example, shared attention frames in order to map an acoustic stimulus to its referent. In this thesis, the model we use will only have access to perceptible cues -in the form of images -but naturally will not be able to use any of the social cues generally used by children. Similarly, our model does not have any Theory of Mind. Thus, we will investigate if using a simple associanistic approach suffices to learn a reliable mapping between an acoustic stimulus and a visual context.

Word Recognition

Once the child has isolated word-forms, they must be stored in the mental lexicon in such a way they can be retrieved afterwards, so as to create her own utterances or to interpret heard utterances. However, what appears to be simple, that is, storing word-forms in the mental lexicon, is far from trivial:

The scope of phonological representation is a fine balance. On one hand, memories for words have to be sufficiently broad to incorporate the rampant variability inherent in natural discourse and to normalize for accents, different voices, emotions, and other factors. On the other hand, memories have to be sufficiently specific to not incorrectly equate minimal pairs or tolerate mispronunciations. [START_REF] Singh | Infant word segmentation and childhood vocabulary development: a longitudinal analysis[END_REF] Spoken word recognition can be defined as the process of accessing lexical items in the mental lexicon from phonological patterns in the speech signal [START_REF] Magnuson | Spoken Word Recognition[END_REF]. That is, it implies mapping what is being heard with lexical items stored in the mental lexicon.

We will review in this section several models of spoken word recognition that have been proposed to explain how humans, children included, manage to do so. This will allow us to compare -later on in this thesis -word activation and recognition in humans and in a neural model of visually grounded speech, and observe if the patterns that are learnt by a neural model are similar to that of humans.

Cohort Model

One of the very first models trying to account for how humans recognise and extract words from the speech stream is the Cohort model by [START_REF] Marslen-Wilson | Processing interactions and lexical access during word recognition in continuous speech[END_REF]. According to this model, spoken word recognition proceeds in three steps that occur simultaneously which are: access, selection, and integration.

Access consists in activating a set of word-forms in the mental lexicon which corresponds to the acoustic input. Each word-form is paired to an activation unit, allowing for the activation of multiple word-forms at once. These units are only activated if the perceived acoustic input exactly matches the internalised phonological form. If there is a mismatch between the perceived acoustic input and the phonological form, the unit is deactivated and the word coded by this unit is not considered a valid candidate anymore. Hence, the initial phone of the speech stream activates a set of units whose word they code for all start with the perceived phone. This set of activated word-forms is called a Cohort. The wordforms of the cohort remain activated as long as the perceived acoustic input matches the internalised phonological form of each word of the cohort. The words comprising the initial cohort that do not match the perceived input are pruned. The process of interactive pruning of the cohort is called selection. The integration process is part of the selection process, but is however more concerned with the nature and function of the words of the cohort rather than with form. Indeed, not only should the phonological form of the words in the cohort match what is perceived, but they should also be be semantically coherent with the context and syntactically fit within the sentence. Thus, words that do not fit either syntactically or semantically are removed from the word cohort. Hence, this model integrates bottom-up activation signals -from the speech signal to the phonological form of a word -as well as top-down inhibition signal -essentially by controling for the syntactic and semantic validity of the activated words.

This initial version of the Cohort model has several problems. First, it requires the perceived acoustic stimuli to exactly match the internalised phonological forms. However, it is well known that this is not how humans process speech. Indeed, humans are able to recognise a word even when it is mispronounced and might not even notice any mispronunciation [START_REF] Cole | Listening for mispronunciations: A measure of what we hear during speech[END_REF]; when for example one sound is replaced by a similar sound ([m]/[n],

[s]/[S], etc.). As in this model, mismatching words are removed from the cohort, it would thus be impossible to recognise mispronounced words. Second, it requires the onset of the speech stream to necessarily be the onset of the word, which is unrealistic as one might drop in a conversation or overhear a conversation and still identify what word was being uttered when dropping in. Finally, this model implies that words should necessarily be recognised once their offset has passed, or there are otherwise deemed unrecognised and unrecognisable and removed from the cohort. However, [START_REF] Grosjean | The recognition of words after their acoustic offset: Evidence and implications[END_REF] showed that in some cases word recognition could only occur after word offset and not at word offset. This is the case for words that may constitute the beginning of another word. For example, in the sentence "I saw the bun in the store" [START_REF] Grosjean | The recognition of words after their acoustic offset: Evidence and implications[END_REF]) the recognition of "bun" can only occur when the /ð/ of "the" is perceived. Indeed, the word "bunny" ("[...] bun in [...]") could be a possible word. Similarly, the /n/ could be the beginning of another coherent word (such as in "[...] bunny nibbling [...]"). However, once the /ð/ is perceived, the only option left is to analyse the sequence as "bun in the". Hence, word recognition might only occur well after the word's offset has passed, situation for which the Cohort model does not account for.

The Revised Cohort model (Marslen-Wilson 1987a) relaxes some of the constraints of the first model. This model does allow for a word to be activated even if the perceived form does not match the internalised form. Nonetheless, only slight mispronunciations are tolerated and are able to activate a word (i.e. such as swapping /p/ for /b/ that only differ by one articulatory feature: voicing). Also, this version of the cohort relaxes the effect of context (integration) as "inappropriate words can [...] be readily perceived and identified, so long as they are unambiguously specified in the signal". 27 Therefore, in this version, topdown inhibition is removed so as to allow for non semantically or syntactically expected words to be recognised. However, to compensate for this top-down inhibition, this model incorporates the effect of word frequency, such that more frequent words may be activated more readily. Yet, this version of the cohort does still not account for the fact that a large majority of words can only be recognised passed their offset and not at word offset.

TRACE

The Trace model builds upon the Cohort model by conserving its "major positive features" (McClelland & Elman 1986a), which are simultaneous word activation and interactive selection and pruning, while trying to overcome the problems already discussed there above. The Trace model is structured in three layers, where each layer represents a particular linguistic unit: feature, phoneme, and word.

The feature layer constitutes one of the major differences with the Cohort model as Cohort supposes the input is perceived as a sequence of discrete units. Here, the acoustic input is perceived sequentially in terms of features: power, vocalic, consonant, voiced, etc.28 Each time slice (25ms) of the speech signal is represented by these features that can take a value ranging from 1 to 8: 8 representing the highest activation possible, and 1 the lowest. This feature layer is connected to the phone layer. The phones that match to a certain extent the activated features are themselves activated. For example, /b/ would be activated by the features voiced and burst (among other features). This feature layer allows for several phonemes -that only differ by one or two features -to be activated at once by the same acoustic input. For example, a [b] sound would also activate /p/. Indeed /p/ and /b/ only differ by the value of the voicing feature, all the others being equal. However, the phoneme that does not exactly correspond to the acoustic input will be less activated than the one that exactly matches the input. Hence, this model, by simply adding a feature layer allows for mispronunciations to activate the mispronounced phoneme, which was not the case in the Cohort model, making the model much more flexible. Finally, the phone layer is connected to the word layer. The activated phonemes activate the words in which they appear. Contrary to the Cohort model, which gives the word onset a very high importance, in this model words can be activated from any point on. For example, the phoneme string /poôt/ is able to simultaneously activate words such "port", "airport", "important" or "portable".

Units inside the same layer (phoneme, or word) are linked through inhibitory connexions so that the units that are the most activated inhibit to a certain extent the activation of the others if they do not fit the input as well as the most activated units. The proportion of inhibition is based on the proportion of overlap: "the strength of the inhibition between two word units depends on the number of time slices in which they overlap". A word is deemed recognised if its activation value is above a given threshold. Contrary to the Cohort model, this model is able to handle string of connected words. In this model, the speech stream is implicitly segmented when the words are recognised. This model brings a major improvement over the Cohort or Revised Cohort: words can be recognised after their offset, and words may be activated from any point on, thus relaxing the constraint on exact matching word onset the Cohort supposes. However, we believe the biggest improvement the Trace model brings is the active competition between words, as words really compete between one another. Indeed, in the Cohort model, words are considered as competitors only because they belong to the same cohort. Yet, there is no active competition between words, that is, there is no process by which a very activated word inhibits the activation of another. In the Trace model however, there is an active competition between words, where highly activated words inhibit the activation of other words, until the one that matches the input the most eventually "wins".

Shortlist

The computational implementation of the Trace model, though functional, made it unrealistic. Indeed, theoretically, this model supposes that for each time slice, each and every word of the vocabulary is considered a potential match, even if it hardly matched the spoken input. Hence, the computational load -both in a computational implementation and for a human brain -makes this model unrealistic when using a large vocabulary. The Shortlist model [START_REF] Norris | Shortlist: a connectionist model of continuous speech recognition[END_REF]) is a two-stage model that circumvents this limitation. The first step resembles the Cohort model, as it consists in building a shortlist (i.e. akin to a cohort) of words that match the current input. The second stage is a competition phase akin to that found in the Trace model.

At each time step, words that begin with the current phoneme as well as words that start with the previously perceived phonemes are activated and added to the shortlist: taking the word /kaet/ as an example, the initial /k/ activates a list of words that start with [k], the following [ae] activates words that start either with [ae] or [kae], etc. (Norris 1994, p. 204). The words figuring in this initial list are removed if their activation score -which represents how well the phonological representation they stand for matches the perceived acoustic input -is below a certain threshold. The second phase is the competition phase, where the words that most closely match the input compete between one another for recognition. As in the Trace model, the proportion of competition is moduled by the overlap -here, in terms of phonemes -between the competing words.

Consequently, this model is computationally lighter than the Trace model -as only a subset of the lexicon is considered a valid candidate -while preserving most of its original features. The Shortlist model however is purely bottom-up (i.e. there are excitatory connexions between the phonemic level and lexical level), but does not implement the top-down excitatory (from the lexical level to the phonemic level)29 that is found in the Trace model. They deem this constraint "redundant" with lexical inhibition. Nonetheless, this decision does not conform with reality, as lexical processing influence phoneme perception (see [START_REF] Warren | Perceptual Restoration of Missing Speech Sounds[END_REF] for the phoneme restoration effect or [START_REF] Ganong | Phonetic categorization in auditory word perception[END_REF] for the so-called "Ganong effect"). Furthermore, contrary to the Trace model that operates principally on handcrafted pseudo-acoustic features, Shortlist operates on strings of phonemes. They also (see §10 in Norris 1994) introduce an input representation that is a bit more realistic than discrete phonemes: mid-class transcription (i.e. coarse-grained transcription: (un)voiced fricative/stop/etc.) to account for "uncertainty, or ambiguity in the input". Nevertheless, this solution is still less realistic than the feature input adopted by the Trace model.

Distributed Cohort Model

The Distributed Cohort Model (DCM) sets apart from the previously presented models [START_REF] Gaskell | Integrating Form and Meaning: A distributed Model of Speech Perception[END_REF]. 30 While the preceding models explicitaly incorporated several levels of computation (feature layer, phone layer, lexical layer), the DCM does not. Indeed, the DCM uses a simple recurrent neural network [START_REF] Elman | Finding Structure in Time[END_REF], see Section 2.2.3) where information is distributed over several processing units. Hence, instead of using discrete representations, this model uses continuous representations. Even though artificial neural models are structured in layers, these layers cannot be interpreted as representing solely a single type of linguistic unit, where for example the first layer would represent features, the second phonemes, etc. Instead, each layer might represent several linguistic units at once; the network learning on its own which is the best representation that each layer should output in order to solve its task.

The network is inputted with a sequence of phonetic features and is trained to predict a semantic vector and a phonological vector. While the target phonological vector was constructed based on the phonemes of the target word, the target semantic vector is simply a vector of ones and zeros randomly assigned to each word of the training set. Given its nature, the model is purely bottom-up, and consequently no top-down contraints were implemented. However, given its recurrent nature, the prediction at a given timestep depending on the previous ones, this constraint is somehow implicit in the network.

In this model, the concept of activation is different from the other models. Indeed, while in the other models a scalar value could be attributed to each word of the lexicon -scalar value representing the strength of the activation -it is not possible here to have such value. Instead, activation is represented by the closeness of the predicted semantic vector to that of the words of the lexicon. Simultaneous activation of several words is still possible, but is however implicit. Indeed, the predicted semantic vector may be considered as "a 'blend' of the relevant representations". Activating a word more than another one can be thought of "modifying this blend" and "can be viewed in terms of movement through semantic space". A word is deemed recognised when the semantic vector corresponds exactly to the semantic vectors of one of the words of the lexicon.

An interesting property of this model is that it is easy to test if word frequency influences recognition. Indeed, to do so, it is only necessary to let the network see more instances of one word than another at training time. The authors find that their network tends to favour more frequent words by predicting a semantic vector which is closest to the most frequent word, particularly when the number of competitor words is high, that is, when many words start with the same sequence of phonemes. This behaviour is coherent with previous research: [START_REF] Goldinger | Priming lexical neighbors of spoken words: Effects of competition and inhibition[END_REF] indeed showed that more frequent words are more easily recognised than less frequent words.

Hence, in such model, the notion of feature, phone, and lexical unit is implicit as all the layers are able to reprensent this information simultaneously. Similarly, simultaneous activation of lexical units is pervasive, by the prediction of a semantic vector which "blends" the representation of possible words. However, in their experiment, they only test the recognition of individual words and not full sentences.

Conclusion on Word Recognition

All word recognition models assume that word recognition proceeds in several steps: an activation, a competition, and the final recognition. Nonetheless, the way these three steps are carried out varies from one model to another. While the Cohort model requires the acoustic input to exactly match the internalised representations, other models, such as Trace and Shortlist do not, and allow for a word to be activated even if the internalised representation does not exactly match the perceived acoustic realisations. The competition step also varies. In the Cohort, there is no active competition between words as the degree of activation of a given word does not influence that of others. This is not the case in the other models where there is a true competition for attention between words.

The DCM sets apart from the other models and shows that word recognition may be done using continuous representations only, and does not need, as for the other models, for the intermediate representations to be discrete (i.e. phones). This model is relatively close to recent models of speech recognition which rely on RNNs to decode speech.

However, all these models agree on the fact that word recognition necessarily implies the simultaneous activation of a set of word candidates. This set of words is iteratively pruned if the acoustic input is too distant from the internalised representation so that ultimately only one word remains.

Conclusion

In this chapter, we reviewed the strategies used by children to segment the spoken input into sub-units. We showed that children use a wide range of strategies to find words. They first start by using supra-segmental cues (i.e. prosody) as they are very sensitive to variation of the fundamental frequency from the moment they are born. We also showed that children hear a high proportion of words in isolation. Using these two strategies, they are able to infer the canonical shape of the word-forms of their native language, so as to later apply more advanced segmentation strategies. Indeed, they are able to compute forward and backward transition probabilities between units in order to posit word boundaries, or use phonotactic rules in order to extract segments.

We then presented several word recognition models that try to account for how humans are able to activate and retrieve lexical units from the mental lexicon. All word recognition models agree on the fact that a successful recognition strategy consists in simultaneously activating several lexical units at once. Yet, the process by which these lexical units are activated varies from one model to another. While some models require the acoustic input to exactly match the internal representations, some are less strict, and allow for mispronunciations.

Finally, we showed that language acquisition is a multimodal phenomemon that involves all of the child's perceptual abilities. We showed that intention reading and shared attention were crucial components for the child to acquire its native language. Lastly, we showed that vision provided the child with invaluable information to acquire her mother tongue. Children that are deprived from this perceptual input, even though they are eventually able to acquire their native language, are hindered, as it is harder to conceptualise the world when such perceptual input is lacking. The work we conduct in this thesis fits into the scheme of unsupervised speech processing. Speech Processing has always required a large amount of human supervision, for example in Automatic Speech Recognition (ASR), in the form of the priors incorporated into the models themselves -when for example using Hidden Markov Models (HMM) and Gaussian Mixture Models (GMMs) which explicitly model phonemes (through GMMs) or syllables (through the use of triphone-based HMMs) -and more importantly in the form of manual transcriptions. However, gathering and having human experts transcribe and annotate data is a time-consuming process which requires expert-knowledge and is costly. Also, not all languages have a written form, or some may have several non standardised written forms, which make current approaches not usable on such languages. Hence, while traditional approaches are sustainable for developed languages,1 they are not for less-developed languages.

Another motivation, which also constitutes the motivation of the work carried out in this thesis, is to devise speech processing models that are closer to human speech processing. Indeed, humans do not require much supervision to learn how to process speech, especially when acquiring their native language. Notably, they do not require textual labels, but instead use weak supervision signals, such as visual cues. These signals constitute weak supervision signals as they constrain the learning process by contextually grounding it.

The work carried out in this thesis belongs to the sensory-based approaches of speech processing models, as defined by [START_REF] Glass | Towards unsupervised speech processing[END_REF]. Sensory-based models of speech processing are models that only require speech paired with sensory data in order to operate. That is, such models do not require any annotated data nor human expertise to operate and "closely match that of human spoken language acquisition" according to [START_REF] Glass | Towards unsupervised speech processing[END_REF]. In the following sections of this introduction, we will present models that also belong to the class of sensory-based approaches to speech processing. We will focus more specifically on models targeted on unsupervised term discovery and speech segmentation. We will then argue that grounding speech to another modality is necessary if one wants to be able to associate a meaning to the extracted word-forms, and hence be able to devise computational models of lexical acquisition.

Spoken Term Discovery and Speech Segmentation

Most of the work done on unsupervised speech processing aims at segmenting an audio signal into sub-units -generally phones or words -or without fully segmenting the audio, aims at finding large lexical units, such as words or multiple word expressions. [START_REF] Lee | Unsupervised Lexicon Discovery from Acoustic Input[END_REF] distinguishes between two terms: spoken term discovery, and unsupervised lexicon discovery. 2 The first term, spoken term discovery simply refers to a keyword spotting task. That is, given a spoken utterance, the goal is to find similar acoustic segments across a collection of utterances. According to [START_REF] Lee | Unsupervised Lexicon Discovery from Acoustic Input[END_REF], this task typically involves using "unsupervised pattern discovery methods" such as Dynamic Time Warping (DTW, [START_REF] Sakoe | Dynamic programming algorithm optimization for spoken word recognition[END_REF]. This task only aims at finding keywords, or a least chunks of words, but does not seek to obtain a full segmentation, where the boundary of each word in the speech stream would be marked. In such keyword spotting task, the goal is only to find one or more keywords, which results in the discovery of two boundaries -one at the beginning of the word, one at the end -leaving the rest of the speech stream unanalysed. Unsupervised lexicon discovery refers to the process which results in a complete segmentation where ideally all word boundaries are found. According to the authors, such task usually implies a two stage modelling, where a model simultaneously learns a sub-word (i.e. syllabic, morphemic, etc.) distribution and a word distribution. Such two stage modelling generates top-down and bottom-up constraints that result in a better segmentation overall, as it explicitly takes into account the hierarchical linguistic structure of the input.

Spoken Term Discovery. [START_REF] Park | Towards unsupervised pattern discovery in speech[END_REF] introduced segmental dynamic time warping (S-DTW) for unsupervised term discovery. This method is based on dynamic time warping which is a pattern similarity measure between two sequences (Rabiner & Juang 1993, p. 226). While DTW operates at a global scale, giving the similarity measure of two sequences globally, S-DTW operates at a local scale, giving a similarity measure for two sub-sequences taken from two larger sequences. The idea behind S-DTW is simple: even though two spoken utterances are different, they might contain similar words. Hence, even though the similarity measure might be low at a global scale, it might be high at a local scale. Such methods allow to detect and eventually extract word-like units from raw acoustic input without the need to result to a transcription at any point [START_REF] Park | Unsupervised Pattern Discovery in Speech[END_REF]. [START_REF] Jansen | Efficient spoken term discovery using randomized algorithms[END_REF] further improved on vanilla S-DTW by introducing algorithms with lower complexities, making S-DTW more scalable on large data sets. Recently, [START_REF] Räsänen | Unsupervised Discovery of Recurring Speech Patterns Using Probabilistic Adaptive Metrics[END_REF] proposed a two-staged probabilistic variation of S-DTW making S-DTW usable on larger data sets.

Finally, we should mention the work of [START_REF] Räsänen | Unsupervised word discovery from speech using automatic segmentation into syllable-like units[END_REF] which introduced a method for unsupervised term discovery without the need for S-DTW, by first slicing the input speech signal into syllables -by considering local minima in the amplitude envelop as boundaries -and then grouping the syllables together by finding frequent n-grams.

Unsupervised lexicon discovery. The goal of unsupervised lexicon discovery is to obtain a full segmentation, and this usually involves Bayesian approaches which model several levels of linguistic units at once. These methods are, to a large extent, inspired by prior work on dealing with the segmentation of strings of discrete units [START_REF] Goldwater | Nonparametric Bayesian Models of Lexical Acquisition[END_REF][START_REF] Johnson | Adaptor grammars: A framework for specifying compositional nonparametric Bayesian models[END_REF]. [START_REF] Taniguchi | Nonparametric Bayesian Double Articulation Analyzer for Direct Language Acquisition From Continuous Speech Signals[END_REF] introduced the Bayesian Double Articulation Analyser that aims at learning in an unsupervised fashion a language model and an acoustic model using a non-parametric Bayesian approach. [START_REF] Lee | Unsupervised Lexicon Discovery from Acoustic Input[END_REF] combined unsupervised acoustic unit discovery with Bayesian segmentation in order to unsupervisedly segment a speech stream into lexical units. [START_REF] Kamper | Unsupervised neural and Bayesian models for zero-resource speech processing[END_REF] propose a model that jointly segments and clusters, where first, boundaries are sampled, the resulting segments are then embedded and clustered by a Bayesian GMM, which in turn gives information on the likelihood of the initial segmentation, which is used to sample better boundaries.

Recently, neural architectures were proposed to solve this task; most of them using autoencoders. For example, [START_REF] Bhati | Self-Expressing Autoencoders for Unsupervised Spoken Term Discovery[END_REF] proposed a new auto-encoder architecture that allows to segment the speech stream into phone-like units which are then clustered. Their idea relies on unsupervisedly learning similar embeddings for audio segments that belong to the same phone unit. Similarly to [START_REF] Räsänen | Unsupervised word discovery from speech using automatic segmentation into syllable-like units[END_REF], words are recomposed using a n-gram strategy. [START_REF] Chen | Audio Word2vec: Sequenceto-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation[END_REF] propose an auto-encoder in which discrete boundary decisions are taken at each time step. The intuition being that if the models take good boundary decisions (i.e. place boundaries near true boundaries), the reconstruction process should be facilitated. A similar approach was also proposed by [START_REF] Elsner | Speech segmentation with a neural encoder model of working memory[END_REF].

Grounding Language

A surprising fact of the previously mentioned approaches to unsupervised term discovery and lexicon discovery, is that word spotting and segmentation are only done on the basis of form alone, without any other contextual cue being used. To this extent, these works are very reminiscent of the work carried out in linguistics that also abstract meaning from the segmentation task (see Section 1.2). This is even truer since the creation of the Zero Recource Speech Challenge. 3 The original goal of this challenge is "unsupervised discovery of linguistic units from raw speech in an unknown language" without using any linguistics resource -hence the name -such as phonetic or orthographic transcriptions. The motivation of such challenge is twofold: first, a technological one, which is to develop methods to handle languages which do not have a written form (or language that might have one, but which do not have a standardised orthography); and second, a psycholinguistic motivation, which is to reproduce the cognitive processes at work in the human child when she learns her mother tongue, as she does so without any supervision. 4 Even though the website mentions the term zero resource should be understood in the sense of "zero linguistic resource" and not "zero information besides audio (visual, limited human feedback)", the only data which is given to the participants is audio data. Hence, the entries to the contest rarely use any other external resource than the one provided, and consequently rely solely on form.

When examining the literature, we notice that the words term and lexicon in unsupervised term discovery and lexicon discovery are used as synonyms for word-form, when from a linguistics perspective, a term or a lexicon is much more than a word-form (or a list of word-forms respectively), but also includes information about meaning (among other information). Language is not an independent system, and what gives it its substance is that it is tied to the physical world. We used the example of blind children to show that, when deprived from one type of sensory input, namely vision, language acquisition was substantially affected. Notably, blind children show a propensity to repeat phrases and sentences tied to a particular context (see 1.3.5.2) without further analysing them. To draw an analogy, term discovery and segmentation models are actually trained as sensory-deprived toddlers, that are only trained on form. Word segmentation might lead to erroneous results if done without contextual information: multi-word expression might be considered as a sequence of several words instead of unique words (e.g. French for potato "pomme de terre"). The decision as to whether these words should be considered as one unit or as several can only be done on the basis of contextual information, that is, their linguistic referent. Similarly, the phonological system of a language can only be inferred in light of the surrounding context and whether the two word-forms that use these similar sounds have the same referent or not. Therefore, it seems an adequate speech segmentation strategy should include a meaning component. In their work, [START_REF] Bender | Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data[END_REF] argue that: if form is augmented with grounding data of some kind, then meaning can conceivably be learned to the extent that the communicative intent is represented in that data Consequently, grounding linguistic data to external knowledge seems to be a necessary step to gain access to meaning. Yet, what does grounding more precisely mean? If grounding is simply adding external knowledge, ASR can be thought of textually grounded model, as text, which is external knowledge, is added to the model. [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF] defines grounding as follows:

The relationship between words and the physical world, and consequently our ability to use words to refer to entities in the world, provides the foundations for linguistic communication. Current approaches to the design of language processing systems are missing this critical connection, which is achieved through a process I refer to as grounding. [...] Language grounding refers to processes specialized for relating words and speech acts to a language user's environment. [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF] Grounding thus implies adding to linguistic data external non-linguistic data which somehow reflects the physical world. The external data should also reflect to some extent the communicative intent in the linguistic data as mentioned by [START_REF] Bender | Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data[END_REF]. This latter constraint is also visible in Roy (2005)'s definition, as "speech acts" necessarily result from a communicative intent. Grounding language to another source of knowledge acknowledges the simple fact that language cannot be considered in vacuo -that is, as a whole complete in itself and disconnected from any context -but is actually produced in and tied to a particular context. Even though considering language in vacuo does not preclude from learning anything, accurate segmentation requires access to meaning. It should be noted that grounding language to another source of information does not necessarily imply a change in modality. For example, it is possible to ground the word-form /kaU/ ("cow") to the sounds [mu:] ("moo") in which case it is the nature -linguistic v. non-linguistic -of the acoustic stimulus that changes, but not the modality itself.

Hence, grounded models refer to a class of computational models that process some linguistic form -either text or speech -in conjunction with another source of information from the physical world. The fact that two modalities are processed in conjunction is necessary, but is not sufficient: both modalities should occur simultaneously in the physical world so that both are tied. Thus, grounded models do not process form alone, but are able -or at least given the ability -to link this form to referents in the physical world, or representation thereof.

Background

In the following section, we will present computational models that are able to ground language, and more specifically speech, by using visual data. We will show that such models constitute viable test beds to study lexical acquisition. But first, we will present artificial neural networks, which are used to build visually grounded computational models.

Machine Learning

Machine Learning (ML) is a sub-field of Computer Science concerned with the creation and "study of computer algorithms that improve automatically through experience" (Mitchell 1997, p. xv). Instead of designing a specific algorithm to solve a specific task, ML seeks to design algorithms that learn how to solve a specific task:

A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E. (Mitchell 1997, p. 3) Experience E consists of a set of examples -called a data set -which the program will learn from. Learning occurs through a process of trial and error where the program tries to solve the task T using the available data E and changes its parameters so as to increase its performance measure P.

Among the several learning schemes that exist, two important ones can be distinguished: supervised learning, and unsupervised learning. We use the latter in this thesis. In the supervised case, the data set consists of a set of annotated (i.e. labeled) examples and the task of the algorithm is to learn the relationship between the examples and their labels. An example of such a task would be to learn to segment a string of characters given a training set where each character is labelled so as to indicate if it constitutes a boundary or not. In the unsupervised case however, the data set would consist of unannotated examples and the latent structure of the data should be inferred by the algorithm. For example, in this case the task would be to infer word boundaries from the string of characters alone.

Hence, contrary to the unsupervised case, where the model is only given raw data, in the supervised case, the data is augmented with supplementary labels which will be the learning target of the algorithm (Goodfellow et al. 2016, p. 105). More formally, Goodfellow et al. (2016, p. 105) define unsupervised and supervised learning as follows: given a set of examples X , unsupervised learning consists in learning the probability distribution p(x) given several random vectors x ∈ X , while supervised learning consists in learning to predict a set of values y given several random vectors x and their associated label y where (x, y) ∈ X , which can be interpreted as learning p(y|x). Goodfellow et al. (2016, p. 105) argues that the lines between supervised and unsupervised learning is blurry. It is the case for example of self-supevised learning. In this case, the targets that the model should predict are drawn from the input data itself. That is, the input data acts as its own label. Given a set of examples X , it could be more formally defined as learning p(x) where x ∈ X by learning to model p(y|x \y ) where y ∈ x. This is the case of most recent language models (BERT), audio embeddings models (wav2vec), or unsupervised learning of visual representations.

The data set used for learning is commonly split in three uneven sets: the training set which is the largest (≈ 80%), the development (or validation) set (≈ 10%), and the test set (≈ 10%). The training set, as its name suggests, is used to train the model. The development set is used to test the model so as to select the best model checkpoint, while the test set is only used once the best model is selected to report the final score.

Artificial Neural Networks

Artificial Neural Networks (ANN) are a type of ML algorithms which are based on the work of the the American psychologist [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF]. ANNs belong to the connectionist approach of Artificial Intelligence (AI):

These models assume that information processing takes place through the interactions of a large number of simple processing elements called units, each sending excitatory and inhibitory signals to other units. (Rumelhart, McClelland & PDP Research Group 1986, p. 10) The central idea of connexionism is that a complex behaviour can be approximated by a complex computation, that can in turn be broken down into smaller and simpler computations realised by individual processing units. The inter-connexion of these units (and hence computation) gives rise to the global behaviour. Connectionism was also referred to as "parallel distributed processing" (PDP), where knowledge is distilled and distributed over each processing units that constitute the network.

φ( n i=1 w i x i ) ŷ x 3 w 3 x 2 w 2 x 1 w 1 . . . x n w n b Figure 2.

2: Example of an artificial neuron

These processing units, originally called perceptrons, are now commonly called neurons and may be represented as in Figure 2.2. A neuron receives n inputs X = x 0 , x 1 , ..., x n from which an output ŷ is computed. Given the input features, the neuron is trained to predict an output ŷ that should be as close as possible to the desired output y. In order to do so, the input vector is weighted by a set of weights W = w 0 , w 1 , ..., w n which are learnable parameters, and the output is further transformed by a non-linear activation function φ. A bias neuron may also be added. 5 The final computation of a single neuron can thus be summarised as follows:

ŷ = φ(W • x + b) = φ( n i=1 w i x i + b) (2.1)
Real world problems are too complex to be solved by a single neuron. Thus, an arbitrary number of neurons can be used in conjunction so as to learn complex transformation functions. These neurons are grouped into layers and form what is commonly referred to as a Multilayer Perceptron (MLP). The first layer is called the input layer, the final layer the output layer, and the one or more layers in-between are called the hidden layers. The topology of the network is the result of how the different layers are connected. There are several ways to connect consecutive layers of neurons, the simplest being by connecting all the neurons of a layer to all the neurons of the next layer (fully connected layers). However, this is not the only option: it is possible to introduce cycles (recurrent neural networks) where the output of a layer serves as input for the same layer; or a layer can be connected to the next layer but also the second-next or n-next layer (skip connections); or only partially connect one layer to the next (sparse connections).

Recurrent Neural Networks

Theoretically, MLP can handle any type of data provided that the input layer is large enough to fit it. For example, it is possible to train a simple image recognition model that uses a MLP provided the size of the input image is fixed and known in advance (e.g. only 32×32 pixels images). In such cases, the data is flattened to a one-dimensional vector (e.g. 1×1024 for a 32×23 grey-scale image) which is then fed to the MLP. Such solution might be satisfactory for simple problems, but it is unusable when handling sequential data, as usually the size of the sequence to be processed is unknown.

Recurrent neural networks (RNN) are neural networks that are able to process sequential data [START_REF] Elman | Finding Structure in Time[END_REF]. RNN can be considered as feed-forward networks augmented with feedback connections. 6 RNN can be formalised as follows:

h t = φ(W x t + U h t-1 + b) (2.2)
where h t is the hidden state at timestep t, x t is the current input, h t-1 is the previous hidden state, b is a bias term, φ is a non-linear activation function (usually sigmoid or hyperbolic tangent), and where W , U and b are learnable parameters. Note that the only difference with a feed-forward layer is simply the additional term U h t-1 : the output at a time step t does not only depend on the current input but also on the output at the previous time step t -1. Thus, such computation allows to model the temporal dependency that exists between consecutive vectors. The output of a RNN consists of a sequence T of vectors. The final vector of the sequence (at timestep T ) can be viewed as the compact representation of the whole input sequence, as the final vector depends on the computation of all the previous vectors of the sequence. When the first element of the sequence x 1 is processed, h t-1 does not exist. In such case, this previous hidden state, noted h 0 , is set to be a vector of 0, though in some cases the initial state might also be a learnable parameter of the network. Note that processing a sequence with an RNN is different from running a simple feedforward network over each vector of the sequence. Indeed, in such case, the output at a given time step would be totally independent of the previous output. Such computation would not be able to model the sequential nature of the data, and would be a simple projection of the input vectors. Hence, the feed-back connection of RNNs really constitute the backbone of this neural model.

Even if RNNs are able to model sequential data and keep track of local dependencies relatively well, they are unable to capture long term dependencies in the input sequence. Indeed, as the sequence size T grows longer, the vector h t contains less and less information about the beginning of the sequence. The information about the past fades away as the previous hidden vector h t-1 is combined at each timestep with new information. Gated units were introduced to solve this problem.

Gated Recurrent Units

LSTM [START_REF] Hochreiter | Long short-term memory[END_REF]) are a type of gated units and are able to learn both long term and short term dependencies.7 LSTM are equipped with a memory cell that is used to store information about the past timesteps. Two gates (the input gate and the forget gate) control which information is added to the memory cell and which information should be removed. Such gating mechanism thus enable the LSTM cell to keep track of both long term and short term dependancies by adding to the memory cell information that needs to be preserved over several timesteps. However, LSTM are hard to train as they have many parameters, notably because of the parameters used for the memory cell and additional gates. [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF] introduced Gated Recurrent Units (GRUs) that are also able to keep track of long term and short term dependencies such as LSTMs, but with fewer parameters which make them easier to train. A GRU is formally defined as follows:

z t = σ(W z x t + U z h t-1 + b z ) r t = σ(W r x t + U r h t-1 + b r ) ĥt = tanh(W h x t + U h (r t * h t-1 ) + b h ) h t = (1 -z t ) * h t-1 + z t * ĥt (2.3)
where W z , U z , W r , U r , W h , U h are learnable matrices, b z , b r , b h are bias terms, and σ represents the sigmoid activation function. A GRU has two gates: an update gate z, and a reset gate r that both independently compute a scalar value at each time step t (z t and r t ∈ [0, 1]). Given the current input x t , the value of z t represents a ratio between how much information should be carried on from the previous time step [(1 -z t ) * h t-1 ] and how much new information should be integrated [z t * ĥt ]. This gate allows to control for long term dependencies. r t represents how much the new state should depend on the previous time step [r t * h t-1 ] thus controlling for local dependencies.

Even though LSTM and GRU are able to keep track of long-and short-term dependencies, encoding the meaning of a whole sequence in a single vector is still challenging and the recurrent unit might forget the beginning of the sequence, especially when the sequence is particularly long. One solution is to encode the sequence from both ends, where one GRU processes the sequence from left to right and another GRU processes the input sequence from right to left. The final vector is a concatenation of both vectors. Such processing is know as bidirectional processing, as opposed to unidirectional processing (which is usually done from left to right).

Attention Mechanism

Even when using bidirectional recurrent cells, encoding the meaning of a whole sequence in a single vector remains a difficult task. Attention Mechanisms [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] were introduced to solve this problem. This solution was originally thought for machine For illustration purposes, we only depicted three positions where the kernel could be, when naturally the convolutional kernel scans the whole input. Usually more than one kernel is used.

translation purposes, but found a wide echo and is now used for many other tasks. The intuition of such attention mechanisms is simple: instead of keeping the last vector of the sequence -which should encode the meaning of the whole sentence -all the vectors that were computed at each time step are kept and a weight is assigned to each of them so as to give more importance to some vectors than others.

The weights are computed by the network itself and learnt at training time. Hence, the network learns which vector(s) in the input sequence should be given more importance in the final representation without any supervision. An attention mechanism can be formalised as follows:

c = T t=1 α t h t α t = exp(score(h t )) T t =1 exp(score(h t )) (2.4)
where c, the context vector, is the weighted sum of the hidden vectors, and α t is the attention weight for time step t, where α t ∈ [0, 1] and

T t=1 α t = 1.
The formalisation we present is very general: the scoring function score (usually a MLP) computes a scalar value for each h t . However, the precise way this scalar value is computed depends on the particular implementation of the attention mechanism and may not only be a function of h t . With such attention mechanism, vectors for which the attention weight α is high will be highly represented in the final vector c, while vectors which were assigned a low attention weight won't.

Convolutional Neural Network

Convolutional cells [START_REF] Fukushima | Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[END_REF][START_REF] Lecun | Object Recognition with Gradient-Based Learning[END_REF] were originally introduced for image processing. As already mentioned, even though MLP may be used to process images, they become unusable in practice when the size of the image grows, as the size of the input of the MLP has to fit the whole image (e.g. 32M for a 32Mpixel image). This makes MLP unusable in practice on such large images. Also, as the input data is flattened, information about which pixel is adjacent to which in the original becomes harder to pick up.

Convolutional cells enable to solve this problem by repetitively applying a convolutional kernel (also called a filter ) to the input image. The image is hence processed piece-wise, one portion at a time. While such kernel processing was known before neural networks,8 the convolutional neural network allows for the kernels to be learnt: the kernel becomes part of the trainable parameters of the network. It hence allows the network to learn the kernels, as usually more than one kernel are used, so as to best solve the task the network is trained for. Convolutions are also easy to implement, as they just consist in matrix multiplication: the sub-matrix that represents the portion of the image being processed and one for the convolutional kernel.

Three types of convolutions exist, depending on the axes on which the kernel slides: 1D convolution (where the kernel slides along one axis), 2D convolution (slides along 2 axes), and 3D convolutions (along 3 axes). For our purpose here, we will only consider 1D convolutions as they constitute the most used type of convolutions for speech processing9 when working with MFCC vectors; such as we do in this thesis.

Convolutions are parametrised by:

• k the number of convolutional kernels used to scan the input (in Figure 2.5 on the Kernel Axis, k = 4). If only one kernel is used, the output of the convolution will be a 1D vector of length S out if more than one kernel is used, the output will be of dimension S out × k.

• size which is the height of the kernel which represents how many input vectors are processed at once (in Figure 2.5, size = 2). In 1D convolutions, the length of the kernel usually is the same as the length of the input vectors (in Figure 2.5, the lenght of the kernel is equal to the dimension of the MFCC vectors: 13 ); though depending on the implementation, the convolutional kernel can be split over several groups that only see one part of the vectors.10

• stride represents the shift along the Time axis (in Figure 2.5, stride = 1), meaning the kernel will slide along the Time axis by moving by stride vectors at each step.

• padding represents how many blank vectors are added before and after the actual vectors to be processed. This allows to control for the output size of the sequence S out given the original input size S in .

The length of the output sequence S out can be computed as follows:

S out ≈ S in -size + padding start + padding end stride + 1 (2.5)
where padding start refers to the number of padding vectors added before the input sequence, and padding end to the vector added after the input sequence; the precise number of vectors added at each end depending on the padding mode used. 1D-convolutions are widely used for speech processing as each convolutional kernel aggregates several vectors along the time dimension into a single value. This is particularly useful for speech, as each phone of the speech stream straddles over several MFCC vectors 2.5 on how to compute Sout) when using 4 convolutional kernels of size 2 × 13 sliding along the Time axis with a stride of 1, and zero padding. The position of the convolutional kernels are depicted with borders of different colours -blue and red -to materialise the evolution of their position over time; and the result of the convolution operation is shown by a shaded box with the same colour in the output matrix. For illustration purposes, we only depicted two positions where the kernels could be (at kernels start position and kernels end position), when naturally the convolutional kernels scan the whole input. We also only depicted 4 kernels -shown on the Kernel axis, with different shades of the same colour -when usually more are used.

-which usually represent 10ms of the original input signal, that is, less than the length of a phone. The use of different kernels also allows the network to select different features in the original input, and encode different correlations between the input frames. Also, the convolved input is usually shorter than the original input given appropriate choice of the kernel size, stride, and padding (the longer the stride and the kernel, the shorter the output sequence). This allows for the network to learn to properly downsample the input, and hence reduce the computational load, particularly when using recurrent layers afterwards.

Loss Function and Backpropagation

The perceptron, and neural networks in general, have a biological motivation (i.e. the nervous system) and authors regularly use biological terms (e.g. "stimuli", "excitatory/inhibitory inpulse", see [START_REF] Rosenblatt | The perceptron: a probabilistic model for information storage and organization in the brain[END_REF] to describe how such models work and compare them to biological systems. Nevertheless, artificial neurons widely differ from their biological counterparts in the way the computation is realised, in their organisation, and in the way actual learning takes place: gradient descent. Using gradient descent as a learning scheme imposes certain constraints when designing the network: every operation should be differentiable. 11The goal of the network is given an input x to predict an output ŷ which is as close as possible to the desired (true) output y. The success of the operation is measured using a loss function (see Equation 2.6). During training, the network is encouraged to minimise the difference between the predicted output and the true output, and the value computed by the loss function will serve as the basis to update all the weights of the network so as to reach its goal.

L(y, ŷ) = |y -ŷ| (2.6)
Because the output ŷ directly depends on the output of the network, which in turn depends on the parameters of the network θ, it is possible to change them so as to tune the network so that the predicted output is closer to the desired output. This operation is done through gradient descent using the backopropagation algorithm [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. This operation consists in computing the derivative of the loss function with respect to the parameters of the network. Intuitively, this measures how responsible a given weight is for the discrepancy between the predicted output and the expected output:

θ t+1 = θ t -η dL(y, ŷ) dθ (2.7)
where the weights at θ t+1 are updated by a factor η which is called the learning rate.12 The learning rate controls the strengths of the updates of the network's weights. If it is set too high, the network might miss the optimal solution, while a too small learning rate will make learning slower and additionally may have the network stuck in a local optimum. Note that the equation 2.7 corresponds to stochastic gradient descent, which corresponds updates the weights of the network based on the loss value of one training example (see Goldberg 2017, p. 31). Even though this approach is effective, it might take time to reach the global minimum, hence, usually batched gradient descent is used, where the loss is averaged over a batch of examples, that is, over several examples randomly sampled from the training set.

Hence, for each backward pass (i.e. gradient descent) all the weights of the network are updated so as to try to minimise the training cost. Even though neural networks are biologically inspired, the way learning takes place is not cognitively plausible (forward pass and backward pass use the same path, when biological synapses are unidirectional; artificial neurons are organised in well defined layers, and not in a biological brain, etc.).

Visually Grounded Speech

In this section, we then present Visually Grounded Speech (VGS) models and the analysis methods that have been developed in order to understand what such models have learnt. Finally, we will examine to what extent VGS models are able to model lexical acquisition.

Models

The CELL Model

One of the very first models of Visually Grounded Speech (VGS) is the CELL (Crosschannel Early Lexical Learning) model developed by [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF] and [START_REF] Roy | Grounded spoken language acquisition: experiments in word learning[END_REF]. This model was explicitly developed so as to understand how the interaction of visual and auditory stimuli enabled lexical acquisition. The goal of the CELL model is to learn audio-visual mappings (called audio-visual prototype) between various objects and their spoken word-forms so as to constitute a proto-lexicon such as a child would. This model strives to implement components that reproduced known cognitive abilities such as short term memory (STM) and long term memory (LTM) so as to simulate the memory abilities of a child.

In order to learn audio-visual prototypes, the CELL model is inputted with a visual context (image) and its paired utterance. The data used to train the model was gathered specially for this experiment and consists of child directed speech uttered by a caregiver while a child was playing with various objects. The CELL model searches for recurrent audio patterns, which are then stored in the STM along with the object considered at the time the discovered pattern was uttered. The LTM then scans the STM so as to find repeated occurrences of a given audio-visual pair -called AV event -and if several are found, they are copied into the LTM. The AV events stored in the LTM are then further processed to remove spurious pairs. Indeed, some words may occur frequently (e.g. determiners) while not being linked to a specific visual context. Hence, spurious AV events are removed based on a mutual information criterion: the lower it is, the less the two modalities (i.e. speech and vision) are tied, and hence do not form a valid AV pair. If mutual information is above a certain threshold, the AV event becomes a lexical item which is stored permanently in the LTM.

The results show that the CELL model is able to learn semantically valid AV items (e.g. shoe, key, dog, doggie) and reports 85% semantic accuracy of the discovered pairs. Some of the AV items also contain non standard "words" -which are not counted as semantically accurate pairs -such as onomatopoeic sounds (e.g. barking, engine). This is an interesting result, as onomatopoeias are also acquired by young children and used both in perception and production (see [START_REF] Laing | A role for onomatopoeia in early language: evidence from phonological development[END_REF]. In order to compare the lexicon that would be acquired when no visual stimuli is used, the authors also ran experiments using a "blind" model, that is, a model that does not use visual context. The results are very different with 0% semantic accuracy. Most of the AV pairs in this setting only comprise onomatopoeias and recurrent patterns not linked to the object in consideration (e.g. "what you gonna do", "really good", etc.). Hence, the key finding of the CELL model is that visually grounding speech increases the accuracy of word learning and speech segmentation compared to a model that would operate on speech only; and this model reproduced some behaviours observed in children (such as learning onomatopoeias).

This model however made several simplifying assumptions, notably the fact that speech is perceived categorically in terms of phonemes13 and images had to be pre-processed so as to first detach the background from the foreground, and then isolate the object from the image. Also, the visual input was deliberetaly simplified so that it only contained one object only, hence facilitating the task of the model.

In a model similar to [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF], [START_REF] Yu | The Role of Embodied Intention in Early Lexical Acquisition[END_REF] test if audio-visual mapping was enhanced by gaze information. To test this, they had a picture book in a foreign language read by a native speaker and recorded. The speaker also had a gaze tracker so as to pinpoint where in the picture they were looking while reading a particular sentence. Their model is inputted with a visual context (image from the picture book) and an acoustic stimuli (sentences from the story, in the form a string of phonemes). The model then tries to find repeated patterns in the acoustic stimuli and learns to pair them with the visual context using an EM (expectation maximisation) algorithm. They train their model on two different conditions: the intention-cued condition, and the audio-visual condition. In the first case, the model is inputted with a string of phonemes and the particular object that was watched by the speaker; in the second case, the model is still inputted with a string of phonemes, but instead of a particular object watched by the speaker, the model is given all instances of the objects in the image. Therefore, in the latter, finding a good AV pair is more challenging, as the model has to learn which object is most likely to have occurred with a given recurring audio sequence.

Their results shows that audio-visual mapping was easier when gaze information was available for the model than when it was not. Consequently, their experiment shows that co-occurrence statistics do not seem to be enough, even though they enable computational models to learn a few reliable word-object mappings such as in [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF]. Word-object mapping can be learned more reliably when additional information (such as attention of the speaker) is available. Their results are consistent with research in child language acquisition which shows that shared-attention is a critical parameter for children to acquire their language. Recall that for blind children, it is the lack of shared attention (or its impoverishment due to the lack of a visual context) that leads them to acquire their native language slower than their sighted peers.

CNN-based Neural Models

Neural Networks enabled researchers to model even more complex interactions between the visual and the spoken modalities. [START_REF] Gabriel | Learning Words from Images and Speech[END_REF] introduced, to the best of our knowledge, the first neuronal (CNN-based) VGS model, where the model is trained to map images to isolated spoken words. Their model has two branches, an audio branch and a visual branch which vectorises the input image and word. The model is trained to minimise the cosine squared distance between matching vectors, while making this distance greater for mismatching vectors. Their model is then evaluated on a speech image retrieval task: retrieve the matching image given an input word, or vice-versa. Their results indicate that their network was indeed able to capture cross-modal links and effectively learn to map a spoken word to its visual context. Their model paved the way for models handling more complex acoustic stimuli, such as full captions instead of isolated words. [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] building upon the work of [START_REF] Gabriel | Learning Words from Images and Speech[END_REF] proposed the first model that could handle full spoken captions instead of isolated words. Their model can be considered as an upgraded version of the CELL model as it makes also a few simplifying assumptions. First, the images are processed by a CNN object detector that outputs 20 bounding boxes (19 bounding boxes around each detected object in an image and an additional bounding box for the the whole image); and second, the captions are pre-segmented at word level. The goal of the network is to align each word in the audio caption to each bounding box in the image. Finally, a global similarity score is computed for each image/utterance pair, and the network is encouraged to make the similarity score 2017a) and [START_REF] Khorrami | Can phones, syllables, and words emerge as sideproducts of cross-situational audiovisual learning? -A computational investigation[END_REF]. The difference between CNN-based and RNNbased model depends on neural cells used for the speech encoder. Note that the embedded images and utterances need not be vectors but may also be matrices. The dashed box represents the shared embedding space in which the similarity measures between matching (green arrow) and mismatching (red arrows) image and utterance pairs is realised. Note that the shape of the embedding space depends on the nature of the encoders. The embedding space used by RNN-based models is typically a d-dimensional hypersphere, where d denotes the dimension of the embedding vectors (see Section 3.3.2). Transcription of the audio signal is only given for illustration purposes and is not inputted to the model. Black arrows show the direction of the information flow through the network.

between a matching image/utterance pair higher than with mismatching pairs.14 Similarly to [START_REF] Gabriel | Learning Words from Images and Speech[END_REF] their model is evaluated on a speech image retrieval task. Their results show similar trends to that of [START_REF] Gabriel | Learning Words from Images and Speech[END_REF]: it is easier to find a caption given an image than vice-versa. Nevertheless, in both cases the results are much better than what randomness would predict. [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] observed their model reliably learnt to map isolated words to their correct visual referent via the bounding boxes. [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF] improved on their previous work so that their model could use full captions instead of pre-segmented captions. Also, instead of running an object detector on the image, they simply use the penultimate activation of a pre-trained VGG network, yielding a 4096-dimensional vector representing the input image. The audio input is processed by three layers of 1D-convolutions with multiple filters at each layer. Contrary to their previous work, they use the dot-product of the image vector and the speech vector as similarity measure. This work allowed them observe that some parts of the captions have a higher degree of similarity with the image than others, suggesting their model was able to detect highly relevant acoustic sequences in the speech signal.

Harwath & Glass (2017) built upon their previous work by trying to find image regions regularly associated with spectrogram regions, so as to build audio-visual pairs. While they used the same model as their previous work, their loss function is different and is the same as in their first work [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF], that is a contrastive loss function. In order to infer an audio-visual collection, they first apply a grid over the image so as to divide the original image into sub-sections. They then run a VGG network over all possible groupings of consecutive image blocks so as to have several embeddings for one image. They proceed similarly for the captions, slicing them into subsections and embedding each subsection. They then compute a similarity measure between the representation of each image region embedding and each caption region embedding to find which have the highest degree of similarity. This effectively allows them to extract fine-grained image-audio pairs in an unsupervised fashion. This is even more impressive when one recalls that the network was trained on unsegmented images and unsegmented captions, hence showing the network somehow segmented the spoken input into word-like units.

Finally, in their latest work, Harwath, Recasens, Surís, Chuang, Torralba & Glass ( 2018) completely change their model so as to have a fully convolutional model: instead of using VGG vectors, they use the full VGG network up until the last convolutional layer (i.e. removing the fully connected layers), and use 1D convolutions for the input caption. Also, contrary to their previous work where the output of their audio branch is a 1D vector, they here keep the whole feature map, yielding a 2D matrix. In order to join the output of both branches, they use a dot product as in their previous work. Keeping the feature maps of both branches allows them to detect saliency regions between the convolved image and the convolved spoken input. Hence, they are able to highlight specific regions in the spectrogram that correspond to specific regions in the image, allowing them to build finegrained image/audio pairs in an unsupervised fashion, contrary to their previous work which required computing similarity for each possible subpart of the image and spectrogram manually.

In their work, Harwath and colleagues thus went from finding coarse audio/visual pairs from pre-segmented images and pre-segmented captions to finding fine-grained audio/visual pairs without any supervision from raw speech paired to raw images. Overall, their work shows it is possible to extract word-like units from raw speech using images as a form of weak supervision. Most importantly, while their network was trained to minimise the distance between an image and its matching caption at a global scale, the network derived similarities at a local scale. Hence, lexical acquisition seems to be a by-product of the main task, and appears "naturally" in the model. [START_REF] Kamper | Semantic Query-by-example Speech Search Using Visual Grounding[END_REF] also proposed a visually grounded CNN-based model to learn speech segment embeddings in a query-by-example search task: given an audio excerpt, the network should find semantically, and not only phonetically, related audio excerpts among a collection of utterances. Contrary to the model of [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF] that uses raw images, here images are represented as a "bag-of-visual (semantic) tags": a vector, where each dimension represents a given object in the picture and is assigned 1 if the object that is represented at a given dimension is in the image, 0 otherwise. Speech utterances are sliced into sub-parts ranging from a minimum length up to a defined maximum length. The goal of the network is to embed each speech excerpt and predict the bag-of-visual semantic tags -which boils down to a classification task. Here, the images only serve a grounding purpose and are only used when training the network. The intuition is that the network will be able to learn how to generate similar embeddings for speech segments which have the same referent in the image. For example, the words "flower" and "rose", while being phonetically different, are semantically related, and might refer to the same object in the paired image. Hence, the embeddings of these two words should be similar. Therefore, at testing time, the network should be able to retrieve audio excerpts corresponding to the word "flower" even when prompted with the word "rose". This is in fact what is observed at testing time, the network is able to retrieve semantically related words which are not phonetically close. Using images to ground the meaning of the audio excerpts proves successful. [START_REF] Räsänen | A computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants[END_REF] recently introduced a CNN model specifically designed to study child language acquisition. Their model is trained to encode the spoken input, in the form of a spectrogram, into a bottleneck representation that is used simultaneously in two sub-tasks: the first task is to predict a visual context (in the form of a 60-dimensional vector representing objects) as traditional VGS models do; and the second task, a decoding task, which consists in predicting the original spectrogram. The task thus assumes that the ideal representation should both be able to correctly predict the visual context, and should incorporate enough details so as to be able to reconstruct the spoken input. Their model is able to learn to correctly map a spoken input to the visual referent while making some plausible mistakes such as confusing similar sounding words (e.g. cat/hat) or confusing semantically related words (e.g. nose/mouth). Most importantly, they show that removing the prediction branch of their model yields worse results, suggesting that joint learning of perception and production is essential.

RNN-based Neural Models

Contrary to CNNs, RNNs are designed to process sequences such as speech. Even though CNNs are also able to process sequences, RNNs are more plausible from a cognitive perspective. Indeed, as already mentioned (see Section 2.2.3), the output at a given timestep depends on the predictions of the past timesteps, which is not the case for CNNs. While RNNs are able to model long-term dependencies, CNNs are unable to so, as their predictions do not rely on the predictions made at the previous timesteps. Consequently, the way speech is processed in RNN-based models is closer to how humans process speech, that is, sequentially and incrementally. The first RNN-based VGS model we know of is that of Chrupa la et al. (2017a). This model builds upon their previous work on textual models [START_REF] Chrupa La | Learning language through pictures[END_REF] and their analysis [START_REF] Kádár | Lingusitic Analysis of Multi-Modal Recurrent Neural Networks[END_REF]. The architecture used is very similar to that of the previously mentioned works and has two branches: an audio branch and a visual branch; and uses the same type of loss as [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF]. Similarly to [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], they use VGG vectors as image features. Nevertheless, contrary to [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], the speech encoder consists of stacked recurrent highway networks (RHN, [START_REF] Zilly | l'utilisation conjointe de syllabes et de mots)[END_REF]) instead of convolutional layers. 15 The final embedding of the audio branch is computed by an attention mechanism (see Section 3.4) which learns how to weight the vectors of each timestep, so as to give more weight to specific parts in the speech signal. They evaluate their model on a speech→image retrieval task, that is, given an input spoken caption, the network should retrieve the matching image. They report better results than [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF], showing RNN-based model are also able to adequately map both modalities.

More recently, Merkx et al. (2019) built upon the architecture of Chrupa la et al. (2017a), adding an attention mechanism at each layer (implemented as in Chrupa la et al. 2017a), and using bidirectional recurrent cells instead of unidirectional cells. Their model reached even higher performance levels than Chrupa la et al. (2017a) and [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] on the same data set. Several optimisations were also made, such as the use of cyclic learning rate [START_REF] Smith | Cyclical Learning Rates for Training Neural Networks[END_REF] which consists in having a learning rate which increases and decreases several times between an upper and lower bound, instead of a having a strictly decreasing learning rate as what is usually done. This allows the model to get out of local minima it could be stuck in and converge to a overall better solution.

However, these better results come at the cost of cognitive plausibility. Indeed, as this model uses bidirectional cell, it processes the spoken input from both ends at the same time, which is of course not possible for humans. Furthermore, contrary to the previously mentioned approaches (be they CNN of RNN-based), their model uses bottleneck features [START_REF] Fér | Multilingually trained bottleneck features in spoken language recognition[END_REF] extracted from a multilingual neural network trained to predict phonemes. Hence, by using such features, their model has some knowledge about the basic units of speech which might help the model converge more rapidly as the input is more informative than plain MFCCs.

Recently, [START_REF] Krishnamohan | Audiovisual Correspondence Learning in Humans and Machines[END_REF] explored if VGS models are able to do few-shot learning of novel noun/object pairs. The goal of few shot learning is to give to a neural network as few examples as possible -1 for one-shot learning, n for n-shot learningand see if the model is able to learn from these few examples and generalise. To do so, they used novel (fake) objects from the NOUN (Novel Object and Unsual Name, [START_REF] Horst | The Novel Object and Unusual Name (NOUN) Database: A collection of novel images for use in experimental research[END_REF] data base where fake objects (3D generated images with various views of the same object) are given fake names (e.g. kakimense, tanzerposk, etc.). They define n-shot learning in their experiment as presenting "n augmented variants [i.e. views] of the stimulus [i.e. novel image/noun pair] before the evaluation". Their model also has two branches, a visual branch which extracts bottleneck features, and an audio branch which is inputted with ASR bottleneck features which are then processed by an LSTM. As in the aforementioned models, their model is trained to minimise the distance between the encoded image and the encoded isolated word referring to the object in the image. Their network is first pre-trained on a set of existing noun/object pairs using a computer vision data set, and they then perform transfer learning on the novel objects. Their result shows that the VGS model they train is able to do 1-shot learning, that is, to learn the association between a novel object and associated noun when presented with one instance, and generalise to novel instances of the same object/noun pair. They compare their result with human data and show their model is on par with human results. We could however argue that they consider an idealised case of few-shot learning, where the label is presented in isolation. This is rarely the case, as usually the novel word is embedded in a full sentence, and humans, and more specifically chidren, are still able to do few-shot learning in such case. It remains to be investigated if it is also the case for neural networks.

We should mention a last model before ending this section which, even though it uses text instead of speech, remains interesting in its approach: that of [START_REF] Hill | Understanding Early Word Learning in Situated Connectionist Agents[END_REF] and [START_REF] Hermann | Grounded Language Learning in a Simulated 3D world[END_REF] which also aims at studying lexical acquisition. Contrary to the previous approaches, their model is trained using a 3D virtual environment, which the model, in the form of a virtual agent, can actively explore. The goal of the model is, given an instruction (e.g. "Find and bump into a pencil"), to learn to distinguish the target object (pencil ) in the virtual environment from a distractor object (e.g. fridge). If the model bumps into the correct object, then it is said to have learnt the word for the target object. The model is not only trained to distinguish object based solely on their shape, but also on their colour (e.g. "Find and bump into the blue object"), pattern, or position (e.g. "Find and bump into the object furthest to the left as you look"). The model is free to explore (i.e. "walk") in this virtual world so as to fulfil the instruction it is given. Hence, this task requires the model to understand the instruction it is given and to actively explore its environment so as to understand what the target object looks like. Contrary to the previous approaches were the model is given static data (i.e. the model cannot change the data it is given so as to gain more insight), here the model is free to get a better understanding of the object by seeing different views of the same object by moving in the virtual environment. Thus, even though the input is less realistic and does not reflect the complexity of the real world, the model here is given the possibility of exploring its environment, such as what a child would do, which is impossible with the previous approaches.

The visual branch of their model is a CNN which processes the images of this virtual world (of shape 84 × 84 × 3[RGB colours]). The language branch of their model is a simple feed forward layer which embeds the instruction, presented as a single world. The output of both branches are then merged and further fed to a LSTM which, given the previously taken action (e.g. move right) and the current input, predicts what to do next. Their results are interesting as they correlate well with what is observed for children. Their model also display a slow learning curve at the beginning and then has a vocabulary spurt such as what is observed for children. The authors however observe a color bias when humans usually display a shape bias (i.e. children tend to overgeneralise based on the shape of the object and not so much on their colour). The authors explain this by the fact that colours and shapes are perfectly balanced in their training data, while in real life shape terms are more used than colour terms. Hence, shape bias seems only to depend on the input the child receives and not by its perceptible environment. Hence, RNN-based models are able to learn speech/image mapping as well as CNNbased models. They also seem to be quite flexible models as they are able to quickly learn the mapping between novel object/noun pairs. The fact that these models use RNN cells instead of CNNs also make them more cognitively plausible, and hence constitute ideal test-beds to simulate lexical acquisition.

Representation Analysis

Chrupa la et al. (2017a) tried to understand the representations learnt by their model. First, they used a classifier to test if the embedding of a full spoken caption contains information about the individual words of the caption, and at which layer this information is the most reliably encoded. They found that not all layers are equally informative about the presence of a word. Particularly, they found the lower layers of their architecture to be the least informative, while the second-to-last was the most reliable. This suggests that word-like units are progressively constructed as the information flows through the network, and that a certain amount of computation is necessary so that information appears. A similar observation was made more recently by Merkx et al. (2019).

Chrupa la et al. (2017a) also explored to what extent the learnt embeddings encoded semantics. Their study reveal that the lower layers of their architecture encodes forms while higher layers encode semantics. Surprisingly, their results show that the last layer of their architecture as well as the final embedding encode the utterance semantics less reliably than the previous layers. The authors explain this is the case because the final embedding should correspond as closely as possible to the embedding of the paired image. Hence, the network might tune the final vector according to the visual modality and discard the information which is not useful in the speech signal. [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF] explore to what extent such VGS model encoded phonology. Their study shows that the network's encodings approximately group the English phones by sound class (plosives, fricatives, affricates, etc.). However, the grouping is not perfect and seems to be done on the basis of acoustic factors (formants) rather than on deeper linguistic factors: we could have expected vowel/consonant dichotomy, a clear voiced/unvoiced dichotomy which is not the case here. This study also investigates the encoding of synonyms (e.g. store/shop) and whether the network is able to distinguish them or not, and at which layer precisely. Their results reveal that synonym discrimination is very low for the representations extracted from all recurrent layers except for the last and the final embedding. It thus shows that form is encoded in the lower recurrent layers while meaning is encoded in the top recurrent layer and the final embedding, making synonym detection much harder as no (or little) information on form is present.

Harwath and colleagues also performed analysis on the representations learnt by their

• an up close picture of an elephants [sic] trunk and eye.

• a baby elephant holding it's [sic] trunk out on a dirt ground. • an elephant raising its truck [sic] with a tree in background. • a close up of an elephant's face showing eyelashes and opening at end of trunk. • an elephant pointing its trunk upward as it looks down. 2016) explored if their network reliably encoded several occurrences of the same word, which are pronounced by different speakers. Their results reveal that the network is able to cluster several occurrences of a given word together. This shows the network was able to remove inter-and intra-speaker variability. Their analysis also suggests that their network performs an implicit segmentation of the audio input into sub-units, which was confirmed in a further study [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF]. [START_REF] Drexler | Analysis of Audio-Visual Features for Unsupervised Speech Recognition[END_REF] showed that some neurons were specifically activated by certain sequences of phonemes. Interestingly, and similarly to the observation of Chrupa la et al. (2017a) and [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF], the lower layers of the network are more concerned with form than with sense. They observe this by clustering the activation extracted at different levels of the architecture and show that the lower activations tend to cluster according to speaker identity while activation from the upper layers cluster according to meaning. [START_REF] Harwath | Towards Visually Grounded Sub-word Speech Unit Discovery[END_REF] showed that the second layer of their architecture was particularly sensitive to phone boundaries. Similarly to [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF], their study shows that the representation learnt by the network encodes coarse phonetic categories (fricatives, plosives, etc.).

Data Sets

Because the number of freely available data sets that feature both images and spoken descriptions is limited, most of the aforementioned models [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF], Chrupa la et al. 2017a, Merkx et al. 2019) are trained on extensions of data sets initially created for image vision purposes. The data sets these models use were conceived to train image captioning models, that is, models that generate textual descriptions of images passed as input. The main data sets used for this task are Flickr8k [START_REF] Rashtchian | Collecting Image Annotations Using Amazon's Mechanical Turk[END_REF][START_REF] Hodosh | Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics[END_REF] and COCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF]) which feature images paired with 5 descriptive captions written by humans (see Figure 2.7). These data sets naturally feature grounded language, as each descriptive caption is paired to an image, which constitutes, to some extent, knowledge of the physical world. As the captions were written by annotators upon having seen the image, the image naturally reflects the communicative intent expressed in the captions. An extensive presentation of the audio extension of these data sets is done in Section 3.2 as we use these data sets in this thesis.

Recently, new data sets that closely capture what children see were introduced. Slone et al. ( 2018) introduced a methodology to collect audio-visual data from children interacting with objects using eye-tracking devices and head-mounted cameras. Such data can then be used to train visually grounded speech or text models and study lexical acquisition. [START_REF] Tsutsui | A computational Model of Early Word Learning from the Infant's Point of View[END_REF] for example use such data to study word-object mapping. In their experiment, 56 Chapter 2. Background on Speech Processing and Term Discovery they used videos captured with a head-mounted camera that shows the perspective of the child. They also used an eye tracker to pinpoint where the child was looking at each time frame and simulate visual acuity by blurring the image except at the focal point. Hence, the data is much more realistic than any other computer vision data set. The model used in Tsutsui et al. ( 2020) is a supervised model which is trained to classify an image as featuring (or not) a given object (e.g. bed, turtle, etc.). Their results show that wordobject association is more successful when using data with the visual perspective of the child than with the visual perspective of the caregiver. Following a similar methodology, the SEEDLingS project [START_REF] Bergelson | Nature and origins of the lexicon in 6-mo-olds[END_REF] 17 gathered data of child/caregiver interaction using a head mounted camera and microphones. This data was used to train the model of [START_REF] Räsänen | A computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants[END_REF] presented above.

The results obtained by [START_REF] Tsutsui | A computational Model of Early Word Learning from the Infant's Point of View[END_REF] and [START_REF] Räsänen | A computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants[END_REF] highlight the need for more realistic data sets if one wants to use computational simulations to understand child language acquisition as advocated by [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF]. Indeed, data sets such as Flickr8k or COCO only reflect the perspective of adults and not the perspective a child could have of the same situation. Hence, models trained only on these data sets might have sub-optimal results. However, realistic data sets are often not publicly available, which explains why so few models are trained using realistic data.

Neural Models and Language Acquisition 2.3.2.1 Simulation or Modelling?

Most of the aforementioned models are inspired by child language acquisition, or at least make an explicit link between the ultimate goal of their model and child language acquisition. It seems reasonable to ask to what extent these models do model child language acquisition or do simulate child language acquisition, or in fact do something else. This distinction is important as it might influence the weight of the conclusions that will be drawn.

We may try to answer this question by referring to Marr's [START_REF] Marr | Artificial intelligence-A personal view[END_REF]) levels of information processing (see more specifically Marr 1983, p. 24). Marr distinguishes three levels of information processing: the computational level, the algorithmic level, and the implementation level. The computational level is concerned with the problem to be solved. It defines what is the object of the computation (i.e. the input and the output) and why those are the objects of the computation. This level also defines abstractly the constraints that should be satisfied by the computational process. The second level, the algorithmic level, is concerned more precisely with how the problem should be solved. Each step of the computational process is precisely defined. Note that this level is different from the computational level, as the computational level defines the task rather abstractly and what constraints it should obey to, while the algorithmic level precisely defined how the computational process takes place. Indeed, "there may be many algorithms that implement the same computation" [START_REF] Marr | Artificial intelligence-A personal view[END_REF]. The final level, which is for us of little concern, is the implementation level, which also defines how the computations are done, this time at the physical level: are the computations supported by biological objects (i.e. real neurons) or artificial objects (i.e. silicon chips).

The difference between simulation and modelling is concerned with the first two levels only, that is the computational level and the algorithmic level. Rieder (2003, p. 818) makes a difference between the two terms: "simulation conveying the action of imitating reality and [the] model representing the vehicle". Simulation is thus "the act of presenting the appearance, or interacting with the behavior, of a system without the reality" while modeling would be "the generation of a facsimile or representation of the real system", the latter being "physical, mathematical, procedural [i.e. algorithmic], or some combination". 18Hence, referring to Marr's hierarchy, the model is defined at the algorithmic level while the simulation is defined at the computational level. There could well be several possible models (i.e. several algorithms) possible that would generate the same simulation.

We believe that if one wants to model child language acquisition, the model should be as close as possible to what humans do algorithmically, and for example incorporate the same priors as humans, which is not the case with any of the aforementioned models. Hence, even though they do simulate child language acquisition to some extent, and more specifically lexical acquisition, they do not model child language acquisition. Nonetheless, not implementing the same "vehicle" as humans does not preclude these implementations to display similar patterns in the final simulation, hence making their implementation and study worthwhile so as to test hypotheses.

Perfect Simulation and Groundedness

According to Dupoux (2018) (and others), a successful approach to study child language acquisition using computational approaches should satisfy the following constraints: construct[ing] scalable computational systems that can, when fed with realistic input data, mimic language acquisition as it is observed in infants [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF] If one wants to mimic language acquisition as it is observed in infants -which would be what we define as a perfect simulation -the aforementioned models are not entirely suitable. Indeed, the models we presented are inputted with an audio caption which they have to learn to map to a paired image. Consequently, these models only incorporate the perceptive abilities of a child but not its productive skills. Hence, this leads us to question to what extent these models really are grounded. [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF] further defines grounding as "an interactive process of predictive control and causal feedback." and describes the ideal grounding model (which he calls a computational semiotics framework ) in a figure which we reproduced in Figure 2.8. The feedback loopslanguage production, and physical action -enable the child to act on the physical world, either by moving and gaining another view of what she is looking at, or by speaking to the persons around her, which prompts a response back from them. Therefore, the ideal grounded model should be both able of perception and production so as to really mimic child language acquisition.

None of the models we presented do implement all the feed-back loops necessary to have a fully-fledge grounded model. They all only implement a sub-set of it, and hence are not able to entirely simulate child language acquisition. Indeed, the models of [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], Chrupa la et al. 2017a, Merkx et al. 2019) only have a perceptive ability. While the model of [START_REF] Räsänen | A computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants[END_REF] 19 is trained to reconstruct the input utteranceand somehow incorporating productive skills -it does not allow for the model to actively explore its environment, whereas the situation is opposite for the model of [START_REF] Hill | Understanding Early Word Learning in Situated Connectionist Agents[END_REF]. However, despite not incorporating all the aspects of child language acquisition, they do incorporate a sub-set of it, which is lexical acquisition. This process indeed implies building schemas about the world (i.e. neural representations) using a lingusitic input (i.e. audio captions) and a non linguistic input (i.e. a visual context in the form of still images).

In this thesis, we will focus on studying lexical acquisition in RNN-based models, and more specifically in a model similar to that of Chrupa la et al. (2017a). Note that the conclusions we will be able to draw and the links we will make with lexical acquisition as observed in children will necessarily be limited. We further describe this issue in Section 3.3.4.

Conclusion

Previous research shows that VGS models, be they CNN or RNN-based, are able to successfully learn how to map a spoken stimulus to a visual stimulus. These works also reveal that, despite having only been trained to minimise a distance between an image and its spoken description so as to map both modalities accurately, these models have developed deeper linguistics abilities. This is an interesting property, as these linguistic abilities only appear as a by-product of the main task. This seems somehow similar to how children learn. They indeed do not have an explicit linguistic task laid out before them, but their linguistic abilities rather emerge as by-product of the interactions with the world around them.

While these abilities were widely explored for CNN-based models, they were not for RNN-based model. We thus aim in this thesis to better understand what linguistic abilities VGS RNN-based models are able to develop. More specifically, we aim at answering the following questions:

• Do RNN-based models highlight specific part(s) of the spoken input that are particularly relevant to predict the target image, such as what was shown for CNN-based models? [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] • If so, what specific parts of the input are highlighted (which specific word-forms, or part-of-speech)?

• If they do highlight specific parts in the input, is this ability cross-linguistically valid,

Conclusion 59

even if the model is trained with a language typologically different from English, say Japanese?

• To what extent such linguistic abilities develop over time? Are VGS models able to quickly acquire some linguistic abilities (i.e. with a relatively small amount of data) or not?

• Previous research (Chrupa la et al. 2017a;Merkx et al. 2019) shows that the internal representations of such network encode the presence of the individual words of the spoken captions. This raises the question whether this behaviour is valid for all the words of a given caption, or only specific words?

• If the presence of individual words are encoded in the internal representations of the network, it means the network has learnt what constitutes a word and has stored this information in its weights. This raises the question of how the network recognises (i.e. activates) the representation of a given word based on an acoustic input.

• VGS models are trained with full unsegmented captions and obtain fair results. However, their textual counterparts (i.e. networks trained on written captions) do better. This raises several questions, the main one being: if VGS networks were presented with segmented captions, would they do better?

Part II

Contributions

Chapter 3

Visually Grounded Speech Architectures and Data

Introduction

Children learn their native language in context. That is, raw speech alone is not sufficient for them to acquire a language: language needs to be grounded in order for them to make sense of what is being said. Grounding is necessary to map linguistics signs -signifier/signified pairs -to their referents. Linguistic signs are not readily available objects that are to be found in the real world but they also have to be constructed from it instead. Through repeated exposure to referents in conjunction with linguistic signs, the child is able to build form/meaning pairs. Grounding may thus occur through various sensory-motor experiences: vision, touch, smell, taste, hearing, thermoception, social interactions, etc.

Neural architectures of Visually Grounded Speech (VGS), be they CNN-based or RNNbased, have recently become popular as they enable to model complex interactions between two modalities, namely speech and vision. Such architectures can be used to model child language acquisition and more specifically, lexical acquisition. Indeed, these architectures are trained to solve a speech-image retrieval task. This task involves identifying lexical units that might be relevant in the spoken input, detecting which objects are present in the image, and finally pairing the detected objects to the detected spoken lexical units. Their task is very close to that of a child learning her mother tongue, who is surrounded by a visually perceptible context and who tries to match parts of the acoustic input to surrounding visible scenes.

In this chapter, we present the VGS architecture that is studied throughout this thesis. We also present the data sets that are used to train such architecture and introduce a new data set in Japanese consisting of image paired to audio descriptive captions. Finally, we mention the assumptions we make on language learning by using the particular architecture we use in conjunction with the data sets we present.

Data

Deep learning architectures require large amounts of data in order to be trained effectively and eventually converge. In our case, the architecture we train requires images that are paired to spoken descriptions. While image recognition and captioning data sets are numerous, only a few number of data sets featuring images paired with audio descriptions are freely available. We here present the three data sets that are used in the experiments of this thesis, two of them featuring synthetic speech (COCO and STAIR) and one featuring real human speech (Flickr8k).

COCO and STAIR

Two of the data sets we use for our experiments are based on the Microsoft Common Objects in Context (MSCOCO) data set [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF]). 1 This data set, initially introduced for computer vision purposes, consists of a set of images paired to five descriptive textual captions. The images show scenes of everyday life which feature at least one instance of the 80 target objects the data set was conceived around. The images comprising the data set were gathered on Flickr,2 thus insuring a diversity of contexts and views in which the 80 object instances are presented. The captions were written by native English speakers recruited on Amazon Mechanical Turk.

In order to have a data set consisting of images and spoken descriptions, Chrupa la et al. (2017a) introduced the Synthetically Spoken COCO data set. It consists in a synthesised version of the original MSCOCO captions using Google's Text-to-Speech (TTS) system [START_REF] Chrupa La | Synthetically Spoken Coco[END_REF]). The audio captions were synthesised using one synthetic female voice with an American accent. This resulted in 616,435 spoken captions for 123,287 images. The original training, validation and test splits of [START_REF] Vendrov | Order-Embeddings of Images and Language[END_REF] were kept and consist of 566,435, 25,000, and 25,000 captions respectively. 3 From now on, this data set will be referred to as COCO.

One of today's pitfall of NLP and SP research is that most of the work is carried out on mainstream languages4 that are, for most of them, Indo-European languages. This has several impacts, the main one in our case being that it could bias our analysis. Indeed, Indo-European languages are very close typologically. Consequently, it might be that the patterns that are uncovered when analysing neural network might only stem from the fact that the languages are typologically related and the analysis might miss the bigger picture. Hence, in this thesis we introduce a speech/image data set in Japanese.

Our Japanese data set is based on the STAIR data set by [START_REF] Yoshikawa | STAIR captions: Constructing a Large-Scale Japanese Image Caption Dataset[END_REF]. Using the same methodology as [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF], [START_REF] Yoshikawa | STAIR captions: Constructing a Large-Scale Japanese Image Caption Dataset[END_REF] collected five captions in Japanese for each image of the original MSCOCO data set. We chose this data set as it has several advantages over other text/image data sets in other languages. First, it uses the same set of images as the English data set (as opposed to the English-German Multi30k data set [START_REF] Elliott | Multi30K: Multilingual English-German Image Descriptions[END_REF]) which uses a different set of images). Second, the STAIR data set has captions for the whole original MSCOCO data set (as opposed to the Chinese version [START_REF] Li | COCO-cN for Cross-Lingual Image Tagging, Captioning, and Retrieval[END_REF]) which only has captions for a small subset of the original data set). Finally, the captions were written by native Japanese speakers. They are original captions and are not translations of the English captions (as opposed to the Japanese captions introduced by [START_REF] Havard | SPEECH-cOCO: 600k Visually Grounded Spoken Captions Aligned to MSCOCO data Set[END_REF] which used computer generated translations). Following the same methodology as Chrupa la et al. (2017a), we used Google's TTS system to synthesise speech for each of the Japanese captions [START_REF] Havard | Synthetically Spoken STAIR[END_REF]. In order to enable a fair comparison between the models trained on the English version of the data set and those trained on the Japanese version, we kept the exact same training, validation, and testing splits. 5From a typological point of view, Japanese is widely different from English, be it phonetically, phonologically (mora-based language), morphologically (agglutinative) or syntactically (OV language, explicit function marking with particles). This will thus enable us to test whether the behaviour of our architecture varies according to the language used for training, and if so, what kind of language-specific strategies the resulting models develop.

FLICKR8k

The audio quality of both the COCO and STAIR data sets is very high as both feature synthetic speech. However, as noted by Chrupa la et al. (2017a), the generated speech is much clearer and simpler than real human speech: there is no inter-speaker variation, as only one synthetic voice is used, and there is very few intra-speaker variation: two occurrences of a word in the same context will be pronounced exactly the same. Also, real human speech is characterised by false starts, repetitions, corrections, and hesitations which are not present in synthetic speech. We therefore felt the need to validate our conclusion on real human speech using the Flickr8k data set.

As for MSCOCO, the Flickr8k data set [START_REF] Rashtchian | Collecting Image Annotations Using Amazon's Mechanical Turk[END_REF][START_REF] Hodosh | Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics[END_REF]) was originally conceived for computer vision purposes. It contains 8,000 images, each paired to five written descriptions written by humans also recruited on Amazon Mechanical Turk. [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] had each of the captions read by native English speakers, resulting in 40,000 audio captions. As this data set features 183 different speakers, it is much more challenging than the previous ones. The audio quality is also uneven from a caption to another, some being very neatly recorded while other have a lot of background noise.

As Chrupa la et al. (2017a), who also used this data set in their original experiments, we kept the original splits provided by [START_REF] Karpathy | Deep Visual-Semantic Alignments for Generating Image Descriptions[END_REF] resulting in a training, validation, and testing set consisting of 30,000, 5,000 and 5,000 audio captions respectively. 6 Needless to say, we expect the results obtained using this data set to be lower than when using either COCO or STAIR, as modelling real speech is notably harder than synthetic speech and also because this data set is much smaller than the other two.

Data Format

Contrary to the VGS model of [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] which used raw images and spectrograms, the architecture we used requires the data to be pre-processed before being inputted to the network.

For COCO and STAIR, which both use the same set of images, we use image vectors extracted from a VGG network [START_REF] Simonyan | Very Deep Convolutional Networks for Large-Scale Image Recognition[END_REF] trained on ImageNet. The features we used for our experiments are those provided by [START_REF] Vendrov | Order-Embeddings of Images and Language[END_REF] which are averaged over 10 crops.7 For Flickr8k, we used the image vectors provided by [START_REF] Karpathy | Deep Visual-Semantic Alignments for Generating Image Descriptions[END_REF],8 also extracted from a VGG network.

The vectors we use do not correspond to the activation of the last layer of the VGG network,9 but we use the activations of the last fully connected layer of the VGG network. Thus, the vectors computed at this layer are informative of the content of the image without being restricted to a fixed number of objects to predict.

Both for COCO and Flickr8k we use the acoustic features provided by Chrupa la et al. (2017b) which were extracted using Python Speech Features. 10 The audio features for COCO consist of 12 Mel-Frequency Cepstral Coefficients (MFCC, Picone 1993) and energy. For Flickr8k, audio features consist of 12 MFCCs with deltas, delta-deltas and energy. We extracted MFCC features for the synthetic STAIR data set using the same parameters as for COCO.

Metadata

For each data set, we force-aligned the speech with its transcription at word and phone level using the Maus forced aligner (Kisler et al. 2017). 11 We also tagged each caption: we used TreeTagger [START_REF] Schmid | Probabilistic Part-of-Speech Tagging Using Decision Trees[END_REF] for English part-of-speech (POS) tagging using the default model and KyTea [START_REF] Neubig | Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis[END_REF] for Japanese POS-tagging using the default model. Because both taggers use different tag sets, that the tag sets are unnecessarily fine-grained (especially for English) and for comparison purposes, we converted each tag to its Universal POS equivalent [START_REF] Petrov | A universal Part-of-Speech Tagset[END_REF]) which uses a coarser tag set.

While mapping PennTreeBank tags to their Universal POS equivalent was rather straightforward, it was not the case for KyTea's POS tag set. KyTea's analysis is closer to that of a morphological tagger than to that of a coarse-grained POS tagger as exemplified by the example thereafter: For example, the verb "積ま" (base form 積まる) is split in two parts by KyTea, respectively "積" ("tsu", VERB)" and "ま" ("ma", TAIL). For our analyses, we considered both parts as forming a single token which we labelled as a VERB. Such overanalysis also occurs for adjectives. In such cases, we considered the sequence ADJ+TAIL as forming only one token of type ADJ.

Another example of KyTea's over-segmentation and analysis is shown in Table 3.2 where the word "work vehicle" is split in two parts. In our analysis, we considered such suffixes as having the same POS as the preceding word. . Such models are trained to solve a speech/image retrieval task: given a input spoken description, they retrieve the image that matches the description the closest. To do so, such models project the image and its matching description in a common vector space so that matching pairs lie close in the representation space while mismatching pairs lie far apart. Therefore, the speech encoder and the image encoder have to learn how to trasnform these two modalities appropriately so the resulting image vectors and matching speech vector lie near in the final embedding space. (2020).

Japanese

The speech encoder consists of a 1D-convolutional layer that subsamples the input vectors followed by five layers of Gated Recurrent Units (GRU) [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF]) with residual connections [START_REF] He | Deep Residual Learning for Image Recognition[END_REF]. Contrary to the original architecture of Chrupa la et al. ( 2017a) which has only one attention mechanism, we use two attention mechanisms (see section 3.3.3 for details): one after the first recurrent layer, and a second after the fifth recurrent layer. The final vector produced by the speech encoder is an element-wise product of the vectors produced by both attention mechanisms. This vector is then normalised to the unit 2 -norm. Contrary to the original implementation that uses Recurrent Highway Units (RHN) [START_REF] Zilly | l'utilisation conjointe de syllabes et de mots)[END_REF], we decided to use GRUs as these cells are widely used by the research community, and also because their behaviour is better understood. Furthermore, it is to be noted we used unidirectional GRUs and not bidirectional GRUs. Indeed, unidirectional GRUs process the input sequentially from left to right, thus respecting the temporal dimension of speech, whereas bidirectional GRUs process input both from left to right and from right to left at the same time. We thus decided to use unidirectional GRUs as those are more cognitively plausible as humans process speech from left to right and not both ends at the same time. 14

Contrastive Loss Function

The network is trained to minimise the following triplet loss function (as used by Chrupa la et al. (2017a) and orignally proposed by [START_REF] Weinberger | Distance Metric Learning for Large Margin Nearest Neighbor Classification[END_REF]):

L(u, i, α) = u,i u max[0, α + d( u, i) -d( u , i)] + i max[0, α + d( u, i) -d( u, i )] (3.1)
where u is an encoded utterance, i an encoded image, u and i are mismatching utterances (respectively images) with respect to image i (respectively utterance u). This loss function

13 that is x = x x 2 where 2 -norm = x 2 ∆ = d i=1
x 2 i so that all vector lie on a d-dimensional hypersphere.

14 Humans can also process the spoken input in reverse (in case one misunderstood what was said for example). However, this happens once the spoken utterance has been processed once, thus the reverse processing only takes place after, whereas for bidirectional RNNs, the input is processed from both ends, independently and at the same time.

encourages the network to minimise by a margin α the distance d( u, i) between the encoded image i and the encoded utterance u belonging to matching image/utterance pairs while making the distance greater for mismatching image/utterance pairs. The loss is computed at batch level, that is, all the images inside a batch (except image i) are considered as contrastive mismatching examples for utterance u, while all the other utterances (except image u) inside the same batch serve as contrastive mismatching example for image i. 15 In our case, the distance d used is the cosine distance,16 defined as follows:

d( u, i) ∆ = 1 -cosine similarity ∆ = 1 - u • i u i (3.2)
The domain of cosine similarity is [-1, 1], where cos(0 • ) = 1; cos(90 • ) = 0; and cos(180 • ) = -1. Hence, if two vectors are collinear (with the same sign), their cosine similarity will be 1, and consequently the cosine distance will be 0. Consequently, orthogonal vectors will have a cosine distance of 1, and opposite vectors (collinear with different signs) a cosine distance of 2. The loss function encourages the network to produce collinear vectors for matching speech/image pairs. Even though the batches are created by randomly selecting examples in the training set, it could be that, in the same batch, two images are adequately described by the same audio caption. Such pairs would act as mismatching examples even though in reality they are not truly mismatching pairs. In practice, it does not impede learning.

Attention Mechanism

One of the key components of this architecture is its attention mechanism (as presented in Section 3.4). We remind to the reader that the attention mechanism computes the context vector c as follows:

c = T t=1 α t h t (3.3)
where T is the length of the sequence, h t is the hidden stated produced by a recurrent cell at time step t, and α t is a learnable parameter. Recall that the attention mechanism learns how to assign a weight α t to each input vector. The higher the weight, the more importance is given to the vector in the final representation. In our case, high attention weights over specific vectors means the network has paid more attention to specific portions of the input signal.

We decided to use two attention mechanisms at two different levels of the network in order to see if the network highlighted the same type of information at different layers or not. The final vector is the dot product of the vectors computed by both attention mechanisms:

c = T t=1 α GRU 1 t h GRU 1 t • T t=1 α GRU 5 t h GRU 5 t (3.4)
where T it the total sequence length, h GRU 1 t and h GRU 5 t respectively represent the hidden vector computed at time step t by the 1 st and 5 th layer, and α GRU 1 t and α GRU 5 t is the attention weight computed respectively by the attention mechanism following the 1st and the 5th recurrent layer. The final vector is normalised to the unit 2 -norm. 17The idea of using more than one attention mechanism stemmed from the article by [START_REF] Mattys | Integration of Multiple Speech Segmentation Cues: A hierarchical Framework[END_REF] which shows that human attend to different cues when processing speech in order to isolate words from the speech stream. 18 The first attention mechanism we use is placed after the first recurrent layer, hence very close to the acoustic information. The network could thus use acoustic-related information (such as pitch) to highlight relevant units. The second attention mechanism being higher in the architecture, we expect the information to be more abstract, and thus we except the attention mechanism to highlight units based on semantic cues. We hence aimed at checking if the neural network highlighted different type of units by analysing the weights of the two attention mechanisms.

Assumptions in the Model 19

In this thesis, we will draw analogies between the learning processes and strategies of a neural network and those of a child learning her mother tongue. However, it is worth mentioning that our comparisons will be limited as there are many differences between how learning occurs in a machine and in a human. By design, the model and data we use also makes certain (limiting) assumptions which we will review here. We may classify the assumptions that are made in this thesis in three categories: assumptions on the data, assumptions on the task, and assumptions on the computations.

We first examine the limitations that ensue from the data set we use. Our data set is biased by the nature of the data, as first, it only contains images that, by nature, are still while the real world is not; and second, because these images were selected according to a predefined set of objects they should depict, they do not reflect the distribution of the objects children encounter in their daily life. Therefore, the salient parts of the images are static objects and not (or rarely) actions the persons depicted in the images could undertakeare undertaking. Thus, the descriptions are equally biased and mainly describe the objects present in the image and not actions. Additionally, in our experiments every picture is paired to a spoken description. This is a very strong assumption which occurs quite rarely in real life. Indeed, even though children and adults tend to pay attention to the same things in their environment (see Section 1.3.2) and thus tend to speak about what is being attended to, it is often the case that adults refer to absent persons, objects, or situations when they speak. In short, it is not because we speak in the context of a particular situation that the discourse that is held necessarily refers to this situation, and if it refers to the particular situation, it might not be a description of the situation.

Second, in our experiments we make assumptions on how learning occurs. This assumption ensues from the task our neural network is trained to solve. The neural network we use is trained to solve a very specific and well-defined task which is a speech-image retrieval task: given a spoken description of an image, the network should retrieve the image that matches the input description the closest. This task is artificial and does not correspond to what humans do when they learn their language. Humans do not restlessly try to predict a visual context from what they hear. Nonetheless, the core skill that solving such task implies, that is, learning a mapping between a visual stimulus and an acoustic stimulus, is a skill that children develop when they learn their language. However, this core skill is just one of the many skills children need to develop to learn their language, but not the only one. Even though visual stimuli play an important role in language learning, vision is not the only way to ground speech: smell, touch, taste, social interactions, for example, are equally (or even more) important than vision to learn a language (see Section 1.3.5.2). Yet, for the particular neural network we study, vision will be the only modality used to ground speech.

Finally, in this thesis we make assumptions on the memory capacities and the nature of the computation done by humans. One assumption we make by using the model we use is that image processing is decoupled from audio processing. Indeed, in our model, both processes are done in parallel, but independently while in human cognition both visual and audio processing is done jointly (see [START_REF] Mcgurk | Hearing lips and seeing voices[END_REF] for an example). Thus, in our model there is no true interaction between the visual stimuli and the audio stimuli. The interaction between both modalities only occurs through the loss function. Moreover, as our network uses pre-trained VGG vectors to represent the content of the images it implies that one should first learn how to see before mapping what is visually perceived to what is heard. It would thus imply that children have from the beginning on a clear and robust categorical perception of their visual surroundings, which we know is not the case: babies have a blurry vision and shorter sight angle, which implies they are not able to perceive distant objects as clearly as adults.

The way the spoken input is processed by the network makes also false assumptions on what we know of human speech processing. MFCC vectors are hand-engineered features that aim at reproducing how speech is perceived by humans. However, the speech encoder we use does not implement several innate priors which we know children have, such as categorical (discrete) perception of speech. Our speech encoder must thus be considered as a blank slate that needs to learn how to encode speech, which is not the case when children are born.

The loss function we use (presented in Section 3.3.2) also makes several (false) assumptions about the memory capacities of a child or humans. This function computes a loss value at batch level. This implies a child would be able to compare a given situation to several other situations, and would have, by doing so, stored in mind the exact sentence and image representation of theses situations. This is of course impossible at a large scale.

Hence, the conclusions we draw should be understood in light of the presented assumptions and limitations. If vision -in form of still images -were to be the only modality used to make sense of the surrounding speech, and that the surrounding speech always referred to the visual context, what regularities should we expect children to have picked up?

Chapter Summary

In this chapter, we presented the VGS speech model we used and detailed the modifications we made to the original architecture of Chrupa la et al. (2017a), and detailed each part of our architecture. We also showed the assumptions using such a model implied and showed the limitations of using the model and data we use in our conclusions.

We presented the data sets used to train such architectures (Flickr8k and COCO). The true contribution we made in this chapter was introducing a new audio/image data set based on the original text STAIR data set [START_REF] Yoshikawa | STAIR captions: Constructing a Large-Scale Japanese Image Caption Dataset[END_REF]) that we created following the same methodology as Chrupa la et al. (2017a). This data set enables us to compare the performance of our network on two comparable corpora, the only changing factor being the language: either English (COCO) or Japanese (STAIR). This data set, entitled Synthetically Spoken STAIR [START_REF] Havard | Synthetically Spoken STAIR[END_REF], is openly available to the research community: https://zenodo.org/record/1495070#.Xyv1dCgzZhE.

Finally, we also made available the POS tags and forced alignments of both the Sythetically Spoken COCO and Synthetically Spoken STAIR data sets: https://github. com/William-N-Havard/VGS-dataset-metadata so that the research community is able to reproduce our results.

Introduction

In the previous chapter, we presented the model of VGS we used and mentionned that this model is trained to solve a speech-image retrieval task. The main hypothesis we have in this thesis is that, in order to learn a reliable mapping between and image and its spoken description, the model should implicitly learn to segment the spoken input into sub-units. As the images used mainly figure objects and not actions, we hypothesise that the model should implicitly learn to segment noun-like units.

In this chapter, we study the attention mechanisms of our models and analyse which parts of the input signal are highlighted. Indeed, as previously mentioned, attention is a tool the network is given so as to favour some part of the input signal over other. We hypothesise that our network uses this tool as a form of segmentation module in order to highlight particularly relevant words in the spoken input.

We focus on analysing where attention is located, that is, we study which parts of the input speech signal are more favourably highlighted than others. We first start by showing that the learnt attention weights are not random and do highlight specific parts of the speech signal. We then study how the distribution of the attention weights evolves over time. Indeed, as the attention mechanism is a trainable component of our network, the distribution of the attention weights evolves over time. Finally, we compare what is known of language acquisition in children to the behaviour we observe in our models.

We believe the main originality of this chapter resides in the fact that the work we present is the first to study the behaviour of attention in two typologically different languages: English and Japanese. Such methodology enables us to isolate the language general behaviours from the language specific behaviours of our models.

Is Attention Explanation?

Before diving into the experiments of this chapter, we have to address an issue that was recently raised regarding the reliability of studying the attention weights of a given model. [START_REF] Jain | Attention is not Explanation[END_REF] claim in their paper "Attention is not Explanation" that attention weights do not provide a meaningful explanation of a neural network's predictions. Notably, they show it is possible to find alternative distributions of the attention weights while keeping the predictions intact. They introduce two ways of doing so: the first one consists in randomly shuffling the attention weights and the other one consists in constructing adversarial attention weights whose distribution differs as widely as possible from the original distribution while leaving the final prediction unchanged. They come to the conclusion that because attention weights can be modified without changing the ouput of the neural network they study, attention weights do not constitute a reliable source of information to understand the predictions of a neural network.

Wiegreffe & Pinter (2019) in their response paper "Attention is not not Explanation"1 however mitigate the initial statement of [START_REF] Jain | Attention is not Explanation[END_REF]. They first state that the goal of studying attention is not to explain all of a model's behaviour by only looking at its attention weights: "attention scores are used as providing an explanation; not the explanation". We totally subscribe to this view. Notably, they observe that the adverserially computed attention weights usually perform worse than the initial attention weights showing "the relationship between tokens and prediction [...] cannot be easily 'hacked' adversarially". The experiments of both papers suggest that manipulating the attention weights yields uneven results depending on the data set, suggesting that in some cases it is legitimate to consider attention weights as a form of explanation.

Even though the experimental settings of Jain & Wallace (2019) widely differ from ours -their task is sentiment analysis while ours is speech-image retrieval, they use discrete units as input whereas we use acoustic vectors that are continuous by nature, their output consists in a binary classification while ours consists in predicting a vector in a continuous space, etc. -we believe the issue they raised is important. In order to ascertain that the attention weights in our models do constitute a form of explanation, we will perform a few sanity checks. We will randomly shuffle the attention weights of both attention mechanisms. If the scores obtained with shuffled weights are worse than those obtained with the original weights, we will be able to conclude the original attention weights are useful for the models' prediction and can thus constitute a good way of understanding our models' behaviour.

Studying Attention

In this chapter, we aim at understanding the behaviour of the attention mechanisms of our architecture. More specifically, we study which parts of the spoken input the model pays attention to by analysing the attention weights. Our analyses focus on the following points:

• Which POS are highlighted;

• Which words are highlighted;

• Which parts of a given word are more specifically highlighted;

• How the distribution of the attention weights evolves over time.

After having trained models on either the English data set or the Japanese data set, we encode each caption of the test set and extract the attention weights α (see 3.3.3) for both attention mechanisms. Recall that the higher the weight is for a given vector, the more importance this vector will have in the final representation. We then use a peak detection algorithm2 to detect the local maxima (that from now on will be referred to as peaks) in the attention weights. We considered as peaks local maxima that were at least 60% as high as the highest detected peak. For the rest of this thesis, we will say that attention highlights a specific part of a unit (POS, word, etc.) if there is a local maxima among the attention weights assigned to the vectors that comprise this unit. A visualisation of how attention weights are distributed on a specific caption is shown in Figure 4.1 where high attention weights that are considered to be peaks are marked with a large orange marker.

In order to understand if the highlighted units are different from what chance would predict, we could compare the proportions of peaks highlighting a given unit to the token frequency of the same unit and see if both are close or not. However, doing so would introduce a bias. Indeed, spoken units vary in length (nouns are typically longer than determiners or prepositions), and tokens do not account for this length difference. In order to have a baseline reference, we instead use random peaks as a baseline. For each caption, we sample 10 × n random peaks, where n is the number of true detected peaks in the caption. A random peak is thus a randomly selected time step in the vector sequence, where each time step is equally likely to be selected as the other. We then compute word and POS distribution under such random peaks. Doing so enables us to account for the size difference of different words and enables us to estimate peak distribution if peaks were to be randomly distributed.

For brievety reasons, from now on, the attention weights computed by the attention mechanism following the first layer and fifth GRU layers will be referred to as "GRU1" and "GRU5" respectively. Therefore, the sentence "GRU1 highlights more prepositions than determiners" should be understood as "the attention mechanism computing the attention weights for the hidden states of GRU1 highlights more prepositions than determiners".

Experiments on Synthetic Speech: COCO & STAIR

Experimental Settings

We used the same experimental settings for both COCO and STAIR in order to enable a fair comparison and used the same experimental parameters as Chrupa la et al. (2017a): a 1D convolution layer with 64 filters, a window size of 6 and a stride of 3, followed by 5 recurrent layers consisting of 512 units with residual connections. Attention dimension was set to 512, Adam optimiser with a initial learning rate of 0.0002, margin size α of 0.2. As in the original implementation, we did not use any dropout as we found out that it hindered performance. The model was trained for 15 epochs.

Results

In order to insure that our results were not due to chance, we trained five models, each with a different seed. The results presented in Table 4.1 are an average (± standard deviation) of the results obtained across the five models. Models are evaluated in terms of recall@k (R@k) on a speech-image retrieval task. For each spoken query, the images are ranked from the closest matching to the least matching image. R@k evaluates if the target image (i.e. the image truly paired with the query caption) is ranked among the first k images. We also report median rank r which informs us at which rank the true images are ranked on average.

Model

R@1 R@5 R@10 r English (COCO) Overall, our results are worse than the results obtained by (Chrupa la et al. 2017a). This is to be explained by the fact we swap the RHN cells for GRU cells. Even though our results are lower, they still remain much better than chance scores, showing the models we trained did effectively learn to map an image with its spoken description.

Random Attention

The first question we need to answer before analysing the patterns of our attention mechanisms is the following: is attention really useful? To do so, we either shuffled the attention weights of both attention mechanisms at the same time, or only shuffled alternatively one of the two so as to estimate the contribution of each in the final prediction. We did so on the best trained model (selected on the validation set) of each of the five runs for both COCO and STAIR. We present the results averaged over five runs (± standard deviation) of randomly shuffling the attention weights in Table 4.2.

When the attention weights of both attention mechanisms are shuffled, we observe that R@1 is barely above 0% showing the network is barely able to find the correct image given a spoken query. It clearly shows that the learned attention weights do highlight very specific parts in the spoken input, and that the highlighted parts are essential for the network to correctly encode the spoken input.

We observe a different pattern when we only shuffle the weights of only one of the two attention mechanisms. When we shuffle the attention weights for GRU1 but leave those of GRU5 intact, we notice that the network obtains better results than when we do the opposite. In both cases, the results are worse than when we leave the attention weights of both attention mechanisms intact (see Table 4.1), but better than when both are shuffled. This shows that the attention weights of GRU5 are more important than those of GRU1, as shuffling the weights of the former has much more (negative) impact than shuffling the weights of the latter. Thus, this experiments allows us to conclude that both attention mechanisms highlight units that are useful for the networks' predictions and hence, that analysing the attention weights is a legitimate endeavour.

Rand

Highlighted POS

Following the methodology introduced in Section 4.3 we analyse which POS both attention mechanisms highlight.

For COCO (Figure 4.2), we notice a large asymmetry in the POS that are highlighted by GRU1 (Figure 4.2a) and GRU5 (Figure 4.2b). We observe that GRU5 highly focuses on nouns (85.89% ± 0.38 of the peaks) and barely focuses on other POS. We also notice that the proportion of highlighted nouns is very different from what randomness would predict (47.15% ± 0.09), showing the network has learnt to detect and focus specifically on nouns. The behaviour of GRU1 appears different and seems much closer to a random behaviour: 52.37% ± 29.46 peaks are located above nouns when randomness would predict 46.92% ± 0.15. However, this result is due to an outlier run (hence the large standard deviation) where attention was concentrated at the end of the captions. Once this outlier For STAIR (Figure 4.3), we also notice that there is a large difference between the random peaks and the true attention peaks, showing that for Japanese also, the attention mechanisms highlight different units than what chance would predict. Both attention mechanisms have learnt to focus on nouns: 71.33% ± 5.94 for GRU1 (Figure 4.3a) and 63.30% ± 2.76 for GRU5 (Figure 4.3b) when randomness would predict 45%. We notice a slight asymmetry in the highlighted POS between both attention mechanisms: while GRU5 focuses on particles 28.63% ± 2.15 (where randomness would predict 15.88% ± 0.07), GRU1 highlights such POS less than what randomness would predict and only focuses on nouns instead. In Japanese, particules are clitics that may follow a noun, a verb, an adjective, or even a sentence and that cover a wide range of functions: some have clear grammatical functions and act as case markers such as "は" /wa/ (topic marker), "が" /ga/ (subject marker), or "を" /o/ (object marker) while others are only interjections such as "ね" (tag question, used to express surprise).

The study of which POS are highlighted by the attention mechanisms of the English and Japanese models reveals that the models have adopted a specific behaviour according to the language of the captions. In English, nouns are the most highlighted POS as they refer to objects present in the image. It should be noted that verbs are not completely ignored as there are some cases in which they are the only informative words of the captions. (e.g. "a woman skiing down a track beside some trees" for which the action of "skiing" is the main information). In Japanese however, the models took advantage of how the language works by not only highlighting nouns, but by adopting a language-specific behaviour when highlighting particles.

Highlighted Words

In this section, we take a closer look at which words are specifically highlighted by the best model of each language. To do so, we compute the proportion of peaks that are above a given word-form. This gives us a finer understanding of which specific words are the most highlighted by each attention mechanism for each language. The 40 most highlighted words for both COCO and STAIR are shown in Appendix B.

For COCO (Tables B.1 and B.2), we notice a slight difference in the words highlighted by the two attention mechanisms. For GRU5, all of the top 10 highlighted words are nouns referring to concrete objects in the pictures (train, tennis, toilet, baseball, etc.). For GRU1, while there are a few nouns (table, train, baseball), the network also highlighted prepositions such as "in" and "of". We believe the network did so because the words preceding preposition are nouns that refer to the main object of the image (e.g. a stop sign in a town). Surprisingly, GRU1's attention mechanism also highlighted the determiner "a" even though such word (or the preceding word) does not convey much information on the content of the image.

When we take a closer look at which words are highlighted by the best STAIR model (Tables B.3 and B.4), we notice that out of the top 10 highlighted words, respectively 3 and 4 words for GRU1 and GRU5 are particles. For GRU1 the most highlighted particles are "ga" (subject marker), "no" (genitive marker) and "o" (object marker). For GRU5, the most highlighted particles are "ga", "no", "o" and "ni" (locative marker). Even though GRU1 highlights less particles overall, the 3 most highlighted words remain particles. For GRU5, we notice that the network has learnt to highlight what seems to be the most useful particle overall which is the "ga" particle which is used to signal that the preceding word is the subject of the sentence. Focusing on this particle is especially useful as there is a strong probability that the preceding word refers to the main object of the image. For GRU5, 13.18% of the attention peaks are located above this "ga" particle, when randomness would predict 3.45% of peaks on this word. This shows that the network has learnt to deliberately focus on this particle. Otherwise, as for the models trained on the COCO data set, the other words among the top 10 words refer to nouns which are concrete objects widely represented in the corpus: skateboards, dogs, cats, etc.

The strategy of the network to highlight particles is very interesting as, by design, because of the uni-directional nature of the recurrent units used; and because particles are suffixed words, the vectors that constitute a particle contain a lot of information concerning the previous word. Consequently, highlighting particles is the optimal strategy.

Peak Position

In this section, we analyse specifically where the attention peaks are located above words.

To do so, we divided each word beneath a peak into four equal parts and we count the percentage of peaks located above a given part. This allows us to analyse if attention waits until the end of a word to peak (and thus attention peaks could be considered as word boundary markers) or if attention tends to peak before the end of a word. Results are shown in Table 4. 3.

For the models trained on English, we observe that the attention peaks are not located precisely at the end of a word, but are rather located between the middle and the end of the highlighted words. This seems to indicate that the network does not focus on the representation of the entire word, but rather on the vectors representing the first half or two thirds of a word. We also observe that GRU1 and GRU5 have a different pattern: while GRU5's peaks clearly highlight word endings, attention peaks for GRU1 seem to highlight in a higher proportion the middle of words.

For Japanese, we observe a similar behaviour as both GRU1 and GRU5's peaks are globally located above word endings. However, the situation is less as clear-cut as for English as we notice that also a large part of the peaks are located above other word parts and peak distribution tends to be more uniform. We explain this by the fact that attention peaks over Japanese captions are located above particles. We suspect that particles are able to trigger the attention mechanism as usually attention peaks are located at the very beginning of the particle or at the boundary with the preceding word (see both peak of Figure 4.1b located at the beginning of the "ni" and "ga" particles). Thus, the distribution of the attention peaks above a given part of word tends to be more uniform. The fact that attention mechanisms peak before the end of a word seems to show that the models do not need to have access to the full version of the word in order to recognise it, but rather peak as soon as the minimum information necessary to recognise it is present. This matter is explored further in Chapter 5.

Longitudinal Study

In the previous sections, we showed that the models trained on the COCO and the STAIR data sets used attention to highlight specific words in the captions. We showed that such behaviour was learnt as the highlighted units differ widly from what would be expected if attention peaks were randomly positionned. This begs however the question of when and how quickly this behaviour is acquired. To explore this question, we regularly saved the Results obtained with randomly initialised weights correspond to the first tick of both x-axes labelled "0". models during training so as to have a longitudinal vision of how learning takes place. To do so, we saved the models' weights every time the computed loss decreased by 4% of its initial value. We also save the model at each epoch.

We will here focus on the learning pattern of the best model (out of the five trained models), 3 and more specifically we will analyse the learning pattern of GRU5 as we have shown that, for both COCO and STAIR, this is the attention mechanism that has the most interesting pattern.

We first notice that for both COCO (4.4a) and STAIR (4.4b) the savings steps belonging to the first epoch represent a large part of the graph, showing the loss dropped quite fast and that the most important learning phase is actually concentrated in the first epoch. This impression is confirmed when we observe the evolution of the mean rank r (reported in Appendix D) which drops from 2504 to 42 at the end of the first epoch for COCO and from 2529 to 44 for STAIR, while the minimum r of 27 is obtained 7 epochs later for COCO and the minimum r of 29 for STAIR is obtained 6 epochs later.

For COCO, we notice a clear gap between the proportion of peaks highlighting nouns when the model uses randomly initialised weights (first tick on the x-axis) and the next saving steps where peaks above nouns have gone from 37.4% up to 55.9% after only 8 batches (256 examples). After 30 batches (960 examples), the proportion of nouns under attention peaks already reaches 64.2%. This shows that it is possible for the network to focus on very specific parts of the spoken input with very a few number examples. At the end of the first epoch, the proportion of peaks above nouns does not evolve much and remains steady, hovering at 85%. We may also notice that, while there are few peaks above other POS during the first epoch, they are then barely highlighted past the first epoch. Interestingly, it seems the network uses verbs at the beginning (as the proportion slightly increases in the first part of the first epoch) and stop highlighting such units later on (-10pp when we compare the proportion at the end of the second epoch with the middle of the first epoch).

For STAIR, the behaviour of attention displays a very different pattern compared to COCO and seems to be much more exploratory. Indeed, we notice that in the first steps of the first epoch, there is a competition between POS, whereby attention highlights simultaneously nouns, verbs, and particles. In the first steps (up to 4.9k examples), the proportion of peaks highlighting particles decreases while the proportion of verbs increases. Then, this behaviour stops, and the number of peaks highlighting particles surpasses that of verbs. We notice such behaviour on 4 out of the 5 models trained on Japanese. Therefore, it seems that in order to properly converge, the models first need to highlight verbs, and then switch to highlighting particles. Also, contrary to COCO where attention does not evolve much after the first epoch, we observe the contrary for STAIR. Nevertheless, we also notice that after the first epoch, the proportion of peaks above nouns decreases as much as the proportion of peaks above particles increases. It shows the model is still adapting its decision strategy after the first epoch and favours particles over nouns.

Experiments on Natural Speech: Flickr8k

In the previous sections, we presented the results we obtained on two data sets that consist of synthetically spoken captions in English and in Japanese. We justified the use of synthetically spoken captions as no bilingual data set is currently openly available. 4 This enabled us to put forward that neural models adapt their learning strategy according to the language used. However, using synthetic speech introduces a bias as it is clearer and simpler than human speech. We thus wanted to test the network's behaviour with real human speech to see if we obtained consistent results.

Experimental Settings and Results

We trained the model on the Flickr8k data set which is much smaller than the COCO and the STAIR data sets (see 3.2.2). This data set consists of real human speech, which features multiple speakers. The quality of the recordings is very uneven (some have a very high background noise), making this data set really challenging. We thus expect the results to be very lower than those previously presented. However, as previously shown, only a few examples are necessary for the network to highlight nouns. Consequently, if the network has a similar behaviour on natural speech, we should also be able to observe this fact.

The network's architecture used for this experiment is the same as in the previous experiments. However, we used 1024 recurrent units instead of 512 as the acoustic vectors are larger (13 MFCC coefficients, delta, and delta-deltas). This also leaves enough space for the network to take into account and encode intra-and inter-speaker variation.

Model

R@1 R@5 R@10 r Our Models As previously observed for the COCO data set, our results are lower than those reported by Chrupa la et al. (2017a). The results we obtain are also worse than those obtained on the COCO and STAIR data sets. This demonstrates how difficult it is for the network to model the variation found in real speech. Nevertheless, even though the results are not particularly good, they are still far better than chance, showing the network was still able to make sense of the data and has learnt a reliable speech-to-image mapping.

Random Attention

As with the previous experiments, we shuffled the attention weights in order to understand if they were meaningful or not. The results are presented in Table 4.5.

Here also, we notice that randomly shuffling the attention weights of both attention mechanisms yields a R@1 which is much lower than the original R@1 (-1.54pp). Once again, this shows that attention plays a vital role for the models. However, contrary to the previous experiments, we notice that while randomly shuffling the attention weights of the first attention mechanism really hurts the performance of the model, shuffling the attention weights of the fifth attention mechanism results in a very small loss of the R@1. This behaviour was observed across all 5 runs. We observed the opposite pattern for COCO and STAIR.

set [START_REF] Ohishi | Trilingual Semantic Embeddings of Visually Grounded Speech with Self-Attention Mechanisms[END_REF]. However, these data sets are not publicly available, and it was thus impossible to use them.

This tends to show that the majority of the models' predictions rely on the context vector computed by the first attention mechanism rather than that of the fifth. As for COCO, we notice that the models have learnt to focus primarily on nouns. Both attention mechanisms highlight more nouns than what randomness would predict (which would be 37%) also demonstrating the network deliberately focuses on nouns rather than any other POS. However, in this case and contrary to COCO and STAIR, GRU1 highlighted more nouns overall (72.58% ± 4.94) than GRU5 (58.73% ± 5.30). The difference in the R@1 we observed when shuffling the attention weights of GRU1 and GRU5 tends to show that highlighting nouns is not sufficient, otherwise we should have observed much of a gap in R@1. What thus seems most important is to highlight the key nouns in the captions.

When looking more closely at which specific words are highlighted by each attention mechanism of the best model (shown in Appendix B), we notice that the top 10 words highlighted by GRU1 are only nouns referring to objects present in the images (dog, man, girl, boy, dogs, people, woman, child, ball, water). These words are much more highlighted than what randomness woud predict: for example, 13.49% of GRU1's peaks highlight the word "dog" when only 1.53% random peaks highlight the same word. This shows the model has learnt to detect and highlight specific acoustic patterns corresponding to what most likely is the main object of the image. GRU5 top 10 words also contain nouns, but also contain silences, prepositions and determiners (</s>, water, <sil>, beach, snow, grass, shirt, street, a, in). Hence, this attention mechanism seems less specialised than the first one. 5.29% of GRU5's peak highlight the final silence of the caption, showing this attention mechanism favours a representation that accounts for the whole sentence rather than specific parts of the input signal.

Peak Position and Longitudinal Study

In this experiment also, we analysed above which word parts the attention peaks are specifically located. We observe that peaks also tend to favour word endings. As for COCO and STAIR, peaks are not located at the very end of the highlighted words, but seem to be more concentrated in the middle as the highest proportions of peaks are to be found in the middle-beginning and middle-end parts of words. Once again, these results tend to show that the model does not need to have access to the full word in order to recognise it and properly highlight it. We however notice for GRU5 that the peaks seem to be more evenly distributed over word parts, which confirms our previous intuition that GRU5 is an attention mechanism that highlights less specific units when dealing with natural speech. 4.6 shows how the proportion of highlighted POS evolves over time. We will focus on GRU1 as it is the most interpretable attention mechanism of the model. We notice also the same phenomenon as what was observed for GRU5 for COCO and STAIR: we observe a large gap between the proportion of peaks highlighting nouns when the model uses randomly initialised weights (first tick on the x-axis) and the next saving steps, going from 40% up to 60% two saving steps later (amounting to 2,912 seen examples). The proportion of peaks highlighting other POS varies slightly in the first few steps and quickly decreases and stabilises to its final value. This shows that the model identified nouns as being useful to find the matching image. The proportion of peaks above nouns increases steadily during the first epoch. Yet, contrary to COCO and STAIR where the proportion did not evolve much after the first epoch, we notice here that the proportion of peaks above nouns still evolves after the first epoch. Indeed, at the end of the first epoch, the proportion of peaks above nouns is 72%, while it is 77% at the last epoch. This confirms that the model needs more time in order to precisely identify all the important words in the caption. In our opinion, this is to be explained by the fact that Flickr8k consists of real human speech. Thus, the model needs more time in order to account for intra-and inter-speaker variation.

Relationship with Language Acquisition

One question we are drawn to ask at this point is the following: are there any commonalities between what we know of language acquisition in children and the learning patterns we put forward in the VGS models we studied?

The fact that our models preferably put forward nouns over all other POS seems coherent to what is observed in the language acquisition literature. This phenomenon is commonly referred to as the "noun bias". Indeed, it has been found that children's lexicon -both in perception and production -contains a higher proportion of nouns than verbs or any other POS. [START_REF] Gentner | Why nouns are learned before verbs: Linguistic relativity versus natural partitioning[END_REF] was the first to postulate that nouns are easier to learn than verbs. She explains this by the fact that "words that refer to concepts are easy to learn because the child has already formed object concepts, and need only match words and concepts". Our neural network is in such case as it uses pre-trained VGG vectors that encode which objects are present or not in the image. Also, she explains that "noun bias" is also to be explained by the fact that nouns represent objects that are "more salient, or more stable" as opposed to verbs, whose main function is to describe action and therefore whose referent is more evanescent. [START_REF] Mcdonough | An image is worth a thousand words: why nouns tend to dominate verbs in early word learning[END_REF] use the notion of "imageability" to explain this behaviour, by which nouns or verbs for which it is easier to recall a mental image are learnt before other words. Even though a few studies have concluded to the absence of a "noun bias", recent studies [START_REF] Bornstein | Cross-Linguistic Analysis of Vocabulary in Young Children: Spanish, Dutch, French, Hebrew, Italian, Korean, and American English[END_REF]) have shown that [START_REF] Gentner | Why nouns are learned before verbs: Linguistic relativity versus natural partitioning[END_REF] conclusions hold true for a wide range of languages. Nevertheless, we should not lose sight of the fact that the task our models solve is not the same as that of a child discovering the world and that our data is biased. First, our data set consists of static images and not video clips. Second, our data sets consist of images where there are more pictures depicting static scenes than dynamic scenes, thus prompting the annotators to use more nouns than verbs. Given these biases, it seems only natural that the network highlighted more nouns than verbs. We suppose that VGS models that operated on video clips, such as that recently proposed by [START_REF] Rouditchenko | AVLnet: Learning Audio-Visual Language Representations from Instructional Videos[END_REF] should highlight a higher proportion of verbs.

We showed that our Japanese model develops a language-specific behaviour when it mainly focuses on "ga" particles. [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF] observed that Japanese children (from 15 months on) also make use of "ga" to segment speech. Our models thus adopted the same strategy as Japanese children to segment the adjacent noun. [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF] state that "it is clear that noun particles are not the earliest cue infants use for word segmentation". We do observe this type of pattern for our models where particles are barely used in the first learning steps, but become more and more highlighted, while nouns are less prevalent than at the beginning.

Finally, is there any evidence that children pay more attention to word endings, such as what we observed with attention peaks which tend to highlight last half of the words? Such a proposal was made by Slobin (Ferguson & Slobin 1973, p. 191) in his list of Operating Principles, where he states that children "Pay attention to the ends of words". He proposed this operating principle as he noticed that post-verbal and post-nominal grammatical markers (and more generally every grammatical inflexion that is suffixed) are acquired by children before other prefixed grammatical realisations. Clark (Slobin 1985, Chapter 7, p. 761) notes that this principle "pertains to children who are trying to understand the language being spoken around them" and does not apply to production. This principle seems to also be used by our models, as they highlight word endings more than any other word parts. However, this behaviour is more clearly illustrated by our Japanese model where 42.6% of GRU5' peak are located at the end of the word. This is to be explained by the fact that our models favour particles which are postposed.

Chapter Summary

In this chapter, we trained VGS models on two languages (English and Japanese) and we analysed the behaviour of their attention mechanisms. Our analysis revealed that attention adopted a language general behaviour by which it learns to detect and highlight specific nouns in the spoken input. Our experiments thus confirm the intuition of Chrupa la et al. (2017a) stating that "the speech model's attention mechanism enables it to cherry pick key fragments of [...] utterances". But we also showed that attention could also adopt a specific behaviour based on the language the network is processing, such as what we observed for Japanese where attention mainly heavily relies on particles. Our analyses also revealed that attention quickly learns to focus on nouns and that such behaviour does not require much training data. With less than a thousand examples, attention already focuses on nouns more than what randomness would predict. Also, we observed that contrary to what we expected, attention does not peak at word boundaries but rather peaks between the middle and the end of words. This shows that attention does not need to have access to the full word. This however begs the question of whether the implicit segmentation carried out by the network segments word into units that correspond to written chunks, or rather segments words into shorter units. We explore this question in the following chapter.

The fact we do not observe much difference between the two attention mechanisms (be it between the highlighted POS, peak position, etc.) might only be a consequence of the way both attented vectors are merged. Indeed, we used dot-product to merge both vectors which equivalates an un-normalised cosine similarity. Indeed, recall that cosine similarity is u• i u i , which normalises the dot-product by the norm of both vectors u i . This surely explains why the difference in attention weights and peaks is minimal.

Finally, we tried to show the relationships that existed between what is known of language acquisition in children and the behaviour of attention in a neural network and have shown that some behaviours seem to be similar, the main one being that nouns are the most favoured POS categories.

Introduction

In the previous chapter, we showed that VGS models were able to highlight specific words in the speech stream using their attention mechanisms. This implies that the model is able to recognise the words which are being highlighted. This aptitude raises a few questions which we will try to answer in the present chapter.

First, we explore if the model is able to map an individual word to its visual referent. That is, instead of presenting full captions to the network we present individual words. If the network is able to retrieve images that feature the object the spoken word refers to, it means the network was able to segment that word from the speech stream and thus proceeded to an implict segmentation.

Second, we explore if all the words that correspond to the main objects the COCO data set was conceived around are equally well recognised by the network or not. We explore a few factors that could influence speech/image mapping in the network.

Third, we examine how words are recognised by the network. The fact that specific words can be highlighted implies that the model was able to store a certain representation of the most important words of the data set and is able to activate these representations when processing the spoken caption. To explore this question, we introduce a methodology originating from psycholinguistics -the gating paradigm [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF]) -and use it to analyse the representation learned by our model. We put forward in the previous chapter that the attention mechanisms need not see full words in order to highlight them, but rather peaked before the end of the words. This seems to imply that the network is able to recognise a specific word from a partial input. This is what we will study, namely how much of a word the network needs to see in order to be able to recognise it and map it to its visual referent.

Finally, we examine if there is a form of lexical competition when the network activates words that start with the same sequence of phonemes, such as it was found in human speech processing.

Word Recognition

The fact that VGS models are able to recognise individual words was already explored by several studies. For instance, Chrupa la et al. (2017a) and more recenlty Merkx et al. (2019) showed that the utterance embeddings computed by RNN-based VGS models contain information about the presence of individual words. To do so, they simply encode a full caption using the speech encoder of their network, encode an individual word, and train a probing classifier to predict whether the individual word was present or not in the full caption.

However, these studies did not show for what type of words this behaviour holds true and if the model had learnt to map these individual words to their visual referents. Also, none of these studies explored the factors that influence word recognition and why some words seem to be fairly well recognised while other are not recognised at all. It should be noted that [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] and Harwath, Recasens, Surís, Chuang, Torralba & Glass (2018) already observed that CNN-based models can reliably map word-like units to their corresponding visual reference. Therefore, we expect RNN-based VGS model to display a similar behaviour.

Isolated Word Mapping

To explore if the model is able to map isolated words to their visual referent, we selected a set of 80 words corresponding to the name of 80 object categories in the MSCOCO data set. 1 We expect our model to be very efficient with these specific words, as these refer to the main objects featured in the COCO data set.

We generated speech signals for these 80 isolated words using Google's TTS system and then extracted MFCC features for each of the generated words. In this experiment, we evaluate the ability of the model to rank the images so that at least one image in the first top 10 images contains an object instance corresponding to the target word (Precision@10, abbreviated P@10). 2 Contrary to Chrupa la et al. (2017a) who uses Recall@k, we use 1 List available at https://github.com/amikelive/coco-labels/blob/master/coco-labels-2014_2017. txt

2 Evaluation is performed on the test set containing 5000 images.

Precision@k as there are several images that may be correctly associated to a single target word: e.g. if the network is prompted with the word "elephant", every image that features an elephant should be considered a valid association. An important point to remember is that at training time, the network was only given full captions and not isolated words. Hence, if the network is able to retrieve images featuring instances of the target word, it shows that implicit segmentation was carried out at training time.

Results are shown in Figure 5.1. 40 words out of the 80 target words have a P@10≥ 0.8. This shows that the network is able to reliably map isolated words to their visual referent despite never having seen them in isolation. Conversely, we also notice that 15 words are not mapped to their visual referents. A list of the 80 words used for this experiment along with P@10 for each of them is available in Appendix D. Among the words that are the best recognised are animals (elephant, zebra, sheep, giraffe) and objects (truck, bus, boat, airplane) that are very frequent in our data set. Among the words that are the least well recognised are objects (fork, knife, vase, toaster) or animals (mouse) that are quite infrequently mentionned in the captions as they might be too small and not described by the annotators. However, among the least well recognised words are also common objects (such as skis or frisbee) that are very frequent in the captions. Manual exploration of the attention weights shows the model also peaks on such words (see Appendix B where "skis" and "frisbee" are respectively the 28 th and 33 rd most highlighted words). We explain this absence of recognition here by a competition effect whereby the network favours the representation of other words. For example, "skis", "snow", "hill" and "snowboard" frequently cooccur. The network might thus favour the other words in order to learn a reliable image mapping instead of our target word.

Factors Influencing Word Mapping

We explore here in more details the factors that could come at play in the recognition of isolated words. We explore 2 types of factors: image-related factors and speech-related factors. For the latter, we consider word frequency in the training set and length of the speech signal. Concerning image related factors we consider object instance frequency in the images of the training set, average number of neighbouring object instances, and average area of each object. We decided to consider these variables as failures to map a spoken word to its visual referent could only be due to the fact that the pre-trained VGG network, from which the image features are extracted, fails to detect and thus encode the presence of a given object.

To model the relationship between all these variables, we fit a multiple linear regression model with R where we try to predict Precision@10 using the previously mentioned factors.

Results are shown in Table 5.1. We notice that the only effect that plays a significant role in word recognition is the frequency of the word in the training set. The positive effect shows that the more frequent the word is in the training set, the better it is recognised. Word length has a mild positive effect which tends to show that longer words are better recognised than shorter words. Object frequency and number of neighbouring object have no effect. Object size seems to play a mild effect also. This seems to be coherent with the fact that more frequent words are better recognised, as we expect objects that occupy a large place in picture to be more often described than smaller objects. Consequently, such results seem to confirm our previously exposed hypothesis stating that words denoting small objects are often not described. Our results thus show that individual words are indeed reliably mapped to their visual referent by the network. The main factor of success in this task being the frequency of the target words in the caption as well as the size of the objects in the image. Hence, words that are very frequent and refer to large objects are better recognised than others.

Factors

Word Activation

In this section we describe how individual words are activated by the network. To do so, we perform an ablation experiment (similar to that of [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] which was conducted on humans) where the neural model is inputted with only a truncated version of the 80 target words. Such a method is also called gating in the psycholinguistic literature.

Gating Paradigm

The gating paradigm was introduced by [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] and involves the following procedure:

The gating paradigm involves the repeated presentation of a spoken stimulus (in this case, a word) such that its duration from onset is increased with each successive presentation. This is done until the entire word has been presented. After each presentation (or gate), subjects are asked to write down the word being presented and to rate their confidence in each guess. [START_REF] Cotton | The gating paradigm: A comparison of successive and individual presentation formats[END_REF] In our case, it means the neural model is fed with truncated versions of a given target word, each truncated version comprising a larger part of the target word. Contrary to [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] who only proceeds to a truncation that preserves the beginning of a word, we also proceed to a truncation that only preverses the end of a word. That is, in our case, truncation is either done left to right (the model only has access to the end of the word) or right to left (the model only has access to the beginning of the word). Also, contrary to the original setting where experimenters are asked to write the target word after each successive presentation, we evaluate the models' ability to rank the images so that the top k images contain instances of the target object. 3 The cohort model, in its initial version (Marslen-Wilson 1987b), stipulates that word onset plays a crucial role in word recognition. The aim of this experiment is to test whether word onset plays a role in word recognition for the network or not. If it is the case, we expect the network to fail to recover images of the target word if the word is truncated left to right, but not -or less -when the word is truncated right to left, hence motivating the truncation from both ends.

Truncation is operated on the MFCC vectors computed for each individual word, meaning that MFCC vectors are iteratively removed either from the beginning of the word or the end of the word, but not from both sides at the same time. Each truncated version of the word is then fed to the speech encoder which outputs an embedding vector. As in our previous experiment, the model's performance is evaluated in terms of P@10. Figure 5.2b shows the evolution of P@10 for one of the target words ("zebra"). When MFCC vectors corresponding to the first phoneme are removed (/z/), precision plummets from 1 to 0. However, when MFCC vectors belonging to the end of the word are removed, precision plateaus at 1 until /i:/ is reached and then plunges to 0. This shows the model successfully retrieved pictures of zebras when only prompted with /zi:b/ but not when prompted with /i:bK2/ even though the latter comprises a longer part of the target word. It also seems that little acoustic differences may yield very different representation as depicted by the red trough, were the only difference with the previous two points being the addition or deletion of an MFCC vector.

Effect of Gating

Figure 5.2a shows evolution of P@10 averaged over the 80 test words. As can be seen from the graph, precision evolves differently according to which part of the word was truncated. When the target words are truncated left to right, precision drops quicker than when truncated right to left. These results show that the model is robust to truncation when it is carried out right to left but not when it is carried out left to right: when the initial phonemes of the words are removed, the network fails to retrieve the target image, but when only presented with the initial phonemes, the network is globally able to retrieve images of the target word. These results suggest that the model does not rely on a vague acoustic pattern to activate the semantic representation of a given concept, but needs to have access to the first phonemes in order to yield an appropriate representation.

Activated Pseudo-Words

Such ablation experiments also enable us to infer on what units the network relies on to make its predictions. Figure 5.3 for example allows us to see what are the pseudo-words that were internalised by the network for the word "tennis racket". When truncation is done left to right (blue curve), we notice that at the beginning precision is high (1.0), then reaches 0 when only /EnIsraekIt/ is left, but suddenly increases up to 1 when the only part left is /raekIt/. This suggests that the network mapped both "tennis racket" as a whole and "racket" as referring to the same object.

Figure 5.3b shows that removing the first part of the word "fire hydrant" has nearly no effect on P@10. Indeed, even if a large part of the word "fire" is removed, P@10 remains high. Yet, once the /Ç/ is reached, P@10 abruptly decreases. We also notice it is possible to remove a large part of the word "hydrant" up to the /d/ without hurting much the performances of the network. This tends to show that only a sub-part of the word is necessary for the network to produce the appropriate representation and not the word "fire hydrant" as a whole. It thus seems that the minimal units that are necessary to activate the word fire hydrant are /ÇhaI/. Therefore, we need to take caution when stating that the network has isolated words, as the words internalised by the network might not always match the human gold reference.

Finally, Figure 5.2b shows that when only prompted with /zi:b/ the network manages to find pictures of zebras. This suggests that only the first part of the word is necessary to activate the representation of "zebra". This might be due to the fact that the vocabulary the network has to deal with is very limited and thus the network is confident from that point on that the rest of the word will refer to a zebra. This is similar to what happens in French with the pseudo-word "coquelic" which activates the word "coquelicot" (poppy) as this is the only word in French which starts with "coquelic".

This experiment hence shows that the pseudo-words internalised by the network do not necessarily correspond to a graphic word. It also shows that only sub-parts of a given word are necessary for the network to activate the target representation.

Gradual or Abrupt Activation?

As previously mentioned, little acoustic differences yield wide differences in the final representation. Thus, in this section we analyse how representation is being constructed over time and explore if some MFCC vectors play a more important role than others in the activation of the final representation.

We progressively let the network see more and more of the MFCC vectors composing the word, iteratively feeding it with MFCC vectors starting from the beginning of the word until the network has had access to the full word. We then compute the cosine similarity between the embedding computed for each of the truncated version of the word and the embedding corresponding to the full word. The closer the cosine similarity is to 1, the more similar the two representations are. Thus, if each MFCC vector equally contributes to the final representation of the word, we expect cosine similarity to evolve linearly. However, if some MFCC vectors have a determining factor in the final representation, we expect cosine similarity to evolve in steps rather than linearly. To detect steps that could occur in the evolution of cosine similarity, we approximate its derivative by computing first order difference. High steps should thus translate into peaks (e.g. Figure 5.4b). We compute the evolution of cosine similarity for the 80 target words encoded with the best trained model (e.g. Figure 5.4c) and also consider a baseline evolution by encoding the 80 target words with an untrained model (e.g. Figure 5.4a). 4 To avoid micro-steps of yielding peaks and hence creating noise, we smooth cosine evolution curves with a gaussian filter. We consider peaks higher than 0.025 as translating a high step in the evolution of cosine similarity.

On average, they are 1.25 peaks per word for the trained model against 0.1 peak per word that are above our 0.025 threshold for our baseline condition (untrained model), showing that cosine evolution is linear in the latter, but not in the former. Consequently, in our baseline condition (untrained model), each MFCC vector equally contributes to the final representation, whereas in our trained model some MFCC vectors are more decisive for the final representation than others. Indeed, some MFCC vectors trigger a high step in the cosine evolution suggesting that the embedding suddenly gets closer to its final value. Figure 5.4c shows the evolution of the cosine similarity for the word "elephant". As it can be seen, cosine similarity does not tend linearly towards 1, but rather evolves in steps. Adding the MFCC vectors corresponding to the transition from /I/ to /f/ triggers a large difference in the embedding as the cosine similarity suddenly jumps to a higher value, showing it is getting closer to its final representation. After the vectors of /f/ are added, the evolution of cosine similarity is modest, suggesting the final part of the words plays a less important role than the initial part in the final representation of the word. .4a shows evolution of the cosine similarity between the embeddings produced for each truncated version of the target word and the embedding for the full word using a model with randomly initialised weights. Figure 5.4c shows the same measure with the embeddings produced by a trained model. Figure 5.4b shows peaks indicating the inflection points of curve 5.4a (green) and 5.4c (red). For our experiments, we only considered inflection point to be significant if the resulting peak was higher than 0.025 (blue line).

We conclude from this experiment that the network needs not see the full word in order to activate the representation that enables images featuring this very word to be retrieved. More specifically, we put forward that the network needs to have access to the first part of the target word in order to activate the correct representation and that some word parts are more crucial than other in order to compute the appropriate representation.

Word Competition

Some psycholinguistic models (see Section 1.4) assume that the first phoneme of a word activates all the words starting by the same phoneme. The word that the speaker wants to pronounce and gradually utters is called the "target" word. The words that are activated but which do not correspond to the target word are called "competitors". As the listener perceives more and more of the target word, some competitors are deactivated. This means that they are not considered as the potential word any more, as they do not match what is being perceived. 5For example, let us consider that a speaker has started to utter the following sentence "I have a ..." and the following lexicon that only consists of three words that start with a /b/: /beIbi/ (baby), /beIsm@nt/ (basement), and /beIsbO:l baet/ (baseball bat). The first sound of last word of the sentence all start with /b/. Thus, all words would be activated and at this point the sentence could as well be "I have a baby", "I have a basement" or "I have a baseball bat". Once /beIs/ is reached, "baby" would not be considered a competitor any more, and once /beIsb/ is reached the only word activated would be "baseball bat" as it is the only word whose beginning corresponds to the perceived sounds. The speaker would know from that point on that the sentence is "I have a baseball bat".

Methodology

We tested if the network displays such lexical competition patterns. To do so, we selected a set of 57 word pairs that could potentially compete between one another. We selected the word pairs according to the following criteria:

• Words should at least appear 500 times or more in the captions of the training set, so that the network would have been able to learn a mapping between these words and their referent;

• Words forming a pair should at least start with the same phoneme;6

• Words should not be synonyms and clearly refer to a different visual objects (thus excluding pairs such as "motorcycle" and "motorbike").

Out of these 53 word pairs, we assessed if each word of each word pair was reliably mapped to a visual referent. We considered it was the case if at least 5 of the first 50 ranked images contained an instance of this word. Finally, out of the initial 57 word pairs, only 12 remained. 7For each word pair, we selected one of the word which we consider as the target word, progressively let the network see more and more of the MFCC vectors composing this word. At each time step, the network produces an embedding, which we use to rank the images from the closest matching image to the least matching image.8 Then, for the 50 closest matching images, we check if at least one of the captions contains either the target word or the competitor. We have to use the captions to check if the concept is present in the images, as among the word pairs, most contain words that do not belong to the 80 object categories of MSCOCO. We thus have to rely on the caption to check whether the object is present in the image or not.

As the competitor and the target word start with the same phonemes, we expect the network to produce an embedding that activates both the competitor and the target word at the beginning and then, when the acoustic signal does not match the competitor any more, we expect the network to be able to activate only the target word. For each word pair, each word is alternatively used as the target word.

Results

We present the competition plots of three specific word pairs as they are the most representative of the patterns observed with the other word pairs. 5.4.2.1 Cat/Cattle: mild competition Figure 5.5 shows an example of competition plots between two words: "cat" and "cattle". In Figure 5.5a, the network is prompted with iteratively longer spans of the word "cat". Figure 5.5: Illustration of lexical competition between "cat" and "cattle". 5.5a shows competition plots when the word "cat" is used as target. 5.5b shows competition plots when the word "cattle" is used as target. Numbers in 1 st x-axis correspond to the number of MFCC frames of the target word; 2 nd x-axis corresponds to time-aligned phonemic transcription of the target word; y-axis shows number of images for which at least one caption (out of 5) contains the target or competitor word. Vertical colour bars are projection of phoneme boundaries of the target word. In order to have smoother curves, we used a gaussian blur with a standard deviation of 2.

We notice that this word only activates the representation of "cat" as out the 50 images, 49 contain a caption with the word "cat" and none of them with the word "cattle". Figure 5.5b shows what happens when we use the word "cattle" as target, where the network is prompted with iteratively longer spans of the word "cattle". We notice that at the beginning, the representation which is activated is that of "cat" as once again, more than 45 of the top 50 images have a caption that contains the word "cat". However, at the end of [ae] and the beginning of [R], we notice that the number of images with a cat strongly decreases and the number of images with cattle increases. Once [2] is reached, no more picture of cats are retrieved, showing this representation was totally deactivated. This example is interesting as it seems the network was able to notice fine acoustic variation such as [t]/[R] in order to recognise the target word.

Train/Truck: strong competition

Figure 5.6 shows the competition plots between "train" and "truck". Contrary to the previous example, we notice a clear competition between "train" and "truck" when "truck" is used as target. We observe that both representations are activated at the same time (after [r]). However, we notice that the magnitude of the activation is very different as more pictures of trains are retrieved than pictures of trucks. We also notice that even though the network activated the representation of "train" when prompted with "truck", there are less pictures of "train" than when the network is prompted with the word "train". We also observe that at first the activation is very similar (both blue curves start to increase at the beginning of [r]), but once [2] is reached, the curve is less steep in Figure 5.6b than in Figure 5.6a, showing the network notice it is not the canonical pronunciation of the word "train". It is only at the very end of the word that the representation of "train" deactivates and that the network retrieves pictures of trucks.

(a) (b)

Figure 5.6: Illustration of lexical competition between "train" and "truck". 5.6a shows competition plots when the word "train" is used as target. 5.6b shows competition plots when the word "truck" is used as target.

5.4.2.3

Frigde/Frisbee: no competition (a) (b)

Figure 5.7: Illustration of lexical competition between "fridge" and "frisbee". 5.7a shows competition plots when the word "fridge" is used as target. 5.7b shows competition plots when the word "frisbee" is used as target.

Figure 5.7 shows the competition (or lack thereof) between "fridge" and "frisbee". We observe that even though both words start with the same sequence of phonemes, there is no competition between them. "frigde" only activates pictures of fridges and none of "frisbee" and vice-versa. In both cases, the network needs to have access to large part of the word in order to retrieve the target word.

Is there any Competition?

(revised) cohort [START_REF] Marslen-Wilson | Processing interactions and lexical access during word recognition in continuous speech[END_REF], [START_REF] Marslen-Wilson | Functional parallelism in spoken word-recognition[END_REF] and trace McClelland & Elman (1986b) both state that competing words are all activated at the same time, that is when the first phoneme is perceived. Even though some examples conform to this statement -such as that presented in Figure 5.6b -it is not the case of all word pairs. In some cases, there is only little competition as the representation of both words is activated sequentially (as in Figure 5.5) and in some other cases, there is literally no competition between the two words (such as 5.7) despite both words starting in a similar way.9 Therefore, the behaviour of the network appears to be very unclear. It seems however that the network activates the word that is the most common. For example, there are many more pictures of trains than pictures where trucks constitute the main object of the image. Thus, the network seems to activate preferentially the representation of the object that is the most frequent in the images and captions.

Replication on Natural Speech

We wanted to replicate our initial experiments on natural speech using the Flickr8k data set. Nevertheless, we were not able to do so as this was done before we had the time to do it ourselves. Indeed, Scholten et al. ( 2020) followed the methodology we introduced in our article Havard et al. (2019c) and applied it to analyse word activation and recognition in the VGS model of Merkx et al. (2019). The results they obtain confirm the trends we presented in section 5.2: they also observe that not all words are equally well recognised when presented in isolation, and report a median P@10 of 0.4 (when we report a median P@10 of 0.8). Such difference is to be expected as natural speech is harder to model than synthetic speech. Contrary to us, where we observed that longer words are better recognised than shorter words, they observe the reverse pattern. However, they also found that frequency had a positive effect such as we did, that is, words that occur more frequently in the training set are those that are recognised better in isolation.

In their study, they also find that word activation operates through a process of competition. Indeed, they show that word recognition is harder for words that have a high number of similar-sounding words -that is, words which start with the same sequence of phonemes.

Their study also puts forward that word recognition is possible even with a partial input, such as we did (see the "cat" example in Figure 5.5a, where /k/ only is enough to activate the representation of the full word). Finally, their study also show that word recognition also occurs in steps (which they call "leaps") where P@10 abruptly jumps from 0 to 1.

Chapter Summary

In this chapter, we showed that a VGS model is able to map individual words to their visual referents despite having been trained on full captions. van Zon (1997, p. 8) notes that in the COHORT and TRACE models "segmentation is the result of recognition". We believe that indiviual word recognition shows that the model implicitly segmented its inputs into subunits, and propose the converse formulation "recognition is proof of segmentation". An important point we made in this chapter is however that the resulting segmentation might not always correspond to the gold standard.

In this chapter, we introduced several methodologies to analyse how the representation of individual words is built over time. Notably, we adapted the gating paradigm of [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] so as to analyse how words are activated by the network. We observed that word representation is not built linearly by the network and that recognition may occur with a partial input, thus corroborating that "recognition often occurs before the acoustic offset of the word" (van Zon 1997, p. 8). We showed that in order to be able to activate the correct representation of a given word, the network needs to have access to the first phonemes of this word, as when they are removed, the network is unable to activate the correct representation. Thus, when word recognition is observed with a partial input, it is only when the partial input encompasses the first part of the word, but not when it only encompasses the final part of the word.

Finally, we tried to see if word activation was done through a process of word competition, such as what was postulated in human word recognition. We observed mixed results, as we did observe competition in some cases, but this behaviour was far from systematic. Also, word competition models assume that competitor words are totally deactivated when the acoustic input does not match the internalised representation, but we found it was not the case for most of the word pairs we tested.

The work we presented in this chapter has some limitations. The first one being that we only used synthetic speech, however, this issue was addressed by [START_REF] Scholten | Learning to Recognise Words using Visually Grounded Speech[END_REF] and confirmed the results we present. One of the issue we faced was that we were limited to a given set of words. Indeed, we were limited by the data set we used and the most frequent words it contains. We chose words according to the 80 object types the data set was conceived around. Nonetheless, all of those words are not equally well recognised by the network, and therefore the activation pattern of some words was rather unclear.

Also, the analysis of the network's representation is uneasy as we predict a vector in a continuous space, and the only way of interpreting the network's behaviour is by looking at which images are retrieved. Studying if there is any form of competition in an ASR model would be much easier as the last layer consists in a softmax over the vocabulary.

Introduction

In the previous chapters, we showed that RNN-based VGS speech models use their attention mechanism to highlight words that are relevant to retrieve the correct image, and that such models implicitly segment the spoken input into sub-units. In this chapter, instead of understanding what types of units were implicitly segmented by the network, we approach the problem the other way round and ask ourselves the following question: what segmentation maximises the performance (recall@k) of a VGS model if speech were to be segmented?

In order to answer this question, we explore how boundary information can be integrated, which type of boundary is the most efficient (either phone, syllable, or word), and finally where -that is, at which layer -such boundary should be introduced in the network. Finally, we also explore hierarchical models for which we provide multiple segmentation levels at the same time to understand the effect of explicitly modelling the structure nature of speech.

Another motivation for this experiment stems from the linguistic litterature, which shows that litteracy (i.e. the ability to read) affects language acquisition, and more specifically lexical acquisition. [START_REF] Havron | Literate and preliterate children show different learning patterns in an artificial language learning task[END_REF] indeed showed in an artificial language learning task that preliterate children have difficulties associating a novel word to its referent while litterate children were easily able to do so. Several factors may explain this, the main one being that prelitterate infant struggle to segment the speech stream, while litterate infants are aided in this task by having seen visual cues (i.e. blanks) separating words.

Contrary to the previous chapters where we used the Theano-based model of Chrupa la et al. (2017a), we switch to a PyTorch-based model.1 Also, in this chapter, we only used one attention mechanism instead of two as in the previous chapters.

Boundary Information

Boundary Types

As previously stated, we wish to give linguistic information to our network, and more specifically with segment boundary information. In this chapter, we define a segment as either being a phone, a syllable, or a word. Segment boundaries were derived from the forced alignment metadata (see § 3.2.4) so as to know which MFCC vector constitutes a boundary or not. As the force aligner does not provide alignment at the syllable level, we wrote a custom script to recreate syllables from the phonemic transcription.

We consider two different types of syllables in this work: indeed, when we speak, words are not uttered one after the other in a disconnected fashion, but are rather blended together through a process called "resyllabification". In English, this phenomenon is visible when a word ending with a consonant is followed by a word starting with a vowel. In this case, the final consonant of the first word tends to be detached from it and attached to the next word, thus crossing the word boundary. This phenomenon is illustrated in Example (1) where phonemes in red indicate a resyllabification phenomenon.

(1)

This is an example. Transcription2 /DIs#Iz#@n#Igzaemp@l/ a. No resyllabification /DIs.Iz.@n.Ig.zaem.p@l/ b. With resyllabification /DI.sI.z@.nIg.zaem.p@l/

The two types of syllables that we consider in this work are the following: "syllable-word" that refers to syllables that result from a segmentation that does not take into account resyllabification (1-a), and "syllable-connected" that refers to syllables that result from a segmentation that takes into account resyllabification (1-b). It should be noted that in the syllable-connected condition, most word boundaries are lost. 3 In the syllable-word condition however all word boundaries are preserved and the segmentation inside a word may occasionally result in a morphemic segmentation (as for example in "runway" /ô2n.weI/ or "air.plane" /Eô.plein/). Yet, this is not always the case, especially for longer words that are of non-germanic origin (such as "elephant" /E.lE.fant/ or "computer" /k@m.pju.t@~/).

Thus, for each caption we have a sequence X of length T of d-dimensional acoustic vectors X = x

Integrating Boundary Information

In order to integrate boundary information in our network, we take advantage of its design, and more specifically of the recurrent cells and how such cells compute their output. Recurrent cells, as already mentioned in the first part of this thesis (see 2.2.3), are particularly suited to handle sequences -such as speech -where each acoustic vector cannot be independently considered, but rather depends on the preceding acoustic vectors. Recurrent neural networks can thus be formalised as follows:

h t = f (h t-1 , x t ; θ) (6.1)
where the hidden state at timestep t, noted h t is a function f of the previous timestep h t-1 and the current input vector x t , and where θ is a set of learnable parameter of the function f . Hence, the final vector computed at timestep T depends on all the previous vectors, thus effectively modelling the sequential nature of the input. A special case arises for the first timestep t = 1 as the previous hidden state h t-1 does not exist. In such case, the initial state h t-1 -noted h 0 and called the initial state -is set to be vector of zeros.

Our approach to integrate boundary information into the recurrent layers of our network can be formalised as follows:

h t = f (h 0 , x t ; θ) , if b t-1 = 1 f (h t-1 , x t ; θ) , otherwise (6.2)
In our approach, h t is only dependant on the previous timestep h t-1 if the previous timestep was not an acoustic vector corresponding to segment boundary (b t-1 = 0). If the previous timestep corresponds to a segment boundary (b t-1 = 1), we reset the hidden state so that it is equal to h 0 . Hence, vectors in the same segment are temporally dependent, but vectors belonging to two different segments are not. The GRUs that use this computing scheme will from now on be referred to as GRU PACK. , as vectors belonging to the same segment are "packed" together.

As for our previous experiments, we use GRUs, but our methodology could be applied to any other type of recurrent cells such as LSTM of vanilla RNN.

All and Keep Conditions

From this initial GRU PACK. setting, we derived to different conditions: all and keep. In the all condition (see Figure 6.1b), all the vectors belonging to a segment are forwarded to the next layer (which can either be a recurrent layer, or an attention mechanism depending in the keep condition where only the last vector of a segment is passed on to the next layer, thus resulting in an output sequence shorter than the input sequence. The red crosses inscribed in a square ( ) signal that the output vector computed at a given timestep is not passed on to the next timestep and that the initial state h 0 is passed on instead. The red crosses inscribed in a circle ( ) signal that the output vector computed at a given timestep is not passed on to the next layer. Dotted lines group vectors belonging to a same segment (either phone, syllable-connected, syllable-word, or word). Note that h 0 is only passed on to the next state at the end of a segment, thus effectively materialising a boundary by manually resetting the history. Also note that the x 1 , x 2 , ..., xt figured in this representation could either be the original input sequence (in our case, acoustic vectors) or could also be the output of the previous recurrent layer on the position of the GRU PACK. layer). In the keep condition, only the last vector of each segment is forwarded to the next layer (see Figure 6.1c). The length of the output and input sequence stays the same in the all condition. However, it should be noted that in the keep condition, the length of the output sequence is shorter than the input sequence. 4The difference between all and keep is motivated by the fact that we believe that keeping the last vector of a segment could constrain the network to learn more consistent representations for different occurrences of the same segment. Indeed, in the all condition, even though a vector at timestep t is different from its neighbours, we expect consecutive vectors to share a substantial amount of information as the acoustic vectors are extracted within a very short timeframe with an overlapping window (see 3.2.3, footnote 10). Thus, boundary information might be watered down by the fact that the preceding and following vector contain redundant information, and the network might not use this information effectively. However, in the keep condition, as only the last vector of a given segment is forwarded to the next layers, the network is forced to "cram" as much information as possible in a single vector so as to pack the information effectively. Also, as only the final vector is kept, it should be as informative as possible as the subsequent layers will have less information to rely on, making it difficult to reconstruct the missing data.

Our GRU PACK. -keep segmentation approach was inspired by prior work by [START_REF] Kreutzer | Learning to Segment Inputs for NMT Favors Character-Level Processing[END_REF] on dynamic segmentation for MT. However, we also found that a nearly identical approach was proposed by [START_REF] Chen | Audio Word2vec: Sequenceto-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation[END_REF] in an Audio-Word2Vec experiment. Instead of being given gold segment boundaries, a classifier outputs a probability that a given frame constitutes a segment boundary, and if so, the RNN's history is reset and only the last vector of a segment is forwarded to the next layers. Working with presegmented speech is also analogous to prior work done by [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] where the spectrogram was broken into words. Nonetheless, our work is different from theirs as we also explore other types of segments (not only words, but also phones and syllables) and also because we introduce this information at different layers of the architecture.

Experimental Settings

GRU PACK. Position

In order to understand where boundary information should be introduced (that is, at which level of the architecture), we train as many models as the number of recurrent layers, where each time one layer of GRUs is replaced with one GRU PACK. layer. For example, "GRU PACK. -3" refers to a model where the third layer of GRUs is a GRU PACK. layer and other layers (1 st ,2 nd ,4 th , and 5 th layer) are vanilla GRU layers. This setting will allow to explore where introducing boundary information is the most efficient.

Random Boundaries

In order to understand if introducing boundary information helps the network in its task, we compare the performance of the models using boundary information with a baseline model which does not use any (thus, all the recurrent layers of the baseline architecture are Vanilla GRU layers). This model will from now on be referred to as baseline. We also introduce another condition, where, instead of training models with real segment boundaries (which from now on will be referred to as true), we train models with random boundaries (which from now on will be referred to as random). Indeed, it could be that randomly slicing speech into sub-units leads to better results, even though the resulting units do not constitute any linguistically meaningful units. Therefore, training models with random boundaries will enable us to verify this claim. Random boundaries were generated by simply shuffling the position of the real boundaries (vector B introduced in §6.2.1), resulting in as many randomly positioned boundaries as they are real boundaries. Note that we do still expect the models to have reasonable results even when using random boundaries, as the acoustic vectors are kept untouched. However, we expect that placing random boundaries will hinder the network's learning process and thus yield results significantly lower than when using true boundaries. We expect the results to be significantly lower in the randomkeep condition as this condition is equivalent to randomly subsampling the input, and thus removing a lot of information.

Evaluation

Models are evaluated in term of Recall@k (R@k). Given a spoken query, R@k evaluates the models' ability to rank the target paired image in the top k images. In order to evaluate if the results observed in our different experimental conditions (true-all, truekeep, random-all, random-keep) are different from one another and from the baseline condition, we used a two-sided proportion Z-Test. This test is used to assess if there is a statistical difference between two independant proportions. As for each spoken query there is only one target image, R@k becomes a binary value which equals 1 if the target image is ranked in the top k images and 0 otherwise. In our case, the proportion that we test is the number of successes over the number of trials (which corresponds to the number of different caption/image pairs in the test set).

Results

Overall, our experimental settings led to the training of 81 different models per data set. 5baseline results are shown in Table 6.1. Results for the true/random conditions obtained Data set R@1 R@5 R@10 COCO 9.0 27.0 39.5 Flickr8k 4.3 13.4 21.4 on the COCO and Flickr8k data sets are shown in Table 6.2 and 6.3 respectively. We obtain lower results on Flickr8k than on COCO which shows how difficult the task is on natural speech. Note that the results obtained on synthetic speech are also very low compared to their textual counterpart.6

True and Random Boundaries

The first question our experiments aim at answering is whether introducing boundary information helps the network in solving its task or not. Overall, models trained on Flickr8k with true boundary information have a significantly better R@1 than their baseline counterparts and models trained with random boundaries (which are either on a par with the baseline, or worse). This indicates that the models did effectively use the provided boundary information.

For COCO, we observe that some models trained with random boundaries have significantly better scores than the baseline (particularly when using phone boundaries in the keep condition). We explain this by the fact that randomly subsampling the input signal might act as a form of regularisation for the network. This effect disappears when using larger units, such as words, for which the results are not significantly different from the baseline. This regularisation effect might only be due to the fact that COCO uses synthetic speech with only one voice and hence, has very low intra-speaker variation. Thus, even though we randomly subsample the input, as there is very little intra-speaker variation, the network is much more likely to figure out from which units the subsampled vector came from.

All and Keep

The results shown above should also be analysed in regards to the all and keep conditions. Indeed, there is an interaction effect between using true and random boundaries either in the all or keep condition.

This effect is particularly noticeable for Flickr8k. Indeed, in the random-all condition, no result is statistically better than the baseline, while in the random-keep condition, the results are statistically worse than the baseline. This clearly shows that keeping the last vector only of segment rather than all of the vectors has a real effect, and that this effect is particularly noticeable when using random boundaries. Hence, using random boundaries 6.2: Maximum R@1 (in %) for each model trained on the COCO data set. "T" stands for true (boundaries) and "R" stands for random (boundaries). "Syl-Co." and "Syl-Word" stand for "Syllable-Connected" and "Syllable-Word" respectively. Each line shows the results for when a specific recurrent layer is a GRU PACK. layer. The 1 st layer is the lowest layer (right after the 1D convolutions and acoustic vectors) and the 5 th the highest (right after the four preceding recurrent layers and before the attention mechanism). The highest R@1 in the table is shown in red. The best results between each true and random pair (columnwise) are shown in bold. ◌ + and ◌ -indicate that the results are statistically better (respectively worse) than the baseline. Results in italics show statistical significance (two-sided Z-Test, p-value < 1e -2 , see §6. 2.4.3) between each true and random pair (columnwise).

which do not delimit meaningful linguistic units really hurts the performance of the network. Furthermore, we notice that the results between all either in the random or true condition are not statistically different from one another while they are in the keep condition. This tells us that boundary information is not used effectively in the all condition, even when true boundaries are given. The only exception to this statement is when using word boundaries at the first layer in the all condition. It seems that boundaries are only effectively used here. This confirms our intuition that the keep condition effectively constrains the network to learn better representations, and that in the all condition boundary information is watered-down by the neighbouring vectors, thus leading to a suboptimal use of such information.

For COCO as well, we notice an asymmetry between the all and keep conditions. In the all condition, the results are rarely significant while there are significantly better than the baseline in the keep condition. Here also, we observe an interaction between the true-random and keep-all condition. In the all condition, the results between true and random are rarely significantly different from one another, while they are in the keep condition.

Phone, Syllable, or Word

In our experiments, we used four different types of segments corresponding two different types of linguistic units: phones, syllables-connected, syllables-word, and words. These different types of segments vary in length (words and syllables are longer than phones), quantity (there are more phones and syllables than words), and intrinsic linguistic information: phones only show which are the basic acoustic units of the language, while word segments represent meaningful units, and syllables-word and syllables-connected are a higher form of acoustic units that may contain morphemic information. Given the task the network is trained for (speech-image retrieval), we do not expect these different units to perform equally well. Indeed, as this task implies mapping an image vector describing which objects are present in a picture and a spoken description of an image, we expect word-like segments (or segments that preserve word boundaries and that bear a substantial amount of semantic information) to perform better. This is in fact what we observe in practice. Word units obtain statistically better results than the baseline for both Flickr8k and COCO (R@1 = 5.4, +1.1pp and R@1 = 11.3, +2.3pp respectively). Syllables-word also bring significant improvement (R@1 = 5.3 for Flickr8k and R@1 = 10.9 for COCO), however, slightly less than when using word units. It should be noted that syllables-connected segments obtain also statistically significant improvement over the baseline (GRU PACK. -2 for both data sets) despite not preserving all word boundaries. However, these results are slightly worse than the syllables-word and word segments suggesting that preserving word boundaries is a property that helps the network. Using syllable-connected segments yields results that are on par with phone units. It appears that the size of a segment is a very important parameter. Indeed, phone segments (naturally) preserve word boundaries but of course naturally lack the internal cohesion of a morpheme or a word as nothing links two adjacent phonemes together, while syllables-connected do not preserve word boundaries, but present a higher internal cohesion. Hence, it seems that segments that preserve both meaning and word boundaries (such as words) or from which meaning can be more easily recomposed (syllable-word) may facilitate the network's task. The fact that syllable-word segments perform as well as word segments might only be an artefact of using English where a high proportion of word is monosyllabic.7 Working on a language where the syllable-to-morpheme ratio is higher would be a future line of work that would enable to test this hypothesis.

GRU PACK. Layer Position

We introduced boundary information at different levels of our architecture in order to better understand at which layer it is the most useful to add such information. We will focus in this section on the results obtained in the keep condition, as the all condition brings little improvement over the baseline condition.

Our results clearly show that introducing boundary information at different layers has a substantial impact on the results: using such information at the first or the fifth layer is useless, as we notice it either yields similar results as the baseline (GRU PACK. -1) or worsens the results regardless of the type of boundary used (GRU PACK. -5). When using syllables-word segments the best results are obtained when introducing the information at GRU PACK. -3 (GRU PACK. -2 for COCO), and GRU PACK. -2 for word segments. This results are in line with that of Chrupa la et al. (2017a) who found that the representations learnt by the fifth layer is the less informative in predicting word presence, while lower layers encode this information better. This confirms that the middle layers of our architecture are better suited to deal with lexical units whereas the fifth layer encodes information that disregards that type of information.

All in all, for Flickr8k word-like segments seem to be the most robust representation to be used as they yield significantly better results at three different layers (GRU PACK. -2,3,4). For COCO, we observe mixed results where introducing boundary information at the first four layers improve our baseline results. However, the results tend to be higher when boundary information is introduced at the second layer, confirming this layer is the one that benifit the most from boundary information overall.

Segmentation as a means for compression

Recall that in the keep condition, only the last vector comprising a segment is kept while the other vectors are discarded. This can be interpreted as a form of "guided" subsampling, as usually subsampling does not take into consideration linguistic factors. To understand how much information is kept between the input and the output of a GRU PACK. layer in the keep condition, we compute an average compression rate (in %) for each of the segment types for Flickr8k. The results are the following: phones = 90.57%, syllablesconnected = 93.41%, syllables-word = 94.36%, and words = 94.90%. 8 When we re-analyse our results in light of this information, it appears we can remove a large part of the original input (up to 94.90% if using word segments) while conserving or increasing the original R@1. It is not simply the effect of subsampling that helps, but subsampling with meaningful linguistic units. The effect of informed subsampling is striking when we compare R@1 for random-keep, which are always below the baseline, while true-keep are on a par with the baseline or better. This effect is particularly visible for GRU PACK. -1 in the random condition where the more vectors are discarded (syllable-like and word segments), the worse the results are. This can only be explained by the fact that randomly subsambling removes important information that the network is unable to recover in the four subsequent layers. A counter-intuitive finding of our experiments is that it is better to subsample early on (in the first layers) and thus remove most of the information early on than later on. Subsampling with word segments in GRU PACK. -2 (and thus only keeping 5.1% of the original amount of information for the subsequent layers) yields better results than subsampling with the same resolution at GRU PACK. -5.

Hierarchical Information

Integrating Hierarchical Information

In the aforementioned experiments, we supply the network with only one type of boundary (either phone, syllable, or word) but not multiple at the same time, as if several units could not coexist at the same time. Yet, this does not correspond to how speech is really structured. Indeed, multiple spoken units exist at the same time, and they are structured hierarchically: words can be broken down into syllables, which can be in turn be broken down into phones.9 In order to model such hierarchical nature of speech, we can stack as many GRU PACK. as desired, where one layer handles one type of segment (e.g. phone) and the following GRU PACK. layer handles another type of segment, that is hierarchically above the preceding (e.g. syllable, or word).10 A graphical representation of a hierarchical GRU PACK. architecture is shown in Figure 6.2.

We expect such hierarchical architecture to perform even better than a single-layered GRU PACK. . Indeed, a two-layered architecture should constrain the network to learn even better representations than a single-layered architecture, especially for longer segments such as words. Words might be long units and capturing all their details in a single pass might be challenging. By breaking them first into sub-units, such as phones or syllables, we expect the network to learn more consistent representation and help the network better distinguish between words differing by only one phoneme. [START_REF] Harwath | Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech[END_REF] explored such hierarchical architecture using a CNN-based model that incorporated multiple vector quantisation layers and found that it improved the network ability to retrieve the target image. Our work thus attempts to verify if it is also the case for an RNN-based model, when provided with gold boundaries.

Experimental Settings

We explore the effect of using a hierarchical architecture on the Flickr8k data set only, as it features real human speech and might thus be more challenging, but also more realistic. Contrary to our previous experiments, we will only consider hierarchical architectures that use GRU PACK. -keep layers, as we have shown that GRU PACK. -all bring little improvement over vanilla GRUs.

We test various architectures that handle different type of boundaries simultaneously with two out of the five recurrent layers being GRU PACK. layers (6.3.3) or with three GRU PACK. layers (6.3.4). In both cases, we test all possible positions as well as boundary type to understand what the best combination is. layers. The 1 st GRU PACK. layer is supplied with phone boundary information. This layer outputs vectors representing phones that are recomposed from the input vectors. The next layer, GRU PACK. -2 takes these phone vectors as input and recombines them into words and outputs vector representing whole words. This architecture effectively models the hierarchical nature of speech, where words consist of a sequence of phones. Note that both GRU PACK. are in the keep condition where only the last vector of a given unit is kept. As in Figure 6.1, note that x 1 , x 2 , ..., xt need not be acoustic vectors, but could also be the output of the previous vector.

Two GRU PACK. Layers

In this section, we experiment an architecture with two GRU PACK. each handling different boundary types: phones and words (6.3.3.1), phones and syllables (6.3.3.1), and syllables and words (6.3.3.3). We vary the position of the GRU PACK. layers used in our architecture and test all the combinations of possible positions.

Phones and Words

When using phones and words together, the results (Table 6.4) are higher than the baseline architecture and than the single-layered GRU PACK. architecture. Indeed, we obtain a maximum R@1 of 8.2% when using GRU PACK. at the layer 2 and 3 which is +3.9pp over the baseline and +2.8pp over a single-layered architecture. We also notice that using a hiearchical architecture also allows us to train shallower networks while improving the results over our baseline architecture. For example, a two-layered architecture (where both layers are GRU PACK. ) has a R@1 of 6.4 which is +2.1pp over our 5-layered baseline architecture that does not use any boundary information.

Here also, we observe that placing a GRU PACK. layer at the last layer yields worse results than when the last layer is a vanilla GRU layer. Indeed, we observe that for all the architectures, when the last layer is a GRU PACK. layer, the results are substantially lower when placed one layer before. Once again, this confirms that the last layer of the network -which is just before the attention mechanism -handles semantic information and is not concerned with form any more. These results also indicate that the GRU PACK. layers are more effective when placed in the middle of the architecture, than when used at the beginning: both for a five-layered architecture or a four-layered architecture, the best results are obtained when the GRU PACK. are placed at layers 2 and 3, suggesting that speech requires a certain form of pre-processing in the lower layers in order to use boundary information more effectively. Table 6.4: R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers using phone and word segments (models were selected based on the maximum R@1 on the validation set). Best score overall is shown in red. Best score (layer-wise) is shown in bold. Greyed out cells signal impossible configurations. We also indicate R@1 obtained on a baseline architecture that does not use any GRU PACK. layers.

In our previous experiments, we observed that even random boundaries could yield statistically better results than the baseline and showed it could only be explained by a regularisation effect. It could be that stacking two GRU PACK. layers yields better results not because of the hierarchical nature of the architecture, but simply because of a double regularisation effect. To investigate if such effect arises or not, and in order to confirm our previous observations, we select our best performing architecture (5-layered architecture with GRU PACK. layers at layer 2 and 3, see Table 6.4) and retrain it using random boundaries. 11 We thus have the following training and testing settings:

• true phones + true words : the first GRU PACK. layer receives true phone boundaries and the second GRU PACK. layer receives true word boundaries. This is the ideal condition, and we expect models trained with such boundaries to have the best results (this setting corresponds to the R@1 = 8.2 in the previous table).

• true phones + random words : the first GRU PACK. layer receives true phone boundaries while the second GRU PACK. layer receives random word boundaries, randomly sampled from the phone boundaries.

• random phones + true words : the first GRU PACK. layer receives random phone boundaries while the second GRU PACK. layer receives true word boundaries. Because the second layer requires true word boundaries, we also need to keep some boundaries intact in the phone boundaries provided to the first GRU PACK. layer. Thus, the random boundaries provided to the first layer are not entirely random, and they are as many true phone boundaries as the number of words. The true phone boundaries we keep correspond to the last phoneme of a word.

• random phones + random words : the first GRU PACK. receives random phone boundaries and the second GRU PACK. receives random word boundaries, randomly sampled from the phone boundaries. This is the worst condition, and we expect models trained with such boundaries to have the worst results.

The results we obtained using random phones and/or random word boundaries are shown in Table 6.5. We notice that all the results are lower when using random phones and/or random words than when using true phones and true words together. This thus shows that the better results obtained with a hierarchical architecture are not to be explained by a double regularisation effect, as if it had been the case, we should have observed similar results with random boundaries. Hence, we conclude that using a hierarchical structure is indeed beneficial because it accounts for the hierarchical nature of the spoken units and helps the network build better (more consistent) representations.

Phones and Syllables

The results we obtain when using phones and syllables is shown in Table 6.6. Here also, we notice that the results are better than the baseline results (without any GRU PACK. ) and also better than when using only one GRU PACK. : R@1 = 7.9, +2.5pp. Contrary to the previous experiment, the 4-layered architecture converges better than the 5-layered architecture. Nonetheless, the results are lower than when using jointly phones and word (-0.3pp) indicating that using jointly phones and syllables is not the ideal combination.

Once again, we do observe the same tendencies as we previously put forward: weaker results Table 6.6: R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers using phones and syllable-word (models were selected based on the maximum R@1 on the validation set). The same naming conventions of Table 6.4 are used for this table

when the last layer is a GRU PACK. layer; and the best results are obtained when GRU PACK. layers are placed in the middle layers. We however observe that the two-layered architecture performs worse than in the previous experimental setting. Indeed, in the previous setting, the result obtained with such an architecture was +2.1pp over our baseline result, while the gain here is negligible: only +0.3pp over the baseline. This shows that even though using word boundaries or syllable boundaries results in a higher compression rate (see 6.2.6), it does not compensate for the inherently lower semantic contribution of the provided segments.

Syllables and Words

The results we obtain when using syllables and words is shown in Table 6.7. Once again, we notice that R@1 is higher than a single-layered GRU PACK. architecture (R@1 = 7.6, +2.2pp) but worse than a two-layered architecture handling phones and words: -0.6pp.

The best result is also lower than the best result when using phones and syllables -0.3pp. However, we observe that the two-layered architecture performs better than the baseline (+1.0pp) and better than the two-layered architecture that uses jointly phones and syllables (+0.7pp). These results suggest two things: first, in order to use a very shallow architecture, segments that bear a lot of semantic information should be preserved (as we observe better results with such architectures when word segments are preserved). Second, keeping lowlevel segments such as phones is also necessary when training deeper models. The fact that we do not observe improvement when using jointly syllables and words with deeper architecture might show that the information brought by both levels overlap and might be redundant. Table 6.7: R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers using syllable-word and word segments (models were selected based on the maximum R@1 on the validation set). The same naming conventions of Table 6.4 are used for this table

Section Conclusion

Our experiments show that the best result using two GRU PACK. layers is obtained when using jointly phone and word boundaries, when the GRU PACK. layers are placed in the middle of the recurrent stack. We also observe that overall our results are maximal when the layers immediately follow one another. Finally, this set of experiments allowed us to show that the network converges best when low-level segments (phones) and high-level segments (words) are used jointly. We explain this by the fact that this allows the model to learn robust representations for the phone units, while having high level units that bear a lot of semantic information. Consequently, using intermediate segments such as syllables is not useful as they are not short enough to learn a consistent representation while being too short in regards to the amount of semantic information they bear.

Three GRU PACK. Layers: Phones, Syllables, and Words

Finally, we integrated three segment levels in a single model. As in our previous experiments, we experiment with a different number of layers (from 3 to 5), each time with 3 GRU PACK. layers at each possible position. The results of this experiment are presented in Table 6.8. We observe that the best result obtained with this architecture (R@1 = 9.6) is far better than the baseline (+5.3pp), better than the best result of a single-layered architecture (+4.2pp) but also better than the best result of a double-layered architecture (+1.4pp over the phone-word architecture). Our best result is obtained by a five-layered architecture with GRU PACK. in position 1, 3 and 4. However, we notice that the four-layered architecture obtains more consistent results across all layers, the maximum result being only -0.3pp away from best five-layered architecture. We also notice that the 3 layered architecture obtains a very high R@1 of 8.0 which is about two times over the baseline results. However, the result obtain with such settings is worse than what was obtained when using a threelayered architecture with two GRU PACK. layer (-0.1pp). We believe this is to be explained by the fact that in such setting, there necessarily is a GRU PACK. at the last layer, and we observe such architecture degraded the results overall.

GRU PACK.

Architecture 5 layers 4 layers 3 layers Table 6.8: R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of three GRU PACK. layers using phone, syllable-word and word segments (models were selected based on the maximum R@1 on the validation set). The same naming conventions of Table 6.4 are used for this table

Even though we notice that R@1 improved when adding another boundary level in this last experiment, we also observe that the leap in the results is not as big as what we observed before. When using one GRU PACK. , we observe an improvement of +1.1pp in R@1 compared to our baseline architecture. When using two GRU PACK. , we observe an improvement of +2.8pp compared to using one GRU PACK. . And finally, when using a third GRU PACK. , we observed an improvement of +1.4pp compared to using two GRU PACK. . Thus, even though introducing more structure into the network is beneficial, we also observe that some levels are more critical than others and have a bigger effect on the final result.

Chapter Summary

In this chapter we studied the impact of prior speech segmentation in a VGS model. We presented a simple method to introduce boundary information at any recurrent layer of our architecture. We do so by simply resetting the RNN's history every time there is a segment boundary. From this initial condition, we derived two conditions, the all and keep conditions, where we either keep all the vectors of a segment or only the last vector. We showed that the latter is more efficient, as, first, it helps the network to learn consistent representations, and second, it reduces the computational charge on the upper layers (as less vectors are forwarded to the next layers).

The goal of this chapter was to see if segmenting speech in sub-units was beneficialthat is, enabled the network to learn a better speech-to-image mapping -and if so, which units maximise the performance. It is indeed the case that segmenting speech into subunits helps. This results is coherent with prior linguistic observation as well as previous results with textual models. As to which segment obtains the best performance we observe mixed results. Indeed, word segmentation yields better results than phone segmentation, but we do also observe that syllable-like segmentation also gives results that are in the same ballpark as word segmentation. Nevertheless, word segmentation seems to be a more robust representation compared to syllable as such word segments consistently yield better results at various levels of our architecture. Our experiments thus allowed us to observe that, such as for humans, the use of large units, such as words, is indeed the most efficient solution to learn a reliable speech-to-image mapping.

Additionally, we observed different results depending on the level at which segmentation information is introduced. We observed negative effects if boundary information is introduced too late (last layer of our architecture). This tends to show that the last layer is not concerned with form any more and thus that boundary information should be introduced before. This result is coherent with previous observation by Chrupa la et al. ( 2017a) and Merkx et al. (2019) who showed that the last layer of such network does not encode word presence (or absence) effectively. Surprisingly, if boundary information is introduced too early (first layer), we do not observe any improvement over our baseline results. Thus, in order to be used effectively, boundary information should be provided to the middle layers. This shows that speech requires a certain amount of pre-processing before it is segmented.

Nonetheless, even though if introducing boundary information is useful, it only mildly improves the performance of the network. It is only when different levels are combined that the performance of the network reaches its peak. Our GRU PACK. setting allowed us to simply introduce such hierarchy in a neural network by simply stacking GRU PACK. layers and providing different boundary information to each of them. Our experiments reveal that having a structure that uses low-level segments (i.e. phones) jointly with high level segments (i.e. words) is better than using segments that are more or less of the same size (i.e. jointly using syllables and words). Adding boundary information also allowed us to reduce the number of layers while increasing the performance of the network. Therefore, explicitly taking into account the hierarchical nature of speech proves useful.

Conclusion 7.1 Summary of Findings

In this thesis, we studied an RNN-based model of Visually Grounded Speech. Our goal was to analyse the representations learnt by our model in order to better understand what type of linguistic knowledge neural models are able to acquire in an unsupervised fashion. We compared these representations to what is known of human speech processing. More specifically, we focused on lexical acquisition and found commonalities between the processes at work in the models we studied and the processes reported in the child language acquisition literature.

More specifically, the main contribution of this thesis is threefold:

• Synthetically Spoken STAIR data set. We introduced the "Synthetically Spoken STAIR data set" which is based on the STAIR data set [START_REF] Yoshikawa | STAIR captions: Constructing a Large-Scale Japanese Image Caption Dataset[END_REF]. This data set constitutes the Japanese equivalent of the "Synthetically spoken COCO data set" (Chrupa la et al. 2017a) for English. We introduced this data set as we believe working on typologically distinct languages is of paramount importance if one wants to truly understand the modelling capacities of deep neural networks. Despite recent work on this subject (Linzen et al. 2018, 2019[START_REF] Alishahi | Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP[END_REF][START_REF] Belinkov | Analysis Methods in Neural Language Processing: A Survey[END_REF], most of the analyses focus on analysing deep neural networks trained on data sets in English. As both the STAIR data set and the COCO data set use the same set of images, this allowed us to compare the learning process of neural models trained on the exact same data, the only changing parameter being the language spoken.

• Analysis of attention. We showed that RNN-based models are able to detect recurring specific patterns in the acoustic input. More specifically, we showed such models did so by tuning the weights of their attention mechanism so as to give more importance to parts of the acoustic input that are particularly discriminative to predict a visual context. The models we trained focus specifically on concrete words such as nouns, as those refer to objects that are particularly salient in the images. We observed this behaviour for two typologically distinct languages, English and Japanese, hence showing this behaviour does not depend on a particular language. We concluded that the models we studied displayed a noun bias, such as what is also found in humans during the process of lexical acquisition, but we however highlighted that such preference might only be a consequence of the data set used in the experiments, which consists of still images.

We also showed that despite adopting a language-general processing behaviour, the model could also develop a language-specific behaviour in order to better solve its task. Specifically, we put forward in this thesis that the Japanese models have learned to detect and highlight particles (such as the "ga" particule) showing a language-specific behaviour. By doing so, the model adopted the same behaviour as Japanese toddlers in order to segment the speech stream, who also use particles to detect nouns in the speech stream.

The networks' ability to highlight nouns -and particles for Japanese -is a behaviour that the networks learned to adopt with a very few examples -less than 500 image/caption pairs -showing the network quickly learns which are the most important parts of the captions. This is an interesting finding, as it is generally admitted that large neural networks require a large amount of data to be trained effectively.

We indeed find that the models we train require large amounts of data to solve their primary task effectively (see Appendix D), but they are however able to focus on the important parts of the speech inputs with very few examples. This suggests such networks could be used on low-resourced languages by linguists so as to automatically detect nouns in the spoken input.

• Analysis of individual word knowledge. We showed that the network was able to map individual nouns to their correct visual referent. This suggest that the network implicitly segmented its input into sub-units, and later mapped them to a visual context. However, we observed that the network was not able to map all isolated nouns to their visual referents equally well. Indeed, while the network was able to learn very reliable mappings between a word-form and its visual referent for about half of the most important words of the data set, the other half were not mapped to their visual referent, suggesting the model's lexicon is restricted to a set of words. We observed that this phenomemon was mainly due to the frequency of the word in the caption: the more frequent a word-form is, the better the model maps it to its visual referent.

We then studied how the network activates an isolated word and compared it to model of word activation and recognition in humans. We showed that the network necessarily needed to have access to the first phoneme of a word in order to activate the representation of the target word. This result is similar to what is postulated in the Cohort model of speech recognition, where word onsets are of particular importance to activate and recognise a word. Using an algorithmic equivalent of the gating paradigm [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF][START_REF] Grosjean | The recognition of words after their acoustic offset: Evidence and implications[END_REF], a methodology stemming from psycholinguistics, we were able to observe that word activation does not occur linearly, but rather evolves in steps. This enabled us to conclude the model was able to recognise a word from a partial input, that is before its offset. We investigated if word recognition was carried out through a process of simultaneous activation of a cohort of words, that would then compete for recognition. We found some evidence of simultaneous activations for some words and competition between them, however, this process seems far from systematic, as we have shown that some words are activaed without competing with similar sounding words.

• Introduction of prior linguistic information. Finally, we investigated if introducing prior linguistic information in the form of boundary information was beneficial. We indeed found the network benefited from such information, particularly when the network was given word boundaries. The network benefited from phone and syllable boundary information, however, not as much as word boundaries. More importantly, we found that taking into account the hierarchical nature of speech, by simultatneously giving to the network phone, syllable and word boundaries, yielded even better results.

We observed that introducing this information at different layers impacted significantly the results, and that this information was better handled by the intermediate layers. This experiment also allowed us to observe that keeping only the last vector of a segment (keep condition) which is equivalent to subsampling the speech signal in a linguistically meaningful way was more effective than keeping all the vectors of a segment. Hence, compressing the information in such a way that it is linguistically meaningful enables the network to acquire better representations.

In his book, Bloom (2002, p. 60) argues that "children learn the meanings of words through theory of mind. If this is right, then a direct connectionist implementation of word learning, in which sounds are associated with percepts, is unfeasible. (And this does preclude all connectionist theories of word learning that [he is] aware of.)". We believe the experiments we carried in this thesis, as well as previous work by [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF]), Chrupa la et al. (2017a), Merkx et al. (2019) (among others) show that purely connexionist models are able to directly associate sounds with percepts, here, in the form of vector representations of visual stimuli. Hence, connexionist models are able to learn words to some extent. Of course, we do not argue that child lexical acquisition is only done via a purely associative mechanism, but it might be that a purely associative learning mechanism bootstraps lexical acquisition for children. An argument against this fact could be that connexionist approaches require large amounts of data to be trained effectively. However, our experiments show that our models learned to focus on specific nouns with a very few number of examples, suggesting associationist bootstrapping constitutes a viable mechanism to acquire a lexicon.

Given the architecture we use in our experiments, the lexical acquisition process we simulate is necessarily limited, as our architecture is not fully grounded (see Figure 2.8 in Chapter 2). Also, an ideal model would not only include visual stimuli, but also olfactory, tactile, etc. and would be a model that would be possible to interact with. In this thesis, we consider word-form/object mapping as a proxy for word learning. Yet, word learning entails much more than simply mapping a word-form to its visual referent, but should also include morphological information, syntactic information, and pragmatic information. It is nonetheless possible the models we trained learned other information, but this remains to be tested.

The experiments we conducted in this thesis allow us to conclude that VGS models implicitly segment their input into sub-units and associate these sub-units to their visual referent. This process seems to only emerge as a by-product of their main task which is to minimise a distance between an acoustic and a matching visual stimulus. Our conclusion is in line with the very recent work of [START_REF] Khorrami | Can phones, syllables, and words emerge as sideproducts of cross-situational audiovisual learning? -A computational investigation[END_REF] who conclude that "both sub-lexical and lexical representations can gradually emerge from the interaction of rich multimodal experiences available to the learner" and that these representations emerge as "a byproduct of multimodal sensing and interaction with the environment". However, even though an implicit segmentation seems to take place in such networks, this segmentation seems to be less effective than explicitly segmenting the input, as our experiments suggest. This result is in line prior researches in language acquisition: [START_REF] Havron | Literate and preliterate children show different learning patterns in an artificial language learning task[END_REF] show that literacy -and consequently the knowledge of word boundaries -facilitates the acquisition of novel nouns.

Future Works

Given the work we conducted in this thesis, several future works could be carried out:

• Word Activation in CNN-based models. We studied how RNN-based models store lexical units and activate the representation of individual words by using the gating paradigm. This methodology could also be applied to analyse the representations learnt by CNN-based models in order to understand how the representation of a given word is activated in such models. It would be interesting to know if CNN-based and RNN-based models converge to the same type of representations, and for example, if word activation in CNN-based models proceeds in steps as what we observed for RNN-based models.

Also, CNN-based models such as that of [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] are able to highlight specific parts in the visual input.12 This would allow us to better experiment the competition phase by constructing images that contain instances of objects that could compete with a chosen target word, also figured in the same image. By observing the attention maps over the image, one could precisely measure competition between words. Moreover, using such a CNN-model would enable us to directly reproduce linguistic experiments that measure word recognition and competition using eye-tracking devices such as that of [START_REF] Huettig | The tug of war between phonological, semantic and shape information in language-mediated visual search[END_REF]. Such an experiment was recently attempted by [START_REF] Duta | A neural Network Model of Lexical Competition during Infant Spoken Word Recognition[END_REF], however, the model they use does not work on raw images. Hence, a CNN-based model would allow us to exactly reproduce psycholinguistic experiments.

• Image-To-Speech. During an intership we did at the Nara Institute of Science and Technology (NAIST) in 2019, we tried to implement an Image-To-Speech model similar to those recently proposed by [START_REF] Hsu | Text-Free Image-to-Speech Synthesis Using Learned Segmental Units[END_REF][START_REF] Wang | Show and Speak: Directly Synthesize Spoken Description of Images[END_REF]) which directly produce a spoken description using an image as input, without requiring an intermediary textual representation. The model we implemented -which we did not describe in this thesis nor in any article -was only able to produce isolated sequences and not full captions. We believe our model was not able to learn how to produce full captions as we did not use any discrete units such as in the model proposed by [START_REF] Hsu | Text-Free Image-to-Speech Synthesis Using Learned Segmental Units[END_REF], and we used an image encoder that processed raw images, instead of using an object detector first, as in the work of [START_REF] Wang | Show and Speak: Directly Synthesize Spoken Description of Images[END_REF].

We would like to further work on such model to understand why our model did not succeed in predicting full captions. We would also like to study how such networks gradually learn to produce their first sentences, and compare this evolution to that of children. Indeed, when children learn how to speak, do not start by uttering full sentences, but rather start by producing isolated words. Later on, they produce twoword-long sentences and from that point on start to produce full sentences. It would be interesting to investigate if an Image-To-Speech model goes through the same steps as children, and if not, investigate why it is the case. Similarly, such experiment would provide insight on the linguistic development children go through by studying the representations learnt by such a model.

• Discrete segmentation. The experiments conducted in Chapter 6 reveal that giving an explicit segmentation improves the models' ability to corretly map a spoken caption to its visual context. Even though our experiments show that when a explicit segmentation is not given, the models implicitily segment the spoken input into sub-units, this implicit segmentation appears to be less effective.

Hence, it would be desirable that the network learns to explicitily segment the spoken input into sub-units. Several options, which we started to explore, could be possible. [START_REF] Kreutzer | Learning to Segment Inputs for NMT Favors Character-Level Processing[END_REF] and Chen et al. ( 2019) introduced a mechanism that allows recurrent neural networks to learn how to segment an input stream into subunits. These methods essentially consist in predicting a binary value (1 or 0) if the current vector constitutes a segment boundary or not. Hence, given the GRU PACK.

architecture we develop, it would simply consist in adding a binary classifier for each time frame. Yet, our initial experiments reveal that the definition of the loss function is critical so that the network does not learn to either consider each input vector as a segment of its own, or on the contrary, discard all the vectors except the last one and consider the full caption as one segment.

Recent work by [START_REF] Shain | Acquiring language from speech by learning to remember and predict[END_REF] suggest such approach is possible. However, their work reveal that their model is unable to learn large linguistic units such as words but only phoneme-like units. We believe this is the case as the spoken input is not grounded. Hence, by using a corpus similar to those used in this thesis, we believe such a model would be able to learn to segment both phoneme-like and word-like units.

In conclusion, neural models of visually grounded speech models offer invaluable opportunities to study and test hypotheses about child language acquisition, thanks to their ability to model complex interactions across several modalitites. New data sets, such as the SEEDLingS data set [START_REF] Bergelson | Nature and origins of the lexicon in 6-mo-olds[END_REF] or the data set recently collected by [START_REF] Tsutsui | A computational Model of Early Word Learning from the Infant's Point of View[END_REF], which are large scale recordings collected in ecological environments [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF], will allow researchers to simulate language acquisition with more realistic data than ever before. List of Tables 6.2 are used for this table. . . . . . . . . 6.4 R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers (phones and words) . . . . 6.5 R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. (random phones and/or random words) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.6 R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. (phones and syllable-word) . . . . 6.7 R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers (syllable-word and word) . 6.8 R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of three GRU PACK. layers (phone, syllable-word, and word) . Dans un laps de temps relativement court, les enfants sont capables d'acquérir leur langue maternelle, ce qui leur permet de la comprendre et de la parler sans effort. Il s'agit d'un exploit étonnant, car ils sont capables de le faire sans beaucoup de supervision. Au contraire, le contexte perceptible environnant semble leur fournir toutes les informations nécessaires à l'acquisition de leur langue maternelle. Landau & Gleitman (1985, p. 7) affirment que "l'input de l'enfant consiste en des paires son/situation, mais que son output final est un ensemble de paires forme/sens, appropriées à un ensemble infini de situations nouvelles mais bien circonscrites". La question est la suivante : comment les enfants passent-ils de paires son/situation à des paires forme/sens ? L'une des étapes que les enfants doivent franchir est de comprendre que les sons qu'ils perçoivent constituent des signes conventionnels utilisés à des fins de communication, et non des sons aléatoires. Cette étape n'est peut-être pas la première que les enfants franchissent, mais elle est essentielle pour passer de son à forme. La transition des paires son/situation aux paires forme/sens implique également une étape de segmentation. La première étape de segmentation à laquelle nous pouvons penser est la segmentation du flux de parole en formes. En effet, le flux de parole contient des séquences de formes sont cependant connectées les unes aux autres. Les enfants doivent donc apprendre à segmenter le flux de parole de manière appropriée afin de découvrir les formes conventionnelles des mots utilisés dans leur langue maternelle. La deuxième étape de segmentation à laquelle on peut penser consiste à analyser l'environnement afin d'identifier et d'extraire les acteurs : qui parle, à qui, à propos de quoi, etc. Alors que la première étape consiste à identifier des schémas récurrents dans la modalité parlée, l'autre étape consiste à identifier des schémas récurrents dans d'autres modalités (visuelles, haptiques, etc.). Une fois que les formes ont été extraites du flux de la parole et que les référents ont été extraits de l'environnement, l'enfant doit apprendre à les mettre en correspondance. À ce stade, l'enfant est passé de paires son/situation à des paires forme/référent. L'enfant devient capable d'abstraire les points communs entre différents référents de la même forme et d'en déduire une signification et cela lui permet finalement à l'enfant de construire un ensemble de paires forme/sens. Les paires forme/sens que l'enfant construit doivent être stockées afin d'être facilement accessibles pour analyser le flux de parole et pour construire ses propres énoncés. Par conséquent, la forme des mots qui est stockée dans le lexique mental doit être suffisamment spécifique pour ne pas permettre la reconnaissance de formes de mots dont la prononciation serait similaire, sans pour autant être trop spécifique afin de tenir compte des variations et d'éventuelles erreurs de prononciation. Cette étape est communément appelée reconnaissance des mots parlés.

Dans cette thèse, nous nous proposons d'étudier un modèle neuronal de la parole visuellement contextualisée (visually grounded speech models). Les modèles de parole visuellement contextualisée sont des modèles qui sont entraînés pour résoudre une tâche de recherche d'image à partir d'une requête orale. C'est-à-dire qu'étant donné la description orale d'une image, ils doivent trouver l'image la plus proche de la description fournie parmi une collection d'images (ou vice-versa).Pour ce faire, ces réseaux doivent apprendre à transformer de manière appropriée l'image d'entrée et la description orale d'entrée afin qu'il soit facile de trouver l'une en fonction de l'autre. La tâche que ces réseaux doivent résoudre est donc très proche de celle d'un enfant qui acquiert sa langue maternelle. En effet, de tels réseaux ont comme input des paires son/situation (c'est-à-dire une description parlée et son image correspondante), et afin de trouver l'image correspondante correcte, nous faisons l'hypothèse que le réseau doit apprendre à transformer cette paire son/situation en une paire forme/sens. En effet, pour trouver l'image correspondante parmi une collection d'images à partir d'un énoncé, le réseau doit d'une manière ou d'une autre segmenter le flux de parole en sous-unités, de sorte que les sous-unités résultantes se réfèrent à des objets dans l'image. La représentation des unités extraites doit être suffisamment spécifique pour que, lorsqu'une de ces unités est donnée en entrée au réseau, il ne récupère que les images présentant des instances auxquelles l'unité parlée fait référence. De même, comme pour les humains, du côté de l'image, le réseau doit être capable d'abstraire le référent, de sorte que l'image cible puisse être récupérée, même si l'objet principal de l'image est présenté de manière non canonique ou dans un environnement encombré. Par conséquent, la tâche pour laquelle les modèles neuronaux de parole visuellement contextualisée sont entraînés est très proche de celle des enfants qui apprennent leur langue maternelle, et plus précisément de la tâche d'acquisition lexicale. Ils doivent passer par les mêmes étapes qu'un enfant, que sont : la segmentation, le appariement forme/référent, et la reconnaissance.

Contributions

Dans cette thèse, nous étudions un modèle récurent de parole visuellement contextualisée (PVC). Comme de tels modèles résolvent une tâche similaire à celle des enfants, qui découvrent l'ensemble des paires forme/sens de leur langue maternelle, nos analyses visent à comprendre si les modèles neuronaux le font de la même manière. Dans ce manuscrit, nous présentons les contributions suivantes :

(i) Nous présentons une extension vocale d'un ensemble de données de description textuelle d'images (image captioning) qui permet d'entraîner de modèles de parole visuellement contextualisée. Contrairement à la plupart des jeux de données image/parole qui sont en anglais, ce jeu de données est en japonais, ce qui nous permet d'étudier, dans une approche contrastive, l'impact de la langue d'entrée (anglais ou japonais) sur les représentations apprises par le modèle.

(ii) Nous étudions les poids d'attention des mécanismes d'attention de nos modèles, afin de comprendre quelles parties du signal de parole sont mises en avant, et dans quelle mesure elles diffèrent de ce que le hasard prédirait. Nous proposons également une investigation longitudinale, où nous étudions comment les poids d'attention évoluent pendant la phase d'entraînement.

(iii) Nous introduisons une méthodologie issue de la littérature psycholinguistique, le paradigme du gating [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF], qui nous permet d'étudier facilement l'activation et la compétition des mots parlés dans notre modèle. A notre connaissance, c'est la première fois qu'une telle méthodologie est utilisée pour étudier les représentations apprises par les modèles neuronaux de traitement de la parole.

(iv) Nous proposons une méthode permettant d'introduire simplement des informations linguistiques préalables sous la forme de frontières de segments (frontières de phonèmes, de syllabes ou de mots) dans un modèle neuronal de traitement de la parole. La méthode que nous proposons permet d'intégrer plusieurs types de frontières, à différents niveaux de l'architecture neuronale, ce qui permet de prendre en compte la nature hiérarchique de l'entrée parlée. Cela nous permet d'étudier si la segmentation du signal vocal en sous-unités permet aux modèles d'apprendre une meilleure correspondance entre les images à leurs descriptions orales.

2 Informations générales sur l'acquisition du langage chez l'enfant L'acquisition du langage désigne le processus par lequel un enfant apprend sa langue maternelle. Afin d'acquérir sa langue maternelle, une étape que l'enfant doit franchir est la construction d'un lexique mental [START_REF] Emmorey | The mental lexicon[END_REF]. Cependant, la construction de ce lexique mental est loin d'être facile. En effet, le flux de la parole ne contient pas de mots bien séparés auxquels l'enfant n'aurait qu'à attribuer un sens. L'enfant doit d'abord segmenter le flux de la parole en sous-unités et finalement associer un sens à chacune des sous-unités résultantes. Ainsi, les étapes fondamentales du traitement et de la compréhension de la parole que les enfants doivent acquérir pour pouvoir correctement analyser le flux de la parole et construire un lexique mental -et finalement acquérir leur langue maternelle -sont identifiées par Di Cristo (2013, pp. 51-52) comme la segmentation, la reconnaissance et l'interprétation ; étapes que nous détaillons ci-après.

Segmentation

Les indices utilisés par les enfants pour segmenter le signal de parole en sous-unités peuvent être classés ainsi : les indices suprasegmentaux, les indices segmentaux, et les indices lexicaux. On peut également y ajouter d'autres types d'indices qui seront détaillés à la fin de cette présente section, qui ne rentrent nettement dans aucune des catégories pré-citées.

Indices suprasegmentaux

Les indices suprasegmentaux désignent les indices qui sont présents dans les caractéristiques suprasegmentales de la parole. Les caractéristiques suprasegmentales sont des modulations du signal vocal qui peuvent s'étendre sur plus d'un segment (c'est-à-dire un phone ou une syllabe). Ces caractéristiques suprasegmentales, regroupées sous le terme générique de prosodie, comprennent le rythme (tempo et pauses), l'intonation et l'accentuation. La prosodie assure de nombreux rôles fonctionnels, le principal qui peut être utile au nourrisson étant sa fonction de démarcation, qui regroupe les unités parlées et que les enfants peuvent utiliser pour inférer les frontières des mots.

Les enfants utilisent notamment des informations rythmiques pour segmenter le flux sonore en sous unités. [START_REF] Nazzi | Language discrimination by newborns: Toward an understanding of the role of rhythm[END_REF] ont montré que les enfants sont sensibles aux informations rythmiques portées par le signal de parole puisqu'ils sont en effet capables de distinguer deux langues appartenant à deux classes rythmiques différentes, mais sont incapables de distinguer deux langues différentes si celles-ci appartiennent à la même classe rythmique.

Les informations rythmiques étant (principalement) portée par les syllabes accentuées en anglais, [START_REF] Jusczyk | Infants' Preference for the Predominant Stress Patterns of English Words[END_REF] ont exploré si les enfants segmentaient le flux sonore selon l'accentuation des syllabes ; la présence d'une frontière de mot étant fortement liée à la présence d'une syllabe accentuée [START_REF] Cutler | The role of strong syllables in segmentation for lexical access[END_REF]). [START_REF] Jusczyk | Infants' Preference for the Predominant Stress Patterns of English Words[END_REF] concluent que les enfants de 7,5 mois segmentent le signal sonore de sorte que les unités résultantes aient le motif accentué/non-accentué confirmant ainsi l'hypothèse des auteurs. Ce résultat a également été trouvé pour des enfants néerlandophones [START_REF] Houston | Crosslanguage word segmentation by 9-month-olds[END_REF]. Pour les enfants français, les travaux de Nazzi et al. (2006[START_REF] Nazzi | Segmentation précoce de la parole continue en mots : évaluation inter-linguistique de l'hypothèse d'initialisation rythmique[END_REF] (et de [START_REF] Bosch | Rapid gains in segmenting fluent speech when words match the rhythmic unit: evidence from infants acquiring syllable-timed languages[END_REF] pour le castillan et catalan) montrent que l'unité de segmentation adoptée par les enfants est la syllabe ; puisque ce sont des langues où l'unité de battement utilisée pour marquer le rythme est la syllabe et non l'accentuation. Ainsi, les sous unités dégagées ne sont pas extraites selon l'accentuation des mots, mais uniquement selon la conformation à une unité syllabique correcte. Cependant, [START_REF] Nazzi | Early segmentation of fluent speech by infants acquiring French: Emerging evidence for crosslinguistic differences[END_REF] notent que les enfants francophones segmentent le flux sonore de manière systématique plus tardivement que les enfants anglophones ou néerlandophones (16 mois chez les enfants francophones contre 10 pour les enfants anglophones sur une tâche de segmentation de mots dissyllabiques).

Outre les indices suprasegmentaux déjà mentionnés, les enfants sont également sensibles à des indices suprasegmentaux affectant des unités plus larges, telles que celles des phrases phonologiques. [START_REF] Christophe | Discovering words in the continuous speech stream: the role of prosody[END_REF] ont montré que la reconnaissance des mots chez les enfants de 13 mois est effectivement contrainte par les frontières des phrases phonologiques. Les enfants reconnaissent effectivement un mot cible familier (par exemple, paper ) s'il fait partie d'une phrase phonologique ("[The college][with the biggest paper forms][is best] ") mais pas s'il est à cheval sur deux phrases phonologiques ("[The butler][with the highest pay][performs the most] "). Ainsi, en prêtant attention aux indices signalant les frontières de phrases prosodiques, les enfants peuvent déduire les frontières de mots, puisqu'une frontière de phrase prosodique correspond nécessairement à une frontière de mot. [START_REF] Johnson | The Edge Factor in Early Word Segmentation: Utterance-Level Prosody Enables Word Form Extraction by 6-Month-Olds[END_REF] montrent d'ailleurs que les enfants prêtent une attention particulière aux fins de phrases phonologiques et sont capables de reconnaitre un mot si celui-ci est présent au début ou à la fin d'une phrase phonologique.

Indices segmentaux

Les indices segmentaux sont des indices directement liés aux segments (c'est-à-dire phone[me]s) d'une langue. Comme [START_REF] Mattys | Integration of Multiple Speech Segmentation Cues: A hierarchical Framework[END_REF], nous regroupons sous le terme d'indices segmentaux les indices phonotactiques et les indices acoustico-phonétiques. [START_REF] Jusczyk | Infants' Sensitivity to the Sound Patterns of Native Language Words[END_REF] ont montré que les enfants de 9 mois étaient sensibles aux indices phonotactiques, c'est-à-dire aux informations qui indiquent si une séquence de phonèmes constitue une séquence légale ou non dans son enchaînement. Ainsi les enfants de 9 mois préfèrent les pseudo-mots dont les séquences de phonèmes sont légales et non des pseudo-mots constitués de séquences illégales. [START_REF] Jusczyk | Infants' Sensitivity to Phonotactic Patterns in the Native Language[END_REF] montrent que les enfants préfèrent les séquences légales les plus fréquentes face à celles qui sont légales, mais moins fréquentes. Mattys & Jusczyk (2001b) 2012) ont montré que la connaissance d'informations phonotactiques contraint l'acquisition de mots nouveaux. Ainsi, chez des enfants anglophones de 12 mois, des (pseudo-)mots nouveaux ne peuvent être associés à des objets nouveaux que si ceux-ci comportent des séquences phonotactiques légales. L'harmonie vocalique, qui peut être considérée comme un cas spécial de contrainte phonotactique ; les segments concernés n'étant simplement pas adjacents ; est également un indice qui peut être utilisé par les enfants afin d'extraire des mots du signal sonore. Ketrez (2013) a en effet montré qu'en turc, lorsque l'harmonie vocalique est rompue, cela signale très probablement une limite de mot, et suggère que les enfants pourraient utiliser cette propriété pour inférer les limites des mots. [START_REF] Mintz | Infants' sensitivity to vowel harmony and its role in segmenting speech[END_REF] montrent ainsi dans une tâche d'apprentissage d'une langue artificielle que des enfants anglophones segmentent effectivement les mots en fonction de l'harmonie des voyelles et que cet indice constitue donc un indice viable pour extraire des unités lexicales.

Les enfants sont également sensibles à de fines variations acoustiques et se servent d'indice de coarticulation ainsi que d'indices d'ordre allophonique afin de déterminer la présence d'une frontière de mot. [START_REF] Christophe | Do infants perceive word boundaries? An empirical study of the bootstrapping of lexical acquisition[END_REF] ont par exemple montré que des nourrissons de 3 jours font la différence entre un séquence dissyllabique CVCV extraite d'un mot (par ex. /mÃta/ dans "sédimentation") et la même séquence CVCV à cheval sur une frontière (par exemple /mÃ#ta/ dans "déguisement talentueux "). Ils sont donc capables de différencier deux réalisations différentes d'une même séquence phonologique à partir d'indices acoustiques fins (expliquant ainsi la sensibilité aux phrases phonologiques). Des résultats similaires ont été observés en anglais par [START_REF] Hohne | Two-month-old infants' sensitivity to allophonic differences[END_REF]

(différence "nitrate" [naIú h ô ˚eIt^] et "night rate" [naIt^ôeIt^]
). Plus récemment, Mattys & Jusczyk (2001a) ont montré que les enfants utilisent cette sensibilité aux variations allophoniques pour déterminer si une séquence constitue un mot ou non, et ainsi segmenter le flux sonore en sous-unités.

Finalement, les enfants utilisent également des indices statistiques afin segmenter le flux sonore en sous-unités. [START_REF] Saffran | Statistical Learning by 8-Month-Old Infants[END_REF] ont notamment montré que les enfants utilisent les probabilités de transition entre syllabes afin de poser des frontières de mots, considérant les séquences des syllabes qui ont une forte probabilité de transition entre elles comme des unités, tandis que les séquences de syllabes ayant une faible probabilité de transition entre elles ne sont pas considérées comme formant une unité. Pelucchi et al. (2009b) ont confirmé ces résultats sur de l'italien. Les travaux de Pelucchi et al. (2009a) montrent que les enfants se servent aussi bien des probabilités de transitions progressives (forward transition probability) que de probabilités de transition régressives (backward transition probability)

Indices lexicaux

Dès que l'enfant a isolé un certain nombre de formes de mots, il peut les utiliser afin d'isoler les mots qui précèdent et/ou suivent le mot connu. [START_REF] White | Segmentation by lexical subtraction in Hungarian speakers of second-language English[END_REF] ont inventé le terme "segmentation par soustraction lexicale" et le définit comme "l'utilisation de la connaissance lexicale pour imposer une structure de segmentation à l'entrée de la parole". [START_REF] Bortfeld | Mommy and Me: familiar names help launch babies into speech-stream segmentation[END_REF] ont ainsi montré que les nourrissons de 6 mois utilisaient déjà des mots familiers ("mommy", "daddy", ou le propre nom de l'enfant) pour segmenter le mot suivant dans le flux de parole. Plus récemment, [START_REF] Bergelson | At 6-9 months, human infants know the meanings of many common nouns[END_REF] ont montré que les très jeunes enfants connaissent de nombreux mots courants (yaourt, banane, etc.), mots qui peuvent ainsi être utilisés pour aider la segmentation. [START_REF] Shi | Frequency and form as determinants of functor sensitivity in English-acquiring infants[END_REF] et [START_REF] Shi | The effect of functional morphemes on word segmentation in preverbal infants[END_REF] ont montré respectivement que chez des enfants anglophones et francophones, les déterminants sont utilisés par les enfants afin de segmenter le signal sonore en sous-unités. Un phénomène similaire a été mis en avant en japonais par [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF], en montrant notamment que les enfants japonais se servent de la particule "ga" afin de déterminer des frontières de mots. Finalement, [START_REF] Johnson | The Edge Factor in Early Word Segmentation: Utterance-Level Prosody Enables Word Form Extraction by 6-Month-Olds[END_REF] rapportent que 80% des mots entendus en isolation par les enfants sont des interjections (oh, ah, etc.) et que ceux-ci pourraient également servir de support à la segmentation.

Autres indices

Les enfants peuvent également se servir d'autres indices afin de déterminer les frontières de mots. Notamment, le langage bébé (également appelé mamanais) permettrait aux enfants de mieux segmenter le flux sonore. En effet, le langage bébé est caractérisé par de nombreuses variations par rapport au langage adulte, notamment, une exagération de la variation de la fréquence fondamentale ou bien encore des pauses plus longues. [START_REF] Thiessen | Infant-Directed Speech Facilitates Word Segmentation[END_REF] concluent dans leur étude que le langage bébé permet aux enfants de 7,5 mois de mieux segmenter le flux sonore en mots, et [START_REF] Ma | Word Learning in Infant-and Adult-Directed Speech[END_REF] ont montré que le langage bébé facilite l'acquisition (en termes de rétention) de mots nouveaux. Cependant, même si le langage bébé peut aider les enfants il n'est en rien nécessaire. Il existe en effet des populations dans lesquels la parole est peu adressée aux enfants tant qu'ils ne sont pas capables de parler (voir Cristia et al. 2019), et qui arrivent malgré tout à apprendre leur langue maternelle.

Les indices que nous avons présentés l'ont été en isolation, comme si les enfants faisaient usage d'un seul indice de segmentation à la fois. Or, ce n'est pas le cas et les enfants utilisent de nombreux indices de manière concomitante cela permettant de "rendre une 'explosion combinatoire' moins probable" (Thiessen & Erickson 2009, p. 44). [START_REF] Johnson | Testing the limits of statistical learning for word segmentation[END_REF] montrent ainsi que l'utilisation de probabilités de transition ne semble être pleinement efficace que lorsqu'elles sont associées à d'autres indices. Cela est confirmé par les travaux de Mersad & Nazzi (2012) qui ont montrés dans leur expérience que les enfants utilisaient correctement les probabilités de transition entre syllabes seulement après être parvenus à extraire de la séquence présentée un mot connu (dans leur expérience, le mot maman). Si ce mot connu ne figurait pas dans la séquence, alors les enfants ne parvenaient pas à utiliser les probabilités de transitions.

Conclusion

Ainsi, les enfants utilisent de nombreux indices pour segmenter le flux de parole. Ceux-ci sont de nature variée, allant d'indice supra-segmentaux à des indices infra-segmentaux en passant par des indices segmentaux. Il semble donc que les enfants utilisent une combinaison d'indices descendants (par exemple, des mots isolés) et ascendants (par exemple, la phonotactique, les transitions de probabilité, etc.) afin de segmenter le flux de parole. Ces deux approches sont nécessaires -car, par exemple, la découverte des schémas phonotactiques réguliers dans les mots nécessite la connaissance d'un ensemble de mots individuels -et complémentaires -car en retour, les schémas phonotactiques déduits permettent à l'apprenant de segmenter encore plus de mots.

Appariement

Dans les sections précédentes nous avons considéré le langage in vacuo comme si les mots existaient pour et par eux-mêmes, déconnectés de tout contexte. Ce n'est bien sûr pas le cas :

Le point important[...] est que le langage n'a d'existence que par rapport à l'ordre physique et mental des choses. Le langage n'est pas une sorte de système indépendant dans lequel on peut puiser à volonté. Les aspects de la langue n'ont de signification que dans la mesure où ils se rapportent à des aspects du monde. (Dixon 2012, p. 434) Ainsi, les informations contextuelles jouent un rôle décisif dans l'acquisition lexicale : on peut notamment citer les informations contextuelles liées à des percepts (vision, toucher, odorat) qui vont venir guider l'enfant dans l'acquisition de sa langue maternelle.

Théorie de l'esprit et attention conjointe

Des chercheurs tels que Bloom affirme que "la covariation statistique entre le mot et le percept n'est ni nécessaire ni suffisante pour l'apprentissage des mots." (Bloom 2002, p. 59) et que la mise en correspondance des mots et des référents et, en fin de compte, l'acquisition du sens semblent exiger quelque chose de plus. En effet, il existe un nombre infini de possibilités parmi lesquelles l'enfant doit choisir (voir le célèbre exemple de Quine, gavagai, Quine 1964).

Pour [START_REF] Tomasello | Constructing a Language[END_REF], la lecture d'intention (intention reading) est l'ingrédient nécessaire à l'acquisition de la langue par les enfants, notamment lorsqu'il s'agit d'inférer le référent d'une forme verbale donnée. De la même manière, [START_REF] Bloom | How Children Learn the Meanings of Words, A Bradford Book[END_REF] soutient que "les enfants utilisent leur psychologie naïve ou théorie de l'esprit pour comprendre à quoi les gens font référence lorsqu'ils utilisent des mots" et qu'il est nécessaire de disposer d'une théorie de l'esprit pour contextualiser de manière appropriée ce qui est dit. Le fait de posséder une théorie de l'esprit permet aux enfants d'entrer dans des moments d'attention conjointe. Un moment d'attention conjointe peut être défini comme "un épisode triadique d'interaction impliquant un interlocuteur, un nourrisson et un objet" [START_REF] Rudd | Joint Focus of Attention, in 'Encyclopedia of Child Behavior and Development[END_REF]. La nécessité de partager des moments d'attention conjointe semble être confirmée par des travaux tels que ceux de Tomasello & Todd (1983) qui montrent que les enfants qui partagent de longues périodes d'attention partagée avec leurs interlocuteurs ont un vocabulaire perceptif globalement plus étendu que ceux pour lesquels les moments d'attention conjointe sont peu fréquents. Les travaux de Moore et al. (1999) semblent aller également dans ce sens en montrant que les enfants sont plus aptes à apprendre un nouveau mot si celui-ci a été vu dans un moment d'attention conjointe.

Assomption, biais et erreurs

Même si une théorie de l'esprit et les moments d'attention conjointe semblent décisifs pour acquérir une langue, il semble que l'acquisition du lexique soit guidée par de nombreux biais et assomptions que les enfants feraient sur le monde. On peut notamment citer les travaux de [START_REF] Markman | Constraints Children Place on Word Meanings[END_REF] qui postule l'existence de plusieurs biais que sont l'hypothèse de l'objet entier, l'hypothèse taxonomique, et l'hypothèse d'exclusivité mutuelle. Ainsi le premier guiderait les enfants afin que les étiquettes des mots se réfèrent aux objets dans leur globalité et non à des portions de ceux-ci. Le second stipule que les étiquettes de mot s'étendent aux objets similaires, c'est-à-dire aux objets ayant des caractéristiques similaires (par exemple, chien pour un berger allemand ou un bouledogue). Le dernier affirme qu'il existe une bijection entre une forme verbale et ses référents : pour une étiquette donnée, il n'existe qu'un seul référent (conceptuel) associé, et un référent ne peut être désigné que par une seule étiquette canonique. [START_REF] Clark | The principle of contrast: A constraint on language acquisition[END_REF]) mentionne deux principes complémentaires : le principe de contraste et le principe de conventionnalité. Le principe de contraste dit que "chaque paire de mots entre en contraste par leur sens". Autrement dit, l'enfant considère que deux formes verbales ne sont pas des synonymes exacts. Ce principe encouragerait l'enfant à explorer son environnement afin de trouver un référent possible afin de ne pas assigner deux étiquettes différentes à un même mot. Le principe de conventionnalité stipule que "pour certaines significations, il existe une forme conventionnelle que les locuteurs s'attendent à voir utilisée dans la communauté linguistique".

Il arrive parfois que les enfants fassent des erreurs d'appariement et attribuent la mauvaise étiquette à un référent donné. [START_REF] Rescorla | Overextension in early language development[END_REF] fournit une typologie des principales causes de surextensions : sur-inclusions catégorielles où une étiquette est utilisée pour un référent proche du référent réel (bébé pour les enfants), sur-extensions analogiques où une étiquette est utilisée pour un référent qui présente une similarité avec le référent réel (textitballe pour les billes ou les pommes), et énoncés prédicats qui sont holophrastiques (par exemple chien en montrant un panier désigner un chien absent). [START_REF] Rescorla | Overextension in early language development[END_REF] estime qu'un tiers des premiers mots de l'enfant sont des surextensions, et qu'environ trois quarts des extensions sont soit des surinclusions catégorielles, soit des surextensions analogiques, la plupart d'entre elles étant faites sur la base de la "similarité perceptuelle" (par opposition à la similarité fonctionnelle). Cependant, bien souvent, la similarité perceptuelle évoquée dans les études se limite à une similarité visuelle.

Modalité visuelle

La vision joue un rôle central dans l'acquisition du langage. En effet, "c'est en grande partie la vision qui dirige l'attention du nourrisson sur les personnes, les objets et les événements ; la vision est importante pour la coordination de l'attention entre le parent et l'enfant ; et elle contraint souvent (surtout dans la culture anglo-américaine) le contenu du langage adressé à l'enfant" [START_REF] Andersen | The impact of input: language acquisition in the visually impaired[END_REF].

Il semble que la capacité des nourrissons à associer systématiquement un référent visuel à mot n'apparaisse que vers l'âge de 12 mois. [START_REF] Werker | Acquisition of word-object associations by 14-month-old infants[END_REF] montrent par exemple qu'à 8 et 12 mois les enfants en sont incapables, cependant ils le sont à 13 ou 14 mois. [START_REF] Smith | Infants rapidly learn word-referent mappings via crosssituational statistics[END_REF] quant à eux trouvent que des enfants de 12 mois en sont capables. Ce dernier résultat semble donc être en contradiction avec [START_REF] Werker | Acquisition of word-object associations by 14-month-old infants[END_REF]. Cependant, comme cette capacité semble se développer dans un laps de temps très court (entre le 12e et 13e mois), il est raisonnable d'observer de légères différences d'une expérience à l'autre, surtout si les paramètres expérimentaux sont différents.

Pour mesurer l'important de l'input visuel sur l'acquisition du langage, nous pouvons nous intéresser à l'acquisition du langage chez les enfants aveugles de naissance. La cécité, même si elle n'empêche une maitrise parfaite de la langue maternelle à l'âge adulte [START_REF] Landau | Language and Experience: Evidence from the Blind Child, Cognitive Science Series[END_REF], peut entraîner des retards dans la prime enfance. Notamment, on constate chez les enfants aveugles une tendance à parler d'événements passés plutôt que de "l'ici et maintenant" . Ainsi, en se concentrant sur des événements passés, l'enfant essaie de recréer une situation dans laquelle il sait que la personne qui s'occupe de lui et lui-même ont partagé une expérience commune dont ils peuvent parler et essaye ainsi de créer une situation d'attention partagée, à défaut de pouvoir en créer ou percevoir une dans "l'ici et le maintenant" [START_REF] Andersen | The impact of input: language acquisition in the visually impaired[END_REF]. Pour les mêmes raisons, les enfants aveugles souffrent également d'un manque de décentration et ont une forte tendance à initier des conversations centrées sur eux-mêmes ou leur environnement immédiat [START_REF] Andersen | Blind children's language: resolving some differences[END_REF][START_REF] Peltzer-Karpf | Spracherwerb bei hörenden, sehenden, hörgeschädigten, gehörlosen und blinden Kindern, number 403[END_REF]. On constate également chez les enfants aveugles une capacité de généralisation moindre. En effet, le fait d'être privé de la vue les prive de l'accès simultané à de nombreuses caractéristiques (forme, couleur, texture, etc.) qui constituent de précieuses informations pour généraliser. De plus, pour l'enfant aveugle, explorer le monde requiert un effort conscient d'exploration de son environnement, ce qu'il ne peut faire que lorsqu'il est en mesure de se déplacer, alors que pour l'enfant voyant, percevoir le monde (au moins visuellement) ne nécessite pas d'efforts conscients (Dunlea 1989, p.10). De même que pour la généralisation, on constate une capacité de conceptualisation moindre chez l'enfant aveugle. Selon [START_REF] Dunlea | Vision and the emergence of meaning: blind and sighted children's early language[END_REF], ils ne parviennent pas à saisir la nature symbolique du langage et ne parviennent pas à appliquer de manière appropriée les mots ou les morceaux de mots qu'ils apprennent à de nouvelles situations et d'après [START_REF] Andersen | Blind children's language: resolving some differences[END_REF] ont tendance à répéter mot pour mot ce qu'ils entendent sans analyse supplémentaire (c'est-à-dire sans segmentation) et tendent à répéter des routines pré-faites.

Ainsi, la vision semble fournir des informations essentielles au jeune enfant afin de lui permettre d'acquérir sa langue maternelle. Plus particulièrement, la vision semble aider l'enfant à conceptualiser le monde et à établir des généralisations.

Reconnaissance

La reconnaissance des mots parlés peut être définie comme le processus d'accès aux éléments lexicaux du lexique mental à partir des motifs phonologiques perçus dans le signal de parole [START_REF] Magnuson | Spoken Word Recognition[END_REF]. En d'autres termes, cela implique de mettre en correspondance ce qui est entendu avec les éléments lexicaux stockés dans le lexique mental. Nous passons en revue dans cette section plusieurs modèles de reconnaissance des mots.

Modèle de la Cohorte

Le modèle de la Cohort (Marslen-Wilson & Welsh 1978) est l'un des tout premier modèle tentant de rendre compte de la manière dont les humains reconnaissent et extraient les mots du flux de la parole. Selon ce modèle, la reconnaissance des mots parlés se déroule en trois étapes simultanées : accès, sélection, et intégration.

L'accès consiste à activer un ensemble de mot-formes dans le lexique mental qui correspondent à l'entrée acoustique. Chaque mot-forme est associée à une unité d'activation, ce qui permet l'activation de plusieurs formes de mots à la fois. Ces unités ne sont activées que si l'entrée acoustique perçue correspond exactement à la forme phonologique internalisée. Tous les mots qui commencent par la séquence sonore perçue sont activés, et forment ainsi une cohorte. Lorsqu'il y a une incompatibilité entre la forme internalisée d'un mot et l'input perçu, alors ce mot est retiré de la cohorte. Ce processus d'élagage des mots de la cohorte constitue l'étape de sélection. Finalement l'intégration consiste à vérifier que les propriétés sémantiques et syntaxiques des mots activés concordent avec le reste de la phrase, sinon, le ou les mots incompatibles sont retirés de la cohorte. Ce processus continue jusqu'à ce qu'il ne reste plus qu'un mot dans la cohorte.

On peut citer plusieurs problèmes avec ce modèle de cohorte, notamment le fait qu'il suppose que les stimuli acoustiques perçus correspondent exactement aux formes phonologiques internalisées. Or ceci n'est pas réaliste puisque les humains peuvent reconnaitre un mot même mal prononcé. De même il est possible de reconnaitre un mot bien prononcé mais sémantiquement incorrect dans le contexte phrastique, bousculant ainsi l'hypothèse de l'étape d'intégration.

Modèle TRACE

Le modèle Trace s'appuie sur le modèle Cohort en conservant ses "caractéristiques positives majeures" (McClelland & Elman 1986a), que sont l'activation simultanée, la sélection, et l'élagage interactif, tout en essayant de surmonter les problèmes qui ont été abordés ci-dessus.

L'input acoustique, contrairement au modèle de la Cohort, n'est plus perçu en termes de phonèmes, mais en termes de stimuli acoustiques. Le modèle TRACE est organisé en couches, la première traitant les stimuli acoustiques, la deuxième les phonèmes et la troisième les mots. La couche acoustique active certaines unités représentant des phonèmes dans la couche des phonèmes, qui eux-mêmes activent des unités représentant des mots dans la couche lexicale. L'activation dans ce modèle dépend de l'adéquation entre les percepts acoustiques avec les représentations internalisés, mais ne nécessite pas une adéquation parfaite ; autorisant ainsi les erreurs de prononciation et/ou de perception. Les phonèmes activent les mots dans lesquels ils apparaissent, peu importe leur position. Les unités à l'intérieur d'une même couche (phonème, ou mot) sont reliées par des connexions inhibitrices de sorte que les unités les plus activées inhibent dans une certaine mesure l'activation des autres si elles ne correspondent pas à l'entrée acoustique aussi bien que les unités les autres unités plus activées.

Contrairement au modèle Cohort, ce modèle est capable de gérer des chaînes de mots connectés. Dans ce modèle, le flux vocal est implicitement segmenté lorsque les mots sont reconnus. Ce modèle apporte une amélioration majeure par rapport au modèle Cohort : les mots peuvent être reconnus après leur fin, et les mots peuvent être activés à partir de n'importe quel point, relâchant ainsi la contrainte de correspondance exacte de l'apparition des mots que suppose le modèle de la Cohort. La plus grande amélioration apportée par le modèle Trace est la compétition active entre les mots, car les mots sont réellement en compétition les uns avec les autres grâce aux connexions inhibitrices entre les unités d'une même couche.

Shortlist

Le modèle TRACE dans son implémentation est computationnellement irréaliste. En effet, il suppose qu'un phonème donné active l'ensemble des mots dans lequel il apparait. Cela est possible pour un vocabulaire limité, mais pas pour un vocabulaire d'une taille plus élevée.

Le modèle Shortlist peut être vu comme un modèle hybride entre le modèle de la Cohort et Trace. Certes, un phonème active, comme dans TRACE, l'ensemble des mots dans lequel il apparait, mais les mots pour lesquels l'activation est trop faible ne sont plus considérés comme des cibles potentielles. Ainsi, le nombre de mots activés de manière simultanée est limité à un petit nombre (a shortlist) et la compétition inter-mots limitée à ceux de la shortlist. Par conséquent, ce modèle est plus léger en termes de calculs que le modèle Trace-car seul un sous-ensemble du lexique est considéré comme un candidat validetout en préservant la plupart de ses caractéristiques originales.

Modèle de la Cohorte distribuée

Le modèle de la cohorte distribuée (MCD, Gaskell & Marslen-Wilson 1997) constitue un changement de paradigme. Alors que les modèles précédents incorporaient explicitement plusieurs niveaux de calcul (couche de caractéristiques, couche phonétique, couche lexicale), le MCD ne le fait pas. En effet, le MCD utilise un simple réseau neuronal récurrent [START_REF] Elman | Finding Structure in Time[END_REF].

Le réseau reçoit en entrée une séquence de mesures phonétiques et est entraîné à prédire un vecteur sémantique et un vecteur phonologique. Dans ce modèle, le concept d'activation est différent des autres modèles. En effet, alors que dans les autres modèles, une valeur est attribuée à chaque mot du lexique -cette valeur représentant la force de l'activationil n'est pas possible ici d'avoir une telle valeur. Au lieu de cela, l'activation est représentée par la proximité du vecteur sémantique prédit avec celui des mots du lexique. L'activation simultanée de plusieurs mots est toujours possible, mais elle est cependant implicite. En effet, le vecteur sémantique prédit peut être considéré comme un "mélange" des représentations pertinentes. Activer un mot plus qu'un autre peut être considéré comme "modifier ce mélange" et "peut être interprété en termes de mouvement dans l'espace sémantique".

Ainsi, dans ce modèle, la notion de caractéristique acoustique, de phonème et d'unité lexicale est implicite puisque toutes les couches sont capables de représenter cette information simultanément. Cependant, dans leur expérience, les auteurs ne testent que la reconnaissance de mots individuels et non de phrases complètes. Ainsi, même si ce modèle obtient des résultats intéressants (en montrant par exemple que les mots les plus fréquents sont ceux qui sont activés prioritairement), il ne réalise aucune segmentation de l'entrée parlée.

Conclusion sur la reconnaissance de mots parlés

Tous les modèles de reconnaissance des mots supposent que la reconnaissance des mots se déroule en plusieurs étapes : une activation, une compétition et la reconnaissance finale. Néanmoins, la manière dont ces trois étapes sont réalisées varie d'un modèle à l'autre. Cependant, tous ces modèles s'accordent sur le fait que la reconnaissance de mots implique nécessairement l'activation simultanée d'un ensemble de mots candidats. Cet ensemble de mots est itérativement élagué si l'entrée acoustique est trop éloignée de la représentation internalisée, de sorte qu'il ne reste finalement qu'un seul mot.

Conclusion

Dans ce chapitre, nous avons examiné les stratégies utilisées par les enfants pour segmenter l'entrée parlée en sous-unités. Nous avons montré que les enfants utilisent un large éventail de stratégies pour trouver des mots en se servant d'indices suprasegmentaux, segmentaux et infrasegmentaux. Nous avons ensuite présenté plusieurs modèles de reconnaissance de mots qui tentent de rendre compte de la manière dont les humains sont capables d'activer et de récupérer les unités lexicales du lexique mental. Tous les modèles de reconnaissance de mots s'accordent sur le fait qu'une stratégie de reconnaissance réussie consiste à activer simultanément plusieurs unités lexicales à la fois. Enfin, nous avons montré que l'acquisition du langage est un phénomène multimodal qui implique toutes les capacités perceptives de l'enfant, et notamment que la vision fournissait à l'enfant des informations précieuses pour acquérir sa langue maternelle. Le travail effectué dans cette thèse appartient donc aux "approches sensorielles" des modèles de traitement de la parole, telles que définies par [START_REF] Glass | Towards unsupervised speech processing[END_REF]. Les modèles "sensoriels" de traitement de la parole sont des modèles qui n'ont besoin que de parole couplée à des données sensorielles pour fonctionner. En d'autres termes, ces modèles ne nécessitent pas de données annotées ni d'expertise humaine pour fonctionner et "se rapprochent de l'acquisition du langage oral humain" selon [START_REF] Glass | Towards unsupervised speech processing[END_REF].

L'une des premières approches de traitement non supervisé de la parole est celle introduite par [START_REF] Park | Towards unsupervised pattern discovery in speech[END_REF]. La méthodologie qu'ils proposent permet de découvrir des unités lexicales à partir d'un signal de parole, sans avoir besoin que celui-ci soit préalablement transcrit [START_REF] Park | Unsupervised Pattern Discovery in Speech[END_REF]. La méthodologie proposée de Segmental-DTW est basée sur la mesure du DTW (Dynamic Time Warping) qui permet de mesurer la similarité de deux séquences. Les autres modèles de traitement non supervisé de la parole, qui ont pour but d'extraire des unités lexicales ou de segmenter l'entrée parlée se base principalement sur des modèles bayésiens. [START_REF] Lee | Unsupervised Lexicon Discovery from Acoustic Input[END_REF] proposent par exemple de combiner la découverte non supervisée d'unités acoustiques avec des modèles de segmentation bayésien tels que celui de [START_REF] Goldwater | Nonparametric Bayesian Models of Lexical Acquisition[END_REF]). Plus récemment, on peut citer le travail de [START_REF] Kamper | Unsupervised neural and Bayesian models for zero-resource speech processing[END_REF] qui propose un modèle bayésien permettant de conjointement inférer des frontières dans le signal de parole tout en clusterisant les unités qui résultent de la segmentation menée.

Un fait surprenant des travaux de découverte non supervisée de termes et de lexique évoquée ci-dessus est que le repérage et la segmentation des mots sont effectués sur la base de la forme uniquement, sans qu'aucun autre indice contextuel ne soit utilisé. Le langage n'est pas un système indépendant, et ce qui lui donne sa substance est qu'il est lié au monde physique. Nous avons utilisé l'exemple des enfants aveugles pour montrer que, lorsqu'ils sont privés d'un type d'entrée sensorielle, à savoir la vision, l'acquisition du langage est considérablement affectée. Pour établir une analogie, les modèles de découverte et de segmentation des termes sont en fait entraînés comme des nourrissons privés de sens, qui ne se fondent que sur la forme. Il semble ainsi qu'une stratégie de segmentation de la parole adéquate doive inclure une composante sémantique. Dans leurs travaux, [START_REF] Bender | Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data[END_REF] soutiennent que : si la forme est complétée par des données contextuelles de quelque nature que ce soit, il est concevable que le sens puisse être appris dans la mesure où l'intention communicative est représentée dans ces données.

Par conséquent, l'ancrage des données linguistiques à des connaissances externes semble être une étape nécessaire pour accéder au sens. Pourtant, que signifie plus précisément le terme grounding ? Si l'ancrage contextuel consiste simplement à ajouter des connaissances Le grounding implique donc d'ajouter aux données linguistiques des données externes non-linguistiques qui reflètent d'une certaine manière le monde physique. Les données externes doivent également refléter dans une certaine mesure l'intention communicative des données linguistiques, comme le mentionne [START_REF] Bender | Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data[END_REF]. Cette dernière contrainte est également visible dans la définition de [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF], puisque les "actes de parole" résultent nécessairement d'une intention communicative. Par conséquent, les modèles grounded désignent une classe de modèles informatiques qui traitent une certaine forme linguistique -du texte, ou bien de la parole -en conjonction avec une autre source d'information provenant du monde physique. Le fait que deux modalités soient traitées en conjonction est nécessaire, mais pas suffisant : les deux modalités doivent se produire simultanément dans le monde physique pour que les deux soient liées. Ainsi, les modèles grounded ne traitent pas uniquement la forme, mais sont capables -ou du moins ont la capacité -de lier cette forme à des référents dans le monde physique, ou à une représentation de celui-ci.

Apprentissage automatique

L'apprentissage machine (ML) est un sous-domaine de l'informatique qui s'intéresse à la création et à "l'étude d'algorithmes informatiques qui s'améliorent automatiquement grâce à l'expérience" (Mitchell 1997, p. xv). Au lieu de concevoir un algorithme spécifique pour résoudre une tâche spécifique, le ML cherche à concevoir des algorithmes qui apprennent comment résoudre une tâche spécifique : On dit d'un programme informatique qu'il apprend de l'expérience E sur une certaine classe de tâches T et selon une mesure de performance P, si sa performance à la tâche T, telle que mesurée par P, s'améliore avec l'expérience E. (Mitchell 1997, p. 3) L'expérience E consiste en un ensemble d'exemples -appelé ensemble de données -à partir duquel le programme va apprendre. L'apprentissage se fait par un processus d'essais et d'erreurs au cours duquel le programme tente de résoudre la tâche T à l'aide des données disponibles E et modifie ses paramètres de manière à augmenter sa mesure de performance P.

L'ensemble de données utilisé pour l'apprentissage est généralement divisé en trois ensembles inégaux : l'ensemble d'apprentissage qui est le plus important (≈ 80%), l'ensemble de développement (ou de validation) (≈ 10%), et l'ensemble de test (≈ 10%). L'ensemble d'entraînement, comme son nom l'indique, est utilisé pour entraîner le modèle. L'ensemble de développement est utilisé pour tester le modèle afin de sélectionner le meilleur point de contrôle du modèle, tandis que l'ensemble de test n'est utilisé qu'une fois le meilleur modèle sélectionné pour rapporter le score final.

Réseaux neuronaux artificiels

Les réseaux de neurones artificiels (ANN) sont un type d'algorithmes d'apprentissage qui appartiennent à l'approche connexionniste de l'intelligence artificielle (IA). L'idée centrale du connexionnisme est qu'un comportement complexe peut être approximé par un calcul complexe, qui peut à son tour être décomposé en calculs plus petits et plus simples réalisés par des unités de traitement individuelles. L'interconnexion de ces unités (et donc du calcul) donne naissance au comportement global.

Ces unités de traitement, appelées à l'origine perceptrons, sont maintenant communément appelées neurones. Un neurone reçoit n entrées X = x 0 , x 1 , ..., x n à partir desquelles une sortie ŷ est calculée. Étant donné les caractéristiques d'entrée, le neurone est entraîné à prédire une sortie ŷ qui doit être aussi proche que possible de la sortie souhaitée y. Pour ce faire, le vecteur d'entrée est pondéré par un ensemble de poids W = w 0 , w 1 , ..., w n qui sont des paramètres entraînables, et la sortie est ensuite transformée par une fonction d'activation non linéaire φ. Un neurone de biais peut également être ajouté. Le calcul final d'un seul neurone peut donc être résumé comme suit :

ŷ = φ(W • x + b) = φ( n i=1 w i x i + b) (D.1)
Les problèmes du monde réel sont trop complexes pour être résolus par un seul neurone. Ainsi, un nombre arbitraire de neurones peut être utilisé en conjonction afin d'apprendre des fonctions de transformation complexes. Ces neurones sont regroupés en couches et forment ce que l'on appelle communément un perceptron multicouche (MultiLayer Perceptron, MLP). La première couche est appelée la couche d'entrée, la dernière couche couche de sortie, et la ou les couches intermédiaires sont appelées les couches cachées.

Réseaux de neurones récurrents et unités récurrente à portes

Les réseaux neuronaux récurrents (RNN) sont des réseaux neuronaux capables de traiter des données séquentielles [START_REF] Elman | Finding Structure in Time[END_REF]. Les RNN peuvent être considérés comme des réseaux feed-forward complétés par des connexions de rétroaction. Les RNN peuvent être formalisés comme suit :

h t = φ(W x t + U h t-1 + b) (D.2)
où h t est l'état caché au pas de temps t, x t est l'entrée actuelle, h t-1 est l'état caché précédent, b est un terme de biais, φ est une fonction d'activation non linéaire (généralement sigmoïde ou tangente hyperbolique), et où W , U et b sont des paramètres entraînables. Notez que la seule différence avec une couche feed-forward est simplement le terme supplémentaire U h t-1 : la sortie à un pas de temps t ne dépend pas seulement de l'entrée courante mais aussi de la sortie au pas de temps précédent t -1. Ainsi, un tel calcul permet de modéliser la dépendance temporelle qui existe entre des vecteurs consécutifs. La sortie d'un RNN consiste en une séquence T de vecteurs. Le vecteur final de la séquence (au pas de temps T ) peut être considéré comme la représentation compacte de toute la séquence d'entrée, car le vecteur final dépend du calcul de tous les vecteurs précédents de la séquence. Lorsque le premier élément de la séquence x 1 est traité, h t-1 n'existe pas. Dans ce cas, cet état caché précédent, noté h 0 , est défini comme un vecteur de 0, bien que dans certains cas, l'état initial puisse également être un paramètre entraînable du réseau.

Informations générales sur le traitement automatique de la parole et la découverte d'unités lexicales 181

Même si les RNN sont capables de modéliser des données séquentielles et de suivre les dépendances locales relativement bien, ils sont incapables de capturer les dépendances à long terme dans la séquence d'entrée. En effet, à mesure que la taille de la séquence T s'allonge, le vecteur h t contient de moins en moins d'informations sur le début de la séquence. L'information sur le passé s'estompe à mesure que le vecteur caché précédent h t-1 est combiné à chaque pas de temps avec de nouvelles informations. Les unités à porte ont été introduites pour résoudre ce problème. [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF] a introduit les Gated Recurrent Units (GRU) qui sont des cellules récurrentes à ports (gated) qui sont également capables de garder la trace des dépendances à long terme et à court terme. Un GRU est formellement défini comme suit :

z t = σ(W z x t + U z h t-1 + b z ) r t = σ(W r x t + U r h t-1 + b r ) ĥt = tanh(W h x t + U h (r t * h t-1 ) + b h ) h t = (1 -z t ) * h t-1 + z t * ĥt (D.3) où W z , U z , W r , U r , W h , U h sont des matrices entraînables, b z , b r , b h sont
des termes de biais, et σ représente la fonction d'activation sigmoïde. Un GRU possède deux portes : une porte de mise à jour z, et une porte de réinitialisation r qui calculent toutes deux indépendamment une valeur scalaire à chaque pas de temps t (z t et r t ∈ [0, 1]). Compte tenu de l'entrée courante x t , la valeur de z t représente un rapport entre la quantité d'informations à reporter depuis le pas de temps précédent [(1 -z t ) * h t-1 ] et la quantité de nouvelles informations à intégrer [z t * ĥt ]. Cette porte permet de contrôler les dépendances à long terme. r t représente le degré de dépendance du nouvel état par rapport au pas de temps précédent [r t * h t-1 ], ce qui permet de contrôler les dépendances locales.

Mécanisme d'attention

Même en utilisant des cellules récurrentes bidirectionnelles, l'encodage de la signification d'une séquence entière dans un seul vecteur reste une tâche difficile. Les mécanismes d'attention [START_REF] Bahdanau | Neural Machine Translation by Jointly Learning to Align and Translate[END_REF] ont été introduits pour résoudre ce problème. L'intuition de tels mécanismes d'attention est simple : au lieu de conserver le dernier vecteur de la séquence -qui devrait encoder le sens de la phrase entière -tous les vecteurs qui ont été calculés à chaque pas de temps sont conservés et un poids est attribué à chacun d'entre eux de façon à donner plus d'importance à certains vecteurs qu'à d'autres.

Les poids sont calculés par le réseau lui-même et appris au moment de l'entraînement. Ainsi, le réseau apprend quel(s) vecteur(s) de la séquence d'entrée doit (vent) Comme la sortie ŷ dépend directement de la sortie du réseau, qui dépend elle-même des paramètres du réseau θ, il est possible de les modifier afin que la sortie prédite soit plus proche de la sortie souhaitée. Cette opération se fait par une opération appelée descente de gradient en utilisant l'algorithme de rétro-propagation du gradient d'erreur (backpropagation, [START_REF] Rumelhart | Learning representations by back-propagating errors[END_REF]. Cette opération consiste à calculer la dérivée de la fonction de coût par rapport aux paramètres du réseau. Intuitivement, cela mesure la responsabilité d'un poids donné dans l'écart entre la sortie prédite et la sortie attendue :

θ t+1 = θ t -η dL(y, ŷ) dθ (D.6)
où les poids à θ t+1 sont mis à jour par un facteur η qui est appelé le taux d'apprentissage. Le taux d'apprentissage contrôle la force des mises à jour des poids du réseau. S'il est trop élevé, le réseau risque de rater la solution optimale, tandis qu'un taux d'apprentissage trop faible ralentira l'apprentissage et risque en outre de bloquer le réseau dans un optimum local. On utilise généralement la descente de gradient par lots, où la perte est moyennée sur un lot (batch) d'exemples, c'est-à-dire sur plusieurs exemples échantillonnés de manière aléatoire dans l'ensemble d'apprentissage.

Modèle de parole visuellement contextualisée

Modèle CELL

Le premier modèle de parole visuellement contextualisée (PVC, en anglais Visually Grounded Speech Models ou VGS models) est le modèle CELL (Cross-channel Early Lexical Learning) développé par [START_REF] Roy | Learning words from sights and sounds: a computational model[END_REF] et [START_REF] Roy | Grounded spoken language acquisition: experiments in word learning[END_REF]. Ce modèle a été explicitement développé afin de comprendre comment l'interaction des stimuli visuels et auditifs permet l'acquisition lexicale. Le but du modèle CELL est d'apprendre des appariements audiovisuels (appelés prototypes audio-visuels) entre divers objets et leurs formes verbales afin de constituer un proto-lexique comme le ferait un enfant. Afin d'apprendre des prototypes audiovisuels, le modèle CELL reçoit en entrée un contexte visuel (image) et son énoncé apparié. Le modèle CELL recherche des séquences audio récurrentes qui sont stockées avec l'objet qui était manipulé lorsque la séquence a été extraite. Les paires objet/son très fréquentes sont conservées puis ensuite filtrées selon un critère d'information mutuelle afin d'enlever les paires de mauvaise qualité (par exemple les paires objets/déterminant, objet/pronom, qui sont certes fréquentes mais non discriminantes).

Informations générales sur le traitement automatique de la parole et la découverte d'unités lexicales 183

Les résultats montrent que le modèle CELL est capable d'apprendre des paires audiovisuelle (AV) sémantiquement valides (par exemple, chaussure, clef, chien, toutou) et indique une précision sémantique de 85% des paires découvertes. Certains des éléments AV contiennent également des "mots" non standard -qui ne sont pas comptés comme des paires sémantiquement exactes -tels que des sons onomatopéiques (par exemple aboiement, bruit de moteur), bien que de telles paires soit exactes d'un point de vue de l'acquisition du langage, le vocabulaire des enfants étant aussi constitué d'onomatopées.

Ce modèle a cependant fait plusieurs hypothèses simplificatrices, notamment le fait que la parole est perçue de manière catégorielle en termes de phonèmes et les images devaient être prétraitées de manière à détacher d'abord l'arrière-plan du premier plan, puis à isoler l'objet de l'image. En outre, l'entrée visuelle était délibérément simplifiée de manière à ne contenir qu'un seul objet, ce qui facilitait la tâche du modèle. Cependant, le modèle CELL constitue l'un des tout premier modèle computationnel d'acquisition du langage en contexte, et est le précurseur des modèles neuronaux actuels.

Modèles neuronaux

Les réseaux neuronaux ont permis aux chercheurs de modéliser des interactions encore plus complexes entre les modalités visuelles et orales. [START_REF] Gabriel | Learning Words from Images and Speech[END_REF] a introduit, à notre connaissance, le premier modèle PVC neuronal (basé sur un CNN), dans lequel le modèle est entraîné à faire correspondre des images à des mots parlés isolés. Leur modèle comporte deux branches, une branche acoustique et une branche visuelle qui vectorise respectivement le mot d'entrée et l'image. Le modèle est entraîné à minimiser la distance cosinus entre les vecteurs correspondants, tout en augmentant cette distance pour les vecteurs non correspondants (approche contrastive). Leur modèle est ensuite évalué sur une tâche de recherche image/parole : retrouver l'image correspondante à partir d'un mot en entrée, ou vice-versa. Leurs résultats indiquent que leur réseau est effectivement capable de capturer les liens intermodaux et d'apprendre efficacement à associer un mot parlé à son contexte visuel. Leur modèle a ouvert la voie à des modèles traitant des stimuli acoustiques plus complexes, tels que des légendes complètes au lieu de mots isolés et a posé les bases de l'ensemble des modèles neuronaux utilisés aujourd'hui, par l'utilisation d'une fonction de coût contrastive notamment. [START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF], s'appuyant sur les travaux de Gabriel et al. (2014), ont proposé le premier modèle capable de gérer des légendes complètes au lieu de mots isolés. Leur modèle peut être considéré comme une version améliorée du modèle CELL car il fait également quelques hypothèses simplificatrices (les légendes sont segmentées en mots et les images pré-traitées afin d'en extraire les objets). L'objectif du réseau est d'aligner chaque mot de la légende audio sur chaque boîte de délimitation de l'image. [START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF] ont amélioré leurs travaux précédents de sorte que leur modèle puisse utiliser des légendes complètes au lieu de légendes pré-segmentées. De plus, au lieu d'exécuter un détecteur d'objets sur l'image, ils utilisent simplement l'avant-dernière activation d'un réseau VGG pré-entraîné. Finalement, [START_REF] Harwath | Jointly Discovering Visual Objects and Spoken Words from Raw Sensory Input[END_REF] changent complètement leur modèle afin d'avoir un modèle entièrement convolutif : au lieu d'utiliser des vecteurs VGG, ils utilisent le réseau VGG complet jusqu'à la dernière couche convolutive (c'est-à-dire en supprimant les couches entièrement connectées), et utilisent des convolutions 1D pour la légende d'entrée. Afin de joindre les résultats des deux branches, ils utilisent un produit scalaire entre la matrice représentant l'image et celle représentant le signal de parole. Le fait de conserver les feature maps des convolutions des deux branches leur permet ainsi de détecter les zones de saillance entre l'image convoluée et l'entrée parlée convoluée avec un niveau de précision très fin. Leur modèle est en mesure de mettre en évidence des régions spécifiques du spectrogramme qui correspondent à des régions spécifiques de l'image, ce qui leur permet de construire des paires image/audio de manière non supervisée.

Dans leur travail, Harwath et collègues sont donc passés de la recherche de paires audiovisuelles à partir d'images pré-segmentées et de légendes pré-segmentées à la recherche de paires audio-visuelles à un fin niveau de granularité sans aucune supervision à partir de parole brute appariée à des images brutes. Globalement, leur travail montre qu'il est possible d'extraire des unités de type mot à partir de la parole brute en utilisant des images comme une forme de supervision faible. Plus important encore, alors que leur réseau a été entraîné à minimiser la distance entre une image et sa légende correspondante à une échelle globale, le réseau a déduit des similarités à une échelle locale. Par conséquent, l'acquisition lexicale semble être un sous-produit de la tâche principale, et apparaît "naturellement" dans le modèle. [START_REF] Alishahi | Encoding of phonology in a recurrent neural model of grounded speech[END_REF], leur étude révèle que les couches inférieures du réseau encode des représentations liées plus à la forme qu'à la sémantique.

Jeux de données

Étant donné que le nombre d'ensembles de données librement disponibles qui comportent à la fois des images et des descriptions parlées est limité, la plupart des modèles mentionnés ci-dessus sont entraînés sur des extensions d'ensembles de données initialement conçus pour entraîner des modèles qui génèrent des descriptions textuelles d'image. Les principaux ensembles de données utilisés pour cette tâche sont Flickr8k [START_REF] Rashtchian | Collecting Image Annotations Using Amazon's Mechanical Turk[END_REF][START_REF] Hodosh | Framing Image Description as a Ranking Task: Data, Models and Evaluation Metrics[END_REF] et MSCOCO [START_REF] Lin | Microsoft COCO: Common Objects in Context[END_REF]) qui présentent des images associées à 5 légendes descriptives écrites par des humains. Ces ensembles de données présentent naturellement un langage contextualisé, car chaque légende descriptive est associée à une image, ce qui constitue, dans une certaine mesure, une connaissance du monde physique. Comme les légendes ont été écrites par des annotateurs après avoir vu l'image, l'image reflète naturellement l'intention communicative exprimée dans les légendes.

Cependant, ces données sont artificielles, dans le sens où elles ne reflètent pas véritablement ce à quoi un enfant est confronté. De récents résultats, utilisant des jeux de données réalistes (voir Tsutsui et al. 2020[START_REF] Räsänen | A computational Model of Early Language Acquisition from Audiovisual Experiences of Young Infants[END_REF]) montrent que leur modèle respectif peine à apprendre. Cela souligne la nécessité de disposer de jeux de données plus réalistes si l'on veut utiliser des simulations informatiques pour comprendre l'acquisition du langage chez l'enfant, comme le préconise Dupoux (2018).

Simulation ou modélisation de l'acquisition du langage ?

La plupart des modèles susmentionnés s'inspirent de l'acquisition du langage chez l'enfant, ou du moins établissent un lien explicite entre le but ultime de leur modèle et l'acquisition du langage chez l'enfant. Il semble raisonnable de se demander dans quelle mesure ces modèles modélisent l'acquisition du langage chez les enfants ou simulent l'acquisition du langage chez les enfants.

La différence entre simulation et modélisation ne concerne que les deux premiers niveaux de la hiérarchie de Marr, à savoir le niveau computationnel , et le niveau algorithmique ; le premier s'intéressant au problème à résoudre et à l'objet du calcul, tandis que le second s'intéresse à la manière dont le calcul est mené et définit chaque étape du calcul précisément. Rieder (2003, p. 818) fait une différence entre simulation et modélisation. Pour lui, la "simulation traduit l'action d'imiter la réalité et [le] modèle représentant le véhicule". La simulation est donc "l'action de présenter l'apparence, ou d'interagir avec le comportement, d'un système sans la réalité", tandis que la modélisation serait "la génération d'un fac-similé ou d'une représentation du système réel", ce dernier pouvant être "physique, mathématique, procédural [c'est-à-dire algorithmique], ou une combinaison de ces éléments".

Nous pensons que si l'on souhaite modéliser l'acquisition du langage chez l'enfant, le modèle doit être aussi proche que possible de ce que font les humains sur le plan algorithmique, et par exemple incorporer les mêmes priors que les humains, ce qui n'est le cas d'aucun des modèles PVC susmentionnés. Par conséquent, même s'ils simulent dans une certaine mesure l'acquisition du langage chez l'enfant, et plus particulièrement l'acquisition lexicale, ils ne modélisent pas l'acquisition du langage chez l'enfant. Néanmoins, le fait de ne pas implémenter le même "véhicule" que les humains n'empêche pas ces implémentations d'afficher des résultats similaires dans la simulation finale, ce qui rend leur implémentation et leur étude intéressantes pour tester des hypothèses.

Simulation parfaite et grounding

Selon Dupoux (2018) (et d'autres), une approche réussie pour étudier l'acquisition du langage chez l'enfant à l'aide d'approches computationnelles devrait satisfaire aux contraintes suivantes : construire des systèmes informatiques évolutifs capables, lorsqu'ils sont alimentés par des données d'entrée réalistes, d'imiter l'acquisition du langage telle qu'elle est observée chez les enfants.Il faut [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF] Si l'on veut imiter l'acquisition du langage telle qu'elle est observée chez les nourrissons -ce qui serait ce que nous définissons comme une simulation parfaite -les modèles susmentionnés ne sont pas entièrement adaptés. En effet, ces modèles n'intègrent que les capacités perceptives et non les capacités productives d'un enfant. Ainsi, pour [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF] le modèle idéal devrait être à la fois capable de perception et de production afin d'imiter réellement l'acquisition du langage chez l'enfant et ayant des boucles de rétroactions. Aucun des modèles que nous avons présentés ne met en oeuvre toutes les boucles de rétroaction nécessaires pour obtenir un modèle totalement grounded au sens de [START_REF] Roy | Semiotic schemas: A framework for grounding language in action and perception[END_REF]. Cependant, bien qu'ils n'intègrent pas tous les aspects de l'acquisition du langage chez l'enfant, ils en intègrent un sous-ensemble, à savoir l'acquisition lexicale. Ce processus implique en effet de construire des schémas sur le monde (c'est-à-dire des représentations neuronales) en utilisant une entrée linguistique (c'est-à-dire des sous-titres audio) et une entrée non linguistique (c'est-à-dire un contexte visuel sous forme d'images fixes). Ainsi, ces modèles se prêtent à la simulation de l'acquisition lexicale chez l'enfant. • Les modèles basés sur les RNN mettent-ils en évidence des parties spécifiques de l'entrée parlée qui sont particulièrement pertinentes pour prédire l'image cible, comme cela a été montré pour les modèles basés sur les CNN ? [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] • Si oui, quelles parties spécifiques de l'entrée sont mises en évidence ?

• S'ils mettent effectivement en évidence des parties spécifiques de l'entrée, cette capacité est-elle valable d'une langue à l'autre, même si le modèle est entraîné avec une langue typologiquement différente de l'anglais, par exemple le japonais ?

• Dans quelle mesure ces capacités linguistiques se développent-elles avec le temps ? Les modèles PVC sont-ils capables d'acquérir rapidement certaines capacités linguistiques (c'est-à-dire avec une quantité de données relativement faible) ou non ? [START_REF] Karpathy | Deep Visual-Semantic Alignments for Generating Image Descriptions[END_REF].

FLICKR8k Enfin, le dernier jeu de données est Flickr8k (Harwath & Glass 2015) qui contient de la parole naturelle : les descriptions ont été enregistrées par plus de 180 locuteurs. Ce corpus est le plus petit et ne contient que 8 000 images pour 40 000 descriptions. Les descriptions orales sont des verbatims des descriptions écrites, qui ont simplement été lues par des locuteurs natifs de l'anglais sur Amazon Mechanical Turk. 30 000 paires images/audio sont utilisées pour l'entraînement, 5 000 pour la validation et enfin 5000 pour le test.

Métadonnées Nous avons procédé à un alignement forcé des signaux de parole avec le texte pour chacun des trois corpus afin d'avoir des informations de frontières précises pour chacun des mots des descriptions. Cet alignement a été fait au moyen de l'outil en ligne Maus Forced Aligner [START_REF] Kisler | Multilingual processing of speech via web services[END_REF]. De plus, nous avons également procédé à une annotation en partie du discours. Pour les corpus en anglais, cette annotation a été faite en utilisant TreeTagger [START_REF] Schmid | Probabilistic Part-of-Speech Tagging Using Decision Trees[END_REF], et pour le japonais, celle-ci a été faite en utilisant KyTea. Puisque l'annotation faite par KyTea [START_REF] Neubig | Pointwise Prediction for Robust, Adaptable Japanese Morphological Analysis[END_REF]) était plus proche d'une annotation morphologique que morphosyntaxique, nous avons pris le parti d'éditer les annotations en fusionnant notamment certaines étiquettes entre elles (morphèmes "TAIL" et "SUF" notamment). Afin d'avoir des étiquettes comparables entre anglais et français, nous avons porté les étiquettes de chacun des deux étiqueteurs vers leur équivalent Universal POS [START_REF] Petrov | A universal Part-of-Speech Tagset[END_REF]). Encodeurs L'encodeur d'image consiste en une simple couche linéaire, chargée de réduire la dimension du vecteur d'image d'entrée. Ainsi, le vecteur VGG (Simonyan & Zisserman 2015) représentant l'image, extrait à partir d'un réseau de vision par ordinateur, se voit réduit pour avoir la même dimension que le vecteur représentant le signal acoustique. Concernant l'encodeur de parole, celui-ci prend en entrée des vecteurs MFCC qui sont ensuite passés à une couche convolutionnelle 1D. Cette couche convolutionnelle est suivie par 5 couches récurrentes composées de GRU (Gated Recurrent Units, [START_REF] Cho | Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation[END_REF]. Contrairement à l'implémentation de base qui utilise des cellules RHN [START_REF] Zilly | l'utilisation conjointe de syllabes et de mots)[END_REF], nous avons fait le choix d'utiliser des GRU car ceux-ci sont mieux documentés dans la littérature. Nous utilisons deux mécanismes d'attention, l'un produisant un vecteur à partir des vecteurs de la première couche récurrente, et un second produisant un vecteur à partir des vecteurs de la cinquième couche récurrente. Le vecteur final de l'encodeur de parole correspond au produit terme à terme de chacun des deux vecteurs d'attention. Pour finir, le vecteur de parole et le vecteur d'image sont normalisés à la norme L2.

Architecture

Fonction de coût La fonction de coût que nous utilisons est la fonction de coût contrastive (contrastive loss, également appelée hinge loss ou triplet loss Weinberger & Saul 2009):

L(u, i, α) = u,i u max[0, α + d( u, i) -d( u , i)] + i max[0, α + d( u, i) -d( u, i )]
(D.7) où u est un vecteur représentant l'énoncé, i un vecteur représentant l'image, u et i sont des énoncés (respectivement des images) ne formant pas une paire avec l'image i (respectivement l'énoncé u). Cette fonction de coût encourage le réseau à minimiser la distance d( u, i) entre l'image i et l'énoncé u appartenant à des paires image/énoncé concordantes pour que celle-ci soit inférieure à une marge α, tout en augmentant la distance pour les paires image/énoncé non concordantes afin qu'elles soient éloignées d'une distance au moins égale à α dans l'espace de représentation. Le coût est calculé au niveau du batch, c'est-à-dire que toutes les images à l'intérieur d'un batch (à l'exception de l'image i) sont considérées comme des exemples d'inadéquation contrastive pour l'énoncé u, tandis que tous les autres énoncés (à l'exception de l'image u) à l'intérieur du même batch servent d'exemple d'inadéquation contrastive pour l'image i. Présupposés et Biais Nous pouvons classer les hypothèses et biais que nous faisons implicitement en trois catégories : hypothèses sur la nature des données, hypothèses sur la tâche, et enfin hypothèses la nature des opérations computationnelles.

c = T t=1 α GRU 1 t h GRU 1 t • T t=1 α GRU 5 t h GRU 5 t (D.
De par la nature même des images d'entrée, le contexte visuel fourni au modèle sera nécessairement limité. En effet, nous ne percevons pas en tant qu'humain des images fixes, mais des séquences d'images. De même, les images des corpus ont été soigneusement sélectionnées afin de mettre en avant des objets particuliers, et ainsi ne reflètent que peu le contexte visuel qu'un enfant pourrait percevoir, et notamment ne sont pas appropriées pour dépeindre des actions. Aussi, les descriptions liées aux images traduisent cela, et ne décrivent que peu les éventuelles actions qui ont lieu dans les images. De plus, en choisissant de telles données, nous faisons implicitement l'hypothèse que l'enfant entendrait systématiquement une description de son environnement. Cela est évidemment loin d'être le cas. Il arrive en effet très souvent que l'on évoque des personnes ou des objets qui ne sont pas physiquement présents.

Nous faisons également des hypothèses la tâche. En effet, le réseau est entraîné à résoudre une tâche très spécifique : retrouver une image à partir d'une description orale. Cela est une tache peu réaliste et ne correspond pas à ce que font des humains qui apprennent leur langue maternelle. Cependant, cette tâche met en jeu une capacité similaire à celle des humains, à savoir apprendre à faire un appariement entre un percept visuel et un percept acoustique. De plus, les humains ne se limitent pas à une seule modalité, mais utilisent de nombreuses autres modalités (odorat, toucher, proprioception, etc.), rendant ainsi la tâche que fait le réseau très spécifique et peu réaliste.

Enfin, nous faisons de nombreuses hypothèses sur la nature des opérations faites. Notre réseau traite les images et les signaux de paroles de manière indépendante, et ainsi la perception visuelle n'influe pas sur la perception auditive. Ce n'est pas le cas chez l'humain qui traite conjointement les deux modalités et où la perception d'une modalité peut influer sur la manière dont est perçue l'autre. Ainsi, il n'y a au sein de notre réseau pas de véritable interaction entre les deux modalités.

Ainsi, de par la nature des données que nous utilisons, de la tâche pour laquelle est entraîné le réseau et de la nature des opérations faites, les comparaisons que nous ferons avec l'acquisition du langage chez l'humain ne pourront être que limitées. Aussi, les conclusions que nous tirerons doivent être comprises à la lumière des hypothèses et des limites présentées : si la vision -sous la forme d'images fixes -devait être la seule modalité utilisée pour donner un sens à la parole environnante, et si la parole environnante se référait toujours au contexte visuel, quelles régularités devrions-nous attendre des enfants ?

Mesure de l'attention

Dans ce chapitre, nous étudions les parties de l'entrée parlée auxquelles le modèle prête attention en analysant les poids d'attention. Nos analyses se concentrent sur les points suivants :

• Quels parties du discours (POS) sont mises en évidence ;

• Quels mots sont mis en évidence ;

• Quelles parties d'un mot donné sont plus spécifiquement mises en évidence ;

• Comment la distribution des poids d'attention évolue-t-elle dans le temps.

Après avoir entraîné les modèles sur l'ensemble de données anglais ou japonais, nous encodons chaque légende de l'ensemble de test et extrayons les poids d'attention α pour les deux mécanismes d'attention. Nous utilisons ensuite un algorithme de détection de pics pour détecter les maxima locaux dans les poids d'attention. Nous dirons que l'attention se focalise sur ou met en évidence une unité spécifique (POS, mot, etc.) s'il existe un maximum local parmi les poids d'attention attribués aux vecteurs qui composent cette unité. Pour des raisons de brièveté, à partir de maintenant, les poids d'attention calculés par le mécanisme d'attention suivant la première couche et la cinquième couche de GRU seront appelés respectivement "GRU1" et "GRU5".

Attention dans un modèle de parole visuellement contextualisée 193

Afin de comprendre si les unités mises en évidence sont différentes de ce que le hasard pourrait prédire, nous utilisons des pics aléatoires comme base de référence. Pour chaque légende, nous échantillonnons des pics aléatoires et calculons la distribution des mots et des POS sous ces pics aléatoires et regardons dans quelle mesure celle-ci est différente du hasard.

Résultat sur les corpus synthétiques

Résultats de référence

Afin de nous assurer que nos résultats ne sont pas dus au hasard, nous avons entraîné cinq modèles, chacun avec une graine différente. Les résultats présentés dans le tableau 4.1 sont une moyenne (± écart-type) des résultats obtenus pour les cinq modèles. Les modèles sont évalués en termes de rappel@k (R@k) sur une tâche de recherche d'images à partir d'une requête vocale. Pour chaque requête vocale, les images sont classées de l'image la plus proche à l'image la moins proche. Le R@k évalue si l'image cible (c'est-à-dire l'image réellement appariée avec la légende de la requête) est classée parmi les k premières images. Nous indiquons également le rang médian r qui nous informe sur le rang moyen des images réelles. Nos résultats (Table 4.1) révèlent que le réseau est parvenu à apprendre un appariement correct dans les deux langues. Cependant, nos résultats en anglais sont moins bons que ceux de Chrupa la et al. (2017a) (R@1 de 5.5 contre 11.1) ce qui s'explique par le changement des cellules RHN par des GRU. Cependant, nos résultats sont bien meilleurs que le hasard (R@1 de 0.02).

Attention aléatoire

Pour vérifier la pertinence des poids d'attention nous avons soit mélangé les poids des deux mécanismes d'attention en même temps, soit mélangé alternativement l'un des deux afin d'estimer la contribution de chacun dans la prédiction finale. Nous l'avons fait sur le meilleur modèle entraîné (sélectionné sur l'ensemble de validation) de chacun des cinq entraînements pour COCO et STAIR (voir Table 4.2).

Lorsque les poids d'attention des deux mécanismes d'attention sont mélangés, nous observons que le R@1 est à peine supérieur à 0%, ce qui montre que le réseau est à peine capable de trouver l'image correcte à partir d'une requête orale. Lorsque nous mélangeons les poids d'attention de GRU1 mais laissons ceux de GRU5 intacts, nous remarquons que le réseau obtient de meilleurs résultats que lorsque nous faisons l'inverse. Dans les deux cas, les résultats sont moins bons que lorsque nous laissons les poids d'attention des deux mécanismes d'attention intacts (voir Tableau 4.1), mais meilleurs que lorsque les deux sont mélangés. Cela montre que les poids d'attention de GRU5 sont plus importants que ceux de GRU1, car le remaniement des poids du premier a un impact négatif beaucoup plus important que le remaniement des poids du second.

Ainsi, ces expériences nous permettent de conclure que les deux mécanismes d'attention mettent en évidence des unités utiles pour les prédictions des réseaux et que, par conséquent, l'analyse des poids d'attention est une entreprise légitime.

Parties du discours (POS) et mots

Pour COCO (Figure 4.2), nous observons que GRU5 se concentre fortement sur les noms (85.89% ± 0.38 des pics) et à peine sur les autres POS, et ce d'une façon très différente de ce que le hasard prédirait (47.15% ± 0.09). Le comportement de GRU1 est différent et semble beaucoup plus proche d'un comportement aléatoire : 52.37% ± 29.46 des pics sont situés au-dessus des noms alors que l'aléatoire prédirait 46.92% ± 0.15. Cependant, ce résultat est dû à un entraînement fautif (d'où l'écart-type important) où l'attention était concentrée à la fin des légendes.

Pour STAIR (Figure 4.3), les deux mécanismes d'attention ont appris à se concentrer sur les noms : 71.33%±5.94 pour GRU1 (Figure 4.3a) et 63.30%±2.76 pour GRU5 (Figure 4.3b) alors que le hasard prédirait 45%. Nous remarquons que GRU5 se concentre principalement sur les particules 28.63% ± 2.15 alors que l'aléatoire prédirait 15.88% ± 0.07 (en japonais, les particules sont des clitiques qui indiquent notamment la fonction du mot précédent dans la phrase. Par exemple, "は" /wa/ marque le topique, et "が" /ga/ marque le sujet), tandis que GRU1 se concentre principalement sur les noms.

L'étude des POS mis en évidence par les mécanismes d'attention des modèles anglais et japonais révèle que les modèles ont adopté un comportement spécifique en fonction de la langue des légendes. En anglais, les noms sont les POS les plus mis en avant car ils font référence aux objets présents dans l'image. En japonais, cependant, les modèles ont tiré parti du fonctionnement de la langue en ne mettant pas seulement en évidence les noms, mais en adoptant un comportement spécifique à la langue lors de la mise en évidence des particules.

Pour COCO (Tableaux B.1 et B.2), les mots les plus mis en évidence sont des noms faisant référence à des objets concrets dans les images (train, tennis, toilettes, baseball, etc.). Pour STAIR (Tableaux B.3 et B.4), on remarque que sur les 10 premiers mots mis en évidence, respectivement 3 et 4 des mots pour GRU1 et GRU5 sont des particules. ("ga" (marqueur sujet), "no" (marqueur génitif) et "o" (marqueur objet) et aussi "ni" (marqueur locatif) pour GRU5) les autres mots étant des noms. La stratégie du réseau de mettre en évidence les particules est très intéressante car, en raison de la nature unidirectionnelle des unités récurrentes utilisées ; et parce que les particules sont des mots suffixés, les vecteurs qui constituent une particule contiennent beaucoup d'informations concernant le mot précédent. Par conséquent, la mise en évidence des particules est la stratégie optimale.

Position des pics

Nous analysons spécifiquement où les pics d'attention sont situés au-dessus des mots. Pour ce faire, nous avons divisé chaque mot situé sous un pic en quatre parties égales et nous comptons le pourcentage de pics situés au-dessus d'une partie donnée. Les résultats sont présentés dans le tableau 4.3.

Pour les modèles entraînés sur l'anglais, les pics d'attention ne sont pas situés précisément à la fin d'un mot, mais sont plutôt situés entre le milieu et la fin des mots mis en évidence. Cela semble indiquer que le réseau ne se concentre pas sur la représentation du mot entier, mais plutôt sur les vecteurs représentant la première moitié ou les deux tiers d'un mot. Pour le japonais, nous observons un comportement similaire puisque les pics de GRU1 et GRU5 sont globalement situés au-dessus des fins de mots. Cependant, la distribution des pics a tendance à être plus uniforme. Nous expliquons cela par le fait que les pics d'attention sur les légendes japonaises sont situés au tout début de la particule ou à la frontière avec le mot précédent (voir les deux pics de la figure 4.1b situés au début des particules "ni" et "ga"). Ainsi, la distribution des pics d'attention au-dessus d'une partie donnée du mot tend à être plus uniforme.

Étude longitudinale

Nous avons régulièrement sauvegardé les modèles pendant l'entraînement afin d'avoir une vision longitudinale de la façon dont l'apprentissage se déroule et avons donc sauvegardé 5. Attention dans un modèle de parole visuellement contextualisée 195 les poids des modèles chaque fois que le coût diminuait de 4% de sa valeur initiale. Nous avons sauvegardé également le modèle à chaque époque.

Nous remarquons d'abord que pour COCO (4.4a) et STAIR (4.4b), le nombre d'étapes de sauvegarde appartenant à la première époque représente une grande partie du graphique, ce qui montre que le coût a chuté assez rapidement et que la phase d'apprentissage la plus importante est en fait concentrée dans la première époque. Pour COCO, nous remarquons une évolution claire dans la proportion de pics mettant en évidence des noms où les pics au-dessus des noms sont passés de 37,4% à 55,9% après seulement 8 batches (256 exemples). Cela montre qu'il est possible pour le réseau de se concentrer sur des parties très spécifiques de l'entrée parlée avec très peu d'exemples. À la fin de la première époque, la proportion de pics au-dessus des noms n'évolue pas beaucoup et reste stable, oscillant autour de 85%. Pour STAIR, nous remarquons que dans les premières étapes de la première époque, il y a une compétition entre les POS, l'attention mettant en évidence simultanément les noms, les verbes et les particules. Puis à la fin de la première epoch, le nombre de pics mettant en évidence les particules dépasse celui des verbes, pour ne plus mettre en avant que des noms ou des particules à la fin de l'apprentissage. Contrairement à COCO, où l'attention n'évolue plus après la première epoch, on constate pour STAIR que l'attention et les unités sur lesquelles elle se focalise évolue tout au long de l'entraînement. 2017a) qui rapportent un R@1 de 5,5 alors que le nôtre n'est que de 2,08. Les résultats que nous obtenons sont également moins bons que ceux obtenus sur les ensembles de données COCO et STAIR. Néanmoins, même si les résultats ne sont pas particulièrement bons, ils sont toujours bien meilleurs que le hasard, ce qui montre que le réseau a été capable de donner un sens aux données et a appris une correspondance parole-image fiable.

Résultat sur le corpus de parole naturelle

Comme pour les expériences précédentes, nous avons mélangé les poids d'attention afin de comprendre s'ils étaient significatifs ou non. Les résultats sont présentés dans le tableau 4.5. Ici aussi, nous remarquons qu'en mélangeant aléatoirement les poids d'attention des deux mécanismes d'attention, nous obtenons un R@1 beaucoup plus faible que le R@1 original (-1.54pp). Une fois encore, cela montre que l'attention joue un rôle essentiel pour les modèles. Cependant, contrairement aux modèles COCO et STAIR, la majorité des prédictions des modèles reposent sur le vecteur contextuel calculé par le premier mécanisme d'attention plutôt que sur celui du cinquième.

Parties du discours (POS) et mots

Comme pour COCO, nous remarquons que les modèles ont appris à se concentrer principalement sur les noms. Les deux mécanismes d'attention mettent en évidence plus de noms que ce que le hasard aurait prédit (qui serait 37%), ce qui démontre également que le réseau se concentre délibérément sur les noms plutôt que sur tout autre POS. Cependant, dans ce cas et contrairement à COCO et STAIR, GRU1 a mis en évidence plus de noms dans l'ensemble (72, 58% ± 4, 94) que GRU5 (58, 73% ± 5, 30).

En examinant de plus près les mots spécifiquement mis en évidence par chaque mécanisme d'attention du meilleur modèle (présentés dans l'annexe B), nous remarquons que les 10 premiers mots mis en évidence par GRU1 sont uniquement des noms faisant référence à des objets présents dans les images (chien, homme, fille, garçon, chiens, personnes, femme, enfant, balle, eau). Les 10 premiers mots de GRU5 contiennent également des noms, mais aussi des silences, des prépositions et des déterminants (</s>, eau, <sil>, plage, neige, herbe, chemise, rue, a, in). GRU5 semble donc moins spécialisé que GRU1.

Position des pics et étude longitudinale

Dans cette expérience également, nous avons analysé au-dessus de quelles parties de mots les pics d'attention sont spécifiquement situés (Tableau 4.6). Nous observons que les pics ont également tendance à favoriser les fins de mots. Comme pour COCO et STAIR, les pics ne sont pas situés à la toute fin des mots mis en évidence, mais semblent être plus concentrés au milieu, puisque les plus fortes proportions de pics se trouvent au milieu du début et de la fin des mots. Une fois de plus, ces résultats tendent à montrer que le modèle n'a pas besoin d'avoir accès au mot complet pour le reconnaître et le mettre en évidence correctement.

La figure 4.6 montre comment la proportion de POS mis en évidence évolue dans le temps. Nous nous concentrerons sur GRU1 car c'est le mécanisme d'attention le plus interprétable du modèle. Comme pour COCO et STAIR, on constate que le modèle se focalise vite sur les noms et ce avec peu d'exemples (400 exemples suffisent). On constate que la proportion de pics au-dessus des noms augmente ensuite régulièrement au cours de la première époque. Cependant, contrairement à COCO et STAIR où la proportion n'a pas beaucoup évolué après la première époque, nous remarquons ici que la proportion de pics au-dessus des noms évolue encore après la première époque. Cela confirme que le modèle a besoin de plus de temps afin d'identifier précisément tous les mots importants de la légende. Cela s'explique par le fait que Flickr8k est composé de parole naturelle. Ainsi, le modèle a besoin de plus de temps pour prendre en compte les variations intra-et inter-locuteurs.

Acquisition du langage

Le fait que nos modèles mettent préférentiellement en avant les noms semble cohérent avec ce qui est observé dans la littérature sur l'acquisition du langage. Ce phénomène est communément appelé le "noun bias". En effet, il a été constaté que le lexique des enfantstant en perception qu'en production -contient une plus grande proportion de noms que de verbes ou de tout autre POS. [START_REF] Gentner | Why nouns are learned before verbs: Linguistic relativity versus natural partitioning[END_REF] a été la première à postuler que les noms sont plus faciles à apprendre que les verbes. Elle explique cela par le fait que "les mots qui se réfèrent à des concepts sont faciles à apprendre parce que l'enfant a déjà formé des concepts d'objet, et n'a besoin que de faire correspondre les mots et les concepts". Notre réseau neuronal est dans ce cas puisqu'il utilise des vecteurs VGG pré-entraînés qui codent les objets présents dans l'image.

Nous avons montré que notre modèle japonais développe un comportement spécifique à la langue lorsqu'il se concentre principalement sur les particules "ga". [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF] ont observé que les enfants japonais (à partir de 15 mois) font également usage de la particule "ga" pour segmenter le discours. Nos modèles ont donc adopté la même stratégie que les enfants japonais pour segmenter le nom adjacent. [START_REF] Haryu | Use of bound morphemes (noun particles) in word segmentation by Japanese-learning infants[END_REF] affirment qu'"il est clair que les particules liées au nom ne sont pas les premiers indices que les enfants utilisent pour la segmentation des mots". Nous observons effectivement ce type de schéma pour nos modèles où les particules sont à peine utilisées dans les premières étapes d'apprentissage, mais deviennent de plus en plus mises en avant, alors que les noms sont moins présents qu'au début. 6 Activation, compétition et reconnaissance lexicale Dans le chapitre précédent, nous avons montré que les modèles PVC étaient capables de mettre en évidence des mots spécifiques dans le flux de la parole en utilisant leurs mécanismes d'attention. Ceci implique que le modèle est capable de reconnaître les mots qui sont mis en évidence. Cette aptitude soulève quelques questions auxquelles nous allons tenter de répondre dans le présent chapitre. Cependant, ces études n'ont pas montré pour quel type de mots ce comportement est vrai et si le modèle a appris à associer ces mots individuels à leurs référents visuels. De plus, aucune de ces études n'a exploré les facteurs qui influencent la reconnaissance des mots et pourquoi certains mots semblent être assez bien reconnus alors que d'autres ne le sont pas du tout. C'est ce que nous cherchons à faire ici.

Reconnaissance de mots

Appariement de mots isolés

Afin de déterminer si le modèle est capable d'associer des mots isolés à leurs référents visuels, nous avons sélectionné un ensemble de 80 mots qui correspondent aux noms de 80 catégories d'objets dans l'ensemble de données MSCOCO. Nous nous attendons à ce que notre modèle soit très efficace avec ces mots spécifiques, car ils font référence aux principaux objets présentés dans l'ensemble de données MSCOCO. Nous évaluons la capacité du modèle à classer les images de manière à ce qu'au moins une image parmi les 10 premières contienne l'instance de l'objet correspondant au mot cible présenté (Précision@10, abrégé P@10). Il est important de rappeler qu'au moment de l'entraînement, le réseau n'a reçu que des légendes complètes et non des mots isolés. Par conséquent, si le réseau est capable de retrouver des images contenant des occurrences du mot cible, cela montre qu'une segmentation implicite a été effectuée au moment de l'entraînement.

Les résultats sont présentés dans la figure 5.1. 40 mots sur les 80 mots cibles ont une P@10≥ 0.8. Cela montre que le réseau est capable d'associer de manière fiable des mots isolés à leurs référents visuels, même s'il ne les a jamais vus isolément. À l'inverse, on remarque également que 15 mots ne sont pas mis en correspondance avec leurs référents visuels. Parmi les mots les mieux reconnus, on trouve des animaux (éléphant, zèbre, mouton, girafe) et des objets (camion, bus, bateau, avion) qui sont très fréquents dans notre jeu de données. Parmi les mots les moins bien reconnus, on trouve des objets (fourchette, couteau, vase, grille-pain) ou des animaux (souris) qui sont assez rarement mentionnés dans les légendes car ils peuvent être trop petits et non décrits par les annotateurs.

Facteurs influençant l'appariement

Nous explorons 2 types de facteurs : les facteurs liés à l'image et les facteurs liés à la parole. Pour ces derniers, nous considérons la fréquence des mots dans l'ensemble d'entraînement et la longueur du signal vocal. En ce qui concerne les facteurs liés à l'image, nous considérons la fréquence des instances d'objets dans les images de l'ensemble d'entraînement, le nombre moyen d'instances d'objets voisins et la surface moyenne de chaque objet. Pour modéliser la relation entre toutes ces variables, nous avons ajusté un modèle de régression linéaire multiple avec R où nous essayons de prédire la Précision@10 en utilisant les facteurs mentionnés précédemment.

Les résultats sont présentés dans le Tableau 5.1. Nous remarquons que le seul effet qui joue un rôle significatif dans la reconnaissance des mots est la fréquence du mot dans l'ensemble d'entraînement : plus le mot est fréquent dans l'ensemble d'entraînement, mieux il est reconnu. La longueur du mot a un léger effet positif qui tend à montrer que les mots longs sont mieux reconnus que les mots courts. La fréquence des objets et le nombre d'objets voisins n'ont aucun effet. La taille des objets semble également avoir un léger effet.

Nos résultats montrent donc que les mots individuels sont effectivement mis en correspondance de manière fiable avec leur référent visuel par le réseau. Le principal facteur de réussite dans cette tâche est la fréquence des mots cibles dans la légende ainsi que la taille des objets dans l'image. Ainsi, les mots qui sont très fréquents et qui font référence à des objets de grande taille sont mieux reconnus que les autres.

Activation lexicale 6.2.1 Paradigme du gating

Le paradigme de gating a été introduit par [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] et implique la procédure suivante :

Le paradigme du gating implique la présentation répétée d'un stimulus oral (dans ce cas, un mot) de telle sorte que sa durée depuis l'apparition est augmentée à chaque présentation successive. On procède ainsi jusqu'à ce que le mot entier ait été présenté. Après chaque présentation (ou gate), on demande aux sujets de noter le mot présenté et d'évaluer leur confiance dans chaque supposition. [START_REF] Cotton | The gating paradigm: A comparison of successive and individual presentation formats[END_REF] Dans notre cas, cela signifie que le modèle neuronal est alimenté par des versions tronquées d'un mot cible donné, chaque version tronquée comprenant une plus grande partie du mot cible. Dans notre cas, la troncation se fait soit de gauche à droite (le modèle n'a accès qu'à la fin du mot), soit de droite à gauche (le modèle n'a accès qu'au début du mot). Nous évaluons la capacité des modèles à classer les images de façon à ce que les k premières images contiennent des instances de l'objet cible (nous utilisons la Précision@10, P@10). Le but de cette expérience est de tester si le début du mot joue un rôle dans la reconnaissance des mots pour le réseau ou non. Si c'est le cas, nous nous attendons à ce que le réseau ne parvienne pas à récupérer les images du mot cible si le mot est tronqué de gauche à droite, mais pas -ou moins -lorsque le mot est tronqué de droite à gauche, ce qui motive la troncature des deux côtés.

Effets du gating

La figure 5.2a montre l'évolution moyenne de la P@10 sur les 80 mots de test. Comme on peut le voir sur le graphique, la précision évolue différemment selon la partie du mot qui a été tronquée. Lorsque les mots cibles sont tronqués de gauche à droite, la précision chute plus rapidement que lorsqu'ils sont tronqués de droite à gauche. Ces résultats montrent que le modèle est robuste à la troncation lorsqu'elle est effectuée de droite à gauche mais pas lorsqu'elle est effectuée de gauche à droite : lorsque les phonèmes initiaux des mots sont supprimés, le réseau ne parvient pas à retrouver l'image cible, mais lorsqu'on ne lui présente que les phonèmes initiaux, le réseau est globalement capable de retrouver des images correspondant au mot cible. Nous laissons progressivement le réseau voir de plus en plus de vecteurs MFCC composant un mot donné, en lui donnant itérativement des segments de vecteurs MFCC de plus en plus longs en commençant par le début du mot jusqu'à ce que le réseau ait eu accès au mot complet. Nous calculons ensuite la similarité cosinus entre l'embedding calculé pour chacune des versions tronquées du mot et l'embedding correspondant au mot complet. Plus la similarité cosinus est proche de 1, plus les deux représentations sont similaires. Si chaque vecteur MFCC contribue de manière égale à la représentation finale du mot, la similarité cosinus évoluera linéairement sinon la similarité cosinus évoluera par "sauts" plutôt que linéairement. Pour détecter les étapes qui pourraient se produire dans l'évolution de la similarité cosinus, nous approximons sa dérivée en calculant la différence de premier ordre. Les "sauts" élevées devraient ainsi se traduire par des pics (par exemple, figure 5.4b). Nous calculons l'évolution de la similarité cosinus pour les 80 mots cibles codés avec le meilleur modèle entraîné.

En moyenne, il y a 1,25 pic par mot pour le modèle entraîné contre 0,1 pic par mot pour notre condition de base, montrant que l'évolution du cosinus est linéaire dans le second, mais pas dans le premier. Par conséquent, dans notre modèle entraîné, certains vecteurs MFCC sont plus déterminants pour la représentation finale que d'autres. En effet, certains vecteurs MFCC déclenchent un "saut" élevé dans l'évolution du cosinus, ce qui suggère que l'embedding se rapproche soudainement de sa valeur finale.

Compétition lexicale

Certains modèles psycholinguistiques (voir section 1.4) supposent que le premier phonème d'un mot active tous les mots commençant par le même phonème. Le mot que le locuteur veut prononcer et qu'il prononce progressivement est appelé le mot "cible". Les mots qui sont activés mais qui ne correspondent pas au mot cible sont appelés "concurrents". Lorsque l'auditeur perçoit de plus en plus le mot cible, certains concurrents sont désactivés. Cela signifie qu'ils ne sont plus considérés comme le mot potentiel, car ils ne correspondent pas à ce qui est perçu.

Méthodologie

Nous avons sélectionné 12 paires de mots commençant par la même séquence de phonèmes et avons testé si elles étaient activées par un processus de compétition. 1Pour chaque paire de mots, nous avons sélectionné un des mots que nous considérons comme le mot cible, en laissant progressivement le réseau voir de plus en plus de vecteurs MFCC composant ce mot. À chaque pas de temps, le réseau produit un vecteur, que nous utilisons pour classer les images de l'image la plus proche à l'image la moins proche. Ensuite, pour les 50 images les plus proches, nous vérifions si au moins une des légendes contient soit le mot cible, soit le concurrent.

Comme le concurrent et le mot cible commencent par les mêmes phonèmes, nous nous attendons à ce que le réseau produise un vecteur qui active à la fois le concurrent et le mot cible au début, puis, lorsque le signal acoustique ne correspond plus au concurrent, nous nous attendons à ce que le réseau soit capable d'activer uniquement le mot cible. Pour chaque paire de mots, chaque mot est utilisé alternativement comme mot cible.

Résultats

Trois résultats sont présentés pour 3 paires de mots : "Train/Truck" (5.6) pour laquelle une forte compétition existe, "Cat/Cattle" (5.5) pour laquelle une compétition modérée existe, et "Frigde/Frisbee" (5.7) pour laquelle aucun phénomène de compétition n'a pu être mis en évidence. Les modèles de la cohorte [START_REF] Marslen-Wilson | Processing interactions and lexical access during word recognition in continuous speech[END_REF][START_REF] Marslen-Wilson | Functional parallelism in spoken word-recognition[END_REF]) et trace (McClelland & Elman 1986b) affirment tout deux que les mots concurrents sont tous activés en même temps, c'est-à-dire lorsque le premier phonème est perçu. Même si certains exemples sont conformes à cette affirmation -comme celui présenté dans la figure 5.6b -ce n'est pas le cas de toutes les paires de mots. Dans certains cas, la compétition est minime car la représentation des deux mots est activée de manière séquentielle (comme dans la figure 5.5) et dans d'autres cas, il n'y a absolument aucune compétition entre les deux mots (comme dans la figure 5.7) bien que les deux mots commencent de manière similaire. Par conséquent, le comportement du réseau semble très peu clair. Il semble toutefois que le réseau active le mot qui est le plus courant. Par exemple, il y a beaucoup plus de photos de trains que de photos où les camions constituent l'objet principal de l'image. Ainsi, le réseau semble activer préférentiellement la représentation de l'objet qui est le plus fréquent dans les images et les légendes.

Conclusion

Dans ce chapitre, nous avons montré qu'un modèle PVC est capable de faire correspondre des mots individuels à leurs référents visuels bien qu'il ait été entraîné sur des légendes complètes. van Zon (1997, p. 8) note que dans les modèles COHORT et TRACE "la segmentation est le résultat de la reconnaissance". Nous pensons que la reconnaissance de mots individuels montre que le modèle a implicitement segmenté ses entrées en sous-unités, et proposons la formulation inverse :"la reconnaissance est la preuve de la segmentation". Un point important que nous avons souligné dans ce chapitre est cependant que la segmentation résultante peut ne pas toujours correspondre à des mots graphiques.

Dans ce chapitre, nous avons introduit plusieurs méthodologies pour analyser comment la représentation des mots individuels se construit au cours du temps. Notamment, nous avons adapté le paradigme de gating de [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF] afin d'analyser comment les mots sont activés par le réseau. Nous avons observé que la représentation des mots n'est pas construite linéairement par le réseau et que la reconnaissance peut se produire avec une entrée partielle, corroborant ainsi que "la reconnaissance se produit souvent avant la fin du mot" (van Zon 1997, p. 8). Nous avons montré que pour pouvoir activer la représentation correcte d'un mot donné, le réseau doit avoir accès aux premiers phonèmes de ce mot, car lorsqu'ils sont supprimés, le réseau est incapable d'activer la représentation correcte. Ainsi, lorsque la reconnaissance de mots est observée avec une entrée partielle, c'est uniquement lorsque l'entrée partielle englobe la première partie du mot, mais pas lorsqu'elle n'englobe que la partie finale du mot.

Finalement, nous avons cherché à étudier la compétition lexicale sans avoir pu parvenir à mettre en avant une systématicité de ce phénomène dans notre réseau.

Impact de l'introduction d'information linguistiques

Dans les chapitres précédents, nous avons montré que les modèles de parole visuellement contextualisée basés sur des RNN utilisent leur mécanisme d'attention pour mettre en évidence les mots qui sont pertinents pour retrouver l'image correcte, et que ces modèles segmentent implicitement l'entrée parlée en sous-unités. Dans ce chapitre, au lieu de comprendre quels types d'unités ont été implicitement segmentés par le réseau, nous abordons le problème dans l'autre sens et nous posons la question suivante : quelle segmentation maximise la performance (rappel@k) d'un modèle PVC si la parole devait être segmentée ? Afin de répondre à cette question, nous explorons comment l'information de frontière peut être intégrée, quel type de frontière est le plus efficace (soit le phone, la syllabe ou le mot), et enfin où -c'est-à-dire, à quelle couche -une telle frontière devrait être introduite dans le réseau. Enfin, nous explorons également des modèles hiérarchiques pour lesquels nous fournissons plusieurs niveaux de segmentation en même temps afin de comprendre l'effet de la modélisation explicite de la nature hiérarchique de la parole.

7.1 Information de frontières 7.1.1 Types de frontières Comme indiqué précédemment, nous souhaitons donner des informations linguistiques à notre réseau, et plus particulièrement des informations sur les frontières des segments. Dans ce chapitre, nous définissons un segment comme étant soit un phone, soit une syllabe, soit un mot. Les frontières des segments ont été dérivées des métadonnées de l'alignement forcé (voir § 3.2.4) afin de savoir quel vecteur MFCC constitue une frontière ou non.

Nous considérons deux types différents de syllabes dans ce travail : en effet, lorsque nous parlons, les mots ne sont pas prononcés les uns après les autres de manière déconnectée, mais sont plutôt connectés par un processus appelé "resyllabification". En anglais, ce phénomène est visible lorsqu'un mot se terminant par une consonne est suivi d'un mot commençant par une voyelle. Dans ce cas, la consonne finale du premier mot tend à s'en détacher et à se rattacher au mot suivant, franchissant ainsi la frontière du mot. Les deux types de syllabes que nous considérons dans ce travail sont les suivants : "syllabe-mot" qui fait référence aux syllabes qui résultent d'une segmentation qui ne prend pas en compte la resyllabification, et "syllabe-connectée" qui fait référence aux syllabes qui résultent d'une segmentation qui prend en compte la resyllabification.

Ainsi, pour chaque légende, nous avons une séquence X de longueur T de vecteurs acoustiques de dimension d : X = x

Intégrer des frontières de segments

Afin d'intégrer des informations de frontière dans notre réseau, nous tirons parti de sa conception, et plus particulièrement des cellules récurrentes et de la manière dont ces cellules calculent leur sortie, qui peut être formalisée comme suit : h t = f (h t-1 , x t ; θ) (D.10) où l'état caché au temps t, noté h t , est une fonction f du vecteur précédent h t-1 et du vecteur d'entrée actuel x t , et où θ est un ensemble de paramètres entraînables de la fonction f .

Impact de l'introduction d'information linguistiques 203

Notre approche pour intégrer l'information de frontière dans les couches récurrentes de notre réseau peut être formalisée comme suit :

h t =      f (h 0 , x t ; θ) , if b t-1 = 1
N f (h t-1 , x t ; θ) , otherwise (D.11) Dans notre approche, h t ne dépend du vecteur précédent h t-1 que si le vecteur précédent n'est pas un vecteur acoustique correspondant à la frontière d'un segment (b t-1 = 0). Si le pas de temps précédent correspond à la frontière d'un segment (b t-1 = 1), nous réinitialisons l'état caché pour qu'il soit égal à h 0 . Ainsi, les vecteurs d'un même segment sont temporellement dépendants, mais les vecteurs appartenant à deux segments différents ne le sont pas. Les GRU qui utilisent ce schéma de calcul seront dorénavant appelés GRU PACK. , car les vecteurs appartenant au même segment sont "regroupés" (packed ) ensemble. La longueur de la séquence de sortie et d'entrée reste la même dans la condition all mais dans la condition keep, la longueur de la séquence de sortie est plus courte que la séquence d'entrée.

Méthodologie

Afin de comprendre où les informations de frontière doivent être introduites (c'est-à-dire à quel niveau de l'architecture), nous entraînons autant de modèles que le nombre de couches récurrentes, où chaque fois une couche de GRU est remplacée par une couche GRU PACK. . Par exemple, "GRU PACK. -3 " fait référence à un modèle où la troisième couche de GRU est une couche GRU PACK. et les autres couches (1 e ,2 e ,4 e , et 5 e ) sont des couches de GRU classiques.

Afin de comprendre si l'introduction d'informations sur les frontières aide le réseau dans sa tâche, nous comparons les performances des modèles utilisant des informations sur les frontières avec un modèle de base qui n'en utilise aucune (ainsi, toutes les couches récurrentes de l'architecture de base sont des couches de GRU normaux). Ce modèle sera dorénavant appelé baseline. Nous introduisons également une autre condition, où, au lieu d'entraîner des modèles avec des limites de segment réelles (que l'on appellera désormais true), nous entraînons des modèles avec des limites aléatoires (que l'on appellera désormais random). Les frontières aléatoires ont été générées en mélangeant simplement la position des frontières réelles, ce qui donne autant de frontières positionnées aléatoirement que de frontières réelles.

Les modèles sont évalués en termes de Rappel@k (R@k). Pour une requête orale, le R@k évalue la capacité des modèles à classer l'image jumelée cible parmi les k meilleures images. Afin d'évaluer si les résultats observés dans nos différentes conditions expérimentales (trueall, true-keep, random-all, random-keep) sont différents les uns des autres et de la condition baseline, nous avons utilisé un Z-test de proportion binomiale. Ce test est utilisé pour évaluer s'il existe une différence statistique entre deux proportions indépendantes. Dans notre cas, la proportion que nous testons est le nombre de succès sur le nombre 204 Résumé Étendu en Français d'essais (qui correspond au nombre de paires légende/image différentes dans l'ensemble de test).

Résultats

Dans l'ensemble, nos paramètres expérimentaux ont conduit à l'entraînement de 81 modèles différents par ensemble de données. Les résultats de base sont présentés dans le tableau 6.1. Les résultats pour les conditions true/random obtenus sur les jeux de données COCO et Flickr8k sont présentés dans le Tableau 6.2 et 6.3 respectivement. Nous obtenons des résultats plus faibles sur Flickr8k que sur COCO, ce qui montre la difficulté de la tâche sur la parole naturelle.

Dans l'ensemble, les modèles entraînés sur Flickr8k avec les frontières true ont un R@1 significativement meilleur que leurs homologues de base et les modèles entraînés avec les frontières random, ce qui indique que les modèles ont utilisé efficacement l'information sur les frontières. Pour COCO, nous observons que certains modèles entraînés avec des frontières aléatoires ont des scores significativement meilleurs que baseline (en particulier lors de l'utilisation de frontières de phone dans la condition keep), cependant cet effet disparaît lors de l'utilisation d'unités plus grandes, telles que des mots.

Il existe un effet d'interaction entre l'utilisation de frontières true et random, que ce soit dans la condition all ou keep. En effet, dans la condition random-all, aucun résultat n'est statistiquement meilleur que baseline, alors que dans la condition random-keep, les résultats sont statistiquement moins bons que baseline. Par conséquent, l'utilisation de frontières aléatoires qui ne délimitent pas d'unités linguistiques significatives nuit réellement aux performances du réseau. Cependant, dans la condition true-keep, les résultats sont meilleurs que baseline. Par conséquent, la condition keep contraint le réseau à apprendre de meilleures représentations, alors que dans la condition all, les informations sur les frontières sont diluées par les vecteurs voisins, ce qui conduit à une utilisation sous-optimale de ces informations.

Dans nos expériences, nous avons utilisé quatre types différents de segments correspondant à deux types différents d'unités linguistiques : les phones, les syllabes-connectées, les syllabes-mots et les mots. Nous nous attendons à ce que les segments de type mot (ou les segments qui préservent les frontières du mot et qui portent une quantité substantielle d'informations sémantiques) obtiennent de meilleurs résultats. Les unités de mots obtiennent en effet des résultats statistiquement meilleurs que baseline pour Flickr8k et COCO (R@1 = 5.4, +1.1pp et R@1 = 11.3, +2.3pp respectivement). Les syllabesmots apportent également une amélioration significative (R@1 = 5.3 pour Flickr8k et R@1 = 10.9 pour COCO), toutefois légèrement inférieure à celle obtenue avec les unités de mots.

Nos résultats montrent clairement que l'introduction d'informations sur les frontières à différentes couches a un impact substantiel sur les résultats : l'utilisation de ces informations à la première ou à la cinquième couche est inutile, car nous remarquons qu'elle donne des résultats similaires à ceux de baseline (GRU PACK. -1) ou qu'elle détériore les résultats quel que soit le type de frontière utilisé (GRU PACK. -5), alors que les résultats sont meilleurs lorsque ces informations sont introduites dans les couches intermédiaires.

Informations hiérarchiques

Dans les expériences susmentionnées, nous fournissons au réseau un seul type de frontière (soit phone, syllabe ou mot) mais pas plusieurs en même temps, comme si plusieurs unités ne pouvaient pas coexister en même temps. Pourtant, de multiples unités parlées existent 7. Impact de l'introduction d'information linguistiques 205 en même temps, et elles sont structurées de manière hiérarchique : les mots peuvent être décomposés en syllabes, qui peuvent à leur tour être décomposées en phones. Afin de modéliser cette nature hiérarchique de la parole, nous pouvons empiler autant de GRU PACK. que nous le souhaitons, où une couche gère un type de segment (par exemple, les phones) et la couche suivante du GRU PACK. gère un autre type de segment, qui est hiérarchiquement au-dessus du précédent (par exemple, des mots, voir Figure 6.2).

Nous explorons l'effet de l'utilisation d'une architecture hiérarchique sur l'ensemble de données Flickr8k. Contrairement à nos expériences précédentes, nous ne considérons que les architectures hiérarchiques qui utilisent des couches GRU PACK. -keep. Nous testons plusieurs architectures qui gèrent différents types de frontières simultanément dont deux des cinq couches récurrentes sont des couches GRU PACK. , puis dans l'expérience suivante, dont trois des cinq couches sont GRU PACK. . Dans les deux cas, nous testons toutes les positions possibles ainsi que le type de frontière pour comprendre quelle est la meilleure combinaison.

7.5 2 couches GRU Packager 7.5.1 Phones et mots En utilisant les phones et les mots ensemble, les résultats (Tableau 6.4) sont supérieurs à ceux de l'architecture de base et à ceux de l'architecture GRU PACK. à couche unique. En effet, nous obtenons un R@1 maximal de 8.2% lorsque nous utilisons GRU PACK. au niveau des couches 2 et 3, ce qui représente +3.9pp par rapport à l'architecture de base et +2.8pp par rapport à une architecture monocouche. Nous remarquons également que l'utilisation d'une architecture hiérarchique nous permet d'entraîner des réseaux moins profonds tout en améliorant les résultats par rapport à notre architecture de base.

Phones et syllabes

Les résultats que nous obtenons en utilisant des phones et des syllabes sont présentés dans le tableau 6.6. Ici aussi, nous remarquons que les résultats sont meilleurs que ceux de la baseline (sans aucun GRU PACK. ) et également meilleurs que lorsque nous utilisons un seul GRU PACK. : R@1 = 7.9, +2.5pp. Contrairement à l'expérience précédente, l'architecture à 4 couches converge mieux que l'architecture à 5 couches. Néanmoins, les résultats sont inférieurs à ceux obtenus en utilisant conjointement les phones et les mots (-0.3pp), ce qui indique que l'utilisation conjointe des phones et des syllabes n'est pas la combinaison idéale.

Syllabes et mots

Les résultats que nous obtenons en utilisant des syllabes et des mots sont présentés dans le tableau 6.7. Une fois de plus, nous remarquons que R@1 est supérieur à une architecture à une couche GRU PACK. (R@1 = 7.6, +2, 2pp) mais pire qu'une architecture à deux couches traitant les phones et les mots : -0, 6pp. Le meilleur résultat est également inférieur au meilleur résultat obtenu en utilisant des phones et des syllabes : -0.3pp. Cependant, nous observons que l'architecture à deux couches est plus performante que la baseline (+1.0pp) et plus performante que l'architecture à deux couches qui utilise conjointement les phones et les syllabes (+0.7pp).

Conclusion

Nos expériences montrent que le meilleur résultat en utilisant deux couches GRU PACK. est obtenu en utilisant conjointement les frontières des phones et des mots, lorsque les 206 Résumé Étendu en Français couches GRU PACK. sont placées au milieu de la pile de cellules récurrentes. Nous observons également que, globalement, nos résultats atteignent leur maximum lorsque les couches se suivent. Enfin, cette série d'expériences nous a permis de montrer que le réseau converge mieux lorsque les segments de bas niveau (phones) et les segments de haut niveau (mots) sont utilisés conjointement. Nous expliquons cela par le fait que cela permet au modèle d'apprendre des représentations robustes pour les phones, tout en ayant des unités de haut niveau qui portent beaucoup d'informations sémantiques. Par conséquent, l'utilisation de segments intermédiaires tels que les syllabes n'est pas utile car elles sont trop longues pour apprendre une représentation cohérente tout en étant trop courtes par rapport à la quantité d'informations sémantiques qu'elles portent.

3 couches GRU Packager

Enfin, nous avons intégré trois niveaux de segments dans un seul modèle. Comme dans nos expériences précédentes, nous expérimentons avec un nombre différent de couches (de 3 à 5), avec à chaque fois 3 couches GRU PACK. à chaque position possible. Les résultats de cette expérience sont présentés dans le tableau 6.8.

Nous observons que le meilleur résultat obtenu avec cette architecture (R@1 = 9.6) est bien meilleur que la baseline (+5.3pp), supérieur au meilleur résultat d'une architecture à une seule couche (+4.2pp) mais aussi supérieur au meilleur résultat d'une architecture à deux couches (+1.4pp sur l'architecture phonème-mot). Notre meilleur résultat est obtenu par une architecture à cinq couches avec GRU PACK. en position 1, 3 et 4. Bien que nous remarquions que le R@1 s'est amélioré lors de l'ajout d'un autre niveau de frontière dans cette dernière expérience, nous observons également que le saut dans les résultats n'est pas aussi important que ce que nous avons observé auparavant. En utilisant un seul GRU PACK. , nous observons une amélioration de +1.1pp dans R@1 par rapport à notre architecture de base. En utilisant deux GRU PACK. , nous observons une amélioration de +2.8pp par rapport à l'utilisation d'un GRU PACK. . Enfin, en utilisant un troisième GRU PACK. , nous avons observé une amélioration de +1.4pp par rapport à l'utilisation de deux GRU PACK. . Ainsi, même si l'introduction de plus de structure dans le réseau est bénéfique, nous observons également que certains niveaux sont plus critiques que d'autres et ont un effet plus important sur le résultat final.

Conclusion du chapitre

Dans ce chapitre, nous avons étudié l'impact de la segmentation préalable de la parole dans un modèle PVC. Nous avons présenté une méthode simple pour introduire des informations de frontière dans n'importe quelle couche récurrente de notre architecture. Pour ce faire, il suffit de réinitialiser l'historique du RNN chaque fois qu'il y a une limite de segment. Nous avons constaté que la segmentation de la parole en sous-unités est utile : la segmentation en mots donne de meilleurs résultats que la segmentation en phonèmes, mais nous observons également que la segmentation en syllabes donne des résultats similaires à la segmentation en mots. Néanmoins, la segmentation en mots semble être une représentation plus robuste. Nous avons observé des résultats différents selon le niveau auquel les informations de segmentation sont introduites. Nous avons observé des effets négatifs si l'information de frontière est introduite trop tard (dernière couche de notre architecture).

Néanmoins, même si l'introduction d'informations sur les frontières est utile, elle n'améliore que légèrement les performances du réseau. Ce n'est que lorsque différents niveaux sont combinés que les performances du réseau atteignent leur maximum. Nos expériences révèlent qu'une structure qui utilise des segments de bas niveau (c'est-à-dire 8 Conclusion

Résumé des contributions

Dans cette thèse, nous avons étudié un modèle récurrent de parole visuellement contextualisée. Notre objectif était d'analyser les représentations apprises par notre modèle afin de mieux comprendre quelles connaissances linguistiques les modèles neuronaux sont capables d'acquérir de manière non supervisée. Nous avons comparé ces représentations à ce qui est connu du traitement de la parole humaine. Plus spécifiquement, nous nous sommes concentrés sur l'acquisition lexicale et avons trouvé des points communs entre les processus à l'oeuvre dans les modèles que nous avons étudiés et les processus rapportés dans la littérature sur l'acquisition du langage chez l'enfant.

Plus précisément, les principales contributions de cette thèse sont :

• Le jeu de données image/parole STAIR. Nous avons introduit le jeu de données image/parole STAIR qui est basé sur le jeu de données image/texte STAIR [START_REF] Yoshikawa | STAIR captions: Constructing a Large-Scale Japanese Image Caption Dataset[END_REF]. Cet ensemble de données constitue l'équivalent japonais de l'ensemble de données "Synthetically spoken COCO" (Chrupa la et al. 2017a) pour l'anglais et permet d'entraîner des modèles de parole visuellement contextualisée.

• Analyse de l'attention. Nous avons montré que les modèles basés sur les RNN sont capables de détecter des motifs récurrents spécifiques dans l'entrée acoustique. Les modèles que nous avons entraînés se concentrent spécifiquement sur les mots concrets tels que les noms, car ils se réfèrent à des objets qui sont particulièrement saillants dans les images. Nous avons observé ce comportement pour deux langues typologiquement distinctes, l'anglais et le japonais, montrant ainsi que ce comportement ne dépend pas d'une langue particulière. Nous avons conclu que les modèles que nous avons étudiés présentaient une préférence pour les noms, comme celle que l'on trouve également chez les humains au cours du processus d'acquisition lexicale. Nous avons mis en évidence que les modèles japonais ont appris à détecter et à mettre en évidence des particules (comme la particule "ga"), et montrent ainsi un comportement spécifique à la langue. Ce faisant, les modèles ont adopté le même comportement que les enfants japonais afin de segmenter le flux de parole. La capacité des réseaux à mettre en évidence les noms -et les particules pour le japonais -est un comportement que les réseaux ont appris à adopter avec très peu d'exemples -moins de 500 paires image/légende -montrant que le réseau apprend rapidement quelles sont les parties les plus importantes des légendes.

• Analyse de la connaissance des mots isolés. Nous avons montré que le réseau était capable d'associer des noms individuels à leurs référents visuels corrects. Cela suggère que le réseau segmente implicitement son entrée en sous-unités, puis les associe à un contexte visuel. Cependant, nous avons observé que le réseau n'était pas capable d'associer tous les noms isolés à leurs référents visuels de manière égale ce qui suggère que le lexique du modèle est limité à un ensemble spécifique de mots. Nous avons observé que ce phénomène était principalement dû à la fréquence du mot dans la légende : plus une forme de mot est fréquente, mieux le modèle la relie à son référent visuel.

Nous avons ensuite étudié comment le réseau active un mot isolé et l'avons comparé aux modèles d'activation et de reconnaissance des mots chez l'homme. En utilisant un équivalent algorithmique du paradigme du gating de [START_REF] Grosjean | Spoken word recognition processes and the gating paradigm[END_REF][START_REF] Grosjean | The recognition of words after their acoustic offset: Evidence and implications[END_REF], une méthodologie issue de la psycholinguistique, nous avons pu observer que l'activation Dans son livre, Bloom (2002, p. 60) soutient que "les enfants apprennent le sens des mots par la théorie de l'esprit. Si cela est vrai, alors une mise en oeuvre connexionniste directe de l'apprentissage des mots, dans laquelle les sons sont associés à des percepts, est irréalisable. (Et cela exclut toutes les théories connexionnistes de l'apprentissage des mots dont [il] a connaissance)". Nous pensons que les expériences que nous avons menées dans cette thèse, ainsi que les travaux précédents de Harwath et al. (2016), [START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF]), Chrupa la et al. (2017a), Merkx et al. (2019) (entre autres) montrent que les modèles purement connexionnistes sont capables d'associer directement les sons aux percepts, ici sous la forme de représentations vectorielles de stimuli visuels. Par conséquent, les modèles connexionnistes sont capables d'apprendre des mots dans une certaine mesure. Bien entendu, nous ne prétendons pas que l'acquisition lexicale chez l'enfant se fait uniquement via un mécanisme purement associatif, mais il se pourrait qu'un mécanisme d'apprentissage purement associatif permette d'amorcer l'acquisition lexicale chez l'enfant. Un argument contre ce fait pourrait être que les approches connexionnistes nécessitent de grandes quantités de données pour être entraînées efficacement. Cependant, nos expériences montrent que nos modèles ont appris à se concentrer sur des noms spécifiques avec un très petit nombre d'exemples, ce qui suggère que l'amorçage associationniste constitue un mécanisme viable pour acquérir un lexique.

Les expériences que nous avons menées dans cette thèse nous permettent de conclure que les modèles PVC segmentent implicitement leur entrée en sous-unités et associent ces sousunités à leur référent visuel. Ce processus ne semble émerger que comme un sous-produit de leur tâche principale qui est de minimiser la distance entre un stimulus acoustique et un stimulus visuel correspondant. Notre conclusion est en accord avec le travail très récent de [START_REF] Khorrami | Can phones, syllables, and words emerge as sideproducts of cross-situational audiovisual learning? -A computational investigation[END_REF].

Futurs travaux

Compte tenu des travaux que nous avons menés dans cette thèse, plusieurs travaux futurs pourraient être réalisés : En conclusion, les modèles neuronaux de parole à base visuelle offrent des opportunités inestimables pour étudier et tester des hypothèses sur l'acquisition du langage chez l'enfant, grâce à leur capacité à modéliser des interactions complexes à travers plusieurs modalités. De nouveaux ensembles de données, tels que l'ensemble de données SEEDLingS [START_REF] Bergelson | Nature and origins of the lexicon in 6-mo-olds[END_REF]) ou l'ensemble de données récemment collecté par [START_REF] Tsutsui | A computational Model of Early Word Learning from the Infant's Point of View[END_REF], qui sont des enregistrements à grande échelle recueillis dans des environnements écologiques [START_REF] Dupoux | Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner[END_REF], permettront aux chercheurs de simuler l'acquisition du langage avec des données plus réalistes que jamais.

Figure 1 . 1 :

 11 Figure 1.1: Figure taken from[START_REF] Mattys | Speech segmentation[END_REF] presenting the cues used in a mature speech segmentation (i.e. adult) system and in a developing system (i.e. child) with the black arrows illustrating the back and forth interactions between top-level cues (semantics, lexical) and low-level cues (prosody). The grey triangle indicates the importance of each set of cues (with the wider end signalling the greater importance of such cues).

Figure 1 . 2 :

 12 Figure 1.2: Illustration of shared attention between a caregiver (top left) and a toddler (top right), attending to the same objet, a chicken (bottom).

Figure 2 . 1 :

 21 Figure 2.1: Unsupervised Speech Processing Hierarchy. Figure reproduced from Glass (2012) except for the annotations in red which we added. The boxed text shows where the work conducted in this thesis fits into this hierarchy.

Figure 2 . 3 :

 23 Figure 2.3: Illustration of a RNN folded (right) and unfolded over time (left). The diagram is annotated according to Equation 2.2. Note the initial state h 0 which is added for the first timestep.

Figure 2 . 4 :

 24 Figure 2.4: Illustration of a 2D-convolution on a 7 × 7 input producing a 5 × 5 output when using 1 kernel of size 3 × 3 sliding from left to right from top to bottom with a stride of 1, and zero padding. The convolutional kernel is depicted with borders of different colours at different places; the result of the convolution is shown by a shaded box with the same colour.For illustration purposes, we only depicted three positions where the kernel could be, when naturally the convolutional kernel scans the whole input. Usually more than one kernel is used.

Figure 2 . 5 :

 25 Figure 2.5: Illustration of a 1D-convolution on a S in × 13 input MFCC matrix producing a Sout × 4 output (see Equation2.5 on how to compute Sout) when using 4 convolutional kernels of size 2 × 13 sliding along the Time axis with a stride of 1, and zero padding. The position of the convolutional kernels are depicted with borders of different colours -blue and red -to materialise the evolution of their position over time; and the result of the convolution operation is shown by a shaded box with the same colour in the output matrix. For illustration purposes, we only depicted two positions where the kernels could be (at kernels start position and kernels end position), when naturally the convolutional kernels scan the whole input. We also only depicted 4 kernels -shown on the Kernel axis, with different shades of the same colour -when usually more are used.

Figure 2 . 6 :

 26 Figure 2.6: Traditional architecture of neuronal VGS models. Figure inspired by Chrupa la et al. (2017a) and[START_REF] Khorrami | Can phones, syllables, and words emerge as sideproducts of cross-situational audiovisual learning? -A computational investigation[END_REF]. The difference between CNN-based and RNNbased model depends on neural cells used for the speech encoder. Note that the embedded images and utterances need not be vectors but may also be matrices. The dashed box represents the shared embedding space in which the similarity measures between matching (green arrow) and mismatching (red arrows) image and utterance pairs is realised. Note that the shape of the embedding space depends on the nature of the encoders. The embedding space used by RNN-based models is typically a d-dimensional hypersphere, where d denotes the dimension of the embedding vectors (seeSection 3.3.2). Transcription of the audio signal is only given for illustration purposes and is not inputted to the model. Black arrows show the direction of the information flow through the network.

Figure 2 .

 2 Figure 2.7: Example image taken from MSCOCO (left) along with five descriptive captions (right).16

Figure 2

 2 Figure 2.8: Figure reproduced from Roy (2005). We annotated the figure according to the models we presented, so as to reflect to what extent these models are fully-fledged grounded models.

3. 3

 3 ArchitectureThe model we use for our experiments is based on that of Chrupa la et al. (2017a).12 The modifications we made to the architecture are detailed in the following sections. This implementation uses Python's Theano deep learning library (James[START_REF] Bergstra | Theano: A CPU and GPU Math Compiler in Python[END_REF]. As most VGS architectures[START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF][START_REF] Kamper | Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech[END_REF], Merkx et al. 2019), the architecture we use has two main components: an image encoder, and a speech encoder (seeFigure 3.1)

Figure 3 . 1 :

 31 Figure 3.1: Architecture of the visually grounded speech models used in the following experiments. The red boxes show the position of the two attention mechanisms used in the following experiments.

Figure 3 . 2 :

 32 Figure 3.2: Illustration of Cosine Similarity on a d-dimensional hypersphere between two vectors.

Figure 4 . 1 :

 41 Figure 4.1: Example of the distribution of GRU5's attention weights over (4.1a) an English caption and (4.1b) Japanese caption along with (4.1c) reference picture.

Figure 4 . 2 :

 42 Figure 4.2: Bar plots showing the proportion of attention peaks above each POS for (4.2a) GRU1 and (4.2b) GRU5 on the COCO data set. Real peaks are shown in blue and random peaks are shown in red. The results are averaged over five runs with different seeds. Error bars represent ± standard deviation.

Figure 4 . 3 :

 43 Figure 4.3: Bar plots showing the proportion of attention peaks above each POS for (4.3a) GRU1 and (4.3b) GRU5 on the STAIR data set. Real peaks are shown in blue and random peaks are shown in red. The results are averaged over five runs with different seeds. Error bars represent ± standard deviation.

Figure 4 . 4 :

 44 Figure 4.4: Evolution across epochs of the proportion of peaks above a given POS for (4.4a) COCO and (4.4b) STAIR. The percentage of peaks above a given POS is shown on the y-axis. The number of training examples seen at any given point of the graph is shown on the bottom x-axis. The point marking the last saving step of an epoch is shown on the top x-axis. Best saving time step (selected on the validation set) is shown by a bold dotted red vectical line. Results obtained with randomly initialised weights correspond to the first tick of both x-axes labelled "0".

Figure 4 . 5 :

 45 Figure 4.5: Bar plots showing the proportion of attention peaks above each POS for (4.3a) GRU1 and (4.3b) GRU5 on the Flickr8k data set. Real peaks are shown in blue and random peaks are shown in red. The results are averaged over five runs with different seeds. Error bars represent ± standard deviation.

Figure

  Figure 4.6 shows how the proportion of highlighted POS evolves over time. We will focus on GRU1 as it is the most interpretable attention mechanism of the model. We notice also the same phenomenon as what was observed for GRU5 for COCO and STAIR: we observe a large gap between the proportion of peaks highlighting nouns when the model uses randomly initialised weights (first tick on the x-axis) and the next saving steps, going from 40% up to 60% two saving steps later (amounting to 2,912 seen examples). The proportion of peaks highlighting other POS varies slightly in the first few steps and quickly decreases and stabilises to its final value. This shows that the model identified nouns as being useful to find the matching image. The proportion of peaks above nouns increases steadily during the first epoch. Yet, contrary to COCO and STAIR where the proportion did not evolve much after the first epoch, we notice here that the proportion of peaks above nouns still evolves after the first epoch. Indeed, at the end of the first epoch, the proportion of peaks above nouns is 72%, while it is 77% at the last epoch. This confirms that the model needs more time in order to precisely identify all the important words in the caption. In our opinion, this is to be explained by the fact that Flickr8k consists of real human speech. Thus, the model needs more time in order to account for intra-and inter-speaker variation.

Figure 4 . 6 :

 46 Figure 4.6: Evolution across epochs of the proportion of peaks above a given POS on the Flickr8k data set.

Figure 5 . 1 :

 51 Figure 5.1: Precision@10 for the 80 isolated words corresponding to MSCOCO categories.

Figure 5 . 2 : 5 .

 525 Figure 5.2: 5.2a Evolution of Precision@10 averaged over 80 test words as a function of the percentage of MFCC vectors removed for each word. 5.2b Evolution of Precision@10 for each ablation step of the word "zebra", with time-aligned phonemic transcription /zi:bK2/ at the bottom. "SIL" signals silences. For both 5.2a and 5.2b, blue lines show P@10 when ablation is carried out left to right, meaning that at any given part on the blue curve, the model has only had access to the rightmost part of the word. (e.g. /i:bK2/ without initial /z/). Red line displays scores when ablation was carried out right to left, meaning that at any given part on the red curve, model has only had access to the leftmost part of the word. (e.g. /zi:/ without the final /bK2/).

Figure 5 . 3 :

 53 Figure 5.3: Evolution of P@10 for each ablation step of the word 5.3a "tennis racket" and 5.3b "fire hydrant" with time aligned phonemic transcription at the bottom

Figure 5 . 4 :

 54 Figure 5.4: Figure5.4a shows evolution of the cosine similarity between the embeddings produced for each truncated version of the target word and the embedding for the full word using a model with randomly initialised weights. Figure5.4c shows the same measure with the embeddings produced by a trained model. Figure5.4b shows peaks indicating the inflection points of curve 5.4a (green) and 5.4c (red). For our experiments, we only considered inflection point to be significant if the resulting peak was higher than 0.025 (blue line).

d 1 , x d 2 ,

 2 ..., x d T and a corresponding sequence of scalars B of length T representing boundaries B = [b 1 , b 2 , ..., b T ], b t ∈ {0, 1}, where b t 1 if x t is a segment boundary, 0 otherwise.

Figure 6 . 1 :

 61 Figure 6.1: Graphical representation of the different GRUs used in our experiments: Figure6.1a shows a Vanilla GRU. Figure6.1b shows GRU PACK. in the all condition where all the vectors produced at each time step are passed on to the next layer. 6.1c shows GRU PACK. in the keep condition where only the last vector of a segment is passed on to the next layer, thus resulting in an output sequence shorter than the input sequence. The red crosses inscribed in a square ( ) signal that the output vector computed at a given timestep is not passed on to the next timestep and that the initial state h 0 is passed on instead. The red crosses inscribed in a circle ( ) signal that the output vector computed at a given timestep is not passed on to the next layer. Dotted lines group vectors belonging to a same segment (either phone, syllable-connected, syllable-word, or word). Note that h 0 is only passed on to the next state at the end of a segment, thus effectively materialising a boundary by manually resetting the history. Also note that the x 1 , x 2 , ..., xt figured in this representation could either be the original input sequence (in our case, acoustic vectors) or could also be the output of the previous recurrent layer

Figure 6 . 2 :

 62 Figure 6.2: Graphical representation of a hierarchical architecture that uses two GRU PACK.layers. The 1 st GRU PACK. layer is supplied with phone boundary information. This layer outputs vectors representing phones that are recomposed from the input vectors. The next layer, GRU PACK. -2 takes these phone vectors as input and recombines them into words and outputs vector representing whole words. This architecture effectively models the hierarchical nature of speech, where words consist of a sequence of phones. Note that both GRU PACK. are in the keep condition where only the last vector of a given unit is kept. As in Figure6.1, note that x 1 , x 2 , ..., xt need not be acoustic vectors, but could also be the output of the previous vector.

3. 1

 1 Example sentence in Japanese taken from the test set. . . . . . . . . . . . . 3.2 Example word oversegmented by KyTea . . . . . . . . . . . . . . . . . . . . 4.1 Results obtained on a speech-image retrieval task on the COCO and STAIR data sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Recall at 1 (averaged over 5 runs ± standard deviation) of trained models where attention weights are randomly shuffled. . . . . . . . . . . . . . . . . 4.3 Dsitribution of the attention peaks above words for the COCO and STAIR data sets. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Results obtained on a speech-image retrieval task on the Flickr8k data set . 4.5 R@1 of models where attention weights are randomly shuffled. . . . . . . . 4.6 Repartition of the attention peaks above words for the Flickr8k data sets. . 5.1 Factors influencing word recognition performance in our model. . . . . . . . 6.1 Mean recalls at 1, 5, and 10 (in %) on a speech-image retrieval task COCO and Flickr8k in the baseline condition on the test set . . . . . . . . . . . . 6.2 Maximum R@1 (in %) for each model trained on the COCO data set with one GRU PACK. layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Maximum R@1 (in %) for each model trained on the Flickr8k data set. The same naming conventions of Table

  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . B.1 Top 40 highlighted words on the COCO data set by GRU1. . . . . . . . . . B.2 Top 40 highlighted words on the COCO data set by GRU5. . . . . . . . . . B.3 Top 40 highlighted words on the STAIR data set by GRU1. . . . . . . . . . B.4 Top 40 highlighted words on the STAIR data set by GRU5. . . . . . . . . . B.5 Top 40 highlighted words on the Flickr8k data set by GRU1. . . . . . . . B.6 Top 40 highlighted words on the Flickr8k data set by GRU5. . . . . . . .

  ont montré que les enfants se servent effectivement d'indices phonotactiques afin de segmenter la parole en sous unités et segmentent une séquence [...]C•CVC•C[...] en [...]C#CVC#C[...] si C•C constituent des séquences phonotactiques plus susceptibles d'apparaître en frontière de mot qu'à l'intérieur d'un mot. MacKenzie et al. (

L

  'architecture que nous utilisons est basée sur celle de Chrupa la et al. (2017a) et est construite au moyen de la librairie Python Theano (James Bergstra et al. 2010). Comme toutes les architectures neuronales de parole visuellement contextualisée, cette architecture à deux parties principales, un encodeur d'image et un encodeur de parole. Cette architecture est entraînée pour réaliser une tâche de recherche d'image à partir d'une description audio. Ainsi, lorsqu'une description orale d'une image est donnée en entrée, le réseau retrouve l'image qui correspond à la description. Pour ce faire, le réseau projette l'image et le signal de parole dans un espace de représentation commun, de sorte que l'image et sa description correspondante soient proches dans l'espace de représentation alors qu'une image et une description ne formant pas une paire (la description correspond à la description d'une autre image) soit éloignées dans l'espace de représentation.

  9) où T est la longueur totale de la séquence, h GRU 1 t et h GRU 5 t représentent respectivement le vecteur caché calculé au moment t par la 1 e et la 5 e couche récurrente, et α GRU 1 t et α GRU 5 t sont les poids d'attention calculés respectivement par le mécanisme d'attention suivant la 1 e et la 5 e couche récurrente. Le vecteur final est normalisé à la norme unitaire 2 .

  FLICKR8k

5. 4 . 1

 41 Résultats de référence et attention aléatoire Comme observé précédemment pour l'ensemble de données COCO, nos résultats sont inférieurs à ceux rapportés par Chrupa la et al. (

5 .

 5 Attention dans un modèle de parole visuellement contextualisée 1975.6 ConclusionDans ce chapitre, nous avons entraîné des modèles PVC sur deux langues (anglais et japonais) et nous avons analysé le comportement de leurs mécanismes d'attention. Notre analyse a révélé que l'attention a adopté un comportement général par lequel elle apprend à détecter et à mettre en évidence des noms spécifiques dans l'entrée parlée. Nos expériences confirment ainsi l'intuition de Chrupa la et al. (2017a) selon laquelle "le mécanisme d'attention du modèle de parole lui permet de sélectionner des fragments clefs des [...] énoncés". Nous avons également montré que l'attention pouvait aussi adopter un comportement spécifique à la langue que le réseau traite en mettant par exemple en avant les particules du japonais. Finalement, nous avons montré que la focalisation de l'attention sur les noms est un comportement appris, et qui apparaît avec relativement peu d'exemples.

  Le fait que les modèles PVC soient capables de reconnaître des mots individuels a déjà été exploré par plusieurs études. Par exemple, Chrupa la et al. (2017a) et plus récemment Merkx et al. (2019) ont montré que les embeddings d'énoncés calculés par les modèles PVC basés sur les RNN contiennent des informations sur la présence des mots de l'énoncé d'entrée.

3

 3 Activation abrupte ou graduelle ?Nos expériences révèlent que de petites différences acoustiques produisent de grandes différences dans la représentation finale. Il semble que certains vecteurs MFCC jouent un rôle plus important que d'autres dans l'activation de la représentation finale.

d 1 , x d 2 ,

 2 ...., x d T ; et une séquence correspondante de scalaires B de longueur T représentant les frontières B = [b 1 , b 2 , ..., b T ], b t ∈ {0, 1}, où b t 1 si x t est une limite de segment, 0 sinon.

7. 1 . 3

 13 Condition ALL et KEEP À partir de cette configuration initiale de GRU PACK. , nous en avons proposé deux versions différentes : all et keep. Dans la version all (voir Figure 6.1b), tous les vecteurs appartenant à un segment sont transmis à la couche suivante. Dans la version keep, seul le dernier vecteur de chaque segment est transmis à la couche suivante (voir Figure 6.1c).

•

  Activation des mots dans les modèles basés sur les CNN. Nous avons étudié comment les modèles basés sur les RNN stockent les unités lexicales et activent la représentation des mots individuels en utilisant le paradigme de gating. Cette méthodologie pourrait également être appliquée pour analyser les représentations apprises par les modèles basés sur les CNN (tel que celuide Harwath & Glass (2017)) afin de comprendre comment la représentation d'un mot donné est activée dans ces modèles. Un modèle CNN tel que[START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] nous permettrait de reproduire directement les expériences linguistiques qui mesurent la reconnaissance et la compétition des mots à l'aide de dispositifs d'oculométrie tels que celuide Huettig & McQueen (2007).• Image-To-Speech. Nous aimerions également étudier comment des réseaux Image-To-Speech (tels que ceux de Hsu et al. 2020 et Wang et al. 2020), qui produisent de la parole à partir d'une image d'entrée, apprennent progressivement à produire leurs premières phrases, et comparer cette évolution à celle des enfants. En effet, lorsque les enfants apprennent à parler, ils ne commencent pas par prononcer des phrases complètes, mais plutôt par produire des mots isolés. Plus tard, ils produisent des phrases de deux mots et, à partir de ce moment, ils commencent à produire des phrases complètes. Il serait intéressant d'étudier si un modèle Image-To-Speech passe par les mêmes étapes que les enfants et, dans le cas contraire, d'en étudier les raisons. De même, une telle expérience permettrait de mieux comprendre le développement linguistique des enfants en étudiant les représentations apprises par un tel modèle. • Segmentation discrète. Les expériences menées dans le chapitre 6 révèlent que le fait de donner une segmentation explicite améliore la capacité des modèles à faire correspondre correctement une légende parlée à son contexte visuel. Il serait donc souhaitable que le réseau apprenne à segmenter explicitement l'entrée parlée en sousunités. Plusieurs options, telles que celles proposées par Kreutzer & Sokolov (2018), Chen et al. (2019) et Shain & Elsner (2020) sont à explorer.

  Contents 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67

3.2 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.2.1 COCO and STAIR . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.2.2 FLICKR8k . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.2.3 Data Format . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65 3.2.4 Metadata . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 3.3 3.3.1 Encoders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 3.3.2 Contrastive Loss Function . . . . . . . . . . . . . . . . . . . . . . . 68 3.3.3 Attention Mechanism . . . . . . . . . . . . . . . . . . . . . . . . . . 70 3.3.4 Assumptions in the Model . . . . . . . . . . . . . . . . . . . . . . . 70 3.4 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

Table 3 .

 3

1: Example sentence in Japanese taken from the test set.

Table 3 . 2

 32

			作業車
	KyTea Tags sagyō/N sha/SUF
	Gloss	work	vehicle
	Translation	work vehicle

: Example word oversegmented by KyTea

  3.3.1 EncodersContrary to[START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF] and[START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF] who use a full VGG network as image encoder, the image encoder of our architecture simply consists of a linear layer. Indeed, as previously mentioned (see3.2.3), our architecture uses pre-computed VGG vectors instead of raw images. The image encoder thus learns how to reduce the original 4096 dimensional input vector to a lower dimension. The resulting vector is then normalised to the unit 2 -norm.13 Note that it would still be possible to use full image encoder -such as a VGG or ImageNet encoder. However, it would substantially lengthen the training time and would be computationally heavier. As we wish to focus on the representation learnt for speech, we deemed such modification unnecessary and kept the original architecture. Using vector extracted from a trained neural network (such as done by Chrupa la et al. 2015 and hence this thesis, orMerkx et al. (2019)) or using a the full pretrained network (such as[START_REF] Harwath | Learning Word-Like Units from Joint Audio-Visual Analysis[END_REF]) is a common practise known as transfer learning. Nonetheless, such transfer learning is not necessary, and VGS models are still able to converge -sometimes even better -when the image encoder is trained from scratch such as shown by Mortazavi

Table 4 .

 4 1: Recall at 1, 5, and 10 (in %) as well as median rank r (±standard deviation) on a speech-image retrieval task on the test set of our data sets (5k images) averaged over five runs with different seeds. Models were selected according to the highest R@1 on the validation set. We report the results obtained by Chrupa la et al. (2017a) with the original RHN implementation. Chance R@k are 0.0002/0.001/0.002, chance median rank r is 2500.5.

		5.52 ± 0.31 18.26 ± 0.82 28.64 ± 1.15 27.4 ± 1.51
	Japanese (STAIR)	5.3 ± 0.16 17.88 ± 0.41 27.92 ± 0.36 29.2 ± 0.44
	English (Chrupa la et al. 2017a)	11.1	31.0	44.4	13

Table 4 . 2

 42

		. Attn GRU1	Rand. Attn GRU1	True Attn. GRU1
		Rand. Attn GRU5	True Attn GRU5	Rand Attn GRU5
	COCO	0.32 ± 0.13	2.66 ± 1.17	1.02 ± 0.18
	STAIR	0.20 ± 0.07	1.94 ± 0.65	1.00 ± 0.16

: Recall at 1 (averaged over 5 runs ± standard deviation) of trained models where attention weights are randomly shuffled.

Table 4 . 3 :

 43 Dsitribution of the attention peaks above words for the COCO and STAIR data sets.

	Data Set Attn. Pos.	Beginning	Middle-Beginning	Middle-End	End
	COCO	GRU1 GRU5	17.47 ± 10.95 3.90 ± 0.48	23.31 ± 11.65 14.11 ± 2.07	44.46 ± 28.58 14.75 ± 7.25 43.32 ± 1.73 38.66 ± 2.50
	STAIR	GRU1 GRU5	17.98 ± 4.29 20.69 ± 1.29	25.03 ± 0.77 14.56 ± 1.02	35.71 ± 2.18 21.28 ± 4.08 22.07 ± 2.57 42.68 ± 0.87

Table 4 .

 4 2.08 ± 0.24 7.96 ± 0.57 12.94 ± 1.20 113 ± 9.08

	Chrupa la et al. (2017a)	5.5	16.3	25.3	48

4: Recall at 1, 5, and 10 (in %) as well as median rank r (±standard deviation) on a speech-image retrieval task on the test set of our data sets (1k images) averaged over five runs with different seeds. Models were selected according to the highest R@1 on the validation set. We report the results obtained by Chrupa la et al. (2017a) with the original RHN implementation. Chance R@k are 0.001/0.005/0.01, chance median rank r is 500.

Table 4 .

 4

	Data Set Attn. Pos.	Beginning	Middle-Beginning Middle-End	End
	Flickr8k	GRU1 GRU5	12.09 ± 2.28 18.72 ± 7.95	31.18 ± 2.75 29.38 ± 5.03	42.35 ± 2.00 14.38 ± 2.74 31.05 ± 6.58 23.84 ± 2.98

6: Repartition of the attention peaks above words for the Flickr8k data sets.

Table 5 .

 5

	Estimate	p-value

1: Factors influencing word recognition performance in our model.

Table 6 .

 6 1: Mean recalls at 1, 5, and 10 (in %) on a speech-image retrieval task COCO and Flickr8k in the baseline condition on the test set (based on the results of the best epoch on the validation set). Chance scores are 0.0002/0.001/0.002 for COCO and 0.001/0.005/0.01 for Flickr8k. Results are different than those presented in Chapter 4 as we use a different code that uses a different Deep Learning framework.

Table

				COCO | all condition
	GRU	Phones	Syl.-Co.	Syl.-Word	Word
	Pack.	T	R	T	R	T	R	T	R
	5	09.7 + 09.4	09.1	09.5	09.5	09.4	09.3	09.5
	4	09.3	09.2	09.4	09.1	09.6	09.2	09.0	09.4
	3	09.5	09.2	09.2	09.1	09.4	09.1	09.4	09.2
	2	09.4	08.9	09.7 + 09.1	09.6	09.2	09.7 + 08.9
	1	09.8 + 09.4	09.6	09.4	10.0 + 09.1	09.5	09.1
				COCO | keep condition
	GRU	Phones	Syl.-Co.	Syl.-Word		Word
	Pack.	T	R	T	R	T	R	T	R
	5	9.4	9.6	9.1	9.1	9.6	9.1	9.4	8.7
	4	10.0 +	10.5 + 10.2 +	9.6	10.4 +	9.9 +	10.6 + 9.5
	3	10.5 +	10.1 + 10.4 +	9.8 + 10.5 +	10.1 + 11.0 + 9.7
	2	10.7 + 9.8 +	10.5 + 9.4	10.9 + 9.3	11.3 + 8.8
	1	10.1 + 7.9 -	9.7	7.1 -10.2 + 7.0 -	10.3 + 7.0 -

Table 6 .

 6

				Flickr8k | all condition
	GRU	Phones	Syl.-Co.	Syl.-Word	Word
	Pack.	T	R	T	R	T	R	T	R
	5	4.0	3.9	4.1	4.1	4.3	3.9	3.4	4.2
	4	4.0	4.4	3.9	4.1	4.3	3.8	4.5	4.5
	3	4.5	4.4	4.3	4.2	4.4	4.2	4.5	3.8
	2	4.5	3.8	4.8	3.6	4.4	4.2	4.7	4.1
	1	4.3	3.4	4.0	4.0	4.4	4.3	5.3 + 4.1
				Flickr8k | keep condition
	GRU	Phones	Syl.-Co.	Syl.-Word		Word
	Pack.	T	R	T	R	T	R	T	R
	5	3.6	3.7	3.6	2.5 -3.3	3.0	3.2	3.2
	4	3.8	3.8	4.4	3.5	3.9	2.6 -5.2 + 2.5 -
	3	4.9 + 3.8	4.5	3.1	5.3 + 3.1	4.9 + 3.3
	2	4.8 +	3.9	5.1 + 3.6	4.8	3.4	5.4 + 3.4
	1	4.8	2.4 -3.4	1.9 -4.4	2.0 -3.9	1.9 -

3: Maximum R@1 (in %) for each model trained on the Flickr8k data set. The same naming conventions of Table

6

.2 are used for this table.

Table 6 .

 6 GRU PACK. Pos.

	Configuration true Phones	true Phones	random Phones	random Phones
		true Words	random Words	true Words	random Words
	2 and 3	8.2	4.3	3.6	1.0

5: R@1 obtained on the test set of the Flickr8k data set with a hierarchical architecture consisting of two GRU PACK. layers using random phones and/or random word boundaries (models were selected based on the maximum R@1 on the validation set). Best score overall is shown in red. Best score (layer-wise) is shown in bold.

  Table B.1: Top 40 highlighted words on the COCO data set by GRU1. Table B.2: Top 40 highlighted words on the COCO data set by GRU5. Table B.3: Top 40 highlighted words on the STAIR data set by GRU1. Table B.4: Top 40 highlighted words on the STAIR data set by GRU5. Table B.5: Top 40 highlighted words on the Flickr8k data set by GRU1. Table B.6: Top 40 highlighted words on the Flickr8k data set by GRU5.

												129 131 133
	Word Word			Word TRUE Peaks TRUE Peaks TRUE Peaks Word TRUE Peaks	RANDOM Peaks RANDOM Peaks RANDOM Peaks RANDOM Peaks
	Rank 1 Rank ga 2 no 1 2	Word Word ga no	1 Translation table Rank 1 train Word Translation Rank Word SUBJ SUBJ GEN GEN 1 dog 1 </s>	3.19 % peak 2.04 % peak 3.88 </s> </s> Word </s> Word END OF SENTENCE 9.23 % Peak 9.15 Translation % peak Word Translation % Peak Word % Peak 3.17 i+ru to be 13.18 </s> END OF SENTENCE 5.35 i+ru to be 13.49 <s> 15.64 5.29 <s> 15.51	% Peak % Peak 9.96 5.73 9.97 5.65
	3	o 3	o	OBJ	2 OBJ 2	2 2	a man tennis water	2.22 1.73 2.78 7.34 4.01 no 3.51	a </s> a no </s> GEN 3.48 5.35 3.47 GEN 5.25	3.99	4.00
	4 5 6	3 4 5 sukeetoboodo skateboard neko cat 4 ni LOC, ALL train baseball room 3 toilet 4 baseball 5 skateboard 1.46 1.89 1.64 1.61 1.53 1.50 2.18 2.00 3 girl 5.42 3 <sil> 2.80 ga ni 3.34 hikou part of "hikouki" ("aeroplane") "hikou" ("aviation") 1.62 <s> 5 </s> END OF SENTENCE 1.67 6 kirin giraffe 1.61 4 boy 5.38 4 beach 1.91 7 inu dog 1.39 5 dogs 4.02 5 snow 1.77	<s> on of <s> on with ga <sil> <sil> SUBJ 2.48 2.04 1.30 LOC, ALL 2.47 1.96 1.26 SUBJ 3.51 3.41 START OF SENTENCE ni LOC, ALL <s> a 3.02 a 2.95 START OF SENTENCE 2.47 3.50 3.45 2.58 2.50 2.55 o OBJ 2.05 in 1.80 in 1.73
	7 8 9 13	6 7 8 9 10 11 12 13 14 15 16 17 18 19 Personnal Computer bathroom of in sitting giraffe bus skateboard 0.94 1.52 1.23 1.11 1.03 0.97 0.95 on 0.93 kitchen 0.90 with 0.84 ball 0.84 cat 0.77 people 0.76 </s> 0.76 6 dog 1.45 7 cat 1.44 8 giraffe 1.39 9 pizza 1.35 kitchen 1.35 street 1.34 sign 1.26 bench 1.15 clock 1.12 bed 1.10 snow 1.10 cake 0.99 motorcycle 0.98 room 0.96 ski 1.27 1.26 horse 6 people 3.37 6 grass 1.74 o dansei with in man standing 1.07 1.27 1.21 1.12 and 1.02 sitting 1.00 is 0.88 <sil> 0.86 the 0.84 table 0.83 people 0.77 woman 0.70 next 0.65 holding in of man and standing sitting <sil> is the people table street woman next OBJ man 1.27 dansei man 1.65 1.22 1.18 1.08 1.04 1.02 0.93 0.87 0.87 0.86 0.79 0.76 0.66 0.63 man dog 1.65 pizza 1.21 te particle of reason, state bike 1.18 te particle of reason, state 7 woman 3.23 dog 1.53 7 shirt 1.46 man 1.59 surfboard 1.17 de LOC bus 1.17 a+ru to be bike 1.16 josei woman sukeetoboodo skateboard pasokon sukii 8 uma piza 9 baiku saafuboodo basu baiku 1.11 a+ru to be keeki cake 1.09 de de 8 child 2.12 on 1.21 8 street 1.35 on 1.15 shimauma zebra 0.98 josei woman 9 ball 1.82 the 1.05 9 a 1.13 the 1.09 uma horse 1.16 shiro+i white dansei man 0.97 to enumeration particle 10 water 1.49 is 0.89 in 0.96 is 0.95 banana banana 0.98 to enumeration particle pasokon PC 0.96 shiro+i white 11 girls 1.38 two 0.84 standing 0.89 two 0.89 zou elephant 0.96 <sil> <sil>ENCE shingou traffic light 0.81 takusan lots of to enumeration part. 0.81 sukeetoboodo skateboard nuigurumi stuffed toy 0.96 <sil> SILENCE sha 12 children 1.26 with 0.80 is 0.69 woman 0.79 end of jitensha ("bus") 0.93 takusan lots of 13 boys 1.09 water 0.79 player 0.67 with 0.75 mistakenly tagged as particle saafin surf 0.80 kuro+i black tokei watch 0.88 no+tte riding 14 basketball 0.94 and 0.78 to 0.63 boy 0.75 shimauma zebra 0.77 shi hyoushiki sign, mark 0.88 aka+i red 15 men 0.91 people 0.77 field 0.62 black 0.74 part of "shiteiru" ("is/are doing"), inflected form "suru" (to do) mo+tte holding 0.76 no+tte riding de LOC 0.73 aka+i red kuruma car 0.73 hito man furisubii frisbee 0.85 kuro+i black piza pizza 0.85 shi four 16 a 0.90 black 0.76 dirt 0.61 girl 0.74 kuruma car 0.85 hito man 17 on 0.90 boy 0.71 air 0.59 and 0.72 shingou sign 0.75 teeburu table 18 while 0.87 woman 0.70 pool 0.57 people 0.69 0.61 kasa umbrella 0.72 ta+tte even if benchi bench 0.74 pasokon PC 19 walking 0.80 white 0.66 swing 0.57 white 0.67 0.58 20 and 0.74 two 0.58 21 horse 0.74 street 0.57 22 board 0.73 large beach 0.92 holding 0.59 bus 0.91 person 0.57 bathroom 0.86 two 0.56 kuma bear 0.71 toma+tte stopping kiiro+i white 0.70 naka inside benchi bench 0.67 mo+tte with denwa train 0.63 suwa+tte sitting kitchin kitchen 0.73 naka inside de LOC 0.71 ta+tte even if 20 wall 0.80 girl 0.64 snowboarder 0.56 wearing 0.65 burokkorii broccoli 0.71 suwa+tte sitting 21 person 0.79 wearing 0.62 snowy 0.54 dogs 0.62 woman woman 0.67 ue above 22 black 0.76 playing 0.56 sitting 0.53 water 0.57 0.57 bear 0.85 large 0.52 kitchin kitchen 0.63 ue above tenisu tennis 0.62 mo+tte with 23 white 0.75 dogs 0.53 sunglasses 0.53 playing 0.57	2.02 1.68 1.61 0.97 0.95 0.85 0.79 0.74 0.64 0.62 0.54 0.46 0.44 0.44 0.44 0.41 0.41 0.39 0.37 0.36 0.36 0.33	1.79 1.64 0.95 0.95 0.91 0.75 0.74 0.66 0.65 0.46 0.45 0.45 0.44 0.42 0.40 0.38 0.36 0.36 0.35 0.35 0.35
		orenji doonatsu douro sunooboodo toire hitsuji beddo	23 24 25 snowboard orange donut road toilet 24 sheep 25 bed 26		woman parked truck sink horse laptop snowboarder 0.73 0.72 0.69 0.69 0.84 0.84 0.83 0.61 0.59 0.59 0.59 soccer 0.48 ookina person white to white to bathroom 0.56 0.55 0.54 big ta PST hashi+tte running re part of "reru" (passive) voice 0.52 0.51 0.47 0.62 re causative particle standing 0.51 standing 0.53 used for verbs 0.62 ta PST rock 0.72 of 0.48 through 0.47 young 0.53 0.61 toma+tte stopping football 0.71 young 0.46 ocean 0.47 child 0.46	0.33 0.33 0.32 0.31	0.34 0.34 0.33
		beddo to	bed	26 enumeration particle umbrella phone 27 bike sand	0.68 0.79 0.59 0.69 0.46 kodomo field field 0.61 neko child person children 0.47 0.46 0.44 cat 0.46	0.31	0.32
		suwa+tte yuki sara takusan ryouri raketto doonatsu re ha kanban ni mae densha terebi senmen	27 28 29 30 snowboard sitting plate lots of cookery racket 28 donut 29 nominalising suffix snow standing car grass skis food elephant plate little the snow surfer 30 wearing on TOP signboard 31 women outside 31 teddy 32 dog 33 boat 34 bowl giraffes frisbee boat building LOC, ALL in front train 32 skateboarder 0.51 0.68 0.67 0.67 0.65 0.73 0.73 0.73 0.72 0.57 0.53 0.52 0.50 0.64 0.44 furisubii plate cat building 0.37 0.44 0.37 top train skateboard 0.43 0.44 baseball 0.42 riding frisbee teeburu table tokei watch kirin giraffe 0.60 sukeetoboodo skateboard 0.58 ookina big person 0.42 of 0.45 0.58 kuruma car 0.61 ball 0.41 0.44 brown 0.44 0.55 nuigurumi stuffed toy 0.55 grass 0.40 0.42 shirt 0.43 0.41 0.50 saafuboodo surboard 0.55 kodomo children 0.55 red 0.40 0.42 red 0.42 0.37 0.60 group 0.36 0.59 tennis 0.36 0.58 water 0.35 0.57 down 0.71 down 0.41 0.70 cat 0.37 0.70 water 0.35 0.67 top 0.49 ao+i blue 0.48 neko cat 0.54 ki+ta wear.PST brown 0.40 her 0.42 running 0.42 television 0.52 sukii ski 33 <sil> 0.47 beach 0.39 his 0.42 small 0.40 part of "senmendai" ("washbasin"), 0.51 o+ka to put 34 with 0.46 shirt 0.38 with 0.41 grass 0.40 senmen∼washing 0.34 35 skateboard 0.46 blue 0.38 grassy 0.41 little 0.38 0.35 35 surfboard 0.55 walking flying 0.65 group 0.34 36 large 0.41 little 0.36 bicycle 0.41 are 0.38 0.35 36 park 0.51 riding surfboard 0.64 small 0.34 37 walks 0.41 jumping 0.35 camera 0.40 blue 0.37 0.35 37 traffic 0.51 dog sheep 0.59 walking 0.34 38 rides 0.41 running 0.34 skateboard 0.40 children 0.37 0.34 38 motorcycle 0.50 pizza is 0.59 building 0.33 39 walk 0.36 large 0.34 small 0.40 ball 0.37 0.34 39 tower 0.50 small 0.34 sandwich 0.57 young 0.33 40 in 0.36 are 0.33 together 0.39 jumping 0.35	0.30 0.30 0.29 0.29 0.28 0.28 0.28	0.31 0.31 0.31 0.30 0.30 0.29 0.29 0.28
					40		hydrant	0.49	black	0.33

3

  Informations générales sur le traitement automatique de la parole et la découverte d'unités lexicales 3.1 Traitement non supervisé de la parole & notion de grounding Le travail que nous menons dans cette thèse s'inscrit dans le cadre du traitement non supervisé de la parole, et la principale motivation de celle-ci est de concevoir des modèles de traitement automatique de la parole plus proches du traitement humain de la parole, qui ne requiert pas ou peu de supervision. Par exemple, un enfant qui apprend sa langue maternelle n'a pas besoin d'étiquettes textuelles, mais utilise plutôt des signaux de supervision faibles, tels que des indices visuels. Ces signaux constituent des signaux de supervision faibles, car ils limitent le processus d'apprentissage en l'ancrant dans le contexte.

  3. Informations générales sur le traitement automatique de la parole et la découverte d'unités lexicales 179 externes, les modèles de reconnaissance vocale peuvent être considérés comme des modèles textuellement ancrés, puisque le texte, qui est une connaissance externe, est ajouté au modèle. Roy (2005) définit l'ancrage comme suit : La relation entre les mots et le monde physique, et par conséquent notre capacité à utiliser des mots pour faire référence à des entités dans le monde, fournit les fondements de la communication linguistique. Les approches actuelles de la conception des systèmes de traitement du langage passent à côté de cette connexion critique, qui est obtenue par un processus que j'appelle grounding. [...] Le grounding linguistique fait référence aux processus spécialisés dans la mise en relation des mots et des actes de parole avec l'environnement de l'utilisateur de la langue. (Roy 2005)

  se voir accorder plus d'importance dans la représentation finale, et ce de manière non supervisée. Un mécanisme d'attention peut être formalisé comme suit : très générale : la fonction de notation score (généralement un MLP) calcule une valeur scalaire pour chaque h t . Cependant, la manière précise dont cette valeur scalaire est calculée dépend de l'implémentation particulière du mécanisme d'attention et peut ne pas être uniquement une fonction de h t . Avec un tel mécanisme d'attention, les vecteurs pour lesquels le poids de l'attention α est élevé seront fortement représentés dans le vecteur final c, tandis que les vecteurs auxquels on a attribué une faible valeur scalaire α seront peu ou pas représentés dans le vecteur final c.Le but du réseau est de prédire, à partir d'une entrée x, une sortie ŷ qui soit aussi proche que possible de la sortie désirée y. Le succès de l'opération est mesuré à l'aide d'une fonction de coût. Pendant l'apprentissage, le réseau est encouragé à minimiser la différence entre la sortie prédite et la sortie réelle, et la valeur calculée par la fonction de coût servira de base pour mettre à jour tous les poids du réseau afin d'atteindre son objectif.

	c = où c, le vecteur de contexte, est la somme pondérée des vecteurs cachés, et α t est le T t=1 α t h t avec α t = exp(score(h t )) T t =1 exp(score(h t )) (D.4) poids d'attention pour le pas de temps t, où α t ∈ [0, 1] et T t=1 α t = 1. La formalisation que nous présentons est 3.4.1 Fonction de coût et rétropropagation L(y, ŷ) = |y -ŷ| (D.5)

  Contrairement aux CNN, les RNN sont conçus pour traiter des séquences telles que la parole. Même si les CNN sont également capables de traiter des séquences, les RNN sont plus plausibles d'un point de vue cognitif. En effet, comme nous l'avons déjà mentionné, la sortie à un temps donné dépend des prédictions des pas de temps précédents, ce qui n'est pas le cas pour les CNNs.Le premier modèle PVC à base de RNN que nous connaissons est celui de Chrupa la et al. (2017a). L'architecture utilisée est très similaire à celle des travaux précédemment cités et comporte deux branches : une branche audio et une branche visuelle ; et utilise le même type de fonction de coût que[START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF]. De même que[START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], ils utilisent des vecteurs VGG pour représenter l'image. Néanmoins, contrairement à[START_REF] Harwath | Unsupervised Learning of Spoken Language with Visual Context[END_REF], l'encodeur acoustique est constitué de Recurrent Highway Networks (RHN,[START_REF] Zilly | l'utilisation conjointe de syllabes et de mots)[END_REF]) empilés au lieu de couches convolutives. L'intégration finale de la branche audio est calculée par un mécanisme d'attention (voir section 3.4) qui apprend à pondérer les vecteurs d'entrée, de manière à donner plus de poids à des parties spécifiques du signal vocal. Ils évaluent leur modèle sur une tâche de recherche d'images vocales, c'est-à-dire qu'à partir d'une légende parlée, le réseau doit retrouver l'image correspondante. Ils obtiennent de meilleurs résultats que[START_REF] Harwath | Deep multimodal semantic embeddings for speech and images[END_REF], montrant que les modèles basés sur les RNN sont également capables de mettre en correspondance les deux modalités de manière adéquate.Plus récemment,Merkx et al. (2019) se sont basés sur l'architecture de Chrupa la et al. (2017a), en ajoutant un mécanisme d'attention à chaque couche (implémenté comme dans Chrupa la et al. 2017a), et en utilisant des cellules récurrentes bidirectionnelles au lieu de cellules unidirectionnelles. Leur modèle a atteint des niveaux de performance encore plus élevés que Chrupa la et al. (2017a) et Harwath & Glass (2015) sur le même jeu de données. Plusieurs optimisations ont également été faites, comme l'utilisation du taux d'apprentissage cyclique (cyclic learning rate, Smith 2017) qui consiste à avoir un taux d'apprentissage qui augmente et diminue plusieurs fois entre une borne supérieure et une borne inférieure, au lieu d'avoir un taux d'apprentissage strictement décroissant comme ce qui est fait habituellement. Cela permet au modèle de sortir des minima locaux dans lesquels il pourrait être bloqué et de converger vers une solution globalement meilleure.Cependant, ces meilleurs résultats se font au détriment de la plausibilité cognitive. En effet, comme ce modèle utilise des cellules bidirectionnelles, il traite l'entrée parlée des deux côtés en même temps, ce qui n'est bien sûr pas possible pour les humains.Ainsi, les modèles basés sur les RNN sont donc capables d'apprendre la correspondance parole/image de manière aussi fiable que les modèles basés sur les CNN. Ils semblent également être des modèles assez flexibles, car ils sont capables d'apprendre rapidement la 3. Informations générales sur le traitement automatique de la parole et la découverte d'unités lexicales 185 correspondance entre de nouvelles paires objet/nom comme le montre les travaux de Krishnamohan et al. (2020). Le fait que ces modèles utilisent des cellules RNN au lieu de CNN les rend également plus plausibles sur le plan cognitif, et constituent donc des bancs d'essai idéaux pour simuler l'acquisition lexicale. Chrupa la et al. (2017a) ont essayé de comprendre les représentations apprises par leur modèle. Ils ont constaté que toutes les couches ne sont pas également informatives sur la présence d'un mot. En particulier, ils ont constaté que les couches inférieures de leur architecture étaient les moins informatives, tandis que l'avant-dernière était la plus fiable. Cela suggère que les unités ressemblant à des mots sont progressivement construites au fur et à mesure que l'information circule dans le réseau, et qu'une certaine quantité de calcul est nécessaire pour que l'information apparaisse. Une observation similaire a été faite plus récemment par Merkx et al. (2019). Leur étude révèle également que les couches inférieures de leur architecture encodent la forme (c'est à dire, des informations phonétique) tandis que les couches supérieures encodent la sémantique. Alishahi et al. (2017) explorent dans quelle mesure un tel modèle PVC encode la phonologie. Leur étude montre que les encodages du réseau regroupent approximativement les phonèmes de l'anglais par classe de sons (plosives, fricatives, affricées, etc.). Cependant, le regroupement n'est pas parfait et semble se faire sur la base de facteurs acoustiques (formants) plutôt que sur des facteurs linguistiques plus profonds.Harwath et collègues ont également effectué une analyse des représentations apprises par leur modèle. Les travauxde Harwath et al. (2016) suggèrent que leur réseau effectue une segmentation implicite de l'entrée audio en sous-unités.[START_REF] Drexler | Analysis of Audio-Visual Features for Unsupervised Speech Recognition[END_REF] a montré que certains neurones étaient spécifiquement activés par certaines séquences de phonèmes. De façon intéressante, et de façon similaire à l'observation de Chrupa la et al. (2017a) et

	3.5.3 Analyse des représentations

  Des recherches antérieures montrent que les modèles PVC, qu'ils soient basés sur des CNN ou des RNN, sont capables d'apprendre avec succès comment apparier un stimulus acoustique à un stimulus visuel. Ces travaux révèlent également qu'en dépit du fait qu'ils n'ont été entraînés qu'à minimiser la distance entre une image et sa description orale afin d'apparier les deux modalités avec précision, ces modèles ont développé des capacités linguistiques plus profondes. Il s'agit d'une propriété intéressante, car ces capacités linguistiques apparaissent comme un sous-produit de la tâche principale. Cela ressemble en quelque sorte à la façon dont les enfants acquièrent leur langue maternelle.Alors que ces capacités ont été largement explorées pour les modèles basés sur les CNN, elles ne l'ont pas été pour les modèles basés sur les RNN. Nous cherchons donc dans cette thèse à mieux comprendre quelles capacités linguistiques les modèles PVC à base de RNN sont capables de développer. Plus précisément, nous cherchons à répondre aux questions suivantes :

	3. Informations générales sur le traitement automatique de la parole et la découverte
	d'unités lexicales	187
	3.6 Conclusion

•

  Des recherches antérieures(Chrupa la et al. 2017a ; Merkx et al. 2019) montrent que les représentations internes d'un tel réseau encodent la présence des mots individuels des légendes parlées. Cela soulève la question de savoir si ce comportement est valable pour tous les mots d'une légende donnée, ou seulement pour des mots spécifiques ?• Si la présence de mots individuels est encodée dans les représentations internes du réseau, cela signifie que le réseau a appris ce qui constitue un mot et a stocké cette information dans ses poids. Cela soulève la question de savoir comment le réseau reconnaît (c'est-à-dire active) la représentation d'un mot donné sur la base d'une entrée acoustique. • Les modèles PVC sont entraînés avec des légendes complètes non segmentées et obtiennent des résultats corrects. Cependant, leurs homologues textuels (c'est-à-dire les réseaux entraînés sur des légendes écrites) obtiennent de meilleurs résultats. Cela soulève plusieurs questions, la principale étant : si les réseaux PVC étaient présentés avec des légendes segmentées, obtiendraient-ils de meilleurs résultats ? 4 Modèles de parole visuellement contextualisée et jeux de données Les enfants acquièrent leur langue maternelle en contexte. Ainsi, l'accès à des stimuli uniquement oraux n'est pas suffisant pour qu'ils acquièrent leur langue maternelle. En effet, il est nécessaire que ces stimuli soient contextualisés (grounded ) afin que les enfants puissent faire sens de ce qui est dit autour d'eux. Le contexte leur permet ainsi d'apparier des signes linguistiques (paire signifié/signifiant) à leurs référents. Les modèles neuronaux de parole visuellement contextualisée permettent de modéliser les interactions complexes qui ont lieu entre la modalité visuelle et la modalité auditive. Ainsi, ces modèles peuvent être utilisés afin de simuler l'acquisition du langage chez l'enfant. Ce chapitre présente donc les modèles qui seront utilisés pour les expériences décrites dans cette thèse ainsi que les données nécessaires à l'entraînement de ceux-ci. Les jeux de données que nous utilisons dans cette thèse sont des jeux de données initialement conçus pour une tâche de description textuelle d'images (image captioning). Ces corpus contiennent un ensemble d'images, chacune décrite par 5 descriptions textuelles écrites par des humains. Ces jeux de données sont le corpus MSCOCO (Lin et al. 2014), STAIR Yoshikawa et al. (2017) et Flickr8k (Rashtchian et al. 2010, Hodosh et al. 2013). Nous utilisons des extensions des jeux de données originaux, qui en plus d'avoir une description textuelle de chacune des images, incluent une version parlée de ces mêmes descriptions. L'extension audio de MSCOCO et de STAIR inclut une description audio générée par un système de synthèse vocale, tandis que l'extension audio de Flickr8k n'utilise pas de voix de synthèse mais des voix naturelles. COCO L'extension audio de MSCOCO (Synthetically Spoken COCO Chrupa la et al. 2017b que nous appellerons désormais COCO) que nous utilisons a été créée par Chrupa la et al. (2017a). Celle-ci a été générée en utilisant le système de synthèse vocale de Google et ne comprend qu'une seule voix de synthèse. Ainsi, le jeu de données COCO comprend 616 435 descriptions audio pour un total de 123 287 images. Nous avons conservé le partitionnement original de données, à savoir 566 435 paires audio/image pour l'entraînement, 25 000 pour la validation et 25 000 pour le test. STAIR L'extension audio de STAIR (Synthetically Spoken STAIR Havard et al. 2019b que nous appellerons désormais STAIR) est une extension que nous avons nous-mêmes créée pendant notre thèse. Le jeu de données original STAIR utilise les mêmes images que MSCOCO, mais contient, au lieu de descriptions textuelles en anglais, des descriptions textuelles en japonais. Il est important de préciser que les descriptions en japonais sont des descriptions originales et non des traductions des descriptions anglaises. Nous avons suivi la même méthodologie que Chrupa la et al. (2017a) et avons généré des signaux acoustiques pour chacune des descriptions japonaises en utilisant le système de synthèse vocale de Google. Nous avons conservé le même partitionnement de données que Chrupa la et al. (2017a) et

	4.1 Données

  Dans notre cas, la distance d utilisée est la distance cosinus. Attention L'un des éléments clef de notre modèle est son mécanisme d'attention. On rappelle au lecteur que l'attention se calcule ainsi : est la longueur de la séquence, h t est l'état caché produit par une cellule récurrente au moment t, et α t est un paramètre qui est appris. Rappelons que le mécanisme d'attention apprend à attribuer un poids α t à chaque vecteur d'entrée. Plus le poids est élevé, plus le vecteur a de l'importance dans la représentation finale. Dans notre cas, des poids d'attention élevés sur des vecteurs spécifiques signifient que le réseau a accordé plus d'attention à des parties spécifiques du signal d'entrée. Comme mentionné précédemment, nous utilisons deux modèles d'attentions, l'un après la première couche récurrente, l'autre après la cinquième couche récurrente, le vecteur final étant calculé ainsi :

	T
	c =	α t h t	(D.8)
	t=1
	où T

5

  Attention dans un modèle de parole visuellement contextualisée L'hypothèse principale de cette thèse est que, afin d'apprendre une correspondance fiable entre une image et sa description orale, le modèle devrait implicitement apprendre à segmenter l'entrée orale en sous-unités. Comme les images utilisées représentent principalement des objets et non des actions, nous supposons que le modèle devrait apprendre implicitement à segmenter des unités nominales. Ainsi, nous étudions les mécanismes d'attention de nos modèles et analysons quelles parties du signal d'entrée sont mises en évidence, pour voir si effectivement le modèle met en avant des unités lexicales nominales.

  8. Conclusion 209des mots ne se produit pas de manière linéaire, mais évolue plutôt par étapes. Ceci nous a permis de conclure que le modèle était capable de reconnaître un mot à partir d'une entrée partielle, c'est-à-dire avant la fin de celui-ci. Nous avons montré que le réseau devait nécessairement avoir accès au premier phonème d'un mot pour pouvoir activer la représentation du mot cible. Ce résultat est similaire à ce qui est postulé dans le modèle Cohort de reconnaissance de la parole, où le début des mots est d'une importance particulière pour activer et reconnaître un mot. Nous avons cherché à savoir si la reconnaissance des mots s'effectuait par un processus d'activation simultanée d'une cohorte de mots, qui seraient ensuite en compétition pour la reconnaissance. Nous avons trouvé des preuves d'activations simultanées pour certains mots et de compétition entre eux, cependant, ce processus semble loin d'être systématique, car nous avons montré que certains mots sont activés sans entrer en compétition avec des mots à consonance similaire.• Introduction d'informations linguistiques préalables. Enfin, nous avons cherché à savoir si l'introduction d'informations linguistiques préalables sous la forme d'informations sur les frontières de segments était bénéfique. Nous avons en effet constaté que le réseau pouvait utiliser de manière adéquate de telles informations, en particulier lorsque le réseau recevait des frontières de mots. Le réseau utilise de manière adéquate les informations sur les limites des phonèmes et des syllabes, mais moins que les frontières des mots. Plus important encore, nous avons constaté que la prise en compte de la nature hiérarchique de la parole, en donnant simultanément au réseau les frontières des phonèmes, des syllabes et des mots, donnait des résultats encore meilleurs.

As measured, among other measures, by their preference of well-formed prosodic units over ill-formed units.

C = consonant sound, V = vowel

Though /l/ tends to be realised syllabically [stô aeNgl " z] or [stô aeNg@lz].

Note that both lists featured words comprising very frequent sounds. Hence, the difference can only be explained by the frequency of the sound sequences and not the frequency of the individual sounds.

Most notably in their study, the spoken sample should be pronounced in a Child Directed fashion in order to observe this phenomenon.

The symbol •marks a syllable boundary.

For example, in Turkish "Türkler" (Turkish men) where "ü" /y/ and "e" /E/ are front vowels and "Fransızlar" (Frenchmen) where "a" /A/ and "ı" /W/ are back vowels.

Note that this is a very restrictive (and maybe facilitative) case of vowel harmony as all the vowels in a given word are the same. Such vowel harmony, as far as we know, has never been reported in any of the world's languages.

Probability that the previous syllable is X given the fact that the current syllable is Y is given by BT P = P (X|Y ) = f req.XY f req.Y .

This term, originally introduced in the aforementioned publication, did not refer to child language acquisition (the original study explored the segmentation strategies used by Hungarian that are L2 speakers of English). However, we do believe that this term also applies to child language acquisition as it adequately describes the method used by children to segment the spoken input.

The syllables mo and fa was added respectively at the beginning and the end of the sequence so that the sequence beginning or end would not be used as a cue

They are usually discarded in the statistics of IDS/CDS speech corpora.

From Landau et al. (1988): "subjects were shown a single standard (i.e. the Dax) and told its name ("This is a Dax"). The standard remained in view throughout the entire procedure. Subjects were then shown the seven test objects one by one" Emphasis added.

"Dieses Ergebnis ist hinsichtlich der Art der Sinnesminderung nicht überraschend, weil anzunehmen ist, dass das Fragen ein wichtige Strategie der Blinden darstellt, die Welt zu erforschen." ("This result is not surprising in view of the sensory impairment, because it is understandable that questioning represents an important strategy for the blind [child] to explore the world.").

Original citation in German: "Während sehende Kinder noch vor dem Krabbelalter durch Beobachtung Konzepte wie Ich, Andere, Agent, Handlung und Objekt formulieren, fehlen dem blinden Kind hier entscheidenden Möglichkeiten"

26 "Für das blinde Kind stellt der Wortschatzerwerb eine enorme intellektuelle Leistung dar, da es aufgrund des fehlenden visuellen Inputs Wörter, die sich auf Farbeindrücke, räumliche Distanzen oder Grössordnungen beziehen, nur schwer begreifen kann." ("For the blind child, the acquisition of vocabulary represents an enormous intellectual achievement, as the lack of visual input makes it difficult for him or her to grasp words that refer to colour impressions, spatial distances or dimensions.")[START_REF] Peltzer-Karpf | Spracherwerb bei hörenden, sehenden, hörgeschädigten, gehörlosen und blinden Kindern, number 403[END_REF]

The authors give the sentence "John slept the guitar" as example, where the listener is able to activate the word "guitar" and hence recognise it even if it is grammatically incorrect to use it in this context.

For more details on the features see(McClelland & Elman 1986a, p. 15).

Called "feedback" in the original Trace paper.

It also sets apart from other word recognition models not presented here, see[START_REF] Weber | Models of spoken-word recognition[END_REF] and[START_REF] Magnuson | Computational Models of Spoken Word Recognition[END_REF] for an extensive review.

"developed" should be understood in the sense given by Ferguson(Fishman et al. 1968, p. 31-32), see https://www.ethnologue.com/language-development.

Actually, a distinction is made with a third term, word segmentation, which is similar to unsupervised lexicon discovery, but carried out on string of symbols (e.g. graphemes or phonological transcriptions).

https://zerospeech.com/

The pyscholinguistic motivation tends to disappear in the recent editions(2020, 2021), but is explicit in the first edition (2015): "[provide] adaptable algorithms that [...] aid infant language acquisition research by providing scalable quantitative models that can be compared to psycholinguistic data"[START_REF] Versteegh | The Zero Resource Speech Challenge 2015[END_REF].

Similarly to the b term of a linear regression y = ax + b, where b controls the y-intercept of the function.

as defined by[START_REF] Hochreiter | Long short-term memory[END_REF].

LSTM were introduced to solve the vanishing/exploding gradient problem which we will not discuss here. However, the fact that vanilla RNN cells are unable to keep track of long term dependencies is a direct consequence of the vanishing gradient problem.

Where each type of kernel (edge detector, blur, etc.) was hand crafted. See example https://en. wikipedia.org/wiki/Kernel_(image_processing).

Though 2D convolution may also be used when working with spectrograms.

see https://pytorch.org/docs/stable/generated/torch.nn.Conv1d.html#torch.nn.Conv1d.

This is not entirely true: for example, Reinforcement Learning does not require the loss function to be differentiable. However, in most cases, all the operations are differentiable.

We here take the simplifying assumption that gradient descent is done for only one instance x and not a batch of items.

Children do in fact perceive speech categorically. However, categorical perception for children should be understood in terms of phones (e.g. [b] is perceived differently from [p], or [p] from [p^] or [p h ]) but not in terms of phonemes. It is only later that they learn that the acoustic difference between [b] and [p] is meaningful -and are thus two phonemes /b/ and /p/ -while the acoustic difference between [p] and [p^] and [p h ] is meaningless in English.

Following the objective function used by[START_REF] Karpathy | Deep Visual-Semantic Alignments for Generating Image Descriptions[END_REF] for image/caption pairs, which will also be used in this thesis (see section3.3.2).

RHN are recurrent cells, similar to LSTMs or GRUs, which are however able to perform more transformation steps for each recurrent transition. This allows to learn better representations while reducing the size of the stack of recurrent cells, which in turn makes learning more efficient.

Image Credit: Josh More, Flickr, BY-NC-ND 2.0, MSCOCO ID 4477.

[START_REF] Rieder | Simulation and Modeling[END_REF] however notes that "another viewpoint is that the term modeling includes both the construction of models and the manipulations of these models (the simulations)".

and other models such as[START_REF] Wang | Show and Speak: Directly Synthesize Spoken Description of Images[END_REF] and a model we have been working on during an internship in Japan.

https://cocodataset.org/#home

https://www.flickr.com/

Representing 551h, 25.27h, and 25.24h of speech respectively; with an average duration of 3.6s±0.8/caption

In terms of political power of their native countries, not in terms of absolute number of speakers.

Representing 729h, 32.53h, and 32.14h of speech respectively; with an average duration of 4.6s±1.2/caption

Representing 34.39h, 5.76h, and 5.73h of speech respectively; with an average duration of 4.1s±2.0/caption

http://www.cs.toronto.edu/~vendrov/order/coco.zip

https://cs.stanford.edu/people/karpathy/deepimagesent/flickr8k.zip

which is a softmax over all the dimensions of the vector, which represents the probability that the object represented at dimension n is present in the image.

https://github.com/jameslyons/python_speech_features with default configuration. Default configuration: window size: 25ms, window step: 10ms, pre-emphasis: 0.97

Available at https://clarin.phonetik.uni-muenchen.de/BASWebServices/interface/WebMAUSBasic (visited onFebruary 28, 2020)

https://github.com/gchrupala/visually-grounded-speech

Note that it is not the only possibility.Merkx et al. (2019), for example, uses importance sampling in order to only select the hardest examples inside a batch and obtains better results.

It should be noted that despite being referred to as a distance in the literature, it is not a true distance. However, as in our case both vector A and B are 2 -normed, it is proportional to the Euclidean distance.

We also tried to do a simple sum to merge the two vectors computed by the two attention mechanisms, but we found dot product to be more efficient.

At the time we introduced this modification, using more than one attention mechanism was not a common practice and was popularised by[START_REF] Vaswani | Attention is All you Need[END_REF]. Using multiple levels of attention in a model of VGS was then adopted byMerkx et al. (2019) who uses an attention mechanism after each recurrent layer.

Emphasis added.

Peak detection is done by taking the first-order difference of the input sequence. Python module available at https://bitbucket.org/lucashnegri/peakutils/src/master/.

All models have a similar behaviour.

Bilingual speech-image data sets actually do exist and are based on the data set introduced by Harwath et al. (2016): namely, a Hindi data set Harwath, Chuang & Glass (2018) and more recently a Japanese data

In fact, it is impossible to reproduce the original experimental settings with our model as it does not predict discrete labels. It would be however possible to use the original settings with a network trained to predict word labels, such as an Automatic Speech Recognition model.

Thus consisting only of randomly initialised weights.

Note that phonetic factors are not the only factors taken into account to deactivate a set competitors, morpho-syntactic factors also come at play.

Phonemic transcription found in CMU Pronouncing Dictionary was used

wii/window, cat/cattle, cat/cow, cat/catcher, cattle/catcher, floor/flower, fridge/frisbee, kid/kitchen, player/plate, tree/train, meat/meter, and train/truck.

That is, we compute the cosine distance between the embedding produced at time step t and all the images (5000) of our collection.

However, it could be that those two words compete with other words, but not specifically these two.

https://github.com/gchrupala/vgs

We use "#" to signal word boundaries and "." to signal syllable boundaries.

Word boundaries are not lost in the following cases: V#V and C#C when CC is not an allowed complex onset. C and V respectively refer to "consonant" and "vowel".

Potentially, the length of the sequences can be different for different items inside a batch as the captions have a different number of segments (be they phones, syllables or words). For this reason, and as the subsequent layers expect a 3D rectangular matrix (of shape batch size × sequence × embedding dimension) we add padding vectors on the sequence dimension until all the elements of the batch have the same sequence length.

(Seg. type ∈ {phone,syl.-connected,syl.-word,word}× GRU PACK. {1,2,3,4,5}× {true,random}× {all,keep}) + baseline

Merkx & Frank (2019) reports R@1 = 27.5 on a GRU-based model using characters as input.

[START_REF] Jespersen | Monosyllabism in English : Biennial Lecture on english philology[END_REF] estimates that at least

8,000 commonly frequent words are monosyllabic in English. 8 Note that the compression rate for syllables-word and words is very close, suggesting there is a significant overlap between syllables-word units and word units.

Note that for Japanese, we could add another intermediate step between syllables and phones which would be morae

Note that it could also be possible to use larger units, such as chunks.

When using random phone or random word boundaries, we make sure to keep the number of random boundaries the same as the number of true boundaries.

They implicitly do so as the final matrix is a dot-product between the feature map of the image and the feature map of the spoken caption. Hence, some parts of the final matrix reflect close proximity between the two modalities.

wii/window, cat/cattle, cat/cow, cat/catcher, cattle/catcher, floor/flower, fridge/frisbee, kid/kitchen, player/plate, tree/train, meat/meter, and train/truck.

Attention et méthodologie

Jain & Wallace (2019) affirment dans leur article "Attention is not Explanation" que les poids d'attention ne fournissent pas une explication significative des prédictions d'un réseau neuronal. Ils montrent notamment qu'il est possible de trouver des distributions alternatives des poids d'attention tout en conservant les prédictions intactes et arrivent à la conclusion que les poids d'attention ne constituent pas une source d'information fiable pour comprendre les prédictions d'un réseau neuronal. Wiegreffe & Pinter (2019) dans leur document de réponse "Attention is not not explication" atténuent toutefois la déclaration initiale de Jain & Wallace (2019) : pour eux les "scores d'attention sont utilisés comme fournissant une explication ; pas l 'explication". Ils affirment qu'il est légitime de considérer les poids d'attention comme une forme d'explication étant donné qu'il est peu facile de trouver une distribution alternative qui conserve la prédiction du réseau intact ou qui ne crée pas de chute de performance trop importante.

Afin de nous assurer que l'attention de notre modèle explique en partie les prédictions, et pour mesurer objectivement son importance, nous allons mélanger de façon aléatoire les poids d'attention des deux mécanismes d'attention. Si les scores obtenus avec les poids mélangés sont moins bons que ceux obtenus avec les poids originaux, nous pourrons conclure que les poids d'attention originaux sont utiles pour la prédiction des modèles.

Chapter 6

Impact of Prior Linguistic Information

The work presented in this chapter is based on the article we published at CoNLL2020 [START_REF] Havard | Catplayinginthesnow: Impact of Prior Segmentation on a Model of Visually Grounded Speech[END_REF].

Personal Bibliography

List of Figures