Marion Dohen
email: marion.dohen@gipsa-lab.inpg.fr

Speech Through the Ear, the Eye, the Mouth and the Hand

Keywords: multimodal speech, auditory-visual perception, sensori-motor, prosody, hand-mouth coordination, speech and language development, pointing

This chapter aims at describing how speech is multimodal not only in its perception but also in its production. It first focuses on multimodal perception of speech segments and speech prosody. It describes how multimodal perception is linked to speech production and explains why we consider speech perception as a sensory-motor process. It then analyses some aspects of hand-mouth coordination in spoken communication. Input from evolutionary perspectives, ontogenesis and behavioral studies on infants and adults are combined to detail how and why this coordination is crucial in speech.

Introduction

Speech is multimodal and is produced with the mouth, the vocal tract, the hands and the entire body and perceived not only with our ears but also with our eyes (and even our hands in the Tadoma method for example). The aim of this chapter is to draw a general working framework for the analysis of speech in a multimodal perspective.

It will focus on two main points. The first one will be multimodal perception of speech segments and prosody which will be related to their production in order to explain how and why speech perception is a sensory-motor process. The second part of the chapter will explore some aspects of hand-mouth coordination in adult spoken communication. It will describe why this coordination is so important for understanding cognitive communicative processes. A conclusion will then summarize all the ideas developed throughout and describe their implications for studying speech in a multisensory perspective. The framework presented here is based on an analysis and synthesis of previous works which have led to the conception of speech as a multisensory process in nature. It aims at putting forward the importance of conceiving speech as multisensory to analyze and describe the cognitive processes involved.

2 Multimodal Perception of Speech Segments and Speech Prosody in Relationship with their Production "Speech is rather a set of movements made audible than a set of sounds produced by movements."

Raymond H. Stetson [START_REF] Stetson | Motor Phonetics: A Study of Speech Movements in Action[END_REF] As R.H. Stetson put it, speech is truly multisensory and does not consist of sounds which are produced, somehow, just to be heard.

Speech is Also Visual

It is possible to perceive and understand speech with the eyes: 40 to 60% of the phonemes and 10 to 20% of the words (up to 60%) can be recovered through lipreading (data from [START_REF] Schwartz | La parole multisensorielle: Plaidoyer, problèmes, perspective[END_REF]). This ability is highly inter-speaker dependent: the deaf are often better than the hearing for example. As shown in many studies [START_REF] Sumby | Visual Contribution to Speech Intelligibility in Noise[END_REF][START_REF] Miller | An Analysis of Perceptual Confusions Among Some English Consonants[END_REF][START_REF] Neely | Effects of Visual Factors on the Intelligibility of Speech[END_REF][START_REF] Erber | Interaction of Audition and Vision in the Recognition of Oral Speech Stimuli[END_REF][START_REF] Binnie | Auditory and Visual Contributions to the Perception of Consonants[END_REF][START_REF] Erber | Auditory-Visual Perception of Speech[END_REF][START_REF] Summerfield | Use of Visual Information for Phonetic Perception[END_REF][START_REF] Macleod | Quantifying the Contribution of Vision to Speech Perception in Noise[END_REF][START_REF] Grant | Evaluating the Articulation Index for Audiovisual Input[END_REF][START_REF] Benoît | Effects of Phonetic Context on Audio-Visual Intelligibility of French[END_REF], the use of vision for speech perception is obvious when the acoustic modality is degraded by noise. Many other situations however put forward the advantage of seeing a speaker while he/she is speaking. For example, when spelling something over the phone, you often have to disambiguate: M as in 'made', N as in 'nanny'…. Reisberg and colleagues [START_REF] Reisberg | Easy to Hear but Hard to Understand: A Lipreading Advantage with Intact Auditory Stimuli[END_REF] showed that vision also helped for perceiving speech in a foreign language, speech produced by a non-native speaker or semantically complex utterances.

This could lead to think that vision simply provides redundant information: when some of the auditory information is missing, it can be recovered by vision. As shown by Summerfied [START_REF] Summerfield | Comprehensive Account of Audio-Visual Speech Perception[END_REF], perceptual confusions between consonants are different and complementary in the visual and the auditory modalities. Another clear and very famous example of the fact that the visual information is not just redundant is the McGurk effect [START_REF] Mcgurk | Hearing Lips and Seeing Voices[END_REF]. An audio [ba] dubbed onto a visual [ga] results in a [da] percept. This effect is very robust: it works in many languages and, even when people are aware of it, it can not be suppressed [START_REF] Mcgurk | Hearing Lips and Seeing Voices[END_REF]. All these observations suggest that the visual and auditory information are fused rather than the visual information being superimposed on the auditory one.

If the auditory and the visual information are not redundant, it suggests that speaking is producing gestures aiming at being both heard and seen. This is further backed by the fact that there is a marked preference for bilabials (consonants articulated with an initial contact of the superior and inferior lips), which are visually salient, at the beginning of babbling [START_REF] Vihman | From Babbling to Speech: A Re-Assessment of the Continuity Issue[END_REF]. This preference is reinforced in deaf children [START_REF] Stoel-Gammon | Prelinguistic Vocalizations of Hearing-Impaired and Normally Hearing Subjects: A Comparison of Consonantal Inventories[END_REF] and unsalient in blind children [START_REF] Mulford | First Words of the Blind Child[END_REF]. Moreover, the [m]/[n] contrast which exists in almost all the languages in the world [START_REF] Boë | Les tendances des structures phonologiques : le poids de la forme sur la substance[END_REF], is not very audible but very visible. Moreover, the fact that we use vision for speech perception does not seem to be a learnt skill. Infants 18 to 20 weeks-old, can indeed identify a speaking face corresponding to a speech signal out of two movies of speaking faces (one congruent to the audio and another incongruent) displayed at the same time [START_REF] Kuhl | The Bimodal Perception of Speech in Infancy[END_REF]. Stetson's statement could therefore be adjusted to:

Speech is (…) a set of movements made audible and visible.

Auditory-Visual Fusion: How and When?

If the auditory and visual information are fused in speech perception, one can wonder which cognitive processes underlie this fusion: how and when does it happen? After analyzing the fusion models proposed in the literature, Robert-Ribes [START_REF] Robert-Ribes | Modèles d'intégration audiovisuelle de signaux linguistiques : de la perception humaine à la reconnaissance automatique des voyelles[END_REF], Schwartz and colleagues [START_REF] Schwartz | Ten Years After Summerfield: A Taxonomy of Models for Audiovisual Fusion in Speech Perception[END_REF] presented four main potential fusion architectures summarized in Fig. 1: -Direct Identification (DI): the auditory and visual channels are directly compiled. -Separate Identification (SI): the phonetic classification is operated separately on both channels and fusion occurs after this separate identification. Fusion is therefore relatively late and decisional. -Recoding in the dominating modality (RD): the auditory modality is considered to be dominant and the visual channel is recoded under a compatible format to that of the auditory representations. This is an early fusion process. -Recoding in the motor modality (RM): the main articulatory characteristics are estimated using the auditory and visual information. These are then fed to a classification process. This corresponds to an early fusion.

Fig. 1. The four main types of auditory-visual fusion models as presented in [START_REF] Robert-Ribes | Modèles d'intégration audiovisuelle de signaux linguistiques : de la perception humaine à la reconnaissance automatique des voyelles[END_REF][START_REF] Schwartz | Ten Years After Summerfield: A Taxonomy of Models for Audiovisual Fusion in Speech Perception[END_REF].

As stated in [START_REF] Schwartz | La parole multisensorielle: Plaidoyer, problèmes, perspective[END_REF], the models which have been mostly tested in speech recognition applications are the DI and SI models because they are easier to implement. Some studies of brain activations however suggest that the interactions between modalities may occur very early (e.g. [START_REF] Calvert | Activation of the Auditory Cortex During Silent Lipreading[END_REF]). Moreover, as suggested in [START_REF] Schwartz | La parole multisensorielle: Plaidoyer, problèmes, perspective[END_REF], a number of behavioral studies provide valuable information on the fusion process. Grant and Seitz [START_REF] Grant | The Use of Visible Speech Cues for Improving Auditory Detection of Spoken Sentences[END_REF] suggested the fact that, not only does vision help us better understand speech in noise, it also improves auditory detection of spoken sentences in noise. Actually, when we see a speaker, we perceive him/her as speaking louder. In the line of this observation, Schwartz and colleagues [START_REF] Schwartz | Seeing to Hear Better: Evidence for Early Audio-Visual Interactions in Speech Identification[END_REF] designed an experiment to further test whether there are early interactions between audition and vision. They tested the intelligibility in noise of sequences which are not distinguishable by lip-reading. They found that visual only perception was not significantly better than chance which confirmed that nothing could be recovered from lip-reading. They however found that adding vision significantly improved auditory perception (AV>A). They interpreted this as potentially corresponding to reinforcement of the voicing feature by vision. The beginning of the labial gesture starts about 240ms before the vowel target is reached. In French, voicing is preceded by a prevoicing of about 100ms. Visual perception of the beginning of the labial gesture would therefore announce to the listener that he/she should lend an ear to detect the potential prevoicing. This temporal marker could reduce by a few dBs the detection threshold. The authors also conducted a second experiment in which the visual information of the speaker was replaced with visual information corresponding to a volume bar moving just as the speaker's lips. In this case, they found no improvement of perception when vision was added to audition suggesting that the effect described above is "speech specific". Put together, these behavioral data further suggest that the auditory-visual interactions are early which is incompatible with the DI and SI models.

Another question raised in [START_REF] Schwartz | La parole multisensorielle: Plaidoyer, problèmes, perspective[END_REF] is that of the control of the fusion process: is there a context bias? For example, as shown by Tiippana and colleagues [START_REF] Tiippana | Visual Attention Modulates Audiovisual Speech Perception[END_REF], visual attention can modulate audiovisual speech perception. If visual attention is distracted, the McGurk effect is indeed weaker. There also are strong inter-individual variations (see e.g. [START_REF] Schwartz | Modeling Audio-Visual Speech Perception: Back on Fusion Architectures and Fusion Control[END_REF]) as well as inter-linguistic differences (see e.g. [START_REF] Sekiyama | Inter-Language Differences in the Influence of Visual Cues in Speech Perception[END_REF]).

The data presented above suggest that the RD and RM models appear to be more likely to reflect the cognitive processes underlying auditory-visual fusion. The RM model states that the auditory and visual channels would be combined and recoded into motor representations. This suggests that there would be a link between speech perception and speech production.

Evidence for Perceptuo-Motor Links in the Human Brain

The first thing we should talk about here is mirror neurons: not because it is fashionable but because this system could be important for speech… The mirrorneuron system was found in monkeys (area F5 of the monkey premotor cortex) by Rizzolati and colleagues (for reviews, see: [START_REF] Rizzolatti | Neurophysiological Mechanisms Underlying the Understanding and Imitation of Action[END_REF][START_REF] Rizzolatti | The Mirror-Neuron System[END_REF]). These neurons are neurons which fire during both the production and the perception of goal-directed actions as well as during watching a similar action made by another individual (e.g. pick up a peanut on a tray). Kohler, Keysers and colleagues [START_REF] Kohler | Hearing Sounds, Understanding Actions: Action Representation in Mirror Neurons[END_REF][START_REF] Keysers | Audiovisual Mirror Neurons and Action Recognition[END_REF] suggested the existence of audiovisual mirror-neurons. These particular neurons fire when the monkey hears the sound produced by an action (e.g. experimenter breaking a peanut), when it sees the action being produced, when it sees and hears the action being produced and when it produces the action itself. Moreover, these neurons seem to be specialized: each neuron responds to one particular type of action (e.g. peanut breaking as opposed to ring grasping). Mirror neurons play a role in orofacial actions [START_REF] Ferrari | Mirror Neurons Responding to the Observation of Ingestive and Communicative Mouth Actions in the Monkey Ventral Premotor Cortex[END_REF]: for example, the same neuron fires when the monkey grasps food with the mouth and when the experimenter grasps food with the mouth. This is also the case for "communicative" orofacial actions such as lip-smacking/lip protrusion [START_REF] Ferrari | Mirror Neurons Responding to the Observation of Ingestive and Communicative Mouth Actions in the Monkey Ventral Premotor Cortex[END_REF][START_REF] Gil-Da-Costa | Species-Specific Calls Activate Homologs of Broca's and Wernicke's Areas in the Macaque[END_REF][START_REF] Taglialatela | Communicative Signaling Activates 'Broca's' Homolog in Chimpanzees[END_REF]. All these observations were however made for monkeys: is there any evidence for a mirror neuron system in humans? It is not possible to record the activity of a single neuron in humans. A number of neurophysiological (EEG, MEG, TMS) and neuroimaging (fMRI) studies (for reviews see: [START_REF] Rizzolatti | Neurophysiological Mechanisms Underlying the Understanding and Imitation of Action[END_REF][START_REF] Rizzolatti | The Mirror-Neuron System[END_REF][START_REF] Rizzolatti | Language and Mirror Neurons[END_REF]) have however showed activations of motor regions involved in performing specific actions in the perception of these actions performed by others. Rizzolatti and colleagues [START_REF] Rizzolatti | The Mirror-Neuron System and its Role in Imitation and Language[END_REF][START_REF] Rizzolatti | Language and Mirror Neurons[END_REF] have proposed that the mirror neuron system would play a fundamental role in speech processing by providing a neurophysiological mechanism that creates parity between the speaker and the listener and allows for direct comprehension. A number of neurobiological models of speech perception [START_REF] Hickok | Towards a Functional Neuroanatomy of Speech Perception[END_REF][START_REF] Scott | The Neuroanatomical and Functional Organization of Speech Perception[END_REF][START_REF] Callan | Multisensory Integration Sites Identified by Perception of Spatial Wavelet Filtered Visual Speech Gesture Information[END_REF][START_REF] Hickok | Dorsal and Ventral Streams: A Framework for Understanding Aspects of the Functional Anatomy of Language[END_REF][START_REF] Scott | The Functional Neuroanatomy of Prelexical Processing in Speech Perception[END_REF][START_REF] Wilson | Neural Responses to Non-Native Phonemes Varying in Producibility: Evidence for the Sensorimotor Nature of Speech Perception[END_REF][START_REF] Hickok | The Cortical Organization of Speech Processing[END_REF][START_REF] Skipper | Hearing Lips and Seeing Voices: How Cortical Areas Supporting Speech Production Mediate Audiovisual Speech Perception[END_REF] actually consider that motor areas linked to speech production (the so-called dorsal stream) intervene in speech perception either always or only under certain circumstances (speech acquisition for example).

Some studies using single-pulse transcranial magnetic stimulation (TMS) have shown a motor resonance in speech perception. This technique consists in exciting a specific motor region (in this case linked to speech production) in order for the hypothesized motor response, which is weak, to go over a threshold and produce a motor evoked potential (MEP) which can be recorded using electromyography (EMG). Using this technique, Fadiga and colleagues [START_REF] Fadiga | Speech Listening Specifically Modulates the Excitability of Tongue Muscles: A TMS Study[END_REF] showed that, during the perception of a speech sequence, there is an increase in MEPs recorded from the listeners' tongue muscles when the segmental content of speech involves tongue movements compared to when it does not. Another study using this technique [START_REF] Watkins | Seeing and Hearing Speech Excites the Motor System Involved in Speech Production[END_REF] showed that there is an increase in MEPs recorded from the listeners' lip muscles during auditory alone perception of speech compared to the perception of nonverbal audio signals and during the visual perception of speaking lips compared to that of moving eyes and brows. Some neuroimaging studies (fMRI: functional Magnetic Resonance Imaging) have shown that the dorsal stream is activated in speech perception [START_REF] Callan | Neural Processes Underlying Perceptual Enhancement by Visual Speech Gestures[END_REF][START_REF] Skipper | Lending a Helping Hand to Hearing: Another Motor Theory of Speech Perception[END_REF].

All these observations suggest that speech perception is tightly linked to speech production and that it can be considered as a sensori-motor process.

What About Suprasegmentals?

All the observations described above were made for segmental perception of speech, that is, perception of which phonemes/words are produced by a speaker. Prosody (intonation, rhythm and phrasing) is however crucial in spoken communication as illustrated by a nice example from L. Truss1 . The two sentences below correspond to exactly the same segmental content, but, pronounced differently, they have completely different meanings (in this example, punctuation is the written equivalent of prosody):

A woman, without her man, is nothing.

A woman: without her, man is nothing. For a long time, prosody was uniquely considered as acoustic/auditory. However, recent studies conducted in our lab, as well as studies conducted elsewhere (e.g. [START_REF] Kelso | A Qualitative Dynamic Analysis of Reiterant Speech Production: Phase Portraits, Kinematics, and Dynamic Modeling[END_REF][START_REF] Summers | Effects of Stress and Final-Consonant Voicing on Vowel Production: Articulatory and Acoustic Analyses[END_REF][START_REF] Vatikiotis-Bateson | Rhythm Type and Articulatory Dynamics in English, French and Japanese[END_REF][START_REF] De | The Supraglottal Articulation of Prominence in English: Linguistic Stress as Localized Hyperarticulation[END_REF][START_REF] Harrington | Coarticulation and the Accented/Unaccented Distinction: Evidence from Jaw Movement Data[END_REF][START_REF] Loevenbruck | An Investigation of Articulatory Correlates of the Accentual Phrase in French[END_REF][START_REF] Erickson | Some Articulatory and Acoustic Changes Associated with Emphasis in Spoken English[END_REF][START_REF] Loevenbruck | Effets articulatoires de l'emphase contrastive sur la Phrase Accentuelle en français[END_REF][START_REF] Erickson | Articulation of Extreme Formant Patterns for Emphasized Vowels[END_REF][START_REF] Keating | Optical Phonetics and Visual Perception of Lexical and Phrasal Stress in English[END_REF][START_REF] Cho | Prosodic Strengthening and Featural Enhancement: Evidence from Acoustic and Articulatory Realizations of /a,i/ in English[END_REF][START_REF] Beskow | Visual Correlates to Prominence in Several Expressive Modes[END_REF]) have showed that prosody also has potentially visible correlates (articulatory or other facial correlates). We analyzed contrastive informational focus as described by Di Cristo for French [START_REF] Di Cristo | Vers une modélisation de l'accentuation du français (deuxième partie)[END_REF]. It is used to put forward a constituent pragmatically and contrastingly to another as in the example below (capital letters signal focus):

Did Bob eat the apple? No, SARAH ate the apple. It can be realized using prosody (as in the example). The acoustic (intonational, durational and phrasing) correlates of prosodic focus have been widely studied [START_REF] Dahan | Interspeaker Variability in Emphatic Accent Production in French[END_REF][START_REF] Di Cristo | Intonation in French[END_REF][START_REF] Di Cristo | Prosodic Organisation and Phrasing after Focus in French[END_REF][START_REF] Rossi | La focalisation[END_REF][START_REF] Jun | A Phonological Model of French Intonation[END_REF][START_REF] Delais-Roussarie | The Prosody of Post Focus Sequences in French[END_REF][START_REF] Dohen | Pre-focal Rephrasing, Focal Enhancement and Post-focal Deaccentuation in French[END_REF]. Several production studies [START_REF] Dohen | Visual Perception of Contrastive Focus in Reiterant French Speech[END_REF][START_REF] Dohen | Audiovisual Production and Perception of Contrastive Focus in French: A Multispeaker Study[END_REF][START_REF] Dohen | Visual Correlates of Prosodic Contrastive Focus in French: Description and Inter-Speaker Variabilities[END_REF] in which we measured the articulatory and facial correlates of contrastive prosodic focus in French showed that there are visible correlates to contrastive focus in French. Two main visible articulatory strategies (inter-individual differences) were found:

-Absolute strategy: the speakers hyper-articulate and lengthen the focused constituent; -Differential strategy: the speakers both slightly hyper-articulate the focused constituent and hypo-articulate the following constituent(s). Both these strategies result in a visible contrast within the utterance between what is focused and what is not. In the line of other studies ( [START_REF] Thompson | On the Detection of Emphasis in Spoken Sentences by Means of Visual, Tactual, and Visual-Tactual Cues[END_REF][START_REF] Risberg | On the Identification of Intonation Contours by Hearing Impaired Listeners[END_REF][START_REF] Risberg | Prosody and Speechreading[END_REF][START_REF] Bernstein | Single-Channel Vibrotactile Supplements to Visual Perception of Intonation and Stress[END_REF][START_REF] Keating | Optical Phonetics and Visual Perception of Lexical and Phrasal Stress in English[END_REF]] also see [START_REF] Krahmer | Perceiving Focus[END_REF][START_REF] Granström | Audiovisual Representation of Prosody in Expressive Speech Communication[END_REF] for audiovisual perception studies using animated talking heads), we found that prosodic contrastive focus was detectable from the visual modality alone and that the cues used for perception at least partly corresponded to those identified in the production studies [START_REF] Dohen | Visual Perception of Contrastive Focus in Reiterant French Speech[END_REF][START_REF] Dohen | Audiovisual Production and Perception of Contrastive Focus in French: A Multispeaker Study[END_REF].

In another study [START_REF] Dohen | Interaction of Audition and Vision for the Perception of Prosodic Contrastive Focus[END_REF], we designed an experiment to avoid the ceiling effect: auditory only perception of prosodic focus is close to a 100% and a potential advantage of adding vision cannot be measured. The speech in noise paradigm is not adequate here since voicing is a robust feature to noise [START_REF] Miller | An Analysis of Perceptual Confusions Among Some English Consonants[END_REF]. We used whispered speech for which there is no F0. We found that auditory only perception was degraded and that adding vision clearly improved prosodic focus detection for whispered speech (see Fig. 2a). Reaction time (RT) measurements showed that adding vision reduced processing time. A further analysis of the data suggested that audition and vision are actually integrated for the perception of prosodic contrastive focus in French. Fig. 2. Percentages of correct answers for each modality (AV, A, V) for loud and whispered speech.

These results suggest that, not only is segmental perception of speech multimodal, but supra-segmental perception of speech is as well.

Some Aspects of Hand-Mouth Coordination in Speech Communication

Speech is vocal and gestural by nature. As shown in the previous section, articulatory gestures are produced aiming at being both heard and seen. Articulatory gestures are however not the only gestures we produce when we speak. We move our entire body and especially our hands [START_REF] Kendon | Gesticulation and Speech: Two Aspects of the Process of Utterance[END_REF][START_REF] Mcneill | Hand and Mind[END_REF][START_REF] Kendon | [END_REF][START_REF]Pointing: Where Language, Culture, and Cognition Meet[END_REF] and this is meaningful as well.

Gesture and Speech in an Evolutionary Perspective: Towards a Crucial Role of Hand-Mouth Coordination in Language Evolution

From an evolutionary perspective, theories defending vocal origins of language are opposed to those defending gestural origins of language. Vocal origin theories argue that speech and language would derive from animal calls such as alarm calls. In the framework of the Frame then Content theory and deriving phylogenesis (as the study of the evolution of a species) from ontogenesis (as the study of the development of a human being from birth to adulthood), MacNeilage [START_REF] Macneilage | The Frame/Content Theory of Evolution of Speech Production[END_REF][START_REF] Macneilage | On the Origin of Internal Structure and Word Forms[END_REF] argues that the motor control of the articulators used in chewing would have led to speech and language. This suggests an orofacial (related to the face and mouth) basis for language. On the other hand, theories of gestural origins of language argue that the hand was used for communication before the mouth. Corbalis [START_REF] Corballis | From Mouth to Hand: Gesture, Speech, and the Evolution of Right-Handedness[END_REF] argues that non-human primates (great apes) produce unvoluntary vocalizations whereas they control and understand manual gestures which are moreover often produced in dyads unlike calls. In his sense, this would have led to a gestural form of communication which would have led to a more complex control of the articulators and to vocal communication. What Arbib [START_REF] Arbib | The Evolving Mirror System: A Neural Basis for Language Readiness[END_REF][START_REF] Arbib | From Monkey-Like Action Recognition to Human Language: An Evolutionary Framework for Neurolinguistics[END_REF] and Holden [START_REF] Holden | The Origin of Speech[END_REF] argue is that there would have been a co-evolution of manual and speech gestural systems towards language and communication. This is in line with ontogenesis observations by Iversen and Thelen [START_REF] Iverson | The Hand Leads the Mouth in Ontogenesis Too[END_REF] who showed that the motor control of the vocal tract is longer to acquire than that of the hand but that the two systems develop in cooperation. This view underlines the fundamental link there would be between hand gestures and articulation and argues for a hand-mouth gestural basis for language. This suggests that hand-mouth coordination would have played a key role in communication evolution.

Gestures and Language Development: The Special Role of Pointing

As demonstrated, among others, by the researchers from Goldin-Meadow's team, gestures play a crucial role in language development (e.g. [START_REF] Iverson | Gesture Paves the Way for Language Development[END_REF][START_REF] Ozcaliskan | Gesture is at the Cutting Edge of Early Language Development[END_REF]). A key stage in language development is the combination of a word and a gesture carrying complementary (non redundant) information [START_REF] Goldin-Meadow | Pointing Toward Two-Word Speech in Young Children[END_REF]. An example of such a combination is a baby saying "monster" and producing a gesture miming "big" to mean "big monster". These combinations lead to the production of two-word utterances. [START_REF] Iverson | Hand, Mouth, and Brain: The Dynamic Emergence of Speech and Gesture[END_REF] underlines the fact that the motor links between the hand and the mouth develop progressively from birth until they reach synchrony (around 16 to 18 months). This association between speech and hand gestures seems to be a motor rather than a perceptive association. Blind children indeed produce gestures as well [START_REF] Iverson | Why People Gesture as They Speak[END_REF].

As pointed out by Butterworth [START_REF] Butterworth | Pointing is the Royal Road to Language for Babies[END_REF]: "pointing is the royal road toward language for babies". Pointing indeed appears to play a special role at many stages of language development (see e.g. [START_REF] Goldin-Meadow | Pointing Toward Two-Word Speech in Young Children[END_REF][START_REF] Pizzuto | Gestural-Vocal Deixis and Representational Skills in Early Language Development[END_REF][START_REF] Volterra | Gesture and the Emergence and Development of Language[END_REF]). The first gestures, produced around 9-11 months and predictive of first word onset, are deictic (gestures that show). These are not synchronized to vocal productions. Then babies combine words and pointing gestures carrying the same information as in pointing at a dog and saying "dog" at the same time. Words are then combined to pointing gestures with no information redundancy as in pointing at a dog and saying "eat" to mean "the dog is eating". This leads to the production of two-word utterances at around 18 months. These observations show that pointing plays a special role in speech development in infants.

Hand-Mouth Coordination in Adult Spoken Communication

Many studies have analyzed hand-mouth coordination in adults' speech. The two main approaches commonly used are "controlled" experiments and analysis of spontaneous productions. "Controlled" experiments make precise quantitative evaluation of the links between gestures and speech possible but involve using controlled material and recordings in an artificial environment such as a lab. Analyzing spontaneous productions enables the evaluation of the "natural" links between gestures and speech but are not replicable and do not allow for quantitative analyses. In this chapter, I will mainly focus on the description of results from "controlled" experiments. For these experiments, two main protocols have been used: dual and single task paradigms and rhythmic tasks. I will describe results from dual and single task studies. In these studies, several conditions are compared to each other: speech alone, speech and gesture and gesture alone.

Hollender [100] conducted several experiments using a dual-task paradigm in which participants were asked to name a letter which appeared on a computer screen and press a key on a keyboard. This showed that, when speech and gesture were combined, the vocal response was delayed compared to the speech alone condition potentially for the two systems to be synchronized. The author concluded that there was an adaptation of the systems due to limitations in processing capacities. Castiello and colleagues [101] also conducted a dual-task study in which the participants uttered the syllable /tah/ while grasping an object. The authors found that "the tworesponses were generated independently of each other" (see also [102]). These studies tend to suggest that hand-mouth coordination may not be so strict. However a crucial issue should be addressed: if the hand and mouth are coordinated in adult spoken communication, what could this coordination be like?

McNeill [START_REF] Mcneill | Hand and Mind[END_REF] described hand-mouth coordination as a synchronization of the hand gesture to the word which it is related to semantically and co-expressively. This raises a question as for the studies described above: there was no "semantic" relationship between the gestures and the speech. In this sense and as stated in [103,104], it seems particularly interesting to study pointing. For pointing, there should be synchrony between the gesture (which points at something) and the part of speech that shows (determiner, focus…). Levelt and colleagues [105] conducted a dual-task study on pointing and speech in which participants uttered "this lamp" while they pointed at the corresponding lamp. They observed a delay in both the manual and vocal responses in the gesture and speech condition. This delay was greater for the vocal response. These findings were replicated by Feyereisen [106]. These authors concluded that there were different processes which were competing for the same resources. This suggests that the hand and mouth systems would use the same resources and that one system would therefore have to wait for the other to free the resources to be able to use them. This would explain the delay measured for vocal responses. In a slightly different view, this delay could simply be due to coordination requirements: the vocal and gestural responses would have to be synchronized at some point and when a gesture is produced at the same time as speech, speech would wait in order for the synchrony to be achieved. Another important point is that the studies described above analyzed voice onset time. No articulatory measurements were conducted. If the coordination is a hand-mouth motor coordination, it seems unsatisfactory to analyze voice onset as an assessment of the vocal motor response. Finally, in the studies described above, the "part of speech that shows" (mostly a determiner) was always at the beginning of the utterance which probably has an effect on coordination.

Rochet-Capellan and colleagues [104] conducted a study in which they tested the effect of stress on hand-jaw coordination. They used a deictic task in which subjects pointed at a target while naming it. The name of the target was a CVCV bisyllable (2 consonant-vowel syllables) in which stress position was varied (e.g. /pápa/ vs. /papá/) as well as consonantal context (e.g. /pata/ vs. /tapa/). The target was always located in the right visual field, either near or far. Twenty native Brazilian Portuguese speakers were tested. The finger and jaw motion were captured using an IRED tracking system (Optotrak). The hypothesis was that, when stress was on the first (resp. second) syllable, the pointing apex would be synchronized with the maximum jaw displacement corresponding to the first (resp. second) syllable. This was actually the case when stress was on the first syllable. When stress was on the second syllable, the pointing apex (furthest position of the index finger corresponding to the extension of the arm in the pointing gesture) was located half way between the two jaw apices and the return of the pointing gesture was synchronized with the maximum jaw displacement corresponding to the second syllable. There was no effect of target position or consonant on the coordinative pattern. A finger adaptation was however observed: a longer plateau (delay between pointing apex and beginning of the return stroke) was measured when stress was on the second syllable. There was also a high subject variability and the authors suggested an adaptation of the jaw from the first to the second stress condition: the jaw response onset was 70ms earlier but the maximum jaw displacements were 100ms earlier. The main conclusion the authors made was that the jaw was synchronized to the finger pointing plateau (the part of the pointing gesture that shows). This study shows that there seems to be a tight coordination of the hand and jaw which serves communication purposes. This study also confirms that the coordination is rather between the hand and the articulatory gestures than between the hand and the sound. However it only analyzed hand-jaw coordination and one can wonder about coordination with other articulators.

The Co-Development of Speech and Sign

Ducey-Kaufman and colleagues [107,108] designed a theoretical framework to analyze the co-development of speech and sign in speech acquisition. This theory mainly puts forward a "developmental rendez-vous" between what the authors call the sign frame (cyclicity of hand gesture in pointing) and the speech frame (jaw oscillations in speech). This meeting point occurs at about 12 months before when the speech and sign frame develop in parallel. This development leads to the possibility of producing two syllables within one pointing gesture and puts forward the potentially crucial role of the foot (phonological unit larger than or equal in size to the syllable and corresponding to the smallest rhythmic group) in speech. They tested this with a longitudinal study on six babies from 6 to 18 months. They found a 400ms mean duration of the syllable and a 800ms mean duration of the pointing gestures. Rochet-Capellan and colleagues tested this on adults [109]. The task they used was to name and show a target at a GO signal twice in a row. The name of the target was either a 1, or a 2, or a 3, or a 4 syllable word (/pá/ vs. /papá/ vs. /papapá/ vs. /papapapá/). They recorded finger motion using an IRED tracking system (Optotrak). The predictions were that the delay between the two pointing apices would remain constant for target names consisting of 1 or 2 syllables. This delay would then increase from 2 to 3 syllables and remain constant from 3 to 4 syllables. This is what they observed. They also analyzed the durations of the jaw gestures (from gesture initiation to the last maximum displacement, corresponding to the last syllable). They found a ratio between the duration of the pointing gesture and that of the jaw gestures of 0.5 whatever the number of syllables. These preliminary observations could confirm the two-syllable for one pointing gesture hypothesis.

Conclusion

The aim of this chapter was to draw a general multimodal framework for studying speech. The first aspect presented was the fact that speech is not only auditory. We also perceive speech through the eyes and the role of vision is not only that of a backup channel. Auditory and visual information are actually fused for perceptual decision. A number of potential cognitive fusion architecture models were presented. Some data suggest that audiovisual fusion takes place at very early stages of speech perception. Studies of the cognitive and brain mechanisms show that motor representations are most probably used in speech perception. Speech perception is a sensori-motor process: speech consists of gestures produced to be heard and seen. The multisensory aspect of speech is not limited to segmental perception of speech. It appears that there are visible articulatory correlates to the production of prosodic information such as prosodic focus and that these are perceived visually. Moreover, when the acoustic prosodic information is degraded, it is possible to put forward an auditory-visual fusion which enhances speech perception.

The second main aspect of this paper is the fact that we do not communicate only with our mouths. Our entire body, and especially our hands, is involved in this process. An analysis of the evolutionary theories of language development suggests that hand-mouth coordination may have played a crucial role in language evolution. This is at least the case for language development in babies in which the pointing gesture plays a crucial role. In adults, several studies seem to point out that hand and mouth (articulatory gestures) are tightly coordinated for communication purposes (for pointing at least). This still has to be analyzed for "real" speech (not only bisyllables) and for other communicative gestures: is the coordination gesture specific? Is it communication specific? Is it linked to grammar and prosody? Another important point is the fact that it appears that, "naturally" one pointing gesture would embed two syllables (in infants and maybe adults).

Truss, L.: Eats, Shoots & Leaves -The Zero Tolerance Approach to Punctuation. Profile Books Ltd, London (2003)

Acknowledgments. First of all, I would like to thank Anna Esposito for giving me the opportunity to take part in the International School "E. R. Caianiello" XIII course on Multimodal Signals: Cognitive and Algorithmic Issues. I also thank the two anonymous reviewers who provided insightful comments on a previous version of this chapter. For all the input he provided as well as for his advice and support, I thank Jean-Luc Schwartz. I also thank Amélie Rochet-Capellan for the inspiration I found in her work as well as for some material she provided. I also thank Marc Sato, Hélène Loevenbruck and Gérard Bailly.