Oliver Roesler

Amir Aly

Tadahiro Taniguchi

Yoshikatsu Hayashi

A Probabilistic Framework for Comparing Syntactic and Semantic Grounding of Synonyms through Cross-Situational Learning

Natural human-robot interaction requires robots to link words to objects and actions through grounding. Although grounding has been investigated in previous studies, none of them considered grounding of synonyms. In this paper, we try to fill this gap by introducing a Bayesian learning model for grounding synonymous object and action names using crosssituational learning. Three different word representations are employed with the probabilistic model and evaluated according to their grounding performance. Words are grounded through geometric characteristics of objects and kinematic features of the robot joints during action execution. An interaction experiment between a human tutor and HSR robot is used to evaluate the proposed model. The results show that representing words by syntactic and/or semantic information achieves worse grounding results than representing them by unique numbers.

I. Introduction

In 2016, 67,000 service robots have been sold worldwide resulting in a slowly growing number of human-robot interactions in peoples everyday life. To enable robots to efficiently collaborate with human users in complex environments, they must be able to converse in natural language and understand the instructions of a user so that they execute the desired actions appropriately, such as pick up a drink or grab a box [START_REF] Kemp | Challenges for robot manipulation in human environments[END_REF][START_REF]World robotics 2017 -service robots[END_REF]. To achieve this goal, robots have to do "Symbol Grounding", which was first described in Harnad [START_REF] Harnad | The symbol grounding problem[END_REF], to relate words and sensory data that refer to the same object or action to each other. Although, previous studies investigated the use of cross-situational learning for grounding of objects [START_REF] Fontanari | Cross-situational learning of objectword mapping using neural modeling fields[END_REF][START_REF] Taniguchi | Crosssituational learning with bayesian generative models for multimodal category and word learning in robots[END_REF] as well as spatial concepts [START_REF] Aly | A Generative Framework for Multimodal Learning of Spatial Concepts and Object Categories: An Unsupervised Part-of-Speech Tagging and 3D Visual Perception Based Approach[END_REF][START_REF] Dawson | A generative probabilistic framework for learning spatial language[END_REF][START_REF] Tellex | Approaching the symbol grounding problem with probabilistic graphical models[END_REF], they ensured that one word appears several times together with the same perceptual feature vector to allow the creation of a corresponding mapping [START_REF] Fontanari | A cross-situational algorithm for learning a lexicon using neural modeling fields[END_REF]. However, natural language is ambiguous due to homonymy, i.e. one word refers to several objects or actions, and synonymy, i.e. one object or action can be referred to by several different words. The latter does not need to be actual synonyms, especially, considering that according to the "Principle of Contrast" no two words refer to the exact same meaning, i.e. there are no true synonyms [START_REF] Clark | The principle of contrast: A constraint on language acquisition[END_REF]. Consequently, words are only synonyms as references to an object or action in a particular set of situations. Examples are words that refer to the purpose or content of an object, instead of the object itself, such as: tea or coffee instead of cup.

In this paper, we take a step towards grounding synonyms through a developmentally plausible approach so as to infer the meaning of objects and actions. More specifically, we present an unsupervised learning model for sensory-motor coupling using a probabilistic learning model and a robot. In the model, words are represented in three different ways: as indices, Part-of-Speech tags, or syntactic-semantic vectors, thereby, allowing the investigation of the influence of syntactic and semantic information on grounding of synonyms. The rest of this paper is structured as follows: Section (II) provides an overview of the framework. The experimental design and the obtained results are described in Sections (III and IV). Finally, Section (V) concludes the paper.

II. System Overview

The used grounding system consists of five parts: (1) Neural Network Language Model (Word2Vec), which creates a vector space in which the distance between two vectors represents their syntactic and semantic similarity, (2) Partof-Speech (POS) tagging system, which grammatically tags words in an unsupervised manner (i.e., it assigns numerical tags to words without using any pre-tagged corpus or tagging dictionary), (3) 3D object segmentation system, which determines the geometric characteristics of objects by segmenting them into point clouds, (4) Action recording system, which creates action feature vectors by recording the state of several joints while the robot is executing actions, and (5) Multimodal probabilistic learning model, which grounds object and action names through visual perception and proprioception. The inputs and outputs of the individual parts are highlighted in Figure [START_REF] Aly | Towards understanding objectdirected actions: A generative model for grounding syntactic categories of speech through visual perception[END_REF], and described in detail in the following subsections.

A. Syntactic-Semantic Representation of Words

Neural Network Language Models (NNLM) represent words as high-dimensional real-valued vectors. Several different NNLM architectures are described in the literature [START_REF] Bengio | A neural probabilistic language model[END_REF][START_REF] Mikolov | Recurrent neural network based language model[END_REF][START_REF] Schwenk | Continuous space language models[END_REF]. One of the main advantages of these models is the level of generalization, which is not possible to attain with simple n-gram models [START_REF] Mikolov | Linguistic regularities in continuous space word representations[END_REF]. Word2Vec, a recently developed NNLM, uses a 2-layer neural network to create word embeddings, i.e. a vector space, for a given text corpus. Syntactically and semantically similar words are located close together [START_REF] Mikolov | Efficient estimation of word representations in vector space[END_REF][START_REF] Mikolov | Distributed representations of words and phrases and their compositionality[END_REF]. A corpus of 100MB of Wikipedia articles was used to train Word2Vec 1 . Several names used Fig. 1: Overview of the language grounding system. in this study are bigrams, i.e. they consist of two words, which would lead to two separate word vectors. Therefore, an underscore has been inserted between the two words to convert the original bigrams into unigrams as shown in Tables (I and II). Multi-Dimensional Scaling (MDS) has been used to transform the vector space generated by Word2Vec to an Euclidean space [START_REF] Cox | Multidimensional Scaling[END_REF]. Afterwards, the high vector dimensionality (100 dimensions) has been reduced through PCA to 7 dimensions so as to efficiently ground vectors in perception.

B. Unsupervised Part-of-Speech Tagging

Through Part-of-Speech (POS) tagging words in sentences are marked with grammatical attributes (e.g., noun, verb, adjective, etc.). The literature reveals a variety of supervised, semi-supervised, and unsupervised POS tagging approaches [START_REF] Brill | A simple rule-based part of speech tagger[END_REF][START_REF] Brill | Unsupervised learning of disambiguation rules for part-of-speech tagging[END_REF][START_REF] Toutanova | A bayesian LDA-based model for semi-supervised part-of-speech tagging[END_REF]. In this study, an unsupervised POS tagging approach is used that induces grammatical tags for word sequences through a first-order Bayesian Hidden Markov Model (HMM), i.e., without using any pre-tagged training corpus2 . The POS tagging model assigns a grammatical tag T = (t 1 , . . . , t n ) to each word in the sequence w = (ω 1 , . . . , ω n ). The first-order Bayesian HMM uses words as observations and tags as hidden states (Figure 2) [START_REF] Gao | A comparison of bayesian estimators for unsupervised hidden markov model pos taggers[END_REF].

The probability distribution of tag states for the word sequence w is defined as follows:

(t 1 , . . . , t n ) = n i =1 (t i | t i-1 ) (1)
TABLE II: Overview of the used actions.

Name 1 Name 2 Description lift up raise The object will be lifted up. grab take The object will be grabbed, but not displaced. push poke The object will be pushed with the closed gripper i.e. it will not be grabbed. pull drag The object will be grabbed and moved towards the robot. move shift The object will be grabbed and moved. where the transition probability to the tag t i is conditioned on the tag t i-1 . This could encode the intuitive grammar that parts of speech might follow, like having a noun after a determiner. Emission distributions of numerical tags over words are defined through the probability (ω i | t i ) of the word ω i being conditioned on the tag t i . For each tag state the generative transition and emission parameters of the proposed HMM model (φ, θ) are characterized through multinomial distributions with Dirichlet priors (α φ , α θ ) (where K denotes the number of tag states):

t i t i-1 = t ∼ Mult (φ t ) , φ t α φ ∼ Dir (α φ ) ω i t i = t ∼ Mult (θ t ) , θ t α θ ∼ Dir (α θ ) (2)
For an unannotated training corpus containing a set of m sentences W = {w 1 , . . . , w m }, the POS tagging model tries to induce the most likely numerical tag set T = {T 1 , . . . , T m } for each sentence in the corpus that maximizes the following expression:

(T, W) = (T ,w) ∈ (T ,W) (T , w | φ, θ) = (T ,w) ∈ (T ,W) n i=1 (t i | t i-1 , φ t ) (ω i | t i , θ t ) (3)
Inferring the latent tag variables uses the Gibbs sampling algorithm [START_REF] Geman | Stochastic relaxation, gibbs distributions, and the bayesian restoration of images[END_REF][START_REF] Neubig | Simple, correct parallelization for blocked gibbs sampling[END_REF], which produces a set of samples from the posterior distribution (T |W), i.e., it loops over the possible tag assignments to words that could maximize Equation (3) expressed as follows, where -i denotes all samples except the i-th sample:

(T i , T (i) T -i , W , T (-i) , w, α φ , α θ ) (4)

C. 3D Object Features

In order to obtain object feature vectors a model based 3D point cloud segmentation approach is used due to its speed, reliability, and the fact that not much prior knowledge about the environment is required, such as object models and the number of regions to process [START_REF] Craye | Environment exploration for object-based visual saliency learning[END_REF][START_REF] Nguyen | 3D point cloud segmentation: A survey[END_REF]. The applied model detects the major plane in the environment 3 via the RANSAC algorithm [START_REF] Fischler | Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography[END_REF], and keeps track of it in consecutive frames. Planes that are orthogonal to the major plane and touch at least one border of the image are defined as wall planes, while points that are neither part of the major nor the wall planes are voxelized and clustered into blobs. Blobs of reasonable size, i.e. neither extremely small nor large, are treated as objects. Each point cloud of a segmented object is characterized through a Viewpoint Feature Histogram (VFH) [START_REF] Rusu | Fast 3D recognition and pose using the viewpoint feature histogram[END_REF] descriptor, which represents the geometry of the object taking into account the viewpoint and ignoring scale variance. Figure [START_REF] Bengio | A neural probabilistic language model[END_REF] shows an example of the obtained 3D point cloud information.

D. Action Features

Action feature vectors were formulated to represent the dynamic characteristics of actions during execution through teleoperation, which could afford variations in the obtained action feature vectors. Overall, five different characteristicseach representing a possible subaction -are recorded using the sensors of the robot [START_REF] Hsr | [END_REF]. The employed characteristics are:

1) The distance from the actual to the lowest torso position in meters.

2) The angle of the arm in radians.

3) The angle of the wrist in radians. 4) Binary state of the gripper.

(1: closing, 0: opening or no change) 5) Velocity of the base. They are then combined into the following vector

            a 1 1 a 2 1 a 3 1 a 4 1 a 5 1 . . . . . . . . . . . . . . . a 1 6 a 2 6 a 3 6 a 4 6 a 5 6            
where a 1 represents the difference of the distances from the lowest torso position in meters, while a 3 and a 4 represent the difference in the angles of the arm and wrist in radians, respectively. The differences are calculated by subtracting 3 The major plane in the conducted experiment is a tabletop. the value at the beginning of the subaction from the value at the end of the subaction. a 2 represents the mean velocity of the base (forward/backward), and a 5 represents the binary gripper state. Each action is characterized through six subactions. Consequently, if an action consists of less than six subactions, rows with zeros will be added at the end.

E. Probabilistic Learning Model

Figure ( 4) outlines the employed Bayesian learning model, which grounds object and action names through perception. A Bayesian network is a directed acyclic graph representing a set of probability distributions that can handle uncertainty represented by noisy perceptual data obtained from the environment [START_REF] Jordan | Graphical models[END_REF]. Three different versions of the Bayesian learning model are employed that represent words differently: (1) Word indices, i.e. each word is represented by a unique number, e.g. (coke, 1) and (lemonade, 2), (2) POS tags, i.e. each word is represented by the grammatical category it belongs to, (3) syntactic-semantic vectors, i.e. each word is represented by a vector in a syntactic-semantic vector space. When words are represented by indices or POS tags a categorical and Dirichlet distribution are used for w i and θ m,Z , respectively. If words are represented by syntacticsemantic vectors a Gaussian and Gaussian Inverse-Wishart distribution are used instead.

In the probabilistic learning model, words are represented by the observed state w i , which can be syntactic-semantic vectors, POS tags or indices (Sections II-A and II-B). The observed state g represents the geometric characteristics of objects expressed through the VFH descriptor (Section II-C). Actions are represented by the observed state a (Section II-D). Table (III) provides a summary of the definitions of the learning model parameters. The corresponding probability distributions, which characterize the different modalities in the graphical model, are defined in Equation [START_REF] Brill | Unsupervised learning of disambiguation rules for part-of-speech tagging[END_REF], where N denotes a multivariate Gaussian distribution, GIW denotes a Gaussian Inverse-Wishart distribution, Dir denotes a Dirichlet distribution, and Cat denotes a categorical distribution. The latent variables of the Bayesian learning model are inferred using the Gibbs sampling algorithm [START_REF] Geman | Stochastic relaxation, gibbs distributions, and the bayesian restoration of images[END_REF].

                                                                   w i ∼ Cat(θ m i ,Z m i ) N(θ m i ,Z m i ) θ m,Z L 1 ∼ Dir(γ) , L 1 = (1, . . . , L) GIW(γ) φ a K 1 ∼ GIW(β a ) , K 1 = (1, . . . , K a ) φ g K 2 ∼ GIW(β g ) , K 2 = (1, . . . , K g ) π w ∼ Dir(λ) π g ∼ Dir(α g ) π a ∼ Dir(α a ) m i ∼ Cat(π v ) Z g ∼ Cat(π g ) Z a ∼ Cat(π a ) g ∼ N(φ Z g ) a ∼ N(φ Z a ) (5)

III. Experimental Setup

A human tutor and HSR robot 4 are interacting in front of a tabletop. The robot does not have any preexisting knowledge about the world, and its syntactic-semantic knowledge is limited to the word vector space and the HMM-based POS tagging model, which were created prior to training. One of the five different objects {bottle, cup, box, car, and book} is placed on the table (Figures 3 and5). Each of the objects can be referred to by five different names as shown in Table (I). During the cross-situational learning phase [START_REF] Fontanari | Cross-situational learning of objectword mapping using neural modeling fields[END_REF], the robot performs five different actions on each object (Figure 5), where each action can be described by two different names as illustrated in Table (II). A total of 75 different sentences are given to the robot 4 The Human Support Robot from Toyota is used for the experiment. It has a cylindrical shaped body, which can move omnidirectional, and is equipped with one arm and a gripper to grasp objects. The robot has 11 degrees of freedom and is equipped with stereo and wide-angle cameras, a microphone, a display screen, and a variety of different sensors. [Official Toyota HSR Website] by the human tutor in order to allow it to ground object and action names using the recorded perceptual data. Each sentence consists of either two or three words and has one of the following two structures: "action the object" or "action object", respectively 5 , where action and object are substituted by the corresponding names (Tables I andII). The experimental procedure consists of three phases as described below:

1) Collection of syntactic, semantic, and perceptual information for the different situations. a) An object is placed on the table and the robot determines its geometric characteristics so as to calculate its feature vector. b) A sentence is given by the human tutor to the robot, and the corresponding index, POS tag or vector of each word is obtained (Sections II-A and II-B). c) The human tutor teleoperates the robot to execute the given action while several kinematic characteristics are recorded and converted into an action feature vector (Section II-D).

2) The probabilistic model is used to ground words using the geometric characteristics of objects and the action feature vectors (Section II-E). 3) For the test phase, a total of 50 sentences are used to evaluate the learning framework. In the investigated scenario, the sentences are assigned randomly to the training and test sets. Consequently, most words are used during the training and test phases, thereby, allowing the investigation whether syntactic and semantic information provides any benefit when most synonyms have already been encountered before in the cross-situational learning phase.

IV. Results and Discussion

In several previous studies, probabilistic models have been used for language grounding [START_REF] Aly | Towards understanding objectdirected actions: A generative model for grounding syntactic categories of speech through visual perception[END_REF][START_REF] Dawson | A generative probabilistic framework for learning spatial language[END_REF][START_REF] Tellex | Approaching the symbol grounding problem with probabilistic graphical models[END_REF]. However, to the best of our knowledge, none of them included synonyms and they differed in their approaches, experimental setups, or corpora from the current study, which makes the comparison of results between our study and these studies, among many others in the literature, difficult to attain. 20 fold crossvalidation has been used, i.e. 20 different training and test Three different word representations have been investigated (Section II-E). The obtained accuracies show that the Word Indices representation achieved the best grounding for all modalities (Table IV). For the POS Tags representation the model did not learn the Object modality, which might be caused by the different object positions in the two sentence structures, i.e. "action the object" or "action object". We argue that this performance is due to the relatively short sentences with less syntactic word categories, which did not allow the first-order Bayesian HMM to learn the Object modality with respect to the previous parts of speech. For example, in the first sentence structure, the object was preceded by an action and a determiner, while in the second sentence structure, it was directly preceded by an action. Unlike our previous studies [START_REF] Aly | Towards understanding objectdirected actions: A generative model for grounding syntactic categories of speech through visual perception[END_REF][START_REF] Aly | A Generative Framework for Multimodal Learning of Spatial Concepts and Object Categories: An Unsupervised Part-of-Speech Tagging and 3D Visual Perception Based Approach[END_REF], where the POS tagging model achieved better results using more informative sentences containing more syntactic word categories. When the Word Vectors representation was used the model did not learn the Others modality 6 because it only contains one word (the article the), which is not sufficient to create an independent cluster for the learning model. Figure [START_REF] Clark | The principle of contrast: A constraint on language acquisition[END_REF] shows the probability distributions of all word categories over the different modalities. For the Word Indices representation, all object categories, except the car names, have been correctly assigned to the Object modality, while the action names and the article the had the highest probability for the Action modality. In comparison, for the POS Tags representation the Action modality achieved the highest probability for all categories, while for the Word Vectors representation the Object modality achieved the highest probability for all categories, except the car names, which had the highest probability for the Action modality. The performance of the POS Tags was, generally, worse than the performance of the Word Indices due to the short sentences of the employed corpus that could not give the model enough syntactic information to learn, as explained earlier. Similarly, Word Vectors achieved less results with respect to Word Indices due to the clustering performance for the different categories based on the encoded syntacticsemantic information in vectors. Overall, this study shows that grounding of synonyms in perception, so as to allow the robot to collaborate efficiently with human users, does not necessarily benefit from the use of syntactic and/or semantic information based on the actual experimental setup of this study. However, in our future work, we will try to enhance the syntactic and semantic representation so as to achieve better results that could allow the robot to better infer the syntactic and semantic structures of a sentence.

V. Conclusions and Future Work

We investigated a multimodal framework for grounding synonymous object and action names through the robot visual perception and proprioception during its interaction with a human tutor. Our Bayesian learning model was set up to learn the meaning of object and action names using geometric characteristics of objects obtained from point cloud information and kinematic features of the robot joints recorded during action execution. Our proposed model allowed the grounding of synonyms, while also showing that representing words by simple indices, instead of syntactic tags or syntactic-semantic vectors, achieves the best grounding. In future work, we will obtain grounding results for more complex sentences containing more syntactic word categories. Furthermore, we will investigate the effect of different syntactic and semantic information as well as the combination of both on grounding.

Fig. 2 :

 2 Fig. 2: Graphical representation of the HMM-based Part-of-Speech tagging model.

Fig. 3 :

 3 Fig. 3: Examples of the used objects and the corresponding 3D point cloud information: (A) car, (B) bottle, and (C) cup.

Fig. 4 :

 4 Fig. 4: Graphical representation of the probabilistic model. Indices i, g and a denote the order of words, object geometric features and action features, respectively.

Fig. 5 :

 5 Fig. 5: Illustration of action lift up executed by the robot in a tabletop scene.

Fig. 6 :

 6 Fig. 6: Probability distributions of word categories over the different modalities for all word representations, where dark blue represents high probability.

TABLE I :

 I Overview of the objects with their corresponding names.

	Object			Names
	Bottle	coca cola	soda	pepsi	coke	lemonade
	Cup	latte	milk	milk tea	coffee	espresso
	Box	candy	chocolate	confection	sweets	dark chocolate
	Car	audi	toyota	mercedes	bmw	honda
	Book	harry potter	the godfather	narnia	lord of the rings	the hobbit

TABLE III :

 III Definitions of learning parameters in the graphical model.

	Parameter	Definition
	λ	Hyperparameter of the distribution πw
	αg	Hyperparameter of the distribution πg
	αa	Hyperparameter of the distribution πa
	mi	Modality index of each word. (modality index ∈ {Action, Object, Others})
	Za	Index of action feature vector distributions
	Zg	Index of object geometry distributions
	vi	Word vectors
	g	Observed state representing geometric characteristics of object using VFH descriptor
	a	Observed state representing characteristics of action
	γ	Hyperparameter of the distribution θm,Z
	βa	Hyperparameter of the distribution φa
	βg	Hyperparameter of the distribution φg

TABLE IV :

 IV Accuracies for all modalities.

		Object	Action	Others
	Word Indices	76.71%	55.14%	41.76%
	POS Tags (Syntactic)	4.58%	49.56%	33.84%
	Word Vectors (Syntactic-Semantic)	66.19%	33.13%	0.00%
	sets have been created. 75 sentences have been used for
	training, while the remaining 50 sentences have been used
	for the test phase.

O. Roesler and Y. Hayashi are with Biomedical Engineering, School of Biological Sciences, University of Reading, UK (email: oliver@roesler.co.uk, y.hayashi@reading.ac.uk).

A. Aly and T. Taniguchi are with the Emergent Systems Laboratory, Ritsumeikan University, Japan (email: amir.aly@em.ci.ritsumei.ac.jp, taniguchi@em.ci.ritsumei.ac.jp).

The corpus can be downloaded at http://mattmahoney.net/dc/text8.zip.

For example, the POS tagging system could assign these numerical tags to words of the sentence: (Push,7) (the,5) (Coffee,9).

The latter is only used for sentences with the book object. For example: "lift up harry potter" represents the structure "action object", while "lift up the lemonade" represents the structure "action the object".

In fact, it does not assign the Others modality to any word as shown in Figure[START_REF] Clark | The principle of contrast: A constraint on language acquisition[END_REF].


Amélie Rochet-Capellan
email: amelie.rochet-capellan@icp.inpg.fr

Jean-Luc Schwartz
email: jean-luc.schwartz@icp.inpg.fr

Rafael Laboissière
email: laboissiere@cbs.mpg.de

Arturo Galvàn
email: galvan@cbs.mpg.de

Pointing to a target while naming it with /pata/ or /tapa/: the effect of consonants and stress position on jaw-finger coordination

Keywords: speech-hand coordination, jaw-finger coordination, deixis, pointing, Optotrak

This study investigates jaw-finger coordination in a task consisting in pointing to a target while naming it with a /pata/ or a /tapa/ utterance stressed either on the first ('CVCV) or on the second (CV'CV) syllable. Optotrack measurements of jaw and finger displacements show that for 'CVCV names, the moment at with the finger reaches the target alignment is synchronized with the maximum of the first jaw opening motion. For CV'CV names, the synchronization occurs between the moment at which the finger leaves the target-alignment position and the maximum of the jaw opening motion for the second vowel. This pattern of synchronization does not depend on the target position or on the consonants order. These results add some support to theories involving the coordination of orofacial and brachiomanual gestures in the development and phylogeny of human languages. They call for more investigations on the link between speech and brachiomanual gestures in face-to-face communication.

Introduction 1.Substance-based deixis

The deictic function of language (refer to an object) has been assumed to be first supported by the hand and later connected to speech [1][2]. This connection could constitute a substrate for the emergence of language indexical signs (e.g. "this") in the course of ontogeny and phylogeny. Hence, a close link might exist between speech and pointing deictic sites, enabling to show respectively by the hand and by the voice. Synchronization of speech and hand pointing in face-to-face communication [START_REF] Mcneill | Action, thought and language[END_REF] [START_REF] Levelt | Pointing and voicing in deictic expressions[END_REF] could consist in coherence between the speech focus and the pointing focus, with a trend to align the arm-hand-finger system with the speech focus. Moreover, developmental studies displayed a link between the frequencies of arm-hand gestures and jaw oscillations in babbling [START_REF] Petitto | Language rhythms in baby hand movements[END_REF]. This link would evolve toward a "rendez-vous" between the motor control of jaw oscillations in babbling, the speech frame [START_REF] Macneilage | On the origins of internal structure of word forms[END_REF] [START_REF] Macneilage | The frame/content theory of evolution of speech production[END_REF] and brachiomanual oscillations in pointing, the sign frame [START_REF] Ducey-Kaufmann | When the Speech Frame meets the Sign Frame in a developmental framework[END_REF]. This "rendez-vous" would play a key role in language acquisition [START_REF] Pizzuto | Gestural-vocal deixis and representational skills in early language development[END_REF] [START_REF] Volterra | Gesture and the emergence and development of language[END_REF]. In addition, it leads to suspect that synchronization between speech and pointing gestures could rest on a close coordination of the jaw and arm-hand oscillatory systems, anchored in motor control acquisition.

Focus and jaw-finger coordination

In this framework, we previously studied jaw-finger coordination in a task consisting in pointing to a target while naming it with a CVCV word (/papa/ vs. /tata/) stressed either on the first ('CVCV) or on the second (CV'CV) syllable [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF]. The hypothesis was that the instant of finger-target alignment (pointing apex, P A ) would be synchronized with the instant of maximum jaw opening (apex) for the stressed vowel; that is, with the first jaw apex (J A1 ) for 'CVCV names and the second one (J A2 ) for CV'CV ones. The results showed that P A was actually close to J A1 for 'CVCV names. For CV'CV ones, P A occurred after J A1 , but before J A2 , roughly at an equal delay between J A1 and J A2 , contrary to the prediction. However, in that case, J A2 was synchronized with the end of the period of finger-target alignment, when finger began its return stroke. Hence, there seems to be clear jaw-finger coordination, though different from our original hypothesis. The effect of stress position on finger-jaw synchronization mainly results from a speech adaptation: jaw movement begins earlier for CV'CV than for 'CVCV names. However, it also involves the duration during which the finger points on the target: the finger stays longer in its apex position for CV'CV than 'CVCV names, "expecting", in some sense, jaw arrival on the stressed vowel. Further analyses showed that stress modifies amplitude and duration of the two jaw opening gestures, but has no effect on initiation and apex times of the pointing gesture. Conversely, increasing target distance delays both pointing apex and jaw apices, but has no effect on amplitude and duration of the two jaw opening strokes. Altogether, this previous experiment showed that the speech focus tends to be achieved during the finger-target alignment and that jaw events tend to be anchored in pointing events. Moreover, the pointing gesture aims at reaching the target alignment; it is mainly "target-driven". On the contrary, jaw motions are adapted both to the phonetic goal and to the pointing task.

Jaw profiles and jaw-finger coordination

The previous study used CVCVs words with a single consonantal articulation place. However, in another study focused on jawtongue-lips coordination in speech sequences, we found different jaw motion profiles in duplicated (/tata/-/papa/) vs. variegated (/pata/-/tapa/) utterances produced with rate increase [START_REF] Rochet-Capellan | The Labial-Coronal effect and CVCV stability during reiterant speech production: An articulatory analysis[END_REF] [START_REF] Rochet-Capellan | An articulatory basis for the Labial-to-Coronal effect: /pata/ seems a more stable articulatory pattern than /tapa[END_REF]. The main tendency is to realize variegated CVCV sequences with large asymmetries between the two jaw cycles, evolving towards a single jaw cycle with rate increase. On the contrary, duplicated CVCV always required two jaw cycles. The present study aims at testing if the pattern of jaw-finger coordination observed with /papa/ and /tata/ with different stress positions resists to the modifications in jaw dynamics imposed by variegation in /pata/ and /tapa/ sequences. This would provide an argument for the independence of internal speech planning relative to the adaptation of speech gestures to pointing gestures.

Methods

Participants. Twenty native Brazilian Portuguese speakers (4 men and 16 women), right-handed with normal or correctedto-normal vision, participated in the experiment. The Brazilian Portuguese language was chosen because it is possible to find pairs of words in this language that differ only by the position of the stress. The participants and the procedure were the same as those of our previous study [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF].

Figure 1. Experimental setup

Procedure. The participant was seated at a table. She/he was informed that a word and a red smiley  sign (the target) would appear on the board in front of her, projected by a beamer. The word was introduced as the name of the person represented by the smiley-target. It could be either /pata/ or /tapa/ with a stress on the first (e.g. /pàta/) or on the second (e.g. /patà/) vowel. The vowel /a/ was selected since its realization requires a large jaw opening gesture. The target appeared in the right visual field, either at the near or at the far position (see Figure 1). The task was to point to the smiley and name it with the word as soon as its color changed from red to green. The presentation of the red target lasted for 3.5 s plus a Gaussian variation with zero mean and 0.15 s standard deviation. Then, the target became green (the GO signal) and lasted 1 s on the board. A black square on the midline of the table indicated the onset position for the finger-pointing gesture. The experiment was divided into four blocks separated by a 30 s pause. Data recording and processing. Finger and jaw movements were recorded using an Optotrak. Two markers were pasted at the tip of index finger, such that at least one of them was always visible by the cameras during the course of the pointing movement. The jaw position was tracked by a third marker attached to the chin. Three other markers were pasted on the table to provide a referential for the moving markers. Head motion was measured by three markers attached to a plastic triangle fixed around the subject's head. Jaw positions were computed in relation to the head moving reference frame. The markers positions were sampled against time at 100 Hz. In order to reduce the amount of data to process, we conducted a Principal Component Analysis (PCA) on the Optotrak data. The first principal component appears as a good representation of markers motions. In average, for the 20 participants, it explained 98% of the variance for the finger markers and 95% for the jaw marker. These signals were lowpass filtered at 15 Hz with a Butterworth filter. The temporal measurements are indicated in Figure 2. Motion onset (initiation) and apex times correspond to 10% of the peak velocity at the beginning and the end of the movement, respectively. P I and P A are the initiation and apex times for the forward movement of the finger, while P R is the onset of the return movement. For the jaw, J I1 , J A1 and J I2 , J A2 are the initiation and the apex times of the first and second opening gestures, respectively.

Results

Absolute positions of jaw and finger events are displayed in Figure 3. As in [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF], we also computed positions of P A and P R relative to the two jaw apices, defined as: P A/J = (P A -J A1 )/(J A2 -J A1 ) and P R/J = (P R -J A1 )/(J A2 -J A1 ) A value of 0 (respectively 1) indicates that the finger event is synchronized with J A1 (respectively J A2 ). Finger positions relative to jaw apices are displayed in Figure 4. Amplitude and duration of jaw and finger strokes, were also computed as the difference of position and duration between the initiation and the apex of the stroke, respectively. They are displayed in Figure 5. The effects of the three experimental factors on each measure were tested using three-way within-subject ANOVAs (effects are considered significant for p < .05 with F(1, 19)). Figure 3. Mean durations from the GO-signal to jaw and finger events according to experimental conditions.

Pattern of jaw-finger synchronization

The effect of stress position on P A/J is significant, with P A/J smaller for 'CVCV (-0.02) than for CV'CV (0.41) (p < .0001). The effect of stress position on P R/J is also significant, with P R/J smaller for 'CVCV (0.44) than for CV'CV (0.95) (p < .0001). Target position and consonant order effects on P A/J are not significant, but P R/J depends on the target position with greater value in the far (0.73) than in the near (0.66) target condition (p < .01). Interaction effects are not significant. Hence, the synchronization pattern is similar to the one found previously [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF]. Firstly, the pointing apex is close to the first jaw apex for 'CVCV sequences while it occurs at about an equal delay from the two jaw apices for CV'CV ones. Secondly, the pointing return occurs close to the second jaw apex for CV'CV names while it occurs at about an equal delay from the two jaw apices for 'CVCV ones. This pattern of synchronization does not depend on target position or consonant order.

Finger motion

Increase in target distance results in a significant 11 ms advance of P I , and a significant delay of P A and P R (respectively 11 ms and 30 ms, p < .05). Thus, the pointing forward motion (from P I to P A ) is 20 ms longer in the far-than in the near-target condition (p < .0001; 5, top right). The stroke amplitude is also 147 mm greater in the far-than in the near-target condition (p < .0001, Figure 5, top left). Finally, the plateau (P R -P A ) duration is 19 ms longer in the far-than in the near-target condition (p < .01, see Figure 3). The stress position has no significant effect on P I and P A and on the amplitude and the duration of the forward motion. On the contrary, P R is 35 ms later for CV'CV than for 'CVCV sequences (p < .001). Hence the plateau (P R -P A ) duration is 22 ms longer for CV'CV than 'CVCV sequences (p < .001). The consonant order has no significant effect on P I and P A and on amplitude and duration of the forward motion. Then, this factor has a significant but weak effect on the (P R -P A ) duration (4 ms longer for /tp/ than /pt/, p < .05). No interaction effect is significant. Altogether, the finger forward motion just depends on the target position. Then, once the alignment with the target is achieved, the plateau duration adapts to the stress position: the finger can wait for the jaw.

Jaw motion

Figure 3 shows that all jaw events J I1 , J A1 , J I2 and J A2 occur significantly earlier for CV'CV than 'CVCV names (p < .001). The J A1 to J A2 duration is larger for 'CVCV utterances (290 ms) than for CV'CV ones (275 ms). Duration of the first jaw motion is 35 ms greater for 'CVCV than for CV'CV utterances (p < .0001). Conversely, the second jaw motion is 44 ms longer for CV'CV than for 'CVCV utterances (p < .0001). Similarly, the amplitude of the first stroke is 3.1 mm larger for 'CVCV than for CV'CV utterances (p < .0001) while the amplitude of the second stroke is 3.8 mm larger for CV'CV than for 'CVCV utterances (p < .0001). The consonant order has a significant effect on J I1 and J I2 , occurring respectively 34 ms (p < .001) and 20 ms (p < .01) earlier for /tp/ than for /pt/, but no significant effect on J A1 and J A2 . It does not significantly affect the first stroke amplitude while the second stroke is 3.2 mm larger for /tp/ than /pt/ (p < .0001). Furthermore, the first and second jaw strokes are, respectively, 29 ms (p < .001) and 26 ms (p < .0001) longer for /tp/ than /pt/. The interaction between stress and consonant order is significant only for J I2 with a greater stress effect for /pt/ than for /tp/ (p < .05). Jaw events occur about 10 ms earlier in the near-than in the fartarget condition. This target effect is significant for J A1 , J I2 and J A2 (p < .05) but not for J I1 . Moreover, it does not significantly interact with the two other factors and it has no significant effect on amplitude and duration of jaw strokes. Altogether, jaw motion is mainly driven by the phonetic goal with different profiles according to the stress position and to the consonant order. The timing of jaw events is also adapted to the finger pointing motion, as showed by the target position effects on jaw timing.

Discussion

A stable jaw-finger coordinative pattern

Globally, the results confirm those we previously obtained with /papa/ and /tata/ target names [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF]. First, the stress position has the same effect on jaw-finger synchronization. The jaw apex for the stressed vowel is included in the period of finger-target alignment. More precisely, for 'CVCV names, the jaw apex for the stressed vowel is aligned with the onset of finger-target alignment. For CV'CV names, the jaw apex for the stressed vowel is aligned with the end of finger-target alignment. This pattern of synchronization mainly results from both an adaptation of the jaw motion and of the duration for which the finger points to the target. Indeed, jaw response begins earlier for CV'CV as compared to 'CVCV. This lead makes the second jaw apex closer to the pointing apex. However, it is not enough to make the second jaw apex synchronized with the pointing apex. Consequently, the finger has to stay longer in its apex position for CV'CV names as compared to 'CVCV ones: it waits for speech focus achievement. This mainly results from the fact that the pointing motion for reaching the target alignment is "target driven": it may begin as soon as possible in order to reach the target. This "externally-driven" system would take priority over the "internally-driven" and more flexible speech system, which agrees with previous results [START_REF] Levelt | Pointing and voicing in deictic expressions[END_REF][14] [START_REF] Feyereisen | The competition between gesture and speech production in dual-task paradigms[END_REF].

How does jaw motion adapt to the phonetic and deictic requirements?

The jaw motion is affected by the stress position, the consonant order and the target position. The stress position modifies both the motion timing and parameters. The initiation and apex of opening strokes occur earlier for CV'CV than for 'CVCV sequences. Unsurprisingly, the duration and amplitude of the first stroke are greater for 'CVCV than for CV'CV utterances while the reverse is observed for the second stroke. The consonant order affects the initiation times of jaw opening strokes but not their apices times. It also affects the amplitude and duration of jaw strokes with a tendency to observe longer opening strokes for /tp/ than for /pt/ and a larger second opening stroke for /tp/ as compared to /pt/. However these changes do not affect jaw-finger apices synchronization. The target position mainly affects the timing of jaw motion. The lag of initiation and apex times of the two jaw strokes from the near-to the far-target condition is 10 ms, which is about that observed in [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF] and corresponds to the delay of the finger apex from the near-to the far-target. Hence, the increase of target distance induces a translation in time of jaw motion while amplitudes and durations remain the same. This adaptation allows preserving the jaw-finger synchronization pattern for both target positions.

Conclusion

Taken together, these results agree with the idea of an anchoring of speech focus in pointing gesture in deictic expressions. Moreover, this anchoring seems to be supported by a synchronization of the speech-frame with the sign-frame [START_REF] Ducey-Kaufmann | When the Speech Frame meets the Sign Frame in a developmental framework[END_REF]. This happens to result from two independent levels of speech adaptation. Firstly, the effects of stress position and consonant order indicate an "internal" adaptation of the gestures to the phonetic goal. Secondly, the spatial target effect suggests an "external" adaptation for the synchronization with the pointing motion. These results agree with those obtained on /papa/ and /tata/ sequences [START_REF] Rochet-Capellan | Finger-jaw coordination during a deictic gesture with CVCV utterances: the effect of stress position[END_REF]. Within the hotly debated topic of the gestural vs. vocal origin of human language [START_REF] Corballis | From mouth to hand : Gesture, speech and the evolution of right-handedness[END_REF], recent theories suggest a reconciliation of gestures and vocalisations in proposals where the coordination of sounds and gestures would be a crucial step [START_REF] Abry | Introduction: Vocalize to Localize? A call for better crosstalk between auditory and visual communication systems researchers: From meerkats to humans[END_REF] [START_REF] Arbib | Interweaving protosign and protospeech: Further developments beyond mirror[END_REF]. Future experiments allowing to better understand the coordination of the orofacial and brachiomanual systems in face-to-face communication should provide very important inputs in this framework.

  A block contained 4 practice trials and 40 experimental trials, 10 for each [stress position]*[target position]*[consonants order] experimental condition. The order of the trials was randomized for each block and each participant.

Figure 2 .

 2 Figure 2. Record and labeling for a /patà/ trial.

Figure 4 .

 4 Figure 4. Position of pointing apex (P A/J , left) and pointing return onset (P R/J , right) relatively to jaw apices according to experimental conditions.

Figure 5 .

 5 Figure 5. Amplitude and duration of finger forward stroke (top) and of first (middle) and second (bottom) jaw opening strokes according to experimental conditions.

Acknowledgements

This work is part of the "Patipapa" project funded by the French Ministry of Research (Action Concertée Incitative "Systèmes Complexes en Sciences Humaines et Sociales"). We also wish to thank Christian Abry for his comments on this work.


• Indigenous/local communities : « populations at disproportionally higher risk of adverse consequences with global warming of +1,5°C »; In particular those dependant on agricultural or coastal livelihoods (high confidence). (IPCC, SR15 SPM, 2018)

Northern saami

• Sitnu

• Ràssi

• Jeagil

• Lahppu

• Gatna

Inga, B. ( 2008). Traditional ecological knowledge among reindeer herders in northern Sweden.

Sitnu :

• Litteraly « short grass » Rássi :

• Litteraly « Tundra flowers » "The name used for grass and sedges, and also for forbs"

• Graminoids plants, dicotyledons; mainly Apiaceae family?

• Mainly hygrophilous plants, describing ground cover plants in :

• Wooded tundras (Betula pubescens sp forests) Jeagil :

• Litteraly « lichen »

• Terricolous lichens communities (dozen of species/subspecies)

• Mainly consumed during winter

• Wide distribution along taïga -toundra ecosystems Inga, B. (2008). Traditional ecological knowledge among reindeer herders in northern Sweden.

Lahppu :

• Litteraly : « thick thread »

• Epiphytic/arboreal lichens (Usni, Bryoria…)

• Mainly found on spruce and pine trees => Boreal distribution

• Highly sensible to pollution and loss of habitats (intensive forestry)

• Example: pine forest composition Sitnu Ràssi Jeagil Lahppu Gatna

• Diversity of life = diversity of way to describe life

• Linguistic categories -scientific categories (empirical)

Taxonomy of vegetal formations… applied to a space (specie coverage; community; vegetal formation; vegetal landscape; biome…)

• Plants important for semi-domesticated reindeer : not species, but functional groups

• Ecosystem services might be adapted according to linguistic contexts