R Haenel

Q Semler

E Semin

P Grussenmeyer
email: pierre.grussenmeyer@insa-strasbourg.fr

E Alby

INTEGRATION OF DEPTH MAPS FROM ARCORE TO PROCESS POINT CLOUDS IN REAL TIME ON A SMARTPHONE

Keywords: 3D scanning, smartphone, real-time, depth map, on-the-fly-reconstruction, ARCore

Real-world three-dimensional reconstruction is a project of long-standing interest in global computer vision. Many tools have emerged these past years to accurately perceive the surrounding world either through active sensors or through passive algorithmic methods. With the advent and popularization of augmented reality on smartphones, new visualization issues have emerged concerning the virtual experience. Especially a 3D model seems to be essential to provide more realistic AR effects including consistency of occlusion, shadow mapping or even collision between virtual objects and real environment. However, due to the huge computation of most of current approaches, most of these algorithms are working on a computer desktop or high-end smartphones. Indeed, the reconstruction scale is rapidly limited by the complexity of both computation and memory. Therefore, our study aims to find a relevant method to process real time reconstruction of close-range outdoor scenes such as cultural heritage or underground infrastructures in real time locally on a smartphone.

INTRODUCTION

Real-world three-dimensional reconstruction is a subject of longstanding interest in computer vision. It has become even more predominant with the advent and popularization of autonomous cars or augmented reality. Especially a 3D model seems to be essential to provide more realistic AR effects including consistency of occlusion, shadow mapping or even collision between virtual objects and real environment. This opened up the need for real time scanning at scale with continuous integration of the accumulated 3D data.

Initially, reconstructions were created by fusing depth measurements from specific active sensors such as structured light, time of flight [START_REF] Izadi | KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera[END_REF] or LiDAR (Lightning Detection and Ranging) into 3D models. Although these sensors provide accurate results, their expansive aspect and the need for adequate equipment make them less attractive. Therefore, multiple approach have emerged to reconstruct a scene based on monocular (Yang et al., 2011), binocular or multi-view stereo methods that predict depth maps according to RGB images only. A depth map represents an image where pixels does not contain colour information, but a distance to objects in the scene from a given viewpoint (Figure 1). This data describes a pseudoreconstruction of space that does not directly represent a surface, but rather a sample of discrete values that allows understanding the geometry of the environment captured from a specific camera position.

Most of the existing solutions are relying on massive data exchange and cloud computing because the smartphone alone cannot provide sufficient computational performance. Therefore, the challenge of our research is to imagine a new approach of producing outdoor 3D models from a smartphone-based acquisition and a full processing performed on the device for the * Corresponding author SYSLOR Company. We aim to reach a productivity in real-time in order to provide a continuous overview of the 3D reconstruction from the recording. The interest of this visualisation is to assist continuously the user during the survey by providing him security and confidence regarding both the quality of the acquisition and the exhaustiveness of the data produced.

The algorithmic procedure has thus induced a succession of constraints to be taken into account during the developments:

-Ensuring a computational reliability of the process efficiently supported by the performances offered by the current smartphones, -Enabling optimization of the computational process to achieve real-time productivity, -Ensuring accessibility of the acquisition for everyone.

STATE OF THE ART OF EXISTING SOLUTIONS

Before presenting how reconstruction can be processed, we need to take a closer look at scene representation. As depicted by [START_REF] Seitz | A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms[END_REF], it is possible to consider a differentiation of the reconstruction approaches in terms of scene representation, since the geometry of an object can easily be described according to:

-A voxel grid (3D equivalent of pixel), used for its simplicity and its ability to approximate any surface, -A polygonal mesh (usually triangular), very popular for simplicity of storage and convenience of data visualization, -A depth map, which avoids the need to resample the 3D geometry.

Then, more generally, the reconstruction approaches can be subdivided into four ways according to how they are undertaken:

1. Extraction and mapping of feature points used to fit a surface to the generated features surface, 2. Calculation of a cost function associated with a 3D volume, then extraction of a surface from it, 3. Recurrent evolution of a surface to minimize a cost function, 4. Computation of a set of depth maps.

2.5 reconstructiondepth maps

In the absence of specific depth sensors, depth data can come from the study of a colorimetric similarity, also called photoconsistency criterion, between the pixels of a stereoscopic pair. A first well-known approach is the planesweep algorithm [START_REF] Collins | A space-sweep approach to true multi-image matching[END_REF]. The idea of this algorithm is to project backwards the whole image on successive virtual planes, perpendicular to the center of projection, indented according to a certain sampling step in 3D space. According to each virtual plane established, each pixel will be assigned a depth value derived from a similarity calculated between a reference view and the adjacent viewing position (Figure 2). Especially [START_REF] Muratov | 3DCapture: 3D Reconstruction for a Smartphone[END_REF] have reused and improved this method to generate a highresolution mesh within minutes on a smartphone. The nature of the photo-consistency indicator determining the depth value has a significant impact on the relevance of the results in the case of changes in illumination or edge effects as indicated by [START_REF] Molas | Multi-view 3D Reconstruction via Depth Map Fusion for a Smartphone Application[END_REF]. More recently, depth maps find a relevant use in augmented reality application, especially to offer management of occlusion effects between the real world and the virtual contents [START_REF] Valentin | Depth from motion for smartphone AR[END_REF]. In the Depth API provided by ARCore, depth maps are generated from the hybrid optimization of two algorithms: PatchMatch [START_REF] Bleyer | PatchMatch Stereo -Stereo Matching with Slanted Support Windows[END_REF] and HashMatch [START_REF] Fanello | Low Compute and Fully Parallel Computer Vision with HashMatch[END_REF]. On the basis that an image is made up of regions of constant depth, the idea is to alternate generating an isolated depth value and propagating it to neighbouring pixels (Figure 3).

Finally, the popularization of machine learning has led to the emergence of new solutions for creating depth maps, notably by integrating convolutional neural networks (CNN) such as MVSNet [START_REF] Yao | MVSNet: Depth Inference for Unstructured Multi-view Stereo[END_REF] or DORN [START_REF] Fu | Deep Ordinal Regression Network for Monocular Depth Estimation[END_REF].

Figure 3. Process for creating depth maps within ARCore

Depth maps fusion

Usually, to fully reconstruct a 3D model, accumulated depth maps need to be gathered and combined. The integration of all these new data allows to continuously update the model and can be approached from different perspectives.

A first one is to represent the model as a surfel (surface element) cloud. Adopted by [START_REF] Kolev | Turning Mobile Phones into 3D Scanners[END_REF], this representation seems more adequate for interactive applications running in real time in contrast to triangular meshes [START_REF] Piazza | Real-Time CPU-Based Large-Scale Three-Dimensional Mesh Reconstruction[END_REF]. Indeed, the unstructured set of surfels can easily be kept consistent. A very interesting alternative to the one proposed above is based on Truncated Signed Distance Functions (TSDF). Inspired by (Zach, 2008), the idea is to convert the depth maps into a Signed Distance Field (SDF) where pixels are projected as voxel (3D point) with a value between [-1;1]. This value describes the distance of the voxel to the true surface of the object. This volumetric fusion of maps, taken up by [START_REF] Izadi | KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera[END_REF] in KinectFusion and [START_REF] Graber | Real time 3D-Reconstruction[END_REF], is interesting since it offers a relatively robust management of outliers while guaranteeing an efficient update of the model.

Then, the geometry of the model can be extracted using known methods such as raycasting [START_REF] Graber | Real time 3D-Reconstruction[END_REF]) [START_REF] Ondruska | MobileFusion: Real-Time Volumetric Surface Reconstruction and Dense Tracking on Mobile Phones[END_REF] or marching cubes [START_REF] Lorensen | Marching cubes : A high resolution 3D surface construction algorithm[END_REF]. The raycasting approach is generally preferred over the marching cubes method because the latter relies on a binary decision to determine the surface position. However, as a result, the geometry of the 3D model may contain false positives, i.e. nonexistent artefacts, as well as false negatives, i.e. poorly extracted features. In addition, the marching cubes method is more favourable because it offers directly a mesh model against a point cloud for raycasting.

3D volume creation

Rather than going through an intermediate representation of the data, it is possible to build a volumetric geometry directly. From CNNs constructed as graphs, Pixel2Mesh [START_REF] Wang | Pixel2Mesh: Generating 3D Mesh Models from Single RGB Images[END_REF] with and GEOMetrics [START_REF] Smith | GEOMetrics: Exploiting Geometric Structure for Graph-Encoded Objects[END_REF] are extracting spatial deformations that will shape a predefined mesh model to converge to the captured geometry. Pixel2Mesh uses dissociation layers to refine the facets of the mesh by uniform oversampling.

In GeoMetrics, the mesh is redefined only in target areas, which allows a better adaptation to the local complexity of an object. Nevertheless, these two methods are built from an initial mesh model, either a sphere or an ellipse, which directly induce topological biases.

To bypass this specific constraint, [START_REF] Gkioxari | Mesh R-CNN[END_REF] developed Mesh R-CNN. The 3D mesh structure is obtained indirectly by passing through an intermediate description, namely a voxel occupation grid. This coarse description of the object will be transformed into a triangular mesh using the operation called cubify. Based on the voxel occupancy probabilities within the grid, each voxel is substituted by a triangular cubic mesh with 8 vertices, 18 edges and 12 faces. Then, the vertices and edges common to adjacent entities are merged while shared interior faces are eliminated.

Comparison of reconstruction methods

The specifications defined within our problematic enabled us to classify the various methods. Some of them have already been implemented on old generation smartphones [START_REF] Ondruska | MobileFusion: Real-Time Volumetric Surface Reconstruction and Dense Tracking on Mobile Phones[END_REF][START_REF] Muratov | 3DCapture: 3D Reconstruction for a Smartphone[END_REF] or latest generation smartphone [START_REF] Valentin | Depth from motion for smartphone AR[END_REF]. This observation confirms that 3D processing can be potentially embedded on this type of device.

Although most of the presented methods based on deep learning represent the concrete future of real-time reconstruction for computer vision, they are still too limited for direct implementation on a smartphone.

We have thus confronted the identified methods with our specifications (Table 1). The results provided are the outcome of tests undertaken, but for many an investigation carried out in relation to the remarks reported and correlated between the various authors cited. During the study, we have especially noticed that depth maps are particularly relevant data to bypass the triangulation steps related to collinearity equations while being an interesting trade-off between computation complexity and effective results. Especially, we highlighted the relevancy of the planesweep method implemented on smartphone by (Muratov et al., 2017) and the ARCore Depth API [START_REF] Valentin | Depth from motion for smartphone AR[END_REF] to generate accurate depth maps. Therefore, these methods seem to be very interesting for the development of our solution. Concerning data fusion, we prefer the method using surfels (Schöps et al., 2017), as it seems to be quite relevant for an interactive application while being within reach of our objectives from an algorithmic point of view.

PRODUCTION OF DEPTH DATA

Planesweep algorithm

The planesweep method presented by [START_REF] Collins | A space-sweep approach to true multi-image matching[END_REF], operates locally to estimate the photo-consistency between two adjacent views. Its interest is to promote a result relatively quickly by ensuring robustness to incongruous changes in brightness, but also in the event of large bases. The indicator used to measure photo consistency directly influences the calculation of colorimetric homogeneity and therefore affects the overall quality of the depth maps.

We tested several indicators very well described by (Mari [START_REF] Molas | Multi-view 3D Reconstruction via Depth Map Fusion for a Smartphone Application[END_REF] in order to determine the relevance of the method as a whole. Among those studied (SAD Sum of Absolute Differences, SSD Sum of Squared Differences, NCC Normalized Cross Correlation, Census transform, Rank transform), the one that seems to be the most relevant because of a favourable ratio accurate results / computation time is the SAD indicator (see Figure 4). Indeed, it allows describing the photographed environment more efficiently with a minimum error of 13.1 cm. This result was obtained after comparing computed depth map to a ground truth map obtained from a famous benchmark: fountain-P11 [START_REF] Strecha | On benchmarking camera calibration and multi-view stereo for high resolution imagery[END_REF].

After testing the influence of the parameters composing the planesweep, it appears that this method is quite legitimate for generalized multi-view reconstruction applications. However, given that it is based on a purely sequential computational procedure, it results in a latency of the calculations. Therefore, we have discarded this solution and turned to a more localized approach of stereo-correspondence, which uses a simpler procedure to implement, namely the API integrated in Google's ARCore application.

Depth API ARCore

The development of mobile technologies, has allowed the emergence and democratization of augmented reality applications. Google services have especially developed an API integrating the measurements of depth maps generated from a monocular camera [START_REF] Valentin | Depth from motion for smartphone AR[END_REF] at a rate of 30 Hz. In order to understand how these depth maps are generated, it is necessary to look at the overall working principle of ARCore. This application is based on three fundamental concepts: motion tracking, environment understanding and light estimation. Using an approach similar to conventional SLAM, feature points are detected along the movement to locate the smartphone in space. Using a methodology similar to RANSAC, these points are used to determine average planes ensuring an accurate understanding of the surrounding environment.

To use the Depth API from ARCore wisely, it is important to take into account the following factors that can influence the quality of a recording: -Illumination conditions, -Potential phenomena of reflection or scattering specific to the materiel, -Poor textured scenes.

Figure 5 shows the analysis of the data produced by ARCore. The first case was performed in a relatively poor textured environment. The differences in depth observed are of the order of ten centimeters. This is interesting but it highlights the limitations of the solution, which relies solely on visual colorimetric information to calculate the different distances. The second case represents a more realistic scene with different depths. We can see right away that the results are better than the first case. In particular, the difference in depth is estimated to be about 8 cm. It is important to understand that these results depend on the quality of the movement performed, so a redundancy of views will provide more accurate results than a single view as showed in Figure 5. ARCore applies a smoothing process to its depth maps in order to express a value for all the pixels of an image. This means that the depth gradient is quite small; implying that around the edges of the objects discontinuities will appear causing a slight difference in accuracy within the comparison. To get rid of these distorted values, we set up a post-processing of the data consisting in the elaboration of a discontinuity mask based on a Canny detection [START_REF] Canny | A Computational Approach to Edge Detection[END_REF] followed by mathematical morphology operators to remove a maximum of uninteresting pixels. The size of the kernel is chosen so that only the dominant contours are kept in the image.

Finally, the API provided by ARCore meets our expectations more favourably in terms of both the quality of the results and the rate of data production. This method has therefore been selected in our developments.

3D RECONSTRUCTION METHODOLOGY

ARCore can been used in the reconstruction process. Nevertheless, [START_REF] Manni | Snap2cad: 3D indoor environment reconstruction for AR/VR applications using a smartphone device[END_REF] decided to bypass the classical reconstruction process by using an object classification method to determine the most similar synthetic object from a database.

The contribution of depth maps alongside image data, resulting in so-called RGB-D data, opens the way to the creation of 3D models. Thanks to all the research carried out, were able to design an optimal processing chain that converts RGB-D data to point clouds based on the intrinsic parameters of the camera. Figure 6 illustrates some reconstructed point clouds created with our method.

Characteristics of point clouds

The characteristics of the produced point clouds are directly influenced by the resolution of the RGB-D images. Initially, the resolution of the depth maps produced by ARCore is 160x120 pixels to ensure a high productivity. Given that each pixel in the image produces a single 3D point, this will lead to a sparse point cloud of 19200 points with a spacing of about 2 cm (Figure 7a). Therefore, we have increased the size of depth maps to 320x240 and 640x480 pixels using linear interpolation to oversample the data. This operation intends to increase the amount of data from 19200 to 76800 and 307200 points respectively while providing a better density of points with a spacing reduced to 1 cm and 0.5 cm respectively. Even if more data to be processed implies more calculations to be carried out, a simple parallelization of the projection will be enough to guarantee a real time productivity. Moreover, visually, a higher resolution makes sense when we wish to have a finer detail of the recorded scene (Figure 7b-7c).

Analysis of results obtained

We built a proof of concept that can be generalized to several environments. Our solution has been designed to digitize all elements within a maximum range of 4 m. Actually, beyond a certain range, depth data provided by ARCore are going to be less accurate. Indeed, as one moves away from the scene or the object, the depth values will become more and more inaccurate. In particular, the ARCore documentation informs us that the data estimation error increases quadratically with the distance.

Therefore, to quantify the value of the models resulting from our method, we conducted comparisons with a ground truth built from Agisoft Metashape. The results are quite interesting and encouraging since we obtain an average deviation of 6-7 cm (Figure 8). The accuracy of our point clouds is directly linked to factors influencing depth estimation without forgetting the acquisition procedure. Indeed, to provide good estimation, a solid motion tracking at an adapted speed is required so that the algorithms can correctly understand the surrounding environment. Moreover, since the depth data results from colorimetric similarity between the pixels of a stereoscopic pair, deformations can be visible on the facade or the ground. For a first sketch of the result, the latter is quite promising, especially since the generation of the models is done almost in real time.

Computational bottleneck

Producing 3D model through cloud processing induces data exchange that can be time-consuming. Therefore, after conducting the acquisition, users have often to wait a certain amount of time to have a feedback. By relying on simple but effective methodology, we designed a proof of concept to ensure a permanent monitoring of the reconstruction with reasonable computation time.

At the time we present these results, our solution has not yet been fully integrated into the smartphone due to code conversion issues. Therefore, time consumptions are given according to computer computation using the minimum amount CPU (Central Processing Unit) and by doing without the GPU (Graphical Processing Unit). In Table 2, we provide an overview of the average computation time of creating 3D model from a single view according to different input data resolutions. Results are very promising because we are able to produce very efficiently complex data consisting of several hundred thousand points all in a fraction of a second. In comparison, with a basic photogrammetry process, it can took several minutes to the corresponding model. Even if progress in automation has been made, allowing reducing the production time, the results of our method are still produced more quickly and requires less computing power.

During our we also compared computational time while including post processing of the depth maps provided by ARCore (Table 2). As we already mentioned, these depth maps are smoothed to provide sufficient data all over the scene, but this induces flying points when we compute the corresponding 3D model (Figure 9).These particular points are outliers that damage the visual representation of the scene. The process described to remove these points seems to lead to an increase in calculation time of about 20%. Therefore, further research have to be conducted to find some alternatives that suppress these outliers without increasing that much the computational time. However, in the case of more accurate depth data, we can easily imagine doing without this process.

CONCLUSION

Designing a proof of concept is not an easy thing to do, especially when significant material constraints are imposed. It implies having to find a computational approach simple enough while being as effective as possible compared to a reference method.

For our study, we had three major constraints on which our developments had to rely:

-Ensuring a computational reliability of the process efficiently supported by the performances offered by the current smartphones, -Enabling optimization of the computational process to achieve real-time productivity, -Ensuring accessibility of the acquisition for everyone.

Thanks to all the researches and tests carried out, we ended up with a first processing chain, which works quite well. We are able to create dense point clouds within an accuracy of 7 cm for a single view, which can further be refined by the addition of more surrounding views. Even if the depth maps generated by ARCore can be noisy in the context of scattering surfaces or around edges, we still managed to develop a very satisfactory method that provides quite encouraging results of about 8 cm. The initial objectives of the project led by the SYSLOR Company have been well outlined since we are able to provide results very quickly with limited computing power compared to usual method such as photogrammetry, which provide relevant information about the completeness and conformity of the acquisition. Although real time has not yet been concretely reached since the implementation of the algorithm has not been completely finished on smartphone, the provided initial results are very promising.

FUTURE WORKS

Our ambitions go beyond a simple ephemeral visualization as found in most AR applications integrating reconstruction methodologies. We want to establish a 3D model used to concretely map the environment around us, and more specifically underground infrastructures. For this, we wish to integrate extrinsic elements to our methodology, in particular GNSS data from a low cost antenna developed by (Quentin et al., 2019) within the company SYSLOR for permanent geolocalized restitution.

Figure 1 .

 1 Figure 1. Example of depth map Source: (Wojciech Mo, 2016)

Figure 2 .

 2 Figure 2. Planesweep methodSource:[START_REF] Graber | Real time 3D-Reconstruction[END_REF]

Figure 5 .

 5 Figure 5. Two cases of analysis of depth results provided by ARCore

Figure 4 .

 4 Figure 4. Impact study of photo-consistency measurements on depth maps quality

Figure 6 .

 6 Figure 6. Example of reconstructions provided by our solution

Figure 7 .

 7 Figure 7. RGB-D resolution impact on point clouds

Figure 8 .

 8 Figure 8. Qualitative analysis of a resulting point cloud

RGB

Figure 9 .

 9 Figure 9. Flying points removal

Table 1 .

 1 Conclusion of our study concerning reconstruction methods

		Real time productivity	Computation optimization	Reconstruction quality	Reference
	3D-R2N2	*	*	*	(Choy et al., 2016)
	Depth API ARCore	***	***	***	(Valentin et al., 2019)
	DORN	*	*	*	(Fu et al., 2018)
	GEOMetrics	*	*	***	(Smith et al., 2019)
	KinectFusion	***	**	*	(Izadi et al., 2011)
	Marching cubes	**	**	***	(Lorensen and Cline, 1987)
	Mesh R-CNN	*	*	***	(Gkioxari et al., 2019)
	MobileFusion	*	*	***	(Ondruska et al., 2015)
	MonoFusion	*	*	***	(Pradeep et al., 2013)
	PatchMatch	***	**	*	(Bleyer et al., 2011)
	Pixel2Mesh	*	*	***	(Wang et al., 2018)
	Planesweep	***	***	**	(Collins, 1996) (Muratov et al., 2017)
	MVSNet	*	*	***	(Yao et al., 2018)
	Raycasting	***	***	**	(Graber, 2012)
	RoutedFusion	**	*	**	(Weder et al., 2020)
	SurfelMeshing	*	**	***	(Schöps et al., 2019)
	Delaunay Tétraédization	*	**	*	(Piazza et al., 2018)
	TGV	**	**	**	(Graber, 2012)
	Visual hull constraints	***	***	**	(Prisacariu et al., 2015)

* : Poor consistency, ** : Acceptable consistency, *** : particular adequacy to the criteria

Table 2 .

 2 Computational time of our method for a single view

This contribution has been peer-reviewed. https://doi.org/10.5194/isprs-archives-XLIII-B2-2022-201-2022 | © Author(s) 2022. CC BY 4.0 License.


Pauline Beaupoil

Aliyah Morgenstern

A child's multimodal negations from 1 to 4: the interplay between cognitive and linguistic processes

, children's first negative constructions seem to take over from early gestures of rejection and avoidance.

This study explores the status and evolution of gestures of negation produced with and without words as well as the role of co-verbal gestures used in combination with negative verbal productions. Our multimodal analyses are conducted on the longitudinal data of an English monolingual girl recorded at home interacting with her mother between 10 months and 4 years old.

We categorised 1) all her negative gestures produced alone and 2) all the gestures she used with words, from their pre-linguistic to their co-verbal uses. Our analyses show that gestures and words are associated and complement each other throughout the corpus when the child needs to express negation.

Results show that the role of gestures changes between 10 months and 4 years old and that their transformation could be divided into five periods. 1) Gestures seem to emerge from actions and are not quite distinctive; 2) the child uses gestures alone; 3) she uses gestures with one or two words; 4) her speech becomes more complex and she uses fewer gestures; 5) she reintroduces gestures but uses them as co-verbal gestures along with longer, more complex negative spoken utterances. Her multimodal negative communication develops from embodied negation to symbolic negation. At the end of the data, the use of gestures becomes fully integrated into multimodal complex productions with a rich diversification of functions as well as forms. Once the child can speak fluently, gestures are no longer used to fill in a lexical gap but rather to illustrate, specify and reinforce the meanings of her negative spoken productions and are part of her multimodal multifunctional linguistic system.

Introduction

Negation, as one of the first linguistic phenomena that is grammatically marked in a child's language, is a popular subject of study in language acquisition [START_REF] Dimroth | The acquisition of Negation[END_REF]. Studies on children's early negative utterances have used different perspectives and criteria. Some have analysed the use of one negative linguistic form such as "non" (no in French) [START_REF] Dodane | La prosodie dans l'acquisition de la negation : étude de cas d'une enfant monolingue française[END_REF]; others have concentrated on the construction of the negative polarity in children's spoken productions (Bloom 1970;[START_REF] Wode | Four early stages in the development of LI negation[END_REF]; some others have used a semantic perspective and tried to link each spoken form with a specific negative function [START_REF] Choi | The semantic development of negation: acrosslinguistic study[END_REF][START_REF] Cameron-Faulkner | What part of no do children not understand? A usage-based account of multiword negation[END_REF]). Children's acquisition of negation has therefore been analysed from different methodological perspectives. Previous studies on children's negative spoken productions have shown that no is the most consistently used word throughout the single word utterance period (Pea, 1980:170) and that the first negative functions children express are rejection, refusal and protest (Spitz, 1957;Bloom, 1970;[START_REF] Clark | From gesture to word, on the natural history of deixis in language acquisition[END_REF][START_REF] Vaidyanathan | Development of forms and functions of negation in the early stages of language acquisition: A study in Tamil[END_REF][START_REF] Dodane | La prosodie dans l'acquisition de la negation : étude de cas d'une enfant monolingue française[END_REF][START_REF] Beaupoil | International conference of the AFLiCO Empirical Approaches to Multi-modality and to Language Variation[END_REF]. These studies showed that children begin using spoken productions for negation around 1;071 (Tomasello, 2003:228-229). However, other studies have analysed children's actions and gestures used before they speak and have shown that children express negation long before 1;07. Spitz (1957) observed that first negative constructions emerge thanks to early actions of rejection and avoidance. [START_REF] Guidetti | Yes or no? How do young children combine gestures and words to agree and refuse[END_REF] demonstrated that gestures of negation are among the first symbolic gestures used by children and when they fully enter the spoken modality, they then mostly use words. Several studies have analysed the role of gestures prior to speech as well as during the one-and two-word utterance period and have concluded that gestures trigger language development [START_REF] Iverson | Gesture Paves the Way for Language Development[END_REF][START_REF] Iverson | Developing language in a developing body: The relationship between motor development and language development[END_REF]. Since there is continuity between actional/gestural expression and spoken productions, the use of gestures and actions ought to be analysed when studying children's linguistic development. As pointed out by Eve Clark (1978:86):

"Children start with gestures alone -pushing away unwanted food, resisting being dressed -then replace these with single word utterances like 'no' or 'not'. Later they make what is being negated more explicit, e.g. 'no go out', 'no bed', and later still produce even more complex expressions like 'I don't want to go out', or 'it's not bedtime yet'". Continuities like the ones Eve Clark describes allow us to trace the developmental paths children follow as they acquire their first language. Therefore, studies on the various channels of expression -whether visual with actions and gestures or auditory with words and vocal productions -indicate that the topic of negation is a good candidate to understand how children acquire speech and manage to communicate with adult interlocutors even when they are not yet proficient speakers of their mother tongue.

The functions of negation are numerous and researchers have tried to organize them into typologies in order to account for their emergence and development. Typologies have often been divided into three or four categories in which several negative functions could be classified [START_REF] Mcneill | What does a child mean when he says[END_REF]Bloom, 1970;[START_REF] Volterra | Negation in child language: a pragmatic study[END_REF][START_REF] Pea | The Development of Negation in Early Child Language. The Social Foundations of Language and Thought: Essays in Honor of Jerome[END_REF]. In their 1968 typology, McNeill & McNeill did not distinguish between classical negative functions such as refusal, denial, nonexistence or epistemic negation. Instead, they classified children's spoken productions into categories dealing with the syntactic and semantic value of the negation. Their three categories were existence/truth, external/internal, entailment/nonentailment. Volterra & Antinucci's study (1979) was the first one to propose a pragmatic typology of the acquisition of negation. It was divided into four categories and the authors convincingly demonstrated that children are able to understand the notion of polarity around 1;06. Volterra and Antinucci explain that around 1;06, children learn to negate "a corresponding positive presupposition attributed to the listener " (1979: 283). At first children's negations are about objects or people present in the situation and the immediate environment. But they must progressively learn to consider that their interlocutor's intentions or beliefs may differ from theirs. The authors also claimed that a detailed analysis of the situation and of the context in which negative forms were produced was the most accurate way of classifying children's negative productions. Later, [START_REF] Choi | The semantic development of negation: acrosslinguistic study[END_REF] and more recently Cameron-Faulkner and colleagues (2007) conducted thorough multilevel corpus-driven analyses on the emergence and development of negative functions and they created typologies that draw distinctions between several negative functions like refusal, denial, failure, epistemic negation, nonexistence and negative assertion. In line with [START_REF] Volterra | Negation in child language: a pragmatic study[END_REF], they classified the occurrences according to syntactic, semantic and pragmatic parameters.

The present paper presents a multilevel study of the emergence and development of negation in a child acquiring English. It unites the syntactic, semantic and pragmatic perspectives used in the work reviewed above. However, we also adopted a multimodal methodological approach by analysing all negative communicative acts the child expressed. This involved coding all spoken negative productions in the child's data as well as all Acts of Bodily Communication [START_REF] Zlatev | Stages and transitions in children's semiotic development[END_REF] that could be conveyed with actions or gestures. Our research investigates the interface between the visual modalitiesactions and gestures -and the auditory modalities -speech and vocal productions -and it takes into account the combinations of these modalities in the construction of negative meaning in one English monolingual child.

This paper will provide a qualitative overview of the multimodal development of negation in an English-speaking child and can serve as a starting point for other studies on the blossoming of multimodal negation. Through the lens of negation, this study also addresses the role and impact of multimodality in the development of language.

Data and Methodology

The topic of the construction of spoken negation in young children has caught researchers' attention because the understanding of the notion of polarity has been described in the literature as a complex cognitive activity [START_REF] Bruner | A study of thinking[END_REF]De Boysson-Bardies, 1976). The development of negation is often considered to be an intricate cognitive and linguistic process [START_REF] Choi | The semantic development of negation: acrosslinguistic study[END_REF] and has therefore been studied from various methodological perspectives. However, little work has tried to investigate how each paradigm of this complex process influences the other. We analyse the expression of negation in the utterances of one typically developing English monolingual child from the CoLaJE 2 Corpus 3 named Ellie. She was born in England on March 6 th , 2009; her grandmother has filmed her and her mother in their natural environment for one hour a month since the child was 0;10. Both the child's parents are college educated middle class adults.

We developed a specific multimodal coding system combining the use of Excel and CLAN with video data and transcriptions to analyse Ellie's negations according to context in dialogue from 0;10 to 4;02 We coded all the child's negative communicative acts and which modality of expression -spoken, vocal, gestural or actional -she used in sessions filmed monthly from 0;10 to 2;00 and biannually from 2;00 to 4;02. We analysed all negative functions and the context in which they were used. We categorised the functions following [START_REF] Choi | The semantic development of negation: acrosslinguistic study[END_REF] and Cameron-Faulkner and colleagues' (2007) typologies. We studied negative utterances produced by the child and her mother. The categories we used were the following: rejection, refusal, protest, prohibition, epistemic negation, incapacity, nonexistence/absence, negative assertion, denial.

Our coding system allowed us to conduct macro-and micro-analyses of the child's negative occurrences. From a large-scale perspective we divided the modalities of expression into two distinct categories that interact with each other: the first category refers to the channel of expression -either visual or auditory; the second category addresses the semiotic status of the preceding category. They correspond to whether the modalities are symbolic/linguistic, or non-symbolic/non-linguistic. Previous work on language acquisition has shown that parents often use a lot of scaffolding when interacting with their children [START_REF] Bruner | From communication to language: A psychological perspective[END_REF]Hodapp et al., 1984;[START_REF] Foster | The impact of scaffolding and overhearing on young children's use of the spatial terms between and middle[END_REF]. When parents verbally reformulate what children have just expressed by the means of gestures or actions, they encourage children to use words rather than actions or gestures to express themselves. Previous studies also observed that a transfer from the visual to the vocal channels as well as from non-symbolic to symbolic modalities occurs in children's communication system around 2;00 (see [START_REF] Morgenstern | From gesture to sign and from gesture to word: pointing in deaf and hearing children[END_REF]. This is why our coding system investigates the interface between channels of expression and their semiotic status in children's early acts of communication. Fig 1 In our coding, we make a distinction between actions and gestures but the difference between the two modalities is not always easy to draw when it comes to young children. We coded an action when the movement produced by the child is a reaction towards the environment and is not intentional and conventionalised as in the following example: In the above example, Ellie is 1;10 and she is engaged in a shared book reading activity with her grandmother (GDM). The grandmother suggests they count the number of horses on the page and the child's actions show her intention not to get involved in this activity. Her actions are negative and we coded them as a refusal and then as a protest. She uses her body to express her refusal to start the activity but her actions are not conventionalised by her cultural community as a systematic way of expressing a disagreement. Therefore, we coded these occurrences as actions and not as gestures.

In example (2), however, Ellie does not use actions to convey her negations; she uses the gestural modality with the headshake. Kendon has shown that this gesture is culturally understood as negative [START_REF] Kendon | Some uses of the headshake[END_REF] and for Spitz (1957) it is a ritualised form derived from the act of avoiding food. In the corpus, we considered that the child and the mother produced a gesture when the movement was 1) conscious and intentional, 2) culturally understood as expressing a negation. In example (1) the child's negations are unimodal because she only uses actions to negate, whereas in (2) she uses multimodal means of expression as well. In the first negative utterance combining < no + headshake > simultaneously, both the gesture and the spoken production are negative, thus we coded this multichannel utterance as a combination of modalities: speech and gesture. When the child uses several modalities but with only one that expresses negation, we coded which modality conveys the negative meaning and we added what accompanies the negative message in another category. We used this coding system when the child and the mother produce co-verbal gestures expressing another function than negation for instance.

Our work on children's language development therefore led us to adopt a multimodal approach in order to take into account all the dimensions of children's communicative system. We do not restrict the definition of language to the use of words. Words are one paradigm of a complex system involving other paradigms such as gestures, actions, facial expressions, gaze and intonation. During interactions between two or more participants, words are uttered in a specific context and are often accompanied with gestures, actions or facial expressions: they rarely stand alone.

Therefore, we consider that actions, gestures, words and vocal productions are modalities of expression. These modalities have two types of semiotic status (linguistic/symbolic vs. non-linguistic/non-symbolic) and function differently according to the channel they belong to (visual vs. auditory. If we focus on how children produce linguistic/symbolic expressions, we can observe that on the one hand the vocal/verbal modality and its syntax are constructed progressively as the number of words per utterance increase linearly with age. On the other hand, in the visual modality, gestures are not often strung together in a linear way (as opposed to what can be observed in signing children) but each of its components (configuration, movement, location) is endowed with more and more specific meaning.

Results

In the following part we will first present how the child conveys her negation and then we will focus on the emergence of the functions of negation.

Multimodality

Our analyses of the corpus show that the first spoken forms of negation Ellie produces appear at 1;01 when she expresses refusals and rejections. However, from 1;01 to 1;04 she rarely uses speech and mostly relies on actions or vocal productions with shouts and grunts. From 1;01 to 1;04 we coded 7 occurrences of the negative marker no. At 1;05 we observed a high increase in her use of the negative polarity item no with 20 occurrences in only one session. From 1;05 onwards, Ellie uses speech more often to express herself; in the session at 1;06 we coded 35 occurrences of the negative marker no, and after 1;08 Ellie starts using other markers such as gone and done, which are understood as negative polarity items in their context of use. We thus observed that the child enters the spoken modality for negation around 1;05 -1;06 and very soon diversifies the verbal means she uses. These results are in line with what has been presented in the literature.

However, when we examined gestures and actions before 1;06 we noted that negation is expressed well before this age. In the corpus, we coded 404 occurrences of spoken negation out of 646 negative communicative acts, which suggests that a significant number of negations are expressed by other modalities than speech used alone or in combination. At 10 months, we coded 59 negative communicative acts conveyed exclusively by non-symbolic modalitiesactions and vocal productions. The category we call "combination of modalities" in graphs 1 and 2 represents all sorts of combinations of modalities of expressions -actions, gestures, vocal productions and speech -and does not mark a distinction between symbolic/non-symbolic, linguistic/non-linguistic and visual/auditory. These results show that even at 0;10 Ellie is able to express negation with multimodal means (graph 1). She uses modalities that belong to two different channels of expression -visual with actions and vocal productions. Our analysis of the negative acts she conveys shows that she combines actions with vocal productions most of the time.

Graph 1. Modalities used to express negation throughout the corpus

Graph 1 presents the development of the expression of negation from 0;10 to 4;02. At 1;02 the child begins to use gestures and enters a symbolic mode of expression. It is however interesting to note that only three months after the child has entered the gestural modality and concurrently a symbolic mode of communication, she starts using speech (in black in the graph). There is a transition from a non-symbolic to a symbolic mode of expression in Ellie's production of negation from 1;00 to 1;05 that leads to the emergence of speech. Overall, the child uses multimodal means to express negation in 43% of the total amount of occurrences. Multimodal means are used throughout the corpus but the child's productions evolve and we were able to draw five main periods in her development of the expression of negative meaning in the corpus from 0;10 to 4;02 (graph 2). The first period extends from 0;10 to 1;01 when the child's expression of negation is non-symbolic. Period two is a transitional phase from a non-symbolic to a more symbolic mode of expression from 1;02 to 1;04. During period three, from 1;05 to 2;00, the child starts producing spoken negations. During period four -from 2;06 to 3;00 -we observe that the use of the spoken modality in contexts of negation has become predominant and Ellie's system of communication relies almost exclusively on a symbolic mode of expression. During period 5 -from 3;06 to the end of the corpus -she reintroduces gestures into her communicative system. Close to 60% of her negations are expressed solely with words and her negative system becomes very similar to the input (graph 2). Graph 2. The five periods in the child's expression of negation and the expression of negation in the input.

The mother's semiotic means to express negation do not vary from one session to the other in the corpus, thus we represented her productions over the entire span of the data in one single column in graph 2. Results show that in the input, negation is predominantly expressed with the spoken modality without combining it to any other modality (gestures, actions or vocal productions). From 0;10 to 4;02, we coded 343 negative communicative acts in the input (mother) among which 70% are conveyed by speech used in isolation and all the combinations are with speech. Among the 30% remaining utterances, we categorised 31 occurrences of speech combined with actions, 7 occurrences of speech combined with vocal productions like shouting, 46 occurrences of speech with gestures and 1 occurrence of a combination of speech with gestures and actions. Only 2% of the mother's utterances are expressed by the visual channel alone (actions or gesture) but overall, the expression of negation in the input is symbolic and mostly spoken. The graph for the expression of negation in the input (graph 2) is very similar to the fifth column presenting the distribution in the child's productions during the last period of the data. We observe that around 4;00, the child's use of speech has become predominant and at the end of the corpus, she is already heading towards a system of expression similar to her mother's.

Graph 3 shows the average number of words the child produces in contexts of negation in each session. We observe that her spoken productions for negation are constantly getting richer. At 4;02 the child has an MLU (mean length of utterance) of 3,045, which means that, in average, during one hour, she produces three-word-utterances to express negation. Therefore, her utterances are longer and also more complex than during period 2 when she produces only one-word utterances in contexts of negation. Thus, in only four years, the child acquires her mother tongue and manages to express negation in a way that is comparable to her mother's.

Diversification of the functions of negation

At the beginning of the corpus at 0;10, Ellie already produces a significant number of refusals, rejections and protests. The emergence of the other negative functions begins later, around 1;02 when the child starts using gestures in contexts of negation and around 1;05 when she enters the spoken modality (graph 2). During period 2, she starts expressing epistemic negation, denial and negative assertion and graph 4 shows that during period 3 she produces all the negative functions present in the input, except for prohibition (which is pragmatically coherent since a child is not frequently in a position to use prohibition when interacting with her mother). We observe a strong correlation between the emergence of the functions of negation (graph 4) and the semiotic status of the expression the child uses during period 3 (graph 2). Indeed, all the negative functions emerge in Ellie's corpus when she enters a symbolic mode of expression with the use of gestures and speech. Graphs 3 and 4 highlight the fact that the enrichment of the child's speech coincides with the development of the functions of negation in her productions. The case of negative assertion is revealing. Ellie starts producing her first negative assertions during period 2 and at 1;03, she simply uses the word no to express this function. Among all her productions of negative assertions from 1;02 to 4;02, 87% of the occurrences are expressed with speech used alone or coupled with another modality. Therefore, negative assertion is a function of negation linked to the spoken modality. As the child's speech grows syntactically and semantically more complex (graph 3), we observe that the proportion of negative assertions in her productions increases. Concurrently, the proportion of refusals, rejections and protests (in black in the graph) gradually decreases after 1;02.

Graph 4. Ellie's multimodal productions of negative functions in the corpus

Graph 5. Multimodal productions of negative functions in the input (the mother).

In the input (graph 5) the production of the functions of negation differs from one period to the other and this variation may be linked to several factors such as the context of use and the age of the child. Just as we noted in the child's use of negative functions, the mother's production of refusals, rejections and protests (in black) decreases throughout the corpus. Negative assertion and denial are the negative functions the mother predominantly expresses and it appears that the child's productions of negative functions are moving towards the same distribution as she gets older.

These preliminary remarks confirm that negation is a relevant topic to study multimodality in children's data. Moreover, as graph 1 illustrates, the child uses negation in every session and therefore provides us with a large amount of productions. It therefore also highlights the fact that negation is very often used by children before 4;00.

In the following sections, we will illustrate how the transfer from nonsymbolic to symbolic modalities occurs and materialises in the child's communicative system. We will also analyse the impact this transfer has on the development of her language skills.

Analyses of Ellie's negative productions in five distinctive periods

4.1. Period one 0;10 -1;1: a non-symbolic system of expression During the first period the child exclusively relies on non-symbolic expressions of negation. 99% of the occurrences are expressed by action with 51% conveyed with actions alone and 48% with actions coupled with vocal productions. During this period, the child's system of expression is based on Acts of Bodily Communication [START_REF] Zlatev | Stages and transitions in children's semiotic development[END_REF] to avoid objects. The following example illustrates how the child manages to convey her negation when she is not yet able to speak. In this sequence, Ellie is 0;10 and she is having lunch but she is not hungry and refuses to eat the rest of her food. Example (3) shows that the child relies on actions to indicate that she is not hungry. The mother offers the child food twice and the child responds by an action twice. We also observe that already at 0;10 Ellie can be considered as a full participant in the interaction and can communicate her intentions as she deliberately avoids looking at her mother to signal that she does not want to eat her food. As graphs 1 and 2 illustrate, period 1 is governed by non-symbolic modalities but the child's communicative system undergoes a crucial change in period 2 as she starts to use gestures alone and to combine words with actions and gestures. 4.2. Period two 1;02 -1;04: the transition from a non-symbolic to a symbolic system of expression During period two, the child investigates the gestural modality. We observe continuity between the first and the second period as the child enters a symbolic mode of expression inside the visual modality. The transition from actions to gestures is rendered possible because the child is becoming more careful and has a greater mastery of her fine motor skills at 1;02 than at 0;10. In the example below, Ellie is walking in front of her mother. She is heading for the kitchen and the mother asks her if she wants to have some milk. The child refuses but very differently from what we described in example (3). Indeed, the child is more careful and controls her actions. We observe that her action is more deliberate and restrained and we expect this action to be soon ritualised and transformed into a gesture. The mother interprets that movement as a negative construction and takes it up with the word "no" in her verbal reaction to her daughter's production.

Example (4) -Ellie, 1;2 *MOT : do you want some milk ? %act :

MOT brings a cup of milk towards CHI . *CHI : 0 . %act :

CHI looks up at the glass of milk and with her right hand, (Open Hand Prone gesture9 ) she carefully and deliberately pushes the glass of milk away from her . *MOT : no (.) okay .

Between 1;02 and 1;04, Ellie starts using gestures to express her negations and she adds epistemic negation to the negative functions she can express. All the epistemic negations she produces are conveyed by gestures; these gestures are either a headshake or an Open Hand Supine gesture (see fig 2 .), also called Palm-up gesture (Kendon, 2004:264). Example (5) illustrates the way she uses the gestural modality to convey her epistemic negations. In the corpus, we coded occurrences of palm-up gestures (Kendon, 2004:264) produced alone and in combination with vocal productions as in the following example. In example (5), the child indicates that she does not know where the cat is with a palm-up gesture coupled with a vocal production. The word she utters is not recognizable and the element that is understood as the negative item in her utterance is the gesture. The gesture in this sequence carries the full meaning of the negation.

Period two is quite short but it is nonetheless crucial, as gestures are very closely followed by the emergence of speech and of new negative functions two months later. The use of gestures from 1;02 to 1;04 marks the child's entry into a symbolic communicative system. 4.3. Period three: 1;05 -2;00, emergence of the spoken modality.

During period 3 the child's use of the gestural and spoken modalities becomes more significant. She produces gestures more often than during period 2. During period 1 the child uses no gesture, during period 2 we coded 17 productions expressed by gestures used alone and in combinations and during period 3 we found 83 productions expressed by gestures. She also starts to use words in contexts of negation as well as negative polarity items like the adverb no. During the third period, the child has a MLU of 0.977 (with a standard deviation at 0.82), which means that she produces mostly single-word utterances in that context. However, half of single word productions are combined with visual modality, in the shape of a gesture or an action. Graph 2 shows that from 1;05 to 2;00 the child relies on multimodal simultaneous means 48% of the time when she conveys her negations and only 20% of all her utterances are conveyed by words used in isolation. Example (6) shows how she combines several modalities to express negative meaning. Example ( 6) highlights that at 1;05 the child starts combining single-word utterances and gestures to convey her negations but she still relies on actions from time to time. Graph 2 shows that 21% of her negations are still conveyed through actions only during period 3. These observations illustrate that during the third period, the child's speech is still in construction. In order to express her negations, she needs to use several modalities. Among the utterances expressed with multimodal means, the child uses speech 87% of the time; she uses a combination of actions and speech in 43% of the total amount of utterances and a combination of speech and gestures in 39% of the cases. The child therefore uses multimodal means during the first three periods to express negation. We observe that at 1;05 she is heading towards a symbolic system of expression even though she still uses actions in combination quite significantly. Period 3 is a transitional phase from a child-like expression of negation relying on bodily actions and gestures to a more adult-like expression almost exclusively based on speech, as shown in graph 2.

Period 3 is also crucial as new negative functions emerge in the child's expression of negation. As graph 4 illustrates, during this period, she begins to express incapacity, absence and nonexistence, denial and negative assertion and 53% of these negative functions are expressed multimodaly. Among these 53%, 94% are combinations with gestures. As we hypothesised during period 2, the gestural modality appears as a facilitator in the acquisition of speech and seems to lead to the development of the spoken modality. Indeed, we looked at the functions Ellie expressed with manual gestures and we observed that during period 2, she always expresses epistemic negation with an Open Hand Supine gesture (Palm up gestures), very often used without combining it to speech. During period 3, all her occurrences of epistemic negation and absence or nonexistence are conveyed by an Open Hand Supine gesture very often combined with the word gone. During period 3, Ellie starts using Open Hand Prone (Palm down gestures) [START_REF] Kendon | Gesture: Visible Action as Utterance[END_REF]:248, Boutet & Morgenstern, 2012) but she restricts their use to the functions of refusal and protest as in example (4). During periods 2 and 3, gestures can carry the whole semantic meaning of an utterance, as was the case in example ( 5), but simultaneously they help the child construct her speech and enter the semantic, pragmatic and syntactic complexities of language.

Fourth period 2;06 -3;00: the development of the spoken modality

Our study of the use of modalities during periods 2 and 3 showed that the child increasingly uses gestures but during period 4 we observe that her production of gestures significantly decreases (graphs 1 and 2). Ellie uses speech in isolation in 52% of the utterances she produces whereas gestures and actions used alone have been reduced to 2% of all utterances each (graph 2). She still combines modalities but in a lower proportion than during the previous period and graph 2 shows that the spoken modality increases each month. At 3;00, Ellie has fully entered symbolic communication; she still uses actions from time to time but in a non-significant proportion compared to her production of words. In the following example the child is 3;00 and she is playing with a game of lego (Duplo) with her mother. This type of game requires that the child move and play with the various pieces and thus, involves the use of her body. Yet, interestingly, the child uses her body actions to play but answers her mother with words and gestures. Her speech, associated to one gesture, is enough to convey her negations. Example (7) can be compared to example (6) in which the child prevents her mother from touching her cards; but we observe a difference in her use of the various modalities. At 3;00 Ellie exclusively relies on symbolic modalities and has stopped using actions alone to act on a tier, as was the case at 1;05. In example (7) she is talking about her cat, Pepper, and later she speaks to her because she does not want the cat to mess with her toys. Ellie scolds the cat and the cat leaves the room. Example ( 7 would it be very funny though ! Example (7) illustrates that during period four the child's speech is more complex than during period three. Graph 3 also shows that at 3;00, her MLU in negative contexts has increased. At the end of the fourth period, the child has a MLU of 2.784. She is able to produce long and complex sentences, as when she says no, Pepper, you mustn't move my toys! in which she utters seven different words, which proves that her speech has considerably developed from 2;00 to 3;00. The sequence presented in example 5 also highlights the child's mastery of the symbolic modalities in interactional contexts. At 3;00, the child understands the notion of turn-taking in dialogue. Indeed, when the mother says because you didn't want her to play with your toys!, Ellie produces several headshake and does not interrupt her mother. She uses a gesture and therefore resorts to the visual channel while the mother uses the auditory channel. This strategy enables the child to express herself and to show that she agrees with her mother without interrupting her. Her ability to switch from the auditory to the visual channel allows her to secure the interaction and to construct several symbolic layers of interactions.

During period 4, Ellie does not introduce new negative functions into her communicative system but she consolidates all the others. She becomes more expert at expressing epistemic negation, incapacity, refusal, absence and nonexistence, denial and negative assertion in more complex situations, because her communicative system relies on symbolic modalities and most specifically on speech, sometimes combined with gestures. 4.5. Fifth period 3;06 -4;02: the emergence of a fully integrated symbolic communicative system.

Graph 3 shows that when the child enters the fifth period of her communicative system for negation, her speech is quite elaborate. During period five, a crucial change happens in her use of the gestural modality. Graph 2 illustrates that after 3;06, Ellie's use of the spoken modality continues to increase but we also observe that she reintroduces gestures alone and in combination. Among the combinations of modalities presented in graph 2 for the fifth period, 68% are with gestures. The child's use of gestures understood as negative as in example ( 7) intensifies during period five. Simultaneously, a new phenomenon emerges. Indeed, after 3;06, Ellie starts using gestures that are not negative and she combines them to her negative spoken productions to enrich her utterances. These combinations are not represented in graph 3 because her co-verbal gestures are not negative. As graph 3 illustrates, when Ellie enters period five, she masters a large range of negative polarity items to convey her negations. She no longer needs to combine her spoken utterances to gestures to express her negations because her speech is rich enough. However, the reintroduction of the gestural modality either as negative gestures or co-verbal gestures shows that her acquisition of negation has reached a new threshold. From 3;06 onwards, Ellie uses gestures to convey more complex pragmatic meaning than in the previous periods.

The role of gestures in the blossoming of symbolic multimodal interactions

Our study of the development of the expression of negation in a longitudinal corpus shows that an interesting transition from nonlinguistic/symbolic modalities to linguistic/symbolic modalities occurs after 1;06, when the child starts using words. Ellie enters negation by using actions. She then adds gestures to those actions as if they could create a bridge from nonsymbolic modalities to symbolic modalities and facilitate the emergence of speech. We divided Ellie's pathway into five main periods. During the first two periods, Ellie uses the visual channel with actions and gestures more than the auditory channel with speech, however screams are present in vocal-gestural combinations. During period three -from 1;05 to 2;00 -the use of actions in isolation decreases and simultaneously, the child progressively uses words. During period 4, she does not stop using actions but in proportion, they are less predominant than during the first period. She still uses actions in the third and fourth periods but almost only in combination with either speech or gestures. During the fourth period, we noted that gestures are less frequently used than in the two previous periods and at the same time we observed that the child's speech becomes richer. She relies on symbolic modalities only and her speech becomes more complex. During the fifth period -from 3;06 to 4;02 -the child enters a new phase in which the symbolic modalities (auditory and visual) combine to enrich her language.

Overall, the child expresses negation at first with actions and vocal productions, then she introduces symbolic gestures used alone or in combination with vocal productions and after 2;00 she uses words. After 2;06 her speech is stable and she relies almost exclusively on symbolic modalities. At the end of our corpus, most negations are expressed with speech used without combining it to other modalities. Just before 4;00, 60% of the child's negative productions are expressed by speech alone but graph 1 shows that the child reintroduces the gestural modality after 3;06. Indeed, after 3;06, Ellie's speech is more fluid and she easily expresses herself with words. After 4;00 she starts combining words with gestures but in a different way from the multimodal combinations she produced before 2;00. Gestures do not carry the whole negative meaning of her utterances; they accompany her speech and are fully integrated to the message she expresses. The gestures she uses are not necessarily negative: they are coverbal gestures with other semantic and pragmatic functions that accompany her negative speech (see example 8). After 3;00, the context is less important to understand the child's utterances and interlocutors who are not familiar with the child can understand her. After 3;06, the gestures she uses have a different function from the ones she produced at 2;06. They are not redundant with speech or not used because of a lack of vocabulary. Rather, they are used to enrich the verbal utterance of the child. From 3;06 to 4;02 we coded 10 negative spoken utterances accompanied with co-verbal gestures, which represent 8% of Ellie's negative productions during this period. This number may seem very low in comparison with the total number of negative spoken productions but the child's use of co-verbal gestures only emerges during period 5 and is therefore not completely established and not yet part of the child's communicative system. In most cases, the child produces a pointing gesture or a nod. Previous studies on co-verbal gestures have made attempts at classifying gestures used with speech and three main categories have been proposed: equivalent or redundant, complementary and supplementary. A gesture is equivalent to speech when it has the same semantic role as the spoken production; the referent for the gesture and the spoken production is the same in the utterance. Goldin-Meadow and Morford (1990:252) explained that a gesture and a spoken production are complementary when they both refer to the same referent but when they have different semantic roles. [START_REF] Olga | Gestures and words during the transition to two-word speech[END_REF] showed that gestures can be supplementary to words when the gesture and the spoken production have different meanings. After 3;06, Ellie starts combining gestures to her negative spoken utterances but the gestures are never "redundant"; half are "complementary" and the others are "supplementary" according to this categorisation. They add a semantic layer to her utterance (see example 8).

Our analyses of the development of negation in Ellie's corpus show that the child uses multimodality differently according to her age and linguistic development. Before 2;00 the three predominant modalities -actions, gestures and words -have the same pragmatic function since the child only wants to convey her global meaning and to make herself understood. During the language acquisition process, negative speech progressively incorporates the functions held by negative actions and gestures at the beginning of our data. The gestural modality seems to have a ratchet effect on non-symbolic modalities and therefore prevents the child from returning to a non-symbolic/non-linguistic mode of expression (action) once her communicative system is symbolic/linguistic. Simultaneously, the spoken modality may have the same effect on the gestural modality since after 3;06 gestures are kinaesthetically, semantically and pragmatically more complex than before 2;00. These ratchet effects may explain the succession of modalities observed for the expression of negation in Ellie's corpus and the transition we found in her communicative system from a non-symbolic to a symbolic semiotic status. This effect prevents the child from "regressing" to the previous non-symbolic mode of expression and therefore provides her with a stable basis for each channel. This new basis allows her to develop her communicative system until she reaches a new transitory phase with a new ratchet effect. The phenomenon is divided into two cumulative periods: 1) the expression of negation extends to the two channels; 2) each channel, one after the other, reaches a new semiotic status.

As a consequence, once the child has mastered the complexity of the adult speech system she starts using co-verbal gestures with her negative productions and becomes a skilful multimodal conversationalist. The following example illustrates the complexity of the child's communicative system at 3;06. The transcription shows that the child first combines a negative utterance with a gesture that can be classified as a Palm-up gesture [START_REF] Kendon | Gesture: Visible Action as Utterance[END_REF], which is usually produced in negative contexts. At 3;06 Ellie still uses gestures she introduced in her communicative system when she was 1;04. We can observe that she also uses gestures that are not necessarily negative as she produces negative utterances. At the end of this sequence the child shrugs and shakes her head but interestingly, the gestures are not produced simultaneously to the negative verbal markers. They are simultaneous to the word "car" as if the gestures were isolating the word "car" and focusing on the idea that the people are coming but not in their car. The gestures are contrastive here and are not merely redundant with speech. Therefore, at the end of the corpus the child combines speech with gestures and produces complex multifunctional and multimodal utterances. That was not the case during period 2, 3 and 4 when the gestures were equivalent to her spoken productions. At the end of period 5, the child has acquired the ability to make efficient use of gestures interactively and with more precision. The finer use of the direction of her body and her movements, along with the orientation of her palms -upwards or downwardsand their location in relation the child herself, her interlocutor and the situation, are subtle parameters that allow the child to convey complex semantic or pragmatic meaning. To a certain extent, the ability to use her body appropriately and efficiently to produce complex gestures could be compared to complex syntax in the realm of speech. The main difference is that for speech we observe a linear organisation of the items whereas for gestures we note a multilinear syntactic, semantic and pragmatic organisation of the items.

Conclusion

The analysis of Ellie's pathway illustrates how she first relies on visual modalities with actions and gestures. During that first period, actions and gestures are used instead of words and convey the whole semantic meaning of her utterances. Then she enters a multimodal period during which she combines several modalities to express her negations but gestures are generally redundant with words or actions. From 2;00 to 3;00 her communicative system is reduced to a mostly unimodal mode of expression during which she learns to master speech with a longer MLU and a diversification of the production of negative functions. After 3;06, when her speech is elaborate enough to convey all her negations she reintroduces the visual modality with supplementary co-verbal gestures. During this fifth period, gestures do not replace words and are not redundant; they add communicative and semantic meaning to her speech. Overall, there are two phases when she relies on multimodality. Before 2;06 the child's use of multimodality is rather mono-functional since all the modalities serve the same end, that is to convey one single negative meaning and function. After 3;06 the child's use of multimodality is multi-functional since each modality is used on a different semantic or pragmatic layer. Before 2;06 the child's negative multimodal utterances are simple and monofunctional whereas after 3;06 they are complex and multifunctional.

The present paper shows that gestures used prelinguistically are qualitatively different from gestures used once speech is already quite elaborate. Not only are they more complex in terms of articulatory phases, but from a pragmatic perspective, they are also richer and multifunctional. Our study also demonstrates that there are different uses of multimodality and therefore, the status of multimodality changes throughout the corpus.

In the case of this child, a multimodal approach to the acquisition of negation has proven that negative meaning is expressed long before she enters the spoken modality. It also adduces evidence that early use of gestures seem to pave the way to speech and the development of a vast set of negative functions, and that co-speech gestures add semantic and pragmatic layers to speech.

This study seems to indicate that children may understand the notion of polarity quite early on and that they start to produce simple negative unimodal and multimodal constructions at a very young age. We observed that as the child gets older, her negations become syntactically, semantically and pragmatically more complex and diversified. Ellie expresses a larger range of negative functions from 2;00 onwards. Our study therefore illustrates how the child's cognitive and fine motor development is key to the development of complex multimodal negative utterances in more elaborate interactions in which the child has become a skilful multimodal conversationalist.

Fig 1 .

 1 Fig 1. Interaction between the channels and the semiotic status of the modalities of expression.

  10-1;01] [1;02-1;04] [1;05-2;00] [2;06-3;00] [3;06-4;02] Mother vocal prod. words gestures actions combinations Graph 3. Ellie's Mean Length of Utterance in negative productions from 0;10 to 4;02.

Example

Fig 2 .

 2 Fig 2. Representation of a Palm-up Gesture, Open Hand Supine Gesture(Kendon, 2004:280)

shows the interaction between those two parameters.

	3	The	recordings	and	transcriptions	can	be	downloaded	from
	http://colaje.risc.cnrs.fr/index.php/corpus/corpus-colaje.

CHI: 0. %act: CHI touches the book and tries to take it .

	Example (1) -Ellie, 1;10
	*MOT 4 :	how many are there Ellie ?
	*GDM:	shall we count ?
	*CHI:	0 . 5
	%act:	CHI moves forward and looks at the book GDM is holding .
	*GDM:	one +…
	**GDM:	two (.) oh@i no !
	*CHI:	0 .
	%act:	CHI

takes the book in her hands and closes it .

	Channel of expression
	Visual Auditory	ACTIONS VOCAL PRODUCTIONS	GESTURES WORDS	Modalities
				Semiotic status
				of the
		Non-Symbolic/	Symbolic/	expression
		non-linguistic	linguistic

bowl of porridge *CHI: 0 . %gpx: headshake .

	Example (2) -Ellie, 2;00
	*MOT:	telephone !
	%sit:	the telephone is ringing .
	*CHI:	telephone !
	%act:	CHI looks at her aunt Marianne .
	*MAR 6 :	it's the telephone !
	*LAU:	+< 7 telephone oh@i yeah !
	*MAR:	0 .
	%gpx:	points to the child's bowl of porridge
	*CHI:	no .
	%gpx:	headshake .
	*MAR:	shall we eat the porridge ?
	*CHI:	xxx 8 .
	*MAR:	some of your porridge .
	%xpnt:	points to the child's

headshake no . %act : CHI grabs a card and puts it behind her and away from MOT .

	Example (6) -Ellie, 1;05
	*MOT :	are you doing some tidying now then ?
	%sit :	MOT and CHI are playing cards .
	*CHI :	no .
	%gpx :
	*MOT :	0 .
	%act :	MOT tries to take the card from CHI's hands .
	*OBS :	no ?
	*CHI :	0 .
	%act :	CHI firmly holds the card and prevents MOT from taking it
		away from her .
	*MOT :	no ?
	%act :	MOT puts her hand on another card laying on the floor .
	*CHI :	0 .
	%act :	CHI reaches MOT'

s hand with the card and pushes it away, then she grabs the card .

f=name of the cat, you mustn't move my toys ! %gpx : nod on « toys » %act : CHI stops playing with her toys and looks in the cat's direction .

		) -Ellie, 3;00
	*MOT :	if you're not gonna let me play then I'm not gonna play .
	%gpx :	headshake on not gonna .
	%com :	MOT is prentending to be the cat .
	*CHI :	why I'm not letting her play ?
	*MOT :	because you didn't want her to play with your toys !
	*CHI :	+< 0 .
	%gpx :	several headshakes .
	*MOT :	is she a pickle monster ?
	%sit :	CHI plays with her toys .
	*CHI : no Pepper@*OBS : what would you say if Pepper said woof@o ?
	*MOT :	0 [=!laughs] .
	*CHI :	+< 0 [=!laughs] .
	*CHI :	that would be naughty !
	%gpx :	shrugs and tilts her head on the right side [=!laughs] .
	*OBS :	0 [=!laughs] .
	*MOT :

shakes her head and shrugs at

	Example (8) -Ellie, 3;6
	*MOT:	come back .
	*CHI:	big massive box .
	%gpx:	iconic gesture: CHI opens her arms to show how big the box was .
	*MOT:	big massive box (.) big enough for y(ou) [//] me to get in ?
	*CHI:	+< they don't +..
	%gpx:	abduction of her arms and extension of the hands and shrugs
	*CHI:	0 .
	%gpx:	CHI nods
	*OBS:	and so what did you do +..
	*CHI:	<why are they> [//] they're <not c(oming)> they're <going to> [//]
		they're not coming in their car .
	%gpx:	CHI brings her hands before her,

the same time as she says "car" .

To refer to the child's age we adopt the conventions used in language acquisition : X;Y where X is the number of years and Y the number of months (1;07 means the child is one year and seven months).

Projet ANR CoLaJE (http://colaje.scicog.fr), a project funded by the French National Agency, see[START_REF] Morgenstern | The Paris Corpus[END_REF]

The CHAT[START_REF] Mcwhinney | The CHILDES Project: Tools for analyzing talk[END_REF]; http://childes.psy.cmu.edu) transcription system includes main tiers indicated by * and a three letter name for the speaker (MOT : mother ; CHI : child and GDM : grand-mother) and secondary tiers indicated by %act describing the actions).

0 . is a convention from the CHAT format that indicates that the participant does not utter a word but makes an action or a gesture that is coded in the following lines. Actions are signaled by %act and gestures by %gpx.

MAR is MARIANNE and LAU is LAURA, Ellie's aunts.

This convention is used to signal that two participants are speaking or making an action at the same time. It is a marker for overlapping utterances.

This convention is used whenever the coder cannot understand what a participant says.

Kendon, 2004:251, also called VP gesture for Vertical Palm gesture.

This convention is used whenever the coder cannot understand what a participant says but can provide a phonetic transcription.

We use the secondary tier %pho to indicate phonetic transcriptions of the child's productions.