Antoine Lima

Anthony Welte

Philippe Bonnifait

Philippe Xu

LiDAR Observations by Motion Compensation and Scan Accumulation

come

LiDAR Observations by Motion Compensation and Scan Accumulation

Antoine Lima*, Anthony Welte*, Philippe Bonnifait* and Philippe Xu* Abstract-In dynamic localization problems, the observations used from exteroceptive sensors are usually obtained from a single measurement. However, there are cases where the current measurement is not sufficient to detect the referenced landmark or to get a sufficient level of accuracy. In this study, a point cloud accumulation strategy is used to improve the resolution of a LiDAR sensor along its sparse axis. In particular, we are interested in the detection of markings transverse to the road axis in order to improve the accuracy of localization of an autonomous vehicle when approaching intersections or roundabouts. We present a method that allows the construction of an accurate observation with an associated observation model based on a High-Definition (HD) map through an accumulation of scans as the vehicle moves, by compensating the vehicle motion. The parameters of the accumulator are studied in terms of detection and accuracy. The quality of the observations and their impact on the localization quality are analyzed using real experiments carried out with an experimental vehicle equipped with a low-cost GNSS receiver, dead-reckoning sensors and a ground truth system.

I. INTRODUCTION

Accurate localization is a prerequisite in many navigation tasks of autonomous vehicles ranging from vehicle control to path following. A Global Navigation Satellite System (GNSS) receiver coupled with Dead-Reckoning (DR) is often used as a basis for the localization scheme. Unfortunately, GNSS signals can be degraded by non line-of-sight and multi-path in particular in urban environments, and a DR pose estimate drifts over time. A solution is to embed perception sensors such as cameras, RADARs or LiDARs to detect visual features in the surrounding environment and to use them to localize the vehicle. These features are usually either mapped in a previously acquired map or localized and mapped simultaneously, i.e., SLAM-based approaches.

Our approach falls in the former category, in particular, we make use of a High-Definition (HD) map containing accurate geo-referenced locations of road markings. Because the reflectivity of road markings is higher than the rest of the road, as illustrated in Figure 1b, a LiDAR sensor can extract such features using the intensity of reflected laser beams. Simple shapes such as lane markings are usually detected by single point cloud methods and used as observations to improve the lateral localization but, more complex markings, in particular in the longitudinal direction, accumulation of scans is required since a single scan does not contain enough information. Our proposal aims at improving the longitudinal resolution of LiDAR data by scan accumulation while compensating the motion of the vehicle during the Université de Technologie de Compiègne, CNRS UMR 7253 Heudiasyc, France. scanning thanks to its odometry. To study the accuracy of this approach, we focus on pedestrian crossings, as their shape does not allow for single frame detection while multi-frame provides longitudinal information, as illustrated in Figure 1a. Note that throughout this paper, only zebra crossings, which are designed with several strips parallel to the lane, are considered, even though detection of other types of marking such as stop or yield lines, or even road arrows could also be helped by this approach. The detector applies ground and markings filtering and incorporates a coarse pre-filtering around crosswalk areas that makes use of the HD map. Strips are then detected and used to compute the crosswalk pose. Detections are integrated in a localization scheme by an observation model in which measurements constrain the position along the crosswalks axis, i.e., in the longitudinal direction of the road.

The main contributions of this paper are:

• An accumulation method to get an accurate observation in the sparse direction of the LiDAR sensor; • A detector and a model associated to this observation;

• A discussion on strategies to integrate such temporally correlated observations in a localization scheme.

The paper is organized as follows. In Section II, a state of the art of localization using visual cues from LiDARs is exposed. Then in Section III, the crosswalk detection process is explained. The localization scheme and crosswalk observation model are presented in Section IV and then experimental results and parameter tuning are given in Section V.

II. STATE OF THE ART

The field of localization using visual features is widely studied but remains an open issue. Two sensors are usually used as sources of information for visual cues: cameras and LiDARs. Localization using LiDARs can be separated into two families: dense and sparse. In dense methods, there is no feature detection and measurement are registered against a dense representation. Levinson and Thrun [START_REF] Levinson | Robust vehicle localization in urban environments using probabilistic maps[END_REF] used a grid to store intensities as Gaussian distributions and formulated the registration problem as finding a transformation between the measured point cloud and the grid in the estimated body frame. An extension of this idea was introduced in [START_REF] Wolcott | Robust LIDAR localization using multiresolution Gaussian mixture maps for autonomous driving[END_REF] by storing both intensity and height as Gaussian mixtures then using a branch-and-bound registration algorithm. More generally, dense methods store a 3D map previously built by a LiDAR and formulate the localization problem as a registration one in that map using algorithms such as Iterative Closest Point (ICP) [START_REF] Segal | Generalized-ICP[END_REF], Normal Distributions Transform (NDT) [START_REF] Magnusson | The Three-Dimensional Normal-Distributions Transform -an Efficient Representation for Registration, Surface Analysis, and Loop Detection[END_REF] or derivatives.

Sparse approaches, on the contrary, extract features from measurements and match them to a prior map to refine a global pose given by a GNSS receiver, for example. Feature detection on LiDAR can once again be grouped into two categories: 2D grid-based or 3D. In grid-based approaches, the point cloud is projected onto a 2D grid placed on the ground. These grids are assimilated to bird-eye-view images and interpolation such as Inverse Distance Weighting is used to fill in gaps between impacts. Castorena and Agarwal [START_REF] Castorena | Ground-Edge-Based LIDAR Localization Without a Reflectivity Calibration for Autonomous Driving[END_REF] used the intensity gradient to extract markings and register them against a prior map. In [START_REF] Yao | Automatic extraction of road markings from mobile laser-point cloud using intensity data[END_REF], markings are extracted by computing an integral image ( [START_REF] Bradley | Adaptive Thresholding using the Integral Image[END_REF]) and clustered. They are then detected either by testing their size for linear markings, or by matching patterns for more complex markings. Yu et al. [START_REF] Yu | Learning Hierarchical Features for Automated Extraction of Road Markings From 3-D Mobile LiDAR Point Clouds[END_REF] extracted markings using an extension of the Otsu dynamic thresholding [START_REF] Otsu | A Threshold Selection Method from Gray-Level Histograms[END_REF] incorporating the distance to the sensor and the point density. Markings are then clustered and refined before being classified using a combination of deep learning for small markings and Principal Component Analysis (PCA) for rectangular markings. Finally, in [START_REF] Ghallabi | LIDAR-Based Lane Marking Detection For Vehicle Positioning in an HD Map[END_REF], Hough transforms are applied to the grid to detect alongtrack and cross-track features that are then tracked using a Kalman filter. The use of grids was introduced to reduce the number of points and simplifying the problem from a 3D to 2D one, but this has the disadvantage of losing accuracy depending on the desired cell size.

Finally, in 3D-based detection approaches, the scanning pattern can be used to detect changes such as curbs. In [START_REF] Hata | Feature Detection for Vehicle Localization in Urban Environments Using a Multilayer LIDAR[END_REF], Hata and Wolf detected curbs using ring compression robust to obstacles and extracted markings using a modified robust version of Otsu. An occupancy grid is then used to integrate these detections to a Monte Carlo localization. In [START_REF] Wang | Intelligent vehicle self-localization based on double-layer features and multilayer LIDAR[END_REF], a collision-based curb detection is introduced and detections are accumulated using an Inertial Measurement Unit (IMU) before being refined by a two-pass beam model. Non-SLAM point cloud accumulation methods have been explored in studies such as [START_REF] Tatebe | Can we detect pedestrians using low-resolution lidar? integration of multi-frame point-clouds[END_REF], where the combination of successive frames as well as the change between them has been used to detect pedestrians.

Most of these methods use the intensity measure, but its sensitivity is a common LiDAR issues. Out of the factory, LiDARs are generally not sufficiently well calibrated and intensities depend on the beam which did the measurement. The intensity can also vary with distance, as the incident angle diminishes. Those issues can be tackled by offline calibrations such as the ones proposed by Levinson and Thrun in [START_REF] Levinson | Robust vehicle localization in urban environments using probabilistic maps[END_REF], [START_REF] Levinson | Unsupervised Calibration for Multi-beam Lasers[END_REF]. However, the major issue appears in bad weather. While rain and snow can produce light noise in the geometrical measurements, it can greatly degrade the intensity readings, in particular on pavement, as studied in [START_REF] Kutila | Automotive LiDAR performance verification in fog and rain[END_REF].

III. CROSSWALK DETECTOR A. Problem Statement

Detection of complex markings on the ground requires good along-road measurement accuracy and resolution in order to accurately measure where the markings begins and stops. This can be an issue with low density LiDARs or certain sensor placements. For example, despite their good measurement accuracy and along-ring resolution, LiDARs, when placed on the roof parallel to the ground, can have a ring resolution up to several meters when hitting the ground because of their construction using stacks of lasers, as can be observed in Figure 1a. In this section, a crosswalk detector capable of computing observables on crosswalks from LiDAR scans thanks to DR information and buffers is detailed. We chose not to use grids nor machine learning in order to build an accurate observation model for the localization filter.

B. Pre-Processing

Typical scans being composed of tens of thousands of points, a buffer can have to hold millions of points, making it hard to guarantee real-time processing with any subsequent algorithm. We apply ground filtering, based on the ground plane fitting algorithm from [START_REF] Zermas | Fast segmentation of 3D point clouds: A paradigm on LiDAR data for autonomous vehicle applications[END_REF], and coarse road filtering that uses a prior HD map and a localization estimate to remove points outside the road closed polygon. For this, an area-based test is realized for each point, where sums of triangle areas are compared to that of the polygon. This significantly reduces the number of points without too much information loss, contrary to ray-casting approaches, polygon enlargement [START_REF] Frisch | High integrity lane level localization using multiple lane markings detection and horizontal protection levels[END_REF] or others detailed in [START_REF] Haines | Point in Polygon Strategies[END_REF]. As our approach uses ground points in the close vicinity, we only focus on lower rings. We have noticed that the factory intensity calibration was sufficient for this use case. Therefore, we have not used a calibration method such as described in [START_REF] Levinson | Unsupervised Calibration for Multi-beam Lasers[END_REF].

C. Buffering

The classical solution to accumulate or "buffer" multiple point clouds is to transform the point clouds in a static working frame and taking their union. The issue with this method is that bad transformations (i.e., imprecise localization) lead to incoherent point cloud. To dampen this issue, grids have been developed [START_REF] Levinson | Robust vehicle localization in urban environments using probabilistic maps[END_REF], [START_REF] Hata | Feature Detection for Vehicle Localization in Urban Environments Using a Multilayer LIDAR[END_REF] and work by projecting points in a 2D grid placed on the ground.

Our proposal is to remain in the vehicle frame by storing untransformed point clouds alongside their relative motion provided by the vehicle DR sensors, wheel encoders and yaw rate gyro. Point clouds are accumulated by bringing past scans to the present using the stored motion and merging them together, resulting in a denser point cloud referenced in the current sensor frame. The advantage of this method is that the vehicle does not have to know its global pose and can rely exclusively on its on-board DR sensors. In the absence of accurate DR sensors, other approaches could use ICP or derivatives to register these scans prior to ground and road filtering. However, as our DR model, described in [START_REF] Welte | Four-Wheeled Dead-Reckoning Model Calibration using RTS Smoothing[END_REF], is fairly stable over short periods of time (e.g: two seconds), we had no need for such registrations.

This has the effect of progressively scanning the surface of the road as illustrated in Figure 2, using movement to produce an instantaneous measurement. The buffer duration, or number of accumulated point clouds, can be changed to match the resolution and placement of the LiDAR, the accuracy of the DR model or the application of its output, depending on whether short bursts of accurate points are preferable to long term but less accurate points. This is detailed in Section V-B but for our use, an accumulation of 1 to 3 seconds is sufficient to drastically increase the longitudinal resolution using a LiDAR scanning at 10 Hz. The vehicle speed is also an important factor, as it directly impacts the resulting longitudinal resolution. Too slow and the rolling buffer will have lost the start by the time the end is reached and too fast and the distance between consecutive point clouds will be greater than the marking. This limits possible speeds between 10 and 50 km/h with our sensor placement.

D. Strips Detection

In our approach, crosswalk strips are first detected from the accumulated point cloud filtered around crosswalks before being fused in a single crosswalk detection. This allows for a more robust detection as strips constitute redundant information and can be easily rejected based on their shape.

The LiDAR points hitting the strips are extracted using the traditional Otsu dynamic thresholding [START_REF] Otsu | A Threshold Selection Method from Gray-Level Histograms[END_REF]. This method works by maximizing a weighted between-class variance inside a bimodal histogram. Assuming measured intensities are separated into a "pavement" and "marking" distributions, which is reasonable given the previous filtering steps, markings can be extracted by retaining points whose intensity is above the computed threshold. If some strips are weathered, they will thus be classified as pavement and not be used for crosswalk estimation, which is not an issue as long as there are other good strip detections.

These markings are then clustered together in order to process each strip independently. The chosen algorithm for this is called DBSCAN [START_REF] Ester | A density-based algorithm for discovering clusters in large spatial databases with noise[END_REF], and is similar to classic Euclidean distance-based clustering, to the difference of a density constraint. With DBSCAN, a class is propagated only when there is enough neighbors, otherwise, a point is considered noise. This choice is driven by the presence of between-strips outliers caused by rectification errors. The propagation of one strip to its neighbors is thus prevented by the low point density between strips.

Oriented bounding boxes are then fitted on each cluster using PCA. It returns the cluster center (x s , y s ) and principal orientation θ s that we will assume to be that of the strip, as strips tend to be longer than they are wide. However, in order to reject obvious outliers, the length L S and width W S of the bounding box are compared to their theoretical shape L, W and θ R by using confidence intervals. We consider that, after the successive outlier rejection mechanisms, the strip poses are used to compute the pose of the crosswalk as the average of its constituent strips, C = x C , y C , θ C T in the body frame of the car. Depending on sensor placement, markings can be seen from tens of meters ahead to tens of meters behind. Thanks to the accumulation, observation can be made as long as there are enough points, up to several seconds even with temporary occlusion such as when the vehicle passes over the crosswalk.

IV. LOCALIZATION FILTER

In order to evaluate the quality of crosswalk detection and to test the observation model, a state filtering scheme has been set up. The state and evolution model are taken from [START_REF] Welte | Four-Wheeled Dead-Reckoning Model Calibration using RTS Smoothing[END_REF]. An Extended Kalman Filter (EKF) has been implemented with measurements from a low-cost Ublox GNSS receiver that provides a global but inaccurate pose, the four wheel-encoders and the yaw rate gyro of the vehicle.

A. Marking Representation

In our HD map, illustrated in the background of Figure 1a, crosswalks are represented by a single closed polygon that covers all the crosswalk strips (and not each strips independently), as visible in gray on Figure 1a. It is attached to a series of nodes describing the center of the lane from which the orientation of the crosswalks can be extracted. In the following section, the polygon center [x O , y O ] alongside the road orientation θ O , expressed in the working frame, are used. Their accuracy is studied in [START_REF] Welte | Estimating the reliability of georeferenced lane markings for map-aided localization[END_REF] and is in the order of centimeters. It is worth noting that this information is also accessible in OpenStreetMap (OSM), where crosswalks are simply points referenced by lanes.

B. Crosswalk Observation Model

In order to incorporate the crosswalk detection into a localization filter, we developed an observation model based on the line-to-line constraint proposed in [START_REF] Choi | A line feature based SLAM with low grade range sensors using geometric constraints and active exploration for mobile robot[END_REF]. In this paper, representing lines as distances and orientations came naturally as a consequence of using Hough transform. However, in our case, we chose this representation because a crosswalk (as well as most other markings) can only give accurate information in one of its directions, here along the strips and this representation thus serves as a 1D constraint over this direction.

In this observation model, illustrated in Figure 3, we assume the LiDAR to be rigidly attached to the vehicle body frame and properly calibrated so that the output of the detector is expressed in the body frame M . X is the state to estimate that defines the frame M and is composed of its position x, y and its heading θ defined in the world frame as well. Let

z = M ρ C θ C = M x C cos θ C + y C sin θ C θ C (1)
be the measurement, as described in previous sections and

h(x) = M ρ O θ O -θ = M (x O -x) cos θ O + (y O -y) sin θ O θ O -θ (2)
be the estimated measurement projected in the body frame.

Then, H, the Jacobian of h(X) can be calculated as

H = -cos θ O -sin θ O 0 0 0 -1 (3)
and letting σ ρ and σ θ respectively be the longitudinal and orientation standard deviations (determined from the detector uncertainty analysis of Section V-B), the observation covariance matrix R of the filter can be defined as:

R = σ 2 ρ 0 0 σ 2 θ (4)
Because observations are produced from overlapping buffers, observation errors for the same crosswalk are correlated. This has been mitigated by enlarging the evolution model noise.

V. EXPERIMENTAL RESULTS

A. Experimental Setup

The method has been evaluated on data recorded with a Renault ZOE experimental vehicle from the Heudiasyc laboratory. The vehicle was equipped with a Velodyne VLP-32C LiDAR sensor. The vehicle internal sensors (gyro, wheel encoders) were accessed through its CAN bus. Ground truth positioning has been obtained by a Novatel SPAN-CPT system providing localization by fusing high accuracy IMU and Real Time Kinematics (RTK) corrections. The sensors were interfaced and recorded using the Robot Operating System (ROS) framework. The trajectory used in these experiments is located in Rambouillet, France and is illustrated in Figure 4. It consists of a stretch of 400 m of mostly straight roads where 5 crosswalks can be observed. This section is followed by 450 m of trajectory on a succession of roundabouts where over five crosswalks can be observed. Speeds during the recording were about 40 km/h on straight roads and 25 km/h in the vicinity of roundabouts.

B. Accuracy analysis of the crosswalk observations

The accuracy and number of observations of the detector depend on several factors, such as the buffer length or the odometry accuracy. We studied these factors through crosswalk detection using ground truth as position so that the observation error can be assumed to be entirely caused by the detector. We evaluate using the observation residual instead of using the classical precision and recall metrics by lack of observation ground truth.

1) Influence of buffer length: The buffer length is also a critical parameter of the detector. Intuitively, one may expect that longer buffer would lead to better accuracy and more observations. The experimental results in Figure 5 shows that on the contrary, a slight bias appears above 5 s. This behavior can be explained by several reasons. Longer buffers means that more DR measurements are accumulated, which in turns also accumulates more of their respective errors, increasing the overall one. Moreover, a long buffer results in observations being produced even though the crosswalk has not been observed for some time. Therefore, it is preferred to have a shorter buffer but sufficiently large to be able to detect crosswalks and provide an acceptable number of observations to the localization process. The small number of crosswalks detected at length 0.5 s confirms the benefits of accumulating point clouds. For the rest of the experiments, the buffer size has been chosen at 2 seconds as it is well balanced for crosswalks of 2 or 3 meters at nominal speeds.

2) Influence of DR measurements accuracy: The vehicle motion between two scans used for accumulating is sensitive to the accuracy of the DR system. This is all the more relevant since automobile grade sensors can be biased, especially gyros. We evaluate the effect of bad inertial measurements on the detector by artificially adding a yaw rate bias to the measurements. Figure 6 presents the loss of accuracy and availability that has been observed for different bias values (for a 2 s buffer).

For this size of buffer, the detector experiences little effect of a gyro bias up until a bias of 0.025 rad/s. This remains acceptable since typical observed biases are around 0.004 rad/s [START_REF] Welte | Four-Wheeled Dead-Reckoning Model Calibration using RTS Smoothing[END_REF]. A 2 second buffer can therefore be assumed short enough not to be significantly affected by inertial errors. It is important to note that the longer the buffer, the more sensitive it is to this effect.

C. Localization accuracy and consistency analysis

The contribution of marking information to global localization is evaluated here. In the following results, depicted in Figure 7, the ground truth positioning is only used to evaluate the localization accuracy of the filter. Here, the point clouds are accumulated using speeds and yaw rates computed using the vehicle on-board sensors. Since the filter here uses inaccurate GNSS positioning, the accuracy gained from crosswalks is easy to observe. In general, a notable improvement in localization error can be observed when crosswalks are detected, such as at times t = 20, 42 and 105 s, where the 2D error drops from 0.6 m to 0.1 m on average. At times t = 46, 117 and 130 s, crosswalks are observed while turning, as can be seen from the significant angle change in the bottom figure of Figure 7. This results in the marking being seen from different angles which does not improve localization, as the DR model is more sensitive to rotation error than translation error. When navigating in roundabouts (starting at t = 95 s), observing a single crosswalk does not have a significant impact on the localization error, but when two are seen at the same time, the state is constrained in two different directions, resulting in significant error reduction. On the whole trajectory, the average 2D error is reduced from 0.68 m to 0.49 m and the longitudinal error from 0.30 m to 0.19 m when in the presence of crosswalks. Finally, the filter and thus the accumulation and detection approach are consistent, as the East and North errors are always inside their 3σ confidence interval.

VI. CONCLUSION AND PERSPECTIVES

The local accumulation of LiDAR scans along a sparse axis of the sensor has shown to be efficient for accurately using geo-referenced markings for localization. We have proposed a method that uses movement to produce a punctual observation that can be used to localize a crosswalk in realtime. Our accumulation takes place in the body frame and produces point clouds that can be used to detect sparse features without loss of accuracy. It has been experimentally shown to improve the localization error in the vicinity of crosswalks, correcting the localization DR drift and reducing uncertainty while maintaining consistency up to 3σ. This indicates that LiDARs can provide accurate along-track measurements of road surface markings.

We have observed that incorporating an observation obtained through buffering of LiDAR scans must be done carefully. Observations produced in such a way can have correlated errors. The degree of correlation is difficult to assess as new scans are not guaranteed to change the measurement, e.g. if the new points are inside the already buffered strips. There are several ways to deal with this problem. The solution we have studied is to enlarge the evolution model noise to diminish the correlation impact. Another solution would be to use a filter capable of handling unknown correlation. The first time a crosswalk is detected, a Kalman update would be done. For all subsequent measurements on this crosswalk, a Covariance Intersection update would address the problem.

Fig. 1 :

 1 Fig. 1: Comparison between a single LiDAR scan and an accumulation around crosswalks. Colors from red to green represent low to high reflectivity. In the background are the road borders and crosswalk areas of an HD map.

Fig. 2 :

 2 Fig. 2: Sweeping of the crosswalk as the vehicle moves.

Fig. 3 :

 3 Fig. 3: Illustration of the observation model. In blue are vehiclerelated frames and coordinates, while green is for measurement and red for map-relative information.

Fig. 4 :

 4 Fig. 4: Satellite view and trajectory of the experiment. Crosswalks are highlighted in red.

Fig. 5 :

 5 Fig. 5: Error distribution of the crosswalk detector and number of detections for different buffer lengths.

Fig. 6 :

 6 Fig. 6: Error distribution of the crosswalk detector and number of detections for yaw rate biases.

Fig. 7 :

 7 Fig. 7: At the top, absolute 2D localization error without crosswalk observations (red), and using crosswalk observations (blue). Below are the 2D error decomposition in East and North components, along with their associated 3σ confidence interval. Gray areas indicate when a crosswalk is observed. At the bottom are the angle at which crosswalks are observed relative to the vehicle, where colors represent a given crosswalk.

ACKNOWLEDGMENT

This work has been carried out within SIVALab, a shared laboratory between Renault and Heudiasyc (UTC/CNRS), and cofinanced by the Hauts-de-France region (ERDF grant) and Renault.


Luma Miranda

Marc Swerts

João Antônio De Moraes

Albert Rilliard

João A Moraes

The Role of the Auditory and Visual Modalities in the Perceptual Identification of Brazilian Portuguese Statements and Echo Questions

Keywords: Audiovisual perception, Prosody, Statement, Echo question, Brazilian Portuguese Audiovisual perception. Prosody. Statement. Echo question. Brazilian

published or not. The documents may come   L'archive ouverte pluridisciplinaire

Note: The following files were submitted by the author for peer review, but cannot be converted to PDF. You must view these files (e.g. movies) online.

Introduction

When people process auditory information, like incoming speech, they tend to also rely on visual input, such as facial expressions a speaker produces while talking. That the visual and auditory channels can have a strong impact on speech perception has been shown in the well-known McGurk effect (McGurk & McDonald, 1976): that is, the presentation to participants of incongruent auditory and facial cues about specific segments tends to lead to an illusionary percept that integrates cues from the two modalities. Visual speech (lipreading) has also been studied as a source of information to help listeners with cochlear implants comprehend speech, while more recent studies have demonstrated that the visual channel is beneficial for listeners with normal hearing as well [START_REF] Rosenblum | Primacy of Multimodal Speech Perception[END_REF].

Previous studies in this area of research focused on the recognition of individual speech sounds, but there is a growing awareness that auditory and visual cues may also interact at higher levels. For instance, the speaker's face has been documented as a reliable indicator of emotions and attitudes in speech [START_REF] Barkhuysen | Cross-modal and incremental perception of audiovisual cues to emotional speech[END_REF]Crespo Sendra, Kaland, Swerts and Prieto, 2013;Ekman, Friesen and Hager, 2002;[START_REF] Moraes | Facial gestures in the expression of prosodic attitudes in Brazilian Portuguese[END_REF]. Studies have also shown that the visual channel can serve as a marker of communicatively relevant information that is often associated with prosodic features. The visual channel may also signal prominence [START_REF] Hadar | Head movement correlates of juncture and stress at sentence level[END_REF][START_REF] Krahmer | The effects of visual beats on prosodic prominence: acoustic analyses, auditory perception and visual perception[END_REF]Swerts & Krahmer, 2008), that is the marking of some words as being more salient than others in an utterance, as well as signal the focus of a sentence (Dohen & Loevenbruck, 2009;[START_REF] Krahmer | Pitch, eyebrows and the perception of focus[END_REF]. In this way, visual cues may signal the information status of sentence elements, for example, to distinguish broad from contrastive focus (e.g., see [START_REF] Lambrecht | Information structure and sentence form[END_REF], for a discussion). Visual cues are also used for the identification of statements versus questions [START_REF] Borràs-Comes | Seeing tunes. The role of visual gestures in tune interpretation[END_REF]Cruz, Swerts and Frota, 2017;[START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF][START_REF] Nicholson | Impaired processing of prosodic and musical patterns after right hemisphere damage[END_REF][START_REF] Srinivasan | Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English[END_REF]. In other words, given the multimodal nature of speech production, listeners perceive linguistic functions through the combination of acoustic and visual cues, both at segmental and suprasegmental levels of speech [START_REF] Gili Fivela | Multimodal analyses of audio-visual information: Some methods and issues in prosody research[END_REF].

In this paper, the bimodal perception of statements and echo questions in Brazilian Portuguese (henceforth, BP) is investigated. The choice of these two sentence types is motivated by the fact that (i) in BP the same syntactic structure is used for both sentence modes, which makes it a function prominently carried out by prosodic cues and (ii) few studies have analyzed the prosodic production of echo questions. This paper is based on insights from previous studies that have investigated the distinction between statements and questions in different languages. These studies are consistent in claiming that, although the auditory signal tends to be a more informative resource to determine the pragmatic status of an utterance, listeners also extract information from the visual channel to identify sentence types.

A number of studies experimentally investigated audiovisual processing of sentence types. [START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF] analysed the visual recognition of questions in Swedish, using an animated talking head whose visual cues were systematically manipulated to convey statements and questions in an interaction with humans. The experiments showed that listeners relied more on intonational than on visual cues for the identification of questions.

For American English, [START_REF] Srinivasan | Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English[END_REF] ran a set of experiments with both natural and virtual speakers based on one sentence produced as a statement or as an echo question; the lexico-syntactic structure of the sentence was kept constant in order to make sure that listeners did not rely on possible syntactic or semantic cues. Similarly to the Swedish study, the authors concluded that the intonation was more informative than the facial expressions; yet, in the final remarks of their work, they suggested that a larger corpus with natural speakers should be used to obtain additional auditory and visual cues that were absent in their experiments.

The integration of the auditory and visual cues in speech is also supported by experiments measuring reaction times. In European Portuguese, Cruz et al. (2017) compared the production and perception of statements and yes-no questions. The results showed that reaction times of the identification task were lower in the audiovisual condition compared to the monomodal conditions, including the auditory condition.

Borràs-Comes and Prieto (2011) analysed the audiovisual recognition of echo questions and statements with contrastive focus using a McGurk style paradigm (presenting incongruent auditory and visual cues) and collecting reaction times from Catalan listeners. Identification was faster when participants were exposed to congruent stimuli, supporting the bimodal integration of speech perception. The study also concluded that the visual channel was predominant over intonational cues for Catalan listeners' decisions. In their discussion, the authors stress the importance of the integration of visual and intonational cues for an accurate and rapid interpretation of these prosodic meanings.

The exceptionally high level of recognition of visual cues in Borràs-Comes and Prieto's (2011) study may be linked to the fact that the authors selected stimuli that "best characterized the contrast" (Borràs-Comes & Prieto, 2011, p. 362) between the sentence types upon which they focused. While this result provides insight into the integration of Most of the previously mentioned research reveals that visual cues have some effect on sentence type classification, but that their contribution is relatively small when compared to auditory cues. This raises the question of how the effect would play out for a language as BP, which is peculiar in that it does not mark yes-no questions by syntactic means (e.g., word order), but only by intonation. [START_REF] Moraes | Intonation in Brazilian Portuguese[END_REF] showed that the perceptually pertinent opposition is made by means of intonation, with a fundamental frequency (F0) change on the final pre-stressed and stressed syllables of a sentence nucleus used to mark assertive (F0 rise on the pre-stressed syllable followed by a fall on the stressed one) and interrogative (F0 fall on the pre-stressed syllable followed by a rise on the stressed one) meanings. [START_REF] Moraes | The pitch accents in Brazilian Portuguese: analysis by synthesis[END_REF] described nuclear pitch accents for these two modes, following the Autosegmental-Metric notation [START_REF] Pierrehumbert | The phonology and phonetics of English intonation[END_REF], as H+L*L% for statements and L+<H*L% for yes-no questions.

As mentioned before, instead of neutral yes-no questions, which are informationseeking questions, that is, the speaker asks for an information unknown to him [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF], we decided to analyze echo questions, because this type of yes-no question is underexplored in BP. According to [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF], the pragmatic function of echoquestions is twofold: to trigger a yes-no answer, just like the neutral yes-no question, and to signal a lack of comprehension of a previous utterance. When speakers produce echo questions, they repeat the previous utterance in order to verify if it was correctly for both types of questions was found [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF]. So, given the attested use of intonation patterns for distinguishing statements and echo questions in BP, it remains to be seen whether, in addition to these prosodic cues, and to what extent BP speakers may rely on facial cues for such an identification task.

In a pilot project, Peres, Raposo de Medeiros, Ferreira Netto and Baia (2011) analysed the role of the visual channel for the identification of statements and yes-no questions in BP. Their first experiment utilized a McGurk set-up whereby they created stimuli by combining the audio from one source with the video from another source. A second experiment proposed a video-only condition in which listeners had to assess muted versions of recorded clips. The first experiment indicated a dominance of intonational information over visual information to identify statements and questions, while the second showed that yes-no questions and statements could be recognized by Brazilian participants through the visual channel only. Still, the number of participants of this study was limited to ten listeners, for all the experiments. Hence, the authors' conclusions about the contribution of each channel for the perception of statements and questions in BP may be considered preliminary.

The current study extends the analysis of the prosodic and visual cues that have been proposed to be valuable for the identification between statements and questions in BP. In comparison with prior work, we increased the number of speakers, participants and presentation conditions of audiovisual stimuli in order to obtain a better understanding of the multimodal perception process involved in such a task. As it was our aim to compare the relative capacities of prosodic and visual cues in conveying this distinction, we choose to keep the lexical and syntactic material constant, despite a possible bias, or lack of variability, this may introduce. Note that we decided to include stimuli from multiple speakers to make sure that the perceptual results had general and ecological validity, as opposed to previous studies that were based on evaluations of a schematic or highly stylized animated talking head with specific auditory and visual parameter settings. Additionally, we semi-randomly selected speakers' productions from their ten realizations, without explicitly choosing their "best performances".

The phonetic analysis of this study served to describe relevant prosodic cues that can be also used to identify the statements and echo questions in the initial and final regions of the intonational contours regarding not only the F0, but also the intensity and duration parameters. A visual analysis of the recorded data was conducted in order to identify and categorize prototypical gestures that were related to the production of each sentence type in BP and which listeners could recognize in perceptual tasks.

In addition, considering that the intonational cues for these contours in BP are known to be highly intelligible, the role of the visual channel was also explored through the identification tasks of statements and questions using degraded audio conditions. As information from the face is known to improve speech intelligibility in noise [START_REF] Benoît | Audiovisual speech synthesis from French text: Eight years of models, designs and evaluation at the ICP[END_REF][START_REF] Ouni | Visual contribution to speech perception: Measuring the intelligibility of animated talking heads[END_REF]Sumby & Pollack, 1954), the acoustic signal of the recorded data was degraded with two levels of Signal-to-Noise Ratio (SNR), using a noise based on a multitalker babbling speech recording.

The main goal of this paper is to understand the contribution of the speaker's face in the perceptual integration of auditory and visual channels for the identification of statements and echo questions in Brazilian Portuguese. This study aims to investigate the following research questions:

(i)
To which extent the visual cues-alone or combined with auditory cuesallow for the correct identification of statements and questions? On the basis of the literature on BP, we may hypothesize a higher recognition rate in the bimodal presentation condition as well as a lower identification ratio for visual cues than for auditory ones, but still above chance.

(ii) What is the extent of the influence of the visual channel when the auditory information is impoverished? Based on prior studies, we expect a higher contribution of the visual channel in this condition.

(iii) Which channel will lead the subjects' interpretation when the voice and the face are signaling different information? Here again, a dominant role of auditory cues is foreseen.

The article is divided into the following sections: we first describe how we obtained audiovisual recordings and how we analysed the prosodic and visual properties of these data. Then, the designs of Experiments 1 and 2 are explained followed by a combined statistical analysis of the results of both tests. Next, the setup of Experiment 3 is presented along with the statistical analysis, combining the results of Experiments 1 and 3. Finally, the outcomes of the three experiments are discussed and followed by a general conclusion.

Methods

Structure of the corpus and participants

We invited ten native speakers of BP (five male) to produce different versions of the sentence "Como você sabe", both as an assertion (with the meaning "As you know.") or as an echo question (with the meaning "As you know?", which can also be rephrased as "Is it 'as I know' that you are asking me?"). Note that the sentence does not contain any lexical or morphosyntactic indices related to the mode; studies so far (Frota et al.,

Recording procedure

Speakers were asked to read and sign a consent form in which the audiovisual recording process and the future use of the data were explained. Speakers were seated against a dark background in a sound-attenuated room at the Acoustic Phonetics Laboratory of UFRJ. None of them wore glasses. They were recorded using a SONY NEX-F3 video camera, placed 90 cm from the speakers. The camera was positioned to frame the upper body and face. To increase the quality of the acoustic signal, a Zoom H4 recorder was used to jointly record each voice, with a microphone located 20 cm from the speaker's mouth, outside the frame of the camera. Although the speakers are not actors, they were told to express themselves as naturally as possible. The experimenter was in the room during the whole recording session, instructing the speaker about the sentence types. Each subject repeated the sentence ten times, alternating statements and echo questions. This process resulted in 200 recorded utterances. After the recording sessions, the audio waves were synchronized with the video in the software Vegas Pro 14.0 (Magix, 2014), using a handclap produced at the beginning of each recording session.

Acoustic analysis

The 200 recorded sentences were manually segmented at the phonemic level using the Praat software [START_REF] Boersma | Praat: doing phonetics by computer[END_REF]. Prosodic correlates of fundamental frequency (F0, expressed in semitones (ST) relative to 1 Hz, using Praat's default algorithm), intensity (expressed in dB) and syllabic duration (in seconds) were estimated from the acoustic signal. Some pre-processing of the raw measurements was made: a Praat script produced time normalized F0 contours [START_REF] Arantes | Time-normalization of fundamental frequency contours: A hands-on tutorial[END_REF] that allowed the comparison of F0 prominences between speakers and modes. The same script was adapted to time normalize the intensity measures. Another Praat script was used for the duration analysis by transforming the raw phonemic duration values in a smoothed zscore values that removed intrinsic differences in phoneme duration [START_REF] Barbosa | Semi-automatic and automatic tools for generating prosodic descriptors for prosody research[END_REF].

The results of these measurements indicate that the assertive and interrogative contours bear distinct F0 movements. These distinguishing patterns are quite consistently expressed across our recordings: to implement the declarative contour, all speakers produce a rising F0 movement from the beginning of the contour until the nuclear prestressed syllable ("cê"), followed by a fall on the stressed one ("sa"), whereas the echo question is characterized by a low F0 from the prenuclear region of the contour until the pre-stressed syllable ("cê"), followed by a rise on the stressed ("sa") and a fall on the post-stressed syllable ("be"), as illustrated in Fig. 1.

[Insert Figure 1]

The patterns presented in Fig. 1 are consistent with the descriptions in the literature on BP intonation [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF][START_REF] Miranda | Análise da entoação do português do Brasil segundo o modelo IPO[END_REF][START_REF] Moraes | The pitch accents in Brazilian Portuguese: analysis by synthesis[END_REF]. Moreover, it can be

F o r P e e r R e v i e w
noticed that the final pre-stressed syllable ("cê") in the declarative contour is high, whilst the same syllable is low in the interrogative contour, marking the contrast between the final pre-stressed syllables ("cê") in both contours. This behavior of the final pre-stressed syllable ("cê") is corroborated by Couto, Silva and Miranda (2017) who analysed the declaratives and interrogative contours in three BP varieties-Rio de Janeiro, Fortaleza

and Salvador. This difference between the nuclear region of the contours supports Moraes's descriptions (1998Moraes's descriptions ( , 2008) ) that can lead in representing the pitch accents of statements as H+L* L% and those of questions as L+<H*L%, as already stated in the introduction of this study.

The acoustic analysis also shows that the intonational contours of both the statement and the echo question are accompanied by specific intensity patterns and similar duration patterns (cf. Fig. 2).

[Insert Figure 2]

Regarding intensity, the main differences between statements and echo questions are found around the tonic syllable ("sa"), with (i) a stronger pre-tonic syllable ("cê") and

(ii) a weaker post-tonic syllable ("be") for the assertive contour, as compared to the interrogative one. The tonic syllable ("sa") is the stronger and the longer in both cases,

showing comparable values across modes. In addition, based on the bottom panel of Fig.

2, it is possible to observe an increasing syllabic duration along sentences for both modes, up to the tonic syllable ("sa") of the nucleus.

The analysis of the prosodic parameters shows that there are clear prosodic features discriminating statements from echo questions in BP, most clearly in terms of their F0 contours.

Facial movements analysis

F o r P e e r R e v i e

w

Analyses of facial movements during the production of statements and questions in different languages generally reveal a recurrent set of visual cues. For interrogatives, the literature shows that the head and the eyebrows movements are commonly employed [START_REF] Cavé | About the relationship between eyebrow movements and F0 variations[END_REF][START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF] and provide reliable cues when manipulated in an animated talking head [START_REF] Granström | Multimodal feedback cues in humanmachine interaction[END_REF][START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF]. For instance, eyebrow raising and head tilt were the visual cues found in English echo questions by [START_REF] Srinivasan | Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English[END_REF], while eyebrow lowering and vertical head tilting conveyed yes-no questions in the Swedish study conducted by [START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF], using an animated talking head.

In multimodal communication studies, various body movements have been considered as well (hand gestures, head movements, speakers' gaze, etc.). Torreira and Valtersson (2015) investigated conversational data in French to check whether polar questions and continuation statements, which are similar in morphosyntactic and intonational forms, could be discriminated by prosodic and visual cues. They concluded that the distinction of the sentence types is based on gaze towards the addressee during questions, but eyebrow raising, head movements and other interactive manual gesture types were also identified.

In the analysis of our video data, movements on the speakers' faces were described using the FACS annotation system (Ekman et al., 2002) Two of the authors independently analysed forty videos, corresponding to two repetitions of statements and echo questions produced by ten speakers, and annotated the facial movements in terms of AUs. The selected stimuli correspond to those used in the perceptual experiments described hereafter. A moderate agreement between the two annotators was reached, as measured by Cohen's Kappa (κ = 0.488). Although significantly above 0 (i.e., no agreement above chance level), this kappa is described as "moderate" by [START_REF] Landis | The measurement of observer agreement for categorical data[END_REF]. Thus, to maintain the possibility of accounting for possible divergences between the two annotators, the AU analysis was made keeping the annotations of both authors separated. A correspondence analysis was run on the count of AUs observed by speech act, speaker and annotator. The analysis extracts the main dimensions that explain attribution of AUs to the stimuli. The ten first dimensions, which explain about 85% of the variance, were selected to run an algorithm of hierarchical clustering which was used to group together stimuli that have similar characteristics [START_REF] Husson | Exploratory multivariate analysis by example using R[END_REF]. The cluster analysis proposed a three-cluster solution.

Cluster I is composed of stimuli showing AU 4 (brow lowerer) and AU 7 (lid tightener) and expressing echo question. Cluster II contains AU 1 (inner brow raiser), AU 2 (outer brow raiser) and AU 55 (head tilt left) associated with the production of assertions. An example of these two facial expressions can be seen in Fig. 3.

[Insert Figure 3]
Cluster III links AU 17 (chin raiser) and AU 57 (head forward) to the production of echo question by speaker 7. These last two AUs may be considered part of this speaker's idiosyncrasy and are illustrated in Fig. 4. These three clusters are essentially related to the two speech acts under investigation, plus an idiosyncratic behavior for echo questions by a particularly visually expressive speaker. Note that some of the gestural features produced in co-occurrence with echo questions seem to share similarities with the recurrent "shrug" gestures described in Debras (2017). In addition, differences in labelers' perception do not introduce a major bias at this broad level of clustering.

This analysis of the facial movements produced by Brazilian speakers uttering statements and echo questions shows that Brazilian speakers in our dataset convey the meaning of statements and questions with different and recurrent facial gestures.

Therefore, in principle, these speech acts may thus be possibly identified not only by prosodic cues (especially regarding F0 patterns), but also by visual cues. The video recordings of Fig. 3 and 4 can be accessed in the supplementary materials of this paper.

The findings of our visual analysis are partially in line with the visual description of statements and yes-no questions in European Portuguese (EP) provided by [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF]. EP speakers produce statements with an up-down head movement, whereas yesno questions are produced with either up-down or back-forward head movements plus eyebrow raising. Thus, the head movement is argued to be a visual cue produced by EP speakers for both statements and yes-no questions. In our data, BP speakers also produced head movement with the assertion meaning, just like in EP. On the other hand, the eyebrow raise was observed in the production of BP assertions, instead of questions, as described in EP. In BP speakers' production of echo questions, the eyebrows are typically lowered and the eyes tightened.

Therefore, our results show that eyebrow lowering can be related to question marker in BP, as opposed to the eyebrow raise described by [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF] for EP. It is worth

F o r P e e r R e v i e w
to note that this difference in the facial performance can be related to the speech acts explored in both studies, since echo-questions were analyzed in BP and neutral yes-no questions in EP. Although an echo-question is also considered a type of yes-no question [START_REF] Frota | Intonational variation in Portuguese: European and Brazilian varieties[END_REF], as stated in the introduction of this study, the visual production of echo-questions may express a component of doubt or uncertainty, which was shown in Fig. 4 with the facial expression of cluster III.

Summarizing the prosodic and visual descriptions, brow lowering along with tightening of the eyes is associated with the echo questions that present a rising nuclear pitch accent and, in terms of intensity prominence, a weaker final pre-stressed syllable ("cê") as well as a stronger final post-stressed syllable ("be"). The eyebrow raising plus head movement is related to the assertion contour with a falling pitch accent and a stronger final pre-stressed syllable ("cê") plus a weaker final post-stressed syllable ("be").

The final stressed syllable ("sa") is the strongest in relation to intensity and the longest in both contours.

Perceptual experiments

In this section, the described prosodic and visual cues related respectively to statements and echo questions are perceptually validated. The design of Experiments 1 and 2 are presented separately and the results of both are commented together. Afterwards, the setup of Experiment 3 is exposed along with the outcome that was also combined with Although the majority of participants were either students or professors in Linguistics at UFRJ, some of them were working or studying in different scientific fields (e.g., Philosophy and Engineering). They participated in the experiment through a web interface, using either their personal computer or one provided by the Acoustic Phonetics Lab at UFRJ. For each presentation condition, a new group of listeners was recruited.

None reported hearing or sight impairments. Use of headset was required for the auditory conditions implying audio presentations.

Stimuli

The eighth and ninth repetitions of the assertion and echo question produced by all speakers were selected. This way, we gained a randomization of the video material.

Unlike Borràs-Comes and Prieto (2011), we did not select the visual data qualitatively.

In other words, we did not select the most expressive productions from each speaker to avoid potentially over-acted data. The video clips were cut in the software Vegas Pro so that their length was approximately two seconds. It is worth noting that the production of the whole utterance contained in the clip was often preceded or followed by a short silent pause. In total, forty utterances were selected for the perceptual experiments.

Procedure

We used the Qualtrics platform to run this perceptual experiment. Stimuli were presented one by one, in a randomized order as generated by the software for each subject.

Subjects were tested individually. They had to identify the sentence mode between the two alternatives (forced choice), by pressing a "statement" or "question" button.

Participants were told that, depending on the experimental condition, they would see, hear, or see and hear speakers uttering either a statement or a question and had to judge the intended meaning of the utterance "Como você sabe".

Experiment 1 presented forty utterances in three conditions: audio-only, video-only and audiovisual, with each condition presented to a different group of participants. This experiment had a between-subject design to avoid possible learning effects that could come into play if participants were presented with all experimental conditions. Each group of participants in the three different conditions were presented with forty trials. The remaining number of trials was always indicated at the top of the screen. A typical run lasted approximately fifteen minutes.

Experiment 2: Audiovisual recognition with degraded audio

The goal of the second experiment was to analyze the robustness of the audiovisual presentations. Thus, the role of the prosodic and visual cues in the recognition of statements and echo questions in Brazilian Portuguese was assessed in degraded audio conditions.

Participants

Stimuli

The video material of Experiment 2 is the same used in Experiment 1, except that the auditory channel is degraded by mixing it with a multi-speaker babble noise taken from [START_REF] Steeneken | Description of the RSG-10 noise database[END_REF]. Note that the speakers were not recorded in noisy conditions. In this study, we are interested in noise for its capacity to degrade the given prosodic cues, so to measure the relative power of visual cues. Recording the speakers in noise would probably change their communication strategies as a consequence of the Lombard effects, leading to an increased F0, duration and energy as well as bigger motions of the face and the head (Fitzpatrick, Kim and Davis, 2015;[START_REF] Kim | Hearing speech in noise: seeing a loud talker is better[END_REF], which is not our aim here.

Regarding the addition of noise in the recording material, [START_REF] Cooke | A glimpsing model of speech perception in noise[END_REF] recommends a fixed signal-to-noise ratio (SNR, henceforth) as being more effective when multispeaker babble noise rather than a single competing talker is employed. Furthermore, the expected limits of SNR for individuals with normal hearing suggested by a study with Brazilian listeners (Costa, Daniel and dos Santos, 2011) showed a range of the SNR that goes from -3,03 dB to -7,55 dB with a mean of -5,29 dB. Based on the latter study and after a pilot experiment using SNR of +3 dB, 0 dB and -6 dB, we selected the last two levels of SNR (0 dB and -6 dB) , and applied the continuous multi-speaker babble noise to the original utterance: a favorable SNR of 0 dB and an adverse SNR of -6 dB.

The degraded audio was produced in Praat and later combined again within the original videos in Vegas Pro software. To keep the task feasible, only one repetition from each speaker was used, thus presenting forty stimuli with degraded audio to listeners. In the supplementary materials of this paper, four samples of the audiovisual stimuli with degraded audio can be accessed (two statements and echo questions with SNR of 0 dB and -6 dB produced by one speaker). It is worth remembering that the segmental material did not change in these experiments: listeners had to decode prosody, a different type of information than the segmental one studied in literature using such masking noise.

Procedure

The second perceptual experiment followed the same procedure as described in Experiment 1. The only difference is that Experiment 2 presented forty stimuli with two SNR levels (0 dB and -6 dB) in only two conditions: audiovisual and audio-only conditions, each condition presented to a different group of participants as well. This experiment also had a between-subject design, avoiding potential learning effects for the same reason presented in Experiment 1. Each group of subjects was presented with forty trials and, just like the previous experiment, the remaining number of trials was always indicated at the top of the screen. Participants took fifteen minutes to complete the perceptual task as well. case. The fact that the AV, AV-0 and AV-6 conditions did not significantly differ from the AO condition also advocates for a more general "ceiling effect" for all these conditions: they shall thus all be regarded as presenting comparably high recognition scores. Bimodal presentations (AV) achieve a comparable performance level, with nonsignificant variations (that thus may be related to, e.g., subjects sampling). The bimodal presentations with degraded audio (AV-0, AV-6) also reach a level of performance comparable to the audio-only and audiovisual conditions. On the contrary, conditions based on visual information alone (VI) and degraded audio alone (AO-0, AO-6) reach a significantly lower identification level. This result supports the view of a cross-modal fusion of information to the interpretation of sentence mode.

Results of

Regarding the factor Type, assertions presented an identification accuracy ratio about six percent higher than echo questions, which was a significant improvement. The fact that there is no interaction between the Type and Condition factors indicates a general tendency of subjects to prefer the "assertion" answer in any presentation condition. One may speculate that assertion functions as a default answer when no clear or interpretable information is available for participants.

Experiment 3: Congruent and incongruent modalities

The third experiment presented congruent and incongruent audio and visual modalities to evaluate which channel exerted a dominant influence on listeners' identification of sentence type.

Participants

Experiment 3 was applied to 24 participants with mean age of 29,4 years old. Once again, listeners were either students or professors of Linguistics from UFRJ. They We decided to construct mixed stimuli for both the congruent and incongruent cases in order to generate a total set of data that were artificially constructed. Using this procedure, we obtained forty artificial videos, twenty with congruent auditory and visual streams, twenty with incongruent streams.

Procedure

The same procedure used in Experiments 1 and 2 was applied in the third experiment, but here all participants would hear and see the videoclips. Experiment 3 presented forty stimuli, containing either congruent or incongruent auditory and visual information (twenty stimuli for each of these two types). The congruent audiovisual condition (AVc) is in principle similar to the audiovisual presentation (AV) of Experiment 1, except that the audio and visual channels were taken from two different repetitions of the same speaker for the same targeted sentence mode. The incongruent condition (AVi) consisted of combinations of audio and visual modalities that were retracted from distinct sentence modes (e.g., the auditory information from an echo question paired with the visual information from a statement). The forty stimuli were presented in audiovisual condition to one group of 24 participants (within-subjects design). There were forty trials in this experiment and, just like the previous perceptual tests, the remaining number of trials was also indicated at the top of the screen. A typical run also lasted fifteen minutes. As shown in Fig. 6, stimuli in congruent audiovisual presentations were better recognized than in audiovisual incongruent ones, which caused the identification rate of the sentence types to decrease.

Results

Results of

[Insert Figure 6]

Based on Experiment 3, we can conclude that the visual channel interferes in the process of listeners' identification of statements and questions, although the auditory component is dominant for the perceptual recognition of these sentence types. This finding means that the speech perception of the intonation of sentence types is bimodal even when the auditory channel is clear.

General discussion

The main goal of this study was to analyze the contribution of the visual channel, in combination with the auditory one, as a possible resource to identify statements and echo questions in BP. To this end, we made use of one carrier sentence produced repetitively by ten speakers of the Rio de Janeiro variety of BP. We chose to use only one sentence because our primary aim is to validate the relative role of the visual modality, controlling for other potential variables (e.g., variation in articulation induced by different sequences of phonemes). We did record various speakers with variable productions to make sure that our findings have general validity; note also that these speakers produced prosodic patterns that have been reported in the literature to be representative of the sentence modes we explored. First, the recorded data was described both acoustically and visually. The outcome of the prosodic and visual descriptions brought to light relevant features that could function as cues for this identification in perceptual tasks. The most relevant prosodic parameters to distinguish both modes are the F0 contours, as described in the literature, combined with intensity patterns. The most salient visual cues for questions were the brow lowerer (AU 4) and lid tightener (AU 7), while chin raiser (AU 17) and head forward (AU 54)

were more consistently found for one of the speakers, which may be considered an idiosyncratic performance or something unrelated to the distinction of modes under investigation here. For statements, the eyebrow raising movements (AU 1 and AU 2) and head movements (AU 55) turned out to be characteristic for this sentence type. Different eyebrow and head movements seem to visually distinguish statements from echo questions in BP.

Next, we were interested in analyzing the perceptual identification of these two sentence types to evaluate the relative strength of the described auditory and visual cues when clear or impoverished audio signal were presented, as well as when both modalities signaled conflicting information.

Experiment 1 investigated the capacity of participants to identify statements and echo questions in clear audio conditions, in the three presentation conditions: audio-only (AO), video-only (VI) and audiovisual (AV). A ceiling effect was observed in the results, that implies identification performances in audiovisual condition (AV) that could not outperform the audio-only condition (AO) in which the outcome was already quasi perfect. Comparably high-performance levels were found for all the conditions containing clear or degraded audio cues and coherent visual ones, as for clear audio cues only. It is worth mentioning that this type of effect obtained in audio-only (AO) and audiovisual (AV) presentation conditions of Experiment 1 is in line with previous studies that looked One could hypothesize that the visual benefit in the interpretation of the intonation of sentence types is higher when the analyzed intonational contours are similar (i.e., more difficult to identify) compared to contours with clearly different F0 configurations, as in our study. For instance, in [START_REF] Borràs-Comes | Seeing tunes. The role of visual gestures in tune interpretation[END_REF], the difference between a contrastive focus statement and an echo question in Catalan is given by pitch range, since both contours have the same F0 contour (rising-falling nuclear configuration). In the audiovisual experiments, the authors concluded that the visual cues had a stronger effect than the audio cues in the perceptual identification of both utterance types. Also in Catalan, Tubau, González-Fuente, Prieto and Spinal (2015) found that the identification of confirmation and contradicting yes-answers, both with falling F0 patterns, was higher in the audiovisual modality than in the audio-only modality. Similarly, in Miranda, [START_REF] Miranda | Audiovisual perception of whquestions and wh-exclamations in Brazilian Portuguese[END_REF], the distinction between wh-questions and wh-exclamations in BP is characterized by the inclination of the falling pitch accent in the nuclear region of both contours, which is steeper in wh-questions than in wh-exclamations. Results from the perceptual identification test showed an improved recognition in the audiovisual modality compared to monomodal ones (video-only and audio-only). All these studies demonstrated that facial gestures enhanced the interpretation of the intonation of the sentence types, without limitation due to a "ceiling effect".

In sum, results of Experiment 1 lend support to the conclusions of previous studies that highlight that the same functions are performed by prosody and facial gestures, although visual cues alone are less efficient than the prosodic ones [START_REF] House | Intonational and visual cues in the perception of interrogative mode in Swedish[END_REF] [START_REF] Srinivasan | Perceiving prosody from the face and voice: Distinguishing statements from echoic questions in English[END_REF]. Nonetheless, the speaker's face can still be considered a reliable source of prosodic information at the sentence level. For instance, in Fisher's study (1969) regarding the "visibility of final pitch" variation in English, listeners had to indicate in a perceptual experiment the direction of terminal pitch contour in the production of questions with either a rising or a falling contour, while watching the visual performances of speakers without sound. Results showed that listeners were able to identify the direction of the intonational contours based on the visual cues. Our findings also support the view that participants are able to identify the intonational contours of different utterance types by relying only on the speaker's face. Fisher attributes the listeners' visual discrimination of final pitch contours to certain motor activities of the face such as the movement of the lips. Still, he also suggests in the final remarks of the paper that the whole face could be a source of information for the listener.

In Experiment 2, participants again had to identify statements and echo questions but this time with noisy stimuli. Noisy audio-only presentations (audio-only in 0 dB SNR/AO-0 and audio-only in -6 dB SNR/AO-6) show a significant decrease in identification performances compared to the audio-only (AO) ones, but there still exist some cues to identify mode, with performances comparable to video-only (VI)

presentations. The auditory (AO-0, AO-6) and visual (VI) channels share their respective cues in the audiovisual in 0 dB SNR (AV-0) and audiovisual in -6 dB SNR (AV-6) conditions to attain identification levels comparable to the ideal situation of a clear audio (or audiovisual) presentation. The visual enhancement of the identification of intonational contours in adverse auditory conditions found in our study supports findings of earlier work on visual speech. For instance, [START_REF] Benoît | Audiovisual speech synthesis from French text: Eight years of models, designs and evaluation at the ICP[END_REF] and [START_REF] Nicholson | Impaired processing of prosodic and musical patterns after right hemisphere damage[END_REF] state that there is a lower impact of the visual channel on speech perception compared to a clear acoustic signal, but the visual speech can increase identification scores when the auditory channel is impoverished. Therefore, our findings confirm that speech intelligibility in a noisy environment is improved when participants have access to the speaker's face.

The results of Experiment 2 also showed that babble noise can efficiently degrade prosodic cues to sentence mode (and probably to other prosodic functions), opening a possible type of evaluation for the contribution of multimodal cues to linguistic meaning (cf. Dohen & Lovenbruck's, 2009, for an opposite prediction). However, since the speakers of our study were not recorded in noisy conditions, their productions do not fit those that would have been produced in a noisy environment. In such a situation, a Lombard speech effect would probably have occurred. It has been found that in speakers' recordings in adverse auditory conditions (Fitzpatrick et al., 2015;[START_REF] Kim | Hearing speech in noise: seeing a loud talker is better[END_REF], there are differences regarding not only the prosody, such as increased energy, F0, duration and flatter spectral tilt, but also the facial gestures, like bigger motions of the face and head.

These are different from the more neutral expressions we elicited in our set of data.

Meanwhile, it is not our goal here to evaluate the capacity of a speaker to communicate in adverse conditions, but rather to measure the relative role of both modalities.

From a perception point of view, Vatikiotis-Bateson, [START_REF] Vatikiotis-Bateson | Audiovisual Lombard speech: reconciling production and perception[END_REF] verified that both the auditory intelligibility and the visual enhancement are greater when speech is recorded in noise compared to speech recorded in quiet and then added noise to it. However, in the same study, when stimuli were presented to listeners in an "easy listening level", the authors found results in which the recognition of words in the condition of speech recorded in quiet with addition of noise was similar to the condition of stimuli recorded in noise. [START_REF] Vatikiotis-Bateson | Audiovisual Lombard speech: reconciling production and perception[END_REF] state that, depending on the used listening levels of SNR, results can be substantially altered. acoustic presentations, as the visual information degrades their decoding when incoherent. When stimuli are presented in a noisy context, the results reveal that visual cues also add robustness in the speech decoding process.

Therefore, these results are in line with Massaro and Cohen's assumptions (1983),

regarding not only the integration of auditory and visual cues as a fundamental process of speech comprehension, but also regarding their viewpoint that we can expect a greater contribution of the visual channel when either the verbal information is ambiguous or the audio information is degraded.

Conclusion

In summary, this article shows that the auditory and visual channels are integrated in the perception of sentence modes in BP. Brazilian listeners identify statements and questions using both auditory and visual cues. Listeners rely more on the auditory cues than the visual ones when presented under clear acoustic conditions. Contrary to our expectations, the recognition rate of the sentences mode in the bimodal condition was not higher than in the audio-only condition. Thus, the first hypothesis written in the introduction section of this study was partially confirmed. However, the visual channel has a beneficial effect when stimuli consists of degraded auditory cues, which confirms the second hypothesis. Finally, although a dominant role of the auditory cues was verified in the experiment with congruent and incongruent audio and visual modalities, the auditory and visual channels are integrated in the perceptual identification process of sentence types, confirming the third hypothesis as well.

We intend to further explore other types of sentences in additional experiments, such as the continuation statement, which is also an underexplored sentence type in BP. For instance, [START_REF] Miranda | Audiovisual perception of whquestions and wh-exclamations in Brazilian Portuguese[END_REF] analysed the bimodal perception of BP wh-questions and Fitzpatrick, M., Kim, J. and Davis, C. (2015). The effect of seeing the interlocutor on auditory and visual speech production in noise. Speech Communication, 74, 37-51.

Fig. 3 -

 3 Fig. 3-Assertion production of a female speaker.mp4 Fig. 3-Echo question production of a female speaker.mp4 Fig. 4-Echo question production of a male speaker.mp4 Assertion with SNR of 0dB produced by a female speaker.mp4 Echo question with SNR of 0dB produced by a female speaker.mp4 Assertion with SNR of -6dB produced by a female speaker.mp4 Echo question with SNR of -6dB produced by a female speaker.mp4

  sentence type identification, it may not really be representative as it makes use of very prototypical and possibly exaggerated expressions that may not generalize to more natural data. It would thus be useful to investigate to what extent similar findings could be obtained with recordings of more natural and spontaneous occurrences of the facial gestures accompanying the production of echo questions and statements with contrastive focus.

  European and Brazilian Portuguese varieties, a similar nuclear rising F0

  The first experiment was designed to verify the relative importance of visual cues in comparison to the auditory ones as signals of the statement and echo question distinction in Brazilian Portuguese.ParticipantsExperiment 1 was applied to 64 participants (21 in the audiovisual, 22 in the videoonly and 21 in the audio-only condition) with mean age of 29,1 years old. Listeners originated from various regions of Brazil, though predominantly from Rio de Janeiro.

  Results of Experiment 2 were analyzed along with Experiment 1. Subjects' responses were expressed as success or failure to identify the presented mode (assertive or interrogative). The proportion of success was expressed for each of the seven presentation conditions in Experiments 1 and 2 (audio-only/AO; video-only/VI; audiovisual/AV; audio-only in 0 dB SNR/AO-0; audio-only in -6 dB SNR/AO-6; audiovisual in 0 dB SNR/AV-0; audiovisual in -6 dB SNR/AV-6-see Fig.5that shows this ratio in each condition as well as each type of sentence, assertive or interrogative). The ratio of success was used as the dependent variable for a logistic regression model (quasibinomial errors were used to deal with overdispersion in binomial model residuals), using two categorical explanatory factors (Condition and Type, and their interaction) as predictors.The logistic regression shows that the interaction between both factors is not significant (F(6,290) = 1.6, p = 0.15), while both Condition and Type factors do have a significant impact on the accuracy of judging the sentence mode (Type: F(1,290) = 9.28, p < 0.01; Condition: F(6,290) = 10.14, p < 0.01). A post-hoc Tukey test was applied to the Condition factor so to test for differences between levels of the two factors, showing that three presentation conditions (video-only/VI, audio-only in 0 dB/AO-0 and audio-only in -6 dB/AO-6) do significantly degrade the listener's identification performances compared to the best presentation condition: audio-only (AO). A comparison of participants' performance in each experimental condition of Experiments 1 and 2 is shown in Fig.5:[Insert Fig. 5]The results show a group of presentation conditions containing the AO, AV, AV-0 and AV-6 conditions with homogeneous and high identification capabilities. The audio presentation conveys identification information about sentence mode: the AO condition reaches a high level of correct answers (90% of correct answers, with a confidence interval that nears a perfect score)-thus there is a strong suspicion of ceiling effect in this

  experiment through a web interface in the computer of the Acoustic Phonetics Lab at UFRJ. The participants do not report any hearing or sight impairments from the participants. The use of headset was also required for the audiovisual presentation of this experiment.StimuliThe set of 40 videos (2 repetitions x 2 sentence types x 10 speakers) was used to mix modalities from different stimuli. Using the Vegas Pro software, the audio utterances were matched with video materials from other recordings. The congruent videos were based on the audio of the first repetition synchronized with the video of the second repetition of the same mode; that is, they consisted of both an audio and video channels that cued the same sentence type. Incongruent videos are based on the audio of the first repetition of one mode matched with the video from the second repetition of the other mode, so that there was a mismatch between what the audio and video signals would cue.

  Experiment 3 are analysed in relation to the outcome of Experiment 1. Answers were expressed as success or failure to correctly identify the assertive or interrogative mode of the acoustic part of the stimuli (because in incongruent presentations, the visual information signals the alternative category). Results for the two conditions of Experiment 3 are presented in Fig. 6. The proportion of success was calculated for the five presentation conditions (audiovisual/AV, audio-only/AO and video-only/VI from Experiment 1; congruent audiovisual/AVc and incongruent audiovisual/AVi from Experiment 3) and each sentence type (statement or question).Variations in the ratio of success according to Condition and Type of sentence factors were again analysed by a logistic regression (with quasibinomial errors).The logistic regression shows that the interaction between the two factors is not significant (F(4,218) = 0.83, p = 0.51), while the two main factors have a significant impact on ratio of success (Type: F(1,226) = 8.08, p < 0.01; Condition: F(6,290) = 8.85, p < 0.01). A post-hoc Tukey test was applied to Condition factor and shows that the video-only (VI) (AVi) levels do significantly degrade the listener's identification performances, compared to the audio-only (AO) presentations.

  discourse items are signaled. For instance, as the contrastive focus in French(Dohen & Loevenbruck, 2009) is accurately marked by prosodic cues, the visual cues in bimodal presentations do not have an extra beneficial effect and so could not increase performances that are already at a ceiling level.

  Although the results of Experiment 2 cannot be used to evaluate the degradation of prosodic cues in noisy conditions, the findings remain interesting to show the role of visual cues to enhance the audio decoding process in adverse auditory conditions from the listeners' point of view. Apart from these caveats, Experiment 2 confirms that listeners also rely on the speaker's face to perceive what type of sentence is being uttered even when the speech recorded in quiet has noise added to it, giving support to the multimodal perception of the intonation of sentence types. The parallel processing of both modalities was shown in Experiment 3, as incongruent presentations degraded the performance of a clear acoustic presentation, introducing significant interferences. Congruent stimuli show accuracy ratio levels comparable to the original stimuli (audio-only/AO and audiovisual/AV), showing that the manipulation does not induce incoherencies. Incongruent presentations of auditory and visual cues show that the visual channel has an impact in the listeners judgment, even if auditory cues are still dominant for the perceptual recognition of sentence mode. The outcome of Experiment 3 confirms earlier findings regarding the process of information fusion from auditory and visual modalities to form a judgment over pragmatic meanings of sentence types such as statements and echo questions. More specifically, the results are in line with previous research using artificially constructed stimuli in BP (Peres et al., 2011), as well as in other languages, such as Dutch (Swerts & Krahmer, 2008), Catalan (Borràs-Comes & Prieto, 2011) and European Portuguese (Cruz et al., 2017).Based on these three perceptual experiments, results support previous audiovisual studies that conclude that the auditory channel can convey all the necessary information about sentence mode, while visual channel alone cannot. Yet, at the same time, the results also show that visual cues are decoded by the participants even in the presence of clear

  . The results of the perceptual identification test confirm the integration of the auditory and visual channels in signaling wh-questions and wh-exclamations meanings in BP. These positive results suggest a further investigation of the bimodal perception of the intonational contours of other sentence types. Moreover, it may be possible to go further in the understanding of audiovisual integration by monitoring a participant's reaction time, a measure that was not included in these experimental settings. As shown in the introduction of this paper, previous studies[START_REF] Borràs-Comes | Seeing tunes. The role of visual gestures in tune interpretation[END_REF] Cruz et al., 2017) already indicated that listeners' identification of sentence types is faster in audiovisual presentation conditions.Experiments analyzing participant's reaction time to audiovisual recognition of sentence types in Brazilian Portuguese are needed.

  . G.(1969). The visibility of terminal pitch contour. Journal ofSpeech and Hearing Research, 12 (2),[379][380][381][382]

  . O., Raposo deMedeiros, B., Ferreira Netto, W. & Baia, M. F. A. (2011). The role of the visual stimuli in the perception of prosody in Brazilian Portuguese. . & Pollack, I.(1954). Visual contribution to speech intelligibility in noise.Journal of Acoustical Society ofAmerica, 26, 212-215. Swerts, M. & Krahmer, E. (2008). Facial expressions and prosodic prominence: effects of modality and facial area. Journal of Phonetics, 36 (2),219-238. Torreira, F. & Valtersson, E. (2015). Phonetic and visual cues to questionhood in French.Phonetica, 72, 20-42. Tubau, S.,González-Fuente, S., Prieto, P. and Espinal, M. A. (2015). Prosody and gesture in the interpretation of yes-answers to negative yes/no-questions. The Linguistic Review, 32(1), 115-142.

Figure 1 :

 1 Figure 1: Plot of the mean F0 values (in ST) of normalized time over each modality: assertion (above) and echo question (below).

Figure 2 :

 2 Figure 2: Plot of the mean intensity (in dB, top panel) and duration (in z-score, bottom panel) over each modality: assertion (left) and echo question (right).

Figure 3 :

 3 Figure 3: Stills from a female speaker producing an assertion (above) and an echo question (below).

Figure 4 :

 4 Figure 4: Stills from a male speaker producing an echo question.

Figure 5 :

 5 Figure 5: Recognition rate of Experiments 1 and 2 with the following conditions: audio-only (AO), videoonly (VI) and audiovisual (AV) from Experiment 1; audiovisual in 0 dB SNR (AV-0), audiovisual in -6 dB SNR (AV-6), audio-only in 0 dB SNR (AO-0) and audio-only in -6 dB SNR (AO-6) from Experiment 2.

Figure 6 :

 6 Figure 6: Recognition rate of Experiments 1 and 3 with the following conditions: audio-only (AO), videoonly (VI) and audiovisual (AV) from Experiment 1; congruent audio and video (AVc) and incongruent audio and video (AVi) from Experiment 3.

  which has been shown to be able to taxonomize and label facial muscular movements made by the speakers. The following eighteen Action Units (AU, henceforth) were selected to describe the facial movements:

	head forward (AU 57), head back (AU 58) and, finally, up and down head movement
	(AU 85). Only the presence of AUs was recorded in this study; intensity was not
	measured.
	F
	o r
	P
	e
	e r
	R
	e v
	i
	e
	w
	inner brow raiser (AU 1), outer brow raiser (AU 2), brow lowerer (AU 4), upper lid raiser
	(AU 5), cheek raiser and lid compressor (AU 6), lid tightener (AU 7), lip corner depressor

(AU 15)

, chin raiser (AU 17), blink (AU 45), head turn left (AU 51), head turn right (AU 52), head up (AU 53), head down (AU 54), head tilt left (AU 55), head tilt right (AU 56),

  Experiment 2 involves 43 participants (21 in the audiovisual and 22 in the audio-only condition) with mean age of 24 years old. Differently from the previous experiment, listeners were only from Rio de Janeiro. Participants used a computer of the Acoustic Phonetics Lab at UFRJ to complete the perceptual task. They were either students or professors of Linguistics at UFRJ. Since listeners were not allowed to participate in multiple experiments, for each presentation condition, a new group of listeners was recruited. None of the participants reported hearing or sight impairments. Again, the use of headset was required for the auditory conditions of the experiment.

	F
	o r
	P
	e
	e r
	R
	e
	v i
	e
	w

https://mc.manuscriptcentral.com/las Language and Speech

Acknowledgments

We wish to thank Dr. Katy Carlson and the three anonymous reviewers for their thoughtful comments on an earlier version of the manuscript. This research has been funded by the scholarship 88881.134778/2016-01 (awarded by the Brazilian Federal Agency for Support and Evaluation of Graduate Education-CAPES) through a PhD exchange program which allowed the first author to organize a research visit to Tilburg University (NL).

https://mc.manuscriptcentral.com/las Language and Speech