Thierry Malon

Geoffrey Roman-Jimenez

Patrice Guyot

Sylvie Chambon

Vincent Charvillat

Alain Crouzil

André Péninou

Julien Pinquier

Florence Sèdes

Christine Sénac

Toulouse Campus Surveillance Dataset: scenarios, soundtracks, synchronized videos with overlapping and disjoint views

Keywords: scenarios, soundtracks, synchronized videos with overlapping and disjoint views Video and audio dataset, forensics, multi-source, synchronization, annotations, overlapping fields of view, disjoint fields of view

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Cameras used in video surveillance are of multiple types and models. Thus, they may or may not include soundtrack, GPS coordinates, timestamps, and can present significant differences in pixel resolution and image quality. Using multiple cameras for surveillance is quite unusual, and, when several cameras are used, the fields of view of the different cameras may or not overlap. As analyzing multiple videos simultaneously can bring more relevant information about the scene, there has been an increasing interest, in particular in the domain of surveillance, in developing pattern recognition tools enabling to automatically extract and to summarize all the relevant information in a user-friendly form. In order to have a better understanding of the scene, it can be useful to recognize and to track an object (human or vehicle) in different views over time. Numerous algorithms have been proposed in the literature to perform automatic object detection and tracking across a video sequence [START_REF] Bedagkar-Gala | A survey of approaches and trends in person re-identification[END_REF][START_REF] Li | Crowded scene analysis: A survey[END_REF][START_REF] Parekh | A survey on object detection and tracking methods[END_REF]. However, in the case of a scene filmed by multiple cameras across several places (streets, train stations, buildings), global object tracking remains a challenging task because of the possible multiple points of view and the configurations of the cameras (video resolution, light exposition, color). Besides, audio surveillance is a very active field [START_REF] Crocco | Audio surveillance: A systematic review[END_REF], proposing numerous algorithms for detection of single or compound audio events (glass breaking, motor noise, explosion, aggression) that could be used in complementarity with video to recognize a given object. More generally, the use of multiple sources of information (video, soundtrack, GPS locations) would help the automatic recognition over time of visual, audio or audio-visual objects and their tracking across various places.

During the last decade, public datasets become more and more available, helping for evaluation and comparison of algorithms and so, contributing to improvements of human and vehicle detection and tracking. However, most of the datasets focus on a specific task and do not allow to evaluate approaches mixing multiple sources of information. Only few datasets provide synchronized videos with overlapping fields of view and rarely provide more than 4 different views while more and more approaches can benefit from more views. Moreover soundtracks are almost never provided while they are a rich source of information as voices and motor noises can help to recognize, respectively, a person or a car.

In consequence, we produced a large dataset composed of synchronized videos of the same scene recorded from multiple viewpoints with both overlapping and non-overlapping fields of view. Soundtracks are also included. The dedicated applications of this dataset are objects detection and matching, 4D scene reconstruction, sound event detection and (meta-)data modeling and querying.

In Section 2, after presenting existing datasets, we introduce our own and conduct a comparison. We also briefly present our two detailed scenarios (scripts) implying vehicles and humans coming and going around a building. These scenarios were then played in real conditions and recorded. The way these videos were recorded and synchronized is detailed in Section 3. For each video, we provide a large amount of annotations detailed in Section 4. Finally, in Section 5, the conclusion and perspectives, we describe potential applications that are suited to be evaluated on this dataset.

EXISTING DATASETS

Our dataset stands at the intersection of multiple fields. On the one hand, image related topics cover video surveillance, multi-view, object detection, recognition and tracking, as well as action and event detection. On the other hand, audio related fields cover salient sound detection and recognition as well as multi-source processing.

To our knowledge, there exists no dataset suitable for both video and audio methods evaluation. Thus, we will review visual datasets and audio datasets separately.

Visual datasets

We only reference works that the are closest to ours. For an exhaustive survey, the reader can find details in [START_REF] Chaquet | A survey of video datasets for human action and activity recognition[END_REF]. The HumanEva dataset [START_REF] Sigal | HumanEva: Synchronized Video and Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated HumanMotion[END_REF] consists in an indoor dataset composed of 4 people moving over a scene while being filmed by 7 synchronized cameras (4 color cameras + 3 black and white cameras) with largely overlapping fields of view. People are filmed one by one and their body is fully visible. The Utrecht Multi-Person Motion (UMPM) benchmark [START_REF] Van Der Aa | UMPM benchmark: A multi-person dataset with synchronized video and motion capture data for evaluation of articulated human motion and interaction[END_REF] provides 4 color cameras with 30 persons. Video sequences are more challenging than video sequences of HumanEva dataset as they contain several persons walking at the same time and occluding each other. These two datasets are designed for articulated human motion recognition. Purely action oriented datasets can be found in Multicamera Human Action Video (MuHAVi) dataset [START_REF] Singh | MuHAVi: A Multicamera Human Action Video Dataset for the Evaluation of Action Recognition Methods[END_REF] where 14 actors are performing 17 different action classes (such as "kick", "punch", "gunshot collapse") while 8 cameras capture the indoor scene and in [START_REF] Li | Multi-Camera Action Dataset for Cross-Camera Action Recognition Benchmarking[END_REF] with 20 actors, 18 action categories divided into 4 groups (micro or intense action with or without an object), and 5 cameras with both indoor and outdoor scenes, as well as different illumination settings. Likewise, Human3.6M [START_REF] Ionescu | Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments[END_REF] contains videos where 11 actors perform 15 different classes of actions while being filmed by 4 digital cameras. Its specificity lies in the fact that 1 time-of-flight sensor and 10 motion cameras were also used to estimate and to provide the 3D pose of the actors on each frame. Both background subtraction and bounding boxes are provided at each frame. In total, more than 3.6M frames are available. In all these cases, actions are performed in unrealistic conditions as actors follow one by one a scenario consisting in performing actions one after the other.

The Video Image Retrieval and Analysis Tool (VIRAT) dataset [START_REF] Oh | A large-scale benchmark dataset for event recognition in surveillance video[END_REF] provides a large amount of surveillance videos with a pixel resolution of 1920 × 1080. In this dataset, 16 scenes were recorded for hours and, at the end, only 25 hours with significant activities were kept. Moreover, only two pairs of videos present overlapping fields of view. Moving objects have been annotated by workers with bounding boxes, as long as some buildings or areas. Three types of events are also annotated, namely single person events, person and vehicle events and person and facility events, leading to 23 classes of events. Most actions were performed by general population with minimal scripted actions, resulting in realistic scenarios with frequent incidental movers and occlusions.

The 3D People Surveillance Dataset (3DPeS) [START_REF] Baltieri | 3DPeS: 3D People Dataset for Surveillance and Forensics[END_REF] comprises 8 cameras with disjoint views and 200 different people. Each person appears, on average, in 2 views. More than 600 video sequences are available. Thus, it is well-suited for people re-identification. Cameras parameters are provided, as well as a coarse 3D reconstruction of the surveilled environment.

Audio datasets

The AudioSet dataset [START_REF] Gemmeke | Audio set: An ontology and human-labeled dataset for audio events[END_REF] consists of YouTube videos annotated in audio events. This dataset holds numerous video references (2 million of 10s videos) and a wide range of audio classes (632). However, the audio events are not temporally annotated. Moreover, the dataset contains misinterpreted sounds resulting in sounds labeled with a wrong class, for example a "cracking whip" labeled as "gunshot, gunfire". Figure 1: A subset of all the synchronized videos for a particular frame of the first scenario. First row: cameras located in front of the building. Second and third rows: cameras that face the car park. A car is circled in red to highlight the largely overlapping fields of view. When a frame is missing in a video, it is replaced with a black screen (camera 24).

Other datasets contain only audio files. The Freesound project [START_REF] Font | Freesound technical demo[END_REF] holds nearly 400 thousand of recordings updated by users with tags. Even if several datasets are hosted on Freesound [START_REF] Kons | Audio event classification using deep neural networks[END_REF][START_REF] Salamon | A Dataset and Taxonomy for Urban Sound Research[END_REF], most of the sounds are not annotated. The Urbansound dataset [START_REF] Salamon | A Dataset and Taxonomy for Urban Sound Research[END_REF] is composed of 1302 audio files of complex field recordings that are temporally annotated.

Finally, to our knowledge, there is no audio dataset which is specifically dedicated to surveillance applications. Moreover, the existing corpora are either poorly annotated, either unrealistic or too complex. Therefore, in the case of a realistic sound mixture, the acoustic properties of the sound are hard to model [START_REF] Gloaguen | Creation of a corpus of realistic urban sound scenes with controlled acoustic properties[END_REF]. The use of audio data from different recording points offers unprecedented opportunities for a relevant pattern of audio events.

Limitations of video surveillance datasets

Existing multi-view surveillance datasets are often limited for several of the following reasons:

(1) unrealistic actions due to acting in constrained scenes, (2) few or no overlapping in the fields of view of the cameras, (3) lack of diversity in terms of events or objects, (4) lack of ground truth annotations, (5) absence of audio track, (6) poor pixel resolution or image quality, (7) composed of only short videos consisting of a single action, [START_REF] Gloaguen | Creation of a corpus of realistic urban sound scenes with controlled acoustic properties[END_REF] poor or no synchronization between the different viewpoints.

These limitations make it difficult to combine different approaches on a single dataset. Table 1 compares our dataset to others that we have found being the closest accorded to the intended applications. The characteristics used in this comparison are, in particular, the pixel resolution, the number of overlapping and disjoint views, the audio track availability, the number of objects to be detected and the realism of the scene in the context of surveillance. In the next section, we will develop all the aspects of this new dataset.

PROPOSED TOCADA DATASET

The Toulouse Campus surveillance Dataset, named ToCaDa, contains two sets of 25 temporally synchronized videos corresponding to two scripted scenarios. Figure 1 shows a subset of the 25 views of the same frame. With the help of about 50 persons (actors and camera holders), these videos were shot on July 17th 2017 at 9:50 a.m. and 11:04 a.m. respectively. Among the cameras:

• 9 were located inside the main building and shot from the windows at different floors. All these cameras are focusing the car park and the path leading to the main entrance of the building with large overlapping fields of view. • 8 were located in front of the building and filmed it with large overlapping fields of view too (these 9+8=17 overlapping view cameras can be seen on Figure 2). • 8 cameras were arranged further, scattered around the university campus, see Figure 3. Each of their views is disjoint from all the others.

About 20 actors were asked to follow two realistic scenarios by performing scripted actions, like driving a car, walking, entering or leaving a building, or holding an item in hand while being filmed. In addition to ordinary actions, some suspicious behaviors are present. More precisely:

• In the first scenario, a suspect car (C) with two men inside (D the driver and P the passenger) arrives and parks in front of the main building (at the sight of the cameras with overlapping views). P gets off of the car C and enters the building. Two minutes later, P leaves the building holding a packet and gets in C. C leaves the parking and gets away from the university campus (passing in front of some of the disjoint fields of view cameras). • In the second scenario starts similarly with a suspect car (C) and two men inside (D the driver and P the passenger) which arrives and parks in front of the main building (again To take full advantage of the videos, it seems helpful to temporally synchronize them so that, for a given time, all the frames of the different cameras match.

Camera holders used their own mobile devices to record the scene, leading to a large variety of resolutions, image quality, frame rates and video duration. In order to coordinate this heterogeneous disposal, three foghorns were blown:

(1) The first one stands for a warning 20 seconds before the start, to let enough time to start shooting. (2) The second one is the actual starting time, useful to temporally synchronize the videos. (3) The third one indicates the ending time. All the videos were collected and were manually synchronized using the second and the third foghorn blows as starting and ending times. Indeed, the second one can be heard at the beginning of every video. All the videos of scenario 1 last 4:48 and all the videos of scenario 2 last 5:40. We used kdenlive 1 to cut the original videos and to produce the synchronized ones. Due to the wide variety of devices used during the shooting of the two scenarios, issues were encountered on some cameras, leading to videos where a few seconds are lacking. To ensure temporal synchronization between videos, black frames were added on the missing intervals of time as it can be seen with camera 24 on Figure 1.

Limits of our dataset are the following. Few simultaneous actions: there are rarely more than two actions occurring at the same time. People are not very challenging to detect and do not appear in groups.

Few salient sound events: our videos do not contain screams, horns, explosions or gun shots. Sound events are mainly categorized as motor type sounds.

The video files with audio soundtracks, the detailed scenarios with the list of actions and corresponding times, the files containing ground truth annotations (that will be detailed in the next section) and the list of the irregularities concerning the videos (lacking times or different pixel resolution) can be found on the following link: http://ubee.enseeiht.fr/dokuwiki/doku.php?id=public:tocada. Additional soundtracks recorded with specific audio devices are also provided.

ANNOTATIONS

Ground truth annotations are stored in json files. Each file corresponds to a video and shares the same title but not the same extension, namely <video_name>.mp4 annotations are stored in <video_name>.json. Both visual and audio annotations are stored in each file.

By annotating, our goal is to detect the visual objects and the salient sound events and, when possible, to associate them. Thus, we have grouped them into the generic term audio-visual object. This way, the appearance of a vehicle and its motor sound will constitute a single coherent audio-visual object and is associated to a same ID. An object that can be seen but can not heard is also an audio-visual object but with only a visual component, and similarly for an object that can only be heard.

We have developed a program for navigating through the frames of the synchronized videos and for identifying audio-visual objects by drawing bounding boxes at particular frames and/or specifying starting and ending times of a salient sound. Each audio-visual object is associated to a unique ID. Regarding bounding boxes, the coordinates of top-left and bottom-right corners of the bounding boxes are given, as in Figure 4. Bounding boxes were drawn such that the object is fully contained inside the box and as tight as possible. For this purpose, our annotation tool allows the user to draw an initial approximate bounding box and then to adjust its boundaries at a pixel-level. As drawing one bounding box for each object on every frame requires a huge amount of time, we have drawn bounding boxes on a subset of frames, so that the intermediate bounding boxes of an object can be linearly interpolated using its previous and next drawn bounding boxes. In average, we have drawn one bounding box per second for humans and two for vehicles due to their speed variation. For objects with irregular speed or trajectory, we have drawn more bounding boxes.

Each bounding box also contains a flag "fully_visible" set to 0 if the object is occluded, even partially, and set to 1 otherwise. On a given frame, an object can be mostly occluded but still present, in particular when it is located near a border of the video. In such case, drawing a bounding box is not relevant. Also, an object can appear and can disappear several times. Instead of considering it as several different objects, we have defined a list visible_times that contains the time segments (defined by a tuple of times "from" and "to") between which an object becomes visible (even partially) and totally disappears from the screen. The visual category of each object is given, consisting in a general term (human, bike, car), along with a list of details like gender and clothes in the case of a human. An example of the json structure of the visual component of an object in a particular video is given in Listing 1.

Regarding the audio component of an audio-visual object, namely the salient sound events, an ID is also given. If the sound comes from an object whose visual component was annotated, the visual and audio IDs have to be the same. An audio category (voice, motor sound) is also given, as long as a list of details and time bounds. In case of an object presenting different sound categories (a car with door slams, music and motor sound for example), one object is created for each category and the same ID is given. In Listing 2, an example of the audio component corresponding to the visual component from Listing 1 is given. Listing 1: json file structure of the visual component of an object in a video, visible from 13.8s to 18.2s and from 29.72s to 32.28s and associated with id 11.

{

"id": 11, "category": "MOTOR", "details": ["motorbike", "very noisy"], "audible_times" : [ { "from": 11.24, "to": 21.32 } ], } Listing 2: json file structure of an audio event in a given video. As it is associated to id 11, it corresponds to the same audio-visual object as the one of Listing 1.

CONCLUSIONS AND PERSPECTIVES

We have introduced a new dataset composed of two sets of 25 synchronized videos of the same scene with 17 overlapping views and 8 disjoint views. Videos are provided with their associated soundtracks. We have annotated the videos by manually drawing bounding boxes on moving objects. We have also manually annotated audio events. Our dataset offers simultaneously a large number of both overlapping and disjoint synchronized views and a realistic environment. It also provides audio tracks with sound events, high pixel resolution and ground truth annotations. The originality and the richness of this dataset come from the wide diversity of topics it covers and the presence of scripted and nonscripted actions and events. Therefore, our dataset is well-suited for numerous pattern recognition applications related to, but not restricted to, the domain of surveillance. We describe below, some multidisciplinary applications that could be evaluated using this dataset:

3D and 4D reconstruction: the multiple cameras sharing overlapping fields of view along with some provided photographs of the scene allow to perform a 3D reconstruction of the static parts of the scene, see Figure 5, and to retrieve intrinsic parameters and poses of the cameras using a Structure-from-Motion algorithm [START_REF] Agarwal | Building Rome in a day[END_REF]. Beyond a 3D reconstruction, the temporal synchronization of the videos could enable to render dynamic parts of the scene as well and to obtain a 4D reconstruction. Object recognition and consistent labeling: evaluation of algorithms for human and vehicle detection and consistent labeling across multiple views can be performed using the annotated bounding boxes and IDs. To this end, overlapping views provide a 3D environment that could help to infer the label of an object in a video knowing its position and label in an other video.

Sound event recognition: the audio events recorded from different points and manually annotated provide opportunities to evaluate the relevance of consistent acoustic models by, for example, launching the identification and indexing of a specific sound event. Looking for a particular sound by similarity is also feasible.

(Meta-)data modeling and querying: the multiple layers of information of this dataset, both low-level (audio/video signal) and high-level (semantic data available in the ground truth files) enable to handle information at different resolutions of space and time, allowing performing queries from heterogeneous information.

Figure 2 :

 2 Figure 2: The main building which concentrates 17 cameras with overlapping fields of view. 8 other cameras are located out of this field of view, see Figure 3.

  1 https://kdenlive.org

Figure 3 :

 3 Figure 3: The positions of the different cameras within a parcel of Toulouse Campus (Paul Sabatier site). These 8 cameras have disjoint views. The red area corresponds to Figure 2.

Figure 4 :

 4 Figure 4: Coordinates of the bounding boxes top-left and bottom-right corners are given according to this reference.

Figure 5 :

 5 Figure 5: 3D reconstruction of the main building from the overlapping views using Structure-from-Motion [1].

Table 1 :

 1 Comparison of the properties of several datasets.

	Soundtrack	No	No	No	No	Yes
	# of microphones 0	0	0	0	25+2
	Overlapping FOV Very partially	2+2	8	4	17
	Disjoint FOV	8	12	0	0	4
	Synchronized	No	No	Partially	Yes	Yes
	Pixel resolution	704 × 576	1920 × 1080	720 × 576	1000 × 1000	Mostly 1920 × 1080
	# visual objects	200	Hundreds (N/A)	14	11	30
	# action types	0	23	17	15	0
	# bounding boxes 0	≈ 1/object/second	0	≈ 1/object/frame	≈ 1/object/second
	In/Outdoor	Outdoor	Outdoor	Indoor	Indoor	Outdoor
	With scenario			×	×	×
	Realistic	×	×			×


Lorenzo Vianello
email: lorenzo.vianello@univ-lorraine.fr

Waldez Gomes
email: waldez@ieee.org

Freek Stulp
email: freek.stulp@dlr.de

Alexis Aubry
email: alexis.aubry@univ-lorraine.fr

Pauline Maurice
email: pauline.maurice@loria.fr

Serena Ivaldi
email: serena.ivaldi@inria.fr

Latent ergonomics maps: Real

Keywords: time visualization of estimated ergonomics of human movements ergonomics tools, digital human model, wearable sensors

come

Introduction

Poor ergonomics conditions in work environments may lead to serious work-related musculoskeletal disorders (WMSDs), including severe disabilities [START_REF] Vos | Global, regional, and national incidence, prevalence, and years lived with disability for 301 acute and chronic diseases and injuries in 188 countries, 1990-2013: A systematic analysis for the Global Burden of Disease Study 2013[END_REF]. The development of WMSDs is an issue not only for the workers' health and well-being but also represents an important cost for companies and society [START_REF] Punnett | Work-related musculoskeletal disorders: the epidemiologic evidence and the debate[END_REF][START_REF] Schneider | Occupational Safety and Health in Figures: Work-Related Musculoskeletal Disorders in the EU-Facts and Figures[END_REF]. In recent years, there has been a surge in robotic solutions for ergonomics interventions, notably using industrial manipulators conceived for collaboration with humans (i.e., cobots) and exoskeletons [START_REF] Shafti | Real-time robot-assisted ergonomics[END_REF][START_REF] Spaa | Predicting and optimizing ergonomics in physical human-robot cooperation tasks[END_REF]. These robotics solutions require ergonomics specialists to identify dangerous conditions and develop adequate interventions, whilst maintaining operational safety and productivity.

Classic kinematics ergonomics evaluation tools such as RULA, REBA and OWAS [START_REF] Mcatamney | RULA: a survey method for the investigation of work-related upper limb disorders[END_REF][START_REF] Hignett | Rapid entire body assessment (REBA)[END_REF][START_REF] Karhu | Correcting working postures in industry: a practical method for analysis[END_REF] use human joint positions to produce an ergonomics score for a given body posture. Kinematics-based scores are fast to compute, but dynamics aspects of the task may not be negligible [START_REF] Busch | Postural optimization for an ergonomic human-robot interaction[END_REF]. Dynamics estimation may be an important complementary evaluation to the classic tools throughout an entire task execution, as they are more suitable for evaluating more accurately varied body morphologies, and external wrenches applied to the human body. Many recent works evaluate dynamic aspects of the task execution, such as internal and external human wrenches [START_REF] Peternel | Towards ergonomic control of human-robot co-manipulation and handover[END_REF][START_REF] Kim | Anticipatory robot assistance for the prevention of human static joint overloading in human-robot collaboration[END_REF][START_REF] Maurice | Assessing and improving human movements using sensitivity analysis and digital human simulation[END_REF][START_REF] Latella | Whole-body human inverse dynamics with distributed micro-accelerometers, gyros and force sensing[END_REF].

Recently, there has been much attention on improving the intuitiveness of ergonomics evaluation tools, as industrial operators should not be expected to have a background in ergonomics. Previous works have used digital human models (DHMs) alongside different types of visual cues for ergonomics evaluation: visualization of the human DHM with colored joints [START_REF] Kim | Adaptable workstations for human-robot collaboration: A reconfigurable framework for improving worker ergonomics and productivity[END_REF] or displaying relevant information such as COP [START_REF] Kim | A real-time and reduced-complexity approach to the detection and monitoring of static joint overloading in humans[END_REF], overloaded joint torque [START_REF] Kim | Anticipatory robot assistance for the prevention of human static joint overloading in human-robot collaboration[END_REF], level of fatigue [START_REF] Peternel | Robot adaptation to human physical fatigue in human-robot comanipulation[END_REF].

The data relevant to assessing ergonomics is high-dimensional, including kinematic and dynamic state variables related to posture and efforts. This high-dimensional data is difficult to interpret, even for experts. Physiological sensors such as surface EMG or EKG often measure critical information for ergonomics assessment, but they require considerable post-processing and are difficult to interpret promptly. They are mostly used for post-experimental analysis [START_REF] Maurice | Objective and Subjective Effects of a Passive Exoskeleton on Overhead Work[END_REF], while few applications of online use in human-robot collaboration exist [START_REF] Peternel | Robot adaptation to human physical fatigue in human-robot comanipulation[END_REF]. Also, the dimensionality of the data may become a bottleneck real-time computation and interpretation. For this reason, many works proposed to reduce the dimensionality of a data set, reducing it to a set of representative principal features [START_REF] Mandery | Dimensionality reduction for whole-body human motion recognition[END_REF]. The reduced human representation has been used to execute for instance activity recognition and prediction [START_REF] Chaveroche | A Variational Time Series Feature Extractor for Action Prediction[END_REF] or ergonomic optimization [START_REF] Marin | Optimizing contextual ergonomics models in human-robot interaction[END_REF].

Here, we focus on providing ergonomics visualization tools that are intuitive for most users-experts or otherwise-by using state-of-the-art methods for dimensionality reduction. We present a novel and potentially easy-to-use visualization tool for ergonomics assessment: Latent Ergonomics Maps (LEM). To construct the LEM, we project a given ergonomics score onto a 2D latent space that represents the human posture. In our previous work, we proposed to use latent representations to capture the human movement/posture [START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF].

Here, we extend the use of latent maps to encode ergonomics evaluation scores.

In this article, we present two supportive tools for the evaluation of strenuous efforts and postures:

•

Online visualization of joint angles and torques for ergonomic feedback. Given a human posture, the framework calculates the ergonomic assessment and an estimation of the human effort. This latter estimation is derived from inversion of the Lagrangian model using variables (e.g., Intertia, Coriolis) extracted from a simulated DHM. This enables to quickly verify a body posture, captured online using a motion capture suit. This visualization consists of a DHM with color-coded visual cues that express specific locations and joints of the body postures that are particularly non-ergonomic, further facilitating the ergonomics assessment.

•

Latent Ergonomic Maps(LEMs) for immediate overall 2D visual feedback on RULA and RULA-based (RULA-continuous) scores. The algorithm uses a state-of-the-art method for dimensionality reduction and generative network, namely Variational Auto-Encoders (VAE). VAE allows us to encode high dimension postures and to sample and decode variations of the same postures. The latter allows creating a LEM by sampling the latent space, decoding the posture, and applying ergonomic assessment to the posture.

We demonstrate the feasibility of the proposed frameworks in two different scenarios: first, we demonstrate the feasibility of our visualization tool for typical industrial activities, such as walking, bending, and overhead manipulation; second, we showcase our tool during a physical human-robot interaction.

Related Work

Dimensionality Reduction for Human State Representation

DHMs typically have many degrees of freedom. A depiction of the human status, therefore, requires high-dimensional vectors, which in turn increase the computational cost of the ergonomics status evaluation [START_REF] Malaisé | Activity Recognition for Ergonomics Assessment of Industrial Tasks With Automatic Feature Selection[END_REF]. To reduce the dimensionality of a data set, a solution is to extract a reduced set of representative principal features from the original data set [START_REF] Ghojogh | Factor analysis, probabilistic principal component analysis, variational inference, and variational autoencoder: Tutorial and survey[END_REF]. This procedure is usually applied to reduce the amount of data necessary to train a model and to reduce the risk of overfitting during the training [START_REF] Dilokthanakul | Deep unsupervised clustering with gaussian mixture variational autoencoders[END_REF]. Recent works have applied this concept to compactly represent human motion. For instance, Mandery et al. [START_REF] Mandery | Dimensionality reduction for whole-body human motion recognition[END_REF] were able to reach high accuracy in a human motion classification task using only four features. Human models with a high number of degrees of freedom impact the time performance of motion analysis. To improve speed in the motion processing of the DHM data, recent works have used a representation of the human motion in a latent space [START_REF] Marin | Optimizing contextual ergonomics models in human-robot interaction[END_REF][START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF]. Marin et al. [START_REF] Marin | Optimizing contextual ergonomics models in human-robot interaction[END_REF] showed that the use of a latent space human representation improved the performance of their application. Ikemoto et al. [START_REF] Ikemoto | Physical interaction learning: Behavior adaptation in cooperative human-robot tasks involving physical contact[END_REF] use principal components analysis (PCA) to reduce the dimensionality of the postures of a humanoid robot.

Dimensionality reduction techniques applied to human poses are also used in movement planning or prediction [START_REF] Liu | Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction[END_REF]. This type of approach uses latent space representations for selecting points that complete a given path (movement prediction) or for joining two extremes (movement planning). By inverting the relationship which maps the high-order dimensional to the latent space, it is possible to recover the poses having the latent space representations. For instance, variational autoencoders have been used to predict the next pose given the current one [START_REF] Ling | Character controllers using motion vaes[END_REF][START_REF] Li | Task-generic hierarchical human motion prior using vaes[END_REF].

Among dimensionality reduction techniques, autoencoders (AEs) and variational-AEs (VAEs) have high reconstruction ability and produce compact latent spaces. For this reason, they have been widely used for reducing the dimensionality of the human state and for movement generation [START_REF] Chaveroche | A Variational Time Series Feature Extractor for Action Prediction[END_REF]. Dermy et al. [START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF] address the problem of predicting future human whole-body movements given prior observations. They map high-dimensionalality trajectories into a reduced latent space using AE. Then the prediction is based on a probabilistic description of the movement primitives in the latent space, which reduces the computational time for the prediction to occur. In our work, we proposed a method for visualizing the ergonomic scores in a latent space obtained using VAE.

Ergonomics Evaluation and Human-Robot Collaboration

To plan appropriate collaborative actions, collaborative robots need to have an estimation of the current human posture [START_REF] Marin | Optimizing contextual ergonomics models in human-robot interaction[END_REF][START_REF] Meng | Dynamic Projection of Human Motion for Safe and Efficient Human-Robot Collaboration[END_REF]. They must be able to calculate the physical, physiological, and/or cognitive state of the human to act accordingly. The perception of the human state relies on sensors that can be placed in the environment, embedded in the robot, or worn by the human. State-of-the-art motion capture techniques based on infrared cameras and reflective markers, or more recently on inertial technology, remain widely used to provide high-fidelity and high-frequency measurements of human kinematics [START_REF] Maurice | Human movement and ergonomics: An industry-oriented dataset for collaborative robotics[END_REF]. The captured data are fitted to a DHM designed to have similar properties to the human operator (height, weight, and structure). Ergonomics scores typically rely on kinematics and dynamics information about the human's movement, which are extracted from the DHM. There are two main types of DHMs: musculoskeletal models, they have many degrees of freedom, and allow the analysis of the human movement by simulating the muscular efforts [START_REF] Bassani | Validation of the AnyBody full body musculoskeletal model in computing lumbar spine loads at L4L5 level[END_REF]; rigid-body models, which are simplified models with fewer degrees of freedom, where the human is represented as made of rigid body links [START_REF] Maurice | Digital human modeling for collaborative robotics[END_REF]. While musculoskeletal models can be very accurate on a biomechanical standpoint, less accurate rigid-body models are much faster to simulate. As such, they are better suited for real-time applications such as model-based prediction, control, and ergonomics assessment [START_REF] Maurice | Digital human modeling for collaborative robotics[END_REF].

The need to reduce musculoskeletal disorders in the industry leads to extensive research into measures of ergonomics. Many ergonomics assessment tools use the kinematic of a given worker to evaluate, and score a given task execution [START_REF] Kong | Comparisons of ergonomic evaluation tools (ALLA, RULA, REBA and OWAS) for farm work[END_REF]. For instance, RULA [START_REF] Mcatamney | RULA: a survey method for the investigation of work-related upper limb disorders[END_REF] and REBA [START_REF] Hignett | Rapid entire body assessment (REBA)[END_REF] evaluate the upper-body and full-body posture by scoring how far the worker's current joint angle positions are from a neutral and safe position. Some works have also used assessments typically used in robotics, such as manipulability measures that can be associated with the user's operational comfort at executing the task [START_REF] Doty | Robot manipulability[END_REF][START_REF] Tanaka | Analysis of operational comfort in manual tasks using human force manipulability measure[END_REF].

Postural-based approaches, however, do not consider the dynamics properties of the human model nor any dynamics interactions with the environment. In many ergonomics scoring sheets, external forces and loads can be considered (e.g., in EAWS it is possible to account for manipulated weights), but a more general approach considering efforts due to physical interaction with loads and robots is required to inform the human. For this reason, many works in human-robot collaboration use DHMs to estimate the status of human dynamics. Latella et al. [START_REF] Latella | Whole-body human inverse dynamics with distributed micro-accelerometers, gyros and force sensing[END_REF][START_REF] Latella | Towards real-time whole-body human dynamics estimation through probabilistic sensor fusion algorithms[END_REF] estimate the balance, and the internal force distribution of the human during a human-robot collaboration using F/T sensors at the robot's end-effector, force plates at the feet, and/or sensorized insoles. Peternel et al. [START_REF] Peternel | Towards ergonomic control of human-robot co-manipulation and handover[END_REF] estimate the human joint overloading torque, a quantification of the effect an external load has on a given body joint. Maurice et al. [START_REF] Maurice | Assessing and improving human movements using sensitivity analysis and digital human simulation[END_REF] estimate the human body joint torques by solving an optimization problem where the objective function encodes the human posture and the optimization variables include torques and wrenches of the human model.

In this article, we present intuitive visualization tools to provide online feedback to industrial operators about the ergonomics of their movements. In the next sections, we present the methods and technologies used to build our tools (Section 3) and then discuss how to use them in two experimental scenarios inspired by industrial activities, where the human is executing different activities with challenging whole-body postures, with or without the assistance of a robot (Section 4).

Methods

The main goal of this paper is to develop intuitive visualization tools to provide online ergonomic feedback to industrial operators. The system is designed to be applicable to diverse industrial scenarios in which a human operator may execute non-ergonomic movements, either working alone, or in interaction with a robot. The intuitive visualization tools enable the human to easily visualize the risks of their posture even if they are not ergonomic experts. The human operator could, visualize the latent maps and the DHM along with ergonomics suggestions on a display during a training session, where time can be dedicated to analyzing and improving the gestures and the postures associated with a workstation. Looking at the display is hardly possible during the regular work because the display could divert the attention of the worker and increase the risk of accidents; however, the display can be used for post-hoc analysis immediately after a gesture if the production time allows for it. The visual feedback could be also displayed to the worker using augmented reality glasses [START_REF] Nee | Augmented reality applications in design and manufacturing[END_REF]. Interestingly, the visual feedback can be used directly by the human worker, as well as by colleagues and ergonomics experts that evaluate the worker's activity at a workstation.

Here, we project the ergonomic information into a 2-dimensional latent space that is learned to represent the human posture. In previous work, we used latent representations to capture the human movement/posture [START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF]. Moreover, we showed they could be used to predict future whole-body movements and optimize movements to improve ergonomics in specific contexts [START_REF] Marin | Optimizing contextual ergonomics models in human-robot interaction[END_REF]. Here, the latent space is used to build a map that is used to display ergonomics information: we train the latent space to maximize the human posture representation and the ergonomics characterization of the space. The result is an intuitive visualization tool, called Latent Ergonomics Map (LEM), which provides a synthetic representation of the ergonomics of motions. The LEM is coupled with a real-time simulation of the human using a DHM that displays localized ergonomics information using colored spheres placed in relevant body parts. Both the LEM and the simulation are updated online, with the information coming from the human motion tracking system and eventually with the robot's data if the human is working in interaction with a robot. The ergonomics visualization tools are shown in Figure 1. The only requirement of our system is that it requires a real-time human motion tracking system like wearable or environmental sensors. In this work, the human posture is tracked using an Xsens MVN motion tracking suit. Other devices could be used, but reviewing them is beyond the scope of this article.

Digital Human Model

Our DHM is a rigid multi-body system, similar to a humanoid robot, with anthropometrics properties of the human (height, weight) to mimic their kinematic and dynamics. It has 66 segments and is based on the Xsens MVN model (Figure 2). Using such a model, it is possible to reproduce the human movement as recorded by the Xsens MVN motion tracking system, a wearable set of 17 sensors distributed along the human body. We modeled the human spherical joints collected by the motion capture suit as a series of 3 onedimensional revolute joints, where each DoF is controlled by a single actuator. The resulting DHM posture is represented by the 66 joints: the links are modeled with geometric shapes (parallelepiped, cylinder, sphere) scaled with the human height. The dynamic properties (e.g., mass) are computed from anthropometric data available in the literature [START_REF] Maurice | Digital human modeling for collaborative robotics[END_REF], assuming a homogeneous density of the links and scaling with the human body mass. The posture of the DHM is calculated using a one-to-one mapping from the original human poses, which are measured in our case by a human skeleton or motion tracking system. The position of the floating base of the DHM is calculated using the position of the pelvis, which is provided by the motion tracking system.

The DHM enables the estimation of the human internal torques, as described in previous work [START_REF] Maurice | Assessing and improving human movements using sensitivity analysis and digital human simulation[END_REF]. However, here we do not use force plates to retrieve feet wrenches: instead, we approximate the wrench acting on the human feet with the weight force generated by the body. This approximation introduces an error in the joint torque estimation, but it is tolerable for our visualization purposes as long as the joint torques are not used in a control loop. This allows transporting easily the framework while having an issue with the movement quality, especially for upper-body movements.

The internal torques are calculated starting from the classical Lagrangian formulation:

M( q) q + C( q, q) q + G( q) = S τ + n k ∑ i=1 J p i ( q) T f i (1)
where q = (x F , q) is the DHM extended posture composed by the floating base pose (x F ∈ R 6 ) and the joints angles (q ∈ R 66 ), M, C, G are respectively the inertia, the Coriolis and the gravity matrices, S is the actuation selection matrix due to the free-floating base and τ is the vector of the DHM joint torques. Concerning the computation of the external wrenches acting on the ith link ( f i ), we consider the force sensed by the robot through the torque sensors ( f RH = J F (q)τ F,ext , where J F is the robot Franka's Jacobian matrix and τ F,ext are the robot external torques) and the gravitational forces acting on the feet. The external wrenches acting on the ith link are multiplied by the Jacobian matrix from the world frame to the i-th link frame (J p i ). The choice of using only the gravitational forces acting on the feet helps to improve the speed of the algorithm and facilitates the deployment of our tools for ergonomics feedback to situations where only the human skeleton information is accessible. We solve the Equation (1) for τ.

Ergonomics Scores

To estimate the motion ergonomics, we use scores based on kinematics information (RULA, RULA-C, manipulability) and dynamics information (joint torques). The Rapid Upper Limb Assessment (RULA) tool [START_REF] Mcatamney | RULA: a survey method for the investigation of work-related upper limb disorders[END_REF] is often used by ergonomists to evaluate work activities involving upper-body motion. It consists of a score ranging from 1 to 7, calculated based on the joint positions (posture), the known force/load applied to the worker's arm, and how many times the activity is repeated. The time evolution of RULA during a work activity is likely to have discontinuities and plateaus that make it inconvenient to use for motion optimization or continuous postural assessment. To alleviate this problem, we define a continuous version of RULA: RULA-C (ε rc ∈ R + ). To compute RULA-C, we fit a second-degree polynomial function to calculate intermediate scores for the RULA joints. The joint scores for each limb are combined with weighted sums, where the weights are computed from linear regressions of the standard RULA tables. To account for the comfort of movement, especially in presence of physical interaction, we consider the arm manipulability: it provides information about the velocity and force production capacity of the limb endpoint in different configurations [START_REF] Peternel | Towards ergonomic control of human-robot co-manipulation and handover[END_REF]. It provides a piece of complementary information about the human posture to RULA and RULA-C about the capacity of the human to produce forces.

Finally, we compute human joint torques using DHM, to account for the postural efforts and the effect of external wrenches acting on the human. To display the ergonomic assessments intuitively (especially targeting end-users that do not have a comprehensive background in ergonomics), we choose to visualize the local ergonomics scores in colorcoded spheres attached to relevant body parts in the DHM, updated at each time step. The mapping from the ergonomic score to the colors follows existing recommendations when available: green is associated with low ergonomics scores and indicates low risks, red with high scores indicating high risks, and yellow with scores in between the two.

Each ergonomic score (kinematic and dynamic) can be normalized between zero and one by using the maximum value for the specific task, computed in an offline calibration phase. Normalizing the values enables to highlight for a specific task the differences in the ergonomics while executing the same movement in different ways. We are aware that this normalization has some limitations (e.g., encountering new movements that are particularly risky). In future work, we want to overcome these limitations and integrate activity recognition [START_REF] Malaisé | Activity recognition with multiple wearable sensors for industrial applications[END_REF] into our framework, to change normalization values according to the current activity and movement.

Latent Ergonomics Maps (LEMs)

Latent Ergonomics Maps (LEMs) project an ergonomics score on a 2-dimensional latent space that is trained to represent the human posture (note that this means that each ergonomics score listed in Section 3.2 leads to a different map). Previous work showed the advantages of reducing the dimensionality of the human postural state (Section 2.1). In particular, Dermy et al. [START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF] showed that low-dimensional (2 to 7) latent representations of a human posture are sufficient for probabilistic models computed over the latent space to predict future whole-body movements with accuracy for activity recognition and postural evaluation. Malaisé [START_REF] Malaisé | Activity Recognition for Ergonomics Assessment of Industrial Tasks With Automatic Feature Selection[END_REF] showed that a 2D latent space is sufficient for activity prediction and to some extent for postural evaluation. In this article, we propose a synthetic representation of the human postures using a 2D latent space created by a Variational Auto-Encoder, and then we project ergonomics scores of sampled human postures on the latent space, in the form of a map. The resulting map is the Latent Ergonomics Map. This representation allows us to visualize in an immediate way when a person is in a non-ergonomic (associated with red color) or ergonomic (associated with green color) posture. In future work, we would like to use the presented space to provide corrections to humans and plan better ergonomically optimized trajectories.

The LEM is created in an offline phase, requiring a dataset of human movements and the definition of the ergonomics score. Training a LEM proceeds in two steps. The first step is to create and train a Variational Auto-Encoder (VAE) to represent the human postures. A schematic of the training procedure is shown Figure 3. A VAE is an autoencoder based on variational inference [START_REF] Hong | How generative adversarial networks and their variants work: An overview[END_REF]. Let [x k ] K k=1 be a data-set of K independent and identically distributed samples of some continuous observation variable x of unknown distribution. It is assumed that x is generated by some process involving the latent variable z and the parametric functions of distribution p θ * (x|z) and p θ * (z): x ∼ z p θ * (x|z)p θ * (z)dz where θ * is a set of parameters. In a VAE, the so-called decoder neural network tries to fit the function which maps z to x, and so learns from data the values of the weights θ. The distribution is assumed to be Gaussian: p θ (z) = N (0, I) and p θ (x|z) = N (µ x , σ 2

x ). However, the transition function p θ (z|x) is not known. A recognition model q φ (z|x) is used to approximate true posterior p θ (z|x). In our case the distribution is also assumed to be Gaussian for simplicity, but without loss of generality: q φ (z|x) = N (µ z , σ 2 z ), where φ is represented by weights and biases of a neural network (encoder). Training the VAE aims at recovering values of the parameters (θ, φ) in such a way to approximate as much as possible the optimal parameters (θ * , φ * ).

We choose a relational VAE to define a loss function which represents both the reconstruction loss and a loss on the relation reconstruction [START_REF] Meng | Relational autoencoder for feature extraction[END_REF]:

L = (1 -α)D kl (q φ (z|x)||p θ (x|z)) + αD kl (q φ (z|ε(x))||p θ (ε(x)|z)) (2)
where D kl (.) is the Kullback-Leibler divergence, ε(.) is the relation function (in our work corresponding to the ergonomic loss function RULA-C) and, finally, the parameter α is a scale parameter to control the weights of the data reconstruction loss and the relationship reconstruction loss. This enables us to obtain a latent space that not only represents the postures, but also, tries to encode the space to minimize the error in reconstructing the correct ergonomics score for the compressed posture. Training the VAE requires a dataset of human movements. For example, the AnDy dataset [START_REF] Maurice | Human movement and ergonomics: An industry-oriented dataset for collaborative robotics[END_REF] contains many examples of human activities, recorded with the Xsens motion tracking suit, which can provide both wearable sensing data and estimated postural data. To simplify, let us assume that we input human postures fitted to the DHM (R 66 ). We train the VAE with batches of human postures from the training dataset to find the parameters of the encoder-decoder networks, to minimize the loss function Equation (2) via backpropagation.

Once the VAE is trained, the second step consists in creating the ergonomics landscape projected on the latent map. The procedure is illustrated in Figure 4. We uniformly sample in the latent space a set of 2D vectors (z i,j = [z i , z j ] ∈ R 2 ). We reconstruct the samples using the decoder. Then, for each reconstructed human posture (q i,j ∈ R 66 ) we calculate the ergonomic score ε(q i,j ) (e.g., RULA or RULA-C) from the estimated human joint angles, following the method presented in Section 2.2. Note that one map is created for each ergonomics score (for instance, one map for the standard RULA score, and another map for the RULA-C score). If the data used for the training are subject to a normalization function ( f n (.)), it is necessary to invert the normalization (ε( f -1 n (q i,j ))) before calculating the ergonomic scores. The result is a height-map composed by the 2D coordinated in the latent space and the ergonomic score ([z i , z j , ε(q i,j )] ∈ R 3 ).

The LEM can be used online to display the movement that the human operator is executing, which appears as a trace on the landscape: since every point is a posture, the height of each point indicates the ergonomics evaluation associated with the corresponding human posture. The human posture is recorded with wearable sensors (e.g., in our case the Xsens MVN motion tracking suit) and retargeted into the DHM (q c ∈ R 66 ). The encoder neural network reduces the input to its 2D representation (z i,j ), the ergonomics score is then retrieved. We adopt a triadic color code to make the height map more intuitive to read: we associate low ergonomic scores (e.g., ergonomic postures, with low risk) with green; high ergonomic scores (e.g., non-ergonomic postures, with high risk) are associated with red; middle scores are associated with yellow. This kind of representation allows to visualize in an immediate way when a person is in a non-ergonomic (and therefore associated with red color) or ergonomic (associated with green color) posture space. Similarly, different variations of the same movement (and therefore different postures) can be mapped in the latent space, as it was done in Figure 5, to visualize which trajectory is better from an ergonomic point of view. In future work, we would like to use this representation to provide corrections to humans and plan better ergonomically optimized trajectories.

Experiments

In this section we elaborate on the structure of the VAE and the specific training procedure for the experiments. We then describe the use of the trained LEM to provide online ergonomics feedback in two different experiments, where a human executes pick and place activities with and without robot assistance.

Setup and Scenarios: Experiment 1

We demonstrate our ergonomics visualization tools to evaluate human movements executed in two experiments, with two work-related scenarios. In the first experiment, the human performs a pick and place task. The task is inspired by packaging tasks on assembly lines in the manufacturing industry and consists of picking, carrying, and placing a 6 kg bar. One male participant performed 8 sequences of the task, with each sequence consisting of 6 to 8 pick-and-place actions. Each sequence started and ended in the same neutral pose. The bar was initially placed at a height of 45 cm on a 100 × 50 cm flat support. The participant was instructed to take the bar with both hands, carry it to the other side of the support, place the bar there and return to the initial position to perform the next iteration. Each sequence lasted around one minute. To add variability to the data, the participant was instructed to change the position of his hands on the bar and to follow two different paths when going to and coming from the bar's final position.

Setup and Scenarios: Experiment 2

In the second experiment, the human carries loads in collaboration with a robot. They move a box together from a point A to a point B and then backward, along a trajectory that lasts about one minute. The box is initially placed at a height of 85 cm on a table. The box is fixed to the robot end-effector which carries the majority of its weight; the human grasps the object through a handle as shown in Figure 6. Each sequence lasts around one minute. At the beginning of the activity, the robot and the human grasp the object, and each sequence starts and ends in the same neutral pose. The robot control is based on a Cartesian impedance controller:

τ = J(q) (D( ẋd -ẋ) + K(x d -x)) + g(q) (3)
where K ∈ R 6×6 and D ∈ R 6×6 are respectively the stiffness and the damping matrices and g(.) is the gravity compensation term. Cartesian impedance control generates a torque proportional to the error between the end-effector pose and the desired end-effector pose (x, x d ∈ R 6 ) and their derivatives ( ẋd , ẋ ∈ R 6 ). We implemented two Cartesian impedance behaviors changing the values of the stiffness matrix: in the first, the robot was more compliant (K = 500 N/m) and in the second the robot was stiff (K = 1000 N/m). The desired Cartesian damping D was calculated proportional to K using factorization design as in [START_REF] Albu-Schaffer | Cartesian impedance control of redundant robots: Recent results with the DLR-light-weight-arms[END_REF]. The robot's trajectory is predefined by the robot planner, the latter selects a trajectory of Cartesian points using a parabolic curve passing through the initial and final points while the orientation is maintained constant. A participant repeated the movements 15 times, with 3 different robot trajectories (5 times for each trajectory) to add variability to the movement; in particular, we designed different robot trajectories in such a way to induce a variety of human postures and ergonomics. Some movement examples are shown in the video attachment. The robot used is the Panda (Franka Emika) robot controlled using libfranka and FrankaROS libraries.

In both experiments, the human is equipped with an Xsens MVN motion tracking suit, which provides real-time (240Hz) information about the current human pose: joint angles and link's positions, orientations, velocities, and accelerations. Both the human's and the robot's sensors measurements are streamed online to the module, to visualize and simulate their movements in the simulation (Dart [START_REF] Lee | Dart: Dynamic animation and robotics toolkit[END_REF]) as shown in Figure 7.

An overview of the main modules used in the two scenarios is shown in Figure 8, which details as well the flow of information from the Xsens sensors and the Franka robot. The simulation receives the sensed information, from both humans and robots, and their corresponding models are updated. All the items in the scene are modeled (i.e., a Unified Robotic Description Format (URDF) is available for all the elements: robot, experimental setup-table, object, etc.-as well as human) and simulated. Precisely, the robot communicates the following information to the simulation: (1) Robot joint configuration, (2) Robot end-effector pose, (3) External measured torques, and (4) Internal torques. The robot pose (joint angles, end-effector pose, and torque) are used to update the simulated robot. Meanwhile, the measured external torques are used to calculate the wrenches that the human is applying to the robot and then used to calculate the effort estimation Equation (1) Similarly, the motion tracking suit communicates: (1) the human posture expressed as the joint angles, and (2) the positions of the human links. The human joint angles are used to generate the LEM and to simulate the DHM; the links' poses are used to locate the DHM in the reference frame of the simulation. . Overview of the system in a human-robot collaboration setting. In the left box: the robot modules that enable to control the robot and retrieve information about the contact forces exchanged with the human. The human-robot interaction force is measured on-line thanks to the joint torque sensors embedded in the robot. Each of the 7 axis is equipped with a torque sensor, and based on these torques measurements, the Franka API provides an estimation of the interaction force at the end-effector. In the right box: modules for online estimation of the human kinematics, dynamics, ergonomics scores and visualization tools. Green boxes: the framework includes visualization tools to plot the online ergonomics scores and other relevant quantities, as well as visualizing the human and the robot interacting in a digital twin based on Dart physics engine.

The main interest of the second scenario is to use a DHM with visual clues of the joint torques and efforts, to account for the physical interaction between the human and the robot. The contact wrenches are input in the DHM in real-time, using the Franka robot's sensor measurements, which enables a better representation of the ergonomics status of the human from the point of view of efforts.

Creating and Visualizing the LEMs

To obtain the LEMs for every ergonomics score, we first created and trained the VAE representing the human body postures. The VAE architecture consists of 5 layers with 66 inputs, 200 hidden neurons, 2-dimensional latent space, and 66 outputs, where rectifier, tanh and identity activations are used for the hidden layers and the output layer. The hyperparameters of the VAE are chosen based on the reconstruction error and the training time by grid search. To train the latent space, we perform first a pre-training with a large human motion dataset (AndyData-lab-onePerson dataset [START_REF] Maurice | Human movement and ergonomics: An industry-oriented dataset for collaborative robotics[END_REF], containing more than 5 h of recorded data), then a fine-tuning with smaller task-specific datasets acquired for each experiment (see below). This latter step is optional. The first training with a large dataset is sufficient to generate a LEM, in particular, it enables to obtain a generic LEM that is task-independent and that captures a variety of different postures. In our experiments, however, the fine-tuning enables us to obtain a task-specific LEM that is more specialized to represent the postures of a specific task, while maintaining the generalization capabilities. All the datasets were normalized between zero and one and augmented using intermediate and mirrored poses before training.

The VAE training was carried out according to the procedure explained in Section 3.3. During training, we optimized both the weight of the encoder and the decoder. The weight optimization was done on an Intel Core TM i7-8850H with 6 cores at 2.6GHz, requiring about 20 min. The trained encoder can be used to visualize example movements in the latent space. This can be useful for preliminary insights into the postures for different activities. For example, Figure 9 shows the sequence of postures, projected in the 2D latent space, associated with the movements from the dataset of [START_REF] Malaisé | Activity recognition with multiple wearable sensors for industrial applications[END_REF], which contains activities such as bending, kicking, lifting objects, and walking. Some postures are also displayed, to clarify that each 2D point is representing a different posture. Some 2D points belong to different activities, and this is normal since the same body posture can be observed in different activities. It is also possible to display ergonomics information in this activities representation: as shown in Figure 5, it is possible to color-code the points associated with a movement according to the ergonomics scores. In this case, the activity information is lost, but it is possible to inspect one or more movements. Of course, the limit of this visualization is that it only shows the execution of a movement.

The trained decoder is used to create the LEM, which is a latent space height-map for a given ergonomics score. To do so, we uniformly sample the latent space to obtain a set of 10,000 points (100 × 100). For each point in the latent space we apply the decoder to reconstruct the original posture, we invert the normalization function and then we apply the ergonomic score to the estimated pose. The shape of the latent space is influenced by the choice of the activation functions, for instance, the sigmoid activation function maps the input to [0, 1] while the tanh maps to [-1, 1]. In Figure 10, we show the LEMs for RULA and RULA-C. The triadic color code (green, yellow, red) is applied to the postures classified according to the RULA recommendations, while a continuous color coding based on the triad is used for RULA-C. The LEM created using the RULA-C score creates, by definition, a latent space that is similar to that of the RULA but continuous. While there is no immediate advantage in using one of the two LEMs for ergonomics visual feedback, the continuous LEM can be used in future work as a reduced model for planning ergonomics movements using gradient-based methods [START_REF] Gomes | Multi-Objective Trajectory Optimization to Improve Ergonomics in Human Motion[END_REF].

In the online phase, the LEM is used to visualize the overall ergonomics score of a posture. Figure 1 shows the RULA LEM (center) corresponding to the human motion (left) tracked in real-time (240 Hz) by the Xsens MVN suit. During the execution of the movement, the current human posture is projected into the ergonomic map, after being encoded by the encoder into a 2D point in the latent space. The result is a trace, i.e., a sequence of 2D points moving in the ergonomic map, representing the human motion and its ergonomics evaluation. Another example of the trace onto a LEM is shown in Figure 6, and other examples are available in the video attachment.

Visualization of Local Ergonomics Scores in the DHM

The DHM is used to display in real-time the ergonomics information associated with human movement. The real-time information about the human posture is used to update the DHM pose and compute the local ergonomics score, where "local" indicates the specific body location. Each score is calculated using the partial scores presented in the RULA table. In the case of RULA-C, these have continuous values because they are the result of polynomials that approximate the RULA values. The local ergonomic scores are then visualized on the DHM to provide intuitive feedback about the current posture. Specifically, we color some spheres placed at relevant joints (knees, ankles, back, torso, shoulders, and elbows) with colors that are proportional to the ergonomic score. The latter is normalized in the range [0, 1] → [GREEN, RED], using the maximum score value that is set after the maximum score registered for that body part on the dataset used for training the LEM in the offline phase. The reason for the latter normalization is to better spot the difference in ergonomic scores between similar movements. In our setup, it is possible to select the ergonomic score to visualize on the DHM by interacting with an easy-to-use GUI (some examples in the video attachment). Figure 1 shows the DHM displayed during a manipulation task, together with the RULA LEM: while the LEM provides a synthetic visualization of the overall ergonomics score, the spheres on the DHM enable to visualize the individual scores on the different body parts. This is convenient to identify the body parts that are more at risk, from the ergonomics standpoint, which are colored in red; those that are not at risk are colored in green, and the intermediate values in yellow. Figure 11 shows more examples of the DHM during a pick and place movement where the human takes on different body postures: back joints are red when the human is bent forward, the knees are red during squats, while shoulders are red during over-head work. In particular, some are usually classified as non-ergonomic (e.g., hands over shoulders, bent back, squat). The color-coded spheres on the DHM show the body parts that have a high risk (ergonomics score: RULA) during these postures.

Discussions and Conclusions

In this article, we presented a set of tools for providing online ergonomics feedback to human workers during their activities, also when they physically interact with robots. A Digital Human Model is used to visualize, with color-coded spheres, the body areas and joints that are subject to efforts and non-ergonomic postures according to state-of-the-art ergonomics scores, such as RULA. Several plotting tools were also developed to provide an intuitive visualization of ergonomics scores associated with human movement.

Our contribution, Latent Ergonomics Maps, are synthetic representations of the overall ergonomics scores projected onto a bidimensional latent space that maps human postures. The result is an intuitive color-coded map where the human posture is a point, a movement is a line, and their associated color is an estimation of the ergonomics score of choice. LEMs can be used for bio-feedback or self-correction, as a visual tool for teaching, or simply to inform the human. Their potential goes beyond the online feedback for the human, as they can be used to inform the robot as well, which can find applications in planning ergonomically optimal collaborative motions. Advantages of LEMs include their ease of interpretation also for non-experts, and the computational efficiency, enabling online feedback. A limitation of projecting the map on a 2D latent space, which is necessary for visualization purposes, is the information loss that may result from such a strong dimensionality reduction. However, the error is tolerable for ergonomics scores based on postural information, and otherwise acceptable if coupled with the visualization of efforts on the Digital Human Model.

In the future, we plan to combine the prediction of intended movement [START_REF] Dermy | Prediction of human whole-body movements with AE-ProMPs[END_REF] with LEMs, therefore predicting future ergonomics scores for the intended movement. This will enable us to alert the human of possible risks associated with ergonomics. Moreover, we want to leverage the LEM for ergonomics optimization of robot motions to improve human-robot collaboration. The idea is to inform the robot about the ergonomics risk associated with planned collaborative robot trajectories as in [START_REF] Vianello | Human Posture Prediction during Physical Human-Robot Interaction[END_REF], and then to optimize these trajectories using ergonomics optimization as in [START_REF] Gomes | Multi-Objective Trajectory Optimization to Improve Ergonomics in Human Motion[END_REF][START_REF] Gomes | Improving Ergonomics at Work with Personalized Multi-Objective Optimization of Human Movements[END_REF].

Figure 1 .

 1 Figure 1. The ergonomics visual feedback tools used to measure the executed motion in real-time. (Left): the human movement is tracked with the Xsens MVN motion tracking suit. At the same time the RULA Latent Ergonomics Map is calculated, with the visualization of the current human movement (the magenta-colored line). (Right): the DHM with colored spheres showing the RULA scores at relevant body locations.

Figure 2 .

 2 Figure 2. Digital Human Model (DHM) with 66 degrees of freedom. For each DHM's link we display its origin axis (x in red, y in blue and z in green).

Figure 3 .

 3 Figure 3. Training the VAE of the LEM requires a dataset of human postures. For each posture an ergonomics score exists. In our experiments, as a dataset, we used: (1) the AnDy dataset [30] and (2) a task-specific dataset acquired for each experiment. The training is done to find the VAE weights which simultaneously minimize the reconstruction error of the postures and the associated ergonomics score.

Figure 4 .

 4 Figure 4. Creation of the LEM: the 2D latent space of the VAE is sampled. The decoder reconstructs sampled latent points. The reconstructed postures are used to compute the ergonomics scores. The ergonomics score is therefore associated with the original 2D points, thus creating a heightmap.

Figure 5 .

 5 Figure 5. Latent space representation: in this plot, some examples of movements from the dataset of [38] colored according to the ergonomic evaluation. The postures associated with some 2D points in the latent space are shown. On the right, the scale of the colors corresponds to the ergonomic scores where: 1 (green) corresponds to the safer postures while 7 (red) represents the postures less safe.

Figure 6 .

 6 Figure 6. Experiment 2: collaborative object transportation. The human is physically interacting with the Franka robot. On the right of each photo, the Latent Ergonomic Map. The current human posture is a point on the map while the line that is attached to it represents the previous human posture.

Figure 7 .Figure 8

 78 Figure 7. Experiment 2: collaborative object transportation. The DHM with the colored spheres indicating non-ergonomic joint values.

Figure 9 .Figure 10 .

 910 Figure9. Latent space representation of the human movement during some activities from the dataset of[START_REF] Malaisé | Activity recognition with multiple wearable sensors for industrial applications[END_REF]. The movements presented in this latent space are: bent forward (strongly), kicking, lifting a box, standing, walking, open a window. A movement is a sequence of points in the latent space. A color code enables to distinguish the 2D path associated with each sequence. The postures associated with some 2D points in the latent space are shown.

Figure 11 .

 11 Figure 11. Experiment 1: pick and place. Across the pick and place task, the human takes on different postures. In particular, some are usually classified as non-ergonomic (e.g., hands over shoulders, bent back, squat). The color-coded spheres on the DHM show the body parts that have a high risk (ergonomics score: RULA) during these postures.

Acknowledgments:

The authors would like to thank Adrien Malaisé and Francis Colas for their contribution to the initial prototype of LEM; Eloise Dalin, Ivan Bergonzani, and Edoardo Ghini for their help with the DHM and Xsens streaming software; the AnDy team for insightful discussions.

Funding: This work was partially supported by the European Union Horizon 2020 Research and Innovation Program under Grant Agreement No. 731540 (project AnDy), the French Research Agency (ANR) under Grant No. ANR-20-CE33-0004 (project ROOIBOS), the CHIST-ERA project HEAP, the European FEDER in the context of the CPER Sciarat and the CreativLab, and the Lorraine University of Excellence (LUE) project C-Shift.

Data Availability Statement:

The dataset of human motions and postures is available at https: //zenodo.org/record/3254403, accessed on 23 May 2022. A detailed description of the dataset and its content is presented in [START_REF] Maurice | Human movement and ergonomics: An industry-oriented dataset for collaborative robotics[END_REF].

Author Contributions: S.I., F.S., P.M., L.V., W.G. conceived the presented ideas and elaborated the method. L.V., W.G., F.S., A.A., P.M. and S.I. wrote the manuscript. L.V. and W.G. implemented the methods and performed the experiments. S.I. supervised the research. All authors have read and agreed to the published version of the manuscript.

Conflicts of Interest:

The authors declare no conflict of interest.

Abbreviations

The


Thomas Pellegrini
email: pellegri@irit.fr

Patrice Guyot
email: patrice.guyot@irit.fr

Baptiste Angles
email: baptiste.angles@gmail.com

Christophe Mollaret
email: christophe.mollaret@irit.fr

Christophe Mangou
email: christophe.mangou@ensemble-amalgammes.fr

Towards soundpainting gesture recognition

Keywords: Gesture recognition, soundpainting, Hidden Markov Models, music, interactive systems ACM Classification H.5.5 Sound and Music Computing

In this article, we describe our recent research activities on gesture recognition for soundpainting applications. Soundpainting is a multidisciplinary live composing sign language for musicians, actors, dancers, and visual artists. These gestures are produced by a soundpainter, which plays the role of a conductor, in order to lead a live performance. Soundpainting gestures are normalized and well defined, thus they are a very interesting case study in automatic gesture recognition. We describe a first gesture recognition system based on hidden Markov Models. We also report on the creation of a pilot corpus of soundpainting RGB/depth videos. The use of a computer could have many interesting applications listed in the paper. These applications are not limited to live performance, in which the computer would act as a performer. It could also help to investigate the balance between improvisation and planned creation in the particular context of soundpainting.

Introduction

In music perfomances, the balance between improvisation, as a spontaneous decision, and prepared material such as music scores has always been a source of reflexions, investigations, and creation. This exploration has produced, over the years, many different music styles and practices, from traditional and classical music to blues and free-jazz. Encyclopedia Britannica1 defines music improvisation as "the extemporaneous composition or free performance of a musical passage, usually in a manner conforming to certain stylistic norms but unfettered by the prescriptive features of a specific musical text".

In our work, we are interested by an improvisation framework called Soundpainting, which was invented about fourty years ago and that is gaining momentum nowadays. It is a multidisciplinary live composing sign language for musicians, actors, dancers, and visual artists 2 . Presently, this language comprises more than 1000 gestures, although a hundred already allow to perform. These gestures are produced by a soundpainter, which plays the role of a conductor, in order to lead an improvisation. In a musical band or orchestra, the interaction between musicians is always a key point, which becomes crucial in improvised music [START_REF] Moran | Measuring musical interaction: Analysing communication in embodied musical behaviour[END_REF]. Thus, the different roles and decisions to take in real time produce complex situations that are difficult to analyze and evaluate [START_REF] Eisenberg | A matter of taste: Evaluating improvised music[END_REF]. This remains true in the context of soundpainting, even if some supervision is brought by the soundpainter. Computer-assisted improvisation systems have attracted a lot of attention over the last ten years [START_REF] Blackwell | Swarm music: improvised music with multi-swarms[END_REF][START_REF] Yee-King | An automated music improviser using a genetic algorithm driven synthesis engine[END_REF]. Examples are the framework provided by the Omax project, where an interactive system learns in real time from human performers [START_REF] Assayag | Omax brothers: a dynamic topology of agents for improvization learning[END_REF], or the Continuator musical instrument that learns and interactively plays with a user in the user's style [START_REF] Pachet | The Continuator: Musical Interaction with Style[END_REF]. Recently, with the availability of RGB/Depth cameras at a low cost, gesture recognition/modeling real-time systems were developed with applications in music composition and improvisation. For instance, a gesture follower system was developed at IRCAM [START_REF] Bevilacqua | Continuous realtime gesture following and recognition[END_REF]. Nevertheless, to the best of our knowledge, no system was designed specifically for soundpainting applications.

In this paper, we report on a pilot corpus of soundpainting RGB/Depth videos, and on a first prototype of a gesture automatic recognition system in the context of soundpainting applied to music improvisation. We first briefly describe soundpainting and its main concepts. We give an overview on techniques used in gesture recognition and present our prototype.

A proof of concept is reported along with the description of the pilot corpus created with a professional music conductor.

The last section lists ideas of potential applications.

Bird's-eye view on soundpainting

Soundpainting is a sign language developed in the mid-1980s by the American Jazz composer Walter Thompson. Using a set of gestures, soundpainting allows the spontaneous conduction of an orchestra. The signs indicate the type of material desired by the conductor, known as the soundpainter, but it can also be used to conduct dancers, actors, poets, and visual artists. Soundpainting becomes more and more established as a code to be used in improvised performances, as shown by the emergence of dedicated orchestras and music festivals 3 . It gives an official and normed language to compose and improvise in real time, with the possibility to use previously prepared music as well as free improvisation. Beyond musical performance, soundpainting also provides tools for musical teaching. In practice, the soundpainter realizes sequences of signs using hand and body gestures that can be thought of as gesture sentences that give instructions to the group of perform-

Gesture recognition

Gesture recognition opens new perspectives for natural human-machine interactions. Indeed, we constantly use gestures to communicate without requiring special attention or effort memory. It is therefore quite natural that they can be used to interact efficiently with a computer [START_REF] Biswas | Gesture Recognition using Microsoft Kinect[END_REF]. Applications of gesture recognition are manifold. Examples of the most common applications are video games, smart video surveillance systems, sign language, and virtual reality. A broad review of the approaches used in automatic gesture recognition can be found in [START_REF] Mitra | Gesture Recognition: A survey[END_REF]. Since then, the arrival on the market a few years ago of inexpensive cameras that combine one or several RGB cameras with a depth camera (such as the Microsoft's Kinect device) allowed the real essort of 3D gesture recognition methods. Depth images are useful to ease the retrieval of the third dimension (depth) and unlike RGB cameras, they are not impacted by the color fluctuations induced by clothing, skin, or the environment. Machine learning techniques range from particle filtering to sequential generative models such as hidden Markov Models [START_REF] Mitra | Gesture Recognition: A survey[END_REF]. Recently, systems involving neural networks such as multilayer perceptrons or recurrent networks are investigated for their discriminative power. Deep learning is also a resurging paradigm at the core of actual studies in computer vision and other domains such as speech recognition [START_REF] Arel | Deep Machine Learning -A New Frontier in Artificial Intelligence Research[END_REF].

HMM-based prototype

Figure 1 shows the architecture of our system. The soundpainter is filmed by a device that captures RGB and depth videos. The RGB and depth image frames are processed to retrieve the skeleton joints of the signer (we used the NITE and OpenNI libraries for this purpose). The frame-based skeleton absolute coordinates feed our gesture recognition software. Several sliding windows of different frame lengths are used in parallel. The analysis window sizes range from 5 to 10, along with windows of 20, 30 and 40 frames. The use of several frame lengths improved recognition performance as it allows to recognize a gesture performed at different paces.

Then, the skeleton coordinates' frames from each sliding window are normalized. This normalization is a generalization to three dimensions of the normalization applied in the "$1" algorithm [START_REF] Wobbrock | Gestures without libraries, toolkits or training: A $1 recognizer for user interface prototypes[END_REF]. It consists of a resampling of the gesture to 32 coordinate vectors for each sliding window. Different kinds of spatio-temporal resampling methods were implemented and we use one that smoothly interpolates the joint trajectories. Follow a rotation according to the soundpainter shoulders' orientation and a translation to center the acquisition to a reference point. A symmetry to merge left-and right-handed gestures is then performed, and a final rescaling to a reference cube is done. We use the x-y-z spatial coordinates of 6 skeleton joints: the elbow, the wrist and the hand joints for both the left and right hands. Our system is based on Hidden Markov Models, one per gesture. We chose this approach for the temporal sequence modeling capabilities of HMMs. Initially, we did some Matlab prototyping based on the HMM implementation from Dan Ellis 4 . Then, we implemented a Java version from scratch. Our final HMM approach is similar to the one reported in [START_REF] Bevilacqua | The augmented string quartet: experiments and gesture following[END_REF].

The main difference lies in the fact that we model each gesture by HMMs with a few states only, typically two or three states, instead of using one state per frame. Gesture recognition can be performed with Viterbi decoding or with the forward algorithm that gives slightly better results. We use a multivariate Gaussian distribution as observation probability density. We plan to implement Gaussian Mixture models and also non-parametric methods in a near future.

Proof of concept

Our objective is to adapt our system to soundpainting gesture recognition for music creation. We initiated a fruitful collaboration with a certified soundpainter, Christophe Mangou5 .

He is one of the directors of the reknown French classical orchestra "Orchestre national du Capitole de Toulouse" 6 . Besides his activity in classical music, he started a soundpainting project in 2005 named Amalgammes, which is a variablesize orchestra that combines the use of written scores and improvisation led by the soundpainting vocabulary 7 . We recorded a pilot corpus with him, which consists of video recordings of Christophe Mangou performing about 20 gestures repeated five times each, such as 'Play', 'Whole Group', etc. We also recorded a set of soundpainting "sentences", which are sequences of gestures that could be used in a performance. Isolated gestures and gesture sequences present differences in their realization that reminds co-articulation in speech or in sign language. Figure 2 shows an example of a realization of 'Play'. This gesture was modeled by three states, from state 0 to state 2. The right-hand side of the figure shows the visited states as found by a Viterbi decoding. As one can see, the three states are sequentially visited. We used this kind of analysis to determine the best number of states for each gesture. When a state is not visited, it is removed from the model.

We further enhanced our system by adding features derived from the tracking of the hands such as hand blob areas, hand barycenter coordinates and spatial moments. We are cur-rently participating to a competition named ChaLearn that focuses on the recognition of a vocabulary of 20 Italian cultural/anthropological signs 8 .

Future applications Computer as a musician

A first direct application to soundpainter gesture recognition is the real time production of sound and music by a computer.

In the case of gestures corresponding to function signals (who and when), the interpretation of these gestures is straightforward. For example, with a given sign or simply by pointing towards the camera, the soundpainter could designate the computer directly, indicating that a sound should be played by the machine.

The of sculpting (what and how) is a more complex question. Indeed, the use of a computer to produce or play sounds offers a wide variety of possibilities. These possibilities could be reduced to a single instrument for instance.

In this case, a wide range of sculpting gestures such as long tones or notes in a pointillist style could be used. We can also imagine that the computer could play the role of several instruments. In this case, specific gestures should be used. The soundpainter could then ask for a drum or a double bass, but also for iconic sounds, recorded music or environmental sounds such as rain or broken glass.

The improvised aspects are an important point to take into consideration as it is a key point in soundpainting. It can be guided by random propositions, but could also depend of the context and previous material, as it is the case in the Omax project [START_REF] Assayag | Omax brothers: a dynamic topology of agents for improvization learning[END_REF]. Moreover, the interaction between musicians is fundamental in a soundpainting performance. This raises very interesting and difficult questions when introducing a computer in the loop.

New tool for music composition

Beyond live performance, it would be interesting to use the computer in music composition led by soundpainting. Indeed, such a framework could be used to build new interfaces for sound design based on soundpainting gestures.

Computer-assisted soundpainting learning

Gesture recognition could be used by the soundpainter as a tool to learn or to improve his gestures. The sounpainters can be assisted in real time by a feedback given by the computer. Specific strategies should nevertheless be implemented in order to provide relevant and informative feedback.

Analysis of practice

As a recent musical practice, soundpainting aroused the interest of musicologists and social scientists. It provides a context where improvisation and live interaction can be analyzed. A first PhD Thesis on the subject was written by Marc Duby, who investigated how soundpainting operates as a system for collaborative creation of music in performative situations [START_REF] Duby | Soundpainting as a system for the collaborative creation of music in performance[END_REF]. Moreover, soundpainting opens the way to a sociosemiotic analysis, in which it is susceptible to examination in the light of some theories of language. Lastly, practice of soundpainting raises questions, such as the evaluation of a live performance by an audience [START_REF] Eisenberg | A matter of taste: Evaluating improvised music[END_REF]. A soundpainter gesture recognition system could help in these studies by indexing performance video recordings automatically, and also by reproducing and creating new performances when the computer is used as a performer.

Conclusions

In this paper, we described a new application of gesture recognition to the context of soundpainting. Soundpainting gestures are a very interesting case study in automatic gesture recognition as showed by the encouraging experiments that we conducted with an HMM-based system. The intro-duction of a computer could have many exciting applications that we listed in the paper. The computer could act as a performer, thus, very interesting and difficult questions would be raised by the fact that performers interact with the soundpainter but also between each other. These applications are not limited to live performance. A machine would also help to investigate the balance between improvisation and prepared music creation in the particular context of soundpainting.

  ers. Basic gestures depicte musical concepts such as volume, tempo, pitch, and duration. More complex signals encompass notions of genre, style, key, memory, and more. The gestures are signed using the syntax of Who, What, How and When. They are grouped in two basic categories: sculpting gestures and function signals. Sculpting gestures indicate What type of material and How it should to be produced and function signals indicate Who will execute a set of instructions instructions and When. In our project, these different types of gestures must be recognized and semantically understood by an automatic system.

Figure 1 :

 1 Figure 1: HMM-based prototype

Figure 2 :

 2 Figure 2: HMM states visited during a realization of a 'Play' gesture.

Encyclopaedia Britannica Online, http://www.britannica.com/EBchecked/topic, last visit: May

2 http://www.soundpainting.com, last visit: May 2014 3 http://www.soundpaintingfestival.fr

http://www.ee.columbia.edu/ dpwe/e4896/practicals.html#prac10

http://www.christophemangou.com/en.html

http://onct.toulouse.fr

http://www.ensemble-amalgammes.fr

http://gesture.chalearn.org/

Acknowledgment

This work was partly supported by a grant from the ANR (Agence Nationale de la Recherche) with reference ANR-12-CORD-003.