Alexis Clay

Nadine Couture
email: n.couture@estia.fr°laurence.nigay@imag.frjb.delariviere

Laurence Nigay

Jean-Baptiste De

La Rivière

Jean-Claude Martin

Matthieu Courgeon
email: courgeon@limsi.fr±vincent.girondel@gipsa-lab.grenoble-inp.fr

Myriam Desainte-Catherine
email: myriam.desainte-catherine@labri.frog.domenger@malandainballet.com

Emmanuel Orvain
email: emmanuel.orvain@immersion.fr§martin

Vincent Girondel

Gaël Domenger

Laurence Nigay°

Estia

Interactions and systems for augmenting a live dance performance

Keywords: Augmented reality, augmented dance, emotion recognition, interaction for performing arts [Computer Applications]: Arts and Humanities -Arts, fine and performing; H.5.2 [Information Systems]: Information interfaces and presentation-User Interface

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

The results presented in this article were realized within the CARE (Cultural Experience: Augmented Reality and Emotion) project. CARE brought together seven academic and industrial partners in order to develop, adapt and integrate augmented reality related tools to enhance the emotion produced in cultural performances. The project was built around two application cases: augmenting a museum (the Natural History Museum of Toulouse, France) and augmenting a dance show (with the French National Choreographic Centre Malandain Ballet Biarritz). This second case gathered research in various fields around a single applicative goal: to augment a live dance show by adding virtual elements (visual as well as audio ones) to the stage and give performers new ways of interacting with those virtual elements. Augmented Reality (AR) can be divided in two types. Seethrough AR can be performed using handheld or headworn devices. Space AR uses projective displays. We place ourselves in the latter case. The dancer is tracked and can interact with virtual elements projected on stage. In our case, the virtual elements are projected for the audience, while the performer is the one that interacts with them. In this paper, we present the various interaction techniques that we developed and how we integrated them with augmented reality tools and techniques in order to create a full system for an interactive augmented live dance show.

In this paper we focus on the overall system and the interactions that were proposed to the choreographer. The system can be considered as an assembly of modules developed by the various partners involved. The assembly of those modules allowed us to create various interactions and augmentations, using the artistic proposal as a guideline, and finally create a system to use in a real augmented show entitled "CARE Project: staging of a research process". This show was choreographed and performed by Gaël Domenger from Malandain Ballet Biarritz, and was presented twice, in front of a live audience, on march 4 th , 2011 in the Casino de Biarritz. The artistic proposal of the show was to portray the process that led to the development of the technologies and tools that were used on stage. It consisted of two parts: a danced conference, where the project coordinator presented the project and the dancer illustrated the speech (figure 1.a); and an augmented show, where the technologies were staged in an aesthetic way (figure 1.b). A short video of the show and a movie are available online at [36]. The contribution presented in this paper is the result of the collaboration of researchers and industrials from four different areas: human-computer interaction, sound and music modeling, augmented reality, and arts. In human-computer interaction, we have a special interest in interaction design theory, movementbased interaction, and especially in movement-based emotion recognition. Within the context of augmented dance, our interest is to recognize the emotions the dancer expresses, and translate them into various output modalities. In music modeling, our interest was to define a set of high-level perceptive parameters that could easily be mapped to emotional concepts and then drive a real-time generative music system from emotion recognition. In augmented reality, we focused on the issues of creating a system for augmenting a large space, meaning capturing and tracking the dancer and develop a physical setup able to add virtual augmentations over the whole stage. For artists, combining those fields unveiled a brand new field of exploration for improvisation in dance, where the dancer could manipulate the sound and the visuals and use them for improvisation at the same time.

The combination of those various interests led us to design and develop various interactions for the context of augmented reality on stage that we present in this paper.

We therefore developed five modules that could be combined together to create interactions and augmentations. The first one is motion tracking, i.e. being able to know the exact conformation of the dancer's body on stage in real time, and his position on stage.

The second one is a system performing movement analysis for recognizing emotions, which can extract movement characteristics and the emotions that the dancer portrays through his movement.

The third one is a musical generator that creates emotional pieces from a given emotion. The fourth one is a system that makes a humanoid robot take an emotionally-expressive pose, from a given emotion. Finally, the fifth one is the physical and software setup to display augmentations on stage. The assembly of these different systems allowed us to develop several movement-based interactions. Two of them are direct, such as the direct manipulation of virtual elements by the dancer, and the animation of an avatar that copies the dancer's movements. Two of them go through a high level of abstraction: movements from the dancer are translated into emotions, which are used to generate emotional pieces of music and poses for the humanoid robot. The interactions we designed are achieved by the combination of those different modules. The set up of the show allowed us to test those interactions in a real-life context. An originality of our work is that every module was designed to work in real-time. Emotion recognition, music generation, 3D elements animation or generation, and the robot animation run simultaneously in real-time, for being used in an improvised dance. We also introduced the position of the "augmented reality engineer", whose role is to monitor and sometimes control the augmentations, as the light or sound technicians monitor and control the lighting and sound. We will first review some related work to highlight the originality of our work regarding each element and then regarding the set up of augmented shows. We will then proceed to define the various interactions (input and output) that we propose. To implement those interactions, we developed several independent and reusable modules that we will present in the following section. We will then review the proposed interactions from a technical point of view. Finally, we will present the show that allowed us to test those interactions and the feedback we received from the audience and the choreographer, before presenting our conclusions.

STATE OF THE ART

For the scope of augmenting a dance show, we naturally focused on movement-based interaction. Movement is the natural language of dance, which we sought to expand using augmented reality techniques.

Designing movement-based interaction for performance

Designing interaction for the context of a live performance falls out of the traditional design spaces. Interactions in performances can be taskless, and should bear aesthetics of their own to better engage the performer and the audience. Petersen [START_REF] Petersen | Aesthetic interaction: a pragmatist's aesthetics of interactive systems[END_REF], for example, suggests the addition of the aesthetic perspective in usercentered design: an aesthetic interaction seeks to challenge and intrigue the user to encourage improvisation. The BodyBug [START_REF] Moen | From hand-held to body-worn: embodied experiences of the design and use of a wearable movement-based interaction concept[END_REF] is an applicative example of an aesthetic interaction. This device was designed in order to explore movement as an interaction, and especially the movement qualities that can be drawn from modern dance.

These works consider the aesthetics of an interaction relatively to the user. In our case, interaction is indeed performed by the artist, but also perceived by the audience. Interactions on stage should hence be considered in an artistic way and be coherent with the choreography. As such, we consider an interaction on stage as a whole, ranging from input to output. As such, designing movement-based interaction for dance involves three main issues.

The first one is related to the input: what movements should be captured and how? This issue encompasses the extent of the capture (e.g. position and orientation on stage, a single limb, the whole body), the measures that are used (e.g. rotation or position information), and the technology to obtain those measures (e.g. motion capture suit, video-based techniques [START_REF] Castellano | Recognising human emotions from body movement and gesture dynamics[END_REF], or gyroscopic sensors [START_REF] Latulipe | Evaluating longitudinal projects combining technology with temporal arts[END_REF]).

The second issue is the interaction language used to manipulate the output. This language can be direct (e.g. direct use of the hand's position to manipulate a virtual object) or expressive. In the latter case, the system extracts movement qualities that are then used to manipulate the output. In particular, Laban's Theory of Effort [START_REF] Hodgson | Mastering movement : the life and work of Rudolf Laban[END_REF] is a grounding work for many existing systems relying on expressive interaction. For example, A light touch [START_REF] Alaoui | Movement qualities as Interaction modality[END_REF] is an artistic installation where movement qualities from the user (breathing, expanding and reducing) impact the behavior of a light output. Expressive interaction can be extended to emotional interaction, where movement characteristics and qualities are further analyzed to extract the emotion conveyed by the dancer. The Infomus Lab in Genoa conducted extensive work on the subject [START_REF] Castellano | Recognising human emotions from body movement and gesture dynamics[END_REF] [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF].

The third issue is the output modality and the technical and artistic choices that are made in output generation or manipulation.

These issues closely relate to Latulipe et al.'s design dimensions for creating interactive dances [START_REF] Latulipe | Evaluating longitudinal projects combining technology with temporal arts[END_REF]. In our methodology however, we also view the interaction language as a design dimension to consider.

The interactions presented in this paper cover different design choices. Namely, the show and its underlying technical system feature both direct and expressive interaction languages, and several output modalities (e.g. music generation and 3D manipulation). To be able to implement those interactions, our primary goal was to be able to capture the dancer's movement and position on stage.

Capturing movement

Many interactions for the stage involve capturing at least part of the performer's movement (e.g. his position on the stage). Traditional means of tracking a person's movement include videobased techniques or the use of more expensive motion tracking devices. Video-tracking systems are not adapted to the stage as they usually rely on controlled lighting; a typical show on stage involves different lights or moving light spots that render these technologies unusable. Welch et al. published a review of the different motion tracking techniques [START_REF] Welch | Motion tracking: no silver bullet, but a respectable arsenal[END_REF]. Although from 2002, this review is of great help to better choose the motion capture technology suited to our context.

For capturing a dancer's movement on stage we chose the MVN motion capture suit from XSens [35]. This suit features magnetometers, gyroscopes and accelerometers, making the motion capture resistant to lighting conditions and occlusions.

Emotion recognition and movement-based emotion recognition

Affective computing involves decoding emotions expressed or felt by a user (emotion recognition) and generating emotional feedback to the user, for example through animating virtual characters or generating emotional pieces of music. Both fields must rely on a theoretical background on emotions. There are many definitions and representations of emotions in the field of psychology, but few of them are actually used in computer science. The systems and the interactions described here rely on the two preponderant representations. Our emotion recognition system, called eMotion, and the humanoid robot pose generator were built around Ekman's six basic emotions [START_REF] Ekman | Basic emotions. The handbook of cognition and 45-60[END_REF]. The music generation system, called MuZICO, relies on a two-dimensional plane of emotions [START_REF]Russell A circumplex model of affect[END_REF]: emotions are then characterized by their valence (pleasant -unpleasant) and their arousal (high -low), those two characteristics being the axes of the 2D plane. As such, a translation from discrete representation to continuous representation was developed to successfully connect eMotion and MuZICO.

Emotion recognition can be divided in three consecutive steps: monitoring the subject (through his facial expression, voice, movements or physiological reactions), extract cues of emotional expression (e.g. a smile, a raise of the eyebrows) to finally infer an emotion. There is a large literature about bodily expression of emotions. In psychology, humanly-perceived cues are identified and validated as conveying emotional information. For instance, De Meijer [START_REF] Demeijer | The contribution of general features of body movement to the attribution of emotions[END_REF] adopted a top-down approach to identify and validate, through human evaluations of actor performances, affect-expressive movement cues such as trunk curvature, position of hands or velocity of a movement. Wallbott [START_REF] Wallbott | Bodily expression of emotion[END_REF] conducted a bottom-up study to identify typical movement characteristics for the following emotions: elated joy, happiness, sadness, despair, fear, terror, cold anger, hot anger, disgust, contempt, shame, guilt, pride, and boredom. Coulson [START_REF] Coulson | Attributing emotions to static body postures: recognition accuracy, confusions, and viewpoint dependence[END_REF] worked on affective cues in static body postures. In computer science, the Infomus lab created systems able to recognize emotions from movements. Using video-analysis techniques, they identified a set of movement characteristics to be extracted from video sequences. Their work was applied in particular to dance; the system could classify dance sequences into four emotions [START_REF] Castellano | Recognising human emotions from body movement and gesture dynamics[END_REF] [START_REF] Volpe | Computational models of expressive gesture in multimedia systems[END_REF].

Expressive pose generation

There are multiple ways to augment a dance performance with affective displays. In two preliminary performances, we augmented the dance performance with a virtual character that expresses emotions with its facial expressions using the MARC platform [START_REF] Courgeon | Impact of expressive wrinkles on perception of a virtual character's facial expressions of emotions[END_REF]. In a second experiment, we were willing to 1) use an embodied device in the real world with which the dancer could play with, and 2) use bodily expressions of emotions so as to be symmetrical with the dancer. Using a robot that could take expressive poses allowed fulfilling those two goals.

There is a large literature on expressive robots. Most of the work, however, relates to the face (e.g. [START_REF] Riek | Cooperative Gestures: Effective Signaling for Humanoid Robots[END_REF], [START_REF] Matsui | Generating Interactive Facial Expression of Communication Robots Using Simple Recurrent Network[END_REF]), as facial expressions are the most direct channel for conveying emotions. Those robots are usually designed to be close enough so that humans can "read their faces". In our case, were the audience sat far from the robot, exaggerated body movements were necessary to better convey the emotions. Using human body movements to directly generate the robot's movements is an explored issue [START_REF] Boutin | Biomimétisme : Génération de trajectoires pour la robotique humanoïde à partir de mouvements humains[END_REF] [START_REF] Kleinsmith | Postural expressions of emotion in motion captured database and in a humanoid robot[END_REF]. However, more studies remains to be done to explore the potential of affective interaction between a dancer and a humanoid robot during a real-time performance.

Music generation from emotions

Generating music directly from gestures or from emotions are two different problems. Some systems are instrument-oriented whereas others are more generative. In our approach we use an instrument-oriented paradigm to process movement characteristics, and a generative paradigm to generate the musical flows from emotions. In the instrument-oriented paradigm, various works can be cited about gestural control of electronic music: for example, A. Tanaka [START_REF] Tanaka | Musical performance practice on sensor-based instruments[END_REF] exploited muscular signals thanks to B. Knapp's Bio-muse system [START_REF] Knapp | A Bioelectric Controller for Computer Music Applications[END_REF] to play music in realtime. Serge de Laubier proposed the meta-instrument [START_REF] Laubier | Meta-Instrument 3: a look over 17 years of practice[END_REF], a general-purpose gestural controller that involves the full body and can be used as a real musical instrument. In the generative paradigm, a famous example of music generator is Omax [START_REF] Dubnov | Improvisation planning and jam session design using concepts of sequence variation and flow experience[END_REF], an IRCAM software using Markov chains to generate melodies (which has become a standard technique in generative music), allowing the interaction with a performer. Eduardo R. Miranda [START_REF] Miranda | Interfacing the Brain Directly with Musical Systems: On developing systems for making music with brain signals[END_REF] set up a brain-computer interface for generating music, which is not far from emotion analysis.

It is established that composed musical structures, as well as actual performance, influence the perceived emotions [START_REF] Gabrielson | The influence of musical structure on emotional expression. Music and Emotion, theory and research[END_REF]. Several empirical studies have focused on which emotions can be expressed by music and which musical factors contribute to the perceived emotional expression. Most of these studies focused especially on Western classical music. A review of articles about analysis of music features and their effects on the perceived emotion has been published in [START_REF] Livingstone | Changing musical emotion: A computational rule system for modifying score and performance[END_REF], providing rules related both to composition and performance for eliciting a particular emotion. Similar experiments in music generation led by emotion have been conducted before [START_REF] Wallis | Computer-Generating emotional music: The design of an affective music algorithm[END_REF]. Our main contribution regarding these articles is that we rely on the valence emotional dimension (as well as the arousal dimension) to generate music. Dimensional approaches for affective computing have recently gained some interest [START_REF] Gunes | Emotion representation, analysis and synthesis in continuous space: a survey[END_REF]. In addition, this music is generated in real-time, and we rely on both expressive movement characteristics and emotion recognition for generating sound and music.

Augmented & interactive shows

We seek to explore the potential of using AR in the context of a ballet dance show to better convey the choreographer's message and suggest innovative artistic situations. Several augmented shows were conducted since about fifteen years ago. The evolution of technologies and systems in the field of AR allowed performance artists to use them as tools for their performances. First, The Plane [START_REF] Ranch | The plane[END_REF] unified dance, theatre and computer media in a duo between a dancer and his own image. With more interactive features, Hand-Drawn Spaces [START_REF] Kaiser | Hand-Drawn spaces[END_REF] presented a 3D choreography of hand-drawn graphics, where the real dancer's movements were captured and applied to virtual characters. Such interaction coupled with real time computing were achieved in "The Jew of Malta" [START_REF] Marlowe | The jew of malta[END_REF] where virtual buildings architecture cuts and virtual costumes were generated, in real time, depending on the music and the opera singer's position on the stage. More recently, Latulipe et al. conducted the Dance.Draw project, producing three different pieces over the course of the project. "A Mischief of mus musculus" features projected geometrical shapes linked to the dancers' motion. "Whispering to Ophiuchus" combined interactive and pre-programmed visuals. Finally, "Bodies/Antibodies" was a restaging of a modern choreography, adding biological-themed visuals linked to gyroscopes and accelerometers on the dancer's body [START_REF] Latulipe | Exploring the design spaces in technology-augmented dance[END_REF].

The use of augmented reality in arts in general (and performing arts in particular), and the design of interaction techniques for this context has witnessed a rising interest in the past few years, as shown by the appearance of various interest groups and special tracks in major human-machine interaction and augmented reality conferences (e.g. the digital arts community in CHI or the Arts, Media and Humanities program at ISMAR 2012).

SYSTEM ELEMENTS

The overall setup used for the show is composed of several interconnected modules, each fulfilling a particular function. The combination of those modules allowed giving birth to various interactions for the stage.

Capturing movement

The motion tracking system we selected (the MVN motion capture suit [35], see figure 2.a) is based on 16 inertial motion trackers positioned on the most important body joints. The body model is a biomechanical model composed of 23 segments. Sensors allow for absolute orientation of each segment, while positions are extrapolated from each inertial sensor. The position and orientation values are available in real-time. Such a technology allows a fast calibration of the system that fits the dancer's requirements. The two main drawbacks are the sensitivity to ferromagnetic objects that we may encounter on a stage floor, and the position drift.

Furthermore, a specific issue that we encounter with motion capture integrated within an augmented reality application is the need of a precise calibration to ensure a good correspondence between the motion tracking coordinate system and the stage coordinate system. It required making our own calibration system, in which we equipped the dancer with an IR LED footstrap to overcome the position drift problem. An IR camera with tracking algorithm has been developed to provide the position of the dancer on the stage. It compensates the drift each second by assuming that the foot is on the floor.

We also developed a vision-based system as an application case for previous research in video objects extraction and tracking. This allowed us to explore video-based emotion recognition from movement as well (although this second use of the system will not be covered in this paper). The vision system involves five different steps. Acquisition was performed using a static firewire digital camera (Point Grey Dragonfly 2 DR2-HICOL) at 30 frames per second and a resolution of 640x480 pixels. The system is then calibrated by clicking in the video on four markers on the stage whose coordinates in the real world were preliminary measured. This step allows mapping each pixel in the video on its corresponding location on the stage ground and vice-versa, by computing the two homographies between the stage ground and the camera's image plane. The stage ground itself is linked to the MVN frame. While running, the vision system performs an extraction of the spatio-temporal segmentation of the dancer. The aim is to extract the region of interest (the dancer) in the video. It is based on a background removal algorithm which computes the difference between the current image and a background reference image updated over time [START_REF] Caplier | Robust fast extraction of video objects combining frame differences and adaptative reference image[END_REF]. We used a Kalman filter [START_REF] Kalman | A new approach to linear filtering and prediction problems[END_REF] with a pseudo 2D constant speed model to track the center of gravity of the dancer. This absolute coordinate of the dancer's basin is used as a reference to recompute the MVN coordinates: the vision system acts as a proxy that receives the MVN coordinates, reprocesses the position of each segment according to the detected basin position, and sends the corrected skeleton over the network.

eMotion : recognizing emotions from movements

One of the goals of the CARE project was to use emotions as a parameter for generating and manipulating audio or visual elements. Our target was to recognize the emotions that the dancer expresses through his movement. For the sake of the show presented here, we did not try to recognize the emotions that the dancer was actually feeling (which can be very different from what he expresses), nor did we analyze the spectators' emotional responses to the show. Our computer-based gestural emotion recognition system relies on 3 modules. The Capture module has the role of acquiring data, the Analysis module has the role of extracting gestural and postural emotional cues, and the Interpretation module has the role of interpreting them as an emotion. From an architecture point of view, the eMotion software is a component-based system; each atomic operation (data acquisition from a sensor, extraction of a feature, interpretation from a set of features) is embedded in a pluggable/unpluggable communicative component which allows easy modifications of the system. We did not focus on identifying new expressive movement cues for emotion recognition but instead considered characteristics proposed by de Meijer [START_REF] Demeijer | The contribution of general features of body movement to the attribution of emotions[END_REF]. eMotion was developed in C++/Qt, and is fully described in [START_REF] Clay | La branche émotion, un modèle conceptuel pour l'intégration de la reconnaissance multimodale d'émotions dans des applications interactives : application au mouvement et à la danse augmentée[END_REF]. eMotion features a graphical interface (see figure 2

.b).

As explained above, the dancer wears the XSens MVN motion capture suit. The coordinates of each of these segments are refreshed and sent through the network at a frequency of 10 Hz. From the flow of coordinates, the eMotion software computes five movement characteristics: trunk curvature (bowed or stretched), arms positions (expanded or folded), vertical and sagittal directions of the movement, and velocity of the basin. In DeMeijer's work [START_REF] Demeijer | The contribution of general features of body movement to the attribution of emotions[END_REF] those movement characteristics are used to generate movement sequences that are classified by human evaluators within a discrete set of emotions. A statistical analysis allowed isolating each movement characteristic and its weight in the attribution of each emotion. We directly transposed these weights in a weight matrix within our Interpretation module, where each line corresponds to a movement characteristic and each column correspond to an emotion. Taking the movement characteristics as an input, the interpretation module chooses the maximum sum of each weighted movement characteristics over each of the six basic emotions defined by Ekman [START_REF] Ekman | Basic emotions. The handbook of cognition and 45-60[END_REF]: joy, fear, anger, sadness, disgust, and surprise. We focused on these emotions as their bodily expressions are already documented in the literature. At each frame, the eMotion software delivers an emotion label through the network as well as movement characteristics such as the speed vectors of the basin, hands and feet.

For example, in a movement where the dancer folds himself from a standing position to a kneeling position with his head on his knees: if the movement is initiated by the head, the trunk and shoulders drop, then the knees bend. The body coordinates are analyzed each tenth of second. To know whether the trunk is stretched or bowed, eMotion computes the center of curvature of the spine. If this center is in the back, the trunk is stretched; if it is in the front, the trunk is bowed. In this manner, eMotion extracts the characteristics of the folding movement: the trunk is bowed, arms are closed, movement is slow, downward, and without a sagittal direction. The weights corresponding for each value and each emotions are summed for each emotion (i.e. the weight matrix columns are summed). The maximum sum is elected as the recognized emotion. A key point of this system is that it relies on robust emotion theories and experimental work from the field of psychology [START_REF] Demeijer | The contribution of general features of body movement to the attribution of emotions[END_REF] [START_REF] Ekman | Basic emotions. The handbook of cognition and 45-60[END_REF]. The second one is that it functions in real-time. Each tenth of second, movement characteristics are computed and an emotion is interpreted, to better react to the dance improvisation. This provides a challenging issue as many relevant emotional cues are detected over longer periods of time. It also impairs the efficiency of recognition of quick emotional expressions; for example surprise or disgust are well recognized during the short time of their expression, but not always taken into account by subsequent treatments due to their brevity. Our system delivers both the recognized emotions and the expressive movement characteristics that led to this recognition; subsequent systems can rely on both outputs. Finally, the extracted movement characteristics are generic and not specific movements from the dance vocabulary.

Generating poses for a humanoid robot

Generating movement in robots from a user's movement is a well-studied issue. In our case however, we propose an interaction goes through a higher level of abstraction: emotions are extracted from the dancer's body expressions, and those extracted emotions act as triggers to generate the robot's poses. We selected the NAO humanoid robot as a display device [START_REF]Aldebaran robotics[END_REF]. Nao features several libraries that allow defining movements and postures, and its design is suitable for the stage. Several questions arise when using such an augmented modality for a performance: which posture should be displayed for which emotion? Should the robot express a given emotion always with the same posture? What if the system detects several times the same emotion from the dancer? The artistic goal was to have a physical, human-like, and expressive device to serve as a counterpart to the dancer. The robot took typical poses for each emotion, delivering a purely emotional expression to the audience.

The pose generation module is constituted of two separate submodules. The main sub-module is the NaoDriver sub-module. It takes an emotion label within the basic emotions set (anger, joy, sadness, fear, disgust and surprise) as an input. A set of poses was coded for each emotional state. The emotional poses were taken from [START_REF] Kleinsmith | Cross-Cultural Differences in Recognizing Affect from Body Posture[END_REF]. At the reception of an emotion label, the NaoDriver randomly chooses a pose in the corresponding set, and sends it to the robot. The second sub-module (called EmoState) is an adapter. As the eMotion software sends a continuous stream of analyzed emotions, its role is to smoothe that input over time and pick the more recognized emotional state to send to the NaoDriver. As such, emotion data from eMotion is first sent to EmoState, which then sends an emotion label to NaoDriver every 30 seconds. These modules were successfully tested and tuned during a preliminary interactive session with a dancer on March 29th, 2010 (figure 3.a).

The small size of the Nao robot, its limited battery capacity and the risk that it might fall during the performance made it difficult to use for the real performance show. As such, we replaced the real robot with small video sequences representing Nao shifting positions for the live show. The video sequences were triggered by the emotion recognized from the dancer's movements. (figure 3.b).

MuZICO: generating music from emotions

The music generation was performed with the goal of highlighting the emotions expressed by the dancer, and reinforcing the impact of those emotions on the audience. We were led to the creation of such software by former research by Jean-Louis Di-Santo [START_REF] Clay | Movement to Emotions to Music: Using Whole Body Emotional Expression as an Interaction for Electronic Music Generation[END_REF] towards the formalization of perceptual parameters describing electroacoustic music. The model that arose from this study mainly described sonic properties rather than musical ones, but the interesting part was the perceptual point of view. When we started generating more tonal music, we needed this perceptual approach to be able to control the generator intuitively. The field of emotions was the best start we could find to work at a higher level than tonality, rhythmic signature and other purely musical concepts.

The music generator was developed as a set of pureData patches. PureData was chosen as it already has most of the features we need built-in as integrated libraries, and it is easily expandable by creating libraries of abstractions and/or writing new external objects in C or C++. The audio part of the software was designed to be as versatile as possible, including beatboxes, chords players, melodies players, texture generators, soundscapes players and audio effects.

The software can be decomposed in two sets of modules. The first set contains the modules for musical rules generation. The rules created by the modules within this set are then fed to the other set, which regroups sound producing modules. MuZICO uses the valence-arousal (2D) continuous space of emotions as an input. As such, it features a translator that transforms emotion labels produced by the eMotion software into coordinates in this 2D space. Each musical rules generation module takes the coordinate along one axis (valence or arousal) to generate musical rules. For example, arousal is translated through rythmicity, whereas the valence coordinate of an emotion will be translated into a set of musical scales. Sound producing modules take the musical rules as inputs to produce the final piece of music. MuZICO and its interaction with eMotion is fully described in [START_REF] Clay | Movement to Emotions to Music: Using Whole Body Emotional Expression as an Interaction for Electronic Music Generation[END_REF].

Setting up an augmented reality system for the stage

We chose the 3DVIA Virtools software to edit the 3D world and its behavior. This software supports the main features we needed to realize an augmented reality setup. The first requirement was to have a collocated correspondence between the real and the virtual worlds. We also quickly had a system that supported stereoscopy, 3D video streams and generated the matching point of view for a specific seat in the theater. The behavioral aspect of the software gave us a strong flexibility to support the creativity of the dancer. Thanks to the editor part of the software, the dancer could see the 3D world he would later on be engaged with. 3D therefore provides the necessary common material to begin the discussion between the dancer and the "augmented reality engineer" during the many repetitions. We also added a component that binds input data from Moven Studio, eMotion and the 2D emotion space over the network.

With augmented reality and monoscopic rendering, we are only able to give the illusion to bring 3D content behind the screen. In the final part of the show, we implemented 3D stereoscopic rendering with projection on the backwall. Such techniques give us the capability to bring the 3D models out of the screen. It gives the spectator, who then wears 3D stereoscopic glasses, the impression that the dancer and the 3D content evolve in the same volume. The one constraint however is that the projection is made for one point of view in the theater, implying that the further the spectator is from this ideal viewpoint, the more disturbed the 3D scene will Today technologies are not capable to handle such a feature (it would require having as many stereo viewpoints as spectators), but our experiments will help us to define guidelines for optimal 3D scene creation. Great care also had to be taken during the live representation to avoid occlusion of the 3D content by the dancer, which may really disturb the spectators' depth perception.

To further take advantage of those capabilities during the show, we implemented a basic interaction technique that consisted in attaching the position of a 3d model to the hand of the dancer. To overcome the occlusion problem, we created a vertical offset between the hand of the dancer and the 3D model. The 3D model therefore appeared 1.5 meter above the dancer.

CREATING INTERACTIONS FOR MULTIMODAL RENDERING

As seen in section 2.1, in our context, there are three main issues for creating interaction combined with artistic choices: capturing the movement, defining the interaction language, and designing the output modalities.

First of all, we conceptualized the augmented ballet with the choreographer (figure 4 is an example of preliminary drawing). We stated that the gradual progress of the dancer, and more generally of a person in the virtual space, is the discovery of oneself through the machine. Through his explorations, the dancer realizes that he can find emotion and experiences in such a binary world. The question of the nature of the emotion arises: is it the emotion that the dancer feels, or the one that he expresses? The dancer might not at all feel the emotions he expresses and viceversa. Is it the emotion that the audience feels while watching the show? In this show, we focused on the emotion the dancer was expressing, to better communicate it through music and visuals. Psychologists see the feeling as a component of the emotion [START_REF] Sander | A system approach to appraisal mechanisms in emotion[END_REF] What about the other components, such as motor expression, and the neurophysiologic processes?

We then elaborated the scenic environment. In its common representation, the computer world has an orderly, smooth, clear and clean aspect. The virtual world is a blank sheet for the imagination of the fruitful creator, where everything that is done can be undone. We built the scenic space as a blank and empty space, considering the dancer as an experimental subject. The augmented reality engineer was positioned on the stage just next to the dancer, between art and science: the two fields were merged into a single one. Figure 4 is an exploratory drawing illustrating this concept, which conducted to the staging of the augmented ballet.

Designing interactions for performing arts involves taking into account their artistic impact. On the other hand, the artist's proposal can be based on the proposed interactions: in this case, the artist seeks to show the work behind the results. The dancer exposes the technology both as a process and as a result, which heavily engages the audience. In the show "CARE project: staging of the research process", we deployed five main interactions. Two were direct interactions. The dancer manipulated a virtual cube attached to his right hand, and avatars copied his movements. Two were indirect. The dancer's expressed emotions were recognized, and used to generate sound, music, and expressive poses in a humanoid robot. Finally, a last direct interaction was staged, where the dancer generated 3D ribbons from his hands (see Figure 2). This particular interaction proved very interesting, as it gave to the audience the illusion to superpose the action and perception spaces of the dancer: thanks to stereoscopy, the 3D ribbons seemed to appear out of the dancer hands. This superposition of action and perception spaces were just in appearance though; as the stereoscopy was calculated to appear from the audience point of view, the dancer had to watch the back screen to follow the evolution of the ribbons. From the dancer's point of view, the action and perception spaces were separated.

SETTING UP THE SHOW: ASSEMBLING THE PIECES

In March 2011, we presented at the Casino de Biarritz an augmented dance show: "CARE project, staging of a research process". The show was composed of two parts. The first part was a danced conference. The project coordinator presented the CARE project and its different findings, from the theoretical ones to its concrete applications. This presentation was accompanied by demonstrative applications and videos, and a dancer improvised a choreography on the sound of the presenters voice. The dancer also included demonstration of both software and hardware developed during the project within his dance. The second part was an augmented dance show showing the research process that had been undergone both by scientists and artists in the preceding years. It was itself divided in seven parts, showing the gradual improvements and findings.

Artistic proposal: choice of augmentations

The "scenario" of the ballet involved a dive from the real and organic world to the virtual and synthetic world as the show went on, before brutally reversing to reality at the end of the show. As such, visual augmentations were more and more prevalent as the show went on. For the audio part, we defined a musical line where instruments were added along the show and where sounds gradually translated from organic sounds to more synthetic ones. A documentary of the show is available online at [36].

Description of the show

The first part of the show took the form of a danced conference explaining the project. In this conference we demonstrated the interaction techniques that were developed during the CARE project. In particular, due to the size of the robot, we decided to include the dancer-robot interaction in this first part. As the robot was too small for the stage, we used a video media: the dancer's expressed emotions were captured, smoothed with the EmoState sub-module, and instead of directly animating the robot on stage, a video of the corresponding pose was dynamically selected and played on stage, allowing the audience to clearly see the robot's movements (as shown in figure 3.b).

The second part of the show took the form of an augmented dance improvisation (see figure 5). In a first part, the dancer without any augmentation. In the second part, the only augmentation to appear on stage was a virtual shadow (figure 5.a). This allowed the dancer to play with his real shadow (from spot lights) and the virtual shadow (which was voluntarily incoherent with the direction of the spot lights). The third part of the show was announced by the appearance of a virtual set in the background that extended the real stage. The dancer could then play with virtual elements on stage: moving a virtual cube and dancing with virtual avatars (figures 5.b and 5.c). Avatars were either animated in real-time from the dancer's movement or playing pre-recorded movements. After this gradation, all augmentations were shut off again (figure 5.d). The final part of the show portrayed the future works of the project, where the dancer could generate 3D ribbons from his hands (figure 5.e).

Theater light technician and augmented reality setup

Because the light technicians and the augmented reality engineer's works both deal with the light brightness, it was necessary and very important that they took each other into account. Compared to a theater spotlight, video projectors are often less luminous. In our case we chose to have spotlights at both side of the stage that would highlight the dancer. Videoprojection displayed the assets with which the dancer interacted during the show.

In this setup, we introduce the augmented reality engineer, who controls the transitions between the different parts of the show. This new actor in the show production has to be synchronized with all the other existing theatre technicians (light and sound).

Physical setup and element connections

The core of our setup was composed of four modules: the motion capture module to capture the dancer's movements, the eMotion software for extracting movement characteristics and emotions, MuZICO for generating music and sounds, and Virtools for displaying and organizing the different visual augmentations. The system was composed of three PCs on a network (see figure 6). The first PC gathered the MVN motion capture application and the eMotion software. This PC sent three kinds of data over the network: motion capture data from the MVN studio application, movement characteristics (e.g. trunk curvature, arms expansion) and emotions from the eMotion software. The second PC featured the interactive virtual world. It received the motion capture data and the recognized emotions. The visual output was then derived to two video-projectors. This set of two video-projectors allowed to project on both the back and the floor of the stage. The third PC received movement characteristics and emotions. It featured the music generation program that used recognized emotions as an input to create adequate pieces of music. Specific sounds were also played when receiving some movement characteristics.

Feedback from the dancer/choreographer

The artist with whom we collaborated during this project is both a choreographer and the dancer that performed during the show. As a long-term user of the technologies we developed, his feedback is highly relevant from a qualitative aspect. The performer was asked several questions about the interaction, both from a dancer and a choreographer point of view. As the show was improvised, the choreography was quite high-level and mainly consisted in cutting the show into several parts, and defining improvisation guidelines for each part. As the performer highly contributed to the development of the eMotion software and its use over time, no question was asked on this part of the system. The questionnaire mixed rated questions (e.g. "Do you take the equipment into account when improvising? 1-not at all, 10-completely") and open questions (e.g. "How do you approach the musical interaction, where you movement and expressed emotions generates the music?"). The performer was asked to freely comment on rated questions as well. Questions were about the equipment, the interactions ("Do simple interactions allow for better creativity?"), and the augmentations ("How did you take the visual augmentations into account?"). As a choreographer, the technology set on stage (motion capture suit, emotion recognition, music generation and visual augmentations) did not restrain creativity but rather oriented it by setting a set of constraints, which could be played with. Music generation was not really a decisive point for the choreography; it was more important from the dancer's point of view, as a support for improvisation.

As a dancer (sentences within quotation marks are translations from the performer's answers), it appeared that even if the motion capture suit was indeed a disturbance (7/10, 10 being "extremely disturbing"), the constraints it gave also "acted as original guidelines for improvisation, which then leant towards expressing the relationship between dance and technologies". As such the dancer saw the suit "rather as a source for improvisation than as a disturbance" ("Do you take the equipment into account when improvising? How?"). The performer felt he had only an imprecise control over the generated sounds and music (6/10, 10 being "complete control"). However, he heavily used it as a basis for improvisation ("Did you take the generated music into account for improvising?" 9/10, 10 being "extensive use of the generated music for improvisation"). Immobility was particularly interesting, "as it triggered silence, and allowed creating subtle sounds with subtle movements. The generated sounds were a feedback of which emotion was recognized by the system, giving [the performer] cues about his interaction with the emotion recognition system". The performer could identify which sounds were related to his gestures and which sounds were related to the emotion he was portraying, allowing him to play with both.

As for visual augmentations, the dancer was really taking into account the visual augmentations for improvising (9/10, 10 being "create a full relationship with the visuals"), as the visuals "provided a feedback from the systems". In particular, the dancer heavily relied on the relationship that he created between the virtual avatars and himself (9/10). Virtual avatars were either copying the dancer's movements or replaying pre-recorded choreographies. Each dancer having some personal movement subtleties, the virtual avatars were a copy of the real dancer. This particular interaction gave the dancer an opportunity "to improvise a choreography portraying the impacts that the proposed technology could have on [his] own movement, playing with the technological issues to better show the limitations of [our] setup". Simpler interactions (moving the cube and the 3D ribbons) also held a great creative potential (8/10, 10 being "great creative potential) as their simplicity did not overpower the choreography. As such, they allowed "setting the dance as the primary message, only supplementing it with effects to be played with".

The performer's point of view

Gaël Domenger, choreographer and dancer in the show, provided his insights about the show:

The first time I got to work, as a dancer, with new technologies and motion capture, was in the years 2000 and 2001 on a project call "STAMINA", conducted by the choreographer Bianca Van Dillen in Holland. STAMINA was about an observation of the involvement of the skeleton in the movement. The CARE project was the occasion to bring back the reflection from STAMINA; it also challenged me and forced me to go deeper in my questioning

about dance and new technologies, and to come back on my practice of dance with an analytical eye. The questions about emotion and movement pushed me to go far back to the fundamentals of dance and question myself on what "interaction" meant in this new frame of work and research. It has been the

occasion for me to discover that scientists had a similar thirst to mine about exploring and experimenting and dig out from their mind new ideas and new concepts. I chose to develop, on my side, new techniques of improvisation based on observations of the relation that my movement was establishing with space and time and an analysis of my breathing while I was dancing. When projection and music got in, to interact with me, I forced myself to stay focus on the line I drew during that first period of research. The work I did to raise awareness about my breathing while I was dancing and use it as a tool to express emotion through motion gave interesting results when music got out of my movements with the help of motion capture. Then again I had to stay focus and not let the music lead me out from the path of expressing emotion strictly with my movement. Interacting with the robot "Nao" helped me a lot to understand the state of the research we were in understand the necessity for me to draw a line of research that strictly concerned dance and emotion and let technology have its own reaction without trying to adapt to it, but instead give clear and true impulse motivated only by my emotions.

Feedback from the audience

The second representation of the show was presented in front of an audience of about 120 people. As the show was a premiere for most of the scientists involved, we focused on the show setup and could not conduct a full validation process with the audience. We did, however, gather some feedback through the form of questionnaires that were distributed at the beginning of the show and informal interviews in order to obtain guidelines and lead hypotheses for future works. The questionnaires were meant to be filled out quickly, with three open questions: "What did you like?" what did you not like?" and "what improvements would you bring to the show?". We collected 97 questionnaires. From these feedbacks we realized some points and drew some hypotheses to be verified in future works. For a non-researcher, it is quite difficult to apprehend the scientific world. The show presented the CARE project scientific results in a both artistic and playful way; scientists were directly in front of the audience, which the audience appreciated. People enjoyed the "magical aspect" of the show, which kept them involved. The audience enjoyed the augmentations and the use of various technologies in a show. It also appreciated seeing the technology itself and not only its results. Visuals were favored over the generated music. Direct interactions were preferred, especially the 3D ribbons projected in stereoscopy (the audience wore stereoscopy glasses). The audience hence had a very strong interest in following this collaboration between art and science; this interest went further, as it sparkled reaction from medias in the form of articles and an interview on local television. It appears from those feedbacks that direct interactions were more widely favored. Moving a cube by attaching it to the dancer's right hand is a straightforward interaction, understood by everyone. Such simple interactions can still be very interesting from an artistic point of view: their simplicity does not overpower the role of the choreography, and they merge in a smoother way in the artistic proposal. Our hypotheses are that "innovative and magical" interpretation by the audience is not correlated with the directness of an interaction, but rather with its staging. Simpler, direct interactions allow the audience to focus on the artistic output.

CONCLUSION AND FUTURE WORKS

In this paper, we presented five interactions used in augmenting an improvised dance show. The system that allows those interactions is composed of several interconnected modules. This system was used to produce an augmented piece entitled "CARE project: staging of a research process " is both a performance and a demonstration that aims at presenting, in an aesthetic way, the interactions presented in this paper.

Feedback from the dancer and choreographer was extremely positive. The generated music and sounds, and the manipulation of the visual augmentations had three roles. It was a feedback for the emotional recognition of the dancer's expression; a new channel for improvisation, since the dancer could experiment with the sounds and visuals by using his body; and a basis for the movement. Far from restraining the creativity, the different kinds of interaction (direct or expressive) have different strengths and weaknesses. In particular, interactions based on emotion recognition proved difficult to perceive for the audience, who did not react as much as with direct interactions. This has led us to realize that producing an augmented show raises the difficulty of staging the technologies: the audience must be given the freedom to interpret what is rendered on stage, should it be visual or audio.

Validating an augmented show, and assessing how the technology is perceived by the audience and their impact on the audience is another critical issue. Questionnaires have some limitations and may be hard to manage in an uncontrolled environment, where the audience is eager to leave once the show is over. This issue has been tackled by Latulipe et al. [START_REF] Latulipe | Evaluating longitudinal projects combining technology with temporal arts[END_REF].

Future works involve focus on this issue. Producing an augmented show gathering so many technological elements proved difficult but rewarding. The experience gained will help us in the production of future shows, for which we will be able to focus more on the validation issues.

Our current works involve designing new interactions for augmenting a stage. The feedback from both the audience and the choreographer led us to focus on simple, direct interaction, as they work better when presented to an audience. Moreover, they are more easily apprehended by the artists, who can concentrate on artistic exploration. In particular, we are now working on 3D painting on stage. The success of the generation of 3D ribbons from the hand of the dancer led us to develop a system where the dancer can generate 3D surfaces with his hands. The choreography hence mixes with painting on stage in a 3D space.

  (a) The motion capture suit. (b) The eMotion software.

  (a) G. Domenger in training with the Nao robot. (b) Use of dynamically triggered videos.

Figure 2 :Figure 3 :

 23 Figure 2: Capturing the movement (a) and recognizing emotions (b)

Figure 4 :

 4 Figure 4: Drawing of a scene of the augmented show. The rightest drawing pictures the scene shown in figure 1.b. © Elodie Decarsin

(a) Scene 2 :

 2 The dancer, with a real (bottom) and virtual (up) shadows. Scene 7: Generation in real time of 3D ribbons from dancers hands.

Figure 5 :

 5 Figure 5: Illustration of the different scenes in the augmented show.

Figure 6 :

 6 Figure 6: The physical setup for the show.

ACKNOWLEDGEMENTS

The work presented in the article was funded by the French National Research Agency under contract CARE (Cultural Experience: Augmented Reality and Emotion) from December 2007 to March 2011. The authors want to thank Issam Rebbai for his implementation work on the Nao robot and Elodie Decarsin, the stage designer of this experimental show. The authors would also like to thanks the reviewers for their fruitful comments.


Daniel Tudball-Smith

Justin S Leontini

John Sheridan

David Lo Jacono

Streamwise forced oscillations of circular and square cylinders

The modification of a cylinder wake by streamwise oscillation of the cylinder at the vortex shedding frequency of the unperturbed cylinder is reported. Recent numerical simulations [J. S. Leontini, D. Lo Jacono, and M. C. Thompson, "A numerical study of an inline oscillating cylinder in a free stream," J. Fluid Mech. 688, 551-568 (2011)] showed that this forcing results in the primary frequency decreasing proportionally to the square of the forcing amplitude, before locking to a subharmonic at higher amplitudes. The experimental results presented here show that this behavior continues at higher Reynolds numbers, although the flow is three-dimensional. In addition, it is shown that this behavior persists when the body is a square cross section, and when the frequency of forcing is detuned from the unperturbed cylinder shedding frequency. The similarity of the results across Reynolds number, geometry, and frequency suggests that the physical mechanism is applicable to periodic forcing of the classic von Kármán vortex street, regardless of the details of the body which forms the street.

This study reports on an experimental study of cylinders undergoing forced streamwise oscillations. Streamwise forcing is just one method of active open-loop flow control that can be employed on the cylinder wake. Others include, but are not limited to, transverse (cross-stream) oscillation, [START_REF] Bishop | The lift and drag forces on a circular cylinder oscillating in a flowing fluid[END_REF][START_REF] Williamson | Vortex formation in the wake of an oscillating cylinder[END_REF][START_REF] Blackburn | A study of two-dimensional flow past an oscillating cylinder[END_REF][START_REF] Leontini | Wake-state and energy transitions of an oscillating cylinder at low Reynolds number[END_REF] rotational oscillation of the cylinder, [START_REF] He | Active control and drag optimization for flow past a circular cylinder[END_REF][START_REF] Poncet | Topological aspects of three-dimensional wakes behind rotary oscillating cylinders[END_REF][START_REF] Thiria | The wake of a cylinder performing rotary oscillations[END_REF][START_REF] Adamo | Spatiotemporal spectral analysis of a forced cylinder wake[END_REF] constant rotation of the cylinder, [START_REF] Badr | Unsteady flow past a rotating circular cylinder at Reynolds numbers 10 3 and 10 4[END_REF][START_REF] Mittal | Flow past a rotating cylinder[END_REF] and actuation by synthetic jets from the surface of the cylinder and base bleed. [START_REF] Monkewitz | Absolute instability in the near-wake of two-dimensional bluff bodies[END_REF][START_REF] Karniadakis | Frequency selection and asymptotic states in laminar wakes[END_REF][START_REF] Feng | Circular cylinder vortex-synchronization control with a synthetic jet positioned at the rear stagnation point[END_REF] Previous studies of streamwise forced oscillation [START_REF] Griffin | Vortex shedding from a cylinder vibrating in line with an incident uniform flow[END_REF][START_REF] Barbi | Vortex shedding and lock-on of a circular cylinder in oscillatory flow[END_REF][START_REF] Konstantinidis | The timing of vortex shedding in a cylinder wake imposed by periodic inflow perturbations[END_REF][START_REF] Konstantinidis | Symmetric vortex shedding in the near wake of a circular cylinder due to streamwise perturbations[END_REF] have investigated the synchronization between the forcing and the vortex shedding. These studies report that the vortex shedding locks to a subharmonic of the forcing (vortex shedding at half the frequency of the forcing), particularly for forcing frequencies around twice f so , the shedding frequency from the unperturbed cylinder. Barbi et al. [START_REF] Barbi | Vortex shedding and lock-on of a circular cylinder in oscillatory flow[END_REF] and Ongoren and Rockwell 18 also report that a synchronized symmetric wake structure, different from the classic von Kármán street, can be achieved depending on the amplitude and frequency of forcing, typically being found for high values of the frequency. Yokoi and Kamemoto [START_REF] Yokoi | Vortex shedding from an oscillating circular cylinder in a uniform flow[END_REF] found a series of synchronized states, depending on forcing frequency, amplitude, and Reynolds number. These states could be synchronized to either the forcing frequency or to its first subharmonic, particularly when the forcing frequency was close to f so , a finding confirmed and extended by Cetiner and Rockwell. [START_REF] Cetiner | Streamwise oscillations of a cylinder in a steady current. Part 1. Locked-on states of vortex formation and loading[END_REF] Perdikaris, Kaitsis, and Triantafyllou 21 numerically investigated a cylinder oscillating at the unperturbed shedding frequency, f so , for a Reynolds number Re = 400. This study found that, for low amplitudes, the flow remained in a state similar to that of an unperturbed cylinder. At high values of amplitude, they found locking to the first subharmonic of the forcing. At moderate amplitudes, they a) Electronic mail: justin.leontini@monash.edu. reported chaos, with disorganized vortex shedding. This chaos was attributed to mode competition between the reflection-symmetric forcing, and the spatio-temporally symmetric Kármán street. A recent numerical study [START_REF] Leontini | A numerical study of an inline oscillating cylinder in a free stream[END_REF] at Re = 175 using fine increments of amplitude and forcing at f so found that, for low to moderate amplitudes, the primary frequency of vortex shedding, f s , decreased with increasing non-dimensional amplitude A * = A/D, where A is the amplitude of forcing and D is the cylinder diameter, scaling as f s ∝ A *2 as suggested in Perdikaris, Kaitsis, and Triantafyllou. [START_REF] Perdikaris | Chaos in a cylinder wake due to forcing at the Strouhal frequency[END_REF] The dynamics were shown to be dictated by the interaction of this new primary frequency f s and the forcing frequency, typically resulting in a quasi-periodic, amplitude and frequency modulated flow. However, nonlinear synchronization was found when the primary frequency and driving frequency made an integer ratio, and a series of subharmonic modes was discovered from the 1/8th mode at low amplitudes all the way to the first 1/2 subharmonic.

The current study has two primary aims. The first is to investigate experimentally whether the dynamics reported in the previous numerical studies [START_REF] Perdikaris | Chaos in a cylinder wake due to forcing at the Strouhal frequency[END_REF][START_REF] Leontini | A numerical study of an inline oscillating cylinder in a free stream[END_REF] persist at higher Reynolds numbers. The second is to investigate whether the dynamics are similar for differently shaped bodies. Also presented are results from experiments where the forcing frequency is detuned from f so .

Experiments were conducted using the Fluids Laboratory for Aeronautical and Industrial Research (FLAIR) water channel at Monash University. This facility is a free surface water channel with 800 mm depth, 600 mm width, and 4000 mm length with a test section of 500 mm width and 650 mm depth. This facility has been previously used for similar bluff-body wake studies. [START_REF] Nazarinia | Flow behind a cylinder forced by a combination of oscillatory translational and rotational motions[END_REF][START_REF] Sherry | An experimental investigation of the recirculation zone formed downstream of a forward facing step[END_REF] Square/circular cylinders of 25 mm width/diameter were oscillated using a high resolution stepper motor connected to a traverse, which allowed for linear translational motion. [START_REF] Nazarinia | The three-dimensional wake of a cylinder undergoing a combination of translational and rotational oscillation in a quiescent fluid[END_REF] Position was confirmed using a hollow bore optical encoder attached to the shaft of the stepper. Lift and drag forces on the cylinders were measured using strain gauges on a load cell positioned above the free surface at a sampling frequency of 100 Hz. Both square and circular cylinders were attached to the load cell such that they passed through the test section with the end of the cylinder finishing <10 mm from the bottom of the channel. Reynolds number was defined using the freestream velocity and the cross-stream width of the body. The range of Reynolds number examined was Re ∈ [1500-5500] and [3750-6300] for the square and circular cylinders, respectively.

The shedding frequency f so was determined from a stationary body using frequency analysis of the lift force.

The frequency content of the lift force was determined using a method based on a fast Fourier transform (FFT). The total time signal was broken into a series of segments, which overlapped by half of the length of the segment. An FFT was taken of each segment, and the results of all segments were then averaged. This process essentially adds a type of high-pass filtering, removing the influence of very low frequencies. Spectra produced using this method are shown on the left-hand side of Figure 1. For all values of Re, f s initially decreases with A *2 and synchronizes to a period doubled state at higher amplitudes. However, the transition between these two states is more abrupt for the circular cylinder case. Note that the symbols for each Re in this figure overlap when f s /f d = 0.5.

Figure 1 shows the development of the frequency content for a circular cylinder, at Re = 5000, with increasing amplitude of oscillation. The three example spectra on the left are for A * = 0.05, A * = 0.12, and A * = 0.20, respectively. The frequencies have been normalized by the driving frequency f d = f so . These images show that as the amplitude is increased, the primary frequency of response f s (marked with the square point) is lowered away from f so , before reaching f d /2. The map on the right-hand side of Figure 1 shows this trend very clearly. The contours on this image represent the energy for a given frequency component as a function of A * . This figure was produced by essentially "stacking" the spectra for each value of A * next to each other. These contours show that the frequency content of the flow response is a smooth function of the amplitude, up to A * ≃ 0.12. For A * < 0.12, f s (again highlighted with the square points) decreases apparently quadratically. The contours also show that over this range of A * , two other frequencies contain small, but non-negligible energy; one at f d -f s , and another at f d + f s . At A * ≃ 0.12, there is a sudden drop in the primary frequency to f d /2, and the flow response remains locked to this subharmonic of the driving frequency with further increases in amplitude. This behavior is in agreement with the numerical work of Leontini, Lo Jacono, and Thompson 22 despite an order of magnitude difference in Re. This observation is not necessarily expected, given the high level of three-dimensionality in the experiments.

The aforementioned numerical work reported that the initial decrease in f s is of the form f s /f d = 1 -aA *2 . Figure 2(a) shows f s as a function of A * for three values of Re. The data for A * lower than that at which the flow locks to the subharmonic of the driving frequency have been fitted with a power-law curve of the form f s /f d = c -aA *b in a least-squares sense allowing all 3 constants (a, b, and c) to vary.

Letting the constant c "float" accounted for any experimental error in determining f d from the unperturbed cylinder experiments. In all cases, the value of c was seen to vary slightly but remained close to unity. The curves corresponding to these fits are plotted in Figure 2(a), and the coefficients are shown in Table I. Table I shows that for all the values of Re, the exponent b remains close to 2. The small variation in b indicates that even at these higher values of Re, the primary frequency for the circular cylinder decreases with A *2 .

To further test the generality of the findings, a similar approach was followed using a square cylinder. Figure 3 shows examples of spectra for the square cylinder at Re = 4500; similar to Figure 1, three example spectra are shown on the left-hand side, while the image on the right shows contours of energy as a function of frequency of response and amplitude, produced by stacking the spectra for each A * next to each other. The square points on the figure mark the primary frequency of response for each value of A * . Similar to the circular cylinder, this figure shows that for a square cylinder, the strongest frequency decreases from f ≈ f so as the amplitude of oscillation is increased, before locking to f = f so /2. Again, energy at the secondary frequencies of f d -f s and f d + f s is evident in the contours.

Figure 2(b) presents the primary frequency of shedding f s as a function of A * , for a series of values of Re. The square cylinder follows a similar trend to the circular cylinder. At low amplitudes f s decreases from the driving frequency with increasing A * , and transitions to a period doubled locked-in state at higher amplitudes. Though some differences are observed in the transition to this locked subharmonic state, the results suggest that this behavior is generic for spatio-temporally symmetric vortex street wakes, regardless of the details of the body, which formed them. However, wakes which do not possess this symmetry (such as those generated from non-symmetric bodies), or which have different vortex configurations (such as those that could be produced by more complex forcing functions) could behave in a different manner.

The trend of f s decreasing with A *2 , found for the circular cylinder, is recovered from the square cylinder data, at least for A * < 0.15. As for the circular cylinder, making a power-law fit to f s as a function of A * results in exponents close to 2. The exponents of these fits, for a range of values of Re, are presented in Table II. Close inspection of the data shows some impact of the body geometry. First, for the square cylinder the locking to a period doubled state occurs at a higher non-dimensional amplitude (A * ) compared to the circular cylinder. Second, for the circular cylinder, the transition to this period doubled region occurs abruptly. However, for the square cylinder, the frequency initially follows the trend of A *2 until around A * = 0.15, and then rapidly, but smoothly, decreases to f d /2 over the range 0.15 ≤ A * ≤ 0.2. This transitional region appears to have increased disorder with a broader spectrum response than the other regions. The spectrum shown in Figure 3(b), for A * = 0.20 when Re = 4500 shows this. Although it is still possible to find a peak frequency in the signal, it is not as distinct or dominant. A broad spectrum response is observed at amplitudes in the transition range, with no significantly dominant peak. It is possible that this broadband response is due to a competition between modes, one with vortex shedding at a frequency dictated by A *2 , the other at the subharmonic frequency of the driving, f d /2. However, this mode competition does not simply lead to periods of time oscillating in one mode, then periods in the other mode. Rather, the flow oscillations are quite irregular, and so a definitive conclusion on the cause of this chaos has not been found.

The circular cylinder data of Figure 1 and the square cylinder data of Figure 3 show that there is some dependence on Re, in terms of the amplitude at which locking to the period doubled state occurs, and the steepness of the initial variation of f s with A * (this is also reflected in the variation of a in Table II). This variation appears to be rather complex, as the amplitude at the period doubled locking varies non-monotonically. This perhaps is not surprising, considering that there are a number of subtle changes in the wake and shear layer formation that occur in the range of Re tested, even for a stationary cylinder. [START_REF] Bloor | The transition to turbulence in the wake of a circular cylinder[END_REF][START_REF] Roshko | Perspectives on bluff body aerodynamics[END_REF] This complicated dependence on Re highlights the underlying complexity of even the canonical flow of the cylinder wake, and the difficulties faced when trying to devise any control method, including periodic forcing. The studies of Parezanović and Cadot, [START_REF] Parezanović | The impact of a local perturbation on global properties of a turbulent wake[END_REF][START_REF] Parezanović | Experimental sensitivity analysis of the global properties of a two-dimensional turbulent wake[END_REF] using a passive control method of a small cylinder positioned close to the main cylinder, also highlighted this complexity, showing that the global frequency of the flow could be modified independently of the length scale of the flow.

Regardless, it is clear that the dominant features, namely a decrease in primary frequency, followed by synchronization to a period doubled state, are the same for all values of Re tested. The similarity of this behavior to that from low-Re two-dimensional simulations suggests that the dynamics of the wake are still governed by structures that are essentially two-dimensional, even at higher Re. A number of studies exist that indicate forcing can increase the correlation in the spanwise direction. Poncet [START_REF] Poncet | Topological aspects of three-dimensional wakes behind rotary oscillating cylinders[END_REF] and Thiria, Goujon-Durand, and Wesfried 7 both show that by rotationally oscillating the cylinder near the Strouhal frequency (the vortex shedding frequency from the unperturbed cylinder), spanwise correlation in the wake can be significantly increased. Leontini, Thompson, and Hourigan 30 also showed that transverse oscillations near the Strouhal frequency could delay the onset of three-dimensional structures. A significant difference between rotational or transverse oscillation, and the streamwise oscillation of this study, is the symmetry of the forcing compared to that of the vortex street. The vortex street is spatio-temporally symmetric; evolution forward by half a period plus a reflection about the wake centreline gives the same solution. Rotational and transverse forcing share this symmetry. However, streamwise oscillation does not. Despite this, it might be expected that the forcing leads to an increase in spanwise correlation, and this might go some way to explain the similarity of the behavior at these higher Re to the behavior in two-dimensional simulations.

The data presented for the circular and square cylinder thus far are for driven oscillations at the natural shedding frequency, i.e., f d = f so . An outstanding question from these studies is that of whether it is the driving frequency, or the unperturbed cylinder frequency, that controls the dynamics, or if there is any interaction between them. To investigate this, a series of experiments have been conducted at a frequency slightly detuned from the natural shedding frequency, for the circular cylinder, at Re = 5000.

Forcing at a slightly detuned frequency across the tested range ( f d = f so ± 7.5%) exhibits the same overall trend in reducing shedding frequency until locking to a period doubled state at higher amplitude as shown in Figure 4. Figure 4(a) shows that subharmonic locking is to the driving frequency f d , causing the variation in locking frequency value. If these curves are normalized by forcing frequency instead of natural shedding frequency, as shown in Figure 4(b), the locking frequency collapses on to f s /f d = 0.5. However, the forcing frequency has no effect at small amplitudes on the shedding frequency as shown in Figure 4(a). The shedding frequency is equal or close to that of the natural shedding. For low amplitudes, the trend is still well approximated by a quadratic, regardless of the forcing frequency over the range tested. The coefficients of the power-law fits for the detuned shedding frequencies are presented in Table III. For all the values of the frequency tested, the exponent b remains close to 2.

In conclusion, this study has investigated the modes of response and trends in shedding frequency from a cylinder performing streamwise oscillations, where the frequency of this oscillation is equal, or close, to the vortex shedding frequency from an unperturbed cylinder.

For a circular cylinder, it has been shown that the initial trend of the primary frequency of shedding decreasing with A *2 holds at values of Re 6300. The critical amplitude at which this trend ends with a sudden transition to a period doubled state has been shown to be a function of Re.

This initial trend of decreasing shedding frequency with A *2 also applies when the cylinder cross section is changed from circular to square. However, the transition to the period doubled state is not as abrupt with increasing A * for the square, as the primary frequency decreases smoothly to the subharmonic of the driving frequency over a small range of A * . This may be due to competition between two modes of shedding.

At moderate amplitudes, it is the driving frequency, and not the unperturbed shedding frequency, that controls the dynamics. Experiments where the driving frequency for the circular cylinder is

FIG. 1 .

 1 FIG.1. Power spectral density of the lift force on the circular cylinder at Re = 5000 as a function of A * . Line plots labeled (a), (b), and (c) are taken at A * = 0.05, 0.12, and 0.19, respectively, and are indicated by the dashed lines on the power spectral density isocontour. The square points represent the primary frequency, f s for each A * .

FIG. 2 .

 2 FIG. 2. Primary shedding frequency, f s , normalized by the driving frequency f d , as a function of A * for (a) the circular cylinder and (b) the square cross section, at various Re. Points are measured frequency, lines are a power-law fit of the form f s /f d = c -aA *b (see TableIand II for the coefficient values). For (a): Re = 3750 (•), Re = 5000 ( ) Re = 6300 ( ). For (b): Re = 1500 ( ); Re = 2500 (△); Re = 3550 (G); Re = 4500 ( ); Re = 5500 ( ). For all values of Re, f s initially decreases with A *2 and synchronizes to a period doubled state at higher amplitudes. However, the transition between these two states is more abrupt for the circular cylinder case. Note that the symbols for each Re in this figure overlap when f s /f d = 0.5.

FIG. 3 .

 3 FIG. 3. Power spectral density of the lift force on the square cylinder at Re = 4500 as a function of A * . Line plots labeled (a), (b), and (c) are taken at A * = 0.11, 0.20, and 0.30, respectively, and are indicated by the dashed lines on the power spectral density isocontour. The dotted symbols represent the maximum frequency, f s for each A * .

15 FIG. 4 .

 154 FIG.4. The behaviour of f s as a function of A * , when the driving frequency f d is detuned from the unperturbed shedding frequency f so , for the circular cylinder at Re = 5000, for values of f d / f so : 0.925 (▽), 0.950 ( ), 0.975 (△), 1.000 (•), 1.025 (•), 1.050 ( ), 1.075 ( ). (a) The primary shedding frequency, f s , normalized by f so . (b) The primary shedding frequency, f s , normalized by f d . The plots clearly show that the shedding frequency varies smoothly with A *2 from the unperturbed cylinder case as A * is increased from A * = 0, and that the flow finally synchronizes to a subharmonic of the driving frequency, regardless of the driving frequency.

TABLE II .

 II Values of the coefficients in the fit f s /f d = c -aA *b , for A * < 0.2 for the square cylinder. For the range of Re tested, the exponent b remains close to 2. Note that the constant parameter c is close to unity.

	Re	a	b	c
	1500	8.05	1.93	1.02
	2500	11.4	2.26	1.01
	3550	9.26	2.25	0.99
	4500	3.78	1.88	0.99
	5500	6.40	2.15	0.99

TABLE III .

 III Values of the coefficients of the power-law fit f s /f d = c -aA *b , for the circular cylinder at Re = 5000 as a function of the driving frequency f d . The exponent b constantly remains close to 2.

	f d / f so	a	b	c
	0.925	10.9	2.03	0.991
	0.950	14.4	2.11	0.992
	0.975	11.8	2.02	0.993
	1.000	16.2	2.10	0.993
	1.025	14.8	2.03	0.993
	1.050	28.2	2.25	0.990
	1.075	12.8	1.91	0.995

detuned from the unperturbed shedding frequency show the same trends as the case where the driving frequency is equal to the unperturbed shedding frequency, although the critical amplitude for the transition to the period doubled state, and the slope of the primary frequency of shedding prior to this transition, are functions of the driving frequency.

The authors would like to acknowledge the financial support of the Australian Research Council through Grant Nos. DP0774525 and DP110102141 as part of the discovery projects scheme, and the Australian Postdoctoral Fellowship of J.L.