Ngurah Indra
email: indra@unud.ac.id

Jean-Marie Bonnin
email: jm.bonnin@imt-atlantique.fr

On the Performance of VDTN Routing Protocols with V2X Communications for Data Delivery in Smart Cities

Keywords: VDTN, routing protocols, V2X, smart cities, energy efficiency

VDTN concept and V2X communications capabilities can be utilised for an efficient data collection and delivery in smart cities. Here we present performance comparison of four lightweight VDTN routing protocols: First Contact, Epidemic, Spray and Wait, and MaxProp in terms of its percentage of message delivered, latency, overhead ratio, and hop counts. Overall results show that Spray and Wait and MaxProp show high performance in most of the scenarios, while each also shows potentials for improvements. We also present simulation results where buses with predetermined routes were introduced to the network and observe that it generally increases the delivery performance. These results are our starting point towards a work-in-progress for a bus and cars assisted data delivery protocol for VDTN in smart cities.

Introduction

Low energy devices, such as wireless sensors and mobile devices have become an integral part of our daily lives, and their potential to extract data from their surroundings is only limited by their designed sensing capabilities and power sources. A very low power consuming sensor can acquire data such as temperature, humidity, and level of pollution for years without the need for maintenance. While on the other hand, a higher power consuming device can retrieve position, motion, and even image, before its battery needs to be recharged.

Typical ways to connect devices to the central systems are by using cellular networks (3G/4G), or dedicated networks such as SigFox, LoRa and Qowisio, which incur subscription cost. Yet, on the other hand, a common feature in the liveliness of a city is the population of vehicles, either public or private, roaming on the streets and highways. This together with the maturing technology of Vehicle-to-Everything (V2X) communications opens the possibility of utilising cars for data collection from devices and deliver them to the nearest sink for lower cost, or even for free.

The utilisation of vehicles to deliver data will be affected by the intermittent connection available due to the mobility. This is why the Vehicular Delay Tolerant Network (VDTN) concept need to be incorporated within the process of delivering data from the source to the sink. In this paper, we present performance evaluation of four VDTN routing protocols, namely: First Contact (FC), Epidemic (EP), Spray and Wait (SNW), and MaxProp (MP). Previous studies on similar performance evaluations can be found in [START_REF] Benamar | Are VDTN Routing Protocols Suitable for Data Collection in Smart Cities: A Performance Assessment[END_REF], [START_REF] Soares | Improvement of Messages Delivery Time on Vehicular Delay-Tolerant Networks[END_REF] and [START_REF] Spaho | Performance Comparison of Different Routing Protocols in Sparse and Dense VDTNs[END_REF], but they do not consider different communication technologies. We improve upon this work by implementing and studying some of the latest wireless access technologies with farther communication ranges achievable today, Zigbee for the sensors and IEEE 802.11p for the vehicles. Another difference is that we additionally consider public vehicles and study their impact on the performance. We take into account the fact that well-established bus routes and schedules are common in most modern cities today. This predetermined mobility of buses in a smart city, among other potential functions that the buses can be assigned to, is one key factor that we will try to exploit in our work-in-progress towards a VDTN routing protocol for an efficient data collection in smart cities, which we name as Bus Assisted data deLIvery or BALI in short.

Routing Protocols in VDTN

Store-carry-and-forward mechanism is used in scenarios with sparse and intermittent connectivity in VDTN [START_REF] Benamar | Routing Protocols in Vehicular Delay Tolerant Networks: A Comprehensive Survey[END_REF]. The routing protocols that try to tackle these challenges have evolved from the simplest to the more elaborate in terms of message replication policy and process of clearing the networks from redundant messages which are already received by the intended destination. Here we describe and focus on four protocols that do not need priori knowledge of the network connectivity nor nodes coordinates. First Contact, is a single copy forwarding approach where each node forwards messages randomly to the first node they encounter. Nodes erase messages that they already forwarded, which means only a single copy of a message exists in the network. This single copy of messages continues to hop until it reaches its destination. Epidemic, is a multiple copy forwarding approach where each node keeps copies of every message while also forwarding them to every other nodes they encounter until the messages reach the destination. Each node receives messages that they do not already have, with their buffering capacity as the only limitation. This approach ensures that at least one copy of each message will reach its destination in the earliest possible time, with the expense of flooding the networks with redundancy. Spray and Wait, is a more controlled multiple copy forwarding approach, where the number (L) can be assigned to the protocol to determine limit of copies that can be created per message by a node, as described in [START_REF] Spyropoulos | Spray and Wait: An Efficient Routing Scheme for Intermittently Connected Mobile Networks[END_REF]. This protocol has two spraying phase modes: the normal mode and the binary mode. In the normal spray mode, each originating node that received a new message can have L copies of the message and forward each copies to other nodes until it has only the last copy when it switches to the wait phase. In the binary spray mode, the source node initially starts with L copies of a message. When the source node encounters another node which has no copies of the message, it forwards L/2 copies and keeps the other L/2 copies. When each of the nodes subsequently encounter another node, they forward half of copies that they still have and keep the other half. This process continues until each node is left with only the last copy when they switch to the wait phase. In the wait phase, each node carries the last copy until it reaches the destination. MaxProp, is also another multiple copy forwarding approach with explicit mechanism of sending acknowledgements to clear redundant messages left on the network once a copy is received by the destination [START_REF] Burgess | MaxProp: Routing for Vehicle-Based Disruption-Tolerant Networks[END_REF]. MaxProp also uses messages ordering policy in the buffer to give priority of transfer and deletion. The ordering policy includes the calculation of delivery likelihood based on nodes encounters.

Performance Evaluation

We evaluate the performance of the four routing protocols for VDTN by using the Opportunistic Networking Environment (ONE) simulator [START_REF] Keränen | The ONE Simulator for DTN Protocol Evaluation[END_REF]. We use the city of Helsinki's map, with an area size of 4500m x 3400m, to set up our smart city data collection scenario, where data from one sensor on one side of the city is needed to be delivered to an Access Point at the other side of the city. We replicate the scenario three times, each with different placement of sensor and Access Point, but identical vehicles mobility. We then accumulate all results for the performance evaluation to avoid isolated outcomes. A sensor which has a Zigbee connection (250m communications range and data rate of 250kbps) transmits data with a varying size of 10kB -100kB once every minute during the 12 hours simulation time and a Time to Live (TTL) of 5 hours. Each car with pseudo-random initial placement and mobility, has an identical Zigbee connection with the sensor to communicate, and also has an 802.11p connection (500m communications range and data rate of 6Mbps) for V2X communications. The destination Access Point has an 802.11p connection with the cars for communications. We only implement 5MB of buffer size for the cars to evaluate the performance of the routing algorithms under strict constraint, since messages only come from one sensor. We also evaluate the scenario with the increasing number of cars roaming the city, from 10 up to 100 with an increase of 10 each time.

All the performance comparison results for a density of 100 cars are summarised in Table I. The results on the Percentage of Messages Delivered, show that FC, SNW, and MP can deliver slightly above 81% of messages, while EP delivered only 42%. These results show that EP performance relies heavily on buffer size, where 5MB of buffer rapidly becomes full and lots of messages are dropped. The results on Latency show that MP has the lowest latency of around 3 minutes, while EP and SNW have around 12 minutes, and FC has the highest latency of around 18 minutes. The results of latency combined with the Hop Counts, where FC had 46 hop counts on average, suggested that messages are being forwarded back and forth between cars a lot of times, although in the end about 81% of them finally reach the destination. On the contrary, EP, SNW and MP had average hop counts of as little as 4, 3, and 6 respectively. On the Overhead Ratio results, where the value of this parameter gives indication on how many copies of a message were relayed on the network before it reaches its destination, SNW has the lowest value of 3, while EP has the highest value of 634, FC has 46, and MP has 249. These results show that SNW is the most efficient in term of limiting useless replication, and that MP needs to be improved on this.

In the next simulation scenario, we include a bus route where the buses will pass the Sensor and the Access Point on their way. The buses have the same specification as the cars, with the exception of the lower maximum speed and the larger buffer size of 25MB. The comparisons of Delivery Percentage for MaxProp routing algorithm, between the cases where buses were included in the network and the previous simulations without buses, can be seen in Figure I. It shows that delivery percentage is increasing in most of the cases where buses were included. This emphasises the potential for improvement if the bus network is considered in the algorithm design.

Conclusion and Future Works

In this paper, we present the performance comparison of four basic VDTN routing protocols and identify some parameters that can be improved. We also show that the inclusion of buses with predetermined routes in the network has the potential to improve the performance. This is our starting point towards a work-in-progress on BALI, a bus assisted data delivery protocol for VDTN in smart cities.

  FIGURE I. THE EFFECT OF BUSES ON THE PERCENTAGE OF MESSAGES DELIVERED USING MAXPROP

Acknowledgment

This work was supported by the Indonesian Endowment Fund for Education (LPDP) of the Indonesian Ministry of Finance, under Contract Number PRJ-43/LPDP.3/2016.


Tadanori Matsui

Nishi Hiroaki

Georgios Z Papadopoulos
email: georgios.papadopoulos@telecom-bretagne.eu

Pascal Thubert
email: pthubert@cisco.com

Thomas Watteyne
email: thomas.watteyne@inria.fr

Nicolas Montavont
email: nicolas.montavont@telecom-bretagne.eu

Poster: 4th Industrial Revolution: Toward Deterministic Wireless Industrial Networks

Keywords: Multi-path Routing Algorithm, Route Diversity, Determinism, Leapfrog Collaboration

Critical applications such as industrial process control, smart grid and vehicle automation require networks which offer on-time data delivery and wire-like end-to-end reliability. This paper proposes to exploit path diversity to compensate for the lossy nature of the wireless medium. We introduce "Leapfrog Collaboration", a communication mechanism in which multiple copies of the same packet traverse the network on disjoint paths. Emulation results on Cooja show that, compared to single-path transmission, this technique reduces end-to-end delay by up to 28%, and jitter by up to 58%.

Motivation

Industry 4.0 re-uses Internet of Things (IoT) technology to simplify production chains, simplify deployment, and make the factory floor more flexible and adaptable. It heavily relies on deterministic networking technology, in which the network guarantees the information is carried with predefined and predictable delay, regardless of link quality and the network usage. One important goal for the network is to offer near-zero jitter. Current IoT technology builds on best-effort packet switched network, which introduces variable network delays because of retransmissions and queuing.

In 2016, a new version of the IEEE802.15.4 standard [1] was published. It introduces Time-Slotted Channel Hoping (TSCH), a new Medium Access Control (MAC) mode designed for demanding industrial applications. In a regular TSCH network, a packet follows a multi-hop path between source and destination. Between some neighbors, when the link is poor, the same link-layer frame can be retransmitted multiple times. This increases end-to-end delay and jitter. This paper proposes to enhance the forwarding mechanism: multiple copies of the same packet travel on disjoint paths through the network. This technique exploits path diversity. Because the first copy that reaches the destination is the one that counts, Leapfrog Collaboration lowers end-toend delay and jitter.

2 Leapfrog Collaboration IEEE802.15.4 TSCH offers a resource reservation mechanism which allows hop-by-hop scheduling when combined with the RPL [2] routing protocol. Since wireless is unreliable in nature, frames are retransmitted if no link-layer acknowledgment is received. Retransmissions increase delay, energy consumption and bandwidth. In some environments, losses can even be bursty, i.e. the link quality between two neighbors suddenly drops for some time. Consecutive retransmissions on the same frequency do not help in this case. RPL does come with a fail-over mechanism, but the delay to discover and use an alternate path can be long (in the order of seconds). Rather than recomputing the path, we duplicate packets along disjoint paths. We use Fig. 1 to illustrate how Leapfrog Collaboration works. In this figure, node 8 wants to send data to node 1. Each node in the network maintains two parents: its Default Parent (DP) and Alternate Parent (AP). Leapfrog collaboration works best if the two paths of default parents (8 → 6 → 4 → 2 → 1 and 8 → 7 → 5 → 3 → 1 in Fig. 1) are disjoint. It also works best if the default and alternate parents hear one another and if they have the same ancestor.

Each node on the Leapfrog Collaboration path does the same: it sends two copies of the same packet, one to the DP, one to the AP. Packets contain a unique counter, and node maintain a buffer of recently-heard (source IPv6 address, destination IPv6 address, counter) tuples. If a node received a packet already present in its buffer, it silently drops it (elimination). If, however, the packet is new, it forwards it to both DP and AP (replication).

From a TSCH point of view, each node has dedicated cells scheduled to its DP and to its AP. A node's sibling also schedules cells to overhear those cells. If, when listening to those cells, siblings hear a packet it has not heard before, it participates in Leapfrog Collaboration as if it had received it from its child.

By having multiple copies travel from source to destination in a "band" of nodes -rather than a single string of nodes -, latency and jitter are reduced. This is because latency/jitter are measured on the first copy of the packet. Leapfrog Collaboration exploits path diversity so the first copy arrives faster than if a single copy was sent in the first place.

Performance Evaluation

Setup. We implement Leapfrog Collaboration on Contiki OS [START_REF]IEEE Standard for Low-Rate Wireless Personal Area Networks (LR-WPANs)[END_REF] and run simulations on Cooja, with emulated Zolertia Z1 2 nodes. We use the topology from Fig. 1, deployed on a 40 m × 10 m area. Node 8 transmits 1 data packet every 10 seconds to node 1. Transmission power of all nodes is set to 0 dBm. We use the RPL protocol to construct the routing structure. We use a single IEEE802.15.4 TSCH slotframe with 101 timeslots. We compare the performance of Leapfrog Collaboration against that of the default RPL protocol (i.e. a single copy travels through the network). On the base implementation, we vary the maximum number of link-layer retransmissions: RT2 means that the TSCH implementation will re-transmit at most twice in case no link-layer acknowledgment is received.

Delay. Fig. 2 shows the average end-to-end delay [START_REF]Delay and jitter are calculated on successful packet receptions[END_REF] for both Leapfrog Collaboration and the base TSCH implementation (RTx, where x is the maximum number of link-layer retransmissions). Leapfrog Collaboration reduces the endto-end delay when compared to the retransmission-based approaches of IEEE802.15.4-TSCH (i.e. RT2, RT4, RT6 and RT8). Indeed, it decreases by up to 28%, 41%, 46% and 54%, respectively.

Jitter. Jitter is calculated as the time of consecutive packet inter-arrival from the source node to the destination. Fig. 2 shows the average end-to-end jitter. The results confirm that packet retransmissions impact jitter: the higher the retransmissions, the higher is the jitter (i.e. RT8 shows results over 5000 ms). Leapfrog Collaboration exhibits low (i.e. 600 ms) and stable jitter throughout the simulation lifetime. It reduces jitter by up to 58%, 71%, 77% and 84% when compared to RT2, RT4, RT6 and RT8, respectively.

Conclusion

Industry 4.0 is considering standard IoT technology to replace or extend wired networking systems. Recent IEEE802.15.4 changes provide a framework for QoS, but delays are hard to guarantee in lossy packet switched network. This poster introduces Leapfrog Collaboration: sending multiple copies of the same packet on disjoint paths lowers latency and jitter by exploiting path diversity, a statement which is confirmed by simulation results.

Figure 1 :

 1 Figure 1: Replication, Elimination and Overhearing operations.

Figure 2 :

 2 Figure 2: Average end-to-end delay (top) and jitter (bottom).


Cagan Arslan
email: cagan.arslan@univ-lille.fr

Florent Berthaut
email: florent.berthaut@univ-lille.fr

Ioan Jean Martinet
email: jean.martinet@univ-lille.fr

Marius Bilasco
email: marius.bilasco@univ-lille.fr

Laurent Grisoni
email: laurent.grisoni@univ-lille.fr

The Phone with the Flow: Combining Touch + Optical Flow in Mobile Instruments

Keywords: Mobile instruments, Optical flow, touchscreen CCS Concepts, Applied computing → Sound and music computing;, Computing methodologies → Motion capture;

Mobile devices have been a promising platform for musical performance thanks to the various sensors readily available on board. In particular, mobile cameras can provide rich input as they can capture a wide variety of user gestures or environment dynamics. However, this raw camera input only provides continuous parameters and requires expensive computation. In this paper, we propose combining camera based motion/gesture input with the touch input, in order to filter movement information both temporally and spatially, thus increasing expressiveness while reducing computation time. We present a design space which demonstrates the diversity of interactions that our technique enables. We also report the results of a user study in which we observe how musicians appropriate the interaction space with an example instrument.

INTRODUCTION

Many sensing capabilities of mobile devices have been explored for musical interaction [START_REF] Essl | Interactivity for mobile music-making[END_REF]. Most of the mobile devices today includes various sensors such as a microphone, motion sensors, a touch screen, and multiple cameras. Hence, numerous musical interaction devices were designed around smartphones and tablets [START_REF] Geiger | Using the touch screen as a controller for portable computer music instruments[END_REF] [2] [START_REF] Wang | Ocarina: Designing the iphone's magic flute[END_REF], in particular using the discrete and continuous input capabilities of the touchscreen. Among these sensors however, the use of built-in camera of mobile devices has been little explored. The simplest approach is to use the camera as a tone-hole. Ananya et al. [START_REF] Ananya | Microphone as sensor in mobile phone performance[END_REF] use the average gray-scale value of the camera input image to detect if the lens is covered. Covering or uncovering of the lens modifies the pitch of the sound similarly to a tone-hole. Keefe and Essl [START_REF] Keefe | The visual in mobile music performance[END_REF] used low-level visual features of the input image such as the edginess to create mobile music performances with visual contributions. Camera image has also been used to track visual reference points Licensed under a Creative Commons Attribution 4.0 International License (CC BY 4.0). Copyright remains with the author(s). NIME'18, June 3-6, 2018, Blacksburg, Virginia, USA.

to gather position and motion information that are mapped to MIDI controls [START_REF] Rohs | Camus: Live music performance using camera phones and visual grid tracking[END_REF] [START_REF] Rohs | Camus 2 -optical flow and collaboration in camera phone music performance[END_REF].

However, to our knowledge, built-in cameras of mobile devices have not been used for sonification of moving elements in the camera image. Controlling sound through movement has always been of interest and can be traced back to Kurenniemi's DIMI-O [START_REF] Ojanen | Design principles and user interfaces of erkki kurenniemi's electronic musical instruments of the 1960's and 1970's[END_REF] and Rokeby's Very Nervous System [START_REF]Very nervous system[END_REF]. More recent examples such as [START_REF] Pelletier | Sonified motion flow fields as a means of musical expression[END_REF][7] [8] [4] use difference between images and optical flow to represent the moving parts of the image, but the methods have been either too simple to extract rich information or too heavy to be run on mobile devices. The details of optical flow methods will be further discussed in section 2.2.

In this paper, we propose combining visual movement detection with the touchscreen of mobile devices in order to reduce the computation time and to open expressive opportunities. We analyze these possibilities using a design space. We also study how users appropriate the instrument for two different movement sources: the user's gestures and the environment.

THE PHONE WITH THE FLOW

In this section, we first present the main approach behind our system. It is currently implemented as an Android App (www.caganarslan.info/pwf.html). Optical flow features, extracted as described in the next section, are then mapped to sound synthesis parameters. The synthesis can be done either directly on the mobile device, with restrictions on the complexity of the synthesis due to limited computing capabilities of the mobile device, or the features can be sent to external musical software via OpenSoundControl messages.

We also describe a design space of the interaction and mappings opportunities afforded by our system.

From the Scene to the Touchscreen

Our system relies on real-time spatial and temporal filtering of rich motion information provided by the built-in cameras of mobile devices. Fig. 1 depicts an example scenario of use of The Phone with the Flow.

Movement-rich Environment: The physical scene captured by the built-in camera offers a large range of movements, with various periodicity, directions and sources. The sources can be artificial (displays, mechanisms), natural (weather, animals, plants), or originate from people (user's body, other musicians, spectators, by-standers).

Choosing the Interaction Space Mobile cameras have a limited field of view but their portability enables exploration of the surroundings by a point-and-shoot approach. Unlike fixed installations, the mobile camera's field of view can be changed without effort by simply changing its position and orientation. When the camera aims at a part of the movement-rich environment, objects in the field of view are captured, providing the user with a visual feedback of their movement sources. The user is then free to interact in the combined interaction volume [START_REF] Marquardt | The continuous interaction space: Interaction techniques unifying touch and gesture on and above a digital surface[END_REF] of the camera and the touchscreen.

Filtering the Movements Once the movement sources are displayed on the touchscreen, the user focuses on a region of interest by touching it. The touch selects a region of the image to be analyzed further for detailed movement information. The user can change the position of the region by dragging their finger on the screen, alter its size, combine multiple regions and switch between them. The touch input thus enables filtering of the movements.

Optical Flow

Optical flow is the distribution of apparent velocities of moving brightness patterns in an image [START_REF] Horn | Determining optical flow[END_REF]. The movement indicates a motion resulting from the relation between captured objects and the viewer. The motion at a point can be represented with ∆x and ∆y, the horizontal and the vertical displacement respectively. Therefore displacement vector can be expressed as v(∆x, ∆y), but also in polar coordinates v(r, Θ) to represent the amplitude and the direction of the movement for a pixel. Fig. 2 shows a color coded representation of the optical flow. Optical flow has been used in various musical interfaces in the past. ParticleTecture [START_REF] Jakovich | Particletecture : Interactive granular soundspaces for architectural design[END_REF] uses Horn-Schunck's optical flow method to determine the apparent movement in the image. The flow output is interpreted as a Game of Life, where living cells correspond to pixels in motion. The cells are tied to sonic grains that emit sound when activated. However, the only direction information they use is the horizontal component, which is used to trigger grains into proliferation or stasis. Thus, while their work uses optical flow for granular synthesis, they did not fully take advantage directional richness of flow properties. In Sonified Motion Fields [START_REF] Pelletier | Sonified motion flow fields as a means of musical expression[END_REF] Pelletier uses FAST corner detector [START_REF] Rosten | Machine learning for high-speed corner detection[END_REF] to extract feature points and estimates the optical flow by using a pyramidal Lucas-Kanade method [START_REF] Bouguet | Pyramidal implementation of the lucas kanade feature tracker[END_REF] to track the sparsely identified points. He also discusses the potential mappings of the flow field to sound parameters. However he concludes that the flow fields' temporal resolution is poor and the feature detection is not robust. CaM uS 2 [START_REF] Rohs | Camus 2 -optical flow and collaboration in camera phone music performance[END_REF] estimates the optical flow from a camera phone. A 176x144 pixels image is divided into 22x18 blocks and cross-correlations between successive pairs of block images are calculated to obtain the optical flow. Because only 4 points are sampled in each block, the algorithm performs quickly. The simplicity of the method allows obtaining the global translation and rotation at 15fps, but the system is unable to provide rich motion information from the moving objects in the image.

In recent years, there have been advances both in smartphone hardware (improved cameras, CPU and GPU) and optical flow algorithms, which open up new possibilities for movement detection in real-time.

In 2016, Kroeger et al. [START_REF] Kroeger | Fast optical flow using dense inverse search[END_REF] introduced DIS-Flow, an optical flow estimation method that is one order of magnitude faster than others and produces roughly similar flow quality. Their objective is to trade-off a less accurate flow estimation for large decreases in run-time for time critical tasks. This method provides a dense flow output, in which every pixel is associated to a displacement vector v(∆x, ∆y). This advancement enables us to envision the creation of an innovative design space that allows richer interaction and sonification methods.

Optical flow computation is noisy and presents important motion discontinuities as it measures the displacement in the 2D image plane of the real 3D world. Filtering out noise and focusing on the coherent motion information is the key to successful optical flow interaction. Our approach to extracting features from raw flow data is as follows. First of all, for every region of interest, the pixels that have a smaller displacement value than a threshold are discarded to eliminate the noise (2). We identified key information to gather from the optical flow output. We collect three global values: the amount of moving pixels, the average direction and the average magnitude. The amount of moving pixels corresponds to the ratio of the number of remaining pixels to the image area. The average direction and magnitude are the direction and amplitude of average displacement of the moving pixels. In order to obtain the distribution of the movement within the region of interest, we construct a normalized 8-bin histogram of directions. Adding the three global values to the histogram information, a feature vector of eleven elements is obtained for various mappings.

Design Space

We present a four dimensional design space to explore the musical expression possibilities presented by Phone with the Flow. The dimensions can be combined to provide a large set of interaction techniques.

Movement Source

A smartphone camera enables the user to freely change the subject of the motion. We distinguish two values for this dimension: Self-motion: The holder of the camera is also the creator of the movements. The user can use its body parts to perform dynamic gestures. In the case where the user holds the camera with one hand, the other hand is free to create optical flow. The fingers permit creating movement in multiple directions simultaneously. Feet, legs, head and even the torso may also be the source of the movement. It is also possible to perform gestures by moving the device, as its relative motion with respect to the environment also creates an optical flow.

World-motion: The camera aims at an external source of motion. These sources can act intentionally to create controlled movements, such as a dancer, or be unaware of how they contribute to the process, such as spectators and other musicians on the stage. Objects that are continuously in motion may also be used to create sound, for instance, a busy highway, a waterfall, an assembly line and so on. Additionally, visual sources such as projections and displays may serve as sources. In a rich environment where there are multiple sources, the mobility of the device allows transition between them.

Camera Movement

This dimension is about how we adapt camera movement to complement the movement search.

Fixed: The camera is held at a fixed position. All of the optical flow is created by the moving objects in the static field of view.

Independent: The camera moves independently from the content of the environment. The optical flow is created by the motion of the camera with respect to the environment. Horizontal and vertical movements of the camera produce an output similar to that of an accelerometer, but the movement in the depth results in an optical flow in every direction as it produces a zoom-in-zoom-out effect. It can also be rotated around three axis.

Adaptive: The camera can track a moving object to strip it from either its translation or rotation. Or, if the object in motion is not rigid, its deformation can be stripped by adapting the camera movement to its trajectory. However, if the object does not fill the camera's field of view, the independent motion mentioned above distorts the optical flow. Hence, adaptive movement requires precise control.

Touch

The touchscreen offers valuable possibilities for filtering the image to discard the movements that are not desirable for the performer. The touch dimension has three values.

None: The touchscreen is not used along with the camera. All the movements in the camera image are processed continuously. This is ideal in controlled environments where no intervention is required.

Activation: The touchscreen is used for temporal-only selection of movement. All movements captured in the camera image are therefore processed, but only while there is a finger touching the screen. This allows for discrete activation/deactivation of otherwise mostly continuous changes in the captured movements.

Isolation: Fingers are used to select a region of the camera image to focus on a movement source and discard the rest of the movements in the view. When there is no contact between the finger and the screen, nothing is processed. This enables both spatial and temporal filtering, making the system more robust to noise. Another advantage of isolation is decreasing the number of pixels to process in order to lower the latency. Multiple regions may be activated at the same time.

Mapping

The mapping dimension describes the mappings between the finger position on the screen and sound parameters.

Independent from spatial position: Sound parameters are influenced only by the optical flow. Regions that encounter the same movement sound the same independently from their position.

Linked to spatial position: X and Y axes of the touchscreen are mapped to the sound parameters. The position of the finger determines the values along these axes. Consequently, the same movement results in different sounds in different regions of the camera image.

EVALUATION

We evaluate what techniques are used by musicians when exploring the instrument. The goal of the evaluation is to determine how the users utilize the interaction space created by the combination of the camera and the touch screen.

10 participants (2 left-handed) volunteered to take part in the experiment. Their ages varied between 23-52 (mean 35, sd 9). All participants were involved in musical activities and were regular users of smartphones. The experiment was conducted on a 5.8" Galaxy Edge S8 smartphone running Android. The built-in camera provided images of 320x240 pixels at 30 fps. The feature vectors were sent via OSC messages to a standard PC running our Pure Data patches.

Protocol

We presented a mobile instrument which demonstrated our design space to the participants. In this instrument, the sound is produced by granular synthesis with 3 voices (one per finger). The mapping is linked to spatial position such that the horizontal and vertical touch position on the screen respectively changes the initial position in a sample and the initial frequency of the grains. While the finger is touching, the grains that make up each voice then move around the sound (in position and frequency, as if browsing a spectrogram) according to the optical flow. The flow histogram gives the amplitude of displacement in 8 directions for the grains. The average flow amplitude changes the duration of the grains and the amount of moving pixels controls the volume. Participants were also able to change the size of the region that is activated by a touch.

The experiment was conducted in two stages. In the first, the participants were provided with a display that could generate animations and a small fan as world movement sources. In the second they were instructed to use their bodies to create movement. For each stage they were given 5-10 minutes to explore and to prepare a 5 minute performance. During the performance, all parameter variations

Results and Discussion

Without any instruction, all of the participants used their dominant hand to hold the smartphone and to use the touch input. They stated that orienting the smartphone and using the touchscreen at the same time was a difficult task for the non-dominant hand.

We measured the frequency of simultaneous fingers used on the screen. While performing hand gestures in the selfmotion condition, the participants mostly used only one finger (55% 1 finger, 32% 2 fingers, 13% 3 fingers) to isolate the regions. However, when participants moved the camera with two hands, they were able to use multiple fingers. In the world-motion, they often used both thumbs (42% 1 finger, 48% 2 fingers, 10% 3 fingers).

Figure 4 shows the average time it takes to calculate optical flow and to produce the feature vector for regions of different size. We calculated the average region size used by the participants to be 16800 (140x120) pixels. This shows that using image regions instead of the entire image (76800 pixels) significantly decreases the computational time from 23ms to 5ms. 70% of the participants preferred using world objects instead of gestures. While the participants who preferred world objects argued that the instrument required concentration when performing hand gestures and using the touch input at the same time. The opponents all stated that gestural interaction allowed a finer control on the created movement.

CONCLUSIONS

In this paper, we presented The Phone With the Flow, a novel approach to sonify visual movements by combining the touchscreen and the built-in cameras of mobile devices. In this context, the touchscreen proves to be a valuable tool to filter the visual input temporally and spatially. We also discussed the different interaction techniques enabled by our design space. An interesting future work would be to combine the use of optical flow, i.e. motion features, with the other features in the images captured by the built-in camera, bringing objects textures, colors and shapes to the sonification process.

  (a) A person, a computer screen, a fan, objects passing by the window are movement sources. (b) The user chooses a movement source (c) The user selects a region on the screen

Figure 1 :

 1 Figure 1: From scene to the the touchscreen.

Figure 2 :

 2 Figure 2: Color representation of the optical flow. (Top) Color space: hue indicates direction, saturation indicates amplitude. (Bottom) Grasp gesture, estimated flow, filtered flow

Figure 3 :

 3 Figure 3: Movement sources. (Left) Self-motion, (Right) World-motion.

Figure 4 :

 4 Figure 4: Computation time of the features vs region size

Table 1 :

 1 Likert Scale resultswere logged in the Pure Data patch. The sound samples were altered between the steps. Both the order of steps and the samples were counterbalanced across participants. After the open exploration and the performances, participants responded to 7-point Likert-scale statements for which the results can be seen in table1.

	Questions	Mean SD
	I made a lot of physical effort	2.65	1.05
	I needed to concentrate	5.25	0.60
	I felt frustrated/annoyed	2.6	0.99
	I enjoyed interacting	5.75	0.84
	with the instrument
	I was able to create a	4.35	1.18
	wide variety of sounds
	I found my own techniques to play	4.9	1.14