Madeleine Baalsrud Jannicke

Hauge

Ioana Andreea Stefan
email: ioana.stefan@ats.com.ro

Antoniu Stefan
email: antoniu.stefan@ats.com.ro

Exploring Pervasive Entertainment Games to Construct Learning Paths

Keywords: Pervasive mechanics, entertainment, learning, lesson path

Digital Educational Games (DEGs) aim to provide motivating, personalized play experiences that blend learning and engagement, while addressing pedagogical requirements. The ultimate challenge is to enable and stimulate knowledge acquisition by creating rich environments that employ Entertainment Games (EGs) mechanics in order to accommodate learning objectives and support skills' development. In recent years, the gap between EGs and DEGs has started to close, with studies looking not only into lessons learnt from EGs, but also into how EGs can be used in learning settings. This research analyses the possibility to integrate EGs mechanics into the pedagogical flows, in order to potentate learning. The paper further outline the design of a learning path that will be used as a unit on logistics and production means. The unit will be used for letting high school students explore functions of logistics and production (as a recruitment tool during the Open University days), as well as for the first introductionary course on production logistics.

Introduction

The massive success of Entertainment Games (EGs), such as Ingress and Pokemon GO, prove the games' capacity to act as a medium to motivate and engage individuals in various types of activities, ranging from strategizing, planning, and resource collecting to collaborating and socializing. Digital Educational Games (DEGs) aim to explore this potential in educational settings, in order to provide students with more attractive means to explore learning paths. In order to recruit new students to STEM subjects on activities like Open University days or to engage young museums visitors, several games have been used with great success, but often the games used for the classical course teaching, the focus on the pedagogical component has often altered the fun of DEGs or have prevented them from reaching their full engaging potential. Moreover, learning paths are individual and do not follow pre-defined scenarios, requiring in-depth levels of personalization.

The emergence of pervasive EGs has opened up new possibilities to explore userpersonalized scenarios [START_REF] Costanza | Simulation games that integrate research, entertainment, and learning around ecosystem services[END_REF], balancing randomness, individually-driven play with preestablished game goals and rules. They are able to captivate users to willingly and repeatedly play for extensive periods. Therefore, EGs offer an excellent ground for defining best practices [START_REF] Ham | Tabletop Game Design for Video Game Designers[END_REF] and fundament the design of DEGs that stimulate learners [START_REF] Marsh | Special issue title: Entertainment in serious games and entertaining serious purposes[END_REF]. This paper describes a way of how EG can be integrated in an introductory class or also as a stand-alone unit to be used for recruitment settings like Open University Days.

Approach for Constructing BEACONING Learning Paths

The shift towards more flexible learning implies the adoption of new methodologies and practices. The emergence of gamification and gaming technologies offer opportunities to construct new approaches to learning, giving learners more freedom, strengthening collaboration skills, and stimulating their creative mind [START_REF] Ștefan | Approaches to reengineering digital games[END_REF]. Pervasive Learning is such new way of learning and is defined as "learning at the speed of need through formal, informal and social learning modalities" [START_REF] Pontefract | Learning by Osmosis[END_REF]. The BEACONING project aims to take advantage of this playful pervasive learning and integrate such informal ways of learning into curricula to provide personalized learning paths. The main focus is STEM and problem-based learning [START_REF]D3.3 Learning Environment System Specification[END_REF]. The BEACONING platform will provide the users (teachers, students) diverse applications, so that the teacher with support of the integrated authoring tool and learning analytics can personalize the learning units to the specific student's needs. In order to ensure the reusability of the different learning paths, a set of templates as well as a taxonomy have been developed. The constructed learning paths have similarities to lessons plans, and will therefore be easy to use for teachers since they can easily adapt. The templates and the taxonomy are described in [START_REF]D3.3 Learning Environment System Specification[END_REF], and section 3 provides an overview on how such a path can look like.

Adapted Learning Path for Understanding Industry 4.0 Applications in Production Logistics

As described in the introduction, we have experienced that it is difficult for students to understand the technical systems and new concepts (like industry 4.0 and IoT) can support logistics operation [START_REF] Baalsrud Hauge | Deploying Serious Games for Management in Higher Education: lessons learned and good practices[END_REF][START_REF] Riedel | Evaluation of Simulation Games for Teaching Production (Engineering)[END_REF][START_REF] Baalsrud Hauge | An educational framework for supporting the implementation of the Intelligent Cargo concept[END_REF], without experiencing how it works, specifically it they lack experience in logistics operations. This also makes it hard for future po-tential students to know what a study on production engineering and logistics is about.

Based upon the long experience of the authors in game-based learning for the given application field, we decided to construct a new learning unit that can be used as an introduction part to a course on production logistics, as well as a single unit for workshops on open university days as the previous mentioned Open University Days. The unit is designed as a part of the small scale pilot that the BIBA will have as a part of the BEACONING project.

E. Play -Lesson Path

The lesson path is divided in four missions presenting basic knowledge on the underlying concepts and technologies for Industrie4.0. The lesson starts with a small (toy size) forklift able to move around move all around and to do stock-in and stock-out in a miniature warehouse. The forklift (see pict.) has sensors and actuators installed. Depending on which sensors (tilt, temperature, humidity, gravity etc.), the students will be able to carry out different operations. During the first small part the student does not get access to the data during operation, whereas these are provided in the second part. The data are collected and provided as feedback for analysis so that the students can use this experience in solving the next quests and challenges. This part is carried out in a safe environment in the gaming lab. The second part of the unit is organized as a treasure hunt with quests and challenges related to the equipment and tools (warehouse, large forklift, conveyer belt, picking tools) we have in different parts of the BIBA building. The students will get different challenges (i.e. on what material to store or to provide at the conveyer belt, lead times etc. relevant for production logistics tasks). With their mobile they will search for the right object to solve this tasks best (these objects are outside or in our production hall). When collected enough information on the real devices, the students will return to the lab and apply what they have learned by developing similar services equipping the miniature world with relevant sensors, actuators and define the interfaces between the PPC, ERP, etc. systems for ensuring the access to the right information. The students will test out if their application works.

Table 1 describes the learning scenario. It is based on our experiences with Ingress and Pokemon, and uses the same mechanics [START_REF] Söbke | Prime Example Ingress Reframing the Pervasive Game Design Framework (PGDF)[END_REF]. Table 2 describes the missions that are implemented. The game environment for mission 1 is illustrated below. Different objects will comprise different information which the students need in order to solve their tasks efficiently. For this main introductionary session is the focus on awareness raising, motivation and engagement. We therefore use game mechanics like time, competitions and exploration more than if the main learning outcome should be specific knowledge about the different system components.

Discussion and Next Steps

The design of the curricula and learning path, as well as of the game is now completed and we are currently adapting the game environment described in [START_REF] Hauge | Analysis on Educating Mechanical Engineers Through Serious Games Using Pervasive Technologies[END_REF]. For the modding we use part of the ATMSG framework [START_REF] Carvalho | An activity theory-based model for se-rious games analysis and conceptual design[END_REF]. The digital part of the game is realized in Unity, and the real world data are imported into the unity scenario. (i.e. this part can be seen as a game in a game). This part is also used alone. The minigames and the overall gamification (including the treasure hunting) are integrated as a part of the overall narrative. The small scale piloting starts in autumn 2017.

Fig. 1

 1 Fig. 1 Experimental Environment

A

  . Domain / Area / Subject: Production logistics B. Topic: IT tools for production logistics and warehousing systems C. Age Group / Key Stage / Year / Background Undergraduate students starting specialization in production logistics, high school students in the phase of deciding on what to study D. What is it about? / What's in your mind? / What's the matter? Concepts like Industry 4.0 and Internet of Things have an increased usage with logistics and production, especially in Germany.The improved possibility to support the logistics processes by allowing access to data and information throughout the whole chain gives rise to a lot of new possibilities, but the concept itself is often difficult to grasp for students hardly knowing how logistics operations are carried out and which tools currently used. In addition, there have been a rapidly development of new technologies. This unit will let the students experience different picking technologies (picking by light, by voice, digital etiquettes etc.) as well as the stock-in and out. The lesson is problem based and the students shall experience the differences with and without technology support.

Table 1

 1 Learning Scenario on IoT in Production Logistics

Table 2

 2 Mission Descriptions

	Mission A: Exploring how information from information from an IoT environment
	can support in-and out-stocking
	Background: No prior knowledge on production and logistics
	Skills: logical thinking
	Quest 1: Discover the in-and out-stocking process in a warehouse
	Brief

overview of Quest 1 activities.

	Brief overview of Quest 2 activities.
	Time Frame	Participants	Location(s)	Resources
	1 hour in one ses-	The teacher	Lab and hall	Information stored
	sion.	Their classmates	On their mobile	in the objects
				(technical data and
				description)
				Videos on usage
	Evidence	Rewards	Beacons
	End result (analy-	Points awarded in	Will allow access
	sis of collected	game are used for	to the game within
	information)	a class/school	building and
		wide leaderboard	around
	Quest 3
	Equip the miniature truck with different sensors and actuators and combine into
	new services
	Brief overview of Quest 3 activities.
	Time Frame	Participants	Location(s)	Resources
	2 hours in one	The teacher	Gaming and Fab-	Sensor and actua-
	session.	The technicians	Lab	tor data
		Their classmates	On their mobile	Description of
			On the lab PC	system and service
				components (in an
				LMS)
	Evidence	Rewards	Beacons
	End result, how	Points awarded in	Will allow access
	the different sen-	game are used for	to the game at
	sors can be used to	a leaderboard	around the hall
	provide different	(Different points	and outside the
	services	for performance,	building
		innovation, etc.)
	Time Frame	Participants	Location(s)	Pedagogical Re-
	1 hour of work in	The teacher	Gaming Lab	sources
	1 session.	Their classmates	BIBA hall	Introductionary
			Home	(on line)
				Books,Websites
				Interactive mater
	Evidence	Rewards	Beacons
	End result,	Points to have	The beacons will
	Data from exer-	accessed/read the	indicate the physi-
	cise (i.e time used,	lesson material	cal location of the
	tilt and numbers	Points according	students
	of errors etc).	to performance
	Quest 2
	Discover different equipment work
	Reverse classroom style and learning by doing.

Acknowledgement

The work presented herein is partially funded under the Horizon 2020 Framework Program of the European Union, BEACONING -Grant Agreement 68676 and by Unitatea Executiva pentru Finantarea Invatamantului Superior, a Cercetarii, Dezvoltarii si Inovarii (UEFISCDI) in Romania, Contract no. 19/ 2014 (DESiG).


Kei Kobayashi
email: kobayashi.kei@entcomp.esys.tsukuba.ac.jp

Junichi Hoshino
email: jhoshino@esys.tsukuba.ac.jp

Augmented Reality Media for Cultural Experience in Shrines

Keywords: Culture, Interaction, Media Art, Texture, Augmented Reality

While intangible culture still remains as a custom, it is gradually losing the substance making it difficult for people to understand the value by their sensibility. Sensibility is important for conservation and dissemination of culture as well as cultural industry, and the augmented reality media convey meaning of shrine rituals, style of ritual manners and meaning as well as information about enshrined deities while appealing to people's sensibility. Media for the purpose consist of five works mixed with attractiveness of materials and system development. Users are supposed to experience the works sequentially by holding a scroll painting. The scroll painting is a centerpiece to be used, for example, as a screen for projecting works guide and images.

Introduction

Japanese culture is expected to be disseminated to the world as a cultural industry's strategy by reaffirming and evaluating its own attractiveness. In this paper, we propose augmented reality media for shrines to convey intangible culture. As a part of traditionally continuous Japanese culture, shrines have taken roots in customs including New Year's visit and tourism even in modern times. However, less people know how to offer prayers correctly and what is enshrined. With a scroll drawing in their hands, users are able to obtain information easier in a step-by-step manner by sequentially experiencing spatial media consisting of multiple works based on images and graphic design as well as to learn subjects, styles and meaning of manners in shrines with their sensibility by experiencing prayer.

System Overview

Media to be proposed is centered by scroll drawing Fig. 1. Space consists of five elements; i.e. Origami paper of pigeon as shinshi (Divine servant) of subject shrine with the meaning of rituals described thereon, flag to show information of enshrined deity, image operated by projecting detail of the enshrined deity and laying on of hands, animation of manner at Chozusha (purification trough) similarly projected on the scroll drawing, and animation of manner at a hall for worship projected on an acryl plate in order. In work Fig. 2 operated by projecting images on the scroll drawing for constructing a system, a framework was detected by RGB-D camera. With a coordinate designated for an icon projected on the scroll drawing by a projector, it was set up to replay a special effect by switching projected image when right or left hand entered into a range of the coordinate. With an idea that visual presentation is better to convey behaviors than a text, manners at Chozusha were expressed by animation and position at starting animation was changed by coordinates evenly divided for seek bar operation. In addition, such a function to temporary stop by grabbing a hand and to replay by unclenching the fi st was equipped with in addition to the seek bar.

For instruct manners at a hall for worship, a method to project images on a rear permeable fi lm put on a transparent acryl plate was adopted. The screen is capable of making users feel uncanny aura of mystery without rupturing space while making matters appear to be fl oating visually compared with ordinary screen (Fig. 3).

Experiments

Data was gathered from 12 persons who actually experienced a work by questionnaire about before and after they experienced the work (Fig. 4). In order to confi rm their degree of understanding of shrines before they view the work, we asked them to select from among four grades, i.e. "1. I know," "2. If anything, I know," "3. If anything, I don't know," and "4. I don't know," for questions on style and meaning of manners and what is enshrined as deity, respectively. As a result, about half of them responded as "If anything, I know" about style of the manners and what is enshrined but majority of them responded as "I don't know" about meaning of the manners (Fig. 5).

After they viewed the work, we asked them to evaluate by four grades, i.e. "1. I understood," "2. I understood to some extent," "3. I didn't understand well," and "4. I didn't understand," and to describe the reasons in order to confi rm their degree of understanding about style and meaning of the manners. The result was that all responses were made up by "1. I understood," and "2. I understood to some extent" (Fig. 6). As the reason why they could understand, many of them described effect of animation. Some of them responded that they obtained information, which they might have not read by a text alone, with an interest due to the interactive method.

In order to recognize importance f manners relatively, we asked them to evaluate seven items in total, i.e. "Omikuji (paper fortune)", "Items including Omamori (charm) and Ema (votive horse tablets)", "Money offerings", "wish", "manners", "Timing for visiting" and "Existence as a specific shrine" on a 5-point scale (Fig. 7). The result was compared by average values. Before viewing work, "wish" was the highest by a score of 4 and "manners" was the fifth position by 2.66. In the same question after viewing the work, "manners" increased to 3.66. "Existence as a specific shrine" showed large difference next to manners before and after viewing the work.

It is considered that it had an effect to make them conscious about characteristics of the shrines such as enshrined deity because the work took up a specific shrine as the theme.

Conclusion

In the current study, we have created sequentially augmented reality media taking up shrines as a theme based on a notion that something intangible and sensitivity for it is important for conservation and dissemination of culture and cultural industry. From a questionnaire survey conducted for those who experienced, it has been proved that shrines are a culture remaining as a part of custom but its meaning is fading while losing the substance in reality. In addition, it is interactive to convey manners and their meanings through targeted works and it has been achieved sufficiently by dynamic visual manner resulting in its increased importance.

Reference

Fig. 2

 2 Fig.2 Works using a system

Fig. 1

 1 Fig.1 The scroll drawing Fig.1 The scroll drawing

Fig. 3

 3 Fig.3 Manners at a hall for worship

  Fig.6 Data after the works

[ 1 ]

 1 Fig.7 Comparison of importance


Laura Giarr É
email: laura.giarre@unimore.it

Ilenia Tinnirello
email: ilenia.tinnirello@unipa.it

Letizia Jaccheri
email: letizia@idi.ntnu.no

In.Line: A Navigation Game For Visually Impaired People

Keywords: Navigation, Assistive Technology, Game, Visually Impaired People

In.line is a novel game based on a navigation system, called ARIANNA (pAth Recognition for Indoor Assisted NavigatioN with Augmented perception, [1]), primarily designed for visually impaired people permitting to navigate and find some points of interests in an indoor and outdoor environment by following a path painted or stuck on the floor. The aim of the game is twofold: 1) let the users learn and familiarize with the system, 2) improve blind people spatial skills to let them learn and acquire an allocentric spatial representation. The impact stands in the possibility of enhancing the social inclusion of a large part of the society that is increasing with the aging of the population by augmenting their autonomous mobility. The paper presents the concept design and preliminary evaluation of a game specifically designed for blind people. The evaluation has been performed via qualitative and quantitative tests.

Introduction And Related Work

Worldwide, 161 million people are visually impaired (according to the 2002 World Health Organization statistics), with an average of 3.4 low-vision patients for each blind person. Visual impairments pose a number of challenges related to mobility and hand-eye coordination. With technology we try to face the challenge of helping Visually Impaired Person to live a more independent life, working and socializing via autonomously and safely moving in an unknown environment and retrieving information about the places nearby.An important aspect of the design of assistive technology for Visually Impaired People, underlined in [START_REF] Elli | Is There a Future for Sensory substitution Outside Academic Laboratories[END_REF], is the possibility for the blind community to be taught and accompanied by the use of the novel device. A learning phase trough games is envisioned as a possible solution to let the blind community to accept such aids, by learning the use of them. This reasoning is the first motivation for the developing of In.line to let the blind people being familiar with the ARIANNA system and its use. The system permits to autonomously navigate and find some points of interests in an indoor and outdoor environment. A path stuck or painted on the floor is detected by the smartphone camera and a vibration signal providing a feedback to the user for correcting his/her direction is generated. Some special landmarks have been deployed for coding additional information, using Beacon technology or Qr codes. With ARIANNA, then, by following pre-determined paths located in the indoor and outdoor environment the visually impaired people can move independently in unfamiliar environments.

The accessibility of Internet to blind persons, and in particular the design of accessible game has been analyzed in the survey [START_REF] Yuan | Game accessibility: a survey[END_REF] where a large number of accessible games are listed for different types of impairments, across several game genres, from which a number of high-and low-level accessibility strategies are distilled for game developers to inform their design. In that analysis it has been stressed how the accessibility problems can be related to a) not being able to receive feedback; b) not being able to determine in-game responses; c) not being able to provide input using conventional input devices. The design of our game is taking into account some of the suggestions presented here. Many blind persons, especially if young or adolescent, have a very high technological ability and many are the commercial games designed specifically for blinds. The pervasive presence of low-cost smart devices with increased computation ability allows the growth of innovative solution in several application fields. As [START_REF] Connors | Virtual environments for the transfer of navigation skills in the blind: a comparison of directed instruction vs. video game based learning approaches[END_REF] reports that for profoundly blind individuals navigating in an unfamiliar building represents a significant challenge, and investigated the use of an audio-based, virtual environment called Audio-based Environment Simulator that can be explored for the purposes of learning the layout of an unfamiliar, complex indoor environment, we were pushed towards the developing of our game. The highly interactive and immerse exploration of the virtual environment greatly engages a blind user to develop skills akin to positive near transfer of learning. Learning through a game play strategy appears to confer certain behavioral advantages with respect to how spatial information is acquired and ultimately manipulated for navigation.

Concept Design Of In.line

In.line is a game primarily designed for visually impaired people which is based on the navigation system ARIANNA.

ARIANNA System Description

ARIANNA system permits to navigate and find some points of interests in an indoor environment by following a path on the floor. Descriptions of the system were presented in [START_REF] Croce | ARIANNA: A smartphone-based navigation system with human in the loop[END_REF], [START_REF] Croce | Enhancing tracking performance in a smartphone-based navigation system for visually impaired people[END_REF], and the Patent [START_REF] Patent | A000054, Sistema di navigazione per non vedenti, presented 2015[END_REF]. The system works as follows: the camera of the smartphone continuously captures the scene in front of the person (see Fig. 1) and identifies the path by recognizing special colored strips painted or stuck on the floor. The users, then, receives a vibration signal through his cell phone. The vibration is related to the position and the direction of the user. In any installations of the system, the paths are marked with colored lines on the floor; QRcodes or beacon are also settled close to points of interest and they provide information on the right line to follow in order to get to the desired destination. The user interface employs tactile stimuli to receive feedback on the heading corrections to be employed, as better described in the following.

Aim Of The Game

The aim of the game is twofold: 1) let the users learn and familiarize with ARI-ANNA 2) to improve blind people spatial skills. The latter is obtained with the design of some 'find short-cuts' task during the play, (navigate trough locations not linked by pathways directly experienced during the game) in order to let them learn and acquire an allocentric spatial representation instead of the egocentric one. We recall that blind people have an egocentric spatial representation (based on themselves) while sighted people have also an allocentric representation, linked to a reference based on the external environment and independent of their current location in it. At first, we have started building up our game to fulfill the following scope: let the blind community play&learn. This will achieve also the secondary goal that is letting ARIANNA become a tool that the blind associations are happy with, by enforcing its use by the game helping blinds to familiarize with it. But learning trough games and amusement, is really the main scope of In.line because, as reported in a recent panel, [START_REF] Mackay | Playing to learn Stanford Report[END_REF], the use of games as an educational tool provides opportunities for deeper learning.

During the process of designing the game, we then asked ourselves if the game could be also a tool that could offer more to the blind community than a simple game to learn and be familiar with an app. Then, along what has been studied by [START_REF] Cattaneo | Blind Vision. The Neuroscience of Visual Impairment[END_REF], we have decided to use In.line to improve blind people spatial skills. To this aim, we design some 'find short-cuts' tasks during the play, (navigate trough locations not linked by pathways directly experienced during the game). While understanding the presence of short cuts may let them learn, acquire and improve an allocentric spatial representation. Egocentric type of representation is a route-based representation, while allocentric one is a survey-based one. For example, the reproduction of previously explored pathway only requires memory trace of the route, while finding shortcuts requires to build a survey representation and to compute a completely new pathway in the brain. Many have been the studies regarding whether the congenital blind individual prefers egocentric over allocentric spatial representation, [START_REF] Cattaneo | Blind Vision. The Neuroscience of Visual Impairment[END_REF] and references therein, but many studies are also suggesting the use of educational programs helping and training blind person in developing such an allocentric skill. Similar experiments have been already carried out, see [START_REF] Connors | Real world navigation independence in the early blind correlates with differential brain activity associated with virtual navigation[END_REF], [START_REF] Connors | Development of an audio-based virtual gaming environment to assist with navigation skills in the blind[END_REF] and [START_REF] Connors | Virtual environments for the transfer of navigation skills in the blind: a comparison of directed instruction vs. video game based learning approaches[END_REF].

The Game In.line

The first prototype of In.Line was a game where a virtual line was projected on the floor. The person stands on the path and follows the line that is moving in front. During the projection of the line, some QR codes were also depicted and the blind (or blind folded) users had to catch it. Clearly, this previous release could be used only in a specific settings, with external teachers guiding the user trough the game. The line was projected on the floor, and the area was restricted so that no one could enter it and be an obstacle for the gamers. The aim of this first release was to teach the blind person how to use the system. We wanted to achieve the following: to be able to follow tactile indication from a common device such as a smartphone. The main objective of the game is to collect max possible scores in a precise time interval, start and the end of the game are reported by different audio signals. During the play, the ARIANNA path or line is projected in front of the player. During the game, the user is not walking on the path but the path flow moves in front of it. The game consists of following the line movements with the smartphone, the player tries to follow the movements of the line in front of him/her. If the line is captured the phone vibrates, the skill of the player is to predict the line changes and keeping on the smartphone vibration. The game has many levels of difficulty related to different paths, every level has a speed line runs and many changes direction different from other levels. The maximum score is reached when the player never looses the line, to all the game time, and consequently, the smartphone never stops the vibration. If the player obtains at least 3/4 of the maximum score, then it can move to the next level. Moreover, the player can intercept markers, such as QR code on the path. If this occurs further information about the point of interests are reported, with more scores and bonus to quickly go at next level.

The second release of the game was designed to be entirely developed on the smartphone. The Game is designed according to the second modalities of ARIANNA, that is the user is following the virtual line with the finger and the phone vibrates only if the finger captures the line. There is a virtual path on the smartphone screen, the finger is used to intercept the line; an on/off vibration feedback will tell the user if he/she gets the line or not. This second release could be used by the blind people in complete autonomy. The main objective of the game is to collect max possible scores during a session, no time limits are considered in a session. Every session is composed of three phases, during the In.line virtual path in the second release of the game first phase a line shape is shown on smartphone touch screen, the smartphone vibrates when the player finger intersects the line. The first phase consists of following the line shape with the player finger, from start point to end point and store the shape. In the second phase, the player must remember the line and uses the finger to reproduce the line on smartphone touch screen. If the reproduced line is completely matched with the original line shape, the player reaches the maximum score and it moves to the third stage. Otherwise, the player can retry unlimited time, any attempt produces a penalty and the total score is reduced. In the third stage, the player must remember the start and stop line points, the player skill is to connect them together through a shortcuts path. Lower path enables additional scores for the player. The game has different levels with different difficulty related to paths shape. An example of the virtual path to follow is depicted in Fig. 2.a). To let the player acquire more points the virtual paths is intended to be followed in a more complicated way, in order to design also the shortcuts necessary for the developing of Allocentric vs. egocentric spatial perception (see Fig. 2

.b)).

To let the player acquire more points the virtual paths is intended to be followed in a more complicated way, in order to design also the shortcuts necessary for the developing of Allocentric vs. egocentric spatial perception (see Fig.

Evaluation And Discussion

The game, in its first release, was tested during the smart city exhibition in Bologna. ARIANNA system has been deployed in various test environments, and also in real installations, such as in the Farm Cultural Park in Favara (AG) ([12]) and in the Blind Institute of Palermo Florio Salamone ([13]), (see [11]). The latter has been designed to allow blind people to move completely independently from one area of the institute complex to another. Specifically, the route crosses both courtyards, from the entrance to the computer classroom, and it enables to reach all the building entrances that are located within the perimeter of the three courtyards. On the way, 10 points of interests are placed allowing the users to orient themselves and achieve full autonomy in any part of the building. Qualitative and quantitative tests with blind and low vision people have been collected and analyzed. The complete analysis has been described in details in [15]. To design our tests, we decided to use Google Drive for our online survey, because it is compatible with many accessibility features for visually impaired people and is supported by a variety of platforms and smartphones. To get information on the user's experience, we created a survey where the scoring of the results provides an overview of the consumer satisfaction. The data can be analyzed and processed to verify the strengths and weaknesses of the system To determine the consumer opinions we use the Likert scale [14], a psychometric scale commonly involved in research that employs questionnaires. It consists of a series of statements linked to the attitudes that are investigated. A similar approach will be used to design the tests for the In-Line game.

Conclusions

Overall, the survey shows that the system was received with good enthusiasm and the users enjoyed the tests. They consider it a breakthrough and would definitely use it. An interesting aspect that emerged is that all users wish to obtain as much information as possible about the environment around them.

A key aspect of the formation of mental maps in blind subjects is linked to the memory of the paths and the location of objects that they develop with repetition and practice. If a person follows in his mind a small-scale model of the external reality, he is able to think of various alternatives, choose the best among them, react to future situations before they happen The obtained results stimulate us to further develop the game and in particular to let the blind people at the Blind Institute use the game together with ARIANNA and then running more tests and collect qualitative and subjective feedback trough questionnaires.

Fig. 1 .

 1 Fig. 1. The line captured by the phone

  Fig. 2. In.line virtual path in the second release of the game

  2.b))


Felix Born
email: felix.born@uni-due.de

Maic Masuch
email: maic.masuch@uni-due.de

Masking Distracting Ambient Sound in an Adaptive VR-Application to Increase Presence

Keywords: Virtual Reality Gaming, Immersion, Presence, Adaptivity, Auditory Immersion, Audio Masking

The perception of disruptive outside sound while being inside a virtual reality simulation can break the experience of presence. Even with noise cancelling headphones external sound cannot be blocked completely. In this paper, we present an acoustic compensation method to sustain the virtual illusion. We developed a testbed VR prototype that allows to classify real-life sound and to adapt the virtual world accordingly by activating pre-defined playable content. The application analyzes and classifies outside sound in real time and triggers a suitable in-game object that matches the outside sound. Our implementation is a first approach, we want to use it to further examine the possibility of adaptive audio to mask external disruptive sounds resulting in an enhanced VR experience.

Introduction

Being inside an immersive Virtual Reality (VR) simulation allows the user to fade out the real surrounding and focus on the presented virtual illusion. In full-immersive VR, visual stimuli of the real surrounding are blocked out and thus allow to create the experience of presence. Presence is often described as the feeling of being there [START_REF] Heeter | Being there: The subjective experience of presence[END_REF], accompanied by fading out the real surrounding and experiencing the virtual world as the real world. Besides the importance of different visual aspects [START_REF] Cummings | How immersive is enough? a meta-analysis of the effect of immersive technology on user presence[END_REF], a realistic auditory environment is also a major influence on presence in virtual worlds [START_REF] Hendrix | The sense of presence within auditory virtual environments[END_REF][START_REF] Poeschl | Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence[END_REF][START_REF] Wood | The structural characteristics of video games: A psycho-structural analysis[END_REF]. In contrast to visual perception, auditory cues of the reality can still reach the user and thus affect the user. Having two different sources of auditory information can lead to a problem due to human's nature of processing different input modalities. According to Wickens [START_REF] Wickens | Processing resources in attention, dual task performance, and workload assessment[END_REF] this will lead to a conflict and to an increased cognitive demand to fade out the disruptive sound. This instance is a major problem for applications that mostly rely on the experience of the player being part of the virtual world, such as VR-systems that are used during local surgeries to distract the patient from potential stress inducing stimuli [START_REF] Hoffman | Virtual reality pain control during burn wound debridement in the hydrotank[END_REF]. One solution would be to integrate the distracting sound elements in the VR world, maintaining the illusion by pretending there is a reason for the sound. This idea raises the question to what degree such a workaround would be beneficial to the immersion or presence? To answer this, we developed a testbed prototype that allows to recognize real-life sounds, and thus to adapt the visual environment accordingly. The virtual adaptive world is based on classification algorithms that allow an analysis of the auditory environment of the real world. Hence, the virtual world changes according to the classified sound in order to match the virtual visual elements with the real auditory surrounding. Offering this match of sensory information ought to help players experience presence and ignoring possible disruptive perceptions.

Related Work

Effects of Disruptive Audio

One of the main prerequisite for the experience of presence is sensory immersion [START_REF] Witmer | Measuring presence in virtual environments: A presence questionnaire[END_REF]. Sensory immersion is defined by the ability of a system to deliver sensory cues that are close to real perceptions [START_REF] Slater | A note on presence terminology[END_REF]. Thus, immersion can be described as an objective quality of a medium. The level of immersion increases if more senses and cognitions of the player are addressed. Hence, using modern VR-systems allows to create a high level of sensory immersion by presenting nearly photorealistic stereoscopic images and thus creating a greater experience of presence. In-game sound is a core element of video games and modern VR-applications and a crucial factor for experiencing presence [START_REF] Hendrix | The sense of presence within auditory virtual environments[END_REF][START_REF] Poeschl | Integration of spatial sound in immersive virtual environments an experimental study on effects of spatial sound on presence[END_REF][START_REF] Wood | The structural characteristics of video games: A psycho-structural analysis[END_REF]. While modern VR-systems use head-mounted displays (HMD) that render the stereoscopic image and shield external visual cues from the user, intense sound of the real world can still reach the user and thus disrupt the presence of the player. Though this argument also applies for other sensory perception such as haptics, olfaction, or gustation, auditory perception and the experienced realism of the auditory cues are besides visual perception, the main influencing factors of a virtual experience [START_REF] Wood | The structural characteristics of video games: A psycho-structural analysis[END_REF]. In an experiment of Wharton and Collins [START_REF] Wharton | Subjective measures of the influence of music customization on the video game play experience: A pilot study[END_REF] participants reported that the main reason for listening to additional music while playing games was to mask external sounds and therefore increase their immersion. This underlines the need for users to immerse with the virtual world and to fade out external stimuli. Furthermore, Wharton and Collins [START_REF] Wharton | Subjective measures of the influence of music customization on the video game play experience: A pilot study[END_REF] examined, that the wrong selection of music can have an adverse effect and reduces the experienced immersion, especially if the music does not fit the visual content. Hence, non-matching visual and auditory cues can break the experience of presence.

Adaptive Audio

Active Noise Cancelling (ANC) Headphones are a common approach to fade out disruptive sounds. These headphones eliminate sound using destructive interference and are often used to shield the user from distracting ambient sound [START_REF] Karalar | Reduced pain and anxiety with music and noise-canceling headphones during shockwave lithotripsy[END_REF][START_REF] Riecke | To move or not to move: can active control and userdriven motion cueing enhance self-motion perception (vection) in virtual reality?[END_REF].

Due to different ear forms of every person and to the circumstance that sound is also transferred by the cranial bone, ANC Headphones can not eliminate the sound completely but rather reduce it. In addition, ANC headphones have a varying effectiveness according to the present sound pitches [START_REF] Gan | An integrated audio and active noise control headset[END_REF][START_REF] Karalar | Reduced pain and anxiety with music and noise-canceling headphones during shockwave lithotripsy[END_REF][START_REF] Molesworth | The effect on recognition memory of noise cancelling headphones in a noisy environment with native and nonnative speakers[END_REF]. Furthermore, the use of ANC headphones can filter out important information that must be perceived, i.e. instructions during medical procedures, and are therefore not applicable in several fields of application. Hence disruptive sound can still reach the user. To counter the problem of disruptive sound, the approach of adaptive audio exists. Adaptive audio in games refers to the ability of a game to change the in-game sound or music according to the game play [START_REF] Eladhari | The soundtrack of your mind: mind music-adaptive audio for game characters[END_REF]. A well-known example for the first application of this technique can be found in the game "Super Mario Bros." 1 , where the tempo of the music increases when the timer begins to run out. However, most of the digital games only consider the game state as a possible music changing event and do not include external auditory sources such as music or ambient sound of the real world. Nevertheless, games like "Audiosurf"2 , "Beat Hazard"3 or "Audio Shield"4 generate game content according to preselected music. These are first approaches to adapt the game world to external influences, however, the chosen music will be analyzed prior to the game. Hence, ambient sound changes in the real world are not considered. "The Polynomial"5 takes this one step further and considers microphone input and thus manipulates the visual game world accordingly. This game world is adapted in real-time and is a first approach to match the visual impressions with the auditory cues of the surrounding. However, "The Polynomial" only changes the visual surrounding and does not adapt the gameplay according to the real-life sounds. So far, we could not find any application, that allows such a game play adaptation based on real-life ambient sound. Therefore, we have developed a fully functional testbed prototype of an adaptive VR-Game that generates games and regulates the visual surrounding based on real time classified auditory cues.

Development of an Adaptive VR Game

Based on the concept of an VR application that considers real-life sound and changes game world elements accordingly and thus integrate external sounds in a meaningful way into the virtual world, we examined the idea of an adaptive audio VR-game. Therefore, we developed a fully immersive VR-environment using the HTC Vive to display the rendered virtual world and an external microphone to analyze all auditory cues of the real-world. We created a scenario that allows to explore four different types of environments. Every environment is located in a different area, allows different kinds of playful interaction, and is activated Fig. 1. Each environment and the according mini-game that can be activated is located in a different area in the virtual city. Based on the location, the dominant sound of the environment is different.

when the outside sound pitch is similar to the dominant sound of the according environment. Hence, we try to achieve a fit of the virtual environment and the real-world surrounding. The four different situations are a showcase for four discriminable tone pitches and are a first approach to integrate outside sound in a meaningful way. There is no interactive walkthrough possible since every environment is only active, if the real-life sound is present. Our development is a first approach of a testbed game that allows to examine how the perception of disruptive sound can be reduced and which tone pitches are of particular importance. The virtual world was implemented using the game engine Unity. The game is a 3D-Singleplayer game which takes place in a virtual city that consists of different buildings, railroads, airports, and a street system. The perspective of the player is a top-down view that allows an observation of the complete urban surrounding. The virtual city is located on an island that is surrounded by water. All assets are designed using a low-poly style, since we did not focus on a photo realistic setting but rather on a first testbed prototype. Besides activating virtual environments according to the real-life sound pitches, the volume of the ambient real-world sound regulates the dynamic of the city. Noisy surrounding leads to a high volume of traffic on the streets and in the air.

Adaptive Elements

The game incorporates two different adaptive approaches. On the one hand, mini-games are triggered according to the pitch of the real-world sound. On the other hand, an increase of the total real-world volume increases the city's traffic and thus leading to a higher traffic ambient sound.

Mini-Games There are four acoustic places that come along with a minigame. Each of the mini-games is activated when the real-life sound is classified accordingly. The first mini-game (Game A) takes place in a park near the city Fig. 2. Illustration of the location of the mini-games. Each mini-game is located in a different environment with ambient sound. Game A is located in a natural environment next to a bonfire with a medium-high sound pitch. Game B takes place on top of a building next to an AC system referring to a high sound pitch. Game C is located in the tower of an airport that is similar to medium-low sound pitches and game D sets the player next to rail station that represents low sound pitches. and consists of a bonfire which is about to run out of wood. The player's task is to rapidly gather wood to increase the fire. The dominant sound in this environment consists of medium-high frequency sounds, such as bird cheeping. The second mini-game (Game B) is located on top of a building and allows the player to paint on a wall using the controller as spray can. The sound of the spray can as well as the sound of the air conditions system, that is installed next to the wall are high frequency sounds. In the third mini-game (Game C) the player is inside the airport tower and controls departing airplanes. The sound of the engines is situated in a medium-low frequency. The fourth mini-game (Game D) is near the central station and the player has to clean and repair incoming trains. Here the situation suggests a low frequency sound due to the mechanical surrounding. Every game will be activated when the corresponding sound is classified and only one game is playable at the same time. The surrounding of each mini-game matches a certain real-life tone pitch. If this tone pitch is recognized by the game, the according game is activated. A real-life surrounding that is similar to bird cheeping will lead to the bonfire game, whereas deep industrial noise will trigger the tram repair game.

Traffic As a metaphor, outside sound is mapped to in-game traffic. This mechanism allows to manipulate the visual ambient environment and matches real-life sound to the sound of the driving cars and airplanes. Within the city, a street system with several parking lots exists. At the start of the game, all cars are parked. If the system recognizes a certain volume of the real-world sound, cars will start to drive a random route in the city. The higher the real-world volume is, the more cars are driving. The same principle applies for airplanes. If the realworld sound decreases, cars will park at the nearest parking lot and airplanes start to land.

Discussion and Future Work

Our goal is to improve immersion and presence in VR despite disturbing outside sounds. Hence, we developed a fully functional prototype that allows to analyze real-life ambient sound and to change game elements within the virtual world accordingly. We facilitate the possibility to integrate potential disruptive sounds of the surrounding in a meaningful way into the game world. Thus, we maintain the illusion and the experience of presence. Considering the works of Hoffman [START_REF] Hoffman | Virtual reality as an adjunctive pain control during burn wound care in adolescent patients[END_REF][START_REF] Hoffman | Virtual reality pain control during burn wound debridement in the hydrotank[END_REF][START_REF] Hoffman | Virtual reality helmet display quality influences the magnitude of virtual reality analgesia[END_REF], who examined the use of VR to distract patients from painful and stress inducing medical procedures, our approach can be used to increase the distracting effect of the VR world by countering discomforting ambient sound in such situations. This field of application was our initial motivation for this investigation. Our development thus serves as a testbed game which we will further evaluate. We want to use our testbed game to examine which tone-pitches and sounds are specifically suitable for the adaptive approach. We hope to use the findings of our future research to examine guidelines how a diverse adaptive audio application should be implemented and which factors are crucial to increase presence. For further development, the site of operation should be considered when adapting the game. Thus, the sounds that will occur during the usage of our prototype can be estimated and result in game changes that are closer to the real-life sounds. For this instance, it is advisable to pre-analyze disruptive sounds of the site of operation and create acoustic fingerprints for each sound that might occur. These fingerprints can be used for a comparison at the runtime of the game. If a stored fingerprint matches the current sound, the game can instantly activate game content to match this specific sound. Thus, pre-defining situations as it is implemented in our prototype, would not be necessary. Ideally, such an adaptive virtual world could react to a great variety of outside sounds and generate a more persistent game experience. An extension to the existing idea would be the use of spatial sound. So far, the position of the sound source is not considered. Generating elements in the virtual world that match the real-life sound should be located at the same place in the virtual world as the sound source in the real world. Otherwise the visual representation does not match the acoustic surrounding and can therefore be a presence breaking problem. This research presents a first approach to improve presence and immersion despite disturbing outside sounds. Though shortcomings exist and the prototype has to be adapted for the according field of application the method can contribute to deliver a more compelling VR experience.

Nintendo 1985,Super Mario Bros.

Invisible Handlebar 2008, Audiosurf

Cold Beam Games 2009, Beat Hazard

Dylan Fitterer 2016, Audioshield

Dmytry Lavrov 2010, The Polynomial


Yukihiro Achiha

Takayoshi Yamashita

Mitsuru Nakazawa

Soh Masuko

Yuji Yamauchi

Hironobu Fujiyoshi

Word Recognition by Combining Outline Emphasis and Synthesize Background

Keywords: character recognition, synthesis image, CNN

Character recognition collects item keywords from images from e-commerce websites; however, it requires a huge amount of training data. In this paper, we propose an efficient method to collect the training data by generating synthesis images and emphasizing outlines to obtain realistic images. The proposed method improves recognition accuracy on both generated images and real images from e-commerce websites.

Introduction

Deep Convolutional Neural Network (DCNN) is a common approach in character recognition of handwritten characters and signs in scenes. Character recognition in scene images consists of two parts: character detection and recognition [START_REF] Liu | Lexicon-driven segmentation and recognition of handwritten character strings for Japanese address reading[END_REF]. This technique can be applied to item images from e-commerce websites to collect the item's information. DCNN requires a huge amount of training data in order to obtain high accuracy [START_REF] Lecun | Gradient-Based Learning Applied to Document Recognition[END_REF] [START_REF] Wang | End-to-End Text Recognition with Convolutional Neural Networks[END_REF]. Although public datasets are available, the range of fonts in these datasets is too small. In order to address this problem, a character synthesis method was proposed to reduce the image collection cost [START_REF] Jaderberg | Synthetic data and artificial neural networks for natural scene text recognition[END_REF], which generates synthesis images by using font data and background images. However, it assumes English characters on a simple background. In this paper, we propose a method to generate synthesis characters and word images using Japanese font data and complex background images for item images from ecommerce websites. In addition, we introduce an approach to emphasize the outline of characters. Our method trained with synthesized images, which include both with and without outline emphasis, improves recognition accuracy.

Proposed method

The proposed method consists of three steps: generation of character images, addition of margins and emphasis of outlines, and synthesis of the characters with a complex background. Figure 1 shows the flow of character image generation. First, character images are generated from character lists with font data; these

Fig. 2. Addition of margins and emphasis of outline

!"#$%&'()*+ ,-"%.

/.).&"0.* ,-"%.

12)03.4,4 ,-"%.

Fig. 3. Synthesis of background image

images are then synthesized with a background image. We prepared 22 fonts commonly used in e-commerce websites such as MS Mincho and Yuri Gothic. Character images on complex backgrounds of e-commerce images are sometimes decorated, for example with borders. In order to improve recognition accuracy of such characters, outline emphasis of characters is introduced during character generation. First, a margin is added to the generated image. Then, an outline is added to emphasize characters in the image. At that time, two types of images are generated with different outline thicknesses. Figure 2 shows the flow of the addition of margins to the character image, followed by outline emphasis.

The background region is replaced with a complex background such as an item image. As shown in Fig. 3, the color of the background is green, and the color of the character and the outline are different. The background image is cropped randomly from a banner image of an e-commerce store.

The DCNN is trained by using the synthesis character images. To test the effectiveness of the method, we also applied the proposed method to word synthesis. The word images are generated based on a word list and synthesized with complex backgrounds.

Structure of DCNN

Figure 4 shows the DCNN network structure. The network consists of 4 layers: 3 convolution layers and 1 fully connected layer. The filter size of each layer is 5 × 5. Max pooling is employed for the pooling layer. The fully connected layer has 4,096 units and it employs Dropout [START_REF] Hinton | Improving neural networks by preventing co-adaptation of feature detectors[END_REF] during the learning phase. The activation function of each layer is ReLU [START_REF] Nair | Rectified Linear Units Improve Restricted Boltzmann Machines[END_REF]. The output units have 1,253 classes for AdaGrad [START_REF] John | Adaptive subgradient methods for online learning and stochastic optimization[END_REF] is used for the optimization method. The mini batch size is 32 and the epoch number is 50.

Evaluation

First, we evaluate the effectiveness of outline emphasis and synthesis with background images using top 5 accuracy. We trained 1,253 characters using 145 images for each (Fig. 5). For evaluation, 227 images collected from e-commerce websites were used. Table 1 shows the result for synthesis images. The method with emphasis and synthesis achieves best performance on top 1 accuracy. The evaluation results of character recognition and word recognition on real images of e-commerce are shown in Table 2 andTable 3, respectively. From Table 2, the method with emphasis performs 12.4% better than baseline, which is without emphasis and synthesis, on top 1 accuracy. On the other hand, the method with synthesis also improves 13.8% than baseline on top 1 accuracy. The combination of emphasis and synthesis achieves best performance with an improvement of 14.0%. The results of word recognition on real images are shown in Table 3. The method with emphasis improves about 20.4% and 19.8% than baseline on top 1 and top 5, respectively. Synthesis is also effective for real images; it improves accuracy by 24.4% and 22.3% than baseline on top 1 and top 5, respectively. The combination of emphasis and synthesis achieves best x 99.1 99.6 99.8 99.9 99.9 x x 99.6 99.9 99.9 99.9 99.9

Conclusion

In this paper, we proposed a method to generate outline emphasis of word images and synthesize them with complex background images. The DCNN trained with generated images obtained high accuracy on both synthesized images and real images from e-commerce websites.

Fig. 1 . 3 4

 13 Fig. 1. The flow of character image generation

Fig. 4 .

 4 Fig. 4. The structure of DCNN

!Fig. 5 .

 5 Fig. 5. Example of synthesis images

!Fig. 6 .

 6 Fig. 6. Examples of recognition results

Table 1 .

 1 The comparison of character recognition on synthesis images[%]

	outline background top1 top2 top3 top4 top5
	-	-	99.4 99.8 99.8 99.8 99.9
	x	-	96.0 97.7 98.2 98.6 98.8
	-

Table 2 .

 2 Comparison of character recognition on real images[%]

	outline background top1 top2 top3 top4 top5
	-	-	50.3 62.0 66.2 69.6 71.5
	x	-	62.7 72.5 75.8 77.5 78.7
	-	x	64.1 73.1 76.9 79.4 80.9
	x	x	64.3 74.4 78.2 80.2 81.4

Table 3 .

 3 Comparison

			of word recognition
	on real images[%]
	outline background top1 top2 top3 top4 top5
	-	-	27.6 34.1 36.8 40.2 42.1
	x	-	48.0 56.0 58.8 60.7 61.9
	-	x	52.0 58.5 60.7 63.8 64.4
	x	x	62.8 69.3 70.9 73.4 76.2


Kazuhisa Akimoto
email: akimoto.kazuhisa@entcomp.esys.tsukuba.ac.jp

Junichi Hoshino

RoughON : A Portable Music Sketch Production System by Real Time Input of Melody

Keywords: making music, real-time, production system, portable

In order to produce a music sketch (a combination of melody and accompaniment) in the beginning of musical composition, it is effective to use a composition means with high portability capable of inputting a melody in real time. In this paper, we propose a system that enables a beginner of musical composition or instrument easily and readily to create a music sketch by constructing a system capable of inputting a natural melody on a smartphone in real time in concert with accompaniment. In addition, we implemented the proposed system as an application to evaluate the proposed system through extensive use experiments by releasing it to the general public, showing the effectiveness of the proposed system in musical composition.

Introduction

In recent years, smartphone composing applications and music posting sites are well-supported [START_REF] Christopher | The YouTube effect: How YouTube has provided new ways to consume, create, and share music[END_REF][START_REF]Apple Computer[END_REF][START_REF]Apple Computer[END_REF]. As the environment in which composing activities can be easily done has become improved, novice musical composers and instrument users begin to occupy a part of users who use composition tools. In this paper, we propose a system to support novice musical composers.

For a novice musical composer or instrument user who is not easy to play, it is difficult "to produce a natural combination of melody and accompaniment" such as accompaniment production and the harmony of melody and accompaniment. Musical compositing support research thus far has realized a system to produce a combination of melody and accompaniment (hereinafter referred to as "music sketch") by automatic musical composition such as automatic accompaniment generation [START_REF] Simon | MySong: automatic accompaniment generation for vocal melodies[END_REF] and automatic lead sheet generation [START_REF] Papadopoulos | Assisted Lead Sheet Composition Using FlowComposer[END_REF]. However, for users such as who want to publish a produced song, because they have a main purpose of selfexpression, it is desirable to be able to directly and easily input the composing elements of a melody such as a rhythm, tones and a chord. Furthermore, portability is also necessary to form their inspiration. Portable systems [START_REF]Apple Computer[END_REF][START_REF]Apple Computer[END_REF] are also available, but their support is not enough compared to the existing composition support researches. Therefore, in this paper, we propose a portable system that allows users to input a melody in real time during automatic playing of accompaniment, thereby enabling to directly and easily enter the composition elements of a melody. In this system, we use a smartphone to generate a corrected melody so as to be able to input a natural melody even on the small screen so that it is consistent with the accompaniment and the melody up to the time of input. It also outputs musical score and MIDI, enabling to coordinate with instrument playing and DTM. In this paper, in order to verify its usefulness, we released it to the general public as iOS and Android applications [START_REF] Roughon | [END_REF]

Generated tones

❶ ❷ ❸ ❹ E F A C E C G F A G E G C E G C

Arrangement of tones on screen

Put the same position Output

System Overview

We show the system outline in Fig. 1 to illustrate the flow of the proposed system following the numbers in the Figure .   The user produces a music sketch by 1) Selecting one out of 12 kinds of accompaniments prepared in advance, and 2) Inputting an melody extemporarily in real time during the automatic playing. The input screen at this time is shown in Figs. 2 and During the time of 2), since the system generates 3) pitches constituting a natural melody, input by the user becomes a natural melody. The pitch candidates to generate is multiplied by a key and constraints imposed by the last user's input. Instead of arranging the pitch candidates randomly on the screen, in order to follow the relative pitch, the pitch candidates are arranged so that the higher the sound becomes as it advances in the positive direction of the horizontal axis of the screen, and the lower the sound becomes as it advances in the negative direction. Also, as long as the user continues to enter at the same position, the pitches are arranged from the position the user input last so that the same pitch is played. This pitches arrangement can be defined by a recurrence formula starting from i as follows. The algorithm is shown in Figs. [START_REF] Roughon | [END_REF]. Let S (k) = {s

! " # $ % & ' ! " $ & " & $ !"#$% "$&'()* + ,-".
(k) 1 , s (k) 2 , ..., s (k)
|N | } be the pitch sequence on the input screen at the time of input number k and

P (k) = {p (k) 1 , p (k) 2 , ..., p (k)
|M | } be the usable pitch candidate array. P (k+1) is generated depending on the input pitch s (k) i and chord. At that time, the pitch sequence S (k+1) at the time of input number k + 1 becomes as follows.

s k+1 j =        s k i (i = j) max m,s (k+1) j+1 >p (k+1) m p (k+1) m (i > j) min m,s (k+1) j-1 <p (k+1) m p (k+1) m (i < j).
(1)

After inputting the melody, the system edits the input melody, and 4) outputs the music sketch in the form of music score or MIDI. The score screen is shown in Figs. 53 Experiment We implemented the system as a smartphone application "RoughON" [START_REF] Roughon | [END_REF] to release it to the general public. As of January 27, 2017, the number of unique users of the application reached 4300. In addition, the cumulative number of music sketch productions exceeds 10,000 (music: 9800, chord: 3500), and the cumulative output number in MIDI format reaches 3300. Furthermore, even now it is used about 100 times every day.

!; <+)6$=>+ 04? <+-4=@)+ )6$=>+04? ! " # $ % & '$ '& !ABC !D 84)6$=5 3/:9(1 E4F9)$F90 *(G/10
A questionnaire was used for the evaluation experiment. We show the questionnaire, the each method and the number of subjects in Table . 1. The user who installed the application and made music sketch more than 5 times was the subject, and the questionnaire was divided into 2 times according to the period. The number of subjects at the first time was 46 people, and the number of subjects at the second time was 38 people. However, because of questionnaire by public disclosure, the number of answers is different for each question item.

Question contents are classified into two types: question asking the attributes of subjects and question asking system evaluations obtained by subjects using "RoughON". The question method selectively uses the choices (Q1, 2, 4, 6) and the points (Q3, 5), and furthermore a free answer column was provided. In the scoring system, we made a 5-point evaluation with 1 step (5: very good evaluation, 1: very bad evaluation). The results of the evaluation experiment are shown in Fig. 7.

In Q1 and 2, we asked the subjects' experience in musical composition and instrument. In question Q1 asking the experience in musical composition, those who have no experience in musical composition and beginners accounted for about 80%. For Q4 asking the difficulty of music composition by musical instruments, about 40% answered that musical composition using musical instruments was possible, but nearly all of the respondents felt difficulty in musical composition using musical instruments.

Q3 and Q4 are questions concerning the evaluation as a system for supporting the music composition. Q3 asked the support of initial music composing, which is the purpose of the proposed system, having gotten 3.57 with a 5-point score evaluation. Compared to the score evaluation, in the free description column, many favorable opinions were obtained, saying that "the operation was intuitive and easy." Besides, there are also answers: "we can compose a music with a game feeling," "it is difficult to compose a music from the beginning, but we can easily take the first step if we use RoughON." When Q4 asked which one would you like to use: RoughON or other composition applications?, and 47 people out of 57 people who had use experience of other music composition applications answered that they want to use RoughON. As a reason for that, there were many answers: "simple and easy to use compared to other applications," similarly in Q3. In addition, there was also an answer that "Although other applications inhibit the pleasure of making a music because of a full automatic system, RoughON reasonably supports the composer." Besides, favorable answers for the accompaniment function, such as "it is easier to do if accompaniment is available at first" and "it can superimpose a sound easily" were obtained. In Q3 and 4, it was shown that the proposed system is useful for supporting the early stage of music composition.

Q5 and 6 are questions about the sufficiency of music sketch to develop it into a music. In the score evaluation conducted in Q5 asking whether music sketch would become a clue to complete a music, the average value was 4.0. Also, in Q6 asking if you would like to actually use music sketch, there were 23 subjects who answered that they would use it. They answered various uses, such as "they make use of it to their own song" as a place to use and "they use for the BGM of radio, game and movie." Q5 and 6 showed that the music sketch produced by the proposed system plays a role of helping to complete a music.

Conclusion

In this paper, we proposed a portable system for music composition and instrument novices to produce a music sketch with real time input. In addition, the proposed system was implemented, released to the general public, and evaluated as a smartphone application: "RoughON." As a result, it was shown that the proposed system played a role of supporting the first step of music composition, revealing that its output, music sketch, is suitable as an intermediate product for completing a music.

A wide range of experiments were carried out by public questionnaire, so many subjects were able to be obtained. However, problems such as not being able to confirm the detailed operation of the examinee, not being able to confirm whether the subject used the music sketch produced by the proposed system or not occurred. In the future, we will acquire operation history and plan to investigate more detailedly for users who use the proposed system frequently.

Fig. 1 .Fig. 2 .

 12 Fig. 1. Schematic chart of the proposed system.

Fig. 3 .

 3 Fig. 3. Input screen of application.

  3 and the select screen in shown in Figs. 4. As it is difficult to input it in real time in concert with the accompaniment at sight reading, display of a line showing the tempo(Tempo line and Time line), addition of percussion, and quantization of sound to reproduce were implemented as simplifying functions.

Fig. 4 .

 4 Fig. 4. Screen of selecting an accompaniment.

Fig. 5 .

 5 Fig. 5. Screen of a music score.

Fig. 6 .

 6 Fig. 6. The algorithm of arranging pitches.

Fig. 7 .

 7 Fig. 7. Graph of experimental results.(Q3, Q5 show the average and SE. )

,G C,D,E,F,G,A,B

  to conduct extensive use experiments.

		The Proposed System		Use Music Sketch
	User Choose Accompaniment Input Melody Input by touch any time	System Play Accompaniment Chord of Accompaniment Generation tones become Natural Melody Arrangement of tones Previous Input	Output Music Sketch (Score/MIDI)	DAW SNS Musical Instrument Music
		C,E		Achieve Users purpose
	times			Posting Music on Site
		Portable Device		Playing in the Band

Table 1 .

 1 Quenstions for evaluation.

	Question		Method Number of subjects
	Subjects' Experience
	Q1. Do you have any experience in musical com-	choices	84
	position?
	Q2. Can you compose with a musical instrument? choices	84
		System Evaluation
	Q3. Is RoughON useful in supporting the early	points	46
	stage of composition?
	Q4. Which would you like to use: RoughON or	choices	84
	other applications?
	Q5. Could a music sketch become a clue to music	points	38
	production?
	Q6. Do you want to use a music sketch in the	choices	37
	future?
	!"	!2
	#$%%$&'() *+,$)+-$%%$&'()	34'5678
	./01	7)69:0


Hayaki Kawata
email: kawata.hayaki@entcomp.esys.tsukuba.ac.jp

Kazuki Ookawara
email: okawara.kazuki@entcomp.esys.tsukuba.ac.jp

Masahumi Muta
email: masafumi.muta@rakuten.com

Soh Masuko
email: so.masuko@rakuten.com

Jun ' Ichi Hoshino

Lifestyle Agent : The Chat-oriented Dialogue System for Lifestyle Management

Keywords: Lifestyle improvement, Chat communication, Rule-based dialogue management

In this paper, we propose Lifestyle Agent, which manages user's lifestyle in a chat format. Lifestyle agent counsels users through chat and encourages the formation of motivation with the aim of improving users' lifestyle habits. In an evaluation experiment, we had users experience chat examples with Lifestyle Agent and gained knowledge on dialogue planning to promote improvement of lifestyle habits.

Introduction

Because lifestyle diseases account for the majority of domestic medical expenses in Japan, the Japanese government has promoted the prevention of lifestyle diseases of citizens [START_REF]Welcom to Ministry of Health, Labour and Welfare[END_REF]. Lifestyle-related diseases can be prevented by improving unfavorable daily lifestyle habits. However, because the daily internal changes due to lifestyle habits are extremely small, It is difficult to manage lifestyle habits with awareness of risks that lie a few years ahead. For lifestyle-related diseases caused by social stress, mental health care such as counseling is important.

In this paper, we propose a Lifestyle Agent that can is designed to improve lifestyle while enjoying communication. Lifestyle Agent performs user counseling through chat: the system asks questions about the user's lifestyle and health consciousness, gives advice appropriate to the user's lifestyle, provides support for improving lifestyle habits, etc. Users can improve their lifestyle habits while enjoying communication by counseling that is based on their own lifestyle habits. In evaluation experiments, we had users experience chat examples with the Lifestyle Agent, and gained insight on dialogue planning to make lifestyle habit improvement more effective and enhance the chat sustainability of users.

Chat systems are recently attracting attention in the dialogue field [START_REF] Banchs | IRIS: a chat-oriented dialogue system based on the vector space model[END_REF] [START_REF] Lasguido | Utilizing human-to-human conversation examples for a multi domain chat-oriented dialog system[END_REF]. The chat dialogue is treated as a non-target task: it is important to let the dialogue with the user continue longer. This seems to include the element of enjoying the dialogue itself. In this paper, we consider the dialogue system as a representation of entertainment products.

System Overview

This system is a chat-style dialogue system that is aimed at improving the lifestyle habits of users along with the agents. Fig. 1 shows the operation of the system. The knowledge database of this system is cited from the policies of the Japanese government posted online to prevent lifestyle-related diseases. Of these, "diet," "sleep," "alcohol consumption," "smoking," etc. are mentioned regarding measures for prevention of lifestyle diseases, and information on these four items is used in the dialogue.

Using the buttons at the bottom of the screen, users can talk to the system to encourage communication and record diet and sleep. Based on the user profile and the action history of the user, a common basis for dialogue is constructed, and the system provides knowledge suitable for each user and feedback based on it. The user can live a physically and mentally healthy life by acquiring correct knowledge and lifestyle habits.

Experiments

In constructing such a system, we examined what effect contents related to the user and contents not related to the user have on the user's interest/consciousness. As an impression evaluation experiment, we conducted an investigation of the impressions of persons who experienced the dialogue. A total of 12 subjects (11 men and 1 woman; subjects were in their 20s) experienced chat examples through the application, and we investigated changes in impression and consciousness in subjects.

Based on the results of the preliminary questionnaire, we classified subjects in Table 1 by the categories of eating habits, sleeping habits, alcohol consumption, and smoking. Here, the number represents the number of persons. Results of totaling the questionnaire results under this category are shown in Figs. 2 and3. Fig. 2 shows the degree of interest in lifestyle diseases before and after the experiment and the change in sense of crisis; Fig. 3 shows the feeling of crisis felt by persons in each category in response to the dialogue of each category.

In Fig. 2, in the group without dietary consciousness and the group with smoking habits, there was a significant difference in the improvement of the degree of interest after the experiment. Moreover, in Fig. 3, differences were noted not only concerning categories related to the groups, but also concerning the feeling of sense of crisis for other categories.

These results revealed that it is effective to divide the user type by diet, alcohol consumption, and smoking habits. By constructing dialogue planning based on these user types, it will be possible to increase the value of the user experience in the proposed system.

Conclusion

In this paper, we propose a Lifestyle Agent that manages lifestyle habits of users through chat-style communication. The proposed system understands the user's lifestyle and conducts guidance for improvement based on dialogue. The user can review (be conscious of) his or her own lifestyle habits while enjoying communication through dialogue with the Agent, who tries to connect to his or her life.

The results of the evaluation experiment revealed that it is important to differentiate the user type using three criteria (diet, alcohol consumption, smoking) in order to make the user conscious of his or her own lifestyle. In the proposed system, it is possible to increase the value of the user experience by changing the dialogue contents for each of these user types.

Lifestyle Agent is a dialogue system that can walk the same time as the user. In the future, we would like to make products with more entertainment elements by using various information from the users' daily lives.

Fig. 1 .

 1 Fig. 1. Example of chat usage and system configuration in the proposed system.

Fig. 2 Fig. 3

 23 Fig. 2 Changes in interest level (left) and sense of crisis (right) before and after experiment

Table . 2

 . Criteria of classification of subjects and number of persons in each

	Dietary consciousness	Sleeping consciousness	Dringking frequency	Smoking habits
	Yes	No	Yes	No	High	Low	Yes	No
	5	7	5	7	6	6	6	6


Kazuki Ookawara
email: okawara.kazuki@entcomp.esys.tsukuba.ac.jp

Hayaki Kawata
email: kawata.hayaki@entcomp.esys.tsukuba.ac.jp

Masahumi Muta
email: masafumi.muta@rakuten.com

Soh Masuko
email: so.masuko@rakuten.com

Takehito Utsuro
email: utsuro@iit.tsukuba.ac.jp

Jun ' Ichi Hoshino

Deep Photo Rally: Let's Gather Conversational Pictures

Keywords: Augmented Reality, Anthropomorphic, Deep Neural Networks

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Humans can feel familiarity and connection to personified objects [START_REF] Waytz | Social connection and seeing human[END_REF]. In order for humans to personify something, expressions interacting with the environment (effectance motivation) are considered important [START_REF] Epley | On seeing human: a three-factor theory of anthropomorphism[END_REF]. In recent years, the Deep Neural Networks technology makes it possible to handle complex information such as the state of an object and its surrounding environment.

In this paper, we propose an approach to anthropomorphize a specific object using the state-of-the-art technology of Deep Neural Networks. The anthropomorphization of an object in the proposal approach can be realized by speech expressions suitable for the state of the object and environment. A user can communicate with an anthropomorphized object by photographing the specific object with a mobile terminal. Finally, we introduce examples of application of the proposal approach to entertainment products, and show that it is feasible to realize an anthropomorphization approach capable of interacting with the environment. Fig. 1 shows the relatedness between a user and an object through the proposal approach. In the proposal approach, it is possible to anthropomorphize a specific object the user photographed with a mobile terminal. The object generates speech sentences based on the state of the object and the surrounding environment. The user receives organically changing speech sentences based on the environment. Then the object is anthropomorphized, and the user can feel familiarity with the object. Anthropomorphization based on Environment Recognition

Recognition of environment and object state

The object recognition technology in the proposal approach adopts YOLO 9000 [START_REF] Redmon | YOLO9000: Better, Faster, Stronger[END_REF] which is the state-of-the-art technology capable of detecting an object at high speed. Fig. 2 shows the network configuration of the actually adopted YOLO 9000. YOLO 9000 realizes extraction of input image features, detection of object regions, and estimation of object labels with a single Neural Network. Image features (information on the state of the object and environment) are extracted through multiple network layers. The image features are also used for speech generation described in Section 3.2. Object region coordinates and object label candidates are stored in the output layer. Multiple objects can be detected simultaneously by calculating the probability of the object candidates. For details of the model, see the document [START_REF] Redmon | YOLO9000: Better, Faster, Stronger[END_REF].

Speech generation for anthropomorphization

The technology of Neural Network (i.e. Google NIC [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF]) which automatically generates syntax from images has been proposed recently. In this paper, we adopt Google NIC as a speech generation technology for an object based on environment recognition. Fig. 2 shows a simplified network configuration of Google NIC. This network is constructed very simply: it is realized by network layers having such roles as image feature extraction, feature conversion, and syntax generation. The proposal approach aims to make a speech generation process more efficient by diverting the image features extracted by YOLO9000. By calculating the probability of the estimated word string, it becomes possible to generate a speech sentence reflecting the state of the object and environment. For details of the model, see the document [START_REF] Vinyals | Show and tell: A neural image caption generator[END_REF].

Building a network model

To build a network model in the proposal approach, you need to prepare an image containing an object to be recognized and a speech sentence of the object. You also need to prepare a set of object labels and area coordinates.

The proposal approach begins with learning of the object detection model (YOLO 9000) to establish a network that extracts image features at the beginning. After that, it performs learning of the speech generation model (Google NIC) by diverting the image features extracted by the object detection model.

The application example of the proposal approach introduced in this paper uses a large scale image data set called Microsoft COCO (MSCOCO) [START_REF] Chen | Microsoft COCO captions: Data collection and evaluation server[END_REF] as learning data. MSCOCO is a data set with arbitrary object labels and captions provided for the image. This time, we have changed the captions to speech sentences of the image with arbitrary label combinations, and made it learn them.

4

Application Example

Photo rally at the zoo

The Photo Rally System is shown as an entertainment product example in which the proposal approach has been augmented for a zoo (Fig. 3). In this system, you can photograph animals speaking based on the state of their own (i.e. eating, running) as well as their surrounding environment (i.e. people are gathered, staying near the water). You can also incorporate gamification factors such as creating your original animal encyclopedia, by using the object detection function.

Play house with plush doll

You can enjoy the augmented play house by anthropomorphizing the plush doll with the proposal approach. Fig. 4 shows the Play House System to which the proposal approach applies. Fig. 4 shows how the teddy bear is speaking according to the surrounding environment. When a user changes the surrounding environment, the teddy bear's speech also changes according to the environment. This system enables a user to have pseudo communication with a plush doll by operating the doll and its surrounding environment.

Conclusion

In this paper, we have proposed an approach to anthropomorphize a specific object using the state-of-the-art technology of Deep Neural Networks. The anthropomorphization of an object in the proposal approach can be realized by speech expressions suitable for the state of the object and environment. A user can communicate with an anthropomorphized object by photographing the specific object with a mobile terminal. The proposal approach adopts the Deep Neural Network technology, including object detection and image captioning. We can apply this technology to various products by utilizing the results. This paper could confirm the operation of Photo Rally at the zoo and Play House with plush doll as concrete examples. Photo Rally System is capable of creating an original animal encyclopedia by utilizing the object detection results. Play House System is capable of ensuring pseudo communication between a user and a plush doll by recognizing the environment of the doll. From here, we would like to examine the "effect of playing" in the example already described in this paper.

Fig. 1 .

 1 Fig. 1. The relatedness between a user and an object.

3

 3

Fig. 2 .

 2 Fig. 2. The proposal approach for anthropomorphization.

Fig. 3 .

 3 Fig. 3. The photo rally system with anthropomorphization.

Fig. 4 .

 4 Fig. 4. The play house system with anthropomorphization.