Nuno Jardim Nunes
email: nunojnunes@tecnico.ulisboa.pt

Marko Radeta
email: marko.radeta@iti.larsys.pt

Valentina Nisi
email: valentina.nisi@tecnico.ulisboa.pt

Enhancing Whale Watching with mobile Apps and Streaming Passive Acoustics

Keywords: Whale watching, Environmental Sustainability, Mobile Entertainment, User Experience, Ecotourism

Whale watching is a prime example of ecotourism services reaching millions of people worldwide. In this paper, we describe a prototype of a mobile application designed to enhance the whale watching experience. Integrating eco-acoustics and entertainment components, the authors designed an App that enhances customer satisfaction even in the case of distant sighing, which helps preserve animal well-being in this growing ecotourism activity. This paper presents the design and preliminary evaluation of the system, which involves many different components, from usability and presence known scales. We asked 12 participants to conduct several activities, including (i) report a sighting, (ii) classify a cetacean vocal call, and (iii) gather a cetacean photo. Because of the system's novelty, we report on an extensive evaluation framework, including usability and presence scales reported by 12 whale watchers. We discuss the implications of our findings on the design of future applications for ecotourism services.

Introduction

Environmental sustainability is one of the fastest-growing areas of activity in technology related-research [START_REF] Dourish | Hci and environmental sustainability: the politics of design and the design of politics[END_REF]. Many research efforts are examining the opportunities to use computing technologies to promote environmental protection, and ecological consciousness from various angles, including entertainment [START_REF] Arts | Digital technology and the conservation of nature[END_REF][START_REF] Jepson | Nature apps: Waiting for the revolution[END_REF][START_REF] Uzunboylu | Using mobile learning to increase environmental awareness[END_REF][START_REF] Chen | Travel recommendation by mining people attributes and travel group types from community-contributed photos[END_REF][START_REF] Schaal | Digital geogames to foster local biodiversity[END_REF]. This is a responsible reaction to the twin crisis of climate and nature, which is supported by overwhelming scientific evidence [START_REF] Oreskes | The scientific consensus on climate change[END_REF]. It is causing increasing public concerns [START_REF] Ratter | Between hype and decline: recent trends in public perception of climate change[END_REF]. Put simply, people and wildlife are at risk unless we take urgent action to mitigate and hopefully reverse the loss of plants and animals on which our ecosystems depend for food, clean water, and a stable climate [START_REF] Lavergne | Biodiversity and climate change: integrating evolutionary and ecological responses of species and communities. Annual review of ecology[END_REF].

Oceans are a crucial driver of earth ecosystems. They cover three-quarters of the Earth's surface, contain 97% of the Earth's water, and absorb 30% of the carbon emissions acting as a buffer for global warming. Careful management of this essential global resource is a vital feature of a sustainable future recognized by the United Nations Sustainable Development Goal 14 -Conserve and sustainably use the oceans, seas and marine resources 4 . However, observe a continuous deterioration of coastal waters due to pollution and ocean acidification compromising ecosystems and biodiversity. The UN SDG 14 goal ambitiously targets to protect 10% of the ocean by 2020, preventing or significantly reducing marine pollution [START_REF] Claudet | A roadmap for using the un decade of ocean science for sustainable development in support of science, policy, and action[END_REF].

At the present scale, although 12% of the Earth's land is conserved through arrangements such as national parks, only 5% of the ocean has any protection at all 5 . Marine Protected Areas are a useful management tool towards conservation [START_REF] Edgar | Global conservation outcomes depend on marine protected areas with five key features[END_REF]. However, their size and scope are not always adequate for protecting marine megafauna, like whales, dolphins, and sharks [START_REF] Cisneros-Montemayor | The global potential for whale watching[END_REF]. Conservation measures for these large pelagic animals depend much on data acquisition in the open ocean to ascertain their population's status and detect temporal and spatial patterns in their distribution. Such an endeavor usually relies on dedicated monitoring and vessel-based survey programs, which are typically costly and timeconsuming. Leading to often temporally sparse or sporadic and spatially patchy collected data. We can partially tackle these data gaps by taking advantage of the growing interest and offer of recreational whale-watching service providers, as these already spend time at sea, tracking cetaceans and other large animals. On the other hand, whale-watching activities engage people with cetaceans, one of the most charismatic fauna raising awareness towards the promotion of ocean conservation measures and actions [START_REF] Hoyt | A review of whale-watching and whaling with applications for the caribbean[END_REF].

Digital Entertainment for Nature Conservation

Digital technology is changing nature conservation in increasingly profound ways. Some have called this new movement "Wilderness 2.0" [START_REF] Stinson | Re-creating wilderness 2.0: Or getting back to work in a virtual nature[END_REF] or "Nature 2.0" [START_REF] Büscher | Nature 2.0: Exploring and theorizing the links between new media and nature conservation[END_REF] referring to the articulation of new media, mobile technologies, and locative media, with the practice of wilderness recreation [START_REF] Stinson | Re-creating wilderness 2.0: Or getting back to work in a virtual nature[END_REF] and online activities that stimulate and complicate the commodification of biodiversity and nature [START_REF] Büscher | Nature 2.0: Exploring and theorizing the links between new media and nature conservation[END_REF]. Digital technology is thus promoting a new reality (between the virtual and the real) where people connect and recreate a new social construct of nature and the underlying digital economy [START_REF] Stinson | Re-creating wilderness 2.0: Or getting back to work in a virtual nature[END_REF], reimagining ideas, ideals and experiences of nature [START_REF] Büscher | Nature 2.0: Exploring and theorizing the links between new media and nature conservation[END_REF]. In this new context, environmentalists have long expressed concern that increasing the mediation of non-human interactions [START_REF] Forlano | Posthumanism and design[END_REF] is contributing to diminishing support for conservation [START_REF] Fletcher | Gaming conservation: Nature 2.0 confronts nature-deficit disorder[END_REF]. Consequently, digital entertainment is increasingly promoted by environmental organizations precisely for their potential to inspire more active commitment to environmental causes than the direct experiences most conservationists advocate [START_REF] Fletcher | Gaming conservation: Nature 2.0 confronts nature-deficit disorder[END_REF][START_REF] Sandbrook | Digital games and biodiversity conservation[END_REF]. Using gamification and serious games promotes a similar strategy to engage consumers in pro-environmental behaviors for areas such as nature conservation [START_REF] Nunes | beeco: co-designing a game with children to promote environmental awareness-a case study[END_REF], energy efficiency [START_REF] Morganti | Gaming for earth: Serious games and gamification to engage consumers in proenvironmental behaviours for energy efficiency[END_REF], transport [START_REF] Yen | Gamification in transport interventions: Another way to improve travel behavioural change[END_REF], and sustainability [START_REF] Lee | Greenify: fostering sustainable communities via gamification[END_REF].

Enhancing the Whale Watching Experience

This project aims to test how digital entertainment could be used in whale watching to engage participants in this form of ecotourism and enhance their experience while promoting more environmentally friendly behaviors. From Moby Dick to Free Willy, the conceptualization of whales in entertainment and their societal representations and understandings have changed dramatically in the last century [START_REF] Wearing | Free willy: the whale-watching legacy[END_REF]. Lawrence and Philips propose that whale watching is one key example of an economic activity emerging from the radical changes that occurred in societal understandings and representations of nature and animals [START_REF] Lawrence | From moby dick to free willy: Macro-cultural discourse and institutional entrepreneurship in emerging institutional fields[END_REF]. Humans have been hunting whales since the beginning of history, but it was not until the 1930s that saw the development of international regulations to control the number and type of whales harvested [START_REF] Tønnessen | The history of modern whaling[END_REF]. Notwithstanding, these regulations were developed to manage oversupply and later overexploitation and still conceptualized the animals as "whale stocks" of valuable human resources and not as animals or species [START_REF] Lawrence | Watching whale watching: Exploring the discursive foundations of collaborative relationships[END_REF]. Changes in the regulatory discourse impacted the image of whales in popular culture. Most movies until the 1970s present a negative image of whales as "man-eating and ship destroying monsters" of which Moby Dick (1956) and 20,000 Leagues Under Sea (1954) are the most known examples. The movies produced after 1980 present a very different image of whales and their relationship with people. Perhaps the most emblematic example is Free Willy (1993), which is responsible for an increase in cetacean-related tourism activities like dolphin swimming and whale watching as well as visitations to marine amusement parks [START_REF] Wearing | Free willy: the whale-watching legacy[END_REF].

Our project aims at digitally enhancing the whale watching experience, while raising awareness about ocean sustainability concerns, such as stressing and disturbing the animals in their habitats. Whale watching is a USD 2.1 billion global industry, attracting more than 13 million people worldwide [START_REF] Cisneros-Montemayor | The global potential for whale watching[END_REF]. However, this activity is usually reasonably expensive for participants who, due to the elusive nature of these animals in the wild, often fail to see their (submerged) full-size body. At the same time, whale watching activities and pressure do affect animal behavior, and noise pollution impacts these animals as they much rely on sound to communicate, navigate, and even locate prey [START_REF] Parsons | The negative impacts of whale-watching[END_REF]. Our motivation for this research is to provide on boat activities that enhance the whale watching experience, allowing tourists to perform fulfilling interaction with the cetaceans while minimizing stressful situations for the animals themselves. For this purpose, our prototype enhances the whale watching activity by signaling the presence of cetacean and bringing the experience of their sounds and location to the tourists, preserving distance from the animal, and minimizing the invasion of their territory. In the remainder of this paper, we position our work against the state of the art of, describing the prototype design and preliminary evaluation on system usability and presence scales of whale watchers. Our results inform a discussion on the future of applications for nature engagement and sensing.

Research Questions

This work builds on other reported case studies designing novel interactive systems and services for tourism related applications [START_REF] Teixeira | Customer experience modeling: Designing interactions for service systems[END_REF][3] [START_REF] Nunes | Beanstalk: A community based passive wi-fi tracking system for analysing tourism dynamics[END_REF][42] [START_REF] Redin | Vitflow: A platform to visualize tourists flows in a rich interactive map-based interface[END_REF]. In our previous work [START_REF] Radeta | Poseidon-passive-acoustic ocean sensor for entertainment and interactive data-gathering in opportunistic nauticalactivities[END_REF], we designed a ubiquitous mobile system to enrich the whalewatching activities with passive acoustics. Our reported study was proof of concept in performing the real-time machine learning classification of cetacean vocal calls using low-cost sensing and communication. Here, we expand the work, allowing users to actively report the sightings, sounds classification, and media gathering while not approaching the animals beyond stress generating proximity. In this study, we obtain participants' feedback and the overall results of the systems' usability and presence. Therefore, this research mainly explores how successful the system is in providing a whale-watchers fulfilling and entertaining experience. Throughout several diverse usability and presence scales, we verify the collected feedback from the participants and discuss results in light of the application's future refinement.

Related Work

We situate this research in-between different three main areas of prior work: sensing and engaging nature, digital entertainment, and nature and whale watching as a form of ecotourism. In the following subsections, we address each one in turn.

Sensing and Engaging Nature

The advent of low-cost, ubiquitous sensing and computing leads to increased use of these technologies in biology, ecology, and environmental sciences, e.g., research on Eco acoustics [START_REF] Sueur | Ecoacoustics: the ecological investigation and interpretation of environmental sound[END_REF]. These efforts build on empirical evidence of biotic and non-biotic sounds used as a reliable proxy for investigating ecological complexity [START_REF] Krause | Using ecoacoustic methods to survey the impacts of climate change on biodiversity[END_REF], including biodiversity loss and climate change. These technologies enable the identification of several thematic priorities for research programs that take advantage of novel computer sciences and technologies. One notable example is AudioMoth [START_REF] Hill | Audiomoth: Evaluation of a smart open acoustic device for monitoring biodiversity and the environment[END_REF], a low-cost, small-sized, and low-energy acoustic detector. The device is open-source and programmable, with diverse applications for recording animal calls or human activity. AudioMoth deployed globally reached in 2019 more than 700 projects worldwide [START_REF] Hill | Audiomoth: Evaluation of a smart open acoustic device for monitoring biodiversity and the environment[END_REF]. The device also facilitated public engagement, including citizen science projects for bat surveys 6 , biodiversity distribution assessments7 , BirdsScience8 , cetalingua 9 , and engagement of the general public in wildlife monitoring programs 10 .

Similarly, the HCI community saw an opportunity to apply the same concept to camera traps. My NatureWatch Camera is an inexpensive wildlife camera designed for people to self-build a way to promote environmental engagement and produce digital content [START_REF] Gaver | My naturewatch camera: Disseminating practice research with a cheap and easy diy design[END_REF]. My NatureWatch Camera includes a smartphone interface that allows a facile setup and retrieval of images. The authors of the project worked closely with a TV Show and social media outreach to highlight the importance of co-development for research projects to circulate at scale. The users of the device also reported increased awareness about local nature and animals. POSEIDON [START_REF] Radeta | Poseidon-passive-acoustic ocean sensor for entertainment and interactive data-gathering in opportunistic nauticalactivities[END_REF] is a low-cost passive acoustic monitoring system for citizen science in nautical/marine settings, which provided whale-listening to whale-watching. Collectively, with other projects focusing on different taxa such as insects [START_REF] Potamitis | Classifying insects on the fly[END_REF][START_REF] Vasconcelos | Locomobis: a lowcost acoustic-based sensing system to monitor and classify mosquitoes[END_REF] or birds [START_REF] Sullivan | ebird: A citizen-based bird observation network in the biological sciences[END_REF], it illustrates the possibilities of applying the low-cost IoT technologies to sense nature and engage with the public.

Digital Entertainment and Nature

Entertainment and gaming approaches are increasingly used to raise awareness about serious issues in a wide range of fields, including biodiversity conservation [START_REF] Sandbrook | Digital games and biodiversity conservation[END_REF]. At the same time, environmentalists express concerns about the increase in technological mediation of human and non-human interactions, keeping people away from nature, and interacting with digital artifacts instead. This phenomenon could be contributing to a growing "nature-deficit disorder" (NDD) and thereby diminishing support for conservation [START_REF] Kareiva | Ominous trends in nature recreation[END_REF][START_REF] Fletcher | Gaming conservation: Nature 2.0 confronts nature-deficit disorder[END_REF]. These concerns are grounded in the premise that direct experiences in "natural" spaces, especially those gained in childhood, will inspire people to "love" non humans [START_REF] Milton | Loving nature: Towards an ecology of emotion[END_REF] and subsequently act in support of environmental causes [START_REF] Wells | Nature and the life course: Pathways from childhood nature experiences to adult environmentalism[END_REF].

Nevertheless, digital entertainment artifacts approaching environmental and animal preservation of natural premises are many. Examples range from Defend Your Territory11 , a game inciting player to kill evil animal poachers; Camera Birds12 which requires the player to take photos of birds in a virtual forest; Wounded iWhale Rescue13 inviting players to "save" (i.e. catch) whales. Deep water hero14 a digital puzzle game demanding players to break up an oil slick to save animals; WilderQuest15 , targeting 5 to 8 eight years old children, placing them in an immersive 360 degree virtual environment to encourage them to observe and discover the nature around them by holding the iPad like a camera.

On the other hand, addressing Milton and Kareiva concerns [START_REF] Kareiva | Ominous trends in nature recreation[END_REF][START_REF] Milton | Loving nature: Towards an ecology of emotion[END_REF], digital games encourage players to go into nature in order to interact with it. For example, with Wildtime App children or parents use their mobile phones to discover enjoyable activities in nearby green space 16 . Dionisio et al. developed a rich trans-media entertainment experience to promote the natural capital of a biodiversityrich tourist destination while physically exploring its real settings [START_REF] Dionisio | Transmedia storytelling for exposing natural capital and promoting ecotourism[END_REF]. Loureiro et al. propose a mobile game to be played in real parks and green spaces supporting the biodiversity awareness through bioacoustic data validation of local species [START_REF] Loureiro | Citizen science and game with a purpose to foster biodiversity awareness and bioacoustic data validation[END_REF].

In summary, mobile apps designed to support nature immersed recreation are emerging. They frequently integrate GPS and compass functionality with maps to support navigation, record observations and send alerts of unusual sightings to other users. The App is ActInNature Hunting, is an excellent example, which adds a social dimension to the above-described functionalities. A minority of Apps associated with visiting attractions (e.g. museums, botanical gardens and parks), provide augmented reality enhancements of exhibits [START_REF] Waterson | Delightfully lost: A new kind of wayfinding at kew[END_REF][START_REF] Mann | A study of the iphone app at kew gardens: Improving the visitor experience[END_REF][START_REF] Nisi | Augmented reality museum's gaming for digital natives: Haunted encounters in the carvalhal's palace[END_REF].

Our App, designed to support the recreation of the users while immersed in nature, builds and extends on state of the art by proposing the combination of acoustic sensing and animal sighting while connecting users in building a local database of data, open to a public of scientists and non. We paid particular attention to designing an application that would not stress or endanger animals while enhancing the whale watchers' experience while immersed in nature.

Whale Watching and Ecotourism

Whaling was an essential element of the industrial revolution, providing one of the first commercially viable commodities used as a source of energy. The charismatic nature of these giant mammals played an essential role in supporting the rise of environmentalism and NGOs like Greenpeace [START_REF] Blok | Actor-networking ceta-sociality, or, what is sociological about contemporary whales? Distinktion[END_REF]. More recently, cetaceans embody one of the prime examples of ecotourism services, with the whale watching industry reaching 13 million people worldwide (conducted in more than 119 countries) and accounting for more than 2 billion USD revenue [START_REF] Silva | How ecotourism works at the community-level: the case of whale-watching in the azores[END_REF].

From Moby Dick to Free Willy, the conceptualization of whales and their societal representations and understandings have changed dramatically over time [START_REF] Wearing | Free willy: the whale-watching legacy[END_REF]. Lawrence and Philips [START_REF] Lawrence | Watching whale watching: Exploring the discursive foundations of collaborative relationships[END_REF] propose that whale watching is a key example of an economic activity, which emerges from the radical changes that occurred in the societal understanding of nature and animals [START_REF] Blok | Actor-networking ceta-sociality, or, what is sociological about contemporary whales? Distinktion[END_REF]. Humans have been hunting whales since the beginning of history. However, it was not until the 1930s that international regulations were developed to control the number and type of whales harvested. Notwithstanding, these regulations were developed to manage oversupply and later over-exploitation, which still position the animals as a valuable resource to humans. The argument provided by Latour [START_REF] Braun | To modernise or ecologise? that is the question[END_REF] is that we can witness in human-whale entanglements a demonstration of a more general transformation of society referred to as "ecoligisation" [START_REF] Blok | Actor-networking ceta-sociality, or, what is sociological about contemporary whales? Distinktion[END_REF] (a hybrid world in which the co-production of humans and non-humans are constantly renegotiated) [START_REF] Braun | To modernise or ecologise? that is the question[END_REF]. Whales are thus prime examples of non-human actors, individualized in their "actorhood" and acquiring "personalities", which became increasingly plausible [START_REF] Teubner | Rights of non-humans? electronic agents and animals as new actors in politics and law[END_REF].

Focusing on the design of sustainable interactions with whales, we need to consider the debate between the "short-term" impacts and the "long-term" con-sequences of whale watching [START_REF] Corkeron | Whale watching, iconography, and marine conservation[END_REF]. Indeed, some authors have argued that whale watching corresponds to a metabolic rift of whale hunting, problematizing the pervasive assumption that whale watching correlates primarily and directly with conservation [START_REF] Neves | Cashing in on cetourism: A critical ecological engagement with dominant e-ngo discourses on whaling, cetacean conservation, and whale watching 1[END_REF]. Also, there have been other systems proposed to question the impact of climate change in the distribution and abundance of whales [START_REF] Lambert | Sustainable whale-watching tourism and climate change: towards a framework of resilience[END_REF]. Lambert and colleagues developed a resilience framework of whale watching to these changes. They defined resilience as the degree of change in cetacean occurrence experienced before tourist numbers fall below a critical threshold [START_REF] Lambert | Sustainable whale-watching tourism and climate change: towards a framework of resilience[END_REF]. Their framework combined likelihood of observing a cetacean, trip type and tourist type, which, when quantified, could identify which operators are more likely to experience a change in tourist numbers, given a specific scenario of changing cetacean occurrence. While these studies provided system and insight into the environmental consequences, Bentz and colleagues studied the satisfaction with whale watching tours in relationship to expectations and demographic variables and identified tour aspects that contribute to satisfaction [START_REF] Bentz | Enhancing satisfaction and sustainable management: Whale watching in the azores[END_REF]. Their study was based on expectancy theory, which suggests that participants engage in recreational activities expecting that this will fulfill their needs and motivations [START_REF] Kozak | A critical review of approaches to measure satisfaction with tourist destinations[END_REF]. It then compared importance-performance analysis and performanceonly perspective to measure satisfaction of whale tour participants. Environmental friendly conditions were the most important expectation: Seeing one whale, seeing lots of whales, the cost of the trip and the boat type were the most influential factors contributing to satisfaction [START_REF] Bentz | Enhancing satisfaction and sustainable management: Whale watching in the azores[END_REF].

System Description

In this section, we briefly describe the whole system setup, extending POSEI-DON [START_REF] Radeta | Poseidon-passive-acoustic ocean sensor for entertainment and interactive data-gathering in opportunistic nauticalactivities[END_REF] with a custom made, on-boat mobile application, designed to support and enhance the whale watching activities. The system provides additional adhoc scientific information supporting awareness and inspiring conservation of marine megafauna.

Sensing Hardware and Mobile Application

The eco-acoustic sensing station is encapsulated in an autonomous, buoyant Styrofoam sphere, which contains the microcomputer with the WiFi connection, powered by the solar panels and containing the media equipment (Figure 1). Once at the whale-watching spot, the capsule is thrown into the water, and dragged from the sea vessel with a common cable. The acoustic buoyant system was previously described and discussed in [START_REF] Radeta | Poseidon-passive-acoustic ocean sensor for entertainment and interactive data-gathering in opportunistic nauticalactivities[END_REF].

The novel POSEIDON mobile application (Figure 2), completing the PO-SEIDON system, is designed to provide users with three diverse types of information: (i) general information about cetaceans; (ii) media gallery (showcasing the collected sound and images); and (iii) three action windows (report a sighting, classify a sound, and gather underwater and surface photos). The Watkins Marine Mammal Dataset [START_REF] Sayigh | The watkins marine mammal sound database: An online, freely accessible resource[END_REF] inspired the cetaceans used in our application. Our 1. The App supports the storing and collection of reported marine megafauna sightings, shared by previous users. Once an animal is sighted, its GPS position and the sighted species, can be logged by the user on the map interface of the App (Figure 2. a, g). 2. The App reports sounds recordings of the sensing of megafauna underwater, portraying the live acoustic feed from hydrophone hardware. The sound reports can be access on a separate screen. A detailed description of this part of the app is provided in [START_REF] Radeta | Poseidon-passive-acoustic ocean sensor for entertainment and interactive data-gathering in opportunistic nauticalactivities[END_REF] (Figure 2. b). 3. The App stores the user's photos, captured by means of pointing and clicking the mobile device. Action triggers a contemporary underwater shot, performed by the buoyant hardware system and shared to the App (Figure 2. c). 4. In order to collect information regarding the App usability and the user satisfaction the App presents a screen Survey to be filled by individual users at the end of the experience (Figure 2. d). 5. The App presents the Watkins Marine Mammal Dataset collection [START_REF] Sayigh | The watkins marine mammal sound database: An online, freely accessible resource[END_REF] to facilitate users recognition and reporting of specific species. The user can select the animal that they think they have sighted at any given moment during the whale watching trip (Figure 2. e). 6. The App contains screens with scientific information and details of the species that have been sighted, according to the Watkins Marine Mammal Dataset collection [START_REF] Sayigh | The watkins marine mammal sound database: An online, freely accessible resource[END_REF] (Figure 2. f). Fig. 3: POSEIDON images depicting spotted cetaceans during the user study.

Tasks Performed by Whale Watchers

In summary, the App enables tourists' whale watchers to perform a set of additional tasks that enhance the experience, without disrupting the animal's wildlife:

1. Reporting a cetacean sighting -Whale watchers are presented with the map of possible cetacean sightings (Figure 2. a) and they are asked to report if they spot a cetacean. Once the cetacean is spotted, the user proceeds with selecting the REPORT button (Figure 2. e) and is guided to insert the additional details with the help of the researchers on board (Figure 2. f). These details include the duration and type of the activity, group and calves' size, Beaufort wind scale and the boat type. Once the report has all the information, the user adds the report, which instantly appears as a red spot on the map (Figure 2. g).

Classifying a cetacean vocal call -Participants can hear the cetacean vocal

calling, which is being recorded by the hydrophone and has been verified as one of the five second acoustic samples with the prediction of above 0.75. (click, moan, whistle, engine noise and empty water). Once the sound is played through the participant headphones, participant classifies the sound by pressing the CLASSIFY button. Participants are further guided to select one of the given five options: click, moan, whistle, engine noise and empty water (Figure 2. h). While listening to the sounds, the participants can see the visual wavelength of the acoustic samples (Figure 2. b).

Gathering cetacean images -Participants can gather the images from below

and above the water surface. Participants select the type of camera they want to use and press the COLLECT button to obtain the photo (Figure 2. c). The underwater photo is captured from the action camera installed in the capsule component of the hydrophone, while the surface photo is from the actual mobile phone the participants are using. In the case of the underwater photo, participants need to wait approximately five seconds from when the image is shot to when it reaches the users' mobile phone. We opted for this waiting time in order to prolong the battery life and maintain the highest resolution photos images (Figure 3).

Evaluation

In this section, we describe the evaluation of the App's first working prototype, designed to enhance the whale watching experience. Because of the underlying context and the absence of previous work evaluating systems that combine enhancing user experience and engaging participants in conservation activities in the wild, we used an extensive evaluation framework which includes assessing the usability of the system (SUS [START_REF] Brooke | Sus: a "quick and dirty'usability[END_REF]) and the presence (PRE [START_REF] Witmer | Measuring presence in virtual environments: A presence questionnaire[END_REF]) of whale watchers.

The study was conducted in Madeira (North East Atlantic Ocean), a historically touristic island destination in the North East Atlantic which has a significant occurrence of cetaceans [START_REF] Dinis | Spatial and temporal distribution of bottlenose dolphins, tursiops truncatus, in the madeira archipelago, ne atlantic[END_REF][START_REF] Alves | Analysis of occurrence patterns and biological factors of cetaceans based on long-term and finescale data from platforms of opportunity: Madeira island as a case study[END_REF]. Participants were invited to use the App during a whale watching trip to report the sightings. The data collected through sightings included:

Our sample size (N=12) included 6 males and 6 females. Their age was ranging from 16 to 45 (M = 30.08, SD = 10.93), from 6 different countries of origin: Portugal (3), France (3), Germany (2), UK (2), Netherlands [START_REF] Alves | Analysis of occurrence patterns and biological factors of cetaceans based on long-term and finescale data from platforms of opportunity: Madeira island as a case study[END_REF] and Papua New Guinea [START_REF] Alves | Analysis of occurrence patterns and biological factors of cetaceans based on long-term and finescale data from platforms of opportunity: Madeira island as a case study[END_REF]. All of these participants expressed themselves as knowledgeable and experienced in using mobile smartphone applications. The study included mostly tourist visitors who came to the island on vacation and opted to go for the whale watching activity using sea-vessels. No incentives for recruiting participants were used. The study consisted of two separate tasks: (i) reporting a cetacean; (ii) answering the questionnaires. Both tasks were completed using the mobile application, which included electronic forms for the System usability scale (SUS) and Presence (PRE) questionnaires and also several study-specific questions.

Data Collection

To evaluate the App, we opted to collect participants' data according to two scales and several study-specific questions (listed below), resulting in 40 questions (10 SUS + 24 PRE + 6 study-specific). At the end of the experience, these questions were posed to users, using the mobile application to display seven points Likert-based scales (Figure 1. h). The averages of the participants' feedback are depicted below (Figures 4 and5), while here, we briefly describe the used scales.

1. System Usability Scale [SUS] (7-point Likert scale) -consisting of a 10 item standard SUS questionnaire, which we converted from 5 to 7 point Likerttype scale to keep the consistency with other forthcoming scales. This scale was used to gauge the usability of the application and its interface. 2. Presence Questionnaire [PRE] (7-point Likert scale) -tailored for virtual reality environments, the scale is equally valid if used in real environments, where all subjects experience the same type of environment [START_REF] Usoh | Using presence questionnaires in reality[END_REF]. We used the PRE scale to understand the feeling of presence of users, while engaged with the App. Thus, to assess if they still felt presence in the natural setting of the experience.

Data Analysis

The results of the three questionnaires are summarized in Figure 4. Overall the results from the 12 users are positive in terms of usability and presence. We calculated the overall SUS score according to [START_REF] Lewis | The factor structure of the system usability scale[END_REF] by determining each item's score contribution, from the range of 0-7 as follows: i) for positively-worded items (1, 3, 5, 7 and 9), the score contribution is the answered value minus 1; ii) for negatively-worded items (2, 4, 6, 8 and 10), it is 7 minus the answered value; iii) for missing values assign 5 (the center of the rating scale). To get the overall SUS score, we then multiply the sum of the item score contributions by 1,67 (100/60), thus adjusting the overall SUS score to 0-100. Calculating the overall score, we reached 60.83, which according to [START_REF] Bangor | Determining what individual sus scores mean: Adding an adjective rating scale[END_REF] fits on a grade C ("good") score.

For the Presence questionnaire, we obtained the overall score by averaging the results from the 24 questions, 7-point Likert scale. The final result (M = 5.45, SD = 0.40) is above average. We also performed a Cronbach's alpha to estimate the reliability, determine correlations between the questionnaires' items, and assess each questionnaire's internal consistency. The overall result was good at a = 0.827 (N = 12). We investigated further the results in terms of the known SUS [START_REF] Lewis | Revisiting the factor structure of the system usability scale[END_REF] and PRE [START_REF] Witmer | The factor structure of the presence questionnaire[END_REF] factors. This structure is reported for SUS as usability (all questions except 4 and 10) and learnability (questions 4 and 10) [START_REF] Lewis | Revisiting the factor structure of the system usability scale[END_REF]. For PRE the structure of factors includes realism (questions 3, 4, 5, 6, 7, 10 and 13), possibility to act (questions 1, 2, 8 and 9, quality of the interface (questions 14, 17 and 18), possibility to examine (questions 11, 12 and 19), self-evaluation of performance (questions 15 and 16), sounds (questions 20, 21 and 22) and haptic (questions 23 and 24). We included the study-specific questions in these factors as follows: realism (questions 1), sounds (question 2), possibility to act (questions 3, 4 and 5) and learnability (question 6). The results are summarized in Figure 5 Finally, the study-specific questions showed that the App was not intruding in the experience (where you looking at the whales or the App?) with mean 5. 58 (s=1.24). Users also reported an above average perception of the importance of the sound provided by the App via the hydrophone with mean 5.42 (SD=1.62). Users also reported liking to report the sightings (M=5.83, SD=1.03) and classifying the sounds (M=5.82, SD=1.70) both with higher mean values than gathering cetacean photos (M=5.58, S=1.16). Finally, users were almost balanced in terms of their experience with citizen science (M=5.50, SD=1.88).

Conclusions

Research efforts examining the opportunities to use entertainment technologies to promote environmental protection and ecological consciousness from users is a growing area of interest in research. In this paper, we expand previous work in trying to augment the experience of whale watching using a combination of sensors with a mobile application. Our App allows whale watchers to actively participate in reporting the sightings, sound classification, and media gathering while staying at a safe distance from the animals, so as not to induce stress in the cetacean. Here we report our first attempt to test such a system by combining known usability and presence scales with some study-specific questions to understand how fulfilling this new entertainment experience is. We asked 12 participants to conduct several activities, including (i) report a sighting, (ii) classify a cetacean vocal call, and (iii) gather a cetacean image.

We summarize our results against the know factors of the SUS and PRE scales. We observe that factors such as "quality of the interface", "possibility to act" and "realism" have higher mean values. While factors such as "haptics" and "learnability" have lower mean values. Although our study included the minimum number of participants required for validity of these scales [START_REF] Claudet | A roadmap for using the un decade of ocean science for sustainable development in support of science, policy, and action[END_REF], and our results are valid in terms of reliability and internal consistency, we would like to expand the sample to understand further how to evaluate these complex experiences. Future work includes comparing the results of the overall experience with and without the whale watching App and integrating environmental awareness scales into the evaluation framework. Another potential avenue for research emerging from this work includes designing applications that promote offshore whale watching experiences, either using holographic displays or combining sound and video from previous experiences. Ultimately our goals are to expand the state of the art in novel entertainment experiences that lie within this space of environmental conscious and sustainability in what we framed as eco-centric design [START_REF] Nisi | Towards eco-centric interaction: Urban playful interventions in the anthropocene[END_REF].

  (a) Microcomputer with the power bank, Wi-Fi dongle for communication with GoPro, audio jack and stereo cable, and a hydrophone. (b) Assembled buoyant capsule including all components. Tripod is used for exhibition.

Fig. 1 :

 1 Fig. 1: POSEIDON Hardware System components

Fig. 2 :

 2 Fig. 2: Screenshots from the POSEIDON application. From left to right: (a) Sightings main screen, portraying the reported sightings; (b) Acoustics main screen, portraying the live acoustic feed from hydrophone; (c) Photos main screen, depicting collected underwater and surface photos; (d) Survey at the end of the experience; (e) Sighting report screen; (f) Sighting Reporting screen information collection; (g) Sighting reported at the GPS location. (h) Classification of acoustics with given options (click, moan, whistle, engine or nothing).

( a )

 a Surface image of the pilot-whale sunbathing and taking a rest. (b) Underwater images of the two breastfeeding mothers with one calf.

3 .

 3 Study-Specific Questions [STD] (7-point Likert scale) -several questions targeting the usage of the App: i) Where you looking more to the actual whales and dolphins than the app?; ii) How important to you was the sound during the whale watching activity?; iii) How much you liked reporting the sightings?; iv) How much you liked classifying the cetacean sounds?; v) How much you liked gathering the cetacean photos?; and vi) Please rate your previous experience as citizen scientist.

Fig. 4 :

 4 Fig. 4: Boxplot depicting the results of the three questionnaires (N=12). Red the PRE scale, green the study-specific questions and blue the SUS scale.

Fig. 5 :

 5 Fig. 5: Boxplot depicting the results of the three questionnaires (N=12) consolidated in the known SUS and PRE factors.

  and show that higher average scores are on factors: realism (M = 5.65, SD = 1.19), possibility to act (M = 5.60, SD = 1.35) and quality of the interface (M = 5.75, SD = 1.27). Conversely lower scores are on haptic (M = 4.83, SD = 1.52) and learnability (M = 4.69, SD = 1.92).

https://sustainabledevelopment.un.org

https://sustainabledevelopment.un.org/frameworks/ouroceanourfuture

https://www.bats.org.uk

https://www.soundscapes2landscapes.org

https://www.wesa.fm/post/pitt-researchers-are-eavesdropping-birds-name-science

https://www.cetalingua.com/citizen-science/

https://www.nocmig.com/audiomoth

https://www.geekwire.com/2015/game-about-killing-poachers-vies-for-50000-prizein-microsoft-student-tech-competition/

[START_REF] Claudet | A roadmap for using the un decade of ocean science for sustainable development in support of science, policy, and action[END_REF] https://appadvice.com/app/camera-birds/

520656685 13 https://gamefaqs.gamespot.com/android/701348-wounded-

iwhale-rescue 14 https://www.amazon.com/Deep-Water-Hero-HD-

Free/dp/B007TXSCV0 15 https://

wilderquest.nsw.gov.au/ 16 https://www.thewildnetwork.com/wild-time-ideas

Acknowledgements Authors recognize several grants which supported the study: (i) LARGESCALE, #32474 by FCT and PIDDAC; (ii) LARSyS, #UID/ EEA/50009/2019 by FCT; (iii) LARSyS -FCT Plurianual funding 2020-2023, by FCT; (iv) INTERTAGUA, #MAC2/1.1.a /385 by MAC INTERREG 2014-2020; and (v) MITIExcell, #M1420-01-01450FEDER0000002 by RAM.


Darui Cheng

Honglei Han
email: hanhonglei@cuc.edu.cn

Guangzheng Fei
email: gzfei@cuc.edu.cn

Automatic Generation of Game Levels Based on Controllable Wave Function Collapse Algorithm

Keywords: Procedural Content Generation, Wave Function Collapse, Level Generation, Game Design

Procedural content generation automatically creates game content through methods such as pseudo-random numbers, which helps save labor or create games that can be played repeatedly and indefinitely. The wave function collapse algorithm is an effective procedural content generation algorithm newly proposed in recent years, but it has the problems of complicated rule writing and lack of non-local constraints. In this paper, based on the original wave function collapse algorithm, an automatic rule system is proposed, which can simplify the rule writing. Moreover, we use the three mechanisms, namely global constraint, multi-layer generation, and distance constraint, to establish non-local constraints. Through experiments, compared with the original wave function collapse algorithm, the results show that manual control has been enhanced, and the generated levels have a certain degree of similarity with human-designed levels. By making a real-time dynamic level game demo using this method, it turns out that the controllable wave function collapse algorithm we proposed has great potential in the game level generation field.

Introduction

Procedural Content Generation (PCG) is a significant research field in game development. To automatically create game content, pseudo-random numbers and other methods have been adopted in PCG fields. The application of PCG in the field of game development helps to save manpower, assist game designers to exert their creativity, save storage space, and create games with infinite replayability [START_REF] Shaker | Procedural Content Generation in Games[END_REF]. PCG algorithm can be divided into the following four categories: constructive methods, searchbased methods, constraint-based methods, and machine-learning methods [START_REF] Summerville | Procedural Content Generation via Machine Learning (PCGML)[END_REF]. Among them, the Wave Function Collapse (WFC) algorithm is a newly proposed PCG algorithm based on constraint solving, which has great potential value in the game development field. However, there are four problems related to the WFC algorithm: complicated rules configuration, lack of global control, difficulty to achieve distance constraints, and impossible to generate multi-layer levels. Therefore, we propose a method that adds more constraints to the original WFC algorithm to solves these problems. First, for the complex adjacency rules which need to be manually configured in the original WFC algorithm, we proposed an automatic rule system to reduce the number of rules that need to be written. Second, we added global control in the original WFC algorithm through the global maximum constraints and tiles preset, which provides more capabilities for game designers to control generated results. Then, the distance constraints were integrated into the observation stage of the WFC algorithm. At last, multi-layer levels generation was supported so that game elements with different semantics, such as NPC and interactive items, can be placed on different layers, and there are connections and constraints between each layer.

2

Related Work

Procedural Content Generation

Previous studies have recognized the importance of PCG in the creation of video games in different aspects. The current use of PCG technology in game development is mainly limited to specific types of game elements. Because the design ideas for different types of game levels are not the same, it is challenging to generate the entire game level procedurally. Lawrence Johnson et al. used the idea of the constructive PCG algorithm to create the infinite cave level. The algorithm based on cellular automata is evaluated in the infinite cave game to generate playable and well-designed cave game levels [START_REF] Johnson | Cellular automata for real-time generation of infinite cave levels[END_REF]. Miguel Frade et al. used a genetic algorithm to evolve the terrain to obtain the required game levels with sufficient accessibility [START_REF] Frade | Evolution of Artificial Terrains for Video Games Based on Accessibility[END_REF], which is a searchbased algorithm. The PCG via machine learning (PCGML) has also become a research hotspot, and a lot of research has been carried out in this direction. These researches [START_REF] Dahlskog | Linear levels through n-grams[END_REF][START_REF] Summerville | MCMCTS PCG 4 SMB: Monte Carlo Tree Search to Guide Platformer Level Generation[END_REF][START_REF] Summerville | Super Mario as a String: Platformer Level Generation Via LSTMs[END_REF][START_REF] Hoover | Composing Video Game Levels with Music Metaphors through Functional Scaffolding[END_REF][START_REF] Snodgrass | Learning to Generate Video Game Maps Using Markov Models[END_REF][START_REF] Jain | Autoencoders for Level Generation, Repair, and Recognition[END_REF][START_REF] Guzdial | Learning to Blend Computer Game Levels[END_REF] each used different machine learning methods to train on the VGLC [START_REF] Summerville | The VGLC: The Video Game Level Corpus[END_REF], a small dataset of Super Mario Bros levels, attempting to generate new levels. Nonetheless, none of them meet the requirements of sufficient playability and controllability. Generative Adversarial Networks (GAN) is a machine learning method suitable for generation. Giacomello et al. [START_REF] Giacomello | Searching the Latent Space of a Generative Adversarial Network to Generate DOOM Levels[END_REF] input a manually extracted level feature vector to the level generator by conditional GAN, trying to generate a level that is similar to human-designed in visual and structural aspects. Volz et al. [START_REF] Volz | Evolving Mario Levels in the Latent Space of a Deep Convolutional Generative Adversarial Network[END_REF] combined GAN with the evolutionary search. A two-stage experiment tried to use the generated results after screening for repeating train, in order to achieve the directional generation. However, these two studies did not reach the result of the utterly controllable generation of levels.

The mentioned constructed methods, search-based methods, machine learning methods have various drawbacks and are only suitable for specific generation goals. Constructive algorithms use fixed algorithms to generate game content at once without testing and may generate results that are beyond human control as well as unusable [START_REF] Shaker | Procedural Content Generation in Games[END_REF]. A search-based algorithm specifies an evaluation function and uses the idea of the evolutionary search to obtain a better result through continuous generation and evaluation, whose efficiency is relatively low [START_REF] Togelius | Search-Based Procedural Content Generation: A Taxonomy and Survey[END_REF]. All machine learning methods have a common defect that they all need a large dataset for training, but this is not practical in the field of games. It is difficult to collect a large game level construction dataset manually. Also, different game types require completely different game levels dataset. Each time developing a new game, one needs to reconstruct the dataset. Currently, all PCG methods based on machine learning can only be trained with a very small data set, so that the results obtained generally lack playability, and the failure rate is relatively high. Compared with the above three types of algorithms, the constraint solving algorithm has certain advantages in the generation of game levels, which is specifically reflected in the fact that this type of algorithm can randomly generate levels that fully meet the artificial constraints, and the controllability is higher.

Constraint Solving Algorithm

Constraint solving algorithm is neither a constructive PCG algorithm nor a strictly search-based PCG method. As an algorithm in the field of traditional artificial intelligence, constraint solving uses ideas from knowledge representation and search to model continuous and combinatorial search and optimization problems and solve them with domain-independent algorithms [START_REF] Russell | Artificial intelligence: a modern approach[END_REF]. The problem of game level generation under certain constraints is actually a constraint satisfaction problem (CSP), which is usually defined according to a series of decisive variables and values. The method of PCG based on constraint solving can usually guarantee the output precisely but at the cost of the unpredictability of the total running time [START_REF] Karth | WaveFunctionCollapse is constraint solving in the wild[END_REF]. Michael Cerny Green et al. took the lead in applying the constraint solving algorithm to the generation of dungeons in games. They divided the algorithm into two stages, namely Layout Creator and Game Element Furnisher, using the algorithm based on constraint solving at all of them [START_REF] Green | Two-step Constructive Approaches for Dungeon Generation[END_REF]. However, their method is only applicable to the generation of dungeon rooms or cave levels. It is not possible to control the reasonable adjacency of each tile in the map in more detail, so it is not suitable for outdoor terrain generation. Moreover, because the use of traditional constraint solving methods needs to iterate step by step, a scan has to be run in order to determine the appropriate position when adding each element. So, when there are too many game elements, it is easy to cause confusion and low efficiency.

Wave Function Collapse Algorithm

WFC algorithm divides game levels into a grid and defines all cells in the grid that have all possible values firstly, reference the concept of superposition state in quantum physics. During the operation, it determines the values in each cell one by one through "observing", and then propagates the effects to adjacent cells, and finally achieve the goal of determining a feasible solution that meets all constraints. Although this algorithm has a probability of failure caused by conflict, the efficiency is higher than the traditional search-based constraint solving method. Many researchers have applied the WFC algorithm in different studies. Isaac Karth et al. summarized a data-driven branch in WFC algorithms, namely the overlapping model. This model uses the bitmap file as input, directly reads the pixel-to-pixel adjacency relationship in the bitmap file as a constraint, and is used to guide the propaga-tion stage of the WFC algorithm [START_REF] Karth | WaveFunctionCollapse is constraint solving in the wild[END_REF]. The WFC algorithm of the overlapping model has remarkable advantages when used as a texture generator but is not suitable when generating game levels. Werner Gaisbauer et al. tried to generate a virtual city under a specific game theme by adjusting the parameters used in the basic WFC algorithm [START_REF] Gaisbauer | Procedural Generation of Video Game Cities for Specific Video Game Genres Using WaveFunctionCollapse (WFC)[END_REF]. They only tried to find the best parameters for the city generation in a specific game, but make no algorithm-level improvements. Hugo Scurti et al. applied the WFC algorithm to generate random paths for NPCs in games [START_REF] Scurti | Generating Paths with WFC[END_REF]. Hwanhee Kim et al. extended the WFC algorithm from the existing grid-based system to the graphbased system, breaking through the limitation of the grid in the original WFC algorithm, making the WFC algorithm more widely applicable [START_REF] Kim | Automatic Generation of Game Content using a Graph-based Wave Function Collapse Algorithm[END_REF]. Nevertheless, the above studies have ignored the possibility of integrating non-local constraints and multi-layer generation into the WFC algorithm. In this view, the motivation for our work is addressing this problem.

WFC, an algorithm based on adjacency constraints, lacks non-local constraints in nature. The method described herein combining the original WFC algorithm Maxim Gumin proposed [START_REF] Gumin | Bitmap & tilemap generation from a single example by collapsing a wave function[END_REF] and the dungeon room generator Green et al. presented [START_REF] Green | Two-step Constructive Approaches for Dungeon Generation[END_REF], using the idea of solving CSP, provides a robust control handle for game designers. It can solve problems mentioned above of the WFC algorithm, which is conducive to obtaining a level generation result that is more controllable and more similar to the artificial design.

Method

WFC algorithm can be divided into the overlapping model and the simple tiled model. The overlapping model is suitable for procedural texture generation [START_REF] Karth | WaveFunctionCollapse is constraint solving in the wild[END_REF]. The simple tiled model can manually configure more complex rules to restrict the adjacency between tiles with different semantics, which is more suitable for the game levels generation. Since the problem we need to solve is the generation of game levels, the simple tiled model is selected. The method in this paper is shown in Figure 1 and consists of the following four steps, which contains the key improvements that we introduced to the original WFC algorithm.

1. Rule initialization: In the original WFC algorithm, all allowed adjacency rules in the grid were explicitly written into a configuration file. According to the rule library specified in this file, each time the WFC algorithm runs, it comes up with a solution that meets all the adjacency rules, that is, a game level with certain playability. To reduce the number of rules that need to be manually written, we integrate an automated rule system during the rule initialization phase, which can expand one manual rule into multiple equivalent rules based on the symmetry of tiles. 2. Data initialization: Before each generation, we have to clear and initialize all data in the grid. Before the first observation stage, we insert a global minimum constraint stage to fix some preset tiles at specific positions to enhance the game designer's control of the result.

3. Observation: In terms of information entropy theory, we calculate the information entropy H of all cells with equation ( 1), where P i means the probability of each available tile. A cell with the lowest information entropy is selected as the observation cell. From all the remaining available tiles in the cell, we select one randomly according to the weight. After the value of a cell is determined, global maximum constraint, inter-layer constraint, and the distance constraint are added to solve a series of problems of the original WFC algorithm.

H = -∑ 𝑃 𝑖 𝑙𝑜𝑔𝑃 𝑖 𝑛 𝑖=1 (1)
4. Propagation: The constraint caused by the value determined in the observation stage is propagated to adjacent cells at the propagation stage. Then, the undetermined cell of the lowest information entropy is selected for further observation. This process is repeated until all cells converge to certain observing values.

Automatic Rule System

In order to use transformations of the tile to simplify the rule configuration, as shown in Figure 2, we have summarized 8 commonly used transformation forms of a tile, and each is marked with an index number, which is convenient for search. First, we temporarily ignore the symmetry of tiles and expand each rule. As shown in Figure 3, the No. 0 transformation of road tiles and No. 1 transformation of road turn tiles form an adjacency rule based on manual configuration. By rotation and mirroring, this rule can be expanded into 8 rules. Since the rules between the two tiles are mutual, the rule is expanded into 16 rules for storage finally.

Next, we consider the symmetry of the tiles. The method mentioned above can solve the problem of repeating different transformation forms of the same rule, but there are still too many rules to write. For rules with symmetrical tiles, the equivalent rules can be extended using symmetry to reduce the number of the required rules further. Therefore, on the basis of the original WFC algorithm [START_REF] Gumin | Bitmap & tilemap generation from a single example by collapsing a wave function[END_REF] and the WFC algorithm of Arunpreet Sandhu et al. [START_REF] Sandhu | Enhancing wave function collapse with design-level constraints[END_REF], the common symmetry types of tiles need to be extended. To build a more comprehensive system of symmetry types of tiles, as shown in Table 1, the symmetry types of tiles are summarized into nine types. The types are named by a letter with the same symmetry type.

8 transformation forms and 9 symmetry types will form a complicated mapping relationship. How to quickly find the transformation number of the equivalent tile after the symmetry transformation of the tile becomes the key point to the automatic rule system. For this reason, we have proposed the concept of the symmetry dictionary, which is established by recording the transformation number equivalent to the initial tile of each symmetry type. The symmetry dictionary is shown in Table 1.

The automatic rule system expands manually configured rules into a large number of equivalent rules, eliminating the game designer's workload of configuring these equivalent rules, which significantly simplifies the difficulty of writing the configuration file. For example, if all 4 sides of 8 transformation types of 2 tiles need to be configured to allow adjacency, 8×8×4×2 = 512 rules have to be manually written. With the automatic rule system, 1 to 8 rules are simply needed, then other rules are extended by this system. As a result, the efficiency of rules writing has been improved significantly. X/X/X/X/X/X/X/X

Non-local Constraints

Global Constraint. One of the main problems of the WFC algorithm is the lack of global constraint, which includes two aspects: global maximum constraint and global minimum constraint. Global maximum constraint refers to the limitation of the maximum number of a specific kind of tile, while the global minimum constraint is that a specific type of tile appears at least several times. The lack of global constraints will cause the global number of game elements generated in the level not to be controlled by the WFC algorithm. The effect of applying the global maximum constraint is shown in Figure 4. The constraint is not added in Figure 4(a), which results in the water area is too large. In Figure 4(b), the global maximum constraint of the water tiles is set to 20 units and get the result we expect.

The global minimum constraint can be equivalent to presetting the minimum number of tiles at specified positions or random positions at first. The method of presetting tiles is to set the value of the specified cell as the specified tile after the data initialization stage and then to spread its influence. Game designers can also use this function to presets the outstanding design that they need to set in advance, and the WFC algorithm automatically generates the remaining parts.

The effects of the global minimum constraint are shown in Figure 5. Due to the position (0, 0) is preset to water, as shown in the result of Figure 5, it can be seen that the WFC algorithm generates the rest of the level without violating the preset constraints. Multi-layer Generation. A game scene is often composed of multiple layers of game elements with different semantics and requires different types of game elements to be generated at different layers. Because game elements of different layers can overlap each other, they usually do not apply to adjacency constraint rules but require interlayer constraint. It is difficult to achieve with the original WFC algorithm. Figure 6(a) is the result of generating a level of a single layer, which generally can only be applied to simple types of game scenes. However, in Figure 6(b), the enemy and the treasure chest are added, the enemy can only be generated on the grass, and the treasure chest can only be generated on the road. It is a result of using the method of this study.

To achieve multi-layer generation, after each observation of the bottom layer is completed, the tiles which do not meet the inter-layer constraints in the upper cell are banned. In this way, through the definition of the inter-layer constraint rules, the connection between the two layers is established.

With the function of multi-layer generation, the richness and playability of the game levels have been improved. The automatically generated game levels are no longer just a flat one-layer game map. Distance Constraint. The distance between the game elements is closely related to the playability of the game, so it is a significant factor for game designers to build game scenes. However, the WFC algorithm is mainly built on the adjacency constraints and lacked the constraints of distance, which makes some game design relying on distance constraints cannot be achieved. As shown in Figure 7(a), the distance between the treasure chest and the enemy cannot be effectively constrained, which makes it impossible to achieve the design expectations that the enemy surrounds the treasure chest. In view of this, it is necessary to control the maximum and minimum distance between tiles and combine adjacent constraints to achieve the effect of comprehensive distance constraints.

In Figure 7(b), when game designers limit the distance between treasure chests and enemies less than 10 units and limit the distance between treasure chests and keys greater than 10 units, the generation results can meet the design goals of enemies distributed around the treasure chest and the keys kept away from the treasure chest.

Experiments and Analysis

All experiments in this study were completed on a computer with a CPU of Intel Core i7-8750H and a graphics card of Intel UHD Graphics 630.

Comparison with the Original WFC Algorithm

First, to verify the improvement of this method after adding more constraints on the original WFC algorithm, an experiment was conducted to compare the results of this method with the original WFC algorithm when generating a birds-eye-view 2D game level. According to a game designer's requirements, the automatically generated level needs to meet the following three constraint rules:

1. Global Constraint. In the case of a scene size of 20×20, the maximum number of water tiles is not more than 20 units. Besides, the bottom left corner (0,0) floor tile is water, and the center (10,10) tile is grass. 2. Distance constraint. The minimum distance between a treasure chest and a key is 10 units, and the maximum distance is 20 units. The maximum distance between a treasure chest and an enemy is 10 units. 3. Inter-layer constraint. Trees, rocks, and enemies must be generated on the grass, and treasure chests and keys must be generated on the road.

The final result is shown in Figure 8, the two sub-pictures in (a) use Maxim Gumin's WFC algorithm level generator [START_REF] Gumin | Bitmap & tilemap generation from a single example by collapsing a wave function[END_REF], and the two sub-pictures in (b) use the method of our study. Due to the original WFC algorithm cannot generate multiple layers, the results of the original algorithm do not generate the second layer of elements. In the original WFC algorithm, as shown in Figure 8(a), the global parameters are not controlled, only the visually reasonably map generated. In particular, with the large water area as well as the lack of decorations and interactive items, the map can barely present playability enough. Although a nice visual effect has been achieved, it is actually not suitable for the game generation. In Figure 8(b), the global maximum constraint was used in our study to limit the water area to less than 20 units. At the same time, the multi-layer generation method was used to generate double layers, so that more playable elements are generated above the map layer. The two preset tiles are marked with blue squares in Figure 8(b), and the number of water tiles does not exceed the maximum constraint of 20 units so that the game designer achieve the goal to generate a small pool in the lower-left corner. The distance between the treasure chest and the key is marked with an orange circle in Figure 8(b), which can be seen that meets the constraint of 10-20 units. The light-yellow rectangular frame in Figure 8(b) shows the aggregation effect of the distance constraint, which makes enemies always appear around the treasure chest. Finally, we can find trees, rocks, monsters always generated over grass, while treasure chests and keys are always generated on the road, which meets the strict restriction rules between layers.

Similarity to Human-designed Levels

To analyze the distinction between the game levels we generate automatically and the levels created by the game designers manually, we conducted a user survey. We invit-ed 5 game designers to design 5 levels manually and generated 5 levels by our method automatically under the same constraints, then mixed them and shuffled the order to form a questionnaire. Five game designers were invited to ensure that the survey results are not affected by the personal design style of the game designer. The questionnaire informed the respondents of the underlying meaning of the game elements and the design intention in these levels. Meanwhile, we informed them that there were only 5 human-designed levels. Then the respondents were required to choose 5 levels which they think are manually designed from 10 game levels. Finally, we got 151 valid feedbacks from students with some game experience, and the statistical result is shown in Table 2.

From Table 2, it can be seen that the game levels generated by the algorithm in this paper are similar to the game levels designed by humans. Ideally, 50% levels that were generated automatically should be considered as human-designed, so do the indeed human-designed levels, meaning that the respondents cannot distinguish the generated level at all. In the worst case, 0% levels that were generated automatically should be considered as human-designed, while the human-designed levels can be 100% identified, meaning that the respondents completely distinguish the generated levels. The survey results show that 57.62% of the respondents think that the No. 1 generated level is human-designed, which has exceeded 50%. The percentage of generated levels considered to be human-designed reaches an average of 40.13%, which is close to 50%. It shows that most participants were failed to differentiate the design made automatically or manually.

Real-time Dynamic Level Generation

Finally, we used the controllable WFC algorithm proposed in this paper to implement a real-time game demo that dynamically generates game levels at runtime. It is a 3D top-down roguelike shooter game with rich terrain with plateaus, brooks, and sinuous roads, whose goal is to defeat the enemies and collect treasures from chests. In this game, all levels are not generated before the game starts, but are generated during the player's game. In this way, it is possible to read the player's operation data in real-time during the game and modify the constraint configuration as a parameter, and then dynamically change the difficulty or style of the game level generated later depend on the different performance of players. For example, when the player defeats a lot of enemies, we can increase the difficulty of subsequent levels; when the player takes a large amount of damage from enemies, we will generate easier levels later on. The real-time level generation effect is shown in the attached video.

Conclusion and Future Work

This paper mainly solves two main problems of the WFC algorithm: the complex constraint rules are hard to write; excessively random game levels that lack control are always less playable. We proposed automatic rule system, global constraint, multi-layer generation, distance constraint to solve these problems effectively and provide a controllable game level generation tool for the game designers. Experiments show that the method is better than the original WFC algorithm in the aspects of enhancing the designer's control and generation effect. Moreover, the game levels automatically generated by this method achieve a high similarity with the human-designed game levels. In addition, Real-time dynamic level generation experiment shows that the method in this paper can be successfully applied to practical games generation and get satisfactory results. This method is only applicable to some game types whose game scenes are built on a plane and is hard to used directly into some other game types such as 2D sidescrolling games. In future work, we plan to extend this method to the field of 2D sidescrolling game levels generation. Furthermore, we will also do more research on the real-time generation of levels by this controllable WFC algorithm.

Fig. 1 .

 1 Fig. 1. Main program flow.

Fig. 2 .

 2 Fig. 2. 8 transformation forms of a tile. The first line is obtained by rotating the No. 0 pattern 90 degrees counterclockwise sequentially, and the second line is left-right mirror images of the first line.

Fig. 3 .

 3 Fig. 3. Get 8 equivalent rules from one left-right rule.

Fig. 4 .

 4 Fig. 4. Two generated levels without(a) and with(b) global maximum constraint.

Fig. 5 .

 5 Fig. 5. Global minimum constraint effects.

Fig. 6 .

 6 Fig. 6. Two generated levels, with single-layer generation(a) and double-layer generation(b).

Fig. 7 .

 7 Fig. 7. Two generated levels, without(a) and with(b) distance constraints.

Fig. 8 .

 8 Fig. 8. Comparison between method in study[22](a) and our method(b).

Table 1 .

 1 Symmetry dictionary.

	Name Symmetry type	Initial tile's equivalent	New symmetry type
			transformation number	after transformation
	F	No symmetry	0	F/F/F/F/F/F/F/F
	S	Centrosymmetric	0/2	S/S/S/S/S/S/S/S
	T	Vertical axis symmetry	0/4	T/B/T/B/T/B/T/B
	L	Counter-diagonal axis symmetry	0/5	L/Q/L/Q/Q/L/Q/L
	B	Horizontal axis symmetry	0/6	B/T/B/T/B/T/B/T
	Q	Main diagonal axis symmetry	0/7	Q/L/Q/L/L/Q/L/Q
	I	Horizontal and vertical axis symmetry 0/2/4/6	I/I/I/I/I/I/I/I
	/	Double diagonal axis symmetry	0/2/5/7	'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'/'
	X	All 8 transforms are identical	0/1/2/3/4/5/6/7

Table 2 .

 2 The user survey results.

	Level number	Manual 1 Manual 2 Manual 3 Manual 4 Manual 5 Average
	Percentage considered	61.59%	32.45%	70.20%	66.23%	68.87%	59.87%
	to be human-designed
	Level number	Auto 1	Auto 2	Auto 3	Auto 4	Auto 5	Average
	Percentage considered	57.62%	40.40%	37.75%	33.11%	31.79%	40.13%
	to be human-designed

Acknowledgments. This work was supported by the Fundamental Research Funds for the Central Universities, and the National Key R&D Program of China (2018YFB1403900).


Haoran Xie
email: xie@jaist.ac.jp

Dazhao Xie

Kazunori Miyata

Body2Particles: Designing Particle Systems Using Body Gestures

Keywords: User interface, Particle system, Firework animation, Body tracking

Body2Particles is an interactive design system that allows users to create complex particle systems using body gestures. Even for skilled animators, adjusting the various parameters for the animation of complex physics systems is a boring task. Common users may feel frustrated in attempting to understand these parameters and their proper usages. To solve these issues, Body2Particles provides an embodied design interface for the animation of complex particle systems. This work focuses especially on the animation of fireworks as it requires the hierarchical construction of multiple particle systems. The proposed user interface takes the user's body movements as input from depth sensor and outputs firework animation. In our preliminary study, we identified the relationships among the animation parameters and skeleton joints. The system proposed in this study adapts these relationships using predefined parameter constraints and estimates the simulation parameters based on the captured skeleton from a depth sensor. User studies are also conducted to verify the effectiveness and potential uses of this system such as exercise promotion.

Introduction

Particle systems are used to simulate complex phenomena in virtual worlds, such as fluid, smoke, and fireworks, which are common in computer animations and video games [START_REF] Reeves | Particle systemsa technique for modeling a class of fuzzy objects[END_REF]. Although various sophisticated Euler and Lagrangian fluid simulation approaches are used in computer graphics nowadays, it is convenient for common users to use particle systems due to the low computation cost and the plugin embedded in popular game engines, such as Unity3D. It is not easy for common users to achieve the desired animation results by adjusting various control parameters. Even for professional users, it is boring task to control the parameters with the traditional graphical user interfaces (GUI). If a user wanted to create fireworks animation, they would have to study the flow dynamics of actual fireworks and master the appropriate simulation software, which are not interesting tasks for common users.

We have also observed that lack of exercise is a serious social issue because we are so busy with our daily studies and work that we have few motivation to exercise, leading to a variety of health problems. A embodied design interface for content creation can facilitate bodily self-expression in contrast to the common design interface with keyboards and mouse as system input. To make the user exercise while designing the interesting items, we aim to present a gesture-based embodied interface for designing a complex particle system. Through such an interface, the common users can design particle systems using body gestures that exercise their bodies.

To achieve these goals, we propose Body2Particles, an interactive system that allows any user to easily design complex particle systems using full-body gestures as system input. Body2Particles provides a similar experience as playing games with Microsoft Kinect in the design processes for particle systems. In this work, we especially focus on firework animations which are common and fascinating phenomena which can be represented by particle systems in our daily lives. Different from the conventional design system used for firework animations [START_REF] Cui | Sketch-based shape-constrained fireworks simulation in head-mounted virtual reality[END_REF], the proposed system tracks human gestures as input.

Body2Particles enables the procedural design of particle systems without requiring professional knowledge. As shown in Figure 1, we conduct the parameter mapping from body posture onto the control parameters of a particle system, so users are not required to understand simulation principles or master any simulation skills. The main contribution of this work is to provide an intuitive and playful user experience in animation design using body gestures. We believe that the proposed system can be applied to general procedural modeling techniques in computer graphics, especially for physical animation designs, such as cloth and smoke animations.

Related Work

In this section, we review related work on firework animation, design interfaces. We also discuss the training systems to support daily exercise.

Firework Animation Particle systems are usually used to model fuzzy objects, such as clouds, water, and fireworks [START_REF] Reeves | Particle systemsa technique for modeling a class of fuzzy objects[END_REF]. It is difficult to use particle systems to design the fuzzy objects under animation constraints, such as into particular shapes. A GPU-based firework simulation pipeline has been proposed to animate shape-constrained fireworks [START_REF] Zhao | Fireworks controller[END_REF]. Recently, a similar approach has been proposed for a virtual-reality environment with a head-mounted display [START_REF] Cui | Sketch-based shape-constrained fireworks simulation in head-mounted virtual reality[END_REF]. The authors provided a sketch-based design system for shape-constrained firework animations. All these approaches were proposed for computers with traditional GUI. However, our work aims to provide an embodied design interface for fireworks animation.

Design Interface It is challenging to create a design interface for novice users, especially for the tasks required high-level skills, such as aerodynamics [START_REF] Xie | Precomputed panel solver for aerodynamics simulation[END_REF] and fluid dynamics [START_REF] Zhu | Sketch-based dynamic illustration of fluid systems[END_REF]. The common solutions fall into two categories: sketch-based design and gesture-based design. An interactive sketch system usually adopts a data-driven approach to provide real-time feedbacks to users. Sketch2Photo was proposed to composite realistic pictures with freehand sketches [START_REF] Chen | Sketch2photo: Internet image montage[END_REF]. Sketch2VF can help novice users design a 2D flow field with conditional generative adversarial networks [START_REF] Hu | Sketch2vf: Sketch-based flow design with conditional generative adversarial network[END_REF]. Sketch2Domino provided the spatial sketch design of block management and guiding for real world tasks [START_REF] Peng | Sketch2domino: Interactive chain reaction design and guidance[END_REF]. As a gesture-based design system, BodyAvatar enables the design of freeform 3D avatars using human gestures [START_REF] Zhang | Bodyavatar: Creating freeform 3d avatars using first-person body gestures[END_REF]. This embodied interaction can also be used in 3D furniture design using body posing and acting as system input [START_REF] Lee | Posing and acting as input for personalizing furniture[END_REF]. Unlike these gesture-based design systems, we provide an embodied design interface for dynamical animation rather than 3D modeling.

Training System A training system usually used depth sensors, such as Microsoft Kinect, to support sports training [START_REF] Alabbasi | Human motion tracking evaluation using kinect v2 sensor[END_REF]. A more complex motion capture system has been used to guide tai chi exercise in a virtual environment [START_REF] Iwaanaguchi | Cyber tai chi -cg-based video materials for tai chi chuan self-study[END_REF]. A similar 3D virtual coaching system has been proposed to improve golf swing technique [START_REF] Kelly | A virtual coaching environment for improving golf swing technique[END_REF]. In addition, a VR training system has been proposed to learn the basketball free throw gestures with motion capture devices [START_REF] Qiao | Real-time human gesture grading based on openpose[END_REF]. Due to the rapid development of deep learning based approaches, the human skeleton [START_REF] Cannavò | A movement analysis system based on immersive virtual reality and wearable technology for sport training[END_REF] and 3D meshes [START_REF] Xie | Visual feedback for core training with 3d human shape and pose[END_REF] can be constructed interactively to guide users in sport training. In contrast to previous training systems, our work aims to design particle systems and provide a game experience that promotes user exercise.

System Configuration

The proposed system framework of Body2Particles is illustrated in Figure 2. To explore the relationship between body gestures and the control parameters of particle systems, we conducted a preliminary study to collect motion data, in which the users were asked to perform gestures freely with the recorded fireworks animations of different control parameters. In our embodied design interface for firework animation, we explicitly designed body gestures to correspond with different launching angles and heights and different blooming sizes and shapes. The final firework animations are created based on the users' gestures in real time.

Body Tracking

In this study, we used the RGB-D depth camera for body tracking (Kinect V2 with Kinect SDK 2.0) and obtained the coordinates, velocities and angles of the captured skeletons' joints. As shown in Figure 3(a), we used 21 items of joint data from the 25 joints captured by the Kinect configuration. Note that the data on ankles and hand tips were ignored because they were not clearly visible in this study. Then, we defined the distances between two hands D h , elbows D e , shoulders D s , knees D k , and feet D f for the particle system control. To control blooming shapes, a shape made by two arms is estimated based on the angles between the forearms and upper arms, where θ l and θ r denote the left and right arm joint angles as shown in Figure 3(b).

Firework Animation

A particle system has been proposed to represent fuzzy objects, such as fireworks with a collection of many small particles [START_REF] Reeves | Particle systemsa technique for modeling a class of fuzzy objects[END_REF]. Each particle system specifies various parameters to control the dynamical behaviors of fireworks as shown in Figure 4(a). A particle system normally goes through three stages in a given lifetime: the creation of particles, a change in particle attributes, and disappearance. After each particle in the particle system is rendered, fireworks animations can be generated. In this study, we generate a fireworks animation by dividing the whole life cycle of firework generation into two phases, from setting off the fireworks to their extinguishing. As shown in Figure 4(b), the firework to be launched is the parent particle, and the firework after blooming is a sub-emitter of particle systems. There are different control parameters defined in two phases as follows:

-Launching phase: launching height and angle, and number of fireworks; -Blooming phase: blooming sizes and shapes, number of particles after blooming are specified as sub-emitter particles from blooming to extinction.

Note that we developed both single and multiple fireworks modes in our prototype development in terms of the number of fireworks.

User Interface Design

Body2Particles provides an embodied design interface for particle systems, which takes users' body gestures as input. In this section, we discuss the preliminary study conducted to map body gesture parameters and control parameters, and to define the gesture controls used in our interface to increase the system immersion and user experiences.

Preliminary Study

We conducted a preliminary study to explore how to express fireworks by observing the animations of fireworks from the common users. All the fireworks animations were generated by modifying the control parameters manually using Unity3D. Totally, we collected 18 basic styles of fireworks animations as shown in Figure 5. To cover the feature spaces in designing fireworks, we adopted the following main parameters of particle system. We asked five participants to join our preliminary study (three males and two females). The experiment setup is shown in Figure 2. The participants stood in front of Kinect depth sensor, and a display screen which showed video clips of generated fireworks animations. The participants were asked to express the fireworks using their body gestures. The distance between participants and the screen was 3.0 m.

Parameter Mapping

Figure 6 shows the motion trajectories tracked during our preliminary study, and each figure corresponds to the video clip of firework animation. We found that all participants preferred to use hand movements to express the firework dynamics in both the launching and blooming phases. For different launching angles of fireworks as shown in Figures 5 and6 (a ∼ c), the participants were most likely to tilt their bodies to the left or right. In interviews, participants said that it was difficult to express the differences among blooming numbers of firework animations. In addition, there was no obvious difference in the data on male and female participants' movements. With reference to the results of the preliminary study, we explicitly define the parameter mapping between body gestures and control parameters as follows:

Launching Angles are calculated from the tilting angle of the user's body θ B .

θ B = arccos( y neck -y sb (x neck -x sb ) 2 + (y neck -y sb ) 2 + (z neck -z sb ) 2 ) (1)
where (x neck , y neck , z neck ) is three-dimensional coordinates of the body's neck joint, and (x sb , y sb , z sb ) is the coordinates of spine base joint. Launching Heights are defined by the relative positions of the user's head, spine shoulder and spine middle joints as shown in Figure 3. Because it is difficult to distinguish between launching heights with small variations, we defined three levels: high y head ∈ (y spine-shoulder , ∞), medium y head ∈ (y spine-mid , y spine-shoulder ), and low y head ∈ (-∞, y spine-mid ) heights, where y head , y spine-shoulder , and y spine-mid denote the height (y-coordinate) of the user's body, spine shoulder, and spine middle joints.

Launching Numbers of parent particles in particle system are decided based on the acceleration values of the hand joints. The larger the acceleration of the user's hands, the greater the number of launching fireworks.

Blooming Sizes are calculated as the maximum values among the distances between the two hands, elbows, shoulders, knees and feet, as max(D h , D e , D s , D k , D f ). Then, we adjust the life cycle of sub-emitter particles to modify the blooming sizes. During the preliminary study, the largest fireworks were generated by the maximum distance between two hands. We argue that these size constraints are related to exercise promotion while the users intended to achieve larger fireworks.

Blooming Shapes are set to be hemispheres, rings or weeping-willow shapes in our prototype design, which are the typical and common shapes of fireworks.

The blooming shapes are determined by hand distance D h and the arm angles θ l and θ r . If D h > δ (δ is a threshold value of shape criterion, δ = 40 cm in this study) and θ l,r > 115 • , the blooming shape is set to be hemispherical. If D h < δ and 60 • < θ l,r ≤ 115 • , the blooming shape is set to be a closed shape (ring or weeping-willow shape). We decide the specific shape in terms of the relative position of hands to body core. Therefore, it is ring shape if y hand ∈ (y spine-mid , ∞), and weeping-willow shape if y hand ∈ (y spine-base , y spine-mid ). y hand and y spine-base denote the height (y-coordinate) of user hands and spine base joints. Figure 7 shows the firework animations generated by different body gestures with our proposed parameter mapping approaches. Please refer to the accompanying video for the details of gesture controls.

Gesture Control

In addition to the parameter mapping between body gesture and control parameters, Body2Particles provides the global setting for firework animations, including initial launching conditions, particle colors, and single or multiple fireworks modes. Firework animations can be designed with different colors and numbers using our proposed embodied design interface as shown in Figure 8.

Launching Conditions

In order to increase the recognition rate of body gestures, the design system launches fireworks under the initial conditions. Based on our preliminary study, we set the initial condition as the hand heights are 20 cm above spine middle joints, and the velocity of the user's hands should exceed 1.8 m/s.

Color Selection

The user can select the fireworks colors using the right hand in our design system as shown in Figure 8(a1,a2). The height of right hand positions y hr is used to determine the colors from bottom to up. The fireworks color can be set to orange if y hr ∈ (y head , ∞), yellow if y hr ∈ (y spine-shoulder , y head ), purple if y hr ∈ (y spine-mid , y spine-shoulder ), blue if y hr ∈ (y spine-base , y spine-mid ), dark blue if y hr ∈ (y knee-right , y spine-base ), and green if y hr ∈ (0, y knee-right ).

Mode Selection

The user can select the single or multiple fireworks modes in our design system using the knees as shown in Figure 8(b1, b2). To diversify the gestures and promote exercise, we set the single fireworks mode if y f oot-lef t > 20 and (y knee-kef t -y hip-lef t ) > 10 cm with the user's left leg, and multiple firework mode if y f oot-right > 20 and (y knee-right -y hip-right ) > 10 cm with the user's right leg.

User Study

In our user study, the prototype system of Body2Particles was developed using Unity3D in Windows, Kinect V2 for depth sensor, and HUAWEI Honor band 5 for calorie consumption measurement. The experiment setup and participants are shown in Figure 9. In order to verify the user experiences using the proposed system, we conducted a subjective evaluation through a questionnaire and an objective evaluation by measuring activity intensity. There were 12 graduate students joined this user study (6 males and 6 females). All participants were asked to experience the proposed system for a total of 20 minutes, 10 minutes in both single and multiple fireworks. During the user study, the participants wore the smartwatch device to record calorie expenditure, which was used to calculate the intensity of the 20-minute activity (metabolic rates).

Results

In this section, we discuss the subjective and objective evaluation results from our user study. Subjective Evaluation We have confirmed the effectiveness of three aspects of proposed system through the subjective evaluation: system usage, system effects, and user experience as shown in Figure 10. We adopted 5-Likert scale for all questions (5 for strongly agree, 1 for strongly disagree). We have received quite positive feedback on all evaluation aspects including mean values of 4.4 for system usage, 4.5 for system effects, 4.1 for user experiences. For system usages, it is verified that our proposed system is easy to operate and provide adequate interactive feedback to users. For system effects, the proposed system can encourage the user to exercise, and the participants felt relaxed after using the system and wanted to use it again. For user experience, the participants felt satisfied and interested while using both single and multiple fireworks modes.

In the interviews with participants after they used the proposed system, the following comments were most common: "More types of fireworks shapes", "Allow two users to battle against each other", "Generate fireworks with footprints while walking", and "Design cute shapes for female users". The participants also pointed out that the system sometimes recognized gestures incorrectly. There was a delay between body gestures and fireworks generation because the system had to generate new particle systems after the completion of previous ones.

Objective Evaluation Because our research aimed to combine particle system design with excercise promotion in gesture-based control, we tried to evaluate the effect of using the proposed system on exercise. Figure 11 shows the correlation between the participant weight and calories consumption. In this study, we measured activity intensity using metabolic rate [START_REF] Ainsworth | Compendium of physical activities: an update of activity codes and met intensities[END_REF] as follows.

M ET s = calorie(kcal) weight(kg) × time(h) (2)
In our user study, both male and female participants achieved more than 6.0 METs as shown in right subfigure of Figure 11. This shows that the proposed system had the effects of medium to high intensity physical activity.

Conclusion

In this study, we proposed an interactive system Body2Particles, which designs particle systems using body gestures. In order to explore the relation between the fireworks animations and users' body movements, we conducted a preliminary study to investigate the parameter mapping from body gestures to control parameters of particle systems. Then, we analyzed the body movements of the participants and clarified the correspondence between the body gestures and the control parameters, including launching angles and heights, blooming sizes and shapes. In our user study, we verified that the proposed system is intuitive and enjoyable for common users. Furthermore, the proposed system had the clear effect of promoting exercise equivalent to other moderate-intensity physical activities.

The prototype implementation of the proposed system can be improved in multiple ways. Although current proposed system used the list of built-in skeletal parameters for designs of particle systems, the design flexibility would be improved by providing interactive authoring tool for visual control of simulation parameters. To provide a continuously stimulating environment to system user, we would improve the current system with few constraints on control gestures and more freedom to the user. The system could be made more interesting by increasing the types of fireworks and gamifying elements, such as scores and awards. The current prototype system can support only single user at a time, so we plan to make it possible for a large number of people to participate in designing multiple fireworks. In future work, we would like to explore the use of deep learning approaches to improve gesture recognition and automatic parameter mapping.

Fig. 1 .

 1 Fig. 1. Embodied design interface for firework animations using body gestures.

Fig. 2 .

 2 Fig. 2. System framework with parameter mapping in the preliminary study and the runtime embodied design interface with controlled parameters.

Fig. 3 .

 3 Fig. 3. Skeleton tracked by the depth camera (a) and the defined metrics (b).

Fig. 4 .

 4 Fig. 4. Control parameters of particle systems (a) and generation phases (b).

Fig. 5 .

 5 Fig. 5. Video clips of fireworks animations used in the preliminary study.

Fig. 6 .

 6 Fig. 6. Motion trajectories tracked in the preliminary study corresponding to the firework animations from (a) to (r).

Fig. 7 .

 7 Fig. 7. Body gestures for lanching angles (a), launching heights (b), blooming shapes (c-e), and launching numbers (f) of fireworks animations.

Fig. 8 .

 8 Fig. 8. Gesture controls for color selection (a1, a2) and transition between single and multiple fireworks modes (b1, b2) in the proposed design user interface.

Fig. 9 .

 9 Fig. 9. Experiment setup (a) and participants performances (b) in our user study.

Fig. 10 .

 10 Fig. 10. Results of the subjective evaluation questionnaire.

Fig. 11 .

 11 Fig. 11. Results of objective evaluation based on the calorie consumption (left) and metabolic rates (right) from our user study.

Acknowledgments

We greatly thank reviewers for their valuable comments, and all the participants in our user study. This work was supported by Hayao Nakayama Foundation Research Grant A, and JSPS KAKENHI grant JP20K19845, Japan.