Johanna Pirker
email: jpirker@iicm.edu

Simone Griesmayr
email: simone.griesmayr@student.tugraz.at

Anders Drachen
email: drachen@hum.aau.dk

Rafet Sifa
email: rafet.sifa@iais.fraunhofer.de

How Playstyles Evolve: Progression Analysis and Profiling in Just Cause 2

Keywords: Game Analytics, Progression, Playstyle, Player Behavior, Cluster Analysis, Visualization

Evaluating progression of players in a game can take a variety of forms, but ideally combines playstyle or performance analysis with one or more aspects of progression, e.g. through a level-or mission-based structure. Furthermore, visualization of the results of analysis are essential to ensure that action can be taken on them. In this paper behavioral profiling through Archetype Analysis is combined with progression analysis, expanding on previous work in the area, and extending it into the context of Open-World Games. The proposed methodological framework is applied to the case of the action-adventure title Just Cause 2, focusing on the main storyline. The results show how players navigate the content of the title, and how some playstyles remain constant throughout the game, whereas others emerge or disappear with player progress. Additionally, player performance as a function of progression is evaluated across a number of key metrics.

Introduction

Progression is a key element of game design, vital to balancing, and therefore an important focus of investigation in Game Analytics [START_REF] El-Nasr | Game Analytics -Maximizing the Value of Player Data[END_REF]. The concept of progression in game design can refer to a variety of components, however, adopting the definition of Lopez [START_REF] Lopez | Gameplay Design Fundamentals: Gameplay Progression[END_REF], generally refers to a pattern of advance towards a goal as well as the act of navigating towards the goal. This structure is important in ensuring the user experience and player engagement. Progression ties directly in with other elements of the game, including mechanics, rewards and risk, difficulty/complexity, etc.

From a behavioral analytics standpoint, evaluating progression can be performed in a variety of ways, but fundamentally combines performance analysis and measures of progression, e.g. through a level-or mission-based structure [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF][START_REF] Wallner | An introduction to gameplay data visualization[END_REF][START_REF] El-Nasr | Game Analytics -Maximizing the Value of Player Data[END_REF][START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF][START_REF] Sifa | Predicting Purchase Decisions in Mobile Free-to-Play Games[END_REF]. The performance component can be developed in a variety of ways depending on the involved number of behavioral features and the specific questions asked of the analysis. A commonly adopted approach is profiling, which seeks to condense high-dimensional behavioral datasets into user profiles that contain the underlying patterns of variance in the data. Towards this, a number of studies have investigated approaches such as description, segmentation as well as machine learning, for the latter notably cluster analysis [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF][START_REF] Normoyle | Bayesian Clustering of Player Styles for Multiplayer Games[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF][START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF][START_REF] Sifa | Predicting Purchase Decisions in Mobile Free-to-Play Games[END_REF].

Being able to analyze and visualize progression is notably important in large development teams, where different stakeholders (designers, artists, programmers etc.) need to collaborate on constructing the indented progression ramp, and the need for checking if the design intent is implemented across all of the components of a game. Essentially the behavior of the player needs to be modeled on an ongoing basis as the player navigates through the content of the game, and results compared against design intent or combined with e.g. churn prediction models. This is the case for all digital games, irrespective of their genre, design or platform. A key element in progression analysis is acknowledging the dynamic nature of gameplay and the impact this has on progression, i.e. that as long as players are active, progression is taking place, either directly by moving through the content of the game, or indirectly, by considering and thinking about how to address particular challenges of puzzles. Progression inertia or even cessation is one of the major sources of player frustration, and therefore not a desirable situation [START_REF] Canossa | Arrgghh!!! Blending Quantitative and Qualitative Methods to Detect Player Frustration[END_REF]. Furthermore, visualization of the results of analysis are essential to ensure that action can be taken on them [START_REF] Wallner | An introduction to gameplay data visualization[END_REF][START_REF] El-Nasr | Game Analytics -Maximizing the Value of Player Data[END_REF][START_REF] Cleveland | Visualizing Data[END_REF]. In the work presented here, behavioral profiling and progression analysis are combined towards evaluating player progression and visualizing analysis results.

Contribution

In this paper behavioral profiling through the clustering with Archetypal Analysis (AA) is combined with progression analysis, expanding on previous work in the area, and extending it into the area of Open-World (or Sandbox) Games. The proposed methodological framework is applied to the case of the major commercial action-adventure title Just Cause 2, using the main storyline of the game as a proxy measure of progression along multiple vectors. Results are visualized using Sankey flow diagrams, which permit visual inspection of the patterns in the behavioral data. The results show how players navigate the content of the title, and how some playstyles remain constant throughout the game, whereas others emerge or disappear as a function of player progress. Additionally, player performance as a function of progression and player profile are evaluated across a number of key metrics.

Just Cause 2

Just Cause 2 (JC2) is an action-adventure game set in an open-world environment, also commonly referred to as a sandbox game. Similar to other sandbox titles such as the Elder Scrolls-and Grand Theft Auto-series, it is characterized by free-roaming gameplay in a large 3D environment, in the current case covering over 1000 square-kilometers. The game was published by Square Enix in 2010 and sold over six million units, retaining an active player base today. The gameplay is reminiscent of other action-adventure third-person shooters, with the player controlling the game protagonist as he travels around the Panau environment whether to complete missions, explore, collect items, or just have fun with the mechanics of the game. There are computer-controlled enemies everywhere, and a substantial part of the gameplay is combat-focused, which also means that combat performance metrics are important for for the current aim of developing behavioral profiles.

A complete description of the gameplay of JC2 is out of scope of this paper, but in brief, JC2 is set in the fictional tropical island nation of Panau. Panau is ruled by a ruthless dictator. The player, taking on the role of Rico Rodriguez, an agent for an outfit referred to as the Agency, is tasked removing the dictator from power. This is done via progressing along different vectors notably the seven Agency-related missions, as well as missions from a number of Rebel Factions which helps wrest control from the dictators forces and into the hand of the factions. At the end of each mission chain for each Rebel Faction is a Stronghold mission, which form some of the most difficult tasks in the game. Upon completion of a Stronghold mission, the control of an administrative district is gained by the local Rebel Faction.

Progression in JC2 follows a directed gameplay approach [START_REF] Lopez | Gameplay Design Fundamentals: Gameplay Progression[END_REF], where all mechanics in the game are available from the beginning of the game, but that the player is gradually encouraged to use and become familiar with the mechanics of the game progressively. Overall, progression in the game towards reaching the point where the dictator can be toppled, is determined by chaos, a measure of the overall efforts of the player towards destabilizing Panau, whether through completing faction missions, attacking military posts, searching for and blowing up status of the dictator, etc. Thus the overall chaos achieved by a player provides a proxy measure of progress across the various progression vectors in the game. Furthermore, Agency missions unlock with chaos progression. Throughout the game, the player is via missions encouraged to navigate around Panau, which can be done using a wide variety of vehicles operating in the water, on land and in the air. Barring a few missions players can travel via any means they can capture. The player also has a grappling hook and reuseable para-sail/parachute which can be used for aerial movement as well as in combat.

Profiling and Progression: Related Work

Behavioral profiling in games has been explored by a number of researchers across academia and industry, providing a number of case studies showcasing how profiling can be performed using different techniques and in varied game contexts [START_REF] Drachen | Guns, swords and data: Clustering of player behavior in computer games in the wild[END_REF][START_REF] Normoyle | Bayesian Clustering of Player Styles for Multiplayer Games[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF][START_REF] Bauckhage | k-Maxoids Clustering[END_REF][START_REF] Shim | Behavioral Profiles of Character Types in EverQuest II[END_REF][START_REF] El-Nasr | Game Analytics -Maximizing the Value of Player Data[END_REF].

There are particular challenges associated with behavioral profiling in major commercial game titles, notably related to volume, variety, and volatility [START_REF] Drachen | Game Data Mining[END_REF]. These problems have been approached using a variety of techniques from de-scription and segmentation to machine learning. In the latter category, cluster algorithms [START_REF] Aggarwal | Data Clustering: Algorithms and Applications[END_REF] and other dimensional reduction techniques have been broadly applied across different game formats. Cluster analysis is an exploratory approach to building behavioral profiles, and aims at identifying groups for players with similar behaviors and identify the most important behavioral features in terms of the underlying patterns in the dataset [START_REF] Drachen | Guns, swords and data: Clustering of player behavior in computer games in the wild[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF][START_REF] Drachen | Player Modeling using Self-Organization in Tomb Raider Underworld[END_REF][START_REF] Shim | Behavioral Profiles of Character Types in EverQuest II[END_REF].

The work presented here forms an extension of earlier work on behavioral game research, notably: Sifa et al. [START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF], who investigated player progression in the action-adventure, single-player Tomb Raider: Underworld, and Drachen et al. [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF], who investigated player progression in the browser-based Massively Multiplayer Online Game (MMOG) Glitch. Sifa et al. [START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF] utilized Simplex Volume Maximization (SIVM) to develop behavioral profiles of the seven levels of Tomb Raider: Underworld, finding some behavioral profiles to be present across the levels, others to be present in only some of the levels. They noted that the variations in the profiles present for a level appeared related to the specific challenges and gameplay of these levels. Drachen et al. [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF] adopted a similar approach, but focused on time rather than in-game segments. The authors developed profiles on the auction house behavior of players using k-means clustering, and used Sankey diagrams to develop interactive visualization of the pattern of players moving into and out of profiles across monthly time bins. This visualization approach is adopted here. The work in the above mentioned publications in turn rests on other research in game analytics focused on behavioral profiling in games, e.g. [START_REF] Drachen | Guns, swords and data: Clustering of player behavior in computer games in the wild[END_REF][START_REF] Drachen | Player Modeling using Self-Organization in Tomb Raider Underworld[END_REF][START_REF] Shim | Behavioral Profiles of Character Types in EverQuest II[END_REF].

Dataset and Features

The dataset was provided by Square Enix, the publisher of JC2. The data comprises complete play histories from over 5000 JC2 players who played the game in 2010. A vast amount of behavioral features were collected including the in-game geographical coordinates for all player actions registered, as well as timestamps for these events. The dataset contains a detailed set of metrics from the gameplay, e.g. the total kills, the total chaos gathered by the player, the kilometers driven or the number of stronghold takeovers the player completed, and more.

Feature Definition and Pre-processing

The gameplay of JC2 gives rise to a huge variety of potential behavioral measures, across spatio-temporal navigation, combat performance, progression through the main storyline and numerous side quests to general exploration. This means that progression analysis can target a number of dimensions in the game, however, for all the free-roaming gameplay, JC2 features a central storyline, experienced across seven Agency missions. Progressing along these missions in turn require the player to reach specific levels of Chaos, which in turn means completing Rebel Faction missions and causing mayhem and destruction in general, which means that the Agency missions work as a proxy for multiple vectors of progression in the game, and it was therefore chosen to make these the vector of progression in the work presented here.

In order to keep the profiling task manageable, a subset of features was selected based on the core mechanics of the game, following the principles of Drachen et al. [START_REF] Drachen | Player Modeling using Self-Organization in Tomb Raider Underworld[END_REF], and the focus on the main storyline. This means for example including measures of distance traveled and locations discovered, which informs whether players focus on straight navigation between story points or are exploratory in their behavior, as well as combat performance metrics such as kill/death ratios (K/D ratio) and player death frequencies, and finally features related to more advanced mechanics such as base jumping, stunt driver points and the variety of vehicles used. The specific feature set chosen here does not impact on the analytical framework, which can be employed for any feature-set in any game context, but of course impacts on the kinds of conclusions that can be derived from analyzing them [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF][START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF]. The features are described in Table 1. In further pre-processing, all players with scores outside the 1-99th percentile were excluded from the analysis, in order to remove extreme outliers. Outliers can in other contexts be the focus of study, but in the current case the most extreme outliers were caused by faulty tracking and similar errors, and therefore excluded. The behavioral features were normalized using regular variance normalization, following the principles of [START_REF] Drachen | Guns, swords and data: Clustering of player behavior in computer games in the wild[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF].

Analysis

We applied AA models to all seven agency mission bins. For each the optimal number of clusters (k) was determined. To identify the number of clusters for the AA, we analyzed the residual sum of squares for all k values less than or equal to 20 and chose the number of clusters by utilizing the elbow criterion. The residual sum of squares indicated that the data consists of three main archetypes.

Progression Profiling via Cluster Analysis

Profiling game player behavior is an important practice in game analytics to obtain insights about how players interact with the game [START_REF] Bauckhage | Clustering game behavior data[END_REF][START_REF] Drachen | A Comparison of Methods for Player Clustering via Behavioral Telemetry[END_REF]. Unlike the commonly used predefined-segmentation based profiling methods, clustering based profiling approaches reveal more interpretable and actionable insights about the game play behavior. Cast as low-rank matrix decomposition problems, many clustering methods can be grouped under the same formalism for better comparison. In this work we will concentrate on two factor decompositions as the methods we used here follow that scheme. That is, given a data matrix X ∈ R m×n , our aim is to find two lower rank matrices P ∈ R m×k and C ∈ R k×n such that X ≈ PC. For the purpose of profiling, the factor matrix P contains the prototypical players that define the profiles and are selected by special procedure enforced by the algorithm whereas the matrix C contains the mixing or belongingness coefficients indicating how much a player belongs to the particular profiles defined by P [START_REF] Drachen | A Comparison of Methods for Player Clustering via Behavioral Telemetry[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF]. It is important to note that under this framework we can categorize the algorithms with respect to their constraints. For Archetypal Analysis [START_REF] Cutler | Archetypal Analysis[END_REF] P is constrained to contain extreme vectors that are defined as convex combinations of particular data-points whereas C is constrained to be column stochastic. Additionally, keeping the stochasticity constraint for C, for a more relaxed archetypal decomposition such as Simplex Volume Maximization, P is selected to be a datapoint lying on the data-simplex [START_REF] Thurau | Yes we can: simplex volume maximization for descriptive web-scale matrix factorization[END_REF]. For more information about constrained matrix decomposition and clustering we refer the reader to [START_REF] Drachen | A Comparison of Methods for Player Clustering via Behavioral Telemetry[END_REF][START_REF] Bauckhage | Clustering game behavior data[END_REF][START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF].

In this study we present a profiling analysis with two components: (1) Behavioral profiling of the JC2 players via the application of archetype analysis (by analyzing the resulting P-matrices) and the (2) generation of a Sankey diagrambased visualization (by using the resulting C-matrices) for the temporal behavior of the players across the defined profiles.

Player Profile Descriptions

The analysis with AA shows that the player profiles change over the different missions. Archetypal analysis provides an indication of the number of clusters between 4 and 6. Table 2 gives an overview of the distribution of the various main profiles among the different missions. One particular player group ("Low Performers") stands out in mission 2-3.

With AA we can identify eight main clusters, which appear and disappear in different missions and can have slightly different metrics depending on the mission-design. Average Player: This player cluster only appears in mission 6 and describes players with overall average values. Drivers: Players that drove the most kilometers and also performed the most base jumps and black market orders (cluster appears in mission 2). Elite Players: Players that played on the highest difficulty level with the highest kill/death ratio, for the rest of the features average values. They have an average exploration performance. In later missions (3 and 4) they also have higher values in hijacking cars and stunt driving points and less black market orders, base jumps (mission 5) and completed below average faction missions and stronghold takeovers (missions 6) and have a small number of kilometers driven (mission 7). Explorers: This player type is very constant and can be observed in slightly different forms over all missions. This cluster can be described as players with the slowest pace, a kill/death ratio above average, but the least amount of chaos collected per minute. They hijacked the most vehicles and discovered the most locations. In mission 2 they've already played in average three times longer than other player types and collect most stunt driver point. From mission 3 they have played at least twice as long as other player types and completed far most faction missions and stronghold takeovers and have discovered and completed nearly three times as many locations. Low Performers: Players with the worst kill/death ratio of all player types and most deaths per minute. Average values for the rest. One can also observe average or slightly above average performance regarding exploration and driving (mission 3-7). Low Performance Drivers: This cluster only appears in mission 1. Players with a fast pace, a very low kill/death ratio and most deaths per minute, but collected a lot of chaos per minute. They drove with many different vehicles and on average drove more kilometers than other clusters. Rushers: This player group is also very constant and appears in all missions but mission 6. It described players that completed the game the fastest. They collected the most chaos per minute, achieved a high kill/death ratio and the most kills per minute. They care little about exploring the world and discovered and completed the least locations (mission 3-5). Stunt Drivers: This group is only observed in the first mission and describes players who collected the most stunt driver points, drove with more different vehicles than the other player types, and also drove the most kilometers. Skill scores pretty similar to player type "Low Performance Drivers".

Player Progression along the Missions

Fig 1 gives an overview of various metrics, which can give the players feedback on their progress and/or their performance. Progress describes the a pre-defined ingame metric to describe the actual progress along the game. K/D-ratio describes the ratio between kills and deaths. In particular missions three and four seem to be "rich of action". In particular the last two missions seems to require the players to drive a lot according to the average metrics Kilometers driven. Stunt driver points can be collected by extreme driving activities. In particular in early missions more new locations (Locations discovered ) are discovered. Based on the performed analysis and visualizations different results with focus on player behaviour along the game progress can be presented.

Player Behaviour over the Missions

The analysis clearly indicates variations in the player behaviours over the duration of the game, despite the consistent presence of the three main clusters throughout all agency mission bins. Players do not remain in a single cluster when they progress through the game, they move relatively freely across clusters from agency mission to agency mission. While the dominance in the exploration based features stays the same for the Cluster "Explorer", the other features, like the skill features change over the course of the main storyline from agency mission to agency mission. Another quite consistent feature for all three clusters Fig. 3: How players change clusters along the main storyline: all players change the cluster at least once is the total time played, the "Explorer" cluster yields in the highest values of total play time across the whole main story line.

How many profiles enter players on average over the course of the game? The analysis indicates that all players change their in-game behaviour along the storyline. As illustrated in Fig. 3 all players change the cluster at least once. On average players enter 2.91 clusters over the course of the game.

How does in-game behaviour and performance change over the various missions? The analysis and the visualization clearly indicate that player behaviour changes over the different missions, which is also due to the nature of the mission design. The Sankey Diagram (Fig. 2) illustrates how players change the player types along the course of the seven missions.

How can we describe player behaviour of the different player profiles? Fig 4 refers to different in-game metrics and how these metrics change over the seven missions for the different main player profiles as described above. Progress is a metric, which is in particular high for Explorers. Kill/Death-Ratio is a distinct feature for Elite players. As illustrated in subfigure (c) the total time played (in minutes) is particular high for Explorers. Starting from mission 2 they start to strongly separate from the other clusters. This value is very low for Rushers. Stunt driver points are also very high for Explorers, but also for the group of Drivers. Factions Missions are popular for Explorers, but interestingly, only in Mission 6, also for the Low Performers and Elite Players.

Conclusion and Discussion

In this paper behavioral analysis has been presented for Just Cause 2, focusing on the relationship between playstyle and progression in the game. It is to the best knowledge of the authors the first time an OWG has been the focus of progression analytics, and provides insights into the behavior of players in these kinds of titles. The results and visualizations directly expand and refine the approaches of [START_REF] Drachen | Going Out of Business: Auction House Behavior in the Massively Multi-Player Online Game Glitch[END_REF] and [START_REF] Sifa | Behavior Evolution in Tomb Raider Underworld[END_REF] via combining OWGs, Archetype Analysis, Sankey visualizations and in-depth exploration of key behavioral metrics. The progression of the player clusters exhibiting similar playstyles has been explored across a number of behavioral metrics related to progression in an OWG, such as discovered locations, K/D ratios or distance traveled. The approach provides a template for similar analyses in other games for evaluation of behavior and game design. Future work will explore different perspectives of progression and extend the work presented here to other games. The work presented here focuses initially on using the main story line of JC2 as a proxy measure of progression, expanding to integrate other metrics. However, there are many viewpoints that could be adopted on player progression in games. For example, progression can also mean gradual acquisition and upgrading of weapons, developments of player skill, or exploration of the environment. Similarly, in other games, progression is a viewpoint that can be applied in a range of situations. Irrespective of the game and the specific aspect of progression studied, it is a key aspect of game design and vital to balancing. The work presented here combines profiling and progression towards enabling the detailed evaluation of player progression.

Fig. 1 :Fig. 2 :

 12 Fig. 1: Player progression over the missions based on different in-game metrics.

Fig. 4 :

 4 Fig. 4: Player progression over the missions based on AA clusters.

Table 1 :

 1 Overview of the behavioural features used for AA

	Feature	Description
	Total playtime	Total time played until the completion of the correspond-
		ing agency mission
	Chaos/minute	Chaos gathered per minute (Chaos increases by complet-
		ing missions, destroying buildings).
	Kill/Death Ratio	(K/D ratio), the number of kills the player has scored
		divided by the number of deaths suffered
	Kills/minute	Number of kills the player has scored per minute.
	Deaths/minute	Number of deaths the player has suffered per minute.
	Hijackings	Number of times the player hijacked a vehicle
	Base jumps	Number of times the player used the game feature base
		jumps.
	Stunt driver points Number of points the player gathered by performing
		stunts with vehicles.
	Stronghold takeovers Number of completed stronghold takeovers.
	Kilometers driven	Distance covered by the player in kilometers.
	Locations discovered Number of discovered locations.

Locations completed Number of completed locations.

Table 2 :

 2 Overview of main AA types by missions (in %)

	Mission

https://github.com/timelyportfolio

Acknowledgments The authors would like to express their sincere gratitude to Square Enix for making the Just Cause 2 dataset available for analysis.


Heinrich Söbke
email: heinrich.soebke@uni-weimar.de

Laura Weitze
email: laura.weitze@uni-weimar.de

The Challenge to Nurture Challenge Students' Perception of a Commercial Quiz App as a Learning Tool

Keywords: Quiz Apps, Educational Content, Mobile Learning, Explorative Study

de niveau recherche, publiés ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

Quizzes are established as an assessment tool. Not so common, but remarkable as well are their applications as learning tools [START_REF] Mcdaniel | Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement[END_REF][START_REF] Wiklund-Hörnqvist | Strengthening concept learning by repeated testing[END_REF]. The quiz format can be used regardless of the technical area of the knowledge. In this context we argued for a common technical infrastructure to facilitate quizzes, especially in educational contexts [START_REF] Söbke | One Size Could Fit It All: a Common Infrastructure for Quiz Apps[END_REF]. Furthermore, recently raised proliferation of mobile internet has enabled the ubiquitous and enormous spread of quiz apps (e.g. Quizkampen [START_REF] Feo Media | QuizClash | Challenge your friends![END_REF] or QuizUp [START_REF]Plain Vanilla: QuizUp -Connecting people through shared interests[END_REF]). In a previous study [START_REF] Söbke | Space for seriousness? Player Behavior and Motivation in Quiz Apps[END_REF] we found that quiz app players enter the game with an expectation to learn. Additionally, competition with friends has been identified as one main motivation for users of such apps [START_REF] Söbke | Space for seriousness? Player Behavior and Motivation in Quiz Apps[END_REF]. Thus commercial quiz apps can be considered as potential educational tool.

In 2015 the commercial quiz app QuizUp has been opened for user-defined topics [START_REF] Woods | QuizUp launches tools for creating your own trivia categories and questions[END_REF]. We used this feature in two engineering university courses. As a result, we found that players can be categorized in two groups: learnerswho accomplish their quantum of educational content and then leave the appand gamerswho get stuck in the game after having fulfilled their learning duties [START_REF] Weitze | Quizzing to Become an Engineer -A Commercial Quiz App in Higher Education[END_REF]. Further results have not drawn a clear picture. On one hand the integration of the app into the course has been appreciated by students as an enrichment, on the other hand the actual usage of the apptrackable via public ranking listshas not been overwhelming. This ambivalence provided no answer to the question, to what extent quiz apps can contribute to intrinsic motivation in learning contexts. Furthermore, more detailed insights in usage experiences of such apps within educational settings would be helpful for the purposeful design of effective application scenarios. Therefore, the aim of the following explorative study is to contribute to a set of general conditions, necessities and requirements, which could frame the usage of game-like quiz apps as educational tools.

Study design

The employed app QuizUp uses multiple choice questions with three distractors. Two players can compete in a match of seven questions all belonging to a certain topic. If currently there is no other player available, the system automatically assigns as opponent a bot. A question has to be answered within 10 seconds. The less time is needed for the correct answer, the more points are earned. A match takes on average three minutes.

As a prerequisite, we have developed a corpus of 57 questions about three areas of basic knowledge in the engineering topic of Ecological Sanitation. (The publicly available topic in QuizUp is called "NASS" -New Alternative Sanitation Systems). This corpus contains questions of different difficulty levels: there are questions about simple factual knowledge like abbreviations. On the other hand there are more complex questions which require the comparison of systems. Questions differ in text length, but they cannot exceed a system provided maximum length. During the study the corpus remained unchanged.

As participants 4 students (2 male, 2 female) of engineering courses of study were recruited by means of online pin boards. Prerequisites for participation were being a student, interested in quiz apps and technical topics, the possession of a smartphone and being available for a determined time frame. As a compensation a € 25 onlineshopping voucher was promised. Those four persons, who responded most quickly, have been selected. Except for one person they had no prior experience with QuizUp. However, all of them were active players of Quizkampen. Although such a small sample size clearly limits the reliability of the results, we have chosen this approach to enable a familiar environment during the gaming sessions very quickly.

Within 12 days three meetings have been scheduled. One part of each of these meetings has been an attended gaming session of 30 minutes. Additionally, these meetings have been used to perform semi-structured interviews [START_REF] Schmidt | The Analysis of Semi-structured Interviews[END_REF] with the participants. The interviews have been roughly guided by the main categories Content, Motivation and Learning outcome: These topics constitute main pillars of games in educational contexts. In the last session, participants had to answer a questionnaire. After session 1 and session 2 they have been provided with extra tasks for the next meeting in order to keep up the engagement with the app over time. In the following we refer to the period between session 1 and session 2 as home period 1 and to the period between session 2 and 3 as home period 2.

Session 1. Here we conducted a semi-structured group interview (and asked additionally for prior knowledge, expectations and experience) and a play session of 30 minutes. Participant played against each other, they connected to each other in QuizUp and established a chat group with the purpose of coordination. A 20-page brochure containing an overview of the topic NASS [START_REF] Dwa | Do we need New Alternative Sanitation Systems in Germany ? Deutsche Vereinigung für Wasserwirtschaft[END_REF] was handed out to the participants. The task for home period 1 was to play at least one match a day. Outside of the gaming sessions, players were free to play against other participants or bots The intention of this task was to establish regular play and to point students to a bonus which is issued by QuizUp to reward daily play.

Session 2. This session took place seven days after the first session. Besides the play session, we interviewed them about their experiences since the last sessions. The task for home period 2 was to reach 25,000 points which is equivalent to 50 matches (if students play each day and are rewarded with the bonus). Intention of this instruction was on one side to investigate, how participants react to becoming the game a task instead of being fun. On the other side, we wanted to know if the habit of daily play has been kept up. This was the case, 3 of 4 participants still declared in session 3 to have played on a daily base.

Session 3. In this session, held five days after session 2, we conducted an interview about their experiences in the last five days. After the play session, a questionnaire has been answered.

There have been no variations of the experimental setting during the study. The only condition we changed was the specific task to accomplish during the home periods.

Results

Sessions, interviews and questionnaire revealed various, partially very detailed aspects. In the following we give an overview about selected aspects. Although categorization has been partly ambiguous, we assigned each aspect to one out of three main topics: Content (Knowledge has to be expressed by means of multiple choice questions (MCQs)), Motivational aspects (Quiz apps use game mechanics, therefore fostering of motivation, especially intrinsic motivation is a relevant aspect) and Learning outcomes (Learning is the purpose for employing quiz apps in this context). These were the main categories of the semi-structured interviews. Subcategories have been developed a posteriori according to the found results.

Content

After having defined three basic learning fields, the according questions corpus has been designed due to the recommendations of Haladyna & Rodriguez [START_REF] Haladyna | Developing and Validating Test Items[END_REF]. Nevertheless, there are further observations. In the following we describe them and suggest possible improvements. Text length. Questions with short texts for question stem and answer options were preferred by participants, as they do not require much reading effort. This is especially critical as there is time pressure and limited space on the display. Long questions require more time for reading and are printed in smaller letters. Both increases the felt difficulty of answering. The allowed time to answer could be adjusted programmaticallye.g. linearly according to the length of the question stem and answer options.

Negation. Further, although negation in question should be avoided in general [START_REF] Haladyna | Developing and Validating Test Items[END_REF], one participant gave an example of such a question, which he considered as very informative: Which is not an appropriate usage of treated stormwater? Body care -Washing machine -Toilet flushing -Cleaning. He stated that the correct answer is not a far-fetched appropriate usage, so all options have to be considered carefully, which contributes to learning.

Level design. Not knowing the correct answer frustrates (One participant uttered often apparently almost in desperation "How should I know that?!"). So there is the need to order questions by complexity and release them stepwise. Questions, which refer to further, not included complex knowledge, can be seen as critical. Fig. 1 depicts such a question: the question asks for a specific sanitation system, which fulfills a particular characteristic. Each answer option names such a system. If these systems are not described by other questions, the question pool is not self-contained. External knowledge is required. However, participants have not looked up additional knowledge on their own. Our conclusion is that not self-contained question pools require embedding in a formal educational setting.

3.2

Motivational aspects

Motivation is a key issue for a game based learning app, because motivation encourages engagement with the app and therefore causes indirectly a learning outcome. In general, the app together with its educational content is experienced as a game, as indicated by the results of the Game Experience Questionnaire (GEQ) [START_REF] Ijsselsteijn | The Game Experience Questionnaire: Development of a self-report measure to assess the psychological impact of digital games[END_REF]. We included the GEQin-game variant in the questionnaire. Among the results (see Fig. 2) are high values especially for Positive affect (3.63) and Challenge (3.13), whereas Tension (1.88) is rated low. Although the sample size is by far too small for reliable quantitative findings, received values seem to be game-typical, except for the quite high value of Negative affect (2.88).

One participant compared the regular play of the educational topic with the healthy habit of eating fruits: "Sometimes it tastes good, sometimes not". In the following, further observed aspects are described.

Competition. Matches against other persons have been indicated as more interesting as matches against bots. However, in many cases a specific bot, which imitates a perfect opponent (answering all questions within the lowest time possible) has been assigned. So there was no chance to win a match. A participant stated that such a bot assignment strategy made him quitting the gaming session. Another aspect of competition is caused by ranking lists: One participant reached more than 35.000 points in the first week. This is equivalent to 150 matches or 7.5 hours of gaming. His dedicated intention was to conquer the top rank. Another player experienced a software failure, which prevented him from having a look at the ranking lists. As a consequence, he reported a decreased motivation.

Fig. 2. Mean value and standard deviation of GEQ categories [12] (n=4)

Social Presence. In the play sessions students competed against each other and teased each other verbally. This was described by the participants as highly motivating.

Direct encouragement. In the first play session participants reported to be motivated by a textual app feedback after a match. However, this effect has not been mentioned in later sessions.

Focus and engagement. During a match the participants almost did not talk and seemed to be quite focused, especially in the first session. As an indicator for engagement, they uttered anger about false answers and cursed from time to time. Furthermore, working with the app requires attention: percentage of correct answers lowered, when there was a distraction, e.g. a switched-on TV during the home periods.

App-internal distraction. Besides our educational topic, QuizUp contains the original entertainment topics. In the first home period, three of four participants played entertainment topics at the same rate as technical topics. In the second home period, the number of matches about entertainment topics even increased.

Challenge. Especially in the third session the decreased motivation has been mentioned. Participants stated that the game has been receipted as boring, because 90 percent of the questions could be answered from memory. Fig. 3 mirrors the decrease of motivation over time. Fig. 3. Course of motivation (self-assessment) over study period on a scale between 0 (unmotivated and 1 (highly motivated)

Learning outcome

All participants pointed out, that they have experienced learning in quiz apps and especially during this study. The following detail aspects have been found.

Level of knowledge. Participants stated that mostly factual knowledge can be taught with quiz apps. Simple questions, e.g. naming abbreviations, have been learned easily and could be answered in session 3 without providing answer options. Questions about more complex topics could be answered after a while. However, especially in the case of complex topics, participants admitted, that they knew the correct answer, but did not gather the meaning of the question and missed an overview of the topic.

Learning curve. Already in the course of the first session effects of learning could be observed, participants could provide more and more correct answers. In the beginning of session 1, mostly 1 to 2 questions of a match (seven questions) have been answered correctly. At the end of the first session, after approximately 10 matches, this rate increased to 4 to 5 questions per match.

Intervals of usage. One participant, which played in the home period 2 only on two days, observed fading memorization of knowledge after longer breaks. The answers could not be retrieved easily from memory, but she had to think again about the possible answers to the questions. There are two possible interpretations for this observation. First, it could be a sign of surface learning, which does not lead to long-term memorization. On the other hand, it could be a learning strategy for deeper learning, as repeated considerations for finding the correct answers could help to think thoroughly about the question.

Situated learning. During the play sessions, participants started discussions about missed questions and their correct answers. These face-to-face discussions contribute to learning [START_REF] Lave | Situated learning: legitimate peripheral participation[END_REF].

External knowledge sources. Additional sources for topic-relevant knowledge have almost not been used. The established chat group did not show discussions about the technical topic. The brochure about the topic which has been handed out to participants has not been used. In the second and third session participants had not questions about the contentdespite their statement, that they had not a thorough comprehension of the content of many questions. Although, students indicated that they had a better overview about the topic compared to the beginning of the study, it has to be investigated, if and in how far deeper learning processes can be triggered by a quiz app.

Time pressure. The limit of ten seconds to answer the questions and its relevance for the reward seems to hinder learning. Participants reported that the urge to answer fast, leads at least partly to superficial reading and to a kind of visual pattern recognition: The shape of question and answer options is recognized, but the question is not read thoroughly. However, this would be necessary for learning.

Discussion, conclusions and outlook

The most limiting restriction of this study is the low number of participants. Even for an exploratory approach four persons are quite a small sample. However, this small number of participants led to familiar atmosphere during the gaming sessions, which may have caused a more authentic behavior. Nevertheless, all findings have to be valued in consideration of this small sample size. The relative small size of the question corpus may have influenced findings as well. The grade of challenge may have not fallen that fast in the presence of a larger corpus. However, in this case the grade of frustration may have decreased the motivation of participants. Due to such a potential ambiguity, we consider the findings as hints, which require more detailed investigations.

In general, this study confirms learning effects of quiz apps. However, there are multiple aspects still to be discussed. On a more technical level, such aspects are partially related to the limitations and features of the studied app, QuizUp. So, learning algorithmsas implemented in flash card systems [START_REF] Leitner | So lernt man lernen. Der Weg zum Erfolg[END_REF][START_REF] Fritz | Independent Research Report: outcomes of using phase-6[END_REF] are missing. More statistical data about answered and failed questions has been indicated as potentially being motivational. The role of time in the match process needs to be balanced carefully. Direct time pressure seems to be counterproductive. Further, the foreseen reading time should be adjusted specifically to current text length and question complexity.

In a broader range there is the process of content generation: an important aspect is the kind of knowledge which can be transformed into MCQs. Complex knowledge may have to be conveyed in other ways (e.g. lectures). Questions do not provide sufficient possibilities to teach all kinds of knowledge. An enrichment through graphics and pictures would improve current possibilities. Furthermore, we have found a relatively fast decreasing motivation among participants in this study. As a countermeasure we suggest the embedding of quiz apps into educational settings, which determine e.g. level design, regular play sessions and concrete app-related goals. In general, proven settings still have to be defined. Another important aspect, which has to be discussed is the role of entertainment topics: are they a distraction from or an enrichment for the learning process? Further, there is the basic aspect of type-dependency: the grade of motivation a quiz app can foster in a person depends on their individual preferences. Relevant in an educational context is to reach a sufficient motivation for any learner. Consequentially, a quiz app has to compete with genuine question-based learning tools like Skive [16], StudyBlue [START_REF]Studyblue -Conquer Your Course[END_REF] or Quizlet [START_REF]Quizlet LLC: Simple free learning tools for students and teachers | Quizlet[END_REF]. In a further step we currently conduct a comparative study between a quiz app and a dedicated learning app regarding efficacy and acceptance. Commonly, app supported learningno matter if game-based or notcan offer the advantages of adaptivity, ubiquitous accessibility and -at least basiccollaboration. Therefore, further research should help to mitigate current limitations.

Fig. 1 .

 1 Fig. 1. Sample question with references to external knowledge.

Acknowledgements

We thank Lukas Hartung, who pursued the question design and the interviews with great commitment [19]. Parts of the activities described in this article have been supported by the German Federal Ministry of Education and Research (BMBF) under the grant agreement FKZ 033W011E. The authors gratefully acknowledge this support. Moreover, we thank the anonymous reviewers of this contribution for their valuable and helpful comments.


Volker Settgast
email: volker.settgast@fraunhofer.at

Johanna Pirker
email: jpirker@iicm.edu

Stefan Lontschar
email: stefan.lontschar@student.tugraz.at

Stefan Maggale
email: stefan.maggale@student.tugraz.at

Christian Gütl

Evaluating Experiences in different Virtual Reality Setups

Keywords: virtual reality, immersion, cybersickness, Oculus Rift, CAVE

This paper describes the evaluation of three different scenarios in the fully immersive room-based virtual environment DAVE (Definitely Affordable Virtual Environment) and a head-mounted display, the Oculus Rift. The evaluation focuses on comparing the two immersive environments and three different scenarios (observation, emotion in a roller coaster, and interaction) in regards to typical virtual-reality characteristics, such as immersion, engagement, but also on cybersickness and the overall experience. First results indicate the DAVE environment better supports scenarios, which require the user to directly interact with the environment. The roller coaster scenario creates stronger immersion and a higher nausea-level, while the interactive task is more engaging in terms of fun.

Introduction

Over the last years, the potential of immersive virtual environments (VE) has been described for various application scenarios. In particular the current trend of affordable head-mounted displays (HMD) allows a wide range of users to access different virtual reality (VR) applications. Such immersive experiences are not only interesting for entertainment, gaming, and simulations, but also for training and education scenarios [START_REF] Stone | Applications of virtual environments: An overview. Handbook of virtual environments: Design, implementation, and applications. Human factors and ergonomics[END_REF][START_REF] Kim | Effects of virtual environment platforms on emotional responses[END_REF].

However, in particular in learning and training applications different scenarios often require different interactions and activities in the virtual reality. For example, specific training tasks would require rich and realistic user interactions (e.g. learning how to use a specific machine). Other tasks require more freedom in the environment such as the possibility to freely examine the objects and the environments. For other experiences often only the observation and the experience of the virtual scenario is sufficient.

Different virtual reality devices and setups support different degrees of freedom, of immersion, and interactions with the environment. In a room-based fully immersive virtual environment (such as a CAVE) users are still able to see their own body and set in relation to the virtual world. It is possible to use additional tools in a natural way (e. g. a map or a smart phone) and interact directly with other users. Head-mounted displays support more flexible forms of experiences and activities, for example show a different body for the user or trick the sense of orientation. However, they often give users not the possibility to directly interact with the environment, since the representation of the own body is missing or poorly represented. Different forms of interaction are challenging, since consumer HMDs only give a limited range of sensors for tracking the body. [START_REF] Mcgill | A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted Displays[END_REF].

To design rich learning and training scenarios in a virtual environment it is not only necessary to focus on the different interactivities, but also to design the experience with consideration of different virtual reality characteristics and problems to create a sound user experience. This in particular includes immersion and cybersickness.

In this work we present a first comparison of different activities (observations, strong emotions, interactions) in two virtual reality systems (CAVE, Oculus Rift DK2) with focus on typical virtual-reality characteristics, such as immersion and engagement, but also on the potential issue of cybersickness.

Background and Related Work

Significant research and development efforts of different VR experiences have discussed and shown the potential of immersive VR already very early [START_REF] Sutherland | The ultimate display. Multimedia: From Wagner to virtual reality[END_REF][START_REF] Bishop | Research directions in virtual environments[END_REF].

Comparing Experiences in Virtual Reality Environments

Tan et al. [START_REF] Tan | Exploring Gameplay Experiences on the Oculus Rift[END_REF] compare and evaluate gaming experiences in the Oculus Rift. They used a Oculus Rift DK1 and a traditional computer with monitor setup to play a first-person shooter game (Half-Life 2). The test persons had to play the game on each system and then they answered two main question. Ten test persons were participating. The first main question was about the experience using the Rift compared to a standard PC setup.The second question was about what to take care about when designing games for the Rift based on the findings about the peoples experience.The study shows that most participants experienced cybersickness 8 on a scale from 1 to 10 and that cybersickness only occurs on the Oculus Rift. They say that: "cybersickness was a strong factor in modulating peoples gaming experiences using the Rift." Kim et al. [START_REF] Kim | Comparison of desktop, head mounted display, and six wall fully immersive systems using a stressful task[END_REF] compared three different VR Systems with two tasks for each system ( A 3 x 2 study). The Systems used were a standard PC without any VR elements, a HMD with limited peripheral vision and a fully immerse CAVE environment (DiVE). The two tasks where a low stressful and a high stressful task where the participants had to find cards with a certain word-color combination. The difference was, that in the low stress task the color and name of the words was congruent and in the high stress task they were not. Additionally in the high stress version of the tasks, the participants were influenced by averse simulations (loud noises, flashes and occasional tactile vibrations). The study focused on cybersickness, presence, and emotional changes. Additionally the time to find the cards was tracked and a galvanic skin sensor was used to measure sympathetic autonomic arousal. The result showed, that there were higher emotional arousal values in the DiVE and HMD compared to the desktop system but there were also differences between the VR devices. In the DiVE the emotional changes were mainly positive whereas with the HMD the effects were negative. Overall the participants showed the highest SCR changes with the DiVE, with the HMD the changes were moderate. In the end the findings showed, that a CAVE like environment is best used to evoke happy emotions whereas a HMD device is advantageous to evoke negative emotion. The desktop system resulted in the smallest changes overall and moderate task performance. This knowledge can be used e.g. in psychological studies because it gives insight in which technologies are viable to use for which treatments. A HMD device can therefore maybe be used to treat anxiety disorders.

Cybersickness

As VR-devices become more and more popular, there is still a big obstacle to overcome to make it an enjoyable experience for everyone, namely cybersickness. Cybersickniss symptoms in virtual reality environments are similar so motion sickness (e.g. nausea, disorientation, discomfort, vomiting). The sensitivity and grade of cybersickness differs from person to person and thus is hard to keep track of. Most studies rely on self-reported tests after using VR devices and so it highly depends on every participant, how the results of the study develop.

Davis et al. compare the two HMDs Oculus Rift DK1 and Oculus Rift DK2 [START_REF] Davis | A systematic review of cybersickness[END_REF]. In a follow-up paper Davis, Nesbitt, and Nalivaiko [START_REF] Davis | Comparing the onset of cybersickness using the oculus rift and two virtual roller coasters[END_REF] introduce new techniques to explicitly measure cybersickness. The idea is to create certain tasks and actions that induce cybersickness and with help of these it would be possible to measure psychological conditions and objectively quantify cybersickness symptoms. The paper also refers to older studies and sets different approaches to measure this condition into contrast to find well suited ways of finding an objective measurement. The study was held with 30 participants and with the use of two different virtual roller coasters. With these two different coasters it was possible to find relations between e.g. the level of detail, the level of user interaction and the nausea condition of each participant.

In conclusion the study found, that the coaster with more complex realism (Helix) causes a higher level of nausea offset compared to the other and is therefore suggested for further studies regarding objective measurement of cybersickness. Such studies are necessary to bring VR further to the possibility of everyday usage because only with cost effective and objective measurements, which still have to be found, this will be achievable [START_REF] Davis | Comparing the onset of cybersickness using the oculus rift and two virtual roller coasters[END_REF].

Polcar and Horejsi [START_REF] Polcar | Knowledge Acquisition And Cyber Sickness: A Comparison Of VR Devices In Virtual Tours[END_REF] compared three devices with a group of 45 students. The devices were a regular PC workstation, a Stereoscopic projection wall (CAVE, PowerWall, StereoWall) and an HMD (Oculus Rift DK2). After the test participants had to fill out a questionnaire about cybersickness, their symptoms, and the level of knowledge acquisition. The goal was to find a comparison of these devices. In the third part of the paper, they describe the effects of virtual learning and cybersickness. In conclusion there found no big difference of knowledge acceptance by doing a training exercise in reality or in a virtual world. While different controllers did not have much effect on the results, a larger display enhances the knowledge acceptance rate. In this study males achieved a better performance at this task than females. This effect decreased when the virtual environment was viewed on a bigger screen.

The Setting

For this study we used 2 ((a) Oculus Rift DK2, (b) DAVE) x 3 (tasks based on (1) observation, (2) emotion, (3) interaction) experiment setup with focus on comparing immersion, cybersickness, and the overall experiences.

Virtual environments

Oculus Rift The Oculus Rift is a Head Mounted Display developed by Oculus VR since 2012. The first commercial version was released in March 2016. For this paper the second pre-released developer kit (DK2) of mid 2014 was used 4 . In this version the display has a full HD resolution which is divided vertically showing the stereoscopic image for both eyes. Compared to prior HMDs the Oculus Rift was able to increase the field of view to 110 degree by using lenses and adjust the rendered images accordingly. An optical tracking system is used in combination with an orientation sensor for the localization of the users head. A sitting and a standing setup is possible but the range of movement is limited to less than two meters because of the cable-based video transmission.

DAVE

The Definitely Affordable Virtual Environment (DAVE) is an immersive projection room with three side walls and one floor projection [START_REF] Lancelle | Definitely Affordable Virtual Environment[END_REF]. The projection screens are 3.3 meters wide and 2.7 meters high. (see Figure 1). Stereo projectors with HD resolution are updated at 60 Hz. Stereoscopic shutter glasses are used, similar to the ones also known from 3D TV sets or 3D cinemas. In addition, an optical head tracking system allows a correct parallax and creates an undistorted view for the main user. Within the 3.3 by 3.3 meters the user can walk around an object to see it from all sides. A big advantage compared to most HMDs is the very wide field of view. Such a CAVE provides a visually convincing immersive experience and while allowing the user to see her own body.

The Implemented Scenarios

For the study three different scenarios (see Figure 2) were implemented with the goal to create three different experiences (observation, strong emotion, and interaction). Task 1 (Observation) The first task was mostly focused on letting people become familiar with the systems. It was just a stationary scene where the participants had to find a certain object on a big model (see Figure 3-a). Participants had to move around to find the object. The difference between the task with the Rift and in the DAVE was the object on the model they had to find.

Task 2 (Emotion) This was the dynamical task where the study participants had to take a ride in a virtual roller coaster once with the DK2 (see Figure 3-b) and once inside the DAVE. The users only had to sit on a chair and experience the virtual ride. Afterwards they had to describe their feelings while riding. Some of the test persons experienced quite a lot of cybersickness during this task but for none it was enough to interrupt the ride. The task was exactly the same with the Rift and in the DAVE.

Task 3 (Interaction) The last task was a dynamical scene where the study participants had to interact with the virtual environment. The goal of this task was to catch or deflect as many balls as possible in a certain time. The balls came flying towards the user from a virtual canon. This task created mostly positive reactions from all participants because it was much fun to play. There was no difference between the task with the DK2 (see Figure 3-c) and in the DAVE. In both setups a Microsoft Kinect was used for detecting the users hands.

Preliminary Study

To evaluate the different scenarios we devised a study setup which should shed light on the participants' experience in the two virtual environment setups with focus on different aspects, such immersion, nausea level, and engagement. The tasks, as described above, were designed to cover activities focusing on (1) observations, (2) emotions, and (3) interactions in virtual environments.

In a first study we compared the the two different virtual reality environments with 8 persons. One environment was an HMD and the other one was the DAVE. The participants had to do three tasks in each of these two virtual environment. Before they started with task 1, they had to fill out a pre-questionnaire. After each task they rated the immersion, nausea level, overall experience, and fun. After completing all 3 tasks in the DAVE, they filled out a post-questionnaire At the end they had to fill out the task questionnaire of part two and a specific cybersickness questionnaire.

Participants

To evaluate the scenarios 8 participants (5 female) between 21 and 48 (M = 28.38, SD = 8.29) were recruited. After a first introduction, participants completed an pre-survey with demographic information (e.g. age, gender, profession), experience with games and virtual realities. 6 participants are students. On a Likert-scale between 1 (not at all) and 5 (definitely) the participants rated their experiences in computer usage with a arithmetic mean of 3.13 (SD = 1.36) and their experience wit video games with a mean of 2.0 (SD = 1.6). All of them mentioned that they are not experienced with Virtual Reality. Only 2 play often or relatively often computer games. 3 like playing video games. 4 have heard of a CAVE/DAVE environment, 1 of them has already used one. 4 have heard of the Oculus Rift, 3 have already used one.

Setup

The virtual reality Oculus Rift Developer Kit 2 (DK2) and the DAVE environment as described in Section 3 were used for the evaluation. For the task design

Method

Immersion, Experience, and Engagement To evaluate aspects such as immersion and fun we used two different measures. (1) After each task we asked the participant to rate immersion, fun, and if they have liked the experience on a Likert scale between 1 (not at all) and 10 (very) to receive immediate feedback. (2) After they have completed all three tasks for one device we asked them to complete slightly modified version of the Game Engagement Questionnaire (GEQ) [START_REF] Brockmyer | The development of the Game Engagement Questionnaire: A measure of engagement in video game-playing[END_REF]. GEQ is designed to measure engagement in games. It provides a set of 19 questions (we used 18 for our study) to measure absorption, flow, presence, and immersion. Since we measure the "game engagement" after the interaction with each setup we are able to compare these values for the two different virtual reality setups.

Cybersickness As also described in [START_REF] Davis | Comparing the onset of cybersickness using the oculus rift and two virtual roller coasters[END_REF] we used a subjective individual rating of the participant's perception of their nausea level to evaluate cybersickness. The participants were asked after each task to rate their nausea level between "0no discomfort" to "10 -feeling like vomiting".

Findings

Experiences

Immersion Participants rated their immersion level on a scale from 1-10 after each tasks slightly higher in the DAVE. They rated in particular the rollercoaster experience as immersive (see Figure 5(a). Looking at the GEQ (see Figure 6) the overall immersion-level in the DAVE is also rated higher compared to DK2.

Cybersickness After each task the participants were asked to describe their nausea-level on a scale between 0 -no discomfort" to "10 -feeling like vomiting". Figure 5-b gives an overview of the participants' nausea level in the two different virtual environment. The value was for both devices very high after the Rollercoaster task (Task 2). The nausea level difference between DAVE and DK2 indicates that this feeling is only slightly higher in the DAVE. . Figure 6 illustrates the four main (absorption, presence, flow, and immersion) concepts as a result the 18 different GEQ-questions. All engagement metrics were described as slightly higher in the DAVE compared to DK2. In particular Immersion as factor of engagement was rated extremely high in the DAVE environment.

The tasks

7 out of 8 participants would prefer the DAVE over DK2 for the observation tasks. Reasons for that were described as "more realistic interactions" or "better graphics". Participants rated the difficulty of finding the object in the DK2 (M=2.5, SD=1.41) slightly higher compared to the DAVE (M=2.0, SD=1.07). 7/8 rated their experience in the virtual roller coaster as a fun experience. Three had a fear emotion at some point of the ride. Four would want to use this simulation again. Two would prefer DK2 for this simulation ("movement more realistic in DK2"), six the DAVE ("feels more real").

On a scale between 1 (not at all) and 5 (very) participants described the fun while playing the minigame with an arithmetic mean of 3.88 (SD=1.25). 6/8 would be interest in further developed versions of this game. 7 would prefer the DAVE over the DK2 to play this game ("not so heavy glasses", "display of hands not realistic in DK2", "hand movements more realistic in DAVE").

Conclusion and Discussion

The presented study is designed as the beginning of many tests with this setup and gives a first overview. Early results indicate, that a DAVE environment gives participants more freedom in regard to body perception, small-scale movement, and more realistic images. However, all tasks were mainly designed for small movements in a limited space. While the roller coaster scenario (designed to create strong emotions) gives participants a strong feeling of immersion and creates a higher nausea level, the interactive playful task is rated as a more fun task.

Further studies should also include scenarios, which require participants e.g. to travel distances. Also due to the small study setup and the natural differences between the two virtual reality environments (HMD vs. room-based virtual environment) many key limitations are given. Given the current rapid development of HMDs, it will be important to extend the current study with the latest HMD-technologies and other platforms. Future studies investigating more specific emotions and different forms of interactions with a larger participant base will extend the present findings.

Fig. 1 .

 1 Fig. 1. The DAVE: A four-sided CAVE-like immersive environment

Fig. 2 .

 2 Fig. 2. The three scenarios from left to right: complex model for observation, roller coaster and catch-the-ball game

Fig. 3 .

 3 Fig. 3. The three different tasks as rendered in the Oculus Rift DK2.

Fig. 4 .

 4 Fig. 4. The tasks in the DAVE; from left to right: Task 1 -Observation: The participants were asked to find a specific part of the machinery by observing the scene; task 2 -Emotion: The participants had to ride a roller coaster; task 3 -Interaction: In a mini game the participants had to catch balls shot in a random angle at them.

Fig. 5 .

 5 Fig. 5. Subjective rating of immersion, nausea level, experience, and fun after each tasks -1 (not at all) 10 (very)

Fig. 6 .

 6 Fig. 6. Game Engagement Questionnaire (GEQ).

https://www.oculus.com/dk2/