Veronica Sundstedt
email: veronica@cs.bris.ac.uk

Efstathios Stavrakis
email: efstathios.stavrakis@sophia.inria.fr

Michael Wimmer
email: wimmer@cg.tuwien.ac.at

Erik Reinhard
email: reinhard@cs.bris.ac.uk

A Psychophysical Study of Fixation Behavior in a Computer Game

Keywords: CR Categories: I.3.7 [Computer Graphics]: Three-Dimensional Graphics and Realism-Virtual Reality; Psychophysics, Electronic Games, Eye Tracking, Visual Attention, Eye Movements, Saliency

Prediction of gaze behavior in gaming environments can be a tremendously useful asset to game designers, enabling them to improve gameplay, selectively increase visual fidelity, and optimize the distribution of computing resources. The use of saliency maps is currently being advocated as the method of choice for predicting visual attention, crucially under the assumption that no specific task is present. This is achieved by analyzing images for low-level features such as motion, contrast, luminance, etc. However, the majority of computer games are designed to be easily understood and pose a task readily apparent to most players. Our psychophysical experiment shows that in a task-oriented context such as gaming, the predictive power of saliency maps at design time can be weak. Thus, we argue that a more involved protocol utilizing eye tracking, as part of the computer game design cycle, can be sufficiently robust to succeed in predicting fixation behavior of players.

Introduction

In the design of interactive applications, notably games, a recent trend is toward tapping the potential of psychophysics for the purpose of understanding player behavior. Predicting where players are likely to focus could be a very useful tool in the arsenal of game designers. The purpose of such predictive algorithms would for instance be to help game designers decide how and where to allocate rendering resources, leaving as many computing cycles as possible free to carry out a variety of tasks. Verifying game mechanics or improving game AI could be other uses.

A trend is toward the use of saliency maps [START_REF] Itti | A model of saliencybased visual attention for rapid scene analysis[END_REF]], which analyze the imagery presented to the user for low-level features such as motion, contrast, and local luminance levels. Their success lies in the fact that these features are attention-grabbing, making it likely that users focus their gaze on them. Rendering such features at full fidelity and the remainder of the image at lower quality could therefore go unnoticed, while saving precious computing cycles. However, saliency maps work under the assumption that no Figure 1: The design of the maze using a single user-controlled object, a ball shown here in its starting position (left), as well as the maze with five additional distractor balls (right).

specific task is present. This is an important limitation, as it is well known that the presence of a task may alter the way in which users focus their attention [START_REF] Yarbus | Eye movements during perception of complex objects[END_REF][START_REF] Hayhoe | Visual memory and motor planning in a natural task[END_REF].

In this paper, we perform a psychophysical eye tracking experiment, carried out on an easy-to-understand game (shown in Figure 1), which was purpose-designed for psychophysics. Our experiment demonstrates the extent to which gaze behavior may be altered by the presence of a task, even if participants are only instructed to observe a pre-recorded video, i.e. in the absence of a specific task.

Our main result shows that even passive user behavior is task dominated and cannot be statistically distinguished from active gameplay behavior. In our game design, saliency is therefore a very poor predictor for task relevance. Additionally, we experiment with further game variations where attention distracting or attention focusing elements are present. Neither case had a measurable effect on the observed fixation patterns, and therefore on where attention was focused.

On the basis of our experiment we argue that, at least in the context of electronic games, a task is nearly always implied. Rather than relying on saliency algorithms, we therefore advocate the use of eye tracking during the design cycle of electronic games, as we show that computational methods to predict fixation behavior are potentially unreliable.

Background

For the purpose of living in complex environments, humans rely strongly on vision, which consists of two broad components. The first is perception, which is pre-attentive. The second is cognition, which involves high-level processes such as thought, reasoning and memory [START_REF] Palmer | Vision science: Photons to phenomenology[END_REF]]. The delineation between these two is not sharp, and much feedback exists between the two.

When carrying out a task, human visual perception aggregates lowlevel features into higher level representations, thus informing cognitive processes while affecting gaze direction. In turn, cognitive processes can guide perception, for instance by actively focussing attention on a particular part of the scene [START_REF] Yarbus | Eye movements during perception of complex objects[END_REF]].

The focus of attention can thus be classified as a combination of perceptual (bottom-up) and cognitive (top-down) processes [START_REF] Itti | Computational modeling of visual attention[END_REF]]. Low-level features in the environment that trigger pre-attentive focus are called salient. Features that attract attention as a result of performing a specific task are called task relevant.

The first computational models concentrated on modeling gaze behavior using low-level features such as color, intensity, and orientation [START_REF] Triesman | A feature-integration theory of attention[END_REF][START_REF] Koch | Shifts in selective visual attention: towards the underlying neural circuitry[END_REF][START_REF] Itti | A model of saliencybased visual attention for rapid scene analysis[END_REF]]. Such models compute for each pixel of a frame a measure of saliency, the result of which is called a saliency map. However, it was shown that task-related gaze behavior can dominate over saliency [START_REF] Land | The roles of vision and eye movements in the control of activities of daily living[END_REF]]. Per-pixel measures of task relevance have more recently appeared, and these are called task maps [START_REF] Cater | Selective quality rendering by exploiting human inattentional blindness: looking but not seeing[END_REF][START_REF] Navalpakkam | Modeling the influence of task on attention[END_REF].

There are various application areas, including computer graphics, virtual reality, and games, where saliency and task models have been used with varying degrees of success. In graphics for example, these models have been used to inform global illumination algorithms [START_REF] Yee | Spatiotemporal sensitivity and visual attention for efficient rendering of dynamic environments[END_REF][START_REF] Haber | Perceptually guided corrective splatting[END_REF][START_REF] Cater | Detail to attention: Exploiting visual tasks for selective rendering[END_REF][START_REF] Sundstedt | Perceptual rendering of participating media[END_REF].

Experiments using virtual environments, with stimuli presented to the participant via a head-mounted display, have shown that saliency is not a good predictor when performing natural tasks [START_REF] Rothkopf | Task and context determine where you look[END_REF]]. It has also been shown that when saliency is extended with top-down features, gaze behavior can be predicted more accurately [START_REF] Canosa | Seeing, sensing, and selection: modeling visual perception in complex environments[END_REF][START_REF] Lee | Real-time tracking of visually attended objects in interactive virtual environments[END_REF].

Recent studies suggest that in adventure games, fixation behavior can follow both bottom-up and top-down processes [START_REF] El-Nasr | Visual attention in 3d video games[END_REF]. Visual stimuli are reported to be more relevant when located near objects that fit players' top-down visual search goals. In first-person shooter games, gaze tends to be more focused on the center of the screen than in adventure games [START_REF] Kenny | A preliminary investigation into eye gaze data in a first person shooter game[END_REF][START_REF] El-Nasr | Visual attention in 3d video games[END_REF]. In an experiment involving active video game play, nine low-level heuristics were compared to gaze behavior collected using eye tracking [START_REF] Peters | Applying computational tools to predict gaze direction in interactive visual environments[END_REF]]. This study showed that these heuristics performed above chance, and that motion alone was the best predictor. This was followed by flicker and full saliency (color, intensity, orientation, flicker, and motion). Nonetheless, these results can be improved further by incorporating a measure of task relevance, which could be obtained by training a neural network on eye tracking data matched to specific image features [START_REF] Peters | Beyond bottom-up: Incorporating task-dependent influences into a computational model of spatial attention[END_REF].

Our hypothesis is that the success of low-level image features, collected into saliency maps through computational models, depends strongly on the type of task performed. In the context of games, saliency, as measured by contrast, luminance, edges, and the like, may correlate with task relevance if this is designed explicitly into the game. However, this need not be the case: saliency may correlate very poorly with fixation behavior, and therefore with attention. In this paper we present a study that demonstrates this lack of correlation. The consequence is that task relevance, while more difficult to model, should play a larger role in models of gaze behavior than currently assumed.

Psychophysical Experiment

We designed an experiment to analyze the validity of the concept of saliency in the context of games. Our first hypothesis is that each object type in the maze carries different significance with respect to solving the maze. The distribution we compute therefore relates directly to the importance that each object type carries for executing the task. Within the maze, for instance, we expect players to focus more on the one path that leads from the starting position to the target than they would focus on dead ends, and other task irrelevant areas and objects. Note that such a distribution cannot be predicted by current saliency maps.

The second hypothesis is that solving the maze actively does not lead to statistically different behavior from passively viewing a prerecorded game. This is on the basis that the task involved in solving a maze is directly evident, even if no specific instructions are given to solve the maze. Finally, we make a first attempt at assessing the ability of additional features to add to or distract from the task.

Our aim is therefore to design a game which is sufficiently complex to bear relevance to real games, while being controlled enough to allow rigorous analysis. We have found that a good trade-off between playability and visual complexity on the one hand, and the necessity to obtain robust and reproducible data, is afforded by a maze puzzle (Section 3.2). In conditions involving active game play, the task is to navigate a red ball from its starting position to a designated target position. To make the task more challenging, a physics engine simulates various attributes of the ball, including weight, acceleration, and how it bounces off other surfaces.

After a calibration step, performed for each participant individually, we employ a Tobii x50 eye tracking device to record gaze behavior. The raw eye tracking data are subsequently filtered to detect fixations (Section 3.3), which are mapped back to different object types in the scene (Section 3.4). For each participant, we compute the distribution of fixation points over the different object types that can be discerned in the scene (Section 3.5). This distribution is computed for every participant, over a set of different conditions (Section 3.6), before being analyzed using standard techniques (Section 4).

Experimental Setup

The Tobii x50 eye tracker, running at 50 Hz, was placed in front of the display, and angled toward the participant according to the manufacturer's instructions. We employ a chin rest, ensuring that the participants' eyes are 60 cm away from the display, and remain accurately positioned within the detection region of the eye tracker. The game itself was modified so that it can start and stop the eye tracker automatically at the start and end of each trial.

All experiments were carried out on a Pentium 4 PC clocked at 3.46 GHz with 2 GB RAM, and an NVIDIA 8600 graphics card. This setup is sufficiently powerful to run the game (at 16 frames per second), record the game state, as well as drive the eye tracker. The display is a Viglen TS700 LCD display with dimensions of 34 × 27 cm, and a resolution of 1024 × 768 pixels. The setup was located in a blackened room with fluorescent lighting built into the ceiling, positioned to avoid undue reflections off the screen.

Game Design

The design of the maze is shown in Figure 1. Several features are included to help improve the robustness of the results. In many game designs, the camera follows an avatar representing the player, or, in the case of first-person shooter games, the player has full control over which part of the environment is explored. This would not affect our ability to collect data, but does not let us pool the data obtained from different participants within the same condition. Hence, we have fixed the view point, so that each participant has access to the same information during each trial.

The maze itself includes several design features which help us maintain control over the robustness of the results. The enclosing walls ensure that the playing field is located away from the edges of the display, where we have found the accuracy of the eye tracker to be unsatisfactory. The total lengths of the paths in each quadrant of the screen are approximately equal. In addition, we have inserted five separate regions of equal length in which we may place distracting objects. These regions are not connected to the paths accessible by the player to avoid polluting the task with the possibility of collisions. With the exception of the balls, all surfaces are textured with a random achromatic texture which exhibits 1/ f image statistics [START_REF] Deriugin | The power spectrum and the correlation function of the television signal[END_REF]], bringing the imagery statistically closer to natural scenes. The contrast of the texture was reduced to ensure that paths and walls are easily distinguished, thereby not hindering the task. The balls are colored red to make them easily distinguishable from the remainder of the scene. The scene is illuminated approximately uniformly with four light sources, with one located above each quadrant of the playing field.

Fixation Detection

The eye tracker outputs a sequence of time-stamped (x, y) pixel coordinates, representing momentary gaze points, which can subsequently be analyzed. It is typically assumed that if gaze points remain within a small region during a time exceeding 200 ms, then we may speak of a fixation point [START_REF] Salthouse | Stimulus processing during eye fixations[END_REF]]. The premise is that such fixation points correlate with the focusing of attention, and therefore constitute our features of interest [START_REF] Duchowski | Eye tracking methodology: Theory and practice[END_REF]].

The eye tracker is rated to be accurate to within a solid angle of 0.7 degrees. Taking into account the distance between the observer and the display, and its size and resolution (see Section 3.1), this corresponds to an error of ±11 pixels.

We have implemented a fixation filter which detects fixations of 200 ms or longer (i.e., > 10 consecutive sample points), whereby gaze points remain within a circle with a radius of 11 pixels of each other. The output of this filter is a set of time intervals with an associated center point of the fixation.

Item Buffer

We are interested in determining which objects in the scene are focused on, at any time during an experiment's session. The fixation points are therefore mapped to the objects making up the scene. To this end, each scene object is first assigned a unique number that represents a color value. Each frame of the session is then rendered out as an item buffer image, shown in Figure 2 (left), which color codes all scene objects. Each item buffer is also stamped with a start and end time over which it was displayed. Thus fixation points can be subsequently correlated, both temporally and spatially, with the scene objects using these item buffers.

Fixation Binning

The item buffer is used to map fixation points to objects. During a trial, we count how many times each object was fixated upon. In eye tracking experiments, it is common to use relatively sparsely populated environments, because the analysis of fixation points becomes Name Description Closed paths

These are the parts of the maze that are separate from the main paths and may or may not contain distracting elements.

Correct path

The floor and walls of the path that must be traversed to go from the starting point to the designated destination.

Incorrect path

The floors and walls of dead ends. User-contr. ball

The ball under the participant's control.

Distractor balls

Balls not under participant control.

Main walls

The four enclosing the playing field.

Top surface

The top surface of the walls that are not adjacent to the correct path.

Adj. to correct

The top surface of the walls that are adjacent to the correct path. more complicated if several objects are located in the vicinity of a fixation point [START_REF] Lee | Real-time tracking of visually attended objects in interactive virtual environments[END_REF]. A common technique is to simply select the object directly underneath the fixation point.

However, games environments can be cluttered and complex, making such a simple approach error-prone. We have therefore developed a novel approach, taking the distance between pixels and a fixation point into account. With the accuracy of the eye tracker at 0.7 • , and the foveal region of human vision spanning approximately 2 • of visual angle [START_REF] Palmer | Vision science: Photons to phenomenology[END_REF]], the area over which a fixation point bears relevance is a circle with a radius of around 43 pixels (corresponding to a solid angle of 2.7 • ).

Considering frame number f , we assign a non-zero weight w f to all pixels (x p , y p ) in a window of 400 × 400 pixels (large enough to be accurate, and small enough to be computationally efficient) according to their distance to the fixation point (x f , y f ):

w f (x p , y p ) = exp   -(x f -x p ) 2 + (y f -y p ) 2 2σ 2   (1)
where σ = 43 pixels. With the aid of the item buffer, the weights computed for frame f are then added to the weights of the different objects. This process is repeated for all frames for which a fixation point exists1 . After normalization, necessary to account for the fact that each trial may last for a different amount of time, this produces a weight ŵi associated with object i.

In the design of the maze, different objects may have the same relevance to the task. For instance, the walls are made up of different segments. Therefore, we group objects playing a similar role into object classes. The classes we distinguish are based upon their anticipated relevance to solving the maze (described in Table 1). A color-coded image showing the location of each of these elements is shown in Figure 2 (right). After classification, each participant produces a normalized distribution of fixations per object class. This set of distributions is then subjected to further analysis, as discussed in Section 4. We first describe the collection of conditions used in our experiment.

Conditions

The four conditions are developed to test the distribution of fixation points over object classes in active and passive tasks, as well as in the presence of distracting and focus-enhancing elements. In the first condition, participants were asked to navigate the ball to the destination position, located in the top left corner of the playing field. No distracting objects or sound is present in this condition.

The second condition adds five static distracting spheres to the five separate areas. The task was otherwise identical to the first condition. In the third condition, we additionally add two types of sound, both connected to the sphere under participant control. A continuous sound change pitch in association with the current speed of the ball. A transient sound indicates collisions. It is anticipated that these sounds help with the control of the sphere, rather than distract.

For the remaining condition, the game was modified to enable its state to be recorded. This is achieved by writing all events (such as object movements) which affect the game state to a file. This is less data and compute intensive than encoding a video of the game. It can therefore proceed in parallel to gathering gaze data from the eye tracker, as well as driving the game itself without affecting the responsiveness of the game, or its frame-rate.

Further, the recorded game data can be played back in real time, with the same frame-rate as the original recording, yielding an exact replica of an earlier trial. Each participant in the second condition was recorded in this manner, producing data for the fourth condition. This last condition consists of participants observing pre-recorded game data, without having been set a task. As the same number of participants were used in each condition, each participant in the fourth condition was exposed to a different recording obtained in the second condition.

Participants

Forty participants (36 men and 4 women, age range 18 -42) with normal or corrected to normal vision were recruited for our experiments. As the eye tracker occasionally loses calibration during a trial, especially for participants wearing glasses, trials which have clearly produced unreliable data were replaced with new ones, for which new participants were recruited. No participant was exposed to more than one condition, so that learning effects between trials could be avoided. As a result, each condition was evaluated by exactly ten participants. The participants were asked to read instructions before the start of the experiment. The instructions for the active and passive conditions are shown in Appendix A.

Results

First, we wanted to analyze whether habituation plays an important role in the analysis. It may be anticipated that participants learn the path to the destination first, and then spend the remainder of the trial steering the ball. It is equally likely that participants frequently alternate between navigation and steering. To assess whether participants change their fixation behavior during the course of a trial, we have computed the number of fixations occurring per second for each participant and each condition. For each point t on the time line, we counted the number of fixations in a window spanning t ± 15 seconds (except near the start and end of trials, where the window size was reduced to accommodate for the lack of data).

The results are plotted in Figure 3, and indicate that fixation behavior for each participant is relatively stable over time. As a result, we infer that participants do not change their solution strategy during the course of the trial. It is more likely that most participants use a strategy whereby navigation and steering are alternated. The average trial duration for the four conditions was 72 ± 12 seconds.

Given that the number of fixations per second is stable over the course of each trial, we conclude that habituation and learning ef-fects do not play a major role, and therefore do not complicate our analysis. There is also no significant difference between the four conditions in the total durations of the trials (F(3, 36) = 1.767, p = 0.171), the number of fixations (F(3, 36) = 0.291, p = 0.832), or the total fixation durations (F(3, 36) = 1.092, p = 0.365).

The distribution of fixations over the different object classes for each participant and each condition is shown in Figure 4. We note that these distributions are markedly different from the distribution one would obtain by counting the number of pixels that are covered by each object type (shown in Figure 5). Thus, we conclude that none of the results presented next can be explained by random fixation behavior.

The first important observation is that the number of fixations attracted by the correct path and its neighboring adjacent to correct form the majority of all fixations points, regardless of condition. This can also be seen by plotting the fixation points on top of the maze stimulus, as shown for a representative participant of condition 2 in Figure 6 (a). The fixation points in this figure roughly map out the correct path between start and destination points, showing that the task of solving the maze has a strong impact on where fixation points are likely to occur. The fixation points for a representative participant of condition 4 are shown in Figure 6 (b).

The raw data of Figure 4 were subjected to a one-way analysis of variance (ANOVA), the null hypothesis being that no statistically significant differences between these distributions are found. All preconditions for ANOVA regarding independence and normality were fulfilled. By using the four conditions as groups (our independent variable), we found that the test of homogeneity of variances was met for all the object classes (our dependent variables), bar the distractor balls (p = 0.001).

Tukey's HSD post hoc test shows that there are statistical differences between condition 1 and the other conditions for this object class (p = 0.000). This is explained by the fact that no distractors exist in one of the conditions (their weight was set to 0 in condition 1). If we exclude condition 1, then the test of homogeneity of variances was met for all object classes. The ANOVA values for both tests are shown in Table 2.

We find that the null hypothesis is supported for all object classes over all four conditions, with the exception of the distractors class.

Once again, if we exclude condition 1 from the analysis, then the null hypothesis is supported for all object classes in the three remaining conditions. Thus, we have found no statistical differences in fixation behavior of participants in each of the four conditions.

There are several implications of this result. First, comparing conditions 2 and 4, we see that there is no significant difference in the distribution of fixations, independent of whether the game was actively played or passively observed. This result is important, because it proves that the absence of a task (other than simple observation) does not guarantee that participants will not assume a task. For this particular game, free viewing of a pre-recorded game encourages the observers the solve the task "in their heads". As in this case the distribution of fixation patterns continues to follow that of the active condition, we conclude that any computational metric that assumes the absence of a task cannot be applied. Importantly, this includes saliency algorithms.

Second, comparing conditions 1 and 2, we see that the addition of five distracting objects did not have any effect on task performance: the distribution of fixations over all object classes remains the same, with the exception of the distracting objects themselves which were absent in condition 1. Once more, this result would not be predicted by saliency algorithms, which would indicate a difference between these conditions due to the contrasting color of the distracting balls. Third, conditions 2 and 4 enable us to assess the influence of sound, which was present in condition 4. Here, rather than distract, the sounds were designed to support the steering component of the task. We would anticipate this to have an influence on the distribution of fixations over object classes. However, we did not find such an influence, showing that the task was not facilitated by including an auditory indication of speed and the occurrence of collisions.

Comparison with Saliency Maps

While saliency maps are currently advocated as a useful tool to understand fixation behavior in games [START_REF] Peters | Applying computational tools to predict gaze direction in interactive visual environments[END_REF], on the basis of our experiments, we are now able to argue that such algorithms can only be used with the utmost care, and possibly in the context of games should be avoided.

In a sense, the design of the maze forms a worst-case scenario for saliency maps. These algorithms determine saliency on the basis of low-level image features. In our design, these image features are more or less evenly distributed across the playing field. A saliency map will therefore have a substantially uniform distribution, as shown in Figure 6 (c), which plots the saliency map computed using [START_REF] Itti | A model of saliencybased visual attention for rapid scene analysis[END_REF] algorithm. Comparing this with the actual distribution of the fixation points (Figure 6 (a and b)), we see that the actual fixation pattern is not predicted by the saliency algorithm, even if no task is specified.

The number of times that the fixation point was in the vicinity of the ball that was being controlled was unexpectedly low (see Figure 4). As this was the only object that moves, we cannot attribute the differences between the outcome of our experiment and the saliency maps shown in Figure 6 (c) to the fact that we did not take motion into account for the computation of the saliency map. the saliency map (see Figure 6 (c)) and the eye tracking results.

Comparing conditions 1 and 2, the distracting objects do not alter the distribution of fixations over object classes. However, their contrasting color does produce a pattern of saliency, which is not supported by our experiments.

We assume that for many games, designers frequently adjust the amount of detail, motion, contrast, luminance, etc., to direct players toward their goal: objects more relevant to the task to be accomplished may exhibit more motion, contrast, etc., than objects with lower task relevance. This may explain the fact that in some games, saliency maps are able to predict fixation behavior better than chance [START_REF] Peters | Applying computational tools to predict gaze direction in interactive visual environments[END_REF]]. However, our experiment shows that this correlation does not necessarily generalize to other game scenarios. Finally, there is a possibility that the predictive power of saliency maps is in essence a measure of how much game designers make their task-relevant objects stand out from the background.

Conclusions

The aim of our work is to assess the distribution of fixation points over a set of pre-determined object classes. The object classes were chosen to correspond with the expected relevance to the task of solving a maze and steering a ball through it. We find that the distribution of fixation points deviates significantly from those which would be predicted by current saliency algorithms. Moreover, no difference was found between active game play and passive observation.

We attribute this result to the fact that tasks and objectives in games are typically designed to be easily apparent or understandable. This means that even if no task is specified, observers will assume a task. Related work have shown that when an observer is watching a block stacking task, the gaze prediction is highly similar to when they perform the task themselves [START_REF] Flanagan | Action plans used in action observation[END_REF].

Whereas existing studies explored average-case behavior by using existing games [START_REF] Kenny | A preliminary investigation into eye gaze data in a first person shooter game[END_REF][START_REF] El-Nasr | Visual attention in 3d video games[END_REF][START_REF] Peters | Applying computational tools to predict gaze direction in interactive visual environments[END_REF], i.e. without control of salient features in the game design, our work has for the first time created a worst-case scenario for saliency algorithms. As our maze design shows, actual fixation behavior can deviate by an arbitrary amount from the predicted saliency. Further, objects which a saliency algorithm would flag as distracting do not necessary impede task efficiency.

We hypothesize that the correspondence between fixation behavior and saliency algorithms depends strongly on the efforts of the designer to impose salient characteristics to objects that happen to be task-relevant. This may explain the above chance performance of saliency algorithms in existing games [START_REF] Peters | Applying computational tools to predict gaze direction in interactive visual environments[END_REF]]. As a result, in the context of game design we would argue for caution when using saliency algorithms.

Instead, the direct use of eye tracking during the design of games can be extremely valuable. We have shown that a relatively straightforward modification to the game engine (namely tagging each object with a color-coded identifier), is sufficient to map fixation points back to objects. While eye tracking sessions during actual game design by necessity will be less controlled than our experiment, simply accumulating fixations over different object classes may prove to be a fruitful approach in understanding where game players focus their attention. Such information cannot currently be extracted from an analysis of low-level features alone.

A Appendix

For participants in conditions 1, 2, and 3 (all actively playing the game), the instructions are as follows:

"Please read this information carefully: The task in this experiment is to play a game. The goal of the game is to guide a ball through a maze. The finish point of the maze is indicated with a cross, as shown below. When you reach the finish point the game will stop and the experiment is over. The ball can be moved in four directions by using the keys shown below. Please ask now if you have any questions."

A S D

W

An example maze was also shown to the participants. For participants in the passive viewing condition, the instructions are:

"Please read this information carefully: In this experiment you will be shown an animation. Please watch it. If you have any questions please ask now." . Image (c) shows a saliency map of the maze, computed using a saliency algorithm [START_REF] Itti | A model of saliencybased visual attention for rapid scene analysis[END_REF]] and (d) shows our distribution map over object classes generated from the ten participants in condition 2.

Figure 2 :

 2 Figure 2: Item buffer showing all color coded objects which enable us to relate fixation points to objects and object classes (left), as well as the different object classes in our experiment (right).

Figure 3 :

 3 Figure 3: The number of fixations per second for each participant.

Figure 4 :

 4 Figure 4: The output of our fixation classification algorithm.

Figure 6 Figure 5 :

 65 Figure 6 (d) shows the relative importance assigned to the object classes based on the eye tracking results from condition 2. Note that the unevenness of the importance distribution suggests that our initial division of objects into object classes was highly correlated with the task performed. Such correlation does not exist between

Figure 6 :

 6 Figure 6: The location of fixation points for a representative participant of condition 2 (a) and the passive condition 4 (b). Image (c) shows a saliency map of the maze, computed using a saliency algorithm[START_REF] Itti | A model of saliencybased visual attention for rapid scene analysis[END_REF]] and (d) shows our distribution map over object classes generated from the ten participants in condition 2.

Table 1 :

 1 Description of object classes. The color coding is consistent with Figure 2 (right).

Table 2 :

 2 Statistics for each object class. The second column shows the results if all four conditions are included. Due to the absence of distractor balls in condition 1, we have repeated the analysis for conditions 2, 3, and 4 in the last columns.

Note that not all frames have an associated fixation point, as saccadic eye movements are of non-zero duration.

Acknowledgments

This research was sponsored through the European Union within the CROSSMOD project (EU IST-014891-2). We would also like to thank Daniel Scherzer, Heinrich Fink, Fr ühstück Anton, Matthias Bernhard, Isabelle Viaud-Delmon and Clara Suied.


David Grelaud

Nicolas Bonneel

Michael Wimmer Vut

Manuel Asselot

George Drettakis

Efficient and Practical Audio-Visual Rendering for Games using Crossmodal Perception

Keywords:

Interactive applications such as computer games, are inherently audio visual, requiring high-quality rendering of complex 3D audio soundscapes and graphics environments. A frequent source of audio events is impact sounds, typically generated with physics engines. In this paper, we first present an optimization allowing efficient usage of impact sounds in a unified audio rendering pipeline, also including prerecorded sounds. We also exploit a recent result on audio-visual crossmodal perception to introduce a new level-ofdetail selection algorithm, which jointly chooses the quality level of audio and graphics rendering. We have integrated these two techniques as a comprehensive crossmodal audio-visual rendering pipeline in a home-grown game engine, thus demonstrating the potential utility of our approach.

Introduction

Modern interactive applications, such as computer games, are inherently audio-visual. The sounds in such environments are both pre-recorded sound effects (e.g., motor sounds, ambient sounds such as wind or rain, animal sounds etc.), or can be synthesized on the fly as a result of a physical event such as an impact. We typically want to apply advanced processing to these sounds, such as spatialization. Impact sounds are an important special case, particularly in action games, since they are often responsible for the generation of a large number of sound sources, typically occurring in a short period of time. Such sounds are usually generated by physics-based interaction of objects, using physics engines (e.g., PhysX http//www.ageia.com).

Recently there have been significant advances in contact sound synthesis [van den [START_REF] Van Den Doel | Modal synthesis for vibrating objects[END_REF][START_REF] Raghuvanshi | Interactive sound synthesis for large scale environments[END_REF][START_REF] Raghuvanshi | Physically based sound synthesis for large-scale virtual environments[END_REF], and their combined use with recorded sounds in a full scalable 3D audio pipeline [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]]. In addition, in some of these recent works, crossmodal perception has been used to improve audio clustering [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]], or for scheduling of impact sounds based on perception of asynchrony [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]].

However, the combination of impact and pre-recorded sounds involves a non-negligible overhead due to energy computation, thus mitigating the potential gains from using the techniques of audio clustering, masking and scalable processing developed in [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF][START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]]. Indeed, masking effects potentially allow the culling of a large fraction of sound sources which will * e-mail: FirstName.LastName@sophia.inria.fr Using our approach, we can render sounds for larger number of impact sounds in real time.

We use crossmodal perception to jointly select the level of detail for sound and graphics. not be computed. In the case of impact sounds which are costly to compute, such masking would allow a large speed-up.

While crossmodal perception has been successfully used to improve audio rendering, to our knowledge there is no previous method which uses crossmodal perception simultaneously for graphics and audio rendering, and in particular, level-of-detail selection. This could be beneficial since our experience of virtual environments is inherently bimodal, particularly if there is mutual influence of sound and visuals on perception which we can exploit algorithmically. Such mutual influence has recently been reported for our perception of materials [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF].

In this paper we address the two issues discussed above: We present a new fast approximation to energy estimation for impact sounds, allowing us to fully exploit the benefits of the scalable processing pipeline of [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]]. We also introduce a joint audiovisual level-of-detail selection algorithm based on the aforementioned study on material perception [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]]. We also show how to integrate high-quality "attacks" (e.g., the onset of an impact sound) into the full audio processing pipeline, and present a full crossmodal audio-visual processing pipeline which is suitable for games. We have integrated our results in an experimental, but quite complete, game engine [START_REF] Chiu | Penta G -A Game Engine for Real-Time Rendering Research[END_REF], putting together all of these contributions and showing their applicability to computer games.

Previous Work

The work we report here is related to 3D audio rendering and crossmodal audio-visual rendering. There is a vast body of work in the perception literature on crossmodal effects, but it is beyond the scope of this paper to discuss it extensively. Please see [START_REF] Spence | Crossmodal Space and Crossmodal Attention[END_REF] for a review. In this section, we briefly discuss closely related previous work on 3D audio rendering and synthesis for complex environments followed by a quick overview of the most related perceptually based techniques and crossmodal audiovisual algorithms including those based on perception.

3D Audio Rendering and Synthesis for Virtual Environments

Rendering spatialized sound for 3D virtual environments has been a subject of research for many years. Techniques developed permit real-time rendering of sound reflections [START_REF] Funkhouser | A beam tracing method for interactive architectural acoustics[END_REF][START_REF] Funkhouser | Real-time acoustic modeling for distributed virtual environments[END_REF]] and [START_REF] Lokki | Creating interactive virtual auditory environments[END_REF], mainly for prerecorded sounds. In [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF], sound sources are first culled using perceptually based auditory masking, then clustered to minimize computations. A scalable processing approach was added in [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]], allowing a continuous tradeoff of quality vs. computation time.

While recorded sounds are the most widespread in applications such as computer games, physically based synthesis of impact sounds [van den [START_REF] Van Den Doel | The sounds of physical shapes[END_REF]] often provides much better results. Various techniques have been developed to optimize this approach, notably recursive evaluations [van den Doel and Pai 2003] and mode-culling [START_REF] Raghuvanshi | Interactive sound synthesis for large scale environments[END_REF] which is very effective in reducing the computational overhead. Recently [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]] use the sparsity of modes in the frequency domain to accelerate computation.

Perceptually-based methods for 3D audio and graphics

In recent years there have been many efforts to exploit perception to reduce computation for interactive virtual environments, ultimately with the goal to "render only what you can perceive". A survey of the early work in this domain can be found in [START_REF] Luebke | Level of Detail for 3D Graphics[END_REF]. Examples of such work in graphics include use of visual differences predictors for ray-tracing acceleration (e.g., [START_REF] Ramasubramanian | A perceptually based physical error metric for realistic image synthesis[END_REF][START_REF] Myszkowski | The Visible Differences Predictor: applications to global illumination problems[END_REF]]), or perceptually based level-of-detail (LOD) control [START_REF] Luebke | Perceptually driven simplification for interactive rendering[END_REF][START_REF] Williams | Perceptually guided simplification of lit, textured meshes[END_REF]]. In audio, [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF]] uses perception to optimize masking and clustering, as discussed above.

Material perception has received quite some attention in recent years in computer graphics. Notably, [START_REF] Vangorp | The influence of shape on the perception of material reflectance[END_REF]] studies the effect of geometry shape and lighting on perception of material reflectance, and [START_REF] Ramanarayanan | Visual equivalence: Towards a new standard for image fidelity[END_REF]] introduce the concept of visual equivalence, based on material properties, geometry and illumination. Our crossmodal level of detail selection approach will be based on material perception.

Crossmodal Perception Methods To our knowledge, there has been little work on exploiting crossmodal audiovisual perception to improve audio and graphics rendering. The study in [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]] indicated that it is better to have more audio clusters in the visible frustum; this was used as a new metric for audio clustering.

In [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF], tolerance to delays of audio with respect to the corresponding visual event was used to perform time-delay scheduling of impact sounds, with significant gains in performance and resulting audio quality.

A recent study [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]] indicates a mutual influence of graphics and sound on how users perceive material. In this study, users compared approximations (spherical harmonics (SH) for BRDF rendering, and modal sound synthesis) with a highquality solution. The users were asked to rate the material similarity of the approximation compared to the high-quality audio-visual rendering material. We will be developing a crossmodal algorithm to control both audio and graphics levels of detail, based on the results of this study.

Efficient Energy Computation for Impact Sounds

The use of a combined audio rendering pipeline for both recorded and impact sounds has a high potential for benefit, since we get significant speed benefits using masking and clustering [START_REF] Tsingos | Perceptual audio rendering of complex virtual environments[END_REF]], and we have a smooth quality/cost tradeoff using scalable processing [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF]]. An initial approach integrating this pipeline with impact sounds was presented in [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]. However, the full potential benefit of this combined approach is hindered by the relatively high cost of the computation of impact sound energy, which is required both for masking and scalable processing.

For recorded sounds, the energy is precomputed for each sound file to avoid on-the-fly computations. In the case of impact sounds, this energy cannot be precomputed since sounds are generated on-thefly during the simulation. Online energy computation based on the values in the current audio buffer would be costly and pointless, since the goal is to avoid computing the audio signal if it is masked by other sounds. We thus require a quick estimate of energy without actually computing the impact sound.

Energy Computations for Masking and Scalable Processing

We first briefly summarize the energy computations required to use impact sounds with the combined pipeline. Computations occur at two instants in time: at impact and at each frame. In [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF], at each impact the total energy of each mode is efficiently computed. At each frame, the energy of the sound over the frame is then estimated using scalar products of a subset of the modes of the sound. This approximation was shown to work well for impact sound processing but still requires much computational effort per frame.

The solution we show here, together with the integration of attack processing in clustering (Sect. 5) allows the full and efficient integration of high-quality impact sounds into a practical audio processing pipeline.

An Efficient Energy Approximation for Impact Sounds

We assume that the power (energy per unit time, that we will call "instant energy") of an impact sound decreases exponentially:

E(t) = Ae -αt (1)
Thus if we know the parameters A and α of this exponential, we can easily compute an approximation of the energy in a given frame, by analytically integrating over the desired interval. The two unknown parameters A and α satisfy two equations concerning the energy:

E Tot = ∞ 0 Ae -αt dt (2) E Part = T 0 Ae -αt dt (3)
Thus, given the total energy E Tot of the sound and a partial energy E Part , we are able to exactly determine parameters A and α. These parameters are thus given by the following equations:

α = - 1 T log(1 - E Part E Tot ) (4) A = α E Tot (5)
The energy Es for each frame is computed by integrating Eq. 1, which represents a negligible computation cost per frame:

Es = - 1 α .(E(t + ∆t) -E(t)) (6)
However, computing these values requires the knowledge of a partial energy E Part for the system to be solved. This can be achieved efficiently, also by computing scalar products. We found that the scalar product S of two modes m 1 and m 2 taken from 0 to T can be easily computed via the expression of the scalar product Q of those two modes taken from 0 to infinity:

S =< m 1 , m 2 > = T 0 e -a1t sin(ω 1 t)e -a2t sin(ω 2 t)dt = (1 -e -T (a1+a2 )Q (7) with Q = ∞ 0 e -a1t sin(ω 1 t)e -a2t sin(ω 2 t)dt = 2(a 1 + a 2 )ω 1 ω 2 ((a 1 + a 2 ) 2 + (ω 1 + ω 2 ) 2 )((a 1 + a 2 ) 2 + (ω 1 -ω 2 ) 2 ) (8)
The partial energy E Part is computed by summing the pairwise scalar product of a subset of highest energy modes, similar to the computation of the total sound energy E Tot previously used. A typical value for T is 10ms. This optimization process is performed only once per impact, and does not have to be repeated per frame. Also, since Q is already computed to get the total energy of the sound, the only additional per-impact cost is the computation of an exponential function. Only the simple Eq. ( 6) has to be computed per frame for each impact sound, which represents a negligible overhead. Our new approximation thus allows faster computation of the instant sound energy which is used for masking and budget allocation.

Numerical Evaluation and Speedup

We performed a numerical evaluation of our energy estimation approach on two sequences. These scenes are respectively a can and a crate falling on the ground (see Fig. 3).

We computed the exact energy Es of each impact sound in each frame using all the modes, and we plotted this compared to our approximation in Fig. 2, over 86 frames for both sequences. As we can see, our approximation is overall accurate, with an average L1 relative error of 24% for "Cans" and 27% for "Crates". Each can had a mesh of 415 elements, and used 113 modes; and for each crate there are 497 elements and 362 modes.

If we use the approximation of [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF], the average cost of the energy is 0.2 ms per frame for the "Cans" sequence and 0.2 ms per frame for "Crates". In contrast, our approximation requires about 1µs and 0.8µs respectively for each sequence, corresponding to a speedup of 200 and 250 times. In addition, for sounds with large numbers of elements, which are required for sounds with fast decaying, high frequency modes, a higher number of modes is required for the approximation. Given the low cost of our approach, we can thus significantly improve the energy approximation without significant overhead in this case.

Crossmodal Audio-visual LOD Selection

As mentioned in Section 2, we want to develop a crossmodal audiovisual level-of-detail selection algorithm based on the perceptual study reported in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]]. We use the main result of that study, concerning the mutual influence of sound and graphics on how users perceive material. While this approach only applies when objects are actually sounding, i.e., during impacts, this is when a large number of sound events occur, resulting in a significant additional load to the system. As a result, there is strong potential for gain using our selection method. Indeed, since audio rendering is much cheaper than spherical harmonic (SH) lighting, we can take advantage of the crossmodal influence of audio and vi-suals on the perceived quality, by reducing visual LOD and increasing audio LOD. According to the study, in this case the perceived similarity with a reference rendering will remain very high, despite the significantly reduced computational cost.

In the experiment, the LOD used are the number of bands of the spherical harmonics for the rendering of the BRDF using an environment map, and the number of modes used for modal sound synthesis. Also, two materials were used (gold and plastic), and two different objects were studied (bunny and dragon model). For the case of the golden bunny, we can plot the results as shown in Fig. 4.

In the context of an audio-visual 3D rendering engine, we can inter- pret and use these results in several ways. One option would be to fix the "target quality" and then minimize the cost or computation budget used to render both the graphics and the sound. A second option is to allocate a fixed budget for both graphics and sound, and maximize the perceived quality. We have chosen the latter option, since BRDF rendering with a large number of spherical harmonic bands can be very costly; in the presence of several such objects even the joint LOD with minimum cost predicted using this model could be unacceptably high.

Crossmodal Audio Visual LOD Metric

We perform an optimization step to determine the appropriate graphics and visual LODs once per frame. The constraint is determined by the actual CPU/GPU cost of rendering audio modes and shaded pixels. For graphics, this is the cost of the pixel shader used to query the SH textures and compute the dot product lighting.

The complexity of rendering audio modes is linear in the number of modes, while the cost of rendering the SH lighting is quadratic in the number of spherical harmonic bands, and linear in the number of displayed pixels. We thus use the following cost estimation:

C AV = C M M +C S S 2 P, ( 9
)
where M is the number of modes, S is the number of spherical harmonic bands, P is the number of shaded pixels, and C M and C S are the costs of rendering respectively one audio mode and one band of one SH-shaded pixel. The values for C M and C S can be measured once for every hardware setup. In our case C M = 5.23µs/mode, C S = 0.0368µs per SH coefficient and per pixel. Note that rendering S bands requires computing S 2 such coefficients. These values

were measured with a GeForce8800 GTX, and a 2.3GHz Intel Xeon CPU. To efficiently determine the number of shaded pixels, we use an early-depth pass, deferred shading and occlusion queries. The above expression shows the quadratic increase in SH cost and the linear cost in the number of modes.

The target cost C T is a user-defined parameter typically depending on the hardware setup. We share this parameter across all objects, i.e., if we have N objects, the budget of C T /N ms is assigned to each object in the scene. The audio-visual perceived quality is determined by an interpolation of the tabulated values given in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]], Fig. 7. We evaluate the cost function C AV for each combination of 13 SH bands (0 to 12) and 6 mode budgets given in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]]. We choose the combination which results in the highest "perceived quality" as determined by the values reported in the study. As an example, for a target budget C T /N of 1.4ms, used to render an object occupying 1000 pixels on the screen, we can visualize the operation LOD choice operation in Fig. 4. In this example, we will choose 6 SH bands and level 5 for the modes.

We perform this operation for each object, and select the number of SH bands and the number of modes resulting in the highest quality, while respecting the target cost.

With this approach we determine a budget for impact sounds and spherical harmonics. The audio LOD is used for the current audio frame, and the spherical harmonic LOD is used in the next visual frame. Visual blending is performed in the pixel shader by progressively adding terms in the dot product (without splitting an SH band) during 0.5ms.

Evidently this LOD selection is valid only when the objects are actually making a sound, in this case when impacts occur. When the objects are not sounding, we use a fixed LOD for graphics, i.e., 5 spherical harmonic bands. The choice of 5 bands is also based on the results presented in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]], since we can see that perceived quality with this approximation is not much different from the perceived quality obtained with the highest visual LOD, for all sound LODs. The crossmodal LOD selection mechanism is applied at the first audio frame of an impact, and maintained for a short period of time (typically 7 seconds).

A General Crossmodal Audiovisual Pipeline

The fast energy estimation and the crossmodal audiovisual LOD selection pave the way for us to introduce a general crossmodal rendering pipeline, in which we use crossmodal perception for combined graphics and sound resource management.

Attack processing with Clusters

In [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]], special processing is performed to provide high-quality rendering of the "attacks" or onset of impact sounds. This involves a specially designed windowing operation in the first frame of each attack. This approach was however not integrated into the full pipeline.

The inclusion of high-quality attack processing in the pipeline is important to allow high-quality audio rendering. In each cluster, we have a set of recorded sounds and a set of impact sounds. Using the improved attack processing of [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF], we need to have one frequency domain buffer for attacks, and one for both recorded and impact sounds. Budget allocation for scalable processing is performed only in the latter buffer. Windowing operations are performed separately in each buffer, and an inverse FFT is computed for each impact sound buffer in each cluster. Recorded sound buffers are processed together, resulting in a single inverse FFT. Attack buffers and the recorded/impact sound buffers are then correctly windowed in the time domain to produce the final sound.

The additional overhead of this approach is thus one inverse FFT per audio channel.

General Crossmodal AV Pipeline

We have implemented a complete perceptually based audio-visual rendering pipeline. We have also included the crossmodal audio clustering metric [START_REF] Moeck | Progressive perceptual audio rendering of complex scenes[END_REF], which assigns more audio clusters to the viewing frustum, the crossmodal scheduling approach of [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]] which significantly improves performance, and the audio-visual LOD selection introduced here.

At each frame, both for audio and graphics, we evaluate the parameters used for each of these crossmodal algorithms, and set the appropriate values for clustering, impact sound scheduling and AV LOD selection.

Results

We have implemented the general AV pipeline into a home-grown "production level" game engine developed in our institutions [START_REF] Chiu | Penta G -A Game Engine for Real-Time Rendering Research[END_REF]]. All the examples shown in this paper and the accompanying video are taken from this engine.

Energy approximation and attack integration

The combination of the integration of attacks in the pipeline and the energy approximation now make it possible to have more impact sounds with higher quality in complex game-like settings. In the accompanying video, we show an example game play, in which we have 7 recorded sounds (toys, stereos, door, cashier sounds etc.) and a large number of impact sounds, with a peak of 712 impacts just after the piles of boxes and cans are knocked over.

We first show the sequence using the method of [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]. The cost of the previous energy estimation method results in low quality audio processing, which is clearly audible. If we compare to a similar sequence (we do not have exact path replay in our implementation), using our energy approximation we see that the quality achievable by the scalable processing is significantly improved.

We next show the same sequence first without attack processing in the clusters and using the attacks, both with the energy approximation. As we can hear, the quality of the impact sounds is audibly better using the attack processing approach; thanks to the integration with clustering, the overhead is minimal. Rendering this scene without clustering, given the number of sounds, would be impossible in real time.

We provide a detailed breakdown of the costs of the different stages for the above sequence, using both the energy approximation and the integration of attacks in clusters (see Table 1). In the scenes we tested, we achieve masking of around 60% of the contact sounds. Crossmodal Material LOD We have implemented our crossmodal LOD selection approach in the same game engine [START_REF] Chiu | Penta G -A Game Engine for Real-Time Rendering Research[END_REF]]. In the accompanying video we show a sequence containing two piles of golden bunnies which are accidentally knocked over.

In Fig. 5 we illustrate the choice of visual LODs for two frames of the sequence. We used a budget of 100 ms shared by all SHshaded objects. Note that this budget will be shared by the GPU for SH lighting computations and the CPU for modes generation, both running in parallel. This budget does not reflect the exact running time of the simulation but is used as an indication.

For comparison, we computed an appropriate fixed audio-visual level of detail, corresponding to the perceived similarity to the reference rendering, as predicted by the perceptual study. For example, if the average choice of LOD in the sequence represents 90% similarity to the reference, we will choose 90% of the spherical harmonic bands and the modes used for the reference.

On average across the entire sequence, the model of [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]] predicts that we have a quality level of 90.6%, i.e., the rating of perceived similarity to the high quality rendering. The computation in this sequence is 44% faster; on average we achieved 26 fps using our approach and 18 fps using the equivalent quality with fixed LOD.

Full crossmodal approach There are three main differences with [START_REF] Bonneel | Fast modal sounds with scalable frequency-domain synthesis[END_REF]]: we integrate high-quality attack processing with clustering, the fast energy approximation, and the crossmodal LOD. We integrated these effects with the previous crossmodal metrics (clustering and scheduling) to provide a unified crossmodal pipeline.

The final sequence of the video shows all the elements working together. We selectively turn on and off some of the features, illustrating the various benefits obtained.

Discussion and Conclusion

We have presented a complete perceptually inspired crossmodal audio-visual rendering pipeline for games. We introduced an approximate energy estimation method for contact sounds, and showed how to integrate high-quality attacks with clustering. We also presented a crossmodal LOD selection approach, which is to our knowledge the first algorithm which jointly chooses audio and graphics levels of detail, based on crossmodal perception. We believe that the integrated method presented here offers a practical and useful pipeline for audio visual rendering, which can be useful for games or other similar applications.

Our crossmodal LOD selection approach uses spherical harmonic rendering, since we base our algorithm on the study presented in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF] which used this approach. We expect that similar results can be obtained with other approaches for environment map rendering (e.g., zonal harmonics [START_REF] Sloan | Local, deformable precomputed radiance transfer[END_REF] or sampling techniques [START_REF] Agarwal | Structured importance sampling of environment maps[END_REF]).

The potential for computation gains can be significant, especially during events which result in large numbers of impacts. Nonetheless, the work reported here is only a first step. In particular, we need to determine how well the results of [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF]] generalize to other kinds of materials and objects, and to determine how perceived quality ratings are affected by the more realistic game settings shown here. Our intuition is that perceived quality should actually be higher in the realistic setting compared to the experimental condition. We also expect that a generic material model, based for example on some general material categories, could be applied in a more general setting. One way to do this would be to sample a perceptually uniform material space such as [START_REF] Pellacini | Toward a psychophysically-based light reflection model for image synthesis[END_REF]] and interpolate the results for any material. Also, the environment map used for this game setting was different from the one used in the experiment in [START_REF] Bonneel | Bimodal perception of audio-visual material properties for virtual environments[END_REF], since this new environment is an indoors environment, contrary to the previous experiment. It has been shown that the environment lighting influences the perception of materials [START_REF] Fleming | Real-world illumination and the perception of surface reflectance properties[END_REF]), and a further BRDF prediction could include this parameter. In our case, an indoor environment makes the golden bunnies appear a bit more diffuse than with a higher frequency environment.

Figure 1 :

 1 Figure 1: Illustration of our results.

  Figure2: Plot of the instant energies computed with our approach (red),[Bonneel et al 09] (cyan), and the reference (blue), over the length of an impact sound on two scenes. Also shown, the reference energy computed with 10 modes, simulating what our approach computes. Note that our approximation is much more efficient. Left: "Cans" sequence, right "Crates" sequence.

Figure 3 :

 3 Figure 3: Two frames from the test sequence used for the fast energy estimation evaluation (see text).

Figure 4 :

 4 Figure4: Material rating depending on sound and visual LOD, and the 1.4ms threshold cost (threshold for P = 1000 shaded pixels -the size of the object on the screen). Golden bunny.

Figure 5 :

 5 Figure 5: (a)-(b) Two frames of our test sequence running at 26 fps using crossmodal LOD manager. (c)-(d) Two similar frames using highest LOD running at 18 fps.

Table 1 :

 1 Time in ms for each stage of the unified pipeline

	Stage	Time (ms)
	Masking	3.339E -3
	Clustering	4.652E -3
	Impact sound synthesis 2.759E -3
	Energy	0.824E -3
	Scalable mixing	1.260E -2

Acknowledgments

This research was funded by the EU FET project CROSSMOD (014891-2 http://www.crossmod.org). We thank Autodesk for the donation of Maya, Fernanda Andrade Cabral for modeling and the anonymous reviewers for their helpful suggestions.


Marcio Cabral

Peter Vangorp

Gaurav Chaurasia

Emmanuelle Chapoulie

Martin Hachet

George Drettakis

A Multimode Immersive Conceptual Design System for Architectural Modeling and Lighting

ou non, émanant des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

INTRODUCTION

Creating and manipulating 3D models is one of the hardest tasks in virtual environments. Traditional 3D modeling packages such as the Autodesk Maya and Blender have a very high learning curve, and are inappropriate for fast conceptual design by lay users, especially in immersive settings. Recently several research [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF][START_REF] Igarashi | Teddy: a sketching interface for 3d freeform design[END_REF] and commercial tools, such as Google's SketchUp, efforts have concentrated on providing intuitive and easy-to-use interfaces to build simple 3D models. These results and products are an indication of the widespread interest for such functionality. Intuitively, the creation of 3D models should be an ideal candidate for true 3D interfaces, since the content is inherently 3D. Such considerations have inspired futuristic artistic works 1 in which a 3D gestural interface is used to create a complex and visually rich environment.

Based on this intuition, our goal is to study direct manipulation for basic 3D modeling in an immersive setting, in the context of conceptual or initial design for architecture. An example of a system allowing such conceptual design is the work of Cabral et al. [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF], which allows users to manipulate geometry and texture directly, and automatically updates the model to respond to these changes. In addition to geometry and texture, we are interested in the manipulation of lighting, which is an important element in modeling and design, especially for architectural models. The last element of interest is the interaction paradigm and in particular the "view" or "mode" in which the user performs the modeling task: egocentric (1:1 scale) immersive mode, world-in-miniature or exocentric mode or a combination of both (see Fig. 1).

The goals of this study are to create a prototype system in an immersive setting, which supports casual modeling of textured geometry and lighting, and to investigate the relative merits of the different view modes with a pilot user study. Our system is designed to be simple and easy to learn, targeting novice users. Our main contribution is to incorporate the use of an automated geometry and texture manipulation system, including basic realtime lighting design in an immersive, WIM-like interface setting. In architectural design, users need tools for interaction which are simple yet provide enough flexibility to achieve the desired goal. Architectural design and geometric modeling have traditionally been priviledged applications for virtual reality systems [START_REF] Deisinger | Towards immersive modeling -challenges and recommendations: A workshop analyzing the needs of designers[END_REF][START_REF] Bullinger | Towards user centred design (ucd) in architecture based on immersive virtual environments[END_REF][START_REF] Leigh | Calvin: an immersimedia design environment utilizing heterogeneous perspectives[END_REF]. Our approach draws inspiration from these and others but addresses the problem from a different perspective. The closest previous work is the World-in-Miniature (WIM) system [START_REF] Stoakley | Virtual reality on a WIM: interactive worlds in miniature[END_REF] and derivations [START_REF] Laviola | Handsfree multi-scale navigation in virtual environments[END_REF]. They presented a miniature version mixed with a scale 1:1 model (equivalent to our "mixed mode"), but for simple manipulations of viewpoint and of the virtual scene. In contrast, we base our work on a system [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF] which provides automatic assistance in the context of a "quick prototype" modeling or conceptual design application [START_REF] Anderson | A virtual environment for conceptual design in architecture[END_REF]. More sophisticated modeling/design operations can thus be performed. In addition, we study lighting design in this context, using a simple "one-bounce" global illumination approximation to evaluate the consequences of light source changes in an immersive setting.

PREVIOUS WORK

By targeting specific developments to the universal 3D tasks of manipulation, selection, system control, navigation and symbolic input [START_REF] Bowman | A survey of usability evaluation in virtual environments: Classification and comparison of methods[END_REF] we can focus on providing a resourceful experience for architectural environment design. Our study is related to several aspects of manipulation of objects in large scale and immersive displays; these are usually accomplished with some degree of physical navigation and interaction.

The idea of automatically scaling an object and placing it within reach in front of the user has been proposed by Mine et al. [START_REF] Mine | Moving objects in space: exploiting proprioception in virtual-environment interaction[END_REF] and makes use of the concept of proprioception.

We use a simple form of bimanual interaction for resizing (similar to that used on 2D touch-sensitive devices). In previous work, bimanual interfaces have been used in immersive settings [START_REF] Balakrishnan | Exploring bimanual camera control and object manipulation in 3d graphics interfaces[END_REF]. We try to follow ideas which were initially observed by Guiard [START_REF] Guiard | Asymmetric division of labor in human skilled bimanual action: The kinematic chain as a model[END_REF] regarding the usage of both hands in our resizing gesture.

In our system, the user may switch between an immersive, scale 1:1 view and a world-in-miniature view, which implies a viewpoint change. Ware and Osborne proposed a technique [START_REF] Ware | Exploration and virtual camera control in virtual three dimensional environments[END_REF] called Eyeball in Hand in which the user directly controls the viewpoint with a device that directly maps its position and orientation. Our approach adopts the use of widgets to disambiguate rotation in all axes and provide the user more precise control. Go-Go [START_REF] Poupyrev | The go-go interaction technique: non-linear mapping for direct manipulation in vr[END_REF] and PRISM [START_REF] Frees | Precise and rapid interaction through scaled manipulation in immersive virtual environments[END_REF] switch between direct (1:1) and scaled interaction, either for largescale or precise manipulation, based on the user's intentions which are determined by the velocity of the hand. The HOMER system [START_REF] Bowman | An evaluation of techniques for grabbing and manipulating remote objects in immersive virtual environments[END_REF] attaches a virtual hand to an object selected by a light ray selection, allowing the user to manipulate far objects with no extra work.

Lucas et al [START_REF] Lucas | Resizing beyond widgets: object resizing techniques for immersive virtual environments[END_REF] introduce alternative 3D resizing widgets, including the straightforward generalization of the familiar 2D resizing widgets, the Pointer Orientation based Resize Technique (PORT), and the Gaze-Hand resize technique. Users prefer the familiar widgets, but after a training period they perform faster with the novel widgets.

PROTOTYPE SYSTEM DESCRIPTION

We designed and implemented a system for 3D immersive modeling aimed at initial prototyping or conceptual design, including modification of textured geometry and lighting.

The underlying modeling system is based on [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF], in which the user "fixes" a set of vertices, and then manipulates one or more "variable" vertices. The system solves an optimization and updates the rest of the model accordingly. Different "model pieces" can be assembled, which communicate through portals allowing the propagation of the mesh modifications.

For our prototype, we use a simplified version of the system in [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF] to deal with the specific task of building a simple house from a set of pre-built rooms which can be assembled. The user can add doors and windows which can subsequently be manipulated. Compared to [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF] we have added new functionality for lighting and the control of sun position, which we describe later. Our system is interactive, and thus geometry and textures of 3D models, as well as lighting, are updated on the fly.

The system receives input from the user through the use of a bimanual interface, which is currently implemented using an ART2 wand (dominant hand) and a hand-tracking device (non-dominant hand). The user is head-tracked and all interaction is performed in a BARCO iSpace 3.2x3.2x2.4m 4-wall system3 powered by a QuadroPlex NVidia card. The system was developed in C++ /OpenGL using in house libraries.

The system is best understood by watching the accompanying video which presents a comprehensive view of its usage.

Widgets and Modes

The system supports three different interaction modes, notably table, immersive (1:1 scale, fully immersive) and mixed, which is a combination of the former two modes. The user can interact with the environment through a set of "selectable widgets". Our goal is to provide visual feedback to the user indicating which actions are available at a given point in space. Although widgets introduce an extra level of abstraction [START_REF] Chen | Poster: A hybrid direct visual editing method for architectural massing study in virtual environments[END_REF], we believe that in an architectural environment widgets clearly indicate which actions are available to the user at a given location.

Each widget allows the user to perform a unique action. For instance, a "wall widget" allows the user to grab, then move or resize the wall (Fig. 2(a) and (b)). Alternatively, a "window widget" allows the user to resize or move a window within the limits of a wall (Fig. 2(c) and (d)). A "human avatar" can be dragged and dropped inside a particular room, changing the viewpoint so that the user is transported to immersive mode inside that room (Fig. 2(eg)); a button-press is used to go back. Other mode changes are also accomplished through a button-press on the wand. The list of available widgets are: wall, window, sun widget, add door, add window, connect room, go into immersive mode.

Actions can be executed in three different editing modes. All three modes take advantage of the immersive nature of a iSpace system, putting the user within reach of all tools for prototyping an architectural model. All modes allow co-located interaction based on the widgets described above.

Table Mode

The first mode is based on a table top design, with building blocks and tools available for interaction: the user stands in front of a table with the building blocks available for usage (Fig. 3(a)). The user begins the architectural design by choosing a pre-designed 3D model e.g., a room, corridor, etc. Room selection is provided to the user with a floating rotation menu with options, displayed to his right and within reach (co-located paradigm). (Fig. 3(b). As the user waves his hand over the menu, it rotates and displays all available options; the user presses the wand button to choose the appropriate room (please see video).

Table (and mixed) mode resembles the WIM paradigm, with the addition of the underlying table which gives the user stability and a "real location" to position its prototype. Since the user observes the world from an external perspective, this is an exocentric view.

Immersive Mode

The second editing mode is egocentric: the world is displayed at a 1:1 scale and the user is correctly immersed in the architectural model (Fig. 3(c)). In this second mode the user has the same editing tools available as in the previous mode. Interaction occurs in the same manner as in table mode using the widgets. One of the challenges in this particular mode is navigation: the user is free to walk around in the confined space of the iSpace system to explore and edit the model. This also translates to a 1:1 scale translation inside the virtual environment. However, it is often the case that the VR environment is bigger than the actual physical iSpace room. To overcome this limitation, the user can "fly through" the environment using the flystick.

Mixed mode

The third editing mode is a combination of the two previous editing modes. The user is positioned at a 1:1 scale inside the 3D model. While in this mode, the user is able to visualize a small scale version of the model in front of him.

Lighting Design

A special widget is provided for lighting design: the user can move an orange colored and textured sphere representing the sun over two trajectories (latitude and longitude). The date and time are displayed in 3D next to the sun widget; when these are changed the lighting is updated in the scene (see Figure 2

(h) and (i)).

A simplified one-bounce illumination model is used: a directional light source illuminates the scene through the windows in the room created by the user. The center of the projection of the window onto the floor is used as a secondary light source to update the illumination of the walls in the room, i.e., using a point to point differential form-factor [START_REF] Greenberg | Radiosity: A method for computing global illumination[END_REF]. Since we compute one bounce illumination and assume that all points are visible to the sun patch P, the form factor is computed per pixel on the GPU using the following formula:

F x,P = cos θ cos θ ′ πr 2 • A P (1)
For a sun patch P, we integrate over all points x in the room, where r is the distance between each point x and P, θ and θ ′ are the angles between the line connecting P and x and A P is the area of the projected sun patch P on the surface of the room. This is a simple approximation, but provides an effective first impression of global illumination changes due to modifications in lighting.

USER STUDIES AND EVALUATION

The integration of the prototype modeling system of [START_REF] Cabral | Structure preserving reshape for textured architectural scenes[END_REF] and a simple lighting control interface into an immersive interface allows us to study the relative effectiveness of the different modes (table, immersive and mixed) for several different tasks. Although the results are preliminary, it provides useful indication of what would be interesting to study in detail. Details on the user study procedure and basic statistical analysis are included in the supplemental material.

The study has two parts: a first set of specific tasks and then a more open task in which the user is asked to construct a three room house. The specific tasks are: resize a wall, add/move/resize a window, and move the sun. For placement and resize tasks, the user is presented with a target guide in wireframe which allows objective error in the task to be measured. For the sun task, the user is presented with a square having a target intensity, and is asked to move the sun to match the target intensity (see video).

The following table gives an overview of which actions can be accomplished in which modes:

Table Immersive Mixed Add window ✔ ✔ ✔ Move window ✔ ✔ ✔ Resize window ✔ ✔ ✔ Resize wall ✔ ✔ ✔ Add door ✔ ✔ ✔ Add room ✔ ✗ ✔ Move sun ✔ ✗ ✔
The participants are also presented with a questionnaire with subjective questions after the experiment.

Experimental procedure

The experiment starts with a training session so the participant can learn the interface. The participant is introduced to the interface by an experimenter who guides the user through the process. The participant starts in table mode, with a simple room created and is then guided through each action available in each mode. The participant is then asked to perform the action until the experimenter is satisfied that the knowledge of the action has been acquired. The training session takes between 10-15 minutes.

After a short break, the participant is asked to perform the set of tests involving the actions listed above. The window and sun tests are performed for each of above). The user completes each one of the tasks in a single mode; the user is not allowed to change modes in this part of the experiment.

For all add/move/resize tests, we record time to achieve the task and error compared to the wireframe target. Completion time is also recorded for the sun test and the error in color between the target square and the corresponding square on the wall is computed (see Fig. 4, second row).

For the open task, the participants were instructed to construct a three room house within a time limit of 5 minutes and to make sure that all rooms have enough sunlight in winter but not too much in summer. In this part of the experiment, the user can freely change between interaction modes.

A total of 8 participants completed the study in the iSpace of our institute, all male with ages varying from 25 to 40. All reported normal vision except for one subject who had stereo deficiency.

Experimental Results

Tasks in the immersive mode can take slightly longer if the user first needs to walk, virtually or physically, to get closer to the object. Some participants reported that it was difficult to reach the corners of the large windows when they were standing too far away. On the other hand resizing the windows from a suitable location was a little more accurate in immersive mode than in the other modes.

The widgets and the selection sphere that floated just in front of the flystick cast a shadow on nearby surfaces. For widgets in the house, participants could use these shadows as a depth cue to guide them to the correct position. Most participants initially had some difficulty selecting the sun widget because there was no shadow to guide them and they had to rely on the stereoscopic depth cue.

During the 5 minutes of the open task, participants switched between modes on average 13 times, suggesting that they preferred to perform some operations in specific modes. Most editing was performed in table mode, while immersive or mixed mode were used mainly for inspection and even for fun. Participants used the undo functionality on average only twice since any move or resize operations could easily be corrected manually.

The responses to the questionnaire were all positive about the system overall, finding the experience to be interesting and engaging. There was a small preference for table mode (4 subjects). Overall users appreciated the immersive nature of the system, and found interaction to be natural and pleasant.

DISCUSSION AND CONCLUSION

The results of the pilot study tend to indicate that none of the modes we used demonstrates clear superiority for the tasks proposed. As a result, all modes can be potentially used, depending on user preference, level of immersion required etc. The results also indicate that measured performance times are similar in table and mixed mode. This is interesting since in mixed mode the effects of lighting on the overall environment can be appreciated much better than in table mode only. Given the slight preference for table mode, it may be the case that mixed mode can be used as the "default view", but the user can "switch off" immersive view when they wish to concentrate on a specific task in table mode. Additional study is required to answer this question, with more complex tasks.

In conclusion, we have presented a first immersive system which allows simple conceptual design for textured geometric models and basic lighting design. We implemented and tested this system in an immersive room with a four-sided display, using three different modes: table, immersive (1:1 scale) and mixed, which is a combination of the first two (WIM). When asked to perform a more open task, all modes were used, and novice users all mentioned that they found the experience rewarding and interesting.

We are interested in pursuing these ideas further in the future, and notably investigating more complex modeling and lighting tasks, making the system appropriate for expert users and architects. It is particularly interesting to investigate the use of sophisticated global illumination in the context of lighting design, and the importance of different levels of realism, in the spirit of [START_REF] Slater | Visual realism enhances realistic response in an immersive virtual environment[END_REF].

Figure 1 :

 1 Figure 1: A view of our system in "mixed mode" where a miniature and immersive version are combined. INRIA / Photo Kaksonen.

Figure 2 :

 2 Figure 2: (a) The wall widget can be grabbed, resulting in a resized wall (b). (c) A window widget allows move and resize (bimanual) shown in (d). (e) The user selects the "human avatar" on the left and drags it to a location in the room (f). The user is then transported to that point in 1:1 scale (g). (h) The sun widget allows manipulation of the location of the sun (i), resultsing in an update of one-bounce illumination.

Figure 3 :

 3 Figure 3: (a) Table mode, (b) Room rotational menu (c) Immersive mode (d) Mixed mode

Figure 4 :

 4 Figure 4: (a) Target wireframe for a window resize and (b) a room resize task. (c) Sun target in table mode (start of test) (d) Sun target at the end of the test.

Table ,

 , Immersive and Mixed mode; the wall resize test is only performed in Table and Mixed mode (see table

http://www.ar-tracking.de/

http://www.barco.com/en/virtualreality/product/732.

ACKNOWLEDGEMENTS

We acknowledge the generous support of NVIDIA (Professor Partnership Program), and the Autodesk Research Donation program. This research was partially funded by the ARC NIEVE collaborative project (http://www-sop.inria.fr/reves/NIEVE) and the Regional Council of Provence-Alpes-Côte d'Azur.