Bradley Van Tonder
email: bradley.vantonder@nmmu.ac.za

Janet Wesson
email: janet.wesson@nmmu.ac.za

IntelliTilt: An Enhanced Tilt Interaction Technique for Mobile Map-Based Applications

Keywords: Tilt interaction, mobile map applications, sensor-based interaction

Current interaction techniques for mobile map-based applications suffer from several usability problems. Tilt interaction provides an alternative form of interaction which combines the benefits of one handed interaction with intuitive physical gestures. Research has shown that tilt interaction suffers from a lack of controllability, high mental demand and practical concerns. In this paper, the design and evaluation of a new tilt interaction technique, called IntelliTilt, is described. IntelliTilt incorporates several intelligent techniques to address the shortcomings of tilt interaction. IntelliTilt was compared to a basic tilt interaction technique using a prototype mobile map-based application in an experiment. The results of this experiment showed that IntelliTilt was preferred by the participants and that it offered significant advantages in terms of mental demand, perceived efficiency and controllability.

Introduction

Map-based applications are rapidly becoming standard features of mobile phones. Touch-screen and keypad interaction are currently the preferred interaction techniques for interacting with mobile map-based applications. These techniques, however, suffer from several shortcomings. Touch-screen interaction can result in the display being obscured by the user's fingers during interaction [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF]. Touch-screen interaction also requires both hands. Previous research has shown that users prefer to only use one hand when interacting with maps on mobile phones [START_REF] Karlson | Understanding Single-Handed Mobile Device Interaction[END_REF]. Keypad interaction limits the user's freedom of movement and provides little control over panning speed [START_REF] Cho | Dynamics of Tilt-Based Browsing on Mobile Devices[END_REF][START_REF] Kratz | Navigating Dynamically-Generated High Quality Maps on Tilt-Sensing Mobile Devices[END_REF].

Tilt interaction offers an intuitive alternative, with the user being able to interact with mobile map-based applications using natural gestures [START_REF] Cho | Dynamics of Tilt-Based Browsing on Mobile Devices[END_REF]. Tilt interaction typically allows users to interact with the display using rotations about the x and y axes. Tilt interaction does, however, suffer from several shortcomings. Previous studies have also shown no significant benefit in terms of effectiveness or efficiency [START_REF] Dong | Comparing Two One-handed Access Methods on a PDA[END_REF].

Basic tilt interaction using Speed Dependant Automatic Zooming (SDAZ) was previously shown to have several shortcomings in comparison to keypad interaction for mobile map-based applications [START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF]. This paper proposes an enhanced tilt interaction technique, called IntelliTilt, which incorporates several design features to address the shortcomings of basic tilt interaction. These features include making use of visual and vibrotactile feedback to improve the ease with which users can control panning and selection operations. Attractor mechanisms are included to make it easier to settle the cursor on a particular icon. Sensitivity adaptation is included to compensate for variability in accelerometer data while walking. A gesture zooming technique was developed to provide an intuitive method of zooming. These features were designed to address the shortcomings of basic tilt interaction identified earlier.

A user study was conducted to compare IntelliTilt to basic tilt interaction incorporating SDAZ. The purpose of the user study was to determine whether the design features of IntelliTilt helped to address the previously identified shortcomings of basic tilt interaction. Participants were required to perform typical mobile mapbased tasks while seated and while walking. Touch-screen interaction was not considered in this research as our focus was on investigating one-handed interaction techniques for mobile devices.

This paper begins with a discussion of related work. The design of the IntelliTilt interaction technique is then described. A prototype mobile map-based application, called MapExplorer, which was used for evaluation purposes, is described. Experimental results are then presented, followed by a discussion of the implications of these results. Finally, conclusions and ideas for future work are presented.

Related Work

Mobile Map-Based Applications

Mobile map-based applications typically support users in performing the following tasks [START_REF] Reichenbacher | Mobile Cartography -Adaptive Visualisation of Geographic Information on Mobile Devices[END_REF][START_REF] Von Hunolstein | Towards Task-Oriented Map-based Mobile Guides[END_REF]:  Locating: Identifying the position of something (e.g. where am I?);  Searching: Identifying facilities matching certain criteria (e.g. where is the nearest hotel?);  Navigating: Finding a route between two points or navigating along a route;  Checking: Determining the condition of a person or place (e.g. operating hours of a business); and  Identifying: Identifying and recognizing people, places or objects (e.g. the name of a business). These five high-level tasks are accomplished through several low-level operations. Irrespective of the interaction technique used, three low-level operations are commonly used, namely panning, zooming and selection [START_REF] Looije | Usability Engineering for Mobile Maps[END_REF].

Tilt Interaction

Tilt interaction was first proposed almost 15 years ago as an experimental form of interaction for mobile devices [START_REF] Rekimoto | Tilting Operations for Small Screen Interfaces (Tech Note)[END_REF]. Research into the use of tilt interaction has since been extended to a range of domains including menu navigation, text entry, mobile museum guides, photo browsing and interacting with mobile map-based applications [START_REF] Cho | Dynamics of Tilt-Based Browsing on Mobile Devices[END_REF][START_REF] Kratz | Navigating Dynamically-Generated High Quality Maps on Tilt-Sensing Mobile Devices[END_REF][START_REF] Mantyjarvi | Scan and Tilt -Towards Natural Interaction for Mobile Museum Guides[END_REF]. Most existing research has focused on the use of tilt as a means of performing panning, where the tilt angles along the x and y axes are mapped onto panning speeds in the horizontal and vertical directions. Most of this research relied on accelerometer data to determine tilt angles. Recently, gyroscopes and digital compasses have also started to be integrated into mobile phones, allowing for more accurate measurement of pitch and roll angles [12].

Tilt interaction techniques allow for one-handed interaction with mobile devices. This is desirable in a mobile context of use and allows the user an unobstructed view of the display. Furthermore, one-handed interaction requires less visual attention than bimanual control [START_REF] Eslambolchilar | Interact, Excite, Feel[END_REF]. Existing studies have shown that users like tilt interaction because of its natural, expressive and intuitive nature [START_REF] Cho | Dynamics of Tilt-Based Browsing on Mobile Devices[END_REF]. Unlike keypad interaction, tilt interaction allows for fine-grained control over panning speed.

Different implementations of tilt interaction differ in terms of the mapping between tilt input and the corresponding effect. Some implementations identify discrete inputs, where tilting past a certain threshold is interpreted similarly to a button press [START_REF] Oakley | Tilt and Feel: Scrolling with Vibrotactile Display[END_REF]. Other implementations allow for continuous input, where tilt input is used to continuously perform panning [START_REF] Dachselt | Throw and Tilt -Seamless Interaction across Devices Using Mobile Phone Gestures[END_REF]. The mapping between tilt input and panning effect can be done using position control, rate control or inertial control [START_REF] Oakley | Tilt and Feel: Scrolling with Vibrotactile Display[END_REF].

Rate control maps the rate at which the display pans onto the angle at which the device is tilted relative to the neutral position. For example, the further the device is tilted to the left, the faster the display pans left [START_REF] Hinckley | Foreground and Background Interaction with Sensor-Enhanced Mobile Devices[END_REF]. The function mapping tilt input onto panning speed does not need to be linear. It has been suggested that a linear mapping for small tilt angles allows for fine-grained control, while the amplification of larger tilt-angles makes it easier to pan faster over long distances [START_REF] Dachselt | Throw and Tilt -Seamless Interaction across Devices Using Mobile Phone Gestures[END_REF]. This approach has previously been used in a mobile map-based application [START_REF] Mock | A GPS Tracking Application with a Tilt-and Motion-Sensing Interface[END_REF].

Discretization can be used to improve the level of control users have over tilt interaction. Different tilt angle intervals are mapped onto different tilt speeds to minimize jitter and allow for finer-grained control. Linear, quadratic and sigmoidal discretization are all possible. Existing research has provided evidence that quadratic discretization allows for the highest level of user control [START_REF] Rahman | Tilt Techniques: Investigating the Dexterity of Wrist-based Input[END_REF]. Some mobile map-based applications allow panning and zooming to be performed simultaneously. Most of these applications implement Speed Dependent Automatic Zooming (SDAZ). SDAZ was proposed to address some of the problems and shortcomings of traditional zooming and panning techniques [START_REF] Igarashi | Speed-dependent automatic zooming for browsing large documents[END_REF]. These problems include blurring (and disorientation) when panning at high speeds and the need for the user to switch focus between the display and user interface controls [START_REF] Igarashi | Speed-dependent automatic zooming for browsing large documents[END_REF][START_REF] Cockburn | Comparing speed-dependent automatic zooming with traditional scroll, pan, and zoom methods[END_REF]. SDAZ adjusts the zoom-level automatically in response to the speed at which the user pans the display. SDAZ has since been implemented for touch-screen mobile devices [START_REF] Jones | An evaluation of integrated zooming and scrolling on small screens[END_REF].

SDAZ has been implemented using tilt interaction to perform combined zooming and panning in a mobile document browser [START_REF] Eslambolchilar | Tilt-Based Automatic Zooming and Scaling in Mobile Devices -a state-space implementation[END_REF]. Kratz and Rohs [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF] combined an element of manual control over the zoom level with a tilt-controlled SDAZ implementation. Their technique, called semi-automatic zooming (SAZ), allows users to control the zoom level using a touch-screen slider control. SAZ was shown to offer significant efficiency, workload and user satisfaction advantages over SDAZ.

Problems and Shortcomings

An experiment was previously conducted by the authors to compare tilt interaction (incorporating SDAZ) and keypad interaction [START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF]. Participants were required to perform typical mobile map-based tasks (locating, navigating and checking) using a prototype mobile map-based application, called MapExplorer. This experiment took the form of a lab-based user study and involved 32 participants.

The results of this experiment allowed several shortcomings of basic tilt interaction to be identified:  Controllability: Tilt interaction was shown to be difficult to control precisely, particularly for performing selections. This problem is not restricted to our implementation and is a well-documented shortcoming of tilt interaction [START_REF] Karlson | Understanding Single-Handed Mobile Device Interaction[END_REF]. Tilt interaction used for panning also tends to suffer from overshooting problems, where users pan past their intended target and have to reverse direction [START_REF] Cho | Multi-Context Photo Browsing on Mobile Devices Based on Tilt Dynamics[END_REF].  Zooming: The use of SDAZ in MapExplorer yielded mixed results. While SDAZ did allow users to pan longer distances more rapidly, it was criticized because it was too easy to trigger accidentally and took control away from the user.  Mental Demand: Tilt interaction was reported to be more mentally demanding than keypad interaction, particularly for locating and checking tasks.  Sensitivity: Several participants felt that the sensitivity of the tilt interaction did not match their expectations (either too sensitive or not sensitive enough).  Practicality: Some participants commented that they felt that tilt interaction was unlikely to be feasible in other contexts of use, such as while walking. This is due to the fact that tilt interaction requires users to exert control over their physical movements while interacting with the mobile device.

Several design modifications were made to the basic tilt interaction technique used in the above experiment in order to address the shortcomings identified. The design of this modified tilt interaction technique, called IntelliTilt, is described below.

Intellitilt: Design and Implementation

IntelliTilt was designed to address the shortcomings of basic tilt interaction discussed above. Table 1 summarizes the relationships between the problems identified in basic tilt interaction and the features of IntelliTiltdesigned to address these shortcomings. The design of each of the above features of IntelliTilt is now described in more detail.

Visual and Vibrotactile Feedback

The results of the previous experiment showed that participants struggled to control tilt interaction. Qualitative comments revealed that some participants found the exact effects of tilt gestures (in terms of panning speed and direction) difficult to predict. In order to accurately control tilt interaction, users need to be aware of the effect the current orientation of the mobile phone has on the direction and speed of panning. Various modalities have previously been employed to aid in controlling tilt interaction, including visual, vibrotactile and audio feedback [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF][START_REF] Oakley | Tilt to Scroll: Evaluating a Motion Based Vibrotactile Mobile Interface[END_REF][START_REF] Eslambolichilar | Multimodal Feedback for Tilt Controlled Speed Dependent Automatic Zooming[END_REF]. Audio feedback was considered to be impractical because it is likely to be annoying to other people (and the use of headphones is not always feasible). Previous research has only made limited use of visual feedback to indicate cursor position and to control automatic zooming [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF].

IntelliTilt makes use of visual feedback in the form of arrows attached to the cursor, showing the vertical and horizontal panning speeds (Figure 1). The length of these arrows is used to denote speed (the longer the arrow, the faster the panning speed). A simple linear mapping between panning speed and arrow length is used. IntelliTilt also makes use of vibrotactile feedback to reinforce the visual feedback channel. This form of feedback is particularly useful in situations where the user may be partially distracted (e.g. while walking). Previous research regarding the use of vibrotactile feedback in conjunction with tilt interaction for menu navigation showed that users liked the use of short vibration pulses to indicate movement from one menu item to another [START_REF] Oakley | Tilt to Scroll: Evaluating a Motion Based Vibrotactile Mobile Interface[END_REF]. IntelliTilt uses short vibration pulses (250 ms) to indicate when the cursor moves over point of interest (POI) icons and route markers. This duration was selected as it has previously been employed for similar purposes and experimentation revealed it to be long enough to be noticeable, but short enough to not irritate users [START_REF] Oakley | Tilt to Scroll: Evaluating a Motion Based Vibrotactile Mobile Interface[END_REF]. The available map area is divided into grid cells based on latitude and longitude, with POI icons indexed into these cells using a hash table. This enables IntelliTilt to efficiently determine whether the cursor is close enough to a POI or route marker icon to trigger vibrotactile feedback.

Attractor Mechanisms

One of the most significant problems encountered during the use of tilt interaction in mobile map-based applications is that users struggle to settle the cursor on a particular target icon. In a previous system using tilt interaction to browse photo collections on a mobile phone, attractors were employed to make it easier to settle on a particular photo [START_REF] Cho | Dynamics of Tilt-Based Browsing on Mobile Devices[END_REF].

This idea was extended to the two-dimensional domain of map browsing. The attractors are designed to work in conjunction with a discretization approach, which splits tilt input into discrete speed levels. If both the current horizontal and vertical panning speeds are slow (in which case it is likely that the user is trying to select an icon), the algorithm then determines whether any selectable icons are within a specified distance. Nearby icons are identified using the same indexing method used for vibrotactile feedback. If a nearby icon exists, the position of the cursor is adjusted vertically and horizontally to draw the cursor towards the centre of the nearest icon. Figure 2 illustrates the use of attractors to draw the cursor (shown in blue) towards the nearest POI within range of the cursor.

Fig. 2.Example of the use of attractors to aid selection

The resulting effect is one of drawing the cursor slowly towards selectable icons. The effects of the attractors are easily overcome by increasing panning speed, so as to avoid being drawn onto icons when this is not desired. The use of attractors eliminates the need to exactly position the cursor, as the user is able to approximately position the cursor and is assisted by the attractor to perform exact selection.

Sensitivity Adaptation

One of the problems identified in the previous experiment was that users felt that tilt interaction would be impractical to use in a mobile context due to the sensitivity of this form of interaction [START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF]. If tilt interaction is not sensitive enough, it is inefficient, while if it is too sensitive it is difficult to use while walking. Sensitivity adaptation was implemented in IntelliTilt to address this problem.

Streams of accelerometer data along the x, y and z axes are constantly recorded and monitored. Accelerometer data is sampled at approximately 20 Hz and a sliding window of data from the last second (last 20 samples) is used to identify whether the user is currently stationary or mobile. Such a sampling rate has previously been successfully employed in a similar mobile application [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF]. Several data samples were recorded of users using MapExplorer while stationary and while walking in order to measure the baseline acceleration variances for the three axes. While performing tilt operations, spikes in acceleration along one or more of the axes are recorded, so a single spike in acceleration is not sufficient to identify when the user is walking. Variances were calculated for acceleration along the three axes, as these provide a more accurate measurement of variability in the acceleration data than the raw values themselves. Analysis of the recorded data showed that while walking, all three acceleration variance values were typically greater than 0.125 (where acceleration is measured in g-forces). This is as a result of the linear acceleration detected by the accelerometer while the user is walking. While seated, acceleration is largely as a result of deliberate tilting gestures, which typically involve acceleration along only one or two axes. As a result, all three values rarely exceeded this threshold at the same time while the user was seated. This observation allows a computationally inexpensive method of identifying when the user is walking that can be performed quickly enough not to negatively affect system performance. Figure 3 illustrates the acceleration variances using actual recorded data while seated and walking.In order to improve the accuracy of the algorithm, all observations within a second need to indicate a change in order for the algorithm to transition between walking and stationary states. Preliminary testing revealed this method to be extremely accurate at identifying when users were walking (> 98% accuracy in informal trials). In order to compensate for increased acceleration variability when walking, IntelliTilt decreases the sensitivity of tilt interaction (Figure 7). Tilt interaction techniques typically employ a "dead zone", wheresmall tilt angles are ignored to allow the user to maintain a stable cursor position. IntelliTilt also increases the size of the dead zone when the user is walking. IntelliTilt constantly observes the variation in the acceleration data in order to identify when to implement sensitivity adaptation.

Gesture Zooming

Previous implementations of tilt interaction have largely relied on SDAZ, where the zoom level is linked to the panning speed [START_REF] Kratz | Semi-Automatic Zooming for Mobile Map Navigation[END_REF][START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF][START_REF] Eslambolchilar | Control centric approach in designing scrolling and zooming user interfaces[END_REF]. This form of zooming, however, has been shown to be frustrating for users who feel that it takes control away from them. In an attempt to find a form of zooming that would offer better usability and user satisfaction, a gesture-based zooming technique was implemented in IntelliTilt. Gesture zooming in IntelliTilt relies on physical movement of the mobile phone.

Zooming out was implemented using a backward movement of the device (Figure 4), while zooming in was implemented using a forward movement of the device, relative to the z-axis (which is perpendicular to the phone's screen). This mapping was designed to be conceptually the same as moving a camera away or towards the map. Such an approach has previously been used in a camera-controlled mobile map-based application [START_REF] Mooser | Large Document, Small Screen: A Camera Driven Scroll and Zoom Control for Mobile Devices[END_REF].These gestures are identified by constantly examining the z-axis acceleration data received from the accelerometer to identify spikes in the data. Acceleration data was captured and stored of users performing the zooming gestures using a Nokia N97. Figure 5 shows an example of the z-axis acceleration data captured. The spikes in the data show where zooming gestures were performed. Zooming in and zooming out gestures can be differentiated from each other based on the order in which the lower and upper spikes occur (this can be used to infer the direction the phone is moving). Experimental trials identified changes of 0.15g either side of the baseline z-acceleration (when the phone is at rest) as suitable to identify zooming in and zooming out gestures. While walking, this interval is increased to allow for the greater variability in z-axis acceleration data. The acceleration values reported during zooming gestures must take the starting orientation into account. As shown in Figure 5 (dotted line), the baseline z-acceleration value will differ depending on the orientation of the phone. As a further measure to avoid accidental zooming, zooming gestures have to be completed within 300ms (normal movement is generally more gradual). This threshold was determined after experimentation to identify the most suitable interval.

Dwell-Time Selection

During the previous experiment, it was observed that users would often accidentally pan away from a target icon while touching the screen to perform a selection operation [START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF]. Furthermore, the original design required both hands to perform selection operations. In order to combat this, dwell-time selection was implemented in IntelliTilt. If the cursor is over an icon for longer than 2.75 seconds, the icon is selected without the user having to do anything. Vibration pulses of increasing durations (25ms, 50ms and 75ms) are used to indicate to the user that a selection operation is about to be performed. The duration of 2.75 seconds was selected after experimentation and was set relatively long as the timing starts when the user is within 25 pixels of any icon, rather than directly over an icon. Shorter durations resulted in unintentional selections taking place. Dwell-time selection can be deactivated when not desired and manual touch-screen selection is always possible.

Prototype: MapExplorer

A prototype mobile map-based application supporting both basic tilt and IntelliTilt was developed for experimental purposes. The prototype, called MapExplorer, was implemented in Java ME and Python and tested using a Nokia N97. Microsoft's Bing Maps service was used as the source of the maps in the application, with map tiles being saved to the phone's memory to improve performance. MapExplorer allows users to browse maps at ten different zoom levels (from 1 pixel = 4891.97m to 1 pixel = 9.56m). A caching system is used, whereby the most recently loaded map tiles are stored in memory, allowing these tiles to be efficiently loaded the next time they are needed. When the maximum cache size is reached (40 tiles), the least recently used tiles are removed from memory to make space for the new tiles.The basic functions of MapExplorer are first described, along with the implementation of tilt interaction. The implementation of SDAZ, used in the basic tilt technique, is also described.

Functionality

MapExplorer supports four of the five typical mobile map-based tasks (locating, navigating, identifying and checking). Searching was not implemented in order to ensure that users interacted with the map-based display. Three categories of POIs are included in the system, namely restaurants, hotels and tourist attractions, each denoted by a different icon (Figure 6). Selecting a POI icon displays detailed information such as a description, contact details and opening hours. MapExplorer also allows users to plan and follow routes. A route planning web service was used to calculate routes and retrieve routing instructions. By moving the cursor over route information markers, users are able to read routing instructions (Figure 6). Tilt interaction in MapExplorer is implemented using a rate control approach. The further the device is tilted in a certain direction, the faster the display is panned in that direction. The user is able to finely control the speed and direction of panning by adjusting the tilt angle in any direction.

The accelerometer in the Nokia N97 is able to detect acceleration between -2g and 2g relative to the x, y and z axes. The Python sensor module was used to access the acceleration data. Accelerometer values along the x, y and z axes are used to calculate pitch and roll angles [START_REF] Tuck | Tilt Sensing Using Linear Accelerometers[END_REF]. Discretization is used to map pitch and roll angles onto panning speeds (Figure 7). This process consists of splitting the range of motion into intervals, with tilt angles falling in the same interval resulting in the same effect. Discretization intervalsare increased for larger angles, as previous research has shown that this approach allows for the highest level of user control. Panning speed increases linearly for the first two discretisation intervals and quadratically for the last three. Previous research has shown that this allows for fine control with small tilt angles and faster panning with largerangles. A dead zone is included to avoid unintentional panning as a result of minor orientation adjustments and hand tremor.

Tilt control is activated and deactivated by tapping anywhere on the display. This allows the user to lock the display and prevents accidental panning from taking place.The neutral orientation, relative to which tilt angles are calculated, is the orientation of the phone when tilt interaction is activated.

SDAZ Implementation

The zoom level is linked to panning speed in order to prevent the problem of extreme visual flow, where the display can become blurred when panning at high speeds [START_REF] Igarashi | Speed-dependent automatic zooming for browsing large documents[END_REF]. In order to combat this problem, the display is automatically zoomed out in order to prevent blurring when panning speed exceeds a certain threshold. The zoom level is a function of the panning speed. According to the original implementation of SDAZ, the zoom level is calculated as the normalized displacement between the current panning speed and the minimum panning speed which triggers zooming [START_REF] Igarashi | Speed-dependent automatic zooming for browsing large documents[END_REF]. That is:

(1)
Where s o is the minimum scale, d y is the current panning speed, d 0 is the minimum panning speed which triggers zooming out and d 1 is the maximum panning speed.

When automatic zooming takes place, a red rectangle is displayed to denote the area that will fill the screen when the panning speed is decreased again (Figure 8). A zoom level indicator is also temporarily displayed while the user is zooming. Automatic zooming was limited to only zooming out a single discrete zoom level to minimize user disorientation. Manual zooming can still be performed using the onscreen controls.

User Study

In order to determine whether IntelliTilt helped to address the shortcomings identified in the previous experiment, a user study was conducted to compare the tilt interaction technique developed in the original version of MapExplorer (referred to hereafter as basic tilt) with the modified interaction technique, called IntelliTilt.

Method

Participants.Sixteen participants (eleven male, five female) took part in the experiment. Nine participants had prior experience with tilt interaction, but in most cases this was only limited use of mobile games or the Nintendo Wii. All participants were computer science students between the ages of 20 and 29. Four participants were left handed and most indicated occasional use of mobile map-based applications.

Experimental Design and Tasks.A within-subjects approach was used with all participants using both interaction techniques. Two similar task sets were used to offset any order effects and the order in which the task sets were used was counterbalanced across participants. The order in which participants used the two interaction techniques was counterbalanced to offset any learning effect. The two techniques were simply named "version 1" and "version 2" (depending on the order in which they were used) in all questionnaires and documentation to avoid any bias.Participants were required to complete three types of tasks:  Locating tasks: participants were required to find a particular POI icon (e.g.

Locate the Ritz Hotel). Locating tasks therefore incorporate identify tasks.  Navigating tasks: participants were required to plan and follow a route from a particular start point to a particular end point (e.g. Plan a route from the beach to the airport. Follow the route from start to finish).  Checking tasks: participants were required to check specific information for a particular POI (e.g. check the opening hours of Starlight Restaurant).

Participants had to complete two tasks of each task type while seated followed by two tasks of each type while walking. Walking tasks were included in order to evaluate the two interaction techniques in situations in which there is likely to be more variability in the accelerometer data as a result of movement. Walking tasks were conducted indoors in a laboratory environment. All tasks started at the scale 1 pixel=38.22m. Targets were located in a major metropolitan area.

Metrics.The following three sets of metrics were collected:  Perceived workload: Participants were required to complete a post-task questionnaire after using each interaction technique. This questionnaire included the six perceived workload measures from the NASA-TLX questionnaire (mental, physical and temporal demand, performance, effort and frustration) [START_REF] Hart | Development of NASA-TLX (task load index): results of empirical and theoretical research[END_REF].  User satisfaction: Six questions were included in the post-task questionnaire to measure perceived efficiency, effectiveness, ease of use, controllability, ease of performing selections and ease of use while walking. This section was based on the standard After-Scenario Questionnaire (ASQ) [START_REF] Lewis | Psychometric evaluation of an after-scenario questionnaire for computer usability studies: The ASQ[END_REF].  Performance: Tasks were built into the system, allowing for easy measurement of task times, which were recorded in a log file stored on the phone.

The post-test questionnaire required participants to rate their preferred interaction technique while seated and while walking and to identify their preferred zooming technique. Participants were also asked to rate the usefulness of the visual and vibrotactile feedback and to identify positive and negative aspects of each interaction technique. Seven point semantic differential scales were used throughout.

Perceived Workload and User Satisfaction Results

Figure 9 shows the mean perceived workload ratings for the two interaction techniques. Perceived workload was lower (better) in all categories for IntelliTilt. These differences were significant for mental demand (Wilcoxon Z = 2.78, p = 0.005), temporal demand (Wilcoxon Z = 2.20, p = 0.03) and effort (Wilcoxon Z = 2.55, p = 0.01).

User Satisfaction

Basic Tilt IntelliTilt Figure 10 shows the mean user satisfaction ratings for the two interaction techniques. Participants were asked to rate the two techniques on a seven point semantic differential scale in terms of effectiveness, efficiency, controllability, ease of use, ease of selection and ease of use while walking. The results show that the IntelliTilt received higher user satisfaction ratings for all six satisfaction questions. These differences were statistically significant for effectiveness (Wilcoxon Z = 2.19, p = 0.03), efficiency (Wilcoxon Z = 2.71, p = 0.01), controllability (Wilcoxon Z = 2.17, p = 0.03), ease of selection (Wilcoxon Z = 2.82, p = 0.01) and ease of use while walking (Wilcoxon Z = 2.13, p = 0.03).

Table 2 shows the participant preference ratings from the post-test questionnaire. In Table 2, a value of 1 indicates a preference for basic tilt and 7 indicates a preference for IntelliTilt. Participants preferred IntelliTilt both while seated and while walking. The preference for IntelliTilt was stronger when participants were walking (mean of 5.88 while walking as opposed to 4.94 while seated). The difference between the mean values and the neutral value was statistically significant in both cases. Gesture zooming was preferred over SDAZ, with a mean value of 4.44. The difference between this mean value and the neutral value was not, however, statistically significant. Participants were also asked to indicate whether they felt the vibration feedback and visual feedback were useful. Mean values of 6.25 and 6.00 were recorded for vibration and visual feedback, indicating that participants found these forms of feedback to be very useful. The difference between the mean and the neutral value of four was statistically significant for vibration feedback (t 15 = 7.32, p < 0.01) and for visual feedback (t 15 = 6.32, p < 0.01). Table 3 shows the mean task times while seated and walking for the two interaction techniques. IntelliTilt proved slightly more efficient for seated tasks, while basic tilt proved slightly more efficient for walking tasks. The mean total time for both walking and seated tasks was almost identical. None of the differences between the two techniques were statistically significant using Wilcoxon ranked paired tests and alpha values of 0.05. Mean task times for different task types are shown in Table 4. IntelliTilt achieved marginally better task times for locating and checking tasks, while basic tilt was slightly more efficient for navigating tasks. None of these differences were statistically significant using Wilcoxon ranked paired tests and alpha values of 0.05.

Performance Results

Qualitative Feedback

Positive feedback about IntelliTilt focused on improved controllability. Several participants found the attractor mechanisms to be useful when performing selections. Participants also liked the visual and vibrotactile feedback. Positive feedback regarding basic tilt was received from some participants who felt that the use of SDAZ allowed them to be more efficient when locating POIs, particularly while seated. Some participants also liked the relative simplicity of this technique. Negative feedback regarding IntelliTilt mainly centered on issues with gesture zooming, with some participants struggling to get used to this form of interaction. Negative feedback about basic tilt focused on the controllability of this technique, particularly while walking. Participants complained that SDAZ was not required when following a route and took place too easily while walking.

Participants who preferred gesture zooming found this form of zooming to be more intuitive than SDAZ. Several participants stated that they found SDAZ to be difficult and disorienting to use while walking. Participants who preferred the SDAZ zooming technique stated that they found this form of zooming to be more efficient and some found gesture zooming difficult to learn to use.

Discussion

IntelliTilt provided statistically significant improvements in three aspects of perceived workload, namely mental demand, temporal demand and effort. The improvement in mental demand is particularly encouraging, as this is one of the areas in which tilt interaction was previously identified as being inferior to keypad interaction [START_REF] Van Tonder | Is Tilt Interaction Better Than Keypad Interaction for Mobile Map-Based Applications?[END_REF]. Given the strongly positive comments regarding the use of attractors and visual and vibrotactile feedback, it is reasonable to assume that these features contributed to the improved mental demand.

The user satisfaction results were all higher for IntelliTilt. Of particular interest is the fact that users felt that it was easier to perform selection operations accurately using IntelliTilt. Given that the additional feedback and attractor mechanisms were designed to improve selection, this result is encouraging. It is also encouraging to note that participants felt that IntelliTilt was significantly easier to use while walking, since sensitivity adaptation was designed for this purpose. Participants were noticeably more relaxed when using IntelliTilt while walking and were generally able to maintain their normal walking pace. Using basic tilt, participants frequently had to slow down to maintain control. Participants made frequent use of dwell-time selection, even when both hands were available.

There was very little difference in terms of efficiency for the two interaction techniques and overall task times were almost identical. No significant trends were evident in the task times for the different types of tasks.Despite the fact that there was little difference between the task times of the two interaction techniques, the participants perceived IntelliTilt to be more efficient. Given that controllability has previously been identified as a problem with tilt interaction, it is encouraging to note that participants found IntelliTilt easier to control.

User feedback regarding the use of visual and vibrotactile feedback was very positive. Participants rated both forms of feedback as very useful. Qualitative comments regarding these forms of feedback was also very positive, with only one participant remarking that vibrotactile feedback might become annoying after prolonged use.

IntelliTilt was preferred both while seated and while walking. This preference was stronger while walking. Several possible reasons for this exist. Firstly, IntelliTilt included sensitivity adaptation which makes allowance for the greater variability in accelerometer data while walking. Secondly, it is likely that the attractor mechanisms and feedback played a greater role in improving controllability while the user was mobile. Finally, SDAZ was perceived to be difficult to use while walking.

Opinions regarding the preferred zooming technique were divided. While some users quickly mastered the gesture zooming technique, others found the learnability of this technique to be problematic. Gestures were sometimes not detected due to participants moving the phone vertically, rather than backwards or forwards along the z-axis (which is dependent on the phone's orientation). SDAZ also resulted in divided opinions. SDAZ was identified as being useful for locating POIs, but difficult to use when following a route. Participants also found SDAZ to be difficult to use when walking.

Design Implications

Analysis of the above results provided insight into the design of tilt interaction for mobile map-based applications. The following list contains some implications for the design of tilt interaction for mobile devices: 1. Mechanisms should be provided to compensate for the inexact nature of tilt interaction and facilitate selection by providing automated assistance to the user. 2. Multimodal feedback can be used to improve controllability and reduce mental demand. Vibrotactile feedback allows for non-visual reinforcement that is useful in a mobile context.

3. Compensation should be made for variability in accelerometer data while walking by decreasing the sensitivity of tilt interaction. 4. One-handed selection can be facilitated using tilt interaction. Dwell-time selection can provide one possible implementation.

Conclusions and Future Work

This paper proposed an enhanced tilt interaction technique called IntelliTilt, which was designed to address the current shortcomings of basic tilt interaction. IntelliTilt was compared to basic tilt interaction using a prototype mobile map-based application, which allowed participants to browse maps in a wide geographic area at a wide range of zoom levels. The inclusion of walking tasks allowed IntelliTilt to be evaluated in a realistic context of use which is essential for mobile interaction research. IntelliTilt was designed to address specific shortcomings of basic tilt interaction that were previously identified. Statistically significant improvements were achieved in several important areas, including perceived mental demand, controllability and ease of selection. These results provide empirical evidence that the use of visual and vibrotactile feedback and attractor mechanisms can aid users in controlling tilt interaction and performing selections. Participants also indicated that they preferred IntelliTilt, particularly when walking, suggesting that sensitivity adaptation was beneficial in this regard. Results regarding gesture zooming were less conclusive, however, with participants divided between SDAZ and the new gesture zooming technique.

Future work will involve investigating improvements to gesture zooming. The use of gyroscope and magnetometer sensors in conjunction with accelerometers will be investigated in order to improve gesture recognition and responsiveness.

Fig. 1 .

 1 Fig.1. Design of visual feedback in IntelliTilt (left) and an example of its use (right)

Fig. 3 .

 3 Fig.3.Comparison of the number of axes recording acceleration variances of greater than 0.125 while seated (left) and while walking (right). Acceleration measured in g-forces.

Fig. 4 .

 4 Fig.4.Zooming out using gesture zooming in IntelliTilt

Fig. 5 .

 5 Fig.5.Z-axis acceleration data showing zoom gestures. The dotted line shows the baseline acceleration which changes depending on the orientation of the phone.

Fig. 6 .

 6 Fig.6.Navigating using MapExplorer

Fig. 7 .

 7 Fig. 7.Discretisationwhere the user is (a) stationary and (b) mobile

Fig. 8 .

 8 Fig. 8. Panning at a high speed using basic tilt interaction and SDAZ

Fig. 9 .

 9 Fig. 9.Mean perceived workload ratings (95% confidence intervals shown) (n = 16). Lower performance values indicate better performance.

Fig. 10 .

 10 Fig. 10.Mean user satisfaction ratings (95% confidence intervals shown) (n = 16)

Table 1 .

 1 Shortcomings of tilt interaction and IntelliTilt features designed to address these shortcomings

	Shortcoming	Feature(s)
	Controllability	Visual and vibrotactile feedback, attractor mechanisms
	Zooming	Gesture zooming
	Mental Demand	Visual and vibrotactile feedback
	Sensitivity	Sensitivity adaptation
	Practicality	Sensitivity adaptation, dwell-time selection

Table 2 .

 2 Participant preferences (1=Basic tilt, 7 = IntelliTilt). Standard deviation values are shown in parenthesis (n = 16)

		Mean	T-stat	P-value
	Preferred Technique (seated)	4.94 (1.69)	2.22	0.03
	Preferred Technique (walking)	5.88 (1.45)	5.16	< 0.01
	Preferred Zooming Technique	4.44 (2.00)	0.88	0.39

Table 3 .

 3 Mean task times (in seconds) for seated and walking tasks (standard deviation shown in brackets) (n = 16)

		Seated	Walking	Total
	Basic tilt	275.73 (60.49)	267.74 (66.38)	543.47 (113.13)
	IntelliTilt	259.81 (82.54)	284.90 (91.66)	544.71 (156.98)

Table 4 .

 4 Mean task times (in seconds) for different task types (standard deviation shown in brackets) (n = 16)

		Locating	Navigating	Checking
	Basic tilt	124.38 (20.15)	314.16 (27.20)	104.93 (19.85)
	IntelliTilt	116.47 (16.92)	328.56 (44.90)	99.68 (19.26)


Ravi Kuber
email: rkuber@umbc.edu

Matthew Tretter

Emma Murphy

Developing and Evaluating a Non-Visual Memory Game

Keywords: Audio, blind, haptics, memory games, multimodal, speech

des établissements d'enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

Introduction

The recent surge in the availability of brain and memory training games suggests that individuals are interested in keeping their minds active, and strengthening their motor skills through interacting with the software. While these games are typically popular with children and younger adults, research has shown that cognitive training can in certain instances benefit older individuals (Lustig et al., [START_REF] Lustig | Aging, Training, and the Brain: A Review and Future Directions[END_REF]), some of whom may experience levels of memory loss. As much of the information presented via memory game interfaces is visual in nature, individuals who are blind can experience some difficulties accessing content when using their existing assistive technologies. Screen readers, which are often used to convey information from a graphical user interface through speech, inadequately handle graphics, and can omit structural information, making the process of playing a game more complex. In order to provide a more inclusive experience, a need has been identified to replace the missing structure of the interface through the use of alternative channels, such as audio and haptics.

Related Work

Non-Visual Games

A variety of games have been developed to address the needs of the blind community. For example, Sanchez [START_REF] Sanchez | AudioBattleShip: Blind Learners Cognition through Sound[END_REF] used spatialized audio in the design of a twoplayer game (AudioBattleShip) which provides the user with awareness of current position, and informs him/her of key events (e.g. the resulting outcome of dropping a bomb in a cell of the contender's battlefield). Input is made using a tablet, with a pen device aiding the triggering of events. More recently, arcade-style games have been modified specifically for blind users. Examples include an accessible version of ‗Rockband' (Allman et al., [START_REF] Allman | Rock Vibe: Rock Band® Computer Games for People with No or Limited Vision[END_REF]), where the user is presented with vibrations on the arm and ankle to represent the drum cues, while auditory information is used to provide feedback on performance. Morelli et al. [START_REF] Morelli | VI-Bowling: A Tactile Spatial Exergame for Individuals with Visual Impairments[END_REF] have developed ‗VI Bowling', an exergame where blind individuals interact with a wireless controller which provides vibrotactile cues to the user's hand. The user is able to identify the direction of the virtual pins using this feedback. Findings showed that participants were able to throw a virtual ball with an average error of 9.76 degrees, demonstrating the promise of using the ‗tactile dowsing' method developed, to aid interaction with a gaming interface.

In terms of memory games, Sjostrom [START_REF] Sjostrom | Using Haptics in Computer Interfaces for Blind People[END_REF] developed a non-visual interface specifically enabling children who are blind, to match pairs of sonified buttons together. Haptic feedback is also produced to provide structural cues to users, presented via the PHANTOM device. Once correctly identified, the pair disappear from the interface leaving a smaller selection of buttons to choose from. Wang et al. [START_REF] Wang | A Haptic Memory Game using the STRESS Tactile Display[END_REF] developed a similar game using tactile feedback presented via the STRESS² display. Using a range of tactile rendering techniques, the researchers created a set of discernable effects, which the user can explore using his/her sense of touch, to locate matching stimuli. Evreinova et al. [START_REF] Evreinova | An Alternative Approach to Strengthening Tactile Memory for Sensory Disabled People[END_REF] developed a memory game targeted to the needs of deaf and hard of hearing users, where participants explore vibrotactile patterns using the Logitech IFeel mouse [14]. The parameters of frequency and duration were modified to develop the set of effects for the game. Interestingly, rather than committing the whole tactile patterns to memory, participants were noted to recall the number of bursts of tactile information associated with each tactile icon. Other strategies were also developed by users to aid game play.

Non-Visual Memory

While research has traditionally focused on human abilities to recognize visual stimuli, less is known about our memory for both haptic and auditory (non-speech) items. In terms of audio, parameters of pitch and tempo of melodies can be effectively recalled by users [START_REF] Levitin | Memory for Musical Attributes[END_REF][START_REF] Levitin | Absolute Memory for Musical Tempo: Additional Evidence that Auditory Memory is Absolute[END_REF]. In terms of touch, estimates suggest that we are able to remember between two to six pieces of tactile information (Watkins and Watkins, [START_REF] Watkins | A Tactile Suffix Effect[END_REF]; Mahrer and Miles, [START_REF] Mahrer | Memorial and Strategic Determinants of Tactile Recency[END_REF]). Kuber and Yu [START_REF] Kuber | Feasibility Study of Tactile-based Authentication[END_REF] found that a sequence of four pin-based tactile icons could be recognized from a wider range presented. Participants were provided with an extensive training period to commit tactile stimuli to memory, which aided retention over the month-long period. However, findings from a follow-up study showed that after a gap of four months without practice of tactile passwords, rates of accurate identification reduced to 58.3% [START_REF] Kuber | Tactile vs Graphical Authentication[END_REF].

Non-Visual Design Considerations

Blind users often rely on keystrokes to navigate around an interface using a screen reader (e.g. up arrow to move up and down arrow to move down), rather than using a mouse. Interacting with a mouse requires a strong level of hand-eye coordination, so when developing a non-visual interface using such a device, it is essential to provide additional non-visual support to aid the targeting process. Furthermore, additional assistance is needed to remain positioned upon targets after locating them, as unintentional deviations may be made. Oakley et al. [START_REF] Oakley | Guidelines for the Design of Haptic Widgets[END_REF] found that attraction effects (gravity wells) and recess effects were the most effective methods to aid the targeting process, as it was difficult to slip away from a particular object mapped to these properties. Vitense et al. [START_REF] Vitense | Multimodal Feedback: Establishing a Performance Baseline for Improved Access by Individuals with Visual Impairments[END_REF] highlighted the benefits that force-feedback cues can make when presented alone and in combination with other forms of feedback, when exploring an interface. The researchers have suggested that conditions providing haptic effects were more quickly recognized than conditions that did not provide haptic feedback.

To ensure that abstract auditory cues (earcons) can be appropriately perceived when integrated with an interface, Brewster et al. [START_REF] Brewster | An Evaluation of Earcons for use in Auditory Human-Computer Interfaces[END_REF] have suggested maintaining the level of pitch no higher than 5Khz and no lower than 125-150 Hz. In addition, manipulating the spatial location of the earcon can be beneficial to distinguish between effects. Findings from a later study revealed that earcons can be recalled over longer periods of time, although training techniques were found to affect recall rates (Brewster,[START_REF] Brewster | Navigating Telephone-Based Interfaces with Earcons[END_REF]). Studies have shown that auditory feedback can be used to augment haptic cues, due to the mechanical limitations associated with force-feedback devices (McGee et al., [START_REF] Mcgee | The Effective Combination of Haptic and Auditory Textural Information[END_REF]) However, research has yet to focus on the memorability of multimodal cues within memory games.

The research described in this paper, examines how effectively sequences of nonvisual effects can be replicated, with the long term goal of developing design guidance for interface developers interested in improving access to memory games, through the use of non-visual feedback. More specifically, we have aimed to determine whether haptic feedback can be recalled more effectively if presented independently or in conjunction with other forms of feedback.

Development of Memory Game

‗Simon™' [7] is an electronic game, where the user is presented with a sequence of flashing colored lights and tones from an electronic device. The user is required to replicate the sequence, by pressing the colored buttons on the device in exactly the same order as originally presented. A multimodal game has been developed based on Simon™ (Figure 1). This was designed with the aim of being accessible to users who are blind. The user is presented with the following forms of feedback: speech, nonspeech audio, haptics, and graphics. The user can use one or more forms of each type of feedback to play the game. Blind users have the choice of using the keyboard, or using both the keyboard and the Logitech Wingman force-feedback mouse (Figure 2) to interact with the game. The interface contains four different-colored buttons (labeled ‗red,' ‗blue,' ‗green' or ‗orange') arranged in the shape of a cross. Each button is mapped to an earcon or speech icon, which is presented for a period of 100ms at 60dB using recommendations from [START_REF]Dangerous Decibels[END_REF] (Table 1). The buttons were recreated in the auditory domain using non-speech sound by manipulating the pitch and spatial position of a pure sine tone. Upward movements are indicated by a 1 second sine tone with an upward frequency glissando (220Hz-880Hz) and a sine tone with a downward glissando indicates a downward movement (880Hz-220Hz). Left and right positions were implemented by adjusting the spatial position of a 1 second sine tone (440Hz) earcon accordingly Haptic spring effects were developed and integrated with the interface to provide guidance towards a target (button labeled ‗red,' ‗blue,' ‗green' or ‗orange'). For example, to prompt the user to move rightwards, the mouse gently guides the users' hand towards the right-hand side of the page.

To play the game, the user is presented with a sequence of buttons which flash and/or play one or more non-visual effects described in Table 1. To complete the stage, the user must successfully select the buttons in the same order originally presented. The system presents speech-based feedback to the user, to indicate his/her completion and whether the attempt was successful or not. Other buttons are presented on the interface to enable the user to customize the forms of feedback which he/she wishes to use, and to manipulate the complexity of the game (Figure 1).

Main Study

The study aimed to address the following hypotheses:

 H1. Participants would achieve greater levels of accuracy replicating sequences when haptic cues are presented in composite with audio and/or speech, rather than by itself.  H2. Participants would be able to navigate faster using a mouse while confirming actions using keystrokes, compared to using the keyboard for both activities.

Due to practical difficulties associated with obtaining a large number of blind participants, sighted participants were included within this exploratory study. Ten sighted participants (aged between 20 and 30), were blindfolded when interacting with the game, to simulate conditions of being blind. Two legally-blind individuals (one congenitally blind, one adventitiously blind, aged between 20 and 25) were also selected to participate. Participants were provided with a ten minute period of training to familiarize them with the non-visual icons developed. Both blind participants were provided with an additional five minutes of training using the mouse, due to their lack of experience with the device.

Participants were then asked to play the memory game, by working through a series of different levels, corresponding from ‗easy' (two stimuli) to ‗complex' (seven stimuli) which they needed to replicate in order to progress to the next stage. Each condition presented in Table 2, was randomized to reduce the occurrence of an order effect. A pilot study had revealed that presenting separate speech and auditory effects in composite could cause confusion. Icons were developed presenting speech with variations in pitch to indicate to move upwards or downwards, or speech panned in a particular direction, to convey to the user to move left or right. Each condition was presented twice, to gain a more comprehensive overview of results. Auditory and speech based cues were delivered through noise-cancelling headphones.

For purposes of the study, participants were asked to ‗think-aloud', describing their experience after performing each condition, and suggesting improvements for the feedback presented. At the end of the study, they were asked to rate perceived levels of cognitive workload experienced under each condition using a Likert scale (1-5). To determine the usability of the game, the time taken to complete each level and the number of successful attempts were automatically logged by the software. Participants were also asked to rate the level of enjoyment experienced (1 to 5), and asked to provide any comments on the methods of interaction used (e.g. use of keyboard vs mouse with keyboard).

Results

Replication of Non-Visual Sequences

The percentage of successful attempts to replicate sequences of stimuli, are shown in Figure 3. A repeated measures ANOVA showed that the level of accurate replication of sequences varied by condition (F (2.901, 31.910) = 7.023, p=0.01, Greenhouse-Geisser adjusted). While results from Figure 3 suggested that the KS, MHS and MAHS conditions could be replicated most accurately (i.e. conditions employing speech-based feedback), post-hoc analysis (Bonferroni corrected) could only confirm significant differences between the following conditions (MH vs MHS, MH vs MAH, MH vs MAHS, MH vs KS, MAHS vs KA).

Although findings indicated that the number of accurate replications of sequences did not differ widely depending on method of interaction (Keyboard -M: 74.3%, SD: 25.7%; Mouse with keyboard -M: 74.5%, SD: 28.2%), a significant effect could not be detected using statistical analysis.

Fig. 3. Percentage of successful attempts by condition

The percentage of successful attempts by level, are shown in Table 3. Results have suggested that as the level of the game increased, accuracy was found to decline from 95.2% when replicating a sequence of two stimuli to 32.7% for a sequence of seven stimuli (F(4.245, 55.181) = 72.160, p=0.00, Huyhn-Feldt adjusted). The percentage of successful attempts increased upon the second attempt for Levels 3 (+1% difference) and 4 (+9% difference). However, as the game became more complex resulting in an increase of objects to target, the number of accurate replications on the second attempt appeared to decline. Results suggested that participants were on average faster performing tasks under KS, MH and MHS. Post-hoc analysis (Bonferroni corrected) confirmed differences between all levels. However, significant differences were only found between the following conditions: MH vs KA, MHS vs MAH, MHS vs KA, MAH vs KS, KS vs KA.

Although the average time taken for replicating sequences was not found to vary widely between different methods of interaction with the system (Keyboard -M: 5.9 seconds, SD: 3.5 seconds; Mouse with keyboard -M: 5.8, SD: 3.7), further analysis would be needed to confirm the presence of an effect.

Fig. 4. Time taken to replicate sequences by condition

Performance of Blind and Sighted Participants

Table 4 shows the results of both blind and sighted groups. The two blind participants were on average found to replicate sequences more successfully (78.0%) when compared with their sighted counterparts (73.7%). Results indicated that blind participants were able to perform tasks within a shorter period of time (4.7s, SD: 2.4s) compared to sighted participants (M: 6.1s, SD: 3.8s), when both using the keyboard or a mouse with keyboard. Table 4 shows that additional time was spent replicating sequences containing non-speech auditory feedback. Results indicated that the time taken was found to increase for both groups while progressing through levels, with a steady decrease in success when replicating sequences of stimuli. However, further study would be needed to identify whether these claims can be validated statistically.

Usability of Interface

Results indicated that levels of workload varied by condition, with larger levels experienced when using the mouse with keyboard, in comparison to solely using the keyboard (Table 5). When recalling the sequence of objects presented, participants reported the most difficulties when using the mouse and keyboard with no audio (MH), and when using the keyboard with auditory feedback (KA). Lower levels of workload were reported for conditions where speech-based feedback was presented (e.g. MHS, KS and KAS). When asked to rate each form of feedback, nine out of the twelve participants agreed with the statement that haptic and speech based cues were appropriate for use. However, only seven agreed with the same statement for audio. Four out of the twelve participants agreed with the statement that using the keyboard with the mouse, was more usable than using the keyboard alone. Nine of the twelve suggested that they were able to enjoy the experience of playing the games developed. Findings confirmed that participants were able to replicate sequences more accurately when additional feedback was presented alongside haptic feedback (M: 78.2%, SD: 24.3%) compared to the unimodal presentation of the haptic effects (M: 62.5%, SD: 27.4%), providing support to H1. Although not statistically significant, results suggested that more time was taken when additional feedback was presented alongside the haptic cues (M: 5.9s, SD: 3.6s), in contrast to haptics presented by itself (M: 5.5s, SD: 3.7s). The large levels of deviation experienced would be worthy of further study. Lower levels of perceived cognitive workload were experienced under conditions where speech was presented (MHS and MAHS), compared with MH and MAH. When questioned about their performance, participants suggested that the presentation of speech would provide the most informative cues which they could commit to memory, while the haptic feedback would provide supplementary feedback to aid retention and provide valuable guidance towards a target.

Method of Interaction with Game

Although findings suggested that the number of accurate replications of sequences and time taken did not differ widely depending on method of interaction, no effects could be detected. Therefore, H2 could not be supported. When questioned about their performance, participants suggested that constraints of the device had slowed their progress. For example, haptic directional effects could be missed, unless the force-feedback mouse was aligned towards the center rather than the edge of its base. Absolute positioning was suggested as one method of improving interaction with the device. Lower levels of cognitive workload were reported for keyboard-only interaction (M: 3.2, SD: 1.7), compared to when using the keyboard in conjunction with the mouse (M: 4.6, SD: 1.3). This was in part attributed to the sensitivity of the mouse, meaning that small movements made using the device, would translate into larger on-screen movements which could cause confusion for the users.

As the blind participants were unfamiliar with using a mouse, it was anticipated sighted users would be faster in terms of targeting objects using this method of interaction. Findings showed that blind participants completed tasks on average 1.4 seconds faster (SD: 1.4 seconds), with greater levels of accuracy (4.3%) compared with their sighted counterparts. Blind participants were noted to make careful, controlled movements to interact with the device. The blind participants were asked to comment on the condition which most effectively supports mouse use. The combination of speech, audio with haptic feedback (MAHS) was found by both participants to aid interaction with the game. One blind participant mentioned that the -accessibility of the game would enable her to take turns with her sighted siblings, and not be at a disadvantage due to her visual abilities.‖ This is particularly encouraging as research suggests that there are few immersive and collaborative games for blind and sighted users (Sanchez,[START_REF] Sanchez | AudioBattleShip: Blind Learners Cognition through Sound[END_REF]).

Auditory and Multimodal Feedback

Findings from our study revealed that more time was spent using a combination of audio and haptic cues (MAH -M: 6.2 s, SD: 3.5s) compared to the majority of other conditions presented in Figure 4. Cockburn and Brewster [START_REF] Cockburn | Multimodal Feedback for the Acquisition of Small Targets[END_REF] found that audio and tactile cues presented individually improved targeting times by 4.2% and 3.5%, while the combination of audio and tactile feedback reduced normal targeting times by only 1.7%. The researchers have suggested that by combining tactile feedback with ‗stickiness' (adjustment of control-display gain), could significantly benefit the targeting process, rather than using alternative combinations of feedback. Subjective comments received from participants in our study, suggested that they experienced difficulties processing both sets of cues in tandem, accounting for the larger levels of cognitive workload expended (M: 4.8, SD: 1.8). More specifically, participants had suggested that the pitch mappings used could on occasion cause confusion, requiring them to think more carefully about the direction of movement which they were meant to convey. Walker and Kramer [START_REF] Walker | Mappings and Metaphors in Auditory Displays: An Experimental Assessment[END_REF] highlighted that mental models and experiences from previous metaphors can influence a user's perception of pitch movement and associated mappings. This may explain why some users did not intuitively associate the upwards or downwards glissando with the intended movements. In future versions of the system, the auditory and haptic feedback will be more carefully designed to complement one another, with the aim of reducing the levels of cognitive workload experienced.

Effects of Memory

Although we did not record the stimuli selected by participants when a sequence was erroneously entered, participants were questioned on the most memorable parts of each sequence. All twelve participants strongly agreed with the statement that they found it easier to remember the first few non-visual stimuli, compared to the last few stimuli in each of the longer sequences. Watkins and Watkins [START_REF] Watkins | A Tactile Suffix Effect[END_REF] have suggested that with auditory input, a strong primacy effect is obtained when recalling tactile information, together with a shorter recency effect. Participants in our study were observed pausing midway through longer sequences, attempting to recall the remaining stimuli. Participants suggested that for longer sequences, they would on occasion attempt to guess the order of the last few remaining stimuli, in order to complete the level. While participants suggested that the presence of speech aided their retention of sequences, Watkins and Watkins [START_REF] Watkins | A Tactile Suffix Effect[END_REF] remain cautious about the role which verbal labels play, favoring further study in the area.

Participant Selection

When evaluating accessible interfaces, researchers often encounter challenges recruiting target users. This is in part attributed to the small size of the blind community, and the variability in the levels of residual sight which may impact the way the interface is used. In response, blindfolded sighted users are recruited for preliminary or exploratory studies. Studies have shown that no major differences were identified in response times between groups of sighted and blind users (Ferres et al., [START_REF] Ferres | Evaluating a Tool for Improving Accessibility to Charts and Graphs[END_REF]). However, results from the authors' study revealed that blind participants were found to select more commands, compared to their sighted counterparts. As many blind users opt to use the keyboard, rather than a mouse, so may require additional training to use these type of input devices. Although we thought that it may create difficulties for the blind users who were unfamiliar with the mouse, results in Section 5.3 have revealed that their careful controlled movements, enabled blind participants to complete tasks with and without a mouse in a faster time than their sighted counterparts. However, we acknowledge this may not be the same case for all blind users. Furthermore, the mental structural representation of interfaces differs between both sighted and blind groups (Kurniawan et al., [START_REF] Kurniawan | How Blind Users' Mental Models Affect their Perceived Usability of an Unfamiliar Screen Reader[END_REF]). This is largely due to the restricted output from a screen reader, leading individuals who are blind to perceive that objects on an interface are spatially presented along one dimension. While sighted users knew from previous experience with web pages, that new content is predominantly in the center of the page, they would move the mouse cursor towards this point to locate targets. The case was more difficult for blind users, who were observed moving around the page searching for a landmark to orientate their position, and then search for the particular target within the sequence. Speech-based cues were described as being essential by the blind participants, to aiding interaction with an unfamiliar device such as a mouse. Further study will also need to be performed with representative users, to ensure that the system meets the diverse needs of the blind community.

Conclusion and Future Work

This study has demonstrated the potential of non-visual feedback in the design of accessible memory games. Observations revealed that blind users were able to interact with the Logitech Wingman force-feedback mouse, and complete the games in a comparable time to using the keyboard alone. Findings have shown that blind users would benefit from haptic feedback providing directional information, however, more successful attempts at replicating sequences were found when haptic cues were presented in combination with other effects, particularly speech-based cues. This was thought to offer additional support when using an unfamiliar input device, such as a force-feedback mouse. In the future, we aim to examine whether participants are able to strengthen their auditory and haptic memory abilities through playing the game, and to see whether with more distinctive forms of feedback, participants are able to replicate sequences within a shorter period of time.

Fig. 1 . 2 .

 12 Fig. 1. Accessible multimodal memory game Fig. 2. Logitech Wingman force-feedback mouse [14]

  and by level (F(3.244, 450.929) = 294.940, p = 0.00, Greenhouse-Geisser adjusted).

Table 1 .

 1 Mappings to non-visual stimuli

		Audio			Speech Haptic
	Move	Increase	in	pitch	Up	Continuous directed guidance
	Up	(upward	frequency		(upwards) to target using attraction
		glissando)				effect
	Move	Decrease	in	pitch	Down	Continuous directed guidance
	Down	(downward frequency		(downwards) to target using
		glissando)				attraction effect
	Move	Sine tone panned left	Left	Continuous directed guidance
	Left					(leftwards)	to	target	using
						attraction effect
	Move	Sine tone panned right	Right	Continuous directed guidance
	Right					(rightwards) to target using
						attraction effect

Table 2 .

 2 Conditions presented to participants

	Method of Interaction Feedback	Abbreviation
	Mouse with keyboard	Haptics	MH
		Haptics and speech	MHS
		Audio and haptics	MAH
		Audio, haptics and speech MAHS
	Keyboard	Speech	KS
		Audio	KA
		Audio and Speech	KAS

Table 3 .

 3 Percentage of successful attempts and time taken by level

		Level	Average	Percentage	of	Average Time Taken
			Successful Attempts (%)		(Seconds)
		2	95.2 (SD: 8.5)		2.8 (SD: 1.2)
		3	94.6 (SD: 7.0)		3.5 (SD: 1.4)
		4	86.9 (SD: 14.5)		4.6 (SD: 2.0)
		5	75.6 (SD: 14.0)		5.8 (SD: 2.5)
		6	53.6 (SD: 16.6)		8.2 (SD: 3.3)
		7	32.7 (SD: 12.0)		10.1 (SD: 3.8)
	5.2	Time Taken to Perform Tasks
	Figure 4 shows the time spent replicating sequences by condition. A significant
	effect was detected by condition (F(4.034, 479.92) = 5.227, p = 0.00, Greenhouse-

Table 4 .

 4 Time taken by condition and level of sight (1-easier to 7-difficult)

	Condition Blind	Sighted
	MH	4.4 (SD: 2.1)	5.8 (SD: 4.0)
	MHS	4.5 (SD: 2.3)	5.8 (SD: 3.4)
	MAH	5.0 (SD: 2.5)	6.5 (SD: 3.6)
	MAHS	4.7 (SD: 2.1)	6.4 (SD: 4.4)
	KS	4.2 (SD: 2.7)	5.7 (SD: 3.6)
	KA	5.3 (SD: 2.7)	6.6 (SD: 3.8)
	KAS	4.8 (SD: 2.3)	6.0 (SD: 3.7)

Table 5 .

 5 Ranking of Ease of Recall and Cognitive Workload by Condition (1-easier/lower to 7difficult/higher)

	Condition Recall	Workload
	MH	5.9 (SD: 1.5)	5.8 (SD: 1.9)
	MHS	3.7 (SD: 1.5)	3.6 (SD: 1.6)
	MAH	4.9 (SD: 1.5)	4.8 (SD: 1.8)
	MAHS	3.7 (SD: 2.2)	4.2 (SD: 1.6)
	KS	2.6 (SD: 1.9)	2.3 (SD: 1.6)
	KA	4.5 (SD: 2.2)	3.9 (SD: 2.1)
	KAS	2.7 (SD: 1.3)	3.3 (SD: 1.4)

Acknowledgements

We thank Dr. Henry H. Emurian (UMBC) for his input to this work.


Ilda Ladeira
email: iladeira@cs.uct.az.za

Gary Marsden

Lesley Green
email: lesley.green@uct.ac.za

Designing Interactive Storytelling: A Virtual Environment for Personal Experience Narratives

Keywords: Virtual Reality, Digital Storytelling, User Experience

We describe an ongoing collaboration with the District Six Museum, in Cape Town, aimed at designing a storytelling prototype for preserving personal experience narratives. We detail the design of an interactive virtual environment (VE) which was inspired by a three month ethnography of real-life oral storytelling. The VE places the user as an audience member in a virtual group listening to two storytelling agents capable of two forms of interactivity:

(1) User Questions: users can input (via typing) questions to the agent; and (2) Exchange Structures: the agent poses questions for users to answer. Preliminary results suggest an overall positive user experience, especially for exchange structures. User questions, however, appear to require improvement.

Introduction

Stories of personal experience are a compelling way to learn about historical events [START_REF] Maynes | Telling Stories: The use of Personal Narrative in the Social Sciences and History[END_REF], particularly when told by those who lived the events first-hand. In South Africa, Apartheid enforced extreme racial segregation. Mixed-race neighborhood residents were relocated to areas allocated to their race and, typically, vacated neighborhoods were demolished to make way for new ones. "Forced removals" occurred throughout South Africa and District Six, in Cape Town, is a famous example [START_REF] Welsh | The Rise and Fall of Apartheid: From Racial Domination to Majority Rule[END_REF][START_REF] Ebrahim | Noor's Story: My Life in District Six[END_REF]. At the District Six Museum, visitors are able to hear this history told by two guides who once lived in District Six: Joe Schaffers and Noor Ebrahim. Sadly, the community of District Six ex-residents is now aging, leaving fewer people to tell of first-hand forced removal experiences. When Joe and Noor retire, museum visitors will no longer be able to hear their stories in person. Preserving stories such as theirs is the focus of many digital storytelling and cultural heritage projects [START_REF]Capture Wales: Digital Storytelling[END_REF]5,6]. Digital archives are not a matter of long-term preservation alone, but also of allowing wider dissemination of stories. However, little has been done to preserve real storytellers' dynamic and interactive qualities. Our work explores ways of presenting personal stories, beyond the, more typical, static video or 'talking head' presentations. We describe our multidisciplinary approach to creating a storytelling prototype informed by an ethnographic study of real-life oral storytelling. We propose two interaction techniques for making digital stories engaging by creating the illusion of interacting with real storytellers.

Background

Digital and Virtual Storytelling

Digital storytelling typically seeks to preserve and disseminate real-life, non-fiction stories. Some have focused on supporting user generated content [START_REF] Bidwell | Designing with Mobile Digital Storytelling in Rural Africa[END_REF][START_REF] Jones | Narrowcast yourself: Designing for Community Storytelling in a Rural Indian Context[END_REF] and others on creating publically accessible archives [START_REF]Capture Wales: Digital Storytelling[END_REF]5,6]. And, while some digital storytelling projects have provided interactivity in story creation and capture [START_REF] Bidwell | Designing with Mobile Digital Storytelling in Rural Africa[END_REF] and archive browsing [5], none, to the best of our knowledge, have explored interactive presentations of real-life stories. On the other hand, virtual storytelling projects have explored interactive storytelling extensively [START_REF] Cavazza | Narratology for Interactive Storytelling: A Critical Introduction[END_REF]. But, it has been, almost exclusively, with fictional content where users can manipulate, evolve or completely author narrative plots [START_REF] Brooks | Do story agents use rocking chairs?[END_REF][START_REF] Charles | User Intervention in Virtual Interactive Storytelling[END_REF][START_REF] Mateas | Structuring Content in the Façade Interactive Drama Architecture[END_REF]. Others have also explored allowing users to influence stories' emotional tone, such as Silva et. al.'s system [START_REF] Silva | Tell Me That Bit Again… Bringing Interactivity to a Virtual Storyteller[END_REF], where users could input how they would like the story told via tangible cards placed into an "influencing box". So, digital storytelling is devoted to presenting real-life stories whose plots listeners should, typically, not be able to edit, leading to story experiences with little or no interactivity. Meanwhile, virtual storytelling tends to give users agency over story plots, leading to highly interactive story experiences with little authorial control. We posit our work as something of an intersection, since we aimed to create digital, reallife stories where users can interact with story content, but not in ways that alter narrative plots.

Simulated Museum Guides

In museums, there have been impressive efforts towards digital tour guides which simulate specific guide-visitor interactions. After conducting a brief survey of tour guide experiences, Yii & Aylett [START_REF] Lim | Narrative Construction in a Mobile Tour Guide[END_REF] created digital tour guides on hand-held devices. These mobile guides had different "personalities", which influenced the stories they told. Additionally users were able to continually input their level of interest which influenced the extensiveness of stories told. Yamazaki et al. [START_REF] Yamazaki | Revealing Gauguin: Engaging Visitor's in Robot Guide's Explanation in a Art Museum[END_REF] studied 15 instances of real guide-visitor interactions in an art museum to inform the creation of a robot capable of mimicking some of the interactions they observed. They placed special focus on the direction of a guide's gaze and their robot was able to detect human faces so as to direct its "gaze" in appropriate directions and able to respond to audience reactions and interruptions to a limited degree. The robot guide also periodically posed "involvement questions" where the robot would pose a question about a museum artifact, pause for a preset amount of time and then give the correct answer. However, responses to these questions were preset and, hence, non-interactive, serving rather to create the illusion of interaction. One of the interaction techniques in our storytelling prototype aims to improve on this by creating a more truly interactive form of involvement questions, which we term "exchange structures".

Personal Experience Narratives

Personal experience narratives are those where a storyteller tells of something that happened to themselves or an acquaintance [START_REF] Pridham | The Language of Conversation[END_REF]. Labov [START_REF] Labov | The Transformation of Experience in Narrative Syntax[END_REF] describes a, now wellestablished, linguistic structure which personal experience narratives often follow wherein they are composed of a number of ordered components: abstract (signals a story's beginning); orientation (provides context); complicating action (the main event -usually something noteworthy or unexpected); resolution (the outcome); evaluation (commentary on why the story is interesting or noteworthy, this may appear near the end and throughout the story); and coda (signals the end). Martin and Plum [START_REF] Martin | Construing Experience: Some Story Genres[END_REF] further identified distinct genres of personal experience narratives: recounts are exact descriptions of events akin to a courtroom testimony, exemplums serve to convey moral or pedagogical judgments and anecdotes convey emotional and/or humorous aspects of the teller's experience. With live oral storytelling, a storyteller conveys a story using words and body language and audiences may interact with the storyteller to influence a story's course or tone. Hence, each oral retelling of story differs with the extent of variation ranging from slight to where the audience might be considered as co-storytellers [START_REF] Bauman | Story performance and event: Contextual studies of oral narrative[END_REF]. However, where a storyteller is telling a personal experience narrative, variations would not alter the overall plot. Discourse analyses of classroom conversations between teachers and students sheds light on the specificities of storyteller-audience interactions. A simple interaction might involve a student making a comment or asking a question to which the teacher responds. This can be extended to a longer interaction, called an exchange structure, which the teacher initiates by posing a question to which students may respond, leading to a dialog [START_REF] Pridham | The Language of Conversation[END_REF][START_REF] Bauman | Story performance and event: Contextual studies of oral narrative[END_REF][START_REF] Sinclair | Towards an Analysis of Discourse: The English used by teachers and pupils[END_REF]. These exchanges are not unlike the "involvement questions" implemented in [START_REF] Yamazaki | Revealing Gauguin: Engaging Visitor's in Robot Guide's Explanation in a Art Museum[END_REF], but are, of course, interactive in that a teacher can respond to a number of different student answers. In our work, we used ideas of understanding personal experience narratives from [START_REF] Labov | The Transformation of Experience in Narrative Syntax[END_REF] and [START_REF] Martin | Construing Experience: Some Story Genres[END_REF] and insights on teacher-student interactions from [START_REF] Sinclair | Towards an Analysis of Discourse: The English used by teachers and pupils[END_REF] to analyze ethnographic data and design interactive storytelling agents.

Research Overview

As [START_REF] Brooks | Do story agents use rocking chairs?[END_REF] points out, real storytellers establish a special connection with audiences that is difficult to emulate digitally. Our approach was to study real-life oral storytelling ethnographically in order to observe how experienced storytellers connect with audiences and draw them into their personal stories. To this end, we chose to observe Joe and Noor, who tell their personal stories every day to District Six Museum visitors. We aimed to learn about their techniques for making their personal stories engaging, dynamic and interactive. Our findings inspired the design and implementation of a storytelling prototype as a virtual environment (VE) containing interactive storytelling agents. Most recently, we conducted a large user study to evaluate our prototype's effectiveness.

Our research began with the rather broad aim of observing real-life techniques for making narratives engaging, dynamic and interactive. We wanted to learn how oral storytellers make content vivid and compelling when telling about personal experience and examine real storyteller-audience interactions. The first author spent three months at the District Six Museum unobtrusively observing tours led by Joe, Noor, independent guides not employed by the museum, and, occasionally, other museum staff. We observed and took field notes for thirty-nine tours -of these we audio recorded (via lapel microphone) nine tours and video recorded snippets from a further four tours. Audio recorded tours ranged from nine to sixty minutes; these were transcribed and became a main focus of our analysis. Using the classification framework from Martin & Plum [START_REF] Martin | Construing Experience: Some Story Genres[END_REF], we identified anecdotes and exemplums which appeared most often in Joe and Noor's tours. This led us to five stories (two from Noor and three from Joe), for which we had recorded three to five retellings each. We used Labov's structural framework [START_REF] Labov | The Transformation of Experience in Narrative Syntax[END_REF] to identify each retelling's constituent components, totaling nineteen distinct discourse analyses. This allowed us to discern a general linguistic structure for each of the five stories and examine the storytelleraudience interactions closely. We found that the stories matched the structure reported in [START_REF] Labov | The Transformation of Experience in Narrative Syntax[END_REF] almost exactly and that they, somewhat disappointedly, did not vary much across retellings. Instead there was remarkable consistency in their structure and content. We believe this is due to the frequency with which Joe and Noor tell their stories -their storytelling has become well-rehearsed [START_REF] Ladeira | Story worlds and virtual environments: Learning from oral storytelling[END_REF]. Despite this, their stories were not completely static and the variations we did observe arose, chiefly, due to two types of storyteller-audience interactions, which we translated into our prototype design.

First, audience questions: Audience members were free to ask questions at any point during a story. Typically, we noticed that they either waited for a pause in the storytelling or raised their hand and waited to be called on by the storyteller. Questions were also solicited by the storyteller explicitly inviting the audience to ask questions, usually at the end of a story. If no one asked questions immediately when invited, Noor especially, would drop hints for questions they could ask, for example: "You can ask me anything about District Six... games, gangsters, you name it, right?" Second, were exchange structures; an interaction initiated by the storytellers which matched the exchange structures defined in [START_REF] Sinclair | Towards an Analysis of Discourse: The English used by teachers and pupils[END_REF]. Periodically, a storyteller directed a question to the audience and then waited for responses. They were either looking for one or for a number of correct responses and the interaction would end when the audience gave enough correct answers. Storytellers usually handled incorrect answers by encouraging more answers. For instance, Noor often asked audiences to guess how he felt upon witnessing his home's demolition whereupon audiences offered responses, such as "sad", until reaching the answer Noor is looking for: "angry". In another example, he asks the audience to name some Cape Town townships, soliciting a number of correct answers before ending the interaction. An important observation about audience questions and exchange structures was that they always took place at the end of a narrative component and before the next began.

Our prototype design was based on the ethnography findings discussed in Section 4. We chose to implement a VE with interactive storyteller agents since we wanted a somewhat natural way for users to see the museum objects we were including and the likeness' of the Joe and Noor. However, the storytelling interactions we designed are orthogonal to a VE implementation and could be used in other storytelling applications. Fig. 1 shows the VE at the start: the user is part of a virtual audience facing two storytelling agents, modeled on Joe and Noor, and is able to move and look around using standard keyboard and mouse controls. The agents introduce themselves and, then, begin the first story, eventually telling five stories. A key design consideration in ensuring the prototype's sustainability was to make it possible for the museum to edit and improve on it easily. Thus, we chose a free implementation platform and ensured that it ran on an affordable desktop machine. We used Microsoft's XNA Game Studio and created the models in Blender 3D. The agents' animations were based on Joe and Noor's typical gestures and movements during storytelling. We noted these from video footage and had grown to know them well during the ethnographic study. The soundtrack was composed entirely of recordings gathered during the ethnography. While not our original intention, this allowed us to (a) present the stories as told spontaneously and (b) combine different retellings so that the VE presented, not one particular version of a story, but a composite version. Fig. 1. The storytelling virtual environment including the storytelling agents modeled after Noor Ebrahim (left) and Joe Schaffers (right). Also visible, are other audience members and museum objects: around Noor are pictures of his grandparents, son and former home and near Joe, pictures of his former home, a former main street and Apartheid-era public signs.

User Questions

Here we aimed to mimic the hand raising behavior observed in the museum. So, at any point during a story, the user may press the Space bar to 'raise their hand' and signal the desire to ask a question. The prompt, shown in Fig. 2, "You may press SPACE to put up your hand", serves to remind users of this. When the user presses space bar, a hand icon, also shown in Fig. 2, is displayed to indicate that the user's hand is up. The agent first finishes the narrative component that is in progress before acknowledging the user's question. The typing window, shown in Fig. 3, then appears for the user to type their question. The agent has a reserve of possible question responses and we use simple keyword matching to find an appropriate response. If no matching response is found the agent responds by saying "I don't know". Occasionally the agent invites the user to ask questions by saying something like "If you have any questions, raise your hand", and then waits for the user to press Space. If the user does not do this within a certain timeframe, a virtual audience member asks a question instead. This way, the VE does not wait indefinitely, or unrealistically long, on the user. We also devised a way of supplying the user with question hints: If the user takes longer than a certain time to either type a question or 'raise their hand' when invited, the VE displays keywords hints for questions the agents can answer. Fig. 2. While the Noor agent is telling a story the user is reminded that they may 'put up their hand' by pressing the space key (left screenshot). When the space key is pressed, a hand icon, right, appears and is displayed until the agent acknowledges the user's question.

Exchange Structures

The second interaction type, exchange structures, is initiated by an agent directing a question to the user. When this happens, the typing box is, again, displayed for the user to type an answer. Each exchange structure has a 'final correct answer'; if the user enters this answer, the agent gives positive feedback and the interaction ends. The agent is also able to give feedback to a number of non-terminating answers (which may be correct or incorrect answers). As before, keyword matching is used to recognize the user's input. Whenever the user enters a non-terminating or unrecognized answer, the agent encourages another answer, saying something like "No, try again", and the typing box is again provided. To ensure that this interaction does not cycle indefinitely, the user has three tries at answering a question before a virtual audience member answers. Following on from the exchange structure described Section 4, we created an exchange structure initiated by the Noor agent asking "What do you think? How did I feel?"; if the user types "sad", he responds with "Sad" and waits for another answer; if the user types "angry", he responds "Angry! That's the word! I was so angry!" and the interaction ends. If the user types anything else, he responds "And what else?" and waits for another answer.

Preliminary Results and Conclusion

While it is impossible to simulate all the nuance and interactivity of a real storyteller, we believe our VE recreates two interactions common in oral storytelling. To test the effectiveness of adding user questions and exchange structures to a digital storytelling experience, we conducted a user evaluation with 150 university students. Participants signed up voluntarily, and were each paid 50ZAR. Each participant first experienced a training VE to practice keyboard and mouse navigation as well as user questions and exchange structures, which were presented textually. Participants then experienced the storytelling VE with an unobtrusive experimenter in the room making ad-hoc observations. Preliminary results indicate a surprisingly positive response with many participants reacting to the prototype by giggling, speaking, exclaiming and nodding. Many approached the experimenter afterward to find out more about the prototype. Participants' comments suggest particular enjoyment of the VE's storytelling nature and the exchange structures. However, user questions were less successful with numerous participants reporting frustration at the agents' inability to answer their questions. A superficial examination of usage logs revealed that a large proportion of the user questions were, indeed, unsuccessfully answered. Currently, the prototype is limited by the number of recordings we had of Joe and Noor answering questions. We have since recorded Joe and Noor answering all those questions which the agents were unable to answer. These will be added to the VE to increase the agents' repertoire of answers. The user questions could also be made more effective using artificial intelligence techniques more sophisticated than our simple keyword matching. The final step in this project is a deployment of our prototype as a trial exhibit at the District Six Museum, the original setting that inspired it. We have found that observing real-life storytelling there was an invaluable design starting point. Linguistics and discourse analysis helped us identify structure in the stories we recorded and well-defined interaction patterns, allowing us to ultimately produce a prototype that users responded to positively. This paper describes only the first part of our work; we look forward to conducting more in-depth analyses of our user evaluation and forthcoming museum deployment towards understanding which aspects of our design were truly effective and how it could be improved.

Fig. 3 .

 3 Fig. 3. The typing box in which users enter user question and exchange structure input.

Acknowledgments. Our thanks to the District Six Museum, especially Joe Schaffers and Noor Ebrahim. Research partially supported by South Africa NRF Grant.