Ruobing Li

Bo Zhang

S Shyam Sundar

Henry Been-Lirn

Duh

Interacting with Augmented Reality: how does locationbased AR enhance learning?

Keywords: Location-based AR, modality, learning, transportation

Augmented Reality (AR) can provide additional information about mediated events, but can it enhance our learning and comprehension? We performed a user study of a location-based AR application in order to answer this question. A 2-condition (AR modality vs. non-AR digital book modality) between-subjects experiment with 36 dyads of secondary school students in Singapore was conducted to examine how the use of AR modality in an educational context impacts students' learning performance. Data from the experiment showed that location-based AR improved students' learning performance by catching their attention and enhancing their ability to elaboratively process the information they encountered. Theoretical and practical implications are discussed.

Introduction

Augmented Reality (AR) development is booming today, providing a contextual layer of extra information (e.g., images, locations, sound, etc.) to add on to user perceptions of the physical environment. Numerous applications have adopted the AR technology into their design, such as in the fields of gaming, entertainment, tourism, marketing, and social networking. However, as the industry is busy developing new apps to improve their design, little is known about the psychological effects of AR, and related user-experience outcomes. For example, AR is being touted as a promising tool for enhancing education, but we do not know how the use of an AR app compared with other digital educational tools can impact students' learning performance. Therefore, our study investigates the impact of an AR game, "the Jackson Plan," on students' learning performance, and attempts to explore the psychological factors that may mediate the relationship.

Augmented Reality

Augmented Reality (AR) is a new modality of presenting information, whereby a contextual layer of computer-generated virtual content is superimposed on a mediated representation of the real physical world. AR blends the real and virtual worlds seamlessly, constantly changing the ways we experience and interact with the physical environment. With the arrival of mobile media, this technology is on the verge of being adopted ubiquitously in our daily lives to enhance what we see and feel. In 2010 alone, AR mobile applications were downloaded more than eleven million times. By 2015, the number of downloads is estimated to exceed one billion per year [START_REF]Juniper Research Mobile Augmented Reality[END_REF]. Among these applications, for example, AR has been designed as a tourist guide on mobile devices [START_REF] Choubassi | An Augmented Reality Tourist Guide On Your Mobile Devices[END_REF], a shopping assistant in e-commerce websites [START_REF] Zhu | Personalized In-Store E-Commerce with PromoPad: An Augmented Reality Shopping Assistant[END_REF], and a way to promote learning in an educational environment. According to the 2011 Horizon Report, AR technology will be widely employed in US universities in the next two to three years, as a way to assist teaching inside or even outside the classroom [START_REF]The 2011 Horizontal Report[END_REF]. These educational AR applications include AR books, AR gaming, object modeling, skill training, etc., which aim to enhance students' learning.

Psychological Effects of AR

Due to its unique characteristics, AR may have a psychological effect on users in various ways. The most representative and appealing feature of AR is its ability to superimpose extra information into the real world based on sophisticated display technologies, which, on the one hand, increases the viewability of the interface and contextualizes the information, but, on the other hand, runs the risk of demanding too much user attention to encode the supplemental information. Limited Capacity Theory [START_REF] Lang | The limited capacity model of mediated message processing[END_REF] would argue that cognitive resources directed towards encoding the message come at the cost of storing the contents of the message for later retrieval, thereby resulting in poorer memory, and therefore inferior learning outcomes. While AR certainly runs this risk of privileging encoding over storage, many AR games and apps are tied to real-world locations, allowing users to view the world through a web camera on their devices with additional information superimposed by the app, which may provide users a sense of presence, or physically being in the environment [START_REF] Lombard | At the heart of it all: The concept of presence[END_REF], thus leveraging their engagement with the content (i.e., mental "transportation" into the narrative [START_REF] Green | The role of transportation in the persuasiveness of public narratives[END_REF]), eventually enhancing users' cognitive information processing and memory. Under this alternative view, AR will serve to enhance the user's "perceptual bandwidth" [START_REF] Reeves | Perceptual user interfaces: perceptual bandwidth[END_REF] such that the user is able to process more information more efficiently and thereby achieve an enhanced mental representation of mediated reality. Given these potential psychological paths to user outcomes, how AR can influence users' learning performance is unclear. Therefore, we experimentally examine how the use of AR modality in an educational context impacts students' learning performance as well as their user experience.

Method

Design and Participants

A 2-condition, between-subjects (location-based AR modality vs. non-AR digital book modality) experiment was conducted to answer the research questions. Participants were recruited from 3 Secondary One history classes in a public school in Singapore. The students were selected by a teacher to ensure social and cognitive homogeneity. In all, we had 72 students participating in the study. Since the procedure required students to work in groups of two, we had a total of 36 dyads for random assignment to the two experimental conditions. None of the students had learned about the historical event described in the application, which was a history class chapter for their grade, before the study, and all of them had some experience of playing games on handheld devices.

Stimulus Material

Two applications containing the same interactive game were developed for this study, one of which contained AR features while the other one was a digital book. The content presented by the two different modalities was identical. An Apple iPad 2 was used in both conditions as the primary device via which they experienced the educational content. The content followed a narrative structure. In the story, "The Jackson Plan" map, which was the original city plan map of Singapore, was stolen, and the participants were asked to locate the missing map by talking to different original immigrants (non-player characters or NPCs, see Figure 1) and collecting clues from their conversations. The story was created based on information about the founding of modern Singapore and the life conditions of its original immigrants. As the original immigrants' trading center, Singapore River was chosen as the physical corresponding location for the location-based application. While students were walking, a map of Singapore River (see Figure 2) would show up on the iPad screen, on which students' current location and their next destination were indicated. There were 6 destinations in all, representing the starting point, ancient trading spots of British, Chinese, Indians and Malays, and the final destination. Once the student arrived at a spot, the locationbased service would automatically trigger a NPC's appearance on the screen, and then the NPC started to talk to the student. Besides, four mini-games were embedded in the narrative story to help students understand and learn different immigrants' specific trading activities. Similarly, the mini-games were also triggered by the GPS. AR features were embedded in several steps. For example, when participants arrived at the destination, which was the statue of Raffles, they needed to capture information on the statue using the iPad to trigger a newspaper page on the iPad screen indicating who had stolen the map. While for the digital book application, students just needed to tap the iPad screen to proceed from screen to screen and get the newspaper page at the end, thus featuring no interactions with objects in the real world.

2.3

Procedure

Each dyad of participants was randomly assigned to one of the two conditions. In the experimental condition, students were taken to the Singapore River to complete their session (see Figure 3). A moderator was assigned to accompany each dyad of students. After briefly introducing the game, the moderator started the game and gave each dyad an iPad. The 18 dyads of participants in the control condition completed their session in a classroom in their school. After completing the game, each participant was asked to complete two paper-and-pencil questionnaires. Then, researchers bought each participant a meal, and asked them not to discuss the experiment with others. Two questionnaires were used to measure participants' learning and user experience.

One questionnaire contained 20 multiple-choice questions testing their ability to recognize factual information and knowledge embedded in the game, such as information about the history of modern Singapore's founding and the trading activities of original immigrants. Another questionnaire contained 40 items which participants rated using a 5-point Likert-scale ranging from 1 (not at all) to 5 (very much). Learning motivation was measured by 11 items, two of which pertained to intrinsic goal orientation (r=0.60, e.g., the most satisfying thing for me in playing "The Jackson Plan" is trying to understand the historical content as thoroughly as possible), 5 referred to task value (α=0.84, e.g., I think learning history in playing "The Jackson Plan" is useful for me to learn), and 4 to self-efficacy (α=0.86, e.g., I'm confident I can understand the most complex materials presented by "The Jackson Plan"). Learning strategy was measured by 7 items, among which elaboration was measured by 4 items (α=0.85, e.g., when playing "The Jackson Plan", I try to relate the material to what I already know) and peer learning was measured by 3 items (α=0.75, e.g., when studying for history, I will try to explain the material to a classmate or a friend). In addition, game engagement was measured by 7 items (α=0.80, e.g., I feel my mind wandering when I am playing "The Jackson Plan"). Telepresence was measured by 3 items (α=0.80, e.g., when I am playing "The Jackson Plan", I have a sense of "being there"). Perceived novelty of the application was measured by 5 items (α=0.77, e.g., the way in which "The Jackson Plan" was presented is novel). Transportation was measured by 7 items (α=0.84, e.g., I was mentally involved in the story while playing "The Jackson Plan"). Aside from these self-report measures, the applications in both conditions automatically recorded the time that participants spent on the application.

Results

We first conducted a MANOVA by considering all the dependent variables. The results showed a significant effect for modality, Wilks' Λ=. On all variables, participants in the location-based AR condition scored higher than their counterparts in the digital book condition. Since participants in the location-based AR condition spent much longer time (M=29.51, SD=4.73, counted in minutes) to complete the whole game than their counterparts in the digital book condition (M=9.40, SD=2.78), the analyses were repeated with time as a covariate. In addition, perceived novelty was also controlled to rule out its effect. Results of ANCOVA analyses showed that, with time and perceived novelty controlled, participants in the AR condition still reported higher level of task-value (F (1, 58)=4.06, p<.05), more self-efficacy towards learning history from the application (F (1, 58)=4.66, p<.05), more elaboration (F (1, 58)=3.95, p=.05), more engagement (F (1, 58)=5.03, p<.05) while experiencing the game, and showed better learning performance than those in the non-AR condition (F (1, 58)=3.96, p<.05).

Multivariate regression was performed to test the relationships between measured variables. The results showed that with time and perceived novelty controlled, participants' amount of elaboration during playing the game was significantly correlated with their leaning performance (b=.85, t (51)=2.04, p<.05). Similarly, participants' amount of elaboration (b=-.35, t (52)=-2.00, p<.05), engagement level (b=.24, t (52)=2.52, p<.05), and telepresence (b=.38, t (52)=3.31, p<.01) were significantly correlated with their transportation level.

A series of mediation tests using the PROCESS Macro [START_REF] Hayes | A Versatile Computational Tool for Observed Variable Mediation, Moderation, and Conditional Process Modeling[END_REF] was conducted to explore the mechanism underlying the effect of AR on students' learning performance and transportation level. The results showed that the modality influenced participants' learning performance through a three-step mediation model. The two modalities led to different levels of engagement, and participants' engagement further influenced their amount of elaboration, which in turn affected their learning performance, b=.32, 95% C.I. from .02 to 1.06, SE=.24 (see Figure 4). The modalities influenced participants' transportation level through two paths: firstly, engagement level of participants mediated modality's impact on transportation, b=.28, 95% C.I. from .01 to .92, SE=.21; secondly, modality influenced participants' engagement, and their engagement level influenced telepresence, which then in turn influenced the transportation, b=.30, 95% C.I. from .02 to .88, SE=.20 (see Figure 5). However, transportation did not predict participants' learning performance. Nor did it mediate modality's effect on learning. In summary, this study found that location-based modality could enhance students' learning performance by making them engaged in the task and facilitating them to elaborate on the information they encountered. Similarly, location-based modality was also more likely to transport users away from the physical world and make them immersed in the virtual narrative world. Users' enhanced engagement and telepresence mediated this process.

Discussion

Our findings show that location-based AR modality improves users' learning performance by enhancing their engagement and ability to elaboratively learn information from the application. The elaboration variable mainly referred to students' ability to put together information from different sources (e.g., narrative, mini games) and make connections between new information and their prior knowledge, history and the game event, virtual event and real world activities. The present study provides practical evidence that location-based AR modality, by enabling users to augment the real world environment with virtual features, generates greater engagement with the content, thereby improving learning performance. Therefore, AR holds promise a potentially useful avenue for educators to fully engage students and achieve desirable learning outcomes.

Our study also found that location-based AR modality had an effect on users' transportation level by leveraging their level of engagement and strengthening their feeling of presence in the virtual world. Since transportation is an important determinant of narrative's persuasive effect on audience, AR can serve a persuasive function [START_REF] Green | The Role of Transportation in the Persuasiveness of Public Narratives[END_REF]. Digital content designers who want users to be immersed in their content should try to use location-based AR modality to present the content, so as to provide users a feeling of actually being in the narrative world. This can be particularly useful for digital story-telling and other forms of informational, as well as entertainment, narratives delivered via devices that can support AR. It can also help marketers persuade users about their products and services by affording greater transportation into the narratives constructed.

A common concern with any new technology is that the technology's effect may be caused by the novelty effect, which will fade away as the technology becomes common. However, in our study, participants in the digital book condition reported slightly higher perceived novelty (M=3.54, SD=.76) of the application than their counterparts in the location-based AR condition (M=3.51, SD=.61). Therefore, we can rule out this alternative explanation in this study.

A limitation of this study is that although students worked in pairs, their data were collected and analyzed individually, without controlling for their dyad membership. Due to a technical failure in the study protocol, information about which two participants worked together was not captured by study administrators. Another, more important, limitation of this study is that while participants in the experimental condition completed their session in the outdoor environment, those in the control condition completed all the procedures in a classroom. Therefore, one may argue that it was the actual historical site, but not the AR technology that made the experimental stimulus outperform the stimulus of control condition. However, based on several factors, we believe this alternative explanation is unlikely. First, the Singapore River is totally different from what it looked like 200 years ago. The area used to be a trading center, but now all the old buildings were replaced or used as modern restaurants, so study participants were not able to get any relevant information about the history of Singapore or the immigrants' trading activities by simply being in the environment. Therefore, it is safe to argue that the real-world environment is not likely to influence students' learning performance. Besides, in terms of the transportation level, the real world environment is actually more likely to distract students from concentrating on and immersed in the narrative. Students had to walk for several minutes before they could continue to get new information from the application, so the whole story was not continuous for them. In addition, while they were walking along the river, a lot of students were attracted by lobsters and crabs in some restaurants' aquariums. Each of these two situations could decrease outdoor students' transportation compared to a classroom, but they still reported much higher level of transportation. Therefore, the location-based AR modality appears to be quite powerful in inducing transportation into the narrative despite the presence of formidable environmental distractions. This bodes well for the future of AR apps in the domain of education, especially in distracting out-of-school environments, because the technology itself appears to be quite involving and therefore conducive to learning.

Fig. 1 .Fig. 3 .Fig. 2 .

 132 Fig. 1. Non-player character

Fig. 4 .

 4 Fig. 4. Regression coefficients ( †p < .10, *p < .05, **p < .01, *** p < .001; solid lines indicate a significant indirect path). IV refers to independent variable and DV refers to dependent variable, which are modality and learning performance respectively in this model. The modality variable had two values; with 0 representing digital book condition and 1 representing the location-based AR condition.

Fig. 5 .

 5 Fig. 5. Regression coefficients ( †p < .10, *p < .05, *** p < .001; solid lines indicate a significant indirect path). IV and DV are modality and transportation respectively in this model.


Shota Suto

Susumu Shibusawa
email: sibusawa@mx.ibaraki.ac.jp

A Tabletop System Using Infrared Image Recognition for Multi-user Identification

Keywords: Tabletop system, Image recognition, FTIR, Multi-touch, User position identification, Area extraction

Many tabletop systems have been developed to facilitate face-to-face collaboration and work at small meetings. These systems often require users to attach sensors to their bodies to identify their positions, but attaching a sensor to one's body can be bothersome and annoying, and user position and posture may be restricted depending on where the sensor is attached. We have proposed a technique for estimating user position in a tabletop system by image recognition and implemented a tabletop system having a user position identification function incorporating the proposed technique. This technique first obtains touch points and hand-area information from touch operations performed by the user, and establishes an association between the touch points and hand from those positional relationships. Since the direction in which a hand is extended can be derived from that hand's touch information, the position of the user of the touch points belonging to that hand can be estimated. As part of this study, we also implemented a photo-object manipulation application, which has a function for orienting a photo object to face the user based on the results of the above user-position estimation technique. We performed an experiment to evaluate the position identification rate, and found that the proposed technique could identify user position with high accuracy.

Introduction

A tabletop system facilitates face-to-face collaboration at small meetings or any setting where people come together to work as a group. It enables all users to manipulate or examine displayed information and enables a variety of input operations to be executed by touching displayed information and performing gestures. Research surrounding tabletop systems with these features has been active including studies of interactive techniques [START_REF] Clifton | SketchTop: Design Collaboration on a Multi-touch Tabletop[END_REF], [START_REF] Haller | Shared Design Space: Sketching Ideas Using Digital Pens and a Large Augmented Tabletop Setup[END_REF], development of information display technologies [START_REF] Yamaguchi | An Evaluation of Perspective Correction Regarding Locations of a User and Displays[END_REF], [START_REF] Sakurai | A Tabletop Display with Visibility Control Using Revolving Polarizer[END_REF], and the use of tabletop systems for supporting collaborative work [START_REF] Funamoto | Support and Evaluation for Group Discussions in Seminar Type Lectures Using Table Top Interface[END_REF], [START_REF] Ohashi | Development and Application of Idea Generation Support System Using Table-Top Interface[END_REF], [START_REF] Goh | Interaction Design Patterns for Multi-touch Tabletop Collaborative Games[END_REF].

In a tabletop system, objects can be oriented in various ways by touch gestures performed by multiple users, which means that a user may find it difficult to understand the text or photo of an object that is currently not facing in the user's direction. In response to this problem, several techniques have been developed to identify user positions by attaching sensors to chairs or the users themselves [START_REF] Watanabe | Face-to-Face Collaboration System that Identifies Individual User[END_REF], [START_REF] Dietz | DiamondTouch: A Multiuser Touch Technology[END_REF], [START_REF] Annett | Medusa: a Proximity-ware Multi-touch Tabletop[END_REF], and these technique are used to automatically modify the orientation of an objects according to the position of the user manipulating that objects. For example, Diamond Touch [START_REF] Dietz | DiamondTouch: A Multiuser Touch Technology[END_REF], registers beforehand each user and the user's position by having each user sit on a conductive sheet for user-identification purposes, and uses this information to determine the position of the user whenever the user is identified. However, identifying user position by attaching sensors to chairs or people can be troublesome, and time must be devoted to learning how to use sensor equipment. User posture may also be restricted depending on where the sensor is attached.

On the other hand, research has been performed on an interactive system that extracts images of body extremities using image recognition technology so that physical movements performed by the user can be used as input operations [START_REF] Mamiya | Tabletop Entertainment System with a Finger Opening and Closing Gesture[END_REF]. This kind of interactive system using image recognition negates the need for wearing a sensor thereby enabling users to use the system in a free and natural manner.

We propose a technique for estimating user position by image recognition in a tabletop system and construct a tabletop system incorporating this technique. This technique negates the need for wearing a sensor and removes restrictions depending on where the sensor is attached. This system uses a frustrated total internal reflection (FTIR) multi-touch panel and obtains touch points and hand-area information from user touch operations using image recognition. It establishes an association between the touch points and hand from those positional relationships. Since the direction in which a hand is extended can be derived from that hand's area information, the position of the user of the touch points belonging to that hand can be estimated. This study includes the implementation of a photo-object manipulation application that enables users to manipulate photo objects on the tabletop system by touch operations. The application has a function for orientating a photo object to face the user according to the results of estimating user position from touch gestures.

2

Proposed System

System Configuration

For this study, we designed a tabletop system with a multi-touch panel capable of touch-point recognition by the FTIR method [START_REF] Han | Low-cost Multi-touch Sensing Through Frustrated Total Internal Reflection[END_REF]. This system features an infrared Web camera installed beneath the table to capture infrared images on the tabletop by picking up the acrylic panel on the table via a mirror. The system also includes a projector connected to a personal computer (PC) to display photo objects to users by projecting images onto the acrylic panel from underneath the table. In addition, tracing paper is pasted onto the acrylic panel to act as a screen and users are presented with information by having the projector project images onto this tracing paper. System configuration is shown in Figure 1.

Finally, to obtain information on hand area, an infrared light is installed on the ceiling above the table. Since a hand on the tabletop will block infrared beams emitted from this light, an infrared shadow corresponding to the hand will form. The Web camera is used to pick up this shadow effect.

Fig. 1. System configuration

Overview of User-position Estimation Technique

In a tabletop system, a user typically extends a hand from the edge of the table to manipulate some object. Accordingly, if it were possible to determine the direction from which the hand including touch points is being extended, the position of the user generating those touch points could be estimated.

In FTIR-based touch-point recognition as used here, the system picks up touch points as white light owing to the diffused reflection of infrared light. It also picks up the area occupied by a hand as a shadow generated by the blocking of light emitted from the infrared light installed on the ceiling. At this time, the system uses brightness values in the captured image to differentiate from the background image in two ways. Specifically, it extracts touch areas having a higher brightness value than the background and an area having a change in brightness value as a hand area. This extracted hand area includes the touch areas as a subset. Thus, when focusing on certain touch points, a hand area that includes those touch areas as a subset certainly exists, which means that the direction from which that hand is being extended can be determined from that hand-area information. As a result, the position of the user associated with certain touch points can be estimated since touch points and user position can be indirectly connected through the direction of extension of the hand to which those touch points belong. Extraction of these key areas and estimation of user position are outlined in Figure 2.

User-position Estimation Model

The FTIR touch-point recognition system picks up touch points as white light and the hand area as a shadow of infrared light. Establishing a difference with the background image enables the touch areas and the hand area that includes those touch areas to be extracted. An image of a touch area and that of a hand area extracted by background differentiation are shown in Figures 3 (a

User-position Estimation Technique

More than one hand area may exist on the tabletop at any one time. To recognize individual hand areas, the technique labels areas having connected pixels and assigns the label L to each hand area.

Then, when focusing on a certain touch point TP, that a hand area Hand that includes the coordinates of TP as an element exists. The results of labeling can therefore be used to extract label L[TP(x, y)] corresponding to the coordinates of TP. Here, L[TP(x, y)] is equivalent to label L[Hand] including TP. The hand area corresponding to the targeted touch point can therefore be identified by referring to L[TP(x, y)].

Referring to Figure 3(b), it can be seen that Hand consists of a continuous area connected to a certain edge of the image. Accordingly, if that edge can be determined, the direction from which Hand is being extended can likewise be determined.

System Implementation

Our prototype tabletop system has a height of 70 cm, a panel 100 cm × 90 cm in size, and a display manipulation range of 60 cm × 50 cm. The tabletop is shown in Figure 4.

We also implemented an application for manipulating photo objects by touch gestures. This application reads in image data as photo objects and displays them on the tabletop system. It treats touch gestures made by the user as input and generates results in response to those gestures. Touch gestures are listed and described in Table 1. Change direction is an operation which makes the object face the user when touching it with three fingers. An example of the change-direction gesture is shown in Figure 5.

User-identification Evaluation Experiment

To assess the accuracy of the user-position estimation technique presented in section 3.4, we conducted a subjective experiment using the 3-finger change-direction gesture.

After briefing four subjects on how to perform this gesture, we asked each to perform the gesture ten times on the tabletop system in each of the up, down, left, and right directions. At this time, we recorded actual user position and system-estimated position in order and determined position identification rate by comparing the values obtained. Specifically, denoting the number of times this gesture was performed as d act and the number of times that actual user position agreed with system-estimated position as d correct , we computed position identification rate by Eq. ( 1).

[ ]

The experiment was performed in the evening after sunset considering the possible effects of infrared light contained in sunlight. The prototype tabletop described in section 4.1 was installed in the center of a room and two infrared lights were installed on the ceiling above the table. Each light was 90-cm long incorporating six equally spaced infrared LEDs and the two lights were set 70 cm apart. The distance from the ceiling to the tabletop panel was 185 cm.

Results and Discussion

Average identification rate for the change-direction gesture by four subjects for each of the four tabletop directions and overall average for all directions are shown in Figure 6. Average identification rate for the change-direction gesture was found to be about 96%. Examining these results, it can be seen that hand extension from the up and down tabletop directions could be accurately identified. There were times, however, when hand extension from the left and right tabletop directions could not be accurately In the examples of Figure 7, the number of pixels in the hand shadow area crossing an edge are higher in the up direction than in the left and right directions. As a result, the system erroneously judges those tabletop operations to be those of a user positioned at the top of the tabletop (in the up direction). This is because the touch-point coordinates on the captured image cannot necessarily be used to correctly obtain the touch-point coordinates on the acrylic panel.

Conclusion

We proposed a technique for identifying user position in a tabletop system through image recognition and introduced a prototype tabletop system incorporating this technique. We also implemented a photo-object manipulation application for running on the tabletop and conducted a subjective experiment to evaluate the accuracy of the proposed technique in estimating user position. In this experiment, we recorded actual user position and system-estimated position when subjects made change-direction gestures and calculated the position identification rate by determining the rate of agreement between actual and estimated positions. We found that an average position identification rate of about 96% could be achieved in the case of change-direction gestures, which demonstrates that user position can be estimated by the proposed image Errors in identifying user position occurred in the system as a result of camera image compensation. An offset between the manipulation area and display area on the tabletop prevented the system from correctly recognizing how a hand shadow area was crossing an edge making it easy for erroneous judgments to occur. We considered that using a wide-angle camera would be able to capture the entire tabletop panel head-on, which should eliminate complicated image-compensation processing by the system and suppress this phenomenon. In future research, we plan to evaluate the position identification rate during simultaneous operations by multiple users and to add gesture functions making full use of user position identification. We also look to apply the proposed technique to face-to-face collaborative work systems and to evaluate its usability in such an application.

Fig. 2 .

 2 Fig. 2. Extraction of key areas and estimation of user position

  ) and (b), respectively.

Fig. 3 .

 3 Fig. 3. Extraction of touch-point and hand areas and their positional relationship

Fig. 4 . 5 .

 45 Fig. 4. View of tabletop Fig. 5. Change-direction gesture

Fig. 6 .

 6 Fig. 6. Identification rate for change-direction gesture

  of incorrect hand-extension identification from the left and right directions are shown in Figure 7. Images for a change-direction gesture from the left are shown in Figures 7 (a) and (b) and those for a change-direction gesture from the right are shown in Figures 7 (c) and (d).

Fig. 7 .

 7 Fig. 7. Change-direction gesture from left and right directions

  (a) Captured image (b) Hand area extracted (d) Hand area extracted (c) Captured image recognition technique. The technique is a simple technique for identifying the user position and negates the need for wearing a sensor.

Table 1 .

 1 Touch gestures

			3-finger touch
			Object direction
			changed to face user
	Operation	No. of Touches	Description
	Move	1	Move object
	Zoom in/out Rotate	2	Change object size Rotate object
	Change direction	3	Change object's direction to face user


Filipe Fernandes

Luís Duarte
email: lduarte@lasige.di.fc.ul.pt

Luís Carriço

Flow Specification Patterns of End-User Programmers: lessons learnt from a health mobile application authoring environment design

Keywords: Authoring Tool, Non-Expert Programming, Participatory Design

This paper discusses a set of interaction patterns encountered during the development of an authoring tool for mobile therapeutic applications. Unlike static paper artefacts, mobile applications can be enriched via the inclusion of complex behaviors. Typical examples include the definition of simple sequential interaction among all screens or the involvement of basic rules and triggers. As part of an ongoing project in which we are designing an authoring environment for mobile applications in clinical interventions, we studied how clinicians with no programming background were able to intertwine different screens from an application according to different rules. We were especially interested in comparing the approaches adopted using a low-fidelity prototype and using a high-fidelity version of the authoring tool. Results show that, despite a few technology induced strategies, users tend to mimic their actions using the paper based prototype in the corresponding hi-fi version.

Introduction

Clinical applications for mobile phones have been shown to improve a patient's state [START_REF] Carriço | A Mixed-Fidelity Prototyping Tool for Mobile Devices[END_REF][7] [START_REF] Mattila | Mobile Diary for Wellness Management -Results on Usage and Usability in Two User Studies[END_REF][16] [START_REF] Przeworski | Palmtop computer-assisted group therapy for social phobia[END_REF]. The increasing dissemination and power of smart-phones has further emphasized this possibility [START_REF] Carriço | Therapy: location-aware assessment and tasks[END_REF] [START_REF] Matthews | In the mood: engaging teenagers psychotherapy using mobile phones[END_REF]. Success cases for pathologies and therapy procedures as diverse as autism [START_REF] Boujarwah | REACT: intelligent authoring of social skills instructional modules for adolescents with high-functioning Autism[END_REF], fear therapy [START_REF] De Sá | Fear therapy for children: a mobile approach[END_REF], aphasia [START_REF] Matthews | In the mood: engaging teenagers psychotherapy using mobile phones[END_REF] or obsessivecompulsive disorder [START_REF] Herman | In vivo measurement of obsessive-compulsive disorder symptoms using palmtop computers[END_REF] are testaments on the benefits of technology. However, a significant number of these applications fall short to success for longer intervention periods [START_REF] Robinson | Aftercare intervention through text messaging in the treatment of bulimia nervosa -Feasibility pilot[END_REF]. Several factors can account for this outcome, among which the inability to personalize and adapt content [START_REF] Offer | To whom do adolescents turn for help? Differences between disturbed and non-disturbed adolescents[END_REF]. For instance, an application's presentation is typically the same for all users who download it. Yet, the expectations of a potential 8 year old user are quite different from those of a 45 year old patient [START_REF] De Sá | Fear therapy for children: a mobile approach[END_REF]. Also, the evolution of the patient's health status often requires adjustments that appli-cations are not ready to accompany. For example, monitoring thresholds vary, support messaging and data collection should be adapted to new clinical assessments [START_REF] Morris | Mobile Therapy: Case Study Evaluations of a Cell Phone Application for Emotional Self-Awareness[END_REF] [START_REF] Offer | To whom do adolescents turn for help? Differences between disturbed and non-disturbed adolescents[END_REF].

The origin of this application stiffness builds on many factors. The complexity of the technology and of the application domain is certainly one of those reasons. In fact we believe that it is one of the most important factors: the dichotomy and complexity of knowledge involved. Information technology (IT) engineers and researchers understand technology and are able to handle its complexity. Clinicians on the other hand comprehend patients and the protocols they must put forward to provide them a better quality of life. Combining the two knowledge sources is no easy task. It gets worse because both knowledge domains evolve rapidly as well as the ultimate target, the patient wellbeing.

The usage of authoring tools is a possible solution. Past works prove valuable evidence in tackling similar situations [START_REF] Boujarwah | REACT: intelligent authoring of social skills instructional modules for adolescents with high-functioning Autism[END_REF]. These tools aim at joining two knowledge sources to a middle ground. They provide domain experts the mechanisms to customize and deeply adapt the applications, refining and embedding it with clinical decisions. For that, the tools must hide the technology complexity under a hopefully welldefined set of components, developed by IT staff. In the end they offer means for enduser programming [START_REF]End-User Development[END_REF]. Finding that adequate middle-ground can be complicated. It is not just about usability. It is also about programming and domain concepts, and ultimately the domain experts' perception of its combination. Available systems often recur to state charts or similar representations of the programmable elements [START_REF] Hartmann | tools: Visually Prototyping Physical UIs through Statecharts[END_REF]. Unfortunately and despite the existence of a few systems in this area, some design aspects were left unaddressed by researchers and IT experts alike. Our research focuses on one of these issues: the way information is organized as far as connections between programmable elements is concerned. We believe that sequential transitions between screens of a mobile application may be sufficient for some domains of intervention (e.g. mobile app prototyping [START_REF] Carriço | Therapy: location-aware assessment and tasks[END_REF][11] [START_REF] Landay | SILK: Sketching Interfaces Like Krazy[END_REF]). Nevertheless, other domains possess a more critical nature which requires richer and more complex connections.

In this paper we present the design process of DETACH -DEsign Tool for smartphone Application Composition in the Health domain. The tool targets therapists and clinicians in general, and aims to provide them support to compose ubiquitous applications. For that, it offers a set of building blocks, as predefined, yet customizable, mobile app screens (e.g. a message panel, a mood selection panel). These can be assembled and customized to address the particular patients' needs. As such, domain experts are the tool's end-users and thus responsible to define the applications according to the adequate clinical procedures and evolution. Particularly relevant was the tool development process. We aimed at understanding what concepts and behaviors a domain expert formulates and expresses to grasp the programming endeavor itself. We were particularly focused in assessing how these end-users connect and define the control of flow between these building blocks, as this emerged as the most difficult task to assimilate. The assessment started in the early design stages and the understanding of users' behavior in low-fidelity prototyping, and transitioned into how that behavior was translated into the high-fidelity (hi-fi), and its evaluation with end-users.

Sketching DETACH: Participatory Design of Low-Fidelity Prototypes

The DETACH's development process started with the creation of specific mobile applications, addressing cognitive behavior therapy procedures. Currently several of those applications are being used in clinical trials and for all of them therapists provide continuous screening and comments [START_REF] Carriço | A Mixed-Fidelity Prototyping Tool for Mobile Devices[END_REF]. Several brainstorming sessions were conducted including a team of expert HCI researchers and the therapists specializing in different types of interventions and pathologies. DETACH's development process was heavily rooted in participatory design [START_REF] Bergvall-Kåreborn | Participatory design: one step back or two steps forward?[END_REF][13] and thinking aloud sessions not only with therapists but clinicians in general (Fig. 1). We have conducted a substantial set of meetings to understand: a) their current requirements and expectations for a tool of this nature; b) how modern smart-phones can improve existing therapeutic practices without disrupting established procedures; c) interaction patterns employed by our stakeholders when organizing different elements of an application. For each trial, participants were asked to design an application to support a child undergoing a fear therapy [START_REF] De Sá | Fear therapy for children: a mobile approach[END_REF] treatment and containing 4 screens: screen 1 allowed the child to select his / her current emotional state from a set of pictorial representations (e.g. happy face, frowny face, etc.); screen 2 presented a question (with "yes" / "no" answers) asking whether the child was still feeling scared or not; screen 3 was presented if the child answered "no" in screen 2 and displayed a descriptive support text regarding typical procedures for fear therapy; screen 4 was a congratulatory animation which was displayed either after screen 3 or if the child responded "yes" in screen 2. Subjects were assisted as required if they were insecure about screen details and other technical details. They were given full freedom to organize the screens.

Each session started with the researcher explaining the purpose of the authoring tool and its main features. Clinicians were then briefed regarding the mobile application they would be creating during the session. Finally they were provided with all the necessary material to create the screens: pen, pencil, eraser, different colored post-its and white sheets of paper (Fig. 1). In order for us to study how clinicians would deal with the provided material, some of the sessions had available additional template screen mockups and/or a guidance text of a specific application for them to represent.

Fig. 2.

Examples of screen transition representations on low-fi prototypes.

Results and Discussion

Upon completing the creation of the mobile application using the low-fi DETACH prototype, subjects were asked to explain us the rationale behind their strategy to organize and connect the screens. Some notable approaches emerged from this trial:

-Subjects position each screen at their will on the sheet. Transitions between each screen are represented by arrows, complemented with side notes either using plain text or a small post-it above the arrow line (Fig. 2-right).

-Some participants preferred to organize screens on a sequential fashion, touching each other to pinpoint the transitions. Special connections such as the one stemming from screen 2 to screen 3 were addressed by separating one of the screens slightly from the main group and annotating the rules associated with that transition (Fig. 2left).

-New screens emerged. Interestingly some of them addressing physiologic sensing. More importantly a new type of transition and condition was created by some (more active) clinicians, featuring an interruption of the normal flow. The example in the figure (Fig. 2-right, lower left corner) refers to: whenever heart beat raises above something show this screen.

Testing the High-Fidelity Prototype

Based on stakeholder feedback, we designed the first DETACH's high-fidelity prototype (Fig. 3). The tool is a web application offering a workspace on which therapists can populate screens based on different templates and connect them according to a set of rules whose complexity may vary (e.g. from basic screen sequence to transitions based on previous patient answers). The top section displays the available screen templates (e.g. multiple choice answer, animation display, etc.). Therapists can drag a template into the center canvas and configure each screen's particular elements in the rightmost panel. Transitions between screens (represented by arrows) are defined by selecting a screen, hitting a button and finally stipulating the rules which trigger the transition (e.g. if the transition considers the selected element as the destination or its source, how is the transition triggered).

High-Fidelity Prototype Trials

The high-fidelity prototype participatory design sessions occurred roughly 4 months after the low-fi ones. This trial was performed by the same clinicians who had been present in the previous experimental period. To provide the most pleasant and relaxed experience to our subjects, testing occurred again in the clinicians' offices. The highfidelity prototype trials were carried out in a laptop (HP Pavilion dv9890ep model).

Regarding screen organization on the prototype's canvas, subjects adhered to two main strategies: a) akin to the low-fi trial, users positioned the screens sequentially, typically reflecting creation order; b) users delineated imaginary columns upon which screens were positionedif a screen generates two navigational branches (i.e. users can transition towards two different screens from the same origin) then all destination screens would fit inside the same virtual column in the canvas, with the branches collapsing into another screen if that was the case. A substantial number of participants did not show any particular spatial strategy here (as observed in Fig. 3).

Results and Discussion

We were able to identify a substantial number of strategies employed by our participants regarding the organization of the elements they were able to interact with. While there was a slight degree of variation in the presented approaches, we were able to cluster them into well-defined interaction patterns. Touch to Connect. One of the approaches observed rooted itself in the low-fi version of the prototype. Clinicians (approximately 25%) often aggregated screens which had a sequential transition nature together in the canvas. A particular behavior was noted: subjects often organized the screen post-it in a way resembling a deck of cards, with each screen slightly touching each other it had a connection with. Transition rules were either described using side textual notes or by adding smaller post-its covering the screens involved. On the hi-fi prototype, these same users attempted to link screens using a different strategy: they dragged one screen towards another, "touching" it. The expectation was that a new transition was established between the "touched" screen and the dragged one. This strategy clearly shows a sequence oriented thinking towards building mobile applications. When confronted with the possibility of adding additional rules for these transitions (e.g. based on patient inserted content / answers) they argued that this still felt like the most natural way to interact with the elements in the canvas. They also reiterated the environment should leave the possibility of editing the transitions to add behaviors on top of the basic sequential connections (a feature already present, but not tested by these subjects).

Origin-Destination Paradigm. The most popular strategy adopted by users was inspired by the way they typically fill-in a postcard, an e-mail or a letter: they define the origin of the connection, the destination and then any related content with them. Approximately 55% of our subjects adopted this approach arguing "this is the way I naturally write" and "the way I did on the paper version". One may ask if using the same subjects and the previous experience with the low-fi prototype could influence this result: in part, we agree, but we must also note that a substantial number of clinicians did not follow the same connection strategies; also the timespan between both trials dissipates some of the "training" acquired in the first trial. A minority of the clinicians (roughly 10%) approached this paradigm by completely switching the connection's order definition: they started by selecting the destination screen and then they picked the screens which would transit to the former. When asked to verbalize why they adopted this strategy, they argued "it made sense, considering a patient can reach the same screen from different branches", so "defining the destination first felt straightforward". Here, we must state such decision may have been influenced in part by the mobile application they were asked to create, since it featured a screen which could be reached from two different navigational branches. However, this behavior was only noted in the hi-fidelity prototype.

Connection from Screen Elements. The last adopted strategy pertains only to the hifidelity prototype participatory design sessions. When selecting and configuring each screen, some clinicians attempted to generate connections from the screen's components themselves (e.g. each answer, a button, etc.), justifying their behavior stating "the patient will transition to another screen if he / she presses this button". Even though no participant had previous programming experience, this is an approach reminiscent of existing Integrated Development Environments (IDE) such as Visual Studio or Eclipse and highly connected with event-based programming. In these tools, users may click a component, such as a button, to configure the application's behavior when the button is pressed. It is interesting that despite the absence of experience, some participants actually prefer this strategy.

Implications for DETACH. The main findings from these design sessions pertain to the variety of approaches clinicians were able to adopt to accomplish the same goal. Nevertheless we must prioritize the strategies which gathered more followers while at the same time not neglecting the preferences manifested by some of our participants. As such, DETACH's primary approach towards the definition of screen connections will follow the origin-destination paradigm. All other strategies will be incorporated into the environment as user preferences. Ideally, the application should be configurable for each user, storing their preferences and preferred workflow strategies.

Conclusions & Future Work

Our research points that non-expert programmers embrace desktop and paper metaphors in a virtual environment. We observed that our subjects employed similar screen organization strategies in the authoring environment regardless of operating a low-fi or hi-fi prototype during the participatory design sessions. It is important to stress that DETACH's final design will reflect these findings, to alleviate the technology transition impact which our stakeholders will be subject to.

We are finishing the development of DETACH, currently focusing on the mobile application emulator and the XML specification that will be used in Android smartphones to recreate the therapists' designs. The final step of this research will encompass a set of clinical trials in which we will assess whether authored applications can foster patient commitment when compared to previous non-authored digital artifacts.

Fig. 1 .

 1 Fig. 1. Application representation made by one of the participatory design users.

Fig. 3 .

 3 Fig. 3. Resulting representation of one of the thinking aloud sessions for the initial application.