Zilouchian Roshanak

Brian P Moghaddam

Christina Bailey
email: bpbailey@illinois.edu

Poon
email: cpoon2@illinois.edu

IdeaTracker: An Interactive Visualization Supporting Collaboration and Consensus Building in Online Interface Design Discussions

Keywords: Design, open source software, user interface, visualization

With the rapid growth of open source and other geographically distributed software projects, more interface design discussions are occurring online. Participation in such discussions typically occurs via issue management systems or similar interactive discussion forums. While such systems have a low learning curve, they do not support key elements of design discussion such as comparing alternatives, maintaining awareness of the arguments for and against the alternatives, or building consensus. To better understand these and other challenges, we conducted a study of online interface design discussion. The study consisted of analyzing a large corpus of online discussion content and conducting interviews with designer and developer participants. We discuss the findings of our study and use them to motivate the implementation of an interactive visualization tool -IdeaTracker. The tool offers explicit support for tracking and comparing ideas and gaining an abstract summary of the overall discussion as well as specific alternatives. It also provides a voting system to support consensus building. The tool extracts and visualizes useful information from the discussions that would otherwise be hidden but without interfering with the current method of participation. Our tool is compatible with the issue management system of one open source project but can be extended for others. Initial user feedback is positive and confirms the need for an alternative visual representation of interface design discussions online.

Introduction

The User Interface accounts for a large part of the design and development effort in the production process of any interactive software system [START_REF] Myers | Survey on user interface programming[END_REF]. With the number of geographically distributed software projects and, in particular, open source software projects growing [START_REF] Deshpande | The Total Growth of Open Source[END_REF], more interface design and development discussions are unfolding online. For example, in open source projects, designers typically engage in interface design discussions through the project's issue management system. These systems offer interactive Web-based forums for discussing design issues. These forums are similar to other types of Web-based newsgroups and social media where participants discuss a variety of topics from food and books to religion and politics.

However, interface design discussions have many unique characteristics that are ignored by the current metaphor of interactive forums. For example, common practices that are paramount to producing effective interface designs such as generating, discussing, and comparing multiple ideas [START_REF] Brown | Design problemsolving: Knowledge structures and control strategies[END_REF][START_REF] Vora | Areviewof designmodels and a proposal for a taxonomy of design[END_REF] are not specifically supported. This problem is exacerbated due to the diverse levels of expertise and viewpoints of the discussion participants [START_REF] Muller | Retrospective on a year of participatory design using the PICTIVE technique[END_REF].

A few studies have analyzed design discussions in open source software. For example, Barcellini et al. studied software discussions occurring in mailing lists with the goal of extracting design relevant information [START_REF] Barcellini | A Study of Online Discussions in an Open-Source Software Community[END_REF]. Unlike their study, our work focuses on analyzing and enhancing design discussions relating to the interface of a software project. Twidale et al. analyzed usability reports from Bugzilla [START_REF] Twidale | Exploring Usability Discussions in Open Source Development[END_REF]. They recommended several improvements such as enhancing the classification of usability bugs and adding explicit representations of design arguments to the reports. However, these enhancements were never implemented and their study considered only usability bug reports rather than, e.g., proposals for new interface features.

Various tools have also been developed to support distributed interface design. For instance, Designers' Outpost [START_REF] Everitt | Two worlds apart: bridging the gap between physical and virtual media for distributed design collaboration[END_REF], Synergy [START_REF] Liapis | Synergy: a prototype collaborative environment to support the conceptual stages of the design process[END_REF] and the Envisionment and Discovery Collaboratory [START_REF] Arias | Transcending the individual human mind -creating shared understanding through collaborative design[END_REF] are examples of collaborative design tools that can be used for colocated or distributed team situations. In addition, a myriad of tools have been developed to support the creation of design artifacts in distributed settings [1][2][3]. Though these types of tools can facilitate the early stages of interface design, our work is original in that we are targeting improving online discussions of interface design issues. Though our current focus is on open source projects due to the public availability of discussion data, we believe our results are applicable to any situation where Web-based interactive forums are being used as the primary means for discussing interface design issues.

To better understand designers' challenges participating in online design discussions, we studied a large corpus of interface design discussions from two popular open source projects: Ubuntu and Drupal. In our study we analyzed the content of 1560 messages collected from thirty discussion threads from the Ubuntu and Drupal issue management systems. Figure 1 shows an example of a design discussion extracted from the Drupal issue management system. Our analysis included counting the posts, alternatives, and participants per discussion as well as the fluidity of the topic flow. Our data analysis was complemented by interviews with UI designers (N=6) and developers (N=6) participating in the design discussions. The interviews helped interpret the results from the content analysis and uncover the current nature of participation, idea generation, and the consensus building process.

From data analysis and interviews, we found that designers are struggling to track the design alternatives and the arguments for and against each alternative, want to maintain better awareness of others' preferences, and prefer that the interface design and development activities remain tightly integrated. We have incorporated these and other findings into the design of a new interactive visualization tool called IdeaTracker for managing online interface design discussions. The visualization highlights the proposed alternatives in a visual timeline, allows participants to filter and review messages that refer to each alternative, and provides an abstract visual summary of each message to indicate length and affective tone of each message. IdeaTracker also helps with decision making by enabling designers to compare selected subsets of alternatives and supports consensus building by implementing a voting system. We believe that using IdeaTracker can result in more effective design discussions online, which can lead to higher quality interfaces. Initial user feedback is positive and confirms the need for an alternative visual representation of interface design discussions online.

Our work makes two contributions. First, we identify key challenges participants face when engaging in interface design discussion online, which include tracking and comparing alternatives, maintaining awareness of community opinion, and building consensus around specific alternatives. Second, we demonstrate how these challenges can be addressed through the implementation of a novel interactive visualization tool. The tool extracts and visualizes useful information from the discussions that would otherwise be hidden but without interfering with the current method of participation. The interface design problem discussed in this thread is the lack of visual connection between a selected tab and its content. An alternative has been proposed along with the problem description. In the first few comments, participants are discussing the scope of this problem and whether it is a valid issue.

Related Work

We describe prior work for representing online content in different domains and how our study builds upon and extends that work for online interface design discussion. Also, we describe open source and the usability enhancement movement within open source and how our work can aid this movement.

Distributed Collaborative User Interface Design

A myriad of tools have been developed to support creation of design artifacts in distributed settings. For instance Axure [1], Pencil [3], and Balsamiq [2] are part of a First Alternative large class of tools that have been developed to manage online collaborative creation of interface mockups. These tools assume that collaborators utilize existing tools such as interactive discussion forums or synchronous messaging systems for discussing the designs. Other tools have been developed to support remote collaboration in UI design. For instance, Distributed Designers' Outpost is a collaborative web site design tool that captures the history of an evolving information architecture [START_REF] Everitt | Two worlds apart: bridging the gap between physical and virtual media for distributed design collaboration[END_REF]. Synergy is another collaborative design environment that supports remote collaboration during the early stages of the design process [START_REF] Liapis | Synergy: a prototype collaborative environment to support the conceptual stages of the design process[END_REF]. However, there has been little research into interfaces that specifically support the unique nature of online design discussions. We have taken a first step in this direction by first understanding specific challenges of such discussions and showing how they can be addressed in a novel visualization.

Visual Representation of Online Content

A visual representation of online content expressed in narrative form can foster sense making of the information. Such visual representations have been created for visualizing content in myriad online communities. For instance, Opinion Space is an online interface designed for collecting and visualizing users' opinions from comments in online news articles, blogs, videos, and product reviews [START_REF] Faridani | Opinion space: a scalable tool for browsing online comments[END_REF]. Shared Space is another tool that analyzes students' chat messages to visualize discussion and agreement during online discussions [START_REF] Janssen | Online visualization of agreement and discussion during computer-supported collaborative learning[END_REF]. The Conversation Map system creates a graphical interface by analyzing the text of archived newsgroup messages. The interface can be utilized to read and search all of the messages in the archive [START_REF] Sack | Conversation map: a content-based Usenet newsgroup browser[END_REF].

All of these tools were developed to represent a specific type of discussion content in an online community: Opinion Space represents opinions, Shared Space represents chat messages, and Conversation Map shows newsgroup posts. Similarly, interface design discussions have certain elements that, if incorporated into the design of a visual representation, can foster participation and improve the quality of discussions. Inspired by this previous work, this paper explores the design of an interactive visualization tool that supports interface design discussions, capturing the unique characteristics of this content.

User Interface Design in Open Source

A usability movement has been ongoing in the Open Source community [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF][START_REF] Benson | Professional usability in open source projects: GNOME, OpenOffice.org, NetBeans[END_REF][START_REF] Frishberg | Getting to know you: open source development meets usability[END_REF][START_REF] Muller-Prove | Community experience at OpenOffice.org[END_REF][START_REF] Nicholas | The usability of open source software[END_REF][START_REF] Terry | Perceptions and practices of usability in the free/open source software (FoSS) community[END_REF][START_REF] Thomas | Why Free Software has poor usability, and how to improve it[END_REF]. Corporations were one of the first contributors to this movement by providing interface guidelines and recruiting usability experts to work on open source projects [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF][START_REF] Benson | Professional usability in open source projects: GNOME, OpenOffice.org, NetBeans[END_REF][START_REF] Nicholas | The usability of open source software[END_REF]. The open source community itself helped this movement by promoting new techniques such as "Design-by-blog" where personal blogs are used for design discussions [START_REF] Nicholas | Usability processes in open source projects[END_REF]. In many projects, screenshots can now be posted in mailing lists and issue management systems which facilitate design discussion [START_REF] Twidale | Exploring Usability Discussions in Open Source Development[END_REF]. Also, usability experts have started joining interface design discussions in open source through blogs, issue management systems, and wikis [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF][START_REF] Zilouchian | Open Source Interface Politics: Identity, Acceptance, Trust, and Lobbying[END_REF]. As more designers, usability experts, and developers engage in discussions of interface design issues for open source, there is a growing need for studies to characterize their participation challenges and identify feasible ways to address them.

As a starting point, several studies have been conducted to identify the challenges of improving the user interfaces of open source projects [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF][START_REF] Benson | Professional usability in open source projects: GNOME, OpenOffice.org, NetBeans[END_REF][START_REF] Frishberg | Getting to know you: open source development meets usability[END_REF][START_REF] Muller-Prove | Community experience at OpenOffice.org[END_REF][START_REF] Nicholas | The usability of open source software[END_REF][START_REF] Terry | Perceptions and practices of usability in the free/open source software (FoSS) community[END_REF][START_REF] Thomas | Why Free Software has poor usability, and how to improve it[END_REF]. Some solutions propose to resolve the challenges by funding usability experts to work on open source projects, adapting automatic approaches for evaluating interfaces, encouraging HCI students to join open source, and providing a social and technological infrastructure for usability experts [START_REF] Nicholas | The usability of open source software[END_REF]. Bach et al. expanded this last solution by arguing for a separate design space for usability experts at the same level as developers in the project's information (and social) architecture [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF].

Our work shares the long-term goal of improving the user interface of open source projects, but emphasizes enhancing the discussions of interface design issues, thereby leading to more effective participation. In contrast to Bach et al., our approach is to provide a more effective visual representation of the existing design discussions, which involve both designers and developers, rather than advocate for a separate discussion space targeted only for designers.

Finally, in relation to a prior study of online interface design discussion for open source [START_REF] Twidale | Exploring Usability Discussions in Open Source Development[END_REF], our work is original in that it analyzes a broader sample of design discussions, complements this analysis with results from stakeholder interviews, and implements the findings within an interactive visualization tool.

A Study of Online Interface Design Discussion

We studied the evolution of interface design discussions in two well-known open source projects: Ubuntu and Drupal. Ubuntu is a popular Linux distribution and Drupal is a widely used content management system. We chose these two projects because they have an active interface design team, the design discussions are publicly accessible, and they contain a considerable amount of lively design discussion.

Within both of these projects participants post interface design problems, solution proposals, and related comments to the respective issue management system. These issue management systems are Web-based interactive forums dedicated to discussing software related issues including interface design issues. These forums are similar to the interactive forums used for discussing topics of interest in many other online communities or social media sites.

As shown in Figure 1, each interface design discussion (a new thread) starts when a participant posts the description of a design problem. Other participants can then contribute to the discussion by proposing design alternatives, arguing for or against the proposed alternatives, attaching an implementation of an alternative (called a patch), reporting the results of a patch review, or raising other concerns such as clarifying the scope of the problem or how it relates to other ongoing design efforts.

Generating multiple alternatives and having an engaged discussion are critical for producing an effective solution to the design problem and having confidence in it. Designers also need mechanisms to share opinions and increase awareness of others' perspectives to facilitate consensus. This is difficult even during face-to-face design meetings, yet the issue management systems used for interface design discussions only support text postings along with the option of attaching an image. They do not have explicit support for key elements of the discussion itself such as tracking which ideas are favored, arguments for or against the ideas, or building consensus around specific alternatives. Our study therefore centered on answering these questions:

1. How many ideas are typically shared in online design discussions? How extensive is the debate? How do designers maintain awareness of the ideas? 2. What techniques are used to promote consensus around specific alternatives and how effective are these techniques? 3. What are the strengths and weaknesses of the issue management systems used for discussing and managing ideas? 4. What are other challenges that designers encounter participating in online design discussions and what strategies are used for addressing them?

To answer these questions, we analyzed content from online interface design discussions and conducted interviews with participants. For the quantitative analysis, we examined 1560 messages that spanned thirty discussion threads. Fifteen of the threads came from a pool of 300 in Ubuntu (average number of messages in each thread=35.4 sd=12.9). The other fifteen come from a pool of 500 threads in Drupal (average number of messages in each thread=68.6 sd=23.0). In both cases, we selected the fifteen threads from the most active threads. From inspecting a sample of the threads; we defined two characteristics for 'activity;' (i) the number of images, where we considered an image to be a proxy for a proposed design alternative and (ii) the number of messages in that discussion. We rank ordered the pool of discussion threads based on these criteria and selected fifteen threads from the top fifty threads in each project.

We then analyzed the content of the messages. First, we divided the message into a smaller set of topical chunks. A new chunk was created when there was a transition from one topic to another. Then, we manually coded the chunks into eight categories: Issue, Alternative, Criterion, Clarification, Project Management, Implementation, Digression, and Other. We adapted these categories from the coding scheme developed by Olsen et al. [START_REF] Olson | Small group design meetings: an analysis of collaboration[END_REF] for capturing and analyzing the core elements of design discussions in collocated settings. Though the original coding schema had eleven categories; our adaptation only used seven. When testing the schema and resolving inconsistencies (based on five discussion threads from each project), the evaluators agreed that the other four categories were not applicable. An implementation category was added to capture the technical messages in the discussions.

The data analysis was complemented by a set of semi-structured interviews with twelve participants from both Ubuntu and Drupal projects. We interviewed six active designers, three from Drupal with an average of 7 years of experience (sd = 1) and three from Ubuntu with an average of 7 years of experience (sd = 3.26). We will refer to Drupal designers and Ubuntu designers as DD# and DU# accordingly. We also interviewed six developers participating in resolving UI design and usability issues: three from Drupal with an average of 6.5 years of experience (sd = 3.51) and three from Ubuntu with an average of 7.5 years of experience (sd = 4.5).

Each interview lasted about an hour and was conducted via phone and instant messaging. The latter was used to share Web links, images, and other data artifacts during the interview. The subjects were compensated with a $30 Amazon gift card.

The interview questions reflected the main research questions of our study and were informed by prior work (e.g. [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF]). For example, the questions included: what are the main challenges in discussing design ideas online? How do you maintain awareness of the ideas? What techniques are used to promote consensus? What are the strengths and weaknesses of the issue management systems for discussing ideas?

Challenges and Implications

We discuss key challenges that designers encounter while participating in interface design discussions online and the methods used for addressing those challenges. Although our study focused on open source projects, the challenges identified were mostly due to the intersection of the interaction design of the forums and the topic of the discussion. As a result, we believe these observations apply to any interactive forum which serves as the primary means for discussing interface design issues.

Designers Struggle to Track Design Alternatives

A total of 299 alternatives (ideas) were submitted in the 1560 messages analyzed. On average, nine alternatives were proposed in each thread of discussion (sd=5.88, max=27, min=0), indicating that multiple alternatives were welcomed and considered for each design problem. Of all the alternatives proposed, 63% were described solely in narrative form, 13% included only a patch (a file that updates the implementation to provide a working preview of the idea), and 18% included only a screenshot of a proposed solution. Figure 2 shows a distribution of alternatives for each combination of modality. Although we chose part of our data set from the discussions with the most images, more than half of the alternatives were presented in narrative form. This is partly due to the lower cost of expressing alternatives in narrative form. Other reasons for having fewer visual representations may be the text-based structure of the discussion threads and lack of emphasis on the importance of visually demonstrating alternatives by the discussion interface. However, all of the designers were aware of the importance of screenshots and other visual representations in explaining a design alternative. They mentioned that alternatives with visual representations have a better chance of receiving comments from community members. For instance DU2 said: "…wireframes or other visual stuff legitimize ideas. Because they're very memorable.

[…] People comment on images more, or codes more than paragraphs."

One way to improve the discussion interface is to better emphasize the importance of alternatives and the significance of providing visual representations for them by adopting a more visual structure.

Confirming the importance of alternatives, our quantitative analysis revealed that 48% of the conversation is spent discussing alternatives (Figure 3). Designers also mentioned alternatives as a vital piece of information when contributing to a design discussion. As DD1 said: "[When participating in a new discussion] I would want to know, in its current state, what is the exact problem the issue is trying to deal with, what are the proposed solutions so far, and what direction have people been taking on each of the proposed solutions." However the current systems do not support tracking the proposed alternatives. As DU3 said: "They (bug reports and mailing lists) aren't good for keeping track of all of the ideas. […] There usually isn't anybody keeping track of these are all the possible options and these are some [discussions] about each option." Today, the only way a participant can maintain awareness of the proposed alternatives is to rely on their memory, maintain notes in a separate tool, or scan the entire discussion thread to review the alternatives and rationale for and against each of them. The first method is unreliable as human memory is fragile and has a limited capacity [START_REF] Baddeley | Working memory[END_REF]. The latter two methods are cumbersome and do not promote shared awareness among participants.

One of the consequences of using an interactive forum for managing a design discussion is that many of the proposed alternatives get buried in the midst of the discussion. As DU4 said: "Some ideas that people have are actually really good, but then they kind of get lost in the thread…" This means that not all alternatives will be evaluated thoroughly or may be forgotten during a lengthy discussion. For example, within these threads, the proposal of design alternatives is distributed throughout the discussion, rather than batched at the very beginning reminiscent of commonly applied face-to-face brainstorming techniques. This may be a consequence of the distributed nature of the discussion in time and space as well as its integration with an issue management system. It may also reflect the fact that participants are able to generate and submit new alternatives as a function of the ongoing discussion. However, an important consequence of having the alternatives distributed throughout the discussion is that some may be overlooked or even fade from community memory. As DU3 explained: "Even if 20 or 30 ideas get generated during the mailing list discussion, a few days on people will be discussing [only] one or two which might be the worst ones because they might be the most controversial."

A related pattern is that the topic of the design discussion changes frequently. For example, in the first thread, there is a topic change after A3 (Alternative 3), A6, A8, and A11. Because of these topic changes there may be a reduced chance for these alternatives to be evaluated. In this case, A8 and A11 didn't receive any comments, and A3 only received one. Indeed, the lack of structure in the discussion sometimes prompts a participant to write a summary of the discussion to date, including the alternatives and opinions of those alternatives. One such summary is called out in the first thread in Figure 4. In this case, the participant wrote the summary mainly to compensate for the lack of awareness in the system. Twelve ideas were proposed and people were struggling with deciding which one works best. The summary reminded them of the goals and the description of each alternative. Another interesting pattern is that the majority of the participants joined during the first half of the discussions, but continued to join throughout. As the discussion grows, those who join later or otherwise do not keep up must review the messages to track the alternatives, the arguments for and against them, and the current consensus of the other participants. The common method for acquiring this information is to (re-)read the discussion to date. But, since this is time consuming, some participants will post irrelevant comments that hinder the flow of the discussion. As DD1 said: "one thing that gets very frustrating in this, it gets very frustrating when I'm involved in a long discussion and have been for the whole time, someone will often come in and just kind of jump in to the discussion and either drill it and say "Oh, this is such a great discussion I also noticed this other problem with this other issue or this other thing" and people will go off on a tangent for two weeks talking about this other thing and we've gotten away from what the core issue is which is can be frustrating."

To avoid losing bright ideas and to have a more organized discussion, the current systems should support better tracking of ideas. Highlighting the alternatives and connecting the messages that reference them can greatly aid participants. It could also reduce the time required to identify, compare, and consider the alternatives without having to sift through all of the textual comments in the discussion thread.

Integration of UI Design and Development Activities is Essential

Designers and developers currently participate in the discussions through a centralized issue management system. This centralized venue helps designers in building trust and gaining merit by enabling them to interact with developers and exhibit their skills. Once they gain respect as a designer, they can more easily convince developers to implement their suggested improvements [START_REF] Bach | Designers wanted: participation and the user experience in open source software development[END_REF][START_REF] Terry | Perceptions and practices of usability in the free/open source software (FoSS) community[END_REF][START_REF] Zilouchian | Open Source Interface Politics: Identity, Acceptance, Trust, and Lobbying[END_REF].

Integrating design and development activities also helps designers collaborate with developers. This collaboration is necessary for designers to receive feedback on the feasibility of their desired improvements [START_REF] Muller | Retrospective on a year of participatory design using the PICTIVE technique[END_REF] and for developers to be advised on the interaction design of their implementation. This iterative process of interface design and development is visible in current discussion threads. As shown in Figure 5 the proposed alternatives and submitted patches are distributed throughout the discussion threads, where each alternative is usually followed by a number of patches.

Any new interface for supporting interface design activities online should be fully integrated into the respective issue management systems. This will allow designers and developers to build mutual trust and collaborate more effectively.

Participants Need to Be Aware of Others' Opinions Regarding Alternatives

Discussion participants typically demonstrate agreement by writing "+1" for a favored alternative or they simply state that they like the idea. In order to determine the current direction and the favored ideas, participants must read through the messages. Another option would be to ask others to clarify the current direction.

However, participants may have inconsistent perceptions about the direction of the issue. As DD1 said:

"… It's often hard to figure out what is the current direction. That's definitely hard to do. Often it takes getting someone to clarify it. And not everyone would clarify it the same way. If there are two people, and they're each kind of pushing their own ideas, within a Drupal issue…and you were to go on IRC and ask each of them individually, "So what's the current direction?", you'll get two very different answers."

The current issue management systems lack a formal way of expressing one's preferred idea and visualizing others' preferences. The absence of a mechanism to share opinions can hinder the consensus building process. Today, the consensus building process can be lengthy and it can be difficult to determine whether consensus has been reached at all. As DU3 said: "People can keep on arguing the point, long after the decision was made […] The nature of the way that many online discussions work is that they let the discussion continue [indefinitely]. That's the main difficulty."

Implementing a mechanism to share preferences and formalize the consensus building technique (e.g. a voting system) may help facilitate the decision-making process. Also, it will be effective to highlight the alternative that has the consensus so far. Bringing the consensus to light can help developers determine which alternatives need to be implemented to improve the project or further inform the discussion.

IdeaTracker

We describe how we translated our implications into the implementation of an interactive visualization tool for reviewing online interface design discussions -IdeaTracker. Our tool can be used by designers and developers participating in a design discussion as well as facilitators who may join a discussion to facilitate the decision-making process. Our system was developed through an iterative design process, starting with four different prototypes that addressed the challenges identified in our study. An informal user study was conducted on these prototypes. For the study, each prototype was seeded with data from an actual design discussion. Four users representative of our target audience were recruited and asked to perform similar tasks (e.g., identify the idea that reflects community consensus) with each a b c prototype and the existing interactive forum interface. The users were then asked to explain the strengths and weaknesses of each prototype. From the results, we implemented our prototype of IdeaTracker. We first discuss the main interface components of our system and then illustrate its value through a user scenario. All of the figures illustrating the use of our system are based on data imported from an actual design discussion in Drupal where participants are proposing and debating alternatives for a revised password checker. To facilitate use and learning of the interface, all interactive controls in IdeaTracker have a tooltip which explains their functionality. For the visual elements, the user can access a short description of each element via a context menu.

Tracking Alternatives

IdeaTracker's main screen consists of an interactive visual timeline, highlighting the alternatives proposed in the current thread of discussion and offering an abstract summary of the posted comments (Figure 6). The timeline illustrates when a specific comment or alternative has been posted (Figure 6a). This timeline allows participants to gauge the amount of activity that has occurred within a specific timeframe as well as the overall progress and pace of the discussion. The alternatives are shown in separate callouts so designers can easily identify and track them (Figure 6b). If an alternative has an attached screenshot, the screenshot is shown in the callout; otherwise, the first few sentences describing the alternative are shown. All other comments are represented by a thin rectangle, with the width of the rectangle corresponding to the length of the comment (Figure 6c). The comments are colored based upon their affective tone. If a comment has a negative tone, the rectangle is colored red. If it has a positive tone, the rectangle is colored green. If the comment has both positive and negative words, then it is colored yellow. The number of negative and positive words is computed by looking up each word in a commonly used dictionary. This color coding allows designers to quickly assess the community opinion of a certain alternative. Designers can easily skim through the comments related to a specific alternative without having to read the text of each message. To aid in exploring alternatives and the comments regarding each alternative, three interaction mechanisms have been implemented:

Expand Alternatives and Comments: The user can select the expand/collapse button next to each alternative and read the entire post explaining the proposed alternative (Figure 7). Also, hovering over each comment representation will open a window containing the first few sentences of that comment (Figure 8).

Filter Unrelated Comments: To examine the comments related to a particular alternative, designers can press the filter button next to the alternative. This dims all of the representations that do not reference this alternative (Figure 9). This interaction isolates the pros and cons of an alternative pointed out by other designers. It also aids in detecting the alternatives that have received insufficient or controversial discussion.

Link to the Original Post: The user may want to read the original post corresponding to a comment or an alternative. To make this interaction possible, a link is provided in both expanded versions which redirect the user to the original post corresponding to that particular alternative or comment. Also, the title of the issue at the top of the main screen links to the original discussion thread.

Comparing Alternatives

To compare different ideas, IdeaTracker offers a comparison view. Users select the ideas they would like to compare by clicking on the check-box at the bottom of each idea. Then, selecting the compare link will redirect them to the compare view (Figure 10). This view shows a timeline for each idea and the representations of comments referencing those ideas are available on the timelines for comparison. To provide users with a reference point for comparison, all of the comments are shown under each idea. But, the comments that are not related to a particular idea are dimmed.

Voting System

A voting system has been implemented in IdeaTracker to aid designers in promoting and reaching consensus. The number of votes for each alternative is shown on the vote button next to the alternative. Hovering over the vote button will show the list of people who voted for the idea. The user can vote for an alternative by clicking on the vote button. If the user clicks the vote button, the number of votes for that idea will increase by one and the vote button will be highlighted to indicate which idea the user has voted for. To synchronize IdeaTracker with the original issue, a comment will be automatically posted to the original issue on behalf of the user stating that the user favors that particular alternative. Conversely, if a user posts a comment using the common notion of "+1" for an alternative in the original discussion thread, the number of votes for that alternative will be updated in the IdeaTracker.

Through IdeaTracker each user can only vote for one idea. If a user votes for a different idea, her initial vote will be re-assigned to the new idea. This feature enables users to retract their votes if a better idea is proposed or an existing idea is refined. This way IdeaTracker reflects the participants' recent views about the proposed alternatives. To promote awareness of the current consensus, the idea with the highest number of votes is highlighted in the system (Figure 11).

User Scenario

Bob is a UI designer who contributes to open source projects in his spare time. Looking through the usability issues in Drupal, he finds an issue about improving the usability of Drupal's password checker. He launches IdeaTracker and enters the URL corresponding to the discussion thread for that design issue.

He first wants gain awareness of the proposed alternatives and their pros and cons as pointed out by other participants. He quickly scans the list of alternatives highlighted on the main screen and notices the idea posted by Lisa that suggests borrowing the design of Google's password checker. Bob then expands the idea to read it in detail and selects the filter button to identify the comments that reference that idea. He immediately notices a negative comment colored in red and hovers the cursor over the comment to read it. The comment has been posted by Mark who thinks that copying and pasting a design from Google will spoil Drupal's trade mark. Bob agrees, so he continues to scan the alternatives to determine if someone else has proposed a better solution. At the end of the discussion, he finds Anne's idea. Anne's idea is a tweak of Lisa's idea. She suggests that instead of using red, orange, and green to indicate a weak, medium, and strong password, they can use different shades of green. Bob decides to compare Anne's and Lisa's ideas. He selects the compare checkbox next to these ideas, selects the compare link, and is redirected to the compare view. He reads the comments posted regarding each idea and compares them. He decides that he likes Lisa's idea more than Anne's. He returns to the main screen and votes for Lisa's idea. A comment is generated on Bob's behalf and posted to the original discussion thread indicating that Bob is in favor of Lisa's idea.

In contrast, using only the current interface, Bob needs to review each comment to identify the alternatives and their pros and cons. He then either needs to use another tool to create a summary of the discussion or rely on his memory. Comparing the two alternatives would also be challenging because it is difficult to isolate the comments that specifically address only the desired alternatives. Finally, since there is no running tally of "+1" votes, it is difficult to identify the currently favored idea.

Implementation

IdeaTracker is fully implemented and its interface was written in ActionScript using Adobe Flex 3 interface framework. The software consists of two layers: the data and visualization layers. The data layer parses the collected data and translates it into an internal format understood by the visualization layer. The data layer receives the data in XML format. The XML data consists of a set of <comment> tags, and each <comment> tag should have <author>, <content>, <date>, and <image> tags.

When the user launches IdeaTracker to view a particular discussion thread, an adaptor component parses the html source of the thread and converts it to the XML format readable by the data layer. The data layer is independent of the html format and only depends on the XML format. In order to apply IdeaTracker to design discussions on other interactive forums, an adaptor component needs to be written that translates the html source of that forum to the XML format readable by our tool.

The data layer processes the XML file to find all the posts. In order to find the alternatives we use two heuristics: (i) we consider posts with image attachments as alternatives and (ii) we consider posts that have been referred to by other posts as alternatives. Our testing indicates that these heuristics accurately detect most alternatives in a thread of discussion. In future work, discussion participants could be allowed to insert a simple tag in their comment stating they are posting an alternative.

After detecting the alternatives, we use natural language processing techniques to infer the comments related to each alternative and the tone of each comment. To find the comments related to an alternative, we look for certain key phrases participants commonly use to reference a comment, for instance: "#34" to refer to comment number 34 or "@Lisa" to refer to the latest comment posted by Lisa.

In order to determine the affective tone of a comment, we find the number of negative and positive words in the comment using standard online dictionaries. Based on the percent of negative and positive words, we assign respective values to that comment, which is used by the visualization layer for color coding.

Preliminary Evaluation

We conducted a qualitative evaluation to assess design choices and gauge initial user reactions to IdeaTracker. The evaluation was performed using the implementation of our tool as described in the previous section. It involved eight designers and six developers who actively contribute to interface discussions in Drupal and Ubuntu. The evaluation started with an introduction to IdeaTracker and a demonstration of its main features. Afterward we asked participants about their perceptions of the overall direction and different features of the tool (e.g. what do you think about showing the ideas in separate callouts, providing an abstract visualization of the comments, and using color codes for the affective tone of the comments?) and encouraged them to respond openly. Each session lasted about thirty minutes.

Overall, the participants reacted positively to IdeaTracker. All of the participants appreciated the visual separation of ideas from the other comments and being able to filter comments related to a particular idea. For example, one Drupal designer said: "The most useful feature to me is the callout of the major ideas that cuts through all lot of the crufty comments" while another said "I like this sort of compressing it... here is some big comments, here is a bunch of small comments, and if they are generally in favor or not, sort of at a glance as an overview is very cool." Most of the participants appreciated having access to an abstract visualization of the comments and felt we were using reasonable decision rules for identifying ideas and filtering comments. Participants also appreciated the fact that IdeaTracker was seeking to complement the existing issue management systems rather than trying to replace them.

The evaluation also highlighted several opportunities for improvement. For example, some of the participants were unsure of the utility of the idea-centric comparison view. Instead they preferred the ability to filter comments based on user id, thereby allowing them to see the comments that one user made across all of the ideas. Participants also expressed that the content of the negative and positive arguments for an idea was more important than the number of votes. For example, as one Drupal designer said: "Often there are issues though where an idea has lots and lots of "likes" until one person discovers why it shouldn't be done…" It may therefore be useful to extract the arguments for and against an idea and represent them within the main visualization. Most of the designers were concerned about accuracy of coloring comments based on affective tone and suggested to color comments based on their type (e.g. code review or patch). Participants also asked for more information to be included in the visualization of each comment (e.g. who posted the comment).

With the emergence of open source software and the geographic distribution of many design teams, more interface design discussions are occurring online. The discussions are carried out using typical interactive Web-based forums, which lack support for the unique nature of design discussions. This paper makes two contributions. First, we conducted a study examining the challenges faced by participants when using the current interactive forums. The study included analyzing the discussion content from two popular open source projects and conducting interviews with active participants in these projects. From the results, we identified key challenges of using these types of systems and implications for how they could be addressed. Second, we built a new interactive visualization tool called IdeaTracker to demonstrate how to manage online design discussion more effectively. The interface offers specific support for tracking alternatives, comparing alternatives, and building consensus, all of which were identified as significant challenges in our study but which are not supported by existing interactive forum models. The interface was also implemented such that it can collect actual data from existing discussion threads for several open source projects and can be adapted for others.

We see at least three promising directions for future work. First, we want to conduct a longitudinal study to compare the impact of our system on the process and outcomes of design discussions relative to the use of existing interfaces. Second, we would like to test different techniques for building consensus, for example, voting for versus ranking the ideas. Finally, we would like to integrate more sophisticated machine learning techniques to better identify key elements of the design discussions such as identifying the alternatives and the affective tone of a discussion.

Figure 1 .

 1 Figure1. A sample interface design discussion thread in Drupal. The interface design problem discussed in this thread is the lack of visual connection between a selected tab and its content. An alternative has been proposed along with the problem description. In the first few comments, participants are discussing the scope of this problem and whether it is a valid issue.

Figure 2 .

 2 Figure 2. Distribution of alternatives over each combination of modality, aggregated across all messages (N = 1560).

Figure 3 .

 3 Figure 3. Distribution of each coding category. Summing the categories for Alternative, Criterion, Implementation, and Clarification indicates that 48% of all discussion was devoted to alternatives.

Figure 4 .

 4 Figure 4. An event timeline for five complete threads of interface design discussion from the Drupal project. The timeline shows the fluidity of topics along with visual indicators of when new alternatives were posted and when new community members joined the discussion thread.

Figure 4

 4 Figure 4 demonstrates some of these challenges by showing an event timeline for five complete threads of interface design discussion from the Drupal project. Though we only show five, the other threads analyzed exhibited similar patterns. The timeline

Figure 5 .

 5 Figure 5. The timeline from Figure 4 showing only the introduction of alternatives and patches.

Figure 6 .

 6 Figure 6. The main screen consists of an interactive visual timeline, highlighting alternatives and offering an abstract summary of the comments. (a) The timeline shows the chronological order of comments and alternatives. (b) The alternatives are shown in callouts so designers can easily track them. (c) The comments are represented by a rectangle whose width corresponds to the length of the comment. The rectangles are colored based upon their affective tone.

Figure 7 .

 7 Figure 7. The user can press the expand button to reveal the entire post explaining the proposed alternative.

Figure 8 .

 8 Figure 8. Placing the cursor over a comment's representation will open a window containing its first few sentences.

Figure 9 .

 9 Figure 9. Users select 'filter' to dim comments unrelated to the alternative. Here selecting filter for the alternative in the top left shows one message referencing it in context of the discussion.

Figure 10 .

 10 Figure 10. The compare view shows a timeline for each selected alternative and the comments referencing them. All of the comments are shown under each alternative, but the comments unrelated to the particular alternative are dimmed.

Figure 11 .

 11 Figure 11. To promote awareness, the alternative currently with the most votes is highlighted.


Pablo Llinás

Manuel García-Herranz

Pablo A Haya

Germán Montoro
email: german.montoro@uam.es

Manuel Llinas

Pablo Garciaherranz

Haya

Unifying Events from Multiple Devices for Interpreting User Intentions through Natural Gestures

Keywords: FLING framework, Multi-touch interface, multiple input peripherals, application development

As technology evolves (e.g. 3D cameras, accelerometers, multitouch surfaces, etc.) new gestural interaction methods are becoming part of the everyday use of computational devices. This trend forces practitioners to develop applications for each interaction method individually. This paper tackles the problem of interpreting gestures in a multiple ways of interaction scenario, by focusing on the abstract gesture rather than on the technology or technologies used to generate it. This article describes the Flash Library for Interpreting Natural Gestures (FLING), a framework for developing multigestural applications integrated and running in different gestural-platforms. By offering an architecture for the integration and unification of different types of interaction, FLING eases scalability while presenting an environment for rapid prototyping by novice multi-gestural programmers. Throughout the article we analyse the benefits of this approach, comparing it with state of the art technologies, describe the framework architecture, and present several examples of applications and experiences of use.

Introduction

The evolving progression of HCI interfaces promises a brilliant future to seamless control and handling of intelligent devices. New ways of interaction are invented and integrated in the search for the best and most natural method of interaction between humans and computers. Their goal is to allow for a more intuitive and natural way of expression when communicating with computers. An immediate consequence of this evolution is the creation of new input devices with which we can operate. Today, intelligent homes can be fitted with all sorts of sensors (temperature, movement, pressure, fingerprint…). Also, existing devices are enhanced with extra sensing capabilities [START_REF] Lester | MobileSense-Sensing modes of transportation in studies of the built environment[END_REF], such as multi-touch screens, accelerometers and voice recognition in smartphones like Apple's iPhone or Google's Nexus S.

As computers integrate deeper into our daily lives, the most natural way to use them becomes dependent to each particular scenario. A single application will have to allow different interaction methods across different settings in order to offer the most suitable experience each time. For example, for talking to a friend, video conferencing will be suitable at home, where you can give visual and hearing attention to the other person, while text chatting will be more adequate in a football stadium during a match, where being heard can be very difficult.

From a programmer's point of view, more input devices entail a higher complexity when treating input. Different peripherals offer different interaction possibilities, which generate input events that need to be processed in order to interact with the applications. To reduce the difficulty of dealing with a large (and sometimes unknown) number of input devices, cross-platform frameworks are used to take care of these devices, and for developing applications using a set of abstract input events.

The difficulty of programming for cross-platform ubiquitous control has been shifted from treating each input device, to learning how to work with cross-platform frameworks. In a world of constant upgrades in hardware gadgets and increasing intelligent sensing devices, developers need unification and standardization of user intention recognition regardless of the technology employed.

Related work

The advantages of allowing the user to choose the most adequate input device for interaction with applications are therefore being studied and proven profitable. Multimodal interaction frameworks [START_REF] Dragicevic | Input device selection and interaction configuration with ICON[END_REF][START_REF] Flippo | A framework for rapid development of multimodal interfaces[END_REF][START_REF] Serrano | The openinterface framework: a tool for multimodal interaction[END_REF] pose valuable precedents of frameworks for using different input devices for human-computer interaction, using input events either independently or combined. However, we lack a higher degree of flexibility when translating device actions (fixed for each input peripheral) into application actions (which trigger the available operations on the computer). One programmer could require a sequence of device events to carry out an operation, while another would be looking for a different one. Moreover, different users using the same application may require different responses to similar input patterns.

This issue is of particular importance in multitouch environments in which events are usually composed of several device inputs (e.g. multiple fingers touching a display) that have to be, furthermore, interpreted from complex raw data, such as blob identification and analysis. In order to ease the process of interpreting device inputs in camera-based multitouch systems, several low-level frameworks such as TouchLib [START_REF]Touchlib: an opensource multi-touch framework[END_REF], reacTIVision [START_REF] Kaltenbrunner | reacTIVision: a computer-vision framework for table-based tangible interaction[END_REF], Community Core Vision (CCV) [START_REF]Community Core Vision[END_REF] or Touchè [START_REF] Touchè | [END_REF] are available to identify significant blobs as screen touches ready to process. This interpretation results, in most cases, in TUIO messages codifying each finger and its evolution on the surface. This kind of frameworks provides a first level of abstraction, allowing applications to listen from a single channel to every finger interaction in a unified manner. Advanced examples of these are BBTouch [START_REF] Bederson | J: Toolkit Design for Interactive Structured Graphics[END_REF] and LightTracker [START_REF] Gokcezade | LightTracker: An Open-Source Multitouch Toolkit[END_REF], which improve the aforementioned frameworks by allowing an advance tunning of the recognition parameters, or VVVV [11], which allows associating blob input with different visualization methods using a visual programming paradigm.

Nevertheless, multitouch interactions are most of the times composed of several fingers and, therefore, multiple events have to be reinterpreted, according to the element of the UI over which they are acting, to form a high level global gesture (e.g. two fingers moving are interpreted as moving apart from each other). Finally, the interpreted gestures have to be associated with a particular action. Some systems such as PyMT [START_REF] Hansen | PyMT: a post-WIMP multitouch user interface toolkit[END_REF] or Grafiti [START_REF] Nardi | Grafiti: Gesture Recognition mAnagement Framework for Interactive Tabletop Interfaces[END_REF] provide a transparent mechanism to distribute events among the elements of the UI as well as basic interpreters for the most common gestures such as move, resize or rotate.

However, the number of different gestures in multitouch systems can grow far beyond the basic ones and the actions associated with each of them may vary among applications or, in a single application, from component to component. Thus, a higher degree of abstraction, modularization and composition is needed. Systems such as Surface SDK [START_REF]Surface SDK[END_REF] or DiamondTouch SDK [START_REF] Esenther | DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications[END_REF] provide this kind of flexibility to a reasonable degree but are constrained to a particular platform, limiting their extensibility and scalability in an ever-increasing world of multitouch hardware solutions. Similarly, proprietary frameworks such as GestureWorks [16] constrain their extensibility, compared to their open source counterparts, in an ever-increasing world of gestures.

Nonetheless, while multitouch systems are gaining popularity, they are far from being an alternative to the standard mouse-keyboard paradigm and will have to further cohabitate with new interaction devices such as the Wii remote or Kinect. Thus, systems such as Squidy [START_REF] König | Squidy: a zoomable design environment for natural user interfaces[END_REF] provide a low-level alternative to unify various device drivers, frameworks, and tracking toolkits in one common library, overcoming the limitations of higher level solutions such as GestureWorks [16] or Grafiti [START_REF] Nardi | Grafiti: Gesture Recognition mAnagement Framework for Interactive Tabletop Interfaces[END_REF] designed to work just with multitouch events.

From the high level open source alternatives designed to support different input mechanisms, define new gestures and dynamically associate actions with them, we can distinguish between language dependent and language independent frameworks. Those that rely on a separated event system, allowing to program in the language of our choice, force the programmer to provide a description of the UI and its components to the event interpreter layer. This is done either explicitly, therefore, adding an extra complexity to the programming process that prevents a rapid prototyping, as in SparshUI [START_REF] Ramanahally | Sparsh UI: A Multi-Touch Framework for Collaboration and Modular Gesture Recognition[END_REF] and Midas [START_REF] Scholliers | Midas: a declarative multi-touch interaction framework[END_REF], or implicitly through a widget library, as in libTISCH [START_REF] Echtler | A Multitouch Software Architecture[END_REF], simplifying the communication channel but making very difficult to interpret events outside the boundaries of the widget. MT4j [START_REF] Laufs | MT4j -A Cross-platform Multi-touch Development Framework[END_REF] (MultiTouch for Java) is, on the other hand, a language dependent framework designed for rapid prototyping and gesture extension. Making use of the well-known event-listener java architecture, it allows components to listen to particular gestures without modifying or adding complexity to the UI design and coding.

FLING, the framework described in this article, falls into this last category but relies on Adobe's Flash, instead, as a well-known platform to graphical designers. Thus, graphic design and program logic can be easily separated and distributed among designers and programmers, allowing for good-looking rapid prototyping. In addition, contrary to systems such as MT4j, FLING provides a double distribution mechanism. Through one channel, interpreted gestures are propagated to every component, allowing them to know what is happening to themselves as well as to the rest of the components. Through the second one, raw events are propagated too, allowing to program global or partial interpretations in any component of the UI, whether they fall in or outside its boundaries.

3 The FLING Framework FLING (Flash Library for Interpreting Natural Gestures) is a cross-platform multigesture framework for developing Adobe Air and Flash applications using the ActionScript 3.0 programming language.

FLING has been developed under the following principles:

Platform-independent

FLING shares the "write once, run anywhere" philosophy. In order to run on the highest number of computing devices, a platform-independent programming language is a must. ActionScript 3.0, the language behind Adobe Flash and Air applications, was chosen for this purpose. Applications run on any desktop operating system (Windows, Mac and Linux) and also on Android smartphones. A special version, Flash Lite, can even be used on more basic mobile phones [START_REF] Blom | Write Once, Run Anywhere A Survey of Mobile Runtime Environments[END_REF].

We chose this way of deploying cross-platform applications instead of using adapted interfaces because we believe that technologies such as Java, HTML and Adobe Flash, which in the past have been secluded to being used on standard PCs, will soon be running with the same capabilities on even the most basic portable devices. The processing gap between different computing architectures is closing-in, and today we can find smartphones 1 , tablets 2 , notebooks 3 and desktop computers sharing very similar dual-core processing computational power.

The Adobe family of products provides a very powerful and robust set of tools for the visual design of graphical interfaces. Also, when integrating these graphical elements into the Flash platform for adding programming logic, there is full compatibility and interoperability between multimedia contents. A drawing made in Photoshop can be imported into Illustrator to get a vector graphic that then can be inserted into Flash and get animated. The final symbol can be accessed and manipulated from code using the ActionScript language. And finally, the result will be an Adobe Air or Flash multi-platform application.

Useful for rapid prototyping

For first-time cross-platform multi-gesture application developers, FLING provides a basic manipulation of visual objects using the traditional mouse and keyboard, and multi-touch surfaces. Extending from the base FLING object class, object movement, rotation, resizing and physics engine (inertia, collisions, gravity…) can be enabled with a single line of code, as seen below. The visual symbol of class "NewObject" will respond according to the activated capabilities upon standard input events. In the case of the mouse and keyboard, the object will respond as shown on Table 1. Using a multi-touch surface, the object will respond as shown on Table 2.

This initial functionality allows for basic application interaction without getting into event interpretation or gesture handling. It is oriented towards Adobe Flash and Air developers with minimal knowledge in device input processing, and allows them to create cross-platform multi-gesture applications which can run on multi-touch surfaces and standard computers without making the effort of learning a complex new framework.

FLING requires little adaptation for first-time programmers. One of the basic fundaments of its operation is a tree-structured hierarchy of symbols or visual objects (Figure 1). A unique FLING root object, representing the application itself, contains all device parsers and gesture interpreters. All symbols used in an application must be children to this root object (either directly, or further down in the tree of child objects). This is required because it represents the order in which visual objects are layered in an application. The root object is the background of the application, and it represents the lower-most layer. The next level in the hierarchy of children objects represents the layer on-top of the background layer, and so on. The FLING framework relies on Flash's native visual hierarchy of objects for target identification, allowing disambiguating when objects overlap. As all Flash objects have one (and only one) parent and their insertion order decides among siblings, no inconsistencies can occur between FLING's object targeting and Flash's visual representation. Using the native structure for object nesting presents a logical way to navigate through objects and can, therefore, be already found in most applications.

Allows customization of event interpretation and triggered actions

To extend the default functionality offered as standard when working with FLING, developers can tune and enhance the interpretation of device events into gestures, and the reactions upon receiving interpreted gestures. This can be achieved with minimal code modification because of the predisposition to customization.

The three main elements with which FLING works are events, gestures and actions. Events are signals from input devices that are processed and homogenized into a common stream of events by the device parsers. Each device has an associated event parser that in some cases makes use of external drivers to capture input signals.

Gestures are interpreted by another module which receives events parsed from all the input devices available to the user. The interpreter can use the combination of these input events and knows of all existing interface objects in order to make sense of the user's intentions. It outputs FLING Gestures, which consist of recognized user intentions. FLING is then responsible of propagating and making gesture events reach their correct target. Once an interactive object receives a gesture event (or FLING Gesture), it reacts according to the triggered action for that gesture. Some triggered actions are preconfigured by default (as is the case of the move, rotate and resize actions mentioned in section 3.2) and others are left blank, but all of them are customizable to fit the needs of each application.

As an example of triggered action customization, we will take the resizing gesture and change its default action so that instead of changing size, the target object will change its transparency. The code added to the target object for modifying this behaviour is as follows:

Example of triggered action customization override public function onRotateGesture(gesture:FlingGesture):void{ this.alpha += gesture.varAngle%360; } Normally, the "varAngle" property of the FLING event received is used to rotate the object accordingly. Instead, we are using this value, normalized between -1.0 and 1.0, to alter the alpha value of the object, hence changing its transparency. This is just an example of how easy it is to change the behavior associated to a particular gesture.

Another example of framework customization can be seen when modifying the way device events are interpreted into gestures. We will take the multi-touch gesture interpreter and add some code to recognize a new gesture. The new gesture will consist in a quick slide of two fingers (cursor1 and cursor2) over a visual object, from top to bottom. This gesture will be called the "minimize" gesture, and could be linked to the minimizing action, but this is entirely up to the programmer.

The code needed inside the finger interpreter to recognize this gesture is the following:

Example of gesture customization if(numCursors == 2){ if(cursor1.type == cursor2.type == CURSOR-EXIT){ if(slideDirection(cursorStream1) == slideDirection(cursorStream2) == SLIDE-DOWN){ flingGestures.push(new FlingGesture("MINIMIZE")); } } }

Cross-platform multi-gestures

Another requirement for ubiquitous control is the possibility of multiple device input (Figure 2). FLING comes with parser modules for mouse & keyboard, multi-touch surfaces (such as Microsoft's Surface4 , MultiTouch Cell5 and the reacTable [START_REF] Jordà | The reacTable: exploring the synergy between live music performance and tabletop tangible interfaces[END_REF]), pressure tokens (special objects recognizable by pressure marks) and fiducial markers [START_REF] Bencina | The design and evolution of fiducials for the reactivision system[END_REF]. These input devices can be used right away, and can be and customized to fit the application's needs. For advanced necessities, it also allows for new input devices to be used when needed. The steps for adding a new input device are:

1. Identify the input data format in which the device sends events from the user's interaction. If not provided by the standard Adobe Flash / Air libraries, parse the device signals into atomic non-interpreted events. These atomic events should match the device's interaction possibilities. For example, a joystick will have coordinate events indicating its position, and button events indicating any change in the state of their manipulation. 2. Add a gesture interpretation module for the new input device, which groups atomic input events into recognized gestures. For example, using multi-touch surfaces, two fingers separating from each-other over a same object are recognized as a resize event by default. Gesture metrics (e.g.: resizing value) are also included in the interpreted gesture. The gesture interpretation module sends recognized gestures, called FLING events, as shown on Figure 1, which can be equivalent to those already handled, or new ones which are exclusive to each input device. 3. FLING will forward both atomic (or raw) events and gesture (or FLING) events to all objects, as described in section 3.2, for them to react consequently. All of the aspects of input device reading and interpreting can be openly customized to obtain the required functionality, as stated on section 3.3.

Multi-touch surfaces are one of the input devices currently by FLING (Figure 3). On a typical multi-touch table, fingers placed on-top of the touch panel (A) create light blobs that can be tracked by a video camera [START_REF] Wang | A Robust Blob Recognition and Tracking Method in Vision-Based Multi-touch Technique[END_REF]. The camera sends a video stream (1) to a blob driver (B), such as reacTIVision [START_REF] Kaltenbrunner | reacTIVision: a computer-vision framework for table-based tangible interaction[END_REF], Touchlib [START_REF]Touchlib: an opensource multi-touch framework[END_REF] or Community Core Vision [START_REF]Community Core Vision[END_REF], which then outputs finger events using the TUIO protocol [START_REF] Kaltenbrunner | TUIO: A protocol for tabletop tangible user interfaces[END_REF] through a data socket [START_REF] Dragicevic | Input device selection and interaction configuration with ICON[END_REF]. FLING (C) connects to this data socket and uses the incoming finger events from the panel to begin the interpretation and propagation processes (3). Finally, the application reacts to the intentions expressed by the user, and evolves its visual interface (4) which is displayed on the same surface used as touch panel (D). Inside the FLING framework, a parser module reads unprocessed finger events obtained from the hardware drivers, and homogenizes them into standard input device events. This homogenization consists in a translation of native signals to a common event class, which holds the same information but in a standard form. Then, the finger interpretation module is in charge of interpreting and generating gesture events that FLING will send to the appropriate application object. Interactive objects will react in response to these gesture events, and will also receive the unprocessed input events in case more information is required. The format of gestures produced from a multi-touch surface is compatible with those generated using mouse, fiducial or token devices. Interactive objects from the application receive complete and descriptive gestures with the user's intentions, but don't have to worry about the input used to express those intentions. Additionally, unprocessed events from the input devices are also delivered to application objects for the event of needing detailed information about raw input data.

Progressive learning curve

The idea behind offering both rapid prototyping and advanced customization capabilities is to create a progressive and smooth learning curve when working with the framework. Existing cross-platform frameworks offer extensive functionality and very advanced user expression recognition, but are difficult to use at first and require a lengthy learning period. We experienced these difficulties when trying commercial products such as Gestureworks [16] or the TUIO Flash client library [27] for receiving multi-touch events. It is normal to go through a number of learning steps while getting comfortable using a new framework, and a learning curve similar to the one described by Gaines [START_REF] Gaines | A learning model for forecasting the future of information technology[END_REF] is typically experienced.

Sample Applications

FLING has been thoroughly tested and used for the development of full applications which have made their way into educational and experimental projects. Different working environments with adapted input devices have accommodated these applications, and people with varying levels of knowledge and expertise have given them a try.

Therapy applications

To serve as a complement for people with disorders, games such as Simon ® and Gesture Hero (Figure 4) were developed under particular requirements. They were designed in cooperation with teachers and assistants of people with Down syndrome and Alzheimer's disease to serve as therapy work for these collectives.

This supervised use has served to gather information about interaction habits and difficulties that has been used in FLING's design. Specific gestures were added to the Gesture Interpreter module in order to supply these applications of their interaction requirements.

Educational applications

In addition to the therapy applications, other educational applications have been developed to be included in the regular activities of people with Down syndrome during their classrooms. Examples are the Postman Game and the Price is Right game shown on Figure 5.

The Postman Game consists in handing packages to the right recipient using the same procedure as real postmen do in our university. The user needs to search for the correct floor, office or desk assigned to the recipient from an address file. Then she must drag the package on to the correct mailbox. This application was made to train future university postmen, and it resembles the real procedure faithfully.

The Price is Right game simulates a typical trip to the cafeteria. The waiter offers the user a product at random, and informs her of its price. In the wallet the user has money represented by actual photographs of real Euro bills. She must drag money from the wallet onto the waiter's plate, and the waiter will give her the right change in return. Upon analysing the payment, the game awards the user with a rating which will be better as the payment gets closer to the right price. The change from the previous purchase is used to continue throughout the game. These games are oriented to desktop computer usage, although they have also been tested on multi-touch tables with real-case users. This was possible thanks to the variety of input devices supported by FLING, which made it easy to shift from one platform to another without any modification.

Data visualization and control applications

One of the great advantages of using a development environment as Adobe Air / Flash is its inherited multimedia capabilities. Video and audio content can be accessed in many ways, and FLING adds a rich interaction experience that helps in the complex task of data visualization of large collections of multimedia content.

For these reasons, FLING was employed at Carnegie Mellon University to create a video discovery application that used a geographical information system based on Google Maps TM . Maps can be browsed with multi-touch gestures, and embedded videos can be opened and played with full timeline controls, which are easily triggered with finger gestures. The typical hardware setup for this application is a vertical multi-touch panel on which the graphical interface can be projected and the interaction is executed using the hands. This application is an example of different interaction techniques seamlessly combined through FLING. While the Google Maps TM API listens to mouse events, the multitouch panel generates TUIO events. In addition, the rest of the elements of the interface, such as the opened videos, listen to finger events. FLING integrates all the interactions seamlessly, significantly reducing the programmer's effort. Google Maps TM are integrated naturally, as they do in traditional PC applications, and all the functions of their API can be used directly out of the box. The intelligent environment control application is meant to be run on a multi-touch table that accepts fiducial marker interaction. These fiducial symbols can be placed over appliances drawn on the house map to control some of their properties. For example, the intensity fiducial marker can be placed over lamps, and can be rotated clockwise to increase light brightness, or anti-clockwise to reduce it. A selection fiducial can be placed over the TV to change the channel. This application was developed to show how new input devices (fiducials in this case) can be easily added to FLING, and how its functionality is automatically incorporated to the existing gestures.

Conclusions and Future Work

This paper has focused on the problem of developing applications for an increasing number of interaction mechanisms. In doing so, we have stressed the necessity to distinguish between events, gestures and actions. Events depend on the interaction mechanisms, gestures represent the users' manipulation intentions and actions depend on each particular application.

Following this distinction we have developed FLING, a framework that allows:

• To easily build interaction independent applications without having to deal with interaction events. • To build new complex gestures combining events from multiple interaction sources.

This framework is designed to be:

• Scalable: so that new interaction mechanisms and gestures can be added and integrated with existing ones. • Multi-platform: so that it can run in the varied number of platforms in which the new interaction mechanisms are emerging.

This framework has been tested, seamlessly and over a number of platforms, through a number of applications using both traditional mouse and keyboard, as well as novel multi-touch interaction mechanisms. This has been achieved through a modular design, separating the parsing of each input device from the gesture interpreter process. Thus, new sensors can be added by just incorporating the corresponding parser to the framework.

We are currently working in extending the interaction experience through new interaction mechanisms and merging the interaction of multiple interfaces.

Finally, working in the Ambient Intelligence domain, we look forward to enriching the interactions with context-aware information in a multi-user, distributed, intelligent environment.

  1 http://www.pcworld.com/article/204947/lg_announces_smartphones_with_dualcore_processor.html 2 http://www.tgdaily.com/mobility-features/49854-nvidia-showcases-dual-core-tegra-2-tablet 3 http://liliputing.com/2010/10/samsung-launches-nf310-dual-core-netbook-with-hd-display.html Example of extending from the base FLING object class public class NewObject extends FlingObj{ public function NewObject():void{ movable = resizable = rotatable = physics = true; } }

Fig. 1 .

 1 Fig. 1. Event parsing and interpretation inside the FLING framework.

Fig. 2 .

 2 Fig. 2.A multi-platform running environment and multi-gesture input interaction enable applications run on the device which most suits each scenario.

Fig. 3 .

 3 Fig. 3. Multi-touch surface input handling using the FLING framework.

Fig. 4 .

 4 Fig. 4. On the left, the Simon® game, a memory training application. On the right, the Gesture Hero game, a psychomotor skill game.

Fig. 5 .

 5 Fig. 5. On the left, the Postman Game, a package delivery simulator. On the right, the Price is Right, a money management game.

Fig. 6 .

 6 Fig.6. On the left, a video discovery application using GIS developed at the Instinctive Computing Lab on Carnegie Mellon University. On the right, a control application for an intelligent room at the Universidad Autónoma de Madrid.

Table 1 .

 1 Default actions triggered by mouse and keyboard input events.

	Input device gesture	Action performed on object
	Drag with mouse	Object moves under mouse pointer
	Control key + Drag with mouse	Object resizes in regard to its center and the mouse pointer
	Shift key + Drag with mouse	Object rotates in regard to its center and the mouse pointer

Table 2 .

 2 Default actions triggered by multi-touch finger input events.

	Input device gesture	Action performed on object
	Slide one finger over the object	Object moves according to the finger movement
	Pose two fingers over object and	Object resizes according to distance variation between
	separate or join them	fingers
	Pose two fingers over object and	Object rotates acording to angle variation of line joining
	move one around the other	fingers

http://www.microsoft.com/surface/

http://multitouch.fi/products/cell/

Acknowledgments. This work has been partially funded by the following projects: ASIES: Adapting Social & Intelligent Environments to Support people with special needs (Ministerio de Ciencia y Educación de España, TIN2010-17344), and Vesta (Ministerio de Industria, Turismo y Comercio de España, TSI-020100-2009-828).